Alert to Incident - Insert a 180 second delay to check if the alert closes with a clear event.

dan_tembe · ‎03-07-2018

Hello Folks,

I am trying to figure out where & how I could implement the following behaviour in Event Management. I feel I have the basics down, but now as I get into very specific handling of events to alerts to incidents, I am running into a lot of questions. Hence the questions -

Behaviour -

I have implemented various tools to feed events into SNOW EM. The events turn into alerts based on specific rules and then into Incidents based on Task templates and alert action rules.

As I audit the incidents, I am seeing a high number of incidents are resolving (that is the expected behaviour in EM Properties set by me), within 180 seconds.

What I want to do -

I am thinking about implementing a delay between the alerts to incidents workflow of 180, so the alert closes and the incident is never created if it is in closed state.

What I know - (apparently very little 🙂 )

There are Business rules that can be executed on the em_alert table.

There is a script - EvtMgmtIncidentHandler - that creates incidents from the alert. There are other EvtMgmt scripts but I have only used the Custom Incident Poulator to change some display behaviour in the Incidents.

My thought -

I would update the EvtMgmtIncidentHandler script to execute only if the alert is 180 seconds past the "initial event generation time" if state is "new" or 180 seconds past the "updated" time if the state is "Reopen".

Then again I am not sure what impact this would cause to any other workflows so concerned. I would rather get some opinions before attempting something that could have adverse effects to Incident handling.

My Question -

Maybe you have some other ideas or have a way you handle recurring up/down or error/clear alerts without creating new incidents or new/resolve/new cycling of the same incident.

Where would I go about implementing this delay / timer value so that before the alert passes the rules to create an incident, wait for 180 seconds to check if the alert closes on its own. If it is still in the same state (open) then create Incident.

Also, How would I do implement this logic?

Thanks in advance for your help & responses.

Regards,

Dan

robertgeen · ‎03-07-2018

Sorry answered my own question. Just a note this is only available in Kingston and Jakarta and it's not in previous versions.

if (hashGr.next()){
timestamp = hashGr.getValue('hash'); //get last calculated timestamp
var inProgress = new GlideDateTime();
var secondsBack = gs.getProperty("evt_mgmt.alert_rule_delay", "5");
var gt = new GlideTime(1000 * secondsBack);
inProgress.subtract(gt);
var openStates = ["Open", "Reopen", "Flapping"];
var recentAlerts = new GlideRecord('em_alert');
recentAlerts.addQuery('sys_updated_on', '>=', timestamp);
recentAlerts.addQuery('sys_updated_on', '<=', inProgress);
recentAlerts.addQuery('state', 'IN', openStates);
recentAlerts.query();

This is from the scheduled job so it is indeed global. If you want to make it more flexible you will have to do what I said but if it's for all sources then you should be set to use this.

View solution in original post

roshan17 · ‎06-06-2018

Can you please help me with the exact path to "evt_mgmt.alert_rule_delay" property setting?

I am not able to find it under Event Management >> Alert Aggregation & RCA >> Properties.

Thanks,

roshan

roshan17 · ‎06-19-2018

Hi @vadimshif @robertgreen

Can you please help me with the exact path to "evt_mgmt.alert_rule_delay" property? I am not able to find it under Event Management >> Settings >> Properties.

ayush_saxena · ‎08-30-2018

@roshan you would have to introduce the property manually, if property is not found it takes the value as 5.

Regards,Ayush

dan_tembe · ‎03-07-2018

Thanks! @robertgreen and @vadimshif for your quick and accurate response. I am reviewing the scheduled job "Event Management - create/resolved incidents by alerts" so I can understand the usage of this property.

Best regards,

Dan