Alert to Incident - Insert a 180 second delay to check if the alert closes with a clear event.

[dan_tembe]

Hello Folks, 

I am trying to figure out where & how I could implement the following behaviour in Event Management. I feel I have the basics down, but now as I get into very specific handling of events to alerts to incidents, I am running into a lot of questions. Hence the questions - 

Behaviour -

I have implemented various tools to feed events into SNOW EM. The events turn into alerts based on specific rules and then into Incidents based on Task templates and alert action rules. 

As I audit the incidents, I am seeing a high number of incidents are resolving (that is the expected behaviour in EM Properties set by me), within 180 seconds.

What I want to do - 

I am thinking about implementing a delay between the alerts to incidents workflow of 180, so the alert closes and the incident is never created if it is in closed state. 

What I know -  (apparently very little 🙂 )  

There are Business rules that can be executed on the em_alert table.

There is a script - EvtMgmtIncidentHandler - that creates incidents from the alert. There are other EvtMgmt scripts but I have only used the Custom Incident Poulator to change some display behaviour in the Incidents. 

My thought - 

I would update the EvtMgmtIncidentHandler script to execute only if the alert is 180 seconds past the "initial event generation time" if state is "new" or 180 seconds past the "updated" time if the state is "Reopen".

Then again I am not sure what impact this would cause to any other workflows so concerned. I would rather get some opinions before attempting something that could have adverse effects to Incident handling. 

My Question - 

Maybe you have some other ideas or have a way you handle recurring up/down or error/clear alerts without creating new incidents or new/resolve/new cycling of the same incident. 

Where would I go about implementing this delay / timer value so that before the alert passes the rules to create an incident, wait for 180 seconds to check if the alert closes on its own. If it is still in the same state (open) then create Incident. 

Also, How would I do implement this logic? 

Thanks in advance for your help & responses.

Regards,

Dan

[robertgeen]

Alright Dan I'm going to give out a bit of a trade secret here. Stay away from the em_event table all together for doing this as you can cause a lot issues. Your requirements is one of the #1 requests I have been implementing for Event. To do this you will need to add a custom field to the alert table in which you set a wait timer via event rules. After that you need to clone the scheduled job that creates the incidents so that you have one for normal alerts and one where the wait timer is set so that it looks back at the create time and makes sure that the amount of time passes before you open the incident.

This should get you going on this.

[vadimshif]

Hi

There is a property used in Incident Creation flow. Exactly delay that you spoke about: "evt_mgmt.alert_rule_delay" - seconds to wait before Alert Action rule run on updated alerts

[robertgeen]

Is this a global setting that would affect all alerts? I don't get why these exist but aren't documented at all on the doc website. Also is it in every version or was it introduced in just 1 version?

[robertgeen · Accepted solution]

Sorry answered my own question. Just a note this is only available in Kingston and Jakarta and it's not in previous versions.

if (hashGr.next()){
timestamp = hashGr.getValue('hash'); //get last calculated timestamp
var inProgress = new GlideDateTime();
var secondsBack = gs.getProperty("evt_mgmt.alert_rule_delay", "5");
var gt = new GlideTime(1000 * secondsBack);
inProgress.subtract(gt);
var openStates = ["Open", "Reopen", "Flapping"];
var recentAlerts = new GlideRecord('em_alert');
recentAlerts.addQuery('sys_updated_on', '>=', timestamp);
recentAlerts.addQuery('sys_updated_on', '<=', inProgress);
recentAlerts.addQuery('state', 'IN', openStates);
recentAlerts.query();

This is from the scheduled job so it is indeed global. If you want to make it more flexible you will have to do what I said but if it's for all sources then you should be set to use this.