Event Management - Auto Close Alerts

[Jonny Lord]

Hi Community,
Forgive the newb question please 🙂

We have just had Service Now Kingston installed and have various stream of the implementation project working in tandem. I currently have the event mgmt stream and have a question that i cannot work out the answer to.

I have successfully configured Service Now to receive events from Orion Solar winds and OEM.
Issue
Orion sends an event in that says 'SERVER XYZ has stopped responding' that is then raising an alert in the console.
The events state then changes to Processed.

SERVER XYZ is then rebooted and an event is received into ServiceNow saying 'SERVER XYZ Rebooted @ 10PM'
AND/OR
10 minutes later, a new event is received in ServiceNow to say 'SERVER XYZ is responding again'

How do i tie that back to the Open Alert in the Console that then auto closes and clears it once the reset has been received or once the device is responding again?

My Alert console is littered with open Alerts where resets and devices back online events were received days ago.
Hope that makes sense!
Thanks

[robertgeen · Accepted solution]

Hello Jonny,

There are 2 ways to do this. The first thing you need to understand that if the monitoring source is sending a clear message then what matters is that the message_key that is generated matches on the alert that is already there and it should auto close it. Out of the box if you aren't setting a message key it will concatenate Source, Type, Node, Resource, and Metric Name to make one. The second thing that can be done is if it doesn't typically send a clear but resets back to info or something and there is another attribute which dictates that it's resolved then you can have the monitoring endpoint set the Resolution_state attribute on the event to Closing and this will auto close the alert it matches too.

Hope this helps :).

[Jonny Lord]

Thanks Robert for taking the time to reply,

I just looked into the 'All events' view and none of the events have a Message_Key, that column is blank.
However in the alert console, every one has a message key. If an event record is received stating a reset/reboot has taken place OR the Server is 'responding again' but those are not generating an alert or linkage - Is this the Problem?

Basically what just happened as i was typing this was;

An event came in for a server node not responding, this raised a critical alert (link can be seen in the Alert Column)
2 mins later, the server started responding again and the event came in to say so, however no Alert record was generated or linked.

I'm failing to see how, when the second event comes in to say 'responding' it closes out the original critical alert it raised moments earlier.

There seems to be a disconnect somewhere.
Added info: Im in the DEV instance using complete out of the box rules.

[robertgeen]

So what's happening in this case is that if it's coming in with a clear severity the system shouldn't open up a new alert with that (which is why you aren't seeing one there is code that checks to see if on create the severity is clear it will just garbage it). What's happening here is that more then likely your node field or type are different. Can you check to see if those are the same? If so it should be creating the same message key on alert creation and it should match and close out the already open one. 

Any chance you can provide some screenshots of both of the events and I can probably tell you the problem.

[robertgeen]

Jonny,

Everything end up working out alrgiht? Let me know if I can be anymore of a help and if things are fixed please flag the answer in this thread so others can easily access it. Thanks.