Event Management - Alert closed but Incident opened and is not set to resolved.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-09-2018 08:27 PM
Hello Folks,
I have a question around the alert action rules or Alert to Incident process. I have PRTG integrated into ServiceNOW EM. The events are coming in correctly and are using the correct message_key to open an Alert and then close it. I saw something unique happen yesterday where a PRTG event came in which created and alert and that was a critical so it matched a rule to create an Incident. This worked as designed. There was also a Clear event that came in at the same time the Incident was created. In Seconds, it was the same time. So the Alert closed correctly but the Incident did not go to resolved state as defined in the Event Management Properties.
I am assuming this is because the rule to create an incident might have executed at the same time the clear (OK) event was passed to the alert table. In fact, it almost looks like the OK alert came in and then the Incident was created.
Is there a way, in your environment you are seeing this type of issue happen. I have seen this in the past (rarely) but didn't focus much on it until now.
One thought I had was to add another condition in my alert action rule to make sure the state of the alert is not "closed" before it executes the create incident rule. Not sure if there is any impact to doing this.
Pls. do help me with your input /guidance. Thanks in advance.
Dan
- Labels:
-
Event Management
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-10-2018 04:27 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-10-2018 10:27 PM
Hello Ofer.
I understand this does look like a RACE condition.
I am not sure if you are planning to follow up with this internally with your team, or with me for specifics.
Also, I am assuming you will respond with a solution or will let us know via a community post (if) you choose not to handle this case.
In the meantime, I am testing adding additional conditions such as Alert state has to be "open" & "initial event generation time" values before an Incident is created.
Please do clarify. Thanks! in advance for your help & support.
Regards,
Dan
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-24-2022 03:19 AM
Hi Dan,
Did you find any solution to above issue.
TIA.
Aakanksha B
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-27-2022 06:56 AM
Hello Aakanksha,
Unfortunately, I did not follow through on this one to see if it was fixed.
We actually moved to using the PRTG connector that is maintained by ServiceNow, which has different concerns but I don't recall ever running into this issue again.
Based on the issue we were seeing this would be happen to almost any other tool that was generating volumes of events and were getting processed into Alerts to Incidents, esp. if the state changes at alert levels were happening fast, so I would assume this is resolved. I haven't run into this over the last couple of years for any other high volume event platforms integrations into SN ITOM EMM.
Thanks!
Dan
Dan