Operational process question - Does your NOC watch the Alerts queue or Incidents queue?

[dan_tembe]

Hello Folks, 

FIrst off, Thanks to this community, I am learning a lot about ServiceNOW ITOM. It is certainly helping me transition my skills from other tools to understand the logic and similar logic in SNOW ITOM - more specifically the Event Management part. 

So far in our Dev, when I am testing ServiceNOW event management, using rules, I am able to reduce over 90% of the noise of events into alerts and about a similar % of reduction into Incidents.

I have created task templates to create Incidents based on alert severity or other matches. My biggest concern is that we might miss creating an incident on something and it will get unnoticed until escalated from the asset owner. As with any new tool in the environment, if we have significant misses in the beginning, we will not get complete adoption, so trying to make sure operationally we have the right process in place. 

This brought the question in my mind, in your operations (NOC) do your teams work from the alert console or incident queue?

is there a method that I should look into developing - such as review the alerts every hour but work from Incident queue? Or part of the team works from Incident queue and part reviews the the alert console?

Not sure there is one size fits all answer, but want to understand how others are operating so we can come up with something that will work for us. 

Thanks in advance for reading this and your valuable input. 

Dan

[stevemacamway]

We are still relatively new into Event Management, but we are aiming towards our Operations folks watching the alerts. using the EM Dashboard (out of the box) will give a view to the overall health of an application (either Service Mapped or Manual Service Group), including those alerts that don't result in an Incident. 

I'd be interested to hear other's (experienced) approach to this as well. 

Steve

[dan_tembe]

Hello Steve, 

Thanks for this response. It is good to get a second perspective. On our end, we are not in production with Event Management so we have some time to run reports and understand the alerts that really need an Incident or need to be worked by a NOC engineer. 

Right now, I am evaluating the Alerts daily, creating/ adjusting the Incident task templates to ensure all alerts that need an incident are passing through to the Incident queue. Also, I am looking at auto resolved (for tools that send resolution alerts) Incidents which resolve in 180 seconds, which if possible just not create an Incident in the 1st place. 

It is all uncharted territory so work in progress. Good Luck in your project. 

Thanks!

Dan

[stevemacamway]

I think I saw your other post about 180 second 'delay' on opening Incidents. I'll be interested in seeing what comes out of that. One of our monitoring tools is set up with a lot of email notifications long before they want Incidents being generated for them, and I'd like to be able to consolidate that into ServiceNow instead of dealing with that in multiple monitoring tools. With what you are looking to do, I could use the same idea to implement this. 

We're actually having a philosophical discussion in our group about how to deal with alarms / events that don't have a 'clear' event generated, and whether we should be auto-closing them. (We've turned off the 'Auto-close interval' for Alerts in the EM Properties.) Right now in our main monitoring tool, we auto-close an alarm that hasn't repeated in 4 hours, but haven't integrated that to ServiceNow yet, and need to decide how we are going to handle that since we don't want it sending 'clear' events over. 

Steve

[robertgeen]

You are quickly becoming one of my favourite question askers around here. This is a very interesting question that really realize on the size and the structure of your organization. For larger companies that have their Operations and Support teams broken out there can often we a tier 1 NOC operator who deals with alerts and opening incidents to the proper groups. 

I will say many of the customers I have worked for on the event management side have been small to medium size companies and usually the investigator and fix-it person are one in the same so they tend to focus on the incident queue and briefly look at the alert side of the house. I also think it depends on how your alert to incident process is configured and whether you are auto-opening most of your incidents.

Either way I'm sure someone can chime in with a more ITIL appropriate answer but I think it largely depends on your structure and how you currently operate.