Welcome back to the series: Journey through Event Management, Part Deux! Two weeks ago, we started with the basics of ServiceNow Event Management: the definition of events & alerts; how event rules can be used; and finally the concept of the message key for Event Consolidation. To refresh your memory, last time we covered the left-hand side of the below diagram (feel free to refer to the post here: Journey through Event Management - Introduction of Events.
This week, we will explore Alert Rules in more depth. Specifically, how they can automate task generation and remediation (the right side of the diagram, as highlighted).
From our previous discussion, we know that alerts are potentially actionable items. To define what "actions" can be taken when, we use "Alert Rules". Generally speaking, these are a few things you could do with them:
1 Create a task - such as an incident, problem, change, security incident, or any task within ServiceNow, including your own extensions of the task table.
2 Trigger remediation workflow through ServiceNow's workflow engine, including Orchestration.
3 Assign Knowledge Base articles.
4 Create a launcher link to quickly route back to a web UI of the monitoring tool.
First, let's talk about creating tasks. Not only could you create an incident, which is the most common choice, but you could also create any type of record that extends from the task table. This means that you could automatically create a change request or a security incident based on the Alert Rules defined. Further, you could employ what we call a "task template" to automatically enhance the task you create with additional information, such as assigning the task to the right user or group. Most importantly, this is all part of the ServiceNow platform! That means that you could start leveraging other ServiceNow offerings such as notification (e.g., email or the Notify solution), Livefeed (collaboration tool), Chat (instant message), On-Call Rotation, SLA, and so on, to ensure that the appropriate users are involved, that all the relevant information is captured, and that real-time updates are displayed.
Second, you could trigger a remediation workflow (set to run automatically or manually) based on a given alert. The remediation workflow is simply a regular ServiceNow workflow where you could define business process (e.g., notification, approval, etc). If you own Orchestration, this opens up even more possibilities. You can define technical activities such as rebooting a server, restarting an application, running a pre-defined script, or making a web service call to directly interact with the infrastructure components. In other words, this allows you to combine both the business processes and technical capabilities in the same workflow. For example, to restart a server, depending on the role of the user or the function of the server, approval might need to be obtained and a change request created. It's a powerful concept, and ServiceNow is the only vendor that can offer both in one single integrated platform.
Third, for alerts that have certain standard operating procedures that ought to be followed, associating a Knowledge Base article is a great way to help a NOC technician quickly follow instructions that are appropriate for the alert.
Fourth, for monitoring tools that have some type of web accessible URL, we can construct a "launcher" that will give the NOC technician a quick way to go back to the source of the alert. This URL construction process is all based on information that can be extracted from the alert -- whether it is a simple URL value or the concatenation of various event values, such as the IP address, port, or the name of the CI the alert is bound to.
Now that we've gone through what we can do conceptually, let's take a look at an actual example to see how this can be done.
In the screenshot above, you can see this alert rule is triggered when
• The severity of the alert is critical
• The CI associated with the alert is a server
• The alert is not a secondary alert
The first two conditions are fairly self-explanatory, the third is perhaps not so obvious. In ServiceNow Event Management, there is a concept of "Alert Correlation" that will group alerts together. For example, an alert on a VM and an alert on a hypervisor (like an ESX server) should probably be correlated — it's likely that the alert on the hypervisor is a primary alert and the alert on the VM is a secondary alert (assuming the VM is running on that hypervisor). When the alert is secondary, we don't want to create an incident on it; instead, we want to only create the incident on the primary alert. This is why we have a condition in place to prevent a secondary alert from creating an incident. We'll explore Alert Correlation more in a future post.
Now, once the alert is triggered based on the condition, the sections called "Action", "Remediation", and "Launcher" will come into play. Let me highlight each section below.
ACTION SECTION
• You can see from the screenshot that the Knowledge Article field is where you can define the Knowledge Base record to be associated with the alert.
• The Auto Open checkbox will enable the ability to create any task based records. In this case, I've chosen to create an incident from a Task Template called "Incident Template for Event Management". Using Task Templates, you can define the value for fields on the task record. For example, you might want to use a template that always sets the priority to high or the assignment group to a specific team for a particular alert.
• In addition to the Task Template, there is also the ability leverage scripting to do more complex operations. For example, you might want to dynamically set the incident assignment group to the same group as the CI's support group. This can be achieved through a script include called "EvtMgmtCustomIncidentPopulator." We'll explore this further another time.
REMEDIATION SECTION
In this section, you have a checkbox to enable some type of remediation workflow. Additionally, you could choose to run this automatically or simply expose it as an option for the technician to run it manually. In this example, it will run an Orchestration workflow to increase the memory capacity. Keep in mind that the workflow can be about both the process and the technology. Simply put, it can be as straightforward as creating a task for someone, or as complex as actually running a command/script to accomplish a task.
LAUNCHER SECTION
Finally, in the Launcher section, you can define a dynamic URL for a web-based monitoring tool based on the alert details. This will come in handy in the alert console or event dashboard, where by right clicking on an alert, an option will be available for the technician to view the source monitoring tool's web interface.
Alright folks, this wraps up our discussion on Alert Rules. Hopefully this has been valuable in expanding your understanding of what Alert Rules can do. Next time, we will delve into the "Alert Correlation" capabilities that ServiceNow offers!
