What is the self-health monitoring feature?
Enabling the OOTB available self-health feature can help detect and troubleshoot Event Management health issues. Some examples below may help you further understand this better before we dig into this a little more -
Example 1: A classic example of what a self-monitoring alert would look like. As you can see, this is an alert created when the system observed an issue with MID Server availability or when resource thresholds are detected in the ServiceNow Environment. In the example below, you can see the MID server was not able to connect with the service bus and hence there were two events generated against the alert.
Also, you can see the default values are populated like CI which is the MID server, metric name along with a description of what the issue is in depth.
Example 2: An auto-created alert that shows that there is a delay in the impact calculator job, as you can see it already has a configuration item pre-selected, type as self-monitoring, and source as EMSelfMonitoring.
On the ‘ServiceNow Event Management’ Application service you can see all the CIs which are related to the event management module.
Source: ServiceNow Documentation
How to enable/disable this feature?
This is available OOTB as a feature, it just needs to be enabled/disabled via a property -
System Property - evt_mgmt.self_health_active
The property can have one of these values -
Enabled - The ServiceNow Event Management application service is created and set to monitor self-health. Events are received and alerts are generated. The first time that the self-health monitoring property is enabled, it automatically creates the ServiceNow Event Management application service.
Alerts Only - The ServiceNow Event Management application service and the CIs related to it are not created. Alerts are created but are not bound to a CI.
Disabled - The ServiceNow Event Management application service is not operational. Events are not received, and alerts are not generated.
What are the OOTB self-health monitors and what if I want to create a custom monitor definition?
All the definitions that determine which Event Management components are being monitored are stored in the ‘em_monitor_conf’ table. Below are the listed jobs available out of the box -
|
Sr No |
Name |
Description |
|
1 |
Connector's Idle state monitoring |
This monitor verifies whether any of the connectors was in idle state that elapsed the configured thresholds [minutes] |
|
2 |
Connectors Status |
This monitor tracks the active status of the connectors. Select a severity to invoke. |
|
3 |
Count of non updated alerts |
This monitor tracks the update states of alerts. |
|
4 |
Delay in event processing |
This monitor tracks the duration [minutes] of events that remained in 'ready' state and were not processed. This duration initiates an alert for the 'event processing' CI with severity according to the configured thresholds [minutes] |
|
5 |
Event Processing Job |
Monitors the state of the event processing jobs. Produces an alert when any of the jobs idle time (in seconds) elapses the configured thresholds |
|
6 |
Impact Calculation |
Status of impact calculation. Select a severity to invoke. |
|
7 |
Max Delay Of Impact Jobs |
Monitors the state of the impact of jobs. Produces an alert when any of the job's idle time (in seconds) elapses the configured thresholds |
|
8 |
MID Server Threshold Alerts |
Monitors MID Server health. Events are sent when issues with MID Server availability and resource thresholds are detected. |
|
9 |
Monitor all event management jobs |
This configuration comes with the base system. It monitors all of the jobs in the em_monitor_jobs_state table. |
You can configure a self-health monitor to track Event Management components and see that they do not exceed the specified threshold.
To augment the self-health monitors that are provided with the base system, you can create a self-health monitor to monitor the components that you specify. You can create a custom script to run when called from a monitor. You can use the health monitor scripts provided with the base system as examples of how to author the script and use the script editing tools. Ensure that the custom script includes:
- Run the monitor function, which is called according to the "monitoring job frequency” setting runUpdateHealthMonitorScripts(scriptId). Add this function to run the monitoring logic.
- Specify the configuration function that is called every hour runInsertHealthMonitorScripts(scriptId). Add the configuration logic and add a CI if needed. In the navigation pane search field, enter em_monitor_state.list. In the Monitoring States [em_monitor_state] table, specify the initialization of the monitor record in the required record.
- Use the helper functions defined in EvtMgmtHealthMonitorCommon and EvtMgmtEventAlertGenerator script includes:
- EvtMgmtHealthMonitorCommon
- shouldRun
- calculateSeverity
- EvtMgmtEventAlertGenerator
- openAlert
- closeAlert
I will soon be creating another blog showcasing the steps on how to create a custom self-health monitor script
Schedule jobs to process scripts
The following jobs are scheduled to run by default after the Event Management - self-health monitoring property is enabled:
The Event Management - Insert Health Monitor scheduled job runs the scripts and determines what must be monitored by adding or modifying the script in this job. This job runs once every hour. After the job has run, you can view the ServiceNow Event Management application service which will have all the listed Cis which are getting monitored.
The Event Management - Update Health Monitor scheduled job runs the update scripts. This job runs once every minute.
Thanks
-Harneet Sital
Find all my ServiceNow articles here
