Event Management source Push Heartbeating

[Jim Palmer]

Those of you who are fans of the graphic novel 'Watchmen' or Terry Pratchett's book 'Thud' would recognise the phrase "Quis custodiet ipsos custodes?" 
Who watches the watchmen?

In this case it's ServiceNow.

In previous lives, I've implemented this solution in varying contexts and for other event management tooling for the simple reason of being able to determine when the monitoring source we rely on for operational visibility no longer work.

ServiceNow has out of the box solutions for Self-health monitors but this is specifically for any event integration where the events are sourced from a 3rd party as a push. In this case, if the push connection goes quiet, unless you're paying close attention to the event console, you won't know something's gone wrong until a service manager asks 'where was my incident?"

This is where the push heartbeating solution comes in.

The Event – Push Heartbeat provides automated heartbeat monitoring for ServiceNow Event Management Push Connector instances. It verifies that each configured push connector instance is alive and operating correctly by processing heartbeat events emitted from those connectors, recording their last-received timestamp and creating incidents when a connector becomes unresponsive.

Business need solved:

• Provides continuous health visibility of Event Push Connector instances without requiring manual checks.
• Automatically correlates connector heartbeats with the connector instances & CMDB CIs to identify which monitored assets are affected.
• Automatically raises Incidents via Alert Management Rules so operations teams are notified of connector failures, reducing blind spots in our monitoring ecosystem.

This is a post of the solution I've used, feel free to take, extend or otherwise develop it in the way that you want. You could extend this to other solutions or integrations where a feed is triggered externally to ServiceNow.

The key to this solution is an M2M table that links a pull connector instance with the associated metadata about that connector and it's matching CI record. 

Table [evtpush_hb_m2m_cmdb] 

Column name	Type	Reference Table	Notes
sn_em_connector_push_instance	Reference	sn_em_connector_push_instance
cmdb_ci	Reference	cmdb_ci	Links connector to monitored CI - can be the event management service or the Service instance of the monitoring solution
last_heartbeat	Date/Time	—	Updated by flow on each OK heartbeat
threshold	Integer	—	Configurable alert threshold in seconds
severity	Choice	—	Health status from[em_event.severity]

Each connector instance (you can setup direct to connectors [sn_em_connector_listener] rather than the instances, but I usually have a lot of different sources to deal with) that you want to heartbeat is listed in the M2M

table with three additional fields: last_heartbeat, threshold and severity.

This allows you to individually configure what idle looks like, the CI that's relevant (for the support_group and impact assessment) and how important this connector is to your business (also to disable this specific heartbeat).

Inputs to this table is managed by each monitoring source instance pushing specially crafted events [1] on a schedule via the connector instance [2] and an event rule parsing [3,4] that event into a consistent format to trigger an alert management subflow [5,6] to update the above table [7].

Very straight-forward. The challenge is the event rule and the format of the heartbeat event. You can define a consistent normalised payload format for each of your sources and have a single event rule (e.g. metric=EMSelfMonitoring, event_class=<push instance label>...)  OR setup individual event rules for each source and normalise the alerts using those rules. Make sure you set the order as one of the first rules and do not apply additional rules to speed up event processing.

The normalised attributes are critical to match the heartbeat events and failure events.

Here's a sample of the normalised event attributes:

[source] == EMSelfMonitoring

[event_class] != null (usually the connector instance label)

[resource] == <the connector type>

[metric_name] == Heartbeat
[message_key] == EMSelfMonitoring_${event_class}_${resource}_Heartbeat

[severity] == 5 (OK)

As long as the alert that is created form those event rules is normalised consistently, you can apply a single Alert Management rule filter to trigger the "Record Heartbeat" subflow to update the M2M table values [7].

E.g.

metric_name=Heartbeat^severityIN5,0^event_classISNOTEMPTY^source=EMSelfMonitoring

"Record Heartbeat" subflow 

This is triggered by the Alert management rule to update/create records in the M2M table.

After setting flow variables from properties (see below), as long as the alert is still active, lookup the M2M matching on the connector instance name and definition (from the [resource] & [event_class] in the normalised event) to find an existing record.

If it doesn't exist, check whether we have enabled auto-creation and setup a new M2M record.
If it does, update the [last_heartbeat] with the current Date/Time & check if the threshold was breached (E.g. this heartbeat is a recovery from an outage) and create a 'clearing event' for the instance.

Then close this existing alert - to allow us to re-trigger on the next heartbeat.

Note: the 'clearing event' is to reset any existing push heartbeat incidents generated by the "Event Push Instance - Heartbeat monitoring" flow below and this must follow the normalised heartbeat attributes and message key as mentioned above.

"Event Push Instance - Heartbeat monitoring" Flow 

Generate events for idle push connector instances.

This is scheduled to loop through the M2M table records looking for any record whose [last_update] + [threshold] is greater than Now() && whose Severity NOT IN 0,5.
* By using severity, we can turn off individual heartbeat triggers by setting the [M2M.severity] to 0 or 5 

The event created references the CI record in the M2M table and it's associated [cmdb_ci.support_group] so make sure this is correct and populated properly!

You will ned to create specific Event rules and Alert management rules for these connector instance heartbeat failures so you can raise incidents.

Additions:

I've used system properties to control some of the functions of this solution:

• enable.pull.connector.heartbeat.events=true (completely disables this action)
• auto.create.heartbeat.records=true (automatically creates a M2M record on receipt of a matching heartbeat event)

You could also add domain awareness to take advantage of the 'GlideApplicationProperty' function to get and set application property values for a specific domain.

Conclusion

The major moving parts in this solution are the following:

• M2M table to store the in-scope connector instances their metadata and last heartbeat time.
• The Heartbeat processing event and alert management rules that manage the inbound heartbeat events.
• "Record Heartbeat" subflow. Updates the status of the push connectors in the M2M table.
• "Event Push Instance - Heartbeat monitoring" Flow. Generates events for idle connector instances.
• The Heartbeat failure event and alert management rules - to generate the idle connector events as a actionable incident.

The place where this can trip up is the event normalisation between the inbound Heartbeat processing events that update the M2M and the Heartbeat failure connector events and associated clearing events. 

The Event/Alert rules and the events created by the flows need to meet these rules and match up correctly for each connector instance, otherwise you'll have uncorrelated alert records for each connector instance being monitored.