From Alerting to Observability: Building a ServiceNow Reliability Engineering Dashboard

Jeff Boltz1 · 5 hours ago

From Alerting to Observability: Building a ServiceNow Reliability Engineering Dashboard

Over the past week, I set out to answer a simple question:

How do we know ServiceNow is healthy before users tell us it isn't?

What started as a review of MID Server monitoring evolved into the development of a ServiceNow Reliability Engineering Dashboard built around observability and Site Reliability Engineering (SRE) principles.

The Journey

The initial objective was to improve operational monitoring of our ServiceNow environment.

As part of that effort, I reviewed ServiceNow sysevent activity and evaluated candidate signals that could provide meaningful indicators of platform health, including:

mid_server.down
mid_server.up
glide.heartbeat
notification_engine.process
email_diagnostics.process
Import processing events

Rather than focusing exclusively on error events, I wanted to identify signals that could be used to measure normal system behavior and detect when expected activity stops.

The Most Interesting Discovery

The standout event was:

glide.heartbeat

A seven-day analysis revealed:

Approximately one heartbeat per minute
Nearly perfect consistency
Roughly 1,440 heartbeats per day
Continuous 24x7 activity

The only significant interruption occurred during a planned ServiceNow maintenance window, where heartbeat activity stopped for approximately 34 minutes before resuming normally.

That immediately raised an interesting possibility:

Could the absence of heartbeat activity become a platform health indicator?

Monitoring by Exception

Rather than alerting on failures alone, we began exploring "silence monitoring."

Examples:

No heartbeat for 10 minutes
No scheduled import completion within an expected window
No notification processing activity

The goal is to detect degradation in expected operational behavior before user-facing impact occurs.

Introducing the ServiceNow Reliability Engineering Dashboard

To support this effort, I built a Splunk-based dashboard focused on ServiceNow reliability and observability.

Platform Health

ServiceNow heartbeat continuity
Heartbeat trend analysis
Event-processing health

Availability

MID Server operational status
MID Server lifecycle events
Current platform availability indicators

Import Health

Scheduled import activity
Time since last successful import
Data freshness monitoring

Notification Health

Notification engine activity
Email diagnostics activity

Reliability Indicators

Failure event trending
Authentication failures
Import failures
Execution tracking failures

Applying SRE Concepts

Although this started as a monitoring exercise, it naturally evolved into an observability discussion.

Potential Service Level Indicators (SLIs) identified include:

MID Server Availability: Availability of production MID Servers
Platform Heartbeat Continuity: Ongoing glide.heartbeat activity
Import Processing Health: Successful execution of scheduled imports
Notification Processing Activity: Ongoing notification engine activity
Operational Failure Rate: Failure event frequency

Potential Service Level Objectives (SLOs) are now being discussed around:

MID Server availability
Heartbeat continuity
Import freshness
Notification processing health

Lessons Learned

A few takeaways from this effort:

Not all monitoring signals are errors.
Silence can be a valuable monitoring indicator.
Operational dashboards should focus on service health, not just event volume.
SRE concepts translate surprisingly well to ServiceNow operations.
A reliable heartbeat event can provide a powerful foundation for platform-level monitoring.

Looking Ahead

Next steps include:

Platform heartbeat missing alerting
Scheduled import silence monitoring
Notification processing silence monitoring
Additional SLI/SLO development
Expansion of ServiceNow observability practices

I'm curious how others are approaching observability within ServiceNow.

What events, indicators, or SLIs have you found most useful for measuring overall ServiceNow health and reliability?