Scheduler performance metrics
Summarize
Summary of Scheduler performance metrics
Scheduler performance metrics provide a snapshot of how scheduled jobs are handled within the ServiceNow AI Platform®. These metrics help monitor the efficiency and health of job processing across application nodes, ensuring timely execution and identifying potential performance bottlenecks.
Show less
Key Metrics
- Completed Scheduled Jobs: Tracks the number of scheduled jobs completed per node. These jobs include scheduled reports, script executions, SLA or workflow timers, email sending, event processing, and asynchronous business rules that run on the scheduled job queue. A sudden increase in job counts may overwhelm processing capacity, causing lag.
- Scheduler Mean Queue Age: Measures the average time unprocessed jobs spend in the in-memory scheduler queue after being claimed by a node. A high mean queue age suggests the node is struggling to process jobs promptly, possibly due to long-running jobs occupying worker threads. Since the New York release, nodes can return unprocessed jobs to the systrigger table for other nodes to claim if they cannot process them timely.
- Scheduler Queue Length: Indicates the number of claimed scheduled jobs waiting in each node’s in-memory queue, excluding currently executing jobs. Sustained high queue length and increasing queue age on a node can signal processing issues. If multiple nodes exhibit this behavior, the instance may be overloaded.
Practical Guidance for Customers
- Monitor the volume of completed scheduled jobs to detect unexpected spikes that may impact system responsiveness.
- Investigate long-running scheduled jobs to optimize their frequency, performance, or stagger execution times to reduce queue age and avoid blocking worker threads.
- Track scheduler queue lengths and mean queue age per node to identify and address nodes struggling to keep up with job processing.
- Utilize the job return functionality (available from New York release onwards) to enable load balancing of unprocessed jobs across nodes.
- Coordinate these metrics with related performance data such as job details, node health, and database performance to comprehensively address scheduling issues.
The metrics provide the Schedulers performance snapshot within the ServiceNow AI Platform®.
Completed Scheduled Jobs
The number of scheduled jobs completed per node.
There are many types of scheduled jobs. They could be scheduled reports, scheduled script executions, one-time jobs like SLA or Workflow Timers, or repeating jobs like Email Sender or Event Processor. In addition to what you might think of as traditional scheduled "jobs" there is also a couple special case of operations that run on the scheduled job queue called, Asynchronous business rules. These can be work that was triggered from the UI or an integration but runs as a scheduled job to achieve asynchronous behavior. Typical examples of Asynchronous business rules are Discovery Sensors or perhaps a job that iterates over a large number of child records, making so many updates that it would not be a good user experience to have to wait for them all to complete.
- When scheduled job counts suddenly shoot up this can be concerning as it may overwhelm processing and cause lag.
- This should be considered together with other job metrics like the scheduler age, scheduler queue length, and pending jobs counts.
Scheduler mean queue age
The average age of all unprocessed jobs in an application node in memory scheduler queue (i.e., the delta between the job being claimed by the node or placed in the queue and the point at which metrics were collected). Note that this does not include time the job spent pending in the sys_trigger table between its 'Next Action' time (that is, when it should have run) and when it was claimed.
A large scheduler mean queue age can indicate that an application node is struggling to process claimed jobs (that is, those in its in memory queue) in a timely manner.
Scheduler queue length
The number of claimed scheduled jobs in each application nodes in memory scheduler queue. Note that this does not include scheduled jobs which are currently executing on a worker thread on the application node.
It is common for application nodes to claim multiple jobs at once and therefore have a significant scheduler queue length at any point in time. If a single node has significant queue length for a sustained period (that is, more than 60 minutes), where queue length does not drop down to 0 and scheduler age is increasing, this may indicate the node has an issue processing scheduled jobs.
If multiple nodes see similar issues it may indicate either all nodes are having issues processing scheduled jobs or the instance is simply overwhelmed by scheduled jobs.
- Check scheduled jobs executing on application nodes to determine type or source.
- Address performance or scheduling of jobs as necessary to prevent them overloading worker threads.