How Can I See Current Performance Stats for ServiceNow?

[GTSPerformance]

< Previous Article		Next Article >
Performance Maintenance and Administration		Maintaining Task table Growth

---

This guide is written by the ServiceNow Technical Support Performance team (All Articles). We are a global group of experts that help our customers with performance issues. If you have questions about the content of this article we will try to answer them here. However, if you have urgent questions or specific issues, please see the list of resources on our profile page: ServiceNowPerformanceGTS

The goal of this article is to share some methods that can be used to see the overall performance and health of your ServiceNow instances.

1. Impact Instance Observer

Instance Observer (IO) is an observability and performance monitoring tool that empowers you to keep track of your instance health and performance in near real-time, while also providing historical insights.

Overview

IO helps Instance Administrators, Platform Owners, and DevOps teams accelerate value and drive their desired business outcomes by increasing visibility into instance performance, helping customers push innovation with speed and confidence.

Key Features

• 24/7 Monitoring: Instance performance monitoring provides awareness of issues within an instance before end users  are impacted.
• 24/7 Alerting: Set up custom alerts to notify you when specific thresholds are crossed, so you can take action before an issue becomes critical.
• Multi-instance Availability: Proactively visualize performance  telemetry across instances. Empowering  organizations to identify trends before  end users are impacted.
• Triage: Actionable point-in-time  instance metadata providing additional  context to instance performance and accelerating troubleshooting.
• Off Instance Application: Access to instance performance  telemetry off instance. When an  issue does arise, organizations can  access actionable data independent of  instance availability.
• Reporting: Visualize instance performance  telemetry over time and quickly generate reports to share with your business stakeholders, making  organizations faster, smarter, and better in their  instance performance management.
• Analytics: Compare metrics over time to see how performance is trending in general or to assess the before and after impacts of certain changes.

2. Monitoring XMLStats directly

Another possibility is to set up monitoring against the /xmlstats.do page. Note that running /xmlstats.do requires the use of a semaphore and it is advisable not to run this more than once a minute per node. In addition, you should use the "include" argument to only get the critical information you need to monitor. 

Some examples of the include argument:

xmlstats.do?include=system

• includes all the following more specific includes:
• system.memory
• replication
• transactions
• scheduled workers
• jvm
• discovery
• client metrics

xmlstats.do?include=servlet

• servlet.uptime: milliseconds since node restart
• servlet.metrics:
• <transactions> timing information for UI Pages, Lists, Forms, redirects (e.g. service_catalog.do). N.B. does not include SOAP, REST, (including all ServicePortal), AJAX, Angular, etc.) 
• <sql_response> how long Java thinks* SQL is taking per query in milliseconds
• <job_times>job durations per span (1-minute, 5-minute, 15-minute)
• <processed_jobs> jobs processed per second
• ...lots more...

xmlstats.do?include=diagnostic_events

• plugin.activation – a plugin was activated
• cache.flush – all caches were flushed
• Node Start – current node restarts
• updateset.commit – update set commited on this node
• low.memory.remediation – node was running low on memory, could trigger transaction cancellation or restart

xmlstats.do?include=scheduler

• Background jobs (sys_trigger table)
• Each node claims ≈200 “Ready” jobs per 10 sec
• 8 “workers” (threads) per node process queued jobs
• queue.length: current number of jobs in queue
• mean.queue.age: average age of jobs in the queue – not including jobs that are currently running
• <scheduler.workers> status of each of the 8 “workers”
• current.job: Name of job running right now
• total.jobs: jobs processed on this worker since last restart
• mean.duration: average job processing duration since restart

xmlstats.do?include=semaphores

• max_queue_depth: highest peak queue depth ever
• name: name of the Semaphore “pool” (e.g. Default)
• queue_depth_limit: 150 for Default pool and 50 for other pools
• rejected_executions: How many transactions failed due to queue depth limit
• <semaphore>
• @age: milliseconds this transaction has been occupying the semaphore
• @processor: thread number

---

< Previous Article		Next Article >
Performance Maintenance and Administration		Maintaining Task table Growth

[ITSM Dev]

Very cool, thank you for this info!

BTW a detailed master article on finding the cause behind instance slowness would be great. There are these old articles like KB0516495 that have a lot of info but are useful only to ServiceNow internal support people who have command line access to the background DB. Besides that the article largely discusses the use of the performance graphs which provide an overall view of the system status but help very little in pinpointing the exact cause in a slowdown.

[Joel L]

On NowLearning there is a course Performance Troubleshooting Overview which helps on some of the questions mentioned in ITSM Dev's comment.

[GTSPerformance]

Note regarding Xanadu Feature Deprecation:

• The 'System Performance Dashboard' will be deprecated.
• 'Application Insights' is being prepared for future deprecation. This uses MetricBase database and 'is' available via the Store to give customers improved features above what the Performance Dashboard could.

Instance Observer will be the main source of performance metric information for our customers. Instance Observer is currently available for Guided Customers and upwards.

Xanadu Deprecation Notes - LINK
Impact Instance Observer Docs - LINK

[dlentz]

The xml content returned from /xmlstats.do?include=..... does not include the W3C standard methods to describe the data in the XML. For example XSD (XML Schema Definition) or even the old DTD standard. I've searched everywhere for a technical description of each element returned by xmlstats.do but have only found posts in developer forums and old KB articles which discuss one or two elements.

I appreciate the high level description you've provided of some xmlstats elements in this article. Is there an official comprehensive list of each xmlstats element and their description?

[GTSPerformance]

Unfortunately, we aren't aware of any comprehensive documentation at this time.

[Robin26]

For customers who don't have Impact product, is there a similar offering to the Instance Observer? 

And if we were to create PA dashboards ourselves, any suggestions as to what metrics to use?

[GTSPerformance]

Hi Robin, unfortunately I am not aware of any product similar to Instance Observer at this time. I believe some customers have had some success by building metrics based off the syslog_transaction table. Note that this is a huge table and the data needs to be aggregated very carefully to avoid creating data or performance issues. Also, some customer have built their own dashboards by consuming the metrics from the XMLStats page. I've mentioned some of the useful metrics from the XMLStats page in my article above. In fact, Instance Observers gets much of its information from the XMLStats page. It should also be noted that consuming the XMLStats page is a non-trivial operation and care should be taken not to cause performance issues.