How can I see statistics (xmlstats.do & stats.do) for all my nodes?

Mwatkins · ‎08-28-2015

ServiceNow's backend infrastructure uses a load balanced cluster of "nodes" (JVMs) to scale application processing. For the ServiceNow administrator one thing that can be a bit challenging is to understand the status of all the nodes in their cluster. All ServiceNow users (including admins) can only be logged into one node at a time and can't control what node they are logged into. This makes it difficult to diagnose issues that are occurring on other nodes. For example, multiple users might start to complain about slow load times and the administrator might want to quickly see if any of the nodes are running short on resources. One thing that can help to see the status of all nodes at the same time is the "System Diagnostics" homepage.

The "System Diagnostics" homepage comes out-of-the-box with ServiceNow and already provides many key metrics useful for quickly assessing instance health - like Java heap and permgen memory, MySQL threads, pending events, etc. To make the "System Diagnostics" homepage even more useful it can be extended to provide other useful pieces of information. There are endless customizations you could make to this homepage but for this post I will show you how to add semaphore stats. (To understand more about some of the important metrics track performance in ServiceNow check out https://docs.servicenow.com/bundle/kingston-platform-administration/page/administer/core-configurati...)

The "System Diagnostics" homepage is created from a widget called "Diagnostics". It picks up any UI Page whose name starts with the prefix "diagnostic_". The first gauge on the "System Diagnostics" homepage is the "Cluster Node Status". It is created by a UI Page called "diagnostic_cluster_nodes" and contains, as one might expect, the status of each node in the cluster. The way the UI Page works is that it pulls the whole /xmlstats.do page into an object created by the "Diagnostics" script include. It then selects the items from that object that it wants to use. Since semaphore stats are part of xmlstats we've just got to grab the items we want out of the Diagnostic object and add them to the output object.

To add Semaphore Availability and Queue Depth metrics to the "System Diagnostics" homepage:

1) Open the UI page called "diagnostics_cluster_nodes"

2) In the HTML field (directly before the line "var ss = diagNode.stats.sessionsummary;") add the following code:

var semaps = diagNode.stats['semaphores'];

for (var idx in semaps) {

var semap = semaps[idx];

var poolName = "Semaphores [" + semap["@name"] + "] ";

addValueToJSON(vals, poolName + "Available Semaphores", semap["@available"]);

addValueToJSON(vals, poolName + "Queue Depth", semap["@queue_depth"]);

}

3) Click "Update"

NOTE: If you don't want to alter the out-of-box UI Page what you could do is make a copy of the page and give it a slightly different prefix (e.g. "my_diagnostic_cluster_nodes") and then also copy the widget called "Diagnostics" and just change line 11 that adds the UI Pages to include "my_diagnostic" as the prefix instead.

There is a Product Documentation page about the System Diagnostics homepage: https://docs.servicenow.com/bundle/kingston-platform-administration/page/administer/platform-perform....

Mwatkins · ‎10-30-2015

Here's a screenshot of what it looks like:

Some I showed this to took the principles here and displayed all the scheduled jobs that were currently running. You can display anything here that is shown in the XML stats page (xmlstats.do)

Mwatkins · ‎11-19-2015

If you want to add a section with the status of all your scheduler jobs, including what job is running, how long it has run, and when it started use the following:

//Custom section for scheduler workers

           var workerpool = diagNode.stats['scheduler.workers'];

           for (var idw in workerpool) {

               var details = [];

               var worker = workerpool[idw];

               for (var wDetail in worker) details.push(wDetail + ": " + worker[wDetail]);

               addValueToJSON(vals, idw, details.join(",   "));

           }

It will output results like the following for each node:

Mwatkins · ‎03-30-2016

One more thing. Some of the diagnostic values turn red when a certain condition is met. This can also be customized. The way to do it is to create (or update) one of the sys_properties records that start with the prefix "diagnostics.condition.". The "value" field of these properties expects a javascript snippet that when evaluated will return true or false if the given diagnostic matches the condition.

So, for the sake of demonstration, suppose I create a sys_properties record with name "diagnostics.condition.system.memory.in.use" with value "> 200". In reality you would want to set the threshold much higher (like 1800 or 90%) as 200 MB in use is very low - only 10%. Before anything will happen I need to set up the Cluster Diagnostics page to pass in my new diagnostic condition value. To do this I change line 79 of UI Page "diagnostic_cluster_nodes" from:

79: addValueToJSON(vals, "Memory (MB)", diagNode.stats['system.memory.in.use'] + " of " + maxMemory);

to:

79: addValueToJSON(vals, "Memory (MB)", diagNode.stats['system.memory.in.use'], jelly.jvar_diag.conditions['system.memory.in.use']);

Then, if Java heap memory goes over 200MB the diagnostic output will turn red like this:

DrewW · ‎09-08-2016

I notice on my instance which is running Helsinki the UI Page is called "diagnostic_cluster_nodes".