How can I see statistics (xmlstats.do & stats.do) for all my nodes?

Mwatkins
ServiceNow Employee
ServiceNow Employee

ServiceNow's backend infrastructure uses a load balanced cluster of "nodes" (JVMs) to scale application processing. For the ServiceNow administrator one thing that can be a bit challenging is to understand the status of all the nodes in their cluster. All ServiceNow users (including admins) can only be logged into one node at a time and can't control what node they are logged into. This makes it difficult to diagnose issues that are occurring on other nodes. For example, multiple users might start to complain about slow load times and the administrator might want to quickly see if any of the nodes are running short on resources. One thing that can help to see the status of all nodes at the same time is the "System Diagnostics" homepage.

 

The "System Diagnostics" homepage comes out-of-the-box with ServiceNow and already provides many key metrics useful for quickly assessing instance health - like Java heap and permgen memory, MySQL threads, pending events, etc. To make the "System Diagnostics" homepage even more useful it can be extended to provide other useful pieces of information. There are endless customizations you could make to this homepage but for this post I will show you how to add semaphore stats. (To understand more about some of the important metrics track performance in ServiceNow check out https://docs.servicenow.com/bundle/kingston-platform-administration/page/administer/core-configurati...)

 

The "System Diagnostics" homepage is created from a widget called "Diagnostics". It picks up any UI Page whose name starts with the prefix "diagnostic_". The first gauge on the "System Diagnostics" homepage is the "Cluster Node Status". It is created by a UI Page called "diagnostic_cluster_nodes" and contains, as one might expect, the status of each node in the cluster. The way the UI Page works is that it pulls the whole /xmlstats.do page into an object created by the "Diagnostics" script include. It then selects the items from that object that it wants to use. Since semaphore stats are part of xmlstats we've just got to grab the items we want out of the Diagnostic object and add them to the output object.

 

To add Semaphore Availability and Queue Depth metrics to the "System Diagnostics" homepage:

1) Open the UI page called "diagnostics_cluster_nodes"

2) In the HTML field (directly before the line "var ss = diagNode.stats.sessionsummary;") add the following code:

var semaps = diagNode.stats['semaphores'];

for (var idx in semaps) {

var semap = semaps[idx];

var poolName = "Semaphores [" + semap["@name"] + "] ";

addValueToJSON(vals, poolName + "Available Semaphores", semap["@available"]);

addValueToJSON(vals, poolName + "Queue Depth", semap["@queue_depth"]);

}        

3) Click "Update" 

NOTE: If you don't want to alter the out-of-box UI Page what you could do is make a copy of the page and give it a slightly different prefix (e.g. "my_diagnostic_cluster_nodes") and then also copy the widget called "Diagnostics" and just change line 11 that adds the UI Pages to include "my_diagnostic" as the prefix instead.

 

There is a Product Documentation page about the System Diagnostics homepage: https://docs.servicenow.com/bundle/kingston-platform-administration/page/administer/platform-perform....

8 REPLIES 8

Deepak Kumar5
Kilo Sage

Hello Matthew,



One thing I noticed while using Fuji that many time we have to move to different nodes to check and terminate any long user background transaction.


Can we also add user background transaction which cross any threshold value and not cancelled in it ?


Sorry, I don't understand your question. Can you rephrase, please?


Can we add active user transaction from all nodes in this page. Like you added for all scheduler jobs


find_real_file.png


Yes, absolutely. You would just need to write a little more code to pull out all the <semaphore> elements from the <semaphores> element.


You can pull the age (in milliseconds), processor, start time and even the URL, transaction number and session number.



<semaphores available="15" max_queue_depth="22" maximum_concurrency="16" name="Default" queue_depth="0">


<semaphore age="200" processor="Default-thread-2" started="Fri Oct 14 13:54:24 PDT 2016">


2C7602D0DBAA6A00014CFCEFBF96194D #2653503 /xmlstats.do


</semaphore>


</semaphores>


<semaphores available="4" max_queue_depth="0" maximum_concurrency="4" name="Debug" queue_depth="0"/>


<semaphores available="4" max_queue_depth="50" maximum_concurrency="4" name="AMB" queue_depth="0"/>


<semaphores available="4" max_queue_depth="0" maximum_concurrency="4" name="Event Management" queue_depth="0"/>


<semaphores available="4" max_queue_depth="43" maximum_concurrency="4" name="API_INT" queue_depth="0"/>


<semaphores available="7" max_queue_depth="46" maximum_concurrency="8" name="Presence" queue_depth="0">


<semaphore age="129" processor="Presence-thread-3" started="Fri Oct 14 13:54:24 PDT 2016">


CD53141CDBEAA600014CFCEFBF9619AB #2653507 /api/now/ui/presence


</semaphore>


</semaphores>



You just need to write code similar to the scheduler worker code that I mentioned that will iterate each of the <semaphore> elements.