Semaphores performance metrics
Summarize
Summary of Semaphores Performance Metrics
Semaphores performance metrics in ServiceNow provide critical indicators at the instance level, measuring how transaction processing resources are utilized over a selected time. These metrics help identify bottlenecks, resource contention, and potential performance issues related to transaction processing threads (semaphores) on application nodes.
Show less
Key Metrics and Their Significance
- Semaphore Default Mean: Shows the average number of semaphores (threads) used for standard end user transactions over one minute. Default semaphore pools handle most user transactions and have a base of 16 threads per node. Sustained usage near or above 10 suggests serious performance issues.
- Semaphore Default Qdepth: Indicates how many transactions are queued waiting for a semaphore in the default pool. Normal operation allows some queueing, but spikes or queues reaching the max (150) can cause transaction rejections (429 errors), signaling throughput problems or resource exhaustion.
- Semaphore Session Waiters Mean: Measures the average wait time for a session’s subsequent request until the prior request completes, highlighting delays caused by long-running requests within the same session.
- Semaphore Waiters Mean: Reflects how long requests wait for any semaphore to become available, indicating contention and resource bottlenecks if high.
- Integrations Semaphore In Use and Qdepth: Track APIINT semaphore usage and queue depth for integration transactions. The APIINT pool typically has 4 threads per node and a 50 transaction queue limit. Overloaded queues cause transaction rejections and potential service loss.
- AMB Send and Receive In Use and Qdepth: Measure the usage and queue depth for Asynchronous Message Bus (AMB) send and receive semaphores, each with a default queue limit of 50. High queue depths or full usage can impact message delivery and channel subscription operations.
- Semaphore Rejected Metrics: Count rejected transactions due to exhausted semaphores for default, integration (APIINT), AMB Send, and AMB Receive pools. Rejections often manifest as HTTP 429 errors indicating too many requests.
Practical Guidance for ServiceNow Customers
- Monitor semaphore usage and queue metrics regularly to detect sustained high usage or queue buildup, as these are early signs of performance degradation.
- Small, transient spikes are usually not problematic; focus on sustained high semaphore usage or queue depths close to maximum thresholds.
- If semaphore queues fill and cause rejections, investigate for long-running transactions, unusually high transaction rates, or resource contention like CPU or memory bottlenecks on affected nodes.
- Determine if issues are isolated to single nodes (likely application layer problems) or across all nodes simultaneously (potentially database-related).
- For integration issues, note that MID servers depend on APIINT semaphores; exhaustion can disrupt communication, requiring intervention such as killing long-running transactions to free semaphores.
- Adjust semaphore thread counts only if necessary and with caution, as the default settings reflect optimal base configurations. Additional nodes can be added to increase semaphore capacity if needed.
Expected Outcomes
By effectively using semaphore metrics, ServiceNow customers can maintain smooth transaction throughput, promptly identify and resolve resource bottlenecks, and minimize transaction rejections that lead to service degradation. This ensures reliable instance performance and optimal response times for both end users and integrations.
The metrics provide the key performance indicators calculated at the instance level for the selected duration.
Semaphore Default Mean
The average number of semaphores in use over a 1-minute period. In other words, the number of end user transactions being processed at once. The default semaphore pool is a group of threads that handle any transaction that is not handled by a more specific semaphore pool. Integration transactions, for example, are handled by the integrations pool (API_INT) semaphore pool and thus would not go through the Default Semaphore pool. Most significant end user transactions will go through the Default semaphore pool with the exception of some very fast transactions like the UI Presence, Asynchronous Message Bus or Debugging transactions.
The Default semaphore pool has 16 threads, which is what the base system provides, and thus, the hypothetical max would be 16. In practice you would see extreme performance issues before reaching the 16 thread max and often a sustained usage of 10 or more might be indicative of a severe issue. Small spikes of semaphore usage on a single node can be ignored, but sustained spikes on a single node can be indicative of some type of performance issue. It is very rare that semaphore usage itself is the root cause of an issue — if more semaphores are needed, then a new node with 16 more semaphores is added to the ServiceNow cluster.
- Some of you may have customized the number of Default semaphores or ServiceNow may have done this on your behalf. To find out how many Default semaphore threads are on your instance go to .
- If a dangerously high spike of semaphore usage is observed the next step would be to find out if some resource has been maxed out.
- If only one node is spiking at a time then it is likely an application layer issue. If all nodes are spiking at the same time, then it is likely a database issue.
Semaphore Default Qdepth
The number of transactions which are currently queued in the default semaphore pools request queue awaiting processing. Note that the default request queue holds details of inbound end user transactions.
As transactions arrive at application nodes they are placed into the corresponding semaphore pools request queue. This queue is then drained as corresponding semaphores/threads become free and execute transactions from the request queue. As a result, it is common to see small numbers of requests in a semaphore pools request queue during normal operation. If, however, the request queue on one or more nodes sees spikes where the queue starts to fill, then this indicates either that there are spikes in transaction rates hitting those nodes, default threads on the nodes are busy executing some long running transactions, or performance of the node is periodically degrading such that transactional throughput decreases.
By default the default semaphore pools request queue can hold a maximum of 150 transactions. If the queue fills, additional transactions will be rejected with a 429 error indicating that the transaction could not be processed (and potential loss of service).
Semaphore Session Waiters Mean
Semaphore session wait time refers to the duration a new request from a given session waits for the prior request from the same session to complete. In most scenarios, the ServiceNow AI Platform uses a session-based synchronization mechanism to process requests in the order they are received. High semaphore session wait times can indicate that an individual session may be executing long-running requests, causing other requests from the same session to wait. This is metric is the mean of that wait time.
Semaphore Waiters Mean
Semaphore waiters refer to the duration a new request (user or API) waits for a semaphore to become available. Semaphores are used to process requests from users (typically 16 max per node) and APIs (typically 4 max per node) and are limited in number. If the semaphore waiters mean is high, it indicates semaphore contention and suggests a bottleneck due to resource constraints.
Integrations Semaphore in use
The number of API_INT semaphores/threads which are in-use (i.e., currently executing a transaction) on an application node at a given point in time.
Ideally all application nodes should have some free API_INT semaphores available at all times — this guarantees the best response time for integrations transactions (as there will be a semaphore available to execute a transaction as soon as it hits the semaphore pools request queue).
Integrations Semaphore Qdepth
The number of transactions which are currently queued in the API_INT semaphore pools request queue awaiting processing. Note that the API_INT request queue holds details of inbound integrations (REST/SOAP/JSON) transactions.
As transactions arrive at application nodes they are placed into the corresponding semaphore pools request queue. This queue is then drained as corresponding semaphores/threads become free and execute transactions from the request queue. As a result it is common to see small numbers of requests in a semaphore pools request queue during normal operation. If, however, the request queue on one of more nodes sees spikes where the queue starts to fill this indicates either that there are spikes in transaction rates hitting those nodes, API_INT threads on the nodes are busy executing some long running transactions, or performance of the node is periodically degrading such that transactional throughput decreases. By default the API_INT semaphore pools request queue can hold a maximum of 50 transactions. If the queue fills additional transactions will be rejected with a 429 error indicating that the transaction could not be processed (and potential loss of service).
AMB Send In Use
The number of AMB Send semaphores/threads which are in-use (that is, currently executing a transaction) on an application node at a given point in time.
Ideally all application nodes should have some free AMB Send semaphores available at all times - this guarantees the best response time for corresponding transactions (as there will be a semaphore available to execute a transaction as soon as it hits the semaphore pools request queue).
AMB Send Qdepth
The number of transactions which are currently queued in the AMB Send semaphore pools request queue awaiting processing. Note that the AMB Send request queue holds details of operations such as channel subscription or AMB message publication.
As transactions arrive at application nodes, they are placed into the corresponding semaphore pools request queue. This queue is then drained as corresponding semaphores/threads become free and execute transactions from the request queue. As a result, it is common to see small numbers of requests in a semaphore pools request queue during normal operation. If, however, the request queue on one of more nodes sees spikes where the queue starts to fill, this indicates either that there are spikes in transaction rates hitting those nodes, AMB Send threads on the nodes are busy executing some long running transactions, or performance of the node is periodically degrading such that transactional throughput decreases.
By default the AMB Send semaphore pools request queue can hold a maximum of 50 transactions. If the queue fills, additional transactions will be rejected with a 429 error indicating that the transaction could not be processed (and potential loss of service).
AMB Receive in use
The number of AMB Receive semaphores or threads which are in-use (that is, not currently executing a transaction) on an application node at a given point in time.
Ideally all application nodes should have some free AMB Receive semaphores available at all times — this guarantees the best response time for corresponding transactions (as there will be a semaphore available to execute a transaction as soon as it hits the semaphore pools request queue).
AMB Receive Qdepth
The number of transactions which are currently queued in the AMB Receive semaphore pools request queue awaiting processing. Note that the AMB Receive request queue holds details of operations such as AMB message delivery.
As transactions arrive at application nodes they are placed into the corresponding semaphore pools request queue. This queue is then drained as corresponding semaphores or threads become free and execute transactions from the request queue. As a result it is common to see small numbers of requests in a semaphore pools request queue during normal operation. If, however, the request queue on one of more nodes sees spikes where the queue starts to fill, this indicates either that there are spikes in transaction rates hitting those nodes, AMB Receive threads on the nodes are busy executing some long running transactions, or performance of the node is periodically degrading such that transactional throughput decreases.
By default the AMB Receive semaphore pools request queue can hold a maximum of 50 transactions. If the queue fills, additional transactions will be rejected with a 429 error indicating that the transaction could not be processed (and potential loss of service).
Semaphore Rejected Metrics
- Semaphore Default Rejected
- This represents the number of rejected executions of semaphore default for last minute. Semaphore default means the number of end user transactions processed at once.
- Integration Semaphore Rejected
- This represents the number of rejected executions of API_INT semaphore or thread for last minute on an application node at given point of time.
- AMB Receive Rejected
- This represents the number of rejected executions of AMB receive semaphore or thread for last minute on an application node at given point of time.
- AMB Send Rejected
- This represents the number of rejected executions of AMB send semaphore/thread for last minute on an application node at given point of time.