The semaphore queue depth reaches the max queue depth (150 or 50, depending on the respective semaphore pool) and the system doesn't allow anymore requests to come in, thereby by throwing HTTP 429 "Rejecting request" errors. These requests don't get served in spite of the semaphores being available to execute them and the ServletTransaction does not account for recycled requests and leaves sessions in session sync.
You could be affected by the issue if you are on certain patches of Eureka, Fuji, and early patches of Geneva. The severity may depend on the usage of the ServiceNow instance by customers.
Here are a few customer scenarios that may help determine if you are are experiencing a semaphore queue leak.
- HTTP 429 "Rejecting request"
- Errors in the localhost logs
- Increasing session wait times
| Customer Symptom 1: HTTP 429 "Rejecting request" |
|---|
Your users will experience HTTP 429 "Rejecting request" errors on the browser while trying to access or load a page within a ServiceNow instance. The affected node's stats.do will show transactions waiting in the the queue that are not processing. In a worst case scenario, the "affected" semaphore pool reaches its maximum queue depth and is no longer taking any more requests nor is being served. Consider the following example of the affected default semaphore queue: - Available semaphores: 16
- Queue depth: 150
- Max queue depth: 150
- Maximum transaction concurrency: 16
Maximum concurrency achieved: 12
Note this can affect any pool (AMB, Presence, etc.) |
| Customer Symptom 2: Errors in the localhost logs |
|---|
Error messages in the localhost logs with a similar stack trace 2016-04-12 05:06:24 (094) Default-thread-9 SYSTEM SEVERE *** ERROR *** Problem processing asynchronous servlet request
java.lang.RuntimeException: Stream not closed properly
at com.glide.rest.serializer.impl.ResponseSerializerImpl.closeOutputStream(ResponseSerializerImpl.java:153)
at com.glide.rest.processors.RESTAPIProcessor.endRequestProcessing(RESTAPIProcessor.java:261)
at com.glide.rest.processors.RESTAPIProcessor.handleException(RESTAPIProcessor.java:256)
at com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:172)
at com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:32)
at com.glide.sys.ServletTransaction.run(ServletTransaction.java:34)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) 2016-04-12 05:06:24 (094) Default-thread-9 SYSTEM Transaction.getHttpSession() caught IllegalStateException
2016-04-12 05:06:24 (095) Default-thread-9 SYSTEM SEVERE *** ERROR *** java.lang.reflect.UndeclaredThrowableException
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy27.getSession(Unknown Source)
at com.glide.ui.GlideHttpRequestContext.getHTTPSessionID(GlideHttpRequestContext.java:205)
at com.glide.ui.GlideHttpRequestContext$HTTPSessionMethods.getHTTPSession(GlideHttpRequestContext.java:238)
at com.glide.ui.GlideHttpRequestContext.getHTTPSession(GlideHttpRequestContext.java:199) at com.glide.sys.Transaction.getHttpSession(Transaction.java:1373) ----> This specific call MUST be present in the stack at com.glide.sys.ServletTransaction.finishTransaction(ServletTransaction.java:44) at com.glide.sys.util.FixedQueueThreadPoolExecutor.afterExecute(FixedQueueThreadPoolExecutor.java:259) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.glide.ui.GlideServletRequest.invoke(GlideServletRequest.java:121) ... 10 more Caused by: java.lang.IllegalStateException: The request object has been recycled and is no longer associated with this facade at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:905) ... 14 more Note: If you are seeing a similar error but different stack trace, then this is a different problem.
|
| Customer Symptom 3: Increasing session wait times |
|---|
If you were to investigate the ServiceNow performance graphs for your instance, for the primary nodes you would see a gradual spike in the thread concurrency and session wait times. Consider the following graph, where you can see the session wait increasing due to requests not being processed and waiting in the queue because of sessions "leaking" in the queue. |
We recommend that you upgrade to one of the releases, mentioned here ServiceNow KB: ServletTransaction does not account for recycled requests and leaves sessions in sess..., where the fix is found. Upgrading to one of these releases will prevent or improve performance caused by this particular issue:
- Geneva Patch 5 Hotfix 6
- Geneva Patch 6 Hotfix 2
- Geneva Patch 7
- Helsinki Patch 0 Hotfix 3
- Helsinki Patch 1
You can also contact ServiceNow to restart the affected application node to clear the queue.
