MID Server socket timeout (HTTP 202) errors

[Antony_Alldis]

Recently, several customers were affected by an underlying issue that caused MID servers to crash, and appear "down" on the instance. For example, in Mid Servers Disconnecting since Fuji upgrade   the user reports that the services on their servers are still running but are reported as being down, only upon restart do they reconnect. You could be affected by the MID server crashing issue if you are on certain patches in Eureka, Fuji and Geneva.

If one reviewed the <path_to_MIDserver>\agent\logs\agent0.log.0 file, you would see errors like this:

12/17/15 07:16:38 (014) StartupSequencer WARNING *** WARNING *** Socket timeout&#13;

12/17/15 07:16:38 (014) StartupSequencer WARNING *** WARNING *** Update failed (Socket timeout)&#13;

12/17/15 07:16:38 (170) LogStatusMonitor.60 stats threads: 26, memory max: 508.0mb, allocated: 90.0mb, used: 29.0mb, queued: 0 probes, processing: 0 probes&#13;

12/17/15 07:16:38 (311) RefreshMonitor.65 WARNING *** WARNING *** Method failed: (https://instance.service-now.com/ecc_agent_property.do?SOAP&amp;displayvalue=all&amp;redirectSupported=true)HTTP/1.1 202 Accepted with code: 202&#13;

12/17/15 07:16:38 (311) RefreshMonitor.65 SEVERE *** ERROR *** getRecords failed (Method failed: (https://instance.service-now.com/ecc_agent_property.do?SOAP&amp;displayvalue=all&amp;redirectSupported=true)HTTP/1.1 202 Accepted with code: 202)&#13;

12/17/15 07:16:38 (311) RefreshMonitor.65 SEVERE *** ERROR *** Failed to load remote properties: Method failed: (https://instance.service-now.com/ecc_agent_property.do?SOAP&amp;displayvalue=all&amp;redirectSupported=true)HTTP/1.1 202 Accepted with code: 202&#13;

12/17/15 07:16:38 (358) ECCQueueMonitor.15 WARNING *** WARNING *** Method failed: (https://instance.service-now.com/ecc_queue.do?SOAP&amp;displayvalue=all&amp;redirectSupported=true)HTTP/1.1 202 Accepted with code: 202&#13;

This would be accompanied by the following type of message in the Application Logs on the instance:

2015-12-17 07:53:27 (349) http-44 WARNING *** WARNING *** GlideRequestManager: Request: /ecc_agent.do, run time: 175669, waiters: 0
2015-12-17 07:53:27 (349) http-23 WARNING *** WARNING *** GlideRequestManager: Request: /ecc_agent.do, run time: 175669, waiters: 0
2015-12-17 07:53:27 (355) http-23 SYSTEM WARNING *** WARNING *** GlideRequestManager: Request ignored: /ecc_queue.doSOAP&displayvalue=all&redirectSupported=true
2015-12-17 07:53:27 (355) http-44 SYSTEM WARNING *** WARNING *** GlideRequestManager: Request ignored: /ecc_mi.doSOAP&displayvalue=all&redirectSupported=true
2015-12-17 07:53:27 (416) http-21 WARNING *** WARNING *** GlideRequestManager: Request: /ecc_agent.do, run time: 175736, waiters: 0
2015-12-17 07:53:27 (421) http-21 SYSTEM WARNING *** WARNING *** GlideRequestManager: Request ignored: /ecc_agent_property.doSOAP&displayvalue=all&redirectSupported=true

Upgrading the version of Tomcat (Apache Tomcat 7.0.64 (Orbit 7.2.0-2)) has occasionally reduced the number of occurrences, but it is not foolproof, and has not worked in the majority of cases, as the underlying issue still exists in the core code. Please Note: ServiceNow will no longer upgrade Apache Tomcat to attempt to resolve this issue.

Due to the nature of this issue, ServiceNow recommends upgrading to one of the fixed in releases mentioned in ServiceNow KB: MID Server stops communicating to the instance and continuously throws socket timeout...

• Fuji Patch 13
• Geneva Patch 7
• Helsinki Patch 1

For more MID Server solutions, troubleshooting demos, and implementation documentation see ServiceNow KB: Discovery and MID Server Resources Page (KB0540193).