What is causing Mid Server Errors SEVERE *** ERROR *** <blah blah blah> Accepted with code: 202

joehoh
Kilo Contributor

‎01-07-2016 03:28 PM

A 202 indicates some kind of 'queued' HTTP request:

10.2.3 202 Accepted

The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.

The 202 response is intentionally non-committal. Its purpose is to allow a server to accept a request for some other process (perhaps a batch-oriented process that is only run once per day) without requiring that the user agent's connection to the server persist until the process is completed. The entity returned with this response SHOULD include an indication of the request's current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.

Several times a month, we get many of these errors on our MId Servers that are used for integrations - Push/Pull Mid Servers (as opposed to those we use for discovery):

The last time this happened, we received 90 of them between 15:04:57 and 15:18:23.     Here is an extract of the first instance of the errors:

12/17/15 14:59:31 (639) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 27.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:00:31 (639) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 27.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:01:31 (625) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 27.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:04:33 (096) ECCQueueMonitor.15 WARNING *** WARNING *** Socket timeout.

12/17/15 15:04:33 (096) ECCQueueMonitor.15 SEVERE *** ERROR *** getRecords failed (Socket timeout).

12/17/15 15:04:57 (533) RefreshMonitor.65 WARNING *** WARNING *** Socket timeout.

12/17/15 15:04:57 (533) RefreshMonitor.65 SEVERE *** ERROR *** getRecords failed (Socket timeout).

12/17/15 15:04:57 (533) RefreshMonitor.65 SEVERE *** ERROR *** Failed to load remote properties: Socket timeout.

12/17/15 15:04:57 (908) RefreshMonitor.65 WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_agent_property.do?SOAP&displayvalue=all&redirectSupported=true)HTTP... 202 Accepted with code: 202.

12/17/15 15:04:57 (908) RefreshMonitor.65 SEVERE *** ERROR *** getRecords failed (Method failed: (https://gaig.service-now.com/ecc_agent_property.do?SOAP&displayvalue=all&redirectSupported=true)HTTP... 202 Accepted with code: 202).

12/17/15 15:04:57 (908) RefreshMonitor.65 SEVERE *** ERROR *** Failed to load remote properties: Method failed: (https://gaig.service-now.com/ecc_agent_property.do?SOAP&displayvalue=all&redirectSupported=true)HTTP... 202 Accepted with code: 202.

12/17/15 15:04:58 (190) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 28.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:04:58 (346) ECCQueueMonitor.15 WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:04:58 (346) ECCQueueMonitor.15 SEVERE *** ERROR *** GetKeys failed (Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:04:58 (346) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 28.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:05:09 (330) StatusMonitor.600 Enqueuing: E:\SERVICE-NOW\MIDSERVER_PRD_PULL\agent\work\monitors\ECCSender\output\ecc_queue.3a0b2203591012004ceba3d2a479f9af.xml.

12/17/15 15:05:09 (455) ECCSender.1 Sending ecc_queue.3a0b2203591012004ceba3d2a479f9af.xml.

12/17/15 15:05:09 (534) ECCSender.1 WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:05:09 (534) ECCSender.1 WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://gaig.service-now.com/ with (Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:05:09 (534) ECCSender.1 Attempt to send ecc_queue.3a0b2203591012004ceba3d2a479f9af.xml failed: file remains enqueued for later sending.

12/17/15 15:05:13 (127) ECCQueueMonitor.15 WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:05:13 (127) ECCQueueMonitor.15 SEVERE *** ERROR *** GetKeys failed (Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:05:28 (128) ECCQueueMonitor.15 WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:05:28 (128) ECCQueueMonitor.15 SEVERE *** ERROR *** GetKeys failed (Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:05:43 (128) ECCQueueMonitor.15 WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:05:43 (143) ECCQueueMonitor.15 SEVERE *** ERROR *** GetKeys failed (Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:05:52 (581) ECCSender.1 WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:05:52 (581) ECCSender.1 SEVERE *** ERROR *** getRecords failed (Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:05:52 (581) ECCSender.1 Sending ecc_queue.3a0b2203591012004ceba3d2a479f9af.1.xml.

12/17/15 15:05:52 (675) ECCSender.1 WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:05:52 (675) ECCSender.1 WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://gaig.service-now.com/ with (Method failed: (https://gaig.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:05:52 (675) ECCSender.1 Attempt to send ecc_queue.3a0b2203591012004ceba3d2a479f9af.1.xml failed: file remains enqueued for later sending.

Here is the log lines excluding those with severe errors from just before until a restart - which I think is self-triggered by the mid server (unconfirmed - this could have been done by an admin):

12/17/15 15:01:31 (625) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 27.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:04:58 (190) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 28.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:04:58 (346) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 28.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:05:58 (347) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 26.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:06:58 (347) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 26.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:07:58 (364) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 26.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:08:58 (349) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 127.0mb, used: 27.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:09:58 (381) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 126.0mb, used: 27.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:10:58 (366) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 126.0mb, used: 27.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:11:58 (414) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 126.0mb, used: 27.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:12:58 (352) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 126.0mb, used: 27.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:13:58 (384) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 126.0mb, used: 26.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:14:58 (385) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 126.0mb, used: 26.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:15:05 (401) AutoUpgrade.3600 SEVERE *** ERROR *** ExecuteException: Process exited with an error: 1060 (Exit value: 1060).

12/17/15 15:15:05 (401) AutoUpgrade.3600 Checking to see if MID server needs to upgrade..

12/17/15 15:15:05 (790) AutoUpgrade.3600 Packages refreshed..

12/17/15 15:15:05 (790) AutoUpgrade.3600 Current packages:.

12/17/15 15:15:05 (790) AutoUpgrade.3600     Installed: [mid-core.2015-09-03-1003.universal.universal.zip, mid-jre.2015-05-14-0900.windows.x86-32.zip].

12/17/15 15:15:05 (790) AutoUpgrade.3600     Assigned: [mid-core.2015-09-03-1003.universal.universal.zip, mid-upgrade.2015-09-03-1003.universal.universal.zip].

12/17/15 15:15:05 (790) AutoUpgrade.3600     Missing: [].

12/17/15 15:15:05 (790) AutoUpgrade.3600     Downloaded: [].

12/17/15 15:15:05 (790) AutoUpgrade.3600 Installed packages are up-to-date..

12/17/15 15:15:58 (425) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 126.0mb, used: 26.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:16:58 (425) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 126.0mb, used: 26.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:17:58 (457) LogStatusMonitor.60 stats threads: 28, memory max: 495.0mb, allocated: 126.0mb, used: 26.0mb, queued: 0 probes, processing: 73 probes.

12/17/15 15:18:22 (192) WrapperListener_stop_runner Running under Java version: 1.8.0_40, java PID: 1932, args: stop.

12/17/15 15:18:22 (192) WrapperListener_stop_runner Stopping MID server.

12/17/15 15:18:22 (192) WrapperListener_stop_runner Destroying injector....

12/17/15 15:18:22 (239) WrapperListener_stop_runner Closing com.service_now.mid.extension.container.ExtensionContainer.

12/17/15 15:18:22 (239) WrapperListener_stop_runner ExtensionContainer is shutting down....

12/17/15 15:18:22 (239) WrapperListener_stop_runner ...waiting a maximum shutdown time of 1000ms.

12/17/15 15:18:22 (239) WrapperListener_stop_runner ExtensionContainer is cleanly shut down.

12/17/15 15:18:22 (302) MIDServer MID Server stopping.

12/17/15 15:18:22 (302) MIDServer interrupting thread IdleConnectionMonitor.5.

12/17/15 15:18:22 (317) MIDServer Getting instance ACLs for table: ecc_agent_log.

12/17/15 15:18:22 (583) MIDServer WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:18:22 (583) MIDServer WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://gaig.service-now.com/ with (Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:18:22 (598) MIDServer interrupting thread AutoUpgrade.3600.

12/17/15 15:18:22 (677) MIDServer WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:18:22 (677) MIDServer WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://gaig.service-now.com/ with (Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:18:22 (677) MIDServer interrupting thread StatusMonitor.600.

12/17/15 15:18:22 (770) MIDServer WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:18:22 (770) MIDServer WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://gaig.service-now.com/ with (Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:18:22 (848) MIDServer WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:18:22 (848) MIDServer WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://gaig.service-now.com/ with (Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:18:22 (864) MIDServer interrupting thread LogStatusMonitor.60.

12/17/15 15:18:22 (942) MIDServer WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:18:22 (942) MIDServer WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://gaig.service-now.com/ with (Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:18:23 (020) MIDServer WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:18:23 (020) MIDServer WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://gaig.service-now.com/ with (Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:18:23 (020) MIDServer interrupting thread FileSyncer.1.

12/17/15 15:18:23 (114) MIDServer WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:18:23 (114) MIDServer WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://gaig.service-now.com/ with (Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:18:23 (208) MIDServer WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:18:23 (208) MIDServer WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://gaig.service-now.com/ with (Method failed: (https://gaig.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:18:23 (302) MIDServer WARNING *** WARNING *** Method failed: (https://gaig.service-now.com/ecc_agent.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202.

12/17/15 15:18:23 (317) MIDServer SEVERE *** ERROR *** getRecords failed (Method failed: (https://gaig.service-now.com/ecc_agent.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 202 Accepted with code: 202).

12/17/15 15:18:23 (395) MIDServer Shutting down SSHProtocolEngine.

12/17/15 15:18:23 (411) SSHProtocolEngine SSH Protocol engine interrupted. Terminating..

12/17/15 15:18:23 (411) SSHProtocolEngine Shutting down SSHClientEngine.

12/17/15 15:18:23 (505) MIDServer MID Server stopped.

12/17/15 15:18:23 (505) MIDServer Destroying injector....

12/17/15 15:18:36 (603) WrapperStartStopAppMain Running under Java version: 1.8.0_40, java PID: 2992, args: start.

12/17/15 15:18:36 (603) WrapperStartStopAppMain Initializing MID Server.

12/17/15 15:18:36 (603) MIDServer Creating injector....

12/17/15 15:18:37 (244) MIDServer Using configuration: E:\SERVICE-NOW\MIDSERVER_PRD_PULL\agent\config.xml.

12/17/15 15:18:37 (728) MIDServer ExtensionContainer ThreadPool started with corePoolSize: 25, maximumPoolSize: 25, maximumQueueSize: 500.

12/17/15 15:18:37 (791) MIDServer Setting basic authentication with user midserver.

12/17/15 15:18:37 (807) MIDServer Using proxy host=squid.td.afg, port=3128, user=null.

12/17/15 15:18:37 (994) MIDServer MID Server starting.

12/17/15 15:18:38 (103) MIDServer Agent home path: E:\SERVICE-NOW\MIDSERVER_PRD_PULL\agent.

12/17/15 15:18:38 (447) MIDServer MID Server started.

12/17/15 15:18:39 (572) StartupSequencer Successfully connected to instance:.

12/17/15 15:18:39 (572) StartupSequencer         Install name: Demo Server.

12/17/15 15:18:39 (572) StartupSequencer       Instance name: gaig.

12/17/15 15:18:39 (572) StartupSequencer                         Node: 353759306f38088fb8b52e6603003ae5.

12/17/15 15:18:39 (572) StartupSequencer             Build date: 09-03-2015_1003.

12/17/15 15:18:39 (572) StartupSequencer               Build tag: glide-fuji-12-23-2014__patch8-08-25-2015.

12/17/15 15:18:39 (572) StartupSequencer           Instance ID: 10b37d7a4a36232701835a137861d821.

12/17/15 15:18:39 (572) StartupSequencer               System ID: app128150.sjc6.service-now.com:gaig011.

12/17/15 15:18:39 (572) StartupSequencer           Instance IP: 10.6.128.150.

12/17/15 15:18:39 (572) StartupSequencer           MID buildstamp: 2015-09-03-1003.

12/17/15 15:18:39 (838) StartupSequencer SOAP basic authentication is enabled.

12/17/15 15:18:39 (838) StartupSequencer SOAP strict security is enabled.

12/17/15 15:18:39 (838) StartupSequencer User midserver has all necessary roles.

Labels:
0 Helpfuls
  • 10,196 Views
4 REPLIES 4

twright5
Kilo Expert

‎01-12-2016 10:17 AM

A socket timeout is usually and indication that the communication between the server and client was abruptly stopped.   Meaning that the server or client did not know it went down.   If you pinged ServiceNow from the mid server are you getting pings back?   Try doing it using the T option.  



Looking at the log, I'm wondering if you are experiencing the same issue we are, in that the mid server will stop when an upgrade is in process.   After talking to SN, they were going to force the upgrade but I do not think it worked, because our mid server went down yesterday.



Tom


0 Helpfuls

joehoh
Kilo Contributor
In response to twright5

‎01-12-2016 02:19 PM

No; SN is reachable during these events.   We actually have a monitor setup to alert us when a Mid Server hasn't checked in within 15 minutes.



No upgrades occurring.   Just happens randomly, but frequently (2+ times/month).



Annoying...



Joe


0 Helpfuls

samlee
Tera Contributor
In response to joehoh

‎05-23-2016 01:57 PM

Joe, did you ever find resolution on this?


0 Helpfuls

Antony_Alldis
ServiceNow Employee
ServiceNow Employee

‎07-13-2016 03:47 AM

Hi folks,



The answer to this is in the following Community Article:



https://community.servicenow.com/community/operations-management/discovery/blog/2016/06/02/mid-serve...



Please review that article for the fix to the issue.



Regards,


Tony