JDBC Connection Issue - Did not get a response from the MID Server

[Heiko Bllr]

Hi,

we have a several JDBC datasources configured, for SQL Server and Oracle databases.

From time to time it seems the MID Server is not able to establish a connection to the database after retrying for 5 mins (every 5s).

The errors we see in the agent.log would look like this:

10/22/19 19:20:10 (189) Worker-Standard:JDBCProbeError-a807251fdbec0094497151c6f49619a0 SEVERE *** ERROR *** java.lang.NullPointerException

              at com.service_now.mid.probe.JDBCProbe.removeCachedConnection(JDBCProbe.java:271)

              at com.service_now.mid.probe.JDBCProbe.cancel(JDBCProbe.java:806)

              at com.service_now.mid.message_executors.CancelProbeExecutor.cancelProbe(CancelProbeExecutor.java:58)

              at com.service_now.mid.probe.JDBCProbeError.probe(JDBCProbeError.java:35)

              at com.service_now.mid.probe.AProbe.process(AProbe.java:96)

              at com.service_now.mid.queue_worker.AWorker.runWorker(AWorker.java:121)

              at com.service_now.mid.queue_worker.AWorkerThread.run(AWorkerThread.java:20)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

              at java.lang.Thread.run(Thread.java:748)

...

10/22/19 19:20:10 (194) Worker-Standard:JDBCProbeError-a807251fdbec0094497151c6f49619a0 Worker completed: JDBCProbeError source: Did not get a response from the MID server time: 0:00:00.002

...

10/22/19 19:20:15 (174) Worker-Standard:JDBCProbeError-8507e91fdbec0094497151c6f4961965 Worker starting: JDBCProbeError source: Did not get a response from the MID server

10/22/19 19:20:15 (176) Worker-Standard:JDBCProbeError-8507e91fdbec0094497151c6f4961965 Cancellation called on probe com.service_now.mid.probe.JDBCProbe

10/22/19 19:20:15 (176) Worker-Standard:JDBCProbe-aed5e5dbdbec0094497151c6f49619e5 SEVERE *** ERROR *** Error when trying to execute: SELECT * FROM [removed] 10/22/19 19:20:15 (177) Worker-Standard:JDBCProbe-aed5e5dbdbec0094497151c6f49619e5 SEVERE *** ERROR *** Socket closed

com.microsoft.sqlserver.jdbc.SQLServerException: Socket closed

              at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1667)

              at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1654)

              at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1789)

              at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:4838)

              at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:6150)

              at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteCursored(SQLServerStatement.java:1877)

              at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:766)

              at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(SQLServerStatement.java:689)

              at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:5696)

              at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1715)

              at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:180)

              at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:155)

              at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeQuery(SQLServerStatement.java:616)

              at com.service_now.monitor.jdbc.JDBCRowSet.query(JDBCRowSet.java:64)

              at com.service_now.mid.probe.JDBCProbe.doSelect(JDBCProbe.java:313)

              at com.service_now.mid.probe.JDBCProbe.doQuery(JDBCProbe.java:199)

              at com.service_now.mid.probe.JDBCProbe.probe(JDBCProbe.java:124)

              at com.service_now.mid.probe.AProbe.process(AProbe.java:96)

              at com.service_now.mid.queue_worker.AWorker.runWorker(AWorker.java:121)

              at com.service_now.mid.queue_worker.AWorkerThread.run(AWorkerThread.java:20)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

              at java.lang.Thread.run(Thread.java:748)

Any idea what I can do to fix this?

Thank you,

Heiko

[Michael Fry1]

Are you sure the mid-server is available and not down for maintenance or some other issue?

[Heiko Bllr]

They are all up. I can see this error in real time in the agent log. we just increased the jdbc timeout to 30 mins without success by setting this glide.jdbcprobeloader.retry to 360.

[Heiko Bllr · Accepted solution]

I just realized I never commented again on my question. The root cause for the issue was, that the database query too ktoo long and then the connection basically timed out (I believe the OOTB setting is 300s if I am not mistaken). I took me a while to figure this out, only after I installed a DB client on the MID Server to run the SQL query I could see the issue and show to my client.

So in my case "Did not get a response from the MID server" did mean that the MID server could not respond back to the ECC queue with a proper result.