Sudden MID Server Connectivity Failure

Daisy3
Tera Guru

Our MID server suddenly fails to connect to the instance every now and then. We are looking at understanding what is causing this issue and try to optimise it from our end. I am unable to understand what might have gone wrong as the below log shows a heartbeat and a sudden connection failure.

Can anyone please guide me how to debug or what next steps can be taken to avoid this issue frequently.

Thanks!

 

2024-07-12T17:09:01.817+1000 INFO (Worker-Interactive:HeartbeatProbe-0b99fe1e47d38a90685ada30116d43a9) [AWorker:145] Worker completed: HeartbeatProbe time: 0:00:00.001
2024-07-12T17:09:01.818+1000 INFO (ECCQueueMonitor.1) [ECCQueueMonitor:389] Received message with timestamp: 2024-07-12 07:09:01. Existing Query window is : 2024-07-12 05:05:49, Updated the query window to: 2024-07-12 05:09:01
2024-07-12T17:09:01.819+1000 INFO (ECCQueueMonitor.1) [FileReadWrite:75] Time being written to the file : 1720760941000
2024-07-12T17:09:02.044+1000 INFO (ECCSender.1) [ECCSenderCache:409] Sending ecc_queue.0b99fe1e47d38a90685ada30116d43a9.xml
2024-07-12T17:09:20.394+1000 INFO (LogStatusMonitor.60) [LogStatusMonitor:54] 2024-07-12T07:09:20.394Z, stats threads: 102, memory max: 910.0mb, allocated: 460.0mb, used: 98.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes
2024-07-12T17:10:20.399+1000 INFO (LogStatusMonitor.60) [LogStatusMonitor:54] 2024-07-12T07:10:20.399Z, stats threads: 101, memory max: 910.0mb, allocated: 460.0mb, used: 98.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes
2024-07-12T17:10:49.124+1000 WARN (ECCQueueMonitor.40) [HTTPClient:830] java.net.ConnectException: Connection refused: connect
2024-07-12T17:10:49.124+1000 ERROR (ECCQueueMonitor.40) [RemoteGlideRecord:918] getRecords failed (java.net.ConnectException: Connection refused: connect)
2024-07-12T17:10:49.125+1000 WARN (ECCQueueMonitor.40) [RetryExecutor:114] MIDRemoteGlideRecord.query failed with error: java.net.ConnectException: Connection refused: connect, retrying in 10 seconds
2024-07-12T17:11:20.434+1000 INFO (LogStatusMonitor.60) [LogStatusMonitor:54] 2024-07-12T07:11:20.434Z, stats threads: 102, memory max: 910.0mb, allocated: 460.0mb, used: 98.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes
2024-07-12T17:11:25.165+1000 WARN (ECCQueueMonitor.40) [HTTPClient:830] java.net.ConnectException: Connection refused: connect
2024-07-12T17:11:25.165+1000 ERROR (ECCQueueMonitor.40) [RemoteGlideRecord:918] getRecords failed (java.net.ConnectException: Connection refused: connect)
2024-07-12T17:11:25.166+1000 WARN (ECCQueueMonitor.40) [RetryExecutor:114] MIDRemoteGlideRecord.query failed with error: java.net.ConnectException: Connection refused: connect, retrying in 15 seconds
2024-07-12T17:12:06.200+1000 WARN (ECCQueueMonitor.40) [HTTPClient:830] java.net.ConnectException: Connection refused: connect
2024-07-12T17:12:06.200+1000 ERROR (ECCQueueMonitor.40) [RemoteGlideRecord:918] getRecords failed (java.net.ConnectException: Connection refused: connect)
2024-07-12T17:12:06.200+1000 WARN (ECCQueueMonitor.40) [RetryExecutor:114] MIDRemoteGlideRecord.query failed with error: java.net.ConnectException: Connection refused: connect, retrying in 22 seconds
2024-07-12T17:12:20.395+1000 INFO (LogStatusMonitor.60) [LogStatusMonitor:54] 2024-07-12T07:12:20.394Z, stats threads: 101, memory max: 910.0mb, allocated: 460.0mb, used: 98.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes
2024-07-12T17:12:54.733+1000 WARN (ECCQueueMonitor.40) [HTTPClient:830] java.net.ConnectException: Connection refused: connect
2024-07-12T17:12:54.733+1000 ERROR (ECCQueueMonitor.40) [RemoteGlideRecord:918] getRecords failed (java.net.ConnectException: Connection refused: connect)
2024-07-12T17:12:54.734+1000 WARN (ECCQueueMonitor.40) [RetryExecutor:114] MIDRemoteGlideRecord.query failed with error: java.net.ConnectException: Connection refused: connect, retrying in 33 seconds