Ecc_queue stuck at processing state for Rest Message via Mid server

wclim
Kilo Sage

‎08-20-2020 04:09 AM

Hi,

I encountered an issue regarding sending Rest Message via Mid server, and would like to see if anyone encountered this issue or am able to tell me what is wrong.

I was testing around some rest messages via mid server, using the Test functionality in the Rest Message

find_real_file.png

Everything was working fine and responses were coming in. But then one of the API (which was working fine previously, let's call this API X) returned an error message 

No response for ECC message request with sysid=XXXX after waiting for 60 seconds in ECC Queue

So I went into the ecc_queue table to check, and the related ecc_queue record was stuck in Processing state (even after few hours).

I tested the other rest APIs that was tested previously, but all of them got the No response message now. The related ecc_queue records were all stuck in Processing state too. 

 

The MID server itself is Up, and within the mid server itself, triggering APIs via cURL were all successful.

 

One thing i notice about the response from the triggering API X with the cURL is that the response is very large (about 80,000 characters). But the payload size in the ecc_queue has a limit of 1million character, so I'm not sure where the issue is.

 

 

Would appreciate any help in this issue.

Thanks!

Labels:
Preview file
7 KB
0 Helpfuls
  • 2,055 Views
13 REPLIES 13

wclim
Kilo Sage
In response to Hitoshi Ozawa

‎08-23-2020 07:23 PM

Memory and CPU usage is fine and consistent for the past 30 days.

From ServiceNow side, everything looks fine, except that none of the REST messages works after the one that failed and timeout.

0 Helpfuls

Hitoshi Ozawa
Giga Sage
Giga Sage
In response to wclim

‎08-23-2020 08:56 PM

That happens when the ecc queue stops working correctly.

The only other thing that I can think of that may cause this is when there is a special character in the message.

0 Helpfuls

wclim
Kilo Sage
In response to Hitoshi Ozawa

‎08-24-2020 12:00 AM

Im pretty sure theres no special characters in the response.

The results from our testing points to the issue being caused when the response grows large enough when calling a rest api via mid server.

Anyway this issue feels weird to me, we had to find out about it via trial and error. I can't tell from ServiceNow instance that there is an issue as the Mid server status is always up and green. 

So just checking here if anyone encounters the same issue. 

Have raised a HI ticket to ServiceNow to seek more help

0 Helpfuls

Hitoshi Ozawa
Giga Sage
Giga Sage
In response to wclim

‎08-24-2020 12:22 AM

When MID Server hangs up, the status is still up. It just hangs up and stop further processing. I've encountered that several times too.

Increasing the timeout helped a little bit but it really didn't resolve my problem because I soon encountered it again when I increased the message.

In my case, I just broke the message into smaller pieces and that solved it. Large messages was also hurting performance on my instance so I never looked back in to getting large message to work.

It didn't seem the MID server but the ECC queue issue.

There have been several reported cases but none seems to fit your situation.

https://community.servicenow.com/community?id=community_question&sys_id=18c7d98adbb83f4ca39a0b55ca96...

https://hi.service-now.com/kb_view.do?sysparm_article=KB0718589

https://community.servicenow.com/community?id=community_question&sys_id=29cebaa5db58dbc01dcaf3231f96...

Sorry, hope HI would provide a solution. Appreciate if you would post the solution if you find it.