Ecc_queue stuck at processing state for Rest Message via Mid server

[wclim]

Hi,

I encountered an issue regarding sending Rest Message via Mid server, and would like to see if anyone encountered this issue or am able to tell me what is wrong.

I was testing around some rest messages via mid server, using the Test functionality in the Rest Message

Everything was working fine and responses were coming in. But then one of the API (which was working fine previously, let's call this API X) returned an error message 

No response for ECC message request with sysid=XXXX after waiting for 60 seconds in ECC Queue

So I went into the ecc_queue table to check, and the related ecc_queue record was stuck in Processing state (even after few hours).

I tested the other rest APIs that was tested previously, but all of them got the No response message now. The related ecc_queue records were all stuck in Processing state too. 

The MID server itself is Up, and within the mid server itself, triggering APIs via cURL were all successful.

One thing i notice about the response from the triggering API X with the cURL is that the response is very large (about 80,000 characters). But the payload size in the ecc_queue has a limit of 1million character, so I'm not sure where the issue is.

Would appreciate any help in this issue.

Thanks!

[Hitoshi Ozawa]

Can it be the timeout?

Try setting following properties:

glide.http.outbound.max_timeout.enabled = false

glide.http.outbound.max_timeout (default 30 sec)

https://docs.servicenow.com/bundle/orlando-application-development/page/integrate/web-services/refer...

However, I've found that increasing timeout have performance hit on the instance so the better way is to break the REST message into parts.

[wclim]

The glide.http.outbound.max_timeout.enabled is already set to false.

And the timeout is set at 60secs.

But if it is due to the timeout, it seems strange that a timeout could cause all other Rest Messages to failed subsequently.

And it has to be resolved by going into the mid server and clearing the output_s folder.

Also, the Mid server status in the SNOW instance is always UP, so from SNOW point of view, there is nothing wrong with the mid server. We actually found the the buildup of xml files in the output_s folder by going into the mid server and poking around.

[Hitoshi Ozawa]

Can you check the content of the generated xml files to make sure it's not getting duplicate records?

If you can, try restricting the data being returned to make sure it's not the problem with the API.

[wclim]

The records are not duplicated, they are different.

Yes we did a few tries with less data in the API response.

Our tries:

2600 characters - works fine

14k characters - works fine

23k characters - works fine

78k characters - everything stops working after that and will require going into the mid server to clear the output_s folder

[Hitoshi Ozawa]

Maybe check if MID server is not running out of memory? Default is 512MB so it should be more than enough but you may be a be able to find something by analyzing CPU and memory usage.

https://hi.service-now.com/kb_view.do?sysparm_article=KB0538157#:~:text=By%20default%2C%20the%20maxi...