question Ecc_queue stuck at processing state for Rest Message via Mid server in ServiceNow AI Platform forum

Ecc_queue stuck at processing state for Rest Message via Mid server

wclim — Thu, 20 Aug 2020 11:09:47 GMT

Hi,

I encountered an issue regarding sending Rest Message via Mid server, and would like to see if anyone encountered this issue or am able to tell me what is wrong.

I was testing around some rest messages via mid server, using the Test functionality in the Rest Message

Everything was working fine and responses were coming in. But then one of the API (which was working fine previously, let's call this API X) returned an error message

No response for ECC message request with sysid=XXXX after waiting for 60 seconds in ECC Queue

So I went into the ecc_queue table to check, and the related ecc_queue record was stuck in Processing state (even after few hours).

I tested the other rest APIs that was tested previously, but all of them got the No response message now. The related ecc_queue records were all stuck in Processing state too.

The MID server itself is Up, and within the mid server itself, triggering APIs via cURL were all successful.

One thing i notice about the response from the triggering API X with the cURL is that the response is very large (about 80,000 characters). But the payload size in the ecc_queue has a limit of 1million character, so I'm not sure where the issue is.

Would appreciate any help in this issue.

Thanks!

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

Sukraj Raikhraj — Thu, 20 Aug 2020 13:30:17 GMT

is the payload too large or getting truncated, you can try increasing the size of the control that capture the payload that need to be process/

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

wclim — Fri, 21 Aug 2020 01:21:50 GMT

There is no indication that the payload is too large or truncated, using cURL to trigger the API directly in mid server returns the full response. In the Rest Message test functionality, it just states that No response after 60 secs, and the ecc_queue record remains as processing.

And the payload field in the ecc_queue table is more than enough - it has a limit of 1million characters:

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

Hitoshi Ozawa — Fri, 21 Aug 2020 03:42:51 GMT

Have you tried deleting the stuck records from the ecc queue and retrying?

Just check the checkbox to the left of the records and select "delete" from the pull-down list at the bottom of the list.

Need to determine if it's something that always happens or if it something that just happened this once.

Also, have you checked the system log see if there was any error?

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

wclim — Fri, 21 Aug 2020 04:00:17 GMT

There are no errors in ServceiNow's system logs.

We tried the following: clearing the output_s folder in the mid server(there were alot of xml files being generated there), and then making sure none of the ecc_queue records are in procesing state.

After that, I was able to trigger the other API's successfully, but once I tested the API that returns a large response, the "No Response after 60 sec" message appears again, and the ecc_queue record became stuck at Processing state.

After that I wasn't able to trigger other API's (all of them got the No response message)

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

Hitoshi Ozawa — Fri, 21 Aug 2020 04:17:19 GMT

Can it be the timeout?

Try setting following properties:

glide.http.outbound.max_timeout.enabled = false

glide.http.outbound.max_timeout (default 30 sec)

https://docs.servicenow.com/bundle/orlando-application-development/page/integrate/web-services/reference/r_HTTPConnectionManagementProps.html

However, I've found that increasing timeout have performance hit on the instance so the better way is to break the REST message into parts.

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

wclim — Fri, 21 Aug 2020 05:22:25 GMT

The glide.http.outbound.max_timeout.enabled is already set to false.

And the timeout is set at 60secs.

But if it is due to the timeout, it seems strange that a timeout could cause all other Rest Messages to failed subsequently.

And it has to be resolved by going into the mid server and clearing the output_s folder.

Also, the Mid server status in the SNOW instance is always UP, so from SNOW point of view, there is nothing wrong with the mid server. We actually found the the buildup of xml files in the output_s folder by going into the mid server and poking around.

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

Hitoshi Ozawa — Mon, 24 Aug 2020 01:31:16 GMT

Can you check the content of the generated xml files to make sure it's not getting duplicate records?

If you can, try restricting the data being returned to make sure it's not the problem with the API.

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

wclim — Mon, 24 Aug 2020 01:44:29 GMT

The records are not duplicated, they are different.

Yes we did a few tries with less data in the API response.

Our tries:

2600 characters - works fine

14k characters - works fine

23k characters - works fine

78k characters - everything stops working after that and will require going into the mid server to clear the output_s folder

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

Hitoshi Ozawa — Mon, 24 Aug 2020 01:48:55 GMT

Maybe check if MID server is not running out of memory? Default is 512MB so it should be more than enough but you may be a be able to find something by analyzing CPU and memory usage.

https://hi.service-now.com/kb_view.do?sysparm_article=KB0538157#:~:text=By%20default%2C%20the%20maximum%20memory,configuration%20by%20the%20property%20wrapper.

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

wclim — Mon, 24 Aug 2020 02:23:51 GMT

Memory and CPU usage is fine and consistent for the past 30 days.

From ServiceNow side, everything looks fine, except that none of the REST messages works after the one that failed and timeout.

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

Hitoshi Ozawa — Mon, 24 Aug 2020 03:56:43 GMT

That happens when the ecc queue stops working correctly.

The only other thing that I can think of that may cause this is when there is a special character in the message.

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

wclim — Mon, 24 Aug 2020 07:00:23 GMT

Im pretty sure theres no special characters in the response.

The results from our testing points to the issue being caused when the response grows large enough when calling a rest api via mid server.

Anyway this issue feels weird to me, we had to find out about it via trial and error. I can't tell from ServiceNow instance that there is an issue as the Mid server status is always up and green.

So just checking here if anyone encounters the same issue.

Have raised a HI ticket to ServiceNow to seek more help

Re: Ecc_queue stuck at processing state for Rest Message via Mid server

Hitoshi Ozawa — Mon, 24 Aug 2020 07:22:02 GMT

When MID Server hangs up, the status is still up. It just hangs up and stop further processing. I've encountered that several times too.

Increasing the timeout helped a little bit but it really didn't resolve my problem because I soon encountered it again when I increased the message.

In my case, I just broke the message into smaller pieces and that solved it. Large messages was also hurting performance on my instance so I never looked back in to getting large message to work.

It didn't seem the MID server but the ECC queue issue.

There have been several reported cases but none seems to fit your situation.

https://community.servicenow.com/community?id=community_question&sys_id=18c7d98adbb83f4ca39a0b55ca9619e4

https://hi.service-now.com/kb_view.do?sysparm_article=KB0718589

https://community.servicenow.com/community?id=community_question&sys_id=29cebaa5db58dbc01dcaf3231f961901

Sorry, hope HI would provide a solution. Appreciate if you would post the solution if you find it.