Connection Reset when calling outbound REST API: How to keep connection alive until it finishes?

BabyYoda · ‎12-15-2022

I have an outbound REST API endpoint call in a Scheduled Job Script. At first, I was receiving the HTTP timeout but I fixed that by setting response.setHttpTimeout() to some very high value. Now, however, I am receiving connection reset/timeout errors. The Rest API call is returning several thousand records or more and it could take anywhere from 10 minutes to 40 minutes for the server to send back to the response to the client (ServiceNow).

I have no problems running the API endpiont in a browser tab/window. It eventually finishes and returns a response. However, I'm running into trouble when running the API endpoint in a Background Script, Business Rule, Scheduled Job, Script Include, etc... Does ServiceNow set the Connection header at all? It's hard to debug when I can't just put an HTTP Proxy in between and see the API request/response. Strangely enough, I'm receving ECONNECT errors in Postman. But the fact running the API endpoint via copy/pasting endpoint to the browser's URI address bar works flawlessly leads me to believe that the issue is the client (i.e. Postman, ServiceNow) is not sending the proper request headers to keep the connection alive and the server is closing out the connection prematurely.

I've looked at transaction quotas and I've seen some system properties elsewhere which talk about REST. However, this is an outbound call. I'm not using Mid Server or ECC. And the API endpoint call is executed synchronously.

Not sure what I need to set to avoid the Connection reset/timeout, either as a request header, System Property, or elsewhere, such that the API call stays open until the server is finished and sends back the response.

Any ideas?

Thank you for your help.

ikc_jc · ‎12-15-2022

Hi,

Have you looked at these?

Basically, you can increase the timeout by setting 'glide.http.outbound.max_timeout.enabled' to false and using the 'waitForResponse' function.

But I would not recommend this approach as this can cause a semaphore to be 'stuck' which can cause performance issues. Have you looked into alternative ways of importing the data? e.g. FTP via MID Server

Thanks

BabyYoda · ‎12-16-2022

Hi ikc_jc,

Thank you for the links. I reviewed those links and came up with a similar conclusin regarding switching to async.

Prior to reading your response, I tried a few different things to narrow down the issue.

I took a look at this KB article:

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0716391

This KB article suggests that system properties glide.http.timeout is set to 175 seconds and glide.http.connection_timeout is set to 10 seconds. However, they are not listed under system properties. I'm not sure if they editable. They might be hardcoded. I created the system properties and it didn't matter what value I set for both properties; running a Test of the API via Outbound Rest Message module immediately resulted in a Connection reset/timeout; however, I checked the outbound API (which I developed on an external server) and it received the response and was still building out the response. So, that tells me that the client, ServiceNow, issued a connection reset, HTTP 0. ServiceNow probably coded to throw an exception and close the connection. In this case, because I set the those system properties when it appears that they are not designed to be created and set in the System Properties table. This is something I'm going to follow up with ServiceNow Support and find out if these two system properties can be created and, subsequently, updated to a different (read: higher) value.

I also executed the API endpoint directly in the URI address bar. It eventually completed. At the same time, I had developer tools open and network activity recording. I exported the request i tracked as a curl script and imported it into Postman. Then I ran it and I believe the ECONNECT error might be a network error attributed to VPN. Intermittent. It just so happens I am on VPN on my own computer. I disabled my VPN and it completed successfully. I re-enabled my VPN and it completed successfully, although that could be because the connection was kept alive. But I'm fairly certain the ECONNECT error is related more to network and not server configuration or database configuration.

In the past, yes, I believe we imported this large dataset via XML, CSV, or JSON using FTP or direct import but not with MID Server, no. That was some time ago. We do have MID Server setup now so that's another tool in our toolbox that we can use to solve this problem.

To solve this issue quickly, however, what I did was developed a couple of new API endpoints to page through the records; essentially a bulk insert that requires multiple passes; or data chunking. SOAP has stuff like this built-in but not REST, of course. In the ServiceNow Scheduled Job, I used executeasync() and waitForResponse() methods to turn the API endpoint call into a ECC queue synchronous request. I also employed the setHttpTimeout method and set it to a rather high number. Using this approach, I was able to import this bulk dataset and let the transform map do its thing.

However, I'm still determined to investigate why connection timeout is occurring with a large dataset. Ergo, is ServiceNow throwing an exception and prematurely telling the server to close the connection, or is it not sending the proper request headers to the server; or, is it on my end and there's a server or database configuration issue that needs to be addressed.

This task, to be clear, is really more of an exception than the rule. It's a small set of data most of the time. But for a development requirement, feeding in a large dataset to ServiceNow is required. I will likely redevelop this task in the near future and abandon the outbound API call and, instead, employ MID Server and query the database directly or perhaps, as you say, FTP and MID Server.

That's such a great point about impact to semaphores which, in turn, can severely impact performance. Great tip, thank you!

I wanted to add that those links which mostly discuss system properties, MID Server, and async do not directly apply to the original issue which is connection reset/timeout with execute method. I'd like to know if it's something configurable in the instance or if it's compiled code/hardcoded and that's just the way it is and you must use alternative methods. I will most certainly raise this question to ServiceNow Support to get a clear answer.

Thanks again for your help and suggestions!

ikc_jc · ‎12-18-2022

Hi @BabyYoda ,

Thank you for sharing in such details.

It would be great if you can share the response from Support in the future, very curious to know as well.

Another idea popped into my head, does your API support pagination? If it does, you can break them into chunks and would resolve the timeout issue.

Cheers

srivatsa_tatti · ‎09-27-2023

HI @BabyYoda ,

We are also facing the same issue. Wanted to check if you were able to resolve this issue and how?

Awaiting for your reply.

Thanks in advance

Sri