Receiving HTTP 401 Errors When Multi-threading Data Extracts via REST API

[David Schafer]

All,

We are attempting to extract tables within the APM space into a Data Lake for the extended reporting capabilities we have available there. We are using the OOB 'table API' for this purpose.

During initial extract/load, we regularly run into a situation where one/several queued extract(s) receive a '401' error.

We have found that if we wait 15 minutes or so(somewhat unreliable), we can start again. Programming around this behavior is possible - but certainly not optimal.

Can someone shed some light on what might be happening here?

Thank you.

[Kieran Anson]

Is the tool you're using to pull data from SN re-using the provided cookies? By re-using cookies, the session token stored can be used to bind your API calls to a single node and avoid the need to repeat TLS handshakes and authentication checks. It does come with its risk of overwhelming a single node with API transactions.

[nateanderson · Accepted solution]

Depending on the number of threads, you might be maxing out the maximum simultaneous connections. In which you'll be put into a queue that has a timeout. Also important to note that this connection pool is shared with UI Transactions as well (anything that hits the Application Node via HTTPs). 

It's been a while since I did any real testing on this to recall what HTTP Status would get returned. Ideally you'd get something like a 408 / 504 back... but SN might just dump the generic 401 un-authorized. 

If you check the Service Account when this occurs, does it show locked out = true? This would happen if one of the threads is trying a bad PW for some reason.. 

Regarding the suggestion of cookies, the /api/now/table/{table_name} api does not leverage cookies, but instead requires authentication to be passed on every request. In that regard, are you using Basic Auth or OAuth? If it's possible to swap to some form of OAuth to get an access/refresh token... you might see some speed improvements as well. I've found in testing that the Basic Auth header has to be digested, salted, hashed, and tested on every API Call. Where an access token is just checked against the value on hand in the oauth_credentials table. 

Anywho... bunch of stuff that may or may not help..

From a diagnostics point, if the system your using can "log the response payload" there might be some additional detail in there that could help isolate what's going on.

[David Schafer]

I appreciate the responses. Of which, the only one I can currently speak to is 'basic vs oauth'. We are using basic auth - and we leverage the same credential for all queries(a handful at a time).

Of more interest to me were the instance-level topics you refer to. We are sorely lacking in monitoring/analyzing our instances. For example, Nate mentioned "Depending on the number of threads, you might be maxing out the maximum simultaneous connections. In which you'll be put into a queue that has a timeout. Also important to note that this connection pool is shared with UI Transactions as well (anything that hits the Application Node via HTTPs)."

How would I know this? How can I track/trend it? In general, I don't know what questions to ask or where to find the answers. Where/how do I gain this knowledge?

Thanks for you input!

[Kieran Anson]

The replacement product was Application Insights but that's now been deprecated sadly. There is a strong push to buying Impact and then leveraging Instance Observer to monitor platform health.

/stats.do can give you a point in time view of your transaction volume, but only on the node you're connected too