My recent passion on the Now Platform are the API-based integrations. They are challenging, and thus, quite interesting. So is the case that I describe in the following paragraphs.
Put in simple words, the assignment was to build a searching tool that is using ServiceNow as a place to submit your search request and get the results stored and displayed for further analysis.
The search itself happens in a third-party system. Both systems are talking to each other over REST messages.
In ServiceNow we maintain a container record, which has multiple search items related to it.
Initially, the estimated load was 2500 search items per day (24h), so no one cared about performance.
I started the implementation, which was basically a bunch of rest messages, a business rule to prepare and execute the API calls and a script include to take care for the request preparation and response parsing. So far so good. But… eventually, the estimated load jumped to 130 000 search items per day, which we were not able to handle with the current setup.
So, the analysis began. I put some timestamps here and there within the code (Hate you, GlideDateTime!), and found that the request preparation was taking about 15ms, API call was literally nothing – 0ms, while the response processing ranged from 900 to 1300ms. So, bottle neck was identified, but what next? I asked here and there and got help in the ServiceNow Development Unofficial Discord server. Guys there were helpful as always and proposed few approaches. One of them was asynchronous API calls, but reading through the docs, I decided to skip this one, as it includes MID servers and ecc_queues.
Other suggestions were to move the response processing to custom queues, to enable the nodes to process multiple responses in parallel.
And that is what I did:
- created 10 custom queues, 5 per each node
- shifted the response handling to events, randomly distributed between the queues
Voila, I had about 4 times faster execution. But still, response processing was taking some time.
As suggested in The ServiceNow Development Handbook (thanks, Tim):
- replaced the GlideRecord().newRecord() with the better performing .initialize(), doing one DB-operation less.
- Removed all unnecessary logging to the system log
- Removed all activity log messages that I was previously adding to the container case, to indicate that the search was triggered and completed
Another 3x, so 12x in total. The whole team was so happy. And then the new requirement came: 1 000 000 records per day. Yes, One Million.
Okay, I know that request preparation was not the bottle neck in the beginning, but removing the response parsing out of my way, now it was the new one.
This time it was easy, I knew what to do. Shifted the request preparation to the event queues for parallel processing. 30-35x in total. I couldn’t be happier. The one million records requirements were almost met, in a DEV environment with 2 nodes. Meaning that on a PRODUCTION environment there must be no issues at all.
Put in absolute values, this looks like:
- Before the change – about 3 seconds per search item, or 15 hours for 7000 records.
- After the change, 10ms per search item or 16 minutes per 9000 records.
Now it was time to load some more records in the queue to see how it behaves. I put 50000 records in the container and hit that Search button.
What? It was completed within 20 minutes. What the heck, no way! 120x faster! "Unbelievable..." my manager said.
I called the developer, responsible for the third-party search engine to ask if something has changed on his side. Apparently, he deployed a new version of the algorithm, which added some another 4x to our already quite fast integration. The insignificant API call that I mentioned in the beginning appeared to be significant too! Now we have a confirmed processing speed of 2ms per Search item, or about 3.5 million records per 24h. And both apps are still in DEV environments…
Thanks for reading!
See how I simulated the third party behaviour of this integration in the early stages of the develop...
Consider marking helpful and subscribe for my content, to get more articles like that, right in your inbox.
Martin Ivanov
2022 Community Rising Star
