Performance impact of 450,000 REST calls to create problem task records?

[John Francis]

What is the expected performance impact to our Production instance if we have an external system sending 450,000 REST messages to insert problem task records?     A stress test on our Dev instance today suggests that it would take 32 hours to load.   We know Production would be faster, but not sure by how much.

Also, if we were to multi-thread the sending process to send up to 100 REST messages concurrently to insert problem task records, would there be any performance benefit or concerns?

Thanks in advance for any advice.

[tim_broberg]

When i investigated this for mid server API requests, I found that things started backing up when the rate hit 100 requests per second per node with the default setup of 4 API_INT semaphores per node.

So, for example, if you had 8 nodes, your peak throughput would be about 450,000 requests / 8 nodes / 100 requests per second per node / 60 seconds / minute ~= 10 minutes, assuming you manage to connect to every node at once, and that the SQL itself doesn't take a long time. ...but then, I would expect it to take about 75 minutes just running against one node on your dev instance, so perhaps this is just a heavier operation than my rule of thumb covers.

Do you have timing information on the node log?

There is a set of worker threads, 4 by default, which feed off a queue whose length is 50 by default. If you overload that queue, the node will start sending 429 errors to requests. So, I wouldn't advise sending blocks of 100. You might try juggling 20? You could also try connecting with multiple clients, and manipulate the load balancer cookie such that you get something like an even distribution of clients across nodes - connect, if it's a cookie you know, retry, if not start running.

Crazy, I know, but desperate times...

[david_loo]

To increase performance, you may want to insert multiple objects with 1 POST, see the insertMultiple example

JSON Data Modification API

Good luck!

David

p.s. we solve the performance problem by installing an app that does a "pull" or "GET" instead.