Multiple REST message calls from Servicenow to store data in one table

[Sunil25]

Hello Experts,

We have a requirement to pull data from third party application and store it in single table of Servicenow. Here the complexity is that the data required is not available with single web service. We need to query the app multiple times (around 20 to 30 times) with different end points to pull the data. Once the data is retrieved, need to process it and store it in required columns.

Eg: We need to pull the data by calling multiple API's which has different end points and parameters and capture the same in multiple columns like active, manager, user_name, email etc from sys_user table. Because they do not have single webservice which can provide all columns data in single call. So we need to create either multiple rest messages or scripts to retrieve the data.

I am worried on load and complexity on Servicenow to trigger multiple calIs. Is there any best approach how this data can be pulled with multiple calls and store in some staging table and thereby save it in target table. 

Thanks,

Sunil Safare

[Maik Skoddow]

This is a great question, and not many developers are concerned about "load and complexity".

Basically there is no special best practice approach for your scenario and to be honest, it is impossible to give a generic answer as the solution depends on many aspects which are not known here.

For example:

• Is a direct connection possible or must the requests be routed via MID server?
• How many outbound requests are expected for a certain time range?
• How much data will be retrieved with each of the requests?
• What should happen if one of the requests which has to be performed for a single record is failing? Start from the beginning? Leave everything as it is? 

[Sunil25]

Hi @Maik Skoddow 

Thanks for the inputs. Please find my answers to some of your questions.

• Is a direct connection possible or must the requests be routed via MID server? - The requests must go through Mid server
• How many outbound requests are expected for a certain time range? - Around 20 outbound calls are required to pull the data. 
• How much data will be retrieved with each of the requests? - Around 80K records
• What should happen if one of the requests which has to be performed for a single record is failing? Start from the beginning? Leave everything as it is? - Leave the record for that call and pull again in the next scheduled job.

I am thinking of below possible approaches technically to achieve this requirement.

1) Create multiple data sources() with transform maps and load the data in sys_user table one after another.

2) Use ETL to load the data into sys_user since IRE is supported for sys_user table from Washington DC release.

3) Create a custom table and load the data from different sources using Scheduled script and REST message function into it. Once all the data loading is completed, then transform it to sys_user table.

Could you please suggest the better approach or any best approach other than these.

Thanks,

Sunil Safare

[BillMartin]

Hi @Sunil25 ,

ServiceNow is built to scale for more power and performance requirements. This capability is already available out of the box to meet the needs of the largest Global 2000 enterprises.

ServiceNow data centers and cloud-based infrastructure is designed to be highly available, with redundant components and multiple network paths to avoid single points of failure.

For further reading: Delivering Performance and Scalability with The Now Platform

Please mark as helpful and accept as solution if you find it lucrative.

[BillMartin]

Hi @Sunil25 ,

Did you manage to get your answers yet? I created this post that gives you a picture of how performance testing and platform performance parameters are measured. Hope you find what you are looking for.

Here is How: Performance Engineering in ServiceNow

Here is How: ServiceNow TCOE