Outbound REST Message Pagination

[MdA3]

Hi Folks,

Good day!

I'm integrating a third-party application. In the response I'm getting around 2 lacs record at one run.

How to process these many records or how to do pagination. I have created a custom action through which I'm calling my rest message and using JSON parser step to parse the response.

And then calling this action to a flow which will runs daily and ingest data to a table. 

[Bhuvan]

@MdA3 

Is this a one time activity to pull these many records or you expect this high volume on daily basis ?

I would recommend to import the records initially using import sets and do a delta load on daily basis for new/updated records from third party application to ServiceNow rather than bulk upload on daily basis. If it is mandatory to use Flow Designer, look at option of data stream rather than custom action.

If this helped to answer your query, please mark it helpful & accept the solution.

Thanks,

Bhuvan

[MdA3]

Thanks @Bhuvan  for your quick response.

This is not a one-time activity; we are expecting this much daily on daily basis. 

Is there another way also to implement if not the custom action? As using data stream requires subscriptions if I'm not wrong.  

Even ingesting delta records to system, the action getting stuck and failed.

[Bhuvan]

@MdA3 

We have performed bulk import using import set API. I have provided more details in below thread

https://www.servicenow.com/community/developer-forum/bulk-data-transfer-to-external-system/m-p/33747...

I would recommend to do initial bulk load and going forward, do a delta load on daily basis to limit the number of records being created and updated. Best practices and recommendations are shared in detail in above link.

If this helped to answer your query, please mark it helpful & accept the solution.

Thanks,

Bhuvan

[Ankur Bawiskar]

@MdA3 

it all depends on whether the 3rd party app supports pagination or not.

If yes ask them to share how it can be used via REST and then use that logic.

If they don't have then you can use Data Stream action in flow to handle large responses

check these links

Data Stream actions and pagination 

Using Data Stream Action to ingest large datasets into ServiceNow -> hands-on details

ServiceNow Datastream Helps Conquer Data Challenges Like Never Before! 

If my response helped please mark it correct and close the thread so that it benefits future readers.