Bulk data transfer to External System

Jesufair J · 3 weeks ago

I'm exploring efficient ways to transfer millions of rows of table data to an external system and would appreciate your insights. What approaches have you found effective for handling such large-scale data transfers without compromising performance? I'm currently considering REST APIs as a potential solution, but I'm open to other suggestions. If possible, please share your experiences or recommendations in detail.

Thanks in advance!

Bhuvan · 3 weeks ago

@Jesufair J

As mentioned, in our integration third-party system was sending the data to MID Server and we were processing the file before we import to ServiceNow instance. We did this manually by splitting the CSV to 5 different files and loaded in import set tables and later to target tables. This approach was for initial data load and delta loads were handled as part of MID Server sending the data to instance on periodic basis and records would be inserted/updated

If you do not have Integration Hub license, I would recommend to go with export set as it is tried and tested method and would work fine

I hope you appreciate the efforts to provide you with detailed information. If my response helped to guide you or answer your query, please mark it helpful & accept the solution.

Thanks,

Bhuvan

View solution in original post

Ankur Bawiskar · 3 weeks ago

@Jesufair J

you can use Export Set feature and place the file (excel, csv etc) to mid server.

Ask the customer's team to grab that file from mid server location.

You can schedule your export and configure the table, field, frequency etc for the push

Export sets

OR

Expose a Scripted REST API and endpoint and ask 3rd party to pull the information based on your filter conditions for the table

If my response helped please mark it correct and close the thread so that it benefits future readers.

Regards,
Ankur
✨ Certified Technical Architect || ✨ 9x ServiceNow MVP || ✨ ServiceNow Community Leader

Jesufair J · 3 weeks ago

Thanks Ankur for the reply.

For the Export Set via MID Server option, since you mentioned exporting as a CSV or file, is it practical to handle millions of rows in a single file? Or would you recommend splitting into smaller files for better performance and reliability?

In your experience on Scripted REST API approach, how does it scale for very large datasets (millions of rows)? Would you recommend streaming/pagination over returning bulk payloads to avoid timeouts?

Ankur Bawiskar · 3 weeks ago

@Jesufair J

with export set you can configure the export definition and push only the delta update to MID Server.

Yes CSV file should handle large data

Also for Scripted REST API yes pagination will help to avoid performance impact

If my response helped please mark it correct and close the thread so that it benefits future readers.

Regards,
Ankur
✨ Certified Technical Architect || ✨ 9x ServiceNow MVP || ✨ ServiceNow Community Leader

Ankur Bawiskar · 3 weeks ago

@Jesufair J

Thank you for marking my response as helpful.

As per new community feature you can mark multiple responses as correct.

If my response helped please mark it correct as well so that it benefits future readers.

Regards,
Ankur
✨ Certified Technical Architect || ✨ 9x ServiceNow MVP || ✨ ServiceNow Community Leader