- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
I'm exploring efficient ways to transfer millions of rows of table data to an external system and would appreciate your insights. What approaches have you found effective for handling such large-scale data transfers without compromising performance? I'm currently considering REST APIs as a potential solution, but I'm open to other suggestions. If possible, please share your experiences or recommendations in detail.
Thanks in advance!
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
As mentioned, in our integration third-party system was sending the data to MID Server and we were processing the file before we import to ServiceNow instance. We did this manually by splitting the CSV to 5 different files and loaded in import set tables and later to target tables. This approach was for initial data load and delta loads were handled as part of MID Server sending the data to instance on periodic basis and records would be inserted/updated
If you do not have Integration Hub license, I would recommend to go with export set as it is tried and tested method and would work fine
I hope you appreciate the efforts to provide you with detailed information. If my response helped to guide you or answer your query, please mark it helpful & accept the solution.
Thanks,
Bhuvan
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Hi,
Running REST API for millions of records may impact performance of the platform. You should have wait time in between (e.g. every 1000 records) to avoid overloading the system.
Also there are features like Instance Data Replication which are designed to avoid performance issue. You can also check whether this can be used
Palani
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Thanks Palani.
I agree that running REST APIs for millions of records could create performance issues, and adding wait times/batching is a good safeguard. Have you ever tried? Will it work for large set of data?
I’ll also explore the Instance Data Replication option you mentioned to see if it fits our use case. Feel free to share if have any resource if handy. 🙂
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Thanks Rafael. One quick question, is XML export/import practical for millions of rows, or is it generally better suited for smaller datasets/configuration data?
ServiceNow may run into memory and performance issues when exporting/importing multi-million rows as XML.
What I know XML export/import in ServiceNow is primarily used for configuration data, updates, or smaller datasets. Will it not?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
We have done import of ~4.5 million records from a third-party system to ServiceNow using import set via MID Server. Export set would work similarly and you would need to export file from ServiceNow to MID Server which in turn would be picked up by third-party.
This operation takes hours and would recommend you to test the maximum records that can be exported in a single transaction. We tested bulk import using many methods and on analysis, we noticed when number of records cross ~950,000 we run into issues. So we split the CSV file to max 900,000 records and imported it. Do similar exercise in Development environment to simulate the number of records to be exported. Schedule this activity during offline hours so that there are no resource bottlenecks during this export set.
Another option you can try to use is data stream. Typically we use data stream to process large number of records from external system to ServiceNow. See if it is possible other way around and third-party can consume data from ServiceNow.
As per community guidelines, you can accept more than one answer as accepted solution. If my response helped to answer your query, please mark it helpful & accept the solution.
Thanks,
Bhuvan
