Best Practices for Importing 3 Million Records into ServiceNow
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Hi,
I would like to understand the recommended approach and best practices for handling this volume of data efficiently.
Could you please share your suggestions on the following questions?
What is the recommended approach for importing 3 million records into ServiceNow?
Should we use Import Sets and Transform Maps, or is there a better approach?
What is the recommended batch size for importing such a large volume of records?
Are there any platform limitations or performance considerations we should be aware of?
If anyone has handled a similar high-volume data import, please share your approach, practical experience, and any relevant ServiceNow documentation or Community articles.
Thank you for your guidance!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago - last edited 3 weeks ago
Hello @Khalid9030 ,
Use Batch Processing: Split millions of records into smaller batches instead of processing everything at once.
Minimize Transformations: Avoid unnecessary Business Rules, workflows, and scripts during imports where safe.: Start with smaller batches, measure performance, and gradually increase the batch size.
Also refer this :
If this helps you then mark it as helpful and accept as solution.
Regards,
Aditya
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago - last edited 3 weeks ago
hi!
We have done similar imports in the past. First of all, I wouldn’t recommend doing the transformation on the ServiceNow side if you can avoid it. The less logic runs per record in ServiceNow, the faster the import.
- Do all cleanups and transformations before ServiceNow. Ideally the data arrives already in the target format, so the transform map is mostly 1:1 field mapping.
- I’d even move the queue management to a system before ServiceNow. There are several ETL products that handle this well (retries, error handling, throttling), and ServiceNow only receives clean, ready batches.
- Do the imports in batches and use the Robust Transform Engine. The ServiceNow docs also say that 10 sets of 100k finish faster than one set of 1M, and for this volume Concurrent Imports are worth a look.
A few more things I'd look out for:
- coalesce only on indexed fields, otherwise every row does a slow lookup
- don’t run several large imports at the same time, stagger them (and obviously run off business time)
- test with a smaller sample (e.g. 50-100k) in sub-prod first, measure it, then estimate the full runtime
There is a Community article from @Maik Skoddow exactly this topic, they used events and queueing:
https://www.servicenow.com/community/servicenow-ai-platform-articles/how-to-import-4-million-records...
ServiceNow docs with the import set performance guidelines:
https://www.servicenow.com/docs/r/zurich/application-development/manage-data.html