Best Practices for Importing 3 Million Records into ServiceNow

Khalid9030
Tera Contributor

Hi,

I would like to understand the recommended approach and best practices for handling this volume of data efficiently.

Could you please share your suggestions on the following questions?

What is the recommended approach for importing 3 million records into ServiceNow?

Should we use Import Sets and Transform Maps, or is there a better approach?

What is the recommended batch size for importing such a large volume of records?

Are there any platform limitations or performance considerations we should be aware of?


If anyone has handled a similar high-volume data import, please share your approach, practical experience, and any relevant ServiceNow documentation or Community articles.

Thank you for your guidance!

6 REPLIES 6

glideFather
Tera Patron

ahoy @Khalid9030,

 

could you possibly share what kind of data is that? and if it is really important to have everything of that?

importing is one thing, please share the nature of the data (cmdb or something else?) and also be aware that indexing and time boxing would be necessary after you have the data in your instance to avoid/minimise the loading the data

@glideFather 

Thank you for your suggestions.

This is an interview question rather than an actual project requirement. I wanted to understand the best practices and recommended approach for importing 3 million records into ServiceNow, along with the performance considerations involved.

I agree that the nature of the data, whether all records are rquired, indexing, and timeboxing are important factors to consider. I would appreciate any additional suggestions on how to approach this scenario from a technical and performance perspective.

@Khalid9030,

 

there's no universal answer and the approach might be different based on the data.

 

If it is CMDB, you can import it with integration, if it was not 3M you could have imported that from XML, CSV, ... 

 

For an interview question, I would emphasise the fact that you think about WHAT data it is, WHO owns them, WHO guarantee the accuracy (in 3M records, it will be crucial to set the right conditions to know it is not a duplicate) and HOW to maintain it (is it one time import or scheduled regular updates - hourly, weekly, monthly, on demand...).

 

For 3M, I would perhaps try to review the data and lower the volume. E.g. if it was legacy incidents (silly example) then I would try to import only incidents from last 1 year not from last 15 years, something like this.

 

Indexing and time boxing was meant for already imported data when users will query them - not to load ALL data but somehow pre-filtered.

 

Leading practice about ANY data import: first you import smaller chunk of data (let's say 2000 out of 3M) to see how the data looks, whether all required attributes are populated and then you can slowly add more, rather than to go full volume at once and then realising that some mapping doesn't work for example :))

Hi @Khalid9030 , I echo @glideFather ,

 

Based on the nature of the data, the approach will be decided.

 

In general,

  1. the impact analysis needs to be done.. table level, reference fields, etc.,
  2. Based on the impact analysis, minimize the business rules, notifications,  (if possible)
  3. always use staging table (import set table) with optimized transform map and transform map scripts
  4. perform a pilot run and observe the behavior in sub-production instance via, file upload or importset table api to push from the source system or via Integrationhub connector
  5. Based on the results, split the records into chunks
  6. Iterate
  7. Done.

Hope this clarifies. I will definitely give the credits to @glideFather .

Have a great day!