Large CMDB Data Import Best Practices

[rohitservicenow]

We have our CMDB hosted on Remedy and we need to pull that data very often (every day) to ServiceNow using the integration. We receive around a million records in each file that we receive from Remedy and we are using the Import Sets and Transform Maps to copy the data to CMDB table. Since we import a large data every day, its impacting the system's performance as our company has presence across the globe and our helpdesks work 24/7.

Wondering is there any best practice where the Imports don't take much time and the system's performance is not affected.

Thank you in advance.

[Maik Skoddow]

Hi @rohitservicenow 

hopefully, you do not just import your CIs via import set & transform maps without running the Identification and Reconciliation Engine (IRE) as you will get many issues with duplicated CIs a.s.o.

See the following documentation page, which explains how you can run the IRE also in a common transform map approach: https://docs.servicenow.com/bundle/vancouver-servicenow-platform/page/product/configuration-manageme... 

The better approach would be an import via Integration Hub ETL, but I have no experience with such large data amounts and how it could affect your instance

For importing millions of records, I use another architectural approach I have described at How to import 4 million records in 3 hours

Maik

[Karim-hashish]

You should have a messaging hub like rabbitMQ, remedy to publish CI updates only and then you consume on ServiceNow using import sets and IRE