IntegrationHUB ETL is creating duplicates

[victor21]

Hi Listers,

I am trying to use IntegrationHUB ETL - Import application to load some data into the CMDB from an Excel file. I see that when I run the Import multiple times I get duplicates in the CMDB - seems the application is not using IRE as documented. I enabled IRE logging but did not get any entry.

Can someone enlighten me unto what I am doing wrong or how to debug?

Thanks & regards,

Victor

[Ashok Sasidhara]

I would suggest to do the following analysis:

1. Identify the CI classes for which duplicates are getting created

2. Open each of these CI class from the CI class manager and check their current identification rules

3. Compare the identifier entries for each CI class against the data which you entered in the excel file for that class. If you find that this is the reason for duplication, then you need to either refine the identification rules or modify the data in excel file to prevent duplication.

If the source data and/or identifier entries are not found to be the reason for duplication, then it will be better to raise a hi-ticket as Integration hub ETL is supposed to follow the IRE.

[victor21]

Thanks for your suggestions.

I have already did all these - I did simulations from Identification Simulation application for the data from the excel sheet and I could see from identification engine log that the CIs were correctly identified.

What is bothering me is that I do not receive any identification engine logs when I use run the IntegrationHUB ETL job.

Is there anything else that I need to set?

Thanks,

Victor

[CMDB Whisperer]

An important point to note here is that just because you are getting duplicates does NOT mean the application isn't using IRE.  There are several reasons why an application that uses IRE may still result in the creation of duplicates:

1. Your identification rules may not properly defined.
2. You may have multiple identification rules that are presenting a conflict.  For example, you have two different serial numbers associated with two different devices that have the same name.  Since serial number takes precedence over name, the identification rule matches each item separately, and they are both created with the same name.  However, because name is also a valid identifier, both items are flagged as duplicate CIs.
3. There could be bad data causing this.  For example, to extend the previous scenario further, if your source data contains dummy values for serial numbers like "Missing" or "Unknown", or the devices have generic names like "Lenovo Thinkpad" ServiceNow will still consider those as unique identifiers.

[victor21]

Thanks for your response.

I am running the ETL for the same set of data where at least one identifier is present. My simulations show that they are being identified - and yet I see duplicates