How are duplicate CI's identified?

[jaredscharf]

Are duplicate CI's identified as data is processed through the IRE? Or is there a job that runs daily that looks at the cmdb data and compares it to identification rules to identify duplicates? I'm asking because we have recently modified our identification rules, but there currently are many duplicate CI's/duplicate CI tasks in the CMDB. Ideally what I would like to do is delete the duplicate CI tasks table and have duplicate CI's identified under the new identification rules. What is the best way to do this?

[M Iftikhar]

Hi @jaredscharf, 

Duplicate CIs are identified in real-time as data is processed by the Identification and Reconciliation Engine (IRE), not by a separate daily job. A de-duplication task is created at the moment the IRE receives data that matches multiple existing CIs based on your active identification rules. For detecting duplicate CIs you can refer to the given documentation link:

https://www.servicenow.com/docs/bundle/zurich-servicenow-platform/page/product/configuration-managem... 

Thanks & Regards,
Muhammad Iftikhar
If my response helped, please mark it as the accepted solution so others can benefit as well.

[Mannapuram]

@jaredscharf  Duplicate CI's are created through IRE. There is no job running to identify duplicate CI's. When the CI goes thorough IRE, if IRE finds another CI with most of the details, then a De-Duplication task is created in table - 'reconcile_duplicate_task'.

For your case, it's advisable not to delete. Instead, you can change the status to 'Closed Incomplete' manually. When you are done with updating all the existing tasks, then re-run the data load (to enable the IRE). 

[jaredscharf]

Here's my confusion: If the purpose of identification rules is to prevent duplicate CI's, then why would a duplicate CI get created in the first place? Shouldn't the identification rule prevent duplicate CI creation instead of identifying a duplicate according to its rules?

[Mannapuram]

@jaredscharf  Consider a case, there were couple of manual entries in CMDB for Servers. However, they didn't have all the attributes of identification rule. Later, discovery is installed and run. IRE checks for the attributes in the identification rule and if all the attributes doesn't match, then CI is allowed. For the existing manual CI, all the attributes doesn't exist. Thus, while entering CI, in the initial comparison of Identification rule attributes, it got through. 

However, de-duplication task is created by next day, as the name matches between them. This gives you choice to either keep both OR make them into one CI.

Similar cases are possible between two different discovery sources. The purpose of IRE is to avoid duplicate. But, due to some scenario's duplicates do exist.