Why is IRE setting duplicate_of on CI creation, or setting duplicate_of to an existing duplicate CI

[mbourla]

Hi all,

We have multiple data feeds into our CMDB, and they all go through the Identification and Reconciliation Engine (IRE). As many customers must also do, we constantly get duplicate CIs being created, which we deduplicate in the normal way. But I have seen some behaviour in our instance which I cannot understand.  It created duplicates when I didn't think it would.  I can't understand why it did that, and I'm really hoping someone can explain it for me!  Incidentally, all these CIs use the same CI Identifier, on cmdb_ci_hardware.  There are two related questions - see below.

Thanks
Michael

(1) IRE setting duplicate_of on CI creation

When processing data from a particular data feed, IRE couldn’t match to an existing CI and so created a new CI.  But at the point of creation, IRE set the duplicate_of field. I know it was done at the same time, because in the CI’s audit history the change to duplicate_of has Update Number of zero, meaning on CI creation.

If IRE knew that the record matched an existing CI according to one of the Identifier Entries, it should have matched to that CI and updated it. But instead it created a new CI.

So the question is, why and when would IRE set duplicate_of on a new CI?

(2) IRE setting duplicate_of to an existing duplicate

On a different occasion, again the same data feed created a new CI. It would have matched to an existing CI by several Identifier Entries on the CI Identifier, but that existing CI had non-empty duplicate_of at the time so IRE should not match to it.  But IRE *did* set duplicate_of on the new CI at the point of creation (i.e. when Update Number is zero in the sys_history_line record), and it set duplicate_of to the CI that itself had non-empty duplicate_of.

Why and when would IRE set duplicate_of to a CI that itself had non-empty duplicate_of?

[Ashok Sasidhara]

Based on what is mentioned here, it seems to be against the expected behavior. Full analysis of such a scenario can be done only by seeing the data involved. It's better to raise a hi-ticket explaining the same details and also mention the URL of such records in the case.  

[mbourla]

Many thanks, Ashok.  Actually I already have a HI case open for this, with all the details of the CIs concerned, but it's proving hard to get answers.  So I thought maybe it would be useful or better to post it on Community and see if other people had had the same behaviour, or knew why IRE should be doing this.  

[Pratiksha]

If the CI's been recently created raise the case. In our case when we raised the case for similar behavior we have told told as discovery status is not present they can't debug. Asked as to delete the ci and run the discovery again (test env) 

And it did not create duplicate again. I could show then CI history and everything but they did not agree. 

I am just posting the case notes which might help someone. 

As discussed in our meeting, the CMDB Correctness job will not flag the CIs as duplicates, if both of them had a Discovery source populated.
This pre-requisite is metioned on :
KB0726425 - CMDB Health - Duplicate Metric algorithm \:
"When CMDB Health - Correctness job runs, the Duplicate Metric only checks CIs that have Identification Rules defined, and also meet one of below conditions:
- One of the duplicated CIs has empty Discovery Source attribute"

The records that were discussed in PRD were created in 2023 and some of them became duplicates after the serial number was updated manually.

KB0869861 - Duplicate CMDB CI records

[mbourla]

Hi Pratiksha, many thanks for coming back on this.  I do actually have a HI case open for this, but it's being slow getting to an answer, so I thought posting on Community might be a better way to see if this is known IRE behaviour.  

Regarding that info from your case, it's different really.  That's about the CMDB Health jobs flagged CIs as duplicates, whereas the CIs I've got were flagged as duplicates at the point of creation.  That's what I just can't understand.  I do have one idea about when that might happen, which is if the IRE payload specified a sys_id but it was invalid and didn't appear on any existing CI.  I'm wondering if in that case that would cause IRE to immediately create a new CI without evaluating the CI Identifier rules.  But then IRE might then realise that the new CI was a duplicate based on serial number, for example, and so set the duplicate_of field.  I haven't checked that out so it might not even work like that.  And in any case, for the data source that created those CIs in our instance, it doesn't supply sys_id in the IRE input payload, so it couldn't be the cause anyway.

So, I'll keep on with my HI case and hopefully will get to the bottom of the issue at some stage soon.

Regards

Michael