Attributes field in Reconciliation rules in IRE - does it actually work?

[mbourla]

We're currently looking into IRE (Identification & Reconciliation Engine), to potentially replace some quite clever custom solutions we have developed over the years to do much the same thing as IRE.  However I'm getting strange behaviour with the Attributes field on reconciliation definitions.

Initially, we were seeing a data source being able to update fields that by rights it shouldn't be able to, because those fields weren't included in the relevant reconciliation definition.  

Today I am seeing IRE blocking (rather than allowing) all updates by a data source when the Attributes field on the reconciliation rule is empty.  When in fact Attributes being empty is meant to mean that the data source is authorised to update ALL fields on the target record.  This is taken from https://docs.servicenow.com/bundle/madrid-servicenow-platform/page/product/configuration-management/reference/r_ReconciliationRulesPrinciples.html (it's the section headed "Authorization for all attributes in a table").

If I explicitly added all the attributes to that Attributes field, then it worked fine. Leaving it blank didn't work.

Below I've given the Output Payload from the call to cmdbUtil.getOutputPayload() that the Docs pages show to use, over two trial data imports using IRE in the onBefore transform map script.  The input data was the same, and I deleted the CIs created by the first run before doing the second run. The only difference was the Attributes field in the reconciliation rule.

Notice that in the first one (when the Attributes field was empty) the "maskedAttributes" field lists the 4 fields that it was trying to update, meaning it blocked the update. In the second one (when the Attributes field was not empty) the "maskedAttributes" field is empty, meaning it allowed the data source to update all fields on the target record.

Has anyone else seen this type of behaviour with the Attributes field on reconciliation rules not working as expected?

More generally, I'm really interested to hear feedback on people's experience with using IRE - how well does it work, any disadvantages or gotchas to be aware of, how easy to configure and operate etc.

Many thanks
Michael

Output payload with Attributes field empty

IE Output Payload: {"items":[{"className":"cmdb_ci_linux_server","operation":"UPDATE","sysId":"62f53e7ddbe7b380a20f09b1ba96191f","maskedAttributes":["short_description","asset_tag","name","host_name"],"identifierEntrySysId":"289075addba7ff40a20f09b1ba961918","identificationAttempts":[{"identifierName":"SP Linux Server","attemptResult":"MATCHED","attributes":["serial_number"],"hybridEntryCiAttributes":[],"searchOnTable":"cmdb_ci_linux_server"}]}],"relations":[]}

Output payload with Attributes field non-empty

IE Output Payload: {"items":[{"className":"cmdb_ci_linux_server","operation":"UPDATE","sysId":"339c72f9db2bb380a20f09b1ba9619f8","maskedAttributes":[],"identifierEntrySysId":"289075addba7ff40a20f09b1ba961918","identificationAttempts":[{"identifierName":"SP Linux Server","attemptResult":"MATCHED","attributes":["serial_number"],"hybridEntryCiAttributes":[],"searchOnTable":"cmdb_ci_linux_server"}]}],"relations":[]}

[mbourla]

OK, so with the help of Renan Coelhosilva from ServiceNow, I've found the answer.

Which is that the Attributes field being empty does indeed mean that the data source is authorised to update all fields on the target CI.  But not for fields which are explicitly specified in the Attributes field on reconciliation rules from other data sources that apply to this CI class. In that situation the update is blocked, without reference to data source precedence, and regardless of whether those other data source(s) have updated the CI.  That definitely isn't what the docs site says here.

When the same field appears in the Attributes field on multiple reconciliation rules, that's when IRE uses the data source precedence records to decide whether to allow the current field update to happen, depending on whether the current data source has greater precedence than the one that last was allowed to update the field.  Which, although I haven't actually checked that out, I assume must based on the Data Source History [cmdb_datasource_last_update] table, because that is the only place I can see where data sources are associated with CIs.

So in conclusion, the safest approach is to always explicitly select all the fields that a data source is authorised to update, in the Attributes field of the reconciliation record.  And avoid leaving that Attributes field empty.

Phew!!

[Jay118]

This is still relevant in 2025 on Yokahama. We are used to using IRE, but reconciliation rules are still crazy. I'm amazed at how the unexpected and undocumented behavior still has not been corrected or documented or clearly communicated by ServiceNow. It is still being ignored and swept under the rug. I suppose customers are just figuring it out on their own and dealing with it. I hope ServiceNow comes clean on this issue and explains how it really works. In addition to the above described problem, I've also encountered the problem that inherited rules always get overridden regardless of priority by child rules. It makes it so that you have to repetitively or duplicitously place all of your rules at the lowest child class and not use inheritance!

[mbourla]

Yes, we also encountered that "feature" of rules on child tables always overriding rules on parent tables.  Personally I think ServiceNow have just got the design wrong.  It makes sense where both reconciliation rules are for the same data source, but not when they're different. 

Out of interest, here's a community post I did back about it some years ago:

RE reconciliation rules on child tables override rules on parent tables - how do you overcome this?