ServiceNow Duplicate CIs Solution / Script / Report

[Sohail Khilji]

 ServiceNow Duplicate CIs Fix

Facing Duplicate CIs Post CMDB Implementation ?

Solution: How to resolve CI duplication:

Duplicate CIs are encountered during the CMDB identification and reconciliation process. And once they are encountered, each set of duplicate CIs is grouped in a de-duplication task for remediation. The cause of this might be weak identification rules. There are a number of ways in which IT teams can detect and merge the duplicate CIs.

One way to detect duplicate CIs is by creating background scripts. For instance, you can check the unique field in the form which should be unique for each record of your table. It can be your mac address, IP address or asset tag. Run the script and you will find the dupes in front of you.

The second way is the most practiced; some discovery issues may be resolved by a one-time delete. You may delete a duplicate server record if it matches all the following criteria:
– No tasks are associated with the CI.

If tasks are associated with both of the duplicate records, merge these records using the following steps:

• i. Select the newer record as the record to be deleted.
• ii. Change the name of the record you plan to delete.
• iii. Merge all relevant data to the older record.
• iv. Delete the duplicate record.
• v. After deletion, revisit the duplicate CI after a few days to ensure that the issue did not recur. If so, continue troubleshooting.

During the CI identification method in the CMDB, duplicate CIs are determined by the following properties:

• glide.identification_engine.skip_duplicates: True by default.
• glide.identification_engine.skip_duplicates.threshold: set to 5 by default.

To modify these properties, we need to add them to the System Properties (sys_properties) table, where the numbers of duplicate CIs are detected. You can configure these properties so duplicated CIs are automatically reconciled, skipping duplication.

If glide.identification_engine.skip_duplicates is true, and the numbers of duplicate CIs are lesser than the threshold specified by glide.identification_engine.skip_duplicates.threshold, then the oldest of the duplicate CIs are picked as a match and gets updated.

The rest of the duplicate CIs are tagged as duplicates by setting the cmdb_ci’s discovery_source field as ‘Duplicate’. During matching, the identification engine filters out any CIs in which statediscovery_source=Duplicate. If glide.identification_engine.skip_duplicates is false, then matching of duplicate CIs fails with an error, and none of the duplicate CIs are updated.

ServiceNow Discovery

ServiceNow’s Discovery tool helps to identify the computers and devices that are connected to your network. It functions to configure the connected computers and devices with their current status and updates the CMDB.

Discovery tool enables IT teams, to take a more hands-off approach for their configuration management. Although it generates automated discovery in the IT environment, it does not eliminate manual entry. For instance, the IT teams may still require monitoring of some details such as renewal of hardware and more.

ServiceNow Discovery helps to prevent duplication of CIs. Each CI item records unique identified based field values that are specified in the CI identifier rule. The CI is identified as ‘duplicate’ if the field value of another record is of the same class. By default, the CIs of hardware class and its subclasses including computer, server, Unix server, Windows server, all use an OS serial number as the unique identifier.

For instance, when ServiceNow Discovery discovers a CI, the sensors and patterns use the internal function (SNC.IdentificationEngineScriptableApi.createOrUpdateCI) to send the CI to the CMDB. If there was already an existing CI with the same attributes for that particular class, it is overwritten; else a new CI is generated.

Duplicate CI Remediator

Enterprise can largely benefit from the ServiceNow Duplicate CI Remediator – a system wizard that guides through the reconciliation process. It aims to provide detailed information about duplicate CIs including the attributes, relationships, and related items to retain and to reconcile.

IT teams can leverage the CMDBDuplicateTaskUtils API to manually create a de-duplication task for duplicate CIs which are not configured to be detected in the system. After initiation of this step, you can then remediate those tasks using the Duplicate CI Remediator that is similar to remediating a system generated during the de-duplication task.

Benefits of resolving CI duplication

Resolving CI duplication should be one of the high priorities for organizations as it has an impact on the overall value-addition of the CMDB within an organization. One of the main benefits is that IT teams will be able to leverage populating configuration items into the CMDB, along with the following benefits:
Enterprises will benefit from improved processes such as incident management, change management, and problem management by providing an overall impact analysis of CI to their business applications.

Additionally, they will also benefit from reduced time to recovery of outages of applications because it will be easier to understand the application topology and all their dependencies.

Enterprises will be able to improve the effectiveness of monitoring tools by associating business impact with detected problems associated with infrastructure components.

Importantly, enterprises will be able to practice effective strategic planning with their data. It will be easier to move, consolidate, and optimize the hardware without any disruption to the business that depends on it. Moreover, enterprises will be able to better manage and control costs of IT assets since there is an accurate understanding of what assets are deployed and used.

Organizations will always be prepared with their data compliance and audit reports that are required from IT organizations. And finally, an organization will achieve better control of configuration changes that are needed for on-going maintenance of software and hardware.

What about future duplicates? 
You don’t want to be in the business of manually remediating every single duplicate.  Often, they are created much faster than a person or team could ever clean them up.  It’s important to be strategic and target the root cause.  Don’t start mopping up the water until you’ve fixed the leaky pipe.

• Ensure your identifiers are appropriate for each class
• Utilize Identification Simulation when possible to vet new rules

Understand duplicate CIs are a natural occurrence in a CMDB.  Having duplicates doesn’t mean you’re doing it wrong; it just means you’re paying attention to the data and appreciating how it impacts your processes, or more often, how it impedes your processes. 
 
How you handle these duplicates is one of the key parts of your CMDB where you succeed or fail. 

Make sure this is someone’s job:
 
This isn’t a nice-to-have, this is key to achieving next-level benefit across all of your CMDB-consuming processes.  If you went to the effort and expense of standing up a CMDB, then invest in keeping it clean and healthy.  
 
One of the tools you can use to help you with this is our CI Watch application by AJUVO.  Our CI Watch Usage identifies all records on the platform that are referencing specific CIs.  This is a critical piece of the puzzle.  
 
Deleting a duplicate without understanding where is was used can result in significant data loss.  If the stale CI was associated to operational tasks that data needs to be migrated.  If the CI has an asset associated with it, the asset will automatically be deleted.  By cleaning up one piece of bad data, you may inadvertently create another issue.  CI Watch not only finds these relationships, it also migrates them for you.  Why not take the guesswork out of your CMDB maintenance?

Script to get duplicate CIs : 

gs.print(getDuplicates('cmdb_ci_business_app','correlation_id'));
function getDuplicates(tablename,val) {
var dupRecords = [];
var gaDupCheck = new GlideAggregate(tablename);
gaDupCheck.addAggregate('COUNT',val);
gaDupCheck.addNotNullQuery(val);
gaDupCheck.groupBy(val);
gaDupCheck.addHaving('COUNT', '>', 1);
gaDupCheck.query();

while (gaDupCheck.next()) {
dupRecords.push(gaDupCheck[val].toString());  
}
return dupRecords;
}

More links to fix Duplicate CI:

https://community.servicenow.com/community?id=community_question&sys_id=c20a8369db5cdbc01dcaf3231f96...

https://community.servicenow.com/community?id=community_question&sys_id=27980f21db5cdbc01dcaf3231f96...

https://community.servicenow.com/community?id=community_blog&sys_id=3efc66a5dbd0dbc01dcaf3231f96193b

https://community.servicenow.com/community?id=community_blog&sys_id=f61d62e5dbd0dbc01dcaf3231f96197f

Kindly Consider Helpful or bookmark for future reference.

[PeterIkeda]

This is an excellent write-up, thank you for sharing. Just an FYI, the URLs at the end of the article need to be fixed.

[taahamushtaq]

It is written in easiest way. Can you help with any background script as an example to check the duplicates?

[Gyaneshwar_c01]

Hi Taahamushtaq,

You can use bellow script to find the duplicate record from any table.

gs.print(getDuplicates('cmdb_ci_business_app','correlation_id'));
function getDuplicates(tablename,val) {
var dupRecords = [];
var gaDupCheck = new GlideAggregate(tablename);
gaDupCheck.addAggregate('COUNT',val);
gaDupCheck.addNotNullQuery(val);
gaDupCheck.groupBy(val);
gaDupCheck.addHaving('COUNT', '>', 1);
gaDupCheck.query();

while (gaDupCheck.next()) {
dupRecords.push(gaDupCheck[val].toString());  
}
return dupRecords;
}

[Ahmet Damli]

very helpful article, just the links to more articles do not work anymore

[SaaniyaChugh1]

This is a great article Sohail. Thanks for sharing!