How to create a schedule job to remediate all duplicate CI's at once ?

[soniarajsin]

HI,

In cmdb dashboard i am able to find duplicate records for different CMDB classes but i have around 5000 duplicate records So, i want to create a schedule jo to Remediate all CI's at once 

Is there any recommendations or tips ??

[Amitoj Wadhera]

Hello @soniarajsin ,

1. Plan and Think Before You Automate
Know What Makes a Duplicate: Make sure you understand what counts as a duplicate record—for example, things like matching serial numbers, asset tags, or names.
Sort Your Data: Check out the CMDB Health Dashboard or create a custom report to help you find and sort out duplicate records by their class or other details.
Focus on Important CIs First: Tackle the most important CIs first—like servers or applications—before dealing with those that matter less.
2. Use a Scripted Method for Fixing Issues
Write a Script Include: Create a Script Include that will address the logic behind fixing duplicate CIs. This script could:
Combine records smartly.
Keep the most accurate or complete entry.
Label duplicates for archiving or removal.
Set Up a Scripted Job: Use the Scheduled Script Execution module to run this script automatically during quieter times.
3. Be Careful with Relationships
Make sure that any related information (like relationships, dependencies, or child CIs) is properly assigned to the record you’re keeping before you remove duplicates.
4. Set Rules for Resolving Conflicts
Make rules for deciding which record to keep:
Keep the CI that has been updated most recently.
Keep records that have all their fields filled out or specific characteristics (like a status of "Active").
If you are gathering data from multiple sources for your CMDB, use CMDB Reconciliation Rules.
5. Use CMDB Query Builder to Find Targets
Create a query with the CMDB Query Builder to find duplicate records in real-time.
Double-check the results to make sure you don’t get any false positives.
6. Utilize CMDB Data Manager
If your CMDB Data Manager is active, set it up to automatically follow lifecycle rules that archive or delete old or duplicate CIs.
7. Keep Records and Notify Others
Record all actions taken (like records merged or deleted) for auditing reasons.
Inform the necessary stakeholders about large changes to maintain transparency and quickly resolve any issues.
8. Start with a Test Run
Run your logic on a smaller group of records first to make sure your method works properly without disrupting business activities.

Here’s a sample script to help you get started:

(function executeScheduledScript() {
    var duplicateCIs = new GlideRecord('cmdb_ci'); 
    duplicateCIs.addEncodedQuery('your_duplicate_query'); 
    duplicateCIs.query();

    while (duplicateCIs.next()) {
        var ciToKeep = getRecordToKeep(duplicateCIs); 
        resolveDuplicate(duplicateCIs, ciToKeep); 
    }
})();

function getRecordToKeep(duplicateCIs) {
    return duplicateCIs.sys_id; 
}

function resolveDuplicate(duplicateCIs, ciToKeep) {
    // Update relationships, archive, or delete other duplicates
    // Example: duplicateCIs.deleteRecord();
}

If you find my response helpful, please consider marking it as the 'Accepted Solution' and giving it a 'Helpful' rating. Your feedback not only supports the community but also encourages me to continue providing valuable assistance.

Thanks,

Amitoj Wadhera

[yuvarajkate]

To remediate duplicate Configuration Items (CIs) across CMDB classes using a scheduled job, follow these recommendations:

1. Identify Duplicates: Use CMDB queries to identify duplicates, possibly leveraging the Duplicate CI module or building a script with conditions like matching serial numbers, names, or other unique attributes.

2. Create a Remediation Script: Write a script to handle duplicates, deciding on criteria like:

   • Keeping the most recently updated record.
   • Merging information from duplicates.
   • Deleting unnecessary duplicates.

3. Schedule the Job: Use a Scheduled Script Execution (formerly Scheduled Jobs) to run the remediation script during low-traffic hours. Navigate to System Definition > Scheduled Script Executions to create one.

4. Test Before Production: Always test the script in a sub-production environment to ensure no unintended data loss.

Script:

var gr = new GlideAggregate('cmdb_ci');
gr.addAggregate('COUNT', 'serial_number'); 
gr.addHaving('COUNT', '>', 1);
gr.query();
while (gr.next()) {
    var dupGr = new GlideRecord('cmdb_ci');
    dupGr.addQuery('serial_number', gr.getValue('serial_number')); 
    dupGr.query();
    while (dupGr.next()) {
        // Add logic to delete/merge duplicate records
    }
}

[MrSmitty]

I would try to use the new De-duplicate template feature to de-duplicate the ones you wish to remove.  I had an issue recently where I had an explosion of duplicates that needed remediation and this feature saved me a lot of time.