--- sourceDocument: Australia ServiceNow AI Platform Capabilities sourceDocumentLink: https://www.servicenow.com/docs/r/australia/servicenow-platform Release : - australia ft:locale : - en-US ft:publication_title : - Australia ServiceNow AI Platform Capabilities ft:clusterId : - platcap bundleId : - platcap workflow : - Platform --- # Detecting duplicate CIs # Detecting duplicate CIs {#ariaid-title1} Release version: Australia Updated March 12, 2026 ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 7 minutes to read Summarize ![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete. ## Summary of Detecting duplicate CIs The Identification and Reconciliation Engine (IRE) in ServiceNow detects and manages duplicate Configuration Items (CIs) by grouping them into de-duplication tasks. These tasks enable review and remediation of duplicates to maintain CMDB integrity. Duplicate CI handling behavior depends on configurable system properties and the volume of duplicates detected during IRE processes. Show full answer Show less ## Key Features * **Automatic reconciliation of duplicates:** Controlled by `glide.identificationengine.skipduplicates` (default true) and `glide.identificationengine.skipduplicates.threshold` (default 5). If enabled and duplicates are below the threshold, the oldest CI is updated and designated as the main CI; others are marked as duplicates. * **Duplicate tagging and filtering:** CIs tagged as duplicates are excluded from matching in subsequent identification runs. If any attribute other than `duplicateof` is updated, the duplicate status is cleared. * **Handling of large duplicate volumes:** Governed by `glide.duplicateciremediator.max.cis` to manage performance and processing. * **Duplicate detection in related and lookup tables:** By default, IRE does not create de-duplication tasks for duplicates in lookup (e.g., Serial Number) or related tables (e.g., Software Instance), but this can be enabled by adding entries in the `cmdbduplicatelookuprelatedallowedclass` configuration table. This allows organizations to extend duplicate remediation to these areas. * **Configurable duplicate detection in lookup-based identification:** The property `glide.identificationengine.lookupmatch.createduplicatetaskci.enabled` controls whether de-duplication tasks are created for duplicates found in lookup tables referencing multiple CIs, allowing organizations to tailor remediation based on their specific validation needs. ## Practical Implications for ServiceNow Customers * Customers can fine-tune how duplicates are handled during CI identification by adjusting key system properties to either automatically reconcile duplicates or require manual remediation. * Extending de-duplication task creation to related and lookup tables increases CMDB accuracy by addressing duplicates beyond primary CI tables. * Understanding the impact of properties controlling duplicate detection during lookup-based identification helps align the process with organizational policies on what constitutes a duplication issue. * De-duplication tasks are always created regardless of automatic reconciliation, ensuring visibility and control over duplicate CI remediation. ## Next Steps * Review and configure the relevant system properties based on your organization's tolerance for duplicates and remediation workflow preferences. * Consider populating the `cmdbduplicatelookuprelatedallowedclass` table if you want to manage duplicates in lookup or related tables actively. * Use de-duplication tasks to review and remediate detected duplicate CIs, ensuring data quality in your CMDB. * Refer to the Duplicate CIs remediation documentation for detailed guidance on resolving duplicate CI tasks and using the main CI effectively. When IRE identification process detects duplicate CIs, it groups each set of duplicate CIs into a de-duplication task for review and remediation. A large number of duplicate CIs might be due to weak identification rules. You can configure the identification engine to reconcile duplicate CIs. During Identification and Reconciliation Engine (IRE) processes, handling of duplicate CIs is determined by the properties glide.identification_engine.skip_duplicates (set to true by default) and glide.identification_engine.skip_duplicates.threshold (set to 5 by default), and on the number of duplicate CIs that are detected. You can configure these properties so that duplicate CIs are automatically reconciled, skipping duplication. * If glide.identification_engine.skip_duplicates is true, and the number of duplicate CIs is less than the threshold specified by glide.identification_engine.skip_duplicates.threshold, then the oldest of the duplicate CIs is picked as a match and gets updated. That oldest duplicate CI also becomes the main CI for that set of duplicate CIs. The rest of the duplicate CIs are tagged as duplicates by setting their duplicate_of attribute to the appropriate main CI. During matching, IRE filters out any CI that is tagged as duplicate of any CI. * If glide.identification_engine.skip_duplicates is false, then matching of duplicate CIs fails with an error, and none of the duplicate CIs are updated. {#id-detect-dup-ci__ul_tnw_fcy_mcb} Also, the glide.duplicate_ci_remediator.max.cis property determines de-duplication processing for a large number of duplicate CIs. For more information, see the 'Large number of duplicate CIs' section in the [Duplicate CIs remediation](https://www.servicenow.com/docs/gSpynTBDyN19wsPvnN1TsQ "Duplicate CIs in the CMDB unnecessarily overload the system and interfere with the CMDB's integrity and reliability. It is therefore important to regularly use CMDB tools that detect and remediate duplicate CIs.") topic. In either case, de-duplication tasks are always created. Note: For a duplicate CI, if any of the CI's attributes, other than duplicate_of, is updated by IRE processing, then the CI is no longer considered a duplicate CI. In that situation, the value of duplicate_of is cleared in the CI. For more information about these properties, see [Properties for Identification and Reconciliation](https://www.servicenow.com/docs/HEqbR2g66O8HVoWp93RQrQ "Use the Identification and Reconciliation properties to configure the identification and reconciliation engine (IRE)."). ## Detecting duplicate CIs in related and lookup tables {#id-detect-dup-ci__section_nvy_yss_dhc} During CI identification, IRE might detect duplicate lookup items (Serial Number \[cmdb_serial_number\] table), or duplicate related items (Software Instance \[cmdb_software_instance\] table). By default, IRE doesn't create de-duplication tasks for these duplicate items. However, you can modify this default behavior by configuring IRE to generate de-duplication tasks for specific lookup or related tables, allowing you to then remediate any duplications. Add entries in the CMDB Deduplication Task Lookup or Related item allowed classes \[cmdb_duplicate_lookup_related_allowed_class\] configuration table for each table for which you want IRE to generate de-duplication tasks during identification: * Add an entry for any lookup table in an identifier entry in the respective identification rule. * Add an entry for any related table in a related entry in the respective identification rule. {#id-detect-dup-ci__ul_hx1_qjb_2hc}In the base system, this configuration table has no records, in which case IRE doesn't generate de-duplication tasks for any duplications in lookup or related tables from the identification rule. When detecting duplication in related or lookup tables, IRE will generate de-duplication tasks only for those tables with an active entry in the configuration table: 1. Ensure that you have the required role for the cmdb_duplicate_lookup_related_allowed_class table. * To add or update records: sn_cmdb_admin, cmdb_dedup_admin, or itil_admin * To read or create reports: sn_cmdb_editor, sn_cmdb_admin, itil, itil_admin, cmdb_dedup_admin {#id-detect-dup-ci__ul_uvt_vkc_2hc} 2. In the navigation filter, enter cmdb_duplicate_lookup_related_allowed_class.list. 3. Select New and fill out the CMDB Deduplication Task Lookup or Related item allowed classes table form.{#id-detect-dup-ci__table_gzj_bjt_dhc__entry__2} | Field | Description | |-|-| | Active | Is the entry active. | | Lookup or related item class | The lookup table (such as Serial Number \[cmdb_serial_number\]) or related table (such as Software Instance \[cmdb_software_instance\]) for which you want IRE to create de-duplication tasks when detecting duplicate records. | | Identifier table | The identification rule class, such as Hardware \[cmdb_ci_hardware\] or Operational Technology (OT). | [ ] {#id-detect-dup-ci__table_gzj_bjt_dhc} {#id-detect-dup-ci__ol_tsf_q3t_dhc} The CMDB CI Class Models ServiceNow® Store app adds class models that extend the CMDB class hierarchy, and contains meta data such as identification and reconciliation rules for the added classes. When specifying class models in this app, you can also include the cmdb_duplicate_lookup_related_allowed_class table with entries specific to the added tables, such as the Operational Technology (OT) table. In which case, IRE will generate de-duplication tasks as configured in the table. For more information, see [CMDB CI Class Models app](https://www.servicenow.com/docs/darqAyORPOpoURCcjTGVXA "The CMDB contains base-system classes that store data about configuration items (CIs). The CMDB CI Class Models ServiceNow Store app adds class models that extend the CMDB class hierarchy, including class descriptions, identification rules, identifier entries, and dependent relationships if applicable."). You can also add records to the cmdb_duplicate_lookup_related_allowed_class configuration table for added classes after the CMDB CI Class Models app has been installed. ## Detecting duplicate referenced CIs during lookup-based identification {#id-detect-dup-ci__section_unn_yjr_xgc} In a lookup-based identification (lookup identifier entry in the identification rule for a class), IRE might detect duplicate related items in the lookup table. For example, two records in which the serial numbers are identical but each referencing a different CI. In this situation, IRE selects the record with the older referenced CI as a match in the identification process. Lookup items matching multiple CIs might be valid in some organizations and invalid in others. You can configure IRE behavior of creating de-duplication tasks to remediate such duplications, by using the glide.identification_engine.lookup_match.create_duplicate_task_ci.enabled system property. Add and then set the property to reflect on whether such duplications are valid or not in the organization, as follows: false (default) : IRE selects the older CI as a match but doesn't create a de-duplication task to handle the duplication. Use this setting when such duplication isn't considered a duplication problem. For example, records with identical serial numbers where each is referencing a different CI. However, each serial number actually represents a different type of hardware, and therefore, this situation isn't considered a duplication problem requiring reconciliation. true : IRE selects the older CI for the matching process and generates a de-duplication task to remediate the duplication. Use this setting when such duplication is considered a problem that requires remediation. IRE doesn't create de-duplication tasks even when glide.identification_engine.lookup_match.create_duplicate_task_ci.enabled is true, in the following circumstances: * Enforce exact count match is set to false. For information about where it's set, see the advanced options in the 'Use attributes from another table (Lookup table)' option in [Create a CI identification rule](https://www.servicenow.com/docs/XeHgHGmCieUlQ6rwpgEmdA "Identification rules are used to uniquely identify CIs in the CMDB, as part of Identification and Reconciliation (IRE) processes. Each CMDB class can be associated with a single identification rule."). * During identification, the number of matching CIs in the lookup table is greater than 1. {#id-detect-dup-ci__ul_o3z_tcg_ygc} See the following examples of lookup records, input payloads, and the results of identification processes: * Example 1Lookup table cmdb_serial_number contains duplicate lookup records and multiple CIs matching both lookup items ABC123 and DEF456: {#id-detect-dup-ci__table_ecw_xdg_ygc__entry__2} | Serial Number | Configuration Item | |-|-| | ABC123 | CI1 | | ABC123 | CI1 | | ABC123 | CI2 | | DEF456 | CI1 | | DEF456 | CI2 | | DEF456 | CI2 | [ ] {#id-detect-dup-ci__table_ecw_xdg_ygc} var payload = { "items": [ { "className": "cmdb_ci_linux_server", "lookup": [ { "className": "cmdb_serial_number", "values": { "serial_number": "ABC123", "serial_number_type": "system", "valid": "true" } }, { "className": "cmdb_serial_number", "values": { "serial_number": "DEF456", "serial_number_type": "system", "valid": "true" } } ], "values": { "name": "CI_New" } } ], "relations": [] }; var jsonUntil = new JSON(); var input = jsonUntil.encode(payload); var output = SNC.IdentificationEngineScriptableApi.createOrUpdateCIEnhanced("ServiceNow", input, {}); gs.print(output); {#id-detect-dup-ci__table_bls_v2g_ygc__entry__3} | Exact count match | Property to create de-duplication task = true | Property to create de-duplication task = false | |-|-|-| | True | * CI1, the oldest, is matched and updated. * De-duplication task is created for CI1 and CI2 (Linux Server class.) * De-duplication task is created for ABC123/system \& CI1 (Serial Number lookup table). * De-duplicate task is created for DEF456/system \& CI2 (Serial Number lookup table). {#id-detect-dup-ci__ul_ocb_y2g_ygc} | * CI1, the oldest, is matched and updated. * De-duplication task isn't created for CI1 and CI2. * De-duplication task is created for ABC123/system \& CI1 (Serial Number lookup table). * De-duplicate task is created for DEF456/system \& CI2 (Serial Number lookup table). {#id-detect-dup-ci__ul_pcb_y2g_ygc} | | False | Same as above | Same as above | [Table 1. Identification results] {#id-detect-dup-ci__table_bls_v2g_ygc} * Example 2No duplicate lookup records are found in the lookup table cmdb_serial_number, but there are 2 CIs each matching ABC123 and DEF456 respectively: {#id-detect-dup-ci__table_h1f_2sg_ygc__entry__2} | Serial Number | Configuration Item | |-|-| | ABC123 | CI1 | | DEF456 | CI2 | [ ] {#id-detect-dup-ci__table_h1f_2sg_ygc} var payload = { "items": [ { "className": "cmdb_ci_linux_server", "lookup": [ { "className": "cmdb_serial_number", "values": { "serial_number": "ABC123", "serial_number_type": "system", "valid": "true" } }, { "className": "cmdb_serial_number", "values": { "serial_number": "DEF456", "serial_number_type": "system", "valid": "true" } } ], "values": { "name": "CI_New" } } ], "relations": [] }; var jsonUntil = new JSON(); var input = jsonUntil.encode(payload); var output = SNC.IdentificationEngineScriptableApi.createOrUpdateCIEnhanced("ServiceNow", input, {}); gs.print(output); {#id-detect-dup-ci__table_qrv_vsg_ygc__entry__3} | Exact count match | Property to create de-duplication task = true | Property to create de-duplication task = false | |-|-|-| | True | * No CI match is found. * De-duplication task isn't created for CI1 and CI2. * No de-duplication tasks are created for lookup items. {#id-detect-dup-ci__ul_ah5_xsg_ygc} | * No CI match is found. * De-duplication task isn't created for CI1 and CI2. * No de-duplication tasks are created for lookup items. {#id-detect-dup-ci__ul_bh5_xsg_ygc} | | False | * CI1, the oldest, is matched and updated. * De-duplication task is created for CI1 and CI2. * No de-duplication tasks are created for lookup items. {#id-detect-dup-ci__ul_ch5_xsg_ygc} | * CI1, the oldest, is matched and updated. * De-duplication task isn't created for CI1 and CI2. * No de-duplication tasks are created for lookup items. {#id-detect-dup-ci__ul_dh5_xsg_ygc} | [Table 2. Identification results] {#id-detect-dup-ci__table_qrv_vsg_ygc} {#id-detect-dup-ci__ul_grm_w5g_ygc} ## Remediating de-duplication tasks {#id-detect-dup-ci__section_px4_jvf_ndb} For information about reviewing and remediating de-duplicate tasks, and how the main CI is used, see [Duplicate CIs remediation](https://www.servicenow.com/docs/gSpynTBDyN19wsPvnN1TsQ "Duplicate CIs in the CMDB unnecessarily overload the system and interfere with the CMDB's integrity and reliability. It is therefore important to regularly use CMDB tools that detect and remediate duplicate CIs.").