Managing CMDB data deletion
Summarize
Summary of Managing CMDB data deletion
ServiceNow’s Integration Commons for CMDB (sncmdbintutil) store app provides methods to manage the deletion of Configuration Management Database (CMDB) records that have been removed from source third-party applications. This ensures that your CMDB stays synchronized with external data sources by removing outdated or deleted configuration items (CIs).
Show less
Post-script method
This method checks the Source [sysobjectsource] table for CIs not updated during the latest run for a given source feed. It removes these non-updated CIs from the target CMDB tables one at a time, synchronously. This approach is commonly used by most Service Graph Connectors.
The method uses the RecordRemoval script to support three deletion strategies:
- Hard deletion: Permanently removes records from the CMDB, making them unrecoverable. Suitable when data is no longer needed.
- Soft deletion: Marks records as deleted without removing them, allowing for potential recovery and preserving historical data.
- Optimized soft deletion: Managed by the Identification and Reconciliation Engine (IRE), this uses a specialized extension attributes table and reconciliation rules for efficient handling of deleted records.
Administrators can configure the number of records managed in optimized soft deletion by creating and setting the sncmdbintutil.irepayloaditemssize system property in the Integration Commons app. This property controls the batch size of records processed, defaulting to 100 if not set.
Transform map-based method
This method also checks the Source table for non-updated CIs but uses a multi-threaded deletion approach to concurrently remove these records from target CMDB tables. It handles all target records within an import set simultaneously, improving data processing speed.
The process involves multiple import sets stored in the Integration Commons Remove Record [sncmdbintutilremoverecord] staging table. This table tracks non-updated CIs with detailed information such as discovery source, source feed, target table, and sysid.
An on-start script within the Remove Record Transform Map deletes the records from the target tables. This method is typically used by Service Graph Connectors that require faster processing, such as the SG-Tanium Remove Software data source.
Implications for ServiceNow Customers
- Choose between post-script and transform map-based deletion methods based on your performance needs and connector compatibility.
- Understand deletion strategies to balance between permanent removal and preserving historical CMDB data.
- Configure system properties as needed to optimize deletion batch sizes and performance.
- Use the transform map-based method to reduce data processing time when working with large data imports or connectors like Service Graph Connector for Tanium.
The Integration Commons for CMDB (sn_cmdb_int_util) store app uses post-script and transform map methods to remove imported Configuration Management Database (CMDB) records that were deleted later from the source third-party application.
Post-script method
The post-script method checks the Source [sys_object_source] table for configuration items (CIs) not updated in the last run for a specific source feed. It then removes any non-updated CIs from the associated target CMDB tables synchronously, handling the removal of target records one at a time.
Most Service Graph Connectors use the post-script method to manage the removal of any CMDB records.
The post-script method uses the RecordRemoval script to handle various record deletion strategies. Hard deletion involves permanently and completely removing records from the CMDB, making them unrecoverable and suitable for data that is no longer needed. Soft deletion marks records as deleted without actually removing them from the CMDB, allowing for potential recovery and preserving historical data. Optimized soft deletion, managed by the Identification and Reconciliation Engine (IRE), enhances this soft deletion approach by using a specialized table, such as the extension attributes table, and applying reconciliation rules to manage deleted records more efficiently. By default, the table can hold up to 100 records at a time, but as a user with the admin role, you can adjust this limit by setting the sn_cmdb_int_util.ire_payload_items_size system property. This property is not available by default and you must create this property of type integer within the Integration Commons for CMDB (sn_cmdb_int_util) store app. For more information, see Add a system property.
Transform map-based method
The transform map-based method checks the Source [sys_object_source] table for configuration items (CIs) not updated in the last run for a specific source feed. It then utilizes the multi-threaded deletion approach to remove any non-updated CIs from the associated target CMDB tables concurrently, handling the removal of all target records within an import set at once. Multiple import sets are created to store target record data within the Integration Commons Remove Record [sn_cmdb_int_util_remove_record] staging table enabling multi-threaded deletion of records. Therefore, reducing the data processing time.
The Integration Commons Remove Record [sn_cmdb_int_util_remove_record] staging table tracks non-updated CIs by storing target record details including the discovery source, source feed, target table sys_id, and target table name of these CIs. The on-start script defined within the Remove Record Transform Map then deletes the records from the associated target tables.
The transform map-based method is used by a Service Graph Connector to improve the data processing time. For example, the SG-Tanium Remove Software data source within the Service Graph Connector for Tanium removes software data using the transform map-based method. See Service Graph Connector for Tanium.