Steps to help prevent duplicate or orphaned records after running Vulnerability Response CI lookup rules

  • Release version: Yokohama
  • Updated January 30, 2025
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Steps to Help Prevent Duplicate or Orphaned Records After Running Vulnerability Response CI Lookup Rules

    This guidance focuses on preventing duplicate or orphaned Configuration Item (CI) records in the Configuration Management Database (CMDB) when importing vulnerability data using Vulnerability Response CI lookup rules. Poorly constructed matching rules can cause performance issues and result in inaccurate CI matching, leading to duplicates or unmatched records.

    Show full answer Show less

    Best Practices for Preventing Duplicate or Orphaned Records

    • Test with Small Data Subsets: Use focused, small datasets specific to the CI Lookup Rule under test, and deactivate all other CI Lookup Rules during testing to isolate effects.
    • Analyze Imported CIs: Review the ratio of matched versus unmatched CIs across all data pages, not just the first, to ensure matching behaves as expected. Manually verify CIs for naming or field inconsistencies.
    • Adjust Matching Rules as Needed: If discrepancies arise, consider refining or adding matching rules to improve accuracy.
    • Review Unmatched CIs: Examine the Unmatched CIs table grouped by CI class to identify unexpected classes or mismatches. Decide if certain classes should be excluded or reclassified, and check CI states (e.g., Retired) for relevance.
    • Remove Test Data Before Final Runs: Once confident in the matching behavior, delete test data from relevant tables such as Discovered Items, Vulnerable Items, and Remediation Tasks, then rerun CI matching rules cleanly.

    Deleting Data from Tables

    When imported data is incorrect, especially in development or test environments, it may be necessary to delete relevant records. There are four main methods:

    • Using the Delete All Records option on the table configuration.
    • Configuring Table Cleaner via Auto Flushes (sysautoflush.list) for automatic cleanup.
    • Running gs.truncateTable background scripts, with rollback disabled to avoid long processing times and failures.
    • Important: Avoid using truncateTable in production environments without consulting ServiceNow support, as improper deletions can cause data loss or system issues.

    Additional Considerations

    • Thoroughly test and debug processing scripts within CI lookup rules to minimize performance impacts and matching errors.
    • For deeper understanding of CI Lookup Rules with specific vulnerability scanners like Qualys or Rapid7, refer to respective ServiceNow KB articles.
    • Related tasks include de-duplicating existing CIs, reclassifying unmatched CIs, reconciling discovered items, and resolving remediation tasks to maintain CMDB integrity.

    Take steps to help prevent duplicate or orphan records resulting from matching (configuration items (CIs) within the CMDB.

    Importing vulnerability data can be taxing on an instance and performance issues with resources can occur if rules are not carefully constructed. The logic used to iterate through and perform matching within the CMDB can result in lengthy processing times. Thorough testing and debugging of processing scripts in the rules helps alleviate the potential of issues later in the process.

    Preventing duplicate or orphaned records

    • Use small subsets of data that are specific to the CI Lookup Rule being tested.
      • Set all CI Lookup Rules, other than the one being tested, to Inactive.
      • Analyze the imported CIs to ensure that you are observing the expected behavior and matching is occurring properly.
    • Review Matched CIs
      • Examine the count of matched vs unmatched CIs. Ensure that the percentage is acceptable. Don’t just look at the first page, that is likely the first one inserted.
      • Manually search for some CIs.
      • Check to see if there are any naming or field problems (such as searching for a specific domain).

        If it seems appropriate, add additional matching rules.

    • Review Unmatched CIs
      • Navigate to the Unmatched CIs table.
      • Group by Configuration Item class.
      • Review any classes that don’t look right (certificates, network cards, images).
        • Figure out why didn’t they match the correct CI?
        • Should the class be excluded?
        • Should the class be elevated to a related class?
    • Review CI states such as Retired.
    • Remove Test Data
      • Once you begin to observe the correct or expected behavior in CI matching, start over.
      • Start over by: Deleting the data used for testing: (see the Deleting data from tables section)
        • Discovered Items
        • Vulnerable Items
        • Remediation tasks
      • Manually rerunning all the CI Matching rules.

    For more information on CI Lookup Rules and Qualys, see the KB0750656 article.

    For more information on CI Lookup Rules and Rapid7, see the KB0818096.

    Deleting data from tables

    Sometimes you have imported data and realize something is wrong. If something is wrong in a development or performance environment, you could reclone from a better environment, but that isn’t always an option.
    Note:
    Performing these actions requires ServiceNow expertise.
    There are four options for deleting data from tables:
    • Using Delete All Records on Table Configuration.
    • Configure the Table Cleaner by navigating to Auto Flushes (sys_auto_flush.list) and creating a new Auto-flush record.
    • Truncate the gs.truncateTable using a background script.

      Using truncateTable requires turning off the record for rollback check box in the background scripts. Otherwise, a copy of the table and related cascade tables are created, take too long, and most likely fail.

      Note:
      Never use truncateTable in a production environment. Consult you Support representative before executing large deletions in production or shared environments.