---
sourceDocument: Yokohama Security Management
sourceDocumentLink: https://www.servicenow.com/docs/r/yokohama/security-management

 Release :

    - yokohama

ft:locale :

    - en-US

ft:publication_title :

    - Yokohama Security Management

ft:clusterId :

    - security

bundleId :

    - security

workflow :

    - Technology


---

# Automated cleanup of duplicate records from same source

# Automated cleanup of duplicate records from same source {#ariaid-title1}

* Release version: Yokohama
* 
* Updated March 2, 2026
* 
* ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 1 minute to read

The TISC application includes automated logic to manage records that were received repeatedly from the same source. When identical or matching records are ingested multiple times from same
source, the application ensures that the most recent record remains active while previously stored instances are identified as duplicates.

## Impact of duplicate records {#duplicate-handling-of-records-from-the-same-source__section_amp_wqd_j3c}

When duplicate records from the same source are retained in the application:

* The deduplication jobs must evaluate a larger number of records during execution.
* Increased record volume can extend job runtime.
* Processing latency may occur as the system identifies and resolves redundant entries.

{#duplicate-handling-of-records-from-the-same-source__ul_ivl_mwd_j3c}

Table cleaners are implemented for the Observable Source \[sn_sec_tisc_observable_source\], Indicator Source \[sn_sec_tisc_indicator_source\], and Object Source \[sn_sec_tisc_object_source\] tables.

These cleaners remove records from the respective tables based on the following condition.

Records where the processing Status is set to Duplicate and Duplicate of Record from Same Source flag is set to true and the record was last
updated more than 7 days ago.

Navigate to System MaintenanceTable Cleanup view and manage table cleaners.
**Related concepts**   

* [Observables](https://www.servicenow.com/docs/mtj0sOYtYVd9ZZ2fpM_hMg "Observables represent stateful properties (such as the MD5 hash of a file or the value of a registry key) or measurable events (such as the creation of a registry key or the deletion of a file) that are pertinent to the operation of computers and networks.")
* [Indicators](https://www.servicenow.com/docs/B69N6Wdc7W0nI0ks7FqyYA "Indicators are artifacts observed on a network or operating system that are likely to indicate an intrusion. Typical IoCs are virus signatures and IP addresses, MD5 hashes of malware files or URLs, or domain names.")
* [TISC Data Processing Functional Flow](https://www.servicenow.com/docs/HMvY3F2tzf0vj8Ai5OEG_w "Threat Intelligence Security Center (TISC) provides a solution that automates the data collection and processing which helps reduce the burden on Threat Intel Analysts by avoiding manual steps involved.")
* [TISC Data archival and cleanup](https://www.servicenow.com/docs/bOVUP_Ikit2D3kYryr4h2A "Data grows rapidly in tables with increased adoption of the platform. Some tables come within the base system with various data management policies, but other users are needed to implemented by themselves.")

*[\>]: and then