Automated cleanup of duplicate records from same source
The TISC application includes automated logic to manage records that were received repeatedly from the same source. When identical or matching records are ingested multiple times from same source, the application ensures that the most recent record remains active while previously stored instances are identified as duplicates.
Impact of duplicate records
When duplicate records from the same source are retained in the application:
- The deduplication jobs must evaluate a larger number of records during execution.
- Increased record volume can extend job runtime.
- Processing latency may occur as the system identifies and resolves redundant entries.
Table cleaners are implemented for the Observable Source [sn_sec_tisc_observable_source], Indicator Source [sn_sec_tisc_indicator_source], and Object Source [sn_sec_tisc_object_source] tables.
These cleaners remove records from the respective tables based on the following condition.
Records where the processing Status is set to Duplicate and Duplicate of Record from Same Source flag is set to true and the record was last updated more than 7 days ago.
Navigate to view and manage table cleaners.