Executive Summary
As ServiceNow instances scale across enterprise environments, the volume and complexity of data grow exponentially. Without a strategic data management framework, organizations face performance degradation, compliance risks, and escalating storage costs. This white paper outlines a comprehensive approach to Data Management in ServiceNow, focusing on four key pillars: Data Archiving, Table Rotation and Extension, Table Cleaner, and Ad-Hoc Cleanup Requests. Drawing from real-world implementations, platform architecture workshops, and industry best practices, this paper provides actionable guidance for optimizing data lifecycle, improving system performance, and ensuring governance compliance.
Problem Statement
ServiceNow environments often accumulate vast amounts of operational, transactional, and audit data. This leads to performance bottlenecks such as slow queries, UI lag, and inefficient reporting. Compliance challenges arise due to difficulty enforcing retention policies and audit trails. Operational overhead increases with manual cleanup efforts and administrative burden. Cost escalation is a major concern as higher storage usage results in additional licensing fees. For example, a case study from Large Retail Enterprise revealed a 40TB sys_audit table that required months of cleanup and table rebuilds to reclaim space.
Solution Options
To address data growth and performance challenges in ServiceNow, organizations can adopt several strategies as part of a comprehensive data management approach. These include:
- Data Archiving: Moves inactive records from primary tables to archive tables, improving performance and enabling retention policy enforcement.
- Table Rotation and Extension: Splits high-volume tables into shards to improve performance and manage data lifecycle.
- Table Cleaner: Automates deletion of old records based on age and conditions, reducing manual effort.
- Ad-Hoc Cleanup Requests: Enables customer-initiated data deletions using APIs or scripts for targeted cleanup.
These components work together to ensure efficient data handling, compliance, and cost control.
Choosing the Right Strategy
Selecting the appropriate tool for managing data in ServiceNow can feel overwhelming, especially when dealing with large or complex tables. While exceptions may occur, following general guidelines helps ensure you ask the right questions and choose the best approach. Applying the 80/20 rule simplifies decision-making: for task-related tables, archiving is typically the preferred option; for email tables, use Email Retention; for CMDB tables, leverage CMDB Data Manager; for log tables, implement Table Rotation; and for supporting tables with temporary records, Table Cleaner is ideal. The attached decision tree provides a visual guide to help determine the right strategy based on factors such as data growth rate, retention requirements, and operational needs.
Implementation Strategy
Implementing a robust data management framework in ServiceNow requires a series of structured steps. While detailed instructions are available in the official ServiceNow documentation, the following outlines the key high-level actions that should be prioritized:
- Archiving Setup:
- Activate Archiving Plugin.
- Create Archive Rule.
- Define Related Record Handling.
- Verify Record Impact.
- Schedule and Activate Rule.
- Table Cleaner Setup:
- Define rules in sys_auto_flush.
- Tune system properties for performance.
- Monitor replication lag and adjust configurations.
- Rotation/Extension Configuration:
- Choose appropriate type based on retention needs.
- Configure shard duration and count.
- Use Synchronize Shards for schema updates.
- Ad-Hoc Cleanup Execution:
- Segment large datasets.
- Use efficient queries.
- Validate with slow query diagnostics.
Outcomes and Metrics
Implementing strategic data management yields measurable outcomes:
- Performance Gains: Archiving reduced active rows by up to 80% in incident tables. Table Cleaner reclaimed 32.6TB in Large Retail Enterprise’s sys_audit cleanup.
- Cost Savings: Off-platform archiving avoids additional storage fees. Table rebuilds reduce disk usage and improve query speed.
- Compliance Alignment: Supports GDPR and internal retention policies. Enables audit readiness and secure data lifecycle management.
Lessons Learned or Recommendations
Key recommendations based on real-world implementations:
- Start Early: Enable archiving and cleaner rules at go-live to avoid multi-year backlogs.
- Segment Cleanup: Target one year or month of data at a time.
- Use Multithreaded Cleaner: For large datasets, enable DM Table Cleaner for faster processing.
- Monitor Footprint: Use Telemetry dashboards and sys_physical_table_stats for tracking growth.
- Avoid Complex Conditions: Use indexed fields and simple filters for cleaner rules.
- Test Everything: Always validate in sub-prod before applying to production.
Real-world Use Case
A major global retailer faced a significant data management challenge with its sys_audit table ballooning to over 40TB, severely impacting performance and storage capacity in its production ServiceNow instance. The table had minimal indexing and was hosted on a 56TB sharded server with only 12TB of free space remaining. To address this, the enterprise deployed a multi-pronged strategy involving custom cleanup scripts, index optimization, and parallel job execution to reclaim space. The team ran 12 concurrent jobs, each targeting a month’s worth of data, and used ServiceNow’s Table Cleaner as a secondary pass to maintain cleanliness.
In parallel, the organization began migrating both live and archived data to Azure SQL, aiming to offload reporting and analytics workloads from the production platform. This move was driven by the need to reduce query load, improve performance, and provide a read-only environment for business users via Power BI. The migration also helped enforce retention policies across product teams, each of which had unique data governance requirements. As a result of these efforts, the enterprise successfully reclaimed 32.6TB, stabilized system performance, and avoided any major incidents during the cleanup process.
FAQ
Q. Can Archival be enabled for rotated tables?
A. No, archival cannot be enabled for rotated tables
Q. Can related records be archived?
A. Yes. Use the Archive Related Records list. Related records are always archived regardless of their own rule conditions.
Q. Can all users view archived records?
A. Archive tables inherit ACLs from their source tables (e.g., ar_incident uses incident ACLs)
Q. Can Destroy rules delete records from Archive Log?
A. Destroy rules apply only to archive tables, not the Archive Log.
Q. Can Table Cleaner delete from Archive Log?
A. No. Deleting from Archive Log disables restoration
Q. Are related record archive rule conditions used?
A. No. Related records are archived unconditionally
Q. What happens to reference fields in archived records?
A. They are converted to display strings and do not reflect future changes
Q. Can archived records be restored?
A. Yes, via the Archive Log
Q. What is the impact of Table Cleaner on performance?
A. Improper configuration can cause replication lag, disk space alerts, and high CPU usage
Q. How does Table Rotation differ from Extension?
A. Rotation overwrites old shards; Extension creates new shards indefinitely
Q. Can Table Cleaner be customized?
A. Yes. Multiple entries can be configured per table with specific conditions.
Conclusion
In conclusion, effective data management in ServiceNow is not merely a technical necessity but a strategic enabler for enterprise performance, compliance, and cost optimization. By implementing a holistic framework that includes data archiving, table rotation and extension, table cleaner automation, and ad-hoc cleanup capabilities, organizations can proactively manage data growth, reduce operational overhead, and ensure long-term sustainability of their ServiceNow environments. Real-world scenarios, such as the Large Retail Enterprise sys_audit cleanup, demonstrate the tangible benefits of these strategies—including reclaimed storage, improved system responsiveness, and enhanced governance.
As data continues to grow in volume and complexity, it is imperative for organizations to adopt a forward-looking data management strategy that aligns with business objectives and platform capabilities. ServiceNow provides the tools, frameworks, and best practices to support this journey, and with proper planning and execution, enterprises can achieve measurable outcomes and maintain a healthy, scalable platform.
