Archive records from Large Table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎01-20-2025 11:32 PM
We need to archive records from a table with approximately 2,600,111,136 records. However, archiving records created before 18 months is taking too long. Does anyone have suggestions on how to speed up this process? Considering the number of records, will increasing the batch size cause any performance issues?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
‎04-21-2025 08:33 AM
One system property you should check at may consider adjusting is glide.db.archiving.max_consumer_workers, which sets how many threads run for the archiving process in parallel.
If you don't see this in system properties, it is at the default value which is 4. If you see the property and it is set lower, chances are there was a replication lag issue at one point and this property was set lower to resolve the issue. In conjunction with discussion and monitoring with your platform team, it may now be fine to reset to 4. View this KB for more information:
Customized glide.db.archiving.max_consumer_workers property
You may consider upping the workers to 5 or 6, but again please work closely with your platform team to monitor any performance impacts across the instance.
More properties that affect how archiving processes records can be found in docs here: