PA historical collection for several years in PROD

[sdf1]

Hi Guys,

Need some advice on a PA historical collection requirement.

We have a service health dashboard already live in PROD, covering incidents, cases and SLA, OLA metrics. The business wants historical trends going back to when the instance went live, say early 2023, so roughly 3 years of data.

The automated indicators include things like:

• Incidents and cases created, resolved and closed
• Resolved incidents reopened
• Closed cases without breached SLAs
• Completed SLAs for cases
• Closed incidents without breached OLAs

The indicators have daily collection jobs. One I checked uses Relative start = 1 day ago and Relative end = 1 day ago, so it collects yesterday’s scores. The requirement now is to backfill the older history without disturbing the existing daily collection.

My main worry is the load on PROD. If we create a historical job covering the full period, could it keep running for days and affect normal usage? I don’t have a reliable runtime estimate yet.

I’ve checked some source configurations in DEV and looked at historical data availability, but DEV doesn’t have representative PROD data, so I can’t really use it to prove the performance impact. We also found sparse earlier data for some resolution and task_sla based metrics, while created records go further back. 

What would be the recommended approach here? Is it better to run one historical job for the whole period, or split it into smaller fixed date ranges and run them separately? but for how many collectors, I have like 15.

This is already live in PROD and we need to agree the backfill approach soon, so practical advice from anyone who has done a multi year PA collection would help. 

[AndersBGS]

Hi @sdf1 

As far as I recall, the historical job will not collect 3 years of data, as there is a system property preventing this. If you change that system property it could for sure put some load on the instance. So in turn, I would ratter ask the question why 3 years of data.

[abirakundu23]

Hi @sdf1 ,

It is not recommended to backfill three years of historical data in a single run, as it could increase database load and impact overall system performance.

Instead, break the backfill into smaller date ranges and run them incrementally. Validate the execution time, system performance, and collected scores after each run before proceeding with the next batch.

Please mark helpful & correct answer if it's worthy for you.