Performance Analytics historical metric collection job still running

[Andrew Bettcher]

I'm adding new indicators to the Change Success Score dashboard. I followed the instructions and got to the bit where I clear out historical scores and regather them. The process is described here.

It says to check the PA job logs to ensure each job has finished. When I first did it, all of the jobs finished relatively quickly (i.e. relative to my expectation based on the timeframe and the amount of data). I re-ran the jobs and they collected a larger number of records and, every time I run them they collect more and more. 

I checked the Metric Definition (called "Change Success Score") and refreshed the related list called "Metrics" and noted that it is still gathering data.

My question is then, all 4 of my data collection jobs report as "Collected" but there is clearly something still gathering data from somewhere so, what is it?

I have an awful feeling that whatever is running might be related to some weird results that we've seen. I mean that perhaps there is a huge job running somewhere and that our collection jobs are running each day before it's had a chance to complete.

[Uncle Rob]

I don't know if there's a good way to find this out.
I'd be tempted to brute force it by 
1)  Looking at the range I'm trying to discover.  (I'd hope no more than 6 months)
2)  Taking a good hard look at the Indicators and breakdowns.

[Andrew Bettcher]

I checked the "Active Transactions" log and found a job running that has been going for 37 minutes which is around the time when I first kicked off the collection process after cleaning out the metrics. The question still stands though.

If the PA Collection Jobs are all showing in a state of "Collected", why is this still running and is there a chance that job is interfering with our daily updates?

This is the active transaction with the instance name redacted for client anonymity, if it helps:

[Andrew Bettcher]

I've done some further research and learned a lot about how this works but I still have questions. 

It seems that there is a job scheduled to kick off at 0200 that collects daily metrics. When this job completes it fires an event called pa.job.dc.ended.ok which, in turn, starts of a Script Action called "Daily change success score".

This then calls a script include and we get into ALL of the script includes calling each other all over. 
So, if this is the case then everything must happen at some time after 0200 (N.B. Ours is set 0200 but I think this is the OOTB setting). So, why are all of the metric lines listed as having been created at 00:00 like this:

My next question is, I manually executed all 4 of the Change Success collection jobs. They all reported as "State = Collected" in the PA Job logs. However, for over 3 hours after this, the number of Metrics listed in the related list of the Change Success Score Metric Definition continued to grow indicating that there was still some processing taking place somewhere (I guess this is the ChangeSuccessScoreGeneratorSNC doing it's thing).

Therefore, couldn't my initial hypothesis be correct? The rest of the processing is trying generate scores based on metrics that aren't there yet leading to what were described to me as "weird scores but only for some groups"?