Error in Performance Analytics Job

[mrswann]

Upon running the historical job for Incident - there are many warnings & errors; mostly around the volume of tickets:

Example Error:
Fetched too many rows from indicator source Incidents.Open. Allowed: 50,000 fetched: 214,266
Example Warnings:
Not saving records for Indicator:[SYSID] Breakdown:[SYSID] Element:unmatched Result has 39,467 records. Allowed are 5000 records
Not saving records for Indicator:[SYSID] Result has 40,991 records. Allowed are 5000 records

I have tried to run the job over a shorter period of time which doesn't seem to have any effect. The number of inserts & deletes always matches, so it is removing any duplicate data as one would expect.

The real, current issue is I am unable to get any PA data more recent than 3 months ago (presumably because the volume of OPEN increased beyond the acceptable limits).

I know that we have had some issues with Monitoring & Event Alerting creating erroneous incident tickets which then need to be closed down, creating significant workload for the teams.

I think this has also caused the volume of OPEN to increase exponentially over time.

Do you have any recommendations, thoughts, ideas   what can be done to resolve this?   Has anyone else experienced the same?

[Tim Deniston]

You could always bump up the properties that define the maximum limits. Go to Performance Analytics > System > Properties and you will see there are several properties in there you could adjust (look towards the bottom of the page).

[Chuck Tomasi]

I would check those indicators and indicator sources. 214K of open incidents? Can you verify that? if so, then Tim's suggestion to tweak the properties might be in order.

[Chuck Tomasi]

Let me know if that answered your question. If so, please mark it as correct so that others with the same question in the future can find it quickly and that it gets removed from the Unanswered list. Thank you

[mrswann · Accepted solution]

Hi both - thank you very much. You are both right!

Firstly, the 214k of open incidents was erroneous to a point and this was corrected with the Indicator Sources to ignore the automated event alerts (a different stream of activity to get this right!)

However - in order to get through that and understand the data - it was necessary to update the properties as per Tim's advice. This allowed the collections to complete and then the data was reviewed and thus the collection processes optimized, now around ~150 open. This is still subject to further review but has stripped out the alert data and made it much more manageable.