Error: Performance Analytics Data Collector Job: Fetched too many rows from indicator source.

[Rahul Nagre1]

Subject: Error: Performance Analytics Data Collector Job: Fetched too many rows from indicator source.

Description: while executing Data Collector job for Performance Analytics, we encountered error for  indicator source (as below).

- Fetched too many rows from indicator source <INDICATOR NAME> for <DATE>. Allowed: 1,000,000 fetched: 21,060,694

We understand that the limit is not allowing the indicator source query outcome to be stored in scores table.
We would like to check if there is any suggested solution to tackle this problem

Please note that the indicator source is fetching data from Test Result table (sn_vulc_result) which contains Qualys test data. The total number of records is around 22 million.

Steps to reproduce:
Trigger the Data Collector job manually/check the logs for the scheduled job run.

Anyone has faced similar issue and if yes, can you please suggest any solution.

We are considering the following solution options;

We have reviewed the KBs below and found that the KB- https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0758471 seems to be more helpful.

We have extracted the possible configuration changes that can be done to tackle the problem;

1. Enabling the “Override record collection” for  indicator source and setting the maximum number of fetched records as 25 M . Refer- https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0965819 and https://docs.servicenow.com/en-US/bundle/vancouver-now-intelligence/page/use/performance-analytics/c...

2. Distributing the data collection in different jobs that will run at different times. This will help us to manage the memory usage.
3. Redesigning the indicator sources to facilitate the reduction of the data set count by moving conditions from the Indicator to the Indicator source.

[AndersBGS]

Hi @Rahul Nagre1 ,

Have you considered if all 22 Million rows is needed in the data collection? 

If my answer has helped with your question, please mark my answer as accepted solution and give a thumb up.

Best regards

Anders 

[Rahul Nagre1]

Yes. I have a use case that demands calculation of a % passed test result. For that, I need to count the scores of total test results and total passed test results.

Total test results are 22 M and Passed Test Results are 11.5 M

[Pablo1]

Hi Rahul,

what solution did you end up doing and was there any performance hit to the system and or how long did the data collection job take?