Getting "[PA AppSec] Daily Data Collection completed with warning" emails ever since updating from Quebec to San Diego

[Jeremy Dean]

We are getting "[PA AppSec] Daily Data Collection completed with warning" emails hourly ever since updating from Quebec to San Diego.

Reviewing the job logs, I see a warning:
Not saving all records for Indicator:f83563cab7721010aed8e6e5de11a96b Result has 7,832 records. Only allowed subset of 5,000 records are stored

Searching around on the issue I found this article "[PA AppSec] Daily Data Collection completed with warning" daily email to ServiceNow admin account s... which indicates you can adjust the system property "com.snc.pa.dc.max_row_count_indicator_source" to include as many records as you need. I updated the number to 50,000 and that seems to have stopped the error in our Pre-Prod instance, but the same change was not successful in Dev. And evn if it were, I would still like to understand:

• What does this message mean? (The error Not saving all records for Indicator...)
• What does this job do? ([PA AppSec] Daily Data Collection completed with warning)
• Why did this start when we upgraded to San Diego
• What is the proper resolution (I don't like the idea of randomly increasing numbers on system variables.)
  
  

Any help is most appreciated - thank you!

[JP - Kyndryl]

Hi Jeremy,

If you go in the Schedules Jobs, you will see that "[PA AppSec] Daily Data Collection" that is running daily (every hour) to collect data for some indicators.  You may see all these indicators in that job (failed logins, locked out users, etc...).

This data is used by the Instance Security Center.

Such collection will always give a warning if it tries to collect more than 5000 snapshots.  The scores are well collected, but the snapshots are then limited to 5000 records.  That's fine and usually it is best not to increase that number.  It is rare that someone looking at the dashboard will want to see all the records form one of the number, so 5000 is usually the good maximum.  A number may be 10,000 (the scores) but if you want to see them, you will only be presented 5,000 (the snapshots).

This job is usually collecting data for 25 indicators.  On my system, it collects 120 scores per hour, so not a big deal.   Go into the Job Logs related link of that scheduled job and try to see why you collect so many data.

Regards.

JP

[Jeremy Dean]

Thank you so much Kilo!

When I looked at the job logs, I narrowed it down to three indicators - unfortunately they are alphnumeric strings and I don't see the correlation if I look at the indicators under the actual job (which are all named).

Warnings from log:

Indicators under job (sample - 25 total, all named):

[Jeremy Dean]

UPDATE: After navigating to [PA AppSec] Daily Data Collection and looking under Job Logs under Relate Links, I sorted by date and clicked on the latest run - all had been "collected with warnings" since moving to San Diego.

From there, I filtered out the sequences with Level = Warning and came back with three lines. I clicked into those logs which were all identical besides the Indicator # and the Result count. But I saw one that failed with a message like “Not saving all records for Indicator:42d4af8ab7721010aed8e6e5de11a963 Result has 27,127 records. Only allowed subset of 5,000 records are stored…” and I knew that number was pretty close to the number of users in our system overall.

So I went back to [PA AppSec] Daily Data Collection and began looking at the indicators. I noticed exactly three indciators had and Indicator Source called “System Users”.

I changed the Active value on those three indicators from True to False and now the job is running without warnings.

I am still not sure what the implications will be for the system but I am not too worried as I understand this is basically a logging/reporting tool. All of our users are synced with AD, so I am really not concerned with ServiceNow’s interpretation of that specific data as we monitor and report on it elsewhere.

Let me know if that sounds like a reasonable fix and if you think there will be any negative consequence from changing this setting. I think it will work as a stopgap to prevent those warning emails from going out every hour, if not a permanent solution.

Thank you!

[Adam Stout · Accepted solution]

The log is showing you the sys_id of the indicator that has the issue.  If you search for that in pa_indicators, you will see the indicator.   I would reactivate this indicator but disable the "Collect Records" option.  This warning is related to collecting (storing the sys_ids) more records than allowed based on the system property.  Storing the sys_ids of every user isn't particularly valuable, but the count of active users is important to have as a reference for other indicators.