fix script max execution time?

[kjcv912]

Hi

Running housekeeping scripts on pa_scores table (Performance Analytics) and have observed that the fix scripts ends after 24 hrs and some minutes (2 up to 45 min) even though there is huge number of orphan records to be deleted.
Has disabled business rule "PA Cache buster" (pa_scores / trigger Delete).
Execute the fix scripts in the background.

Progress Workers list view shows that   State is "Complete" and background colour green which doesn't match my view when it has terminated before being removing all orphan records.

When checking messages via "System Diagnostics > Progress Workers Fix Script: PA - Remove orphan BD(5)"

can I see TOTAL time at the bottom and in several cases also NPEs

Evaluator: java.lang.NullPointerException

    Caused by error in script at line 19

16:	while (gr.next()) {
17:		count++;
18:		gs.log("PA: Indicator '" + gr.indicator + "' and element '" + gr.element + "' on " + gr.start_at + " score " + gr.value + " deleted. ( " + count + " of " + gr.getRowCount() + " )", logSource);
==>   19:		gr.deleteRecord();
20:	}
21:
22:	gr.close();

[24:29:23.655] Total Time

line 19 has been executed 4 000 000 times and suddenly this NPE.

Is there a instance / global setting stopping fix scripts after 24 hours of execution time ?

Any ideas?

Stefan

[Uncle Rob · Accepted solution]

You can look into Transaction Quotas, but if you're already at the 24 hour mark of script running, there might be bigger issues.  
I'd be jumping on the horn with SN support.

[kjcv912]

We are a rather large user of PA so our tables are huge.

The total time is not an issue, that is the outcome when deleting some millions of orphan records (100 000 - 250 000 deletions per hour depending in which environment the fix script is running).

Thanks Robert, now have I finally found a quota related message in the log, not at the end of the log there I expected it to be,

rather high up did I found this

Quota exceeded for UI Transactions, requesting cancel

Transaction: Cancelling   transaction #114195 /syslog_list.do (maximum execution time exceeded): Thread Default-thread-39

I am still confused by that the STATE is COMPLETE and COMPLETION CODE is SUCCESS,

would expect some high level WARNING and also that the CANCELLATION text would have been close to the TOTAL TIME

Another new experience:-)

Thx
Stefan

[Uncle Rob]

Wow... I'd be very interested in talking with you further then since I'm on a rabid PA kick lately.

Where do all the orphaned records come from due to PA?

[kjcv912]

Hi Robert,

we have an ongoing conversation with a ServiceNow PA guru helping us to optimize our PA environment.

We have adopted recommendations from him, started to run scripts in DEV and QA environments and today have we finally started to run the first fix script in PROD.

When running the "cleanup" scripts in dev and QA did we face huge number of orphan records, RCA has not been completed why we see so many orphan records. What we suspect can have an impact is that the last clone jobs to DEV and QA did fail on "pa_scores" tables so they might be damaged...

Orphan records could be the outcome when removing breakdowns and indicators. Fuji Patch 11 added a new PA housekeeping script which I guess is aimed to remove orphan records but we have been advised to disable that for the moment.

Stefan