Progress Workers available instance count

atanas8d
Tera Contributor

‎11-22-2023 05:43 AM

Hey,

 

I'm trying to spread a large amount of work into the sys_progress_workers table using GlideScriptedHierarchicalWorker API in the background.

 

Is there a way to access the available workers count from the scoped application in the background script? 

 

Also I'm noticing that some progress workers are starting and end directly in state "Completed" with "0 seconds" execution / run time. Which is not correct and obviously it's kind of a bug or problem. Have anyoene experienced that? 

 

Many thanks! 

 

0 Helpfuls
  • 2,112 Views
3 REPLIES 3

Maik Skoddow
Tera Patron
Tera Patron

‎11-22-2023 07:38 AM

Hi @atanas8d 

 

your question "Is there a way to access the available workers count from the scoped application in the background script?" makes no sense to me, as this table holds transactional data for running or finished workers. That does NOT holds any available workers! Whenever a new progress worker is started a respective record is added to that table. That's it. Later old records will be removed automatically by the table cleaner.

But maybe I did not understand your question correctly.

 

And you should have access to that table from a scoped application. If not you might add any cross-scope access records.

 

Maik

0 Helpfuls

atanas8d
Tera Contributor
In response to Maik Skoddow

‎11-22-2023 08:52 AM

Hi @Maik Skoddow ,

 

First of all, I'm a fan of your work in the field of knowledge-sharing in the ServiceNow world. Kudos for what you do! 

 

You're right, I should have added a bit more details about what I mean by "available workers count". I refer to the available instance resource. They can be found in the /now/nav/ui/classic/params/target/stats.do as well as /background_progress_worker.do
atanas8d_0-1700671292497.png

I'm not sure that the Maximum Worker Pool Size is 16 by default in all instances, so would prefer to get this number dynamically in the code from scoped application.

 

My second question is about the strange behavior of progress workers, which occurs when the worker with a work batch that usually takes about 1 hour and a half ends up in a "Complete" state with "0 Seconds" execution time. I tried to re-initialize the workers who do this with a scheduled job and got the same result, although the stats.do shows that I'm using only 7 out of 16 workers from the pool size:

atanas8d_1-1700671662408.png

In my case, I'm importing an XML file, which cannot be imported using Import Sats as I need to utilize the attributes of the XML nodes as column values, etc and I need to process the nodes. 

 

Checking the system logs, I'm observing an error that seems to correlate to these "falsely completed" progress workers. Strangely, when I try to run the same work batch manually from the background script it starts processing without any errors. The confusing part is that the Progress Worker does not set the status correctly in this situation.  

com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '2' (code 50) in prolog; expected '<'
 at [row,col {unknown-source}]: [1,1]: org.apache.axiom.om.OMException: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '2' (code 50) in prolog; expected '<'
 at [row,col {unknown-source}]: [1,1]: 	org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:297)
	org.apache.axiom.om.impl.dom.DocumentImpl.getOMDocumentElement(DocumentImpl.java:446)
	org.apache.axiom.om.impl.dom.DocumentImpl.getDocumentElement(DocumentImpl.java:458)
	com.glide.util.XMLStreamDocument.getDocumentElement(XMLStreamDocument.java:403)
	com.glide.util.XMLStreamDocument.isValid(XMLStreamDocument.java:231)
	com.glide.util.XMLStreamDocument.parse(XMLStreamDocument.java:217)
	com.glide.script.XMLDocument2.jsFunction_parseXML(XMLDocument2.java:49)
	jdk.internal.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
	java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.base/java.lang.reflect.Method.invoke(Method.java:566)
	org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138)
	org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:675)
	org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614)
	org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2649)
	org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1518)
	org.mozilla.javascript.Interpreter.interpret(Interpreter.java:830)
	org.mozilla.javascript.InterpretedFunction.lambda$call$0(InterpretedFunction.java:160)
	com.glide.caller.gen.sys_script_include_625e2588dbec9050415b178b68961998_script.call(Unknown Source)
	com.glide.script.ScriptCaller.call(ScriptCaller.java:18)
	org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:159)
	org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2734)
	org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2657)
	org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1518)
	org.mozilla.javascript.Interpreter.interpret(Interpreter.java:830)
	org.mozilla.javascript.InterpretedFunction.lambda$call$0(InterpretedFunction.java:160)
	com.glide.caller.gen.sys_script_include_625e2588dbec9050415b178b68961998.call(Unknown Source)
	com.glide.script.ScriptCaller.call(ScriptCaller.java:18)
	org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:159)
	org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:597)
	org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3573)
	org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:172)
	com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:397)
	com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:209)
	com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:137)
	com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:348)
	com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:295)
	com.glide.worker.ScriptedHierarchicalWorker.evaluateScriptInScope(ScriptedHierarchicalWorker.java:100)
	com.glide.worker.ScriptedHierarchicalWorker.startWork(ScriptedHierarchicalWorker.java:76)
	com.glide.worker.AbstractProgressWorker.startAndWait(AbstractProgressWorker.java:126)
	com.glide.worker.HierarchicalProgressWorker.startAndWait(HierarchicalProgressWorker.java:26)
	com.glide.worker.BackgroundProgressJob.execute(BackgroundProgressJob.java:59)
	com.glide.schedule.JobExecutor.lambda$executeJob$0(JobExecutor.java:140)
	com.glide.schedule.JobExecutor.executeJob(JobExecutor.java:143)
	com.glide.schedule.JobExecutor.execute(JobExecutor.java:127)
	com.glide.schedule_v2.SchedulerWorkerThread.executeJob(SchedulerWorkerThread.java:328)
	com.glide.schedule_v2.SchedulerWorkerThread.lambda$process$0(SchedulerWorkerThread.java:219)
	com.glide.worker.TransactionalWorkerThread.executeInTransaction(TransactionalWorkerThread.java:35)
	com.glide.schedule_v2.SchedulerWorkerThread.process(SchedulerWorkerThread.java:219)
	com.glide.schedule_v2.SchedulerWorkerThread.run(SchedulerWorkerThread.java:100)

 

0 Helpfuls

atanas8d
Tera Contributor

‎11-23-2023 07:22 AM - edited ‎11-23-2023 07:26 AM

I'm posting what I found out in the process of researching and debugging. 

 
1. The stats.do information can be retrieved from Global scope with code similar to:

 

 

var diag = new Diagnostics();

while (diag.nextNode()) 
{
    var diagNode = diag.getNode();
    var nodeName = diagNode.name;
    var memUse = diagNode.stats['system.memory.in.use'];
    var memFree = diagNode.stats['system.memory.pct.free'];
    gs.info("Node name: " + nodeName + " Memory in use: " + memUse + " Memory % Free: " + memFree);
}

 

 

As the Diagnostics object is not accessible from scoped applications, a workaround could be an application property (for example instance_progress_workers_pool_count) that could be fulfilled as a setup process on the respective instance.

 

2. The second problem I was facing with the "Complete" state of a progress worker for "0 Seconds" was caused by a misplaced argument order passed to the progress worker while using GlideScriptedHierarchicalWorker API in the code. The workaround to avoid such cases is to use one input argument as an object of parameters:

 

 

var worker = new GlideScriptedHierarchicalWorker();
worker.setProgressName(progressWorkerName);
worker.setBackground(true);
worker.setScriptIncludeName(task.scriptIncludeAPI);
worker.setScriptIncludeMethod(task.scriptIncludeMethod);
worker.putMethodArg('args', argumentsObject);
worker.start();

 

 


Another important detail I think is worth noticing while using progress workers is that if an execution error occurs while executing the passed script including the method with the respective argument/s, the progress workers will end up in a Complete state. The progress worker is running on its own thread and is considered completed by the system, regardless of the execution results in error from what I tested. To overcome this issue, I suggest passing a sys_id of an existing db object responsible for the status of the work passed to the progress worker, which should be used to update the status and potentially log a reason in a try-catch block or in a conditional block which catches unexpected results. 

0 Helpfuls