A large part of "under the hood" communications and notifications in the ServiceNow platform are managed by events. These are triggered by specific actions, such as: an incident being created or updated, a requested item that needs an approval, an SLA that has surpassed a predefined threshold, and many more. All these events need to be processed for the needed actions, such as sending out emails and SMS, updating a Live Feed, etc. are performed. The mechanism responsible for this is the "events processor."
Starting with the Fuji release we have made some changes to improve events processing (not to be confused with the ITOM "Event Management" application). The way in which the platform generates events and the logic behind their processing has not been changed, but we have introduced several improvements in the manner in which "ready events" are picked up and processed. This method reduces the possibility of a single event being the bottleneck for the processing of other events behind it.
One events processor per node
Starting with the Fuji release, we have one events processor (Scheduled Job) available on each active node of the instance. This only applies to the "default" event queue; the "text_index", "metric_update" and any other custom queues will continue to be processed by their respective processors.
On a pre-Fuji instance (Aspen, Berlin, Calgary, Dublin, and Eureka) you would typically see the following default Scheduled Job:
This processor job would pick up all the events in "ready" state and queue them for processing. If there had been a contention in the previous run of the job (due to a stuck event, or a bunch of long-running events), there could easily be a backlog of several thousand events which will cause the next run of the job to also be delayed. It would take several runs of the events processor to return to normality. On an instance that generates very many events, it could easily take a day or more to return to normality.
The biggest difference you will see as an administrator of an instance on the Fuji release or newer are the following scheduled jobs named "events process n" (where n is a number) with the corresponding master job.
You will also notice that the method in the Job Context section is now "gs.processDelegatedEvents()", which depends on the Event Delegator.
Event Delegator
The Event Delegator is an internal process that runs on one of the active nodes and checks every 10 seconds for events it can delegate to the processors. By default, each processor will not be assigned more than 3,000 events (in previous releases, the job would claim all events in "ready" state) and will process them sequentially, in order of creation.
This new mechanism helps handle load and bursts of events in a more robust way. In the example above, the contention scenario would have had a minor impact as the "stuck" queue would not have caused a large backlog of events.
Resources:
Scheduled Script Executions and Events (Developer Training Module)
4 Comments
