The notification from the ServiceNow UI can be extremely flexible with configuration. With all of the configuration capabilities and platform flexibilities for creating email notifications, you can end up overlooking how complex notifications can be. Notifications are created with a purpose to "inform about an event" and they are happily self-contained to integrate with the systems. However, this independence and simplicity can cause oversight on them triggering the same message.
Lets talk about
- The problem with events stacked that send the same notification
- Example of events stacked sending the same notification message
- Using event parm1 values to overcome events sending the same notification message
- Using the notification weight to overcome duplicated email
Most events are not processed straight away. Events will queue up and they have to hold until previous events are processed. Once the previous events are processed, then the following events can fire. The sysevent table holds the event queue generated by the system.
Events allow you to specify:
- Event name
- Target record
- Two parameters
They have four states:
- Ready
- Processed
- Error
- Transferred
Events named 'notification_engine.process' are Engine based. Any other event, not named 'notification_engine.process' is considered Event based. A simplified version to explain the event life cycle looks like this:
Delay | Event cycle | Data changes |
Events are created in "Ready" state | Record expected by event is accurate | |
#1 Event is queued — this is the most likely delay | Events WAIT - this could be seconds, or minutes if there are previous events or the event processor is stopped, etc | Users or workflow may change the record |
Processing start | Events are 'claimed' by a node, then set to "processed" state when completed If they trigger email notifications, the conditions are evaluated here with the 'current' record* | *Record can have changed by this time |
#2 Delay depends on the number of recipients | Then email notification is processed to generate the recipients | Users or workflow may change the record |
Then generate the notification message with the 'current' record** | **Record can have changed again by it is less likely | |
Processing ends | Processed time is set |
The Event Processor in the ServiceNow system is in charge of handling the events that come in. Think of it as the swiss army knife of the notification system. Being that the Event Processor has several actions to review, manage and fire, like any assembly line, if it gets too backed up it could cause performance issues. If the event processor is stopped, or busy, it can cause a long queue list of events. As default, every 30 seconds gs.eventsProcess() runs on each node and clears the 'ready' events.
You recognise this problem because the notification usually has a conditions with data is likely to change (e.g. conditions on assigned group, assigned to, etc.) and it's usually the source of the triggering event itself (e.g. incident.assigned)
The source of the problem with events stacked that send the same notification is that there is a delay between event creation and the event being processed. There is a tiny chance of delay on recipient creation. By the time the notification conditions are processed there is a chance the current record could have changed. The problem here is that an event that should have not passed the conditions, will later pass them because the new values passes the notification conditions. In most cases the result is the same message send twice.
Here is an example of events stacked sending the same notification message:
To better articulate why duplicate notifications due the event stacked is an issue, and passing outdated records through validation is bad, I have put together an example.
I have set three(3) users:
- Alejandra Prenatt - Sysid = 22826bf03710200044e0bfc8bcbe5dec
- David Loo
- Charles Tylor (Duty Manager).
I created a test notification that should only fire with assigned to is "Alejandra Prenatt":
Name | = | test_notification_01 |
Table | = | Incident |
Event name | = | incident.assigned |
Condition | = | assigned_to = "Alejandra Prenatt" |
Subject: | = | This incident Number: ${number} - Assigned to: ${assigned_to} |
Message | = | Assigned to: ${assigned_to} |
Here is how the notification looks:
I tested it by creating the two quick changes in incident:
- Then go to Incident table and assign an incident to "David Loo" (internally, event 1 fires)
- then quickly assign it to "Alejandra Prenatt." (internally, event 2 fires)
You will receive two notifications with the same subject and recipient is "Alejandra Prenatt." Reviewing the logic, you can see the first update 'Assigned to' = "David Loo" (event 1) on which the email notification SHOULD HAVE NOT PASSED. However, another update with 'Assigned to' = "Alejandra Prenatt" happens (event 2). In practice, there is a delay for the event to be processed, the current record 'Assigned to' = "Alejandra Prenatt" which now matches the condition and when processing event 1 and 2, it sends both notifications.
This is a normal but unwanted behavior. Keep that in mind when designing notifications.
However, there are multiple workarounds.
One solution is to create specific events that get used on those specific cases instead of conditions based on data that may change, then limit the notifications to those events
Workaround #1: Using event parm1 values to overcome the events sending the same notification message
You can modify the email notification condition to validate by the data passed on event.parm1 or event.parm2.
Here is an example, changing the notification advanced condition:
Advanced condition =
checkAssignedToUser();
// Check if the parameter provided is the user
function checkAssignedToUser() {
var vbool = (event.parm1 == 'Alejandra Prenatt');
return vbool;
}
Here is how it looks in the notification:
I tested it by creating the two quick changes in incident: Then go to Incident table and assign an incident to "Charles Tylor (Duty Manager)," then quickly assigned it to "Alejandra Prenatt."
Two events were generated on 'incident.assigned.' One contains parm1="Alexandra Prennatt" and the other parm1="Charles Tylor (Duty Manager)." Even when current.assigned is "Alexandra Prenatt," the event that has parm1 parameter = "Charles Tylor (Duty Manager)" does not match the condition and it does not generate a notification, so only one notification is sent to "Alejandra Prenatt."
Workaround #2: Use the notification weight to overcome duplicated email
Another option is to set the weight to a value higher than zero (0). By setting the weight to a value more than zero, it will cause the business rule to "Ignore duplicates" on the sysemail table to trigger and send any duplicated emails into the ignored mailbox. This options is easier and cleaner than the previous workaround but it generates an unwanted email in the ignored mailbox.
On my case, I set the weight to 10. Then retested,
I tested it by creating the two quick changes in incident: Then go to Incident table and assign an incident to "David Loo," then
quickly to "Alejandra Prenatt."
The system generated two emails to "Alejandra Prenatt." However, only one email remains on ready to send. The other was sent to the ignored mailbox.
This only works if both emails are in 'ready-to-send' state at the time the last event is processed.
In conclusion, please be aware of the time gap between when the event was generated and when the event was processed. This is normal behavior that can cause unwanted results like notifications that look like duplicates.
More information can be found here:
2 Comments
