On-call escalations allow you to inform your primary, secondary, tertiary, ... on-call engineer that an issue requires their immediate attention. I had a few customers ask me about on-call escalations these last few weeks. In trying to answer my customer's questions, I found that I needed to dig deeper into the issue and — since doing it anyway — I want to share my findings with this stellar community.
The first issue I encountered, was that when using the base system (out of box) Escalation Email workflow, the initial escalation message that should look similar to this:
is not entered into the incident at all and if the number of reminders is set to 0, it shows NaN:
Additionally, during the escalation through the different on-call levels, it enters into the comments
without mentioning who was reminded.
The next issue was that the workflow through the trigger rule only fired if the assignment group was not filled in. In my system, the assignment groups are automatically filled based on the incident category. I was able to use one or the other rule, but not both. I created a workaround using a business rule that empties out the assignment group, but with that, the workflow then fired, and it kept firing over and over and over again, until I had over a dozen workflows associated to my incident. Ack!
And on top of all of these items, I also attempted to modify the workflow to alert, not just the current escalation point but the past escalation points as well. I had to first understand, what inputs the on-call rotation scripts and functions are using, so that I could then use them myself to work as needed.
Syntax Usage of the OnCallRotation scripts
The following methods are used in scripts and workflows to determine who the current person on call for a specific group is as well as other related information.
1. getEscalationPlan(sys_id pointing to assignment group, current date/time)
inputs:
sys_id for the current record's assignment group
Date/Time value retrieved through GlideDateTime()
Sample usage:
var rota = new SNC.OnCallRotation();
var gdt = new GlideDateTime();
var escalationPlan = rota.getEscalationPlan(workflow.scratchpad.assignment_group, gdt);
output:
gs.log("Escalation Plan " + escalationPlan, "Escalation");
Escalation Plan [Escalatee [Order:1, UserId:46d44a23a9fe19810012d100cca80666, DeviceId:null, IsDevice:false, TimeBetweenReminders:1970-01-01 00:10:00, ReminderNum:2, Roster ID:633a3aeaeb201100fcfb858ad106fe0a], Escalatee [Order:2, UserId:5137153cc611227c000bbd1bd8cd2007, DeviceId:null, IsDevice:false, TimeBetweenReminders:1970-01-01 00:05:00, ReminderNum:1, Roster ID:7a5007eeeb201100fcfb858ad106fe8b], Escalatee [Order:3, UserId:681b365ec0a80164000fb0b05854a0cd, DeviceId:null, IsDevice:false, TimeBetweenReminders:1970-01-01 00:05:00, ReminderNum:0, Roster ID:b36007eeeb201100fcfb858ad106fe98]]
The Escalation Plan is a list (not an array) of all the escalatees defined for this plan. To retrieve elements of this list, use the get(level number in string format) method defined against this list as described in 2).
2. escalationPlan.get(level number representing the current level: 1=Primary, 2=Secondary, …)
input:
current level number (1, 2, ...) but in string format
sample usage:
var level = workflow.scratchpad.currentLevel+"";
var escalatee = escalationPlan.get(level-1);
output:
- gs.log("Escalatee " + escalatee, "Escalation");
Escalatee Escalatee [Order:1, UserId:46d44a23a9fe19810012d100cca80666, DeviceId:null, IsDevice:false, TimeBetweenReminders:1970-01-01 00:10:00, ReminderNum:2, Roster ID:633a3aeaeb201100fcfb858ad106fe0a]
This returns a list containing the information for the escalation on that level. Contents of individual areas can be addressed as escalatee.<field label before the 😆
- escalatee.timeBetweenReminders.getNumericValue()
- escalatee.reminderNum;
3. getCurrentRotaID()
inputs:
none
Sample usage:
workflow.scratchpad.currentRotaID = rota.getCurrentRotaID(); // store the rota sys_id
// Check if we have more people on the escalation lineup
output:
gs.log("Current Rota ID " + workflow.scratchpad.currentRotaID, "Escalation");
Current Rota ID af3a3aeaeb201100fcfb858ad106fe09
This returns the sys_id of the Rota record (cmn_rota) that is linked to the current assignment group.
4. getEscalateeAt(current assignment group, date/time, escalation level: 1=Primary, 2=Secondary, …)
input:
sys_id of the current record's assignment group
date/time in format of GlideDateTime()
escalation level in string format
sample usage:
var currentEscalatee = rota.getEscalateeAt(workflow.scratchpad.assignment_group, gdt, level);
ouptut:
- gs.log("Current Escalatee" + currentEscalatee, "Escalation");
Current Escalatee[object GlideRecord]
This method returns the current (for this level) Escalatee's user (sys_user) record. The current escalatee ID then is the user's sys_id.
5. Updating the current incident
5.1 - Initial Escalation update
writeLineupToIncidentActivityLog(escalation Plan list, Number of reminders for this escalatee, Time inbetween individual reminders, current incident's sys_id)
inputs:
Escalation plan list (returned from escalationPlan.get())
Number of reminders from this escalatee (also retrieved from the escalationPlan)
Time between individual reminders (also retrieved from the escalationPlan)
sys_id of the current incident
Sample Usage:
function writeLineupToIncidentActivityLog(plan, numberOfReminders, timeBetweenReminders, incident) {
var message = gs.getMessage("This incident Escalation is in Process. Escalation Plan is now: ");
var lineBreak = " ";
message += lineBreak;
for(var i = 0; i < plan.size(); i++) {
//show next escalatee and amount of reminders
var userId = plan.get(i).userId;
var userGR = new GlideRecord('sys_user');
userGR.get(userId);
var name = userGR.name;
var reminder = gs.getMessage("reminders");
if(numberOfReminders == 1)
reminder = gs.getMessage("reminder");
if(numberOfReminders == 0)
numberOfReminders = "no";
message += name + " (" + numberOfReminders + " " + reminder + ") ";
message += lineBreak;
if(i < plan.size()) {
//show when next escalation will happen
//take number of reminders times time between reminders
var time = numberOfReminders * timeBetweenReminders;
message += gs.getMessage("In ") + time/60 + " " + gs.getMessage("minutes") + ". ";
}
current.comments = message;
}
output:
A message in the incident's comments
5.2 - Updates whenever an escalation email is sent to escalatee:
writeEscalationToIncidentActivityLog(Name of the current Escalatee, sys_id of the current incident)
inputs:
user name of the current escalatee
sys_id of the current incident
Sample Usage:
function writeEscalationToIncidentActivityLog(name, incidentId) {
var message = gs.getMessage("Incident escalated to " + name);
current.comments = message;
}
output:
A message in the incident's comments
*Incident escalated to David Loo |
Sample setup to alert escalatees when a high priority incident is unassigned
Now that we know how to call the different methods related to the on-call code, this example will show how to create a copy of the out of box workflow to email the escalatees and modify it to alert not just the current level, but also the prior level(s), send an email, and start this flow whenever an incident matches the criteria (eg. P1, P2, or assignment group changes, etc)
Switching from trigger rule to business rule
I started out by changing from the trigger rule to calling my workflow from a business rule instead. That gave me better control under which circumstances I want the workflow called. I chose to start my workflow after the priority changes to a P1 or P2, or when the assignment group changes. This is set up through the Business Rule condition. I am running the business rule after insert or update.
function onAfter(current, previous) {
//This function will be automatically called when this rule is processed.
var wf = new Workflow();
wf.cancel(current);
gs.addInfoMessage(gs.getMessage("Workflows for {0} have been cancelled", current.getDisplayValue()) +" due to changed conditions");
wf.startFlow('043a297c131f5e00d57c50f32244b058', current, current.operation());
}
I chose to cancel the existing workflows, but you might not want to do that in your system, if you are planning to have more than one workflow associated with the record. The startFlow function needs to be passed the sys_id of the wf_workflow (Workflow) record, the current record, and the current operation as above.
Creating and modifying a copy of the On-Call Escalation Email workflow
Next, I created a copy of the out-of-box On-Call Escalation Email workflow and changed it to meet my requirements. I also included some fixes for the issues described in the beginning.
Because I called the workflow from a business rule, rather than creating a trigger rule, the "vars" object was not passed in and I had to change the "Is there any schedule at the time?" activity to use current instead of vars to determine the assignment group:
answer = ifScript();
function ifScript() {
var rota = new SNC.OnCallRotation();
var gdt = new GlideDateTime();
var escalationPlan = rota.getEscalationPlan(current.assignment_group.sys_id.toString(), gdt);
if (JSUtil.nil(rota) || JSUtil.nil(rota.getCurrentRotaID())) {
return 'no';
}
return 'yes';
}
Line 06 was changed from... var escalationPlan = rota.getEscalationPlan(vars.assignment_group.sys_id.toString(), gdt); to... var escalationPlan = rota.getEscalationPlan(current.assignment_group.sys_id.toString(), gdt); |
In the "More Escalation Levels Available?" activity, I had to do the same adjustment on line 08:
answer = ifScript();
function ifScript() {
// Set current level to 0 if we are here for first time
// and save the assignment group on the scratchpad
if(JSUtil.nil(workflow.scratchpad.currentLevel)){
workflow.scratchpad.currentLevel = 0;
workflow.scratchpad.assignment_group = current.assignment_group.sys_id + "";
}
For your reference, the full script in the "More Escalation Levels Available" activity then shows:
answer = ifScript();
function ifScript() {
// Set current level to 0 if we are here for first time
// and save the assignment group on the scratchpad
if(JSUtil.nil(workflow.scratchpad.currentLevel)){
workflow.scratchpad.currentLevel = 0;
workflow.scratchpad.assignment_group = current.assignment_group.sys_id + "";
}
var rota = new SNC.OnCallRotation();
var gdt = new GlideDateTime();
var escalationPlan = rota.getEscalationPlan(workflow.scratchpad.assignment_group, gdt);
workflow.scratchpad.escalationPlan=escalationPlan;
workflow.scratchpad.currentRotaID = rota.getCurrentRotaID(); // store the rota sys_id
// Check if we have more people on the escalation lineup
if(escalationPlan.size() > workflow.scratchpad.currentLevel) {
workflow.scratchpad.current_backup = workflow.scratchpad.currentLevel ? "backup" : "current";
workflow.scratchpad.currentLevel ++;
var level = workflow.scratchpad.currentLevel+"";
var escalatee = escalationPlan.get(level-1);
var currentEscalatee = rota.getEscalateeAt(workflow.scratchpad.assignment_group, gdt, level);
// Handle null due to time-off case
if(!currentEscalatee || !currentEscalatee.sys_id){
return ifScript();
}
workflow.scratchpad.currentEscalateeId = currentEscalatee.sys_id;
workflow.scratchpad.timeBetweenReminders = escalatee.timeBetweenReminders.getNumericValue() / 1000;
workflow.scratchpad.reminderNum = escalatee.reminderNum;
workflow.scratchpad.reminderSentNum = 0;
var user = new GlideRecord('sys_user');
user.get(currentEscalatee.sys_id);
//write to incident activity log
if(level == 1) {
//first message, presenting lineup
writeLineupToIncidentActivityLog(escalationPlan, escalatee.reminderNum, workflow.scratchpad.timeBetweenReminders, current.getUniqueValue());
}
else {
writeEscalationToIncidentActivityLog(user.name, current.getUniqueValue());
}
workflow.scratchpad.escalateeName = user.name;
return 'yes';
}
return 'no';
}
function writeEscalationToIncidentActivityLog(name, incidentId) {
var message = gs.getMessage("Incident escalated to " + name);
current.comments = message;
}
function writeLineupToIncidentActivityLog(plan, numberOfReminders, timeBetweenReminders, incident) {
//var message = "[code] ";
var message = gs.getMessage("This incident Escalation is in Process. Escalation Plan is now: ");
var lineBreak = " ";
message += lineBreak;
for(var i = 0; i < plan.size(); i++) {
//show next escalatee and amount of reminders
var userId = plan.get(i).userId;
var userGR = new GlideRecord('sys_user');
userGR.get(userId);
var name = userGR.name;
var reminder = gs.getMessage("reminders");
if(numberOfReminders == 1)
reminder = gs.getMessage("reminder");
if(numberOfReminders == 0)
numberOfReminders = "no";
message += name + " (" + numberOfReminders + " " + reminder + ") ";
message += lineBreak;
if(i < plan.size()) {
//show when next escalation will happen
//take number of reminders times time between reminders
var time = numberOfReminders * timeBetweenReminders;
message += gs.getMessage("In ") + time/60 + " " + gs.getMessage("minutes") + ". ";
}
}
//message += " [/code]";
current.comments = message;
}
Creating events to prompt the escalation emails
The majority of my changes happened in the "Escalatee has an e-mail?" activity since I switched the workflow from using a workflow notification to creating an event in the event queue. I did this to allow for more flexibility in the recipients as well as the content of the email through scripting. To accomplish this change, I had to move all notification work into this activity "Escalatee has an e-mail", where I create the event record with all needed parameters:
// run script
answer = ifScript();
// script to determine answer
function ifScript() {
// get the current escalatee
var currentEscalatee = new GlideRecord("sys_user");
currentEscalatee.get(workflow.scratchpad.currentEscalateeId);
workflow.scratchpad.escalateeName = currentEscalatee.name;
// determine does user has e-mail address
if (JSUtil.notNil(currentEscalatee.email)) {
//write to activity log
logToIncident(workflow.scratchpad.currentLevel, currentEscalatee.name, currentEscalatee.sys_id);
return 'yes';
} else {
// log error
var reason = gs.getMessage("Cannot escalate to {0}: no email address configured", [currentEscalatee.getValue('name')] );
workflow.error(reason);
activity.state = 'faulted';
activity.fault_description = reason;
return 'no';
}
}
function logToIncident(level, name,esc_id) {
var message = name + " is " + getLevelName(level) + " and has been contacted via Email";
current.comments = message;
var rota = new SNC.OnCallRotation();
var gdt= new GlideDateTime();
gs.eventQueue("OnCall.escalated", current, esc_id, getLevelName(level));
for (var esc_ind=1; esc_ind<level; esc_ind++) {
esc_ind_str=esc_ind+""
var priorEscalatee=rota.getEscalateeAt(workflow.scratchpad.assignment_group, gdt, esc_ind_str);
gs.eventQueue("OnCall.escalated", current, priorEscalatee.sys_id, getLevelName(esc_ind_str));
}
}
function getLevelName(level) {
var levels = [];
levels[1] = "Primary on-call";
levels[2] = "Secondary on-call";
levels[3] = "Tertiary on-call";
if(level > 3)
return level+"th level";
return levels[level];
}
I added line 09 to fix the issue where the incident update did not include the escalatee's name workflow.scratchpad.escalateeName = currentEscalatee.name |
The logToIncident function was modified from:
function writeToIncidentActivityLog(name) {
var message = gs.getMessage("Escalation plan exhausted and failed to assign incident. Communication has been sent to ");
message += name;
current.comments = message;
}
to
function logToIncident(level, name,esc_id) {
var message = name + " is " + getLevelName(level) + " and has been contacted via Email";
current.comments = message;
var rota = new SNC.OnCallRotation();
var gdt= new GlideDateTime();
gs.eventQueue("OnCall.escalated", current, esc_id, getLevelName(level));
for (var esc_ind=1; esc_ind<level; esc_ind++) {
esc_ind_str=esc_ind+""
var priorEscalatee=rota.getEscalateeAt(workflow.scratchpad.assignment_group, gdt, esc_ind_str);
gs.eventQueue("OnCall.escalated", current, priorEscalatee.sys_id, getLevelName(esc_ind_str));
}
}
These changes were done to be able to create the Event record that would then trigger the email notification. The parameters I am passing to the event record are:
esc_id = sys_id of the current Escalatee's user record
getLevelName(level), which will return the string:
- "Primary on-call" for level 1
- "Secondary on-call" for level 2
- "Tertiary on-call" for level 3
- "4th level on-call" for level 4, etc.
Since I wanted to email not only the current Escalatee, but all prior Escalatees up to that point as well, I added a for loop going through all levels prior to the current escalation level and created events for those users as well.
To inform the incident caller that the record was escalated in the "Reminder needed?" activity, I incorporated a fix to include the current escalatee in the incident's comments, eg. "Beth Anglin has been reminded via Email" (this was possible through adding and filling workflow.scratchpad.escalateeName in the "Escalatee has an e-mail?" activity highlighted above):
answer = ifScript();
function ifScript() {
// Send a reminder if needed
if (workflow.scratchpad.reminderSentNum < workflow.scratchpad.reminderNum) {
workflow.scratchpad.reminderSentNum ++ ;
workflow.scratchpad.reminder = "REMINDER: ";
activityLog();
return 'yes';
} else
workflow.scratchpad.reminder = "";
return 'no';
}
function activityLog() {
var message = gs.getMessage(workflow.scratchpad.escalateeName + ' has been reminded via Email');
current.comments = message;
}
I modified line 17 to fix the issue where the incident update did not include the escalatee's name var message = gs.getMessage(workflow.scratchpad.escalateeName + ' has been reminded via Email'); used to be: var message = gs.getMessage(workflow.scratchpad.currentEscalateeId + ' has been reminded via Email'); |
Create a notification and matching event registry
For event based notifications, we will first have to register the event in the Event Registry (System Policy > Events > Registry). I created the
OnCall.escalated event to be against the incident table and fired through a workflow.
I then created a notification (System Policy > Email > Notifications) as outlined below that runs when this event is fired. The mail script is necessary to determine the user's name based on the user's sys_id.
| Subject: |
Quick Assignment Required: ${number} |
Message text: |
${mail_script:UserName} An ${sys_class_name} ${number} has been raised: ${short_description} and you are identified as the ${event.parm2} person now. Please navigate to the ${sys_class_name} and assign it to yourself. Number: ${number} Click here to view the ${sys_class_name}: ${URI_REF} |
| Mail Script: |
var userID=event.parm1 var recipientUser=new GlideRecord("sys_user"); recipientUser.get(userID); var userName=recipientUser.name; template.print("Dear " + userName + ","); template.print("<br />"); |
You can only pass 2 parameters to the event queue, and the first has to be the recipient's (group or user) sys_id, as set by the "Event Parm 1 contains recipient" checkbox in the notification. To be able to inform the recipient of which level escalation this is in regards to, and address him or her by their full name, the mail script is used to retrieve the user's name based on the sys_id of the user record passed in as a parameter.
When all is done, the finished modified workflow show like this - a simplified flow with added functionality:
Armed with this knowledge and examples, you can now use the on-call module as needed by your organization's specific notification requirements. I am looking forward to your feedback and together going deeper in this very interesting topic.
3 Comments
