How Mary Kay tamed the ITOM beast

Task templates and action alerts are powerful tools for automating IT incident response in ServiceNow, especially for organizations that lack a sophisticated Configuration Management Database (CMDB). Using the two in tandem enables filtering of incoming events, prescribes steps for determining severity, and helps determine how incident response should be assigned.

But as Louis Oliver, senior technical consultant at Mary Kay, explained in a breakout session at Knowledge 2019, these advanced automation tools have long had hidden drawbacks. With the ServiceNow Kingston release, task templates often required rigid sets of rules to define sequences of steps taken by various assignment groups. The more steps in sequence and the more groups of users potentially impacted, the greater the chance alerts could become unwieldy to manage and resolve.

The rigidity of task templates also made them brittle. If some aspect of a template needed to be revised—for example, if a personnel change placed a new person in charge of a particular service—changing the filters for that instance would change them for every instance.

As a result, Oliver’s team would frequently get bogged down in manual triage before incidents could be assigned—a process that occasionally went on for weeks. “It breaks the whole thing,” said Oliver. Somebody would call us and say, ‘Hey, I got a ticket that shouldn't go to my team. Why did we get this?’ We were getting those calls all the time.”

With the London release came the solution Oliver was waiting for: a new alert management tool and another tool to manage subflows—workflows that operate beneath a parent workflow. Like parent workflows, subflows are flexible and reusable, but they handle more specific routines. For Oliver, implementing these features prevented alerts from triggering exponentially-expanding branches of task templates. They also created a clearer view of the specific workflow tied to each alert.

The upgrade paid off dramatically. “On average, we used to work through seven or more task templates and order action rules for each alert management rule per element,” said Oliver. Mary Kay has since slashed average incident resolution times from more than seven days to less than 24 hours. In many cases, Oliver’s team can resolve incidents in less than an hour.

“Before, it might take us two to three weeks to figure out who to assign an incident to make sure we were going to the right place, and not accidentally breaking something else,” Oliver said. “Now it can take as little as 15 minutes.”

Oliver advised companies still running the Kingston release to upgrade to London before attempting to create subflows on their own, as these subflows won’t be editable after making the upgrade.

“If you haven't gone to London yet and you're heavily leveraging alert action rules and task templates, be prepared,” said Oliver. “You'll have to change the way you do it.”