In my last blog, I described how you could use Event Rules to create meaningful alerts, while also introducing flexibility to cater for some variations. The example in the blog was fairly simple, so in this blog we'll up the difficulty factor and push the Event Rule Transform operation further.
In order to demonstrate this I'm going to enlist the services of our tame Solution Consultant.
Some say that he wanted to start a rock band called Woven Cries before music was invented, and that he inhales nitrous oxide to clear his nasal passages. All we know is that he's called The Scid!
Possibly a distant relative of The Stig from BBC's Top Gear TV show, The Scid starts off by looking at raw events in the All Events list view, and finds one that's particularly nasty. Nasty as in: no description, no event type, no severity... does it get any worse than this?? It's almost as if the event was created just so a blog could be written...
Anyhoo, he opens the event to see what he's got to work with. The Node field should be a hostname, there's no value for Type, Resource or Severity and the event State is "Error".
If he was able to speak, The Scid might utter a four letter obscenity... like "SNMP!". Fortunately the Description and Additional information fields contain information that could be useful. Additional information contains an itemName field with a Synthetic Transaction Check value that would be a good choice for an alert Type, while the status field could be mapped to Severity as The Scid knows it can be either PROBLEM or OK.
Scid can beat this nasty event into submission through the judicious application of a sledgehammer an Event Rule. He reaches for the keyboard and cunningly turns to regex101.com which he will use to build what he needs to produce an awesome Event Rule. Taking the text from the event's Description field, he pastes it into Regular Expression and Test String sections of the regex101 web page:
Regex101 allows you to build and test regular expressions using sample text and show the results in the Match Information section, plus it includes a useful quick reference guide for commonly used tokens. Regex101 is showing our Scid that there's an error caused by unescaped delimiters, but this barely registers on his care factor scale. The Scid wants to extract the correct Node and a Resource from the description field, so he uses a capture group:
That gives him what looks like a big URL, but he really wants to split that into FQHN and resource, so he first tries matching the forward slash (escaping it with a backslash):
The Match Information shows the result... have to try harder, Sciddy! That earns a chilling stare from the man in the mask and he comes up with a regex that gives the desired result:
The Scid has seen many things over the aeons including an event whose description ends in "returned an ok state". The regex will be changed to accommodate a description that contains either "not return" or "return", however capturing this group greatly displeases The Scid so he wields the non-capture "?:" magik:
Thank you, Regex101, Google and all those awesome nerds regex gurus! While he may be able to Google fast, can the Scid put this regex to any use? He returns to his ServiceNow instance, goes to Event Management > Rules > Event Rules and clicks on the New action button. Gloved fingers fly over the keyboard and before long an Event Rule has been created:
The Scid pastes the regex he devised in regex101.com into the Regular expression column for the description field in the Event Match Fields section. He maps the matching capture groups in the regex to the node and resource fields. Event Rules allow you to map multiple capture groups to multiple fields (OOTB alert fields, or non-OOTB fields stored as JSON pairs in the additional_info field). The Scid then composes the type field to use the itemName value extracted from the additional_info field. To make sure the Event Rule is applied to the correct events, he adds itemName to the Additional Filter and sets it to match Synthetic Transaction Check.
But he's not done yet - astute readers will recall that the State of the processed event was Error. That's not good. So the Scid creates an Event Field Mapping Rule by navigating to Event Management > Rules > Event Field Mapping Rule and clicking on New:
When an event occurs that contains a status field set to PROBLEM, the severity will be set to 2 (i.e. Major) and when an OK is received severity will be set to zero (i.e. Clear). Let's get out the stopwatch and test The Scid's awesomeness. Sending event... GO!!!
... a-a-a-n-n-d the alert crosses the finish line! Not only is the event successfully processed, but the resulting Alert now contains the correct Node, Type and Resource and a Severity.
When a clearing event arrives (it's not made up, right?), the Event Rule ensures the event is correctly matched to and closes the alert:
The Scid may give nothing away when it comes to facial expressions (he wears a mask), but he does like sharing his tool (ahem)... tools of the trade. With the aid of cattle prods The Scid is packed back into into his cozy cryotube ready for the next blog.
So the next time you're confronted by a nasty raw event, remember that you too can become awesome with event rules... just like The Scid .
2 Comments
