Azure "New Metric Alerts" as an Event Source

J0pesh · ‎09-13-2019

Hi.

We're currently in Kingston but about to upgrade to Madrid. So i'll ask this question from a Madrid perspective.

Our Azure Dev Ops team is helping me implement Web Hooks from Azure to ServiceNow, We set a few up already for the "classic Metric Alerts" but they're working to make these redundant and will in the next few months be basing all monitoring within Azure on the "New Metric Alerts" in Azure.

In the ServiceNow documentation, there isn't much (anything) that I can find about new metric alerts.

Anyone got any experience with setting these up and steps I should take to getting it right? I'm still relatively new to ServiceNow and by no means an expert in Event management.. This is a new implementation for us so please be gentle with me 🙂

Thanks

Joe

dbehnood · ‎09-13-2019

Hey Joe,

I posted this on a prior thread, but might be useful for you.

The out of the box integration provided by SNOW, documented here (https://docs.servicenow.com/bundle/newyork-it-operations-management/page/product/event-management/ta...), only works for 'Classic' format alerts which are being deprecated by Microsoft on Aug 31st.

The only options available to you are:

1) Create a custom Event Listener Transform Script that can intake Events in the new formats (metric and log being the most common schemas)

2) Send Events to the GenericJSON listener and use a combination of Event Transform and Event Field Mapping rules (https://<<INSTANCE>>/api/global/em/inbound_event?source=genericJson)

I went with option 2 for expediency, and it is working well for us. If you go this route you will need to disable authentication required for the GenericJSON listener (Transform Generic Events MID).

In case anyone is curious, here is how I built this out.

1) Create an Action Group in Azure monitor that calls a webhook action. Specify the webhook URI as https://<<INSTANCE>>/api/global/em/inbound_event?source=genericJson

2) Associate the Action with an Alert rule, and force it to fire (just to capture an Event in ServiceNow)

3) Create a new Event rule:

- Source = GenericJson

- Event filter should include the following condition: schemaID is AzureMonitorMetricAlert

- Transform and compose the following:

a) Description:

Resource: ${data_context_resourceName}
Resource Group: ${data_context_resourceGroupName}

Alert Name: ${data_context_name}
Metric Name: ${metricName}
Operator: ${operator}
Threshold: ${threshold}
Current Value: ${metricValue}

Portal link: ${data_context_portalLink}

${data_context_description}

b) Node: ${data_context_resourceName}

c) Type: ${data_context_name}

d) Resource: ${data_context_resourceType}

e) Message key: ${data_context_resourceName}_${data_context_name}_${data_context_resourceType}

f) Severity: ${severity}

g) Metric Name: ${metricName}

h) Source instance: ${data_context_resourceGroupName}

i) Source: Microsoft Azure

4) We must use regex within the Event Rule rule to get the following nested objects from within the “data_context_condition_allOf” value and convert them into usable values of their own:

Metric Name (metricName)
Operator (operator)
Threshold (threshold)
Metric Value (metricValue)

The expression used to accomplish this is: .*metricName=(.*), metricN.*operator=(.*), threshold=(.*), time.*metricValue=(.*)}.*

5) Create an Event Field Mapping Rule for Severity (THIS IS A MUST or your Sev 0 Alerts from Azure will force auto-closure of the Alert instead of being most critical)

From field: data_context_severity

To field: Severity

Event Mapping Pairs:

0 -> 1

1 -> 2

2 -> 3

3 -> 4

4 -> 5

6) Create an Event Field Mapping Rule for Resolution State (so that the Alerts auto-close)

From field: data_status

To field: resolution_state

Event Mapping Pairs:

Activated -> New

Deactivated -> Closing

I might have gone overkill on the content I added to the description, but our Ops teams wanted as much info in their Incident as possible. You can tweak this to your organization's needs. Hope it helps!

-Dom

View solution in original post

J0pesh · ‎09-16-2019

Thanks Dom. This looks great.

I've got a meeting in about an hour to start playing around with this, so i'll keep you posted.

dbehnood · ‎09-16-2019

Hey Joe,

You're welcome! I have started creating additional rules for the other JSON schemas for the other Alerts in Azure too - it has worked well for us since implementing it, and it is the best I could do with a narrow runway. SNOW knows that they need to update their solution, so I am sure an OOTB option will be available in a not-so-distant release.

-Dom

dbehnood · ‎09-23-2019

Joe,

Glad this worked out for you. I updated the original Regex used to grab the Alert criteria values. Microsoft doesn't always send the same JSON.

.*metricName=([^,}]*), .*operator=([^,}]*), .*threshold=([^,}]*), .*metricValue=([^,}]*).*.

I will update my original reply to include this, but wanted to ensure you saw it.

-Dom

Daigo1 · ‎10-25-2019

I'm sorry from the side.

The method described here seemed to be better than using "User App for Microsoft OMS integration".

Which is recommended? Please let me know if there are merits and demerits of this Webhook.

Thanks

dbehnood · ‎10-30-2019

I tried the "User App for Microsoft OMS integration" and was not very pleased with the ITSM Connector within Azure. If you were going straight to the Incident table it might be worth looking at, but into the Event table, I was very unimpressed. If I recall correctly, all Events come across with the same Severity, and you had to enter variable values for every rule vs. using standard mappings.