Azure "New Metric Alerts" as an Event Source

J0pesh · ‎09-13-2019

Hi.

We're currently in Kingston but about to upgrade to Madrid. So i'll ask this question from a Madrid perspective.

Our Azure Dev Ops team is helping me implement Web Hooks from Azure to ServiceNow, We set a few up already for the "classic Metric Alerts" but they're working to make these redundant and will in the next few months be basing all monitoring within Azure on the "New Metric Alerts" in Azure.

In the ServiceNow documentation, there isn't much (anything) that I can find about new metric alerts.

Anyone got any experience with setting these up and steps I should take to getting it right? I'm still relatively new to ServiceNow and by no means an expert in Event management.. This is a new implementation for us so please be gentle with me 🙂

Thanks

Joe

dbehnood · ‎09-13-2019

Hey Joe,

I posted this on a prior thread, but might be useful for you.

The out of the box integration provided by SNOW, documented here (https://docs.servicenow.com/bundle/newyork-it-operations-management/page/product/event-management/ta...), only works for 'Classic' format alerts which are being deprecated by Microsoft on Aug 31st.

The only options available to you are:

1) Create a custom Event Listener Transform Script that can intake Events in the new formats (metric and log being the most common schemas)

2) Send Events to the GenericJSON listener and use a combination of Event Transform and Event Field Mapping rules (https://<<INSTANCE>>/api/global/em/inbound_event?source=genericJson)

I went with option 2 for expediency, and it is working well for us. If you go this route you will need to disable authentication required for the GenericJSON listener (Transform Generic Events MID).

In case anyone is curious, here is how I built this out.

1) Create an Action Group in Azure monitor that calls a webhook action. Specify the webhook URI as https://<<INSTANCE>>/api/global/em/inbound_event?source=genericJson

2) Associate the Action with an Alert rule, and force it to fire (just to capture an Event in ServiceNow)

3) Create a new Event rule:

- Source = GenericJson

- Event filter should include the following condition: schemaID is AzureMonitorMetricAlert

- Transform and compose the following:

a) Description:

Resource: ${data_context_resourceName}
Resource Group: ${data_context_resourceGroupName}

Alert Name: ${data_context_name}
Metric Name: ${metricName}
Operator: ${operator}
Threshold: ${threshold}
Current Value: ${metricValue}

Portal link: ${data_context_portalLink}

${data_context_description}

b) Node: ${data_context_resourceName}

c) Type: ${data_context_name}

d) Resource: ${data_context_resourceType}

e) Message key: ${data_context_resourceName}_${data_context_name}_${data_context_resourceType}

f) Severity: ${severity}

g) Metric Name: ${metricName}

h) Source instance: ${data_context_resourceGroupName}

i) Source: Microsoft Azure

4) We must use regex within the Event Rule rule to get the following nested objects from within the “data_context_condition_allOf” value and convert them into usable values of their own:

Metric Name (metricName)
Operator (operator)
Threshold (threshold)
Metric Value (metricValue)

The expression used to accomplish this is: .*metricName=(.*), metricN.*operator=(.*), threshold=(.*), time.*metricValue=(.*)}.*

5) Create an Event Field Mapping Rule for Severity (THIS IS A MUST or your Sev 0 Alerts from Azure will force auto-closure of the Alert instead of being most critical)

From field: data_context_severity

To field: Severity

Event Mapping Pairs:

0 -> 1

1 -> 2

2 -> 3

3 -> 4

4 -> 5

6) Create an Event Field Mapping Rule for Resolution State (so that the Alerts auto-close)

From field: data_status

To field: resolution_state

Event Mapping Pairs:

Activated -> New

Deactivated -> Closing

I might have gone overkill on the content I added to the description, but our Ops teams wanted as much info in their Incident as possible. You can tweak this to your organization's needs. Hope it helps!

-Dom

View solution in original post

Pistle · ‎10-05-2020

Azure Monitor now supports a new metric alert type which has significant benefits over the older classic metric alerts. Metrics are available for large list of Azure services. The newer alerts support a (growing) subset of the resource types. This article lists that subset. You can also use newer metric alerts on popular log data stored in a Log Analytics workspace extracted as metrics.