Unable to create alert from datadog event

[poojashah27]

Hi,

i am using datadog and event management integration to integrate datadog alert into ServiceNow.

when alert gets triggered on datadog with @servicenow it created event under datadog event table and under events (em_events) but it is unable to create alert out of event, even though having event rule (with no specific filters).

can someone please help me find issue here?

[Gianpaolo Pagan]

Have you followed the Datadog Event Integration instructions?

Can you check if all the mandatory fields are correctly populated?

Also check if data is consistent, like severity, which is an enumeration even though it visualizes the Severity strings (warning, critical, etc.), so severity should be one of the following numbers: 1=Critical, 2=Major, 3=Minor, 4=Warning, 5=Information, 0=Clear.

If you could post a sample event that's not producing the desired alert and details about your event rule, the community could be more helpful.

I hope this helps,

Gp

[poojashah27]

Hi Gp,

yes i followed instructions for datadog event integration.

required fields are also getting populated correctly.

please see below sample payload.

{
"alert_metric":"system.mem.pct_usable",
"alert_query":"avg(last_5m):avg:system.mem.pct_usable{*} by {host} * 100 < 10",
"datadog_tags":"[account_id:1231243214, availability-zone:us-west-2b, billingcode:del02054-01-01-01-nb10, client:mdis02, contacts:venkatappaiah_kodali_vivek_lodhi, country:us, cs:pii, csclass:hrc, csqual:sensitive_personal_information, cstype:client, environment:prod, function:con, groupcontact, host:parnwmdiap06, iam_profile:managedservices_role, image:ami-c1daaab9, instance-type:t2.micro, kernel:none, memberfirm:us, monitor, mrp:mitrava_sarkar, name:parnwmdiap06, patch_group:windows, primarycontact:venkatappaiah_kodali, projectname:mdis02, region:us-west-2, requestid, role:app, secondcontact:usciamoperations_deloitte.com, security-group:sg-0e4202c90438756a6, security-group:sg-46ef8636, security-group:sg-7cb27c02]",
"action":"create",
"alert_title":"Mem Alert Triggered",
"alert_transition":"Warn",
"event_msg":"%%% Metric Name: Memory Utilization\n\n @maliedike@deloitte.com @servicenow [](https://app.datadoghq.com/monitors#6285715?to_ts=1552655565000&group=host%3Adsfgsdfgdfsg03&from_ts=1552651905000) `avg(last_5m):avg:system.mem.pct_usable{*} by {host} * 100 < 10` The monitor was last triggered at Fri Mar 15 2019 13:11:55 UTC (**1 sec ago**). - - - [[Monitor Status](https://app.datadoghq.com/monitors#6285715?group=host%3Adsfgsdfgdfsg03)] · [[Edit Monitor](https://app.datadoghq.com/monitors#6285715/edit)] · [[View dsfgsdfgdfsg03](https://app.datadoghq.com/infrastructure?filter=dsfgsdfgdfsg03)] · [[Show Processes](https://app.datadoghq.com/process?sort=memory%2CDESC&to_ts=1552655635000&tags=host%3Adsfgsdfgdfsg03&from_ts=1552654615000&live=false&showSummaryGraphs=true)] %%%",
"event_title":"[Warn] Mem Alert Triggered",
"hostname":"dsfgsdfgdfsg03",
"pretty_event_details":"datadog: \torg_name: test Pod \talert_type: warning \tlast_updated: 1552655516000 \tevent_type: query_alert_monitor \tmonitors_url: https://app.datadoghq.com/monitors/triggered \tuser: None \tdd_tags: \t\tauto_tags: ['dsfgsdfgdfsg03', 'availability-zone:us-west-2b', 'image:ami-c1daaab9', 'instance-type:t2.micro', 'kernel:none', 'dsfgsdfgdfsg03', 'region:us-west-2', 'role:app'] \t\tcustom_tags: ['account_id:12345678910'] \tsource_type_name: Monitor Alert \tmessage: Metric Name: Memory Utilization  @servicenow `avg(last_5m):avg:system.mem.pct_usable{*} by {host} * 100 < 10` Metric value: 23.935 \tclient_url: https://app.datadoghq.com/monitors#6285715?to_ts=1552655565000&group=host%3Adsfgsdfgdfsg03&from_ts=1552651905000 \ttitle: [Warn] Mem Alert Triggered \tevent_id: 4567897651548945132\tsnap_url: https://p.datadoghq.com/snapshot/view/ \tpriority: normal \thost_name: dsfgsdfgdfsg03 \tmetric_query: avg(last_5m):avg:system.mem.pct_usable{*} by {host} * 100 < 10 user_configured: \tHostname: $HOSTNAME \tMetric: $ALERT_METRIC \tOrg ID: $ORG_ID \tDescription: $EVENT_MSG \tTitle: $EVENT_TITLE",
"text_only_msg":"Metric Name: Memory Utilization @servicenow `avg(last_5m):avg:system.mem.pct_usable{*} by {host} * 100 < 10` Metric value: 23.935 Metric Graph: https://app.datadoghq.com/monitors#6285715?to_ts=1552655565000&group=host%dsfgsdfgdfsg03&from_ts=1552651905000 · Monitor Status: https://app.datadoghq.com/monitors#6285715?group=host%3Adsfgsdfgdfsg03 · Edit Monitor: https://app.datadoghq.com/monitors#6285715/edit · Event URL: https://app.datadoghq.com/event/event?id=4838425786361424266 · View dsfgsdfgdfsg03: https://app.datadoghq.com/infrastructure?filter=dsfgsdfgdfsg03"
}

[Gianpaolo Pagan · Accepted solution]

Something seems off with some fields...

state is 1 but it's supposed to show one of Ready, Processed, Ignored or Error

"additional information" value is "mem alert triggered" but it's supposed to be a JSON structure.

Just for the sake of a test, please edit the event you sent me:

1. change additional information from "mem alert triggered" to
2. 1. {"test_field":"mem alert triggered"}
3. what do you see in the field "state" when you open the dropdown list? If you have "Ready" an option, select it.
4. Also the "resource" field value doesn't make much sense, but that shouldn't affect the alert processing.
5. Reprocess the event by saving or inserting the record.

It seems something is not working properly in the datadog to SN event translation. Either the integration is not properly installed or something has changed in the Datadog source payload format.

[poojashah27]

Thanks Gp.

I was able to resolve it by changing transform map for state field it was setting it to value 1/2/3 rather than Ready/Processed/Ignored.