question Unable to create alert from datadog event in ITOM forum

Unable to create alert from datadog event

poojashah27 — Mon, 18 Mar 2019 20:19:14 GMT

Hi,

i am using datadog and event management integration to integrate datadog alert into ServiceNow.

when alert gets triggered on datadog with @servicenow it created event under datadog event table and under events (em_events) but it is unable to create alert out of event, even though having event rule (with no specific filters).

can someone please help me find issue here?

Re: Unable to create alert from datadog event

Gianpaolo Pagan — Tue, 19 Mar 2019 09:27:35 GMT

Have you followed the Datadog Event Integration instructions?

Can you check if all the mandatory fields are correctly populated?

Also check if data is consistent, like severity, which is an enumeration even though it visualizes the Severity strings (warning, critical, etc.), so severity should be one of the following numbers: 1=Critical, 2=Major, 3=Minor, 4=Warning, 5=Information, 0=Clear.

If you could post a sample event that's not producing the desired alert and details about your event rule, the community could be more helpful.

I hope this helps,

Re: Unable to create alert from datadog event

poojashah27 — Tue, 19 Mar 2019 14:21:38 GMT

Hi Gp,

yes i followed instructions for datadog event integration.

required fields are also getting populated correctly.

please see below sample payload.

{
"alert_metric":"system.mem.pct_usable",
"alert_query":"avg(last_5m):avg:system.mem.pct_usable{*} by {host} * 100 < 10",
"datadog_tags":"[account_id:1231243214, availability-zone:us-west-2b, billingcode:del02054-01-01-01-nb10, client:mdis02, contacts:venkatappaiah_kodali_vivek_lodhi, country:us, cs:pii, csclass:hrc, csqual:sensitive_personal_information, cstype:client, environment:prod, function:con, groupcontact, host:parnwmdiap06, iam_profile:managedservices_role, image:ami-c1daaab9, instance-type:t2.micro, kernel:none, memberfirm:us, monitor, mrp:mitrava_sarkar, name:parnwmdiap06, patch_group:windows, primarycontact:venkatappaiah_kodali, projectname:mdis02, region:us-west-2, requestid, role:app, secondcontact:usciamoperations_deloitte.com, security-group:sg-0e4202c90438756a6, security-group:sg-46ef8636, security-group:sg-7cb27c02]",
"action":"create",
"alert_title":"Mem Alert Triggered",
"alert_transition":"Warn",
"event_msg":"%%% Metric Name: Memory Utilization\n\n @maliedike@deloitte.com @servicenow [![Metric Graph](https://p.datadoghq.com/snapshot/view/dd-snapshots-prod/org_207986/2019-03-15/ebb07bae8a368174a3889df1bebdba49b6bab940.png)](https://app.datadoghq.com/monitors#6285715?to_ts=1552655565000&group=host%3Adsfgsdfgdfsg03&from_ts=1552651905000) `avg(last_5m):avg:system.mem.pct_usable{*} by {host} * 100 < 10` The monitor was last triggered at Fri Mar 15 2019 13:11:55 UTC (**1 sec ago**). - - - [[Monitor Status](https://app.datadoghq.com/monitors#6285715?group=host%3Adsfgsdfgdfsg03)] · [[Edit Monitor](https://app.datadoghq.com/monitors#6285715/edit)] · [[View dsfgsdfgdfsg03](https://app.datadoghq.com/infrastructure?filter=dsfgsdfgdfsg03)] · [[Show Processes](https://app.datadoghq.com/process?sort=memory%2CDESC&to_ts=1552655635000&tags=host%3Adsfgsdfgdfsg03&from_ts=1552654615000&live=false&showSummaryGraphs=true)] %%%",
"event_title":"[Warn] Mem Alert Triggered",
"hostname":"dsfgsdfgdfsg03",
"pretty_event_details":"datadog: \torg_name: test Pod \talert_type: warning \tlast_updated: 1552655516000 \tevent_type: query_alert_monitor \tmonitors_url: https://app.datadoghq.com/monitors/triggered \tuser: None \tdd_tags: \t\tauto_tags: ['dsfgsdfgdfsg03', 'availability-zone:us-west-2b', 'image:ami-c1daaab9', 'instance-type:t2.micro', 'kernel:none', 'dsfgsdfgdfsg03', 'region:us-west-2', 'role:app'] \t\tcustom_tags: ['account_id:12345678910'] \tsource_type_name: Monitor Alert \tmessage: Metric Name: Memory Utilization @servicenow `avg(last_5m):avg:system.mem.pct_usable{*} by {host} * 100 < 10` Metric value: 23.935 \tclient_url: https://app.datadoghq.com/monitors#6285715?to_ts=1552655565000&group=host%3Adsfgsdfgdfsg03&from_ts=1552651905000 \ttitle: [Warn] Mem Alert Triggered \tevent_id: 4567897651548945132\tsnap_url: https://p.datadoghq.com/snapshot/view/ \tpriority: normal \thost_name: dsfgsdfgdfsg03 \tmetric_query: avg(last_5m):avg:system.mem.pct_usable{*} by {host} * 100 < 10 user_configured: \tHostname: $HOSTNAME \tMetric: $ALERT_METRIC \tOrg ID: $ORG_ID \tDescription: $EVENT_MSG \tTitle: $EVENT_TITLE",
"text_only_msg":"Metric Name: Memory Utilization @servicenow `avg(last_5m):avg:system.mem.pct_usable{*} by {host} * 100 < 10` Metric value: 23.935 Metric Graph: https://app.datadoghq.com/monitors#6285715?to_ts=1552655565000&group=host%dsfgsdfgdfsg03&from_ts=1552651905000 · Monitor Status: https://app.datadoghq.com/monitors#6285715?group=host%3Adsfgsdfgdfsg03 · Edit Monitor: https://app.datadoghq.com/monitors#6285715/edit · Event URL: https://app.datadoghq.com/event/event?id=4838425786361424266 · View dsfgsdfgdfsg03: https://app.datadoghq.com/infrastructure?filter=dsfgsdfgdfsg03"
}

Re: Unable to create alert from datadog event

Gianpaolo Pagan — Tue, 19 Mar 2019 16:56:30 GMT

Something seems off with some fields...

state is 1 but it's supposed to show one of Ready, Processed, Ignored or Error

"additional information" value is "mem alert triggered" but it's supposed to be a JSON structure.

Just for the sake of a test, please edit the event you sent me:

change additional information from "mem alert triggered" to
1. {"test_field":"mem alert triggered"}
what do you see in the field "state" when you open the dropdown list? If you have "Ready" an option, select it.
Also the "resource" field value doesn't make much sense, but that shouldn't affect the alert processing.
Reprocess the event by saving or inserting the record.

It seems something is not working properly in the datadog to SN event translation. Either the integration is not properly installed or something has changed in the Datadog source payload format.

Re: Unable to create alert from datadog event

poojashah27 — Tue, 19 Mar 2019 18:03:15 GMT

Thanks Gp.

I was able to resolve it by changing transform map for state field it was setting it to value 1/2/3 rather than Ready/Processed/Ignored.

Re: Unable to create alert from datadog event

rohan81chou — Wed, 03 Jun 2026 06:28:00 GMT

Hi @Gianpaolo Pagan

We received all the alert, expect clear event. What we need to configure for clear event come into the servicenow from datadog

Re: Unable to create alert from datadog event

MushtaqMir — Wed, 03 Jun 2026 19:28:54 GMT

Hi,

Please check if clear events are being captured and are present in Datadog. If they are present and you are using the pull (GET) method, then verify the endpoint query parameters. If you are using the push (POST) method, kindly check with the Datadog team to understand why clear events are not being posted.

Kindly keep us updated with your findings.

Re: Unable to create alert from datadog event

rohan81chou — Fri, 05 Jun 2026 05:17:54 GMT

Hi @MushtaqMir

I checked, I get the notification of the Triggred and Recovered into the mailbox,

but not receiving the clear event in servicenow event table. It update the event only but state and severity is not changed.

Re: Unable to create alert from datadog event

rohan81chou — Fri, 05 Jun 2026 05:22:47 GMT

Hi @MushtaqMir

Receiving the Triggered and Recovered email into the mailbox, but clear event is not came into the servicenow. It updates the event count. Separate clear event is not generated into the servicenow.