Event Management - Alert handling

[Patrick Quinlan]

We are attempting to configure event/alert management rules and actions for a particular monitoring tool. We have the events coming into ServiceNow and we have event rules being applied. On the associated alerts, we are seeing a behavior where an event for a particular resource is being attributed to an existing alert for the same resource even though the description of the event differs. When this occurs, it's overwriting the description of the existing alert. Our events currently don't have a "message key" and I believe this is the a contributing factor. My initial thought was to have the Event Rule transform the description into the message key, however by doing that, I'm afraid I will lose the ability to relate the "OK" events to the proper alert. 

Can anyone provide any guidance?

[Mark Manders]

Do you have that 'OK' within the description? In that case, maybe you should utilize some regex to exclude that part. But you are right about the message key being the issue (probably).

You can also check the alert correlation rule(s) applying to these alerts. You could have them correlate on other attributes. 

[Bhuvan]

@Patrick Quinlan 

Go through below links for guidance,

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0756665

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0753955

Thanks,

Bhuvan

[Jeff K1]

You are probably spot on with the message key potentially being your culprit.

When message key isn't set, it gets created with the following: Source, Type, Node, Resource, and Metric Name fields.

If those fields are the same for the differing description events, they will still be considered duplicates of one another and get grouped under the same Alert.

You would need to either have the source send unique message keys (we tend to use this route when onboarding new sources) or create the message key in the Event Rule that would make it unique.

I'm not sure if I'd go with having the description be the message key, unless it's the last resort, but I'd analyze the events closely to look for something else that might be able to make it unique enough to not group them, but still be vague enough for things I do want grouped together.

[AJ-TechTrek]

Hi @Patrick Quinlan ,

As per my understanding why It’s Happening

* ServiceNow groups events into alerts based on:
1. CI association (cmdb_ci)
2. Message Key (if present)
3. Additional grouping rules in Event Rules (alert aggregation criteria)
* If there is no message key, ServiceNow uses resource + node/CI as the correlation key.
* That means multiple events for the same CI but different descriptions are merged into the same alert, overwriting the description.

Solution Options
1. Define a Proper Message Key in Event Rules
* Best practice: The message key should uniquely identify the alert condition but stay consistent between the "problem" and "OK" events.
* Example:
* CPU high event: message_key = ci_sys_id + "|CPU"
* Memory high event: message_key = ci_sys_id + "|Memory"
* That way, events for CPU won't overwrite memory alerts, and vice versa.

Implementation:

1. Go to your Event Rule.
2. In Message Key :
event.parm1 = event.resource + "|" + event.node; // sample
Or:
event.message_key = event.node + "|" + event.metric_name;
3. Ensure the OK/clear event uses the exact same message key as the problem event.

2.Keep Description but Still Use Message Key
You can still preserve the original event description without it being overwritten by:
* Adding the original description to Additional Information or a custom field in the alert instead of description.
* Updating your Event Rule mapping:
* Map the event description to a custom field like u_original_description.
* Keep the alert description static or based on a template.

3. Use Multiple Alert Rules for Different Event Types
If your events have a clear type (e.g., CPU vs Disk vs Network), you can:
* Create separate Event Rules for each type.
* Assign different message key logic for each rule.

4. Prevent Description Overwrite
In the Alert Management configuration:
* Go to Alert Management → Field Mapping.
* For the description field, change mapping so it updates only if empty — or store incoming event description in work_notes instead of overwriting the main alert description.

Best Practice (from ServiceNow docs & TAC recommendations)
* Always have a message key for external events.
* The message key should:
1. Be consistent between "problem" and "OK" events for the same issue.
2. Be unique across different issues on the same CI.
* Avoid using raw description text as the message key — it often changes between events.

Please appreciate the efforts of community contributors by marking appropriate response as Mark my Answer Helpful or Accept Solution this may help other community users to follow correct solution in future.
 

Thank You
AJ - TechTrek with AJ - ITOM Trainer
LinkedIn:- https://www.linkedin.com/in/ajay-kumar-66a91385/
YouTube:- https://www.youtube.com/@learnitomwithaj
Topmate:- https://topmate.io/aj_techtrekwithaj (Connect for 1-1 Session)
ServiceNow Community MVP 2025