Event Management flooding alerts due to missing correlation rules

SandeepKSingh
Kilo Sage

I have  integrated SolarWinds and AWS CloudWatch with ServiceNow Event Management.
During a major outage, 5,000+ events flood the system in minutes, creating multiple duplicate alerts for the same CI. The Event Management dashboard becomes unusable.


How do we design a solution that reduces alert noise without losing visibility?


Any leads will be Appreciated.. 

2 ACCEPTED SOLUTIONS

Ravi Gaurav
Giga Sage

Hi @SandeepKSingh 

Read the below steps carefully crafted by me as I faced the same issue in the past engagement :

 

 

  • Analyze the Incoming Events:

    • Check the em_event table to see if duplicate events differ only by time or metric value.

    • Identify patterns like:

      • Same node, resource, type

      • Different metric_name or description

  • Implement Alert Correlation Rules:

    • Use Alert Correlation Definitions to merge similar alerts under one parent.

    • Example:

      • Rule: If same node and metric_name within 5 minutes → merge alerts.

      • Result: 20 CPU alerts collapse into one.

  • Create Alert Aggregation (De-duplication):

    • Enable the ā€œDe-duplication Keyā€ in event rules (based on node, metric_name, resource).

    • ServiceNow will automatically update existing alerts instead of creating new ones.

  • Use Threshold and Event Filters:

    • Set thresholds in Connector Definition (e.g., ignore metrics with less than 80% utilization).

    • Apply Event Filters to drop non-critical events (like ā€œOKā€ or ā€œInformationalā€).

  • Apply Correlation Engine Rules (optional advanced step):

    • Use Topological Correlation if you have Service Mapping in place.

    • For example, multiple alerts on the same Application Service can roll up to one parent alert for that business service.

 

View solution in original post

3 REPLIES 3

Ravi Gaurav
Giga Sage

Hi @SandeepKSingh 

Read the below steps carefully crafted by me as I faced the same issue in the past engagement :

 

 

  • Analyze the Incoming Events:

    • Check the em_event table to see if duplicate events differ only by time or metric value.

    • Identify patterns like:

      • Same node, resource, type

      • Different metric_name or description

  • Implement Alert Correlation Rules:

    • Use Alert Correlation Definitions to merge similar alerts under one parent.

    • Example:

      • Rule: If same node and metric_name within 5 minutes → merge alerts.

      • Result: 20 CPU alerts collapse into one.

  • Create Alert Aggregation (De-duplication):

    • Enable the ā€œDe-duplication Keyā€ in event rules (based on node, metric_name, resource).

    • ServiceNow will automatically update existing alerts instead of creating new ones.

  • Use Threshold and Event Filters:

    • Set thresholds in Connector Definition (e.g., ignore metrics with less than 80% utilization).

    • Apply Event Filters to drop non-critical events (like ā€œOKā€ or ā€œInformationalā€).

  • Apply Correlation Engine Rules (optional advanced step):

    • Use Topological Correlation if you have Service Mapping in place.

    • For example, multiple alerts on the same Application Service can roll up to one parent alert for that business service.

 

SandeepKSingh
Kilo Sage

Thanks for the Quick Help !!