Alert rules in Health Log Analytics
Summarize
Summary of Alert rules in Health Log Analytics
Health Log Analytics (HLA) automatically detects anomalies in log data using machine learning based on log patterns. However, certain log types—especially those with low frequency or critical conditions—may require custom alert rules to ensure reliable detection and alerting. Custom alert rules allow you to specify metrics, thresholds, and alert properties tailored to your specific monitoring needs.
Show less
Log Pattern Classification and Detection Logic
HLA categorizes incoming logs into three patterns to determine the appropriate anomaly detection method:
- Lively: Logs arrive frequently and consistently (at least once every 20 seconds). Standard anomaly scoring is applied due to sufficient data volume.
- Sparse: Logs arrive infrequently or irregularly (less than once every 60 seconds). Probability distribution analysis is used since standard anomaly scoring may be unreliable. Alerts might not trigger if volume is too low.
- Stopped: Logs have not arrived for a configured period (default 5 minutes). Configuration options allow tuning the detection thresholds for stopped logs.
When to Use Custom Alert Rules
- High-frequency (lively) logs: Custom rules are optional and only needed to capture specific conditions beyond standard anomaly detection.
- Low-frequency (sparse) logs: Custom alert rules are suggested because automatic detection may be unreliable or absent.
- Known critical conditions: Custom rules are required to alert on specific log messages or events that may not trigger automated anomaly detection.
Practical Implications for ServiceNow Customers
By leveraging custom alert rules in HLA, you can enhance alert accuracy and responsiveness for diverse log types across your environment. This enables proactive monitoring of critical services, especially where automated anomaly detection may fall short due to log frequency patterns. Additionally, tuning system properties allows you to customize alert sensitivity for stopped logs according to your operational needs.
Next Steps
- Define, modify, or delete custom Log Analytics alert rules to tailor alerting behavior.
- Configure HLA system properties to adjust detection parameters for stopped logs.
Health Log Analytics (HLA) detects anomalies automatically by learning from your log data. However, some log types require a custom alert rule to generate alerts reliably.
You can use custom alert rules to specify the metric, threshold, and alert properties for generating alerts that HLA might not detect automatically.
Anomaly detection logic by log pattern
HLA classifies incoming logs into three patterns before applying anomaly detection. This classification determines which detection logic is used.
| Pattern | Description |
|---|---|
| Lively | Logs arrive frequently and consistently: at least once in 20 seconds. For example, an application writes hundreds of log entries per hour. The ML Engine has enough volume to build a reliable baseline, so it applies standard anomaly scoring to detect deviations. |
| Sparse | Logs arrive infrequently or irregularly: less than once in 1 minute (60 seconds). For example, a batch job runs every night and writes a small number of log entries. Standard anomaly scoring would produce unreliable results here, so the ML Engine applies probability distribution analysis instead. Sparse logs might not generate alerts if the volume is too low to establish a baseline. |
| Stopped | Logs have not arrived for more than the configured period. Default is 5 minutes (300 seconds). To modify the default value, update the HLA system property detective.resolution.signal_dead.For a log stream to be alerted as stopped or dead it must first be considered alive by running continuously for
a minimum period of time. You can set this time in the HLA system property For information about setting and changing HLA system properties, see Configure global Health Log Analytics system properties. |
When to create custom alert rules
- For high-frequency logs with a lively log pattern, there is no need for a custom rule. However, you can add a rule to generate alerts under specific conditions.
- For low-frequency logs with a sparse log pattern, the system might not generate alerts automatically. If these logs should still generate alerts, define a custom alert rule.
- For known critical conditions that HLA might not flag automatically, define a custom rule. For example, if a specific log message indicates that a critical service has failed, define a rule that generates an alert every time that message appears.
| Scenario | ML detection | Custom rule |
|---|---|---|
| High-frequency logs with a lively log pattern | Likely sufficient | Optional |
| Low-frequency or periodic logs | Unreliable | Suggested |
| Known critical conditions | Insufficient | Required |