Using log correlators to detect relationships in log data
Summarize
Summary of Using log correlators to detect relationships in log data
Log correlators in ServiceNow detect relationships between alerts by identifying common keys or values in log data. They help recognize when specific terms or identifiers appear simultaneously across multiple alerts from different services or log sources. This enables more effective correlation and troubleshooting of related issues within your environment.
Show less
Types of log correlators
- Free text correlators: Analyze the unstructured text portion of log messages to find recurring terms that indicate correlated alerts. These are ideal for detecting unique system, application, or service names, or terms not captured as structured log properties. For example, adding a service name like "teatime" as a free text correlator helps link alerts related to that service.
- Log property correlators: Analyze the structured metadata portion of log lines, such as service instance names, interface IDs, or request IDs. These correlators identify correlations based on specific business-context data, like a network device interface ID appearing in multiple warnings from different sources.
Configuring log correlators
- Log source selection: You can specify which log sources a correlator applies to:
- Only new sources created after activation
- All log sources
- A specific log source you define
- Excluding sources: You can exclude specific log sources from a correlator to prevent their log lines from being analyzed, refining correlation accuracy.
- Customization: The base system includes default log correlators, but you can create custom correlators tailored to your environment and business context.
Benefits for ServiceNow customers
By using log correlators, you can automate the detection of relationships between alerts arising from common log data elements. This improves root cause analysis, reduces alert noise, and helps prioritize incidents linked to shared underlying issues. Properly configured correlators allow you to quickly identify correlated events across diverse systems and services, enhancing operational efficiency and responsiveness.
Log correlators are keys or values in log data that detect correlations between alerts. For example, a log correlator could detect when the interface ID of a particular network device occurs simultaneously in multiple warnings across different service instances.
Types of log correlators
Most log lines include a metadata portion plus a message portion. Some log lines, however, include only message text with metadata included in the text.
The two types of log correlators, free text correlators and log property correlators, analyze the different portions of each log to identify relationships between log data from multiple log sources.
- Free text correlators
Free text correlators analyze the text within the log message portion of log lines that are associated with an anomaly. The system uses free text correlators to identify correlations between alerts. You use free text correlators to add a term that you expect to appear within log messages. A good choice is a term that is not structured and would not otherwise be extracted as a log property. For example, “policy-id” or “ thread-id”.
You also typically add free text correlators for the names of systems, applications, and services that are unique to your environment. Because such a value can be referred to by multiple sources, layers, middleware, or databases, the free text correlator can be an effective detector of correlated alerts. For example, if your organization's service is called TeaTime, then you might add "teatime" as a free text correlator. The correlator would identify alerts that are related because they were generated for resources that support the TeaTime service, such as a database lock or a connection failure between TeaTime components.
- Log property correlators
Log property correlators analyze the metadata portion of log lines. For example, the correlator can analyze the name of a service instance, the interface ID of a network device, or the request ID of a web-facing component. A log property correlator could flag a correlation when the interface ID of a network device simultaneously occurs in multiple warnings in different log sources. Log property correlators are specific to the business context of your environment.
Specifying the log sources for a log correlator
- Only new sources: The system applies the log correlator only to log lines from log sources that were created after this log correlator is activated.
- All sources: The system applies the log correlator to log lines from all log sources.
- Specified source: For a log correlator, the system analyzes only log lines from the log source that you specify.
For instructions for specifying the set of log sources, see Add a log correlator to identify relationships in logs.