Log data auto-mapping and mapping

  • Release version: Xanadu
  • Updated August 1, 2024
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Log data auto-mapping and mapping

    Health Log Analytics in ServiceNow automatically assigns incoming log lines to tags such asservice instance,component, andsource type. This auto-mapping helps organize and analyze log data efficiently by identifying key metadata fields like source, path, and namespace. Customers can customize this process by defining JavaScript functions to manually modify automatic mapping results, enabling fine-tuned control over how logs are categorized.

    Show full answer Show less

    Key Features

    • Auto-mapping: Automatically extracts tags from incoming logs based on fields like source, path, and others, associating them with service instances and components for logical grouping.
    • Manual mapping customization: Define JavaScript functions to override or enhance auto-mapping, allowing precise control over tag assignments and data organization.
    • Test mode: Safely test mapping scripts without affecting production data or Elasticsearch storage by using temporary indices and previewing results before publishing changes.
    • Source limits and alerts: Configurable system properties set warning and critical limits on the number of sources created per data input to prevent data overload and trigger notifications if limits are approached or exceeded.
    • Binding to CMDB: Log data can be linked to Configuration Items (CIs) via service instances, enabling correlation for improved root cause analysis and streamlined incident management.
    • Header properties detection: Separates transport header data from the actual log message to focus analysis on relevant log content.
    • Data extraction control: Ability to stop extraction of redundant or non-descriptive data and ensure extraction of specific important terms to improve log clarity and tagging accuracy.

    Practical Benefits for ServiceNow Customers

    This functionality enables customers to:

    • Automatically organize large volumes of log data into meaningful categories aligned with their service topology.
    • Customize mapping logic to meet unique data structures and operational needs.
    • Test changes safely to optimize log data classification without impacting production environments.
    • Monitor and control the growth of log data sources to avoid system overload.
    • Integrate log data with CMDB records, enhancing the accuracy and efficiency of root cause analysis and operational insights.
    • Improve the relevance and quality of log data through selective extraction of key information.

    By leveraging these capabilities, customers can optimize their log data ingestion and analysis workflows, improve alerting and anomaly detection, and support effective IT operations management within the ServiceNow platform.

    By default, the Health Log Analytics AI engine tries to auto-map every incoming log line to the correct tags. You can change automatic mapping results manually by defining a JavaScript function.

    Auto-mapping incoming log lines

    Health Log Analytics auto-mapping assigns log samples and metadata to three tags: service instance, component, and source type. The service instance assignment is based on the service instance specified in the data input setup. The remaining tags are assigned automatically.

    For example, in the following example log line, Health Log Analytics uses the "source" field to find the component and source type.

    {"beat":{"version":"6.8","name":"abc3.prd.acme.com","hostname":"abc3.prd.acme.com"},"@timestamp":"2020-08-27T10:12:24.792Z","prospector":{"type":"log"},"message":"**** User null is requesting the following page http://www.acme.com PROPS:{"subcategory1":"home pages","httpStatus":"200","loginLevel":"Anonymous","userAgent":"Mozilla5.0", ("pageUrl":\"http://www.acme.com","host":"abc3.prd.acme.com","@version":"1","source":"/opt/oracle/weblogic/abc/online_store3/logs/online_store3.out","offset":3951550786}

    In the example, Health Log Analytics extracts the string "online_store". It analyzes the following fields if they exist in the log line: source, path, channel, namespace_name, name, pod_name, source_name, and aws_lambda_name. When data is sent over Syslog, it also analyzes the syslog tag.

    Stop extraction of unneeded data
    If an extracted string is not descriptive enough or contains redundant text or information, you can stop extracting such expendable data. For more information, see Stop extraction of unneeded log data.
    Ensuring extraction of specific data
    You can make sure that Health Log Analytics extracts specific desired terms. For more information, see Ensure extraction of specific log data.

    Mapping data input sources

    You can change automatic mapping results manually by defining a JavaScript function. Data input mapping enables you to organize your log data by service instance and by availability zone. A single service instance can include multiple components, and a component can receive logs from many different source types. An service instance-component pair, however, is unique. Source types are based on a specific log structure and format. Service instances and components are defined more broadly and are therefore used mainly for logical mapping.

    Activating Test mode avoids blowing up Elasticsearch storage with sample data that is used only for perfecting the log data mapping. When the data input is in Test mode, Health Log Analytics doesn’t create the source types, sources, or any other objects it creates in the standard flow. It saves the streamed data in dedicated temporary Elasticsearch indices that appear as components in the Log viewer. When you publish the script and exit Test mode, these temporary indices are deleted to minimize storage space consumption.

    When you're defining a JavaScript function, select Test to view the outcome of the script as it is currently specified. This functionality enables you to preview the created source types and sources. You can then refine the script until it achieves the desired outcome. For example, it can be useful to compare the test outcome of several versions of the JavaScript function.
    Note:
    By default, the test processes 100 log data samples. You can customize this number in the system properties. For more information, see Configure global Health Log Analytics system properties.
    During the data input setup, the system might create an excessive total number of data input sources. For example, this can be due to a faulty mapping script. You can configure limits for the number of sources created per data input in the system properties:
    System property Description Default
    log_source.sources_warning_limit The warning limit for the number of sources created per data input. 500
    log_source.sources_critical_limit The critical limit for the number of sources created per data input. 600
    The number of log sources that a specific data input has created displays in the Sources count field for that data input. When the total number of sources created during the data input setup reaches the warning limit, the system sends a warning notification by email. It also displays a message on the Data Input mapping, Log sources, and Data input pages. The notification and the message include the total number of sources created so far and the three data inputs that contributed the most sources to this total. If no action is taken, the system continues to create sources until the total number reaches the critical limit. When this happens, the data input setup and streaming from all data inputs stops automatically. You can't start data inputs again manually until the issue has been resolved. You can resolve this state by following the instructions in the How to handle too many sources in data inputs [KB0963067] article in the Now Support Knowledge Base.

    Binding log data

    Binding log data to Configuration Items (CIs) in the Configuration Management Database (CMDB) enables you to search the CMDB for endpoints that match a log. When you configure a data input, you bind log entries to a service instance that is bound to a CI in the CMDB. Binding log entries, service instances, and CIs enables the Health Log Analytics AI engine to correlate them for use in root cause analysis (RCA). For more information, see Configure data inputs (Rsyslog, Filebeat, or Winlogbeat) or Configure data inputs (Elasticsearch).