Configure data inputs (Elasticsearch)

  • Release version: Xanadu
  • Updated August 1, 2024
  • 3 minutes to read

    • Configure a data input for streaming log data from Elasticsearch indices to your ServiceNow instance. Data input configuration is an essential step in setting up the Health Log Analytics (HLA) application.

    Before you begin

    • Ensure that a MID Server is installed and configured with the Log Ingestion capability enabled. For more information, see MID Server system requirements.

      MID Server configuration with Log Ingestion capability enabled.

      Important:
      Health Log Analytics does not support IPv6. To work with the application, configure the MID Server to IPv4.
    • If the MID Server IP address is exposed by network address translation (NAT), a load balancer or a similar device, it must have a public IP address. In the MID Server properties, add a property named mid.public_ip with the public IP address as the value. For more information, see Create a MID Server property.

    Health Log Analytics supports Elasticsearch versions above 7.10.2 and below 8.18.2.

    Note:
    For advanced information about streaming log data from Elasticsearch indices to your instance, see the Stream logs using Elasticsearch data input - Advanced guide [KB1080162] article in the Now Support knowledge base.

    Role required: evt_mgmt_admin

    Procedure

    1. Navigate to All > Health Log Analytics > Data Input > Data Inputs.
    2. On the Data Inputs page, select New.
    3. Choose the Elasticsearch data input.
    4. On the form, fill in the fields.
      For a description of the fields, see Elasticsearch data input configuration fields.
    5. Optional: Select Advanced to set advanced configuration fields.
      For a description of the fields, see Elasticsearch data input configuration fields. For information about configuring the advanced settings later, see Configure advanced settings for Elasticsearch data inputs.
    6. On the Transport tab, fill in the fields.
      For a description of the fields, see Elasticsearch data input configuration fields.
    7. On the Query Settings tab, fill in the fields.
      For a description of the fields, see Elasticsearch data input configuration fields.
    8. Select Save.
      Health Log Analytics adds the data input record to the Data Inputs table.
    9. Ensure that the data input is configured correctly by selecting Test connection.

      Health Log Analytics tries to connect the MID Server to the data repository.

      If the data input is configured to run on a MID Server cluster, the system tries to connect all the MID Servers contained in the cluster to the repository. The cluster passes the test if at least one of its MID Servers gets connected. This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.

      • If the connection was established, the Test connection button is turned off and the Publish button is enabled.
      • If the connection failed, the reason for the failure displays in the Error message field. This field displays only when a streaming error has occurred.

        Resolve the issue, select Save if you modified the configuration, and then select Test connection to test the connection again.

        Note:
        You can only publish the data input configuration when the connection is created successfully.
      Note:
      You can revert to the last published configuration by selecting Revert Changes. This option is available only when you're modifying a configuration that has been published previously.
    10. Select Publish to publish the data input to the MID Server.

    Result

    The data input configuration process is complete. Health Log Analytics adds the data input record to the Data Inputs table and attaches the configuration file to the data input record. The data input starts streaming log data from Elasticsearch indices to your ServiceNow instance.

    For more information about streaming logs using the Elasticsearch data input, see the Stream logs using Elasticsearch data input - Advanced guide [KB1080162] article in the Now Support Knowledge Base.

    Note:
    If the HLA engine is down and data has stopped streaming, a notification appears at the top of the data input configuration page. When this happens, contact ServiceNow support.

    What to do next

    Make sure that the data input is streaming data.

    Note:
    If you experience permissions-related issues with streaming log data from Elasticsearch, refer to the Granting privileges for data streams from Elasticsearch [KB0967366] article in the Now Support Knowledge Base.