Elasticsearch data input configuration fields

  • Release version: Washingtondc
  • Updated February 1, 2024
  • 5 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Elasticsearch Data Input Configuration Fields

    This guide provides detailed information on configuring Elasticsearch data inputs for Health Log Analytics within ServiceNow. It outlines the necessary fields in the connector configuration forms, enabling users to set up data ingestion effectively.

    Show full answer Show less

    Key Features

    • Integration Name: A unique identifier for the connector, automatically reflected on the form.
    • Description: An optional field for a brief description to assist in identification.
    • Execute On: Choose between a specific MID Server or a MID Server cluster for log data processing.
    • MID Server/MID Server Cluster: Required fields to define where log data is pulled from, ensuring failover support for enhanced reliability.
    • Service Instance: Required field to bind log data; create a new service instance if none exists.
    • Data Retrieval Method: Configuration options for server URL, authentication, index prefix, and document timestamp field, critical for proper data access and retrieval.
    • Advanced Settings: Options for maximum connections, proxy settings, certificate checks, and cross-cluster search capabilities to optimize data management.

    Key Outcomes

    By accurately configuring the Elasticsearch data input fields, ServiceNow customers can:

    • Ensure reliable log data ingestion from Elasticsearch indices.
    • Utilize MID Server clusters for failover protection and improved performance.
    • Control data access through authentication methods and privileges, enhancing security.
    • Optimize data retrieval through advanced settings, catering to specific operational needs.

    This setup is essential for effective monitoring and analysis within the IT Operations Management framework, ensuring that relevant log data is available for operational insights.

    Description of the fields on the Elasticsearch connector configuration forms for Health Log Analytics.

    Table 1. Provide details
    Field Description
    Integration name Unique name of this connector. This field is required.
    Note:
    When you fill in this field, the generic name displayed on the form adjusts automatically to match the name you entered.
    Description Option to add a brief description of the connector to help identify it.
    Execute on Option to determine whether to use a specific MID Server or a MID Server cluster.
    MID Server

    (Only when the Execute on field is set to Specific MID Server)

    The MID Server to which log data from Elasticsearch indices is pulled. This field is required.
    MID Server Cluster

    (Only when the Execute on field is set to MID Server Cluster)

    The MID Server cluster to which the log data is pulled. This field is required.

    When you select a cluster, the MID Servers in the selected cluster and their status are displayed.

    The connector runs on a single MID Server in the cluster until that MID Server fails. The system then moves all the connector tasks to the next available MID Server in the cluster according to the configured order.

    Note:
    • Health Log Analytics supports only failover MID Server clusters. In these clusters, multiple MID Servers are grouped together for failover protection. When selecting a cluster from the connector form, the MID Server Clusters list displays only failover clusters.
    • The MID Server cluster must include only MID Servers that support basic authentication. mTLS is not supported for log ingestion.
    • Log ingestion must be enabled for each MID Server in the cluster. If log ingestion is not enabled for the active MID Server, Health Log Analytics enables it automatically.
    • If Elasticsearch uses client certificate or CA certificate authentication, all MID Servers in the cluster must have the appropriate certificates.
    • The default maximum number of connectors streaming logs to a single MID Server is 10. A cluster passes capacity validation if it contains at least one MID Server with fewer than 10 connectors running on it, even when that MID Server is down.
    Service instance The service instance to which to bind the log data. This field is required.
    Note:
    If no relevant service instance exists, Create an application service and add CIs to it. Set the status of the new service instance to Operational.
    Table 2. Data retrieval method
    Field Description
    Server URL The URL used to access the cluster. This field is required.
    Authentication method The authentication method used to authenticate the connector to Elasticsearch. Default is none.
    When you select the authentication method, the corresponding credentials fields display on the form.
    Note:
    As an admin, you can create an authentication method by navigating to All > Health Log Analytics > Authentication Methods and selecting New.
    Index prefix Prefix prepended to the names of the Elasticsearch indices from which you want to read data. The connector only reads data from indices that match the configured prefix. For example: network-logs-* matches indices such as network-logs-2024.01.01.

    This setting ensures that HLA only ingests data from the relevant indices.

    This field is required.
    Document timestamp field Timestamp field in documents stored in the read indices. This field is required.
    Term filter JSON map of the terms to filter.
    Note:
    Avoid using the term query for text fields. If the target field is mapped as both text and keyword, reference the keyword by using fieldname.keyword.
    Table 3. Advanced settings
    Field Description
    Max connections per route The maximum number of connections to be opened per node.
    Max scroll slices The number of shards configured for the relevant index in Elasticsearch.

    This number tells Elastic how many parallel queries to execute in each polling request.
    Proxy host Host name of the HTTP proxy through which requests are sent.
    Proxy port Port of the HTTP proxy through which requests are sent.
    Use MID certificate policy check Option to enable the MID certificate policy check.

    Select this option if you want to ship your logs encrypted using SSL TLS. Then navigate to All > MID Server > MID Security Policy and add the MID certificate policy check to the table. For more information, see MID Server certificate check policies.
    Use cross-cluster search Option for searching for data across Elasticsearch clusters.

    When this check box is selected, the Clusters to search field displays.

    Note:
    Your settings in the Use minimal privileges check box and the Delay in reading current timestamp (seconds) field on the Advanced configuration form affect how data is collected across multiple clusters.
    Use minimal privileges Option for reading log data directly from the Elasticsearch indices with the configured prefix.
    • When selected, the connector reads the log data directly from the Elasticsearch indices with the configured prefix. To perform this task, it needs only read privileges.
      Note:
      When this check box is selected and you're using cross-cluster search, data is collected from all the clusters simultaneously.
    • When clear, the connector fetches all indices with the prefix, filters them, and reads the log data from the filtered indices. Performing this task requires additional privileges.
      Note:
      Leaving this check box clear when using cross-cluster search affects how data is collected from the clusters. For more information, see the Enabling and Using Cross-Cluster Search for Elasticsearch Data Inputs in Health Log Analytics [KB1556079] article in the Now Support Knowledge Base.

    For additional information about streaming logs using the Elasticsearch connector, see the Stream logs using Elasticsearch data input - Advanced guide [KB1080162] article in the Now Support Knowledge Base.
    From Starting date for reading the data. Data older than this date is not read. This field is required.
    Note:
    Setting this value to a past date might require the system to read large amounts of data, causing congestion.
    Timestamp field format Format of the timestamp field in the documents.

    If no format is specified, the default Unix epoch time format is used, in milliseconds.

    For example: 1684168407 (May 15, 2023 4:33:27 PM)
    Term filters JSON map of the terms to filter.
    Note:
    Avoid using the term query for text fields. If the target field is mapped as both text and keyword, reference the keyword by using fieldname.keyword.

    For example: {"severity": ["error", "warning"]}
    Sliced-scrolling tie breaker Value used to slice the data. Each slice is scrolled in parallel. Default: _id
    Search-after tie breaker Unique value per document to use as tiebreaker when sorting log entries by timestamp.
    Max documents per query Maximum number of documents fetched in a single query.
    Proxy port Port of the HTTP proxy through which requests are sent.