Set up data inputs in Health Log Analytics manually

Set up your Health Log Analytics data inputs for Health Log Analytics manually. Data input configuration is an essential step in setting up the Health Log Analytics application.

Before you begin

Note:

Consider using the Health Log Analytics data input guided setup, which ensures that you have the minimum required setup for the data input process. For more information, see Set up data inputs in Health Log Analytics using guided setup.

• Verify that a MID Server is installed and configured with the Log Ingestion capability enabled. For more information, see MID Server system requirements.

  

  Important:

  Health Log Analytics does not support IPv6. To work with the application, configure the MID Server to IPv4.
• Unless the MID Server and external clients are on the same network, the MID Server must have a public IP address. This is required when its IP is exposed through network address translation (NAT), a load balancer, or a similar device. The public IP address enables external clients, such as Filebeat agents located outside its network, to reach the MID Server. Private IP addresses are not routable over the internet. Without a public IP, external clients cannot connect to the MID Server even if they are configured with its address. In the MID Server properties, add a property named mid.public_ip with the public IP address as the value. For more information, see Create a MID Server property. If the MID Server and external clients are on the same network, connections can be made using the private IP address.
• For shipping your logs encrypted using SSL TLS, see the Streaming Data With Rsyslog & Filebeat Using SSL [KB0866319] article in the Now Support Knowledge Base.

• For MID Server proxy requirements, see MID Server proxy preconditions for streaming logs to Health Log Analytics.

Role required: evt_mgmt_admin. For the ServiceNow System Logs data input: admin.

Procedure

1. Setup a data input manually by performing the relevant procedure described in the product documentation.

   Table 1. Data Inputs

   Data Input	Description
   Agent Client Collector	The data input streams log messages to your ServiceNow instance using the ServiceNow Agent Client Collector. This data input is supported for use with the Agent Client Collector Log Analytics application, available from the ServiceNow Store.
   Amazon CloudWatch	The data input streams log data from Amazon CloudWatch to your ServiceNow instance.
   Amazon S3 Bucket	The data input streams log data from Amazon S3 (Simple Storage Service) buckets to your ServiceNow instance.
   Apache Kafka	The data input streams log data from Apache Kafka to your ServiceNow instance.
   Cribl	The data input to enables Health Log Analytics to process Cribl log messages streaming into your ServiceNow instance.
   Edge Delta	The data input enables Health Log Analytics to process Edge Delta log messages streaming into your ServiceNow instance.
   Elasticsearch	The data input pulls log data from Elasticsearch indexes into your instance.
   GCP PubSub	The data input receives log messages that are published to a Google Cloud Pub/Sub topic and streams them to your ServiceNow instance.
   Microsoft Azure Event Hubs	The data input streams events from Microsoft Azure Event Hubs to your ServiceNow instance.
   Microsoft Azure Log Analytics	The data input streams log data from Microsoft Azure Log Analytics to your ServiceNow instance.
   MID Server	The data input collects MID Server log files and streams them to your instance.
   REST API	The data input streams log data to your ServiceNow instance in JSON format.
   Rsyslog or Beats	The data input streams log data into your instance using Rsyslog or Beats.
   ServiceNow System Logs Retriever	The data input streams log data from the ServiceNow System Log table to the Health Log Analytics AI engine. Note: Only a single ServiceNow System Logs Retriever data input can exist in the system, and only users with the admin role can create and configure it. This data input doesn't run on a MID Server.
   Splunk	The data input streams log data into your instance using Splunk.
   Splunk Polling	The data input periodically pulls log data from Splunk by using a query.
   TCP	The data input sends raw log messages to your instance directly over a TCP/SSL socket.
   UDP	The data input streams raw log messages to your ServiceNow instance directly over a UDP socket.
   Vector Agent	The data input enables Health Log Analytics to process log messages that are streaming into your ServiceNow instance via a Vector Agent.

   Note:

   Testing the connection to the MID Server at the end of the data input setup procedure ensures that your data input is configured correctly. You can only publish a data input configuration when the connection between the MID Server and the data repository has been established.
2. Identify and address streaming issues to confirm that the data input is streaming log data to the MID Server from all sources.
   For more information, see Identify and resolve a log streaming issue in Health Log Analytics.
3. Optional: Edit raw log data before Health Log Analytics maps and structures it.
   For more information, see Edit your raw log data before processing.
4. Determine how Health Log Analytics handles raw log data that is streaming into your instance.
   By default, every incoming log line is auto-mapped to the correct tag. If properties aren't discovered automatically, map the data input sources manually by defining a JavaScript function. For more information, see Map the raw data.
5. Optional: Tweak the source type structure to make sure that Health Log Analytics extracts and classifies all properties correctly.
   For more information, see Refine the source type structure in Health Log Analytics.
6. Optional: Perform additional data input setup tasks.
   For more information, see Additional data input setup tasks in Health Log Analytics.

• Configuring data inputs for Health Log Analytics manually
  Configure the data input process manually in Health Log Analytics. Data input configuration is an essential step in setting up the Health Log Analytics application.
• Identify and resolve a log streaming issue in Health Log Analytics
  Find and address log streaming issues to verify that your data inputs are streaming log data to your instance properly.
• Additional data input setup tasks in Health Log Analytics
  After performing the initial data input setup and configuration in Health Log Analytics, continue with the remaining data input setup tasks.
• Advanced data input configuration in Health Log Analytics
  When you have configured a data input successfully, Health Log Analytics adds a record to the Data Inputs table and attaches the configuration file to it. You can configure advanced settings for your data input. Configuring advanced settings is optional.
• Stop or restart a data input in Health Log Analytics
  You can stop using a data input for streaming log messages to your ServiceNow instance. Restart the data input if you want it to resume streaming data.

Related reference

• Supported data inputs for Health Log Analytics