Configure Apache Kafka integrations

Configure an integration for streaming log data from Apache Kafka to your ServiceNow instance for processing by Health Log Analytics.

Before you begin

• You must have an installed and configured MID Server with the log ingestion capability enabled.
• If the IP address of the MID Server is exposed by network address translation (NAT), a load balancer or a similar device, it must have a public IP address. In the MID Server properties, add a property named mid.public_ip with the public IP address as the value. For more information, see Create a MID Server property.

• Consider whether to pull the log data to a single MID Server or to a MID Server cluster.

  In MID Server clusters, multiple MID Servers are grouped together for failover protection. The active integration runs on a single MID Server in the cluster. If that MID Server fails, the system moves its tasks to the next available MID Server in the cluster in a configured order.

  If you decide to use a MID Server cluster, ensure the following:

  • The MID Server cluster includes only MID Servers that support basic authentication.
    Note:

    mTLS is not supported for log ingestion.
  • Log ingestion is enabled for each MID Server in the cluster.
    Note:

    If log ingestion is not enabled for the active MID Server, Health Log Analytics enables it automatically.
  • No more than the default maximum of 10 integrations will pull logs to a single MID Server.

    A cluster passes capacity validation if it contains at least one MID Server with fewer than 10 integrations running on it, even when that MID Server is down.

Role required: evt_mgmt_admin

Procedure

1. Navigate to Workspaces > Service Operations Workspace.
2. From the left pane, select the Integrations Launchpad icon ()
3. In the Browse integrations tab, search for Kafka.
4. Select the Kafka integration tile.
   Note:

   If you started the integration process without meeting all the prerequisites listed in the Before you begin section, a message appears. You have the option to cancel the integration and complete the missing requirements, or continue in draft mode and fulfill them later. Bear in mind that you can only activate a configured integration when all the prerequisites are met. Saved integrations that haven’t been activated are available in the Installed integrations tab on the Integrations Launchpad, under Waiting for your action.
5. On the Provide details form, fill in the fields.
   For a description of the fields, see the Provide details table in Apache Kafka integration configuration fields.
6. Select Next.
7. On the Set data retrieval method form, fill in the fields.
   For a description of the fields, see the Set data retrieval method table in Apache Kafka integration configuration fields.
8. Optional: Select Advanced settings and fill in the advanced configuration fields.
   For a description of the fields, see the Advanced settings table in Apache Kafka integration configuration fields.
9. Ensure that the integration is configured correctly by selecting Test and save.
   Health Log Analytics tests the new integration. If the test fails, a message prompts you to correct the faulty field settings. When you have fixed the configuration, select Save.
10. Select Activate to activate the integration.
    Note:

    You can only activate a configured integration when you have fulfilled all the prerequisites listed in the Before you begin section.
    The integration is activated. Its Overview tab is displayed.
11. Optional: If you installed the integration in draft mode, perform these steps to activate it:
    1. Complete the integration prerequisites.
    2. In the Integrations Launchpad Installed integrations tab, under Waiting for your action, locate and select the integration.
    3. On the Set data retrieval method tab, select Activate to activate the integration.

Result

Log data starts streaming from Apache Kafka to your ServiceNow instance. The tile for the integration is available in the Installed integrations tab on the Integrations Launchpad.

Users with the evt_mgmt_user role can use Event Management to monitor the logs and view the alerts that Health Log Analytics generates from them.

What to do next

Review the log data streaming status and sources of the integration on the Overview tab. Leverage the displayed information to refine how HLA reads the log data by adjusting your integration configuration. For more information, see Review log data streaming status and sources of an integration.

Note:

You can go directly from this tab to the Data Input Mapping, Source Type Structures, and Log Sources pages with context from the integration. If the log data is not properly mapped, structured, or sourced, you can go back and adjust the configuration of the integration.

1. Select the View menu icon ().
2. Choose the appropriate menu option.
3. Review the displayed information.
4. Adjust the integration configuration if needed.

• Apache Kafka integration configuration fields
  Description of the fields on the Apache Kafka integration configuration forms for Health Log Analytics.