--- sourceDocument: Australia IT Operations Management sourceDocumentLink: https://www.servicenow.com/docs/r/australia/it-operations-management Release : - australia ft:locale : - en-US ft:publication_title : - Australia IT Operations Management ft:clusterId : - itom bundleId : - itom workflow : - Technology --- # Apache Kafka data input configuration fields # Apache Kafka data input configuration fields {#ariaid-title1} Release version: Australia Updated March 12, 2026 ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 4 minutes to read Summarize ![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete. ## Summary of Apache Kafka data input configuration fields This guide explains the configuration fields for setting up Apache Kafka data inputs within ServiceNow, specifically for Health Log Analytics. It enables customers to stream log data from Apache Kafka clusters or specific MID Servers into their ServiceNow instance for monitoring and analysis. Show full answer Show less ## Basic Configuration * **Name and Description:** Required fields to identify and describe the new data input. * **Execute on:** Choose between a specific MID Server or a MID Server cluster to pull log data. Only MID Servers supporting basic authentication (not mTLS) are available. * **MID Server / Cluster:** When selecting a specific MID Server, the default maximum is 10 concurrent data inputs per MID Server (modifiable). For clusters, failover is supported, and log ingestion must be enabled on each MID Server. The system automatically enables log ingestion if disabled. * **Service Instance:** Required binding of the log data to an operational service instance. If none exists, customers must create one and add Configuration Items (CIs). ## Read-Only Status Fields * **Status:** Current state of the data input. * **Transport:** Indicates Apache Kafka is used for log streaming. * **Sources Count:** Number of created log sources. * **Disabled Since \& Last Log Time:** Track when data input stopped and last log received. * **Error Message:** Displays streaming errors if any occurred. ## Query Settings Configure the start date/time for reading data. This is essential to avoid system congestion by limiting the amount of historical data read. Example: "Now -1 week." ## Transport Settings * **Kafka Node Names:** Comma-separated list of Kafka cluster servers in HOST:PORT format; does not need to include all servers. * **Topics:** Required comma-separated list of Kafka topics to subscribe to. * **Kafka Credentials:** Reference to credentials with selectable security protocols (SSL, SASLSSL, SASLPLAINTEXT) for authenticating with Kafka. * **Group Id:** Name of the Kafka consumer group managing the subscription. ## Advanced Configuration * **Timeouts:** Polling timeout (default 500 ms) and node discovery timeout (default 30 ms) to control Kafka communication responsiveness. * **Default Timezone:** Applies if logs lack timezone info; default is GMT. * **Sub Sample Ratios:** Settings to reduce or limit the number of events fetched or received to manage data volume. * **Max Length in Bytes:** Maximum event size allowed (default 32766 bytes). * **Character Encoding:** Encoding for ingested logs; default is UTF-8. * **Drop if Queue is Full:** Option to discard logs if MID Server load is high (default is false). ## Practical Benefits for ServiceNow Customers By configuring these fields correctly, customers can efficiently stream and manage Apache Kafka log data into ServiceNow's Health Log Analytics, ensuring reliable log ingestion, failover protection with MID Server clusters, and appropriate authentication. This setup supports scalable, secure, and high-performance log data integration, enabling better operational insight and incident response. Description of the fields on the Apache Kafka data input configuration form. ## Basic configuration {#hla-data-input-kafka-ref__section_k5x_cx4_pvb} {#hla-data-input-kafka-ref__table_vny_z2c_x4b__entry__2} | Field | Description | |-|-| | Name | Name of the new data input. This field is required. | | Description | Description of the data input. | | Execute on | Option to select whether to use a specific MID Server or a MID Server cluster. This field is required. | | MID | (Only when the Execute on field is set to Specific MID Server) MID Server to which log data from Apache Kafka is pulled. Note: * You can select only MID Servers that support basic authentication. MID Servers that support mTLS are not listed. * The default maximum number of data inputs streaming logs to a single MID Server is 10. You can modify this number in the MID Server properties. * If log ingestion is not enabled for the selected MID Server, Health Log Analytics enables it automatically. {#hla-data-input-kafka-ref__ul_ekr_yz3_nsb}This field is required. | | MID Server Cluster | (Only when Execute on is set to Specific MID Server cluster.) The MID Server cluster to which the log data is pulled. This field is required. The data input runs on a single MID Server in the cluster until that MID Server fails. The system then moves all the data input tasks to the next available MID Server in the cluster according to the configured order. Note: * Health Log Analytics supports only failover MID Server clusters. In these clusters, multiple MID Servers are grouped together for failover protection. When selecting a cluster from the data input or integration form, the MID Server clusters list displays only failover clusters. * Log ingestion must be enabled for each MID Server in the cluster. If log ingestion is not enabled for the active MID Server, Health Log Analytics enables it automatically. * The default maximum number of data inputs or integrations streaming logs to a single MID Server is 10. A cluster passes capacity validation if it contains at least one MID Server with fewer than 10 data inputs or integrations running on it. This is true even when that MID Server is down. {#hla-data-input-kafka-ref__ul_htt_wdq_dwb} For more information about MID Server clusters, see [Configure a MID Server cluster](https://www.servicenow.com/docs/access?context=t_ConfigureAMIDServerCluster&version=australia&pubname=australia-servicenow-platform&ft:locale=en-US). | | Service instance | The service instance to which to bind the log data. Note: If no relevant service instance exists, [Create an service instance](https://www.servicenow.com/docs/access?context=create-it-services&version=australia&pubname=australia-servicenow-platform&ft:locale=en-US) and add CIs to it. Set the status of the new service instance to Operational.This field is required. | [ ] {#hla-data-input-kafka-ref__table_vny_z2c_x4b} The following fields show read-only information: {#hla-data-input-kafka-ref__table_hds_jcb_z4b__entry__2} | Field | Description | |-|-| | Status | Status of the data input. | | Transport | Protocol used to stream the log data. This data input uses Apache Kafka to stream log data to your instance. | | Sources count | The number of log sources this data input has created. | | Disabled since | The time when the data input stopped or failed. | | Last log time | The time when the last log streamed in the data input. | | Error message | The streaming error. This field is populated automatically. It displays only when a streaming error has occurred. | [ ] {#hla-data-input-kafka-ref__table_hds_jcb_z4b} {#hla-data-input-kafka-ref__table_z51_vcb_z4b__entry__3} | Field | Description | Example | |-|-|-| | From | Starting date and time for reading the data. Data older than this date and time is not read. Note: Setting this value to a past date might require the system to read large amounts of data, causing congestion. This field is required. | Now -1 week | [Table 1. Query Settings tab] {#hla-data-input-kafka-ref__table_z51_vcb_z4b} {#hla-data-input-kafka-ref__table_m1j_5g4_3pb__entry__3} | Field | Description | Example | |-|-|-| | Kafka node names | A comma-separated list in the format HOST:PORT,HOST:PORT. The list does not have to include all the Apache Kafka Cluster servers. | 123.4.5.6:9092,123.3.4.5:9093 | | Topics | A comma-separated list of topics to which the data input must subscribe. This field is required. | FirstTopic,SecondTopic,ThirdTopic | | Kafka credentials | Reference to the Apache Kafka credentials. You can display the Kafka SSL credentials form by selecting the Search icon (![]()) and opening a record. The form enables you to choose the security protocol used for authenticating with Apache Kafka from the following options: * SSL - SSL channel. * SASL_SSL - SASL authenticated, SSL channel. * SASL_PLAINTEXT - SASL authenticated, non-encrypted channel. {#hla-data-input-kafka-ref__ul_jvv_z11_3tb}For a description of the fields on the Kafka SSL credentials form, see [Kafka SSL credentials fields](https://www.servicenow.com/docs/JZzAZEOLpNPVFTWf0nmPzg "Description of the fields on the Kafka SSL credentials form."). | None | | Group Id | The name of the Apache Kafka Consumer Group. | logs | [Table 2. Transport tab] {#hla-data-input-kafka-ref__table_m1j_5g4_3pb} ## Advanced configuration {#hla-data-input-kafka-ref__section_u5g_dx4_pvb} {#hla-data-input-kafka-ref__table_afg_2z3_x4b__entry__3} | Field | Description | Default value | |-|-|-| | Timeout | The time, in milliseconds, spent waiting in the poll if data is not available in the topics. | 500 | | Node discovery timeout | The time, in milliseconds, before node discovery times out. | 30 | | Default timezone | The default timezone if the log doesn't include timezone information. | GMT | | Sub sample drop ratio | The number of events to batch together, out of which one will be discarded. This setting is used to reduce the number of fetched events. | -1 | | Sub sample receive ratio | The number of events to batch together, out of which all but one will be discarded. This setting is used to decrease the number of received events. | -1 | | Max length in bytes | The maximum length, in bytes, of events. | 32766 | | Character encoding | The character encoding for this data input. | UTF-8 | | Drop if queue is full | Option to discard logs if there is a load on the MID Server. | False | [Table 3. Advanced configuration form] {#hla-data-input-kafka-ref__table_afg_2z3_x4b} * **[Kafka SSL credentials fields](https://www.servicenow.com/docs/JZzAZEOLpNPVFTWf0nmPzg)** Description of the fields on the Kafka SSL credentials form.