---
sourceDocument: Zurich Workflow Data Fabric
sourceDocumentLink: https://www.servicenow.com/docs/r/zurich/integrate-applications

 Release :

    - zurich

ft:locale :

    - en-US

ft:publication_title :

    - Zurich Workflow Data Fabric

ft:clusterId :

    - crint

bundleId :

    - crint

workflow :

    - Creator


---

# Using Stream Connect for Apache Kafka

# Using Stream Connect for Apache Kafka {#ariaid-title1}

* Release version: Zurich
* 
* Updated July 31, 2025
* 
* ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 7 minutes to read

Summarize  
![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI  
This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.  

## Summary of Using Stream Connect for Apache Kafka

Stream Connect for Apache Kafka enables ServiceNow customers to integrate their Kafka environments with their ServiceNow instances.
Apache Kafka is a distributed event-streaming platform, and this integration allows the streaming of data between ServiceNow and external Kafka-based systems with high volume and low latency.
Stream Connect requires subscriptions to Workflow Data Fabric and Stream Connect itself.
Show full answer Show less  
This integration leverages the Hermes Messaging Service to efficiently manage data flows between Kafka and ServiceNow, supporting the production and consumption of Kafka events through low-code workflows and APIs.

## Key Features

* **Publishing and Consuming Kafka Events:** Use Workflow Studio's Kafka Producer step and Kafka Message trigger to publish and consume Kafka messages in low-code flows, with automatic Kafka stream and subscription creation.
* **Consumer Types:** Support for multiple consumer types including Extract Transform Load (ETL), Transform Map, and Script consumers for flexible data processing and import using existing ServiceNow configurations or custom scripts.
* **Message Formats:** Support for Apache Avro format with schema import and management to reduce payload size and simplify integration.
* **Message Replication:** Configure message replication between Kafka and ServiceNow using a MID Server without additional service hosting, with automatic certificate management.
* **Monitoring and Alerting:** Detailed performance metrics, subscription statistics, and alerting mechanisms monitor producers, consumers, and message streams to ensure operational reliability.
* **Domain Separation and Namespaces:** Use topic namespaces to control domain access to Kafka topics in domain-separated ServiceNow instances, enhancing security and governance.
* **Compression Support:** Configure producer message compression (NONE, GZIP, LZ4) to optimize data transmission.
* **Guided Setup:** A step-by-step guided configuration interface simplifies initial setup of Stream Connect components.
* **Stream Connect Dashboard:** Centralized management for producers, consumers, topics, replicators, and viewing integration statistics.

## Practical Application

ServiceNow customers can seamlessly integrate Kafka event streams with their workflows to automate data ingestion, real-time event processing, and data replication tasks. By using the pre-built flow triggers and producer steps, customers can build scalable, low-latency event-driven integrations without extensive coding. Advanced users can leverage script consumers for custom message handling and schema management to support Avro message formats.

Monitoring and alerting features help maintain integration health, while domain separation and namespaces ensure secure multi-domain access control for Kafka topics. The use of the Hermes Messaging Service and MID Server simplifies architecture and operational overhead.

## Expected Outcomes

* High-throughput, low-latency streaming of Kafka events to and from ServiceNow.
* Efficient data transformation and import using existing ServiceNow transform maps and ETL configurations.
* Reduced integration complexity through low-code Workflow Studio components and guided setup.
* Improved operational visibility with detailed statistics and alert notifications for producers and consumers.
* Secure and organized Kafka topic management via namespaces supporting domain separation.
* Reliable message delivery with retry mechanisms for undelivered and unprocessed messages.  
Connect your Apache Kafka environment to your ServiceNow instance with Stream Connect for Apache Kafka.

Apache Kafka is a distributed event-streaming platform that provides a unified way to exchange data across multiple systems. Stream Connect for Apache Kafka links your Kafka environment to your ServiceNow instance, enabling you to stream data between your instance and your external systems.  
Note:  
Stream Connect requires an Workflow Data Fabric subscription and a Stream Connect subscription. For more information, see <https://www.servicenow.com/now-platform/workflow-data-fabric.html>.

## Benefits {#stream-connect-apache-kafka__section_xps_grb_kvb}

* Publish and process Kafka events at scale. Publish events to your Kafka environment
  from your ServiceNow instance and consume Kafka events from your
  external systems at a high volume with low latency.

* Build flows that produce and consume Kafka events. Stream Connect is integrated with Workflow Studio, providing a low-code way to publish and process Kafka messages.
* Import data from your Kafka environment and process that data using your existing Robust Transform Engine (RTE) or transform map configurations.
* Configure a consumer that uses your own scripts to process data from a Kafka topic.
* Monitor your consumers' performance with detailed reporting of statistics and performance metrics.
{#stream-connect-apache-kafka__ul_tvj_dyj_kvb}

## Terminology {#stream-connect-apache-kafka__section_o53_4pt_bvb}

Stream Connect uses the following terms.

Producers

:   A producer publishes events to a Kafka environment. Stream Connect has two producers.

    * Kafka Producer step in Workflow Studio
    * ProducerV2 API
    {#stream-connect-apache-kafka__ul_tnd_l5t_bvb}

Consumers

:   A consumer reads and processes events from a Kafka environment. Stream Connect has several consumers.

    * Kafka Message trigger in Workflow Studio
    * Extract Transform Load (ETL) Consumer
    * Transform Map Consumer
    * Script Consumer
    {#stream-connect-apache-kafka__ul_cbq_1qt_bvb}

Topics and topic namespaces

:   Events are organized and stored in topics. A topic stores events of the same type. Topics are partitioned. Events have a key. Events with the same key are stored in the same partition.

    Topics link to a topic namespace. You can use namespaces to organize topics in logical ways. For example, you can group topics together based on which Kafka cluster they come from. You can also use namespaces to
    configure which domains can access which topics on a domain-separated instance. For more information, see [Managing namespaces and topics in Hermes](https://www.servicenow.com/docs/access?context=managing-namespaces-topics-hermes&version=zurich&pubname=zurich-servicenow-platform&ft:locale=en-US).

Subscriptions

:   A subscription is a record associated with a consumer. It stores configuration information about the consumer, such as the name of the Kafka topic to consume messages from and the number of partitions the topic has. The
    subscription record is created when a Kafka stream is activated.

    Each subscription record has several metrics that enable you to view the performance of the consumer reading from the topic. For more information, see [Viewing Kafka subscriptions and statistics](https://www.servicenow.com/docs/wxCbe1dDBpTrzn58LMUOEg "View detailed information about a Kafka consumer and its performance, including the number of records added to the topic, the number of records processed, and the number of records remaining to be processed.").

Partition groups

:   A partition group is a set of topic partitions. For example, if a topic has six partitions, they can be divided into three partition groups, with two partitions in each group.

Kafka consumer job

:   A job that regularly checks Hermes for any new events in a topic. The job picks a free partition group and retrieves its subscription. The subscription gives the topic name, and the job checks the partitions for
    messages for that topic.

Kafka streams

:   A Kafka stream is a record that defines the data stream for a consumer. If you're using the Kafka Message trigger in Workflow Studio, the Kafka stream is automatically created for you. If you're using a different consumer, you'll need to create one manually.

To link your Kafka environment to your ServiceNow instance, Stream Connect uses the Hermes Messaging Service. The Hermes Messaging Service enables your instance to produce and consume large volumes of Kafka events. It manages the flow of data between your Kafka environment and your instance. For more information, see [Hermes Messaging Service](https://www.servicenow.com/docs/access?context=hermes-messaging-service&version=zurich&pubname=zurich-servicenow-platform&ft:locale=en-US).

The following diagram shows some of the key components of Stream Connect.  
Figure 1. Stream Connect overview

## Using Guided Setup for Stream Connect {#stream-connect-apache-kafka__section_k4h_kjh_nhc}

Guided Setup provides a sequence of tasks that can help you configure Stream Connect. To open Guided Setup, navigate to AllIntegrationHubStream ConnectGuided Setup.

For more information about using the guided setup interface, see [Using guided setup](https://www.servicenow.com/docs/access?context=guided-setup&version=zurich&pubname=zurich-platform-user-interface&ft:locale=en-US).

## Stream Connect and Workflow Studio {#stream-connect-apache-kafka__section_qsm_mfn_nvb}

Build flows that produce and consume Kafka events with Stream Connect and Workflow Studio. Stream Connect has a flow trigger for consuming Kafka events and an action step for producing them.

Use the [Kafka Message trigger](https://www.servicenow.com/docs/access?context=create-flow-kafka&version=zurich&pubname=zurich-build-workflows&ft:locale=en-US) to create flows that process Kafka events. You can build a flow that
consumes data from Kafka and inserts it into a table, or uses spokes to communicate the data
to third-party environments.

The trigger is enabled when the flow is activated. After it's activated, the trigger starts
the flow whenever there's a message in the specified Kafka topic. When you use the Kafka
Message trigger, you don't need to create a Kafka stream or subscription record. The system
automatically creates both when the flow is activated. Messages are read from the topic as
long as the flow is active.

Use the [Kafka Producer step](https://www.servicenow.com/docs/access?context=kafka-producer-action-designer&version=zurich&pubname=zurich-build-workflows&ft:locale=en-US) to create actions that publish events to a
topic in your Kafka environment. For example, you can use the step to create a message about
an update on an incident in ServiceNow, then push the message to a
topic in your Kafka environment.

## Stream Connect alerting {#stream-connect-apache-kafka__section_emf_32l_tdc}

Receive alerts and alert notifications for Stream Connect integrations. Stream Connect uses both active and scheduled monitoring to detect events across multiple components. If an issue is detected, the system creates an alert, logs a message to the Stream Connect Log, and sends out an alert notification. For details, see [Stream Connect alerting](https://www.servicenow.com/docs/~eO9n8Wc_bXF6mt~uqOnug "Receive alerts and alert notifications for Stream Connect integrations.").

## Support for messages in an Avro format {#stream-connect-apache-kafka__section_i55_n4s_3bc}

Import and create schemas to send and receive messages in an Apache Avro format. Using an Avro format can reduce the size of the payload and simplify your integration to your local Kafka instance.

You can import Avro schemas directly from the Confluent Registry, or you can create your own schemas using a JSON file or a JSON-formatted string. The schemas are stored in ServiceNow and enable your producers and consumers to convert plain-text messages to an Avro format and back. For details, see [Schema management in Stream Connect](https://www.servicenow.com/docs/0eqaxlrSMNkzMyYNWf8ldA "Import and create schemas to send and receive messages in an Apache Avro format. Using an Avro format can reduce the size of the payload and simplify your integration to your local Kafka instance.").

## ETL, Transform Map, and Script Consumers {#stream-connect-apache-kafka__section_nkk_1wt_nvb}

Import data from your Kafka environment using your existing RTE or transform map
configurations. The Extract Transform Load (ETL) and Transform Map consumers simplify your
data imports by providing an efficient way to take a payload from a Kafka message, transform
the data, and insert or update a record in a table. You can switch from a scheduled data
import to one using Stream Connect and process the data with the same configurations.

You can also use the Script Consumer to process data from your Kafka environment. The
Script consumer is for more advanced use cases, such as when the data in the message isn't
structured, or it requires data lookups using code.

When you [Configure an Extract Transform Load (ETL) consumer](https://www.servicenow.com/docs/4637IXnvGVKQNZ2mICTV6g "Import and process data from your Kafka environment using your existing Robust Transform Engine (RTE) configurations."), [Configure a Transform Map consumer](https://www.servicenow.com/docs/s52DyV~ypSkDo4e8mSXL1g "Import and process data from your Kafka environment using your existing transform map configurations."), or [Configure a script consumer](https://www.servicenow.com/docs/hOifj0rM23yamGZQCYn7mg "Use a script to import and process data from your Kafka environment."), you also need to [Create a Kafka stream](https://www.servicenow.com/docs/fqla8xvcPW2ryjlI5sTBLw "Define a data stream for an Extract Transform Load (ETL), Transform Map, or Script consumer. A Kafka stream defines the stream of data to your consumer.").

## ProducerV2 API {#stream-connect-apache-kafka__section_gh4_tkq_tvb}

Publish events to a Kafka topic with the [ProducerV2 API](https://www.servicenow.com/docs/access?context=ProducerV2ScopedAPI&version=zurich&pubname=zurich-api-reference&ft:locale=en-US).

## Stream Connect Message Replication {#stream-connect-apache-kafka__section_vqp_crh_zzb}

You can replicate data between your Kafka environment and ServiceNow with Stream Connect Message Replication.

Stream Connect Message Replication enables you to configure and manage message replications directly from your ServiceNow instance. It uses a MID Server or MID Server cluster to run the data replications, so you don't need to configure or host additional replication services. It also simplifies the message replication setup by automatically generating the required
certificates.

For more information, see [Stream Connect Message Replication](https://www.servicenow.com/docs/KJIqdjPmdM_nD6MFozaeNw "Replicate data between your Apache Kafka environment and ServiceNow.").

## Unprocessed and undelivered messages {#stream-connect-apache-kafka__section_ovd_xwr_rvb}

If a message can't be delivered, it's stored in the Kafka Undelivered Messages
\[sys_kafka_undelivered_messages\] table. A scheduled job, Kafka Producer Retry, regularly
reads this table and tries to redeliver any messages.

If a batch of messages can't be processed because it has timed out, it's stored in the
Kafka Unprocessed Messages \[sys_kafka_unprocessed_messages\] table. The time-out for a
message batch can be set with the com.glide.kafka_consumer.timeout
property. The default value is 60 seconds. This table is a rotated table, so it cleans
records automatically.

## Producer compression formats {#stream-connect-apache-kafka__section_d51_czr_1fc}

Specify a compression format for Stream Connect producers with the com.glide.kafka_producer.compression_type system property. It supports the following values:

* NONE
* GZIP
* LZ4
{#stream-connect-apache-kafka__ul_bl4_3zr_1fc}  
This property is not in the System Properties \[sys_properties\] table by default, so it must be [added manually](https://www.servicenow.com/docs/access?context=t_AddAPropertyUsingSysPropsList&version=zurich&pubname=zurich-platform-administration&ft:locale=en-US). This property sets the compression format for all Stream Connect producers.  
Note:  
To consume a compressed message, it must have been compressed using GZIP or LZ4.

## Domain separation {#stream-connect-apache-kafka__section_zmt_2wt_nvb}

Use Stream Connect topic namespaces to configure which domains can access a Kafka topic on a domain-separated instance. Group topics into ServiceNow namespaces, then link the namespaces to specific domains. For more information, see [Domain separation and Stream Connect](https://www.servicenow.com/docs/mM_SbFW0yZdPT~tdblpUgg "Domain separation is supported for Stream Connect. Domain separation enables you to separate data, processes, and administrative tasks into logical groupings called domains. You can control several aspects of this separation, including which users can see and access data.").

## Architecture diagram {#stream-connect-apache-kafka__section_zfg_knd_4gc}

The following diagram shows key components of Stream Connect, how they relate to ServiceNow and third-party applications, and how they connect to your Kafka environment through Hermes.  
Figure 2. Stream Connect architecture diagram

## Plugin {#stream-connect-apache-kafka__section_t1y_3hr_ccc}

Stream Connect requires the ServiceNow
Stream Connect Installer \[com.glide.hub.stream_connect.installer\] plugin. This plugin enables the licensed components for working with message-based streaming data in Stream Connect.
* **[Stream Connect Message Replication](https://www.servicenow.com/docs/KJIqdjPmdM_nD6MFozaeNw)**   
  Replicate data between your Apache Kafka environment and ServiceNow.
* **[Using the Stream Connect Dashboard](https://www.servicenow.com/docs/Lr1j3AiLsYyNJdS1zhI85A)**   
  View detailed statistics for your Stream Connect integrations. Manage producers and consumers, view data usage, and create or edit topics and replicators with the ServiceNow® Stream Connect dashboard.
* **[Stream Connect alerting](https://www.servicenow.com/docs/~eO9n8Wc_bXF6mt~uqOnug)**   
  Receive alerts and alert notifications for Stream Connect integrations.
* **[Schema management in Stream Connect](https://www.servicenow.com/docs/0eqaxlrSMNkzMyYNWf8ldA)**   
  Import and create schemas to send and receive messages in an Apache Avro format. Using an Avro format can reduce the size of the payload and simplify your integration to your local Kafka instance.
* **[Configure an Extract Transform Load (ETL) consumer](https://www.servicenow.com/docs/4637IXnvGVKQNZ2mICTV6g)**   
  Import and process data from your Kafka environment using your existing Robust Transform Engine (RTE) configurations.
* **[Configure a Transform Map consumer](https://www.servicenow.com/docs/s52DyV~ypSkDo4e8mSXL1g)**   
  Import and process data from your Kafka environment using your existing transform map configurations.
* **[Configure a script consumer](https://www.servicenow.com/docs/hOifj0rM23yamGZQCYn7mg)**   
  Use a script to import and process data from your Kafka environment.
* **[Create a Kafka stream](https://www.servicenow.com/docs/fqla8xvcPW2ryjlI5sTBLw)**   
  Define a data stream for an Extract Transform Load (ETL), Transform Map, or Script consumer. A Kafka stream defines the stream of data to your consumer.
* **[Viewing Kafka subscriptions and statistics](https://www.servicenow.com/docs/wxCbe1dDBpTrzn58LMUOEg)**   
  View detailed information about a Kafka consumer and its performance, including the number of records added to the topic, the number of records processed, and the number of records remaining to be processed.
* **[Viewing producer statistics](https://www.servicenow.com/docs/LyIfjVD_iyn2KhyH9JrLiQ)**   
  View detailed information about a Stream Connect producer and its performance, including the producer type and ID, and the total number of bytes and messages produced to a topic.

*[\>]: and then