--- sourceDocument: Zurich Workflow Data Fabric sourceDocumentLink: https://www.servicenow.com/docs/r/zurich/integrate-applications Release : - zurich ft:locale : - en-US ft:publication_title : - Zurich Workflow Data Fabric ft:clusterId : - crint bundleId : - crint workflow : - Creator --- # Quick start guide # Stream Connect quick start guide {#ariaid-title1} * Release version: Zurich * * Updated July 31, 2025 * * ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 15 minutes to read This quick start guide gives you an overview of how to set up and use each of the six Stream Connect producers and consumers to exchange data between your ServiceNow instance and a Kafka environment. Stream Connect has two producers that send data from ServiceNow to a Kafka environment. * Kafka Producer step in Workflow Studio * ProducerV2 API {#stream-connect-quick-start__ul_znj_n3h_2yb} Stream Connect also has four consumers that retrieve data from a Kafka environment. * Kafka Message trigger in Workflow Studio * Extract Transform Load (ETL) consumer * Transform Map consumer * Script consumer {#stream-connect-quick-start__ul_d4k_p3h_2yb} To link your ServiceNow instance to a Kafka environment, Stream Connect uses the Hermes Messaging Service. The following diagram shows each of the producers and consumers and how they connect to your Kafka environment, shown here as the customer site, through Hermes. Figure 1. Stream Connect producers and consumers For more information, see [Using Stream Connect for Apache Kafka](https://www.servicenow.com/docs/Hu4we4SrxN~OwyIrVHavWQ "Connect your Apache Kafka environment to your ServiceNow instance with Stream Connect for Apache Kafka.") and the [Hermes Messaging Service](https://www.servicenow.com/docs/access?context=hermes-messaging-service&version=zurich&pubname=zurich-servicenow-platform&ft:locale=en-US). ## Getting started {#stream-connect-quick-start__section_vwk_y3w_xxb} To represent the customer site, this guide uses Apache Kafka command-line tools run in your computer. The configuration parameters used with this tool can be used to configure any client connection to the Application Delivery Controller, version 2 (ADCv2) gateway using the Kafka protocol. All the given commands are tested with OpenSSL version (LibreSSL 2.8.3) and Apache Kafka binary distribution version kafka_2.13-3.4.0.tgz. However, all the commands should work with any version you have in your local environment. ## Quick start steps {#stream-connect-quick-start__section_oxh_dsw_xxb} This quick start guide has five steps. Steps 1 and 2 cover how to set up and test a connection to Hermes. Steps 3 through 5 show you how to configure and use each of the Stream Connect producers and consumers to send and receive data. 1. The ADCv2 gateway uses mutual authentication to authenticate requests, so the first step is to create a keystore and truststore to use with your favorite Kafka client. The following page shows you how to create the required certificates: [Set up a secure connection to the Hermes Messaging Service](https://www.servicenow.com/docs/KUWVt9_GXVgq0XU~s~UiAQ#set-up-connection-hermes "Secure your Kafka topics by generating a ServiceNow instance-signed certificate."). 2. After you set up a secure connection to Hermes, verify the keystore and truststore have been correctly generated by [Testing the connection to Hermes through the ADCv2 gateway](https://www.servicenow.com/docs/KUWVt9_GXVgq0XU~s~UiAQ#test-connection-hermes "Verify the keystore and truststore have been correctly generated by testing the connection to Hermes through the ADCv2 gateway."). 3. Now you can [Use the Kafka Message trigger and Script consumer to consume messages](https://www.servicenow.com/docs/KUWVt9_GXVgq0XU~s~UiAQ#use-kafka-trigger-script-consumer "Retrieve messages from Hermes with the Kafka Message trigger in Workflow Studio and the Script consumer."). The first part of this page shows you how to use the Kafka Message trigger in Workflow Studio to retrieve messages from Kafka. The second part shows you how to use the Script consumer to retrieve messages. 4. And [Use the ETL and Transform Map consumers to import data](https://www.servicenow.com/docs/KUWVt9_GXVgq0XU~s~UiAQ#use-etl-transform-map-consumers "Use ETL definitions and transform maps to process Kafka events. ETL definitions and transform maps specify the transformation logic to use when pulling data through scheduled imports. You can use the same ETL definitions and transform maps to transform the events received through Kafka."). ETL definitions and transform maps specify the transformation logic to use when pulling data through scheduled imports. You can use the same ETL definitions and transform maps to transform the events received through Kafka. Note: To use the ETL consumer, you need a [robust import set transformer](https://www.servicenow.com/docs/Yb2bF4mJt4pePCEUg31O6Q "Use robust import set transformers instead of transform maps if you want to extract, transform, and load data to one or more target tables."). Likewise, to use the Transform Map consumer, you need a [transform map](https://www.servicenow.com/docs/0I9KmNgBBj~Ov7fmmZcV4Q "A transform map is a set of field maps that determine the relationships between fields in an import set and fields in an existing ServiceNow table, such as Incident [incident] or User [sys_user]."). 5. Finally, you can [Use the Kafka Producer step and the ProducerV2 API to publish messages](https://www.servicenow.com/docs/KUWVt9_GXVgq0XU~s~UiAQ#publish-messages-instance-hermes "Push data from ServiceNow to Hermes so that the Kafka consumers can read them."). This page shows you how to use both the Kafka Producer step in Workflow Studio and the ProducerV2 API to push messages to Kafka. {#stream-connect-quick-start__ol_z2t_2kw_xxb} ## Set up a secure connection to the Hermes Messaging Service {#ariaid-title2} Secure your Kafka topics by generating a ServiceNow® instance-signed certificate. ### Before you begin Setting up the Hermes Messaging Service requires coordination with your network administrator and with your Kafka administrator. Work with your network administrator to obtain required security certificates and open the required ports. Work with your Kafka administrator to ensure that your Kafka environment is configured correctly and that your applications can connect to the Hermes Messaging Service using the standard Kafka protocol. Make sure the following setup is in place: * The Hermes Messaging Service is activated. See [Activating the Hermes Messaging Service](https://www.servicenow.com/docs/access?context=hermes-messaging-service-activation&version=zurich&pubname=zurich-servicenow-platform&ft:locale=en-US). * The Key Management Framework plugin (com.glide.kmf.global) is activated. * The Certificates \[sys_kmf_certificate\] table contains a ServiceNow instance root CA certificate. * The instance isn't configured with a Custom URL. Custom URLs are not supported with the Instance PKI Certificate Generator. {#set-up-connection-hermes__ul_wp2_lfp_fvb} Role required: hermes_admin, sn_kmf.cryptographic_manager, or admin For details on assigning KMF roles, see [Roles installed with Key Management Framework](https://www.servicenow.com/docs/access?context=kmf-roles&version=zurich&pubname=zurich-platform-security&ft:locale=en-US). ### Procedure 1. Navigate to AllCertificate GeneratorInstance PKI Certificate Generator. 2. **Optional:** Control access to topics by configuring Access Control Lists (ACLs) at the namespace or topic-level. | Option | Description | | Apply ACLs to namespaces | 1. Select Configure ACLs. 2. In the Topic ACLs dialog box, select Namespaces. 3. Enter a namespace that you want to configure. 4. Set the permission level by selecting either Read Only or Read/Write. 5. Select Add. {#set-up-connection-hermes__ol_qm5_lkn_zyb} | | Apply ACLs to defined topics | 1. Select Configure ACLs. 2. In the Topic ACLs dialog box, select Defined topics. 3. Enter an existing topic that you want to configure. 4. Set the permission level by selecting either Read Only or Read/Write. 5. Select Add. {#set-up-connection-hermes__ol_etm_n4t_zyb} | |-|-| {#set-up-connection-hermes__choicetable_ebz_1jn_zyb} The bearer of the certificate is granted read or read/write access to the topics in the namespace or the existing topic that you selected. 3. Set up security for the Hermes Messaging Service. 1. Navigate back to the Instance PKI Certificate Generator page. 2. Enter a keystore password in the Certificate Password field. 3. Select Generate. {#set-up-connection-hermes__substeps_gnt_jzd_wxb} The system generates an instance-signed certificate in the Certificates \[sys_kmf_certificate\] table, creates a keystore, and creates a truststore. If Restricted Caller Access isn't allowed for the IPKI Certificate Generator, a cross scope access error appears. Contact Customer Service and Support for assistance with allowing Restricted Caller Access. To resolve this issue, Customer Service and Support can reference source_scope=76f9d51369115083f4ea77aab1677cc0 in the Restricted Caller Access Privileges \[sys_restricted_caller_access\] table. 4. Save a copy of the keystore by selecting Download Keystore. 5. Save a copy of the truststore by selecting Download Truststore. 6. Copy the keystore and truststore files to each producer and consumer client that will connect to the Hermes Messaging Service. {#set-up-connection-hermes__steps_rxf_543_qcc} ### Result You can now create a secure connection to the Hermes Messaging Service. Note: You must use the keystore that you generated using the Instance PKI Certificate Generator to connect to Hermes. Custom-generated keystores that aren't created according to the ServiceNow documentation aren't supported. ## Test the connection to Hermes through the ADCv2 gateway {#ariaid-title3} Verify the keystore and truststore have been correctly generated by testing the connection to Hermes through the ADCv2 gateway. ### Before you begin * Role required: integration_hub_admin {#test-connection-hermes__ul_nhx_ysn_cyb} ### Procedure 1. Download and install Apache Kafka. 1. Download the [Apache Kafka](https://kafka.apache.org/downloads) binaries. 2. Unzip the Kafka package by running the following command: tar -zxvf kafka_2.13-3.1.2.tgz {#test-connection-hermes__substeps_ffg_hvp_twb} 2. Navigate to the Kafka directory by running the following command: cd kafka_2.13-3.1.2 3. Configure a producer. 1. Open the config/producer.properties file. 2. Configure the following SSL properties: security.protocol=SSL ssl.truststore.password= ssl.truststore.location= ssl.truststore.type=PKCS12 ssl.keystore.password= ssl.keystore.location= ssl.keystore.type=PKCS12 ssl.key.password= Replace the following placeholder variables: * `` with your truststore password. * `` with the path to your truststore file. * `` with your keystore password. * `` with the path to your keystore file. {#test-connection-hermes__ul_tvh_twp_twb} {#test-connection-hermes__substeps_jt5_lwp_twb} After following steps 1 through 3, you should have downloaded Apache Kafka and configured the producer.properties correctly. All the commands given in next steps should run within the Apache Kafka home directory. 4. Copy the config/producer.properties file to config/bootcamp.properties. This example uses `bootcamp` as a generic name. cp config/producer.properties config/bootcamp.properties 5. Use the following command to list the topics. bin/kafka-topics.sh --list --command-config config/bootcamp.properties --bootstrap-server .service-now.com:4000,.service-now.com:4001,.service-now.com:4002,.service-now.com:4003 Be sure to replace `` with the name of your instance. If there are topics in the instance, they should be listed here. ## Use the Kafka Message trigger and Script consumer to consume messages {#ariaid-title4} Retrieve messages from Hermes with the Kafka Message trigger in Workflow Studio and the Script consumer. ### Before you begin * Role required: integration_hub_admin * The first part of this section shows you how to use the Kafka Message trigger in Workflow Studio to consume messages. For more information about the trigger, see [Create a flow with a Kafka Message trigger](https://www.servicenow.com/docs/access?context=create-flow-kafka&version=zurich&pubname=zurich-build-workflows&ft:locale=en-US). For more information about Workflow Studio and configuring flows, see [Workflow Studio](https://www.servicenow.com/docs/access?context=workflow-studio&version=zurich&pubname=zurich-build-workflows&ft:locale=en-US). * The second part of this section shows you how to use the Script consumer to consume messages. For more information, see [Configure a script consumer](https://www.servicenow.com/docs/hOifj0rM23yamGZQCYn7mg "Use a script to import and process data from your Kafka environment."). {#use-kafka-trigger-script-consumer__ul_gbw_h5n_cyb} ### Procedure 1. Create a topic to consume messages. 1. Use the following command to create a topic in Hermes. bin/kafka-topics.sh --create --command-config config/bootcamp.properties --bootstrap-server .service-now.com:4000,.service-now.com:4001,.service-now.com:4002, .service-now.com:4003 --topic snc..topic1 Be sure to replace `` with the name of your instance. 2. To view the topic, log in to the instance and navigate to AllIntegrationHubStream ConnectTopics. Note: It takes approximately 10 minutes to create a topic. {#use-kafka-trigger-script-consumer__substeps_ir5_fzd_yxb} 2. Create a flow trigger to consume messages from the topic. 1. Navigate to AllProcess AutomationFlow Designer. 2. Select Create newFlow. 3. On the Flow properties form, enter a name for your flow, then enter Submit. This example uses the name Topic1 Consumer. 4. In the trigger section, select Add a trigger, then select Kafka Message as the trigger type. 5. Select the Topic1 topic. You can use the magnifying glass icon to see the list of available topics. 6. In the Actions section of the flow, navigate to Add an Action, Flow Logic, or SubflowFlow LogicFor Each. 7. Drag the Messages data pill to the Items field, then select Done. 8. In the Actions section of the flow, navigate to Add an Action, Flow Logic, or SubflowActionServiceNow CoreLog. 9. Drag the Payload data pill to the Message field, then select Done. 10. Save the flow trigger. 11. Go to the Settings \[sys_flow_execution_setting\] table to enable reporting for this flow. 12. Create a new entry and enable reporting to the flow created above. 13. Navigate back to your flow and select Activate. 14. To view your topic subscription, navigate to AllIntegration HubStream ConnectSubscriptions. {#use-kafka-trigger-script-consumer__substeps_vrn_szd_yxb} 3. Publish messages to the topic using the Kafka command-line tool. 1. Run the following command to publish messages to the topic. bin/kafka-console-producer.sh --topic snc..topic1 --producer.config config/bootcamp.properties --bootstrap-server .service-now.com:4000,.service-now.com:4001,.service-now.com:4002,.service-now.com:4003 Be sure to replace `` with the name of your instance. This command prompts you for messages. Add a few sample messages. This example uses the following messages. * Test message1 * Test message2 * Test message3 {#use-kafka-trigger-script-consumer__ul_arc_54m_2yb}You should receive the messages in about a minute. 2. Navigate back to the flow and select Executions. 3. Open the flow execution. 4. To view the statistics for this subscription, navigate to AllIntegration HubStream ConnectSubscriptions, open the subscription, and go to the statistics view. {#use-kafka-trigger-script-consumer__substeps_nrq_gkq_yxb} 4. Create a Script consumer to consume messages from the topic. 1. Navigate to IntegrationHubConsumersScript Consumers. 2. Select New. 3. In the Name field, enter a name for your script consumer. This example uses Topic1 Consumer. 4. Set the script to log the message by adding the following logic in the script box. for (var i = 0; i < messages.length; i++) { gs.info('Received message ' + JSON.stringify(messages[i].message)); } 5. Save the form. 6. In the Related Links section, select New to add a Kafka stream to consume the messages. 7. On the Kafka Stream form, set the following field values. * Name to Topic1 Stream. * Topic to the topic created earlier, topic1. * Max concurrency to 1. * Start Consuming to All the messages stored in the topic. This option enables you to read the messages already published to this topic. * Message handling to Automatically optimize. {#use-kafka-trigger-script-consumer__ul_pbp_rqm_2yb} 8. Select the Activate link under Related Links. You should be able to see the activated subscription. 9. To view the messages, navigate to AllSystem logsSystem log and filter messages that start with Received message. 10. You should see all the messages after 1 minute. {#use-kafka-trigger-script-consumer__substeps_mm5_rkq_yxb} ## Use the ETL and Transform Map consumers to import data {#ariaid-title5} Use ETL definitions and transform maps to process Kafka events. ETL definitions and transform maps specify the transformation logic to use when pulling data through scheduled imports. You can use the same ETL definitions and transform maps to transform the events received through Kafka. ### Before you begin * Role required: integration_hub_admin * The first part of this section shows you how to use the ETL Consumer to consume messages. To use the ETL consumer, you need a [robust import set transformer](https://www.servicenow.com/docs/Yb2bF4mJt4pePCEUg31O6Q "Use robust import set transformers instead of transform maps if you want to extract, transform, and load data to one or more target tables."). * The second part of this section shows you how to use a Transform Map consumer to consume messages. To use the Transform Map Consumer, you need a [transform map](https://www.servicenow.com/docs/0I9KmNgBBj~Ov7fmmZcV4Q "A transform map is a set of field maps that determine the relationships between fields in an import set and fields in an existing ServiceNow table, such as Incident [incident] or User [sys_user]."). {#use-etl-transform-map-consumers__ul_gm5_njx_cyb} ### Procedure 1. Create a topic to consume ETL and transform map consumer messages. 1. Use the following command to create a topic in Hermes. In this example, the topic name is topic2. bin/kafka-topics.sh --create --command-config config/bootcamp.properties --bootstrap-server .service-now.com:4000,.service-now.com:4001,.service-now.com:4002, .service-now.com:4003 --topic snc..topic2 Be sure to replace `` with the name of your instance. 2. To view the topic, log in to the instance and navigate to AllIntegrationHubStream ConnectTopics. Note: It takes approximately 10 minutes to create a topic. {#use-etl-transform-map-consumers__substeps_lv3_ylx_cyb} 2. Create an ETL Consumer to consume data from the topic. 1. Navigate to AllIntegrationHubConsumersETL Consumer. 2. Select New. 3. On the Kafka Consumer form, enter a name in the Name field. This example uses the name Member Import. 4. For the Robust import set transformer filed, select an existing Robust import set transformer. For the robust import set transformer, select one that is not using the option to store data in a single column. To be able to see the messages, make sure the robust import set transformer has the Verbose option enabled. 5. Deselect the Data in single column option. This option should only be selected for import set tables with data in a single column. 6. Set the Column mapping to Label. This means the input message's JSON keys are mapped to the column labels of the Import Set table. 7. Save the form. 8. [Create a Kafka stream](https://www.servicenow.com/docs/fqla8xvcPW2ryjlI5sTBLw "Define a data stream for an Extract Transform Load (ETL), Transform Map, or Script consumer. A Kafka stream defines the stream of data to your consumer.") for the consumer. 9. On the Kafka Stream form, set the following field values. * Name to Topic2 Stream. * Topic to the topic created earlier, topic2. * Max concurrency to 1. * Start Consuming to Messages received after the first activation. * Message handling to Automatically optimize. {#use-etl-transform-map-consumers__ul_pbp_rqm_2yb} 10. Select the Activate link under Related Links. You should be able to see the activated subscription. {#use-etl-transform-map-consumers__substeps_ajg_hkx_cyb} 3. Publish messages to topic2. 1. Run the following command to publish messages to topic2. bin/kafka-console-producer.sh --topic snc..topic2 --producer.config config/bootcamp.properties --bootstrap-server .service-now.com:4000,.service-now.com:4001,.service-now.com:4002,.service-now.com:4003 Be sure to replace `` with the name of your instance. This command prompts you for messages. 2. Send the following JSON message. {"city":"San Diego","name":"Jhon","id":"SN001","state":"California"} You can send this message with any values, but the JSON message key should match the Import Set table labels. 3. Navigate back to the ETL Consumer you created. After about one minute, the data should have been imported through the import set. 4. To view detailed information about the import set, select the import set number. 5. To view detailed information about an import set row, select the import set row number. {#use-etl-transform-map-consumers__substeps_uzy_4kx_cyb} 4. Create a Transform Map Consumer to consume data. 1. Navigate to AllIntegrationHubConsumersTransform Map Consumer. 2. Select New. 3. On the Transform Map Consumer form, enter a name in the Name field. 4. For the Transform Map field, select an existing transform map. 5. Save the form. 6. [Create a Kafka stream](https://www.servicenow.com/docs/fqla8xvcPW2ryjlI5sTBLw "Define a data stream for an Extract Transform Load (ETL), Transform Map, or Script consumer. A Kafka stream defines the stream of data to your consumer.") for the consumer. 7. On the Kafka Stream form, set the following field values. * Name to Transform Map Stream. * Topic to the topic created earlier, topic2. * Max concurrency to 1. * Start Consuming to Messages received after the first activation. * Message handling to Automatically optimize. {#use-etl-transform-map-consumers__ul_u4q_1wm_2yb} 8. Select the Activate link under Related Links. You should be able to see the activated subscription. {#use-etl-transform-map-consumers__substeps_yt5_vkx_cyb} 5. Publish messages to topic2. 1. Run the following command to publish messages to topic2. bin/kafka-console-producer.sh --topic snc..topic2 --producer.config config/bootcamp.properties --bootstrap-server .service-now.com:4000,.service-now.com:4001,.service-now.com:4002,.service-now.com:4003 Be sure to replace `` with the name of your instance. This command prompts you for messages. 2. Send the following JSON message. {"city":"San Diego","name":"Jhon","id":"SN001","state":"California"} You can send this message with any values, but the JSON message key should match the Import Set table labels. 3. Go to the Transform Map Consumer you created. After about one minute, the data should have been imported through the import set. 4. To view detailed information about the import set, select the import set number as in step 3d above. {#use-etl-transform-map-consumers__substeps_t4b_clx_cyb} ## Use the Kafka Producer step and the ProducerV2 API to publish messages {#ariaid-title6} Push data from ServiceNow to Hermes so that the Kafka consumers can read them. ### Before you begin * Role required: integration_hub_admin * The first part of this section shows you how to use the Kafka Producer step in Workflow Studio to publish messages to Hermes. For more information about the step, see [Kafka Producer step](https://www.servicenow.com/docs/access?context=kafka-producer-action-designer&version=zurich&pubname=zurich-build-workflows&ft:locale=en-US). * The second part of this section shows you how to use the ProducerV2 API to publish messages. For more information, see [ProducerV2 API](https://www.servicenow.com/docs/access?context=ProducerV2ScopedAPI&version=zurich&pubname=zurich-api-reference&ft:locale=en-US). * To receive all the messages, you need to run two consumers for different datacenters. See [Consume test messages from a Hermes topic using the Kafka client](https://www.servicenow.com/docs/access?context=consume-messages-hermes&version=zurich&pubname=zurich-servicenow-platform&ft:locale=en-US) for details. {#publish-messages-instance-hermes__ul_g1n_yg4_cyb} ### Procedure 1. Start two consumers to receive messages. 1. Run the following commands from the command line to receive the messages. bin/kafka-console-consumer.sh --consumer.config config/bootcamp.properties --topic snc..topic1 --group snc..consumer_group1 --from-beginning --bootstrap-server .service-now.com:4100,.service-now.com:4101,.service-now.com:4102,.service-now.com:4103 bin/kafka-console-consumer.sh --consumer.config config/bootcamp.properties --topic snc..topic1 --group snc..consumer_group1 --from-beginning --bootstrap-server .service-now.com:4200, .service-now.com:4201, .service-now.com:4202, .service-now.com:4203 Be sure to replace `` with the name of your instance. {#publish-messages-instance-hermes__substeps_ix5_z5b_cyb} 2. Publish messages from the instance using a flow step. 1. Navigate to AllProcess AutomationFlow Designer. 2. Create a new action by selecting Create newAction. 3. In the Action properties form, give your action a name, then select Submit. This example uses the name Message Publisher. 4. Add a Kafka Producer step to the action. To add an action step, select the plus sign between the Inputs and Error Evaluation in the Action Outline. 5. In the step, set the Topic field to topic1. 6. In the Message field, enter a message. You don't need to change any other fields. 7. Select Save to save the action. 8. Select the Test button to test the action. Testing the action publishes the message to the topic. 9. To view the message, go to the command-line consoles created in step 1. The message should be present in one of the terminals. {#publish-messages-instance-hermes__substeps_o3b_rvb_cyb} 3. Publish messages from the instance using a script. 1. Run the following script using the scripts background to publish messages to this topic from the instance. var producer = new sn_ih_kafka.ProducerV2(); producer.send('', gs.generateGUID(), 'Test message from ServiceNow', false, null); Be sure to replace `` with the sys_id of the topic record. 2. To view the message, go to the command-line consoles created in step 1. The message should be present in one of the terminals. {#publish-messages-instance-hermes__substeps_vrb_2wb_cyb} *[\>]: and then