Stream Connect quick start guide

Rversion finale: Australia

Mis à jour 12 mars 2026

15 minutes de lecture

This quick start guide gives you an overview of how to set up and use each of the six Stream Connect producers and consumers to exchange data between your ServiceNow instance and a Kafka environment.

Stream Connect has two producers that send data from ServiceNow to a Kafka environment.

Kafka Producer step in Workflow Studio
ProducerV2 API

Stream Connect also has four consumers that retrieve data from a Kafka environment.

Kafka Message trigger in Workflow Studio
Extract Transform Load (ETL) consumer
Transform Map consumer
Script consumer

To link your ServiceNow instance to a Kafka environment, Stream Connect uses the Hermes Messaging Service. The following diagram shows each of the producers and consumers and how they connect to your Kafka environment, shown here as the customer site, through Hermes.

Diagram showing Stream Connect producers and consumers and how they connect to a Kafka environment, described as the "customer site", through the Hermes Messaging Service. — Figure 1. Stream Connect producers and consumers

For more information, see Using Stream Connect for Apache Kafka and the Hermes Messaging Service.

Getting started

To represent the customer site, this guide uses Apache Kafka command-line tools run in your computer. The configuration parameters used with this tool can be used to configure any client connection to the Application Delivery Controller, version 2 (ADCv2) gateway using the Kafka protocol.

All the given commands are tested with OpenSSL version (LibreSSL 2.8.3) and Apache Kafka binary distribution version kafka_2.13-3.4.0.tgz. However, all the commands should work with any version you have in your local environment.

Quick start steps

This quick start guide has five steps. Steps 1 and 2 cover how to set up and test a connection to Hermes. Steps 3 through 5 show you how to configure and use each of the Stream Connect producers and consumers to send and receive data.

The ADCv2 gateway uses mutual authentication to authenticate requests, so the first step is to create a keystore and truststore to use with your favorite Kafka client. The following page shows you how to create the required certificates: Set up a secure connection to the Hermes Messaging Service.
After you set up a secure connection to Hermes, verify the keystore and truststore have been correctly generated by Testing the connection to Hermes through the ADCv2 gateway.
Now you can Use the Kafka Message trigger and Script consumer to consume messages. The first part of this page shows you how to use the Kafka Message trigger in Workflow Studio to retrieve messages from Kafka. The second part shows you how to use the Script consumer to retrieve messages.
And Use the ETL and Transform Map consumers to import data. ETL definitions and transform maps specify the transformation logic to use when pulling data through scheduled imports. You can use the same ETL definitions and transform maps to transform the events received through Kafka.
Remarque :
To use the ETL consumer, you need a robust import set transformer. Likewise, to use the Transform Map consumer, you need a transform map.
Finally, you can Use the Kafka Producer step and the ProducerV2 API to publish messages. This page shows you how to use both the Kafka Producer step in Workflow Studio and the ProducerV2 API to push messages to Kafka.

Set up a secure connection to the Hermes Messaging Service

Secure your Kafka topics by generating a ServiceNow® instance-signed certificate.

Avant de commencer

Setting up the Hermes Messaging Service requires coordination with your network administrator and with your Kafka administrator. Work with your network administrator to obtain required security certificates and open the required ports. Work with your Kafka administrator to ensure that your Kafka environment is configured correctly and that your applications can connect to the Hermes Messaging Service using the standard Kafka protocol.

Make sure the following setup is in place:

The Hermes Messaging Service is activated. See Activating the Hermes Messaging Service.
The Key Management Framework plugin (com.glide.kmf.global) is activated.
The Certificates [sys_kmf_certificate] table contains a ServiceNow instance root CA certificate.
The instance isn't configured with a Custom URL. Custom URLs are not supported with the Instance PKI Certificate Generator.

Role required: hermes_admin, sn_kmf.cryptographic_manager, or admin

For details on assigning KMF roles, see Roles installed with Key Management Framework.

Procédure

Navigate to All > Certificate Generator > Instance PKI Certificate Generator.

Facultatif : Control access to topics by configuring Access Control Lists (ACLs) at the namespace or topic-level.

Option	Description
Apply ACLs to namespaces	Select Configure ACLs. In the Topic ACLs dialog box, select Namespaces. Enter a namespace that you want to configure. Set the permission level by selecting either Read Only or Read/Write. Select Add.
Apply ACLs to defined topics	Select Configure ACLs. In the Topic ACLs dialog box, select Defined topics. Enter an existing topic that you want to configure. Set the permission level by selecting either Read Only or Read/Write. Select Add.

The bearer of the certificate is granted read or read/write access to the topics in the namespace or the existing topic that you selected.

Set up security for the Hermes Messaging Service.
1. Navigate back to the Instance PKI Certificate Generator page.
2. Enter a keystore password in the Certificate Password field.
3. Select Generate.
The system generates an instance-signed certificate in the Certificates [sys_kmf_certificate] table, creates a keystore, and creates a truststore.
If Restricted Caller Access isn't allowed for the IPKI Certificate Generator, a cross scope access error appears. Contact Customer Service and Support for assistance with allowing Restricted Caller Access. To resolve this issue, Customer Service and Support can reference source_scope=76f9d51369115083f4ea77aab1677cc0 in the Restricted Caller Access Privileges [sys_restricted_caller_access] table.
Save a copy of the keystore by selecting Download Keystore.
Save a copy of the truststore by selecting Download Truststore.
Copy the keystore and truststore files to each producer and consumer client that will connect to the Hermes Messaging Service.

Résultats

You can now create a secure connection to the Hermes Messaging Service.

Remarque :

You must use the keystore that you generated using the Instance PKI Certificate Generator to connect to Hermes. Custom-generated keystores that aren't created according to the ServiceNow documentation aren't supported.

Test the connection to Hermes through the ADCv2 gateway

Verify the keystore and truststore have been correctly generated by testing the connection to Hermes through the ADCv2 gateway.

Avant de commencer

Role required: integration_hub_admin

Procédure

Download and install Apache Kafka.
1. Download the Apache Kafka binaries.
2. Unzip the Kafka package by running the following command:
```
tar -zxvf kafka_2.13-3.1.2.tgz
```
Navigate to the Kafka directory by running the following command:
```
cd kafka_2.13-3.1.2
```
Configure a producer.
1. Open the config/producer.properties file.
2. Configure the following SSL properties:
```
security.protocol=SSL

ssl.truststore.password=<truststore password>

ssl.truststore.location=<path to truststore.p12>

ssl.truststore.type=PKCS12

ssl.keystore.password=<keystore password>

ssl.keystore.location=<path to keystore.p12>

ssl.keystore.type=PKCS12

ssl.key.password=<keystore password>
```
  Replace the following placeholder variables:
  - <truststore password> with your truststore password.
  - <path to truststore.p12> with the path to your truststore file.
  - <keystore password> with your keystore password.
  - <path to keystore.p12> with the path to your keystore file.
After following steps 1 through 3, you should have downloaded Apache Kafka and configured the producer.properties correctly. All the commands given in next steps should run within the Apache Kafka home directory.
Copy the config/producer.properties file to config/bootcamp.properties.
This example uses bootcamp as a generic name.
```
cp config/producer.properties config/bootcamp.properties
```

Use the following command to list the topics.

bin/kafka-topics.sh --list --command-config config/bootcamp.properties  --bootstrap-server <instance name>.service-now.com:4000,<instance name>.service-now.com:4001,<instance name>.service-now.com:4002,<instance name>.service-now.com:4003

Be sure to replace <instance name> with the name of your instance.

If there are topics in the instance, they should be listed here.

Use the Kafka Message trigger and Script consumer to consume messages

Retrieve messages from Hermes with the Kafka Message trigger in Workflow Studio and the Script consumer.

Avant de commencer

Role required: integration_hub_admin
The first part of this section shows you how to use the Kafka Message trigger in Workflow Studio to consume messages. For more information about the trigger, see Create a flow with a Kafka Message trigger.
The second part of this section shows you how to use the Script consumer to consume messages. For more information, see Configure a script consumer.

Procédure

Create a topic to consume messages.
1. Use the following command to create a topic in Hermes.
```
bin/kafka-topics.sh --create --command-config config/bootcamp.properties  --bootstrap-server <instance name>.service-now.com:4000,<instance name>.service-now.com:4001,<instance name>.service-now.com:4002, <instance name>.service-now.com:4003 --topic snc.<instance name>.topic1
```
  Be sure to replace <instance name> with the name of your instance.
2. To view the topic, log in to the instance and navigate to All > IntegrationHub > Stream Connect > Topics.
  
  Remarque :
  It takes approximately 10 minutes to create a topic.
Create a flow trigger to consume messages from the topic.
1. Navigate to All > Process Automation > Flow Designer.
2. Select Create new > Flow.
3. On the Flow properties form, enter a name for your flow, then enter Submit.
  This example uses the name Topic1 Consumer.
4. In the trigger section, select Add a trigger, then select Kafka Message as the trigger type.
5. Select the Topic1 topic.
  
  You can use the magnifying glass icon to see the list of available topics.
6. In the Actions section of the flow, navigate to Add an Action, Flow Logic, or Subflow > Flow Logic > For Each.
7. Drag the Messages data pill to the Items field, then select Done.
8. In the Actions section of the flow, navigate to Add an Action, Flow Logic, or Subflow > Action > ServiceNow Core > Log.
9. Drag the Payload data pill to the Message field, then select Done.
10. Save the flow trigger.
11. Go to the Settings [sys_flow_execution_setting] table to enable reporting for this flow.
12. Create a new entry and enable reporting to the flow created above.
13. Navigate back to your flow and select Activate.
14. To view your topic subscription, navigate to All > Integration Hub > Stream Connect > Subscriptions.
Publish messages to the topic using the Kafka command-line tool.
1. Run the following command to publish messages to the topic.
```
bin/kafka-console-producer.sh --topic snc.<instance name>.topic1  --producer.config  config/bootcamp.properties  --bootstrap-server <instance name>.service-now.com:4000,<instance name>.service-now.com:4001,<instance name>.service-now.com:4002,<instance name>.service-now.com:4003
```
  Be sure to replace <instance name> with the name of your instance.
  This command prompts you for messages. Add a few sample messages. This example uses the following messages.
  - Test message1
  - Test message2
  - Test message3
  You should receive the messages in about a minute.
2. Navigate back to the flow and select Executions.
3. Open the flow execution.
4. To view the statistics for this subscription, navigate to All > Integration Hub > Stream Connect > Subscriptions, open the subscription, and go to the statistics view.
Create a Script consumer to consume messages from the topic.
1. Navigate to IntegrationHub > Consumers > Script Consumers.
2. Select New.
3. In the Name field, enter a name for your script consumer.
  This example uses Topic1 Consumer.
4. Set the script to log the message by adding the following logic in the script box.
```
for (var i = 0; i < messages.length; i++) {
      gs.info('Received message ' + JSON.stringify(messages[i].message));
 }
```
5. Save the form.
6. In the Related Links section, select New to add a Kafka stream to consume the messages.
7. On the Kafka Stream form, set the following field values.
  - Name to Topic1 Stream.
  - Topic to the topic created earlier, topic1.
  - Max concurrency to 1.
  - Start Consuming to All the messages stored in the topic. This option enables you to read the messages already published to this topic.
  - Message handling to Automatically optimize.
8. Select the Activate link under Related Links.
  You should be able to see the activated subscription.
9. To view the messages, navigate to All > System logs > System log and filter messages that start with Received message.
10. You should see all the messages after 1 minute.

Use the ETL and Transform Map consumers to import data

Use ETL definitions and transform maps to process Kafka events. ETL definitions and transform maps specify the transformation logic to use when pulling data through scheduled imports. You can use the same ETL definitions and transform maps to transform the events received through Kafka.

Avant de commencer

Role required: integration_hub_admin
The first part of this section shows you how to use the ETL Consumer to consume messages. To use the ETL consumer, you need a robust import set transformer.
The second part of this section shows you how to use a Transform Map consumer to consume messages. To use the Transform Map Consumer, you need a transform map.

Procédure

Create a topic to consume ETL and transform map consumer messages.
1. Use the following command to create a topic in Hermes.
  In this example, the topic name is topic2.
```
bin/kafka-topics.sh --create --command-config config/bootcamp.properties  --bootstrap-server <instance name>.service-now.com:4000,<instance name>.service-now.com:4001,<instance name>.service-now.com:4002, <instance name>.service-now.com:4003 --topic snc.<instance name>.topic2
```
  Be sure to replace <instance name> with the name of your instance.
2. To view the topic, log in to the instance and navigate to All > IntegrationHub > Stream Connect > Topics.
  
  Remarque :
  It takes approximately 10 minutes to create a topic.
Create an ETL Consumer to consume data from the topic.
1. Navigate to All > IntegrationHub > Consumers > ETL Consumer.
2. Select New.
3. On the Kafka Consumer form, enter a name in the Name field.
  This example uses the name Member Import.
4. For the Robust import set transformer filed, select an existing Robust import set transformer.
  For the robust import set transformer, select one that is not using the option to store data in a single column. To be able to see the messages, make sure the robust import set transformer has the Verbose option enabled.
5. Deselect the Data in single column option.
  This option should only be selected for import set tables with data in a single column.
6. Set the Column mapping to Label.
  
  This means the input message's JSON keys are mapped to the column labels of the Import Set table.
7. Save the form.
8. Create a Kafka stream for the consumer.
9. On the Kafka Stream form, set the following field values.
  - Name to Topic2 Stream.
  - Topic to the topic created earlier, topic2.
  - Max concurrency to 1.
  - Start Consuming to Messages received after the first activation.
  - Message handling to Automatically optimize.
10. Select the Activate link under Related Links.
  
  You should be able to see the activated subscription.
Publish messages to topic2.
1. Run the following command to publish messages to topic2.
```
bin/kafka-console-producer.sh --topic snc.<instance name>.topic2  --producer.config  config/bootcamp.properties  --bootstrap-server <instance name>.service-now.com:4000,<instance name>.service-now.com:4001,<instance name>.service-now.com:4002,<instance name>.service-now.com:4003
```
  Be sure to replace <instance name> with the name of your instance.
  
  This command prompts you for messages.
2. Send the following JSON message.
```
{"city":"San Diego","name":"Jhon","id":"SN001","state":"California"}
```
  You can send this message with any values, but the JSON message key should match the Import Set table labels.
3. Navigate back to the ETL Consumer you created.
  
  After about one minute, the data should have been imported through the import set.
4. To view detailed information about the import set, select the import set number.
5. To view detailed information about an import set row, select the import set row number.
Create a Transform Map Consumer to consume data.
1. Navigate to All > IntegrationHub > Consumers > Transform Map Consumer.
2. Select New.
3. On the Transform Map Consumer form, enter a name in the Name field.
4. For the Transform Map field, select an existing transform map.
5. Save the form.
6. Create a Kafka stream for the consumer.
7. On the Kafka Stream form, set the following field values.
  - Name to Transform Map Stream.
  - Topic to the topic created earlier, topic2.
  - Max concurrency to 1.
  - Start Consuming to Messages received after the first activation.
  - Message handling to Automatically optimize.
8. Select the Activate link under Related Links.
  
  You should be able to see the activated subscription.
Publish messages to topic2.
1. Run the following command to publish messages to topic2.
```
bin/kafka-console-producer.sh --topic snc.<instance name>.topic2  --producer.config  config/bootcamp.properties  --bootstrap-server <instance name>.service-now.com:4000,<instance name>.service-now.com:4001,<instance name>.service-now.com:4002,<instance name>.service-now.com:4003
```
  Be sure to replace <instance name> with the name of your instance.
  
  This command prompts you for messages.
2. Send the following JSON message.
```
{"city":"San Diego","name":"Jhon","id":"SN001","state":"California"}
```
  You can send this message with any values, but the JSON message key should match the Import Set table labels.
3. Go to the Transform Map Consumer you created.
  After about one minute, the data should have been imported through the import set.
4. To view detailed information about the import set, select the import set number as in step 3d above.

Use the Kafka Producer step and the ProducerV2 API to publish messages

Push data from ServiceNow to Hermes so that the Kafka consumers can read them.

Avant de commencer

Role required: integration_hub_admin
The first part of this section shows you how to use the Kafka Producer step in Workflow Studio to publish messages to Hermes. For more information about the step, see Kafka Producer step.
The second part of this section shows you how to use the ProducerV2 API to publish messages. For more information, see ProducerV2 API.
To receive all the messages, you need to run two consumers for different datacenters. See Consume test messages from a Hermes topic using the Kafka client for details.

Procédure

Start two consumers to receive messages.

Run the following commands from the command line to receive the messages.

bin/kafka-console-consumer.sh --consumer.config config/bootcamp.properties --topic snc.<instance name>.topic1 --group snc.<instance name>.consumer_group1 --from-beginning --bootstrap-server <instance name>.service-now.com:4100,<instance name>.service-now.com:4101,<instance name>.service-now.com:4102,<instance name>.service-now.com:4103

bin/kafka-console-consumer.sh --consumer.config config/bootcamp.properties --topic snc.<instance name>.topic1 --group snc.<instance name>.consumer_group1 --from-beginning --bootstrap-server <instance name>.service-now.com:4200, <instance name>.service-now.com:4201, <instance name>.service-now.com:4202, <instance name>.service-now.com:4203

Be sure to replace <instance name> with the name of your instance.

Publish messages from the instance using a flow step.
1. Navigate to All > Process Automation > Flow Designer.
2. Create a new action by selecting Create new > Action.
3. In the Action properties form, give your action a name, then select Submit.
  This example uses the name Message Publisher.
4. Add a Kafka Producer step to the action.
  
  To add an action step, select the plus sign between the Inputs and Error Evaluation in the Action Outline.
5. In the step, set the Topic field to topic1.
6. In the Message field, enter a message.
  
  You don't need to change any other fields.
7. Select Save to save the action.
8. Select the Test button to test the action.
  Testing the action publishes the message to the topic.
9. To view the message, go to the command-line consoles created in step 1.
  The message should be present in one of the terminals.
Publish messages from the instance using a script.
1. Run the following script using the scripts background to publish messages to this topic from the instance.
```
var producer = new sn_ih_kafka.ProducerV2();
producer.send('<sys id of the topic record>', gs.generateGUID(), 'Test message from ServiceNow', false, null);
```
  Be sure to replace <sys id of the topic record> with the sys_id of the topic record.
2. To view the message, go to the command-line consoles created in step 1.
  The message should be present in one of the terminals.

Quick start guide

Australia Workflow Data Fabric

Stream Connect quick start guide

Getting started

Quick start steps

Set up a secure connection to the Hermes Messaging Service

Avant de commencer

Procédure

Résultats

Test the connection to Hermes through the ADCv2 gateway

Avant de commencer

Procédure

Use the Kafka Message trigger and Script consumer to consume messages

Avant de commencer

Procédure

Use the ETL and Transform Map consumers to import data

Avant de commencer

Procédure

Use the Kafka Producer step and the ProducerV2 API to publish messages

Avant de commencer

Procédure