Stream Connect quick start guide

  • Release version: Yokohama
  • Updated January 30, 2025
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Stream Connect quick start guide

    This guide provides a concise introduction to setting up and using the six Stream Connect producers and consumers that enable data exchange between your ServiceNow instance and a Kafka environment. Stream Connect integrates with Kafka through the Hermes Messaging Service, facilitating seamless communication between ServiceNow and Kafka via producers and consumers.

    Show full answer Show less

    Key Features

    • Producers: Two options to send data from ServiceNow to Kafka:
      • Kafka Producer step in Workflow Studio
      • ProducerV2 API
    • Consumers: Four options to retrieve data from Kafka into ServiceNow:
      • Kafka Message trigger in Workflow Studio
      • Extract Transform Load (ETL) consumer
      • Transform Map consumer
      • Script consumer
    • Secure connectivity: Uses mutual authentication via the ADCv2 gateway, requiring keystore and truststore certificates for Kafka client connections.
    • Transformation support: ETL definitions and transform maps enable data transformation during import, applicable both for scheduled imports and Kafka event processing.

    Getting Started Steps

    1. Set up secure connection to Hermes Messaging Service: Create required certificates (keystore and truststore) to enable mutual authentication through the ADCv2 gateway.
    2. Test connection: Verify the certificates and connectivity to Hermes using Kafka command-line tools compatible with your local Kafka client environment.
    3. Consume messages: Configure Kafka Message trigger or Script consumer in ServiceNow to receive messages from Kafka.
    4. Import data with transformation: Use ETL and Transform Map consumers to import and transform Kafka data into ServiceNow tables, ensuring you have corresponding import set transformers or transform maps configured.
    5. Publish messages: Use Kafka Producer step in Workflow Studio or ProducerV2 API to send messages from ServiceNow to Kafka.

    Practical Benefits for ServiceNow Customers

    By following this guide, ServiceNow customers can rapidly establish secure, reliable, and bi-directional data flows between ServiceNow and Kafka environments. This enables real-time data integration, automated data processing, and enhanced event-driven workflows leveraging Kafka's messaging capabilities combined with ServiceNow's platform features.

    This quick start guide gives you an overview of how to set up and use each of the six Stream Connect producers and consumers to exchange data between your ServiceNow instance and a Kafka environment.

    Stream Connect has two producers that send data from ServiceNow to a Kafka environment.
    • Kafka Producer step in Workflow Studio
    • ProducerV2 API
    Stream Connect also has four consumers that retrieve data from a Kafka environment.
    • Kafka Message trigger in Workflow Studio
    • Extract Transform Load (ETL) consumer
    • Transform Map consumer
    • Script consumer

    To link your ServiceNow instance to a Kafka environment, Stream Connect uses the Hermes Messaging Service. The following diagram shows each of the producers and consumers and how they connect to your Kafka environment, shown here as the customer site, through Hermes.

    Figure 1. Stream Connect producers and consumers
    Diagram showing Stream Connect producers and consumers and how they connect to a Kafka environment, described as the "customer site", through the Hermes Messaging Service.

    For more information, see Using Stream Connect for Apache Kafka and the Hermes Messaging Service.

    Getting started

    To represent the customer site, this guide uses Apache Kafka command-line tools run in your computer. The configuration parameters used with this tool can be used to configure any client connection to the Application Delivery Controller, version 2 (ADCv2) gateway using the Kafka protocol.

    All the given commands are tested with OpenSSL version (LibreSSL 2.8.3) and Apache Kafka binary distribution version kafka_2.13-3.4.0.tgz. However, all the commands should work with any version you have in your local environment.

    Quick start steps

    This quick start guide has five steps. Steps 1 and 2 cover how to set up and test a connection to Hermes. Steps 3 through 5 show you how to configure and use each of the Stream Connect producers and consumers to send and receive data.

    1. The ADCv2 gateway uses mutual authentication to authenticate requests, so the first step is to create a keystore and truststore to use with your favorite Kafka client. The following page shows you how to create the required certificates: Set up a secure connection to the Hermes Messaging Service.
    2. After you set up a secure connection to Hermes, verify the keystore and truststore have been correctly generated by Testing the connection to Hermes through the ADCv2 gateway.
    3. Now you can Use the Kafka Message trigger and Script consumer to consume messages. The first part of this page shows you how to use the Kafka Message trigger in Workflow Studio to retrieve messages from Kafka. The second part shows you how to use the Script consumer to retrieve messages.
    4. And Use the ETL and Transform Map consumers to import data. ETL definitions and transform maps specify the transformation logic to use when pulling data through scheduled imports. You can use the same ETL definitions and transform maps to transform the events received through Kafka.
      Note:
      To use the ETL consumer, you need a robust import set transformer. Likewise, to use the Transform Map consumer, you need a transform map.
    5. Finally, you can Use the Kafka Producer step and the ProducerV2 API to publish messages. This page shows you how to use both the Kafka Producer step in Workflow Studio and the ProducerV2 API to push messages to Kafka.