Create a Kafka stream

Define a data stream for an Extract Transform Load (ETL), Transform Map, or Script consumer. A Kafka stream defines the stream of data to your consumer.

Before you begin

• Role required: integration_hub_admin
• Kafka streams are part of Stream Connect which requires a Stream Connect subscription. For more information, see https://www.servicenow.com/now-platform/workflow-data-fabric.html.

About this task

You must create a Kafka stream to use an ETL, Transform Map, or Script consumer. After you configure a consumer, and create and activate a stream, you can start receiving messages from your Kafka environment.

Procedure

1. Open the record for your ETL, Transform Map, or Script consumer.
   To find a consumer record, navigate to All > IntegrationHub > Consumers and select ETL Consumer, Transform Map Consumer, or Script Consumer.
2. Locate the Kafka Streams section and select New.
3. In the form, fill in the fields.

   Table 1. Kafka Stream

   Fields	Description
   Name	Name of the Kafka stream.
   Topic	Name of the topic to monitor for messages.
   Max concurrency	Maximum number of parallel processors to use to consume messages, and the number of partition groups to create. Default value: 1
   Relative weight	For each cycle, the maximum amount of time allocated to the consumer for processing messages, relative to other consumers. For example, a consumer with a relative weight that's twice as high as another consumer's gets twice as much time. Consumers that have the same relative weight get the same amount of time. The minimum value is 5. The maximum value is 2000. The default value is equal to the max concurrency multiplied by the value of the glide.ih.kafka.consumer.max_seconds_per_partition_group property. The glide.ih.kafka.consumer.max_seconds_per_partition_group property specifies the maximum time, in seconds, allocated to each partition group. Its default value is 10. The system uses the relative weight to calculate the partition group timeout for the subscription. The partition group timeout specifies the maximum time, in milliseconds, allocated to each partition group in a specific subscription. This field appears only when Advanced is selected.
   Start consuming	Option to specify where to begin consuming messages from the topic. Select one of the following. Messages received after the first activation: Only consumes messages the topic received after the stream was activated. All the messages stored in the topic: Consumes all the messages in the topic, even messages received before the stream was activated.
   Message handling	Option to specify the number of messages processed per run or let the system decide the number of messages. Select one of the following. Automatically optimize: The system determines the number of messages to process per run. Manually override: You specify the number of messages to process per run. This field appears only when Advanced is selected.
   Maximum messages	Maximum number of messages to process per run. This field appears only when Manually override is selected from Message handling.
   Run as	Option to run the consumer with the credentials of a specified user.
   Consumer	The name of the consumer to stream data to. This field is automatically set to the name of the consumer from which you started adding the stream.
   Active	Option to make the stream active.
   Advanced	Option to view the advanced fields Relative weight and Message handling.

4. Under Related Links, select Activate.
   After the stream is activated, the system automatically creates a Kafka subscription for the associated consumer.
5. Select Submit.

Result

When you activate the stream, the consumer begins looking for messages to process in the specified topic. You can view detailed information about the consumer and its performance by visiting its subscription page. For details, see Viewing Kafka subscriptions and statistics.