Create message and Kafka topic replications in Stream Connect

Set up message and topic replications to replicate data between your Apache Kafka environment and ServiceNow.

Before you begin

• Role required: message_replication_admin
• This feature requires a subscription. For more information, see Using Stream Connect for Apache Kafka.
• Verify the Hermes Messaging Service and Instance PKI Certificate Generator are working.
  • Check the status of and connection to the Hermes Kafka cluster.
  • Check the Key Management Framework Health to verify that all three items in the Instance PKI section are operational.

• Configure and start a MID Server or MID Server cluster. For more information, see Configuring MID Servers and Configure a MID Server cluster.

  The MID Server must be able to access Hermes endpoints through the firewall. Work with your network administrator to make sure that the following port ranges are open.

  • Producer: 4000–4050
  • Consumer1: 4100–4150
  • Consumer2: 4200–4250

  Note:

  Proxy settings on the MID Server don't apply to Hermes connectivity. The MID Server proxy configuration applies only to HTTP-based communication. Hermes uses Kafka-native TCP connections, so traffic to the Hermes cluster bypasses proxy settings regardless of how the MID Server is configured.

  If you're using a MID Server cluster, it must have a cluster type of Load Balance. The Failover cluster type is not supported.

• Stream Connect Message Replication uses a Connection & Credential alias to connect to your local Kafka. Create a Connection & Credential alias with a Connection type of Kafka. A Connection & Credential alias requires a Kafka Connection and a Kafka SSL Credential.
• Stream Connect Message Replication requires the ServiceNow Stream Connect Installer [com.glide.hub.stream_connect.installer] plugin.

About this task

This page shows you how to create a Message Replication record and an associated Kafka Topic Replication record.

A Message Replication record represents a single Kafka cluster. For example, if you have two Kafka clusters, you would create two different message replication records, one for each cluster. A Message Replication record is the parent record for all the topics being replicated to or from that cluster. Message Replication records are stored in the Message Replications [sys_sc_message_replication] table.

A Kafka Topic Replication record specifies the replication from a single source topic to a single destination topic. You can't replicate a single source topic to multiple destinations. You can only replicate to each destination once. Kafka Topic Replication records are stored in the Kafka Topic Replications [sys_kafka_topic_replication] table.

Procedure

1. Navigate to All > IntegrationHub > Stream Connect > Message Replications and select New.
2. On the Message Replication form, fill in the fields.

   Table 1. Message Replication form

   Field	Description
   Name	Name of the message replication.
   Messaging service connection	Connection & Credential alias specifying which messaging service to connect to.
   Application	Application scope for the message replication. This field is automatically set.
   Active	Option to activate the message replication.

3. Select Save.
   After you save the form, the Message Replication Statuses and Kafka Topic Replications tabs appear at the bottom of the page.
4. On the Kafka Topic Replications tab, select New.
5. On the Kafka Topic Replication form, fill in the fields.

   Table 2. Kafka Topic Replication form

   Field	Description
   Name	Name of the topic replication. The name is generated automatically based on the values in other fields. To generate the Name, enter the values for the Direction, ServiceNow topic, and Topic name fields, then select Save. When the Direction is To ServiceNow, the Name has the following format: topic name [replication name] -> ServiceNow topic name [SN]. When the Direction is From ServiceNow, the Name has the following format: ServiceNow topic name [SN] -> topic name [replication name].
   Direction	Direction of the topic replication. To ServiceNow replicates a topic from your local Kafka environment to ServiceNow. From ServiceNow replicates a topic from ServiceNow to your local Kafka environment.
   ServiceNow topic	Reference to the Kafka Topics [sys_kafka_topic] table.
   Replication	Message replication that manages this topic replication.
   Topic name	Name of the topic in your local Kafka environment.
   Active	Option to activate the topic replication.
   Track in current update set	When selected, the topic replication is added to the current update set, so it's simpler to promote it from sub-production to production instances. You may need to configure the form to add this field.

6. Select Submit.
7. Follow steps 1–6 to create message and topic replications for all your Kafka clusters and topics.

Result

After you've created all your message and topic replication records, messages should be replicating. You can check the Message Replication Statuses [sys_sc_message_replication_status] tab on the Message Replication form to view the status, MID Server, and error messages for each message replication.

You can also check the Channel Replication Statuses [sys_sc_channel_replication_status] tab on the Kafka Topic Replication record to view status details for each topic replication, including the MID Server, Error message, and Error Since values.

Each Kafka Topic Replication record also has a related list named Message Replication Statistics. Once replications are running, this list displays metrics records, which are created every 60 seconds for each topic replication. Each metrics record has information about the topic replication, including the Message count, which shows the number of messages replicated in each collection interval.

What to do next

If you're replicating messages from your local Kafka into ServiceNow, you can configure a Script, ETL, or Transform Map consumer to process the messages or set up a Kafka Message trigger to start a flow.

If you're replicating messages from ServiceNow to your local Kafka, you can use a Kafka Producer step in Workflow Studio or use the ProducerV2 API to publish messages to the Hermes topic and your local Kafka.