Stream Connect Message Replication

  • Release version: Australia
  • Updated March 12, 2026
  • 3 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Stream Connect Message Replication

    Stream Connect Message Replication enables ServiceNow customers to replicate data seamlessly between their Apache Kafka environments and ServiceNow instances. This feature uses a MID Server or MID Server cluster to handle data replication tasks, eliminating the need for additional replication services and simplifying setup through automatic certificate generation.

    Show full answer Show less

    Configuring MID Servers for Replication

    A MID Server equipped with a replicator extension is required to facilitate data replication with Kafka. The MID Server must have network access to Hermes endpoints on specific port ranges (Producer: 4000–4050, Consumer1: 4100–4150, Consumer2: 4200–4250) without relying on proxy settings, as Hermes uses Kafka-native TCP connections.

    Customers can also deploy a MID Server cluster for improved reliability and load balancing. In this configuration, Kafka topic partitions are distributed across all available MID Servers, ensuring continuity if a MID Server fails. Note that only the Load Balance cluster type is supported for this purpose.

    Connection and Credential Management

    Stream Connect Message Replication connects to Kafka using a Connection & Credential alias, which combines Kafka connection details and authentication credentials. Customers must:

    • Create Kafka credentials with necessary authentication data.
    • Configure a Kafka connection to their Kafka environment.
    • Create a Connection & Credential alias associating the above, ensuring it’s accessible from the MID Server.

    No manual setup is needed for connecting to Hermes, as the MID Server automatically manages keystore, truststore, and certificate creation.

    Setting Up Message and Topic Replications

    Replication setup involves two types of records:

    • Message Replication records: Represent individual Kafka clusters and serve as parent records for associated topic replications.
    • Kafka Topic Replication records: Define replication from a single source topic to a single destination topic; multiple destination replication for a single source topic is not supported.

    These records allow customers to organize and control data replication at both cluster and topic levels effectively.

    Monitoring and Logging

    Once replications are active, the system generates metrics records every 60 seconds for each active topic replication. Metrics include the message count and other relevant statistics, accessible via the Message Replication Statistics table or directly from Kafka Topic Replication records.

    Logging of issues primarily occurs in the MID Server log, with an option to enable additional debug logging through a MID Server property for deeper troubleshooting.

    Requirements and Roles

    The feature requires the ServiceNow Stream Connect Installer [com.glide.hub.streamconnect.installer] plugin. Role-based access controls include:

    • messagereplicationadmin: Full create, modify, and delete permissions across message replication and connection-related tables.
    • messagereplicationuser: Read-only access to message replication records.

    Practical Benefits for ServiceNow Customers

    By using Stream Connect Message Replication, customers can efficiently synchronize and manage Kafka data within ServiceNow without additional infrastructure overhead. The integrated MID Server approach simplifies configuration, enhances reliability with clustering options, and provides comprehensive monitoring to maintain data replication health.

    Replicate data between your Apache Kafka environment and ServiceNow.

    With Stream Connect Message Replication, you can configure and manage message replications directly from your ServiceNow instance.

    Stream Connect Message Replication uses a MID Server or MID Server cluster to run the data replications, so you don't need to configure or host additional replication services. It also simplifies the message replication setup by automatically generating the required certificates.

    Enabling a MID Server to replicate data

    Stream Connect Message Replication uses a MID Server with a replicator extension to replicate data to and from your local Kafka. For instructions on how to configure the MID Server, see Configuring MID Servers.

    The MID Server must be able to access Hermes endpoints through the firewall. Work with your network administrator to make sure that the following port ranges are open.
    • Producer: 4000–4050
    • Consumer1: 4100–4150
    • Consumer2: 4200–4250
    Note:
    Proxy settings on the MID Server don't apply to Hermes connectivity. The MID Server proxy configuration applies only to HTTP-based communication. Hermes uses Kafka-native TCP connections, so traffic to the Hermes cluster bypasses proxy settings regardless of how the MID Server is configured.

    Using a MID Server cluster

    You can use a MID Server cluster, instead of a single MID Server, for message replication. With a MID Server cluster, if one of the MID Servers in the cluster fails, the other MID Servers can share the load of the failed MID Server.

    In a MID Server cluster, the topic partitions are distributed across all the available MID Servers in the cluster. If a MID Server becomes unavailable, the partitions are redistributed on the remaining MID Servers. If an additional MID Server becomes available in the cluster, then the partitions are distributed across the MID Servers again.

    For details on setting up a MID Server cluster, see Configure a MID Server cluster.
    Note:
    The MID Server cluster must have the Load Balance cluster type. The Failover cluster type is not supported.

    Configuring connections and credentials for Kafka

    Stream Connect Message Replication uses a Connection & Credential alias to connect to your local Kafka. To configure a Connection & Credential alias:
    1. Create Kafka credentials with the authentication data required for the connection.
    2. Configure a Kafka connection to connect to your Kafka environment.
    3. Create a Connection & Credential alias, to associate the connection information and credential data. The Connection & Credential alias should have a Connection type of Kafka and must be accessible from the MID Server.

    These steps are for configuring a Connection & Credential alias for connecting to Kafka. You don't need to set up connections or credentials for connecting to Hermes, because the MID Server automatically handles the creation of the required keystore, truststore, and certificates.

    Creating message and topic replications

    Message replication requires Message Replication records and Kafka Topic Replication records.

    A Message Replication record represents a single Kafka cluster. For example, if you have two Kafka clusters, you would create two different Message Replication records, one for each cluster. A Message Replication record is the parent record for all the topics being replicated to or from that cluster. Message Replication records are stored in the Message Replications [sys_sc_message_replication] table.

    A Kafka Topic Replication record specifies the replication from a single source topic to a single destination topic. You can't replicate a single source topic to multiple destinations. You can only replicate to each destination once. Kafka Topic Replication records are stored in the Kafka Topic Replications [sys_kafka_topic_replication] table.

    For a step-by-step guide to creating message and topic replication records, see Create message and Kafka topic replications in Stream Connect.

    Viewing message replication statistics

    Once replications are running, the system creates a metrics record for each active topic replication every 60 seconds. Metrics records provide information about topic replications, including the Message count, which shows the number of messages replicated in each collection interval.

    You can view metrics records on the Message Replication Statistics [sys_sc_channel_replication_metric] table. You can also view metrics records for a particular topic by checking the Message Replication Statistics on its Kafka Topic Replication record.

    For a list of message replication metrics and their descriptions, see Viewing Stream Connect Message Replication statistics.

    Required plugin

    Stream Connect Message Replication requires the ServiceNow Stream Connect Installer [com.glide.hub.stream_connect.installer] plugin.

    Roles

    The message_replication_admin role can create, modify, and delete records in all the message replication tables, including the connection and credential tables, and the message and topic replication tables.

    The message_replication_user role can view records in the message replication tables.

    Logging

    Most issues are logged in the MID Server log. Additional debug logging can be enabled by setting the glide.stream_connect.message_replication.debug MID Server property to true.