Exchanging data using the Hermes Messaging Service

  • Release version: Washingtondc
  • Updated February 1, 2024
  • 3 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Exchanging Data Using the Hermes Messaging Service

    The Hermes Messaging Service allows ServiceNow customers to produce and consume Kafka messages, facilitating seamless data exchange between ServiceNow instances and Kafka environments. This is essential for integrating applications and enabling real-time data flow.

    Show full answer Show less

    Key Features

    • Message Production and Consumption: Messages can be produced from ServiceNow using various methods, such as Producer steps within flow actions or the Producer API, and consumed in external applications. Conversely, messages from external applications can be consumed in ServiceNow via Kafka flow triggers, RTE consumers, transform map consumers, or script consumers.
    • Log Export Service: This service allows logs to be produced from ServiceNow and consumed in external applications.
    • Kafka Standard Protocol: Enables message exchange with any application that supports the standard Kafka protocol.
    • Failover Capabilities: Hermes utilizes two Kafka clusters for failover, ensuring continuous data production and consumption even if one cluster fails.

    Key Outcomes

    To successfully exchange messages, customers must create a topic in the Hermes Kafka cluster and set up specific processes for production and consumption. Proper configuration of consumer bootstrap addresses and port ranges is crucial for connectivity. Following these guidelines will allow ServiceNow customers to leverage Hermes for efficient data integration, enhancing operational workflows and application connectivity.

    You can produce and consume Kafka messages in your ServiceNow instance using the Hermes Messaging Service.

    There are several methods for exchanging data between your ServiceNow instance and your Kafka environment using the Hermes Messaging Service. In all cases, data is produced from one entity and consumed by another.

    • Using Stream Connect, you can produce messages from your ServiceNow instance using a Producer step from a flow action or the Producer API, and then consume the messages in your external application.
      You can also produce messages from an external application and then consume the messages in your ServiceNow instance via any of the following methods:
      • Kafka flow trigger
      • RTE consumer
      • Transform map consumer
      • Script consumer

      See Stream Connect for Apache Kafka for more information.

    • Using the Log Export Service, you can produce logs from your ServiceNow instance, and then consume the logs in your external application. For details on producing and consuming logs for Log Export Service, see Log Export Service (LES).
    • Using the Kafka standard protocol, you can exchange messages with any application that produces messages. For example, you can produce messages from a Java application using the standard Kafka protocol and then consume them in your ServiceNow instance and vice versa.

    Producing and consuming messages

    To start exchanging messages, create a topic in the Hermes Kafka cluster. After you successfully create the topic, you create a total of three processes to communicate with the Hermes Kafka cluster.

    • One process is required to produce messages to Hermes.
    • Two processes are required to consume messages from Hermes. Two processes are required because Hermes uses a pair of Kafka clusters for failover purposes. If one cluster goes down, data is produced to the other Hermes Kafka cluster.
      Important:
      You must configure two distinct consumer bootstrap addresses, one for each consumer client.

    When using the list command to view a list of topics, a list of current topics are returned from one or both clusters. Topics might be returned from only one cluster, depending on when the last synchronization occurred. Topics created for failover purposes are differentiated by a three-letter cluster identification prefix.

    When accessing the Hermes clusters using the CLI, internal topics appear in addition to any topics you've created. For details, see KB1705399.

    You can't delete a topic from both Kafka clusters using a single command in the command-line interface (CLI). However, you can delete the topic from both clusters by deleting the topic record in your instance. See Delete a topic in Hermes.

    Required port ranges

    Work with your network administrator to ensure that the following port ranges are open before you begin exchanging messages using Hermes:
    • Producer: 4000-4050
    • Consumer1: 4100-4150
    • Consumer2: 4200-4250

    Bootstrap addresses

    Use the following port mappings to connect producers and consumers to the Kafka cluster bootstrap addresses. All Application Delivery Controllers used by Hermes follow this same convention.

    Important:
    The following Hermes bootstrap address examples use ports 4000-4003, 4100-4103, and 4200-4203. However, you must open the following port ranges to ensure connectivity as additional clusters are added to the Hermes Messaging Service.
    • Producer: 4000-4050
    • Consumer1: 4100-4150
    • Consumer2: 4200-4250

    Producer clients use ports ranging from 4000 to 4050. For example:

    Producers = "<instance_name>.service-now.com:4000,<instance_name>.service-now.com:4001,<instance_name>.service-now.com:4002,<instance_name>.service-now.com:4003"

    Because Hermes uses a pair of Kafka clusters, you must configure two consumer clients with separate consumer bootstrap addresses.

    Important:
    You must configure two distinct consumer bootstrap addresses, one for each consumer client.
    • The first consumer client uses ports ranging from 4100 to 4150. For example:
      Consumer A = "<instance_name>.service-now.com:4100,<instance_name>.service-now.com:4101,<instance_name>.service-now.com:4102,<instance_name>.service-now.com:4103"
    • The second consumer client uses ports ranging from 4200 to 4250. For example:
      Consumer B = "<instance_name>.service-now.com:4200,<instance_name>.service-now.com:4201,<instance_name>.service-now.com:4202,<instance_name>.service-now.com:4203"

    When you configure producer and consumer properties for your own Kafka client, use this string pattern.