Managing topics in Hermes

Release version: Zurich

Updated July 31, 2025

4 minutes to read

Summarize

Summarized using AI

Summary of Managing topics in Hermes

The Hermes Messaging Service enables ServiceNow customers to manage Kafka topics within their instance and the Hermes Kafka cluster. Topic records are stored in the Kafka Topics [syskafkatopic] table and correspond to Kafka topics in the cluster. These records include details such as the topic's name, namespace, and active status. Synchronization between the instance and Hermes is maintained by a scheduled job that creates, updates, or deactivates topic records based on the actual topics in the Kafka cluster.

Show full answer Show less

Managing Topic Records

Topic names in the Kafka Topics table are not unique across namespaces, but their full names are unique. The system uses namespaces—logical groupings identified by prefixes—to organize topics, including a default prefix-less namespace. Topics can be created, updated, or deleted both from within the ServiceNow instance or externally (e.g., via Kafka CLI), but proper naming conventions including application IDs are important for correct namespace assignment.

Creating and Updating Topics

Creation: Topics can be created directly in the instance or externally. When created externally, including the application ID in the topic name ensures proper namespace assignment.
Updating: The number of partitions in a topic can be increased if needed. However, once a topic is assigned a namespace, it cannot be changed; to move a topic to a new namespace, it must be deleted and recreated with the appropriate prefix.

Deleting Topics

Topics can be deleted either from the Kafka Topics table within the instance or externally using CLI commands. Deleting from the instance is preferred as it automatically removes the topic from all Hermes Kafka clusters. Note that topics with active Stream Connect subscriptions cannot be deleted.

Rescanning Topics

Rescanning synchronizes topic records in the instance with the actual topics in the Hermes Kafka cluster, creating any missing records and assigning them to namespaces based on naming prefixes. This process runs automatically every ten minutes but can be triggered manually to immediately reflect external changes.

Practical Considerations for ServiceNow Customers

Kafka topics are specific to an instance; migration between instances requires manual recreation.
Proper topic naming conventions, especially including application IDs, are critical to avoid synchronization issues and ensure correct namespace assignment.
Namespace assignments are permanent for each topic record, so plan namespaces carefully before creation.
Use the instance to manage topics when possible to leverage automatic synchronization and cluster-wide consistency.
Monitor topics using the Hermes topic inspector for visibility into Kafka cluster status.

Manage topic records in your instance and Kafka topics in the Hermes Kafka cluster using the Hermes Messaging Service.

Topic records

Messages are organized and stored in Kafka topics in the Hermes Kafka cluster. Topic records are stored in the Kafka Topics [sys_kafka_topic] table in your instance. This table shows a list of topics records with a reference to each topic's namespace. Each topic record has a field for the topic's name, related namespace, and whether the topic is active.

Values in the Name column of the Kafka Topics [sys_kafka_topic] table aren't inherently unique. For example, topic records might have the same Name value when they belong to different namespaces or exist in different clusters. However, values in the Full Name column are always unique.

A scheduled job runs regularly to synchronize topic records with the topics found in Hermes. This job performs the following functions:

Creates topic records for any discovered topics, assigning them to the appropriate namespace based on the prefix for the topic name.
Synchronizes topics created from external sources, such as the Kafka command-line interface (CLI). For example, when a topic is created from an external source on the near cluster, the job runs and creates the topic on the far cluster as well.
Logs a warning message if a topic can't be matched to any namespace. If a topic can't be matched to any namespace, the topic record isn't created. There's a default, prefix-less namespace called the Default Namespace that matches all topics. The only time a topic doesn't match any namespace is when the Default Namespace has been deleted or modified.
Marks topics as inactive if no Kafka topic was found for them.

You can monitor and view topics in the Hermes Kafka cluster using the topic inspector. For more information, see Monitoring topics in the Hermes Kafka cluster.

Kafka topics are specific to a particular instance. This means you can't migrate a topic record from the Kafka Topics [sys_kafka_topic] table to another instance using an update set. Instead, you must manually create the topic in the target instance.

Creating topics

You can create topics in the Hermes Kafka cluster using either of the following methods:

Create the topic from your instance. This creates a record the Kafka Topics [sys_kafka_topic] table and creates a Kafka topic in the Hermes Kafka cluster. See Create a topic in Hermes.
Create the topic externally. For example, create the topic using the Kafka command-line interface (CLI).
Note:
To avoid syncing issues between the cluster and your instance, include the application ID when creating topics externally. For example, without the application ID, a topic created from the CLI with a name like snc.instancename.fulfillment.topicname is created in the default namespace instead of the fulfillment namespace. To ensure the topic is created in the fulfillment namespace, include the application ID in the name using the format snc.instancename.fulfillment.app_id.topicname. Replace app_id with the application ID of your choice.

If, for some reason, a topic can't be created in the Hermes Kafka cluster, it won't be created in the Kafka Topics [sys_kafka_topic] table either.

Updating topics

If necessary, you can change the number of partitions in a topic. Note that you can only increase the number of partitions in a topic. For more information, see Update a topic in Hermes.

After it's created, the namespace assigned to a topic record is permanent. If you create a namespace record and want to move existing topics to it, you must delete the topics and recreate them in the Hermes Kafka cluster with the new namespace prefix. If the topics weren't created through the instance, new topic records will be created and assigned to the matching namespace record after a rescan.

Deleting topics

You can delete topics from the Hermes Kafka cluster using either of the following methods:

Delete the topic directly from the Kafka Topics [sys_kafka_topic] table in your instance. This method automatically deletes the topic from all the Hermes Kafka clusters in one action. See Delete a topic in Hermes
Delete the topic externally. For example, delete the topic using the Kafka command-line interface (CLI).
Important:
Deleting the topic externally requires that you delete the topic from both Hermes clusters manually by specifying each set of ports (410x and 420x). If possible, delete the topic through the instance instead.

You can't delete a topic if it has subscriptions in Stream Connect.

Rescanning topics

Rescanning creates topic records for any topics in the Hermes Kafka cluster that don't already have corresponding topic records. For example, if you create topics in Hermes using the CLI, you can rescan to create corresponding topic records in the Kafka Topics [sys_kafka_topic] table immediately instead of waiting for the scheduled job to create them.

The rescan occurs automatically every ten minutes.
Created topic records are assigned to existing namespaces based on prefix matching rules.

Sometimes topic records in your instance don't match what's in Hermes. This situation can happen in the following cases:

When the ServiceNow Integration Hub Kafka Consumer (com.glide.hub.kafka_consumer) plugin is enabled after there are already topics in Hermes.
When topic records are deleted accidentally.
When namespace definitions are changed, and you want to assign existing topics to the new namespaces.