Stream Connect with a Kafka proxy (Kroxylicious): end-to-end setup guide

[Adam Celli]

TL;DR

• The point of this guide: ServiceNow Stream Connect keeps working in both Direct Kafka mode (Scenario A) and Hermes mode (Scenario B) when a Kafka proxy sits in between. Kroxylicious is used as an example proxy.
• Scenario A (main guide): a ServiceNow instance reaches a self-managed Kafka broker through one public TLS endpoint using Stream Connect Direct Kafka (no MID Server). Tested end to end on the instance: connection test, topic sync, topic alias, script publish, Script Consumer, and Stream Producer (CDC).
• Scenario B (short, at the end): a plain Kafka client with no ServiceNow certificates produces to ServiceNow-hosted Hermes through a second proxy that holds the mTLS identity. Produce was tested and confirmed in the Hermes Topic Inspector; reading the records on the instance was not tested. The proxy gateway in this scenario is plaintext and unauthenticated: keep it on a private network.
• Lab versions: Kroxylicious 0.23.0 (MultiTenant filter is a preview feature, one tenant tested), Confluent Platform cp-kafka 8.0.0 (single node), one developer instance.

Introduction

Stream Connect Direct Kafka lets a ServiceNow instance talk to a Kafka cluster without a MID Server. The instance connects to the bootstrap address you give it, then to every broker the cluster advertises. That is easy when the cluster is a managed service with public endpoints. It is harder when the cluster is self-managed or sits in a private network.

Putting a Kafka-aware proxy such as Kroxylicious in front of the cluster gives you:

• One public TLS endpoint. The instance sees a single host name and port (krox.example.com:9192) with a publicly trusted certificate. The brokers stay on a private network.
• Resource-name separation. A MultiTenant filter prefixes topic names per virtual cluster. This guide tested one tenant (tenanta) and verified topic prefixing only.
• Hidden internal broker addresses. The proxy rewrites the broker addresses it hands back to clients, so clients see only proxy host names and ports, not the broker's internal ones. Node ids and the number of brokers are still visible in the metadata.

Kroxylicious is used here as an example; the results that matter are on the ServiceNow side. Scenario A builds the setup in a lab and shows Direct Kafka working through it. It covers traffic in both directions: the instance publishing to Kafka (script publish and table change data capture with a Stream Producer) and the instance consuming from Kafka (Script Consumer with a Kafka Stream). Scenario B at the end reverses the idea for ServiceNow-hosted Hermes. The guide documents only configuration that was tested.

Tested on. Kroxylicious 0.23.0 (the newest published container image; the docs site shows 0.25.0 configuration syntax with clusterDefinitions, which differs from 0.23.0 and is not used here), Confluent Platform cp-kafka 8.0.0 as a single-node KRaft broker, and a personal developer instance. The Kroxylicious multi-tenant filter is marked "(Preview)" in the 0.23.0 multi-tenancy guide, which says it is "currently in incubation" and that the project "would not recommend using it in a production environment". It "isolates operations on resources, it does not isolate user identities across tenants": user authentication and ACLs are shared across all tenants. Treat this as a reference pattern to validate in your own environment, not a supported production design.

Architecture (Scenario A)

ServiceNow instance Direct Kafka Cluster Connection and Credential Alias Kafka credentials: SASL_SSL, SCRAM-SHA-512

→ TLS 9192

Public DNS krox.example.com krox-1.example.com

→

Kroxylicious 0.23.0 SNI gateway :9192, TLS, wildcard cert MultiTenant filter, prefix tenanta.

→ SASL

Kafka broker SASL_PLAINTEXT :9092 SCRAM-SHA-512

The instance connects to krox.example.com:9192. Kroxylicious terminates TLS with a publicly trusted certificate and reads the SNI host name to decide which broker the connection is for. SCRAM authentication passes through to the broker. The MultiTenant filter adds the tenanta. prefix to topic names on the way in and strips it on the way out, so the instance reads and writes sn-orders while the broker holds tenanta.sn-orders. The connection is initiated by the instance in every case; the same connection carries publishes (script or Stream Producer CDC) and consumer fetches.

The hop from the proxy to the broker uses SASL_PLAINTEXT, which is not encrypted. Keep it on a trusted network.

Glossary

Term	Meaning here
Direct Kafka	The Stream Connect mode in which the instance connects to Kafka itself, without a MID Server.
Kroxylicious	An open-source, protocol-aware Kafka proxy. It understands the Kafka wire protocol, so it can rewrite broker addresses and filter requests.
Virtual cluster	A proxy-side cluster definition that maps to one target Kafka cluster. Here, tenanta.
SNI gateway	A virtual cluster gateway (sniHostIdentifiesNode) that uses the TLS Server Name Indication host name to route a connection to a specific broker. One public port serves every broker.
MultiTenant filter	A Kroxylicious filter that prefixes resource names (topic prefixing was verified here) with the virtual cluster name. Incubating; it does not isolate identities.
Connection & Credential Alias	The ServiceNow record that decouples a connection from its credential. Direct Kafka uses an alias of type Connection and Credential with connection type Kafka.

Prerequisites

• Public DNS names. Two A records pointing at the proxy host: krox.example.com (bootstrap) and krox-1.example.com (broker node 1). Add one krox-<n> record per broker if you have more than one.
• A publicly trusted wildcard certificate for *.example.com, for example from Let's Encrypt. Because the instance trusts public CAs, you do not need to upload a truststore.
• Single-label host names. A wildcard certificate covers exactly one label. That is why the broker pattern is krox-$(nodeId).example.com (one label) and not $(nodeId).krox.example.com (two labels, not covered by *.example.com).
• Port 9192 reachable from the instance. The proxy host must accept inbound TCP 9192 from the internet (or from your instance's egress addresses), including any router forward and host firewall rule. On the test instance no egress allowlist blocked a non-443 port; yours may differ.
• No MID Server. Direct Kafka connects from the instance itself, so the proxy must be reachable from the instance directly.
• Docker with Compose on the proxy host, and the certificate private key in PKCS#8 form (BEGIN PRIVATE KEY). OpenSSL 3 default output works. Place the files at ./certs/privkey.pem and ./certs/fullchain.pem.

Step 1: Create the compose file and start the broker

Create docker-compose.yml with the complete file below. It contains every service used in this guide. Only the broker starts in this step; kroxylicious starts in step 3, client (profile tools) in step 4, and kroxy-hermes (profile hermes) only in Scenario B.

The broker is a single-node KRaft broker with an internal PLAINTEXT listener (used only for administration inside the Docker network) and a SASL_PLAINTEXT listener with SCRAM-SHA-512 (the one the proxy talks to).

name: kxlab

services:
  broker:
    image: confluentinc/cp-kafka:8.0.0
    container_name: kxlab-broker
    restart: unless-stopped
    hostname: broker
    environment:
      CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:29093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENERS: CONTROLLER://0.0.0.0:29093,INTERNAL://0.0.0.0:29092,SASL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker:29092,SASL://broker:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,SASL:SASL_PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-512
      KAFKA_LISTENER_NAME_SASL_SCRAM___SHA___512_SASL_JAAS_CONFIG: "org.apache.kafka.common.security.scram.ScramLoginModule required;"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    networks: [kxlab]

  kroxylicious:
    image: quay.io/kroxylicious/kroxylicious:0.23.0
    container_name: kxlab-kroxylicious
    restart: unless-stopped
    command: ["--config", "/opt/kroxylicious/config/proxy-config.yaml"]
    volumes:
      - ./kroxylicious/proxy-config.yaml:/opt/kroxylicious/config/proxy-config.yaml:ro
      - ./certs/privkey.pem:/opt/kroxylicious/tls/privkey.pem:ro
      - ./certs/fullchain.pem:/opt/kroxylicious/tls/fullchain.pem:ro
    ports:
      - "9192:9192"
    depends_on: [broker]
    networks:
      kxlab:
        aliases:
          - krox.example.com
          - krox-1.example.com

  # Scenario B only: second proxy in front of ServiceNow-hosted Hermes.
  kroxy-hermes:
    image: quay.io/kroxylicious/kroxylicious:0.23.0
    container_name: kxlab-kroxy-hermes
    hostname: kroxy-hermes
    profiles: [hermes]
    restart: unless-stopped
    command: ["--config", "/opt/kroxylicious/config/hermes-proxy-config.yaml"]
    volumes:
      - ./kroxylicious/hermes-proxy-config.yaml:/opt/kroxylicious/config/hermes-proxy-config.yaml:ro
      - ./hermes/hermes-client.key:/opt/kroxylicious/tls/hermes-client.key:ro
      - ./hermes/hermes-client.crt:/opt/kroxylicious/tls/hermes-client.crt:ro
      - ./hermes/hermes-client.keypass:/opt/kroxylicious/tls/hermes-client.keypass:ro
      - ./hermes/hermes-ca.pem:/opt/kroxylicious/tls/hermes-ca.pem:ro
    networks: [kxlab]

  client:
    image: confluentinc/cp-kafka:8.0.0
    container_name: kxlab-client
    profiles: [tools]
    entrypoint: ["sleep", "infinity"]
    volumes:
      - ./client:/client:ro
    networks: [kxlab]

networks:
  kxlab: {}

The per-listener JAAS entry (KAFKA_LISTENER_NAME_SASL_SCRAM___SHA___512_SASL_JAAS_CONFIG) is required. Without it the broker fails at startup with Could not find a 'KafkaServer' ... entry.

docker compose up -d broker

You should see: docker ps lists kxlab-broker with status Up, and docker logs kxlab-broker shows no KafkaServer JAAS error.

Step 2: Create the SCRAM user and test topics

SCRAM users can be created after startup. Use the internal PLAINTEXT listener (broker:29092) for this, because the SASL listener cannot authenticate a user that does not exist yet.

docker exec kxlab-broker kafka-configs --bootstrap-server broker:29092 \
  --alter --entity-type users --entity-name <scram-user> \
  --add-config 'SCRAM-SHA-512=[password=<scram-password>]'

The broker holds topics under the tenant prefix, so create the topic with the prefix. A client going through the proxy will see it as sn-orders. Also create an unprefixed sn-orders as an isolation check: it exists on the broker but must not be visible through the proxy.

docker exec kxlab-broker kafka-topics --bootstrap-server broker:29092 \
  --create --topic tenanta.sn-orders --partitions 1 --replication-factor 1
docker exec kxlab-broker kafka-topics --bootstrap-server broker:29092 \
  --create --topic sn-orders --partitions 1 --replication-factor 1

You still need to give the user rights on the topics if you enable an authorizer. The lab broker runs without one.

You should see: Completed updating config for user <scram-user>., Created topic tenanta.sn-orders. and Created topic sn-orders.

Step 3: Configure and start Kroxylicious

Create kroxylicious/proxy-config.yaml. This is the 0.23.0 syntax: the upstream is virtualClusters[].targetCluster.bootstrapServers.

---
filterDefinitions:
  - name: tenant-prefix
    type: MultiTenant
    config:
      prefixResourceNameSeparator: "."
defaultFilters:
  - tenant-prefix
virtualClusters:
  - name: tenanta
    targetCluster:
      bootstrapServers: broker:9092
    gateways:
      - name: sni
        sniHostIdentifiesNode:
          bootstrapAddress: krox.example.com:9192
          advertisedBrokerAddressPattern: krox-$(nodeId).example.com
        tls:
          key:
            privateKeyFile: /opt/kroxylicious/tls/privkey.pem
            certificateFile: /opt/kroxylicious/tls/fullchain.pem

How it works:

• bootstrapAddress is what clients (and the instance) are given.
• advertisedBrokerAddressPattern is what the proxy tells clients to use for each broker. Node 1 becomes krox-1.example.com, so the DNS record from the prerequisites must exist.
• The virtualClusters[].name (tenanta) becomes the topic prefix, so sn-orders is stored as tenanta.sn-orders.
• The image entrypoint is kroxylicious-start.sh, so the compose file passes --config <path> as the command.
• The network aliases in the compose file let the tools container resolve the same names as the public internet does, so you can test the full SNI path from inside Docker.

docker compose up -d kroxylicious

You should see: docker logs kxlab-kroxylicious shows Kroxylicious is started, and docker ps shows 0.0.0.0:9192->9192/tcp.

Step 4: Verify the proxy before touching the instance

Create the client properties file on the host at ./client/client.properties (do not commit it) before starting the client container. Use your own password in place of the placeholder.

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="<scram-user>" password="<scram-password>";

No truststore entry is needed because the certificate chains to a public CA. If you used a private CA instead, add ssl.truststore.* settings or point the tools at your CA file.

4a. From the tools container (proxy path inside Docker):

docker compose --profile tools up -d client
docker exec kxlab-client kafka-topics --bootstrap-server krox.example.com:9192 \
  --command-config /client/client.properties --list

You should see: exactly one topic, sn-orders. That is the tenanta.sn-orders topic with the prefix stripped by the MultiTenant filter. The unprefixed broker topic sn-orders is not visible through the proxy.

4b. Produce and consume through the proxy:

docker exec -i kxlab-client kafka-console-producer --bootstrap-server krox.example.com:9192 \
  --producer.config /client/client.properties --topic sn-orders
# type: {"number":"INC0010001","state":"new"}

docker exec kxlab-client kafka-console-consumer --bootstrap-server krox.example.com:9192 \
  --consumer.config /client/client.properties --topic sn-orders --from-beginning --max-messages 1

You should see: the record {"number":"INC0010001","state":"new"} come back.

4c. Check the public path. Run the same kafka-topics --list from a machine outside your network, using any Kafka client that has the same properties file. Also check the certificate:

openssl s_client -connect krox.example.com:9192 -servername krox.example.com </dev/null

You should see: a negotiated TLS version and Verify return code: 0 (ok). In the lab this showed TLSv1.3 with a private CA; with a public certificate expect the chain to verify the same way. If the client succeeds from outside, the instance is likely to as well, but egress rules can differ.

4d. Check authentication is enforced. Create a second properties file, ./client/client-bad.properties, identical to the first but with a wrong password, and repeat step 4a with --command-config /client/client-bad.properties.

You should see: failed authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512. The SCRAM exchange passes through the proxy to the broker, so the broker is the one rejecting it.

Step 5: Configure the ServiceNow instance

Four records, in this order. Direct Kafka needs the credential type Kafka credentials, not Kafka SSL credentials. The latter is a different type (kafka_ssl_credentials) used as an input for Health Log Analytics and has free-text SASL fields; it is not what Direct Kafka uses.

5.1 Kafka credentials

Go to Connections & Credentials > Credentials, select New, and choose Kafka credentials (table kafka_credentials, which extends Credentials).

 New credential type chooser

Set Security protocol to SASL_SSL. This reveals a SASL Mechanism dropdown (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER). Choose SCRAM-SHA-512.

Security protocol SASL_SSL reveals the SASL mechanism

Enter the username and <scram-password>. Leave Use SSL truststore unchecked, because the proxy certificate is publicly trusted.

Filled-in Kafka credentials form

You should see: the credential record saved, with SASL_SSL and SCRAM-SHA-512 shown.

5.2 Connection & Credential Alias

Go to Connections & Credentials > Connection & Credential Aliases (also available under IntegrationHub) and create an alias with type Connection and Credential and connection type Kafka, and attach the credential from 5.1.

Alias form

You should see: the alias saved. Note its sys_id (right-click the header, Copy sys_id); you may need it in 5.3.

5.3 Kafka Connection

Go to Connections & Credentials > Connections and create a Kafka connection (it extends Connection). Set Credential to the credential from 5.1 (as in the screenshot), Connection alias to the alias from 5.2 (mandatory) and Bootstrap servers to krox.example.com:9192. Leave Use MID server unchecked.

If the alias field autocomplete returns no matches for the alias you just created, preset it from the URL by appending ?sysparm_query=connection_alias=<alias sys_id> to the new-record URL. The form then opens with the alias filled in.

Kafka Connection form

You should see: the Kafka Connection saved with the alias populated. After you submit, check that the record appears in the Connections list. Once during the lab the first Submit did not save and no error was shown; the second Submit saved it.

5.4 Direct Kafka Cluster, then test

Go to IntegrationHub > Direct Kafka Cluster, create a new record and set External Kafka to the alias from 5.2.

New Direct Kafka Cluster

Select Test Direct Kafka Connection.

Connection validated

You should see: a dialog reading Connection validated.

Step 6: Negative control

Prove the test is really exercising the proxy. Stop it:

docker compose stop kroxylicious

Run Test Direct Kafka Connection again.

You should see: a failure containing TimeoutException: Timed out waiting for a node assignment. Call: listTopics.

Start the proxy again (docker compose start kroxylicious) and re-run the test.

You should see: Connection validated again.

Step 7: Topic sync, topic alias, publish, consume and CDC

7.1 Topic sync

After the Direct Kafka Cluster exists, the scheduled job Direct Kafka Cluster Topic Synchronizer (sysauto_script) reads the topic list through the proxy and populates the cluster's Direct Kafka Topics related list (table sys_kafka_direct_topic). The job is under System Definition > Scheduled Jobs; it is active and runs periodically every 5 minutes, so allow up to about 5 minutes after creating the cluster. You can also see the synced topics under IntegrationHub > Direct Topics.

Synced topic in the cluster's Direct Kafka Topics list

You should see: sn-orders in the cluster's Direct Kafka Topics list, without the prefix, exactly as the tenant sees it. A broker-direct kafka-topics --list shows tenanta.sn-orders and the unprefixed sn-orders.

7.2 Topic alias

1. Go to IntegrationHub > Stream Connect > Topic Alias > New (table sys_sc_topic_alias), give the alias a name (for example kxlab-orders), and save.
2. Use the related link Connect Direct Kafka Topic and pick the synced topic sn-orders. The lookup field in the dialog has no autocomplete in this UI, but it accepts typed text.

Topic alias connected to the Direct Kafka topic

You should see: the alias form now shows Disconnect Direct Kafka Topic and Direct Kafka Topics (1), and the topic record links back to the alias (sc_topic_alias).

7.3 Publish from the instance

On an instance that uses Direct Kafka only, publish with KafkaProducerAPI, which takes the sys_sc_topic_alias sys_id. It needs the plugin com.glide.hub.action_step.kafka. Run this as a background script:

var result = new sn_ih_kafka.KafkaProducerAPI().send({
    topicSysId: '<topic-alias-sys-id>',
    key: 'kx1',
    message: '{"number":"INC0010001","state":"new"}',
    isSync: true
});
gs.info(JSON.stringify(result));

You should see: a synchronous delivery result such as {"delivery":"sync","partition":0,"offset":2}. The offset depends on what is already in the topic.

Do not use sn_ih_kafka.ProducerV2().send(...) here. On the test instance, which used Direct Kafka only, it failed with Untrusted GlideRecord for sys_kafka_topic; the sys_kafka_topic table had no rows without Hermes. Passing the Direct topic id or the alias id to it failed the same way.

7.4 Check the broker side

Consume from the broker's internal listener, bypassing the proxy:

docker exec kxlab-broker kafka-console-consumer --bootstrap-server broker:29092 \
  --topic tenanta.sn-orders --from-beginning --property print.key=true

You should see: the record with key kx1 in tenanta.sn-orders. The MultiTenant filter applied the prefix; the unprefixed broker topic sn-orders is untouched.

7.5 Consume messages with ServiceNow as the client

The instance can also read from the topic through the proxy. This takes a Script Consumer and a Kafka Stream.

1. Go to IntegrationHub > Consumers > Script Consumer (table sys_kafka_script_consumer) and create a consumer with a script like this. m.key, m.message and m.metadata.partition / m.metadata.offset are available on each message.

(function process(messages) {
    for (var i = 0; i < messages.length; i++) {
        var m = messages[i];
        gs.info('[orders consumer] key=' + m.key +
                ' partition=' + m.metadata.partition +
                ' offset=' + m.metadata.offset +
                ' message=' + m.message);
    }
})(messages);

2. From the consumer's Kafka Streams related list, create a Kafka Stream (also under IntegrationHub > Kafka Streams). Set Topic Alias to the alias from step 7.2 and Start consuming to All the messages stored in the topic. Save.
3. Select the related link Activate.

You should see: the stream's subscription go to ACTIVE with no error. Within about a minute, the system log list shows one [orders consumer] line per record already in the topic, with its key and offset (filter on your log prefix and a recent time range; unfiltered log queries are slow). Publish another record with the script from 7.3 and a new line follows with the next offset. The reads go through Kroxylicious, so the instance sees the unprefixed topic name.

7.6 Stream a table with Stream Producer (CDC)

A Stream Producer publishes table changes to the topic without any script.

1. Go to IntegrationHub > Stream Connect > Stream Producers and select New (table sys_sc_stream_producer).
2. Set: - Table Name: incident - Filter: a condition that matches only your test records, for example short_description starts with kxlab - On Insert and On Update: selected - Topic Alias: the alias from step 7.2 - Key: number - Serialization: JSON
3. The producer is created inactive. Select the related link Activate.
4. Insert an incident with a short description that matches the filter.

Always set a filter. Without one, every insert and update on the table is published, which can flood the topic.

You should see: within about a minute, one record in the topic with:

• key: the incident number (for example INC0010031)
• headers: action:INSERT, table_name:incident, collection_timestamp:<ISO timestamp>
• value: JSON of the form {"current":{ ...all incident fields... }} (about 2 KB for an incident)

Check it on the broker, bypassing the proxy:

docker exec kxlab-broker kafka-console-consumer --bootstrap-server broker:29092 \
  --topic tenanta.sn-orders --from-beginning \
  --property print.key=true --property print.headers=true

You should see: the same record under tenanta.sn-orders. If the Script Consumer from 7.5 is still active, it logs the record as well.

Troubleshooting

Symptom	Likely cause and fix
TimeoutException: Timed out waiting for a node assignment. Call: listTopics	The instance cannot complete the connection. Check, in order: the proxy is running; DNS for krox.example.com and krox-<n>.example.com resolves publicly; port 9192 is open end to end (router forward, host firewall); the SNI host name the client uses matches a name on the certificate and the advertisedBrokerAddressPattern.
failed authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512	Wrong user or password, or the user was never created. Re-run step 2 and re-enter the password in the Kafka credentials record.
Broker will not start: Could not find a 'KafkaServer' ... entry	The per-listener JAAS variable is missing. Add KAFKA_LISTENER_NAME_SASL_SCRAM___SHA___512_SASL_JAAS_CONFIG.
Proxy fails to start or load the gateway key	A likely cause is the key format: the lab used a PKCS#8 key (BEGIN PRIVATE KEY), which worked. Check the proxy log for the exact error.
Untrusted GlideRecord for sys_kafka_topic when publishing	ProducerV2 did not work on the Direct-Kafka-only test instance. Use KafkaProducerAPI with the topic alias sys_id (step 7.3).
Alias autocomplete is empty on the Kafka Connection form	Preset it with ?sysparm_query=connection_alias=<alias sys_id> (step 5.3).
Kafka Connection does not appear in the list after Submit	Observed once: the first Submit did not save, with no error. Submit again and verify the record in the list.
No Kafka SASL options on the credential form	You chose Kafka SSL credentials. Create a Kafka credentials record instead.

Gotchas and differences

Documented differences:

• Kroxylicious version and syntax. The docs site shows 0.25.0 (clusterDefinitions, target.cluster). The newest published container image is 0.23.0, which uses virtualClusters[].targetCluster.bootstrapServers. Use the syntax that matches the image you run.
• Credential type. The ServiceNow documentation for Direct Kafka credentials uses Kafka credentials (kafka_credentials), not Kafka SSL credentials, which is the Health Log Analytics input type.

Lab gotchas:

• The Kroxylicious image entrypoint is kroxylicious-start.sh; pass --config <path> as the command.
• Confluent Platform 8.0.0 needed a per-listener JAAS entry for the SCRAM listener.
• With a publicly trusted certificate, leaving Use SSL truststore unchecked worked.
• A wildcard certificate covers one label only, which drives the krox-$(nodeId) host name pattern.
• The alias lookup on the Kafka Connection form returned nothing for a new alias; presetting it through the URL worked.
• ProducerV2 failed on a Direct-Kafka-only instance; KafkaProducerAPI with a topic alias sys_id worked.

Scenario B: a Kafka client behind the proxy writing to ServiceNow-hosted Hermes

Scenario A put the proxy in front of your Kafka. Scenario B puts a second Kroxylicious instance in front of ServiceNow-hosted Hermes. Applications on your network use an ordinary Kafka client against the proxy; the proxy presents the instance's mTLS client identity to Hermes, so no application needs ServiceNow certificates, keystores or the Hermes address.

Plain Kafka client no certs no Hermes address

→ plaintext

Kroxylicious 0.23.0 (second instance) Gateway: portIdentifiesNode, plaintext, no filters Upstream TLS client: mTLS key + CA from your Hermes setup

→ mTLS

Hermes (ServiceNow-hosted) topic: snc.<instance>.sn_streamconnect.<hermes-topic>

In the diagram, instance stands for the <instance> part of your Hermes topic prefix.

B.1 Prerequisite: the client identity

The proxy needs three things in PEM format: a client certificate, a private key and a CA certificate. They come from the Hermes client keystore and truststore that your instance generates.

Get the keystore and truststore. The documented procedure is on the page "Set up a secure connection to the Hermes Messaging Service for LES": (https://www.servicenow.com/docs/r/platform-security/les-hermes-cert.html). It is the LES page, but it is the documented certificate procedure for Hermes clients. In summary:

1. Prerequisites: the Hermes Messaging Service is activated, the Key Management Framework plugin (com.glide.kmf.global) is active, an instance root CA exists in sys_kmf_certificate, and the instance does not use a Custom URL.
2. You need the role hermes_admin, sn_kmf.cryptographic_manager or admin.
3. Go to All > Certificate Generator > Instance PKI Certificate Generator.
4. Optionally select Configure ACLs to limit the certificate to namespaces or defined topics, as Read Only or Read/Write. Topic and consumer-group access for this identity is governed by these ACLs.
5. Enter a keystore password in Certificate Password and select Generate.
6. Select Download Keystore and Download Truststore.

Only keystores created by this generator are supported.

Convert to PEM with OpenSSL. Run these on a trusted machine. Replace keystore and truststore with the downloaded file names.

# Client certificate
openssl pkcs12 -in keystore -clcerts -nokeys -out hermes-client.crt

# CA certificate
openssl pkcs12 -in truststore -nokeys -out hermes-ca.pem

Then extract the private key in one of two ways.

Variant 1, encrypted key (matches the configuration in B.2 and the compose file). OpenSSL prompts for the keystore password and then for a PEM passphrase. The result is an ENCRYPTED PRIVATE KEY (PKCS#8). Put the passphrase in a file for the proxy:

openssl pkcs12 -in keystore -nocerts -out hermes-client.key
printf '%s' '<key-passphrase>' > hermes-client.keypass

Variant 2, unencrypted key. Use -nodes instead of -noenc on OpenSSL before 3.0. If you use this variant, remove the keyPassword block from the proxy config in B.2 and the hermes-client.keypass mount from the compose file, and do not create the password file:

openssl pkcs12 -in keystore -nocerts -noenc -out hermes-client.key

Restrict permissions on the key (for example chmod 600 hermes-client.key hermes-client.keypass). If OpenSSL 3 cannot read an older keystore, add -legacy to the openssl pkcs12 commands.

How much was verified: the flags were checked against the OpenSSL 3 openssl pkcs12 -help output, and in the lab the extracted key was an encrypted PKCS#8 key. A full end-to-end run of every variant on every keystore type was not performed. Confirm your files before starting the proxy:

openssl x509 -in hermes-client.crt -noout -subject -enddate

Keep the key, the password file and the CA out of source control. Place the files in ./hermes/ as hermes-client.crt, hermes-client.key, hermes-client.keypass (variant 1 only) and hermes-ca.pem, matching the mounts in the compose file.

B.2 Proxy configuration

Create kroxylicious/hermes-proxy-config.yaml:

---
virtualClusters:
  - name: hermes
    targetCluster:
      bootstrapServers: <instance>.service-now.com:<port>
      tls:
        key:
          privateKeyFile: /opt/kroxylicious/tls/hermes-client.key
          certificateFile: /opt/kroxylicious/tls/hermes-client.crt
          keyPassword:
            passwordFile: /opt/kroxylicious/tls/hermes-client.keypass
        trust:
          storeFile: /opt/kroxylicious/tls/hermes-ca.pem
          storeType: PEM
    gateways:
      - name: local
        portIdentifiesNode:
          bootstrapAddress: kroxy-hermes:9193
          advertisedBrokerAddressPattern: kroxy-hermes
          nodeIdRanges:
            - name: brokers
              start: 0
              end: 5
    filters: []

Notes:

• There is no MultiTenant filter. The topic names are Hermes topic names and must not be rewritten.
• The gateway is portIdentifiesNode: each node id gets its own port on the proxy. The node id range must cover the broker ids Hermes advertises (0 to 5 in the lab). See the Kroxylicious proxy guide for the port allocation rules. On a compose network no port needs to be published.
• advertisedBrokerAddressPattern must be the host name clients can resolve for the proxy (here the compose service host name); it is the same value as the host part of bootstrapAddress.
• This used the same 0.23.0 image and --config command pattern as Scenario A.

docker compose --profile hermes up -d kroxy-hermes

You should see: docker logs kxlab-kroxy-hermes reports the virtual cluster transitioning to Serving.

B.3 Produce with a bare client

The bare client runs in the kxlab-client container from Scenario A, which is on the same compose network and so resolves kroxy-hermes. It needs no properties file and no security settings, only the proxy address. Start it if it is not running (docker compose --profile tools up -d client), then:

echo '{"hello":"hermes"}' | docker exec -i kxlab-client kafka-console-producer \
  --bootstrap-server kroxy-hermes:9193 \
  --topic snc.<instance>.sn_streamconnect.<hermes-topic>

docker exec kxlab-client kafka-producer-perf-test \
  --topic snc.<instance>.sn_streamconnect.<hermes-topic> \
  --num-records 3 --record-size 100 --throughput -1 \
  --producer-props bootstrap.servers=kroxy-hermes:9193 acks=all

You should see: the console producer exit 0, and the perf test report 3 records sent with an average latency in the hundreds of milliseconds (231 ms in the lab). With acks=all that means Hermes acknowledged the writes.

B.4 Negative control and limits

• Point the same bare client directly at the Hermes bootstrap address, from the client container:

bash docker exec kxlab-client kafka-producer-perf-test --topic snc.<instance>.sn_streamconnect.<hermes-topic> --num-records 3 --record-size 100 --throughput -1 --producer-props bootstrap.servers=<instance>.service-now.com:<port> acks=all

You should see: a timeout such as Topic ... not present in metadata after 10000 ms. The mTLS identity comes from the proxy, not from the client. - Consuming back through the proxy failed with GroupAuthorizationException: Not authorized to access group (observed with one arbitrary consumer group). Topic and consumer-group access is governed by the ACLs set in the Instance PKI Certificate Generator (B.1). - Reading the records with a consumer on an instance was not tested in this guide. Instead, confirm delivery in the Hermes Topic Inspector (next section).

B.4a Confirm in the Hermes Topic Inspector

The Hermes Topic Inspector is a documented view of Hermes topics. You need the role hermes_admin, kafka_namespace_admin or admin. Go to All > Hermes Messaging Service > Topic inspector, select the cluster in Select Service, select List Topics, select the topic and select Inspect Topic. Messages expire after 36 hours, so check promptly. Headers, keys and payloads longer than 20 characters open through View or download. See (https://www.servicenow.com/docs/r/servicenow-platform/multi-instance-framework-hermes/view-messages-...) and, for topic monitoring in general, (https://www.servicenow.com/docs/r/servicenow-platform/multi-instance-framework-hermes/monitoring-top...). You should see: the topic with an end offset equal to the number of records you sent, and one row per record with its size. In the lab the console producer record (46 bytes) was offset 0 and the three perf-test records (64 bytes each) were offsets 1 to 3.

Hermes Topic Inspector showing four messages on the test topic

B.5 Security note

The gateway in this scenario is plaintext and unauthenticated. Anyone who can reach it gets the Hermes producer identity. Keep it on a private network segment, restrict who can reach the port, and do not publish it to the internet. Adding TLS and authentication on the gateway was not tested.

Cleanup

• On the instance, deactivate and delete the Stream Producer, deactivate the Kafka Stream, delete the Script Consumer, the test incident and the topic alias, then delete the Direct Kafka Cluster, Kafka Connection, alias and Kafka credentials you created.
• On the proxy host: docker compose --profile tools --profile hermes down -v, then delete the private key and certificate files, client.properties and client-bad.properties.
• Remove the DNS records, the router port forward for 9192 and any host firewall rule you added.
• Scenario B: remove the kxlab-kroxy-hermes container (covered by the down command above), delete the ./hermes/ folder (client private key, key password file, client certificate, CA) and hermes-proxy-config.yaml.

Where to go next

Kroxylicious 0.23.0:

• Proxy guide: https://kroxylicious.io/documentation/0.23.0/html/kroxylicious-proxy/
• Multi-tenancy guide: https://kroxylicious.io/documentation/0.23.0/html/multi-tenancy-guide/

ServiceNow documentation:

• Direct Kafka: https://servicenow.com/docs/r/integrate-applications/integration-hub/direct-kafka.html
• Stream Connect for Apache Kafka: https://servicenow.com/docs/r/integrate-applications/integration-hub/stream-connect-apache-kafka.htm...
• Manage topic aliases: https://servicenow.com/docs/r/integrate-applications/integration-hub/manage-topic-alias.html
• Configure a script consumer: https://servicenow.com/docs/r/integrate-applications/integration-hub/configure-script-consumer.html
• Create a Kafka stream: https://servicenow.com/docs/r/integrate-applications/integration-hub/create-kafka-stream.html
• Create a stream producer: https://servicenow.com/docs/r/integrate-applications/integration-hub/create-stream-producer.html
• KafkaProducerAPI (scoped): https://servicenow.com/docs/r/api-reference/server-api-reference/KafkaProducerAPIScoped.html
• Set up a secure connection to the Hermes Messaging Service (Instance PKI Certificate Generator): https://www.servicenow.com/docs/r/platform-security/les-hermes-cert.html
• View a message in a Hermes topic (Topic Inspector): https://www.servicenow.com/docs/r/servicenow-platform/multi-instance-framework-hermes/view-messages-...
• Monitoring topics in the Hermes Kafka cluster: https://www.servicenow.com/docs/r/servicenow-platform/multi-instance-framework-hermes/monitoring-top...