How do I check the events that I push to the Kafka cluster in Stream Connect?

[Johannes Mweli]

Hi ServiceNow Community Developers,

I am working on a project where we are trying to use Stream Connect to push data from the ServiceNow instance to the Kafka cluster. After calling send method of sn_ih_kafka.ProducerV2() API where do I go to check it’s output and verify that it was executed successfully. I am new to Stream Connect and this is my first project on it however my understanding, based on the research that I have done so far on Stream Connect, is that the sn_ih_kafka.ProducerV2() API sends the messages to the Kafka cluster. What I am therefore asking is where in the platform do I go to check the output in the Kafka cluster. Basically how do I navigate to the Kafka cluster?

Kindly advise please.

Thanks,
Johannes

[kmohammed]

Hi Johannes,

The short answer is: there is nowhere in the ServiceNow platform that lets you browse/read back the messages inside the Kafka cluster. ServiceNow (Stream Connect) is the producer in your design - once ProducerV2 sends, the message lives in the topic on the Kafka/Hermes side, not in a ServiceNow table you can open. So verification is done in two separate places:

1. Verify the SEND in ServiceNow. ProducerV2 sends asynchronously, so "the script ran without throwing an error" is not proof of delivery. Check the result/error handling of your send call, and then check the Stream Connect side: the topic and producer configuration under Stream Connect for Kafka (topic name must match exactly what you sent to), and the system logs / Stream Connect message delivery records for that send - failures like an unknown topic, authentication or a connection error to the cluster show up there as errors, not in the topic.

2. Verify the MESSAGE in Kafka, from a consumer - this is the only authoritative check that your payload arrived and what it looks like:

- Simplest round-trip test: create a Stream Connect consumer in the instance subscribed to the same topic (with a flow/trigger that just logs the incoming message), send one test message from your producer, and confirm the consumer receives it. That proves producer -> cluster -> consumer end to end.

- Or consume directly on the cluster side with a Kafka tool, for example kafka-console-consumer (or kcat / your Kafka UI if your team runs one) against the same topic, reading from the beginning for a test topic so you see the message you just pushed.

A practical tip for testing: put a unique, searchable identifier in the test payload (for example a test ID plus a timestamp in the message body/key). Then search for that exact ID in the consumer output/logs. Matching on that ID proves that specific send() call delivered - just seeing other messages on the topic does not.

So navigate to Stream Connect for configuration and delivery errors, and to a consumer of the topic for the actual message content.