What Actually Goes Inside a Kafka Message?

[Adam Celli]

Stream Connect Series · Part 2 of 3

What Actually Goes Inside a Kafka Message?

Last time, you got the "we want to use Kafka" conversation sorted out. You know what a broker is, you know Stream Connect sits between ServiceNow and Hermes, and you know delivery is at-least-once - messages get through, but duplicates are the price of admission.

That last part matters more than it sounds like it should, because it's about to collide with your next decision: what actually goes inside the message you're publishing?

It feels like it should be obvious. You have a record. You publish the record. Done, right?

Not quite. And getting this wrong doesn't blow up on day one - it blows up three weeks later, when the security team asks why every subscriber to your incident topic can see the caller's phone number, or when your "real-time" integration turns out to be making a Table API call back to ServiceNow for every single message and your instance starts sweating under the load.

First, a quick word on duplicates

Stream Connect producers - whether you're using the Kafka Producer Step or the ProducerV2 API - guarantee at-least-once delivery to Hermes. Under retries and failures, a message can be delivered more than once. That's not a bug you're going to engineer away; it's the contract.

Kafka's broker-side enable.idempotence flag helps with one narrow slice of this - it dedupes retries within a single producer session. But it's session-scoped. Restart the producer and you get a new session, a new Producer ID, and a message published again after that restart looks like a brand-new message to the broker. Broker-level idempotence protects against retry storms; it does not protect against your integration logic.

So the responsibility for handling duplicates lands on the consuming system. Part 1 went deep on all three consumer-side patterns - upsert logic, key-based dedup, and timestamp versioning - including which one to reach for and why; go there for the full breakdown. The short version of Pattern 1 (Upsert logic): key on sys_id, insert if new, update if it already exists, and the second write becomes idempotent by definition.

What's worth adding now is that message key-based deduplication and timestamp versioning - the other two dedup strategies - depend on what's actually in the payload. Key-based dedup needs a stable identifier (sys_id or otherwise) to anchor a seen-key cache or state store. Timestamp versioning needs sys_updated_on or a sequence number in the payload, so that when two messages show up for the same record, the consumer can keep the later one and discard the earlier - which also handles out-of-order delivery, not just duplicates. Your dedup strategy and your payload design are the same decision wearing two hats.

The real question: full record, or just a pointer?

This is the one that surprises people. Message design in Stream Connect isn't a platform-wide setting - it's a per-topic, per-use-case decision. The same source table can publish full records to one topic and lightweight pointers to another, at the same time, for different consumers. Nobody's making you pick one pattern and live with it forever.

There are two baseline patterns, plus a hybrid, plus a way to get the benefits of "full record" without the downside. Let's go through them.

Event-Carried State Transfer: send the whole thing

This is the pattern most people default to, and for good reason - the message carries the complete record. The consumer is self-sufficient. It doesn't need to call back into ServiceNow for anything; everything it needs is already sitting in the payload.

Use it when: the consumer is a data lake, warehouse, or analytics platform; latency matters and you can't afford a round-trip; or the consumer genuinely can't reach ServiceNow directly (common with external, customer-managed infrastructure).

Watch for: every field on that record is now visible to anyone subscribed to the topic. If the record has PII, business-impact classifications, or anything sensitive, topic-level ACLs suddenly matter a lot - or you need to solve access control somewhere else entirely (more on that below).

• This term isn't a ServiceNow invention - it's a standard architectural pattern from Martin Fowler's 2017 article "What do you mean by 'Event-Driven'?", and the name comes up in event-driven architecture discussions well outside the Kafka world.

Event Notification: send a pointer, not the payload

Instead of the record, you publish table_name.sys_id and a timestamp. The consumer gets the pointer, then calls the Table API to retrieve the full record at processing time.

Use it when: access control is better enforced at the API layer (where role checks are already living) than through topic ACLs, or when payload size is a real concern at high message volume.

Watch for: every notification is now an API call back to your instance. At scale, that's inbound load on ServiceNow plus retrieval latency layered on top of the message latency. If a message needs to be processed the instant it lands, pointer-based notification is fighting you.

Hybrid: both, from the same table, at the same time

Nothing stops you from running both patterns off the same source table, split by topic. A single Stream Connect configuration can publish full Event-Carried State Transfer messages to a warehouse-bound topic while publishing lightweight Event Notification messages to an alerting topic - same underlying data, two different consumer needs, two different payload shapes.

This is the part that trips people up when they're planning: you don't need a platform-wide "we do full records" or "we do pointers" policy. You need a per-topic answer to "what does this consumer actually need."

Source table (e.g. incident)	→ Event-Carried State Transfer	Topic: incident-full	→	Data warehouse (self-sufficient, full record)
	→ Event Notification	Topic: incident-pointer	→	Alerting consumer (pointer + Table API call back)

Solving access control without giving up on full records

Say you want the simplicity of Event-Carried State Transfer - one topic, full records, no per-field filtering logic living in your producer - but you can't have every subscriber seeing every field. You don't have to build field-filtered topics for every consumer type at the producer. You can push that problem downstream.

Stream processing fan-out. Publish the full record to one ACL-restricted topic - say, sn.incidents.full. Then let a stream processor read that topic and fan it out into narrower, filtered downstream topics: sn.incidents.ops with PII fields (caller name, contact info, user identifiers) stripped, sn.incidents.security with business-impact classification and assignment-group details removed. Whatever stream-processing tool your organization already runs - ksqlDB, Kafka Streams, Apache Flink, Spark Structured Streaming, or similar - supports this read-filter-write topology. ServiceNow doesn't prescribe one; the filtering rules live entirely in the customer's environment, not in your producer logic.

Data warehouse RBAC at query time. Alternatively, load the full record once - say, into change_requests_full - and apply access control at the query layer instead of the ingestion layer. Define a role_itsm_analyst that sees everything except requested_by_email, approval_set, and work_notes. Define a role_compliance_auditor that sees everything but only for the past 90 days. No duplicate tables, no additional ingestion pipeline. Snowflake Dynamic Data Masking, BigQuery policy tags, and Databricks Unity Catalog row/column filters all support this - ingest once, govern at query time.

Both approaches land on the same principle: producer logic stays simple, and access control ownership shifts to whichever team actually owns that boundary - the data platform team, typically, which is usually the right owner for data governance anyway.

Where this leaves you

Message design isn't a one-time architectural decree - it's a per-topic decision you can revisit as consumers change. Full record when the consumer needs to be self-sufficient, a pointer when API-layer access control or payload size matters more than latency, both at once when different consumers off the same table need different things, and downstream fan-out or RBAC when you want full records without giving up field-level control.

Governance note: who administers topic ACLs, and the cost of standing up a stream-processing fan-out, is a customer-specific architecture conversation - not a platform default. This series covers the design patterns available; the ownership decision for your organization belongs with your integration architecture team.

None of this tells you whether the data actually arrived, though. You've decided what's in the message - next you have to prove every message you sent landed where it was supposed to, in the right count, without anything silently going missing along the way. That's Part 3: producer statistics, consumer lag, and how Zero Copy Connectors let you close the reconciliation loop without moving any data at all.