- Post History
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
2 hours ago
Kafka Fundamentals Through a Stream Connect Lens
Kafka Reconciliation Series · Post 1 of 3
Why this matters before you configure a single topic
You've just been asked to wire up a Stream Connect integration for one of your biggest customers. The conversation starts with "We want to use Kafka" - and you realize: you don't actually know what Kafka is. Not at a depth that lets you design it correctly. This post exists to fix that.
The good news: Kafka is conceptually simple. The bad news: the details matter. Get a single assumption wrong early - about delivery guarantees, or how duplicates work, or when messages can disappear - and you'll find out the hard way when data goes missing at 3 AM.
This is post 1 of a 3-part series. We're starting with the fundamentals: what Kafka IS, why its delivery model matters to you, and how to NOT lose data (or at least, how to recover from it if you do). Posts 2 and 3 go deeper on message design and consumer patterns.
What is Kafka? A mental model
Kafka is a distributed event streaming platform - think of it as a high-speed mailing list with memory. Here's what lives under the hood.
Broker: A single machine (process) that stores messages. It's the actual keeper of the data. You usually run multiple brokers in a cluster for redundancy and scale, but conceptually: one broker = one machine doing the job.
Topic: A named category of messages. Like a mailing list. If you want events from your CRM, you might create a topic called crm-events. Anything publishing to Kafka needs to pick a topic; anything subscribing needs to pick one too.
Partition: The trick that makes Kafka fast. A topic can be split across multiple partitions - think of them as parallel lanes on a highway. If you have 3 partitions for crm-events, messages land on partition 0, 1, or 2 depending on the message key (more on that in a moment). Each partition is a separate queue that preserves order. This parallelism is what lets Kafka handle huge throughput.
Producer: The thing that SENDS messages to Kafka. In Stream Connect, that's the Kafka Producer Step or the ProducerV2 API. The producer picks a topic, optionally a partition (usually via a key), and publishes the message.
Consumer: The thing that READS messages from Kafka. It subscribes to a topic, reads messages in order, and processes them. In Stream Connect, the consumer is your ETL job, Transform Map, or Script step pulling messages off a topic.
One more piece of vocabulary: offset. An offset is a position in a partition - think of it as a line number. Partition 0 might have messages at offset 0, 1, 2, ... 1000. A consumer tracks which offset it's read up to. When it crashes and restarts, it resumes from that offset.
Figure 1
A producer publishes to topic crm-events. The message key determines which of the three partitions the message lands on. A consumer reads across all partitions, tracking offsets independently in each.
| Producer | → |
Topic: crm-events
message key determines partition |
→ | Consumer |
At-least-once delivery: the promise and the catch
Here's what Stream Connect guarantees: every message a producer publishes will be delivered to the broker at least once.
At least once. Not exactly once. At least.
This matters because of failure scenarios. Say a producer publishes a message, the broker accepts it, but the ACK (acknowledgment) gets lost on the network. The producer doesn't see the ACK, assumes the message was lost, and retries. Now the broker has the message twice. Both copies are identical. Both are sitting in the topic.
This is not a bug. This is the contract.
On the consumer side, you need to understand your delivery setting. If you're using an ETL job or Transform Map consumer in Stream Connect, you can choose:
- No lost but duplicates (default): The consumer will not lose messages. But you will see duplicates. Expect them.
- Once or not at all: The consumer eliminates duplicates - but if a node crashes mid-processing, some messages might not get processed at all. You trade duplicate resilience for message loss risk.
Use "Once or not at all" only when you genuinely cannot tolerate a second write for the same message. Otherwise, stick with "No lost but duplicates" and handle duplicates in your consumer logic (which we cover next).
Producer-side idempotence (it's not enough)
Kafka brokers have a built-in deduplication feature: the enable.idempotence flag (on by default). Here's what it does.
The broker assigns each producer a Producer ID and a sequence number for each message. When the producer retries a message batch, the broker recognizes the batch by its ID + sequence and discards the duplicate instead of storing it twice.
This is great. But it only works within a single producer session. The moment a producer dies and restarts, it gets a new Producer ID. A message from the old session looks brand new to the broker, even if it was already written.
This is why you still need deduplication on the consumer side. The broker's idempotence is a network-level safeguard, not an end-to-end guarantee.
Consumer-side idempotence: three patterns
If your consumer might see a message twice (which it will, given the at-least-once contract), here are three ways to make sure the second write doesn't corrupt your data.
Pattern 1: Upsert logic
When you receive a message, use a stable ID as the key. If that ID already exists in your target system, update the row (or skip it). If it doesn't exist, insert it. The second write becomes idempotent by definition: you're updating the same row with the same data.
Example: a customer record arrives with ID 12345. You upsert on that ID. If it comes a second time, the ID already exists, so you update the same row. No duplicate.
This is the simplest pattern and works for most use cases.
Pattern 2: Message key plus deduplication
Set the Kafka message key to a stable identifier (like a sys_id or customer ID). Kafka guarantees that messages with the same key always land on the same partition, which preserves order for that key. On the consumer side, maintain a state store or cache of seen keys. If you see a key you've already processed, skip it.
This pattern is good when you need to guarantee ordering for a specific entity AND want to explicitly deduplicate. It's more complex than upsert but gives you fine-grained control.
Pattern 3: Timestamp versioning
Embed an updated timestamp or sequence number in each message. When two messages arrive for the same record (a duplicate scenario), compare their timestamps. Keep the one with the later timestamp; discard or skip the older one.
This pattern is particularly useful when you're not sure if the second message is a true duplicate or a genuinely newer update to the same record. The timestamp lets you distinguish between the two.
Figure 2
A producer retries after a lost ACK, so the same message reaches the topic twice. Whichever pattern the consumer applies, the outcome is a single correct record instead of a duplicate.
| Producer retries after lost ACK | → | Message A arrives twice in the partition | → | Consumer applies a dedup pattern |
| Pattern | How it works | Result on duplicate |
|---|---|---|
| 1. Upsert | Upsert on ID 12345 | Second write updates the same row |
| 2. Key cache | Check a cache of seen keys before processing | Key 12345 already seen - skip |
| 3. Timestamp compare | Compare embedded timestamps | Keep the latest, discard the older duplicate |
| ↓ Single correct record, no duplicate | ||
Where to go next
You now have the mental model. If you want to dig deeper:
- ServiceNow Docs: Using Stream Connect for Apache Kafka
- ServiceNow Docs: Hermes Messaging Service release notes
- For deeper dives on core Kafka concepts (brokers, partitions, offsets, rebalancing): Apache Kafka's official Introduction to Kafka and Confluent's free Kafka 101 course are both excellent
Post 2 of this series digs into a question you'll face immediately: what data actually goes INSIDE your Kafka messages? Spoiler - it's not always the whole record.
