Kafka and Kafka Connect¶

Apache Kafka is a distributed log that stores streams of events durably and lets many applications write and read them independently; Kafka Connect is its framework for moving data between Kafka and other systems without writing custom code.

Why it matters¶

Most streaming and change data capture pipelines pass through Kafka or a system with the same model. It decouples the systems that produce events from the systems that consume them: a producer writes once, and any number of consumers read at their own pace, now or days later. Knowing how topics, partitions and offsets work explains ordering, scaling and replay in every pipeline built on it.

How it works¶

A topic is a named stream of events, such as shop.clickstream. Each event, called a record, has a key, a value, a timestamp and optional headers. A topic is split into partitions, and each partition is an append-only log: records are added at the end and never changed. Each record gets an offset, its position in the partition.

The key decides the partition. Records with the same key always go to the same partition, so they stay in order relative to each other. There is no order across partitions. Choosing the key is therefore choosing what must stay in order: for clicks, the session; for order changes, the order ID.

flowchart LR
  P[Producer:<br/>web servers] -->|key = session_id| T
  T["Topic shop.clickstream<br/>partition 0: offsets 0, 1, 2, ...<br/>partition 1: offsets 0, 1, 2, ...<br/>partition 2: offsets 0, 1, 2, ..."]
  T --> G1[Consumer group: sessionizer<br/>3 consumers, one per partition]
  T --> G2[Consumer group: lake-sink<br/>1 consumer reads all three]

The diagram shows three partitions; the example below uses twelve. Consumers read in consumer groups. Within a group, each partition is assigned to exactly one consumer, so the partition count caps the parallelism of a group. Kafka 4.2 (February 2026) made share groups production-ready: consumers in a share group can read the same partition, with records acknowledged one by one like a queue, at the cost of ordering (checked October 2026). Each group stores its own committed offsets, the position it has processed up to. Different groups read the same topic independently, and a group can be reset to an earlier offset to read data again.

Records are not deleted when read. A topic keeps them by retention: for a period, such as seven days, or up to a size. A compacted topic keeps instead the latest record for each key, which turns it into a table of current values. A record with a key and an empty value, called a tombstone, marks that key as deleted and is itself removed after a while.

A Kafka cluster is a set of brokers. Each partition is replicated to several brokers, and producers can wait until all in-sync replicas have a record before treating it as written (acks=all). Since Kafka 4.0, released in March 2025, clusters manage their own metadata with the KRaft protocol; the earlier dependency on ZooKeeper has been removed.

Kafka Connect runs connectors, configured with JSON rather than code. Source connectors read from an outside system into topics, for example Debezium reading a database's change log. Sink connectors write topics to an outside system, such as an object store, a database or an Iceberg table. Connect handles offsets, retries, scaling across workers and converting records to and from formats such as Avro or JSON. Single message transforms make small changes to each record on the way, such as renaming a field or routing to another topic. Anything more complex belongs in a stream processor or later in the platform.

Key terms¶

• Topic: a named stream of records.
• Partition: an ordered, append-only part of a topic; the unit of ordering and parallelism.
• Offset: a record's position within a partition.
• Consumer group: consumers that share the work of reading a topic, with their own committed offsets.
• Retention: how long or how much data a topic keeps.
• Compacted topic: a topic that keeps the latest record per key.
• Connector: a configured source or sink in Kafka Connect.

Example¶

The shop's clickstream topic, keyed by session so that each session's clicks stay in order:

kafka-topics.sh --bootstrap-server broker-1:9092 \
  --create --topic shop.clickstream \
  --partitions 12 --replication-factor 3 \
  --config retention.ms=604800000   # 7 days

A sink connector that writes the topic to object storage, as configured in Kafka Connect:

{
  "name": "clickstream-to-lake",
  "config": {
    "connector.class": "<sink connector class>",
    "topics": "shop.clickstream",
    "tasks.max": "3",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "io.confluent.connect.avro.AvroConverter",
    "value.converter.schema.registry.url": "https://registry.example.com"
  }
}

With 12 partitions, up to 12 consumers in one group can share the reading; a thirteenth would sit idle.

Pitfalls¶

• A key that is too coarse. Keying clicks by country puts most traffic on a few partitions while others sit idle.
• Too few partitions. They cap consumer parallelism; adding partitions later changes which partition a key maps to.
• Retention shorter than the longest outage. A consumer down for longer than retention loses data permanently.
• Treating Kafka as the permanent store. Keep history in the lake; topics are a buffer and a transport.
• Logic in single message transforms. They are hard to test and version; keep them to simple per-record changes.

Learn more¶

• Apache Kafka documentation (design and Kafka Connect sections): https://kafka.apache.org/documentation/
• Apache Kafka 4.0 release announcement: https://kafka.apache.org/blog/2025/03/18/apache-kafka-4.0.0-release-announcement/
• Jay Kreps, The Log: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

Related¶

Change data capture · Delivery guarantees and idempotency · Schema evolution · Writing to Iceberg from Kafka · Stream processing