Skip to content

Stream processing#

Optional

Continuous computation over unbounded event streams, keeping state between events, so that results such as counts, joins and alerts are updated within seconds of each event.

Before reading

Late and out-of-order data

Why it matters#

Some uses cannot wait for the next batch: alerting during a sale, detecting fraud before an order ships, updating a live dashboard, scoring a session while the user is still on the site. Stream processors make this possible, but they bring state, time and failure handling that batch jobs do not have. Knowing what they involve helps decide when they are worth it.

How it works#

A stream processor runs a job continuously. It reads events from sources such as Kafka topics, transforms them, and writes results to sinks: other topics, tables, serving stores or alerting systems. Stateless steps, such as filtering or converting one event at a time, are simple. The interesting part is state.

State is what the job remembers between events: running counts per product, the open items of each basket, the last event of each session. It lives inside the processor, partitioned by key across workers, often in an embedded store on local disk, with copies saved regularly so that it survives failures. State is what lets a stream job answer questions that span many events.

Windows group events by time so that aggregates have a defined scope. Tumbling windows are fixed and non-overlapping, such as each minute. Sliding windows (HOP windows in Flink SQL) overlap, such as the last ten minutes, updated every minute. Session windows group events separated by less than a gap, such as a user's clicks with no pause longer than 30 minutes. Windows are defined on event time, closed when the watermark passes their end, and can accept late events for an allowed period, as described in Late and out-of-order data.

Joins on streams combine events by key within a time bound: a payment matched to its order if it arrives within an hour. Joining a stream to a table, such as enriching each click with product category, uses a copy of the table kept in the job's state and updated from its own change stream.

Fault tolerance relies on checkpoints. Periodically the processor saves a consistent snapshot of all its state together with the source offsets it has read. After a failure it restores the last checkpoint and replays from those offsets, so every event affects the state exactly once. Results written to the outside are exactly once only if the sink cooperates, through transactions committed with the checkpoint or through idempotent writes.

flowchart LR
  K[Kafka:<br/>clickstream,<br/>orders] --> J
  subgraph J[Stream job]
    direction TB
    ST[(State per session:<br/>basket items,<br/>last activity)] ~~~ W[Session window:<br/>30-minute gap]
  end
  J --> A[Alerts topic:<br/>abandoned baskets]
  J -.->|checkpoint: state<br/>and offsets| CP[(Durable storage)]

Apache Flink is the most widely used open-source stream processor; its 2.0 release (March 2025) added disaggregated state, a state backend that keeps state in remote object storage, to make large-state jobs easier to scale. Spark Structured Streaming runs streams as micro-batches by default. Kafka Streams is a library for building stream processing into Java applications. Several managed services and streaming databases offer SQL over streams.

Stream processing is worth its cost when results must be available within seconds and the logic needs state or time windows. When minutes are enough, micro-batches into tables are simpler. When no state is needed, a plain consumer is enough.

Key terms#

  • Stream processor An engine that runs continuous jobs over event streams.
  • State Data a job keeps between events, partitioned by key.
  • Window A time-based grouping of events: tumbling, sliding or session.
  • Stream-table join Enriching events with the current version of a table held in state.
  • Checkpoint A consistent snapshot of state and source offsets, used to recover.
  • Exactly-once processing Each event affects state once, even after failures.

Example#

The shop alerts its marketing service when a logged-in customer adds items to a basket and leaves without buying.

A Flink SQL job reads the clickstream, keyed by session:

INSERT INTO basket_abandoned_alerts
SELECT session_id, customer_id, window_start, window_end,
       sum(CASE WHEN event_type = 'add_to_basket' THEN 1 ELSE 0 END) AS items_added
FROM TABLE(
  SESSION(TABLE clickstream PARTITION BY session_id, customer_id,
          DESCRIPTOR(event_at), INTERVAL '30' MINUTES))
WHERE customer_id IS NOT NULL
GROUP BY session_id, customer_id, window_start, window_end
HAVING sum(CASE WHEN event_type = 'add_to_basket' THEN 1 ELSE 0 END) > 0
   AND sum(CASE WHEN event_type = 'checkout_completed' THEN 1 ELSE 0 END) = 0;

event_at is declared as the table's event-time attribute, with a watermark; session window functions of this form need Flink 1.19 or later.

When a session has been quiet for 30 minutes, measured by the watermark, the window closes and an alert is emitted if items were added but no checkout completed. The job checkpoints every minute; after a worker restart, it resumes from the last checkpoint without losing or duplicating alerts, because the alerts topic is written transactionally and its consumers read only committed records. The price is latency: alerts become visible only when the checkpoint that contains them completes, up to a minute later.

Pitfalls#

  • Streaming where micro-batches would do. State, checkpoints and late data add real operating cost.
  • Unbounded state. Keys that are never cleaned up, such as sessions without timeouts, grow until the job fails.
  • Changing a job's logic carelessly. Saved state must be compatible with the new version, or the job must start fresh.
  • Exactly-once assumed at the sink. Outside writes need transactions or idempotency.
  • Different definitions in the stream and in batch. Live and reported numbers disagree.

Learn more#

Processing patterns: batch, micro-batch and streaming Kafka and Kafka Connect Low-latency serving stores Model serving and monitoring