Skip to content

IoT and device data

Optional

Data from many small devices, such as sensors, scanners and vehicles, sent as frequent small messages over unreliable networks, usually through an MQTT broker before it reaches the platform.

Before reading

Kafka and Kafka Connect

Why it matters

Device data behaves differently from database or application data. Volumes are high, each message is tiny, devices go offline and send batches when they reconnect, clocks drift, and firmware versions differ across the fleet. Pipelines built for business events break on these properties unless they are designed for them.

How it works

Devices rarely talk to Kafka directly. Kafka clients are heavy for small devices, and a Kafka cluster is not meant to be exposed to thousands of devices on the internet. Devices use MQTT instead: a light publish and subscribe protocol designed for constrained devices and poor networks. Each device keeps one connection to an MQTT broker and publishes messages to topics.

MQTT topics are hierarchical strings separated by slashes, such as shop/warehouse-1/scanner-17/stock. Subscribers can use wildcards: + matches one level, # matches all remaining levels. Topic design matters, because it decides what can be subscribed to and secured together: a common pattern is site, then device, then message type.

MQTT offers three quality-of-service levels per message. QoS 0 sends once with no acknowledgement. QoS 1 retries until acknowledged, so duplicates are possible. QoS 2 uses a four-step handshake so each message is delivered once between a client and the broker, at a higher cost. Each hop has its own QoS, and the broker delivers to a subscriber at no more than the QoS that subscriber asked for. A retained message is kept by the broker and sent to new subscribers immediately, which suits "last known state" values. A last will message is published by the broker when a device disconnects unexpectedly. MQTT 5, the current version, adds features such as shared subscriptions, which spread messages for one topic across several consumers.

flowchart LR
  D1[Scanner 17] -->|MQTT, QoS 1| B[MQTT broker]
  D2[Scanner 18] -->|MQTT, QoS 1| B
  D3[Temperature sensor] -->|MQTT, QoS 0| B
  B -->|bridge or connector| K[Kafka topic:<br/>shop.device-readings]
  K --> L[(raw_device_readings)]
  K --> S[Stream processor:<br/>alerts]

A bridge or connector forwards MQTT topics into Kafka, where the rest of the platform takes over. The Kafka key is usually the device ID, so each device's readings stay in order.

Device data needs its own care in storage. Every reading should carry the device ID, the device's own timestamp, a sequence number from the device if it has one, and the firmware version. The device timestamp is event time, but device clocks can be wrong, so the broker or bridge also records when the message was received. Sequence numbers detect gaps and duplicates. Readings are appended and rarely updated; they are queried by device and time range, aggregated into minutes or hours, and kept at full detail only for a limited period. In a lakehouse this usually means partitioning by day of receipt, sorting by device ID and time, and building aggregated tables for long-term analysis.

Key terms

  • MQTT: a light publish and subscribe protocol for devices and unreliable networks.
  • Broker: the server that receives MQTT messages and forwards them to subscribers.
  • Topic hierarchy: slash-separated topic names that can be matched with + and # wildcards.
  • QoS level: MQTT's delivery guarantee per message: 0, 1 or 2.
  • Retained message: the last message on a topic, kept by the broker for new subscribers.
  • Telemetry: periodic measurements sent by devices.

Example

Warehouse scanners send stock counts after each shelf scan. One message on shop/warehouse-1/scanner-17/stock:

{
  "device_id": "scanner-17",
  "seq": 48211,
  "firmware": "2.4.1",
  "read_at": "2026-10-28T14:03:22Z",
  "product_id": "p-332",
  "shelf": "A-14",
  "count": 37
}

Scanner 17 loses Wi-Fi for 40 minutes and then sends 120 queued readings in one burst. Because each reading carries read_at and seq, the platform stores them under the right times, detects that none are missing, and drops the two that the scanner resent after a timeout.

Pitfalls

  • Using the arrival time as the reading time. Bursts after reconnects then compress hours of readings into one moment.
  • No sequence numbers. Gaps and duplicates cannot be told apart from real changes.
  • One topic per device in Kafka. Thousands of devices mean thousands of topics; use one topic keyed by device ID.
  • Keeping raw telemetry forever. Aggregate early and expire detail by a retention policy.
  • Ignoring firmware versions. Message formats change across the fleet gradually, not at once.

Learn more

  • OASIS, MQTT version 5.0 specification: https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html
  • MQTT.org, MQTT overview: https://mqtt.org/

Late and out-of-order data · Stream processing · Partitioning · Delivery guarantees and idempotency