Context
Industrial telemetry has an unusual shape: enormous write volume, high cardinality, bursty arrival, and consumers with wildly different latency requirements. A control-room alarm needs sub-second delivery. A monthly efficiency report can tolerate hours. Building both on the same path forces the slowest requirement onto the fastest consumer.
The design questions that actually matter
What is the ordering guarantee, and over what scope? Global ordering across a fleet is neither achievable nor necessary. Ordering per asset is both, and it is what the domain requires — a state transition sequence for a single machine must not be reordered. Partitioning by asset identifier makes this a property of the transport rather than something each consumer must reconstruct.
What happens when a consumer falls behind? This is the question that separates a platform from a demo. The answer here is a durable log with a defined retention window: producers are never blocked by slow consumers, and a consumer that falls behind catches up from its own offset. A consumer that stays behind triggers a lag alert long before it hits the retention edge.
How are duplicates handled? Exactly-once delivery across a network is a marketing term. At-least-once delivery plus idempotent consumers is the design that survives contact with reality: every event carries a deterministic key, and writes are idempotent on that key.
Architecture
Edge. Devices publish over MQTT with QoS matched to the message class, and a store-and-forward buffer sized for the expected connectivity gap. Payloads are Protobuf with an explicit schema — self-describing JSON at this volume is a bandwidth and parsing tax paid on every message forever.
Ingest. A stateless gateway authenticates the device, validates against the registered schema version, stamps ingest time alongside device time — the two differ and both matter — and publishes to the log partitioned by asset.
Transport. Kafka as the durable, replayable backbone. Retention long enough that a downstream rebuild does not require going back to devices for data they no longer hold.
Storage. A columnar store for time-series analytics, where the compression ratio on machine telemetry is the difference between an affordable platform and an unaffordable one. Relational storage for asset metadata, schema registry and anything requiring transactional integrity.
Consumers. Independent, each owning its offset: real-time alarm evaluation, continuous aggregation, and batch analytics — decoupled so the alarm path is unaffected by a heavy reporting query.
Schema evolution
Devices in the field cannot be upgraded in lockstep with the platform, so the platform must accept every schema version currently deployed. Compatibility rules are enforced in CI: additive changes only, no field renumbering, no type changes. A schema registry rejects incompatible producers at deploy time rather than at 3am.
Operations
GitOps delivery with the cluster reconciling towards Git. OpenTelemetry instrumentation so a single event can be traced from device to dashboard. SLOs defined on ingest availability and end-to-end alarm latency, with alerting driven by error-budget burn rate.
Capacity is modelled explicitly — message rate, payload size, partition count, retention and replication factor — so growth is a planned procurement rather than an incident.
What this demonstrates
The hard parts of a data platform are not the technology choices; they are the guarantees. Ordering scope, delivery semantics, backpressure behaviour and schema compatibility determine whether the system holds under load. Choosing them deliberately, and writing them down, is the work.