Kafka in a telemetry system
From the event as something already happened to three crash recovery strategies
Moving to events does not make the system faster: it makes it less coupled. The series starts from what Kafka actually guarantees: ordering holds per key, not in absolute terms. Then it goes through the problems that come afterwards: two services that disagree on the format, actors that block a thread, and a consumer restarting after a crash.
An event is not a request: it is a fact that already happened
A synchronous call asks somebody to do something and waits for the answer. An event declares that something happened, and whoever receives it decides on their own what to do with it. The difference sounds philosophical until you look at what changes in production: a slow service stops slowing down the ones calling it, and a service that is down stops taking them down with it.
The price is there, though, and this series pays it in full instead of naming it and moving on. Message ordering holds only per key. Consumer state becomes your problem. Debugging crosses one more component. And two services that disagree on what a message contains break at runtime, not at compile time.
The system these articles come from is a telemetry platform for construction machinery: sensors publishing odometric data and position, three consumers reading the same topic with different responsibilities. A case small enough to fit in a demo and real enough to have already broken something.
What you will learn
- ✓ Understand what Kafka guarantees about ordering, and under which conditions it stops guaranteeing it
- ✓ Version the message format with Avro and a Schema Registry, instead of by convention
- ✓ Migrate from Akka to Pekko knowing where it actually breaks
- ✓ Take blocking I/O out of the actors without losing backpressure
- ✓ Pick the recovery strategy from the nature of the state, not from preference
Articles in the series
- 01Kafka in Practice 1: Anatomy of an Event Stream 12 min
The foundations of Apache Kafka: partition internals, message keys, replication guarantees, and practical examples in Node.js and Python.
→ - 02Schema Registry with Apache Kafka: From Wild JSON to Avro with Apicurio 14 min
Migrating from schemaless JSON to Avro with Apicurio Registry: infrastructure, Node.js producer, Python consumer, schema evolution
→ - 03Akka Is Dead, Long Live Pekko 8 min
A practical guide to migrating from Akka to Apache Pekko in production: complete checklist, real-world gotchas, and lessons from the field.
→ - 04From Blocking Poll to Reactive Streams with Pekko Connectors Kafka 11 min
Refactoring from blocking actors to Source.queue and dedicated consumer threads: practical patterns with Pekko Streams and Kafka for telemetry systems
→ - 05Kafka Crash Recovery: Three Strategies for Three Types of State 8 min
Full replay, checkpoint-and-skip, or no recovery at all: which one fits depends on whether the consumer state is idempotent, additive or absent.
→