A self-contained streaming platform on a Raspberry Pi — Zigbee sensors, MQTT, Kafka, Spark Structured Streaming, Delta Lake, and Grafana. No cloud required.
A self-contained streaming platform on a Raspberry Pi 400 — Zigbee sensors, MQTT, Kafka, Spark Structured Streaming, Delta Lake, and Grafana, no cloud required. This post covers the architecture and what we are building across the series.
Getting a Raspberry Pi 400 ready to run a full data stack: 64-bit OS Lite, SSH without a screen, system hardening, and Docker Engine. The foundation every subsequent post builds on.
Pairing the SNZB-04P and S60ZBTPF with Zigbee2MQTT, bridging them into Mosquitto, and looking at what MQTT actually guarantees — and doesn't — before this data ever reaches Kafka.
Running Kafka without Zookeeper on a Raspberry Pi 400 — KRaft internals, why partition count matters more than you think, and a Python bridge that moves sensor events from MQTT into a durable, replayable topic.
Add Spark Structured Streaming and Delta Lake on MinIO to the Pi stack. Then open the checkpoint directory and the Delta transaction log and read what is actually inside them — because that is what exactly-once actually means.