Kafka
Topics, connectors, registries and the pipelines built on them. 3 posts
- Design
Logs as a Pipeline, Not a Pile: Running Vector on Kubernetes
Agents that only forward, an aggregator that owns durability, Kafka in the middle, disk buffers on gp3, dead-letter routing for events that fail to parse, and S3 keys designed around the query engine. Plus the three things that bit us: a queue that throttled sends, a metrics label that leaked forever, and two containers sharing one name.
15 minKubernetesObservabilityKafka - Incident
Six-figure Subject Counts: Why Our Schema Registry Fell Over Every Day
Daily 5xx from Schema Registry took down JSON-to-Avro converters and Kafka Connect sinks. The cause was not traffic. It was a registry carrying a few hundred thousand subjects, replaying them on every restart, and a cold-cache stampede whenever Connect rebalanced. What we found, how the registry stores state, and the four fixes.
17 minKafkaDataTroubleshooting - Design
Ten Million Objects a Day: Paying The Small-Files Tax on a Kafka-to-S3 Pipeline
Our Kafka backup to S3 wrote small objects, around ten million a day. The bill was driven by request count, not bytes. Here is how we stopped partitioning at write time, got Parquet out of schemaless JSON, compacted hourly inside Athena around its 100-partition limit, and why five other designs lost.
10 minAWSDataStorage