All topics
Data
Lakes, warehouses and the cost of moving bytes around. 2 posts
- Incident
Six-figure Subject Counts: Why Our Schema Registry Fell Over Every Day
Daily 5xx from Schema Registry took down JSON-to-Avro converters and Kafka Connect sinks. The cause was not traffic. It was a registry carrying a few hundred thousand subjects, replaying them on every restart, and a cold-cache stampede whenever Connect rebalanced. What we found, how the registry stores state, and the four fixes.
17 minKafkaDataTroubleshooting - Design
Ten Million Objects a Day: Paying The Small-Files Tax on a Kafka-to-S3 Pipeline
Our Kafka backup to S3 wrote small objects, around ten million a day. The bill was driven by request count, not bytes. Here is how we stopped partitioning at write time, got Parquet out of schemaless JSON, compacted hourly inside Athena around its 100-partition limit, and why five other designs lost.
10 minAWSDataStorage