Troubleshooting
Incident write-ups: symptoms, dead ends, root causes and what changed afterwards. 4 posts
- Incident
Eight hours for a Ten-Minute Job: EFS Burst Credits, NFS Cache Flags and SQLite
Same image, three regions, one of them eight hours slow. The cause sat in a credit balance, a 4 KiB metering rule and three mount options. The first fix then broke an unrelated upload service, which taught me more about NFS close-to-open consistency than I ever wanted to know.
22 minAWSKubernetesStorage - Incident
79% NXDOMAIN: How ndots:5 Turned One Queue Worker Into a DNS Flood
Three node-local DNS caches were pinned at their CPU limit while CoreDNS looked bored. The cause was a resolver option most people never set, multiplied by 75 replicas on three oversized nodes. A full walk from resolv.conf to packet capture to node-pool design.
23 minKubernetesDNSNetworking - Incident
Six-figure Subject Counts: Why Our Schema Registry Fell Over Every Day
Daily 5xx from Schema Registry took down JSON-to-Avro converters and Kafka Connect sinks. The cause was not traffic. It was a registry carrying a few hundred thousand subjects, replaying them on every restart, and a cold-cache stampede whenever Connect rebalanced. What we found, how the registry stores state, and the four fixes.
17 minKafkaDataTroubleshooting - Incident
Read-Only File System, But Only on Some Spot Nodes: cloud-init, instance store and fstab
Workers in one autoscaling group kept failing with "Read-only file system". Same AMI, same user data, and only some instances affected. The difference was hardware: the Spot mix included instance types with local disks, and cloud-init rewrote fstab on exactly those. How we found it, how cloud-init turns "ephemeral0" into a device path, what ext4 does when it stops trusting a filesystem (and how that changed in 2024), and how to make the AMI boring again.
18 minAWSStorageTroubleshooting