The 3 AM Brew
Incident notes, design write-ups and the occasional tutorial. Most of them start with a pager going off and end with what I'd do differently.
- Issues
- 11
- Incidents
- 4
- Reading
- 156 min
11 posts shown
2026
- Design
Bounding an Agent’s Authority During Kubernetes Incident Response
Intent classification can choose a specialist, but it cannot grant access. A practical design for routing incident work across clusters while keeping authorization in scoped grants and Kubernetes RBAC.
9 minKubernetesPlatformSecurity - Design
Shell Access That Expires on its Own: Building Just-in-Time kubectl exec
Standing exec permissions in production are a liability. This is the design of a just-in-time replacement (request, approval, namespace lock, broker, scheduled expiry), and the long tail of making expiry recoverable as well as fail-closed: retry semantics, durable notifications, CloudFormation imports and evicted pods.
14 minKubernetesSecurityAWS - Incident
Eight hours for a Ten-Minute Job: EFS Burst Credits, NFS Cache Flags and SQLite
Same image, three regions, one of them eight hours slow. The cause sat in a credit balance, a 4 KiB metering rule and three mount options. The first fix then broke an unrelated upload service, which taught me more about NFS close-to-open consistency than I ever wanted to know.
22 minAWSKubernetesStorage - Incident
79% NXDOMAIN: How ndots:5 Turned One Queue Worker Into a DNS Flood
Three node-local DNS caches were pinned at their CPU limit while CoreDNS looked bored. The cause was a resolver option most people never set, multiplied by 75 replicas on three oversized nodes. A full walk from resolv.conf to packet capture to node-pool design.
23 minKubernetesDNSNetworking
2025
2024
2023
- Design
Logs as a Pipeline, Not a Pile: Running Vector on Kubernetes
Agents that only forward, an aggregator that owns durability, Kafka in the middle, disk buffers on gp3, dead-letter routing for events that fail to parse, and S3 keys designed around the query engine. Plus the three things that bit us: a queue that throttled sends, a metrics label that leaked forever, and two containers sharing one name.
15 minKubernetesObservabilityKafka - Incident
Six-figure Subject Counts: Why Our Schema Registry Fell Over Every Day
Daily 5xx from Schema Registry took down JSON-to-Avro converters and Kafka Connect sinks. The cause was not traffic. It was a registry carrying a few hundred thousand subjects, replaying them on every restart, and a cold-cache stampede whenever Connect rebalanced. What we found, how the registry stores state, and the four fixes.
17 minKafkaDataTroubleshooting
2022
- Design
Exactly 100, Not 101: Distributed Rate Limiting with Envoy and Redis
Per-tenant quotas for internal APIs, enforced across many instances, on VMs and Kubernetes, on x86 and ARM. Why we picked Envoy over Kong, Caddy and a managed WAF, how descriptors really compose, how we proved a 100-per-hour quota lets exactly 100 through, and the sidecar plumbing that made it a one-line opt-in.
16 minNetworkingKubernetesPlatform - Incident
Read-Only File System, But Only on Some Spot Nodes: cloud-init, instance store and fstab
Workers in one autoscaling group kept failing with "Read-only file system". Same AMI, same user data, and only some instances affected. The difference was hardware: the Spot mix included instance types with local disks, and cloud-init rewrote fstab on exactly those. How we found it, how cloud-init turns "ephemeral0" into a device path, what ext4 does when it stops trusting a filesystem (and how that changed in 2024), and how to make the AMI boring again.
18 minAWSStorageTroubleshooting