AWS
Notes from running production on AWS: EFS, Lambda, CloudFormation and friends. 7 posts
- Design
Shell Access That Expires on its Own: Building Just-in-Time kubectl exec
Standing exec permissions in production are a liability. This is the design of a just-in-time replacement (request, approval, namespace lock, broker, scheduled expiry), and the long tail of making expiry recoverable as well as fail-closed: retry semantics, durable notifications, CloudFormation imports and evicted pods.
14 minKubernetesSecurityAWS - Incident
Eight hours for a Ten-Minute Job: EFS Burst Credits, NFS Cache Flags and SQLite
Same image, three regions, one of them eight hours slow. The cause sat in a credit balance, a 4 KiB metering rule and three mount options. The first fix then broke an unrelated upload service, which taught me more about NFS close-to-open consistency than I ever wanted to know.
22 minAWSKubernetesStorage - Incident
79% NXDOMAIN: How ndots:5 Turned One Queue Worker Into a DNS Flood
Three node-local DNS caches were pinned at their CPU limit while CoreDNS looked bored. The cause was a resolver option most people never set, multiplied by 75 replicas on three oversized nodes. A full walk from resolv.conf to packet capture to node-pool design.
23 minKubernetesDNSNetworking - Design
Logs as a Pipeline, Not a Pile: Running Vector on Kubernetes
Agents that only forward, an aggregator that owns durability, Kafka in the middle, disk buffers on gp3, dead-letter routing for events that fail to parse, and S3 keys designed around the query engine. Plus the three things that bit us: a queue that throttled sends, a metrics label that leaked forever, and two containers sharing one name.
15 minKubernetesObservabilityKafka - Design
Exactly 100, Not 101: Distributed Rate Limiting with Envoy and Redis
Per-tenant quotas for internal APIs, enforced across many instances, on VMs and Kubernetes, on x86 and ARM. Why we picked Envoy over Kong, Caddy and a managed WAF, how descriptors really compose, how we proved a 100-per-hour quota lets exactly 100 through, and the sidecar plumbing that made it a one-line opt-in.
16 minNetworkingKubernetesPlatform - Incident
Read-Only File System, But Only on Some Spot Nodes: cloud-init, instance store and fstab
Workers in one autoscaling group kept failing with "Read-only file system". Same AMI, same user data, and only some instances affected. The difference was hardware: the Spot mix included instance types with local disks, and cloud-init rewrote fstab on exactly those. How we found it, how cloud-init turns "ephemeral0" into a device path, what ext4 does when it stops trusting a filesystem (and how that changed in 2024), and how to make the AMI boring again.
18 minAWSStorageTroubleshooting - Design
Ten Million Objects a Day: Paying The Small-Files Tax on a Kafka-to-S3 Pipeline
Our Kafka backup to S3 wrote small objects, around ten million a day. The bill was driven by request count, not bytes. Here is how we stopped partitioning at write time, got Parquet out of schemaless JSON, compacted hourly inside Athena around its 100-partition limit, and why five other designs lost.
10 minAWSDataStorage