Kubernetes
Controllers, scheduling, storage and the failure modes between them. 6 posts
- Design
Bounding an Agent’s Authority During Kubernetes Incident Response
Intent classification can choose a specialist, but it cannot grant access. A practical design for routing incident work across clusters while keeping authorization in scoped grants and Kubernetes RBAC.
9 minKubernetesPlatformSecurity - Design
Shell Access That Expires on its Own: Building Just-in-Time kubectl exec
Standing exec permissions in production are a liability. This is the design of a just-in-time replacement (request, approval, namespace lock, broker, scheduled expiry), and the long tail of making expiry recoverable as well as fail-closed: retry semantics, durable notifications, CloudFormation imports and evicted pods.
14 minKubernetesSecurityAWS - Incident
Eight hours for a Ten-Minute Job: EFS Burst Credits, NFS Cache Flags and SQLite
Same image, three regions, one of them eight hours slow. The cause sat in a credit balance, a 4 KiB metering rule and three mount options. The first fix then broke an unrelated upload service, which taught me more about NFS close-to-open consistency than I ever wanted to know.
22 minAWSKubernetesStorage - Incident
79% NXDOMAIN: How ndots:5 Turned One Queue Worker Into a DNS Flood
Three node-local DNS caches were pinned at their CPU limit while CoreDNS looked bored. The cause was a resolver option most people never set, multiplied by 75 replicas on three oversized nodes. A full walk from resolv.conf to packet capture to node-pool design.
23 minKubernetesDNSNetworking - Design
Logs as a Pipeline, Not a Pile: Running Vector on Kubernetes
Agents that only forward, an aggregator that owns durability, Kafka in the middle, disk buffers on gp3, dead-letter routing for events that fail to parse, and S3 keys designed around the query engine. Plus the three things that bit us: a queue that throttled sends, a metrics label that leaked forever, and two containers sharing one name.
15 minKubernetesObservabilityKafka - Design
Exactly 100, Not 101: Distributed Rate Limiting with Envoy and Redis
Per-tenant quotas for internal APIs, enforced across many instances, on VMs and Kubernetes, on x86 and ARM. Why we picked Envoy over Kong, Caddy and a managed WAF, how descriptors really compose, how we proved a 100-per-hour quota lets exactly 100 through, and the sidecar plumbing that made it a one-line opt-in.
16 minNetworkingKubernetesPlatform