Storage
Persistent volumes, network file systems and why I/O is usually the answer. 3 posts
- Incident
Eight hours for a Ten-Minute Job: EFS Burst Credits, NFS Cache Flags and SQLite
Same image, three regions, one of them eight hours slow. The cause sat in a credit balance, a 4 KiB metering rule and three mount options. The first fix then broke an unrelated upload service, which taught me more about NFS close-to-open consistency than I ever wanted to know.
22 minAWSKubernetesStorage - Incident
Read-Only File System, But Only on Some Spot Nodes: cloud-init, instance store and fstab
Workers in one autoscaling group kept failing with "Read-only file system". Same AMI, same user data, and only some instances affected. The difference was hardware: the Spot mix included instance types with local disks, and cloud-init rewrote fstab on exactly those. How we found it, how cloud-init turns "ephemeral0" into a device path, what ext4 does when it stops trusting a filesystem (and how that changed in 2024), and how to make the AMI boring again.
18 minAWSStorageTroubleshooting - Design
Ten Million Objects a Day: Paying The Small-Files Tax on a Kafka-to-S3 Pipeline
Our Kafka backup to S3 wrote small objects, around ten million a day. The bill was driven by request count, not bytes. Here is how we stopped partitioning at write time, got Parquet out of schemaless JSON, compacted hourly inside Athena around its 100-partition limit, and why five other designs lost.
10 minAWSDataStorage