Read-Only File System, But Only on Some Spot Nodes: cloud-init, instance store and fstab

The first signal wasn't an alert from infrastructure monitoring. It was the application error tracker: a new group of Read-only file system (EROFS) exceptions, coming from workers in one autoscaling group in one region.

Nothing in that group's configuration had changed recently. Same AMI, same launch template, same user data. And yet some instances had a root filesystem that the kernel had remounted read-only, while their neighbours were fine.

This is the investigation, the mechanics underneath it, and the fix that stopped it coming back.

The fleet

The workers ran in an Auto Scaling group with a mixed instances policy on Spot. We had listed several instance types across families and generations to improve Spot availability and reduce interruptions. That's standard practice, and it's also the root of this story, because a mixed policy means mixed hardware. Spot allocation picks among the listed types based on capacity and price at launch time, so from the AMI's point of view, the hardware an instance gets is close to random.

Some of the listed types are EBS-only. Others come with instance store: SSDs physically attached to the host, fast and free, and gone when the instance stops. The AMI had been built and tested on an EBS-only type.

Finding the pattern

The affected instances had nothing in common in age, Availability Zone or workload. What they did share showed up as soon as we compared instance types: every affected instance had instance store volumes. No EBS-only instance ever went read-only.

Two quick ways to check what a node has:

# Block devices with model and serial. On Nitro instances every disk is NVMe:
# instance store reports the model "Amazon EC2 NVMe Instance Storage", while EBS
# reports "Amazon Elastic Block Store" and uses the volume ID as its serial number.
lsblk -o NAME,MODEL,SERIAL,SIZE,TYPE,FSTYPE,MOUNTPOINT

# Ask the instance metadata service which virtual devices are ephemeral (instance store).
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
          -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
MD=http://169.254.169.254/latest/meta-data/block-device-mapping
for bd in $(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" $MD/); do
  echo "$bd -> $(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" $MD/$bd)"
done | grep ephemeral

On affected nodes the second command listed an ephemeral0 mapping. On healthy nodes it printed nothing.

The fstab nobody wrote

Then we compared /etc/fstab between an affected node and a healthy one. The affected node had an extra line that wasn't in the AMI, mounting the ephemeral disk at /mnt.

The cloud-init log explained where it came from:

grep -n 'cc_mounts\|ephemeral' /var/log/cloud-init.log

cloud-init's mounts module (cc_mounts) runs on every new instance. Its default configuration includes an entry that mounts ephemeral0 at /mnt. On an instance without instance store, the metadata has no ephemeral0, so the module does nothing. On an instance with instance store, it tries to turn ephemeral0 into a device path, and when that works it writes an entry into /etc/fstab and mounts the disk.

So the behaviour of a single AMI depended on the hardware it landed on, and with a mixed Spot policy, that was effectively random. "cloud-init did it" only names the culprit, though. The next two subsections are how it happens.

When the module runs

cloud-init isn't one program that runs once. It's a set of systemd services that run in stages during boot, each with its own list of modules:

cc_mounts is declared to run once per instance. cloud-init records a semaphore under /var/lib/cloud/instances/<instance-id>/sem/ and skips the module on later boots of the same instance. A new instance from the same AMI has a new instance ID, so the module runs again on its first boot, against whatever hardware that launch got. That's why an AMI tested on one instance type proves nothing about another.

How ephemeral0 becomes a device path

ephemeral0 isn't a Linux device. It's a name from the EC2 metadata service's block-device mapping, and the module has to translate it into something mount understands. Reading the cc_mounts module and cloud-init's EC2 datasource, the translation goes like this:

1. Ask the metadata service. block-device-mapping/ephemeral0 usually answers sdb. EBS-only instances have no ephemeral0 key, so the lookup ends here and nothing is written.
2. Try the name as given. Prefix /dev/ and check whether /dev/sdb exists.
3. Try the aliases. If it doesn't, try the same suffix under other driver prefixes: /dev/vdb, /dev/xvdb, /dev/vtbb.
4. Give up quietly. If none of those nodes exists, the default ephemeral0 entry is dropped. This special case exists so that a disk the metadata advertises, but the hardware doesn't present, can't produce a bogus fstab line.
5. Prefer a partition. If the disk has a first partition (a 1, p1 or -part1 suffix), use that instead.
6. Check it's a block device by looking for it under /sys/block, then write the fstab line.

Two things follow from that code.

First, every check is about names, not identity. Nothing verifies that the node it found is actually the instance-store disk. Any block device with the right name is accepted.

Second, which names exist depends on the hardware and the AMI. Xen-based instance types present disks as xvd*. Nitro types present every disk, EBS and instance store alike, as nvme<x>n<y>, numbered in the order the devices answer during boot. Some AMIs add udev rules that create /dev/sdX symlinks for EBS volumes. Same AMI, same user data, different device nodes, different fstab.

For cloud-init releases of that period (22.x on a systemd distribution), a resolved entry looks like this:

/dev/xvdb  /mnt  auto  defaults,nofail,x-systemd.requires=cloud-init.service,_netdev,comment=cloudconfig  0  2

Every field is doing something:

After rewriting the file, the module runs mount -a if a newly configured mount point isn't mounted yet, then systemctl daemon-reload, so systemd's fstab generator turns the line into a mnt.mount unit on later boots.

From an fstab line to a read-only root

On affected nodes, dmesg showed ext4 errors on the root filesystem, followed by the kernel remounting it read-only:

EXT4-fs error (device ...): ...
EXT4-fs (...): Remounting filesystem read-only

That last step is a safety feature, not a bug. The application only sees the result: every write fails with EROFS. It's worth knowing exactly what ext4 does at that moment, because it decides what evidence exists afterwards and how you can detect it.

What ext4 does when it stops trusting a filesystem

ext4 checks its own metadata as it reads it: a checksum that doesn't match, a directory entry that points at a deleted inode, an extent that points past the end of the device. When a check fails, the kernel calls ext4's error handler, and the error policy decides what happens next:

• errors=continue: log the error, mark the filesystem as having errors, and keep going.
• errors=remount-ro: stop all further writes.
• errors=panic: halt the machine.

The policy has two sources. The superblock stores a default (tune2fs -e remount-ro <dev> sets it; dumpe2fs -h shows it as "Errors behavior"), and the errors= mount option overrides it. Many cloud images put errors=remount-ro on the root entry in /etc/fstab, so the mount option is usually what's in force.

With remount-ro, the handler does four things in order:

1. Marks the filesystem as having errors. That state is written to the superblock, so the next fsck runs a full check instead of trusting the clean flag.
2. Aborts the journal, so no further transaction can commit. The kernel logs Aborting journal on device <dev>-8.; the 8 is the inode number of ext4's internal journal.
3. Records the details in the superblock: an error count, plus the time, kernel function, source line, inode and block of the first and the most recent error.
4. Stops writes and logs Remounting filesystem read-only.

Step 3 is the evidence I'd collect first next time. The superblock names which ext4 function noticed the problem, which is how you tell a damaged directory from a bad extent or a checksum failure. The record stays until fsck repairs the filesystem and clears it, so read it before anything runs fsck:

sudo dumpe2fs -h /dev/nvme0n1p1 | grep -iE 'state|error'
#   Filesystem state:      clean with errors
#   FS Error count:        ...
#   First error function:  ...   <- where ext4 noticed the inconsistency
#   Last error function:   ...

# The same fields while mounted, straight from the kernel:
cat /sys/fs/ext4/nvme0n1p1/{errors_count,first_error_func,first_error_line,last_error_func}

"Read-only" stopped meaning ro in 2024

Until late 2024, step 4 meant setting the superblock's read-only flag. /proc/mounts showed ro, and every write failed with EROFS, which is what we saw in 2022. Kernel developers then found that flipping that flag outside the normal remount path, without its locking, could confuse other kernel code such as filesystem freezing. So ext4 stopped doing it, and now refuses writes through an internal flag instead:

Every version still logs Remounting filesystem read-only. That matters for monitoring. An alert that only reads mount flags sees nothing on the middle row, and has to understand emergency_ro on the last one. The kernel log is the one signal that works on all three.

What we observed, and what we could not prove

Observed: every affected instance had instance-store hardware, an extra /mnt entry in /etc/fstab that was absent from the AMI, ext4 errors on the root filesystem, and a read-only remount. No EBS-only instance in the fleet showed the issue. Removing the default ephemeral mount stopped the recurrence.

Not proven: the incident evidence did not establish the exact device-level or kernel mechanism that connected the extra mount to the root filesystem errors. Device naming was a credible risk in this mixed fleet, but we did not capture enough evidence to claim that a particular device alias resolved to the wrong disk or directly corrupted root. The points below explain why we removed hardware-dependent mount configuration; they are not proof of the failure mechanism:

• On Nitro instances, both EBS and instance store volumes are NVMe devices, and the kernel names them in the order they respond during boot. AWS documents that this order isn't guaranteed to match your block device mapping, and that it can change between starts.
• cloud-init's lookup accepts any block device with the expected name. An fstab entry or mount based on a device name (/dev/xvdb, /dev/nvme1n1) can therefore mean different disks on different instance types, or even across reboots of the same instance.
• ephemeral0 → sdb → xvdb → nvmeXn1 is a chain of translations, and every link is an assumption.

Once the extra mount was gone, the problem was gone. The lesson is that disk configuration in an AMI used by a mixed fleet must not depend on the hardware.

The fix

1. Make cloud-init's mounts explicit

We didn't need /mnt at all. The workloads didn't use instance store. So the base AMI now ships a cloud-init config that removes the default ephemeral mount:

# /etc/cloud/cloud.cfg.d/99-no-ephemeral-mount.cfg
# Listing only the device name removes a default mount entry.
mounts:
  - [ ephemeral0 ]

Why a one-element list works is in the same code path. The module only appends its built-in ephemeral0 entry if no user entry already names that device. Ours does, so the default is never added. Missing fields are then filled in from mount_default_fields, whose default mount point is empty, and entries without a mount point are dropped before the file is written. The result is no line at all, on any hardware.

A blunter option is to remove mounts from the module list in cloud.cfg entirely. That works, but it also disables swap and any other mount handling you might want later. The targeted override documents intent better.

If you do want to use instance store, do it on purpose. Identify the disk by a stable identifier: on Nitro, udev already builds one from the model and serial number, /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_<serial>. Format it with a filesystem label, and mount it by label with nofail, so an instance type without local disks still boots cleanly:

LABEL=scratch  /scratch  xfs  defaults,nofail,noatime  0  2

One more trap: instance store is blank again after a stop and start, but the instance ID doesn't change, so per-instance user data won't run a second time. Format the disk from something that runs on every boot (cloud-init's bootcmd, or a small systemd unit ordered before the mount), and make it a no-op when the label already exists.

2. Make the fleet's hardware a decision

A mixed instances policy is a hardware contract. If the AMI assumes EBS-only, say so in the policy. With attribute-based instance type selection, the instance requirements can exclude local storage outright:

"InstanceRequirements": {
  "VCpuCount":   { "Min": 4, "Max": 8 },
  "MemoryMiB":   { "Min": 16384 },
  "LocalStorage": "excluded"
}

With an explicit type list, review it for instance store variants (the d in names like m5d or c6gd). We did both: the AMI now handles any hardware, and the fleet only asks for the hardware it needs.

3. Detect it before users do

Our error tracker found this one, and that was luck. Cheap infrastructure signals catch it directly, as long as they match the kernel you run:

• The kernel log. node-problem-detector ships a readonly-monitor configuration that watches /dev/kmsg for Remounting filesystem read-only and sets a ReadonlyFilesystem node condition. It works on every kernel in the table above. NPD only reports, though: pair it with something that acts on the condition, such as a remediation controller or your autoscaler's node-repair feature, to cordon and replace the node.
• Mount flags. node_exporter exposes node_filesystem_readonly{mountpoint="/"}. It's reliable on older kernels. On 6.15 and later it needs node_exporter 1.12 or newer, which understands emergency_ro. On the kernels in between, it stays at 0.
• A write probe. A node health check that creates and deletes a file on the root filesystem catches all three cases, because it tests what you actually care about.

Takeaways

• One AMI, many instance types, is many configurations. Mixed instances policies are great for Spot availability, but every boot-time component that inspects the hardware (cloud-init, udev rules, NVMe naming) makes the result hardware-dependent.
• Never mount by device name on Nitro. Use labels, UUIDs or /dev/disk/by-id, and nofail for anything that might not exist.
• Read the defaults of what runs at first boot. cloud-init did exactly what its default config says. We just hadn't read it, or the code that resolves ephemeral0.
• errors=remount-ro is your friend; read the superblock before fsck. The read-only flip turned silent corruption into a loud error, and the superblock recorded where ext4 noticed it. Make sure something besides the application notices, and that nothing repairs the evidence before you've read it.
• Mount flags are no longer a reliable read-only signal. Since late 2024, an ext4 filesystem that refuses writes can still show rw. Alert on the kernel log or a write probe.
• Application error trackers are infrastructure monitors too. A new exception class appearing on a subset of hosts is often the first sign that the hosts, not the code, are different.

Reproduce it yourself

Three short labs. The first two need an AWS account and show the hardware dependence. The third runs on any Linux machine and shows ext4's error policy without risking a real disk.

Lab 1: watch cloud-init decide

Launch two instances from the same AMI with the same user data: one type with instance store and its EBS-only sibling (for example m5d.large and m5.large). On each one:

# What the metadata service advertises
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
          -H 'X-aws-ec2-metadata-token-ttl-seconds: 60')
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/block-device-mapping/ephemeral0; echo

# How cc_mounts resolved it (cloud-init.log is written at debug level)
sudo grep -E 'Unable to convert|Mapped metadata name|Remapped device name|Ignoring nonexistent' \
  /var/log/cloud-init.log

# What it wrote, and what the disks really are
grep cloudconfig /etc/fstab
lsblk -o NAME,MODEL,SERIAL,SIZE,MOUNTPOINT

You'll see one of three outcomes. Which one depends on the hardware and the AMI's device naming, not on your configuration:

Lab 2: prove the override

On an instance that got the line:

sudo tee /etc/cloud/cloud.cfg.d/99-no-ephemeral-mount.cfg >/dev/null <<'EOF'
#cloud-config
mounts:
  - [ ephemeral0 ]
EOF
sudo cloud-init schema -c /etc/cloud/cloud.cfg.d/99-no-ephemeral-mount.cfg   # validate
sudo cloud-init single --name mounts --frequency always                     # re-run one module
grep cloudconfig /etc/fstab || echo "no cloud-init mounts in fstab"

Re-running the module rewrites fstab without the marked line. It doesn't unmount anything that's already mounted, so the real test is the next one: bake the file into an AMI and launch the instance-store type again.

Lab 3: watch ext4 give up, safely

This uses a loopback file, so no real disk is at risk. It needs root and e2fsprogs. ext4 has a test hook in sysfs, trigger_fs_error, that raises a filesystem error with whatever message you write to it:

truncate -s 256M /tmp/lab.img && mkfs.ext4 -q -F /tmp/lab.img
sudo mkdir -p /mnt/lab && sudo mount -o loop,errors=remount-ro /tmp/lab.img /mnt/lab
dev=$(basename "$(findmnt -no SOURCE /mnt/lab)")
echo "lab: simulated corruption" | sudo tee /sys/fs/ext4/$dev/trigger_fs_error >/dev/null
sudo dmesg | tail -n 3
sudo touch /mnt/lab/x
findmnt -no OPTIONS /mnt/lab
cat /sys/fs/ext4/$dev/{errors_count,first_error_func}
sudo umount /mnt/lab && sudo dumpe2fs -h /tmp/lab.img 2>/dev/null | grep -iE 'state|behavior|error count|first error function'
sudo e2fsck -fy /tmp/lab.img     # repairs the filesystem and clears the error record

On a 7.0 kernel, everything before the e2fsck step printed:

EXT4-fs error (device loop0): trigger_test_error:130: comm tee: lab: simulated corruption
Aborting journal on device loop0-8.
EXT4-fs (loop0): Remounting filesystem read-only
touch: cannot touch '/mnt/lab/x': Read-only file system
rw,relatime,errors=remount-ro,emergency_ro
1
trigger_test_error
Filesystem state:         clean with errors
Errors behavior:          Continue
FS Error count:           1
First error function:     trigger_test_error

Three things to notice:

• The superblock says Continue, yet writes stopped. The errors=remount-ro mount option overrode the superblock's default.
• The mount options still say rw. On this kernel, emergency_ro is the only hint. Run the same lab on a kernel from before late 2024 and you'll see ro instead.
• The error record names the function that raised the error. Here it's the test hook. In a real incident, that function name is where the investigation starts.

References

• cloud-init boot stages: which stage runs which module list.
• cloud-init mount configuration examples: mount entries and removing a default entry.
• cloud-init cc_mounts source: device-name resolution, defaults and the comment=cloudconfig marker.
• cloud-init CLI reference: single, schema, status and query.
• Amazon EC2 device names: requested device names and device naming behavior.
• Map Amazon EBS volumes to NVMe device names: NVMe enumeration order and stable identifiers.
• Amazon EC2 block-device mappings: mapping concepts and instance storage.
• Linux ext4 documentation: the errors= mount option.
• ext4: don't set SB_RDONLY after filesystem errors and ext4: show 'emergency_ro': the 2024 and 2025 changes to the read-only behavior.
• node-problem-detector readonly-monitor.json: the kernel-log rule for read-only remounts.
• node_exporter: support the ext4 emergency_ro flag: the change that makes node_filesystem_readonly see the new state.