Eight hours for a Ten-Minute Job: EFS Burst Credits, NFS Cache Flags and SQLite

Same container image, same configuration, three regions. In two of them a nightly import finished in minutes. In the third it took about eight hours, and the pod looked idle the whole time: low CPU, flat memory, nothing in the logs.

This post covers that investigation and the bug the first fix caused. The short version: on a network file system, every consumer needs its own trade-off between consistency and speed, and one mount policy shared by all of them will hurt one of them. The long version needs a few numbers.

The setup

The importer is a single-replica Deployment on EKS. Each run builds a working dataset in a SQLite database under /data, an EFS-backed PersistentVolumeClaim, then pushes results downstream. EFS was chosen years ago because "it survives rescheduling", and nobody revisited the choice.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-shared
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap        # one EFS access point per PVC
  fileSystemId: fs-XXXXXXXX
  directoryPerms: "700"
mountOptions:
  - sync
  - actimeo=0
  - lookupcache=none
reclaimPolicy: Delete

Hold on to those three mount options. Most of this story is in them.

Ruling out the obvious

Same image tag as the fast regions, so the code wasn't the cause. kubectl top showed a few millicores. The tell was the process state:

kubectl exec -n importer deploy/importer -- sh -c '
  for i in $(seq 1 20); do
    for p in /proc/[0-9]*; do
      s=$(awk "{print \$3}" $p/stat 2>/dev/null); [ "$s" = D ] && cat $p/comm
    done; sleep 0.5
  done' | sort | uniq -c

The worker was in D (uninterruptible sleep) in almost every sample. On Linux that nearly always means waiting on I/O, and the only I/O path that mattered here was NFS to EFS. To see which I/O, count syscalls for a short window:

# from a debug container sharing the pod's process namespace
strace -c -f -p <pid> -e trace=%file,%desc -o /tmp/sc.txt & sleep 30; kill %1; sort -k4 -nr /tmp/sc.txt | head

A SQLite-heavy process profiles almost the same way every time: mostly fcntl (locks), single-page pread64/pwrite64, fstat/newfstatat and fsync. Lots of tiny calls, very few bytes. Keep that shape in mind for the metering section.

The kernel keeps a third view, and it's the most direct one. Every NFS mount has per-operation counters in /proc/self/mountstats: how many times each NFS operation was sent, the cumulative round-trip time, and the total time each call spent inside the client. A few lines of awk turn them into averages:

kubectl exec -n importer deploy/importer -- cat /proc/self/mountstats |
awk '$0 ~ " mounted on /data " {m=1; next}
     m && /^device /           {exit}
     m && /per-op statistics/  {p=1; next}
     p && NF >= 9 && $2 > 0    {printf "%-10s %10d ops  rtt %7.2f ms  exec %7.2f ms\n", $1, $2, $8/$2, $9/$2}' |
sort -k2 -nr | head

Each line is one NFS operation type: GETATTR, LOOKUP, LOCK, WRITE and so on. rtt is how long the server took to answer. exec covers the whole call, including any time it sat queued in the client. A throttled file system doesn't return errors; it answers more slowly. So throttling shows up as a climbing rtt while the error column stays at zero. nfsiostat, from nfs-utils, prints the same counters as rates.

How EFS decides how fast you can go

EFS has three throughput modes, and this file system was on the oldest one.

In Bursting, the baseline is 50 KiB/s per GiB in the Standard storage class (50 MiB/s per TiB), with a floor of 1 MiB/s for every file system. Credits accrue whenever you run below baseline and are spent above it. A file system under 1 TiB can bank up to 2.1 TiB of credits, enough for hours of bursting at 100 MiB/s.

Our scratch file system held only tens of GiB. Twenty GiB × 50 KiB/s is exactly the 1 MiB/s floor, so its sustainable rate was 1 MiB/s. Everything above that was borrowed.

The credit bucket, as arithmetic

Bursting is a token bucket measured in bytes. Written down, the incident stops being mysterious. For a file system with S GiB in Standard storage:

baseline       b = max(50 KiB/s × S, 1 MiB/s)       earned all the time
burst ceiling  p = max(100 KiB/s × S, 100 MiB/s)    allowed while the balance is above zero
balance        dB/dt = b − m                         m = metered throughput (reads discounted)
cap            B ≤ 2.1 TiB                           for file systems under 1 TiB

Two consequences follow, and both mattered here.

Spending is fast and earning is slow. At the 100 MiB/s ceiling, a full bucket on a 1 MiB/s file system lasts 2.1 TiB ÷ 99 MiB/s ≈ 6.2 hours. Refilling it from empty while doing nothing at all takes 2.1 TiB ÷ 1 MiB/s ≈ 25 days. A small file system can spend in one long night what it needs most of a month of idleness to earn back.

An empty bucket stays empty under load. With no credits, the permitted throughput drops to the baseline. A job that wants more than the baseline then uses exactly the baseline, so b − m = 0 and the balance doesn't move. Nothing recovers until demand falls below 1 MiB/s, which an importer running for eight hours a night never allowed. The credit metrics further down show exactly that trap.

The metering rule that changes the math

This is the line in the EFS performance documentation that explains the whole incident: every NFS request is accounted as at least 4 KiB of throughput, or its real size if larger. Reads are discounted (metered at up to one-third the rate of writes), but a GETATTR, LOOKUP, LOCK or 1-byte write still costs a full 4 KiB.

Over a ten-hour window, CloudWatch recorded roughly 4.2 million metadata operations on this file system:

4.24 M ops × 4 KiB ≈ 16.2 GiB metered
16.2 GiB ÷ 36,000 s ≈ 0.46 MiB/s

Metadata alone used almost half of the 1 MiB/s floor, before a single byte of real data was read or written. Add SQLite's page I/O and fsync traffic and the file system sat at its permitted ceiling for the whole run.

Two ceilings, and the lower one wins

Permitted throughput is measured in bytes, but SQLite spends it in operations. Dividing one by the other turns the cap into an operations budget:

So with no credits, the whole file system, across every client, can do somewhere between 256 and 768 small operations per second. SQLite's mix of locks, stat calls, lookups and single-page reads and writes lands somewhere in that range. The metadata alone averaged 4.24 M ÷ 36,000 s ≈ 118 operations per second, almost half of the lower figure.

A second ceiling applies even with a full bucket. A process that makes its calls one at a time, as a single SQLite connection does, can't go faster than one call per round trip. AWS quotes best-case latencies of about 1 ms for reads and 2.7 ms for writes on General Purpose file systems, which caps such a process at somewhere between roughly 370 and 1,000 calls per second. With caching switched off, every stat and every lock pays that toll.

For each kind of operation, the no-credit budget sits below what the round trip alone allows. Once the credits ran out, the budget set the runtime. That's the precise sense in which the job was slow but idle: the process wasn't short of CPU, it was waiting for permission to do I/O.

Reading the credit balance

aws cloudwatch get-metric-statistics \
  --namespace AWS/EFS --metric-name BurstCreditBalance \
  --dimensions Name=FileSystemId,Value=fs-XXXXXXXX \
  --start-time 2026-06-01T00:00:00Z --end-time 2026-06-05T00:00:00Z \
  --period 300 --statistics Minimum \
  --query 'length(Datapoints[?Minimum==`0`])'

In the slow region: about 600 five-minute periods at zero, roughly two days straight. The other two regions held ~2 TiB, essentially full buckets, because their importers did less work and their file systems sat idle most of the day. PermittedThroughput in the slow region was flat at 1 MiB/s.

These are the metrics I now put on any EFS dashboard:

• BurstCreditBalance: alert on its slope, not its level. A bucket draining 5% a day will be empty in three weeks, and by the time a threshold alert fires you've usually already lost.
• PermittedThroughput against MeteredIOBytes (as a rate). The gap is your headroom.
• MetadataIOBytes, DataReadIOBytes, DataWriteIOBytes. The ratio of metadata to data tells you whether you have a bytes problem or an operations problem.
• PercentIOLimit on General Purpose file systems, which shows how close you are to the IOPS ceiling.

Why SQLite was the worst possible tenant

SQLite is fast on a local disk because everything it does is a cheap syscall. A single statement can easily cost 5–20 file system operations: take a shared lock, read the header page, check the file size, read B-tree pages, upgrade to a reserved lock, write the rollback journal, fsync, write pages, fsync again, delete or truncate the journal, release locks. On NVMe each of those takes microseconds.

Over NFS each one is a network round trip. Over this NFS mount, each one was also uncached:

With no cache, every one of those small calls goes to the server and is metered at a minimum of 4 KiB. Once credits ran out, each one also queued behind a 1 MiB/s cap. A job that makes a few million tiny calls stops being "a few minutes of CPU" and becomes "a few million network round trips, rate-limited".

You can confirm what a running pod actually mounted. The PV spec isn't proof:

kubectl exec -n importer deploy/importer -- grep ' /data ' /proc/mounts
# ... nfs4 rw,sync,vers=4.1,rsize=1048576,wsize=1048576,...,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,lookupcache=none,...

One write transaction, on the wire

SQLite's documentation walks through a commit in rollback-journal mode step by step. Map each step onto the system calls it makes, and then onto what an uncached NFS mount sends to the server, and a transaction that changes one page looks roughly like this:

That's thirteen round trips for one changed page, before counting anything the fsync calls add. At 256 metered operations per second, this rough model caps the file system at about 20 such commits per second, for all clients combined. Two details make it worse than the table suggests:

• The lock bytes live at 1 GiB. SQLite's POSIX locks sit on a fixed byte range starting at offset 0x40000000, whether or not the database is that big. Every SQLite lock state is a byte-range lock, and on NFSv4 every byte-range lock is a protocol operation, tracked by the server under a lease.
• Every lock is a cache flush. The Linux NFS client treats locking as a coherency point. It writes back dirty data before it takes or releases a lock, and after taking one it throws away its cached attributes and data for that file, unless the server has granted a delegation. EFS never grants delegations. So every transaction starts by discarding whatever the client knew about the database, and its first reads go back over the network.

SQLite on a network file system, in general

Even with good mount options, SQLite on NFS is a compromise its maintainers don't recommend:

• WAL mode is off the table. Write-ahead logging coordinates readers and writers through a small shared-memory index (-shm) that's memory-mapped by every connection. Processes on different hosts can't share memory, so WAL needs every connection on the same machine. Over NFS you're limited to rollback-journal modes (DELETE, TRUNCATE, PERSIST), or to WAL under locking_mode=EXCLUSIVE, which means a single connection.
• Correctness depends on the lock implementation. SQLite uses POSIX advisory locks. NFSv4.1, which EFS speaks, puts locking into the protocol with leases, which is far better than the old NFSv3 sideband lock manager. It's still a distributed lock with network failure modes, so a client that loses its lease can keep running with stale assumptions.
• fsync is expensive by design. On EFS a successful fsync means the data is durable across multiple AZs. That's a great guarantee, and it's paid for in latency on every commit.

If the database is scratch data that's rebuilt every run, there's no reason to pay any of this.

The fix, in two layers

Layer 1: stop the bleeding

We switched the file system to Elastic throughput. That takes effect without remounting, and job times dropped straight away. Choosing between the two non-bursting modes is mostly about the shape of your load:

• Elastic suits spiky work whose average is a small fraction of its peak (AWS's rule of thumb is an average-to-peak ratio of about 5% or less). You pay per byte, metadata included, so a metadata-heavy workload still pays for its chattiness, just without a cap. Small operations are rounded up, too: Elastic meters metadata at a minimum of 4 KiB per operation and data reads and writes at a minimum of 32 KiB each, so a 4 KiB SQLite page write is billed as 32 KiB.
• Provisioned suits steady, predictable throughput. Note the guard rail: after switching to Provisioned or changing the amount, you can't switch back or decrease it for 24 hours.

Elastic was the right emergency lever. It wasn't the fix.

Layer 2: put SQLite where SQLite belongs

The database is rebuilt every run; only the outputs have to last. So:

spec:
  template:
    spec:
      containers:
        - name: importer
          resources:
            requests:
              ephemeral-storage: 10Gi
            limits:
              ephemeral-storage: 25Gi
          volumeMounts:
            - name: scratch
              mountPath: /scratch      # SQLite lives here now
            - name: data
              mountPath: /data         # EFS, for artefacts that must outlive the pod
      volumes:
        - name: scratch
          emptyDir:
            sizeLimit: 20Gi
        - name: data
          persistentVolumeClaim:
            claimName: importer-data

A few details that are easy to get wrong:

• emptyDir without medium: Memory lives on the node's ephemeral storage, usually the root EBS volume, or instance-store NVMe if your node bootstrap places the kubelet directory there. Size the node volume (and its IOPS) for it.
• sizeLimit and ephemeral-storage requests aren't decoration. Exceeding the limit gets the pod evicted, and requests are what the scheduler uses to place it.
• If the scratch data has to survive a pod restart but not a reschedule, a generic ephemeral volume (a per-pod gp3 PVC created and deleted with the pod) is a good middle ground, with IOPS you control.
• Inside SQLite, PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL; PRAGMA temp_store=MEMORY; is safe and fast on local disk. These are the same settings that are unsafe on NFS.

If a database genuinely must be shared, the answer isn't a better mount option. It's a database server.

StorageClass changes don't move bound volumes

parameters and mountOptions on a StorageClass are applied when a volume is provisioned. Editing the class rewrites nothing that already exists, and you can't change a bound PVC's storageClassName. That leaves two honest options:

1. A new, versioned StorageClass and a new PVC, with the workload repointed. The EFS CSI driver's efs-ap mode creates a new access point and an empty directory, so copy the data first, and only with writers stopped. Never cp a live SQLite file; use sqlite3 src.db ".backup dst.db" or VACUUM INTO.
2. Patch the existing PV's spec.mountOptions, then recreate the consumers. Mount options are read at mount time, so running pods keep the old ones until they're replaced, and two pods on the same node may share an existing mount.

One more trap: the importer was at zero replicas when we changed its claim. An HPA whose target is at zero turns itself off (ScalingActive=False, reason ScalingDisabled). Changing the pod template started nothing, and the HPA wasn't going to either. Bringing a workload back from zero is a deliberate step.

The second bug: the same flags were load-bearing elsewhere

With the importer fixed, the obvious next step looked like dropping actimeo=0 and lookupcache=none from the shared EFS mount as well. Several services used it. Less metadata traffic, more credits for everyone.

Within a day, chunked uploads started failing on a separate front-end service. Its replicas all staged uploads in one shared directory on that mount. A client uploads chunks, the load balancer spreads them across replicas, and each replica appends its chunk after checking that the file's current size equals the chunk's offset.

The failure was a race between caches:

t=0.00  replica A  open(), write chunk n, close()       → server size = S + len(n)
t=0.12  replica B  stat(path)  → returns cached size S  ✗
                   chunk n+1 offset = S + len(n) ≠ S    → 416 range mismatch
t=3.5   replica B  stat(path)  → returns S + len(n)     ✓  (attribute cache expired)

Every replica eventually agreed on the right size and the same inode. The data was persisted correctly. Replica B was reading cached attributes.

Close-to-open consistency, precisely

The Linux NFS client doesn't promise that every operation sees the latest server state. It promises close-to-open (CTO) consistency:

• When a client closes a file it wrote, it flushes dirty data to the server.
• When a client opens a file, it revalidates the file's attributes with a GETATTR, and drops cached data if the change attribute moved.
• Between those points, attributes come from the cache for between acregmin and acregmax seconds, 3 to 60 by default for files. The window adapts. The client starts at acregmin, doubles the timeout each time it revalidates and finds the file unchanged, and drops back to acregmin as soon as it sees a change. A file that other replicas keep appending to never earns a long timeout, so its stale window stays near three seconds. That's the gap in the timeline above: wrong at 0.12 seconds, right at 3.5.

So stat() on a path, without opening the file, reads the cache and skips the one place NFS promises freshness. With actimeo=0 every stat happened to go to the server, which hid the application bug. Removing the option exposed it.

NFSv4 does have a way to cache more safely: a delegation, where the server promises to call the client back before anyone else changes the file. EFS doesn't grant them; every OPEN comes back with no delegation. On EFS, close-to-open and the attribute timers are the whole consistency model.

Some tempting explanations don't hold up:

• sync controls when this client's writes reach the server. It doesn't invalidate another client's cache.
• lookupcache governs directory entries, meaning whether a name exists and which inode it maps to. It doesn't affect file sizes. lookupcache=positive is a good middle ground for shared directories: it caches names that exist, but never trusts a cached "not found".
• noac is actimeo=0 plus synchronous writes. It's the right tool when you really need it, but it's very expensive on a metered file system.
• reclaimPolicy: Retain is about deleting volumes. It has nothing to do with coherence.

We put attribute revalidation back on the shared staging mount, verified every replica, and tested a multi-chunk upload deliberately spread across replicas. The proper fix belongs in the application:

// open() is the revalidation point under close-to-open; validate via the descriptor.
func appendChunk(path string, c Chunk) error {
    f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE, 0o600) // never O_TRUNC
    if err != nil {
        return err
    }
    fi, err := f.Stat() // fstat on the open fd: attributes refreshed at open
    if err != nil {
        f.Close()
        return err
    }
    if fi.Size() != c.Offset {
        f.Close()
        return fmt.Errorf("range mismatch: have %d, chunk starts at %d", fi.Size(), c.Offset)
    }
    if _, err := f.WriteAt(c.Data, c.Offset); err != nil {
        f.Close()
        return err
    }
    // On NFS, deferred write errors are reported at close (or fsync). Never ignore them.
    return f.Close()
}

That closes the stale-attribute window, but not the race between two replicas that both pass the check at once. For that you need per-upload serialization: a lock keyed by upload ID (an advisory lock on a sidecar file, or a row in a database), or routing all chunks of one upload to the same replica. Don't fix it with sleeps.

Rollouts mix cache policies

Patching mountOptions doesn't change existing mounts. During a rolling restart, new replicas mount with the new options while survivors keep the old ones, and pods on the same node may reuse an existing mount. One healthy new pod proves nothing. Wait for the rollout to finish, check /proc/mounts in every replica, then run the cross-replica test.

Mount options I'd start from

For a general-purpose EFS mount, these are AWS's recommended baseline (the EFS mount helper and CSI driver mostly set them for you):

nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport

• hard with timeo=600 (60 seconds) and retrans=2: retry instead of returning I/O errors to applications that can't handle them.
• noresvport: reconnect on a new source port. Older kernels reuse the same port, which can block fast reconnection after a network event.
• 1 MiB rsize/wsize: fewer, larger requests, which also means fewer 4 KiB metering minimums.

Then choose the cache policy per consumer, with its own StorageClass:

What I took away

1. Read the throughput-credit metric before profiling code. BurstCreditBalance at zero explains "slow but idle" faster than any flame graph.
2. On EFS, count operations, not bytes. At 4 KiB per request, a few million metadata calls is a real bill and a real ceiling. Divide the permitted throughput by 4 KiB and you have your budget in operations per second.
3. SQLite belongs on local storage. WAL needs shared memory, locks need a reliable lease, and fsync over the network is slow by design.
4. Mount options are part of each consumer's contract. Scratch data wants caching; multi-writer staging needs coherence. Version a StorageClass per pattern.
5. Before removing a mount option, find out who depends on it. The flags we dropped to speed up one service were what kept another one correct.

References

• Amazon EFS performance specifications: throughput modes, burst credits, the 4 KiB per-request accounting and latency figures.
• CloudWatch metrics for Amazon EFS: BurstCreditBalance, PermittedThroughput, MeteredIOBytes and the metadata metrics.
• Understanding Amazon EFS billing and usage reports: metering minimums for Elastic throughput.
• Amazon EFS quotas: unsupported NFSv4 features (including delegations) and lock limits.
• nfs(5): mount options, attribute caching, close-to-open consistency and the sync option.
• Linux NFS client, fs/nfs/file.c: do_setlk, where locking becomes a cache coherency point.
• Atomic commit in SQLite: every step of a rollback-journal commit.
• The SQLite database file format: the lock-byte page, and why WAL needs shared memory.
• Kubernetes ephemeral volumes: emptyDir and generic ephemeral volumes.