79% NXDOMAIN: How ndots:5 Turned One Queue Worker Into a DNS Flood

The report was short: "CoreDNS and node-local DNS are under heavy load on one of the production clusters, can you check?" Nothing was failing yet. No SERVFAIL spikes, no alerts, no application errors. That made it a good problem: there was time to understand it before it became an incident.

It turned out CoreDNS was fine. Three node-local DNS cache pods were not, and the reason went from a CPU limit, to a resolver option, to node sizing, and finally to an autoscaler query quietly counting the wrong thing. This post walks the whole chain, because each layer hid the next.

The DNS path, end to end

You can't read DNS metrics on Kubernetes without a clear map of the path a query takes. In this cluster it looks like this:

 application (getaddrinfo)
        │  /etc/resolv.conf: nameserver = kube-dns ClusterIP, search ×4, ndots:5
        ▼
 node-local-dns (DaemonSet, hostNetwork)          ← bound to 169.254.20.10 AND the kube-dns ClusterIP
        │
        ├── zone cluster.local, in-addr.arpa, ip6.arpa
        │        cache (success 45s / denial 5s)  ── miss ──► CoreDNS via "kube-dns-upstream" (TCP)
        │
        └── zone "."  (everything else)
                 cache 30s                         ── miss ──► node's upstream resolvers (the VPC resolver)

A few details in that picture matter later:

• node-local-dns binds two addresses. It listens on the link-local 169.254.20.10, and it also binds the kube-dns ClusterIP on a dummy interface, with iptables NOTRACK rules. Pods whose resolv.conf still points at the ClusterIP get intercepted on their own node without any change to their configuration. It also means the traffic skips conntrack, which removes a whole class of UDP races (more on that below).
• cluster.local misses go to CoreDNS over TCP (force_tcp). That's intentional: TCP connections clean up their conntrack state when they close, and they avoid the dropped-UDP tail latency that the NodeLocal DNSCache design calls out. The cost is CPU on both ends for connection handling.
• Everything outside the cluster domain never touches CoreDNS. The . server block forwards straight to the node's upstream resolvers. On AWS that's the VPC's built-in resolver.
• Negative answers for cluster.local are cached for only 5 seconds. The denial 9984 5 line caps NXDOMAIN caching at five seconds, no matter what TTL the authority suggests.

Symptom triage: CPU, not errors

kubectl top across kube-system told the first story:

Four node-local-dns pods were running at 53–99% of a 150m CPU limit. On Linux, a CPU limit is enforced by the CFS bandwidth controller: 150m means 15 ms of CPU time in every 100 ms period. When a container uses its quota early in a period, all of its threads are paused until the next period starts. For a DNS server that means queries sit in the socket receive buffer, latency goes up in 100 ms steps, and if the buffer overflows, UDP packets are dropped without any log line. Nothing on the server side retries a dropped query. The client waits out its resolver timeout, five seconds by default in glibc, and then asks again.

The kubelet exports this through cAdvisor:

sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="kube-system", container="node-cache"}[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="kube-system", container="node-cache"}[5m]))

A throttled-period ratio above a few percent on a DNS cache is a reliability problem, even if every query eventually gets an answer.

Mapping the hot pods to nodes showed what they had in common. They were the nodes running replicas of a single high-concurrency queue worker that calls cloud provider APIs across many regions. That workload had 75 replicas, and the node autoscaler had packed them onto three large nodes, 30, 23 and 22 per node. One node-local cache per node meant one 150m budget absorbing the DNS traffic of 30 busy Python processes.

Measuring QPS without a dashboard

The cluster's metrics pipeline had the CoreDNS series, but I wanted per-pod numbers I could trust, sampled right now. Both CoreDNS and node-local-dns expose Prometheus metrics, and the API server can proxy to a pod port, so two snapshots a few seconds apart give you rates without any extra tooling:

pod=node-local-dns-xxxxx
snap() {
  kubectl get --raw "/api/v1/namespaces/kube-system/pods/${pod}:9253/proxy/metrics" |
  awk '
    /^coredns_dns_requests_total\{/   { req  += $NF }
    /^coredns_cache_hits_total\{/     { hit  += $NF }
    /^coredns_cache_misses_total\{/   { miss += $NF }
    /^coredns_dns_responses_total\{/  { match($0, /rcode="[A-Z]+"/); rc[substr($0, RSTART+7, RLENGTH-8)] += $NF }
    END { printf "req %d hit %d miss %d", req, hit, miss; for (k in rc) printf " %s %d", k, rc[k]; print "" }'
}
a=$(snap); sleep 10; b=$(snap)
paste <(echo "$a" | tr ' ' '\n') <(echo "$b" | tr ' ' '\n') |
  awk 'NR%2==1 {k=$1} NR%2==0 {printf "%-10s %8.1f/s\n", k, ($2-$1)/10}'

Across the hot node caches:

• 450–1,688 QPS per pod, with a 77–83% cache hit ratio
• Combined, the three busiest handled about 6,000 QPS
• NXDOMAIN: ~4,770 QPS. NOERROR: ~1,260 QPS. That's 79% of all responses saying "this name does not exist"
• SERVFAIL: zero

CoreDNS behind them received only ~1,300 QPS, and 94% of that was NXDOMAIN.

Two things stood out. First, a healthy service shouldn't spend four-fifths of its DNS answers on names that don't exist. Second, a cache with a ~80% hit ratio was still forwarding a steady stream of negative answers upstream. That's the 5-second denial cap at work: every distinct (name, type) pair that misses can be re-asked after five seconds, on every node, so a large set of distinct names keeps CoreDNS busy even with caching in place. Put a number on it: if a node's workloads keep asking for D distinct failing pairs, each at least once every five seconds, that node's cache forwards about D ÷ 5 queries per second upstream for them, however good its hit ratio looks. The cap sets the floor.

Splitting the responses by zone gave the shape of the problem. The cluster.local zone and the root zone were answering at almost exactly a 3:2 ratio. Keep that number in mind; it falls straight out of the resolver configuration.

ndots, precisely

Every pod with the default dnsPolicy: ClusterFirst gets a resolv.conf similar to this one. The search list has the pod's namespace, the two generic cluster suffixes and, on EKS, the node's VPC domain:

nameserver 172.20.0.10
search <namespace>.svc.cluster.local svc.cluster.local cluster.local <region>.compute.internal
options ndots:5

The resolver rule is short:

• A name containing a trailing dot is absolute. It's queried exactly once, as written.
• A name with at least ndots dots is tried as written first, then with the search suffixes if that fails.
• A name with fewer than ndots dots is tried with every search suffix first, and as written only at the end.

glibc's own default is ndots:1, and it silently caps the value at 15. Kubernetes ships ndots:5 so that a name like db.payments.svc (two dots) resolves through the search list. The cost lands on external names. sqs.us-east-1.amazonaws.com has three dots, which is fewer than five, so a single lookup becomes:

sqs.us-east-1.amazonaws.com.<namespace>.svc.cluster.local   → NXDOMAIN   (cluster.local zone)
sqs.us-east-1.amazonaws.com.svc.cluster.local               → NXDOMAIN   (cluster.local zone)
sqs.us-east-1.amazonaws.com.cluster.local                   → NXDOMAIN   (cluster.local zone)
sqs.us-east-1.amazonaws.com.<region>.compute.internal       → NXDOMAIN   (root zone → VPC resolver)
sqs.us-east-1.amazonaws.com                                 → answer     (root zone → VPC resolver)

Three wasted queries land in cluster.local, one wasted query plus the real one land in the root zone. That's the 3:2 ratio. It was the fingerprint that confirmed the diagnosis before any packet capture.

Now add record types. Most resolvers ask for both A and AAAA, even in IPv4-only VPCs, because the application calls getaddrinfo with AF_UNSPEC. Each candidate name is asked twice:

queries per logical lookup = (search domains + 1) × record types = (4 + 1) × 2 = 10

Ten packets to learn one address, eight of which are guaranteed to fail. There's an even less obvious case. Under ndots:5, a fully qualified service name such as api.payments.svc.cluster.local (four dots) also goes through the search list first, unless it's written with the trailing dot. Being explicit doesn't help you unless you're explicit all the way to the root.

That arithmetic also explains the headline. Eight failures in ten queries is 80% NXDOMAIN, and the node caches measured 79%. The missing percent is ordinary in-cluster traffic: a Service name resolves within its first two candidates, so it adds successful answers and pulls the share down slightly. Split the same ten queries by zone and you get six in cluster.local, all NXDOMAIN, and four in the root zone, two NXDOMAIN and two real answers. That's the 3:2. The 79% wasn't a property of the traffic. It was the resolver configuration, written as a percentage.

Watching the expansion

You don't need a cluster to see this. The script below runs glibc's resolver in a container with the same search list. A small dnsmasq plays the cluster DNS: it answers NXDOMAIN for the cluster and VPC suffixes and forwards everything else. Then it resolves one AWS hostname the way CPython does and records every query on the wire:

cat > ndots-lab.sh <<'EOF'
apt-get update -qq && apt-get install -y -qq tcpdump dnsmasq-base python3-minimal >/dev/null
up=$(awk '/^nameserver/ {print $2; exit}' /etc/resolv.conf)
# Stand-in for cluster DNS: NXDOMAIN for the cluster and VPC suffixes, forward the rest.
dnsmasq --no-resolv --no-hosts --listen-address=127.0.0.1 --bind-interfaces \
        --server="$up" --local=/cluster.local/ --local=/compute.internal/
printf '%s\n' 'nameserver 127.0.0.1' \
  'search demo.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal' \
  "options ${NDOTS:-ndots:5}" > /etc/resolv.conf
tcpdump -n -l -i lo 'udp dst port 53' > /tmp/queries.txt 2>/dev/null &
sleep 1
python3 -c 'import socket; socket.getaddrinfo("sqs.us-west-2.amazonaws.com", 443)'
sleep 1; kill %1; sleep 0.5
cut -d' ' -f3- /tmp/queries.txt
EOF
docker run --rm --cap-add NET_RAW -v "$PWD/ndots-lab.sh:/lab.sh:ro" ubuntu:24.04 bash /lab.sh
docker run --rm --cap-add NET_RAW -e NDOTS=ndots:2 -v "$PWD/ndots-lab.sh:/lab.sh:ro" ubuntu:24.04 bash /lab.sh

With the default ndots:5 (glibc 2.39), one lookup sent ten queries:

127.0.0.1.42221 > 127.0.0.1.53: 59745+ A? sqs.us-west-2.amazonaws.com.demo.svc.cluster.local. (68)
127.0.0.1.42221 > 127.0.0.1.53: 40805+ AAAA? sqs.us-west-2.amazonaws.com.demo.svc.cluster.local. (68)
127.0.0.1.60115 > 127.0.0.1.53: 15253+ A? sqs.us-west-2.amazonaws.com.svc.cluster.local. (63)
127.0.0.1.60115 > 127.0.0.1.53: 5780+ AAAA? sqs.us-west-2.amazonaws.com.svc.cluster.local. (63)
127.0.0.1.58757 > 127.0.0.1.53: 56471+ A? sqs.us-west-2.amazonaws.com.cluster.local. (59)
127.0.0.1.58757 > 127.0.0.1.53: 61332+ AAAA? sqs.us-west-2.amazonaws.com.cluster.local. (59)
127.0.0.1.46918 > 127.0.0.1.53: 58779+ A? sqs.us-west-2.amazonaws.com.us-west-2.compute.internal. (72)
127.0.0.1.46918 > 127.0.0.1.53: 49306+ AAAA? sqs.us-west-2.amazonaws.com.us-west-2.compute.internal. (72)
127.0.0.1.59469 > 127.0.0.1.53: 41330+ A? sqs.us-west-2.amazonaws.com. (45)
127.0.0.1.59469 > 127.0.0.1.53: 39539+ AAAA? sqs.us-west-2.amazonaws.com. (45)

With ndots:2, the same lookup sent two:

127.0.0.1.56151 > 127.0.0.1.53: 18693+ A? sqs.us-west-2.amazonaws.com. (45)
127.0.0.1.56151 > 127.0.0.1.53: 11527+ AAAA? sqs.us-west-2.amazonaws.com. (45)

Two details are worth noticing. Every candidate costs an A and an AAAA query, because CPython calls getaddrinfo with AF_UNSPEC and without the AI_ADDRCONFIG flag. Tools that do set that flag, such as getent ahosts, skip AAAA on a host with no IPv6 address, so they under-count what your application sends. And each A/AAAA pair leaves from the same source port, back to back. That's the parallel, same-socket pattern behind the conntrack race described next.

Resolver behaviour differs by runtime

"The resolver" is really a family of implementations, and they don't all behave the same way:

• glibc walks the search list in order and, by default, sends the A and AAAA queries in parallel from one UDP socket. That pattern used to trigger a kernel conntrack race in which one of the two packets was silently dropped, which shows up as the famous 5-second DNS delay on Kubernetes. The single-request-reopen and single-request options exist to work around it. NodeLocal DNSCache sidesteps it by keeping DNS traffic out of conntrack.
• musl (Alpine) sends queries to all configured nameservers in parallel and takes the first answer. It handles search domains, but it doesn't support glibc's single-request* options.
• Go's pure resolver reads resolv.conf, respects ndots and the search list, and runs A/AAAA lookups concurrently. Go programs that use cgo for DNS go through the system libc instead.
• The JVM keeps its own positive and negative caches (networkaddress.cache.ttl, networkaddress.cache.negative.ttl). That reduces the query rate, but not the number of distinct names asked the first time.
• CPython calls getaddrinfo for every new connection and has no process-level DNS cache. Our worker was Python. Its cloud SDK opens connections to many regional endpoints, such as ec2.<region>, sts.<region>, monitoring.<region> and per-bucket S3 hostnames. A large set of distinct names, each resolved over and over across 75 processes, is the worst case for both amplification and cache efficiency.

The race, precisely. The A and AAAA packets of a pair share a 5-tuple (same source port, same destination), so the kernel's connection tracking needs one entry for both. When two CPUs process them at the same moment, each builds an entry and only one can be inserted. On older kernels the loser's packet could be dropped. The only trace was the insert_failed counter in conntrack -S, and the client sat out its five-second resolver timeout before retrying. Kernel fixes narrowed the race over several releases, and Linux 5.6 closed the case that hit Kubernetes hardest: two packets of one pair DNATed to different CoreDNS pods behind the Service IP. NodeLocal DNSCache avoids the question entirely, because its NOTRACK rules keep DNS traffic out of conntrack.

Getting the actual query names

Response codes told us how much. To fix the right thing I needed which names. The existing telemetry recorded rcode, zone and query type, but not the query name, and neither CoreDNS nor node-local-dns had query logging on.

The options, roughly from most to least invasive:

1. Turn on the CoreDNS log plugin. This changes production DNS configuration and produces a lot of log volume at 6k QPS. And it would only show the CoreDNS side, not the node caches.
2. dnstap, the structured DNS tap. It's lovely, but it needs a collector and a config change.
3. eBPF tracing (Inspektor Gadget's trace dns and similar tools). That would have been ideal if the tooling had already been installed in the cluster.
4. A short, scoped packet capture on just the hot nodes.

I chose the fourth: a 30-second, NXDOMAIN-only capture on the three hot nodes, using hostNetwork pods with tshark that were deleted straight after aggregation. The filter keeps only DNS responses with rcode 3 and extracts just the fields needed:

apiVersion: v1
kind: Pod
metadata:
  name: dns-nxdomain-capture-a
  namespace: kube-system
spec:
  hostNetwork: true
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: <hot-node>
  tolerations:                       # mirror the workload's dedicated-pool taint
    - key: <team-taint-key>
      operator: Equal
      value: <team-taint-value>
      effect: NoSchedule
  containers:
    - name: capture
      image: nicolaka/netshoot:v0.13
      command: ["/bin/sh", "-c"]
      args:
        - >-
          tshark -Q -n -i any -a duration:30 -f 'port 53'
          -Y 'dns.flags.response == 1 && dns.flags.rcode == 3'
          -T fields -e dns.qry.name -e dns.qry.type -E separator=,
      securityContext:
        capabilities:
          add: ["NET_ADMIN", "NET_RAW"]
      resources:
        requests: { cpu: 1m, memory: 64Mi }    # the hot nodes were fully allocated
        limits:   { cpu: 500m, memory: 512Mi }

Three practical lessons from running it:

• Pin with a hostname nodeSelector, not nodeName. nodeName bypasses the scheduler, so taints and resource fit aren't evaluated, and you get confusing kubelet-side rejections. The selector keeps the scheduler involved.
• Requests decide placement, limits don't. The third pod stayed Pending because its node was fully allocated by requests, even though the node had idle CPU. A 1m request with a sane limit got it scheduled.
• Nodes vanish mid-diagnosis. One target node was scaled away while its capture pod was scheduling. The capture script had a trap that deleted every diagnostic pod on any failure, so nothing privileged was left behind. Write the cleanup before the capture.

What the capture said

A clean 20-second sample across the three nodes recorded 75,958 NXDOMAIN responses, about 3,800 per second. Collapsing the search-suffix variants back to their base hostname gave the ranking:

The query types split 50.08% A and 49.92% AAAA, exactly the two-records-per-candidate pattern. None of these hostnames were wrong; every one of them eventually resolved. They were simply being asked four times in the wrong place first.

The second consumer you don't see: the VPC resolver

Look again at the path diagram: the <region>.compute.internal candidate and the real name both leave the node and go to the VPC's built-in resolver. On AWS that resolver has a hard limit: 1,024 packets per second per network interface for traffic to link-local services, including DNS. Packets over that rate are dropped, not queued, the limit can't be raised, and the only signal is a counter on the instance:

ethtool -S eth0 | grep linklocal_allowance_exceeded

node-local-dns's 30-second cache in the root zone protects you from most of this. But ndots amplification doubles the root-zone traffic for every external name, and packing 30 replicas onto one node concentrates it on a single ENI. If you run dense, DNS-heavy nodes on AWS, export that counter (the CloudWatch agent can ship it) before you need it.

Two details make that budget tighter than it looks. It's shared: the same per-interface allowance covers the Instance Metadata Service and the Amazon Time Sync Service, and the counter can't tell you which of the three lost packets. And it's counted in packets sent, so the failing compute.internal candidate spends from it exactly as much as the lookup that succeeds. It doesn't apply to DNS resolvers you run yourself, which is one way out if a single node really needs more than 1,024 packets per second of DNS.

The side effect: autoscaling on the wrong metric

There was one more puzzle. CoreDNS was running 12 replicas while serving ~1,300 QPS, roughly 100 QPS per replica. Its KEDA ScaledObject explained why:

triggers:
  - type: prometheus
    metadata:
      query: sum(rate(coredns_dns_requests_total{namespace="kube-system"}[5m]))
      threshold: "500"

node-local-dns is built on CoreDNS, exports the same metric names, and runs in the same namespace. The query summed node-cache traffic (~6,000 QPS) together with CoreDNS traffic. With a target of 500 per replica, 6000 / 500 = 12 replicas, which matched exactly. The autoscaler was doing its job on the wrong input: it scaled CoreDNS for traffic that, by design, never reaches CoreDNS.

      query: sum(rate(coredns_dns_requests_total{namespace="kube-system", pod=~"coredns-.*"}[5m]))

Scoping by pod (or by scrape job) fixes it. Any metric shared between a component and a sidecar or DaemonSet built from the same codebase deserves the same check.

The fix, layer by layer

1. Stop generating the queries: ndots: 2

The setting belongs to the workload's pod template, not to CoreDNS or node-local-dns. dnsPolicy already defaults to ClusterFirst for ordinary pods, so only dnsConfig is needed:

spec:
  template:
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "2"

Why 2 and not 1:

With ndots:1, the very common service.namespace form would be sent to the internet root first: one extra NXDOMAIN through the VPC resolver for every cross-namespace call. 2 keeps both Kubernetes short names and external names on their fast path. For a below-threshold external lookup, the cost drops from 10 queries to 2.

Two notes. The change only reaches pods created after the rollout, because resolv.conf is written when the sandbox starts. And the alternative people reach for, writing external names with a trailing dot (sqs.us-east-1.amazonaws.com.), works for DNS but leaks into HTTP Host headers and TLS SNI in some clients, where the trailing dot can break certificate matching or virtual hosting. A resolver option is the cleaner lever. CoreDNS's autopath plugin, which answers search-path expansions on the server side, is another option, but it needs pods verified mode, which watches every pod in the cluster. That's more memory and complexity than this problem justified.

There's a third lever for IPv4-only clusters, with a caveat. glibc 2.36 added options no-aaaa, which stops the stub resolver from sending AAAA queries at all. In the lab above it halves the count again, from ten queries to five. But glibc's own man page describes it as a diagnostic option, it only affects glibc-based programs, and an early version of that code path shipped with a security bug (CVE-2023-4527). Use it to measure how much of your load is AAAA, not as the fix.

2. Give the cache room to breathe

A 150m CPU limit on a component that every pod on the node depends on is a single point of throttling. We raised the request to a realistic steady-state value and lifted the limit well above observed peaks. A DNS cache that gets throttled turns a load problem into a latency problem for every workload on its node.

3. Stop concentrating the load: smaller nodes, spread evenly

The workload wasn't wrong to make DNS queries; it was wrong to make them from 30 processes behind one cache. Spreading replicas across more nodes spreads queries across more node-local caches, and more ENIs, without anyone having to tune anything per node:

spec:
  strategy:
    rollingUpdate:
      maxSurge: 10
      maxUnavailable: 0          # wait for new capacity instead of dropping workers
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: queue-executor

Topology spread balances across nodes that exist; it doesn't make the autoscaler create more of them. While I was preparing this change, the eligible pool was five 8-vCPU nodes and one 64-vCPU node holding 45 of the 75 replicas. The real lever was the node pool definition.

Our node pools are Crossplane claims rendered into the node autoscaler's launch specifications. The provider supported dynamic filters (maxVcpu, maxMemoryGib), but the parent cluster already defined a cluster-level instance filter, and the autoscaler's API rejects pool-level filters when cluster-level filters exist. The safe alternative the composition already exposed was an explicit instance-type allowlist, which needs no platform-wide schema change.

Choosing that list was its own small exercise:

• Size cap: exactly 8 vCPU and at most 64 GiB, so no node can host more than a handful of replicas.
• Latest generation first: c8i, m8i and r8i .2xlarge were the only generation-8 types offered in every AZ of all three production regions. m8a.2xlarge and c8a.2xlarge were rejected because they weren't available in one region, and a per-region claim would mean configuration drift.
• Spot diversification: three types is a thin Spot pool, so generation 7 (c7i, m7i, r7i), m7a.2xlarge and c8i-flex.2xlarge were added as fallbacks. A requested c8i-flex.4xlarge was swapped for the .2xlarge because it has 16 vCPU.
• Check before you commit: EC2's Spot placement score API rates how likely a set of instance types is to fulfil a target capacity in a region. The 8-type pool scored 9/10 in every region for ~13 nodes. A narrower two-type pool scored 3/10. Catalog availability isn't Spot availability.
• Architecture: the worker image is amd64-only, and the claim also rendered an empty arm64 pool, so we disabled it. An unused pool with no size constraints is a way for big nodes to come back.

Before migrating anything, a throwaway Deployment with the pool's selector and toleration and a 1 CPU / 2 GiB request confirmed the path. The autoscaler provisioned an m7i.2xlarge Spot node (≈7.9 allocatable vCPU, ≈29.8 GiB) with the expected taint, the pod became Ready, and the test was deleted.

4. Fix the autoscaler's input

The CoreDNS ScaledObject now counts CoreDNS only. Once node-cache traffic is excluded, the replica count reflects the load CoreDNS actually serves, and the threshold can be tuned against real per-replica capacity.

5. Clean up what the investigation found

One node had been NotReady/Unknown for weeks, leaving terminating pods and a permanently unavailable DaemonSet slot. It wasn't related to DNS, but a stale node skews every per-node average you compute. It went on the hygiene list.

Verifying the change

Because resolv.conf is fixed at pod creation, check the new pods, not the Deployment:

kubectl exec deploy/queue-executor -- cat /etc/resolv.conf | grep options     # expect ndots:2

Then repeat the same two-snapshot measurement on the node caches. The signals that should move:

• NXDOMAIN share should collapse. External lookups no longer produce search-path misses, so most remaining NXDOMAINs should be names that genuinely don't exist.
• The 3:2 zone ratio should disappear, because its existence is the search-path expansion.
• Throttled periods on node-local-dns should drop to near zero.
• CoreDNS replicas should settle to what its own traffic needs.

What I'd tell the next person

1. Read the DNS path before the DNS metrics. Node caches, upstream TCP, per-zone server blocks and negative-cache caps all change what a counter means.
2. Ratios are fingerprints. A clean 3:2 split between cluster.local and root-zone NXDOMAINs points straight at the search list.
3. ndots:5 costs you on every external name. For any workload that mostly calls things outside the cluster, set ndots: 2 in its pod template. It's one of the cheapest reliability wins in Kubernetes.
4. Never CPU-limit a per-node critical path too tightly. A throttled DNS cache becomes everyone's latency problem.
5. Bin packing concentrates side effects. Big nodes save money until 30 replicas share one cache, one ENI and one packet budget.
6. Audit autoscaler queries for metric-name collisions. Same codebase, same metric name, same namespace: a sum that looks correct isn't.

References

• resolv.conf(5): search, ndots, timeout, attempts, single-request and no-aaaa.
• DNS for Services and Pods: dnsPolicy, dnsConfig and the resolv.conf a pod gets.
• Using NodeLocal DNSCache in Kubernetes clusters: the node cache's addresses, NOTRACK rules and TCP upstream.
• CoreDNS cache plugin and autopath plugin: success and denial caps, and server-side search paths.
• CFS bandwidth control: how CPU limits turn into throttled periods.
• netfilter: conntrack: allow insertion of clashing entries: the Linux 5.6 change for the DNAT clash.
• Amazon VPC quotas: 1,024 packets per second per network interface to the Route 53 Resolver.
• Monitor network performance for ENA: linklocal_allowance_exceeded and the other allowance counters.
• Spot placement score: checking a Spot type list before you commit to it.