Six-figure Subject Counts: Why Our Schema Registry Fell Over Every Day

Almost every day, Schema Registry returned 5xx for a while. When it did, two things broke downstream:

• JSON-to-Avro converters couldn't look up schemas, crashed and restarted.
• Kafka Connect S3 sinks that turn Avro into Parquet failed their tasks, and consumer lag climbed until the alerts went off.

The registry instances looked idle most of the day: low CPU and few requests. Then they spiked, threw errors, sometimes ran out of memory and took minutes to come back. This is the investigation and what we changed.

How Schema Registry keeps its state

You need this model to understand the failure. Schema Registry is a stateless-looking HTTP service with a Kafka topic as its database:

                ┌───────────────────────────── _schemas (1 partition, compacted) ─────────────────────────────┐
 register ──►  leader ──► append record ──►  │ key: {subject, version}  value: {schema, id, deleted?} ... │
                                              └──────────────────────────────────────────────────────────────┘
                                                        ▲ every instance consumes from offset 0 on startup
 lookups ──►  any instance ──► in-memory indexes (subject → versions, id → schema, schema → id)

• Writes (registering a new schema or version) go to the leader, which appends a record to the _schemas topic. Followers forward writes to it.
• Reads are served by any instance from in-memory indexes built by consuming _schemas.
• On startup, an instance consumes _schemas from the beginning and only starts serving once it has caught up. The topic is compacted, so it holds roughly one record per subject version, but it still holds every subject version.

Two consequences follow. Startup time grows with the number of schema versions, and heap grows with them too, because all the indexes live in memory.

What we found

1. Far too many subjects

Our largest region had several hundred thousand subjects, and the count kept growing. A widely quoted upper bound for even very large, data-heavy companies is on the order of ten thousand unique schemas. We were well over an order of magnitude past that.

The reason was structural. The pipeline registered:

• one subject per tenant for the "users" pipeline, and
• one subject per event type per tenant for the "events" pipeline.

Tenants create event types freely, so subjects grew with customer behaviour, not with our data model. About 98% of the subjects came from events. Most hadn't been used in weeks.

2. Slow restarts became outages

With that many records in _schemas, a restarted instance needed minutes to replay the topic before it could serve. (Rolling deploys of registries with far fewer schemas already need kafkastore.init.timeout.ms raised above its 60-second default.) During that window the cluster ran with fewer instances, so the remaining ones took all the load.

3. Heap pressure and OOMs

The indexes for hundreds of thousands of subjects, many with dozens of versions, needed a large heap. Instances sized for "a small metadata service" ran out of memory under bursty load. An OOM kill triggered a restart, the restart triggered a long replay, and the replay took the instance out of rotation at exactly the wrong moment.

4. A cold-cache stampede from Connect

Converters cache schemas in-process. When a Kafka Connect worker restarts or rebalances, every task starts with an empty cache, and the first record of every event type triggers a registry call.

connect rebalance
  └─ N tasks restart, caches empty
       └─ each task sees records for hundreds of event types
            └─ each first sighting → HTTP request to the registry
                 └─ thousands of near-simultaneous lookups → CPU spike → 5xx → task fails → rebalance

Request metrics showed it clearly: long flat stretches, then sharp peaks of POST /subjects/{subject} ("is this schema registered under this subject, and as which version?") and GET /subjects calls, with get-schema latency climbing during each peak. The POSTs come from the Avro converter itself. It stamps the registry's version number onto every schema it hands to Connect, so the first time a task sees a schema, the converter fetches it by ID and then asks the registry which version it is under its subject.

Partitioning made it worse. Events of one type were spread across many partitions, so many different tasks each needed the same schema, and each of them asked. Total lookups after a rebalance scaled with tasks × event types per task, not with event types.

5. A shared fate

One registry cluster served both pipelines. The users pipeline had a tiny, stable set of subjects (about 2% of the total), yet it went down whenever the events pipeline's churn took the cluster down.

Fixes

Fix 1: separate clusters for separate blast radii

The simplest, highest-value change was to give each pipeline its own registry cluster. The users registry is small: it restarts in seconds, needs a modest heap and doesn't care what the events side is doing. The events registry can be sized, tuned and scaled on its own: bigger heaps, more replicas and a longer init timeout.

Nothing about the data needed to change. Clients changed one URL.

Fix 2: delete subjects nobody uses

Most event subjects belonged to event types that hadn't been seen in weeks. Our metadata store already recorded when each event type was last received, so it became the source of truth for cleanup:

for each subject in the registry:
    if the event type it describes hasn't been seen for > threshold:
        DELETE /subjects/{subject}                      # soft delete
        DELETE /subjects/{subject}?permanent=true       # hard delete, frees the memory
# if the event type reappears, the producer re-registers its schema on first use

Two details matter here.

• Soft delete doesn't save memory. A soft-deleted subject keeps its metadata, and its schema IDs remain reserved. Only a hard delete (?permanent=true, which must come after a soft delete) removes it from the indexes, and eventually from the compacted topic.
• Schema IDs live inside your data. In the Confluent wire format, every Avro message starts with a magic byte (0x00) and then the schema ID as a 4-byte big-endian integer. If you hard-delete a schema and an old message that references its ID is still sitting in a topic, a consumer that tries to read it will fail. The inactivity threshold has to be longer than the retention of any topic carrying those IDs. Parquet files are safe, because they embed their own schema, but the Kafka topics feeding them are not.

Fix 3: cap version history

For the warehouse, only the latest version of each subject matters. Some subjects evolved constantly and had accumulated long version chains. We identified the worst ones and kept only the most recent N versions, deleting older ones.

That is safe only for the right compatibility mode:

• With BACKWARD (the default), a new version is checked against the latest version only. Older versions are history, not constraints.
• With BACKWARD_TRANSITIVE or FULL_TRANSITIVE, a new version is checked against all previous versions, and deleting old ones silently weakens that guarantee.

Check the subject's effective compatibility level before pruning.

Fix 4: fewer lookups per task

Two levers here:

• Partition by event type. If all records of one event type go to one partition, only one task ever needs that schema, and post-rebalance lookups drop from tasks × types toward types. The trade-off is skew: a very hot event type becomes a very hot partition. We prototyped this in a new near-real-time pipeline rather than retrofitting it onto existing topics.
• Size the converter's caches. The registry client inside the Avro converter caches lookups up to max.schemas.per.subject, which defaults to 1,000. Despite the name, that limit also caps how many subjects the client remembers, because its caches are keyed by subject first. A task that handles thousands of event types will evict and re-fetch constantly, even without a rebalance. That turns a cache into a steady trickle of registry traffic. Raise value.converter.max.schemas.per.subject above the number of distinct subjects a task actually sees. schemas.cache.config (also 1,000 by default) is a different cache, for converting between Avro and Connect schemas. Size it to the number of distinct schemas a task sees.

What changed

• An events-side incident no longer takes the users pipeline down, by construction.
• The events registry carries far fewer subjects after cleanup, so restarts replay much less and heaps have room to spare.
• Connect rebalances no longer cause a lookup storm big enough to push the registry into errors.

Lessons

• Your registry's size is a product decision in disguise. If customers can create event types, they can create subjects. Decide which side of that line the schema boundary sits on before it reaches six figures.
• Know where your service keeps its state. "Stateless HTTP service" was the wrong mental model. A registry backed by a log replays that log on every start, and every decision about restarts, heap and deploy strategy follows from that.
• Hard deletes and wire formats interact. Deleting a schema is a data-retention decision, not just a cleanup task.
• Caches have cold starts. Any system that caches remote lookups per process needs a plan for the moment all those processes restart together.

Reproduce it yourself

Three labs that rebuild the mechanism on a laptop. They need Docker, curl, jq and xxd, and use Apache Kafka 4.3.1 with Confluent Schema Registry 8.3.2. Every timing below is a laptop lab result from Docker Desktop on an Apple-silicon Mac with 4 CPUs and 4 GB of memory, not a production measurement.

Start a single Kafka broker and one registry:

# compose.yaml
services:
  kafka:
    image: apache/kafka:4.3.1
    container_name: kafka
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
  schema-registry:
    image: confluentinc/cp-schema-registry:8.3.2
    container_name: schema-registry
    depends_on: [kafka]
    ports: ["8081:8081"]
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092

docker compose up -d
until curl -sf -o /dev/null localhost:8081/subjects; do sleep 1; done

# Prints every record in the registry's database as "key | value".
schemas() {
  docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 \
    --topic _schemas --partition 0 --offset earliest --timeout-ms 2000 \
    --formatter-property print.key=true --formatter-property key.separator=' | ' 2>/dev/null
}

Lab 1: read the registry's database

Register one schema. By default, the value schemas of a topic named orders live under the subject orders-value:

curl -s -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}"}' \
  localhost:8081/subjects/orders-value/versions; echo
docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 \
  --describe --topic _schemas | head -n 1
schemas

{"id":1,"version":1,"guid":"8b8bd78e-4456-4c99-8d72-64cbe6177dae","schemaType":"AVRO","schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}"}
Topic: _schemas	TopicId: RwunGvNOQ6SV_N6V0UKctg	PartitionCount: 1	ReplicationFactor: 1	Configs: min.insync.replicas=1,cleanup.policy=compact
{"keytype":"NOOP","magic":0} | null
{"keytype":"NOOP","magic":0} | null
{"keytype":"SCHEMA","subject":"orders-value","version":1,"magic":1} | {"subject":"orders-value","version":1,"id":1,"schemaType":"AVRO","schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}","deleted":false}

Three things to notice:

• The whole database is one compacted partition. A single partition keeps every write in one order. Compaction means Kafka eventually keeps only the newest record for each key.
• The key is keytype, subject and version. The value carries the schema text, its global id and a deleted flag. Every index the registry answers from is built in memory by reading these records from the first offset.
• The NOOP records are bookmarks. When an instance starts, it writes a NOOP record, notes its offset and waits until its own reader has consumed that far. Only then does it know it has replayed everything. Every start adds two of them, and Lab 3 times this wait.

Now look at what lands in a data topic. Produce one Avro message with schema 1 and print its raw bytes:

echo '{"id":"o-1"}' | docker exec -i schema-registry kafka-avro-console-producer \
  --bootstrap-server kafka:9092 --topic orders \
  --property schema.registry.url=http://localhost:8081 --property value.schema.id=1 >/dev/null 2>&1
docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 \
  --topic orders --from-beginning --max-messages 1 2>/dev/null | xxd

00000000: 0000 0000 0106 6f2d 310a                 ......o-1.

The first five bytes are the Confluent wire format. Byte 0 is the magic byte, 00. Bytes 1 to 4 are the schema ID, 00 00 00 01: the number 1 as a 4-byte big-endian integer. Only then does the Avro data start. 06 is the string's length (Avro writes lengths as zig-zag varints, so 3 becomes 6), followed by the three bytes of o-1. The trailing 0a is the newline the console consumer adds. Nothing can decode this message without schema 1, and the registry is the only place that has it.

Lab 2: soft delete, then hard delete

Try the hard delete on its own first:

curl -s -X DELETE 'localhost:8081/subjects/orders-value?permanent=true'; echo

{"error_code":40405,"message":"Subject 'orders-value' was not deleted first before being permanently deleted"}

The registry insists on two steps. Soft-delete the subject, then look around:

curl -s -X DELETE localhost:8081/subjects/orders-value; echo
curl -s localhost:8081/subjects; echo
curl -s 'localhost:8081/subjects?deleted=true'; echo
curl -s localhost:8081/schemas/ids/1; echo
schemas | tail -n 1

[1]
[]
["orders-value"]
{"subject":"orders-value","version":1,"guid":"8b8bd78e-4456-4c99-8d72-64cbe6177dae","schemaType":"AVRO","schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}","ts":1792002259781,"deleted":true}
{"keytype":"DELETE_SUBJECT","subject":"orders-value","magic":0} | {"subject":"orders-value","version":1}

A soft delete is one extra record, DELETE_SUBJECT, which says "versions up to 1 are deleted". The original SCHEMA record is untouched. The subject drops out of the normal listing, but schema 1 still resolves by ID, now marked "deleted":true. That's why a soft delete frees no memory, and why messages carrying ID 1 stay readable.

Now the hard delete, then read the message from Lab 1 again:

curl -s -X DELETE 'localhost:8081/subjects/orders-value?permanent=true'; echo
curl -s localhost:8081/schemas/ids/1; echo
schemas | tail -n 1
docker exec schema-registry kafka-avro-console-consumer --bootstrap-server kafka:9092 \
  --topic orders --partition 0 --offset earliest --max-messages 1 --timeout-ms 10000 \
  --property schema.registry.url=http://localhost:8081 2>&1 | grep 'Caused by'

[1]
{"error_code":40403,"message":"Schema 1 not found"}
{"keytype":"SCHEMA","subject":"orders-value","version":1,"magic":1} | null
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema 1 not found; error code: 40403

Two things to notice:

• A hard delete writes a tombstone: the original SCHEMA record's key with a null value. The registry forgets the schema at once. Compaction later removes the original record, and after delete.retention.ms the tombstone too.
• The message from Lab 1 is now unreadable. Its first five bytes still point at schema 1, and schema 1 no longer exists. That's why the threshold in Fix 2 has to outlast the retention of every topic that carries those IDs.

Lab 3: watch restarts slow down

Three small scripts. register.sh registers subjects through the API, one schema each, the way the events pipeline did:

#!/usr/bin/env bash
# Usage: ./register.sh FIRST LAST
# One subject per number, each with its own record schema, so every
# registration is a new subject, a new schema ID and a new _schemas record.
seq "$1" "$2" | xargs -P 8 -I{} curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"schema":"{\"type\":\"record\",\"name\":\"Event{}\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}"}' \
  localhost:8081/subjects/event-{}-value/versions | sort | uniq -c

restart.sh stops the registry, starts it again and times how long until it answers. Then it prints the moments from the registry's log that matter: the reader seeking to the start of _schemas, catching up with each NOOP bookmark, and the server starting.

#!/usr/bin/env bash
# Stops the registry, starts it again, and times how long until it answers requests.
records=$(docker exec kafka /opt/kafka/bin/kafka-get-offsets.sh \
            --bootstrap-server kafka:9092 --topic _schemas | cut -d: -f3)
docker stop schema-registry >/dev/null
since=$(date -u +%Y-%m-%dT%H:%M:%SZ)
TIMEFORMAT="records=$records ready after %1Rs"
time (docker start schema-registry >/dev/null
      until curl -sf -o /dev/null localhost:8081/config; do sleep 0.2; done)
docker logs --since "$since" schema-registry 2>&1 |
  grep -E 'Seeking to beginning|Reached offset|Server started' |
  sed -E 's/^\[[0-9-]+ ([0-9:,]+)\] INFO (.*) \(.*$/\1  \2/'

Registering through the API managed about 150 subjects a second on my laptop, and going from 8 to 32 parallel clients barely changed that. Half a million subjects would take about an hour. The registry doesn't care who wrote a record, though: its reader consumes _schemas the same way whoever produced to it. So append.sh writes records straight into the topic, in exactly the format Lab 1 printed. Never do this to a registry you care about.

#!/usr/bin/env bash
# Usage: ./append.sh FIRST LAST
# Writes one SCHEMA record per subject straight into _schemas, in the format Lab 1
# printed. IDs are offset by 1,000,000 so they can't collide with API-assigned ones.
jq -nr --argjson from "$1" --argjson to "$2" '
  range($from; $to + 1) as $i
  | "event-\($i)-value" as $s
  | ({keytype: "SCHEMA", subject: $s, version: 1, magic: 1} | tojson) + "\t" +
    ({subject: $s, version: 1, id: (1000000 + $i), schemaType: "AVRO", deleted: false,
      schema: ({type: "record", name: "Event\($i)", fields: [{name: "id", type: "string"}]} | tojson)}
     | tojson)' |
docker exec -i kafka /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka:9092 \
  --topic _schemas --reader-property parse.key=true --reader-property key.separator=$'\t' 2>/dev/null

Grow the log and restart after each step:

chmod +x register.sh restart.sh append.sh
./register.sh 1 4000      && ./restart.sh
./register.sh 4001 25000  && ./restart.sh
./register.sh 25001 50000 && ./restart.sh
./append.sh 50001 500000  && ./restart.sh

The last restart printed:

records=500011 ready after 8.5s
18:37:38,795  Seeking to beginning for all partitions
18:37:42,064  Reached offset at 500011
18:37:45,131  Reached offset at 500012
18:37:45,461  Server started, listening for requests...

Here are all four restarts side by side. "Replay" is the time from the seek to the first "Reached offset" line.

Three things to notice:

• Only the replay grows. It went from 0.2 s to 3.3 s as the log grew 125-fold. The other 5 seconds or so were the same on every run: starting the JVM, and about 3 seconds between finishing the replay and joining the group the registry uses to elect its leader. The lab shows the shape, not our production numbers.
• Listing everything gets expensive. At 50,000 subjects, one GET /subjects response was already 988,895 bytes. At 500,000, the registry still answered single lookups on the image's default 512 MB heap, but GET /subjects failed with OutOfMemoryError: Java heap space. That's one of the two calls that spiked during our incident.
• A restart can't fix a heap that's too small. When I pushed on toward a million subjects, the registry ran out of heap as the new records arrived, ran out again while replaying the log after a restart, and didn't come back up before I stopped the lab. That's the loop from Heap pressure and OOMs above, compressed: the replay needs the same memory the indexes do.

Clean up with docker compose down -v.

References

• Schema Registry API reference: subjects, versions, lookups by schema ID, and the delete endpoints used in Lab 2.
• Delete schemas in Confluent Platform: soft versus hard deletes, and why a soft delete frees no space.
• Formats, serializers and deserializers for Schema Registry: the wire format, with the magic byte and the big-endian schema ID.
• Apache Avro specification: binary encoding, including zig-zag varints and how strings are written.
• Schema Registry configuration reference: kafkastore.topic, kafkastore.init.timeout.ms and the other storage settings.
• Deploy Schema Registry in production: the _schemas topic and production deployment guidance.
• Integrate schemas from Kafka Connect: converter settings, including max.schemas.per.subject and schemas.cache.config.
• Schema evolution and compatibility: which earlier versions each compatibility mode checks a new schema against.
• Apache Kafka documentation: log compaction: how compaction keeps the newest record per key, and when tombstones go away.
• KafkaStore source: the NOOP bookmark and the catch-up wait at startup.
• CachedSchemaRegistryClient source: the client's caches, keyed by subject and capped by the cache capacity.
• AvroConverter source: where the converter's client takes its capacity from max.schemas.per.subject.
• AbstractKafkaAvroDeserializer source: the converter path that looks up each schema's version.
• AvroData source: the two schema conversion caches that schemas.cache.config sizes.