Skip to content
The 3 AM Brew

Logs as a Pipeline, Not a Pile: Running Vector on Kubernetes

Agents that only forward, an aggregator that owns durability, Kafka in the middle, disk buffers on gp3, dead-letter routing for events that fail to parse, and S3 keys designed around the query engine. Plus the three things that bit us: a queue that throttled sends, a metrics label that leaked forever, and two containers sharing one name.

Shivam Kumar15 min, 2 coffees

Container logs on Kubernetes have a short life. They're written to files on the node, rotated, and deleted when the pod goes away. If you want to read them later, during an incident or for a quarterly audit, something has to move them somewhere durable, and do it without losing data when that something restarts.

We built that "something" with Vector. This post covers the architecture, the configuration decisions that matter, and the three problems that taught us the most.

Requirements

  • Every container's stdout and stderr, enriched with pod metadata: namespace, labels, container name.
  • Durable: a restart of any component must not lose logs.
  • Archive to S3, queryable with Athena. Query cost depends on how keys are laid out.
  • Cheap on the nodes. The agent runs on every node, so every millicore counts.
  • Failures visible and recoverable. An event that fails processing must be kept, not dropped.

Architecture: dumb agents, smart aggregators

Vector supports two deployment roles, and we used both.

 node ─┬─ Vector agent (DaemonSet)          Kafka                 Vector aggregator (StatefulSet)                 S3
       │   source: kubernetes_logs   ──►  logs topic   ──►  source: kafka                                   ┌─► {team}/{deployment_group}/{container}/dt=...
       │   no parsing, no routing                            remap  (reroute_dropped = true)  ── ok ──► route ┤
       │   sink: kafka                                         │                                            └─► fallback prefix (no team label)
       │                                                       └── dropped ──► S3 dead-letter prefix (disk buffer)
       └─ ...                                               sinks: aws_s3, disk buffer on a gp3 PVC

The agent is a forwarder. It runs as a DaemonSet with the kubernetes_logs source, which tails container log files and enriches each event with metadata from the Kubernetes API. Then it ships events to Kafka. It doesn't parse, remap or route anything. Every transform you put on the agent costs CPU on every node, and a config change becomes a fleet-wide rollout.

The aggregator owns processing and durability. It runs as a StatefulSet on dedicated nodes, consumes from Kafka, runs the transforms and writes to S3. Vector's own sizing guidance points aggregators at at least 4 vCPUs, and memory grows with the number of sinks, because each sink batches in memory.

Kafka sits in the middle as a shock absorber. If the aggregators are down for an hour, the agents keep shipping and Kafka keeps the backlog. When the aggregators come back they resume from committed offsets. Kafka is a pull-based source, so anything not yet processed can simply be read again.

Buffers: system of record vs system of analysis

Every Vector sink has a buffer in front of it, and the choice of buffer type is the choice between latency and durability:

  • Memory buffers are fast. Anything in them is lost if Vector is killed.
  • Disk buffers are a write-ahead log. Events are written to disk before the sink reads them back out, so a restarted aggregator resumes where it stopped. Vector syncs to disk periodically (every 500 ms by default), so a crash can still lose a sliver, and each event is checksummed so corruption is detected rather than silently replayed.

Our rule follows Vector's own advice: systems of record use disk buffers, systems of analysis use memory buffers. The S3 archive is the system of record, so its sinks use disk buffers on a gp3 persistent volume:

[sinks.archive]
type   = "aws_s3"
inputs = ["route.has_team"]
bucket = "logs-archive"
key_prefix = "{{ kubernetes.pod_labels.team }}/{{ kubernetes.pod_labels.deployment_group }}/{{ kubernetes.container_name }}/dt=%Y-%m-%d/"
compression = "gzip"
encoding.codec = "json"

[sinks.archive.buffer]
type      = "disk"
max_size  = 10737418240      # 10 GiB; disk buffers have a ~256 MiB floor
when_full = "block"          # back-pressure upstream instead of dropping

Some things we learned:

  • Volume performance matters. A disk buffer is only as fast as its disk, and a slow disk stalls the sink. Vector's documentation calls out gp2 volumes as usually too slow. gp3 provides a 3,000 IOPS / 125 MiB/s baseline regardless of size, and you can provision more independently, which suits a buffer far better than gp2's size-coupled burst model.
  • when_full = "block" is back-pressure, not data loss. When the buffer fills up, Vector stops pulling from Kafka. The backlog stays in Kafka, where it's cheap and visible as consumer lag. drop_newest is for data you're happy to lose, such as debug logs.
  • Watch free space. If writing to the buffer fails, for example because the disk is full, Vector stops itself rather than silently losing data. Alert on buffer volume usage like any other stateful disk.

Dead letters: keep the events that fail

A remap transform runs VRL on each event. If the program fails (bad JSON, a fallible function you !-asserted, a type error), the default is to pass the event through unmodified. With drop_on_error = true, it's dropped and logged instead. Neither is good in a log pipeline. The first sends half-processed events into tables that expect a shape. The second loses exactly the strange event you'll need later.

The remap transform supports reroute_dropped. Combined with drop_on_error, failed events are sent to a separate .dropped output, which you can route like any other stream:

[transforms.kube_parsing]
type            = "remap"
inputs          = ["kafka_source"]
drop_on_error   = true
reroute_dropped = true
source = '''
  del(.file)
  del(.source_type)
  del(.stream)
'''

[sinks.parse_failures]
type       = "aws_s3"
inputs     = ["kube_parsing.dropped"]
bucket     = "logs-archive"
key_prefix = "_dropped/%Y-%m-%d/"
encoding.codec = "json"
compression    = "gzip"

[sinks.parse_failures.buffer]
type      = "disk"
max_size  = 1073741824
when_full = "block"

The dropped events land in S3 with metadata about why they failed. Once the bug is fixed you can inspect them, correct the transform and replay them. Dropped-event counts also get their own alert. A sudden increase almost always means some application changed its log format.

Routing on labels, with a fallback

S3 keys are rendered from templates like {{ kubernetes.pod_labels.team }}. If an event has no team label, the template fails to render and Vector drops the event with an error. Not every pod in a cluster is well-labelled, especially system components and third-party charts.

So the routing stage splits events first:

[transforms.route]
type   = "route"
inputs = ["kube_parsing"]
route.has_team = 'exists(.kubernetes.pod_labels.team)'
route.no_team  = '!exists(.kubernetes.pod_labels.team)'

has_team goes to the templated archive sink, and no_team goes to a fallback prefix keyed only on namespace and container. Nothing is lost because someone forgot a label, and the fallback prefix doubles as a list of workloads that need labelling.

Problem 1: the queue that throttled

The first version used a managed message queue between agents and aggregators instead of Kafka. It worked in staging. In the busiest region, sends started getting throttled: the agents produced faster than the queue's API accepted, batching made little difference, and backlog piled up on the nodes.

Moving to Kafka fixed it. Throughput depends on partitions and brokers, not on a per-API-call quota. Batching is native. And consumer lag is a single, well-understood metric to alert on. If you already run Kafka, it's a better fit for a high-volume pipeline from agents to aggregators.

We also saw occasional connection closed before message completed warnings from the S3 sink, which were retried and succeeded. They're transient, but they're the reason to keep retries and adaptive request concurrency on for object-storage sinks, rather than pinning a high fixed concurrency.

Problem 2: a metrics label that leaked forever

After a few weeks, agent CPU and memory crept up on long-lived nodes, with no change in log volume. The cause was in Vector's own telemetry.

The kubernetes_logs source emits internal metrics per file, and those metrics carried the log file path as a label. Every new pod creates new log files, so every new pod created a new label set. At the time, nothing expired old series. On a node that had run thousands of short-lived pods, the agent was tracking and exporting thousands of dead time series. It's the same cardinality problem as in an application's /metrics, inside the log shipper itself.

Fixes, in order of preference:

  1. Expire stale internal metrics. Newer Vector versions have a global expire_metrics_secs option (and expire_metrics_per_metric_set for finer control) that drops series not updated within a window. Set it longer than your scrape interval.
  2. Drop the label at scrape time. If you can't upgrade, a metric_relabel_configs rule can drop the file label, or the whole per-file metric, before it reaches your TSDB. That protects storage, but not the agent's memory.
  3. Restart agents on a schedule. It's a band-aid, and it works.

Problem 3: two containers, one name, one very slow query

Our original key layout was {team}/{container}/date. Then someone queried logs for a stream-processing job's coordinator and Athena started throttling.

The coordinator and the workers were separate Deployments, but their containers had the same name. Every log line from dozens of busy workers landed under the same prefix as the one quiet coordinator. Athena can only prune on what's in the key, so a query for coordinator logs scanned all of the workers' logs too.

The fix was to put a deployment_group label on every Deployment and add it to the key:

before:  {team}/{container}/dt=YYYY-MM-DD/
after:   {team}/{deployment_group}/{container}/dt=YYYY-MM-DD/

With a matching table definition, partition projection or explicit partitions, Athena now reads only the prefix for the component you asked about. Design object keys for the query engine. Every dimension people filter on and that isn't in the key becomes a full scan.

Takeaways

  • Keep agents dumb. Parse, route and enrich in a tier you can scale and redeploy without touching every node.
  • Put a log in the middle. Kafka between agents and aggregators turns aggregator outages into lag, not loss.
  • Choose buffers by role. Disk buffers for the system of record, on disks that can keep up. Memory buffers for anything you can afford to lose.
  • Route failures, don't drop them. reroute_dropped plus a dead-letter prefix makes parse errors something you can replay.
  • Your pipeline's own metrics have cardinality too. Watch the shipper like you watch the apps it ships for.
  • Object keys are a query plan. Decide the layout with the people who will search it.

Reproduce it yourself

Three labs that need only Docker and jq. They use Vector 0.59.0 and run this post's aggregator transforms, with stand-ins at both ends: a demo_logs source named kafka_source replays three agent-shaped events instead of reading Kafka, and blackhole and console sinks take the place of S3.

The remap in this post only deletes fields, and deleting a field can't fail, so on its own it never produces a dead letter. The lab adds one step that can: parsing the application's message as JSON, the kind of step that breaks when an application changes its log format. The [api] block is there for vector tap in Lab 2.

# pipeline.toml
[api]
enabled = true

# Stands in for the Kafka source: replays three agent-shaped events.
[sources.kafka_source]
type           = "demo_logs"
format         = "shuffle"
interval       = 1.0
decoding.codec = "json"
lines = [
  '{"message":"{\"level\":\"info\",\"msg\":\"payment accepted\"}","file":"/var/log/pods/pay/api/0.log","stream":"stdout","kubernetes":{"container_name":"api","pod_labels":{"team":"payments","deployment_group":"api"}}}',
  '{"message":"{\"level\":\"warn\",\"msg\":\"upstream slow\"}","file":"/var/log/pods/kube-system/coredns/0.log","stream":"stdout","kubernetes":{"container_name":"coredns","pod_labels":{}}}',
  '{"message":"panic: runtime error: index out of range","file":"/var/log/pods/pay/api/0.log","stream":"stderr","kubernetes":{"container_name":"api","pod_labels":{"team":"payments","deployment_group":"api"}}}',
]

[transforms.kube_parsing]
type            = "remap"
inputs          = ["kafka_source"]
drop_on_error   = true
reroute_dropped = true
source = '''
  del(.file)
  del(.source_type)
  del(.stream)
  .app = parse_json!(.message)
'''

[transforms.route]
type   = "route"
inputs = ["kube_parsing"]
route.has_team = 'exists(.kubernetes.pod_labels.team)'
route.no_team  = '!exists(.kubernetes.pod_labels.team)'

[sinks.archive]
type   = "blackhole"
inputs = ["route.has_team"]

[sinks.fallback]
type   = "blackhole"
inputs = ["route.no_team"]

[sinks.parse_failures]
type           = "console"
inputs         = ["kube_parsing.dropped"]
encoding.codec = "json"

Lab 1: unit-test the remap and the route

A Vector unit test inserts a hand-written event at a transform, in place of a real source, and runs VRL assertions against whatever comes out of the outputs you name. These three tests cover the three ways an event can leave the aggregator:

# tests.toml
[[tests]]
name = "labelled JSON log is parsed and archived under its team"

[[tests.inputs]]
insert_at  = "kube_parsing"
type       = "log"
log_fields = { message = '{"level":"info"}', stream = "stdout", "kubernetes.pod_labels.team" = "payments" }

[[tests.outputs]]
extract_from = "route.has_team"
[[tests.outputs.conditions]]
type   = "vrl"
source = '''
  assert_eq!(.app.level, "info")
  assert!(!exists(.stream))
'''

[[tests]]
name = "unlabelled log takes the fallback route"

[[tests.inputs]]
insert_at  = "kube_parsing"
type       = "log"
log_fields = { message = '{"level":"warn"}', "kubernetes.container_name" = "coredns" }

[[tests.outputs]]
extract_from = "route.no_team"
[[tests.outputs.conditions]]
type   = "vrl"
source = 'assert_eq!(.app.level, "warn")'

[[tests]]
name = "plain-text log goes to the dead-letter output, untouched"

[[tests.inputs]]
insert_at  = "kube_parsing"
type       = "log"
log_fields = { message = "panic: runtime error", stream = "stderr", "kubernetes.pod_labels.team" = "payments" }

[[tests.outputs]]
extract_from = "kube_parsing.dropped"
[[tests.outputs.conditions]]
type   = "vrl"
source = '''
  assert_eq!(.stream, "stderr")
  assert_eq!(.metadata.dropped.reason, "error")
'''

docker run --rm -v "$PWD:/etc/vector:ro" timberio/vector:0.59.0-debian \
  test /etc/vector/pipeline.toml /etc/vector/tests.toml

Running tests
test labelled JSON log is parsed and archived under its team ... passed
test unlabelled log takes the fallback route ... passed
test plain-text log goes to the dead-letter output, untouched ... passed

Now make the kind of edit these tests exist to catch. Turn drop_on_error off and run the same tests:

mkdir -p broken && cp tests.toml broken/
sed 's/^drop_on_error   = true$/drop_on_error   = false/' pipeline.toml > broken/pipeline.toml
docker run --rm -v "$PWD/broken:/etc/vector:ro" timberio/vector:0.59.0-debian \
  test /etc/vector/pipeline.toml /etc/vector/tests.toml

Running tests
test labelled JSON log is parsed and archived under its team ... passed
test unlabelled log takes the fallback route ... passed
test plain-text log goes to the dead-letter output, untouched ... failed

failures:

test plain-text log goes to the dead-letter output, untouched:

checks for transforms ["kube_parsing.dropped"] failed: no events received. Topology may be disconnected or transform is missing inputs.

Two things to notice:

  • reroute_dropped does nothing on its own. Without drop_on_error, a failed event isn't dropped, so there's nothing to reroute. The remap passes the original event downstream unchanged, here into the team archive, and .dropped stays empty. The two settings only work as a pair.
  • The failed run exits non-zero (78 here), so CI can block the change. And because extract_from accepts named outputs like route.no_team and kube_parsing.dropped, every branch of the pipeline can have its own test.

Lab 2: watch a running pipeline with vector tap

vector tap connects to a running Vector through its API and prints a sample of the events leaving a component's output: by default, up to 100 events every 500 ms. Start the pipeline, then tap each route. The source picks one of its three lines at random every second, so if a short tap catches nothing, run it again.

docker run -d --name vector -v "$PWD/pipeline.toml:/etc/vector/vector.toml:ro" \
  timberio/vector:0.59.0-debian --config /etc/vector/vector.toml >/dev/null
sleep 8
docker exec vector vector tap --outputs-of route.no_team --quiet --duration-ms 5000 2>/dev/null | head -n 1
docker exec vector vector tap --outputs-of route.has_team --format yaml --quiet --duration-ms 6000 2>/dev/null | head -n 13

{"log":{"app":{"level":"warn","msg":"upstream slow"},"host":"localhost","kubernetes":{"container_name":"coredns","pod_labels":{}},"message":"{\"level\":\"warn\",\"msg\":\"upstream slow\"}","service":"vector","timestamp":"2026-10-14T18:31:09.685823720Z"}}
!log
app:
  level: info
  msg: payment accepted
host: localhost
kubernetes:
  container_name: api
  pod_labels:
    deployment_group: api
    team: payments
message: '{"level":"info","msg":"payment accepted"}'
service: vector
timestamp: 2026-10-14T18:31:19.685443628Z

Two things to notice:

  • The remap ran. file, stream and source_type are gone, and the parsed message sits under app.
  • Each route is its own output. The unlabelled coredns event left through no_team, toward the fallback prefix, and the labelled one through has_team. Tapping one output answers "what is this branch actually sending?" without changing the pipeline.

Lab 3: read a dead letter

Here the parse_failures sink prints dead letters to the container's log, where the real one wrote them under _dropped/ in S3. Tap the .dropped output for 15 seconds, pretty-print the first event, and clean up:

docker exec vector vector tap --outputs-of kube_parsing.dropped --quiet --duration-ms 15000 2>/dev/null > tap-dropped.json
head -n 1 tap-dropped.json | jq .log
docker rm -f vector >/dev/null

{
  "file": "/var/log/pods/pay/api/0.log",
  "host": "localhost",
  "kubernetes": {
    "container_name": "api",
    "pod_labels": {
      "deployment_group": "api",
      "team": "payments"
    }
  },
  "message": "panic: runtime error: index out of range",
  "metadata": {
    "dropped": {
      "component_id": "kube_parsing",
      "component_kind": "transform",
      "component_type": "remap",
      "message": "function call error for \"parse_json\" at (57:78): unable to parse json: expected value at line 1 column 1",
      "reason": "error"
    }
  },
  "service": "vector",
  "source_type": "demo_logs",
  "stream": "stderr",
  "timestamp": "2026-10-14T18:31:46.683451501Z"
}

Two things to notice:

  • It's the original event. file, stream and source_type are still there, because when the program fails, Vector throws away its changes and forwards the event as it arrived. That's what makes replay work: fix the remap, then feed these events back through it.
  • metadata.dropped says why. It names the component, the reason, and the VRL error with its position in the program. Here parse_json failed at the first character, because a plain-text panic isn't JSON. That message is the field to group by when the dead-letter alert fires.

References

Written by Shivam Kumar

DevOps Engineer and Back-End Developer If this matched an incident you're working on, I'd like to hear about it.

EmailLinkedInGitHub

Related reading