Skip to content
Fresh brewBounding an Agent’s Authority During Kubernetes Incident Response

I keep production boring.Even at 3 AM.

I'm Shivam, a platform and SRE engineer in Bengaluru. Seven years of building the platforms, autoscaling and access systems that let teams ship without getting paged, and writing up what broke.

Read the 3 AM BrewSee my work

Currently Member of Technical Staff, SRE at Flexera

Incident console
Monitoring
p99 DNS latency, queue-worker nodesSLO 500 ms
1s2s03:0003:0503:1003:15
    A dramatized replay of the ndots postmortem.
    • Kubernetes
    • AWS
    • Terraform
    • Argo CD
    • KEDA
    • Karpenter
    • Graviton
    • Flink
    • Airflow
    • Kafka
    • Strimzi
    • VictoriaMetrics
    • Prometheus
    • Grafana
    • Loki
    • Vector
    • Go
    • Python
    • eBPF
    • Azure

    What I build

    Infrastructure that behaves the same at 3 PM and 3 AM. These are the systems I keep coming back to.

    Rate limiting at the edge of every service

    Problem:
    Bursty internal API traffic could exhaust shared capacity.
    Contribution:
    Put per-tenant quotas in Envoy with a shared rate-limit service and Redis counters.
    Outcome:
    A 100-requests-per-hour test quota admitted exactly 100 requests.

    Quota behavior

    Illustrative browser demo. Requests do not reach Envoy or Redis.

    • tenant-a (default)0 / 5 per s
    • tenant-b (premium)0 / 50 per s
    • tenant-c (default)0 / 5 per s
    Last burst
    none yet
    Allowed
    0
    Rate limited
    0
    Read the write-up

    Platforms with paved paths

    Problem:
    Data teams needed repeatable operations for streaming and scheduled jobs.
    Contribution:
    Built self-service Flink and Airflow platforms on Kubernetes.
    Outcome:
    Airflow platform work reduced operational effort by 50%.
    See platform case studies

    Access that expires

    Problem:
    Production exec access needed a narrow scope and a hard end time.
    Contribution:
    Built approved, namespace-scoped access with scheduled expiry and failure-safe cleanup.
    Outcome:
    Access expires without a person having to remember to revoke it.
    Read the write-up

    A data lake without the small-files tax

    Problem:
    Tenant path partitioning multiplied small objects and query overhead.
    Contribution:
    Wrote Parquet by Kafka partition, then compacted into query-oriented tables.
    Outcome:
    Object counts fell about 120× on the busiest stream and over 1,000× on two others.
    Read the write-up

    Metrics you can afford

    Problem:
    Monitoring cost was growing with a large Kubernetes and EC2 estate.
    Contribution:
    Moved to VictoriaMetrics and added cardinality controls across ingest and query paths.
    Outcome:
    Monitoring infrastructure cost fell 60%.
    See case study

    Logs as a pipeline, not a pile

    Problem:
    Node-local logs needed durable delivery and replayable parse failures.
    Contribution:
    Built a Vector and Kafka pipeline with buffered S3 sinks and dead-letter routing.
    Outcome:
    Aggregator outages become Kafka lag; parse failures remain available for replay.
    Read the write-up

    Receipts, not adjectives

    The before and after of the work, as an engineer would report it. Costs went down, toil went down, and access stopped being permanent.

    Full case studies
    shivam@prod: ~
    $

    Seven years, one trace

    My career drawn as a distributed trace. Companies are spans, roles are child spans, projects sit underneath. Expand one.

    Career trace6 of 23 spans
    Running
    2019202120232025

    Flexera

    Member of Technical Staff, SRE

    Reliability for multi-region SaaS on Kubernetes and AWS: autoscaling, storage, GitOps at fleet scale, and secure operator access.

    • Event-driven autoscaling across regions
    • Storage and delivery reliability on EKS
    • Time-boxed production access
    Start
    Apr 2024
    End
    Still running
    Duration
    2y 7mo
    Where
    Bengaluru
    Stack
    EKS, KEDA, Argo CD, Lambda, CloudFormation

    Select a span to inspect it. Arrow keys move between spans; right and left expand or collapse them.

    The 3 AM Brew

    Incident notes and design write-ups, brewed after the pager goes quiet.

    All 11 issues
    DesignNo. 11

    Bounding an Agent’s Authority During Kubernetes Incident Response

    Intent classification can choose a specialist, but it cannot grant access. A practical design for routing incident work across clusters while keeping authorization in scoped grants and Kubernetes RBAC.

    9 minKubernetesPlatformSecurityAI
    DesignNo. 10

    Shell Access That Expires on its Own: Building Just-in-Time kubectl exec

    Standing exec permissions in production are a liability. This is the design of a just-in-time replacement (request, approval, namespace lock, broker, scheduled expiry), and the long tail of making expiry recoverable as well as fail-closed: retry semantics, durable notifications, CloudFormation imports and evicted pods.

    14 minKubernetesSecurity
    IncidentNo. 9

    Eight hours for a Ten-Minute Job: EFS Burst Credits, NFS Cache Flags and SQLite

    Same image, three regions, one of them eight hours slow. The cause sat in a credit balance, a 4 KiB metering rule and three mount options. The first fix then broke an unrelated upload service, which taught me more about NFS close-to-open consistency than I ever wanted to know.

    22 minAWSKubernetes

    Got a platform problem that pages people at 3 AM?

    I'm happy to talk platform and SRE roles, or to compare notes on an incident that looks like one of mine.

    Email meLinkedIn