Shell Access That Expires on its Own: Building Just-in-Time kubectl exec

Every production cluster eventually has a "who can kubectl exec here?" problem. Usually the answer is "too many people, permanently", because the alternative is a 2 a.m. page with no way in.

We replaced standing exec rights with a just-in-time (JIT) system. An engineer requests access to one namespace for a bounded window, a reviewer approves it, the platform provisions it, and it disappears on schedule. This post covers the design, then the part that took longest to get right: making expiry recoverable, not only safe.

What pods/exec actually grants

It's worth being precise about the threat, because it shapes every decision after it. RBAC permission on the pods/exec subresource lets you run an arbitrary process inside a running container. From there you inherit everything that container has:

• Its mounted secrets and config, including any projected service-account token, and therefore whatever that service account can do against the API.
• Its cloud identity. On EKS that means the pod's IAM role via IRSA or Pod Identity, with every AWS permission that role holds.
• Its network position: whatever the pod can reach, inside or outside the cluster, regardless of network policy aimed at humans.
• Its filesystem and memory, including data in flight.

In other words, exec is "become this workload". Granting it permanently, across namespaces, to whole teams, is granting all of their workloads' privileges to all of those people, all the time.

Goals and non-goals

What we wanted:

• No standing exec rights for application teams in production namespaces.
• One namespace per grant, with a hard upper bound on duration.
• Expiry that needs nobody: no human revoke step, no cron someone forgets to deploy.
• An audit trail tied to a request that anyone can read later.

What we explicitly don't promise:

• Expiry can't undo anything done during the session, and it can't stop a process someone started with nohup inside the target container. JIT bounds access. It doesn't roll back effects.

How exec actually travels

The broker design makes more sense once you remember what kubectl exec does on the wire:

kubectl ──(HTTPS upgrade)──► kube-apiserver ──► kubelet on the pod's node ──► container runtime (CRI streaming)
          POST/GET .../pods/<p>/exec?command=sh&stdin=true&tty=true

• The client asks the API server to upgrade the connection. Historically that was SPDY/3.1, a long-deprecated protocol that many proxies and load balancers don't handle. Kubernetes has been moving exec, attach and port-forward to WebSockets (KEP-4006), with SPDY kept as a fallback during the transition.
• The API server authorizes the request against the pods/exec subresource, then proxies the stream to the kubelet, which hands it to the runtime's streaming server.
• RBAC subtlety: a WebSocket upgrade begins life as an HTTP GET, while the SPDY path used POST. So the verb an authorizer sees has historically depended on the transport. Grant (and audit) both get and create on pods/exec when you mean "can exec", and don't assume a role with only one of them is safe.
• Audit visibility: API audit logs record the exec request, including the command query parameters, the user and the target pod. They don't record the interactive stream. Typing sh and then running anything inside it shows up as "exec sh".

That last point is the main reason for a broker. If we want session-level enforcement (deadlines, scoping, stream logging), it has to sit somewhere the stream passes through.

The request path

Requests are GitHub issues created from an issue form: cluster, namespace, reason and duration. We didn't need a new UI, and the issue thread doubles as the audit record.

issue form ──► request workflow ──► preflight ──► environment approval ──► executor
   │               │                    │                                   │
   │          validate team ⇄       "is this namespace                  provision grant,
   │          namespace mapping      free right now?"                   schedule expiry
   ▼                                                                         │
 issue thread ◄──────────────── status comments + connection guide ◄────────┘

The executor is a Lambda function in each cluster's AWS account. The workflow reaches it through a role assumed via GitHub's OIDC federation, scoped to this repository and environment, so there are no long-lived AWS keys in CI. It's the only principal allowed to create RBAC objects for JIT in the target cluster.

Approval uses a GitHub environment with required reviewers. The approval is attached to the workflow run, the run is attached to the issue, and the issue carries the request. Every link is visible and timestamped.

What one grant actually covers

The authorization unit is a tuple:

(GitHub team, Kubernetes group, target namespace)

Configuration maps teams to the namespaces they may request; a request produces a grant for exactly one namespace. The list of eligible namespaces describes what you could ask for, not how broad a single grant is.

Client access binds to a Kubernetes Group, not to the individual who opened the issue. That was deliberate: whoever is on call should be able to join the session. It has a consequence we document loudly: different teams mapped to the same group share an approved session. The configuration generator rejects the one ambiguous case, a single team and namespace pair mapping to two groups, at build time rather than at request time.

Broker and client

A grant creates two pods:

• The broker holds a credential scoped to the target namespace only, and it's the only thing that can open exec streams into that namespace's workloads.
• The client is what engineers in the bound group connect to. Their own RBAC allows exec into the client and nothing else.

The broker's credential is scoped to the target namespace and lives only as long as the grant. If you build something similar, issue it as a short-lived, audience-bound token from the TokenRequest API rather than a long-lived Secret, with an expiry no later than the grant's. (TokenRequest has a 10-minute minimum expirationSeconds, so very short grants rely on the broker's own deadline rather than the token's.)

Both pods carry the grant's absolute deadline as immutable configuration and enforce it independently. When the deadline passes, the broker refuses new streams and closes the ones it owns, even if every other part of the cleanup machinery has failed. Expiry is fail-closed by construction: the safety of the system doesn't depend on the scheduler, the cleanup Lambda or GitHub being up.

One session per namespace: the lock

Two overlapping grants in the same namespace are confusing during an incident at best, and an escalation path at worst. So each namespace has a lock: a small Kubernetes object named after the target namespace, holding the grant's immutable scope.

Kubernetes gives you a compare-and-set for free: create either succeeds or returns 409 AlreadyExists. No read-then-write race is possible.

def acquire_lock(api, control_ns: str, target_ns: str, request_id: str, expires_at: str):
    body = {
        "metadata": {
            "name": f"jit-lock-{target_ns}",
            "labels": {"jit/request-id": request_id},
            "annotations": {"jit/expires-at": expires_at},
        },
        "data": {"scope": canonical_scope_json(...)},   # immutable once written
    }
    try:
        return api.create_namespaced_config_map(control_ns, body)
    except ApiException as e:
        if e.status == 409:
            raise NamespaceBusy(read_holder(api, control_ns, target_ns))
        raise

Releasing the lock uses delete preconditions on uid and resourceVersion. A delayed revoke from an old grant then can't delete a lock that now belongs to a newer one:

api.delete_namespaced_config_map(
    name, control_ns,
    body=V1DeleteOptions(preconditions=V1Preconditions(uid=lock.uid,
                                                       resource_version=lock.rv)),
)

Why not a coordination.k8s.io Lease? Leases are built for holders that keep renewing. A JIT grant is the opposite: a fixed, non-renewable deadline. A plain object with create semantics and precondition deletes models "exactly one, until this time" more honestly than a lease nobody is supposed to renew.

Preflight is a contract, not a boolean

The first preflight returned ok: true/false. That was too little information. Every failure looked the same, so a transient API error closed a perfectly valid request. The redesigned response tells the workflow what to do next:

Only an explicit available or replaceable leads to approval. Nothing else authorizes a grant, including a timeout or a response with an unexpected shape. Because the producer's semantics changed, rollout order mattered: we shipped the backward-compatible, fail-closed consumer first.

Expiry: where the real engineering was

At provisioning time, the executor creates a one-time EventBridge Scheduler schedule using an at(...) expression set to the grant's deadline, with the same Lambda as its target and a revoke payload. An hourly sweep acts as a backstop for anything a schedule missed.

Three Scheduler settings matter more than they look:

• ActionAfterCompletion: DELETE removes the schedule after it fires. One-time schedules otherwise pile up indefinitely. But a schedule set to auto-delete is also deleted after its retries are exhausted. If delivery fails and there's no dead-letter queue, the evidence that a revoke ever existed disappears with it.
• Retry policy: Scheduler retries delivery with exponential backoff until it reaches either its maximum attempts or its maximum event age, whichever comes first. Set both deliberately rather than relying on the defaults.
• Flexible time window off. A revoke that may fire "sometime in the next 15 minutes" isn't a deadline.

Revoke follows a strict order:

1. Claim the request ID conditionally. A stale scheduled revoke for a replaced grant becomes a successful no-op.
2. Delete the broker first. That removes the credential and closes live streams.
3. Delete the remaining request objects: client, RoleBindings, secrets and the schedule.
4. Release the lock last.

The broker-first, lock-last order means a crash at any step leaves the namespace locked with no usable credential. That's the safe direction. Our design review found a less safe detail inside step 4.

The bug: notification before release

The original revoke posted "access expired" to the GitHub issue before deleting the lock. If GitHub returned an error, or the token expired, or the issue had been moved, the function failed after removing credentials and before releasing the lock.

Access was correctly gone. The namespace, though, stayed locked, and every new request saw busy. The hourly sweep tried the same notify-then-release sequence and failed the same way. A third-party API outage had turned into an internal availability outage.

The lesson: credential cleanup, lock release and external notification are three separate durability problems, and we had chained them together.

The redesign: enqueue durably, then release

revoke ─► delete broker ─► delete objects ─► enqueue notification (SQS) ─► delete lock
                                                   │                          ▲
                                                   │   enqueue fails? keep a  │
                                                   │   retryable "revoking"   │
                                                   │   lock, sweep retries ───┘
                                                   ▼
                                       notification worker (Lambda)
                                       ├─ validates queue source + immutable scope
                                       ├─ never calls Kubernetes
                                       └─ partial-batch failures → retry → DLQ

The invariants that make this safe:

• A durable enqueue happens before lock deletion. If the enqueue fails, the lock stays in a retryable revoking state. If the callback fails later, the lock has already been released.
• Duplicates are expected. Enqueue and delete aren't transactional, so the worker is idempotent per request ID.
• The notification worker runs on immutable data. It checks issue number, request ID and expired deadline from the message. It never reloads mutable cluster authorization and never contacts Kubernetes. It also compares against the issue's active marker, so a delayed expiry notice can't close a newer request on the same issue.
• Failures are isolated per grant and per cluster. One malformed record or one failing callback can't block unrelated expired grants. A failed broker cleanup also protects its request ID from the orphan sweep. Without that, fault isolation would itself bypass the broker-first order.

Three retry mechanisms, three dead-letter locations

The failure paths around one revoke are easy to blur together. They're separate mechanisms with separate controls:

Scheduler invokes Lambda asynchronously: a successful delivery means Lambda accepted the event, not that the handler succeeded. Configure only one of these and the others drop events silently. We configured failure retention for every path, with each queue encrypted under a customer-managed KMS key. If you do the same, make sure the role writing to each encrypted queue can use the key; otherwise an encrypted DLQ quietly rejects the very messages it exists to keep.

For the SQS-triggered worker we followed AWS's guidance: a visibility timeout of at least 6× the function timeout, ReportBatchItemFailures enabled so one bad message doesn't redeliver a whole batch, and worker concurrency capped so notifications can't starve lifecycle operations.

There's one rollout subtlety. Existing one-time schedules don't pick up a DLQ when the template changes. They get it only when recreated, so grants approved before the rollout expired under the old configuration.

One more failure mode: The Nodes Underneath

A few weeks in, a grant stopped working well before its deadline. The Kubernetes audit log showed a pods/eviction from our node autoscaler, consolidating capacity. The broker and client were bare Pods, and bare Pods have no controller to recreate them. The lock stayed active and the connection instructions in the issue pointed at pods that no longer existed.

Notes for anyone building something similar:

• A PodDisruptionBudget only constrains the Eviction API. It doesn't help with forced node loss, Spot reclamation or an autoscaler that deletes nodes directly. Most autoscalers also honor a per-pod "do not disrupt" signal (in Karpenter, the karpenter.sh/do-not-disrupt annotation). Use it for short-lived, stateful-in-spirit pods like these.
• If you add a controller to recreate the pods, recovery must keep the original absolute deadline, grant generation and scope. A restarted broker that renews access is a privilege-escalation bug with good intentions.
• Preflight should check workload presence, not just lock state and expiry. "Locked but brokerless" is its own state with its own remedy.
• To identify the node afterwards, the audit log's Node responseObject.spec.providerID gave the exact EC2 instance ID. Terminated instances lose their private DNS quickly, so matching on timestamps alone is unreliable. A Spot purchase option also isn't proof of a Spot interruption; establish the autoscaler's actual reason separately.

What an engineer sees

None of this machinery is visible to the person who asked for access. They see:

1. An issue form with four fields.
2. A bot comment with the preflight result: available, busy (held by #NNN until 14:20 UTC, you can join it), or invalid, because ….
3. A reviewer approves.
4. A comment with the exact command to connect, and the time access ends.
5. At that time, the connection closes and a final comment says so.

What I'd tell my past self

1. Model the grant as data with a deadline, not as permissions with a cron job. Immutable scope and an absolute expiry, enforced by the components themselves, make everything else simpler.
2. Fail-closed and recoverable are separate properties. Our expiry never leaked access, but it could still lock a namespace indefinitely.
3. Order operations so that any crash leaves a safe state: remove credentials first, persist intent before releasing anything, release the lock last.
4. Model each retry path explicitly. Delivery retries, execution retries and queue redelivery are three mechanisms with three dead-letter locations.
5. Treat the deploy pipeline's permissions as part of the feature. The bootstrap role, service-specific policy rules and retained resources all blocked the rollout, and none of them was in the application code.