Bounding an Agent’s Authority During Kubernetes Incident Response

An agent can summarize a Kubernetes incident, correlate evidence and suggest a next step. The hard part begins when it can call tools that read from or change a cluster.

The safety question is not whether the model usually chooses the right command. It is whether a wrong classification, misleading event or prompt injection can turn into an API operation the caller was never allowed to make.

I designed and built an agent-assisted incident-response platform around a simple separation: classification chooses a route; authorization decides whether that route may act. The system normalizes operational events, resolves a specialist capability, connects remote cluster agents to a central gateway over Server-Sent Events (SSE), and uses cluster-scoped grants alongside Kubernetes RBAC. The project name, identifiers and deployment-specific settings are omitted here; the architecture and permission boundaries are retained.

The authority boundary

Treat these as three different decisions:

An LLM output such as mode=investigate is not a permission. Neither is an agent's tool description, a retrieved runbook, a confidence score, or the fact that an agent is connected to a cluster. Those can help choose work. They must not expand the set of API operations available to the workload identity.

Request path across clusters

The flow below is conceptual and uses generic names. It shows where data and authority should be checked separately.

alert or operator request
        │
        ▼
normalize event ──► classify intent ──► resolve explicit cluster + namespace
                                             │
                                             ▼
                                grant: caller/team + cluster
                                      → namespace + capability
                                       │                │
                                   denied or        authorized route
                                   unresolved             │
                                                         ▼
central gateway ──SSE commands + heartbeats──────────► cluster agent
central gateway ◄──authenticated result POST───────── cluster agent
                                                         │
                                                         ▼
                                                Kubernetes API server
                                                Role / RoleBinding / SA

In the implementation, a grant resolver maps a team and target cluster to the namespaces and capabilities available there. The router resolves a capability to a visible agent route. A remote agent maintains an SSE connection to the gateway, which streams commands and heartbeats to it; the agent posts results back through a separate authenticated request. At the cluster, Kubernetes RBAC determines which API requests the agent's service account can make.

The connection is a transport path, not an authorization token. Losing and re-establishing an SSE stream must not create a broader identity or permission set. The agent should authenticate as a workload identity, and the gateway should bind dispatch to the cluster represented by that authenticated connection rather than accepting a cluster name supplied by generated text.

Route first, authorize independently

Event normalization gives the router a stable shape to work with. Classification can then choose a domain and mode, such as a read-oriented lookup or a deeper investigation. The router resolves that intent to a capability and a concrete agent route. This makes routing inspectable: a decision can record the selected capability, target context, reason and whether a fallback was used.

That decision still does not authorize the operation. Before dispatch, the application layer should check the caller's grant against the resolved cluster, namespace and required capability. The agent's Kubernetes identity then provides a second, independent enforcement point. If the model is manipulated into asking for an operation outside its role, the Kubernetes API should reject it even if the request passed through the router.

The same rule applies to fallback. A fallback is a new route decision, not permission inherited from the failed primary. It must resolve against the same target constraints and pass its own authorization check. If no unique cluster can be resolved, do not fan out to every connected cluster. Return a request for clarification or use a deliberately narrower path whose authority is independently checked.

Give each workload identity the smallest useful role

For a read-only incident investigator, a namespace Role might start with access to pods, events and pod logs. This is an illustrative baseline, not a complete production manifest:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: incident-reader
  namespace: example
rules:
  - apiGroups: [""]
    resources: ["pods", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]

Bind that role to a dedicated service account with a RoleBinding in the same namespace. Avoid handing a general investigator secrets, pods/exec, wildcard resources or wildcard verbs as a convenience. Kubernetes treats subresources such as pods/log and pods/exec as distinct authorization checks; enumerate the exact subresource and verb that the workflow needs.

Keep read and mutation authority separate. A request classified as a change should not make a read-only identity more powerful. If a workflow supports a mutation capability, define its specific resources and verbs, bind it to a separate identity, and keep the application grant equally narrow. An approval step may be useful for a product, but it does not replace either authorization check.

Multi-cluster ambiguity and fallback

Capability discovery should return only routes visible for the request's authorization context. A route should carry a concrete cluster reference, not just an agent type. If the event does not identify a unique target, selecting the first available cluster or broadcasting the request creates surprising data access.

There are two safe outcomes for an unresolved target:

1. Ask the operator to select the intended cluster or namespace.
2. Continue only with an explicitly defined, low-authority fallback, such as explaining what context is missing without querying cluster data.

In the implementation, unresolved capability selection follows a fallback path with a runtime policy envelope. That envelope can bound execution work; it does not replace a unique target, the normal grant check or target-side RBAC. A smaller token budget or a different model role may reduce work, but neither is an access control.

Revocation and cached grants

Caching grant lookups reduces repeated reads from the grant store, but it changes the revocation guarantee. If a grant is removed while a positive entry remains cached, the application may continue to authorize work until the cache expires or is invalidated. Describe that stale-authorization window honestly and size it to the risk.

For a robust design:

• Keep grant cache lifetimes bounded and invalidate entries on grant changes where possible.
• Make the target-side Kubernetes role binding part of the same lifecycle, so API authorization is removed when access is revoked.
• For high-risk operations, use a fresh authorization check or short-lived workload credential rather than relying only on a cache entry.
• Treat a grant-store error as an authorization failure for operations that need the grant; do not silently fall back to a broad default.

The application grant and Kubernetes RBAC solve related but different problems. The grant expresses which caller may invoke which capability on a cluster and namespace. RBAC limits what the agent's Kubernetes identity can do even if the application makes a mistake. For temporary grants, define expiry and cleanup for both; for longer-lived grants, make the revocation path explicit.

Failure behavior to design before an incident

These are useful review cases because they test the boundary between orchestration and enforcement. A safe response is not always “keep retrying”: on ambiguous scope or missing authority, retrying a broader route is escalation.

Make the decision observable

During an incident, an operator should be able to answer: what intent was classified, which capability was selected, what target scope was used, whether a fallback ran, and whether the operation was allowed or denied. Record those decisions as structured events with a correlation identifier, while redacting secrets and unnecessary prompt content.

Metrics can track bounded dimensions such as mode, capability class, decision and fallback status. Keep high-cardinality or sensitive values—raw prompts, resource names, user identifiers and cluster identifiers—out of metric labels; put the minimum necessary detail in access-controlled logs or traces instead. This keeps telemetry useful without turning it into a second data-exfiltration path.

Useful measures for evaluating a deployment include authorization-denial counts by reason, unresolved-target frequency, fallback frequency, API permission errors and time from accepted request to a useful result. These are measurement candidates, not performance results from this project; I do not claim an MTTR or incident-cost reduction without a recorded baseline and measurement window.

The practical rule

An agent can propose a plan and choose among capabilities that the platform has made visible. It should not choose its own authority. Resolve the target from trusted context, check a scoped application grant, dispatch through an authenticated cluster connection, and let Kubernetes RBAC enforce the final API boundary. When any of those facts is missing, keep the response inside the smaller boundary.

Further reading

• Kubernetes RBAC authorization
• Kubernetes authorization checks for API requests
• Server-Sent Events in the HTML Standard
• Prometheus metric and label naming practices