RBAC & ServiceAccounts

RBAC answers one question per request: may this subject perform this verb on this resource? It is entirely additive — there are no deny rules — which makes it simple to reason about and easy to over-grant.

Topic 1: The Four Objects

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: payments
  name: pod-reader
rules:
  - apiGroups: [""]                       # "" = the CORE group
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["api-config-secret"]  # ONE named secret only
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: payments
  name: read-pods
subjects:
  - kind: ServiceAccount
    name: api-sa
    namespace: payments
  - kind: User
    name: alice@example.com
    apiGroup: rbac.authorization.k8s.io
  - kind: Group
    name: platform-team
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Details that matter:

apiGroups: [""] is the core group — pods, services, secrets, configmaps, nodes. Deployments are in apps; ingresses in networking.k8s.io. Getting the group wrong is the second-most-common RBAC mistake, and the error is an unhelpful “forbidden”.

Subresources are separate resources. pods does not grant pods/log, pods/exec or pods/portforward. Granting pods/exec is effectively granting shell access to whatever runs in those pods — treat it as a privileged verb.

resourceNames narrows to specific objects, which is how you grant access to one Secret rather than all of them. Note it does not work with list or watch (you cannot list a filtered set), only with get, update, patch and delete.

roleRef is immutable. To change which Role a binding grants, delete and recreate it.

The verbs:

get  list  watch  create  update  patch  delete  deletecollection

Plus non-resource verbs for URLs (/healthz), and impersonate, bind, escalate — the last three being privilege-escalation vectors worth auditing specifically.

Topic 2: A ClusterRole Bound Two Ways

This is the piece that unlocks the model. A ClusterRole is just a set of permissions; where they apply depends on the binding:

# Bound cluster-wide → read secrets EVERYWHERE
kind: ClusterRoleBinding
roleRef: { kind: ClusterRole, name: secret-reader }
---
# The SAME ClusterRole, bound in one namespace → read secrets in THAT namespace only
kind: RoleBinding
metadata: { namespace: payments }
roleRef: { kind: ClusterRole, name: secret-reader }

This is why you define common permission sets once as ClusterRoles and bind them per namespace, rather than duplicating a Role into forty namespaces.

Some resources are cluster-scoped (nodes, PVs, StorageClasses, CRDs, namespaces themselves). Access to those requires a ClusterRole and a ClusterRoleBinding — a RoleBinding cannot grant them, because they do not live in a namespace.

The built-in ClusterRoles:

kubectl get clusterroles | grep -v '^system:'

view excluding Secrets is deliberate and worth knowing — people grant view expecting it to be safe, which it largely is, precisely because of that exclusion.

Topic 3: ServiceAccounts — Identity for Pods

apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-sa
  namespace: payments
automountServiceAccountToken: false      # default to OFF
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      serviceAccountName: api-sa
      automountServiceAccountToken: true  # opt IN only where needed

Every pod runs as a ServiceAccount — the namespace’s default one if you do not say otherwise. Its identity in RBAC is:

system:serviceaccount:<namespace>:<name>

Turn off token automounting unless the pod actually calls the API. Most applications never talk to the Kubernetes API, and a mounted token is a credential an attacker can use after compromising the container. Set automountServiceAccountToken: false on the ServiceAccount and opt in per workload.

Tokens changed in 1.24:

Creating a ServiceAccount no longer creates a permanent Secret. Pods receive a projected token that is:

• Time-bound — expires and is rotated by the kubelet automatically.
• Audience-bound — valid only for the intended recipient.
• Object-bound — invalidated when the pod is deleted.

volumes:
  - name: token
    projected:
      sources:
        - serviceAccountToken:
            path: token
            expirationSeconds: 3600
            audience: vault

A leaked bound token expires. A leaked legacy token is valid forever. If you find long-lived kubernetes.io/service-account-token Secrets in your cluster, that is technical debt worth clearing.

Cloud identity federation:

Rather than putting cloud credentials in a Secret, map a ServiceAccount to a cloud role:

metadata:
  name: api-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/api-role      # IRSA
    iam.gke.io/gcp-service-account: api@project.iam.gserviceaccount.com      # Workload Identity

The pod’s projected token is exchanged for cloud credentials via OIDC. No static secret exists to leak, and rotation is automatic. This is the correct pattern on every major cloud.

Topic 4: Debugging Permissions in One Command

kubectl auth can-i create deployments -n payments
kubectl auth can-i delete pods --as=system:serviceaccount:payments:api-sa -n payments
kubectl auth can-i get secrets --as=alice@example.com -n payments
kubectl auth can-i '*' '*' --all-namespaces
kubectl auth can-i --list --as=system:serviceaccount:payments:api-sa -n payments

kubectl auth can-i --list --as=... prints everything a subject may do. It answers most RBAC questions without reading a single YAML file, and it is the first command to run — not the last.

# Who is bound to cluster-admin? (audit this regularly)
kubectl get clusterrolebindings -o json | jq -r '
  .items[] | select(.roleRef.name=="cluster-admin") |
  "\(.metadata.name): \(.subjects // [] | map(.kind+"/"+.name) | join(", "))"'

# What does this Role actually allow?
kubectl describe role pod-reader -n payments
kubectl get clusterrole edit -o yaml

# Who am I right now?
kubectl auth whoami

Reading the error properly:

Error from server (Forbidden): pods is forbidden:
  User "system:serviceaccount:payments:api-sa" cannot list resource "pods"
  in API group "" in the namespace "monitoring"

The message contains everything: the subject, the verb, the resource, the API group, and the namespace. Most fixes are visible in it — here, the binding exists in payments but the request was for monitoring.

Topic 5: Least Privilege in Practice

# A controller that manages only its own CRD, plus the resources it creates
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: widget-operator
rules:
  - apiGroups: ["example.com"]
    resources: ["widgets", "widgets/status", "widgets/finalizers"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]

Principles worth holding to:

1. Start from nothing and add what fails. Deploy with no permissions, read the Forbidden errors, grant exactly those. It is faster than it sounds and produces a genuinely minimal Role.
2. Prefer namespaced Roles. A ClusterRole should be a deliberate decision.
3. Never grant * on *. Enumerate; a new resource type should not be automatically in scope.
4. Watch the escalation verbs: escalate, bind, impersonate, create on pods (you can mount any Secret), pods/exec, and create/patch on ValidatingWebhookConfigurations.
5. create on pods is close to node-level access. Anyone who can create a pod can mount hostPath, run privileged, or use another ServiceAccount’s token — which is why Pod Security Admission (next lesson) exists alongside RBAC.

Auditing:

kubectl get clusterrolebindings,rolebindings -A -o wide | grep -i cluster-admin
kubectl get sa -A -o json | jq -r '.items[] | select(.automountServiceAccountToken != false) | "\(.metadata.namespace)/\(.metadata.name)"'

Tools worth knowing: kubectl-who-can (reverse lookup — who can do X), rbac-tool (visualise and generate least-privilege roles), and kubectl auth can-i --list as the always-available baseline.

Topic 6: Where RBAC Sits Among the Others

RBAC is one of several authorisation modes, and the API server can run more than one:

--authorization-mode=Node,RBAC

• Node — restricts each kubelet to objects relevant to its own node. Without it, one compromised node could read every Secret in the cluster.
• RBAC — everything above.
• ABAC — legacy, file-based, requires an API server restart to change. Avoid.
• Webhook — delegate to an external service.

Modes are evaluated in order and any allow wins. This is why RBAC has no deny rules: a deny would be meaningless when another authoriser could still permit the request.

Beyond RBAC: admission control (next lessons) enforces what an object may look like, which RBAC cannot express. “May create pods” is RBAC; “may not create privileged pods” is admission. You need both.

Topic 7: The Audit Log — Who Actually Did It

RBAC answers who may. Once something has already happened, the question is who did — and the only place that answer exists is the API server audit log. It is disabled by default on several distributions, and it cannot be reconstructed after the fact: if it was not running when the deletion happened, the record does not exist.

Every request produces an event that answers five questions:

apiVersion: audit.k8s.io/v1
kind: Event
level: RequestResponse
stage: ResponseComplete
verb: create                                  # WHAT happened
objectRef:                                    # ON WHAT it happened
  resource: secrets
  namespace: payments
  name: db-credentials
user:                                         # WHO did it
  username: system:serviceaccount:ci:build-bot
  groups: ["system:serviceaccounts", "system:authenticated"]
sourceIPs: ["10.0.4.19"]                      # WHERE it came from
requestReceivedTimestamp: "2025-03-11T09:14:22.104Z"   # WHEN
stageTimestamp: "2025-03-11T09:14:22.139Z"
responseStatus:
  code: 201

The four stages

One request can emit an event at more than one point in its life:

ResponseComplete is the one you want. Leaving RequestReceived on roughly doubles volume and adds nothing unless you are specifically hunting requests that never completed.

The four levels

The policy: first matching rule wins

Rule order is the entire design, and it is where policies go wrong. A permissive catch-all near the top silently disables everything below it.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  # 1. Secrets FIRST, at Metadata only — never record their bodies.
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]

  # 2. Drop the noise: control-plane components reading constantly.
  - level: None
    users: ["system:kube-scheduler", "system:kube-controller-manager"]
    verbs: ["get", "list", "watch"]

  - level: None
    nonResourceURLs: ["/healthz*", "/readyz*", "/livez*", "/version", "/metrics"]

  # 3. Everything that changes state: full bodies.
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete", "deletecollection"]

  # 4. Everything else.
  - level: Metadata

Rule 1 must come before rule 3. Reverse them and create secret matches RequestResponse first, writing every Secret you own into the audit log in plaintext — where it is now replicated to whatever log store you ship to, usually with broader read access than the Secret had. That is an audit control that has become the breach.

Two backends

• log — JSON lines to a file on the control-plane node: --audit-log-path, plus --audit-log-maxage, --audit-log-maxbackup and --audit-log-maxsize. Simple, and lost with the node — which is a problem, because “the node was replaced” is a plausible thing for an attacker to arrange.
• webhook — batches POSTed to an external service. Survives the node, and adds a dependency: a slow endpoint applies backpressure to the API server itself, so it is worth treating as a component that can take the control plane with it.

On a managed control plane (EKS, GKE, AKS) you do not pass these flags. You enable audit logging through the provider, it lands in CloudWatch / Cloud Logging / Azure Monitor, and the policy is fixed or only partly configurable — check what yours actually records before assuming a question is answerable.

What it answers that nothing else can

• Who deleted the Deployment at 02:14 — and was it a human or a controller? (system:serviceaccount: prefix tells you)
• Which identity read that Secret, and from which IP?
• Did the change come from CI or from somebody’s laptop? (sourceIPs with user)
• What did the object look like before the patch that broke it? (RequestResponse keeps both)

Events expire in an hour and tell you what Kubernetes did. The audit log persists and tells you what people and controllers did. They are not substitutes.

A note on cost: RequestResponse on every write is a large volume of data on a busy cluster, and audit logs are usually the second-biggest line in an observability bill after application logs. Filter at the policy — dropping a rule there costs nothing, while dropping it at the log pipeline means you paid to ingest it first.

Try it yourself: Create a ServiceAccount with no bindings at all, run a pod as it, and from inside the pod curl the API server using the projected token. Read the Forbidden response, then grant exactly the one permission it names and try again.

Common mistake: Debugging RBAC by editing YAML and redeploying. kubectl auth can-i --as=<subject> --list answers the question in one command, against the live cluster, with no deployment at all — and it is the same evaluation the API server performs.