Running a cluster

RBAC and multi-tenancy

Roles that are actually least-privilege, and the three places tenancy leaks anyway.

CKASecurityKCSA 10 min read

RBAC is four object types and one rule: permissions are purely additive, and there is no deny. What a subject can do is the union of every binding that mentions it, which means you can never fix an over-permissive grant by adding a narrower one.

Four objects, two scopes

  • Role — permissions, inside one namespace.
  • ClusterRole — permissions, cluster-wide or reusable.
  • RoleBinding — grants a Role or a ClusterRole inside one namespace.
  • ClusterRoleBinding — grants a ClusterRole everywhere.
The combination people miss: a RoleBinding referencing a ClusterRole. That is how you define “developer” once and grant it in thirty namespaces without thirty copies of the rules. The single most useful pattern in RBAC.

Ask the API rather than reading YAML

kubectl auth can-i --list --as=system:serviceaccount:team-a:deployer -n team-a
kubectl auth can-i delete pods --as=jane -n prod
kubectl auth can-i '*' '*' --as=system:serviceaccount:team-a:deployer   # the scary one

Reviewing bindings by eye does not work at any real scale, because the answer depends on role aggregation and on bindings in other namespaces. can-i --list gives you the effective answer, which is the only one that matters.

The escalation paths that are not obvious

  • create pods in a namespace means you can mount any Secret in that namespace and run as any ServiceAccount in it. It is effectively read access to every credential there.
  • create pods/exec is a shell in a running container, bypassing whatever the image entrypoint was.
  • escalate or bind on roles lets a subject grant itself permissions it does not have. RBAC normally prevents privilege escalation; these verbs are the exemption.
  • get secrets cluster-wide is usually equivalent to cluster-admin in practice, because one of those secrets is a token for something that is.
Namespaces are not a security boundary on their own. They scope names and RBAC, and nothing else. By default a pod in one namespace can reach a pod in another over the network, both can exhaust the same node, and both see the same cluster-scoped objects. Real tenancy needs network policy, quotas and admission control on top.

Three leaks to close

  1. Network. Without NetworkPolicy, every pod can reach every pod. A default-deny policy per namespace is the first real wall.
  2. Resources. Without a ResourceQuota and LimitRange, one tenant can take a whole node. Noisy-neighbour problems are tenancy problems.
  3. Node and kernel. Pods share a kernel. A privileged container, a hostPath mount or hostNetwork escapes the namespace entirely — which is what Pod Security Admission is for.
# the baseline worth having in every tenant namespace
apiVersion: v1
kind: Namespace
metadata:
  name: team-a
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted

Default ServiceAccount tokens

Every pod gets a ServiceAccount token mounted unless you say otherwise. Most workloads never call the API, so that token is pure downside — it is the first thing anything inside a compromised container goes looking for.

spec:
  automountServiceAccountToken: false     # on the pod, or on the ServiceAccount

What to actually do with this

  • Run can-i --list for your most-used ServiceAccount. Expect a surprise.
  • Find every binding of cluster-admin and justify each one out loud.
  • Turn off token automounting on one workload that does not need it, and watch nothing break.
This article covers one checkpoint on the roadmap. Open RBAC and multi-tenancy on the roadmap → โ€” it lists what this depends on and everything else written about it.

Something wrong or out of date? Open an issue โ€” corrections are welcome and get credited.