Control plane components and etcd

Four processes make the control plane, and knowing which one owns a decision is the difference between fixing a cluster and restarting things hopefully.

Who owns what

• etcd — the only stateful thing. If it is gone, the cluster is gone.
• kube-apiserver — the only process that talks to etcd. Everything else talks to it. Stateless, so you can run several.
• kube-scheduler — decides which node a pod goes to, then writes that decision down. It does not start anything.
• kube-controller-manager — the built-in reconcile loops: Deployments, ReplicaSets, node lifecycle, endpoints.
• kubelet — on every node, the only thing that actually starts containers.

A useful consequence: if the scheduler is down, running pods are completely unaffected and new pods sit in Pending. If the API server is down, running pods keep running and nothing can be changed or observed. Neither is an outage of your application — which is a surprise to most people the first time.

Follow one apply through the system

1. kubectl POSTs to the API server, which authenticates, authorises via RBAC, runs admission, validates, and writes to etcd.
2. The Deployment controller sees the new object and creates a ReplicaSet.
3. The ReplicaSet controller creates Pod objects with no nodeName.
4. The scheduler notices unassigned pods and writes a binding.
5. The kubelet on that node sees a pod assigned to it, pulls the image and starts containers through the CRI.
6. The kubelet reports status back; the endpoint controller adds the pod to Services once it is ready.

Static pods: the bootstrap trick

On a kubeadm cluster the control plane runs as pods — but something has to start them before the control plane exists. The kubelet reads manifests straight off disk:

ls /etc/kubernetes/manifests/
# etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

# moving a file out of this directory stops that component, immediately
sudo mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/
kubectl get pods -n kube-system | grep scheduler    # gone
sudo mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/    # back

That is also the repair path when the API server will not start: you edit the manifest on disk, because you cannot use the API to fix the API.

Back up etcd, then actually restore it

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-2026-10-08.db --write-out=table

The restore is the part worth rehearsing, because it is not symmetrical with the backup. You stop the control plane, restore into a new data directory, point etcd at it, and start everything again:

sudo mv /etc/kubernetes/manifests/*.yaml /tmp/manifests-backup/   # stop control plane

sudo ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-2026-10-08.db \
  --data-dir=/var/lib/etcd-restored

sudo vim /etc/kubernetes/manifests/etcd.yaml   # point hostPath at /var/lib/etcd-restored
sudo mv /tmp/manifests-backup/*.yaml /etc/kubernetes/manifests/

What to actually do with this

• Build a kind or kubeadm cluster you do not care about and destroy etcd deliberately.
• Stop the scheduler for two minutes and watch what breaks — less than you expect.
• Write the restore procedure down as commands, not prose, and keep it somewhere you can reach without the cluster.