Demystifying the Kubernetes control plane: what happens under the hood

Most developers can deploy a pod. Far fewer can say exactly how Kubernetes turns one command into a running container. Here is the machinery — followed step by step, from a single kubectl apply to a pod on a node.

You type kubectl apply -f deployment.yaml, press enter, and a few seconds later a container is running somewhere in your cluster. It feels like magic, which is another way of saying you have not yet seen the mechanism. There is no magic — there is a small set of components on the control plane passing a piece of desired state between them, each doing one job well. Follow one command through them and Kubernetes stops being mysterious and starts being legible.

01A day in the life of a single command

Here is the whole journey in one breath, before we slow it down. Your kubectl apply becomes an HTTPS request to the API server, which authenticates it, validates it, and writes the desired object into etcd. A controller notices the new Deployment and creates the Pods it implies. The scheduler notices those Pods have no home and assigns each to a node. The kubelet on that node sees a Pod assigned to it and tells the container runtime to actually start the container.¹ Every step is one component reacting to a change in stored state — never a chain of direct commands.

# The command that kicks off the whole dance
kubectl apply -f deployment.yaml

# What it becomes: an authenticated HTTPS write to the API server,
# which persists the object in etcd. Everything else reacts to that.

02The API server: the only door in

The kube-apiserver is the front door to the cluster, and importantly the only door: every other component — the scheduler, the controllers, the kubelets on each node, and your kubectl — talks to etcd exclusively through it, never directly.¹ That single-entrance design is what makes the cluster governable. Every request passes through authentication (who are you?), authorization (are you allowed?), and admission control (is this object acceptable, and should it be mutated or rejected?) before anything is persisted.²

Because it is the only writer to the datastore, the API server is also where consistency is enforced. It is stateless itself, which is a deliberate gift: you can run several replicas behind a load balancer and lose one without losing the cluster, because none of them holds state — the state lives in etcd.

03etcd: the cluster's single source of truth

Everything Kubernetes knows about itself — every object, its spec, and its status — lives in etcd, a distributed, strongly-consistent key-value store.³ This is the most important sentence in the article: etcd is the cluster. The nodes, the API server replicas, the controllers — all of them are replaceable and can be rebuilt. etcd holds the only irreplaceable thing, which is the truth about what should exist.

etcd stays consistent across its replicas using the Raft consensus algorithm, which requires a majority — a quorum — of members to agree before any write is committed.⁸ That is why etcd clusters are sized as odd numbers (typically three or five): a three-member cluster keeps working if one member dies, because two of three is still a majority, but loses write ability if two die. Kubernetes' own documentation is blunt about the stakes and insists on a backup plan, because a cluster whose etcd data is lost and unrecoverable is, for practical purposes, gone.⁴ Taking regular etcd snapshots is not optional housekeeping; it is the difference between a bad afternoon and rebuilding from scratch.

# The one backup that actually matters
ETCDCTL_API=3 etcdctl snapshot save snapshot.db

# Lose your nodes and you reschedule. Lose etcd with no snapshot
# and you have lost the cluster's memory of what should exist.

04The scheduler: matchmaking pods to nodes

When a Pod is created it starts life unscheduled — it exists as desired state but has no node. The kube-scheduler watches for exactly these homeless Pods and, for each one, runs a two-stage decision: filtering (which nodes could even run this Pod, given its resource requests, node selectors, taints and tolerations, and affinity rules?) and then scoring (of the feasible nodes, which is best?).⁵ It picks a winner and writes that choice back — it does not start anything itself. It only decides where.

This is the pattern worth internalising: the scheduler's entire job is to turn one field on the Pod object — the node assignment — from empty to filled. Something else acts on that decision.

05Controllers: the reconciliation loop that never sleeps

The kube-controller-manager runs a collection of controllers, and controllers are the beating heart of Kubernetes. Each one runs the same simple loop forever: observe the current state of some part of the world, compare it to the desired state, and take one step to close the gap — then do it again.⁶ The Deployment controller notices you asked for three replicas but only two exist, and creates one. The moment a node dies and its Pods vanish, the relevant controllers observe the shortfall and create replacements, which the scheduler then places.

This reconciliation loop is why Kubernetes is self-healing rather than merely automated. You never tell it how to recover from a failure; you tell it what should be true, and controllers spend the rest of eternity making reality match — which is also why kubectl apply only ever declares intent and never issues step-by-step commands.⁶

Kubernetes does not execute your deployment. It records what you want, and then a hundred small loops spend forever making the cluster look like your description — including the moment after a machine catches fire.

06Why the control plane must be highly available

Now the payoff. If the whole control plane is down, your already-running Pods keep serving traffic for a while — the kubelets on each node carry on with what they were last told. But the cluster goes blind and deaf: nothing new can be scheduled, no failure can be healed, no change can be applied, because the only door in and the only memory are both unreachable. A control-plane outage is not an inconvenience; it is the loss of the cluster's ability to react.

That is why production clusters run the control plane highly available — multiple API server replicas and a multi-member etcd cluster, spread across failure domains, so losing one machine costs you nothing.⁷ For etcd specifically, high availability and regular snapshots together form the survival plan; the documented disaster-recovery path depends on having those snapshots.⁴ Treat the control plane as the crown jewels, because it is: the nodes are cattle, but the control plane is the herd's memory of what the herd is supposed to be.