cybercyber

Everybody I have explained kubernetes to got stuck in the same place: there is a pile of YAML on one side and a running program on the other, and everything in between is a fog of words — controllers, operators, reconciliation, admission, CRDs.

I have stopped trying to explain the middle. I draw three columns instead.

The three columns

Column 1: documents. The manifests. The things you kubectl apply. They are inert: a Deployment does not do anything, it is a piece of text in a database with a schema attached to it.

Column 2: code. Programs that read documents and act on them. Controllers, the scheduler, the kubelets, operators, CSI and CNI plugins.

Column 3: the real world. Processes, files, routing tables, LUNs, a row in a database, an entry in a firewall. The things that are still there when the API server is down.

That is the whole model.

It is. Let us see how far it gets us anyway.

One Deployment, all the way down

Read it downwards and watch the ping-pong: document, code, document, code, document, code. Every controller in that chain does exactly one thing — it reads a document out of column 1 and writes another document into column 1. None of them talk to each other. They talk to the API server, and the API server is just the place where the documents live.

The step I like best is the scheduler. A whole control-plane component, with a plugin framework and scoring and years of work in it, and its output is one field in one document¹. It starts nothing. It does not talk to a node. It writes spec.nodeName and goes back to sleep.

And note what has not happened yet: three control-plane components have run, and so far the only effect anywhere is that some text changed.

The real world gets touched exactly once, at the very bottom, when containerd finally starts a process². Everything above that line is documents and code.

It does not stop at pods

That chain is the one everybody learns. The reason the model is worth anything is that it does not only describe that chain.

At any point in column 1 you can put a different document. At any point in column 2 you can put different code. And column 3 can be anything that code is able to reach:

The third row is the one that usually gets a reaction. The very same NetworkPolicy document, in a cluster where different code is watching it, ends up as a rule in a piece of hardware in a rack instead of as nftables entries on a node. The document did not change. The column 2 code did.

Once you see that, custom resources stop being an advanced topic. A CRD is “let me put my own documents into column 1”. An operator is “let me put my own code into column 2”. That is not an extension mechanism bolted onto kubernetes afterwards — it is the only mechanism kubernetes has ever had. The built-in controllers work in exactly the same way.

Why this makes my life easier

Because it turns “kubernetes is not doing the thing” into three questions, always in this order:

1. Is the document there, and does it say what I think it says? kubectl get -o yaml. Not the file on my disk — the document in the cluster. Defaulting, mutating webhooks and a colleague’s kubectl edit all live between the two.
2. Is there code watching that document, and is it running? Surprisingly often the answer is no. The CRD is installed but the operator is not. Or it crash-looped an hour ago. Or its RBAC does not let it see that namespace. A document that nobody watches is a text file with extra steps.
3. Did that code manage to change the real world? Now, and only now, I go to the node and look at processes, mounts, nft list ruleset.

In that order. I have not yet needed a fourth question — debugging kubernetes is mostly finding out which of the arrows is not being traversed.

There is one more thing hiding in column 3, by the way: .status. That is the real world being reported back into column 1, written by the same code that did the work. Which is why kubectl describe and its events are so often where the answer is: that is column 2 telling you, in a document, what went wrong out in column 3.

For me, this way of thinking has made working with kubernetes extremely easy.

If you want the same idea taken a lot further by somebody who has clearly been thinking about it longer than I have, watch Carson Anderson’s talk “kubernetes deconstructed”.

1. It does not even write the Pod itself: it POSTs a Binding to pods/<name>/binding and lets the API server put the value into spec.nodeName. Still a document. ↩︎
2. Which is the same point I made in another article, coming from a different direction. ↩︎