This is the first article in a series about setting up kubernetes for personal use. I will probaly follow these steps:

• kubernetes I: Motivation, Network setup and installing kubernetes
• kubernetes II: Installing the metrics server
• kubernetes III: Ingress-Controller, Cert-Manager and DNS
• kubernetes IV: Storage provider
• kubernetes V: Docker registry and Jenkins
• kubernetes VI: Network policies and Monitoring
• kubernetes VII: Updating kubernetes and activating IPv6
• kubernetes VIII: Walkthrough the ansible roles

(I have since abandoned this plan :-( —pr, 2022-11-17)

I am running a number of servers in different places. One of them is hosting this page. We (that is another person crazy enough to self-host and me) are trying to set thing up in such a way that we can easily and quickly change things. That means that it is important to be able to understand what’s going on at a glance. For us that means that using infrastructure- or configuration-as-code is key: It allows one to understand what parts of a systems configuration are incidental (i.e. default configuration by the distribution) and which parts are deliberate.

Me and my co-admin noticed that many things in our infrastructure are hard to change because of the clunky and inflexible work flow surrounding our configuration-as-code tool. We set about defining a set of requirements for our tool and work flow and began designing a system. We were thinking about a modular ansible role-system that would start docker-containers on a central host. We wanted to be able to define tests for those roles or modules that would (e.g.) start some monitoring solution like icinga and only successfully finish ansible if that gives the OK.

Before implementing that though, we noticed that kubernetes would fit our requirements and that helm-charts can contain tests in a way that is very similar to what we wanted to do.

Since we are operating a number of sites (home-nets, rented servers) we want to build something for which no information seems to exist on the internet. Possibly because it’s stupid or possibly because it just works and there is basically no reason to write about it: A (multi-platform¹) multi-homed kubernetes-cluster. Distributed distributed computing!

Network setup

Kubernetes has a few requirements on the networking and in turn promises certain invariants itself:

• Every node and pod has one IP address each
• Every node and every pod can reach any other pod and node using that IP address (no NAT!)

Kubernetes controls the pod IP addresses itself (the network plugin does) and I just have to handle the node IP addresses. Fortunately, it is not a requirement that all nodes are in the same network segment.

So on both participating sites we added a dedicated network for kubernetes. The routers (two of which are pfSense’s and one is just a network namespace on a VPS) establish an IPSec connection between themselves and use BGP to propagate routes into those kubernetes-networks. Since they are already the default router of their respective network, there is no need to do any special configuration on the nodes to have them reach each other as they need to send traffic to the pfSense anyways.

We are using calico as network plugin for kubernetes. Calico uses BGP internally to setup the routing between nodes, pods and services. It assigns a network-segment (/26) to each node and the IP adresses of pods running on that node are picked from that range. If there are more pods than IPs in that range, calico assigns a second (third, …) range to the node. By default calico does a full-mesh, it peers every node with every other node. We disabled that behaviour and peer a node only with it’s pfSense.

apiVersion: crd.projectcalico.org/v1
kind: BGPConfiguration
metadata:
  name: default
spec:
  nodeToNodeMeshEnabled: false
  serviceClusterIPs:
    - cidr: {{ service_cidr }}

This make for a very nice routing setup where each pfSense knows to send traffic to pods of a certain node either to the node itself (if it is in the local network) or over to the other pfSense which in turn will send it to the node.

The service network is announced by all nodes, so traffic to services gets sent to a local node by the routers. For services where the externalTrafficPolicy is set to Local every node that has a pod belonging to that service will also announce the services IP address as /32 — this allows traffic to be sent to the nearest node.

Even without having all nodes in a single network segment and even have (domestic!) WAN connection between the sites, kubernetes runs without any problem.

Installing kubernetes

To install kubernetes without “magic” for better understanding, we opted to follow the guide by Mumshad Mannambeth. Every step was either scripted (e.g. for PKI) or implemented in ansible. The playbook is rather specialized for our setup and we are only slowly carving out publishable parts of it. You should see them appearing on our github.

So this is the final setup:

This concludes everything we have right now. I hope to add a second post when we are further along and perhaps describe our new workflow. Currently we are too far stuck in creating infrastructure. Which is fun :-D.