As I had hinted at in another article, I am currently building a distributed kubernetes cluster from a variety of machines. One of that machines is a virtual private server that (of course) has a public IP configured. Having a node which has a public IP hinders you in using NodePort easily; you would need to add a DNAT entry for incoming traffic. But if you want to do any NAT or firewalling, you need cooperation of the kubernetes network plugin you are using (calico in my case). Firewalling is easy, but NAT is not. Please note, that outgoing traffic from a pod needs to be NATed, as my provider will not handle traffic coming from an internal IP on the public interface. I had experimented with natOutgoing, but that messed up traffic from pods running on the node to pods running on other nodes, as they would suddenly come from a public IP…

The solution? Move the network interface of the machine into its own network namespace, add a virtual ethernet pair between the root namespace and the new namespace and configure the network-stack in that namespace to do routing and NAT.

I have wrapped all necessary steps into an ansible role, but I will go through the steps here.

Step 0: Experience linux namespaces

The namespace support of the linux kernel is what allows such technologies as “containers”. There are a number of namespaced resources in the linux kernel (network, process ids, mountpoints and many others). See this article for an in-depth walkthrough.

If you enter into a network namespace there is absolutely no network connectivity of a sudden:

$ sudo ip netns add foo
$ sudo ip netns exec foo bash
# ip addr show
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

In this namespace everything network is totally separate from the rest of the system: Interfaces, routes, iptables rules, even the sysctl parameters controlling IP-forwarding. Basically, it feels like a totally different computer.

Let’s use that “new computer” as a router for our VPS:

Step 1: Create the namespace, move the interface, configure the interface

I use debian on my servers, so I configured the networking in the file /etc/network/interfaces.

auto ens3
iface ens3 inet manual

First of all, I set the interface to be configured manually. That means that we have to provide commands to be called for each step of setting up the interface.

Before the interface is configured, the namespace is created and the interface is moved into it.

  pre-up ip netns add ens3 && ip link set ens3 netns ens3
  down ip netns del ens3

To de-configure the interface, all we need to do is remove the namespace and the interface will pop back into the root namespace as an unconfigured interface.

  up ip netns exec ens3 ip addr replace 1.2.3.4/24 dev ens3 && ip netns exec ens3 ip -6 addr replace 2123:2134:abc:ddd::3/64 dev ens3 && ip netns exec ens3 ip link set up dev ens3

Now the IP-addresses is configured. For IPv6, I choose the IP ending in :3, where the server had one ending in :2 before.

Next, we configure the routes into the new and old internet.

  post-up ip netns exec ens3 ip route replace default via 1.2.3.1
  post-up ip netns exec ens3 ip -6 route replace default via 2132:2134:abc:ddd::1

And configure routing in the namespace:

  post-up ip netns exec ens3 bash -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'
  post-up ip netns exec ens3 bash -c 'echo 1 > /proc/sys/net/ipv6/conf/all/forwarding'
  post-up ip netns exec ens3 iptables -t nat -A POSTROUTING -o ens3 -j MASQUERADE

As the server still needs to answer to the IPv6 address ending in :2, we ask the namespace to do that:

  post-up ip netns exec ens3 bash -c 'echo 1 > /proc/sys/net/ipv6/conf/all/proxy_ndp'
  post-up ip netns exec ens3 ip -6 neigh add proxy 2132:2134:abc:ddd::2 dev ens3

If we now add the virtual device (see below), this is all we need to have internet access in the root namespace. I added two more lines:

  post-up ip netns exec ens3 iptables -t nat -A PREROUTING -d 1.2.3.4 -p tcp --dport 22 -j DNAT --to-destination 192.168.158.2
  post-up ip netns exec ens3 iptables -t nat -A POSTROUTING -o vens3 -s 192.168.158.2 -j MASQUERADE

The first one does a port forwarding for SSH into the root namespace (which has IP 192.168.158.2, see below). The second one ensures that when the host itself (from the root namespace) tries to contact its own public ip (1.2.3.4), the traffic will be sent back through the virtual interface as if it originated from the non-root namespace. Otherwise the root namespace would send traffic to 1.2.3.4 and receive it back with its own IP address as source.

Step 2: Create virtual interface between the namespaces

auto vens3
iface vens3 inet static
  address 192.168.158.2
  netmask 255.255.255.0

This time we can actually leverage the operating system to configure our IP addresses!

But first, the interface needs to be created:

  pre-up ip link add vens3 type veth peer vens3 netns ens3

And the other half needs to be configured (note that we are also adding a static IPv6 address, that will become important when configuring routing for IPv6):

  pre-up ip netns exec ens3 ip addr add 192.168.158.1/24 dev vens3 && ip netns exec ens3 ip -6 addr add fe80::1 dev vens3 && ip netns exec ens3 ip link set up dev vens3

Now we can configure the default route through the namespace:

  post-up ip route replace default via 192.168.158.1

And tell the operating system to delete the interface if is it deconfigured:

  post-down ip link del vens3

Repeat for IPv6:

iface vens3 inet6 static
  address 2132:2134:abc:ddd::2/64
  post-up ip -6 addr add fe80::2 dev vens3

We add a static link local IPv6 here as well, as that will be the address the namespace will use to route our public IP:

  post-up ip -6 route replace default via fe80::1 dev vens3
  post-up ip netns exec ens3 ip -6 route replace 2134:2134:abc:ddd::2/128 via fe80::2 dev vens3

We add a default router via the static link local IPv6 of the namespace and we tell the namespace how to reach us.

Step 3: Add port-forwardings

The ansible role also support adding port forward rules, they are implemented by adding DNAT iptables rules to the namespace, just as we saw for SSH.

Have fun!