cybercyber

This is part of a series of articles about how one could design a data-centre network, concentrating on Ethernet, although some details can be used for Infiniband or other technologies as well.

• Non-blocking DC network
• Using BGP between hosts and switches
• Achieving leaf redundancy by multi-homing hosts
• Supporting network boot

In this second installment, we will look at how to remove even more L2 networks by using the switches as routers for the hosts.

At the end of the last installment, the fabric between the leaves was routed, but every leaf was still its own little Ethernet: one subnet, one broadcast domain, and every host on that leaf is in it. That is a lot better than one broadcast domain for the whole data centre, and for many setups it’s where you stop. But inside a leaf, it is still a broadcast domain:

• Every host has to receive and look at every ARP broadcast of every other host.
• One misbehaving host floods all of its neighbours.
• The address of a host says which leaf it is connected to, so moving the host means renumbering it anyway.

So let’s make the broadcast domains even smaller.

One network per cable

Every port on the leaf that faces a host gets its own subnet. A /31 has exactly two addresses, one for each end of the cable¹.

The leaf takes the even address, the host takes the odd one and uses the leaf as its default gateway:

ip address add 10.0.1.1/31 dev eth0
ip route add default via 10.0.1.0

The broadcast domain still exists, but it has exactly two members: the host and its leaf. A host can flood as much as it likes; the only one listening is the switch. And the switch is a router, so it does not pass broadcasts on to anybody.

The leaf knows its /31s as connected routes and tells the rest of the fabric about them. In the last article I promised to explain how the switches learn all those routes, so let’s do that first.

BGP in the fabric

Every switch speaks BGP with every switch it has a cable to. This might seem strange, as BGP is what the internet uses between providers, but it is actually the common way to do this; RFC 7938 describes it in detail. OSPF or IS-IS would work as well. I use BGP for two reasons: I can put a filter on every single session, and we will need that as soon as the hosts take part. And it is one protocol from the spine all the way down to the host.

On the switches, that is usually FRR as well (SONiC and Cumulus use it). Other vendors have the same features with a different syntax.

Every router in BGP belongs to an autonomous system, with a number, the ASN. We use external BGP everywhere: every session goes between two different ASNs². The numbers come from the private range (64512 to 65534), and they follow a pattern:

Every leaf has its own ASN, but all spines share one. This is not a typo!

BGP detects loops with the AS path: Every route carries the list of ASNs it has passed through, and a router drops every route that already has its own ASN in it. Now take a route from the leaf in AS 65011. It goes up to spine 1 and down to the leaf in AS 65012, which would offer it up again to spine 2. That path is valid, but it makes no sense: leaf, spine, leaf, spine, leaf. Since the path already contains 65000, spine 2 drops it. So we get this for free: A route never goes up to a spine twice.

Every leaf ends up with exactly two paths to every other leaf, one through each spine. And because both spines are in the same AS, both paths have exactly the same AS path (65000 65012). That is what BGP needs to use both at the same time; FRR does that by default. That is the ECMP from last time³.

So far, so good. Every host can reach every other host, and the fabric figures out by itself how. But go back to the hosts and look at what we have to configure for each one:

• The address on the switch port and on the host. Somebody has to keep a list of which /31 belongs to which port.
• The default gateway on every host.
• The address belongs to the cable, not to the host. Moving a host to another port means renumbering it, and with it every DNS entry, firewall rule and config file that knows that address.
• A single cable or transceiver can take down the host. So can its leaf.

BGP to the host

If the switches can learn their routes with BGP, the host can do the same: Run FRR on the host and give it a BGP session to its leaf.

The address of the host moves away from the cable: It becomes a /32 on the loopback interface, e.g. 10.1.0.21/32 out of 10.1.0.0/16, which is reserved for hosts. While we are at it, the host gets an IPv6 address the same way: 2001:db8:1::21/128 out of 2001:db8:1::/48. The host announces both to its leaf. The /31 stays on the cable for now, but it only carries the BGP session.

The leaf sends a single route back to the host: the default route. So we do not need to configure a default gateway anymore; the host learns it from its leaf. (Where the leaf gets its own default route from — the border routers where the data centre connects to the rest of the world — is not part of this series.)

The hosts get an ASN, too, and they all get the same one: 65100. The loop detection will not hurt us there, because every host only ever gets the default route that its leaf originates, never another host’s route with 65100 in the path⁴.

Yes. The hosts are now part of the routing, and that is the price of this design. A host that announces a default route attracts traffic for the whole internet, and a host that announces another host’s /32 gets some or all of that host’s traffic.

So the leaf has to filter, and you should never rely on the host’s filter. The leaf sends only the default route down and accepts only /32s out of 10.1.0.0/16 from below, and only a few of them.

That does not stop a host from announcing the /32 of its neighbour, though. For that, every port would need a filter with exactly the address of the host behind it, and that is a per-port configuration again. To be fair, plain Ethernet does not stop that either: One gratuitous ARP and the traffic is yours. But in BGP, every announcement arrives at the leaf on a known session, so you can filter it, log it and alert on it, which is a lot easier than watching ARP and neighbour discovery. I am fine with that for hosts that run my own software; I would never let a customer’s machine talk BGP to my switches.

The default gateway is gone from the list. But we have not gained as much as it seems:

• We still configure the /31 on the switch port and the host, now for the BGP session.
• We now also configure ASNs. Every host has to know the ASN of its leaf, and that is a different one on every leaf, so the host’s configuration depends on where it is plugged in.
• The /32 moves with the host, but the /31 and the leaf’s ASN do not. Moving a host still means reconfiguring it.
• A host can announce the /32 of any other host.
• A single cable or transceiver can still take down the host. So can its leaf.

BGP unnumbered

The fix for the first three is two features: no addresses on the cable, and no ASN of the neighbour in the configuration. Both were popularised by Cumulus Networks, and Dinesh G. Dutt describes them in great detail in BGP in the Data Center⁵, which I can only recommend.

The first one removes the addresses from the cable. Every interface with IPv6 has a link-local address in fe80::/10, without anybody configuring it. FRR sends router advertisements on the interface, so each end learns the other’s link-local address, and the BGP session is opened between those two. IPv4 routes are then announced with an IPv6 next hop (RFC 8950, which replaced RFC 5549 that most documentation still cites).

When I first read this, I did not believe that it works. But the gateway’s address is never the destination of a single packet: When a router forwards an IPv4 packet, the destination address in the packet stays the address of the final destination. The router uses the next hop for exactly one thing, finding the MAC address to put into the Ethernet frame. For an IPv4 next hop, ARP answers that question; for an IPv6 next hop, neighbour discovery does.

Either way, what comes out is a MAC address, and the frame with the unchanged IPv4 packet in it goes there. Whether the next hop was written down as IPv4 or IPv6 does not matter at all.

The Linux kernel can install an IPv4 route with an IPv6 next hop since 5.2⁶. The default route on a host looks like this:

default nhid 134 via inet6 fe80::7ec2:55ff:fe51:15e9 dev uplink0 proto bgp metric 20

An IPv4 route via the link-local IPv6 address of the leaf on the other end of uplink0, learned by BGP.

The second one is remote-as external: Instead of the neighbour’s ASN, you only tell FRR to accept any ASN except its own. Please note, that this does not say anything about the router’s own ASN; that one still has to come from somewhere, and in the fabric it is still the pattern from above.

So the first feature takes care of the /31s, the second one of the leaf’s ASN.

On the leaf, this is the whole configuration⁷. Sessions with the same policy are grouped into peer groups: fabric for the two uplinks to the spines (swp51 and swp52), hosts for the ports facing the hosts (swp1 and swp2 here, all of them in a real leaf):

frr defaults datacenter
!
interface lo
 ip address 10.0.0.11/32
!
router bgp 65011
 neighbor fabric peer-group
 neighbor fabric remote-as external
 neighbor swp51 interface peer-group fabric
 neighbor swp52 interface peer-group fabric
 neighbor hosts peer-group
 neighbor hosts remote-as external
 neighbor swp1 interface peer-group hosts
 neighbor swp2 interface peer-group hosts
 !
 address-family ipv4 unicast
  network 10.0.0.11/32
  neighbor hosts default-originate
  neighbor hosts prefix-list DEFAULT-ONLY out
  neighbor hosts prefix-list HOST-LOOPBACKS in
  neighbor hosts maximum-prefix 16 restart 5
 exit-address-family
!
ip prefix-list DEFAULT-ONLY seq 10 permit 0.0.0.0/0
ip prefix-list HOST-LOOPBACKS seq 10 permit 10.1.0.0/16 ge 32

A few things to note:

• frr defaults datacenter is important. Without it, FRR refuses to exchange any routes over an eBGP session without a filter (bgp ebgp-requires-policy), and it waits up to three minutes before it declares a silent neighbour dead. The datacenter profile turns the first off and shortens the timers to a few seconds.
• The leaf needs a loopback address of its own: FRR takes its router ID from it, and without any IPv4 address the router ID is 0.0.0.0 and no session comes up. The leaf announces it, so that I can reach the switch.
• maximum-prefix shuts the session down when the host announces too much; restart 5 brings it back after five minutes, otherwise somebody has to clear it by hand.

There is not a single address of a cable in there! The ports are listed, but the list is the same on every leaf (of the same model), and the only things that differ between two leaves are the ASN and the loopback address.

On the host:

frr defaults datacenter
!
router bgp 65100
 neighbor uplink0 interface remote-as external
 !
 address-family ipv4 unicast
  redistribute connected route-map LOOPBACKS
 exit-address-family
 address-family ipv6 unicast
  neighbor uplink0 activate
  redistribute connected route-map LOOPBACKS
 exit-address-family
!
ip prefix-list LOOPBACKS-V4 seq 10 permit 10.1.0.0/16 ge 32
ipv6 prefix-list LOOPBACKS-V6 seq 10 permit 2001:db8:1::/48 ge 128
route-map LOOPBACKS permit 10
 match ip address prefix-list LOOPBACKS-V4
route-map LOOPBACKS permit 20
 match ipv6 address prefix-list LOOPBACKS-V6

This file contains nothing that belongs to the host. It does not know its own address; it announces whatever is configured on the loopback, and FRR takes its router ID from there, too. It does not know the ASN of the leaf. So the file is the same, byte for byte, on every host⁸. The only things that are the host’s own are the two addresses on lo, and they move with the host⁹. Those two addresses are part of the host’s configuration, which of course lives in git like everything else.

If I unplug a host and plug it into a host port on another leaf, the BGP session comes up and the new leaf learns the /32; the old leaf forgot it as soon as its session went down. I did not have to change a single configuration file!

There is one price I have not mentioned yet: Every switch now has a route for every single host, because the address of a host does not say anything about its leaf anymore, so the leaves cannot aggregate. With 2048 hosts, that is 2048 IPv4 and 2048 IPv6 routes on every switch, plus whatever the VMs and containers announce. Data-centre switches usually handle that just fine, but IPv6 /128 routes are often more expensive in the hardware than IPv4 ones, so have a look at the datasheet before you announce every container.

What is left on the list:

• A host can announce the /32 of another host. For my own hosts, I can live with that.
• A single cable or transceiver can take down the host. So can its leaf, e.g. during a firmware update.

The next article will fix that second one: two cables, two leaves, a software router on the host (in a network namespace, with VPP, or whatever else you like), and ECMP between the host and its leaves. That also means the leaf should stop announcing a default route when it has lost its own uplinks, which it happily does with the configuration above.

And yes, a host without an IPv4 address on its cable cannot PXE boot, and the BMC of a server does not run FRR. That is for the article after that.

1. That is RFC 3021 from 2000. Before that, you needed a /30 for a point-to-point link, so half of the addresses were wasted on network and broadcast address. ↩︎
2. The alternative is internal BGP, where everybody is in the same AS. iBGP does not pass routes learned from one iBGP peer on to another, so you need a full mesh of sessions or route reflectors, and it does not change the next hop by default. eBGP needs none of that. ↩︎
3. With a different ASN on every spine, the two paths would look different to BGP, and you would need bgp bestpath as-path multipath-relax to use both. Keep that in mind for the next article. ↩︎
4. If you ever want to send the hosts more than a default route, it will hurt you: host B would drop the route to host A, because it sees its own ASN in it. Then you either give every host its own ASN (there are enough of them in the 4-byte private range, 4200000000 to 4294967294) or tell it to accept its own ASN in the path (allowas-in). ↩︎
5. O’Reilly 2017, ISBN 978-1-491-98338-6. ↩︎
6. Before that (and in FRR for a while after), FRR installed the route via a made-up IPv4 next hop, 169.254.0.1, and added a static neighbour entry that mapped this address to the peer’s MAC address. It worked, but it was quite a hack. It also shows nicely that the next hop is only ever a key for the MAC lookup: nobody owns 169.254.0.1, and it did not matter. ↩︎
7. Shortened to the IPv4 half. The ipv6 unicast address family looks the same, with IPv6 prefix-lists, and needs the peer groups activated explicitly. ↩︎
8. As long as the interface is called the same everywhere. If it is not, a systemd .link file that renames it to uplink0 fixes that, once per hardware model rather than once per host. ↩︎
9. More than the host’s own address can be announced like that: the addresses of the VMs running on it, or Kubernetes service IPs (MetalLB, Calico, Cilium, kube-router, …). These bring their own BGP speaker, which either talks to the FRR on the host or replaces it. Then an address belongs to whichever host runs the workload right now, and moves when the workload moves. The prefix-list and the maximum-prefix on the leaf need to allow that, of course, with all the trust questions from above. And if several hosts on different leaves announce the same address, the spines need multipath-relax to spread the traffic over all of them. ↩︎