Post

MetalLB on K3S, with a UDM SE

Posted
By Emma Ruby
6 min read

Overview

Kubernetes is great, however with the default K3S setup, it exposes traefik as a nodeport, which is not optimal for High Availibility / Redunancy. As well, the service external IP’s are not routable, and are only accessible from within the cluster. This is where MetalLB comes in, it allows you to expose services as LoadBalancer, and will automatically configure your router to forward traffic to the correct node.

However, the UDM-Pro/SE is lacking on many features….. and this is one of them. The UDM-Pro/SE does not support the use of BGP atleast within the GUI.

Credit to Map59’s Blog Post for some guidance with BGP on the UDM.

Prerequisites

  • A Kubernetes cluster running K3S, however this should work on any, however i will include some k3s specific things.
  • KubeCTL installed and configured to talk to your cluster.
  • A Ubiquiti UDM-Pro/SE with SSH enabled.

UDM BGP Setup

On-Boot-Script

First, you’ll want to SSH into your UDM, and run the following commands:

1

curl -fsL "https://raw.githubusercontent.com/unifi-utilities/unifios-utilities/HEAD/on-boot-script-2.x/remote_install.sh" | /bin/sh

This will install the on-boot-script for the UDM, which will allow us to run commands on boot. This allows the configuration changes we make to survive a software update, as without it the changes would be deleted.

You can checkout their github repo here, they have guidance aswell on how to run things like Pi-Hole etc, natively within the UDM.

FRR Install

Now that we have setup an onboot script, we’re going to create the actual boot script for FRR.

Open /data/on_boot.d/10-frr.sh via vim /data/on_bood.d/10-frr.sh and paste the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#!/bin/bash

# If FRR is not installed then install and configure it
if ! command -v /usr/lib/frr/frrinit.sh &> /dev/null; then
    echo "FRR could not be found"
    rm -f /etc/apt/sources.list.d/frr.list
    curl -s https://deb.frrouting.org/frr/keys.asc | sudo apt-key add -
    echo deb https://deb.frrouting.org/frr $(lsb_release -s -c) frr-stable | sudo tee -a /etc/apt/sources.list.d/frr.list

    apt-get update && apt-get -y install frr frr-pythontools
    if [ $? -eq 0 ]; then
        echo "Installation successful, updating configuration"
        echo > /etc/frr/vtysh.conf
        rm -f /etc/frr/frr.conf
        chown frr:frr /etc/frr/vtysh.conf
    fi
    service frr restart
fi

Now, run this file to install FRR

1

/data/on_boot.d/10-onboot-frr.sh

FRR Configuration

Now, lets enable BGP within FRR, and configure it to talk to our K3S cluster.

Open /etc/frr/daemons via vim /etc/frr/daemons and change bgpd=no to bgpd=yes

Now, open /etc/frr/bgpd.conf via vim /etc/frr/bgpd.conf and paste the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

! -*- bgp -*-
!
hostname $UDMP_HOSTNAME
password zebra
frr defaults traditional
log file stdout
!
router bgp 65510
 bgp ebgp-requires-policy
 bgp router-id --CHANGE_ME_UDM_IP--
 maximum-paths 1
 !
 ! Peer group for MetalLB
 neighbor ML peer-group
 neighbor ML remote-as 65512
 neighbor ML activate
 neighbor ML soft-reconfiguration inbound
 neighbor ML timers 15 45
 neighbor ML timers connect 15
 ! Neighbors for MetalLB
 neighbor --K3S-NODE-IP-- peer-group ML
 neighbor --K3S-NODE-IP-- peer-group ML
 neighbor --K3S-NODE-IP-- peer-group ML

 address-family ipv4 unicast
  redistribute connected
  !
  neighbor DNS activate
  neighbor DNS route-map ALLOW-ALL in
  neighbor DNS route-map ALLOW-ALL out
  neighbor DNS next-hop-self
  !
  neighbor ML activate
  neighbor ML route-map ALLOW-ALL in
  neighbor ML route-map ALLOW-ALL out
  neighbor ML next-hop-self
 exit-address-family
 !
route-map ALLOW-ALL permit 10
!
line vty
!

You’ll want to edit the bgp router-id to be the internal IP address of your UDM, and the K3S-NODE-IP’s to be the internal IP’s of your K3S nodes reachable from the UDM.

MetalLB Setup

Now that we have BGP setup on the UDM, we can install & configure MetalLB to use it.

K3S Disable

By default, k3s ships with the ServiceLB LoadBalancer, which is what we are replacing MetalLB with. However before we install MetalLB we’ll need to remove it from our cluster.

We can remove this by editing the following file /etc/systemd/system/k3s.service and adding --disable servicelb to the ExecStart line.

It should look something like this:

1
2
3

ExecStart=/usr/local/bin/k3s \
    server --disable servicelb \

MetalLB Install

First, we’ll need to install MetalLB, you can do this by running the following command:

1

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.9/config/manifests/metallb-frr.yaml

MetalLB Configuration

Now, we’ll need to configure MetalLB with the IP Range we want it to use, as well as the BGP Information for our UDM.

To start, lets setup the IP-Range, you’ll want to create a yaml file with the following contents: Replace the addresses with the IP Range you want to use.

1
2
3
4
5
6
7
8

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.111.0.0/16

Now apply this to your cluster via kubectl apply -f <filename>

Now, we’ll need to setup the BGP information for MetalLB, you’ll want to create a yaml file with the following contents: Replace the peerAddress with the internal IP of your UDM and the ASN numbers if you had changed them in the FRR config.

1
2
3
4
5
6
7
8
9

apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: udm
  namespace: metallb-system
spec:
  myASN: 65512
  peerASN: 65510
  peerAddress: 10.69.42.1

And we can apply this just like before with kubectl apply -f <filename>

Lastly, we’ll need to tell Metal to advertise our prefixes with one last yaml file:

1
2
3
4
5

apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: advert
  namespace: metallb-system

Once again, apply this with kubectl apply -f <filename>

Voila

Now, your UDM should be reciving the routes from your Cluster, and the LB ip’s should be reachable from your UDM/Network. You can verify this by running vtysh -c 'show ip bgp' on your UDM, and you should see entries with the External IP’s of your services. Which will look something like the following.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

BGP table version is 9, local router ID is 10.69.42.1, vrf id 0
Default local pref 100, local AS 65510
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *  10.111.0.0/32    10.69.42.55              0             0 65512 i
 *>                  10.69.42.51              0             0 65512 i
 *> 10.111.0.1/32    10.69.42.51              0             0 65512 i
 *  10.111.0.2/32    10.69.42.55              0             0 65512 i
 *>                  10.69.42.51              0             0 65512 i
Kubernetes
This post is licensed under CC BY 4.0 by the author.