Kubernetes is great, however with the default K3S setup, it exposes traefik as a nodeport, which is not optimal for High Availibility / Redunancy. As well, the service external IP’s are not routable, and are only accessible from within the cluster. This is where MetalLB comes in, it allows you to expose services as LoadBalancer, and will automatically configure your router to forward traffic to the correct node.
However, the UDM-Pro/SE is lacking on many features….. and this is one of them. The UDM-Pro/SE does not support the use of BGP atleast within the GUI.
Credit to Map59’s Blog Post for some guidance with BGP on the UDM.
- A Kubernetes cluster running K3S, however this should work on any, however i will include some k3s specific things.
- KubeCTL installed and configured to talk to your cluster.
- A Ubiquiti UDM-Pro/SE with SSH enabled.
UDM BGP Setup
First, you’ll want to SSH into your UDM, and run the following commands:
1 curl -fsL "https://raw.githubusercontent.com/unifi-utilities/unifios-utilities/HEAD/on-boot-script-2.x/remote_install.sh" | /bin/sh
This will install the on-boot-script for the UDM, which will allow us to run commands on boot. This allows the configuration changes we make to survive a software update, as without it the changes would be deleted.
You can checkout their github repo here, they have guidance aswell on how to run things like Pi-Hole etc, natively within the UDM.
Now that we have setup an onboot script, we’re going to create the actual boot script for FRR.
Open /data/on_boot.d/10-frr.sh via
vim /data/on_bood.d/10-frr.sh and paste the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 #!/bin/bash # If FRR is not installed then install and configure it if ! command -v /usr/lib/frr/frrinit.sh &> /dev/null; then echo "FRR could not be found" rm -f /etc/apt/sources.list.d/frr.list curl -s https://deb.frrouting.org/frr/keys.asc | sudo apt-key add - echo deb https://deb.frrouting.org/frr $(lsb_release -s -c) frr-stable | sudo tee -a /etc/apt/sources.list.d/frr.list apt-get update && apt-get -y install frr frr-pythontools if [ $? -eq 0 ]; then echo "Installation successful, updating configuration" echo > /etc/frr/vtysh.conf rm -f /etc/frr/frr.conf chown frr:frr /etc/frr/vtysh.conf fi service frr restart fi
Now, run this file to install FRR
Now, lets enable BGP within FRR, and configure it to talk to our K3S cluster.
Open /etc/frr/daemons via
vim /etc/frr/daemons and change
Now, open /etc/frr/bgpd.conf via
vim /etc/frr/bgpd.conf and paste the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 ! -*- bgp -*- ! hostname $UDMP_HOSTNAME password zebra frr defaults traditional log file stdout ! router bgp 65510 bgp ebgp-requires-policy bgp router-id --CHANGE_ME_UDM_IP-- maximum-paths 1 ! ! Peer group for MetalLB neighbor ML peer-group neighbor ML remote-as 65512 neighbor ML activate neighbor ML soft-reconfiguration inbound neighbor ML timers 15 45 neighbor ML timers connect 15 ! Neighbors for MetalLB neighbor --K3S-NODE-IP-- peer-group ML neighbor --K3S-NODE-IP-- peer-group ML neighbor --K3S-NODE-IP-- peer-group ML address-family ipv4 unicast redistribute connected ! neighbor DNS activate neighbor DNS route-map ALLOW-ALL in neighbor DNS route-map ALLOW-ALL out neighbor DNS next-hop-self ! neighbor ML activate neighbor ML route-map ALLOW-ALL in neighbor ML route-map ALLOW-ALL out neighbor ML next-hop-self exit-address-family ! route-map ALLOW-ALL permit 10 ! line vty !
You’ll want to edit the bgp router-id to be the internal IP address of your UDM, and the K3S-NODE-IP’s to be the internal IP’s of your K3S nodes reachable from the UDM.
Now that we have BGP setup on the UDM, we can install & configure MetalLB to use it.
By default, k3s ships with the ServiceLB LoadBalancer, which is what we are replacing MetalLB with. However before we install MetalLB we’ll need to remove it from our cluster.
We can remove this by editing the following file
/etc/systemd/system/k3s.service and adding
--disable servicelb to the
It should look something like this:
1 2 3 ExecStart=/usr/local/bin/k3s \ server --disable servicelb \
First, we’ll need to install MetalLB, you can do this by running the following command:
1 kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.9/config/manifests/metallb-frr.yaml
Now, we’ll need to configure MetalLB with the IP Range we want it to use, as well as the BGP Information for our UDM.
To start, lets setup the IP-Range, you’ll want to create a yaml file with the following contents: Replace the addresses with the IP Range you want to use.
1 2 3 4 5 6 7 8 apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: first-pool namespace: metallb-system spec: addresses: - 10.111.0.0/16
Now apply this to your cluster via
kubectl apply -f <filename>
Now, we’ll need to setup the BGP information for MetalLB, you’ll want to create a yaml file with the following contents:
peerAddress with the internal IP of your UDM and the ASN numbers if you had changed them in the FRR config.
1 2 3 4 5 6 7 8 9 apiVersion: metallb.io/v1beta2 kind: BGPPeer metadata: name: udm namespace: metallb-system spec: myASN: 65512 peerASN: 65510 peerAddress: 10.69.42.1
And we can apply this just like before with
kubectl apply -f <filename>
Lastly, we’ll need to tell Metal to advertise our prefixes with one last yaml file:
1 2 3 4 5 apiVersion: metallb.io/v1beta1 kind: BGPAdvertisement metadata: name: advert namespace: metallb-system
Once again, apply this with
kubectl apply -f <filename>
Now, your UDM should be reciving the routes from your Cluster, and the LB ip’s should be reachable from your UDM/Network. You can verify this by running
vtysh -c 'show ip bgp' on your UDM, and you should see entries with the External IP’s of your services. Which will look something like the following.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 BGP table version is 9, local router ID is 10.69.42.1, vrf id 0 Default local pref 100, local AS 65510 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path * 10.111.0.0/32 10.69.42.55 0 0 65512 i *> 10.69.42.51 0 0 65512 i *> 10.111.0.1/32 10.69.42.51 0 0 65512 i * 10.111.0.2/32 10.69.42.55 0 0 65512 i *> 10.69.42.51 0 0 65512 i