Add a floating virtual IP to Kubernetes cluster nodes for load balancing easily.

Carpathian Mountains by Victoria Skrypnik

Kube Karp allows Kubernetes cluster nodes to share a common virtual IP address in order to provide automatic Kube API Server failover. In beneath, it is using UCarp which is a portable userland implementation of the secure and patent-free Common Address Redundancy Protocol (CARP, OpenBSD's alternative to the patents-bloated VRRP).

Kube Karp is running as a DaemonSet in the cluster which means it is deployed on every node. Hovewer, the image is built on Alpine Linux, it is about 3 MBs in size and requires about the same amount of memory on each node, so it will not be a resource hog :) It requires host network access and NET_ADMIN capabilities to manage the nodes' interface settings to add or remove the virtual IP of the current UCarp leader process. All nodes must be in the same subnet (configurable).

Practical use-case: in your kubeconfig file, the Kubernetes API Server URL can point to the virtual IP, so your cluster is always accessible, not depending on a particluar node's IP address. When the leader node becomes unavailable, another node takes the floating IP in about 3 seconds, so you can always reference the k8s cluster by the same virtual IP. The current leader is elected based on the CARP implementation.

Kube Karp can be deployed to an existing cluster, no need to tear it down then build it up from scratch. It should not require any maintenance, once you set it up, it just works™.

Install Helm v3 on your machine, then run:

git clone https://github.com/immanuelfodor/kube-karp
cd kube-karp/helm

# now edit the values.yaml file to set up your preferences
# and then install Kube Karp with preferences from the file
helm install kube-karp .

# or install using the --set key=value flags,
# the values are just examples
helm install kube-karp . \
    --set envVars.virtualIp=192.168.100.100 \
    --set envVars.interface=enp0s3

# if you want to install it in a new kube-karp namespace, add
# the following flags to any of the above helm install commands
-n kube-karp --create-namespace

# you can check what will be installed with the template command
# beforehand, but do not forget to also add all your custom flags
helm template kube-karp .

Replace your current node-IP in your kubeconfig file (possibly located at ~/.kube/config) with the KARP_VIRTUAL_IP (envVars.virtualIp in the Helm chart). For example, your new config should be:

apiVersion: v1
clusters:
- cluster:
    ...
    server: https://192.168.100.100:6443
    ...

Attention! Changing the IP will possibly result in losing access to your cluster temporarily without adding the insecure flag to further kubectl commands:

# this will not work
kubectl get nodes
# Unable to connect to the server: x509: certificate is valid for
# 192.168.100.110, 192.168.100.111, 192.168.100.112, 127.0.0.1, 10.43.0.1, not 192.168.100.100

# quick and dirty solution to get around it
kubectl get nodes --insecure-skip-tls-verify=true
# NAME        STATUS   ROLES                      AGE    VERSION
# ...

To permanently fix this, you need to add the KARP_VIRTUAL_IP (envVars.virtualIp in the Helm chart) to the Kube API Server certificate as a new SAN (Subject Alternative Name). If you have an RKE cluster, make the following change to to your cluster.yml:

 authentication:
   strategy: x509
-  sans: []
+  sans:
+  - "192.168.100.100"
   webhook: null

Then reconfigure the cluster and check the new IP in the Kube API Server certificate, the kubectl commands should work fine from now on:

# propagate the change to the cluster
rke up

# check the presence of the virtual IP in the API server cert
openssl s_client -connect 192.168.100.100:6443 | openssl x509 -noout -text

# kubectl commands should work fine from now on
kubectl get nodes

One caveat that will probably be solved by RKE maintainers in the future is that you will need to manually update the kubeconfig file with the floating virtual IP after any further rke up command issued. There is an open issue to fix this problem at RKE #1682.

Should you have a cluster bootstrapped with kubeadm, you can find some reading material here as a start but I use RKE, so this method is untested by me:

• A tutorial how to add a new SAN with kubeadm: https://blog.scottlowe.org/2019/07/30/adding-a-name-to-kubernetes-api-server-certificate/
• A related Github issue with more certSANs examples: kubernetes/kubeadm#1447
• A tl;dr of the above: kubernetes/kubernetes#76357

Kube Karp's unique feature is that it can load balance the Kube API Server without external tools which most other LB solutions can not do at the time of writing. It does not provide LoadBalancer resources for exposed services, it only manages a floating virtual IP accross nodes in the same subnet, so you can always reference the cluster with the same IP address. It does one thing and and tries to do it well as an "install and forget" solution. (See the following open issues for the circular dependency problem in Porter #125, MetalLB #168 repositories about why they can't load balance the kube API server.)

You could also use Keepalived with HAProxy as mentioned in the official k8s HA docs but you would also need to manually edit things on the nodes and then do the kubeadm init bootstrapping afterwards, which is not feasible if you already have a running cluster and you have even set it up with another cluster bootstrapping method (e.g., RKE).

The main motivation behind this project is to create an in-cluster solution without separate configs on each node and keep the Unix-philosophy of achieving only one thing with the tool. If you still need LoadBalancer resources in your cluster, go for the previously mentioned tools, I think these can co-exist besides Kube Karp but this scenario is untested.

I do not take responsibility for anything regarding the use or misuse of the contents of this repository. Please use it on your own risk.

• It seems that kube-vip has implemented a similar functionality to Kube Karp in the meantime based on my issue at kube-vip #76. You can check it out if you're already using it to provide a LoadBalancer resource or you plan to have one besides kube API server HA. If you're using or will be using MetalLB or Porter, or you just need the floating virtual IP, Kube Karp to the rescue! :)

Immánuel Fodor
fodor.it | Linkedin