Running a cluster on R620 using the VPS as a network gateway

Plan

In first step to see I my idea will work, I create 2 VMs on my proxmox host, connect VMs to wireguard (to not use local network and BGP).

Creating the VMs

To do my test, I create VMs from Ubuntu 22.04 cloud image, and set their memory to 4GB, 32G of disk and basic cloud-init config (ssh key, and use DHCP).

Installing Wireguard and first ping

As I explain before, my proxmox cluster is connected by BGP to my VPS, so here I need to make sure that when I ping my other VM it will pass by the VPS.

Run this in each VMs.

sudo -i
apt install -y wireguard openresolv
nano /etc/wireguard/wg0.conf # Paste your configuration
systemctl enable --now wg-quick@wg0
# Make it ping
ping 10.6.0.20

Particular options in wireguard

I use PiVPN to create automatically my wireguard configuration, so by default in Peer.AllowedIPs there is 0.0.0.0/0, ::0/0, I will change that with 10.6.0.0/16 to avoid the VMs to connect using the default route and so BGP routing.

I need to add PersistentKeepalive = 25 to not be disconnected and be locked outside of my VMs.

Setup Tailscale

To communicate between my nodes, I’ll use Tailscale. There is an integration with K3s right here.

Create an external DB

In this case, I’ll setup only two nodes, so I setup a database on my existing everything docker host.

docker run --name kube-database -p 3306:3306 -v /root/mysql:/var/lib/mysql -e MYSQL_USER=kube -e MYSQL_DATABASE=kube -e MYSQL_PASSWORD=SOMETHING_REALLY_HARD_TO_GUESS -e MYSQL_ROOT_PASSWORD=MY_SUPER_LONG_PASSWORD --restart always mysql:latest

Kubernetes installation on the first node

There is a bug with k3s installation script that will not download the latest version of k3s. DO NOT forget to specify the latest version.

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.29.2+k3s1 sh -s - server \
     --cluster-init \
     --datastore-endpoint="mysql://kube:SOMETHING_REALLY_HARD_TO_GUESS@tcp(10.1.0.13:3306)/kube" \
     --vpn-auth="name=tailscale,joinKey=tskey-auth-something-anotherthing" \
     --tls-san=10.6.0.19

See if the node is properly configured

When you execute kubectl get nodes, you’ll see your node with control-plane, etcd, master as roles. That’s we want!

Install Kubernetes on the second node

Get the token by executing: cat /var/lib/rancher/k3s/server/token on the first node.

curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET INSTALL_K3S_VERSION=v1.29.2+k3s1 sh -s - server \
    --server https://10.6.0.19:6443 \
    --datastore-endpoint="mysql://kube:SOMETHING_REALLY_HARD_TO_GUESS@tcp(10.1.0.13:3306)/kube" \
    --vpn-auth="name=tailscale,joinKey=tskey-auth-something-anotherthing" \
    --tls-san=10.6.0.19

See if the node is recognized and configured as we want

You should have something like that:

root@kube2:~# kubectl get nodes
NAME    STATUS   ROLES                       AGE     VERSION
kube1   Ready    control-plane,master   2m50s   v1.29.2+k3s1
kube2   Ready    control-plane,master   50s     v1.29.2+k3s1

Our first service

To test the installation, we’re going to create nginx pod.

apiVersion: v1
kind: Pod
metadata:
  name: webserver
  labels:
    app: webserver
spec:
   containers:
   - name: webserver
     image: nginx:latest
     ports:
     - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webserver
spec:
  selector:
    app: webserver
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webserver
spec:
  rules:
  - host: test.legodard.fr
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: webserver
            port:
              number: 80

Test

Now we want to test!

You can run this, and you’ll see “Welcome to nginx”!

curl http://100.127.195.81/ -H "Host: test.legodard.fr" -v

I want to go further

Before going to buy some hardware, I need to be sure if my system can be “easy” (it’s kube, not so easy) and have an automated ingress with the outside.

For this, I will use STRRL/cloudflare-tunnel-ingress-controller project.

Cloudflare configuration

To create the API Token, you’ll need to sign in to cloudflare, and on the top right corner you will find “My profile”, then you can create API keys.

As permission you need:

  • Zone:Zone:Read

  • Zone:DNS:Edit

  • Account:Cloudflare Tunnel:Edit

Then to get your account id, you can go to your website and on the bottom left you can see it!

Ingress controller installation

Everything is executed on the kube2 node, so no 100% kube API LB.

To install the ingress controller, you’ll need to add a helm repository, I know some people out there are not trustable, all I can recommend if it’s your case, it’s to build everything from the GitHub repository.

helm repo add strrl.dev https://helm.strrl.dev
helm repo update

Then, we need the API key and the account id:

helm upgrade --install --wait \
  -n cloudflare-tunnel-ingress-controller --create-namespace \
  cloudflare-tunnel-ingress-controller \
  strrl.dev/cloudflare-tunnel-ingress-controller \
  --set=cloudflare.apiToken="<cloudflare-api-token>",cloudflare.accountId="<cloudflare-account-id>",cloudflare.tunnelName="<your-favorite-tunnel-name>"

Now the final test!

apiVersion: v1
kind: Pod
metadata:
  name: webserver
  labels:
    app: webserver
spec:
   containers:
   - name: webserver
     image: nginx:latest
     ports:
     - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webserver
spec:
  selector:
    app: webserver
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webserver
spec:
  ingressClassName: cloudflare-tunnel
  rules:
  - host: testkube.legodard.fr
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: webserver
            port:
              number: 80