# Running a cluster on R620 using the VPS as a network gateway ## Plan In first step to see I my idea will work, I create 2 VMs on my proxmox host, connect VMs to wireguard (to not use local network and BGP). ## Creating the VMs To do my test, I create VMs from Ubuntu 22.04 cloud image, and set their memory to 4GB, 32G of disk and basic cloud-init config (ssh key, and use DHCP). ## Installing Wireguard and first ping As I explain before, my proxmox cluster is connected by BGP to my VPS, so here I need to make sure that when I ping my other VM it will pass by the VPS. Run this in each VMs. ```bash sudo -i apt install -y wireguard openresolv nano /etc/wireguard/wg0.conf # Paste your configuration systemctl enable --now wg-quick@wg0 # Make it ping ping 10.6.0.20 ``` ### Particular options in wireguard I use PiVPN to create automatically my wireguard configuration, so by default in `Peer.AllowedIPs` there is `0.0.0.0/0, ::0/0`, I will change that with `10.6.0.0/16` to avoid the VMs to connect using the default route and so BGP routing. I need to add `PersistentKeepalive = 25` to not be disconnected and be locked outside of my VMs. ## Setup Tailscale To communicate between my nodes, I’ll use Tailscale. There is an integration with K3s right [here](https://docs.k3s.io/installation/network-options#integration-with-the-tailscale-vpn-provider-experimental). ## Create an external DB In this case, I’ll setup only two nodes, so I setup a database on my existing ~~everything~~ docker host. ```bash docker run --name kube-database -p 3306:3306 -v /root/mysql:/var/lib/mysql -e MYSQL_USER=kube -e MYSQL_DATABASE=kube -e MYSQL_PASSWORD=SOMETHING_REALLY_HARD_TO_GUESS -e MYSQL_ROOT_PASSWORD=MY_SUPER_LONG_PASSWORD --restart always mysql:latest ``` ## Kubernetes installation on the first node :::warning There is a bug with k3s installation script that will not download the latest version of k3s. DO NOT forget to specify the latest version. ::: ```bash curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.29.2+k3s1 sh -s - server \ --cluster-init \ --datastore-endpoint="mysql://kube:SOMETHING_REALLY_HARD_TO_GUESS@tcp(10.1.0.13:3306)/kube" \ --vpn-auth="name=tailscale,joinKey=tskey-auth-something-anotherthing" \ --tls-san=10.6.0.19 ``` ### See if the node is properly configured When you execute `kubectl get nodes`, you’ll see your node with `control-plane, etcd, master` as roles. That’s we want! ## Install Kubernetes on the second node Get the token by executing: `cat /var/lib/rancher/k3s/server/token` on the first node. ```bash curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET INSTALL_K3S_VERSION=v1.29.2+k3s1 sh -s - server \ --server https://10.6.0.19:6443 \ --datastore-endpoint="mysql://kube:SOMETHING_REALLY_HARD_TO_GUESS@tcp(10.1.0.13:3306)/kube" \ --vpn-auth="name=tailscale,joinKey=tskey-auth-something-anotherthing" \ --tls-san=10.6.0.19 ``` ### See if the node is recognized and configured as we want You should have something like that: ```bash root@kube2:~# kubectl get nodes NAME STATUS ROLES AGE VERSION kube1 Ready control-plane,master 2m50s v1.29.2+k3s1 kube2 Ready control-plane,master 50s v1.29.2+k3s1 ``` ## Our first service To test the installation, we’re going to create nginx pod. ```yaml apiVersion: v1 kind: Pod metadata: name: webserver labels: app: webserver spec: containers: - name: webserver image: nginx:latest ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: webserver spec: selector: app: webserver ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: webserver spec: rules: - host: test.legodard.fr http: paths: - path: / pathType: Prefix backend: service: name: webserver port: number: 80 ``` ### Test Now we want to test! You can run this, and you’ll see “Welcome to nginx”! ```bash curl http://100.127.195.81/ -H "Host: test.legodard.fr" -v ``` ## I want to go further Before going to buy some hardware, I need to be sure if my system can be “easy” (it’s kube, not so easy) and have an automated ingress with the outside. For this, I will use [STRRL/cloudflare-tunnel-ingress-controller](https://github.com/STRRL/cloudflare-tunnel-ingress-controller) project. ### Cloudflare configuration To create the API Token, you’ll need to sign in to cloudflare, and on the top right corner you will find “My profile”, then you can create API keys. As permission you need: * `Zone:Zone:Read` * `Zone:DNS:Edit` * `Account:Cloudflare Tunnel:Edit` Then to get your account id, you can go to your website and on the bottom left you can see it! ### Ingress controller installation :::info Everything is executed on the `kube2` node, so no 100% kube API LB. ::: To install the ingress controller, you’ll need to add a helm repository, I know some people out there are not trustable, all I can recommend if it’s your case, it’s to build everything from the GitHub repository. ```bash helm repo add strrl.dev https://helm.strrl.dev helm repo update ``` Then, we need the API key and the account id: ```bash helm upgrade --install --wait \ -n cloudflare-tunnel-ingress-controller --create-namespace \ cloudflare-tunnel-ingress-controller \ strrl.dev/cloudflare-tunnel-ingress-controller \ --set=cloudflare.apiToken="",cloudflare.accountId="",cloudflare.tunnelName="" ``` ### Now the final test! ```bash apiVersion: v1 kind: Pod metadata: name: webserver labels: app: webserver spec: containers: - name: webserver image: nginx:latest ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: webserver spec: selector: app: webserver ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: webserver spec: ingressClassName: cloudflare-tunnel rules: - host: testkube.legodard.fr http: paths: - path: / pathType: Prefix backend: service: name: webserver port: number: 80 ``` --- **Documents** - [AS205126](https://docs.legodard.fr/s/bc661677-ac7d-4fe7-984e-e1f43ddf6287/doc/as205126-UD37o91h51) - [Kubernetes](https://docs.legodard.fr/s/bc661677-ac7d-4fe7-984e-e1f43ddf6287/doc/kubernetes-AYsNjQV7SW) - [Proxmox](https://docs.legodard.fr/s/bc661677-ac7d-4fe7-984e-e1f43ddf6287/doc/proxmox-TnNEgoRFwj) - [DN42](https://docs.legodard.fr/s/bc661677-ac7d-4fe7-984e-e1f43ddf6287/doc/dn42-oTEE68Qn7n) - [Openshift](https://docs.legodard.fr/s/bc661677-ac7d-4fe7-984e-e1f43ddf6287/doc/openshift-428FG3kMtg) - [Openstack](https://docs.legodard.fr/s/bc661677-ac7d-4fe7-984e-e1f43ddf6287/doc/openstack-KJmz0J0klm)