Kubernetes dev environment with K3s, Longhorn and a private registry
Sunday, August 15th 2021

In this article we will set up a Kubernetes dev environment using K3S and Longhorn for dynamic storage provisioning.

The specs:

  • CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
  • Memory: 32GB DDR4 2133 MT/s
  • OS: Debian 10.4.0

Hostname

It should be noted that the hostname has been set to k8s.sm0ke.local:

# cat /etc/hosts  
127.0.0.1 localhost  
127.0.1.1 k8s.sm0ke.local k8s  
  
# The following lines are desirable for IPv6 capable hosts  
::1 localhost ip6-localhost ip6-loopback  
ff02::1 ip6-allnodes  
ff02::2 ip6-allrouters

You should also add a hosts file entry to your desktop computer as such (replace IP address with your own):

192.168.1.200 k8s.sm0ke.local

K3S installation

# curl -sfL https://get.k3s.io | sh

Be careful running scripts off of the internet. Only do so if you trust the source.

At this point K3S has been installed and can be used. However we will perform kubectl operations using our desktop computer so we need to set it up. Download the file /etc/rancher/k3s/k3s.yaml to your local computer. You will need to point kubectl to this file every time you run a kubectl command. Or you can automate this any way you like.

Edit k3s.yaml on your desktop computer and change the server IP address.

apiVersion: v1  
clusters:  
- cluster:  
 certificate-authority-data: -removed-  
 server: https://192.168.1.200:6443  
 name: default  
contexts:  
- context:  
 cluster: default  
 user: default  
 name: default  
current-context: default  
kind: Config  
preferences: {}  
users:  
- name: default  
user:  
 client-certificate-data: -removed-  
 client-key-data: -removed-Verifying we can connect to the server
$ kubectl --kubeconfig path/to/your/k3s.yaml get nodes  
NAME STATUS ROLES AGE VERSION  
k8s Ready control-plane,master 11h v1.21.3+k3s1

Installing Longhorn

We need to set up some kind of default dynamic storage provisioner so that we can use tools such as Helm with ease. For this we will install Longhorn.

$ helm repo add longhorn https://charts.longhorn.io  
$ helm repo update  
$ kubectl create namespace longhorn-system  
$ helm install longhorn longhorn/longhorn --namespace longhorn-system

Now that Longhorn is installed we need to configure a few things. K8S is pre-configured with a default storage provisioner called local-path. This provisioner is also marked as default.

$ kubectl get storageclasses  
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE  
longhorn (default) driver.longhorn.io Delete Immediate true 11h  
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 11h

We need to patch the local-path storage class and mark it as not being default.

$ kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

After that we can see that it worked

$ kubectl get storageclasses  
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE  
longhorn (default) driver.longhorn.io Delete Immediate true 11h  
local-path rancher.io/local-path Delete WaitForFirstConsumer false 11h

Configuring the private registry

In order to deploy anything to our k8s environment we will need to reference an image. You can use any kind of registry such as Github but for the sake of security (and speed) we will set up our own.

First of all we will prepare the registry certificates:

registry.cnf

[req]  
default_bits = 4096  
prompt = no  
default_md = sha256  
x509_extensions = v3_req  
distinguished_name = dn  
  
[dn]  
C = GR  
ST = Attica  
L = Athens  
O = k8s  
emailAddress = -your-email-address-  
CN = k8s.sm0ke.local  
  
[v3_req]  
subjectAltName = @alt_names  
  
[alt_names]  
DNS.1 = k8s.sm0ke.local  
DNS.2 = k8s.sm0ke.local:5000  
DNS.3 = 192.168.1.200  
DNS.4 = 192.168.1.200:5000  

Feel free to tune these settings to your liking.

$ openssl req -new -x509 -newkey rsa:4096 -nodes -sha256 -keyout registry.key -days 3560 -out registry.crt -config registry.cnf  
\# mkdir /usr/local/share/ca-certificates/k8s  
\# cp registry.crt /usr/local/share/ca-certificates/k8s/  
\# update-ca-certificates

We are importing this certificate and marking it as trusted.

You should also do this on your desktop operating system. I am using docker-desktop on Windows10. This will come in handy at a later stage where we will try and push images to the registry.

Right-click the certificate and select Install Certificate. Follow the prompts making sure that you select Current user for the store location then selecting Trusted Root Certification Authorities for the sertificate store.

Make sure you restart docker-desktop after doing this.

K3S Registry configuration

We must now configure K3S to use this registry. On your k8s host create the file /etc/rancher/k3s/registries.yaml

mirrors:  
 "k8s.sm0ke.local:5000":  
 endpoint:  
 - "https://k8s.sm0ke.local:5000"  
configs:  
 "k8s.sm0ke.local:5000":  
 tls:  
 cert_file: /opt/certs/registry.crt  
 key_file: /opt/certs/registry.key

Restart K3S

# systemctl restart k3s

We have now taken care of the certificates and configured k3s to use the registry. We should crack on with running the registry!

# mkdir /opt/registry

private-registry.yaml

apiVersion: apps/v1  
kind: Deployment  
metadata:  
 name: private-repository-k8s  
 labels:  
 app: private-repository-k8s  
spec:  
 replicas: 1  
 selector:  
 matchLabels:  
 app: private-repository-k8s  
 template:  
 metadata:  
 labels:  
 app: private-repository-k8s  
 spec:  
 volumes:  
 - name: certs-vol  
 hostPath:  
 path: /opt/certs  
 type: Directory  
 - name: registry-vol  
 hostPath:  
 path: /opt/registry  
 type: Directory  
 containers:  
 - image: registry:2  
 name: private-repository-k8s  
 imagePullPolicy: IfNotPresent  
 env:  
 - name: REGISTRY_HTTP_TLS_CERTIFICATE  
 value: "/certs/registry.crt"  
 - name: REGISTRY_HTTP_TLS_KEY  
 value: "/certs/registry.key"  
 ports:  
 - containerPort: 5000  
 volumeMounts:  
 - name: certs-vol  
 mountPath: /certs  
 - name: registry-vol  
 mountPath: /var/lib/registry

private-registry-svc.yaml

apiVersion: v1  
kind: Service  
metadata:  
 name: registry-service  
spec:  
 selector:  
 app: private-repository-k8s  
 type: LoadBalancer  
 ports:  
 - name: docker-port  
 protocol: TCP  
 port: 5000  
 targetPort: 5000  
 loadBalancerIP: 192.168.1.200

Apply the registry

$ kubectl apply -f private-registry.yaml  
$ kubectl apply -f private-registry-svc.yaml

You should now see the registry pod running

$ kubectl get pods  
NAME READY STATUS RESTARTS AGE  
private-repository-k8s-686564966d-f2zd8 1/1 Running 0 11h

Run the following commands on your desktop to make sure you can push to the private registry

$ docker pull nginx  
$ docker tag nginx k8s.sm0ke.local:5000/nginx  
$ docker push k8s.sm0ke.local:5000/nginx

You can also check out what your registry holds by visiting: https://k8s.sm0ke.local:5000/v2/_catalog in your browser.

{"{"repositories":["busybox","ispy","nginx"]}"}

You can now reference this repository when deploying workloads to k8s.

deployment.yaml

apiVersion: apps/v1  
kind: Deployment  
metadata:  
 name: nginx-test-deployment  
 labels:  
 app: nginx  
spec:  
 replicas: 3  
 selector:  
 matchLabels:  
 app: nginx  
 template:  
 metadata:  
 labels:  
 app: nginx  
 spec:  
 containers:  
 - name: nginx-1-17  
 image: k8s.sm0ke.local:5000/nginx:latest  
 ports:  
 - containerPort: 80

Apply the deployment

$ kubectl apply -f deployment.yaml

We have now configured a k8s dev cluster (only the master node) with a working dynamic storage provisioner and a private registry. We can now use it for development (or other) purposes:

$ helm repo add minecraft-server-charts https://itzg.github.io/minecraft-server-charts/  
$ helm repo update  
$ kubectl create ns minecraft  
$ helm -n minecraft install minecraft --set minecraftServer.eula=true --set persistence.dataDir.enabled=true minecraft-server-charts/minecraft  
  
💪💪💪

Discovering / Accessing services

K3S uses 10.0.0.0/8 address space by default so make sure you route this class to your K8S installation. On Windows open up a Powershell window as administrator and type the following to add a persistent route entry:

route add -p 10.0.0.0 MASK 255.0.0.0 192.168.1.200

This way we are routing all 10.0.0.0/8 addresses to our K8S cluster at 192.168.1.200

Now we can access services using the internal IP addresses shown in kubectl. Let us access Longhorn's web UI:

$ kubectl -n longhorn-system get svc  
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE  
longhorn-backend ClusterIP 10.43.172.167 <none> 9500/TCP 22h  
longhorn-frontend ClusterIP 10.43.199.16 <none> 80/TCP 22h  
csi-attacher ClusterIP 10.43.156.66 <none> 12345/TCP 22h  
csi-provisioner ClusterIP 10.43.112.101 <none> 12345/TCP 22h  
csi-resizer ClusterIP 10.43.226.21 <none> 12345/TCP 22h  
csi-snapshotter ClusterIP 10.43.222.124 <none> 12345/TCP 22h

We can see that the front-end service is running on 10.43.199.16 port 80. So open up a browser window and navigate to http://10.43.199.16

null

Notes

Originally I had considered installing Harbor which includes a whole lot of goodies such as Artifactory and Chartmuseum but I eventually decided against it since it is an overkill.

This post assumes a lot of knowledge of setting things up, I mostly wrote it to note down all the required steps since I do them again and again from scratch. Feel free to comment if something doesn't add up!

You can find all the relevant files in this github repo.

Edit 1 -- 26/08/2021

  • Adds security notice about running scripts off of the internet.
  • Adds Github repository with source files.