Kubernetes dev environment with K3s, Longhorn and a private registry
Sunday, August 15th 2021

In this article we will set up a Kubernetes dev environment using K3S and Longhorn for dynamic storage provisioning.

The specs:

  • CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
  • Memory: 32GB DDR4 2133 MT/s
  • OS: Debian 10.4.0

Hostname

It should be noted that the hostname has been set to k8s.sm0ke.local:
# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 k8s.sm0ke.local k8s

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
You should also add a hosts file entry to your desktop computer as such (replace IP address with your own):
192.168.1.200 k8s.sm0ke.local

K3S installation

# curl -sfL https://get.k3s.io | sh -

Be careful running scripts off of the internet. Only do so if you trust the source.

At this point K3S has been installed and can be used. However we will perform kubectl operations using our desktop computer so we need to set it up. Download the file /etc/rancher/k3s/k3s.yaml to your local computer. You will need to point kubectl to this file every time you run a kubectl command. Or you can automate this any way you like.

Edit k3s.yaml on your desktop computer and change the server IP address.

apiVersion: v1
clusters:
- cluster:
  certificate-authority-data: -removed-
  server: https://192.168.1.200:6443
 name: default
contexts:
- context:
  cluster: default
  user: default
 name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
  client-certificate-data: -removed-
  client-key-data: -removed-

Verifying we can connect to the server

$ kubectl --kubeconfig path/to/your/k3s.yaml get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,master 11h v1.21.3+k3s1

Installing Longhorn

We need to set up some kind of default dynamic storage provisioner so that we can use tools such as Helm with ease. For this we will install Longhorn.

$ helm repo add longhorn https://charts.longhorn.io
$ helm repo update
$ kubectl create namespace longhorn-system
$ helm install longhorn longhorn/longhorn --namespace longhorn-system

Now that Longhorn is installed we need to configure a few things. K8S is pre-configured with a default storage provisioner called local-path. This provisioner is also marked as default.

$ kubectl get storageclasses
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
longhorn (default) driver.longhorn.io Delete Immediate true 11h
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 11h

We need to patch the local-path storage class and mark it as not being default.

$ kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

After that we can see that it worked

$ kubectl get storageclasses
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
longhorn (default) driver.longhorn.io Delete Immediate true 11h
local-path rancher.io/local-path Delete WaitForFirstConsumer false 11h

Configuring the private registry

In order to deploy anything to our k8s environment we will need to reference an image. You can use any kind of registry such as Github but for the sake of security (and speed) we will set up our own.

First of all we will prepare the registry certificates:

registry.cnf

[req]
default_bits = 4096
prompt = no
default_md = sha256
x509_extensions = v3_req
distinguished_name = dn

[dn]
C = GR
ST = Attica
L = Athens
O = k8s
emailAddress = -your-email-address-
CN = k8s.sm0ke.local

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = k8s.sm0ke.local
DNS.2 = k8s.sm0ke.local:5000
DNS.3 = 192.168.1.200
DNS.4 = 192.168.1.200:5000

Feel free to tune these settings to your liking.

$ openssl req -new -x509 -newkey rsa:4096 -nodes -sha256 -keyout registry.key -days 3560 -out registry.crt -config registry.cnf
# mkdir /usr/local/share/ca-certificates/k8s
# cp registry.crt /usr/local/share/ca-certificates/k8s/
# update-ca-certificates

We are importing this certificate and marking it as trusted.

You should also do this on your desktop operating system. I am using docker-desktop on Windows10. This will come in handy at a later stage where we will try and push images to the registry.

Right-click the certificate and select Install Certificate. Follow the prompts making sure that you select Current user for the store location then selecting Trusted Root Certification Authorities for the sertificate store.

Make sure you restart docker-desktop after doing this.

K3S Registry configuration

We must now configure K3S to use this registry. On your k8s host create the file /etc/rancher/k3s/registries.yaml

mirrors:
 "k8s.sm0ke.local:5000":
    endpoint:
        - "https://k8s.sm0ke.local:5000"
configs:
 "k8s.sm0ke.local:5000":
    tls:
        cert_file: /opt/certs/registry.crt
        key_file: /opt/certs/registry.key

Restart K3S

# systemctl restart k3s

We have now taken care of the certificates and configured k3s to use the registry. We should crack on with running the registry!

# mkdir /opt/registry

private-registry.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: private-repository-k8s
  labels:
    app: private-repository-k8s
spec:
  replicas: 1
  selector:
    matchLabels:
      app: private-repository-k8s
  template:
    metadata:
      labels:
        app: private-repository-k8s
    spec:
      volumes:
      - name: certs-vol
        hostPath:
          path: /opt/certs
          type: Directory
      - name: registry-vol
        hostPath:
          path: /opt/registry
          type: Directory
      containers:
        - image: registry:2
          name: private-repository-k8s
          imagePullPolicy: IfNotPresent
          env:
          - name: REGISTRY_HTTP_TLS_CERTIFICATE
            value: "/certs/registry.crt"
          - name: REGISTRY_HTTP_TLS_KEY
            value: "/certs/registry.key"
          ports:
            - containerPort: 5000
          volumeMounts:
          - name: certs-vol
            mountPath: /certs
          - name: registry-vol
            mountPath: /var/lib/registry

private-registry-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: registry-service
spec:
  selector:
    app: private-repository-k8s
  type: LoadBalancer
  ports:
    - name: docker-port
    protocol: TCP
    port: 5000
    targetPort: 5000
  loadBalancerIP: 192.168.1.200

Apply the registry

$ kubectl apply -f private-registry.yaml
$ kubectl apply -f private-registry-svc.yaml

You should now see the registry pod running

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
private-repository-k8s-686564966d-f2zd8 1/1 Running 0 11h

Run the following commands on your desktop to make sure you can push to the private registry

$ docker pull nginx
$ docker tag nginx k8s.sm0ke.local:5000/nginx
$ docker push k8s.sm0ke.local:5000/nginx

You can also check out what your registry holds by visiting: https://k8s.sm0ke.local:5000/v2/_catalog in your browser.

{"{"repositories":["busybox","ispy","nginx"]}"}

You can now reference this repository when deploying workloads to k8s.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx-1-17
        image: k8s.sm0ke.local:5000/nginx:latest
        ports:
        - containerPort: 80

Apply the deployment

$ kubectl apply -f deployment.yaml

We have now configured a k8s dev cluster (only the master node) with a working dynamic storage provisioner and a private registry. We can now use it for development (or other) purposes:

$ helm repo add minecraft-server-charts https://itzg.github.io/minecraft-server-charts/
$ helm repo update
$ kubectl create ns minecraft
$ helm -n minecraft install minecraft --set minecraftServer.eula=true --set persistence.dataDir.enabled=true minecraft-server-charts/minecraft

💪💪💪

Discovering / Accessing services

K3S uses 10.0.0.0/8 address space by default so make sure you route this class to your K8S installation. On Windows open up a Powershell window as administrator and type the following to add a persistent route entry:

route add -p 10.0.0.0 MASK 255.0.0.0 192.168.1.200

This way we are routing all 10.0.0.0/8 addresses to our K8S cluster at 192.168.1.200

Now we can access services using the internal IP addresses shown in kubectl. Let us access Longhorn's web UI:

$ kubectl -n longhorn-system get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
longhorn-backend ClusterIP 10.43.172.167 <none> 9500/TCP 22h
longhorn-frontend ClusterIP 10.43.199.16 <none> 80/TCP 22h
csi-attacher ClusterIP 10.43.156.66 <none> 12345/TCP 22h
csi-provisioner ClusterIP 10.43.112.101 <none> 12345/TCP 22h
csi-resizer ClusterIP 10.43.226.21 <none> 12345/TCP 22h
csi-snapshotter ClusterIP 10.43.222.124 <none> 12345/TCP 22h

We can see that the front-end service is running on 10.43.199.16 port 80. So open up a browser window and navigate to http://10.43.199.16

Longhorn Web UI

Notes

Originally I had considered installing Harbor which includes a whole lot of goodies such as Artifactory and Chartmuseum but I eventually decided against it since it is an overkill.

This post assumes a lot of knowledge of setting things up, I mostly wrote it to note down all the required steps since I do them again and again from scratch. Feel free to comment if something doesn't add up!

You can find all the relevant files in this github repo.

Edit 1 -- 26/08/2021

  • Adds security notice about running scripts off of the internet.
  • Adds Github repository with source files.