Kubernetes dev environment with K3s, Longhorn and a private registry
Sunday, August 15th 2021

In this article we will set up a Kubernetes dev environment using K3S and Longhorn for dynamic storage provisioning.

The specs:

  • CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
  • Memory: 32GB DDR4 2133 MT/s
  • OS: Debian 10.4.0


It should be noted that the hostname has been set to k8s.sm0ke.local:
# cat /etc/hosts localhost k8s.sm0ke.local k8s

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
You should also add a hosts file entry to your desktop computer as such (replace IP address with your own): k8s.sm0ke.local

K3S installation

# curl -sfL https://get.k3s.io | sh -

Be careful running scripts off of the internet. Only do so if you trust the source.

At this point K3S has been installed and can be used. However we will perform kubectl operations using our desktop computer so we need to set it up. Download the file /etc/rancher/k3s/k3s.yaml to your local computer. You will need to point kubectl to this file every time you run a kubectl command. Or you can automate this any way you like.

Edit k3s.yaml on your desktop computer and change the server IP address.

apiVersion: v1
- cluster:
  certificate-authority-data: -removed-
 name: default
- context:
  cluster: default
  user: default
 name: default
current-context: default
kind: Config
preferences: {}
- name: default
  client-certificate-data: -removed-
  client-key-data: -removed-

Verifying we can connect to the server

$ kubectl --kubeconfig path/to/your/k3s.yaml get nodes
k8s Ready control-plane,master 11h v1.21.3+k3s1

Installing Longhorn

We need to set up some kind of default dynamic storage provisioner so that we can use tools such as Helm with ease. For this we will install Longhorn.

$ helm repo add longhorn https://charts.longhorn.io
$ helm repo update
$ kubectl create namespace longhorn-system
$ helm install longhorn longhorn/longhorn --namespace longhorn-system

Now that Longhorn is installed we need to configure a few things. K8S is pre-configured with a default storage provisioner called local-path. This provisioner is also marked as default.

$ kubectl get storageclasses
longhorn (default) driver.longhorn.io Delete Immediate true 11h
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 11h

We need to patch the local-path storage class and mark it as not being default.

$ kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

After that we can see that it worked

$ kubectl get storageclasses
longhorn (default) driver.longhorn.io Delete Immediate true 11h
local-path rancher.io/local-path Delete WaitForFirstConsumer false 11h

Configuring the private registry

In order to deploy anything to our k8s environment we will need to reference an image. You can use any kind of registry such as Github but for the sake of security (and speed) we will set up our own.

First of all we will prepare the registry certificates:


default_bits = 4096
prompt = no
default_md = sha256
x509_extensions = v3_req
distinguished_name = dn

C = GR
ST = Attica
L = Athens
O = k8s
emailAddress = -your-email-address-
CN = k8s.sm0ke.local

subjectAltName = @alt_names

DNS.1 = k8s.sm0ke.local
DNS.2 = k8s.sm0ke.local:5000
DNS.3 =
DNS.4 =

Feel free to tune these settings to your liking.

$ openssl req -new -x509 -newkey rsa:4096 -nodes -sha256 -keyout registry.key -days 3560 -out registry.crt -config registry.cnf
# mkdir /usr/local/share/ca-certificates/k8s
# cp registry.crt /usr/local/share/ca-certificates/k8s/
# update-ca-certificates

We are importing this certificate and marking it as trusted.

You should also do this on your desktop operating system. I am using docker-desktop on Windows10. This will come in handy at a later stage where we will try and push images to the registry.

Right-click the certificate and select Install Certificate. Follow the prompts making sure that you select Current user for the store location then selecting Trusted Root Certification Authorities for the sertificate store.

Make sure you restart docker-desktop after doing this.

K3S Registry configuration

We must now configure K3S to use this registry. On your k8s host create the file /etc/rancher/k3s/registries.yaml

        - "https://k8s.sm0ke.local:5000"
        cert_file: /opt/certs/registry.crt
        key_file: /opt/certs/registry.key

Restart K3S

# systemctl restart k3s

We have now taken care of the certificates and configured k3s to use the registry. We should crack on with running the registry!

# mkdir /opt/registry


apiVersion: apps/v1
kind: Deployment
  name: private-repository-k8s
    app: private-repository-k8s
  replicas: 1
      app: private-repository-k8s
        app: private-repository-k8s
      - name: certs-vol
          path: /opt/certs
          type: Directory
      - name: registry-vol
          path: /opt/registry
          type: Directory
        - image: registry:2
          name: private-repository-k8s
          imagePullPolicy: IfNotPresent
            value: "/certs/registry.crt"
          - name: REGISTRY_HTTP_TLS_KEY
            value: "/certs/registry.key"
            - containerPort: 5000
          - name: certs-vol
            mountPath: /certs
          - name: registry-vol
            mountPath: /var/lib/registry


apiVersion: v1
kind: Service
  name: registry-service
    app: private-repository-k8s
  type: LoadBalancer
    - name: docker-port
    protocol: TCP
    port: 5000
    targetPort: 5000

Apply the registry

$ kubectl apply -f private-registry.yaml
$ kubectl apply -f private-registry-svc.yaml

You should now see the registry pod running

$ kubectl get pods
private-repository-k8s-686564966d-f2zd8 1/1 Running 0 11h

Run the following commands on your desktop to make sure you can push to the private registry

$ docker pull nginx
$ docker tag nginx k8s.sm0ke.local:5000/nginx
$ docker push k8s.sm0ke.local:5000/nginx

You can also check out what your registry holds by visiting: https://k8s.sm0ke.local:5000/v2/_catalog in your browser.


You can now reference this repository when deploying workloads to k8s.


apiVersion: apps/v1
kind: Deployment
  name: nginx-test-deployment
    app: nginx
  replicas: 3
      app: nginx
        app: nginx
      - name: nginx-1-17
        image: k8s.sm0ke.local:5000/nginx:latest
        - containerPort: 80

Apply the deployment

$ kubectl apply -f deployment.yaml

We have now configured a k8s dev cluster (only the master node) with a working dynamic storage provisioner and a private registry. We can now use it for development (or other) purposes:

$ helm repo add minecraft-server-charts https://itzg.github.io/minecraft-server-charts/
$ helm repo update
$ kubectl create ns minecraft
$ helm -n minecraft install minecraft --set minecraftServer.eula=true --set persistence.dataDir.enabled=true minecraft-server-charts/minecraft


Discovering / Accessing services

K3S uses address space by default so make sure you route this class to your K8S installation. On Windows open up a Powershell window as administrator and type the following to add a persistent route entry:

route add -p MASK

This way we are routing all addresses to our K8S cluster at

Now we can access services using the internal IP addresses shown in kubectl. Let us access Longhorn's web UI:

$ kubectl -n longhorn-system get svc
longhorn-backend ClusterIP <none> 9500/TCP 22h
longhorn-frontend ClusterIP <none> 80/TCP 22h
csi-attacher ClusterIP <none> 12345/TCP 22h
csi-provisioner ClusterIP <none> 12345/TCP 22h
csi-resizer ClusterIP <none> 12345/TCP 22h
csi-snapshotter ClusterIP <none> 12345/TCP 22h

We can see that the front-end service is running on port 80. So open up a browser window and navigate to

Longhorn Web UI


Originally I had considered installing Harbor which includes a whole lot of goodies such as Artifactory and Chartmuseum but I eventually decided against it since it is an overkill.

This post assumes a lot of knowledge of setting things up, I mostly wrote it to note down all the required steps since I do them again and again from scratch. Feel free to comment if something doesn't add up!

You can find all the relevant files in this github repo.

Edit 1 -- 26/08/2021

  • Adds security notice about running scripts off of the internet.
  • Adds Github repository with source files.