In this article we will set up a Kubernetes dev environment using K3S and Longhorn for dynamic storage provisioning.
The specs:
- CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
- Memory: 32GB DDR4 2133 MT/s
- OS: Debian 10.4.0
Hostname
It should be noted that the hostname has been set to k8s.sm0ke.local:
# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 k8s.sm0ke.local k8s
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
You should also add a hosts file entry to your desktop computer as such (replace IP address with your own):
192.168.1.200 k8s.sm0ke.local
K3S installation
# curl -sfL https://get.k3s.io | sh
Be careful running scripts off of the internet. Only do so if you trust the source.
At this point K3S has been installed and can be used. However we will perform kubectl operations using our desktop computer so we need to set it up. Download the file /etc/rancher/k3s/k3s.yaml to your local computer. You will need to point kubectl to this file every time you run a kubectl command. Or you can automate this any way you like.
Edit k3s.yaml on your desktop computer and change the server IP address.
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: -removed-
server: https://192.168.1.200:6443
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
client-certificate-data: -removed-
client-key-data: -removed-Verifying we can connect to the server
$ kubectl --kubeconfig path/to/your/k3s.yaml get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,master 11h v1.21.3+k3s1
Installing Longhorn
We need to set up some kind of default dynamic storage provisioner so that we can use tools such as Helm with ease. For this we will install Longhorn.
$ helm repo add longhorn https://charts.longhorn.io
$ helm repo update
$ kubectl create namespace longhorn-system
$ helm install longhorn longhorn/longhorn --namespace longhorn-system
Now that Longhorn is installed we need to configure a few things. K8S is pre-configured with a default storage provisioner called local-path. This provisioner is also marked as default.
$ kubectl get storageclasses
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
longhorn (default) driver.longhorn.io Delete Immediate true 11h
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 11h
We need to patch the local-path storage class and mark it as not being default.
$ kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
After that we can see that it worked
$ kubectl get storageclasses
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
longhorn (default) driver.longhorn.io Delete Immediate true 11h
local-path rancher.io/local-path Delete WaitForFirstConsumer false 11h
Configuring the private registry
In order to deploy anything to our k8s environment we will need to reference an image. You can use any kind of registry such as Github but for the sake of security (and speed) we will set up our own.
First of all we will prepare the registry certificates:
registry.cnf
[req]
default_bits = 4096
prompt = no
default_md = sha256
x509_extensions = v3_req
distinguished_name = dn
[dn]
C = GR
ST = Attica
L = Athens
O = k8s
emailAddress = -your-email-address-
CN = k8s.sm0ke.local
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = k8s.sm0ke.local
DNS.2 = k8s.sm0ke.local:5000
DNS.3 = 192.168.1.200
DNS.4 = 192.168.1.200:5000
Feel free to tune these settings to your liking.
$ openssl req -new -x509 -newkey rsa:4096 -nodes -sha256 -keyout registry.key -days 3560 -out registry.crt -config registry.cnf
\# mkdir /usr/local/share/ca-certificates/k8s
\# cp registry.crt /usr/local/share/ca-certificates/k8s/
\# update-ca-certificates
We are importing this certificate and marking it as trusted.
You should also do this on your desktop operating system. I am using docker-desktop on Windows10. This will come in handy at a later stage where we will try and push images to the registry.
Right-click the certificate and select Install Certificate. Follow the prompts making sure that you select Current user for the store location then selecting Trusted Root Certification Authorities for the sertificate store.
Make sure you restart docker-desktop after doing this.
K3S Registry configuration
We must now configure K3S to use this registry. On your k8s host create the file /etc/rancher/k3s/registries.yaml
mirrors:
"k8s.sm0ke.local:5000":
endpoint:
- "https://k8s.sm0ke.local:5000"
configs:
"k8s.sm0ke.local:5000":
tls:
cert_file: /opt/certs/registry.crt
key_file: /opt/certs/registry.key
Restart K3S
# systemctl restart k3s
We have now taken care of the certificates and configured k3s to use the registry. We should crack on with running the registry!
# mkdir /opt/registry
private-registry.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: private-repository-k8s
labels:
app: private-repository-k8s
spec:
replicas: 1
selector:
matchLabels:
app: private-repository-k8s
template:
metadata:
labels:
app: private-repository-k8s
spec:
volumes:
- name: certs-vol
hostPath:
path: /opt/certs
type: Directory
- name: registry-vol
hostPath:
path: /opt/registry
type: Directory
containers:
- image: registry:2
name: private-repository-k8s
imagePullPolicy: IfNotPresent
env:
- name: REGISTRY_HTTP_TLS_CERTIFICATE
value: "/certs/registry.crt"
- name: REGISTRY_HTTP_TLS_KEY
value: "/certs/registry.key"
ports:
- containerPort: 5000
volumeMounts:
- name: certs-vol
mountPath: /certs
- name: registry-vol
mountPath: /var/lib/registry
private-registry-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: registry-service
spec:
selector:
app: private-repository-k8s
type: LoadBalancer
ports:
- name: docker-port
protocol: TCP
port: 5000
targetPort: 5000
loadBalancerIP: 192.168.1.200
Apply the registry
$ kubectl apply -f private-registry.yaml
$ kubectl apply -f private-registry-svc.yaml
You should now see the registry pod running
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
private-repository-k8s-686564966d-f2zd8 1/1 Running 0 11h
Run the following commands on your desktop to make sure you can push to the private registry
$ docker pull nginx
$ docker tag nginx k8s.sm0ke.local:5000/nginx
$ docker push k8s.sm0ke.local:5000/nginx
You can also check out what your registry holds by visiting: https://k8s.sm0ke.local:5000/v2/_catalog in your browser.
{"{"repositories":["busybox","ispy","nginx"]}"}
You can now reference this repository when deploying workloads to k8s.
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-test-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx-1-17
image: k8s.sm0ke.local:5000/nginx:latest
ports:
- containerPort: 80
Apply the deployment
$ kubectl apply -f deployment.yaml
We have now configured a k8s dev cluster (only the master node) with a working dynamic storage provisioner and a private registry. We can now use it for development (or other) purposes:
$ helm repo add minecraft-server-charts https://itzg.github.io/minecraft-server-charts/
$ helm repo update
$ kubectl create ns minecraft
$ helm -n minecraft install minecraft --set minecraftServer.eula=true --set persistence.dataDir.enabled=true minecraft-server-charts/minecraft
💪💪💪
Discovering / Accessing services
K3S uses 10.0.0.0/8 address space by default so make sure you route this class to your K8S installation. On Windows open up a Powershell window as administrator and type the following to add a persistent route entry:
route add -p 10.0.0.0 MASK 255.0.0.0 192.168.1.200
This way we are routing all 10.0.0.0/8 addresses to our K8S cluster at 192.168.1.200
Now we can access services using the internal IP addresses shown in kubectl. Let us access Longhorn's web UI:
$ kubectl -n longhorn-system get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
longhorn-backend ClusterIP 10.43.172.167 <none> 9500/TCP 22h
longhorn-frontend ClusterIP 10.43.199.16 <none> 80/TCP 22h
csi-attacher ClusterIP 10.43.156.66 <none> 12345/TCP 22h
csi-provisioner ClusterIP 10.43.112.101 <none> 12345/TCP 22h
csi-resizer ClusterIP 10.43.226.21 <none> 12345/TCP 22h
csi-snapshotter ClusterIP 10.43.222.124 <none> 12345/TCP 22h
We can see that the front-end service is running on 10.43.199.16 port 80. So open up a browser window and navigate to http://10.43.199.16
Notes
Originally I had considered installing Harbor which includes a whole lot of goodies such as Artifactory and Chartmuseum but I eventually decided against it since it is an overkill.
This post assumes a lot of knowledge of setting things up, I mostly wrote it to note down all the required steps since I do them again and again from scratch. Feel free to comment if something doesn't add up!
You can find all the relevant files in this github repo.
Edit 1 -- 26/08/2021
- Adds security notice about running scripts off of the internet.
- Adds Github repository with source files.