Published: Mar 19, 2021 by Isaac Johnson
I was chatting with some folks from Mirantis about Lens and some of their offerings and they highly encouraged me to try out K0s (“K Zeros”). This week I took some time to dig into it. My goal was mostly trying to see how it compared to Rancher’s K3s.
Setup
Let’s start with the pi and go from there. # spoiler: this didn’t go to well.
We first download the binary:
$ curl -sSLf https://get.k0s.sh | sudo sh
Downloading k0s from URL: https://github.com/k0sproject/k0s/releases/download/v0.11.0/k0s-v0.11.0-arm64
Verify the binary is installed:
$ k0s --help
k0s - The zero friction Kubernetes - https://k0sproject.io
This software is built and distributed by Mirantis, Inc., and is subject to EULA https://k0sproject.io/licenses/eula
Usage:
k0s [command]
Available Commands:
api Run the controller api
completion Generate completion script
controller Run controller
default-config Output the default k0s configuration yaml to stdout
docs Generate Markdown docs for the k0s binary
etcd Manage etcd cluster
help Help about any command
install Helper command for setting up k0s on a brand-new system. Must be run as root (or with sudo)
kubeconfig Create a kubeconfig file for a specified user
kubectl kubectl controls the Kubernetes cluster manager
reset Helper command for uninstalling k0s. Must be run as root (or with sudo)
status Helper command for get general information about k0s
token Manage join tokens
validate Helper command for validating the config file
version Print the k0s version
worker Run worker
Flags:
-c, --config string config file (default: ./k0s.yaml)
--data-dir string Data Directory for k0s (default: /var/lib/k0s). DO NOT CHANGE for an existing setup, things will break!
-d, --debug Debug logging (default: false)
--debugListenOn string Http listenOn for debug pprof handler (default ":6060")
-h, --help help for k0s
Use "k0s [command] --help" for more information about a command.
The thing to remember is that there is not an uninstall, however “reset” serves that purpose.
Installing k0s on Pi
Installing is a two step process (https://docs.k0sproject.io/latest/k0s-single-node/) . First we generate a config that drives it. This is very similar to kubespray or kubeadm
ubuntu@ubuntu:~/k0s$ k0s default-config > k0s.yaml
ubuntu@ubuntu:~/k0s$ cat k0s.yaml
apiVersion: k0s.k0sproject.io/v1beta1
images:
konnectivity:
image: us.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-agent
version: v0.0.13
metricsserver:
image: gcr.io/k8s-staging-metrics-server/metrics-server
version: v0.3.7
kubeproxy:
image: k8s.gcr.io/kube-proxy
version: v1.20.4
coredns:
image: docker.io/coredns/coredns
version: 1.7.0
calico:
cni:
image: calico/cni
version: v3.16.2
flexvolume:
image: calico/pod2daemon-flexvol
version: v3.16.2
node:
image: calico/node
version: v3.16.2
kubecontrollers:
image: calico/kube-controllers
version: v3.16.2
installConfig:
users:
etcdUser: etcd
kineUser: kube-apiserver
konnectivityUser: konnectivity-server
kubeAPIserverUser: kube-apiserver
kubeSchedulerUser: kube-scheduler
kind: Cluster
metadata:
name: k0s
spec:
api:
address: 192.168.1.208
sans:
- 192.168.1.208
- 172.17.0.1
storage:
type: etcd
etcd:
peerAddress: 192.168.1.208
network:
podCIDR: 10.244.0.0/16
serviceCIDR: 10.96.0.0/12
provider: calico
calico:
mode: vxlan
vxlanPort: 4789
vxlanVNI: 4096
mtu: 1450
wireguard: false
flexVolumeDriverPath: /usr/libexec/k0s/kubelet-plugins/volume/exec/nodeagent~uds
withWindowsNodes: false
overlay: Always
podSecurityPolicy:
defaultPolicy: 00-k0s-privileged
telemetry:
interval: 10m0s
enabled: true
Now install and start the service
ubuntu@ubuntu:~/k0s$ sudo k0s install controller -c ~/k0s/k0s.yaml --enable-worker
INFO[2021-03-16 18:14:08] creating user: etcd
INFO[2021-03-16 18:14:08] creating user: kube-apiserver
INFO[2021-03-16 18:14:09] creating user: konnectivity-server
INFO[2021-03-16 18:14:09] creating user: kube-scheduler
INFO[2021-03-16 18:14:09] Installing k0s service
ubuntu@ubuntu:~/k0s$ sudo systemctl start k0scontroller.service
Accessing it (or rather, trying to)
Trying to access proved a bit challenging. I couldn’t directly use the built in kubectl because it didn’t see me as logged in
ubuntu@ubuntu:~/k0s$ sudo chmod a+rx /var/lib/k0s/
ubuntu@ubuntu:~/k0s$ sudo chmod a+rx /var/lib/k0s/pki
ubuntu@ubuntu:~/k0s$ sudo chmod a+r /var/lib/k0s/pki/*.*
ubuntu@ubuntu:~/k0s$ k0s kubectl get nodes
Unable to connect to the server: x509: certificate signed by unknown authority
Then i tried a snap install
ubuntu@ubuntu:~/k0s$ cp /var/lib/k0s/pki/admin.conf ~/.kube/config
ubuntu@ubuntu:~/k0s$ sudo snap install kubectl --classic
kubectl 1.20.4 from Canonical✓ installed
ubuntu@ubuntu:~/k0s$ kubectl get nodes --insecure-skip-tls-verify
error: You must be logged in to the server (Unauthorized)
I actually tried a lot of hand rigged paths, making new configs, etc. Nothing would get past either the bad/cert signature issue or login issue.
Trying on WSL
First we download as before
builder@DESKTOP-JBA79RT:~/k0s$ curl -sSLf https://get.k0s.sh | sudo sh
[sudo] password for builder:
Downloading k0s from URL: https://github.com/k0sproject/k0s/releases/download/v0.11.1/k0s-v0.11.1-amd64
Then let’s make a config
builder@DESKTOP-JBA79RT:~/k0s$ k0s default-config > k0s.yaml
builder@DESKTOP-JBA79RT:~/k0s$ cat k0s.yaml
apiVersion: k0s.k0sproject.io/v1beta1
images:
konnectivity:
image: us.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-agent
version: v0.0.13
metricsserver:
image: gcr.io/k8s-staging-metrics-server/metrics-server
version: v0.3.7
kubeproxy:
image: k8s.gcr.io/kube-proxy
version: v1.20.4
coredns:
image: docker.io/coredns/coredns
version: 1.7.0
calico:
cni:
image: calico/cni
version: v3.16.2
flexvolume:
image: calico/pod2daemon-flexvol
version: v3.16.2
node:
image: calico/node
version: v3.16.2
kubecontrollers:
image: calico/kube-controllers
version: v3.16.2
installConfig:
users:
etcdUser: etcd
kineUser: kube-apiserver
konnectivityUser: konnectivity-server
kubeAPIserverUser: kube-apiserver
kubeSchedulerUser: kube-scheduler
kind: Cluster
metadata:
name: k0s
spec:
api:
address: 172.25.248.143
sans:
- 172.25.248.143
storage:
type: etcd
etcd:
peerAddress: 172.25.248.143
network:
podCIDR: 10.244.0.0/16
serviceCIDR: 10.96.0.0/12
provider: calico
calico:
mode: vxlan
vxlanPort: 4789
vxlanVNI: 4096
mtu: 1450
wireguard: false
flexVolumeDriverPath: /usr/libexec/k0s/kubelet-plugins/volume/exec/nodeagent~uds
withWindowsNodes: false
overlay: Always
podSecurityPolicy:
defaultPolicy: 00-k0s-privileged
telemetry:
interval: 10m0s
enabled: true
And while we can install.. WSL cannot do init services..
builder@DESKTOP-JBA79RT:~/k0s$ sudo k0s install controller -c ~/k0s/k0s.yaml --enable-worker
INFO[2021-03-19 08:02:54] creating user: etcd
INFO[2021-03-19 08:02:54] creating user: kube-apiserver
INFO[2021-03-19 08:02:54] creating user: konnectivity-server
INFO[2021-03-19 08:02:54] creating user: kube-scheduler
INFO[2021-03-19 08:02:54] Installing k0s service
builder@DESKTOP-JBA79RT:~/k0s$ sudo systemctl start k0scontroller.service
System has not been booted with systemd as init system (PID 1). Can't operate.
However, we can just start the service manually…
builder@DESKTOP-JBA79RT:~/k0s$ sudo /usr/bin/k0s controller --config=/home/builder/k0s/k0s.yaml --enable-worker=true
INFO[2021-03-19 08:09:26] using public address: 172.25.248.143
INFO[2021-03-19 08:09:26] using sans: [172.25.248.143]
INFO[2021-03-19 08:09:26] DNS address: 10.96.0.10
INFO[2021-03-19 08:09:26] Using storage backend etcd
INFO[2021-03-19 08:09:26] initializing Certificates
…
Note: this is a chatty service so expect a lot of log scroll…
In a fresh bash shell
$ sudo cat /var/lib/k0s/pki/admin.conf > ~/.kube/config
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
desktop-jba79rt Ready <none> 57s v1.20.4-k0s1
Trying on a mac
First inclination was to follow the same steps as linux. But the linux steps really are just arm and x64.
$ curl -sSLf https://get.k0s.sh | sudo sh
Password:
Downloading k0s from URL: https://github.com/k0sproject/k0s/releases/download/v0.11.1/k0s-v0.11.1-amd64
sh: line 39: /usr/bin/k0s: Operation not permitted
However, we can try with the kubectl binary. From releases (https://github.com/k0sproject/k0sctl/releases) we can download the latest k0sctl for mac: https://github.com/k0sproject/k0sctl/releases/download/v0.5.0/k0sctl-darwin-x64
As an unsigned binary, we have to fight macos (because they know better than me what i want to run on my computer…grrr)
You can go to security & privacy to allow. Of course, you used to be able to set “always allow”, but they disabled it (because they know better than me and want to force me to use their walled garden app store)
Once you click allow anyway, you are given the privilege of clicking open in the next window
We can now see the YAML file it created
$ cat k0sctl.yaml
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
name: k0s-cluster
spec:
hosts:
- ssh:
address: 10.0.0.1
user: root
port: 22
keyPath: /Users/isaacj/.ssh/id_rsa
role: controller
- ssh:
address: 10.0.0.2
user: root
port: 22
keyPath: /Users/isaacj/.ssh/id_rsa
role: worker
k0s:
version: 0.11.1
I thought perhaps it would create some local IP vms or something.. But clearly this is just like Kubeadm that plans to SSH as root and update nodes:
$ ./k0sctl apply -d --config k0sctl.yaml
⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███ ███ ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███ ███ ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███ ███ ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████ ███ ██████████
k0sctl v0.5.0 Copyright 2021, k0sctl authors.
Anonymized telemetry of usage will be sent to the authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
DEBU[0000] Preparing phase 'Connect to hosts'
INFO[0000] ==> Running phase: Connect to hosts
ERRO[0075] [ssh] 10.0.0.2:22: attempt 1 of 60.. failed to connect: dial tcp 10.0.0.2:22: connect: operation timed out
ERRO[0075] [ssh] 10.0.0.1:22: attempt 1 of 60.. failed to connect: dial tcp 10.0.0.1:22: connect: operation timed out
ERRO[0080] [ssh] 10.0.0.1:22: attempt 2 of 60.. failed to connect: dial tcp 10.0.0.1:22: connect: connection refused
Docker version
Since i’m looking for a fast solution, let’s move on to the dockerized version and see how well that works.
docker run -d --name k0s --hostname k0s --privileged -v /var/lib/k0s -p 6443:6443 docker.io/k0sproject/k0s:latest
Unable to find image 'k0sproject/k0s:latest' locally
latest: Pulling from k0sproject/k0s
f84cab65f19f: Pull complete
62ee4735a41a: Pull complete
86e9485af690: Pull complete
4fe5e4ec65da: Pull complete
30d8cc0db847: Pull complete
Digest: sha256:5e915f27453e1561fb848ad3a5b8d76dd7efd926c3ba97a291c5c33542150619
Status: Downloaded newer image for k0sproject/k0s:latest
387ade843d505caafdfc6f1234c9b1c1da6317407c961d79ea17bd5c667bb042
We can see this ran
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
387ade843d50 k0sproject/k0s:latest "/sbin/tini -- /bin/…" 37 seconds ago Up 35 seconds 0.0.0.0:6443->6443/tcp k0s
Let’s grab that kubeconfig:
$ docker exec k0s cat /var/lib/k0s/pki/admin.conf
apiVersion: v1
clusters:
- cluster:
server: https://localhost:6443
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBRENDQWVpZ0F3SUJBZ0lVTXpFRDFsTWk5K21jLzYzWCtINmpVUVk5V1Jrd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dERVdNQlFHQTFVRUF4TU5hM1ZpWlhKdVpYUmxjeTFqWVRBZUZ3MHlNVEF6TVRreE1USTBNREJhRncwegpNVEF6TVRjeE1USTBNREJhTUJneEZqQVVCZ05WQkFNVERXdDFZbVZ5Ym1WMFpYTXRZMkV3Z2dFaU1BMEdDU3FHClNJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURocUxTZWdTYU5TY1hJa2JHOEE1eWlCbDhwWjV1M2RtZHkKTHA3R2JMZlRvRVFYazUvRHk2bi9xVXZMQThMcFZKV05xVU1tSDV4MFFMaWZXSmt4am5XNXVWNUFhVkdrdDB2VgpDaVVzazY0bHNJY2xpem5xY09oNFRVb3VQVUZZNVQrQUp4aDV2eUxSTjJVV3B6TkpqZDkvVG4xZ0RJWStSczdkCnZ2TVdBbkt2dkNFUmkxQmxtTStYZWM3WTVmTkJiTzZ0NWZtTGJpMm5UQUFUMHZNdHRYUFNhaUd2K0pqN2ZWbjAKZm1JRTNkanY0VkNTY3pnWU10Tmt1T3dDMzJteHZBSEhVYXNFVzBWbDdZemJ2K1FSaTd6aWNXckhLeGdsaWlGcwo3dy80L3NrSGoySHZkYlRMc2k5RzRKTkJld1UvQ1dlVVNaTFo2ait2eU9peWdnRmdSTkZaQWdNQkFBR2pRakJBCk1BNEdBMVVkRHdFQi93UUVBd0lCQmpBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRS2J3U2wKbFFjUlJHYjZxMmI4NUswZFZaTHByVEFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBakY1MitEOEQrQkpodXFDcQo1dVV4N2RTMFZpcFZzQWtaWmxTZFNmWGc5aUdOVjBzQ0NJbXdBMWdBS2Q0RE5qdjRnTElhSTgwU2hXcnFoY0c5CjhFaElEbW85UzBEWjdmbisySGNMTUIxTnNWMmRNTW5jVE5oSC9SUmRhZWZ3aFE4NnZYdnRvcVlhRURWRm1lUkIKaWorYWdTL0gyTkU1SmsvaEloV2QycDArYTdsbUVKdm9oajQ0Mi9yQnpJcXVNSEdWSVNjaVhvRzNqUUhCUWdzdApKRE56amxZc3V0REpTcXY1N3RTeHkxaFpCTXF1SU5seWQyak5ibEZmaUNCRlp3NUJrV2hSS01MVVl6YTFJYXVsCnpxcHl2N0s3RjhONHA4S2t6VHdjNHMzZmRBNU9jcGJRQUpmdXl1RmhVRXREYk9DMi9HVHpYTncrZEcyN2ZralcKL2xxT3lnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
name: local
contexts:
- context:
cluster: local
namespace: default
user: user
name: Default
current-context: Default
kind: Config
preferences: {}
users:
- name: user
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURXVENDQWtHZ0F3SUJBZ0lVRU9KaE83dG5wa21wYWJqSUFrbFI0TW5wb2o0d0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dERVdNQlFHQTFVRUF4TU5hM1ZpWlhKdVpYUmxjeTFqWVRBZUZ3MHlNVEF6TVRreE1USTBNREJhRncweQpNakF6TVRreE1USTBNREJhTURReEZ6QVZCZ05WQkFvVERuTjVjM1JsYlRwdFlYTjBaWEp6TVJrd0Z3WURWUVFECkV4QnJkV0psY201bGRHVnpMV0ZrYldsdU1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0MKQVFFQTcveENGampVeXZ2VmNIT1JCbFI0cDNsSk1NbWV4NXhNMVVoeU1qYmVjbjhPSm16aS9jNXNIRFZkdi9lRApUaWdVMGFIR3FSeXE2RWdmQmdsWHZPOHJZR3VlUE5mUkp4TU5FcFFhMWp2MnVtY0grRVowMmxKbFJEMDU0RE9KCmdBRzJIM1ZsTmthTFFtVGNNRnorRUNVaTJyeHNmdmJwUkU4WHhDcS9vWXJnZzNvSVozQTZSN21laUdGV1NlbjIKRnZ0WC82Zkd2T3MvaGZ1ekIvamsyV1RkMGthNkJNc241OExLM1BWeDZoc05yL3hVZWtTbkRQdXpjVDVtUE0yUQowdWdoMGZNd0NKVDBtalpwVSszSkpOd3FiblNFZzVCWXhSaEhoYXdqTVZRbmUzaERrK013ZGtxMlRvNWt5RFF5CmRycUl4MFlaMjJBdmVpQ2xEbjZPOUpWWDRRSURBUUFCbzM4d2ZUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0hRWUQKVlIwbEJCWXdGQVlJS3dZQkJRVUhBd0VHQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXdIUVlEVlIwTwpCQllFRkJDMEpSOE4yY2pvbXp0Uk0zRExZTU04TzBYUk1COEdBMVVkSXdRWU1CYUFGQXB2QktXVkJ4RkVadnFyClp2emtyUjFWa3VtdE1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQWxCaFRhMVR4M3VKbnNIY0FEdXVSeFNFNm0KYVkxdlp1K0hBTHo5dFlnTW53eUt5aGZKQnRLS3VtUE43SXlrR21zRDFSNDEydXRsVTQ2QWdqL1BLdVFoYUpHbApacHpoRDcvenI3M29GZ29nQWlSYTd3bW1iYVRxZmh4OHpxR3hqWGtGRnZhRjZRQVZDVzhYVlkzUU8rN1FxSDBXCmtvNyt6Rjl5UG5LdWQvQWZnZHlWUTRheWI5NE9pQ2lmdGNOcTZZU1hWb1lmbHBibXo3NTBKbEV2T2QvUUt6WEoKbHNrQjM3eFNZZytHUlpJWVdtdDRnOWtXZzlQRHMwM0dQVnlEcWJPV1RzU3l0Y3o5MzliTmhCWURXSzFsbTA3RgpReXFVaGdOOUt4L3hYc0NhNmpHMjg2cXBQVGl5UlRUampPa01pUE1mVk9lcFRINXFESGNnU0d0UTBHOVkKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBNy94Q0ZqalV5dnZWY0hPUkJsUjRwM2xKTU1tZXg1eE0xVWh5TWpiZWNuOE9KbXppCi9jNXNIRFZkdi9lRFRpZ1UwYUhHcVJ5cTZFZ2ZCZ2xYdk84cllHdWVQTmZSSnhNTkVwUWExanYydW1jSCtFWjAKMmxKbFJEMDU0RE9KZ0FHMkgzVmxOa2FMUW1UY01GeitFQ1VpMnJ4c2Z2YnBSRThYeENxL29ZcmdnM29JWjNBNgpSN21laUdGV1NlbjJGdnRYLzZmR3ZPcy9oZnV6Qi9qazJXVGQwa2E2Qk1zbjU4TEszUFZ4NmhzTnIveFVla1NuCkRQdXpjVDVtUE0yUTB1Z2gwZk13Q0pUMG1qWnBVKzNKSk53cWJuU0VnNUJZeFJoSGhhd2pNVlFuZTNoRGsrTXcKZGtxMlRvNWt5RFF5ZHJxSXgwWVoyMkF2ZWlDbERuNk85SlZYNFFJREFRQUJBb0lCQUFrcVlqL0ZCZU9Hd0wwZAp2TkU0RXErYXd6L1l3M2F1TW9VRzBrMjJxTUpJdGZxanQvdW5rWnQyTWxGdG01YzFrcTM2OXNKb3RPSlFOdGNkCkxqdXFDc2NROVNjdnV1NDZIUXF6bTE3SlJZQVRCVXZuTFBsVm9rWjc3Q1R5OHZKdm5rd1BTTUZNNm02cnVYa2oKWkJTcnhtd0NQRWFSR1pHaFQ3QVBUZWhXaklLWTZKLzZWWnU3M2VZQVJDQk9JS3ZPY3k4VmRzV2NlMmQvMHcrVQpFYkk5c0k2bmpiTTZmdzA1Y1IxN3FUcUxCWnhWeEdQWTVWa3kzWUdSQ3RsaUZNRGZaeGlYM3AvTThUVzdLalJZCmtGeXNEeklyT0JJUDJlQVVjOGFQa1daK2ExdVJxcTdpNllpQmlTUnVnT081WnBCblViVEdxcHZhWVRtVkdRNzAKZ2ZNaSswRUNnWUVBK3dqbkRYTDAveDNHdG1sWm9qV1krY0E2TDFwa2xkdmhlV2kvSFNwUERxZ01HdFphTTJGVwo4OFRyWGpZVDgveW9SR2MwTmU0eVBqdEFKdXJac0xJckMzVzRuNkwvOUJCZXJncHQ5VjEvU2N3b1lJTklBYllFCkwzWTltQnpHKzhTNGtJaCtQL29odUU4V2pRTkxjTzh3VC9hNlBLeEJJVXdpbmp6NmtPQW5RR2tDZ1lFQTlMdG8KWXUzSEEwQ25TenhIME1TSG1SclpUZGIrYlliL2VUSjkzbHVXWkhSZVlmQ0xIQWhkZkpZUlhqNFRqLyt3Mnp5UApQdVIza0Y5Yi9qcHZuZldQeUI1YXdsZURHajVTNFVneGcwTXRieTlseWFsYTI5SmxBcFNwK0dpS0hoeTNmYzNECk1qVGRwbG1DSTljVnVuNEtRd2hhQS84TldudDhSZ1Uxd3BjLzdMa0NnWUVBMTJkMU0ySWYvY1RrSE9QYktNZ0sKNHN0aVlmMlRiOC9EOHJUQndObnNDbXlDTG9rZHp4YklVTlg1RE5ja1dlakR6aVlzYzlaWFFIVUJBQ1BtOWFwOApLeEl4Z0xHU0pTL2l2ajV1eWVzWGJSQ0UyUVB0UnFLVGh0SlQyZkZmZ092MVh2ZndOUitCemEvM2JycVVBbTBMCnJLSE9mbjlrUjVrWDMyWDlyMURYL1hrQ2dZRUF2SFR3Zm5ZQ01iUUEzOFBNdmF3SmkxSU1rbytEbjQ0OGZ2VHQKem1RUzNNcHJ2OW0vRmRndlBYaEdhWjg5Nno2SHoxdkVKemRDQnpBWHBCOFZ3cnJOZk5vN0k3ckdIMWhzOUVSbQp0R0R2eE5YbitUSHI0S2tVMWJicmFIb0FHZzRkRXNoM2p0cjg4Rk92RHJCYmNDQU5BTytXZWN0WFdoMExadFF1ClFPbTk5U2tDZ1lCMVZMOFFiSXJMVUhKRzhOUnAvV0tvbnI5NnJPRm85Tk80YTBqdFYyT1dVVjUyTk1SeUJUMUcKWjZTMU01SUp2M1o2eHZIalMvVWlHenpseTFUQm93K0ZtbHk4Q2dJWHBCUEZYek5Fb1hEYzQzNWt4NHAybWJhQQpoRWhWVEtyZlcvdTlNUXFhaXUvbGtPSWljZHdpWktoV3BvSXltaUN1SE9jUEp6MXZsaERTWVE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
We can pipe that to our kubeconfig to view the nodes:
$ docker exec k0s cat /var/lib/k0s/pki/admin.conf > ~/.kube/config
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k0s Ready <none> 2m8s v1.20.4-k0s1
Next let’s add a worker
isaacj-macbook:k0s isaacj$ docker exec -t -i k0s k0s token create --role=worker
H4sIAAAAAAAC/2yVzW7jOhKF93kKv0D31Y+Ve21gFi2bjCJHzJAiixJ3tKgbWdRfJMWRNZh3H8TdDcwAsytWHXyHIIg6D3q4QDlOl77bb67uQ9F8THM5TvuHb5tf9f5hs9lspnK8luN+U83zMO3/+MP90/vu/vnd+e7tH7db/64pynG+/H0p9Fx+0x9z1Y+X+fbN6FnvNy+pM7+k7oEJE/NLeGQQi1SomDpYsPvMmQ/WjVPBQobIkUoYlIP9VMShchrg2YDZEU9c2uDkufXLmq/yMD+TdgABNgA3Ho3DmBTDy9kdYh2pmlvMmPg80QaHxjGIgSG0wRHlGJjAWy6CKnHVIJvqZGDIRbvUJcfvEliohPKTqCGA8CMHNpbIFSkPCUNxxbris3wbfs/q/55xEXclUu8U4KCcQFKLCSCWmSNWZ1BB3roywUPOM6YSC77yDNbCDRNkjsLH0aEhcV4TJAQcKGCgAjmpiJFx8F2bOvj69SZCsL4QC1fS8FwEPHerWHtx9IqQW8rmcD5Wn7IGN/HYrCJ74tEPn3lxohp2ZYBzvYori6x3vuwWyFRCeZUUWJ3ACRYQ7pweYZtgnGipsrSdt7oNMpKBJAJXYM1ojqEHb8NRA6x6zZ1zROLcW4ayDZbc2fUEMzj7QEEoRYCOVAxbfQSvFEvK6xjADx+5Hd7V0V7habsoh8UyndNiNfbQKY+DCc929owlWLR2oe0y83TOlUx8yaHlNr7wVTlEqpk/xUPSBoIKLJJIEROxXGBSaWG8kzO8E0/Jc/3jpFo3ZpxY3eUOWMJzf+ikcB3ezi73zTFZ47mMVJiiCvKMYFhDeT4aVbaxd3Jxqrl51JJkha1eyiczadk8FZ+9b25/OS8+GdOn/pZGyuYNSwrPBmxlJ27jxrjiSl3TABDNce/py+yVYjeU0nSsNSm3SlNpCLU4ZN7AtI3Dg3VDgkyYAFgWGUwvO18ICI3THGg7hALFHW/Us7Dugcp8y0QsqGChEG9XLp4/mYMWkAwpF2fUxiz1Yl94n6czxrVo4iiv1ZK0z1sipk+FleZReAOEX2lrxkJWoxZFYNbw6w+uzOKQOyT88tc2d5PLjF4RHakdepPhY0F71wBsicd4gtVQYLVSOWvZLlxh0sqnItDCvEIdrtQh8TkzYSJNmHrUYSh4NzXrOGqqlL99Cq/Kig73uVMEh7rCGjXoLP8KxBoiWZv2fJlu6RNJuHheeEdk0jLCZVADCvr0sEtFyyolla8x2pJO5aZj1wKaigkmWes2wj6ftOzHXBr+4rzduBVu2k5XjZoePHorjj/GnJvpLOBk2l2va+okl92NdkNcZEBSZGQKpNZQXdlK3oWoDkKa1fwYTgwFj7pdVOGDw9DAiyx3ic94GdlF4+HAM+ymIphKSW/aBpczUq0W5MCawSciHsGr0tRxE4DmMec4zjOYDt2wFFHjEWfyWV29kujHNvXmR4hMTaJkVS0LidjVxVPMqBhakzUuaytgGUO53R2Tyy6CaMh5V4wKFTfiqVE3xenFWxbuN90/uXP6tZsxt/SNOpACap5SQUJA4r6XX2j/j4fNptNtud9YZ3oo+m4ul/lnSvysf6bE78i4q74aH9P99HEum3L+du77eZpHPfwv7WMcy27+9pt0b9pLZ/abQ9/9fXl7GMby73Isu6Kc9pt//fvhi3o3/wX5P/i78f0Kc2/Lbr9Zx2nbPn73bD+U29LV3Vju7G5++E8AAAD//9nwda4CBwAA
# save token to env var
isaacj-macbook:k0s isaacj$ export token=`docker exec -t -i k0s k0s token create --role=worker`
Add worker1
docker run -d --name k0s-worker1 --hostname k0s-worker1 --privileged -v /var/lib/k0s docker.io/k0sproject/k0s:latest k0s worker $token
And in a moment we can see it come up…
$ docker run -d --name k0s-worker1 --hostname k0s-worker1 --privileged -v /var/lib/k0s docker.io/k0sproject/k0s:latest k0s worker $token
e98d00411e5258a053d9cfbd107e05f66dda0cd190545da2b263800201392315
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k0s Ready <none> 4m57s v1.20.4-k0s1
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k0s Ready <none> 5m11s v1.20.4-k0s1
k0s-worker1 NotReady <none> 12s v1.20.4-k0s1
This worked, however the documentation from k0s notes that we have some network limitations doing it via docker containers:
Known limitations#
No custom Docker networks
Currently, we cannot run k0s nodes if the containers are configured to use custom networks e.g. with --net my-net. This is caused by the fact that Docker sets up a custom DNS service within the network and that messes up CoreDNS. We know that there are some workarounds possible, but they are bit hackish. And on the other hand, running k0s cluster(s) in bridge network should not cause issues.
Testing it
One of my favourite quick test runs is to use the Azure Vote app: https://github.com/Azure-Samples/azure-voting-app-redis/blob/master/azure-vote-all-in-one-redis.yaml
$ cat azure-vote-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: azure-vote-back
spec:
replicas: 1
selector:
matchLabels:
app: azure-vote-back
template:
metadata:
labels:
app: azure-vote-back
spec:
nodeSelector:
"beta.kubernetes.io/os": linux
containers:
- name: azure-vote-back
image: mcr.microsoft.com/oss/bitnami/redis:6.0.8
env:
- name: ALLOW_EMPTY_PASSWORD
value: "yes"
ports:
- containerPort: 6379
name: redis
---
apiVersion: v1
kind: Service
metadata:
name: azure-vote-back
spec:
ports:
- port: 6379
selector:
app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: azure-vote-front
spec:
replicas: 1
selector:
matchLabels:
app: azure-vote-front
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
minReadySeconds: 5
template:
metadata:
labels:
app: azure-vote-front
spec:
nodeSelector:
"beta.kubernetes.io/os": linux
containers:
- name: azure-vote-front
image: mcr.microsoft.com/azuredocs/azure-vote-front:v1
ports:
- containerPort: 80
resources:
requests:
cpu: 250m
limits:
cpu: 500m
env:
- name: REDIS
value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
name: azure-vote-front
spec:
type: LoadBalancer
ports:
- port: 80
selector:
app: azure-vote-front
Install it
$ kubectl apply -f ./azure-vote-app.yaml
deployment.apps/azure-vote-back created
service/azure-vote-back created
deployment.apps/azure-vote-front created
service/azure-vote-front created
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
azure-vote-back 0/1 1 0 5s
azure-vote-front 0/1 1 0 5s
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
azure-vote-back 1/1 1 1 21s
azure-vote-front 0/1 1 0 21s
The first try at pulling the image failed.. I am going to assume it’s mostly due to my overburdened Macbook and not the software…
Warning Failed 44s kubelet Error: ImagePullBackOff
Normal Pulling 33s (x2 over 2m4s) kubelet Pulling image "mcr.microsoft.com/azuredocs/azure-vote-front:v1"
Normal Pulled 2s kubelet Successfully pulled image "mcr.microsoft.com/azuredocs/azure-vote-front:v1" in 30.1524056s
We can see it’s running now:
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
azure-vote-back 1/1 1 1 2m49s
azure-vote-front 1/1 1 1 2m49s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
azure-vote-back ClusterIP 10.109.184.1 <none> 6379/TCP 3m9s
azure-vote-front LoadBalancer 10.107.123.105 <pending> 80:30368/TCP 3m9s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 12m
And we can see it’s running:
Via K8s
Let’s get our local cluster:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
isaac-macbookpro Ready <none> 82d v1.19.5+k3s2
isaac-macbookair Ready master 82d v1.19.5+k3s2
Can we deploy k0s into k8s?
Let’s create a quick YAML for the master k0s:
$ cat k0s-master.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: k0s-master
spec:
replicas: 1
selector:
matchLabels:
app: k0s-master
template:
metadata:
labels:
app: k0s-master
spec:
nodeSelector:
"beta.kubernetes.io/os": linux
volumes:
- name: k0smount
containers:
- name: k0s-master
image: docker.io/k0sproject/k0s:latest
volumeMounts:
- name: k0smount
mountPath: /var/lib/k0s
ports:
- containerPort: 6443
name: kubernetes
securityContext:
allowPrivilegeEscalation: true
---
apiVersion: v1
kind: Service
metadata:
name: k0s-master
spec:
ports:
- port: 6443
selector:
app: k0s-master
And we can apply:
$ kubectl apply -f ./k0s-master.yaml
deployment.apps/k0s-master created
service/k0s-master created
The first time through, it was mad about the volume mount perms.
kubectl logs k0s-master-6985645f66-v9tpz
time="2021-03-19 12:08:24" level=info msg="no config file given, using defaults"
Error: directory "/var/lib/k0s" exist, but the permission is 0777. The expected permission is 755
Usage:
This can be fixed with a bit of an init container hack
$ cat k0s-master.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: k0s-master
spec:
replicas: 1
selector:
matchLabels:
app: k0s-master
template:
metadata:
labels:
app: k0s-master
spec:
nodeSelector:
"beta.kubernetes.io/os": linux
volumes:
- name: k0smount
containers:
- name: k0s-master
image: docker.io/k0sproject/k0s:latest
volumeMounts:
- name: k0smount
mountPath: /var/lib
ports:
- containerPort: 6443
name: kubernetes
securityContext:
allowPrivilegeEscalation: true
initContainers:
- name: volume-mount-hack
image: busybox
command: ["sh", "-c", "mkdir /var/lib/k0s && chmod 755 /var/lib/k0s"]
volumeMounts:
- name: k0smount
mountPath: /var/lib
---
apiVersion: v1
kind: Service
metadata:
name: k0s-master
spec:
ports:
- port: 6443
selector:
app: k0s-master
Now when applied, we see a running container:
$ kubectl get pods -l app=k0s-master
NAME READY STATUS RESTARTS AGE
k0s-master-b959fd5c4-vj662 1/1 Running 0 82s
$ kubectl get svc k0s-master
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
k0s-master ClusterIP 10.43.111.145 <none> 6443/TCP 2m15s
Now, let’s keep the insanity going and dump the kubeconfig into a local file:
kubectl exec k0s-master-b959fd5c4-vj662 -- cat /var/lib/k0s/pki/admin.conf > ~/.kube/k0s-in-k8s-config
We can now port forward to the master node:
$ kubectl port-forward k0s-master-b959fd5c4-vj662 9443:6443
Forwarding from 127.0.0.1:9443 -> 6443
Forwarding from [::1]:9443 -> 6443
And in another shell, i’ll sed the config to use the portforward (9443) and check on nodes and deployments
$ sed -i '' 's/:6443/:9443/' ~/.kube/k0s-in-k8s-config
$ kubectl get nodes --kubeconfig=/Users/jmyuser/.kube/k0s-in-k8s-config
No resources found
$ kubectl get deployments --kubeconfig=/Users/myuser/.kube/k0s-in-k8s-config
No resources found in default namespace.
$ kubectl get ns --kubeconfig=/Users/myuser/.kube/k0s-in-k8s-config
NAME STATUS AGE
default Active 11m
kube-node-lease Active 11m
kube-public Active 11m
kube-system Active 11m
If we wanted to continue, we could create a token from the master and use that in a deployment of a worker node..
$ kubectexec k0s-master-b959fd5c4-vj662 -- k0s token create --role=worker
H4sIAAAAAAAC/2xV246jPNa9z1PkBao/G5L6vor0XxSJHQKBxMYH8B1gOiQ2hyLk+GvefVTpbmlGmrvtvZbWsizvvSZ5fxTVcD527WJ6hZPSXs5jNZwXk7fp73oxmU6n03M1XKthMa3HsT8v/voLgh8z5wf44bqL99nMfXHKahiPP49lPlZv+WWsu+E4Pt50PuaL6TYB4zaBS8p1wI7eioqAJ1wFBGBOXxgYlwYGCaceRfGKSNErgN2EB54CViSuenCjarmKDWvqmAvs6NZ7j5C+hI7dMYZmGlAqeb8tYB/kvjoxgynlt5BY7GmgERU6Jhb7hGFBOZ4xPq8jqHpp61CLPuPN/VQx/CUF9RRXbuTbWCD8zgQdKgS/aGpjioKatuWtOvR/sNN/YowHbYXUFxFiqcBcEoNjgWiqV1gVQs2zBsoI9xlLqYqMcJWjcc6hFyG94i72lzYOslOMOBdLIrAgHIGEB0gD/OImAF+/34TzyC2a2GhUp4Xp32l6cHbr0okh3RMzg9Tae4VxImXA8tMhzBuNCqRC6pS3THyCJI0vlU9bdorXGft8FEhRbQ9PkdKs8vmDOkGUrCGLnnqVG3Qt/F7mDkR5E4QE2nYn70J89tfc38wSHB91SvdS4D4E+sEaMadQOyXcPEMgEubjkJyUVVa5GTAzZUQdoc2zhDojQKQE3NvcapQAvC9sNiwbsY+OoyNWNBLrD3eX9h1f82d5qq+56NchsCe67nuejDUHCnJfRTr1tkUD86ztM7qO/cygGbeWlSleM0B5LEiomvqqjQgKi7sKqy/iklu5yp7S2Cw3+rQFwsuxavh6TKNn0HBfJxXHULeByWUwYzDYF9iToTMPcwRnBYhuu89uxtNskK2+ZhLz0FFP2dIxt9284AKUEq4qecdc3jeU6z0z3aCWY11AdBOgxiH44MTFFyJ1TAz2qNPT3ATe0kAvRtqLhDDU15gcP1zOhaeBXZKm9zgKWmbVhhu4JDKbUR5wwqnH+eHK+OZGAboLSZGCOCUm4NKMrj6ewxDcccTqPFodbtz3Op5SoLh1k1QBivCONHooZT3kvJzrp/f9B5/UYI+B2Pv2Z6voRpw63vGPZcLsRoEOklvvZ7J7aITvNK3fibl7W0ekAkDLHx/HrevNYiBc7nQPurKXZB3UlRG7GNA7YdEjR2oQT/rkqTUhrFksx20Me7E8jqu8tUnCAl4JdQ2dmGSG+uTxj1u0lkqpW2rmAeHwRK3wRVr3pRFRBOKVgIdB+F6kpIh3vLsKV8yTk71E7BAmWFx5U7scqU2Vbp7stBm4DWDR4oCy+L0y8aZq6lX0tCvNFdnC+pYsP3ZsfX9PcLwnbr8pntirUM24f7jlAj856W6ygVGFaF58zykXX1rGTiG1KfxPyHGQUN9CKuy7bOicuSJKEG5LP4M7XIu8EfuCx+tMiozxeSN93KhGs6W1u2R9Bju/t6z5x5ErvU/QeE3MbEh4EGk5txyWM56ieXI6uLqNbxTUD4pqv1iPc51in5h7XlrblMZ2ie9dQicKicCGJCPdMxD+3s2YGXIgQCQC2XXCY08g/trLW9L932Q6bfOmWkwNOE/Krh2r+/grJX7Vv1LiT2S8WN+Ny/l1uhSVrca3ouvG8zjk/X+rXYahase3P0qvpjm2ejFddu3P42HSD9XPaqjasjovpv//r8m36sv8t8j/kH8Zv64wdqZqF1Nz/Buc9I+P+uv+9/Ewg83jeOt+NpN/BwAA//85gYNTAgcAAA==
and defining a worker YAML using that token
$ cat k0s-worker.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: k0s-worker1
spec:
replicas: 1
selector:
matchLabels:
app: k0s-worker1
template:
metadata:
labels:
app: k0s-worker1
spec:
nodeSelector:
"beta.kubernetes.io/os": linux
volumes:
- name: k0smount
containers:
- name: k0s-worker1
image: docker.io/k0sproject/k0s:latest
command: ["k0s", "worker", "H4sIAAAAAAAC/2xV246jPNa9z1PkBao/G5L6vor0XxSJHQKBxMYH8B1gOiQ2hyLk+GvefVTpbmlGmrvtvZbWsizvvSZ5fxTVcD527WJ6hZPSXs5jNZwXk7fp73oxmU6n03M1XKthMa3HsT8v/voLgh8z5wf44bqL99nMfXHKahiPP49lPlZv+WWsu+E4Pt50PuaL6TYB4zaBS8p1wI7eioqAJ1wFBGBOXxgYlwYGCaceRfGKSNErgN2EB54CViSuenCjarmKDWvqmAvs6NZ7j5C+hI7dMYZmGlAqeb8tYB/kvjoxgynlt5BY7GmgERU6Jhb7hGFBOZ4xPq8jqHpp61CLPuPN/VQx/CUF9RRXbuTbWCD8zgQdKgS/aGpjioKatuWtOvR/sNN/YowHbYXUFxFiqcBcEoNjgWiqV1gVQs2zBsoI9xlLqYqMcJWjcc6hFyG94i72lzYOslOMOBdLIrAgHIGEB0gD/OImAF+/34TzyC2a2GhUp4Xp32l6cHbr0okh3RMzg9Tae4VxImXA8tMhzBuNCqRC6pS3THyCJI0vlU9bdorXGft8FEhRbQ9PkdKs8vmDOkGUrCGLnnqVG3Qt/F7mDkR5E4QE2nYn70J89tfc38wSHB91SvdS4D4E+sEaMadQOyXcPEMgEubjkJyUVVa5GTAzZUQdoc2zhDojQKQE3NvcapQAvC9sNiwbsY+OoyNWNBLrD3eX9h1f82d5qq+56NchsCe67nuejDUHCnJfRTr1tkUD86ztM7qO/cygGbeWlSleM0B5LEiomvqqjQgKi7sKqy/iklu5yp7S2Cw3+rQFwsuxavh6TKNn0HBfJxXHULeByWUwYzDYF9iToTMPcwRnBYhuu89uxtNskK2+ZhLz0FFP2dIxt9284AKUEq4qecdc3jeU6z0z3aCWY11AdBOgxiH44MTFFyJ1TAz2qNPT3ATe0kAvRtqLhDDU15gcP1zOhaeBXZKm9zgKWmbVhhu4JDKbUR5wwqnH+eHK+OZGAboLSZGCOCUm4NKMrj6ewxDcccTqPFodbtz3Op5SoLh1k1QBivCONHooZT3kvJzrp/f9B5/UYI+B2Pv2Z6voRpw63vGPZcLsRoEOklvvZ7J7aITvNK3fibl7W0ekAkDLHx/HrevNYiBc7nQPurKXZB3UlRG7GNA7YdEjR2oQT/rkqTUhrFksx20Me7E8jqu8tUnCAl4JdQ2dmGSG+uTxj1u0lkqpW2rmAeHwRK3wRVr3pRFRBOKVgIdB+F6kpIh3vLsKV8yTk71E7BAmWFx5U7scqU2Vbp7stBm4DWDR4oCy+L0y8aZq6lX0tCvNFdnC+pYsP3ZsfX9PcLwnbr8pntirUM24f7jlAj856W6ygVGFaF58zykXX1rGTiG1KfxPyHGQUN9CKuy7bOicuSJKEG5LP4M7XIu8EfuCx+tMiozxeSN93KhGs6W1u2R9Bju/t6z5x5ErvU/QeE3MbEh4EGk5txyWM56ieXI6uLqNbxTUD4pqv1iPc51in5h7XlrblMZ2ie9dQicKicCGJCPdMxD+3s2YGXIgQCQC2XXCY08g/trLW9L932Q6bfOmWkwNOE/Krh2r+/grJX7Vv1LiT2S8WN+Ny/l1uhSVrca3ouvG8zjk/X+rXYahase3P0qvpjm2ejFddu3P42HSD9XPaqjasjovpv//r8m36sv8t8j/kH8Zv64wdqZqF1Nz/Buc9I+P+uv+9/Ewg83jeOt+NpN/BwAA//85gYNTAgcAAA=="]
volumeMounts:
- name: k0smount
mountPath: /var/lib
ports:
- containerPort: 6443
name: kubernetes
securityContext:
allowPrivilegeEscalation: true
initContainers:
- name: volume-mount-hack
image: busybox
command: ["sh", "-c", "mkdir /var/lib/k0s && chmod 755 /var/lib/k0s"]
volumeMounts:
- name: k0smount
mountPath: /var/lib
Then apply it
$ kubectl apply -f k0s-worker.yaml
deployment.apps/k0s-worker1 created
And we see both working now..
$ kubectl get pods | tail -n2
k0s-master-b959fd5c4-vj662 1/1 Running 0 17m
k0s-worker1-7bb8444844-2z8hv 1/1 Running 0 48s
While it launched, i don’t see the nodes show up:
$ export KUBECONFIG=/Users/isaacj/.kube/k0s-in-k8s-config
$ kubectl get nodes
No resources found
But I am seeing a running control plane..
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5f6546844f-6s676 0/1 Pending 0 17m
kube-system coredns-5c98d7d4d8-h9clg 0/1 Pending 0 17m
kube-system metrics-server-6fbcd86f7b-q7hhk 0/1 Pending 0 17m
It does seem kubenet is having troubles.. But other than that, the pod logs looked fine
$ kubectl logs k0s-worker1-7bb8444844-2z8hv | tail -n5
time="2021-03-19 12:37:50" level=info msg="\t/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:368" component=kubelet
time="2021-03-19 12:37:50" level=info msg="created by k8s.io/kubernetes/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).Start" component=kubelet
time="2021-03-19 12:37:50" level=info msg="\t/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:134 +0x191" component=kubelet
time="2021-03-19 12:37:50" level=warning msg="exit status 255" component=kubelet
time="2021-03-19 12:37:50" level=info msg="respawning in 5s" component=kubelet
I did try adding a command to enable work in the k0s-master and tried again:
containers:
- name: k0s-master
image: docker.io/k0sproject/k0s:latest
command: ["k0s", "controller", "--enable-worker"]
Then got fresh k8s config:
isaacj-macbook:k0s isaacj$ kubectl exec k0s-master-76d596b48f-68x8m -- cat /var/lib/k0s/pki/admin.conf > ~/.kube/k0s-in-k8s-config
isaacj-macbook:k0s isaacj$ sed -i '' 's/:6443/:9443/' ~/.kube/k0s-in-k8s-config
isaacj-macbook:k0s isaacj$ kubectl port-forward k0s-master-76d596b48f-68x8m 9443:6443
Forwarding from 127.0.0.1:9443 -> 6443
Forwarding from [::1]:9443 -> 6443
Handling connection for 9443
Handling connection for 9443
Handling connection for 9443
Handling connection for 9443
And listed nodes and namespaces which showed i was connecting, but still no active nodes
$ kubectl get ns
NAME STATUS AGE
default Active 2m28s
kube-node-lease Active 2m30s
kube-public Active 2m30s
kube-system Active 2m30s
$ kubectl get nodes
No resources found
I had one more nutty idea.. Would this work in AKS where I have Azure CNI instead of kubenet?
This cluster is properly namespaced.. So i’ll need to do this in my own NS:
isaacj-macbook:k0s isaacj$ kubectl get pods -n isaacj
NAME READY STATUS RESTARTS AGE
vote-back-azure-vote-1605545802-6dbb7bbd9-mblhs 1/1 Running 0 35h
vote-front-azure-vote-1605545802-6db7445477-82jct 1/1 Running 0 35h
isaacj-macbook:k0s isaacj$ kubectl apply -f k0s-master.yaml -n isaacj
deployment.apps/k0s-master created
service/k0s-master created
and see it running
$ kubectl get pods -n isaacj
NAME READY STATUS RESTARTS AGE
k0s-master-7667c9976f-kch9q 1/1 Running 0 57s
vote-back-azure-vote-1605545802-6dbb7bbd9-mblhs 1/1 Running 0 35h
vote-front-azure-vote-1605545802-6db7445477-82jct 1/1 Running 0 35h
We can snag the kubeconfig:
kubectl exec k0s-master-7667c9976f-kch9q -n isaacj -- cat /var/lib/k0s/pki/admin.conf > ~/.kube/k0s-in-k8s-config-aks
$ sed -i '' 's/:6443/:9443/' ~/.kube/k0s-in-k8s-config-aks
$ kubectl port-forward k0s-master-7667c9976f-kch9q -n isaacj 9443:6443
Forwarding from 127.0.0.1:9443 -> 6443
Forwarding from [::1]:9443 -> 6443
But still no go
$ export KUBECONFIG=/Users/isaacj/.kube/k0s-in-k8s-config-aks
$ kubectl get nodes
No resources found
$ kubectl get pods
No resources found in default namespace.
$ kubectl get ns
NAME STATUS AGE
default Active 2m43s
kube-node-lease Active 2m45s
kube-public Active 2m45s
kube-system Active 2m45s
I’ll clean this up before logging out
$ kubectl delete -f k0s-master.yaml -n isaacj
deployment.apps "k0s-master" deleted
service "k0s-master" deleted
Summary
K0s is a nice offering that I would see as similar to RKE or other on-prem cluster offerings. The idea that it’s as simple as k3s isn’t quite there. I would argue, however, it was easier to set up than Rancher Kubernetes Engine (RKE).
I’m not sure if it was the particular Pi i had, but it was disappointing that the simple shell script installer just wouldn’t fly on ubuntu on my Pi. However, running it manually on my Ubuntu WSL was just fine and it ran in my k8s which was x86 as well.
It’s likely a kubernetes offering I’ll follow and try again in the future, but for now, I’m keeping k3s on my home clusters.
