Uptime Kuma and RunDeck Revisit

Published: May 9, 2023 by Isaac Johnson

Today we’ll look at Uptime Kuma and Rundeck again. Unlike the last time, we’ll focus on running these via Docker on a Linux host (instead of Kubernetes). We’ll handle exposing Uptime via an “External IP” route in Kubernetes (which will route external TLS traffic through Kubernetes onto a private VM in my network). We’ll setup and configure Discord notifications and PagerDuty service integrations. Then lastly, we’ll setup a simple docker based RunDeck service to act as internal webhook runner.

Uptime Kuma (docker)

Last time I focused on exposing Uptime Kuma via Helm and Kubernetes.

How might we do it with just a local Linux VM or machine?

First, check if you have Docker

builder@builder-T100:~$ docker ps

Command 'docker' not found, but can be installed with:

sudo snap install docker     # version 20.10.17, or
sudo apt  install docker.io  # version 20.10.21-0ubuntu1~20.04.2

See 'snap info docker' for additional versions.

Clearly, I need to add Docker first

Remove any old versions

builder@builder-T100:~$ sudo apt-get remove docker docker-engine docker.io containerd runc
[sudo] password for builder:
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package docker-engine

Install the package index required libraries

$ sudo apt-get update && sudo apt-get install ca-certificates curl gnupg   

Get the official docker gpg key

builder@builder-T100:~$ sudo install -m 0755 -d /etc/apt/keyrings
builder@builder-T100:~$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
builder@builder-T100:~$ sudo chmod a+r /etc/apt/keyrings/docker.gpg

Add the Docker repo

echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Then we can install Docker

$ sudo apt-get update && sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Get:2 https://download.docker.com/linux/ubuntu focal InRelease [57.7 kB]
Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:5 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease
... snip ...
0 upgraded, 11 newly installed, 0 to remove and 6 not upgraded.
Need to get 115 MB of archives.
After this operation, 434 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 pigz amd64 2.4-1 [57.4 kB]
... snip ...
Setting up gir1.2-javascriptcoregtk-4.0:amd64 (2.38.6-0ubuntu0.20.04.1) ...
Setting up libwebkit2gtk-4.0-37:amd64 (2.38.6-0ubuntu0.20.04.1) ...
Setting up gir1.2-webkit2-4.0:amd64 (2.38.6-0ubuntu0.20.04.1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...

Unless we want to use sudo everytime, let’s add ourselves to the docker group

builder@builder-T100:~$ sudo groupadd docker
groupadd: group 'docker' already exists
builder@builder-T100:~$ sudo usermod -aG docker builder

I’ll logout and login to see I can run docker commands

builder@builder-T100:~$ exit
logout
Connection to 192.168.1.100 closed.
builder@DESKTOP-QADGF36:~$ ssh builder@192.168.1.100
Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.15.0-71-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

Expanded Security Maintenance for Applications is not enabled.

0 updates can be applied immediately.

9 additional security updates can be applied with ESM Apps.
Learn more about enabling ESM Apps service at https://ubuntu.com/esm

New release '22.04.2 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Your Hardware Enablement Stack (HWE) is supported until April 2025.
Last login: Tue May  9 06:27:47 2023 from 192.168.1.160
builder@builder-T100:~$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Now let’s create a Docker volume for Uptime Kuma

builder@builder-T100:~$ docker volume create uptime-kuma
uptime-kuma

Lastly, launch a fresh Docker instance on port 3001

builder@builder-T100:~$ docker run -d --restart=always -p 3001:3001 -v uptime-kuma:/app/data --name uptime-kuma louislam/uptime-kuma:1
Unable to find image 'louislam/uptime-kuma:1' locally
1: Pulling from louislam/uptime-kuma
9fbefa337077: Pull complete
c119feee8fd1: Pull complete
b3c823584bd9: Pull complete
460e41fb6fee: Pull complete
6834ffbb754a: Pull complete
784b2d9bfd2e: Pull complete
edeeb5ae0fb9: Pull complete
d6f2928f0ccc: Pull complete
4f4fb700ef54: Pull complete
08a0647bce64: Pull complete
Digest: sha256:1630eb7859c5825a1bc3fcbea9467ab3c9c2ef0d98a9f5f0ab0aec9791c027e8
Status: Downloaded newer image for louislam/uptime-kuma:1
0d276b3055232a1bec04f141501039c1809522ea0ff94d5de03643e19c91b152

We can now create an account

/content/images/2023/05/uptime-01.png

We now have a functional local instance

/content/images/2023/05/uptime-02.png

However, what if I want to reach it externally? Do I need to add a helm chart?

External routing via Kubernetes Ingress

Actually, this is easier than it may seem

Some time ago I had already setup an A record to the ingress external IP

$ cat r53-uptimekuma.json
{
  "Comment": "CREATE uptime fb.s A record ",
  "Changes": [
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "uptime.freshbrewed.science",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "73.242.50.46"
          }
        ]
      }
    }
  ]
}

My cluster is still listening, even though no such service is set up

/content/images/2023/05/uptime-03.png

The first bit we need is an Ingress setup with TLS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  labels:
    app.kubernetes.io/instance: uptimeingress
  name: uptimeingress
spec:
  rules:
  - host: uptime.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: uptime-external-ip
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - uptime.freshbrewed.science
    secretName: uptime-tls

That assumes some service of “uptime-external-ip” is exposed on 80.

We then define that service

apiVersion: v1
kind: Service
metadata:
  name: uptime-external-ip
spec:
  ports:
  - name: utapp
    port: 80
    protocol: TCP
    targetPort: 3001
  clusterIP: None
  type: ClusterIP

Unlike regular k8s services, there is no label to send traffic to some pods… This looks sort of strange.

We lastly create an endpoint that would tie to this

apiVersion: v1
kind: Endpoints
metadata:
  name: uptime-external-ip
subsets:
- addresses:
  - ip: 192.168.1.100
  ports:
  - name: utapp
    port: 3001
    protocol: TCP

We can now see the whole YAML file and apply it

$ cat setup-fwd-uptime.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  labels:
    app.kubernetes.io/instance: uptimeingress
  name: uptimeingress
spec:
  rules:
  - host: uptime.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: uptime-external-ip
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - uptime.freshbrewed.science
    secretName: uptime-tls
---
apiVersion: v1
kind: Service
metadata:
  name: uptime-external-ip
spec:
  ports:
  - name: utapp
    port: 80
    protocol: TCP
    targetPort: 3001
  clusterIP: None
  type: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:
  name: uptime-external-ip
subsets:
- addresses:
  - ip: 192.168.1.100
  ports:
  - name: utapp
    port: 3001
    protocol: TCP

$ kubectl apply -f setup-fwd-uptime.yaml
ingress.networking.k8s.io/uptimeingress created
service/uptime-external-ip created
endpoints/uptime-external-ip created

This got me closer

/content/images/2023/05/uptime-04.png

I need to tell NGinx this is a Websockets service.

This is just an annotation. I moved the Ingress YAML to a file so i could easily delete/update/add back

$ cat uptime-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  labels:
    app.kubernetes.io/instance: uptimeingress
  name: uptimeingress
spec:
  rules:
  - host: uptime.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: uptime-external-ip
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - uptime.freshbrewed.science
    secretName: uptime-tls
$ kubectl delete -f uptime-ingress.yaml
ingress.networking.k8s.io "uptimeingress" deleted
$ cat uptime-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.org/websocket-services: uptime-external-ip
  labels:
    app.kubernetes.io/instance: uptimeingress
  name: uptimeingress
spec:
  rules:
  - host: uptime.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: uptime-external-ip
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - uptime.freshbrewed.science
    secretName: uptime-tls
$ kubectl apply -f uptime-ingress.yaml
ingress.networking.k8s.io/uptimeingress created

I now have a valid HTTPS ingress

/content/images/2023/05/uptime-05.png

Adding Monitors

Let’s start with just monitoring some of my hosts

/content/images/2023/05/uptime-06.png

Since that worked

/content/images/2023/05/uptime-07.png

I can add the rest of the cluster

/content/images/2023/05/uptime-08.png

You can see I added a custom tag for cluster. just click “+” on the Tags

/content/images/2023/05/uptime-09.png

You can give a name, colour and value

/content/images/2023/05/uptime-10.png

Or pick from existing Tags

/content/images/2023/05/uptime-11.png

When I got done, I realized visually I didn’t like the tags co-mingled. I want my primary cluster different than my secondary/test cluster

/content/images/2023/05/uptime-12.png

That looks better

Status page

I could create a status page just for cluster compute.

I click new status page

/content/images/2023/05/uptime-13.png

Give it a name

/content/images/2023/05/uptime-14.png

I’ll give it some details and save

/content/images/2023/05/uptime-15.png

I now have a nice dashboard status page to leave up in a window

/content/images/2023/05/uptime-16.png

Alerts

Since I’ve moved off Teams to Discord, adding a new alert should be easy

I’m going to create a Channel just for alerts

/content/images/2023/05/uptime-17.png

I’ll give it a name

/content/images/2023/05/uptime-18.png

I can click the gear icon by it, and go to Integrations to “Create Webhook”

/content/images/2023/05/uptime-19.png

It automatically creates a new one named “Captain hook”

/content/images/2023/05/uptime-20.png

I’ll rename it and click “Copy Webhook URL” (and click “save changes” at the bottom)

/content/images/2023/05/uptime-21.png

I’ll paste that into Uptime Kuma and set some fields.

/content/images/2023/05/uptime-22.png

Clicking Test shows up in Discord

/content/images/2023/05/uptime-23.png

When I save it, with the settings to enable on all monitors, it will go and add to all the Monitors I setup thus far.

We can see that by clicking Edit and checking the right-hand side

/content/images/2023/05/uptime-24.png

PagerDuty

We can go two ways on this. The first is to create a service with an integration key, the other is to use PagerDuty email.

Let’s start with the first.

I’ll create a new service for UptimeKuma alerts

/content/images/2023/05/uptime-27.png

I’ll give it a name and description

/content/images/2023/05/uptime-28.png

For now I’ll keep my default escalation policy

/content/images/2023/05/uptime-29.png

I’ll need the “Events API V2” for the Key, but I’ll also add Email while I’m at it

/content/images/2023/05/uptime-30.png

With the service created, I now have an Integration Key and URL

/content/images/2023/05/uptime-31.png

I can put those details into UptimeKuma and click Test

/content/images/2023/05/uptime-32.png

Which shows it hit PagerDuty

/content/images/2023/05/uptime-33.png

And I can see it worked in PagerDuty as well

/content/images/2023/05/uptime-34.png

I’ll resolve the alert so we don’t stay in a bad state

/content/images/2023/05/uptime-35.png

And we can see the Incident that was created and now resolved in PD as well

/content/images/2023/05/uptime-36.png

My last step is to enable this just on the Primary Cluster nodes. I specifically do not want PD to be alerted on my test cluster as I tend to cycle it often.

I’ll zip through the nodes in ‘mac77’ the primary and enable Pagerduty

/content/images/2023/05/uptime-37.png

We can now see I get PD alerts just as I would for anything else - in fact I realized I had to mask some real work-related alerts that were co-mingled in the list

/content/images/2023/05/uptime-38.png

A few more settings

You may need to adjust the timezone or set the Base URL. You can find those settings in the “General” field

/content/images/2023/05/uptime-25.png

RunDeck

Let’s follow the same pattern with Rundeck

I’ll create a volume on the Utility Linux box, then launch Rundeck in daemon mode on port 4440

builder@builder-T100:~$ docker volume create rundeck
rundeck
builder@builder-T100:~$ docker run -d --restart=always -p 4440:4440 -v rundeck:/home/rundeck/server/data --name rundeck rundeck/rundeck:3.4.6
Unable to find image 'rundeck/rundeck:3.4.6' locally
3.4.6: Pulling from rundeck/rundeck
b5f4c6494e1c: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:9fa107d48e952ad743c6d2f90b8ee34ce66c73a6528c1c0093ef9300c4389fab
Status: Downloaded newer image for rundeck/rundeck:3.4.6
0c89872282f564ac1c5661f3b23a96ffd1f94419aa33db736db6f6cc42475842

While I can login with admin/admin

/content/images/2023/05/uptime-39.png

RunDeck kept redirecting to 127.0.0.1 because that is the default config

/content/images/2023/05/uptime-40.png

We can review all the env vars we can set including some SMTP ones for email. But the one that we need is the RUNDECK_GRAILS_URL setting

$ docker run -d --restart=always -p 4440:4440 -v rundeck:/home/rundeck/server/data -e RUNDECK_GRAILS_URL=http://192.168.1.100:4440 --name rundeck2 rundeck/rundeck:3.4.6
767c9377f238147fa85e1f1c7d194e22fd1eb16b1b1988c0537711da3e41ef7a

Now when I login

/content/images/2023/05/uptime-41.png

I get properly redirected.

I’ll create a new project

/content/images/2023/05/uptime-43.png

Then I can create webhooks

/content/images/2023/05/uptime-44.png

Give the webhook a name

/content/images/2023/05/uptime-45.png

For handler, I’ll tell it to just log an event

/content/images/2023/05/uptime-46.png

I’ll save and get a URL (http://192.168.1.100:4440/api/40/webhook/tx9upk9nrcQVARAUbIOS0t8OCCM484YD#TestHook)

I’ll save to RunDeck and Test

/content/images/2023/05/uptime-47.png

and if I do docker logs on the rundeck2 container (docker logs rundeck2) I can see the log was written

[2023-05-09T15:41:31,031] INFO  web.requests "GET /plugin/detail/WebhookEvent/log-webhook-event" 192.168.1.160 http admin form 14 OnPremWebhooks [application/json;charset=utf-8] (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36)
[2023-05-09T15:42:27,053] INFO  web.requests "POST /api/40/webhook/tx9upk9nrcQVARAUbIOS0t8OCCM484YD" 172.17.0.1 http admin token 9 ? [text/plain;charset=utf-8] (axios/0.27.2)
[2023-05-09T15:42:27,054] INFO  api.requests "POST /api/40/webhook/tx9upk9nrcQVARAUbIOS0t8OCCM484YD" 172.17.0.1 http admin token 10  (axios/0.27.2)

Let’s do something slightly more interesting…

Jobs

Let’s create a new Job under Jobs

/content/images/2023/05/uptime-48.png

I’ll give it a name

/content/images/2023/05/uptime-49.png

For a workflow, I’ll just have it echo a line

/content/images/2023/05/uptime-50.png

There are lots of other settings, including notifications, but for now, let’s just save it with defaults

/content/images/2023/05/uptime-51.png

Back in our test webhook, let’s change from Log to Run Job

/content/images/2023/05/uptime-52.png

I’ll set it to TestJob and save

/content/images/2023/05/uptime-53.png

Now when I test from Uptime, I can see it fired

/content/images/2023/05/uptime-54.png

At this time, I really don’t need to perform any actions. However, this could be a place I trigger a lightswitch with pykasa, or update a webpage - pretty much any arbitrary workflow I might need to happen in response to a monitor.

Since, by its nature, I cannot reset the admin password without email being setup, I won’t expose this via external ingress. But it shows a straightforward webhook provider we can use in conjunction with UptimeKuma for automations.

Some notes

What is nice, compared to my last setup, is that I have this setup on a local machine tied to a battery backup which lasts for typical summer blips.

This means, IF my Kubernetes cluster has issues, this VM will still monitor and update discord. Moreover, I can hit this URL using the external TLS ingress but also, to minimize vectors, using the internal insecure IP as well (take Kubernetes out of the mix)

/content/images/2023/05/uptime-26.png

Fun mini PC

I got a Vine review item to play with. It was ~$130 I believe. They did the whole typical review-and-switch and now the listing points to a much more expensive model.

That said, this is all running quite well on a SkyBarium Mini Desktop N100.

/content/images/2023/05/20230509_073446.jpg

Not a plug or referral link - just thought I would mention it.

Summary

Today we setup RunDeck and Uptime Kuma on a small i3-based Linux VM running Ubuntu and Docker. We setup external Ingress with TLS to Uptime Kuma and monitors for two different Kubernetes clusters. We configured a new PagerDuty service and set that up to trigger alerts on the primary cluster nodes. We configured alerts for all nodes to go to a new Discord channel. Lastly, we setup an internal RunDeck instance to act as a webhook provider we can use internally.

Overall, this was fun revisit of some well-loved Open Source tools. I hope you got some value and can use some of these in your own projects.

UptimeKuma Kubernetes Monitoring

Have something to add? Feedback? You can use the feedback form

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes