Published: Jan 30, 2024 by Isaac Johnson

Today is a great day to upgrade our Matrix web clients. I’ll walk through upgrading Matrix Synapse, the backend, and the front ends of Element, Cinny and Fluffychat. Then tackle upgrading Immich, the OS equivelant of Google Photos.

Upgrading Element

Element, which we host at https://element.freshbrewed.science/ is really quite easy.

The deployment YAML literally says to get the latest version with each rotation

  spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: element
    strategy:
      rollingUpdate:
        maxSurge: 25%
        maxUnavailable: 25%
      type: RollingUpdate
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: element
      spec:
        containers:
        - image: vectorim/element-web:latest
          imagePullPolicy: Always

So it is of little suprise that when I check for updates, it shows that we are running the latest

Matrix Synapse, however, was a helm deployment set to a version:

$ helm list | grep matrix
matrix-synapse                          default         1               2023-11-01 18:30:11.957425513 -0500 CDT deployed        matrix-synapse-3.7.10           1.95.1

And just to confirm it’s running ‘1.95.1’ we can look up the image ID from the pod

$ kubectl get pod matrix-synapse-7687454496-c8t8b -o yaml | grep -i image
    image: matrixdotorg/synapse:v1.95.1
    imagePullPolicy: IfNotPresent
    image: docker.io/matrixdotorg/synapse:v1.95.1
    imageID: docker.io/matrixdotorg/synapse@sha256:9b6d64057cf0be41370c87523fd9dc9a0b744389aa67cb0462eca4ef34e0c5ec

While we can synapse, in my current deployment, runs its own PSQL DB:

$ kubectl get pods | grep matrix
matrix-synapse-7687454496-c8t8b                          1/1     Running            0                  50d
matrix-synapse-redis-master-9b74d946-v254s               1/1     Running            0                  63d
matrix-synapse-wellknown-lighttpd-7cc44d98f5-nlkng       1/1     Running            0                  63d
matrix-synapse-postgresql-0                              1/1     Running            0                  63d

I didn’t set any values when I originally launched with helm

$ helm get values matrix-synapse
USER-SUPPLIED VALUES:
serverName: matrix.freshbrewed.science
wellknown:
  enabled: true

If we go to Artifact Hub, under matrix-synapse we can see the latest, as of this writing, was from this past December at chart version 3.7.14 and app version 1.98.0

To upgrade, I just need to ensure I have the same helm repo added and updated

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ helm repo add ananace-charts https://ananace.gitlab.io/charts
"ananace-charts" has been added to your repositories
builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ngrok" chart repository
...Successfully got an update from the "azure-samples" chart repository
...Successfully got an update from the "kube-state-metrics" chart repository
...Successfully got an update from the "confluentinc" chart repository
...Successfully got an update from the "jfelten" chart repository
...Successfully got an update from the "nfs" chart repository
...Successfully got an update from the "actions-runner-controller" chart repository
...Successfully got an update from the "adwerx" chart repository
...Successfully got an update from the "novum-rgi-helm" chart repository
...Successfully got an update from the "makeplane" chart repository
...Successfully got an update from the "openfunction" chart repository
...Successfully got an update from the "zabbix-community" chart repository
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "kiwigrid" chart repository
...Successfully got an update from the "lifen-charts" chart repository
...Successfully got an update from the "elastic" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "kubecost" chart repository
...Successfully got an update from the "sumologic" chart repository
...Successfully got an update from the "openzipkin" chart repository
...Successfully got an update from the "argo-cd" chart repository
...Successfully got an update from the "ananace-charts" chart repository
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "rancher-latest" chart repository
...Successfully got an update from the "crossplane-stable" chart repository
...Unable to get an update from the "epsagon" chart repository (https://helm.epsagon.com):
        Get "https://helm.epsagon.com/index.yaml": dial tcp: lookup helm.epsagon.com on 172.22.64.1:53: server misbehaving
...Successfully got an update from the "newrelic" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "prometheus-community" chart repository
...Unable to get an update from the "myharbor" chart repository (https://harbor.freshbrewed.science/chartrepo/library):
        failed to fetch https://harbor.freshbrewed.science/chartrepo/library/index.yaml : 404 Not Found
...Unable to get an update from the "freshbrewed" chart repository (https://harbor.freshbrewed.science/chartrepo/library):
        failed to fetch https://harbor.freshbrewed.science/chartrepo/library/index.yaml : 404 Not Found
...Successfully got an update from the "akomljen-charts" chart repository
...Successfully got an update from the "opencost" chart repository
...Successfully got an update from the "portainer" chart repository
...Successfully got an update from the "rhcharts" chart repository
...Successfully got an update from the "dapr" chart repository
...Successfully got an update from the "kuma" chart repository
...Successfully got an update from the "sonarqube" chart repository
...Successfully got an update from the "longhorn" chart repository
...Successfully got an update from the "gitea-charts" chart repository
...Successfully got an update from the "btungut" chart repository
...Successfully got an update from the "castai-helm" chart repository
...Successfully got an update from the "open-telemetry" chart repository
...Successfully got an update from the "rook-release" chart repository
...Successfully got an update from the "datadog" chart repository
...Successfully got an update from the "nginx-stable" chart repository
...Successfully got an update from the "signoz" chart repository
...Successfully got an update from the "uptime-kuma" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

I can do a dry-run first (lots of output so I trimmed it)

$ helm upgrade --dry-run matrix-synapse ananace-charts/matrix-synapse --set serverName=matrix.freshbrewed.science --set wellknown.enabled=true
Release "matrix-synapse" has been upgraded. Happy Helming!
NAME: matrix-synapse
LAST DEPLOYED: Thu Jan  4 15:35:50 2024
NAMESPACE: default
STATUS: pending-upgrade
REVISION: 2
HOOKS:
---
# Source: matrix-synapse/templates/signing-key-job.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: matrix-synapse-signingkey-job
  labels:
    helm.sh/chart: matrix-synapse-3.7.14
    app.kubernetes.io/name: matrix-synapse
    app.kubernetes.io/instance: matrix-synapse
    app.kubernetes.io/version: "1.98.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: signingkey-job
    ... snip ...

Then upgrade

$ helm upgrade matrix-synapse ananace-charts/matrix-synapse --set serverName=matrix.freshbrewed.science --set wellknown.enabled=true
Release "matrix-synapse" has been upgraded. Happy Helming!
NAME: matrix-synapse
LAST DEPLOYED: Thu Jan  4 15:37:14 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
NOTES:
** Note, this chart may take a while to finish setup, please be patient **
** Also, remember to disable the signingkey job (signingkey.job.enabled=false) **

Your Synapse install is now starting, you should soon be able to access it on
the following URL(s);
http://matrix.freshbrewed.science

You can create a user in your new Synapse install by running the following
command; (replacing USERNAME and PASSWORD)

    export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=matrix-synapse,app.kubernetes.io/instance=matrix-synapse,app.kubernetes.io/component=synapse" -o jsonpath="{.items[0].metadata.name}")
    kubectl exec --namespace default $POD_NAME -- register_new_matrix_user -c /synapse/config/homeserver.yaml -c /synapse/config/conf.d/secrets.yaml -u USERNAME -p PASSWORD --admin http://localhost:8008

You can also specify --no-admin to create a non-admin user.

I can see after the upgrade, just the two app pods rotated which is good

$ kubectl get pods | grep matrix
matrix-synapse-redis-master-9b74d946-v254s               1/1     Running            0                 63d
matrix-synapse-postgresql-0                              1/1     Running            0                 63d
matrix-synapse-wellknown-lighttpd-6c54b5d8cf-wjxkz       1/1     Running            0                 47m
matrix-synapse-57bf8cc6b4-9s8d5                          1/1     Running            0                 47m

I can login and view rooms so I know it’s working

Secure backups

I decided to reset my keys and save a Security Key backup for Element.

This meant generating a new file

This gave me a new key to save

Now I have confirmation

I can now save it to my AKV for secure storage

$ az keyvault secret set --name element-keys --vault-name asdfsadf --subscription 'Pay-As-You-Go' --description 'Matrix Synapse for element.freshbrewed.science, @isaac:matrix.freshbrewed.science, asdfsadf' --file /mnt/c/Users/isaac/Downloads/security-key.txt

To use it, when I have to do secure operations, I need both the contents of that key and my password.

I also took a moment to backup and save Room keys from the browser

Cinny and FluffyChat

We can see we are already using the “latest” tag for cinny

builder@builder-T100:~$ docker ps
CONTAINER ID   IMAGE                                                            COMMAND                  CREATED        STATUS                  PORTS                                                 NAMES
4437fe5178a0   fireflyiii/core:latest                                           "/usr/local/bin/entr…"   21 hours ago   Up 21 hours (healthy)   0.0.0.0:9098->8080/tcp, :::9098->8080/tcp             fireflyiii-firefly-1
15c752146804   harbor.freshbrewed.science/freshbrewedprivate/fluffychat:0.0.6   "/docker-entrypoint.…"   32 hours ago   Up 32 hours             0.0.0.0:9095->80/tcp, :::9095->80/tcp                 fluffychat
b1ca143ab350   productiveops/dokemon:latest                                     "/dokemon"               2 days ago     Up 2 days               0.0.0.0:9090->9090/tcp, :::9090->9090/tcp             dokemon
9fe031d65142   ghcr.io/cinnyapp/cinny:latest                                    "/docker-entrypoint.…"   2 days ago     Up 2 days               0.0.0.0:8088->80/tcp, :::8088->80/tcp                 cinny

FluffyChat was built locally. So if I want to update that, I’ll need to pull down fresh code and compile.

Before I do anything, I want to send back the fix for the Dockerfile to Fluffychat. I created PR774 for that.

I’ll then push my updated versions to my repo

$ git push --set-upstream origin my-updated-version
Enumerating objects: 14, done.
Counting objects: 100% (14/14), done.
Delta compression using up to 16 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 886 bytes | 886.00 KiB/s, done.
Total 8 (delta 6), reused 0 (delta 0)
remote: Resolving deltas: 100% (6/6), completed with 6 local objects.
remote:
remote: Create a pull request for 'my-updated-version' on GitHub by visiting:
remote:      https://github.com/idjohnson/fluffychat/pull/new/my-updated-version
remote:
To https://github.com/idjohnson/fluffychat.git
 * [new branch]        my-updated-version -> my-updated-version
Branch 'my-updated-version' set up to track remote branch 'my-updated-version' from 'origin'.

Here is a case where I will leave the code in Github, but create the pipeline in Azure DevOps.

To start with, I’ll create a new pipeline in a new “MatrixClients” project and point it to my fork

I’ll have it create a new pipeline

Since this is based on main, I do not expect the build to work as it does not include my fix branch

Indeed, we can see the yq issue on that first build:

Locally, I’ll now update my forked repo, pull down the azure-pipelines branch then merge in my updates and push them back

builder@DESKTOP-QADGF36:~/Workspaces/idjohnson-fluffychat$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
builder@DESKTOP-QADGF36:~/Workspaces/idjohnson-fluffychat$ git pull
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), 571 bytes | 571.00 KiB/s, done.
From https://github.com/idjohnson/fluffychat
 * [new branch]        azure-pipelines -> origin/azure-pipelines
Already up to date.
builder@DESKTOP-QADGF36:~/Workspaces/idjohnson-fluffychat$ git checkout azure-pipelines
Branch 'azure-pipelines' set up to track remote branch 'azure-pipelines' from 'origin'.
Switched to a new branch 'azure-pipelines'
builder@DESKTOP-QADGF36:~/Workspaces/idjohnson-fluffychat$ git merge my-updated-version
Merge made by the 'recursive' strategy.
 Dockerfile                 |  9 ++++++++-
 config.json                | 10 ++++++++++
 lib/config/app_config.dart |  4 ++--
 web/manifest.json          |  3 ++-
 4 files changed, 22 insertions(+), 4 deletions(-)
 create mode 100644 config.json
builder@DESKTOP-QADGF36:~/Workspaces/idjohnson-fluffychat$ git push
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 16 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 335 bytes | 335.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/idjohnson/fluffychat.git
   fde80437..c13df22f  azure-pipelines -> azure-pipelines

I can already see my updates being applied in the triggered build

And it completed just fine. As with my on-prem builds, this one takes some time. As we can see below, it took just over 4 minutes for the AzDO agent to build the container.

I can now add a Dockerhub service connection to the project

And use that in an updated pipeline for buildAndPush

trigger:
- main

resources:
- repo: self

variables:
  tag: '$(Build.BuildId)'

stages:
- stage: Build
  displayName: Build image
  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: ubuntu-latest
    steps:
    - task: Docker@2
      displayName: Build an image
      inputs:
        containerRegistry: 'dockerhub'
        repository: 'idjohnson/fluffychat'
        command: 'buildAndPush'
        Dockerfile: '**/Dockerfile'
        tags: |
          $(tag)
          latest

I forgot my PR would automatically kick a build so I had already manually fired one off when PR1 enqueued build 4

The build now showed two tags made

Which now show up in Dockerhub under idjohnson/fluffychat

I can verify the last version running on my Dockerhost

builder@builder-T100:~$ docker ps
CONTAINER ID   IMAGE                                                            COMMAND                  CREATED        STATUS                  PORTS                                                 NAMES
15c752146804   harbor.freshbrewed.science/freshbrewedprivate/fluffychat:0.0.6   "/docker-entrypoint.…"   47 hours ago   Up 47 hours             0.0.0.0:9095->80/tcp, :::9095->80/tcp                 fluffychat

Now I just need to pull my new image, stop the old, remove it and lastly start with the pulled new image

builder@builder-T100:~$ docker pull idjohnson/fluffychat:20366
20366: Pulling from idjohnson/fluffychat
c926b61bad3b: Already exists
fed54a1dc458: Already exists
d4735778d47c: Already exists
8695c106552e: Already exists
dffa16519b51: Already exists
9e50a0e580b1: Already exists
5ddd532e9cec: Already exists
fe117667dcd0: Already exists
ad627240d700: Pull complete
f7183d416424: Pull complete
Digest: sha256:baa2cc42d4d120bc3b8986f1453cc35a67835ce38ae6b961a1c45156d0215edf
Status: Downloaded newer image for idjohnson/fluffychat:20366
docker.io/idjohnson/fluffychat:20366
builder@builder-T100:~$ docker stop fluffychat
fluffychat
builder@builder-T100:~$ docker rm fluffychat
fluffychat
builder@builder-T100:~$ sudo docker run -p 9095:80 --restart unless-stopped --name fluffychat -d idjohnson/fluffychat:20366
a5f62be7736c084bdf7e578bbbe677116385059530e83dba224247491319f617
builder@builder-T100:~$

Since Kubernetes fronts the traffic, nothing has changed from the front-end URL

Upgrading Immich

I have known for some time that Immich is out of date

I can see the version when I look at helm

$ helm list -n immich
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
immichprod      immich          3               2023-10-30 07:29:19.806616713 -0500 CDT deployed        immich-0.1.3    v1.79.1

Luckily, I still have the chart locally (which was based on theirs)

builder@DESKTOP-QADGF36:~/Workspaces/immich-charts$ ls
CONTRIBUTING.md  LICENSE  README.md  Taskfile.yaml  charts

However, if I hadn’t, knowing it’s OCI end-point, I could just as easily pull down a fresh copy

builder@DESKTOP-QADGF36:~/Workspaces$ mkdir new-immich-charts
builder@DESKTOP-QADGF36:~/Workspaces$ cd new-immich-charts/
builder@DESKTOP-QADGF36:~/Workspaces/new-immich-charts$ export HELM_EXPERIMENTAL_OCI=1
builder@DESKTOP-QADGF36:~/Workspaces/new-immich-charts$ helm pull oci://harbor.freshbrewed.science/library/immich --version 0.1.3
Pulled: harbor.freshbrewed.science/library/immich:0.1.3
Digest: sha256:6085d5763fefac3ea18706a3fd1be3f5ffeea40f3a2672271b91a536eefde429

And expand locally

builder@DESKTOP-QADGF36:~/Workspaces/new-immich-charts$ ls
immich-0.1.3.tgz
builder@DESKTOP-QADGF36:~/Workspaces/new-immich-charts$ tar -xzf immich-0.1.3.tgz
builder@DESKTOP-QADGF36:~/Workspaces/new-immich-charts$ ls
immich  immich-0.1.3.tgz
builder@DESKTOP-QADGF36:~/Workspaces/new-immich-charts$ ls immich
Chart.lock  Chart.yaml  charts  ci  templates  values.yaml

I’ll want to fetch my installs values (such as Database password, storageclass and PVC)

$ helm get values immichprod -n immich -o yaml > myvalues.yaml

I have two ways to update the tag. I could use --set image.tag=v1.93.3 or set it in the values:

image:
  tag: v1.79.1

One issue I see is that while the app says the latest is “v1.93.3”, I only see as late as v1.87.0 in Github, which is marked as “release”.

This is verified by double checking the pod image and then trying to pull from ghcr.io

$ kubectl get pods -n immich immichprod-web-dd4b87954-g2n56 -o yaml | grep -i image:
    image: ghcr.io/immich-app/immich-web:v1.79.1
    image: ghcr.io/immich-app/immich-web:v1.79.1

$ docker pull ghcr.io/immich-app/immich-web:v1.93.3
Error response from daemon: manifest unknown

$ docker pull ghcr.io/immich-app/immich-web:v1.87.0
v1.87.0: Pulling from immich-app/immich-web
96526aa774ef: Already exists
8b8b60d56fb8: Pull complete
97f8dfa93eef: Pull complete
aaa59b7b85b6: Pull complete
5577c0df3ca2: Pull complete
8df1e3ab2671: Pull complete
4f4fb700ef54: Pull complete
cd410038ef59: Pull complete
c0dc99d14349: Pull complete
78ddfb8461b3: Pull complete
c39fe12ca550: Pull complete
Digest: sha256:2bef28adbcc60a2ee5dee8cafe109e3d5c6b7bca88d90acdd3eec376200a6d6e
Status: Downloaded newer image for ghcr.io/immich-app/immich-web:v1.87.0
ghcr.io/immich-app/immich-web:v1.87.0

Thus, despite the app thinking the latest is “v1.93.3” we will just upgrade from 1.79.1 to 1.87.0 which is still a big jump.

I had one more fear to assuage - does it need a DB migration? I checked the official charts and see no indication in the templates folder that there is a db migration kubernetes job. If there is one, it must be baked into the main container.

I decided to just update the values file:

builder@DESKTOP-QADGF36:~/Workspaces/new-immich-charts$ helm get values immichprod -n immich -o yaml > myvalues.yaml
builder@DESKTOP-QADGF36:~/Workspaces/new-immich-charts$ helm get values immichprod -n immich -o yaml > myvalues.yaml.bak
builder@DESKTOP-QADGF36:~/Workspaces/new-immich-charts$ vi myvalues.yaml
builder@DESKTOP-QADGF36:~/Workspaces/new-immich-charts$ diff myvalues.yaml myvalues.yaml.bak
1,2d0
< image:
<   tag: v1.87.0

Here we can see the helm upgrade command in action:

$ helm upgrade immichprod -f myvalues.yaml -n immich ./immich/
Release "immichprod" has been upgraded. Happy Helming!
NAME: immichprod
LAST DEPLOYED: Tue Jan 23 07:29:18 2024
NAMESPACE: immich
STATUS: deployed
REVISION: 4
TEST SUITE: None

I can now login and see the reason they don’t have new images. There is a notice in the lower left about a breaking change

The key changes beyond 1.88 is that they removed immich-web and immich-proxy, reducing the total containers from 8 to 6.

Since the charts are nested, with the real common chart actually living in https://bjw-s.github.io/helm-charts/docs/, I’m going to stay here at 1.87 for a bit before upgrading to newer.

Summary

That was a lot less painless that I thought it would be. It just involved some charts, docker containers and a few small backups. I did opt to punt on the breaking change release of Immich. I want to be newer but not neccessarily the newest.