Published: Nov 4, 2022 by Isaac Johnson

Having to replace some cheap smart plugs that all failed, I figured it might be worth getting some that we could program.

I picked up some Kasa Smart Plug HS103P4 from Amazon made by TP-Link. They were just about US$5 more than the cheapest ones that had previously died after a few months.

While TPLink doesn’t exactly expose a public REST API, developers deciphered it and the comapany seems to have no intention to stop them. Open-source developers exposed it as a Python project: Python-Kasa which makes it easy to turn off and on bulbs with a simple python script call.

Today we’ll setup a local environment to build python-kasa (uses “Poetry”) then work to containerize it. Lastly, we’ll add it as containerized microservice to an on-prem Kubernetes cluster, expose an HTTPS webhook URL and tie it into Datadog for alerting.

I won’t really cover building the little lamp you’ll see, but I’m happy to provide the STL i created for the trailer bulb and it’s enclosure. I wired it up with a 5v plug I chopped from a broken IKEA kids nightlight.

Setup

First, we need to pull down and try the python-kasa project as it stands.

Before we do, we need to make sure we have the Python package manager Poetry

$ curl -sSL https://install.python-poetry.org | python3 -
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/home/builder/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.2.2): Done

Poetry (1.2.2) is installed now. Great!

You can test that everything is set up by executing:

`poetry --version`

Then we can clone the repo

builder@DESKTOP-QADGF36:~/Workspaces$ git clone https://github.com/python-kasa/python-kasa.git
Cloning into 'python-kasa'...
remote: Enumerating objects: 2150, done.
remote: Counting objects: 100% (896/896), done.
remote: Compressing objects: 100% (232/232), done.
remote: Total 2150 (delta 739), reused 754 (delta 659), pack-reused 1254
Receiving objects: 100% (2150/2150), 758.71 KiB | 5.13 MiB/s, done.
Resolving deltas: 100% (1471/1471), done.
builder@DESKTOP-QADGF36:~/Workspaces$ cd python-kasa/

We can now install the Kasa binary

builder@DESKTOP-QADGF36:~/Workspaces/python-kasa$ poetry install
Creating virtualenv python-kasa-ZQLIqOJb-py3.10 in /home/builder/.cache/pypoetry/virtualenvs
Installing dependencies from lock file

Package operations: 41 installs, 0 updates, 0 removals

  • Installing certifi (2021.10.8)
  • Installing charset-normalizer (2.0.12)
  • Installing idna (3.3)
  • Installing pyparsing (3.0.7)
  • Installing urllib3 (1.26.9)
  • Installing attrs (21.4.0): Installing...
  • Installing attrs (21.4.0)
  • Installing distlib (0.3.4)
  • Installing filelock (3.6.0)
  • Installing iniconfig (1.1.1)
  • Installing packaging (21.3)
  • Installing platformdirs (2.5.1)
  • Installing pluggy (1.0.0)
  • Installing py (1.11.0)
  • Installing requests (2.27.1)
  • Installing six (1.16.0)
  • Installing tomli (2.0.1)
  • Installing cfgv (3.3.1): Downloading... 0%
  • Installing coverage (6.3.2): Pending...
  • Installing cfgv (3.3.1)
  • Installing coverage (6.3.2)
  • Installing identify (2.4.12)
  • Installing nodeenv (1.6.0)
  • Installing pytest (7.1.1)
  • Installing pyyaml (6.0)
  • Installing sniffio (1.2.0)
  • Installing termcolor (1.1.0)
  • Installing toml (0.10.2)
  • Installing typing-extensions (4.1.1)
  • Installing virtualenv (20.14.0)
  • Installing zipp (3.8.0)
  • Installing anyio (3.5.0): Installing...
  • Installing asyncclick (8.0.3.2): Pending...
  • Installing codecov (2.1.12): Installing...
  • Installing anyio (3.5.0)
  • Installing asyncclick (8.0.3.2)
  • Installing codecov (2.1.12)
  • Installing importlib-metadata (4.11.3)
  • Installing pre-commit (2.18.1)
  • Installing pydantic (1.9.0)
  • Installing pytest-asyncio (0.18.3)
  • Installing pytest-cov (2.12.1)
  • Installing pytest-mock (3.7.0)
  • Installing pytest-sugar (0.9.4)
  • Installing tox (3.24.5)
  • Installing voluptuous (0.13.0)
  • Installing xdoctest (0.15.10)

Installing the current project: python-kasa (0.5.0)

It installed it into ~/.cache/pypoetry. I exported it into my path so I could call kasa (e.g. $ export PATH=$PATH:/home/builder/.cache/pypoetry/virtualenvs/python-kasa-ZQLIqOJb-py3.10/bin/)

While the binary in WSL didn’t find it

builder@DESKTOP-QADGF36:~/Workspaces/python-kasa$ kasa discover
Discovering devices on 255.255.255.255 for 3 seconds

builder@DESKTOP-QADGF36:~/Workspaces/python-kasa$ kasa
No host name given, trying discovery..
Discovering devices on 255.255.255.255 for 3 seconds

Doing it on Windows did work

C:\Users\isaac\Workspaces\hv-packer>pip install python-kasa
Defaulting to user installation because normal site-packages is not writeable
Collecting python-kasa
  Downloading python_kasa-0.5.0-py3-none-any.whl (131 kB)
     |████████████████████████████████| 131 kB 3.3 MB/s
Collecting importlib-metadata
  Downloading importlib_metadata-5.0.0-py3-none-any.whl (21 kB)
Collecting asyncclick>=8
  Downloading asyncclick-8.1.3.4-py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 6.4 MB/s
Collecting pydantic<2,>=1
  Downloading pydantic-1.10.2-cp39-cp39-win_amd64.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 6.8 MB/s
Collecting anyio
  Downloading anyio-3.6.2-py3-none-any.whl (80 kB)
     |████████████████████████████████| 80 kB ...
Requirement already satisfied: colorama in c:\program files\python39\lib\site-packages (from asyncclick>=8->python-kasa) (0.4.4)
Collecting typing-extensions>=4.1.0
  Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting sniffio>=1.1
  Downloading sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting idna>=2.8
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB ...
Collecting zipp>=0.5
  Downloading zipp-3.9.0-py3-none-any.whl (5.8 kB)
Installing collected packages: zipp, typing-extensions, sniffio, idna, pydantic, importlib-metadata, asyncclick, anyio, python-kasa
  WARNING: The script kasa.exe is installed in 'C:\Users\isaac\AppData\Roaming\Python\Python39\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed anyio-3.6.2 asyncclick-8.1.3.4 idna-3.4 importlib-metadata-5.0.0 pydantic-1.10.2 python-kasa-0.5.0 sniffio-1.3.0 typing-extensions-4.4.0 zipp-3.9.0
WARNING: You are using pip version 21.1.3; however, version 22.3 is available.
You should consider upgrading via the 'c:\program files\python39\python.exe -m pip install --upgrade pip' command.

C:\Users\isaac\Workspaces\hv-packer>C:\Users\isaac\AppData\Roaming\Python\Python39\Scripts\kasa.exe
No host name given, trying discovery..
Discovering devices on 255.255.255.255 for 3 seconds
== Office LED lights - HS103(US) ==
        Host: 192.168.1.245
        Device state: ON

        == Generic information ==
        Time:         2022-10-22 18:52:59 (tz: {'index': 13, 'err_code': 0}
        Hardware:     5.0
        Software:     1.0.3 Build 201015 Rel.142523
        MAC (rssi):   6C:5A:B0:BC:40:23 (-44)
        Location:     {'latitude': 44.9305, 'longitude': -92.9119}

        == Device specific information ==
        LED state: True
        On since: 2022-10-22 08:36:04

        == Modules ==
        + <Module Schedule (schedule) for 192.168.1.245>
        + <Module Usage (schedule) for 192.168.1.245>
        + <Module Antitheft (anti_theft) for 192.168.1.245>
        + <Module Time (time) for 192.168.1.245>
        + <Module Cloud (cnCloud) for 192.168.1.245>

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000024F86863EE0>
Traceback (most recent call last):
  File "c:\program files\python39\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "c:\program files\python39\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "c:\program files\python39\lib\asyncio\base_events.py", line 746, in call_soon
    self._check_closed()
  File "c:\program files\python39\lib\asyncio\base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

I can now turn them off and on locally

builder@DESKTOP-QADGF36:~/Workspaces/python-kasa$ kasa --host 192.168.1.245 off
No --type defined, discovering..
Turning off Office LED lights
builder@DESKTOP-QADGF36:~/Workspaces/python-kasa$ kasa --host 192.168.1.245 on
No --type defined, discovering..
Turning on Office LED lights

I should note, installing this with Python10 on a different Windows host worked fine (no python stack dump at end)

C:\Users\isaac>C:\Users\isaac\AppData\Roaming\Python\Python310\Scripts\kasa.exe
No host name given, trying discovery..
Discovering devices on 255.255.255.255 for 3 seconds
== Office LED lights - HS103(US) ==
        Host: 192.168.1.245
        Device state: OFF

        == Generic information ==
        Time:         2022-10-23 09:47:47 (tz: {'index': 13, 'err_code': 0}
        Hardware:     5.0
        Software:     1.0.3 Build 201015 Rel.142523
        MAC (rssi):   6C:5A:B0:BC:40:23 (-38)
        Location:     {'latitude': 44.9305, 'longitude': -92.9119}

        == Device specific information ==
        LED state: True
        On since: None

        == Modules ==
        + <Module Schedule (schedule) for 192.168.1.245>
        + <Module Usage (schedule) for 192.168.1.245>
        + <Module Antitheft (anti_theft) for 192.168.1.245>
        + <Module Time (time) for 192.168.1.245>
        + <Module Cloud (cnCloud) for 192.168.1.245>

== OfficeLamp - HS103(US) ==
        Host: 192.168.1.24
        Device state: OFF

        == Generic information ==
        Time:         2022-10-23 09:47:36 (tz: {'index': 13, 'err_code': 0}
        Hardware:     5.0
        Software:     1.0.3 Build 201015 Rel.142523
        MAC (rssi):   6C:5A:B0:BC:7D:E5 (-47)
        Location:     {'latitude': 44.9305, 'longitude': -92.9119}

        == Device specific information ==
        LED state: True
        On since: None

        == Modules ==
        + <Module Schedule (schedule) for 192.168.1.24>
        + <Module Usage (schedule) for 192.168.1.24>
        + <Module Antitheft (anti_theft) for 192.168.1.24>
        + <Module Time (time) for 192.168.1.24>
        + <Module Cloud (cnCloud) for 192.168.1.24>

Containerizing it

(If you want to skip how I worked through this, feel free to just look at the forked repo)

First, I’ll get a basic Flask app started

$ cat restApi.py
from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    return "Hello, World!"

Then test it

builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa$ export FLASK_APP=restApi.py
builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa$ export FLASK_ENV=development
builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa$ /home/builder/.local/bin/flask run
 * Serving Flask app 'restApi.py' (lazy loading)
 * Environment: development
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 134-193-920
127.0.0.1 - - [29/Oct/2022 09:10:27] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [29/Oct/2022 09:10:27] "GET /favicon.ico HTTP/1.1" 404 -

Adding virtenv

Cbuilder@DESKTOP-72D2D9T:~/Workspaces/python-kasa$ pip install pipenv
Collecting pipenv
  Downloading pipenv-2022.10.25-py2.py3-none-any.whl (3.2 MB)
     |████████████████████████████████| 3.2 MB 1.7 MB/s
Requirement already satisfied: setuptools>=36.2.1 in /usr/lib/python3/dist-packages (from pipenv) (45.2.0)
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from pipenv) (2019.11.28)
Collecting virtualenv
  Downloading virtualenv-20.16.6-py3-none-any.whl (8.8 MB)
     |████████████████████████████████| 8.8 MB 1.4 MB/s
Collecting platformdirs<3,>=2.4
  Using cached platformdirs-2.5.2-py3-none-any.whl (14 kB)
Collecting distlib<1,>=0.3.6
  Using cached distlib-0.3.6-py2.py3-none-any.whl (468 kB)
Collecting filelock<4,>=3.4.1
  Using cached filelock-3.8.0-py3-none-any.whl (10 kB)
Installing collected packages: virtualenv-clone, platformdirs, distlib, filelock, virtualenv, pipenv
  WARNING: The script virtualenv-clone is installed in '/home/builder/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script virtualenv is installed in '/home/builder/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts pipenv and pipenv-resolver are installed in '/home/builder/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed distlib-0.3.6 filelock-3.8.0 pipenv-2022.10.25 platformdirs-2.5.2 virtualenv-20.16.6 virtualenv-clone-0.5.7
builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa$ mkdir cashman-flask-project && cd cashman-flask-project
builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$ pipenv --three

Command 'pipenv' not found, but can be installed with:

sudo apt install pipenv

builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$ /home/builder/.local/bin/pipenv --three
WARNING: --three is deprecated! pipenv uses python3 by default

Creating a virtualenv for this project...
Pipfile: /home/builder/Workspaces/python-kasa/cashman-flask-project/Pipfile
Using /usr/bin/python3.8 (3.8.10) to create virtualenv...
⠋ Creating virtual environment...created virtual environment CPython3.8.10.final.0-64 in 713ms
  creator Venv(dest=/home/builder/.local/share/virtualenvs/cashman-flask-project-0ejrmkiu, clear=False, no_vcs_ignore=False, global=False, describe=CPython3Posix)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/builder/.local/share/virtualenv)
    added seed packages: pip==22.3, setuptools==65.5.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/builder/.local/share/virtualenvs/cashman-flask-project-0ejrmkiu
Creating a Pipfile for this project...
builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$

add flask back

builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$ /home/builder/.local/bin/pipenv install flask
Installing flask...
Adding flask to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (601e48664aae47c8c903f9f070ec23d766dc916b445c91361ab8090b3c62cf9f)!
Installing dependencies from Pipfile.lock (62cf9f)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

This creates a couple files

builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$ ls
Pipfile  Pipfile.lock
builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
flask = "*"

[dev-packages]

[requires]
python_version = "3.8"
python_full_version = "3.8.10"
builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "601e48664aae47c8c903f9f070ec23d766dc916b445c91361ab8090b3c62cf9f"
        },
        "pipfile-spec": 6,
        "requires": {
          ...snip...

Now I’ll add an index.py with some switches

builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project/restapi$ cat index.py
from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    return "Hello, World!"

@app.route("/on")
def turn_on():
    return "Turn On!"

@app.route("/off")
def turn_off():
    return "Turn Off!"

@app.route("/swap")
def turn_swap():
    return "Swap on and off!"

In the parent directory we will make a bootstrap.sh and change its perms

builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$ cat bootstrap.sh
#!/bin/sh
export FLASK_APP=./restapi/index.py
pipenv run flask --debug run -h 0.0.0.0
builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$ chmod u+x bootstrap.sh

Now to test

builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$ export PATH=$PATH:/home/builder/.local/bin
builder@DESKTOP-72D2D9T:~/Workspaces/python-kasa/cashman-flask-project$ ./bootstrap.sh
 * Serving Flask app './restapi/index.py'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:5000
 * Running on http://172.25.86.186:5000
Press CTRL+C to quit
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 130-147-564

Next, we can push an image up

docker push harbor.freshbrewed.science/freshbrewedprivate/kasarest:1.0.0
The push refers to repository [harbor.freshbrewed.science/freshbrewedprivate/kasarest]
5f6d128f32ff: Pushed
add780bbe862: Pushed
91d2c77a2edc: Pushed
712c133ff3aa: Pushed
ebdb09e1ad9c: Pushed
98310229ec44: Pushed
229486f87ac7: Pushed
ec8f0bde2cdf: Pushed
2fef07314e9a: Pushed
15dfb8e06bed: Pushed
83b7d033bde4: Pushed
3d3e9708cef3: Pushed
1fe0699af9f7: Pushed
156568a71809: Pushed
5fca8a94d542: Pushed
6b183c62e3d7: Pushed
882fd36bfd35: Pushed
d1dec9917839: Pushed
d38adf39e1dd: Pushed
4ed121b04368: Pushed
d9d07d703dd5: Pushed
1.0.0: digest: sha256:ba33e37dbf3d550c2c3bfdbec2d25bb58618e5042dd377dffa87116f3d37be47 size: 4734

And I can now see in Harbor

We can “copy” the pull command

docker pull harbor.freshbrewed.science/freshbrewedprivate/kasarest@sha256:ba33e37dbf3d550c2c3bfdbec2d25bb58618e5042dd377dffa87116f3d37be47

But more exactly we’ll be using this in a basic deployment.

I’ll add an A record

$ cat r53-kasarest.json 
{
    "Comment": "CREATE kasarest fb.s A record ",
    "Changes": [
      {
        "Action": "CREATE",
        "ResourceRecordSet": {
          "Name": "kasarest.freshbrewed.science",
          "Type": "A",
          "TTL": 300,
          "ResourceRecords": [
            {
              "Value": "73.242.50.46"
            }
          ]
        }
      }
    ]
  }

$ aws route53 change-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP --change-batch file://r53-kasarest.json
{
    "ChangeInfo": {
        "Id": "/change/C02863662AUH1QQYVNMQI",
        "Status": "PENDING",
        "SubmittedAt": "2022-10-31T17:25:59.145Z",
        "Comment": "CREATE kasarest fb.s A record "
    }
}

Then create a quick deployment yaml

$ cat kasadep.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kasarest
    app.kubernetes.io/name: kasarest
  name: kasarest-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kasarest
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kasarest
    spec:
      containers:
      - env:
        - name: PORT
          value: "5000"
        image: harbor.freshbrewed.science/freshbrewedprivate/kasarest:1.0.0
        name: kasarest
        ports:
        - containerPort: 5000
          protocol: TCP
      imagePullSecrets:
      - name: myharborreg
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: kasarest
  name: kasarest
spec:
  internalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    app.kubernetes.io/name: kasarest
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  name: kasarest
spec:
  rules:
  - host: kasarest.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: kasarest
            port:
              number: 5000
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - kasarest.freshbrewed.science
    secretName: kasarest-tls
---

Then apply

$ kubectl apply -f kasadep.yaml 
deployment.apps/kasarest-deployment created
service/kasarest created
ingress.networking.k8s.io/kasarest created

Then I can check on it’s progress

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get pods | grep kasa
kasarest-deployment-7f9697c654-x8lrs                     0/1     ContainerCreating   0                 55s
builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl describe pod kasarest-deployment-7f9697c654-x8lrs | tail -n20
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-2nqgm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  30s   default-scheduler  Successfully assigned default/kasarest-deployment-7f9697c654-x8lrs to builder-hp-elitebook-850-g2
  Normal  Pulling    30s   kubelet            Pulling image "harbor.freshbrewed.science/freshbrewedprivate/kasarest:1.0.0"

builder@DESKTOP-QADGF36:~/Workspaces/jekyll-blog$ kubectl get pods | grep kasa
kasarest-deployment-7f9697c654-x8lrs                     1/1     Running   0                 3m1s

Adding to Datadog

In Datadog, we go to Integrations

I’ll go to Webhooks where I already have some old Rundeck ones defined

I’ll add my endpoint using the JSON to pass the form var

The resulting webhook:

Next, I’ll add a new Monitor based on Host

and we can then send an alert to that endpoint

Running a test failed (that is, the light did not change)

After several unsuccessful attempts, I realized the issue was i was assuming GET when the webhook in Datadog was using POST

42.2.104 - - [31/Oct/2022 20:30:14] "GET /off?devip=192.168.1.24 HTTP/1.1" 200 -
42.2.104 - - [31/Oct/2022 20:30:19] "POST /on?devip=192.168.1.24 HTTP/1.1" 405 -
42.2.104 - - [31/Oct/2022 20:30:20] "POST /on?devip=192.168.1.24 HTTP/1.1" 405 -

Adding POST

This was an easy fix. I updated my rest functions to add a post block

import subprocess
from subprocess import Popen, PIPE
from subprocess import check_output
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/")
def hello_world():
    return "use /on /off and /swap with ?devip=IP!"

@app.route("/on", methods=["GET","POST"])
def turn_on():
    if request.method == 'GET':
        devip = request.args.get('devip','192.168.1.245')
        typen = request.args.get('type','plug')
        stdout = check_output(['./some.sh',typen,devip,'on']).decode('utf-8')
        return '''<h1>Turning On {}: {}</h1>'''.format(devip, stdout)
    if request.method == 'POST':
        devip = request.form.get('devip')
        typen = request.form.get('type','plug')
        stdout = check_output(['./some.sh',typen,devip,'on']).decode('utf-8')
        return '''<h1>Turning On {}: {}</h1>'''.format(devip, stdout)

@app.route("/off", methods=["GET","POST"])
def turn_off():
    if request.method == 'GET':
        devip = request.args.get('devip','192.168.1.245')
        typen = request.args.get('type','plug')
        stdout = check_output(['./some.sh',typen,devip,'off']).decode('utf-8')
        return '''<h1>Turning Off {}: {}</h1>'''.format(devip, stdout)
    if request.method == 'POST':
        devip = request.form.get('devip')
        typen = request.form.get('type','plug')
        stdout = check_output(['./some.sh',typen,devip,'off']).decode('utf-8')
        return '''<h1>Turning Off {}: {}</h1>'''.format(devip, stdout)

@app.route("/swap", methods=["GET","POST"])
def turn_swap():
    if request.method == 'GET':
        devip = request.args.get('devip','192.168.1.245')
        typen = request.args.get('type','plug')
        #return '''<h1>Swapping On/Off: {}</h1>'''.format(devip)
        stdout = check_output(['./swap.sh',typen,devip]).decode('utf-8')
        return stdout
    if request.method == 'POST':
        devip = request.form.get('devip','192.168.1.245')
        typen = request.form.get('type','plug')
        stdout = check_output(['./swap.sh',typen,devip]).decode('utf-8')
        return 

Once I built and pushed a new container from the Dockerfile and tagged it 1.1.0

I simply updated the deployment YAML to include it (harbor.freshbrewed.science/freshbrewedprivate/kasarest:1.1.0)

$ cat kasadep.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kasarest
    app.kubernetes.io/name: kasarest
  name: kasarest-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kasarest
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kasarest
        app.kubernetes.io/name: kasarest
    spec:
      containers:
      - env:
        - name: PORT
          value: "5000"
        image: harbor.freshbrewed.science/freshbrewedprivate/kasarest:1.1.0
        name: kasarest
        ports:
        - containerPort: 5000
          protocol: TCP
      imagePullSecrets:
      - name: myharborreg
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: kasarest
  name: kasarest
spec:
  internalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    app.kubernetes.io/name: kasarest
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  name: kasarest
spec:
  rules:
  - host: kasarest.freshbrewed.science
    http:
      paths:
      - backend:
          service:
            name: kasarest
            port:
              number: 5000
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - kasarest.freshbrewed.science
    secretName: kasarest-tls
---

This means I can still turn off my office light with a GET (e.g. have fun and harass me by turning off my desk LED lights with https://kasarest.freshbrewed.science/off but then turn them back on with https://kasarest.freshbrewed.science/on)

Then it worked to trigger:

This is a great start. But perhaps we want to turn it off when our cluster comes back to known good.

Now we can do the inverse to create a Monitor on the down condition being missing

But that might get a little weird as we would have a monitor constantly in error.

Instead, I’ll initially flip the light but also send some reminder emails

Which when tested, not only flips the light, but sends an email

And clicking the link flips the light back off

Builds

I can alert on other things as well. For instance, say I want a bulb to kick in when I build things in Github.

I quickly created a simple “Build” logo that I 3d Printed ( IkeaLampBuild.stl ).

Because I push an event with the github workflow (used for this blog), I’ll be able to use that in an event monitor.

      - name: Build count
        uses: masci/datadog@v1
        with:
          api-key: $
          metrics: |
            - type: "count"
              name: "prfinal.runs.count"
              value: 1.0
              host: $
              tags:
                - "project:$"
                - "branch:$"
      - name: Datadog-Pass
        uses: masci/datadog@v1
        with:
          api-key: $
          events: |
            - title: "Passed building jekyll"
              text: "Branch $ passed build"
              alert_type: "info"
              host: $
              tags:
                - "project:$"

My Datadog tags are a bit funny in that I make the “host” my Repo Owner (idjohnson in github). But it’s still a thing we can query.

I can then create some webhooks we’ll use for a event based monitor.

First, I create the On and Off webhooks

The off for ended:

Then I can create the monitor which will trigger the build is running (alert) and completed (alert recovery) using the following message syntax:

{{#is_alert}}
@webhook-BuildStarted 
{{/is_alert}}

{{#is_recovery}}
@webhook-BuildEnded
{{/is_recovery}}

Which we can test

and see in action:

I did need to tweak the threshold to be “above” 0.5 since the initial (above 1) didn’t trigger on 1.

However, if we pause for a moment, we’ll realize this is rather silly.
I could simply trigger the call inline on my GH Runners.

To test, I muted my DD Monitor

In my PR build (the main build), I updated some steps:

# at the start of both test and final builds
    ...
    steps:
      - name: Check out repository code
        uses: actions/checkout@v2
      - name: Turn on build light
        run: |
          curl -s -o /dev/null https://kasarest.freshbrewed.science/on?devip=192.168.1.3
    ...
# for passing
      ...
      - name: Datadog-Pass
        uses: masci/datadog@v1
        with:
          api-key: $
          events: |
            - title: "Passed building jekyll"
              text: "Branch $ passed build"
              alert_type: "info"
              host: $
              tags:
                - "project:$"
      - name: Turn off build light
        run: |
          curl -s -o /dev/null https://kasarest.freshbrewed.science/off?devip=192.168.1.3

# for failures
  run-if-failed:
    runs-on: ubuntu-latest
    needs: [build_deploy_test, build_deploy_final]
    if: always() && (needs.build_deploy_test.result == 'failure' || needs.build_deploy_final.result == 'failure')
    steps:
      - name: Blink and Leave On build light
        run: |
          curl -s -o /dev/null https://kasarest.freshbrewed.science/off?devip=192.168.1.3 && sleep 5 \
          && curl -s -o /dev/null https://kasarest.freshbrewed.science/on?devip=192.168.1.3 && sleep 5 \
          &&  curl -s -o /dev/null https://kasarest.freshbrewed.science/off?devip=192.168.1.3 && sleep 5 \
          && curl -s -o /dev/null https://kasarest.freshbrewed.science/on?devip=192.168.1.3
      - name: Datadog-Fail
      ....

Here we can see it in action:

Summary

We tackled using Python-Kasa to trigger TP-Link smart plugs. We then containerized it with a Flask app and published it to Kubernetes. Lastly, we tied it in as a Webhook (and POST handlers) and used it to trigger a physical light should our Cluster have issues.

To demonstrate other kinds of Monitors, I setup an Event Monitor on a simple metric I push in my Github Workflows. This allowed me to trigger a switch tied to a lightbulb with the word “build” when a build was running (albeit with a slight delay).

There is an obvious issue, if you think about it - I run this containerized service ON the cluster that might be down. If it’s just a Datadog agent issue or something simple, we might still be fine. But if our cluster is out (such as the master going down), then we clearly cannot trigger the bulb.

This might mean we need to move this function to a different cluster or host it in some other way, such as an internal pi or Linux host. We could look into using the TPLink cloud as we can see in guides here and here. The only issue with that is it is easy for a vendor to just deprecate their unofficial cloud API or end support for Apps. My plugs, unless I firmware update, should be accessible on port 80 and UDP 20002 (discovery). One more crazy approach that comes to mind is to expose a port on my external IP and forward traffic directly to the plug. Then I could run the function in the public cloud. However, to expose each bulb would be cumbersome and likely not a scalable solution.