OpenCost

Published: May 17, 2023 by Isaac Johnson

A while back we looked at Kubecost as a good Open-Source tool for monitoring costs in Kubernetes. In the last year the OS version has forked into OpenCost and has had plenty of changes.

Today we’ll set up OpenCost and check out a couple on-prem clusters.

Setup

Since I’ll start by using a demo cluster, I’ll add Prometheus for Metrics collection which is a requirement for OpenCost

$ helm install my-prometheus --repo https://prometheus-community.github.io/helm-charts prometheus --namespace prometheus --create-namespace --set pushgateway.enabled=false --set alertmanager.enabled=false -f https://raw.githubusercontent.com/opencost/opencost/develop/kubernetes/prometheus/extraScrapeConfigs.yaml
NAME: my-prometheus
LAST DEPLOYED: Wed May 17 08:27:51 2023
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
my-prometheus-server.prometheus.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9090


#################################################################################
######   WARNING: Pod Security Policy has been disabled by default since    #####
######            it deprecated after k8s 1.25+. use                        #####
######            (index .Values "prometheus-node-exporter" "rbac"          #####
###### .          "pspEnabled") with (index .Values                         #####
######            "prometheus-node-exporter" "rbac" "pspAnnotations")       #####
######            in case you still need it.                                #####
#################################################################################


The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
my-prometheus-prometheus-pushgateway.prometheus.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus-pushgateway,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prometheus port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/

Once I see it running

$ kubectl get pods -n prometheus
NAME                                                    READY   STATUS              RESTARTS   AGE
my-prometheus-server-597c87ccd6-bjsbl                   0/2     Pending             0          17s
my-prometheus-prometheus-pushgateway-65857c98fb-cl97x   0/1     ContainerCreating   0          17s
my-prometheus-prometheus-node-exporter-87kkr            0/1     ContainerCreating   0          17s
my-prometheus-prometheus-node-exporter-tbqjh            1/1     Running             0          17s
my-prometheus-prometheus-node-exporter-c2gk2            1/1     Running             0          17s
my-prometheus-kube-state-metrics-65dbf85f86-xgrtr       0/1     Running             0          17s
$ kubectl get pods -n prometheus
NAME                                                    READY   STATUS    RESTARTS   AGE
my-prometheus-prometheus-node-exporter-tbqjh            1/1     Running   0          2m5s
my-prometheus-prometheus-node-exporter-c2gk2            1/1     Running   0          2m5s
my-prometheus-kube-state-metrics-65dbf85f86-xgrtr       1/1     Running   0          2m5s
my-prometheus-prometheus-node-exporter-87kkr            1/1     Running   0          2m5s
my-prometheus-prometheus-pushgateway-65857c98fb-cl97x   1/1     Running   0          2m5s
my-prometheus-server-597c87ccd6-bjsbl                   2/2     Running   0          2m5s

I can now install OpenCost

$ kubectl apply --namespace opencost -f https://raw.githubusercontent.com/opencost/opencost/develop/kubernetes/op
encost.yaml
namespace/opencost created
serviceaccount/opencost created
clusterrole.rbac.authorization.k8s.io/opencost created
clusterrolebinding.rbac.authorization.k8s.io/opencost created
deployment.apps/opencost created
service/opencost created

$ kubectl get pods -n opencost
NAME                        READY   STATUS    RESTARTS   AGE
opencost-84c884b5f5-zwkvm   2/2     Running   0          24s

OpenCost will create a service we can use to access the UI

$ kubectl port-forward --namespace opencost service/opencost 9003 9090
Forwarding from 127.0.0.1:9003 -> 9003
Forwarding from [::1]:9003 -> 9003
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

/content/images/2023/05/opencost-01.png

We can view things by Node

/content/images/2023/05/opencost-02.png

which shows the primary node (anna-macbook air) is doing most of the heavy lifting

We can look at data by type of workload (Controller Kind).

/content/images/2023/05/opencost-03.png

and Pod

/content/images/2023/05/opencost-04.png

The other options are Namespace

/content/images/2023/05/opencost-05.png

and service

/content/images/2023/05/opencost-06.png

If I click the “Download CSV” icon, I can get a wholly uninteresting CSV of the data

/content/images/2023/05/opencost-07.png

Prod cluster

I’ll bring down the Github file locally and change to use the Prom server already running for Kubecost

            - name: PROMETHEUS_SERVER_ENDPOINT
              value: "http://kubecost-prometheus-server.kubecost.svc"

I’m not certain it will work as the Job in that Prom server does not have the identical extraconfigs that OpenCost desires

extraScrapeConfigs: |
  - job_name: opencost
    honor_labels: true
    scrape_interval: 1m
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    dns_sd_configs:
    - names:
      - opencost.opencost
      type: 'A'
      port: 9003

though it does expose the /metrics endpoint with similar settings

$ kubectl get cm kubecost-prometheus-server -n kubecost -o yaml | grep -C 10 9003
    - job_name: kubecost
      honor_labels: true
      scrape_interval: 1m
      scrape_timeout: 60s
      metrics_path: /metrics
      scheme: http
      dns_sd_configs:
      - names:
        - kubecost-cost-analyzer
        type: 'A'
        port: 9003
    - job_name: kubecost-networking
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
      # Scrape only the the targets matching the following metadata
        - source_labels: [__meta_kubernetes_pod_label_app]
          action: keep
          regex:  kubecost-network-costs
  recording_rules.yml: |
    {}

Suprisingly, this worked just dandy!

I port-forwarded

$ kubectl port-forward --namespace opencost service/opencost 9003 9090
Forwarding from 127.0.0.1:9003 -> 9003
Forwarding from [::1]:9003 -> 9003
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090

And could see multiple days

/content/images/2023/05/opencost-08.png

So on a weekly basis, if I looked at “Total Cost”, i could see the bigger workloads by service being Datadog and Sonarqube

/content/images/2023/05/opencost-09.png

Let’s ask for a big report, the last 90 days by Pod.

This can only show as us far back as the Prometheus server was running (which was about 2 weeks)

/content/images/2023/05/opencost-10.png

If I dump that to Excel, I have the ability now to do some fun things like charts

/content/images/2023/05/opencost-11.png

An example might be to create a 2 week report on service

/content/images/2023/05/opencost-12.png

which would let me create a chart on, say, CPU

/content/images/2023/05/opencost-13.png

or total cost in a two week period

/content/images/2023/05/opencost-14.png

I was surprised to see when looking on that period that it was not my primary node doing most of the work, rather the derpy old EliteBook providing the most value

/content/images/2023/05/opencost-15.png

Helm

We can setup OpenCost with Helm as well.

Since I already launched with YAML, I’ll remove that instance

$ kubectl delete -n opencost -f  opencost.values.yaml
warning: deleting cluster-scoped resources, not scoped to the provided namespace
namespace "opencost" deleted
serviceaccount "opencost" deleted
clusterrole.rbac.authorization.k8s.io "opencost" deleted
clusterrolebinding.rbac.authorization.k8s.io "opencost" deleted
deployment.apps "opencost" deleted
service "opencost" deleted

Like before, I’ll have it use the Kubecost Prometheus metrics to save time

$ helm install opencost opencost/opencost  --set opencost.exporter.image.registry=gcr.io --set  opencost.exporter.image.repository=kubecost1/opencost --set opencost.exporter.image.tag=kc-eu-2023 --set opencost.prometheus.external.enabled=true --set opencost.prometheus.external.url=http://kubecost-prometheus-server.kubecost.svc --set opencost.prometheus.internal.enabled=false
E0517 09:12:38.429776     506 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0517 09:12:38.693345     506 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0517 09:12:38.761140     506 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0517 09:12:38.824133     506 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0517 09:12:38.916284     506 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E0517 09:12:38.986901     506 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
NAME: opencost
LAST DEPLOYED: Wed May 17 09:12:38 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Now when I port-forward (this time in the default namespace)

$ kubectl port-forward service/opencost 9003 9090
Forwarding from 127.0.0.1:9003 -> 9003
Forwarding from [::1]:9003 -> 9003
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090
Handling connection for 9090

I can see similar data

/content/images/2023/05/opencost-16.png

Comparing to older Kubecost

I went to checkout Kubecost, which was still supposidly running (for a year) but it seems to have gone AWOL

/content/images/2023/05/opencost-17.png

The body is just blank

<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Monitor and reduce Kubernetes spend"/><link rel="apple-touch-icon" href="./logo196.png"/><link rel="manifest" href="./manifest.json"/><script type="text/javascript" src="./jquery.min.3.6.0.js"></script><script type="text/javascript" src="./helper.js"></script><title>Kubecost</title><script defer="defer" src="./static/js/main.53a1d9a1.js"></script><link href="./static/css/main.61d98920.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><div id="portal-root"></div></body></html>

Summary

OpenCost is a very simple tool. There is no alerting, or automated reporting. It just shows calculations on usage from the cluster in a very easy to consume way. We can dump to a CSV if we want more nuanced data.

OpenCost KubeCost Kubernetes K3s

Have something to add? Feedback? You can use the feedback form

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes