Oracle Cloud: VMs, K3s, and Container Registry

Published: Dec 9, 2020 by Isaac Johnson

There is one cloud I’ve avoided. I have feelings about their business policies and treatment of the open source community. However, since Oracle is buying Tik Tok to leverage their Cloud, I decided it was worth looking into. They offer a free trial with 1 month of $300 credit presently, if you are able to sign up. Let me explain…

Signing up

I did most of this on my phone, to be honest. I used oracle.com/cloud/free

Then after fighting through a tedious and buggy interface, you will get the email confirmation moments before a 2min timer on the page expires

They then want a Credit Card for the free tier. Microsoft (a good company) does this for Azure, so i guess i’ll go along

You then get a message that it takes 15m to create

But then, to add to the fun.. I guess they ?manually review these things?

Several days later, around thanksgiving they did get back to me.

I got a call from Ireland, and indeed a real Irish person started to ask me “Why do you want an Oracle account? What will you use it for?”

I explained that I was using it for dev/learning and she said she would go ahead and “release” it… a few minutes later i got a login.

The login page looks like this:

I do like how they make your current costs front and center. I have to assume they intend for single tenancy in the portal, otherwise an “account bill” wouldn’t make sense.

Creating a VM

Lets create a “free” VM. It’s 1 “OCPU” (really, really?), 1 Gb of memory and 500Mbs network connectivity.

We can save the Private key and then create.

Lets test connectivity:

$ chmod 600 ~/Downloads/ssh-key-2020-12-09.key 
$ ssh -i ~/Downloads/ssh-key-2020-12-09.key opc@129.213.164.130
[opc@instance-20201209-0633 ~]$ sudo echo hi
hi
[opc@instance-20201209-0633 ~]$ 

Honestly, that was pretty fast.

Testing it looks to be CentOS/RHEL based:

[opc@instance-20201209-0633 ~]$ sudo yum update && sudo yum upgrade -y
Loaded plugins: langpacks, ulninfo
ol7_UEKR6 | 2.8 kB 00:00:00     
ol7_addons | 2.8 kB 00:00:00     
ol7_developer | 2.8 kB 00:00:00     
ol7_developer_EPEL | 3.4 kB 00:00:00     
ol7_ksplice | 2.8 kB 00:00:00     
ol7_latest | 3.4 kB 00:00:00     
ol7_oci_included | 2.9 kB 00:00:00     
ol7_optional_latest | 2.8 kB 00:00:00     
ol7_software_collections | 2.8 kB 00:00:00     
(1/19): ol7_UEKR6/x86_64/updateinfo | 56 kB 00:00:00     
(2/19): ol7_developer/x86_64/primary_db | 662 kB 00:00:00     
(3/19): ol7_developer_EPEL/x86_64/group_gz | 87 kB 00:00:00     
(4/19): ol7_developer_EPEL/x86_64/updateinfo | 6.3 kB 00:00:00     
(5/19): ol7_addons/x86_64/updateinfo | 95 kB 00:00:00     
(6/19): ol7_ksplice/updateinfo | 6.1 kB 00:00:00     
(7/19): ol7_addons/x86_64/primary_db | 165 kB 00:00:00     
(8/19): ol7_latest/x86_64/group_gz | 134 kB 00:00:00     
(9/19): ol7_UEKR6/x86_64/primary_db | 8.4 MB 00:00:01     
(10/19): ol7_latest/x86_64/updateinfo | 3.1 MB 00:00:00     
(11/19): ol7_ksplice/primary_db | 1.3 MB 00:00:00     
(12/19): ol7_oci_included/x86_64/primary_db | 404 kB 00:00:00     
(13/19): ol7_developer/x86_64/updateinfo | 13 kB 00:00:01     
(14/19): ol7_software_collections/x86_64/updateinfo | 8.6 kB 00:00:00     
(15/19): ol7_optional_latest/x86_64/updateinfo | 1.2 MB 00:00:00     
(16/19): ol7_optional_latest/x86_64/primary_db | 5.2 MB 00:00:00     
(17/19): ol7_developer_EPEL/x86_64/primary_db | 12 MB 00:00:00     
(18/19): ol7_software_collections/x86_64/primary_db | 5.2 MB 00:00:0
…
  tuned-profiles-oci-recommend.noarch 0:2.11.0-10.0.3.el7 uptrack.noarch 0:1.2.70-0.el7                      

Replaced:
  PyYAML.x86_64 0:3.10-11.el7 python-cffi.x86_64 0:1.6.0-5.el7 python-chardet.noarch 0:2.2.1-3.el7   
  python-idna.noarch 0:2.4-1.el7 python-urllib3.noarch 0:1.10.2-7.el7   

Complete!
Loaded plugins: langpacks, ulninfo
No packages marked for update
[opc@instance-20201209-0633 ~]$ 

Can we put k3s on it? (I certainly failed at doing this on Google’s free tier)

[opc@instance-20201209-0633 ~]$ curl -sfL https://get.k3s.io | sh -
[INFO] Finding release for channel stable
[INFO] Using v1.19.4+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.4+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.4+k3s1/k3s
…
Complete!
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink from /etc/systemd/system/multi-user.target.wants/k3s.service to /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
[opc@instance-20201209-0633 ~]$ 

Lets try it.. I cannot imagine this actually works…

[opc@instance-20201209-0633 ~]$ sudo cat /etc/rancher/k3s/k3s.yaml | base64 -w 0 
YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TF…. SNIP ….. WE5VRkJaa1l5YUdsNWRuTXhTbkJ1TVV0c1dRcDFhRGxtVDFWeVkydENhU3RZV21wR1dtOWpkelJaUVZGRUx6WXJNRWMyVkU5UlBUMEtMUzB0TFMxRlRrUWdSVU1nVUZKSlZrRlVSU0JMUlZrdExTMHRMUW89Cg==

# Now locally, 
$ echo YXB…..== | base64 --decode | sed 's/127.0.0.1/129.213.164.130/g' > ~/.kube/config

However, I will timeout unless i open port 6443.

$ kubectl get nodes
Unable to connect to the server: dial tcp 129.213.164.130:6443: i/o timeout

I’ll go to the subnet on the instance details page, clicking on the subnet id

Here we can see the Security Lists, click on the default to edit:

Click Add Ingress rule and add one for 6443:

You can limit to your own CIDR, but for testing I’ll use 0.0.0.0/0 (everywhere).

$ kubectl get nodes
The connection to the server 129.213.164.130:6443 was refused - did you specify the right host or port?

We can disable the firewall for now

[opc@instance-20201209-0633 ~]$ sudo systemctl stop firewalld
[opc@instance-20201209-0633 ~]$ 

However, the appropriate way is to just add 6443 to firewalld:

$ sudo firewall-cmd --zone=public --add-port=6443/tcp
success

Let’s install again, this time specifying the external IP:

$ curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --node-external-ip=129.213.164.130 --advertise-address=129.213.164.130" sh -
[INFO] Finding release for channel stable
[INFO] Using v1.19.4+k3s1 as release
..

# try another way
$ curl -sfL https://get.k3s.io | sh -s - --node-external-ip 129.213.164.130 --advertise-address 129.213.164.130 --tls-san 129.213.164.130
[INFO] Finding release for channel stable
[INFO] Using v1.19.4+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.4+k3s1/sha256sum-amd64.txt
[INFO] Skipping binary downloaded, installed k3s matches hash
Loaded plugins: langpacks, ulninfo
…

# and another way
[opc@instance-20201209-0633 ~]$ curl -sfL https://get.k3s.io | sh -s - --node-external-ip "129.213.164.130" --advertise-address "129.213.164.130" --tls-san "129.213.164.130"
[INFO] Finding release for channel stable
[INFO] Using v1.19.4+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.4+k3s1/sha256sum-amd64.txt
[INFO] Skipping binary downloaded, installed k3s ma

Seems i cannot get k3s to create a proper cert for that IP.. we can skip TLS verify for now…

$ kubectl get nodes --insecure-skip-tls-verify
NAME STATUS ROLES AGE VERSION
instance-20201209-0633 Ready master 49m v1.19.4+k3s1

Let’s try once more, this time on a fresh instance. I’m not convinced it’s actually picking up the settings.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--tls-san x.x.x.x" sh -s -

That worked - i blame that first install for not fixing certs:

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
instance-20201209-0756 Ready master 4m1s v1.19.4+k3s1

##
Create a Registry

You can search for Registry in the search to create a Container Registry.

Let’s create a Repository:

We’ll create a public repo to see how well it fares

It is nice that after creation, one can switch it to private:

What i find fascinating is if you want to change image retention policies, its a change that can take 24h to implement:

Now how do we use it? We see no URL or login credentials. I can expand the User to show Ocid1.user.oc1..aaaaaaaafxdrwzmqhv5jmn4vbip7inbdibhvfpzrj7l74mylfrtvneojdxaa

We need to create an Auth Token to use this before we can login.

Auth Token

Get that in user settings:

Here we can create and delete tokens. No idea when they expire, however:

Clicking Generate Token:

I guess that token can do all the things?

We need to login to the “regional” container registry. Which according to that sheet is:

https://us-ashburn-1.ocir.io
https://iad.ocir.io

Use the instance and your oracle id (email)

$ docker login iad.ocir.io
Username: id5esgrthabn/isaac.johnson@gmail.com
Password: 
Login Succeeded

I did use the Token for the password

Lets tag and push hello-world:

$ docker images | grep hello
hello-world latest bf756fb1ae65 11 months ago 13.3kB
registry-intl.us-east-1.aliyuncs.com/freshbrewed/helloworld 1 bf756fb1ae65 11 months ago 13.3kB
$ docker tag hello-world:latest iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest
$ docker push iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest
The push refers to repository [iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world]
9c27e219663c: Pushed 
latest: digest: sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 size: 525

We can now see that reflected in our Registry:

Testing with the k3s we created…

$ kubectl run helloworld --image=iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest
pod/helloworld created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
helloworld 0/1 ImagePullBackOff 0 88s

Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 15s default-scheduler Successfully assigned default/helloworld to instance-20201209-0756
  Normal Pulling <invalid> (x3 over 14s) kubelet Pulling image "iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest"
  Warning Failed <invalid> (x3 over 14s) kubelet Failed to pull image "iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest": rpc error: code = Unknown desc = failed to pull and unpack image "iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest": failed to resolve reference "iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest": unexpected status code [manifests latest]: 403 Forbidden

Ah, we need to set each image repo to public (it’s not a parent relationship):

We can now see it’s “(Public)”

I tested, but perhaps the change is not immediate:

$ kubectl delete pod helloworld
pod "helloworld" deleted
$ kubectl get pods
No resources found in default namespace.
$ kubectl run helloworld --image=iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest
pod/helloworld created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
helloworld 0/1 ErrImagePull 0 3s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
helloworld 0/1 ImagePullBackOff 0 20s

Let’s work around this… I shouldn’t need a login credential, but this should work for a private registry…

$ kubectl create secret docker-registry ocrcred --docker-server=iad.ocir.io --docker-username='id5esgrthabn/isaac.johnson@gmail.com' --docker-password='UIit<nev9+Ec9+3u>52(' --docker-email=isaac.johnson@gmail.com
secret/ocrcred created
$ cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: ocrprivate
spec:
  containers:
  - name: helloworld
    image: iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest
  imagePullSecrets:
  - name: ocrcred

This time we get a different error:

  Warning Failed <invalid> (x3 over 11s) kubelet Failed to pull image "iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest": rpc error: code = NotFound desc = failed to pull and unpack image "iad.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest": failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 (application/json) from remote: not found

Trying the other endpoint:

Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled <invalid> default-scheduler Successfully assigned default/ocrprivate3 to instance-20201209-0756
  Normal Pulling <invalid> kubelet Pulling image "us-ashburn-1.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest"
  Warning Failed <invalid> kubelet Failed to pull image "us-ashburn-1.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest": rpc error: code = NotFound desc = failed to pull and unpack image "us-ashburn-1.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest": failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042 (application/json) from remote: not found
  Warning Failed <invalid> kubelet Error: ErrImagePull
  Warning Failed <invalid> kubelet Error: ImagePullBackOff
  Normal BackOff <invalid> kubelet Back-off pulling image "us-ashburn-1.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest"

To rule out k3s, i used AKS

Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled <invalid> default-scheduler Successfully assigned johnsi10/ocrprivate3 to aks-agentpool-17684165-vmss000001
  Normal Created <invalid> (x2 over <invalid>) kubelet Created container helloworld
  Normal Started <invalid> (x2 over <invalid>) kubelet Started container helloworld
  Warning BackOff <invalid> (x2 over <invalid>) kubelet Back-off restarting failed container
  Normal Pulling <invalid> (x3 over <invalid>) kubelet Pulling image "us-ashburn-1.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest"
  Normal Pulled <invalid> (x3 over <invalid>) kubelet Successfully pulled image "us-ashburn-1.ocir.io/id5esgrthabn/ijohnsonpublicrepo/hello-world:latest"

Indeed, this worked fine.. And i then tried without a credential in the default namespace and that worked. So we will chalk up k8s issues to the basic k3s i used.

Checking Usage

In Oracle Cloud, we can see the number of pulls thus far:

We can lookup all our resources:

Oracle cloud groups things into “Compartments” which is akin to Azure’s Resource Groups.

A key difference is that compartments are generated strings. For example, all the above are in “ocid1.tenancy.oc1..aaaaaaaa5b4id2a4wsenr67b5eveb6umobdcohh3hqiblubne5lfush5vshq”

Budgets

A nice feature they have is a budget alert. You can denote both actual spend and forecast which is nice.

You can also use both custom and Oracle generated tags for the alerts:

This could be handy in tracking costs for various users or teams. For reference, we can see tags on the Tags tab for most resources:

However, i cannot find a chart that lists prices for “shapes” (vm image sizes). At this time, im gunshy about “upgrading” to pay-per-use during the free trial month;

I will give credit where credit is due, however, the free E2.1.Micro is _very_ fast for such a small class of machine. It’s significantly faster than GCPs free tier machines.

But what are the costs?

We can see the main sheet shows “E3” is $0.03/hour.. But which shape? https://www.oracle.com/cloud/compute/pricing.html

The cost calculator does not help:

I also saw no costs for public IPs.. usually that is a small add on for cloud providers:

Cleanup

Expiring an auth token is pretty immediate

$ docker login iad.ocir.io
Authenticating with existing credentials...
Login did not succeed, error: Error response from daemon: Get https://iad.ocir.io/v2/: unknown: Unauthorized
Username (id5esgrthabn/isaac.johnson@gmail.com): id5esgrthabn/isaac.johnson@gmail.com
Password: 
Error response from daemon: Get https://iad.ocir.io/v2/: unknown: Unauthorized

CLI

We can follow the Installation instructions. On mac we can use brew

$ brew update && brew install oci-cli
…
For pkg-config to find python@3.8 you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/python@3.8/lib/pkgconfig"

==> azure-cli
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

We can then login

$ oci session authenticate
Enter a region (e.g. ap-chiyoda-1, ap-chuncheon-1, ap-hyderabad-1, ap-melbourne-1, ap-mumbai-1, ap-osaka-1, ap-seoul-1, ap-sydney-1, ap-tokyo-1, ca-montreal-1, ca-toronto-1, eu-amsterdam-1, eu-frankfurt-1, eu-zurich-1, me-dubai-1, me-jeddah-1, sa-santiago-1, sa-saopaulo-1, uk-cardiff-1, uk-gov-cardiff-1, uk-gov-london-1, uk-london-1, us-ashburn-1, us-gov-ashburn-1, us-gov-chicago-1, us-gov-phoenix-1, us-langley-1, us-luke-1, us-phoenix-1, us-sanjose-1): us-ashburn-1
    Please switch to newly opened browser window to log in!
    Completed browser authentication process!
Config written to: /Users/johnsi10/.oci/config

    Try out your newly created session credentials with the following example command:

    oci iam region list --config-file /Users/johnsi10/.oci/config --profile DEFAULT --auth security_token

$ oci compute instance list --compartment-id ocid1.tenancy.oc1..aaaaaaaa5b4id2a4wsenr67b5eveb6umobdcohh3hqiblubne5lfush5vshq --config-file /Users/johnsi10/.oci/config --profile DEFAULT --auth security_token | jq '.[] | .[] | .id'
"ocid1.instance.oc1.iad.anuwcljtlidsygqcl43qrkajtpffmte5di6lli32ipd4tlrcj5odrfkjcwoq"
"ocid1.instance.oc1.iad.anuwcljtlidsygqcp2zuoz6cdefjtpkvqaj72dfbju4amopmbmi7d2lk2r5q"

and VNICs

$ oci compute vnic-attachment list --compartment-id ocid1.tenancy.oc1..aaaaaaaa5b4id2a4wsenr67b5eveb6umobdcohh3hqiblubne5lfush5vshq --config-file /Users/johnsi10/.oci/config --profile DEFAULT --auth security_token | jq '.[] | .[] | ."vnic-id"'
"ocid1.vnic.oc1.iad.abuwcljtapl4fxt57yoficrqzrn2wp6mbfiyietho7femto6xjf6767b63xq"
"ocid1.vnic.oc1.iad.abuwcljtda2tyalbzb4dxln4xo3aavbima4fdnch575onxzzh7guwwtsg2vq"

Summary

Oracle Cloud has some rather nice features. It seems that the free tier of VM is actually quite fast and useful. We were able to use a basic E2 instance to run k3s meaning they are container friendly VMs. Creating a container registry was just as easy - and like IBM Cloud, it’s a namespace on their host.

The parts that are challenging are certain updates have very long delays. Other than CloudFront expirations, I’ve never seen 24h delays on any activity in AWS and certainly not Azure.

Additionally, and I say this with a smile - you can tell this is a Database company first. Many IDs look like Oracle DB Object IDs. The cost report is all database query driven:

The costs are a big open topic - will I be charged for Public IP? Network ingress? What about the container registry? I’ll have to watch my emails and check back in the coming weeks.