Published: Jul 25, 2023 by Isaac Johnson
I have been asked about instrumenting Ruby code for New Relic a lot lately and figured now might be a good time to cover some of the scalable patterns.
Today we’ll set up a sample Ruby app configured to access a sample PostgreSQL database. We will then setup APM instrumentation to New Relic; first natively, then through zipkin and lastly using OpenTelemetry and Kubernetes.
We have a lot to cover, so let’s get started!
Setting up a Ruby app
First, we’ll create a Github repo with a Readme and MIT license
We’ll clone it down locally
builder@DESKTOP-QADGF36:~/Workspaces$ git clone https://github.com/idjohnson/demo-containerized-app.git
Cloning into 'demo-containerized-app'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (5/5), done.
Unpacking objects: 100% (5/5), 1.97 KiB | 1.97 MiB/s, done.
remote: Total 5 (delta 0), reused 0 (delta 0), pack-reused 0
builder@DESKTOP-QADGF36:~/Workspaces$ cd demo-containerized-app/
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ ls
LICENSE README.md
Now let’s create a Dockerfile that would run an app. The next few steps started from Don Schencks Demo App should you want to look at the source.
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ vi Dockerfile
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ cat Dockerfile
FROM ruby:latest
# throw errors if Gemfile has been modified
RUN bundle config --global frozen 1
WORKDIR /usr/src/app/
COPY Gemfile Gemfile.lock ./
RUN bundle install
ADD . /usr/src/app/
EXPOSE 4000
CMD ["ruby", "/usr/src/app/helloworld.rb"]
We need a Gemfile and Gemfile.lock
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ cat Gemfile
# frozen_string_literal: true
source "https://rubygems.org"
git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
# gem "rails"
gem "sinatra"
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ cat Gemfile.lock
GEM
remote: https://rubygems.org/
specs:
mustermann (1.0.3)
rack (2.0.6)
rack-protection (2.0.5)
rack
sinatra (2.0.5)
mustermann (~> 1.0)
rack (~> 2.0)
rack-protection (= 2.0.5)
tilt (~> 2.0)
tilt (2.0.9)
PLATFORMS
ruby
DEPENDENCIES
sinatra
BUNDLED WITH
1.17.1
Lastly, and most importantly, a hello world app
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ cat helloworld.rb
require 'sinatra'
set :port, 4000
set :bind, '0.0.0.0'
get '/' do
'Hello World!'
end
I’ll now build it locally. You do not need to specify a tag but I do it out of habit.
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ docker build -t hello-world-ruby:0.0.1 .
[+] Building 19.7s (10/11)
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 293B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ruby:latest 2.3s
=> [auth] library/ruby:pull token for registry-1.docker.io 0.0s
=> [1/6] FROM docker.io/library/ruby:latest@sha256:05ca729e257fc550f4228714a5f6c9f377d4dab92c30b2213340ddb7eec0 11.1s
=> => resolve docker.io/library/ruby:latest@sha256:05ca729e257fc550f4228714a5f6c9f377d4dab92c30b2213340ddb7eec0b 0.0s
=> => sha256:669a9f71cf3b339ad85ba8d4a5efe7511cd57f59f5c2fd492970e683608a8d8b 1.79kB / 1.79kB 0.0s
=> => sha256:0c4b46102a5273d695c1a776990d77dba5939119a29806a958495e9645a73f19 8.15kB / 8.15kB 0.0s
=> => sha256:05ca729e257fc550f4228714a5f6c9f377d4dab92c30b2213340ddb7eec0b597 1.86kB / 1.86kB 0.0s
=> => sha256:d52e4f012db158bb7c0fe215b98af1facaddcbaee530efd69b1bae07d597b711 49.55MB / 49.55MB 1.7s
=> => sha256:7dd206bea61ff3e3b54be1c20b58d8475ddd6f89df176146ddb7a2fd2c747ea2 24.03MB / 24.03MB 3.1s
=> => sha256:2320f9be4a9c605d1ac847cf67cec42b91484a7cf7c94996417a0c7c316deadc 64.11MB / 64.11MB 2.5s
=> => sha256:6e5565e0ba8dfce32b9049f21ceeb212946e0bb810d94cbd2db94ca61082f657 211.00MB / 211.00MB 6.0s
=> => extracting sha256:d52e4f012db158bb7c0fe215b98af1facaddcbaee530efd69b1bae07d597b711 1.1s
=> => sha256:3487b74cbe46c33dbbf395e0b72508708eaf83c574adc8e075d7f445fb076b3e 199B / 199B 2.6s
=> => sha256:d674e4eae0fcd4fe9b8e66aad48dee351b9b97981f2cf6a9a5742e0ced525cab 34.74MB / 34.74MB 4.2s
=> => sha256:22b888c5c84f49fcd6d95f0781bf6b03c7931053a76c64e1d517758a4aaa9f05 176B / 176B 3.5s
=> => extracting sha256:7dd206bea61ff3e3b54be1c20b58d8475ddd6f89df176146ddb7a2fd2c747ea2 0.5s
=> => extracting sha256:2320f9be4a9c605d1ac847cf67cec42b91484a7cf7c94996417a0c7c316deadc 1.5s
=> => extracting sha256:6e5565e0ba8dfce32b9049f21ceeb212946e0bb810d94cbd2db94ca61082f657 4.1s
=> => extracting sha256:3487b74cbe46c33dbbf395e0b72508708eaf83c574adc8e075d7f445fb076b3e 0.0s
=> => extracting sha256:d674e4eae0fcd4fe9b8e66aad48dee351b9b97981f2cf6a9a5742e0ced525cab 0.5s
=> => extracting sha256:22b888c5c84f49fcd6d95f0781bf6b03c7931053a76c64e1d517758a4aaa9f05 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 28.69kB 0.0s
=> [2/6] RUN bundle config --global frozen 1 1.8s
=> [3/6] WORKDIR /usr/src/app/ 0.0s
=> [4/6] COPY Gemfile Gemfile.lock ./ 0.0s
=> ERROR [5/6] RUN bundle install 3.7s
------
> [5/6] RUN bundle install:
#10 0.688 Bundler 2.4.10 is running, but your lockfile was generated with 1.17.1. Installing Bundler 1.17.1 and restarting using that version.
#10 3.174 Fetching gem metadata from https://rubygems.org/.
#10 3.224 Fetching bundler 1.17.1
#10 3.478 Installing bundler 1.17.1
#10 3.682 /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/shared_helpers.rb:272:in `search_up': undefined method `untaint' for "/usr/src/app":String (NoMethodError)
#10 3.682
#10 3.682 current = File.expand_path(SharedHelpers.pwd).untaint
#10 3.682 ^^^^^^^^
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/shared_helpers.rb:259:in `find_file'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/shared_helpers.rb:251:in `find_gemfile'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/shared_helpers.rb:27:in `root'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler.rb:234:in `root'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler.rb:244:in `app_config_path'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler.rb:273:in `settings'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/feature_flag.rb:21:in `block in settings_method'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/cli.rb:97:in `<class:CLI>'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/cli.rb:7:in `<module:Bundler>'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/cli.rb:6:in `<top (required)>'
#10 3.682 from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
#10 3.682 from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/exe/bundle:23:in `block in <top (required)>'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
#10 3.682 from /usr/local/bundle/gems/bundler-1.17.1/exe/bundle:22:in `<top (required)>'
#10 3.682 from /usr/local/bin/bundle:25:in `load'
#10 3.682 from /usr/local/bin/bundle:25:in `<main>'
------
executor failed running [/bin/sh -c bundle install]: exit code: 1
The first time through my bundler was out of date (as I use Ruby for this blog)
I needed to update my bundler (which was clear when I tried to do a bundle update)
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ bundle update
Traceback (most recent call last):
2: from /home/builder/gems/bin/bundle:23:in `<main>'
1: from /usr/lib/ruby/2.7.0/rubygems.rb:294:in `activate_bin_path'
/usr/lib/ruby/2.7.0/rubygems.rb:275:in `find_spec_for_exe': Could not find 'bundler' (1.17.1) required by your /home/builder/Workspaces/demo-containerized-app/Gemfile.lock. (Gem::GemNotFoundException)
To update to the latest version installed on your system, run `bundle update --bundler`.
To install the missing version, run `gem install bundler:1.17.1`
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ gem install bundler:1.17.1
Fetching bundler-1.17.1.gem
Successfully installed bundler-1.17.1
Parsing documentation for bundler-1.17.1
Installing ri documentation for bundler-1.17.1
Done installing documentation for bundler after 2 seconds
1 gem installed
Then I could run update
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ bundle update
Fetching gem metadata from https://rubygems.org/....
Resolving dependencies...
Using bundler 1.17.1
Fetching ruby2_keywords 0.0.5
Installing ruby2_keywords 0.0.5
Fetching mustermann 3.0.0 (was 1.0.3)
Installing mustermann 3.0.0 (was 1.0.3)
Fetching rack 2.2.7 (was 2.0.6)
Installing rack 2.2.7 (was 2.0.6)
Fetching rack-protection 3.0.6 (was 2.0.5)
Installing rack-protection 3.0.6 (was 2.0.5)
Fetching tilt 2.2.0 (was 2.0.9)
Installing tilt 2.2.0 (was 2.0.9)
Fetching sinatra 3.0.6 (was 2.0.5)
Installing sinatra 3.0.6 (was 2.0.5)
Bundle updated!
My Gemfile.lock now has the proper gems (as of this writing)
$ cat Gemfile.lock
GEM
remote: https://rubygems.org/
specs:
mustermann (3.0.0)
ruby2_keywords (~> 0.0.1)
rack (2.2.7)
rack-protection (3.0.6)
rack
ruby2_keywords (0.0.5)
sinatra (3.0.6)
mustermann (~> 3.0)
rack (~> 2.2, >= 2.2.4)
rack-protection (= 3.0.6)
tilt (~> 2.0)
tilt (2.2.0)
PLATFORMS
ruby
DEPENDENCIES
sinatra
BUNDLED WITH
1.17.1
Let’s try again
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ docker build -t hello-world-ruby:0.0.2 .
[+] Building 5.0s (9/10)
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ruby:latest 0.7s
=> [1/6] FROM docker.io/library/ruby:latest@sha256:05ca729e257fc550f4228714a5f6c9f377d4dab92c30b2213340ddb7eec0b597 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 2.77kB 0.0s
=> CACHED [2/6] RUN bundle config --global frozen 1 0.0s
=> CACHED [3/6] WORKDIR /usr/src/app/ 0.0s
=> [4/6] COPY Gemfile Gemfile.lock ./ 0.0s
=> ERROR [5/6] RUN bundle install 3.6s
------
> [5/6] RUN bundle install:
#9 0.480 Bundler 2.4.10 is running, but your lockfile was generated with 1.17.1. Installing Bundler 1.17.1 and restarting using that version.
#9 3.101 Fetching gem metadata from https://rubygems.org/.
#9 3.151 Fetching bundler 1.17.1
#9 3.401 Installing bundler 1.17.1
#9 3.594 /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/shared_helpers.rb:272:in `search_up': undefined method `untaint' for "/usr/src/app":String (NoMethodError)
#9 3.594
#9 3.594 current = File.expand_path(SharedHelpers.pwd).untaint
#9 3.594 ^^^^^^^^
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/shared_helpers.rb:259:in `find_file'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/shared_helpers.rb:251:in `find_gemfile'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/shared_helpers.rb:27:in `root'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler.rb:234:in `root'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler.rb:244:in `app_config_path'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler.rb:273:in `settings'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/feature_flag.rb:21:in `block in settings_method'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/cli.rb:97:in `<class:CLI>'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/cli.rb:7:in `<module:Bundler>'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/cli.rb:6:in `<top (required)>'
#9 3.594 from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
#9 3.594 from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/exe/bundle:23:in `block in <top (required)>'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
#9 3.594 from /usr/local/bundle/gems/bundler-1.17.1/exe/bundle:22:in `<top (required)>'
#9 3.594 from /usr/local/bin/bundle:25:in `load'
#9 3.594 from /usr/local/bin/bundle:25:in `<main>'
------
executor failed running [/bin/sh -c bundle install]: exit code: 1
I read that it could be due to an old Gemfile lock. I removed and tried again.
this worked:
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ rm Gemfile.lock
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ bundle install
Fetching gem metadata from https://rubygems.org/....
Resolving dependencies...
Using bundler 2.2.26
Using ruby2_keywords 0.0.5
Using rack 2.2.7
Using tilt 2.2.0
Using mustermann 3.0.0
Using rack-protection 3.0.6
Using sinatra 3.0.6
Bundle complete! 1 Gemfile dependency, 7 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ docker build -t hello-world-ruby:0.0.3 .
[+] Building 6.6s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ruby:latest 0.7s
=> [1/6] FROM docker.io/library/ruby:latest@sha256:05ca729e257fc550f4228714a5f6c9f377d4dab92c30b2213340ddb7eec0b597 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 2.78kB 0.0s
=> CACHED [2/6] RUN bundle config --global frozen 1 0.0s
=> CACHED [3/6] WORKDIR /usr/src/app/ 0.0s
=> [4/6] COPY Gemfile Gemfile.lock ./ 0.0s
=> [5/6] RUN bundle install 5.0s
=> [6/6] ADD . /usr/src/app/ 0.0s
=> exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:48e32887d4af01521acee8c0d86bbbcaa03392bc87728d15bd9d01829eabcea5 0.0s
=> => naming to docker.io/library/hello-world-ruby:0.0.3 0.0s
Now let’s pause and create our first commit before we get too far.
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git add -A
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
new file: Dockerfile
new file: Gemfile
new file: Gemfile.lock
new file: helloworld.rb
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git commit -m "First Pass"
[main f899da3] First Pass
4 files changed, 55 insertions(+)
create mode 100644 Dockerfile
create mode 100644 Gemfile
create mode 100644 Gemfile.lock
create mode 100644 helloworld.rb
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git push
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 16 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.06 KiB | 1.06 MiB/s, done.
Total 6 (delta 0), reused 0 (delta 0)
To https://github.com/idjohnson/demo-containerized-app.git
2bb51f9..f899da3 main -> main
I can now try and run it
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ docker run -p 4000:4000 hello-world-ruby:0.0.3
/usr/local/bundle/gems/rack-2.2.7/lib/rack/handler.rb:45:in `pick': Couldn't find handler for: thin, falcon, puma, HTTP, webrick. (LoadError)
from /usr/local/bundle/gems/sinatra-3.0.6/lib/sinatra/base.rb:1526:in `run!'
from /usr/local/bundle/gems/sinatra-3.0.6/lib/sinatra/main.rb:47:in `block in <module:Sinatra>'
Seems we missed Puma. We’ll add and update the lock with bundler
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ vi Gemfile
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git diff Gemfile
diff --git a/Gemfile b/Gemfile
index b86565d..2edd1ed 100644
--- a/Gemfile
+++ b/Gemfile
@@ -5,4 +5,5 @@ source "https://rubygems.org"
git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
# gem "rails"
+gem 'puma'
gem "sinatra"
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ bundle install
Fetching gem metadata from https://rubygems.org/....
Resolving dependencies...
Using bundler 2.2.26
Using ruby2_keywords 0.0.5
Using rack 2.2.7
Using tilt 2.2.0
Using mustermann 3.0.0
Using rack-protection 3.0.6
Fetching nio4r 2.5.9
Using sinatra 3.0.6
Installing nio4r 2.5.9 with native extensions
Fetching puma 6.3.0
Installing puma 6.3.0 with native extensions
Bundle complete! 2 Gemfile dependencies, 9 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
Then I’ll build fresh and try again
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ docker build -t hello-world-ruby:0.0.4 .
[+] Building 13.4s (12/12) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ruby:latest 1.4s
=> [auth] library/ruby:pull token for registry-1.docker.io 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 6.62kB 0.0s
=> [1/6] FROM docker.io/library/ruby:latest@sha256:05ca729e257fc550f4228714a5f6c9f377d4dab92c30b2213340ddb7eec0b597 0.0s
=> CACHED [2/6] RUN bundle config --global frozen 1 0.0s
=> CACHED [3/6] WORKDIR /usr/src/app/ 0.0s
=> [4/6] COPY Gemfile Gemfile.lock ./ 0.0s
=> [5/6] RUN bundle install 11.0s
=> [6/6] ADD . /usr/src/app/ 0.0s
=> exporting to image 0.2s
=> => exporting layers 0.2s
=> => writing image sha256:0f24cdc43153ba58816d7a4172c03d6dfa18e2184bd850c4a34acf855cbb161c 0.0s
=> => naming to docker.io/library/hello-world-ruby:0.0.4 0.0s
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ docker run -p 4000:4000 hello-world-ruby:0.0.4
== Sinatra (v3.0.6) has taken the stage on 4000 for development with backup from Puma
Puma starting in single mode...
* Puma version: 6.3.0 (ruby 3.2.2-p53) ("Mugi No Toki Itaru")
* Min threads: 0
* Max threads: 5
* Environment: development
* PID: 1
* Listening on http://0.0.0.0:4000
Use Ctrl-C to stop
Database
For the next part, I’ll go to a Linux host where I have PostgreSQL already running. There I will create a user for the demo app
postgres@isaac-MacBookAir:~$ createuser --pwprompt demoapp
Enter password for new role:
Enter it again:
We’ll now create a simple database with a table and a row of data to query.
postgres@isaac-MacBookAir:~$ createdb demoapp
postgres@isaac-MacBookAir:~$ psql
psql (12.15 (Ubuntu 12.15-0ubuntu0.20.04.1))
Type "help" for help.
postgres=# grant all privileges on database demoapp to demoapp;
GRANT
postgres=# \c demoapp
You are now connected to database "demoapp" as user "postgres".
demoapp=# create table demotable (id INTEGER PRIMARY KEY, name VARCHAR);
CREATE TABLE
demoapp=# INSERT INTO demotable VALUES (1, 'Hello World');
INSERT 0 1
demoapp=# SELECT * FROM demotable;
id | name
----+-------------
1 | Hello World
(1 row)
Before I can add PSQL to the Docker file, the Gem will need libpq-dev locally for some libraries
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ sudo apt update && sudo apt install libpq-dev
[sudo] password for builder:
Get:1 https://apt.releases.hashicorp.com focal InRelease [12.9 kB]
Get:2 https://packages.microsoft.com/repos/azure-cli focal InRelease [3575 B]
Err:1 https://apt.releases.hashicorp.com focal InRelease
... snip ...
I added the ‘pg’ gem and did a bundle install
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git diff
diff --git a/Gemfile b/Gemfile
index b86565d..fe681c4 100644
--- a/Gemfile
+++ b/Gemfile
@@ -5,4 +5,6 @@ source "https://rubygems.org"
git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
# gem "rails"
+gem 'pg'
+gem 'puma'
gem "sinatra"
diff --git a/Gemfile.lock b/Gemfile.lock
index 272decb..8cffe25 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -3,6 +3,10 @@ GEM
specs:
mustermann (3.0.0)
ruby2_keywords (~> 0.0.1)
+ nio4r (2.5.9)
+ pg (1.5.3)
+ puma (6.3.0)
+ nio4r (~> 2.0)
rack (2.2.7)
rack-protection (3.0.6)
rack
@@ -18,6 +22,8 @@ PLATFORMS
x86_64-linux
DEPENDENCIES
+ pg
+ puma
sinatra
BUNDLED WITH
diff --git a/helloworld.rb b/helloworld.rb
index 050c9f7..a4c62c4 100644
--- a/helloworld.rb
+++ b/helloworld.rb
@@ -1,8 +1,31 @@
The ruby code then used the pg gem to connect to the local database and fetch rows
$ cat helloworld.rb
require 'sinatra'
require 'pg'
set :port, 4000
set :bind, '0.0.0.0'
get '/' do
begin
# Initialize connection variables.
host = String('192.168.1.78')
database = String('demoapp')
user = String('demoapp')
password = String('demopass')
# Initialize connection object.
connection = PG::Connection.new(:host => host, :user => user, :dbname => database, :port => '5432', :password => password)
puts 'Successfully created connection to database.'
resultSet = connection.exec('SELECT * from demotable;')
resultSet.each do |row|
puts 'Data row = (%s, %s)' % [row['id'], row['name']]
end
rescue PG::Error => e
puts e.message
ensure
connection.close if connection
end
'Hello World!'
end
Doing a test showed we might have missed a permission or needed to flush permissions in postgres
postgres@isaac-MacBookAir:~$ psql -d demoapp
psql (12.15 (Ubuntu 12.15-0ubuntu0.20.04.1))
Type "help" for help.
demoapp=# GRANT SELECT ON ALL TABLES IN SCHEMA public TO demoapp;
GRANT
demoapp=# GRANT ALL ON ALL TABLES IN SCHEMA public TO demoapp;
GRANT
That seemed to work. We’ll mark that part 2 and push it up
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git add Gemfile
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git add Gemfile.lock
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git add helloworld.rb
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git commit -m "part 2"
[main cb6610d] part 2
3 files changed, 32 insertions(+), 1 deletion(-)
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git push
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 16 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 886 bytes | 886.00 KiB/s, done.
Total 5 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/idjohnson/demo-containerized-app.git
f899da3..cb6610d main -> main
I’ll make one more tweak to just show the row to the user:
require 'sinatra'
require 'pg'
set :port, 4000
set :bind, '0.0.0.0'
get '/' do
begin
# Initialize connection variables.
host = String('192.168.1.78')
database = String('demoapp')
user = String('demoapp')
password = String('demopass')
# Initialize connection object.
connection = PG::Connection.new(:host => host, :user => user, :dbname => database, :port => '5432', :password => password)
puts 'Successfully created connection to database.'
resultSet = connection.exec('SELECT * from demotable;')
outStr = "<table><tr><th>id</th><th>name</th></tr>"
resultSet.each do |row|
puts 'Data row = (%s, %s)' % [row['id'], row['name']]
row_data = "<tr><td>#{row['id']}</td><td>#{row['name']}}</td></tr>"
outStr += "#{row_data}"
end
outStr += "</table>"
outStr
rescue PG::Error => e
puts e.message
ensure
connection.close if connection
end
end
and test
I’ll push as part 3
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git add -A
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git commit -m "part 3"
[main 92c25c7] part 3
1 file changed, 5 insertions(+), 1 deletion(-)
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 16 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 406 bytes | 406.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/idjohnson/demo-containerized-app.git
cb6610d..92c25c7 main -> main
New Relic APM
First, we need the NR License Key (or API Key). We can get that from “API Keys” in Administration
I’ll add the NR Gems
diff --git a/Gemfile b/Gemfile
index fe681c4..4e2c2c1 100644
--- a/Gemfile
+++ b/Gemfile
@@ -8,3 +8,6 @@ git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }
gem 'pg'
gem 'puma'
gem "sinatra"
+
+gem 'newrelic_rpm'
+gem 'newrelic-infinite_tracing'
\ No newline at end of file
diff --git a/Gemfile.lock b/Gemfile.lock
and then bundle install
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ bundle install
Fetching gem metadata from https://rubygems.org/..........
Resolving dependencies...
Using bundler 2.2.26
Using ruby2_keywords 0.0.5
Using nio4r 2.5.9
Using pg 1.5.3
Fetching newrelic_rpm 9.3.1
Using tilt 2.2.0
Using rack 2.2.7
Using rack-protection 3.0.6
Fetching google-protobuf 3.23.4 (x86_64-linux)
Using puma 6.3.0
Using mustermann 3.0.0
Using sinatra 3.0.6
Installing newrelic_rpm 9.3.1
Installing google-protobuf 3.23.4 (x86_64-linux)
Fetching googleapis-common-protos-types 1.7.0
Installing googleapis-common-protos-types 1.7.0
Fetching grpc 1.56.2 (x86_64-linux)
Installing grpc 1.56.2 (x86_64-linux)
Fetching newrelic-infinite_tracing 9.3.1
Installing newrelic-infinite_tracing 9.3.1
Bundle complete! 5 Gemfile dependencies, 15 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
I can now use the local newrelic binary to create the ‘newrelic.yml’ file
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ newrelic install --license_key="***********mykey***********NRAL" "FBDemoApp"
Installed a default configuration file at
/home/builder/Workspaces/demo-containerized-app/newrelic.yml.
Visit support.newrelic.com if you are experiencing installation issues.
Or you can use the boilerplate yaml.
Our Dockerfile allready adds “.” so that should bring in the YAML with our code.
Let’s build and run the Dockerfile
And I can now see an entry in New Relic APM
Make sure to set the transaction type to “All” to see example runs
Before we go and commit things, realize that the New Relic license key is plain text in the newrelic.yml. That is probably something we do not wish to check in.
We’ll add that to our gitignore
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ echo "newrelic.yml" >> .gitignore
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: .gitignore
modified: Gemfile
modified: Gemfile.lock
modified: helloworld.rb
no changes added to commit (use "git add" and/or "git commit -a")
We’ll make this Verison 4
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: .gitignore
modified: Gemfile
modified: Gemfile.lock
modified: helloworld.rb
no changes added to commit (use "git add" and/or "git commit -a")
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git add -A
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git commit -m "With NewRelic"
[main fa12c02] With NewRelic
4 files changed, 18 insertions(+)
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ git push
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 16 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 870 bytes | 870.00 KiB/s, done.
Total 6 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To https://github.com/idjohnson/demo-containerized-app.git
92c25c7..fa12c02 main -> main
We do not need to bundle the YAML. In fact, that is likely a bad idea. For the License Key and Name, we can just as well use the environment variables.
I’ll build a new container, sans newrelic.yaml
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ rm newrelic.yml
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ docker build -t hello-world-ruby:0.1.1 .
[+] Building 2.2s (12/12) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ruby:latest 1.5s
=> [auth] library/ruby:pull token for registry-1.docker.io 0.0s
=> [1/6] FROM docker.io/library/ruby:latest@sha256:05ca729e257fc550f4228714a5f6c9f377d4dab92c30b2213340ddb7eec0b597 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 11.28kB 0.0s
=> CACHED [2/6] RUN bundle config --global frozen 1 0.0s
=> CACHED [3/6] WORKDIR /usr/src/app/ 0.0s
=> CACHED [4/6] COPY Gemfile Gemfile.lock ./ 0.0s
=> CACHED [5/6] RUN bundle install 0.0s
=> [6/6] ADD . /usr/src/app/ 0.0s
=> exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:169bdf90d081e63bc5733eaa759b597c11822dc3399df60dcc2bdf72e5c704fa 0.0s
=> => naming to docker.io/library/hello-world-ruby:0.1.1 0.0s
I can now run it while passing in the vars
$ docker run -e NEW_RELIC_LICENSE_KEY=************************NRAL -e NEW_RELIC_APP_NAME='FBDemoApp (Development)' -p 4000:4000 hello-world-ruby:0.1.0
And hitting refresh a few times, I can now see that in APM
Some wrong paths.
I tried to follow guides to get Zipkin data to NewRelic’s endpoint while adding headers. The AI, guides and searches got me close
require 'sinatra'
require 'pg'
set :port, 4000
set :bind, '0.0.0.0'
get '/' do
begin
# Initialize connection variables.
host = String('192.168.1.78')
database = String('demoapp')
user = String('demoapp')
password = String('demopass')
# Initialize connection object.
connection = PG::Connection.new(:host => host, :user => user, :dbname => database, :port => '5432', :password => password)
puts 'Successfully created connection to database.'
resultSet = connection.exec('SELECT * from demotable;')
outStr = "<table><tr><th>id</th><th>name</th></tr>"
resultSet.each do |row|
puts 'Data row = (%s, %s)' % [row['id'], row['name']]
row_data = "<tr><td>#{row['id']}</td><td>#{row['name']}}</td></tr>"
outStr += "#{row_data}"
end
outStr += "</table>"
outStr
rescue PG::Error => e
puts e.message
ensure
connection.close if connection
end
end
require 'rack'
require 'zipkin-tracer'
ZIPKIN_TRACER_CONFIG = {
service_name: 'FBSampleApp (Zipkin1)',
sample_rate: 1.0,
json_api_host: 'https://trace-api.newrelic.com/trace/v1'
}.freeze
ZipkinTracer::Config.headers = {
'Content-Type' => 'application/json',
'Api-Key' => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxNRAL',
'Data-Format' => 'zipkin',
'Data-Format-Version' => '2'
}
ZipkinTracer::Config.setup do |config|
config.service_name = 'FBSampleApp (Zipkin)'
config.collector_host = 'https://trace-api.newrelic.com/trace/v1'
end
use ZipkinTracer::RackHandler, ZIPKIN_TRACER_CONFIG
However, that style of Headers just doesnt exist in zipkin tracer for me
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ docker run -p 4000:4000 hello-world-ruby:0.1.2
/usr/src/app/helloworldzip.rb:47:in `<main>': undefined method `headers=' for ZipkinTracer::Config:Class (NoMethodError)
ZipkinTracer::Config.headers = {
^^^^^^^^^^
This is where sending Zipkin trace data, whether it is via Faraday or Rack or other to just a local ‘zipkin’ endpoint is going to pay dividends.
We’ll be able to test locally with a zipkin instance (just running java and hitting localhost), then using an OTel client configured for NewRelic.
Using OTel Zipkin Exporters And the NewRelic Collector which they pulled from the common collectors - It used to be here
Seems it was dropped just over a year ago
Zipkin
My next goal is to fire up a local Zipkin instance so we can see Zipkin traces
$ cat helloworldzip.rb
require 'sinatra'
require 'pg'
set :port, 4000
set :bind, '0.0.0.0'
get '/' do
begin
# Initialize connection variables.
host = String('192.168.1.78')
database = String('demoapp')
user = String('demoapp')
password = String('demopass')
# Initialize connection object.
connection = PG::Connection.new(:host => host, :user => user, :dbname => database, :port => '5432', :password => password)
puts 'Successfully created connection to database.'
resultSet = connection.exec('SELECT * from demotable;')
outStr = "<table><tr><th>id</th><th>name</th></tr>"
resultSet.each do |row|
puts 'Data row = (%s, %s)' % [row['id'], row['name']]
row_data = "<tr><td>#{row['id']}</td><td>#{row['name']}}</td></tr>"
outStr += "#{row_data}"
end
outStr += "</table>"
outStr
rescue PG::Error => e
puts e.message
ensure
connection.close if connection
end
end
require 'rack'
require 'zipkin-tracer'
ZIPKIN_TRACER_CONFIG = {
service_name: 'FBSampleApp',
sample_rate: 1.0,
json_api_host: 'http://localhost:9411'
}.freeze
use ZipkinTracer::RackHandler, ZIPKIN_TRACER_CONFIG
I’ll want a local Zipkin which I can run with Docker compose
builder@DESKTOP-QADGF36:~/Workspaces$ git clone https://github.com/openzipkin/zipkin.git
Cloning into 'zipkin'...
remote: Enumerating objects: 211805, done.
remote: Counting objects: 100% (1126/1126), done.
remote: Compressing objects: 100% (245/245), done.
remote: Total 211805 (delta 856), reused 1101 (delta 844), pack-reused 210679
Receiving objects: 100% (211805/211805), 66.91 MiB | 25.24 MiB/s, done.
Resolving deltas: 100% (140600/140600), done.
builder@DESKTOP-QADGF36:~/Workspaces$ cd zipkin/docker/examples$
builder@DESKTOP-QADGF36:~/Workspaces/zipkin/docker/examples$ docker-compose -f docker-compose.yml -f docker-compose-ui.yml up
[+] Running 15/15
⠿ zipkin-ui Pulled 2.9s
⠿ ce92dcb96d54 Pull complete 0.7s
⠿ 18be8ff31039 Pull complete 0.9s
⠿ a0db17684027 Pull complete 1.0s
⠿ 0aa2486fd915 Pull complete 1.1s
⠿ e04a8c584b93 Pull complete 1.2s
⠿ 329c46775779 Pull complete 1.3s
⠿ zipkin Pulled 4.7s
⠿ b255216f6add Pull complete 1.2s
⠿ 5f6acd91b200 Pull complete 2.4s
⠿ 2e98d2dd5e93 Pull complete 2.5s
⠿ 0424e82c174b Pull complete 2.6s
⠿ 72a912b7d968 Pull complete 2.7s
⠿ cbe57ac687a3 Pull complete 2.8s
⠿ 7462568844af Pull complete 3.1s
[+] Running 3/2
⠿ Network examples_default Created 0.8s
⠿ Container zipkin Created 0.2s
⠿ Container zipkin-ui Created 0.1s
Attaching to zipkin, zipkin-ui
zipkin |
zipkin | oo
zipkin | oooo
zipkin | oooooo
zipkin | oooooooo
zipkin | oooooooooo
zipkin | oooooooooooo
zipkin | ooooooo ooooooo
zipkin | oooooo ooooooo
zipkin | oooooo ooooooo
zipkin | oooooo o o oooooo
zipkin | oooooo oo oo oooooo
zipkin | ooooooo oooo oooo ooooooo
zipkin | oooooo ooooo ooooo ooooooo
zipkin | oooooo oooooo oooooo ooooooo
zipkin | oooooooo oo oo oooooooo
zipkin | ooooooooooooo oo oo ooooooooooooo
zipkin | oooooooooooo oooooooooooo
zipkin | oooooooo oooooooo
zipkin | oooo oooo
zipkin |
zipkin | ________ ____ _ _____ _ _
zipkin | |__ /_ _| _ \| |/ /_ _| \ | |
zipkin | / / | || |_) | ' / | || \| |
zipkin | / /_ | || __/| . \ | || |\ |
zipkin | |____|___|_| |_|\_\___|_| \_|
zipkin |
zipkin | :: version 2.24.2 :: commit 3575817 ::
zipkin |
zipkin-ui | nginx: [warn] duplicate extension "woff2", content type: "application/font-woff2", previous content type: "font/woff2" in /etc/nginx/nginx.conf:35
zipkin | 2023-07-21 22:33:50:315 [armeria-boss-http-*:9411] INFO Server - Serving HTTP at /0.0.0.0:9411 - http://127.0.0.1:9411/
I now have Zipkin running
I had some challenges getting the docker running with $ docker run -p 4000:4000 hello-world-ruby:0.1.5 hitting the IP of the other. I tried local NATing, localhost, etc.
172.17.0.1 - - [22/Jul/2023:13:19:15 +0000] "GET / HTTP/1.1" 200 88 0.0290
E, [2023-07-22T13:19:15.204488 #1] ERROR -- : Error while connecting to http://localhost:9411: Faraday::ConnectionFailed with message 'Failed to open TCP connection to localhost:9411 (Cannot assign requested address - connect(2) for "localhost" port 9411)'. Please make sure the URL / port are properly specified for the Zipkin server.
But I mostly want to test the code, not docker.
So when I ran directly
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ ruby helloworldzip.rb
/home/builder/gems/gems/sinatra-3.0.6/lib/sinatra/base.rb:931: warning: constant Tilt::Cache is deprecated
W, [2023-07-22T08:20:01.429775 #19106] WARN -- : Using a boolean in the Sampled header is deprecated. Consider setting sampled_as_boolean to false
== Sinatra (v3.0.6) has taken the stage on 4000 for development with backup from Puma
Puma starting in single mode...
* Puma version: 6.3.0 (ruby 2.7.0-p0) ("Mugi No Toki Itaru")
* Min threads: 0
* Max threads: 5
* Environment: development
* PID: 19106
* Listening on http://0.0.0.0:4000
Use Ctrl-C to stop
Successfully created connection to database.
Data row = (1, Hello World)
127.0.0.1 - - [22/Jul/2023:08:20:06 -0500] "GET / HTTP/1.1" 200 88 0.0393
Successfully created connection to database.
Data row = (1, Hello World)
127.0.0.1 - - [22/Jul/2023:08:20:07 -0500] "GET / HTTP/1.1" 200 88 0.0190
Successfully created connection to database.
Data row = (1, Hello World)
127.0.0.1 - - [22/Jul/2023:08:20:08 -0500] "GET / HTTP/1.1" 200 88 0.0423
Successfully created connection to database.
Data row = (1, Hello World)
127.0.0.1 - - [22/Jul/2023:08:20:08 -0500] "GET / HTTP/1.1" 200 88 0.0159
Successfully created connection to database.
Data row = (1, Hello World)
127.0.0.1 - - [22/Jul/2023:08:20:10 -0500] "GET / HTTP/1.1" 200 88 0.0254
^C- Gracefully stopping, waiting for requests to finish
I could see traces
If we want a multi-service example, we can use this example repo
I can run the frontend.rb and backend.rb and give it a test
And, of course, we can see a dependency map
And since we have more interesting data, we can view specifics of the Trace
Kubernetes
There are a few helm charts out there such as ‘ygqygq2’ here and Openzipkin here.
We’ll use the later:
builder@DESKTOP-QADGF36:~/Workspaces/zipkin-ruby-example$ helm repo add openzipkin https://openzipkin.github.io/zipkin
"openzipkin" has been added to your repositories
builder@DESKTOP-QADGF36:~/Workspaces/zipkin-ruby-example$ helm search repo openzipkin
NAME CHART VERSION APP VERSION DESCRIPTION
openzipkin/zipkin 0.7.0 2.24.1 A Zipkin helm chart for kubernetes
I can now install
builder@DESKTOP-QADGF36:~/Workspaces/zipkin-ruby-example$ helm install zipkin openzipkin/zipkin
NAME: zipkin
LAST DEPLOYED: Sat Jul 22 08:35:44 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=zipkin,app.kubernetes.io/instance=zipkin" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
We can see the service was created for us
builder@DESKTOP-QADGF36:~/Workspaces/zipkin-ruby-example$ kubectl get svc zipkin
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
zipkin ClusterIP 10.43.138.139 <none> 9411/TCP 19s
I’ll now change the URL
ZIPKIN_TRACER_CONFIG = {
service_name: 'FBSampleApp',
sample_rate: 1.0,
json_api_host: 'http://zipkin.default.svc.cluster.local:9411'
}.freeze
Actually, let’s use an ENV Var like “ZIPKIN_URL”
require 'sinatra'
require 'pg'
set :port, 4000
set :bind, '0.0.0.0'
get '/' do
begin
# Initialize connection variables.
host = String('192.168.1.78')
database = String('demoapp')
user = String('demoapp')
password = String('demopass')
# Initialize connection object.
connection = PG::Connection.new(:host => host, :user => user, :dbname => database, :port => '5432', :password => password)
puts 'Successfully created connection to database.'
resultSet = connection.exec('SELECT * from demotable;')
outStr = "<table><tr><th>id</th><th>name</th></tr>"
resultSet.each do |row|
puts 'Data row = (%s, %s)' % [row['id'], row['name']]
row_data = "<tr><td>#{row['id']}</td><td>#{row['name']}}</td></tr>"
outStr += "#{row_data}"
end
outStr += "</table>"
outStr
rescue PG::Error => e
puts e.message
ensure
connection.close if connection
end
end
require 'rack'
require 'zipkin-tracer'
ZIPKIN_TRACER_CONFIG = {
service_name: 'FBSampleApp',
sample_rate: 1.0,
json_api_host: ENV["ZIPKIN_URL"]
}.freeze
use ZipkinTracer::RackHandler, ZIPKIN_TRACER_CONFIG
I’ll want to deploy this so best to build a version and push to my Harbor CR
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ docker tag hello-world-ruby:1.2.0 harbor.freshbrewed.science/library/hello-world-ruby:1.2.0
builder@DESKTOP-QADGF36:~/Workspaces/demo-containerized-app$ docker push harbor.freshbrewed.science/library/hello-world-ruby:1.2.0
The push refers to repository [harbor.freshbrewed.science/library/hello-world-ruby]
93c604d63b20: Pushed
8442fa2dfe02: Pushed
3bf0d91550b9: Pushed
54a0b3e9a828: Pushed
fc0015b23b27: Pushed
aa65f5b2fc19: Pushed
1573adcbec5d: Pushed
24deff53c342: Pushed
28218ecd8008: Pushed
2f66f3254105: Pushed
a72216901005: Pushed
61581d479298: Pushed
1.2.0: digest: sha256:47a329784730c94ca2e28155278963c3f6be494bdf315ff6355e8e178d26731f size: 2834
I can see now
We can pull the image
docker pull harbor.freshbrewed.science/library/hello-world-ruby@sha256:47a329784730c94ca2e28155278963c3f6be494bdf315ff6355e8e178d26731f
We can actually just create a deployment on the fly and expose port 4000 on a service
$ kubectl create deployment rubysample --image harbor.freshbrewed.science/library/hello-world-ruby:1.2
.0
deployment.apps/rubysample created
$ kubectl expose deployment rubysample --type ClusterIP --port 4000
service/rubysample exposed
I realized I should have added --env=ZIPKIN_URL=http://zipkin.default.svc.cluster.local:9411 in the kubectl deployment create command.
I can use kubectl edit deployment and add the absent block instead
containers:
- env:
- name: ZIPKIN_URL
value: "http://zipkin.default.svc.cluster.local:9411"
Or we can just run as a pod and skip deployments
$ kubectl run rubysample --image harbor.freshbrewed.science/library/hello-world-ruby:1.2.0 --restart=Always --env=ZIPKIN_URL="http://zipkin.default.svc.cluster.local:9411"
pod/rubysample created
I had to port-forward to zipkin on 9412 since I’m still binding to 9411 for my local zipkin in docker. That said, I can port-forward and see my traces in Zipkin in Kubernetes just as a had when using Docker
The traces aren’t very interesting, but it is fetching from an on-prem PostgreSQL and sending traces to the Zipkin service inside K8s
Open Telemetry (OTel)
Let’s now add Otel to the mix
I’ll install with ‘deployment’ mode
builder@DESKTOP-QADGF36:~/Workspaces/zipkin-ruby-example$ helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
"open-telemetry" already exists with the same configuration, skipping
builder@DESKTOP-QADGF36:~/Workspaces/zipkin-ruby-example$ helm install my-opentelemetry-collector open-telemetry/opentelemetry-collector --set mode=deployment
NAME: my-opentelemetry-collector
LAST DEPLOYED: Sat Jul 22 09:04:21 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
[INFO] as of chart version 0.47.0 the default collector configuration has been updated to use pod IP instead of 0.0.0.0 for its endpoints. See https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/UPGRADING.md#0460-to-0470 for details.
[WARNING] As of 0.54.0 Collector chart, the default resource limits are removed. See https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/UPGRADING.md#0531-to-0540 for details.
OTel covers the popeular tracing formats, including Zipkin. We can see the service is now running
$ kubectl get svc my-opentelemetry-collector
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-opentelemetry-collector ClusterIP 10.43.184.118 <none> 6831/UDP,14250/TCP,14268/TCP,4317/TCP,4318/TCP,9411/TCP 93s
Routing through Otel
What I want to do next is configure the OTel collector to recieve Zipkin (already set) and to the forward it the Zipkin service in the cluster (just act as a relay)
I’ll fetch the CM and add a section:
$ kubectl get cm my-opentelemetry-collector -o yaml > moc.cm.yaml
$ vi moc.cm.yaml
$ cat moc.cm.yaml
$ cat moc.cm.yaml
apiVersion: v1
data:
relay: |
exporters:
zipkin:
endpoint: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"
logging: {}
extensions:
health_check: {}
memory_ballast:
size_in_percentage: 40
processors:
batch: {}
memory_limiter:
check_interval: 5s
limit_percentage: 80
spike_limit_percentage: 25
receivers:
jaeger:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:14250
thrift_compact:
endpoint: ${env:MY_POD_IP}:6831
thrift_http:
endpoint: ${env:MY_POD_IP}:14268
otlp:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:4317
http:
endpoint: ${env:MY_POD_IP}:4318
prometheus:
config:
scrape_configs:
- job_name: opentelemetry-collector
scrape_interval: 10s
static_configs:
- targets:
- ${env:MY_POD_IP}:8888
zipkin:
endpoint: ${env:MY_POD_IP}:9411
service:
extensions:
- health_check
- memory_ballast
pipelines:
logs:
exporters:
- logging
processors:
- memory_limiter
- batch
receivers:
- otlp
metrics:
exporters:
- logging
processors:
- memory_limiter
- batch
receivers:
- otlp
- prometheus
traces:
exporters:
- logging
- zipkin
processors:
- memory_limiter
- batch
receivers:
- otlp
- jaeger
- zipkin
telemetry:
metrics:
address: ${env:MY_POD_IP}:8888
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"relay":"exporters:\n zipkin:\n endpoint: \"http://zipkin.default.svc.cluster.local:9411/api/v2/spans\"\n insecure: true\n logging: {}\nextensions:\n health_check: {}\n memory_ballast:\n size_in_percentage: 40\nprocessors:\n batch: {}\n memory_limiter:\n check_interval: 5s\n limit_percentage: 80\n spike_limit_percentage: 25\nreceivers:\n jaeger:\n protocols:\n grpc:\n endpoint: ${env:MY_POD_IP}:14250\n thrift_compact:\n endpoint: ${env:MY_POD_IP}:6831\n thrift_http:\n endpoint: ${env:MY_POD_IP}:14268\n otlp:\n protocols:\n grpc:\n endpoint: ${env:MY_POD_IP}:4317\n http:\n endpoint: ${env:MY_POD_IP}:4318\n prometheus:\n config:\n scrape_configs:\n - job_name: opentelemetry-collector\n scrape_interval: 10s\n static_configs:\n - targets:\n - ${env:MY_POD_IP}:8888\n zipkin:\n endpoint: ${env:MY_POD_IP}:9411\nservice:\n extensions:\n - health_check\n - memory_ballast\n pipelines:\n logs:\n exporters:\n - logging\n processors:\n - memory_limiter\n - batch\n receivers:\n - otlp\n metrics:\n exporters:\n - logging\n processors:\n - memory_limiter\n - batch\n receivers:\n - otlp\n - prometheus\n traces:\n exporters:\n - logging\n - zipkin\n processors:\n - memory_limiter\n - batch\n receivers:\n - otlp\n - jaeger\n - zipkin\n telemetry:\n metrics:\n address: ${env:MY_POD_IP}:8888\n"},"kind":"ConfigMap","metadata":{"annotations":{"meta.helm.sh/release-name":"my-opentelemetry-collector","meta.helm.sh/release-namespace":"default"},"creationTimestamp":"2023-07-22T14:04:21Z","labels":{"app.kubernetes.io/instance":"my-opentelemetry-collector","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"opentelemetry-collector","app.kubernetes.io/version":"0.81.0","helm.sh/chart":"opentelemetry-collector-0.62.2"},"name":"my-opentelemetry-collector","namespace":"default","resourceVersion":"519924","uid":"e1197248-35c4-4d97-a946-9559ffe3ff90"}}
meta.helm.sh/release-name: my-opentelemetry-collector
meta.helm.sh/release-namespace: default
creationTimestamp: "2023-07-22T14:04:21Z"
labels:
app.kubernetes.io/instance: my-opentelemetry-collector
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: opentelemetry-collector
app.kubernetes.io/version: 0.81.0
helm.sh/chart: opentelemetry-collector-0.62.2
name: my-opentelemetry-collector
namespace: default
resourceVersion: "533100"
uid: e1197248-35c4-4d97-a946-9559ffe3ff90
Then apply it
$ kubectl apply -f moc.cm.yaml
configmap/my-opentelemetry-collector configured
Then rotate the collector pod to have it take in the fresh CM
$ kubectl delete pod my-opentelemetry-collector-76dbb7d64b-rcsd4
pod "my-opentelemetry-collector-76dbb7d64b-rcsd4" deleted
Next, I’ll just launch a new pod that will forward to the Otel collector instead of Zipkin:
$ kubectl run rubysampleotel --image harbor.freshbrewed.science/library/hello-world-ruby:1.2.0 --restart=Always --env=ZI
PKIN_URL="http://my-opentelemetry-collector.default.svc.cluster.local:9411"
pod/rubysampleotel created
Now I just need to create a forwarder for the ruby sample app and the kubernetes zipkin to see the results:
$ kubectl port-forward pod/rubysampleotel 4000:4000
Forwarding from 127.0.0.1:4000 -> 4000
Forwarding from [::1]:4000 -> 4000
in another window
$ kubectl port-forward svc/zipkin 9412:9411
Forwarding from 127.0.0.1:9412 -> 9411
Forwarding from [::1]:9412 -> 9411
Handling connection for 9412
New Relic
The last mile is to send the traces to NewRelic as well
NewRelic abandoned their custom exporter in favour of using OTLP and packing the API key into the header.
We can see an example config here
Key is to set
exporters:
otlp:
endpoint: https://otlp.nr-data.net:4317
headers:
"api-key": $NEW_RELIC_API_KEY
We can pull down the CM, update and push it back up
$ kubectl get cm my-opentelemetry-collector -o yaml > moc.cm.yaml
$ vi moc.cm.yaml
$ cat moc.cm.yaml | head -n20
$ cat moc.cm.yaml
apiVersion: v1
data:
relay: |
exporters:
zipkin:
endpoint: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"
otlp:
endpoint: "https://otlp.nr-data.net:4317"
headers:
"api-key": **************************NRAL
logging: {}
extensions:
health_check: {}
memory_ballast:
size_in_percentage: 40
processors:
batch: {}
memory_limiter:
check_interval: 5s
limit_percentage: 80
spike_limit_percentage: 25
receivers:
jaeger:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:14250
thrift_compact:
endpoint: ${env:MY_POD_IP}:6831
thrift_http:
endpoint: ${env:MY_POD_IP}:14268
otlp:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:4317
http:
endpoint: ${env:MY_POD_IP}:4318
prometheus:
config:
scrape_configs:
- job_name: opentelemetry-collector
scrape_interval: 10s
static_configs:
- targets:
- ${env:MY_POD_IP}:8888
zipkin:
endpoint: ${env:MY_POD_IP}:9411
service:
extensions:
- health_check
- memory_ballast
pipelines:
logs:
exporters:
- logging
processors:
- memory_limiter
- batch
receivers:
- otlp
metrics:
exporters:
- logging
processors:
- memory_limiter
- batch
receivers:
- otlp
- prometheus
traces:
exporters:
- otlp
- logging
- zipkin
processors:
- memory_limiter
- batch
receivers:
- otlp
- jaeger
- zipkin
telemetry:
metrics:
address: ${env:MY_POD_IP}:8888
kind: ConfigMap
$ kubectl apply -f moc.cm.yaml
configmap/my-opentelemetry-collector configured
I’ll rotate the Otel pod to take effect
$ kubectl delete pods -l app.kubernetes.io/instance=my-opentelemetry-collector
pod "my-opentelemetry-collector-76dbb7d64b-s9z8m" deleted
I can now see traces in both Zipkin and NewRelic
Summary
Let’s review what we did here. We built out a Ruby app that can access PostgreSQL. The Database portion wasn’t that important, mostly just to slow up the app. We added the New Relic Gems and showed native direct tracing to New Relic APM.
Then, we setup tracing with zipkin-tracer and rack. We installed Zipkin locally as a docker container to show how to use tracing locally. We then deployed Zipkin with Helm into a cluster and deployed our ruby sample app with kubectl run.
We installed the OpenTelemetry into Kubernetes and showed it could be a forwarding service to Zipkin. Lastly, we added the New Relic OTLP endpoint and showed traces simultaneously going to Zipkin and NewRelic.
Benefits of this Model
It is easy enough to see how we can expand this to other systems. By setting up our containers to use Zipkin Traces we can:
- Test locally with Docker
- Create a Container once - it has a parameter for Zipkin endpoint - no more compiling with specific 3rd Party bindings - this avoids Vendor lock-in
- Test in Kubernetes without egressing to an APM (and incurring APM Costs)
- Change or Expand APM options by sending traces to the APM that makes the most sense
- Migrate to APMs - this, too, avoids Vendor lock-in
