Aiven Part 2: Redis and AWS Cloudwatch Integrations

Published: Nov 29, 2022 by Isaac Johnson

In our first post we explored to create and manage a hosted Kafka cluster as well as a MirrorMaker instance. We explored sending metrics to Datadog and using MirrorMaker to sync with an on-prem cluster as well as Azure Event Hub.

Today we will explore using Aiven for Redis, and sending Logs from Kafka and Redis to GCP Log Monitor. We’ll then look at AWS CloudWatch for both Metrics and Logs.

Aiven Redis

The first step is to go into services and create a Redis Service


Our next step will be to try and connect via a Redis CLI

Since I really don’t care to install a local redis server, I’ll just build from source:

builder@DESKTOP-72D2D9T:~/Workspaces$ git clone
Cloning into 'redis'...
remote: Enumerating objects: 85618, done.
remote: Counting objects: 100% (31/31), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 85618 (delta 9), reused 9 (delta 0), pack-reused 85587
Receiving objects: 100% (85618/85618), 121.71 MiB | 7.64 MiB/s, done.
Resolving deltas: 100% (62172/62172), done.
builder@DESKTOP-72D2D9T:~/Workspaces$ cd redis/
builder@DESKTOP-72D2D9T:~/Workspaces/redis$ git checkout 3.0
Branch '3.0' set up to track remote branch '3.0' from 'origin'.
Switched to a new branch '3.0'
builder@DESKTOP-72D2D9T:~/Workspaces/redis$ make redis-cli
cd src && make redis-cli
make[1]: Entering directory '/home/builder/Workspaces/redis/src'
... snip

Then fix the perms and link it. We can use ‘-v’ to test

builder@DESKTOP-72D2D9T:~/Workspaces/redis$ chmod 755 src/redis-cli
builder@DESKTOP-72D2D9T:~/Workspaces/redis$ ln -s /home/builder/Workspaces/redis/src/redis-cli /usr/local/bin/redis-cli
builder@DESKTOP-72D2D9T:~/Workspaces/redis$ redis-cli -v
redis-cli 3.0.7 (git:48e24d54)

I actually found troubles connecting with the 3.0 and the 6.0.16 version of redis-cli. But the latest 7.x seems to work fine

builder@DESKTOP-72D2D9T:~/Workspaces/redis$ redis-cli -u 'rediss://'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.>


builder@DESKTOP-72D2D9T:~/Workspaces/redis$ redis-cli -u 'rediss://'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.> set TEST VAL1234
(3.96s)> get TEST
(2.10s)> exit

Next, I’ll send Metrics to Datadog


GCP Logging

We need to ensure the Logging API is enabled


Then create a service account


I’m likely going overkill, but I’m going to add to the Logging Admin Role and Editor role. I really want roles/logging.logWriter and roles/editor but the GCP UI is rather lacking


I’ll want to get the service key JSON next


from there I can create a new key


choose JSON


This then downloads it locally. We can now use that in the Integrations setup window


We can now pick that in our Integrations section of the service


We can now see we are sending Logs to GCP and Metrics to DD for Redis


and I can do the same for Kafka


We can almost immediately see the logs show up in StackDriver


We can see an example of the kind of data being sent into GCP Logging


One thing you may want to do to avoid costs building up is to set a log storage retention policy.

We can go to Logs Storage in Logging (the popup upselling advert is GCPs not mine)


Then we can “edit bucket” from the 3-dot menu


and set a value lower if you need to


We can also create a Sink from GCP Logging if we want to send to pub/sub (to send to Datadog) or directly into Splunk


Scaling a service

We saw how to create a service in the UI, but what if we want to scale up or down our instance?

We just need to change the plan


For instance, I can move from teh US$200/mo plan for Redis down to the smallest business $70/mo


The plans, for Redis, at the moment go from US$19/mo


Up to $8150/mo for the largest in GCP


And again, these prices are by month and I believe (but do not know) that pricing is hourly. Thus, if we wanted to fire up the largest Redis for a test run, an hour would be roughly US$11.16 for that hour which isn’t bad if testing a system under load.

Cloudwatch logs

Before we can send logs to Cloudwatch, we need an IAM user setup

In AWS IAM, I’ll create a user with only programmitic access


The policies you pick largely depend on what you plan to do with this IAM user. If you want to just send logs, CloudWatchLogsFullAccess would do it. If you want Metrics, use CloudWatchEventsFullAccess. If you think you may want both, just use CloudWatchFullAccess.


I’ll just use all three and create my user


You can then get your IAM ID and Key


Lastly, you’ll want to determine the region. For no particular reason, I tend to use us-east-1


Back in, we’ll add the AWS Cloudwatch Logs Endpoint


Since I’m here, I’ll also handly the CW Metrics


I’ll pick a service, such as Kafka, and click Manage Integrations


I’ll then add Cloudwatch Logs and Metrics


When adding Cloudwatch Metrics, you can chose which Metrics you want to send to AWS Cloudwatch (by default, all are enabled)


Immediately, I saw Aiven logs appear as a new Group in AWS


You’ll see that log streams have been added retroactively for past logs - not just logs going forward


If I explore one to see the messages being delivered


That means I can use the logs in Logs Insights to parse and find certain types


As always, I like to point out that logs that are infinitely stored can really start to cost over time. I recommend going to the retention settings:


and reducing to the minimum duration you care about


AWS Cloud Metrics

As you recalled, like logs, we sent Metrics to AWS as well

We can see those in Cloud Metrics


With Metrics, I can create graphs, such as bytes received by Kafka


Perhaps tweaking the style or adding annotations


In time, we can see plenty of data gathered from Aiven



Today we looked at setting up a hosted Redis instance in We setup an IAM user and showed how to set up AWS CloudWatch for Metrics and Logs. We demonstrated integration logs and metrics into AWS and also showed how to minimize costs.

We can now use CloudWatch with SNS to trigger SNS alerts or Lambdas.

In our next blog, we’ll wrap the series looking at Grafana, InfluxDB for monitors and alerts.

aiven redis aws cloudwatch

Have something to add? Feedback? Try our new forums

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes