Published: May 14, 2024 by Isaac Johnson

Today we’ll continue our Rundeck review with part two. We’ll look at Commands and SCM connectivity for DR. In fact, we’ll test restoring using a second Rundeck instance. We’ll look at Ansible in Rundeck and then running Ansible via the REST API. Lastly, we’ll touch on the commercial product (that may be no more).

Let’s dig in!

Commands

An easy way to just run some commands against hosts is to use the “Commands” area.

Here we can pick a node pattern (or all nodes) and then just run a command.

Here it is in action:

SCM

Next, let’s export our Job Definitions to an external SCM (GIT repo).

In Project settings, we can choose to “Setup SCM”

Next, I’ll choose “Setup” under “Git Export”

I could use an HTTPS password file then https:

I also could use SSH or FTP.

First, however, I want to create an identity in Forgejo just for this work.

In Identity and Access, I’ll “Create User Account”

I’ll create a “rundeck” user

Next, I’ll create a new repository

I’ll call the new repo “rundeckbackup”

Because I created this under ‘builderadmin’, I need to add the Rundeck user as a collaborator

And grant that user Administrator (for force push)

Even though I told it to “create the branch”

It seems to fail with no existing branch

I’ll work past this by creating a README.md

and commit to main

This time setup worked

I created a quick job but noted nothing was pushed. I then tried to update an existing and “push to scm”

But got an error

I set the committer name and email (to match the Rundeck user we created)

Then tried a fresh push

This time it seemed to work

In GIT, I could see the created build definition for NewTest

<joblist>
  <job>
    <defaultTab>nodes</defaultTab>
    <description></description>
    <dispatch>
      <excludePrecedence>true</excludePrecedence>
      <keepgoing>false</keepgoing>
      <rankOrder>ascending</rankOrder>
      <successOnEmptyNodeFilter>false</successOnEmptyNodeFilter>
      <threadcount>1</threadcount>
    </dispatch>
    <executionEnabled>true</executionEnabled>
    <id>f0caa71c-913b-430f-8ead-46f16969604d</id>
    <loglevel>INFO</loglevel>
    <name>NewTest</name>
    <nodeFilterEditable>false</nodeFilterEditable>
    <nodefilters>
      <filter>mac.*</filter>
    </nodefilters>
    <nodesSelectedByDefault>true</nodesSelectedByDefault>
    <plugins />
    <scheduleEnabled>true</scheduleEnabled>
    <sequence keepgoing='false' strategy='node-first'>
      <command>
        <exec>whoami</exec>
      </command>
    </sequence>
    <uuid>f0caa71c-913b-430f-8ead-46f16969604d</uuid>
  </job>
</joblist>

Import

I’ll go to my alternate Rundeck instance, login and go to Project Settings.

There I’ll get into the “Setup SCM” section to “Setup” a GIT Import

I’ll use the same GIT URL as before which solves authentication for the Private repo

I can now see it’s imported

When I then went to the jobs, I found an odd error that would suggest I cannot import

I tried some of the steps listed in https://github.com/rundeck/rundeck/issues/8218 but to no avail.

That said, I can import a job using the XML definition directly.

We can see it imported. The warning denotes this other Rundeck host has no “mac81” node

Ansible

We can create a job with an Ansible playbook as a workflow step.

We just need to add the playbook as plain YAML and any extra variables (like those we might set in the Tower/AWX Template definition)

This simple job just does a whoami to show an example

- name: Test
  hosts: all

  tasks:

  - name: QuickTest
    command: whoami
    args:
      chdir: /tmp

I’m going to guess on settings here as I see no place to really define my hosts or inventory. I’ll set the auth I know will work on ‘mac81’ and that I want to ‘become’, or rather - escalate privilege (sudo)

To test, I’ll seek to run on all nodes - I would expect it might work on mac81 and not on the rundeck built-in

When I try and run, I see that it skipped rundeck (local server) and errored on ‘mac81’

I’m going to add ansible to the mac81 host using

$ sudo apt-add-repository ppa:ansible/ansible
$ sudo apt update && sudo apt install ansible


... snip ...

Setting up python3-requests-kerberos (0.12.0-2) ...
Setting up python3-requests-ntlm (1.1.0-1.1) ...
Setting up python3-babel (2.8.0+dfsg.1-7) ...
update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode
Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ...
Setting up python3-winrm (0.3.0-2) ...
Setting up ansible-core (2.16.5-1ppa~jammy) ...
Setting up ansible (9.5.1-1ppa~jammy) ...
Processing triggers for man-db (2.10.2-1) ...

builder@anna-MacBookAir:~$ which ansible-playbook
/usr/bin/ansible-playbook

Generally, one would then setup /etc/ansible/hosts next but I’ll see what happens if we just provide a valid binary

I still saw the error

Failed: IOFailure: ERROR: Ansible IO failure: Cannot run program "ansible-playbook" (in directory "/tmp/ansible-rundeck3006190326939488760"): error=2, No such file or directory

The more I read about this, the more it looks like the ansible binaries need to live on the rundeck instance.

I decided to try and just add the ansible binary to the pod - this obviously would not survive a pod rotate so we would likely need to bake a new container image, but still - it’s worth seeing if this would unblock us with an apt get update && apt get -y install ansible

Setting up python3-ntlm-auth (1.4.0-1) ...
Setting up python3-pycryptodome (3.11.0+dfsg1-3ubuntu0.1) ...
Setting up python3-kerberos (1.1.14-3.1build5) ...
Setting up python3-yaml (5.4.1-1ubuntu1) ...
Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ...
Setting up python3-packaging (21.3-1) ...
Setting up python3-chardet (4.0.0-1) ...
Setting up python3-cryptography (3.4.8-1ubuntu2.2) ...
Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ...
Setting up python3-requests-kerberos (0.12.0-2) ...
Setting up ansible (2.10.7+merged+base+2.10.8+dfsg-1) ...
Setting up python3-requests-ntlm (1.1.0-1.1) ...
Setting up python3-requests-toolbelt (0.9.1-1) ...
Setting up python3-libcloud (3.2.0-2) ...
Setting up python3-winrm (0.3.0-2) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
root@rundeck-7cb68459bf-vssv8:~# history | tail -n 2
    4  apt -y install ansible
    5  history | tail -n 2
root@rundeck-7cb68459bf-vssv8:~#

This time it succeeded

This seemed to skip localhost thus showing ansible ran but didn’t really do anything nor run the playbook I gave it.

I think the key issue is I did not set up the ansible inventory via the ad-hoc node executor.

To do that, we go to project settings/edit configuration

Under “Default Node Executor” change the drop down from SSHJ to ‘Ansible Ad-Hoc Node Executor’

I’ll chose to change the Executable, config path and to Generate Inventory, but for our issue, it’s that last setting that is holding us up

The thing is, there are than one way to solve this need

Ansible with REST

Ansible AWX has a rich REST API I use often for Job creation and other setups.

Let’s start by creating a user we can use with Rundeck in AWX

I now have a regular user

I noticed the user lacked access to Projects, Jobs and Templates.

So my first step was to add to the project

I then gave Use and Read permissions to one Template

I can test the REST API by using CURL. I should only see one Job Template so that should make it easy

builder@LuiGi:~/Workspaces/jekyll-blog$ curl -X GET -u "rundeck:ThisIsNotMyPassword" https://awx.freshbrewed.science/api/v2/job_templates/ | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4972  100  4972    0     0   9050      0 --:--:-- --:--:-- --:--:--  9040
{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "id": 7,
      "type": "job_template",
      "url": "/api/v2/job_templates/7/",
      "related": {
        "created_by": "/api/v2/users/1/",
        "modified_by": "/api/v2/users/1/",
        "labels": "/api/v2/job_templates/7/labels/",
        "inventory": "/api/v2/inventories/1/",
        "project": "/api/v2/projects/6/",
        "organization": "/api/v2/organizations/1/",
        "credentials": "/api/v2/job_templates/7/credentials/",
        "last_job": "/api/v2/jobs/163/",
        "next_schedule": "/api/v2/schedules/6/",
...snip ...

I can use POST to kick out a fresh run of that Template

$ curl -X POST -u "rundeck:ThisIsNotMyPassword" https://awx.freshbrewed.science/api/v2/job_templates/7/launch/ | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2901  100  2901    0     0   2755      0  0:00:01  0:00:01 --:--:--  2757
{
  "job": 166,
  "ignored_fields": {},
  "id": 166,
  "type": "job",
  "url": "/api/v2/jobs/166/",
  "related": {
    "created_by": "/api/v2/users/3/",
    "modified_by": "/api/v2/users/3/",
    "labels": "/api/v2/jobs/166/labels/",

which I can see invoked

Now, back in Rundeck, let’s add a Job that can do the same

I’ll use the curl i did before in the workflow step

I’ll tell it to invoke locally

In fact, we can see a live run below:

Enterprise

I wanted to try Rundeck Enterprise, however I didn’t really see much from their docs page

I had to jump through a few pages, but in the end, I found that, essentially, Rundeck Enterprise has become (or is becoming) Pagerduty Process Automation

I like Pagerduty, I do, but I sure hope they keep Rundeck community around even if they are rebranding the Enterprise offering.

In fact, the email I got when I requested a trial of RD Enterprise was for PD Process Automation:

Summary

Today we continued our Rundeck journey and addressed how to use Commands to run arbitrary steps on hosts and how the SCM connectivity works (leveraging GIT with Forgejo). We looked at imports of the XML files stored in GIT (using an alternate host) then wrapped by looking at running Ansible two ways.