HashiCorp Vault setup SSH Authentication with Ansible and Terraform

Most HashiCorp Vault tutorials - even those from HashiCorp - use the commandline tools or even just curl to configure HashiCorp Vault. But do we want to use curl when we can use terraform Code? Terraform already includes a HashiCorp Vault provider which enables us to do most tasks by using code and apply this to our installation - and maintain also our infrastructure.

We gonna use ansible to rollout the Vault SSH One Time Password authentication to all of our servers within the infrastructure. With this change HashiCorp Vault enables us to

• control who is able to access/authenticate on which machine/subnet
• how long those one time password are valid

So nobody has to copy&paste ssh keys or even worse mail the password of a machine over the network.

Configure HashiCorp Vault with terraform

Let's start with HashiCorp Vault configuration by terraform to provide an endpoint which provides one time passwords for the authentication on our infrastructure. The first code block defines our vault installation and creates a secret backend in vault of type ssh. This is mounted to the path ssh. This can be changed, but has to be adapted later on within the ansible code. Within the terraform code the backend path is passed through the items. As a default we set the lease time of a secret for 4 hours and a maximum of 1 week.

provider "vault" {
  address = "http://vault.local:8200/"
}

resource "vault_mount" "ssh" {
  type = "ssh"
  path = "ssh"

  default_lease_ttl_seconds = "14400"  # 4h
  max_lease_ttl_seconds     = "604800" # 1w

}

Next we gonna create a role within the ssh backend of type otp = One Time Password. Within this resource we set some parameters to allow only certains usernames on the connection - in this sample these are centos and ubuntu. We also allow connecting users to forward a port over this ssh connection and open a interactive shell on the servers. The secret backend role is also defined to be valid for a subnet based on cidr "192.168.0.0/24". And last but not least we limit the lifetime of the otp to 30 minutes.

variable "ssh_cidr" {
    default = "192.168.0.0/24"
}

resource "vault_ssh_secret_backend_role" "otp" {
  name     = "otp"
  backend  = vault_mount.ssh.path
  key_type = "otp"

  allowed_extensions = "permit-pty,permit-port-forwarding"
  default_user       = "centos"
  allowed_users      = "centos,ubuntu"
  cidr_list          = var.ssh_cidr

  ttl = "30m"

}

So that our clients can create an otp we gonna create and policy to allow them access to this endpoint:

resource "vault_policy" "ssh_client_access" {
  name   = "ssh_clients"
  policy = <<EOT
path "${vault_mount.ssh.path}/creds/otp" {
    capabilities = ["create", "read", "update"]
}
EOT
}

Next we can apply this change to our HashiCorp Vault:

1export VAULT_TOKEN="my-vault-token"
2terraform apply

At this point HashiCorp Vault is able to serve One Time Passwords for SSH Authentication and also allow our infrastructure to validate these passwords against HashiCorp Vault.

Ansible to rollout SSH One Time Password authentication

Ansible role

Ansible has already some community written HashiCorp vault roles, but none of them was up-to-date at the time of writing this article. So we forked a solution and updated this with some fixxes and changes to provide a solution for us. You can use this role by following ansible-galaxy command:

1ansible-galaxy install https://github.com/infralovers/ansible-vault-otp

The role uses the HashiCorp vault-ssh-helper in its core to reconfigure the infrastructure authentication mechanism. This helper validates the login prompt by ssh with our vault configuration that the provided password is a valid one time password provided by HashiCorp Vault.

Ansible rollout

To use this role you create a playbook which includes the role and sets the required variables:

1- hosts: all
2  become: yes
3  vars:
4  - vault_addr: http://vault.local:8200
5  - ssh_mount_point: ssh
6  roles:
7  - role: infralovers.vault_otp

When using an https endpoint you also have to set vault_ca_cert_file variable with a path to the certificate file.

Using HashiCorp Vault OTP

When consuming the OTP funcitonality we have to use curl to get this done - but we can write a function and add this content to e.g. .bashrc. The following function requests an one time password from HashiCorp Vault for a certain ip address or hostname. The function lookups the hostname within the ansible inventory. If this did not work the ssh configuration is used for a lookup on the ip address. And if this should not work either, a dns query is done to get the ip address of the host. As optional parameter also the username of the connection can be passed to the function - but it defaults to ubuntu.

 1function vault_otp() {
 2
 3    VAULT_ADDR="http://vault.local:8200"
 4    SSH_MOUNT="ssh"
 5    SSH_ROLE="otp"
 6    ANSIBLE_INVENTORY=${ANSIBLE_INVENTORY:-/full/path/to/hosts.ini}
 7    REMOTE_USER="ubuntu"
 8
 9    if [ -z "$VAULT_TOKEN" ]; then
10        echo "missing vault token!"
11        return
12    fi
13    HOSTNAME=$1
14    if [ -n "$2" ]; then
15        REMOTE_USER="$2"
16    fi
17    if [ -z "$VAULT_TOKEN" ]; then
18        echo "missing vault token!"
19        return
20    fi
21
22    IP=$(ansible-inventory -i "$ANSIBLE_INVENTORY" --host "$HOSTNAME" 2>/dev/null | jq -r '.ansible_host')
23
24    if [ -z "$IP" ]; then
25        IP=$(ssh -G "$HOSTNAME" | awk '$1 == "hostname" p{ print $2 }')
26    fi
27    CONFIGURED_USER=$(ssh -G "$HOSTNAME" | awk '$1 == "user" p{ print $2 }')
28    if [ -n "$CONFIGURED_USER" ]; then
29      REMOTE_USER="$CONFIGURED_USER"
30    fi
31    if [ -z "$IP" ]; then
32        IP=$HOSTNAME
33    fi
34
35    VALIDATE=$(dig +short "$IP")
36    if [ -n "$VALIDATE" ]; then
37        IP=$VALIDATE
38    fi
39    OTP=$( curl \
40            --header "X-Vault-Token: $VAULT_TOKEN" \
41            --request POST \
42            --data "{ \"username\": \"$REMOTE_USER\", \"ip\": \"${IP}\" }" \
43            $VAULT_ADDR/v1/$SSH_MOUNT/creds/$SSH_ROLE | jq -r '.data.key')
44
45    echo $OTP
46
47}

Afterwards you can use this one time password to open a connection to this host:

 1$ vault_otp my-server
 2d7a79b69-b3d6-ccce-9d2d-ce9a177b8e73
 3$ ssh ubuntu@my-server
 4Password: d7a79b69-b3d6-ccce-9d2d-ce9a177b8e73
 5Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1025-raspi aarch64)
 6
 7 * Documentation:  https://help.ubuntu.com
 8 * Management:     https://landscape.canonical.com
 9 * Support:        https://ubuntu.com/advantage
10
11  System information as of Wed Mar  3 10:16:00 UTC 2021
12
13  System load:  0.08               Temperature:           42.3 C
14  Usage of /:   10.6% of 29.05GB   Processes:             147
15  Memory usage: 45%                Users logged in:       0
16  Swap usage:   0%                 IPv4 address for eth0: 1.2.3.4
17
18 * Introducing self-healing high availability clusters in MicroK8s.
19   Simple, hardened, Kubernetes for production, from RaspberryPi to DC.
20
21     https://microk8s.io/high-availability
22
230 updates can be installed immediately.
240 of these updates are security updates.
25
26
27The list of available updates is more than a week old.
28To check for new updates run: sudo apt update
29
30Last login: Wed Mar  3 00:00:00 2021 from 127.0.0.1
31ubuntu at my-server in ~