In a cloud-native environment, secrets management is a critical aspect of security. HashiCorp Vault is a popular tool for managing secrets and protecting sensitive data. When combined with HashiCorp Nomad, an orchestration tool for deploying and managing applications, you can create a powerful and secure platform for running your workloads.
HashiCorp Vault provides a dynamic secrets engine that generates secrets on-demand. This feature allows you to create short-lived credentials for databases, cloud providers, and other services. By using dynamic secrets, you can reduce the risk of exposure and limit the lifespan of sensitive data.
In this example, we will deploy a simple web application using HashiCorp Nomad. The application requires access to a MySQL database, and we will use HashiCorp Vault to generate dynamic credentials for the database, use HashiCorp Vault transit engine to encrypt the database values on the fly, and finally use HashiCorp Consul for service discovery.
The acutal deployment of the MySQL database is done using the following Nomad job file. The MySQL database is started as a service and the root password is set as an environment variable.
1job "mysql-server" {
2 datacenters = ["dc1"]
3 type = "service"
4 namespace = "demo"
5
6 group "mysql-server" {
7 count = 1
8
9 service {
10 name = "mysql-server"
11 port = "db"
12 }
13
14 restart {
15 attempts = 10
16 interval = "5m"
17 delay = "25s"
18 mode = "delay"
19 }
20
21 task "mysql-server" {
22 driver = "docker"
23
24 env = {
25 "MYSQL_ROOT_PASSWORD" = "super-duper-password"
26 }
27
28 config {
29 image = "docker.io/mysql:9"
30 ports = ["db"]
31 }
32
33 resources {
34 cpu = 500
35 memory = 1024
36 }
37 }
38 network {
39 mode = "bridge"
40 port "db" {
41 static = 3306
42 }
43 }
44 }
45}
As a first step, we will deploy the web application with hardcoded credentials. That might be a workflow you are familiar with - start with less secure settings and improve them step by step. The web application is just a simple Python Flask application that connects to the MySQL database and displays the database values on a web page, and as a alternatively view it can display the plain database values.
Within the configuration of the application config.ini you can see the hardcoded database credentials. The application is started as a Nomad job and the configuration file is mounted into the container.
The configuration is file is also using the Consul template engine to get the database instance configuration from the Consul service discovery by the {{ range service "mysql-server" }} statement.
1job "dynamic-app" {
2 datacenters = ["dc1"]
3 type = "service"
4 namespace = "demo"
5
6 group "dynamic-app" {
7 count = 1
8
9 restart {
10 attempts = 10
11 interval = "5m"
12 delay = "25s"
13 mode = "delay"
14 }
15
16 service {
17 name = "dynamic-app"
18 port = "web"
19
20 check {
21 type = "http"
22 method = "GET"
23 interval = "10s"
24 timeout = "2s"
25 path = "/health"
26 }
27 }
28
29 task "dynamic-app" {
30 driver = "docker"
31
32 config {
33 image = "ghcr.io/infralovers/nomad-vault-mysql:1.0.0"
34 volumes = [
35 "local/config.ini:/usr/src/app/config/config.ini"
36 ]
37
38 ports = ["web"]
39 }
40
41 template {
42 destination = "local/config.ini"
43 data = <<EOF
44[DEFAULT]
45Port = 8080
46
47[DATABASE]
48{{ range service "mysql-server" }}
49Address = {{ .Address }}
50Port = {{ .Port }}
51{{end}}
52
53Database = my_app
54User = root
55Password = super-duper-password
56EOF
57 }
58 resources {
59 cpu = 256
60 memory = 256
61 }
62 }
63 network {
64 port "web" {
65 to = 8080
66 }
67 }
68 }
69}
The next step to improve our journey to get more secure deployments is to use the key value secret engine of HashiCorp Vault. This engine allows you to store and retrieve arbitrary secrets. In this example, we will store the database credentials in Vault and retrieve them at runtime.
1$ vault secrets enable -path=dynamic-app/kv kv
2$ vault kv put dynamic-app/kv/database username=root password=super-duper-password
To allow HashiCorp Nomad to access the secrets, you need to create a policy that grants read access to the dynamic-app/kv path. We have named this policy nomad-dynamic-app.
1path "dynamic-app/kv/database" {
2 capabilities = ["read"]
3}
And also our application deployment needs to be adapted- the configuration file is now using the Consul template engine to get the database credentials from the Vault key value store by the {{ with secret "dynamic-app/kv/database" }} statement.
1job "dynamic-app" {
2 datacenters = ["dc1"]
3 type = "service"
4 namespace = "demo"
5
6 group "dynamic-app" {
7 count = 1
8
9 vault {
10 policies = ["nomad-dynamic-app"]
11 change_mode = "signal"
12 change_signal = "SIGINT"
13 }
14
15 service {
16 name = "dynamic-app"
17 port = "web"
18
19 check {
20 type = "http"
21 method = "GET"
22 interval = "10s"
23 timeout = "2s"
24 path = "/health"
25 }
26 }
27
28 restart {
29 attempts = 10
30 interval = "5m"
31 delay = "25s"
32 mode = "delay"
33 }
34
35 task "dynamic-app" {
36 driver = "docker"
37
38 config {
39 image = "ghcr.io/infralovers/nomad-vault-mysql:1.0.0"
40 volumes = [
41 "local/config.ini:/usr/src/app/config/config.ini"
42 ]
43
44 ports = ["web"]
45 }
46
47 template {
48 destination = "local/config.ini"
49 data = <<EOF
50[DEFAULT]
51Port = 8080
52
53[DATABASE]
54{{ range service "mysql-server" }}
55Address = {{ .Address }}
56Port = {{ .Port }}
57{{end}}
58
59Database = my_app
60{{ with secret "dynamic-app/kv/database" }}
61User = {{ .Data.username }}
62Password = {{ .Data.password }}
63{{ end }}
64EOF
65 }
66 resources {
67 cpu = 256
68 memory = 256
69 }
70 }
71 network {
72 mode = "bridge"
73 port "web" {
74 to = 8080
75 }
76 }
77 }
78}
The almost final step in our journey is to use the dynamic secrets engine of HashiCorp Vault. This engine generates short-lived credentials for databases, cloud providers, and other services. In this example, we will use the MySQL database secrets engine to generate dynamic credentials for the database.
1$ vault secrets enable -path=dynamic-app/db database
2$ vault write dynamic-app/db/config/mysql \
3 plugin_name=mysql-database-plugin \
4 connection_url="{{username}}:{{password}}@tcp(mysql-server.service.consul:3306)/" \
5 allowed_roles="*" \
6 username="root" \
7 password="super-duper-password"
8$ vault write -force dynamic-app/db/database/rotate-root/mysql
9$ vault write dynamic-app/db/roles/my-role \
10 db_name=mysql \
11 creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT ALL ON my_app.* TO '{{name}}'@'%';" \
12 default_ttl="1h" \
13 max_ttl="24h"
And of course we need to adapt our Vault policy for HashiCorp Nomad to access the dynamic secrets engine:
1path "dynamic-app/kv/database" {
2 capabilities = ["read"]
3}
4path "dynamic-app/db/creds/app" {
5 capabilities = ["read"]
6}
And finally also the application deployment needs to be adapted, but this time we are using the dynamic secrets engine to get the database credentials from the Vault secrets store by the {{ with secret "dynamic-app/db/creds/app" }} statement. When comparing this actual deployment to the last one, you can see that it is just a minimal change in the deployment code to use the dynamic secrets engine - and you have a more secure deployment with adapting the path of the secrets store.
1job "dynamic-app" {
2 datacenters = ["dc1"]
3 type = "service"
4 namespace = "demo"
5
6 group "dynamic-app" {
7 count = 1
8
9 vault {
10 policies = ["nomad-dynamic-app"]
11 change_mode = "signal"
12 change_signal = "SIGINT"
13 }
14
15 service {
16 name = "dynamic-app"
17 port = "web"
18 }
19 restart {
20 attempts = 10
21 interval = "5m"
22 delay = "25s"
23 mode = "delay"
24 }
25
26 task "dynamic-app" {
27 driver = "docker"
28
29 config {
30 image = "ghcr.io/infralovers/nomad-vault-mysql:1.0.0"
31 volumes = [
32 "local/config.ini:/usr/src/app/config/config.ini"
33 ]
34
35 ports = ["web"]
36 }
37
38 template {
39 destination = "local/config.ini"
40 data = <<EOF
41[DEFAULT]
42Port = 8080
43
44[DATABASE]
45{{ range service "mysql-server" }}
46Address = {{ .Address }}
47Port = {{ .Port }}
48{{end}}
49
50Database = my_app
51{{ with secret "dynamic-app/db/creds/app" }}
52User = {{ .Data.username }}
53Password = {{ .Data.password }}
54{{ end }}
55EOF
56 }
57 resources {
58 cpu = 256
59 memory = 256
60 }
61 }
62 network {
63 port "web" {
64 to = 8080
65 }
66 }
67 }
68}
The final step in our journey is to use the transit engine of HashiCorp Vault to encrypt the database values on the fly. This engine provides a way to encrypt and decrypt data without storing the encryption keys. In this example, we will use the transit engine to encrypt the database credentials before storing them in the database.
1$ vault secrets enable -path=dynamic-app/transit transit
2$ vault write -f dynamic-app/transit/keys/app
And again we need to adapt our Vault policy for HashiCorp Nomad to access the transit engine:
1path "dynamic-app/kv/database" {
2 capabilities = ["read"]
3}
4path "dynamic-app/db/creds/app" {
5 capabilities = ["read"]
6}
7path "dynamic-app/transit/encrypt/app" {
8 capabilities = ["create", "update"]
9}
10path "dynamic-app/transit/decrypt/app" {
11 capabilities = ["create", "update"]
12}
This functionality is an additional security layer to encrypt the database values on the fly, but also requires code changes that your application is able to interact with HashiCorp Vault transit engine on a code level. The previous changes in the adaption to HashiCorp Vault are just changes in the deployment code, but this did not require code changes in your application.
Also, the application deployment needs to be adapted, but this time we need to add the configuration section of [VAULT] to use the transit engine to encrypt the database values by the KeyPath = dynamic-app/transit and KeyName = app statements.
1job "dynamic-app" {
2 datacenters = ["core"]
3 type = "service"
4 namespace = "demo"
5
6 group "dynamic-app" {
7 count = 1
8
9 vault {
10 policies = ["nomad-dynamic-app"]
11 change_mode = "signal"
12 change_signal = "SIGINT"
13 }
14
15 service {
16 name = "dynamic-app"
17 port = "web"
18
19 check {
20 type = "http"
21 method = "GET"
22 interval = "10s"
23 timeout = "2s"
24 path = "/health"
25 }
26 }
27
28 restart {
29 attempts = 10
30 interval = "5m"
31 delay = "25s"
32 mode = "delay"
33 }
34
35 task "dynamic-app" {
36 driver = "docker"
37
38 config {
39 image = "ghcr.io/infralovers/nomad-vault-mysql:1.0.0"
40 volumes = [
41 "local/config.ini:/usr/src/app/config/config.ini"
42 ]
43
44 ports = ["web"]
45 }
46
47 template {
48 destination = "local/config.ini"
49 data = <<EOF
50[DEFAULT]
51Port = 8080
52
53[DATABASE]
54{{ range service "mysql-server" }}
55Address = {{ .Address }}
56Port = {{ .Port }}
57{{end}}
58
59Database = my_app
60{{ with secret "dynamic-app/db/creds/app" }}
61User = {{ .Data.username }}
62Password = {{ .Data.password }}
63{{ end }}
64
65[VAULT]
66Enabled = True
67InjectToken = True
68Namespace =
69Address = {{ env "VAULT_ADDR" }}
70KeyPath = dynamic-app/transit
71KeyName = app
72EOF
73 }
74 resources {
75 cpu = 256
76 memory = 256
77 }
78 }
79 network {
80 port "web" {
81 to = 8080
82 }
83 }
84 }
85}
In this article, we have demonstrated how to use HashiCorp Vault and Nomad to deploy a secure web application with dynamic secrets. By using dynamic secrets, you can reduce the risk of exposure and limit the lifespan of sensitive data. This approach provides a secure and scalable platform for running your workloads in a cloud-native environment.
And even if you are going to transform your deployments by the use of the Key Value Secret Engine you have a good starting point to improve your security settings step by step. The next step is to use the dynamic secrets engine to generate short-lived credentials for your services. And as demonstrated within the example - it is just a minor change in your actual deployment code to use the dynamic secrets engine.
You find all the code examples in the GitHub repository to reproduce the example on your own. As a starting point, you can use the HashiCorp Nomad cluster on the cloud instructions to deploy a Nomad cluster on your own, which is packed by Consul cluster and a Vault cluster.
If you are interested in learning more about HashiCorp Vault and Nomad, check out our Cloud Native Essentials and HashiCorp Vault Enterprise courses. These courses will help you master the tools and techniques needed to build secure and scalable cloud-native applications.
Introduction to HashiCorp Vault.
In previous posts we have already shown multiple ways to use HashiCorp Packer to build Golden Images. In this post we will show how to automate the process with
In dynamic Kubernetes environments, managing DNS records can be a challenge. As services scale, redeploy, or change IP addresses, ensuring that DNS records
In today's rapidly evolving cloud-native landscape, the need for efficient service orchestration, service discovery, and secure communication between services
Securing Communication on HashiCorp Nomad Payloads with HashiCorp Consul Connect In modern cloud-native environments, security is paramount. One of the key
The rise of AI in DevOps has been causing a revolution in how we approach software development and operations. We're witnessing a shift in our industry, where
You are interested in our courses or you simply have a question that needs answering? You can contact us at anytime! We will do our best to answer all your questions.
Contact us