Securing Communication on HashiCorp Nomad Payloads with HashiCorp Consul Connect

In modern cloud-native environments, security is paramount. One of the key challenges is ensuring secure communication between microservices. Whether you’re running a monolithic application or a set of microservices on HashiCorp Nomad, securing inter-service communication is crucial to prevent unauthorized access and ensuring data privacy. Enter HashiCorp Consul Connect, a powerful service mesh solution that allows you to secure communication between services without the need to modify your application code.

In this blog post, we’ll explore how you can secure the communication of your workloads on HashiCorp Nomad using HashiCorp Consul Connect, and why it’s an essential component of your service mesh strategy.

What is HashiCorp Consul Connect?

Consul Connect is part of HashiCorp's Consul service mesh, providing secure service-to-service communication in your infrastructure. It achieves this by enabling mutual Transport Layer Security (mTLS) between services. Each service gets its own identity, and Consul enforces that only trusted services can communicate with each other. By using Connect, service communication is authenticated and encrypted, ensuring that the network’s security is maintained even in hostile or compromised environments.

Key Features of Consul Connect:

• mTLS (Mutual TLS): Ensure that all service-to-service communication is encrypted, and both the client and server verify each other’s identity.
• Service Identity: Each service registered in Consul has a unique identity, which can be used for fine-grained access control.
• Authorization Policies: You can define policies that control which services are allowed to communicate, adding an extra layer of security.
• Automatic Certificate Rotation: Consul manages certificate lifecycle, ensuring that certificates are renewed and rotated automatically to reduce operational overhead.

Why Consul Connect for Nomad?

Nomad is a highly scalable, flexible orchestrator for deploying and managing applications. However, while it schedules and runs workloads efficiently, it lacks native networking security features like encrypted communication between services. This is where Consul Connect comes in. By integrating Nomad with Consul Connect, you can add zero-trust security to your workloads, ensuring secure communication between microservices, even in complex, multi-cloud, or hybrid environments.

Some key reasons to integrate Consul Connect with Nomad:

• Seamless Integration: Nomad has built-in support for Consul Connect, allowing you to easily secure your applications with minimal configuration.
• Simplified Networking: Consul Connect abstracts away complex network configurations, using sidecars to automatically manage encrypted connections.
• Improved Security: All communications between services are encrypted and authenticated, reducing the risk of man-in-the-middle attacks.

Setting Up Consul Connect with Nomad

Prerequisites:

We are following the example we started with securing secrets in HashiCorp Nomad payload with the use of HashiCorp Vault. If you haven't read it yet, you can find it here.

Step 1: Enable Consul Connect in Consul and Nomad

To use Consul Connect, you must enable Consul Connect within your Consul Cluster config.json by adding the following section:

1{
2  "connect": {
3    "enabled": true
4  }
5}

Afterward, you first need to ensure that Nomad is integrated with Consul. In your Nomad server configuration file (nomad.hcl), enable the Consul integration by adding the following section:

1consul {
2  address = "127.0.0.1:8500"  # Address of the Consul agent
3  client_auto_join = true     # Auto-join Consul clients to the service mesh
4  token = "<CONSUL_ACL_TOKEN>" # Optional: ACL token for secure Consul communication
5}

Step 2: Configure the Service to Use Connect - MySQL Database

When defining a job in Nomad, you can enable Consul Connect for each task by adding a connect block. Here’s an example of a job definition with Consul Connect enabled:

 1job "mysql-server" {
 2  datacenters = ["dc1"]
 3  type        = "service"
 4  namespace   = "demo"
 5
 6  group "mysql-server" {
 7    count = 1
 8
 9    service {
10      name = "mysql-server"
11      port = "db"
12      connect {
13        sidecar_service {}
14      }
15    }
16
17    restart {
18      attempts = 10
19      interval = "5m"
20      delay    = "25s"
21      mode     = "delay"
22    }
23
24    task "mysql-server" {
25      driver = "docker"
26      env = {
27        "MYSQL_ROOT_PASSWORD" = "super-duper-password"
28      }
29      config {
30        image = "mysql:9"
31        ports = ["db"]
32      }
33      resources {
34        cpu    = 500
35        memory = 500
36      }
37    }
38    network {
39      mode = "bridge"
40      port "db" {
41        static = 3306
42        to     = 3306
43      }
44    }
45  }
46}

Our database service is now available via Consul Connect, and all communication will be encrypted and authenticated. This is happening with a sidecar proxy that is automatically injected into the task group. Within the sidecar proxy, the necessary certificates are managed and the connection to the database is established.

Step 3: Configure the Service to Use Connect - dynamic-app Service

 1job "dynamic-app" {
 2  datacenters = ["dc1"]
 3  type        = "service"
 4  namespace   = "demo"
 5
 6  group "dynamic-app" {
 7    count = 1
 8
 9    restart {
10      attempts = 10
11      interval = "5m"
12      delay    = "25s"
13      mode     = "delay"
14    }
15
16    network {
17      mode = "bridge"
18      port "dynamic-app" {
19         to = 8080
20      }
21    }
22
23    vault {
24      policies      = ["nomad-dynamic-app"]
25      change_mode   = "signal"
26      change_signal = "SIGINT"
27    }
28
29    service {
30      name = "dynamic-app"
31      port = "dynamic-app"
32
33      connect {
34        sidecar_service {
35          proxy {
36            upstreams {
37              destination_name = "mysql-server"
38              local_bind_port  = 3306
39            }
40          }
41        }
42      }
43      check {
44        type     = "http"
45        name     = "health"
46        method   = "GET"
47        interval = "10s"
48        timeout  = "2s"
49        path     = "/health"
50      }
51    }
52
53    task "dynamic-app" {
54      driver = "docker"
55      config {
56        image = "docker.io/mabunixda/dynamic-vault-app"
57        volumes = [
58          "local/config.ini:/usr/src/app/config/config.ini"
59        ]
60        ports = [ "dynamic-app"]
61      }
62
63      template {
64        destination = "local/config.ini"
65        data        = <<EOF
66    [DEFAULT]
67    LogLevel = DEBUG
68    Port = 8080
69    [DATABASE]
70    Address = 127.0.0.1
71    Port = 3306
72
73    Database = my_app
74    {{ with secret "dynamic-app/db/creds/app" }}
75    User = {{ .Data.username }}
76    Password = {{ .Data.password }}
77    {{ end }}
78
79    [VAULT]
80    Enabled = True
81    InjectToken = True
82    Namespace =
83    Address = {{ env "VAULT_ADDR" }}
84    KeyPath = dynamic-app/transit
85    KeyName = app
86EOF
87      }
88      resources {
89        cpu    = 256
90        memory = 256
91      }
92    }
93  }
94}

In this example:

• The dynamic-app service is set up to use a sidecar proxy that forwards requests to the mysql-server service via Consul Connect.
• The mysql-server service also has a sidecar proxy, but no upstreams are defined since it only accepts requests from the dynamic-app service.

Step 4: Define Intentions for Service Communication

In Consul Connect, intentions define which services are allowed to communicate. By default, no services can communicate, so you must create an intention to allow communication between your services.

For example, to allow the dynamic-app service to communicate with the mysql-server service, you can run the following command:

1consul intention create -allow dynamic-app mysql-server

This command creates a policy that permits the dynamic-app service to connect to the mysql-server service, while any other service attempting to access mysql-server will be blocked unless explicitly allowed.

Step 5: Observe the Secure Communication

Once your job is running, Nomad and Consul will automatically handle the secure communication. The sidecar proxies for the dynamic-app and mysql-server services will establish a secure mTLS connection. You can verify the encrypted communication by observing the logs or by inspecting the traffic between the proxies.

Additionally, you can use Consul’s UI to monitor the health and connections between your services.

Benefits of Using Consul Connect with Nomad

1. End-to-End Encryption: All communication between services is encrypted using mTLS, safeguarding your data in transit.
2. Zero Trust Networking: You can enforce policies that define exactly which services can talk to each other, following the principle of least privilege.
3. Dynamic and Scalable: Consul Connect scales with your infrastructure, handling service registration, discovery, and connectivity dynamically as your workload grows.
4. Simple Integration: Consul Connect integrates seamlessly with Nomad, reducing complexity while improving security.

Conclusion

Securing communication between services is a fundamental aspect of building resilient and secure distributed systems. By leveraging HashiCorp Consul Connect with HashiCorp Nomad, you can ensure that your service mesh follows security best practices without needing to modify application code. The combination of mTLS, service identity, and access control policies provides robust security while maintaining flexibility and scalability in your infrastructure.

Start enhancing your deployments today by integrating Consul Connect, and take advantage of secure, authenticated, and encrypted communication across your services.