Secure Communication between HashiCorp Nomad and Traefik with Consul Connect

In today's rapidly evolving cloud-native landscape, the need for efficient service orchestration, service discovery, and secure communication between services is critical. HashiCorp Nomad, Consul Connect, and Traefik make a powerful combination that simplifies service deployment, discovery, and load balancing with built-in security features. This post will explore how these tools can be used together to efficiently connect and manage services in a dynamic environment.

Overview

• HashiCorp Nomad: A simple yet powerful scheduler designed for deploying and managing containerized and non-containerized applications at scale.
• HashiCorp Consul Connect: A service mesh solution that enables secure service-to-service communication by utilizing mutual TLS (mTLS). It ensures that only authorized services communicate with one another.
• Traefik: A modern HTTP reverse proxy and load balancer that is dynamic, flexible, and integrates well with a variety of orchestration and service discovery tools like Consul.

By integrating these technologies, we can achieve efficient service orchestration, secure service mesh capabilities, and intelligent traffic routing in a dynamic cluster environment.

Why Combine Nomad, Consul Connect, and Traefik?

When running services in a distributed environment, you need more than just an orchestrator. Here’s why each component is essential:

1. Nomad’s Orchestration: Nomad enables you to run jobs across multiple nodes with ease. It handles both containerized and legacy workloads, making it a versatile tool for deployment.

2. Consul Connect’s Security: Consul provides secure service-to-service communication using mTLS, meaning all services communicating through Consul Connect are authenticated and encrypted.

3. Traefik’s Load Balancing: Traefik integrates with Consul’s service discovery to act as an ingress controller and load balancer for your Nomad jobs, routing traffic intelligently based on real-time service information.

This stack eliminates the complexity of managing different layers for service discovery, load balancing, and security by offering a seamless, integrated approach.

How It All Fits Together

We are following our examples we already used in previous blog posts to deploy workload with HashiCorp Vault secured secrets and secure communication between services with Consul Connect.

Step 1: Setting Up HashiCorp Nomad for Service Orchestration

To begin, you need to deploy a Nomad cluster. Nomad is lightweight and can run on any infrastructure, from bare metal to Kubernetes. Once Nomad is running, you can define jobs using its HCL-based job specification format. These jobs describe how and where applications should be deployed.

Here is our updated dynamic-app Nomad job that includes Consul Connect sidecar proxies. This job deploys a simple web service that connects to a MySQL database and uses Vault to retrieve database credentials securely. The service is registered with Consul Connect for secure communication. We also added Traefik tags to make the service available through Traefik - the load balancer and ingress controller.

 1job "dynamic-app" {
 2  datacenters = ["dc1"]
 3  type        = "service"
 4  namespace   = "demo"
 5
 6  group "dynamic-app" {
 7    count = 1
 8
 9    restart {
10      attempts = 10
11      interval = "5m"
12      delay    = "25s"
13      mode     = "delay"
14    }
15
16    network {
17      mode = "bridge"
18    }
19
20    vault {
21      policies      = ["nomad-dynamic-app"]
22      change_mode   = "signal"
23      change_signal = "SIGINT"
24    }
25
26
27    service {
28      name = "dynamic-app"
29      port = "8080"
30      tags = ["traefik.enable=true",
31        "traefik.http.routers.dynamic-app.rule=Host(`dynamic-app.127.0.0.1.nip.io`)",
32        "traefik.http.routers.dynamic-app.entrypoints=http",
33        "traefik.http.routers.dynamic-app.tls=false",
34        "traefik.connsulcatalog.connect=true"
35      ]
36      connect {
37        sidecar_service {
38          proxy {
39            upstreams {
40              destination_name = "mysql-server"
41              local_bind_port  = 3306
42            }
43          }
44        }
45      }
46      check {
47        expose   = true
48        type     = "http"
49        name     = "heatlh"
50        method   = "GET"
51        interval = "10s"
52        timeout  = "2s"
53        path     = "/health"
54      }
55    }
56
57    task "dynamic-app" {
58      driver = "docker"
59
60      config {
61        image = "ghcr.io/infralovers/nomad-vault-mysql:1.0.0"
62        volumes = [
63          "local/config.ini:/usr/src/app/config/config.ini"
64        ]
65        ports = [8080]
66      }
67
68      template {
69        destination = "local/config.ini"
70        data        = <<EOF
71    [DEFAULT]
72    LogLevel = DEBUG
73    Port = 8080
74    [DATABASE]
75    Address = 127.0.0.1
76    Port = 3306
77
78    Database = my_app
79    {{ with secret "dynamic-app/db/creds/app" }}
80    User = {{ .Data.username }}
81    Password = {{ .Data.password }}
82    {{ end }}
83
84    [VAULT]
85    Enabled = True
86    InjectToken = True
87    Namespace =
88    Address = {{ env "VAULT_ADDR" }}
89    KeyPath = dynamic-app/transit
90    KeyName = app
91EOF
92      }
93      resources {
94        cpu    = 256
95        memory = 256
96      }
97    }
98  }
99}

This Nomad job runs an dynamic-app web service and registers it with Consul Connect for secure communication.

Step 2: Secure Service Communication with Consul Connect

Consul Connect allows services registered in Consul to communicate securely via mTLS. In the example job above, we configured a Consul Connect sidecar proxy for the web service. This proxy handles encryption and decryption of traffic between services.

Step 3: Traefik as the Load Balancer and Ingress Controller

Traefik integrates seamlessly with Consul for service discovery and routing. In our setup, Traefik will automatically pick up services registered in Consul and route traffic to them.

Here’s a Traefik configuration snippet for Consul integration:

 1entryPoints:
 2  web:
 3    address: ":80"
 4
 5providers:
 6  consulCatalog:
 7    endpoint:
 8      address: "127.0.0.1:8500"
 9    exposedByDefault: false
10    connectAware: true
11    connectByDefault: true
12
13api:
14  dashboard: true
15  insecure: true

With this configuration, Traefik will act as a reverse proxy and load balancer for all services registered in Consul, including those deployed with Nomad.

You must add tags to your Nomad job to make services available through Traefik. For example:

1tags = ["traefik.enable=true", "traefik.http.routers.web.rule=Host(`example.com`)"]

Traefik will now recognize this service and route HTTP traffic based on the defined rules.

Because we are using Consul Connect, Traefik will also be aware of the secure communication between services and route traffic accordingly, but we need to enable the connect tag in the tag declaration.

1tags = ["traefik.enable=true", "traefik.http.routers.web.rule=Host(`example.com`)", "traefik.connsulcatalog.connect=true"]

Within the traefik configuration you need to define the connectAware and connectByDefault options to make Traefik aware of the Consul Connect services.

1providers:
2  consulCatalog:
3    endpoint:
4      address: "127.0.0.1:8500"
5    exposedByDefault: false
6    connectAware: true
7    connectByDefault: true

Step 4: Putting It All Together

Once you’ve deployed your Nomad jobs, Consul will automatically register the services and Traefik will pick them up for load balancing. Consul Connect ensures that all inter-service communication is encrypted and authenticated.

In summary:

• Nomad deploys and manages the services.
• Consul Connect secures the communication between services.
• Traefik acts as the load balancer and routes incoming traffic to the appropriate services.

Benefits of this Setup

• Scalability: Easily scale services using Nomad’s dynamic orchestration.
• Security: Consul Connect ensures encrypted, authenticated communication between services, mitigating man-in-the-middle attacks.
• Dynamic Load Balancing: Traefik dynamically adjusts traffic routing based on the real-time state of services discovered via Consul.

Conclusion

The combination of HashiCorp Nomad, Consul Connect, and Traefik offers a powerful and flexible solution for deploying, discovering, securing, and routing services in a dynamic environment. Whether you are running a small or large-scale infrastructure, this stack simplifies the complexity of managing service communication, load balancing, and security, allowing you to focus on building reliable and scalable applications.

By leveraging these tools together, you can streamline the process of deploying and managing services with ease while ensuring robust security and performance.