Blue/Green Deployments with HashiCorp Nomad and Waypoint

What is HashiCorp Nomad?

HashiCorp Nomad is not only a lightweight alternative to Kubernetes. It's simple to use, easy to setup and very scalable. I use it on many customer projects and in my home-lab to host applications big and small. You could say it is my favourite container orchestrator.

There is one downside though: When I deploy an application on Nomad, that uses a blue/green deployment approach, I need to manually interact with Nomad to switch to the newly deployed version. If there is something that I do not like in a DevOps and Cloud Native world, it is doing stuff manually!

Don't get me wrong: Nomad fully supports blue/green deployments. The process could just be more...automated.

NOTE: In case you don not know what a blue/green deployment is have a look at this link: https://en.wikipedia.org/wiki/Blue-green_deployment

What is HashiCorp Waypoint?

This is where HashiCorp Waypoint comes in! Waypoint is a tool that can structure and automate your deployment pipelines. It comes with a bunch of integration for different container orchestrators like Kubernetes, Amazon ECS and of course HashiCorp Nomad.

What I will show you in this post

Today, I want to show an example blue/green deployment done with HashiCorp Nomad and Waypoint where all steps are automated. No manual steps necessary! Ok...maybe we will use one manual command to start the pipeline but other than that: All automated. Especially that blue/green version switch in Nomad!

NOTE: We could remove that last manually step by integrating Waypoint with a CI tool like GitLab CI or GitHub Actions. However, let's keep it simple for this example

What you will need

A running Nomad cluster with Docker and Waypoint installed on it. You can find a great step-by-step guide here: https://developer.hashicorp.com/waypoint/tutorials/get-started-nomad/get-started-install

The application

We will deploy a simple NodeJS web app that will be serving a static website on port 8080. All we need to do to prepare this application for deployments, is to add a Dockerfile to the directory. Waypoint will see the Dockerfile and build it during it's build phase.

You can find an example of the NodeJS app we deployed right here: https://github.com/commandemy/colors-app

The app shows you specific info about your browser and can have a different background color. Very good to showcase different versions of the same app as you can e.g. switch the background color between deployments.

However, this will work with any valid app and Dockerfile. Just be sure to adjust the port in your Nomad job template if your application serves under a different port.

The Nomad job

The Nomad job for this application is very simple. The important bit here is that we set the canary argument. This will instruct Nomad to keep the old version of the deployment until the new version has successfully deployed (blue/green deployment). Otherwise, Nomad would delete the old deployment before starting the new one.

 1# colors-app.nomad.tpl
 2job "colors" {
 3  datacenters = ["dc1"]
 4  group "app" {
 5    update {
 6      max_parallel = 1
 7      canary       = 1
 8      auto_revert  = true
 9      auto_promote = false
10      health_check = "task_states"
11    }
12
13    task "app" {
14      driver = "docker"
15      config {
16        image = "${artifact.image}:${artifact.tag}"
17      }
18
19      env {
20        PORT = "8080"
21      }
22    }
23  }
24}

Additionally, we set the health_check to task_states. This means that Nomad will identify a new deployment as "healthy" if the deployed task becomes running. In a real world scenario we might want to exchange this for a more meaningful health check. However, for this example it will do just fine.

Notice that we do net set an explicit container image or tag in this file. Instead, we use ${artifact.image}:${artifact.tag}. These placeholders will be automatically filled out by Waypoint with the most recently built image.

The Waypoint project

The Waypoint HCL contains three stages for our deployment:

• build: Builds our Docker image from a Dockerfile
• deploy: To deploy the image onto Nomad
• release: Switch to new version once it has been deployed successfully

I am a big fan of the release stage here. As mentioned earlier, I do not like that I have to run $ nomad deployment promote to tell Nomad that it should use the new version from now on. I want this to happen automatically once an automated test shows that the new version is healthy. We can make that happen with the release stage of this Waypoint project and the health_check config in the Nomad job specification.

We just need to use the nomad-jobspec-canary stanza from Waypoint's Nomad plugin. You can find the documentation here: https://www.waypointproject.io/plugins/nomad

 1# waypoint.hcl
 2project = "colors-app"
 3
 4app "colors-app" {
 5  build {
 6    use "docker" {
 7      buildkit = false
 8      disable_entrypoint = false
 9    }
10    registry {
11      use "docker" {
12        image = "colors-app"
13        tag   = "latest"
14        local = true
15      }
16    }
17  }
18
19  deploy {
20    use "nomad-jobspec" {
21      jobspec = templatefile("${path.app}/colors-app.nomad.tpl")
22    }
23  }
24
25  release {
26    use "nomad-jobspec-canary" {
27      groups = [
28        "app"
29      ]
30    }
31  }
32}

Running the deployment

Once you have all the files setup, you just switch into the directory which contains the waypoint.hcl file and run $ waypoint init:

 1$ waypoint init
 2
 3✓ Configuration file appears valid
 4✓ Connection to Waypoint server was successful
 5✓ Project "colors-app" and all apps are registered with the server.
 6✓ Project "colors-app" pipelines are registered with the server.
 7
 8Project initialized!
 9
10You may now call 'waypoint up' to deploy your project or
11commands such as 'waypoint build' to perform steps individually.

You will see the project listed on your Waypoint dashboard:

Let's start the deployment!

 1$ waypoint up
 2
 3» Performing operation locally
 4
 5» Building colors-app...
 6✓ Running build v2
 7✓ Initializing Docker client...
 8✓ Building image...
 9 │  ---> Using cache
10 │  ---> da0f57f57968
11 │ Step 4/5 : COPY index.html /
12 │  ---> Using cache
13 │  ---> 5c907ab74170
14 │ Step 5/5 : ENTRYPOINT [ "/usr/local/bin/node", "/colors.js" ]
15 │  ---> Using cache
16 │  ---> 55f3fb91662f
17 │ Successfully built 55f3fb91662f
18 │ Successfully tagged waypoint.local/colors-app:latest
19✓ Injecting Waypoint Entrypoint...
20Image built: waypoint.local/colors-app:latest (amd64)
21✓ Running push build v2
22✓ Tagging Docker image: waypoint.local/colors-app:latest => colors-app:1
23✓ Docker image pushed: colors-app:1
24
25» Deploying colors-app...
26✓ Running deploy v2
27✓ Job registration successful
28✓ Allocation "1c2cc492-1895-9f9e-21ae-816a6ee006df" created: node "79e79364-264f-e159-5e79-69122471e2f0", group "app"
29✓ Allocation "1c2cc492-1895-9f9e-21ae-816a6ee006df" status changed: "pending" -> "running" (Tasks are running)
30✓ Evaluation status changed: "pending" -> "complete"
31✓ Evaluation "256b8490-0a00-b0e8-6ffd-4ab99a057b25" finished with status "complete"
32✓ Deployment successfully rolled out!
33
34✓ Finished building report for Nomad platform
35✓ Getting job info...
36✓ Job "colors" is reporting ready!
37
38» Releasing colors-app...
39✓ Running release v2
40✓ Evaluation status changed: "pending" -> "complete"
41✓ Evaluation "2522b165-dc86-dac6-3afc-3cfc7b96400a" finished with status "complete"
42✓ Release successfully rolled out!
43
44✓ Finished building report for Nomad platform
45✓ Finished building report for Nomad job resource
46
47» Variables used:
48  VARIABLE | VALUE | TYPE | SOURCE
49-----------+-------+------+---------
50
51
52The deploy was successful! A Waypoint deployment URL is shown below. This
53can be used internally to check your deployment and is not meant for external
54traffic. You can manage this hostname using "waypoint hostname."
55
56           URL: https://globally-healthy-krill.waypoint.run
57Deployment URL: https://globally-healthy-krill--v2.waypoint.run

Waypoint builds the image, deploys it to Nomad and then checks for the new version to become active before triggering the promote inside of Nomad.

NOTE: You might get an error message if you deploy for the first time as there is no "old version" that can be replaced. Once you run the deployment again, all should work just fine.

We also get a nifty URL to access our deployment in the browser:

That's it. Super easy automated blue/green deployment.

Improving this example

However, there is one thing I would want to optimize for a professional project: I would not want to build a new Docker image every time this pipeline runs and I also do not want to trigger Waypoint manually!

I would create Git repository and connect it to an Continuous Integration system like GitLab CI or GitHub Actions. This repo would contain the app and the Dockerfile. Whenever a change happens in the repo, a new Docker image is being built and pushed to a container registry. I would NOT use Waypoint to build the image here.

Once the image is built and in a registry, I would trigger the waypoint up command. The build stage would only download the recently built image, but not actually build it. The deploy and release stages would be the same as in the example above.

Why? This way, you can re-trigger deployments without re-building the same image over and over again. It also give you the advantage of storing the image in a Docker registry which might take advantage of security scans etc.

Another improvement would be to keep the "old version" alive until a 3rd version is being deployed. This would make rolling-back quicker. However, Nomad does not seem to support this feature...yet!