Terraform with Terratest in Gitlab Pipeline

Martin Buchleitner | 18.12.2019 Devops, HashiCorp, Cloud Native, CI/CD

Terraform with terratest in Gitlab pipeline

With terraform Cloud and terraform Enterprise you are enabled now to us your custom modules in a way that all in your organization or team can use the same building blocks and must not reinvent all functionality. But that also takes more into account that those modules always do what they are used for - you should have your tests in place to ensure the behaviour of the module over time. At this point, Terratest comes on the stage.

Terratest

Terratest is a Go library that makes it easier to write automated tests for your infrastructure code. Terratest was developed at Gruntwork to help maintain the Infrastructure as Code Library, which code is written in Terraform, Go, Python, and Bash. Terratest is written in Go and also all tests must be written in Go.

Terratest provides a collection of helper functions and patterns for common infrastructure testing tasks, like making HTTP requests and using SSH to access a specific virtual machine. The following list describes some of the major advantages of using Terratest:

It provides convenient helpers to check infrastructure. This feature is useful when you wasnt to verify your real infrastructure in the real environment.
The folder structure is clearly organized. Your test cases are organized clearly and follow the standard Terraform module folder structure.
All test cases are written in Go. Most developers who use Terraform are Go developers. If you're a Go developer, you don't have to learn another programming language to use Terratest. Also, the only dependencies that are required for you to run test cases in Terratest are Go and Terraform.
The infrastructure is highly extensible. You can extend additional functions on top of Terratest

Terratest is designed for integration tests. For that purpose, Terratest provisions real resources in a real environment. Sometimes, integration test jobs can become exceptionally large, especially when you have a large number of resources to provision.

So You should take into account that all tests should always refer to a test account in your cloud provider(s) so that no production code - or even staging environments are not influenced by your development efforts of a terraform module which is under test with Terratest!

Sample AWS Route53 module

Within this post, we gonna try to test a module which encapsulates the creation of subdomains within AWS Route53. Its basic code looks like this:

 1data "aws_route53_zone" "target_zone" {
 2  name = var.domain
 3}
 4
 5resource "aws_route53_record" "target_record" {
 6  depends_on = [null_resource.module_dependency]
 7  zone_id    = data.aws_route53_zone.target_zone.zone_id
 8  name       = "${var.subdomain}.${var.domain}"
 9  type       = var.record_type
10  ttl        = var.record_ttl
11  records    = [var.record_ip]
12}

Unit testing with Terratest

Thanks to the flexibility of Terratest, we can use unit tests. Unit tests are local running test cases (although internet access is required). Unit test cases execute terraform init and terraform plan commands to parse the output of terraform plan and look for the attribute values to compare.

To start with Terratest you must init your terraform module path as a Go module with the current snipped and create a path test for your upcoming tests

1go mod init $(basename $PWD)
2mkdir -p test

In the following unit test, we want to test if a subdomain gets created and nothing gets destroyed, but the zone entry should only be loaded as a data source and never be a resource in our module. The zone entry must be created and maintained in another module.

 1package test
 2
 3import (
 4  "encoding/json"
 5  "fmt"
 6  "path"
 7  "testing"
 8
 9  "github.com/gruntwork-io/terratest/modules/terraform"
10  tfPlan "github.com/hashicorp/terraform/plans/planfile"
11)
12
13
14const domain = "testing.infralovers.com"
15
16type awsRoute53 struct {
17  subdomain     string
18  record_type   string
19  record_ip     string
20}
21
22// Test cases for storage account name conversion logic
23var testCases = map[string]awsRoute53{
24  "terratest.testing.infralovers.com": awsRoute53{subdomain: "terratest", record_type: "A", record_ip: "127.0.0.1"},
25}
26
27func TestUT_AWSRoute53(t *testing.T) {
28  t.Parallel()
29
30  for expected, input := range testCases {
31    // Specify the test case folder and "-var" options
32    tfOptions := &terraform.Options{
33      TerraformDir: "../",
34      Vars: map[string]interface{}{
35        "subdomain":    input.subdomain,
36        "domain":       domain,
37        "record_type":  input.record_type,
38        "retcord_ip":   input.record_ip,
39      },
40    }
41
42    // Terraform init and plan only
43    tfPlanOutput := "terraform.tfplan"
44    terraform.Init(t, tfOptions)
45    terraform.RunTerraformCommand(t, tfOptions, terraform.FormatArgs(tfOptions, "plan", "-out="+tfPlanOutput)...)
46    tfOptions.Vars = nil
47
48      // Read and parse the plan output
49    reader, err := tfPlan.Open(path.Join(tfOptions.TerraformDir, tfPlanOutput))
50    if err != nil {
51      t.Fatal(err)
52    }
53    defer reader.Close()
54    plan, _ := reader.ReadPlan()
55    if plan.Changes.Empty() {
56      t.Fatal("Empty plan outcome")
57      continue
58    }
59    fmt.Printf("Checking %s...", expected)
60    for _, res := range plan.Changes.Resources {
61      if res.ChangeSrc.Action.String() != "Create" {
62        t.Errorf("Found an action which is not create: %s", res.ChangeSrc.Action.String())
63        continue
64      }
65      if res.Addr.String() == "aws_route53_record.target_record" {
66        // do some fancy checks ...
67      }
68    }
69  }
70}

Go developers probably will notice that the unit test matches the signature of a classic Go test function by accepting an argument of type *testing.T.

With this code, we can now run the following command to check if all resources are just generated and none are updated or even destroyed. We can do this by the following command

1go test ./test/

The test will show no problems, but it does not take into account if the correct dns record is generated. To also verify this we gonna add the following code to our terraform module

1output "dns" {
2  value = aws_route53_record.target_record.fqdn
3}

And we must also modify now our test to verify the output in the plan

 1package test
 2
 3import (
 4  "encoding/json"
 5  "path"
 6  "testing"
 7
 8  "github.com/gruntwork-io/terratest/modules/terraform"
 9  tfPlan "github.com/hashicorp/terraform/plans/planfile"
10)
11
12func getJsonMap(m map[string]interface{}, key string) map[string]interface{} {
13  raw := m[key]
14  sub, ok := raw.(map[string]interface{})
15  if !ok {
16    return nil
17  }
18  return sub
19}
20
21func TestUT_AWSRoute53(t *testing.T) {
22  t.Parallel()
23
24  for expected, input := range testCases {
25    // Specify the test case folder and "-var" options
26    tfOptions := &terraform.Options{
27      TerraformDir: "../",
28      Vars: map[string]interface{}{
29        "subdomain":   input.subdomain,
30        "domain":      domain,
31        "target_type": input.record_type,
32        "target_ip":   input.record_ip,
33      },
34    }
35
36    // init and plan
37    tfPlanOutput := "terraform.tfplan"
38    terraform.Init(t, tfOptions)
39    terraform.RunTerraformCommand(t, tfOptions, terraform.FormatArgs(tfOptions, "plan", "-out="+tfPlanOutput)...)
40    tfOptions.Vars = nil
41
42    // read the plan as json
43    jsonplan, err := terraform.RunTerraformCommandAndGetStdoutE(t, tfOptions, terraform.FormatArgs(tfOptions, "show", "-json", tfPlanOutput)...)
44    jsonMap := make(map[string]interface{})
45    err = json.Unmarshal([]byte(jsonplan), &jsonMap)
46    if err != nil {
47      panic(err)
48    }
49    planned := getJsonMap(jsonMap, "planned_values")
50    outputs := getJsonMap(planned, "outputs")
51    dns := getJsonMap(outputs, "dns")
52    actual := dns["value"]
53    if expected != actual {
54      t.Errorf("Planned dns output is not valid: %s, expected: %s", actual, expected)
55    }
56    // Read and parse the plan output
57    reader, err := tfPlan.Open(path.Join(tfOptions.TerraformDir, tfPlanOutput))
58    if err != nil {
59      t.Fatal(err)
60    }
61    defer reader.Close()
62    plan, _ := reader.ReadPlan()
63    if plan.Changes.Empty() {
64      t.Fatal("Empty plan outcome")
65      continue
66    }
67
68    for _, res := range plan.Changes.Resources {
69      if res.ChangeSrc.Action.String() != "Create" {
70        t.Errorf("Found an action which is not create: %s", res.ChangeSrc.Action.String())
71        continue
72      }
73      if res.Addr.String() == "aws_route53_record.target_record" {
74        // do some fancy checks ...
75      }
76    }
77  }
78}

In the above code, the plan is read as json file because the internal processing of the terraform plan is made by go-cty which created some weird types which must be converted and can cause a headache.

Integration testing with Terratest

For integration testing, the test will go one step further and really create resources - and also destroy those afterwards. The actual test code is smaller because now we can read the output of the terraform applied code to verify the generated dns record

 1package test
 2
 3import (
 4  "testing"
 5
 6  "github.com/gruntwork-io/terratest/modules/terraform"
 7)
 8
 9// Test the Terraform module in examples/complete using Terratest.
10func TestIT_AWSRoute53(t *testing.T) {
11  t.Parallel()
12
13  for expected, input := range testCases {
14    // Specify the test case folder and "-var" options
15    tfOptions := &terraform.Options{
16      TerraformDir: "../",
17      Vars: map[string]interface{}{
18        "subdomain":   input.subdomain,
19        "domain":      domain,
20        "target_type": input.dnstype,
21        "target_ip":   input.target,
22      },
23    }
24
25    defer terraform.Destroy(t, tfOptions)
26
27    // Terraform init and plan only
28    terraform.InitAndApply(t, tfOptions)
29
30    actual := terraform.Output(t, tfOptions, "dns")
31
32    if actual != expected {
33      t.Errorf("Expect %v, but found %v", expected, actual)
34    }
35
36  }
37}

When running this test now, make sure you already defined the aws provider credentials as environment variables.

1go test ./test/ -run "TestIT_"

This time will only run integration tests with the command line above and they should create a dns record and also by the defer terraform.Destroy() destroy it after the complete test has run through. At this point our written tests are fine and our code is ok. But you also want to run those tests within a pipeline in Gitlab.

Pitfalls at Integration testing

If we add another test to our input, the test will succeed in the first run but will fail in the upcoming

1// Test cases for storage account name conversion logic
2var testCases = map[string]awsRoute53{
3  "terratest.testing.infralovers.com": awsRoute53{subdomain: "terratest", record_type: "A", record_ip: "127.0.0.1"},
4  "cnamtest.testing.infralovers.com": awsRoute53{subdomain: "cnamtest", record_type: "CNAME", record_ip: "terratest.testing.infralovers.com"},
5}

At the current test code, we will produce resources, which will not be destroyed because the destroy process just uses the latest created terraform plan! In our example now only the CNAME record will be removed correctly, the A type record will still exist. This is of course not a problem of Terraform or Terratest, it is the simple example code which produces this behaviour!

Keep in mind to verify that your integration tests always clean up correctly. Otherwise, they will fail and eventually produce also costs!

Terratest in a Gitlab Pipeline

 1terratest:
 2  stage: test
 3  image:
 4    name: "hashicorp/terraform:full"
 5    entrypoint:
 6      - "/usr/bin/env"
 7      - "PATH=/go/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
 8  script:
 9    - go test ./test/ -run "TestUT_" # running unit tests first
10    - go test ./test/ -run "TestIT_" # and afterwards integration tests

A full gitlab Pipeline for terraform and terratest

In the following code snippet is a gitlab pipeline which also validates and lints the containing code. In this example also shellcheck is running for some scripts in the module and the terraform is linted by tflint. The testing is done within an advanced version using mage by the following magefile.

 1stages:
 2  - validate
 3  - lint
 4  - test
 5
 6validate:
 7  stage: validate
 8  image:
 9    name: "hashicorp/terraform:0.12.8"
10    entrypoint:
11      - "/usr/bin/env"
12      - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
13  script:
14    - terraform init
15    - terraform validate
16  artifacts:
17    paths:
18      - .terraform
19
20scriptlint:
21  stage: lint
22  image:
23    name: "koalaman/shellcheck-alpine"
24  script:
25    - shellcheck scripts/*
26
27terralint:
28  stage: lint
29  image:
30    name: "wata727/tflint"
31    entrypoint:
32      - "/usr/bin/env"
33      - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
34  script:
35    - tflint
36
37terratest:
38  stage: test
39  image:
40    name: "hashicorp/terraform:full"
41    entrypoint:
42      - "/usr/bin/env"
43      - "PATH=/go/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
44  script:
45    - currdir=$(pwd)
46    - apk add --no-cache gcc libc-dev bind-tools
47    - go get -u -d github.com/magefile/mage
48    - magedir=$(find /go/pkg/mod/ -name magefile -type d | grep -v cache)
49    - cd $magedir/$(ls $magedir/)
50    - go run bootstrap.go
51    - cd $currdir
52    - mage full

The testing is done with in a advanced version using mage by the following magefile.

 1package main
 2
 3import (
 4  "fmt"
 5  "os"
 6  "path/filepath"
 7
 8  "github.com/magefile/mage/mg"
 9  "github.com/magefile/mage/sh"
10)
11
12// The default target when the command executes `mage` in Cloud Shell
13var Default = Full
14
15// A build step that runs Clean, Format, Unit and Integration in sequence
16func Full() {
17  mg.Deps(Unit)
18  mg.Deps(Integration)
19}
20
21// A build step that runs unit tests
22func Unit() error {
23  mg.Deps(Clean)
24  mg.Deps(Format)
25  fmt.Println("Running unit tests...")
26  return sh.RunV("go", "test", "./test/", "-run", "TestUT_", "-v")
27}
28
29// A build step that runs integration tests
30func Integration() error {
31  mg.Deps(Clean)
32  mg.Deps(Format)
33  fmt.Println("Running integration tests...")
34  return sh.RunV("go", "test", "./test/", "-run", "TestIT_", "-v")
35}
36
37// A build step that formats both Terraform code and Go code
38func Format() error {
39  fmt.Println("Formatting...")
40  if err := sh.RunV("terraform", "fmt", "."); err != nil {
41    return err
42  }
43  return sh.RunV("go", "fmt", "./test/")
44}
45
46// A build step that removes temporary build and test files
47func Clean() error {
48  fmt.Println("Cleaning...")
49  return filepath.Walk(".", func(path string, info os.FileInfo, err error) error {
50    if err != nil {
51      return err
52    }
53    if info.IsDir() && info.Name() == "vendor" {
54      return filepath.SkipDir
55    }
56    if info.IsDir() && info.Name() == ".terraform" {
57      os.RemoveAll(path)
58      fmt.Printf("Removed \"%v\"\n", path)
59      return filepath.SkipDir
60    }
61    if !info.IsDir() && (info.Name() == "terraform.tfstate" ||
62      info.Name() == "terraform.tfplan" ||
63      info.Name() == "terraform.tfstate.backup") {
64      os.Remove(path)
65      fmt.Printf("Removed \"%v\"\n", path)
66    }
67    return nil
68  })
69}

Go Back explore our courses

Agile Testing CI/CD Bootcamp

Run your tests on autopilot with CI/CD pipelines.

Terraform Foundations

Introduction to modern infrastructure provisioning.

Cloud Native Essentials

This is an introduction to Kubernetes, an open-source system for automating deployment, scaling, and management of containerized applications.

Paul Strebenitzer | 18.12.2024 Devops

AI for DevOps Engineers - Part 1: The Building Blocks of DevOps AI

DevOps is a key success factor for modern software development and we have most definitely come across AI in one way or another. The intersection of AI and

Matthias Theuermann | 14.11.2024 Devops

Understanding DevOps and Cloud Maturity Models: A Guide to Elevating Your IT Strategy

In today’s fast-paced technological landscape, DevOps and Cloud practices are integral to accelerating software delivery and optimizing cloud resources. But as

Martin Buchleitner | 04.11.2024 Devops, IAC

Efficient Testing Environments with Vagrant and Ansible: Simplifying Automation

Setting up consistent testing environments is essential for development, and Vagrant combined with Ansible is a powerful, flexible solution for automating this

Martin Buchleitner | 17.10.2024 Infrastructure as Code, Devops, HashiCorp, Packer, Vault, Mondoo

Github Action to Build Golden Images with HashiCorp Packer

In previous posts we have already shown multiple ways to use HashiCorp Packer to build Golden Images. In this post we will show how to automate the process with

Martin Buchleitner | 16.10.2024 Infrastructure as Code, Devops, HashiCorp, Consul

Automating DNS Updates from Kubernetes - HashiCorp Style

In dynamic Kubernetes environments, managing DNS records can be a challenge. As services scale, redeploy, or change IP addresses, ensuring that DNS records

We are here for you

You are interested in our courses or you simply have a question that needs answering? You can contact us at anytime! We will do our best to answer all your questions.