Expanding Terraform's Horizons: The Call for Custom Functions

Terraform functions have become a cornerstone in managing infrastructure as code. These built-in functions empower developers to manipulate data, perform calculations, and streamline resource configurations within their Terraform projects. As the complexity of infrastructure grows, so does the need for more advanced and customizable functions to handle unique requirements and optimize workflows.

The current set of Terraform functions provides a solid foundation, but it has limitations when dealing with specific use cases. This blogpost explores the potential of expanding Terraform's capabilities through custom functions. It delves into the current state of Terraform functions, discusses the impact of custom functions on infrastructure as code, and looks at approaches to implement this feature. By the end, readers will gain insights into how custom functions could revolutionize Terraform's flexibility and efficiency in managing complex infrastructure setups.

Current State of Terraform Functions

Overview of Built-in Functions

Terraform functions are built-in, reusable code blocks that perform specific tasks within Terraform configurations. These functions make code more dynamic and ensure configurations remain DRY (Don't Repeat Yourself). They allow users to manipulate data, perform calculations, and streamline resource configurations within Terraform projects.

The syntax for built-in functions generally follows the pattern: <function_name>(arg 1, arg 2). These functions accept a predefined number and type of arguments, though some, like min() and max(), can handle undefined arguments of the same type. For improved efficiency, users can pass arguments using list or tuple variables with expansion syntax: (<list/tuple variable> ...).

Terraform offers a variety of built-in functions, including:

• Type conversion functions
• Length calculation functions
• Complex variable building functions

While these functions are not frequently used, they are essential for creating more flexible and maintainable configurations.

Limitations and Pain Points

Despite their usefulness, Terraform's current function system has several limitations:

• Lack of custom functions: Users cannot create custom functions to add further processing logic beyond the built-in capabilities. This restriction limits the ability to tailor functions to specific use cases.
• Module system constraints: Terraform's module system, while useful for code reuse, has limitations. The inability to do partial application can complicate module interfaces.
• Type restrictions: When defining a module's input variable types, using map or object types can be impractical. The map type requires all values to have the same type, while the object type mandates specifying all key names and value types.

Community Workarounds

To address these limitations, the Terraform community has developed several workarounds:

• Provider-defined functions: Some providers implement custom functions that can be used from HCL, although this feature is relatively new with limited examples.

• Using modules as pseudo-functions: Users often create modules to replicate function-like behavior, using variable, locals, and output primitives to mimic parameters, function body, and return concepts.

• Code generation tools: Third-party tools like Terramate have emerged to address Terraform's limitations. Terramate introduces concepts like stacks, global variables, and code generation to simplify environment management and reduce code complexity.

  1globals {
  2  a = tm_try(global.b, null)
  3}
  

• Creative use of existing functions: Users often combine built-in functions in complex ways to achieve desired outcomes, though this can lead to less readable code.

While these workarounds provide some relief, the Terraform community continues to advocate for the introduction of user-defined functions to enhance flexibility and code reusability in infrastructure management.

In a statement, a contributor emphasizes that due to HCL's nature as a non-general programming language, any new function-creation capabilities within it would be inherently limited. They mention that partial application and function currying might be achievable, but this could represent the extent of native enhancements. Instead of expanding HCL directly, the development focus is on extending function capabilities through the RPC (Remote Procedure Call) layer, which will remain the primary method for enhancing functionality for the foreseeable future. This statement comes in response to ongoing requests from the Terraform community for more flexible and reusable code options through user-defined functions, as noted in the prior communication.

Custom Functions: A Game-Changer for IaC

Enhanced Flexibility in Configuration

Custom functions have the potential to revolutionize Infrastructure as Code (IaC) by providing enhanced flexibility in configuration. Currently, Terraform's built-in functions offer a solid foundation for manipulating data and performing calculations. However, the inability to create custom functions limits users' ability to add further processing logic beyond what is already available. This restriction often leads to complex workarounds and less efficient code.

The introduction of custom functions would allow users to address complex requirements more effectively. For instance, custom functions could handle specific use cases that are currently difficult to manage with existing built-in functions. This enhancement would enable users to create more dynamic and adaptable configurations, tailoring their infrastructure code to specific needs without resorting to convoluted solutions.

Improved Code Reusability

Custom functions would significantly improve code reusability in Terraform projects. Currently, to reuse code, users often have to create entire modules, which can be more work than necessary for simple operations. With custom functions, users could define reusable pieces of logic that can be easily shared across different parts of their configuration.

For example, a custom function could be defined to perform a specific calculation or data transformation that is used repeatedly throughout a project. Instead of duplicating this logic or creating a module, users could simply call the custom function wherever needed. This approach would lead to more concise, maintainable, and DRY (Don't Repeat Yourself) code.

Domain-Specific Optimizations

Custom functions would enable domain-specific optimizations, allowing users to create functions tailored to their particular industry or use case. This feature would be particularly valuable for handling complex, industry-specific calculations or data manipulations that are not covered by Terraform's built-in functions.

Moreover, the ability for providers to include context-specific functions would be a game-changer. This would allow cloud providers or third-party vendors to offer specialized functions that are optimized for their specific services or platforms. For instance, a cloud provider could offer custom functions for advanced networking calculations or security policy generation, streamlining the process of configuring complex infrastructure components.

By enabling these domain-specific optimizations, custom functions would bridge the gap between Terraform's general-purpose capabilities and the specific needs of different industries and cloud environments, ultimately leading to more efficient and effective infrastructure management.

Implementing Custom Functions: Possible Approaches

Extending HCL

While Terraform's core functionality includes a robust set of built-in functions, there may be cases where specific logic or data manipulation is required that isn't covered by the defaults. Although Terraform does not natively support user-defined functions within HCL itself, developers can effectively simulate this capability through alternative means.

One approach to extend HCL's capabilities is through the use of external data sources. The external data source allows users to call out to external programs, which can be written in any language capable of returning JSON output. This method effectively allows the creation of custom functions that are executed externally, providing a flexible way to incorporate complex logic into Terraform configurations.

Another method to implement custom operations is through the use of local-exec provisioners. These provisioners can execute scripts on the local machine running Terraform when resources are created or changed. This approach is particularly useful for scenarios that require custom operations tied to specific resource lifecycle events.

Go-based Function Libraries

For more advanced customization, developers can create Terraform plugins using Go. This approach involves creating a provider that includes custom resources and data sources. The process begins with setting up a basic project structure and scaffolding for a Terraform plugin.

To implement a custom function as a resource, developers create a Go file (e.g., resource_server.go) that defines the data schema and CRUD operations for the resource. The schema defines the structure of the resource, while the CRUD functions implement the logic for creating, reading, updating, and deleting the resource.

If you would like to learn more about creating providers, we recommend you take a look at our blog series in which we have covered the creation and expansion of providers based on our experience with the Mondoo provider.

External Function Execution

External function execution provides a powerful way to incorporate custom logic into Terraform workflows. This approach allows Terraform to interact with external programs and process their output for use within configurations.

For example, developers can create a PowerShell script that retrieves specific data and use it within a Terraform configuration. This method is particularly useful when needing to interact with local tools or process data from external APIs that are not directly supported by Terraform's built-in functions or available providers.

However, it's important to note that using external programs can introduce platform-specific dependencies and may limit the portability of Terraform configurations across different environments. Therefore, careful consideration should be given to the trade-offs between flexibility and cross-platform compatibility when implementing custom functions through external execution.

Conclusion

The introduction of custom functions in Terraform has the potential to further develop Infrastructure as Code. This enhancement would provide users with the flexibility to create tailored solutions for complex requirements, leading to more efficient and maintainable code. The ability to implement domain-specific optimizations and improve code reusability would have a significant impact on how developers manage their infrastructure, enabling them to handle unique scenarios with ease.

While Terraform doesn't natively support user-defined functions, there are several approaches to simulate this functionality. These include extending HCL through external data sources, creating Go-based function libraries, and leveraging external function execution. Each method has its own strengths and trade-offs, allowing developers to choose the most suitable approach for their specific needs. As Terraform continues to evolve, the community's push for custom functions highlights the ongoing need to adapt and improve this essential tool for modern infrastructure management.