replacing unreadable terraform 11 workarounds with beautiful terraform 12 constructs
On some greenfield infrastructure efforts at work I recently decided to take the not so recently released major update of Terraform 0.12 out for a spin. Specifically, they're up to 0.12.6 so many of the initial snags have been ironed out and it seemed like a good opportunity to kick the tires.
For some years now I have been saying that (pre 0.12) Terraform is better thought of more like XML than a DSL. Originally the project founder wanted to keep HCL "as simple as possible" and the more familiar programming language like constructs of looping and conditionals were conceded to later in the projects life. However since these were addons and the project was not designed with them in mind they lead to some scary issues. That one in particular bit me hard in a way that deleted IAM accounts for 2/3rds of our engineering team.
While working with 0.12 though I have managed to add three dramatic improvements to some of our modules by leveraging some awesome new features.
First some background. We have a module which creates GKE clusters and due to the varying needs of our projects sometimes we want one with a default node pool and sometimes we don't. (It doesn't matter if none of that means anything to you, we'll go with simple examples).
Like many Terraform resources, the config for a GKE cluster is complex and requires both arguments and config blocks.
resource "my_resource" "foo" {
argument = "value"
config_block {
config_a = 1
config_b = 2
}
}
It has always been straightforward with Terraform to pass a variable in and assign it to a value, but until 0.12 it was not possible to optionally specify a config block. It's either on the resource or it's not. In our case we wanted to support the creation of resources which had mutually exclusive combinations of arguments and config blocks. So the workaround looked like this.
resource "my_resource" "variant_a" {
count = "${var.enable_variant_a ? 1 : 0}"
argument = "value"
}
resource "my_resource" "variant_b" {
count = "${var.enable_variant_a ? 0 : 1}"
config_block {
config_a = 1
config_b = 2
}
}
In this way we can achieve the desired result, but it's kinda a gross workaround that results in code duplication and is generally the sort of thing you find all over a complex legacy Terraform code base.
We also needed this module to output the name of the resource regardless of which variant was created so we defined an output.
output "resource_name" {
value = "${var.enable_variant_a ? join("", my_resource.variant_a.*.name) : join("", my_resource.variant_b.*.name) }"
}
Hideous.
This is where we introduce our first dramatic improvement with a new Terraform 0.12 feature, string templates.
The addition of conditional directives in strings allowed us to take the above monstrosity and change it to this.
output "resource_name" {
value = "%{ if var.enable_variant_a }${my_resource.variant_a[0].name}%{ else }${my_resource.variant_b[0].name}%{ endif }
}
Still burly but infinitely more readable.
Our second dramatic improvement is enabled by combining three new features, complex types, the new null type, and dynamic blocks. Using these we eliminated the need for duplicate resource variants.
First we constructed a complex type to describe all the configuration required for our resource.
variable "my_resource_config" {
type = object({
name = string
location = string
enable_variant_a = bool
config_block = list(object({
config_a = number
config_b = number
}))
})
}
Note the inclusion of the mutually exclusive options. Also note that this variable definition needs to exist both in the module as a definition of the interface to the module and at the calling site so a correctly typed instance of the variable can be populated. I opted to separate this complex type from the other variables being passed to the module and put it in it's own file so I could symlink it into the calling Terraform space to avoid duplication (we use symlinks as Terraform "include" statements a fair bit).
Next we populate the values of this variable type in our terraform.tfvars file.
my_resource_config = {
"name" = "sweet_resource"
"location" = "us-central1"
"enable_variant_a" = true
"config_block" = []
}
Finally we add the dynamic block to the resource itself.
resource "my_resource" "resource" {
name = var.my_resource_config.name
location = var.my_resource_config.location
argument = var.my_resource_config.enable_variant_a
dynamic "config_block" {
for_each = var.my_resource_config.config_block
content {
config_a = config_block.value.config_a
config_b = config_block.value.config_b
}
}
}
Note that to access the elements of the config_block on our complex type we reference the name of the dynamic block (also config_block) followed by .value followed by the element name. If this is confusing you can add the line iterator = some_other_name to use in place of the name of the dynamic block.
With this approach the values we've set in my_resource_config will cause the boolean value to be passed to our argument to enable whatever feature we want and the empty list in the config_block field will mean the for_each loop in the dynamic block never runs and the block is not added.
To flip this around we define the config this way.
my_resource_config = {
"name" = "sweet_resource"
"location" = "us-central1"
"enable_variant_a" = null
"config_block" = [
"config_a" = 1
"config_b" = 2
]
}
The presence of the null type means that Terraform will not actually set the value of the argument and will not complain about a type mismatch, while the presence of a list of objects in config_block means the loop will now execute and the dynamic block is populated. Note that there are some resources which support multiple copies of a dynamic block (eg. ingress rules on a security group) and in those cases you can just add more elements to the list.
This second improvement also means that the first improvement with the string templates actually gets tossed out because we no longer have multiple variants of our resource we can just output my_resource.resource.name, another win!
The third improvement happened on another module that combines the elements of the second improvement with the addition of a for_each construct on a resource itself as well as on a dynamic block.
For this resource our config object looks like this.
variable "my_resource_configs" {
type = map(object({
name = string
location = string
enable_variant_a = bool
config_block = list(object({
config_a = number
config_b = number
}))
}))
}
Note that in contrast to our first config object the root begins with a map of objects rather than an object. This allows us to pass multiple configs in and iterate over them.
my_resource_configs = {
"default" = {
"name" = "default"
"location" = "us-central1"
"enable_variant_a = null
"config_block" = [
{
"config_a" = 1
"config_b" = 2
}
]
}
"variant" = {
"name" = "variant"
"location" = "us-central1"
"enable_variant_a = true
"config_block" = []
}
}
And in our module we define our resource like so.
resource "my_resource" "resource" {
for_each = var.my_resource_config
name = each.value.name
location = each.value.location
argument = each.value.argument
dynamic "config_block" {
for_each = each.value.config_block
content {
config_a = config_block.value.config_a
config_b = config_block.value.config_b
}
}
}
Now we iterate over each of the config items safely (we could use this approach to create IAM accounts and not nuke them when somebody quits and we remove an username from a list of users passed in) and when we come to the dynamic block we further iterate over the elements of that list.
These improvements only touched two Terraform modules in our codebase but resulted in the elimination of over 150 lines of legacy boilerplate and workarounds. Probably my favourite pull request to date