Upgrading Terraform Configurations

While HCL 2 is broadly compatible with its predecessor HCL 1, incorporating HIL expression features into the language required some compatibility compromises that mean some existing Terraform configurations will require slight modifications for compatibility with Terraform 0.12. Terraform itself also has some new capabilities in 0.12 that required minor incompatibilities, as a result of correcting some unintentional inconsistencies in the old implementation.

For example, Terraform 0.12 allows whole resource instances and collections of instances to be used as list values, where before only individual attributes could be accessed:

resource "aws_instance" "example" {
  count = 5

  ami           = "ami-123123"
  instance_type = "t2.micro"
}

output "ip_addresses" {
  value = [for x in aws_instance.example : {
    private = x.private_ip
    public  = x.public_ip
  }]
}

In Terraform 0.11 and earlier, aws_instance.example was not a valid reference alone, and so Terraform could get away with accepting aws_instance.example.id as an alias for aws_instance.example.0.id.

In order to make aws_instance.example usable in isolation, we had to make some decisions about how it would interact with the count feature in order to have reference like this behave consistently. The final design is that the presence of count, regardless of value, causes aws_instance.example to be a list value, and thus aws_instance.example.id is no longer valid: any existing configurations using that form would need to be rewritten as aws_instance.example[0].id.

Terraform 0.12 will include a tool to automatically rewrite the configuration files in an existing module, both to fix up these small incompatibilities and to make use of other general 0.12 and HCL 2 changes to improve readability.

Upgrade Methodology

Since some of the changes are at the level of HCL syntax, the first problem to solve is that the new HCL 2 parser will not be able to parse some existing configurations. As a result, the upgrade tool begins by using the HCL 1 parser to produce an HCL 1 AST.

As discussed in the earlier article on validation, the normal way to work with HCL 1 immediately loses details from the source file such as block structure and source locations. However, HCL 1 internally uses an AST which does retain the exact source block structure and any comments, and the upgrade tool is essentially a tree walk of that AST, applying migration rules along the way based on context.

An important requirement is to retain the relative ordering of configuration constructs and associated comments. The information retained by the HCL 1 AST in order to support terraform fmt allows us to weave the original files' comment tokens into the output at suitable locations, with the same constraints as terraform fmt itself has.

Structural Walk

The top-level flow of the upgrade process is to walk over each block and argument in the input files, and use the block type and argument names to activate different upgrade rules based on context.

Each of Terraform's top-level blocks has some different details to handle in the upgrade process, due to Terraform's special handling of its various "meta-arguments". Those special cases aside though, some general mapping rules can be derived from the providers' configuration schemas.

Expression Upgrades

One of the most obvious improvements in Terraform 0.12 is the ability to use expressions directly as argument values, rather than having to wrap everything in string interpolations:

resource "aws_instance" "example" {
  ami           = var.amis[data.aws_region.current.name]
  instance_type = var.instance_type
}

This is achieved by merging features from the interpolation language HIL into HCL itself, parsing both at once. However, this means the upgrade tool must also deal with HIL sequences, creating a further layer of upgrade rules.

The upgrade tool uses a recursive walk through both the HCL and HIL ASTs, switching over to HIL whenever it encounters a HCL string in a context where interpolation is allowed. The mapping logic unwraps strings containing single interpolation sequences to be naked expressions like the above, improving readability.

We also have mapping rules that replace several interpolation functions from Terraform 0.11 with equivalent native syntax features in 0.12. For example, the list(a, b, c) function is now better written as a HCL list constructor, [a, b, c]. We're also taking this opportunity to fix up the long-deprecated usage of the lookup function for map lookups, which are better written using map syntax foo[bar].

Preserving Comments

The HCL AST retains comments as a separate set of tokens alongside the main syntax tree, so reconstructing the original configuration requires comparing the source locations of each new syntax construct to the next comment to see if the comment should be emitted first.

This means that comments can, in practice, only appear between the significant language constructs.

When Static Analysis is Insufficient

The upgrade tool exclusively uses static analysis of configuration (no access to state or real remote objects) and so there are certain upgrade rules that cannot happen fully automatically. For these, the tool emits a warning and leaves a comment marked with TF-UPGRADE-TODO: in the updated source code.

For this reason, upgrading is an interactive development task rather than a fully unattended operation: the user should run the tool against a clean version control work tree and then use the VCS diff tools to review what it changed.

One particularly-tricky example was the long-standing oddity in Terraform where a redundant set of list brackets could be used to work around Terraform's inability to properly detect dynamic list expressions from interpolation:

  vpc_security_group_ids = ["${var.security_group_ids}"]

Although Terraform can often achieve the expected result without those redundant brackets, if there were any unknown values in the list Terraform 0.11 would incorrectly fail with the error "must be a list". This is fixed in Terraform 0.12, and so the above expression now results in a list of lists.

In many cases the upgrade tool can use resource type schema to prove both that the argument expects a list of strings and that the expression produces a list of lists, and so automatically remove the redundant brackets. In some cases though -- like the example above -- the upgrade tool may not be able to decide whether the given expression produces a single list or a nested list, and so it must instead just settle for marking up the problem with a comment so the user can decide how to adjust it.