Decentralized Terraform Provider Distribution

In the very early days of Terraform, the main providers were included in the main Terraform CLI distribution and maintenence was done via similar principles as followed by the Linux kernel: a single shared codebase with responsibility for different parts delegated to different people.

In Terraform 0.10 we switched to a different model where providers were built and distributed separately, and Terraform CLI grew the ability to download and install those separate provider packages.

However, the providers were all still distributed from a centralized single source: HashiCorp's "releases" site. Since then, HashiCorp's partnerships with provider developers have grown but HashiCorp has retained the singular responsibility for building, signing, and releasing all of the providers that Terraform can automatically install. Although it's always been possible to build and manually install third-party Terraform providers, the need for manual installation tends to discourage the use of these.

Work is currently underway to enable automatic installation of third-party providers in a future release, though, and this project is my current focus. With that in mind, this article is an overview of some of the internal changes to Terraform that are needed to get there.

The overall technical design of this feature was led by fellow Terraform Core team member Kristin Laemmert.

Provider Namespacing

The first hurdle we hit on this journey was deciding how to model an ecosystem with many separate parties releasing into it. Terraform's model of providers thus far has been that each one has a single short name, like "aws" or "rundeck", that uniquely identifies it both for installation and throughout the internal processing of Terraform.

Some other ecosystems have had good success with a flat namespace like this, allowing an individual to take ownership of a name by publishing to it. However, a model like that works best when everyone is publishing their work publicly in a single registry. Real-world use of Terraform includes a number of bespoke providers for in-house APIs whose authors have no interest in making them available for others to use.

We've also frequently encountered a user experience issue where users find it hard to discover who is responsible for maintaining a particular provider, and have tended to assume that HashiCorp teams are maintaining all of the ones that terraform init has been able to auto-install. In practice, many of those providers are maintained by partner companies already, and some have no current maintainers at all.

With those concerns in mind, and taking inspiration from the existing model for Terraform module registries, we're establishing a three-level heirarchy for provider names in this new, decentralized ecosystem model:

registry.terraform.io/hashicorp/aws

A fully-qualified provider name consists of the hostname of the registry that is authoritative for its distribution, an organization namespace within that registry, and a name for the provider that is unique within a particular organization on a particular registry host.

A lot of providers will continue to be distributed via the main HashiCorp-run registry at registry.terraform.io though, so Terraform will also support a shorthand with only two segments: hashicorp/aws is equivalent to registry.terraform.io/hashicorp/aws. Furthermore, to retain compatibility for existing modules using providers that will continue to be maintained by and distributed by HashiCorp, the naked name "aws" will be treated as registry.terraform.io/hashicorp/aws by default.

Just as with the module registry protocol introduce in Terraform 0.11, we will be publishing documentation of the protocol Terraform expects to see from a provider registry so that it's possible for third parties to run their own registries, including private registries within a particular organization.

Provider Names in Modules

Adopting a heirarchical namespace for talking about providers for distribution led to a secondary challenge, though: the Terraform language already uses provider "short names" in many places, and it would be mightily inconvenient to have to use a long name like tf.example.com/awesomecorp/happycloud every time you need to mention the "happycloud" provider.

After considering a few different options, we ultimately took inspiration from how Go deals with package imports, establishing a separation between a fully-qualified provider name that uniquely identifies a provider in the global ecosystem and a module-specific local name that is used as a convenient shorthand for a provider within a single module. The existing required_providers block takes on a new role as the mapping from local name to fully-qualified name, along with still providing version constraints:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 2.0.0"
    }
    happycloud = {
      source  = "tf.example.com/awesomecorp/happycloud"
      version = "~> 1.2.0"
    }
  }
}

Once a local name is established within a module, all other references to the provider within the module will use that local name. For example:

provider "aws" {
  region = "us-west-2"
}

provider "happycloud" {
  endpoint = "https://happycloud.example.com/"
}

resource "aws_instance" "example" {
  instance_type = "t2.micro"
  ami           = "ami-abc123"
}

We expect that most modules will refer to modules using the same provider type name that appears at the end of the provider's fully-qualified name, as shown above, but because the namespace is decentralized it is likely that at some point there will be two providers that have the same local type name but different hostname or organization namespace. In that case, a module that uses both would need to give at least one of them a different local name to retain the local uniqueness.

An important part of this model is that Terraform uses the local names only within a single module, and uses the fully-qualified names between modules. That means that implicit inheritance or explicit passing of a provider from a parent module into a child module only requires the two to agree on the fully-qualified name; they can each select a different local name with no problem:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 2.0.0"
    }
    other_aws = {
      source  = "tf.example.com/awesomecorp/aws"
      version = "~> 5.0.0"
    }
  }
}

module "other" {
  source = "./modules/other"

  providers = {
    # The child module could use a name other than other_aws for this provider,
    # but it will still work as long as they both agree on the FQN being
    # tf.example.com/awesomecorp/aws .
    other_aws.foo = other_aws
  }
}

Adding this indirection has been quite a difficult change to Terraform Core because historically it was able to use a single identifier for a provider throughout configuration, state, plan, etc. In this new model, Terraform now needs to consult a mapping table within each module to find the corresponding fully-qualified name for any reference in configuration.

This work is largely internal only, but one place it will be somewhat visible is in state snapshots, where the original provider configuration for a resource will now use a fully-qualified name instead of a local name to ensure that it's possible to change the local name for a provider in the configuration without creating ambiguity for existing state snapshots:

module.foo.provider["tf.example.com/awesomecorp/aws"]

Internally, Terraform will now be tracking providers by their fully-qualified names throughout, using the local names only when interpreting references in the configuration for a single module and translating them as soon as possible into fully-qualified names.

Mirroring Providers

When deploying Terraform in production, it's common and reasonable to want to isolate the process running Terraform from the overhead and risk of repeatedly downloading the providers from their origins. Instead, we'd prefer to keep copies of the providers locally.

Terraform has historically supported this by scanning a set of local directories for executables named with the prefix terraform-provider-, followed by a short provider name and a version number giving something like terraform-provider-aws_v2.0.0.

That naming convention does not give enough information to infer a fully-qualified name, so we'll be switching to a new heirarchical directory structure that captures all of the components of the fully-qualified name along with the target platform and version number, giving a path like this:

registry.terraform.io/hashicorp/aws/2.0.0/windows_amd64/terraform-provider-aws_v2.0.0.exe

This new structure also addresses another latent concern that we've been wanting to address for a while: putting the packaged files for each provider in a separate directory so that they can potentially include other files in addition to the executable, such as metadata or license information. Therefore in this new structure it's the directories leading to the file that identify which provider the directory is for, and then the contents of that directory are the full contents of the distribution .zip file for that particular release.

We've also heard lots of requests over the years for the possibility to mirror providers in a network service rather than local disk, such as in JFrog Artifactory. Although this won't be complete in time for the first release of the functionality, we do eventually intend to support publishing a local repository of mirrored providers on a local HTTP server in addition to mirroring in a local filesystem directory.

Trust and Verification

One important advantage of having a centralized repository of providers has been that HashiCorp has been able to verify and sign them all with a HashiCorp-controlled private key, and then Terraform CLI can easily verify that signature during installation.

In a world of potentially many provider authors and provider registries, a centralized trust model is no longer appropriate: you wouldn't want to have to contact HashiCorp each time you want to make a new release of the provider you wrote to interact with some internal system that nobody outside of your company has never heard of.

Once Terraform CLI is able to install third-party providers, it will need to broaden its idea of trust to allow for third-party distributors. The details of this part are still being finalized by the Terraform Registry team in collaboration with the HashiCorp Product Security team, but it's likely to feel similar to the strategies used by Linux distributions like Debian, where Terraform will continue to trust certain root keys by default and you'll be able to opt in to trust others after doing due diligence about the issuer.

Lots of Refactoring!

As is often the case with wide-reaching features like this, we've had to do quite a lot of refactoring to prepare for implementing it. That refactoring work remains underway but will hopefully conclude soon and allow us to start working on exposing these features for real use.

The forthcoming Terraform 0.13 release should include at least the result of the refactoring, and hopefully also at least part of the functionality. It's likely that full support for installation from arbitrary sources will follow in a later minor release just because we want to make sure that the new trust and verification model is solid before moving away from the existing centralized trust model.