Terraform: How to support releasing new versions of the code, running in parallel with the last stable release?
There are a few different ways to achieve goals of this sort, each with some different tradeoffs. I'm going to describe the most common ones below.



The simplest approach is to use Terraform's create_before_destroy mechanism with autoscaling groups. An example of this pattern is included in the aws_launch_configuration documentation.

In this scenario, changing the AMI id causes the launch configuration to be re-created. Due to create_before_destroy, the new configuration is created first, then a new autoscaling group is created, adding the new instances to an attached ELB. The min_elb_capacity argument to aws_autoscaling_group can be used to ensure that a given number of instances are present and healthy in the attached ELB before considering the autoscaling group to be created, thus delaying the destruction of the old autoscaling group and launch configuration until the new one is serving requests.

The downside of this approach is the lack of control it represents. Since Terraform is thinking of the entire set of changes as a single run, it's impossible to pause after creating the new instances to allow other checks to be carried out before destroying the old ones. As a consequence, the ELB healthcheck is the only input to deciding if the new release is "good", and rolling back is impossible once the old resources have been destroyed.



A second common approach is to adopt a sort of "blue/green deployment" pattern with explicit changes to two clusters. This is done by putting all of the per-release resources in a child module, and instantiating that module twice with different arguments. In the top-level module this would look something like the following:

resource "aws_elb" "example" {
  instances = "${concat(module.blue.ec2_instance_ids, module.green.ec2_instance_ids)}"

  # ...
}

module "blue" {
  source = "./app"

  ami_id = "ami-1234"
  count  = 10
}

module "green" {
  source = "./app"

  ami_id = "ami-5678"
  count  = 0
}


The principle of operation here is that in the "steady state" (no deployment in progress) only one of these modules has a non-zero count, and the other one has zero. During a deployment, they are both set to the same non-zero count, but with different ami_id values. Each deployment swaps which of the modules is the "active" module, with both being active during the deployment.

When using this approach, each step is a distinct Terraform operation:


change count of the inactive module to nonzero and set its AMI id
apply the change with Terraform, thus activating the new module
verify that the new release is good
change count of the older module to zero
apply the change with Terraform, thus deactivating the old module


Although this has more steps, it allows arbitrary verification and an arbitrary amount of time to pass during step 3. It also allows "rolling back" by resetting the previously-inactive cluster count to zero.

Since both the old and new clusters exist in the same configuration, there is the risk of using this pattern incorrectly and prematurely destroying the active cluster. This can be mitigated by carefully reviewing Terraform's plan to make sure it leaves the old cluster untouched, but Terraform itself can't guarantee this.

Also, since both clusters are using the same child module configuration, it can be tricky to make updates to that configuration while retaining the blue/green separation. If changes are made that would require Terraform to replace the running instances, it's necessary to temporarily have two copies of the module code on disk, make the source arguments point to separate copies, and make the change only to the copy used by the inactive module.



The final approach I'll present is the most extreme and manual, but it does the best job of meeting your requirements and retaining control. This is, in effect, the most literal interpretation of your current CloudFormation workflow, and is a more concrete version of the approach you talked about in your question.

In this approach, there are two entirely-separate Terraform configurations, which I will call "version-agnostic" (things that must survive between versions, such as your ELB) and "version-specific" (the resources that are re-created for each new version).

The version-agnostic configuration will contain the ELB and will, as you suspected, export its id for consumption by the version-specific configuration:

terraform {
  required_version = "&gt;= 0.9.4"
  backend "s3" {
    bucket = "example-company-terraform-state"
    key    = "exampleapp/version-agnostic"
    region = "eu-central-1"
  }
}

resource "aws_elb" "example" {
  # ...
}

output "elb_id" {
  value = "${aws_elb.example.id}"
}


This configuration can be initialized, planned and applied as usual, creating an ELB with no attached instances to start.

The version-specific configuration would be similar to the "app" child module in the previous approach, but this time as a top-level module. The backend configuration for this module would omit the S3 key, since this will change for each new release as you expected:

terraform {
  required_version = "&gt;= 0.9.4"
  backend "s3" {
    bucket = "example-company-terraform-state"
    region = "eu-central-1"
  }
}


The specific key can then be set (or re-set) when running terraform init:

$ terraform init -reconfigure -backend-config="key=exampleapp/20170808-1"


Here I chose to use a "current date, release index" tuple as an identifier for a release. By running terraform init with a new value for this argument, an entirely separate state is created, independent of the last. Using -reconfigure tells Terraform that you don't wish to migrate the old state to the new, but rather to just switch directly to the new state path, possibly creating a new state in the process.

You can then run terraform show to confirm that indeed the state is empty (and thus operations won't affect existing resources) and then run a plan/apply cycle as normal.

Once you're satisfied with the new release, you can switch back to the previous version and destroy it.

The version-specific configuration will need the id of the ELB from the version-agnostic configuration in order to populate the load_balancers attribute of aws_autoscaling_group. To get access to this, we can use the terraform_remote_state data source to read the values from its state in S3:

data "terraform_remote_state" "version_agnostic" {
  backend = "s3"
  config {
    bucket = "example-company-terraform-state"
    key    = "exampleapp/version-agnostic"
    region = "eu-central-1"
  }
}

resource "aws_autoscaling_group" "example" {
  # ...

  load_balancers = ["${data.terraform_remote_state.version_agnostic.elb_id}"]
}


With a system of this complexity, it would likely be best to run Terraform via some sort of wrapper script or orchestration to make the release process less onerous. For example, such a script might automatically generate the new version number to avoid the risk of a human operator mistyping the date or accidentally conflicting with an existing one. There's some recommendations and caveats about running Terraform via scripts in the guide Running Terraform in Automation.



Although the third option here is the most direct mapping of your CloudFormation approach, the second option is more commonly used due to it striking a reasonable compromise between control and workflow overhead.