Table of Contents

• Scenario
• Approach
• Example: Expected Plan
• Example: Auto-Approve Apply
• Conclusion
• About the Author ✍🏻

Scenario

Consider this scenario: You have a few Terraform modules that are used to deploy (AWS) infrastructure in several environments / regions / accounts. In the absence of a managed “CI/CD for IaC” platform like Terraform cloud or Spacelift, you use your application CI/CD tooling for IaC as well, like Jenkins, GitLab or GitHub. Maybe you even have Terragrunt & Atlantis in there.

In most cases, you want a human to review the Terraform plan before applying it, even if it’s the same plan in every environment. But what if you need to push out a fairly trivial change / update to all environments ASAP? Reviewing all plans would be cumbersome, but you also can’t blindly auto-approve Terraform apply with the -auto-approve flag, in case something has changed in AWS that the Terraform plan might undo!

This article describes a custom, scripted solution to address this niche use case.

Approach

The script takes the following steps as it iterates through every environment:

1. terraform init
2. terraform plan — ideally targeted to specific resource(s)
   • Capture this plan in a file
3. Convert the plan to JSON
4. Compare this plan to the “expected” plan:
   • If they match, terraform apply with -auto-approve
   • If not, just run terraform apply & wait for human approval

The “expected” plan in step 4 is something you capture before running this script by:

1. Running terraform plan in one environment
2. If the plan is as expected, save the plan to a file & convert it to JSON
3. If the plan has environment-specific values, remove them

For example, if you’re deploying a Terraform module to multiple AWS regions or accounts & the region or account ID is included in the plan (like resource ARN), remove it.

Example: Expected Plan

Let’s walk through an example: You have a Terraform-managed IAM policy in several AWS accounts & you need to add a permission to all of them.

First to capture the expected plan, run:

$ terraform init
$ terraform plan -out my-policy.tfplan -target aws_iam_policy.my_policy

$ terraform show -json my-policy.tfplan | \
jq .resource_changes > expected-changes.json

$ rm my-policy.tfplan

terraform show -json my-policy.tfplan generates a large JSON; we’re only interested in its resource_changes section. The final expected-changes.json looks like this:

[
  {
    "address": "aws_iam_policy.my_policy",
    "mode": "managed",
    "type": "aws_iam_policy",
    "name": "my_policy",
    "provider_name": "aws",
    "change": {
      "actions": [
        "update"
      ],
      "before": {
        "attachment_count": 1,
        "name": "MyPolicy",
        "name_prefix": "",
        "path": "/",
        "policy": "<JSON-policy>"
      },
      "after": {
        "attachment_count": 1,
        "name": "MyPolicy",
        "name_prefix": "",
        "path": "/",
        "policy": "<JSON-policy>"
      },
      "after_unknown": {}
    }
  }
]

The embedded JSON policy changes between the before & after blocks when you add an IAM permission. Note that the before & after blocks also had account-specific values (ARN etc) which were removed by running:

jq '.[0].change.before.arn       |= empty |
    .[0].change.before.id        |= empty |
    .[0].change.before.policy_id |= empty |
    .[0].change.after.arn        |= empty |
    .[0].change.after.id         |= empty |
    .[0].change.after.policy_id  |= empty' \
  expected-changes.json > expected-changes.json

Example: Auto-Approve Apply

Now to iterate over the accounts & run Terraform apply on each account with or without auto-approval:

$ terraform init
$ terraform plan -out my-policy.tfplan -target aws_iam_policy.my_policy

$ terraform show -json my-policy.tfplan | \
jq .resource_changes > resource-changes.json

$ rm my-policy.tfplan

jq '.[0].change.before.arn       |= empty |
    .[0].change.before.id        |= empty |
    .[0].change.before.policy_id |= empty |
    .[0].change.after.arn        |= empty |
    .[0].change.after.id         |= empty |
    .[0].change.after.policy_id  |= empty' \
  resource-changes.json > changes.json

$ rm resource-changes.json
$ pip install DeepDiff

$ DIFF=$(deep diff changes.json expected-changes.json)
$ rm changes.json

$ APPROVE=''
$ if [ "$DIFF" = '{}' ]; then APPROVE=-auto-approve; fi

$ terraform apply $APPROVE -target aws_iam_policy.my_policy

Here we use Python’s DeepDiff package to deep compare JSONs. The output diff is an empty JSON {} if no differences are found.

Warning: Do not provide my-policy.tfplan to terraform apply as an input. If you do, -auto-approve is implied, even if there are unexpected changes!

Conclusion

This article explored a simple solution to automate infrastructure changes at scale. This works best for smaller changes. If you manage large-scale infrastrcuture, it might be worth investing in a managed IaC CI/CD solution like Terraform cloud or Spacelift.

About the Author ✍🏻

Harish KM is a Principal DevOps Engineer at QloudX & a top-ranked AWS Ambassador since 2020. 👨🏻‍💻

With over a decade of industry experience as everything from a full-stack engineer to a cloud architect, Harish has built many world-class solutions for clients around the world! 👷🏻‍♂️

With over 20 certifications in cloud (AWS, Azure, GCP), containers (Kubernetes, Docker) & DevOps (Terraform, Ansible, Jenkins), Harish is an expert in a multitude of technologies. 📚

These days, his focus is on the fascinating world of DevOps & how it can transform the way we do things! 🚀

Automation AWS Terraform