Enforce EBS Encryption Across an AWS Organization

Table of Contents

Introduction

My team & I manage an AWS organization with 90+ AWS accounts. We recently embarked on a campaign to enforce EBS encryption across our organization. This is the story of the approach we took, the challenges we faced & the decisions we made along the way.

EBS Encryption by Default

EBS has a feature to enable encryption by default for all EBS volumes & snapshots. Before this feature was released, the only way to enforce encryption was to deny creation of unencrypted resources using IAM policies (that’s what we do to enforce tagging).

Configure your AWS account to enforce the encryption of the new EBS volumes and snapshot copies that you create. For example, Amazon EBS encrypts the EBS volumes created when you launch an instance and the snapshots that you copy from an unencrypted snapshot.

EBS Encryption by Default — EC2 User Guide

Encryption by Default — Considerations

However, there are a few considerations to keep in mind:

  • Encryption by default has no effect on existing EBS volumes or snapshots.
  • Encryption by default is a Region-specific setting. If you enable it for a Region, you cannot disable it for individual volumes or snapshots in that Region.
  • When you enable encryption by default, you can launch an instance only if the instance type supports EBS encryption.
  • If you copy a snapshot and encrypt it to a new KMS key, a complete (non-incremental) copy is created. This results in additional storage costs.
  • If after turning on encryption by default, you need to launch an unencrypted EC2 instance or create an unencrypted EBS volume, you must first turn off encryption by default completely, launch the EC2 instance or create the EBS volume, & then turn on encryption by default again.

Enable Encryption by Default

To enable encryption by default, go to https://console.aws.amazon.com/ec2/home#Settings:tab=ebsEncryption & turn it on:

You can also choose your own encryption key (CMK) instead of the default EBS key.

Challenges in Enforcing Encryption

Being a large enterprise, enforcing EBS encryption for us, wasn’t as easy as enabling it in every account. There were a number of use cases to be considered.

Cross-Account Snapshot Sharing

EBS snapshots encrypted using the account’s default key, cannot be shared with other accounts. Since sharing snapshots across accounts is a very common use case for us, using the default key wasn’t an option.

Single Global Master CMK for Entire AWS Org

Since using the default key wasn’t feasible, the next best solution would be to create a single master CMK in a central account & share it with all accounts in the organization. We decided against it since we would rather have separate keys for separate accounts / business units.

Golden AMIs

My team & I are also responsible for periodically building & distributing “golden AMIs”. These are AMIs of approved operating systems, hardened & bootstrapped with base configuration (CIS compliance etc) & security & monitoring agents like Qualys, SentinelOne, CloudWatch, CloudHealth, Datadog, etc.

Anyone wanting to launch an EC2 instance in the company, must do so using one of these AMIs. This presented a unique challenge since if we encrypt the golden AMIs’ snapshots with a CMK, we’d have to share the CMK with the entire org, which we really did not want to do.

Our Final Approach

Considering all of the above, here is the approach we designed to enforce EBS encryption across our AWS organization.

The Encryption Key

First, there’s the matter of the encryption key. Our AWS accounts are divided by application & environment. Every business unit (BU) / application / team typically gets 2 or 3 AWS accounts: 1 for production, 1 for all non-production or pre-production environments like dev, QA, stage, perf, etc & 1 sandbox account (optional) for trying out stuff.

With such a design, it’s logical to provision a dedicated encryption key (CMK) per app/BU/team. We went 1 step further. We provisoned 1 CMK per account & made them usable from accounts of the same team. For example:

  • CMK in analytics prod account is usable by analytics nonprod & sandbox
  • CMK in analytics nonprod account is usable by analytics prod & sandbox
  • CMK in analytics sandbox is usable by analytics prod & nonprod

This design facilitates common use cases like teams building AMIs in nonprod & sharing to prod, or teams sharing snapshots of test data across their accounts. If you’re wondering, all this cross-account key sharing is accomplished using KMS key policies.

Each CMK must also allow use by:

  • Users of your AWS accounts, assuming they assume certain predefined roles to work in the accounts.
  • AWS service roles:
    • service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling
    • service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot
    • Other service roles like the ones used by EMR

And as far as provisioning 90+ CMKs across 90+ AWS accounts & setting each CMK’s key policy is concerned, we use Terraform. Since the very beginning, we’ve used so called “bootstrap” modules written in Terraform, to bootstrap every new AWS account we provision. So it was just a matter of adding the CMK to it & reapplying it on all accounts!

Golden AMIs

Given all the challenges & limitations around encrypting golden AMI snapshots, we eventually decided not to encrypt them at all. Here’s how it would work:

The central account where we build our golden AMIs will have “EBS encryption by default” turned OFF! All AMIs will come out of the factory with unencrypted snapshots. They’re then shared with the entire AWS organization. Since “encryption by default” is turned ON in all other accounts, when someone uses the golden AMIs to launch an EC2 instance, it’s EBS volumes will automatically be encrypted using their own CMK!

Testing the Approach

If you’re considering enforcing EBS encryption in your org, you’ll certainly need to test it out thoroughly before you roll it out to every account. Here are a few basic test cases that will give you confidence in your encryption design.

The following tests were performed after enabling encryption by default. The expected result in each case is for all newly created EBS volumes & snapshots to be encrypted.

TESTS
Create an EBS volume using CMK. Modify its size.
Create a snapshot from it. Delete the volume.
Create volume (encrypted by CMK) from snapshot & delete volume.
Create volume (encrypted by aws/ebs) from snapshot & delete volume.
Copy snapshot using CMK & delete the copy.
Copy snapshot using aws/ebs & delete the copy.
Delete the original snapshot.
Create an EBS volume (using aws/ebs) & delete it
Create an EBS volume (from an unencrypted snapshot) & delete it
Launch an EC2 instance & terminate it
Launch an EC2 instance (from an ASG) & terminate it
Launch an EC2 spot instance & terminate it
Launch an EC2 spot instance (from an ASG) & terminate it
Create an EC2 instance (from a Golden AMI) & terminate it
(Unencrypted golden AMI is shared to this account from another account)
Create a Beanstalk app & environment & delete it
Create an EKS cluster with an EKS-managed node group
Create an ECS cluster with EC2 as the capacity provider
Create an “EMR on EC2” cluster & terminate it
Create an RDS cluster/instance
(encrypted using the aws/rds key, not the EBS CMK)
Add other EC2-based services here…

Operational Considerations

Cross-Team Snapshot Sharing

When EBS snapshots or AMIs are to be shared across non-peer accounts, such as the analytics team wanting to share with the datalake or some application team, teams will need to reach out to you to manually update the CMK policy to allow this.

EC2s Launched by Non-Standard Roles

In cases where EC2 instances are launched by IAM roles other than your company’s standard IAM roles for users, or service roles like AutoScaling, you’ll need to grant CMK access to the role in question. One such example might be EC2s launched by Karpenter in EKS.

EMR (& Other Special Cases)

Most EC2-based services like Beanstalk, EKS, ECS, etc manage EC2 instances via the AutoScalingRole, but EMR is different. EMR manages EC2 instances directly using the roles configured on the EMR cluster so these roles must be added to the CMK policy as well.

Also, teams using EMR can create & use custom roles for their EMR clusters. In that case, the custom roles must also be added to the CMK policy!

If you have a centralized cloud inventory management system like CloudHealth or Wiz, you can leverage them to get an idea of how many custom roles are in use in your org & how much effort would be required to make them work with EBS encryption.

Resources

These resources were very helpful in designing our encryption strategy & understanding how encryption in AWS works:

Conclusion

I hope this article gives you an idea of what it takes to enforce something as simple as EBS encryption across an enterprise. If you’re in the same boat & looking to enforce encryption in your org (EBS or otherwise), I hope this article can serve as a guide & reference in your journey. Good luck!

About the Author ✍🏻

Harish KM is a Principal DevOps Engineer at QloudX & a top-ranked AWS Ambassador since 2020. 👨🏻‍💻

With over a decade of industry experience as everything from a full-stack engineer to a cloud architect, Harish has built many world-class solutions for clients around the world! 👷🏻‍♂️

With over 20 certifications in cloud (AWS, Azure, GCP), containers (Kubernetes, Docker) & DevOps (Terraform, Ansible, Jenkins), Harish is an expert in a multitude of technologies. 📚

These days, his focus is on the fascinating world of DevOps & how it can transform the way we do things! 🚀

Leave a Reply

Your email address will not be published. Required fields are marked *