Almost every service in AWS that stores data on disk provides the ability to encrypt it at rest. And in almost all of those cases, the encryption key you use comes from KMS, either the AWS-provided default key for the service or one that you generated in or uploaded to KMS.

Having such a centralized system for any and all encryption needs throughout AWS is awesome for auditability, maintainability, enforcing security policies, and monitoring all encryption/decryption-related activities. In this article, we will explore how to set up a system to closely monitor all decryption activity going through KMS, and alert us in case of suspicious activity.

What are we building?

Our objective here is to build a system as shown in the architecture diagram below:

The approach is as follows:

• AWS services call KMS for encryption/decryption all the time.
• We will turn on CloudTrail to capture all KMS activities.
• The CloudTrail logs are sent to CloudWatch.
• A CloudWatch Logs metric filter continuously scans the CloudTrail logs looking for failed decryption calls.
• If found, CloudWatch increments the count of a CloudWatch custom metric we define.
• When the metric’s value crosses a predefined threshold, it sends an email alert to admins who can take appropriate actions.

Enable CloudTrail

The first step is to enable CloudTrail to capture all those KMS API calls used to encrypt/decrypt data. For this, go to console.aws.amazon.com/cloudtrail/home#/trails and click Create Trail:

Next, provide a name for the trail, provide an S3 bucket (new or existing), and most importantly, enable CloudWatch Logs:

On the next screen, leave all defaults as it is:

Finish creating the trail.

CloudWatch Logs, Filter, and Alarm

Perform some random tasks in your AWS account, preferably involving encryption/decryption like creating/retrieving secrets in Secrets Manager. This will generate some API calls for CloudTrail to capture. CloudTrail can take up to fifteen minutes to deliver logs so we’ll have to wait a while before the CloudWatch logs are generated.

Once the CloudWatch log group is created, open it:

Scroll down to the Metric Filters tab and click Create Metric Filter:

A typical KMS event captured by CloudTrail looks like this:

{
    "eventVersion": "1.05",
    "userIdentity": {},
    "eventTime": "2020-11-16T22:25:39Z",
    "eventSource": "kms.amazonaws.com",
    "eventName": "Decrypt",
    "awsRegion": "ap-southeast-2",
    "errorCode": "IncorrectKeyException",
    "errorMessage": "The key ID in the request does not identify a CMK that can perform this operation.",
    "requestID": "11748bbd-ddcd-4ee2-9f42-9cec69f414b1",
    "eventID": "1f620618-46e5-4f78-93cc-0b7bccfff5d2",
    "readOnly": true,
    "eventType": "AwsApiCall"
}

Since we are looking to monitor for all KMS errors, enter this filter pattern in the create metric filter screen:

{ $.eventSource = "kms.amazonaws.com" && $.errorCode = "*" }

Provide a filter name, metric namespace, metric name, and metric value of 1 on the next screen:

Finish creating the filter.

You can now create a CloudWatch alarm on this custom metric. It would notify you of KMS errors if the number of errors in a time window exceeds the threshold you specify.

Conclusion

This article was a walkthrough on how to set up alerts on KMS errors that might potentially indicate malicious activity in your account. We set up CloudTrail, CloudWatch logs, metric filter, and alarm to accomplish this.

About the Author ✍🏻

Harish KM is a Principal DevOps Engineer at QloudX & a top-ranked AWS Ambassador since 2020. 👨🏻‍💻

With over a decade of industry experience as everything from a full-stack engineer to a cloud architect, Harish has built many world-class solutions for clients around the world! 👷🏻‍♂️

With over 20 certifications in cloud (AWS, Azure, GCP), containers (Kubernetes, Docker) & DevOps (Terraform, Ansible, Jenkins), Harish is an expert in a multitude of technologies. 📚

These days, his focus is on the fascinating world of DevOps & how it can transform the way we do things! 🚀