Serverless functions & relational databases do not mix! If you use RDS databases with Lambda functions, there are a lot of features & optimizations you’re missing out on.

Traditional relational databases were designed to be used by applications via relatively long-lived connections with several queries coming in over the same database connection over a few minutes or even hours. Traditional server-based applications like website backends optimize this further by maintaining a pool of database connections & reusing connections from this pool to talk to the database when needed.

Serverless is the polar opposite of this. Lambda functions are very short-lived themselves & might open a new connection to the database on every invocation. These connections are typically used for just 1 query & closed within seconds. Clearly, there’s scope for improvement here.

The standard best practice when maintaining database connections in Lambda functions is to open the database connection outside the main Lambda function handler method. This way, you can at least reuse a single database connection for multiple invocations of a Lambda function. But of course, this is still far from ideal. Multiple Lambda functions connecting to the same database cannot share database connections & every Lambda needs to open a new connection if it’s been idle for a few minutes.

Opening database connections is time & resource-intensive both for the application involved (serverless or otherwise) & the database in question. Reusing an already open database connection, on the other hand, is very light on resources.

Wouldn’t it be great if you could get the benefits of a connection pool with Lambda functions? That’s where RDS proxies come in!

What are RDS Proxies?

How would you build your own connection pool for Lambda functions? You could use your preferred database client library to maintain a connection pool in your Lambda code itself like in traditional applications. But that doesn’t help much, mainly because no other Lambda function can share this connection pool.

The next obvious solution is to have a server always running that keeps a bunch of database connections open & provides these to Lambda functions. Lambda functions use this server as a forward proxy to connect your database. Now all Lambda functions can share the connection pool!

You don’t want this 1 server to become a single point of failure in your architecture. So you make it into an auto-scaling group of servers with a bit more intelligence built-in to optimize database access. Well, that’s exactly what an RDS proxy is!

RDS Proxy is a pool of serverless, auto-scaling compute resources, deployed over multiple availability zones, that establishes a database connection pool & reuses connections in this pool without the memory and CPU overhead of opening a new database connection each time. RDS Proxy is fully compatible with MySQL & PostgreSQL. You can enable RDS Proxy for most applications with no code changes.

Why use RDS Proxies?

There are many obvious (& some not so obvious) benefits of using RDS proxies instead of connecting to RDS databases directly:

• You can pool & share database connections to improve your application’s scalability.
• In case of database failures, the RDS proxy will automatically connect to a standby database instance without dropping your app’s connections to the proxy.
  • This makes your app very resilient & takes away the need for you to implement complex connection retry logic in your app.
• RDS proxies can enforce IAM authentication for databases if required.
• RDS proxies also force you to store your database credentials in AWS Secrets Manager, making them much more secure than storing them in say, Lambda environment variables.
  • Secrets Manager also comes with cool features like automatic periodic database credential rotation without affecting your application in any way.
• If a surge in traffic to your app causes a surge in database usage, RDS proxies can absorb it & keep your database happy.
  • An RDS proxy will either queue or throttle application connections that can’t be served immediately. Your app will keep scaling seamlessly except for a small latency spike. That’s way better than failing queries or overwhelming the database.
• You can configure RDS proxies to accept a max number of application connections. If connection requests exceed this limit, the RDS proxy rejects the connection, thus maintaining ideal performance for the existing connections.

How to Use RDS Proxies?

Now that you’re convinced of the benefits of RDS proxies, let’s see how to set 1 up. We’ll start with a Lambda function that connects to an RDS Aurora MySQL database directly (without a proxy). We’ll then see how to convert this into a Lambda that uses an RDS proxy. Along the way, we’ll see how to store the database credentials in Secrets Manager & how to create the proxy itself. So let’s get started!

The Lambda Function

Here’s is the code of a simple Python 3.8 Lambda function that connects to an RDS Aurora MySQL database directly & queries it for some data:

import pymysql

try:
    conn = pymysql.connect(user = 'admin', passwd = 'adminadmin', db = 'my_database',
        host = 'my-database.cluster-a1b2c3d4e5f6.ap-south-1.rds.amazonaws.com')
except pymysql.MySQLError as e:
    print(e)
    exit()

def handler(event, context):
    with conn.cursor() as cur:
        cur.execute("select * from employee")
        for row in cur:
            print(row)

Notice how the database connection is created outside the Lambda handler so subsequent invocations of the Lambda function can reuse it.

Test out this function to make sure it works as expected & then move on to creating the secret & the proxy.

The Secret & The Proxy

Before creating the RDS proxy, you need to store the database credentials in a secret in AWS Secrets Manager. Start by visiting https://console.aws.amazon.com/secretsmanager/home#!/newSecret?step=selectSecret, select RDS, enter the username & password & select your RDS database. Next, name the secret, configure rotation & finish creating it.

Now to create the RDS proxy, go to https://console.aws.amazon.com/rds/home#create-proxy:, name the proxy, select the database & the secret & create the proxy. The security groups you select should allow your Lambda functions to connect to the proxy.

Now go to your Lambda function’s configuration & add the database proxy to it. Go to https://console.aws.amazon.com/lambda/home#/add/database-proxy & simply select the RDS proxy you created above. When you save this config change, the AWS console will update your Lambda function’s execution role to include the permissions needed to connect to the RDS proxy.

The only change remaining now is to update the Lambda function’s code to use the RDS proxy’s endpoint instead of the database’s. The updated Lambda function code is as follows:

import pymysql

try:
    conn = pymysql.connect(user = 'admin', passwd = 'adminadmin', db = 'my_database',
        host = 'my-rds-proxy.proxy-a1b2c3d4e5f6.ap-south-1.rds.amazonaws.com')
except pymysql.MySQLError as e:
    print(e)
    exit()

def handler(event, context):
    with conn.cursor() as cur:
        cur.execute("select * from employee")
        for row in cur:
            print(row)

The only change is the database host endpoint. Everything else stays as it was. This was just for demo but in real apps, the endpoint will most probably be an environment variable so no code change will be required in switching to an RDS proxy.

Conclusion

That brings us to the end of this journey. We saw how immensely beneficial RDS proxies are, especially for use with serverless services like Lambda functions. You can also use them with traditional apps to eliminate database connection management code from your apps.

About the Author ✍🏻

Harish KM is a Principal DevOps Engineer at QloudX & a top-ranked AWS Ambassador since 2020. 👨🏻‍💻

With over a decade of industry experience as everything from a full-stack engineer to a cloud architect, Harish has built many world-class solutions for clients around the world! 👷🏻‍♂️

With over 20 certifications in cloud (AWS, Azure, GCP), containers (Kubernetes, Docker) & DevOps (Terraform, Ansible, Jenkins), Harish is an expert in a multitude of technologies. 📚

These days, his focus is on the fascinating world of DevOps & how it can transform the way we do things! 🚀