Here’s How To Setup AWS RDS-proxy With Lambda Using Serverless Framework

AWS Relational Database Service (RDS) is a managed database service that was launched almost 10 years ago and over these years application patterns have evolved, which has led to various challenges when it comes to interacting with the database.

My traditional approach towards application development using AWS RDS in serverless -

The database question has been one of the bigger issues in Serverless for a few years now. I have seen many people writing on this numerous times, including posts on the data layer in Serverless, choosing a database in serverless. Another major challenge is “Connection limits” when working with relational databases like MySQL, PostgresSQL, etc.

Not only this, but the way I use to approach was also affecting things. Some common or maybe you can say silly mistakes were —
1. Creating database sessions for RDS in every handler.
2. Creating a pool of DB connections and creating database sessions for RDS in every handler.

Now as we know APIs in AWS are deployed on AWS Lambda and AWS lambda requests are then given to a container to handle. A container contains the code the user has provided to satisfy the query. With an increased number of requests, an increasing number of containers are created. If the number of requests reduces, the number of containers is reduced as well.
Imagine how fast the RDS connections used to get exhausted!

How this traditional approach was hampering my performance -

Hitting the maximum amount of connections will cause the database to reject new connections. This can be especially problematic when deploying application code because the code might take up more connections than usual, and fail to run when the database does not allow connections. As a result, that application failure may occur at any given point in time.
But not to worry! I have found one of the approaches amongst many others which is more native to the AWS environment.
Introducing RDS-proxy!

What is RDS-proxy?

AWS RDS Proxy (Source: AWS Documents)

RDS proxy is a fully-managed database proxy for Amazon RDS. RDS proxy works by pooling and sharing DB connections and thus makes applications more scalable as well as resilient to database failures.

How does RDS-proxy work?

RDS Proxy establishes a database connection pool and reuses connections in this pool without the memory and CPU overhead of opening a new database connection each time. To protect the database against oversubscription, you can control the number of database connections that are created.RDS Proxy queues or throttles application connections that can’t be served immediately from the pool of connections. If connection requests exceed the limits you specify, RDS Proxy rejects application connections (that is, it sheds load). At the same time, it maintains predictable performance for the load that can be served with the available capacity.

Security using RDS proxy -

AWS RDS security

RDS proxy can provide an additional layer of security between the application and the underlying database. AWS recommends enforcing IAM authentication while connecting to the proxy as that eliminates the need to specify database credentials anywhere in the code.

How to monitor RDS proxy?

RDS proxy can be monitored by using Amazon CloudWatch. CloudWatch is well integrated with RDS proxy and provides useful metrics that can be used to understanding the performance and behavior of the proxy.

Some key metrics to keep an eye are:

• DatabaseConnections: Number of database connections to the backend database
• DatabaseConnectionsCurrentlyBorrowed: Number of connections currently being used by your application. Important to set an alarm on this metric.
• DatabaseConnectionsCurrentlySessionPinned: Number of connections in the pinned state. This number should ideally be as low as possible to maximize RDS proxy performance.

Points to keep in mind.

1. RDS proxy must be in the same VPC as the database instance. The proxy cannot be publicly accessible even if the database instance is.
2. A proxy can only be associated with 1 Database instance.
3. If you are using AWS lambda along with RDS proxy, make sure your AWS lambda, RDS instance, and RDS proxy lie in the same VPC network.
4. RDS proxy is only available for PostgresSQL and Aurora DB instances.

Serverless Architecture with RDS Proxy —

Application Architecture with Serverless Framework and RDS proxy

How to implement this approach?

Pre-requisites:
1. Install NodeJs
2. Install Serverless package.
3. Install AWS CLI

Note — Here I have used a serverless “aws-python3” template. You can find the list of all the supported templates for serverless here.

Step 1: Configure AWS Secret Manager to manage RDS credentials.

a. Sign into AWS Secrets Manager and choose Store a new Secret.
b. Choose Credentials for the RDS database.
c. Enter the user name and password of the RDS instance created.
d. Select the RDS Database this secret is valid for. Choose Next.

Store a new secret

e. Enter a Secret Name and choose Next.

Save the secret

f. Accept all defaults and choose Store. Save the ARN assigned to this secret, as you need it later.

Secret Details

g. Now create an IAM role that allows RDS Proxy to read this secret. RDS Proxy uses this secret to maintain a connection pool to your database. Go to your IAM console and create a new role. Add a policy that provides secretsmanager permissions to the secret you created in the previous step. For example:

{
“Version”: “2012–10–17”,
“Statement”: [
{
“Sid”: “VisualEditor0”,
“Effect”: “Allow”,
“Action”: [
“secretsmanager:GetResourcePolicy”,
“secretsmanager:GetSecretValue”,
“secretsmanager:DescribeSecret”,
“secretsmanager:ListSecretVersionIds”
],
“Resource”: [
“arn:aws:secretsmanager:us-east-2:[your-account-number]:secret:gmao-rds-secret-YZ2MMN”
]
},
{
“Sid”: “VisualEditor1”,
“Effect”: “Allow”,
“Action”: [
“secretsmanager:GetRandomPassword”,
“secretsmanager:ListSecrets”
],
“Resource”: “*”
}
]
}

h. Add the following Trust Policy to allow RDS to assume the role. Save the role and take note of the IAM Role ARN, as you need it later.

{
“Version”: “2012–10–17”,
“Statement”: [
{
“Sid”: “”,
“Effect”: “Allow”,
“Principal”: {
“Service”: “rds.amazonaws.com”
},
“Action”: “sts:AssumeRole”
}
]
}

Step 2(a): Create and attach a proxy to a Lambda function (Manually)-

a. Use the Lambda console to Add a Database proxy to a Lambda function. Scroll to the bottom of your Lambda configuration page and choose Add Database Proxy.

Add database proxy

b. Follow the Add database proxy wizard, and fill in the Proxy Identifier and select your RDS Database. Then choose the Secrets Manager secret and the IAM role you created earlier. RDS Proxy uses this secret to connect to your database. Choose Add.

Configure database proxy

c. Wait until the status changes to “Available”.

Database proxy available

d. Choose your proxy to view the details. Note the Proxy endpoint. You need this later in the Lambda function code.

Available Proxy configurations

Now the Lambda function has permission to use the configured RDS Proxy, and you are ready to connect to the proxy.

Step 2(b): Create and attach a RDS proxy to a Lambda function (serverless framework)

a. Create the RDS proxy from the above-mentioned steps. Once the proxy is created, copy the “Proxy ARN”. Now navigate to your “serverless.yaml” file and use the following configurations.

service: rdsproxy-demo

provider:
name: aws
runtime: python3.6
timeout: 60
stage: ${opt:stage}
environment: ${file(env.yml):${self:custom.stage}}
region: ${opt:region}
iamRoleStatements:
— Effect: ‘Allow’
Action:
— ‘s3:*’
Resource: “*”
— Effect: ‘Allow’
Action:
— “rds-db:connect”
Resource: “arn:aws:rds-db:us-east-1:***********:dbuser:prx-0d1efa13172fea793/*”
vpc:
securityGroupIds:
— <secutiryGroup-id>
subnetIds:
— <subnet-id>
— <subnet-id>

functions:

demo:
handler: src/item/demo.give_demo
events:
— http:
path: /item/list_item
method: get
cors:
origin: ‘*’
headers:
— Content-Type
— X-Amz-Date
— Authorization
— X-Api-Key
— X-Amz-Security-Token
— X-Amz-User-Agent
allowCredentials: false

plugins:
— serverless-python-requirements
— serverless-offline

package:
individually: true
exclude:
— venv/**
— node_modules/**
— layer/**

custom:
stage: “${opt:stage}”
pythonRequirements:
zip: true
slim: true
noDeploy:
— botocore
— docutils
— jmespath
— pip
— python-dateutil
— s3transfer
— setuptools
— six

Conclusion -

Thus the AWS RDS proxy came to my rescue. Amazon RDS proxy is a database proxy that helps improve application availability and performance. It is particularly helpful for applications that have the following requirements:

• Unpredictable workloads
• Frequently open and close database connections
• Higher availability during transient database failures

Hope you enjoyed it! Thank You!