Access Amazon RDS across VPCs using Amazon PrivateLink and Network Load Balancer

{"value":"<p>In this post, we provide a solution to access <a href=\\"https://aws.amazon.com/rds/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon Relational Database Service</a> ([Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail)) across AWS accounts and VPCs, without using <a href=\\"https://docs.aws.amazon.com/vpc/latest/userguide/vpc-peering.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">VPC peering</a> with <a href=\\"https://aws.amazon.com/vpc/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon Virtual Private Cloud</a> ([Amazon VPC](https://aws.amazon.com/cn/vpc/?trk=cndc-detail)) or <a href=\\"https://aws.amazon.com/transit-gateway/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">AWS Transit Gateway</a>.</p>\\n\\n<p>We use <a href=\\"https://aws.amazon.com/privatelink/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">AWS PrivateLink</a> and <a href=\\"https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Network Load Balancer</a> to redirect database traffic to [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail), <a href=\\"https://aws.amazon.com/rds/aurora/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon Aurora</a>, or <a href=\\"https://aws.amazon.com/rds/proxy/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon RDS Proxy</a>. We also use <a href=\\"https://aws.amazon.com/lambda/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">AWS Lambda</a> to automate Network Load Balancer target group IP address synchronization in the event of RDS failover.</p>\\n\\n<p>AWS PrivateLink provides private connectivity between VPCs, AWS services, and your on-premises networks, without exposing your traffic to the public internet. AWS PrivateLink makes it simple to connect services across different accounts and VPCs to significantly simplify your network architecture.</p>\n\n<p>Network Load Balancer functions at the fourth layer of the Open Systems Interconnection (OSI) model. It can handle millions of requests per second. After the load balancer receives a connection request, it selects a target from the target group for the default rule. It attempts to relay a TCP connection to the selected target on the port specified in the listener configuration.</p>\n\n<p>Failover is a high availability feature that replaces a database instance with another one when the original instance becomes unavailable. A failover might happen because of a problem with a database instance. It might also be part of normal maintenance procedures, such as during a database upgrade. Failover applies to RDS DB instances in a Multi-AZ configuration, and Aurora DB clusters with one or more reader instances in addition to the writer instance. <a href=\\"https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Target groups</a> for Network Load Balancer use IPv4 and IPv6 target IP addresses. [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) and Aurora change their IP address in the event of database failover. Therefore, database clients connecting to [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) using a Network Load Balancer endpoint receive an error if the RDS database has failed over and the new database instance IP address hasn’t been updated to the Network Load Balancer target group.</p>\\n\\n### **Why Network Load Balancer for Amazon RDS?**\\n\\n<p>Let’s say an organization’s cloud infrastructure consists of multiple AWS accounts and VPCs. In this scenario, Amazon RDS or Aurora run under private subnets in one of the VPCs, and applications run in a different VPC. To access Amazon RDS across VPCs, you can either use VPC peering or AWS Transit Gateway, or you can create <a href=\\"https://docs.aws.amazon.com/vpc/latest/privatelink/endpoint-service.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">VPC endpoint services</a> using [AWS PrivateLink](https://aws.amazon.com/cn/privatelink/?trk=cndc-detail). Unlike VPC peering which logically merges two VPCs and exposes all the services across the VPCs, the VPC endpoint service option creates an [AWS PrivateLink](https://aws.amazon.com/cn/privatelink/?trk=cndc-detail) route to access only the RDS service to other [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail)-hosted VPCs. However, the VPC endpoint service talks to only the Network Load Balancer service in the RDS VPC.</p>\\n\\n### **Challenges**\\n\\n<p>Amazon provides <a href=\\"https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Concepts.AuroraHighAvailability.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">high availability</a> and automatic failover support for Aurora clusters with Regions and <a href=\\"https://aws.amazon.com/rds/features/multi-az/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Multi-AZ deployments</a> for RDS instances. [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) uses several technologies to provide failover support. The failover mechanism automatically changes the <a href=\\"https://aws.amazon.com/route53/what-is-dns/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Domain Name System</a> (DNS) record in <a href=\\"https://aws.amazon.com/route53/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon Route 53</a> for the new appropriate (writer or reader) instance. Therefore, client connections are redirected to a new primary instance immediately (or depending on the DNS TTL and <a href=\\"https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.BestPractices.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">TCP keepalive parameters settings</a>) after the failover. When you use RDS Proxy for connection pooling, RDS Proxy uses its internal mechanism to detect the failover and redirect the client request to the new writer instance. However, RDS Proxy works within the same AWS account and Region where the RDS service is hosted.</p>\\n\\n<p>If you’re using Network Load Balancer to redirect client connections to the Amazon RDS or Aurora database, you must update the IP address of the new primary instance to the Network Load Balancer target group manually when database failover occurs.</p>\n\n### **Solution overview**\n\n<p>The following diagram illustrates the solution architecture.</p>\n\n\n\n<p>The Amazon RDS connection route workflow is as follows:</p>\n\n<p>A. Database users or applications connect to Amazon RDS using VPC endpoints.<br> B. The endpoints establish the user connection to VPC endpoint services (AWS PrivateLink) in other VPCs.<br> C. The VPC endpoint services establish the connection request to the Network Load Balancer.<br> D. The Network Load Balancer forwards the connection to the RDS primary instance.</p>\n\n<p>The workflow to update the new IP to the Network Load Balancer target group is as follows:</p>\n\n<ol> \\n <li>The Amazon RDS failover process updates the new primary instance IP to Route 53.</li> \n <li>The failover process generates a failover event, which triggers an <a href=\\"https://aws.amazon.com/sns/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon Simple Notification Service</a> ([Amazon SNS](https://aws.amazon.com/cn/sns/?trk=cndc-detail)) topic.</li> \\n <li>The SNS topic has a subscription that triggers an <a href=\\"https://aws.amazon.com/lambda/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">AWS Lambda</a> function.</li> \\n <li>The Lambda function gets the current IP from Route 53, and gets the current registered IP from Network Load Balancer.</li> \n <li>The function checks if both of the IPs are the same. If not, it registers the new IP received from Route 53 to Network Load Balancer, and deregisters the old IP from Network Load Balancer.</li> \n <li>Now all the new user connections are redirected to the new primary instance.</li> \n </ol>\\n\\n<p>You use the following AWS services and features to implement this solution:</p>\n\n<ul> \\n <li><a href=\\"https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">AWS CloudFormation</a> – [AWS CloudFormation](https://aws.amazon.com/cn/cloudformation/?trk=cndc-detail) helps you model and set up your AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle. You use a template to describe your resources and their dependencies, and launch and configure them together as a stack, instead of managing resources individually.</li> \\n <li><a href=\\"https://aws.amazon.com/eventbridge/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon EventBridge</a> – [Amazon EventBridge](https://aws.amazon.com/cn/eventbridge/?trk=cndc-detail) delivers a near-real-time stream of system events that describe changes in AWS resources. For more information, refer to <a href=\\"https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Scheduled-Rule.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Creating a CloudWatch Events Rule That Triggers on a Schedule</a>.</li> \\n <li><a href=\\"https://aws.amazon.com/lambda/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">AWS Lambda</a> – Lambda is a compute service that supports running code without provisioning or managing servers.</li> \\n <li><a href=\\"https://aws.amazon.com/rds/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon RDS</a> – [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) is a managed relational database service supporting MySQL, PostgreSQL, MariaDB, Oracle, or SQL Server engines.</li> \\n <li><a href=\\"https://aws.amazon.com/rds/proxy/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon RDS Proxy</a> –RDS Proxy is a fully managed, highly available database proxy for [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) that makes applications more scalable, more resilient to database failures, and more secure.</li> \\n <li><a href=\\"https://aws.amazon.com/route53/what-is-dns/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon Route 53</a> – Route 53 is a highly available and scalable cloud DNS web service. It’s designed to give developers and businesses an extremely reliable and cost-effective way to route end-users to internet applications by translating names like www.example.com into the numeric IP addresses like 192.0.2.1 that computers use to connect to each other.</li> \\n <li><a href=\\"https://aws.amazon.com/sdk-for-python/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">AWS SDK for Python</a> (Boto3) – The SDK for Python provides a Python API for AWS infrastructure services.</li> \\n <li><a href=\\"https://aws.amazon.com/sns\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon SNS</a> – [Amazon SNS](https://aws.amazon.com/cn/sns/?trk=cndc-detail) coordinates and manages the delivery or sending of messages between publishers and clients, including web servers and email addresses. Subscribers receive all the messages published to the topics to which they subscribe, and all the subscribers to a topic receive the same messages.</li> \\n </ul>\n\n### **Prerequisites**\n\n<p>For this post, we use two AWS accounts that will be refer as a VPC-A for client account, and VPC-B for target account for RDS service.</p>\n\n<p>We assume that you have cloud infrastructures such as a VPC, subnet groups, and a security group in both AWS accounts.</p>\n\n<p>You can use this solution with Amazon RDS, Amazon Aurora, and RDS Proxy, assuming that these AWS service components are in place.</p>\n\n### **Pricing**\n\n<p>Pricing of AWS services depends on many factors, for example AWS region of service, hourly/ monthly resource price, service utilization, and data transfer by service.</p>\n\n<p>You can check the pricing in detail and calculate the pricing estimate on AWS service page or the links given below.</p>\n\n<p><a href=\\"https://aws.amazon.com/privatelink/pricing/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">AWS PrivateLink pricing</a><br> <a href=\\"https://aws.amazon.com/elasticloadbalancing/pricing/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Elastic Load Balancing pricing</a><br> <a href=\\"https://aws.amazon.com/rds/pricing/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Amazon RDS pricing</a><br> <a href=\\"https://aws.amazon.com/lambda/pricing/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">AWS Lambda pricing</a></p>\\n\\n### **Create an IAM role in VPC-A**\\n\\n<p>We create an IAM role in client account, later we will use the Amazon Resource Name (ARN) of the role to establish trust relationship with Amazon RDS VPC.</p>\n\n<p>To create an IAM role in VPC-A, you <a href=\\"https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">create an IAM role</a> with AssumeRole trust relationships. Then you attach the following IAM policy with least required privileges to the role:</p>\\n\\n<div class=\\"hide-language\\"> \\n <div class=\\"code-toolbar\\"><pre class=\\" language-json\\"><code class=\\" language-json\\">{\\n \\"Version\\": \\"2012-10-17\\",\\n \\"Statement\\": [\\n {\\n \\"Sid\\": \\"VisualEditor0\\",\\n \\"Effect\\": \\"Allow\\",\\n \\"Action\\": [\\n \\"ec2:Describe*\\",\\n \\"ec2:CreateTags\\",\\n \\"ec2:CreateVpcEndpoint\\",\\n \\"route53:AssociateVPCWithHostedZone\\"\\n ],\\n \\"Resource\\": \\"*\\"\\n }\\n ]\\n}</code></pre><div class=\\"toolbar\\"><div class=\\"toolbar-item\\"><span></span></div></div></div> \\n </div>\n\n### **Launch a CloudFormation stack in VPC-B**\n\n<p>You can deploy this solution using a CloudFormation template. For more information, refer to <a href=\\"https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Creating a stack on the AWS CloudFormation console</a>.</p>\\n\\n<ol> \\n <li>Download the CloudFormation template, <code>CrossAccountRDSAccess.yml</code>, from the <a href=\\"https://github.com/aws-samples/amazon-rds-crossaccount-access/blob/main/CrossAccountRDSAccess.yml\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">GitHub repository</a>.</li> \\n <li>Open the AWS CloudFormation console in the same Region where Amazon RDS is running.</li> \n <li><a href=\\"https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-console-create-stack-template.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">Deploy</a> the template <code>CrossAccountRDSAccess.yml</code>.</li> \\n <li>For <strong>Stack name</strong>, enter a name (for this post, RDSAccess).</li> \\n <li>For <strong>Lambda Function Name</strong>, enter a name for your Lambda function.</li> \\n <li>For <strong>Select RDS Type</strong>, specify if you’re using an Aurora cluster or RDS instance.</li> \\n <li>For <strong>Enter (Amazon Aurora/Amazon RDS/Amazon RDS Proxy) endpoint</strong>, enter an RDS writer endpoint.</li> \\n <li>For <strong>Enter RDS Port</strong>, enter an RDS port number (This port number is used in Network Load Balancer to redirect connections to [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail)).</li> \\n <li>For <strong>Select VPC from the list</strong>, choose the VPC name or ID where you want to deploy the stack.</li> \\n <li>For <strong>Select Subnets</strong>, choose the private subnets for your resources.<br> Make sure that the subnets belong to the VPC that you specified.</li> \n <li>For <strong>Select Security Group for Lambda</strong>, choose the appropriate security group for Lambda.</li> \\n <li>For<strong> Provide Role ARN of target Account/VPC</strong>, enter the ARN of the role that you created in the VPC-A account. (The role ARN is used to establish a trust relationship in the endpoint).</li> \\n <li>Chose <strong>Next</strong>.</li> \\n <li>Select the acknowledgement check box and choose <strong>Create stack</strong>. (CFT Stack creation will take up to 10 minutes to complete).</li> \\n <li>After the stack is created, open the Lambda console and perform a test run to update the RDS primary instance IP address to the Network Load Balancer.</li> \n <li>On the Amazon VPC console, choose <strong>Endpoint services</strong> and copy the service name to use in the next steps.</li> \\n </ol>\n\n\n<p id=\\"caption-attachment-23152\\" class=\\"wp-caption-text\\">Sample of CloudFormation parameters</p>\n\n### **Create the VPC endpoint in VPC-A**\n\n<p>To create your VPC endpoint in VPC-A, complete the following steps:</p>\n\n<ol> \\n <li>Sign in to your account for VPC-A.</li> \n <li>Switch to the role you created earlier.</li> \n <li>On the Amazon VPC console, choose <strong>Endpoints</strong> in the navigation pane.</li> \\n <li>Choose <strong>Create endpoint</strong>.</li> \\n <li>Enter a name.</li> \n <li>Select <strong>Other endpoint services</strong> and enter the endpoint service name you copied in the last section.</li> \\n <li>Choose <strong>Verify service</strong>.<br> If you followed all the preceding steps correctly, you should get a message that the service name is verified.</li> \n <li>Choose the appropriate VPC.</li> \n <li>Choose at least two private subnets in different Availability Zones for high availability.</li> \n <li>Choose the appropriate security groups for the endpoint.</li> \n <li>Add optional tags.</li> \n <li>Choose <strong>Create endpoint</strong>.</li> \\n </ol>\n\n<p>When the endpoint is ready, you can retrieve the DNS names under the endpoint details. You use common DNS names (first on the list) to connect to Amazon RDS from VPC-A.</p>\n\n<p>For an example, to access Amazon RDS for PostgreSQL from VPC-A account. You may install PostgreSQL client on your EC2 instance in VPC-A account and access the database using VPC Endpoint as hostname.</p>\n\n<div class=\\"hide-language\\"> \\n <div class=\\"code-toolbar\\"><pre class=\\" language-bash\\"><code class=\\" language-bash\\">psql -p <span class=\\"token number\\">5432</span> -d <span class=\\"token operator\\">&lt;</span>dbname<span class=\\"token operator\\">&gt;</span> -U <span class=\\"token operator\\">&lt;</span>dbuser<span class=\\"token operator\\">&gt;</span> <span class=\\"token punctuation\\">\\\\</span> \n-h vpce-xxxxxxxxxxx-xxxxxxxx.vpce-svc-xxxxxxxxxxxxxx.us-east-1.vpce.amazonaws.com\n</code></pre><div class=\\"toolbar\\"><div class=\\"toolbar-item\\"><span></span></div></div></div> \\n </div>\n\n\n\n### **Limitations**\n\n<p>This solution works across AWS accounts and VPCs within the same Region. To make this solution works across Regions, use <a href=\\"https://aws.amazon.com/about-aws/whats-new/2019/03/aws-privatelink-now-supports-access-over-vpc-peering/\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">AWS PrivateLink access over VPC peering</a>.</p>\\n\\n<p>Depending on DNS TTL or <a href=\\"https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.BestPractices.html\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\">TCP keepalive parameter settings</a> on the client side, client traffic may still route to previous primary IP address for a while. Therefore, if the client session is connected to the reader instance (the previous primary instance, which is now the reader instance after the failover) and tries to run, the DML operation receives an error: <code>ERROR: cannot execute INSERT in a read-only transaction</code>.</p>\\n\\n### **Conclusion**\\n\\n<p>Amazon RDS Multi-AZ database deployment makes Amazon RDS highly available, and RDS failover handles Route 53 updates. RDS Proxy is also a highly available database proxy service that makes your applications more scalable and resilient to database failover, and doesn’t requires any additional mechanisms for handling RDS failover.</p>\n\n<p>In this post, you learned how to implement a custom solution to connect to Amazon RDS hosted under different AWS accounts or VPCs, without VPC peering. The solution is also capable of handling database failovers.</p>\n\n<p>Leave your feedback in the comments section to further improve this post.</p>\n\n#### **About the author**\n\n\n\n<p><strong>Jay Singh</strong> is a Database Consultant&nbsp;with the Professional Services Team at Amazon Web Services. He works as database migration specialist to help Amazon customers to move their on-premises database environment to AWS cloud database solutions.</p>","render":"<p>In this post, we provide a solution to access <a href=\\"https://aws.amazon.com/rds/\\" target=\\"_blank\\">Amazon Relational Database Service</a> ([Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail)) across AWS accounts and VPCs, without using <a href=\\"https://docs.aws.amazon.com/vpc/latest/userguide/vpc-peering.html\\" target=\\"_blank\\">VPC peering</a> with <a href=\\"https://aws.amazon.com/vpc/\\" target=\\"_blank\\">Amazon Virtual Private Cloud</a> ([Amazon VPC](https://aws.amazon.com/cn/vpc/?trk=cndc-detail)) or <a href=\\"https://aws.amazon.com/transit-gateway/\\" target=\\"_blank\\">AWS Transit Gateway</a>.</p>\\n<p>We use <a href=\\"https://aws.amazon.com/privatelink/\\" target=\\"_blank\\">AWS PrivateLink</a> and <a href=\\"https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html\\" target=\\"_blank\\">Network Load Balancer</a> to redirect database traffic to [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail), <a href=\\"https://aws.amazon.com/rds/aurora/\\" target=\\"_blank\\">Amazon Aurora</a>, or <a href=\\"https://aws.amazon.com/rds/proxy/\\" target=\\"_blank\\">Amazon RDS Proxy</a>. We also use <a href=\\"https://aws.amazon.com/lambda/\\" target=\\"_blank\\">AWS Lambda</a> to automate Network Load Balancer target group IP address synchronization in the event of RDS failover.</p>\\n<p>AWS PrivateLink provides private connectivity between VPCs, AWS services, and your on-premises networks, without exposing your traffic to the public internet. AWS PrivateLink makes it simple to connect services across different accounts and VPCs to significantly simplify your network architecture.</p>\n<p>Network Load Balancer functions at the fourth layer of the Open Systems Interconnection (OSI) model. It can handle millions of requests per second. After the load balancer receives a connection request, it selects a target from the target group for the default rule. It attempts to relay a TCP connection to the selected target on the port specified in the listener configuration.</p>\n<p>Failover is a high availability feature that replaces a database instance with another one when the original instance becomes unavailable. A failover might happen because of a problem with a database instance. It might also be part of normal maintenance procedures, such as during a database upgrade. Failover applies to RDS DB instances in a Multi-AZ configuration, and Aurora DB clusters with one or more reader instances in addition to the writer instance. <a href=\\"https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html\\" target=\\"_blank\\">Target groups</a> for Network Load Balancer use IPv4 and IPv6 target IP addresses. [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) and Aurora change their IP address in the event of database failover. Therefore, database clients connecting to [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) using a Network Load Balancer endpoint receive an error if the RDS database has failed over and the new database instance IP address hasn’t been updated to the Network Load Balancer target group.</p>\\n<h3><a id=\\"Why_Network_Load_Balancer_for_Amazon_RDS_10\\"></a><strong>Why Network Load Balancer for Amazon RDS?</strong></h3>\\n<p>Let’s say an organization’s cloud infrastructure consists of multiple AWS accounts and VPCs. In this scenario, Amazon RDS or Aurora run under private subnets in one of the VPCs, and applications run in a different VPC. To access Amazon RDS across VPCs, you can either use VPC peering or AWS Transit Gateway, or you can create <a href=\\"https://docs.aws.amazon.com/vpc/latest/privatelink/endpoint-service.html\\" target=\\"_blank\\">VPC endpoint services</a> using [AWS PrivateLink](https://aws.amazon.com/cn/privatelink/?trk=cndc-detail). Unlike VPC peering which logically merges two VPCs and exposes all the services across the VPCs, the VPC endpoint service option creates an [AWS PrivateLink](https://aws.amazon.com/cn/privatelink/?trk=cndc-detail) route to access only the RDS service to other [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail)-hosted VPCs. However, the VPC endpoint service talks to only the Network Load Balancer service in the RDS VPC.</p>\\n<h3><a id=\\"Challenges_14\\"></a><strong>Challenges</strong></h3>\\n<p>Amazon provides <a href=\\"https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Concepts.AuroraHighAvailability.html\\" target=\\"_blank\\">high availability</a> and automatic failover support for Aurora clusters with Regions and <a href=\\"https://aws.amazon.com/rds/features/multi-az/\\" target=\\"_blank\\">Multi-AZ deployments</a> for RDS instances. [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) uses several technologies to provide failover support. The failover mechanism automatically changes the <a href=\\"https://aws.amazon.com/route53/what-is-dns/\\" target=\\"_blank\\">Domain Name System</a> (DNS) record in <a href=\\"https://aws.amazon.com/route53/\\" target=\\"_blank\\">Amazon Route 53</a> for the new appropriate (writer or reader) instance. Therefore, client connections are redirected to a new primary instance immediately (or depending on the DNS TTL and <a href=\\"https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.BestPractices.html\\" target=\\"_blank\\">TCP keepalive parameters settings</a>) after the failover. When you use RDS Proxy for connection pooling, RDS Proxy uses its internal mechanism to detect the failover and redirect the client request to the new writer instance. However, RDS Proxy works within the same AWS account and Region where the RDS service is hosted.</p>\\n<p>If you’re using Network Load Balancer to redirect client connections to the Amazon RDS or Aurora database, you must update the IP address of the new primary instance to the Network Load Balancer target group manually when database failover occurs.</p>\n<h3><a id=\\"Solution_overview_20\\"></a><strong>Solution overview</strong></h3>\\n<p>The following diagram illustrates the solution architecture.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/35a5d92156aa4fd0a44c19b70b4bcc96_image.png\\" alt=\\"image.png\\" /></p>\n<p>The Amazon RDS connection route workflow is as follows:</p>\n<p>A. Database users or applications connect to Amazon RDS using VPC endpoints.<br> B. The endpoints establish the user connection to VPC endpoint services (AWS PrivateLink) in other VPCs.<br> C. The VPC endpoint services establish the connection request to the Network Load Balancer.<br> D. The Network Load Balancer forwards the connection to the RDS primary instance.</p>\n<p>The workflow to update the new IP to the Network Load Balancer target group is as follows:</p>\n<ol> \\n <li>The Amazon RDS failover process updates the new primary instance IP to Route 53.</li> \n <li>The failover process generates a failover event, which triggers an <a href=\\"https://aws.amazon.com/sns/\\" target=\\"_blank\\">Amazon Simple Notification Service</a> ([Amazon SNS](https://aws.amazon.com/cn/sns/?trk=cndc-detail)) topic.</li> \\n <li>The SNS topic has a subscription that triggers an <a href=\\"https://aws.amazon.com/lambda/\\" target=\\"_blank\\">AWS Lambda</a> function.</li> \\n <li>The Lambda function gets the current IP from Route 53, and gets the current registered IP from Network Load Balancer.</li> \n <li>The function checks if both of the IPs are the same. If not, it registers the new IP received from Route 53 to Network Load Balancer, and deregisters the old IP from Network Load Balancer.</li> \n <li>Now all the new user connections are redirected to the new primary instance.</li> \n </ol>\\n<p>You use the following AWS services and features to implement this solution:</p>\n<ul> \\n <li><a href=\\"https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html\\" target=\\"_blank\\">AWS CloudFormation</a> – [AWS CloudFormation](https://aws.amazon.com/cn/cloudformation/?trk=cndc-detail) helps you model and set up your AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle. You use a template to describe your resources and their dependencies, and launch and configure them together as a stack, instead of managing resources individually.</li> \\n <li><a href=\\"https://aws.amazon.com/eventbridge/\\" target=\\"_blank\\">Amazon EventBridge</a> – [Amazon EventBridge](https://aws.amazon.com/cn/eventbridge/?trk=cndc-detail) delivers a near-real-time stream of system events that describe changes in AWS resources. For more information, refer to <a href=\\"https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Scheduled-Rule.html\\" target=\\"_blank\\">Creating a CloudWatch Events Rule That Triggers on a Schedule</a>.</li> \\n <li><a href=\\"https://aws.amazon.com/lambda/\\" target=\\"_blank\\">AWS Lambda</a> – Lambda is a compute service that supports running code without provisioning or managing servers.</li> \\n <li><a href=\\"https://aws.amazon.com/rds/\\" target=\\"_blank\\">Amazon RDS</a> – [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) is a managed relational database service supporting MySQL, PostgreSQL, MariaDB, Oracle, or SQL Server engines.</li> \\n <li><a href=\\"https://aws.amazon.com/rds/proxy/\\" target=\\"_blank\\">Amazon RDS Proxy</a> –RDS Proxy is a fully managed, highly available database proxy for [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) that makes applications more scalable, more resilient to database failures, and more secure.</li> \\n <li><a href=\\"https://aws.amazon.com/route53/what-is-dns/\\" target=\\"_blank\\">Amazon Route 53</a> – Route 53 is a highly available and scalable cloud DNS web service. It’s designed to give developers and businesses an extremely reliable and cost-effective way to route end-users to internet applications by translating names like www.example.com into the numeric IP addresses like 192.0.2.1 that computers use to connect to each other.</li> \\n <li><a href=\\"https://aws.amazon.com/sdk-for-python/\\" target=\\"_blank\\">AWS SDK for Python</a> (Boto3) – The SDK for Python provides a Python API for AWS infrastructure services.</li> \\n <li><a href=\\"https://aws.amazon.com/sns\\" target=\\"_blank\\">Amazon SNS</a> – [Amazon SNS](https://aws.amazon.com/cn/sns/?trk=cndc-detail) coordinates and manages the delivery or sending of messages between publishers and clients, including web servers and email addresses. Subscribers receive all the messages published to the topics to which they subscribe, and all the subscribers to a topic receive the same messages.</li> \\n </ul>\n<h3><a id=\\"Prerequisites_54\\"></a><strong>Prerequisites</strong></h3>\\n<p>For this post, we use two AWS accounts that will be refer as a VPC-A for client account, and VPC-B for target account for RDS service.</p>\n<p>We assume that you have cloud infrastructures such as a VPC, subnet groups, and a security group in both AWS accounts.</p>\n<p>You can use this solution with Amazon RDS, Amazon Aurora, and RDS Proxy, assuming that these AWS service components are in place.</p>\n<h3><a id=\\"Pricing_62\\"></a><strong>Pricing</strong></h3>\\n<p>Pricing of AWS services depends on many factors, for example AWS region of service, hourly/ monthly resource price, service utilization, and data transfer by service.</p>\n<p>You can check the pricing in detail and calculate the pricing estimate on AWS service page or the links given below.</p>\n<p><a href=\\"https://aws.amazon.com/privatelink/pricing/\\" target=\\"_blank\\">AWS PrivateLink pricing</a><br> <a href=\\"https://aws.amazon.com/elasticloadbalancing/pricing/\\" target=\\"_blank\\">Elastic Load Balancing pricing</a><br> <a href=\\"https://aws.amazon.com/rds/pricing/\\" target=\\"_blank\\">Amazon RDS pricing</a><br> <a href=\\"https://aws.amazon.com/lambda/pricing/\\" target=\\"_blank\\">AWS Lambda pricing</a></p>\\n<h3><a id=\\"Create_an_IAM_role_in_VPCA_70\\"></a><strong>Create an IAM role in VPC-A</strong></h3>\\n<p>We create an IAM role in client account, later we will use the Amazon Resource Name (ARN) of the role to establish trust relationship with Amazon RDS VPC.</p>\n<p>To create an IAM role in VPC-A, you <a href=\\"https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html\\" target=\\"_blank\\">create an IAM role</a> with AssumeRole trust relationships. Then you attach the following IAM policy with least required privileges to the role:</p>\\n<div> \\n <div><pre><code>{\\n \\"Version\\": \\"2012-10-17\\",\\n \\"Statement\\": [\\n {\\n \\"Sid\\": \\"VisualEditor0\\",\\n \\"Effect\\": \\"Allow\\",\\n \\"Action\\": [\\n \\"ec2:Describe*\\",\\n \\"ec2:CreateTags\\",\\n \\"ec2:CreateVpcEndpoint\\",\\n \\"route53:AssociateVPCWithHostedZone\\"\\n ],\\n \\"Resource\\": \\"*\\"\\n }\\n ]\\n}</code></pre><div><div><span></span></div></div></div> \\n </div>\n<h3><a id=\\"Launch_a_CloudFormation_stack_in_VPCB_95\\"></a><strong>Launch a CloudFormation stack in VPC-B</strong></h3>\\n<p>You can deploy this solution using a CloudFormation template. For more information, refer to <a href=\\"https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html\\" target=\\"_blank\\">Creating a stack on the AWS CloudFormation console</a>.</p>\\n<ol> \\n <li>Download the CloudFormation template, <code>CrossAccountRDSAccess.yml</code>, from the <a href=\\"https://github.com/aws-samples/amazon-rds-crossaccount-access/blob/main/CrossAccountRDSAccess.yml\\" target=\\"_blank\\">GitHub repository</a>.</li> \\n <li>Open the AWS CloudFormation console in the same Region where Amazon RDS is running.</li> \n <li><a href=\\"https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-console-create-stack-template.html\\" target=\\"_blank\\">Deploy</a> the template <code>CrossAccountRDSAccess.yml</code>.</li> \\n <li>For <strong>Stack name</strong>, enter a name (for this post, RDSAccess).</li> \\n <li>For <strong>Lambda Function Name</strong>, enter a name for your Lambda function.</li> \\n <li>For <strong>Select RDS Type</strong>, specify if you’re using an Aurora cluster or RDS instance.</li> \\n <li>For <strong>Enter (Amazon Aurora/Amazon RDS/Amazon RDS Proxy) endpoint</strong>, enter an RDS writer endpoint.</li> \\n <li>For <strong>Enter RDS Port</strong>, enter an RDS port number (This port number is used in Network Load Balancer to redirect connections to [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail)).</li> \\n <li>For <strong>Select VPC from the list</strong>, choose the VPC name or ID where you want to deploy the stack.</li> \\n <li>For <strong>Select Subnets</strong>, choose the private subnets for your resources.<br> Make sure that the subnets belong to the VPC that you specified.</li> \n <li>For <strong>Select Security Group for Lambda</strong>, choose the appropriate security group for Lambda.</li> \\n <li>For<strong> Provide Role ARN of target Account/VPC</strong>, enter the ARN of the role that you created in the VPC-A account. (The role ARN is used to establish a trust relationship in the endpoint).</li> \\n <li>Chose <strong>Next</strong>.</li> \\n <li>Select the acknowledgement check box and choose <strong>Create stack</strong>. (CFT Stack creation will take up to 10 minutes to complete).</li> \\n <li>After the stack is created, open the Lambda console and perform a test run to update the RDS primary instance IP address to the Network Load Balancer.</li> \n <li>On the Amazon VPC console, choose <strong>Endpoint services</strong> and copy the service name to use in the next steps.</li> \\n </ol>\n<p><img src=\\"https://dev-media.amazoncloud.cn/6722f0fee39e4cd0935ab060b9c9d315_image.png\\" alt=\\"image.png\\" /></p>\n<p>Sample of CloudFormation parameters</p>\n<h3><a id=\\"Create_the_VPC_endpoint_in_VPCA_121\\"></a><strong>Create the VPC endpoint in VPC-A</strong></h3>\\n<p>To create your VPC endpoint in VPC-A, complete the following steps:</p>\n<ol> \\n <li>Sign in to your account for VPC-A.</li> \n <li>Switch to the role you created earlier.</li> \n <li>On the Amazon VPC console, choose <strong>Endpoints</strong> in the navigation pane.</li> \\n <li>Choose <strong>Create endpoint</strong>.</li> \\n <li>Enter a name.</li> \n <li>Select <strong>Other endpoint services</strong> and enter the endpoint service name you copied in the last section.</li> \\n <li>Choose <strong>Verify service</strong>.<br> If you followed all the preceding steps correctly, you should get a message that the service name is verified.</li> \n <li>Choose the appropriate VPC.</li> \n <li>Choose at least two private subnets in different Availability Zones for high availability.</li> \n <li>Choose the appropriate security groups for the endpoint.</li> \n <li>Add optional tags.</li> \n <li>Choose <strong>Create endpoint</strong>.</li> \\n </ol>\n<p>When the endpoint is ready, you can retrieve the DNS names under the endpoint details. You use common DNS names (first on the list) to connect to Amazon RDS from VPC-A.</p>\n<p>For an example, to access Amazon RDS for PostgreSQL from VPC-A account. You may install PostgreSQL client on your EC2 instance in VPC-A account and access the database using VPC Endpoint as hostname.</p>\n<div> \\n <div><pre><code>psql -p <span>5432</span> -d <span>&lt;</span>dbname<span>&gt;</span> -U <span>&lt;</span>dbuser<span>&gt;</span> <span>\\\\</span> \n-h vpce-xxxxxxxxxxx-xxxxxxxx.vpce-svc-xxxxxxxxxxxxxx.us-east-1.vpce.amazonaws.com\n</code></pre><div><div><span></span></div></div></div> \\n </div>\n<p><img src=\\"https://dev-media.amazoncloud.cn/b5dc6cec4fc24186868378e9eb07aa22_image.png\\" alt=\\"image.png\\" /></p>\n<h3><a id=\\"Limitations_152\\"></a><strong>Limitations</strong></h3>\\n<p>This solution works across AWS accounts and VPCs within the same Region. To make this solution works across Regions, use <a href=\\"https://aws.amazon.com/about-aws/whats-new/2019/03/aws-privatelink-now-supports-access-over-vpc-peering/\\" target=\\"_blank\\">AWS PrivateLink access over VPC peering</a>.</p>\\n<p>Depending on DNS TTL or <a href=\\"https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.BestPractices.html\\" target=\\"_blank\\">TCP keepalive parameter settings</a> on the client side, client traffic may still route to previous primary IP address for a while. Therefore, if the client session is connected to the reader instance (the previous primary instance, which is now the reader instance after the failover) and tries to run, the DML operation receives an error: <code>ERROR: cannot execute INSERT in a read-only transaction</code>.</p>\\n<h3><a id=\\"Conclusion_158\\"></a><strong>Conclusion</strong></h3>\\n<p>Amazon RDS Multi-AZ database deployment makes Amazon RDS highly available, and RDS failover handles Route 53 updates. RDS Proxy is also a highly available database proxy service that makes your applications more scalable and resilient to database failover, and doesn’t requires any additional mechanisms for handling RDS failover.</p>\n<p>In this post, you learned how to implement a custom solution to connect to Amazon RDS hosted under different AWS accounts or VPCs, without VPC peering. The solution is also capable of handling database failovers.</p>\n<p>Leave your feedback in the comments section to further improve this post.</p>\n<h4><a id=\\"About_the_author_166\\"></a><strong>About the author</strong></h4>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/f35f606c9aaf44fe81c0332fc629025f_image.png\\" alt=\\"image.png\\" /></p>\n<p><strong>Jay Singh</strong> is a Database Consultant&nbsp;with the Professional Services Team at Amazon Web Services. He works as database migration specialist to help Amazon customers to move their on-premises database environment to AWS cloud database solutions.</p>"}