Understanding data transfer costs for Amazon container services

{"value":"### **Overview**\nData transfer costs can play a significant role in determining the overall design of a system. The [Amazon Elastic Container Registry (Amazon ECR)](https://aws.amazon.com/ecr/), [Amazon Elastic Container Service (Amazon ECS)](https://aws.amazon.com/ecs/), and [Amazon Elastic Kubernetes Service (Amazon EKS)](https://aws.amazon.com/eks/) can all incur data transfer charges depending on a variety of factors. It can be difficult to visualize what that means relative to an [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) or [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) deployment. This blog illustrates common deployment patterns for AWS container services and explains the resulting data transfer charges that may be incurred. Service charges are out of scope for this blog, but should be carefully considered when designing any architecture.\n\n### **Amazon ECR**\nAn Amazon ECR private registry hosts container images in a highly available and scalable architecture. There are two types of registries in Amazon ECR—private and public—and each one has different data transfer charges.\n\n#### **Amazon ECR private registry**\nYou can use an ECR private registry to manage private image repositories consisting of Docker and Open Container Initiative (OCI) images and artifacts.\n\n\n\nFigure 1. ECR private registry\n\nData transfer into the Amazon ECR private registry incurs no charge from Amazon ECR. Data transferred between Amazon ECR and other services within the same Region is free. Data transferred between Amazon ECR and other services in different Regions will be charged at Internet Data Transfer rates on both sides of the transfer. Note that this is aggregated with other outbound data transfer across multiple services, and rate tiers will apply based on the amount of data transferred.\n\nAmazon ECR offers built-in functionality to replicate container images to different locations. This could be useful for disaster recovery purposes, a promote-to-production pipeline, or to help reduce the image pull time and data transfer costs when running containers in different Regions. This data transfer is charged at the cross-Region data transfer charge rates described on the [Amazon EC2 pricing page](https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer)\n\n\n\nFigure 2. ECR public registry\n\nRefer to the [Amazon ECR pricing page](https://aws.amazon.com/ecr/pricing/) and [Amazon EC2 pricing page](https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer) for more information.\n\n#### **Amazon ECR Registry Public**\n[Amazon Elastic Container Registry Public](https://docs.aws.amazon.com/AmazonECR/latest/public/what-is-ecr.html) is a managed AWS container image registry service that is secure, scalable, and reliable. Amazon ECR supports public image repositories with resource-based permissions using [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/AmazonECR/latest/public/security_iam_service-with-iam.html) so that specific users can access your public repositories to push images. Developers can use their preferred CLI to push and pull container images and OCI-compatible artifacts. Images are publicly available to pull, either anonymously or using an Amazon ECR public authentication token.\n\nAll data transfer into Amazon ECR Public incurs no charge from Amazon ECR. Data transfer out is subject to charges when more than 5 TB/month is transferred out to non-AWS destinations, and you have authenticated to Amazon ECR with an AWS account. Up to 500 GB/month can be transferred out to non-AWS destinations to clients that have not authenticated (that is, anonymously). After that limit is reached, no further anonymous data transfer is allowed.\n\n### **Data transfer for [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail)**\nThree common deployment models for [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) are clusters with external network access, clusters without external network access, and ECS Anywhere.\n\n#### **[Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) clusters with external access**\nCompute capacity for container instances in [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) can be deployed within VPCs that allow access to the [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) service endpoint using the internet. For example, the compute capacity for container instances can be deployed in public subnets (with an [internet gateway](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html)), private subnets (with a [NAT gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html), shown in Figure 3), or can route to the internet through another [VPC](https://aws.amazon.com/vpc/) using the [AWS Transit Gateway](https://aws.amazon.com/transit-gateway/). Both [Amazon EC2](https://aws.amazon.com/ec2/) and [AWS Fargate](https://aws.amazon.com/fargate/) can be used as compute for this type of deployment, and there is no difference in data transfer costs based on which is chosen.\n\n\n\nFigure 3. ECS deployment with internet access\n\nThe following data transfer is not charged in the sample deployment:\n\n- Data transfer in from the [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) control plane (that is, responses to API calls from the data plane) and from Amazon ECR (that is, “image pulls”)\n- Communication between tasks and the Application Load Balancer\n- Communication between tasks and the database instance in the same Availability Zone\n\n\nIn this deployment, data transfer charges accrue for:\n\n- Data transfer out through the NAT gateway, including polling traffic from the ECS agent to the [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) control plane and outbound data to Amazon ECR\n- Cross-Availability Zone traffic to the database\n\n\nIt is important to note that although the NAT gateway does not charge for data transfer in, there will still be a per-GB data processing charge on data that flows through the NAT gateway, regardless of direction. The data transfer out charges are in addition to this charge. NAT gateway pricing can be found on the [Amazon VPC pricing page](https://aws.amazon.com/vpc/pricing/).\n\n#### **[Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) with no external access**\nAnother common pattern to deploy [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) workloads is to restrict all external network access (Figure 4). Because container instances in [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) clusters need external network access to communicate with the [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) service endpoint, [AWS PrivateLink](https://aws.amazon.com/privatelink/) must be used to communicate with the service endpoints. Again, both [Amazon EC2 ](https://aws.amazon.com/cn/ec2/?trk=cndc-detail)and [AWS Fargate](https://aws.amazon.com/cn/fargate/?trk=cndc-detail) can be used for the compute capacity in this type of deployment. However, there is a difference in data transfer costs based on which is chosen.\n\n\n\n\nFigure 4. ECS deployment in private network\n\nThe following data transfer is not charged in the sample deployment:\n\n- Communication between tasks and the Application Load Balancer\n- Communication between tasks and the database instance in the same Availability Zone\n\n\nIn this deployment, data transfer charges accrue for cross-AZ traffic to the database.\n\nFor clarity, the diagram depicts the [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) and Amazon ECR PrivateLink VPC interface endpoints as single objects. However, there are actually multiple endpoints required for each. For a description of endpoint requirements, consult [this AWS Compute Blog](https://aws.amazon.com/blogs/compute/setting-up-aws-privatelink-for-amazon-ecs-and-amazon-ecr/) or the [Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/vpc-endpoints.html) and [Amazon ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html) documentation.\n\nAlthough there is no data transfer fee, [AWS PrivateLink](https://aws.amazon.com/cn/privatelink/?trk=cndc-detail) imposes both a per-hour service charge and a per-GB data processing charge for data processed through each VPC endpoint, billed per Availability Zone. More details can be found on the [AWS PrivateLink pricing page](https://aws.amazon.com/privatelink/pricing/).\n\n##### **Using [AWS Fargate](https://aws.amazon.com/cn/fargate/?trk=cndc-detail) in private [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) clusters**\nIf [AWS Fargate](https://aws.amazon.com/cn/fargate/?trk=cndc-detail) provides the compute capacity for the [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) cluster, the [AWS PrivateLink](https://aws.amazon.com/cn/privatelink/?trk=cndc-detail) endpoints for [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) are not required. This eliminates the service charges—hourly and data processing—for communication to the [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) service endpoint. Note that [AWS PrivateLink](https://aws.amazon.com/cn/privatelink/?trk=cndc-detail) VPC Endpoints for Amazon ECR are still required to pull images from Amazon ECR.\n\n#### **[Amazon ECS Anywhere](https://aws.amazon.com/cn/ecs/anywhere/?trk=cndc-detail)**\n[Amazon Elastic Container Service Anywhere](https://aws.amazon.com/ecs/anywhere/) is a feature of [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) that enables you to easily run and manage container-based applications on-premises, including on your own virtual machines (VMs) and bare metal servers. With [Amazon ECS Anywhere](https://aws.amazon.com/cn/ecs/anywhere/?trk=cndc-detail), you do not need to install or operate local container orchestration software, thus reducing operational overhead. ECS Anywhere offers a completely managed solution that enables you to standardize container management across all of your environments.\n\n\n\nFigure 5. ECS Anywhere over the internet\n\n\n\nFigure 6. ECS Anywhere with Direct Connect\n\nData transfer fees are accrued based on how the customer-managed infrastructure connects to the [Amazon ECS Anywhere](https://aws.amazon.com/cn/ecs/anywhere/?trk=cndc-detail) control plane:\n\n- **If connecting via the internet (Figure 5)** – There is no charge for data transfer when communication between the [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) control plane and ECS agent occurs over the open internet.\n- **If connecting via [AWS Site-to-Site VPN](https://aws.amazon.com/vpn/) or [AWS Direct Connect](https://aws.amazon.com/directconnect/) (Figure 6)** – Standard data transfer fees (that is, “Data Transfer Out”) apply to AWS Site-to-Site VPN or [AWS Direct Connect](https://aws.amazon.com/cn/directconnect/?trk=cndc-detail) for communication between the ECS Anywhere control plane and ECS agent that occurs through Site-to-Site VPN or [AWS Direct Connect](https://aws.amazon.com/cn/directconnect/?trk=cndc-detail) link.\n\n\nRefer to the [ECS Anywhere pricing page](https://aws.amazon.com/ecs/anywhere/pricing/), [AWS Site-to-Site VPN pricing page](https://aws.amazon.com/vpn/pricing/), and the [AWS Direct Connect pricing page](https://aws.amazon.com/directconnect/pricing/) for more detail.\n\n### **Data transfer for [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail)**\nSimilar to [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail), [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) data transfer charges follow the guidelines described on the [Amazon EC2 ](https://aws.amazon.com/cn/ec2/?trk=cndc-detail)pricing page. Figure 7 represents a sample [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) workload with two pods deployed to different [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) worker nodes in different Availability Zones.\n\n\n\nFigure 7. Sample application EKS deployment\n\nThe following data transfer is not charged in this example:\n\n- Traffic in/out of control plane (not shown on diagram)\n- Image pulls from ECR within the same Region (not shown on diagram)\n- Communication between pods and the Application Load Balancer\n- Communication between pods and the database instance in the same Availability Zone\n\n\nThe following data transfer charges accrue for this example:\n\n- Data transfer between the Kubernetes pod and the database in a different Availability Zone\n- Data out (egress) from the Application Load Balancer\n- Communication between pods in different Availability Zones\n\n\nThere are several other configuration options and deployment strategies to consider in an [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) deployment relative to data transfer costs: cluster access (public/private), pod-to-pod communication, and communication between the pods and the load balancer.\n\n#### **Public EKS clusters**\nPublic EKS clusters (Figure 8) have the public endpoint access enabled and the private endpoint disabled. Communication between the control plane and worker nodes will exit the VPC based on the VPC’s routing. For worker nodes in private subnets, communication will traverse a NAT gateway and exit via an internet gateway. There is no data transfer charge associated with this. However, there is a per-GB NAT gateway data processing fee. In addition, worker nodes not in the same Availability Zone as the NAT gateway will incur a per-GB data transfer charge for cross-Availability Zone traffic. For worker nodes in public subnets, communication will exit the internet gateway and will not be charged.\n\n\n\nFigure 8. Public EKS cluster\n\n#### **Private EKS clusters**\nIn a cluster with private [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) endpoints (Figure 9), worker nodes will communicate with private endpoints. The Kubernetes API Server Endpoint URL will resolve to [elastic network interfaces](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html) (ENIs) within the (customer) VPC. Worker nodes inside the VPC will communicate with these ENIs. In this deployment model, there is a chance that a worker node communicates with an ENI in a different Availability Zone. This communication would be subject to a cross-Availability Zone data transfer charge.\n\n\n\nFigure 9. Private EKS cluster\n\n#### **Pod-to-pod communication**\nKubernetes applications often leverage communication between containers and between pods (Figure 10).\n\n\n\nFigure 10. Pod-to-pod communication\n\nContainers within the same pod are guaranteed to be on the same worker node and will communicate over the loopback device, resulting in no data transfer charges.\nData transfer charges between pods depend on the pod placement:\n\n- There is no data transfer charge for pods deployed on the same node or within the same Availability Zone\n- Pods that communicate and are placed in different Availability Zones will incur a data transfer charge\n\n\n#### **Load balancer-to-pod communication**\n[Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) workloads often include a load balancer to distribute the traffic evenly across pods. Pods are deployed alongside Kubernetes Service and Ingress objects. The add-on [AWS Load Balancer Controller](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) monitors these objects and automatically provisions the AWS load balancer resource. The path traffic takes (and resulting data transfer charges) depend on how the Service and Ingress objects are configured. There are two common methods of configuration.\n\nThe first method involves having a target group consisting of all worker nodes within the cluster (Figure 11). This grouping includes Service:LoadBalancer, Service:NodePort, and Ingress TargetType: Instance. Here, an ephemeral port (NodePort) is opened on each node in the cluster, and traffic is distributed evenly across all nodes. If the destination pod is on the node, no additional data transfer occurs. If the destination pod was scheduled on a different node, data transfer charges may accrue if the target pod is in a different Availability Zone.\n\n\n\nFigure 11. Load balancer-to-pod using TargetType “instance”\n\nThe second method involves a target group consisting of the pod IP addresses (Figure 12). In this method, the load balancer targets are the IP address of the pods. Communication bypasses the kube-proxy and targets the pod directly, keeping traffic in the same Availability Zone and avoiding data transfer charges.\n\n\n\nFigure 12. Load balancer-to-pod using TargetType “IP”\n\n### **Tips**\nBelow are some tips to avoid excess data transfer charges and data processing charges in your container workloads:\n\n- Additional components in the network path, such as NAT gateways, [AWS PrivateLink](https://aws.amazon.com/cn/privatelink/?trk=cndc-detail), or [AWS Transit Gateway](https://aws.amazon.com/cn/transit-gateway/?trk=cndc-detail), may incur additional data transfer or data processing charges.\n- Review potential data transfer charges at both the source and the target of your communication channel. Remember that “Data Transfer In” to a destination is also “Data Transfer Out” from a source.\n- Limit cross-Region data transfer where possible. Use the Amazon ECR built-in cross-Region replication features to limit what is transferred across Regions.\n- Use cross-Region replication to replicate images in Amazon ECR into additional Regions and then pull images from the local Region.\n- Use container image build [best practices](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/application.html) and limit container images to only the essentials required to run your workload. Images with extraneous binaries cost more to store, transfer, and increase startup time.\n- Determine the most efficient network path for your traffic. For example, do your requirements indicate a need for a private cluster with no external connectivity? As more networking components are added, data transfer costs will increase.\n- Consider consolidating [AWS PrivateLink](https://aws.amazon.com/cn/privatelink/?trk=cndc-detail) endpoints in a central VPC connected by the Transit Gateway described in this [AWS PrivateLink whitepaper](https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/aws-privatelink.html).\n- In an [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) deployment with no external network connectivity, consider using [AWS Fargate](https://aws.amazon.com/cn/fargate/?trk=cndc-detail) to host your containers. This will eliminate the need for the [AWS PrivateLink](https://aws.amazon.com/cn/privatelink/?trk=cndc-detail) endpoint for ECS.\n- When using a load balancer in your [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) workload, try to avoid NodePort services and target the pods directly using the IP-based TargetType for target groups.\n\n\n### **Conclusion**\nIn order to determine the most cost-efficient architecture for your container-based workloads, it’s important to understand how data transfer charges are calculated. Design decisions made related to compute capacity deployment, access to public AWS services, and general network architecture have an impact on data transfer charges.\n\nInterested in learning more? Check out the blog post [Overview of Data Transfer Costs for Common Architectures](https://aws.amazon.com/blogs/architecture/overview-of-data-transfer-costs-for-common-architectures/) for an explanation of data transfer charges for common network architectures, and be sure to visit the [AWS Containers Blog](https://aws.amazon.com/blogs/containers/) for more great container-related content!\n\n\n\n**Dennis Schmidt**\n\nDennis Schmidt is a Solutions Architect based in Rochester, New York. He works with Greenfield customers, helping them get started on their cloud journey with AWS, and is passionate about containers, serverless, and cost optimization. Outside of work, Dennis enjoys spending time with family and attending various sporting events.\n\n\n\n**Steve Wolfe**\n\nSteve Wolfe is a Senior Solutions Architect for AWS. Steve has a specialty in Containers and Kubernetes. He works with customers in Wisconsin getting started with AWS.","render":"<h3><a id=\\"Overview_0\\"></a><strong>Overview</strong></h3>\\n<p>Data transfer costs can play a significant role in determining the overall design of a system. The <a href=\\"https://aws.amazon.com/ecr/\\" target=\\"_blank\\">Amazon Elastic Container Registry (Amazon ECR)</a>, <a href=\\"https://aws.amazon.com/ecs/\\" target=\\"_blank\\">Amazon Elastic Container Service (Amazon ECS)</a>, and <a href=\\"https://aws.amazon.com/eks/\\" target=\\"_blank\\">Amazon Elastic Kubernetes Service (Amazon EKS)</a> can all incur data transfer charges depending on a variety of factors. It can be difficult to visualize what that means relative to an [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) or [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) deployment. This blog illustrates common deployment patterns for AWS container services and explains the resulting data transfer charges that may be incurred. Service charges are out of scope for this blog, but should be carefully considered when designing any architecture.</p>\\n<h3><a id=\\"Amazon_ECR_3\\"></a><strong>Amazon ECR</strong></h3>\\n<p>An Amazon ECR private registry hosts container images in a highly available and scalable architecture. There are two types of registries in Amazon ECR—private and public—and each one has different data transfer charges.</p>\n<h4><a id=\\"Amazon_ECR_private_registry_6\\"></a><strong>Amazon ECR private registry</strong></h4>\\n<p>You can use an ECR private registry to manage private image repositories consisting of Docker and Open Container Initiative (OCI) images and artifacts.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/40b3f76892ea4816a5ca1190c19df21c_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 1. ECR private registry</p>\n<p>Data transfer into the Amazon ECR private registry incurs no charge from Amazon ECR. Data transferred between Amazon ECR and other services within the same Region is free. Data transferred between Amazon ECR and other services in different Regions will be charged at Internet Data Transfer rates on both sides of the transfer. Note that this is aggregated with other outbound data transfer across multiple services, and rate tiers will apply based on the amount of data transferred.</p>\n<p>Amazon ECR offers built-in functionality to replicate container images to different locations. This could be useful for disaster recovery purposes, a promote-to-production pipeline, or to help reduce the image pull time and data transfer costs when running containers in different Regions. This data transfer is charged at the cross-Region data transfer charge rates described on the <a href=\\"https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer\\" target=\\"_blank\\">Amazon EC2 pricing page</a></p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/75f56e1250e2481eb7e0b298fdbd0fed_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 2. ECR public registry</p>\n<p>Refer to the <a href=\\"https://aws.amazon.com/ecr/pricing/\\" target=\\"_blank\\">Amazon ECR pricing page</a> and <a href=\\"https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer\\" target=\\"_blank\\">Amazon EC2 pricing page</a> for more information.</p>\\n<h4><a id=\\"Amazon_ECR_Registry_Public_23\\"></a><strong>Amazon ECR Registry Public</strong></h4>\\n<p><a href=\\"https://docs.aws.amazon.com/AmazonECR/latest/public/what-is-ecr.html\\" target=\\"_blank\\">Amazon Elastic Container Registry Public</a> is a managed AWS container image registry service that is secure, scalable, and reliable. Amazon ECR supports public image repositories with resource-based permissions using <a href=\\"https://docs.aws.amazon.com/AmazonECR/latest/public/security_iam_service-with-iam.html\\" target=\\"_blank\\">AWS Identity and Access Management (IAM)</a> so that specific users can access your public repositories to push images. Developers can use their preferred CLI to push and pull container images and OCI-compatible artifacts. Images are publicly available to pull, either anonymously or using an Amazon ECR public authentication token.</p>\\n<p>All data transfer into Amazon ECR Public incurs no charge from Amazon ECR. Data transfer out is subject to charges when more than 5 TB/month is transferred out to non-AWS destinations, and you have authenticated to Amazon ECR with an AWS account. Up to 500 GB/month can be transferred out to non-AWS destinations to clients that have not authenticated (that is, anonymously). After that limit is reached, no further anonymous data transfer is allowed.</p>\n<h3><a id=\\"Data_transfer_for_Amazon_ECS_28\\"></a><strong>Data transfer for Amazon ECS</strong></h3>\\n<p>Three common deployment models for Amazon ECS are clusters with external network access, clusters without external network access, and ECS Anywhere.</p>\n<h4><a id=\\"Amazon_ECS_clusters_with_external_access_31\\"></a><strong>Amazon ECS clusters with external access</strong></h4>\\n<p>Compute capacity for container instances in Amazon ECS can be deployed within VPCs that allow access to the Amazon ECS service endpoint using the internet. For example, the compute capacity for container instances can be deployed in public subnets (with an <a href=\\"https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html\\" target=\\"_blank\\">internet gateway</a>), private subnets (with a <a href=\\"https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html\\" target=\\"_blank\\">NAT gateway</a>, shown in Figure 3), or can route to the internet through another <a href=\\"https://aws.amazon.com/vpc/\\" target=\\"_blank\\">VPC</a> using the <a href=\\"https://aws.amazon.com/transit-gateway/\\" target=\\"_blank\\">AWS Transit Gateway</a>. Both <a href=\\"https://aws.amazon.com/ec2/\\" target=\\"_blank\\">Amazon EC2</a> and <a href=\\"https://aws.amazon.com/fargate/\\" target=\\"_blank\\">AWS Fargate</a> can be used as compute for this type of deployment, and there is no difference in data transfer costs based on which is chosen.</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/acf9988e04844dc6b877872a34e8531a_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 3. ECS deployment with internet access</p>\n<p>The following data transfer is not charged in the sample deployment:</p>\n<ul>\\n<li>Data transfer in from the Amazon ECS control plane (that is, responses to API calls from the data plane) and from Amazon ECR (that is, “image pulls”)</li>\n<li>Communication between tasks and the Application Load Balancer</li>\n<li>Communication between tasks and the database instance in the same Availability Zone</li>\n</ul>\\n<p>In this deployment, data transfer charges accrue for:</p>\n<ul>\\n<li>Data transfer out through the NAT gateway, including polling traffic from the ECS agent to the Amazon ECS control plane and outbound data to Amazon ECR</li>\n<li>Cross-Availability Zone traffic to the database</li>\n</ul>\\n<p>It is important to note that although the NAT gateway does not charge for data transfer in, there will still be a per-GB data processing charge on data that flows through the NAT gateway, regardless of direction. The data transfer out charges are in addition to this charge. NAT gateway pricing can be found on the <a href=\\"https://aws.amazon.com/vpc/pricing/\\" target=\\"_blank\\">Amazon VPC pricing page</a>.</p>\\n<h4><a id=\\"Amazon_ECS_with_no_external_access_53\\"></a><strong>Amazon ECS with no external access</strong></h4>\\n<p>Another common pattern to deploy Amazon ECS workloads is to restrict all external network access (Figure 4). Because container instances in Amazon ECS clusters need external network access to communicate with the Amazon ECS service endpoint, <a href=\\"https://aws.amazon.com/privatelink/\\" target=\\"_blank\\">AWS PrivateLink</a> must be used to communicate with the service endpoints. Again, both [Amazon EC2 ](https://aws.amazon.com/cn/ec2/?trk=cndc-detail)and [AWS Fargate](https://aws.amazon.com/cn/fargate/?trk=cndc-detail) can be used for the compute capacity in this type of deployment. However, there is a difference in data transfer costs based on which is chosen.</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/ba7122ee9c8048b4abe7798fc18c2d1c_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 4. ECS deployment in private network</p>\n<p>The following data transfer is not charged in the sample deployment:</p>\n<ul>\\n<li>Communication between tasks and the Application Load Balancer</li>\n<li>Communication between tasks and the database instance in the same Availability Zone</li>\n</ul>\\n<p>In this deployment, data transfer charges accrue for cross-AZ traffic to the database.</p>\n<p>For clarity, the diagram depicts the Amazon ECS and Amazon ECR PrivateLink VPC interface endpoints as single objects. However, there are actually multiple endpoints required for each. For a description of endpoint requirements, consult <a href=\\"https://aws.amazon.com/blogs/compute/setting-up-aws-privatelink-for-amazon-ecs-and-amazon-ecr/\\" target=\\"_blank\\">this AWS Compute Blog</a> or the <a href=\\"https://docs.aws.amazon.com/AmazonECS/latest/developerguide/vpc-endpoints.html\\" target=\\"_blank\\">Amazon ECS</a> and <a href=\\"https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html\\" target=\\"_blank\\">Amazon ECR</a> documentation.</p>\\n<p>Although there is no data transfer fee, AWS PrivateLink imposes both a per-hour service charge and a per-GB data processing charge for data processed through each VPC endpoint, billed per Availability Zone. More details can be found on the <a href=\\"https://aws.amazon.com/privatelink/pricing/\\" target=\\"_blank\\">AWS PrivateLink pricing page</a>.</p>\\n<h5><a id=\\"Using_AWS_Fargate_in_private_Amazon_ECS_clusters_73\\"></a><strong>Using AWS Fargate in private Amazon ECS clusters</strong></h5>\\n<p>If AWS Fargate provides the compute capacity for the Amazon ECS cluster, the AWS PrivateLink endpoints for Amazon ECS are not required. This eliminates the service charges—hourly and data processing—for communication to the Amazon ECS service endpoint. Note that AWS PrivateLink VPC Endpoints for Amazon ECR are still required to pull images from Amazon ECR.</p>\n<h4><a id=\\"Amazon_ECS_Anywhere_76\\"></a><strong>Amazon ECS Anywhere</strong></h4>\\n<p><a href=\\"https://aws.amazon.com/ecs/anywhere/\\" target=\\"_blank\\">Amazon Elastic Container Service Anywhere</a> is a feature of [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) that enables you to easily run and manage container-based applications on-premises, including on your own virtual machines (VMs) and bare metal servers. With [Amazon ECS Anywhere](https://aws.amazon.com/cn/ecs/anywhere/?trk=cndc-detail), you do not need to install or operate local container orchestration software, thus reducing operational overhead. ECS Anywhere offers a completely managed solution that enables you to standardize container management across all of your environments.</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/1585afa0d9e74fbfb40b37891a8da792_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 5. ECS Anywhere over the internet</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/7c4dba117aca452b811c3ff508180aca_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 6. ECS Anywhere with Direct Connect</p>\n<p>Data transfer fees are accrued based on how the customer-managed infrastructure connects to the Amazon ECS Anywhere control plane:</p>\n<ul>\\n<li><strong>If connecting via the internet (Figure 5)</strong> – There is no charge for data transfer when communication between the [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) control plane and ECS agent occurs over the open internet.</li>\\n<li><strong>If connecting via <a href=\\"https://aws.amazon.com/vpn/\\" target=\\"_blank\\">AWS Site-to-Site VPN</a> or <a href=\\"https://aws.amazon.com/directconnect/\\" target=\\"_blank\\">AWS Direct Connect</a> (Figure 6)</strong> – Standard data transfer fees (that is, “Data Transfer Out”) apply to AWS Site-to-Site VPN or AWS Direct Connect for communication between the ECS Anywhere control plane and ECS agent that occurs through Site-to-Site VPN or AWS Direct Connect link.</li>\n</ul>\\n<p>Refer to the <a href=\\"https://aws.amazon.com/ecs/anywhere/pricing/\\" target=\\"_blank\\">ECS Anywhere pricing page</a>, <a href=\\"https://aws.amazon.com/vpn/pricing/\\" target=\\"_blank\\">AWS Site-to-Site VPN pricing page</a>, and the <a href=\\"https://aws.amazon.com/directconnect/pricing/\\" target=\\"_blank\\">AWS Direct Connect pricing page</a> for more detail.</p>\\n<h3><a id=\\"Data_transfer_for_Amazon_EKS_95\\"></a><strong>Data transfer for Amazon EKS</strong></h3>\\n<p>Similar to Amazon ECS, Amazon EKS data transfer charges follow the guidelines described on the Amazon EC2 pricing page. Figure 7 represents a sample Amazon EKS workload with two pods deployed to different Amazon EKS worker nodes in different Availability Zones.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/c02799d3698c48798349c50bae002f39_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 7. Sample application EKS deployment</p>\n<p>The following data transfer is not charged in this example:</p>\n<ul>\\n<li>Traffic in/out of control plane (not shown on diagram)</li>\n<li>Image pulls from ECR within the same Region (not shown on diagram)</li>\n<li>Communication between pods and the Application Load Balancer</li>\n<li>Communication between pods and the database instance in the same Availability Zone</li>\n</ul>\\n<p>The following data transfer charges accrue for this example:</p>\n<ul>\\n<li>Data transfer between the Kubernetes pod and the database in a different Availability Zone</li>\n<li>Data out (egress) from the Application Load Balancer</li>\n<li>Communication between pods in different Availability Zones</li>\n</ul>\\n<p>There are several other configuration options and deployment strategies to consider in an Amazon EKS deployment relative to data transfer costs: cluster access (public/private), pod-to-pod communication, and communication between the pods and the load balancer.</p>\n<h4><a id=\\"Public_EKS_clusters_119\\"></a><strong>Public EKS clusters</strong></h4>\\n<p>Public EKS clusters (Figure 8) have the public endpoint access enabled and the private endpoint disabled. Communication between the control plane and worker nodes will exit the VPC based on the VPC’s routing. For worker nodes in private subnets, communication will traverse a NAT gateway and exit via an internet gateway. There is no data transfer charge associated with this. However, there is a per-GB NAT gateway data processing fee. In addition, worker nodes not in the same Availability Zone as the NAT gateway will incur a per-GB data transfer charge for cross-Availability Zone traffic. For worker nodes in public subnets, communication will exit the internet gateway and will not be charged.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/ca76c96b774448d2955070082887d861_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 8. Public EKS cluster</p>\n<h4><a id=\\"Private_EKS_clusters_126\\"></a><strong>Private EKS clusters</strong></h4>\\n<p>In a cluster with private Amazon EKS endpoints (Figure 9), worker nodes will communicate with private endpoints. The Kubernetes API Server Endpoint URL will resolve to <a href=\\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html\\" target=\\"_blank\\">elastic network interfaces</a> (ENIs) within the (customer) VPC. Worker nodes inside the VPC will communicate with these ENIs. In this deployment model, there is a chance that a worker node communicates with an ENI in a different Availability Zone. This communication would be subject to a cross-Availability Zone data transfer charge.</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/715e431ea45449c092a486b08f21b95e_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 9. Private EKS cluster</p>\n<h4><a id=\\"Podtopod_communication_133\\"></a><strong>Pod-to-pod communication</strong></h4>\\n<p>Kubernetes applications often leverage communication between containers and between pods (Figure 10).</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/9bcfb80fec064cd89bccc2ed3938b22c_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 10. Pod-to-pod communication</p>\n<p>Containers within the same pod are guaranteed to be on the same worker node and will communicate over the loopback device, resulting in no data transfer charges.<br />\\nData transfer charges between pods depend on the pod placement:</p>\n<ul>\\n<li>There is no data transfer charge for pods deployed on the same node or within the same Availability Zone</li>\n<li>Pods that communicate and are placed in different Availability Zones will incur a data transfer charge</li>\n</ul>\\n<h4><a id=\\"Load_balancertopod_communication_147\\"></a><strong>Load balancer-to-pod communication</strong></h4>\\n<p>Amazon EKS workloads often include a load balancer to distribute the traffic evenly across pods. Pods are deployed alongside Kubernetes Service and Ingress objects. The add-on <a href=\\"https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html\\" target=\\"_blank\\">AWS Load Balancer Controller</a> monitors these objects and automatically provisions the AWS load balancer resource. The path traffic takes (and resulting data transfer charges) depend on how the Service and Ingress objects are configured. There are two common methods of configuration.</p>\\n<p>The first method involves having a target group consisting of all worker nodes within the cluster (Figure 11). This grouping includes Service:LoadBalancer, Service:NodePort, and Ingress TargetType: Instance. Here, an ephemeral port (NodePort) is opened on each node in the cluster, and traffic is distributed evenly across all nodes. If the destination pod is on the node, no additional data transfer occurs. If the destination pod was scheduled on a different node, data transfer charges may accrue if the target pod is in a different Availability Zone.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/d9585ccef3c645c5b80a1707aae826da_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 11. Load balancer-to-pod using TargetType “instance”</p>\n<p>The second method involves a target group consisting of the pod IP addresses (Figure 12). In this method, the load balancer targets are the IP address of the pods. Communication bypasses the kube-proxy and targets the pod directly, keeping traffic in the same Availability Zone and avoiding data transfer charges.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/98d8b44efce14e11ab4765416d8d2843_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 12. Load balancer-to-pod using TargetType “IP”</p>\n<h3><a id=\\"Tips_162\\"></a><strong>Tips</strong></h3>\\n<p>Below are some tips to avoid excess data transfer charges and data processing charges in your container workloads:</p>\n<ul>\\n<li>Additional components in the network path, such as NAT gateways, AWS PrivateLink, or AWS Transit Gateway, may incur additional data transfer or data processing charges.</li>\n<li>Review potential data transfer charges at both the source and the target of your communication channel. Remember that “Data Transfer In” to a destination is also “Data Transfer Out” from a source.</li>\n<li>Limit cross-Region data transfer where possible. Use the Amazon ECR built-in cross-Region replication features to limit what is transferred across Regions.</li>\n<li>Use cross-Region replication to replicate images in Amazon ECR into additional Regions and then pull images from the local Region.</li>\n<li>Use container image build <a href=\\"https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/application.html\\" target=\\"_blank\\">best practices</a> and limit container images to only the essentials required to run your workload. Images with extraneous binaries cost more to store, transfer, and increase startup time.</li>\\n<li>Determine the most efficient network path for your traffic. For example, do your requirements indicate a need for a private cluster with no external connectivity? As more networking components are added, data transfer costs will increase.</li>\n<li>Consider consolidating AWS PrivateLink endpoints in a central VPC connected by the Transit Gateway described in this <a href=\\"https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/aws-privatelink.html\\" target=\\"_blank\\">AWS PrivateLink whitepaper</a>.</li>\\n<li>In an Amazon ECS deployment with no external network connectivity, consider using AWS Fargate to host your containers. This will eliminate the need for the AWS PrivateLink endpoint for ECS.</li>\n<li>When using a load balancer in your Amazon EKS workload, try to avoid NodePort services and target the pods directly using the IP-based TargetType for target groups.</li>\n</ul>\\n<h3><a id=\\"Conclusion_176\\"></a><strong>Conclusion</strong></h3>\\n<p>In order to determine the most cost-efficient architecture for your container-based workloads, it’s important to understand how data transfer charges are calculated. Design decisions made related to compute capacity deployment, access to public AWS services, and general network architecture have an impact on data transfer charges.</p>\n<p>Interested in learning more? Check out the blog post <a href=\\"https://aws.amazon.com/blogs/architecture/overview-of-data-transfer-costs-for-common-architectures/\\" target=\\"_blank\\">Overview of Data Transfer Costs for Common Architectures</a> for an explanation of data transfer charges for common network architectures, and be sure to visit the <a href=\\"https://aws.amazon.com/blogs/containers/\\" target=\\"_blank\\">AWS Containers Blog</a> for more great container-related content!</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/8077de2a432c4400a3a9a01a377cf994_image.png\\" alt=\\"image.png\\" /></p>\n<p><strong>Dennis Schmidt</strong></p>\\n<p>Dennis Schmidt is a Solutions Architect based in Rochester, New York. He works with Greenfield customers, helping them get started on their cloud journey with AWS, and is passionate about containers, serverless, and cost optimization. Outside of work, Dennis enjoys spending time with family and attending various sporting events.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/e8a4e89eb9b6464692631fa60e498473_image.png\\" alt=\\"image.png\\" /></p>\n<p><strong>Steve Wolfe</strong></p>\\n<p>Steve Wolfe is a Senior Solutions Architect for AWS. Steve has a specialty in Containers and Kubernetes. He works with customers in Wisconsin getting started with AWS.</p>\n"}