New – ENA Express: Improved Network Latency and Per-Flow Performance on EC2

{"value":"We know that you can always make great use of all available network bandwidth and network performance, and have done our best to supply it to you. Over the years, network bandwidth has grown from the 250 Mbps on the original **m1** instance to 200 Gbps on the newest **m6in** instances. In addition to raw bandwidth, we have also introduced advanced networking features including [Enhanced Networking](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html), [Elastic Network Adapters](https://aws.amazon.com/blogs/aws/elastic-network-adapter-high-performance-network-interface-for-amazon-ec2/) (ENAs), and (for tightly coupled HPC workloads) [Elastic Fabric Adapters](https://aws.amazon.com/blogs/aws/now-available-elastic-fabric-adapter-efa-for-tightly-coupled-hpc-workloads/) (EFAs).\n\n\n### **++Introducing ENA Express++**\nToday we are launching ENA Express. Building on the [Scalable Reliable Datagram](https://ieeexplore.ieee.org/document/9167399) (SRD) protocol that already powers Elastic Fabric Adapters, ENA Express reduces P99 latency of traffic flows by up to 50% and P99.9 latency by up to 85% (in comparison to TCP), while also increasing the maximum single-flow bandwidth from 5 Gbps to 25 Gbps. Bottom line, you get a lot more per-flow bandwidth and a lot less variability.\n\nYou can enable ENA Express on new and existing ENAs and take advantage of this performance right away for TCP and UDP traffic between **c6gn** instances running in the same Availability Zone.\n\n### **++Using ENA Express++**\nI used a pair of **c6gn** instances to set up and test ENA Express. After I launched the instances I used the [Amazon Web Services Management Console](https://console.aws.amazon.com/) to enable ENA Express for both instances. I find each ENI, select it, and choose **Manage ENA Express** from the **Actions** menu:\n\n\n\nI enable **ENA Express** and **ENA Express** UDP and click **Save**:\n\n\n\nThen I set the [Maximum Transmission Unit](https://en.wikipedia.org/wiki/Maximum_transmission_unit) (MTU) to 8900 on both instances:\n\n```\\n\$ sudo /sbin/ifconfig eth0 mtu 8900\\n```\n\nI install [iperf3](https://iperf.fr/) on both instances, and start the first one in server mode:\n\n\n```\\n\$ iperf3 -s\\n-----------------------------------------------------------\\nServer listening on 5201\\n-----------------------------------------------------------\\n```\n\nThen I run the second one in client mode and observe the results:\n\n```\\n\$ iperf3 -c 10.0.178.46\\nConnecting to host 10.0.178.46, port 5201\\n[ 4] local 10.0.187.74 port 35622 connected to 10.0.178.46 port 5201\\n[ ID] Interval Transfer Bandwidth Retr Cwnd\\n[ 4] 0.00-1.00 sec 2.80 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 1.00-2.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 2.00-3.00 sec 2.80 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 3.00-4.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 4.00-5.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 5.00-6.00 sec 2.80 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 6.00-7.00 sec 2.80 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 7.00-8.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 8.00-9.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 9.00-10.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n- - - - - - - - - - - - - - - - - - - - - - - - - -\\n[ ID] Interval Transfer Bandwidth Retr\\n[ 4] 0.00-10.00 sec 28.0 GBytes 24.1 Gbits/sec 0 sender\\n[ 4] 0.00-10.00 sec 28.0 GBytes 24.1 Gbits/sec receiver\\n```\n\nThe [ENA driver](https://github.com/amzn/amzn-drivers/tree/master/kernel/linux/ena) reports on metrics that I can review to confirm the use of SRD:\n\n```\\nethtool -S eth0 | grep ena_srd\\n ena_srd_mode: 3\\n ena_srd_tx_pkts: 25858313\\n ena_srd_eligible_tx_pkts: 25858323\\n ena_srd_rx_pkts: 2831267\\n ena_srd_resource_utilization: 0\\n```\n\nThe metrics work as follows:\n\n\n ```\\nena_srd_mode\\n```\n indicates that SRD is enabled for TCP and UDP.\n\n```\\nena_srd_tx_pkts\\n```\n denotes the number of packets that have been transmitted via SRD.\n\n```\\nena_srd_eligible_pkts\\n```\n denotes the number of packets that were eligible for transmission via SRD. A packet is eligible for SRD if ENA-SRD is enabled on both ends of the connection, both connections reside in the same Availability Zone, and the packet is using either UDP or TCP.\n\n```\\nena_srd_rx_pkts\\n```\n denotes the number of packets that have been received via SRD.\n\n```\\nena_srd_resource_utilization\\n```\n denotes the percent of allocated Nitro network card resources that are in use, and is proportional to the number of open SRD connections. If this value is consistently approaching 100%, scaling out to more instances or scaling up to a larger instance size may be warranted.\n\n### ++Thing to Know++\nHere are a couple of things to know about ENA Express and SRD:\n\n\n**Access** – I used the Management Console to enable and test ENA Express; CLI, API, CloudFormation and CDK support is also available.\n\n**Fallback** – If a TCP or UDP packet is not eligible for transmission via SRD, it will simply be transmitted in the usual way.\n\n**UDP – SRD** takes advantage of multiple network paths and “sprays” packets across them. This would normally present a challenge for applications that expect packets to arrive more or less in order, but ENA Express helps out by putting the UDP packets back into order before delivering them to you, taking the burden off of your application. If you have built your own reliability layer over UDP, or if your application does not require packets to arrive in order, you can enable ENA Express for TCP but not for UDP.\n\n**Instance Types and Sizes** – We are launching with support for the **16xlarge** size of the **c6gn** instances, with additional instance families and sizes in the works.\n\n**Resource Utilization** – As I hinted at above, ENA Express uses some Nitro card resources to process packets. This processing also adds a few microseconds of latency per packet processed, and also has a moderate but measurable effect on the maximum number of packets that a particular instance can process per second. In situations where high packet rates are coupled with small packet sizes, ENA Express may not be appropriate. In all other cases you can simply enable SRD to enjoy higher per-flow bandwidth and consistent latency.\n\n**Pricing** – There is no additional charge for the use of ENA Express.\n\n**Region**s – ENA Express is available in all commercial [Amazon Web Services Regions](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/).\n\n### **All About SRD**\nI could write an entire blog post about SRD, but my colleagues beat me to it! Here are some great resources to help you to learn more:\n\n**[A Cloud-Optimized Transport for Elastic and Scalable HPC](https://ieeexplore.ieee.org/document/9167399)** – This paper reviews the challenges that arise when trying to run HPC traffic across a TCP-based network, and points out that the variability (latency outliers) can have a profound effect on scaling efficiency, and includes a succinct overview of SRD:\n\nScalable reliable datagram (SRD) is optimized for hyper-scale datacenters: it provides load balancing across multiple paths and fast recovery from packet drops or link failures. It utilizes standard ECMP functionality on the commodity Ethernet switches and works around its limitations: the sender controls the ECMP path selection by manipulating packet encapsulation.\n\nThere’s a lot of interesting detail in the full paper, and it is well worth reading!\n\n**[In the Search for Performance, There’s More Than One Way to Build a Network](https://aws.amazon.com/blogs/hpc/in-the-search-for-performance-theres-more-than-one-way-to-build-a-network/)** – This 2021 blog post reviews our decision to build the Elastic Fabric Adapter, and includes some important data (and cool graphics) to demonstrate the impact of packet loss on overall application performance. One of the interesting things about SRD is that it keeps track of the availability and performance of multiple network paths between transmitter and receiver, and sprays packets across up to 64 paths at a time in order to take advantage of as much bandwidth as possible and to recover quickly in case of packet loss.\n\n— [Jeff](https://twitter.com/jeffbarr);\n\n\n\n### **[Jeff Barr](https://aws.amazon.com/blogs/aws/author/jbarr/)**\nJeff Barr is Chief Evangelist for Amazon Web Services. He started this blog in 2004 and has been writing posts just about non-stop ever since.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","render":"<p>We know that you can always make great use of all available network bandwidth and network performance, and have done our best to supply it to you. Over the years, network bandwidth has grown from the 250 Mbps on the original <strong>m1</strong> instance to 200 Gbps on the newest <strong>m6in</strong> instances. In addition to raw bandwidth, we have also introduced advanced networking features including <a href=\\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html\\" target=\\"_blank\\">Enhanced Networking</a>, <a href=\\"https://aws.amazon.com/blogs/aws/elastic-network-adapter-high-performance-network-interface-for-amazon-ec2/\\" target=\\"_blank\\">Elastic Network Adapters</a> (ENAs), and (for tightly coupled HPC workloads) <a href=\\"https://aws.amazon.com/blogs/aws/now-available-elastic-fabric-adapter-efa-for-tightly-coupled-hpc-workloads/\\" target=\\"_blank\\">Elastic Fabric Adapters</a> (EFAs).</p>\\n<h3><a id=\\"Introducing_ENA_Express_3\\"></a><strong><ins>Introducing ENA Express</ins></strong></h3>\n<p>Today we are launching ENA Express. Building on the <a href=\\"https://ieeexplore.ieee.org/document/9167399\\" target=\\"_blank\\">Scalable Reliable Datagram</a> (SRD) protocol that already powers Elastic Fabric Adapters, ENA Express reduces P99 latency of traffic flows by up to 50% and P99.9 latency by up to 85% (in comparison to TCP), while also increasing the maximum single-flow bandwidth from 5 Gbps to 25 Gbps. Bottom line, you get a lot more per-flow bandwidth and a lot less variability.</p>\\n<p>You can enable ENA Express on new and existing ENAs and take advantage of this performance right away for TCP and UDP traffic between <strong>c6gn</strong> instances running in the same Availability Zone.</p>\\n<h3><a id=\\"Using_ENA_Express_8\\"></a><strong><ins>Using ENA Express</ins></strong></h3>\n<p>I used a pair of <strong>c6gn</strong> instances to set up and test ENA Express. After I launched the instances I used the <a href=\\"https://console.aws.amazon.com/\\" target=\\"_blank\\">Amazon Web Services Management Console</a> to enable ENA Express for both instances. I find each ENI, select it, and choose <strong>Manage ENA Express</strong> from the <strong>Actions</strong> menu:</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/3aca842b4a6343e7bcba906d1602c8bf_image.png\\" alt=\\"image.png\\" /></p>\n<p>I enable <strong>ENA Express</strong> and <strong>ENA Express</strong> UDP and click <strong>Save</strong>:</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/4c7218ba564b48899acf2ef3b4fe5bb7_image.png\\" alt=\\"image.png\\" /></p>\n<p>Then I set the <a href=\\"https://en.wikipedia.org/wiki/Maximum_transmission_unit\\" target=\\"_blank\\">Maximum Transmission Unit</a> (MTU) to 8900 on both instances:</p>\\n<pre><code class=\\"lang-\\">\$ sudo /sbin/ifconfig eth0 mtu 8900\\n</code></pre>\\n<p>I install <a href=\\"https://iperf.fr/\\" target=\\"_blank\\">iperf3</a> on both instances, and start the first one in server mode:</p>\\n<pre><code class=\\"lang-\\">\$ iperf3 -s\\n-----------------------------------------------------------\\nServer listening on 5201\\n-----------------------------------------------------------\\n</code></pre>\\n<p>Then I run the second one in client mode and observe the results:</p>\n<pre><code class=\\"lang-\\">\$ iperf3 -c 10.0.178.46\\nConnecting to host 10.0.178.46, port 5201\\n[ 4] local 10.0.187.74 port 35622 connected to 10.0.178.46 port 5201\\n[ ID] Interval Transfer Bandwidth Retr Cwnd\\n[ 4] 0.00-1.00 sec 2.80 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 1.00-2.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 2.00-3.00 sec 2.80 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 3.00-4.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 4.00-5.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 5.00-6.00 sec 2.80 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 6.00-7.00 sec 2.80 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 7.00-8.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 8.00-9.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n[ 4] 9.00-10.00 sec 2.81 GBytes 24.1 Gbits/sec 0 1.43 MBytes\\n- - - - - - - - - - - - - - - - - - - - - - - - - -\\n[ ID] Interval Transfer Bandwidth Retr\\n[ 4] 0.00-10.00 sec 28.0 GBytes 24.1 Gbits/sec 0 sender\\n[ 4] 0.00-10.00 sec 28.0 GBytes 24.1 Gbits/sec receiver\\n</code></pre>\\n<p>The <a href=\\"https://github.com/amzn/amzn-drivers/tree/master/kernel/linux/ena\\" target=\\"_blank\\">ENA driver</a> reports on metrics that I can review to confirm the use of SRD:</p>\\n<pre><code class=\\"lang-\\">ethtool -S eth0 | grep ena_srd\\n ena_srd_mode: 3\\n ena_srd_tx_pkts: 25858313\\n ena_srd_eligible_tx_pkts: 25858323\\n ena_srd_rx_pkts: 2831267\\n ena_srd_resource_utilization: 0\\n</code></pre>\\n<p>The metrics work as follows:</p>\n<pre><code class=\\"lang-\\">ena_srd_mode\\n</code></pre>\\n<p>indicates that SRD is enabled for TCP and UDP.</p>\n<pre><code class=\\"lang-\\">ena_srd_tx_pkts\\n</code></pre>\\n<p>denotes the number of packets that have been transmitted via SRD.</p>\n<pre><code class=\\"lang-\\">ena_srd_eligible_pkts\\n</code></pre>\\n<p>denotes the number of packets that were eligible for transmission via SRD. A packet is eligible for SRD if ENA-SRD is enabled on both ends of the connection, both connections reside in the same Availability Zone, and the packet is using either UDP or TCP.</p>\n<pre><code class=\\"lang-\\">ena_srd_rx_pkts\\n</code></pre>\\n<p>denotes the number of packets that have been received via SRD.</p>\n<pre><code class=\\"lang-\\">ena_srd_resource_utilization\\n</code></pre>\\n<p>denotes the percent of allocated Nitro network card resources that are in use, and is proportional to the number of open SRD connections. If this value is consistently approaching 100%, scaling out to more instances or scaling up to a larger instance size may be warranted.</p>\n<h3><a id=\\"Thing_to_Know_95\\"></a><ins>Thing to Know</ins></h3>\\n<p>Here are a couple of things to know about ENA Express and SRD:</p>\n<p><strong>Access</strong> – I used the Management Console to enable and test ENA Express; CLI, API, CloudFormation and CDK support is also available.</p>\\n<p><strong>Fallback</strong> – If a TCP or UDP packet is not eligible for transmission via SRD, it will simply be transmitted in the usual way.</p>\\n<p><strong>UDP – SRD</strong> takes advantage of multiple network paths and “sprays” packets across them. This would normally present a challenge for applications that expect packets to arrive more or less in order, but ENA Express helps out by putting the UDP packets back into order before delivering them to you, taking the burden off of your application. If you have built your own reliability layer over UDP, or if your application does not require packets to arrive in order, you can enable ENA Express for TCP but not for UDP.</p>\\n<p><strong>Instance Types and Sizes</strong> – We are launching with support for the <strong>16xlarge</strong> size of the <strong>c6gn</strong> instances, with additional instance families and sizes in the works.</p>\\n<p><strong>Resource Utilization</strong> – As I hinted at above, ENA Express uses some Nitro card resources to process packets. This processing also adds a few microseconds of latency per packet processed, and also has a moderate but measurable effect on the maximum number of packets that a particular instance can process per second. In situations where high packet rates are coupled with small packet sizes, ENA Express may not be appropriate. In all other cases you can simply enable SRD to enjoy higher per-flow bandwidth and consistent latency.</p>\\n<p><strong>Pricing</strong> – There is no additional charge for the use of ENA Express.</p>\\n<p><strong>Region</strong>s – ENA Express is available in all commercial <a href=\\"https://aws.amazon.com/about-aws/global-infrastructure/regions_az/\\" target=\\"_blank\\">Amazon Web Services Regions</a>.</p>\\n<h3><a id=\\"All_About_SRD_113\\"></a><strong>All About SRD</strong></h3>\\n<p>I could write an entire blog post about SRD, but my colleagues beat me to it! Here are some great resources to help you to learn more:</p>\n<p><strong><a href=\\"https://ieeexplore.ieee.org/document/9167399\\" target=\\"_blank\\">A Cloud-Optimized Transport for Elastic and Scalable HPC</a></strong> – This paper reviews the challenges that arise when trying to run HPC traffic across a TCP-based network, and points out that the variability (latency outliers) can have a profound effect on scaling efficiency, and includes a succinct overview of SRD:</p>\n<p>Scalable reliable datagram (SRD) is optimized for hyper-scale datacenters: it provides load balancing across multiple paths and fast recovery from packet drops or link failures. It utilizes standard ECMP functionality on the commodity Ethernet switches and works around its limitations: the sender controls the ECMP path selection by manipulating packet encapsulation.</p>\n<p>There’s a lot of interesting detail in the full paper, and it is well worth reading!</p>\n<p><strong><a href=\\"https://aws.amazon.com/blogs/hpc/in-the-search-for-performance-theres-more-than-one-way-to-build-a-network/\\" target=\\"_blank\\">In the Search for Performance, There’s More Than One Way to Build a Network</a></strong> – This 2021 blog post reviews our decision to build the Elastic Fabric Adapter, and includes some important data (and cool graphics) to demonstrate the impact of packet loss on overall application performance. One of the interesting things about SRD is that it keeps track of the availability and performance of multiple network paths between transmitter and receiver, and sprays packets across up to 64 paths at a time in order to take advantage of as much bandwidth as possible and to recover quickly in case of packet loss.</p>\n<p>— <a href=\\"https://twitter.com/jeffbarr\\" target=\\"_blank\\">Jeff</a>;</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/a3bed60f23364e5290581aac9a40be67_image.png\\" alt=\\"image.png\\" /></p>\n<h3><a id=\\"Jeff_Barrhttpsawsamazoncomblogsawsauthorjbarr_128\\"></a><strong><a href=\\"https://aws.amazon.com/blogs/aws/author/jbarr/\\" target=\\"_blank\\">Jeff Barr</a></strong></h3>\n<p>Jeff Barr is Chief Evangelist for Amazon Web Services. He started this blog in 2004 and has been writing posts just about non-stop ever since.</p>\n"}