Identification of replication bottlenecks when using Amazon Application Migration Service

{"value":"Enterprises frequently begin their journey by re-hosting (lift-and-shift) their on-premises workloads into [Amazon Web Services and running Amazon Elastic Compute Cloud (Amazon EC2)](https://aws.amazon.com/ec2/) instances. A simpler way to re-host is by using [Amazon Web Services Application Migration Service (Application Migration Service)](https://aws.amazon.com/application-migration-service/), a cloud-native migration service.\n\nTo streamline and expedite migrations, automate reusable migration patterns that work for a wide range of applications. Application Migration Service is the recommended migration service to lift-and-shift your applications to Amazon Web Services.\n\nIn this blog post, we explore key variables that contribute to server replication speed when using Application Migration Service. We will also look at tests you can run to identify these bottlenecks and, where appropriate, include remediation steps.\n\n\n#### **Overview of migration using Application Migration Service**\n\n\nFigure 1 depicts the end-to-end data replication flow from source servers to a target machine hosted on Amazon Web Services. The diagram is designed to help visualize potential bottlenecks within the data flow, which are denoted by a black diamond.\n\n\n\nFigure 1. Data flow when using Amazon Web Services Application Migration Service (black diamonds denote potential points of contention)\n\n\n#### **Baseline testing**\n\n\nTo determine a baseline replication speed, we recommend performing a control test between your target Amazon Web Services Region and the nearest Region to your source workloads. For example, if your source workloads are in a data center in Rome and your target Region is Paris, run a test between eu-south-1 (Milan) and eu-west-3 (Paris). This will give a theoretical upper bandwidth limit, as replication will occur over the Amazon Web Services backbone. If the target Region is already the closest Region to your source workloads, run the test from within the same Region.\n\n\n#### **Network connectivity**\n\n\nThere are several ways to establish connectivity between your on-premises location and Amazon Web Services Region:\n\n- Public internet\n- VPN\n- [Amazon Web Services Direct Connect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html)\n\nThis section pertains to options 1 and 2. If facing replication speed issues, the first place to look is at network bandwidth. From a source machine within your internal network, run a speed test to calculate your bandwidth out to the internet; common test providers include [Cloudflare](https://www.cloudflare.com/), [Ookla](https://www.ookla.com/), and Google. This is your bandwidth to the internet, not to Amazon Web Services.\n\nNext, to confirm the data flow from within your data center, run a ```traceroute``` (Windows) or ```tracert``` (Linux). Identify any network hops that are unusual or potentially throttling bandwidth (due to hardware limitations or configuration).\n\nTo measure the maximum bandwidth between your data center and the Amazon Web Services subnet that is being used for data replication, while accounting for Security Sockets Layer (SSL) encapsulation, use the [CloudEndure SSL bandwidth tool](https://docs.cloudendure.com/Content/FAQ/FAQ/Replication_Related.htm#How) (refer to Figure 1).\n\n\n#### **Source storage I/O**\n\n\nThe next area to look for replication bottlenecks is source storage. The underlying storage for servers can be a point of contention. If the storage is maxing-out its read speeds, this will impact the data-replication rate. If your storage I/O is heavily utilized, it can impact block replication by Application Migration Service. In order to measure storage speeds, you can use the following tools:\n\n- Windows: WinSat (or other third-party tooling, like [AS SSD Benchmark](https://as-ssd-benchmark.en.softonic.com/))\n- Linux: hdparm\n\nWe suggest reducing read/write operations on your source storage when starting your migration using Application Migration Service.\n\n\n#### **Application Migration Service EC2 replication instance size**\n\n\nThe size of the EC2 replication server instance can also have an impact on the replication speed. Although it is recommended to keep the default instance size (t3.small), it can be increased if there are business requirements, like to speed up the initial data sync. Note: using a larger instance can lead to increased compute costs.\n\n\n\nCommon replication instance changes include:\n\n- Servers with <26 disks: change the instance type to m5.large. Increase the instance type to m5.xlarge or higher, as needed.\\n- Servers with <26 disks (or servers in Amazon Web Services Regions that do not support m5 instance types): change the instance type to m4.large. Increase to m4.xlarge or higher, as needed.\\n\\nNote: Changing the replication server instance type will not affect data replication. Data replication will automatically pick up where it left off, using the new instance type you selected.\\n\\n\\n#### **Application Migration Service Elastic Block Store replication volume**\\n\\n\\nYou can customize the [Amazon Elastic Block Store (Amazon EBS)](https://aws.amazon.com/ebs/) volume type used by each disk within each source server in that source server’s settings ([change staging disk type](https://docs.aws.amazon.com/mgn/latest/ug/disk-settings.html#staging-disk)).\\n\\nBy default, disks <500GiB use Magnetic HDD volumes. Amazon Web Services best practice suggests not change the default Amazon EBS volume type, unless there is a business need for doing so. However, as we aim to speed up the replication, we actively change the default EBS volume type.\\n\\nThere are two options to choose from:\\n\\n1. The lower cost, Throughput Optimized HDD (st1) option utilizes slower, less expensive disks.\\n\\n\\n\\n2. The faster, General Purpose SSD (gp2) option utilizes faster, but more expensive disks.\\n\\n\\n\\nConsider this option if you(r):\\n- Source server has disks with a high write rate, or if you need faster performance in general\\n- Want to speed up the initial sync process\\n- Are willing to pay more for speed\\n\\n\\n#### **Source server CPU**\\n\\n\\nThe Application Migration Service agent that is installed on the source machine for data replication uses a single core in most cases (agent threads can be scheduled to multiple cores). If core utilization reaches a maximum, this can be a limitation for replication speed. In order to check the core utilization:\\n\\n- Windows: Launch the Task Manger application within Windows, and click on the “CPU” tab. Right click on the CPU graph (this is currently showing an average of cores) > select “Change graph to” > “Logical processors”. This will show individual cores and their current utilization (Figure 2).\\n\\n\\n\\nFigure 2. Logical processor CPU utilization\\n\\nLinux: Install htop and run from the terminal. The htop command will display the Application Migration Service/CE process and indicate the CPU and memory utilization percentage (this is of the entire machine). You can check the CPU bars to determine if a CPU is being maxed-out (Figure 3).\\n\\n\\n\\nFigure 3. Amazon Web Services Application Migration Service/CE process to assess CPU utilization\\n\\n\\n#### **Conclusion**\\n\\n\\nIn this post, we explored several key variables that contribute to server replication speed when using Application Migration Service. We encourage you to explore these key areas during your migration to determine if your replication speed can be optimized.\\n\\n\\n#### **Related information**\\n\\n\\n- [What Is Amazon Web Services Application Migration Service](https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html)?\\n- [Amazon Web Services Application Migration Service Documentation](https://docs.aws.amazon.com/mgn/?id=docs_gateway)\\n\\n\\n\\n\\n#### **Tobias Reekers**\\n\\n\\nTobias Reekers is a Migrations Consultant from Amazon Web Services Professional Services, Zurich, Switzerland. He enjoys to support customers solving meaningful business problems applying cutting edge technology at scale. Tobias supports different types of customers with onboarding to the Amazon Web Services Cloud.\\n\\n\\n\\n\\n#### **Kishan Dave**\\n\\n\\nKishan Dave is a Migration & Modernisation Specialist Solutions Architect based in London, UK. Kishan helps Amazon Web Services customers across all industries migrate to Amazon Web Services and take advantage of cloud-native technologies to support business challenges. In his spare time, Kishan enjoys photography, films and travel.","render":"<p>Enterprises frequently begin their journey by re-hosting (lift-and-shift) their on-premises workloads into <a href=\\"https://aws.amazon.com/ec2/\\" target=\\"_blank\\">Amazon Web Services and running Amazon Elastic Compute Cloud (Amazon EC2)</a> instances. A simpler way to re-host is by using <a href=\\"https://aws.amazon.com/application-migration-service/\\" target=\\"_blank\\">Amazon Web Services Application Migration Service (Application Migration Service)</a>, a cloud-native migration service.</p>\\n<p>To streamline and expedite migrations, automate reusable migration patterns that work for a wide range of applications. Application Migration Service is the recommended migration service to lift-and-shift your applications to Amazon Web Services.</p>\n<p>In this blog post, we explore key variables that contribute to server replication speed when using Application Migration Service. We will also look at tests you can run to identify these bottlenecks and, where appropriate, include remediation steps.</p>\n<h4><a id=\\"Overview_of_migration_using_Application_Migration_Service_7\\"></a><strong>Overview of migration using Application Migration Service</strong></h4>\\n<p>Figure 1 depicts the end-to-end data replication flow from source servers to a target machine hosted on Amazon Web Services. The diagram is designed to help visualize potential bottlenecks within the data flow, which are denoted by a black diamond.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/d04dd8c516f5464cb126fa35131e2126_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 1. Data flow when using Amazon Web Services Application Migration Service (black diamonds denote potential points of contention)</p>\n<h4><a id=\\"Baseline_testing_17\\"></a><strong>Baseline testing</strong></h4>\\n<p>To determine a baseline replication speed, we recommend performing a control test between your target Amazon Web Services Region and the nearest Region to your source workloads. For example, if your source workloads are in a data center in Rome and your target Region is Paris, run a test between eu-south-1 (Milan) and eu-west-3 (Paris). This will give a theoretical upper bandwidth limit, as replication will occur over the Amazon Web Services backbone. If the target Region is already the closest Region to your source workloads, run the test from within the same Region.</p>\n<h4><a id=\\"Network_connectivity_23\\"></a><strong>Network connectivity</strong></h4>\\n<p>There are several ways to establish connectivity between your on-premises location and Amazon Web Services Region:</p>\n<ul>\\n<li>Public internet</li>\n<li>VPN</li>\n<li><a href=\\"https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html\\" target=\\"_blank\\">Amazon Web Services Direct Connect</a></li>\\n</ul>\n<p>This section pertains to options 1 and 2. If facing replication speed issues, the first place to look is at network bandwidth. From a source machine within your internal network, run a speed test to calculate your bandwidth out to the internet; common test providers include <a href=\\"https://www.cloudflare.com/\\" target=\\"_blank\\">Cloudflare</a>, <a href=\\"https://www.ookla.com/\\" target=\\"_blank\\">Ookla</a>, and Google. This is your bandwidth to the internet, not to Amazon Web Services.</p>\\n<p>Next, to confirm the data flow from within your data center, run a <code>traceroute</code> (Windows) or <code>tracert</code> (Linux). Identify any network hops that are unusual or potentially throttling bandwidth (due to hardware limitations or configuration).</p>\\n<p>To measure the maximum bandwidth between your data center and the Amazon Web Services subnet that is being used for data replication, while accounting for Security Sockets Layer (SSL) encapsulation, use the <a href=\\"https://docs.cloudendure.com/Content/FAQ/FAQ/Replication_Related.htm#How\\" target=\\"_blank\\">CloudEndure SSL bandwidth tool</a> (refer to Figure 1).</p>\\n<h4><a id=\\"Source_storage_IO_39\\"></a><strong>Source storage I/O</strong></h4>\\n<p>The next area to look for replication bottlenecks is source storage. The underlying storage for servers can be a point of contention. If the storage is maxing-out its read speeds, this will impact the data-replication rate. If your storage I/O is heavily utilized, it can impact block replication by Application Migration Service. In order to measure storage speeds, you can use the following tools:</p>\n<ul>\\n<li>Windows: WinSat (or other third-party tooling, like <a href=\\"https://as-ssd-benchmark.en.softonic.com/\\" target=\\"_blank\\">AS SSD Benchmark</a>)</li>\\n<li>Linux: hdparm</li>\n</ul>\\n<p>We suggest reducing read/write operations on your source storage when starting your migration using Application Migration Service.</p>\n<h4><a id=\\"Application_Migration_Service_EC2_replication_instance_size_50\\"></a><strong>Application Migration Service EC2 replication instance size</strong></h4>\\n<p>The size of the EC2 replication server instance can also have an impact on the replication speed. Although it is recommended to keep the default instance size (t3.small), it can be increased if there are business requirements, like to speed up the initial data sync. Note: using a larger instance can lead to increased compute costs.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/29c593921a6e40689c4d4d09725ed160_image.png\\" alt=\\"image.png\\" /></p>\n<p>Common replication instance changes include:</p>\n<ul>\\n<li>Servers with &lt;26 disks: change the instance type to m5.large. Increase the instance type to m5.xlarge or higher, as needed.</li>\n<li>Servers with &lt;26 disks (or servers in Amazon Web Services Regions that do not support m5 instance types): change the instance type to m4.large. Increase to m4.xlarge or higher, as needed.</li>\n</ul>\\n<p>Note: Changing the replication server instance type will not affect data replication. Data replication will automatically pick up where it left off, using the new instance type you selected.</p>\n<h4><a id=\\"Application_Migration_Service_Elastic_Block_Store_replication_volume_65\\"></a><strong>Application Migration Service Elastic Block Store replication volume</strong></h4>\\n<p>You can customize the <a href=\\"https://aws.amazon.com/ebs/\\" target=\\"_blank\\">Amazon Elastic Block Store (Amazon EBS)</a> volume type used by each disk within each source server in that source server’s settings (<a href=\\"https://docs.aws.amazon.com/mgn/latest/ug/disk-settings.html#staging-disk\\" target=\\"_blank\\">change staging disk type</a>).</p>\\n<p>By default, disks &lt;500GiB use Magnetic HDD volumes. Amazon Web Services best practice suggests not change the default Amazon EBS volume type, unless there is a business need for doing so. However, as we aim to speed up the replication, we actively change the default EBS volume type.</p>\n<p>There are two options to choose from:</p>\n<ol>\\n<li>The lower cost, Throughput Optimized HDD (st1) option utilizes slower, less expensive disks.</li>\n</ol>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/6fccad70a25c471e988309da065e86f8_image.png\\" alt=\\"image.png\\" /></p>\n<ol start=\\"2\\">\\n<li>The faster, General Purpose SSD (gp2) option utilizes faster, but more expensive disks.</li>\n</ol>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/1946ccd44e494290bd722f16bb030b79_image.png\\" alt=\\"image.png\\" /></p>\n<p>Consider this option if you®:</p>\n<ul>\\n<li>Source server has disks with a high write rate, or if you need faster performance in general</li>\n<li>Want to speed up the initial sync process</li>\n<li>Are willing to pay more for speed</li>\n</ul>\\n<h4><a id=\\"Source_server_CPU_88\\"></a><strong>Source server CPU</strong></h4>\\n<p>The Application Migration Service agent that is installed on the source machine for data replication uses a single core in most cases (agent threads can be scheduled to multiple cores). If core utilization reaches a maximum, this can be a limitation for replication speed. In order to check the core utilization:</p>\n<ul>\\n<li>Windows: Launch the Task Manger application within Windows, and click on the “CPU” tab. Right click on the CPU graph (this is currently showing an average of cores) &gt; select “Change graph to” &gt; “Logical processors”. This will show individual cores and their current utilization (Figure 2).</li>\n</ul>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/6fe0fc12a07644f9bfd744c2dd031dbc_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 2. Logical processor CPU utilization</p>\n<p>Linux: Install htop and run from the terminal. The htop command will display the Application Migration Service/CE process and indicate the CPU and memory utilization percentage (this is of the entire machine). You can check the CPU bars to determine if a CPU is being maxed-out (Figure 3).</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/be110df964124eacbf2ca7e0d89e1e3c_image.png\\" alt=\\"image.png\\" /></p>\n<p>Figure 3. Amazon Web Services Application Migration Service/CE process to assess CPU utilization</p>\n<h4><a id=\\"Conclusion_106\\"></a><strong>Conclusion</strong></h4>\\n<p>In this post, we explored several key variables that contribute to server replication speed when using Application Migration Service. We encourage you to explore these key areas during your migration to determine if your replication speed can be optimized.</p>\n<h4><a id=\\"Related_information_112\\"></a><strong>Related information</strong></h4>\\n<ul>\\n<li><a href=\\"https://docs.aws.amazon.com/mgn/latest/ug/what-is-application-migration-service.html\\" target=\\"_blank\\">What Is Amazon Web Services Application Migration Service</a>?</li>\\n<li><a href=\\"https://docs.aws.amazon.com/mgn/?id=docs_gateway\\" target=\\"_blank\\">Amazon Web Services Application Migration Service Documentation</a></li>\\n</ul>\n<p><img src=\\"https://dev-media.amazoncloud.cn/51dc9cf99a1d4ce39faac528c7509801_image.png\\" alt=\\"image.png\\" /></p>\n<h4><a id=\\"Tobias_Reekers_121\\"></a><strong>Tobias Reekers</strong></h4>\\n<p>Tobias Reekers is a Migrations Consultant from Amazon Web Services Professional Services, Zurich, Switzerland. He enjoys to support customers solving meaningful business problems applying cutting edge technology at scale. Tobias supports different types of customers with onboarding to the Amazon Web Services Cloud.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/9ac4c676e40e4daebd944df1422b3a57_image.png\\" alt=\\"image.png\\" /></p>\n<h4><a id=\\"Kishan_Dave_129\\"></a><strong>Kishan Dave</strong></h4>\\n<p>Kishan Dave is a Migration &amp; Modernisation Specialist Solutions Architect based in London, UK. Kishan helps Amazon Web Services customers across all industries migrate to Amazon Web Services and take advantage of cloud-native technologies to support business challenges. In his spare time, Kishan enjoys photography, films and travel.</p>\n"}