Introducing Amazon S3 Transfer Manager in the Amazon SDK for Java 2.x

{"value":"We are pleased to announce the Developer Preview release of the Amazon S3 Transfer Manager – a high level file transfer utility for the Amazon Simple Storage Service (Amazon S3) in the [AWS SDK for Java 2.x](https://github.com/aws/aws-sdk-java-v2).\n\nUsing Transfer Manager’s simple API, you can now perform accelerated uploads and downloads of objects to and from Amazon S3 and benefit from enhanced throughput and reliability, which is achieved through concurrent transfers of a set of small parts from a single object. The Transfer Manager is built on top of the Java bindings of the [AWS Common Runtime S3 client](https://github.com/awslabs/aws-crt-java) and leverages Amazon S3 multipart upload and byte-range fetches for parallel transfers.\n\n#### **Parallel upload via multipart upload**\n\nFor upload operation, the Transfer Manager uses the Amazon S3 [multipart upload API](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html); it converts one single [PutObjectRequest](PutObjectRequest) to multiple [MultiPartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) requests and sends those requests concurrently to achieve high performance.\n\n#### **Parallel download via byte-range fetches**\n\nFor download operation, the Transfer Manager utilizes [byte-range fetches](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html). It splits a [GetObjectRequest](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) to multiple smaller requests, each of which retrieves a specific portion of the object. Those requests are also executed through concurrent connections to Amazon S3.\n\nIf you are using Transfer Manager 1.x, an object must be uploaded using[ multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) for it to be eligible to be downloaded with optimized performance. If an object was originally uploaded as a single object, the Transfer Manager will not be able to accelerate the downloading process.\n\nWith the new Transfer Manager 2.x, this is no longer a limitation – downloading an object does not depend on how the object was originally uploaded, and all downloads can benefit from high throughput and concurrency.\n\n#### **Getting Started**\n\n#### **Add a dependency for the Transfer Manager**\n\nFirst, you need to include the separate dependency in your project.\n\n```\nXML\n```\n```\n<dependency>\n <groupId>software.amazon.awssdk</groupId>\n <artifactId>s3-transfer-manager</artifactId>\n <version>2.17.123-PREVIEW</version>\n</dependency>\n```\n\n#### **Instantiate the Transfer Manager**\n\nYou can instantiate the Transfer Manager easily using the default settings\n\n```\nJava\n```\n```\nS3TransferManager transferManager = S3TransferManager.create();\n```\n\nIf you wish to configure settings, we recommend using the builder instead:\n\n```\nJava\n```\n```\nS3TransferManager transferManager =\n S3TransferManager.builder()\n .s3ClientConfiguration(cfg -> cfg.credentialsProvider(credentialProvider)\n .region(Region.US_WEST_2)\n .targetThroughputInGbps(20.0)\n .minimumPartSizeInBytes(10 * MB))\n .build();\n```\n\n#### **Upload a file to Amazon S3**\n\nTo upload a file to S3, you need to provide the source file path and the PutObjectRequest that should be used for the upload.\n\n ```\nJava\n```\n```\nFileUpload upload = transferManager.uploadFile(b -> b.source(Paths.get(\"myFile.txt\"))\n .putObjectRequest(req -> req.bucket(\"bucket\")\n .key(\"key\")));\n\nupload.completionFuture().join();\n```\n\n\n#### **Download an Amazon S3 object to a file**\n\nTo download an object, you need to provide the destination file path and the GetObjectRequest that should be used for the download.\n\n```\nJava\n```\n```\nFileDownload download = \n transferManager.downloadFile(b -> b.destination(Paths.get(\"myFile.txt\"))\n .getObjectRequest(req -> req.bucket(\"bucket\")\n .key(\"key\")));\ndownload.completionFuture().join();\n```\n\n#### **Conclusion**\n\n\nTo learn how to set up and begin using the Transfer Manager for the AWS SDK for Java 2.x, visit our [Developer Guide](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html). If you want to learn more, you can check out the [source code](https://github.com/aws/aws-sdk-java-v2/tree/master/services-custom/s3-transfer-manager) on GitHub. Try out the new Transfer Manager today and let us know what you think via the GitHub issues page!\n\n\n[**Zoe Wang**](https://github.com/zoewangg)\n\nZoe is a Software Development Engineer working on the AWS SDK for Java. She is passionate about building tools to improve the developer experience. You can find her on GitHub @zoewangg.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","render":"<p>We are pleased to announce the Developer Preview release of the Amazon S3 Transfer Manager – a high level file transfer utility for the Amazon Simple Storage Service (Amazon S3) in the <a href=\"https://github.com/aws/aws-sdk-java-v2\" target=\"_blank\">AWS SDK for Java 2.x</a>.</p>\n<p>Using Transfer Manager’s simple API, you can now perform accelerated uploads and downloads of objects to and from Amazon S3 and benefit from enhanced throughput and reliability, which is achieved through concurrent transfers of a set of small parts from a single object. The Transfer Manager is built on top of the Java bindings of the <a href=\"https://github.com/awslabs/aws-crt-java\" target=\"_blank\">AWS Common Runtime S3 client</a> and leverages Amazon S3 multipart upload and byte-range fetches for parallel transfers.</p>\n<h4><a id=\"Parallel_upload_via_multipart_upload_4\"></a><strong>Parallel upload via multipart upload</strong></h4>\n<p>For upload operation, the Transfer Manager uses the Amazon S3 <a href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html\" target=\"_blank\">multipart upload API</a>; it converts one single <a href=\"PutObjectRequest\" target=\"_blank\">PutObjectRequest</a> to multiple <a href=\"https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html\" target=\"_blank\">MultiPartUpload</a> requests and sends those requests concurrently to achieve high performance.</p>\n<h4><a id=\"Parallel_download_via_byterange_fetches_8\"></a><strong>Parallel download via byte-range fetches</strong></h4>\n<p>For download operation, the Transfer Manager utilizes <a href=\"https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html\" target=\"_blank\">byte-range fetches</a>. It splits a <a href=\"https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html\" target=\"_blank\">GetObjectRequest</a> to multiple smaller requests, each of which retrieves a specific portion of the object. Those requests are also executed through concurrent connections to Amazon S3.</p>\n<p>If you are using Transfer Manager 1.x, an object must be uploaded using<a href=\"https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html\" target=\"_blank\"> multipart upload</a> for it to be eligible to be downloaded with optimized performance. If an object was originally uploaded as a single object, the Transfer Manager will not be able to accelerate the downloading process.</p>\n<p>With the new Transfer Manager 2.x, this is no longer a limitation – downloading an object does not depend on how the object was originally uploaded, and all downloads can benefit from high throughput and concurrency.</p>\n<h4><a id=\"Getting_Started_16\"></a><strong>Getting Started</strong></h4>\n<h4><a id=\"Add_a_dependency_for_the_Transfer_Manager_18\"></a><strong>Add a dependency for the Transfer Manager</strong></h4>\n<p>First, you need to include the separate dependency in your project.</p>\n<pre><code class=\"lang-\">XML\n</code></pre>\n<pre><code class=\"lang-\">&lt;dependency&gt;\n &lt;groupId&gt;software.amazon.awssdk&lt;/groupId&gt;\n &lt;artifactId&gt;s3-transfer-manager&lt;/artifactId&gt;\n &lt;version&gt;2.17.123-PREVIEW&lt;/version&gt;\n&lt;/dependency&gt;\n</code></pre>\n<h4><a id=\"Instantiate_the_Transfer_Manager_33\"></a><strong>Instantiate the Transfer Manager</strong></h4>\n<p>You can instantiate the Transfer Manager easily using the default settings</p>\n<pre><code class=\"lang-\">Java\n</code></pre>\n<pre><code class=\"lang-\">S3TransferManager transferManager = S3TransferManager.create();\n</code></pre>\n<p>If you wish to configure settings, we recommend using the builder instead:</p>\n<pre><code class=\"lang-\">Java\n</code></pre>\n<pre><code class=\"lang-\">S3TransferManager transferManager =\n S3TransferManager.builder()\n .s3ClientConfiguration(cfg -&gt; cfg.credentialsProvider(credentialProvider)\n .region(Region.US_WEST_2)\n .targetThroughputInGbps(20.0)\n .minimumPartSizeInBytes(10 * MB))\n .build();\n</code></pre>\n<h4><a id=\"Upload_a_file_to_Amazon_S3_59\"></a><strong>Upload a file to Amazon S3</strong></h4>\n<p>To upload a file to S3, you need to provide the source file path and the PutObjectRequest that should be used for the upload.</p>\n<pre><code class=\"lang-\">Java\n</code></pre>\n<pre><code class=\"lang-\">FileUpload upload = transferManager.uploadFile(b -&gt; b.source(Paths.get(&quot;myFile.txt&quot;))\n .putObjectRequest(req -&gt; req.bucket(&quot;bucket&quot;)\n .key(&quot;key&quot;)));\n\nupload.completionFuture().join();\n</code></pre>\n<h4><a id=\"Download_an_Amazon_S3_object_to_a_file_75\"></a><strong>Download an Amazon S3 object to a file</strong></h4>\n<p>To download an object, you need to provide the destination file path and the GetObjectRequest that should be used for the download.</p>\n<pre><code class=\"lang-\">Java\n</code></pre>\n<pre><code class=\"lang-\">FileDownload download = \n transferManager.downloadFile(b -&gt; b.destination(Paths.get(&quot;myFile.txt&quot;))\n .getObjectRequest(req -&gt; req.bucket(&quot;bucket&quot;)\n .key(&quot;key&quot;)));\ndownload.completionFuture().join();\n</code></pre>\n<h4><a id=\"Conclusion_90\"></a><strong>Conclusion</strong></h4>\n<p>To learn how to set up and begin using the Transfer Manager for the AWS SDK for Java 2.x, visit our <a href=\"https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/transfer-manager.html\" target=\"_blank\">Developer Guide</a>. If you want to learn more, you can check out the <a href=\"https://github.com/aws/aws-sdk-java-v2/tree/master/services-custom/s3-transfer-manager\" target=\"_blank\">source code</a> on GitHub. Try out the new Transfer Manager today and let us know what you think via the GitHub issues page!</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/663ddb2f2aaa42839c852d3f380763e4_image.png\" alt=\"image.png\" /><br />\n<a href=\"https://github.com/zoewangg\" target=\"_blank\"><strong>Zoe Wang</strong></a></p>\n<p>Zoe is a Software Development Engineer working on the AWS SDK for Java. She is passionate about building tools to improve the developer experience. You can find her on GitHub @zoewangg.</p>\n"}