Target cross-platform Go builds with Amazon CodeBuild Batch builds

{"value":"Many different operating systems and architectures could end up as the destination for our applications. By using a [AWS CodeBuild](http://aws.amazon.com/codebuild) batch build, we can run builds for a Go application targeted at multiple platforms concurrently.\n\nCross-compiling Go binaries for different platforms is as simple as setting two environment variables ``` $GOOS``` and ``` $GOARCH ```\n, regardless of the build’s host platform. For this post we will build all of the binaries on Linux/x86 containers. You can run the command ``` go tool dist list``` to see the Go list of supported platforms. We will build binaries for six platforms: Windows+ARM, Windows+AMD64, Linux+ARM64, Linux+AMD64, MacOS+ARM64, and Mac+AMD64. Note that AMD64 is a 64-bit architecture based on the Intel x86 instruction set utilized on both AMD and Intel hardware.\n\nThis post demonstrates how to create a single AWS CodeBuild project by using a batch build and a single build spec to create concurrent builds for the six targeted platforms. Learn more about batch builds in AWS CodeBuild in the documentation: [Batch builds in AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/batch-build.html)\n\n### **Solution Overview**\nAWS CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces ready-to-deploy software packages. A batch build is utilized to run concurrent and coordinated builds. Let’s summarize the 3 different batch builds:\n\n- Build graph: defines dependencies between builds. CodeBuild utilizes the dependencies graph to run builds in an order that satisfies the dependencies.\n- Build list: utilizes a list of settings for concurrently run builds.\n- Build matrix: utilizes a matrix of settings to create a build for every combination.\n\t\n\nThe requirements for this project are simple – run multiple builds with a platform pair of``` $GOOS ``` and``` $GOARCH ```environment variables. For this, a build list can be utilized. The buildspec for the project contains a batch/build-list setting containing every environment variable for the six builds.\n\nYAML\n```\nbatch:\n build-list:\n - identifier: build1\n env:\n variables:\n GOOS: darwin\n GOARCH: amd64\n - identifier: build2\n env:\n variables:\n GOOS: darwin\n GOARCH: arm64\n - ...\n```\nThe batch build project will launch seven builds. See the build sequence in the diagram below.\n\n- Step 1 – A build downloads the source.\n- Step 2 – Six concurrent builds configured with six sets of environment variables from the batch/build-list setting.\n- Step 3 – Concurrent builds package a zip file and deliver to the artifacts [Amazon Simple Storage Service](https://aws.amazon.com/s3/) (Amazon S3) \nbucket.\n\n\n\nThe [supplied buildspec file](https://github.com/aws-samples/cross-platform-go-builds-with-aws-codebuild/blob/main/buildspec.yml) includes commands for the install and build phases. The install phase utilizes the ```phases/install/runtime-versions``` phase to set the version of Go is used in the build container.\n\nThe build phase contains commands to replace source code placeholders with environment variables set by CodeBuild. The entire list of environment variables is documented at [Environment variables in build environments](https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-env-vars.html). This is followed by a simple ```go build``` to build the binaries. The application getting built is an [AWS SDK for Go](https://aws.amazon.com/sdk-for-go/) sample that will list the contents of an S3 bucket.\n\nYAML\n```\n build:\n commands:\n - mv listObjects.go listObjects.go.tmp\n - cat listObjects.go.tmp | envsubst | tee listObjects.go\n - go build listObjects.go\n```\nThe artifacts sequence specifies the build outputs that we want packaged and delivered to the artifacts S3 bucket. The name setting creates a name for ZIP artifact. And the name combines the operating system, architecture environment variables, as well as the git commit hash. We use Shell command language to expand the environment variables, as well as command substitution to take the first seven characters of the git hash.\n\nYAML\n```\nartifacts:\n files:\n - 'listObjects'\n - 'listObjects.exe'\n - 'README.md'\n name: listObjects_${GOOS}_${GOARCH}_$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7).zip \n```\n\nLet’s walk through the CodeBuild project setup process. When the builds complete, we’ll see zip files containing the builds for each platform in an artifacts bucket.\n\nHere is what we will create:\n\n- Create an S3 bucket to host the built artifacts.\n- Create the CodeBuild project, which needs to know:\n\t1. Where the source is\n\t2. The environment – a docker image and a service role\n\t3. The location of the build spec\n\t4. Batch configuration, a service role used to launch batch build groups\n\t5. The artifact S3 bucket location\n\n\nThe code that is built is available on github here: [https://github.com/aws-samples/cross-platform-go-builds-with-aws-codebuild](https://github.com/aws-samples/cross-platform-go-builds-with-aws-codebuild). For an alternative to the manual walkthrough steps, a CloudFormation template that will build all of the resources is available in the git repository.\n\n### **Prerequisites**\nFor this walkthrough, you must have the following prerequisites:\n\n- An [AWS account](https://portal.aws.amazon.com/billing/signup)\n- Access to an AWS account with administrator or PowerUser (or equivalent) [AWS Identity and Access Management](https://aws.amazon.com/iam/) (IAM) role policies attached\n\n### **Create the Artifacts S3 Bucket**\nAn S3 bucket will be the destination of the build artifacts.\n\n- In the Amazon S3 console, create a bucket with a unique name. This will be the destination of your build artifacts.\n\n\n\n### **Create the AWS CodeBuild Project**\n\n- n the AWS CodeBuild console, create a project named ```multi-arch-build```.\n\n\n\n- For **Source Provider**, choose **GitHub**. Choose the **Connect to GitHub** button and follow the authorization screens. For repository, enter\n\n```https://github.com/aws-samples/cross-platform-go-builds-with-aws-codebuild```\n\n\n\n- For **Environment image**, choose **Managed Image**. Choose the most recent Ubuntu image. This image contains every tool needed for the build. If you are interested in the contents of the image, then you can see the Dockerfile used to build the image in the GitHub repository here: [https://github.com/aws/aws-codebuild-docker-images/tree/master/ubuntu](https://github.com/aws/aws-codebuild-docker-images/tree/master/ubuntu)\n\n\n\n- For **Service Role**, keep the suggested role name.\n\n\n\n- For **Build specifications**, leave **Buildspec** name empty. This will use the default location ```buildspec.yml``` in the source root.\n- Under **Batch configuration**, enable **Define batch configuration**. For the **Role Name**, enter a name for the role: \n```batch-multi-arch-build-service-role```. There is an option here to combine artifacts. CodeBuild can combine the artifacts from batch builds into a single location. This isn’t needed for this build, as we want a zip to be created for each platform.\n\n\n\n- Under Artifacts, for **Type**, choose **Amazon S3**. For **Bucket name**, select the S3 bucket created earlier. This is where we want the build artifacts delivered. Choose the checkbox for **Enable semantic versioning**. This will tell CodeBuild to use the artifact name that was specified in the buildspec file.\n\n\n\n- For **Artifacts Packaging**, choose **Zip** for CodeBuild to create a compressed zip from the build artifacts.\n\n\n\n- Create the build project, and start a build. You will see the ```DOWNLOAD_SOURCE``` build complete, followed by the six concurrent builds for each combination of OS and architecture.\n\n\n\n### **Run the Artifacts**\nThe builds have completed, and each packaged artifact has been delivered to the S3 bucket. Remember the name of the ZIP archive that was built using the buildspec file setting. This incorporated a combination of the operating system, architecture, and git commit hash.\n\n\n\nBelow, I have tested the artifact by downloading the zip for the operating system and architecture combination on three different platforms: MacOS/AMD64, an AWS Graviton2 instance, and a Microsoft Windows instance. Note the system information, unzipping the platform artifact, and the build specific information substituted into the Go source code.\n\n\n\n\n\n\n\n### **Cleaning up**\nTo avoid incurring future charges, delete the resources:\n\n- On the Amazon S3 console, choose the artifacts bucket created, and choose Empty. Confirm the deletion by typing ‘permanently delete’. Choose **Empty**.\n- Choose the artifacts bucket created, and **Delete**.\n- On the IAM console, choose **Roles**.\n- Search for ```batch-multi-arch-build-service-role``` and **Delete**. Search for\n ```codebuild-multi-arch-build-service-role``` and **Delete**.\n- Go to the CodeBuild console. From Build projects, choose ```multi-arch-build```, and choose **Delete build project**.\n\n### **Conclusion**\nThis post utilized CodeBuild batch builds to build and package binaries for multiple platforms concurrently. The build phase used a small amount of scripting to replace placeholders in the code with build information CodeBuild makes available in environment variables. By overriding the artifact name using the buildspec setting, we created zip files built from information about the build. The zip artifacts were downloaded and tested on three platforms: Intel MacOS, a Graviton ARM based EC2 instance, and Microsoft Windows.\n\nFeatures like this let you build on CodeBuild, a fully managed build service – and not have to worry about maintaining your own build servers.\n","render":"<p>Many different operating systems and architectures could end up as the destination for our applications. By using a <a href=\"http://aws.amazon.com/codebuild\" target=\"_blank\">AWS CodeBuild</a> batch build, we can run builds for a Go application targeted at multiple platforms concurrently.</p>\n<p>Cross-compiling Go binaries for different platforms is as simple as setting two environment variables <code> $GOOS</code> and <code>$GOARCH</code><br />\n, regardless of the build’s host platform. For this post we will build all of the binaries on Linux/x86 containers. You can run the command <code> go tool dist list</code> to see the Go list of supported platforms. We will build binaries for six platforms: Windows+ARM, Windows+AMD64, Linux+ARM64, Linux+AMD64, MacOS+ARM64, and Mac+AMD64. Note that AMD64 is a 64-bit architecture based on the Intel x86 instruction set utilized on both AMD and Intel hardware.</p>\n<p>This post demonstrates how to create a single AWS CodeBuild project by using a batch build and a single build spec to create concurrent builds for the six targeted platforms. Learn more about batch builds in AWS CodeBuild in the documentation: <a href=\"https://docs.aws.amazon.com/codebuild/latest/userguide/batch-build.html\" target=\"_blank\">Batch builds in AWS CodeBuild</a></p>\n<h3><a id=\"Solution_Overview_7\"></a><strong>Solution Overview</strong></h3>\n<p>AWS CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces ready-to-deploy software packages. A batch build is utilized to run concurrent and coordinated builds. Let’s summarize the 3 different batch builds:</p>\n<ul>\n<li>Build graph: defines dependencies between builds. CodeBuild utilizes the dependencies graph to run builds in an order that satisfies the dependencies.</li>\n<li>Build list: utilizes a list of settings for concurrently run builds.</li>\n<li>Build matrix: utilizes a matrix of settings to create a build for every combination.</li>\n</ul>\n<p>The requirements for this project are simple – run multiple builds with a platform pair of<code>$GOOS</code> and<code>$GOARCH</code>environment variables. For this, a build list can be utilized. The buildspec for the project contains a batch/build-list setting containing every environment variable for the six builds.</p>\n<p>YAML</p>\n<pre><code class=\"lang-\">batch:\n build-list:\n - identifier: build1\n env:\n variables:\n GOOS: darwin\n GOARCH: amd64\n - identifier: build2\n env:\n variables:\n GOOS: darwin\n GOARCH: arm64\n - ...\n</code></pre>\n<p>The batch build project will launch seven builds. See the build sequence in the diagram below.</p>\n<ul>\n<li>Step 1 – A build downloads the source.</li>\n<li>Step 2 – Six concurrent builds configured with six sets of environment variables from the batch/build-list setting.</li>\n<li>Step 3 – Concurrent builds package a zip file and deliver to the artifacts <a href=\"https://aws.amazon.com/s3/\" target=\"_blank\">Amazon Simple Storage Service</a> (Amazon S3)<br />\nbucket.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/3268af9f2a1e4a748d109894249411cc_image.png\" alt=\"image.png\" /></p>\n<p>The <a href=\"https://github.com/aws-samples/cross-platform-go-builds-with-aws-codebuild/blob/main/buildspec.yml\" target=\"_blank\">supplied buildspec file</a> includes commands for the install and build phases. The install phase utilizes the <code>phases/install/runtime-versions</code> phase to set the version of Go is used in the build container.</p>\n<p>The build phase contains commands to replace source code placeholders with environment variables set by CodeBuild. The entire list of environment variables is documented at <a href=\"https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-env-vars.html\" target=\"_blank\">Environment variables in build environments</a>. This is followed by a simple <code>go build</code> to build the binaries. The application getting built is an <a href=\"https://aws.amazon.com/sdk-for-go/\" target=\"_blank\">AWS SDK for Go</a> sample that will list the contents of an S3 bucket.</p>\n<p>YAML</p>\n<pre><code class=\"lang-\"> build:\n commands:\n - mv listObjects.go listObjects.go.tmp\n - cat listObjects.go.tmp | envsubst | tee listObjects.go\n - go build listObjects.go\n</code></pre>\n<p>The artifacts sequence specifies the build outputs that we want packaged and delivered to the artifacts S3 bucket. The name setting creates a name for ZIP artifact. And the name combines the operating system, architecture environment variables, as well as the git commit hash. We use Shell command language to expand the environment variables, as well as command substitution to take the first seven characters of the git hash.</p>\n<p>YAML</p>\n<pre><code class=\"lang-\">artifacts:\n files:\n - 'listObjects'\n - 'listObjects.exe'\n - 'README.md'\n name: listObjects_${GOOS}_${GOARCH}_$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7).zip \n</code></pre>\n<p>Let’s walk through the CodeBuild project setup process. When the builds complete, we’ll see zip files containing the builds for each platform in an artifacts bucket.</p>\n<p>Here is what we will create:</p>\n<ul>\n<li>Create an S3 bucket to host the built artifacts.</li>\n<li>Create the CodeBuild project, which needs to know:\n<ol>\n<li>Where the source is</li>\n<li>The environment – a docker image and a service role</li>\n<li>The location of the build spec</li>\n<li>Batch configuration, a service role used to launch batch build groups</li>\n<li>The artifact S3 bucket location</li>\n</ol>\n</li>\n</ul>\n<p>The code that is built is available on github here: <a href=\"https://github.com/aws-samples/cross-platform-go-builds-with-aws-codebuild\" target=\"_blank\">https://github.com/aws-samples/cross-platform-go-builds-with-aws-codebuild</a>. For an alternative to the manual walkthrough steps, a CloudFormation template that will build all of the resources is available in the git repository.</p>\n<h3><a id=\"Prerequisites_81\"></a><strong>Prerequisites</strong></h3>\n<p>For this walkthrough, you must have the following prerequisites:</p>\n<ul>\n<li>An <a href=\"https://portal.aws.amazon.com/billing/signup\" target=\"_blank\">AWS account</a></li>\n<li>Access to an AWS account with administrator or PowerUser (or equivalent) <a href=\"https://aws.amazon.com/iam/\" target=\"_blank\">AWS Identity and Access Management</a> (IAM) role policies attached</li>\n</ul>\n<h3><a id=\"Create_the_Artifacts_S3_Bucket_87\"></a><strong>Create the Artifacts S3 Bucket</strong></h3>\n<p>An S3 bucket will be the destination of the build artifacts.</p>\n<ul>\n<li>In the Amazon S3 console, create a bucket with a unique name. This will be the destination of your build artifacts.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/b25718e8cf074547b20ebe23e57d737c_image.png\" alt=\"image.png\" /></p>\n<h3><a id=\"Create_the_AWS_CodeBuild_Project_94\"></a><strong>Create the AWS CodeBuild Project</strong></h3>\n<ul>\n<li>n the AWS CodeBuild console, create a project named <code>multi-arch-build</code>.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/26d27fcaf22849a2b861055522d78b74_image.png\" alt=\"image.png\" /></p>\n<ul>\n<li>For <strong>Source Provider</strong>, choose <strong>GitHub</strong>. Choose the <strong>Connect to GitHub</strong> button and follow the authorization screens. For repository, enter</li>\n</ul>\n<p><code>https://github.com/aws-samples/cross-platform-go-builds-with-aws-codebuild</code></p>\n<p><img src=\"https://dev-media.amazoncloud.cn/bf246bf97b104ff5aa6ee7dcbd4dd7a0_image.png\" alt=\"image.png\" /></p>\n<ul>\n<li>For <strong>Environment image</strong>, choose <strong>Managed Image</strong>. Choose the most recent Ubuntu image. This image contains every tool needed for the build. If you are interested in the contents of the image, then you can see the Dockerfile used to build the image in the GitHub repository here: <a href=\"https://github.com/aws/aws-codebuild-docker-images/tree/master/ubuntu\" target=\"_blank\">https://github.com/aws/aws-codebuild-docker-images/tree/master/ubuntu</a></li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/53c76cc114ec4f17b0f8d0f2ffd92856_image.png\" alt=\"image.png\" /></p>\n<ul>\n<li>For <strong>Service Role</strong>, keep the suggested role name.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/a0a09bdcbddf42249bfdff592469588d_image.png\" alt=\"image.png\" /></p>\n<ul>\n<li>For <strong>Build specifications</strong>, leave <strong>Buildspec</strong> name empty. This will use the default location <code>buildspec.yml</code> in the source root.</li>\n<li>Under <strong>Batch configuration</strong>, enable <strong>Define batch configuration</strong>. For the <strong>Role Name</strong>, enter a name for the role:<br />\n<code>batch-multi-arch-build-service-role</code>. There is an option here to combine artifacts. CodeBuild can combine the artifacts from batch builds into a single location. This isn’t needed for this build, as we want a zip to be created for each platform.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/b38a9098c0394dc383d4466e9115291b_image.png\" alt=\"image.png\" /></p>\n<ul>\n<li>Under Artifacts, for <strong>Type</strong>, choose <strong>Amazon S3</strong>. For <strong>Bucket name</strong>, select the S3 bucket created earlier. This is where we want the build artifacts delivered. Choose the checkbox for <strong>Enable semantic versioning</strong>. This will tell CodeBuild to use the artifact name that was specified in the buildspec file.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/8e9af87e22d54b0fb677d0a640feac6e_image.png\" alt=\"image.png\" /></p>\n<ul>\n<li>For <strong>Artifacts Packaging</strong>, choose <strong>Zip</strong> for CodeBuild to create a compressed zip from the build artifacts.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/744fe4acc4534e9abeeaaf11c15ec948_image.png\" alt=\"image.png\" /></p>\n<ul>\n<li>Create the build project, and start a build. You will see the <code>DOWNLOAD_SOURCE</code> build complete, followed by the six concurrent builds for each combination of OS and architecture.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/14fecfc1b1d54bc789ea12856a520c5d_image.png\" alt=\"image.png\" /></p>\n<h3><a id=\"Run_the_Artifacts_132\"></a><strong>Run the Artifacts</strong></h3>\n<p>The builds have completed, and each packaged artifact has been delivered to the S3 bucket. Remember the name of the ZIP archive that was built using the buildspec file setting. This incorporated a combination of the operating system, architecture, and git commit hash.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/450ddcb1e850441b84fefbb749190972_image.png\" alt=\"image.png\" /></p>\n<p>Below, I have tested the artifact by downloading the zip for the operating system and architecture combination on three different platforms: MacOS/AMD64, an AWS Graviton2 instance, and a Microsoft Windows instance. Note the system information, unzipping the platform artifact, and the build specific information substituted into the Go source code.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/66aad46880f34bbb9c9897a82c45665f_image.png\" alt=\"image.png\" /></p>\n<p><img src=\"https://dev-media.amazoncloud.cn/71438679ad014bfa9b76a678b585cef6_image.png\" alt=\"image.png\" /></p>\n<p><img src=\"https://dev-media.amazoncloud.cn/786faa6fe02d4970ad1e8703df6e7a44_image.png\" alt=\"image.png\" /></p>\n<h3><a id=\"Cleaning_up_145\"></a><strong>Cleaning up</strong></h3>\n<p>To avoid incurring future charges, delete the resources:</p>\n<ul>\n<li>On the Amazon S3 console, choose the artifacts bucket created, and choose Empty. Confirm the deletion by typing ‘permanently delete’. Choose <strong>Empty</strong>.</li>\n<li>Choose the artifacts bucket created, and <strong>Delete</strong>.</li>\n<li>On the IAM console, choose <strong>Roles</strong>.</li>\n<li>Search for <code>batch-multi-arch-build-service-role</code> and <strong>Delete</strong>. Search for<br />\n<code>codebuild-multi-arch-build-service-role</code> and <strong>Delete</strong>.</li>\n<li>Go to the CodeBuild console. From Build projects, choose <code>multi-arch-build</code>, and choose <strong>Delete build project</strong>.</li>\n</ul>\n<h3><a id=\"Conclusion_155\"></a><strong>Conclusion</strong></h3>\n<p>This post utilized CodeBuild batch builds to build and package binaries for multiple platforms concurrently. The build phase used a small amount of scripting to replace placeholders in the code with build information CodeBuild makes available in environment variables. By overriding the artifact name using the buildspec setting, we created zip files built from information about the build. The zip artifacts were downloaded and tested on three platforms: Intel MacOS, a Graviton ARM based EC2 instance, and Microsoft Windows.</p>\n<p>Features like this let you build on CodeBuild, a fully managed build service – and not have to worry about maintaining your own build servers.</p>\n"}