Orchestrate Jenkins Workloads using Dynamic Pod Autoscaling with Amazon EKS

{"value":"This blog post will demonstrate how to leverage [Jenkins](https://www.jenkins.io/) with [Amazon Elastic Kubernetes Service (EKS)](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html) by running a Jenkins Manager within an EKS pod. In doing so, we can run Jenkins workloads by allowing Amazon EKS to spawn dynamic Jenkins Agent(s) in order to perform application and infrastructure deployment. Traditionally, customers will setup a Jenkins Manager-Agent architecture that contains a set of manually added nodes with no autoscaling capabilities. Implementing this strategy will ensure that a robust approach optimizes the performance with the right-sized compute capacity and work needed to successfully perform the build tasks.\n\nIn setting up our Amazon EKS cluster with Jenkins, we’ll utilize the ```eksctl``` simple CLI tool for creating clusters on EKS. Then, we’ll build both the Jenkins Manager and Jenkins Agent image. Afterward, we’ll run a container deployment on our cluster to access the Jenkins application and utilize the dynamic Jenkins Agent pods to run pipelines and jobs.\n\n### **Solution Overview**\nThe architecture below illustrates the execution steps.\n\n\n\n*Figure 1. Solution overview diagram*\n\n***Disclaimer(s): (Note: This Jenkins application is not configured with a persistent volume storage. Therefore, you must establish and configure this template to fit that requirement).***\n\nTo accomplish this deployment workflow, we will do the following:\n\n#### **Centralized Shared Services account**\n1. Deploy the Amazon EKS Cluster into a Centralized Shared Services Account.\n2. Create the Amazon ECR Repository for the Jenkins Manager and Jenkins Agent to store docker images.\n3. Deploy the kubernetes manifest file for the Jenkins Manager.\n\n#### **Target Account(s)**\n1. Establish a set of [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html) roles with permissions for cross-across access from the Share Services account into the Target account(s).\n\n#### **Jenkins Application UI**\n1. Jenkins Plugins – Install and configure the [Kubernetes Plugin](https://plugins.jenkins.io/kubernetes/) and [CloudBees AWS Credentials Plugin](https://plugins.jenkins.io/aws-credentials/) from Manage Plugins (*you will not have to manually install this since it will be packaged and installed as part of the Jenkins image build*).\n2. Jenkins Pipeline Example—Fetch the Jenkinsfile to deploy an S3 Bucket with CloudFormation in the Target account using a Jenkins parameterized pipeline.\n\n\n### **Prerequisites**\nThe following is the minimum requirements for ensuring this solution will work.\n\n#### **Account Prerequisites**\n- Shared Services Account: The location of the Amazon EKS Cluster.\n- Target Account: The destination of the CI/CD pipeline deployments.\n\n#### **Build Requirements**\n\n- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)\n- [Docker CLI](https://docs.docker.com/get-docker/) — *Note: The docker engine must be running in order to build images.*\n- [aws-iam-authenticator](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html)\n- [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html)\n- [eksctl](https://github.com/weaveworks/eksctl)\n- Configure your [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) with the associated region.\n- Verify if the build requirements were correctly installed by checking the version.\n- [Create an EKS Cluster using eksctl](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/EKS.md).\n- Verify that EKS nodes are running and available.\n- [Create an AWS ECR Repository for the Jenkins Manager and Jenkins Agent](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/ECR.md).\n\n### **Clone the Git Repository**\n\nBash\n```\ngit clone https://github.com/aws-samples/jenkins-cloudformation-deployment-example.git\n```\n### **Security Considerations**\nThis blog provides a high-level overview of the best practices for cross-account deployment and isolation maintenance between the applications. We evaluated the cross-account application deployment permissions and will describe the current state as well as what to avoid. As part of the security best practices, we will maintain isolation among multiple apps deployed in these environments, e.g., Pipeline 1 does not deploy to the Pipeline 2 infrastructure.\n\n### **Requirement**\nA Jenkins manager is running as a container in an EC2 compute instance that resides within a Shared AWS account. This Jenkins application represents individual pipelines deploying unique microservices that build and deploy to multiple environments in separate AWS accounts. The cross-account deployment utilizes the target AWS account admin credentials in order to do the deployment.\n\nThis methodology means that it is not good practice to share the account credentials externally. Additionally, the deployment errors risk should be eliminated and application isolation should be maintained within the same account.\n\nNote that the deployment steps are being run using AWS CLIs, thus our solution will be focused on AWS CLI usage.\n\nThe risk is much lower when utilizing CloudFormation / CDK to conduct deployments because the AWS CLIs executed from the build jobs will specify stack names as parametrized inputs and the very low probability of stack-name error. However, it remains inadvisable to utilize admin credentials of the target account.\n\n### **Best Practice — *Current Approach***\n\nWe utilized cross-account roles that can restrict unauthorized access across build jobs. Behind this approach, we will utilize the assume-role concept that will enable the requesting role to obtain temporary credentials (from the STS service) of the target role and execute actions permitted by the target role. This is safer than utilizing hard-coded credentials. The requesting role could be either the inherited EC2 instance role OR specific user credentials. However, in our case, we are utilizing the inherited EC2 instance role.\n\nFor ease of understanding, we will refer the target-role as execution-role below.\n\n\n\n*Figure 2. Current approach*\n\n- As per the security best practice of assigning minimum privileges, we must first create execution role in IAM in the target account that has deployment permissions (either via CloudFormation OR via CLI’s), e.g., app-dev-role in Dev account and app-prod-role in Prod account.\n- For each of those roles, we configure a trust relationship with the parent account ID (Shared Services account). This enables any roles in the Shared Services account (with assume-role permission) to assume the execution-role and deploy it on respective hosting infrastructure, e.g., the app-dev-role in Dev account will be a common execution role that will deploy various apps across infrastructure.\n- Then, we create a local role in the Shared Services account and configure credentials within Jenkins to be utilized by the Build Jobs. Provide the job with the assume-role permissions and specify the list of ARNs across every account. Alternatively, the inherited EC2 instance role can also be utilized to assume the execution-role.\n#### \n**Create Cross-Account IAM Roles**\n\n[Cross-account IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html#tutorial_cross-account-with-roles-prereqs) allow users to securely access AWS resources in a target account while maintaining the observability of that AWS account. The cross-account IAM role includes a trust policy allowing AWS identities in another AWS account to assume the given role. This allows us to create a role in one AWS account that delegates specific permissions to another AWS account.\n\n- Create an IAM role with a common name in each target account. The role name we’ve created is \n```AWSCloudFormationStackExecutionRole```. The role must have permissions to perform CloudFormation actions and any actions regarding the resources that will be created. In our case, we will be creating an S3 Bucket utilizing CloudFormation.\n- This IAM role must also have an established trust relationship to the Shared Services account. In this case, the Jenkins Agent will be granted the ability to assume the role of the particular target account from the Shared Services account.\n- In our case, the IAM entity that will assume the ```AWSCloudFormationStackExecutionRole``` is the EKS Node Instance Role that associated with the EKS Cluster Nodes.\n\nJSON\n```\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"cloudformation:CreateUploadBucket\",\n \"cloudformation:ListStacks\",\n \"cloudformation:CancelUpdateStack\",\n \"cloudformation:ExecuteChangeSet\",\n \"cloudformation:ListChangeSets\",\n \"cloudformation:ListStackResources\",\n \"cloudformation:DescribeStackResources\",\n \"cloudformation:DescribeStackResource\",\n \"cloudformation:CreateChangeSet\",\n \"cloudformation:DeleteChangeSet\",\n \"cloudformation:DescribeStacks\",\n \"cloudformation:ContinueUpdateRollback\",\n \"cloudformation:DescribeStackEvents\",\n \"cloudformation:CreateStack\",\n \"cloudformation:DeleteStack\",\n \"cloudformation:UpdateStack\",\n \"cloudformation:DescribeChangeSet\",\n \"s3:PutBucketPublicAccessBlock\",\n \"s3:CreateBucket\",\n \"s3:DeleteBucketPolicy\",\n \"s3:PutEncryptionConfiguration\",\n \"s3:PutBucketPolicy\",\n \"s3:DeleteBucket\"\n ],\n \"Resource\": \"*\"\n }\n ]\n}\n```\n### **Build Docker Images**\nBuild the custom docker images for the Jenkins Manager and the Jenkins Agent, and then push the images to AWS ECR Repository. Navigate to the ```docker/``` directory, then execute the command according to the required parameters with the AWS account ID, repository name, region, and the build folder name ```jenkins-manager/``` or ```jenkins-agent/ ```that resides in the current docker directory. The custom docker images will contain a set of starter package installations.\n\n- [Build and push the Jenkins Manager and Jenkins Agent docker images to the AWS ECR Repository](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/BUILDIMAGE.md)\n\n### **Deploy Jenkins Application**\nAfter building both images, navigate to the ```k8s/ ```directory, modify the manifest file for the Jenkins image, and then execute the Jenkins [manifest.yaml](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/k8s/manifest.yaml) template to setup the Jenkins application. (Note: This Jenkins application is not configured with a persistent volume storage. Therefore, you will need to establish and configure this template to fit that requirement).\n\n- [Deploy the Jenkins Application to the EKS Cluster](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/MANAGER.md)\n\nBash\n```\n# Fetch the Application URL or navigate to the AWS Console for the Load Balancer\nkubectl get svc -n jenkins\n\n# Verify that jenkins deployment/pods are up running\nkubectl get pods -n jenkins\n\n# Replace with jenkins manager pod name and fetch Jenkins login password\nkubectl exec -it pod/<JENKINS-MANAGER-POD-NAME> -n jenkins -- cat /var/jenkins_home/secrets/initialAdminPassword\n```\n- The [Kubernetes Plugin](https://plugins.jenkins.io/kubernetes/) and[CloudBees AWS Credentials Plugin](https://plugins.jenkins.io/aws-credentials/) should be installed as part of the Jenkins image build from the Managed Plugins.\n- Navigate: Manage Jenkins → Configure Global Security\n- Set the Crumb Issuer to remove the error pages in order to prevent Cross Site Request Forgery exploits.\n\n\n\n*Figure 3. Configure Global Security*\n\n### **Configure Jenkins Kubernetes Cloud**\n- Navigate: Manage Jenkins → Manage Nodes and Clouds → Configure Clouds\n- Click: Add a new cloud → select Kubernetes from the drop menus\n\n\n\n*Figure 4a. Jenkins Configure Nodes and Clouds*\n\n***Note: Before proceeding, please ensure that you can access your Amazon EKS cluster information, whether it is through Console or CLI.***\n\n- Enter a Name in the Kubernetes Cloud configuration field.\n- Enter the Kubernetes URL which can be found via AWS Console by navigating to the Amazon EKS service and locating the API server endpoint of the cluster, or run the ```command kubectl cluster-info```.\n- Enter the namespace that will be utilized in the Kubernetes Namespace field. This will determine where the dynamic kubernetes pods will spawn. In our case, the name of the namespace is ```jenkins```.\n- During the initial setup of Jenkins Manager on kubernetes, there is an environment variable ```JENKINS_URL``` that automatically utilizes the Load Balancer URL to resolve requests. However, we will resolve our requests locally to the cluster IP address.\n\n\t The format is as follows: \n```https://<service-name>.<namespace>.svc.cluster.local```\n\n\n\n\n*Figure 4b. Configure Kubernetes Cloud*\n\n### **Set AWS Credentials**\nSecurity concerns are a key reason why we’re utilizing an IAM role instead of access keys. For any given approach involving IAM, it is the best practice to utilize temporary credentials.\n\n- You must have the AWS Credentials Binding Plugin installed before this step. Enter the unique ID name as shown in the example below.\n- Enter the IAM Role ARN you created earlier for both the ID and IAM Role to use in the field as shown below.\n\n\n\n*Figure 5. AWS Credentials Binding*\n\n\n\n*Figure 6. Managed Credentials*\n\n### **Create a pipeline**\n- Navigate to the Jenkins main menu and select new item\n- Create a Pipeline\n\n\n\n*Figure 7. Create a pipeline*\n\n### **Configure Jenkins Agent**\nSetup a Kubernetes YAML template after you’ve built the agent image. In this example, we will be using the [k8sPodTemplate.yaml](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/k8s/k8sPodTemplate.yaml) file stored in the ```k8s/``` folder.\n\n- [https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/k8s/k8sPodTemplate.yaml](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/AGENT.md)\n\n### **CloudFormation Execution Scripts**\nThis [deploy-stack.sh](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/scripts/deploy-stack.sh) file can accept four different parameters and conduct several types of CloudFormation stack executions such as deploy, create-changeset, and execute-changeset. This is also reflected in the stages of this [Jenkinsfile](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/Jenkinsfile) pipeline. As for the [delete-stack.sh](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/scripts/delete-stack.sh) file, two parameters are accepted, and, when executed, it will delete a CloudFormation stack based on the given stack name and region.\n\n- [Understanding Pipeline Execution Scripts for CloudFormation](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/JENKINS.md)\n\n### **Jenkinsfile**\nIn this [Jenkinsfile](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/Jenkinsfile), the individual pipeline build jobs will deploy individual microservices. The [k8sPodTemplate.yaml](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/k8s/k8sPodTemplate.yaml) is utilized to specify the kubernetes pod details and the inbound-agent that will be utilized to run the pipeline.\n\n- [Jenkinsfile Parameterized Pipeline Configurations](https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/JENKINS.md)\n\n### **Jenkins Pipeline: Execute a pipeline**\n- Click Build with Parameters and then select a build action.\n\n\n\n*Figure 8a. Build with Parameters*\n\n- Examine the pipeline stages even further for the choice you selected. Also, view more details of the stages below and verify in your AWS account that the CloudFormation stack was executed.\n\n\n\n\n*Figure 8b. Pipeline Stage View*\n\n- The Final Step is to execute your pipeline and watch the pods spin up dynamically in your terminal. As is shown below, the Jenkins agent pod spawned and then terminated after the work completed. Watch this task on your own by executing the following command:\n\nBash\n```\n# Watch the pods spawn in the \"jenkins\" namespace\nkubectl get pods -n jenkins -w\n```\n\n\n\n*Figure 9. Watch Jenkins Agent Pods Spawn*\n\n### **Code Repository**\n- [Amazon EKS Jenkins Integration](https://github.com/aws-samples/jenkins-cloudformation-deployment-example.git)\n### **References**\n- [Jenkins Kubernetes Plugin](https://plugins.jenkins.io/kubernetes/)\n- [CloudBees AWS Credentials Plugin](https://plugins.jenkins.io/aws-credentials/)\n### **Cleanup**\n\nIn order to avoid incurring future charges, delete the resources utilized in the walkthrough.\n\n- Delete the EKS cluster. You can utilize the ```eksctl``` to delete the cluster.\n- Delete any remaining AWS resources created by EKS such as AWS LoadBalancer, Target Groups, etc.\n- Delete any related IAM entities.\n\n### **Conclusion**\nThis post walked you through the process of building out Amazon EKS based infrastructure and integrating Jenkins to orchestrate workloads. We demonstrated how you can utilize this to deploy securely across multiple accounts with dynamic Jenkins agents and create alignment to your business with similar use cases. To learn more about Amazon EKS, see our [documentation](https://aws.amazon.com/eks/getting-started/) pages or explore our [console](https://console.aws.amazon.com/eks/home?region=us-east-1).\n\n#### **About the Authors**\n\n\n\n**Vladimir P. Toussaint**\nVladimir is a DevOps Cloud Architect at Amazon Web Services. He works with GovCloud customers to build solutions and capabilities as they move to the cloud. Previous to Amazon Web Services, Vladimir has leveraged container orchestration tools such as Kubernetes to securely manage microservice applications for large enterprises.\n\n\n\n**Matt Noyce**\nMatt is a Sr. Cloud Application Architect at Amazon Web Services. He works primarily with health care and life sciences customers to help them architect and build applications, data lakes, and DevOps pipelines that solve their business needs. In his spare time Matt likes to run and hike along with enjoying time with friends and family.\n\n\n\n**Nikunj Vaidya**\nNikunj is a DevOps Tech Leader at Amazon Web Services. He offers technical guidance to the customers on AWS DevOps solutions and services that would streamline the application development process, accelerate application delivery, and enable maintaining a high bar of software quality. Prior to AWS, Nikunj has worked in software engineering roles, leading transformation projects, driving releases and improvements in the software quality and customer experience.\n","render":"<p>This blog post will demonstrate how to leverage <a href=\"https://www.jenkins.io/\" target=\"_blank\">Jenkins</a> with <a href=\"https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html\" target=\"_blank\">Amazon Elastic Kubernetes Service (EKS)</a> by running a Jenkins Manager within an EKS pod. In doing so, we can run Jenkins workloads by allowing Amazon EKS to spawn dynamic Jenkins Agent(s) in order to perform application and infrastructure deployment. Traditionally, customers will setup a Jenkins Manager-Agent architecture that contains a set of manually added nodes with no autoscaling capabilities. Implementing this strategy will ensure that a robust approach optimizes the performance with the right-sized compute capacity and work needed to successfully perform the build tasks.</p>\n<p>In setting up our Amazon EKS cluster with Jenkins, we’ll utilize the <code>eksctl</code> simple CLI tool for creating clusters on EKS. Then, we’ll build both the Jenkins Manager and Jenkins Agent image. Afterward, we’ll run a container deployment on our cluster to access the Jenkins application and utilize the dynamic Jenkins Agent pods to run pipelines and jobs.</p>\n<h3><a id=\"Solution_Overview_4\"></a><strong>Solution Overview</strong></h3>\n<p>The architecture below illustrates the execution steps.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/ff7655bb8db842878908e82f66bbc38b_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 1. Solution overview diagram</em></p>\n<p><em><strong>Disclaimer(s): (Note: This Jenkins application is not configured with a persistent volume storage. Therefore, you must establish and configure this template to fit that requirement).</strong></em></p>\n<p>To accomplish this deployment workflow, we will do the following:</p>\n<h4><a id=\"Centralized_Shared_Services_account_15\"></a><strong>Centralized Shared Services account</strong></h4>\n<ol>\n<li>Deploy the Amazon EKS Cluster into a Centralized Shared Services Account.</li>\n<li>Create the Amazon ECR Repository for the Jenkins Manager and Jenkins Agent to store docker images.</li>\n<li>Deploy the kubernetes manifest file for the Jenkins Manager.</li>\n</ol>\n<h4><a id=\"Target_Accounts_20\"></a><strong>Target Account(s)</strong></h4>\n<ol>\n<li>Establish a set of <a href=\"https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html\" target=\"_blank\">AWS Identity and Access Management (IAM)</a> roles with permissions for cross-across access from the Share Services account into the Target account(s).</li>\n</ol>\n<h4><a id=\"Jenkins_Application_UI_23\"></a><strong>Jenkins Application UI</strong></h4>\n<ol>\n<li>Jenkins Plugins – Install and configure the <a href=\"https://plugins.jenkins.io/kubernetes/\" target=\"_blank\">Kubernetes Plugin</a> and <a href=\"https://plugins.jenkins.io/aws-credentials/\" target=\"_blank\">CloudBees AWS Credentials Plugin</a> from Manage Plugins (<em>you will not have to manually install this since it will be packaged and installed as part of the Jenkins image build</em>).</li>\n<li>Jenkins Pipeline Example—Fetch the Jenkinsfile to deploy an S3 Bucket with CloudFormation in the Target account using a Jenkins parameterized pipeline.</li>\n</ol>\n<h3><a id=\"Prerequisites_28\"></a><strong>Prerequisites</strong></h3>\n<p>The following is the minimum requirements for ensuring this solution will work.</p>\n<h4><a id=\"Account_Prerequisites_31\"></a><strong>Account Prerequisites</strong></h4>\n<ul>\n<li>Shared Services Account: The location of the Amazon EKS Cluster.</li>\n<li>Target Account: The destination of the CI/CD pipeline deployments.</li>\n</ul>\n<h4><a id=\"Build_Requirements_35\"></a><strong>Build Requirements</strong></h4>\n<ul>\n<li><a href=\"https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html\" target=\"_blank\">AWS CLI</a></li>\n<li><a href=\"https://docs.docker.com/get-docker/\" target=\"_blank\">Docker CLI</a> — <em>Note: The docker engine must be running in order to build images.</em></li>\n<li><a href=\"https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html\" target=\"_blank\">aws-iam-authenticator</a></li>\n<li><a href=\"https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html\" target=\"_blank\">kubectl</a></li>\n<li><a href=\"https://github.com/weaveworks/eksctl\" target=\"_blank\">eksctl</a></li>\n<li>Configure your <a href=\"https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html\" target=\"_blank\">AWS credentials</a> with the associated region.</li>\n<li>Verify if the build requirements were correctly installed by checking the version.</li>\n<li><a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/EKS.md\" target=\"_blank\">Create an EKS Cluster using eksctl</a>.</li>\n<li>Verify that EKS nodes are running and available.</li>\n<li><a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/ECR.md\" target=\"_blank\">Create an AWS ECR Repository for the Jenkins Manager and Jenkins Agent</a>.</li>\n</ul>\n<h3><a id=\"Clone_the_Git_Repository_48\"></a><strong>Clone the Git Repository</strong></h3>\n<p>Bash</p>\n<pre><code class=\"lang-\">git clone https://github.com/aws-samples/jenkins-cloudformation-deployment-example.git\n</code></pre>\n<h3><a id=\"Security_Considerations_54\"></a><strong>Security Considerations</strong></h3>\n<p>This blog provides a high-level overview of the best practices for cross-account deployment and isolation maintenance between the applications. We evaluated the cross-account application deployment permissions and will describe the current state as well as what to avoid. As part of the security best practices, we will maintain isolation among multiple apps deployed in these environments, e.g., Pipeline 1 does not deploy to the Pipeline 2 infrastructure.</p>\n<h3><a id=\"Requirement_57\"></a><strong>Requirement</strong></h3>\n<p>A Jenkins manager is running as a container in an EC2 compute instance that resides within a Shared AWS account. This Jenkins application represents individual pipelines deploying unique microservices that build and deploy to multiple environments in separate AWS accounts. The cross-account deployment utilizes the target AWS account admin credentials in order to do the deployment.</p>\n<p>This methodology means that it is not good practice to share the account credentials externally. Additionally, the deployment errors risk should be eliminated and application isolation should be maintained within the same account.</p>\n<p>Note that the deployment steps are being run using AWS CLIs, thus our solution will be focused on AWS CLI usage.</p>\n<p>The risk is much lower when utilizing CloudFormation / CDK to conduct deployments because the AWS CLIs executed from the build jobs will specify stack names as parametrized inputs and the very low probability of stack-name error. However, it remains inadvisable to utilize admin credentials of the target account.</p>\n<h3><a id=\"Best_Practice__Current_Approach_66\"></a><strong>Best Practice — <em>Current Approach</em></strong></h3>\n<p>We utilized cross-account roles that can restrict unauthorized access across build jobs. Behind this approach, we will utilize the assume-role concept that will enable the requesting role to obtain temporary credentials (from the STS service) of the target role and execute actions permitted by the target role. This is safer than utilizing hard-coded credentials. The requesting role could be either the inherited EC2 instance role OR specific user credentials. However, in our case, we are utilizing the inherited EC2 instance role.</p>\n<p>For ease of understanding, we will refer the target-role as execution-role below.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/118f3ccc2725430e86cf60c7e6423c12_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 2. Current approach</em></p>\n<ul>\n<li>As per the security best practice of assigning minimum privileges, we must first create execution role in IAM in the target account that has deployment permissions (either via CloudFormation OR via CLI’s), e.g., app-dev-role in Dev account and app-prod-role in Prod account.</li>\n<li>For each of those roles, we configure a trust relationship with the parent account ID (Shared Services account). This enables any roles in the Shared Services account (with assume-role permission) to assume the execution-role and deploy it on respective hosting infrastructure, e.g., the app-dev-role in Dev account will be a common execution role that will deploy various apps across infrastructure.</li>\n<li>Then, we create a local role in the Shared Services account and configure credentials within Jenkins to be utilized by the Build Jobs. Provide the job with the assume-role permissions and specify the list of ARNs across every account. Alternatively, the inherited EC2 instance role can also be utilized to assume the execution-role.</li>\n</ul>\n<h4><a id=\"_79\"></a></h4>\n<p><strong>Create Cross-Account IAM Roles</strong></p>\n<p><a href=\"https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html#tutorial_cross-account-with-roles-prereqs\" target=\"_blank\">Cross-account IAM roles</a> allow users to securely access AWS resources in a target account while maintaining the observability of that AWS account. The cross-account IAM role includes a trust policy allowing AWS identities in another AWS account to assume the given role. This allows us to create a role in one AWS account that delegates specific permissions to another AWS account.</p>\n<ul>\n<li>Create an IAM role with a common name in each target account. The role name we’ve created is<br />\n<code>AWSCloudFormationStackExecutionRole</code>. The role must have permissions to perform CloudFormation actions and any actions regarding the resources that will be created. In our case, we will be creating an S3 Bucket utilizing CloudFormation.</li>\n<li>This IAM role must also have an established trust relationship to the Shared Services account. In this case, the Jenkins Agent will be granted the ability to assume the role of the particular target account from the Shared Services account.</li>\n<li>In our case, the IAM entity that will assume the <code>AWSCloudFormationStackExecutionRole</code> is the EKS Node Instance Role that associated with the EKS Cluster Nodes.</li>\n</ul>\n<p>JSON</p>\n<pre><code class=\"lang-\">{\n &quot;Version&quot;: &quot;2012-10-17&quot;,\n &quot;Statement&quot;: [\n {\n &quot;Effect&quot;: &quot;Allow&quot;,\n &quot;Action&quot;: [\n &quot;cloudformation:CreateUploadBucket&quot;,\n &quot;cloudformation:ListStacks&quot;,\n &quot;cloudformation:CancelUpdateStack&quot;,\n &quot;cloudformation:ExecuteChangeSet&quot;,\n &quot;cloudformation:ListChangeSets&quot;,\n &quot;cloudformation:ListStackResources&quot;,\n &quot;cloudformation:DescribeStackResources&quot;,\n &quot;cloudformation:DescribeStackResource&quot;,\n &quot;cloudformation:CreateChangeSet&quot;,\n &quot;cloudformation:DeleteChangeSet&quot;,\n &quot;cloudformation:DescribeStacks&quot;,\n &quot;cloudformation:ContinueUpdateRollback&quot;,\n &quot;cloudformation:DescribeStackEvents&quot;,\n &quot;cloudformation:CreateStack&quot;,\n &quot;cloudformation:DeleteStack&quot;,\n &quot;cloudformation:UpdateStack&quot;,\n &quot;cloudformation:DescribeChangeSet&quot;,\n &quot;s3:PutBucketPublicAccessBlock&quot;,\n &quot;s3:CreateBucket&quot;,\n &quot;s3:DeleteBucketPolicy&quot;,\n &quot;s3:PutEncryptionConfiguration&quot;,\n &quot;s3:PutBucketPolicy&quot;,\n &quot;s3:DeleteBucket&quot;\n ],\n &quot;Resource&quot;: &quot;*&quot;\n }\n ]\n}\n</code></pre>\n<h3><a id=\"Build_Docker_Images_126\"></a><strong>Build Docker Images</strong></h3>\n<p>Build the custom docker images for the Jenkins Manager and the Jenkins Agent, and then push the images to AWS ECR Repository. Navigate to the <code>docker/</code> directory, then execute the command according to the required parameters with the AWS account ID, repository name, region, and the build folder name <code>jenkins-manager/</code> or <code>jenkins-agent/ </code>that resides in the current docker directory. The custom docker images will contain a set of starter package installations.</p>\n<ul>\n<li><a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/BUILDIMAGE.md\" target=\"_blank\">Build and push the Jenkins Manager and Jenkins Agent docker images to the AWS ECR Repository</a></li>\n</ul>\n<h3><a id=\"Deploy_Jenkins_Application_131\"></a><strong>Deploy Jenkins Application</strong></h3>\n<p>After building both images, navigate to the <code>k8s/ </code>directory, modify the manifest file for the Jenkins image, and then execute the Jenkins <a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/k8s/manifest.yaml\" target=\"_blank\">manifest.yaml</a> template to setup the Jenkins application. (Note: This Jenkins application is not configured with a persistent volume storage. Therefore, you will need to establish and configure this template to fit that requirement).</p>\n<ul>\n<li><a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/MANAGER.md\" target=\"_blank\">Deploy the Jenkins Application to the EKS Cluster</a></li>\n</ul>\n<p>Bash</p>\n<pre><code class=\"lang-\"># Fetch the Application URL or navigate to the AWS Console for the Load Balancer\nkubectl get svc -n jenkins\n\n# Verify that jenkins deployment/pods are up running\nkubectl get pods -n jenkins\n\n# Replace with jenkins manager pod name and fetch Jenkins login password\nkubectl exec -it pod/&lt;JENKINS-MANAGER-POD-NAME&gt; -n jenkins -- cat /var/jenkins_home/secrets/initialAdminPassword\n</code></pre>\n<ul>\n<li>The <a href=\"https://plugins.jenkins.io/kubernetes/\" target=\"_blank\">Kubernetes Plugin</a> and<a href=\"https://plugins.jenkins.io/aws-credentials/\" target=\"_blank\">CloudBees AWS Credentials Plugin</a> should be installed as part of the Jenkins image build from the Managed Plugins.</li>\n<li>Navigate: Manage Jenkins → Configure Global Security</li>\n<li>Set the Crumb Issuer to remove the error pages in order to prevent Cross Site Request Forgery exploits.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/fce2ccda16b548cb81b0e4da1786855f_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 3. Configure Global Security</em></p>\n<h3><a id=\"Configure_Jenkins_Kubernetes_Cloud_155\"></a><strong>Configure Jenkins Kubernetes Cloud</strong></h3>\n<ul>\n<li>Navigate: Manage Jenkins → Manage Nodes and Clouds → Configure Clouds</li>\n<li>Click: Add a new cloud → select Kubernetes from the drop menus</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/c825913270634e39afef414ec6d0a328_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 4a. Jenkins Configure Nodes and Clouds</em></p>\n<p><em><strong>Note: Before proceeding, please ensure that you can access your Amazon EKS cluster information, whether it is through Console or CLI.</strong></em></p>\n<ul>\n<li>\n<p>Enter a Name in the Kubernetes Cloud configuration field.</p>\n</li>\n<li>\n<p>Enter the Kubernetes URL which can be found via AWS Console by navigating to the Amazon EKS service and locating the API server endpoint of the cluster, or run the <code>command kubectl cluster-info</code>.</p>\n</li>\n<li>\n<p>Enter the namespace that will be utilized in the Kubernetes Namespace field. This will determine where the dynamic kubernetes pods will spawn. In our case, the name of the namespace is <code>jenkins</code>.</p>\n</li>\n<li>\n<p>During the initial setup of Jenkins Manager on kubernetes, there is an environment variable <code>JENKINS_URL</code> that automatically utilizes the Load Balancer URL to resolve requests. However, we will resolve our requests locally to the cluster IP address.</p>\n<p>The format is as follows:<br />\n<code>https://&lt;service-name&gt;.&lt;namespace&gt;.svc.cluster.local</code></p>\n</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/4a2d39aafd8c4cfab1d2c728464adb59_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 4b. Configure Kubernetes Cloud</em></p>\n<h3><a id=\"Set_AWS_Credentials_178\"></a><strong>Set AWS Credentials</strong></h3>\n<p>Security concerns are a key reason why we’re utilizing an IAM role instead of access keys. For any given approach involving IAM, it is the best practice to utilize temporary credentials.</p>\n<ul>\n<li>You must have the AWS Credentials Binding Plugin installed before this step. Enter the unique ID name as shown in the example below.</li>\n<li>Enter the IAM Role ARN you created earlier for both the ID and IAM Role to use in the field as shown below.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/76d7a30f705f495ca75ac611498c3725_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 5. AWS Credentials Binding</em></p>\n<p><img src=\"https://dev-media.amazoncloud.cn/3df35cd47d6347069e6db88f25fd7dc1_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 6. Managed Credentials</em></p>\n<h3><a id=\"Create_a_pipeline_192\"></a><strong>Create a pipeline</strong></h3>\n<ul>\n<li>Navigate to the Jenkins main menu and select new item</li>\n<li>Create a Pipeline</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/4f1f275edbe44fb6875ee7fec958c412_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 7. Create a pipeline</em></p>\n<h3><a id=\"Configure_Jenkins_Agent_200\"></a><strong>Configure Jenkins Agent</strong></h3>\n<p>Setup a Kubernetes YAML template after you’ve built the agent image. In this example, we will be using the <a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/k8s/k8sPodTemplate.yaml\" target=\"_blank\">k8sPodTemplate.yaml</a> file stored in the <code>k8s/</code> folder.</p>\n<ul>\n<li><a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/AGENT.md\" target=\"_blank\">https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/k8s/k8sPodTemplate.yaml</a></li>\n</ul>\n<h3><a id=\"CloudFormation_Execution_Scripts_205\"></a><strong>CloudFormation Execution Scripts</strong></h3>\n<p>This <a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/scripts/deploy-stack.sh\" target=\"_blank\">deploy-stack.sh</a> file can accept four different parameters and conduct several types of CloudFormation stack executions such as deploy, create-changeset, and execute-changeset. This is also reflected in the stages of this <a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/Jenkinsfile\" target=\"_blank\">Jenkinsfile</a> pipeline. As for the <a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/scripts/delete-stack.sh\" target=\"_blank\">delete-stack.sh</a> file, two parameters are accepted, and, when executed, it will delete a CloudFormation stack based on the given stack name and region.</p>\n<ul>\n<li><a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/JENKINS.md\" target=\"_blank\">Understanding Pipeline Execution Scripts for CloudFormation</a></li>\n</ul>\n<h3><a id=\"Jenkinsfile_210\"></a><strong>Jenkinsfile</strong></h3>\n<p>In this <a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/Jenkinsfile\" target=\"_blank\">Jenkinsfile</a>, the individual pipeline build jobs will deploy individual microservices. The <a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/k8s/k8sPodTemplate.yaml\" target=\"_blank\">k8sPodTemplate.yaml</a> is utilized to specify the kubernetes pod details and the inbound-agent that will be utilized to run the pipeline.</p>\n<ul>\n<li><a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example/blob/main/JENKINS.md\" target=\"_blank\">Jenkinsfile Parameterized Pipeline Configurations</a></li>\n</ul>\n<h3><a id=\"Jenkins_Pipeline_Execute_a_pipeline_215\"></a><strong>Jenkins Pipeline: Execute a pipeline</strong></h3>\n<ul>\n<li>Click Build with Parameters and then select a build action.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/5f3b3146ca664b5db8ffb05d4ea421ea_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 8a. Build with Parameters</em></p>\n<ul>\n<li>Examine the pipeline stages even further for the choice you selected. Also, view more details of the stages below and verify in your AWS account that the CloudFormation stack was executed.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/1012f5937f624fd58ed75d2e26d1f5cc_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 8b. Pipeline Stage View</em></p>\n<ul>\n<li>The Final Step is to execute your pipeline and watch the pods spin up dynamically in your terminal. As is shown below, the Jenkins agent pod spawned and then terminated after the work completed. Watch this task on your own by executing the following command:</li>\n</ul>\n<p>Bash</p>\n<pre><code class=\"lang-\"># Watch the pods spawn in the &quot;jenkins&quot; namespace\nkubectl get pods -n jenkins -w\n</code></pre>\n<p><img src=\"https://dev-media.amazoncloud.cn/ef832373c0c642fb944063741f178d9a_image.png\" alt=\"image.png\" /></p>\n<p><em>Figure 9. Watch Jenkins Agent Pods Spawn</em></p>\n<h3><a id=\"Code_Repository_241\"></a><strong>Code Repository</strong></h3>\n<ul>\n<li><a href=\"https://github.com/aws-samples/jenkins-cloudformation-deployment-example.git\" target=\"_blank\">Amazon EKS Jenkins Integration</a></li>\n</ul>\n<h3><a id=\"References_243\"></a><strong>References</strong></h3>\n<ul>\n<li><a href=\"https://plugins.jenkins.io/kubernetes/\" target=\"_blank\">Jenkins Kubernetes Plugin</a></li>\n<li><a href=\"https://plugins.jenkins.io/aws-credentials/\" target=\"_blank\">CloudBees AWS Credentials Plugin</a></li>\n</ul>\n<h3><a id=\"Cleanup_246\"></a><strong>Cleanup</strong></h3>\n<p>In order to avoid incurring future charges, delete the resources utilized in the walkthrough.</p>\n<ul>\n<li>Delete the EKS cluster. You can utilize the <code>eksctl</code> to delete the cluster.</li>\n<li>Delete any remaining AWS resources created by EKS such as AWS LoadBalancer, Target Groups, etc.</li>\n<li>Delete any related IAM entities.</li>\n</ul>\n<h3><a id=\"Conclusion_254\"></a><strong>Conclusion</strong></h3>\n<p>This post walked you through the process of building out Amazon EKS based infrastructure and integrating Jenkins to orchestrate workloads. We demonstrated how you can utilize this to deploy securely across multiple accounts with dynamic Jenkins agents and create alignment to your business with similar use cases. To learn more about Amazon EKS, see our <a href=\"https://aws.amazon.com/eks/getting-started/\" target=\"_blank\">documentation</a> pages or explore our <a href=\"https://console.aws.amazon.com/eks/home?region=us-east-1\" target=\"_blank\">console</a>.</p>\n<h4><a id=\"About_the_Authors_257\"></a><strong>About the Authors</strong></h4>\n<p><img src=\"https://dev-media.amazoncloud.cn/cbedb331fb804bf38b22b75b9df955e3_image.png\" alt=\"image.png\" /></p>\n<p><strong>Vladimir P. Toussaint</strong><br />\nVladimir is a DevOps Cloud Architect at Amazon Web Services. He works with GovCloud customers to build solutions and capabilities as they move to the cloud. Previous to Amazon Web Services, Vladimir has leveraged container orchestration tools such as Kubernetes to securely manage microservice applications for large enterprises.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/dea08cd2b08f42a99eae12888937057e_image.png\" alt=\"image.png\" /></p>\n<p><strong>Matt Noyce</strong><br />\nMatt is a Sr. Cloud Application Architect at Amazon Web Services. He works primarily with health care and life sciences customers to help them architect and build applications, data lakes, and DevOps pipelines that solve their business needs. In his spare time Matt likes to run and hike along with enjoying time with friends and family.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/0dfae5a8a7a64426bbd2e169e44c429f_image.png\" alt=\"image.png\" /></p>\n<p><strong>Nikunj Vaidya</strong><br />\nNikunj is a DevOps Tech Leader at Amazon Web Services. He offers technical guidance to the customers on AWS DevOps solutions and services that would streamline the application development process, accelerate application delivery, and enable maintaining a high bar of software quality. Prior to AWS, Nikunj has worked in software engineering roles, leading transformation projects, driving releases and improvements in the software quality and customer experience.</p>\n"}