Govern CI/CD best practices via Amazon Service Catalog

{"value":"#### **Introduction**\n\n[Amazon Web Services Service Catalog](https://aws.amazon.com/servicecatalog/) enables organizations to create and manage Information Technology (IT) services catalogs that are approved for use on Amazon Web Services. These IT services can include resources such as virtual machine images, servers, software, and databases to complete multi-tier application architectures. Amazon Web Services Service Catalog lets you centrally manage deployed IT services and your applications, resources, and metadata , which helps you achieve consistent governance and meet your compliance requirements. In addition, this configuration enables users to quickly deploy only approved IT services.\n\nIn large organizations, as more products are created, Service Catalog management can become exponentially complicated when different teams work on various products. The following solution simplifies Service Catalog products provisioning by considering elements such as shared accounts, roles, or users who can run portfolios or tags in the form of best practices via Continuous Integrations and Continuous Deployment (CI/CD) patterns.\n\nThis post demonstrates how Service Catalog Products can be delivered by taking advantage of the main benefits of CI/CD principles along with reducing complexity required to sync services. In this scenario, we have built a CI/CD Pipeline exclusively using Amazon Web Services Services and the [Amazon Web Services Cloud Development Kit (CDK)](https://aws.amazon.com/cdk/) Framework to provision the necessary Infrastructure.\n\nCustomers need the capability to consume services in a self-service manner, with services built on patterns that follow best practices, including focus areas such as compliance and security. The key tenants for these customers are: the use of infrastructure as code (IaC), and CI/CD. For these reasons, we built a scalable and automated deployment solution covered in this post.Furthermore, this post is also inspired from another post from the Amazon Web Services community, [Building a Continuous Delivery Pipeline for Amazon Web Services Service Catalog.](https://aws.amazon.com/blogs/devops/aws-service-catalog-sync-code/)\n\n#### **Solution Overview**\n\nThe solution is built using a unified Amazon Web Services CodeCommit repository with CDK v2 code, which manages and deploys the Service Catalog Product estate. The solution supports the following scenarios: 1) making Products available to accounts and 2) provisioning these Products directly into accounts. The configuration provides flexibility regarding which components must be deployed in accounts as opposed to making a collection of these components available to account owners/users who can in turn build upon and provision them via sharing.\n\n\n\nThe pipeline created is comprised of the following stages:\n\n1. Retrieving the code from the repository\n2. Synthesize the CDK code to transform it into a CloudFormation template\n3. Ensure the pipeline is defined correctly\n4. Deploy and/or share the defined Portfolios and Products to a hub account or multiple accounts\n\n#### **Deploying and using the solution**\n\n##### **Deploy the pipeline**\n\nWe have created a Python [Amazon Web Services Cloud Development Kit (Amazon Web Services CDK)](http://aws.amazon.com/cdk) v1 application hosted in a [Git Repository](https://github.com/aws-samples/aws-cdk-service-catalog-pipeline). Deploying this application will create the required components described in this post. For a list of the deployment prerequisites, see the project [README](https://github.com/aws-samples/aws-cdk-service-catalog-pipeline/blob/main/README.md).\n\nClone the repository to your local machine. Then, bootstrap and deploy the CDK stack following the next steps.\n\n```\\ngit clone https://github.com/aws-samples/aws-cdk-service-catalog-pipeline\\ncd aws-cdk-service-catalog-pipeline\\npip install -r requirements.txt\\ncdk bootstrap aws://account_id/eu-west-1\\ncdk deploy\\n```\n\nThe infrastructure creation takes around 3-5 minutes to complete deploying the [Amazon Web Services CodePipelines](https://aws.amazon.com/codepipeline/) and repository creation. Once CDK has deployed the components, you will have a new empty repository where we will define the target Service Catalog estate. To do so, clone the new repository and push our sample code into it:\n\n```\\ncd ..\\ngit clone https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/service-catalog-repo\\ncd service-catalog-repo\\ngit checkout -b main\\ncp -aR ../aws-cdk-service-catalog-pipeline/* .\\ngit add .\\ngit commit -am \\"First commit\\"\\ngit push --set-upstream origin main\\n```\n\n##### **Review and update configuration**\n\nOur [cdk.json](https://github.com/aws-samples/aws-cdk-service-catalog-pipeline/blob/main/cdk.json) file is used to manage [context settings](https://docs.aws.amazon.com/cdk/v1/guide/context.html) such as shared accounts, permissions, region to deploy, etc.\n\n```\\nshared_accounts_ecs: Amazon Web Services account IDs where the ECS portfolio will be shared\\nshared_accounts_storage: Amazon Web Services account IDs where the Storage portfolio will be shared\\nroles: ARN for the roles who will have permissions to access to the Portfolio\\nusers: ARN for the users who will have permissions to access to the Portfolio\\ngroups: ARN for the groups who will have permissions to access to the Portfolio\\nhub_account: Amazon Web Services account ID where the Portfolio will be created\\npipeline_account: AAmazon Web ServicesWS account ID where the main Infrastructure Pipeline will be created\\nregion: the Amazon Web Services region to be used for the deployment of the account\\n```\n\n```\\n\\"shared_accounts_ecs\\":[\\"012345678901\\",\\"012345678902\\"],\\n \\"shared_accounts_storage\\":[\\"012345678901\\",\\"012345678902\\"],\\n \\"roles\\":[],\\n \\"users\\":[],\\n \\"groups\\":[],\\n \\"hub_account\\":\\"012345678901\\",\\n \\"pipeline_account\\":\\"012345678901\\",\\n \\"region\\":\\"eu-west-1\\"\\n```\n\nThere are two mechanisms that can be used to create Service Catalog Products in this solution: 1) providing a CloudFormation template or 2) declaring a CDK stack (that will be transformed as part of the pipeline). Our sample contains two Products, each demonstrating one of these options: an [Amazon Elastic Container Services (ECS)](https://aws.amazon.com/ecs/) deployment and an [Amazon Simple Storage Service (S3)](https://aws.amazon.com/s3/) product.\n\nThese Products are automatically shared with accounts specified in the shared_accounts_storage variable. Each product is managed by a CDK Python file in the cdk_service_catalog folder.\n\n\n\n\n\n\n\nThe Pipeline stages that Amazon Web Services CodePipeline runs through are as follows:\n\n1. Download the Amazon Web Services CodeCommit code\n2. Synthesize the CDK code to transform it into a CloudFormation template\n3. Auto-modify the Pipeline in case you have made manual changes to it\n4. Display the different Portfolios and Products associated in a Hub account in a Region or in multiple accounts\n\n#### **Adding new Portfolios and Products**\n\nTo add a new Portfolio to the Pipeline, we recommend creating a new class under [cdk_service_catalog](https://github.com/aws-samples/aws-cdk-service-catalog-pipeline/tree/main/cdk_service_catalog) similar to [cdk_service_catalog_ecs_stack.py from our sample](https://github.com/aws-samples/aws-cdk-service-catalog-pipeline/blob/main/cdk_service_catalog/cdk_service_catalog_ecs_stack.py). Once the new class is created with the products you wish to associate, we instantiate the new class inside [cdk_pipelines.py](https://github.com/aws-samples/aws-cdk-service-catalog-pipeline/blob/main/cdk_pipelines/cdk_pipelines.py), and then add it inside the wave in the stage. There are two ways to create portfolio products. The first one is by creating a CloudFormation template, as can be seen in the [Amazon Elastic Container Service (ECS)](https://aws.amazon.com/ecs/) example. The second way is by creating a CDK stack that will be transformed into a template, as can be seen in the Storage example.\n\n##### **Product and Portfolio definition:**\n\n```\\nclass ECSCluster(servicecatalog.ProductStack):\\n def __init__(self, scope, id):\\n super().__init__(scope, id)\\n # Parameters for the Product Template\\n cluster_name = cdk.CfnParameter(self, \\"clusterName\\", type=\\"String\\", description=\\"The name of the ECS cluster\\")\\n container_insights_enable = cdk.CfnParameter(self, \\"container_insights\\", type=\\"String\\",default=\\"False\\",allowed_values=[\\"False\\",\\"True\\"],description=\\"Enable Container Insights\\")\\n vpc = cdk.CfnParameter(self, \\"vpc\\", type=\\"AWS::EC2::VPC::Id\\", description=\\"VPC\\")\\n ecs.Cluster(self,\\"ECSCluster_template\\", enable_fargate_capacity_providers=True,cluster_name=cluster_name.value_as_string,container_insights=bool(container_insights_enable.value_as_string),vpc=vpc)\\n cdk.Tags.of(self).add(\\"key\\", \\"value\\")\\n```\n\n#### **Clean up**\n\nThe following will help you clean up all necessary parts of this post: After completing your demo, feel free to delete your stack using the CDK CLI:\n\n```\\ncdk destroy --all\\n```\n\n#### **Conclusion**\n\nIn this post, we demonstrated how Service Catalog deployments can be accelerated by building a CI/CD pipeline using self-managed services. The Portfolio & Product estate is defined in its entirety by using Infrastructure-as-Code and automatically deployed based on your configuration. To learn more about Amazon Web Services [CDK Pipelines](https://docs.aws.amazon.com/cdk/api/v1/) or [Amazon Web Services Service Catalog](https://docs.aws.amazon.com/servicecatalog/), visit the appropriate product documentation.\n\n##### **Authors:**\n\n\n\n##### **César Prieto Ballester**\nCésar Prieto Ballester is a Senior DevOps Consultant at Amazon Web Services. He enjoys automating everything and building infrastructure using code. Apart from work, he plays electric guitar and loves riding his mountain bike.\n\n\n\n##### **Daniel Mutale**\nDaniel Mutale is a Cloud Infrastructure Architect at Amazon Web Services Professional Services. He enjoys creating cloud based architectures and building out the underlying infrastructure to support the architectures using code. Apart from work, he is an avid animal photographer and has a passion for interior design.\n\n\n\n##### **Raphael Sack**\nRaphael is a technical business development manager for Service Catalog & Control Tower. He enjoys tinkering with automation and code and active member of the management tools community.\n","render":"<h4><a id=\\"Introduction_0\\"></a><strong>Introduction</strong></h4>\\n<p><a href=\\"https://aws.amazon.com/servicecatalog/\\" target=\\"_blank\\">Amazon Web Services Service Catalog</a> enables organizations to create and manage Information Technology (IT) services catalogs that are approved for use on Amazon Web Services. These IT services can include resources such as virtual machine images, servers, software, and databases to complete multi-tier application architectures. Amazon Web Services Service Catalog lets you centrally manage deployed IT services and your applications, resources, and metadata , which helps you achieve consistent governance and meet your compliance requirements. In addition, this configuration enables users to quickly deploy only approved IT services.</p>\\n<p>In large organizations, as more products are created, Service Catalog management can become exponentially complicated when different teams work on various products. The following solution simplifies Service Catalog products provisioning by considering elements such as shared accounts, roles, or users who can run portfolios or tags in the form of best practices via Continuous Integrations and Continuous Deployment (CI/CD) patterns.</p>\n<p>This post demonstrates how Service Catalog Products can be delivered by taking advantage of the main benefits of CI/CD principles along with reducing complexity required to sync services. In this scenario, we have built a CI/CD Pipeline exclusively using Amazon Web Services Services and the <a href=\\"https://aws.amazon.com/cdk/\\" target=\\"_blank\\">Amazon Web Services Cloud Development Kit (CDK)</a> Framework to provision the necessary Infrastructure.</p>\\n<p>Customers need the capability to consume services in a self-service manner, with services built on patterns that follow best practices, including focus areas such as compliance and security. The key tenants for these customers are: the use of infrastructure as code (IaC), and CI/CD. For these reasons, we built a scalable and automated deployment solution covered in this post.Furthermore, this post is also inspired from another post from the Amazon Web Services community, <a href=\\"https://aws.amazon.com/blogs/devops/aws-service-catalog-sync-code/\\" target=\\"_blank\\">Building a Continuous Delivery Pipeline for Amazon Web Services Service Catalog.</a></p>\\n<h4><a id=\\"Solution_Overview_10\\"></a><strong>Solution Overview</strong></h4>\\n<p>The solution is built using a unified Amazon Web Services CodeCommit repository with CDK v2 code, which manages and deploys the Service Catalog Product estate. The solution supports the following scenarios: 1) making Products available to accounts and 2) provisioning these Products directly into accounts. The configuration provides flexibility regarding which components must be deployed in accounts as opposed to making a collection of these components available to account owners/users who can in turn build upon and provision them via sharing.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/4e99bca6f740459e80e876a219c2a4a3_image.png\\" alt=\\"image.png\\" /></p>\n<p>The pipeline created is comprised of the following stages:</p>\n<ol>\\n<li>Retrieving the code from the repository</li>\n<li>Synthesize the CDK code to transform it into a CloudFormation template</li>\n<li>Ensure the pipeline is defined correctly</li>\n<li>Deploy and/or share the defined Portfolios and Products to a hub account or multiple accounts</li>\n</ol>\\n<h4><a id=\\"Deploying_and_using_the_solution_23\\"></a><strong>Deploying and using the solution</strong></h4>\\n<h5><a id=\\"Deploy_the_pipeline_25\\"></a><strong>Deploy the pipeline</strong></h5>\\n<p>We have created a Python <a href=\\"http://aws.amazon.com/cdk\\" target=\\"_blank\\">Amazon Web Services Cloud Development Kit (Amazon Web Services CDK)</a> v1 application hosted in a <a href=\\"https://github.com/aws-samples/aws-cdk-service-catalog-pipeline\\" target=\\"_blank\\">Git Repository</a>. Deploying this application will create the required components described in this post. For a list of the deployment prerequisites, see the project <a href=\\"https://github.com/aws-samples/aws-cdk-service-catalog-pipeline/blob/main/README.md\\" target=\\"_blank\\">README</a>.</p>\\n<p>Clone the repository to your local machine. Then, bootstrap and deploy the CDK stack following the next steps.</p>\n<pre><code class=\\"lang-\\">git clone https://github.com/aws-samples/aws-cdk-service-catalog-pipeline\\ncd aws-cdk-service-catalog-pipeline\\npip install -r requirements.txt\\ncdk bootstrap aws://account_id/eu-west-1\\ncdk deploy\\n</code></pre>\\n<p>The infrastructure creation takes around 3-5 minutes to complete deploying the <a href=\\"https://aws.amazon.com/codepipeline/\\" target=\\"_blank\\">Amazon Web Services CodePipelines</a> and repository creation. Once CDK has deployed the components, you will have a new empty repository where we will define the target Service Catalog estate. To do so, clone the new repository and push our sample code into it:</p>\\n<pre><code class=\\"lang-\\">cd ..\\ngit clone https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/service-catalog-repo\\ncd service-catalog-repo\\ngit checkout -b main\\ncp -aR ../aws-cdk-service-catalog-pipeline/* .\\ngit add .\\ngit commit -am &quot;First commit&quot;\\ngit push --set-upstream origin main\\n</code></pre>\\n<h5><a id=\\"Review_and_update_configuration_52\\"></a><strong>Review and update configuration</strong></h5>\\n<p>Our <a href=\\"https://github.com/aws-samples/aws-cdk-service-catalog-pipeline/blob/main/cdk.json\\" target=\\"_blank\\">cdk.json</a> file is used to manage <a href=\\"https://docs.aws.amazon.com/cdk/v1/guide/context.html\\" target=\\"_blank\\">context settings</a> such as shared accounts, permissions, region to deploy, etc.</p>\\n<pre><code class=\\"lang-\\">shared_accounts_ecs: Amazon Web Services account IDs where the ECS portfolio will be shared\\nshared_accounts_storage: Amazon Web Services account IDs where the Storage portfolio will be shared\\nroles: ARN for the roles who will have permissions to access to the Portfolio\\nusers: ARN for the users who will have permissions to access to the Portfolio\\ngroups: ARN for the groups who will have permissions to access to the Portfolio\\nhub_account: Amazon Web Services account ID where the Portfolio will be created\\npipeline_account: AAmazon Web ServicesWS account ID where the main Infrastructure Pipeline will be created\\nregion: the Amazon Web Services region to be used for the deployment of the account\\n</code></pre>\\n<pre><code class=\\"lang-\\">&quot;shared_accounts_ecs&quot;:[&quot;012345678901&quot;,&quot;012345678902&quot;],\\n &quot;shared_accounts_storage&quot;:[&quot;012345678901&quot;,&quot;012345678902&quot;],\\n &quot;roles&quot;:[],\\n &quot;users&quot;:[],\\n &quot;groups&quot;:[],\\n &quot;hub_account&quot;:&quot;012345678901&quot;,\\n &quot;pipeline_account&quot;:&quot;012345678901&quot;,\\n &quot;region&quot;:&quot;eu-west-1&quot;\\n</code></pre>\\n<p>There are two mechanisms that can be used to create Service Catalog Products in this solution: 1) providing a CloudFormation template or 2) declaring a CDK stack (that will be transformed as part of the pipeline). Our sample contains two Products, each demonstrating one of these options: an <a href=\\"https://aws.amazon.com/ecs/\\" target=\\"_blank\\">Amazon Elastic Container Services (ECS)</a> deployment and an <a href=\\"https://aws.amazon.com/s3/\\" target=\\"_blank\\">Amazon Simple Storage Service (S3)</a> product.</p>\\n<p>These Products are automatically shared with accounts specified in the shared_accounts_storage variable. Each product is managed by a CDK Python file in the cdk_service_catalog folder.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/8ed67a0bcdea452692007d3babe9bc4f_image.png\\" alt=\\"image.png\\" /></p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/2607643bc26f467b8d25215c00c65491_image.png\\" alt=\\"image.png\\" /></p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/e3036f2bef294e89847e1b0bef194cec_image.png\\" alt=\\"image.png\\" /></p>\n<p>The Pipeline stages that Amazon Web Services CodePipeline runs through are as follows:</p>\n<ol>\\n<li>Download the Amazon Web Services CodeCommit code</li>\n<li>Synthesize the CDK code to transform it into a CloudFormation template</li>\n<li>Auto-modify the Pipeline in case you have made manual changes to it</li>\n<li>Display the different Portfolios and Products associated in a Hub account in a Region or in multiple accounts</li>\n</ol>\\n<h4><a id=\\"Adding_new_Portfolios_and_Products_95\\"></a><strong>Adding new Portfolios and Products</strong></h4>\\n<p>To add a new Portfolio to the Pipeline, we recommend creating a new class under <a href=\\"https://github.com/aws-samples/aws-cdk-service-catalog-pipeline/tree/main/cdk_service_catalog\\" target=\\"_blank\\">cdk_service_catalog</a> similar to <a href=\\"https://github.com/aws-samples/aws-cdk-service-catalog-pipeline/blob/main/cdk_service_catalog/cdk_service_catalog_ecs_stack.py\\" target=\\"_blank\\">cdk_service_catalog_ecs_stack.py from our sample</a>. Once the new class is created with the products you wish to associate, we instantiate the new class inside <a href=\\"https://github.com/aws-samples/aws-cdk-service-catalog-pipeline/blob/main/cdk_pipelines/cdk_pipelines.py\\" target=\\"_blank\\">cdk_pipelines.py</a>, and then add it inside the wave in the stage. There are two ways to create portfolio products. The first one is by creating a CloudFormation template, as can be seen in the <a href=\\"https://aws.amazon.com/ecs/\\" target=\\"_blank\\">Amazon Elastic Container Service (ECS)</a> example. The second way is by creating a CDK stack that will be transformed into a template, as can be seen in the Storage example.</p>\\n<h5><a id=\\"Product_and_Portfolio_definition_99\\"></a><strong>Product and Portfolio definition:</strong></h5>\\n<pre><code class=\\"lang-\\">class ECSCluster(servicecatalog.ProductStack):\\n def __init__(self, scope, id):\\n super().__init__(scope, id)\\n # Parameters for the Product Template\\n cluster_name = cdk.CfnParameter(self, &quot;clusterName&quot;, type=&quot;String&quot;, description=&quot;The name of the ECS cluster&quot;)\\n container_insights_enable = cdk.CfnParameter(self, &quot;container_insights&quot;, type=&quot;String&quot;,default=&quot;False&quot;,allowed_values=[&quot;False&quot;,&quot;True&quot;],description=&quot;Enable Container Insights&quot;)\\n vpc = cdk.CfnParameter(self, &quot;vpc&quot;, type=&quot;AWS::EC2::VPC::Id&quot;, description=&quot;VPC&quot;)\\n ecs.Cluster(self,&quot;ECSCluster_template&quot;, enable_fargate_capacity_providers=True,cluster_name=cluster_name.value_as_string,container_insights=bool(container_insights_enable.value_as_string),vpc=vpc)\\n cdk.Tags.of(self).add(&quot;key&quot;, &quot;value&quot;)\\n</code></pre>\\n<h4><a id=\\"Clean_up_113\\"></a><strong>Clean up</strong></h4>\\n<p>The following will help you clean up all necessary parts of this post: After completing your demo, feel free to delete your stack using the CDK CLI:</p>\n<pre><code class=\\"lang-\\">cdk destroy --all\\n</code></pre>\\n<h4><a id=\\"Conclusion_121\\"></a><strong>Conclusion</strong></h4>\\n<p>In this post, we demonstrated how Service Catalog deployments can be accelerated by building a CI/CD pipeline using self-managed services. The Portfolio &amp; Product estate is defined in its entirety by using Infrastructure-as-Code and automatically deployed based on your configuration. To learn more about Amazon Web Services <a href=\\"https://docs.aws.amazon.com/cdk/api/v1/\\" target=\\"_blank\\">CDK Pipelines</a> or <a href=\\"https://docs.aws.amazon.com/servicecatalog/\\" target=\\"_blank\\">Amazon Web Services Service Catalog</a>, visit the appropriate product documentation.</p>\\n<h5><a id=\\"Authors_125\\"></a><strong>Authors:</strong></h5>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/269fa4d254774623acc53ca32bff6751_image.png\\" alt=\\"image.png\\" /></p>\n<h5><a id=\\"Csar_Prieto_Ballester_129\\"></a><strong>César Prieto Ballester</strong></h5>\\n<p>César Prieto Ballester is a Senior DevOps Consultant at Amazon Web Services. He enjoys automating everything and building infrastructure using code. Apart from work, he plays electric guitar and loves riding his mountain bike.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/53d8b94690ac47f7bbec5f2e54c58cd2_image.png\\" alt=\\"image.png\\" /></p>\n<h5><a id=\\"Daniel_Mutale_134\\"></a><strong>Daniel Mutale</strong></h5>\\n<p>Daniel Mutale is a Cloud Infrastructure Architect at Amazon Web Services Professional Services. He enjoys creating cloud based architectures and building out the underlying infrastructure to support the architectures using code. Apart from work, he is an avid animal photographer and has a passion for interior design.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/a981c0067f5949128b9bc348d0b6fa37_image.png\\" alt=\\"image.png\\" /></p>\n<h5><a id=\\"Raphael_Sack_139\\"></a><strong>Raphael Sack</strong></h5>\\n<p>Raphael is a technical business development manager for Service Catalog &amp; Control Tower. He enjoys tinkering with automation and code and active member of the management tools community.</p>\n"}