{"value":"In this blog post, I describe the recommended [AWS Cloud Development Kit (AWS CDK)](https://aws.amazon.com/cdk/) project structure for Python applications. This is based on [Best practices for developing and deploying cloud infrastructure with the AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/best-practices.html).\n\n\nThe AWS CDK is an open source software development framework for modeling and provisioning your cloud application resources through AWS CloudFormation by utilizing familiar programming languages, including TypeScript, JavaScript, Python, C#, Java, and Go.\n\n\nThe AWS CDK application maps to a component as [defined](https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html#introduction) by the AWS Well-Architected Framework. A component usually includes logical units (e.g., api, database), and optionally can have a continuous deployment pipeline. The logical units should be implemented as [constructs](https://docs.aws.amazon.com/cdk/latest/guide/constructs.html), including the infrastructure (e.g., Amazon S3 buckets, Amazon RDS databases, [Amazon VPC](https://aws.amazon.com/vpc/) network), runtime (e.g., [AWS Lambda](https://aws.amazon.com/lambda/) function code), and configuration code.\n\n\nFor an example, I will walk through a user management backend component that utilizes [Amazon API Gateway](https://aws.amazon.com/api-gateway/), [AWS Lambda](https://aws.amazon.com/lambda/), and [Amazon DynamoDB ](https://www.google.com/aclk?sa=L&ai=DChcSEwix8t-arPPxAhUYwdUKHZUtDNUYABAAGgJ3cw&ae=2&sig=AOD64_1AMRtOs1siISFeGaALLwqQULgQQA&q&adurl&ved=2ahUKEwi4ttiarPPxAhVBhlwKHcyPCQAQ0Qx6BAgCEAE)to provide basic CRUD operations for managing users. The project also includes a continuous deployment pipeline. This essentially contains everything required for managing the component as a unit of ownership, including the specific deployment environments.\n\n### **Concepts**\n\nWe recommend organizing the project directory structure based on the component’s logical units. Each logical unit should have a directory and include the related infrastructure, runtime, and configuration code. For example:\n\n```\n.\n|-- api\n| |-- infrastructure.py\n| `-- runtime\n| |-- app.py\n| `-- requirements.txt\n|-- database\n| `-- infrastructure.py\n```\n\nThis way, if I need to make API changes, then I can easily find the code related to that logical unit. If I need to refactor the code, or make it a separate unit of ownership, it can be changed in a single place. In other words, it is a [self-contained](https://en.wikipedia.org/wiki/Self-contained_system_(software)) unit.\n\nThe logical units should be implemented as [constructs ](https://docs.aws.amazon.com/cdk/latest/guide/constructs.html)and not as [stacks](https://docs.aws.amazon.com/cdk/latest/guide/stacks.html). Constructs are the basic building blocks of AWS CDK applications, while stacks are deployment units. All of the AWS resources defined within the scope of a stack, either directly or indirectly, are provisioned as a single unit. Implementing logical units as constructs provides the flexibility to support different deployment layouts and enables future reuse as construct libraries. I will further discuss the deployment layout later.\n\n- **Note:** When refactoring constructs, consider [logical ID](https://docs.aws.amazon.com/cdk/latest/guide/identifiers.html#identifiers_logical_ids) stability to avoid unexpected infrastructure changes.\n\n\nBefore the AWS CDK arrived, runtime and infrastructure code remained two separate concepts. The AWS CDK abstraction lets you to combine the infrastructure and runtime code of a logical unit behind a single construct interface.\n\n### **Project structure**\n\n\nLet’s look at the recommended project structure in detail. Clone the example I use in this blog post from [https://github.com/aws-samples/aws-cdk-project-structure-python](https://github.com/aws-samples/aws-cdk-project-structure-python). Note that I have left out source code snippets and files in the blog post, such as linters and the full pipeline structure. The focus is on the code that illustrates the project structure recommendations, while the source code still provides a fully functional project for reference. Below is a snapshot of the project structure, excluding files not in the scope of this blog post:\n\n\n```\n# The recommended project structure example\n.\n|-- api\n| |-- __init__.py\n| |-- infrastructure.py\n| `-- runtime\n| |-- app.py\n| `-- requirements.txt\n|-- database\n| |-- __init__.py\n| `-- infrastructure.py\n|-- monitoring\n| |-- __init__.py\n| `-- infrastructure.py\n|-- app.py\n|-- constants.py\n|-- deployment.py\n|-- pipeline.py\n`-- requirements.txt\n```\n\nThree logical units compose the user management backend: API, database, and monitoring. Each logical unit contains an ```infrastructure.py``` [module](https://docs.python.org/3/tutorial/modules.html). If the infrastructure implementation were more complex, then I would replace the ```infrastructure.py``` module with ```infrastructure``` [package](https://docs.python.org/3/tutorial/modules.html#packages), which contains multiple modules. Some logical units also have a ```runtime``` directory. For example, the API has a ```runtime``` directory containing Lambda function code.\n\nNext, I will cover ```app.py``` (the AWS CDK application entry point), ```deployment.py``` (the user management backend deployment layout), and ```pipeline.py``` (the continuous deployment pipeline) modules in order to show the implementation of the recommended project structure. We’ll start with ```app.py```.\n\n**app.py**\n\nPython\n\n```\n# The AWS CDK application entry point\n...\nimport constants\nfrom deployment import UserManagementBackend\nfrom pipeline import Pipeline\n\napp = cdk.App()\n\n# Development\nUserManagementBackend(\n app,\n f\"{constants.CDK_APP_NAME}-Dev\",\n env=constants.DEV_ENV,\n api_lambda_reserved_concurrency=constants.DEV_API_LAMBDA_RESERVED_CONCURRENCY,\n database_dynamodb_billing_mode=constants.DEV_DATABASE_DYNAMODB_BILLING_MODE,\n)\n\n# Production pipeline\nPipeline(app, f\"{constants.CDK_APP_NAME}-Pipeline\", env=constants.PIPELINE_ENV)\n\napp.synth()\n```\n\nThe module contains the ```app``` object, followed by the development environment definition and the continuous deployment pipeline.\n\n- **Note:** ```constants.CDK_APP_NAME``` is utilized as part of the construct [identifier](https://docs.aws.amazon.com/cdk/latest/guide/identifiers.html) (e.g., ```f\"{constants.CDK_APP_NAME}-Dev\"``` above) in order to set a unique prefix for a CloudFormation stack name.\n\n\nIn this case, I utilize [CDK Pipelines](https://aws.amazon.com/blogs/developer/cdk-pipelines-continuous-delivery-for-aws-cdk-applications/) for continuous deployment, and instantiate the pipeline stack here. Then, the pipeline deploys the user management backend to Prod environment.\n\n- **Note:** We recommend deploying the pipeline in a separate production deployment account. See [Separating CI/CD management capabilities from workloads](https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/deployments-ou.html#separating-cicd-management-capabilities-from-workloads) for more details.\n\nDuring development, I will iterate quickly and deploy changes to my development environment. The ```UserManagementBackend``` definition above enables this. It defines the user management backend deployment layout for my development environment. As you can see, the ```UserManagementBackend``` class is imported from the ```deployment.py``` module. Let’s look into it.\n\n#### **deployment.py**\n\nPython\n\n```\n# The user management backend deployment layout\n...\nfrom api.infrastructure import API\nfrom database.infrastructure import Database\nfrom monitoring.infrastructure import Monitoring\n\nclass UserManagementBackend(cdk.Stage):\n def __init__(\n self,\n scope: cdk.Construct,\n id_: str,\n *,\n database_dynamodb_billing_mode: dynamodb.BillingMode,\n api_lambda_reserved_concurrency: int,\n **kwargs: Any,\n ):\n super().__init__(scope, id_, **kwargs)\n\n stateful = cdk.Stack(self, \"Stateful\")\n database = Database(\n stateful, \"Database\", dynamodb_billing_mode=database_dynamodb_billing_mode\n )\n stateless = cdk.Stack(self, \"Stateless\")\n api = API(\n stateless,\n \"API\",\n dynamodb_table=database.table,\n lambda_reserved_concurrency=api_lambda_reserved_concurrency,\n )\n Monitoring(stateless, \"Monitoring\", database=database, api=api)\n\n self.api_endpoint_url = api.endpoint_url\n\nPython\n\n```\n\nThe ```UserManagementBackend``` class inherits from ```cdk.Stage```—an abstract deployment unit consisting of one or more [stacks](https://docs.aws.amazon.com/cdk/latest/guide/stacks.html) that should be deployed together. This is where the separation between constructs as logical units, and their deployment layout as stacks, reveals its flexibility. In this example, I deploy the stateful database logical unit in a separate stack from the stateless API and monitoring logical units. The ```UserManagementBackend``` class combines the stateful and stateless stacks into a single deployment stage.\n\nFinally, let’s look at the pipeline definition.\n\n\n**pipeline.py**\n\nPython\n\n```\n# The continuous deployment pipeline\n...\nfrom aws_cdk import core as cdk\nfrom aws_cdk import pipelines\n\nimport constants\nfrom deployment import UserManagementBackend\n\nclass Pipeline(cdk.Stack):\n def __init__(self, scope: cdk.Construct, id_: str, **kwargs: Any):\n super().__init__(scope, id_, **kwargs)\n ...\n codepipeline = pipelines.CodePipeline(...)\n self._add_prod_stage(codepipeline)\n ...\n def _add_prod_stage(self, codepipeline: pipelines.CodePipeline) -> None:\n prod_stage = UserManagementBackend(\n self,\n f\"{constants.CDK_APP_NAME}-Prod\",\n env=constants.PROD_ENV,\n api_lambda_reserved_concurrency=constants.PROD_API_LAMBDA_RESERVED_CONCURRENCY,\n database_dynamodb_billing_mode=constants.PROD_DATABASE_DYNAMODB_BILLING_MODE,\n )\n ...\n codepipeline.add_stage(prod_stage, post=[smoke_test_shell_step])\n```\n\nThis time, the ```UserManagementBackend``` stage is utilized for deployment to a Prod environment via pipeline. This lets me keep my development environment similar to the Prod environment, all while remaining able to add customizations. For example, I use the ```database_dynamodb_billing_mode``` argument to set DynamoDB capacity mode to on-demand for the development environment and to [provisioned mode](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html) for the Prod environment.\n\n\n### **Conclusion**\n\n\nThe AWS CDK allows for infrastructure code to be located in the same repository with runtime code. This leads to additional considerations, such as how to structure the project. In this blog post, I have described the recommended AWS CDK project structure for Python applications, thereby aiming to ease the maintenance and evolution of your projects.\n\nIf you think I’ve missed something, or you have a use case that I didn’t cover, we would love to hear from you on the [aws-cdk](https://github.com/aws/aws-cdk) GitHub repository. Happy coding!\n\n\n#### **About the author**\n\n![image.png](https://dev-media.amazoncloud.cn/9cd8aad1ea944172845637fa77587aac_image.png)\n\n\n**Alex Pulver is a Sr. Partner Solutions Architect at AWS SaaS Factory team.** He works with AWS Partners at any stage of their software-as-a-service (SaaS) journey in order to help build new products, migrate existing applications, or optimize SaaS solutions on AWS. His areas of interest include builder experience (e.g., developer tools, DevOps culture, CI/CD), containers, security, IoT, and AWS multi-account strategy.","render":"<p>In this blog post, I describe the recommended <a href=\"https://aws.amazon.com/cdk/\" target=\"_blank\">AWS Cloud Development Kit (AWS CDK)</a> project structure for Python applications. This is based on <a href=\"https://docs.aws.amazon.com/cdk/latest/guide/best-practices.html\" target=\"_blank\">Best practices for developing and deploying cloud infrastructure with the AWS CDK</a>.</p>\n<p>The AWS CDK is an open source software development framework for modeling and provisioning your cloud application resources through AWS CloudFormation by utilizing familiar programming languages, including TypeScript, JavaScript, Python, C#, Java, and Go.</p>\n<p>The AWS CDK application maps to a component as <a href=\"https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html#introduction\" target=\"_blank\">defined</a> by the AWS Well-Architected Framework. A component usually includes logical units (e.g., api, database), and optionally can have a continuous deployment pipeline. The logical units should be implemented as <a href=\"https://docs.aws.amazon.com/cdk/latest/guide/constructs.html\" target=\"_blank\">constructs</a>, including the infrastructure (e.g., Amazon S3 buckets, Amazon RDS databases, <a href=\"https://aws.amazon.com/vpc/\" target=\"_blank\">Amazon VPC</a> network), runtime (e.g., <a href=\"https://aws.amazon.com/lambda/\" target=\"_blank\">AWS Lambda</a> function code), and configuration code.</p>\n<p>For an example, I will walk through a user management backend component that utilizes <a href=\"https://aws.amazon.com/api-gateway/\" target=\"_blank\">Amazon API Gateway</a>, <a href=\"https://aws.amazon.com/lambda/\" target=\"_blank\">AWS Lambda</a>, and <a href=\"https://www.google.com/aclk?sa=L&ai=DChcSEwix8t-arPPxAhUYwdUKHZUtDNUYABAAGgJ3cw&ae=2&sig=AOD64_1AMRtOs1siISFeGaALLwqQULgQQA&q&adurl&ved=2ahUKEwi4ttiarPPxAhVBhlwKHcyPCQAQ0Qx6BAgCEAE\" target=\"_blank\">Amazon DynamoDB </a>to provide basic CRUD operations for managing users. The project also includes a continuous deployment pipeline. This essentially contains everything required for managing the component as a unit of ownership, including the specific deployment environments.</p>\n<h3><a id=\"Concepts_11\"></a><strong>Concepts</strong></h3>\n<p>We recommend organizing the project directory structure based on the component’s logical units. Each logical unit should have a directory and include the related infrastructure, runtime, and configuration code. For example:</p>\n<pre><code class=\"lang-\">.\n|-- api\n| |-- infrastructure.py\n| `-- runtime\n| |-- app.py\n| `-- requirements.txt\n|-- database\n| `-- infrastructure.py\n</code></pre>\n<p>This way, if I need to make API changes, then I can easily find the code related to that logical unit. If I need to refactor the code, or make it a separate unit of ownership, it can be changed in a single place. In other words, it is a <a href=\"https://en.wikipedia.org/wiki/Self-contained_system_(software)\" target=\"_blank\">self-contained</a> unit.</p>\n<p>The logical units should be implemented as <a href=\"https://docs.aws.amazon.com/cdk/latest/guide/constructs.html\" target=\"_blank\">constructs </a>and not as <a href=\"https://docs.aws.amazon.com/cdk/latest/guide/stacks.html\" target=\"_blank\">stacks</a>. Constructs are the basic building blocks of AWS CDK applications, while stacks are deployment units. All of the AWS resources defined within the scope of a stack, either directly or indirectly, are provisioned as a single unit. Implementing logical units as constructs provides the flexibility to support different deployment layouts and enables future reuse as construct libraries. I will further discuss the deployment layout later.</p>\n<ul>\n<li><strong>Note:</strong> When refactoring constructs, consider <a href=\"https://docs.aws.amazon.com/cdk/latest/guide/identifiers.html#identifiers_logical_ids\" target=\"_blank\">logical ID</a> stability to avoid unexpected infrastructure changes.</li>\n</ul>\n<p>Before the AWS CDK arrived, runtime and infrastructure code remained two separate concepts. The AWS CDK abstraction lets you to combine the infrastructure and runtime code of a logical unit behind a single construct interface.</p>\n<h3><a id=\"Project_structure_35\"></a><strong>Project structure</strong></h3>\n<p>Let’s look at the recommended project structure in detail. Clone the example I use in this blog post from <a href=\"https://github.com/aws-samples/aws-cdk-project-structure-python\" target=\"_blank\">https://github.com/aws-samples/aws-cdk-project-structure-python</a>. Note that I have left out source code snippets and files in the blog post, such as linters and the full pipeline structure. The focus is on the code that illustrates the project structure recommendations, while the source code still provides a fully functional project for reference. Below is a snapshot of the project structure, excluding files not in the scope of this blog post:</p>\n<pre><code class=\"lang-\"># The recommended project structure example\n.\n|-- api\n| |-- __init__.py\n| |-- infrastructure.py\n| `-- runtime\n| |-- app.py\n| `-- requirements.txt\n|-- database\n| |-- __init__.py\n| `-- infrastructure.py\n|-- monitoring\n| |-- __init__.py\n| `-- infrastructure.py\n|-- app.py\n|-- constants.py\n|-- deployment.py\n|-- pipeline.py\n`-- requirements.txt\n</code></pre>\n<p>Three logical units compose the user management backend: API, database, and monitoring. Each logical unit contains an <code>infrastructure.py</code> <a href=\"https://docs.python.org/3/tutorial/modules.html\" target=\"_blank\">module</a>. If the infrastructure implementation were more complex, then I would replace the <code>infrastructure.py</code> module with <code>infrastructure</code> <a href=\"https://docs.python.org/3/tutorial/modules.html#packages\" target=\"_blank\">package</a>, which contains multiple modules. Some logical units also have a <code>runtime</code> directory. For example, the API has a <code>runtime</code> directory containing Lambda function code.</p>\n<p>Next, I will cover <code>app.py</code> (the AWS CDK application entry point), <code>deployment.py</code> (the user management backend deployment layout), and <code>pipeline.py</code> (the continuous deployment pipeline) modules in order to show the implementation of the recommended project structure. We’ll start with <code>app.py</code>.</p>\n<p><strong>app.py</strong></p>\n<p>Python</p>\n<pre><code class=\"lang-\"># The AWS CDK application entry point\n...\nimport constants\nfrom deployment import UserManagementBackend\nfrom pipeline import Pipeline\n\napp = cdk.App()\n\n# Development\nUserManagementBackend(\n app,\n f"{constants.CDK_APP_NAME}-Dev",\n env=constants.DEV_ENV,\n api_lambda_reserved_concurrency=constants.DEV_API_LAMBDA_RESERVED_CONCURRENCY,\n database_dynamodb_billing_mode=constants.DEV_DATABASE_DYNAMODB_BILLING_MODE,\n)\n\n# Production pipeline\nPipeline(app, f"{constants.CDK_APP_NAME}-Pipeline", env=constants.PIPELINE_ENV)\n\napp.synth()\n</code></pre>\n<p>The module contains the <code>app</code> object, followed by the development environment definition and the continuous deployment pipeline.</p>\n<ul>\n<li><strong>Note:</strong> <code>constants.CDK_APP_NAME</code> is utilized as part of the construct <a href=\"https://docs.aws.amazon.com/cdk/latest/guide/identifiers.html\" target=\"_blank\">identifier</a> (e.g., <code>f"{constants.CDK_APP_NAME}-Dev"</code> above) in order to set a unique prefix for a CloudFormation stack name.</li>\n</ul>\n<p>In this case, I utilize <a href=\"https://aws.amazon.com/blogs/developer/cdk-pipelines-continuous-delivery-for-aws-cdk-applications/\" target=\"_blank\">CDK Pipelines</a> for continuous deployment, and instantiate the pipeline stack here. Then, the pipeline deploys the user management backend to Prod environment.</p>\n<ul>\n<li><strong>Note:</strong> We recommend deploying the pipeline in a separate production deployment account. See <a href=\"https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/deployments-ou.html#separating-cicd-management-capabilities-from-workloads\" target=\"_blank\">Separating CI/CD management capabilities from workloads</a> for more details.</li>\n</ul>\n<p>During development, I will iterate quickly and deploy changes to my development environment. The <code>UserManagementBackend</code> definition above enables this. It defines the user management backend deployment layout for my development environment. As you can see, the <code>UserManagementBackend</code> class is imported from the <code>deployment.py</code> module. Let’s look into it.</p>\n<h4><a id=\"deploymentpy_106\"></a><strong>deployment.py</strong></h4>\n<p>Python</p>\n<pre><code class=\"lang-\"># The user management backend deployment layout\n...\nfrom api.infrastructure import API\nfrom database.infrastructure import Database\nfrom monitoring.infrastructure import Monitoring\n\nclass UserManagementBackend(cdk.Stage):\n def __init__(\n self,\n scope: cdk.Construct,\n id_: str,\n *,\n database_dynamodb_billing_mode: dynamodb.BillingMode,\n api_lambda_reserved_concurrency: int,\n **kwargs: Any,\n ):\n super().__init__(scope, id_, **kwargs)\n\n stateful = cdk.Stack(self, "Stateful")\n database = Database(\n stateful, "Database", dynamodb_billing_mode=database_dynamodb_billing_mode\n )\n stateless = cdk.Stack(self, "Stateless")\n api = API(\n stateless,\n "API",\n dynamodb_table=database.table,\n lambda_reserved_concurrency=api_lambda_reserved_concurrency,\n )\n Monitoring(stateless, "Monitoring", database=database, api=api)\n\n self.api_endpoint_url = api.endpoint_url\n\nPython\n\n</code></pre>\n<p>The <code>UserManagementBackend</code> class inherits from <code>cdk.Stage</code>—an abstract deployment unit consisting of one or more <a href=\"https://docs.aws.amazon.com/cdk/latest/guide/stacks.html\" target=\"_blank\">stacks</a> that should be deployed together. This is where the separation between constructs as logical units, and their deployment layout as stacks, reveals its flexibility. In this example, I deploy the stateful database logical unit in a separate stack from the stateless API and monitoring logical units. The <code>UserManagementBackend</code> class combines the stateful and stateless stacks into a single deployment stage.</p>\n<p>Finally, let’s look at the pipeline definition.</p>\n<p><strong>pipeline.py</strong></p>\n<p>Python</p>\n<pre><code class=\"lang-\"># The continuous deployment pipeline\n...\nfrom aws_cdk import core as cdk\nfrom aws_cdk import pipelines\n\nimport constants\nfrom deployment import UserManagementBackend\n\nclass Pipeline(cdk.Stack):\n def __init__(self, scope: cdk.Construct, id_: str, **kwargs: Any):\n super().__init__(scope, id_, **kwargs)\n ...\n codepipeline = pipelines.CodePipeline(...)\n self._add_prod_stage(codepipeline)\n ...\n def _add_prod_stage(self, codepipeline: pipelines.CodePipeline) -> None:\n prod_stage = UserManagementBackend(\n self,\n f"{constants.CDK_APP_NAME}-Prod",\n env=constants.PROD_ENV,\n api_lambda_reserved_concurrency=constants.PROD_API_LAMBDA_RESERVED_CONCURRENCY,\n database_dynamodb_billing_mode=constants.PROD_DATABASE_DYNAMODB_BILLING_MODE,\n )\n ...\n codepipeline.add_stage(prod_stage, post=[smoke_test_shell_step])\n</code></pre>\n<p>This time, the <code>UserManagementBackend</code> stage is utilized for deployment to a Prod environment via pipeline. This lets me keep my development environment similar to the Prod environment, all while remaining able to add customizations. For example, I use the <code>database_dynamodb_billing_mode</code> argument to set DynamoDB capacity mode to on-demand for the development environment and to <a href=\"https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html\" target=\"_blank\">provisioned mode</a> for the Prod environment.</p>\n<h3><a id=\"Conclusion_188\"></a><strong>Conclusion</strong></h3>\n<p>The AWS CDK allows for infrastructure code to be located in the same repository with runtime code. This leads to additional considerations, such as how to structure the project. In this blog post, I have described the recommended AWS CDK project structure for Python applications, thereby aiming to ease the maintenance and evolution of your projects.</p>\n<p>If you think I’ve missed something, or you have a use case that I didn’t cover, we would love to hear from you on the <a href=\"https://github.com/aws/aws-cdk\" target=\"_blank\">aws-cdk</a> GitHub repository. Happy coding!</p>\n<h4><a id=\"About_the_author_196\"></a><strong>About the author</strong></h4>\n<p><img src=\"https://dev-media.amazoncloud.cn/9cd8aad1ea944172845637fa77587aac_image.png\" alt=\"image.png\" /></p>\n<p><strong>Alex Pulver is a Sr. Partner Solutions Architect at AWS SaaS Factory team.</strong> He works with AWS Partners at any stage of their software-as-a-service (SaaS) journey in order to help build new products, migrate existing applications, or optimize SaaS solutions on AWS. His areas of interest include builder experience (e.g., developer tools, DevOps culture, CI/CD), containers, security, IoT, and AWS multi-account strategy.</p>\n"}