Throttling a tiered, multi-tenant REST API at scale using API Gateway: Part 2

{"value":"In [Part 1](https://aws.amazon.com/blogs/architecture/throttling-a-tiered-multi-tenant-rest-api-at-scale-using-api-gateway-part-1/) of this blog series, we demonstrated why tiering and throttling become necessary at scale for multi-tenant REST APIs, and explored tiering strategy and throttling with [Amazon API Gateway.](https://aws.amazon.com/api-gateway/)\n\nIn this post, Part 2, we will examine tenant isolation strategies at scale with API Gateway and extend the sample code from Part 1.\n\n### **Enhancing the sample code**\nTo enable this functionality in the sample code (Figure 1), we will make manual changes. First, create one API key for the Free Tier and five API keys for the Basic Tier. Currently, these API keys are private keys for your Amazon Cognito login, but we will make a further change in the backend business logic that will promote them to pooled resources. Note that all of these modifications are specific to this sample code’s implementation; the implementation and deployment of a production code may be completely different (Figure 1).\n\n\n\nFigure 1. Cloud architecture of the sample code\n\nNext, in the business logic for the ```createKey()```, find the AWS Lambda function in ```lambda/create_key.js```. It appears like this:\n\nXML\n```\nfunction createKey(tableName, key, plansTable, jwt, rand, callback) {\n const pool = getPoolForPlanId( key.planId ) \n if (!pool) {\n createSiloedKey(tableName, key, plansTable, jwt, rand, callback);\n } else {\n createPooledKey(pool, tableName, key, jwt, callback);\n }\n}\n```\nThe ```getPoolForPlanId()``` function does a search for a pool of keys associated with the usage plan. If there is a pool, we “create” a kind of reference to the pooled resource, rather than a completely new key that is created by the API Gateway service directly. The ```lambda/api_key_pools.js``` should be empty.\n\n```exports.apiKeyPools = [];```\n\n\nIn effect, all usage plans were considered as siloed keys up to now. To change that, populate the data structure with values from the six API keys that were created manually. You will have to look up the IDs of the API keys and usage plans that were created in API Gateway (Figures 2 and 3). Using the AWS console to navigate to API Gateway is the most intuitive.\n\n\n\nFigure 2. A view of the AWS console when inspecting the ID for the Basic usage plan\n\n\n\nFigure 3. A view of the AWS Console when looking up the API key value (not the ID)\n\nWhen done, your code in ```lambda/api_key_pools.js``` should be the following, but instead of ellipses (```…```), the IDs for the user plans and API keys specific to your environment will appear.\n\nXML\n```\nexports.apiKeyPools = [{\n planName: \"FreePlan\"\n planId: \"...\",\n apiKeys: [ \"...\" ]\n },\n {\n planName: \"BasicPlan\"\n planId: \"...\",\n apiKeys: [ \"...\", \"...\", \"...\", \"...\", \"...\" ]\n }];\n```\nAfter making the code changes, run ```cdk deploy``` from the command line to update the Lambda functions. This change will only affect key creation and deletion because of the system implementation. Updates affect only the user’s specific reference to the key, not the underlying resource managed by API Gateway.\n\nWhen the web application is run now, it will look similar to before—tenants should not be aware what tiering strategy they have been assigned to. The only way to notice the difference would be to create two Free Tier keys, test them, and note that the value of the ```X-API-KEY ``` header is unchanged between the two.\n\nNow, you have a virtually unlimited number of users who can have API keys in the Free or Basic Tier. By keeping the Premium Tier siloed, you are subject to the 10,000-API-key maximum (less any keys allocated for the lower tiers). You may consider additional techniques to continue to scale, such as replicating your service in another AWS account.\n\n### **Other production considerations**\nThe sample code is minimal, and it illustrates just one aspect of scaling a Software-as-a-service (SaaS) application. There are many other aspects be considered in a production setting that we explore in this section.\n\nThe throttled endpoint, ```GET /api``` rely only on API key for authorization for demonstration purpose. For any production implementation consider [authentication](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html) options for your REST APIs. You may explore and extend to require authentication with Cognito similar to ```/admin/*``` endpoints in the sample code.\n\nOne API key for Free Tier access and five API keys for Basic Tier access are illustrative in a sample code but not representative of production deployments. Number of API keys with service quota into consideration, business and technical decisions may be made to minimize noisy neighbor effect such as setting blast radius upper threshold of 0.1% of all users. To satisfy that requirement, each tier would need to spread users across at least 1,000 API keys. The number of keys allocated to Basic or Premium Tier would depend on market needs and pricing strategies. Additional allocations of keys could be held in reserve for troubleshooting, QA, tenant migrations, and key retirement.\n\nIn the planning phase of your solution, you will decide how many tiers to provide, how many usage plans are needed, and what throttle limits and quotas to apply. These decisions depend on your architecture and business.\n\nTo define API request limits, examine the system API Gateway is protecting and what load it can sustain. For example, if your service will scale up to 1,000 requests per second, it is possible to implement three tiers with a 10/50/40 split: the lowest tier shares one common API key with a 100 request per second limit; an intermediate tier has a pool of 25 API keys with a limit of 20 requests per second each; and the highest tier has a maximum of 10 API keys, each supporting 40 requests per second.\n\nMetrics play a large role in continuously evolving your SaaS-tiering strategy (Figure 4). They provide rich insights into how tenants are using the system. Tenant-aware and [SaaS-wide metrics](https://youtu.be/j7Sqt8GpYl0?t=1692) on throttling and quota limits can be used to: assess tiering in-place, if tenants’ requirements are being met, and if currently used tenant usage profiles are valid (Figure 5).\n\n\n\nFigure 4. Tiering strategy example with 3 tiers and requests allocation per tier\n\n\n\nFigure 5. An example SaaS metrics dashboard\n\nAPI Gateway provides options for different levels of granularity required, including [detailed metrics](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html), and [execution and access logging](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html) to enable observability of your SaaS solution. Granular usage metrics combined with underlying resource consumption leads to managing optimal experience for your tenants with throttling levels and policies per method and per client.\n\n### **Cleanup**\nTo avoid incurring future charges, delete the resources. This can be done on the command line by typing:\n\nXML\n```\ncd ${TOP}/cdk\ncdk destroy\n\ncd ${TOP}/react\namplify delete\n```\n```${TOP}``` is the topmost directory of the sample code. For the most up-to-date information, see the README.md file.\n\n### **Conclusion**\nIn this two-part blog series, we have reviewed the best practices and challenges of effectively guarding a tiered multi-tenant REST API hosted in AWS API Gateway. We also explored how throttling policy and quota management can help you continuously evaluate the needs of your tenants and evolve your tiering strategy to protect your backend systems from being overwhelmed by inbound traffic.\n\nFurther reading:\n\n- [AWS Well-Architected SaaS Lens Performance Efficiency pillar](https://docs.aws.amazon.com/wellarchitected/latest/saas-lens/monitoring.html)\n- [AWS Serverless SaaS Workshop](https://catalog.us-east-1.prod.workshops.aws/workshops/b0c6ad36-0a4b-45d8-856b-8a64f0ac76bb/en-US)\n- [SaaS Factory Serverless SaaS reference solution](https://github.com/aws-samples/aws-saas-factory-ref-solution-serverless-saas) \n\n*This series was co-authored by Gary Kumfert, PhD, former Principal Solutions Architect at AWS.*\n#### **Nick Choi**\n\n\nNick Choi is a Solutions Architect with Amazon Web Services. He works with software and SaaS customers to design, build, operate, and evolve their SaaS service on AWS. He is passionate about understanding why and how SaaS businesses are changing way we live, work, and play.","render":"<p>In <a href=\"https://aws.amazon.com/blogs/architecture/throttling-a-tiered-multi-tenant-rest-api-at-scale-using-api-gateway-part-1/\" target=\"_blank\">Part 1</a> of this blog series, we demonstrated why tiering and throttling become necessary at scale for multi-tenant REST APIs, and explored tiering strategy and throttling with <a href=\"https://aws.amazon.com/api-gateway/\" target=\"_blank\">Amazon API Gateway.</a></p>\n<p>In this post, Part 2, we will examine tenant isolation strategies at scale with API Gateway and extend the sample code from Part 1.</p>\n<h3><a id=\"Enhancing_the_sample_code_4\"></a><strong>Enhancing the sample code</strong></h3>\n<p>To enable this functionality in the sample code (Figure 1), we will make manual changes. First, create one API key for the Free Tier and five API keys for the Basic Tier. Currently, these API keys are private keys for your Amazon Cognito login, but we will make a further change in the backend business logic that will promote them to pooled resources. Note that all of these modifications are specific to this sample code’s implementation; the implementation and deployment of a production code may be completely different (Figure 1).</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/1c22b4bc6f4144ecbc981e953693d4d4_image.png\" alt=\"image.png\" /></p>\n<p>Figure 1. Cloud architecture of the sample code</p>\n<p>Next, in the business logic for the <code>createKey()</code>, find the AWS Lambda function in <code>lambda/create_key.js</code>. It appears like this:</p>\n<p>XML</p>\n<pre><code class=\"lang-\">function createKey(tableName, key, plansTable, jwt, rand, callback) {\n const pool = getPoolForPlanId( key.planId ) \n if (!pool) {\n createSiloedKey(tableName, key, plansTable, jwt, rand, callback);\n } else {\n createPooledKey(pool, tableName, key, jwt, callback);\n }\n}\n</code></pre>\n<p>The <code>getPoolForPlanId()</code> function does a search for a pool of keys associated with the usage plan. If there is a pool, we “create” a kind of reference to the pooled resource, rather than a completely new key that is created by the API Gateway service directly. The <code>lambda/api_key_pools.js</code> should be empty.</p>\n<p><code>exports.apiKeyPools = [];</code></p>\n<p>In effect, all usage plans were considered as siloed keys up to now. To change that, populate the data structure with values from the six API keys that were created manually. You will have to look up the IDs of the API keys and usage plans that were created in API Gateway (Figures 2 and 3). Using the AWS console to navigate to API Gateway is the most intuitive.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/8e79ff5b9b794bca9ac7aab043d8893e_image.png\" alt=\"image.png\" /></p>\n<p>Figure 2. A view of the AWS console when inspecting the ID for the Basic usage plan</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/fc841800775540cd9fca5899f727947c_image.png\" alt=\"image.png\" /></p>\n<p>Figure 3. A view of the AWS Console when looking up the API key value (not the ID)</p>\n<p>When done, your code in <code>lambda/api_key_pools.js</code> should be the following, but instead of ellipses (<code>…</code>), the IDs for the user plans and API keys specific to your environment will appear.</p>\n<p>XML</p>\n<pre><code class=\"lang-\">exports.apiKeyPools = [{\n planName: &quot;FreePlan&quot;\n planId: &quot;...&quot;,\n apiKeys: [ &quot;...&quot; ]\n },\n {\n planName: &quot;BasicPlan&quot;\n planId: &quot;...&quot;,\n apiKeys: [ &quot;...&quot;, &quot;...&quot;, &quot;...&quot;, &quot;...&quot;, &quot;...&quot; ]\n }];\n</code></pre>\n<p>After making the code changes, run <code>cdk deploy</code> from the command line to update the Lambda functions. This change will only affect key creation and deletion because of the system implementation. Updates affect only the user’s specific reference to the key, not the underlying resource managed by API Gateway.</p>\n<p>When the web application is run now, it will look similar to before—tenants should not be aware what tiering strategy they have been assigned to. The only way to notice the difference would be to create two Free Tier keys, test them, and note that the value of the <code>X-API-KEY </code> header is unchanged between the two.</p>\n<p>Now, you have a virtually unlimited number of users who can have API keys in the Free or Basic Tier. By keeping the Premium Tier siloed, you are subject to the 10,000-API-key maximum (less any keys allocated for the lower tiers). You may consider additional techniques to continue to scale, such as replicating your service in another AWS account.</p>\n<h3><a id=\"Other_production_considerations_60\"></a><strong>Other production considerations</strong></h3>\n<p>The sample code is minimal, and it illustrates just one aspect of scaling a Software-as-a-service (SaaS) application. There are many other aspects be considered in a production setting that we explore in this section.</p>\n<p>The throttled endpoint, <code>GET /api</code> rely only on API key for authorization for demonstration purpose. For any production implementation consider <a href=\"https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html\" target=\"_blank\">authentication</a> options for your REST APIs. You may explore and extend to require authentication with Cognito similar to <code>/admin/*</code> endpoints in the sample code.</p>\n<p>One API key for Free Tier access and five API keys for Basic Tier access are illustrative in a sample code but not representative of production deployments. Number of API keys with service quota into consideration, business and technical decisions may be made to minimize noisy neighbor effect such as setting blast radius upper threshold of 0.1% of all users. To satisfy that requirement, each tier would need to spread users across at least 1,000 API keys. The number of keys allocated to Basic or Premium Tier would depend on market needs and pricing strategies. Additional allocations of keys could be held in reserve for troubleshooting, QA, tenant migrations, and key retirement.</p>\n<p>In the planning phase of your solution, you will decide how many tiers to provide, how many usage plans are needed, and what throttle limits and quotas to apply. These decisions depend on your architecture and business.</p>\n<p>To define API request limits, examine the system API Gateway is protecting and what load it can sustain. For example, if your service will scale up to 1,000 requests per second, it is possible to implement three tiers with a 10/50/40 split: the lowest tier shares one common API key with a 100 request per second limit; an intermediate tier has a pool of 25 API keys with a limit of 20 requests per second each; and the highest tier has a maximum of 10 API keys, each supporting 40 requests per second.</p>\n<p>Metrics play a large role in continuously evolving your SaaS-tiering strategy (Figure 4). They provide rich insights into how tenants are using the system. Tenant-aware and <a href=\"https://youtu.be/j7Sqt8GpYl0?t=1692\" target=\"_blank\">SaaS-wide metrics</a> on throttling and quota limits can be used to: assess tiering in-place, if tenants’ requirements are being met, and if currently used tenant usage profiles are valid (Figure 5).</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/fc20235226ba47698f2ce271db84e7c6_image.png\" alt=\"image.png\" /></p>\n<p>Figure 4. Tiering strategy example with 3 tiers and requests allocation per tier</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/85fc040264dd4f3fbacc9ada108904db_image.png\" alt=\"image.png\" /></p>\n<p>Figure 5. An example SaaS metrics dashboard</p>\n<p>API Gateway provides options for different levels of granularity required, including <a href=\"https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html\" target=\"_blank\">detailed metrics</a>, and <a href=\"https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html\" target=\"_blank\">execution and access logging</a> to enable observability of your SaaS solution. Granular usage metrics combined with underlying resource consumption leads to managing optimal experience for your tenants with throttling levels and policies per method and per client.</p>\n<h3><a id=\"Cleanup_83\"></a><strong>Cleanup</strong></h3>\n<p>To avoid incurring future charges, delete the resources. This can be done on the command line by typing:</p>\n<p>XML</p>\n<pre><code class=\"lang-\">cd ${TOP}/cdk\ncdk destroy\n\ncd ${TOP}/react\namplify delete\n</code></pre>\n<p><code>${TOP}</code> is the topmost directory of the sample code. For the most up-to-date information, see the README.md file.</p>\n<h3><a id=\"Conclusion_96\"></a><strong>Conclusion</strong></h3>\n<p>In this two-part blog series, we have reviewed the best practices and challenges of effectively guarding a tiered multi-tenant REST API hosted in AWS API Gateway. We also explored how throttling policy and quota management can help you continuously evaluate the needs of your tenants and evolve your tiering strategy to protect your backend systems from being overwhelmed by inbound traffic.</p>\n<p>Further reading:</p>\n<ul>\n<li><a href=\"https://docs.aws.amazon.com/wellarchitected/latest/saas-lens/monitoring.html\" target=\"_blank\">AWS Well-Architected SaaS Lens Performance Efficiency pillar</a></li>\n<li><a href=\"https://catalog.us-east-1.prod.workshops.aws/workshops/b0c6ad36-0a4b-45d8-856b-8a64f0ac76bb/en-US\" target=\"_blank\">AWS Serverless SaaS Workshop</a></li>\n<li><a href=\"https://github.com/aws-samples/aws-saas-factory-ref-solution-serverless-saas\" target=\"_blank\">SaaS Factory Serverless SaaS reference solution</a></li>\n</ul>\n<p><em>This series was co-authored by Gary Kumfert, PhD, former Principal Solutions Architect at AWS.</em></p>\n<h4><a id=\"Nick_Choi_106\"></a><strong>Nick Choi</strong></h4>\n<p><img src=\"https://dev-media.amazoncloud.cn/8d051fd9758941b3908349f2178e311b_image.png\" alt=\"image.png\" /></p>\n<p>Nick Choi is a Solutions Architect with Amazon Web Services. He works with software and SaaS customers to design, build, operate, and evolve their SaaS service on AWS. He is passionate about understanding why and how SaaS businesses are changing way we live, work, and play.</p>\n"}