New for Amazon SageMaker – Perform Shadow Tests to Compare Inference Performance Between ML Model Variants

海外精选
re:Invent
Amazon SageMaker
海外精选的内容汇集了全球优质的亚马逊云科技相关技术内容。同时,内容中提到的“AWS” 是 “Amazon Web Services” 的缩写,在此网站不作为商标展示。
0
0
{"value":"As you move your machine learning (ML) workloads into production, you need to continuously monitor your deployed models and iterate when you observe a deviation in your model performance. When you build a new model, you typically start validating the model offline using historical inference request data. But this data sometimes fails to account for current, real-world conditions. For example, new products might become trending that your product recommendation model hasn’t seen yet. Or, you experience a sudden spike in the volume of inference requests in production that you never exposed your model to before.\n\nToday, I’m excited to announce **Amazon SageMaker [support for shadow testing!](https://aws.amazon.com/sagemaker/shadow-testing/)**\n\nDeploying a model in shadow mode lets you conduct a more holistic test by routing a copy of the live inference requests for a production model to the new (shadow) model. Yet, only the responses from the production model are returned to the calling application. Shadow testing helps you build further confidence in your model and catch potential configuration errors and performance issues before they impact end users. Once you complete a shadow test, you can use the [deployment guardrails](https://docs.aws.amazon.com/sagemaker/latest/dg/deployment-guardrails.html) for SageMaker inference endpoints to safely update your model in production.\n\n### ++Get Started with Amazon SageMaker Shadow Testing++\nYou can create shadow tests using the new SageMaker Inference Console and APIs. Shadow testing gives you a fully managed experience for setup, monitoring, viewing, and acting on the results of shadow tests. If you have existing workflows built around SageMaker endpoints, you can also deploy a model in shadow mode using the existing SageMaker Inference APIs.\n\nOn the SageMaker console, select **Inference** and **Shadow tests** to create, monitor, and deploy shadow tests.\n\n![image.png](https://dev-media.amazoncloud.cn/b0cb676a63864116b923e6f3823a9351_image.png)\n\nTo create a shadow test, select an existing (or create a new) SageMaker endpoint and production variant you want to test against.\n\n![image.png](https://dev-media.amazoncloud.cn/701941592bd74c5fa15a85814f3037cf_image.png)\n\nNext, configure the proportion of traffic to send to the shadow variant, the comparison metrics you want to evaluate, and the duration of the test. You can also enable data capture for your production and shadow variant.\n\n![image.png](https://dev-media.amazoncloud.cn/99b8f551e2af46b18993603c7e413368_image.png)\n\nThat’s it. SageMaker now automatically deploys the new variant in shadow mode and routes a copy of the inference requests to it in real time, all within the same endpoint. The following diagram illustrates this workflow.\n\n![image.png](https://dev-media.amazoncloud.cn/fef9d30c47b2406082d8bac746902479_image.png)\n\nNote that only the responses of the production variant are returned to the calling application. You can choose to either discard or log the responses of the shadow variant for offline comparison.\n\nYou can also use shadow testing to validate changes you made to any component in your production variant, including the serving container or ML instance. This can be useful when you’re upgrading to a new framework version of your serving container, applying patches, or if you want to make sure that there is no impact to latency or error rate due to this change. Similarly, if you consider moving to another ML instance type, for example, [Amazon EC2 C7g instances](https://aws.amazon.com/ec2/instance-types/c7g/) based on [AWS Graviton processors](https://aws.amazon.com/ec2/graviton/), or [EC2 G5 instances](https://aws.amazon.com/ec2/instance-types/g5/) powered by NVIDIA A10G Tensor Core GPUs, you can use shadow testing to evaluate the performance on production traffic prior to rollout.\n\nYou can monitor the progress of the shadow test and performance metrics such as latency and error rate through a live dashboard. On the SageMaker console, select **Inference** and **Shadow tests**, then select the shadow test you want to monitor.\n\n![image.png](https://dev-media.amazoncloud.cn/bcadeb88e3f1449e940342d6caf0de86_image.png)\n\n![image.png](https://dev-media.amazoncloud.cn/e2c5ae594dc44c1da531054761fbcfb6_image.png)\n\n\nIf you decide to promote the shadow model to production, select **Deploy shadow variant** and define the infrastructure configuration to deploy the shadow variant.\n\n![image.png](https://dev-media.amazoncloud.cn/807f35ed7edd465e89c8fe1fc50824e3_image.png)\n\n![image.png](https://dev-media.amazoncloud.cn/470db99918514f75bc0f1f91c1608ccd_image.png)\n\n\nYou can also use the SageMaker deployment guardrails if you want to add linear or canary traffic shifting modes and auto rollbacks to your update.\n\n### ++Availability and Pricing++\nSageMaker support for shadow testing is available today in all [AWS Regions](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/) where SageMaker hosting is available except for the AWS GovCloud (US) Regions and AWS China Regions.\n\nThere is no additional charge for SageMaker shadow testing other than usage charges for the ML instances and ML storage provisioned to host the shadow variant. The pricing for ML instances and ML storage dimensions is the same as the real-time inference option. There is no additional charge for data processed in and out of shadow deployments. The SageMaker [pricing page](https://aws.amazon.com/sagemaker/pricing) has all the details.\n\nTo learn more, visit [Amazon SageMaker shadow testing](https://aws.amazon.com/sagemaker/shadow-testing/).\n\n**[Start validating your new ML models with SageMaker shadow tests today!](https://console.aws.amazon.com/sagemaker)**\n\n— [Antje](https://twitter.com/anbarth)\n\n![image.png](https://dev-media.amazoncloud.cn/036397a36876438faf7aa73825152f29_image.png)\n\n### Antje Barth\nAntje Barth is a Principal Developer Advocate for AI and ML at AWS. She is co-author of the O’Reilly book – Data Science on AWS. Antje frequently speaks at AI/ML conferences, events, and meetups around the world. She also co-founded the Düsseldorf chapter of Women in Big Data.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","render":"<p>As you move your machine learning (ML) workloads into production, you need to continuously monitor your deployed models and iterate when you observe a deviation in your model performance. When you build a new model, you typically start validating the model offline using historical inference request data. But this data sometimes fails to account for current, real-world conditions. For example, new products might become trending that your product recommendation model hasn’t seen yet. Or, you experience a sudden spike in the volume of inference requests in production that you never exposed your model to before.</p>\n<p>Today, I’m excited to announce <strong>Amazon SageMaker <a href=\"https://aws.amazon.com/sagemaker/shadow-testing/\" target=\"_blank\">support for shadow testing!</a></strong></p>\n<p>Deploying a model in shadow mode lets you conduct a more holistic test by routing a copy of the live inference requests for a production model to the new (shadow) model. Yet, only the responses from the production model are returned to the calling application. Shadow testing helps you build further confidence in your model and catch potential configuration errors and performance issues before they impact end users. Once you complete a shadow test, you can use the <a href=\"https://docs.aws.amazon.com/sagemaker/latest/dg/deployment-guardrails.html\" target=\"_blank\">deployment guardrails</a> for SageMaker inference endpoints to safely update your model in production.</p>\n<h3><a id=\"Get_Started_with_Amazon_SageMaker_Shadow_Testing_6\"></a><ins>Get Started with Amazon SageMaker Shadow Testing</ins></h3>\n<p>You can create shadow tests using the new SageMaker Inference Console and APIs. Shadow testing gives you a fully managed experience for setup, monitoring, viewing, and acting on the results of shadow tests. If you have existing workflows built around SageMaker endpoints, you can also deploy a model in shadow mode using the existing SageMaker Inference APIs.</p>\n<p>On the SageMaker console, select <strong>Inference</strong> and <strong>Shadow tests</strong> to create, monitor, and deploy shadow tests.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/b0cb676a63864116b923e6f3823a9351_image.png\" alt=\"image.png\" /></p>\n<p>To create a shadow test, select an existing (or create a new) SageMaker endpoint and production variant you want to test against.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/701941592bd74c5fa15a85814f3037cf_image.png\" alt=\"image.png\" /></p>\n<p>Next, configure the proportion of traffic to send to the shadow variant, the comparison metrics you want to evaluate, and the duration of the test. You can also enable data capture for your production and shadow variant.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/99b8f551e2af46b18993603c7e413368_image.png\" alt=\"image.png\" /></p>\n<p>That’s it. SageMaker now automatically deploys the new variant in shadow mode and routes a copy of the inference requests to it in real time, all within the same endpoint. The following diagram illustrates this workflow.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/fef9d30c47b2406082d8bac746902479_image.png\" alt=\"image.png\" /></p>\n<p>Note that only the responses of the production variant are returned to the calling application. You can choose to either discard or log the responses of the shadow variant for offline comparison.</p>\n<p>You can also use shadow testing to validate changes you made to any component in your production variant, including the serving container or ML instance. This can be useful when you’re upgrading to a new framework version of your serving container, applying patches, or if you want to make sure that there is no impact to latency or error rate due to this change. Similarly, if you consider moving to another ML instance type, for example, <a href=\"https://aws.amazon.com/ec2/instance-types/c7g/\" target=\"_blank\">Amazon EC2 C7g instances</a> based on <a href=\"https://aws.amazon.com/ec2/graviton/\" target=\"_blank\">AWS Graviton processors</a>, or <a href=\"https://aws.amazon.com/ec2/instance-types/g5/\" target=\"_blank\">EC2 G5 instances</a> powered by NVIDIA A10G Tensor Core GPUs, you can use shadow testing to evaluate the performance on production traffic prior to rollout.</p>\n<p>You can monitor the progress of the shadow test and performance metrics such as latency and error rate through a live dashboard. On the SageMaker console, select <strong>Inference</strong> and <strong>Shadow tests</strong>, then select the shadow test you want to monitor.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/bcadeb88e3f1449e940342d6caf0de86_image.png\" alt=\"image.png\" /></p>\n<p><img src=\"https://dev-media.amazoncloud.cn/e2c5ae594dc44c1da531054761fbcfb6_image.png\" alt=\"image.png\" /></p>\n<p>If you decide to promote the shadow model to production, select <strong>Deploy shadow variant</strong> and define the infrastructure configuration to deploy the shadow variant.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/807f35ed7edd465e89c8fe1fc50824e3_image.png\" alt=\"image.png\" /></p>\n<p><img src=\"https://dev-media.amazoncloud.cn/470db99918514f75bc0f1f91c1608ccd_image.png\" alt=\"image.png\" /></p>\n<p>You can also use the SageMaker deployment guardrails if you want to add linear or canary traffic shifting modes and auto rollbacks to your update.</p>\n<h3><a id=\"Availability_and_Pricing_45\"></a><ins>Availability and Pricing</ins></h3>\n<p>SageMaker support for shadow testing is available today in all <a href=\"https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/\" target=\"_blank\">AWS Regions</a> where SageMaker hosting is available except for the AWS GovCloud (US) Regions and AWS China Regions.</p>\n<p>There is no additional charge for SageMaker shadow testing other than usage charges for the ML instances and ML storage provisioned to host the shadow variant. The pricing for ML instances and ML storage dimensions is the same as the real-time inference option. There is no additional charge for data processed in and out of shadow deployments. The SageMaker <a href=\"https://aws.amazon.com/sagemaker/pricing\" target=\"_blank\">pricing page</a> has all the details.</p>\n<p>To learn more, visit <a href=\"https://aws.amazon.com/sagemaker/shadow-testing/\" target=\"_blank\">Amazon SageMaker shadow testing</a>.</p>\n<p><strong><a href=\"https://console.aws.amazon.com/sagemaker\" target=\"_blank\">Start validating your new ML models with SageMaker shadow tests today!</a></strong></p>\n<p>— <a href=\"https://twitter.com/anbarth\" target=\"_blank\">Antje</a></p>\n<p><img src=\"https://dev-media.amazoncloud.cn/036397a36876438faf7aa73825152f29_image.png\" alt=\"image.png\" /></p>\n<h3><a id=\"Antje_Barth_58\"></a>Antje Barth</h3>\n<p>Antje Barth is a Principal Developer Advocate for AI and ML at AWS. She is co-author of the O’Reilly book – Data Science on AWS. Antje frequently speaks at AI/ML conferences, events, and meetups around the world. She also co-founded the Düsseldorf chapter of Women in Big Data.</p>\n"}
目录
亚马逊云科技解决方案 基于行业客户应用场景及技术领域的解决方案
联系亚马逊云科技专家
亚马逊云科技解决方案
基于行业客户应用场景及技术领域的解决方案
联系专家
0
目录
关闭
contact-us