Preview: Amazon OpenSearch Serverless – Run Search and Analytics Workloads without Managing Clusters

{"value":"Most AWS analytics services have compelling serverless offerings that make it even easier for customers to analyze vast amounts of data without having to configure, scale, or manage the underlying infrastructure.\n\nAlong with other serverless analytics, such as [Amazon QuickSight](https://aws.amazon.com/quicksight/) for business intelligence and [AWS Glue](https://aws.amazon.com/glue) for data integration, we have introduced [Amazon EMR Serverless](https://aws.amazon.com/blogs/aws/amazon-emr-serverless-now-generally-available-run-big-data-applications-without-managing-servers/), [Amazon MSK Serverless](https://aws.amazon.com/blogs/aws/amazon-msk-serverless-now-generally-available-no-more-capacity-planning-for-your-managed-kafka-clusters/), and [Amazon Redshift Serverless](https://aws.amazon.com/blogs/aws/amazon-redshift-serverless-now-generally-available-with-new-capabilities/) this year.\n\nToday, we announce the preview release of a **new serverless option for** **[Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/)** that makes it easy for customers to run large-scale search and analytics workloads without managing clusters. It automatically provisions and scales the underlying resources to deliver fast data ingestion and query responses for even the most demanding and unpredictable workloads, eliminating the need to configure and optimize clusters.\n\nWith [Amazon OpenSearch Serverless](https://aws.amazon.com/opensearch-service/features/serverless), you do not need to account for factors that are hard to know in advance, such as the frequency and complexity of queries or the volume of data expected to be analyzed. Instead of managing infrastructure, you can focus on using OpenSearch for exploring and deriving insights from your data. You can also get started using familiar APIs to load and query data and use OpenSearch Dashboards for interactive data analysis and visualization.\n\n### **++Configure Your OpenSearch Serverless Collection++**\nTo get started with Amazon OpenSearch Serverless, you create a **Collection** via the [AWS Management Console](https://console.aws.amazon.com/esv3/), AWS Command-Line Interface (AWS CLI), or AWS API.\n\n\n\n\nBefore the launch of OpenSearch Serverless, you created a managed cluster, specifying instance types, counts, and storage options, and then managed the lifecycle and shard strategy for indices within that cluster. With OpenSearch Serverless, you create a Collection, which manages a group of indices that work together to support a specific workload. You no longer need to specify the hardware or manage the indices directly.\n\n\n\n\nTo create an OpenSearch Serverless collection and secure data, set up **Encryption policies** to assign AWS KMS keys to one or more collections and attach **Network policies** to collections to control the access from specified VPCs and public IP addresses.\n\n\n\nTo create an encryption policy, choose **Encryption policies** in the left navigation pane and **Create encryption policy**. Encryption at rest secures the indices within your collection. For each collection, AWS KMS generates a unique, symmetric encryption key. Encryption policies are the optimal way to manage AWS KMS keys across multiple collections. You can define the target collection name or a prefix that automatically applies the encryption settings from this policy to the collection.\n\n\n\n\nIn order for users to access a collection, choose **Network policies** in the left navigation pane and **Create network policy**. Network policies determine whether your collection is accessible over the internet from public networks or whether it must be accessed through OpenSearch Serverless–managed VPC endpoints.\n\n\n\n\nYou can define multiple rules for each collection, either the **Public** or **VPC**, as a recommended option for the **Access Type**. If you select a public option, you can access the collection from OpenSearch Dashboards.\n\nAlso, you can configure access for OpenSearch Dashboards and the OpenSearch endpoint. For the **Resource typ**e, enable both Access to OpenSearch endpoints and Access to OpenSearch Dashboards. In both input boxes, select the ```Collection Name```property and your collection name or prefix.\n\n\nFinally, to create an OpenSearch Serverless collection, choose **Create collection** in the home page or choose **Collections** in the left navigation pane and choose **Create collection**.\n\n\n\n\nInput your collection name, description, and collection type, either **Time series** or **Search** by your data type.\n\n- **Time series** – The log analytics segment that focuses on analyzing large volumes of semistructured, machine-generated data in real time for operational, security, user behavior, and business insights.\n- **Search** – Full-text search that powers applications in your internal networks (content management systems, legal documents) and internet-facing applications such as e-commerce website search and content search.\n\nWhen you choose **Create**, a collection typically takes less than a minute to initialize.\n\n### ++Upload and Search Data in Your Collection++\nBefore uploading and searching data in your collection, configure the IAM policy to access the actual data within a collection. Choose **Data access policies** in the left navigation pane and **Create data access policy**.\n\n\nYou can apply multiple policies simultaneously to the same resource. Each policy contains a set of rules. Each rule has a resource (collection or index), permissions for the resource, and a list of principals (IAM users, role ARNs, or SAML identities).\n\nHere is a sample policy that provides a single user the minimum permissions required to create an index in your collection, index some data, and search for it. Replace the principal ARN with the ARN of the account that you’ll use to sign in to OpenSearch Dashboards.\n\n\n```\n[\n {\n \"Rules\": [\n {\n \"ResourceType\": \"index\",\n \"Resource\": [\n \"index/books/*\"\n ],\n \"Permission\": [\n \"aoss:CreateIndex\",\n \"aoss:ReadDocument\",\n \"aoss:UpdateIndex\",\n \"aoss:DeleteIndex\",\n \"aoss:WriteDocument\"\n ]\n }\n ],\n \"Principal\": [\n \"arn:aws:iam::123456789012:user/admin\"\n ]\n }\n]\n```\n\nNow, you can upload data to an OpenSearch Serverless collection using Postman or curl. You can also use Dev Tools within the OpenSearch Dashboards console. Choose **OpenSearch Dashboards** on the detail page of your collection.\n\n\n\nSign in to OpenSearch Dashboards using the AWS access and secret keys for the principal that you specified in your data access policy. Within OpenSearch Dashboards, open the left navigation menu and choose **Dev Tools**.\n\nTo create a single index called ```books-index```, run ```PUT books-index```, and index your first single document into books-index.\n\n\n\nYou can also query search data in Dev Tools.\n\n```\nGET books-index/_search\n{\n \"query\": {\n \"simple_query_string\": {\n \"query\": \"Jeff\",\n \"fields\": [\"author\"]\n } \n }\n}\n```\n\nIn the case of time-series data, you can ingest data with all of the streaming ingestion options, such as native [OpenSearch streaming APIs](https://opensearch.org/docs/1.0/opensearch/data-streams/), [Amazon Kinesis Data Firehose](https://aws.amazon.com/kinesis/data-firehose/), [AWS Glue](https://aws.amazon.com/glue/), and a wide range of open-source streaming ingestion pipelines like Logstash, FluentBit, Fluentd, and Data Prepper.\n\nIn addition, you can snapshot your data from a managed cluster on OpenSearch Service and restore it to your collection, making it easy to migrate your workloads. Once your data is in your collection, you can then query it using your favorite OpenSearch client and interactively analyze and visualize your data using OpenSearch Dashboards.\n\n### ++Things to Know++\nHere are a couple of things to keep in mind about additional features and considerations when you choose Amazon OpenSearch Serverless:\n\n- **SAML Authentications** – You can use your existing identity provider to offer single sign-on (SSO) for the OpenSearch Dashboards endpoints of OpenSearch Serverless SAML authentication lets you use third-party identity providers to sign in to OpenSearch Dashboards to index and search data. OpenSearch Serverless supports providers that use the SAML 2.0 standard, such as Okta, Keycloak, Active Directory Federation Services, and Auth0.\n- **Private VPC Endpoints** – You can use [AWS PrivateLink](https://aws.amazon.com/privatelink/) to create a private connection between your VPC and OpenSearch Serverless. You can access your collections as if they were in your VPC without the use of an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection. To create an interface endpoint, choose **VPC endpoints** in the left navigation pane of OpenSearch Service.\n- **Managed Clusters** – You may prefer to use an option of Amazon OpenSearch Service’s managed clusters in scenarios where you need tight control over cluster configuration or specific customizations. For example, your workloads may need custom plugins that run best on accelerated computing instances and need more control on configuration such as data sharding strategy. You can choose either provisioned instances or serverless according to the requirements of your workload.\n\n### **++Join the Preview++**\nThe preview release of [Amazon OpenSearch Serverless](https://aws.amazon.com/opensearch-service/features/serverless) is now available in the US East (N. Virginia, Ohio), US West (Oregon), EU (Ireland), Asia Pacific (Tokyo). With OpenSearch Serverless, there are no upfront costs, and you pay only for the data that is ingest and the queries you run. For pricing details, see the [OpenSearch Service pricing page](https://aws.amazon.com/opensearch-service/pricing/). To learn more, visit the [Amazon OpenSearch Service User Guide](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html).\n\nWe want to hear more feedback during the preview. Please send feedback to [AWS re:Post for Amazon OpenSearch Service](https://repost.aws/tags/TA6VFzFFY6QQa_KlHRKR-WsA/amazon-open-search-service) or through your usual AWS support contacts.\n\n– [Channy](https://twitter.com/)\n\n\n\n### **[Channy Yun](https://aws.amazon.com/blogs/aws/author/channy-yun/)**\nChanny Yun is a Principal Developer Advocate for AWS, and passionate about helping developers to build modern applications on latest AWS services. A pragmatic developer and blogger at heart, he loves community-driven learning and sharing of technology, which has funneled developers to global AWS Usergroups. His main topics are open-source, container, storage, network & security, and IoT. Follow him on Twitter at @channyun.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","render":"<p>Most AWS analytics services have compelling serverless offerings that make it even easier for customers to analyze vast amounts of data without having to configure, scale, or manage the underlying infrastructure.</p>\n<p>Along with other serverless analytics, such as <a href=\"https://aws.amazon.com/quicksight/\" target=\"_blank\">Amazon QuickSight</a> for business intelligence and <a href=\"https://aws.amazon.com/glue\" target=\"_blank\">AWS Glue</a> for data integration, we have introduced <a href=\"https://aws.amazon.com/blogs/aws/amazon-emr-serverless-now-generally-available-run-big-data-applications-without-managing-servers/\" target=\"_blank\">Amazon EMR Serverless</a>, <a href=\"https://aws.amazon.com/blogs/aws/amazon-msk-serverless-now-generally-available-no-more-capacity-planning-for-your-managed-kafka-clusters/\" target=\"_blank\">Amazon MSK Serverless</a>, and <a href=\"https://aws.amazon.com/blogs/aws/amazon-redshift-serverless-now-generally-available-with-new-capabilities/\" target=\"_blank\">Amazon Redshift Serverless</a> this year.</p>\n<p>Today, we announce the preview release of a <strong>new serverless option for</strong> <strong><a href=\"https://aws.amazon.com/opensearch-service/\" target=\"_blank\">Amazon OpenSearch Service</a></strong> that makes it easy for customers to run large-scale search and analytics workloads without managing clusters. It automatically provisions and scales the underlying resources to deliver fast data ingestion and query responses for even the most demanding and unpredictable workloads, eliminating the need to configure and optimize clusters.</p>\n<p>With <a href=\"https://aws.amazon.com/opensearch-service/features/serverless\" target=\"_blank\">Amazon OpenSearch Serverless</a>, you do not need to account for factors that are hard to know in advance, such as the frequency and complexity of queries or the volume of data expected to be analyzed. Instead of managing infrastructure, you can focus on using OpenSearch for exploring and deriving insights from your data. You can also get started using familiar APIs to load and query data and use OpenSearch Dashboards for interactive data analysis and visualization.</p>\n<h3><a id=\"Configure_Your_OpenSearch_Serverless_Collection_8\"></a><strong><ins>Configure Your OpenSearch Serverless Collection</ins></strong></h3>\n<p>To get started with Amazon OpenSearch Serverless, you create a <strong>Collection</strong> via the <a href=\"https://console.aws.amazon.com/esv3/\" target=\"_blank\">AWS Management Console</a>, AWS Command-Line Interface (AWS CLI), or AWS API.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/f8f0dec9830a4b16a337e7cf2e31cfb1_image.png\" alt=\"image.png\" /></p>\n<p>Before the launch of OpenSearch Serverless, you created a managed cluster, specifying instance types, counts, and storage options, and then managed the lifecycle and shard strategy for indices within that cluster. With OpenSearch Serverless, you create a Collection, which manages a group of indices that work together to support a specific workload. You no longer need to specify the hardware or manage the indices directly.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/c851d62cceb04ed182569f2fbeb753a3_image.png\" alt=\"image.png\" /></p>\n<p>To create an OpenSearch Serverless collection and secure data, set up <strong>Encryption policies</strong> to assign AWS KMS keys to one or more collections and attach <strong>Network policies</strong> to collections to control the access from specified VPCs and public IP addresses.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/84f3c773eabf4af1a49c6167e5ccf801_image.png\" alt=\"image.png\" /></p>\n<p>To create an encryption policy, choose <strong>Encryption policies</strong> in the left navigation pane and <strong>Create encryption policy</strong>. Encryption at rest secures the indices within your collection. For each collection, AWS KMS generates a unique, symmetric encryption key. Encryption policies are the optimal way to manage AWS KMS keys across multiple collections. You can define the target collection name or a prefix that automatically applies the encryption settings from this policy to the collection.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/b5904a94d3f64737b60dfb90f547cbbd_image.png\" alt=\"image.png\" /></p>\n<p>In order for users to access a collection, choose <strong>Network policies</strong> in the left navigation pane and <strong>Create network policy</strong>. Network policies determine whether your collection is accessible over the internet from public networks or whether it must be accessed through OpenSearch Serverless–managed VPC endpoints.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/bbe3bf6e1dc34f8dbede3ce8cc9f3851_image.png\" alt=\"image.png\" /></p>\n<p>You can define multiple rules for each collection, either the <strong>Public</strong> or <strong>VPC</strong>, as a recommended option for the <strong>Access Type</strong>. If you select a public option, you can access the collection from OpenSearch Dashboards.</p>\n<p>Also, you can configure access for OpenSearch Dashboards and the OpenSearch endpoint. For the <strong>Resource typ</strong>e, enable both Access to OpenSearch endpoints and Access to OpenSearch Dashboards. In both input boxes, select the <code>Collection Name</code>property and your collection name or prefix.</p>\n<p>Finally, to create an OpenSearch Serverless collection, choose <strong>Create collection</strong> in the home page or choose <strong>Collections</strong> in the left navigation pane and choose <strong>Create collection</strong>.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/8d4c800a10454468814df08f5c32107d_image.png\" alt=\"image.png\" /></p>\n<p>Input your collection name, description, and collection type, either <strong>Time series</strong> or <strong>Search</strong> by your data type.</p>\n<ul>\n<li><strong>Time series</strong> – The log analytics segment that focuses on analyzing large volumes of semistructured, machine-generated data in real time for operational, security, user behavior, and business insights.</li>\n<li><strong>Search</strong> – Full-text search that powers applications in your internal networks (content management systems, legal documents) and internet-facing applications such as e-commerce website search and content search.</li>\n</ul>\n<p>When you choose <strong>Create</strong>, a collection typically takes less than a minute to initialize.</p>\n<h3><a id=\"Upload_and_Search_Data_in_Your_Collection_50\"></a><ins>Upload and Search Data in Your Collection</ins></h3>\n<p>Before uploading and searching data in your collection, configure the IAM policy to access the actual data within a collection. Choose <strong>Data access policies</strong> in the left navigation pane and <strong>Create data access policy</strong>.</p>\n<p>You can apply multiple policies simultaneously to the same resource. Each policy contains a set of rules. Each rule has a resource (collection or index), permissions for the resource, and a list of principals (IAM users, role ARNs, or SAML identities).</p>\n<p>Here is a sample policy that provides a single user the minimum permissions required to create an index in your collection, index some data, and search for it. Replace the principal ARN with the ARN of the account that you’ll use to sign in to OpenSearch Dashboards.</p>\n<pre><code class=\"lang-\">[\n {\n &quot;Rules&quot;: [\n {\n &quot;ResourceType&quot;: &quot;index&quot;,\n &quot;Resource&quot;: [\n &quot;index/books/*&quot;\n ],\n &quot;Permission&quot;: [\n &quot;aoss:CreateIndex&quot;,\n &quot;aoss:ReadDocument&quot;,\n &quot;aoss:UpdateIndex&quot;,\n &quot;aoss:DeleteIndex&quot;,\n &quot;aoss:WriteDocument&quot;\n ]\n }\n ],\n &quot;Principal&quot;: [\n &quot;arn:aws:iam::123456789012:user/admin&quot;\n ]\n }\n]\n</code></pre>\n<p>Now, you can upload data to an OpenSearch Serverless collection using Postman or curl. You can also use Dev Tools within the OpenSearch Dashboards console. Choose <strong>OpenSearch Dashboards</strong> on the detail page of your collection.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/0e60ff12a654408fa2eeaa1363ea34ab_image.png\" alt=\"image.png\" /></p>\n<p>Sign in to OpenSearch Dashboards using the AWS access and secret keys for the principal that you specified in your data access policy. Within OpenSearch Dashboards, open the left navigation menu and choose <strong>Dev Tools</strong>.</p>\n<p>To create a single index called <code>books-index</code>, run <code>PUT books-index</code>, and index your first single document into books-index.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/c293e3166be5428fa2575a444d638019_image.png\" alt=\"image.png\" /></p>\n<p>You can also query search data in Dev Tools.</p>\n<pre><code class=\"lang-\">GET books-index/_search\n{\n &quot;query&quot;: {\n &quot;simple_query_string&quot;: {\n &quot;query&quot;: &quot;Jeff&quot;,\n &quot;fields&quot;: [&quot;author&quot;]\n } \n }\n}\n</code></pre>\n<p>In the case of time-series data, you can ingest data with all of the streaming ingestion options, such as native <a href=\"https://opensearch.org/docs/1.0/opensearch/data-streams/\" target=\"_blank\">OpenSearch streaming APIs</a>, <a href=\"https://aws.amazon.com/kinesis/data-firehose/\" target=\"_blank\">Amazon Kinesis Data Firehose</a>, <a href=\"https://aws.amazon.com/glue/\" target=\"_blank\">AWS Glue</a>, and a wide range of open-source streaming ingestion pipelines like Logstash, FluentBit, Fluentd, and Data Prepper.</p>\n<p>In addition, you can snapshot your data from a managed cluster on OpenSearch Service and restore it to your collection, making it easy to migrate your workloads. Once your data is in your collection, you can then query it using your favorite OpenSearch client and interactively analyze and visualize your data using OpenSearch Dashboards.</p>\n<h3><a id=\"Things_to_Know_112\"></a><ins>Things to Know</ins></h3>\n<p>Here are a couple of things to keep in mind about additional features and considerations when you choose Amazon OpenSearch Serverless:</p>\n<ul>\n<li><strong>SAML Authentications</strong> – You can use your existing identity provider to offer single sign-on (SSO) for the OpenSearch Dashboards endpoints of OpenSearch Serverless SAML authentication lets you use third-party identity providers to sign in to OpenSearch Dashboards to index and search data. OpenSearch Serverless supports providers that use the SAML 2.0 standard, such as Okta, Keycloak, Active Directory Federation Services, and Auth0.</li>\n<li><strong>Private VPC Endpoints</strong> – You can use <a href=\"https://aws.amazon.com/privatelink/\" target=\"_blank\">AWS PrivateLink</a> to create a private connection between your VPC and OpenSearch Serverless. You can access your collections as if they were in your VPC without the use of an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection. To create an interface endpoint, choose <strong>VPC endpoints</strong> in the left navigation pane of OpenSearch Service.</li>\n<li><strong>Managed Clusters</strong> – You may prefer to use an option of Amazon OpenSearch Service’s managed clusters in scenarios where you need tight control over cluster configuration or specific customizations. For example, your workloads may need custom plugins that run best on accelerated computing instances and need more control on configuration such as data sharding strategy. You can choose either provisioned instances or serverless according to the requirements of your workload.</li>\n</ul>\n<h3><a id=\"Join_the_Preview_119\"></a><strong><ins>Join the Preview</ins></strong></h3>\n<p>The preview release of <a href=\"https://aws.amazon.com/opensearch-service/features/serverless\" target=\"_blank\">Amazon OpenSearch Serverless</a> is now available in the US East (N. Virginia, Ohio), US West (Oregon), EU (Ireland), Asia Pacific (Tokyo). With OpenSearch Serverless, there are no upfront costs, and you pay only for the data that is ingest and the queries you run. For pricing details, see the <a href=\"https://aws.amazon.com/opensearch-service/pricing/\" target=\"_blank\">OpenSearch Service pricing page</a>. To learn more, visit the <a href=\"https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html\" target=\"_blank\">Amazon OpenSearch Service User Guide</a>.</p>\n<p>We want to hear more feedback during the preview. Please send feedback to <a href=\"https://repost.aws/tags/TA6VFzFFY6QQa_KlHRKR-WsA/amazon-open-search-service\" target=\"_blank\">AWS re:Post for Amazon OpenSearch Service</a> or through your usual AWS support contacts.</p>\n<p>– <a href=\"https://twitter.com/\" target=\"_blank\">Channy</a></p>\n<p><img src=\"https://dev-media.amazoncloud.cn/369b9b98ecb749db967018da1aa79f2d_image.png\" alt=\"image.png\" /></p>\n<h3><a id=\"Channy_Yunhttpsawsamazoncomblogsawsauthorchannyyun_128\"></a><strong><a href=\"https://aws.amazon.com/blogs/aws/author/channy-yun/\" target=\"_blank\">Channy Yun</a></strong></h3>\n<p>Channy Yun is a Principal Developer Advocate for AWS, and passionate about helping developers to build modern applications on latest AWS services. A pragmatic developer and blogger at heart, he loves community-driven learning and sharing of technology, which has funneled developers to global AWS Usergroups. His main topics are open-source, container, storage, network &amp; security, and IoT. Follow him on Twitter at @channyun.</p>\n"}