Open source news and updates #90

{"value":"##### **November 15th, 2021 - Instalment #90**\nNewsletter #90.\n\nAnother packed edition of his newsletter, with some great new open source projects such as ploomber (a project to help you build production pipelines for your notebooks), slic-starter (a complete starter project for production-grade serverless applications) and many more interesting projects covering RStudio, Spring Boot, a serverless software vending solution, Kubernetes on the edge and more. Amazon Web Services and Community content this week covers Amazon Web Services CDK, Steampipe, Amazon Distro for OpenTelemetry, Dask, Flagger, Cortex, Apache Airflow, Apache Flink, OpenSearch, Reddis, RabbitMQ and more. Make sure you check out the videos too, this week featuring open source Observability and OpenSearch.\n\n##### **Celebrating open source contributors**\n\nThe articles posted in this series are only possible thanks to contributors and project maintainers and so I would like to shout out and thank those folks who really do power open source and enable us all to build on top of what they have created.\n\nSo thank you to the following open source heroes: Azmi Mengü, Pawel Grzybek, Jozef Jaroščiak, fourTheroem, Kyle Davis, Dotan Horovits, Matt Hanson, Ariel Walcutt, Imtranur Rahman, Mary-Elizabeth Morin, Kinnar Sen, Nathaniel Ruiz Nowell, David Boeke, Bob Tordella, Jon Udell, Behram Irani, Madhav Vishnubhatta, Ethan Fahy, Zac Flamig, Nima Kaviani, Manabu McCloskey, Nicholas Thomson, Tom Zach, Kevin Bell, Stefan Prodan, Ramesh Sencha, Nate Bachmeier, Wallace Printz, Christian Mueller, Alex Pulver, Nick Lynch, Dzenan Softic, Sam Dengler, James Greenwood, Bikash Behera, and Kevin Higgins.\n\nMake sure you find and follow these builders and keep up to date with their open source projects and contributions.\n\n##### **Community noticeboard**\n###### **Research Papers**\n\nCheck out the latest research papers released from Amazon focused on natural-language understanding and question answering are areas of focus, with additional topics ranging from self-learning to text summarisation. You can check these out on the post, ++[Amazon’s 23 papers at EMNLP](https://aws-oss.beachgeek.co.uk/13g)++\n\n###### **Amazon Web Services CDK**\n\nIf you are looking for examples CDK projects, then Amazon Web Services Community Builder Azmi Mengü has plenty of good examples you can get inspiration from. Head on over to ++[his GitHub repos](https://aws-oss.beachgeek.co.uk/13f)++ and see for yourself.\n\n##### **Latest open source projects**\n###### **slic-starter**\n\n++[slic-starter](https://aws-oss.beachgeek.co.uk/136)++ Whilst I was watching the great sessions during Amazon Web Services Community Day, I came across this project from fourTheorem. SLIC Starter is a complete starter project for production-grade serverless applications on Amazon Web Services.\n\n\n\n###### **lambda-runtimes-stack**\n\n++[lambda-runtimes-stack](https://aws-oss.beachgeek.co.uk/135)++ Pawel Grzybek has put together this project that provides detailed information about the Amazon Web Services Lambda runtimes versions. Deploy it and every few hours it will show record details of important version information about the Amazon Web Services Lambda runtimes.\n\n###### **Amazon Web Services-fargate-with-rstudio-open-source**\n\n++[amazon-fargate-with-rstudio-open-source](https://aws-oss.beachgeek.co.uk/134)++ this project delivers Amazon CDK Python code to provision serverless infrastructure in Amazon Web Services Cloud to run Open Source RStudio Server and Shiny. This is pretty nice, so make sure you bookmark this. I am also going through the code and going to re-use parts of it for other open source projects too.\n\n\n\n###### **ploomber**\n\n++[ploomber](https://aws-oss.beachgeek.co.uk/13a)++ this project helps you with the challenges of working with .ipynb files allowing teams to develop collaborative, production-ready pipelines using JupyterLab or any text editor. This project is integrated with Amazon Web Services so you can leverage it in your environments to develop locally, and then deploy to the cloud.\n\n##### **Amazon Web Services-Connect-Manager**\n\n++[Amazon Web Services-Connect-Manager](https://aws-oss.beachgeek.co.uk/137)++ this project from Jozef Jaroščiak provides Windows users with an easy to use tool to connect with your Amazon EC2 instances.\n\n\n\n##### **Amazon Web Services-kubernetes-edge-greengrass**\n\n++[amazon-kubernetes-edge-greengrass](https://aws-oss.beachgeek.co.uk/13b)++ this project demonstrates how to collect data from edge locations using Kubernetes (k3s) and Amazon Web Services IoT Greengrass. The code shows you how to get up and running on a Raspberry Pi, and uses Amazon Web Services CDK - two of my favourite things.\n\n\n##### **Amazon Web Services-iam-temporary-elevated-access-broker**\n\n++[Amazon Web Services-iam-temporary-elevated-access-broker](https://aws-oss.beachgeek.co.uk/13e)++ this projects provides a minimal reference implementation for providing temporary elevated access to your Amazon Web Services environment. James Greenwood, Bikash Behera, and Kevin Higgins walk you through how to use this project in the blog post, ++[Managing temporary elevated access to your Amazon Web Services environment](https://aws-oss.beachgeek.co.uk/13l)++\n\n\n\n##### **java-spring-boot-ecs-fargate-redis-caching**\n\n++[java-spring-boot-ecs-fargate-redis-caching](https://aws-oss.beachgeek.co.uk/13h)++ this sample shows how a Spring Boot application can be deployed on ECS Fargate with Redis Caching Support to persist session data and improve application performance. The solution also provides the necessary templates to automatically provision the required infrastructure using CloudFormation to your desired Amazon Web Services account.\n\n\n\n##### **Amazon Web Services-severless-vending-machine**\n\n++[Amazon Web Services-severless-vending-machine](https://aws-oss.beachgeek.co.uk/13i)++ this repository shows you how to create a solution to developer teams consume IaC Products, like a \"Vending Machine\", to automated deploy cloud service in their Amazon Web Services Accounts without open a ticket or request in IT department. You can read the blog post ++[Criando um portal de autosserviços para o provisionamento na Amazon Web Services](https://aws-oss.beachgeek.co.uk/13j)++ (in Portugese, but I tried translating with Amazon Translate and it did a great job)\n\n\n\n##### **amazon-connect-twitter-listener**\n\n++[amazon-connect-twitter-listener](https://aws-oss.beachgeek.co.uk/13k)++ I was recently chatting with a builder who was trying to implement social media integration with Amazon Connect, so I was pleasantly surprised when this project popped up on my open source radar. This project contains source code and supporting files for twitter streaming to Amazon Connect Tasks. Tweets are filtered on the specified tag and analyzed using comprehend to identify language and sentiment. Both tags are added as attributes to the task so additional branching decisions can be made. You need to set up your Amazon Connect project up first, but details are in the README\n\n\n\n#### **Amazon Web Services and Community blog posts**\n##### **Amazon Web Services CDK**\n\nA couple of good Amazon CDK posts this week, starting off with ++[Comparing Amazon Cloud Development Kit and Amazon Web Services Controllers for Kubernetes](https://aws.amazon.com/blogs/opensource/comparing-aws-cloud-development-kit-and-aws-controllers-for-kubernetes/)++ in which Nima Kaviani, Manabu McCloskey, and Nicholas Thomson come together to compare strategies, tools and technologies to automate infrastructure and software rollout strategies when working with Kubernetes. They look at six areas of focus: Programming, Validation, Execution, Interaction, Operation, and Reconciliation.\n\nA must read post from Alex Pulver and Nick Lynch is their post, ++[Experimental construct libraries are now available in Amazon Web Services CDK v2](https://aws-oss.beachgeek.co.uk/133)++ where they share an update on the Amazon Web Services CDK v2 experimental APIs, and how this can make it even easier to migrate from v1.\n\n##### **Steampipe**\n\nSteampipe is an open source tool for querying cloud APIs in a universal way and reasoning about the data in SQL. David Boeke, Bob Tordella, Jon Udell, and Nathan Wallace have collaborated together on this post, ++[Querying Amazon Web Services at scale across APIs, Regions, and accounts](https://aws-oss.beachgeek.co.uk/12u)++ where they share a number of different approaches for how you can use this project to deliver the best possible speed and throughput. [hands on]\n\n\n\n\n##### **Amazon Web Services Distro for OpenTelemetry**\n\nNathaniel Ruiz Nowell explains how to get telemetry data from Amazon Web Services Lambda Python functions, without having to change a line of code, in his post ++[Auto-instrumenting a Python application with an Amazon Web Services Distro for OpenTelemetry Lambda layer](https://aws-oss.beachgeek.co.uk/12t)++ [hands on]\n\n\n\nWe also had with the post, ++[How To Use OpenTelemetry With Amazon Web Services Lambda](https://aws-oss.beachgeek.co.uk/12z)++ from Tom Zach at Aspecto, that also shows you how to deploy a tracing-enabled lambda with OpenTelemetry.\n\n\n\nTo wrap up things in ADOT we had Alolita Sharma write ++[Amazon Web Services Distro for OpenTelemetry 0.14 is now available with updated Lambda layers](https://aws-oss.beachgeek.co.uk/138)++, providing the release highlights and details on how you can download this image.\n\n##### **Flagger**\n\nFlagger is a CNCF incubating project that takes care of much of the undifferentiated heavy lifting of canary deployments on Kubernetes. In the post ++[Performing canary deployments and metrics-driven rollback with Amazon Managed Service for Prometheus and Flagger](https://aws-oss.beachgeek.co.uk/130)++ Kevin Bell and Stefan Prodan explain how to perform canary deployments on Kubernetes using Flagger to orchestrate the rollout, promotion, and rollback of deployments. Taking care of metrics we have Amazon Managed Service for Prometheus, and letting you observe the canary deployment process, we have Amazon Managed Grafana. Nice! [hands on]\n\n\n\n##### **Cortex**\n\nIn the post, ++[Scaling Cortex with parallel compaction](https://aws-oss.beachgeek.co.uk/12y)++ Albert Choi, an intern on the Amazon Managed Service for Prometheus team, shares his experience of designing and implementing parallel compactors inside of the Cortex open source project.\n\n\n\n##### **Dask and Open Data**\n\nThis week we have a couple of interesting posts on how to analyse open data sets using an open source project called Dask (an open source project in Python that provides parallelism for analytics). Combining open data sets together with open source software running on Amazon Web Services is a growing trend to help provide insights that can help make the lives of emergency managers, climate scientists, and policy makers that much easier.\n\nFirst up we have the post, ++[Transforming Geospatial Data to Cloud-Native Frameworks with Element 84 on Amazon Web Services](https://aws-oss.beachgeek.co.uk/12r)++, where Matt Hanson and Ariel Walcutt from Element 84\nand Imtranur Rahman and Mary-Elizabeth Morin from Amazon Web Services walk you through a solution you can put together than combines open data sets from Sentinel-2 (satellite imagery) with Dask, and a number of Python libraries to analyse and visualise the data. [hands on]\n\n\n\nFollowing that we have ++[Analyze terabyte-scale geospatial datasets with Dask and Jupyter on Amazon Web Services](https://aws-oss.beachgeek.co.uk/12w)++, where Ethan Fahy and Zac Flamig show you how to set up this Kubernetes, Dask and Jupyter Pangeo solution on Amazon Web Services to automatically scale your cloud compute resources and parallelise your workload across multiple Dask worker nodes. In this example they show you how you can run an example notebook that performs sample analyses on the Coupled Model Intercomparison Project 6 (CMIP6) dataset from the Registry of Open Data on Amazon Web Services. [hands on]\n\n\n\n##### **Apache Airflow**\n\nDzenan Softic and Sam Dengler demonstrate how to use a CodeArtifact repository to load your Python libraries in Apache Airflow in the post, ++[Amazon MWAA with Amazon Web Services CodeArtifact for Python dependencies](https://aws-oss.beachgeek.co.uk/139)++\n\n\n\n##### **Apache Flink**\n\nIn this post, ++[Optimizing Apache Flink on Amazon EKS using Amazon EC2 Spot Instances](https://aws-oss.beachgeek.co.uk/12s)++ Kinnar Sen shows you how to deploy a scalable, highly available, resilient, and cost optimised Flink application using Kubernetes via Amazon EKS and Amazon EC2 Spot Instances. [hands on]\n\n\n\n##### **OpenSearch**\n\nIf you have a use case where you need to be able to run a single query against many different data sources, then you will find this post from Behram Irani and Madhav Vishnubhatta will be of interest. Using a capability of Amazon Athena called Amazon Athena Federated Queries, you can use an existing connector for OpenSearch to query your data in both OpenSearch and (in this example) Amazon S3. To find out more, read the post ++[Query data in Amazon OpenSearch Service using SQL from Amazon Athena](https://aws-oss.beachgeek.co.uk/12v)++ [hands on]\n\n\n\n##### **RabbitMQ**\n\nAmazon MQ provides two managed broker deployment connection options: public brokers and private brokers.\nNate Bachmeier, Wallace Printz, and Christian Mueller have come together to share how to provision private Amazon MQ brokers behind a secure public load balancer endpoint using an example subdomain in the blog post, ++[Creating static custom domain endpoints with Amazon MQ for RabbitMQ ](https://aws-oss.beachgeek.co.uk/132)++[hands on]\n\n\n\n##### **Redis**\n\nThis post from Ramesh Sencha at Coupa, ++[How Coupa migrated from a self-hosted Redis to fully managed Amazon ElastiCache](https://aws-oss.beachgeek.co.uk/131)++ details why Coupa migrated self managed Redis workloads to Amazon ElastiCache; the benefits, the approach and the architecture they chose.\n\n#### **Quick updates**\n##### **Fluent Bit**\n\nAmazon Elastic Kubernetes Service (EKS) on Fargate now supports the use of Kubernetes Fluent Bit filters which provide enriched Kubernetes-specific metadata to Fluent Bit logs. Customers can now more easily observe and troubleshoot their applications by using the Kubernetes pod, container, or namespace name, among other Kubernetes metadata, to associate with their applications’ logs. Without support for the Kubernetes filter for Fluent Bit on EKS Fargate, customers had to manually read through log files to find the events they were interested in and it was difficult to associate a given log line with a Kubernetes application. Some customers who needed this kind of observability were unable to use Fargate with EKS and couldn’t benefit from its fully managed compute environment.\n\n#### **Video of the week**\n##### **OpenSearch**\n\nIn this recording from the Amazon Web Services Summit, Kyle Davis will give you an understanding of the capabilities of OpenSearch, the first steps to get started (on or off Amazon Web Services), a picture of how OpenSearch fits in your stack, a look into what’s in store for the future of the project as well as a guide to join the community and contribute.\n\n<video src=\"https://dev-media.amazoncloud.cn/56803396e1964778b1c0f653524a6368_AWS%20AMER%20Summit%20Aug%202021%EF%BC%9A%20The%20top%20five%20things%20to%20know%20about%20OpenSearch.mp4\" class=\"manvaVedio\" controls=\"controls\" style=\"width:160px;height:160px\"></video>\n\n\n##### **Open source observability**\n\nAnother great video from Dotan Horovits over at Logz.io. In this session he introduces observability, the role of open source in this area and then shows you some of the tools and projects in this space. Well worth 41 minutes of your time.\n\n<video src=\"https://dev-media.amazoncloud.cn/93f3390882694d06992e25195134a037_Open%20Source%20for%20Better%20Observability%20-%20Dotan%20Horovits%2C%20Logz.io.mp4\" class=\"manvaVedio\" controls=\"controls\" style=\"width:160px;height:160px\"></video>\n\n#### **Events for your diary**\n##### **Databricks | Amazon Web Services Lakehouse Dev Day Live Workshop**\n##### **November 16th 9:00 AM PT**\n\nDelta Lake is an open source storage layer that provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. You can use Delta Lake on top of your existing data lake. During this workshop you will learn how to:\n\n- Make your existing Amazon S3 data lakes into a lakehouse with Delta Lake.\n- Provide an easy-to-use platform for analysts to directly query data on your data lake using SQL Analytics\n- Simplify and automate data pipelines for streaming and batch data to lower costs and boost productivity for your data teams\n\n++[Read more and sign up here](https://aws-oss.beachgeek.co.uk/zs)++\n\n##### **Stay in touch with open source at Amazon Web Services**\nI hope this summary has been useful. Remember to check out the ++[Open Source homepage](https://aws.amazon.com/opensource/?opensource-all.sort-by=item.additionalFields.startDate&opensource-all.sort-order=asc)++ to keep up to date with all our activity in open source by following us on ++[@Amazon Web ServicesOpen](https://twitter.com/AWSOpen)++\n\n\n","render":"<h5><a id=\"November_15th_2021__Instalment_90_0\"></a><strong>November 15th, 2021 - Instalment #90</strong></h5>\n<p>Newsletter #90.</p>\n<p>Another packed edition of his newsletter, with some great new open source projects such as ploomber (a project to help you build production pipelines for your notebooks), slic-starter (a complete starter project for production-grade serverless applications) and many more interesting projects covering RStudio, Spring Boot, a serverless software vending solution, Kubernetes on the edge and more. Amazon Web Services and Community content this week covers Amazon Web Services CDK, Steampipe, Amazon Distro for OpenTelemetry, Dask, Flagger, Cortex, Apache Airflow, Apache Flink, OpenSearch, Reddis, RabbitMQ and more. Make sure you check out the videos too, this week featuring open source Observability and OpenSearch.</p>\n<h5><a id=\"Celebrating_open_source_contributors_5\"></a><strong>Celebrating open source contributors</strong></h5>\n<p>The articles posted in this series are only possible thanks to contributors and project maintainers and so I would like to shout out and thank those folks who really do power open source and enable us all to build on top of what they have created.</p>\n<p>So thank you to the following open source heroes: Azmi Mengü, Pawel Grzybek, Jozef Jaroščiak, fourTheroem, Kyle Davis, Dotan Horovits, Matt Hanson, Ariel Walcutt, Imtranur Rahman, Mary-Elizabeth Morin, Kinnar Sen, Nathaniel Ruiz Nowell, David Boeke, Bob Tordella, Jon Udell, Behram Irani, Madhav Vishnubhatta, Ethan Fahy, Zac Flamig, Nima Kaviani, Manabu McCloskey, Nicholas Thomson, Tom Zach, Kevin Bell, Stefan Prodan, Ramesh Sencha, Nate Bachmeier, Wallace Printz, Christian Mueller, Alex Pulver, Nick Lynch, Dzenan Softic, Sam Dengler, James Greenwood, Bikash Behera, and Kevin Higgins.</p>\n<p>Make sure you find and follow these builders and keep up to date with their open source projects and contributions.</p>\n<h5><a id=\"Community_noticeboard_13\"></a><strong>Community noticeboard</strong></h5>\n<h6><a id=\"Research_Papers_14\"></a><strong>Research Papers</strong></h6>\n<p>Check out the latest research papers released from Amazon focused on natural-language understanding and question answering are areas of focus, with additional topics ranging from self-learning to text summarisation. You can check these out on the post, <ins><a href=\"https://aws-oss.beachgeek.co.uk/13g\" target=\"_blank\">Amazon’s 23 papers at EMNLP</a></ins></p>\n<h6><a id=\"Amazon_Web_Services_CDK_18\"></a><strong>Amazon Web Services CDK</strong></h6>\n<p>If you are looking for examples CDK projects, then Amazon Web Services Community Builder Azmi Mengü has plenty of good examples you can get inspiration from. Head on over to <ins><a href=\"https://aws-oss.beachgeek.co.uk/13f\" target=\"_blank\">his GitHub repos</a></ins> and see for yourself.</p>\n<h5><a id=\"Latest_open_source_projects_22\"></a><strong>Latest open source projects</strong></h5>\n<h6><a id=\"slicstarter_23\"></a><strong>slic-starter</strong></h6>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/136\" target=\"_blank\">slic-starter</a></ins> Whilst I was watching the great sessions during Amazon Web Services Community Day, I came across this project from fourTheorem. SLIC Starter is a complete starter project for production-grade serverless applications on Amazon Web Services.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/517c497497eb4fa7a3ffba66cdac545b_image.png\" alt=\"image.png\" /></p>\n<h6><a id=\"lambdaruntimesstack_29\"></a><strong>lambda-runtimes-stack</strong></h6>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/135\" target=\"_blank\">lambda-runtimes-stack</a></ins> Pawel Grzybek has put together this project that provides detailed information about the Amazon Web Services Lambda runtimes versions. Deploy it and every few hours it will show record details of important version information about the Amazon Web Services Lambda runtimes.</p>\n<h6><a id=\"Amazon_Web_Servicesfargatewithrstudioopensource_33\"></a><strong>Amazon Web Services-fargate-with-rstudio-open-source</strong></h6>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/134\" target=\"_blank\">amazon-fargate-with-rstudio-open-source</a></ins> this project delivers Amazon CDK Python code to provision serverless infrastructure in Amazon Web Services Cloud to run Open Source RStudio Server and Shiny. This is pretty nice, so make sure you bookmark this. I am also going through the code and going to re-use parts of it for other open source projects too.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/b8bb3c5b8f5e4434b4a9763607a90f52_image.png\" alt=\"image.png\" /></p>\n<h6><a id=\"ploomber_39\"></a><strong>ploomber</strong></h6>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/13a\" target=\"_blank\">ploomber</a></ins> this project helps you with the challenges of working with .ipynb files allowing teams to develop collaborative, production-ready pipelines using JupyterLab or any text editor. This project is integrated with Amazon Web Services so you can leverage it in your environments to develop locally, and then deploy to the cloud.</p>\n<h5><a id=\"Amazon_Web_ServicesConnectManager_43\"></a><strong>Amazon Web Services-Connect-Manager</strong></h5>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/137\" target=\"_blank\">Amazon Web Services-Connect-Manager</a></ins> this project from Jozef Jaroščiak provides Windows users with an easy to use tool to connect with your Amazon EC2 instances.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/44e8b2d9494047638910980962be28b0_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"Amazon_Web_Serviceskubernetesedgegreengrass_49\"></a><strong>Amazon Web Services-kubernetes-edge-greengrass</strong></h5>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/13b\" target=\"_blank\">amazon-kubernetes-edge-greengrass</a></ins> this project demonstrates how to collect data from edge locations using Kubernetes (k3s) and Amazon Web Services IoT Greengrass. The code shows you how to get up and running on a Raspberry Pi, and uses Amazon Web Services CDK - two of my favourite things.</p>\n<h5><a id=\"Amazon_Web_Servicesiamtemporaryelevatedaccessbroker_54\"></a><strong>Amazon Web Services-iam-temporary-elevated-access-broker</strong></h5>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/13e\" target=\"_blank\">Amazon Web Services-iam-temporary-elevated-access-broker</a></ins> this projects provides a minimal reference implementation for providing temporary elevated access to your Amazon Web Services environment. James Greenwood, Bikash Behera, and Kevin Higgins walk you through how to use this project in the blog post, <ins><a href=\"https://aws-oss.beachgeek.co.uk/13l\" target=\"_blank\">Managing temporary elevated access to your Amazon Web Services environment</a></ins></p>\n<p><img src=\"https://dev-media.amazoncloud.cn/b41f2d1fdb9844d8bad689c1b2e0fdd4_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"javaspringbootecsfargaterediscaching_60\"></a><strong>java-spring-boot-ecs-fargate-redis-caching</strong></h5>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/13h\" target=\"_blank\">java-spring-boot-ecs-fargate-redis-caching</a></ins> this sample shows how a Spring Boot application can be deployed on ECS Fargate with Redis Caching Support to persist session data and improve application performance. The solution also provides the necessary templates to automatically provision the required infrastructure using CloudFormation to your desired Amazon Web Services account.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/edfa6c5c1ada40d19d8c79b2c7c9e969_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"Amazon_Web_Servicesseverlessvendingmachine_66\"></a><strong>Amazon Web Services-severless-vending-machine</strong></h5>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/13i\" target=\"_blank\">Amazon Web Services-severless-vending-machine</a></ins> this repository shows you how to create a solution to developer teams consume IaC Products, like a “Vending Machine”, to automated deploy cloud service in their Amazon Web Services Accounts without open a ticket or request in IT department. You can read the blog post <ins><a href=\"https://aws-oss.beachgeek.co.uk/13j\" target=\"_blank\">Criando um portal de autosserviços para o provisionamento na Amazon Web Services</a></ins> (in Portugese, but I tried translating with Amazon Translate and it did a great job)</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/e0fe8eb348ab4385b25e68aa7f7da674_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"amazonconnecttwitterlistener_72\"></a><strong>amazon-connect-twitter-listener</strong></h5>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/13k\" target=\"_blank\">amazon-connect-twitter-listener</a></ins> I was recently chatting with a builder who was trying to implement social media integration with Amazon Connect, so I was pleasantly surprised when this project popped up on my open source radar. This project contains source code and supporting files for twitter streaming to Amazon Connect Tasks. Tweets are filtered on the specified tag and analyzed using comprehend to identify language and sentiment. Both tags are added as attributes to the task so additional branching decisions can be made. You need to set up your Amazon Connect project up first, but details are in the README</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/5e5a4540203d4d15b50f2a3cabaaae88_image.png\" alt=\"image.png\" /></p>\n<h4><a id=\"Amazon_Web_Services_and_Community_blog_posts_78\"></a><strong>Amazon Web Services and Community blog posts</strong></h4>\n<h5><a id=\"Amazon_Web_Services_CDK_79\"></a><strong>Amazon Web Services CDK</strong></h5>\n<p>A couple of good Amazon CDK posts this week, starting off with <ins><a href=\"https://aws.amazon.com/blogs/opensource/comparing-aws-cloud-development-kit-and-aws-controllers-for-kubernetes/\" target=\"_blank\">Comparing Amazon Cloud Development Kit and Amazon Web Services Controllers for Kubernetes</a></ins> in which Nima Kaviani, Manabu McCloskey, and Nicholas Thomson come together to compare strategies, tools and technologies to automate infrastructure and software rollout strategies when working with Kubernetes. They look at six areas of focus: Programming, Validation, Execution, Interaction, Operation, and Reconciliation.</p>\n<p>A must read post from Alex Pulver and Nick Lynch is their post, <ins><a href=\"https://aws-oss.beachgeek.co.uk/133\" target=\"_blank\">Experimental construct libraries are now available in Amazon Web Services CDK v2</a></ins> where they share an update on the Amazon Web Services CDK v2 experimental APIs, and how this can make it even easier to migrate from v1.</p>\n<h5><a id=\"Steampipe_85\"></a><strong>Steampipe</strong></h5>\n<p>Steampipe is an open source tool for querying cloud APIs in a universal way and reasoning about the data in SQL. David Boeke, Bob Tordella, Jon Udell, and Nathan Wallace have collaborated together on this post, <ins><a href=\"https://aws-oss.beachgeek.co.uk/12u\" target=\"_blank\">Querying Amazon Web Services at scale across APIs, Regions, and accounts</a></ins> where they share a number of different approaches for how you can use this project to deliver the best possible speed and throughput. [hands on]</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/9b4228ed80524b8fbc6246dfdbe1b2ef_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"Amazon_Web_Services_Distro_for_OpenTelemetry_92\"></a><strong>Amazon Web Services Distro for OpenTelemetry</strong></h5>\n<p>Nathaniel Ruiz Nowell explains how to get telemetry data from Amazon Web Services Lambda Python functions, without having to change a line of code, in his post <ins><a href=\"https://aws-oss.beachgeek.co.uk/12t\" target=\"_blank\">Auto-instrumenting a Python application with an Amazon Web Services Distro for OpenTelemetry Lambda layer</a></ins> [hands on]</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/93fb6675c0014146b4672bde6892c6c8_image.png\" alt=\"image.png\" /></p>\n<p>We also had with the post, <ins><a href=\"https://aws-oss.beachgeek.co.uk/12z\" target=\"_blank\">How To Use OpenTelemetry With Amazon Web Services Lambda</a></ins> from Tom Zach at Aspecto, that also shows you how to deploy a tracing-enabled lambda with OpenTelemetry.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/a7baa9fe4b0640afbf338f4f434c4b68_image.png\" alt=\"image.png\" /></p>\n<p>To wrap up things in ADOT we had Alolita Sharma write <ins><a href=\"https://aws-oss.beachgeek.co.uk/138\" target=\"_blank\">Amazon Web Services Distro for OpenTelemetry 0.14 is now available with updated Lambda layers</a></ins>, providing the release highlights and details on how you can download this image.</p>\n<h5><a id=\"Flagger_104\"></a><strong>Flagger</strong></h5>\n<p>Flagger is a CNCF incubating project that takes care of much of the undifferentiated heavy lifting of canary deployments on Kubernetes. In the post <ins><a href=\"https://aws-oss.beachgeek.co.uk/130\" target=\"_blank\">Performing canary deployments and metrics-driven rollback with Amazon Managed Service for Prometheus and Flagger</a></ins> Kevin Bell and Stefan Prodan explain how to perform canary deployments on Kubernetes using Flagger to orchestrate the rollout, promotion, and rollback of deployments. Taking care of metrics we have Amazon Managed Service for Prometheus, and letting you observe the canary deployment process, we have Amazon Managed Grafana. Nice! [hands on]</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/0be44719a4ac472d8880d7eddfcc85ea_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"Cortex_110\"></a><strong>Cortex</strong></h5>\n<p>In the post, <ins><a href=\"https://aws-oss.beachgeek.co.uk/12y\" target=\"_blank\">Scaling Cortex with parallel compaction</a></ins> Albert Choi, an intern on the Amazon Managed Service for Prometheus team, shares his experience of designing and implementing parallel compactors inside of the Cortex open source project.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/dba186f151ed42a981ffeba2769a026f_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"Dask_and_Open_Data_116\"></a><strong>Dask and Open Data</strong></h5>\n<p>This week we have a couple of interesting posts on how to analyse open data sets using an open source project called Dask (an open source project in Python that provides parallelism for analytics). Combining open data sets together with open source software running on Amazon Web Services is a growing trend to help provide insights that can help make the lives of emergency managers, climate scientists, and policy makers that much easier.</p>\n<p>First up we have the post, <ins><a href=\"https://aws-oss.beachgeek.co.uk/12r\" target=\"_blank\">Transforming Geospatial Data to Cloud-Native Frameworks with Element 84 on Amazon Web Services</a></ins>, where Matt Hanson and Ariel Walcutt from Element 84<br />\nand Imtranur Rahman and Mary-Elizabeth Morin from Amazon Web Services walk you through a solution you can put together than combines open data sets from Sentinel-2 (satellite imagery) with Dask, and a number of Python libraries to analyse and visualise the data. [hands on]</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/747617887b4d4715ab7d5b2e70d8f89b_Element-84-RBG-Image.gif\" alt=\"Element84RBGImage.gif\" /></p>\n<p>Following that we have <ins><a href=\"https://aws-oss.beachgeek.co.uk/12w\" target=\"_blank\">Analyze terabyte-scale geospatial datasets with Dask and Jupyter on Amazon Web Services</a></ins>, where Ethan Fahy and Zac Flamig show you how to set up this Kubernetes, Dask and Jupyter Pangeo solution on Amazon Web Services to automatically scale your cloud compute resources and parallelise your workload across multiple Dask worker nodes. In this example they show you how you can run an example notebook that performs sample analyses on the Coupled Model Intercomparison Project 6 (CMIP6) dataset from the Registry of Open Data on Amazon Web Services. [hands on]</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/ffe0cb59daf04116995ce20468bc47d1_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"Apache_Airflow_129\"></a><strong>Apache Airflow</strong></h5>\n<p>Dzenan Softic and Sam Dengler demonstrate how to use a CodeArtifact repository to load your Python libraries in Apache Airflow in the post, <ins><a href=\"https://aws-oss.beachgeek.co.uk/139\" target=\"_blank\">Amazon MWAA with Amazon Web Services CodeArtifact for Python dependencies</a></ins></p>\n<p><img src=\"https://dev-media.amazoncloud.cn/cc5e1b23541543609128aa345041a4fb_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"Apache_Flink_135\"></a><strong>Apache Flink</strong></h5>\n<p>In this post, <ins><a href=\"https://aws-oss.beachgeek.co.uk/12s\" target=\"_blank\">Optimizing Apache Flink on Amazon EKS using Amazon EC2 Spot Instances</a></ins> Kinnar Sen shows you how to deploy a scalable, highly available, resilient, and cost optimised Flink application using Kubernetes via Amazon EKS and Amazon EC2 Spot Instances. [hands on]</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/1692ef1e50794a8ab6464ebd61e1b4d2_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"OpenSearch_141\"></a><strong>OpenSearch</strong></h5>\n<p>If you have a use case where you need to be able to run a single query against many different data sources, then you will find this post from Behram Irani and Madhav Vishnubhatta will be of interest. Using a capability of Amazon Athena called Amazon Athena Federated Queries, you can use an existing connector for OpenSearch to query your data in both OpenSearch and (in this example) Amazon S3. To find out more, read the post <ins><a href=\"https://aws-oss.beachgeek.co.uk/12v\" target=\"_blank\">Query data in Amazon OpenSearch Service using SQL from Amazon Athena</a></ins> [hands on]</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/6ff9dffaa33a4dd5b540f26ee04f4438_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"RabbitMQ_147\"></a><strong>RabbitMQ</strong></h5>\n<p>Amazon MQ provides two managed broker deployment connection options: public brokers and private brokers.<br />\nNate Bachmeier, Wallace Printz, and Christian Mueller have come together to share how to provision private Amazon MQ brokers behind a secure public load balancer endpoint using an example subdomain in the blog post, <ins><a href=\"https://aws-oss.beachgeek.co.uk/132\" target=\"_blank\">Creating static custom domain endpoints with Amazon MQ for RabbitMQ </a></ins>[hands on]</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/87f660904c5c4b1d896bcada78859243_image.png\" alt=\"image.png\" /></p>\n<h5><a id=\"Redis_154\"></a><strong>Redis</strong></h5>\n<p>This post from Ramesh Sencha at Coupa, <ins><a href=\"https://aws-oss.beachgeek.co.uk/131\" target=\"_blank\">How Coupa migrated from a self-hosted Redis to fully managed Amazon ElastiCache</a></ins> details why Coupa migrated self managed Redis workloads to Amazon ElastiCache; the benefits, the approach and the architecture they chose.</p>\n<h4><a id=\"Quick_updates_158\"></a><strong>Quick updates</strong></h4>\n<h5><a id=\"Fluent_Bit_159\"></a><strong>Fluent Bit</strong></h5>\n<p>Amazon Elastic Kubernetes Service (EKS) on Fargate now supports the use of Kubernetes Fluent Bit filters which provide enriched Kubernetes-specific metadata to Fluent Bit logs. Customers can now more easily observe and troubleshoot their applications by using the Kubernetes pod, container, or namespace name, among other Kubernetes metadata, to associate with their applications’ logs. Without support for the Kubernetes filter for Fluent Bit on EKS Fargate, customers had to manually read through log files to find the events they were interested in and it was difficult to associate a given log line with a Kubernetes application. Some customers who needed this kind of observability were unable to use Fargate with EKS and couldn’t benefit from its fully managed compute environment.</p>\n<h4><a id=\"Video_of_the_week_163\"></a><strong>Video of the week</strong></h4>\n<h5><a id=\"OpenSearch_164\"></a><strong>OpenSearch</strong></h5>\n<p>In this recording from the Amazon Web Services Summit, Kyle Davis will give you an understanding of the capabilities of OpenSearch, the first steps to get started (on or off Amazon Web Services), a picture of how OpenSearch fits in your stack, a look into what’s in store for the future of the project as well as a guide to join the community and contribute.</p>\n<p><video src=\"https://dev-media.amazoncloud.cn/56803396e1964778b1c0f653524a6368_AWS%20AMER%20Summit%20Aug%202021%EF%BC%9A%20The%20top%20five%20things%20to%20know%20about%20OpenSearch.mp4\" controls=\"controls\"></video></p>\n<h5><a id=\"Open_source_observability_171\"></a><strong>Open source observability</strong></h5>\n<p>Another great video from Dotan Horovits over at Logz.io. In this session he introduces observability, the role of open source in this area and then shows you some of the tools and projects in this space. Well worth 41 minutes of your time.</p>\n<p><video src=\"https://dev-media.amazoncloud.cn/93f3390882694d06992e25195134a037_Open%20Source%20for%20Better%20Observability%20-%20Dotan%20Horovits%2C%20Logz.io.mp4\" controls=\"controls\"></video></p>\n<h4><a id=\"Events_for_your_diary_177\"></a><strong>Events for your diary</strong></h4>\n<h5><a id=\"Databricks__Amazon_Web_Services_Lakehouse_Dev_Day_Live_Workshop_178\"></a><strong>Databricks | Amazon Web Services Lakehouse Dev Day Live Workshop</strong></h5>\n<h5><a id=\"November_16th_900_AM_PT_179\"></a><strong>November 16th 9:00 AM PT</strong></h5>\n<p>Delta Lake is an open source storage layer that provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. You can use Delta Lake on top of your existing data lake. During this workshop you will learn how to:</p>\n<ul>\n<li>Make your existing Amazon S3 data lakes into a lakehouse with Delta Lake.</li>\n<li>Provide an easy-to-use platform for analysts to directly query data on your data lake using SQL Analytics</li>\n<li>Simplify and automate data pipelines for streaming and batch data to lower costs and boost productivity for your data teams</li>\n</ul>\n<p><ins><a href=\"https://aws-oss.beachgeek.co.uk/zs\" target=\"_blank\">Read more and sign up here</a></ins></p>\n<h5><a id=\"Stay_in_touch_with_open_source_at_Amazon_Web_Services_189\"></a><strong>Stay in touch with open source at Amazon Web Services</strong></h5>\n<p>I hope this summary has been useful. Remember to check out the <ins><a href=\"https://aws.amazon.com/opensource/?opensource-all.sort-by=item.additionalFields.startDate&amp;opensource-all.sort-order=asc\" target=\"_blank\">Open Source homepage</a></ins> to keep up to date with all our activity in open source by following us on <ins><a href=\"https://twitter.com/AWSOpen\" target=\"_blank\">@Amazon Web ServicesOpen</a></ins></p>\n"}