Open source news and updates #85

{"value":"#### **October 11th, 2021 - Instalment #85**\nNewsletter #85.\n\nThis week we have an interesting gaming flavour to this newsletter, with a couple of projects that are influenced or are directly related to gaming. We also have other new open source projects including one that shows you how you can run an OpenVSCode Server for Amazon Web Services Graviton2. This weeks blog post and tutorials cover Apache Airflow, Apache Spark, Apache Kafka, Apache Ranger, Apache Flink, OpenSearch, Apache Lucene, PostgreSQL, Amazon Web Services CDK, and Kubernetes. For those interested in machine learning, we have a literal Feast for you, in a nice post covering the open source project Feast. We also have a great post for those of you looking who are applying the open source way of working, internally (sometimes referred to as Innersource) so make sure you check that post out as it is really nice. Finally we have some new events added, so check those out and a couple of videos that you will want to bookmark/view.\n\nWith KubeCon happening this week, make sure you check out the Container Day. These are always packed full of the latest updates and great presentations from the containers team at Amazon Web Services.\n\n##### **Container Day x KubeCon**\n\nNext week, the Amazon Web Services Kubernetes team is back for the next instalment of Container Day x KubeCon, a full day of sessions on [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) and Kubernetes at Amazon Web Services. In this live and virtual Twitch event, the Amazon Web Services Kubernetes team, hosted by developer advocates Justin Garrison and Jesse Butler, will be covering new launches, demoing products and features, and answering your questions. Read more in the post, ++[Save the date: Container Day x KubeCon](https://aws-oss.beachgeek.co.uk/zd)++ to find out about the sessions and you can ++[sign up here](https://aws-oss.beachgeek.co.uk/cr)++ to be reminded when things kick off.\n\n##### **Celebrating open source contributors**\n\nThe articles posted in this series are only possible thanks to contributors and project maintainers and so I would like to shout out and thank those folks who really do power open source and enable us all to build on top of what they have created.\n\nSo thank you to the following open source heroes: Starr Shaw, John Kennedy, Nico Triballier, Michael Hamilton, Joe Losinski, Eng-Hwa Tan, Raffa aka Picster, Debashish Chakrabarty, Akash Verma, Ian Mckay, Naga Appani, Chinmayi Narasimhadevara, Uma Ramadoss, Willem Pienaar, Matt Asay, Marcus Eagan, Henry Robalino, Varun Rao Bhamidimarri, Jalpan Randeri, Virginia Chu, Hunter Tom, Om Prakash Jha and Michael Hausenblas\n\nMake sure you find and follow these builders and keep up to date with their open source projects and contributions.\n\n##### **Latest open source projects**\n###### **vscode-spoty-army**\n\n++[vscode-spoty-army](https://aws-oss.beachgeek.co.uk/z5)++ Gitpod is a great open source project that helps you to tun upstream VS Code on a remote machine with access through a modern web browser from any device, anywhere, via the ++[OpenVSCode Server project](https://aws-oss.beachgeek.co.uk/z6)++. This project from Eng-Hwa Tan will help you to deploy an OpenVSCode Server for Amazon Web Services Graviton2 (arm64) with spot instance using [Amazon ECS](https://aws.amazon.com/cn/ecs/?trk=cndc-detail).\n\n###### **cloudkeeper-ckui**\n\n++[cloudkeeper-ckui](https://aws-oss.beachgeek.co.uk/z7)++ this project really impressed me last week and got me thinking should game designers be hired by all projects/organisations to help design their UI's. Cloudkeeper is an open source project that indexes resources, captures dependencies and maps out your Amazon Web Services infrastructure in a graph so that it’s understandable for a human. Raffa aka Picster shared news last week on Reddit about a new project he is working on, a UI for this project. Check out the ++[thread in Reddit](https://aws-oss.beachgeek.co.uk/z7)++, or the video that provides a glimpse of what you can see.\n\n<video src=\\"https://dev-media.amazoncloud.cn/4839f0d57e8442958ef40d8b22640c5f_SciFi%20UI%20in%20Godot%20-%20UI%20prototype%20for%20Cloudkeeper.io.mp4\\" class=\\"manvaVedio\\" controls=\\"controls\\" style=\\"width:160px;height:160px\\"></video>\n\n###### **Planet_Storm**\n\n++[Planet_Storm](https://aws-oss.beachgeek.co.uk/zo)++ from application of a games UI to an actual game. This project provides an example game called Planet Storm, that is imagined as a 3D, physics-based, side-scrolling platformer inspired by the 1980’s IREM classic “Moon Patrol”. It uses the Open 3D Engine (O3DE) engine and the developers behind this, Starr Shaw and John Kennedy, share how you can build this yourself in the blog post, ++[We Also Make Games: Amazon Web Services’ First Internal O3DE Game is Now Available!](https://aws-oss.beachgeek.co.uk/zp)++\n\n\n\n###### **Amazon Web Services-instance-scheduler**\n\n++[amazon-instance-scheduler](https://aws-oss.beachgeek.co.uk/z4)++ if you ever needed to schedule the stopping and starting of EC2 and RDS resources across your Amazon Web Services accounts, then this project is what you have been looking for. This helps reduce operational costs by stopping resources that are not in use and starting resources when their capacity is needed. It provides a solution for Cross-Account and Cross-Region scheduling EC2 and RDS instances/clusters using SSM Automation Runbooks.\n\n\n\n###### **glue-workflow-amazon-cdk**\n\n++[glue-workflow-amazon-cdk](https://aws-oss.beachgeek.co.uk/z2)++ this repository contains an example that creates a Glue Workflow containing multiple crawlers, glue jobs and triggers for the workflow. The workflow is manually triggered, but the script can be updated to run on a cron schedule.\n\nFollow the blog post ++[Field Notes: How to Build an Amazon Glue Workflow using the Amazon Web Services Cloud Development Kit](https://aws-oss.beachgeek.co.uk/z3)++ from Michael Hamilton and Joe Losinski to get started.\n\n\n\n##### **Amazon Web Services and Community blog posts**\n###### **Kubernetes**\n\nVirginia Chu and Hunter Tom write in their post, ++[Simplifying Kubernetes configurations using Amazon Web Services Lambda](https://aws-oss.beachgeek.co.uk/zl)++ how to create a multi-stage Dockerfile that uses eksctl, kubectl, and amazon-auth. Why would you want to do that? This will help you manage your [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) clusters dynamically using a Lambda function rather than installing kubectl or eksctl on a local machine. Additionally, all container images are stored in a versioned format as infrastructure as code.\n\nI also read this post from Okta, ++[Secure Access to Amazon Web Services EKS Clusters for Admins](https://aws-oss.beachgeek.co.uk/zq)++ from Nico Triballier. In this tutorial, Nico shows you how you can use OpenID Connect (OIDC) to allow developers to securely access [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) clusters on Amazon Web Services.\n\n###### **OpenSearch**\n\nAmazon Web Services Hero and open sourcerer Ian Mckay put together this blog post, ++[Migrating to OpenSearch with CloudFormation](https://aws-oss.beachgeek.co.uk/zc)++ on how to migrate from an Elasticsearch domain to an OpenSearch domain. [hands on]\n\n###### **Apache Lucene**\n\nIt was great to see this post from former colleague Matt Asay and Marcus Eagan from MongoDB, diving deep on how MongoDB Atlas uses Apache Lucene within Atlas Search. Read the post, ++[How Amazon Web Services and MongoDB collaborate to unlock the power of Apache Lucene](https://aws-oss.beachgeek.co.uk/zi)++\n\n\n\n###### **Innersource**\n\nInnerSource is the term for the emerging practice of organisations adopting the open source methodology and tools in order to build their applications. I was very excited to read ++[Building an InnerSource ecosystem using Amazon Web Services DevOps tools](https://aws-oss.beachgeek.co.uk/z9)++ from Debashish Chakrabarty and Akash Verma. The post discusses how to build a model InnerSource ecosystem that leverages multiple Amazon Web Services services, such as CodeBuild, CodeCommit, CodePipeline, CodeArtifact, and CodeGuru, along with other Amazon Web Services services and open source tools such as the ++[InnerSource Portal from SAP](https://aws-oss.beachgeek.co.uk/zb)++. The post provides a nice tool, the ++[codecommit-crawler-innersource](https://aws-oss.beachgeek.co.uk/za)++ that provides a custom crawler for Amazon Web Services CodeCommit that generates the repos.json that can be used by the SAP InnerSource Portal. [hands on]\n\n\n\n\n###### **Feast**\n\nFeast is an open source feature store and a fast, convenient way to serve machine learning (ML) features for training and online inference. Feast lets you build point-in-time correct training datasets from feature data, allows you to deploy a production-grade feature serving stack to Amazon Web Services and simplifies tracking features models are using. Willem Pienaar, Principal Engineer at Tecton and creator of Feast created a post, ++[Getting started with Feast, an open source feature store running on Amazon Web Services Managed Services](https://aws-oss.beachgeek.co.uk/zg)++ that does what it says on the tin - walks you through how you can use Feast using a real-time credit scoring example.\n\n\n\n###### **Apache Airflow**\n\nA couple of Apache Airflow posts this week (neither by me!), covering some cool topics that I often hear developers wanting to know how to do.\n\nFirst up we have Uma Ramadoss who has put together this post, ++[Automating a DAG deployment with Amazon Managed Workflows for Apache Airflow](https://aws-oss.beachgeek.co.uk/zh)++ that goes beyond just creating a simple CI/CD pipeline for your workflows, and includes how you test them and then how you automate the deployment and updating of your MWAA environments. Super useful. [hands on]\n\n\n\nTo follow that, we have Henry Robalino who wrote ++[Using Okta as an identity provider with Amazon MWAA](https://aws-oss.beachgeek.co.uk/zj)++ which shows you how you can integrate an existing identity provider (in this instance, Okta) to authenticate via Amazon Web Services SSO to your [Amazon Managed Workflows for Apache Airflow](https://aws.amazon.com/cn/managed-workflows-for-apache-airflow/?trk=cndc-detail) (Amazon MWAA) environments. Very nice post indeed. [hands on]\n\n###### **Apache Flink/Apache Kafka**\n\nChinmayi Narasimhadevara walks you through how to create stream processing applications using Apache Flink, in this hands on post, ++[Query your Amazon MSK topics interactively using Amazon Kinesis Data Analytics Studio](https://aws-oss.beachgeek.co.uk/zk)++. Apache Flink is an open-source framework and engine for processing data streams, and in the post she walks you through creating some example applications to interact with streaming data from Apache Kafka in Scala, Python and SQL.\n\n###### **Apache Ranger**\n\nApache Ranger is an open source project that provides security capabilities for Hadoop clusters, and in this post ++[Authorize SparkSQL data manipulation on Amazon EMR using Apache Ranger](https://aws-oss.beachgeek.co.uk/zk)++ from Varun Rao Bhamidimarri and Jalpan Randeri, you will learn more about how [Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail) 6.4 has introduced additional authorising capabilities for data manipulation statements with Apache Ranger 2.0.\n\n\n\n###### **PostgreSQL**\n\nNaga Appani has put together this post, ++[Tune sorting operations in PostgreSQL with work_mem](https://aws-oss.beachgeek.co.uk/ze)++ where he talks about how and where to identify queries that are suboptimal, such as ones that require sorting on disk, and how to use various optimisation techniques to improve performance. What is work_mem? Read the post to find out what it is and why it is important in how you optimise your queries.\n\n###### **Amazon Web Services CDK**\n\nOm Prakash Jha writes in his post, ++[Align with best practices while creating infrastructure using CDK Aspects](https://aws-oss.beachgeek.co.uk/zm)++, how if you are utilising Amazon CDK to provision your infrastructure, you can apply some best practices to not only to evaluate compliance of your resources against the best practices, but also modify their state to make them compliant before they are created.\n\n\n\n##### **Quick updates**\n###### **Apache Spark**\n\n[Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail) now supports Apache Spark SQL to insert data into and update Apache Hive metadata tables when Apache Ranger integration is enabled. In January we launched [Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail) integration with Apache Ranger, a feature that allows you to define and enforce database, table, and column-level permissions when Apache Spark users access data in [Amazon S3](https://aws.amazon.com/cn/s3/?trk=cndc-detail) through the Hive Metastore.\n\nPreviously, with Apache Ranger is enabled, you were limited to only being able to read data using Spark SQL statements such as SHOW DATABASES and DESCRIBE TABLE. **Now**, you can also insert data into, or update the Apache Hive metadata tables with these statements: INSERT INTO, INSERT OVERWRITE, and ALTER TABLE.\n\n###### **PostgreSQL**\n\nAmazon Relational Database Service ([Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail)) for PostgreSQL now supports PostGIS major version 3.1. This new version of PostGIS is available on PostgreSQL versions 13.4, 12.8, 11.13, 10.18, and higher.\n\nPostGIS allows you to store, query and analyse geospatial data within a PostgreSQL database. PostGIS 3.1 significantly improves performance such as spatial joins, which now run up to 6.8X faster on PostgreSQL 13. As an example, you could use a spatial join to count the number of people living in an area defined by the reception of mobile phones from radio towers.\n\nPostGIS 3.1 is the new default version on PostgreSQL 10 and higher starting with the new minor versions. However, you can still create older versions of PostGIS in your PostgreSQL database, e.g., if you require version stability.\n\n###### **Ubuntu**\n\n[Amazon EC2 ](https://aws.amazon.com/cn/ec2/?trk=cndc-detail)now supports Hibernation for Ubuntu 20.04 LTS operating system. Hibernation allows you to pause your EC2 Instances and resume them at a later time, rather than fully terminating and restarting them. Resuming your instance lets your applications continue from where they left off so that you don’t have to restart your OS and application from scratch. Hibernation is useful for cases where rebuilding application state is time-consuming (e.g., developer desktops) or an application’s start-up steps can be prepared in advance of a scale-out.\n\nFor Ubuntu 20.04 LTS, Hibernation is supported for On-Demand Instances running on C3, C4, C5, C5d, I3, M3, M4, M5, M5a, M5ad, M5d, R3, R4, R5, R5a, R5ad, R5d, T2, T3, and T3a with up to 150 GB of RAM.\n\n##### **Video of the week**\n\nThis week we have Michael Hausenblas' session at the Swiss Cloud Native Day, where he gave this session How open source made observability mainstream.\n\n<video src=\\"https://dev-media.amazoncloud.cn/79453f10a5764ae38acb24c0a2847c0e_How%20open%20source%20made%20observability%20mainstream%20-%20Michael%20Hausenblas%20-%20Swiss%20Cloud%20Native%20Day%202021.mp4\\" class=\\"manvaVedio\\" controls=\\"controls\\" style=\\"width:160px;height:160px\\"></video>\n\nFollowing that we have this recorded workshop from Provectus, [Amazon SageMaker](https://aws.amazon.com/cn/sagemaker/?trk=cndc-detail) and Open-Source Tools for ML: Better Together.\n\n<video src=\\"https://dev-media.amazoncloud.cn/02425145c38542728d0c7d8957368947_Amazon%20SageMaker%20and%20Open-Source%20Tools%20for%20ML%EF%BC%9A%20Better%20Together.mp4\\" class=\\"manvaVedio\\" controls=\\"controls\\" style=\\"width:160px;height:160px\\"></video>\n\n#### **Workshops**\n##### **Hadoop on Kubernetes**\n\nIf you have not seen or worked through this workshop, ++[EMR on EKS Workshop](https://aws-oss.beachgeek.co.uk/zn)++ then it provides a nice entry point into running some of the open source analytics tools such as Apache Spark on Kubernetes.\n\n#### **Events for your diary**\nComing up later this week we have...\n\n##### **Container Day x KubeCon**\n##### **October 12th, 8:00 AM – 4:00 PM (PDT)**\n\nThe Amazon Web Services Kubernetes team is back for the next instillment of Container Day x KubeCon, a full day of sessions on [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) and Kubernetes at Amazon Web Services. In this live and virtual Twitch event, the Amazon Web Services Kubernetes team, hosted by developer advocates Justin Garrison and Jesse Butler, will be covering new launches, demoing products and features, and answering your questions. ++[Sign up](https://aws-oss.beachgeek.co.uk/cr)++ to be reminded when things kick off.\n\n##### **GraphQL API security best practices with Amazon Web Services**\n##### **AppSync and Amazon Web Services Amplify**\n##### **14th October, 11am AEST**\n\nAs a developer, the most important parts of managing your applications should always include enhancing performance while strengthening security. In this webinar, we take you through security best practices for your GraphQL API’s with Amazon Web Services AppSync and Amazon Web Services Amplify, providing you with an understanding of how these can be applied to your applications. In this session, you will learn about:\n\n- GraphQL Protocol and how to configure a schema\n- Possible ways to authenticate and authorise access to GraphQL APIs\n- How to configure network security for your API\n- How to enable observability for your API with logging, tracing or auditing\n\nTo ++[register for this event, use this link](https://aws-oss.beachgeek.co.uk/ue)++.\n\n##### **Flink Forwards Global 2021**\n##### **October 26th/27th**\n\nFlink Forward Global 2021 is a 2-day virtual conference for the Apache Flink and stream processing communities. Apache Flink is an open-source distributed engine for processing data streams that can support both streaming and batch workloads. Flink Forward has keynote presentations and talks on production Flink use cases, technical deep dive sessions, and the growth of the Flink ecosystem. You can meet core Flink committers, new and experienced users, and thought leaders who share experiences and best practices in stream processing, real-time analytics, and the management of mission-critical Flink deployments in production.\n\n++[Read more and sign up here](https://aws-oss.beachgeek.co.uk/wh)++.\n\n##### **Databricks | Amazon Web Services Lakehouse Dev Day Live Workshop**\n##### **November 16th 9:00 AM PT**\n\nDelta Lake is an open source storage layer that provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. You can use Delta Lake on top of your existing data lake. During this workshop you will learn how to:\n\n- Make your existing [Amazon S3](https://aws.amazon.com/cn/s3/?trk=cndc-detail) data lakes into a lakehouse with Delta Lake.\n- Provide an easy-to-use platform for analysts to directly query data on your data lake using SQL Analytics\n- Simplify and automate data pipelines for streaming and batch data to lower costs and boost productivity for your data teams\n\n++[Read more and sign up here.](https://aws-oss.beachgeek.co.uk/zs)++\n\n#### **Stay in touch with open source at Amazon Web Services**\nI hope this summary has been useful. Remember to check out the ++[Open Source homepage](https://aws.amazon.com/opensource/?opensource-all.sort-by=item.additionalFields.startDate&opensource-all.sort-order=asc)++ to keep up to date with all our activity in open source by following us on ++[@Amazon Web ServicesOpen](https://twitter.com/AWSOpen)++\n\n\n\n","render":"<h4><a id=\\"October_11th_2021__Instalment_85_0\\"></a><strong>October 11th, 2021 - Instalment #85</strong></h4>\\n<p>Newsletter #85.</p>\n<p>This week we have an interesting gaming flavour to this newsletter, with a couple of projects that are influenced or are directly related to gaming. We also have other new open source projects including one that shows you how you can run an OpenVSCode Server for Amazon Web Services Graviton2. This weeks blog post and tutorials cover Apache Airflow, Apache Spark, Apache Kafka, Apache Ranger, Apache Flink, OpenSearch, Apache Lucene, PostgreSQL, Amazon Web Services CDK, and Kubernetes. For those interested in machine learning, we have a literal Feast for you, in a nice post covering the open source project Feast. We also have a great post for those of you looking who are applying the open source way of working, internally (sometimes referred to as Innersource) so make sure you check that post out as it is really nice. Finally we have some new events added, so check those out and a couple of videos that you will want to bookmark/view.</p>\n<p>With KubeCon happening this week, make sure you check out the Container Day. These are always packed full of the latest updates and great presentations from the containers team at Amazon Web Services.</p>\n<h5><a id=\\"Container_Day_x_KubeCon_7\\"></a><strong>Container Day x KubeCon</strong></h5>\\n<p>Next week, the Amazon Web Services Kubernetes team is back for the next instalment of Container Day x KubeCon, a full day of sessions on Amazon EKS and Kubernetes at Amazon Web Services. In this live and virtual Twitch event, the Amazon Web Services Kubernetes team, hosted by developer advocates Justin Garrison and Jesse Butler, will be covering new launches, demoing products and features, and answering your questions. Read more in the post, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zd\\" target=\\"_blank\\">Save the date: Container Day x KubeCon</a></ins> to find out about the sessions and you can <ins><a href=\\"https://aws-oss.beachgeek.co.uk/cr\\" target=\\"_blank\\">sign up here</a></ins> to be reminded when things kick off.</p>\n<h5><a id=\\"Celebrating_open_source_contributors_11\\"></a><strong>Celebrating open source contributors</strong></h5>\\n<p>The articles posted in this series are only possible thanks to contributors and project maintainers and so I would like to shout out and thank those folks who really do power open source and enable us all to build on top of what they have created.</p>\n<p>So thank you to the following open source heroes: Starr Shaw, John Kennedy, Nico Triballier, Michael Hamilton, Joe Losinski, Eng-Hwa Tan, Raffa aka Picster, Debashish Chakrabarty, Akash Verma, Ian Mckay, Naga Appani, Chinmayi Narasimhadevara, Uma Ramadoss, Willem Pienaar, Matt Asay, Marcus Eagan, Henry Robalino, Varun Rao Bhamidimarri, Jalpan Randeri, Virginia Chu, Hunter Tom, Om Prakash Jha and Michael Hausenblas</p>\n<p>Make sure you find and follow these builders and keep up to date with their open source projects and contributions.</p>\n<h5><a id=\\"Latest_open_source_projects_19\\"></a><strong>Latest open source projects</strong></h5>\\n<h6><a id=\\"vscodespotyarmy_20\\"></a><strong>vscode-spoty-army</strong></h6>\\n<p><ins><a href=\\"https://aws-oss.beachgeek.co.uk/z5\\" target=\\"_blank\\">vscode-spoty-army</a></ins> Gitpod is a great open source project that helps you to tun upstream VS Code on a remote machine with access through a modern web browser from any device, anywhere, via the <ins><a href=\\"https://aws-oss.beachgeek.co.uk/z6\\" target=\\"_blank\\">OpenVSCode Server project</a></ins>. This project from Eng-Hwa Tan will help you to deploy an OpenVSCode Server for Amazon Web Services Graviton2 (arm64) with spot instance using Amazon ECS.</p>\n<h6><a id=\\"cloudkeeperckui_24\\"></a><strong>cloudkeeper-ckui</strong></h6>\\n<p><ins><a href=\\"https://aws-oss.beachgeek.co.uk/z7\\" target=\\"_blank\\">cloudkeeper-ckui</a></ins> this project really impressed me last week and got me thinking should game designers be hired by all projects/organisations to help design their UI’s. Cloudkeeper is an open source project that indexes resources, captures dependencies and maps out your Amazon Web Services infrastructure in a graph so that it’s understandable for a human. Raffa aka Picster shared news last week on Reddit about a new project he is working on, a UI for this project. Check out the <ins><a href=\\"https://aws-oss.beachgeek.co.uk/z7\\" target=\\"_blank\\">thread in Reddit</a></ins>, or the video that provides a glimpse of what you can see.</p>\n<p><video src=\\"https://dev-media.amazoncloud.cn/4839f0d57e8442958ef40d8b22640c5f_SciFi%20UI%20in%20Godot%20-%20UI%20prototype%20for%20Cloudkeeper.io.mp4\\" controls=\\"controls\\"></video></p>\\n<h6><a id=\\"Planet_Storm_30\\"></a><strong>Planet_Storm</strong></h6>\\n<p><ins><a href=\\"https://aws-oss.beachgeek.co.uk/zo\\" target=\\"_blank\\">Planet_Storm</a></ins> from application of a games UI to an actual game. This project provides an example game called Planet Storm, that is imagined as a 3D, physics-based, side-scrolling platformer inspired by the 1980’s IREM classic “Moon Patrol”. It uses the Open 3D Engine (O3DE) engine and the developers behind this, Starr Shaw and John Kennedy, share how you can build this yourself in the blog post, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zp\\" target=\\"_blank\\">We Also Make Games: Amazon Web Services’ First Internal O3DE Game is Now Available!</a></ins></p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/ea359f49c64342929c947b05847a3a29_image.png\\" alt=\\"image.png\\" /></p>\n<h6><a id=\\"Amazon_Web_Servicesinstancescheduler_36\\"></a><strong>Amazon Web Services-instance-scheduler</strong></h6>\\n<p><ins><a href=\\"https://aws-oss.beachgeek.co.uk/z4\\" target=\\"_blank\\">amazon-instance-scheduler</a></ins> if you ever needed to schedule the stopping and starting of EC2 and RDS resources across your Amazon Web Services accounts, then this project is what you have been looking for. This helps reduce operational costs by stopping resources that are not in use and starting resources when their capacity is needed. It provides a solution for Cross-Account and Cross-Region scheduling EC2 and RDS instances/clusters using SSM Automation Runbooks.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/e061b3e16ca448628aad8b6615548bbb_image.png\\" alt=\\"image.png\\" /></p>\n<h6><a id=\\"glueworkflowamazoncdk_42\\"></a><strong>glue-workflow-amazon-cdk</strong></h6>\\n<p><ins><a href=\\"https://aws-oss.beachgeek.co.uk/z2\\" target=\\"_blank\\">glue-workflow-amazon-cdk</a></ins> this repository contains an example that creates a Glue Workflow containing multiple crawlers, glue jobs and triggers for the workflow. The workflow is manually triggered, but the script can be updated to run on a cron schedule.</p>\n<p>Follow the blog post <ins><a href=\\"https://aws-oss.beachgeek.co.uk/z3\\" target=\\"_blank\\">Field Notes: How to Build an Amazon Glue Workflow using the Amazon Web Services Cloud Development Kit</a></ins> from Michael Hamilton and Joe Losinski to get started.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/c6b03ff46fa347c2a57d60763bfc51f2_image.png\\" alt=\\"image.png\\" /></p>\n<h5><a id=\\"Amazon_Web_Services_and_Community_blog_posts_50\\"></a><strong>Amazon Web Services and Community blog posts</strong></h5>\\n<h6><a id=\\"Kubernetes_51\\"></a><strong>Kubernetes</strong></h6>\\n<p>Virginia Chu and Hunter Tom write in their post, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zl\\" target=\\"_blank\\">Simplifying Kubernetes configurations using Amazon Web Services Lambda</a></ins> how to create a multi-stage Dockerfile that uses eksctl, kubectl, and amazon-auth. Why would you want to do that? This will help you manage your Amazon EKS clusters dynamically using a Lambda function rather than installing kubectl or eksctl on a local machine. Additionally, all container images are stored in a versioned format as infrastructure as code.</p>\n<p>I also read this post from Okta, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zq\\" target=\\"_blank\\">Secure Access to Amazon Web Services EKS Clusters for Admins</a></ins> from Nico Triballier. In this tutorial, Nico shows you how you can use OpenID Connect (OIDC) to allow developers to securely access Amazon EKS clusters on Amazon Web Services.</p>\n<h6><a id=\\"OpenSearch_57\\"></a><strong>OpenSearch</strong></h6>\\n<p>Amazon Web Services Hero and open sourcerer Ian Mckay put together this blog post, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zc\\" target=\\"_blank\\">Migrating to OpenSearch with CloudFormation</a></ins> on how to migrate from an Elasticsearch domain to an OpenSearch domain. [hands on]</p>\n<h6><a id=\\"Apache_Lucene_61\\"></a><strong>Apache Lucene</strong></h6>\\n<p>It was great to see this post from former colleague Matt Asay and Marcus Eagan from MongoDB, diving deep on how MongoDB Atlas uses Apache Lucene within Atlas Search. Read the post, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zi\\" target=\\"_blank\\">How Amazon Web Services and MongoDB collaborate to unlock the power of Apache Lucene</a></ins></p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/8ce3db7d4249440888640b54535d9c6c_image.png\\" alt=\\"image.png\\" /></p>\n<h6><a id=\\"Innersource_67\\"></a><strong>Innersource</strong></h6>\\n<p>InnerSource is the term for the emerging practice of organisations adopting the open source methodology and tools in order to build their applications. I was very excited to read <ins><a href=\\"https://aws-oss.beachgeek.co.uk/z9\\" target=\\"_blank\\">Building an InnerSource ecosystem using Amazon Web Services DevOps tools</a></ins> from Debashish Chakrabarty and Akash Verma. The post discusses how to build a model InnerSource ecosystem that leverages multiple Amazon Web Services services, such as CodeBuild, CodeCommit, CodePipeline, CodeArtifact, and CodeGuru, along with other Amazon Web Services services and open source tools such as the <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zb\\" target=\\"_blank\\">InnerSource Portal from SAP</a></ins>. The post provides a nice tool, the <ins><a href=\\"https://aws-oss.beachgeek.co.uk/za\\" target=\\"_blank\\">codecommit-crawler-innersource</a></ins> that provides a custom crawler for Amazon Web Services CodeCommit that generates the repos.json that can be used by the SAP InnerSource Portal. [hands on]</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/01355c489cf848119bd172d83eebcfec_image.png\\" alt=\\"image.png\\" /></p>\n<h6><a id=\\"Feast_74\\"></a><strong>Feast</strong></h6>\\n<p>Feast is an open source feature store and a fast, convenient way to serve machine learning (ML) features for training and online inference. Feast lets you build point-in-time correct training datasets from feature data, allows you to deploy a production-grade feature serving stack to Amazon Web Services and simplifies tracking features models are using. Willem Pienaar, Principal Engineer at Tecton and creator of Feast created a post, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zg\\" target=\\"_blank\\">Getting started with Feast, an open source feature store running on Amazon Web Services Managed Services</a></ins> that does what it says on the tin - walks you through how you can use Feast using a real-time credit scoring example.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/edd27805d8d3427c96bc928784308c0b_image.png\\" alt=\\"image.png\\" /></p>\n<h6><a id=\\"Apache_Airflow_80\\"></a><strong>Apache Airflow</strong></h6>\\n<p>A couple of Apache Airflow posts this week (neither by me!), covering some cool topics that I often hear developers wanting to know how to do.</p>\n<p>First up we have Uma Ramadoss who has put together this post, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zh\\" target=\\"_blank\\">Automating a DAG deployment with Amazon Managed Workflows for Apache Airflow</a></ins> that goes beyond just creating a simple CI/CD pipeline for your workflows, and includes how you test them and then how you automate the deployment and updating of your MWAA environments. Super useful. [hands on]</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/8849c4e5e971401f930daddfd625e456_image.png\\" alt=\\"image.png\\" /></p>\n<p>To follow that, we have Henry Robalino who wrote <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zj\\" target=\\"_blank\\">Using Okta as an identity provider with Amazon MWAA</a></ins> which shows you how you can integrate an existing identity provider (in this instance, Okta) to authenticate via Amazon Web Services SSO to your Amazon Managed Workflows for Apache Airflow (Amazon MWAA) environments. Very nice post indeed. [hands on]</p>\n<h6><a id=\\"Apache_FlinkApache_Kafka_90\\"></a><strong>Apache Flink/Apache Kafka</strong></h6>\\n<p>Chinmayi Narasimhadevara walks you through how to create stream processing applications using Apache Flink, in this hands on post, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zk\\" target=\\"_blank\\">Query your Amazon MSK topics interactively using Amazon Kinesis Data Analytics Studio</a></ins>. Apache Flink is an open-source framework and engine for processing data streams, and in the post she walks you through creating some example applications to interact with streaming data from Apache Kafka in Scala, Python and SQL.</p>\n<h6><a id=\\"Apache_Ranger_94\\"></a><strong>Apache Ranger</strong></h6>\\n<p>Apache Ranger is an open source project that provides security capabilities for Hadoop clusters, and in this post <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zk\\" target=\\"_blank\\">Authorize SparkSQL data manipulation on Amazon EMR using Apache Ranger</a></ins> from Varun Rao Bhamidimarri and Jalpan Randeri, you will learn more about how Amazon EMR 6.4 has introduced additional authorising capabilities for data manipulation statements with Apache Ranger 2.0.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/4a4b1385a143447e98f73caaa78120ba_image.png\\" alt=\\"image.png\\" /></p>\n<h6><a id=\\"PostgreSQL_100\\"></a><strong>PostgreSQL</strong></h6>\\n<p>Naga Appani has put together this post, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/ze\\" target=\\"_blank\\">Tune sorting operations in PostgreSQL with work_mem</a></ins> where he talks about how and where to identify queries that are suboptimal, such as ones that require sorting on disk, and how to use various optimisation techniques to improve performance. What is work_mem? Read the post to find out what it is and why it is important in how you optimise your queries.</p>\n<h6><a id=\\"Amazon_Web_Services_CDK_104\\"></a><strong>Amazon Web Services CDK</strong></h6>\\n<p>Om Prakash Jha writes in his post, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zm\\" target=\\"_blank\\">Align with best practices while creating infrastructure using CDK Aspects</a></ins>, how if you are utilising Amazon CDK to provision your infrastructure, you can apply some best practices to not only to evaluate compliance of your resources against the best practices, but also modify their state to make them compliant before they are created.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/f3531bfe52f94f8fa0c5b2f4118ec1e5_image.png\\" alt=\\"image.png\\" /></p>\n<h5><a id=\\"Quick_updates_110\\"></a><strong>Quick updates</strong></h5>\\n<h6><a id=\\"Apache_Spark_111\\"></a><strong>Apache Spark</strong></h6>\\n<p>Amazon EMR now supports Apache Spark SQL to insert data into and update Apache Hive metadata tables when Apache Ranger integration is enabled. In January we launched Amazon EMR integration with Apache Ranger, a feature that allows you to define and enforce database, table, and column-level permissions when Apache Spark users access data in Amazon S3 through the Hive Metastore.</p>\n<p>Previously, with Apache Ranger is enabled, you were limited to only being able to read data using Spark SQL statements such as SHOW DATABASES and DESCRIBE TABLE. <strong>Now</strong>, you can also insert data into, or update the Apache Hive metadata tables with these statements: INSERT INTO, INSERT OVERWRITE, and ALTER TABLE.</p>\\n<h6><a id=\\"PostgreSQL_117\\"></a><strong>PostgreSQL</strong></h6>\\n<p>Amazon Relational Database Service (Amazon RDS) for PostgreSQL now supports PostGIS major version 3.1. This new version of PostGIS is available on PostgreSQL versions 13.4, 12.8, 11.13, 10.18, and higher.</p>\n<p>PostGIS allows you to store, query and analyse geospatial data within a PostgreSQL database. PostGIS 3.1 significantly improves performance such as spatial joins, which now run up to 6.8X faster on PostgreSQL 13. As an example, you could use a spatial join to count the number of people living in an area defined by the reception of mobile phones from radio towers.</p>\n<p>PostGIS 3.1 is the new default version on PostgreSQL 10 and higher starting with the new minor versions. However, you can still create older versions of PostGIS in your PostgreSQL database, e.g., if you require version stability.</p>\n<h6><a id=\\"Ubuntu_125\\"></a><strong>Ubuntu</strong></h6>\\n<p>Amazon EC2 now supports Hibernation for Ubuntu 20.04 LTS operating system. Hibernation allows you to pause your EC2 Instances and resume them at a later time, rather than fully terminating and restarting them. Resuming your instance lets your applications continue from where they left off so that you don’t have to restart your OS and application from scratch. Hibernation is useful for cases where rebuilding application state is time-consuming (e.g., developer desktops) or an application’s start-up steps can be prepared in advance of a scale-out.</p>\n<p>For Ubuntu 20.04 LTS, Hibernation is supported for On-Demand Instances running on C3, C4, C5, C5d, I3, M3, M4, M5, M5a, M5ad, M5d, R3, R4, R5, R5a, R5ad, R5d, T2, T3, and T3a with up to 150 GB of RAM.</p>\n<h5><a id=\\"Video_of_the_week_131\\"></a><strong>Video of the week</strong></h5>\\n<p>This week we have Michael Hausenblas’ session at the Swiss Cloud Native Day, where he gave this session How open source made observability mainstream.</p>\n<p><video src=\\"https://dev-media.amazoncloud.cn/79453f10a5764ae38acb24c0a2847c0e_How%20open%20source%20made%20observability%20mainstream%20-%20Michael%20Hausenblas%20-%20Swiss%20Cloud%20Native%20Day%202021.mp4\\" controls=\\"controls\\"></video></p>\\n<p>Following that we have this recorded workshop from Provectus, Amazon SageMaker and Open-Source Tools for ML: Better Together.</p>\n<p><video src=\\"https://dev-media.amazoncloud.cn/02425145c38542728d0c7d8957368947_Amazon%20SageMaker%20and%20Open-Source%20Tools%20for%20ML%EF%BC%9A%20Better%20Together.mp4\\" controls=\\"controls\\"></video></p>\\n<h4><a id=\\"Workshops_141\\"></a><strong>Workshops</strong></h4>\\n<h5><a id=\\"Hadoop_on_Kubernetes_142\\"></a><strong>Hadoop on Kubernetes</strong></h5>\\n<p>If you have not seen or worked through this workshop, <ins><a href=\\"https://aws-oss.beachgeek.co.uk/zn\\" target=\\"_blank\\">EMR on EKS Workshop</a></ins> then it provides a nice entry point into running some of the open source analytics tools such as Apache Spark on Kubernetes.</p>\n<h4><a id=\\"Events_for_your_diary_146\\"></a><strong>Events for your diary</strong></h4>\\n<p>Coming up later this week we have…</p>\n<h5><a id=\\"Container_Day_x_KubeCon_149\\"></a><strong>Container Day x KubeCon</strong></h5>\\n<h5><a id=\\"October_12th_800_AM__400_PM_PDT_150\\"></a><strong>October 12th, 8:00 AM – 4:00 PM (PDT)</strong></h5>\\n<p>The Amazon Web Services Kubernetes team is back for the next instillment of Container Day x KubeCon, a full day of sessions on Amazon EKS and Kubernetes at Amazon Web Services. In this live and virtual Twitch event, the Amazon Web Services Kubernetes team, hosted by developer advocates Justin Garrison and Jesse Butler, will be covering new launches, demoing products and features, and answering your questions. <ins><a href=\\"https://aws-oss.beachgeek.co.uk/cr\\" target=\\"_blank\\">Sign up</a></ins> to be reminded when things kick off.</p>\n<h5><a id=\\"GraphQL_API_security_best_practices_with_Amazon_Web_Services_154\\"></a><strong>GraphQL API security best practices with Amazon Web Services</strong></h5>\\n<h5><a id=\\"AppSync_and_Amazon_Web_Services_Amplify_155\\"></a><strong>AppSync and Amazon Web Services Amplify</strong></h5>\\n<h5><a id=\\"14th_October_11am_AEST_156\\"></a><strong>14th October, 11am AEST</strong></h5>\\n<p>As a developer, the most important parts of managing your applications should always include enhancing performance while strengthening security. In this webinar, we take you through security best practices for your GraphQL API’s with Amazon Web Services AppSync and Amazon Web Services Amplify, providing you with an understanding of how these can be applied to your applications. In this session, you will learn about:</p>\n<ul>\\n<li>GraphQL Protocol and how to configure a schema</li>\n<li>Possible ways to authenticate and authorise access to GraphQL APIs</li>\n<li>How to configure network security for your API</li>\n<li>How to enable observability for your API with logging, tracing or auditing</li>\n</ul>\\n<p>To <ins><a href=\\"https://aws-oss.beachgeek.co.uk/ue\\" target=\\"_blank\\">register for this event, use this link</a></ins>.</p>\n<h5><a id=\\"Flink_Forwards_Global_2021_167\\"></a><strong>Flink Forwards Global 2021</strong></h5>\\n<h5><a id=\\"October_26th27th_168\\"></a><strong>October 26th/27th</strong></h5>\\n<p>Flink Forward Global 2021 is a 2-day virtual conference for the Apache Flink and stream processing communities. Apache Flink is an open-source distributed engine for processing data streams that can support both streaming and batch workloads. Flink Forward has keynote presentations and talks on production Flink use cases, technical deep dive sessions, and the growth of the Flink ecosystem. You can meet core Flink committers, new and experienced users, and thought leaders who share experiences and best practices in stream processing, real-time analytics, and the management of mission-critical Flink deployments in production.</p>\n<p><ins><a href=\\"https://aws-oss.beachgeek.co.uk/wh\\" target=\\"_blank\\">Read more and sign up here</a></ins>.</p>\n<h5><a id=\\"Databricks__Amazon_Web_Services_Lakehouse_Dev_Day_Live_Workshop_174\\"></a><strong>Databricks | Amazon Web Services Lakehouse Dev Day Live Workshop</strong></h5>\\n<h5><a id=\\"November_16th_900_AM_PT_175\\"></a><strong>November 16th 9:00 AM PT</strong></h5>\\n<p>Delta Lake is an open source storage layer that provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. You can use Delta Lake on top of your existing data lake. During this workshop you will learn how to:</p>\n<ul>\\n<li>Make your existing Amazon S3 data lakes into a lakehouse with Delta Lake.</li>\n<li>Provide an easy-to-use platform for analysts to directly query data on your data lake using SQL Analytics</li>\n<li>Simplify and automate data pipelines for streaming and batch data to lower costs and boost productivity for your data teams</li>\n</ul>\\n<p><ins><a href=\\"https://aws-oss.beachgeek.co.uk/zs\\" target=\\"_blank\\">Read more and sign up here.</a></ins></p>\n<h4><a id=\\"Stay_in_touch_with_open_source_at_Amazon_Web_Services_185\\"></a><strong>Stay in touch with open source at Amazon Web Services</strong></h4>\\n<p>I hope this summary has been useful. Remember to check out the <ins><a href=\\"https://aws.amazon.com/opensource/?opensource-all.sort-by=item.additionalFields.startDate&amp;opensource-all.sort-order=asc\\" target=\\"_blank\\">Open Source homepage</a></ins> to keep up to date with all our activity in open source by following us on <ins><a href=\\"https://twitter.com/AWSOpen\\" target=\\"_blank\\">@Amazon Web ServicesOpen</a></ins></p>\n"}