Amazon Web Services open source newsletter, #203

Welcome to issue #203 of the Amazon Web Services open source newsletter, the newsletter where we try and provide you the best open source on Amazon Web Services content.As always, we have more great new projects to check out, which include how you can proxy OpenAI requests through [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail), security tools that help you stay one step ahead of bad actors, a way of implementing CDK Pipelines in a less opinionated way, a tool that helps you validate your Amazon IAM policies, a toolkit to help get you started with good practices when creating CloudFormation templates, some demo code that demonstrate how you can implement zero downtime updates to your applications, as well as some really cool demos and use cases of generative AI in action (too many to mention, so check them all out!). Once you have had time to take a look at the projects, there is still lots to check out. In this edition we feature content covering some of your favourite open source projects, including Cedar, Langfuse, Steampipe, Ray, Apache Spark, Amazon Amplify, Flutter, Valkey, O3DE, Amazon CDK, LangChain, Kubernetes, Argo Workflows, OpenTofu, Bottlerocket, Karpenter, OpenTelemetry, Apache Flink, Apache Pinot, Apache Kafka, openCypher, Apache Iceberg, Apache Airflow, MySQL, PostgreSQL, Deequ, OCSF, GraphStorm, OpenShift, [Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail), ActiveMQ, and Red Hat Enterprise Linux. As always, get in touch if you want me to feature your projects in this open source newsletter. Until the next time, I will leave you to dive into the good stuff! ### Latest open source projects *The great thing about open source projects is that you can review the source code. If you like the look of these projects, make sure you that take a look at the code, and if it is useful to you, get in touch with the maintainer to provide feedback, suggestions or even submit a contribution. The projects mentioned here do not represent any formal recommendation or endorsement, I am just sharing for greater awareness as I think they look useful and interesting!* ### Tools **bedrock-litellm** [bedrock-litellm](https://aws-oss.beachgeek.co.uk/43m?trk=cndc-detail) is an awesome project that provides a way of proxying requests in the OpenAI format, so that they will work with [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail). OpenAI is often one of the default options for integrating various generative AI tools and libraries, and now you have a way of being able to point those to use foundational models managed by [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail). It uses [litellm](https://www.litellm.ai/?trk=cndc-detail) to do this, and is deployed in a Kubernetes cluster. **aws-config-rule-rds-logging-enabled-remediation** [aws-config-rule-rds-logging-enabled-remediation](https://aws-oss.beachgeek.co.uk/43l?trk=cndc-detail) provides code that will help you checks if respective logs of Amazon Relational Database Service ([Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail)) are enabled using Amazon Config rule 'rds-logging-enabled'. The rule is NON_COMPLIANT if any log types are not enabled. Amazon Systems Manager Automation document used as remediation action to enable the logs export to CloudWatch for the RDS instances that are marked NON_COMPLIANT. **apeman** [apeman](https://aws-oss.beachgeek.co.uk/43n?trk=cndc-detail) is a new tool that helps security people to map and visualise Amazon Web Services identity attack paths. The README provides detailed instructions on how to get up and running, but I also found the blog post, [Project Apeman : A Comprehensive Guide To Graph-Based Amazon Security Analysis](https://aws-oss.beachgeek.co.uk/43o?trk=cndc-detail) very helpful. Daniel Heinsen talks more about this in depth in his talk at SpecterOps. **aws-mine** [aws-mine](https://aws-oss.beachgeek.co.uk/43k?trk=cndc-detail) is a project from Steven Smiley that should interest security folk. It provides a [honey](https://uk.norton.com/blog/iot/what-is-a-honeypot?trk=cndc-detail) token system for Amazon Web Services, that allows you to create Amazon Web Services access keys that can be placed in various places to tempt bad guys. If used, you will be notified within \~4 minutes. You can then investigate that asset to determine if it may have been compromised. [demo of aws-mine running](https://media.dev.to/cdn-cgi/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1j5l956bx0xzs2stvdrh.gif) **bedrock-embed-web** [bedrock-embed-web](https://aws-oss.beachgeek.co.uk/43i?trk=cndc-detail) is a new project from my colleague Olivier Leplus that makes it easier than ever to embed [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail) foundation models within a chat interface in your web applications.  **cdk-express-pipeline** [cdk-express-pipeline](https://aws-oss.beachgeek.co.uk/43g?trk=cndc-detail) is the latest project from Amazon Web Services Hero Rehan van der Merwe (who's project I use to track usage metrics of this newsletter!) which provides an alternative to those of you who want to use some of the features of Amazon CDK Pipelines, but perhaps need it less opinionated. CDK Express Pipelines is a library built on the Amazon CDK, allowing you to define pipelines in a CDK-native method. It leverages the CDK CLI to compute and deploy the correct dependency graph between Waves, Stages, and Stacks using the .addDependency method, making it build-system agnostic and an alternative to Amazon CDK Pipelines. Check out the clear documentation which will help you get started in no time. **improve-employee-productivity-using-genai** [improve-employee-productivity-using-genai](https://aws-oss.beachgeek.co.uk/43h?trk=cndc-detail) is an innovative code sample designed to elevate the efficiency and effectiveness of writing tasks through the integration of AI technologies. Tailored for Amazon Web Services users, this assistant example utilises [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail) and generative AI models to automate the creation of initial templates, that can also be customised for your own needs. Users can input both text and images, benefiting from the multimodal capabilities of cutting-edge AI like the Claude 3 family, which supports dynamic interaction with multiple data types. The README is very comprehensive and covers not only how to set up and configure this project, but also has lots of great short videos of it in action. Looks really cool, and a project for my weekend I think.  **validate-aws-policies** [validate-aws-policies](https://aws-oss.beachgeek.co.uk/42r?trk=cndc-detail) is a Python command line tool from Alejandro Velez to validate aws policies using boto3 and Access Analyzer API. You can check out his supporting blog post (including demo video) [Continuous Delivery applied to Authorization with IAM Identity Center and Amaozn IAM Access Analyzer – Part 1](https://aws-oss.beachgeek.co.uk/42s?trk=cndc-detail) that shows you how you can incorporate this tool as part of your CI/CD pipeline.  **aws-cloudformation-starterkit** [aws-cloudformation-starterkit](https://aws-oss.beachgeek.co.uk/434?trk=cndc-detail) is a new project from Amazon Web Services Community Builder Danny Steenman that should accelerate Amazon Web Services infrastructure deployment for CloudFormation users. It's designed for both beginners and seasoned pros, featuring quick CI/CD setup, multi-environment support, and automated security checks. Very nice repo, clear and detailed documentation, so make sure you check this project out. ### Demos, Samples, Solutions and Workshops **build-an-agentic-llm-assistant** [build-an-agentic-llm-assistant](https://aws-oss.beachgeek.co.uk/43e?trk=cndc-detail) this repo provides code that you can follow along as part of the "Build an agentic LLM assistant on Amazon Web Services" workshop. This hands-on workshop, aimed at developers and solution builders, trains you on how to build a real-life serverless LLM application using foundation models (FMs) through [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail) and advanced design patterns such as: Reason and Act (ReAct) Agent, text-to-SQL, and Retrieval Augemented Generation (RAG). It complements the [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail) Workshop by helping you transition from practicing standalone design patterns in notebooks to building an end-to-end llm serverless application. Check out the README for additional links to the workshop text, as well as more details on how this repo works.  **deploy-langfuse-on-ecs-with-fargate** [deploy-langfuse-on-ecs-with-fargate](https://aws-oss.beachgeek.co.uk/43j?trk=cndc-detail) This repository contains the Amazon CDK Python code for deploying the Langfuse application using [Amazon Elastic Container Registry ](https://aws.amazon.com/cn/ecr/?trk=cndc-detail)(ECR) and [Amazon Elastic Container Service](https://aws.amazon.com/cn/ecs/?trk=cndc-detail) (ECS). If you are not familiar with Langfuse, it is is an open-source LLM engineering platform that helps teams collaboratively debug, analyze, and iterate on their LLM applications.  **webapp-form-builder** [webapp-form-builder ](https://aws-oss.beachgeek.co.uk/43d?trk=cndc-detail)- this repo was built to accelerate the development of web forms on the frontend using the Amazon Cloudscape Design System. Cloudscape is an open source design system to create web applications. It was built for and is used by Amazon Web Services products and services. This solution provides you with a sample application that utilises components of the Cloudscape Design System that are commonly used in web-forms where users are required to input data. Check out the more expansive README for more details of how this works and how to get started.  **sample-code-for-a-secure-vault-using-aws-nitro-enclaves** [sample-code-for-a-secure-vault-using-aws-nitro-enclaves](https://aws-oss.beachgeek.co.uk/43c?trk=cndc-detail) - This repository contains a sample secure vault solution built using Amazon Nitro Enclaves, a feature available exclusively through the Amazon Nitro System hypervisor on supported [Amazon EC2 ](https://aws.amazon.com/cn/ec2/?trk=cndc-detail)instances. A vault solution is useful when you need to ensure sensitive data (such as Protected Health Information (PHI)/Personally Identifiable Information (PII)) is properly secured at rest and can only be decrypted through approved channels. Check out more details about how cool Nitro Enclaves are by reading the supporting documentation for this project, [About the Amazon Nitro Enclaves Vault](https://aws-oss.beachgeek.co.uk/43b?trk=cndc-detail).  **aws-genai-rfpassistant** [aws-genai-rfpassistant](https://aws-oss.beachgeek.co.uk/43a?trk=cndc-detail) this repository contains the code and infrastructure as code for a Generative AI-powered Request for Proposal (RFP) Assistant leveraging [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail) and Amazon Cloud Development Kit (CDK). This could be very hand if responding to RFP's is something that you do and you want to look at ways of optimising your approach. The documentation in the repo is very comprehensive. I have not tried this one out for myself, but I have been involved in both writing and reviewing RFPs in the past, so understand the pain that led to the creation of this project.  **zero-downtime-deployment-tofu** [zero-downtime-deployment-tofu](https://aws-oss.beachgeek.co.uk/438?trk=cndc-detail) is a repo from Amazon Web Services Community Build Jorge Tovar that contains code examples using OpenTofu that shows how you can achieve zero downtime using a number of different approaches. Check out the supporting blog post for more details, [Zero Downtime Deployment in Amazon Web Services with Tofu/Terraform and SAM](https://aws-oss.beachgeek.co.uk/439?trk=cndc-detail). This is this weeks essential repo to check out, and a good opportunity to learn about and become familiar with the different techniques and options you have. **streamlit-bedrock-claude-sample** [streamlit-bedrock-claude-sample](https://aws-oss.beachgeek.co.uk/437?trk=cndc-detail) - I have featured Gary Stafford's open source projects and blog posts regularly in this newsletter. Gary has built a number of simple Streamlit applications to make it easy access the latest models and features of Amazon Web Services [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail) as part of several talks, workshops, and demonstrations he has done. As part these, he has put together a simple Streamlit application that uses the [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail) boto3 Python SDK to call the latest Anthropic Claude 3 family of multimodal foundation models. The application accepts a system and user prompt and generates a text-based response. The Streamlit app can be easily modified to incorporate new Bedrock features or as a starting point for your own applications. **aws-chatbot-fargate-python** [aws-chatbot-fargate-python](https://aws-oss.beachgeek.co.uk/43f?trk=cndc-detail) is a new repo from Amazon Web Services Hero Ran Isenberg that deploys a Streamlit Chatbot in an Amazon Fargate-based ESC cluster web application using Amazon CDK (Cloud Development Kit). The infrastructure includes an ECS cluster, Fargate service, Application Load Balancer, VPC, and WAF and includes security best practices with CDK-nag verification. The chatbot is based on an implementation by Streamlit and the initial prompt is that the chatbot is me, Ran the builder, a serverless hero and attempts to answer as me. The Chatbot uses custom domain (you can remove it or change it to your own domain) and assume an OpenAI token exists in the account in the form of a secrets manager secret for making API calls to OpenAI. **my-steampipe-arsenal** [my-steampipe-arsenal](https://aws-oss.beachgeek.co.uk/436?trk=cndc-detail) Sometimes GitHub Gists or snippets are super useful, and Gabriel Soltz shares one such useful snippet in his repo of Steampipe queries that allows you to check for inactive Route53 alias'. Check out some of the other ones he shares too. ### Amazon Web Services and Community blog posts **This weeks essential reading** * [The Amazon SDK for Java 1.x is in maintenance mode, effective July 31, 2024](https://aws-oss.beachgeek.co.uk/41y?trk=cndc-detail) essential you read this if you are using the v1.x version of the Amazon SDK for Java to find out what your options are, including test out the... * [Preview Release of the Migration Tool for the Amazon SDK for Java 2.x](https://aws-oss.beachgeek.co.uk/42g?trk=cndc-detail) looks at how this tool might be able to help in migrating your 1.x application to 2.x. \[hands on] * [Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2](https://aws-oss.beachgeek.co.uk/423?trk=cndc-detail) provides a fantastic look into what it takes to migrate from Apache Spark over to Ray at scale, and how this helped reduce both data processing time and cost - essential reading  **What I am reading from around the Community** Each week I spent a lot of time reading posts from across the Amazon Web Services community on open source topics. In this section I share what personally caught my eye and interest, and I hope that many of you will also find them interesting. It was really tough though, as there were so many great posts. Starting things off this week is Muhammed Salih Guler, who shares how to build and deploy a Flutter web application using Amazon Amplify in his post, [Deploy a Flutter Web Application Using Amazon Amplify](https://aws-oss.beachgeek.co.uk/42p?trk=cndc-detail). One of my colleagues (Derek Bingham) has put together a bunch of Flutter content, and I have been meaning to try this out. I am going to give this a try over the weekend, so will let you know how I get on (watch out via socials). I have been writing and play with Valkey over the past few weeks (check out some of my blog posts on this topic) and so found Dan Touitou's deep dive, [Unlock 1 Million RPS: Experience Triple the Speed with Valkey](https://aws-oss.beachgeek.co.uk/42q?trk=cndc-detail), super interesting. Valkey 8.0 is available for folks to play with, and hopefully will go GA soon. I always have time to read up on robotics and IoT, so Mike's latest post, [Simulating Robots in the Cloud with EC2 and O3DE](https://aws-oss.beachgeek.co.uk/42u?trk=cndc-detail), shows you how you can speed up your robotic development using simulators, in this instance the Open 3D Engine (O3DE) open source project. Go check it out. Next up we have a series of great blog posts covering all things Amazon CDK from Pahud Hsieh. From diving deep into how grantRead works in the post, [What happens when bucket.grantRead() in Amazon CDK](https://aws-oss.beachgeek.co.uk/42t?trk=cndc-detail), or how you can begin contributing to the Amazon CDK project in [Contributing to Amazon CDK](https://aws-oss.beachgeek.co.uk/42v?trk=cndc-detail), dive deeper into more advanced CDK topics such as a look at some real-world use cases to override L1 props from L2 constructs in [Customize CDK L1 construct props from L2](https://aws-oss.beachgeek.co.uk/42w?trk=cndc-detail). Pahud has also embraced generative AI, and shows how using tools like [Amazon Q](https://aws.amazon.com/cn/q/?trk=cndc-detail) Developer (not open source sadly), you can use it to both create your CDK applications in [Writing CDK like a Pro using Amazon Q Customizations](https://aws-oss.beachgeek.co.uk/42x?trk=cndc-detail), or how to build L2 CDK Construct Methods in [Generate CDK Construct Methods Using Amazon Q Customizations](https://aws-oss.beachgeek.co.uk/42y?trk=cndc-detail). It's not only Pahud that has been creating CDK content though, my colleague Abhishek Gupta has also been exploring this topic in his latest blogging adventure, [Use AWS Generative AI CDK constructs to speed up app development](https://aws-oss.beachgeek.co.uk/42z) where he explores how to use the AWS Generative AI Constructs Library to deploy a complete RAG application. I guess that is a good segway into a series of posts covering the latest open source libraries and frameworks for generative AI. First up is Michele Ricciardi who shares how to enable prompt versioning and management, using LangChain and [Amazon Bedrock](https://aws.amazon.com/cn/bedrock/?trk=cndc-detail) Prompt management in his post, [Integrate Amazon Bedrock Prompt Management in LangChain applications](https://aws-oss.beachgeek.co.uk/430?trk=cndc-detail). Next up is a post from John Wick (yup, there was no way I was NOT going to post this right!) who shares how to work with generative AI in Javascript in his post, [GenAI, from your Local machine to Amazon Web Services with LangChain.js](https://aws-oss.beachgeek.co.uk/431?trk=cndc-detail). We conclude this round up with some cloud native posts, starting with Marcelo Andrade who shares how to find out which Kubernetes Pod owns a specific container in his post [From which Kubernetes pod (and namespace!) is this process that I see on my host? ](https://aws-oss.beachgeek.co.uk/432?trk=cndc-detail). I love these kinds of posts as they cover the things that I always need to know at some point (so I have bookmarked this for future reference). Amazon Web Services Community Builder Saifeddine Rajhi shares how to use and run Argo Workflows to orchestrate jobs within your Kubernetes environments in the post, [Data on Kubernetes: Part 4 - Argo Workflows: Simplify parallel jobs : Container-native workflow engine for Kubernetes](https://aws-oss.beachgeek.co.uk/433?trk=cndc-detail). This is part of a series which I found myself scrolling through, so check those other posts out too. Finally, we have Amazon Web Services Community Builder Sunny Bhambhani shares how to get multiple helm charts installed in your Kubernetes clusters in his post, [Installing multiple helm charts in one go using simple bash utility](https://aws-oss.beachgeek.co.uk/435?trk=cndc-detail). **Cloud Native** * [Automate monitoring for your Amazon EKS cluster using CloudWatch Container Insights](https://aws-oss.beachgeek.co.uk/41z?trk=cndc-detail) presents an event-driven, automation solution for monitoring your [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) cluster using [Amazon CloudWatch](https://aws.amazon.com/cn/cloudwatch/?trk=cndc-detail) Container Insights metrics as well as bunch of other Amazon services and resources \[hands on]  * [Diving into OCI Image and Distribution 1.1 Support in Amazon ECR](https://aws.amazon.com/blogs/opensource/diving-into-oci-image-and-distribution-1-1-support-in-amazon-ecr/?trk=cndc-detail) looks into how Amazon ECR is one of the first registry cloud services to support the Referrers API and OCI 1.1 (read the post to find out what that means!) * [Cordial’s journey implementing Bottlerocket and Karpenter in Amazon EKS](https://aws-oss.beachgeek.co.uk/42e?trk=cndc-detail) is a case study in how Cordial implemented Bottlerocket as the Operating System (OS) and Karpenter as the Node Lifecycle Manager within their [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) environments to achieve operational efficiency and improve security posture (graphs ahoy!) * [How Getir optimized their Amazon EKS compute using Karpenter](https://aws-oss.beachgeek.co.uk/42h?trk=cndc-detail) provides another case study, this time Getir, and how Karpenter helped them address the challenges of spiky workloads, which require faster responses for dynamic scaling * [Developing Custom Processors using OpenTelemetry in .NET 8](https://aws-oss.beachgeek.co.uk/42f?trk=cndc-detail) explores how to develop a custom processor using OpenTelemetry in the .NET ecosystem \[hands on] * [Making sense of secrets management on Amazon EKS for regulated institutions](https://aws-oss.beachgeek.co.uk/42m?trk=cndc-detail) provides a summary of three options for managing secrets in [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) - ESO, Sealed Secrets, and Amazon Secrets and Configuration Provider (ASCP), and the key considerations for FSI customers when choosing between them * [How to Run WebAssembly on Amazon EKS](https://aws-oss.beachgeek.co.uk/42n?trk=cndc-detail) is a step by step guid on how Amazon Web Services empowers users to harness the full potential of Wasm by providing a seamless integration with [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) \[hands on] **Data and Analytics** * [Build a real-time analytics solution with Apache Pinot on Amazon Web Services](https://aws-oss.beachgeek.co.uk/42c?trk=cndc-detail) shows how to deploy a scalable Apache Pinot-based near real-time user facing analytics solution on Amazon Web Services \[hands on] * [Stream change data in a multicloud environment using Amazon DMS, Amazon MSK, and Amazon Managed Service for Apache Flink](https://aws-oss.beachgeek.co.uk/42i?trk=cndc-detail) dives deep into architecture, approaches, and considerations for streaming data changes from the transactional databases deployed in other cloud providers to a streaming data solution deployed on Amazon Web Services \[hands on]  * [Improve the resilience of Amazon Managed Service for Apache Flink application with system-rollback feature](https://aws-oss.beachgeek.co.uk/42k?trk=cndc-detail) looks at how to use the system-rollback feature in Managed Service for Apache Flink, and how this functionality improves your application’s resilience by providing a highly available Flink application \[hands on] * [Build and deploy knowledge graphs faster with RDF and openCypher](https://aws-oss.beachgeek.co.uk/42j?trk=cndc-detail) shows you how to use openCypher with RDF graphs \[hands on] * [Monitoring Apache Iceberg metadata layer using Amazon Lambda, Amazon Glue, and Amazon CloudWatch](https://aws-oss.beachgeek.co.uk/41w?trk=cndc-detail) discusses how the metadata layer of Apache Iceberg can be used to make data lakes more efficient. You will learn about an open-source solution that can collect important metrics from the Iceberg metadata layer \[hands on]  * [Integrate Amazon MWAA with Microsoft Entra ID using SAML authentication](https://aws-oss.beachgeek.co.uk/420?trk=cndc-detail) illustrates how to configure an Amazon MWAA environment deployed in private network access mode with customer managed VPC endpoints and authenticate users using SAML federated identity using Microsoft Entra ID and Application Load Balancer (ALB) \[hands on]  * [Optimize cost and performance for Amazon MWAA](https://aws-oss.beachgeek.co.uk/421?trk=cndc-detail) provides guidance on how you can optimise performance and save cost of your Apache Airflow environments by following some good practices \[hands on] * [How Amazon GTTS runs large-scale ETL jobs on Amazon Web Services using Amazon MWAA](https://aws-oss.beachgeek.co.uk/42a?trk=cndc-detail) is a great case study on how Amazon GTTS were able to see 70% improvement in end-to-end runtime on their ETL jobs, among other benefits - a great read  * [Synchronize data lakes with CDC-based UPSERT using open table format, Amazon Glue, and Amazon MSK](https://aws-oss.beachgeek.co.uk/422?trk=cndc-detail) illustrates the construction of a comprehensive CDC system, enabling the processing of CDC data sourced from an [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) for MySQL source \[hands on] * [Improve Apache Kafka scalability and resiliency using Amazon MSK tiered storage](https://aws-oss.beachgeek.co.uk/425?trk=cndc-detail) dives deep into how tiered storage helps with faster broker recovery and quicker partition migrations, facilitating faster load balancing and broker scaling in your Apache Kafka clusters (plenty of graphs in this one!) * [Amazon Glue mutual TLS authentication for Amazon MSK](https://aws-oss.beachgeek.co.uk/42d?trk=cndc-detail) describes how to set up Amazon Glue jobs to produce, consume, and process messages on an MSK cluster using mutual TLS authentication \[hands on] * [Migrate an on-premises MySQL database to Amazon Aurora MySQL over a private network using Amazon DMS homogeneous data migration and Network Load Balancer](https://aws-oss.beachgeek.co.uk/427?trk=cndc-detail) guides you through the steps of performing a homogeneous migration from an on-premises MySQL database to [Amazon Aurora](https://aws.amazon.com/cn/rds/aurora/?trk=cndc-detail) MySQL using Amazon DMS homogeneous data migrations over a private network using network load balancer \[hands on] * [Implement a rollback strategy after an Amazon Aurora MySQL blue/green deployment switchover](https://aws-oss.beachgeek.co.uk/42b?trk=cndc-detail) looks at the steps needed to perform a blue/green deployment switchover and how to set up and perform a rollback strategy post switchover for [Amazon Aurora](https://aws.amazon.com/cn/rds/aurora/?trk=cndc-detail) MySQL-Compatible Edition \[hands on] * [Build a serverless data quality pipeline using Deequ on Amazon Lambda](https://aws-oss.beachgeek.co.uk/42l?trk=cndc-detail) shows how to run Deequ (an open source framework for data quality) on Lambda \[hands on] * [Build a custom HTTP client in Amazon Aurora PostgreSQL and Amazon RDS for PostgreSQL: An alternative to Oracle’s UTL_HTTP](https://aws-oss.beachgeek.co.uk/42o?trk=cndc-detail) is a neat post that demonstrates how you can use PL/pgSQL custom wrapper functions to convert Oracle UTL_HTTP referenced custom code to the [Amazon Aurora](https://aws.amazon.com/cn/rds/aurora/?trk=cndc-detail) PostgreSQL and [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) for PostgreSQL equivalent \[hands on] **Other posts to check out** * [From Data Chaos to Cohesion: How OCSF is Optimizing Cyber Threat Detection](https://aws-oss.beachgeek.co.uk/429?trk=cndc-detail) is an in depth overview of the OCSF’s progress since its inception in 2022, and share insights from organisations participating in the project and adopting OCSF * [GraphStorm 0.3: Scalable, multi-task learning on graphs with user-friendly APIs](https://aws-oss.beachgeek.co.uk/426?trk=cndc-detail) provides an overview of what GraphStorm is and how it can help you tackle your large-scale graph ML challenges \[hands on] * [Installing Red Hat OpenShift on Amazon Web Services in a restricted network using Amazon Secure Token Service](https://aws-oss.beachgeek.co.uk/41v?trk=cndc-detail) is a detailed walk through on how to install a Red Hat OpenShift Container Platform cluster in a restricted [Amazon VPC](https://aws.amazon.com/cn/vpc/?trk=cndc-detail) network to comply with regulatory requirements and organisational controls \[hands on]  * [How DXC creates application consistent EBS Snapshots for Linux](https://aws-oss.beachgeek.co.uk/428?trk=cndc-detail) looks at how DXC and Amazon Web Services collaborated to implement a serverless solution to create application consistent backups for Linux instances \[hands on] ### Quick updates **Karpenter** Karpenter’s APIs graduate out of beta with the release of Karpenter version 1.0.0. Karpenter is a flexible, efficient, and high-performance Kubernetes compute management solution that helps improve application availability, reduce operational overhead, and increase cluster compute utilisation. This release also includes three new features which provide customers greater control over how and when Karpenter disrupts Kubernetes applications. Customers can use Karpenter with [Amazon Elastic Kubernetes Service](https://aws.amazon.com/cn/eks/?trk=cndc-detail) (EKS) or any conformant Kubernetes cluster. Like other Kubernetes open-source projects, Karpenter’s APIs follow a maturity progression from alpha to beta then stable. In October 2023, the Karpenter project graduated its APIs from alpha to beta. This release marks the final milestone in the project’s maturity and customers can be assured that all Karpenter APIs will remain available in future 1.0 minor versions and not modified in any way that results in breaking changes. Alongside the graduation from beta, this 1.0 release includes three new features for Karpenter: 1/ the ability to specify disruption reasons, e.g. underutilisation, emptiness, drift, for disruption budgets, 2/ a forceful disruption mode that helps customers balance application availability against security requirements, and 3/ an expansion of consolidateAfter which lets customers better tune Karpenter’s consolidation feature to meet their cost-efficiency and application availability requirements. Alex Kestner, Robert Northard, and Rajdeep Saha share more details in the post, [Announcing Karpenter 1.0](https://aws-oss.beachgeek.co.uk/41u) **MySQL** Amazon Relational Database Service ([Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail)) for MySQL now supports MySQL minor version 8.0.39. We recommend that you upgrade to the latest minor versions to fix known security vulnerabilities in prior versions of MySQL, and to benefit from the bug fixes, performance improvements, and new functionality added by the MySQL community. Learn more about the enhancements in RDS for MySQL 8.0.39 in the [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) user guide. [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) for MySQL now supports version 8.4 in the [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) Database Preview Environment, allowing you to evaluate the latest Long-Term Support Release on [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) for MySQL. You can deploy MySQL 8.4 in the [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) Database Preview Environment that has the benefits of a fully managed database, making it simpler to set up, operate, and monitor databases. MySQL 8.4 is the latest Long-Term Support Release from the MySQL community. **PostgreSQL** There were a number of PostgreSQL related updates since the last newsletter. Here are the highlights: [Amazon Aurora](https://aws.amazon.com/cn/rds/aurora/?trk=cndc-detail) PostgreSQL-Compatible Edition now supports PostgreSQL versions 16.3, 15.7, 14.12, 13.15, and 12.19. These releases contain product improvements and bug fixes made by the PostgreSQL community, along with Aurora-specific improvements. Databases now startup faster after upgrades and restarts. Version 16.3 with IO-Optimized configuration includes performance enhancements that improve write throughput for 8xl and larger instances. These releases also contain Babelfish’s new features and improvements such as support for group AD, logical replication, Blue/Green Deployments, and LIKE operator for AI collations. As a reminder, [Amazon Aurora](https://aws.amazon.com/cn/rds/aurora/?trk=cndc-detail) PostgreSQL 12 support ends on Feb 29, 2025. Upgrade to a newer major version. You can initiate a minor version upgrade by modifying your DB cluster. [Amazon Aurora](https://aws.amazon.com/cn/rds/aurora/?trk=cndc-detail) PostgreSQL-Compatible Edition now supports pgvector 0.7.0, an open-source extension for PostgreSQL for storing vector embeddings in your database. pgvector provides vector similarity search capabilities enabling Aurora usage for semantic search and retrieval-augemented generation (RAG) in generative artificial intelligence (AI) applications. pgvector 0.7.0 adds parallelism to improve the Hierarchical Navigable Small Worlds (HNSW) index build time in Aurora. pgvector 0.7.0 adds two new vector data types: halfvec for storing dimensions as 2-byte floats, and sparsevec for storing up to 1,000 nonzero dimensions, and now supports indexing binary vectors using the PostgreSQL-native bit type. These additions let you use scalar and binary quantization for the vector data type using PostgreSQL expression indexes, which reduces index storage size and lowers index build time. Quantization also lets you increase the maximum dimensions of vectors you can index: 4,000 for halfvec and 64,000 for binary vectors. pgvector 0.7.0 is available in [Amazon Aurora](https://aws.amazon.com/cn/rds/aurora/?trk=cndc-detail) clusters running PostgreSQL 16.3, 15.7, 14.12, 13.15, and 12.19 and higher in all applicable Amazon Regions. [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) for PostgreSQL 17 Beta 3 is now available in the [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) Database Preview Environment, allowing you to evaluate the pre-release of PostgreSQL 17 on [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) for PostgreSQL. You can deploy PostgreSQL 17 Beta 3 in the [Amazon RDS](https://aws.amazon.com/cn/rds/?trk=cndc-detail) Database Preview Environment that has the benefits of a fully managed database. PostgreSQL 17 includes updates to vacuuming that reduces memory usage, improves time to finish vacuuming, and shows progress of vacuuming indexes. With PostgreSQL 17, you no longer need to drop logical replication slots when performing a major version upgrade. PostgreSQL 17 continues to build on the SQL/JSON standard, adding support for `JSON_TABLE` features that can convert JSON to a standard PostgreSQL table. The `MERGE` command now supports the `RETURNING` clause, letting you further work with modified rows. PostgreSQL 17 also includes general improvements to query performance and adds more flexibility to partition management with the ability to SPLIT/MERGE partitions. Amazon Relational Database Service (RDS) for PostgreSQL now supports the latest minor versions 16.4, 15.8, 14.13, 13.16, and 12.20. This release of RDS for PostgreSQL also includes updates for PostgreSQL extensions such as pgvector 0.7.3, hypopg 1.4.1, and others. **[Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail)** [Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail) 7.2 release is now generally available and includes Apache Spark 3.5.1, Trino 436, and PrestoDB 0.285, Apache Iceberg 1.5.0 and Delta 3.1. Furthermore, with [Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail) 7.2, you can view additional [Amazon CloudWatch](https://aws.amazon.com/cn/cloudwatch/?trk=cndc-detail) metrics for enhanced monitoring in the [Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail) console, which provides comprehensive monitoring capabilities, allowing you to track the performance and health of your cluster more effectively. You can configure the [Amazon CloudWatch](https://aws.amazon.com/cn/cloudwatch/?trk=cndc-detail) Agent to publish metrics for Apache Hadoop, YARN, and Apache HBase applications running on your [Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail) on EC2 clusters, and track the metrics of each cluster within the EMR console. In addition, [Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail) 7.2 adds support for Apache Flink Operator 1.8 with [Amazon EMR](https://aws.amazon.com/cn/emr/?trk=cndc-detail) on EKS. **ActiveMQ** [Amazon MQ](https://aws.amazon.com/cn/amazon-mq/?trk=cndc-detail) now supports ActiveMQ minor version 5.18, which introduces several improvements and fixes compared to the previous version of ActiveMQ supported by [Amazon MQ](https://aws.amazon.com/cn/amazon-mq/?trk=cndc-detail). These enhancements include initial support for the JMS 2.0 simplified APIs, such as JMSContext, JMSProducer, and JMSConsumer, as well as the implementation of methods for XA transactions. Starting from ActiveMQ 5.18, [Amazon MQ](https://aws.amazon.com/cn/amazon-mq/?trk=cndc-detail) will manage patch version upgrades for your brokers. All brokers on ActiveMQ version 5.18 will be automatically upgraded to the next compatible and secure patch version in your scheduled maintenance window. If you are utilising prior versions of ActiveMQ, such as 5.17, 5.16, or 5.15, we strongly recommend you to upgrade to ActiveMQ 5.18. You can easily perform this upgrade with just a few clicks in the Amazon Management Console. **Red Hat Enterprise Linux** [Amazon AppStream 2.0](https://aws.amazon.com/cn/appstream2/?trk=cndc-detail) now offers support for Red Hat Enterprise Linux, enabling ISVs and central IT organisations to stream Red Hat Enterprise Linux apps and desktops to users while leveraging the flexibility, scalability, and cost-effectiveness of the Amazon Cloud. With this launch, customers have the flexibility to choose from a broader set of operating systems including Red Hat Enterprise Linux, Amazon Linux 2, and Microsoft Windows. This launch enables organisations operating in highly regulated industries to accelerate time to market, scale resources up or down with demand, and manage the entire fleet centrally through the Amazon Console by streaming Red Hat Enterprise Linux apps from AppStream 2.0. Red Hat Enterprise Linux on AppStream 2.0 also enables traditional desktop apps to be converted to a SaaS delivery model without the cost of refactoring, while pay-as-you-go billing and license-included images ensure you only pay for the resources you use. Red Hat Enterprise Linux is currently supported in all Amazon Regions where AppStream 2.0 is available. Red Hat Enterprise Linux-based instances use per second billing (with a minimum of 15 minutes). Check out this related post, [Announcing Red Hat Enterprise Linux 8 on Amazon WorkSpaces and Amazon AppStream 2.0](https://aws-oss.beachgeek.co.uk/41x?trk=cndc-detail) ### Videos of the week **Getting Started with OpenTofu in Amazon Web Services** In this video Chris Hill shows you how to provision infrastructure, manage state effectively, and implement best practices for automation. We'll also cover importing existing resources into OpenTofu to optimise your cloud management. You can follow along with supporting code. **An Intro to Amazon Deployments with OpenTofu, Scalr, & GitHub!** Keeping with the OpenTofu theme, David Hundley does a live coding session that covers using OpenSearch as a vector store for a RAG solution, using OpenTofu as the infrastructure as code layer to simplify deployment. **Cedar Policies: Default Deny and Avoiding Evaluation Errors with Validation** A new video from Cedar Craft, a series of brilliant short videos that walk you through key concepts of Cedar and then shows you how these work with hands on code examples. In this latest video, they look at Policy errors. Policies that error can be difficult to reason about. We should a small example here where two seemingly equivalent policies give different answers if you make the "same" mistake in both. The fix is easy though: always validate your policies! ### Events for your diary If you are planning any events in 2024, either virtual, in person, or hybrid, get in touch as I would love to share details of your event with readers. **Open Source Summit**\ **September 16-18th, Vienna, Austria** Come join my colleagues at the Amazon Web Services booth at the Open Source Summit Europe, which is being held in the wonderful city of Vienna. There will be a bunch of around, doing talks, open source technology demos, and just hanging out with the open source community. Be great to see some of you there. **All Things Open**\ **27-29th October, Raleigh, North Carolina** Make sure you attend All Things Open this coming Autumn if you can, it is one of my favourite tech conferences, with the amazing community that comes year in, year out. As always my colleagues will be manning the Amazon Web Services booth, and I am sure we will have some cool stuff and SWAG to share with the community. Check out and grab your ticket while they are still available at [2024.allthingsopen.org](https://2024.allthingsopen.org/?trk=cndc-detail) **Manchester Tech Festival**\ **29th October** I will be speaking about Cedar at the [Manchester Tech Festival](https://www.manchestertechfestival.co.uk/cyber-security/?trk=cndc-detail) in October. Looking forward to sharing info about Cedar with the local developers attending. Hope to see some of you there. **Cortex**\ **Every other Thursday, next one 16th February** The Cortex community call happens every two weeks on Thursday, alternating at 1200 UTC and 1700 UTC. You can check out the GitHub project for more details, go to the [Community Meetings](https://aws-oss.beachgeek.co.uk/2h5?trk=cndc-detail) section. The community calls keep a rolling doc of previous meetings, so you can catch up on the previous discussions. Check the [Cortex Community Meetings Notes](https://aws-oss.beachgeek.co.uk/2h6?trk=cndc-detail) for more info. **OpenSearch**\ **Every other Tuesday, 3pm GMT** This regular meet-up is for anyone interested in OpenSearch & Open Distro. All skill levels are welcome and they cover and welcome talks on topics including: search, logging, log analytics, and data visualisation. Sign up to the next session, [OpenSearch Community Meeting](https://aws-oss.beachgeek.co.uk/1az?trk=cndc-detail) ### Celebrating open source contributors The articles and projects shared in this newsletter are only possible thanks to the many contributors in open source. I would like to shout out and thank those folks who really do power open source and enable us all to learn and build on top of what they have created. So thank you to the following open source heroes: Jorge Tovar, Gabriel Soltz, Eduardo Monich Fronza, Rishit G Barochia, Steve Mirman, Sunil Gajula, Michael Greenshtein, Dan Garibay, Matthew Miller, Manish Garg, Navdeep Pareek, Sushant Jagdale, Satya Chikkala, Vijay Velpula, Sriharsh Adari, Jeetendra Vaidya, Retina Satish, Shubham Purwar, Nitin Kumar, Patrick Ames, Jules Damji, Zhe Zhang, Jesse Butler, Michael Brown, Sai Maddali, Masudur Rahaman Sayem, Nagarjuna Koduru, Xiang Song, Florian Saupe, Jian Zhang, Alok Srivastava, A. Mosaad, Asif Fouzi, Prahal N P, Daxeshkumar Patel, Bhavesh Rathod, Kamal Singh, Raj Ramasubbu, Francisco Morillo, Ismail Makhlouf, Edward Ondari, Emmanuel Mashandudze, Sai Charan Teja Gopaluni, Daniel Schaaff, Martin Guy Lapointe, Neelendra Bhandari, Richard Kelly, Nisha Gambhir, Bibhuti Sahu, Mayank Kumar, Simran Singh, Amandeep Bajwa, Deevanshu Budhiraja, Sasha Luthra, Ora Lassila, Charles Ivie, Willem Broekema, Michael Schmidt,Piyush Mattoo, Chetan Pawar, Ruy Cavalcanti, Alex Koenig, David Surey, Bhanu Ganesh Gudivada, Vamsikrishna Jammula, Rajeshkumar Sabankar, Sumana Yanamandra, Alejandro Velez, Danny Steenman, Gary Stafford, Muhammed Salih Guler, Dan Touitou, Pahud Hsieh, Abhishek Gupta, Michele Ricciardi, John Wick, Saifeddine Rajhi, Sunny Bhambhani, and Marcelo Andrade. **Feedback** Please please please take 1 minute to [complete this short survey](https://www.pulse.aws/promotion/10NT4XZQ). ### Stay in touch with open source at Amazon Web Services Remember to check out the [Open Source homepage](https://aws.amazon.com/opensource/?opensource-all.sort-by=item.additionalFields.startDate\\&opensource-all.sort-order=asc) for more open source goodness. One of the pieces of feedback I received in 2023 was to create a repo where all the projects featured in this newsletter are listed. Where I can hear you all ask? Well as you ask so nicely, you can meander over to[ newsletter-oss-projects](https://aws-oss.beachgeek.co.uk/3l8?trk=cndc-detail). Made with ♥ from DevRel