Leverage Amazon secrets stores from EKS Fargate with External Secrets Operator

{"value":"Secrets management is a challenging but critical aspect of running secure and dynamic containerized applications at scale. To support this need to securely distribute secrets to running applications, Kubernetes provides native functionality to manage secrets in the form of [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/). However, many customers choose to centralize the management of secrets outside of their Kubernetes clusters by using external secret stores such as [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) to improve the security, management, and auditability of their secret usage.\n\nConsuming secrets from external secret stores often requires modifications to your application code to support API-based calls to the external store to retrieve a secret at application run time. When running applications on [Amazon EKS](https://aws.amazon.com/eks/), you can use Kubernetes’s flexibility to expose secrets directly to pods without requiring any application code changes. One example of how to accomplish this is by using the [AWS Secrets and Configuration Provider](https://github.com/aws/secrets-store-csi-driver-provider-aws) (ASCP) for the [Kubernetes Secrets Store CSI Driver](https://secrets-store-csi-driver.sigs.k8s.io/). ASCP uses the Secrets Store CSI driver to expose secrets from [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) to your pods as a [mounted storage volume](https://aws.amazon.com/blogs/security/how-to-use-aws-secrets-configuration-provider-with-kubernetes-secrets-store-csi-driver/).\n\nCustomers using [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) to orchestrate their applications often will use [AWS Fargate](https://aws.amazon.com/fargate/) as their compute layer to reduce the management complexity of operating their containerized workloads. For customers who have EKS clusters with [AWS Fargate](https://aws.amazon.com/cn/fargate/?trk=cndc-detail) nodes, a different method of consuming external secrets will be required since ASCP with the Secrets Store CSI is deployed to your EKS cluster as a daemonset. As of today, daemonsets are not supported on Fargate.\n\nOne such option is the open-source [External Secrets Operator](https://github.com/external-secrets/external-secrets) (External Secrets) project. External Secrets manages your secrets in a different manner from the Secrets Store CSI driver. Instead of mounting secrets as volumes, External Secrets reads secrets from your external secret store and automatically stores the values as native Kubernetes Secrets in the Kubernetes control plane. External Secrets is installed as a deployment on your cluster and works with [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) Fargate-only clusters as well as those with Amazon EC2-based nodes.\n\nIn this post, I walk through using the External Secrets Operator on an EKS Fargate cluster to consume secrets stored in [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail). This same method could apply to secrets stored in an [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) but will not be covered directly in this post.\n\n### **Solution overview**\nExternal Secrets will be deployed to your EKS cluster as a standard Kubernetes deployment, allowing Fargate to be used as your container compute provider. Once configured, External Secrets will sync defined secrets from [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) back to your EKS cluster. The secrets will be automatically synced on a recurring basis to capture any updates or changes, such as periodic credential rotations.\n\nThe following diagram outlines how the sync process will occur.\n\n1. External Secrets will make API calls to [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) on a recurring basis to copy specified secrets values.\n2. External Secrets will take the copied values and create native Kubernetes Secrets based on the external secret values.\n3. Kubernetes Secrets will be available for consumption by specified applications running on your EKS cluster.Kubernetes RBAC will define what applications can consume the secrets.\n4. Pods will consume the secrets as either volume mounts or environment variables as defined in the pod specifications.\n\n\n\n*Figure 1. Solution overview*\n\n### **Walkthrough**\nThis walkthrough will guide you through the steps of using the External Secrets Operator to sync a secret from [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) to your EKS Fargate cluster, then consuming that secret in an application pod running on Fargate.\n\nThe steps to be outlined in the following sections include:\n\n- Deploying External Secrets Operator on a EKS Fargate cluster\n- Configuring External Secrets Operator resources\n- Consuming synced Kubernetes Secret in an application pod\n\n\n#### **Prerequisites**\nFor this walkthrough, you should have the following prerequisites:\n\n- An AWS account\n- An IAM policy with permissions to [retrieve a secret from Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/auth-and-access_examples.html)\n- Your [secret stored in Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/manage_create-basic-secret.html)\n- An existing [Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html)\n- AWS [CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) installed\n- [kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html), [Helm](https://helm.sh/docs/intro/install/), and [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html) installed\n\n\n#### **Step 1. Create an EKS Fargate profile**\nWith [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail), [Fargate profiles](https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html) allow administrators to specify which pods in your cluster should be designated to run on Fargate. Profile selectors declare namespaces and optionally labels that will be evaluated against to determine if a pod should be scheduled on Fargate. Fargate profiles can be created via numerous CLI and Infrastructure as Code (IaC) methods. The following example will showcase the eksctl method. To create your Fargate profile using the AWS console, see [creating a Fargate profile](https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html#create-fargate-profile) in the [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) user guide.\n\nExternal Secrets will, by default, deploy its resources in a namespace called “external-secrets.” To ensure our External Secrets pods run on Fargate, we will specify this namespace in our Fargate profile selector.\n\nFrom the terminal where you have eksctl installed, run the following command to create your Fargate profile. Replace “CLUSTERNAME” with the name of your EKS cluster.\n\n```\\neksctl create fargateprofile \\\\\\n --cluster <CLUSTERNAME> \\\\\\n --name externalsecrets \\\\\\n --namespace external-secrets\\n```\n\nOnce the profile creation is complete, run the following command to verify.\n\n```\\neksctl get fargateprofile --cluster <CLUSTERNAME> -o yaml\\n```\n\nThe expected output should look similar to the following with your “externalsecrets” profile listed.\n\n```\\n- name: externalsecrets\\n podExecutionRoleARN: arn:aws:iam::<ACCOUNTNUM>:role/eksctl-blogdemo-cluster-FargatePodExecutionRole-1GY5JZHKR1993\\n selectors:\\n - namespace: external-secrets\\n status: ACTIVE\\n subnets:\\n - subnet-072cad201ce783be1\\n - subnet-086d0835e05be3d89\\n```\n\n#### **Step 2. Deploy External Secrets Operator**\nThe External Secrets Operator provides Helm Charts for ease of deployment. The Helm Charts can be found in the project’s [Github repository](https://github.com/external-secrets/external-secrets/tree/main/deploy/charts/external-secrets). The following commands will perform a default installation and are further outlined in the [External Secrets Getting Started guide](https://external-secrets.io/v0.5.7/guides-getting-started/#option-1-install-from-chart-repository).\n\nFrom the terminal where you have Helm installed, run the following commands.\n\n```\\nhelm repo add external-secrets https://charts.external-secrets.io\\n```\n\n```\\nhelm install external-secrets \\\\\\n external-secrets/external-secrets \\\\\\n -n external-secrets \\\\\\n --create-namespace \\\\\\n --set installCRDs=true \\\\\\n --set webhook.port=9443 \\n```\n\nOnce the deployment is complete, run the following command to verify.\n\n```\\nkubectl get pods -n external-secrets\\n```\n\nThe expected output should look like the following with the External Secrets pod running.\n\n```\\nNAMESPACE NAME READY STATUS RESTARTS AGE\\nexternal-secrets external-secrets-8cdbf85cd-k4hvs 1/1 Running 0 87s\\nexternal-secrets external-secrets-cert-controller-655b7b7d45-bxh24 1/1 Running 0 87s\\nexternal-secrets external-secrets-webhook-75db54d748-85l8p 1/1 Running 0 87s\\n```\n\n#### **Step 3. Set up IAM roles for service accounts**\n[IAM roles for service accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) is a feature of EKS that allows you to map [AWS IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) to [Kubernetes Service Accounts](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). This feature provides a strategy for managing AWS credentials for your applications running on EKS without having to directly manage static credentials. Reference this AWS blog post, Diving into [IAM Roles for Service Accounts](https://aws.amazon.com/blogs/containers/diving-into-iam-roles-for-service-accounts/), for more information on IRSA.\n\nFor our use case with External Secrets, we will be using IRSA to provide AWS credentials to the External Secrets pods for fine-grained access control to our [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) secret.\n\nThe first step in enabling IRSA is to create an [IAM OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) for your cluster, if one is not already created. You can follow the steps outlined in [create an IAM OIDC provider for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) in the [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) user guide to create an IAM OIDC provider for your cluster or use the following eksctl command. Once the OIDC provider is associated with your EKS cluster, we can create an IRSA service account for External Secrets to use.\n\nFrom the terminal where you have eksctl installed, run the following command.\n\n```\\neksctl utils associate-iam-oidc-provider --cluster=<CLUSTERNAME> --approve\\n```\n\nFrom the terminal where you have eksctl installed, run the following commands. IAMPOLICYARN will be replaced with the Amazon Resource Name (ARN) of the IAM policy that has permissions to access your [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) secret.\n\n```\\neksctl create iamserviceaccount \\\\\\n --name blogdemosa \\\\\\n --namespace default \\\\\\n --cluster <CLUSTERNAME> \\\\\\n --role-name \\"blogdemosa\\" \\\\\\n --attach-policy-arn <IAMPOLICYARN> \\\\\\n --approve \\\\\\n --override-existing-serviceaccounts\\n```\n\nThe IAM policy associated with the IRSA service account will be used to provide granular access to the secrets stored in [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail). Limiting the scope of your IAM policy will be important for following security best practices. The following is an example IAM policy used for this blog post. This example policy limits actions to describing and retrieving the single specified secret.\n\n```\\n{\\n \\"Version\\": \\"2012-10-17\\",\\n \\"Statement\\": [\\n {\\n \\"Sid\\": \\"VisualEditor0\\",\\n \\"Effect\\": \\"Allow\\",\\n \\"Action\\": [\\n \\"secretsmanager:GetSecretValue\\",\\n \\"secretsmanager:DescribeSecret\\"\\n ],\\n \\"Resource\\": \\"arn:aws:secretsmanager:<REGION>:<ACCOUNTNUM>:secret:prod/blogdemo/mysql1-Ft6k18\\"\\n }\\n ]\\n}\\n```\n\nTo verify successful creation of the Kubernetes Service Account, run the following kubectl command.\n\n```\\nkubectl get sa\\nYou should see the specified service account name listed.\\n\\nNAME SECRETS AGE\\nblogdemosa 1 2m49s\\ndefault 1 18h\\n```\n\nYou can further inspect the service account by running the following kubectl command.\n\n```\\nkubectl describe sa blogdemosa\\n```\n\nThe output will provide extended service account details. You should see the EKS annotation that references the IAM role created by the eksctl command.\n\n```\\nName: blogdemosa\\nNamespace: default\\nLabels: app.kubernetes.io/managed-by=eksctl\\nAnnotations: eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNTNUM>:role/blogdemosa\\nImage pull secrets: <none>\\nMountable secrets: blogdemosa-token-8d46d\\nTokens: blogdemosa-token-8d46d\\nEvents: <none>\\n```\n\nWith our IRSA service account created, we can now configure External Secrets resources.\n\n#### **Step 4. Configuring External Secrets to sync [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) secrets**\nExternal Secrets provides custom resources (CRDs) for configuring the required operator functionality to sync external secrets to your cluster.\n\n[SecretStore](https://external-secrets.io/v0.5.7/api-secretstore/) is used to define the external secrets store and the authentication mechanisms to access the declared store. [ExternalSecret](https://external-secrets.io/v0.5.7/api-externalsecret/) defines what data to fetch from the secret store defined in the SecretStore resource. Reference the [External Secret resource model documentation](https://external-secrets.io/v0.5.7/api-overview/#resource-model) for additional details.\n\nFor this example, we are going to create a SecretStore object that references our existing [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) store. We will specify the IRSA-based service account we previously created to define the AWS credentials that will be used to access the secret store.\n\nUse the following object definition YAML to create the resource using kubectl.\n\n```\\napiVersion: external-secrets.io/v1beta1\\nkind: SecretStore\\nmetadata:\\n name: blogdemo\\nspec:\\n provider:\\n aws:\\n service: SecretsManager\\n region: <REGION>\\n auth:\\n jwt:\\n serviceAccountRef:\\n name: blogdemosa\\n```\n\nFrom the terminal where you have kubectl installed and the YAML file created, run the following commands.\n\n```\\nkubectl apply -f secretstore.yaml\\n```\n\nYou can validate your SecretStore was created by running the following kubectl command.\n\n```\\nkubectl get secretstore\\n```\n\nYou should see the created SecretStore referenced with STATUS of Valid.\n\n```\\nNAME AGE STATUS\\nblogdemo 82s Valid\\n```\n\nWe will now create our ExternalSecret resource, specifying the secret we want to access and referencing the previously created SecretStore object. You will specify your existing [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) secret name and keys where highlighted. For details on creating an [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) secret, see [Create and manage secrets with AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/managing-secrets.html) in the [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) user guide. For additional External Secrets configuration options, see the [AWS Secrets Manager provider](https://external-secrets.io/v0.4.4/provider-aws-secrets-manager/) in the External Secrets documentation.\n\nUse the following object definition YAML to create the resource using kubectl. Replace the highlight areas with your specific Secrets Manager values\n\n```\\napiVersion: external-secrets.io/v1beta1\\nkind: ExternalSecret\\nmetadata:\\n name: blogdemo\\nspec:\\n refreshInterval: 1h\\n secretStoreRef:\\n name: blogdemo\\n kind: SecretStore\\n target:\\n name: blogdemosecret\\n creationPolicy: Owner\\n data:\\n - secretKey: blogdemo-mysql-username\\n remoteRef:\\n key: prod/blogdemo/mysql1 #AWS Secrets Manager secret name\\n property: username #AWS Secrets Manager secret key\\n - secretKey: blogdemo-mysql-password\\n remoteRef:\\n key: prod/blogdemo/mysql1 #AWS Secrets Manager secret name\\n property: password #AWS Secrets Manager secret key\\n```\n\nHere is the example secret used for this blog as it is stored in Secrets Manager. Only the username and password are being referenced in the previous ExternalSecret definition.\n\n\n\n*Figure 2. [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) Secret value example*\n\nFrom the terminal where you have kubectl installed and the YAML file created, run the following commands.\n\n```\\nkubectl apply -f externalsecret.yaml\\n```\n\nOnce you have created the ExternalSecret resource, you will be able to view the newly created Kubernetes Secret that is being synced with the Secrets Manager store. Run the following kubectl command.\n\n```\\nkubectl describe secret blogdemosecret\\n```\n\nThe output will describe the secret as in the following:\n\n```\\nName: blogdemosecret\\nNamespace: default\\nLabels: <none>\\nAnnotations: reconcile.external-secrets.io/data-hash: a12b24f34338b71e8d118c6a04107b0e\\n\\nType: Opaque\\n\\nData\\n====\\nblogdemo-mysql-password: 8 bytes\\nblogdemo-mysql-username: 8 bytes\\n```\n\nYou now have a synced Kubernetes Secret that can be used within your pod specification for consumption by your applications.\n\nOne thing to consider with Kubernetes Secrets is that by default, they are stored in etcd unencrypted in base64 encoded form. With EKS, you are able to leverage [AWS Key Management Service](https://aws.amazon.com/kms/) (AWS KMS) keys to provide envelope encryption of Kubernetes secrets stored in EKS. This feature, along with the fact that we operate the etcd volumes encrypted at disk-level using AWS-managed encryption keys, provides a defense in-depth strategy for protection of your Kubernetes secrets stored in etcd. Reference this [AWS blog post](https://aws.amazon.com/blogs/containers/using-eks-encryption-provider-support-for-defense-in-depth/) for more information on EKS envelope encryption of Kubernetes Secrets.\n\n#### **Step 4. Consuming secret in pod**\nNow that External Secrets has synced your [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) secret to a Kubernetes Secret, you can consume this secret by referencing it in your Pod specification. Refer to [Using a Secret](https://kubernetes.io/docs/concepts/configuration/secret/#using-a-secret) in the Kubernetes documentation for additional information on different secret consumption options available.\n\nUse the following pod spec as an example for using your secret via environment variables.\n\n```\\napiVersion: v1\\nkind: Pod\\nmetadata:\\n name: busybox\\n namespace: default\\nspec:\\n containers:\\n - image: busybox\\n command:\\n - sleep\\n - \\"3600\\"\\n imagePullPolicy: IfNotPresent\\n name: busybox\\n env:\\n - name: BLOG_SECRET_USERNAME\\n valueFrom:\\n secretKeyRef:\\n name: blogdemosecret\\n key: blogdemo-mysql-username\\n optional: false \\n - name: BLOG_SECRET_PASSWORD\\n valueFrom:\\n secretKeyRef:\\n name: blogdemosecret\\n key: blogdemo-mysql-password\\n optional: false\\n restartPolicy: Always\\n```\n\nFrom the terminal where you have kubectl installed and the YAML file created, run the following commands.\n\n```\\nkubectl apply -f podexample.yaml\\n```\n\nOnce the pod has been deployed, you can directly access the running container’s shell using the following command.\n\n```\\nkubectl exec --stdin --tty busybox -- /bin/sh\\n```\n\nFrom there, you can test your environment variables to view your secrets.\n\n```\\n/ # echo \$BLOG_SECRET_USERNAME\\nblogdemo\\n/ # echo \$BLOG_SECRET_PASSWORD\\nTest1234\\n```\n\nBy exposing your secrets as environment variables, you are able to consume these secrets in your application as required.\n\n#### **Cleaning up**\nTo avoid incurring continued charges, delete any created resources from your cluster, especially those that are running as Fargate pods. Make sure you run the following commands from the same machine used to originally deploy the resources.\n\n```\\nkubectl delete -f podexample.yaml\\nkubectl delete -f externalsecret.yaml\\nkubectl delete -f secretstore.yaml\\neksctl delete iamserviceaccount \\\\\\n --name blogdemosa \\\\\\n --namespace default \\\\\\n --cluster <CLUSTERNAME>\\nhelm uninstall external-secrets -n external-secrets\\neksctl delete fargateprofile --cluster <CLUSTERNAME> --name externalsecrets\\n```\n\nAlso delete any IAM or Secrets Manager resources you created specifically for this walkthrough.\n\n### **Conclusion**\nThis blog post showcased how customers using EKS Fargate can consume secrets stored in AWS external secrets managers using the External Secrets Operator without requiring any code changes to your applications. This same solution can also apply to EC2 and hybrid EC2/Fargate clusters.\n\nFor more information on the External Secrets Operator, see the [External Secrets Operator documentation](https://external-secrets.io/) or the [External Secrets Github](https://github.com/external-secrets/external-secrets/).\n\n\n\n**Ryan Stebich**\n\nRyan Stebich is a Senior Solutions Architect with Amazon Web Services (AWS) based out of North Carolina. He leverages his broad range of IT experience and technical knowledge to design and implement Cloud based solutions to help customer solve their business challenges.","render":"<p>Secrets management is a challenging but critical aspect of running secure and dynamic containerized applications at scale. To support this need to securely distribute secrets to running applications, Kubernetes provides native functionality to manage secrets in the form of <a href=\\"https://kubernetes.io/docs/concepts/configuration/secret/\\" target=\\"_blank\\">Kubernetes Secrets</a>. However, many customers choose to centralize the management of secrets outside of their Kubernetes clusters by using external secret stores such as <a href=\\"https://aws.amazon.com/secrets-manager/\\" target=\\"_blank\\">AWS Secrets Manager</a> to improve the security, management, and auditability of their secret usage.</p>\\n<p>Consuming secrets from external secret stores often requires modifications to your application code to support API-based calls to the external store to retrieve a secret at application run time. When running applications on <a href=\\"https://aws.amazon.com/eks/\\" target=\\"_blank\\">Amazon EKS</a>, you can use Kubernetes’s flexibility to expose secrets directly to pods without requiring any application code changes. One example of how to accomplish this is by using the <a href=\\"https://github.com/aws/secrets-store-csi-driver-provider-aws\\" target=\\"_blank\\">AWS Secrets and Configuration Provider</a> (ASCP) for the <a href=\\"https://secrets-store-csi-driver.sigs.k8s.io/\\" target=\\"_blank\\">Kubernetes Secrets Store CSI Driver</a>. ASCP uses the Secrets Store CSI driver to expose secrets from [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) to your pods as a <a href=\\"https://aws.amazon.com/blogs/security/how-to-use-aws-secrets-configuration-provider-with-kubernetes-secrets-store-csi-driver/\\" target=\\"_blank\\">mounted storage volume</a>.</p>\\n<p>Customers using Amazon EKS to orchestrate their applications often will use <a href=\\"https://aws.amazon.com/fargate/\\" target=\\"_blank\\">AWS Fargate</a> as their compute layer to reduce the management complexity of operating their containerized workloads. For customers who have EKS clusters with [AWS Fargate](https://aws.amazon.com/cn/fargate/?trk=cndc-detail) nodes, a different method of consuming external secrets will be required since ASCP with the Secrets Store CSI is deployed to your EKS cluster as a daemonset. As of today, daemonsets are not supported on Fargate.</p>\\n<p>One such option is the open-source <a href=\\"https://github.com/external-secrets/external-secrets\\" target=\\"_blank\\">External Secrets Operator</a> (External Secrets) project. External Secrets manages your secrets in a different manner from the Secrets Store CSI driver. Instead of mounting secrets as volumes, External Secrets reads secrets from your external secret store and automatically stores the values as native Kubernetes Secrets in the Kubernetes control plane. External Secrets is installed as a deployment on your cluster and works with [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) Fargate-only clusters as well as those with Amazon EC2-based nodes.</p>\\n<p>In this post, I walk through using the External Secrets Operator on an EKS Fargate cluster to consume secrets stored in AWS Secrets Manager. This same method could apply to secrets stored in an <a href=\\"https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html\\" target=\\"_blank\\">AWS Systems Manager Parameter Store</a> but will not be covered directly in this post.</p>\\n<h3><a id=\\"Solution_overview_10\\"></a><strong>Solution overview</strong></h3>\\n<p>External Secrets will be deployed to your EKS cluster as a standard Kubernetes deployment, allowing Fargate to be used as your container compute provider. Once configured, External Secrets will sync defined secrets from AWS Secrets Manager back to your EKS cluster. The secrets will be automatically synced on a recurring basis to capture any updates or changes, such as periodic credential rotations.</p>\n<p>The following diagram outlines how the sync process will occur.</p>\n<ol>\\n<li>External Secrets will make API calls to AWS Secrets Manager on a recurring basis to copy specified secrets values.</li>\n<li>External Secrets will take the copied values and create native Kubernetes Secrets based on the external secret values.</li>\n<li>Kubernetes Secrets will be available for consumption by specified applications running on your EKS cluster.Kubernetes RBAC will define what applications can consume the secrets.</li>\n<li>Pods will consume the secrets as either volume mounts or environment variables as defined in the pod specifications.</li>\n</ol>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/74759f166dca45078b8fe9d80ddd70b5_image.png\\" alt=\\"image.png\\" /></p>\n<p><em>Figure 1. Solution overview</em></p>\\n<h3><a id=\\"Walkthrough_24\\"></a><strong>Walkthrough</strong></h3>\\n<p>This walkthrough will guide you through the steps of using the External Secrets Operator to sync a secret from AWS Secrets Manager to your EKS Fargate cluster, then consuming that secret in an application pod running on Fargate.</p>\n<p>The steps to be outlined in the following sections include:</p>\n<ul>\\n<li>Deploying External Secrets Operator on a EKS Fargate cluster</li>\n<li>Configuring External Secrets Operator resources</li>\n<li>Consuming synced Kubernetes Secret in an application pod</li>\n</ul>\\n<h4><a id=\\"Prerequisites_34\\"></a><strong>Prerequisites</strong></h4>\\n<p>For this walkthrough, you should have the following prerequisites:</p>\n<ul>\\n<li>An AWS account</li>\n<li>An IAM policy with permissions to <a href=\\"https://docs.aws.amazon.com/secretsmanager/latest/userguide/auth-and-access_examples.html\\" target=\\"_blank\\">retrieve a secret from Secrets Manager</a></li>\\n<li>Your <a href=\\"https://docs.aws.amazon.com/secretsmanager/latest/userguide/manage_create-basic-secret.html\\" target=\\"_blank\\">secret stored in Secrets Manager</a></li>\\n<li>An existing <a href=\\"https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html\\" target=\\"_blank\\">Amazon EKS cluster</a></li>\\n<li>AWS <a href=\\"https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html\\" target=\\"_blank\\">CLI</a> installed</li>\\n<li><a href=\\"https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html\\" target=\\"_blank\\">kubectl</a>, <a href=\\"https://helm.sh/docs/intro/install/\\" target=\\"_blank\\">Helm</a>, and <a href=\\"https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html\\" target=\\"_blank\\">eksctl</a> installed</li>\\n</ul>\n<h4><a id=\\"Step_1_Create_an_EKS_Fargate_profile_45\\"></a><strong>Step 1. Create an EKS Fargate profile</strong></h4>\\n<p>With Amazon EKS, <a href=\\"https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html\\" target=\\"_blank\\">Fargate profiles</a> allow administrators to specify which pods in your cluster should be designated to run on Fargate. Profile selectors declare namespaces and optionally labels that will be evaluated against to determine if a pod should be scheduled on Fargate. Fargate profiles can be created via numerous CLI and Infrastructure as Code (IaC) methods. The following example will showcase the eksctl method. To create your Fargate profile using the AWS console, see <a href=\\"https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html#create-fargate-profile\\" target=\\"_blank\\">creating a Fargate profile</a> in the [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) user guide.</p>\\n<p>External Secrets will, by default, deploy its resources in a namespace called “external-secrets.” To ensure our External Secrets pods run on Fargate, we will specify this namespace in our Fargate profile selector.</p>\n<p>From the terminal where you have eksctl installed, run the following command to create your Fargate profile. Replace “CLUSTERNAME” with the name of your EKS cluster.</p>\n<pre><code class=\\"lang-\\">eksctl create fargateprofile \\\\\\n --cluster &lt;CLUSTERNAME&gt; \\\\\\n --name externalsecrets \\\\\\n --namespace external-secrets\\n</code></pre>\\n<p>Once the profile creation is complete, run the following command to verify.</p>\n<pre><code class=\\"lang-\\">eksctl get fargateprofile --cluster &lt;CLUSTERNAME&gt; -o yaml\\n</code></pre>\\n<p>The expected output should look similar to the following with your “externalsecrets” profile listed.</p>\n<pre><code class=\\"lang-\\">- name: externalsecrets\\n podExecutionRoleARN: arn:aws:iam::&lt;ACCOUNTNUM&gt;:role/eksctl-blogdemo-cluster-FargatePodExecutionRole-1GY5JZHKR1993\\n selectors:\\n - namespace: external-secrets\\n status: ACTIVE\\n subnets:\\n - subnet-072cad201ce783be1\\n - subnet-086d0835e05be3d89\\n</code></pre>\\n<h4><a id=\\"Step_2_Deploy_External_Secrets_Operator_78\\"></a><strong>Step 2. Deploy External Secrets Operator</strong></h4>\\n<p>The External Secrets Operator provides Helm Charts for ease of deployment. The Helm Charts can be found in the project’s <a href=\\"https://github.com/external-secrets/external-secrets/tree/main/deploy/charts/external-secrets\\" target=\\"_blank\\">Github repository</a>. The following commands will perform a default installation and are further outlined in the <a href=\\"https://external-secrets.io/v0.5.7/guides-getting-started/#option-1-install-from-chart-repository\\" target=\\"_blank\\">External Secrets Getting Started guide</a>.</p>\\n<p>From the terminal where you have Helm installed, run the following commands.</p>\n<pre><code class=\\"lang-\\">helm repo add external-secrets https://charts.external-secrets.io\\n</code></pre>\\n<pre><code class=\\"lang-\\">helm install external-secrets \\\\\\n external-secrets/external-secrets \\\\\\n -n external-secrets \\\\\\n --create-namespace \\\\\\n --set installCRDs=true \\\\\\n --set webhook.port=9443 \\n</code></pre>\\n<p>Once the deployment is complete, run the following command to verify.</p>\n<pre><code class=\\"lang-\\">kubectl get pods -n external-secrets\\n</code></pre>\\n<p>The expected output should look like the following with the External Secrets pod running.</p>\n<pre><code class=\\"lang-\\">NAMESPACE NAME READY STATUS RESTARTS AGE\\nexternal-secrets external-secrets-8cdbf85cd-k4hvs 1/1 Running 0 87s\\nexternal-secrets external-secrets-cert-controller-655b7b7d45-bxh24 1/1 Running 0 87s\\nexternal-secrets external-secrets-webhook-75db54d748-85l8p 1/1 Running 0 87s\\n</code></pre>\\n<h4><a id=\\"Step_3_Set_up_IAM_roles_for_service_accounts_111\\"></a><strong>Step 3. Set up IAM roles for service accounts</strong></h4>\\n<p><a href=\\"https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html\\" target=\\"_blank\\">IAM roles for service accounts (IRSA)</a> is a feature of EKS that allows you to map <a href=\\"https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html\\" target=\\"_blank\\">AWS IAM roles</a> to <a href=\\"https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/\\" target=\\"_blank\\">Kubernetes Service Accounts</a>. This feature provides a strategy for managing AWS credentials for your applications running on EKS without having to directly manage static credentials. Reference this AWS blog post, Diving into <a href=\\"https://aws.amazon.com/blogs/containers/diving-into-iam-roles-for-service-accounts/\\" target=\\"_blank\\">IAM Roles for Service Accounts</a>, for more information on IRSA.</p>\\n<p>For our use case with External Secrets, we will be using IRSA to provide AWS credentials to the External Secrets pods for fine-grained access control to our AWS Secrets Manager secret.</p>\n<p>The first step in enabling IRSA is to create an <a href=\\"https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html\\" target=\\"_blank\\">IAM OIDC provider</a> for your cluster, if one is not already created. You can follow the steps outlined in <a href=\\"https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html\\" target=\\"_blank\\">create an IAM OIDC provider for your cluster</a> in the [Amazon EKS](https://aws.amazon.com/cn/eks/?trk=cndc-detail) user guide to create an IAM OIDC provider for your cluster or use the following eksctl command. Once the OIDC provider is associated with your EKS cluster, we can create an IRSA service account for External Secrets to use.</p>\\n<p>From the terminal where you have eksctl installed, run the following command.</p>\n<pre><code class=\\"lang-\\">eksctl utils associate-iam-oidc-provider --cluster=&lt;CLUSTERNAME&gt; --approve\\n</code></pre>\\n<p>From the terminal where you have eksctl installed, run the following commands. IAMPOLICYARN will be replaced with the Amazon Resource Name (ARN) of the IAM policy that has permissions to access your AWS Secrets Manager secret.</p>\n<pre><code class=\\"lang-\\">eksctl create iamserviceaccount \\\\\\n --name blogdemosa \\\\\\n --namespace default \\\\\\n --cluster &lt;CLUSTERNAME&gt; \\\\\\n --role-name &quot;blogdemosa&quot; \\\\\\n --attach-policy-arn &lt;IAMPOLICYARN&gt; \\\\\\n --approve \\\\\\n --override-existing-serviceaccounts\\n</code></pre>\\n<p>The IAM policy associated with the IRSA service account will be used to provide granular access to the secrets stored in AWS Secrets Manager. Limiting the scope of your IAM policy will be important for following security best practices. The following is an example IAM policy used for this blog post. This example policy limits actions to describing and retrieving the single specified secret.</p>\n<pre><code class=\\"lang-\\">{\\n &quot;Version&quot;: &quot;2012-10-17&quot;,\\n &quot;Statement&quot;: [\\n {\\n &quot;Sid&quot;: &quot;VisualEditor0&quot;,\\n &quot;Effect&quot;: &quot;Allow&quot;,\\n &quot;Action&quot;: [\\n &quot;secretsmanager:GetSecretValue&quot;,\\n &quot;secretsmanager:DescribeSecret&quot;\\n ],\\n &quot;Resource&quot;: &quot;arn:aws:secretsmanager:&lt;REGION&gt;:&lt;ACCOUNTNUM&gt;:secret:prod/blogdemo/mysql1-Ft6k18&quot;\\n }\\n ]\\n}\\n</code></pre>\\n<p>To verify successful creation of the Kubernetes Service Account, run the following kubectl command.</p>\n<pre><code class=\\"lang-\\">kubectl get sa\\nYou should see the specified service account name listed.\\n\\nNAME SECRETS AGE\\nblogdemosa 1 2m49s\\ndefault 1 18h\\n</code></pre>\\n<p>You can further inspect the service account by running the following kubectl command.</p>\n<pre><code class=\\"lang-\\">kubectl describe sa blogdemosa\\n</code></pre>\\n<p>The output will provide extended service account details. You should see the EKS annotation that references the IAM role created by the eksctl command.</p>\n<pre><code class=\\"lang-\\">Name: blogdemosa\\nNamespace: default\\nLabels: app.kubernetes.io/managed-by=eksctl\\nAnnotations: eks.amazonaws.com/role-arn: arn:aws:iam::&lt;ACCOUNTNUM&gt;:role/blogdemosa\\nImage pull secrets: &lt;none&gt;\\nMountable secrets: blogdemosa-token-8d46d\\nTokens: blogdemosa-token-8d46d\\nEvents: &lt;none&gt;\\n</code></pre>\\n<p>With our IRSA service account created, we can now configure External Secrets resources.</p>\n<h4><a id=\\"Step_4_Configuring_External_Secrets_to_sync_AWS_Secrets_Manager_secrets_188\\"></a><strong>Step 4. Configuring External Secrets to sync AWS Secrets Manager secrets</strong></h4>\\n<p>External Secrets provides custom resources (CRDs) for configuring the required operator functionality to sync external secrets to your cluster.</p>\n<p><a href=\\"https://external-secrets.io/v0.5.7/api-secretstore/\\" target=\\"_blank\\">SecretStore</a> is used to define the external secrets store and the authentication mechanisms to access the declared store. <a href=\\"https://external-secrets.io/v0.5.7/api-externalsecret/\\" target=\\"_blank\\">ExternalSecret</a> defines what data to fetch from the secret store defined in the SecretStore resource. Reference the <a href=\\"https://external-secrets.io/v0.5.7/api-overview/#resource-model\\" target=\\"_blank\\">External Secret resource model documentation</a> for additional details.</p>\\n<p>For this example, we are going to create a SecretStore object that references our existing AWS Secrets Manager store. We will specify the IRSA-based service account we previously created to define the AWS credentials that will be used to access the secret store.</p>\n<p>Use the following object definition YAML to create the resource using kubectl.</p>\n<pre><code class=\\"lang-\\">apiVersion: external-secrets.io/v1beta1\\nkind: SecretStore\\nmetadata:\\n name: blogdemo\\nspec:\\n provider:\\n aws:\\n service: SecretsManager\\n region: &lt;REGION&gt;\\n auth:\\n jwt:\\n serviceAccountRef:\\n name: blogdemosa\\n</code></pre>\\n<p>From the terminal where you have kubectl installed and the YAML file created, run the following commands.</p>\n<pre><code class=\\"lang-\\">kubectl apply -f secretstore.yaml\\n</code></pre>\\n<p>You can validate your SecretStore was created by running the following kubectl command.</p>\n<pre><code class=\\"lang-\\">kubectl get secretstore\\n</code></pre>\\n<p>You should see the created SecretStore referenced with STATUS of Valid.</p>\n<pre><code class=\\"lang-\\">NAME AGE STATUS\\nblogdemo 82s Valid\\n</code></pre>\\n<p>We will now create our ExternalSecret resource, specifying the secret we want to access and referencing the previously created SecretStore object. You will specify your existing AWS Secrets Manager secret name and keys where highlighted. For details on creating an AWS Secrets Manager secret, see <a href=\\"https://docs.aws.amazon.com/secretsmanager/latest/userguide/managing-secrets.html\\" target=\\"_blank\\">Create and manage secrets with AWS Secrets Manager</a> in the [AWS Secrets Manager](https://aws.amazon.com/cn/secrets-manager/?trk=cndc-detail) user guide. For additional External Secrets configuration options, see the <a href=\\"https://external-secrets.io/v0.4.4/provider-aws-secrets-manager/\\" target=\\"_blank\\">AWS Secrets Manager provider</a> in the External Secrets documentation.</p>\\n<p>Use the following object definition YAML to create the resource using kubectl. Replace the highlight areas with your specific Secrets Manager values</p>\n<pre><code class=\\"lang-\\">apiVersion: external-secrets.io/v1beta1\\nkind: ExternalSecret\\nmetadata:\\n name: blogdemo\\nspec:\\n refreshInterval: 1h\\n secretStoreRef:\\n name: blogdemo\\n kind: SecretStore\\n target:\\n name: blogdemosecret\\n creationPolicy: Owner\\n data:\\n - secretKey: blogdemo-mysql-username\\n remoteRef:\\n key: prod/blogdemo/mysql1 #AWS Secrets Manager secret name\\n property: username #AWS Secrets Manager secret key\\n - secretKey: blogdemo-mysql-password\\n remoteRef:\\n key: prod/blogdemo/mysql1 #AWS Secrets Manager secret name\\n property: password #AWS Secrets Manager secret key\\n</code></pre>\\n<p>Here is the example secret used for this blog as it is stored in Secrets Manager. Only the username and password are being referenced in the previous ExternalSecret definition.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/d31015854e8546c6a64cdfed0502d9ac_image.png\\" alt=\\"image.png\\" /></p>\n<p><em>Figure 2. AWS Secrets Manager Secret value example</em></p>\\n<p>From the terminal where you have kubectl installed and the YAML file created, run the following commands.</p>\n<pre><code class=\\"lang-\\">kubectl apply -f externalsecret.yaml\\n</code></pre>\\n<p>Once you have created the ExternalSecret resource, you will be able to view the newly created Kubernetes Secret that is being synced with the Secrets Manager store. Run the following kubectl command.</p>\n<pre><code class=\\"lang-\\">kubectl describe secret blogdemosecret\\n</code></pre>\\n<p>The output will describe the secret as in the following:</p>\n<pre><code class=\\"lang-\\">Name: blogdemosecret\\nNamespace: default\\nLabels: &lt;none&gt;\\nAnnotations: reconcile.external-secrets.io/data-hash: a12b24f34338b71e8d118c6a04107b0e\\n\\nType: Opaque\\n\\nData\\n====\\nblogdemo-mysql-password: 8 bytes\\nblogdemo-mysql-username: 8 bytes\\n</code></pre>\\n<p>You now have a synced Kubernetes Secret that can be used within your pod specification for consumption by your applications.</p>\n<p>One thing to consider with Kubernetes Secrets is that by default, they are stored in etcd unencrypted in base64 encoded form. With EKS, you are able to leverage <a href=\\"https://aws.amazon.com/kms/\\" target=\\"_blank\\">AWS Key Management Service</a> (AWS KMS) keys to provide envelope encryption of Kubernetes secrets stored in EKS. This feature, along with the fact that we operate the etcd volumes encrypted at disk-level using AWS-managed encryption keys, provides a defense in-depth strategy for protection of your Kubernetes secrets stored in etcd. Reference this <a href=\\"https://aws.amazon.com/blogs/containers/using-eks-encryption-provider-support-for-defense-in-depth/\\" target=\\"_blank\\">AWS blog post</a> for more information on EKS envelope encryption of Kubernetes Secrets.</p>\\n<h4><a id=\\"Step_4_Consuming_secret_in_pod_298\\"></a><strong>Step 4. Consuming secret in pod</strong></h4>\\n<p>Now that External Secrets has synced your AWS Secrets Manager secret to a Kubernetes Secret, you can consume this secret by referencing it in your Pod specification. Refer to <a href=\\"https://kubernetes.io/docs/concepts/configuration/secret/#using-a-secret\\" target=\\"_blank\\">Using a Secret</a> in the Kubernetes documentation for additional information on different secret consumption options available.</p>\\n<p>Use the following pod spec as an example for using your secret via environment variables.</p>\n<pre><code class=\\"lang-\\">apiVersion: v1\\nkind: Pod\\nmetadata:\\n name: busybox\\n namespace: default\\nspec:\\n containers:\\n - image: busybox\\n command:\\n - sleep\\n - &quot;3600&quot;\\n imagePullPolicy: IfNotPresent\\n name: busybox\\n env:\\n - name: BLOG_SECRET_USERNAME\\n valueFrom:\\n secretKeyRef:\\n name: blogdemosecret\\n key: blogdemo-mysql-username\\n optional: false \\n - name: BLOG_SECRET_PASSWORD\\n valueFrom:\\n secretKeyRef:\\n name: blogdemosecret\\n key: blogdemo-mysql-password\\n optional: false\\n restartPolicy: Always\\n</code></pre>\\n<p>From the terminal where you have kubectl installed and the YAML file created, run the following commands.</p>\n<pre><code class=\\"lang-\\">kubectl apply -f podexample.yaml\\n</code></pre>\\n<p>Once the pod has been deployed, you can directly access the running container’s shell using the following command.</p>\n<pre><code class=\\"lang-\\">kubectl exec --stdin --tty busybox -- /bin/sh\\n</code></pre>\\n<p>From there, you can test your environment variables to view your secrets.</p>\n<pre><code class=\\"lang-\\">/ # echo \$BLOG_SECRET_USERNAME\\nblogdemo\\n/ # echo \$BLOG_SECRET_PASSWORD\\nTest1234\\n</code></pre>\\n<p>By exposing your secrets as environment variables, you are able to consume these secrets in your application as required.</p>\n<h4><a id=\\"Cleaning_up_356\\"></a><strong>Cleaning up</strong></h4>\\n<p>To avoid incurring continued charges, delete any created resources from your cluster, especially those that are running as Fargate pods. Make sure you run the following commands from the same machine used to originally deploy the resources.</p>\n<pre><code class=\\"lang-\\">kubectl delete -f podexample.yaml\\nkubectl delete -f externalsecret.yaml\\nkubectl delete -f secretstore.yaml\\neksctl delete iamserviceaccount \\\\\\n --name blogdemosa \\\\\\n --namespace default \\\\\\n --cluster &lt;CLUSTERNAME&gt;\\nhelm uninstall external-secrets -n external-secrets\\neksctl delete fargateprofile --cluster &lt;CLUSTERNAME&gt; --name externalsecrets\\n</code></pre>\\n<p>Also delete any IAM or Secrets Manager resources you created specifically for this walkthrough.</p>\n<h3><a id=\\"Conclusion_373\\"></a><strong>Conclusion</strong></h3>\\n<p>This blog post showcased how customers using EKS Fargate can consume secrets stored in AWS external secrets managers using the External Secrets Operator without requiring any code changes to your applications. This same solution can also apply to EC2 and hybrid EC2/Fargate clusters.</p>\n<p>For more information on the External Secrets Operator, see the <a href=\\"https://external-secrets.io/\\" target=\\"_blank\\">External Secrets Operator documentation</a> or the <a href=\\"https://github.com/external-secrets/external-secrets/\\" target=\\"_blank\\">External Secrets Github</a>.</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/aadbceefc83f4c33b89b59a87222c462_image.png\\" alt=\\"image.png\\" /></p>\n<p><strong>Ryan Stebich</strong></p>\\n<p>Ryan Stebich is a Senior Solutions Architect with Amazon Web Services (AWS) based out of North Carolina. He leverages his broad range of IT experience and technical knowledge to design and implement Cloud based solutions to help customer solve their business challenges.</p>\n"}