Protect Sensitive Data with Amazon CloudWatch Logs

{"value":"Today we are announcing [Amazon CloudWatch Logs data protection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html), a new set of capabilities for [Amazon CloudWatch Logs](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) that leverage pattern matching and machine learning (ML) to detect and protect sensitive log data in transit.\n\nWhile developers try to prevent logging sensitive information such as Social Security numbers, credit card details, email addresses, and passwords, sometimes it gets logged. Until today, customers relied on manual investigation or third-party solutions to detect and mitigate sensitive information from being logged. If sensitive data is not redacted during ingestion, it will be visible in plain text in the logs and in any downstream system that consumed those logs.\n\nEnforcing prevention across the organization is challenging, which is why quick detection and prevention of access to sensitive data in the logs is important from a security and compliance perspective. Starting today, you can enable [Amazon CloudWatch](https://aws.amazon.com/cn/cloudwatch/?trk=cndc-detail) Logs data protection to detect and mask sensitive log data as it is ingested into CloudWatch Logs or as it is in transit.\n\nCustomers from all industries that want to take advantage of native data protection capabilities can benefit from this feature. But in particular, it is useful for industries under strict regulations that need to make sure that no personal information gets exposed. Also, customers building payment or authentication services where personal and sensitive information may be captured can use this new feature to detect and mask sensitive information as it’s logged.\n\n**++Getting Started++**\n\nYou can enable a data protection policy for new or existing log groups from the [Amazon Web Services Management Console](https://console.aws.amazon.com/), [Amazon Web Services Command Line Interface (CLI)](https://aws.amazon.com/cli/), or [Amazon Web Services CloudFormation](https://aws.amazon.com/cloudformation/). From the console, select any log group and create a data protection policy in the **Data protection** tab.\n\n\n\nWhen you create the policy, you can specify the data you want to protect. Choose from over 100 managed data identifiers, which are a repository of common sensitive data patterns spanning financial, health, and personal information. This feature provides you with complete flexibility in choosing from a wide variety of data identifiers that are specific to your use cases or geographical region.\n\n\n\nYou can also enable audit reports and send them to another log group, an [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) bucket, or [Amazon Kinesis Firehose](https://aws.amazon.com/kinesis/firehose/). These reports contain a detailed log of data protection findings.\n\nIf you want to monitor and get notified when sensitive data is detected, you can create an alarm around the metric```LogEventsWithFindings```. This metric shows how many findings there are in a particular log group. This allows you to quickly understand which application is logging sensitive data.\n\nWhen sensitive information is logged, CloudWatch Logs data protection will automatically mask it per your configured policy. This is designed so that none of the downstream services that consume these logs can see the unmasked data. From the Amazon Web Services Management Console, Amazon Web Services CLI, or any third party, the sensitive information in the logs will appear masked.\n\n\n\nOnly users with elevated privileges in their IAM policy (add```logs:Unmask``` action in the user policy) can view unmasked data in [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html), [logs stream search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SearchDataFilterPattern.html), or via [FilterLogEvents](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_FilterLogEvents.html) and [GetLogEvents](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html) APIs.\n\nYou can use the following query in CloudWatch Logs Insights to unmask data for a particular log group:\n\n```\\nfields @timestamp, @message, unmask(@message)\\n| sort @timestamp desc\\n| limit 20\\n```\n\n**++Available Now++**\n\nData protection is available in US East (Ohio), US East (N. Virginia), US West (N. California), US West (Oregon), Africa (Cape Town), Asia Pacific (Hong Kong), Asia Pacific (Jakarta), Asia Pacific (Mumbai), Asia Pacific (Osaka), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Milan), Europe (Paris), Europe (Stockholm), Middle East (Bahrain), and South America (São Paulo) Amazon Web Services Regions.\n\n[Amazon CloudWatch](https://aws.amazon.com/cn/cloudwatch/?trk=cndc-detail) Logs data protection pricing is based on the amount of data that is scanned for sensitive information. You can check the [CloudWatch Logs pricing page](https://aws.amazon.com/cloudwatch/pricing/) to learn more about the pricing of this feature in your Region.\n\nLearn more about [data protection on the CloudWatch Logs User Guide.](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html)\n\n— [Marcia](https://twitter.com/mavi888uy)\n\n\n\n### **Marcia Villalba**\n\nMarcia Villalba is a Principal Developer Advocate for Amazon Web Services. She has almost 20 years of experience working in the software industry building and scaling applications. Her passion is designing systems that can take full advantage of the cloud and embrace the DevOps culture.","render":"<p>Today we are announcing <a href=\\"https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html\\" target=\\"_blank\\">Amazon CloudWatch Logs data protection</a>, a new set of capabilities for <a href=\\"http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html\\" target=\\"_blank\\">Amazon CloudWatch Logs</a> that leverage pattern matching and machine learning (ML) to detect and protect sensitive log data in transit.</p>\\n<p>While developers try to prevent logging sensitive information such as Social Security numbers, credit card details, email addresses, and passwords, sometimes it gets logged. Until today, customers relied on manual investigation or third-party solutions to detect and mitigate sensitive information from being logged. If sensitive data is not redacted during ingestion, it will be visible in plain text in the logs and in any downstream system that consumed those logs.</p>\n<p>Enforcing prevention across the organization is challenging, which is why quick detection and prevention of access to sensitive data in the logs is important from a security and compliance perspective. Starting today, you can enable Amazon CloudWatch Logs data protection to detect and mask sensitive log data as it is ingested into CloudWatch Logs or as it is in transit.</p>\n<p>Customers from all industries that want to take advantage of native data protection capabilities can benefit from this feature. But in particular, it is useful for industries under strict regulations that need to make sure that no personal information gets exposed. Also, customers building payment or authentication services where personal and sensitive information may be captured can use this new feature to detect and mask sensitive information as it’s logged.</p>\n<p><strong><ins>Getting Started</ins></strong></p>\n<p>You can enable a data protection policy for new or existing log groups from the <a href=\\"https://console.aws.amazon.com/\\" target=\\"_blank\\">Amazon Web Services Management Console</a>, <a href=\\"https://aws.amazon.com/cli/\\" target=\\"_blank\\">Amazon Web Services Command Line Interface (CLI)</a>, or <a href=\\"https://aws.amazon.com/cloudformation/\\" target=\\"_blank\\">Amazon Web Services CloudFormation</a>. From the console, select any log group and create a data protection policy in the <strong>Data protection</strong> tab.</p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/706fbf61690443b185a2e4d89eeeb3b8_image.png\\" alt=\\"image.png\\" /></p>\n<p>When you create the policy, you can specify the data you want to protect. Choose from over 100 managed data identifiers, which are a repository of common sensitive data patterns spanning financial, health, and personal information. This feature provides you with complete flexibility in choosing from a wide variety of data identifiers that are specific to your use cases or geographical region.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/bcb957ee638248fe80d2252d48358af7_image.png\\" alt=\\"image.png\\" /></p>\n<p>You can also enable audit reports and send them to another log group, an <a href=\\"https://aws.amazon.com/s3/\\" target=\\"_blank\\">Amazon Simple Storage Service (Amazon S3)</a> bucket, or <a href=\\"https://aws.amazon.com/kinesis/firehose/\\" target=\\"_blank\\">Amazon Kinesis Firehose</a>. These reports contain a detailed log of data protection findings.</p>\\n<p>If you want to monitor and get notified when sensitive data is detected, you can create an alarm around the metric<code>LogEventsWithFindings</code>. This metric shows how many findings there are in a particular log group. This allows you to quickly understand which application is logging sensitive data.</p>\\n<p>When sensitive information is logged, CloudWatch Logs data protection will automatically mask it per your configured policy. This is designed so that none of the downstream services that consume these logs can see the unmasked data. From the Amazon Web Services Management Console, Amazon Web Services CLI, or any third party, the sensitive information in the logs will appear masked.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/0cbbaae1e65e4c868613495e1778dce0_image.png\\" alt=\\"image.png\\" /></p>\n<p>Only users with elevated privileges in their IAM policy (add<code>logs:Unmask</code> action in the user policy) can view unmasked data in <a href=\\"https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html\\" target=\\"_blank\\">CloudWatch Logs Insights</a>, <a href=\\"https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SearchDataFilterPattern.html\\" target=\\"_blank\\">logs stream search</a>, or via <a href=\\"https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_FilterLogEvents.html\\" target=\\"_blank\\">FilterLogEvents</a> and <a href=\\"https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html\\" target=\\"_blank\\">GetLogEvents</a> APIs.</p>\\n<p>You can use the following query in CloudWatch Logs Insights to unmask data for a particular log group:</p>\n<pre><code class=\\"lang-\\">fields @timestamp, @message, unmask(@message)\\n| sort @timestamp desc\\n| limit 20\\n</code></pre>\\n<p><strong><ins>Available Now</ins></strong></p>\n<p>Data protection is available in US East (Ohio), US East (N. Virginia), US West (N. California), US West (Oregon), Africa (Cape Town), Asia Pacific (Hong Kong), Asia Pacific (Jakarta), Asia Pacific (Mumbai), Asia Pacific (Osaka), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Milan), Europe (Paris), Europe (Stockholm), Middle East (Bahrain), and South America (São Paulo) Amazon Web Services Regions.</p>\n<p>Amazon CloudWatch Logs data protection pricing is based on the amount of data that is scanned for sensitive information. You can check the <a href=\\"https://aws.amazon.com/cloudwatch/pricing/\\" target=\\"_blank\\">CloudWatch Logs pricing page</a> to learn more about the pricing of this feature in your Region.</p>\\n<p>Learn more about <a href=\\"https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html\\" target=\\"_blank\\">data protection on the CloudWatch Logs User Guide.</a></p>\\n<p>— <a href=\\"https://twitter.com/mavi888uy\\" target=\\"_blank\\">Marcia</a></p>\\n<p><img src=\\"https://dev-media.amazoncloud.cn/46791385c07e426d82a58c4dcb07f496_6465c675a51d44aca59965f281504693_image%282%29.png\\" alt=\\"6465c675a51d44aca59965f281504693_image2.png\\" /></p>\n<h3><a id=\\"Marcia_Villalba_48\\"></a><strong>Marcia Villalba</strong></h3>\\n<p>Marcia Villalba is a Principal Developer Advocate for Amazon Web Services. She has almost 20 years of experience working in the software industry building and scaling applications. Her passion is designing systems that can take full advantage of the cloud and embrace the DevOps culture.</p>\n"}