Analyze logs with Dynatrace Davis AI Engine using Amazon Kinesis Data Firehose HTTP endpoint delivery

海外精选
海外精选的内容汇集了全球优质的亚马逊云科技相关技术内容。同时,内容中提到的“AWS” 是 “Amazon Web Services” 的缩写,在此网站不作为商标展示。
0
0
{"value":"*This blog post is co-authored with Erick Leon, Sr. Technical Alliance Manager from Dynatrace.*\n\n[Amazon Kinesis Data Firehose](https://aws.amazon.com/kinesis/data-firehose/) is the easiest way to reliably load streaming data into data lakes, data stores, and analytics services. With just a few clicks, you can create fully-managed delivery streams that auto scale on demand to match the throughput of your data. Customers already use Kinesis Data Firehose to ingest raw data from various data sources, including logs from AWS services. Kinesis Data Firehose now supports delivering streaming data to Dynatrace. Dynatrace begins analyzing incoming data within minutes of [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) data generation.\n\nStarting today, you can use Kinesis Data Firehose to send CloudWatch Metrics and Logs directly to the Dynatrace observability platform to perform your explorations and analysis. Dynatrace, an [AWS Partner Network](http://aws.amazon.com/partners) (APN) has provided full observability into AWS Services by ingesting CloudWatch metrics that are published by AWS services. Dynatrace ingests this data to perform root-cause analysis using the Dynatrace Davis AI engine.\n\nIn this post, we describe the Kinesis Data Firehose and related Dynatrace integration.\n\n\n#### **Prerequisites**\n\n\nFor this walkthrough, you should have the following prerequisites:\n\n- [AWS account.](https://signin.aws.amazon.com/signin?redirect_uri=https%3A%2F%2Fportal.aws.amazon.com%2Fbilling%2Fsignup%2Fresume&client_id=signup)\n- Access to the CloudWatch and Kinesis Data Firehose with permissions to manage HTTP endpoints.\n- Dynatrace Intelligent Observability Platform account, or get a free 15 day trial [here](https://www.dynatrace.com/trial).\n- Dynatrace version 1.182+.\n- An updated [AWS monitoring policy](https://www.dynatrace.com/support/help/setup-and-configuration/setup-on-cloud-platforms/amazon-web-services/aws-monitoring-with-dynatrace-saas#monitoring-policy) to include the additional AWS services.\nTo [update the AWS Identity and Access Management (IAM) policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-edit.html), use the JSON in the link above, containing the monitoring policy (permissions) for all supporting services.\n- [Dynatrace API token](https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication): create token with the following permission and keep readily available in a notepad.\n\n![image.png](https://dev-media.amazoncloud.cn/7a8e58c2a82f46c58e919add1cfdda0e_image.png)\n\n\nFigure 1 – Dynatrace API Token\n\n\n#### **How it works**\n\n\n![image.png](https://dev-media.amazoncloud.cn/bcf2de0fcba440229a6a3dd824ff0852_image.png)\n\nFigure 2 – Amazon Kinesis Data Firehose HTTP endpoint delivery\n\nSimply create a log stream for your Amazon services to deliver your context rich logs to the [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html) service. Next, select your Dynatrace HTTP endpoint to enhance your logs streams with the power of the Dynatrace Intelligence Platform. Finally, you can also back up your logs to an [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) bucket.\n\n\n#### **Setup instructions**\n\n\n[To add a service to monitoring](https://www.dynatrace.com/support/help/how-to-use-dynatrace/infrastructure-monitoring/cloud-platform-monitoring/amazon-web-services-monitoring/aws-supporting-service-metrics/kinesis#expand--to-add-a-service-to-monitoring), follow these steps:\n\n1. In the [Dynatrace menu](https://www.dynatrace.com/support/help/get-started/navigation), go to **Settings > Cloud and virtualization**, and select **AWS**.\n2. On the **AWS overview page**, scroll down and select the desired AWS instance. Select the **Edit** button.\n3. Scroll down and select **Add service**. Choose the service name from the drop-down, and select **Add service.**\n4. Select **Save changes**.\n\n**To process and deliver AWS CloudWatch Metrics to Dynatrace, follow these steps.**\n\n1. Log in to the AWS console and type “Kinesis” in the text search bar. Select **Kinesis**\n\n![image.png](https://dev-media.amazoncloud.cn/953307036ecf4902aac81695022a6740_image.png)\n\nFigure 3 – AWS Console\n\n2. On the [Amazon Kinesis](https://aws.amazon.com/kinesis/) services page, select the radio button for **Kinesis Data Firehose** and select the **Create delivery stream** button.\n\n![image.png](https://dev-media.amazoncloud.cn/aa1a6b9ac4dd423da70831d57dbf32f9_image.png)\n\nFigure 4 – Amazon Kinesis\n\n3. Choose the “**Direct PUT**” from the drop down, and from Destination drop down, choose “**Dynatrace**”.\n\n![image.png](https://dev-media.amazoncloud.cn/58ee9804b3df4e59a8a246ee683a4c77_image.png)\n\n\nFigure 5 – Amazon Kinesis Data Firehose\n\n4. **Delivery stream name** – Give your stream a new name, for example: – “KFH-StreamToDynatrace”\n\n![image.png](https://dev-media.amazoncloud.cn/6d73ef27461940a18afb61117ce8ea1e_image.png)\n\nFigure 6 – Delivery stream name\n\n5. In the section “**Destination settings**”:\n\n![image.png](https://dev-media.amazoncloud.cn/cbaba692384e4520a3f8ee7d408236d9_image.png)\n\n\nFigure 7 – Destination settings\n\n6. **HTTP endpoint name** – “Dynatrace”.\n7. **HTTP endpoint URL** – From the drop down, select “Dynatrace – US”.\n8. **API token** – Enter Dynatrace API TOKEN created in the prerequisite section.\n9. **API URL** – enter the Dynatrace URL for your tenant, for example: [xxxxx.live.dynatrace.com](https://xxxxx.live.dynatrace.com/)\n10. **Back Up Settings** – Either select an existing S3 bucket or create a new one and add details and select the **Create delivery stream** button.\n\n![image.png](https://dev-media.amazoncloud.cn/464de8d7f0e641bd99d8a67927c1064c_image.png) \n\nFigure 8 – Backup settings\n\nOnce successful, your [AWS Console](https://aws.amazon.com/console/) will look like the following:\n\n![image.png](https://dev-media.amazoncloud.cn/4c722844bfeb4580a65d7d4e0b6577ff_image.png)\n\nFigure 9 – Amazon Kinesis Data Firehose\n\n\n#### **The Dynatrace Experience**\n\nOnce the initial setups are completed in both Dynatrace and the AWS Console, follow these steps to visualize your new KHF stream data in the Dynatrace console.\n\n1. Log in to the Dynatrace Console, and on the left side menu expand the “**infrastructure**” section, and select “**AWS**”\n2. From the screen, select the AWS account that you want to add the KFH stream to.\n3. Next, you’ll see a virtualization of your AWS assets for the account selected. Select the box marked “**Supporting Services**”.\n4. Next, press the “**Configure services**” button.\n5. Next, select “**Add service**”.\n6. From the drop down, select “Kinesis Data Firehose”.\n7. Next, select the “**Add metric**” button, and select the metrics that you want to see for this stream. Dynatrace has a comprehensive list of metrics that can be selected from the UI. The list can be found in this [link](https://www.dynatrace.com/support/help/shortlink/kinesis#available-metrics).\n\n\n#### **Troubleshooting**\n\n\n1. After configuration, load to the new KFH stream no data in the Dynatrace Console.\na.Check the Error Logs tab check to make sure that the Destination URL is correct for the Dynatrace Tenant.\n\n![image.png](https://dev-media.amazoncloud.cn/60a3ac4ef641408e85080033de111a98_image.png)\n\nFigure 10 – Destination error logs\n\n2. Invalid or misconfigured Dynatrace API token or scope isn’t properly set.\n\n![image.png](https://dev-media.amazoncloud.cn/4cc82cb79e5b45eb90cd2ca2f5f9e89b_image.png)\n\n\nFigure 11 – Destination error logs\n\n\n\n#### **Conclusion**\n\nIn this post, we demonstrate the Kinesis Data Firehose and related Dynatrace integration. In addition, engineers can use CloudWatch Metrics to explore their production systems alongside events in Dynatrace. This provides a seamless, current view of your system (from logs to events and traces) in a single data store.\n\nTo learn more about CloudWatch Service, see the [Amazon CloudWatch home page](https://docs.aws.amazon.com/cloudwatch/index.html). If you have any questions, post them on the [AWS CloudWatch service forum](https://forums.aws.amazon.com/forum.jspa?forumID=138).\n\nIf you haven’t yet signed up for Dynatrace, then you can try out Kinesis Data Firehose with Dynatrace with a [free Dynatrace trial](https://www.dynatrace.com/trial/).\n\n\n#### **About the Authors**\n\n![image.png](https://dev-media.amazoncloud.cn/0f183442e23340f3a11b1abdfc4fbd3a_image.png)\n\n**Erick Leon** is a Technical Alliances Sr. Manager at Dynatrace, Observability Practice Architect, and Customer Advocate. He promotes strong technical integrations with a focus on AWS. With over 15 years as a Dynatrace customer, his real-world experiences and lessons learned bring valuable insights into the Dynatrace Intelligent Observability Platform.\n\n![image.png](https://dev-media.amazoncloud.cn/677c6e351c5545409e55c313b98d6c4f_image.png)\n\n**Shashiraj Jeripotula (Raj)** is a San Francisco-based Sr. Partner Solutions Architect at AWS. He works with various independent software vendors (ISVs), and partners who specialize in cloud management tools and DevOps to develop joint solutions and accelerate cloud adoption on AWS.","render":"<p><em>This blog post is co-authored with Erick Leon, Sr. Technical Alliance Manager from Dynatrace.</em></p>\n<p><a href=\"https://aws.amazon.com/kinesis/data-firehose/\" target=\"_blank\">Amazon Kinesis Data Firehose</a> is the easiest way to reliably load streaming data into data lakes, data stores, and analytics services. With just a few clicks, you can create fully-managed delivery streams that auto scale on demand to match the throughput of your data. Customers already use Kinesis Data Firehose to ingest raw data from various data sources, including logs from AWS services. Kinesis Data Firehose now supports delivering streaming data to Dynatrace. Dynatrace begins analyzing incoming data within minutes of <a href=\"https://aws.amazon.com/cloudwatch/\" target=\"_blank\">Amazon CloudWatch</a> data generation.</p>\n<p>Starting today, you can use Kinesis Data Firehose to send CloudWatch Metrics and Logs directly to the Dynatrace observability platform to perform your explorations and analysis. Dynatrace, an <a href=\"http://aws.amazon.com/partners\" target=\"_blank\">AWS Partner Network</a> (APN) has provided full observability into AWS Services by ingesting CloudWatch metrics that are published by AWS services. Dynatrace ingests this data to perform root-cause analysis using the Dynatrace Davis AI engine.</p>\n<p>In this post, we describe the Kinesis Data Firehose and related Dynatrace integration.</p>\n<h4><a id=\"Prerequisites_9\"></a><strong>Prerequisites</strong></h4>\n<p>For this walkthrough, you should have the following prerequisites:</p>\n<ul>\n<li><a href=\"https://signin.aws.amazon.com/signin?redirect_uri=https%3A%2F%2Fportal.aws.amazon.com%2Fbilling%2Fsignup%2Fresume&amp;client_id=signup\" target=\"_blank\">AWS account.</a></li>\n<li>Access to the CloudWatch and Kinesis Data Firehose with permissions to manage HTTP endpoints.</li>\n<li>Dynatrace Intelligent Observability Platform account, or get a free 15 day trial <a href=\"https://www.dynatrace.com/trial\" target=\"_blank\">here</a>.</li>\n<li>Dynatrace version 1.182+.</li>\n<li>An updated <a href=\"https://www.dynatrace.com/support/help/setup-and-configuration/setup-on-cloud-platforms/amazon-web-services/aws-monitoring-with-dynatrace-saas#monitoring-policy\" target=\"_blank\">AWS monitoring policy</a> to include the additional AWS services.<br />\nTo <a href=\"https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-edit.html\" target=\"_blank\">update the AWS Identity and Access Management (IAM) policy</a>, use the JSON in the link above, containing the monitoring policy (permissions) for all supporting services.</li>\n<li><a href=\"https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication\" target=\"_blank\">Dynatrace API token</a>: create token with the following permission and keep readily available in a notepad.</li>\n</ul>\n<p><img src=\"https://dev-media.amazoncloud.cn/7a8e58c2a82f46c58e919add1cfdda0e_image.png\" alt=\"image.png\" /></p>\n<p>Figure 1 – Dynatrace API Token</p>\n<h4><a id=\"How_it_works_28\"></a><strong>How it works</strong></h4>\n<p><img src=\"https://dev-media.amazoncloud.cn/bcf2de0fcba440229a6a3dd824ff0852_image.png\" alt=\"image.png\" /></p>\n<p>Figure 2 – Amazon Kinesis Data Firehose HTTP endpoint delivery</p>\n<p>Simply create a log stream for your Amazon services to deliver your context rich logs to the <a href=\"https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html\" target=\"_blank\">Amazon CloudWatch Logs</a> service. Next, select your Dynatrace HTTP endpoint to enhance your logs streams with the power of the Dynatrace Intelligence Platform. Finally, you can also back up your logs to an <a href=\"https://aws.amazon.com/s3/\" target=\"_blank\">Amazon Simple Storage Service (Amazon S3)</a> bucket.</p>\n<h4><a id=\"Setup_instructions_38\"></a><strong>Setup instructions</strong></h4>\n<p><a href=\"https://www.dynatrace.com/support/help/how-to-use-dynatrace/infrastructure-monitoring/cloud-platform-monitoring/amazon-web-services-monitoring/aws-supporting-service-metrics/kinesis#expand--to-add-a-service-to-monitoring\" target=\"_blank\">To add a service to monitoring</a>, follow these steps:</p>\n<ol>\n<li>In the <a href=\"https://www.dynatrace.com/support/help/get-started/navigation\" target=\"_blank\">Dynatrace menu</a>, go to <strong>Settings &gt; Cloud and virtualization</strong>, and select <strong>AWS</strong>.</li>\n<li>On the <strong>AWS overview page</strong>, scroll down and select the desired AWS instance. Select the <strong>Edit</strong> button.</li>\n<li>Scroll down and select <strong>Add service</strong>. Choose the service name from the drop-down, and select <strong>Add service.</strong></li>\n<li>Select <strong>Save changes</strong>.</li>\n</ol>\n<p><strong>To process and deliver AWS CloudWatch Metrics to Dynatrace, follow these steps.</strong></p>\n<ol>\n<li>Log in to the AWS console and type “Kinesis” in the text search bar. Select <strong>Kinesis</strong></li>\n</ol>\n<p><img src=\"https://dev-media.amazoncloud.cn/953307036ecf4902aac81695022a6740_image.png\" alt=\"image.png\" /></p>\n<p>Figure 3 – AWS Console</p>\n<ol start=\"2\">\n<li>On the <a href=\"https://aws.amazon.com/kinesis/\" target=\"_blank\">Amazon Kinesis</a> services page, select the radio button for <strong>Kinesis Data Firehose</strong> and select the <strong>Create delivery stream</strong> button.</li>\n</ol>\n<p><img src=\"https://dev-media.amazoncloud.cn/aa1a6b9ac4dd423da70831d57dbf32f9_image.png\" alt=\"image.png\" /></p>\n<p>Figure 4 – Amazon Kinesis</p>\n<ol start=\"3\">\n<li>Choose the “<strong>Direct PUT</strong>” from the drop down, and from Destination drop down, choose “<strong>Dynatrace</strong>”.</li>\n</ol>\n<p><img src=\"https://dev-media.amazoncloud.cn/58ee9804b3df4e59a8a246ee683a4c77_image.png\" alt=\"image.png\" /></p>\n<p>Figure 5 – Amazon Kinesis Data Firehose</p>\n<ol start=\"4\">\n<li><strong>Delivery stream name</strong> – Give your stream a new name, for example: – “KFH-StreamToDynatrace”</li>\n</ol>\n<p><img src=\"https://dev-media.amazoncloud.cn/6d73ef27461940a18afb61117ce8ea1e_image.png\" alt=\"image.png\" /></p>\n<p>Figure 6 – Delivery stream name</p>\n<ol start=\"5\">\n<li>In the section “<strong>Destination settings</strong>”:</li>\n</ol>\n<p><img src=\"https://dev-media.amazoncloud.cn/cbaba692384e4520a3f8ee7d408236d9_image.png\" alt=\"image.png\" /></p>\n<p>Figure 7 – Destination settings</p>\n<ol start=\"6\">\n<li><strong>HTTP endpoint name</strong> – “Dynatrace”.</li>\n<li><strong>HTTP endpoint URL</strong> – From the drop down, select “Dynatrace – US”.</li>\n<li><strong>API token</strong> – Enter Dynatrace API TOKEN created in the prerequisite section.</li>\n<li><strong>API URL</strong> – enter the Dynatrace URL for your tenant, for example: <a href=\"https://xxxxx.live.dynatrace.com/\" target=\"_blank\">xxxxx.live.dynatrace.com</a></li>\n<li><strong>Back Up Settings</strong> – Either select an existing S3 bucket or create a new one and add details and select the <strong>Create delivery stream</strong> button.</li>\n</ol>\n<p><img src=\"https://dev-media.amazoncloud.cn/464de8d7f0e641bd99d8a67927c1064c_image.png\" alt=\"image.png\" /></p>\n<p>Figure 8 – Backup settings</p>\n<p>Once successful, your <a href=\"https://aws.amazon.com/console/\" target=\"_blank\">AWS Console</a> will look like the following:</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/4c722844bfeb4580a65d7d4e0b6577ff_image.png\" alt=\"image.png\" /></p>\n<p>Figure 9 – Amazon Kinesis Data Firehose</p>\n<h4><a id=\"The_Dynatrace_Experience_99\"></a><strong>The Dynatrace Experience</strong></h4>\n<p>Once the initial setups are completed in both Dynatrace and the AWS Console, follow these steps to visualize your new KHF stream data in the Dynatrace console.</p>\n<ol>\n<li>Log in to the Dynatrace Console, and on the left side menu expand the “<strong>infrastructure</strong>” section, and select “<strong>AWS</strong>”</li>\n<li>From the screen, select the AWS account that you want to add the KFH stream to.</li>\n<li>Next, you’ll see a virtualization of your AWS assets for the account selected. Select the box marked “<strong>Supporting Services</strong>”.</li>\n<li>Next, press the “<strong>Configure services</strong>” button.</li>\n<li>Next, select “<strong>Add service</strong>”.</li>\n<li>From the drop down, select “Kinesis Data Firehose”.</li>\n<li>Next, select the “<strong>Add metric</strong>” button, and select the metrics that you want to see for this stream. Dynatrace has a comprehensive list of metrics that can be selected from the UI. The list can be found in this <a href=\"https://www.dynatrace.com/support/help/shortlink/kinesis#available-metrics\" target=\"_blank\">link</a>.</li>\n</ol>\n<h4><a id=\"Troubleshooting_112\"></a><strong>Troubleshooting</strong></h4>\n<ol>\n<li>After configuration, load to the new KFH stream no data in the Dynatrace Console.<br />\na.Check the Error Logs tab check to make sure that the Destination URL is correct for the Dynatrace Tenant.</li>\n</ol>\n<p><img src=\"https://dev-media.amazoncloud.cn/60a3ac4ef641408e85080033de111a98_image.png\" alt=\"image.png\" /></p>\n<p>Figure 10 – Destination error logs</p>\n<ol start=\"2\">\n<li>Invalid or misconfigured Dynatrace API token or scope isn’t properly set.</li>\n</ol>\n<p><img src=\"https://dev-media.amazoncloud.cn/4cc82cb79e5b45eb90cd2ca2f5f9e89b_image.png\" alt=\"image.png\" /></p>\n<p>Figure 11 – Destination error logs</p>\n<h4><a id=\"Conclusion_131\"></a><strong>Conclusion</strong></h4>\n<p>In this post, we demonstrate the Kinesis Data Firehose and related Dynatrace integration. In addition, engineers can use CloudWatch Metrics to explore their production systems alongside events in Dynatrace. This provides a seamless, current view of your system (from logs to events and traces) in a single data store.</p>\n<p>To learn more about CloudWatch Service, see the <a href=\"https://docs.aws.amazon.com/cloudwatch/index.html\" target=\"_blank\">Amazon CloudWatch home page</a>. If you have any questions, post them on the <a href=\"https://forums.aws.amazon.com/forum.jspa?forumID=138\" target=\"_blank\">AWS CloudWatch service forum</a>.</p>\n<p>If you haven’t yet signed up for Dynatrace, then you can try out Kinesis Data Firehose with Dynatrace with a <a href=\"https://www.dynatrace.com/trial/\" target=\"_blank\">free Dynatrace trial</a>.</p>\n<h4><a id=\"About_the_Authors_140\"></a><strong>About the Authors</strong></h4>\n<p><img src=\"https://dev-media.amazoncloud.cn/0f183442e23340f3a11b1abdfc4fbd3a_image.png\" alt=\"image.png\" /></p>\n<p><strong>Erick Leon</strong> is a Technical Alliances Sr. Manager at Dynatrace, Observability Practice Architect, and Customer Advocate. He promotes strong technical integrations with a focus on AWS. With over 15 years as a Dynatrace customer, his real-world experiences and lessons learned bring valuable insights into the Dynatrace Intelligent Observability Platform.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/677c6e351c5545409e55c313b98d6c4f_image.png\" alt=\"image.png\" /></p>\n<p><strong>Shashiraj Jeripotula (Raj)</strong> is a San Francisco-based Sr. Partner Solutions Architect at AWS. He works with various independent software vendors (ISVs), and partners who specialize in cloud management tools and DevOps to develop joint solutions and accelerate cloud adoption on AWS.</p>\n"}
目录
亚马逊云科技解决方案 基于行业客户应用场景及技术领域的解决方案
联系亚马逊云科技专家
亚马逊云科技解决方案
基于行业客户应用场景及技术领域的解决方案
联系专家
0
目录
关闭
contact-us