Dream11: Blocking application attacks using Amazon WAF at scale

{"value":"As the world’s largest fantasy sports platforms with more than 120 million registered users, Dream11 runs multiple contests simultaneously while processing millions of user requests per minute. Their user-centric and data-driven teams make it a priority to ensure that the Dream11 application (app) remains protected against all kinds of threats and vulnerabilities.\n\n#### **Introduction to Amazon Web Services WAF Security Automations**\n\n[Amazon Web Services WAF](https://aws.amazon.com/waf/) is a web application firewall that helps protect apps and APIs against common web exploits and bots. These attacks may affect availability, compromise security, or consume excessive resources. Amazon Web Services WAF gives you control over how traffic reaches your applications. You can create security rules that control bot traffic and block common attack patterns, such as SQL injection or cross-site scripting (XSS.)\n\n[Amazon Web Services WAF Security Automations](https://aws.amazon.com/solutions/implementations/aws-waf-security-automations/) use [ Amazon Web Services CloudFormation ](https://aws.amazon.com/cloudformation/) to quickly configure Amazon Web Services WAF rules that help block the following common types of attacks:\n\n- SQL injection\n- Cross-site scripting\n- HTTP floods\n- Scanners and probes\n- Known attacker origins (IP reputation lists)\n- Bots and scrapers\n\nIn this blog post, we will explain how Dream11 uses Amazon Web Services WAF Security Automations to protect its application from scanners and probes attacks.\n\n#### **Scanner and probe automation**\nTo understand the scanner and probe automation, let’s look at a realistic attack scenario for a standard app that is protected by Amazon Web Services WAF. Let’s assume that a malicious user is trying to scan the app and identify loopholes using their custom tool. They plan to conduct injection attacks (such as SQLi, XSS) or directory brute force attacks.\n\nThe app, secured by Amazon Web Services WAF, has rules in place to block requests if certain signatures and patterns are matched. Amazon Web Services WAF cannot have all possible payload lists for each attack vector. This means that after some trial and errors, an attacker may find the payload that doesn’t get blocked by Amazon Web Services WAF and try to exploit the vulnerability.\n\nIn this case, what if Amazon Web Services WAF can detect the behavior of malicious user IPs and block it for a certain time period? Wouldn’t it be great if Amazon Web Services WAF blocks the IP of a malicious user after receiving a couple of malicious requests? That way, new requests coming from that IP will be blocked without Amazon Web Services WAF having to check all the rules in the web ACL. Any successful bypass attempts will also get blocked from that IP. Rather than permanently blocking the IP, this feature blocks the offending IP for a certain time period, discouraging the attacker from any further attempts. It acts as a first step of incident response. Here’s where automation can help.\n\nScanner and probe automation monitors [Amazon CloudFront](https://aws.amazon.com/cloudfront/) logs and analyses HTTP status codes for requests coming from different IPs. Based on the configured threshold of HTTP status codes, scanner and probe automation will update the malicious IP directly to the Amazon Web Services WAF rule IPSet. It then blocks subsequent requests from that IP for a configured period of time.\n\nThe Amazon Web Services WAF Security Automations solution creates an Amazon Web Services WAF rule, an[ Amazon Web Services Lambda ](https://aws.amazon.com/lambda/) function, and a Scanner and Probes [ Amazon Athena](https://aws.amazon.com/athena/) query. The Athena query parses Amazon CloudFront or[Application Load Balancer ](https://aws.amazon.com/elasticloadbalancing/application-load-balancer/) access logs at regular intervals. It counts the number of bad requests per minute from unique source IP addresses. The Lambda function updates the Amazon Web Services WAF IPSet rule to block further scans from IP addresses with a high error rate.\n\n\n\nFigure 1. Solution architecture for scanner and probe automation (xxx represents the numbers as defined by the use case)\n\nThe workflow of the solution is as follows, shown in Figure 1:\n\n- CloudFront logs are pushed to the [Amazon S3 ](https://aws.amazon.com/s3/) bucket\n- Log Parser Lambda will run the Athena query to find the error code threshold for each unique IP\n- If the HTTP error threshold is crossed for any IP, the Lambda function will update the IP into an Amazon Web Services WAF IPSet for a certain time\n- The IPSet is unblocked automatically after the time period is over\n\n#### **Customizing the Amazon Web Services WAF Security Automation solution**\nScanner and probe automation with rules will block traffic if the error rate for a particular IP crosses the threshold. It then adds the IP in the blocked IPSet. This IP is blocked for a configurable amount of time (for example, 12 hours, 2 days, 1 week).\n\nDuring the customization of Amazon Web Services WAF for Dream11, there were instances which required exceptions to the preceding rule. One was to prevent internal services/gateway IPs from getting blocked by the security automation. We needed to customize the rules for these predefined thresholds. For example: the solution should block the external traffic, but exclude any internal IP addresses.\n\nThe Dream11 Security team customized the Lambda logic to approve all internal NAT gateway IPs. Scanner and probe automation ignores these IPs even if there is a high number of errors from the approved IPs. Sample code is as follows:\n\n```\nlog.info(\"[update_ip_set] \\tIgnore the approved IP \")\n\nif ip_type == \"IPV4\" and source_ip not in outstanding_requesters['ApprovedIPs']: \n addresses_v4.append(source_ip)\nelif ip_type == \"IPV6\" and source_ip not in outstanding_requesters['ApprovedIPs']: addresses_v6.append(source_ip)\n```\n\n**Note:** Create a JSON file with list of approved IPs and store it in APP_ACCESS_LOG_BUCKET\nWe will use the same S3 bucket to put our office-approved IPs as xyz.json file where we store our CloudFront access logs. This is configurable during CloudFormation template for Security Automation.\n\n##### *Code explanation:*\n1. The custom code first validates the particular IP for which the error threshold is crossed against the approved IPs.\n2. If the IP belongs to the IPV4 or IPV6 format and isn’t an approved IP, it will be appended to the blocked IPSet for a certain period of time.\n\nThe customization of the Lambda function provides a security automation solution that doesn’t block any legitimate request. At the same time, it provides protection against scanner and probe attacks. Amazon Web Services WAF security automation is an open-source solution and is hosted on [GitHub](https://github.com/awslabs/aws-waf-security-automations).\n\n#### **Conclusion**\nIn this blog post, we’ve given a brief overview of how you can reduce attacks by using Amazon Web Services WAF Security Automations against scanners and probes. We’ve also illustrated the customization implemented by the Dream11 security team.\n\nBy automating your security operations, you will improve effective incident response. You can prioritize threats and handle cyber attacks automatically with automated courses of action. This reduces the need for human intervention, reduces response time, and addresses security issues without manual effort.\n\nAfter implementing this at Dream11, we were able to create custom, application-specific rules that blocked attack patterns. This has provided application availability, secure resources, and has prevented excessive resource consumption. With this solution, we are able to provide the best fantasy sports experience for over 120 million users.\n\nRead more about [Security Automations in Amazon Web Services WAF](https://aws.amazon.com/solutions/implementations/aws-waf-security-automations/).\n\n\n\n**Vatsal Shah**\nVatsal Shah is a Senior Solutions Architect at Amazon Web Services based out of Mumbai, India. Vatsal has more than 8 years of industry experience, including roles in cloud architecture, systems engineering, and software design. He currently focuses on enabling large startups to streamline their cloud operations and optimize their cloud spend. His area of interest is in machine learning.\n\n\n\n**Parag Suchak**\nParag Suchak is a Security Engineer at Dream11 in Mumbai, India, with over six years of experience is Web Application Security, Mobile Security, and Automation. His professional expertise includes application penetration testing, secure code review, Network VAPT, and Cloud Security.\n\n\n\n**Ritesh Kumar**\nRitesh Kumar is a Security Engineer at Dream11 in Mumbai, India. He has a background in Application Security, and provides expertise in architectures that reduce the likelihood of data leakage, application, and denial-of-service attacks. He enjoys working on complex topics and processes, and breaking them down into simpler components.\n\n\n\n**Sanjana Potdar**\nSanjana Potdar is a cloud Security Engineer at Dream11 in Mumbai, India. She has a master’s degree in Computer Science with a major in System Security. She is passionate about security, cloud services, and building solutions around them. Outside of work, she is an avid reader and traveler.","render":"<p>As the world’s largest fantasy sports platforms with more than 120 million registered users, Dream11 runs multiple contests simultaneously while processing millions of user requests per minute. Their user-centric and data-driven teams make it a priority to ensure that the Dream11 application (app) remains protected against all kinds of threats and vulnerabilities.</p>\n<h4><a id=\"Introduction_to_Amazon_Web_Services_WAF_Security_Automations_2\"></a><strong>Introduction to Amazon Web Services WAF Security Automations</strong></h4>\n<p><a href=\"https://aws.amazon.com/waf/\" target=\"_blank\">Amazon Web Services WAF</a> is a web application firewall that helps protect apps and APIs against common web exploits and bots. These attacks may affect availability, compromise security, or consume excessive resources. Amazon Web Services WAF gives you control over how traffic reaches your applications. You can create security rules that control bot traffic and block common attack patterns, such as SQL injection or cross-site scripting (XSS.)</p>\n<p><a href=\"https://aws.amazon.com/solutions/implementations/aws-waf-security-automations/\" target=\"_blank\">Amazon Web Services WAF Security Automations</a> use <a href=\"https://aws.amazon.com/cloudformation/\" target=\"_blank\"> Amazon Web Services CloudFormation </a> to quickly configure Amazon Web Services WAF rules that help block the following common types of attacks:</p>\n<ul>\n<li>SQL injection</li>\n<li>Cross-site scripting</li>\n<li>HTTP floods</li>\n<li>Scanners and probes</li>\n<li>Known attacker origins (IP reputation lists)</li>\n<li>Bots and scrapers</li>\n</ul>\n<p>In this blog post, we will explain how Dream11 uses Amazon Web Services WAF Security Automations to protect its application from scanners and probes attacks.</p>\n<h4><a id=\"Scanner_and_probe_automation_17\"></a><strong>Scanner and probe automation</strong></h4>\n<p>To understand the scanner and probe automation, let’s look at a realistic attack scenario for a standard app that is protected by Amazon Web Services WAF. Let’s assume that a malicious user is trying to scan the app and identify loopholes using their custom tool. They plan to conduct injection attacks (such as SQLi, XSS) or directory brute force attacks.</p>\n<p>The app, secured by Amazon Web Services WAF, has rules in place to block requests if certain signatures and patterns are matched. Amazon Web Services WAF cannot have all possible payload lists for each attack vector. This means that after some trial and errors, an attacker may find the payload that doesn’t get blocked by Amazon Web Services WAF and try to exploit the vulnerability.</p>\n<p>In this case, what if Amazon Web Services WAF can detect the behavior of malicious user IPs and block it for a certain time period? Wouldn’t it be great if Amazon Web Services WAF blocks the IP of a malicious user after receiving a couple of malicious requests? That way, new requests coming from that IP will be blocked without Amazon Web Services WAF having to check all the rules in the web ACL. Any successful bypass attempts will also get blocked from that IP. Rather than permanently blocking the IP, this feature blocks the offending IP for a certain time period, discouraging the attacker from any further attempts. It acts as a first step of incident response. Here’s where automation can help.</p>\n<p>Scanner and probe automation monitors <a href=\"https://aws.amazon.com/cloudfront/\" target=\"_blank\">Amazon CloudFront</a> logs and analyses HTTP status codes for requests coming from different IPs. Based on the configured threshold of HTTP status codes, scanner and probe automation will update the malicious IP directly to the Amazon Web Services WAF rule IPSet. It then blocks subsequent requests from that IP for a configured period of time.</p>\n<p>The Amazon Web Services WAF Security Automations solution creates an Amazon Web Services WAF rule, an<a href=\"https://aws.amazon.com/lambda/\" target=\"_blank\"> Amazon Web Services Lambda </a> function, and a Scanner and Probes <a href=\"https://aws.amazon.com/athena/\" target=\"_blank\"> Amazon Athena</a> query. The Athena query parses Amazon CloudFront or<a href=\"https://aws.amazon.com/elasticloadbalancing/application-load-balancer/\" target=\"_blank\">Application Load Balancer </a> access logs at regular intervals. It counts the number of bad requests per minute from unique source IP addresses. The Lambda function updates the Amazon Web Services WAF IPSet rule to block further scans from IP addresses with a high error rate.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/436ec66c606b4f4f998cd0c6178f463a_image.png\" alt=\"image.png\" /></p>\n<p>Figure 1. Solution architecture for scanner and probe automation (xxx represents the numbers as defined by the use case)</p>\n<p>The workflow of the solution is as follows, shown in Figure 1:</p>\n<ul>\n<li>CloudFront logs are pushed to the <a href=\"https://aws.amazon.com/s3/\" target=\"_blank\">Amazon S3 </a> bucket</li>\n<li>Log Parser Lambda will run the Athena query to find the error code threshold for each unique IP</li>\n<li>If the HTTP error threshold is crossed for any IP, the Lambda function will update the IP into an Amazon Web Services WAF IPSet for a certain time</li>\n<li>The IPSet is unblocked automatically after the time period is over</li>\n</ul>\n<h4><a id=\"Customizing_the_Amazon_Web_Services_WAF_Security_Automation_solution_39\"></a><strong>Customizing the Amazon Web Services WAF Security Automation solution</strong></h4>\n<p>Scanner and probe automation with rules will block traffic if the error rate for a particular IP crosses the threshold. It then adds the IP in the blocked IPSet. This IP is blocked for a configurable amount of time (for example, 12 hours, 2 days, 1 week).</p>\n<p>During the customization of Amazon Web Services WAF for Dream11, there were instances which required exceptions to the preceding rule. One was to prevent internal services/gateway IPs from getting blocked by the security automation. We needed to customize the rules for these predefined thresholds. For example: the solution should block the external traffic, but exclude any internal IP addresses.</p>\n<p>The Dream11 Security team customized the Lambda logic to approve all internal NAT gateway IPs. Scanner and probe automation ignores these IPs even if there is a high number of errors from the approved IPs. Sample code is as follows:</p>\n<pre><code class=\"lang-\">log.info(&quot;[update_ip_set] \\tIgnore the approved IP &quot;)\n\nif ip_type == &quot;IPV4&quot; and source_ip not in outstanding_requesters['ApprovedIPs']: \n addresses_v4.append(source_ip)\nelif ip_type == &quot;IPV6&quot; and source_ip not in outstanding_requesters['ApprovedIPs']: addresses_v6.append(source_ip)\n</code></pre>\n<p><strong>Note:</strong> Create a JSON file with list of approved IPs and store it in APP_ACCESS_LOG_BUCKET<br />\nWe will use the same S3 bucket to put our office-approved IPs as xyz.json file where we store our CloudFront access logs. This is configurable during CloudFormation template for Security Automation.</p>\n<h5><a id=\"Code_explanation_57\"></a><em>Code explanation:</em></h5>\n<ol>\n<li>The custom code first validates the particular IP for which the error threshold is crossed against the approved IPs.</li>\n<li>If the IP belongs to the IPV4 or IPV6 format and isn’t an approved IP, it will be appended to the blocked IPSet for a certain period of time.</li>\n</ol>\n<p>The customization of the Lambda function provides a security automation solution that doesn’t block any legitimate request. At the same time, it provides protection against scanner and probe attacks. Amazon Web Services WAF security automation is an open-source solution and is hosted on <a href=\"https://github.com/awslabs/aws-waf-security-automations\" target=\"_blank\">GitHub</a>.</p>\n<h4><a id=\"Conclusion_63\"></a><strong>Conclusion</strong></h4>\n<p>In this blog post, we’ve given a brief overview of how you can reduce attacks by using Amazon Web Services WAF Security Automations against scanners and probes. We’ve also illustrated the customization implemented by the Dream11 security team.</p>\n<p>By automating your security operations, you will improve effective incident response. You can prioritize threats and handle cyber attacks automatically with automated courses of action. This reduces the need for human intervention, reduces response time, and addresses security issues without manual effort.</p>\n<p>After implementing this at Dream11, we were able to create custom, application-specific rules that blocked attack patterns. This has provided application availability, secure resources, and has prevented excessive resource consumption. With this solution, we are able to provide the best fantasy sports experience for over 120 million users.</p>\n<p>Read more about <a href=\"https://aws.amazon.com/solutions/implementations/aws-waf-security-automations/\" target=\"_blank\">Security Automations in Amazon Web Services WAF</a>.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/ed6a59503a5d4593aa8d68ca0bea83ae_image.png\" alt=\"image.png\" /></p>\n<p><strong>Vatsal Shah</strong><br />\nVatsal Shah is a Senior Solutions Architect at Amazon Web Services based out of Mumbai, India. Vatsal has more than 8 years of industry experience, including roles in cloud architecture, systems engineering, and software design. He currently focuses on enabling large startups to streamline their cloud operations and optimize their cloud spend. His area of interest is in machine learning.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/d98f5eb6bcc6414597ba7b2891c2b2b6_image.png\" alt=\"image.png\" /></p>\n<p><strong>Parag Suchak</strong><br />\nParag Suchak is a Security Engineer at Dream11 in Mumbai, India, with over six years of experience is Web Application Security, Mobile Security, and Automation. His professional expertise includes application penetration testing, secure code review, Network VAPT, and Cloud Security.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/f39d6f12f6954d6caa3a5ec53358ac83_image.png\" alt=\"image.png\" /></p>\n<p><strong>Ritesh Kumar</strong><br />\nRitesh Kumar is a Security Engineer at Dream11 in Mumbai, India. He has a background in Application Security, and provides expertise in architectures that reduce the likelihood of data leakage, application, and denial-of-service attacks. He enjoys working on complex topics and processes, and breaking them down into simpler components.</p>\n<p><img src=\"https://dev-media.amazoncloud.cn/56c433bb1b524517abe0af396457110a_image.png\" alt=\"image.png\" /></p>\n<p><strong>Sanjana Potdar</strong><br />\nSanjana Potdar is a cloud Security Engineer at Dream11 in Mumbai, India. She has a master’s degree in Computer Science with a major in System Security. She is passionate about security, cloud services, and building solutions around them. Outside of work, she is an avid reader and traveler.</p>\n"}