Astro’s Intelligent Motion brings state-of-the-art navigation to the home

{"value":"<video src=\\"https://dev-media.amazoncloud.cn/38bc442de6f3436e8c8fc28d9cea6eec_The%20Science%20Behind%20Astro%20_%20Amazon%20Astro.mp4\\" class=\\"manvaVedio\\" controls=\\"controls\\" style=\\"width:160px;height:160px\\"></video>\n\n*At a virtual event today, Amazon’s senior vice president for devices, Dave Limp, unveiled his organization’s new lineup of devices, which included Astro, a household robot with home monitoring and Alexa.*\n\nBuilding a robot that can move intelligently around your home is no easy task. When building self-driving cars or robots for industrial applications, you can predefine high-definition maps of the environments they will encounter on the open road or factory floor. But in the home, nothing is predefined, with frequently rearranged furniture and belongings and people and pets always in motion.\n\nWhen we set out to build Astro, we knew we wanted its motion to be intuitive and graceful, and we wanted it to be able to interact naturally with humans. That meant that we had to account for the dynamism of the home when deciding on Astro’s design, its sensor configuration, its algorithms, and the speed at which it moves. In addition, we had to deliver Astro at a consumer-accessible price point with a highly optimized suite of sensors and sufficient processing power, when the sensors and processors for other industrial robots that operate at similar speeds can cost thousands of dollars.\n\nSo how did we create Intelligent Motion for Astro? \n\n\n\n#### **Perception and mapping**\n\nTo be able to move around your home, Astro needs to effectively map its surroundings and understand where it is at any given point: this is perception. Astro’s computer vision system observes the world with both visible and infrared light, which gives it robust perception in dynamic environments and varying lighting conditions. As it perceives where it is in a space, Astro uses its suite of navigation and obstacle sensors as inputs to its on-device ++[simultaneous localization and mapping (SLAM)](https://www.amazon.science/blog/a-more-useful-way-to-measure-robotic-localization-error)++ and obstacle avoidance systems. \n\nThe navigation sensors help identify the positions of key landmarks in 3-D space for the SLAM system, such as corners of tables and doorframes, so that Astro can figure out where it is relative to these landmarks. Astro builds a map of the relative positions of these sparse landmarks when it explores your home and then uses the landmarks to update its location as it moves through the home. \n\nThe obstacle sensors help Astro build a detailed map of its immediate surroundings, capturing the distance to obstacles like couches, chairs, walls, and stairs (see figure below). Astro then uses its knowledge of its position from SLAM and its map of obstacles to path plan and interact with its environment, performing complex tasks such as exploring the home and determining boundaries between spaces, following and approaching people, and figuring out where to hang out. We’ll dive deeper into Intelligent Motion’s SLAM and obstacle avoidance systems in a future science blog post.\n\n\n\nAstro’s Intelligent Motion algorithms build a depth map of Astro's surroundings for mapping and path planning.\n\n#### **Real-time planning**\n\nIntelligent Motion is all about having Astro make decisions quickly and autonomously. Homes are ever-changing and full of moving obstacles. For that reason, Astro’s knowledge of its world is rarely perfect, so its navigation system has to be able to handle variability. \n\n\n\nAstro’s path-planning algorithm tests hundreds of options in real time. Blue arrows indicate longer-range route guidance; colored lines represent options for close-range trajectories within the next three seconds. The colors represent scoring of the trajectories across many weighted factors.\n\nAs Astro navigates the home, the Intelligent Motion system generates several hundred potential paths several times a second, evaluates each of them, and then makes a determination on how to move. This process factors in the possibility of changes in the environment (e.g., a book bag dropped on the floor), the desired smoothness of the motion path, and the potential for encountering obstacles.\n\nAstro weighs how each choice contributes toward achieving its current goal, whether that’s reaching a person or heading back to its charger. Astro keeps repeating this process while it is navigating, intelligently optimizing based on its latest knowledge of its world. This approach involves novel methods for dimensionality reduction and probabilistic planning that advance the state of the art in the field of consumer robotics. We’ll also cover this more in a future science blog post.\n\n#### **Body language and communication of intent**\n\nControlling speed, acceleration, and the curvature of Astro’s path are important for making sure Astro can move safely, gracefully, and confidently through the home, but Astro needs to do even more when it interacts with humans. Human-robot interaction (HRI) is a rapidly growing area of research, one that Amazon has invested in in its study of consumer robotics. \n\nAstro builds trust with customers by moving with predictable behaviors, such as signaling its intents through body language. People and pets do the same thing — signaling, for instance, how they plan to move with a slight turn of the head, change in shoulder angle, or change in eye direction. These are signals people pick up on without even realizing it. \n\nEmulating these patterns, Astro uses natural changes in head angle as it moves around, indicating which way it is going to turn, pointing at the person it is approaching, and more. When we tested these features, the difference in customer experience with and without them was clear. A simple signal executed via well-coordinated screen and body movements is a powerful tool for communicating intent in real time and making Astro’s behavior more natural.\n\n#### **More coverage of devices and services announcements**\n\n- \"++[Unlocking AI for everyone](https://www.aboutamazon.com/news/devices/unlocking-ai-for-everyone)++\"\n- \"++[How Amazon is using self-service to democratize AI](https://www.amazon.science/blog/how-amazon-is-using-self-service-to-democratize-ai)++\"\n- \"++[A more useful way to measure robotic localization error](https://www.amazon.science/blog/a-more-useful-way-to-measure-robotic-localization-error)++\"\n- \"++[The science behind visual ID](https://www.amazon.science/blog/the-science-behind-visual-id)++\"\n\n#### **Moving at humanlike speeds**\n\nAstro’s ability to interact naturally with people helps make it even more useful in customers’ homes. Astro can tell when an obstacle is a person and make decisions about how to interact appropriately. To do this, Astro has to operate at human-scale speeds and have an awareness of social norms. \n\n\n\nWhen following a person, Astro maintains a socially appropriate distance.\n\nFor example, when Astro approaches a person, Intelligent Motion uses computer vision signals like the approximate position of that person relative to Astro and the direction the person is facing, the stored map for the area, and other inputs from Astro’s navigation and depth sensors to plan a smooth, graceful path that will enable Astro to end up in front of the person, in the person’s line of sight, at a socially appropriate distance. \n\nIf Astro is following a person, Intelligent Motion helps Astro follow at a comfortable strolling pace for an adult, maintaining a socially appropriate distance, and estimating where that person goes when moving out of view so that Astro can move to a point where the person can be seen and followed again. Astro can determine when an obstacle it detects is a person and follow that obstacle instead of avoiding it, even when it moves in and out of Astro’s field of view. This approach involves dynamic obstacle recognition and tracking, path planning, proxemics, and HRI that we’re excited to share more about soon. \n\n#### **Recovering from difficult situations**\n\nDespite its navigation prowess, Astro will still encounter situations that require it to problem-solve to avoid the need for human intervention. Intelligent Motion includes a set of recovery behaviors that can help when Astro encounters challenges to normal path planning, such as a narrow path that is currently blocked. \n\nTo continue with its task in the face of a blocked path, Astro might try backing up until there is enough space to turn around. As part of this process, Astro also determines when it is time ask for help. We know from our internal testing that people don’t mind occasionally helping Astro, though we have also learned that people have limited patience for a robot that gives up too often and is always asking for help. \n\n\n\nAstro heads for a gap but is blocked, so the planner calculates new waypoints (blue arrows), and the recovery planner finds a way out and onto the new path.\n\n#### **How Intelligent Motion is designed to protect customer privacy**\n\nMoving and reacting quickly requires a very fast system, making local processing of data essential. The raw data from the navigation and obstacle sensors is locally processed into a distance measurement and then discarded, without being sent to the cloud.\n\nWhen Astro saves a new map at the completion of exploration, information derived from its navigation and depth sensors, including a copy of the 2-D obstacle map, is sent to the cloud, where a map of the home is created and stored. A rendering of the map can then be shown in the Astro app. \n\nThis map contains derived information such as the location of walls, rooms, boundaries, furniture, and objects, plus related data such as customer-provided room names. Map data is encrypted in transit to the cloud, where it is securely stored with 256-bit keys, an industry standard for secure encryption. For more information about the way Astro protects customer privacy, visit ++[amazon.com/astroprivacy](https://www.amazon.com/astroprivacy)++. \n\n#### **What's next?**\n\nAstro is Amazon’s first household robot to use Intelligent Motion to gracefully and intuitively interact with people, help customers monitor their homes, bring the power of Alexa to them, and give them back time in their busy lives. \n\nThis is just the beginning for Intelligent Motion, with its navigation and HRI capabilities. We have exciting plans for advancing the science and engineering of Intelligent Motion so that it will improve over time at navigating in homes and serving customers’ needs. We also expect to learn a lot from our customers, who have never had a product quite like Astro in their homes before. Astro’s Intelligent Motion is a brand-new experience that we can’t wait for you to try, and we’re excited to have you join us on the journey.\n\nABOUT THE AUTHOR\n\n#### **[Jong Jin Park](https://www.amazon.science/author/jong-jin-park)**\n\nJong Jin Park is a senior applied scientist at Amazon.\n\n#### **[Arnie Sen](https://www.amazon.science/author/arnie-sen)**\n\nArnie Sen is a senior manager of software development at Amazon.\n\n","render":"<p><video src=\\"https://dev-media.amazoncloud.cn/38bc442de6f3436e8c8fc28d9cea6eec_The%20Science%20Behind%20Astro%20_%20Amazon%20Astro.mp4\\" controls=\\"controls\\"></video></p>\\n<p><em>At a virtual event today, Amazon’s senior vice president for devices, Dave Limp, unveiled his organization’s new lineup of devices, which included Astro, a household robot with home monitoring and Alexa.</em></p>\\n<p>Building a robot that can move intelligently around your home is no easy task. When building self-driving cars or robots for industrial applications, you can predefine high-definition maps of the environments they will encounter on the open road or factory floor. But in the home, nothing is predefined, with frequently rearranged furniture and belongings and people and pets always in motion.</p>\n<p>When we set out to build Astro, we knew we wanted its motion to be intuitive and graceful, and we wanted it to be able to interact naturally with humans. That meant that we had to account for the dynamism of the home when deciding on Astro’s design, its sensor configuration, its algorithms, and the speed at which it moves. In addition, we had to deliver Astro at a consumer-accessible price point with a highly optimized suite of sensors and sufficient processing power, when the sensors and processors for other industrial robots that operate at similar speeds can cost thousands of dollars.</p>\n<p>So how did we create Intelligent Motion for Astro?</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/8e9a2cb7de0b44f1990deccc9cfb1540_image.png\\" alt=\\"image.png\\" /></p>\n<h4><a id=\\"Perception_and_mapping_12\\"></a><strong>Perception and mapping</strong></h4>\\n<p>To be able to move around your home, Astro needs to effectively map its surroundings and understand where it is at any given point: this is perception. Astro’s computer vision system observes the world with both visible and infrared light, which gives it robust perception in dynamic environments and varying lighting conditions. As it perceives where it is in a space, Astro uses its suite of navigation and obstacle sensors as inputs to its on-device <ins><a href=\\"https://www.amazon.science/blog/a-more-useful-way-to-measure-robotic-localization-error\\" target=\\"_blank\\">simultaneous localization and mapping (SLAM)</a></ins> and obstacle avoidance systems.</p>\n<p>The navigation sensors help identify the positions of key landmarks in 3-D space for the SLAM system, such as corners of tables and doorframes, so that Astro can figure out where it is relative to these landmarks. Astro builds a map of the relative positions of these sparse landmarks when it explores your home and then uses the landmarks to update its location as it moves through the home.</p>\n<p>The obstacle sensors help Astro build a detailed map of its immediate surroundings, capturing the distance to obstacles like couches, chairs, walls, and stairs (see figure below). Astro then uses its knowledge of its position from SLAM and its map of obstacles to path plan and interact with its environment, performing complex tasks such as exploring the home and determining boundaries between spaces, following and approaching people, and figuring out where to hang out. We’ll dive deeper into Intelligent Motion’s SLAM and obstacle avoidance systems in a future science blog post.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/3862ff2a38984711833cc96e9d6cd29b_image.png\\" alt=\\"image.png\\" /></p>\n<p>Astro’s Intelligent Motion algorithms build a depth map of Astro’s surroundings for mapping and path planning.</p>\n<h4><a id=\\"Realtime_planning_24\\"></a><strong>Real-time planning</strong></h4>\\n<p>Intelligent Motion is all about having Astro make decisions quickly and autonomously. Homes are ever-changing and full of moving obstacles. For that reason, Astro’s knowledge of its world is rarely perfect, so its navigation system has to be able to handle variability.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/a09580b66c364404a75111a951a84ec3_image.png\\" alt=\\"image.png\\" /></p>\n<p>Astro’s path-planning algorithm tests hundreds of options in real time. Blue arrows indicate longer-range route guidance; colored lines represent options for close-range trajectories within the next three seconds. The colors represent scoring of the trajectories across many weighted factors.</p>\n<p>As Astro navigates the home, the Intelligent Motion system generates several hundred potential paths several times a second, evaluates each of them, and then makes a determination on how to move. This process factors in the possibility of changes in the environment (e.g., a book bag dropped on the floor), the desired smoothness of the motion path, and the potential for encountering obstacles.</p>\n<p>Astro weighs how each choice contributes toward achieving its current goal, whether that’s reaching a person or heading back to its charger. Astro keeps repeating this process while it is navigating, intelligently optimizing based on its latest knowledge of its world. This approach involves novel methods for dimensionality reduction and probabilistic planning that advance the state of the art in the field of consumer robotics. We’ll also cover this more in a future science blog post.</p>\n<h4><a id=\\"Body_language_and_communication_of_intent_36\\"></a><strong>Body language and communication of intent</strong></h4>\\n<p>Controlling speed, acceleration, and the curvature of Astro’s path are important for making sure Astro can move safely, gracefully, and confidently through the home, but Astro needs to do even more when it interacts with humans. Human-robot interaction (HRI) is a rapidly growing area of research, one that Amazon has invested in in its study of consumer robotics.</p>\n<p>Astro builds trust with customers by moving with predictable behaviors, such as signaling its intents through body language. People and pets do the same thing — signaling, for instance, how they plan to move with a slight turn of the head, change in shoulder angle, or change in eye direction. These are signals people pick up on without even realizing it.</p>\n<p>Emulating these patterns, Astro uses natural changes in head angle as it moves around, indicating which way it is going to turn, pointing at the person it is approaching, and more. When we tested these features, the difference in customer experience with and without them was clear. A simple signal executed via well-coordinated screen and body movements is a powerful tool for communicating intent in real time and making Astro’s behavior more natural.</p>\n<h4><a id=\\"More_coverage_of_devices_and_services_announcements_44\\"></a><strong>More coverage of devices and services announcements</strong></h4>\\n<ul>\\n<li>“<ins><a href=\\"https://www.aboutamazon.com/news/devices/unlocking-ai-for-everyone\\" target=\\"_blank\\">Unlocking AI for everyone</a></ins>”</li>\n<li>“<ins><a href=\\"https://www.amazon.science/blog/how-amazon-is-using-self-service-to-democratize-ai\\" target=\\"_blank\\">How Amazon is using self-service to democratize AI</a></ins>”</li>\n<li>“<ins><a href=\\"https://www.amazon.science/blog/a-more-useful-way-to-measure-robotic-localization-error\\" target=\\"_blank\\">A more useful way to measure robotic localization error</a></ins>”</li>\n<li>“<ins><a href=\\"https://www.amazon.science/blog/the-science-behind-visual-id\\" target=\\"_blank\\">The science behind visual ID</a></ins>”</li>\n</ul>\\n<h4><a id=\\"Moving_at_humanlike_speeds_51\\"></a><strong>Moving at humanlike speeds</strong></h4>\\n<p>Astro’s ability to interact naturally with people helps make it even more useful in customers’ homes. Astro can tell when an obstacle is a person and make decisions about how to interact appropriately. To do this, Astro has to operate at human-scale speeds and have an awareness of social norms.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/9a639d104ac64ac6a5a2a164c31e8a5a_image.png\\" alt=\\"image.png\\" /></p>\n<p>When following a person, Astro maintains a socially appropriate distance.</p>\n<p>For example, when Astro approaches a person, Intelligent Motion uses computer vision signals like the approximate position of that person relative to Astro and the direction the person is facing, the stored map for the area, and other inputs from Astro’s navigation and depth sensors to plan a smooth, graceful path that will enable Astro to end up in front of the person, in the person’s line of sight, at a socially appropriate distance.</p>\n<p>If Astro is following a person, Intelligent Motion helps Astro follow at a comfortable strolling pace for an adult, maintaining a socially appropriate distance, and estimating where that person goes when moving out of view so that Astro can move to a point where the person can be seen and followed again. Astro can determine when an obstacle it detects is a person and follow that obstacle instead of avoiding it, even when it moves in and out of Astro’s field of view. This approach involves dynamic obstacle recognition and tracking, path planning, proxemics, and HRI that we’re excited to share more about soon.</p>\n<h4><a id=\\"Recovering_from_difficult_situations_63\\"></a><strong>Recovering from difficult situations</strong></h4>\\n<p>Despite its navigation prowess, Astro will still encounter situations that require it to problem-solve to avoid the need for human intervention. Intelligent Motion includes a set of recovery behaviors that can help when Astro encounters challenges to normal path planning, such as a narrow path that is currently blocked.</p>\n<p>To continue with its task in the face of a blocked path, Astro might try backing up until there is enough space to turn around. As part of this process, Astro also determines when it is time ask for help. We know from our internal testing that people don’t mind occasionally helping Astro, though we have also learned that people have limited patience for a robot that gives up too often and is always asking for help.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/6f0168b05bc345e7bb3939303e9a3cb9_image.png\\" alt=\\"image.png\\" /></p>\n<p>Astro heads for a gap but is blocked, so the planner calculates new waypoints (blue arrows), and the recovery planner finds a way out and onto the new path.</p>\n<h4><a id=\\"How_Intelligent_Motion_is_designed_to_protect_customer_privacy_73\\"></a><strong>How Intelligent Motion is designed to protect customer privacy</strong></h4>\\n<p>Moving and reacting quickly requires a very fast system, making local processing of data essential. The raw data from the navigation and obstacle sensors is locally processed into a distance measurement and then discarded, without being sent to the cloud.</p>\n<p>When Astro saves a new map at the completion of exploration, information derived from its navigation and depth sensors, including a copy of the 2-D obstacle map, is sent to the cloud, where a map of the home is created and stored. A rendering of the map can then be shown in the Astro app.</p>\n<p>This map contains derived information such as the location of walls, rooms, boundaries, furniture, and objects, plus related data such as customer-provided room names. Map data is encrypted in transit to the cloud, where it is securely stored with 256-bit keys, an industry standard for secure encryption. For more information about the way Astro protects customer privacy, visit <ins><a href=\\"https://www.amazon.com/astroprivacy\\" target=\\"_blank\\">amazon.com/astroprivacy</a></ins>.</p>\n<h4><a id=\\"Whats_next_81\\"></a><strong>What’s next?</strong></h4>\\n<p>Astro is Amazon’s first household robot to use Intelligent Motion to gracefully and intuitively interact with people, help customers monitor their homes, bring the power of Alexa to them, and give them back time in their busy lives.</p>\n<p>This is just the beginning for Intelligent Motion, with its navigation and HRI capabilities. We have exciting plans for advancing the science and engineering of Intelligent Motion so that it will improve over time at navigating in homes and serving customers’ needs. We also expect to learn a lot from our customers, who have never had a product quite like Astro in their homes before. Astro’s Intelligent Motion is a brand-new experience that we can’t wait for you to try, and we’re excited to have you join us on the journey.</p>\n<p>ABOUT THE AUTHOR</p>\n<h4><a id=\\"Jong_Jin_Parkhttpswwwamazonscienceauthorjongjinpark_89\\"></a><strong><a href=\\"https://www.amazon.science/author/jong-jin-park\\" target=\\"_blank\\">Jong Jin Park</a></strong></h4>\n<p>Jong Jin Park is a senior applied scientist at Amazon.</p>\n<h4><a id=\\"Arnie_Senhttpswwwamazonscienceauthorarniesen_93\\"></a><strong><a href=\\"https://www.amazon.science/author/arnie-sen\\" target=\\"_blank\\">Arnie Sen</a></strong></h4>\n<p>Arnie Sen is a senior manager of software development at Amazon.</p>\n"}