A more useful way to measure robotic localization error

{"value":"Simultaneous localization and mapping (SLAM) is one of the core capabilities of autonomous robots, with a rich literature behind it. “Mapping” means building a map of the robot’s environment, “localization” means identifying the robot’s location on the map, and “simultaneous” means the robot has to do both at once.\n\nOn the basis of SLAM, the robot can plan a trajectory through its environment. But no SLAM algorithm is perfect. How accurate does localization have to be to ensure workable trajectory planning?\n\nIn a ++[paper](https://www.amazon.science/publications/overlap-displacement-error-are-your-slam-poses-map-consistent)++ we’re presenting at this year’s International Conference on Intelligent Robots and Systems (++[IROS](https://www.amazon.science/conferences-and-events/iros-2021)++), we argue that existing metrics for measuring SLAM error aren’t well suited to the problem of navigation. In particular, minimizing global error can leave open the possibility of local errors that make trajectory planning extremely difficult.\n\n#### **More coverage of devices and services announcements**\n\n- \"++[Unlocking AI for everyone](https://www.aboutamazon.com/news/devices/unlocking-ai-for-everyone)++\"\n- \"++[Astro’s Intelligent Motion brings state-of-the-art navigation to the home](https://www.amazon.science/blog/astros-intelligent-motion-brings-state-of-the-art-navigation-to-the-home)++\"\n- \"++[How Amazon is using self-service to democratize AI](https://www.amazon.science/blog/how-amazon-is-using-self-service-to-democratize-ai)++\"\n- \"++[The science behind visual ID](https://www.amazon.science/blog/the-science-behind-visual-id)++\"\n\nInstead, we propose a new metric, which we call the overlap displacement error (ODE). ODE may sometimes lead to solutions with a slightly larger global metric error, but it enforces greater local consistency, which we believe will enable robust trajectory planning.\n\n#### **Think locally, act globally**\n\nBelow is an example showing why conventional SLAM metrics are not ideal for purposes of navigation. The figure shows two different maps of a large building with two wings, produced by two different SLAM algorithms applied to the same sensor data. \n\nOn the left, we show a SLAM algorithm that is globally very accurate; the error in its estimate of the distance from the edge of the left wing to the edge of the right is only 1 centimeter. However, in the left top corner it gets confused. The absolute trajectory error (ATE) is still very low, but the algorithm produces two different estimates of the location of a doorway (red and blue) that leave the robot believing that the opening is impassably narrow. The robot will thus have trouble planning a path through the opening if it wants to move from one wing to another.\n\nOn the right, we see the results of another SLAM algorithm. The robot’s estimate of its trajectory obviously drifts, and the algorithm has a huge absolute mapping error; the error from left to right wing is 100 centimeters. However, the SLAM algorithm appears to be locally consistent with itself, which means that although the map is skewed, it is still completely navigable.\n\n\n\nTwo maps of a large building with two wings, produced by applying two different SLAM algorithms to the same sensor data from a robot traversing the building. In blue and red, we show the algorithms’ estimates of wall locations, and in black, we show their estimates of the robot’s trajectory.\n\nThe point to make here is that absolute error is not the most relevant measure for autonomous navigation. What is important is that whenever the robot passes through the same area, it remains consistent with its own knowledge. If it is consistent, then it won't delete walls (or create false openings), and it won't build up false walls (which block off corridors). Self-consistency is precisely the aim of the metric we proposed.\n\nConsider a physical obstacle (the blue circle in the figure below) seen by a robot at two different times, T0 (mustard view) and T1 (green view). A SLAM algorithm maps this obstacle twice (mustard circle and green circle), but due to the localization inaccuracy, the two mapped circles do not overlap with each other as they should. ODE measures the displacement error (red arrow) from the mustard circle to the green circle. \n\n\n\nThis figure illustrates how displacement error can be calculated for each obstacle viewed twice by the same robot.\n\nThis displacement error can be calculated for every pair of camera views of the same sensing area. If we superimpose a grid on the SLAM map, the error can be aggregated by placing a virtual obstacle on each cell of the grid. \n\n\n\nGiven the ground truth for a robot’s trajectory (left) and the trajectory estimated by a SLAM algorithm (center), ODE can measure displacement for virtual objects (black dots) placed on each cell of grid overlaying the SLAM map.\n\nIn the figure below, we compare ODE to three conventional metrics: absolute trajectory error (ATE) and two types of relative trajectory error (RTE), linear (m) and angular (deg). On the left, we show a ground truth trajectory (green line), a SLAM algorithm’s estimated trajectory (white line), and a heat map that indicates the error that inaccurate localization would introduce into the map, if we used a 360° range sensor with a two-meter range. The numbers along the estimated trajectory indicate time steps.\n\nOn the right, we show the plots for different localization error metrics. Note that ODE focuses on map consistency, with clear peaks at the discontinuity in the trajectory estimate (100th step) and the region where the trajectory imperfectly intersects itself (40th and 220th steps). With the other metrics, however, local inconsistencies are harder to deduce.\n\n\n\nODE measures map consistency of localization, which conventional metrics do not provide.\n\nABOUT THE AUTHOR\n\n#### **[Jianbo Ye](https://www.amazon.science/author/jianbo-ye)**\n\nJianbo Ye is a senior applied scientist at Amazon.\n\n#### **[Christian Mostegel](https://www.amazon.science/author/christian-mostegel)**\n\nChristian Mostegel is a senior applied scientist at Amazon.\n\n\n\n\n\n","render":"<p>Simultaneous localization and mapping (SLAM) is one of the core capabilities of autonomous robots, with a rich literature behind it. “Mapping” means building a map of the robot’s environment, “localization” means identifying the robot’s location on the map, and “simultaneous” means the robot has to do both at once.</p>\n<p>On the basis of SLAM, the robot can plan a trajectory through its environment. But no SLAM algorithm is perfect. How accurate does localization have to be to ensure workable trajectory planning?</p>\n<p>In a <ins><a href=\\"https://www.amazon.science/publications/overlap-displacement-error-are-your-slam-poses-map-consistent\\" target=\\"_blank\\">paper</a></ins> we’re presenting at this year’s International Conference on Intelligent Robots and Systems (<ins><a href=\\"https://www.amazon.science/conferences-and-events/iros-2021\\" target=\\"_blank\\">IROS</a></ins>), we argue that existing metrics for measuring SLAM error aren’t well suited to the problem of navigation. In particular, minimizing global error can leave open the possibility of local errors that make trajectory planning extremely difficult.</p>\n<h4><a id=\\"More_coverage_of_devices_and_services_announcements_6\\"></a><strong>More coverage of devices and services announcements</strong></h4>\\n<ul>\\n<li>“<ins><a href=\\"https://www.aboutamazon.com/news/devices/unlocking-ai-for-everyone\\" target=\\"_blank\\">Unlocking AI for everyone</a></ins>”</li>\n<li>“<ins><a href=\\"https://www.amazon.science/blog/astros-intelligent-motion-brings-state-of-the-art-navigation-to-the-home\\" target=\\"_blank\\">Astro’s Intelligent Motion brings state-of-the-art navigation to the home</a></ins>”</li>\n<li>“<ins><a href=\\"https://www.amazon.science/blog/how-amazon-is-using-self-service-to-democratize-ai\\" target=\\"_blank\\">How Amazon is using self-service to democratize AI</a></ins>”</li>\n<li>“<ins><a href=\\"https://www.amazon.science/blog/the-science-behind-visual-id\\" target=\\"_blank\\">The science behind visual ID</a></ins>”</li>\n</ul>\\n<p>Instead, we propose a new metric, which we call the overlap displacement error (ODE). ODE may sometimes lead to solutions with a slightly larger global metric error, but it enforces greater local consistency, which we believe will enable robust trajectory planning.</p>\n<h4><a id=\\"Think_locally_act_globally_15\\"></a><strong>Think locally, act globally</strong></h4>\\n<p>Below is an example showing why conventional SLAM metrics are not ideal for purposes of navigation. The figure shows two different maps of a large building with two wings, produced by two different SLAM algorithms applied to the same sensor data.</p>\n<p>On the left, we show a SLAM algorithm that is globally very accurate; the error in its estimate of the distance from the edge of the left wing to the edge of the right is only 1 centimeter. However, in the left top corner it gets confused. The absolute trajectory error (ATE) is still very low, but the algorithm produces two different estimates of the location of a doorway (red and blue) that leave the robot believing that the opening is impassably narrow. The robot will thus have trouble planning a path through the opening if it wants to move from one wing to another.</p>\n<p>On the right, we see the results of another SLAM algorithm. The robot’s estimate of its trajectory obviously drifts, and the algorithm has a huge absolute mapping error; the error from left to right wing is 100 centimeters. However, the SLAM algorithm appears to be locally consistent with itself, which means that although the map is skewed, it is still completely navigable.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/137ca6954aa040a0916f45211e7f12f4_image.png\\" alt=\\"image.png\\" /></p>\n<p>Two maps of a large building with two wings, produced by applying two different SLAM algorithms to the same sensor data from a robot traversing the building. In blue and red, we show the algorithms’ estimates of wall locations, and in black, we show their estimates of the robot’s trajectory.</p>\n<p>The point to make here is that absolute error is not the most relevant measure for autonomous navigation. What is important is that whenever the robot passes through the same area, it remains consistent with its own knowledge. If it is consistent, then it won’t delete walls (or create false openings), and it won’t build up false walls (which block off corridors). Self-consistency is precisely the aim of the metric we proposed.</p>\n<p>Consider a physical obstacle (the blue circle in the figure below) seen by a robot at two different times, T0 (mustard view) and T1 (green view). A SLAM algorithm maps this obstacle twice (mustard circle and green circle), but due to the localization inaccuracy, the two mapped circles do not overlap with each other as they should. ODE measures the displacement error (red arrow) from the mustard circle to the green circle.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/b973c82557014152843cfddf001b431d_image.png\\" alt=\\"image.png\\" /></p>\n<p>This figure illustrates how displacement error can be calculated for each obstacle viewed twice by the same robot.</p>\n<p>This displacement error can be calculated for every pair of camera views of the same sensing area. If we superimpose a grid on the SLAM map, the error can be aggregated by placing a virtual obstacle on each cell of the grid.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/721c2981363e44acb6cb7d79ca3b3372_image.png\\" alt=\\"image.png\\" /></p>\n<p>Given the ground truth for a robot’s trajectory (left) and the trajectory estimated by a SLAM algorithm (center), ODE can measure displacement for virtual objects (black dots) placed on each cell of grid overlaying the SLAM map.</p>\n<p>In the figure below, we compare ODE to three conventional metrics: absolute trajectory error (ATE) and two types of relative trajectory error (RTE), linear (m) and angular (deg). On the left, we show a ground truth trajectory (green line), a SLAM algorithm’s estimated trajectory (white line), and a heat map that indicates the error that inaccurate localization would introduce into the map, if we used a 360° range sensor with a two-meter range. The numbers along the estimated trajectory indicate time steps.</p>\n<p>On the right, we show the plots for different localization error metrics. Note that ODE focuses on map consistency, with clear peaks at the discontinuity in the trajectory estimate (100th step) and the region where the trajectory imperfectly intersects itself (40th and 220th steps). With the other metrics, however, local inconsistencies are harder to deduce.</p>\n<p><img src=\\"https://dev-media.amazoncloud.cn/802a8e6e821841c8a18837f1c9512244_image.png\\" alt=\\"image.png\\" /></p>\n<p>ODE measures map consistency of localization, which conventional metrics do not provide.</p>\n<p>ABOUT THE AUTHOR</p>\n<h4><a id=\\"Jianbo_Yehttpswwwamazonscienceauthorjianboye_51\\"></a><strong><a href=\\"https://www.amazon.science/author/jianbo-ye\\" target=\\"_blank\\">Jianbo Ye</a></strong></h4>\n<p>Jianbo Ye is a senior applied scientist at Amazon.</p>\n<h4><a id=\\"Christian_Mostegelhttpswwwamazonscienceauthorchristianmostegel_55\\"></a><strong><a href=\\"https://www.amazon.science/author/christian-mostegel\\" target=\\"_blank\\">Christian Mostegel</a></strong></h4>\n<p>Christian Mostegel is a senior applied scientist at Amazon.</p>\n"}