Performance evaluation of autonomous mobile robots

Abstract

The National Institute of Standards and Technology (NIST) has initiated a program to develop quantitative metrics for machine intelligence. One of the possible approaches to evaluating machine intelligence is task‐based performance testing, like a mouse in a maze. A series of application‐specific testbeds is envisioned. NIST has created a set of reference test arenas for evaluating the performance of autonomous mobile robots performing urban search and rescue tasks. Robots must explore the maze‐like test course, map the environment, find the simulated victims, and then report back their findings. In this paper, we describe our approach toward developing performance metrics for autonomous mobile robots through standardized testing within representative environments and objective performance evaluations. Our intent is to help accelerate the robotic research communities’ advancement of mobile robot capabilities, thereby improving the effectiveness of robots performing within industrial settings, hazardous environments, and in exploration.