Evaluating success in autonomous multi-robot teams: experiences from ALLIANCE architecture implementations

Abstract

A current limitation in multi-robot systems research is the lack of metrics that enable the performance comparison of various approaches to cooperative control. Ideally, metrics would be available that provide quantitative evaluations of cooperative robot teams based upon high-level design goals, such as fault tolerance, adaptivity, and so forth. In current research, however, metrics are limited to application-dependent measures. This paper summarizes the metrics that have been used in eight implementations of the ALLIANCE architecture in multi-robot teams. We explore the implications of the chosen metrics and offer suggestions for developing less application-dependent metrics in the future.