Workshop at IEEE/RSJ IROS 2009 (St.Louis, MO, USA, 15 October, 2009).
High-level cognitive competencies encompassing knowledge representation, perception, control, and learning are considered essential elements that will allow robots to perform complex tasks within highly uncertain environments with an appropriate level of autonomy in various domains (service, manufacturing, and search and rescue). As the complexity and variety of required tasks and of the targeted environments increase exploring new application capabilities and domains, it becomes necessary to develop well principled procedures that allow quantitative comparison of the solutions provided by robotics research thereby facilitating exchanges of methods and solutions between different research groups and assessment of the state of the art.
New more successful implementations of concepts already presented in literature, but not implemented with exhaustive experimental methodology, run the risk of being ignored, if appropriate benchmarking procedures allowing to compare the actual practical results with reference to standard accepted procedures, are not in place.
The emphasis of the workshop will be on principles, methods, and applications expanding beyond the current limit of robotics applications in terms of cognitive capabilities and autonomy. Another key issue will be a capability-led understanding of cognitive robots: how to define shared ontologies or dictionaries to discuss robotic cognitive systems in terms of their performance, relationships between different cognitive robotics capabilities, requirements, theories, architectures, models and methods that can be applied across multiple engineering and application domains, detailing and understanding better the requirements for robots in terms of performance, the approaches to meeting these requirements and the trade-offs in terms of performance.
The proper definition of benchmarking is related to the problem of measuring capabilities of robots in a context in which, in many cases, the 'robotics experiments' themselves are difficult to 'replicate'.
List of topics
- Knowledge representation, perception (sensing), and learning
- Uncertainty management in robot navigation, path-planning and control
- Cognitive manipulation
- Metrics for sensory motor coordination and visual servoing effectiveness/efficiency
- Benchmarking autonomy and robustness to changes in the environment/task
- Scalable autonomy measurements
- Shared ontologies to discuss robotic cognitive systems in terms of their performance capabilities
- Relationships between different cognitive robotics capabilities
- Requirements, theories, architectures, models and methods that can be applied across multiple engineering and application domains
- Detailing and understanding better the requirements for robots in terms of performance, the approaches to meeting these requirements, the trade-offs in terms of performance
- The development of experimental scenarios to evaluate performance, demonstrate generality, and measure robustness
- Performance modeling of the relationship between a task and the environment where it is performed
- Relationship between benchmarking and replication of experiments with robots
- Robotics experiment
The workshop will consist of invited presentations and regular presentations. INTERACTION IS HIGHLY ENCOURAGED.
The papers are included in the IROS 2009 Tutorials and Workshops DVD.
You can register for the workshop via the IROS'09 website at http://www.iros2009.org/.
Previous workshops on the topic of Benchmarking have been organized successfully at IROS (3 in a row) and RSS conferences.
This workshop is under the auspices of the IEEE-TC on Performance Evaluation and Benchmarking of Robotic and Automation Systems (TC-PEBRAS) and of the EURON Good Experimental Methodology and Benchmarking SIG.
F Bonsignorio Raj Madhavan
(keynote) Publishing Identifiable Experiment Code And Configuration Is Important, Good and Easy
School of Computing Science, Simon Fraser University
Distributed Access to a Centralized Testing and Benchmarking Facility via the DESIRE Webportal
A. Bubeck U. Reiser K. Pfeiffer M. Hägele
OlfMark: A benchmark for mobile robot odour source localisation
Institute of Systems and Robotics, University of Coimbra
Redefining Fan out Metric: Estimating Human Capability in Controlling Multiple Robots
R. E. Mohan C. A. A. Acosta C. Zhou
W. Sardha Wijesoma
Nanyang Technological University
The Articulated Head: An Intelligent Interactive Agent as an Artistic Installation
C. Kroos D. C. Herath Stelarc
Thinking Head Project MARCS Auditory Laboratories, University of Western Sydney
(keynote) Robotic Competitions as Catalysts for Intelligent System Advancements
Reverse KLD-Sampling for Measuring Uncertainty in Rao-Blackwellized Particle Filters SLAM
L. Carlone M. Kaouk Ng
CSPP Laboratorio di Meccatronica, Politecnico di Torino
J. Du B. Bona M. Indri
Dipartimento di Automatica e Informatica, Politecnico di Torino
Ground Truth Free Evaluation of Segment Based Maps
Temple University, Philadelphia
Refutable Robotics Research: the Road Ahead
Fabio Bonsignorio John Hallam Angel P. del Pobil