Workshop at IEEE/RSJ IROS 2009 (St.Louis, MO, USA, 15 October, 2009).

Organizers

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.    This e-mail address is being protected from spambots. You need JavaScript enabled to view it.    This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Motivation

High-level cognitive competencies encompassing knowledge representation, perception, control, and learning are considered essential elements that will allow robots to perform complex tasks within highly uncertain environments with an appropriate level of autonomy in various domains (service, manufacturing, and search and rescue). As the complexity and variety of required tasks and of the targeted environments increase exploring new application capabilities and domains, it becomes necessary to develop well principled procedures that allow quantitative comparison of the solutions provided by robotics research thereby facilitating exchanges of methods and solutions between different research groups and assessment of the state of the art.

New more successful implementations of concepts already presented in literature, but not implemented with exhaustive experimental methodology, run the risk of being ignored, if appropriate benchmarking procedures allowing to compare the actual practical results with reference to standard accepted procedures, are not in place.

The emphasis of the workshop will be on principles, methods, and applications expanding beyond the current limit of robotics applications in terms of cognitive capabilities and autonomy. Another key issue will be a capability-led understanding of cognitive robots: how to define shared ontologies or dictionaries to discuss robotic cognitive systems in terms of their performance, relationships between different cognitive robotics capabilities, requirements, theories, architectures, models and methods that can be applied across multiple engineering and application domains, detailing and understanding better the requirements for robots in terms of performance, the approaches to meeting these requirements and the trade-offs in terms of performance.

The proper definition of benchmarking is related to the problem of measuring capabilities of robots in a context in which, in many cases, the 'robotics experiments' themselves are difficult to 'replicate'.

Workshop Highlights:

List of topics

• Knowledge representation, perception (sensing), and learning
• Uncertainty management in robot navigation, path-planning and control
• Cognitive manipulation
• Metrics for sensory motor coordination and visual servoing effectiveness/efficiency
• Benchmarking autonomy and robustness to changes in the environment/task
• Scalable autonomy measurements
• Shared ontologies to discuss robotic cognitive systems in terms of their performance capabilities
• Relationships between different cognitive robotics capabilities
• Requirements, theories, architectures, models and methods that can be applied across multiple engineering and application domains
• Detailing and understanding better the requirements for robots in terms of performance, the approaches to meeting these requirements, the trade-offs in terms of performance
• The development of experimental scenarios to evaluate performance, demonstrate generality, and measure robustness
• Performance modeling of the relationship between a task and the environment where it is performed
• Relationship between benchmarking and replication of experiments with robots
• Robotics experiment

Format:

The workshop will consist of invited presentations and regular presentations. INTERACTION IS HIGHLY ENCOURAGED.

Proceedings:

The papers are included in the IROS 2009 Tutorials and Workshops DVD.

You can register for the workshop via the IROS'09 website at http://www.iros2009.org/.

Previous workshops:

Previous workshops on the topic of Benchmarking have been organized successfully at IROS (3 in a row) and RSS conferences.

Backing:

This workshop is under the auspices of the IEEE-TC on Performance Evaluation and Benchmarking of Robotic and Automation Systems (TC-PEBRAS) and of the EURON Good Experimental Methodology and Benchmarking SIG.

Program

09:00

Introduction
F Bonsignorio Raj Madhavan

09:15

(keynote) Publishing Identifiable Experiment Code And Configuration Is Important, Good and Easy
R.T. Vaughan
School of Computing Science, Simon Fraser University

10:00

Distributed Access to a Centralized Testing and Benchmarking Facility via the DESIRE Webportal
A. Bubeck U. Reiser K. Pfeiffer M. Hägele
Fraunhofer IPA

10:30

Coffee Break

11:00

OlfMark: A benchmark for mobile robot odour source localisation
L. Marques
Institute of Systems and Robotics, University of Coimbra

11:30

Redefining Fan out Metric: Estimating Human Capability in Controlling Multiple Robots
R. E. Mohan C. A. A. Acosta C. Zhou
Singapore Polytechnic
W. Sardha Wijesoma
Nanyang Technological University

12:00

The Articulated Head: An Intelligent Interactive Agent as an Artistic Installation
C. Kroos D. C. Herath Stelarc
Thinking Head Project MARCS Auditory Laboratories, University of Western Sydney

12:30-13:00

Open Discussion

13:00

Lunch

14:00

Morning Recap
F.Bonsignorio R.Madhavan

14:15

(keynote) Robotic Competitions as Catalysts for Intelligent System Advancements
E.Messina
NIST

15:00

Reverse KLD-Sampling for Measuring Uncertainty in Rao-Blackwellized Particle Filters SLAM
L. Carlone M. Kaouk Ng
CSPP Laboratorio di Meccatronica, Politecnico di Torino
J. Du B. Bona M. Indri
Dipartimento di Automatica e Informatica, Politecnico di Torino

15:30

Coffee Break

16:00

Ground Truth Free Evaluation of Segment Based Maps
R. Lakaemper
Temple University, Philadelphia

16:30

Refutable Robotics Research: the Road Ahead
Fabio Bonsignorio John Hallam Angel P. del Pobil

17:00

Open discussion