Workshop at ICRA 2011 (Shanghai, China, May 13, 2011)

Organizers

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Content

In all field of science experiments play an important role, in order to confirm/refute a theory and to discover new theories. It is a widespread opinion that experimental methodologies in robotics have not yet achieved a level of maturity comparable with that in traditional science. In this workshop, we will discuss fundamental issues about the role of experiments in robotics, such as how can results be replicable and refutable on the one hand, and quantitatively comparable according to community-endorsed metrics to enable a faster cumulative progress, or even appreciate disruptive changes. We will particularly focus on how might be possible, by providing the proper kind and amount of data to enable the replication of experiments as a prerequisite to quantitative comparison of capabilities. A key point to allow replication and comparison of results is having adequate data support: all the data necessary to repeat a given experiment, how to achieve it with today's digital media will be addressed. These issues, when viewed in the context of some general principles about experiments in science and engineering, will allow us to do some insightful considerations on the role of experiments in robotics and it scientific and epistemological foundations. This workshop is a joint initiative of the IEEE Technical Committee on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS) and the EURON Special Interest Group on Good Experimental Methodology for Robotics (SIG GEM).

A major output of the EURON SIG GEM is a set of guidelines for good experimental and reporting practice: http://www.heronrobots.com/eurongemsig/downloads/GemSigGuidelinesBeta.pdf An open and deep discussion of these issues, with regards to some general principles about experiments in science and engineering, will lead us to derive some insightful considerations on the role of experiments in robotics. We also aim to lay some foundations toward a replicable robotics research publishing thread based on the publication of fully replicable experiments

Workshop Highlights

List of topics

• Design of Experiments in Robotics
• Execution of Experiments in Robotics
• Reporting Experiments in Robotics
• Epistemological issues
• Examples of Good Practice
• Evaluation of Experimental Robotics Work
• Proposals for Promotion of Good Experimental Work
• Metrics for sensory motor coordination and visual servoing effectiveness/efficiency
• Benchmarking autonomy and robustness to changes in the environment/task
• Scalable autonomy measurements
• Requirements for robots in terms of performance, the approaches to meeting these requirements, the trade-offs in terms of performance
• Experimental scenarios to evaluate performance, demonstrate generality, and measure robustness
• Performance modeling of the relationship between a task and the environment where it is performed
• Relationship between benchmarking and replication of experiments with robots

Format:

The workshop will consist of presentations interleaved with a significant amount of additional time for discussions between the presentations and at the end of the full day single track sessions.

Proceedings:

The workshop contribution will be included in the ICRA 2010 Workshop/Tutorial CD; selected contributions may be invited to submit to a journal special issue on good experimental robotics.

Intended Audience:

Robotics researchers from any subfield of the discipline from both academia and industry.
Industry members interested in the exploitation of research results, others interested in methodologies in scientific and engineering disciplines.

Background and previous workshops:

This workshop is a joint initiative of the IEEE-TC on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS) and the EURON Special Interest Group on Good Experimental Methodology for Robotics (SIG GEM). The proposers are co-chairs of IEEE-TC PEBRAS and/or EURON SIG GEM and have jointly co-organized over 10 successful related events in the last 5 years, such as 5-in-a-row workshops on Benchmarking and Performance Metrics at IROS (2006-2010), three workshops at RSS (2008-2010) and the Performance Metrics for Intelligent Systems Workshop (PERMIS) series (20060-2010). A list of previously organized events can be found here: http://www.heronrobots.com/EuronGEMSig/GEMSIGEvents.html The latest workshop at RSS 2010 had a certified attendance of over 50 participants.

Unconference style interaction is encouraged:

This will be a 'working' :-) workshop interaction in person and by social networking media with people in the room and outside is welcome.

Program

09:00

Introduction

09:10

Wolfram Burgard Autonomous Lifelong Navigation: Experiment replication and Benchmarking
University of Freiburg
Germany

9:50

Weidong Chen Experimental method and benchmarking in Mobile Robot Networks
Shanghai Jiao Tong University
China

10:30 - 11:00

Coffee break

11:00

Libor Kral Performance evaluation and benchmarking in EU-funded activities
EU Commission, Challenge 2 Head of Unit
Luxembourg

11:40

Vincent C. Müller Benchmarking benchmarks: What should count as 'success' towards a benchmark?

EUCogII coordinator

Anatolia College/ACT, Thessaloniki

Greece

12:20 - 14:10

Lunch break

14:10

Jeff Trinkle On Grasping in Simulation and the 'Real World'

Renssellaer Politechnic Institute

USA

14:50

Sven Wachsmuth Performance Assessment and System Design in Human Robot Interaction

Citec, University of Bielefeld

Germany

15:30-16:00

Coffee break

16:00

Organizers' 'editorial': Time for action!

16:40

Panel and discussion

17:40

Wrap up

IEEE copyright disclaimer

Personal use of this material (the papers) is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.