Workshop at RSS 2010 (Zaragoza, Spain, June 28, 2010)

Organizers

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Content

Good experimental work is not novel in robotics: there are many good experimental scientists in our community. However, uniformly good experimental work and reporting has not yet been achieved.

The purpose of this workshop is to encourage groups with ideas for good experimental practice and reporting to discuss, evaluate and share their experimental designs and techniques, to the benefit of the whole community.
In particular we will investigate opportunities and issues of designing, performing and PUBLISHING fully replicable research.

As the complexity of current robotic and embodied intelligent systems grows, it is more and more necessary to define proper experimental approaches and benchmarking procedures. The Special interest group on Good Experimental Methodology was formed by EURON in response to a wide perception — evidenced by some 90 responses to an email call for interest — that roboticists could do much better at performing and, particularly, reporting experimental work. A major output of the Special Interest Group is a set of guidelines for good experimental (reporting) practice, which we believe supports the community’s wish to produce better experimental work. Equally significant is a series of workshops on Benchmarking and Good Experimental Methodology held at major Robotics conferences. Good experimental work is not novel in robotics: there are many good experimental scientists in our community. However, uniformly good experimental work and reporting has not yet been achieved. The evidence of community interest in the topic mentioned above suggests that the community is willing to take steps to improve in this area. Thus we propose to take the work of good experimental robotics groups and use it to illustrate high-quality experimental work and reporting, using the workshop format to draw out the strengths (and weaknesses) of presented work and encourage sharing and adoption of good practices.

We will above all analyse the issues (from practical to epistemological ones) involved by the suitable reporting process for fully replicable robotics research.

Workshop Highlights

List of topics

• Design of Experiments in Robotics
• Execution of Experiments in Robotics
• Reporting Experiments in Robotics
• Examples of Good Practice
• Evaluation of Experimental Robotics Work
• Proposals for Promotion of Good Experimental Work
• Proposals for replicable robotics research reporting processes

Format:

The workshop will consist of presentations interleaved with a significant amount of additional time for discussions between the presentations and at the end of the full day single track sessions.

Proceedings:

The workshop will generate online proceedings; selected contributions may be invited to submit to a journal special issue on good experimental robotics.

Previous workshops:

Previous workshops on the related topic of Benchmarking have been organized successfully at IROS (4 in a row) and RSS conferences, while successful ones on the same main topic were organized at RSS 2009 and ICRA 2010.

Unconference style interaction is encouraged:

This will be a 'working' :-) workshop interaction in person and by social networking media with people in the room and outside is welcome

Program

09:00 - 09:10

Welcome and Introduction

09:10 - 9:40

FP7 Challenge 2 : Benchmarking and Standardization in FP7 Challenge 2 - Cognitive Systems and Robotics
A Bajart
Unit E5 - Cognitive Systems, Interaction and Robotics
EU Commission  DG INFSO
Luxembourg

9:40 - 10:10

Grasping in GRASP: Benchmarking and evaluation
D Kragic
KTH Royal Institute of Technology
Sweden

10:10 -10:30

Comparison of grasping forces in multi-fingered hands
H Borgstrom
University of California Los Angeles
USA

10:30 - 11:00

On grasping
V Kumar
University of Pennsylvania
USA

11:00 - 11:30

Coffee Break

11:30 - 12:00

On Comparing SLAM
W Burgard
University of Freiburg
Germany

12:00 - 12:20

Information metrics for assistive human-in-the-loop cognitive systems
M Stolen A Jardon C Balaguer
Universidad Carlos III de Madrid
Spain
F Bonsignorio Heron Robots and Universidad Carlos III de Madrid
Italy

12:20 - 12:40

Benchmark Tools for Evaluating AGVs at Industrial Environments
H Yuste, L Armesto J Tornero
Universidad Politecnica de Valencia
Spain

12:40

Open discussion: Replicable and comparable robotics research, how to perform and to report it?