Workshop at RSS 2010 (Zaragoza, Spain, June 28, 2010)
Good experimental work is not novel in robotics: there are many good experimental scientists in our community. However, uniformly good experimental work and reporting has not yet been achieved.
The purpose of this workshop is to encourage groups with ideas for good experimental practice and reporting to discuss, evaluate and share their experimental designs and techniques, to the benefit of the whole community.
In particular we will investigate opportunities and issues of designing, performing and PUBLISHING fully replicable research.
As the complexity of current robotic and embodied intelligent systems grows, it is more and more necessary to define proper experimental approaches and benchmarking procedures. The Special interest group on Good Experimental Methodology was formed by EURON in response to a wide perception — evidenced by some 90 responses to an email call for interest — that roboticists could do much better at performing and, particularly, reporting experimental work. A major output of the Special Interest Group is a set of guidelines for good experimental (reporting) practice, which we believe supports the community’s wish to produce better experimental work. Equally significant is a series of workshops on Benchmarking and Good Experimental Methodology held at major Robotics conferences. Good experimental work is not novel in robotics: there are many good experimental scientists in our community. However, uniformly good experimental work and reporting has not yet been achieved. The evidence of community interest in the topic mentioned above suggests that the community is willing to take steps to improve in this area. Thus we propose to take the work of good experimental robotics groups and use it to illustrate high-quality experimental work and reporting, using the workshop format to draw out the strengths (and weaknesses) of presented work and encourage sharing and adoption of good practices.
We will above all analyse the issues (from practical to epistemological ones) involved by the suitable reporting process for fully replicable robotics research.
List of topics
- Design of Experiments in Robotics
- Execution of Experiments in Robotics
- Reporting Experiments in Robotics
- Examples of Good Practice
- Evaluation of Experimental Robotics Work
- Proposals for Promotion of Good Experimental Work
- Proposals for replicable robotics research reporting processes
The workshop will consist of presentations interleaved with a significant amount of additional time for discussions between the presentations and at the end of the full day single track sessions.
The workshop will generate online proceedings; selected contributions may be invited to submit to a journal special issue on good experimental robotics.
Previous workshops on the related topic of Benchmarking have been organized successfully at IROS (4 in a row) and RSS conferences, while successful ones on the same main topic were organized at RSS 2009 and ICRA 2010.
Unconference style interaction is encouraged:
This will be a 'working' :-) workshop interaction in person and by social networking media with people in the room and outside is welcome
09:00 - 09:10
Welcome and Introduction
09:10 - 9:40
FP7 Challenge 2 : Benchmarking and Standardization in FP7 Challenge 2 - Cognitive Systems and Robotics
Unit E5 - Cognitive Systems, Interaction and Robotics
EU Commission DG INFSO
9:40 - 10:10
Grasping in GRASP: Benchmarking and evaluation
KTH Royal Institute of Technology
Comparison of grasping forces in multi-fingered hands
University of California Los Angeles
10:30 - 11:00
University of Pennsylvania
11:00 - 11:30
11:30 - 12:00
On Comparing SLAM
University of Freiburg
12:00 - 12:20
Information metrics for assistive human-in-the-loop cognitive systems
M Stolen A Jardon C Balaguer
Universidad Carlos III de Madrid
F Bonsignorio Heron Robots and Universidad Carlos III de Madrid
12:20 - 12:40
Benchmark Tools for Evaluating AGVs at Industrial Environments
H Yuste, L Armesto J Tornero
Universidad Politecnica de Valencia
Open discussion: Replicable and comparable robotics research, how to perform and to report it?