Workshop at ICRA 2010 (Anchorage, Alaska, USA, May 3, 2010)
Organizers
This e-mail address is being protected from spambots. You need JavaScript enabled to view it. This e-mail address is being protected from spambots. You need JavaScript enabled to view it. This e-mail address is being protected from spambots. You need JavaScript enabled to view it. This e-mail address is being protected from spambots. You need JavaScript enabled to view it.
Content
Experiments are essential ingredients of science, both to confirm/refute a theory and to discover new theories.
It is a common belief that experimentation in robotics has not yet reached a level of maturity comparable with that reached in traditional science. In this workshop, we will discuss fundamental issues about the role of experiments in robotics, such as how can results be replicable and refutable on the one hand, and quantitatively comparable according to community-endorsed metrics to enable a faster cumulative progress, or even appreciate disruptive changes.
These issues, when viewed in the context of some general principles about experiments in science and engineering, will lead us to derive some insightful considerations on the role of experiments in robotics. A key point to allow replication and comparison of results is having adequate data support: all the data necessary to repeat a given experiment, how to achieve it with today's digital media will be addressed.
This workshop is a joint initiative of the new IEEE-TC on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS) and the EURON Special Interest Group on Good Experimental Methodology for Robotics (SIG GEM).
Most forms of robotic system performance measurement, evaluation, comparison, characterization etc. involve practical experimentation, which must be carried out responsibly and reported well. Recently, the interest in experimental methodologies increased dramatically within the robotics community, both from researchers and from funding agencies, according to the idea that good experimental activities could reduce the gap between research and industrial applications.
Some projects have been funded by the European Commission and a series of workshops (see below) have been held in the latest years. Despite the ongoing efforts and the recognized importance of experiments for rigorously evaluating new approaches and for reporting them in an objective and complete manner, these ideas have not yet become really part of current practice, as it can be seen by having a look at recently published papers. This may be due to the diffculty to perform time-consuming experiments and to the weak awareness of experiments as fundamental elements in the development of a robotic system.
The aim of this workshop is to discuss fundamental issues about the role of experiments in robotics, and namely as already noticed above, issues such as how can results be replicable and refutable on the one hand, and quantitatively comparable according to community-endorsed metrics, on the other hand. As both replication and benchmarking are needed to foster a cumulative advancement of robotics, and even to correctly appreciate disruptive innovation in the science and technology of robots. The recent evidence of community interest suggests that it is willing to take steps to improve in this area.
A major output of the Euron SIG GEM is a set of guidelines for good experimental and reporting practice: GemSigGuidelinesBeta.pdf
A deep discussion of these issues, when viewed in the context of some general principles about experiments in science and engineering, will lead us to derive some insightful considerations on the role of experiments in robotics.
Workshop Highlights:
List of topics
- Design of Experiments in Robotics
- Execution of Experiments in Robotics
- Reporting Experiments in Robotics
- Epistemological issues
- Examples of Good Practice
- Evaluation of Experimental Robotics Work
- Proposals for Promotion of Good Experimental Work
- Metrics for sensory motor coordination and visual servoing effectiveness/efficiency
- Benchmarking autonomy and robustness to changes in the environment/task
- Scalable autonomy measurements
- Requirements for robots in terms of performance, the approaches to meeting these requirements, the trade-offs in terms of performance
- Experimental scenarios to evaluate performance, demonstrate generality, and measure robustness
- Performance modeling of the relationship between a task and the environment where it is performed
- Relationship between benchmarking and replication of experiments with robots
Format:
The workshop will consist of presentations interleaved with a significant amount of additional time for discussions between the presentations and at the end of the full day single track sessions.
Proceedings:
The workshop contribution will be included in the ICRA 2010 Workshop/Tutorial CD; selected contributions may be invited to submit to a journal special issue on good experimental robotics.
Intended Audience:
Robotics researchers from any subfield of the discipline from both academia and industry.
Industry members interested in the exploitation of research results, others interested in methodologies in scientific and engineering disciplines.
Background and previous workshops:
This workshop is a joint initiative of the new IEEE-TC on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS) and the EURON Special Interest Group on Good Experimental Methodology for Robotics (SIG GEM). The proposers are co-chairs of IEEE-TC PEBRAS and/or EURON SIG GEM and have jointly co-organized over 10 successful related events in the last 5 years, such as 4-in-a-row workshops on Benchmarking and Performance metrics at IROS (2006-2009), two workshops at RSS (2008-2009) and the Performance Metrics for Intelligent Systems Workshop (PERMIS) series.
Unconference style interaction is encouraged
This will be a 'working' :-) workshop interaction in person and by social networking media with people in the room and outside is welcome
Program
08:45 - 09:00
Introduction
09:00 - 9:30
Simulations Used as Experiments in Autonomous Mobile Robotics
F Amigoni V Schiaffonati Politecnico di Milano
M Reggiani Universita' di Padova
9:30 - 10:00
Robotic Wheelchairs: Scientific Experimentation or Social Intervention?
T Carlson Y Demiris Imperial College
10:00 - 10:30
Benchmarking Usability of Assistive Robotic Systems: Methodology and Application
J G Victores A Jardón, M F Stoelen C Balaguer University Carlos III de Madrid
F Bonsignorio HeronRobots and University Carlos III de Madrid
10:30 - 11:00
Coffee Break
11:00 - 11:30
The Importance of Accuracy Measurement Standards for Computer-Integrated Interventional Systems
T Haidegger B Benyó Z Benyó Budapest University of Technology and Economics, Dept. of Control Engineering and Information Technology (BME - IIT) – Biomedical Engineering Laboratory, Budapest, Hungary
P. Kazanzides Center for Computer Integrated Surgical Systems and Technology, (CISST ERC),The Johns Hopkins University
I. Rudas Óbuda University
11:30 - 12:00
The Effect of Multiple Robots on Simulation Accuracy
S Dawson, B Lowe Wellman M Anderson Department of Computer Science, University of Alabama
12:00 - 12:30
Testing and Validation of Large Scale Collaborative Robotics Projects Lessons from the GUARDIANS experience
L Marques Dept. of Electrical and Computer Engineering, Institute of Systems and Robotics, University of Coimbra
V Gazi Dept. of Electrical and Electronics Engineering, TOBB University of Economics and Technology
J Gancet U Witkowski, Heinz Nixdorf Institute, University of Paderborn
R Marin, Robotic Intelligence Lab, Jaume-I University,
J Penders Materials and Engineering Research Institute, Sheffield Hallam University
12:30 - 14:00
Lunch
Experimental methodology for benchmarking ICP-like algorithms
L Armesto Universidad Politecnica de Valencia
L Montesano J Mınguez Computer Science Department, Universidad de Zaragoza
14:30 - 15:00
Using Hardware-in-the-Loop Simulation for Parametric Analysis of Robotic Systems
J P Gonzalez, Autonomous Perception Research, General Dynamics Robotic Systems,
W Dodson R Dean General Dynamics Robotic Systems
A Lacaze L Sapronov Robotic Research LLC
M Childers Robotics Program Office, U.S. Army Research Laboratory
15:00 - 15:30
The Teleworkbench – A Platform for Performing and Comparing Experiments in Robot Navigation
F Werner Ulrich Rueckert Cognitive Interaction Technology Centre of Excellence (CITEC), Bielefeld University,
A Tanoto J Welzel Heinz-Nixdorf Institute, Paderborn University
15:30 - 16:00
Coffee Break
The Role of Experiments in Robotics Research
F P Bonsignorio A P del Pobil J Hallam R Madhavan
16:30 - 17:30
Open discussion
17:30 - 17:45
Wrap-up
IEEE copyright disclaimer
Personal use of this material (the papers) is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
