Workshop at ICRA 2012 (St. Paul, Minnesota, USA, May 18, 2012)

Proposers:

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.  with the support of  This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

In all field of science experiments play an important role, in order to confirm/refute a theory and to discover new theories. It is a widespread opinion that experimental methodologies in robotics have not yet achieved a level of maturity comparable with that in traditional science. On the other hand being able of objectively measure and comparing performances is a critical aspect of modern engineering.

In this workshop, we will discuss fundamental issues about the role of experiments in robotics, such as how can results be replicable and refutable on the one hand, and quantitatively comparable according to community-endorsed metrics to enable a faster cumulative progress, or even appreciate disruptive changes, on the other end.

We will particularly focus on how might be possible, by providing the proper kind and amount of data to enable the replication of experiments as a prerequisite to quantitative comparison of capabilities.

A key point to allow replication and comparison of results is having adequate data support: all the data necessary to repeat a given experiment, how to achieve it with today's digital media will be addressed.

We will concentrate on three main subfields: visual servoing and grasping, slam and navigation.

In these fields it is already possible to outline several compelling criteria. These issues, when viewed in the context of some general principles about experiments in science and engineering, will allow us to do some insightful considerations on the role of experiments in robotics and its scientific and epistemological foundations.

This workshop is a joint initiative of the IEEE Technical Committee on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS) and the EURON Special Interest Group on Good Experimental Methodology for Robotics (SIG GEM).

PROGRAM

Friday, May, 18th (time CDT GMT+6)

8:45 Opening

8:55 The RCC Flagship: scientific and experimental challenges
P. Dario
Scuola Superiore S. Anna, Italy

9:40 Competitions: One pathway to replicable and comparable results
S.Balabirsky,
National Institute of Standards, USA

10:10 Coffee break

10:30 Practical and Theoretical Motivations for Replication of Experiments in Robotics: The complementary roles of benchmarks, challenges and competitions
F.Bonsignorio
Universidad Carlos III of Madrid, Spain and Heron Robots

11:00 Benchmarking Grasping
D. Kragic
Royal Technology Institute KTH, Sweden

11:30 Analyzing Performance of Visual Servoing Approaches
V. Kyrki
Lappeenranta University of Technology, Finland

12:00 Lunch

13:30 Characterization of Textile Grasping Experiments
G. Alenya, A. Ramisa, F. Moreno-Noguer, C. Torras
CSIC- Universidad Politecnica de Catalunia CSIC-UPC, Spain

14:00 An Analysis of Experimental Trends in Autonomous Robotics Papers
F. Amigoni, V. Schiaffonati, Politecnico di Milano, Italy
M.Verdicchio Universita' di Bergamo, Italy

14:30 Towards Stochastic Characterization of Robot Traction in Granular Terrain
J. Williams, J.Trinkle
Rensselaer Polytechnic Institute, USA

15:00 Coffee break

15:30 Energy in robots: measuring performance and efficiency
C. Stefanini
Scuola Superiore S. Anna, Italy

16:00 Discussion

17:00 Conclusions and future initiatives

Workshop Highlights

List of topics

• Design of Experiments in Robotics
• Execution of Experiments in Robotics
• Reporting Experiments in Robotics
• Epistemological issues
• Examples of Good Practice
• Evaluation of Experimental Robotics Work
• Proposals for Promotion of Good Experimental Work
• Metrics for sensory motor coordination and visual servoing effectiveness and efficiency
• Metrics for SLAM effectiveness and efficiency

• Metrics for grasping effectiveness and efficiency:

• Benchmarking autonomy and robustness to changes in the environment/task
• Scalable autonomy measurements
• Requirements for robots in terms of performance, the approaches to meeting these requirements, the trade-offs in terms of performance
• Experimental scenarios to evaluate performance, demonstrate generality, and measure robustness
• Performance modeling of the relationship between a task and the environment where it is performed
• Relationship between benchmarking and replication of experiments with robots

Motivation and objectives

In most cases the measurement robotic system performance, their evaluation, comparison, characterization etc. involve practical experimentation, which has to be carried out responsibly and reported well.

Recently, the interest in experimental methodologies increased dramatically within the robotics community, both from researchers, aiming at more grounded and fast research advancement, and from public funding agencies, according to the idea that good experimental activities could reduce the gap between research and industrial applications.

Some projects have been funded by the European Commission and a series of workshops (see below) have been held in the latest years. While the definition of proper benchmarking procedures has become a key requisite of European project funding application.

The aim of this workshop is to discuss fundamental issues about the role of experiments in robotics, such as how can results be replicable and refutable on the one hand, and quantitatively comparable according to community endorsed metrics.

A major output of the EURON SIG GEM is a set of guidelines for good experimental and reporting practice:

http://www.heronrobots.com/EuronGEMSig/Downloads/GemSigGuidelinesBeta.pdf

An open and deep discussion of these issues, with regards to some general principles about experiments in science and engineering, will lead us to derive some insightful considerations on practical issues and best practices, as well of theoretical natures on the role of experiments in robotics.

We also aim to lay some foundations toward a replicable robotics research publishing thread based on the publication of fully

replicable experiments.

* We believe it is now possible to draw some conclusions on a number of subfields to the benefit of the community. *

The best contributions will be invited to submit to a refereed edited book summarizing the activities carried on in the latest years in this perspective. 

Primary/secondary audience

Primary: Robotics researchers from any subfield of the discipline from both academia and industry.

Secondary: industry members interested in the exploitation of research results, others interested in methodologies in scientific and engineering disciplines.

Relation to previous ICRA or IROS workshops

This workshop is a joint initiative of the IEEE-TC on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS) and the EURON Special Interest Group on Good Experimental Methodology for Robotics (SIG GEM). The proposers are co-chairs of IEEE-TC PEBRAS and/or EURON SIG GEM and have jointly co-organized over 10 successful related events in the last 5 years, such as 5-in-a-row workshops on Benchmarking and Performance Metrics at IROS (2006-2010), three workshops at RSS (2008-2010) and the Performance Metrics for Intelligent Systems Workshop (PERMIS) series (2006-2010) and two organized at ICRA 2010 and 2011 on replication of experiment in robotics research. 

Support

This workshop is supported by IEEE RAS TC-Pebras and the Euron GEM Sig