Workshop at IROS 2015
(Hamburg, Germany, September 28, 2015)
In Robotics Research the replicability of results and their objective evaluation and comparison is very difficult to put into practice. Furthermore, there is no solid theoretical foundation for experimental replicability in robotics.
This situation impairs at the same time research progress and technology transfer.
Significant progress has been made in this respect in recent years. The current RAM issue is a special issue composed of replicable experiments and guest edited by some of the proposers will be published at the end of 2015, while in September the 2015 IEEE-RAS Summer School on Experimental Methodology, Performance Evaluation and Benchmarking in Robotics, has been organized in Benicassim, Castellon, Spain dedicated to same topics and organized by the proposers and John Hallam.
This workshop aims to gather researchers active in the academia and the industry to share the ideas so far developed and discuss the challenges still ahead.
The best contributions will be invited to submit to a refereed edited book or special issue on an high impact robotics journal.
8:40 Do we really need to replicate experiments?
Matteo Matteucci, Francesco Amigoni, Viola Schiafonati, Andrea Bonarini
Politecnico di Milano, Italy
9:10 Intelligence testing and robotics
Vincent C. Mueller
Amerikaniko Kollegio Anatolia, Greece
9:40 The RoboCup Logistics League as a Holistic Multi-Robot Smart Factory Benchmark
Tim Niemueller1, Alexander Ferrein2, Sebastian Reuter3, Sabina Jeschke3, Gerhard Lakemeyer1
1Knowledge-Based Systems Group, RWTH Aachen University, Aachen,Germany
2Mobile Autonomous Systems & Cognitive Robotics Institute, FH Aachen University of Appl. Sc., Aachen, Germany
3Cluster IMA/ZLW & IfU, RWTH Aachen University, Aachen, Germany
10:00 Coffee Break
10:10 Benchmarking Service Robots
Luca Iocchi, Daniele Nardi
Sapienza University of Rome, Italy
10:40 Standards for service robotics
Univ. of Gävle & KTH, Sweden
11:10 Benchmarking of bipedal functions in humanoids and wearable robots
Spanish National Research Council (CSIC), Spain
11:40 NAO race: a simulation benchmark for walking humanoids
Cyberbotics Ltd., Switzerland
12:10 Dataset Suite for Benchmarking Perception in Robotics
Aamir Ahmad1,2 , Pedro Lima
1Max Planck Institute for Biological Cybernetics, Tubingen, Germany
2Institute for Systems and Robotics, Instituto Superior Tecnico, Portugal
12:30 - 14:00 Lunch Break
14:00 Functional and Non-Functional Expressive Dimensions: Classiﬁcation of
the expressiveness of the face of humanoid robots
Francois Ferland, Adriana Tapus
ENSTA-ParisTech, Paris, France
14:30 The challenges in benchmarking robot's social intelligence and some ideas: an industrial perspective
Amit Kumar Pandey
Aldebaran Robotics, France
15:00 Probabilistic approaches to robot task benchmarking
Institute for Systems and Robotics Instituto Superior Técnico, Portugal
15:30 Coffee Break
15:45 The Journal of Computational Research Replication (JCRR)
Benjamin Kuipers, Michael Wellman
Division of Computer Science & Engineering, University of Michigan, Ann Arbor, USA
16:15 Towards Reproducible Robotics Research
The BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa and Heron Robots, Italy
16:45 Panel and Plenary Discussion on RRR (Reproducible Robotics Research)
17:45 Conclusions and Wrap up
Robotics is a broad science and though we will try to cover different aspects of the discipline, the emphasis of the workshop will be on principles, methods, and applications in terms of cognitive capabilities and autonomy. We will address the issue of how to define and measure system level characteristics like autonomy, cognition or intelligence.
Another key topic will be a capability-led understanding of cognitive robots: how to define shared ontologies or dictionaries to discuss robotic cognitive systems in terms of their performance, relationships between different cognitive robotics capabilities, requirements, theories, architectures, models and methods that can be applied across multiple engineering and application domains, detailing and understanding better the requirements for robots in terms of performance, the approaches to meeting these requirements and the trade-offs in terms of performance.
Finally, epistemological issues in robotics research and its evaluation will be presented and discussed, related to performance measurement, methods for the objective comparison of different algorithms and systems, and the replication of published results.
The workshop will provide a forum for the discussion of these issues and will show a number ofexamples of replicable experiments in Robotics Research.
Topics of interest
- Metrics of Dexterity, adaptivity, flexibility, robustness
- Replication of experiments in robotics
- Metrics for visual servoing effectiveness and efficiency
- Metrics for shared control effectiveness and efficiency
- Benchmarking autonomy and robustness to changes in the environment/task
- Scalable autonomy measurements
- Reporting experiments in Robotics
- Epistemological issues
- Examples of Good Practice
- Evaluation of Experimental Robotics Work
- Proposals for Promotion of Good Experimental Work
This workshop is supported by IEEE RAS Technical Committee on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS).
This workshop is a joint initiative of the IEEE-TC on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS) and the SPARC TG on Good Experimental Methodology, Benchmarking, Challenges and Competitions.
The proposers have jointly co-organized over 20 related events in the last few years, such as workshops on Benchmarking and Performance Metrics at IROS (2006-2010, 2012-2013), several workshops at RSS (2008-2013) and the Performance Metrics for Intelligent Systems Workshop (PERMIS) series (2006-2012) and some organized at ICRA 2010 and 2011, 2012 on replication of experiments in robotics research.
The BioRobotics Institute, SSSA
Viale Rinaldo Piaggio 34,
56025 Pontedera (Pisa), Italy