Workshop at IROS 2015

(Hamburg, Germany, September 28, 2015)

Proposers:

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.    

Objectives

In Robotics Research the replicability of results and their objective evaluation and comparison is very difficult to put into practice. Furthermore, there is no solid theoretical foundation for experimental replicability in robotics.

This situation impairs at the same time research progress and technology transfer.
Significant progress has been made in this respect in recent years. The current RAM issue is a special issue composed of replicable experiments and guest edited by some of the proposers will be published at the end of 2015, while in September the 2015 IEEE-RAS Summer School on Experimental Methodology, Performance Evaluation and Benchmarking in Robotics, has been organized in Benicassim, Castellon, Spain dedicated to same topics and organized by the proposers and John Hallam.

This workshop aims to gather researchers active in the academia and the industry to share the ideas so far developed and discuss the challenges still ahead.

The best contributions will be invited to submit to a refereed edited book or special issue on an high impact robotics journal.

Program

8:30 Introduction


8:40 Do we really need to replicate experiments?

Matteo Matteucci, Francesco Amigoni, Viola Schiafonati, Andrea Bonarini

Politecnico di Milano, Italy

 

9:10  Intelligence testing and robotics

Vincent C. Mueller

Amerikaniko Kollegio Anatolia, Greece 

 

9:40 The RoboCup Logistics League as a Holistic Multi-Robot Smart Factory Benchmark

Tim Niemueller1, Alexander Ferrein2, Sebastian Reuter3, Sabina Jeschke3, Gerhard Lakemeyer1

1Knowledge-Based Systems Group, RWTH Aachen University, Aachen,Germany

2Mobile Autonomous Systems & Cognitive Robotics Institute, FH Aachen University of Appl. Sc., Aachen, Germany

3Cluster IMA/ZLW & IfU, RWTH Aachen University, Aachen, Germany

 

10:00 Coffee Break


10:10 Benchmarking Service Robots

Luca Iocchi, Daniele Nardi

Sapienza University of Rome, Italy

 

10:40  Standards for service robotics

Gurvinder Virk

Univ. of Gävle & KTH, Sweden

 

11:10 Benchmarking of bipedal functions in humanoids and wearable robots

Diego Torricelli

Spanish National Research Council (CSIC), Spain

 

11:40 NAO race: a simulation benchmark for walking humanoids

Olivier Michel

Cyberbotics Ltd., Switzerland

 

12:10  Dataset Suite for Benchmarking Perception in Robotics

Aamir Ahmad1,2 , Pedro Lima

1Max Planck Institute for Biological Cybernetics, Tubingen, Germany

2Institute for Systems and Robotics, Instituto Superior Tecnico, Portugal

 

12:30 - 14:00 Lunch Break


14:00 Functional and Non-Functional Expressive Dimensions: Classification of

the expressiveness of the face of humanoid robots

Francois Ferland, Adriana Tapus

ENSTA-ParisTech, Paris, France

 

14:30 The challenges in benchmarking robot's social intelligence and some ideas: an industrial perspective

Amit Kumar Pandey

Aldebaran Robotics, France

 

15:00 Probabilistic approaches to robot task benchmarking

Pedro Lima

Institute for Systems and Robotics Instituto Superior Técnico, Portugal

 

15:30 Coffee Break


15:45  The Journal of Computational Research Replication (JCRR)

Benjamin Kuipers, Michael Wellman

Division of Computer Science & Engineering, University of Michigan, Ann Arbor, USA

 

16:15 Towards Reproducible Robotics Research

Fabio Bonsignorio

The BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa and Heron Robots, Italy

 

16:45 Panel and Plenary Discussion on RRR (Reproducible Robotics Research)

 

17:45 Conclusions and Wrap up


Robotics is a broad science and though we will try to cover different aspects of the discipline, the emphasis of the workshop will be on principles, methods, and applications in terms of cognitive capabilities and autonomy. We will address the issue of how to define and measure system level characteristics like autonomy, cognition or intelligence.

Another key topic will be a capability-led understanding of cognitive robots: how to define shared ontologies or dictionaries to discuss robotic cognitive systems in terms of their performance, relationships between different cognitive robotics capabilities, requirements, theories, architectures, models and methods that can be applied across multiple engineering and application domains, detailing and understanding better the requirements for robots in terms of performance, the approaches to meeting these requirements and the trade-offs in terms of performance.

Finally, epistemological issues in robotics research and its evaluation will be presented and discussed, related to performance measurement, methods for the objective comparison of different algorithms and systems, and the replication of published results.

The workshop will provide a forum for the discussion of these issues and will show a number ofexamples of replicable experiments in Robotics Research.

 

Topics of interest

  • Metrics of Dexterity, adaptivity, flexibility, robustness
  • Replication of experiments in robotics
  • Metrics for visual servoing effectiveness and efficiency
  • Metrics for shared control effectiveness and efficiency
  • Benchmarking autonomy and robustness to changes in the environment/task
  • Scalable autonomy measurements
  • Reporting experiments in Robotics
  • Epistemological issues
  • Examples of Good Practice
  • Evaluation of Experimental Robotics Work
  • Proposals for Promotion of Good Experimental Work
  • ...

 

Support

This workshop is supported by IEEE RAS Technical Committee on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS).
This workshop is a joint initiative of the IEEE-TC on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS) and the SPARC TG on Good Experimental Methodology, Benchmarking, Challenges and Competitions.
The proposers have jointly co-organized over 20 related events in the last few years, such as workshops on Benchmarking and Performance Metrics at IROS (2006-2010, 2012-2013), several workshops at RSS (2008-2013) and the Performance Metrics for Intelligent Systems Workshop (PERMIS) series (2006-2012) and some organized at ICRA 2010 and 2011, 2012 on replication of experiments in robotics research.

 

Contact

F. Bonsignorio
The BioRobotics Institute, SSSA
Viale Rinaldo Piaggio 34,
56025 Pontedera (Pisa), Italy
This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it. ,
This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Additional information