Workshop at IROS 2015

(Hamburg, Germany, September 28, 2015)

Proposers:

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.   This e-mail address is being protected from spambots. You need JavaScript enabled to view it.    

Objectives

In Robotics Research the replicability of results and their objective evaluation and comparison is very difficult to put into practice. Furthermore, there is no solid theoretical foundation for experimental replicability in robotics.

This situation impairs at the same time research progress and technology transfer.
Significant progress has been made in this respect in recent years. The current RAM issue is a special issue composed of replicable experiments and guest edited by some of the proposers will be published at the end of 2015, while in September the 2015 IEEE-RAS Summer School on Experimental Methodology, Performance Evaluation and Benchmarking in Robotics, has been organized in Benicassim, Castellon, Spain dedicated to same topics and organized by the proposers and John Hallam.

This workshop aims to gather researchers active in the academia and the industry to share the ideas so far developed and discuss the challenges still ahead.

The best contributions will be invited to submit to a refereed edited book or special issue on an high impact robotics journal.

Program

8:30 Introduction

8:40 Do we really need to replicate experiments?

Matteo Matteucci, Francesco Amigoni, Viola Schiafonati, Andrea Bonarini

Politecnico di Milano, Italy

9:10  Intelligence testing and robotics

Vincent C. Mueller

Amerikaniko Kollegio Anatolia, Greece 

9:40 The RoboCup Logistics League as a Holistic Multi-Robot Smart Factory Benchmark

Tim Niemueller¹, Alexander Ferrein², Sebastian Reuter³, Sabina Jeschke³, Gerhard Lakemeyer¹

¹Knowledge-Based Systems Group, RWTH Aachen University, Aachen,Germany

²Mobile Autonomous Systems & Cognitive Robotics Institute, FH Aachen University of Appl. Sc., Aachen, Germany

³Cluster IMA/ZLW & IfU, RWTH Aachen University, Aachen, Germany

10:00 Coffee Break

10:10 Benchmarking Service Robots

Luca Iocchi, Daniele Nardi

Sapienza University of Rome, Italy

10:40  Standards for service robotics

Gurvinder Virk

Univ. of Gävle & KTH, Sweden

11:10 Benchmarking of bipedal functions in humanoids and wearable robots

Diego Torricelli

Spanish National Research Council (CSIC), Spain

11:40 NAO race: a simulation benchmark for walking humanoids

Olivier Michel

Cyberbotics Ltd., Switzerland

12:10  Dataset Suite for Benchmarking Perception in Robotics

Aamir Ahmad^1,2 , Pedro Lima

¹Max Planck Institute for Biological Cybernetics, Tubingen, Germany

²Institute for Systems and Robotics, Instituto Superior Tecnico, Portugal

12:30 - 14:00 Lunch Break

14:00 Functional and Non-Functional Expressive Dimensions: Classification of

the expressiveness of the face of humanoid robots

Francois Ferland, Adriana Tapus

ENSTA-ParisTech, Paris, France

14:30 The challenges in benchmarking robot's social intelligence and some ideas: an industrial perspective

Amit Kumar Pandey

Aldebaran Robotics, France

15:00 Probabilistic approaches to robot task benchmarking

Pedro Lima

Institute for Systems and Robotics Instituto Superior Técnico, Portugal

15:30 Coffee Break

15:45  The Journal of Computational Research Replication (JCRR)

Benjamin Kuipers, Michael Wellman

Division of Computer Science & Engineering, University of Michigan, Ann Arbor, USA

16:15 Towards Reproducible Robotics Research

Fabio Bonsignorio

The BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa and Heron Robots, Italy

16:45 Panel and Plenary Discussion on RRR (Reproducible Robotics Research)

17:45 Conclusions and Wrap up

Robotics is a broad science and though we will try to cover different aspects of the discipline, the emphasis of the workshop will be on principles, methods, and applications in terms of cognitive capabilities and autonomy. We will address the issue of how to define and measure system level characteristics like autonomy, cognition or intelligence.

Another key topic will be a capability-led understanding of cognitive robots: how to define shared ontologies or dictionaries to discuss robotic cognitive systems in terms of their performance, relationships between different cognitive robotics capabilities, requirements, theories, architectures, models and methods that can be applied across multiple engineering and application domains, detailing and understanding better the requirements for robots in terms of performance, the approaches to meeting these requirements and the trade-offs in terms of performance.

Finally, epistemological issues in robotics research and its evaluation will be presented and discussed, related to performance measurement, methods for the objective comparison of different algorithms and systems, and the replication of published results.

The workshop will provide a forum for the discussion of these issues and will show a number ofexamples of replicable experiments in Robotics Research.

Topics of interest

• Metrics of Dexterity, adaptivity, flexibility, robustness
• Replication of experiments in robotics
• Metrics for visual servoing effectiveness and efficiency
• Metrics for shared control effectiveness and efficiency
• Benchmarking autonomy and robustness to changes in the environment/task
• Scalable autonomy measurements
• Reporting experiments in Robotics
• Epistemological issues
• Examples of Good Practice
• Evaluation of Experimental Robotics Work
• Proposals for Promotion of Good Experimental Work
• ...

Support

This workshop is supported by IEEE RAS Technical Committee on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS).
This workshop is a joint initiative of the IEEE-TC on Performance Evaluation and Benchmarking of Robotic and Automation Systems (PEBRAS) and the SPARC TG on Good Experimental Methodology, Benchmarking, Challenges and Competitions.
The proposers have jointly co-organized over 20 related events in the last few years, such as workshops on Benchmarking and Performance Metrics at IROS (2006-2010, 2012-2013), several workshops at RSS (2008-2013) and the Performance Metrics for Intelligent Systems Workshop (PERMIS) series (2006-2012) and some organized at ICRA 2010 and 2011, 2012 on replication of experiments in robotics research.

Contact

F. Bonsignorio
The BioRobotics Institute, SSSA
Viale Rinaldo Piaggio 34,
56025 Pontedera (Pisa), Italy
This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it. ,
This e-mail address is being protected from spambots. You need JavaScript enabled to view it.