Workshop at ICRA in Montreal, May 20-24, 2019




The BioRobotics Institute, SSSA

Viale Rinaldo Piaggio 34,

56025 Pontedera (Pisa), Italy

tel: +39 339 84 06 011

This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.


Signe A. Redfield

Naval Research Lab

4555 Overlook Ave, SW

Washington, D.C.  20375

This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.


A. P. del Pobil

Department of Engineering and Computer Science, Universitat Jaume I

12071 Castellon, Spain,

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.


In Robotics research the replicability and reproducibility of results and their objective evaluation and comparison is very difficult to put into practice. Controlling for environmental considerations is hard, and definingcomparable  metrics and identifying goal similarity across various domains is poorly understood. Even the minimal goal of determining the information required to enable replication of results has been the subject of extensive discussion. Even worse, there is still no solid theoretical foundation for experimental replicability of experiments in robotics. This situation harms both research progress and technology transfer.

Thanks to the R-article process in IEEE RAM, we are now able to make progress towards a widespread and much needed practice of Reproducible Research in Robotics. This workshop will provide necessary information, knowledge and motivation to support the community’s transition to an environment where reproducible research is possible and common.


IEEE Robotics and Automation Magazine solicit Reproducible Article submissions, for more info please check here:  (check in particular section 9.), here and here

Main Details


The 2018 issue (Vol. 25, Issue 3) of the IEEE RAM highlights the importance and timeliness of this topic in its   editorial and  turning point column interview. 

This special issue is composed of replicable experiments, demonstrating the improvements in the state of the art and identifying areas where further work is still needed.

The workshop proposers organized and held the 2015 IEEE RAS Summer School dedicated to these topics in  September of 2015 in Benicassim, Castellon, Spain. The attendees could verify the improved understanding of the problem and the maturity of the approaches at least against specific topics or interest in the topic, as it is also shown by the RAM Special Issue quotedabove.

This workshop aims to gather researchers active in academia and industry to share the ideas so far developed and discuss the challenges still ahead. We will discuss how to encourage researchers to engage in reproducible and measurable research in robotics, how to improve reproducibility of robotics experiments, and above all, focus on issues regarding not just replication but generalization of results. We will also consider how the need forreproducibility affects publication formats, problem definitions and behavior specifications, and evaluation tools. Robotics is a wide and diverse science and engineering field and we will address at least some of the different  approaches to reproducibility required by different aspects of the discipline. Finally, we will include presentations and discussions addressing epistemological issues in robotics research and its evaluation related to performance measurement, methods for the objective comparison of different algorithms and  systems including shared concepts for task and capability representation, and the replication of published results,

Prelimary Agenda


F. Bonsignorio, Heron Robots and the BioRobotics Institute, Italy

Topic: The R-Article Process in IEEE Robotics & Automation Magazine


Session 1:  Problem Definition


Signe Redfield , Naval Research Laboratory, USA

Topic: The Case for Modular Task Specification: Specification via the future 1872.1 Task Ontology


Neil Dantam, Colorado School of Mines, USA

Topic: The Case for Platform-Specific Task Specification: The Task-Motion Kit

Contributed Presentations


Coffee Break


Session 2:  Performance Measurement

Amit Kumar-Pandey, SoftBank Robotics, France

Topic: Benchmarking and Reproducible Robotics in Social Robots


D. Scaramuzza, University of Zurich, Zurich, Switzerland

Topic: How to run reproducible visual SLAM experiments


Stefano Carpin , University of California Merced, USA

Topic: How to Run Reproducible Visual Grasping Experiments

Contributed Presentations

Lunch Break


Session 3:  Objective Comparison

Matteo Matteucci, Politecnico di Milano, Italy

Topic: European Robotics League: Toward Sustainable Benchmarking Competitions


J. Dias, Khalifa University, UAE

Topic: Benchmarking and Assessment In Robotics Acting in Dynamic Environments - Facts and   Figures from MBZIRC International Robotic Challenges



W. Burgard (tbc), Toyota Research Institute, CA, USA and University of Freiburg, Freiburg, Germany

Topic:  Accurately Benchmarking SLAM and SLAM Experiment Reproducibility



Contributed Presentations


Coffee Break


Session 4:  Replication of Published Results

Francesco Amigoni, Politecnico di Milano, Milano, Italy

Topic: Experiments, tests, and benchmarks in robotics: Is a demarcation possible?


Ken Goldberg (tbc), University of California Berkeley, USA

Topic: Reproducible Results and Benchmarking in Robotic Manipulation


Contributed Presentations




Fabio Bonsignorio, Heron Robots and The BioRobotics Institute, Scuola Superiore Sant'Anna, Italy


Topic: Reproducible Research in Robotics: State of the Art and Road Ahead

Discussion and Wrap-up

List of Accepted Contributed Presentations

Reproducible Experimental Results of Swarming Behavior Using Mixed Reality

Victoria Edwards, Ioana Triandaf, Loy McGuire, Donald Sofge, Ira B. Schwartz

Naval Research Laboratory, USA

Chris Taylor

George Mason University, USA


Performance Benchmarking via Adversarial Test Generation for Autonomous Surface Vehicles

Galen Mullins, Paul Stankiewicz

Johns Hopkins Applied Physics Laboratory


Statistical Methods are Foundational to Reproducibility

P. Michael Furlong

Brian Coltin

Intelligent Robotics Group, NASA Ames (SGT), USA


ROSjects: reproducible and shareable robotics paper results based on ROS

Ricardo Tellez

Miguel A. Rodriguez

The Construct, Spain


Reproducibility Tools for Robotics Code: ROS, Docker, and Jupyter

Enric Cervera,

UJI,  Spain


DevOps@MECH - a cloud infrastructure for reproducible research

Johan Philips, Herman Bruyninckx

Department of Mechanical Engineering, KU Leuven, Belgium


Reproducing Industrial Robot Performance Observed in Lab Experiments in Simulations

Marek Franaszek, Geraldine S. Cheok, and Jeremy A. Marvel

National Institute of Standards and Technology, USA

Intended Audience

The intended audience for this workshop includes researchers and practitioners from academia, the military, and industry, as well as people involved in research publishing inside and outside RAS. Reproducibility of research results and performance evaluation are key for research progress, result exploitation, and publishing across all disciplines.


Each session will include 15-20 minutes talks interleaved with discussions.  The discussions will be based on both preformatted questions for the speakers and the audience distributed weeks before the event and questions solicited from the audience during the event and from the web audience before it. We will encourage remote participation and interactions through streaming, hangouts and social media.

The talks provide context for the discussion periods and ensure the audience, both early-career and experienced, have enough information to participate and interact, becoming engaged in the workshop and ideally continuing the discussions through the breaks.


The proposed workshop is meant as a community town hall on the state of the art and the road ahead.  The best contributions will be invited to submit to a refereed edited book or special issue in a high impact robotics journal.


Topics of interest         

  • Replication of experiments inrobotics
  • Metrics of dexterity, adaptivity, flexibility,robustness
  • Metrics for visual servoing effectiveness andefficiency
  • Metrics for shared control effectiveness andefficiency
  • Benchmarking autonomy and robustness to changes in theenvironment/task
  • Shared concept development for comparison across tasks andcapabilities
  • Scalable autonomymeasurements
  • Reporting experiments inRobotics
  • Epistemological issues


This workshop is supported by the IEEE RAS TC-Pebras. It is also meant as the TC-Pebras TC meeting.

These issues are core issues for RAS and IEEE and they certainly are for TC-Pebras.


Additional information