Workshop at ICRA in Montreal, May 23rd, 2019




HeronRobots (Genoa), Italy

tel: +39 339 84 06 011

This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.


Signe A. Redfield

Naval Research Lab

4555 Overlook Ave, SW

Washington, D.C.  20375

This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.


A. P. del Pobil

Department of Engineering and Computer Science, Universitat Jaume I

12071 Castellon, Spain,

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.


In Robotics research the replicability and reproducibility of results and their objective evaluation and comparison is very difficult to put into practice. Controlling for environmental considerations is hard, and definingcomparable  metrics and identifying goal similarity across various domains is poorly understood. Even the minimal goal of determining the information required to enable replication of results has been the subject of extensive discussion. Even worse, there is still no solid theoretical foundation for experimental replicability of experiments in robotics. This situation harms both research progress and technology transfer.

Thanks to the R-article process in IEEE RAM, we are now able to make progress towards a widespread and much needed practice of Reproducible Research in Robotics. This workshop will provide necessary information, knowledge and motivation to support the community’s transition to an environment where reproducible research is possible and common.


IEEE Robotics and Automation Magazine solicit Reproducible Article submissions, for more info please check here:  (check in particular section 9.), here and here

Main Details


The 2018 issue (Vol. 25, Issue 3) of the IEEE RAM highlights the importance and timeliness of this topic in its   editorial and  turning point column interview. 

This special issue is composed of replicable experiments, demonstrating the improvements in the state of the art and identifying areas where further work is still needed.

The workshop proposers organized and held the 2015 IEEE RAS Summer School dedicated to these topics in  September of 2015 in Benicassim, Castellon, Spain. The attendees could verify the improved understanding of the problem and the maturity of the approaches at least against specific topics or interest in the topic, as it is also shown by the RAM Special Issue quotedabove.

This workshop aims to gather researchers active in academia and industry to share the ideas so far developed and discuss the challenges still ahead. We will discuss how to encourage researchers to engage in reproducible and measurable research in robotics, how to improve reproducibility of robotics experiments, and above all, focus on issues regarding not just replication but generalization of results. We will also consider how the need forreproducibility affects publication formats, problem definitions and behavior specifications, and evaluation tools. Robotics is a wide and diverse science and engineering field and we will address at least some of the different  approaches to reproducibility required by different aspects of the discipline. Finally, we will include presentations and discussions addressing epistemological issues in robotics research and its evaluation related to performance measurement, methods for the objective comparison of different algorithms and  systems including shared concepts for task and capability representation, and the replication of published results,


Room 518a   8:30 – 17:30


8:30 F. Bonsignorio, Heron Robots

Topic: The R-Article Process in IEEE Robotics & Automation Magazine


8:45 J. Dias, Khalifa University, UAE


Topic: Benchmarking and Assessment In Robotics Acting in Dynamic Environments - Facts and   Figures from MBZIRC International Robotic Challenges


9:00 Stefano Carpin, University of California Merced, USA


Topic: R-Paper: Time Constrained Exploration Using Toposemantic Models


9:40  Reproducible Experimental Results of Swarming Behavior Using Mixed Reality


Victoria Edwards, Ioana Triandaf, Loy McGuire, Donald Sofge, Ira B. Schwartz

Naval Research Laboratory, USA


Chris Taylor


George Mason University, USA



10:00 – 10:30 Coffee Break



10:30 D. Scaramuzza, University of Zurich, Zurich, Switzerland


Topic: How to run reproducible visual SLAM experiments


11:10 Signe Redfield , Naval Research Laboratory, USA


Topic: The Case for Modular Task Specification: Specification via the future 1872.1 Task Ontology


11:50 Statistical Methods are Foundational to Reproducibility


P. Michael Furlong


Brian Coltin


Intelligent Robotics Group, NASA Ames (SGT), USA



12:10  DevOps@MECH - a cloud infrastructure for reproducible research


Johan Philips, Herman Bruyninckx


Department of Mechanical Engineering, KU Leuven, Belgium



12:30 – 14:00 Lunch Break



14:00  Neil Dantam, Colorado School of Mines, USA


Topic: Task-Motion Specification: Progress and Challenges



14:40 Performance Benchmarking via Adversarial Test Generation for Autonomous Surface Vehicles


Galen Mullins, Paul Stankiewicz


Johns Hopkins Applied Physics Laboratory



15:00 – 15:30 Coffee Break



15:30 M. Matteucci, Politecnico di Milano, Italy


Topic:  Plug&Bench: Proposal for a Benchmark Composability Roadmap


16:10 Reproducing Industrial Robot Performance Observed in Lab Experiments in Simulations


Marek Franaszek, Geraldine S. Cheok, and Jeremy A. Marvel


National Institute of Standards and Technology, USA



16:30 Mike Danielczuk (on behalf of K. Goldberg), University of California Berkeley, USA


Topic: Reproducible Results and Benchmarking in Robotic Manipulation



17:10 ROSjects: reproducible and shareable robotics paper results based on ROS


Ricardo Tellez

Miguel A. Rodriguez

The Construct, Spain



17:30 Wrap-up


Intended Audience

The intended audience for this workshop includes researchers and practitioners from academia, the military, and industry, as well as people involved in research publishing inside and outside RAS. Reproducibility of research results and performance evaluation are key for research progress, result exploitation, and publishing across all disciplines.


Each session will include 15-20 minutes talks interleaved with discussions.  The discussions will be based on both preformatted questions for the speakers and the audience distributed weeks before the event and questions solicited from the audience during the event and from the web audience before it. We will encourage remote participation and interactions through streaming, hangouts and social media.

The talks provide context for the discussion periods and ensure the audience, both early-career and experienced, have enough information to participate and interact, becoming engaged in the workshop and ideally continuing the discussions through the breaks.


The proposed workshop is meant as a community town hall on the state of the art and the road ahead.  The best contributions will be invited to submit to a refereed edited book or special issue in a high impact robotics journal.


Topics of interest         

  • Replication of experiments inrobotics
  • Metrics of dexterity, adaptivity, flexibility,robustness
  • Metrics for visual servoing effectiveness andefficiency
  • Metrics for shared control effectiveness andefficiency
  • Benchmarking autonomy and robustness to changes in theenvironment/task
  • Shared concept development for comparison across tasks andcapabilities
  • Scalable autonomymeasurements
  • Reporting experiments inRobotics
  • Epistemological issues


This workshop is supported by the IEEE RAS TC-Pebras. It is also meant as the TC-Pebras TC meeting.

These issues are core issues for RAS and IEEE and they certainly are for TC-Pebras.


Additional information