Workshop at ICRA in Montreal, May 23rd, 2019

Organizers

F. Bonsignorio (CORRESPONDING CONTACT PERSON)

HeronRobots (Genoa), Italy

tel: +39 339 84 06 011

This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Signe A. Redfield

Naval Research Lab

4555 Overlook Ave, SW

Washington, D.C.  20375

This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

A. P. del Pobil

Department of Engineering and Computer Science, Universitat Jaume I

12071 Castellon, Spain,

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

In Robotics research the replicability and reproducibility of results and their objective evaluation and comparison is very difficult to put into practice. Controlling for environmental considerations is hard, and definingcomparable  metrics and identifying goal similarity across various domains is poorly understood. Even the minimal goal of determining the information required to enable replication of results has been the subject of extensive discussion. Even worse, there is still no solid theoretical foundation for experimental replicability of experiments in robotics. This situation harms both research progress and technology transfer.

Thanks to the R-article process in IEEE RAM, we are now able to make progress towards a widespread and much needed practice of Reproducible Research in Robotics. This workshop will provide necessary information, knowledge and motivation to support the community’s transition to an environment where reproducible research is possible and common.

Content

The 2018 issue (Vol. 25, Issue 3) of the IEEE RAM highlights the importance and timeliness of this topic in its   editorial and  turning point column interview. 

This special issue is composed of replicable experiments, demonstrating the improvements in the state of the art and identifying areas where further work is still needed.

The workshop proposers organized and held the 2015 IEEE RAS Summer School dedicated to these topics in  September of 2015 in Benicassim, Castellon, Spain. The attendees could verify the improved understanding of the problem and the maturity of the approaches at least against specific topics or interest in the topic, as it is also shown by the RAM Special Issue quotedabove.

This workshop aims to gather researchers active in academia and industry to share the ideas so far developed and discuss the challenges still ahead. We will discuss how to encourage researchers to engage in reproducible and measurable research in robotics, how to improve reproducibility of robotics experiments, and above all, focus on issues regarding not just replication but generalization of results. We will also consider how the need forreproducibility affects publication formats, problem definitions and behavior specifications, and evaluation tools. Robotics is a wide and diverse science and engineering field and we will address at least some of the different  approaches to reproducibility required by different aspects of the discipline. Finally, we will include presentations and discussions addressing epistemological issues in robotics research and its evaluation related to performance measurement, methods for the objective comparison of different algorithms and  systems including shared concepts for task and capability representation, and the replication of published results,

Agenda

Room 518a   8:30 – 17:30

Introduction: 

8:30 F. Bonsignorio, Heron Robots

Topic: The R-Article Process in IEEE Robotics & Automation Magazine

8:45 J. Dias, Khalifa University, UAE

Topic: Benchmarking and Assessment In Robotics Acting in Dynamic Environments - Facts and   Figures from MBZIRC International Robotic Challenges

9:00 Stefano Carpin, University of California Merced, USA

Topic: R-Paper: Time Constrained Exploration Using Toposemantic Models

9:40  Reproducible Experimental Results of Swarming Behavior Using Mixed Reality

Victoria Edwards, Ioana Triandaf, Loy McGuire, Donald Sofge, Ira B. Schwartz

Naval Research Laboratory, USA

Chris Taylor

George Mason University, USA

10:00 – 10:30 Coffee Break

10:30 D. Scaramuzza, University of Zurich, Zurich, Switzerland

Topic: How to run reproducible visual SLAM experiments

11:10 Signe Redfield , Naval Research Laboratory, USA

Topic: The Case for Modular Task Specification: Specification via the future 1872.1 Task Ontology

11:50 Statistical Methods are Foundational to Reproducibility

P. Michael Furlong

Brian Coltin

Intelligent Robotics Group, NASA Ames (SGT), USA

12:10  DevOps@MECH - a cloud infrastructure for reproducible research

Johan Philips, Herman Bruyninckx

Department of Mechanical Engineering, KU Leuven, Belgium

12:30 – 14:00 Lunch Break

14:00  Neil Dantam, Colorado School of Mines, USA

Topic: Task-Motion Specification: Progress and Challenges

14:40 Performance Benchmarking via Adversarial Test Generation for Autonomous Surface Vehicles

Galen Mullins, Paul Stankiewicz

Johns Hopkins Applied Physics Laboratory

15:00 – 15:30 Coffee Break

15:30 M. Matteucci, Politecnico di Milano, Italy

Topic:  Plug&Bench: Proposal for a Benchmark Composability Roadmap

16:10 Reproducing Industrial Robot Performance Observed in Lab Experiments in Simulations

Marek Franaszek, Geraldine S. Cheok, and Jeremy A. Marvel

National Institute of Standards and Technology, USA

16:30 Mike Danielczuk (on behalf of K. Goldberg), University of California Berkeley, USA

Topic: Reproducible Results and Benchmarking in Robotic Manipulation

17:10 ROSjects: reproducible and shareable robotics paper results based on ROS

Ricardo Tellez

Miguel A. Rodriguez

The Construct, Spain

17:30 Wrap-up

Intended Audience

The intended audience for this workshop includes researchers and practitioners from academia, the military, and industry, as well as people involved in research publishing inside and outside RAS. Reproducibility of research results and performance evaluation are key for research progress, result exploitation, and publishing across all disciplines.

Format

Each session will include 15-20 minutes talks interleaved with discussions.  The discussions will be based on both preformatted questions for the speakers and the audience distributed weeks before the event and questions solicited from the audience during the event and from the web audience before it. We will encourage remote participation and interactions through streaming, hangouts and social media.

The talks provide context for the discussion periods and ensure the audience, both early-career and experienced, have enough information to participate and interact, becoming engaged in the workshop and ideally continuing the discussions through the breaks.

Topics

The proposed workshop is meant as a community town hall on the state of the art and the road ahead.  The best contributions will be invited to submit to a refereed edited book or special issue in a high impact robotics journal.

Topics of interest         

• Replication of experiments inrobotics

• Metrics of dexterity, adaptivity, flexibility,robustness

• Metrics for visual servoing effectiveness andefficiency

• Metrics for shared control effectiveness andefficiency

• Benchmarking autonomy and robustness to changes in theenvironment/task

• Shared concept development for comparison across tasks andcapabilities

• Scalable autonomymeasurements

• Reporting experiments inRobotics

• Epistemological issues

Support 

This workshop is supported by the IEEE RAS TC-Pebras. It is also meant as the TC-Pebras TC meeting.

These issues are core issues for RAS and IEEE and they certainly are for TC-Pebras.