Workshop at ICRA in Montreal, May 23rd, 2019

 

Organizers

F. Bonsignorio (CORRESPONDING CONTACT PERSON)

HeronRobots (Genoa), Italy

tel: +39 339 84 06 011

This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

 

Signe A. Redfield

Naval Research Lab

4555 Overlook Ave, SW

Washington, D.C.  20375

This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

 

A. P. del Pobil

Department of Engineering and Computer Science, Universitat Jaume I

12071 Castellon, Spain,

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

In Robotics research the replicability and reproducibility of results and their objective evaluation and comparison is very difficult to put into practice. Controlling for environmental considerations is hard, and definingcomparable  metrics and identifying goal similarity across various domains is poorly understood. Even the minimal goal of determining the information required to enable replication of results has been the subject of extensive discussion. Even worse, there is still no solid theoretical foundation for experimental replicability of experiments in robotics. This situation harms both research progress and technology transfer.

Thanks to the R-article process in IEEE RAM, we are now able to make progress towards a widespread and much needed practice of Reproducible Research in Robotics. This workshop will provide necessary information, knowledge and motivation to support the community’s transition to an environment where reproducible research is possible and common.

 

IEEE Robotics and Automation Magazine solicit Reproducible Article submissions, for more info please check here:  (check in particular section 9.), here and here



Main Details

Content

The 2018 issue (Vol. 25, Issue 3) of the IEEE RAM highlights the importance and timeliness of this topic in its   editorial and  turning point column interview. 

This special issue is composed of replicable experiments, demonstrating the improvements in the state of the art and identifying areas where further work is still needed.

The workshop proposers organized and held the 2015 IEEE RAS Summer School dedicated to these topics in  September of 2015 in Benicassim, Castellon, Spain. The attendees could verify the improved understanding of the problem and the maturity of the approaches at least against specific topics or interest in the topic, as it is also shown by the RAM Special Issue quotedabove.

This workshop aims to gather researchers active in academia and industry to share the ideas so far developed and discuss the challenges still ahead. We will discuss how to encourage researchers to engage in reproducible and measurable research in robotics, how to improve reproducibility of robotics experiments, and above all, focus on issues regarding not just replication but generalization of results. We will also consider how the need forreproducibility affects publication formats, problem definitions and behavior specifications, and evaluation tools. Robotics is a wide and diverse science and engineering field and we will address at least some of the different  approaches to reproducibility required by different aspects of the discipline. Finally, we will include presentations and discussions addressing epistemological issues in robotics research and its evaluation related to performance measurement, methods for the objective comparison of different algorithms and  systems including shared concepts for task and capability representation, and the replication of published results,

Agenda

Room 518a   8:30 – 17:30


Introduction: 

8:30 F. Bonsignorio, Heron Robots

Topic: The R-Article Process in IEEE Robotics & Automation Magazine

 

8:45 J. Dias, Khalifa University, UAE

 

Topic: Benchmarking and Assessment In Robotics Acting in Dynamic Environments - Facts and   Figures from MBZIRC International Robotic Challenges


 

9:00 Stefano Carpin, University of California Merced, USA

 

Topic: R-Paper: Time Constrained Exploration Using Toposemantic Models


 

9:40  Reproducible Experimental Results of Swarming Behavior Using Mixed Reality

 

Victoria Edwards, Ioana Triandaf, Loy McGuire, Donald Sofge, Ira B. Schwartz

Naval Research Laboratory, USA

 

Chris Taylor

 

George Mason University, USA

 

 

10:00 – 10:30 Coffee Break

 

 

10:30 D. Scaramuzza, University of Zurich, Zurich, Switzerland

 

Topic: How to run reproducible visual SLAM experiments


 

11:10 Signe Redfield , Naval Research Laboratory, USA

 

Topic: The Case for Modular Task Specification: Specification via the future 1872.1 Task Ontology


 

11:50 Statistical Methods are Foundational to Reproducibility

 

P. Michael Furlong

 

Brian Coltin

 

Intelligent Robotics Group, NASA Ames (SGT), USA

 

 

12:10  DevOps@MECH - a cloud infrastructure for reproducible research

 

Johan Philips, Herman Bruyninckx

 

Department of Mechanical Engineering, KU Leuven, Belgium

 

 

12:30 – 14:00 Lunch Break

 

 

14:00  Neil Dantam, Colorado School of Mines, USA

 

Topic: Task-Motion Specification: Progress and Challenges

 

 

14:40 Performance Benchmarking via Adversarial Test Generation for Autonomous Surface Vehicles

 

Galen Mullins, Paul Stankiewicz

 

Johns Hopkins Applied Physics Laboratory

 

 

15:00 – 15:30 Coffee Break

 

 

15:30 M. Matteucci, Politecnico di Milano, Italy

 

Topic:  Plug&Bench: Proposal for a Benchmark Composability Roadmap


 

16:10 Reproducing Industrial Robot Performance Observed in Lab Experiments in Simulations

 

Marek Franaszek, Geraldine S. Cheok, and Jeremy A. Marvel

 

National Institute of Standards and Technology, USA

 

 

16:30 Mike Danielczuk (on behalf of K. Goldberg), University of California Berkeley, USA

 

Topic: Reproducible Results and Benchmarking in Robotic Manipulation

 

 

17:10 ROSjects: reproducible and shareable robotics paper results based on ROS

 

Ricardo Tellez

Miguel A. Rodriguez

The Construct, Spain

 

 

17:30 Wrap-up

 


Intended Audience

The intended audience for this workshop includes researchers and practitioners from academia, the military, and industry, as well as people involved in research publishing inside and outside RAS. Reproducibility of research results and performance evaluation are key for research progress, result exploitation, and publishing across all disciplines.

Format

Each session will include 15-20 minutes talks interleaved with discussions.  The discussions will be based on both preformatted questions for the speakers and the audience distributed weeks before the event and questions solicited from the audience during the event and from the web audience before it. We will encourage remote participation and interactions through streaming, hangouts and social media.

The talks provide context for the discussion periods and ensure the audience, both early-career and experienced, have enough information to participate and interact, becoming engaged in the workshop and ideally continuing the discussions through the breaks.

Topics

The proposed workshop is meant as a community town hall on the state of the art and the road ahead.  The best contributions will be invited to submit to a refereed edited book or special issue in a high impact robotics journal.

 

Topics of interest         

  • Replication of experiments inrobotics
  • Metrics of dexterity, adaptivity, flexibility,robustness
  • Metrics for visual servoing effectiveness andefficiency
  • Metrics for shared control effectiveness andefficiency
  • Benchmarking autonomy and robustness to changes in theenvironment/task
  • Shared concept development for comparison across tasks andcapabilities
  • Scalable autonomymeasurements
  • Reporting experiments inRobotics
  • Epistemological issues

Support 

This workshop is supported by the IEEE RAS TC-Pebras. It is also meant as the TC-Pebras TC meeting.

These issues are core issues for RAS and IEEE and they certainly are for TC-Pebras.

 

Additional information