Workshop at ICRA in Montreal, May 23rd, 2019
Organizers
F. Bonsignorio (CORRESPONDING CONTACT PERSON)
HeronRobots (Genoa), Italy
tel: +39 339 84 06 011
This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.
Signe A. Redfield
Naval Research Lab
4555 Overlook Ave, SW
Washington, D.C. 20375
This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.
A. P. del Pobil
Department of Engineering and Computer Science, Universitat Jaume I
12071 Castellon, Spain,
This e-mail address is being protected from spambots. You need JavaScript enabled to view it.
Abstract
In Robotics research the replicability and reproducibility of results and their objective evaluation and comparison is very difficult to put into practice. Controlling for environmental considerations is hard, and definingcomparable metrics and identifying goal similarity across various domains is poorly understood. Even the minimal goal of determining the information required to enable replication of results has been the subject of extensive discussion. Even worse, there is still no solid theoretical foundation for experimental replicability of experiments in robotics. This situation harms both research progress and technology transfer.
Thanks to the R-article process in IEEE RAM, we are now able to make progress towards a widespread and much needed practice of Reproducible Research in Robotics. This workshop will provide necessary information, knowledge and motivation to support the community’s transition to an environment where reproducible research is possible and common.
Content
The 2018 issue (Vol. 25, Issue 3) of the IEEE RAM highlights the importance and timeliness of this topic in its editorial and turning point column interview.
This special issue is composed of replicable experiments, demonstrating the improvements in the state of the art and identifying areas where further work is still needed.
The workshop proposers organized and held the 2015 IEEE RAS Summer School dedicated to these topics in September of 2015 in Benicassim, Castellon, Spain. The attendees could verify the improved understanding of the problem and the maturity of the approaches at least against specific topics or interest in the topic, as it is also shown by the RAM Special Issue quotedabove.
This workshop aims to gather researchers active in academia and industry to share the ideas so far developed and discuss the challenges still ahead. We will discuss how to encourage researchers to engage in reproducible and measurable research in robotics, how to improve reproducibility of robotics experiments, and above all, focus on issues regarding not just replication but generalization of results. We will also consider how the need forreproducibility affects publication formats, problem definitions and behavior specifications, and evaluation tools. Robotics is a wide and diverse science and engineering field and we will address at least some of the different approaches to reproducibility required by different aspects of the discipline. Finally, we will include presentations and discussions addressing epistemological issues in robotics research and its evaluation related to performance measurement, methods for the objective comparison of different algorithms and systems including shared concepts for task and capability representation, and the replication of published results,
Agenda
Room 518a 8:30 – 17:30
Introduction:
8:30 F. Bonsignorio, Heron Robots
Topic: The R-Article Process in IEEE Robotics & Automation Magazine
8:45 J. Dias, Khalifa University, UAE
Topic: Benchmarking and Assessment In Robotics Acting in Dynamic Environments - Facts and Figures from MBZIRC International Robotic Challenges
9:00 Stefano Carpin, University of California Merced, USA
Topic: R-Paper: Time Constrained Exploration Using Toposemantic Models
9:40 Reproducible Experimental Results of Swarming Behavior Using Mixed Reality
Victoria Edwards, Ioana Triandaf, Loy McGuire, Donald Sofge, Ira B. Schwartz
Naval Research Laboratory, USA
Chris Taylor
George Mason University, USA
10:00 – 10:30 Coffee Break
10:30 D. Scaramuzza, University of Zurich, Zurich, Switzerland
Topic: How to run reproducible visual SLAM experiments
11:10 Signe Redfield , Naval Research Laboratory, USA
Topic: The Case for Modular Task Specification: Specification via the future 1872.1 Task Ontology
11:50 Statistical Methods are Foundational to Reproducibility
P. Michael Furlong
Brian Coltin
Intelligent Robotics Group, NASA Ames (SGT), USA
12:10 DevOps@MECH - a cloud infrastructure for reproducible research
Johan Philips, Herman Bruyninckx
Department of Mechanical Engineering, KU Leuven, Belgium
12:30 – 14:00 Lunch Break
14:00 Neil Dantam, Colorado School of Mines, USA
Topic: Task-Motion Specification: Progress and Challenges
14:40 Performance Benchmarking via Adversarial Test Generation for Autonomous Surface Vehicles
Galen Mullins, Paul Stankiewicz
Johns Hopkins Applied Physics Laboratory
15:00 – 15:30 Coffee Break
15:30 M. Matteucci, Politecnico di Milano, Italy
Topic: Plug&Bench: Proposal for a Benchmark Composability Roadmap
16:10 Reproducing Industrial Robot Performance Observed in Lab Experiments in Simulations
Marek Franaszek, Geraldine S. Cheok, and Jeremy A. Marvel
National Institute of Standards and Technology, USA
16:30 Mike Danielczuk (on behalf of K. Goldberg), University of California Berkeley, USA
Topic: Reproducible Results and Benchmarking in Robotic Manipulation
17:10 ROSjects: reproducible and shareable robotics paper results based on ROS
Ricardo Tellez
Miguel A. Rodriguez
The Construct, Spain
17:30 Wrap-up
Intended Audience
The intended audience for this workshop includes researchers and practitioners from academia, the military, and industry, as well as people involved in research publishing inside and outside RAS. Reproducibility of research results and performance evaluation are key for research progress, result exploitation, and publishing across all disciplines.
Format
Each session will include 15-20 minutes talks interleaved with discussions. The discussions will be based on both preformatted questions for the speakers and the audience distributed weeks before the event and questions solicited from the audience during the event and from the web audience before it. We will encourage remote participation and interactions through streaming, hangouts and social media.
The talks provide context for the discussion periods and ensure the audience, both early-career and experienced, have enough information to participate and interact, becoming engaged in the workshop and ideally continuing the discussions through the breaks.
Topics
The proposed workshop is meant as a community town hall on the state of the art and the road ahead. The best contributions will be invited to submit to a refereed edited book or special issue in a high impact robotics journal.
Topics of interest
- Replication of experiments inrobotics
- Metrics of dexterity, adaptivity, flexibility,robustness
- Metrics for visual servoing effectiveness andefficiency
- Metrics for shared control effectiveness andefficiency
- Benchmarking autonomy and robustness to changes in theenvironment/task
- Shared concept development for comparison across tasks andcapabilities
- Scalable autonomymeasurements
- Reporting experiments inRobotics
- Epistemological issues
Support
This workshop is supported by the IEEE RAS TC-Pebras. It is also meant as the TC-Pebras TC meeting.
These issues are core issues for RAS and IEEE and they certainly are for TC-Pebras.