Workshop at Iros 2021 ONLINE, Prague, Czech Republic
on October 1st and October 8th, 2021
Organizers
F. Bonsignorio (CORRESPONDING CONTACT PERSON)
HeronRobots (Genoa), Italy
tel: +39 339 84 06 011
This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.
Signe A. Redfield
Naval Research Lab
4555 Overlook Ave, SW
Washington, D.C. 20375
This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.
A. P. del Pobil
Department of Engineering and Computer Science, Universitat Jaume I
12071 Castellon, Spain,
This e-mail address is being protected from spambots. You need JavaScript enabled to view it.
Tomoyuki Yamamoto
ROBOCIP
https://robocip.or.jp
Email: This e-mail address is being protected from spambots. You need JavaScript enabled to view it.
Angela Faragasso
Department of Precision Engineering,
School of Engineering, The University of Tokyo
Hongo 7-3-1, Bunkyo, Tokyo 113-8656, JAPAN
Email: This e-mail address is being protected from spambots. You need JavaScript enabled to view it.
Objectives
Reproducibility of results is a core characteristic of the scientific method. However, in Robotics and AI research the replicability and reproducibility of results and their objective evaluation and comparison are usually lacking. Defining comparable metrics and identifying goal similarity across various domains is still poorly understood. In the past determining the information required to enable replication of results has been the subject of extensive discussion. This situation has impaired and still impairs both research progress and technology transfer.
Reproducible Research in Robotics is now possible, thanks to the R-article process in IEEE RAM. From 2017 IEEE Robotics & Automation Magazine solicits R-articles (i.e., papers that report experiments aim-ing to be fully reproducible), short articles reporting on the replication of r-Article results, and author short article replies.
All papers are peer reviewed. The result will be a two stage very high-quality review process. The first stage will be the ordinary rigorous review process of a top-tier publishing venue. The second stage will be the replication of the experiments by the community (what is usually the core of the scientific meth-od).
This event will provide necessary information, knowledge and motivation to allow the community to tran-sition to a reproducible research environment where objective evaluation of the performances of intelligent robotics and automation systems will be mainstream practice.
We will focus on the prospective for the future, analysing which test beds are currently available and for which specific field, what is missing and what, as a community, we need to provide for having more re-producible and comparable research.
Invited Speakers
- F. Bonsignorio (Heron Robots, Italy)
- G. Gillini and G. Antonelli (University of Cassino, Italy)
- E. Zereik (CNR,INM, Italy)
- S. Carpin (University of California Merced, USA)
- A. del Pobil (University Jaume I, Spain)
- A. Faragasso (University of Tokyo, Japan)
- D.Torricelli (CSIC, Spain)
- P. Corke and J. Haviland (Queensland University of Technology, Australia)
- Signe Redfield (Naval Research Laboratory, USA)
- Tomoyuki Yamamoto (ROBOCIP, Japan)
- Pedro Lima (IST, Portugal)
- Liam Paull (University of Montreal, Canada) (pending)
- Andrea Censi(ETH Zurich, Switzerland) (pending)
Selected RAS TC representatives of the other TCs give their position statements.
The workshop will happen in two steps: a teaser 2 hours event on October 1st and an 8 hours event on October 8th.
Only registered participants for IROS Workshops will be allowed to the October 8th event which is integral part of the IROS workshop.
Check back soon for more details.
Draft Schedule (please check back before the workshop)
October 8th, 2021
All times in CEST
09:00 09:05 Where we are and the road ahead
F. Bonsignorio (Heron Robots, Italy)
09:05 09:40 The Python Robotics Toolbox
P. Corke and J. Haviland (Queensland University of Technology, Australia)
9:40 10:15 Robot Dynamics Identification: a Reproducible Comparison with experiments on the KINOVA Jaco 2
G. Gillini and G. Antonelli (University of Cassino, Italy)
10:15 10:50 A simple visual servoing reproducible experiment
E. Zereik (CNR,INM, Italy)
10:50 11:35 Relax and chat
11:35 12:10 Reproducibility and Benchmarking in the field (Health care, Nuclear, …)
A. Faragasso (University of Tokyo, Japan)
12:10 12:45 Benchmarking locomotion in humans and machines
D.Torricelli (CSIC, Spain)
12:45 14:20 Relax and chat
14:20 14:55 Comparing Different Approaches to Performance Evaluation of Robot Systems
Pedro Lima (IST, Portugal)
14:55 15:30 Duckietown and AI Self-driving competitions
15:30 16:05 A Reproducibile experiment in mobile robotics
S. Carpin (University of California Merced, USA)
16:05 16:40 Reproducibility and benchmarking in the verification of automous systems
Signe Redfield (Naval Research Laboratory, USA)
16:40 17:15 Guided discussion
17:15 17:30 Wrap up and invitation to the Winter School
Chair: F. Bonsignorio (Heron Robots, Italy)
October 1st, 2021
All times in CEST
14:00 14:05 Where we are and the road ahead
F. Bonsignorio (Heron Robots, Italy)
14:05 14:10 The Python Robotics Toolbox
P. Corke and J. Haviland (Queensland University of Technology, Australia)
14:10 14:15 Reproducibility and Benchmarking in the field (Health care, Nuclear, …)
A. Faragasso (University of Tokyo, Japan)
14:15 14:20 Robot Dynamics Identification: a Reproducible Comparison with experiments on the KINOVA Jaco 2
G. Gillini and G. Antonelli (University of Cassino, Italy)
14:20 14:25 Relax and chat
14:25 14:30 A simple visual servoing reproducible experiment
E. Zereik (CNR,INM, Italy)
14:30 14:35 Benchmarking
A. del Pobil (University Jaume I, Spain)
14:35 14:40 Benchmarking locomotion in humans and machines
D.Torricelli (CSIC, Spain)
14:40 14:45 Comparing Different Approaches to Performance Evaluation of Robot Systems
Pedro Lima (IST, Portugal)
14:45 14:50 Duckietown and AI Self-driving competitions
Andrea Censi(ETH Zurich, Switzerland) (pending)
14:50 14:55 Relax and chat
14:55 15:00 Benchmarking Autonomous Vehicles
Liam Paull (University of Montreal, Canada) (pending)
15:00 15:05 Reproducibility and benchmarking in the verification of automous systems
Signe Redfield (Naval Research Laboratory, USA)
15:05 15:10 A Reproducibile experiment in mobile robotics
S. Carpin (University of California Merced, USA)
15:10 15:15 Relax and chat
15:15 15:55 Guided discussion with other TC Representatives
15:55 16:00 Wrap up and invitation to next session on October 8th
Chair: F. Bonsignorio (Heron Robots, Italy)
Topics
The proposed workshop is meant as a community town hall on the state of the art and the road ahead. The best contributions will be invited to submit to a refereed edited book or special issue in a high impact robotics journal.
Topics of interest
• Replication of experiments in robotics
• Metrics of dexterity, adaptivity, flexibility, robustness
• Metrics for visual servoing effectiveness and efficiency
• Reproducibility and Benchmarking in Aerial Intelligent Robotics
• Reproducibility and Benchmarking in Marine Intelligent Robotics
• Reproducibility and benchmarking in Mobile Manipulation
• Relevance of Reproducibility and benchmarking for practicalities in robotics
• Metrics for shared control effectiveness and efficiency
• Benchmarking autonomy and robustness to changes in the environment/task
• Shared concept development for comparison across tasks and capabilities
• Scalable autonomy measurements
• Reporting experiments in Robotics
• Test beds for robotic research
• Field specific benchmarking procedures and comparisons with other fields
• Generalization of methodologies issues
• Epistemological issues
• Examples of good practice
• Evaluation of experimental Robotics work
• Proposals for promotion of good experimental work
Support
This workshop is supported by the IEEE RAS TC-Pebras. It is also meant as the TC-Pebras TC meeting.
These issues are core issues for RAS and IEEE and they certainly are for TC-Pebras.
Other supporting TCs are:
- Human Movement Understanding
- Marine Robotics
- Robotics Research for Practicalities
- Mobile Manipulation
- Verification of Autonomous Systems