Workshop at Iros 2021 ONLINE, Prague, Czech Republic

on October 1st and October 8th, 2021

Organizers

F. Bonsignorio (CORRESPONDING CONTACT PERSON)

HeronRobots (Genoa), Italy

tel: +39 339 84 06 011

This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Signe A. Redfield

Naval Research Lab

4555 Overlook Ave, SW

Washington, D.C.  20375

This e-mail address is being protected from spambots. You need JavaScript enabled to view it. , This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

A. P. del Pobil

Department of Engineering and Computer Science, Universitat Jaume I

12071 Castellon, Spain,

This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Tomoyuki Yamamoto

ROBOCIP

https://robocip.or.jp

Email: This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Angela Faragasso

Department of Precision Engineering,

School of Engineering, The University of Tokyo

Hongo 7-3-1, Bunkyo, Tokyo 113-8656, JAPAN

Email: This e-mail address is being protected from spambots. You need JavaScript enabled to view it.

Objectives

Reproducibility of results is a core characteristic of the scientific method. However, in Robotics and AI research the replicability and reproducibility of results and their objective evaluation and comparison are usually lacking. Defining comparable metrics and identifying goal similarity across various domains is still poorly understood. In the past determining the information required to enable replication of results has been the subject of extensive discussion. This situation has impaired and still impairs both research progress and technology transfer.

Reproducible Research in Robotics is now possible, thanks to the R-article process in IEEE RAM. From 2017 IEEE Robotics & Automation Magazine solicits R-articles (i.e., papers that report experiments aim-ing to be fully reproducible), short articles reporting on the replication of r-Article results, and author short article replies.

All papers are peer reviewed. The result will be a two stage very high-quality review process. The first stage will be the ordinary rigorous review process of a top-tier publishing venue. The second stage will be the replication of the experiments by the community (what is usually the core of the scientific meth-od).

This event will provide necessary information, knowledge and motivation to allow the community to tran-sition to a reproducible research environment where objective evaluation of the performances of intelligent robotics and automation systems will be mainstream practice.

We will focus on the prospective for the future, analysing which test beds are currently available and for which specific field, what is missing and what, as a community, we need to provide for having more re-producible and comparable research.

REGISTER HERE!!!

Invited Speakers

• F. Bonsignorio (Heron Robots, Italy)
• G. Gillini and G. Antonelli (University of Cassino, Italy)
• E. Zereik (CNR,INM, Italy)
• S. Carpin (University of California Merced, USA)
• A. del Pobil (University Jaume I, Spain)
• A. Faragasso (University of Tokyo, Japan)
• D.Torricelli (CSIC, Spain)
• P. Corke and J. Haviland (Queensland University of Technology, Australia)
• Signe Redfield (Naval Research Laboratory, USA)
• Tomoyuki Yamamoto (ROBOCIP, Japan)
• Pedro Lima (IST, Portugal)
• Liam Paull (University of Montreal, Canada) (pending)
• Andrea Censi(ETH Zurich, Switzerland) (pending)

Selected RAS TC representatives of the other TCs give their position statements.

The workshop will happen in two steps: a teaser 2 hours event on October 1st and an 8 hours event on October 8th.

Only registered participants for IROS Workshops will be allowed to the October 8th event which is integral part of the IROS workshop.

Check back soon for more details.

REGISTER HERE!!!

Draft Schedule  (please check back before the workshop)

October 8th, 2021

All times in CEST

09:00 09:05 Where we are and the road ahead 
F. Bonsignorio (Heron Robots, Italy)

09:05 09:40 The Python Robotics Toolbox 
P. Corke and J. Haviland (Queensland University of Technology, Australia)

9:40 10:15 Robot Dynamics Identification: a Reproducible Comparison with experiments on the KINOVA Jaco 2 
G. Gillini and G. Antonelli (University of Cassino, Italy)

10:15 10:50 A simple visual servoing reproducible experiment 
E. Zereik (CNR,INM, Italy)

10:50 11:35 Relax and chat

11:35 12:10 Reproducibility and Benchmarking in the field (Health care, Nuclear, …) 
A. Faragasso (University of Tokyo, Japan)

12:10 12:45  Benchmarking locomotion in humans and machines 
D.Torricelli (CSIC, Spain)

12:45 14:20 Relax and chat

14:20 14:55 Comparing Different Approaches to Performance Evaluation of Robot Systems 
Pedro Lima (IST, Portugal)

14:55 15:30 Duckietown and AI Self-driving competitions

15:30 16:05  A Reproducibile experiment in mobile robotics 

S. Carpin (University of California Merced, USA)

16:05 16:40 Reproducibility and benchmarking in the verification of automous systems

Signe Redfield (Naval Research Laboratory, USA)

16:40 17:15   Guided discussion

17:15 17:30 Wrap up and invitation to the Winter School 

Chair: F. Bonsignorio (Heron Robots, Italy)

REGISTER HERE!!!

October 1st, 2021

All times in CEST

14:00 14:05 Where we are and the road ahead
F. Bonsignorio (Heron Robots, Italy)

14:05 14:10 The Python Robotics Toolbox
P. Corke and J. Haviland (Queensland University of Technology, Australia)

14:10 14:15 Reproducibility and Benchmarking in the field (Health care, Nuclear, …)
A. Faragasso (University of Tokyo, Japan)

14:15 14:20 Robot Dynamics Identification: a Reproducible Comparison with experiments on the KINOVA Jaco 2
G. Gillini and G. Antonelli (University of Cassino, Italy)

14:20 14:25 Relax and chat

14:25 14:30 A simple visual servoing reproducible experiment
E. Zereik (CNR,INM, Italy)

14:30 14:35 Benchmarking
A. del Pobil (University Jaume I, Spain)

14:35 14:40 Benchmarking locomotion in humans and machines
D.Torricelli (CSIC, Spain)

14:40 14:45 Comparing Different Approaches to Performance Evaluation of Robot Systems
Pedro Lima (IST, Portugal)

14:45 14:50 Duckietown and AI Self-driving competitions
Andrea Censi(ETH Zurich, Switzerland) (pending)

14:50 14:55 Relax and chat

14:55 15:00 Benchmarking Autonomous Vehicles
Liam Paull (University of Montreal, Canada) (pending)

15:00 15:05 Reproducibility and benchmarking in the verification of automous systems

Signe Redfield (Naval Research Laboratory, USA)

15:05 15:10 A Reproducibile experiment in mobile robotics
S. Carpin (University of California Merced, USA)

15:10 15:15 Relax and chat

15:15 15:55 Guided discussion with other TC Representatives

15:55 16:00 Wrap up and invitation to next session on October 8th
Chair: F. Bonsignorio (Heron Robots, Italy)

Topics

The proposed workshop is meant as a community town hall on the state of the art and the road ahead.  The best contributions will be invited to submit to a refereed edited book or special issue in a high impact robotics journal.

Topics of interest         

• Replication of experiments in robotics

• Metrics of dexterity, adaptivity, flexibility, robustness

• Metrics for visual servoing effectiveness and efficiency

• Reproducibility and Benchmarking in Aerial Intelligent Robotics

• Reproducibility and Benchmarking in Marine Intelligent Robotics

• Reproducibility and benchmarking in Mobile Manipulation

• Relevance of Reproducibility and benchmarking for practicalities in robotics

• Metrics for shared control effectiveness and efficiency

• Benchmarking autonomy and robustness to changes in the environment/task

• Shared concept development for comparison across tasks and capabilities

• Scalable autonomy measurements

• Reporting experiments in Robotics

• Test beds for robotic research

• Field specific benchmarking procedures and comparisons with other fields

• Generalization of methodologies issues

• Epistemological issues

• Examples of good practice

• Evaluation of experimental Robotics work

• Proposals for promotion of good experimental work

Support  

This workshop is supported by the IEEE RAS TC-Pebras. It is also meant as the TC-Pebras TC meeting.

These issues are core issues for RAS and IEEE and they certainly are for TC-Pebras.

Other supporting TCs are:

• Human Movement Understanding
• Marine Robotics
• Robotics Research for Practicalities
• Mobile Manipulation
• Verification of Autonomous Systems