Workshop at RSS 2009 (Seattle, WA, USA, June 28, 2009)


This e-mail address is being protected from spambots. You need JavaScript enabled to view it.    This e-mail address is being protected from spambots. You need JavaScript enabled to view it.    This e-mail address is being protected from spambots. You need JavaScript enabled to view it.


As the complexity of current robotic and embodied intelligent systems grows, it is more and more necessary to define proper experimental approaches and benchmarking procedures. The Special interest group on Good Experimental Methodology was formed by EURON in response to a wide perception — evidenced by some 90 responses to an email call for interest — that roboticists could do much better at performing and, particularly, reporting experimental work. A major output of the Special Interest Group is a set of guidelines for good experimental (reporting) practice, which we believe supports the community’s wish to produce better experimental work. Equally significant is a series of workshops on Benchmarking and Good Experimental Methodology held at major Robotics conferences. Good experimental work is not novel in robotics: there are many good experimental scientists in our community. However, uniformly good experimental work and reporting has not yet been achieved. The evidence of community interest in the topic mentioned above suggests that the community is willing to take steps to improve in this area. Thus we propose to take the work of good experimental robotics groups and use it to illustrate high-quality experimental work and reporting, using the workshop format to draw out the strengths (and weaknesses) of presented work and encourage sharing and adoption of good practices.

Workshop Highlights:

List of topics

  • Design of Experiments in Robotics
  • Execution of Experiments in Robotics
  • Reporting Experiments in Robotics
  • Examples of Good Practice
  • Evaluation of Experimental Robotics Work
  • Proposals for Promotion of Good Experimental Work


The workshop will consist of presentations interleaved with a significant amount of additional time for discussions between the presentations and at the end of the full day single track sessions.


The workshop has proceedings HERE BELOW!; selected contributions may be invited to submit to a journal special issue on good experimental robotics.

Previous workshops:

Previous workshops on the topic of Benchmarking have been organized successfully at IROS (3 in a row) and RSS conferences.

Unconference style interaction is encouraged:

This will be a 'working' :-) workshop interaction in person and by social networking media with people in the room and outside is welcome

Program & Pre-Proceedings


Coffee and registration


09:00 - 09:10


F. Bonsignorio


09:10 - 9:30

Publishing Identifiable Experiment Code And Configuration Is Important, Good and Easy

J. Wawerla R. T. Vaughan

School of Computing Science, Simon Fraser University


9:30 - 9:50

Toward a Science of Robotics: Goals and Standards for Experimental Research

L. Takayama

Willow Garage


9:50 -10:10

Experiment Design for Large Multi-Robot Systems

J. McLurkin

Computer Science and Engineering

University of Washington


10:10 - 10:30

Towards Establishing Clinical Credibility for Rehabilitation and Assistive Robots Through Experimental Design

K. M. Tsui H. A. Yanco

University of Massachusetts Lowell



Coffee Break


11:00 - 11:20

Experimental Identification of Friction and Dynamic Coupling in a Dual Actuator Testbed

D. Rabindran D. Tesar

Robotics Research Group

Department of Mechanical Engineering

The University of Texas at Austin, USA


11:20 - 11:40

Good Experimental Methodologies for Mobile Robot Olfaction

L. Marques

Institute of Systems and Robotics

University of Coimbra


11:40 - 12:00

A Proposal of a Set of Metrics for Collective Movement of Robots

I. Navarro F. Matia

Intelligent Control Group

Universidad Politecnica de Madrid


12:00 - 12:20

A Traceable Inertial Calibration Parameter Estimation Procedure Suited for MEMS Sensing

S. P. N. Singh

Australian Centre for Field Robotics, at the School of Aerospace, Mechanical, and Mechatronic Engineering

University of Sydney


12:20 - 13:00

Open Discussion


13:00 - 14:00



14:00 - 14:10

Morning recap

F. Bonsignorio


14:10 - 14:30

Datasets for the Evaluation of Multi-Sensor Perception in Natural Environments with Challenging Conditions

T. Peynot S. Scheding

ARC Centre of Excellence for Autonomous Systems

Australian Centre for Field Robotics

The University of Sydney


14:30 - 14:50

Acquisition of 2-D ground truth data in multirobot experiments

M. Delaine Anderson

The University of Alabama


14:50 - 15:10

Evaluating the Performance of Robot Mapping Systems

E. Olson, University of Michigan

M. Kaess, MIT


15:10 - 15:30

Evaluation of Adaptive Technology Algorithms in Small Robotic Platforms

M. Farmer J. Pippine C. Sullivan C. Valdez A. Watson

System Planning Corporation


15:30 - 16:00

Coffee Break


16:00 - 16:30

Defining the requisites of a replicable robotics experiment

F.P. Bonsignorio Heron Robots

A.P. Del Pobil University Jaume I Castellon

J. Hallam South Denmark University Odense


16:30 - 17:45

Open discussion


17:45 - 18:00


Additional information