Working Session on
Experimental Settings in Program Comprehension: Challenges and Open Issues

June 16, 2006 Athens, Greece

Working Session Organizers:

Giuseppe Antonio Di Lucca
RCOST - University of Sannio
Benevento, Italy
dilucca@unisannio.it

Massimiliano Di Penta
RCOST - University of Sannio
Benevento, Italy
dipenta@unisannio.it

Important Dates:

Deadline for contributions:
May 15, 2006

Author's notification:
May 20, 2006

Call for Contributions

The success and the adoption of any research approach highly depends on the confidence the potential user has on its effectiveness. Evaluation and comparison with other approaches is therefore vital for any new research product. For program comprehension techniques, this is particularly important: when the aim is aiding a developer to better understand a software artifact, the researcher need to actually assess whether the techniques or the tool introduces a benefit. Moreover, the diffusion of alternative program comprehension techniques (e.g., alternative ways to visualize similar information) systematic comparisons are needed.
When designing and carrying out experiments, a series of issues have to be addressed, such as the proper instrumentation to be used, the factors affecting the study, the most adequate software system to experiment with, and the best way to facilitate replications of the study.
The aim of this working session is to discuss issues and open problems that can be encountered when planning and carrying out experiments on program comprehension techniques involving human subjects. Moreover, the session aims to act as a forum for the promotion of joint experimental studies that researchers believe to be relevant and worth to be performed/replicated.
Conceiving, designing and carrying out a controlled experiment requires a series of steps: experiment definition, planning, operation, analysis and interpretation, presentation and packaging. The discussion within the working session is articulated, but not limited to, into the following main dimensions:

• Variables worth to be investigated and factors affecting the study
• Experimental design
• Experimental Subjects
• Experimental Objects
• Instrumentation
• Packaging

We invite you to submit a position paper about one or more of the mentioned topics and to participate to the working session to discuss and propose ideas and solutions.

Working session organization
The working session starts with a limited set (four-five) of very short presentations (5 minutes each) either presenting (i) provocative statements or ideas related to any of the main dimensions reported before, (ii) proposals for coordinated empirical studies that can be carried out together with other participants. The aim is not to present results, but rather to drive on the discussion, propose challenging idea to plan experimentations and favor networking within participants.
Each presentation is followed by clarification questions and by a short discussion. One of the organizers acts as a moderator while the other collects a list of issues and experiment proposals, taking into account presentations, questions and the discussion held.
Subsequently, a broader discussion about the collected issues takes place and further ideas, comments and suggestion are collected.
A final post-conference document will summarize the results of the working session; the document is to distribute to the attendees and somebody else interested in the session topics.

SUBMISSION OF POSITION STATEMENTS

Interested participants are expected to submit informal abstracts or position papers, not exceeding 3 pages in length, and then to give a short presentation (10 min.) at the working session.
The submission deadline is May, 15, 2006. The official language is English. Authors are requested to submit a PostScript or PDF version of their position papers (IEEE proceeding format) by e-mail to the Working Session Organizers.
Submissions should be sent by email to Giuseppe A. Di Lucca at the address: dilucca@unisannio.it