Operation Software Engineering II Sharif University of Technology - - PowerPoint PPT Presentation

Empirical Software Engineering Steps in the Experiment Process: Operation

Software Engineering II Sharif University of Technology MohammadAmin Fazli

Operation

 When an experiment has been designed and planned it must be carried

• ut in order to collect the data that should be analyzed.

 In the operational phase of an experiment, the treatments are applied

to the subjects.

 This part of the experiment is the part where the experimenter

actually meets the subjects.

 The operation deals to some extent with how to motivate people to

participate and take part in experiments.

 Even if an experiment has been perfectly designed and the collected

data is analyzed with the appropriate analysis methods, the result will be invalid if the subjects have not participated seriously in the experiment.

2

Operation Phases

 The operational phase of an experiment consists of three

steps:

 Preparation: where subjects are chosen and forms etc. are prepared  Execution: where the subjects perform their tasks according to

different treatments and data is collected

 Data validation: where the collected data is validated.

3

Preparation

 Before the experiment is actually executed there are some

preparations that have to be made.

 Preparation has two main steps:

 Select and inform participants  Prepare materials such as forms and tools

4

Commit Participants

 Before an experiment can be started, people who are willing

to act as subjects have to be found. It is essential that the people are motivated and willing to participate throughout the whole experiment.

 In many cases it is important to find people who work with

tasks in the experiment that are similar to their ordinary work tasks.

 C-coders for experiments involving C-coders, not java-coders

 When the right people are found and it is necessary to

convince these people to participate in the experiment.

5

Commit Participants

 Ethics:

 Obtain Consent:

 The participants have to agree to the research objectives.  It is important to describe how the result of the experiment will be used and

published

 Sensitive Results:

 If the results obtained in the experiment are sensitive for the participants, it is

important to assure the participants that the results of their personal performance in the experiment will be kept confidential.

 Inducement:

 One way to attract people to an experiment is to offer some kind of

inducement.

 Disclosure:

 Disclosure means to reveal all details of the experiment as openly as possible

to the experiment subjects.

6

Instrumentation Concerns

 Before the experiment can be executed, all experiment instruments

must be ready.

 Experiment Objects  Guidelines  Measurement forms & tools

 If the subjects themselves should collect data, this means in most

cases that some kind of forms must be handed out to the participants.

 One thing to determine when forms are constructed is whether

they should be personal or the participants should fill them out anonymously.

 In many cases it is appropriate to prepare one personal set of

instruments for every participant.

 This is because many designs deal with randomization and repeated tests,

such that different participants should be subject to different treatments.

7

Execution

 The experiment can be executed in a number of different

ways.

 Some experiments, such as simple inspection experiments can be

carried out at one occasion when all participants are gathered at, for example, a meeting.

 The result of the data collection can be obtained directly at the meeting  There is no need to contact the participants and later on ask for their

respective results

 The experimenter is present during the meeting and if questions arise they

can be resolved directly.

 Some experiments are, however, executed during a much longer time

span, and it is impossible for the experimenter to participate in every detail of the experiment and the data collection.

8

Data Collection

 Data can be collected either manually by the participants that

fill out forms, manually supported by tools, in interviews, or automatically by tools.

 Forms:

 Advantage: it does not require so much effort for the experimenter,

since the experimenter does not have to actively take part in the collection.

 Drawback: there is no possibility for the experimenter to directly

reveal inconsistencies, uncertainties and flaws in the forms.

 Interview:

 Advantage: the experimenter has the possibility to communicate

better with the participants during the data collection.

 Drawback: it needs much effort

9

Experimental Environment

 If an experiment is performed within a regular development

project, the experiment should not affect the project more than necessary.

 In some cases it is appropriate with some interaction between

the experiment and the project.

 Ex. Feedbacks for improving the project  This type of direct feedback from the experiment to the project can

help to motivate project personnel to participate in the experiment

10 10

Data Validation

 When data has been collected, the experimenter must check

that the data is reasonable and that it has been collected correctly.

 This deals with aspects such as if the participants have

understood the forms and therefore filled them out correctly.

 It is important to review that the experiment has actually been

conducted in the way that was intended.

 One way to check that the participants have not

misunderstood the intentions of the experimenter, is to give a seminar, or in some other way present the results of the data collection.

11 11

Example

 Preparation:

 Ph.D. and M.Sc. students are invited as subjects.  It is important to convince them to participate and get their commitment to

participate

 Consent must be ensured from the participants.  A suitable room booked and copies of all documents and forms must be

available for all subjects

 Given that time is going to be collected, a clock is needed in the room.

 Execution:

 Ensuring that the people are suitably spread out in the room.  It should be possible to run the experiment once with all subjects doing the

inspection at the same time.

 It is easy to provide support for any questions that may arise during the

experiment.

 Depending on whether the data should be collected by filling in forms by

hand or by use of a computer, preparation has to be done accordingly.

12 12

Example

 Data validation:

 It may be the case that one or several subjects leave the experiment

very early and their data forms have to be checked carefully to ensure that they have filled in the forms in a reasonable way.

 It must be checked that everybody has understood how to fill in the

data in a correct way.

13 13