EXPERIMENTAL RESEARCH (Cont.):True Experimental Designs

<< EXPERIMENTAL RESEARCH:The Language of Experiments

EXPERIMENTAL RESEARCH (Cont.):Validity in Experiments >>

Research Methods STA630

Lesson 34

EXPERIMENTAL RESEARCH (Cont.)

Steps in Conducting an Experiment

Following the basic steps of the research process, experimenters decide on a topic, narrow it into a

testable research problem or question, and then develop a hypothesis with variables. Once a researcher

has the hypothesis, the steps of experimental research are clear. Broadly there are about 12 steps in

conducting an experiment, which are as below:

1. Begin with a straightforward hypothesis that is appropriate for experimental research.

2. Decide on an experimental design that will test the hypothesis within practical limitations. The

researcher decides the number of groups to use, how and when to create treatment conditions,

the number of times to measure the dependent variable, and what the groups of subjects will

experience from beginning till end.

3. Decide how to introduce the treatment or create a situation that induces the independent

variable.

4. Develop a valid and reliable measure of the dependent variable.

5. Set up an experimental setting and conduct a pilot test of the treatment and dependent variable

measures.

6. Locate appropriate subjects or cases.

7. Randomly assign subjects to groups (if random assignment is used in the chosen research

design) and give careful instructions.

8. Gather data for the pretest measure of the dependent variable for all groups (if pretest is used in

thee chosen design).

9. Introduce the treatment to the experimental group only (or to the relevant groups if there are

multiple experimental groups) and monitor all groups.

10. Gather data for posttest measure of the dependent variable.

11. Debrief the subjects by informing them of the true purpose and reasons for the experiment. Ask

subjects what they thought was occurring. Debriefing is crucial when subjects have been

deceived about some aspect of the treatment.

12. Examine data collected and make comparisons between different groups. Where appropriate,

use statistics and graphs to determine whether or not the hypothesis is supported.

Types of Designs

Researchers combine parts of experiment (e.g. pretests, control groups, etc.) together into an

experimental design. For example some designs lack pretests, some do not have control groups, and

others have many experimental groups. Certain widely used standard designs have names.

Classical Experimental Design: All designs are variations of the classical experimental design, which

has random assignment of subjects, a pretest and a posttest, an experimental group, and a control group.

Quasi-Experimental Designs:

One-shot Case Study Design: Also called the one-group posttest-only design, the one-shot case study

design has only one group, a treatment, and a posttest. Because it is only one group, there is no random

assignment. For example, a researcher shows a group of students a horror film, then measures their

attitude with a questionnaire. A weakness of this design is that it is difficult to say for sure that the

treatment caused the dependent variable. If subjects were the same before and after the treatment, the

researcher would not know it.

One Group Pretest-posttest Design: This design has one group, a pretest, a treatment, and a posttest. It

lacks a control group and random assignment. Continuing with the previous example, the researcher

gives a group of students an attitude questionnaire to complete, shows a horror film, then has them

117

Research Methods STA630

complete the same questionnaire second time. This is an improvement over the one-shot case study

because the researcher measures the dependent variable both before and after the treatment. But it lacks

the control group for comparison. The researcher cannot know whether something other than the

treatment occurred between the pretest and the posttest to cause the outcome.

Two Groups Posttest-only Design: It has two groups, a random assignment of subjects, a posttest, and

a treatment. It has all parts of the classical design except a pretest. Continuing with our previous

example, the researcher forms two groups through randomization process. He shows group a horror film

to one group i.e. the experimental group. The other group is not shown any film. Both groups then

complete the questionnaire. The random assignment reduces the chance that the groups differed before

the treatment, but without a pretest, a researcher cannot be as certain that the groups began the same on

the dependent variable.

True Experimental Designs

Experimental designs, which have at least two groups, a random assignment of subjects to experimental

and control groups, only experimental group is exposed to treatment, both groups record information

before and after the treatment, are known as ex-post facto experimental designs.

Pretest and Posttest Experimental and Control Group Design: Two groups, one control group and the

other experimental group, are formed randomly. Both the groups are exposed to pretest and posttest.

The experimental group is exposed to treatment while the control group is not. Measuring the

difference between the differences in the post- and pretests of the two groups would give the net effects

of the treatment.

Experimental Group: Pretest (O1) X

Posttest (O2)

Control Group: Pretest (O3) -

Posttest (O4)

Randomization used for setting up the group.

[(O2 O1) (O4 O3)] = Treatment effect (could be anywhere between 0 to -1 or +1).

Solomon's Four Group Design: To gain more confidence in internal validity in experimental designs,

it is advisable to set up two experimental groups and two control groups. One experimental group and

one control group can be given the both pretest and the posttest. The other two groups will be given

only the posttest. Here the effects of treatment can be calculated in several different ways as shown in

figure 1:

Figure 1: Solomon's four group design

Group

Pretest

Treatment

Posttest

1. Experimental

2. Control

3. Experimental

4. Control

(O2 O1) = E

(O4 - O3) = E

(O5 O6) = E

(O5 - O3) = E

[(O2 O1) (O4 O3)] = E

E = Effect

118

Research Methods STA630

If all Es are similar, the cause and effect relationship is highly valid.

Interaction Effect

The effect of two variables together is likely to be greater than the individual effect of each put together.

The idea of an interaction effect is familiar, especially in the area of medicine or illness. As an example,

imagine that for a given population of 100 persons, all of the same age and sex, it was found that if all

100 smoked cigarettes the effect would be a lung cancer rate of 20 percent. Assume that for an identical

group of 100 persons who did not smoke but lived in a smoggy environment, 10 percent would get lung

cancer. Now consider a third identical group of 100 persons all of whom smoke and also live in a

smoggy environment. The additive effect of both smoking and smog would be 20 percent plus 10

percent, or a total of 30 percent (30 people) having cancer. However, imagine that an actual medical

survey of the population shows a cancer rate of 37 percent among persons experiencing both smoking

and smog. This extra 7 percent can be computed residually as:

Interaction Effect = Total effect (smoking effect + smog effect) = 37 percent

= 37 percent - (20 percent + 10 percent)

= 37 percent - 30 percent

= 7 percent

In experiments we have the pretests and posttests, in which case we use the same instrument for

measuring the dependent variable, for example racial prejudice as an effect of a movie. In pretest is a

questionnaire in which items forming the prejudice scale are dispersed at random among other items so

that the subject does not know that his or her level of racial prejudice is being measured. Nevertheless,

the measurement of this variable (prejudice) itself, by presenting questions about race relations may

stimulate the subject's thinking and actually cause a change in his or her level of racial prejudice. Any

pretest effect that occurs will be visible as part of extraneous change (change caused by the test

stimulus) in the control group, as the pretest is also presented to the control group. Any change between

the pretest and posttest for measuring the dependent variable in the control group may be attributed to

the sensitization of the subjects with the instrument. In the experimental group of course a movie (an X

variable) was shown due to which we expect a change in the racial prejudice of the subjects. But that is

not all. The subjects in the experimental group were also exposed to the instrument for measuring the

racial prejudice, hence they were also sensitized. Their posttest results include the combined effect of

exposure to a movie and that of sensitization to the instrument. In other words the racial prejudice of

the subjects in the experimental group exhibits the interaction effect of the treatment plus that of

sensitization of the instrument.

In order to calculate the interaction effect in the experiment we shall have two experimental groups and

one control group created by using the randomization process. It may look like this:

Experimental group 1: Pretest (O1)

Posttest (O2)

Control group:

Pretest (O3)

Posttest (O4)

Why O4 be different from O3? The difference may be due to sensitization. So let us figure it out. Let us

take another experimental group and we do not pretest i.e. no sensitization with the instrument.

Experimental group 2: No pretest

Posttest (O5)

Let us work out the results:

(O2- O1) = D

(O4- O3) = D/

(O5 O3)= D// (Since all groups are identical, so we can use the pretest of any of the

Other two groups)

Interaction effect = D [D/ + D//]. Substituting it with our example of lung cancer

37 - [10 + 20] = 37 30 = 7

There are many other experimental designs like the randomized block design, Latin square design,

natural group design, and factorial design.

119

Table of Contents: