EXPERIMENTAL RESEARCH (Cont.):Validity in Experiments

<< EXPERIMENTAL RESEARCH (Cont.):True Experimental Designs

NON-REACTIVE RESEARCH:Recording and Documentation >>

Research Methods STA630

Lesson 35

EXPERIMENTAL RESEARCH (Cont.)

Validity in Experiments

Experiments are judged by two measures. The first, internal validity indicates whether the independent

variable was the sole cause of the change in the dependent variable. It implies the researcher's ability to

eliminate alternative explanations of the dependent variable. Variables, other than the treatment, that

affect the dependent variable are threats to internal validity. They threaten the researcher's ability to say

that the treatment was the true causal factor producing change in the dependent variable. The second

measure, external validity, indicates the extent to which the results of the experiment are applicable in

the real world.

Internal validity is high in the laboratory experiment, reason being the control over all the confounding

factors. External validity (generalisability) is not sure because of the effect of variety of factors. Field

experiments have more external validity but less internal validity because it is closer to the real

situations.

Factors Affecting Internal Validity

In choosing or evaluating experimental research design, researchers must determine whether they have

internal and external validity. There are eight major types of extraneous variables that may jeopardize

internal validity: History effect, maturation effect, testing effect, instrumentation effect, selection bias

effect, selection bias effect, statistical regression, mortality, and mechanical loss.

1. History Effect: A specific event in the external environment occurring between the first and second

measurement that is beyond the control of the experimenter and that affects the validity of an

experiment. Advertisement of a particular product (mineral water) and its sale is affected by an event in

the society (contamination of drinking water). The researcher does not have control on such happenings

which have an impact on the X and Y relationship.

2. Maturation Effect: Cause and effect relationship can also be contaminated by the effects of the

passage of time another uncontrollable variable. Such contamination is called maturation effect. The

maturation effects are a function of the processes biological and psychological operating within the

subjects as a result of the passage of time. Examples of maturation processes could include growing

older, getting tired, feeling hungry, and getting bored. In other words there could be maturation effect

on the dependent variable purely because of the passage of time. For example, let us say that an R & D

director intends that an increase in the efficiency of workers would result within three months' time if

advanced technology is introduced in the work setting. If at the end of three months increased

efficiency is indeed found, I will be difficult to claim that the advanced technology (and it alone)

increased the efficiency of workers, because with the passage of time, employees would also gained

experience, resulting in better performance and therefore improved efficiency. Thus, the internal

validity also gets reduced owing to the effects of maturation in as much as it is difficult to pinpoint how

much of the increase is attributable to the introduction of the enhanced technology alone.

3. Testing Effects: Frequently, to test the effects of treatment, subjects are given what is called a

pretest (say, a short questionnaire eliciting their feelings and attitudes). That is, a measure of the

dependent variable is taken (pretest), then the treatment given, and after that a second test, called

posttest, administered. The difference between the posttest and the pretest scores is then attributed to

the treatment. However, the very fact that the subjects were exposed to the pretest might influence their

responses on the posttest, which will adversely impact on internal validity. It is also called sensitization

through previous testing.

120

Research Methods STA630

4. Instrumentation Effects: Instrumentation effects are yet another source of threat to internal

validity. These might arise because of a change in the measuring instrument between pretest and

posttest, and not because of the instrument's differential impact at the end. For example, in a weight-

loss experiment, the springs on the scale weaken during the experiment, giving lower readings in the

posttest.

A change in the wording of questions (may be done to avoid testing effects), change in interviewers, or

change in other procedures to measure the dependent variable can cause instrumentation effect.

Performance of the subjects measured by the units of output in the pretest, but when measuring the out

put in posttest the researcher measures it by "the number of units rejected, and the amount of resources

expended to produce the units.

5. Selection Bias Effect: Selection bias is the threat that subjects will not form equivalent groups. It is

a problem in design without random assignment, hence differential selection of the subjects for the

comparison groups. It occurs when subjects in one experimental group have a characteristic that affects

the dependent variable. For example, in an experiment on physical aggressiveness, the experimental

group unintentionally contains subjects who are sportsmen, whereas the control group is made up of

musicians, chess players, and painters.

6. Statistical Regression: Statistical regression is not easy to grasp intuitively. It is a problem of

extreme values or a tendency for random error to move group results towards the average. If extremes

are taken then they tend to regress toward the mean. Those who are on either end of the extreme would

not truly reflect the cause and effect relationship.

One situation arises when subjects are unusual with regard to dependent variable. Because they begin

as unusual or extreme, subjects are likely to respond further in the same direction. For example, a

researcher wants to see whether violent films make people act violently. The researcher chooses a

group of violent criminals from a high security prison, gives them a pretest, shows violent films, and

then administers a posttest. To the researcher's surprise, the criminals are slightly less violent after the

film, whereas a control group of non-prisoners who did see the film are slightly more violent than

before. Because the violent criminals began at an extreme, it is unlikely that a treatment could make

them more violent; by random chance alone, they appear less extreme when measured a second time.

If participants chosen for experimental group have extreme scores on the dependent variable to begin

with then the laws of probability say that those with very low scores on a variable have a greater

probability to improve and scoring closer to mean on the posttest after treatment. This phenomenon of

low scorers tending to score closer to the mean is known as "regressing toward the mean."

Likewise, those with high scores have a greater tendency to regress toward the mean will score lower

on the posttest than on pretest. Thus the extremes will not "truly" reflect the causal relationship a

threat to internal validity.

7. Mortality: Mortality, or attrition, arises when some subjects do not continue throughout the

experiment. Although the word mortality means death, it does not necessarily mean that subjects have

died. If a subset of subjects leaves partway through an experiment, a researcher cannot whether the

results would have been different had the subjects stayed. Even with departure of few subjects, the

groups do not remain balanced.

Consider for example of a training experiment that investigates the effects of close supervision of

salespersons (high pressure) versus low supervision (low supervision). The high pressure condition may

misleadingly appear to be superior if those subjects who completed the experiment did very well. If,

however, the high-pressure condition caused more subjects to drop-out than the other condition, this

apparent superiority may be due to a self-selection bias (those who could not bear the pressure had left

121

Research Methods STA630

mortality) perhaps only very determined and/or talented salespersons made it through the end of the

experiment.

8. Mechanical Loss: A problem may be experienced due to equipment failure. For example, in an

experiment if the subjects are told that their behavior is being video taped, and during the experiment

the video equipment failed to work for some subjects, then the validity of the results could become

doubtful.

9. Experimenter Expectancy: In addition to the usually listed eight factors affecting the internal

validity some times experimenter expectancy may threaten the causal logic of the relationship between

the variables. A researcher may threaten internal validity, not purposefully unethical behavior but by

indirectly communicating experimenter expectancy to the subjects. Researchers may highly committed

to the hypothesis and indirectly communicate desired findings to subjects. For example, a researcher

studying reactions towards disabled deeply believes that females are more sensitive toward the disabled

than the males are. Through eye contact, tone of voice, pauses, and other nonverbal communication, the

researcher unconsciously encourages female subjects to report positive feelings toward the disabled; the

researcher's nonverbal behavior is the opposite for male subjects.

The double-blind experiment is designed to control experimenter expectancy. In it, people who have

direct contact with subjects do not know the details of the hypothesis or the treatment. It is double blind

because both the subjects and those in contact with them are blind to details of the experiment. For

example a researcher wants to see if new drug is effective. Using capsules of three colors green,

yellow, and pink -- the researcher puts the new drug in the yellow capsule, puts an old drug in the pink

one, and take the green capsule a placebo a false treatment that appears to be real (e.g., a sugar capsule

without any physical effects). Assistants who give the capsules and record the effects do not know

which color contains the new drug. Only another person who does not deal with subjects directly knows

which colored capsule contains the drug and examines the results.

External Validity

Even if the researcher eliminates all concerns for internal validity, external validity remains a potential

problem. External validity is the ability to generalize experimental findings to real life situations.

Without external validity, the findings are of little use for both basic and applied research i.e. we shall

not be able to develop any theories that could be applicable to similar other situations.

Reactivity: A Threat to External Validity

Subjects may react differently in an experiment than they would in real life; because they know they are

in a study. The Hawthorn Effect, a specific kind of reactivity to the experimental situation is a good

example in this respect. The experiment was conducted in the Hawthorn Electric Company where the

performance of the participants was supposed to change due to the change in the environmental

conditions i.e. improvement on the environmental conditions will have a positive effect on thee

performance. The researchers modified many aspects of the working conditions and measured

productivity. Productivity rose after each modification. Productivity rose even if there was no real

modification but it was announced that there is a modification. The behavior change was simply a

reaction to the announcement of modification and some other factors like the participants were being

watched and had a feeling of being `very important persons.'

Here the workers did not respond to treatment (modification of working conditions) but to the additional

attention they received (being in the experiment ad being the focus of attention).

Demand characteristic (discussed earlier) is another type of reactivity. Here the participants change

their behavior as a reaction to the demands of the experimenter who may have inadvertently told the

subjects about the expected outcome of the treatment. They change their behavior as demanded by the

experimenter.

122

Research Methods STA630

Ethical Issues in Lab Experiments

We have already discussed the ethical issues in research. Just for the sake of emphasis, it may be

appropriate to very briefly repeat some of those which are specifically relevant to experimental designs.

The following actions may be unethical:

Putting pressure on individuals to participate in experiments through coercion, or apply social

pressure.

Asking demeaning questions from the subjects that hurt their self respect or giving menial task

to subjects that diminish their self respect.

Deceiving subjects by deliberately misleading them as to the true purpose of research.

Exposing participants to physical or mental stress.

Not allowing subjects to withdraw from the experiment when they want to.

Using research results to disadvantage the participants, or for purposes not to their liking.

Not explaining the procedures to be followed in the experiment.

Exposing subjects to hazardous and unsafe environments.

Not debriefing the participants fully and accurately after the experiment is over.

Not preserving the privacy and confidentiality of the information given by the participants.

Withholding benefits from the control group.

Human Subjects Committee

In order to protect the rights of participating subjects the research institutions have usually set up Ethics

Committees. Sometime project specific ethics committees are also formed. Such committees try to

look after the rights of the subjects participating in the experiments, as well as in other research

techniques.

123

Table of Contents: