PILOT TESTING OF THE QUESTIONNAIRE:Discovering errors in the instrument

<< TOOLS FOR DATA COLLECTION:Guidelines for Questionnaire Design

INTERVIEWING:The Role of the Interviewer, Terminating the Interview >>

Research Methods STA630

Lesson 24

PILOT TESTING OF THE QUESTIONNAIRE

Pilot testing also called pre-testing means small scale trial run of a particular component; here we are

referring to pilot testing of the questionnaire.

Conventional wisdom suggests that pre-testing not only is an established practice for discovering errors

but also is useful for extra training the research team. Ironically, professionals who have participated in

scores of studies are more likely to pretest an instrument than is a beginning researcher hurrying to

complete a project. Revising questions five or more times is not unusual. Yet inexperienced

researchers often underestimate the need to follow the design-test-revise process.

It is important to pilot test the instrument to ensure that the questions are understood by the respondents

and there are no problems with the wording or measurement. Pilot testing involves the use of a small

number of respondents to test the appropriateness of the questions and their comprehension. Usually, the

draft questionnaire is tried out on a group that is selected on a convenience and that is similar in makeup

to the one that ultimately will be sampled. Making a mistake with 25 or so subjects can avert the

disaster of administering an invalid questionnaire to several hundred individuals. Hence the main

purpose of pilot testing is to identify potential problems with the methods, logistics, and the

questionnaire.

Administering a questionnaire exactly as planned in the actual study often is not possible. For example,

mailing out a questionnaire might require several weeks. Pre-testing a questionnaire in this manner

might provide important information on response rate, but it may not point out why questions were

skipped or why respondents found certain questions ambiguous or confusing. The ability of personal

interviewer to record requests for additional explanation and to register comments indicating

respondent's difficulty with question sequence or other factors is the primary reason why interviewers

are often used for pretest work.

What aspects to be evaluated during pilot testing?

1. Reactions of Respondents:

The reactions of the respondents can be looked at from different angles. The researcher may be familiar

with the local culture; still getting the first hand experience is always useful. Going to the field,

contacting the people, and their reactions to the different aspects of research may be a learning

experience.

�

Availability of study population timing. In case we are doing interviewing then pre-testing might

help to find out the most appropriate time when the respondent shall be available. The researcher

can plan the interviewing accordingly.

�

Acceptability of the questions asked. An important purpose of pre-testing is to discover

participants' reaction to the questions. If the participants do not find the experience stimulating

when an interviewer is physically present, how will they react on the phone, or in the self

administered mode? Pre-testing should help to discover where repetitiveness or redundancy is

bothersome or what topics were not covered that the participant expected. An alert interviewer will

look for questions or even sections that the participant perceives to be sensitive or threatening or

topics about which the participant knows nothing.

�

Pre-testing will also provide the opportunity to see the acceptability of the wording of the questions

in the local cultural context. Some of the issues may be discussed openly while for others people

use a disguised language. If people consider the use of certain phrases as offensive, then it is high

time to change the wording.

�

Willingness of the respondents to co-operate. Field testing of the questionnaire will give the idea

about the level of cooperation the research team is likely to get from the respondents, particularly if

they have to interview them.

Research Methods STA630

2. Discovering errors in the instrument:

�

Do the tools provide you the information? Reliability. Suitability for analysis. Tabulation

of the results /of a pretest helps determine whether the questionnaire will meet the objectives

of the research. A preliminary analysis often illustrates that although respondents can easily

comprehend and answer a given question, it is an inappropriate question because it does not

help solving the issue. The information may not be suitable for analysis.

�

Time taken/needed to interview/conduct the observation. Pre-testing can indicate the time

taken for interview or to conduct the observation. Too long questionnaires may not be

recommended and, therefore, need modification. It can also help in estimating average time

being taken to collect information form a respondent. Such an exercise can help in budget

estimations.

�

If there is any need to revise the format of the tool. Question arrangement can play a

significant role in the success of the instrument. May be we should start with stimulating

questions and place sensitive questions last. Such a situation might be handled through pre-

testing. Therefore, pre-testing may help in putting questions in proper sequence, using

acceptable wording, doing appropriate translation, question spacing, structuring of

answers, coding system, and needing instructions for interviewers (probing).

3. Sampling procedure can be checked:

�

The extent to which instructions given are followed. Field functionaries are given the

instructions for following a sampling procedure. Depending upon the type of sampling to be

followed, the field worker must follow the guidelines otherwise the quality of the study will be

hampered. During the pre-testing one could see not only the extent to which the instructions

are being followed but also locate the problems in carrying out those instructions. Also what

could be the solutions to those problems?

�

How much time is needed to locate the respondents? By following the instructions how

easy it is to locate the respondents, and how much time is needed to do that activity. It could

help in calculating the overall time for data collection, having relevancy for budgeting thee

resources.

4. Staffing and activities of research team can be checked:

�

How successful the training has been? Pre-testing can be seen as a period of extra training.

The pre-testing exercise can provide a good opportunity to make an evaluation of the

achievement of the objectives of training. For any deficiencies additional training may be

provided.

�

What is the work output of each member? The researcher can calculate the average output

of each fieldworker and accordingly calculate the number of workers needed to finish the

work on time. It can also help in making the budget estimates.

�

How well the research team works together? It is a good opportunity to observe the kind of

coordination the research team has. The integrated work is likely affect the efficiency of the

team. Any shortcomings could be looked after.

�

Is the logistical support adequate? Of course we are leaving the field functionaries in

isolation. They shall be in need of other logistical support like the transportation, boarding,

lodging, guidance and supervision. Some of these aspects could also be appraised during the

pre-testing

5. Procedure for data processing and analysis can be evaluated:

�

Make dummy tables. See how can we tabulate the data and use the appropriate statistics for

purposes of interpretations

Table of Contents: