Chapter Eleven: Questionnaire Design
Chapter 11
Questionnaire Design
AT-A-GLANCE
I. Introduction
II. Basic Considerations in Questionnaire Design
A. What should be asked?
B. Questionnaire relevancy
C. Questionnaire accuracy
III. Question Phrasing: Open- or Close-Ended Statements?
A. Open-ended response versus fixed-alternative questions
2. Fixed-alternative questions
B. Types of fixed-alternative questions
C. Phrasing questions for self-administered, telephone, and personal interview surveys
IV. Avoiding Mistakes
A. Simpler is better
B. Avoid leading and loaded questions
C. Avoid ambiguity: Be as specific as possible
D. Avoid double-barreled items
E. Avoid making assumptions
F. Avoid taxing respondent’s memory
V. Order Bias
A. Question sequence
B. Randomized presentations
C. Randomized response techniques
D. Survey flow
E. Traditional questionnaires
VI. Survey Technology
A. Response quality
B. Timing
C. Randomized assignment
D. Physical features
1. Tracking interest
3. Prompting
4. Piping
VII. Pretesting and Revising Questionnaires
Chapter Eleven: Questionnaire Design
LEARNING OUTCOMES
2. Choose between open-ended and fixed-alternative questions.
4. Minimize problems with order bias.
6. Use the latest survey technology to reduce respondent error
7. Appreciate the importance of pretesting survey instruments.
CHAPTER VIGNETTE: Ask a Sensitive Question, Get a Sensitive
Answer
Survey researchers believe that the responses people give to questionnaire items provide a valid
representation of the interests, opinions, or behaviors studied. Researchers should always pay
careful attention to how a question is asked, but particularly so when the subjects are sensitive
and reveal potentially self-incriminating or embarrassing facts about an individual. Marketing
SURVEY THIS!
As you went through the questionnaire, did you spot any problem questions? Students should be
able to better describe the issues after finishing this chapter. Some considerations include topics
that would be better covered through a phone interview or a personal interview. Students are
asked to examine a section of the questionnaire for potential problems using the terminology from
the chapter. That is, do the items display characteristics that should be avoided?
RESEARCH SNAPSHOTS
➢ Who Really Does Housework?
Married women have been part of the workforce for decades, but they continue to note that
when both spouses hold jobs, women bear the heavier responsibility for housework.
Researchers have considered whether the problem is one of fairness or misperception, and
some have suggested that the difference in estimates depends partly on data gathering. First,
it matters whom the researcher questions when they ask. Both men and women overestimate
the time they devote to housework when asked to recall what they do at home. Also, defining
time spent on housework turns out to be more complex. Sometimes, behavioral evidence can
validate (or invalidate) survey results. People who do more housework do not have more
children than other couples. Does this behavioral result say anything about potential bias in
the survey results?
Chapter Eleven: Questionnaire Design
➢ I Give Up!
The questionnaire design not only aims to get valid data—but design features also assist in
getting data at all. No matter what the interview mode, a large portion of respondents give up
TIPS OF THE TRADE
➢ Keep questionnaire wording simple
▪ Use shorter words that have concrete meaning
▪ Use shorter statements—one line is best—never more than two lines
▪ When questioning about frequency, try to use numbers instead of vague terms like
rarely.
➢ Design questionnaire for least capable respondent in the sampling frame
➢ In general, funnel respondents from general to specific questions
➢ Randomize question order when possible to avoid order bias
OUTLINE
I. INTRODUCTION
A. The questionnaire is the primary tool for building responses to research questions.
B. Questionnaire design is one of the most critical stages in the survey research process.
C. Ask a bad question and you get bad results.
II. BASIC CONSIDERATIONS IN QUESTIONNAIRE DESIGN
A. Good questionnaire design requires far more that correct grammar.
B. To fulfill a researcher’s purposes, the questions must meet the basic criteria of
relevance and accuracy.
C. To achieve these ends, the decisions in the following order must be made:
1. What should be asked?
2. How should questions be phrased?
3. In what sequence should the questions be arranged?
4. What questionnaire layout will best serve the research objectives?
5. How can the questionnaire encourage complete responses?
6. How should the questionnaire be pretested and then revised?
Chapter Eleven: Questionnaire Design
D. What Should Be Asked?
2. The later stages of the research process will have an important impact on the
questionnaire wording.
3. When designing the questionnaire, the researcher must also be thinking about the
types of statistical analyses that will be conducted.
E. Questionnaire Relevancy
1. A questionnaire is relevant to the extent that all information collected addresses a
3. Irrelevant questions are more than a nuisance because they make the survey
needlessly long.
4. When planning the questionnaire design, researchers must think about possible
omissions.
F. Questionnaire Accuracy
2. While it is generally believed that one should use simple, understandable,
3. Respondents tend to be most cooperative when the subject of the research is
interesting.
5. Question wording and sequence substantially influence accuracy.
III. QUESTION PHRASING: OPEN- OR CLOSE-ENDED STATEMENTS?
A. There are many ways to phrase questions, and many standard question formats have
been developed in previous research.
B. Open-Ended Response versus Fixed-Alternative Questions
1. We may categorize two basic types of questions asked on the amount of freedom
respondents are given in answering.
3. Open-ended response questions are free-answer questions.
4. In contrast, fixed-alternative questions (a.k.a., closed questions) give
5. Using Open-Ended Response Questions
a. Open-ended response questions are most beneficial when the researcher is
conducting exploratory research.
b. By gaining free and uninhibited responses, the researcher may find some
Chapter Eleven: Questionnaire Design
© 2016 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, in whole or in
part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website for classroom use.
e. Another potential disadvantage of open-ended response questions is the
possibility of interviewer bias to influence the answer; even the best
interviewer can take shortcuts in recording answers.
f. Also, articulate individuals tend to give longer answers and such respondents
often are better educated and from higher income groups, which may not be
representative of the entire population but gives them a large share of the
responses.
C. Fixed-Alternative Questions
2. Answers to closed questions are classified into standardized groupings prior to
3. However, when a researcher is unaware of the potential responses to a question,
fixed-alternative questions cannot be used.
5. Unanticipated alternatives emerge when respondents believe that closed answers
6. Respondents may check off obvious alternatives if they do not see the choice
they would prefer.
8. Most questionnaires mix open-ended and closed questions, providing a change of
pace that can eliminate respondent boredom and fatigue.
D. Types of Fixed-Alternative Questions
1. Simple-dichotomy (dichotomous-alternative) questions – require the
2. Several types of questions provide the respondent with multi-choice alternatives:
a. The multiple-choice question requires the respondent to choose one
response from among multiple alternatives.
b. The frequency-determination question is a determinant-choice question
Chapter Eleven: Questionnaire Design
G. Avoid Making Assumptions
1. The researcher should not place the respondent in a bind by including an implicit
assumption in the question.
2. Another frequent mistake is assuming that the respondent had previously thought
about an issue; research that induces people to express attitudes on subjects that
they do not ordinarily think about is meaningless.
H. Avoid Taxing Respondent’s Memory
2. Questions that give no clue as to the brand of interest are referred to as unaided
recall questions.
4. While aided recall is not as strong a test of attention or memory as unaided recall,
it is less taxing to the respondent’s memory.
6. Squishing occurs when respondents think that recent events took place longer ago
than they really did.
7. In situations in which “I don’t know” or “I can’t recall” is a meaningful answer,
simply including a “don’t know” response category may solve the question
writer’s problem.
V. ORDER BIAS
A. Question Sequence
2. For example, if respondents’ curiosity is not aroused at the outset, they can
become disinterested and terminate the interview.
3. It is generally not advisable to ask demographic or classificatory questions (i.e.,
5. Order bias tends to distort survey results.
7. It is advisable to ask the general questions before the specific questions to obtain
the freest of open-ended responses.
a. This technique is known as the funnel technique, and it allows researchers
to understand the respondent’s frame of reference before asking more
specific questions about the level of the respondent’s information and
intensity of opinion.
B. Randomized presentations
1. With attitude scales, there also may be an anchoring effect.
a. The first concept measured tends to become a comparison point from which
2. A related problem is bias caused by the order of alternatives on closed questions.
3. The order of these choices should be rotated if producing alternative forms of the
© 2016 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, in whole or in
part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website for classroom use.
5. A more common practice is to pencil X’s or check marks on the printed
questionnaires to indicate where the interviewer should start a series of repetitive
questions at a certain point.
C. Randomized response techniques
1. Randomized response techniques involve randomly assigning respondents to
2. The use of randomized response techniques remains controversial based in part
on the willingness and ability of respondents to follow procedures.
D. Survey flow
1. Survey flow refers to the ordering of questions through a survey.
3. Asking a question that does not apply to the respondent or that the respondent is
4. A breakoff means the respondent stops answering questions resulting in an
incomplete survey.
6. Branching directs respondents to alternative portions of the questionnaire, based
on their response to a filter question.
E. Traditional Questionnaires
1. In mail questionnaires, often the rate of return can be improved by adding the
2. Questionnaires should be designed to appear as short as possible.
4. Instructions are often capitalized or printed in bold to alert the interviewer that it
may be necessary to proceed in a certain way.
5. Layout is extremely important when questionnaires are long or require the
respondent to fill in a large amount of information.
VI. SURVEY TECHNOLOGY
A. Survey software programs like Qualtrics allow several special features that facilitate
design and allow important data to be collected that may be difficult otherwise.
B. Response quality
1. The researcher can build in questions to test whether a respondent is paying
attention, which can increase response quality.
C. Timing
2. The timers produce potentially useful behavioral data and also help identify types
of respondent error such as speeding.
3. The presumption is that speeders, those who move through a survey much faster
Chapter Eleven: Questionnaire Design
D. Randomized assignment
2. In implementing experimental designs, the researcher assigns a certain set of
conditions to each respondent.
3. Electronic survey software facilitates randomized assignment.
E. Physical features
1. Several physical features of questionnaires can influence responses.
2. Tracking interest
a. Electronic questionnaires sometimes allow us to come close to tracking
3. Status bar
a. With a paper questionnaire, a respondent can flip through a questionnaire and
know about how many questions are in a survey.
4. Prompting
a. Web-based surveys allow limited interactivity.
b. One useful piece of feedback comes in the form of a prompt message.
5. Piping
a. Survey software can systematically or randomly manipulate the questions a
respondent sees.
b. Piping software allows responses to a previous question to be inserted into
later questions.
VII. PRETESTING AND REVISING QUESTIONNAIRES
A. Typically, the questionnaire is tried out on a group selected on a convenience basis
that is similar in makeup to the one that ultimately will be sampled.
B. It is not necessary to get a statistical sample for pretesting.
C. Pretesting allows the researcher to determine if the respondents have any difficulty
Chapter Eleven: Questionnaire Design
© 2016 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, in whole or in
part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website for classroom use.
E. A preliminary tabulation of the pretest results often illustrates that while a question
is easily comprehended and answered by the respondent, it is inappropriate because it
does not solve the marketing problem.
F. Administering a questionnaire exactly as planned in the actual study often is not
possible.
G. Interviewers are often used for pretest work because they can record requests for
additional explanation or comments that indicate respondents’ difficulty with
question sequence or other factors.
H. Pretests are typically conducted to answer questions about the questionnaire such as:
1. Can the questionnaire format be followed by the interviewer?
2. Does the questionnaire flow naturally and conversationally?
3. Are the questions clear and easy to understand?
4. Can respondents answer the questions easily?
5. Which alternative forms of questions work best?
6. What overall and item response rates can be expected?
I. Pretests also provide the means for testing the sampling procedure and estimates for
the response rates for the mail surveys and completion rates for telephone surveys.
J. Designing Questionnaires for Global Markets
2. The most common problem involves translation into another language.
3. Back translation is the process of translating the questionnaire from one
language to another and then having it translated back again by a second,
independent translator. The back translator is often a person whose native tongue
is the language that will be used on the questionnaire.
QUESTIONS FOR REVIEW AND CRITICAL THINKING/ANSWERS
1. What are six critical questions for a researcher in designing a questionnaire?
1) What should be asked?
2) How should questions be phrased?
3) In what sequence should the questions be arranged?
4) What questionnaire layout will best serve the research objectives?
5) How can the questionnaire encourage complete responses?
6) How should the questionnaire be pretested and then revised?
2. Evaluate and comment on the following questions, taken from several different questionnaires:
See the textbook for the actual questions.
Chapter Eleven: Questionnaire Design
b. This question about the Federal Trade Commission’s corrective advertising policy is
c. The first question asks the respondents to imagine that they do 90 percent of their driving
d. Questions that ask the percentage of usage, time, and the like are fine if every respondent
makes sure that his or her answer amounts to 100 percent. Unfortunately, many times
respondents will only partially answer the question or incorrectly account for 100
percent. They may indicate that they walk 60 percent of the time without filling in any of
e. For this question, it may be that not all respondents will believe that the cooperative
f. This survey asks retirees to rate how often they use various media and devices for
product information and discounts. Some retirees may become frustrated and exit the
survey, because many of the devices and sites are uncommon to them.
g. This survey question has answers that are not mutually exclusive, and may be confusing
to respondents.
h. This question seems to be somewhat loaded and emotionally charged.
i. Although the question asks for one of two alternatives, “up” or “down,” if the
respondents answer “stays the same” or “don’t know,” these responses will be recorded.
j. The wording on this question is a little strange. It also asks students to state their GPA,
which they may not truthfully report.
k. This question starts with an assumption: “Since agriculture is vital to our state’s economy”;
it assumes that the respondent will agree with this statement.
3. The following question was asked of a sample of television viewers:
Chapter Eleven: Questionnaire Design
▪ Diehard Fan: Watch games, follow-up on scores and sports news multiple times a
day
▪ Avid Fan: Watch games, follow-up on scores and sports news once a day
Does this question do a good job of avoiding ambiguity? If this is exactly how it looked, what
improvements would you suggest?
4. What is the difference between a leading question and a loaded question?
5. Design one or more open-ended response questions to measure reactions to public
smartphone usage by others. What advantages and disadvantages does the open-ended format
provide?
6. What are some general guidelines for avoiding common mistakes in writing questionnaire
items?
Here are the general guidelines for avoiding common mistakes in writing questionnaire
items:
2) Avoid leading or loaded questions
4) Avoid double-barreled items
6) Avoid taxing respondents’ memory
7. Do you think the order of the questions below might affect responses? Explain.
© 2016 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, in whole or in
part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website for classroom use.
8. When might a researcher wish to randomize both question order and the order of
responses? Provide an example.
9. Define randomized response technique? Provide an example company that might
benefit from using the technique in its marketing research.
10. Design a short telephone survey interview form that measures what types of programs
residents of Peoria would like to see available through DIRECTV.
11. What advantages do Internet surveys offer in terms of order bias and survey flow?
The advantages of Internet surveys include:
2. Use of timers to detect “speeders”
4. Physical features that are not available in print surveys
12. Define the term survey “break–off.”
13. List at least three factors that increase nonresponse through break-offs.
Below are three factors that increase break-offs:
1. Questions that are irrelevant to a particular respondent
Chapter Eleven: Questionnaire Design
14. Provide an example where a filter-item is needed to implement branching in a survey.
Student answers may vary for this question, but the text provides a good example:
A filter question can serve as a branching mechanism directing respondents to an
appropriate part of the questionnaire using skip logic. Asking: “Where do you generally
have check-cashing problems in Springfield?” may elicit a response even though the
15. What is a status bar? When might a status bar be inappropriate?
16. The Apple Assistance Center exists to solve problems for users of MacBook, iPhone, and
iPad products. Design a text-message questionnaire that assesses users’ satisfaction with the
Apple Assistance Center.
17. [Ethics Question] Respondents have a duty to follow the survey instructions once they agree
to participate? Are Internet or personal surveys better for making sure respondents maintain
integrity in responding?
18. Visit pollseverywhere.com. What types of surveys may benefit from questions administered
via pollseverywhere.com?
19. Define pretesting. Pretests costs time and money. How should a researcher decide if further
pretesting is needed?
© 2016 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, in whole or in
part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website for classroom use.
20. What is back translation?
RESEARCH ACTIVITY
1. Search the Internet to find out a little about Nando’s Chicken Restaurant. Design a
Web-based survey with a five-item questionnaire assessing how consumers in your
town would react to a Nando’s location.
2. Using Qualtrics, design an eight-question Internet survey that assesses how effective the
introductory Marketing course at your school has been. Include a filter question that first asks
respondents whether they have completed the course and branch those that have not
completed it out of the survey.
3. Try to find two friends that know the same foreign language. Write ten Likert
questions that measure how exciting a retail store environment is to shop in. Have one
of your friends interpret the question into the foreign language. Have the other take
the translation and state each question in English. How similar is the translated
English to the original English? Comment.
4. Use the survey flow feature in Qualtrics to set up an assignment of subjects to questions that
ask each about their attitude toward either Instagram, Tumblr, Pinterest or Snapchat, which
includes randomization so that only each subject answers about only one of the social media
outlets.
© 2016 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, in whole or in
part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website for classroom use.
2. What might be an alternative approach?
3. How many questionnaire items would be needed if Sandy moved forward as suggested?
4. What type of branching might be useful in a survey of this type?