Chapter 15
Questionnaire Design
AT-A-GLANCE
I. Questionnaire Quality and Design: Basic Considerations
A. What should be asked?
B. Questionnaire relevancy
C. Questionnaire accuracy
II. Wording Questions
A. Open-ended response versus fixed-alternative questions
Using open-ended response questions
Using fixed-alternative questions
B. Types of fixed-alternative questions
C. Phrasing questions for self-administered, telephone, and personal interview surveys
III. Guidelines for Constructing Question
A. Avoid complexity: Use simple, conversational language
B. Avoid leading and loaded questions
C. Avoid ambiguity: Be as specific as possible
D. Avoid double-barreled items
E. Avoid making assumptions
F. Avoid burdensome questions that may tax the respondent’s memory
G. Make certain questions generate variance
IV. What Is the Best Question Sequence?
V. What Is the Best Layout?
A. Traditional questionnaires
B. Internet questionnaires
Layout issues
Software that makes questionnaires interactive
VI. How Much Pretesting and Revising Are Necessary?
VII. Designing Questionnaires for Global Markets
IX. Appendix 15A: Question Wording and Measurement Scales for Commonly Researched Topics
LEARNING OUTCOMES
1. Explain the significance of decisions about questionnaire design and wording
2. Define alternatives for wording open-ended and fixed-alternative questions
3. Summarize guidelines for questions that avoid mistakes in questionnaire design
4. Describe how the proper sequence of questions may improve a questionnaire
5. Discuss how to design a questionnaire layout
6. Describe criteria for pretesting and revising a questionnaire and for adapting it to global markets
CHAPTER VIGNETTE: J.D. Power Asks: It’s Interesting, But Do You
Really Want It?
As car makers consider adding new features, they have to evaluate not only which ones appeal to
consumers but also which ones will actually sell, considering their likely cost. In a J.D. Power survey,
about 17,000 consumers were asked whether they were familiar with 22 different emerging technologies
and their interest in each technology. Next, the study indicated the likely price of each technology, and
consumers were asked their interest, given the price. Learning price information often changed
consumers’ interest levels, but a few features remained in the top five (run-flat tires and stability control).
Automakers can use findings such as these to determine which features are price-sensitive and which
might be appealing even at a higher price.
SURVEY THIS!
Students are asked to critique the survey instrument from a standpoint of the learning objectives of this
chapter. Comment on the order of questions or on the presence of any leading or double-barreled items.
What are the strengths and weaknesses of the questionnaire?
RESEARCH SNAPSHOTS
Corporate Reputations: Consumers Put Johnson & Johnson, Microsoft, and Google on Top
Since 1999, the Wall Street Journal has sponsored an annual research project, the Corporate
Reputation Survey, to see what U.S. adults think about corporations. The study has two phases.
In the first phase, researchers identified the most “visible” companies by using open-ended
questions asking respondents to name two companies they felt had the best reputation and two
that had the worst. The researchers selected the 60 named most often for the second phase of the
study. The second phase was aimed at generating rankings of the corporations, so the researchers
identified six dimensions of corporate reputation: products and services, financial performance,
workplace environment, social responsibility, vision and leadership, and emotional appeal.
Within each dimension, 20 attributes were identified, and respondents in an online survey were
asked to rate one company an all 20 attributes. The top-ranked company for each of the first
seven years was Johnson & Johnson. J&J also was the firm from which the largest share of
people said they would “definitely purchase” products.
What Citizens (Don’t) Know about Climate Change
Climate change as a result of global warming has frequently been featured in the news, and
scientists at the Commonwealth Scientific and Industrial Research Organization in Australia
conducted an online survey of 5,000 Australians to learn their views on climate change and the
degree to which they engaged in behaviors that are environmentally-friendly. The researchers
found that there was considerable confusion about what Australians believe is the underlying
cause of climate change. There appeared to be clear differences between those who believe that
climate change is happening, and is simply a naturally occurring cycle of the Earth’s climate,
versus those that believe that climate change is human-induced. The results suggest that further
research on human attitudes about climate change is necessary.
Pretesting the CAHPS Hospital Survey
The federal government’s Centers for Medicare and Medicaid Services (CMS) is supposed to
make information about hospital performance available to the public, so the CMS spent several
years creating and modifying a questionnaire, the Consumer Assessment of Health Providers and
Systems (CSHPS) Hospital Survey. The researchers put it through extensive pretesting. The first
version (68 questions) was given to a sample of 18 individuals. Based on their reactions, the
survey was modified and tested on 13 more people, resulting in 66 items. Next, the 66-item
survey was pilot tested with almost 50,000 patients in hospitals in three states, and the
questionnaire was reduced to 32 items based on the results. It was tested at several more
hospitals and reviewed by the National Quality Forum, resulting in a 27-item survey that was
ready for use nationwide.
OUTLINE
I. QUESTIONNAIRE QUALITY AND DESIGN: BASIC CONSIDERATIONS
A research survey is only as good as the questions it asks.
Questionnaire design is one of the most critical stages in the survey research process.
Good questionnaire design requires far more that correct grammar.
To fulfill a researcher’s purposes, the questions must meet the basic criteria of relevance and
accuracy.
To achieve these ends, the decisions in the following order must be made:
1. What should be asked?
2. How should questions be phrased?
3. In what sequence should the questions be arranged?
4. What questionnaire layout will best serve the research objectives?
5. How should the questionnaire be pretested? Does the questionnaire need to be revised?
What Should Be Asked?
The specific questions to be asked will be a function of the previous decisions.
The later stages of the research process will have an important impact on the
questionnaire wording.
When designing the questionnaire, the researcher must also be thinking about the types of
statistical analyses that will be conducted.
Questionnaire Relevancy
A questionnaire is relevant to the extent that all information collected addresses a
research question that will help the decision maker address the current business
problem.
The researcher should be specific about data needs and have a rationale for each item
requesting information.
Irrelevant questions are more than a nuisance because they make the survey
needlessly long.
When planning the questionnaire design, researchers must think about possible
omissions.
Questionnaire Accuracy
Accuracy means that the information is reliable and valid.
While it is generally believed that one should use simple, understandable, unbiased,
unambiguous, and nonirritating words, no step-by-step procedure can be generalized.
Respondents tend to be most cooperative when the subject of the research is
interesting.
If questions are lengthy, difficult to answer, or ego threatening, there is a high
probability of biased answers.
Question wording and sequence substantially influence accuracy.
II. WORDING QUESTIONS
There are many ways to phrase questions, and many standard question formats have been
developed in previous research.
Open-Ended Response versus Fixed-Alternative Questions
We may categorize two basic types of questions asked on the amount of freedom
respondents are given in answering.
Open-ended response questions pose some problem or topic and ask respondents to
answer in their own words.
Open-ended response questions are free-answer questions.
In contrast, fixed-alternative questions (sometimes called closed-ended questions) give
respondents specific limited-alternative responses and ask them to choose the one closest
to their own viewpoint.
Using Open-Ended Response Questions
Open-ended response questions are most beneficial when the researcher is
conducting exploratory research.
They may also be useful at the beginning of an interview as they allow the
respondent to warm up to the questioning process.
The cost of open-ended response questions is substantially higher than that of
fixed-alternative questions because the job of coding, editing, and analyzing the data
is quite extensive.
Another potential disadvantage of open-ended response questions is the possibility of
interviewer bias to influence the answer; even the best interviewer can take shortcuts
in recording answers.
Also, articulate individuals tend to give longer answers and such respondents often
are better educated and from higher income groups, which may not be representative
of the entire population but gives them a large share of the responses.
Using Fixed-Alternative Questions
Fixed-alternative questions require less interviewer skill, take less time, and are
easier for the respondent to answer.
Answers to closed questions are classified into standardized groupings prior to data
collection, which provides comparability of answers and facilitates coding,
tabulating, and interpreting the data.
However, when a researcher is unaware of the potential responses to a question,
fixed-alternative questions cannot be used.
If the researcher assumes the responses and is wrong, he or she will have no way of
knowing the extent to which the assumption was incorrect.
Unanticipated alternatives emerge when respondents believe that closed answers do
not adequately reflect their feelings, causing them to make comments to the
interviewer or write additional answers on the questionnaire.
Respondents may check off obvious alternatives if they do not see the choice they
would prefer.
May tempt respondents to check an answer that is more prestigious or socially
acceptable than the true answer.
Most questionnaires mix open-ended and closed questions, providing a change of pace
that can eliminate respondent boredom and fatigue.
Types of Fixed-Alternative Questions
Simple-dichotomy (dichotomous) questions – require the respondent to choose one of
two alternatives. The answer can be a simple “yes” or “no” or a choice between “this”
and “that.”
Several types of questions provide the respondent with multi-choice alternatives:
The determinant-choice question requires the respondent to choose one–and only
one—response from among several possible alternatives.
The frequency-determination question is a determinant-choice question that asks
for an answer about the general frequency of occurrence.
Attitude rating scales, such as the Likert scale, semantic differential, Stapel scale, and
so on, are also fixed-alternative questions and were discussed in Chapter 14.
The checklist question allows respondents to provide multiple answers to a single
question. In many cases, the choices are adjectives that describe a particular object.
Alternatives should be totally exhaustive, meaning that all the response options are
covered and that every respondent has an alternative to check.
There should be no overlap among categories in the checklist—each alternative
should be mutually exclusive, that is, only one dimension of an issue should be
related to that alternative.
Phrasing Questions for Self-Administered, Telephone, and Personal Interview Surveys
The means of data collection—telephone interview, personal interview, self-administered
questionnaire—will influence the question format and question phrasing.
In general, questions for mail, Internet, and telephone surveys must be less complex than
those used in personal interviews.
Questionnaires for telephone and personal interviews should be written in a
conversational style.
Exhibit 15.1 illustrates how a question may be revised for a different medium.
III. GUIDELINES FOR CONSTRUCTING QUESTIONS
In developing a questionnaire, there are no hard-and-fast-rules.
Some guidelines have been developed to avoid the most common mistakes.
Avoid Complexity: Use Simple, Conversational Language
Words used in questionnaires should be readily understandable to all respondents.
The technical jargon of top executives should be avoided (e.g., “brand image,” “marginal
analysis,” etc.).
Avoid Leading and Loaded Questions
Leading and loaded questions are a major source of bias in question wording.
Leading questions suggest or imply certain answers.
Such questions may result in a “bandwagon effect” which threatens the study’s
validity.
Partial mention of alternatives is a variation of this phenomenon.
Loaded questions suggest a socially desirable answer or are emotionally charged.
Certain answers to questions are more socially desirable than others.
Invoking the status quo is a form of loading that results in bias because the
majority of people tend to be resistant to change.
Asking respondents “how often” leads them to generalize about their behavior and
one is more likely to portray one’s ideal behavior rather than one’s average behavior.
An introductory counterbiasing statement or preamble to a question that
reassures respondents that their “embarrassing” behavior is not abnormal may
help yield truthful responses. Also, an assurance of anonymity may help elicit
honest responses to embarrassing questions.
A question statement may be leading because it is phrased to reflect either the
negative or positive aspects of an issue.
To control for this bias, split-ballot technique, which reverses the wording of
attitudinal questions for 50 percent of the sample, can be used.
Avoid Ambiguity: Be as Specific as Possible
Items on questionnaires are often ambiguous because they are too general.
Indefinite words such as frequently, occasionally, often, etc., have many different
meanings.
Avoid Double-Barreled Items
A question covering several items at once is referred to as a double-barreled
question and should always be avoided.
The results may be exceedingly difficult to interpret.
Avoid Making Assumptions
The researcher should not place the respondent in a bind by including an implicit
assumption in the question.
Another frequent mistake is assuming that the respondent had previously thought
about an issue; research that induces people to express attitudes on subjects that they
do not ordinarily think about is meaningless.
Avoid Burdensome Questions That May Tax the Respondent’s Memory
In many situations respondents cannot recall the answer to a question.
Questions that give no clue as to the brand of interest are referred to as unaided recall
questions.
Often researchers will devise aided recall questions that provide a clue to jog the
respondent’s memory.
While aided recall is not as strong a test of attention or memory as unaided recall, it
is less taxing to the respondent’s memory.
Telescoping occurs when respondents recall past events as more recent than they
actually were.
Squishing occurs when respondents think that recent events took place longer ago
than they really did.
In situations in which “I don’t know” or “I can’t recall” is a meaningful answer,
simply including a “don’t know” response category may solve the question writer’s
problem.
Make Certain Questions Generate Variance
We want our variables to vary!
The response categories provided must cover the breadth of possibilities (totally
exhaustive) but also yield variance across respondents.
If all the respondents check the same box, we have not generated usable information.
IV. WHAT IS THE BEST QUESTION SEQUENCE?
The order of questions may serve several functions for the researcher.
For example, if respondents’ curiosity is not aroused at the outset, they can become
disinterested and terminate the interview.
It is generally not advisable to ask demographic or classificatory questions (i.e., education,
income, etc.) at the beginning because it may embarrass or threaten respondents.
Order bias can result from an alternative answer’s position in a set of answers or from the
sequencing of questions.
Order bias can distort survey results.
Asking specific questions before asking about broader issues is a common cause of order
bias.
It is advisable to ask the general questions before the specific questions to obtain the freest of
open-ended responses.
This technique is known as the funnel technique, and it allows researchers to understand
the respondent’s frame of reference before asking more specific questions about the level
of the respondent’s information and intensity of opinion.
With attitude scales, there also may be an anchoring effect.
The first concept measured tends to become a comparison point from which subsequent
evaluations are made.
Randomization of these items on a questionnaire helps to minimize this order bias.
A related problem is bias caused by the order of alternatives on closed questions.
The order of these choices should be rotated if producing alternative forms of the
questionnaire is possible.
Unfortunately, researchers rarely print alternative questionnaires to eliminate problems
arising from order bias.
One advantage of Internet surveys is the ability to reduce order bias by having the computer
randomly order questions and/or response alternatives.
Filter questions minimize the asking of questions that may be inapplicable
Another form of filter question, the pivot question, can be used to obtain information that the
respondent may be reluctant to provide.
V. WHAT IS THE BEST LAYOUT?
The layout and attractiveness of the questionnaire are of crucial importance in mail,
Internet, and other self-administered questionnaires.
For different reasons, a good layout in questionnaires designed for personal and telephone
interviews is also important.
Traditional Questionnaires
In mail questionnaires, often the rate of return can be improved by adding the money
that might have been used as an incentive to improve the attractiveness and quality of
the questionnaire instead.
Questionnaires should be designed to appear as short as possible.
The multiple-grid question presents several similar questions and corresponding
response alternatives arranged in a grid format.
The title of a questionnaire should be phrased carefully to:
capture the respondent’s interest
underline the importance of the research
emphasize the interesting nature of the study
appeal to the respondent’s ego
emphasize the confidential nature of the study
Instructions are often capitalized or printed in bold to alert the interviewer that it may
be necessary to proceed in a certain way.
Skip question – either skip instructions or an arrow drawn pointing to the next
question informs the respondent which question comes next.
Layout is extremely important when questionnaires are long or require the respondent
to fill in a large amount of information.
Internet Questionnaires
Questionnaires on a website should be easy to use, flow logically, and have a graphic
look and overall feel that motivate the respondent to cooperate from start to finish.
With graphical user interface (GUI) software, the researcher can exercise control over
the background colors, fonts, and other visual features displayed on the screen so as
to create an attractive and easy-to-use interface between the computer user and the
Internet survey.
GUI software allows respondents to click on the appropriate answer rather than
having to type answers or codes.
The possibility that the questionnaire the researcher/designer constructs on his or her
computer may look different from the one that appears on the respondent’s computer
should always be considered when designing Internet surveys.
One solution is to limit the horizontal width of the questions to 70 characters or less.
Layout Issues
The first decision is whether the questionnaire will appear page by page, with
individual questions on separate screens, or on a scrolling basis, with the entire
questionnaire appearing on a single web page that the respondent scrolls from top
to bottom.
The paging layout (going from screen to screen) greatly facilitates skip patterns.
The scrolling layout gives the respondent the ability to read any portion of the
questionnaire at any time, but the absence of page boundaries can cause
problems.
Push buttons with labels should clearly describe the actions to be taken.
Decisions must be made about the use of color, graphics, animation, sound, and
other special features that the Internet makes possible.
Because many Internet surveys offer no visual clues about the number of
questions to be asked, it is important to provide a status bar or some other visual
indicator of questionnaire length.
An Internet questionnaire uses windows known as dialog boxes to display
questions and record answers.
Four common ways of displaying questions on a computer screen (see Exhibit
15.8):
Radio buttons – clicking on an alternative response deactivates the first
choice and replaces it with the new response (i.e., like push buttons on an
automobile radio).
Drop-down box – a space-saving device that allows the researcher to
provide a list of responses that are hidden from view until they are
needed.
Checklist questions may be followed by check boxes, several, none, or
all of which may be checked by the respondent.
Open-ended boxes are boxes in which respondents type their answers to
open-ended questions, and they may be designated as one-line text boxes
or scrolling text boxes depending on the breadth of the expected answer.
Pop-up boxes are message boxes that can be used to highlight important
information.
Software That Makes Questionnaires Interactive
Internet software allows the branching off of questioning into two or more
different lines, depending on a particular respondent’s answer, and the skipping
or filtering of questions.
Most of these programs have hidden skip logic so that respondents never see any
evidence of skips.
Variable piping software allows variables, such as answers from previous
questions, to be inserted into unfolding questions.
Other software can randomly rotate the order of questions, blocks of questions,
and response alternatives.
Software can be used to control the flow of a questionnaire (e.g., preventing
respondents from backing up or allowing them to stop and return later).
Error trapping – error message appears if the respondent fails to answer a
question or answers it with an incorrect type of response.
Forced answering software – program will not let respondents continue if they
fail to answer a question.
Some designers include interactive help desks in their web questionnaires so
that respondents can solve problems they encounter.
Respondents leaving the questionnaire website, prematurely terminating the
survey may receive an e-mail message to encourage them to revisit the site and
complete the questionnaire, often picking up where they left off.
VI. HOW MUCH PRETESTING AND REVISING ARE NECESSARY?
Usually, the questionnaire is tried out on a group, selected on a convenience basis, that is
similar in makeup to the one that ultimately will be sampled.
It is not necessary to get a statistical sample for pretesting.
Pretesting allows the researcher to determine if the respondents have any difficulty
understanding the questionnaire and whether there are any ambiguous or biased questions.
This process can save the potential disaster of administering an invalid questionnaire to
several hundred individuals.
A preliminary tabulation of the pretest results often illustrates that while a question is easily
comprehended and answered by the respondent, it is inappropriate because it is does not solve
the problem.
Administering a questionnaire exactly as planned in the actual study often is not possible.
Interviewers are often used for pretest work because they can record requests for additional
explanation or comments that indicate respondents’ difficulty with question sequence or other
factors.
Pretests are typically conducted to answer questions about the questionnaire such as:
Can the questionnaire format be followed by the interviewer?
Does the questionnaire flow naturally and conversationally?
Are the questions clear and easy to understand?
Can respondents answer the questions easily?
Which alternative forms of questions work best?
Pretests also provide the means for testing the sampling procedure and estimates for the
response rates for the mail surveys and completion rates for telephone surveys.
VII. DESIGNING QUESTIONNAIRES FOR GLOBAL MARKETS
International researchers must take cultural factors into account when designing
questionnaires.
The most common problem involves translation into another language.
Back translation is the process of translating the questionnaire from one language to another
and then having it translated back again by a second, independent translator. The back
translator is often a person whose native tongue is the language that will be used on the
questionnaire.
Literacy influences the designs of self-administered questionnaires and interviews, so
knowledge of the literacy rates in foreign countries is vital.
IX. APPENDIX 15A: QUESTION WORDING AND MEASUREMENT SCALES FOR
COMMONLY RESEARCHED TOPICS
This appendix provides a bank of questions and scales to provide easy reference for satisfying
traditional research objectives.