1
Chapter 08 – Designing the Questionnaire
Chapter 8
Designing the Questionnaire
Learning Objectives (PPT slide 8-2)
1. Describe the steps in questionnaire design.
2. Discuss the questionnaire development process.
Key Terms and Concepts
1. Bad questions
2. Call records
3. Common methods variance (CMV)
4. Cover letter
5. Interviewer instructions
6. Introductory section
Chapter Summary by Learning Objectives
Describe the steps in questionnaire design.
Researchers follow a systematic approach to designing questionnaires. The steps include: confirm
research objectives, select appropriate data collection method, develop questions and scaling,
2
Chapter 08 – Designing the Questionnaire
determine layout and evaluate questionnaire, obtain initial client approval, pretest, revise and
finalize questionnaire, and implement the survey.
Discuss the questionnaire development process.
A number of design considerations and rules of logic apply to the questionnaire development
process. The process requires knowledge of sampling plans, construct development, scale
measurement, and types of data. A questionnaire is a set of questions/scales designed to collect
data and generate information to help decision makers solve business problems. Good
Summarize the characteristics of good questionnaires.
Survey information requirements play a critical role in the development of questionnaires. For
each objective, the researcher must choose types of scale formats (nominal, ordinal, interval, or
ratio); question formats (open-ended and closed-ended); and the appropriate scaling. Researchers
Understand the role of cover letters.
The primary role of a cover letter should be to obtain the respondent’s cooperation and willingness
to participate in the project. Ten factors should be examined in developing cover letters. Observing
these guidelines will increase response rates.
Explain the importance of other documents used with questionnaires.
When data are collected using interviews, supervisor and interviewer instructions must be
developed as well as screening forms and call record sheets. These documents ensure the data
3
Chapter 08 – Designing the Questionnaire
interviews. Interviewer instructions are used to train interviewers to correctly select a prospective
respondent for inclusion in the study, screen prospective respondents for eligibility, and how to
properly conduct the actual interview. Screening forms are a set of preliminary questions used to
confirm the eligibility of a prospective respondent for inclusion in the survey. Quota sheets are
tracking forms that enable the interviewer to collect data from the right type of respondents. All of
these documents help improve data collection and accuracy.
Chapter Outline
Opening Vignette: Can Surveys Be Used to Develop University Residence Life Plans
The opening vignette in this chapter describes a survey instrument, “Residence Life” program,
developed to determine the housing needs of its current and future students at a large university
(43,000 students). The objectives of the program evolved around the idea that high-quality
on-campus living facilities and programs could help attract new students to the university. MPC
Consulting Group, Inc., which specializes in the assessment of on-campus housing programs,
prepared a self-administrated survey instrument to be delivered to existing students using the
I. Value of Questionnaires in Marketing Research (PPT slide 8-3)
Most surveys are designed to be descriptive or predictive. Descriptive research designs use
questionnaires to collect data that can be turned into knowledge about a person, object, or issue. In
contrast, predictive survey questionnaires require the researcher to collect a wider range of data
that can be used in predicting changes in attitudes and behaviors as well as in testing hypotheses.
4
Chapter 08 – Designing the Questionnaire
unchanged. Whether developing a survey to use online or offline, the steps researchers follow in
designing questionnaires are similar.
II. Pilot Studies and Pretests
A pilot study is a small-scale version of the intended main research study, including all the
subcomponents that make up the main study, including the data collection and analysis from about
III. Questionnaire Design (PPT slides 8-4 to 8-16)
Exhibit 8.1 lists the steps followed in developing survey questionnaires (PPT slide 8-4):
A. Step 1: Confirm Research Objectives (PPT slide 8-5)
In the initial phase of the development process, the research objectives are agreed upon by the
researcher and bank management.
B. Step 2: Select Appropriate Data Collection Method (PPT slide 8-6)
To select the data collection method, the researcher first must determine the data requirements
C. Step 3: Develop Questions and Scaling (PPT slides 8-7 to 8-9)
Questionnaire design is systematic and includes a series of logical activities. Researchers select
the appropriate scales and design the questionnaire format to meet the data requirements. The
researcher decides on the following:
• question format (unstructured or structured)
5
Chapter 08 – Designing the Questionnaire
Question format: Unstructured questions are open-ended questions that enable respondents to
reply in their own words. There is no predetermined list of responses available to aid or limit
respondents’ answers. Open-ended questions are more difficult to code for analysis. Perhaps
more importantly, these questions require more thinking and effort on the part of respondents.
As a result, with quantitative surveys there are generally only a few open-ended questions.
Unless the question is likely to be interesting to respondents, open-ended questions are often
skipped.
Structured questions are closed-ended questions that require the respondent to choose from a
Wording: Researchers must carefully consider the words used in creating questions and scales.
Ambiguous words and phrases as well as vocabulary that is difficult to understand must be
avoided. Researchers must select words carefully to make sure respondents are familiar with
them, and when unsure, questionable words should be examined in a pretest.
Words and phrases can influence a respondent’s answer to a given question.
Questions and scaling: Questions and scale format directly impact survey designs. To collect
accurate data researchers must devise good questions and select the correct type of scale.
Whenever possible, metric scales should be used. Also, researchers must be careful to maintain
consistency in scales and coding to minimize confusion among respondents in answering
questions.
Once a particular question or scale is selected, the researcher must ensure it is introduced
6
Chapter 08 – Designing the Questionnaire
properly and easy to respond to accurately. Bad questions prevent or distort communications
between the researcher and the respondent. If the respondent cannot answer a question in a
meaningful way, it is a bad question. Some examples of bad questions are those that are:
• Unanswerable either because the respondent does not have access to the information
needed or because none of the answer choices apply to the respondent.
When designing specific questions and scales, researchers should act as if they are two different
people: one thinking like a technical, systematic researcher and the other like a respondent. The
questions and scales must be presented in a logical order. After selecting a title for the
questionnaire, the researcher includes a brief introductory section and any general instructions
prior to asking the first question. Questions should be asked in a natural general-to-specific
order to reduce the potential for sequence bias. Also, any sensitive or more difficult questions
should be placed later in the questionnaire after the respondent becomes engaged in the process
of answering questions.
Prior to developing the layout for the survey questionnaire, the researcher should assess the
reliability and validity of the scales. Once this is completed, the focus is on preparing
instructions and making required revisions.
D. Step 4: Determine Layout and Evaluate Questionnaire (PPT slides 8-9 to 8-12)
In good questionnaire design, questions flow from general to more specific information, and
end with demographic data. Questionnaires begin with an introductory section that gives the
7
Chapter 08 – Designing the Questionnaire
respondent an overview of the research. The section begins with a statement to establish the
legitimacy of the questionnaire. Screening questions (also referred to as screeners or filter
questions) are used on most questionnaires. Their purpose is to identify qualified prospective
respondents and prevent unqualified respondents from being included in the study. It is difficult
to use screening questions in many self-administered questionnaires, except for
computer-assisted surveys. Screening questions are completed before the beginning of the main
portion of the questionnaire that includes the research questions. The introductory section also
includes general instructions for filling out the survey.
The last section includes demographic questions for the respondents. Demographic questions
are placed at the end of a questionnaire because they often ask personal information and many
people are reluctant to provide this information to strangers. Until the “comfort zone” is
established between the interviewer and respondent, asking personal questions could easily
bring the interviewing process to a halt. The questionnaire ends with a thank-you statement.
Some researchers have recently expressed concerns that questionnaire design might result in
common methods variance (CMV) being embedded in respondent data collected from surveys.
Common methods variance is biased variance that results from the measurement method used
in a questionnaire instead of the scales used to obtain the data. CMV is present in survey
8
Chapter 08 – Designing the Questionnaire
responses when the answers given by respondents to the independent and dependent variable
questions are falsely correlated. The bias introduced by CMV is most likely to occur when the
same respondent answers at the same time both independent and dependent variable questions
that are perceptual in nature, and the respondent recognizes a relationship between the two
types of variables.
A summary of the major considerations in questionnaire design is given in Exhibit 8.4.
The design of online surveys requires additional planning. A primary metric for traditional data
collection methods is response rate. To determine response rates the researcher must know the
number of attempts to contact respondents and complete a questionnaire. With mail or phone
surveys this task is relatively easy. With online surveys researchers must work with the online
data collection field service to plan how respondents will be solicited. If the data collection
service sends out a “blanket” invitation to complete a survey there is no way to measure
response rates. Even if the invitation to complete a survey is sent to an organized panel of
respondents, calculating response rates can be a problem. This is because panels may involve
millions of individuals with broad criteria describing them. Another issue is recruiting of
participants. If individuals are invited to participate and they decline, should they be included in
the response rate metric? Or should the response rate metric be based on just those individuals
9
Chapter 08 – Designing the Questionnaire
Another problem with online surveys is the length of time it takes some respondents to
complete the survey. To deal with this issue with online data collection vendors, first make sure
“time for completion” is a metric that is measured.
Research firms are just beginning to analyze online questionnaire design issues. Following
three issues have been addressed to date.
• The effect of response box size on length of answer in open-ended
questions—respondents will write more when boxes are bigger. Some research firms
cope with this by offering respondents the choice between small, medium, and large
boxes so that the box size does not affect their response.
Online formats facilitate the use of improved rating and ranking scales, as well as extensive
graphics and animation. Programming surveys in the online environment offers the opportunity
to use graphics and scales in new ways, some of which are helpful and some of which are not.
As with traditional data collection methods and questionnaire design, the best guideline is the
KISS test—Keep It Simple and Short! Complex formats or designs can produce biased findings
and should be avoided.
E. Step 5: Obtain Initial Client Approval (PPT slide 8-13)
Copies of the questionnaire should be given to all parties involved in the project. This is the
client’s opportunity to provide suggestions of topics overlooked or to ask any questions.
F. Step 6: Pretest, Revise, and Finalize the Questionnaire (PPT slides 8-14 and 8-15)
The final version of the questionnaire is evaluated using a pretest. In the pretest, the survey
questionnaire is given to a small, representative group of respondents that are asked to fill out
the survey and provide feedback to researchers. The number of respondents is most often
10
Chapter 08 – Designing the Questionnaire
answer for all questions within a group. The pretest helps the researcher determine how much
time respondents will need to complete the survey, whether to add or revise instructions, and
what to say in the cover letter.
G. Step 7: Implement the Survey (PPT slide 8-16)
The focus here is on the process followed to collect the data using the agreed-upon
questionnaire. The process varies depending on whether the survey is self-administered or
IV. The Role of a Cover Letter (PPT slides 8-17 and 8-18)
A cover letter is used with a self-administered questionnaire. The primary role of the cover letter
is to obtain the respondent’s cooperation and willingness to participate in the research project.
With personal or telephone interviews, interviewers use a verbal statement that includes many of
the points covered by a mailed or drop-off cover letter. Self-administered surveys often have low
response rates (25 percent or less). Good cover letters increase response rates. Exhibit 8.5 provides
guidelines for developing cover letters (PPT slide 8-18).
• Personalization
• Identification of the organization
• Clear statement of the study’s purpose and importance
• Anonymity and confidentiality
11
Chapter 08 – Designing the Questionnaire
IV. Other Considerations in Collecting Data (PPT slides 8-19 to 8-21)
When data are collected, supervisor and interviewer instructions often must be developed as well
as screening questions and call records. These mechanisms ensure the data collection process is
successful. Some of these are needed for interviews and others for self-administered
questionnaires.
A. Supervisor Instructions (PPT slide 8-19)
A supervisor instruction form serves as a blueprint for training people on how to execute the
interviewing process in a standardized fashion; it outlines the process by which to conduct a
study that uses personal and telephone interviewers. They include the following:
• detailed information on the nature of the study
• start and completion dates
Exhibit 8.6 displays a sample page from a set of supervisor instructions for a restaurant study.
B. Interviewer Instructions (PPT slide 8-19)
Interviewer instructions are used for training interviewers to correctly select a prospective
respondent for inclusion in the study, screen prospective respondents for eligibility, and to
properly conduct the actual interview. The instructions include the following:
• detailed information about the nature of the study
• start and completion dates
• sampling instructions
• screening procedures
12
Chapter 08 – Designing the Questionnaire
• verification form procedures
C. Screening Questions (PPT slide 8-20)
Screening questions ensure the respondents included in a study are representative of the defined
target population. Screening questions are used to confirm the eligibility of a prospective
D. Quotas (PPT slide 8-20)
Quota is a tracking system that collects data from respondents and helps ensure that subgroups
are represented in the sample as specified. When a particular quota for a subgroup of
respondents is filled, questionnaires for that subgroup are no longer completed. If interviewers
E. Call or Contact Records (PPT slide 8-21)
Call records, also referred to as either reporting or tracking approaches, are used to estimate
the efficiency of interviewing. These records typically collect information on the number of
attempts to contact potential respondents made by each interviewer and the results of those
attempts. Call and contact records are often used in data collection methods that require the use
of an interviewer, but also can be used with online surveys. Information gathered from contact
records include the following:
• number of calls or contacts made per hour
• number of contacts per completed interview
• length of time of the interview
Marketing Research in Action
Designing a Questionnaire to Survey Santa Fe Grill Customers (PPT slides 8-22 and 8-23)
13
Chapter 08 – Designing the Questionnaire
The Marketing Research in Action in this chapter extends the discussion on questionnaire design.
It includes actual Screening Questions (Exhibit 8.7) and a Questionnaire (Exhibit 8.8) for Santa Fe
Grill. The research objectives guiding the survey are provided.
Answers to Hands-On Exercise
1. Based on the research objectives, does the self-administered questionnaire, in its current
form, correctly illustrate sound questionnaire design principles? Please explain why or why
not.
No, the self-administered questionnaire does not correctly illustrate the sound questionnaire
design principles. It does not flow from the general to the specific. A better order might be
Selection Factors, Perception Measures, Relationship Measures, Life Style Questions, and
2. Overall, is the current survey design able to capture the required data needed to address all
the stated research objectives? Why, or why not? If changes are needed, how would you
change the survey’s design?
Keeping the six research objectives in mind, Santa Fe Grill could try to bring the following
changes in its survey design.
(i) To identify the factors people consider important in making casual dining restaurant
choice decisions
The restaurant decision factors are fairly well covered. It is a good idea to include an
(ii) To determine the characteristics customers use to describe the Santa Fe Grill and its
competitor, Jose’s Southwestern Café
Customers’ description of the Santa Fe Grill is covered, but there is still the question
concerning coverage of the important factors. Are all possible descriptors included in
14
Chapter 08 – Designing the Questionnaire
the questionnaire? Which descriptors do customers find most attractive?
(iii) To develop a psychographic/demographic profile of the restaurant customers
The relevant demographic information seems to be pretty well covered. One might
question if the most relevant psychographic characteristics and statement were
included, as there were only 11 statements included in the questionnaire (Section 1).
(iv) To determine the patronage and positive word-of-mouth advertising patterns of the
restaurant customers
(v) To assess the customer’s willingness to return to the restaurant in the future
Likelihood of return is covered in one of the questions.
(vi) To assess the degree to which customers are satisfied with their Mexican restaurant
experiences
There is just one question on satisfaction with an open-ended follow-up for people
3. Evaluate the “screener” used to qualify the respondents. Are there any changes needed? Why,
or why not?
The first screener is fine. It gets right to the characteristic of interest. The second screener
about having eaten at some other Mexican restaurant isn’t particularly useful. Why is it
4. Redesign questions # 26 to 29 on the survey using a rating scale that will enable you to
obtain the “degree of importance” a customer might attach to each of the four listed
attributes in selecting a restaurant to dine at.
Listed below are some reasons that many people might use in selecting a restaurant where
15
Chapter 08 – Designing the Questionnaire
they want to dine. Think about your visits to casual dining restaurants in the last three
months. For each attribute listed below, please indicate by circling the number which
Attribute
1
2
3
4
5
Price
1
2
3
4
5
Food quality
1
2
3
4
5
Atmosphere
1
2
3
4
5
Service
1
2
3
4
5
Answers to Review Questions
1. Discuss the advantages and disadvantages of using unstructured (open-ended) and structured
(closed-ended) questions in developing an online, self-administered survey instrument.
The type of question format (structured or unstructured) has a direct impact on survey design.
The advantage of using unstructured questions is that they provide a rich array of
information to the researcher, since there are no predetermined responses available to aid (or
2. Explain the role of a questionnaire in the research process. What should be the role of the
client during the questionnaire development process?
A questionnaire is an essential tool in the research process, especially if a secondary data
scan fails to address the research question and decision problem with precision and clarity.
Questionnaires capture the new (primary) information from respondents. Clearly, as with
16
Chapter 08 – Designing the Questionnaire
3. What are the guidelines for deciding the format and layout of a questionnaire?
Questions and decisions about “format” and “layout” are directly related to function; for
example, constructing a survey which captures primary information from the target
population to address business or marketing information problems. Questionnaire
4. What makes a question bad? Develop three examples of bad questions. Rewrite your
examples so they could be judged as good questions.
Bad questions are defined as queries which cause a fundamental “disconnect” between
researcher and respondent. They can be unanswerable, leading (or loaded), and/or
double-barreled. Listed below are three examples of “bad” questions and the remedies which
flow from the guidelines presented.
(i) “What do you think of the exceptionally poor customer service provided by the
circulation desk at our campus library?” This is a leading or loaded question. The phrase
“exceptionally poor” should be eliminated.
5. Discuss the value of a good questionnaire design.
A good questionnaire increases the probability of collecting high-quality primary data that
can be transformed into reliable and valid information. The layout of the instrument can
6. Discuss the main benefits of including a brief introductory section in questionnaires.
If the questionnaire is intended to be self-administered, it is necessary to explain to
respondents the main objectives of the research. They also need some general instructions
7. Unless needed for screening purposes, why shouldn’t demographic questions be asked up
front in questionnaire?
Gathering demographic data is not the main purpose of the survey, so it should not be
covered in the main part of the questionnaire. Because demographic information is
Answers to Discussion Questions
1. Assume you are doing exploratory research to find out students’ opinions about the purchase
of a new digital music player. What information would you need to collect? What types of
questions would you use? Suggest six to eight questions you would ask, and indicate in what
sequence you would ask them. Pretest the questions on a sample of students from your class.
18
Chapter 08 – Designing the Questionnaire
2. Assume you are conducting a study to determine the importance of brand names and features
of mobile phone handsets. What types of questions would you use—open-ended,
closed-ended, or scaled—and why? Suggest six to eight questions you would ask and in
what sequence you would ask them. Pretest the questions on a sample of students from your
class.
3. Discuss the guidelines for developing cover letters. What are some of the advantages of
developing good cover letters? What are some of the costs of a bad cover letter?
The following are the guidelines for developing cover letters.
(i) Personalization of the cover letter
(ii) Identification of the organization
(iii) Clear statement of the study’s purpose and importance
The advantages associated with a “good” cover letter are numerous, but the primary one is
the ability of a cover letter to act as a behavioral incentive to get the respondent to complete
and return the survey in a manner which is aligned with the “deadlines” associated with
capturing data agreed upon by the research team and decision makers.
4. Using the questions asked in evaluating any questionnaire design (see Exhibit 8.4), evaluate
the Santa Fe Grill restaurant questionnaire. Write a one-page assessment.
Students’ responses will vary, but all should follow the considerations given in Exhibit 8.4.
19
Chapter 08 – Designing the Questionnaire
These are as follows:
• Are the questions in the questionnaire appropriate and complete for addressing the
research objectives?
• Will the questions as designed provide the data in a sufficient form to address each
objective?
• Does the introduction section include a general description of the study?
• Are the instructions clear?
• Do the questions and scale measurements follow a logical order?
5. What are the critical issues involved in pretesting a questionnaire?
It is crucial to pretest a questionnaire since no amount of input from the research team and
client can ever be considered “final” until the instrument is subjected to a first “litmus test”;
for example, getting it in front of the target population who will eventually be asked to fill it
out in a more expensive and time-consuming fashion. The critical issue involved in