CHAPTER 7
EVALUATING SURVEY DATA COLLECTION METHODS
LEARNING OBJECTIVES
In this chapter you will learn:
7-1 To learn the four basic alternative modes for gathering survey data
7-2 To understand the advantages and disadvantages of each of the various data
collection modes
7-3 To become knowledgeable about the details of different types of survey data
collection methods, such as personal interviews, telephone interviews, and computer-
assisted interviews, including online surveys
7-4 To acquire knowledge of how online panels greatly expedite and improve data
collection
7-5 To comprehend the factors researchers consider when choosing a particular survey
method
CHAPTER OUTLINE
Advantages of Surveys
• Experience the Advantages of Surveys
o Standardization
o Ease of analysis
o Ease of administration
o Get beneath the surface
o Subgroup differences
Modes of Data Collection
• The Data Collection Dilemma and Impact of Technology
• Data Collection and Computer Technology
• Person-Administered Surveys
o Advantages of person-administered surveys
▪ Feedback
▪ Rapport
▪ Quality control
▪ Adaptability
o Disadvantages of person-administered surveys
▪ Humans make errors
▪ Slow speed
▪ High cost
▪ Fear of interview evaluation
• Computer-Assisted Surveys
o Advantages of computer-assisted surveys
▪ Speed
▪ Relatively error-free interviews
▪ Use of pictures, audio-visuals, and graphics
▪ Immediate capture of data
o Disadvantages of Computer-Assisted Surveys
▪ Technical skills may be required
▪ Setup costs can be high
• Self-Administered Survey
o Advantages of Self-Administered Surveys
▪ Reduced cost
▪ Respondent control
▪ Reduced interview evaluation apprehension
o Disadvantages of Self-Administered Surveys
▪ Respondent control
▪ Lack of monitoring
▪ High questionnaire requirements
• Computer-Administered Surveys
o Advantages of Computer-Administered Surveys
▪ Many user-friendly features
▪ Relatively inexpensive
▪ Reduction of interview evaluation concern in respondents
o Disadvantage of Computer-Administered Surveys
▪ Require computer literate and Internet-connected respondents
• Mixed-Mode Surveys
o Advantage of Mixed-Mode Surveys
▪ Multiple avenues to achieve data collection goal
o Disadvantage of Mixed-Mode Surveys
▪ The survey mode may affect response
▪ Additional complexity
Descriptions of Data Collection Methods
• Person-Administered Interviews /Computer-Assisted Interviews
o In-home surveys
o Mall-intercept surveys
o In-office surveys
o Telephone surveys
• Computer-Administered Interviews
o Fully automated survey
o Online surveys
• Self-Administered Surveys
o Group self-administered survey
o Drop-off survey
o Mail survey
Working with a Panel Company
• Advantages of Using a Panel Company
o Fast turnaround
o High quality
o Database information
o Access to targeted respondents
o Integrated features
• Disadvantages of Using a Panel Company
o Not random samples
o Overused respondents
o Cost
• Top Panel Companies
Choice of the Survey Method
• How Fast Is the Data Collection?
• How Much Does the Data Collection Cost?
• How Good Is the Data Quality?
• Other Considerations
KEY TERMS
Survey Person-administered survey
Computer-assisted survey Self-administered survey
Computer-administered survey Interview evaluation
Mixed-Mode survey In-home survey
Mall-intercept survey In-office survey
Central location telephone survey Fully automated survey
Computer-assisted telephone interviews (CATI)
Completely automated telephone survey (CATS)
Internet-based questionnaire Group self-administered survey
Drop-off survey Mail survey
Nonresponse Self-selection bias
Incidence rate Panel company
TEACHING SUGGESTIONS
1. A field trip to a local telephone interview company does a great deal in the way of
illustrating how this survey mode operates. If there is an interview service in your
city, consider such a field trip, or if this is not feasible, have a representative from the
company speak to the class about the company’s operations.
2. It is possible to illustrate self-selection bias in students with a quick exercise. Give
students the following set of topics and have them indicate how they would respond
to a mail survey request in each case. Use a show of hands or actual count of the
responses to illustrate that some topics are inherently interesting and garner more
response than uninteresting topics.
Instructions. You pick up your mail, and you find a mail survey in it. How likely
would you be to take 15 minutes to fill out the questionnaire and return it if it
concerned each of the following topics?
Would you fill out the survey and send it back?
Topic Definitely Would Definitely Would Unsure
Not
Automobile maintenance _________ __________ ____ ___
Dental hygiene _________ __________ _____ __
Personal safety on campus _________ __________ ____ ___
Baby care _________ __________ ____ ___
Recycling _________ __________ ____ ___
Foreign trade with Chile _________ _________ ____ ___
3. A large number of studies have addressed mail survey response rates. Have each
student select a single incentive tactic such as advance postcard or monetary
incentive, and perform research on it, ideally with some database search service
maintained by your university library or otherwise available to your students. Instruct
students to prepare a 3-5 minute summary class presentation on their topics.
(1) a cat, (2) a Ford pickup truck, (3) a digital video camera, (4) a riding mower, (5)
5. Ask students what creative approaches can be taken when a researcher requires a
large amount of information. Suppose, for instance, that a researcher is investigating
attitudes toward the preservation of natural habitats, and the survey includes a battery
of life style questions, many attitude questions on wildlife and habitat preservation,
extensive demographics, plus many opinion questions on specific state parks, national
parks, public lands, and recreational areas. The questionnaire takes 60 minutes to
complete when self-administered, and 90 minutes to complete when administered by
an interviewer. What approaches can be used to overcome this amount of
information difficulty?
Some suggestions that may arise from class discussion include:
a. Divide the questions into 3 or 4 surveys and administer them at separate times to
the same respondents
b. Provide a very large incentive (say, a lottery of $500)
c. Pay respondents for their time
d. Send out a huge number of mail surveys and see what comes back
e. Pay interviewers extra compensation for completing more than a certain minimum
number
6. Describe to your students the survey method selection phase of a recent survey with
7. Privacy issues continually challenge marketing researchers who conduct surveys.
Students protect their privacy just as do typical consumers. Ask students to identify
the various ways they protect their telephone privacy (such as answering machines,
caller identification, etc.), and for each one have them discuss the challenge(s) posed
to marketing researchers. What can researchers do to entice consumers to take part in
surveys? This topic can quickly degenerate to telemarketer bashing, so be prepared to
8. Computer-assisted interview and web-based formats are steadily advancing and
evolving. There may be articles in the Marketing News, Quirk’s Marketing Research
descriptions of their capabilities and many offer trial usage. Have students share their
findings either with short presentations or in a discussion format.
9. We did not include the “traditional telephone interview” where interviewers make
calls from their homes as this method is rarely used. If students inquire about the
omission, this is the answer. Alternatively, you might ask them to add traditional
telephone interviews. The key advantage is fast turnaround relative to face-to-face or
mail alternatives, while the key disadvantage is control as interviewers cannot be
monitored or supervised while working. A comment is that traditional telephone
interviewing is an option when other telephone data collection methods are
unavailable or too expensive.
10. Marketing Research Insight (MRI) 7.1 illustrates how technology and mobile phones
have greatly influenced data collection. Something to point out to students is that
much of the transition from old to new methods took place in the first decade of the
21st century, so it has been a very rapid transition to computer-based and mobile data
collection methods by the marketing research industry.
ACTIVE LEARNING EXERCISES
Experience the Advantages of Surveys
Standardization.
How have the response options for Questions 2 and 3 standardized the survey? In other
words, what answers might have occurred if you did not give your respondents these
specific response categories from which to pick?
Ease of Analysis.
What percentage of your friends who took part in the survey watched television last
night? What percentage of them watched TV for 4 or more hours? How long did it take
you to tabulate the answers to these questions? Also, depending on whether the
respondents checked off or voiced the answers, how did this affect the task of analysis?
Ease of Administration.
How difficult was it to administer the survey? One way to answer this is to estimate how
long it took you, on average, to obtain each respondent’s answers.
Get Beneath the Surface.
Do your friends watch television mostly for entertainment, education, or escape?
Tabulate the answers to Question 3 to find out. Notice that with a single question you
have discovered the reasons (i.e., motivations) for your friends’ television viewing.
Subgroup Differences.
Do the answers from the two males differ from those of the two females? Do separate
percentage tabulations for each gender and compare them. In a matter of a few minutes,
you can spot whether or not differences exist in the subgroups and what the differences
are.
Setting Up Controls for a Telephone Interview
Quality Assurance Procedure
Recommended Procedure
The student team member
interviewers should call the right
students at the proper times of the
day.
Require them to keep detailed logs of their
calling, and do random callbacks of respondents
and ask what time of day the survey took place.
Ensure that they conduct the
interviews correctly, meaning read
the instructions and “skip”
questions as required by the
respondent’s answers.
Require that they turn in completed
questionnaires daily; check their work carefully at
first until you are sure they are doing them
properly.
The 40 interviews should be
conducted on schedule.
Set up a quota system or expect them to turn in
“x” completed questionnaires every so many days
How to handle “no answers” and
answering machines?
Keep a calling log and call back these individuals.
It is not appropriate to leave a message as the
respondents will probably not call when the
interviewer is in.
Make sure that their completed
interviews are not bogus, that is, that
they do not cheat.
Call back a random sample of each interviewer’s
completions to verify that the respondents did, in
fact, take part in the survey
ANSWERS TO END-OF-CHAPTER QUESTIONS
1. List the major advantages of survey research methods over qualitative methods. Can
you think of any drawbacks, if so, what are they?
The advantages are listed in Table 7.1 and repeated following.
Advantage Description
For person-administered surveys, personal interviewers are slower. They are prone
to errors, and although personal interviewers can handle pictures, videos, and
Copyright © 2017 Pearson Education, Inc.
not complete the survey, will fill it out incompletely, will answer questions
erroneously, will not respond in a timely manner, or will refuse to return the survey at
all.
2. How and why has technology impacted data collection from the recent past through
the present?
In the past two decades, there has been a dramatic decline in the willingness of the
general public to take part in surveys, and, second, computer and telecommunications
3. What aspects of person-administered surveys make them attractive to marketing
researchers? What aspects make them unattractive?
4. What aspects of computer-administered surveys make them attractive to marketing
researchers?
5. What are the advantages of person-administered over computer-administered
surveys, and vice versa?
6. What would be the motivation for a researcher to consider a mixed-mode survey?
7. Indicate the differences between: (a) in-home surveys, (b) mall-intercept surveys, and
(c) in-office surveys. What do they share in common?
8. Why were telephone surveys popular before widespread Internet access?
9. Indicate the pros and cons of self-administered surveys.
It is important that students realize that “self–administered” refers to paper-and-pencil
questionnaires as well as online surveys.
Paper-and-pencil self-administered surveys include group self-administered surveys,
10. What advantages do online surveys have over various types of self-administered
surveys?
Speed
Very Fast
Fast
Moderate
Slow
Cost/Response
Very low
Low
Low
Low
Control
High
Moderate
Low
Low
Scope
Can be world-
wide
Local
Local
Can be
national
11. What are the major disadvantages of a mail survey?
12. How does a drop-off survey differ from a regular mail survey?
13. What is a panel company and why are such companies dominating survey data
collection today?
14. How does the incidence rate affect the choice of a data- collection mode?
15. Is a telephone interview inappropriate for a survey that has as one of its objectives a
complete listing of all possible advertising media a person was exposed to in the past
week? Why or why not?
This question can be answered by applying the survey time horizon and budget
16. NAPA Car Parts is a retail chain specializing in stocking and selling both domestic
and foreign automobile parts. To learn about the company’s customers, the
marketing director sends instructions to all 2000 store managers telling them that
whenever a customer makes a purchase of $150 or more, they are to write down a
description of the customer who made that purchase. They are to do this just for the
second week in October, writing each description on a separate sheet of paper. At the
end of the week, they are to send all sheets to the marketing director. Comment on
this data collection method.
This is really an observation method rather than a survey method of data collection.
17. Discuss the feasibility of each of the types of survey mode for each of the following
cases:
a. Faberge, Inc. wants to test a new fragrance called “Lime Brut.”
b. Kelly Services needs to determine how many businesses expect to hire temporary
administrative assistants for those who go on vacation in the summer.
c. Funbrain, an online educational games company, requires information on the
degree to which parents of children in elementary school see online math and
reading games as worthwhile purchases for their children’s education.
d. Samsung is considering a screen view phone watch system and wants to know
people’s reaction to it.
a. Lime Brut
b. Kelly Services
Copyright © 2017 Pearson Education, Inc.
Mail Survey – works but response rate will be low
Group Self-Administered Survey – could be used at several national business
conventions
Drop-Off Survey – works if target companies are in a concise geographic area
such as an industrial park or large office building
c. Funbrain
In-Home Interview – appropriate assuming mothers need to see examples of
Funbrain materials
Mall-Intercept Interview – appropriate and less costly than in-home
In-Office Interview – inappropriate as will miss “at–home” mothers
Central Location Telephone Interview – works unless examples and materials
must be viewed by respondent
Computer-Assisted Telephone Interview – works unless examples and materials
must be viewed by respondent
Fully Computerized Interview – works unless examples and materials must be
viewed by respondent
Online Survey – works best with an online panel, but could work if there were a
list of qualified respondents to contact via e-mail
Mail Survey – could work with proper incentives to increase response rate
Group Self-Administered Survey – could work at PTA or PTO meetings
Drop-Off Survey – could work but in a limited geographical area
d. Samsung
In-Home Interview – appropriate as system would need to be viewed
Copyright © 2017 Pearson Education, Inc.
Central Location Telephone Interview – would work only as a “concept” test
Computer-Assisted Telephone Interview – would work only as a “concept” test
Fully Computerized Interview – would work only as a “concept” test
Online Survey – would work as graphics and even video can be embedded in an
online survey. Also, Samsung has a very diverse market, so the wide scope of
online surveys is appropriate.
Mail Survey – would work only as a “concept” test but subject to nonresponse
Group Self-Administered Survey – works if the system can be demonstrated in
front of a large group of people
Drop-Off Survey – Too limited in scope, might work as a very limited “concept”
test
18. With a telephone survey, when a potential respondent refuses to take part or is found
to have changed his or her telephone number or moved away, it is customary to
simply try another prospect until a completion is secured. It is not standard practice
to report the number of refusals or noncontacts. What are the implications of this
policy for the reporting of nonresponse?
The two concepts involved here are nonqualifiers and nonresponse. With a changed
or no-longer-working phone number, it is a case of nonqualification, meaning that the
19. Compu-Ask Corporation has developed a stand-alone computerized interview system
which can be adapted to almost any type of survey. It can fit on a handheld tablet
computer, and the respondent answers questions directly using a stylus once the
interviewer has turned on the tablet and started up the program. Indicate the
appropriateness of this interviewing system in each of the following cases.
a. A survey of plant managers concerning a new type of hazardous waste disposal
system
b. A survey of high school teachers to see if they are interested in a company’s
videotapes of educational public broadcast television programs
Appropriate. Assuming the videos could be previewed on the computer screen
c. A survey of consumers to determine their reactions to a nonrefrigerated variety of
yogurt
20. A researcher is pondering what survey mode to use for a client that markets a home
security system for home-owners. The system consists of tiny motion sensors that are
pressed onto all of the windows and doors. Once activated with an app, the system
emits a loud alarm and simulates a barking guard dog when an intruder trips one of
the sensors. The client wants to know how many home-owners in the United States
are aware of the system, what they think of it, and how likely they are to buy it in the
coming year. Which consideration factors are positive and which ones are negative
for each of the possible survey modes: (a) in-home interview, (b) mall intercept, (c)
online survey, (d) drop-off survey, and (e) CATI survey?
Students will need to make assumptions about the circumstances of the client and the
Copyright © 2017 Pearson Education, Inc.
owners; mall intercepts will have incidence issues to the extent that shoppers are not
home-owners. With online surveys, respondents will qualify or select themselves to
participate, and drop-offs in neighborhood mail boxes will have high incidence
(assuming few already own alarm systems). CATI will result in a great many calls to
find qualified respondents unless lists of home owners are used.
CASE SOLUTIONS
Case 7.1 Machu Picchu National Park Survey
Case Objective
This case will necessitate analysis of the circumstances surrounding a survey and
recommending and defending a data collection method.
Answers to Case Questions
1. If the questionnaire is an online survey, would it be successful? Why or why not?
2. If the park department uses a mail survey, what issues must be resolved? Would it be
successful? Why or why not?
3. If the seven hotels in the Machu Picchu area each desired to know how its customers
felt about the hotel’s services, prices, and accommodations, how might the park
department and the hotels work together on data collection to affect a mutually
beneficial survey?
4. Using the knowledge that the Peruvian national park department has very meager
resources for marketing research, suggest a different method (not online, not mail,
and not partnering with the local hotels) with the potential of effecting a high
response rate and high-quality responses.
Case 7.2 Integrated Case: Advantage Research, Inc
Case Objective
Students must assess the appropriateness of various data collection methods with the case
of Advantage Research, Inc. The survey should yield a respondent profile that reflects
the demographic appropriate to various and specific customers.
Answers to Case Questions
1. For each client’s survey, take each of the nine data collection methods identified in
this chapter and specify what you think is the strongest and weakest aspects of using
that data collection method for the client’s survey situation.
1. In-home survey – In-home surveys are useful when the research objective
Copyright © 2017 Pearson Education, Inc.
test the physical product, and a mall setting may not be conducive to that level of
interaction.
3. In-office survey – This method often works well with businesspeople, and may
4. Telephone survey – If physical contact is not necessary, a telephone interview
5. Fully automated survey – This type of survey may not work well for any of the
7. Group self-administered survey – This survey methodology is best when working
for economies of scale. All respondents are in the same location and respond to a
8. Drop-off survey – This type of survey is given to a prospective respondent and
then left with the respondent to fill out on their own. This might work well when
Copyright © 2017 Pearson Education, Inc.
9. Mail survey – This type of survey does not use interviewers, and are inexpensive
on a mass-mailing basis. However, there is often a lack of response and a natural
self-selection bias with this type of survey. It will not be as effective a survey tool
for any of the clients, as respondents will have to feel strongly about something,
either positively or negatively, in order to be moved to respond.
2. Based on your analysis from the preceding question, and taking into consideration
any other relevant aspects, decide which is the best data collection method for each
client’s survey. Defend your choice in each case.
Assume that you are the project director recommending the data collection method
you have chosen for each client’s survey in the preceding question, but during the
next daily meeting, Joe and the other project directors vote your recommendations
down. What is the next best data collection method in your mind for each client’s
survey? Defend your choice in each case.