CHAPTER 11
DEALING WITH FIELDWORK AND DATA QUALITY ISSUES
LEARNING OBJECTIVES
In this chapter you will learn:
11-1 What constitutes sampling error
11-2 About errors that occur during field data collection
11-3 How to control field data collections error
11-4 All about nonresponse error
11-5 How panel companies control errors
11-6 What is a dataset, coding of data, and the data code book
11-7 Data quality issues in data sets
CHAPTER OUTLINE
Data Collection and Nonsampling Error
Possible Errors in Field Data Collection
• Intentional Fieldworker Errors
• Unintentional Fieldworker Errors
• Intentional Respondent Errors
• Unintentional Respondent Errors
Field Data Collection Quality Controls
• Control of Intentional Fieldworker Error
• Control of Unintentional Fieldworker Error
• Control of Intentional Respondent Error
• Control of Unintentional Respondent Error
• Final Comment on the Control of Data Collection Errors
Nonresponse Error
• Refusals to Participate in the Survey
• Break-Offs During the Interview
• Refusals to Answer Specific Questions (Item Omission)
• What is a Completed Interview?
• Measuring Response Rate in Surveys
How Panel Companies Control Error
Dataset, Coding Data, and the Data Code Book
Data Quality Issues
• What to Look for in Raw Data Inspection
o Incomplete Responses
o Nonresponses to Specific questions (Item Omissions)
o Yea- or Nay-Saying Patterns
o Middle-of-the-Road Patterns
o Other Data Quality Problems
o How to Handle Data Quality Issues
KEY TERMS
Nonsampling error Data collection
Fieldworker error Respondent error
Intentional fieldworker errors Interviewer cheating
Leading the respondent Unintentional interviewer errors
Interviewer misunderstanding Fatigue-related mistakes
Intentional respondent errors Falsehoods
Nonresponse Unintentional respondent error
Respondent misunderstanding Guessing
Attention loss Distractions
Respondent fatigue Supervision
Validation Orientation sessions
Role-playing sessions Anonymity
Confidentiality Incentives
Validation checks Third-person technique
Questionnaire instructions and examples Reversals of scale endpoints
Prompters Refusals
Break-offs Item omission
Completed interview CASRO response rate formula
Dataset Data coding
Data code book Incomplete response
Yea-saying Nay-saying
Middle-of-the-road pattern Panel company
TEACHING SUGGESTIONS
1. The distinction between sampling error and nonsampling error is presented in a
definitional matter-of-fact way because of space constraints. Instructors who want
2. Implicit throughout the chapter is the message that professional data collection
companies will control for fieldwork errors. If you have a nearby data collection
company, invite the manager to your class to talk about the company’s operations,
particularly training, orientation sessions, and other quality control procedures.
Alternatively, a field trip to a data collection company may be feasible.
3. Students may not appreciate the need for interviewer training. A way to combat this
belief is to bring a (complicated) telephone or personal interview questionnaire to
class and require two students to role play the interviewer and the respondent. It will
become apparent that reading the instructions, following skip patterns, and making
notations of the respondent’s answers on the questionnaire takes practice.
4. There is a danger when talking about respondent and interviewer errors that students
will come away with a negative opinion about surveys. That is, class discussion
sometimes dwells on the cheating, falsehoods, and other misrepresentations that occur
in surveys, and students may acquire a belief that these problems are rampant. The
point to be made is that there may be significant problems if controls are not used, but
most researchers will apply controls even if an ad hoc group is used. So the errors are
kept to a minimum.
5. The section on “Measuring Nonresponse Error in Surveys” is based on the 1982
CASRO report that attempted to standardize nonresponse error measures. The report
and its formula are complicated, and the treatment in this section is our best attempt
to make them understandable to students. We recommend that you take up each
“outcome” carefully and fully discuss what its assumptions about data collection are,
are eligibility of potential respondents, substitutions, and so forth. We have also
included a Marketing Research Insight with specific calculations that you may want
to go over with students to help them understand how the formula handles each code.
Active Learning Exercise “How to Calculate a Response Rate Using the CASRO
Formula” and end–of-chapter question 17 requires students to use the CASRO
formula.
6. An experiential way to have students learn about the need for preliminary
questionnaire screening is to bring a stack of completed questionnaires to class, pass
them out to students, and have them inspect each for problems such as incomplete
questionnaires, nonresponses to specific questions, yea-saying, nay-saying, or middle-
of-the-road patterns. Outlier questionnaires can be spotted, and class discussion can
be used to determine how to handle the outliers.
ACTIVE LEARNING EXERCISES
What Type of Cheater Are You?
What Type of Error is It?
Situation
Type of Error
A respondent says “no opinion” to every question
asked.
Intentional interviewee error
When a mall-intercept interviewer is suffering from a
bad cold, few people want to take the survey.
Unintentional interviewer error
Because a telephone respondent has an incoming call,
he asks his wife to take the phone and answer the rest
of the interviewer’s questions.
Intentional interviewee error
A respondent grumbles about doing the survey, so an
interviewer decides to skip asking the demographic
questions.
Intentional interviewer error
A respondent who lost her job last year gives her last
year’s income level rather than the much lower one she
will earn for this year.
Intentional interviewee error
How to Calculate a Response Rate Using the CASRO Formula
Students must determine the outcome designation of each telephone number and use
these to calculate the response rate using the CASRO formula supplied in the chapter.
Here are the designations for the 19 numbers.
Telephone
Number
1st Attempt
2nd Attempt
3rd Attempt
474-2892
No Answer
No Answer
Completed
474-2668
Busy
Ineligible
Respondent
488-3211
Disconnected
488-2289
Completed
672-8912
Wrong Target
263-6855
Busy
Busy
Busy
265-9799
Terminate
234-7160
Refusal
619-6019
Call Back
Busy
Busy
619-8200
Ineligible
Respondent
474-2716
Ineligible
Respondent
774-7764
No Answer
No Answer
474-2654
Disconnected
488-4799
Wrong Target
619-0015
Busy
Completed
265-4356
No Answer
No Answer
Completed
265-4480
Wrong Target
263-8898
No Answer
No Answer
No Answer
774-2213
Completed
The tallies are as follows
Result
Number
Completed
5
Ineligible Respondent
3
Wrong Target
3
Busy
2
Disconnected
2
Terminate
1
Refusal
1
No Answer
2
%5863.8
5
)21122)(
335
(5
=
=
++++
++
+
ANSWERS TO END-OF-CHAPTER QUESTIONS
1. Distinguish sampling error from nonsampling error.
Both the sample plan and sample size concern sampling error, and the amount of
sampling error can be measured by using a standard error formula. Sampling error is
2. Because we cannot easily calculate nonsampling errors, how must the prudent
researcher handle nonsampling error?
3. Identify different types of intentional field worker error and the controls used to
minimize them. Identify different types of unintentional fieldworker error and the
controls used to minimize them.
4. Identify different types of intentional respondent error and the controls used to
minimize them. Identify different types of unintentional respondent error and the
controls used to minimize them.
5. Define “nonresponse.” List three types of nonresponse found in surveys.
Nonresponse is defined as a failure on the part of a prospective respondent to take
6. How should a researcher define a “completed interview”?
Almost all surveys have some item omissions, break-off s, and partially completed
surveys. Nonetheless, these respondents did provide some information. At which
7. Why is it necessary to perform preliminary screening of a dataset?
8. Identify five different problems that a researcher might find while screening a
dataset.
9. What is an “exception” and what is typically done with each type of exception
encountered?
Problem respondents fall into five categories: incomplete responses (break-offs), non
responses to specific questions (item omissions), yea- or nay-saying patterns, and
10. Your church is experiencing low attendance with its Wednesday evening Bible
classes. You volunteer to design a telephone questionnaire aimed at finding out why
church members are not attending these classes. Because the church has limited
funds, members will be used as telephone interviewers. List the steps necessary to
ensure high data quality in using this do-it-yourself option of field data collection.
This is an instance of no controls in place whatsoever, so students will need to
trained in telephone interviewing.
11. A new mall-intercept company opens its offices in a nearby discount mall, and its
president calls on the insurance company where you work in an effort to solicit
business. It happens that your company is about to do a study on the market reaction
to a new whole life insurance policy it is considering adding to its line. Make an
outline of the information you would want to receive from the mall-intercept company
president to assess the quality of the company’s services.
Because the company is new to you, it is necessary to gain information on its service
quality. The five advantages of using a professional data collection company can be
12. Acme Refrigerant Reclamation Company performs large-scale reclamation of
contaminated refrigerants as mandated by the U.S. Environmental Protection
Agency. It wishes to determine what types of companies will use this service, so the
marketing director designs a questionnaire intended for telephone administration.
Respondents will be plant engineers, safety engineers, or directors of major
companies throughout the United States. Should Acme use a professional field data
collection company to gather the data? Why or why not?
13. You work part-time in a telemarketing company. Your compensation is based on the
number of credit card applicants you sign up. The company owner has noticed that
the credit card solicitation business is slowing down, so she decides to take on some
marketing research telephone interview business. When you start work on Monday,
she assigns you to do telephone interviews and gives you a large stack of
questionnaires to have completed. What intentional field worker errors are possible
under the circumstances described here?
14. Indicate what specific intentional and unintentional respondent errors are likely with
each of the following surveys.
a. The Centers for Disease Control and Prevention sends out a mail questionnaire
on attitudes and practices concerning the prevention of AIDS.
b. Eyemasters has a mall-intercept survey performed to determine opinions and uses
of contact lenses.
Refusals are possible, but falsehoods are not likely as contact lenses are not a
c. Boy Scouts of America sponsors an online survey on Americans’ views on
humanitarian service agencies.
If BSA is given as the sponsor, refusals will be minimized, and knowledge of the
15. On your first day as a student marketing intern at the O-Tay Research Company, the
supervisor hands you a list of yesterday’s telephone interviewer records. She tells
you to analyze them and to give her a report by 5 p.m. Well, get to it!
Ron Mary Pam Isabelle Designation
CASE SOLUTIONS
Case 11.1 Cass Corridor Food Co-Op
Case Objective
This case involves drafting the control measures necessary to ensure that interviews
conducted by student interviewers from their residences will have good quality.
Answers to Case Questions
Case 11.1: Skunk Juice
1. Take the role of the marketing research projects director. Draft all of the interviewer
controls you believe are necessary to ensure data collection comparable in quality to
that gathered by a professional telephone interviewing company.
Here are the controls.
a. Orientation. Have orientation sessions to go over the questionnaires
2. The AMA chapter president calls the marketing research projects director and says,
“I’m concerned about the questionnaire’s length. It will take over 20 minutes for the
typical respondent to complete it. Isn’t that length going to cause problems?” Again
take the role of the marketing research projects director. Indicate what nonresponse
problems might result from the questionnaire’s length, and recommend ways to
counter each of these problems.
Case 11.2 Sony Televisions Ultra HS TV Survey
Case Objective
Answers to Case Questions
Keeping the quality of the data as a foremost concern, compare the practices of these two
competing panel companies and recommend one that you think Sony research should use
for the survey. Why have you selected your choice over the competitor?
The table is provided below with an “Analysis” row after each set of company responses.
Question 1. What experience does your company have with providing online samples for
market research?
Company A: We have conducted market research
since 1999. We are the only panel company to take
advantage of computer technology and provide a
truly nationally representative U.S. sample online.
Company B: We have supplied online U.S. samples
since 1990, Europe samples since 2000, and our
Asian Panel went “live” in 2005. We have supplied
approximately 5,000 online samples to our clients in
the past 10 years.
Company B is older and it has global coverage whereas A is younger an only has a U.S.
sample
Question 2. Please describe and explain the type(s) of online sample sources from which
you recruit respondents.
Company A: Individuals volunteer for our online
panel via our website where they are informed that
they will be compensated with redemption points
based on the number of surveys in which they take
part.
Company B: We recruit household members by
asking them to join our panel, telling them they can
have a say in the development of new products and
services. They are rewarded with “credits” that they
can use to claim products.
Company B is more selective in that it has direct recruitment. Company A seems to have
an “anyone” recruitment system.
Question 3. What steps do you take to achieve a representative sample of the target
population?
Company A: Our master panel of over 100,000
individuals mirrors the population distribution of
the U.S. Census with respect to 10 demographic
factors such as gender, education, income, marital
status, etc.
Company B: The client specifies the target market
population using any one of 1,000 variables,
including demographic, ownership, purchase
behavior, and other variables. We invite panelists
who meet the client’s criteria to participate in the
survey.
Company B’s drop-out is low and its replacement system is superior to that of Company
A.
Question 4. What profiling data are held on respondents? How is it done?
Company A: We maintain extensive individual–
level data, in the form of about 1,000 variables
including demographics, household characteristics,
financials, shopping and ownership, life styles, and
more. All are updated every other year.
Company B: For each panelist, we have about 2,500
data points on demographics, assortment of goods
and services owned, segmentation/life style factors,
health-related matters, political opinions, travel,
financials, Internet usage, leisure activities,
memberships, etc. Our updating is done annually.
It is obvious that Company B has more “stored” variables in their panel dataset. Plus, the
annual update is more current than Company A’s every other year update.
Question 5. Please describe your survey invitation process.
Company A: Typically a survey
invitation is sent via email and posted on every
selected panel member’s personal member page. In
either case, we have a link to the online survey
location: “Click here to start your survey.” The
email invitation is sent daily to selected panelists
until the survey quota is filled.
Company B: Based on the client’s sample
requirements, we email selected panelists with a link
to the online survey. After 48 hours, if the panelist
has not participated, we send a reminder, and again
48 hours after the reminder.
The systems are almost identical except Company A’s reminder is daily while Company
B’s reminder is every other day.
Assuming there are no other factors to consider (such as price), select Company B as it is
superior to Company A in 4 out of the 5 areas.