Chapter 9: Conducting Marketing Experiments
Chapter 9
Conducting Marketing Experiments
AT-A-GLANCE
I. Introduction
II. The Characteristics of Experiments
A. Subjects
B. Experimental conditions
C. Effects
2. Interactions
D. Summary of experimental characteristics
III. Basic Issues in Experimental Design
A. Manipulation of the independent variable
1. Experimental and control groups
3. More than one independent variable
B. Selection and measurement of the dependent variable
C. Selection and assignment of test units
1. Sample selection and random sampling errors
3. Matching
4. Repeated measures
D. Extraneous variables
1. Experimental confounds
2. Identifying extraneous variables
IV. Demand Characteristics and Experimental Validity
A. Experimenter bias and demand effects
B. Reducing demand characteristics
2. Isolate experimental subjects
4. Administer only one experimental condition per subject
5. Paying for performance
C. Establishing Control
D. Basic versus factorial experimental designs
Chapter 9: Conducting Marketing Experiments
V. Internal versus External Validity
A. Internal validity
2. History
4. Testing
6. Selection
7. Mortality
B. External validity
1. Convenient subjects
C. Trade-offs between internal and external validity
VI. Test-Marketing
A. Forecasting new product success
B. Testing the marketing mix
C. Identifying product weaknesses
D. Projecting test-market results
E. Estimating sales volume: some problems
1. Overattention
3. Reading the competitive environment incorrectly
4. Time lapse
2. Time
3. Loss of secrecy
VII. Ethical Issues in Experimentation
LEARNING OUTCOMES
2. Design an experiment using the basic issues of experimental design.
4. Weigh the trade-off between internal and external validity.
6. Avoid unethical experimental practices.
CHAPTER VIGNETTE: Warning! This Product May Cause…
Does smoking cause cancer? Does smoking cause death? Does advertising cause people to
smoke? In U.S. courts, plaintiff’s attorneys and in some cases, state governments, have
successfully argued that cigarette companies are responsible for the health problems and even
deaths associated with long-term smoking.
Chapter 9: Conducting Marketing Experiments
But a U.K.-based tobacco company is basing a legal defense on the notion that a lack of certainty
remains over whether or not cigarettes cause cancer. They claim that the only evidence for this is
statistical association and that many other factors are also statistically associated with cancer (i.e.,
socioeconomic status, childhood experiences, ethnicity, personality, and diet). The company
claimed that no experimental evidence isolates the extent to which smoking truly causes cancer,
SURVEY THIS!
Students have had an opportunity to explore the editing features of the Qualtrics survey platform.
What types of variables can be measured using survey items created with Qualtrics? Would it be
RESEARCH SNAPSHOTS
➢ Goldfishing or Bluefishing?
Sea Snapper brand frozen fish products claimed in advertising that their fish sticks are
preferred more than two to one over the most popular brand, Captain John’s, and included a
statement that research existed to substantiate this claim. Captain John’s sued Sea Snapper
claiming that the ads include false claims based on faulty research. Consumer preference
was measured, and the results showed the average preference score for Sea Snapper was
➢ Mechanical Students
The bulk of experimental psychological research relies on student subjects and their ability
to execute the instructions provided faithfully and honestly. Students may become
mechanical and simply move through the procedures, motivated primarily by completing the
task and receiving credit. Enter Mechanical Turk. Amazon provides opportunities for
Chapter 9: Conducting Marketing Experiments
Most “workers” from India report being male (almost 2 of 3), college-educated (8 of 10),
and report little concern about privacy. US “workers” tend slight toward reporting being
➢ The Hidden in Hidden Valley Ranch
Hidden Valley Ranch (HVR) conducted a field market experiment to test three new flavors
of salad dressings (i.e., three levels of the experimental variable flavor). These types of tests
are expensive because small batches of each flavor are produced, bottled, and shipped to
sales representatives who stock the dressings in the stores (almost $20 per bottle!). The first
TIPS OF THE TRADE
➢ Experiments test for causal evidence and thus represent the primary tool for causal research
designs.
➢ Experimental manipulations in marketing research should possess the following
characteristics:
➢ Experimental graphs are useful in displaying results, particularly when interactions are
involved.
➢ Laboratory experiments provide greater internal validity at the cost of lower external validity.
➢ Field experiments, including test markets, increase external validity at the cost of internal
validity.
Chapter 9: Conducting Marketing Experiments
OUTLINE
I. INTRODUCTION
A. Experiments are widely used in causal research designs.
B. The marketing experimenter manipulates one or more independent variables and
holds constant all other possible independent variables while observing effects on
dependent variables.
C. A researcher can control variables in an experiment to a degree not possible in a
survey.
II. THE CHARACTERISTICS OF EXPERIMENTS
A. Subjects
1. Subjects – the sampling units for an experiment, usually human participants in
B. Experimental Conditions
1. An experimental condition refers to one of the possible levels of an
experimental variable manipulation.
3. We refer to a continuous variable expected to show a statistical relationship with
a dependent variables as a covariate.
C. Effects
1. Effects are the characteristics of experiments that allow hypotheses to be tested.
2. Main Effects
3. Interactions
a. An interaction effect is a change in a dependent variable due to a specific
D. Summary of Experimental Characteristics
1. Experiments use subjects instead of respondents. The experimental
2. Experimental variables become the key independent variables. The researcher
3. Experimental effects exist to the extent that treatment groups exhibit differences
III. BASIC ISSUES IN EXPERIMENTAL DESIGN
A. Experimental designs involve no less than four important design elements:
2. Selection and measurement of the dependent variable
4. Control over extraneous variables
Chapter 9: Conducting Marketing Experiments
B. Manipulation of the Independent Variable
1. An experimental treatment is the term referring to the way an experimental
variable is manipulated.
2. Experimental and Control Groups
3. Several Experimental Treatment Levels
a. The advertising/sales experiment with one experimental and one control
4. More than One Independent Variable
a. An experiment can also be made more complicated by including the effect of
another experimental variable.
b. Often, the term cell refers to a treatment combination associated with an
experimental group…
c. The formula for computing the number of cells involved in any experiment
is:
d. Since there are multiple experimental variables, this design involves both
main effects and interactions.
C. Selection and Measurement of the Dependent Variable
1. Unless the dependent variables are relevant and truly represent an outcome of
D. Selection and Assignment of Test Units
1. Test units are the subjects or entities whose responses to the experimental
treatment are measured.
3. Sample Selection and Random Sampling Errors
4. Randomization
5. Matching
Chapter 9: Conducting Marketing Experiments
F. Basic versus Factorial Experimental Designs
2. Factorial experimental designs are more sophisticated than basic experimental
designs and allow for an investigation of the interaction of two or more
independent variables.
G. Laboratory Experiments
1. In a laboratory experiment the researcher maximizes control over the research
setting and extraneous variables.
H. Field Experiments
2. In field experiments, as in test-markets, subjects do not even know they have
taken part in an experiment.
I. Advantages of Between-Subjects Designs
2. Between-subjects design – each subject receives only one treatment
combination
3. Advantages
V. INTERNAL VERSUS EXTERNAL VALIDITY
A. Internal validity – exists to the extent that an experimental variable is truly
responsible for any variance in the dependent variable
1. Manipulation Checks
2. History
a. History effect – occurs when some change other than the experimental
3. Maturation
4. Testing
a. Testing effects (pretesting effects) – nuisance effects occurring when the
Chapter 9: Conducting Marketing Experiments
F. Projecting Test-Market Results
1. Researchers do a trial test on a small scale with the hope that the small scale
results will match those achieved in a full-scale product introduction.
G. Estimating Sales Volume: Some Problems
1. Overattention
2. Unrealistic Store Conditions
3. Reading the Competitive Environment Incorrectly
a. Another common mistake is to assume that the competitive environment will
4. Time Lapse
a. One relatively uncontrolled problem results from the time lapse between the
test-market experiment and the national introduction of the product.
H. Advantages of Test-Marketing
2. Researchers can easily communicate results to management.
I. Disadvantages of test-marketing
2. Time
3. Loss of secrecy
VII. Ethical Issues in Experimentation
A. The question of deception
B. Researchers should debrief experimental subjects following an experimental
procedure, because it returns the subjects to their normal state.
QUESTIONS FOR REVIEW AND CRITICAL THINKING/ANSWERS
1. Define experimental condition, experimental treatment, and experimental group. How are
these related to the implementation of a valid manipulation?
© 2016 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, in whole or in
part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website for classroom use.
2. A tissue manufacturer that has the fourth-largest market share plans to experiment with a 50¢
off coupon during November and a buy one, get one free coupon during December. The
experiment will take place at Target stores in St. Louis and Kansas City. In addition, coupons
will be issued either through Groupon.com or in the local newspaper. Sales will be recorded
by scanners from which mean tissue sales for each store for each month can be computed and
interpreted.
a. What are the experimental variable and the dependent variable?
The experimental, or independent, variable is the price promotion, either a 50¢ off coupon or a
buy one, get one free coupon. The dependent variable is sales.
b. Prepare a mock experimental graph that shows hypothetical results (simply guess at
what the mean values for each experimental condition would be).
Treatment
Sales
Price Promotion
50¢ off coupon (Nov)
$2,000,000
Buy one, get one coupon (Dec)
$2,590,000
c. What types of people would make good subjects for this experiment?
3. What is the difference between a main effect and an interaction in an experiment? In question
2, what will create a main effect? Is an interaction possible?
4. How can experimental graphs be used to show main effects and interactions?
5. What purpose does the random assignment of subjects serve?
© 2016 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, in whole or in
part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website for classroom use.
6. Why is an experimental confound so damaging to the conclusions drawn from an
experiment?
7. What are demand characteristics? How can they be minimized?
8. [Ethics Question] Suppose researchers were experimenting with how much more satisfied
consumers are with a “new and improved” version of some existing product. How might the
researchers design a placebo within an experiment testing this research question? Is using
such a placebo ethical or not?
9. If a company wanted to know whether to implement a new management training program
based on how much it would improve ROI in its southwest division, would you recommend a
field or lab experiment?
© 2016 Cengage Learning. All Rights Reserved. May not be scanned, copied or duplicated, in whole or in
part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website for classroom use.
14. The idea of nonspurious association was introduced in Chapter 3. How does a confounding
variable affect whether an association is spurious or nonspurious?
15. Why is a test-market usually considered an experiment?
16. When is test-marketing likely to be conducted? When is it not as appropriate?
17. Which type of validity is most relevant to test-marketing?
18. What are the advantages and disadvantages of test-marketing?
Chapter 9: Conducting Marketing Experiments
Test-markets also have disadvantages:
2) Time – average test-market requires about 12 months.
3) Loss of secrecy – the marketplace is a public forum, so competitors know about the
19. [Ethics Question] What role does debriefing play in ensuring that experimental procedures
are consistent with good ethical practice?
20. [Ethics Question] A university researcher asks students to participate in an experiment that
takes 4 hours to complete in the last week of a semester. Each student receives 5 points extra
credit. Another researcher conducts the same experiment using Internet crowdsourced
subjects paid $0.50 to participate. The subjects know they will be paid after the researcher
checks the quality of each subject’s response. Contrast these two approaches including a
discussion of any ethical issues that might be involved.
RESEARCH ACTIVITIES
1. Consider the situation of a researcher approached by Captain John’s in the Research Snapshot
of this chapter.
One could argue that color of the plate on which the fish sticks were offered to subjects was a
confounding variable that influenced the dependent variable.
Chapter 9: Conducting Marketing Experiments
b. Design an experiment that would provide a more valid test of the research question,
“Do consumers prefer Sea Snapper fish sticks compared to Captain John’s fish
sticks?”
2. Conduct a taste test involving some soft drinks with a group of friends. Pour them several
ounces of three popular soft drinks and simply label the cups A, B, and C. Make sure they are
blind to the actual brands. Then, let them drink as much as they want and record how much of
each they drink. You may also ask them some questions about the drinks. Then, allow other
subjects to participate in the same test, but this time, let them know what the three brands are.
Record the same data and draw conclusions. Does brand knowledge affect behavior and
attitudes about soft drinks?
3. Consider the information in the chapter describing Facebook’s experiments with their own
users. Do you see any evidence of such experiments going on today at Facebook, Instagram
or Twitter? Would you consider Facebook’s experimentation lab or field experiments? Given
the backlash Facebook experienced from conducting the experiments, what advice would you
have for them to improve the way they go about conducting experiments through their
platform.
CASE 9.1 Examining Product Failure at No-Charge Electronics
Objective: To encourage students to think about the ethical issues related to experimental
manipulations.
Summary: No-Charge Electronics owner Buzz Auphf wants to know how much product failure
Chapter 9: Conducting Marketing Experiments
arrives indicating that a severe virus has infected their computers, and this goes on for forty-five
minutes with no remedy. At that time, a debriefing message pops up telling subjects that it was all
part of an experiment and that nothing is wrong with their computers.
Question
CASE 9.2 Tooheys
Objective: To encourage students to think about validity and ethical issues surrounding
experiments.
Summary: Sixty-six willing Australian drinkers helped a Federal Court judge decide that
Tooheys did not engage in misleading or deceptive advertising for its 2.2 beer, which contains 2.2
Questions:
1. Would a lab experiment or a field experiment be more “valid” in determining whether
Tooheys could cause a normal beer consumer to become intoxicated? Explain.
2. Describe an alternative research design that would have higher validity.
