Unlock document.
This document is partially blurred.
Unlock all pages and 1 million more documents.
Get Access
Chapter 12
Experimental Research
AT-A-GLANCE
I. Creating an Experiment
A. An illustration: Can a Self-Efficacy Intervention Enhance Job Attitude?
Experimental subjects
Independent variables
Experimental outcome
Independent variable main effects and interaction
II. Designing an Experiment to Minimize Experimental Error
A. Manipulation of the independent variable
Experimental and control groups
Several experimental treatment levels
More than one independent variable
Repeated measures
B. Selection and measurement of the dependent variable
C. Selection and assignment of test units
Sample selection and random sampling error
Randomization
Matching
Control over extraneous variables
Experimental confounds
Extraneous variables
III. Demand Characteristics
A. What are demand characteristics?
B. Experimenter bias and demand effects
C. Hawthorne effect
D. Reducing demand characteristics
Experimental disguise
Isolate experimental subjects
Use a “blind” experimental administrator
Administer only one experimental condition per subject
IV. Establishing Control
A. Problems controlling extraneous variables
V. Ethical Issues in Experimentation
VI. Practical Experimental Design Issues
A. Basic versus factorial experimental designs
B. Laboratory experiments
C. Field experiments
D. Within-subject and between-subjects designs
VII. Issues of Experimental Validity
A. Internal validity
Manipulation checks
History
Maturation
Testing
Instrumentation
Selection
Mortality
B. External validity
Student subjects
C. Trade-offs between internal and external validity
VIII. Classification of Experimental Designs
A. Symbolism for diagramming experimental designs
B. Three examples of quasi-experimental designs
One-shot design
One-group pretest-posttest design
Static group design
C. Three alternative experimental designs
Pretest-posttest control group design (Before-after with control)
Posttest-only control group design (After-only with control)
Compromise designs
D. Time series designs
E. Complex experimental designs
Completely randomized design
Randomized-block design
Factorial designs
LEARNING OUTCOMES
1. Identify the independent variable, dependent variable, and construct a valid simple experiment to
assess a cause and effect relationship
2. Understand and minimize the systematic experimental error
3. Know ways of minimizing experimental demand characteristics
4. Avoid unethical experimental practices
5. Understand the advantages of between-subjects experimental design
6. Weigh the trade-off between internal and external validity
7. Use manipulations to implement a completely randomized experimental design, a randomized-block
design, and a factorial experimental design
CHAPTER OPENING VIGNETTE: Testing Web Protocols for Financial
Markets
Technology has drastically changed the way we conduct banking and related financial services. For most
of us, how it all happens is not that relevant. However, for information technology directors and financial
managers, how this process occurs—and how to make it occur faster and cheaper—is vitally important.
An experiment of two protocols (FIX and SOAP) for transferring information over computer networks is
described. Researchers designed a laboratory experiment to compare the relative performance of the two
in business computing scenarios. While the results are not important to us, an understanding of the
process of testing these two approaches is very important to business researchers.
SURVEY THIS!
Students are asked to look at the questions on prospective careers. The questions actually represent a very
simple experimental design in which the type of occupation described may cause the subjects’ responses
to these questions. Students are asked to look at the data and see if they can determine whether or not
students’ beliefs about careers are altered by the type of occupation they were assigned to rate. Here, job
type becomes the experimental manipulation. Do you know the treatment levels?
RESEARCH SNAPSHOTS
Talking While Driving: Are Cell Phone Conversations Different from Passenger Conversations?
The increased use of cell phones while driving has led to increased driver distractions and the
potential for accidents and even fatalities. But people engage in conversations with fellow
passengers as they drive as well. Researchers designed an experiment with a driving simulator to
test whether engaging in a conversation on a cell phone is more dangerous. Students were
assigned to one of three groups: a group with no passenger as a control group, a group that had a
passenger who asked them questions while they simulated driving, and a group where the driver
engaged in a cell phone conversation. The results indicate that driving while engaged in a
conversation on a cell phone was indeed much more distracting and dangerous than having a
conversation with a passenger, perhaps because a passenger can anticipate changing driving
conditions and adjust their engagement in conversation accordingly.
The Hidden in Hidden Valley Ranch
Hidden Valley Ranch (HVR) conducted a field market experiment to test three new flavors of
salad dressings (i.e., three levels of the experimental variable flavor). These types of tests are
expensive because small batches of each flavor is produced, bottled, and shipped to sales
representative who stock the dressings in the stores. The first day involved placing the products
in retail store. On the second day, the reps returned to record sales and saw that all flavors had
sold. It turns out that a competitor bought every bottle, and HVR was unable to produce any sales
data (the dependent variable). Moreover, the competitor was able to determine the recipe in their
labs. So the risks with field tests are: no more secrets once the product is for sale and the risk of
espionage rendering the experiment invalid.
OUTLINE
I. CREATING AN EXPERIMENT
The purpose of experimental research is to allow the investigator to control the research
situation so that causal relationships among variables may be evaluated.
The experimenter manipulates one or more independent variables and holds constant all other
possible independent variables while observing the effects on dependent variables.
Independent variables are expected to determine the outcomes of interest.
Dependent variables are the outcomes of interest.
An Illustration: Can a Self-Efficacy Intervention Enhance Job Attitude?
An experiment investigating how self-efficacy might influence employee’s attitude
toward their job is described (Self-efficacy is a person’s confidence and belief in their
own abilities to accomplish tasks at hand).
Experimental Subjects
Participants in experimental research are referred to as subjects rather than
respondents because the experimenter subjects them to some experimental treatment.
In this experiment, 35 of the subjects were given positive feedback and
encouragement from their supervisors as the experimental treatment. The other 36
subjects were not provided the positive feedback.
Independent Variables
Experiment involved one relevant independent variable—whether or not the
employee received positive feedback.
While not a true independent variable, the length of time each employee had worked
with the firm was also important to the researchers.
oVariables such as this are referred to as blocking variables—a categorical (less
than interval) variable that is not manipulated as in an experimental variable but
is included in the statistical analysis of experiments.
Experimental condition refers to one of the possible levels of an experimental
variable manipulation.
Subjects were divided into “newcomers” (the new employees) and “insiders” (current
employees) and then randomly assigned to either the treatment condition or the
control group.
Differences between groups are analyzed.
Experimental Outcome
The key outcomes, or dependent variables, is the subject’s job satisfaction (only one
looked at in this illustration), organizational commitment, professional commitment,
intent to quit the organization, and intent to quit the profession.
In addition, the researchers followed up later to see if the subjects had actually left
the firm.
A rating scale assessed job satisfaction, with higher scores representing higher job
satisfaction.
oMeans are given in Exhibit 12.2.
oLooks like the difference in job satisfaction is primarily between the current and
new employees.
Independent Variable Main Effects and Interaction
Length of time that the employee works at the firm clearly appears to matter, but
attempts to enhance self-efficacy shouldn’t be dismissed so quickly.
The research must examine both the effects of each experimental variable alone and
the effects due to combinations of variables.
A main effect refers to the experimental difference in means between the different
levels of any single experimental variable.
An interaction effect is due to a specific combination of independent variables.
oIt’s possible that the combination of length of employment and self-efficacy
treatment creates effects that are not clearly represented in the main effects.
Experimental results are often shown with a line graph (see Exhibit 12.3).
oMain effects are illustrated when the lines are at different heights.
oWhen the lines have very different slopes, an interaction is likely present.
In this example:
oThe worst situation is the current employees who do not receive positive
feedback.
oThe best scenario occurs when the treatment is given to new employees.
oIt appears that job satisfaction decreases over time, and the benefit of the
self-efficacy treatment is greater for the employees that have been with the
organization than for new employees.
II. DESIGNING AN EXPERIMENT TO MINIMIZE EXPERIMENTAL ERROR
Experimental designs involve no less than four important design elements:
1. Manipulation of the independent variable(s).
2. Selection and measurement of the dependent variable(s).
3. Selection and assignment of experimental subjects.
4. Control over extraneous variables.
Manipulation of the Independent Variable
The researcher creates the values of the independent variables.
Experimental independent variables are hypothesized to be causal influences.
An experimental treatment is the term referring to the way an experimental variable is
manipulated, and it often involves treatments with more than two levels.
Experimental variables are often categorical variables that take on a value to represent
some classifiable or qualitative aspect (e.g., protocol is either FIX or SOAP – see chapter
vignette).
Independent variables may truly be continuous variables, and the researcher must select
appropriate levels of that variable as experimental treatments.
Researcher decides on levels that would be relevant to study.
Levels should be noticeably different and realistic.
Experimental and Control Groups
In the simplest experiment, an independent variable is manipulated over two
treatment levels resulting in two groups: an experimental group and a control
group.
Experimental group – one in which an experimental treatment is administered.
Control group – one in which no experimental treatment is administered.
By holding conditions constant in the control group, the researcher controls for
potential sources of error in the experiment.
Several Experimental Treatment Levels
By analyzing more groups each with a different treatment level, a more precise
result may be obtained than in the simple experimental group-control group
experiment described above.
May still involve a control group.
Design can produce only a main effect.
More Than One Independent Variable
Include the effect of another experimental variable.
The term cell is used to refer to treatment combinations within an experiment.
The number of cells involved in any experiment can be easily computed as
follows: K = (T1)(T2)…(Tm).
K = the number of cells
T1 = the number of treatment levels for experimental group number one.
T2 = the number of treatment levels for experimental group number two
and so on.
Including multiple variables allows a comparison of experimental treatments on
the dependent variables.
This design involves both main effects and interactions.
Repeated Measures
Repeated measures designs – experiments in which an individual subject is
exposed to more than one level of an experimental treatment.
Possesses several drawbacks even though it is more economical.
Selection and Measurement of the Dependent Variable
Choosing the right dependent variable is part of the problem definition process.
The amount of time needed for effects to become evident should be considered.
Selection and Assignment of Test Units
Test units are the subjects or entities whose responses to the experimental treatment are
measured or observed (i.e. individual consumers, employees, organizational units, sales
territories, market segments, brands, stores, etc.).
People are the most common test units in most business experiments.
Sample Selection and Random Sample Errors
As in other forms of research, random sampling errors and sample selection
errors may occur.
Sample selection error occurs because of flaws in procedures used to assign
experimental test units.
Systematic or non-sampling error may occur if the sampling units in an
experimental cell are somehow different than the units in another cell, and this
difference affects the dependent variable.
Randomization
Randomization – the random assignment of subject and treatments to groups –
is one device for equally distributing the effects of extraneous variables to all
conditions.
Nuisance variables—items that may affect the dependent measures but are not
of primary interest—will not be eliminated, but they will be controlled because
they are likely to exist to the same degree in every experimental cell.
Matching
Assigning subjects in a way that their characteristics are the same in each group.
This is best thought of in terms of demographic characteristics (e.g., If a subject’s
sex is expected to influence the dependent variable, make sure there are equal
numbers of men and women in each experimental cell.).
While useful, the researcher can never be sure that sampling units are matched on
all characteristics.
Control Over Extraneous Variables
This is related to the various types of experimental error.
Recall that total survey error was classified into two basic types: random
sampling error and systematic error.
Same dichotomy applies to all research designs, but the terms random (sampling)
error and systematic error are more frequently used when discussing
experiments.
Experimental Confounds
A confound in an experiment means that there is an alternative explanation
beyond the experimental variables for any observed differences in the dependent
variable.
Once identified, the validity of the experiment is severely questioned.
Extraneous Variables
Marketing mix variables (i.e., price, product, promotion, and distribution)
interact with uncontrollable forces in the market (i.e., competitors’ activities,
consumer trends).
Thus, many marketing experiments are subject to the effect of extraneous
variables.
Must be identified before the experiment if at all possible so the experimenter
can control or eliminate such variables.
III. DEMAND CHARACTERISTICS
What Are Demand Characteristics?
Demand characteristic – an experimental design element that unintentionally provides
subjects with hints about the research hypothesis.
Knowledge of the experimental hypothesis creates a particular type of confound known
as a demand effect.
Experimenter Bias and Demand Effects
Demand characteristics are aspects of an experiment that demand (encourage) that the
subjects respond in a particular way a source of systematic error.
Prominent demand characteristics are often presented by the person administering
experimental procedures.
Experimenter bias – an experimenter’s presence, actions, or comments influence the
subjects’ behavior or sway the subjects to slant their answers to cooperate with the
experimenter.
Hawthorne Effect
Hawthorne Effect – people will perform differently when they know they are
experimental subjects.
Named after a famous management experiment that attempted to study the effects on
productivity of various working conditions (i.e., hours of work, rest periods, lighting,
methods of pay) at the Western Electric Hawthorne plant in Cicero, IL.
Researchers found that workers’ productivity increased regardless of the
manipulation.
Investigators realized that the workers’ morale was higher because they were
aware of being part of a special experimental group.
Social interaction should be restricted in laboratory experiments because conversations
might produce joint decisions rather than a desired individual one.
Reducing Demand Characteristics
It is practically impossible to eliminate demand characteristics, but the following steps
can by taken to reduce them:
1. Use an experimental disguise.
2. Isolate experimental subjects.
3. Use a “blind” experimental administrator.
4. Administer only one experimental treatment level to each subject.
Experimental Disguise
Subjects can be told that the purpose of the experiment is somewhat different
than the actual purpose.
Most often, they are simply told less than the complete “truth” about what is
going to happen.
In other cases, more deceit may be needed.
A placebo is an experimental deception involving a false treatment.
A placebo effect refers to the corresponding effect in a dependent
variable that is due to the psychological impact that goes along with
knowledge of the treatment.
Isolate Experimental Subjects
Discussion among subjects may lead them to guess the experimental hypotheses.
Integrity will be higher when each only knows enough to participate in the
experiment.
Use a “Blind” Experimental Administrator
The people administering the experiment may not be told the hypotheses.
Less likely to give off clues that result in demand effects.
Administer Only One Experimental Treatment Per Subject
When subjects know more than one experimental treatment condition, they are
much more likely to guess the experimental hypotheses.
IV. ESTABLISHING CONTROL
The major difference between experimental research and descriptive research is an
experimenter’s ability to control variables by either holding conditions constant or
manipulating the experimental variable.
When extraneous variables cannot be eliminated, experimenters may strive for constancy of
conditions, which means that subjects in all experimental groups are exposed to identical
conditions except for the differing experimental treatments.
If an experimental method requires that the same subjects be exposed to two or more
experimental treatments, an error may occur due to the order of presentation.
Counterbalancing attempts to eliminate the confounding effects of order of presentation.
Problems Controlling Extraneous Variables
It is not always possible to control every possible extraneous variable (e.g., competitors
may bring out a product during a test market).
Competitors aware of a company’s test market may knowingly change its marketing to
confound the test.
V. ETHICAL ISSUES IN EXPERIMENTATION
Although deception is necessary in most experiments, when subjects can be returned to their
prior condition through debriefing, then it is probably consistent with high moral standards.
If debriefing will not return subjects to their former condition because they are injured
significantly or truly psychologically harmed, then the experiment should not proceed.
In test-markets, when a company puts a product out for public consumption, competitors may
also now freely consume the product.
When attempts to interfere with a test market are aimed solely at invalidating test results or
infringing on some copyright protection, those acts are ethically questionable.
