Unlock document.
This document is partially blurred.
Unlock all pages and 1 million more documents.
Get Access
VI.PRACTICAL EXPERIMENTAL DESIGN ISSUES
Basic Versus Factorial Experimental Designs
In basic experimental designs a single independent variable is manipulated to observe its
effect on a single dependent variable.
Factorial experimental designs are more sophisticated than basic experimental designs
and allow for an investigation of the interaction of two or more independent variables.
Laboratory Experiments
An experiment can be conducted in a natural setting (a field experiment) or in an artificial
setting (a laboratory experiment).
In a laboratory experiment the researcher has more complete control over the research
setting and extraneous variables.
Some laboratory experiments may be more controlled or artificial.
An example of a device used in a very controlled experiment is a tachistoscope,
which can be used to experiment with the visual impact of advertising,
packaging, and so on, by controlling the amount of time a visual image is
exposed to a subject.
Field Experiments
Field experiments are research projects involving manipulations that are implemented in
a natural environment.
Test markets are field experiments.
A researcher manipulates experimental variables but cannot possibly control all the
extraneous variables.
Generally, subjects know when they are participating in a laboratory experiment, but in
field experiments, subjects do not even know they have taken part in an experiment.
With field experiments, the consent is implied since subjects are not asked to do anything
other than their normal behavior.
Controlled store tests – products are put into stores in a number of small cities or into
selected supermarket chains.
Within-Subjects and Between-Subjects Designs
A basic question is how many treatments should a subject receive?
For economical reasons, the researcher may wish to apply multiple treatments to the same
subject, which is called a within-subjects design that involves repeated measures
because with each treatment the same subject is measured.
Between-subjects design – each person receives only one treatment combination and
each dependent variable is measured only once for every subject.
More costly but more advantageous.
Validity is higher because demand characteristics are greatly reduced.
Statistical analyses are simpler.
Results are easier to report and explain to management.
VII. ISSUES OF EXPERIMENTAL VALIDITY
Internal Validity
Internal validity exists to the extent that an experimental variable is truly responsible for
any variance in the dependent variable.
If the observed results were influenced or confounded by extraneous factors, the
researcher will have problems making valid conclusions about the relationship between
the experimental treatment and the dependent variable.
A lab experiment enhances internal validity because it maximizes control of outside
forces.
Manipulation Checks
Internal validity depends in large part on successful manipulations.
Manipulations should be carried out in a way that varies the experimental
variable over meaningfully different levels.
If the levels are too close together, the experiment may lack the power necessary
to observe differences in the dependent variable.
The validity of manipulations can often be checked with a manipulation check.
In business, it is often done by asking a survey question or two.
Should always be administered after dependent variables in self-response
format experiments to keep the manipulation check item from becoming
a troublesome demand characteristic.
Extraneous variables can jeopardize internal validity, and six types are:
1. history
2. maturation
3. testing
4. instrumentation
5. selection
6. mortality
History Effect
A history effect occurs when some change other than the experimental treatment
occurs during the course of an experiment that affects the dependent variable.
A common history effect occurs when competitors change their strategies during
a test marketing experiment.
Particularly prevalent in repeated measures experiments that take place over an
extended time.
A special case is the cohort effect, which refers to a change in the dependent
variable that occurs because members of one experimental group experienced
different historical situations than members of other experimental groups.
Maturation
Maturation effects are effects that are function of time and the naturally
occurring events that coincide with growth and experience.
Testing
Testing effects are also called pretesting effects because the initial measurement
or test alerts subjects in a way that affects their response to the experimental
treatments.
Only occur in a before-and-after study (i.e., one in which an initial baseline
measure is taken before an experimental treatment is administered).
May increase awareness of socially approved answers, increase attention to
experimental conditions, or make the subject more conscious than usual of the
dimensions of a problem.
Instrumentation
A change in the wording of questions, a change in interviewers, or a change in
other procedures used to measure the dependent variable causes an
instrumentation effect.
Problematic with any type of repeated measures design.
Selection
The selection effect is a simple bias that results from differential selection of
respondents for the comparison groups, or sample selection error, discussed
earlier.
Mortality
If an experiment is conducted over a period of a few weeks or more, some
sample bias may occur due to mortality, or sample attrition.
Attrition occurs when some subjects withdraw from the experiment before it is
completed.
Mortality effects may occur if subjects drop from one experimental treatment
group disproportionately from other groups.
External Validity
External validity is the accuracy with which experimental results can be generalized
beyond the experimental subjects.
Increased when the sample truly represents some population and when the results extend
to market segments or other groups of people.
The higher the external validity, the more one can count on the fact that any results
observed in an experiment will also be seen in the “real world.”
Lab experiments are associated with low external validity because the limited set of
experimental conditions does not adequately represent all the influences existing in the
real world.
Lack of external validity results in difficulty repeating the experiment with any change in
subjects, settings or time.
Student Subjects
Basic researchers often use college students as subjects.
Students are easily accessible, but they often are not representative of the total
population.
However, when behaviors are studied for which students have some particular
expertise, then they are appropriate.
Trade-Offs Between Internal and External Validity
Laboratory experiments with many controlled factors usually are high in internal validity,
while field experiments generally have less internal validity, but greater external validity.
Ideally, results from lab experiments would be followed up with some type of field test.
VIII. CLASSIFICATION OF EXPERIMENTAL DESIGNS
There are various types of experimental designs.
If only one variable is manipulated, the experiment is a basic experimental design.
If the experimenter wishes to investigate several levels of the independent variable (e.g., four
price levels), or to investigate the interaction effects of two or more independent variables,
then the experiment requires a complex, or statistical, experimental design.
Symbolism for Diagramming Experimental Designs
The following symbolism facilitates the description of the various experimental
designs:
X = exposure of a group to an experimental treatment.
O = observation or measurement of the dependent variable. If more than one
observation is taken, subscripts will be given to indicate temporal order.
R= random assignment of test units.
As we diagram designs utilizing these symbols, the reader should assume a time flow
from left to right.
Three Examples of Quasi-Experimental Designs
Quasi-experimental designs do not involve random allocation of subjects to treatment
combinations.
They do not qualify as true experimental designs because they do not adequately
control for the problems associated with loss of internal validity.
One-Shot Design
Also known as after-only design and is diagrammed as follows:
X O1
This one-shot design is a case study fraught with problems:
subjects participate because of voluntary self-selection or arbitrary
assignment
study lacks any kind of comparison or any means of controlling
extraneous influences
Under certain circumstances, though, it is the only viable choice.
One-Group Pretest-Posttest Design
O1X O2
This design offers a comparison on the same individuals before and after
treatment (i.e., training).
Although this is an improvement over the one-shot design, this design still has
several weaknesses that may jeopardize internal validity (i.e., maturation, testing
effect, and mortality).
However, despite its weaknesses, this design is used frequently in business
research.
Static Group Design
Each subject is identified as a member or either an experimental group or a
control group.
Experimental group is measured after being exposed to the treatment, and the
control group is measured without having been exposed to the treatment:
Experimental group: X O1
Control group: O2
The results of a static control group are computed by subtracting the observed
results in the control group from those in the experimental group (O1 - O2 ).
A major weakness of this design is that we have no assurance that the groups
were equal on variables of interest before the experimental group received the
treatment.
If the groups were selected arbitrarily by the investigator or if entry into either
group was voluntary, systematic differences between the groups could invalidate
the conclusions about the effect of the treatment.
Random assignment of subjects may minimize problems with group differences.
If the groups are established by the experimenter rather than existing as a
function of some other causation, the static group design is referred to as an
after-only design with control group.
On many occasions, an after-only design is the only possible option (i.e.,
conducting use tests for new products or brands).
Three Alternative Experimental Designs
In the next three designs, the symbol R to the left of the diagram indicates that
the first step in a true experimental design is the randomization of subject
assignment.
Pretest-Posttest Control Group Design (Before-After With Control)
This is the classic experimental design:
Experimental group: R O1X O2
Control group: R O3X O4
This design has the advantage of the before-after design with the additional
advantages gained from having a control group.
The effect of the experimental treatment equals (O2 - O1) - (O4 - O3).
It is assumed that the effect of extraneous variables will be the same on both
the experimental and the control groups.
This assumption is also made for effects of other events between the
before and after measurements (history), changes within the subjects that
occur with the passage of time (maturation), testing effects, and
instrumentation effects.
However, a testing effect is possible when subjects are sensitized to the
subject of the research.
This weakness in the before-after with control group design can be
corrected (see the next two designs).
Posttest-Only Control Group Design (After-Only With Control)
Experimental group: R X O1
Control group: R O2
The effect of the experimental treatment is equal to O2 – O1..
With only posttest measurement, the effects of testing and instrument
variation are eliminated.
Further, all the same assumptions about extraneous variables are made; that
is, they operate equally on both groups.
Compromise Designs
In many instances of business research true experimentation is not possible—
the best the researcher can do is approximate an experimental design.
A compromise design is one that falls short of assigning subjects or
treatments randomly to groups.
The alternative to the compromise design when random assignment of
subjects is not possible is to conduct the experiment without a control group.
Generally this is considered a greater weakness than utilizing groups that
have already been established.
When the experiment involves a longitudinal study, circumstances usually
dictate a compromise with true experimentation.
Time Series Designs
Experiments that are investigating long-term structural change may require a time
series design.
When experiments are conducted over long periods of time, they are most vulnerable
to historical changes.
In such cases the following quasi-design is utilized:
O1O2O3X O4O5 O6
Several observations are taken to identify trends before the treatment is
administered.
After the treatment, several observations are made to determine if the
patterns after the treatment are similar to those before.
Of course, this time series design cannot give the researcher complete
assurance that the treatment caused the change in the trend, but it does enable
the researcher to distinguish temporary changes from permanent changes.
Complex Experimental Designs
Complex experimental designs are statistical designs that isolate the effects of
confounding extraneous variables or allow for the manipulation of more than one
independent variable.
Completely Randomized Designs
A completely randomized design is an experimental design that uses a
random process to assign subjects to treatment levels of an experimental
variable.
Randomization is an attempt to control extraneous variables while
manipulating potential causes.
A random number process can be used to assign subjects to one of the
treatment groups.
Randomized Block Design
The randomized block design is an extension of the completely randomized
design.
A form of randomization is utilized to control for most extraneous variables.
However, if the researcher has identified a single extraneous variable that
might affect subjects’ responses systematically, then the researcher will
attempt to isolate the single variable by blocking out its effects.
A blocking variable is a categorical variable that is expected to be associated
with different values of a dependent variable for each group (e.g., biological
sex).
The term randomized block originated in agricultural research that applied
several levels of a treatment variable to each of several blocks of land.
In business research, the researcher may wish to isolate block effects such as
store size, territory location, market shares of the test brand or its major
competition, per capita consumption levels for a product class, city size, etc.
Factorial Designs
Allow for the testing of the effects of two or more treatments (factors) at
various levels.
Main effects are differences (in the dependent variable) between treatment
levels.
Interactions produce differences (in the dependent variable) between
experimental cells based on combinations of variables.
For example, with an experiment with three levels of one factor (e.g., three
levels of price) and two levels of another (e.g., two packaging designs), we
have a 3 x 2 (read “three by two”) factorial design because the first factor is
varied in three ways and the second factor is varied in two ways.
A 3 x 2 design requires six cells, or six experimental groups (3 x 2 =
6).
If the subjects each receive only one combination of experimental
variables, then we use the term 3 x 2 between-subjects design to
describe the experiment.
The number of treatments (factors) and the number of levels of each
treatment identify the factorial design.
The important idea is that each treatment level is combined with every other
treatment level.
In addition to the advantage of investigating two or more independent
variables simultaneously, factorial designs allow researchers to measure
interaction effects.
If the effect of one treatment differs at various levels of another
treatment, interaction occurs.
