978-1111826925 Chapter 12 Lecture Note Part 2

Unlock access to all the studying documents.

View Full Document

VI.PRACTICAL EXPERIMENTAL DESIGN ISSUES

Basic Versus Factorial Experimental Designs

In basic experimental designs a single independent variable is manipulated to observe its

effect on a single dependent variable.

Factorial experimental designs are more sophisticated than basic experimental designs

and allow for an investigation of the interaction of two or more independent variables.

Laboratory Experiments

An experiment can be conducted in a natural setting (a field experiment) or in an artificial

setting (a laboratory experiment).

In a laboratory experiment the researcher has more complete control over the research

setting and extraneous variables.

Some laboratory experiments may be more controlled or artificial.

An example of a device used in a very controlled experiment is a tachistoscope,

which can be used to experiment with the visual impact of advertising,

packaging, and so on, by controlling the amount of time a visual image is

exposed to a subject.

Field Experiments

Field experiments are research projects involving manipulations that are implemented in

a natural environment.

Test markets are field experiments.

A researcher manipulates experimental variables but cannot possibly control all the

extraneous variables.

Generally, subjects know when they are participating in a laboratory experiment, but in

field experiments, subjects do not even know they have taken part in an experiment.

With field experiments, the consent is implied since subjects are not asked to do anything

other than their normal behavior.

Controlled store tests – products are put into stores in a number of small cities or into

selected supermarket chains.

Within-Subjects and Between-Subjects Designs

A basic question is how many treatments should a subject receive?

For economical reasons, the researcher may wish to apply multiple treatments to the same

subject, which is called a within-subjects design that involves repeated measures

because with each treatment the same subject is measured.

Between-subjects design – each person receives only one treatment combination and

each dependent variable is measured only once for every subject.

More costly but more advantageous.

Validity is higher because demand characteristics are greatly reduced.

Statistical analyses are simpler.

Results are easier to report and explain to management.

VII. ISSUES OF EXPERIMENTAL VALIDITY

Internal Validity

Internal validity exists to the extent that an experimental variable is truly responsible for

any variance in the dependent variable.

If the observed results were influenced or confounded by extraneous factors, the

researcher will have problems making valid conclusions about the relationship between

the experimental treatment and the dependent variable.

A lab experiment enhances internal validity because it maximizes control of outside

forces.

Manipulation Checks

Internal validity depends in large part on successful manipulations.

Manipulations should be carried out in a way that varies the experimental

variable over meaningfully different levels.

If the levels are too close together, the experiment may lack the power necessary

to observe differences in the dependent variable.

The validity of manipulations can often be checked with a manipulation check.

In business, it is often done by asking a survey question or two.

Should always be administered after dependent variables in self-response

format experiments to keep the manipulation check item from becoming

a troublesome demand characteristic.

Extraneous variables can jeopardize internal validity, and six types are:

1. history

2. maturation

3. testing

4. instrumentation

5. selection

6. mortality

History Effect

A history effect occurs when some change other than the experimental treatment

occurs during the course of an experiment that affects the dependent variable.

A common history effect occurs when competitors change their strategies during

a test marketing experiment.

Particularly prevalent in repeated measures experiments that take place over an

extended time.

A special case is the cohort effect, which refers to a change in the dependent

variable that occurs because members of one experimental group experienced

different historical situations than members of other experimental groups.

Maturation

Maturation effects are effects that are function of time and the naturally

occurring events that coincide with growth and experience.

Testing

Testing effects are also called pretesting effects because the initial measurement

or test alerts subjects in a way that affects their response to the experimental

treatments.

Only occur in a before-and-after study (i.e., one in which an initial baseline

measure is taken before an experimental treatment is administered).

May increase awareness of socially approved answers, increase attention to

experimental conditions, or make the subject more conscious than usual of the

dimensions of a problem.

Instrumentation

A change in the wording of questions, a change in interviewers, or a change in

other procedures used to measure the dependent variable causes an

instrumentation effect.

Problematic with any type of repeated measures design.

Selection

The selection effect is a simple bias that results from differential selection of

respondents for the comparison groups, or sample selection error, discussed

earlier.

Mortality

If an experiment is conducted over a period of a few weeks or more, some

sample bias may occur due to mortality, or sample attrition.

Attrition occurs when some subjects withdraw from the experiment before it is

completed.

Mortality effects may occur if subjects drop from one experimental treatment

group disproportionately from other groups.

External Validity

External validity is the accuracy with which experimental results can be generalized

beyond the experimental subjects.

Increased when the sample truly represents some population and when the results extend

to market segments or other groups of people.

The higher the external validity, the more one can count on the fact that any results

observed in an experiment will also be seen in the “real world.”

Lab experiments are associated with low external validity because the limited set of

experimental conditions does not adequately represent all the influences existing in the

real world.

Lack of external validity results in difficulty repeating the experiment with any change in

subjects, settings or time.

Student Subjects

Basic researchers often use college students as subjects.

Students are easily accessible, but they often are not representative of the total

population.

However, when behaviors are studied for which students have some particular

expertise, then they are appropriate.

Trade-Offs Between Internal and External Validity

Laboratory experiments with many controlled factors usually are high in internal validity,

while field experiments generally have less internal validity, but greater external validity.

Ideally, results from lab experiments would be followed up with some type of field test.

VIII. CLASSIFICATION OF EXPERIMENTAL DESIGNS

There are various types of experimental designs.

If only one variable is manipulated, the experiment is a basic experimental design.

If the experimenter wishes to investigate several levels of the independent variable (e.g., four

price levels), or to investigate the interaction effects of two or more independent variables,

then the experiment requires a complex, or statistical, experimental design.

Symbolism for Diagramming Experimental Designs

The following symbolism facilitates the description of the various experimental

designs:

X = exposure of a group to an experimental treatment.

O = observation or measurement of the dependent variable. If more than one

observation is taken, subscripts will be given to indicate temporal order.

R= random assignment of test units.

As we diagram designs utilizing these symbols, the reader should assume a time flow

from left to right.

Three Examples of Quasi-Experimental Designs

Quasi-experimental designs do not involve random allocation of subjects to treatment

combinations.

They do not qualify as true experimental designs because they do not adequately

control for the problems associated with loss of internal validity.

One-Shot Design

Also known as after-only design and is diagrammed as follows:

X O1

This one-shot design is a case study fraught with problems:

subjects participate because of voluntary self-selection or arbitrary

assignment

study lacks any kind of comparison or any means of controlling

extraneous influences

Under certain circumstances, though, it is the only viable choice.

One-Group Pretest-Posttest Design

O1X O2

This design offers a comparison on the same individuals before and after

treatment (i.e., training).

Although this is an improvement over the one-shot design, this design still has

several weaknesses that may jeopardize internal validity (i.e., maturation, testing

effect, and mortality).

However, despite its weaknesses, this design is used frequently in business

research.

Static Group Design

Each subject is identified as a member or either an experimental group or a

control group.

Experimental group is measured after being exposed to the treatment, and the

control group is measured without having been exposed to the treatment:

Experimental group: X O1

Control group: O2

The results of a static control group are computed by subtracting the observed

results in the control group from those in the experimental group (O1 – O2 ).

A major weakness of this design is that we have no assurance that the groups

were equal on variables of interest before the experimental group received the

treatment.

If the groups were selected arbitrarily by the investigator or if entry into either

group was voluntary, systematic differences between the groups could invalidate

the conclusions about the effect of the treatment.

Random assignment of subjects may minimize problems with group differences.

If the groups are established by the experimenter rather than existing as a

function of some other causation, the static group design is referred to as an

after-only design with control group.

On many occasions, an after-only design is the only possible option (i.e.,

conducting use tests for new products or brands).

Three Alternative Experimental Designs

In the next three designs, the symbol R to the left of the diagram indicates that

the first step in a true experimental design is the randomization of subject

assignment.

Pretest-Posttest Control Group Design (Before-After With Control)

This is the classic experimental design:

Experimental group: R O1X O2

Control group: R O3X O4

This design has the advantage of the before-after design with the additional

advantages gained from having a control group.

The effect of the experimental treatment equals (O2 – O1) – (O4 – O3).

It is assumed that the effect of extraneous variables will be the same on both

the experimental and the control groups.

This assumption is also made for effects of other events between the

before and after measurements (history), changes within the subjects that

occur with the passage of time (maturation), testing effects, and

instrumentation effects.

However, a testing effect is possible when subjects are sensitized to the

subject of the research.

This weakness in the before-after with control group design can be

corrected (see the next two designs).

Posttest-Only Control Group Design (After-Only With Control)

Experimental group: R X O1

Control group: R O2

The effect of the experimental treatment is equal to O2 – O1..

With only posttest measurement, the effects of testing and instrument

variation are eliminated.

Further, all the same assumptions about extraneous variables are made; that

is, they operate equally on both groups.

Compromise Designs

In many instances of business research true experimentation is not possible—

the best the researcher can do is approximate an experimental design.

A compromise design is one that falls short of assigning subjects or

treatments randomly to groups.

The alternative to the compromise design when random assignment of

subjects is not possible is to conduct the experiment without a control group.

Generally this is considered a greater weakness than utilizing groups that

have already been established.

When the experiment involves a longitudinal study, circumstances usually

dictate a compromise with true experimentation.

Time Series Designs

Experiments that are investigating long-term structural change may require a time

series design.

When experiments are conducted over long periods of time, they are most vulnerable

to historical changes.

In such cases the following quasi-design is utilized:

O1O2O3X O4O5 O6

Several observations are taken to identify trends before the treatment is

administered.

After the treatment, several observations are made to determine if the

patterns after the treatment are similar to those before.

Of course, this time series design cannot give the researcher complete

assurance that the treatment caused the change in the trend, but it does enable

the researcher to distinguish temporary changes from permanent changes.

Complex Experimental Designs

Complex experimental designs are statistical designs that isolate the effects of

confounding extraneous variables or allow for the manipulation of more than one

independent variable.

Completely Randomized Designs

A completely randomized design is an experimental design that uses a

random process to assign subjects to treatment levels of an experimental

variable.

Randomization is an attempt to control extraneous variables while

manipulating potential causes.

A random number process can be used to assign subjects to one of the

treatment groups.

Randomized Block Design

The randomized block design is an extension of the completely randomized

design.

A form of randomization is utilized to control for most extraneous variables.

However, if the researcher has identified a single extraneous variable that

might affect subjects’ responses systematically, then the researcher will

attempt to isolate the single variable by blocking out its effects.

A blocking variable is a categorical variable that is expected to be associated

with different values of a dependent variable for each group (e.g., biological

sex).

The term randomized block originated in agricultural research that applied

several levels of a treatment variable to each of several blocks of land.

In business research, the researcher may wish to isolate block effects such as

store size, territory location, market shares of the test brand or its major

competition, per capita consumption levels for a product class, city size, etc.

Factorial Designs

Allow for the testing of the effects of two or more treatments (factors) at

various levels.

Main effects are differences (in the dependent variable) between treatment

levels.

Interactions produce differences (in the dependent variable) between

experimental cells based on combinations of variables.

For example, with an experiment with three levels of one factor (e.g., three

levels of price) and two levels of another (e.g., two packaging designs), we

have a 3 x 2 (read “three by two”) factorial design because the first factor is

varied in three ways and the second factor is varied in two ways.

A 3 x 2 design requires six cells, or six experimental groups (3 x 2 =

6).

If the subjects each receive only one combination of experimental

variables, then we use the term 3 x 2 between-subjects design to

describe the experiment.

The number of treatments (factors) and the number of levels of each

treatment identify the factorial design.

The important idea is that each treatment level is combined with every other

treatment level.

In addition to the advantage of investigating two or more independent

variables simultaneously, factorial designs allow researchers to measure

interaction effects.

If the effect of one treatment differs at various levels of another

treatment, interaction occurs.