978-1111826925 Chapter 12 Lecture Note Part 1

Unlock access to all the studying documents.

View Full Document

Chapter 12

Experimental Research

AT-A-GLANCE

I. Creating an Experiment

A. An illustration: Can a Self-Efficacy Intervention Enhance Job Attitude?

Experimental subjects

Independent variables

Experimental outcome

Independent variable main effects and interaction

II. Designing an Experiment to Minimize Experimental Error

A. Manipulation of the independent variable

Experimental and control groups

Several experimental treatment levels

More than one independent variable

Repeated measures

B. Selection and measurement of the dependent variable

C. Selection and assignment of test units

Sample selection and random sampling error

Randomization

Matching

Control over extraneous variables

Experimental confounds

Extraneous variables

III. Demand Characteristics

A. What are demand characteristics?

B. Experimenter bias and demand effects

C. Hawthorne effect

D. Reducing demand characteristics

Experimental disguise

Isolate experimental subjects

Use a “blind” experimental administrator

Administer only one experimental condition per subject

IV. Establishing Control

A. Problems controlling extraneous variables

V. Ethical Issues in Experimentation

VI. Practical Experimental Design Issues

A. Basic versus factorial experimental designs

B. Laboratory experiments

C. Field experiments

D. Within-subject and between-subjects designs

VII. Issues of Experimental Validity

A. Internal validity

Manipulation checks

History

Maturation

Testing

Instrumentation

Selection

Mortality

B. External validity

Student subjects

C. Trade-offs between internal and external validity

VIII. Classification of Experimental Designs

A. Symbolism for diagramming experimental designs

B. Three examples of quasi-experimental designs

One-shot design

One-group pretest-posttest design

Static group design

C. Three alternative experimental designs

Pretest-posttest control group design (Before-after with control)

Posttest-only control group design (After-only with control)

Compromise designs

D. Time series designs

E. Complex experimental designs

Completely randomized design

Randomized-block design

Factorial designs

LEARNING OUTCOMES

1. Identify the independent variable, dependent variable, and construct a valid simple experiment to

assess a cause and effect relationship

2. Understand and minimize the systematic experimental error

3. Know ways of minimizing experimental demand characteristics

4. Avoid unethical experimental practices

5. Understand the advantages of between-subjects experimental design

6. Weigh the trade-off between internal and external validity

7. Use manipulations to implement a completely randomized experimental design, a randomized-block

design, and a factorial experimental design

CHAPTER OPENING VIGNETTE: Testing Web Protocols for Financial

Markets

Technology has drastically changed the way we conduct banking and related financial services. For most

of us, how it all happens is not that relevant. However, for information technology directors and financial

managers, how this process occurs—and how to make it occur faster and cheaper—is vitally important.

An experiment of two protocols (FIX and SOAP) for transferring information over computer networks is

described. Researchers designed a laboratory experiment to compare the relative performance of the two

in business computing scenarios. While the results are not important to us, an understanding of the

process of testing these two approaches is very important to business researchers.

SURVEY THIS!

Students are asked to look at the questions on prospective careers. The questions actually represent a very

simple experimental design in which the type of occupation described may cause the subjects’ responses

to these questions. Students are asked to look at the data and see if they can determine whether or not

students’ beliefs about careers are altered by the type of occupation they were assigned to rate. Here, job

type becomes the experimental manipulation. Do you know the treatment levels?

RESEARCH SNAPSHOTS

Talking While Driving: Are Cell Phone Conversations Different from Passenger Conversations?

The increased use of cell phones while driving has led to increased driver distractions and the

potential for accidents and even fatalities. But people engage in conversations with fellow

passengers as they drive as well. Researchers designed an experiment with a driving simulator to

test whether engaging in a conversation on a cell phone is more dangerous. Students were

assigned to one of three groups: a group with no passenger as a control group, a group that had a

passenger who asked them questions while they simulated driving, and a group where the driver

engaged in a cell phone conversation. The results indicate that driving while engaged in a

conversation on a cell phone was indeed much more distracting and dangerous than having a

conversation with a passenger, perhaps because a passenger can anticipate changing driving

conditions and adjust their engagement in conversation accordingly.

The Hidden in Hidden Valley Ranch

Hidden Valley Ranch (HVR) conducted a field market experiment to test three new flavors of

salad dressings (i.e., three levels of the experimental variable flavor). These types of tests are

expensive because small batches of each flavor is produced, bottled, and shipped to sales

representative who stock the dressings in the stores. The first day involved placing the products

in retail store. On the second day, the reps returned to record sales and saw that all flavors had

sold. It turns out that a competitor bought every bottle, and HVR was unable to produce any sales

data (the dependent variable). Moreover, the competitor was able to determine the recipe in their

labs. So the risks with field tests are: no more secrets once the product is for sale and the risk of

espionage rendering the experiment invalid.

OUTLINE

I. CREATING AN EXPERIMENT

The purpose of experimental research is to allow the investigator to control the research

situation so that causal relationships among variables may be evaluated.

The experimenter manipulates one or more independent variables and holds constant all other

possible independent variables while observing the effects on dependent variables.

Independent variables are expected to determine the outcomes of interest.

Dependent variables are the outcomes of interest.

An Illustration: Can a Self-Efficacy Intervention Enhance Job Attitude?

An experiment investigating how self-efficacy might influence employee’s attitude

toward their job is described (Self-efficacy is a person’s confidence and belief in their

own abilities to accomplish tasks at hand).

Experimental Subjects

Participants in experimental research are referred to as subjects rather than

respondents because the experimenter subjects them to some experimental treatment.

In this experiment, 35 of the subjects were given positive feedback and

encouragement from their supervisors as the experimental treatment. The other 36

subjects were not provided the positive feedback.

Independent Variables

Experiment involved one relevant independent variable—whether or not the

employee received positive feedback.

While not a true independent variable, the length of time each employee had worked

with the firm was also important to the researchers.

oVariables such as this are referred to as blocking variables—a categorical (less

than interval) variable that is not manipulated as in an experimental variable but

is included in the statistical analysis of experiments.

Experimental condition refers to one of the possible levels of an experimental

variable manipulation.

Subjects were divided into “newcomers” (the new employees) and “insiders” (current

employees) and then randomly assigned to either the treatment condition or the

control group.

Differences between groups are analyzed.

Experimental Outcome

The key outcomes, or dependent variables, is the subject’s job satisfaction (only one

looked at in this illustration), organizational commitment, professional commitment,

intent to quit the organization, and intent to quit the profession.

In addition, the researchers followed up later to see if the subjects had actually left

the firm.

A rating scale assessed job satisfaction, with higher scores representing higher job

satisfaction.

oMeans are given in Exhibit 12.2.

oLooks like the difference in job satisfaction is primarily between the current and

new employees.

Independent Variable Main Effects and Interaction

Length of time that the employee works at the firm clearly appears to matter, but

attempts to enhance self-efficacy shouldn’t be dismissed so quickly.

The research must examine both the effects of each experimental variable alone and

the effects due to combinations of variables.

A main effect refers to the experimental difference in means between the different

levels of any single experimental variable.

An interaction effect is due to a specific combination of independent variables.

oIt’s possible that the combination of length of employment and self-efficacy

treatment creates effects that are not clearly represented in the main effects.

Experimental results are often shown with a line graph (see Exhibit 12.3).

oMain effects are illustrated when the lines are at different heights.

oWhen the lines have very different slopes, an interaction is likely present.

In this example:

oThe worst situation is the current employees who do not receive positive

feedback.

oThe best scenario occurs when the treatment is given to new employees.

oIt appears that job satisfaction decreases over time, and the benefit of the

self-efficacy treatment is greater for the employees that have been with the

organization than for new employees.

II. DESIGNING AN EXPERIMENT TO MINIMIZE EXPERIMENTAL ERROR

Experimental designs involve no less than four important design elements:

1. Manipulation of the independent variable(s).

2. Selection and measurement of the dependent variable(s).

3. Selection and assignment of experimental subjects.

4. Control over extraneous variables.

Manipulation of the Independent Variable

The researcher creates the values of the independent variables.

Experimental independent variables are hypothesized to be causal influences.

An experimental treatment is the term referring to the way an experimental variable is

manipulated, and it often involves treatments with more than two levels.

Experimental variables are often categorical variables that take on a value to represent

some classifiable or qualitative aspect (e.g., protocol is either FIX or SOAP – see chapter

vignette).

Independent variables may truly be continuous variables, and the researcher must select

appropriate levels of that variable as experimental treatments.

Researcher decides on levels that would be relevant to study.

Levels should be noticeably different and realistic.

Experimental and Control Groups

In the simplest experiment, an independent variable is manipulated over two

treatment levels resulting in two groups: an experimental group and a control

group.

Experimental group – one in which an experimental treatment is administered.

Control group – one in which no experimental treatment is administered.

By holding conditions constant in the control group, the researcher controls for

potential sources of error in the experiment.

Several Experimental Treatment Levels

By analyzing more groups each with a different treatment level, a more precise

result may be obtained than in the simple experimental group-control group

experiment described above.

May still involve a control group.

Design can produce only a main effect.

More Than One Independent Variable

Include the effect of another experimental variable.

The term cell is used to refer to treatment combinations within an experiment.

The number of cells involved in any experiment can be easily computed as

follows: K = (T1)(T2)…(Tm).

K = the number of cells

T1 = the number of treatment levels for experimental group number one.

T2 = the number of treatment levels for experimental group number two

and so on.

Including multiple variables allows a comparison of experimental treatments on

the dependent variables.

This design involves both main effects and interactions.

Repeated Measures

Repeated measures designs – experiments in which an individual subject is

exposed to more than one level of an experimental treatment.

Possesses several drawbacks even though it is more economical.

Selection and Measurement of the Dependent Variable

Choosing the right dependent variable is part of the problem definition process.

The amount of time needed for effects to become evident should be considered.

Selection and Assignment of Test Units

Test units are the subjects or entities whose responses to the experimental treatment are

measured or observed (i.e. individual consumers, employees, organizational units, sales

territories, market segments, brands, stores, etc.).

People are the most common test units in most business experiments.

Sample Selection and Random Sample Errors

As in other forms of research, random sampling errors and sample selection

errors may occur.

Sample selection error occurs because of flaws in procedures used to assign

experimental test units.

Systematic or non-sampling error may occur if the sampling units in an

experimental cell are somehow different than the units in another cell, and this

difference affects the dependent variable.

Randomization

Randomization – the random assignment of subject and treatments to groups –

is one device for equally distributing the effects of extraneous variables to all

conditions.

Nuisance variables—items that may affect the dependent measures but are not

of primary interest—will not be eliminated, but they will be controlled because

they are likely to exist to the same degree in every experimental cell.

Matching

Assigning subjects in a way that their characteristics are the same in each group.

This is best thought of in terms of demographic characteristics (e.g., If a subject’s

sex is expected to influence the dependent variable, make sure there are equal

numbers of men and women in each experimental cell.).

While useful, the researcher can never be sure that sampling units are matched on

all characteristics.

Control Over Extraneous Variables

This is related to the various types of experimental error.

Recall that total survey error was classified into two basic types: random

sampling error and systematic error.

Same dichotomy applies to all research designs, but the terms random (sampling)

error and systematic error are more frequently used when discussing

experiments.

Experimental Confounds

A confound in an experiment means that there is an alternative explanation

beyond the experimental variables for any observed differences in the dependent

variable.

Once identified, the validity of the experiment is severely questioned.

Extraneous Variables

Marketing mix variables (i.e., price, product, promotion, and distribution)

interact with uncontrollable forces in the market (i.e., competitors’ activities,

consumer trends).

Thus, many marketing experiments are subject to the effect of extraneous

variables.

Must be identified before the experiment if at all possible so the experimenter

can control or eliminate such variables.

III. DEMAND CHARACTERISTICS

What Are Demand Characteristics?

Demand characteristic – an experimental design element that unintentionally provides

subjects with hints about the research hypothesis.

Knowledge of the experimental hypothesis creates a particular type of confound known

as a demand effect.

Experimenter Bias and Demand Effects

Demand characteristics are aspects of an experiment that demand (encourage) that the

subjects respond in a particular way  a source of systematic error.

Prominent demand characteristics are often presented by the person administering

experimental procedures.

Experimenter bias – an experimenter’s presence, actions, or comments influence the

subjects’ behavior or sway the subjects to slant their answers to cooperate with the

experimenter.

Hawthorne Effect

Hawthorne Effect – people will perform differently when they know they are

experimental subjects.

Named after a famous management experiment that attempted to study the effects on

productivity of various working conditions (i.e., hours of work, rest periods, lighting,

methods of pay) at the Western Electric Hawthorne plant in Cicero, IL.

Researchers found that workers’ productivity increased regardless of the

manipulation.

Investigators realized that the workers’ morale was higher because they were

aware of being part of a special experimental group.

Social interaction should be restricted in laboratory experiments because conversations

might produce joint decisions rather than a desired individual one.

Reducing Demand Characteristics

It is practically impossible to eliminate demand characteristics, but the following steps

can by taken to reduce them:

1. Use an experimental disguise.

2. Isolate experimental subjects.

3. Use a “blind” experimental administrator.

4. Administer only one experimental treatment level to each subject.

Experimental Disguise

Subjects can be told that the purpose of the experiment is somewhat different

than the actual purpose.

Most often, they are simply told less than the complete “truth” about what is

going to happen.

In other cases, more deceit may be needed.

A placebo is an experimental deception involving a false treatment.

A placebo effect refers to the corresponding effect in a dependent

variable that is due to the psychological impact that goes along with

knowledge of the treatment.

Isolate Experimental Subjects

Discussion among subjects may lead them to guess the experimental hypotheses.

Integrity will be higher when each only knows enough to participate in the

experiment.

Use a “Blind” Experimental Administrator

The people administering the experiment may not be told the hypotheses.

Less likely to give off clues that result in demand effects.

Administer Only One Experimental Treatment Per Subject

When subjects know more than one experimental treatment condition, they are

much more likely to guess the experimental hypotheses.

IV. ESTABLISHING CONTROL

The major difference between experimental research and descriptive research is an

experimenter’s ability to control variables by either holding conditions constant or

manipulating the experimental variable.

When extraneous variables cannot be eliminated, experimenters may strive for constancy of

conditions, which means that subjects in all experimental groups are exposed to identical

conditions except for the differing experimental treatments.

If an experimental method requires that the same subjects be exposed to two or more

experimental treatments, an error may occur due to the order of presentation.

Counterbalancing attempts to eliminate the confounding effects of order of presentation.

Problems Controlling Extraneous Variables

It is not always possible to control every possible extraneous variable (e.g., competitors

may bring out a product during a test market).

Competitors aware of a company’s test market may knowingly change its marketing to

confound the test.

V. ETHICAL ISSUES IN EXPERIMENTATION

Although deception is necessary in most experiments, when subjects can be returned to their

prior condition through debriefing, then it is probably consistent with high moral standards.

If debriefing will not return subjects to their former condition because they are injured

significantly or truly psychologically harmed, then the experiment should not proceed.

In test-markets, when a company puts a product out for public consumption, competitors may

also now freely consume the product.

When attempts to interfere with a test market are aimed solely at invalidating test results or

infringing on some copyright protection, those acts are ethically questionable.