Unlock document.
This document is partially blurred.
Unlock all pages and 1 million more documents.
Get Access
1
CHAPTER 10
QUASI-EXPERIMENTAL DESIGNS AND PROGRAM EVALUATION
CHAPTER OUTLINE AND OBJECTIVES
I. Overview
II. True Experiments
A. Characteristics of True Experiments
In true experiments, researchers manipulate an independent variable with treatment and
comparison condition(s) and exercise a high degree of control (especially through random
assignment to conditions).
B. Obstacles to Conducting True Experiments in Natural Settings
Researchers may experience difficulty obtaining permission to conduct true experiments in
natural settings and gaining access to participants.
Although random assignment is perceived by some as unfair because it may deprive individuals
of a new treatment, it is still the best way and fairest way to determine if a new treatment is
effective.
C. Threats to Internal Validity Controlled by True Experiments
Threats to internal validity are confounds that serve as plausible alternative explanations for a
research finding.
Major classes of threats to internal validity include history, maturation, testing, instrumentation,
regression, subject attrition, selection, and additive effects with selection.
D. Problems That Even True Experiments May Not Control
Threats to internal validity that can occur in any study include contamination, experimenter
expectancy effects, and novelty effects.
Contamination occurs when information about the experiment is communicated between groups
of participants, which may lead to resentment, rivalry, or diffusion of treatment.
Experimenter expectancy effects occur when researchers’ biases and expectancies
unintentionally influence the results of a study.
Novelty effects, including Hawthorne effects, occur when people’s behavior changes simply
because an innovation (e.g., a treatment) produces excitement, energy, and enthusiasm.
Threats to external validity occur when treatment effects may not be generalized beyond the
particular people, setting, treatment, and outcome of the experiment.
2
III. Quasi-Experiments
Quasi-experiments provide an important alternative when true experiments are not possible.
Quasi-experiments lack the degree of control found in true experiments; most notably, quasi-
experiments typically lack random assignment.
Researchers must seek additional evidence to eliminate threats to internal validity when they do
quasi-experiments rather than true experiments.
The one-group pretest-posttest design is called a pre-experimental design or a bad experiment
because it has so little internal validity.
A. The Nonequivalent Control Group Design
In the nonequivalent control group design, a treatment group and a comparison group are
compared using pretest and posttest measures.
If the two groups are similar in their pretest scores prior to treatment but differ in their posttest
scores following treatment, researchers can more confidently make a claim about the effect of
treatment.
Threats to internal validity due to history, maturation, testing, instrumentation, and regression can
be controlled in a nonequivalent control group design.
B. Nonequivalent Control Group Design: The Langer and Rodin Study
Quasi-experiments often assess the overall effectiveness of a treatment that has many
components; follow-up research may then determine which components are critical for achieving
the treatment effect.
C. Threats to Internal Validity in the Nonequivalent Control Group Design
To interpret the findings in quasi-experimental designs, researchers examine the study to
determine if any threats to internal validity are present.
The threats to internal validity that must be considered when using the nonequivalent control
group design include additive effects with selection, differential regression, observer bias,
contamination, and novelty effects.
Although groups may be comparable on a pretest measure, this does not ensure that the groups
are comparable in all possible ways that are relevant to the outcome of the study.
D. The Issue of External Validity
Similar to internal validity, the external validity of research findings must be critically examined.
The best evidence for the external validity of research findings is replication with different
populations, settings, and times.
3
E. Interrupted Time-Series Designs
In a simple interrupted time-series design, researchers examine a series of observations both
before and after a treatment.
Evidence for treatment effects occurs when there are abrupt changes (discontinuities) in the time
series data at the time treatment was implemented.
The major threats to internal validity in the simple interrupted time-series design are history
effects and changes in measurement (instrumentation) that occur at the same time as the
treatment.
F. Time Series with Nonequivalent Control Group
In a time series with nonequivalent control group design, researchers make a series of
observations before and after treatment for both a treatment group and a comparable comparison
group.
V. Program Evaluation
Program evaluation is used to assess the effectiveness of human service organizations and provide
feedback to administrators about their services.
Program evaluators assess needs, process, outcome, and efficiency of social services.
The relationship between basic research and applied research is reciprocal.
Despite society’s reluctance to use experiments, true experiments and quasi-experiments can provide
excellent approaches for evaluating social reforms.
VI. Summary
REVIEW QUESTIONS AND ANSWERS
These review questions appear in the textbook (without answers) at the end of Chapter 10, and can be
used for a homework assignment or exam preparation. Answers to these questions appear in italic.
1. Identify two reasons why it might be especially important to carry out experiments in natural settings.
2. Explain how laboratory experiments and those in natural settings differ in control, external validity,
goals, and consequences.
9. Explain why it is essential to use a pretest in the nonequivalent control group design.
10. Explain how one threat to internal validity is controlled in the nonequivalent control group design, and
describe a threat to internal validity that is not controlled in this design.
11. Identify two reasons why we cannot conclude that the treatment and control groups in a
nonequivalent control group design are equivalent even when the pretest scores are the same for
both groups.
12. Explain the difference between a history threat to internal validity and what is called a “local history
effect” in the nonequivalent control group design.
13. What is the major evidence for an effect of the treatment in a simple interrupted time-series design,
and what are the major threats to internal validity in this design?
14. Explain how the addition of a nonequivalent control group to a simple interrupted time-series design
reduces the threat to the internal validity of the design.
15. Describe the type of information sought when evaluators ask each of the four questions typically
addressed in program evaluation.
CHALLENGE QUESTIONS AND ANSWERS
These questions appear in the textbook at the end of Chapter 10, and can be used for a homework
assignment, in-class discussion, or exam preparation. Answers to these questions appear in italic below.
[Answer to Challenge Question 1 also appears in the text.]
1. A quasi-experiment was used to determine whether multimedia instruction is effective. Two sections
of introductory psychology were taught by the same instructor, both in the early afternoon. In one
section (the treatment group), the instructor used multimedia instruction. In the other section, the
instructor covered the same material but did not use multimedia instruction. Students did not know
when they registered for the course whether multimedia instruction would be used, but the students
were not randomly assigned to sections. Students’ knowledge of the course material was assessed
using two forms of a comprehensive introductory psychology test. The comprehensive test can be
considered a reliable and valid test that can be used to compare the effectiveness of the instruction in
the two sections. The students in both sections were tested on the second day of class (the pretest)
and at the final (the posttest). Different forms of the test were used at the pretest and at the posttest.
A. What quasi-experimental design is used in the study?
B. The instructor initially considered doing a true experiment rather than a quasi-experiment.
Comment critically on the fairness of random assignment if you were arguing in favor of doing a
true experiment to test the effectiveness of multimedia instruction.
C. Explain why the quasi-experimental design used by the instructor is more effective than if the
instructor had tested only students who had received multimedia instruction. Identify one threat to
internal validity that was controlled in this study that would not have been controlled if only
students who received multimedia instruction had been tested.
7
9
B. What questions about the process of the program would be useful once the program was under
way to help ensure that the evaluation of the outcome of the program could be interpreted
appropriately?
monitor both these aspects of the program.
C. Explain how you would test the effectiveness of the proposed program if it were not possible to do
a true experiment.
ISSUES AND PROBLEMS FOR CLASS DISCUSSION
1. Just How Do They (We) Decide?
In this chapter students are introduced to the idea of using true experiments and quasi-experiments to
make decisions about the effectiveness of social reforms. We have found that one way to help
students appreciate the application of these ideas is to ask them to consider how decisions are made
at their institution. We have done this in the context of a large group discussion in class. The steps we
follow in this discussion are outlined below.
Step 1. We ask the students to identify recent changes that have been made in the past year or so at
their college or university. Students at one of our institutions generated the following list of campus
changes in the past year: introduction of a first-year seminar course in the general education
curriculum; revised guidelines for new member education in fraternal organizations (i.e., new
guidelines for pledging); review of the chapel program; selection of a new president; and the adoption
of a combined residential and academic program emphasizing diversity issues.
Step 2. We then ask students to use one or more of the changes to describe how decisions are
made. Students may need some help in articulating the decision-making process that the institution
uses. Our experience is that students readily recognize that there is an institutional mechanism for
making decisions and that the mechanism often relies on voting (or reaching consensus) about
adopting proposals that come forth from different groups on the campus (e.g., student congress,
faculty boards and committees, student development).
Step 3. Perhaps the most useful aspect of this discussion for teaching research methods occurs
when students are asked what types of evidence decision-makers on campus use to decide to make
a change in policy. The students typically have to work to answer this question; few of them are likely
to have direct experience with decision making on campus. One helpful guide for this part of the
10
discussion is to ask students to consider how decisions are made by our government institutions.
There are two keys to this phase of the discussion. The first is that decision-makers do use evidence
to make decisions (e.g., documentation submitted with proposals; opinions of recognized experts;
comparisons to similar programs at other institutions; discussion and debate among people making
the decision). The second, and more important, key to this phase of the discussion is that decision
makers rarely use true experiments or quasi-experiments to make institutional decisions—even more
than 40 years after Donald Campbell advocated for the use of true and quasi-experiments to test
social reforms. Or, as illustrated in Box 10.2, even when controlled experiments are used, decision
makers may base their decisions on factors other than treatment outcomes.
Step 4. The final step in the discussion begins by asking students to consider good reasons why
institutions do not make use of true experiments and quasi-experiments in decision making. Students
are then asked to consider what benefits could be realized if institutions used an experimental
approach, and under what conditions these benefits are most likely to be realized (i.e., what types of
programs are most amenable to experimental tests).
Step 5. The discussion can conclude by reminding students that (1) good decisions can be made
without using experimental tests, (2) program evaluation is a good example of how research methods
can be used in deciding what social programs are needed and which ones are most effective and
efficient and (3) Campbell’s recommended approach to reforms as experiments is underutilized.
2. Reading Research Critically
This research summary and the accompanying questions could be used in class for small group
discussion. The answers appear below. The research summary and questions appear on subsequent
pages to facilitate photocopying for class handouts.
Answers to Reading Research Critically: Creating New Recycling Habits
A. Describe the quasi-experimental design used in this study and the critical features of their
methodology that allow you to classify their design.
B. Explain (1) how contamination could be a problem in this study, (2) why unobtrusive
measurement was important, and (3) how the questionnaire could be used to rule out whether
Hawthorne effects could account for their findings.
11
12
Reading Research Critically
Read the following description of a research study to answer the questions that follow. [Based on an
article by: Holland, R. W., Aarts, H., & Langendam, D. (2006). Breaking and creating habits on the
working floor: A field-experiment on the power of implementation intentions. Journal of Experimental
Social Psychology, 42, 776-783. doi: 10.1016/j.jesp.2005.11.006]
Creating New Recycling Habits
The purpose of this research was to test a theory regarding how to change behavior habits; in this case,
Holland et al. (2006) wanted to break the “bad” habit of throwing used paper and plastic cups into the
trash in an office setting and create a new habit of throwing these things into recycling bins. According to
the theory they tested, changing habits requires “implementation intention,” which requires individuals to
develop a plan that links their behavioral goal with specific situational cues for that behavior (i.e., the
where, how, and when of the new behavior).
Holland et al. (2006) used a quasi-experimental design to test whether they could improve recycling at a
telecommunications company in the Netherlands. The company’s first effort at encouraging recycling was
not very successful. They had placed recycling bins throughout the company and communicated to
employees the recycling goal using meetings and letters. Unobtrusive investigations of employees’
personal wastebaskets, however, showed that high rates of paper and plastic cups in the trash persisted.
Some form of treatment was needed. Holland and his colleagues used six conditions to test their theory
of behavior change. In the implementation-intention (“Plan”) condition, following a questionnaire about
employees’ satisfaction with the building environment (to divert attention from the recycling goal of the
study), participants completed a measure designed to address the specific steps and situational cues
they would use to recycle paper and plastic. In a second implementation-intention condition, participants
were also given their own personal “eye-catching” recycling bin for their office after completing the
questionnaire and plan (“Plan + Bin”). Participants in the “Bin” condition received just the personal
recycling bin (to see if an eye-catching situational cue is sufficient for behavior change), and in a fourth
condition, participants received the bin plus completed the questionnaire. Finally, in two control
conditions, participants’ daily trash was observed without any of the treatments; one control group did,
however, complete the questionnaire. The control participants were expected to use the centrally-located
recycling bins.
Participants were 109 employees from six different departments of the company. Intact departments, not
individuals, were randomly assigned to the conditions, making this a quasi-experimental design. All of the
participants held administrative jobs and spent roughly the same amount of time at the office. Each had a
personal desk and wastebasket. The dependent variables were amount of paper trash and plastic cups in
the wastebaskets for these 109 employees, measured 5 days before the intervention, during Weeks 1
and 2 after the intervention, and 2 months following the intervention. The measures of trash were
unobtrusive–participants did not know their trash was examined after working hours.
The findings for the study were impressive. There were no differences in paper and plastic cups in the
trash among the six groups prior to the intervention. Following the intervention, however, the number of
plastic cups in employees’ trash dropped overall by 75% and paper (in kg) dropped by 80%, both to near
zero levels, for participants in the Plan and Plan + Bin conditions (a large discontinuity in the data). This
effect was observed at Week 1, Week 2, and 2 months later. There were no changes in the trash for
participants in the two control conditions. Interesting findings were observed for the Bin conditions:
Participants’ paper trash decreased for the 3 post-intervention times to levels similar to the Plan
conditions, but their plastic trash remained similar to the control conditions over time.
13
After summarizing their findings, Holland et al. (2006) noted that while simply providing employees a
personal recycling bin increased recycling (although findings were mixed for paper and plastic), this
intervention can be costly for companies. In contrast, simply asking individuals to develop a behavioral
plan, with specific situational cues for implementing their intentions, produced dramatic changes in their
recycling habits. The change in recycling behavior for these individuals was not short (i.e., just a brief
function of the intervention), but habit-forming.
A. Describe the quasi-experimental design used in this study and the critical features of their
methodology that allow you to classify their design.
B. Explain (1) how contamination could be a problem in this study, (2) why unobtrusive measurement
was important, and (3) how the questionnaire could be used to rule out whether Hawthorne effects
could account for their findings.
C. Why do you think Holland et al. (2006) believed it was important to go to the expense and trouble to
conduct follow-up observations, even up to 2 months following the intervention?
D. Identify the threats to internal validity that can be ruled out because of the quasi-experimental design
used in this study, and which threats remain. Explain how you would rule out these remaining threats
to internal validity.
3. More Challenges
A. Laboratory research has consistently shown that treatment X1 results in reliably better test
performance than treatment X2. The typical mean test performance for these two treatments is
presented in each of the two tables that follow. Your task is to complete the table on the left such
that the laboratory results would be shown to generalize to the nonlaboratory situation. Then,
complete the table on the right such that the laboratory results would be shown not to generalize.
Finally, state the role of the concept of an interaction effect (see Chapter 8) in differentiating
between the two tables you have completed.
Lab Results Do Generalize Lab Results Do Not Generalize
Lab Nonlab Lab Nonlab
X1 60 ? X1 60 ?
Treatment
X2 30 ? X2 30 ?
B. At a recent meeting of the Academic Affairs Board of your school, a proposal was submitted for a
program to implement a new interdisciplinary approach to the teaching of Spanish and the
cultural history component of required courses. The program will be modeled after existing
programs in French and German. The Spanish program will begin in the fall of next year.
Because of your extensive expertise, you have been asked to serve as a consultant to the
Academic Affairs Board. Your task is to develop a procedure whereby the effectiveness of this
new program can be evaluated. The board will obviously make the final decision, so what they
need from you is some alternative approaches and the pros and cons of each approach.
(1) The first plan you are to develop should use a “true” experimental approach, so you should
outline what a random groups design test of the new program would look like.
(2) Explain why the board is likely to reject this first plan. Then develop a quasi-experimental test
of the program, using either a nonequivalent control group design or a simple interrupted
time-series design.
15
C. One central issue in debates concerning health care is the issue of accountability. In the context
of psychotherapy, accountability could lead insurance companies to require that reimbursement
would be provided only for procedures that had been proven “clinically effective.” Clinically
effective treatments would be those that had been tested in controlled clinical trials conducted
and evaluated under generally accepted principles of scientific research.
(1) Consider first that you are a psychotherapist who has developed a treatment for a simple
phobia (an unreasonable fear of something, e.g., snakes). Explain briefly but clearly how you
would show that your treatment was “clinically effective” if you had only a few patients
available who were suffering from this particular phobia.
(2) Next, explain how you would test your treatment if you had access to a large metropolitan
clinic at which there was a backlog of 50 clients awaiting treatment for the specific phobia for
which your treatment is intended.
D. The chairperson of the student campus life committee at a nearby college wants to test whether a
proposed action by the committee will have the desired effect on student life. Specifically, the
committee has proposed to establish “quiet hours” each weekday evening in the dorms to try to
foster an atmosphere more conducive to studying. The chairperson wants to do a study to
determine if students’ grades improve when quiet hours are established. The Registrar’s Office
has agreed to allow access anonymously to students’ grades and this procedure has been
approved by the college's Institutional Review Board. The critical question is how to test the
comparison between quiet hours (treatment) and the typical atmosphere in the dorms on a
weekday evening (control). The chairperson asks you for advice about how best to carry out the
test of quiet hours. You and the chairperson agree that doing a true experiment is not likely to be
acceptable to the students. Given that you have decided not to do a true experiment, explain to
the chairperson how a quasi-experiment involving a time series design with a nonequivalent
control group could be used to provide a good test of the effectiveness of quiet hours.
