1
Introduction to Econometrics, 3e (Stock)
Chapter 13 Experiments and Quasi–Experiments
13.1 Multiple Choice
1) The following are reasons for studying randomized controlled experiment in an econometrics course,
with the exception of
A) at a conceptual level, the notion of an ideal randomized controlled experiment provides a benchmark
against which to judge estimates of causal effects in practice.
B) when experiments are actually conducted, their results can be very influential, so it is important to
understand the limitations and threats to validity of actual experiments as well as their strength.
C) randomized controlled experiments in economics are common.
D) external circumstances sometimes produce what appears to be randomization.
2) Program evaluation
A) is conducted for most departments in your university/college about every seven years.
B) is the field of study that concerns estimating the effect of a program, policy, or some other intervention
or “treatment.”
C) tries to establish whether EViews, SAS or Stata work best for your econometrics course.
D) establishes rating systems for television programs in a controlled experiment framework.
3) In the context of a controlled experiment, consider the simple linear regression formulation Yi = β0 +
β1Xi + ui. Let the Yi be the outcome, Xi the treatment level, and ui contain all the additional determinants
of the outcome. Then
A) the OLS estimator of the slope will be inconsistent in the case of a randomly assigned Xi since there
are omitted variables present.
B) Xi and ui will be independently distributed if the Xi be are randomly assigned.
C) β0 represents the causal effect of X on Y when X is zero.
D) E(Y X = 0) is the expected value for the treatment group.
4) In the context of a controlled experiment, consider the simple linear regression formulation Yi = β0 +
β1Xi + ui. Let the Yi be the outcome, Xi the treatment level when the treatment is binary, and ui contain
all the additional determinants of the outcome. Then calling a differences estimator
A) makes sense since it is the difference between the sample average outcome of the treatment group and
the sample average outcome of the control group.
B) and the level estimator is standard terminology in randomized controlled experiments.
C) does not make sense, since neither Y nor X are in differences.
D) is not quite accurate since it is actually the derivative of Y on X.
5) The following does not represent a threat to internal validity of randomized controlled experiments:
A) attrition.
B) failure to follow the treatment protocol.
C) experimental effects.
D) a large sample size.
6) The Hawthorne effect refers to
A) subjects dropping out of the study after being randomly assigned to the treatment or control group.
B) the failure of individuals to follow completely the randomized treatment protocol.
C) the phenomenon that subjects in an experiment can change their behavior merely by being included in
the experiment.
D) assigning individuals, in part, as a result of their characteristics or preferences.
7) The following is not a threat to external validity:
A) the experimental sample is not representative of the population of interest.
B) the treatment being studied is not representative of the treatment that would be implemented more
broadly.
C) experimental participants are volunteers.
D) partial compliance with the treatment protocol.
8) Assume that data are available on other characteristics of the subjects that are relevant to determining
the experimental outcome. Then including these determinants explicitly results in
A) the limited dependent variable model.
B) the differences in means test.
C) the multiple regression model.
D) large scale equilibrium effects.
9) All of the following are reasons for using the differences estimator with additional regressors, with the
exception of
A) efficiency.
B) providing a check for randomization.
C) providing an adjustment for “conditional” randomization.
D) making the difference estimator easier to calculate than in the case of the differences estimator without
the additional regressors.
10) Experimental data are often
A) observational data.
B) binary data, in that the subject either does or does not respond to the treatment.
C) panel data.
D) time series data.
11) With panel data, the causal effect
A) cannot be estimated since correlation does not imply causation.
B) is typically estimated using the probit regression model.
C) can be estimated using the “differences–in–differences” estimator.
D) can be estimated by looking at the difference between the treatment and the control group after the
treatment has taken place.
12) Causal effects that depend on the value of an observable variable, say Wi,
A) cannot be estimated.
B) can be estimate by interacting the treatment variable with Wi.
C) result in the OLS estimator being inefficient.
D) requires use of homoskedasticity–only standard errors.
13) To test for randomization when Xi is binary,
A) you regress Xi, on all W‘s and compute the F–statistic for testing that all the coefficients on the W‘s are
zero. (The W‘s measure characteristics of individuals, and these are not affected by the treatment.)
B) is not possible, since binary variables can only be regressors.
C) requires reordering the observations randomly and re–estimating the model. If the coefficients remain
the same, then this is evidence of randomization.
D) requires seeking external validity for your study.
14) The following estimation methods should not be used to test for randomization when Xi, is binary:
A) linear probability model (OLS) with homoskedasticity–only standard errors.
B) probit.
C) logit.
D) linear probability model (OLS) with heteroskedasticity–robust standard errors.
15) In a quasi–experiment
A) quasi differences are used, i.e., instead of △Y you need to use ( – λ × ), where 0 < λ < 1.
B) randomness is introduced by variations in individual circumstances that make it appear as if the
treatment is randomly assigned.
C) the causal effect has to be estimated through quasi maximum likelihood estimation.
D) the t–statistic is no longer normally distributed in large samples.
16) Your textbooks gives several examples of quasi experiments that were conducted. The following is
not an example of a quasi experiment:
A) labor market effects of immigration.
B) effects on civilian earnings of military service.
C) the effect of cardiac catheterization.
D) the effect of unemployment on the inflation rate.
17) A repeated cross–sectional data set
A) is also referred to as panel data.
B) is a collection of cross–sectional data sets, where each cross–sectional data set corresponds to a different
time period.
C) samples identical entities at least twice.
D) is typically used for estimating the following regression model
Yit = β0 + β1Xit + β2W1,it + … + β1+ rWr,it + uit
18) For quasi–experiments,
A) there is a particularly important potential threat to internal validity, namely whether the “as if”
randomization in fact can be treated reliably as true randomization.
B) there are the same threats to internal validity as for true randomized controlled experiments, without
modifications.
C) there is little threat to external validity, since the populations are typically already different.
D) OLS estimation should not be used.
19) Experimental effects, such as the Hawthorne effect,
A) generally are not germane in quasi–experiments.
B) typically require instrumental variable estimation in quasi–experiments.
C) can be dealt with using binary variables in quasi–experiments.
D) are the most important threat to internal validity in quasi–experiments.
20) Heterogeneous population
A) implies that heteroskedasticity–robust standard errors must be used.
B) suggest that multiple characteristics must be used to describe the population.
C) effects can be captured through interaction terms.
D) refers to circumstances in which there is unobserved variation in the causal effect with the population.
21) If the causal effect is different for different people, then the population regression equation for a
binary treatment variable Xi, can be written as
A) Yi = β0 + β1Xi + ui.
B) Yi = β0 + β1iXi + ui.
C) Yi = β0i β1iXi + ui.
D) Yi = β0 + β1Gi + β2Dt + ui.
22) In the case of heterogeneous causal effects, the following is not true:
A) in the circumstances in which OLS would normally be consistent (when E(ui Xi) = 0), the OLS
estimator continues to be consistent.
B) OLS estimation using heteroskedasticity–robust standard errors is identical to TSLS.
C) the OLS estimator is properly interpreted as a consistent estimator of the average causal effect in the
population being studied.
D) the TSLS estimator in general is not a consistent estimator of the average causal effect if an individual’s
decision to receive treatment depends on the effectiveness of the treatment for that individual.
23) One of the major lessons learned in the chapter on experiments and quasi–experiments
A) is that there are almost no true experiments in economics and that quasi–experiments are a poor
substitute.
B) you should always use TSLS when estimating causal effects in quasi–experiments.
C) populations are always homogeneous.
D) is that the insights of experimental methods can be applied to quasi–experiments, in which special
circumstances make it seem “as if” randomization has occurred.
24) Quasi–experiments
A) provide a bridge between the econometric analysis of observational data sets and the statistical ideal
of a true randomized controlled experiment.
B) are not the same as experiments, and lessons learned from the use of the latter can therefore not be
applied to them.
C) most often use difference–in–difference estimators, which are quite different from OLS and
instrumental variables methods studied in earlier chapters of the book.
D) use the same methods as studied in earlier chapters of the book, and hence the interpretation of these
methods is the same.
25) The major distinction between the experiments and quasi–experiments chapter and earlier chapters is
the
A) frequent use of binary variables.
B) type of data analyzed and the special opportunities and challenges posed when analyzing experiments
and quasi–experiments.
C) superiority of TSLS over OLS.
D) use of heteroskedasticity–robust standard errors.
26) A potential outcome
A) is the outcome for an individual under a potential treatment.
B) cannot be observed because most individuals do not achieve their potential.
C) is the same as a causal effect.
D) is none of the above.
27) A causal effect for a single individual
A) can be deduced from the average treatment effect.
B) cannot be measured.
C) depends on observable variables only.
D) is observable since it is used as part of calculating the mean of individual causal effects.
28) Randomization based on covariates is
A) not of practical importance since individuals are hardly ever assigned in this fashion.
B) dependent on the covariances of the error term (serial correlation).
C) a randomization in which the probability of assignment to the treatment group depends on one of
more observable variables W.
D) eliminates the omitted variable bias when using the difference estimator based on Yi = β0 + β1Xi + ui,
where Y is the outcome variable and X is the treatment indicator.
29) Testing for the random receipt of treatment
A) is not possible, in general.
B) entails testing the hypothesis that the coefficients on W1i, …, Wri are non–zero in a regression of Xi on
W1i, …, Wr.
C) is not meaningful since the LHS variable is binary.
D) entails testing the hypothesis that the coefficients on W1i, …, Wri are zero in a regression of Xi on W1i,
…, Wr.
30) Failure to follow the treatment protocol means that
A) the OLS estimator cannot be computed.
B) instrumental variables estimation of the treatment effect should be used where the initial random
assignment is the instrument for the treatment actually received.
C) you should use the TSLS estimator and regress the outcome variable Y on the initial random
assignment in the first stage to get predicted values of the outcome variable.
D) the Hawthorne effect plays a crucial role.
31) Small sample sizes in an experiment
A) biases the estimators of the causal effect.
B) may pose a problem because the assumption that errors are normally distributed is dubious for
experimental data.
C) do not raise threats to the validity of confidence intervals as long as heteroskedasticity–robust standard
errors are used.
D) may affect confidence intervals but not hypothesis tests.
32) A repeated cross–sectional data set is
A) a collection of cross–sectional data sets, where each cross–sectional data set corresponds to a different
time period.
B) the same as a balanced panel data set.
C) what Card and Krueger used in their study of the effect of minimum wages on teenage employment.
D) time series.
33) In a sharp regression discontinuity design,
A) crossing the threshold influences receipt of the treatment but is not the sole determinant.
B) the population regression line must be linear above and below the threshold.
C) Xi will in general be correlated with ui.
D) receipt of treatment is entirely determined by whether W exceeds the threshold.
34) Threats to internal validity of quasi–experiments include
A) failure of randomization.
B) failure to follow the treatment protocol.
C) attrition.
D) all of the above with some modifications from true randomized controlled experiments.
13.2 Essays and Longer Questions
1) You want to study whether or not the use of computers in the classroom for elementary students has
an effect on performance. Explain in some detail how you would ideally set up such an experiment and
what threats to internal and external validity there might be.
2) Canada and the United States had approximately the same aggregate unemployment rates from the
1920s to 1981. In 1982, a two percentage point gap appears, which has roughly persisted until today, with
the Canadian unemployment rate in the third quarter of 2002 being 7.6 percent while the American rate
stood at 5.9 percent in the same period. Several authors have investigated this phenomenon. One study,
published in 1990, contained the following statement: “It is a clichė that, as compared to analysis in the
physical sciences, economic analysis is hampered by the lack of controlled experiments. In this regard,
study of the Canadian economy can be much facilitated by comparison with the behavior of the US …”
Discuss what the authors may have had in mind. List some potential threats to internal and external
validity when comparing aggregate unemployment rate behavior between countries.
3) Earnings functions provide a measure, among other things, of the returns to education. It has been
argued these regressions contain a serious omitted variable bias due to differences in abilities.
Furthermore, ability is hard to measure and bound to be highly correlated with years of schooling. Hence
the standard estimate of about a 10 percent return to every year of schooling is upward biased. Suggest
some ways to address this problem. One famous study looked at earnings of identical twins. Explain how
this can be viewed as a quasi–experiment, and mention some of the threats to internal and external
validity that such a study might encounter.
4) Describe the major differences between a randomized controlled experiment and a quasi–experiment.
5) Roughly ten percent of elementary schools in California have a system whereby 4th to 6th graders
share a common classroom and a single teacher (multi–age, multi–grade classroom). Suggest an
experimental design that would allow you to assess the effect of learning in this environment. Mention
some of the threats to internal and external validity and how you would attempt to circumvent these.
6) Assume for the moment that the student–teacher ratio effect on test scores was large enough that you
would advocate reducing class sizes in elementary schools. In 1996, the State of California reduced class
sizes from K–3 to no more than 20 students across all public elementary schools (Class Size Reduction Act) at
a cost of approximately $2 billion. In a short essay, discuss why the general equilibrium effects might
differ from the results obtained using experiments.