1
CHAPTER 5
SURVEY RESEARCH
CHAPTER OUTLINE AND OBJECTIVES
I. Overview
II. Uses of Surveys
Survey research is used to assess people’s thoughts, opinions, and feelings.
Surveys can be specific and limited in scope or more global in their goals.
The best way to determine whether results of a survey are biased is to examine the survey
procedures and analyses.
III. Characteristics of Surveys
Survey research involves selecting a sample (or samples) and using a predetermined set of
questions.
IV. Sampling in Survey Research
Careful selection of a survey sample allows researchers to generalize findings from the sample to the
population.
A. Basic Terms of Sampling
The identification and selection of elements that will make up the sample is at the heart of all
sampling techniques; the sample is chosen from the sampling frame, or list of all members of the
population of interest.
Researchers are not interested simply in the responses of those surveyed; instead, they seek to
describe the larger population from which the sample was drawn.
The ability to generalize from a sample to the population depends critically on the
representativeness of the sample.
A biased sample is one in which the characteristics of the sample are systematically different
from the characteristics of the population.
Selection bias occurs when the procedures used to select a sample result in the
overrepresentation or underrepresentation of some segment(s) of the population.
B. Approaches to Sampling
Two approaches to selecting a survey sample are probability sampling and nonprobability
sampling.
2
Probability sampling is the method of choice for obtaining a representative sample.
In simple random sampling, each element of the population has an equal chance of being
included in the sample; in stratified random sampling, the population is divided into
subpopulations (strata), and random samples are drawn from the strata.
Nonprobability sampling (such as convenience sampling) does not guarantee that every element
in the population has an equal chance of being included in the sample.
V. Survey Methods
Four methods for obtaining survey data are mail surveys, personal interviews, telephone interviews,
and Internet surveys.
A. Mail Surveys
Although mail surveys are quick and convenient, there may be a problem with the responses rate
when individuals fail to complete and return the survey.
Due to problems with response rate, the final sample for a mail survey may not represent the
population.
B. Personal Interviews
Although costly, personal interviews allow researchers to gain more control over how the survey
is administered.
Interviewer bias occurs when survey responses are recorded inaccurately or when interviewers
guide individuals’ responses.
C. Telephone Interviews
Despite some disadvantages, telephone interviews are used frequently for brief surveys.
D. Internet Surveys
The Internet offers several advantages for survey research because it is an efficient, low-cost
method for obtaining survey responses from large, potentially diverse and underrepresented
samples.
Disadvantages associated with Internet survey research include the potential for response rate
bias and selection bias, and lack of control over the research environment.
VI. Survey-Research Designs
The three types of survey design are the cross-sectional design, the successive independent samples
design, and the longitudinal design.
A. Cross-Sectional Design
3
In the cross-sectional design, one or more samples are drawn from the population(s) at one time.
Cross-sectional designs allow researchers to describe the characteristics of a population or the
differences between two or more populations, and correlational findings from cross-sectional
designs allow researchers to make predictions.
B. Successive Independent Samples Design
In the successive independent samples design, different samples of respondents from the
population complete the survey over a time period.
The successive independent samples design allows researchers to study changes in a population
over time.
The successive independent samples design does not allow researchers to infer how individual
respondents have changed over time.
A problem with the successive independent samples design occurs when the samples drawn
from the population are not comparable—that is, not equally representative of the population.
C. Longitudinal Design
In the longitudinal design, the same respondents are surveyed over time in order to examine
changes in individual respondents.
Because of the correlational nature of survey data, it is difficult to identify the causes of
individuals’ changes over time.
As people drop out of the study over time (attrition), the final sample may no longer be
comparable to the original sample or represent the population.
VII. Questionnaires
A. Questionnaires as Instruments
Most survey research relies on the use of questionnaires to measure variables.
Demographic variables describe the characteristics of people who are surveyed.
The accuracy and precision of questionnaires requires expertise and care in their construction.
Self-report scales are used to assess people’s preferences or attitudes.
B. Reliability and Validity of Self-Report Measures
Reliability refers to the consistency of measurement and is frequently assessed using the test–
retest reliability method.
Reliability is increased by including many similar items on a measure, by testing a diverse sample
4
of individuals, and by using uniform testing procedures.
Validity refers to the truthfulness of a measure: Does it measure what it intends to measure?
Construct validity represents the extent to which a measure assesses the theoretical construct it
is designed to assess; construct validity is determined by assessing convergent validity and
discriminant validity.
C. Constructing a Questionnaire
Constructing a questionnaire involves deciding what information should be sought and how to
administer the questionnaire, writing a draft of the questionnaire, pretesting the questionnaire,
and concluding with specifying the procedures for its use.
The wording of questionnaires should be clear and specific using simple, direct, and familiar
vocabulary.
The order in which questions are asked on a questionnaire needs to be considered seriously
because the order can affect respondents’ answers.
VIII. Thinking Critically About Survey Research
A. Correspondence Between Reported and Actual Behavior
Survey research involves reactive measurement because individuals are aware that their
responses are being recorded.
Social desirability refers to pressures that respondents sometimes feel to respond as they
“should” believe rather than how they actually believe.
Researchers can assess the accuracy of survey responses by comparing these results with
archival data or behavioral observations.
B. Correlation and Causality
When two variables are related (correlated), we can make predictions for the variables; however,
we cannot, simply knowing a correlation, determine the cause of the relationship.
When a relationship between two variables can be explained by a third variable, the relationship
is said to be “spurious.”
Correlational evidence, in combination with a multimethod approach, can help researchers
identify potential causes of behavior.
IX. Summary
REVIEW QUESTIONS AND ANSWERS
These review questions appear in the textbook (without answers) at the end of Chapter 5, and can be
used for a homework assignment or exam preparation. Answers to these questions appear in italic.
5
1. Briefly identify the goal of survey research and how correlations are used within survey research.
2. Describe the information you would examine to determine whether survey results are biased because
the sponsoring agency of the survey has a vested interest in how the results turn out.
3. What two characteristics do surveys have in common regardless of the purpose for which the survey
has been done?
predetermined questions used for all respondents. (p. 138)
4. Explain the relationship between the homogeneity of the population from which a sample is to be
drawn and the size of a sample needed to ensure representativeness.
5. Explain why there is likely to be a serious threat to the interpretability of the results of a survey when
a convenience sample is used.
6. Explain why you would choose to use a mail survey, personal interviews, telephone interviews, or an
Internet survey for your survey-research project.
6
7. Explain why it is not possible to conclude a sample does not represent a population simply by
knowing that the response rate was 50%.
8. What are the major advantages and disadvantages of Internet surveys?
9. Describe the relationship that would need to exist among the samples in a successive independent
samples design in order to be able to interpret population changes in attitudes over time.
10. You are interested in assessing the direction and extent of change over time in the opinions of
individual respondents. Identify the survey-research design you would choose, and explain why you
would make this choice.
11. Describe one method for determining the reliability and one method for determining the validity of a
self-report measure.
12. Describe three factors that affect the reliability of self-report measures in survey research.
7
13. How would you respond if someone told you that survey results were useless because people do not
respond truthfully to questions on surveys?
14. Explain why “correlation does not imply causation,” and explain how correlational evidence can be
useful in identifying potential causes of behavior.
15. Define mediator and moderator and provide an example of each.
CHALLENGE QUESTIONS AND ANSWERS
These questions appear in the textbook (without answers) at the end of Chapter 5, and can be used for a
homework assignment, in-class discussion, or exam preparation. Answers to these questions appear in
italic.
[Answer to Challenge Question 1 also appears in the text.]
1. Survey research is difficult to do well, and this can be especially the case when the topic is people’s
sexual attitudes and practices. For a book focusing in part on women’s sexuality, an author mailed
100,000 questionnaires to women who belonged to a variety of women’s groups in 43 states. These
groups ranged from feminist organizations to church groups to garden clubs. The author’s
questionnaire included 127 essay questions. The author received responses from 4,500 women.
8
Findings in this survey included that 70% of respondents married 5 years or more reported having
extramarital affairs and that 95% of respondents felt emotionally harassed by the men they love.
A. The final sample in this study is large (4,500). Is this sufficient to ensure the representativeness of
the sample? If not, what survey–research problem could lessen the sample’s representativeness?
B. Is it possible on the basis of your response to Part A of this question to argue that any
conclusions drawn by the author from her data are incorrect? What could you do to determine
whether the results are correct?
2. With the increasing use of electronic communications and Internet, a growing problem among youth
is the experience of online harassment (cyberbullying). Estimates of the extent of cyberbullying vary.
In one national (U.S.) survey conducted in 2005, professional interviewers used random-digit dialing
to construct a sample of 1,500 English-speaking households with youth between the ages 10 and 17
who use the Internet (Youth Internet Safety Survey). Youth were interviewed if their parents provided
consent, and the youth assented. Nine percent of the youth in this sample reported being targets of
online harassment within the past year. In another study conducted in 2008, 20,406 high school
students (9th –12th grades) in the Boston metropolitan area completed an anonymous, paper–and–
pencil survey about health and behavior topics one day in school (MetroWest Adolescent Health
Survey). One question asked, “How many times has someone used the Internet, a phone, or other
electronic communications to bully, tease, or threaten you?@ In this sample, 15.8% of the students
reported being victimized within the past year.
Decide whether each of the following statements is True or False, then explain your answer.
A. The much larger sample size of the Boston survey indicates that 15.8% is a more reliable
estimate of the amount of cyberbullying experienced by youth in the U.S. population.
B. The different ages included in the two samples (10-17 and 9th–12th graders) may account for
differences in the reported rate of cyberbullying.
9
C. One reason youth reported more cyberbullying in the Boston study could be because an
anonymous, self-report survey was used, compared to the telephone interview for the national
sample.
D. The findings for Internet bullying in the two surveys indicate that the Boston area has more
Internet bullying than the overall national rate.
E. The findings for the two surveys indicate that the percentage of cyberbullying increased nearly
7% points from 2005 to 2008.
3. The human resources department of a large corporation decides it wants to survey employees=
knowledge of procedures for requesting alternative work schedules (e.g., flex-time,
maternity/paternity leave), as well as employees= perceptions of the fairness of these corporate
policies. The questionnaire also asked employees whether they have ever requested alternative
scheduling. A stratified random sample of 1,000 employees was drawn from payroll=s list of 5,000
full-time employees. Strata were defined based on number of years employed with the corporation
and were sampled proportionally. Via interoffice mail, employees in the sample were mailed a survey
to be completed and returned anonymously by dropping it in a box. Questionnaires were returned by
600 employees. Results indicated that 200 (33%) of the respondents rated the corporate policies for
alternative schedules as unfair.
A. Was the initial sample of 1,000 employees likely to be representative of the population of 5,000
full-time employees? Why or why not?
10
B. Suppose the head of the human resources department decides the sample was biased because
of the response rate, and the perception by one-third of the sample respondents that the policies
are unfair is an overestimate. Is this a correct decision? Why or why not?
C. Suppose the following survey results were obtained:
Have you ever requested alternative
scheduling?
Believe
schedule
policies are:
Yes (42%)
No (58%)
Unfair
150
50
Fair
100
300
In addition, data from the human resources department indicate that overall, 1,250 full-time
employees (25%) have requested alternative work schedules. Based on these data, what do you
conclude about whether the sample was biased?
4. Suppose you are an intern at a hospital with the task of developing a survey regarding patients’
attitudes toward their care while they were hospitalized. Hospital administration will provide a list of
patients who were treated and discharged during the past month, and who agreed to be contacted for
follow-up surveys. Administrators would like to know whether patients= level of satisfaction with
hospital procedures (e.g., admission/discharge, pharmacy, billing) change over the 2, 6, and 12
months post-discharge.
A. Describe the two survey research designs that could be used to answer the administrators’
questions. Outline how each of these designs could be implemented for this project, and describe
the advantages and possible limitations of each design.
11
12
13
first 100 students who arrive for dinner at the dining hall on campus.
(1) Identify the type of sampling your student representative plans to use, and briefly describe
the most serious limitation of this approach.
(2) Suggest how the sampling should be done. Remember that there are likely to be differences
in students’ opinions as a function of their year in school (freshman, sophomore, junior,
senior), so be sure that the sampling plan you propose allows for a breakdown of the results
(3) Your student representative then asks you to suggest how she should administer the survey
questionnaire. List the four choices she has, state which method you would suggest, and
explain why.
D. A physician from an inner-city hospital wrote a critique of a newspaper article describing a survey
that showed a lowered incidence of child abuse in the past year. The sample for the phone
survey was 1,400 families, most of which were intact and economically advantaged. What survey
research problem did the physician likely use as the basis of his critique?
2. Reading Research Critically
This research summary and the accompanying questions could be used in class for small group
discussion or as a homework assignment. Answers for the research summary are provided below.
14
The research summary and questions are presented on subsequent pages to facilitate copying for
class handouts.
Answers for Expressing Gratitude in Relationships
A. Describe the sampling method used in these two studies. What is the likely population the
researchers wish to generalize their findings to and to what extent is their sample representative
of that population (i.e., do they potentially have a biased sample)?
B. Identify the survey design in each of these two studies and how Lambert et al. (2010) used these
designs to satisfy conditions for causal inference. Which conditions did they meet and which did
they not meet in these two studies?
that explain the relationship between expressions of gratitude and communal strength.
C. Lambert et al. (2010) used four questionnaires in their study to examine the relationship between
expressions of gratitude and communal strength in a relationship. The two remaining measures,
relationship satisfaction and social desirability, were used to address questions associated with
the construct validity of gratitude and communal strength. That is, the researchers sought
evidence to demonstrate that expressions of gratitude and communal strength are not the same
as relationship satisfaction, nor simply socially desirable responding. Describe the pattern of
correlations they likely observed to support the construct validity of their measures.
15
Reading Research Critically
Read the following description of a research study to answer the questions that follow. [Based on an
article by Lambert, N. M., Clark, M. S., Durtschi, J., Fincham, F. D., & Graham, S. M. (2010). Benefits of
expressing gratitude: Expressing gratitude to a partner changes one’s view of the relationship.
Psychological Science, 21, 574-580. doi: 10.1177/0956797610364003]
Expressing Gratitude in Relationships
From the time we are children, we are taught to say “thank you,” write thank-you notes, and more
generally, appreciate what others do for us. Gratitude is an emotional state in which a person is aware of
and appreciates that which is valuable and meaningful to oneself. Empirical research suggests that
expressions of gratitude are related to relationship satisfaction, and that people like to feel appreciated.
While it’s clear that the recipient of gratitude benefits from such expressions, Lambert and his colleagues
sought to learn whether the person who expresses gratitude also benefits. They hypothesized that when
people express gratitude to a relationship partner, they will perceive greater communal strength in that
relationship. “Communal strength” refers to a feeling of responsibility for meeting the partner’s needs.
In their first study, 137 students (116 women, 21 men) in an undergraduate psychology course completed
questionnaires for extra credit. Age of the students ranged from 18 to 37; the median age was 19.
Participants completed four measures: Expression of Gratitude in Relationships, Communal Strength, a
Social Desirability measure, and Relationship Satisfaction. [Students completed the measures thinking
about a romantic partner or a close friend.] The researchers obtained scores on the measures for each
participant using accepted procedures. The results indicated that expression of gratitude was correlated
with communal strength of the relationship, and that this relationship was maintained even when other
variables (e.g., social desirability, relationship satisfaction, participant gender, type of relationship, etc.)
were statistically controlled (i.e., these variables did not mediate or moderate the correlation).
In a second study, Lambert et al. sought to determine whether expressions of gratitude predict
individuals’ subsequent perceptions of communal strength (i.e., at a later time). In this study, 218
undergraduates (171 women, 47 men) first completed the measures at the beginning of a semester (Time
1) and then again 6 weeks later (Time 2). Using correlational procedures, their results indicated that
expressions of gratitude at Time 1 predicted perceptions of communal strength at Time 2 (statistically
controlling for initial feelings of communal strength, relationship satisfaction, etc.). Lambert et al.
concluded that their “findings provide important support for a time–order relationship” between
expressions of gratitude and subsequent feelings of communal strength in a relationship. These
researchers suggest that expressing gratitude helps the expressers communicate to themselves (in
addition to their partner) their desire for the relationship and care for the partner, and that these
expressions might result in more relationship-enhancing behaviors.
A. Describe the sampling method used in these two studies. What is the likely population the
researchers wish to generalize their findings to and to what extent is their sample representative of
that population (i.e., do they potentially have a biased sample)?
B. Identify the survey design in each of these two studies and how Lambert et al. (2010) used these
designs to satisfy conditions for causal inference. Which conditions did they meet and which did they
not meet in these two studies?
16
C. Lambert et al. (2010) used four questionnaires in their study to examine the relationship between
expressions of gratitude and communal strength in a relationship. The two remaining measures,
relationship satisfaction and social desirability, were used to address questions associated with the
construct validity of gratitude and communal strength. That is, the researchers sought evidence to
demonstrate that expressions of gratitude and communal strength are not the same as relationship
satisfaction, nor simply socially desirable responding. Describe the pattern of correlations they likely
observed to support the construct validity of their measures.
3. More Challenge Questions
These additional challenge questions can be used for class discussion or as test questions.
A. Two different national organizations that conduct research on higher education did independent
surveys asking faculty how well prepared they thought their students were. The results of these
two surveys drew attention when they were reported in the Chronicle of Higher Education
because the findings from the two surveys were very different. Researchers from Research
Foundation A found that nearly 75% of professors said that their students were “seriously
underprepared.” Researchers from Research Foundation B found that only 18.8% of the faculty
they surveyed said that their students were “not at all prepared.” Survey-research findings can be
expected to vary from one survey to another, but the large discrepancy found in these two
surveys could make one wonder about the reliability and credibility of survey findings. Before
reaching this conclusion, it is useful to consider several details of the two surveys. [Note. This
question is based on a report from the NCRIPTAL Update, Spring 1990, Vol 3, No. 1, pp. 2-3.]
(1) Who was asked? The original sample for Foundation A included 10,000 college professors
who taught undergraduate and graduate students in all types of institutions. Of the original
sample, 54.5% responded. Foundation B omitted research universities (25% of Foundation
A’s sample). Foundation B had a final sample of 2,311 (62% response rate). Approximately
90% of the final sample were teaching introductory–level students. How might the
characteristics of the samples surveyed by Foundations A and B affect the findings obtained
in the two surveys?
at research universities and for students taking introductory courses.
(2) What was asked? Foundation A asked its respondents: “The undergraduates with whom I
have close contact are seriously underprepared in basic skills such as those required for
written and oral communication.” The responses for this statement were: strongly agree,
agree with reservations, neutral, disagree with reservations, and disagree. Foundation B
asked its respondents: “In their background preparation, students who enroll in this course
are most typically…” The response choices were: not at all prepared, somewhat prepared,
very well prepared, and extremely well prepared. How might the nature of these questions
affect the findings obtained in the two surveys?
17
18
concluded that the final sample was not representative of the population of students. They
further decided that the ratings of the judicial system as unfair by more than a third of the
students was an incorrect overestimate. Do you agree that the finding represents an incorrect
estimate? Why or why not?
many students think that the judicial system is unfair.
(4) While the task force was meeting to discuss their final report, one member of the task force
expressed the opinion that students’ responses were unlikely to have been truthful and so the
results of the survey were useless and should not be reported at all. The director of the task
force calls on you to respond to this statement. What would you say?
C. As an intern with the alumni relations office at a small college, one of your assignments is to help
develop a survey–research project. The college is interested in finding out about the alumni’s
attitudes toward their academic and extracurricular experiences while enrolled in college. The
director also wants to include questions to assess the alumni’s opinions about the different
activities the college sponsors for them (e.g., reunions) and how they prefer to be kept informed
about issues and activities on campus (e.g., newsletters, e-mails, postings on the college
website). One of the major goals of the survey-research project is to determine how the attitudes
of alumni change 1, 5, or 10 years after graduation.
(1) The first step is to select the survey-research design for the project. Describe the two designs
that can be used to measure changes in attitudes over time. Outline how each of these
designs would be implemented for this project, and identify the advantages and possible
limitations of each design.