21
By doing this assignment students gain experience with developing a brief questionnaire, selecting a
random sample of respondents, and conducting phone interviews. When students write their report of
their survey they address important survey research issues such as the effects of wording of questions,
representativeness of samples, and response bias. Comparing their survey results to those of published
surveys gives students the opportunity to consider the generality of survey results (when the students’
survey results are similar to those of the published survey). This comparison process also allows students
to explore what characteristics of their sample may have led to the results that differ between their sample
and the published survey results. This comparison process is especially effective when the students have
included at least some items on their questionnaire that are drawn from the published survey.
The amount of class time that will be required for this assignment depends on the extent to which
students are involved in the development of the questionnaire, how much discussion occurs in class of
the obtained results, and the nature of the report students are required to write.
Steps in Conducting a Survey Research Project
1. Select a topic for the survey and a source of a survey to use as a comparison for the students’
survey.
2. Construct a brief questionnaire appropriate for a 5-8 minute phone interview (about 10 questions). It
is desirable to include items from the comparison survey whenever possible.
3. Select the sample for the survey.
A. We have used the student phone directory as the sampling frame of the population of our
students.
B. We have tried to obtain a sample of 60-100 respondents. If there are 25 students in a research
methods class, then each student in class will try to obtain up to 4 respondents.
C. Because the methods students may not be able to reach each student they try to call, it is best if
they randomly identify 8 possible students to call in an effort to obtain 4 respondents.
D. In order to reduce the number of phone directories we need to have available in class when we
do this part of the assignment, students work in pairs to select their potential respondents. Pairs
of students can be assigned to certain pages of the directory such that across all the pairs the
entire directory is represented. Each student randomly selects a number that is equal to or less
than the number of names on a page of the phone directory. Each student in the pair repeats this
process until both students in the pair have selected the desired number of phone numbers (in
our example, 8).
E. Prior to making their phone calls, students will need copies of the questionnaire, instructions as to
how to complete their calls, and a record sheet showing the number of surveys successfully
completed, the number of people who could not be reached, and the number who refused to
participate.
4. The instructions for the phone interviews could include:
A. Recording the date and time when each call was made.
B. If someone answers, read from the survey script which begins by asking if the person is willing to
respond to a few questions about (the topic). [The person answering need not be asked to
identify themselves; anyone answering the phone at the selected number is an eligible
respondent.]
C. Read each question from the questionnaire. As the person responds to the questions, the
interviewer circles the corresponding response on the questionnaire.
22
D. If no one answers or if the person is unwilling to participate, the student tries a second number
and follows the same procedure.
E. Students should not try to call a number a second time if a person at that number has declined to
participate. Students should, however, try again to reach someone at a number where there was
no answer. Messages should not be left on voice mail or answering machines. [The instructor
may want to specify how many times a student needs to call a number before giving up.]
5. After completing their calls each student should fill out the record sheet summarizing the outcomes of
their calls: number of calls successfully completed, number of respondents they could not reach, and
number of people who declined to participate.
6. Students then return their completed questionnaires and record sheets to the instructor by the
specified due date.
Outline of Report for Survey Research Assignment
Note: This outline follows the major sections of an APA report but can be done most easily in the form of
a take-home assignment with short-answer essay questions.
1. Description of the nature of the topic and why it is of interest (including description of the comparison
survey, if one is used)
2. Outline of the procedures for developing the questionnaire
3. Brief description of the sampling plan that was used
4. Report of the results, including which results were surprising or unexpected and which results were
most interesting (especially in terms of comparisons between the students’ sample and the
comparison sample)
5. Consideration of survey research issues such as representativeness of the students’ sample, the
comparability of the students’ sample and the comparison sample, response bias, influence of the
particular wording used in items on the questionnaire
6. Implications of the results of the survey for the student and for society
INSTRUCTOR’S LECTURE/DISCUSSION AIDS
The following pages reproduce content from Chapter 5 and may be used to facilitate lecture or
discussion.
1. Survey Research: This page summarizes key points about survey research.
2. Sampling in Survey Research: This page introduces terms such as “representative” and “generalize.”
3. Basic Terms of Sampling: This page defines population, sampling frame, sample, and element.
4. Biased Samples: This page describes the characteristics of biased samples and defines selection
bias and response-rate.
23
5. Approaches to Sampling: Nonprobability sampling and probability sampling are described on this
page.
6. Survey Methods: This page identifies four methods for obtaining survey data and ways to increase
response rate.
7. Survey Research Designs: The three survey designs are summarized on this page.
8. Measures in Survey Research: This page introduces questionnaires.
9. Reliability and Validity: Essential features of reliability and validity are described on this page.
10–11. Constructing a Questionnaire: Important steps for preparing a questionnaire are described on
these two pages.
12–13. Thinking Critically About Survey Research: These two pages describe issues associated with the
correspondence between reported and actual behavior, correlation and causality, and path
analysis.
24
Survey Research
• Surveys
▪ Describe people’s thoughts, opinions, and feelings
▪ Measure naturally occurring variables
▪ Allow predictions based on correlations
▪ Questionnaires
o Predetermined set of questions
▪ Sample represents a population
▪ Examine survey procedures and analyses for sources of bias
25
Sampling in Survey Research
• Use sample to represent population
▪ “representative”: similar to
• Requires careful selection of a sample
• Goal: Generalize survey findings from representative sample to the
population
26
Basic Terms of Sampling
• Population
▪ Set of all cases of interest
• Sampling Frame
▪ List of members of a population
• Sample
▪ Subset of a population drawn from sampling frame
• Element
▪ Each member of the population
27
Biased Samples
• Characteristics of the sample differ systematically from those of the
population
▪ Sample over-represents or under-represents segment(s) of the
population
• Two sources
▪ Selection bias: Researcher’s procedures for selecting sample cause
bias
▪ Response-rate bias: Individuals selected for the sample do not
complete the survey
28
Approaches to Sampling
• “Sampling”
▪ Procedures used to obtain a sample
• Two basic approaches: probability sampling and nonprobability sampling
• Probability sampling
▪ All members of the population have a specified chance of being
selected for the sample
▪ Simple random sample
o random selection, random-digit dialing
▪ Stratified random sample
o Divide population into strata and sample proportionally
o Improves representativeness of sample
• Nonprobability sampling
▪ No guarantee each member of the population has an equal chance of
being in the sample
▪ “Convenience sampling”
o Individuals are available and willing to respond to the survey
o Example: magazine surveys, call-in radio shows
▪ Sample likely not representative of the population
29
Survey Methods
• Four methods for obtaining survey data
▪ Mail surveys
▪ Personal interviews
▪ Telephone interviews
▪ Internet surveys
• Each method has advantages and disadvantages.
• Choose method based on research question.
• Ways to increase response rate
▪ Questionnaire has a “personal touch”
o Use name, not “Dear student”
▪ Responding requires minimal effort
▪ Topic of survey is interesting to respondents
▪ Respondents identify with organization or sponsor of survey
30
Survey Research Designs
• “Research Design”
▪ A plan for conducting a research project
▪ Choose method best suited for answering a particular question
• Three types of survey research designs: cross-sectional, successive
independent samples, longitudinal
• Cross-sectional Design
▪ Select sample from one or more populations at one time
▪ Use survey responses to describe population(s), make predictions for
the population (correlations) at that one point in time.
▪ Compare populations
▪ Cannot assess change over time
• Successive Independent Samples Design
▪ A series of cross-sectional samples over time
▪ A different sample from the population completes the survey each
time.
▪ Each sample is selected from the same population.
▪ Responses from each sample are used to describe changes in the
population over time.
▪ Problem: noncomparable samples
• Longitudinal Design
▪ Same sample of individuals completes the survey at different points
in time
▪ Assess how individuals change over time
▪ Responses from the sample are generalized to describe changes
over time in the population.
▪ Problems: attrition and reactivity
31
Measures in Survey Research
• Questionnaires
▪ Most frequently used to collect survey data
▪ Measure different types of variables
o Demographic variables using checklists
o Preferences, opinions, and attitudes
Self-report scales
Rating scales (assume interval level of measurement)
• Measures must be reliable and valid.
32
Reliability and Validity
Reliability
• Reliability refers to consistency of measurement.
▪ Test-retest reliability
o Administer measure two times to same sample
o High correlation between the two sets of scores indicates good
reliability (r > .80)
• How to improve reliability?
▪ More items (observations)
▪ Greater variability among individuals on the factor being measured
▪ Testing situation free of distractions
▪ Clear instructions
• A measure can be reliable but not valid.
Validity
• Validity refers to the truthfulness of a measure.
▪ Assesses what it is intended to measure
• Construct validity
▪ Instrument measures the theoretical construct it was designed to
measure.
• Establishing construct validity: Demonstrate convergent and discriminant
validity
▪ Convergent validity
o Extent to which two measures of the same construct are correlated
(go together)
▪ Discriminant validity
o Extent to which two measures of different constructs are not
correlated (do not go together)
33
Constructing a Questionnaire
• Best choice for selecting a measure
▪ Use measure already shown to be reliable and valid in previous
research.
▪ If no suitable measure is found, create a questionnaire or measure.
• Creating a reliable and valid measure is not easy.
• Important first steps for constructing a questionnaire
▪ Decide what information should be sought.
▪ Decide how to administer the questionnaire.
▪ Write a first draft of the questionnaire.
▪ Reexamine and revise the questionnaire based upon expert advice.
▪ Pretest the questionnaire.
▪ Review results and edit the questionnaire.
• Next steps: Establish reliability and validity
▪ Reliability
o Test and re-test questionnaire using sample and conditions similar
to planned survey.
▪ Validity
o Convergent: Administer questionnaire with measures of
theoretically related constructs
o Discriminant: Administer questionnaire with measures of
theoretically unrelated constructs
34
Constructing a Questionnaire, continued
• Guidelines for writing survey questions
▪ Choose how participants will respond.
o free-response questions or closed questions
▪ Use simple, familiar vocabulary.
▪ Write clear, specific questions.
o Avoid double-barreled questions.
o Place conditional phrases at the beginning of sentences.
o Avoid leading questions and loaded questions.
o Avoid response bias.
• Ordering of questions
▪ Self-administered questionnaires
o Place most interesting questions first
▪ Personal and telephone interviews
o Demographic questions first
▪ Use funnel questions and filter questions as needed.
35
Thinking Critically About Survey Research
• Correspondence between reported and actual behavior
▪ Survey responses may not be truthful.
▪ Reactivity
▪ Social desirability
▪ Accept people’s responses as truthful unless there’s reason to
suspect otherwise.
▪ Use a multimethod approach to answering research questions.
• Correlation and causality
▪ “Correlation does not imply causality.”
▪ Three possible causal inferences for any correlation
o A causes B
o B causes A
o Variable C causes A and B
36
Thinking Critically About Survey Research, continued
• Path analysis
▪ Statistical procedure to tease apart complex correlational
relationships among variables
▪ Mediators
o Variables used to explain a correlation between two variables
▪ Moderators
o Variables that affect direction or strength of correlation between
two variables
▪ Diagram of direct and indirect relationships
“Chaos” mediates the relationship between
poverty and psychological distress among children.
Chaos
path b path c
Poverty Psychological Distress
path a (direct)
▪ A moderator may affect the direction and strength of these
relationships.
o Possible moderators: sex of the child, population density,
personality features of the children
▪ Path analysis helps us to understand relationships among variables.
▪ But these relationships are still correlational
o Cannot make definitive causal statements
o Other untested variables may be important