20
(4) What type of sampling did the observers use to decide when to make their observations?
(5) Identify one aspect of their procedure that likely increased the reliability of their observations.
(6) Identify one aspect of their procedure that likely decreased the problem of reactivity.
(7) Identify one aspect of their procedure that would likely limit the external validity of their
findings.
B. As part of an investigation of student attitudes toward their university, a researcher performs a
content analysis of the letters to the editor published in the campus newspaper. Specifically, over
the course of the fall semester, every letter to the editor published in the school paper is classified
according to whether it reflects a “favorable,” “unfavorable,” or “neutral” attitude toward the
school.
(1) What procedures should be followed to ensure that the classification of the letters is reliable?
How could the reliability of the classification be measured?
(2) What are the possible limitations in assessing students’ attitudes toward the university solely
on the basis of this one archival source?
21
(3) Identify other possible archival sources that would provide evidence regarding students’
attitudes toward the university.
(4) What other methods could researchers use to assess attitudes toward the university that
would complement archival records?
C. A bright female graduate student in psychology has been offered a job with both Newsweek and
Time. The salary offers of the two companies are basically the same, and it appears that both the
working conditions and the job responsibilities are similar. To help her decide which job to accept,
she resolves to determine whether one organization has a better attitude toward women that the
other. She appeals to you to help her with a content analysis of these two news sources. What
specific advice would you give her regarding each of the following steps of her content analysis?
(1) Sampling
(2) Coding
(3) Reliability
(4) Quantitative and qualitative measures
22
LEARNING BY DOING RESEARCH
One of the best ways to help students appreciate the strengths and limitations of observational research
is by having them carry out a naturalistic observation study to test an idea generated by the students.
There are several advantages of having students work together on smaller scale projects they develop.
Students are typically more interested in testing ideas they develop, and they often enjoy the opportunity
to work in small groups. They report that abstract concepts such as sampling and reliability “come to life”
as they encounter issues while doing their observation project. Of course, there are limitations to small–
scale observation projects. The research questions students test may not measure up to Sternberg’s
(1997) criteria for important research questions. Working in small groups can make it difficult for students
to complete a large number of observations within a limited time frame for the assignment. At times,
“learning by doing” in this assignment means learning what they would do differently next time. Yet,
students respond positively to this hands-on approach to learning and much is gained when students
appreciate some of the complications involved in conducting good research.
One way to complete the assignment is to have students work in teams of four. Allow about an hour of
class time for them to develop an idea for their project, and require each student to spend an hour outside
of class time to make their observations. The report of their project could be prepared collaboratively or
separately. It’s also helpful to allow some class time for students to share their project results with their
classmates.
The following pages present a worksheet the students can use to guide the development of their research
project. A second page has questions for students to answer to report the outcome of their project. Each
can easily be adapted to other ways for doing this assignment.
23
Worksheet for Naturalistic Observation Project
1. For this assignment you will work as a team of four students to complete a naturalistic observation
study. Your project should include the following characteristics:
A. You should not intervene in the situation you observe (i.e., no participant observation, no
structured observation, and no field experiments).
B. You should select a naturally occurring independent variable with two levels (e.g., time of day–
morning, afternoon; campus location—library, student union). Choose an independent variable
you believe will produce a difference in the behavior you wish to observe (e.g., studying). You
may use gender as a secondary independent variable, but you must use some other naturally
occurring variable as your primary independent variable.
C. You should select a behavior to record that you think will be influenced by your independent
variable. Do not attempt a study that involves a comprehensive record of behavior in a setting.
D. Develop a prediction regarding the outcome you expect in your study (e.g., students will study
more effectively in the afternoon in the morning). You may also provide a possible reason for your
expected outcome (e.g., because they are more alert in the afternoon). If you provide a possible
reason for your expected outcome, you should include in your study a measure as directly related
as possible to the stated reason your propose (e.g., a measure of alertness).
E. You should observe only public behavior–choose a setting that is clearly in the public domain.
2. When deciding on your project, it is most important that you determine:
A. the research question you are trying to answer in your naturalistic observation project
B. why this research question is important to you
C. the operational definition for each of the variables you plan to study
D. the goal(s) of the scientific method you intend to meet by doing your observation project (note
that because you will not manipulate an independent variable, “explanation” is not an achievable
goal)
3. In deciding on your research project you should also consider the following issues:
A. Reactivity: how will you assure your observations are unobtrusive?
B. Situation sampling: where will you make your observations? Although situation sampling may be
desirable, due to time constraints, you may need to select only one or two locations.
C. Time sampling: when will you make your observations? Remember, your team should try to
complete the assignment with no more than four hours of observation (one hour for each team
member), so it’s important that the times you select will yield as many observations as possible
for the behavior you wish to study.
24
D. Subject sampling: how will you select individuals for observation? If you plan to observe in a
setting where there are more people than you can observe (e.g., a crowded cafeteria), how will
you decide whom to include?
Worksheet for Naturalistic Observation Project, page 2
E. Data reduction: before beginning your observations, decide what kind of measurement you are
using (i.e., scale of measurement, frequency counts, duration, ratings) and plan how you will
summarize your results (e.g., proportion/percentage, rankings, means, standard deviations).
F. Reliability: before beginning your observations, decide the methods you will use to make sure
your observations will be reliable.
4. Prepare a checklist or some other method for recording observations. Be sure to include space for
recording for each observation: the level of independent variable and observations for the dependent
variable(s) measures. For example, if observations require categorization, your checklist should
include relevant categories that observers can check or circle easily for each observation. If
observations require ratings, then rating scales with endpoints (“anchors”) should be provided (e.g.,
1= “not at all alert” and 5 = “very alert”).
5. Data analysis: Use appropriate descriptive statistics to describe the results of your observational
study. If you seek to identify relationships between your variables, the analysis you choose will
depend on the level of measurement. Check with your instructor regarding the statistics required, if
any, to assess relationships between variables in your study.
6. After collecting your observations, consider alternative interpretations for your findings. Are there
other variables that may be related to your naturally occurring independent variable that may account
for any relationship you observe between your variables? You are unlikely to be able to identify
precisely a single interpretation of your results. It is more important that you are able to articulate
arguments with evidence from your observations for and against alternative interpretations. Also
consider issues of external validity when interpreting your findings. That is, to what extent are you
able to generalize your findings to populations, settings, and conditions beyond those you studied?
25
Report of Naturalistic Observation Project
Questions 1-9 will be graded. Responses to these questions can be written by each member of the team
in separate reports or by the team working together to produce a single report. Questions 10 and 11 are
not graded, but each member of the team must submit responses to these two questions for the team’s
report to be considered complete. Students’ responses to Questions 10 and 11 should be submitted
separately if a group report is submitted from the team.
1. What research question did you try to answer in your naturalistic observation project, and why was
this question important to you?
2. What were the operational definitions for the naturally occurring independent variable and the
dependent variable(s) in your project?
3. What specific prediction did you test in your study? Include any reason you identified for your
prediction, if applicable and state your prediction in the form of a hypothesis.
4. How did you avoid intervening in the situation you observed so as to avoid reactivity?
5. What specific procedures did you use for time sampling, situation sampling, and subject sampling? If
you observed at only one time or in one situation, please describe and explain why you made this
choice.
6. Describe the aspects of the study that affected the reliability of your observations.
7. Briefly summarize the results of your study using the appropriate descriptive statistics for your
dependent variable. With two levels of the independent variable, a table or figure typically is not
necessary for reporting the results; a verbal statement including the appropriate numerical values is
sufficient.
8. Describe your conclusions based on the results of your study. Was your prediction supported?
9. Describe any characteristics of your study that would limit your interpretation of the findings. Describe
characteristics of your study that limit external validity of the findings. Explain how these
characteristics affect the conclusions you have made based on your observational study.
10. Comment critically on what you have learned from doing this assignment. Your response to this
question will not be included when determining your grade on this assignment.
11. Comment on the positive and negative aspects of doing this assignment in a group. Please do not
refer to any group members by name. Again, this question will not be graded.
26
INSTRUCTOR’S LECTURE/DISCUSSION AIDS
The following pages reproduce content from Chapter 4 and may be used to facilitate lecture or
discussion.
1. Observational Research: This page describes the goal of sampling behavior, with an illustration that
asks students about their television viewing.
2. External Validity: This page defines external validity and raises the issue of representative samples.
3. Time Sampling and Situation Sampling: This page introduces two methods for sampling behavior.
4. Observational Methods: A diagram outlines the observational methods discussed in the chapter.
5. Direct Observation without Intervention: Naturalistic observation is described on this page.
6. Direct Observation with Intervention: The methods of intervention in observational studies are
outlined.
7. Indirect (Unobtrusive) Observational Methods: This page identifies the two main methods for indirect
observation: physical traces and archival records.
8. Recording Behavior: This page distinguishes between comprehensive records of behavior and
selected records of behavior, and describes measurement scales.
9. Analysis of Observational Data: Qualitative and quantitative data analysis are described.
10. Thinking Critically about Observational Research: This page outlines problems associated with
influence of the observer.
11. Thinking Critically about Observational Research: Problems associated with observer bias are
described on this page.
27
Observational Research
• Researchers can’t observe
▪ All people
▪ All of one individual’s behavior
• Researchers obtain samples
▪ Samples of individuals
▪ Samples of individuals’ behaviors at particular times, in different
settings and conditions
• Goal: The sample should represent the larger population of people,
settings, conditions, and behaviors.
• Example: How many hours of television did you watch last week?
▪ Is this number representative of how much you typically watch tv?
For the entire class:
▪ Is the average number of hours watched representative of hours
watched by all students on campus?
▪ … representative of other college students?
▪ … representative of the larger population of adults?
28
External Validity
• Extent to which results of a research study can be generalized to other
populations, settings, conditions
▪ Validity: “truthfulness”
• Sample must be representative of the population.
▪ “Representative”: similar to
• Issue in Psychological research
▪ College student samples frequently tested
▪ Representative?
• How do we obtain representative samples?
29
Time Sampling
• Researchers choose time intervals for observations.
▪ Systematic: schedule observations to occur at a regular time
▪ Random: use some random means for selecting observation intervals
▪ Event sampling: used when events of interest occur infrequently
Situation Sampling
• Researchers choose different settings, circumstances, conditions for
observations.
▪ Enhances external validity of observations
▪ Subject sampling: observe some people in the setting
30
Observational Methods
Observational
Methods
Direct
Observation
Observation
Without
Intervention
Observation
With
Intervention
Participant
Observation
Structured
Observation
Field
Experiment
Indirect
(Unobtrusive)
Observation
Physical Traces Archival
Records
31
Direct Observation
without Intervention
• Naturalistic observation
▪ Observation in “real–world” setting
▪ No attempt to intervene or change the situation
• Goals
▪ Describe behavior as it normally occurs
▪ Examine relationships among naturally occurring variables
▪ Establish external validity of laboratory findings
▪ Use when ethical considerations prevent experimental manipulation
32
Direct Observation
with Intervention
• Characterizes most psychological research
• Intervene to exert control over observations
• Three methods
▪ Participant observation
o Disguised
o Undisguised
▪ Structured observation
▪ Field experiment
• Problem of reactivity
▪ People often change their usual behavior when they know they’re
being observed.
33
Indirect (Unobtrusive) Observational Methods
• Examine evidence of past behavior
• Nonreactive
• Two types of methods
▪ Physical traces: remnants, fragments, products of past behavior
o Use traces: natural or controlled
o Products
▪ Archival records
o Running records
o Episodic records
• Possible biases
▪ Selective deposit
▪ Selective survival
• Seek converging evidence using multimethod approach
34
Recording Behavior
• Comprehensive records
▪ Narrative records: reproduction of behavior as it originally occurred
o Written descriptions, videotapes, audiotapes
▪ Code behaviors after observation is completed
• Selected records of behavior
▪ Focus on certain behaviors or specific aspects of individuals or
settings
• Measurement scales
▪ Nominal: categorize behavior or events
▪ Ordinal: rank-order behavior or events
▪ Interval: specify distance between behavior or events on a dimension
o Rating scales used in psychological research are treated as
interval scales
▪ Ratio: specify distance plus value of zero is meaningful
35
Copyright © 2015 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written
consent of McGraw-Hill Education.
Analysis of Observational Data
• Qualitative data analysis
▪ Data reduction to summarize comprehensive records
▪ Coding: identify units of behavior or events according to specific
criteria
▪ Qualitative analysis emphasizes verbal summary of observations
▪ Content analysis: coding procedures for archival records
o Identify relevant archival source
o Obtain representative sample from source
▪ Code content using descriptive categories
• Quantitative data analysis
▪ Statistical summary of observations
▪ Use of descriptive statistics depends on measurement scale
o Nominal: relative frequency
o Ordinal: rank percentages
o Interval and ratio: mean, standard deviation
▪ Interobserver reliability: measure of agreement between observers
o Nominal: percent agreement
o Ordinal: Spearman rank-order correlation
o Interval and ratio: Pearson correlation
▪ Improve reliability by
o Using clear operational definitions
36
o Training observers
o Giving observers practice and feedback
o Minimize observer fatigue, boredom
37
Thinking Critically about Observational Research
• Influence of the observer
▪ Reactivity: presence of behavior causes people to change their usual
behavior
o Demand characteristics: people use cues in research setting to
determine what is expected of them
▪ Controlling reactivity
o Disguised participant observation and indirect (unobtrusive)
methods
o Adapt individuals to observer’s presence (habituation,
desensitization)
o Limit information about hypotheses, participants’ role in study
• Ethical issues
▪ Privacy and informed consent issues when observe people without
their knowledge
▪ Potential risk associated with structured observation and field
experiments
▪ Risk/benefit ratio: balance psychologists’ ethical obligation to benefit
individuals and society with potential risks of observing individuals’
behavior
38
Thinking Critically about Observational Research
• Observer bias
▪ Systematic errors in observation resulting from observer’s
expectations
▪ Experimenter expectancy effects: research hypotheses can lead
researchers to “observe” what they expect to see
▪ Placement of cameras and automated equipment can be biased
▪ Controlling observer bias
o Be aware of possible presence of observer bias
o Keep observers “blind” regarding hypotheses, expectations