Reading Social Science Methods

Chapter-by-chapter summaries

1. Introduction

Reisner introduces scientific literacy as the ability to judge what a study establishes and how it establishes it. Useful inventions and findings can bring risks as well as benefits, making uncritical acceptance inadequate. The chapter distinguishes a research result from a decision about how to act on it. Readers need to understand methodological rules before treating a conclusion as dependable. Its examples establish the book’s larger purpose: a person can evaluate research without first becoming an expert practitioner of every method.

2. Judgment Rule 1 for Surveys

The first survey rule asks whether questioning people can answer the research question. Respondents must possess the relevant information and be willing to disclose it. A survey may investigate reported beliefs, experiences, knowledge, or behavior, but its usefulness depends on the connection between those answers and the claim under investigation. Reisner begins with the research question because an inappropriate method cannot be rescued by an elaborate questionnaire. Readers should identify what the study is trying to learn before evaluating its technical details.

3. Judgment Rule 2 for Surveys

The second rule compares the intended population with the people actually sampled. A sample is not automatically representative merely because it is large or described as random. Choices about lists, recruitment channels, eligibility, and participation can systematically exclude certain people. The chapter traces how such choices narrow the claims a study can support. Readers are asked to examine the sampling process step by step, identifying who remains reachable and who disappears from consideration. Generalization must stay within the boundaries justified by that process.

4. Judgment Rule 3A for Surveys

This chapter moves from choosing respondents to choosing questions. The answers to the questionnaire, considered together, must provide evidence relevant to the overarching research question. Individual items can appear sensible while collectively failing to measure the concept at issue. Reisner encourages readers to reconstruct the relationship between a broad idea and its concrete indicators. The resulting appraisal asks whether the survey covers the necessary dimensions, whether essential information is missing, and whether the researchers’ conclusions answer the question their instrument was actually capable of addressing.

5. Judgment Rule 3B for Surveys

The quality of each question matters independently of the questionnaire’s overall topic. Leading language, confusing phrasing, excessive demands on memory, or unfamiliar terminology can shape responses in ways unrelated to the intended concept. A suitable item must be understandable to its particular respondents, not merely to the researcher. Reisner distinguishes a localized defective question from a pattern that compromises the instrument more broadly. The chapter teaches readers to inspect wording and response possibilities before assigning evidential weight to the answers.

6. Judgment Rule 4 for Surveys

Response rates introduce another possible gap between a recruited sample and the people whose answers become data. Those who respond may differ meaningfully from those who do not. Reisner presents a relatively demanding response-rate benchmark as part of her appraisal framework, while also discussing the practical difficulty of obtaining participation. The interpretive issue is nonresponse and the range of perspectives it may remove. The chapter should be read as the book’s methodological guidance, rather than treating one percentage as a universal guarantee of validity.

7. Summary of Judgment Rules for Survey Methods

The survey summary assembles the earlier checks into an ordered judgment. First establish that a survey fits the question, then examine sampling, item quality, and participation. Different flaws have different implications: some limit a particular result, while others undermine a study’s broader conclusions. The chapter discourages an all-or-nothing response to research. A careful reader specifies which findings remain useful, which population they concern, and where confidence weakens. The outcome is a qualified account of the evidence rather than a simple label of good or bad.

8. Content Analysis Introduction

Content analysis examines cultural materials such as news, films, images, games, and other meaningful artifacts. Researchers turn features of those materials into observations through a coding scheme. The chapter explains why apparently straightforward counting involves conceptual decisions: readers must know what counts as an instance and how that judgment is made. The method is useful for describing patterns within the material studied. Its introduction establishes the need for explicit categories and repeatable coding before the later chapters assess sampling and reliability.

9. Judgment Rule 1 for Content Analysis

The first content-analysis rule insists on matching the method to the object of the claim. Examining messages can establish something about those messages, but does not by itself demonstrate what audiences believe or how they behave after exposure. Reisner uses this distinction to identify conclusions that move beyond the evidence collected. Readers should restate the research question and determine whether the required observations concern texts, people, or effects. A well-executed coding exercise still cannot answer a different kind of question.

10. Judgment Rule 2 for Content Analysis

Sampling cultural materials requires the same care as sampling people, although the selection mechanisms differ. Archives, platforms, publication periods, technical requirements, and definitions of popularity can each change the set available for analysis. The chapter asks readers to follow the movement from a broad population of artifacts to the final sample. Each restriction may be reasonable, but it needs an interpretive consequence. Findings about a selected group of texts should not silently become claims about every text in a medium or culture.

11. Judgment Rule 3 for Content Analysis

Coding categories connect an abstract research interest with observable features. They must be precise enough for consistent use while remaining relevant to the concept being investigated. Ambiguous definitions can produce counts that look objective but conceal inconsistent judgments. The chapter encourages readers to inspect both the coding instructions and the relationship between categories and the research question. A clear measurement of the wrong feature remains inadequate. Good appraisal therefore considers conceptual appropriateness and operational clarity together.

12. Judgment Rule 4 for Content Analysis

Intercoder reliability concerns whether different people applying a coding scheme reach compatible judgments. Agreement provides evidence that results are not merely one coder’s private interpretation. Reisner discusses reliability benchmarks and the need to recognize the particular index reported. Different measures and research situations complicate the idea of one automatic cutoff. Readers should identify how reliability was assessed, what variables were tested, and which disagreements remain. The chapter treats consistency as an essential part of evaluating the measurement process.

13. Summary of Judgment Rules for Content Analysis

This summary brings the question, sample, categories, and coder agreement into one appraisal. A content-analysis study is strongest when its claims stay close to the materials observed and its coding procedure is both meaningful and reproducible. The reader should distinguish limits arising from selection from limits arising from measurement. The chapter’s decision structure also makes rejection more specific: an unsupported claim about audience effects is different from an uncertain estimate within a poorly described archive. Conclusions should retain those distinctions.

14. Experimental Analysis Introduction

The experimental section begins with the importance of causal questions in personal and public decisions. People often change behavior because they expect a benefit or hope to avoid harm, making evidence about causes consequential. An experiment attempts to isolate a change and examine its effects under controlled conditions. The introduction prepares readers to ask what was manipulated, what was measured, and what alternative explanations were addressed. It establishes the difference between finding an association and constructing a defensible causal argument.

15. Judgment Rule 1 for Experimental Analysis

The first experimental rule asks whether the researchers are investigating causality. A descriptive question about the characteristics of a population does not require the same design as a question about whether one factor changes another. Experimental reasoning gains its force from controlling competing explanations while varying the proposed cause. The chapter encourages readers to identify the causal claim before inspecting details. This keeps the evaluation focused on what an experiment is designed to establish rather than assuming that an experimental label automatically makes a study superior.

16. Judgment Rule 2 for Experimental Analysis

Research design must make alternative explanations sufficiently implausible. Reisner contrasts comparisons between groups with repeated measurements involving the same participants. Random assignment can help make groups comparable before treatment, while within-person designs introduce concerns about order and prior exposure. The chapter directs attention to the actual comparison on which a causal conclusion depends. Readers should determine whether a difference attributed to treatment might instead reflect pre-existing differences, timing, or the sequence of conditions. Design quality is therefore a matter of reasoning about plausible alternatives.

17. Judgment Rule 3 for Experimental Analysis

A proposed cause must be represented by a concrete treatment. Researchers cannot manipulate a broad category in the abstract; they choose a particular message, activity, object, or experience. The chapter asks whether that choice reasonably represents the intended factor and whether the control condition provides a meaningful comparison. An effect produced by an unusually narrow or confounded treatment may support a more limited conclusion than the researchers announce. Readers should visualize what participants actually encountered rather than evaluating the experiment solely through its theoretical vocabulary.

18. Judgment Rule 4 for Experimental Analysis

The dependent variable must be able to detect the kind of change the treatment is supposed to produce. Different measures can refer to intentions, reported preferences, observable choices, or other outcomes, and these are not interchangeable. Reisner asks readers to inspect how the outcome was defined and measured. A null result may be difficult to interpret if the instrument could not register the relevant change. Conversely, a measurable difference should not automatically be described as a broader transformation that the measure never assessed.

19. Summary Judgment Rules for Experiments

The experimental summary combines causal purpose, design, treatment, and outcome measurement. Its central task is to evaluate whether the observed difference can reasonably be attributed to the manipulated factor. Readers must also separate the causal inference supported by a design from questions about how widely it applies. A persuasive appraisal reconstructs the comparison and identifies the conditions under which it was made. The chapter turns the earlier rules into a practical sequence for judging a study’s evidential strengths and remaining uncertainties.

20. Compiling a Summary of Research Findings

Reisner shifts from appraising one study to explaining what a collection of studies establishes. A synthesis needs a clearly defined question because research, policy, and moral questions ask different things. Empirical findings can inform a decision without supplying its entire justification. The chapter emphasizes relevance and methodological soundness when deciding which findings belong in an account. A useful summary should preserve the boundaries of each result while showing how several results contribute to a wider picture.

21. An Example: Building a Summary

The worked example demonstrates synthesis as a sequence of decisions rather than a collage of abstracts. Starting from a specific question, the reader examines notes on methods and findings and decides what each study can contribute. Differences in definitions, materials, and procedures affect whether apparently similar results can be combined. The example makes the earlier appraisal rules operational: limitations remain attached to the evidence when it is incorporated into a larger account. The objective is a coherent description of knowledge whose reasoning remains traceable.

22. Conclusions

The conclusion returns to science as a method-dependent way of knowing. Different procedures answer different questions and possess characteristic strengths and weaknesses. Research literacy therefore involves recognizing both what evidence permits and what it leaves unsettled. The earlier judgment rules support reasoned confidence rather than either unquestioning acceptance or blanket skepticism. Reisner’s final lesson is that the quality of a claim depends on the path from question to observation to conclusion, and that an informed reader can examine that path.

Study instructions

1. Choose one published survey article and restate its research question in a single sentence before reading its conclusions.
2. Record the intended population, the participants actually questioned, and the information each survey item could reasonably obtain.
3. Apply the first judgment rule: explain whether the desired information is accessible through respondents’ answers. Distinguish what people report from independently observed behavior.
4. Write a short appraisal with two separate decisions: whether the method fits the question and how broadly the result can be interpreted. Use the later method-specific chapters to extend the appraisal.

Questions for reflection

• Could the same questionnaire be suitable for one research question and unsuitable for another?
• What additional evidence would justify extending a finding beyond the people studied?

Sources and coverage

These original summaries draw on chapter introductions, selected explanatory passages, examples, and concluding material across the chapters or units listed above. They are a condensed study guide, rather than a page-by-page account of the complete text. Supplementary sections are identified separately.

Official book and contents; official PDF.