3 Designs That Support Causal Claims
Where this chapter fits. Step 3 of the research process is to choose the research design. This chapter covers the designs that can support a claim about cause, and what the other designs can support.
3.1 The Same Resume With Two Names
In 2001 and 2002, two economists answered help-wanted ads in Boston and Chicago newspapers. Marianne Bertrand and Sendhil Mullainathan sent 4,870 resumes in response to more than 1,300 ads. The jobs were in sales, administrative support, clerical work, and customer service.
The resumes were fictitious. The researchers built them from real resumes and then assigned a first name to each one at random. Half of the resumes received a name that is more common among white Americans, such as Emily or Greg. The other half received a name that is more common among Black Americans, such as Lakisha or Jamal. The researchers then recorded which resumes received a call or an email from an employer.
Resumes with white-sounding names received a callback 9.65 percent of the time. Resumes with Black-sounding names received a callback 6.45 percent of the time. Figure 3.1 shows the two rates.
A difference of 3.2 percentage points may look small. In relative terms, a resume with a white-sounding name was about 50 percent more likely to receive a callback. An applicant with a white-sounding name needed to send about 10 resumes for each callback. An applicant with a Black-sounding name needed to send about 15.
The size of the gap is one reason the study is widely cited. The design is the other reason. Because the names were assigned at random, the study supports a claim that most studies of discrimination cannot support. The name on the resume caused the difference in callbacks. This chapter explains why the design allows that claim.
3.2 Why Comparing Groups Is Not Enough
Suppose a researcher took a different approach. The researcher collects records of real job applicants and compares callback rates for Black and white applicants. Suppose the white applicants received more callbacks.
That result would be an association. Two variables are associated when the values of one tend to go with certain values of the other. An association does not show that one variable causes the other.
Real applicants differ in many ways besides race. They differ in education, work history, the neighborhoods they live in, and the jobs they apply for. Any of those differences could account for a gap in callbacks. A variable that is related to both the supposed cause and the outcome is a confounder. With records of real applicants, the researcher cannot rule out confounders, because no two applicants are identical in everything except race.
Researchers generally require three conditions before they accept that one variable causes another.
- Association. The two variables are related.
- Time order. The cause comes before the effect.
- No alternative explanation. No third variable accounts for the relationship.
A comparison of real applicants can meet the first two conditions. It rarely meets the third.
3.3 What Random Assignment Does
An experiment is a study in which the researcher controls who receives which condition. The variable the researcher controls is the independent variable, also called the treatment. The outcome the researcher measures is the dependent variable. In the resume study, the type of name is the independent variable and the callback is the dependent variable.
Random assignment means that chance alone decides which unit receives which condition. In the resume study, a computer assigned the names. The quality of a resume had no influence on which name it received.
Random assignment addresses the third condition. Because chance decided the names, the two groups of resumes are alike, on average, in every other respect. They have similar education, similar experience, and similar formatting. The same is true for characteristics the researchers did not think to measure. Only the name differs in a consistent way. If callbacks differ between the groups by more than chance would produce, the name is the only remaining explanation.
The data file allows a check. Table 3.1 compares the two groups on three resume characteristics.
| Name group | Resumes | Mean years of experience | With a college degree | Rated high quality |
|---|---|---|---|---|
| Black-sounding name | 2,435 | 7.8 | 72.3% | 50.2% |
| White-sounding name | 2,435 | 7.9 | 71.6% | 50.2% |
The two groups match closely on all three. This is what random assignment is expected to produce when the groups are large.
3.4 Work With the Data: Random Assignment Lab
The study found a gap of 3.2 percentage points in callbacks. Chance alone produces some gap between any two groups. The tool shows how large a gap chance produces here. It uses the callback totals from the study.
Open Random Assignment Lab in its own window
- Press “Shuffle 1,000 times.” The tool deals the 392 callbacks into two groups of 2,435 at random and plots the gap from each shuffle.
- Read how many of the 1,000 shuffles gave a gap of 3.2 points or more.
- Read the largest gap that chance produced. Compare it with the dark line at 3.2 points.
- Press the button several more times. Note how little the answer changes.
In most runs, no shuffle reaches 3.2 points. Chance is therefore a poor explanation for the gap in the study. Random assignment is what permits this reasoning. The names were assigned by chance, so chance and the names are the only two explanations left.
The full file of 4,870 resumes is ch03-resume-audit.csv at quant.marginoferrormedia.com/data.
These two terms sound alike and do different jobs. Random sampling decides who is in the study. It supports the claim that the results describe a larger population. Random assignment decides who receives which condition. It supports the claim that the treatment caused the outcome. A study can have one, both, or neither. The resume study had random assignment. It did not have a random sample of all employers.
3.5 Internal and External Validity
Two questions apply to any study that claims a cause.
Internal validity concerns whether the treatment caused the outcome within the study. Random assignment gives the resume study strong internal validity. Few alternative explanations remain for the gap in callbacks.
External validity concerns whether the result holds for other people, places, and times. Here the resume study is more limited. It covered two cities, four job categories, and ads placed in newspapers in 2001 and 2002. It measured callbacks, which are the first step in hiring, and it did not measure job offers or pay.
The two kinds of validity often compete. A tightly controlled experiment has strong internal validity and may not resemble ordinary conditions. A large survey of real workers resembles ordinary conditions and cannot rule out confounders. No single study settles a question. Confidence grows when studies with different strengths reach the same conclusion.
Later research has addressed the external validity of the resume study. Kline, Rose, and Walters (2022) sent more than 83,000 fictitious applications to 108 of the largest employers in the United States. Applications with distinctively Black names were contacted about 2 percentage points less often. The gap was smaller than in the original study, and it was concentrated in a subset of firms. The direction of the result was the same twenty years later, with different employers and online applications.
3.6 A Second Factor
The researchers varied a second characteristic. Each resume was written as either higher quality or lower quality. The higher-quality resumes listed more experience, fewer gaps in employment, and more skills. This gave the study four groups in place of two. A design that crosses two independent variables is a factorial design.
Figure 3.2 shows the four rates. A better resume helped applicants with white-sounding names more than it helped applicants with Black-sounding names. When the effect of one variable depends on the level of another, the two variables interact. A study with only one factor could not have found this result.
3.7 What the Study Does Not Show
A careful reader separates what a design supports from what it does not.
The study shows that the name on a resume changed the callback rate. It does not show why employers responded as they did. The study cannot separate a preference against Black applicants from an assumption about applicants with those names.
Several researchers have also asked what a name signals. Fryer and Levitt (2004) found that distinctively Black names are more common among children of mothers with less education. An employer could read a name as a signal of social class as well as of race. Gaddis (2017) tested this directly and found that people do not perceive all such names in the same way. The treatment in the study was a name. The conclusion concerns race. The step from one to the other is a validity question of the kind raised in Chapter 2.
The study also measured the first step in hiring only. A gap in callbacks may grow or shrink at the interview and the offer.
These limits do not weaken the central result. They mark where it ends.
3.8 When an Experiment Is Not Possible
Many questions cannot be studied with an experiment. A researcher cannot assign people to lose their housing or to grow up in a poor neighborhood. In those cases, researchers use other designs.
- A cross-sectional study measures a group once. Most surveys are cross-sectional. They describe a population and show associations.
- A longitudinal study measures the same people more than once. It can show that a change in one variable came before a change in another, which addresses time order.
- A quasi-experiment compares groups that were formed by an event the researcher did not control. A new law that applies in one state and not in a neighboring state is an example. The groups were not assigned at random, and so the researcher has to argue that they were similar before the event.
Each design supports some claims and not others. Table 3.2 summarizes them.
| Design | Researcher assigns the treatment | Association | Time order | Rules out confounders |
|---|---|---|---|---|
| Cross-sectional survey | No | Yes | Rarely | No |
| Longitudinal study | No | Yes | Yes | Partly |
| Quasi-experiment | No | Yes | Yes | Partly, with an argument |
| Experiment with random assignment | Yes | Yes | Yes | Yes, on average |
The resume study is a special kind of experiment called an audit study. An audit study sends matched testers or matched applications into a real market and records how they are treated. It combines the control of an experiment with the conditions of a real market.
The employers in the resume study did not know they were in a study. They spent time reading applications from people who did not exist. Research ethics normally require informed consent, and this study did not obtain it. Review boards allow a waiver of consent when three conditions hold. The study could not be done if participants knew about it. The risk to participants is small. The knowledge gained is important. The researchers also did not publish the names of the employers. Reasonable people disagree about whether these conditions were met. A researcher who plans a study that involves deception has to make the case to a review board before the study begins.
A language model can explain random assignment and can suggest a design for your question. It cannot make your data come from that design. If you give a model a file of survey responses and ask what causes what, it will often answer in causal language. The language does not change what the data support. Before you accept a causal sentence from a model, ask how the treatment was assigned. If no one assigned it, treat the sentence as a statement about association.
3.9 Putting It All Together
An association between two variables does not show that one causes the other. A causal claim requires an association, the right time order, and no alternative explanation. Random assignment addresses the third condition by making the groups alike, on average, in everything except the treatment.
The resume study assigned names at random to 4,870 resumes. Resumes with white-sounding names received callbacks at a rate of 9.65 percent, and resumes with Black-sounding names at 6.45 percent. The design supports the claim that the name caused the gap. It does not show why employers responded as they did, and it measured only the first step in hiring.
Internal validity concerns whether the treatment caused the outcome within the study. External validity concerns whether the result holds elsewhere. When an experiment is not possible, longitudinal studies and quasi-experiments can meet some of the conditions for a causal claim.
3.10 Looking Ahead
The resume study measured a simple outcome. An employer called back or did not. Many research questions concern constructs that are much harder to measure, such as hunger, health, or trust. Chapter 4 covers how researchers build a measure from several questions and how they check that it works.
3.11 Further Reading and References
- Bertrand, M., and Mullainathan, S. (2004). Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American Economic Review, 94(4), 991 to 1013. The source of the case and the data.
- Kline, P., Rose, E. K., and Walters, C. R. (2022). Systemic discrimination among large U.S. employers. Quarterly Journal of Economics, 137(4), 1963 to 2036. A much larger audit study of 108 large firms.
- Fryer, R. G., and Levitt, S. D. (2004). The causes and consequences of distinctively Black names. Quarterly Journal of Economics, 119(3), 767 to 805.
- Gaddis, S. M. (2017). How Black are Lakisha and Jamal? Racial perceptions from names used in correspondence audit studies. Sociological Science, 4, 469 to 489.
- Shadish, W. R., Cook, T. D., and Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin. The standard reference on validity and design.
3.12 Key Terms
- Association. A relationship in which the values of one variable tend to go with certain values of another.
- Audit study. An experiment that sends matched testers or applications into a real market.
- Confounder. A variable related to both the supposed cause and the outcome.
- Cross-sectional study. A study that measures a group once.
- Dependent variable. The outcome a researcher measures.
- Experiment. A study in which the researcher controls who receives which condition.
- External validity. Whether a result holds for other people, places, and times.
- Factorial design. A design that crosses two or more independent variables.
- Independent variable. The variable a researcher controls. Also called the treatment.
- Interaction. A pattern in which the effect of one variable depends on the level of another.
- Internal validity. Whether the treatment caused the outcome within the study.
- Longitudinal study. A study that measures the same people more than once.
- Quasi-experiment. A comparison of groups formed by an event the researcher did not control.
- Random assignment. The use of chance alone to decide which unit receives which condition.
3.13 Exercises
Check Your Understanding
- State the three conditions for a causal claim. Which one does random assignment address?
- Explain the difference between random assignment and random sampling. Which did the resume study use?
- A study finds that people who own dogs report better health. Name two possible confounders.
- Define internal validity and external validity. Rate the resume study on each and give one reason for each rating.
Apply It
- In the resume study, 235 of 2,435 resumes with white-sounding names received a callback, and 157 of 2,435 with Black-sounding names did. Calculate each rate, the difference in percentage points, and the ratio of the two rates. Show each step.
- Use the file
ch03-resume-audit.csv. Calculate callback rates by gender and type of name. Describe the pattern in two sentences. - A city wants to know whether a text-message reminder raises attendance at housing court. Design an experiment. Name the independent variable, the dependent variable, the unit that is assigned, and how you would assign it.
Think Deeper
- The treatment in the resume study was a name, and the conclusion concerns race. What does a reader have to assume to move from one to the other? How could a new study test that assumption?
- Employers in the study did not consent. Argue for and against the waiver of consent. What would you require of the researchers?
- A company uses software to screen resumes. Describe an audit study that would test whether the software treats applicants differently by the type of name. What would the study show, and what would it leave open?