5 Writing Questions
Where this chapter fits. Step 4 of the research process continues. This chapter covers the wording of the questions that go on the form.
5.1 A Question That Took Years to Write
Each year, state health departments in the United States interview more than 400,000 adults by telephone. The survey is the Behavioral Risk Factor Surveillance System. The Centers for Disease Control and Prevention has coordinated it since 1984. One of its questions reads as follows.
Now thinking about your physical health, which includes physical illness and injury, for how many days during the past 30 days was your physical health not good?
The question looks simple. Each part of it reflects a decision. “Physical health” is defined inside the question, so that respondents do not have to guess what it covers. The time period is stated as 30 days. The answer is a number of days, which is easier to report than a rating. The wording has been kept the same for decades, so that results can be compared across years and across states.
Suppose a county health department wants to survey its own residents. The staff could write new questions. A better plan is to borrow tested questions such as this one and add new questions only where no tested question exists. This chapter covers both tasks. It explains how respondents answer a question, how wording can change the answers, and how to test a questionnaire before using it.
5.2 How a Respondent Answers
Tourangeau, Rips, and Rasinski (2000) describe four steps that a respondent goes through to answer a survey question.
- Comprehension. The respondent works out what the question is asking.
- Retrieval. The respondent searches memory for the relevant information.
- Judgment. The respondent combines what was recalled into an answer.
- Response. The respondent fits that answer to the choices offered and decides whether to report it.
An error can enter at each step. A respondent may misread a term. The respondent may not remember, may estimate carelessly, or may choose an answer that sounds better than the truth. Good question writing makes each step easier.
The health question above supports each step. It defines its term, which helps comprehension. It limits the period to 30 days, which helps retrieval. It asks for a count of days, which makes the judgment concrete. It accepts any number from 0 to 30, so the respondent does not have to fit the answer to categories.
5.3 Wording Changes the Answers
Small changes in wording can produce large changes in results. Researchers have shown this with split-ballot experiments. In a split-ballot experiment, a random half of the sample receives one version of a question and the other half receives a second version. Because the halves were formed at random, a difference in the answers can be attributed to the wording.
Figure 5.1 shows two well-known results.
In 1940, one half of a national sample was asked whether the United States should forbid public speeches against democracy. Fifty-four percent said yes. The other half was asked whether the United States should allow such speeches. Seventy-five percent said no. To forbid and to not allow mean the same thing. The share that chose that position differed by 21 points.
In the 1980s, a national survey asked about government spending. Half the sample was asked about spending on “welfare.” The other half was asked about spending on “assistance to the poor.” In 1986, about 23 percent said too little was spent on welfare. About 63 percent said too little was spent on assistance to the poor.
Neither result means that respondents were careless. Words carry associations, and respondents react to them. A researcher cannot write a question that has no wording. The researcher can choose neutral words, keep the wording fixed over time, and report the exact question with every result.
5.4 Common Problems in Questions
Most weak questions have one of a small number of problems. Table 5.1 lists them with an example and a revision.
| Problem | Example | Revision |
|---|---|---|
| Double-barreled. Two questions in one. | “How satisfied are you with the cost and quality of your health care?” | Ask about cost and quality in two questions. |
| Leading. The wording suggests an answer. | “Do you agree that the county should do more to protect children’s health?” | “Should the county spend more, less, or about the same on children’s health programs?” |
| Loaded. A word carries a strong association. | “Do you support handouts for people who do not work?” | “Do you favor or oppose cash assistance for adults who are unemployed?” |
| Vague term. A word that people read differently. | “Do you exercise regularly?” | “In the past 7 days, on how many days did you exercise for at least 30 minutes?” |
| Double negative. Two negatives in one sentence. | “Do you disagree that smoking should not be allowed in parks?” | “Should smoking be allowed in public parks?” |
| Jargon. A term the respondent may not know. | “Have you used telehealth modalities?” | “In the past 12 months, have you had a medical visit by phone or video?” |
| Long recall period. The period is too long to remember. | “How many times did you see a doctor in the past five years?” | “In the past 12 months, how many times did you see a doctor?” |
| Assumed knowledge. The question assumes a fact. | “How would you rate the county’s new clinic?” | First ask whether the respondent has visited the clinic. |
A useful habit is to read each question aloud and ask what a respondent would need to know to answer it. If the answer includes a definition you did not supply, add it to the question.
5.5 Work With the Data: Question Reviewer
Table 5.1 lists eight problems. The tool checks a draft question for several of them. It uses fixed rules, so it can miss a problem and can flag wording that is fine.
Open Question Reviewer in its own window
- Press “Review the question” for the sample question. The tool marks two problems. Find each one in Table 5.1.
- Rewrite the sample as two neutral questions. Review each one.
- Press “Example 2” and “Example 3.” Predict the problems before you read what the tool marks.
- Type a question of your own and review it. Revise until the rules find nothing.
A question with no marks has passed a first check. It has not yet been tested. Only people from the population can show whether they understand the question as you intended. A later section of this chapter covers how to test with them.
5.6 The Answer Choices
A question is not complete without its answer choices. An open-ended question lets the respondent answer in any words. A closed-ended question offers a fixed set of choices.
Open-ended questions collect answers the researcher did not expect. They take longer to answer and have to be coded before they can be counted. Closed-ended questions are quick to answer and easy to count. They limit the respondent to the choices the researcher wrote.
Fixed choices have to meet two requirements. They should be mutually exclusive, so that no answer fits two choices. They should be exhaustive, so that every respondent can find a choice that fits. Age groups of “18 to 30” and “30 to 45” fail the first test for a respondent who is 30. A list of health insurance types without “none” fails the second.
Rating scales raise three further choices.
Number of points. Five to seven points work well for most rating scales. Fewer points lose detail. More points ask respondents for finer judgments than most can make.
Balance. A scale should offer the same number of positive and negative choices. “Excellent, very good, good, fair, poor” has three positive choices and two that are not. The health survey has used this scale for decades, and its results are well understood. A new scale should be balanced.
Labels. Label every point with words where possible. A scale with labels only at the two ends leaves respondents to decide what the middle points mean.
A final choice is whether to offer “don’t know.” Without it, respondents who have no opinion will often pick one anyway. With it, some respondents who do have an opinion will choose it to save effort. Offer “don’t know” for questions of fact that a respondent might not know. Consider leaving it out for questions about the respondent’s own experience.
5.7 Bias in the Answers
Some patterns in answers come from the respondent and not from the wording.
Social desirability bias is the tendency to give the answer that makes a good impression. Respondents tend to report more voting and exercise than they do, and less drinking and smoking. The bias is stronger when an interviewer is present and weaker on a self-completed form. It can be reduced with neutral wording and a credible promise that answers are confidential.
Acquiescence is the tendency to agree with statements regardless of content. A scale made only of agree-disagree items will inflate agreement. One remedy is to mix items worded in both directions, as described in Chapter 4. A second is to ask a direct question. “How would you rate your health?” is better than “My health is good. Agree or disagree?”
Satisficing is the tendency to give an answer that is good enough with little thought. A respondent who is tired or bored may pick the first reasonable choice or give the same rating to every item in a grid. Short questionnaires and varied formats reduce it. Chapter 7 covers how to detect it in the data.
5.8 The Order of Questions
A question can be affected by the questions that come before it. This is a question-order effect.
In 1948, Hyman and Sheatsley asked Americans about Communist reporters from other countries. Should they be allowed into the United States to report the news as they saw it? When this question came first, 36 percent said yes. A second version first asked whether American reporters should be allowed into Russia. When the question about Communist reporters followed that one, 73 percent said yes. Respondents who had just endorsed access for American reporters applied the same standard to the other side.
Order effects cannot be removed. They can be managed with a few rules.
- Begin with questions that are easy and that relate to the stated topic.
- Move from general questions to specific ones within a topic.
- Group questions on the same topic and use a short sentence to introduce each group.
- Place sensitive questions after the respondent has had time to build trust.
- Place demographic questions such as age and income at the end.
- Keep the order fixed across years if results will be compared.
A filter question routes respondents past questions that do not apply to them. A respondent who has not visited the county clinic should skip the questions that rate it. Online survey software applies this routing automatically, and the routing has to be tested like any other part of the questionnaire.
5.9 Test Before You Field
No researcher can predict every way a question will be read. A questionnaire has to be tested with people before it is used. Three methods are standard.
Expert review. People who know the topic and people who know survey design read the draft. They catch missing content and known wording problems.
Cognitive interviewing. A small number of people from the target population answer the draft questions and explain their thinking aloud. The interviewer asks follow-up questions such as “What does ‘physical health’ mean to you in this question?” Five to fifteen interviews usually reveal the main problems. This method tests comprehension directly, and it is the most useful of the three.
Pilot test. The full questionnaire is given to a small sample under the conditions of the real survey. The pilot shows how long the survey takes and where respondents quit. It also shows which questions are skipped and whether the routing works.
Table 5.2 shows how the county in the example could put its questionnaire together.
| Section | Source of the questions | Testing needed |
|---|---|---|
| General health and healthy days | National health survey, used as written | None for wording. Check that they fit the mode. |
| Access to care | National health survey, used as written | None for wording. |
| Use of the county clinic | New questions | Expert review, cognitive interviews, pilot. |
| Housing and food | Six-item food security scale from Chapter 4 | None for wording. |
| Demographics | Census Bureau wording | None for wording. |
The same question can perform differently by phone, on paper, and on a screen. A long list of choices is hard to follow when read aloud. A grid of ratings is easy to fill in carelessly on a phone screen. When you borrow a question, check the mode in which it was tested. If your mode differs, include the question in your cognitive interviews.
The experiments in this chapter show that wording can move answers by 20 points or more. A sponsor who wants a particular result can choose the wording that produces it. A survey written this way is sometimes called a push poll or an advocacy survey. A researcher has two duties. The first is to write neutral questions even when a client would prefer otherwise. The second is to publish the exact wording of every question with the results, so that readers can judge it for themselves.
A language model can draft questions quickly and can find many of the problems in Table 5.1. Both uses save time. Two limits apply. A model tends to write questions that are long and formal, and they need editing for the reading level of your respondents. A model also cannot replace a cognitive interview. It can predict how a typical reader might interpret a question. It cannot tell you how a retired farmer or a recent immigrant in your county will read it. Use a model for drafting and for a first review, and then test the questions with people from the population you will survey.
5.10 Putting It All Together
A respondent answers a question in four steps. The steps are comprehension, retrieval, judgment, and response. Good wording makes each step easier. Split-ballot experiments show that small changes in wording and order can change results by 20 points or more.
Most weak questions are double-barreled, leading, loaded, vague, or too demanding of memory. Answer choices have to be mutually exclusive and exhaustive, and rating scales should be balanced and labeled. Social desirability, acquiescence, and satisficing affect answers regardless of the wording.
Borrow tested questions where they exist and use them as written. Test new questions with expert review, cognitive interviews, and a pilot.
5.11 Looking Ahead
A well-written questionnaire still has to reach the right people. Chapter 6 covers how a sample is drawn and what happens when part of the population has no chance of being included. The case is the 2020 Census, which counted the country accurately in total and missed several groups at high rates.
5.12 Further Reading and References
- Tourangeau, R., Rips, L. J., and Rasinski, K. (2000). The Psychology of Survey Response. Cambridge University Press. The source of the four-step model.
- Schuman, H., and Presser, S. (1981). Questions and Answers in Attitude Surveys. Academic Press. The standard collection of experiments on wording and order.
- Rugg, D. (1941). Experiments in wording questions: II. Public Opinion Quarterly, 5(1), 91 to 92.
- Rasinski, K. A. (1989). The effect of question wording on public support for government spending. Public Opinion Quarterly, 53(3), 388 to 394.
- Hyman, H. H., and Sheatsley, P. B. (1950). The current status of American public opinion. In J. C. Payne (Ed.), The Teaching of Contemporary Affairs. National Council for the Social Studies.
- Dillman, D. A., Smyth, J. D., and Christian, L. M. (2014). Internet, Phone, Mail, and Mixed-Mode Surveys (4th ed.). Wiley. A practical guide to questionnaire design.
- Willis, G. B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage.
- Centers for Disease Control and Prevention. Behavioral Risk Factor Surveillance System Questionnaires. The questionnaires for every year are public.
5.13 Key Terms
- Acquiescence. The tendency to agree with statements regardless of content.
- Closed-ended question. A question with a fixed set of answer choices.
- Cognitive interviewing. A test in which respondents explain their thinking as they answer draft questions.
- Double-barreled question. A question that asks two things at once.
- Exhaustive. A set of choices in which every respondent can find one that fits.
- Filter question. A question that routes respondents past questions that do not apply.
- Leading question. A question whose wording suggests an answer.
- Mutually exclusive. A set of choices in which no answer fits more than one.
- Open-ended question. A question the respondent answers in any words.
- Pilot test. A trial of the full questionnaire with a small sample.
- Question-order effect. A change in answers caused by the questions that came before.
- Satisficing. Giving an answer that is good enough with little thought.
- Social desirability bias. The tendency to give the answer that makes a good impression.
- Split-ballot experiment. A test in which random halves of a sample receive different versions of a question.
5.14 Exercises
Check Your Understanding
- Name the four steps a respondent goes through to answer a question. Give one error that can enter at each step.
- Identify the problem in each question. (a) “How often do you eat fruits and vegetables?” (b) “Don’t you agree that clinics should stay open later?” (c) “Are you satisfied with your doctor and your pharmacy?”
- Explain why a split-ballot experiment supports a causal claim about wording. Use the terms from Chapter 3.
- The age groups on a form are “under 25, 25 to 40, 40 to 65, over 65.” Which requirement do they fail?
Apply It
- Rewrite each question from Exercise 2 so that the problem is removed.
- Write a question that measures how many days in the past week a respondent ate breakfast. Include the answer choices. State which of the four steps your wording is meant to help.
- Use the file
ch05-wording-experiments.csv. For each of the three experiments, calculate the difference in percentage points between the two versions.
Think Deeper
- The general health question uses an unbalanced scale and has been asked the same way for decades. Should the survey switch to a balanced scale? What would be gained and what would be lost?
- A client asks you to word a question so that support for a policy looks as high as possible. Describe what you would say and what you would offer to do in place of the request.
- Draft three questions on a topic you care about. Ask two people to answer them and to explain what each question meant to them. Report what you learned and how you would revise the questions.