Is a Chance of Immediate Death Tolerable in Exchange for a Longer Expected Life?
by Paul R. Rosenbaum
In the daily news and the scientific literature, we are faced with conflicting claims about the effects caused by some treatments, behaviors, and policies. A daily glass of wine prolongs life, or so we are told. Yet we are also told that alcohol can cause life-threatening cancer and that pregnant women should abstain from drinking. Some say that raising the minimum wage decreases inequality while others say it increases unemployment. Investigators once confidently claimed that hormone replacement therapy reduces the risk of heart disease but today investigators confidently claim it raises that risk. How should we study such questions? Observation and Experiment is an introduction to causal inference from one of the field’s leading scholars. Using minimal mathematics and statistics, Paul Rosenbaum explains key concepts and methods through scientific examples that make complex ideas concrete and abstract principles accessible. Here is a brief excerpt.
A patient who is severely ill may face a choice of treatments. One treatment, perhaps a surgical treatment, places the patient at increased risk of death for a short period of time but offers much improved prospects if the patient does survive that short period of time. Another treatment, say, a form of radiation treatment, carries little immediate risk but a smaller improvement in longer-term prospects. That is a consequential choice, not a trivial one. Moreover, reasonable people might reasonably have different preferences. How can this choice be used to demonstrate that people often harbor consequential but irrational preferences?
Barbara McNeil and colleagues divided a small population of 583 individuals into two groups. The first group was given some background information and offered the following choice.
Surgery: Of 100 people having surgery, 90 live through the postoperative period, 68 are alive at the end of the first year, and 34 are alive at the end of five years.
Radiation Therapy: Of 100 people having radiation therapy, all live through the treatment, 77 are alive at the end of one year, and 22 are alive at the end of five years.
In the group that was offered this choice, 18% preferred radiation therapy, and 82% preferred surgery.
The second group was offered almost exactly the same choice. The only difference was that instead of speaking of how many people would live, the description spoke of how many people would die. For instance, instead of saying that 90/100 people receiving surgery would survive the postoperative period, this same mortality rate was described as 10/100 people receiving surgery would die in the postoperative period. The mortality rates were identical all the way through the two descriptions, as was the background information. The only difference was the emphasis on living in the first description and on dying in the second description.
In the group that received the second description, 44% of people preferred radiation, and 56% of people preferred surgery. So many more people preferred radiation — a guarantee of short-term survival — when the mortality rates emphasized dying than when they emphasized living, even though the actual mortality rates are the same, and only the emphasis had changed.
In other words, we estimate a shift of 44% − 18% = 26% in population preference based on something totally inconsequential, namely, speaking of 90/100 surviving rather than 10/100 dying. This 26% shift estimates that at least 26% of the individuals in the population would change their individual preferences about a grave choice based on something inconsequential, surely a behavior that everyone in the population would regard as irrational. Yet every individual in the population can claim, without fear of contradiction, to be among the remaining 100% − 26% = 74% of the population that avoids irrationality, holding the same preference no matter how it is described.
The 583 individuals in this experiment came from three groups: patients with chronic medical problems, radiologists, and students in business school. The pattern of preference shifts was similar for all three groups.
The experiment demonstrates that a preference in a grave decision is responsive to something to which no rational preference should be responsive. If your preference in a grave matter changes in response to something you yourself regard as inconsequential, then your preference is irrational in your own eyes. Of course, you never see this about yourself, nor would anyone else. We see an irrational responsiveness to something inconsequential only for the population of 583 individuals as a whole. That is, 100% of 583 individuals are in a position to ridicule the foolish 26%. We can all agree that there are a lot of crazy people out there, but of course you and I are perfectly rational.
Let us write this argument out a little more carefully. As in Chapter 2, every individual i, i =1, 2, . . . , I = 583, has two potential preferences, rTi and rCi. Here, rTi =1 if person i would prefer radiation if mortality rates were described in terms of the chance of surviving, as in the quote above, and rTi = 0 if person i would prefer surgery in this situation. In parallel, rCi =1 if person i would prefer radiation if mortality rates were described in terms of the chance of dying, and rCi = 0 if person i would prefer surgery in this situation. If people were rational in their preferences about grave decisions, then their preferences would not shift based on something inconsequential. In particular, people would not change their preferences if the factual situation was unchanged but was described in a different but obviously equivalent way. You might prefer one thing, and I might prefer something else, but if we were rational, neither of us would change our preferences in response to an inconsequential change in the description of unchanged facts. That is, if the I = 583 people were rational, then rTi = rCi for every individual i, i =1, 2, . . . , I = 583. In other words, McNeil and colleagues built their experiment in such a way that rational preferences correspond with Fisher’s null hypothesis of no difference in treatment effects, namely, H0: rTi = rCi, i =1, 2, . . . , I = 583. We saw rTi for people who heard the mortality rates described in terms of survival, the treated group with Zi =1, and we saw rCi for people who heard the mortality rates described in terms of dying, the control group with Zi = 0, but we never saw both rTi and rCi for the same person i, so we never saw an individual exhibit an irrational preference, rTi ≠ rCi.
Now the average value of rTi is 3T = 0.18 =18% for people who heard mortality rates described in terms of surviving, people with Zi = 1. Also, the average value of rCi is 3C = 0.44 = 44% for people who heard mortality rates described in terms of dying, people with Zi = 0. So our estimate of the average treatment effect is 3T − 3C = 18% − 44% = −26%. These two averages,
3T and 3C, describe the preferences of different people, so we need to ask whether this 26% difference could occur by chance or whether it constitutes strong evidence against Fisher’s hypothesis of no effect — that is, strong evidence that some individuals i harbor irrational preferences with rTi ≠ rCi. Reasoning as in Chapter 3, we find that the two-sided P-value testing Fisher’s hypothesis is 1.4 × 10−11, so a difference of 26% or more is very improbable with I = 583 people if Fisher’s hypothesis were true. We conclude that Fisher’s hypothesis is implausible, that many of the 583 people changed their preferences in a grave decision in response to something inconsequential.
Obviously, if you want to demonstrate that a preference can be irrational, then you cannot offer one person both choices. If you offered one person both choices, then that one person would recognize the irrationality of offering inconsistent responses to identical mortality rates described differently. However, if you offer random halves of a single population one description or the other, then you can demonstrate that the population must contain individuals who would change their preferences in response to something inconsequential. This is true even though neither the investigator nor the 583 experimental subjects knows whose preferences are consistent and whose are irrational.
This experiment and many similar experiments demonstrate that people often arrive at consequential preferences in an irrational way. However, these experiments also exhibit forms of irrationality that are fragile. Presumably, few people, perhaps no one, would have rTi ≠ rCi if people were forced to re-write the description they were given in its equivalent but alternative form. The irrationality exhibited in this experiment might vanish if people were asked to consider the matter more carefully, more thoughtfully, from more than one perspective. The exhibited irrationality might vanish if people were encouraged and guided to think rationally about their preferences.
