Thus far, we have used statistical methods to reach conclusions by seeing how compatible the observations were with the null hypothesis that the treatment had no effect. When the data were unlikely to occur if this null hypothesis was true, we rejected it and concluded that the treatment had an effect. We used a test statistic (F, t, z, or χ2) to quantify the difference between the actual observations and those we would expect if the null hypothesis of no effect were true. We concluded that the treatment had an effect if the value of this test statistic was bigger than 95% of the values that would occur if the treatment had no effect. When this is so, it is common for medical investigators to report a statistically significant effect. On the other hand, when the test statistic is not big enough to reject the hypothesis of no treatment effect, investigators often report no statistically significant difference and then discuss their results as if they had proven that the treatment had no effect. All they really did was fail to demonstrate that it did have an effect. The distinction between positively demonstrating that a treatment had no effect and failing to demonstrate that it did have an effect is subtle but very important, especially in the light of the small numbers of subjects included in most clinical studies.*

As already mentioned in our discussion of the t test, the ability to detect a treatment effect with a given level of confidence depends on the size of the treatment effect, the variability within the population, and the size of the samples used in the study. Just as bigger samples make it more likely that you will be able to detect an effect, smaller sample sizes make it harder. In practical terms, this fact means that studies of therapies that involve only a few subjects and fail to reject the null hypothesis of no treatment effect may arrive at this result because the statistical procedures lacked the power to detect the effect because of a too small sample size, even though the treatment did have an effect. Conversely, considerations of the power of a test permit you to compute the sample size needed to detect a treatment effect of given size that you believe is present.

* This problem is particularly encountered in small clinical studies in which there are no “failures” in the treatment group. This situation often leads to overly optimistic assessments of therapeutic efficacy. See Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA. 1983;249:1743–1745.

Now, we make a radical departure from everything that has preceded: we assume that the treatment does have an effect.

Figure 6-1 shows the same population of people we studied in Figure 4-3 except that this time the drug given to increase daily urine production works. It increases the average urine production for members of this population from 1200 to 1400 mL/day. Figure 6-1A shows the distribution of daily urine production for all 200 members of the population in the control (placebo) group, and Figure 6-1B shows the distribution of urine production for all 200 members of the population in the diuretic group.

More precisely, the population of people taking the placebo consist of a normally distributed population with mean μpla = 1200 mL/day and the population of people taking the drug consist of a normally distributed population with a mean of μdr = 1400 mL/day. Both populations have the same standard deviation, σ = 200 mL/day.

Of course, an investigator cannot observe all members of the population, so he or she selects two groups of 10 people at random, gives one group the diuretic and the other a placebo, and measures their daily urine production. Figure 6-1C shows what the investigator would see. The people receiving a placebo produce an average of 1180 mL/day, and those receiving the drug produce an average of 1400 mL/day. The standard deviations of these two samples are 144 and 245 mL/day, respectively. The pooled estimate of the population variance is

The value of t associated with these observations is

which exceeds 2.101, the value that defines the most extreme 5% of possible values of the t test statistic when the two samples are drawn from the same population. (There are ν = ndr + npla − 2 = 10 + 10 − 2 = 18 degrees of freedom.) The investigator would conclude that the observations are not consistent with the assumption that two samples came from the same population and report that the drug increased urine production. And he or she would be right.

Of course, there is nothing special about the two random samples of people selected for the experiment. Figure 6-2 shows two more groups of people selected at random to test the drug, together with the results as they would appear to the investigator. In this case, the mean urine production is 1216 mL/day for the people given the placebo and 1368 mL/day for the people taking the drug. The standard deviations of urine production in the two samples are 97 and 263 mL/day, respectively, so the pooled estimate of the variance is 1/2(972 + 2632) = 1982. The value of t associated with these observations is

which is less than 2.101. Had the investigator selected these two groups of people for testing, he or she would not have obtained a value of t large enough to reject the hypothesis that the drug had no effect and probably reported “no significant difference.” If the investigator went on to conclude that the drug had no effect, he or she would be wrong.

Notice that this is a different type of error from that discussed in Chapters 3, 4, 5. In the earlier chapters, we were concerned with rejecting the hypothesis of no effect when it was true. Now we are concerned with not rejecting it when it is not true. This situation is called a Type II error or β error.

What are the chances of making this second kind of error?

Just as we could repeat this experiment more than 1027 times when the drug had no effect to obtain the distribution of possible values of t (compare with the discussion of Fig. 4-4), we can do the same thing when the drug does have an effect. Figure 6-3 shows the results of 200 such experiments; 111 out of the resulting values of t fall at or above 2.101, the value we used to define a “big” t. Put another way, if we wish to keep the P value at or below 5%, there is a 111/200 = 56% chance of concluding that the diuretic increases urine output when average urine output actually increases by 200 mL/day. We say the power of the test is .56. The power quantifies the chance of detecting a real difference of a given size.

Alternatively, we could concentrate on the 89 of the 200 experiments that produced t values below 2.101, in which case we would fail to reject the hypothesis that the treatment had no effect and be wrong. Thus, there is an 89/200 = 44% = .44 chance of continuing to accept the hypothesis of no effect when the drug really increased urine production by 200 mL/day on the average.

Now we have isolated the two different ways the random-sampling process can lead to erroneous conclusions. These two types of errors are analogous to the false-positive and false-negative results one obtains from diagnostic tests. Before this chapter we concentrated on controlling the likelihood of making a false-positive error, that is, concluding that a treatment has an effect when it really does not. In keeping with tradition, we have generally sought to keep the chances of making such an error below 5%; of course, we could arbitrarily select any cutoff value we wanted at which to declare the test statistic “big.” Statisticians denote the maximum acceptable risk of this error by α, the Greek letter alpha. If we reject the hypothesis of no effect whenever P <.05, α = 0.05 or 5%. If we actually obtain data that lead us to reject the null hypothesis of no effect when the null hypothesis of no effect is true, statisticians say that we have made a Type I error. All this logic is relatively straightforward because we have specified how much we believe the treatment affects the variable of interest, that is, not at all.

What about the other side of the coin, the chance of making a false-negative conclusion and not reporting an effect when one exists? Statisticians denote the chance of erroneously accepting the hypothesis of no effect by β, the Greek letter beta. The chance of detecting a true positive, that is, reporting a statistically significant difference when the treatment really produces an effect, is 1 − β. The power of the test that we discussed earlier is equal to 1 − β. For example, if a test has power equal to .56, there is a 56% chance of actually reporting a statistically significant effect when one is really present. Table 6-1 summarizes these definitions.

So far we have developed procedures for estimating and controlling the Type I, or α, error. Now we turn our attention to keeping the Type II, or β, error as small as possible. In other words, we want the power to be as high as possible. In theory, this problem is not very different from the one we already solved with one important exception. Since the treatment has an effect, the size of this effect influences how easy it is to detect. Large effects are easier to detect than small ones. To estimate the power of a test, you need to specify how small an effect is worth detecting.

Just as with false positives and false negatives in diagnostic testing, the Type I and Type II errors are intertwined. As you require stronger evidence before reporting that a treatment has an effect, that is, make α smaller, you also increase the chance of missing a true effect, that is, make β bigger or power smaller. The only way to reduce both α and β simultaneously is to increase the sample size, because with a larger sample you can be more confident in your decision, whatever it is.

In other words, the power of a given statistical test depends on three interacting factors:

• The risk of error you will tolerate when rejecting the hypothesis of no treatment effect.
  
• The size of the difference you wish to detect relative to the amount of variability in the populations.
  
• The sample size.

To keep things simple, we will examine each of these factors separately.

The Size of the Type I Error α

Figure 6-3 showed the complementary nature of the maximum size of the Type I error α and the power of the test. The acceptable risk of erroneously rejecting the hypothesis of no effect, α, determines the critical value of the test statistic above which you will report that the treatment had an effect, P < α. (We have usually taken α = 0.05.) This critical value is defined from the distribution of the test statistic for all possible experiments with a specific sample size given that the treatment had no effect. The power is the proportion of possible values of the test statistic that fall above this cutoff value given that the treatment had a specified effect (here a 200 mL/day increase in urine production). Changing α, or the P value required to reject the hypothesis of no difference, moves this cutoff point, affecting the power of the test.

Figure 6-4 illustrates this point further. Figure 6-4A essentially reproduces Figure 6-3 except that it depicts the distribution of t values for all 1027 possible experiments involving two groups of 10 people as a continuous distribution. The top part, copied from Figure 4-4D, shows the distribution of possible t values (with ν = 10 + 10 − 2 = 18 degrees of freedom) that would occur if the drug did not affect urine production. Suppose we require P < .05 before we are willing to assert that the observations were unlikely to have arisen from random sampling rather than the effect of the drug. According to the table of critical values of the t distribution (see Table 4-1), for ν = 18 degrees of freedom, 2.101 is the (two-tail) critical value that defines the most extreme 5% of possible values of the t test statistic if the null hypothesis of no effect of the diuretic on urine production is true. In other words, when we make α = 0.05, in which case −2.101 and +2.101 delimit the most extreme 5% of all possible t values we would expect to observe if the diuretic did not affect urine production.