Wednesday, March 28, 2007

A final word on Number Needed to Treat

In my previous post in this series I discussed how to create confidence intervals for the Number Needed to Treat (NNT). I just left it as taking the reciprocal of the confidence limits of the absolute risk reduction. I tried to find a better way, but I suppose there's a reason that we have a rather unsatisfactory method as a standard practice. The delta method doesn't work very well, and I suppose methods based on higher-order Taylor series will not work much better.

So, what happens if the treatment has no statistically significant effect (sample size is too small or the treatment simply doesn't work). The confidence interval for absolute risk reduction will cover 0, say, maybe -2.5% to 5%. Taking reciprocals, you get an apparent NNT confidence interval of -40 to 20. A negative NNT is easy enough to interpret: -40 NNT means that for every 40 people you "treat" with the failed treatment, you get a reduction of 1 in favorable outcomes. A 0 absolute risk reduction results in NNT=∞. So if the confidence interval of absolute risk reduction covers 0, the confidence interval must cover ∞. In fact, in the example above, we get the bizarre confidence set of -∞ to -40 and 20 to ∞, NOT -40 to 20. The interpretation of this confidence set (it's no longer an interval) is that either you have to treat at least 20 people but probably a lot more to help one, or if you treat 40 or more people then you might harm one. For this reason, for a treatment that doesn't reach statistical significance (i.e. whose absolute risk reduction includes 0), the NNT is often reported as a point estimate. I would argue that such a point estimate is meaningless. In fact, if it were left up to me, I would not report an NNT for a treatment that doesn't reach statistical significance, because the interpretation of statistical non-significance is that you can't prove with the data you have that the treatment helps anybody.

Douglas Altman, heavy hitter in medical statistics, has the gory details.

Technorati Tags: ,

Wednesday, March 21, 2007

SAS weirdness

From time to time, I'll complain about the weirdness of SAS, the statistical analysis program of choice for much of the pharmaceutical industry. This post is one such complaint.

Why, oh why, does SAS not directly give us the asymptotic variance of the Mantel-Haenszel odds ratio estimate? It does, however, give the confidence interval. Though the default is a 95% confidence interval, by specifying alpha=31.4 in the TABLES statement in the FREQ procedure and using ODS output to get these values into a dataset, you can compute the asymptotic variance by either dividing the upper confidence limit by the Mantel-Haenszel odds ratio estimate, or dividing the MH estimate by the lower confidence limit (both should give the same answer). The point is, SAS has to compute the asymptotic variance to calculate the confidence interval, so why not just go ahead and display it? (Yes, I understand that the confidence interval is symmetric only on a log scale.)

Addendum: R doesn't either. Same story. Weird.