Wednesday, March 28, 2007

A final word on Number Needed to Treat

In my previous post in this series I discussed how to create confidence intervals for the Number Needed to Treat (NNT). I just left it as taking the reciprocal of the confidence limits of the absolute risk reduction. I tried to find a better way, but I suppose there's a reason that we have a rather unsatisfactory method as a standard practice. The delta method doesn't work very well, and I suppose methods based on higher-order Taylor series will not work much better.

So, what happens if the treatment has no statistically significant effect (sample size is too small or the treatment simply doesn't work). The confidence interval for absolute risk reduction will cover 0, say, maybe -2.5% to 5%. Taking reciprocals, you get an apparent NNT confidence interval of -40 to 20. A negative NNT is easy enough to interpret: -40 NNT means that for every 40 people you "treat" with the failed treatment, you get a reduction of 1 in favorable outcomes. A 0 absolute risk reduction results in NNT=∞. So if the confidence interval of absolute risk reduction covers 0, the confidence interval must cover ∞. In fact, in the example above, we get the bizarre confidence set of -∞ to -40 and 20 to ∞, NOT -40 to 20. The interpretation of this confidence set (it's no longer an interval) is that either you have to treat at least 20 people but probably a lot more to help one, or if you treat 40 or more people then you might harm one. For this reason, for a treatment that doesn't reach statistical significance (i.e. whose absolute risk reduction includes 0), the NNT is often reported as a point estimate. I would argue that such a point estimate is meaningless. In fact, if it were left up to me, I would not report an NNT for a treatment that doesn't reach statistical significance, because the interpretation of statistical non-significance is that you can't prove with the data you have that the treatment helps anybody.

Douglas Altman, heavy hitter in medical statistics, has the gory details.

Technorati Tags: ,

Wednesday, March 21, 2007

SAS weirdness

From time to time, I'll complain about the weirdness of SAS, the statistical analysis program of choice for much of the pharmaceutical industry. This post is one such complaint.

Why, oh why, does SAS not directly give us the asymptotic variance of the Mantel-Haenszel odds ratio estimate? It does, however, give the confidence interval. Though the default is a 95% confidence interval, by specifying alpha=31.4 in the TABLES statement in the FREQ procedure and using ODS output to get these values into a dataset, you can compute the asymptotic variance by either dividing the upper confidence limit by the Mantel-Haenszel odds ratio estimate, or dividing the MH estimate by the lower confidence limit (both should give the same answer). The point is, SAS has to compute the asymptotic variance to calculate the confidence interval, so why not just go ahead and display it? (Yes, I understand that the confidence interval is symmetric only on a log scale.)

Addendum: R doesn't either. Same story. Weird.

Tuesday, February 13, 2007

Alternative medicine use might be affecting the results of trials

From here.



At issue is whether remedy-drug interactions are skewing the results of Phase I cancer trials. At present, this is hard to determine because it is hard to elicit (alternative) remedy use. To me, it's pretty clear that having remedy use out in the open is better than being secretive. However, what's interesting to me is the following, surveyed from 212 patients with advanced cancer enrolled in Phase I clinical trials:



  • 72 (34 percent) use alternative remedies, similar to general US population usage
  • 41 (19.3 percent) take vitamins and minerals
  • 40 (18.9 percent) take herbal preparations
In addition, we have the following:

Sometimes, patients are reluctant to tell the doctor they are taking
alternative medicines, either because they don't think it's important,
or they don't want to be told to stop taking them, Daugherty said.

Also,

And, since it's often difficult to get cancer patients to take part in
phase 1 trials, some researchers may be reluctant to turn any potential
patient away. "In addition, most doctors don't know very much about
alternative medicine," Daugherty said.
For research, I think these matters need to be out in the open. For one thing, we need to understand our drugs we are developing. For another, we need to understand the alternative remedies.





Technorati Tags: ,

Friday, February 9, 2007

Big news that's easy to miss -- FDA clears a molecular prognostic tool for breast cancer metastasis

The FDA just approved a prognostic device. So what's the big deal?

I'll let the press release speak:

It is the first cleared molecular test that profiles genetic activity.

That's right. Despite many years (ok, about a decade to a decade and a half) of use in basic research, microarray technology has matured (along with the analysis methodologies) enough to be used in clinical practice, and this approval marks a big step toward that. Microarrays were a buzz in the statistical community a few years ago when there were still some methodological hurdles to overcome (and were being rapidly overcome).

What's more, this marks the first time a genetic expression test (different from a genetic test -- this one identifies which genes are expressed [active] at a particular time) has been approved for the prognosis of a disease. 70 gene expressions are analyzed. Agendia has blazed some trails, and I expect to see more of this kind of test in the coming years. And that's a good thing.

And, hopefully for women with breast cancer, this will help a bit in the decision making for treatment.

Tuesday, February 6, 2007

Dichloroacetate (DCA) - not an easy road to "cheap, patentless cancer cure"

Posts that contain Dca per day for the last 30 days.
Technorati Chart
Get your own chart!

Ginger, curcumin, and DCA have recently been touted for their anticancer properties, and, granted, in the Petri dish, they look pretty good. However, it's a long road from the Petri dish to the pharmacy shelves. Abel Pharmboy's coverage of DCA seems to be the most spot-on, and his points are worth repeating. Many compounds show promise in the Petri dish and animal models, but when it comes to human trials, they bomb. It is entirely possible, in fact, from drug development experience I would say very likely, that DCA will do very little for humans in trials. It may be ineffective when we actually inject it into human beings (for anticancer purposes; it's already approved for some metabolic disorders).

Remember, cancer is a complex disease. To "cure cancer" is really to cure a whole lot of different diseases, which is why our "war on cancer" applies some rather naive assumptions.

I'm all for supporting DCA research, and, unlike some of the more paranoid commenters on this issue, I think that Pharma companies are taking notice. It's not unthinkable that some NIH oncology funding is in the pipeline for the compound, or some small pharma company will in-license the compound/formulation/use and perhaps even bring some of Big Pharma's research dollars in if the compound passes Phase II trials (Phase I safety testing should be a breeze relatively speaking, since it's already approved for some indications). Before we get up in arms about what is patentable and whether research dollars will be spent on a promising compound, realize that nearly everybody in this industry is for helping others, and we will find a way to get the most promising compounds studied and, if they work, on the market.

Tuesday, January 2, 2007

Confidence limits on NNTs - a guide to comparing NNTs

Previously in this series I discussed the definition of the NNT (i.e. when comparing therapy to placebo, it's 1/absolute risk reduction of therapy) and how to interpret it (it's the expected number of people that you would have to treat to prevent one unfavorable outcome). It's evil twin, the NNH, is similarly calculated and interpreted in association with an adverse event.

In my first entry on the subject, a commenter asked whether the NNT is a number or a statistic with an error. The answer is that it's a statistic with an error. Problem is, most people do not report the error (or confidence interval) along with the NNT. The error or confidence interval helps us answer questions such as "Drug A has NNT of 21 and Drug B has NNT of 22. Is there really a difference?"

To begin with, I'll assume that all needed data is available in this form:
  • Risk of unfavorable outcome for placebo group
  • Risk of unfavorable outcome for treatment group
  • Sample size
To calculate the NNT itself, just do 1/(risk in placebo-risk in treatment). The sample size is not needed. However, to calculate the error, the sample size is necessary.

Then, we calculate the error for (risk in placebo-risk in treatment) (i.e. 1/NNT). The expression is a little more complicated, but not hard to put into a spreadsheet or calculator:

std error = sqrt(risk placebo * (1 - risk placebo) / (# in placebo group) + risk treatment * (1 - risk treatment) / (# in treatment group)),

where sqrt means take the square root of the whole thing. A simple explanation goes as follows:

risk group A * (1 - risk group A) / (# in group A)

is the variance of the estimate in group A. Add the variances in the placebo and treatment groups to get the variance of the treatment, and then take the square root to get the error. So two principles: variances (often) add, and the error is the square root of the variance.

To get a 95% confidence interval of the risk reduction, you take the difference and add/subtract 2 times the error1.

Example. In the last entry I compared niacin and simvastatin. The article has some of the information we need:

DrugRisk placeboRisk treatmentSample size
Niacin36.5%31.5%1390
Simvastatin21.5%13.5%2221 (sim)
2223 (placebo)


I had to do some sleuthing to get the sample size numbers. For niacin a Google search for the Coronary Heart Disease project landed this draft of a report, from which I found a total sample size and divided by 6 (there were six groups). For the simvastatin number I used the Wikipedia entry on Scandinavian Simvastatin Survival Study to get the sample size. But anyway, we're able to do a confidence interval calculation. We start with niacin:
  • risk reduction = (36.5% - 31.5%) = 5% (so NNT = 1/5% = 20)
  • error = sqrt(0.365 * 0.635/1390 + 0.315*0.685/1390) = 0.018 = 1.8%
  • 95% confidence interval of risk reduction is 5%-2*1.8% to 5%+2*1.8% = 2.4% to 8.6%
The 95% confidence interval for risk reduction for simvastatin is 5.8% to 10.2%. (I got a risk reduction of 8% and standard error of 1.1%).

End example.

The simplest way to get a 95% confidence interval for the NNT is to just do 1/confidence limits. You will also have to invert the order of the limits. Granted, this isn't necessarily the best way, and I'll probably show how to do another way one day, but it's easy and are actual (approximate) 95% confidence limits. So the NNT limits for niacin are (1/8.6% to 1/2.4%) = (11.63 to 41.7). The NNT limits for simvastatin are (9.8 to 17.2).

From this quick and dirty calculation, it's not absolutely clear that niacin has higher efficacy than simvastatin. Part of the reason for this is the wide error in the risk reduction estimate for niacin, which comes from the fact that 31.5% of subjects in the niacin group of the study had a cardiovascular event.

A few other issues are worth point out here, and they cloud the issue even more. I took these numbers from two different studies: the 4S study and the Coronary Heart Disease project. If you look at these two studies, they have different inclusion criteria (e.g. the CHD project had an inclusion criterion of men only). Eventually, in trying to get the information we need, we come across such barriers. Preferably, the numbers I used above would have come from the same study, and given the differences between objectives and populations in the studies, the comparison between the simvastatin and niacin NNTs are not as straightforward as back-of-the-envelope calculations as given above can lead you to believe. It's important to keep the limitations of both the data and the statistics in mind.

1Technical details: this is an approximate interval, and those who have been through stats classes may prefer to use z0.025=1.965. I don't think it matters too much except in cases that accuracy is very important such as academic reports and regulatory submissions. Also, this confidence interval has fallen out of favor with statisticians, but is easy and useful for the kinds of back-of-the-envelope things I'm doing here. [back]

Monday, January 1, 2007

Not quite a repost - Number Needed to Treat (NNT)

Rather than repost this entry on the NNT, I thought I would discuss the issue a little further. For background, here are some references:
For the individual, the NNT doesn't really matter. After all, when you take a drug, it doesn't matter what happens to other people on the drug. It only matters what happens to you. However, for public policy makers and insurance companies, the NNT has become very important. The reasoning goes as follows:

Say you, as an insurance company, wanted to compare two therapies to prevent serious cardiovascular events (e.g. cardiovascular death, myocardial infarction): niacin and simvastatin. The cost of a cardiac event is high both economically (in terms of health care cost and days missed) and in pain and suffering. Then we can answer the following question: what is the cost of preventing one event using niacin and simvastatin?

You can organize the work as follows:
Therapy Source Duration (years) Cost/Day NNT Total Cost/1 prevention
Niacin Coronary Drug Project 6.2 $0.21 20 $9,511.11
Simvastatin 4S 5.4 $0.93 13 $23,845.71

The only calculated column is the last one (you can easily set this kind of table up in any spreadsheet). The calculation is total=duration*365.25*cost/day*NNT. In addition, I used a favorable cost for simvastatin (although it will get more favorable when more generics hit the market) and an unfavorable cost for niacin. Source of data is Tables 1 and 2 from Therapeutics Letter, May 1998 as shown here. Note that cerivastatin has been withdrawn since then, and a generic form of simvastatin has hit the market.

You can interpret the last column as follows: to prevent one cardiovascular event, you expect to pay $9,511.11 for niacin therapy or $23,845.71 for simvastatin therapy.

Simvastatin actually doesn't compare too unfavorably with niacin therapy. Other measures such as safety profile (including NNH - Number Needed to Harm - for the more serious adverse events) are needed in the decision making process, but under this measure we should not rule out simvastatin as a valid and effective therapy. The main issue is unit cost, something that will change as generics come on the market or something that can be negotiated down (especially in the case of transporting drugs to developing countries).

Speaking of safety profile, that is something that hasn't been figured into the table above. The costs associated with flushing, gastrointestinal problems, skin problems, and acute gout, -- all associated with niacin -- along with the NNH for these issues, need to be addressed. On the simvastatin side, creatine kinase elevation with muscle weakness and rhabdomyolysis need to be addressed, along with the other adverse events that have been found to be associated to statin therapy in the last couple of years. These expected costs are to be added to the costs in the table above.

There are a few caveats to the NNT a couple of which I mention below:
  • First, the NNT is a number derived from an estimate, i.e. 1/absolute risk reduction. Though most estimates are reported with a standard error, NNTs are not (and this is a flaw). Likewise, the costs above have a range deriving from several sources: range of cost, range of NNT, range of durations studied.
  • The NNT is based on population-based statistics. For an individual making an individual decision about healthcare, it carries less weight than it would for an insurance company deciding which therapies to cover or a healthcare NGO deciding which therapies to pay for transport into developing countries. Side effect risk factors, metabolism profile, and other individual factors carry more weight (and those carry less, but higher than zero, weight with policy makers).
So next on my plate in the NNT series is how to compute a standard error from measures that you see in the literature.

Saturday, December 16, 2006

Adaptive trials page is up

The presentations given at the FDA/PhRMA conference on adaptive designs in November is up. A lot of interest in this workshop was stimulated when Scott Gottlieb mentioned it back in July during a keynote at another conference. Unfortunately, registration seemed limited to PhRMA members.



The content looks pretty good. I'll have more to say on it as time goes by.

Wednesday, December 13, 2006

CETP inhibitors don't seem to be all bad

Merck's CETP inhibitor MK-859, which is in the same family as torcetrapib which Pfizer discontinued after an excess of cardiovascular problems and deaths, does not seem to cause heart attacks and strokes. So, apparently torcetrapib had some other mechanism than CETP inhibition which caused the problems, or perhaps Merck's version has some other cardioprotective side effect.

(via Pharmagossip)

Realizations

This blog grew out of a need to separate my professional blogging from personal blogging. With the public trust in clinical trials waning in the wake of Vioxx and Ketek, there is a need for public information and transparency on how clinical trials are conducted. While more information won't solve all of the problems -- this requires a commitment on the part of drug and research companies as well -- I do find that a lot of mistrust stems from a simple misunderstanding of the scientific process in general and how drugs are studied in particular.

Most other blogging on this subject seems to come from doctors. I've found very few statistical bloggers, and I'm the only person I know of that is blogging with a purely biostatistical focus. Yet it is the biostatistician who has to determine whether a clinical trial can give a statistically valid answer, and the doctor or pharmacologist who decides if the statistically valid answer has any meaning or relevance. This blog is about clinical trials and other research, and how to extract conclusions from them.

The material in this blog will draw from the news, my own personal experience, and even a bit of research, and is intended for a general and professional audience.

As for the title: in statistics a "realization" is one instance of data. We dream up these statistical models, use mathematics to say what we can about them without looking at real data, and then examine realizations to see how well our models and methods work in practice. The data that comes out of a clinical trial can be said to be a realization.