On Avandia: The difference between statistically significant and not statistically significant ...

... is not statistically significant.

I'm placing less and less trust in p-values. Perhaps the Bayesians are getting to me. But hear me out.

In the Avandia saga, GSK got their drug approved on two adequate and well-controlled trials (and have studied their drug even more!). There was some concern over cardiovascular risks (including heart attack), but apparently the risk did not outweigh the benefits. Steve Nissen performs a meta-analysis on GSK's data from 42 (!) randomized control trials, and now the lawyers are lining up, and the FDA's favorite congressmen are keeping the fax lines busy with request letters and investigations.

Here's how the statistics is shaking out: the results from the meta-analysis shows a 43% increase in relative risk of myocardial infarction, with a p-value of 0.03. The (unspecified) increase in deaths didn't reach statistical significance with a p-value of 0.06.

Argh. Seriously, argh. Does this mean that the relative risk of myocardial infarction is "real" but the increase in deaths is "not real"? Does the 43% increase in relative risk even mean anything? (C'mon people, show the absolute risk increase as well!)

According to the Mayo clinic, the risk is 1/1000 (Avandia) vs. 1/1300 (other medications) in the diabetic study populations. That works out to a 30% increase in relative risk, not the same as what MedLineToday reported. The FDA's safety alert isn't very informative, either.

Fortunately, the NEJM article is public, so you can get your fill of statistics there. So, let me reference Table 4. My question: was the cardiovascular risk real in all studies combined (p=0.03), but not in DREAM (p=0.22), ADOPT (p=0.27), or all small trials combined (p=0.15)? That seems to be a pretty bizarre statement to make, and is probably why the European agencies, the FDA, and Prof. John Buse of UNC-Chapel Hill (who warned the FDA of cardiovascular risks in 2000) have urged patients not to switch right away.

The fact of the matter is if you look for something hard enough, you will find it. It apparently took 42 clinical trials, 2 of them very large, to find a significant p-value. Results from such a meta-analysis on the benefits of a drug probably wouldn't be taken as seriously.

Let me say this: the cardiovascular risks may be real. Steve Nissen's and John Buse's words on the matter are not to be taken lightly. But I think we need to slow down and not get too excited over a p-value that's less than 0.05. This needs a little more thought, not just because I'm questioning whether the statistical significance of the MI analysis means anything, but also because I'm questioning whether then non-significance of the mortality analysis means the death rates aren't different.

Update: Let me add one more thing to this post. The FDA realizes that p-values don't tell the whole story. They have statistical reviewers, medical reviewers, pharmacokinetic reviewers, and so forth. They look at the whole package, including the p-values, medical mechanism of action, how the drug moves through the body, and anything else that might affect how the drug changes the body. Likewise, Nissen and companies discusses the medical aspects of this drug, and doesn't let the p-values tell the whole story. This class of compounds -- the -glitazones (also known as PPAR agonists) -- are particularly troublesome for reasons described in the NEJM article. So, again, don't get too excited about p-values.

A plunge into the dark side

I'm referring, of course, to Bayesian statistics. My statistical education is grounded firmly in frequentist inference, though we did cover some Bayesian topics in the advanced doctorate classes. I even gave one talk on empirical Bayes. However, in the last 8 or so years, all that knowledge (such as it was) was covered over.

No more. I've had as a goal to get my feet wet again, because I knew some time or another I would have to deal with it. Well, that some time or another is now, and it probably won't be another eight years after this time before I have to do it again. So off to a short course I go, and equipped with books by Gelman, et al. and Gamerman, I'll be a fully functional Bayesian imposter in no time. I'm looking forward to it.

A tale of two endpoints

Some time ago, when Gardasil was still in clinical trials, I congratulated the Merck team for a product with 100% efficacy. After all, getting anything with 100% efficacy is a rare event, especially in drug/biologic development.

Apparently, that congratulations was a little too soon. Looks like Merck may have found a surrogate endpoint that their vaccine managed very well, but if you look at the important endpoint, the story doesn't look quite so rosy.

So, to be specific, Gardasil is marketed to protect against two strains of human pampilloma virus (HPV) that account for 70% of cervical cancer cases. (Types 16 and 18, for those keeping track.) Merck is going for 80% now by asking the FDA to add types 6 and 11 to the label.

Ed from Pharmalot notes that in clinical trials, among women that already have HPV, the vaccine reduces precancerous lesions (no time limit given) by 14%. For women that don't have HPV, the occurrence of precancerous lesions is reduced 46%. Presumably this is because the vaccine is ineffective against strains that already infect the body. Merck's spin engine is carefully worded to tout that 70%, even though that number is only of secondary importance. It's the 14% and 46% that really matter.

Addendum: I looked at the Gardasil PI, and they already mention 6 and 11. They also mention all other sorts of efficacy measures. The patient product information is less informative. My guess is Merck is overplaying the efficacy in their soundbites by shoving that 70% front and center, but its detractors are overplaying the gap between the 70% and the real story by shoving the 14% front and center.

I'm glad I'm a biostatistician, else I wouldn't be able to understand all this jockeying the numbers.

Simple, but so complex

So, in addition to statistics, I've been dabbling a little in fractal/chaos theory. Nothing serious, but enough to know that behind even the simplest functions there lies an amazing complex landscape. Who knew that z²+c could be so rich?

At any rate, I did all this stuff back in college, but in specializing I've forgotten most of it (except for the occasional admonition that it's often easy to confuse the complexity of dynamic systems for noise of a stochastic [random] system).

As I get older, it's become easier to lose the wonder. However, beneath every simple surface could be a world of complexity that will inspire a new round of curiosity.

Regulatory fallout from Tegenero's ill-fated TGN1412 trial

While biostatistics does not get used very much in early human clinical trials, any regulatory changes can have an effect on the practice. The EMEA has published new guidelines (pdf - in draft form, to be finalized after public comment and consultation with industry) about the conduct of Phase I trials for "high-risk" compounds. This comes in the wake of the infamous TGN1412 trial, in which a monoclonal antibody caused severe adverse reactions in all of the six otherwise healthy trial participants. (All six participants suffered multiple organ failure, along with gangrene. They will all probably contract and die of cancer within a few short years.)

The EMEA concluded that the trial was conducted in accordance with current regulations. These new recommendations are changes to avoid another similar disaster.

Among the recommendations:
- stronger pre-clinical data, and a stronger association between pre-clinical data and choice of dosing in humans (e.g. using minimal dose for biological activity), as opposed to the no observed adverse-event dose
- the use of independent data safety monitoring boards, along with well-defined stopping rules for subjects, cohorts, and trials
- well-defined provisions for dose-escalation
- increasing follow-up length for safety monitoring
- use of sites with appropriate medical facilities

(via Thomson Centerwatch)

A final word on Number Needed to Treat

In my previous post in this series I discussed how to create confidence intervals for the Number Needed to Treat (NNT). I just left it as taking the reciprocal of the confidence limits of the absolute risk reduction. I tried to find a better way, but I suppose there's a reason that we have a rather unsatisfactory method as a standard practice. The delta method doesn't work very well, and I suppose methods based on higher-order Taylor series will not work much better.

So, what happens if the treatment has no statistically significant effect (sample size is too small or the treatment simply doesn't work). The confidence interval for absolute risk reduction will cover 0, say, maybe -2.5% to 5%. Taking reciprocals, you get an apparent NNT confidence interval of -40 to 20. A negative NNT is easy enough to interpret: -40 NNT means that for every 40 people you "treat" with the failed treatment, you get a reduction of 1 in favorable outcomes. A 0 absolute risk reduction results in NNT=∞. So if the confidence interval of absolute risk reduction covers 0, the confidence interval must cover ∞. In fact, in the example above, we get the bizarre confidence set of -∞ to -40 and 20 to ∞, NOT -40 to 20. The interpretation of this confidence set (it's no longer an interval) is that either you have to treat at least 20 people but probably a lot more to help one, or if you treat 40 or more people then you might harm one. For this reason, for a treatment that doesn't reach statistical significance (i.e. whose absolute risk reduction includes 0), the NNT is often reported as a point estimate. I would argue that such a point estimate is meaningless. In fact, if it were left up to me, I would not report an NNT for a treatment that doesn't reach statistical significance, because the interpretation of statistical non-significance is that you can't prove with the data you have that the treatment helps anybody.

Douglas Altman, heavy hitter in medical statistics, has the gory details.

Technorati Tags: number needed to treat, NNT

SAS weirdness

From time to time, I'll complain about the weirdness of SAS, the statistical analysis program of choice for much of the pharmaceutical industry. This post is one such complaint.

Why, oh why, does SAS not directly give us the asymptotic variance of the Mantel-Haenszel odds ratio estimate? It does, however, give the confidence interval. Though the default is a 95% confidence interval, by specifying alpha=31.4 in the TABLES statement in the FREQ procedure and using ODS output to get these values into a dataset, you can compute the asymptotic variance by either dividing the upper confidence limit by the Mantel-Haenszel odds ratio estimate, or dividing the MH estimate by the lower confidence limit (both should give the same answer). The point is, SAS has to compute the asymptotic variance to calculate the confidence interval, so why not just go ahead and display it? (Yes, I understand that the confidence interval is symmetric only on a log scale.)

Addendum: R doesn't either. Same story. Weird.

Alternative medicine use might be affecting the results of trials

From here.



At issue is whether remedy-drug interactions are skewing the results of Phase I cancer trials. At present, this is hard to determine because it is hard to elicit (alternative) remedy use. To me, it's pretty clear that having remedy use out in the open is better than being secretive. However, what's interesting to me is the following, surveyed from 212 patients with advanced cancer enrolled in Phase I clinical trials:

• 72 (34 percent) use alternative remedies, similar to general US population usage
• 41 (19.3 percent) take vitamins and minerals
• 40 (18.9 percent) take herbal preparations

In addition, we have the following:

Sometimes, patients are reluctant to tell the doctor they are taking
alternative medicines, either because they don't think it's important,
or they don't want to be told to stop taking them, Daugherty said.

Also,

And, since it's often difficult to get cancer patients to take part in
phase 1 trials, some researchers may be reluctant to turn any potential
patient away. "In addition, most doctors don't know very much about
alternative medicine," Daugherty said.

For research, I think these matters need to be out in the open. For one thing, we need to understand our drugs we are developing. For another, we need to understand the alternative remedies.





Technorati Tags: cancer, alternative medicine

Big news that's easy to miss -- FDA clears a molecular prognostic tool for breast cancer metastasis

The FDA just approved a prognostic device. So what's the big deal?

I'll let the press release speak:

It is the first cleared molecular test that profiles genetic activity.

That's right. Despite many years (ok, about a decade to a decade and a half) of use in basic research, microarray technology has matured (along with the analysis methodologies) enough to be used in clinical practice, and this approval marks a big step toward that. Microarrays were a buzz in the statistical community a few years ago when there were still some methodological hurdles to overcome (and were being rapidly overcome).

What's more, this marks the first time a genetic expression test (different from a genetic test -- this one identifies which genes are expressed [active] at a particular time) has been approved for the prognosis of a disease. 70 gene expressions are analyzed. Agendia has blazed some trails, and I expect to see more of this kind of test in the coming years. And that's a good thing.

And, hopefully for women with breast cancer, this will help a bit in the decision making for treatment.

Dichloroacetate (DCA) - not an easy road to "cheap, patentless cancer cure"

Posts that contain Dca per day for the last 30 days.

Get your own chart!

Ginger, curcumin, and DCA have recently been touted for their anticancer properties, and, granted, in the Petri dish, they look pretty good. However, it's a long road from the Petri dish to the pharmacy shelves. Abel Pharmboy's coverage of DCA seems to be the most spot-on, and his points are worth repeating. Many compounds show promise in the Petri dish and animal models, but when it comes to human trials, they bomb. It is entirely possible, in fact, from drug development experience I would say very likely, that DCA will do very little for humans in trials. It may be ineffective when we actually inject it into human beings (for anticancer purposes; it's already approved for some metabolic disorders).

Remember, cancer is a complex disease. To "cure cancer" is really to cure a whole lot of different diseases, which is why our "war on cancer" applies some rather naive assumptions.

I'm all for supporting DCA research, and, unlike some of the more paranoid commenters on this issue, I think that Pharma companies are taking notice. It's not unthinkable that some NIH oncology funding is in the pipeline for the compound, or some small pharma company will in-license the compound/formulation/use and perhaps even bring some of Big Pharma's research dollars in if the compound passes Phase II trials (Phase I safety testing should be a breeze relatively speaking, since it's already approved for some indications). Before we get up in arms about what is patentable and whether research dollars will be spent on a promising compound, realize that nearly everybody in this industry is for helping others, and we will find a way to get the most promising compounds studied and, if they work, on the market.