Clinical trials: enrollment targets vs. valid hypothesis testing

The questions raised in this Scientific American article ought to concern all of us, and I want to take some of these questions further. But let me first explain the problem.

Clinical trials and observational studies of drugs, biologics, and medical devices are a huge logistical challenge, not the least of which is finding physicians and patients to participate. The thesis of the article is that the classical methods of finding participants – mostly compensation – lead to perverse incentives to lie about one’s medical condition.

I think there is a more subtle issue, and it struck me when one of our clinical people expressed a desire not to put enrollment caps on large hospitals for the sake of a fast enrollment. In our race to finish the trial and collect data, we are biasing our studies toward larger centers where there may be better care. This effect is exactly the opposite of that posited in the article, where treatment effect is biased downward. Here, treatment effect is biased upward, with doctors more familiar with best delivery practices (many of the drugs I study are IV or hospital-based), best treatment practices, and more efficient care.

We statisticians can start to characterize the problem by looking at treatment effect by different sites, or using hierarchical models to separate out center effect from drug. But this isn’t always a great solution, because low-enrolling sites, by definition, have a lot fewer people, and pooling is problematic because low-enrolling centers tend to have way more variation in level and quality of care than high-enrolling centers.

We can get creative on the statistical analysis end of studies, but I think the best solution is going to involve stepping back at the clinical trial logistics planning stage and recasting the recruitment problem in terms of a generalizability/speed tradeoff.

How to waste millions of dollars on clinical trials

When I started working for my current company (a clinical research organization), I was presented with a project that had discontinued due to lack of continued funding. The sponsor simply wanted to stop enrolling and get any conclusions from the study they could find. We presented some basic summary statistics, and then they presented me with an ethical dilemma: they found a couple of numbers that "looked good" and wanted me to generate a p-value that they could put into a press release so that they could get more funding. In statistics this is known as data dredging or p-value hunting. P-values generated this way are known to be extremely biased in favor of the drug being studied simply because the brain is very good at picking out patterns. Recently, however, I read that this company had a failed Phase 3 trial, which cost millions of dollars.

In a previous life, I worked with someone whose large Phase 2 trial failed on its primary endpoint. However, a secondary endpoint looked very good. They commissioned a Phase 3 study with thousands of patients to study and hopefully confirm the new endpoint. However, that study ended up failing as well, and I believe development of the drug was discontinued.

In my opinion, those failed studies could have been avoided. A Phase 2 study need not reach statistical significance (and certainly should not be designed so that it has to), but results in Phase 2 should be robust enough and strong enough to inspire confidence going into Phase 3. For example, estimated treatment effect should be clinical relevant, even if confidence intervals are wide enough to extend to 0. Related secondary endpoints should show similar trends. Relevant subgroups should show similar effects, and different clinical sites should show a solid trend.

I personally would prefer Bayesian methods which can quantify these concepts I just listed, and can even give a probability of success in a Phase 3 trial (with given enrollment) based on the treatment effect and variation present in the Phase 2 trial. However, these methods aren't necessary to apply the concepts above.

In both of the cases I listed above, the causes were extremely worthy, and products that are able to accomplish what the sponsors wanted would have been useful additions to medical practice. However, these products are probably now on the shelves, millions of dollars too late. The end of Phase 2 can be a very difficult soul-searching time, especially when a Phase 2 trial gives equivocal or negative results. It's better to shelve the compound or even run further small proof-of-concept studies than waste such large sums of money on failed large trials.

Is it just me, or is Eli Lilly asking for a torcetrapib-like epic fail?

From Lilly's press release:

INDIANAPOLIS, May 21, 2009 /PRNewswire-FirstCall via COMTEX News Network/ -- Eli Lilly and Company (NYSE: LLY) today announced it will begin enrolling patients this month in two separate but identical Phase III clinical trials of solanezumab(i), previously referred to as LY2062430, an anti-amyloid beta monoclonal antibody being investigated as a potential treatment to delay the progression of mild to moderate Alzheimer's disease. The trials, called EXPEDITION and EXPEDITION 2, will each include a treatment period that lasts 18 months and are expected to enroll a total of 2,000 patients age 55 and over from 16 countries.

Ok, fine. I'm sure the Lilly solanezumab team is excited and nervous, as this is their moment in the spotlight. But then later on:

Alzheimer's disease theory suggests that amyloid beta clumps together and eventually kills brain cells. Solanezumab binds specifically to soluble amyloid beta and thereby may draw the peptide away from the brain through the blood. In short-term clinical studies, solanezumab appeared to have dose-dependent effects on amyloid beta in blood and cerebrospinal fluid. The clinical studies to date have been too short to evaluate any potential delay in the progress of Alzheimer's disease.

Ok, hit the brakes. I am acquainted with some of the statisticians over at Lilly, and they are very bright people who have contributed a lot to the field. I can't help but think they are more nervous than the people who made the decision to run with a long-term endpoint in Phase 3 based on a short-term endpoint in Phase 2. I've known a couple of epic fails falling into the same category. Throw on top of that that the theory behind it, while still king of the hill, is starting to be thrown into doubt, and Alzheimer's is a graveyard anyway. (Just read this series to see the associated problems.)

It may be standard practice in some circles to run a large trial on a long term endpoint based on a shorter term endpoint or a biomarker (or even 2), but, while I hope this trial succeeds (we definitely need an effective Alzheimer's treatment), I can't put my money on it at this point. I wish them luck.