Tuesday, April 20, 2010

How to waste millions of dollars with clinical trials, Part II: Rexahn

Lots of people have been calling BS on Rexahn's press release about its Phase 2a data on Serdaxin and subsequent "additional statements." Read the articles below; they offer good background and analysis.

Here, I think Rexahn really did it to themselves, but I also think the environment around drug development is partly to blame. Everybody loves the p-value less than 0.05 [to Adam Fueurstein's credit in the first reference below, he blows off the "not statistically significant" issue], and companies are ready to comply by cherry-picking the best-looking p-value to present. (I know: I've been asked to do this cherry picking.)

Why shouldn't we care about statistically significant? Simply because it's a Phase 2a study. This is the place where we don't know anything about a drug's efficacy, and we're starting to learn. We don't know how to power a study to reach statistical significance, simply because we don't know the best endpoint, the best dosing schedule, the best dose, or really anything except what is observed in preclinical. And we know that the drug development graveyard is littered with drugs that did well in preclinical and bombed in the clinic. So how can we expect to know how many subjects to enroll to show efficacy? We could also use some nouveau Bayesian adaptive design (and I could probably design one for most circumstances), but tweaking more than two or three things in the context of one study is a recipe for disaster.

Here's what I would prescribe (while ignoring the needs of press release consumers):
1. Forget about statistically significant. Whatever calculations are made for power analysis or number of subjects are usually a joke, anyway. The real reason for a sample size usually has to do with budget, and the desire to collect at least the minimum amount of information needed to design a decent Phase 2 trial. If there is a stated power calculation, it has usually been reverse engineered. Instead, it is sometimes possible to do sample size calculations based on size of confidence intervals (to reflect the certainty of an estimate) of different endpoints (see #2).
2. Forget about a primary endpoint. There is no need. Instead, use several clinically relevant instruments, and pick one after the study that is the best from a combination of resolution (i.e. ability to showcase treatment effect) and clinical relevance.
3. Set some criteria for "success," i.e. decision criteria for further development of the drug, that does not include statistical significance. This might include an apparent dose effect (say, 80% confidence interval around a parameter in a dose-response design that shows positive dose-related effect), tolerability at a reasonably high dose, or, if you implement a Bayesian design (adaptive or otherwise), a given probability (say 80%) of a successful adequate and well-controlled trial of reasonable size with clinically relevant criteria. What these criteria are of course needs to be carefully considered -- they have to be reasonable enough to be believable to investors and scientifically sound enough to warrant further treatment of patients with your unproven drug, and yet not so ridiculously high that your press release is almost definitely going to kill your ability to woo investors.
4. Be transparent and forthcoming in your press release, because enough data is out there and there are enough savvy investors and bloggers who can call BS on shenanigans.

Also see:

Friday, April 16, 2010

R - not the epic fail we thought

I usually like AnnMaria's witty insight. I can relate to a lot of what she is saying. After all SAS and family life are large parts of my life, too. But you can imagine the reaction she provoked in saying the following:

I know that R is free and I am actually a Unix fan and think Open Source software is a great idea. However, for me personally and for most users, both individual and organizational, the much greater cost of software is the time it takes to install it, maintain it, learn it and document it. On that, R is an epic fail. 
 With the exception of the last sentence, I am in full agreement. Especially in the industry I work in, qualification and documentation is hugely important, and a strength of SAS is a gigantic support department who has worked through these issues. Having some maverick install and use R, as I do, simply does not work for the more formal work that we do. (I use R for consulting and other things that do not necessarily fulfill a predicate rule.)

However, another company, REvolution Computing, has recognized this need as well. With R catching on at the FDA, larger companies in our industry have taken a huge interest in R, partly because of the flexibility in statistical calculations (and, frankly, the S language beats SAS/IML hands down for the fancier methods), and mostly to save millions of dollars in license fees. Compare REvolution's few hundred dollars a year for install and operation qualification on an infinite-seat license to SAS's license model, and it's not hard to see how.

And SAS, to their credit, has made it easier to interact with R.

Epic win for everybody.

Friday, April 2, 2010

FDA guidance on adaptive trials

The FDA guidance on adaptive trials (group sequential designs, sample size re-estimation, and so forth) can be found here. I've only skimmed through it but it looks fairly informative. So far so good. Hopefully this will be a strong document that will lead to an increase in these kinds of studies.

I'll discuss more after a more thorough look.

Adventures in graduate school

I was recently reflecting at, basic classes aside, I use information from mainly three graduate classes. Two of them were special topics classes, and one was a class that had finally evolved from a special topics class.

In one of the special topics class, we were given a choice of two topics: one field survey of Gaussian processes, which would have been useful but that was not so interesting to the professor, and local time (i.e. the amount of time a continuous process spends in the neighborhood of a point), which was much more specialized (and for which I did not meet the prerequisites) and much more interesting to the professor. I chose the local time because I figured if the professor was excited about it, I would be excited enough to learn what I needed to to understand the class. As a result, I have a much deeper understanding of time series and stochastic processes in general.

The second special topics class seemed to have a very specialized focus, pattern recognition. It covered the abstract Vapnik-Chervonenkis theory in detail, and we discussed rates of convergence, exponential inequalities on probabilities, and other hard-core theory. I could have easily forgotten that class, but the professor was excited about it, and because of it I am having a much easier time understanding data mining methods than I would have otherwise.

The third class, though it was not labeled a special topics class, was a statistical computing class where the professor shared his new research in addition to the basics. There I learned a lot about scatterplot smoothing, Fourier analysis, local polynomial and other nonparametric regression methods that I still use very often.

In each of these cases, I decided to forgo a basic or survey class for a special topics class. Because of the professor's enthusiasm toward the subject in each case, I was willing to go the extra mile and learn whatever prerequisite information I needed to understand the class. In each case as well, that willingness to go the extra mile and fill in the gaps has carried over to over a decade later where I have kept up my interest and am always looking to apply these methods to new cases, when appropriate.

I am currently taking the bootstrapping course at statistics.com and am happy to say that I am experiencing the same thing. (I was introduced to the bootstrap in fact in my computing class mentioned above but we never got beyond the basics due to time.) We are getting the basics and current research, and I'm already able to apply it to problems I have now.