Reproducible research in the drug development industry

1 What is reproducible research?

Reproducible research, in a nutshell, is the process of publishing
research in such a way that a person can pick up the materials and
reproduce the research exactly. This is an ideal in
science. Essentially, all data, programming code, and interpretation
is presented in such a way that it is easy to see what was done, how
it was done, and why.

A report written in reproducible research style is written in such a
way that any result that comes from analyzing data is written in some
programming language inside the report. The written report is then
processed by software that will interpret the programming code and
replace it with both the code and the output from the code. The reader
of the report then sees exactly what code is executed to produce the
results, and the results that are shown in the report are guaranteed
to be from the code that is shown. This is different from, for
example, writing the code in the document and running it separately to
generate results which are copied and pasted back into the report. In
essence, the report and the analysis are done together, at the same
time, as a unit. An demo of how this works using the LaTeX, Sweave,
and R packages can be found here, and another example using R and
LaTeX, but not Sweave, can be found at Frank Harrell's rreport page.

Further information can be found at some of the links below (and the
links from those pages).

When should you write your statistical analysis plans?

Up over at Ask Cato.

Analysis of the new draft guidance of adaptive trials

Now up over at Ask-cato.com.

Perverse incentives in clinical trials

Decisions whether to progress to Phase 3 are not always based on the effiacy and safety of a drug or the feedback on the drug from Phase 2. Derek Lowe notes that because Phase 3 is a milestone for smaller companies and their valuations is dependent on how advanced their products are.

A common mistake I see is that a Phase 2 study will fail soundly, but the sponsor still runs a Phase 3 trial without further tests. To "justify" these studies, a case will often be made that a secondary endpoint is "trending" and, if enough subjects are enrolled, it will be significant. Of course, most such Phase 3 trials will fail. Unfortunately, this same strategy carries forth into the investment arena with press releases chock full of meaningless p-values. P-values that savvy investors are double-checking.

We have statistical techniques available or under development that will aid in Phase 3 go/no-go decision-making without requiring statistical significance or baseless guesswork. We can compute a probability of Phase 3 success, with an estimate of uncertainty of that probability, using techniques we have now. I guess too many small companies want to go to Phase 3 at any cost rather than make a cool-headed decision. Maybe investors should start requesting a failure plan for drugs under development in which they invest to reduce this perverse incentive.