Trials with bolted on adaptive components

All too often, I get a request to make a trial adaptive. In a lot of cases, adaptations were considered but rejected, but the sample size was too large given considerations such as dropout. Of course, this is a delicate time in sponsor-CRO relations, because emotions are already running high due to the frustration in spending the time considering a lot of alternatives that are already rejected. There is further danger in that the sponsor is, in fact, asking for a fundamental change to a trial that has already been designed.




Adaptive trials are best designed with the adaptation already in mind. This is because the adaptive component affects many aspects of the trial. In addition, the additional planning required for an additive trial can be more easily done if it is worked in from the beginning.

In the case where adaptation is used to rescue a trial, it's probably best to take the time to effectively start from the beginning, at least in making sure the time and events table makes sense. Barring that, I will often recommend one futility analysis be performed. The reason I do this are as follows:

* no adjustment to stated Type 1 error rate is required

* it's relatively easy to "bolt on" to an existing trial

* under the most common circumstances under which this late consideration is done (late Phase 1 or Phase 2 trial) this strategy will prevent wasting too much money on a worthless compound

Of course, not all trials benefit from a futility analysis, but I recommend this strategy almost categorically in cases where a sponsor wants to add one interim analysis to an otherwise designed trial.

Posted via Blogaway

Future of clinical trials recap

Clinical trials are complex, so any meeting about the future of trials is going to be complex. Indeed, the  Future of Clinical Trials meeting had something from many perspectives from recruitment to ethics to statistics. Of course, I viewed most of the presentations with an eye for how to apply them to adaptive trials. So, here's the themes of what I heard in the presentations:

• Relationships are going to be the most important key to the success of any clinical trial. Pharma companies are starting to outsource in such a way that they expect a strategic partner-level participation by the vendor (such as a clinical research organization-CRO), and the CRO had best bring its A-game regarding project management, design and execution of trials.
• I had not thought about this particular area, but business development is going to play a key role as well. We discussed several aspects, but one that sticks in my mind is structuring contracts in such a way to minimize change orders. I think this will be helpful because change orders take precious time away from the team and make the relationship more difficult to maintain.
• Regulatory uncertainty drives us to be more efficient, but we are also uncertain about the changes that are required to make us more efficient. We can expect the difficult regulatory environment to get worse before it gets better because of the recent politicization of drug safety.
• I think a new wave of technologies is going to make designing and running trials more efficient. Improvements are being made to study startup, clinical trial management, patient recruitment, site selection, and ethics approval of protocols. It may take a while, but any company wanting to stay competitive will need to either employ some of these technologies or use something else to make up the lag in efficiency.

This is only a small overview. I think we will be hearing a lot more about these issues in the years to come.

Adaptive trials can be hard on the team

Clinical trials are hard enough to do as it is, because many people coming from many different backgrounds and having many different focuses have to coordinate their efforts to make a good quality finished product--a clinical trial with good data that answers the research questions in a persuasive and scientifically valid way. Add to that mix several interim analyses with tight turnaround times (required to make the interim analysis results useful enough to adapt the trial) and you really are putting your sites, clinical, data management, and statistical teams in the pressure cooker. Making stupid mistakes that your teams would not ordinarily make is a real danger (believe me, it is and I have made a few of those myself), and one that can endanger the results of the interim analysis. Here are some ideas to cut down on those stupid mistakes:

• Overplan during study startup.
• Get the whole trial execution team, including data management and stats, together around the table in the beginning.
• Do a dry run of the interim analysis, with everybody around the table. Personally, I think it's worth it to fly people in if they are scattered around the world, but at the very least use the web conferencing technologies.
• Draw a diagram of data flow for the interim analysis. Use Visio, a white board, note cards and string, or whatever is useful. The process of making this diagram is more important than the diagram itself, but the diagram is important as well. Of course, this process will more than likely change during the course of the study but these diagrams can be updated as well.
• Fuss over details. Little details can trip up the team when the chips are down. Make the process as idiot-proof as possible. I once had a situation where I screwed up an interim analysis because I forgot to change the randomization directory from a dummy randomization (so blinded programmers to write programs) to the real randomization (so I could produce the reports). After that, I talked with the lead programmer and refined the report production process even further.
• Plan for turnover. You like members of your team, and some of them will go away during the execution of the trial. New members will come on board. Business continuity planning is very important and is increasingly being scrutinized. Scrutinize it on your trials. Because you've overplanned, done some dry runs, drawn diagrams, and fussed over details, you've written all these down, so the content's readily available to put together in a binder (or pdf). You might even repeat the dry run process with new staff.
• And, for the statisticians, run clinical trial simulations. A well-done simulation will not only show how the trial performs, but also illuminate the assumptions behind the trial. Then simulations can be performed to show how robust the trial is regarding those assumptions.

Running adaptive trials is hard, but a thoughtful process and a prepared staff will help you realize the potential gains that adaptive trials can bring.

Great things coming up

In just a couple of weeks, I'll be giving my talk at the  Future of Clinical Trials  conference. For the next few weeks, I'll be posting material here and at Ask Cato about the best ways to negotiate with the FDA, design, and execute adaptive clinical trials so they can reach their potential.

Five keys to successful adaptive trials

Up over at Ask Cato. Enjoy.

New edition of Jennison and Turnbull

I just found out that there will probably be a second edition to Jennison and Turnbull's book on group sequential designs (with a slightly different title) coming out next year. Rock on.

Lan-DeMets in R is easier

I found this little gem today. Makes simulating group sequential designs a lot easier, since I don't have to run ld98 every time or something silly like that.

In other news, I got some cool toys from Google at JSM.

My O'Brien-Fleming design is not the same as your O'Brien-Fleming design

I know this discussion is a little technical, and nonstatisticians can probably skip this, but I hope that a statistician struggling with the O'Brien-Fleming design and its implementation in SAS/IML (notably the SEQ, SEQSHIFT, and SEQSCALE functions) can find this from a search engine and save hours of headache.

There are two ways of designing an O'Brien-Fleming design, a popular design for conducting interim analyses of clinical trials. The first method is to use an error (or alpha) spending function, which essentially gives you a "budget" of error you can spend at each interim analysis. The second is to realize that, if you are looking at cumulative sums in the trial, the O'Brien-Fleming design terminates if you cross a constant threshhold. In the popular design programs LD98 and PASS 2007, the spending function approach is used. In the book Analysis of Clinical Trials using SAS, (a book I highly recommend, by the way), the cumulative sum approach is used at the design stage (the spending function is used at the monitoring stage). When interim analyses are equally spaced, the two approaches give the same answer. When interim analyses are not equally spaced, the two approaches seem to give different answers. What's more, the spending function for O'Brien-Fleming as implemented in LD98 and PASS are different from what they show you in the books. They use:

4 - 4*PHI(z(1-alpha/4)/sqrt(tau))

for two-sided designs.

They don't tell you these things in school. Or in the books.

Update: Steve Simon's post on the topic has moved as of 11/21/2008. Please see the third comment below.

Bias in group sequential designs - site effect and Cochran-Mantel-Hanszel odds ratio

It is well known that estimating treatment effects from a group sequential design results in a bias. When you use the Cochran-Mantel-Haenszel statistic to estimate an odds ratio, the number of patients within each site affects the bias in the estimate of the odds ratio. I've presented the results of a simulation study, where I created a hypothetical trial and then resampled from this trial 1000 times. I calculated the approximate bias in the log odds ratio (i.e. log of the CMH odds ratio estimate) and plotted that versus the estimated log odds ratio. The line is cubic smoothing spline, made by the statement symbol i=sm75ps in SAS. The actual values are underprinted in light gray circles just to get some idea of the variability.

A good joint statistical meetings week

While I was not particularly enthralled with the location, I found this years Joint Statistical Meetings to be very good. By about 5 pm yesterday, I thought it was going to be so-so. There were good presentations on adaptive trials and Bayesian clinical trials, and even a few possible answers to some serious concerns I have about noninferiority trials. Last night I went to the biopharmaceutical section business meeting, and struck up conversations with a few people from the industry and the FDA (including the speaker who had some ideas on how to improve noninferiority trials). And shy, bashful me who had to drink 3 glasses of wine a couple of years ago to get up the courage to approach a (granted rather famous) colleague was one of the last ones to leave the mixer.

This morning, I was still feeling a little burned out, but decided to drag myself to a section on Bayesian trials in medical devices. I found the speakers (which came from both industry and FDA) top notch, and at the end the session turned into a very nice dialog on the CDRH draft guidance.

I then went to a session on interacting with the FDA in a medical device setting, and again speakers from both the FDA and industry were top notch. Again, the talks turned into very good discussions about how to most effectively communicate with the FDA, especially from a statistician/statistical consultant's point of view. I asked the question of how to handle the situation where, though it's not in the best interest, a sponsor wants to kick the statistical consults out of the FDA interactions. The answer: speak the sponsor's language, which is in dollars. Quite frankly, statistics is a major part of any clinical development plan, and unless the focus is specifically on chemistry, manufacturing, and controls (CMC), a statistician needs to be present for any contact with the FDA. (In a few years, it might be true for CMC as well.) If this is not the case, especially if it's consistently not the case throughout the development cycle of the product, the review can be delayed, and time is money. Other great questions were asked on use of software and submission of data. We all got an idea of what is required statistically in a medical device submission.

After lunch was a session given by the section on graphics and International Biometric Society (West N America Region). Why it wasn't cosponsored by biopharmaceutical, I'll never know. The talks were all about using graphs to understand effects of drugs, and how to use graphs to effectively support a marketing application or medical publication. The underlying message was get out of the 60's line printer era with the illegible statistical tables, and take advantage of new tools available. Legibility is key in producing a graph, followed by the ability to present a large amount of data in a small area. In some cases, many dimensions can be included on a graph, so that the human eye can spot potential complex relationships among variables. Some companies, notably big pharma, are far ahead in this arena. (I guess they have well-paid talent to work on this kind of stuff.)

These were three excellent sessions, and worth demanding more of my aching feet. Now I'm physically tired and ready to chill with my family for the rest of the week/weekend before doing "normal" work on Monday. But professionally, I'm refreshed.

Adaptive trials page is up

The presentations given at the FDA/PhRMA conference on adaptive designs in November is up. A lot of interest in this workshop was stimulated when Scott Gottlieb mentioned it back in July during a keynote at another conference. Unfortunately, registration seemed limited to PhRMA members.



The content looks pretty good. I'll have more to say on it as time goes by.