Data cleaning is harder than statistical analysis

Statistical analysis is relatively hard, but it is a piece of cake compared to data collection, cleaning, and manipulation. In fact, in clinical trials research, we spend millions of dollars to develop and advance the capability to effectively manage data. Just about any clinical research organization worth the price has a strong data management department that they’ve spent a lot of time cultivating.

It’s time to take this a step further. In my workplace, we have a very close integration of the statistics group (consisting of statisticians and statistical programmers) and the data management group. In the latest issue of their newsletter, the Society for Clinical Data Management has included an article for the optimal collaboration between statisticians and data managers.  (I take this a step further and include the medical writer.) This collaboration takes a lot of time – time I could be spending doing statistical analysis. However, if the statistical analysis involves working around fewer data issues, it’s all worth it.

Exercise helps statisticians

Statistics is a rather sedentary job, and, over the years, I found my effectiveness decreasing as I found fewer “peak” hours in the day. I also gained a lot of weight. The number of migraines I experienced went from about two a year to about once a month.

In the last two or three years, I’ve been getting out of my chair to go for runs, I’ve taken up taekwondo, and also I a small gym that provides small-group personal training. In addition to adding who knows how many years to my life, they’ve really helped my focus and concentration when I’m doing statistics. I’ve also decided to take once a week or so off of thinking about statistics, which I’m finding helpful.

I only wish I had established these habits years ago.