Wednesday, May 30, 2012

Statistical leadership, part IV: the world needs you

Read this, even if you are not a statistician. Go on, I'll be here when you get back.

This article was adapted from Roger Hoerl's excellent Deming Lecture at the Joint Statistical Meetings in 2011. This is a call to action, of course, but a call to something even deeper. In a time when we are running very short on critical thinking, we need more people to think critically and speak up. Critical thought, of course, implies more than just speaking against something (the status quo, proposed solutions, or other object of thought), but rather seeking a deeper understanding of the problems that face us, and what the most effective solutions are. In this imperfect world of tradeoffs, we have to understand the impact of solutions and of solving problems.

This is a call to understand our world, and to make it better. (The former does not necessarily precede the latter.)

Tuesday, May 15, 2012

Thoughts on privacy

As this world gets more connected, and as data storage and analysis advances, we have to change our notions of privacy and data stewardship. About 25 years ago, right before email hit the big time and data analysis methods were limited to small datasets or Cray supercomputers, having data was a huge deal. Coverups, such as Watergate, were characterized by hiding data from others. While still true, it’s a lot harder, and, with increases in computing speed and availability of data, it’s a lot harder to hide from the rest of the world.

Whether we like it or not, our notions of privacy have to change. In a recent instance, Target knew of a daughter’s pregnancy before her father did. (Mailings to the house were the source of a lot of consternation and an uncomfortable chat.) Doing this is fairly easy: you assign an ID number to each customer based on credit cards or loyalty cards, mine purchase data for what can predict not just pregnancy but also a due date, and apply it to future customers. Many first year statistics grad students have already learned the basic methods for doing this. This cat is out of the bag, and it’s not going back in. We will not be able to legislate this practice out of existence (and perhaps we shouldn’t be, anyway).

So what now? How does privacy have to change? It appears that a new attitude toward privacy is rising, but this is equally disturbing. In the link, teens were given Blackberries, with the understanding that everything they did on it would be monitored and analyzed, and they still went for it. They even did drug deals using these devices!

I think our privacy laws have to evolve to deal with this new reality. We require de-identified data for data released to the public, but even that strategy will only be useful for so long. No, the bounds of acceptable behavior based on data have to be re-thought. For example, is it ok to drop insurance coverage based on FB postings of drunken parties or certain tweets? Is it ok to terminate an employee because a manager did some social network analysis of public data and found some badmouthing of the company? Is it ok for a car insurance company to bump up your premium because you blogged about Top Gear? Answering these kinds of questions, which really are just a couple of steps away from product recommendations, with legislation will just be the start.