Showing posts with label big data. Show all posts
Showing posts with label big data. Show all posts

Wednesday, February 20, 2013

The burst of the Big Data bubble, and do we need the hype, anyway?

So, now I'm seeing some buzz over Twitter that the Big Data disillusionment is starting now. Frankly, I've been wondering when this would happen. Of course, the next stage involves making strategic investments in Big Data resources, and having these resources quietly being used effectively, at least if Big Data follows technologies such as neural networks, Java, etc. So the theory goes, all surviving technologies follow a pattern of hype, disillusionment, and then quiet acceptance.

Did we really need this period of hype? I can understand companies hype up a technology to maintain interest while they try to make their offerings mature, and overhyping usually leads to disillusionment, but I wonder if there is a different path. R, Python, and some other open projects seem to have flattened the hype hill and disillusionment valley, probably because the larger number of people hacking the inside generates its own interest and maturity mechanism.

Anyway, I look forward to the maturing of big data at least until the privacy concerns generate widespread panic.

Monday, August 27, 2012

Romney’s “secretive data mining”–could the same techniques be used for clinical trial enrollment?

Romney has been “exposed” as using “secretive data mining techniques” to find donors to his campaign in traditional Democratic strongholds. (These techniques can be learned in any of these free online courses offered through Coursera and Udacity along with the massive databases collected by the different parties.)

Of course, my thought is, can we use these techniques to find potential participants in clinical trials? I think that if we can work out the privacy issues, this represents a useful tool for clinicians to find not just trial participants, but patients who need to be treated, but for some reason are not being treated. This could be a win for everybody.

Other ideas:

  • using Google trends, much like Google uses to identify flu outbreaks
  • mining discussion boards
  • identifying need through blog networks

I’ll be taking the Web Intelligence and Big Data class through Coursera, so maybe I’ll get more ideas.