Interesting interview with Michael Jordan (no, not that Michael Jordan) about the the challenges with Big Data. It reminds me of a conversation that I had with a good friend of mine just the other day. We discussed how Big Data is meaningless in the absence of context. Likewise, Michael Jordan’s point touches on the other side of the issues with Big Data and BD Analytics in such that meaning can be obtained from spurious connections, and that confidence can be over assumed.
A quote:
“If I have no principles, and I build thousands of bridges without any actual science, lots of them will fall down, and great disasters will occur.
Similarly here, if people use data and inferences they can make with the data without any concern about error bars, about heterogeneity, about noisy data, about the sampling pattern, about all the kinds of things that you have to be serious about if you’re an engineer and a statistician—then you will make lots of predictions, and there’s a good chance that you will occasionally solve some real interesting problems. But you will occasionally have some disastrously bad decisions. And you won’t know the difference a priori. You will just produce these outputs and hope for the best.”