Missing from my taxonomy of types of data (as summarized in this post) leaves out mention of dirty data. The different types of data have different levels of trustworthiness where bright data is highly trusted, while model-generated data is less trusted witness of reality. My descriptions of the dim data or model-forbidden data is not…
Data-driven analytics currently thrives on wealth of available data that happened to be freely volunteered by others. The ultimate realization of the value of the data comes after the analytics and visualization presents a story that attracts an audience. To be relevant to the difficult and urgent problems, data science projects need to find ways to propagate the financial benefits of the final results back to the information sources in order to provide the necessary incentive to uncover the difficult to obtain data. The journalism market needs new incentives to redirect their skills toward collecting a vast number of first-person accounts. As I described earlier, the journalist’s skills are exemplified by the input and the output stages of data science projects. The mathematics and software for analytics and visualization need data to work with and story-tellers to attract audiences to the results.
We need the opportunity to discover a hypothesis that practices informed by our best theories may actually be making matters worse. We can never learn that lesson if we keep substituting theory-based explanations for otherwise mysterious transmissions. The theory itself may be the problem.
My earlier Ebola post … implied that [data science] participation is optional. With this post, I think the employment of big data predictive analytics is not optional. This disease will spread to affluent areas where people will learn of their degree of contact separation from the infected individual. We urgently need predictive analytics to inform these people of quantitatively verified risk of contracting the disease given that degree of contact separation.
We have the capacity to supply the necessary technologies to this region to begin collecting social media data. We just need the incentive that this data will provide valuable contributions to the fight against the Ebola epidemic and that the data science community is available to devote their efforts and put their reputations on the line to come up with big-data recommendations that really make a difference for an urgent and life-critical critical. First of all, we need data scientists to present solid proposals for how big data can help the management of the populations with the goal of improving our prospects for controlling the epidemic.
Despite the challenges for exhaustively identifying possible bad outcomes, I still think it is valuable to invest some portion of our data collection and analysis to seek out worst case scenarios. Even when we find results that lack sufficient evidence to change our decisions, the results can inform us of what to be aware of as we continue to watch the arrival of new data… we cannot find what we don’t want to find.
To realize the benefits of big data analytics (descriptive, predictive, or prescriptive), we need to obligate everyone to participate in order to be measured in order to avoid the selection biases for those analytics. In many cases, this participation requires following the recommended course of actions so that we can measure the results to tune the algorithms. In some sense we need an obedient population to follow the recommendations of big data analytics in much the same way that authoritarian regimes demand.
In earlier posts, I distinguished data into two categories: operational and historical. Operational, or present-tense, data is about obtaining the best record of what is occurring in the immediate moment. Ideally, operational data is well documented and controlled observations of the real world so that information may be used to influence the events in a…
A definition of what separates data from big data is the triple challenge of large volume, high velocity, and wide variety. The technologies of big data system (data collection, storage, retrieval, analytics, and visualization) present new opportunities for making positive decisions that predict where to find benefits, or that prescribe actions that can beneficially…
As people consider their continued investment in big data and its associated technologies, they are probably interested in knowing what the downsides might be. Can the project fail? How bad can a failed project be? Promoters of big data and associated technologies frequently release some news about some organization realizing some major benefit from their…
Hypothesis Discovery 