We need a new approach to governance at all scales in order to sustain and build upon our culture. I think a new approach is possible by recognizing that narratives are expendable. We do not need consistency of narratives over time, for all narratives at all scales from nations to individuals.
The advantage of data on read strategy is that it separates the processes of data collection from the processes of applying a schema in order to interpret the results. We can learn more easily that our prior knowledge was wrong when we get prior knowledge out of the data store.
For the project of knowledge or hypothesis discovery, this sharding of history is more valuable than attempting a historical report using the operational database. The sharded history retains the context of the data. For a business example, assume a report for the previous period involved some action by an employee who has since been promoted to a different position. Using the operational database for this historical information will naturally return the erroneous result that the new position was responsible for the prior action when in fact that action was done in capacity of the older position.
The potential return for exploiting operational data will not justify the investment. This return is naturally limited by the short time period available to take advantage of the opportunity. The window of opportunity is naturally short because new operational data will present distractions of new opportunities to pursue. Also, the competitors and customers also are employing their own operational data intelligence so that they will quickly close any advantage gap. Unfortunately, this investment distracts the organization away from historical data that offers more durable knowledge discovery.
Legacy applications can benefit from big data approaches without the need to replace the legacy architecture with new technologies. Instead the big data can augment the application by collecting higher volume, variety, and velocity data about the user’s activity using the application. Analysis of this data can inform decision makers where there may be problems with the work-products. Correspondingly, it can provide requirements analysts with information about where improvements are needed or with more complete library of edge cases to consider for new designs.
Agile practices are shown to work. Such practices are integral to many modern successful businesses. Similarly, machine learning algorithms are showing their success. At the time of this writing, it appears that both will play a large role in the future economy.
My point is that we have the technology to debate over this volume of data and this debate is likely to be more productive than elevating the issues into something that fits the ancient model of sophistic debate. With modern data mining and visualization tools, the public can discuss the details of all of the relevant features of the healthcare debate. We should learn from how learning arises in machine learning from feature selection among a large set of features with diverse measurements. In analogy to neural networks, for example, the same volume of data can be presented to networks of humans each with access to tools empowering them to observe the vast richness of the data and then use debate to mimic forward propagation of accumulating conclusions and backward propagation of derivatives of errors.