Exploiting sharding capabilities in cloud databases for better preservation of history

For the project of knowledge or hypothesis discovery, this sharding of history is more valuable than attempting a historical report using the operational database. The sharded history retains the context of the data. For a business example, assume a report for the previous period involved some action by an employee who has since been promoted to a different position. Using the operational database for this historical information will naturally return the erroneous result that the new position was responsible for the prior action when in fact that action was done in capacity of the older position.

Perspective of real time analytics

The potential return for exploiting operational data will not justify the investment. This return is naturally limited by the short time period available to take advantage of the opportunity. The window of opportunity is naturally short because new operational data will present distractions of new opportunities to pursue. Also, the competitors and customers also are employing their own operational data intelligence so that they will quickly close any advantage gap. Unfortunately, this investment distracts the organization away from historical data that offers more durable knowledge discovery.

Business activity tracking can improve requirement analysis for maintaining legacy applications

Legacy applications can benefit from big data approaches without the need to replace the legacy architecture with new technologies. Instead the big data can augment the application by collecting higher volume, variety, and velocity data about the user’s activity using the application. Analysis of this data can inform decision makers where there may be problems with the work-products. Correspondingly, it can provide requirements analysts with information about where improvements are needed or with more complete library of edge cases to consider for new designs.

The meaning of persuasion in the age of data

A data-driven economy is not a free economy. While there remains promise that algorithms acting on vast amounts of rapidly arriving data can produce a better economy, I am suspicious that such an economy will eventually languish because it robs the human actors of their ability to negotiate. The vitality of a free economy derives from individual freedom to negotiate terms of engagement. Eventually, A data-driven economy may prove to be superior but it will succeed only by suppressing natural human negotiation. Human actors negotiating in a data-driven economy must negotiate with machines. Applying approaches that work for other humans to machines instead is criminalized as cybercrime. Human negotiation involves coming to terms with weaknesses as well as strengths. Exploiting weaknesses of machines is a crime.

Playing with some data: Capital Bikeshare data

With modern speed of data retrieval, analysis, and visualization, we may be encountering a new form of logical fallacy of appealing to authority where the authority comes from the speed at which we can present affirming data for our theses. Assuming that human behavior is a product of evolution, there has not been enough time for evolution to adapt to the new reality of nearly instant affirmation of some consequent. Historically, we recognized a pattern that we can trust affirming data if it arrives quickly. Before modern data technologies, the speed of finding affirming data was an indication that affirming data is abundant around us so it didn’t take long to find. That particular mode of thinking is no longer valid with modern data technologies. The instant access to a wide variety of data makes it possible to find affirming data very quickly. It will take a few generations for evolution to catch up to teach us to not trust speed of affirmation as proof of some hypothesis.