In previous posts, I’ve challenged what I called dark data.   I introduced the concept as a general complaint about interpreting data when the data have varying degrees of separation from an actual observation.   Generally speaking, my use of the term dark data was a synonym for scientific theories.   The theories provide predictive data that replace gaps in observations.   Also, theories will encourage us to reject observations that are always prone to error or randomness.  I wanted to explore what can be learned from observational data (of various levels of brightness).

My initial definitions of dark data to represent the bias that scientific theories impose on observations were later expanded to include journalism stories.  I could build upon my lexicon for data by distinguishing different types of dark data.   Many will argue there is a difference between scientific theories such as laws of motion or thermodynamics, and stories told in journalism, and gossip told between friends.   From my perspective, I see a benefit of not making the distinction at all.   In terms of offering any new observations about the world, there is no difference between Einstein’s theory of relativity or the speculation of antediluvian advanced civilizations.   I fully agree that there is a difference in utility, but that utility is distinct from observational data.  Part of the reason why we needed theories is that theories are good at data reduction.   When we didn’t have access to big data technologies, the only way to convey information about observations was to condense the observations into a theory, a formula, or a statistical model.   Now that we are entering an age when vast amounts of observational data is available, there is less need for data reduction.   Keeping theories out of data offers the potential of discovering new theories that otherwise would be forbidden by existing theories but not by previous observations.

Dark data, in my usage of the term, is still data.  I am not really arguing to reject it from the data store.   Instead, I do welcome it but only as long as it is clearly separated from observational data.   We need stories that humans tell each other and we need objective observations of the world.  We need to protect the two from cross contamination.  Observational data should be kept apart from stories.   Analytics can compare the two, for example, observations substantiate stories or stories can give us confidence in observations.   However, I consider analytics to be a dynamic concept to be applied against both the observations and the stories at exactly the time when the reader attempts to interpret them.   The answer can change from one reading to the next.   I propose that this is a good thing.

For this discussion, I want to expand the idea of dark data to include all forms of human narratives.   Dark data includes the entire ranges of stories that humans tell each other to give meaning to the world and in particular their experiences and lives.   As with my earlier definition of dark data, there are times when I want to consider narratives but there also times when I want full freedom to consider observations in the absence of any narratives.    This is consistent with the idea of just-in-time narratives, narratives constructed at the moment when I’m prepared to think about a particular topic.   The narrative should be free to accommodate all observations and stand apart from other narratives.

In the ideal form of automated narrative building of all available information, the new narrative would accommodate all of this information.   In this ideal form, the new narrative would out-compete older narratives because the newer narrative would consider the older narratives as pre-existing dark data, while the older narratives would not have access to the dark data of narratives that haven’t yet been constructed.   Of course in practical terms, newer narratives will not automatically be superior to older narratives.  We will still require rhetoric to evaluate competing narratives.

I think that the modern era of capability of storage and rapid retrieval of vast amounts of information make possible a more dynamic system of producing narratives that guide our lives or at least to interpret our lives.   Given our access to observational data that otherwise would require narratives to reduce, we can consult the data instead of narratives.   We can then make decisions more dynamically, and free from the need for consistency with prior narratives.

Modern controversies, such as the much debated multi-gender revolution, can make more sense when viewed as an inevitable consequence of modern information.   In modern times, there is access to body modification technologies permitting customizing a person’s appearance to accomplish the person’s goals (whatever those are).   Also in modern times, there is less priority on reproduction, at least at the species level.   There is an excess of breeding-capable individuals, so there is little existential danger from the loss of breeders resulting from these choices.

My point is that there probably would not be any controversy at all if we adapted a more dedomenocratic approach of evaluating our conditions by data alone.   The controversies are the result of demanding obedience to prior narratives whether they are based on the sciences (biological or psychological) or based on morality or religion.

It seems to me that we should be able to live in a world where the social narratives can adapt to the current conditions.   Indeed, it is possible to live in a world where personal narratives can change over time as conditions change.

The gender-controversies serve as a good example of a dynamic narrative.   In the specific cases of transgender people undergoing body modification through hormones or surgery, these decisions can result in eliminating the possibility of future narratives.   In particular, these actions can result in sterilization that will prevent the biological-parenting narrative should the person seek that in the future.

In the context of an imposed biological narrative, we can at least recognize the possibility of regret, particularly for the younger population seeking these options.  Also, our historic deference to personal narratives would impose a narrative onto the person that the person can not escape.   True, a person going through surgical and hormonal changes will be limited to future narratives that are constrained by these changes, but that does not mean the new narrative has to include the old narrative.

In the data-rich culture, we can adopt a dynamic narrative that considers only the current conditions and not the past at all.   Each time we need a narrative, we can construct a new one based on all of the available information at this moment of time.

Certainly earlier choices of narratives to follow will have consequences for what is possible in the present.   On the other hand, being free from consistency with the prior narratives permits taking advantage of new narratives that would not have been available previously to anyone or of new narratives made possible by the current conditions that are a consequence of prior choices.

Another way to describe this new opportunity of freedom is greater acceptance of what otherwise would have been condemned as a disappointment.  Historically, humans have always tolerated more disappointment from their machines than they have in each other.  I think the reason we demanded more from people than we did from machines is that we were limited in our access to data.  When considering a person’s value for a future endeavor, it was impossible to evaluate the extensive and detailed experiences of the person’s life until this point.  Instead we were forced to evaluate the merits of the person’s narrative.  That narrative may be more formal such as in a resume or curriculum vitae, or informal in the form of reputation or recommendations.  These are narratives that attempt to reduce the abundant data about the person into something that can be communicated quickly.

Today, we have access to technologies that can quickly communicate the raw data about people.   This presents the possibility of constructing a new narrative based on that data and the current circumstances.  With this just-in-time narrative building, there is no necessity to insist on consistency with earlier narratives.   This disentanglement with the past is the opportunity that is newly available to the human species.   This is an opportunity for a new kind of culture that is not possible when we respect established narratives.

Worldwide, humanity faces a broad level of crisis of how to sustain the advances made in the past couple centuries.   In particular, we as a world have an economy based on debt where that debt is no longer sustainable at the world-wide scale if not at all individual nation-state levels.   We need a new approach to governance at all scales in order to sustain and build upon our culture.   I think a new approach is possible by recognizing that narratives are expendable.  We do not need narratives any more.   More specifically we do not need consistency of narratives over time, for all narratives at all scales from nations to individuals.

Modern data technologies provide us an unique opportunity to reinvent the vary nature of governance as well as interpersonal relationships to be free from narrative precedence and consider only the current needs and the available data of objective observations.  Until now, we needed narratives because of their data-reduction.  Now, we have superior methods of data reduction, where that reduction occurs at the moment we need to make a decision and that reduction need not consider outdated narratives.