Navigating dark data: Ever Given in Suez

This navigation reminds me of the hyperspace short cuts in science fiction. In both cases, the ship is in a short cut where spatial properties are different from more routine conditions of open seas in deep waters. In both cases, the navigator must rely on information he had when he entered the short cut. The navigator has very little if any relevant measurements of what will really matter to the outcome of the journey.

Dark nothing, dark data’s blind spot

There is a benefit to opening our processes to the possibility that the reality may be changing, where the changing is from an evolving intelligence or even from a plethora of competing intelligences that have transitions of power much like our political systems. Admitting dark data into our algorithms blinds us to this possibility, especially when we allow dark data to have priority over observations.

Urgency is dark data

A government by data could consider the observations of iatrogenic complications and deaths. The public’s fear of a virus could grant this government permission to impose some new authoritarian policy that would do something, but that something would exploit the opportunity to improve the future prospects based on all observations of the current world. Such a government would be free to decide to tackle the problem of iatrogenesis instead of the problem of the virus. Fixing the overextension of medicine may ultimately benefit more people than overreacting to a virus that is not as threatening as the population perceived.

Dark data betrays bright data

The current democratic government is run by politicians, bureaucrats, and electorate who all are gaslighted into distrusting their own observations that disprove the original explanations and projections.   We need to ignore what we are seeing and continue on the original plan because it was based on infallible science, science we know is proven because we can replicate the experiments that prove it.

Holding dark data accountable

The biggest failing of science in the current COVID situation is its inability to react to new evidence that its original conclusions were wrongly decided, and the assurances to governments were incompetent.   We implicitly accept that any initial science-based decisions attains some law-like status that is automatically presumed to be true until there is overwhelming evidence that it is wrong.   In particular, such decision making does not permit a simple apology for making a mistake following new data that clearly disproves the original science. 

Pandemic of Dark Data

Perhaps the real agent behind this pandemic is science itself.   Given a sufficiently dire circumstance, science can shut down our natural defenses of critically thinking about observations we can clearly see.   Science tells us that if this is the disease it warned us would come, then we have no choice but place all trust on science at the expense of paying attention to what we are seeing.

Indifference to dark data

We should learn from recent experience of large data technologies the lesson that decision making can benefit from streaming data in addition to (and often instead of) the publication science of one-time experiments.    It is clear now that policy making needs access to a continuous stream fresh data about old ideas, especially when that data accumulates over time.   With access to the technologies to do this work, it is unacceptable to base policies on the failed approaches of the past that rely on published studies.