Dark data: fall of civilizations

The histories of human civilizations demonstrate that the nature of humanity permits it to create and strengthen large civilizations.    There have been many examples of great civilizations with various methods of governments where even those with repressive authoritarian rule nonetheless created civilizations that were for a large portion of the population preferential to the lack of civilization experienced by the ancestors.    There may have been limitations on individual expression, but individuals were able to enjoy the fruits of the civilization and often find ways to uniquely contribute to that civilization, in recognizable ways despite often being minuscule in the grand scheme of things.

Assuming as accurate that modern humans existed for the past 50-100 millennia, most of human history was during periods when there were no nation or empire scaled civilizations, and even when they did exist, often the majority of the human population existed outside of their domain.   Based on this, it would seem that the default human behavior is to not cooperate to building large scale civilizations.

Also, I assume as accurate that genetics play a major role in determining the animal behavior at least in terms of tendencies.    Based on the entire history of all modern humans that have ever lived, it would seem that this genetic predisposition is to not cooperate outside of very limited population defined by tribes of closely related people.

Yet, the same genetic makeup does occasionally construct, strengthen, and preserve vast civilizations of peoples who are so distantly related so as to have no recognized familial relationship.   The fact that such civilizations arise is exceptional when compared with the evidence that most historic humans lived outside of such civilizations.

The historic evidence is that humans are by nature relatively simple in terms of adequate social arrangements.   Tribes or extended families appears adequate to satisfy humans, including their desire to innovate or explore.   The history demonstrates the resourcefulness of tribes (or individuals in those tribes) to take advantage of the luxuries of environments that they may encounter.   There is no contradiction of imagining a tribe assimilating into a larger-scale civilization if they would encounter it.   The initial introduction may include some animosity, but this can be as short lived as learning how to get along with a new environment with different sources of food and with different hazards.

Primitive peoples managed to populate a large number of environments on Earth, each with its own unique survival challenges.   Clearly, these same peoples would similarly adapt to civilizations that they may encounter.   These civilizations are simply artificial environments where the challenges of survival involve negotiation with other humans in addition to the natural elements.   However, this scenario involves a preexisting civilized society.

Something exceptional to human nature has to occur to establish such a civilization in the first place.   It may not be within human nature to build civilizations.   Instead, the proto-civilizations occur accidentally as a consequence of local environment that either is especially conducive for human survival, thus negating the need for migration, or holds some natural attraction worthy of regular visits or extended stays.   Multiple tribes may congregate in same area and the area has sufficient resources to sustain all of them without major conflicts.    Gradually such a settlement may grow.   The citizens will begin to specialize to assure their survival by exchanges with other humans instead of (or at least in addition to) confronting nature directly.   Eventually, the civilization will have some form that can be recognized as being distinct from being uncivilized.

In short, this idea is that civilization is not a direct consequence of human nature, but instead an accident.   Once one begins to form, though, the resourcefulness of human nature will take advantage of the new environment, and allow their tribes to integrate into it.

Personally, I think civilization building is more deliberate.   As far as I know, there is no evidence that this is true.   The records we have for the ancient civilizations are most likely later rationalization for something that by then has already existed.

Even in the modern age, civilizations like that of the USA may be rationalized after the fact.   Certainly, the historic evidence of the original rebellion from England and the writing of the constitution demonstrates an intentional action.   However, that intentional action did not conceive of a world spanning power it has become.   My understanding is that the original intention was for a self-governing body of primarily farmers with very little interest in interactions with neighboring states, let alone overseas lands.

Despite evidence to the contrary, I think that the initiation and building of a new civilization consists of deliberate choices by anonymous actors who had the full intent of building something larger than a tribe.   Most likely these are deliberate choices in many different interactions: some may be to open opportunities for trade (especially to acquire something not available locally), while others may be to cooperate to build common facilities that can benefit many, such as the building of a meeting hall, a bakery, or a brewery.  Once these get started, there will be further choices to specialize as carpenters to build the buildings or carts, as administrators for the halls, or as artisans to operate the bakeries or taverns.

I may be far off from what gets things started.   My point is that these are deliberate choices made with intentional calculations that such initiatives would be long-term beneficial at least to the individual, but mostly like the tribe as well.   These are choices that not only envision future benefit, but also recognize what is lost.   Devoting time to carpentry and the betterment of that skill will come at the detriment of that person’s ability to fight off or to evade predators, or to seek out foods available at different parts of the year.

At the root of the above discussion is whether creating and building civilizations is an inherent human behavior.   It is hard to argue that it is inherent when so much of history lacked organizations beyond tribal scales, and even when examples of civilizations did exist that most contemporary people neither participated nor attempted to mimic them.   On the other hand, nothing else but humans created these civilizations.   No other life form on earth created these for humans to later take over.  I’ll also presume the absence of some supernatural or extraterrestrial beings with a benevolent penchant for building human habitats.

From the above, I conceive of a human nature that can be satisfied fully with tribal-type existence subsiding on the surrounding natural or artificial environment.   Clearly, there will be benefits and costs for different environments, but the survival of the tribe will be satisfying to the members of the tribe.   Out of this same human nature will arise a small minority of individuals who will intentionally strive for something larger and longer-lasting than the immediate tribe.   Judging from the evidence of history, most of these attempts will fail, and often these failures will not disrupt the rest of the tribe.   Occasionally, however, a large civilization will emerge, or an existing civilization will enlarge.   The component tribes will move in to fill the niches created by this civilization, and they may even celebrate those benefits.

In other words, the discussion described the civilization-creation capability of humans to be a contradiction.   Human nature defined by genetics can be fully satisfied with a tribal hunter-gatherer type of existence.   The exact same human nature can occasionally create large-scale civilizations where once established many inhabitants will not be able to conceive of surviving outside of that civilization, and indeed most will be incapable of such independent survival.

Creating and building civilizations is the exception to human history.    But it is not exceptional that humans inevitably destroy their civilization, leaving behind a much smaller population of isolated tribes who may occasionally rummage among abandoned ruins for something of value.

From a genetically-defined human-nature perspective, it is even harder to explain why people will allow civilizations to fail, than it is to explain why people who are satisfied with tribal hunter-gather existence will build the civilization in the first place.   In the case of the fall of civilization, the civilization already exists.  It could have been just as easily been built by benevolent extraterrestrials, or some supernatural being, built as a human habitat.   The tribal humans could have moved in just as they would have moved into a natural habitat, learning to living within the constraints the ensure occupation by an unlimited number of future generations.

Something different happens in artificial habitats.   Inevitably, the inhabitants will destroy the artificial habitat of civilization.   In contrast, there is evidence that humans can persevere indefinitely in natural habitats, even when it results in much lower quality of life.

How can the same creature that builds great civilizations also be the same creature that eventually eagerly dismantles it, even with obvious consequence of losing all the benefits.

It is as if the norm for human nature is to live at tribal levels, often at the thin edge between survival and extinction.   It is unusual for people to build large civilizations, but when they do, they often persist for multiple generations, but there will come a time long after the civilization’s founding generation is long forgotten, the civilization will fall, and the inhabitants will abandon the cities forever.

My theory of data science may explain what is happening.   In this theory, I propose that large-scale civilizations are not natural, it is not a natural consequence of basic human animal behavior.    Civilizations occur from the collection of data.   In particular, this data is observational in nature.   People observe that location that predictably provides shelter or resources, or they observe the locations that offer some sense of delight.   Based on this observational data, they adjust their customs to revisit the sites, or to settle the sites, even when the conditions become challenging.    Because of this historically observed data, new arrangements are made to make the location more habitable during the harder times even though it may be easier to simply migrate to a more habitable site.

Once an initial settlement arises, the inhabitants observe more data about what is needed to make the site more suitable for survival.  One such observation may be a simple trend of increased difficulty in surviving with a growing population.   Other observations are to discover that the larger population creates new opportunities for survival strategy through the building of common structures or common laws, and through specialization to realize these ideas.

In summary, both the initiation and the growth of a civilization is about the retrieval and retention of observational data.   In earlier discussions, I described as bright data the directly observed data.   In this context, the growth of the civilization includes adaptation, replacing older constructs with newer ones because of newer observations that informs the population the old observations are no longer valid or were somehow incomplete.   A developing civilization builds new structures on top of ruins of previous structures instead of renovating the old structures.   They will even demolish the old structures if they are no longer relevant.   This is behavior that is driven by new observations.

I contrasted bright data from dark data.   Dark data is data calculated from models about the world.

At first, civilizations will use theories sparingly to substitute for missing observations or observations that are impossible to acquire.   When given the choice of a fresh observation or a derivation from a model, the fresh observation is both more valuable and less expensive.

As time progresses, the getting information from theories becomes easy enough to compete with obtaining new observations.   Also, we grow to trust the theories based on past performance.  For successful civilizations, such as the current-day USA, people perceive the theories about the nature of government as valid by evidence of the past success of those theories.   For USA, those theories include concepts of representative-democratic rule, independent bureaucracies, equal rights under the law, and free trade in a capitalist model.

I suggest that at some point, the theories become more important than observations.  As I have complained about before, theories can give rise to forbidden data: these are the outliers we feel justified in omitting because they don’t fit the theory.   On a broader scale, we can discard observations that contradict the foundation theories about our government.   Our confidence in our theories authorizes us to reject contrary evidence.

Once civilizations mature to the point of being governed by theories instead of observations, they open the opportunity for others to propose alternative theories.   These alternatives may be adjusted to fit the past evidence about as well as the established theory.

At some point in the evolution of civilizations, the civilization make a fundamental change by rejecting the past practice of being governed by observations and instead accepting the practice of being governed by theories.   Government by dark data instead of bright data.   I propose that it is at this point when civilizations begin to fail.   Furthermore, I propose that this change is inevitably fatal.

Notice that this change can occur without any disruption to the nature of the government.  In the context of the USA, we can change from a bright-data method to a dark-data method without any change to the mechanisms of democracy or rule of law.   All we have changed is the nature of evidence.

Specifically, we allow model-generated dark data to substitute for bright observed data.   More importantly, we allow model-generated data to override observed data.   Much of current public debate and politics are basically blind to observed data because we put full faith in dark data from our competing models.

Once we prefer theories to observations, we offer the opportunity to replace one theory for another.  The new theory may convince us by its ability to explain past observations.  However, the two (or more) theories present different propositions about the present, often contrary to actual observations we can see if we would allow ourselves to trust our observations.

The switch from bright to dark data convinces us to doubt our observations, giving us no option but to be governed by theory instead of observations.   This is the start of the decline of a civilization.   While I can describe this in semi-scientific terms for current civilization, similar models in the past were religious or hereditary in nature.   In both cases, we seek guidance from theories or ancestors.   Any such guidance will override any contrary evidence from current observations.

The civilization in decline makes the mistake the the risk of following inferences from current observations outweighs the risk of following guidance from established theories.  The decline is inevitable because it takes too long for present observations modify the theories, if they are allowed to do so at all.   Unfortunately, total collapse of the civilization may be the only evidence permitted to abandon the theories.   At that point, the people will abandon the civilization entire because they will see it as the embodiment of that failed theory, instead of original brilliance of what started it.

I strongly suspect that our current civilization is at least at the inflection point of decline, if it is not already well on its way into decline.   Clearly, we have a lot of confidence in our theories, scientific in origin instead of religious or customary.   We are also dismissive of observed evidence that contradicts or at least challenges established theories.   We base policy decisions on established theory, and the decline has started because we are permitting competing theories into the debate and people are segregating in terms of what theory they will follow.   The debates can not be resolved with commonly observed data when at least one of the theories rejects the validity of the observation.

The poison in the system is the dark data of theory-generated data from competing theories.

Unlike past civilizations (I presume), we have an opportunity to dethrone the primacy of dark data.   The modern big-data technologies permit pervasive and extensive collection of current observations as well as their persistent storage and rapid retrieval for analysis.   With big data analytics we can make decisions that are heavily influenced by observational data, minimizing the relevance of model-generated data.

I described this in earlier posts as dedomenocracy, or government by data and urgency.  This concept is a replacement of the current democratic republic (and administrative state) with a system of government that prioritizes observations over theories.   In such a government, analytics of observations dictate the rules governing society.   Just as a republic can have a democratic element by democratic election and merit-based admission into the administrative state, a dedomenocracy can be democratic in terms of public validation and verification of data sources and analytic algorithms.    In addition, the primacy of observed data requires minimizing the laws to the most urgent issues in order to permit the most unbiased observations of current human conditions including behavioral tendencies of individuals and groups.

Such a government by data runs the risk of making bad decisions based on spurious correlations lacking any underlying causality.   The large variety and volume of data minimizes this risk.   Also, freed from debate on actual policy will direct the public’s attention on the data and the algorithms to select those that are most trusted to be causally relevant.   There will be debates especially to counter the algorithm outcomes, but the debates are best focused on the validity of observations and the algorithms instead of the outcomes directly.    More importantly, such a government is free to adapt to current conditions, taking full advantage of current opportunities without the constraints of consistency with past ideas or decisions.

Dedomenocracy, I believe, recreates the cause of the vitality that created and grew the civilization in the first place.  As such, it offers the potential of a greater future as unexpected to us as the current civilization would be to the the authors of the US constitution.

The alternative is to stick to the current reliance on theories and consistency with past.   Maybe this time is different and we will not fail or fail as spectacularly as past civilizations because this time our theories are so complete so we can dismiss any observations that don’t align with theories or aren’t consistent with past proclamations.

Dedomenocracy is more promising.





Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s