Over the four decades of my adult life there has been a recurring theme in my education and profession and that theme is that the world works on fundamental principles and atomic units.   In college, I recall the confidence that we can understand everything from quantum mechanics if only we had the computing power to solve the equations for large systems.   Later, there was the phase of object-oriented software programming that when applied to simulations gave the hope that with enough computing power, we could simulate entire cities by modeling the motions of the sum total of all the component parts, each with its own object behavior and properties including those inherited from ancestor objects.   Even later, with big data, there is the promise that having a giant repository of all available information in its raw form can support discovering anything useful that can be derived from that data by clever querying of the entirety of the raw data.

Each of these had some notable successes in making steps toward the ultimate goal.   These successes became evidence of the validity of our expectations.   With enough resources and diligence we can get arbitrarily close to understanding the world from this deduction from first principles, from root objects, from raw data.

In parallel with this recurring theme in my experience, there was my contrary fascination with the concept of rhetoric.   This fascination may have been lifelong but I recall it becoming explicit upon my introduction to the words of Plato and his disdain for the sophists.   At the time I sided with Plato, in that I was convinced that there was an ultimate truth to the world that can not be dismissed by mere persuasion.   However even at that time, I had some sympathy with the sophists, and in particular I admired those who can master the informal logic of persuasion and rhetoric.   Even at a young age, if given the choice between being an professional in the mastery of evidence and deductive reasoning from first principles, or being a professional in rhetoric of context dependent persuasion, I would hesitate.   That hesitation would follow me throughout my life and with the unsurprising consequence of being neither.

In the later part of my career, I’ve been content in getting useful information out of data of large variety and substantial size, not big-data but big enough.   This data variety include within-type variety of confidence in the individual samples as well as the variety of sources that overlap in relevance often in contradictory ways and often leaving uncomfortable gaps.   Many of the contradictions and gaps were visible only to myself.  I had to learn how to make assertions with reasonable allowance to uncertainty and yet with enough confidence to allow the larger discussion to move forward.   Even when successful at getting past a certain question, I would constantly return to prior work checking up on the progress of those contradictions or omissions, like a doctor checking up on the recovery of a cured patient.

Through this work, my internal thinking about the world changed or maybe it settled on what naturally occurred to me.   This thinking does not dismiss the fundamental knowledge of principles and hard evidence, but it does place it aside as almost a diversion, comparable to the game of Trivia Pursuit.   All of science or other assertions of absolute truth is a similar form of trivia even if it requires a lifetime of dedicated and diligent study and practice.   It is true that we can use science and first principles to design and build durable systems,  even such major achievements as sending a probe for a close flyby of the distant Pluto and getting back high resolution photographs.   No matter what we build, there always remains an element of something coming up that we didn’t account for or didn’t expect would occur during our lifetime.

There may be a single Truth but we will never be able to have a total grasp of its entirety.   We push our knowledge of truth as far as possible but we also use probability and statistics to bound the portion of truth that we have yet to discover.

In the field of engineering, I get the impression that my education and introduction to the field occurred at the early stage of the appreciation of probability and statistics in what I would call hard engineering: engineering where every question had one and only one closed form solution (or if there were multiple solutions, there would only be on mathematically optimal one).   I recall the conflict in the education where I suspect the professors were trying to integrate the more recent appreciation for statistics with established principles of mathematical solutions based on empirically tested principles.   Maybe this was just my imagination as I struggled to absorb both analytic and probabilistic analysis of the world.  Maybe it is easier to learn now where assertions of a single truth is accompanied by some assessment of uncertainty or doubt that may be minuscule but never zero.

But my thinking has moved in a different direction.   Instead of seeing the world in terms of uncertainty in the one Truth, I see a world of multiple versions of truth.   There is no one truth.   When I say that, I’m saying it from from the perspective of data, not of some principle of relativism (that everyone is entitled to their own truth).   The ideal of a data warehouse is to have a single version of truth for all decision making where that truth is a gathering of all available data and cleansed and conformed to the degree that everyone agrees is adequate.

While making large decisions will always be difficult, there is the expectation that it can be easier if everyone is using the same set of facts.  Clearly there is a time savings whenever we can avoiding arguments about what facts to use, but there is a risk that information excluded from this single version of truth could have led to a more profitable or less costly decision.  In recent years there has been a transition to replace data warehouse (a single version of truth) to a data lake (a refuse pool of all available data).

I have always liked the later approach but that is more likely a consequence of my personality of hating to throw anything away.   In my personal life, I accumulate things until some point where I decide to clean everything out, and then I clean out everything.   I approached my career the same way, being in a job struggling to maintain some coherent story that includes my first experience on the job with my current experience.   At some point that doesn’t work, but my solution to that is to walk out and work somewhere else.

I approached data the same way, clinging on the early data even though later data is also available and often superior.   Over time, I have both recent data and old data.   I also have first hand experience working with that old data and of becoming convinced of its adequacy.   Eventually, I confront the fact that my current understanding of what the data is telling me is completely different from my understanding of what earlier data told me.   There is a conflict in the notion of truth.   The current understanding based on current data (and usually much more abundant data) is the best one for coming to some conclusion, but this conclusion could contradict a conclusion I made earlier.   That earlier conclusion may have been just a couple months ago.

I had wrote earlier about my admiration of the New Horizons mission to Pluto where the initial engineering of the probe was separated from the eventual successful completion of that mission by at least 2 decades.   For that to work, there had to be a stable Truth for an immense set of facts.   Stable for that duration was not only the existence and orbit of Pluto, but also of orbital navigation to get there, and of all the various technologies that were required to work at the right time.

In my work, the truth seems to change every month.   I’m similarly asked to help make decisions for benefits that may be years in the future.   At the same time, I’ve been at this job for long enough to experience the results of decisions I’ve helped make in the past.   The results didn’t meet the predictions and I am not the least bit surprised.   Now, I need to help make new decisions.   The lessons of the past motivate me to work even harder to get it right this time, but they also inform me to be prepared for surprises (and likely disappointment).

The discussion above described the element of statistics or of calculated doubt to add to any conclusion.   I would not describe what I feel as doubt and I am reluctant to advertise a confidence level or of probability of being wrong even if I were to properly calculate it.   Instead, I have a certainty that the truth that appears to me given the data I have available is not the truth that would emerge in the future in part as a consequence of the decisions made.   From an outside perspective, of course, this is doubt, but the doubt I have that I am working with the right Truth overrides the doubt I have in interpreting the current Truth.

I do have doubt in my ability to analyze all of the current data.   I just have a bigger doubt that the current data is relevant to describe what will emerge in the future.

If I had successfully worked in a field of science or of engineering, I likely would be more comfortable with the notions of a single truth and of a calculated certainty of how close I am to that truth.   Instead, I worked in the realm of data, not experimental data or data from lab testing, but data discarded after its primary use for many other often unrelated purposes.   From this experience, the world appears to work very differently from the clockwork universe model taught to me in my youth.

If someone is going to design a mission to another planet, land there safely, and complete its entire mission, they would not success unless they expertly apply the clockwork model of how things work.   Despite that, I feel the world does not really work like that.   Their assumptions are sound in that they are likely to succeed with that model, and the evidence of the past century or so of technological achievement proves this has a high rate of success.

It does not make sense to assume that the universe would suddenly change in some substantial way tomorrow.   I suspect it could.  I may not live to see it happen, but I would not be surprised if it does.

Earlier I had wrote about a form of government I labeled as dedomenocracy and described as governing by data and urgency.   In particular, to make laws only when urgently necessary and then retiring those laws the moment the urgency has passed.  Underlying this concept is the appreciation for the necessity to gather new data that is uninfluenced by imposition of arbitrary rules.  Underlying that appreciation is the sense that the Truth is a living thing, never staying exactly the same.   The data reveals that.  Perhaps what the newer data reveals is a more accurate picture of the unchanging single truth, but that just means our earlier understanding was based on more flawed data.   Our understanding of the Truth is changing because our data is better, or it is because Truth itself is changing.   I don’t know how to tell the difference, but I’m more inclined to believe the later.

I’m reminded of stories about ancient civilizations such as the ancient Romans.   They managed to build large structures of materials that are still standing today, and in some cases are even in use today.   They must have got something right about how the world works.   Also, I know that modern humans populated that civilization.   I don’t think I can ever understand what it would have been like to live and work in that civilization.   When I read books or watch documentaries or fictions about that period, I’m always left with the impression of our imposing a modern experience into the constraints of the period they lived in.   I am convinced they experienced a very different world, a world forever outside of our ability to comprehend.   That’s just how I think about it.

Clearly the Romans left a legacy that affects our lives today.  They left hard evidence of their existence through structures.  The left historic evidence of laws, religions, philosophies, and language.   They left soft evidence of how later periods responded to the conditions they left behind.

I look at history as an illustration of how the entire universe works.   The reason why things are the way they are today is partly a vestige of earlier events kept around either because they are still useful, or there is no great benefit to justify replacing them.   We can describe these lasting influences as physical laws and be correct because there is a lack of sufficient motivation to replace them.    Just as in human history, there sometimes appears a change (whether an innovation or a catastrophe) that justifies replacing something old with something newer.   When that happens, it is necessary to adapt with that change.   To do that, we need new data and we need to interpret that data unhampered by old data, especially since that old data measured an aspect of the universe that no longer is true.

From my thinking, the world works from a different model of the clockwork type description of modern science.   Instead the world works through a set of established rules agreed upon in response to past negotiations following some innovation or catastrophe.   The different parts of the way the universe works will continue to work that way as long as the other parts cooperate and the evidence is that there is little reason not to cooperate at least in time scales humans can experience.

The various parts work relatively independently of each other.  In an earlier post, I described the independence of time scales where nanoseconds don’t listen to milliseconds.   Milliseconds are at the mercy of nanoseconds but they don’t listen to seconds.   In physical terms, the protons within the atoms within my body will survive into tomorrow no matter what I chose to do.

This is a model that comes from my practice with data.  When I have a problem to solve, I could approach it by deriving some one-step software program (perhaps consisting of many algorithms in succession) to apply against the available pool of raw data.   Instead I end up with an assembly line approach that I arrive at through iterations developing each step and returning to earlier steps to make the better.   The eventual solution has a customized approach to consume the rawest of data but that approach produces a new persistent pool of refined data.   Later processes work only on that refined data to produce persistent pools of even more refined data that eventually supports the final product.   Once working I can visit each of these steps like different factories in a supply chain, inspecting their operations, testing their stockpile of products for what they should do with what they have.

This is how I do my work.   It seems naturally that this is influencing how I see how the world works.  If had been an engineer building spaceships for decades-long-missions, I would see the world differently.   Instead I work with hand-me-down data recycled from other processes that change without any regard to what I am trying to do.   In this data world, those processes are the truth from which the data comes, that truth is always changing.

I can’t help but to think that perhaps this is how the entire universe works.  There is an ecosystem of cooperating parts working they way they are as a consequence of some type of negotiation to do what they can with what they can get from upstream suppliers in order to satisfy their downstream consumers.    What we call physical laws is actually a very stable economy for now.   I wouldn’t be surprised if it changed tomorrow.