In this public release paper from MITRE, they describe a tool they developed that adjusts windows of time to allocate for certain operations on the ground and immediate airspace of an airport.   In particular, this tool strives to reduce the already rare occurrences of near collisions of arriving and departing aircraft through the use of simulations.

Initially the story caught my attention because I had done some work for the same group at MITRE in the early 1990s where I used simulations to simulate higher altitude terminal airspace near collisions in order to evaluate and refine the algorithms used in the TCAS (Traffic Collision Avoidance System) avionics.    At the time I developed and ran software to simulate millions of encounters to assess the effectiveness of the algorithms.   However, a fundamental difference about the more recent announcement concerns the velocity of the simulation.   The newer work appears to run simulations to adjust arrival/departure windows in real time while my earlier work was for presentation to committees to evaluate for decisions of future recommendations.

In recent posts (recent one is here), I expressed a concern about the big data projects obligating decision makers to adopt the automatically generated recommendations.  To get the best benefit from the recommendations, the decision maker needs to use them as soon as possible.  However, the volume and velocity of data renders the recommendation beyond human comprehension.    To make these decisions in a timely manner, the decision maker has to place faith in the correctness and more importantly in the defensibility of the recommendation.

I suggested that a conscientious decision maker will demand human comprehension of why the recommendation is superior for all other options because the community will hold him accountable for any unintended negative consequences.   My conclusion is the inevitability of eliminating the human accountable decision maker in order to enjoy the benefits of the sophisticated algorithms acting on high volumes and varieties of data.   In other words, we can not simultaneously enjoy the benefits of big data prescriptive algorithms and the benefits of having an accountable decision maker who will be able to defend the decision in light of unfortunate consequences.

The above paper suggest a continued assurance that we can have both.    The simulations run on current conditions to estimate the best windows to use.   To enjoy the maximal benefit, these recommendations will need to be adopted immediately.  In a few moments, the conditions will change allowing for fresher simulations to estimate a different recommendation.    Despite this need for immediate adoption, the authors describe the tool as a decision aid.   It is a suggestion for accountable decision makers who retain the option of rejecting the suggestion.

We continue to insist on holding the decision makers accountable for the decision despite the near impossibility of the decision maker being able to comprehend the merits of the recommendation in the time available for him to make the decision.   There really is no option for the decision maker.  He is obligated to follow the recommendation unless he can see some very obvious flaw in the recommendation.    That obligation comes from the limited time to act on the recommendation and from the fact that he will encounter even more difficult position of defending the consequences of failing to follow a recommendation from the computed recommendation.  It would be more realistic to describe the tool as an automated decision instead of a decision aid.

In my earlier blog posts, I described recommendations based on predictive analytic algorithms acting on big data.   Implicit in those discussions was that the big data involved historic observations.    The strongest case for trusting the generated recommendation comes from large data sets of high quality observations.

In much earlier posts, I distinguished different levels of data quality in terms of how much real-world information they provide.   I used the analogy of light to describe these levels.   Bright data was well-documented and controlled observations.   More commonly we have dim data that is not so well documented or controlled: we know we are observing something about the real world, but we have some doubts about exactly what that is.   Dim data results from things like uncertainty and errors.   At the other end of this scale, I described dark data that I defined to be model-generated data lacking any fresh real world observations.    Model generated data substitutes for missing observations similar to how we may interpolate a value that must exist between two observed values.    We may have a lot of trust in the accuracy and relevance of our models, but I still distinguish it from observation data because it remains our best guess about the real world instead of an actual observation.

We prefer to use real world observations to feed predictive algorithms that are based on statistical models that assume they are working with real data.    The problem of using model-generated data to feed predictive algorithms is that they may be self confirming.  Because the same understanding of the world informs both the model generating the data and the model used to predict future results, there may be nothing new to learn.    Using model-generated data instead of fresh observations will essentially teach us what we could have calculated far in advance with enough time to comprehend the results.

The above project describes a particular type of problem that addresses a very rare possibility of a near collision.   The operations of the airport are already tuned through experience and planning to avoid these events.   As a result, there are very few if any bright-data recorded near collisions to use to calculate a recommendation that refines a standard practice.  In particular, because of the rarity of these events in a managed airport, there may be no recorded events that match the current conditions.   A near collision will be a surprising event that we had not anticipated or experienced before.

In order to use a big data approach of large volume and velocity, the above project resorts to dark data.   It simulates millions of scenarios based on some observed current condition to estimate a refinement of the standard practice to further minimize the risk of an already rare occurrence.

This project that looks like a big data project with its promise of high velocity but it uses simulated data instead of observed historical data.   Given this appearance, we are led to the same inevitability of obligating the decision maker to follow the recommendation and thus delegate his accountability to this algorithm.   I find this to be even more troubling than the similar delegation based on historical data, because these recommendations come from predictive analytic algorithms using predictive simulation algorithms where both are based on the same human understanding.   Such a recommendation is similar to dead-reckoning or flying blind.

Like dead-reckoning, the simulations project what might happen based on models extrapolating future conditions from some observed starting point.  The resulting data relies entirely on our understanding of the world.  In contrast, the same predictive algorithms run on actual historic observations (if they existed) gives us the opportunity to learn something new.

This project is about refining the operations of an airport that has the benefit of planning and experience to already virtually eliminate the possibility of a near collision.    Despite this maturity in operations, we recognize there remains the possibility of a near collision, but such scenarios will only occur in surprising circumstances.    When an actual near collision does occur, it will tell us something that was missing from our earlier understanding of what might happen.

Simulations that produce millions of variations of a scenario will use randomness to explore the possible extremes of various combinations.   Simulations can come up with surprising results to the operator but these are surprises only in to the experience of that individual operator.   The fact that the simulation was able to produce this result is proof that the result was predictable from prior understanding.   It does not provide the same kind of information that we’d learn from an observation of an actual occurrence of a near collision in an environment that takes such efforts to prevent it.   A real instance of a near collision in that environment will surprise not only the operators but also the models.

I recognize there is some value to the simulated data approach.   Some rare near collisions may be predictable given the possibilities of human or machine failures.   Because these are predictable, they can be addressed far in advance to allow time to scrutinize the recommendations and make an accountable decision.    This is similar to how I approached the simulation analysis projects 25 years ago.  The results are prepared and advance, diligently analyzed, and presented to persuade a committee to make a recommendation for future operations.

This project promotes the expectation these simulations can provide something of value in real time.    A simulation can run based on the current conditions so that we can apply statistical algorithms to determine the best window size for the current situation.    This real time expectation parallels the high velocity expectation of big data analytic algorithms.    Indeed, these expectations may be evolving in parallel because there has always been a goal to make simulations more real-time.   However, I think the simulation technologies is encouraged by the recently promoted success of big data projects.

The big data analytic technologies can process simulated data instead of observed data and get similar kinds of results with the accompanying support of graphic visualizations.   It is technically feasible to populate big data with simulated data.   The same algorithms and technologies operate as efficiently with simulated data as with observed data.

The distinction concerns relevance to the real world.  The big data project based on observed data offers the opportunity of hypothesis discovery: finding something new about the real world we did not know before.    When based on simulated data, the same algorithms at best remind us what we already understand about about real world.  Rare events we want to prevent are very likely to result from something we did not previously understand about the world.

In the context of finding a way to eliminate an occurrence already made highly unlikely by mature operational practices, a big data approach using observed data is more likely to be effective than one using simulated data.    The simulated data may even introduce counter-productive possibility of making the event more likely by reinforcing misconceptions about the real world.   Those misconceptions are built into both the simulation that generates the data and the algorithms that generates the recommendation.

The danger comes when we impose on the decision maker the same expectation of inevitable obligation to follow high velocity recommendations from simulated data that we expect from decision makers using big data systems that use observed data.

The fact that the simulation can run fast enough to be relevant in real time is proof that the simulations can be run quickly.  The big data technologies allow for storage and analysis of all of the intermediate values in each simulated scenario.   Those same simulations could be run far in advance feed big-data type technologies to allow for time for diligent scrutiny of the results in order to support valid human accountable decisions for recommendations.

We continue to place a premium on human accountability.  With simulated data, there is less of an argument to eliminate this accountability because it can be performed in real time leaving no opportunity for human scrutiny.   We could have studied the simulated results of the same scenario long in advance.