In my last post, I described story telling as a professional skill for someone who needs to persuade someone of a concept neither party comprehends.   In particular, in the profession of data science, we seek talent who combine technical know-how for selecting and assembling algorithms with the non-technical story telling skills to persuade an audience that the incomprehensible recommendation is worth following.

In that post, I criticized the notion of story-telling as something a decision maker can detect and dismiss as irrelevant to the actual problem.   In many recent posts, I emphasize the need for accountability for the consequences (even if unintended) of a decision.   The decision maker is in that role to defend his decision after the consequences of the decision become historic fact.   The best decision makers can anticipate possible criticisms of his decision.   Anticipating a criticism or grievance is distinct from anticipating the actual consequences.   Even when the decision process failed to anticipate a certain outcome, the decision maker can still anticipate how a population may react and what kind of explanation they will demand.

A great political leader, for example, understands the diversity of viewpoints of the population he leads.  He uses this broad understanding to build and maintain a super-majority coalition of support for his leadership despite differences of opinions within the coalition.   This understanding informs his anticipation of the types of criticism he will need to defend against in case a decision does not work out well.  A great leader is a great communicator who strives to find the right answers to the tough criticisms.   Even if these answers do not satisfy his critics, they can silence the critics by building a super-majority coalition of support that overwhelms the critics.   An effective leader will craft a response to best answer a particular criticism, but the successful response often will draw upon his learning of the rationale of the recommendation before making the decisions.

I believe a similar dynamic of super-majority coalition of support applies also within organizations and within markets, although it is much easier to describe the public political example.  This discussion itself is an example of story telling.   Running a business or operating in a market is very complex and involves a lot information that is not well known or well understood.   My suggestion to compare it with something more familiar, a political process, is an example of the rhetorical device of metaphor.

A metaphor is story telling.   Story telling is a substitution of a difficult to comprehend concept with a concept (the story) that is easy to understand because we designed the story around the understanding we want to convey.   In other words, the story’s truth is easy to demonstrate because we built the story to be consistent with that truth.

A metaphor is a rhetorical device.   In earlier posts, I complained about modern education’s lack of study of classical rhetoric.   In particular, in the STEM (science, technology, engineering, math) fields, we tend to draw a distinction where rhetoric is largely outside of the discipline.   This is especially true for informal logic that persuades through devices such as story telling.    STEM is about precision and accuracy.   The truth is not subject to persuasive argument that appeals to emotions including the audience’s opinion of the speaker.  As a result, people who have STEM education generally have limited skills in rhetoric especially in the informal logic of persuasion.

To the STEM practitioner, trusted algorithms used on trusted data can only produce incontrovertible results, even if the results are beyond human comprehension or independent verification.   A good example is my favorite example of cosmology’s inference of dark matter.   The combination of highly trusted data from astronomers with highly trusted theories of physics produces a conclusion of missing mass.   The matter that accounts for this missing mass is incomprehensible but we are lead to conclude it must exist.   This concept became widely accepted after someone came up with the name of dark matter to describe it.   The mere labeling this incomprehensible stuff provided the metaphor needed to be persuasive. In fact, we do not know anything about what can account for this missing mass, but we are sure it exists in some kind of undefined form.

I think this example of inference of dark matter is a good example.   It is the result of complex statistically correct algorithms applied to a large volume of data.  Our trust in the algorithms and the data leads us to trust the incomprehensible result.   Except for the cost of labor to perform additional research inspired by this finding, there is little at stake with this conclusion.  The concept of dark matter applies to galactic scale observations that have little relevance to human condition.  There is no prospect of humans being able to have any influence at this scale.    With so little at stake, the general population readily accepts the dark matter story.

Replacing incomprehensible results with easy to understand stories is a rhetorical skill.  Although most astronomers and physicists accept the story of dark matter as a legitimate topic of study, it took a special talent to come up with the metaphor in the first place.   We require a similarly special talent from data scientists.    In contrast to the dark matter story that emerged over many years within a very large community, the data scientist must work within a very small group (or in a solitary role) and invent new stories quickly, perhaps as frequently as daily, in order to persuade a decision maker to follow an incomprehensible recommendation.

The basic problem is defending an incomprehensible result coming from trusted algorithms acting on trusted data.   The incomprehensibility is absolute.   The very goal of big data is to discover results that are incomprehensible to humans, at least in the sense of being able to independently derive the same conclusion.   As I discussed in the last post, the presumed beneficial recommendations from big data analytic are worthless unless we can convince the decision maker to act on the recommendation.   In that post, I suggested we may have no choice but to automate the decision making process and eliminate human accountability.   This last resort becomes necessary when the data scientist is unable to invent a persuasive metaphoric story.

In recent years, there has been a recurring complaint that despite high unemployment numbers, industry has a persistent shortage of appropriately skilled labor.   The evidence comes in the form of many job openings that remain unfilled for very long times.   There is also evidence of a large number of unemployed with relevant training but apparently their skills are juts not good enough to fill the challenging positions that exist in large numbers.

Often this is mentioned as a failing of the educational system to prepare appropriate skills to fill these abundant job opportunities.   The educational infrastructure has no shortage of critics.   As a result it is rhetorically very effective to blame the labor shortage on the education system.    I don’t think this is fair because despite the number of unfilled positions, most of these positions are highly unique and specialized.   For most of these challenging positions, a person well qualified for one of these positions would be unqualified for any other.

It is unrealistic to expect a continental or global scale education system to prepare solitary individuals to fit specific job openings that do not exist at the start of that individual’s training.   Demanding an education solution results in labor bubbles such as what is emerging today in the data science field: a huge number of data scientists being developed in educational systems because there has been a lack of those skills when the field was more challenging.  I am convinced we will soon hear of complaints of large numbers of unemployed data scientists.   The educational system is very good at mass producing talent.  This talent may in fact be very relevant.   The trouble is that there really are not that many openings for this talent.

Even when we fill all of the open data science positions so that we end up with large number of unemployed (and unemployable) data scientists, there will remain a complaint that there is a huge number of skilled jobs that remain unfilled.

The explanation of a failed education system was a story that we invented to explain this disconnect of labor and the market.  The educational scapegoat story was persuasive.   Everyone has experience in the educational system.   There is no shortage of critics about the failings of the education system.

There may indeed be a partial explanation of inappropriate training from educational systems.   The recommendation to improve the education system to better prepare modern skills was valid.  Using the story of a broken educational system was effective in convincing us to adjust our educational systems.

The problem with story telling is that it introduces invented evidence into our future thinking.   The reason for the disparity of many unfilled positions and many unemployed is hard to comprehend.   It is easy to comprehend an educational system explanation to persuade a change in educational policies.   Unfortunately, the disparity will remain but now we have in our evidence that education is part of the problem.  We will continue to challenge educational systems to mass produce talent for what in reality will be just a few openings.  The danger of story telling is that it can distract us from other opportunities to address the problem.

The problem with story telling is that it become evidence for future decision making.   In analogy to big data solutions, the story that persuades adoption of a previous recommendation will become part of future algorithms to help make future recommendations easier to adopt.   The success of the story encourages us to incorporate the story as evidence or algorithms for future recommendations.   Prior story telling inevitably influences future recommendations to the point where we can reach a point when we begin to question the trust worthiness of the algorithms or data.   Like astronomy’s dark matter, we will perpetually treat a story as fact.

This article illustrates the above result.   The article defends the complaint that industry can not fill challenging positions.  Despite high unemployment, there are many job openings left unfilled because there is no one available with the right skills.  Underlying this argument is the assumption that industry is a victim of some outside failing.   The prior story telling of failed educational systems have become incorporated into the data and the algorithms for explaining the labor market.   This has reached the point where industries are helpless and innocent victims of an outside failure.

The data and models for the labor crisis point to causes outside of the industry. There is at least the possibility that the crisis in labor is a direct result of industry practices.  Industry created this problem.  Industry can fix this problem.

We often point to independent and free enterprise as the most efficient method of innovation and the source of most of our wealth.  Such brilliant creatures can not be powerless when it comes to filling its most challenging labor positions.  I find it incredible that we readily accept the notion that industry is a helpless victim of a labor pool that is unsuited to its needs.

We need to strip out the notion of educational failing from the data and the algorithms to explain the mismatch of labor to jobs before we can observe real solutions.

I really doubt the modern job requirements are unusually challenging in historical terms.   When we look at the early years of industrial and scientific revolution of a couple centuries ago, we see many people performing brilliantly with new concepts that were very challenging to them.   Although today we only remember a few names, the era experienced a broad participation in challenging skills across the entire population with very little if any specialized formal education.   As I discussed in my earlier post on observations from a recent reading of Melville’s Moby Dick, there was a popularity of amateur science that rivaled today’s popularity of social media.   The amateur scientists, even those who are unknown today, made many good contributions to scientific knowledge or inventions.

For a good part of USA’s early history, there was a culture of talent that obtained specialized skills with the need for formal education.   We developed this talent through a process of challenging promising individuals.

I think a turning point occur about the time of the publication of the Peter principle that stated that people get promoted to their level of incompetence.   I think the book was more descriptive of a new intolerance of incompetence than a prescription for celebrating competence.   At about that same time there was a sense that failure can not be an option.   In other words, we need to fill positions with people who are previously proven to be competent for the task.

Take a simple retail example of asking a floor associate where to find a particular item.   If that associate can not give exact and correct answer, we declare that individual to be incompetent unworthy of occupying that job.   The result is that those job openings can only be filled by skilled floor associates.   They need to be prepared before they can be hired.   It is the job of schools or certification trainers to qualify staff.   Once on the job they indefinitely will perform the same work that they were hired to do.

Our culture no longer tolerates the incompetence that happens when we promote a promising individual into a new challenge.   This intolerance ignores the human capacity to grow to meet to new challenges.   It is human nature to respond positively to challenges by striving to be excellent in a new job.

I believe the earlier period of history approached staff development in a way that observed competence as an opportunity to tackle a more challenging position.   They actively promoted competent individuals into positions where they would be incompetent but with promise that they will quickly become competent.    Back then, demonstrated competence was evidence of capability to take on bigger challenges.   A more challenging position would not remain unfilled if we observed competent staff already on our team but who were performing some other unrelated task.    Competence was evidence of future competence and made the individual a prime candidate to assign to a more challenging role.

Today we approach our workforce differently.   In particular, we interpret competence as evidence that the person and job are well matched.   We strive to preserve competence by keeping the competent staff happy and keeping his skills current to the needs of that specific skill.   Our goal is retaining competent people in their place of competence.

Our admiration of competence is so strong that we narrow specialties to make it easier to celebrate special competence.  In software engineering for example, we recognize competence in the use of one specific language through tools from one specific vendor that runs on one specific type of computer.   Once we identify the competence, we encourage longevity with recognition awards, bonuses, and salary increases.   The modern ideal is for every position to be filled with competence and to stay that way as long as those positions are profitable.

The difference between modern and older practices is that modern practices seeks to preserve competence while the older practices sought to create new competence.    Today, competence is evidence that the staff is in the right job.  In the past that same competence was evidence of being prepared for a new challenge that results in initial incompetence.

Perhaps the difference between present and past practices is more subtle than that, but I do think that there is room for improvement within modern businesses to solve the problem of unfilled skilled positions using their own resources.   Most organization have a large supply of highly competent people who are performing tasks that are delivering real value.   Instead of exploiting this productivity benefit by constraining the talent to an area of already proven competence, companies can elevate this talent to fit the unmet needs.

Companies can fill challenging unfilled positions by identifying a competent staff and place him in a role where he will not be as competent.  Where it is possible, this can be a direct assignment with a tolerance for some initial incompetence for the interim while the individual is learning the new challenges.   Alternatively, the company can place the person into a training program to meet the minimal competence requirements for the new position.    The experience of seeing competence in the earlier job provides confidence that this investment will be worthwhile.

There may be an objection that moving someone from an area of profitable competence can result in a loss of profitable performance from that group.    An answer to that objection may be the observation of the complete unrealized profit of the hard-to-fill position that may in fact be more profitable once filled.   Also, the company already has evidence that someone can be competent in this current position so they can identify someone elsewhere who can learn with the benefit of the availability of an experienced mentor.  In other words, a competent worker for a task is evidence that it is feasible to find another competent worker to take his place.

Another objection is that the company is too small to have that kind of skill diversity.   Certainly, the above recommendation works better in large organizations (many of which are complaining about the skills gap).    If this approach were widely practiced in the 18th and 19th century, it occurred in an environment dominated by relatively small businesses with very focused needs.   The same challenges existed then as it did now.   I believe they approached meeting this challenge very differently than we do today.   In particular, they did not look to schools or certification training programs to fill unmet needs.  How this worked out may involve an implicit cooperation of different corporations that benefit by mutually developing talent that would be useful across corporate boundaries.   Perhaps, a regret in losing a valued competent person would be compensated by obtaining a more person competent in a much more valued role.

I am just speculating on the mechanics, but I continue to believe that earlier eras had more impatience with demonstrated competence than we have today.   Assume that someone is competent in one area.  The earlier era would perceive this as an excellent candidate to take on a more challenging role as soon as one becomes available.   The present era perceives this as a perfect fit that should be preserved indefinitely.

I’m describing a cultural level concept for internally developing talent throughout an organization.    In contrast to the culture that sees itself as a victim of ineffective external educational systems, the alternative culture exploits its own opportunities to build talent internally.

Solving the skilled labor shortage may simply involve a shift from celebrating and conserving competence to identifying talent to build future competence.

The skilled labor shortage comes from the story telling that external educational systems are failing the deliver desired skills.  We introduced the story telling initially to motivate a recommendation to improve education.   Unfortunately, the story became part of the evidence and data so that future decision making considers education (external skill development) to be their primary challenge.   The story obscures other opportunities that they can pursue, such as identifying promising talent to train for a more challenge and difficult challenge that remains unfilled.

Update 9/6/2014: A very similar observation on Washington Post site is here.