I started this morning posting this update to my LinkedIn account:

A curious thing about my listing of skills in my profile is that I did use the listed skills productively in my last job and in many cases within just a few days of starting that job, but I had nearly none of those skills on the day I was hired. To put them into the profile is to say here I am ready to do what I did before, but this misses the real skill/talent of what really made a difference: I learned immediately the skills necessary to get the present job done without forcing my prior experience on the new opportunity.

My formal college education was in engineering.   I chose engineering as a compromise between my desires to do science and mathematics and yet have something that had some flexibility of where I could work.    I made an effort to take advanced courses in physics and mathematics as optional electives in my degree program.    I don’t recall hearing the term at the time, but it seems I had all of the bases covered in the current term STEM: science, technology, engineering, and math (apparently current usages says they are distinct).

I had a heavy course load and for some classes I rebelliously under-performed so I didn’t get the highest of grade averages, but I did pretty good.   My personality is uncomfortable with a top-of-class designation.   I’m most comfortable in the A- territory.   That minus sign is very meaningful.   It means I tend to rebel.

Upon graduating undergraduate school I was strongly encouraged to continue on to graduate studies with a promising path to getting a PhD.   By that I mean, I received encouragement appropriate to an A- student: they acknowledged that I would have to shape up a bit more to be a successful candidate.   They simply suggested it was something I should strongly consider.

In hindsight, I should have taken their advice but I didn’t.   I went off to work, picking one of a half dozen offers all of which were at least a 1000 miles away (so much for flexibility of where to work) but I was young so I gave it a try.   I showed up to work and settled into an interesting job in STEM.

My first task, if not my primary task, was to start shopping.   As a defense for sounding sexist, I was basing it on depictions in popular situation comedies of the time, but I immediately thought of why there aren’t more women in engineering because of the utmost importance of shopping to getting things done.

Shopping is an under-appreciated talent and skill in the application of technology.   If you want to get something done, you need to be able to obtain resources.   Unless you have the luxury of building everything from raw ore extracted from the earth, you will need to pick products from catalogs of competing vendors no two of which offer exactly the same item on the same terms.

Shopping involves a lot of show time.   This may involve travel to sales conventions or vendor show rooms.   Often it involves sales people arriving with samples an promotional presentations.  Most frequently it involved browsing catalogs and calling a number to ask questions and establish a relationship with an assigned sales person.

In the pre-Internet era, the catalogs were printed and often woefully out of date.   These were often thick catalogs and so they were expensive.   I recall a couple times when our request for a catalog was denied because our company had a history of asking for catalogs but never making orders.  In our case, we were in the process of moving into a new area of technology and in my particular case I was on a project that finally needed this technology.

To get the catalog I needed, I had to spend a lot of time talking to the salesperson and even arranging to meet him in person.   My point is that I hated it.   It is laughable now, but I really felt abused to have to devote so much energy to shopping when my skills were in synthesis and analysis.   I just wanted someone to send me the things I calculated that I needed.

That’s the problem, ultimately.   No one makes the precise things I calculated I needed.

In this particular first job experience, we were tasked to produce a one-time test bed to demonstrate the concepts of what we will later engineer into a product that can be manufactured.   In this particular case, we needed stuff that we could obtain quickly.   This stuff had to already have been built, in a currently supported product line, and can be obtained from stock instead of requiring a contract to a manufacturer to build to specification.    As I’m writing about this experience from 30 years ago, I realize the situation was a little more nuanced than that.   Some of what we needed had to be built from specification but it had to come from a shop that was willing to do small quantities without a formal commitment for larger quantities later.

As much as I hated it, shopping around, seeking and comparing obtainable products, was an essential part of the job.   This was not merely a nuisance.   The success of the product required excellence in shopping skills.   Excellent shopping skills demand the performance of shopping.   Shopping means seeking out all of the available vendors, not just the first one to answer the phone or the one that sent the only catalog you can find that a peer had at his desk from 5 years ago.   Shopping meant studying the specifications of the offered products to understand everything about them, not just the specific features required for the product.    Shopping meant paying attention to features that the vendor insisted were important but wasn’t important for this project.  Shopping meant learning that in fact what the vendor was selling was important after all.

The most central fact of shopping is to make a shopping decision.  A shopping decision is a decision where there are multiple legitimate choices each with strong considerations for and against their purchase.   Those considerations went beyond the original requirements such as whether the vendor is one that we want to build a lasting relationship with.  Such considerations may be irrelevant to the immediate technical needs, but are important in case we decide we actually like the product.   We need to know whether we can do business with this vendor for the larger but not too large quantities we anticipated needing.

In hindsight, this is just the nature of the engineering business.   I’m deliberately highlighting the notion of shopping because I think this is very key skill and talent.   Excellent shopping skills are key to successful projects.

Many people drawn to STEM are terrible shoppers.

I recall my undergrad experience when digital calculators were first becoming easily accessible and personal computers (8 inch floppies) were just beginning to offer an option to waiting in mainframe lines holding stacks of punch cards at midnight when CPU time was available to undergrads (I’m proud to have been in the last class year required to do that!).    The point of the computing capability was to be able to compute with great precision the exact quantities needed for each component to have a perfect solution.  We can get the math to work out because we had increasingly easy access to math machines.

Some of our labs involved variable components that we would adjust and then measure to confirm our calculations.    Underlying much of the undergraduate education of the time was this sense that precision and specificity were not only possible but mandatory.

My undergraduate degree was in electrical engineering where many of the early exercises involved electrical networks of resistors (later impedances modeled as complex number valued resistors).   For design problems, the goal was identifying the precise values for each component to get the desired results.   We did that and got good grades.

If we did a lab, we might use variable resistors, capacitors, and inductors along with meters to measure the precise calculated values and then run the experiment to confirm or calculations (or inevitably go back to explain why the experiment fell short of expectations).   These were simple experiments and the lab period was generous so there was a luxury to tweak and measure and retweak.   The component counts were small and variable devices were readily available because we reused the same ones the previous group used.

This lab scenario had no resemblance at all to most real jobs, except maybe for the very rare pure research labs (that I once hoped would be available to me).    Real jobs do have labs for prototypes.  But real jobs have budgets both in money and time.   Real jobs generally don’t have a stock of readily available components needed for the current innovative project.   Even in such prototyping labs, the key to success was shopping for available and affordable components.   Available components do not exist in a continuum of values.  There are very limited number of distinct values available.   Nor are available components precisely accurate, their values have tolerances, and often very generous or loose tolerances.

Even for the design of the manufactured products, we had to adhere to standards for defining values and tolerances.   These jobs did require the STEM skills to adjust the design based on what is available.   However, these jobs required shopping to understand everything that is available from any source and to obtain solid information about each option to support a defensible decision.

Shopping is a key skill to successful engineering.

My personal story is that I hate shopping.  I hate it to this very day.  For example, when I go clothes shopping in some mall I go straight to a store I know I found something I liked before and in that store I got straight to the section that has the brand or style that I liked before.   Often if I find an exact match to what worked before, I grab it and get out.  The entire energy inside me is to get it over with as soon as possible.    I will shop in a true sense of the word only if the store goes out of business, rearranges its floor, or cycles through to some other brand.   Even then I will not linger after finding something that is good enough.

I admit that often I will not be completely satisfied with my choice.   I take it as a fact of life that I’ll had to alter something or adjust something else to make it work for me.   An example occurred recently when I quickly bought something comfortable to wear around the house.   The outfit was the right feel, the right chest and waist size.   The problem is that the sleeves and pant legs are mysteriously 4 inches too long.  I resigned to the fact that I have to roll them up.  I’m not going to alter something I only wear in the house.

Despite the essential importance of shopping skills for successful projects, I don’t recall ever seeing a job listing that points out the need for shopping skills.    Either hiring managers take for granted that people can shop, or (more likely) they don’t appreciate how important the ability to shop is to their tasks.

My career deviated from hardware and landed solidly in software by way of simulation of physical systems.    In the current software environment, it is almost overwhelming that there are so many different options.    For any particular project there are many different technologies that can do the job and many different vendors with variations of those technologies.   Even when you select a particular software vendor and technology, their libraries provide many options to tackle the same job.

My personal experience is my amazement at how many different ways I can write an SQL script to perform the same job.  Each approach does get the basic job done, but each has it own good and bad points in terms of load on the database, speed of operation, ease of maintenance, or number of users that can be supported.

One example is the SQL cursor that allows building a loop to apply some logic to each item one at a time.   For someone trained in procedural languages, this is an obvious choice if the job must be done in SQL.    I learned to avoid cursors entirely.   Elaborate SQL select statements with temporary tables, views, or common table expressions can almost always do the same job.   The advantage of the select statement approach is that it gives the query planning engine more clues for building an efficient strategy to select and process the minimal records needed to supply the results.   The cursor approach (or equivalent in procedural languages) forces the database to deliver the entirety of results to the script and pace the delivery to match the loop execution speed (although I recognize there are techniques to do this efficiently).   My point here is when focusing just on one language (SQL), there is a need for shopping to identify and diligently compare different approaches for solving the same problem.

This shopping for options is very apparent when considering modern programming languages with access to large number of libraries where each library contains multiple options for doing roughly the same task.   The best work product will come from the designer who is aware of all of the options and picks the right one for the job.    Being a good developer in part means having good shopping skills.    A good developer does not grab the first option that he finds and then does what ever is necessary to make it work for him (like my clothes store experience).

One of my observations when interviewing people is that a claim for understanding a particular technology often will consist of a long period of applying just a small subset of that technology that was discovered very early in their introduction to that technology.   (I hasten to add that this is often a consequence of how jobs are managed where we excessively compartmentalize individuals.  If a programmer becomes good at using the communication socket and protocol libraries, he may spend all of his time working on those types of projects and never have a need to use the other unique features of the programming language and its libraries.)

I also observe that despite the plethora of options of other technologies that can be better suited for a task, there is a tendency to use what one already is comfortable using.   This can be a sign of individual strength because he is able to master a technology to successfully deliver a result despite that technology requiring more effort to get the job done.    However, it can be a lost opportunity of the team or employer to take advantage of something that is simpler and better suited to the task.

The main motivation for this post came from a comment I heard from a recorded YouTube tutorial on machine learning libraries in a particular language.   The language library was designed so that a wealth of machine learning algorithms follow a common structure.   As far as programming is concerned, once you know how to invoke one algorithm, you pretty much know how to invoke the rest.    The key comment concerned the question of which algorithm do you use for a particular kind of data.

This question cuts at the very core of what I’m talking about.   There are many different algorithms, assumptions, and even broad approaches that all fall under the umbrella of the term machine learning and predictive analytics.   Most options have a great deal of research and experience that define the type of problems they were designed to tackle or where they have been found to be most appropriate.   The challenge of becoming a machine learning expert is not being able to call the algorithm from code.   The challenge is being aware of all of the options and their relative merits.

Most jobs have a specific type or nature of data so that they can select one or just a few appropriate algorithms.   Often these options are already selected before a candidate is hired.   The job is to apply one of these algorithms.   The challenge to the organization occurs when they need to address a new type of data or the nature of their old data has fundamentally changed due to natural evolution of data.    Do these organizations have the right skills for shopping for the best algorithm to match these new needs?

The comment from the tutorial instructor caught my attention.   In short, he said there are too many algorithms to learn.   Because the library design provides consistent methods of invoking each algorithm, a useful strategy is to apply the algorithms one at a time to a sample data set until you find one that seems to work well.   Then you study that selected algorithm’s documentation to learn about it.

The presumption is that the developer will read the apparently well-suited algorithm’s documentation with diligence to be sure it is appropriate for the data.   That  presumption is that the developer is an excellent and discriminating shopper.