In my days as a low-level manager I occasionally needed to hire new staff. Based on various constraints, I had to choose staff specifically with technical degrees. But my work dealt with historical data. While it is true that there are technical skills required to write software, to understand server capabilities, to understand the technologies for collecting and assembling data, the real challenge was in making sense of historical data. By that time, I had already met several people with non-technical degrees who were good at working with historical data and basically figured out the technical stuff as needed. These people had degrees in things like economics, history, sociology, and the like.
Although I was not aware of the term at the time, today we refer to the technical degrees as the STEM programs of Science, Technology, Engineering, Math. Allow me to gripe a bit in saying the Technology only makes sense because someone wants to elevate the term engineering. Technology and Engineering are the same thing especially today when most engineers are mostly reduced to being technologists. Also, the urge to make a cute acronym results in a misconception. A common advice today is if a young person wants a career that will be easy to find a job and make good money then he should choose one of the STEM fields. Easy to remember, but misleading. It is extremely hard to find jobs in pure science and pure math, and for the lucky few that do find those jobs, they are more likely to find themselves working very long hours for very little pay. There are exceptions, of course, but the majority are working more than 40 hour workweeks with few benefits and with low pay. The market demand is in the TE part and as I mentioned are really the same thing. I would abbreviate the college-program recommendation for good work and good pay to be E/MS (engineering over a grounding of math and science). This is where the jobs are.
Today the distinction of careers are STEM and whatever term is used for non-STEM. We still use the older terms of hard science and soft science. As someone aspired to and trained in hard sciences, I learned the prejudice that hard is better than soft.
But now with some experience all these labels are misleading. I see three areas of academics. Science based on the present (operations or experimentation), Science based on the past (making the best conclusions of historical data), and Science based on persuading other people (broadly Rhetoric, but it includes various *-studies).
Hard sciences or STEM focus on the present. They are primarily characterized by observation, experimentation, building and maintaining systems that provide the tools for present events. I agree that they also encounter duties involving historical data and persuading others, but their stereotype is that these are their strong skills.
Soft sciences are essentially historians in the broadest sense. These are the sciences that are constrained by the available record of the past. Practitioners include historians, archaeologists, forensics, criminal investigators, medical coroners, financial auditors, etc. They need special skills in working with historical data. They need skills in finding historical data (research), handling that data, preparing that data, evaluating the relevance of the data, and most importantly interpreting the data’s contribution to the questions they are seeking.
Allow me to set aside the persuasive sciences for another post.
I want to focus on the divisions of science based on tense: the present is the realm of the hard sciences, the past is the realm of the soft sciences. Since my focus is on this time aspect, allow me to use use the terms present-science and history-sciences.
The skills and training of these two sciences are very different. The laboratory symbolizes present-science while the library symbolizes the history-sciences. One of the more important distinctions is the difference in temperament. History science is inherently more patient than the present science. History science tends to attract more patient people or people are comfortable being patient. In contrast the present science is all about racing the clock whether it is the proper running of an experiment or having limited time and money budgets: the emphasis is on delivering their product quickly and decisively.
My previous post is an illustration of a task of the history scientist. In particular it is about finding missing data where the emphasis is on finding the most definitive data no matter how long it takes to find it. A past event is frozen into the historical record as material or circumstantial evidence. The task is to exhaustively search that historical material to best fill in the missing data. This requires discipline, temperament, and a passion for getting it right.
Back to my dilemma as a manager. My project involving data processing ideally requires both present-sciences (STEM) and history-sciences. At the time, I was lacking both but I only had the opportunity to hire one person. In the ideal world, that person would do his thing and the other thing would be left unfilled. My plan was to get someone on board and then to encourage him to grow into other vacancy. Also in reality, I was not given a choice: I had to find someone with a technical degree, a present-scientist.
It is hard to find a person who works well with both present and history sciences. Certainly my own experience taught me that it is very difficult to train on-the-job a present-scientist to do the work of history-science. I did encounter some history-scientists who were able to learn on the job the present-science, but they were not very good at doing it in a timely and cost-effective manner.
My project worked with historical data. The issue with historical data is the demon of the missing data. Lacking time-travel, the available record is frozen in history. But missing data is a serious problem. As I described it earlier, missing data can become dark data: made up data that we treat equal to supported data in order to allow us to move on to other work (as is our nature as impatient present-scientists).
Dark data is very dangerous. Dark data can never be trusted. When it appears, we need to treat it with perpetual suspicion and continuously subject it to challenges. Basically the domain of history-science.
The conclusion is that what I call data science is requires in equal parts the history-sciences and the present-sciences. I called the object of my work “historical data”, I approached it as present-scientist where the project was simply another application of STEM. Ultimately, I was overwhelmed by the problem of missing data and mired in the problems of dark data. I unwittingly fell into the realm of history sciences.
I learned to appreciate history sciences.