In an earlier post, I mentioned a bad weather call a couple weeks ago in the DC area where the forecasters and their media counterparts suggested that a morning-commute snowfall would not be a problem and the resulting morning was a complete mess due to a dry snow falling on cold and untreated surfaces that might as well have been coated in ice. It was unfair that everyone criticized the city and the school officials for being unprepared. I recalled the forecast even the night before that seemed to laugh of this minor event. Given that all of DC’s roads are near capacity during normal rush hour traffic, road conditions can deteriorate quickly especially if everyone is encouraged to treat the commute as any typical winter day. The result was a complete mess.
Today, there is news of another forecast going wrong, this time involving New Jersey and New York with predictions of a historic and likely record-breaking snowfall. The city planners in NYC responded appropriately by shutting down roads and even the metro to keep people home and out of the way of this major event. Forecasters continued to predict a huge event even a few hours away but the result was that the bulk of the storm passed to the east.
I understand what it is like to see computer models predict an extraordinary event. It is hard not to get excited and to raise all of the alarms to draw attention to this once in a lifetime opportunity to show the value of the simulation. This is a chance to save the day and get credit for it. On the other hand, the alternative of being skeptical of the computer model is very scary because the model might just be right. Not giving the model enough credibility and having the city caught unprepared will be devastating to the career and perhaps even the profession as a whole. It is humanly natural to go all in on the forecast even if it predicted a magnitude of event that never happened before. I understand that.
Modern weather technology with exotic data visualization with real-time satellite and radar imagery can be overwhelmingly convincing. It is reasonable to trust all this technology to offer a dire prediction and encourage the city to prepare to the maximum extent possible.
Ultimately the forecast was wrong and we are supposed to just brush if off as a minor error. Even the governor of NY refuses to criticize the weather service for a bad call. These are just people doing their job and their job just happened to cause entire cities to be shut down for the better part of a day. Everyone needs to accept their losses that were a consequence of the bad decision, but the weather forecaster jobs were not inconvenienced at all and will continue to work uninterrupted.
My observation is that criticizing the weather service for making a bad prediction is fully justified. I accept the arguments that weather is hard to predict, that the models are imperfect, and that the input data is incomplete and also imperfect due to insufficient funds. However, the reason we invest in a weather service is to provide actionable weather intelligence so that city and emergency planners can apply resources where they will be needed but only when they will be needed. The benefit of the investment in weather service is to improve the operation of the city to avoid expenses of false alarms or the costs of being unprepared.
A bad forecast that causes excessive expenses or fails to prepare a city should have consequences. At a minimum, the failure deserves abundant criticism without apology for making that criticism.
I don’t see what would be so bad for firing senior forecasters in each of the meteorological centers supporting the predictions. Getting a forecast wrong is a good reason to invoke an early retirement or abrupt termination from the senior managers who are expected to have experience to recognize that the predictions were unlikely (too much concern in the case of NYC, and too little concern in our local earlier example).
Accountability involves acknowledging errors that were made. A reasonable remedy for errors by accountable decision makers is to remove them from their jobs.
In the case of senior weather forecasters, much of their jobs are highly automated. The skill of their task is mostly in the area of being skeptical about models and learning what models to trust and when to trust them. The models themselves run without a need for the senior forecasters to be present. Also, the field of forecasting has a deep pool of junior talent who could do the senior job as well as the current incumbent. In each of the organizations, a forced departure of a senior forecaster will provide a probably long-awaited opportunity of promotion for the next in line, and that in turn will cascade for lower level promotions as well.
A failed forecast is an ideal time to give advancement opportunities to the people who have been waiting for the opportunities. In my last post, I described my skepticism that the more senior professional will be any better than the next in line. For a healthy stable society, we need to take advantage of opportunities to make vacancies for younger people so that they can have their chance to contribute. I don’t doubt that there is a deep pool of very qualified weather professionals who are ready and eager to fill the vacancies of resulting from dismissing the senior-most positions when bad forecasts cause significant harm by causing the consumer urban and emergency planners to over- or under-react.
Hypothesis Discovery 
From FiveThirtyEight.com another view of the forecasting failure:
I think this is a very insightful observation. We are becoming too enamored of computer models and their impressive command of huge volumes of data, ability to run massive simulations, and talent to present very captivating visualizations. The job previously known as meteorology (a human science) has become modelogy (a human spectator sport).
The model spectators should be held accountable for failing to be human meteorologists. Their replacements will still have access to the same models.
Pingback: Grammatic fallacies in data | kenneumeister
Pingback: Dedomenocracy in action: forecast and response to DC snow event of 2/17 | kenneumeister
On Feb 26, we had another snowfall (mildly inconvenient because it impacted morning commutes and caused delayed openings in cancellations) that lacked good forecasts until immediately prior to the event as summarized in this note:
The forecasters were able to recognize this event just a half day before it happened. It was a low impact event so there was little harm. However, I think that we had similar forecasting capabilities 50 years ago. Are we getting any return for the immense investment we’ve been putting into the weather forecasting industry since then?
On March 5, we had another snowfall. This one was significant but not as much snow as predicted 24 hours in advance, and at the lower end of the later more accurate forecasts.
Locally, the winter started with a small snow storm that caught everyone by surprise to result in a major traffic problem during morning commute. After that point, the area has responded aggressively to pre-treat roads for future forecasts and to get early start on preparing snow plows and salt trucks. There were at least two times when pre-treatment occurred for a storm that didn’t arrive and this resulted in a waste of salt.
The latest storm occurring late in the season needs more salt because of the unusually cold temperatures for this time of year. It turns out that the county is running out of its supply of salt.
A similar problem is happening with retail stores running out of ice melt for consumers.
This year has been unusual in the number of winter weather events that would deplete salt stocks. But there is a limit to how much salt is available during a winter season because the salt is shipped during the non-winter seasons and it is very difficult or impossible to ship salt during winter season when roads can be iced.
I mention this here because salt stock is an example of a resource that county managers need to manage carefully. In order to be prepared for a late season winter snow fall that are not uncommon for this area, they need the best forecasts so that they will only apply salt when it will be needed. The couple of times that the forecasts predicted a storm that didn’t arrive resulted in a waste of salt that was then unavailable when a more significant storm did arrive.
In addition, the unnecessary use of salt also has environmental consequences. Treating roads for a storm that does not arrive means that the ice will be washed away with a small rainfall that will result in run-off with a toxic concentration of salt entering our streams and eventually the bay.
Road salt usage provides a good illustration of the real costs and consequences of the quality of weather forecasts. Salt needs to be applied in advance of the winter weather and this depends on forecasts. When forecasts predict a storm that eventually turns out to not have required as much salt treatment, the county managers lose out in depleting their salt stocks and becoming less prepared for the next storm.
Weather forecasters should be held accountable for these errors in forecasts.
An update on Arlington’s road salt supply:
Being in a region that has experiences a lot of variability in winter weather from season to season, part of the planning is to mid-season resupplies instead of relying on a single supply for the entire season as suggested in my earlier reply.
Still, this shipment is out of the normal and requires additional expenses, especially for the need to rush the shipment to deal with a current snowfall. It raises the question of whether the salt that has been used so far this season has been used efficiently and how much salt was wasted in pre-treatment for forecasts that did not result in winter precipitation.
Pingback: Better context information needed to understand breaking new stories and crisis management | kenneumeister