Forecasting as (Fallible) Science and Art
Methods and Applications -Aman khan and KENNETH A. KRIZ
Recently I was at a holiday gathering. A person I've known for some time asked me what I was doing at work. Among other projects, I remarked that my long-awaited book on forecasting was about to be published (for the record, the book is Forecasting Government Budgets: Methods and Applications, co-authored with Dr. Aman Khan of Texas Tech University, published this month by Lexington Books). My acquaintance then asked, "Well, what will the economy do next year?" I felt by the tone of the question that he meant to insert the invective "Einstein" between Well and what. I then shocked him by not predicting in some Nostradamus manner what the exact growth rate of the economy would be but by giving him a range of potential outcomes and a discussion of risks faced by anyone trying to forecast the economy. He didn't seem pleased, but I explained that a good forecaster should provide a range of predicted values in a forecast, even in the best of times. And given the high level of uncertainty in the world now, forecasters should definitely admit that they have less of an idea about the potential outcomes in the economy than they usually would.
This is one of the lessons that one can glean by reading our book on forecasting. Regrettably, given some really awful media coverage of forecasts, the general public thinks either that forecasters are Carnac-like prognosticators who can forecast simply by holding cards to their forehead (for the younger audience, the reference is to the old Johnny Carson Show) or they are charlatans pushing their forecasts for political or financial purposes. And then there are the jokes about economic forecasters making astrologers look good (with apologies to astrologers). The reality is, of course, somewhere in-between swami and charlatan. Forecasters are humans who are subject to reasoning errors but who can often be quite right about things. And forecasters rely on models which are by their nature not right 100% of the time. There is always some amount of error in models. Another big lesson in our book is that error in our models is inevitable and must be included in a forecast. Otherwise, it is not a forecast but a projection of past outcomes into the future. Much of what comes from budget forecasters, for example, is projection because it takes past outcomes and simply extrapolates them into the future. They prefer not to disclose potential errors because they fear what citizens will perceive about risks and how those risks can be spun as problems with an administration's policies.
Another lesson our book offers is the tremendous range of models that forecasters can use. There are several dozen, if not hundreds of, potential techniques, and the numbers grow each year. The book includes sections on models that I have recently helped develop for use by revenue and budget forecasters: ensemble forecasts and forecasts based on artificial intelligence techniques. Ensemble forecasts take other predictions and combine them through various weighting techniques to produce a single forecast that often outperforms each of the individual estimates. If you have ever seen a "spaghetti plot" of predictions of hurricane paths with a combined forecast built from them, that is an ensemble forecast. In a 2019 paper, I created an ensemble model that outperformed individual Chicago sales tax revenue forecasts. And working with a team of researchers at UIS, we recently developed a model based on "deep learning" techniques (a type of Artificial Intelligence model) that outperformed traditional state sales tax receipts forecasts. Adding these models to several traditional forecasting models like linear regression, time-series modeling, and naive models like trend and cyclical decomposition models produces an extensive menu for forecasters to choose from. Yet another lesson from the book is that there is not one best technique. For several economic and budget variables, simple techniques may work fine. Others require more sophisticated methods.
Forecasting is simultaneously a highly technical science and an art. For example, estimating the individual or institutional responses to economic or fiscal conditions may tell the forecaster about important variables that should be included in a forecast. Recently, most forecasters had predictions of economic activity in the first and second quarters of 2020 that were too high due to the COVID-19 pandemic and associated lockdowns. This is perfectly understandable. The pandemic was an "exogenous" event that caused a break in historical patterns. However, most forecasters also had economic forecasts for the third and fourth quarters that were too low. This is because they did not expect the Federal Government's response in the form of pandemic relief loans and checks for businesses and households. This response was so robust as to create a situation where Disposable Personal Income, the measure of how much money households have to spend, went up during the pandemic even as Gross Domestic Product, the broadest measure of economic activity, fell, indicating a recession. This has never before happened during a recession.
The robust response to the 2020 pandemic recession creates even more uncertainty about the current economic situation. Most forecasters show the economy slowing and perhaps entering a recession in early 2023. One big unknown about that scenario is how the Federal Government will respond. Suppose it responds as it did in 2020 and 2021, creating a sizeable fiscal impulse by sending checks to people and businesses. In that case, the downturn may be followed by a strong rebound. If not, the economy may languish in a low growth path. Returning to the conversation with my friend, this is just one of the many uncertainties facing forecasters in 2022. Which is my reason to be explicit about the range of potential outcomes.
One final lesson is related to the others. That is the power of humility and learning from mistakes. I often tell the story of when I first started doing forecasts for a large city. In one of the first years, my estimates for sales tax revenues were less than 0.5 percent off. I nearly broke my arm, patting myself on the back, as anything under 2 to 3 percent is considered a good forecast. The following year I confidently gave my forecasts using the same model, which then proceeded to be off substantially due in part to a slower-than-expected recovery from the 2008-09 financial crisis. Was the second forecast a lousy one created by a bad forecaster? Or was the first forecast a good one made by a great forecaster? In truth, both were likely a mix of good and bad. For forecasters, the appropriate response is not to celebrate good forecasts and laugh or curse at bad ones. The best approach to the next forecasting exercise is to be humble, learn from one's mistakes, apply the best model at your disposal, and acknowledge the limits of knowing too much about the future.
And of course, provide ranges of outcomes along with single numbers. So, if you see me at a gathering, please ask my opinion about the economy. But please don't be disappointed if I respond with a range of figures and several caveats. And I promise not to be disappointed if your eyes glaze over.
Kenneth A. Kriz, Ph.D. is the University of Illinois Distinguished Professor of Public Administration in the School of Public Management and Policy at the University of Illinois at Springfield. Dr. Kriz conducts research focusing on subnational debt policy and administration, public pension fund management, government financial risk management, economic and revenue forecasting, and behavioral public finance. A nationally recognized scholar for his work on public finance and quantitative data analysis, Dr. Kriz has published over 50 academic journal articles and book chapters along with books on quantitative research methods in public administration and tax increment financing, a frequently used economic development incentive. Dr. Kriz' newest book: Forecasting Government Budgets, will be published in Fall 2022 by Lexington Books. Dr. Kriz has consulted with several public and nonprofit organizations on financial and economic matters, including the cities of New York City, Minneapolis, St. Paul, Omaha, and Wichita, and the states of Nebraska and Kansas. He served as Vice-Chairperson of the City of Omaha, Nebraska Civilian Employees Retirement System from 2006 to 2011 and on the Board of Trustees of the Wichita, Kansas Police & Fire Retirement System, and on the Joint Investment Committee for the city’s pension funds from 2014 to 2018. Dr. Kriz was a Fulbright Scholar in the Republic of Estonia during the academic year 2004-05 and a Fulbright Senior Specialist in the Czech Republic in 2008.