Few economists have a sophisticated theoretical education in macroeconometric modeling — many are the equivalent of children playing with a computer game they really don’t understand. Arnold Kling is not one of these. In a series of essays Kling has been sharing a practitioner’s inside view of the lost knowledge of a generation which had spilled scientific blood in the effort to come to grips with the scientific limitations and deceptions of macroeconomic modeling. In his latest autopsy of the “science” of macroeconomic modeling Kling provides advanced versions of arguments about complex phenomena and statistics found in Hayek against the feasibility of “scientific” macroeconomic modelings of the economy across time. Here’s a taste of Kling:
Economists cannot construct controlled experiments to test all of our interesting hypotheses. We have abundant data, but we did not create the circumstances that produced the data. In statistical jargon, we are making observational studies.
An observational study can be of scientific use if the conditions are right. One condition is that there are many observations relative to the number of factors that must be controlled for. In statistical jargon, this is known as the degrees of freedom.
In macroeconomics, there are more factors to be controlled for than there are observations. There are negative degrees of freedom, which should cause your statistical software to give you an error message.
Instead, the modeler limits the way that factors enter the model. For example, the modeler probably will not control for changes in the educational attainment of the labor force over time. That is not because the educational attainment over time does not matter. It is because the modeler does not want to put in so many factors that the computer spits out an error message.
There are thousands of ways to specify the “consumption function,” which is the equation that predicts consumer spending. Should durable goods spending be separated from spending on nondurable goods and services? Should previous periods’ income be used in addition to current income, and with what weight? Should a measure of anticipated future income be used? How should wealth enter the equation? Is there a way to account for the role of credit market conditions? How do tax considerations enter? Are there different propensities to consume out of wage income and out of transfer payments? How do consumers respond to changes in oil prices? How do they form expectations for oil prices in the future? What factors that are trending over time, such as population changes and shifts in the mix of consumption, need to be controlled for? Which time periods are affected by special factors, such as the recent snowstorms along the east coast?
If you have about 80 quarters of data to work with, and you have thousands of factors to control for, there is no conceivable way for the model’s specification to reflect the data. Instead, the specification depends on the opinion of the modeler.
The conditions under which statistical techniques are scientifically valid are not satisfied with macroeconomic data. There is no reason to take model results as reflecting anything other than the opinion of the modeler.
UPDATE: Kevin Hassett explains how a 3rd grader with a pencil and ruler easily outperforms the “professionals”:
The large-scale Keynesian forecasting models were discarded by most of the profession because they didn’t work. One of the first to demonstrate this was Charles R. Nelson of the University of Chicago, who in 1972 showed that forecasts based on simple extrapolations significantly outperformed theory-laden macroeconomic models in competitions.
About a decade later, Virginia Tech economist Richard Ashley performed a similar exercise that found the big macro models “so inaccurate that simple extrapolation of historical trends is superior for forecasts more than a couple of quarters ahead.” To paraphrase Ashley, all you need to outperform the fancy models is a ruler and a pencil.