Persuading our Peers
Soren asked why I don’t think predictive accuracy is necessarily the best way to test whether someone is a good economist. First let me point out that economic theory itself predicts that it should almost impossible to make unconditional predictions of asset prices, or to predict the business cycle beyond a few months. So the fact that Irving Fisher didn’t predict the stock market crash, for instance, is totally irrelevant to his ability as an economist. But there is certainly something to Soren’s point. If economic models cannot make at least conditional predictions, what use are they?
I do believe that predictive power is very important, indeed I have argued that what most people call “objective truth,” is nothing more than statements with highly predictable implications. So I don’t want to dismiss prediction, but I think there is more to knowledge than just predictive power. It’s been a while since I read any methodology, so forgive me if I have forgotten the correct terminology, or just offer warmed up Richard Rorty (via McCloskey.)
Rorty said that “truth is what my peers let me get away with.” At first glance that seems wrong, surely truth is what’s true—not simply what people believe? Unfortunately we don’t have any direct access to reality, all we know about the world is what we believe to be true based on observation, experiment, etc. So this raises two questions; what criteria do my peers use to judge what is true, and can we think of better criteria? I believe most people are methodological pluralists (regardless of what they say), and I cannot imagine how it could be otherwise. Let’s look at some techniques that one might employ to persuade, which do not necessarily involve prediction. (I am no expert here, so I will probably make a few mistakes. But I think my general approach would withstand close scrutiny.)
1. Model elegance
2. Logic
3. Theoretical justification
4. Coherence
5. Accuracy of assumptions
6. Metaphors
Model Elegance: I don’t know if this story is apocryphal, but I recall reading that immediately after Copernicus first developed his sun-centered model of the solar system, it did not predict as well as the established Ptolemaic model, which had the earth at the center. Nevertheless, the Copernican system offered the advantage of simplicity. And it was not simply a matter of scientists preferring the easier model to work with, but also a deep, almost religious, intuition that the structure of the universe is aesthetically elegant.
Logic: In Pop Internationalism, Paul Krugman defends some of his views with simple accounting. For instance, he dismisses the fear that America will suffer from both a trade deficit and a capital outflow, by noting that the capital account is the mirror image of the current account. So if the data are collected accurately, it should be impossible for both accounts to be simultaneously in deficit. The appeal of Krugman’s model is not its predictive power, but its logic.
Theoretical Justification: Over time, the field of economics has developed an array of well-established models, most famously supply and demand. In my study of the price level during the Great Depression I applied the model of supply and demand to the international gold market. I did not pick this market because gold market shocks predicted inflation better than other markets. Indeed, I am almost certain that if one searched thousands of other markets, one would probably find some commodities that would better explain price level changes than the gold market. I chose the gold market because we have strong theoretical priors suggesting that the market might be pivotal. Unlike other commodities, gold was the medium of account throughout almost all of the 1930s. So changes in the price level were, by definition, changes in the value of gold. Thus one would expect supply and demand shocks in the gold market to be especially important, and it would make no difference to me if someone found the interwar price level was better predicted by shocks in the market for strawberries.
To make this point clearer, let me mention an article I once read (in the AER no less!) that in my view lacked theoretical justification. If my memory is correct, the article showed that over a certain period of time the return on equities on the New York Stock Exchange was slightly lower on rainy days. When I first read this article I immediately thought “data mining.” When I learned a bit more about research I realized that data mining is an inevitable (and necessary) part of the research process. In principle, one should use one set of data to construct a model, and another set to test it. But that is not always easy to do. In any case, perhaps the real problem was not data mining, but rather that there was no well-established economic theory linking rainy days and stock returns, so it looked a lot like the researcher had simply searched thousands of possible correlations, and found two variables that were significant at the 95% confidence level. (And I seem to recall that even that required a one-tailed test!)
I would not be impressed if told that the predictive power of the rainy day study is slightly higher than my gold standard study. We can never know for certain how well a model that has performed well with one set of data, will do with a new set of data. But if the model is theoretically justified, I will have a bit more confidence that any previous successes were not just a fluke.
Accuracy of Assumptions: In my research on 1933 I often refer to government statistics that show wages jumped about 20% between July and September. Such an increase in the midst of a Depression is nearly incredible, and thus I have sometimes wondered if the data are accurate. Perhaps businesses reported these numbers to please the Roosevelt administration. It seems to me that my study would be more convincing if I had outside evidence for this wage shock—such as narrative accounts from business or labor leaders indicating that something like this did occur in 1933. Adding such a narrative to my study of the Depression would give the reader more confidence that my data was accurate. It would not change the R-squared in any of the regressions, but it would make my findings more persuasive.
Coherence: Rorty argued against the “correspondence theory of truth,” the idea that our models of reality in some sense correspond to or mirror nature. He was a pragmatist who preferred to speak in terms of a “coherence theory of truth.” This means that we find explanations persuasive if they help us make sense out of a previously bewildering set of observations. Thus a historian might make sense out of the complex political changes between the 19th and 20th centuries by utilizing abstract concepts such as nationalism. If nationalism is no longer the problem it was 100 years ago, then the theory may not predict anything useful today. But it might nonetheless provide a unifying explanation for seemingly unrelated phenomena during the earlier period of history.
Milton Friedman was a forceful proponent of the view that predictive accuracy is the true test of a model. But did he really practice what he preached? His most famous book (co-authored with Anna Schwartz) was the Monetary History of the U.S. What makes that book so famous? The statistical analysis is extremely rudimentary by contemporary standards. But then why is it that when people want to argue that monetary forces caused the Great Depression, they generally refer to Friedman and Schwartz, and not some more recent VAR study of the Depression? I would argue that it is precisely the methodological pluralism of the Monetary History (and the fact that it is done so well) that makes it so persuasive. Even the footnotes are well worth reading, as they embellish the narrative in ways that support Friedman and Schwartz’s argument that causation went from money to spending, i.e. they provide a sense of which policy decisions should be regarded as exogenous. Their work is impressive on almost every level, it’s accuracy, its theoretical justification, its coherence, its rigorous logic and sophisticated understanding of economic theory. I don’t know about anyone else, but I find a study to be much more persuasive when I can see that it has been produced by superb scholars.
Metaphors: I seem to recall that McCloskey once argued that even metaphors play an important role in persuading our peers. Thus Adam Smith’s “invisible hand” metaphor nicely conveys the intuition behind his argument that markets allocate resources efficiently. On the other hand, the “trickle down” metaphor does a poor job of illustrating the intuition behind supply-side economics, which may be why it is more often employed by opponents of supply-side polices. You might say that metaphors don’t really show whether a model is correct or not, they merely persuade economists to accept a model. But what is the difference?
To summarize, economists are persuaded by all sorts of arguments. Prediction is certainly important, perhaps the most important test of a theory. But it has its limits. We cannot do a series of controlled experiments of the Great Depression, changing one factor at a time, so its not clear how prediction could be used to establish which model of the Depression is best.
My methodological pluralism does have one limit, however, any argument should be honest. Unfortunately the ethics of my profession are shameful—many economists keep searching for more “predictive power,” defined in terms of misleading indicators such as t-statistics, with little regard for whether the results actually mean anything. For too many economists, getting published is the only thing that matters.
Tags:
18. March 2009 at 21:49
I have to agree. Scientific methods are constrained by the subject matter about which they are employed. For example prediction in geology is fairly unimportant because it takes a long time for the geological “landscape” to change. Instead we rely on more historical-explanatory approaches. (I ran across a good, relatively short essay on this topic here, which you may find interesting.)
Outsiders of economics sometimes denigrate economists for their lack of predictive ability. But because of the very nature of economies, I don’t think prediction should be the measure of the profession. I think explanation has (and should have) center stage.
19. March 2009 at 05:35
Scott
You are linked here:
– http://www.economist.com/blogs/freeexchange/
Easing Does It.
I am glad the Fed did what they did. A price level announcement would have been better – but acting by surprise and on this scale is nice to see.
Jim
19. March 2009 at 06:27
I think it is fair to say that most economists believe that investment and/or asset market fluctuations are responsible for are the biggest part of business cycle fluctuations. Keynesians like to think in terms of changes in gross investment arising from attendant fluctuations in animal spirits. The Austrians think that changes in the relative prices of capital goods arising from credit expansions and contractions are responsible. You like to rely on changes in AD arising from monetary fluctuations.
In all cases it is a logical consequence of the “No Free Lunch” principle that business cycle fluctuations are almost impossible to foresee. They cannot be quantitatively predicted using our theoretical models. Consequently, economists are doomed to be no better at predicting turns in the business cycle than Jim Cramer is at predicting the future course of stock prices.
The interesting question is why this implication of virtually any form of the efficient markets hypothesis (or of rational/consistent expectations) is not more widely appreciated.
I ascribe this to the pervasive view, arising (I think) from logical positivism, that you are not doing science if you cannot make refutable predictions.
19. March 2009 at 07:40
I have recently become aware (and a fan) of Rorty (new to me, since any serious exposure to philosophy is as well) and to monetarism and this blog (ditto re economics). So it is with surprise and delight that I find them combined here. Thanks!
As you may know, Simon Blackburn (et al) has criticized Rorty’s “peers” statement in a style close to ridicule, unfairly IMO since I assume it was said partly in jest and not meant to be taken literally, and in any event properly interpreted made perfectly good sense to me. I probably couldn’t have expressed that last reaction coherently, but you have with your list of “criteria”. Thanks for that as well.
19. March 2009 at 08:14
Scott: as an economic historian what’s your take on praxeology?
Inferring from Mises’s own style, the core idea behind praxeology is that statistical analysis of historical data may inspire but cannot replace the need for logical deduction from simple premises.
19. March 2009 at 10:17
First of all, thanks for the in-depth response!
Your perspective on this seems to be judging your peers in an academic environment, which is fair enough for a man in your position. As an outsider (and a skeptic and an empiricist), I’m more inclined to judge economists purely from how well the tools and models they provide work when dealing with real-world problems, so predicitve value is really the real test of any model in my book, alongside the underlying logic and accuracy of assumptions. If a model fails in any of those areas, it either needs to be revised or discarded. Anything else is gravy.
Well, if something fails to conform to well-understood and proven theory, I’d be leery of actually relying on it as well.
That’s really my worry. How much can you rely on these things? You point out that current economic theory states that it’s impossible make anything but conditional short-term predictions with current models, and that’s fine provided people use them with that understanding. But how many people actually have that understanding, and consequently grossly misjudge the risk when insuring bonds for instance? And since we can’t run controlled experiments on it, how sure can we be that we’ve actually learnt the correct lessons from the great depression when trying to deal with the current mess? Etc, etc.
Dan in Euroland:
“I have to agree. Scientific methods are constrained by the subject matter about which they are employed. For example prediction in geology is fairly unimportant because it takes a long time for the geological “landscape” to change. Instead we rely on more historical-explanatory approaches.
Outsiders of economics sometimes denigrate economists for their lack of predictive ability. But because of the very nature of economies, I don’t think prediction should be the measure of the profession. I think explanation has (and should have) center stage.”
But doesn’t the value in explaining past events lie in the fact that it equips us to better deal with future events of a similar nature? If you impose a narrative on a sequence of events, you are going to use look to that narrative when you see a similar sequence of events in order to attempt to predict the outcome. Finding patterns is how we work as a species.
19. March 2009 at 10:30
I’ll reiterate my view of value of economics. it’s in coming up with better systems (including incentives) for allocation or resources. First, the economist must understand history, then suggest a course of action/system. That system oughta be better, but will fail for a new/different reason. I would liken it to an engineer that realizes why a bridge collapsed and improves the structure, but guess what… something new will cause the bridge to collapse (wind – Tacoma Narrows). rinse and repeat 🙂 if you really want, it can be framed as a PREDICTIVE value – the old cause of the crisis will not cause the next crisis (unless we change the incentives back to they way they were).
19. March 2009 at 11:14
I ran into Rorty’s views some years ago, but I don’t find them satisfactory. If I am a civil engineer, and build a bridge that my peers in civil engineering let me get away with, that’s all very well, but the real, physical world may not let me get away with it: My bridge may fall down. Indeed, the main distinction between the solidly established sciences and the debatable sciences appears to be that the former are based on an ongoing dialogue with physical reality in which physical reality can say No to our beliefs. No doubt objective economic reality says No to many of the theories of economists, but it’s not so straightforward to identify how precisely it has done so; the alloy of economic policy contains a lot more handwavium, as the past year has demonstrated.
19. March 2009 at 13:50
But, Mr. Stoddard, like Simon Blackburn you are taking Rorty’s “peers” statement too literally, thereby making it appear rather foolish – IMO a strange thing to do given that Rorty was manifestly not a fool.
His point, as I understand it, is that while the bridge/economy collapsing despite peer concurrence in methods that predicted it wouldn’t certainly suggests a deficient analytical methodology, further claiming that the underlying models failed to adequately “correspond to reality” adds nothing. And when an alternative methodology emerges, gains peer approval, and appears to result in fewer collapses, it is sufficient to say that the methodology is “better” (in that sense) than the old one. Again, claiming that the underlying models better “correspond to reality” or are “closer to the truth” adds nothing helpful.
In these fairly “concrete” examples (especially bridges!), the distinction isn’t very important. For purposes of crossing a river, a bridge as seen by civil engineers is “reality” (although the “reality” of the bridge would be quite different from the perspective of the peers of a particle physicist, a researcher in cognitive studies, or even an economist). Obviously, the more subjective the discipline, the more significant the distinction is. And Rorty’s peers were, after all, philosophers, not civil engineers.
19. March 2009 at 14:43
But Charles, the fact that a lot of philosophers approach their discipline in such a way that this distinction can be taken seriously is precisely my complaint about philosophy, and more broadly about a lot of the humanities. We have had people practice this sort of dialectical activity at least since ancient Greece; and the results are not impressive when set against the results of the natural sciences and the applied sciences that are based on them. I take this to be the case precisely because the natural sciences have worked out how to carry on an actual dialogue with objective reality in which reality can tell them, “No, that’s crap,” whereas the humanists have not. So taking humanistic discourse as a normative standard for the pursuit of truth does not strike me as useful. Indeed, I anticipate hopefully the day when the progress of cognitive neuroscience allows us to hear reality saying “No, that’s crap” about epistemological and psychological claims.
19. March 2009 at 16:22
Dan, Thanks for the support. His attached link is a nice summary of how different fields use different methods of inquiry, in case other commenters are interested.
JimP, Thanks for the Economist link (my favorite magazine–although I don’t agree with them on monetary policy.) I have a new post today on the recent Fed action.
Carl, I think most economists are wrong about investment cycles causing business cycles. They may be the trigger on occasion (such as this cycle) but not usually. And even when they are the trigger, it also requires bad monetary policy. I agree with you about the implications of the EMH. I think there is a certain amount of hubris in economists, that makes them think they can predict better than markets. But they can’t. I also agree with your comments on logical positivism.
Charles, I am afraid I haven’t read Simon Blackburn. I wonder how long it will be before people notice that the only philosopher I know anything about is Rorty. BTW, I just read a short book where Rorty debates Pascal Engel (what a name!) on the question of objective truth. It is a good way of better understanding where Rorty is coming from (as he is easy to ridicule if misunderstood.)
Jon, “Human Action” has been sitting (unread) on my shelf for 34 years–another book to read this summer. I can’t really comment other than to say that logic is important, but can only take you so far in the social sciences. One needs an eclectic approach.
Soren, You have some good points that make me think again, but first a few comments where I slightly disagree:
I don’t think economists should be judged harshly because they can’t predict asset prices or business cycle turning points. Economic theory suggests these should be unpredictable. We don’t look down on statisticians because they can’t predict roulette.
Conditional predictions can be made, even long term. If you tell me that Obama puts a $5 gallon tax on gas, I’ll predict a long term movement toward much more energy efficient cars.
Regarding failures (and I view economists as being more to blame for this crisis than perhaps anyone else on the planet–as I think it was tight money, not the banking crisis that sent NGDP sharply lower last fall) I still think it is one of those glass half full/half empty situations. Economics doesn’t seem able to predict all that well, but we understand enough to make conditional predictions useful for policy, and which allow for policy choices that keep us far above the worst case (say N. Korea, where policy is not at all informed by the science of economics.)
I think your strongest point was in your response to Dan. Your are right that if my work on the Depression has any value it should tell us something about the modern day world. I do think that it helped be understand and predict what happened in Argentina after year 2000, a bit better than if I hadn’t studied the Depression. I also think it helps me better understand the current crisis, at least in terms of conditional predictions. I think my original thinking was confused, mixing up the issue of whether prediction is important, and whether it is easy to tell whether a model of something like the Depression can actually predict current events. Those are conceptually different issues.
Alex, I like your pragmatic take on this issue. Another good example is passenger jets, which were really safe in the 1980s, really really safe in the 1990s, and really really really safe since 9/11. How do they keep getting safer? Every time there is a crash they find out what caused it and try to prevent that specific problem from happening again.
William, I don’t see much conflict between your views and Rorty’s views. Rorty isn’t a realist or an anti-realist. Rorty is a pragmatist. He isn’t telling scientists what to think, he is telling other philosophers to stop telling scientists what to think. He says we should let each field decide what they consider knowledge, don’t impose a one size fits all standard. He understands how average people define “truth” or “objective truth”. (He thinks they aren’t useful epistomological categories–they may be useful in everyday discourse.) He understands that the sciences have a great track record at predicting certain things, and that that fact gives people a high level of confidence in the models used to make predictions. But ultimately all we have is beliefs. We don’t directly observe “objective reality” in some fields and not others, rather we simply make more accurate predictions in some fields than others. There isn’t really more that can me said, without resorting to some sort of religious leap of faith. But given that even Newton’s elegant model has now been shown to be only true in the limiting case, we should be very modest before claiming that our scientific knowledge goes beyond prediction into some sort of mirror of ultimate reality.
Regarding your observation that nature constrains what beliefs are acceptable in science, as when bridges collapse, I completely agree. But there is always the human interpretation. Did the bridge collapse because of wind, or because a bomber dropped bombs on it? That will make all the difference when we decide how to design the next bridge. And that decision will be made by our peers, other civil engineers. His statement about “peers” is really a statement about how society operates, and how it uses terms like ‘truth.’
19. March 2009 at 18:33
Interesting. I assume you mean the first one as an analogue to Occam’s Razor (http://cgm.cs.mcgill.ca/~soss/cs644/projects/jacob/)
The six criteria you sum up are similar to the criteria used in medicine to determine whether some therapy is scientifically sound.
An example: showing in a single clinical trial that a given therapy is superior to another, will not be enough to convince most physicians to use it. Consider this the medical analogue of predictive accuracy. For physicians to accept the therapy, other criteria also have to be met. This is often a matter of contention with alternative medicine.
20. March 2009 at 04:51
The passenger plane story may be a symptom of what’s wrong with a lot of planning. Maybe the focus on making safer planes has distracted us from innovation. Perhaps if we were willing to tolerate defects at ten to twenty times the current rate, we might have much faster or more fuel efficient or just — in general — more amazing air travel. Look at how lawsuits/torts killed Cessna and the entire small plane industry. Science and engineering work well when the focus is narrow and agreed upon. When the focus is broader (global warming) or harder to test (string theory), either political values or aesthetic/logical ones become predominant.
20. March 2009 at 06:32
I concur with Mr. Sumner’s comment re Rorty’s position, especially on “objective truth”. Rorty made his non-belief in it quite clear, and how seriously he meant his “peers” statement to be taken should be judged with that in mind.
And Rorty may very well have agreed with Mr. Stoddard’s assessment of philosophy. In his later years he switched to teaching literature.
20. March 2009 at 09:41
The difficulty of social sciences not being as testable as physical sciences is a not a failure of the field, but a difficulty and requires a different aproach as well as expectations. It is NOT a science, but Consequences and causes are highly complex allowing for statements like “quantitative or fiscal stimulus have never been attempted” because they never occur in
20. March 2009 at 09:49
Economics is NOT a science, it is a field of study. It seems to fail precisely when people take a scientific aproach to it. Obviously finance is the area where people fall most susceptible to trying to apply scientific method to the madness. All the problems listed are not failures of the study, but difficulties and reasons for an aproach and expectations different from the scientific method. Now on the second question of whether the current non scientific and more philosophical aproach practiced by few (taleb, soros) is valuable cannot be answered by using scientific reasoning. the systems are too complex, and scientific hypothesis too simplistic and linear. BUT any economist/financier posing as Copernicus deserves the same fate. Ironic ain’t it!
20. March 2009 at 11:58
Michael, It is similar to Occam’s Razor, although you could also give it a mystical/religious slant, if you are so inclined. I think some physicists have a gut feeling that the true model of reality is simple, and don’t gravitate toward the simpler model solely for the reasons in Occam’s Razor.
ajb, I totally agree. I just picked the jet example as a learning from experience case. But we often do tend to put too much emphasis on safety, at the expense of other worthy goals.
Charles, I know that Rorty grew disenchanted with much of philosophy, as his theories challenge its usefulness (in some areas.) I think that he thought the most valuable philosphers were akin to “strong poets,” those that make us see the world in a new way, That’s also how I think about great economists, by the way.
Alex, I basically agree about how economics cannot make precise predictions. I would just add one point. The debate over whether it is a science completely depends on how one defines ‘science.’ The root of the word is actually something like “knowledge,” which is pretty vague.
Economics is probably as precise as meteorology, which is generally regarded as a science. So I think the term “social science” is fine. We do some scientific-type things, but with much less precision than physics or chemistry.
21. March 2009 at 13:38
Scott: I don’t mean to dismiss economists just because they can’t predict short term prices and business cycles, conditional long term predictions are perfectly acceptable, as long as they’re all people claim to be able to make. I just want a clear picture of what a tool is and isn’t able to do.
Nitpick about Occam’s Razor: Occam’s Razor doesn’t state that we should prefer the simplest explanation, but the one that introduces the fewest new assumptions. Often they’re one and the same, but that’s not always the case.
23. March 2009 at 06:01
Soren, Thanks for clarifying Occam’s Razor. Perhaps it is slightly different from the concept of theoretical “elegance” that some theorists intuitively feel to be advantageous. What do you think?
29. March 2009 at 00:10
Unfortunately we don’t have any direct access to reality
That has never made sense to me. We cannot act on any other basis than that we do have direct access to reality. Which, within limits, seems to work for us most of the time.
It is perfectly true our access is not certain, complete, etc. But that is a very different thing than saying we have no direct access to reality. But I think the entire neo-Kantian push in philosophy is a disaster. (The best text on why is Stephen Hicks’ Explaining Postmodernism: Skepticism and Socialism from Rousseau to Foucault which I review here.)
Besides, I think microeconomics does quite well with pretty robust predictions of general tendency.
29. March 2009 at 09:58
Lorenzo, I think we act on the basis of a sort of mental model of the world that we all have in our minds. It doesn’t have to reflect reality in order to be effective. Early man thought the world was flat, and that model served their purposes quite well. What most people regard as “objective” is merely what predicts with a high degree of accuracy. I share your disDain for much of post-modernism. Many post-modernists don’t understand economics, and therefore their analysis of economic issues has little value. For instance, their theories don’t predict very well, just as Marx’s theories didn’t predict very well.
31. March 2009 at 04:10
Science does not require prediction, per se, but it does require falsifiable hypotheses. Discounting inability to predict market prices is one thing, but retrospective explanation must contain enough precision to allow at least theoretical testing. It would seem that if economists are to provide a service of any worth, they must provide more than hand-waving explanations of the past- we get that much from astrologers.
31. March 2009 at 05:05
Will, I don’t think many modern epistomologists would agree with your views about falsifiability being essential to science. Of course economics does make some predictions (and thus is partly falsifiable), I did emphasize that predictive power is important. But it’s not the only thing. The astrology comparison seems odd to me. Surely astrology does make predictions—indeed I thought that that was pretty much all it did. It’s just that astrology’s predictive power is weak. So astrologers are doing science in the Popperian sense, just not very good science (they haven’t noticed yet that their theories have already been falsified.) History would be a better example. And I think that even economic history that is not falsifiable, can be very useful.
1. April 2009 at 04:57
My point about astrology was not that astrologers don’t make predictions: they certainly make plenty of them. It is the explanation after the fact, in an attempt to make the prediction fit the subsequently revealed reality, that I was referring to. Falsifiable claims are necessary for any field of inquiry, if it is to avoid falling into pseudoscience.
3. April 2009 at 16:10
Will, Well I don’t know much about astrology. But I thought it was based on the idea that your character reflected (to some extent) what sign you were born under. If so, there should be a strong correlation between birth dates and membership in professions characteristic of that type person. (Such as Congressmen.) But I’m sure someone who does know astrology will tell me where I am wrong.