I see dead patterns.
Everywhere I look I see patterns. Because of the off year elections I started thinking last night about the odds of Obama being re-elected. I was familiar with the data showing that when presidents run for re-election, more often than not they succeed. But then I started noticing another pattern; the more interesting correlation was between success in being re-elected and the number of years a given party has held the presidency.
I have finally memorized all the Presidents since 1900, and noticed that only in 1980 was a party rejected after 4 years in the presidency (out of 11 chances.) So I thought “wow, it looks like Obama’s got a 91% chance of being re-elected.”
Since my readers are curious about the world, and numbers-oriented, I’m sure you have reasoned through issues in much the same way. And I’m sure you know what’s coming next; I’m going to accuse myself of data mining. But there is a solution to data mining—do an out-of-sample test.
Well that didn’t take long to blow my lovely theory out of the water. In 1888, 1892, and 1896 the presidency was lost by a party that had held it for just 4 years. So my 91% estimate is falling fast. But wait a minute, wasn’t there something wrong with the 1888 election? Yes, Wikipedia shows that Cleveland was robbed in 1888. Cleveland (who I am shamefully ignorant of, but Dems seem embarrassed by him so I figure he was a good president) actually won the popular vote in three consecutive elections, 1884, 1888, and 1892. So now let’s refine my theory a bit. We can pretend Cleveland served three terms by using the popular vote as the criterion, but then of course we have to take away one Bush term. And what about President Tilden? So that won’t work.
Do you see what my mind was trying to do? I was looking for some pattern that would allow me to predict the future. If I tweak the “model” enough I could probably get 100% accuracy, after all there haven’t been all that many elections where the incumbent party had held office for just four years.
And all I really had to do was just look at Intrade, and we can see that the Dems have a nearly 2-1 shot of winning again. The 20th century was a fluke. The odds of replacing a party after 4 years are long, but they aren’t that long.
PS. I woke up this morning with a black president in a country that is 14% African-American, with a black governor in a state that is 5% African-American, and with a black mayor in a city that is maybe 2% African-American. Hmmm, what are the odds of that . . . .
Update 11/5/09: Patterns, patterns, everywhere. Here is another misuse of probability theory.
Update 11/6/09: Russ Roberts sent me this post on Cleveland. No wonder modern Democrats are embarrassed by Cleveland—he took seriously his oath to uphold the Constitution.
Tags:
4. November 2009 at 20:21
You might check out Ray Fair’s model that tries to get at the factors that convince people to vote one way or another. Basically – the economy is a huge factor.
http://fairmodel.econ.yale.edu/rayfair/pdf/2006chtm.htm
4. November 2009 at 20:21
it is interesting how the quality of the study depends entirely on the rigor of the researcher, but there is no way to judge after the fact how rigorous the research was. the data is useless the moment it is published. unlike in physics, there is no way for someone else to come along and attempt to repeat the experiment — if the data has been mined. does this mean statistical data is a finite resource in the universe and we are in danger of using it all up if we data mine everything now? is a conservation plan in order? how about a data-mining tax?
4. November 2009 at 20:34
See here
http://www.jstor.org/stable/420023
Abramowitz has about 96% predictive power using GDP, a dummy variable if the party in power has been there for more than 8 years, and the popularity of the incumbent
4. November 2009 at 20:46
I guess this is a break from all the commentary about Obama losing re-election if unemployment is still high.
5. November 2009 at 02:57
Given that only 1/44th of our presidents have been black (2.27%), while the percent of the population that is black has ranged from 9.7% (1930) to 19.3% (1790), I think it’s actually surprising how few black politicians we can point to…
… unless of course we know anything about American history.
5. November 2009 at 05:23
Tom and Kevin, Those models are good at predicting the past. Predicting dead patterns. But I want to predict the future, and for that purpose Intrade is far better.
Tom, kevin, and rob, When I look at a study I make a mental adjustment for data mining, that varies according to how arbitrary the hypothesis seems. Thus if the model said all Presidents in power for 4 years except former Georgia peanut farmers get re-elected, I would not be impressed. I also think about whether the parameters make sense a priori. We have independent evidence that the economy is important, so it makes sense that the two models mentioned would include the economy.
Mike, I think unemployment will be the key. It would be useful to get some conditonal contracts up and running on Intrade. Probability of Obama winning given unemployment is above 8%, for instance.
Daniel, That makes things even more surprising. I noticed really long odds even if the chance of a black winning reflected their share of the population. Another example of the mirage of statistical significance.
5. November 2009 at 07:27
I added an update on Arnold’s dirty words.
5. November 2009 at 16:03
‘ Those models are good at predicting the past. Predicting dead patterns. But I want to predict the future….’
Huh? Here’s Ray Fair’s prediction on October 30, 2008 for the Presidential election less than a week ahead (remember, his model uses quarterly data, so is only current through the end of September):
http://fairmodel.econ.yale.edu/vote2008/vot1008.htm
‘…the predicted Republican vote share (of the two-party vote) is 48.09 percent. So the prediction is 51.91 for the Democrats and 48.09 for the Republicans, for a spread of 3.82.
‘ The current situation is unusual in that the economy since the end of the third quarter appears to have gotten much worse. People may perceive the economy to be worse than the economic values through the third quarter indicate, which, other things being equal, suggests that the vote equation may overpredict the Republican share.’
Which was just about dead bang on.
5. November 2009 at 16:21
SS: “If I tweak the ‘model’ enough I could probably get 100% accuracy…”
I want to say “that’s common practice, I think” but I don’t want to be rude. I see false patterns.
PS. What do I know? After the fact, I’d say your chances of waking up were 100%.
5. November 2009 at 17:11
We’re back, I have no idea why the blog broke down this afternoon. Perhaps we are frightening some very important people.
Patrick, Check out that model in the year 2000, and you’ll see (if I am not mistaken) that it was way of base.
I predicted Obama would win by about 5%, so the result you cite doesn’t impress me all that much. Yes, it was close, but there are other techniques that were probably closer.
I do agree, however, that economics plays a siginficant role in elections. So I expect the Fair model would do better than a coin flip, but worse than Intrade.
The Arthurian, what do you make of the vicious attack on Arnold Schwartzenegger that I added to the post in an update?
Here’s how I think Arnold should respond:
Understanding the principles of
Probability theory would allow
You to avoid making such
Outrageous charges. Data mining is often
Used to create the impression of a meaningful
Relationship where the only
Significance is for those with a dirty mind
5. November 2009 at 18:08
Speaking of dead patterns:
Here is a paper that argues that left wing governments finance what they do with regressive taxes. They do not steal from the rich to give to the poor – they tax the poor to give the poor benefits. Right wing governments tax income and capital (like here). Left wing tax consumption. Lets put on a vat and finance health care with that.
http://allman.rhon.itam.mx/~jtimmons/Papers/J%5B1%5D._Timmons_Taxes___Partisanship_APSA_2005.pdf
5. November 2009 at 19:50
@Scott, @arthurian: I see misuse of probability theory constantly. I actually see it misused more than I see it used properly.
@JimP thats not a dead pattern, it seems to be a very alive and well pattern.
Economists should start using skillscores to categorize their predictions, because I see a lot of useless and or horrible predictions made.
5. November 2009 at 20:25
Doc
True. I meant to say that my idea of liberal politics was a dead pattern. If liberal politics consists of poor people taxing themselves to buy benefits that they can best afford in that way I have no objection. Krugman Obama DeLong etc think liberal politics is soak the rich class war. If it not but rather is social insurance financed by regressive taxes then I am fine with it.
5. November 2009 at 20:43
@JimP: I should clarify. VAT is very popular in governmental circles for leftists because in order for social spending to work you need a very stable tax base. Taxing the rich isn’t as stable because the rich are very mobile and also suffer a lot (in absolute terms if not, in terms of being destitute) during downturns, which hurts your tax base. Regressive taxes like social security tax, medicare tax, VAT, etc make a very stable tax base. Thus politicians are able to buy votes from left leaning low income folks by taking their own money from them first. This also has the added benefit of not angering politically powerful higher income folks.
5. November 2009 at 20:44
@JimP: Got yah. Yah that makes more sense.
6. November 2009 at 05:49
JIm and Doc, I couldn’t open that link, but I am aware of that tendency. I recall Peter Lindert did research on this topic. Because social welfare states in Europe need to raise close to 50% of GDP, taxes on the rich aren’t enough. In addition, such a high tax burden would severely damage an economy if it was loaded on to capital. So the extra taxes they collect are mostly from consumption, and European taxes on capital are generally lower than in the US.
6. November 2009 at 06:56
Unemployment at over 10%
Phelps has an interview on Bloomberg saying that the PIMCO new normal of deflation and depression is our fate – apparently no matter what we do. And Krugman has a column up calling for higher and higher stimulus – the Japan road – with the Japan ending.
It is either Sumner or we are really toast.
6. November 2009 at 07:13
Early on I wrote that until Obama himself explicitly embraces, as Roosevelt did in 1933, higher inflation – or higher nominal prices or a weaker dollar or higher NGDP growth or whatever you wish to call it – until Obama finds the political courage to explicitly call for that Bernanke wont have the votes on the board to do it. I beieve that is where we are now. I do agree with Krugman – Obama is too careful. He is just not bold enough.
6. November 2009 at 08:31
Or another way to think about this: all those economies look approximately the same, were this to be so then the effect of policy generally + taxes must respectively be the same on capital and consumption — to achieve equilibrium.
Generally disfavorable environment to capital -> lower capital taxes.
General encouragement of consumption without production -> higher consumption taxes.
Are the Swiss Leftist or Rightist? They sure have low taxes there.
6. November 2009 at 08:37
‘ Patrick, Check out that model in the year 2000, and you’ll see (if I am not mistaken) that it was way of base.’
Nope, here was his last word before the 2000 election:
‘ The Democrats are thus predicted to win with 50.8 percent of the two party vote. Given that the standard error of the equation is 2.15 percentage points, the election is essentially predicted to be too close to call.’
Again, remembering that Gore got slightly more of the popular vote than Bush, that’s pretty good shooting.
His model only faltered twice. It predicted a landslide for W. in 2004 which didn’t materialize, and had H.W. winning in 92. But that election was odd in that it had the well-financed third party candidate, Ross Perot. So, I’d not expect a two-party model to perform in that case.
So, we’re left only to explain why it over=predicted for W. in 2004. My guess is that Fair is wrong that war and peace have nothing to do with presidential elections.
It’s real weakness though, is that he doesn’t have the minimum 30 data points, and given that we only have elections every four years, he never will.
6. November 2009 at 09:37
And The Economist seems to agree:
http://www.economist.com/blogs/freeexchange/
November
6th
15:59 GMT +00:00
Crank up the helicopter
Posted by: Economist.com | WASHINGTON
Categories: Monetary policy
………………………..
Begin quote:
Does this describe America? I don’t actually think it does. I am persuaded, I think, by something Tyler Cowen once wrote on the subject:
For another view, Paul Krugman argues people may not expect the inflation to continue for long enough, and therefore won’t spend their money but will instead expect a future deflation further down the road. I think that creating and maintaining the inflationary expectations is quite easy, especially if the inflation will boost output and employment and thereby make politicians popular with voters. If you print money, people don’t think “hmm…that is inflationary…that means someday the central bank will have to deflate, I’ll wait six years and spend this new money when prices are really low.” Yes, I see the intertemporal equilibrium concept, but nope, that fails Psych 101.
Of course, as Mr Krugman notes, the Fed can’t print money. But that doesn’t mean it’s powerless. Let’s go now to a 2002 speech by one Ben Bernanke:
Each of the policy options I have discussed so far involves the Fed’s acting on its own. In practice, the effectiveness of anti-deflation policy could be significantly enhanced by cooperation between the monetary and fiscal authorities. A broad-based tax cut, for example, accommodated by a program of open-market purchases to alleviate any tendency for interest rates to increase, would almost certainly be an effective stimulant to consumption and hence to prices. Even if households decided not to increase consumption but instead re-balanced their portfolios by using their extra cash to acquire real and financial assets, the resulting increase in asset values would lower the cost of capital and improve the balance sheet positions of potential borrowers. A money-financed tax cut is essentially equivalent to Milton Friedman’s famous “helicopter drop” of money.
I believe that if you put enough money in the hands of American consumers, then eventually they’ll begin spending it. Now, is it necessary to do as Mr Bernanke recommends above? I don’t think so; as Mr Krugman himself has written, a credible promise by the Fed to allow future inflation would probably do the trick
End quote
6. November 2009 at 09:38
A credible promise by the Fed to allow future inflation.
Do it Ben.
6. November 2009 at 09:46
And do it Obama as well.
Don’t promise free health care financed by 60% income tax brackets. Finance health care reform (which we sure do need) with a solid regressive vat.
Cut income taxes and finance that with the helicopter. Get us some inflation.
6. November 2009 at 09:49
I try to avoid philosophical comments, but:
When change is manifesting, logic about underlying processes will contest with historical data. If the logic of change wins, then it will have always been obvious. If the logic of change loses, it will only prove that the ideas underlying the change were stupid from the start. The winners always rebuild history to suit their needs.
IMHO, the most effective use of historical data patterns is to test _historical_ narratives which people use to marshall arguments that affect current/future policy. OFTEN, there is a dominant narrative that people use to explain history (based on some presumed logic). If we can demonstrate that historical data is inconsistent with the historical narrative, then that forces us to reconstruct the historical narrative, and this can help us build better theories about how things work – and it’s these theories that help us try to predict events.
All models are a combination of intuition/logic about underlying processes (the “data generating mechanism”), and the data. All are inherently partial models – which is to say all of them throw everything not in the model into the error. All imply some sort of causality. When using logic/intuition, all models borrow accross intellectual domains (many great innovations are inspired by lateral thinking – applying insights in one domain into another).
Cognitively, by the way, humans are PROGRAMMED to see patterns. It takes only 2 data points to suggest a pattern, and 3 to prove it to the human mind. This has some negative consequences…
http://www.eduratireview.com/2009/07/conspiracy-theories-patterns-teaching.html
If you go back further, many anthropologists attribute our advanced pattern-detecting ability as an explanation for the belief in magic and superstition. (Twice, I shook a stick and it rained… is there a correlation?)
Also, once a pattern gets embedded – either in a single person or even worse a set of people – it’s hard to erradicate. This is why the scientific method marks such a great change in history.
Many educated folks believe themselves to be less vulnerable to these pitfalls of the human brain. (We’re educated, and liberated, and self-aware…)
However, I recall reading a study comparing students and professors in the realm of foreign policy. The conclusions were that neither group was more willing to change their views when confronted with new data, however the professors were much better at justifying their current beliefs.
BTW, I don’t place much faith in Wisdom of the Crowds for unconventional thinking… There are any number of mechanisms that can uncover Crowd views when they are conventional. That’s _easy_. The advantage of markets is that they allow people to bet AGAINST the crowd, and reward people who are right (sometimes). The great virtue of markets is not merely that they predict better; it’s that they keep account of who made what predictions and reward agents of change.
6. November 2009 at 12:23
Regarding VAT,
I think the reasons leftists like is so much is that it helps make welfare look more important since the cost of living is artifically raised.
Here in Ireland we have 22.5% VAT, it one of the worst things about the place.
6. November 2009 at 18:16
JimP, Those are good points. One of Obama’s mistakes was to not focus on the economy. He got distracted looking at other issues and yet he won’t achieve any of his objectives unless the economy improves a great deal. Otherwise he will be another Jimmy Carter.
It’s interesting to read the news from Europe, given how Krugman seems to like their model. The Germans have reached the conclusion that they need to inject more competition into their health care system. And a number of the continental countries have cut payroll taxes as a way of keeping unemployment down. Indeed they are generally avoiding the scattershot approach to massive fiscal stimulus used here. And their unemployment rates has risen less than ours (although I would concede that their labor markets are also quite different. But Obama seems to be following a 1970s style government model that is out of step with the modern world. And if the economy doesn’t get better quickly he will soon find out that it isn’t going to be possible to pay for it.
Jon, Yes, the European countries are smaller and more aware of their need to compete with their neighbors. They lie somewhere between our national government and our state governments.
Patrick, I guess my memory was wrong, it was 2004. The 8 year point must be a big negative in the model, if the very strong US economy in 2000 only gave the Dems a 50.8 predicted vote share.
JimP, Thanks, I like that Tyler Cowen quotation.
Statsguy, On the wisdom of crowds, I like one dollar one vote more than one man one vote. I think public opinion polls would show much higher inflation expectations than CPI futures markets.
I agree with you that change is hard to predict. My hunch is that the Dems are right about some things (more stimulus needed, some form of universal health insurance needed) and wrong about some things (fiscal stimulus isn’t very effective, and we need actual (catastrophic) health insurance, not open-ended pre-paid health care with no cost controls.)
Current, I’d prefer a VAT over an income tax, but I would definitely oppose a VAT combined with an income tax. That would open the door to the government taking 40% to 50% of GDP.