The father of behavioral economics explains why you can't predict the market

When Richard Thaler and Werner De Bondt published a 1985 study in the Journal of Finance called "Does the Market Overreact?," it was borderline heresy to suggest the markets weren't completely rational and efficient. Anyone who's paid even passing attention the past 20-odd years knows financial markets are prone to bouts of what Alan Greenspan called "irrational exuberance" and it's pessimistic counterpart. And, yet, "there are still people clinging to the belief markets are perfectly rational all the time despite what we've seen the last 20 years," Thaler tells me in the accompanying video.

Ironically, the idea that supposedly learned people would believe something despite mountains of contrary evidence is something behavioral economics, the field Thaler and De Bondt helped birth, would at least try to factor into the analysis.  

Thaler's last book, Misbehaving, is an attempt to explain the origins of behavioral economics and "to explain some of the things we learned along the way." Among those lessons are two the University of Chicago Booth School professor says are particularly relevant to investors:

• "One, you can learn a lot about yourself by studying the mistakes everybody else makes because you're no different than everyone else. You're probably not going to beat the market - even if you watch Yahoo Finance religiously you don't really know more than everyone else."

• "Two, markets have these wild swings and people have a tendency to extrapolate the past into the future. When real estate prices were booming in places like Vegas...you'd hear people say 'Real estate prices always go up, I can't lose'. As soon as you hear that, it's time to get out."

These truths may seem self evident but, again, "the idea that there are people out there and they're not all as smart as the smartest economists wouldn't seem to be particularly controversial," Thaler quips. But it is in certain circles, including the realm of public policy where Thaler notes economists have a "virtual monopoly" on advising elected officials and policymakers.

So does a belief in Eugene Fama's Efficient Market Hypothesis explain why we have so much bad policy making?

"To be fair, economists are the only ones willing to do it -- most behavioral scientists are not that interested in what the Fed should be doing or whether austerity is good or bad for us," Thaler says. "We're stuck with economists but I think we can make economists and economics more realistic by including humans in the picture. The biggest policy mistakes we've made, like assuming there can't be a bubble because our theories say it's impossible, that sort of ignorance -- there's no excuse for that."

Aaron Task is Editor-at-Large of Yahoo Finance. You can follow him on Twitter at @aarontask or email him at atask@yahoo-inc.com.