Should soothsayers be regulated?

I have a new post at Econlog that is far more important that this throwaway effort.  I also recommend a recent post on China by David Beckworth, which I initially overlooked.  And excellent posts on the Fed by Tim Duy and Evan Soltas. But if you insist on continuing . . .

Scott Alexander is perturbed that 80% to 90% of doctors don’t know how to answer an extremely simple statistics question like this:

Ten out of every 1,000 women have breast cancer. Of these 10 women with breast cancer, 9 test positive. Of the 990 women without cancer, about 89 nevertheless test positive. A woman tests positive and wants to know whether she has breast cancer for sure, or at least what the chances are. What is the best answer?

(OK, technically 26% got it right, but it was multiple choice with 5 choices—you do the math.)  Nobody who’s taught in college should be surprised by this.  Unless you test students on material that they are told they’ll be tested on, and that they studied for the night before, most students basically can’t answer anything, even 8th grade level questions. Heck, I’d probably get some elementary stat questions wrong (although at least I can do the “story problems” like the one above) Alexander is concerned about the implications of this (here’s he’s referring to another question):

Good news! 42% of doctors can correctly answer a true-false question on p-values! That’s only 8% worse than a coin flip!

And this paragraph is your friendly reminder that six months after this study was published, the FDA decided it was unsafe for individuals to look at their own genome since they might misunderstand the risks involved. Instead, they must rely on their doctor. I am sure that statisticians and math professors making life-changing health or reproductive decisions feel perfectly confident being at the mercy of people whose statistics knowledge is worse than chance.

Obviously I agree.  But the FDA is model of rationality compared to the SEC.  At least with the FDA you can sort of understand the logic of the regulation.  If doctors actually knew what they theoretically should know, what they were taught in college, then they would have more expertise than the average person.  But not even that excuse is true for the regulation of stock pickers:

The Investment Advisers Act of 1940 is a United States federal law that was created to regulate the actions of those giving investment advice for compensation as means to protect the public.

The Act defines an “investment adviser” as anyone who, for compensation engages in the business of advising others about the value of securities or the advisability of investing in, purchasing, or selling securities.

So you go to your investment adviser for stock picking advice, not to some idiot like me. And that’s because your investment advisor is better able to pick the right stocks and mutual funds than I can.  Or at least that’s the theory.  In fact, they do worse than I’d do, in all but a few cases.  The rest of the post will exclude the tiny number of investment advisors that simply tell you to put everything into low cost index funds.

For the rest, the vast majority of regulated investment advisers, they either understand than indexed funds outperform managed accounts, and hence are dishonest, or they don’t understand and are incompetent.  So the SEC regulation virtually forces people like my mother to go to investment advisers who are either knaves or fools.

(On the other hand if doctors profit from needless cancer tests, then maybe medicine isn’t so different from the investment industry.  Maybe the doctors are just pretending not to understand statistics, as a cover.)

Progressives like to talk about the “science” of an issue (at least sometimes, not the science of gender differences, or GMO foods, or taxing capital income, or free trade, or that studies show that voucher schools are cheaper, but at least for global warming.) OK, the science of finance says that indexed funds outperform managed funds.  That stock pickers are no more helpful to investors than soothsayers. So if we force stock pickers to be regulated, why not do the same for palm readers, fortune tellers, soothsayers, tea leaf readers, and all the rest?  Alternatively, is the SEC going after unlicensed stock pickers any different from the leaders of Salem going after witches?  Aren’t both types of prosecution equally “anti-science”?



24 Responses to “Should soothsayers be regulated?”

  1. Gravatar of Topher Hallquist Topher Hallquist
    7. September 2015 at 06:49

    A wrinkle in your analysis is that some financial advisors *do* recommend index funds, and the bulk of their time ends up being spent on details like how much weight to give different asset classes (e.g. bonds vs. domestic stocks vs. emerging market stocks), rebalancing, tax issues, etc.

    Some people think we should have *tighter* regulation, to require all advisors to follow a fiduciary standard:

  2. Gravatar of Niklas Blanchard Niklas Blanchard
    7. September 2015 at 06:51

    You are behind the times, Scott…,8599,2015676,00.html

  3. Gravatar of Steve Steve
    7. September 2015 at 07:14

    “OK, technically 26% got it right”

    The answer 9/98 wasn’t provided as a choice. I suppose “about 10%” is close enough for medical work–but I wonder if the study authors themselves calculated it correctly..

    “the science of finance says that indexed funds outperform managed funds.”

    Actually managed funds outperform before fees and taxes, but usually underperform after fees and almost always underperform after fees AND taxes.

    And managed bond funds are a little better since it is very hard to replicate a bond index do to illiquidity. It turns out illiquidity, private markets, and costs of short-selling are keys sources of inefficiency in an otherwise near-EMH world.

    “On the other hand if doctors profit from needless cancer tests, then maybe medicine isn’t so different from the investment industry.”

    Sadly, there is truth to this. The people on the front lines are doing very repetitive work, seeing 1000s of customers per year and offering advice based on a few minutes of observation and questioning. That advice often includes palliative and placebos to anxious customers, because the remuneration for education isn’t there.

  4. Gravatar of Market Fiscalist Market Fiscalist
    7. September 2015 at 07:33

    Where does 80% to 90% come from ?

    74% got the answer wrong so another 18% (25% of 74%) probably guessed luckily.

    Which means that the ones who actually knew the answer was 26% -18% = 8%.

  5. Gravatar of ssumner ssumner
    7. September 2015 at 07:36

    Topher, I’ve seen that proposal. While I sympathize with the motivation, in practice it would probably just make things worse. The unintended consequences that flow from that sort of attempt to micromanage business are almost always far worse than the problem itself.

    I think a better solution would be for the government to use a Singapore style forced saving system (for health care and retirement) and require really low management fees on those saving plans, and then allowing a free market in any other savings done outside the plan.

    Someone has to make markets efficient, and it won’t be index funds. So we don’t want to completely ban high cost asset management. (Yes, I know the regulation doesn’t do that, but on the other hand to the extent it doesn’t, it’s toothless.)

    Thanks Niklas, I did a post.

    Steve, Odd, they didn’t offer “about 9%” as an option. What were the other 4 options?

    You make some good points, and I’m not qualified to comment on bond funds.

  6. Gravatar of ssumner ssumner
    7. September 2015 at 07:42

    Market, You might be right, but I was trying to be conservative. You were assuming random guesses. But on “tricky questions” errors may not be random.

    Ask people which way do ships travel when they pass through the Panama Canal, from the Atlantic to the Pacific–east or west? Even if 20% know the correct answer is “East”, I’ll bet less than 50% get it right. (Not the 60% expected with random guesses.)

  7. Gravatar of Market Fiscalist Market Fiscalist
    7. September 2015 at 08:17

    Good point. A lot of people (including me till now) would have been “certain” that the answer is “west” and I can believe that similar biases would have been at work in the answers the doctors gave, and this would indeed reduce the proportion of lucky guesses amongst the right answers.

  8. Gravatar of Steve Steve
    7. September 2015 at 08:28

    The five choices were

    1- 90%
    2- 9/10
    3- 81%
    4- 1/10
    5- 1%

  9. Gravatar of Ray Lopez Ray Lopez
    7. September 2015 at 08:51

    This problem sounds like a standard “Bayesian” question where you have to do the math and make sure the denominator is correct. I’d guess, without doing the math, a smaller number than intuition tells you, so pick either 4 or 5. But since tests tend to be accurate, and shift the curve to the left, it’s not 1% (choice 5) but it’s probably choice 4, hence the answer is 10%.

  10. Gravatar of collin collin
    7. September 2015 at 10:10

    Thinking about financial advisors and records, isn’t one of the problems is a lot of Financial Advisors can be right today but really wrong tomorrow. Didn’t you debate Peter Schiff a few years back on CNBC? In 2006 – 2009 he had a rate track record and was one of the loudest voices of the Finanical Crisis. Today, he has hardly changed his tune and he is endlessly whining that the US Fed is the ruin of us all and hyperinflation is coming.

    Anyway, if you want to help people the most in financial advisors, I would go after 401K rules and fees. The employer based system for 401K into a monopoly investment firm is a terrible way for people to build for retirement.

  11. Gravatar of Kevin Erdmann Kevin Erdmann
    7. September 2015 at 10:43

    I know a lot of financial advisors, and I don’t think any of them see their roles as anything like a stock picker. They may make marginal tactical asset re-weightings based on cyclical factors. Or, they might encourage clients to put some money into some idiosyncratic asset class that they determine has some structural advantage. (Buying homes for rent, IMO, is an obvious area now where people with access to capital can earn higher returns for structural reasons.) But, almost all of their work involves managing their clients’ risk profiles with regard to the various risk premiums available to them, which is not easy. In fact, it’s so hard that many advisors probably don’t do that great of a job. Curing cancer is pretty hard, too – maybe not that much harder than understanding the breadth of risk premiums in a free economy. (We can cure some cancer. I’m not sure if any of us understand 5% of how risk is manifest in financial markets.) But, we don’t throw up our hands and tell people to treat themselves or say that doctors’ efforts aren’t worth it because their patients pay a fee for their treatments versus self-treating patients.

    I am a speculator – a stock picker – and financial advisors I compare notes with will sometimes put some of their own money on my ideas, but they all consider it outside the realm of their professional duties, and I don’t know anyone who has put their clients’ money on my ideas.

    I think pretense about EMH and admittedly true anecdotal evidence leads market-friendly economists to be too dismissive of financial agents. Financial advisor stock picker. And, it may be worth paying a fee to manage risk. Without professional input, even intelligent people can get really out of whack. Even if an advisor takes 1% a year, think of the difference they make if they can make a young client more comfortable putting money into stocks instead of something less risky, like treasuries.

    The Dunning-Kruger effect looms large, for both clients and advisors, and I think fiduciary duty standards can actually be useful here, whether public or professionally administered (I highly recommend using a CFA certified advisor). But, treating this as an issue driven by stock-picker advisors misses the difficult and interesting factors at play, I think.

  12. Gravatar of Kevin Erdmann Kevin Erdmann
    7. September 2015 at 10:45

    There should be a “not equal” sign between financial advisor and stock picker. I think I accidentally put in html code that disappeared there

  13. Gravatar of W. Peden W. Peden
    7. September 2015 at 10:50

    You can see the problem as tricking people because of the number of reference classes involved: we know that the person is a woman, and that on the basis of this alone, the probability is 10/1000 = 1% that she has breast cancer. We know that she was tested, and this provides a more specific reference class, and using this reference class you’d think that this test is reliable in 91% of cases, so the probability would be 9% if you don’t take into account the distribution of the inaccurate diagnoses.

    The odd this is that you’d think that doctors would be good at putting together the relevant information to find the right reference class (since it’s a problem-solving/basic reading task) rather than the mathematics.


    That tests is general are accurate does not require adjusting upwards the probability for this test, because (on the information provided) the test is accurate, so there’s no reason to modify the posterior, unless you have strong background reasons to think that medical tests are usually MORE than 91% accurate etc.

    Even if the test wasn’t accurate, you would need reasons to think that medical tests are usually more accurate than we estimate. Perhaps I’m cynical, but my suspicion is the opposite!

    That’s granting what need not be granted, which is that it’s a question where you’re supposed to bring in your priors. Like Steve, I read it as a question where you’re just supposed to use the info. provided, and in that case 9/98 is the obvious answer. Priors are important when there are unorthodox reasoning tasks that are used to draw major psychological conclusions, as in behavioural economics; it’s tempting to infer that people are irrational when they might just have a different prior distribution from you.

    However, if you check the question carefully, it asks, “Which is the best answer?”, and 1/10 is closest to 9/98. I agree with Steve that 9/98 should have been an option, since I would hope that the sum to get the denominator isn’t too hard for doctors!

    Still, probabilistic reasoning is difficult, and most of us (myself included) are disadvantaged by the fact that English, like all natural languages, does not provide a good framework for talking probabilistically.

  14. Gravatar of Christian List Christian List
    7. September 2015 at 15:11

    Bashing doctors. I’m *loving* it.

    The study with the gynecologists is actually pretty famous around physicians. The original study is from 2007 and was conducted by Gerd Gigerenzer, a German psychologist.

    The original study is quite different to the *results* you mentioned. Gigerenzer gave 160 gynecologists the following information:

    Assume you conduct breast cancer screening using mammography in a certain region. You know the following information about the women in this region:
    – The probability that a woman has breast cancer is 1% (prevalence)
    – If a woman has breast cancer, the probability that she tests positive is 90% (sensitivity)
    – If a woman does not have breast cancer, the probability that she nevertheless tests positive is 9% (false-positive rate = 1-specificity)

    Then they got this task:

    A woman tests positive. She wants to know from you whether
    that means that she has breast cancer for sure, or what the
    chances are. What is the best answer?

    A. The probability that she has breast cancer is about 81%.
    B. Out of 10 women with a positive mammogram, about 9
    have breast cancer.
    C. Out of 10 women with a positive mammogram, about 1 has
    breast cancer.
    D. The probability that she has breast cancer is about 1%.

    21% have chosen C.

    Only after this test the same 160 gynecologists got some extra education and the information you mentioned:

    Assume you conduct breast cancer screening using mammography in a certain region. You know the following information about the women in this region:
    – Ten out of every 1,000 women have breast cancer
    – Of these 10 women with breast cancer, 9 test positive
    – Of the 990 women without cancer, about 89 nevertheless test positive.

    Now 87% of the doctors got it right.

    Now what’s the conclusion of all this? Gigerenzer concludes that we should use natural frequencies instead of conditional probabilities, because natural frequencies are way easier for our brains “to digest”.

    Gigerenzer mentions his study in detail for example in his book “Simply Rational” on page 24-25. The link is way too long. Just google parts of the test questions in Google Books and you’ll find it.

  15. Gravatar of Christian List Christian List
    7. September 2015 at 15:38

    I think Gigerenzer got a point. Our mind can process natural frequencies way easier than conditional probabilities.

    But I think your explanation is way better because it is more general. Every doctor got statistics during his college. We learn it, we passed the test the next day and after that we forgot most of it as soon as possible.

    This is quite natural. Things that you have learnt in the past but don’t use in a daily routine, are mostly forgotten quite fast.

    Both findings are nothing new, so I never understood why Gigerenzer got so much applause for his study.

    It might be because the subjects were gynecologists that are suppossed to perform well in such a situation. Gigerenzer *caught* them with their pants down.

    I bet you could do pretty much the exact same thing with scientists like Gigerenzer as long as it is a task that they don’t perform (in some form) in a near-daily routine.

  16. Gravatar of Topher Hallquist Topher Hallquist
    7. September 2015 at 16:20


    Fair points, but when make those arguments you’re no longer arguing that financial advising is basically astrology.

  17. Gravatar of W. Peden W. Peden
    8. September 2015 at 01:53

    Christian List,

    What you say is persuasive, but there’s still an interesting psychological point from the studies, as you summarise them, in that if you are correct then presumably we tend to think in terms of natural frequencies moreso than conditional probabilities.

    Arguably, we also usually think in terms of imprecise probabilities than real-valued probabilities, which is another reason to be careful with a lot of psychological work that assesses people in terms of real-valued Bayesian problems. Interpreted as a real world problem rather than a statistical exercise, you could argue that something like “85-95%” is the correct answer.

  18. Gravatar of ssumner ssumner
    8. September 2015 at 05:21

    Steve, Thanks, so 1/10 is the only answer even close.

    Ray, For me I find it easier to dispense with the equations, and just consider that 9 out of those 98 diagnosed with cancer, actually have cancer.

    Collin, Good point.

    Kevin, I agree that financial advisers play a useful role in managing risk. But there are lots of stock and mutual fund pickers in the financial industry. They should be telling their clients to just put the equity portion of their portfolios into index funds, and leave it at that. Those managing mutual funds should hold costs down by not trying to pick winners. That’s not what I tend to see.

    Christian, Good points.

    Topher, In this post I was specifically focusing on the part of regulation that says you must be licensed in order to recommend stocks. That part is like licensing astrology. The proposal you mention is a different type of regulation, and much more defensible.

  19. Gravatar of Njnnja Njnnja
    8. September 2015 at 06:21

    I think that there should be a little more care in dealing with the difference between “stock pickers” and “financial advisers.” But a couple pertinent facts to frame the issue better:

    Except in Wealth Management, financial advisors are generally not saying to their clients, “buy MSFT now!” At best, they are helping with asset allocation as others have described, and at worst, they are getting fat commissions for peddling overpriced options in things like insurance and commodities products to people who fear negative outcomes and are willing to pay a lot for protection. It really has little bearing on stock picking and EMH.

    So who is recommending individual stocks to retail clients? Wealth Management advisors and full service brokers. But ever since Charles Schwab set up shop you don’t have widows and orphans (or even moderately successful dentists and doctors) using full service brokers, and with internet trading even less so. So it is only the wealthy who “pay” for these services.

    One might object to even the wealthy getting ripped off by full service brokers but they aren’t really using them for professional advice. Actually, it is more like a country club membership that knits together a particular social group. For example, a full service broker might be able to get his wealthy client hard to get tickets to a big game. Or when Jim’s son, Johnny, gets a job in Wealth Management at a big investment bank, some of Jim’s friends start to use Johnny for “portfolio advice” a.k.a. getting on Jim’s good side. Or the owner of a privately held company will use a full service broker to get initial allocations on some IPOs in return for using the bank when his own company goes public.

  20. Gravatar of Doug M Doug M
    8. September 2015 at 09:30

    The SEC acts were passed it was to prevent fraudulent activities that were prevalent in the 1920’s. There has been very little that has been done to modernize the intent of these acts.

    There have been re-interpretations that had to be made due to technological change, (both info-tech and financial engineering). But the SEC acts are otherwise little changed.

    It is assumed that your broker understands what a mutual fund is, but no one should be under any illusion that a broker can pick a better fund than you can.

  21. Gravatar of Floccina Floccina
    8. September 2015 at 18:28

    On the MD’s would it not be better to state the results to them rather than depend on them to do the logic and math.

  22. Gravatar of Scott Freelander Scott Freelander
    8. September 2015 at 19:50

    Scott Freelander,

    Scott, you’ve really struck a nerve here for me. I’m a stockbroker and I’ve long felt FINRA should be abolished and that registration, rather than licensing, would make sense for investment advisers. That is, free online registration, just so those caught committing fraud can be stripped of a registration, if need be.

    FINRA exists for one reason, and that’s to protect the broker-dealers that fund it. This comes at the expense of consumers, especially those with modest means, and lower-level employees in the industry.

    FINRA has suitability requirements for recommendations, rather than fiduciary requirements, like registered investment advisors, the former of which allows brokers to recommend products they earn commissions on, while the latter does not.

    Also, there are certain types of investments only open to wealthier investors, like pre-market IPO opportunities or private placements, that simply leave smaller investors at a disadvantage.

    Also, regulation T, when it comes to free riding rules and margin limits, along with day trading rules, unnecessarily complicates investing.

    And when it comes to margin, it’s illegal to use a credit card, for example, to buy shares of stock, but many people start their own businesses, wholly or partly funded by credit cards. Which is more likely? That an upstart company will survive the first year or two or Apple and Google?

    I’m not saying it’s a good idea to buy stock with credit cards, but in a free market people should have the right to leverage anyway they want. Unfortunately, FDR and Congress at the time wrongly blamed Wall Street for the Great Depression. Investors have been paying a terrible price since.

  23. Gravatar of Scott Freelander Scott Freelander
    8. September 2015 at 19:56


    When it comes to recommendations for people to be in index funds, surely you’re aware that if the number of people you desire took that advice, they wouldn’t be good investments anymore. There has to be a role for arbitrage, otherwise there are no markets. There’s no incentive for anyone to actually try to allocate capital efficiently.

    That being said, more people should have more money in index funds.

  24. Gravatar of ssumner ssumner
    9. September 2015 at 07:15

    Njnnja, I specifically excluded financial advisers who put their client’s money into index funds.

    Doug, You said:

    “It is assumed that your broker understands what a mutual fund is, but no one should be under any illusion that a broker can pick a better fund than you can.”

    That’s right, but their job (as they see it) is to convince people they can do so.

    Scott, Excellent comment. I agree on index funds, which is why I don’t want Washington regulators to mandate that advisers put people into index funds, even though it’s often in the client’s interest. My point was just about regulation.

    (I don’t know if the Obama bill actually requires index funds (I presume it doesn’t) but that’s clearly the intent of the bill.)

Leave a Reply