Proof Negative

DisconfirmationI have regularly argued that in investing, as in most things in life, disconfirmation is more valuable than confirmation (see here, for example). In other words, we learn more from what doesn’t work than from what does. That’s largely because induction is the way science advances.

We want deductive proof, but have to settle for induction. That’s because science never fully proves anything. It analyzes the available data and, when the force of the data is strong enough, it makes tentative conclusions. But these conclusions are always subject to modification or even outright rejection based upon further evidence gathering. The great value of data is not so much that it points toward the correct conclusion (even though it does), but that it allows us the ability to show that some things are conclusively wrong.

In other words, confirming evidence adds to the inductive case but doesn’t prove anything conclusively. Correlation is not causation and all that. Thus disconfirming evidence is immensely (and far more) valuable. It allows us conclusively to eliminate some ideas, approaches or hypotheses.

That said, we don’t like disconfirming evidence and we tend to neglect the limits of induction. Few papers get published establishing that something doesn’t work. Instead, we tend to spend the bulk of our time looking (and data-mining) for an approach that seems to work or even for evidence we can use to support our preconceived notions.

We should be spending much more of our time focused upon a search for disconfirming evidence for what we think (there are excellent behavioral reasons for doing so too). But we don’t, as illustrated by the following question (a variation of the Wason selection task).

Disconfirming

Most people answer with E and 4, but that’s wrong. For the posited statement to be true, the E card must have an even number on the other side of it and the 7 card must have a consonant on the other side. It doesn’t matter what’s on the other side of the 4 card. But we turn the 4 card over because we intuitively want confirming evidence. And we don’t think to turn over the 7 card because we tend not to look for disconfirming evidence, even when it would be “proof negative” that the given hypothesis is incorrect. In a variety of test environments, fewer than 10 percent of people get the right answer to this type of question.

I suspect that this cognitive failing is a natural result of our constant search for meaning in an environment where noise is everywhere and signal vanishingly difficult to detect. Randomness is difficult for us to deal with. We are meaning-makers at every level and in nearly every situation. Yet, as I have noted often and as my masthead proclaims, information is cheap while meaning is expensive and elusive. Therefore, we tend to short-circuit good process to get to the end result – typically and not so coincidentally the result we wanted all along.

As noted above, science progresses not via verification (which can only be inferred) but by falsification (which, if established and itself verified, provides relative certainty only as to what is not true). Thank you, Karl Popper. In the investment world, as in science generally, we need to build our investment processes from the ground up, with hypotheses offered only after a careful analysis of all relevant facts and tentatively held only to the extent the facts and data allow. Accordingly, we need always to be on the look-out for disconfirming evidence — proof negative — even though doing so is oh so counterintuitive pretty much all the time.

26 thoughts on “Proof Negative

  1. Pingback: “So you’re telling me there’s a chance” | Above the Market

  2. In the above card problem you also need to turn up the K card; because if the K card has a vowel on its opposite side, the hypothesis being tested will be proved false.

  3. Pingback: 10 Monday PM Reads | The Big Picture

  4. Fewer than 10 percent of people get the right answer because you asked the wrong question…You have to be very careful how you phrase statements like these…

    I would parse the statement as “There exists a card with a vowel on one side and an even number on the other side”, in which case the correct strategy is to turn E and 4.

    What you meant to write is probably “, For all cards, If a card has a vowel on one side, then it has a number on the other side”. If this is what you had in mind, then the correct answer is indeed to turn E, and then turn 7 to check the contrapositive statement. But, this is not what is written, nor it is how the statement as written should be interpreted.

    Your line of reasoning to make your point was very confusing…

      • “Subjects are aware that on the particular set of cards, each one has a letter on one side and a number on the other side.” You left that part out so I would say Neal is technically right. I assumed that part, so I was technically wrong too. On the other hand, I was nice enough to play along as you intended until Neal pointed out my error; that’s got to count for something.

      • To be fair to Neal and Hugo, the original question as stated via your link, has an added constraint that, ‘each one has a letter on one side and a number on the other side’ – hence why Neal rightfully posited that you must turn over the card K, if you strictly follow they way you constructed the same card question.

        That aside, I agree with the points you’ve raised regarding falsification – confirmation bias is an easy trap to fall into.

      • Neither of you are right, actually. The stated manoeuvre doesn’t tell you what you will find if you come across all the vowels except E. Turning over the 7 doesn’t prove anything useful in terms of the statement. I agree that it is pointless to turn over the 4 but somebody needs to convince me of the validity of the given statement, seeing that no action gives us any idea of what is on the other side of all the other vowels. Comments, please…

      • No, Neal is correct. You framed the question incorrectly. Note that the question in the link you provide includes the condition “each one has a letter on one side and a number on the other side”.

        Without that condition you must turn the K card for exactly the same reason that you must turn the 7.

      • Bob, I hope you wont mind me pointing out that the mistake you made here was exactly the kind of mistake that you warned against in your article. You brushed aside the negative evidence (Neal’s comment) and looked for confirming evidence (the link you provided).

        The comic irony of this situation shows just how difficult it is to avoid the mistake you described in the article. That only underlines the importance of the article and shows that it applies to us all equally.

  5. i’m not sure neal is wrong… you might have dropped this statement from the original problem “Subjects are aware that on the particular set of cards, each one has a letter on one side and a number on the other side.” I could be wrong so I’ll spare the attitude.

  6. Pingback: Best of the Web: 14-08-12, nr 1041 | Best of the Web

  7. Sorry Mr Seawright; I still think Neal’s comment is correct (and quite astute). From your link to the Wason Selection test, there is an additional piece of information given which is not present in your article: “Subjects are aware that on the particular set of cards, each one has a letter on one side and a number on the other side”. Now try reading Neal’s comment again.
    As always, the devil’s in the detail.

    • Correct, Solon. As stated by Seawright, the ONLY correct answer is E and 4. There is no reason to know what is supposed to be on the other side of 7 or K; it could be 9 and W, respectively. Or clown faces.

      • I was just a repair mechanic on industrial machinery. All 4 cards must be turned over and checked or the exercise is meaningless. As Lloyd Clucas said, there could be anything including something irrelevant or a blank on any one of the reverse faces of the cards and that would negate the whole exercise. And I have seen ‘tests’ like this where something like that was done deliberately.

  8. Pingback: 10 Monday PM Reads | FAN FICTION PLANET

  9. Pingback: The New Headline Indicators - A Wealth of Common SenseA Wealth of Common Sense

  10. Pingback: The New Headline Indicators | PRAGMATIC CAPITALISM

  11. Pingback: Saturday links: extreme scarcity | Abnormal Returns

  12. Pingback: A Dash of InsightWeighing the Week Ahead: Time for Central Bankers to Unwind? -

  13. Pingback: Weighing The Week Ahead: Time For Central Bankers To Unwind? | Social Dashboard

  14. Pingback: Math is Different | Above the Market

  15. Pingback: Horrid Facts, Stubborn Facts | Above the Market

  16. Pingback: Underestimating the Density of the Fog | Above the Market

  17. Pingback: Diruangan classrom | aperiusndruru

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s