Proof Negative

I have regularly argued that in investing, as in most things in life, disconfirmation is more valuable than confirmation (see here, for example). In other words, we learn more from what doesn’t work than from what does. That’s largely because induction is the way science advances.

We want deductive proof, but have to settle for induction. That’s because science never fully proves anything. It analyzes the available data and, when the force of the data is strong enough, it makes tentative conclusions. But these conclusions are always subject to modification or even outright rejection based upon further evidence gathering. The great value of data is not so much that it points toward the correct conclusion (even though it does), but that it allows us the ability to show that some things are conclusively wrong.

In other words, confirming evidence adds to the inductive case but doesn’t prove anything conclusively. Correlation is not causation and all that. Thus disconfirming evidence is immensely (and far more) valuable. It allows us conclusively to eliminate some ideas, approaches or hypotheses.

That said, we don’t like disconfirming evidence and we tend to neglect the limits of induction. Few papers get published establishing that something doesn’t work. Instead, we tend to spend the bulk of our time looking (and data-mining) for an approach that seems to work or even for evidence we can use to support our preconceived notions.

We should be spending much more of our time focused upon a search for disconfirming evidence for what we think (there are excellent behavioral reasons for doing so too). But we don’t, as illustrated by the following question (a variation of the Wason selection task).

Most people answer with E and 4, but that’s wrong. For the posited statement to be true, the E card must have an even number on the other side of it and the 7 card must have a consonant on the other side. It doesn’t matter what’s on the other side of the 4 card. But we turn the 4 card over because we intuitively want confirming evidence. And we don’t think to turn over the 7 card because we tend not to look for disconfirming evidence, even when it would be “proof negative” that the given hypothesis is incorrect. In a variety of test environments, fewer than 10 percent of people get the right answer to this type of question.

I suspect that this cognitive failing is a natural result of our constant search for meaning in an environment where noise is everywhere and signal vanishingly difficult to detect. Randomness is difficult for us to deal with. We are meaning-makers at every level and in nearly every situation. Yet, as I have noted often and as my masthead proclaims, information is cheap while meaning is expensive and elusive. Therefore, we tend to short-circuit good process to get to the end result – typically and not so coincidentally the result we wanted all along.

As noted above, science progresses not via verification (which can only be inferred) but by falsification (which, if established and itself verified, provides relative certainty only as to what is not true). Thank you, Karl Popper. In the investment world, as in science generally, we need to build our investment processes from the ground up, with hypotheses offered only after a careful analysis of all relevant facts and tentatively held only to the extent the facts and data allow. Accordingly, we need always to be on the look-out for disconfirming evidence — proof negative — even though doing so is oh so counterintuitive pretty much all the time.

Advertisement