How Not to Be Data-Driven

Earlier today I wrote about the nature of scientific progress — how it isn’t always linear and incremental.  Sometimes it moves in drastic and revolutionary ways, dramatically shifting the fundamental working paradigm of the subject. I then related this idea (from Thomas Kuhn’s seminal work, The Structure of Scientific Revolutions) to economics and markets, recognizing that this is not “hard” science, making the application of the idea at least a bit problematic.

That post got me to wondering how fact-based (or as I say with respect to this blog’s aspirations, how data-driven) economists are when developing their ideas and hypotheses.  At first brush, my question seemed too cynical by half.  But a bit of research suggests that I may not be nearly cynical enough.

Nobel Prize winning economist Gary Becker of the University of Chicago is famous for seeking to apply the economic concept of utility maximization essentially everywhere, so as (or so the claim goes) to understand virtually all human behavior.  Despite the gains of behavioral economics and just plain common sense, Becker and his ilk see inherent rationality behind essentially every damaging and ridiculous human endeavor. 

In that context, Ole Rogeberg and Hans Melberg surveyed a group of economists in a particular area of study to examine the extent to which they looked to actual empirical tests of a theory’s predictions to evaluate the success of that theory. It sounds like the answer ought to be right out of The Scientific Method for Dummies, right? Of course the data controls. Unfortunately, the answer was that actual data didn’t matter all that much to the economists surveyed — internal consistency (elegance?) seems to have been much more important.

Rogeberg and Melberg looked specifically at the literature of “rational addiction,” which postulates that those who suffer life-destroying addictions (such as an addiction to heroin) are actually acting in what they perceive to be their own best interest. Intuitively, I would expect the evidence needed to support this idea would be pretty high.  But my intuition seems to be clearly in error.  Ironically, the facts simply don’t support it. 

Instead, the surveyed economists’ ignoring actual evidence to adhere to elegant theories (dare I say preconceived notions) led to what Rogeberg and Melberg concluded were “absurd and unjustified claims being made and accepted in even highly ranked journals.”  As their abstract states:  “A majority of the respondents believe the literature is a success story that demonstrates the power of economic reasoning. At the same time, they also believe the empirical evidence to be weak, and they disagree both on the type of evidence that would validate the theory and the policy implications.”

Significantly, this isn’t a matter of people interpreting the evidence differently.  Indeed, in many cases evidence isn’t even gathered:

“The core of the causal insight claims from rational addiction research is that people behave in a certain way (i.e. exhibit addictive behavior) because they face and solve a specific type of choice problem. Yet rational addiction researchers show no interest in empirically examining the actual choice problem – the preferences, beliefs, and choice processes – of the people whose behavior they claim to be explaining.”

In fact, supporting evidence was deemed unnecessary (in that it likely doesn’t exist):  “Becker has even suggested that the rational choice process occurs at some subconscious level that the acting subject is unaware of, making human introspection irrelevant and leaving us no known way to gather relevant data….”  In other words, “a choice problem people neither face nor would be able to solve prescribes an optimal consumption plan no one is aware of having.  The gradual implementation of this unknown plan is then claimed to be the actual explanation for why people over time smoke more than they should according to the plans they actually thought they had.” 

Unless I’m making some glaring error or omission here, I can only conclude that the entire “rational addiction” enterprise is simply nuts.  It should be a good reminder for us always to check and re-check both our assumptions and our data, perhaps especially when it comes from economists.  Theories can be helpful, surely.  But unless and until they are actually supported by evidence, they must remain irrelevancies — elegant irrelevancies perhaps, but irrelevancies nonetheless.

Advertisement