More on replicating results

The New Yorker article I mentioned last post has had me thinking more about this idea that economics is not terribly well-suited to the increasingly questionable rigors of the scientific method. Poking around on the interwebs led me to a paper by Daniel Hamermesh of UT Austin, who some bio reading indicates is something of an authority on some of the mores/norms of the economics profession. The paper, entitled “Viewpoint: Replication in Economics” (link is a PDF download) discusses the evolution of authors making available their data for examination and replication and the incentives and lack thereof for actually doing so. The first part seems to be going rather well, with the American Economic Review making inclusion of data on a companion website mandatory for some years now and the author finding a general increase in the sharing of such data from a survey of requests and their timely fulfillment.

However, his main finding from surveys of a few journals is of the extreme paucity of attempts at replication by anyone. As he puts it, “Economists treat replication the way teenagers treat chastity – as an ideal to be professed but not to be practised.” Hamermesh attributes this to a number of disincentives that include:

-Journals not at all incentivizing researchers to do such relatively mundane follow up by being institutionally more interested in further novel new work which will increase readership and prestige.

-The opportunity cost to researchers of taking the time to simply replicate a study rather than working on their own material. This is considerable for any busy researcher who is likely already facing considerable time constraints regarding what to work on.

-The difficulty in “replicating results” when there is “pure replication” i.e. using the same data set and methodology and also “scientific replication” where an attempt is made to procure a similar data set and use the same methodology, but which method may have marginal value based on singular economic conditions and mitigating circumstances surrounding the tested hypothesis.

This led me to think of a few recent comments (as the author generally terms rebuttals that are not strictly speaking “replications”) back and forth between economists on some papers discussing the influence / danger of government debt size on GDP growth. In both of these cases, what was at odds was really the assumptions made on the data.

Here are some links to the recent spats I’m referring to:

1) Mike Konczal and Arjun Jayadev putting (what I must admit to thinking is) a pretty good smackdown on a paper by Alberto F. Alesina and Silvia Ardagna paper (“Large Changes in Fiscal Policy: Taxes Versus Spending” 2009) that seems constantly cited by the goldbug / government-is-the-problem set. It purports to show that cutting government spending relative to GDP leads in increased economic growth. They effectively slay this paper by pointing out pretty huge fallacies of composition in the examples chosen by the author:

New Working Paper: “The Boom Not The Slump: The Right Time For Austerity”

2) Next up is Paul Krugman taking on Kenneth Rogoff and Carmen Rienhart’s influential paper “Growth In A Time Of Debt” (2009) for much the same issue, choosing to find a common thread among many disparate and largely non-correllated historical examples by implying that governments with a debt ratio at or above 90% of annual GDP will experience lower to non-existent growth. Krugman also rightly points out the lack of any real attempt to show causation running the way the authors posit and states quite sensibly that it may just as easily run the other way (i.e. slow growth leads to large government deficits, i.e. stimulate the economy and you won’t find yourself drowning in government debt). Numerous others have attacked this fairly dubious use of highly heterogeneous historical examples to draw what seems a very arbitrary conclusion.

Krugman Reinhart and Rogoff Are Confusing Me – NYTimes

Yeva Nersisyan – More Reasons To Doubt Rogoff and Reinhart

Let’s call it “selective reporting.” At least Jonah Lehrer does in the New Yorker piece. Discussing the sort of “fad” of doing studies on fluctuating asymmetry (whereby in a nutshell during mating and other processes relating to natural selection, selectors from birds to humans tend to favor symmetry in their choice as a proxy for indicating “good genes”) one of the biggest casualties of “the decline effect”:

Palmer emphasizes that selective reporting is not the same as scientific fraud. Rather, the problem seems to be one of subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results. Stephen Jay Gould referred to this as the “shoehorning” process. “A lot of scientific measurement is really hard,” Simmons told me. “If you’re talking about fluctuating asymmetry, then it’s a matter of minuscule differences between the right and left sides of an animal. It’s millimetres of a tail feather. And so maybe a researcher knows that he’s measuring a good male”—an animal that has successfully mated—“and he knows that it’s supposed to be symmetrical. Well, that act of measurement is going to be vulnerable to all sorts of perception biases. That’s not a cynical statement. That’s just the way human beings work.

This seems to be much more the norm than the exception in the economics profession. Institutional affiliation rarely disappoints when it comes to POV, so choose carefully. It’s also a strong indictment of the hands-off approach with regards to morality displayed by many economic “scientists.” Morality is what makes the reporting of economists selective, the sad part is how many economists cannot come to terms with that fact.

From another recent New Yorker essay by Adam Gopnik (October 18th 2010) on Adam Smith and the nearly constant misreading of his celebrated invisible hand in The Wealth of Nations. Gopnik points to the importance of reading this seminal book in tandem with Smith’s The Theory of Moral Sentiments, which shows clearly that Smith viewed economics as a subset of philosophy and also the primacy Smith placed on morally informed actors in achieving functional markets. Gopnik:

Smith’s real question, it turns out, was not the economist’s question, How do we get richer and poorer?, or even the philosopher’s question, How should one live? It was the modern question, Darwin’s question: How do you find and make order in a world with God?

…All those moral judges are what let the invisible hand act. Our faith, before we get to the market, that our fellow-man is like us and will seek to bargain and wheedle as we do … is primary. The narrow animal instinct is not to trade and exchange and invest; it is to hoard and guard and pillage. The acquired human trait is the market trait, and it depends on trust, sympathy. To see what happens … if there is not … a network of trust, already in place, ‘privatization’ just produces kleptocracy [as recently in the developing world].

…What moved men to make markets was ultimately their love of pleasure and happiness, and who, Smith wondered, could live happily in a society where all the wealth has been confiscated and kept in a few hands? He believed not that markets make men free but that free men move toward markets. The difference is small but decisive; it is most of what we mean by humanism.

Indeed, it seems as if Smith would get a hearty laugh from the idea of economics as a hard science with the attendant hand-wringing over replication rather than simple Socratic analysis, but the fraying camouflage of the scientific method has been used for most of the last century to disguise the selective reporting of economists as they have buttressed the pillars of injustice as the peer-reviewed result of rigorous science.

About theunlikelyeconomist

theunlikelyeconomist is a 41-year avid economics, news and politics addict. I am currently in the midst of a long-delayed attempt to attain a college degree after 18 years of audio engineering. I currently co-own and master records and CDs at Chicago Mastering Service in Chicago, IL.
This entry was posted in Uncategorized and tagged , , , , , . Bookmark the permalink.

One Response to More on replicating results

  1. Pingback: How did I get here? | theunlikelyeconomist

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s