On the limitations of empirical studies

Here's an excerpt from a blog entry by James Kwak, co-author with Simon Johnson at the Baseline Scenario:

I finally got around to listening to Planet Money’s interview with Russ Roberts from December. Russ Roberts and I are pretty sure to disagree on almost any actual policy question. But what I liked about his interview was that he basically admitted that policy questions cannot be settled by looking at the empirical studies. On whether the minimum wage increases or decreases employment for example, he says that he can poke holes in the studies whose conclusions he doesn’t agree with, but other people can poke holes in the studies he agrees with. In Roberts’s view, people’s policy positions are determined by their prior normative commitments.

I don’t completely agree. I don’t think that these questions, like the one about the minimum wage, are inherently unanswerable in the sense that the answer does not exist. But I agree that empirical studies are unlikely to get to the truth, particularly on a politically charged question, because there are so many ways to fudge an empirical study. As one of my professors said, there are a million ways you can screw up a study, and only one way to do it right. But I agree with the general sentiment. We are living in an age of numbers, where people think that statistics can answer any question. Statistics can answer any question, but they can answer it in multiple ways depending on who is sitting at the keyboard.

No comments: