Saturday, May 23, 2009

inference has to be done from model-based extrapolation rather than from clean "natural experiments" whose results we can all agree on

TO BE NOTED: From Statistical Modeling, Causal Inference, and Social Science:

"
Difficulty of forecasting, and of causal inference when n=1
| 9 Comments

Greg Mankiw has a nice little discussion of the difficulty of evaluating the effects of interventions in the n=1 setting:

stimulus-vs-unemployment-april.gif

As Mankiw points out, the bad news about the unemployment rate is bad news with or without the recovery plan and thus--although it certainly seems to knock down the predictions shown in that graph--it does not provide much information on the causal effect of the fiscal stimulus. Especially given that the graph comes from a report released in early January, before anyone knew what would end up being included in the final version of the stimulus plan.

As Mankiw writes, this is "a reflection of the inherent uncertainties associated with macroeconomics." It's a tough problem, because decisions have to be made, and these decisions will implicitly or explicitly be based on causal inference. It's just that much of this inference has to be done from model-based extrapolation rather than from clean "natural experiments" whose results we can all agree on.

P.S. I'm wondering a little bit about the placements of the points on the graph. First, I'd recommend against using triangles as a graphing symbol! Is the exact value supposed to be in the middle of the triangle, or maybe the top vertex? I think dots would be better. Second, I can't quite follow the x-axis. The graph places March halfway between Q1 and Q3, but is that right? I'd think that halfway between Q1 and Q3 would be the center of the year, that is, June/July. But I could be missing something here; I don't have experience working with this sort of data.

P.P.S. This is an interesting example of a graph where perhaps it's actually appropriate for the y-axis not to go down all the way to zero, if we accept 3% as some sort of lower bound on unemployment in normal circumstances."

No comments: