I joke with my graduate students they need to get as many technical skills as possible as PhD students because the moment they graduate it's a slow decline into obsolescence. And of course by "joke" I mean "cry on the inside because it's true".

Take experiments. Every year the technical bar gets raised. Some days my field feels like an arms race to make each experiment more thorough and technically impressive, with more and more attention to formal theories, structural models, pre-analysis plans, and (most recently) multiple hypothesis testing. The list goes on. In part we push because want to do better work. Plus, how else to get published in the best places and earn the respect of your peers?

It seems to me that all of this is pushing social scientists to produce better quality experiments and more accurate answers. But it's also raising the size and cost and time of any one experiment.

This should lead to fewer, better experiments. Good, right? I'm not sure. Fewer studies is a problem if you think that the generalizabilty of any one experiment is very small. What you want is many experiments in many places and people, which help triangulate an answer.

The funny thing is, after all that pickiness about getting the perfect causal result, we then apply it in the most unscientific way possible. One example is deworming. It's only a slight exaggeration to say that one randomized trial on the shores of Lake Victoria in Kenya led some of the best development economists to argue we need to deworm the world. I make the same mistake all the time.

We are not exceptional. All of us---all humans---generalize from small samples of salient personal experiences. Social scientists do it with one or two papers. Usually ones they wrote themselves.

The latest thing that got me thinking in this vein is an amazing new paper by Alwyn Young. The brave masochist spent three years re-analyzing more than 50 experiments published in several major economics journals, and argues that more than half the regressions that claim statistically significant results don't actually have them.

My first reaction was "This is amazingly cool and important." My second reaction was "We are doomed."

Read more of this post