The Importance of Reporting the Null

If there’s one thing I took from Quantitative Methods I it’s that findings indicating a null hypothesis aren’t findings at all, and are rarely (if ever) publishable. But why? Aren’t null findings just as important as finding a relationship that is statistically significant? By reporting these null findings, future researchers will know what relationships don’t work. This could save millions in wasted research funds and hours.

In international development, we encounter something similar to the lack of null reporting in research. Rarely, if ever to organizations publish information about projects that can be considered failures. However, if this information was freely available – say an assessment of the project and what went wrong – well both the organization itself and others in the field would know which projects simply don’t work in certain contexts or what pitfalls to avoid when implementing similar projects.

So why are we so afraid of admitting failure when it could be beneficial both to ourselves and to our colleagues? One reason could be that failure simply isn’t sexy. Organizations fear that by publishing a null finding or a unsuccessful result, financial support for similar future ventures might evaporate. It’s a mindset we need to change.

But there is hope. Charles Kenney at the Center for Global Development published a blog yesterday about movement within USAID to be more transparent about project failures and to encourage better reporting from the field when a project is going poorly. He shares some interesting ideas in the column. In particular, the idea that project managers who report problems with project implementation from the onset will be allowed to implement a similar project in the same country with the remaining project funds should USAID decide to cancel the failing project. Of course there will be issues regarding moral hazard with this approach to reporting – PMs could be inclined to report failures on projects they simply don’t like so that they can craft a project they do like in the same country. All in all, though, it sounds like a fair approach in a field where PMs typically sign on for short-term contracts and have little job security after the contract is finished.

Another concept discussed in Kenney’s blog is a USAID FailFaire, where USAID reports various projects that have failed and the reasons why they did not work. This also sounds like an excellent idea to encourage transparency and to allow for better reporting on what does and does not work.

Personally, I am a bit skeptical about whether or not these improvements in reporting will come about. First, Kenney supports short-term appraisal of project success, but things take time in the developing world. How short is short-term? Without giving projects and PMs enough time to mature, it’s dangerous to consider a project a failure. Projects take months to get off the ground. A 5-year contract might not see any real progress until the 5th year, or even after the project is completed. Learning what works in development requires patience. Too many times innovations have been given the stamp of approval too early and were later shown to be ineffective.

Second, the current budgeting climate in Washington certainly does not encourage negative reporting from federal agencies. In the scramble for funding, agencies have more of an incentive than ever to exaggerate their successes and request future funding based on their efficiency and effectiveness. USAID is not immune to these pressures.

Still, the dialogue is a step closer to the real thing. I am hopeful that in the future there will be a forum for all major players in development to discuss both successful and failed projects for the enhancement of the overall success of development worldwide. Now we just need to work on getting that Journal of All Things Null off the ground.

(H/T xkcd for comic)


One Reply to “The Importance of Reporting the Null”

  1. This is actually fairly disturbing to me.

    Then again, I am used to medical research and reports that report generally most details involved with outcomes. Reporting both successful and unsuccessful results allows us to see the true effectiveness of interventions, no matter what field you’re in. The idea that international research is afraid to take a non conclusive result and publish it is appalling.

    It is mainly so because it is a lack of effort on the part of the person writing up the research to be published. They could have easily found things that could have altered their results from the environment to account for the failure and included them in the publication. This would, 1) cover their ass, and 2) allow for future researchers to account for this issue, and 3) show that are actually good researchers that notice these things.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s