RSS

RCTs and aid effectiveness: Much to be said

In @Saundra_S’s absence, it’s only fitting that someone step in to compile and share the barrage of articles that have been regarding or related to RCTs in the past month or so due to the release of More Than Good Intentions: How a New Economics is Helping to Solve Global Poverty and Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty. En homage to Good Intentions Are Not Enough, here’s my attempt to do so, in no particular order. Please do share additional posts and articles via the comments section.

I am also including at the bottom the content of Friday’s “tweet debate” regarding RCTs that demonstrates some of the polarities in thinking, to which I hope I’ve done justice to those who participated.

Here’s to the continued conversations…

***

Dean Karlan’s Q&A on the Freakonomics blog and with Jacob Appel in Foreign Policy

World Bank’s Development Impact Portal: “Dean Karlan’s New Book: RCTs – this time it’s personal” and “Stuff You Cannot Randomize” and “Esther Duflo’s refreshing perspective on fighting poverty” and “Is it the program or is it participation? Randomization and placebos

Boston Review: “Small Changes, Big Results,” a forum on applying behavioral economics to global development.

Tom Murphy in the Huffington Post: Proven Impact: Better International Development Through Rigorous Research

Innovation for Poverty Action’s blog: “Helping Ugandan families save for school fees” and “What should NGOs do when they can’t run an RCT?” and “RCTs: The Backlash

Nicholas Kristof in the New York Times: “Getting Smart on Aid

Arvind Subramanian, Center for Global Development: “Nicholas Kristof and Aid

The Wall Street Journal: Is ‘Nudging’ really enough? and Bill Easterly’s review of More Than Good Intentions

Owen Barder: “Do Economists Have Better Tools?

Rachel Strohm: “Development Aid in Context

Ben Goldacre in The Guardian: “How can you tell if a policy is working? Run a trial

Madeleine Bunting in The Guardian: “Duflo and Banerjee take the guesswork out of policies that help the poor” (on the glaring omission of an understanding of power)

Guardian Editorial: “In praise of…Esther Duflo and Abhijit Banerjee

Duncan Green of Oxfam GB reviews Poor Economics and guest post by Karl Hughes “Can we demonstrate effectiveness without bankrupting our NGO and/or becoming a randomista?

Rick Davies on MandE News “Do we need a minimum level of failure?” and “Relative rather than absolute counterfactuals: A more useful alternative?

Ian Thorpe: “The problem with the big picture

Tales from the Hood on the necessity of “good vibes”

Chris Blatmann: “Go Short on Randomized Control Trials?” and “Aspiring PhD students: Should you become a field research assistant for an RCT?” and “Development Aid 3.0?

Development Policy Blog: “Why abundance evidence won’t inevitably lead to abundance

Lawrence Haddad of IDS: “A New Harvest of RCTs?

Find What Works’ series on the Limitations of RCTs – Part 1, Part 2, Part 3, Postscript

Good Intentions Are Not Enough: “’More Than Good Intentions’ and ‘Three Cups of Tea’

OpenAid: “Randomized trials are NOT the hottest thing in the fight against poverty

Ben Ramalingam at ODI: “When will we learn how to learn?

Tate Watkins at Short Sentences: “Why most aid won’t be ‘good aid’

Edward Carr: “On explanation in development research” and “The Qualitative Research Challenge to RCT4D

Interaction Institute for Social Change: “95% Failure

Mike Klassen at Doorway to the Universe: “At Second Glance

GiveWell blog: “Suggestions for the Social Sciences

Marc Bellemare: “Thoughts on the RCT Debate

Chasing Fat Tails: “Don’t Bite the Silver Bullet

David Week: “Who guards the RCT guardians?

The Monkey Cage: “Why aren’t there institutional review boards for the development industry?

Centro Studi Luca d’Agliano: “Lentils for Vaccines” and “Dangerous Liaisons: On the ambiguous relationship between development and development economics

Stuff Expat Aid Workers Like: “#60 Randomized Controlled Trials

Dennis Whittle: “If not randomized trials, then what?

Ian Thrope: “The development ideas hype cycle

Rasmussen, Malchow-Møller & Andersen, University of Southern Denmark: “Walking the talk: the need for a trial registry for development interventions

Mark Engler: “On Nicholas Kristof’s Boneheaded ‘Paean to Economists‘”

Kate McKee: “Why Poor People Don’t Use Savings Accounts: Can qualitative research (RCT’s homely stepsister) help answer the question?

Philip Auerswald: “Why Randomized Controlled Trials Work in Public Health…and Not Much Else.”

Tim Ogden: “Revisiting Objections to Randomized Control Trials” and “Making RCTs Better” (love the call for humility)

David Roodman: “The Smartest RCT Critic

Marc Maxson: “What Aid Experts Could Learn From Fantasy Football

14 Points: “Randomized Control Trials on Trial: Evaluating the Efficacy of RCTs” and “Vying with Velker: RCTs Reconsidered

Brendan Rigby: “The decline effect: a storm on the horizon for RCTs?

VOX, Anders Olofsgård: Why political short-sightedness and randomised control trials can be a deadly mix for aid effectivness

How Matters: “RCTs: A band-aid on a deeper issue? and “RCTs: some ‘how matters’ advice for donors”

***

@Dochasnetwork May 19, 10:29pm @intldogooder @owenbarder I’ve a huge problem with Randomised Trials. Immoral, in my book. Overkill. Measuring Outcomes should suffice.

@intldogooder 4:18am Moral ? is how to justify control group that doesn’t get services & how they are later brought into progs @dochasnetwork @owenbarder #RCTs

@viewfromthecave 4:23am @intldogooder From my understanding of RCTs there are multiple ways of dealing with ethics of control groups. One is lack of funds which…means that only so many can get the services, so some will naturally be without. If enough resources for all, interventions…can be gradually timed to get a comparative, but not deny services. Or all can get different kinds of services, etc…Maybe @poverty_action will be kind enough to post on the ethics of RCTs and determining control groups…

@intldogooder 4:55am: The “how” discussion is important given attn RT @viewfromthecave: Maybe @poverty_action will be kind enough to post on the ethics of #RCTs

@txtpablo 7:10am Yes, pls

@DevEconHealth 4:31am @intldogooder @dochasnetwork @owenbarder Not sure that is a serious moral ?. Same issue with any proj trgt group and med. trial.

@intldogooder 4:55am Counter-balance to Kristof’s article @HumDevNGOs: better ways to measure complex #socialchange http://bit.ly/ilPqfm #complexity #smartaid

@viewfromthecave 5:24am via @intldogooder You read Karlan’s book, right? [Ethics] is addressed a bit in the beginning

@intldogooder 5:33am @viewfromthecave I did and you’re right, they include that & many other caveats. Important for me to keep beating the “how matters” drum…Have seen such poorly-executed evals/research in the field that I have visions of that now for wide-scale #RCTs

@poverty_action 6:32am @intldogooder @dochasnetwork how about the ethics of potentially wasting limited money on unproven projects?

@intldogooder 7:00am Also vital but deficit mentality has its drawbacks RT @poverty_action: @intldogooder @dochasnetwork ethics of potentially wasting limited $?

@poverty_action 7:04am @intldogooder which are..?

@intldogooder 7:19am @poverty_action econ tools “often used unhelpfully due to hubris & a shocking level of ignorance about being studied” http://ow.ly/4Z9UN

@poverty_action 7:19am @txtpablo @viewfromthecave @intldogooder any ethical issues with using control group for medical trials? Or should we just hope for the best

@intldogooder 7:20am @poverty_action thus why sharing more of your experience on HOW to do #RCTs well so important, many of us grounded in assets-based approach

@txtpablo 7:21am @poverty_action @viewfromthecave @intldogooder Well, let us know! For sure there are some discussions around.

@poverty_action 7:27am @intldogooder we run trainings for partner orgs & run exec ed with J-PAL http://bit.ly/mwVNkY this WB book is great http://bit.ly/iP5848…but always grateful for suggestions of how we can do more!

@intldogooder 7:33am Lets acknowledge differences btwn social change & medical science @txtpablo @poverty_action @viewfromthecave re: ethics of control groups

@poverty_action 7:37am @intldogooder @txtpablo @viewfromthecave true, but not an argument against striving for the greatest rigor in evaluation possible

@txtpablo 7:40am @poverty_action: @intldogooder @viewfromthecave agree, but rigor in #evaluation is not only RCT, it is much more

@poverty_action @txtpablo there are other rigorous methods but RCTs generally agreed to be the best way of attributing causal impact

http://bit.ly/...

@txtpablo @poverty_action I agree it is good for causal impacts but in simple contexts, where you can manipulate few variables, but in complex ones?

@poverty_action @txtpablo do you have a specific example you are thinking of?

@txtpablo @poverty_action what causes a revolution? what causes development in a region? RCT only tells us part of what we need to know

@poverty_action @txtpablo spot on. Lots of important questions that RCTs can’t answer. But for the ones they can, they are best method going.

@intldogooder 7:41am Does genuine learning on an individual level always come from rigor? RT @poverty_action: @intldogooder @txtpablo @viewfromthecave @itsjina

@BonnieKoenig 7:44am @intldogooder I don’t believe in ‘always’, ‘best’ etc.. See previous tweets @poverty_action @txtpablo @viewfromthecave…Yes these differences r impt 4 diff situations

@viewfromthecave 7:42am @intldogooder Both impact livelihoods of many and known solutions can lead to improvement for many in the future #DevilsAdvocate

@intldogooder 7:47am At times this is like atheism vs. faith debate. What is comfort level with not knowing? @txtpablo @poverty_action @viewfromthecave @itsjina

@sabes_a_arim 7:50am @intldogooder what?

@viewfromthecave 7:50am @intldogooder There has to be comfort in not knowing everything, but that does not mean we cant know more @txtpablo @poverty_action @itsjina

@poverty_action 7:53am @viewfromthecave @intldogooder @txtpablo @itsjina will always be unknowns, but not knowing what works to fight poverty IS uncomfortable

@ithorpe 7:58am @poverty_action @viewfromthecave @intldogooder @txtpablo @itsjina exchange reminded me of “The truth is out there” http://wp.me/p110pY-12

@intldogooder 8:12am What works is investing in people, not just ideas- must pay attn to that 2 @poverty_action @viewfromthecave @intldogooder @txtpablo @itsjina

@viewfromthecave 8:25am @intldogooder no disagreement on that.

@poverty_action 8:15am @intldogooder we should test that theory….

***

Related Posts

Remembering George Harrison on the road to Busan

Aid, Africa, Corruption and Colonialism: An Honest Conversation

Egypt’s Thundering Wave & Questions for Development Aid

Confessions of a Recovering Neocolonialist

Changing the aid system: 5 more ideas from the inside

Reuters: New report, model of best practice for aid world


3 Comments Add Yours ↓

  1. Pablo Rodriguez-Bilella #
    1

    I have tried to copy and paste some of the interchange we had with @poverty_action last Friday, which I think it is missing in the post (sorry if it is not the case). I wonder also if there is a SIMPLE method for saving and ordering tweets!

    txtpablo @poverty_action: @intldogooder @viewfromthecave agree, but rigor in
    #evaluation is not only RCT, it is much more

    poverty_action @txtpablo there are other rigorous methods but RCTs generally agreed to be the best way of attributing causal impact
    http://bit.ly/

    txtpablo @poverty_action I agree it is good for causal impacts but in simple contexts, where you can manipulate few variables, but in complex ones?

    poverty_action @txtpablo do you have a specific example you are thinking of?

    txtpablo @poverty_action what causes a revolution? what causes development in a region? RCT only tells us part of what we need to know

    poverty_action @txtpablo spot on. Lots of important questions that RCTs can’t answer. But for the ones they can, they are best method going.

  2. 2

    Back in May 2011, Lawrence Haddad, Director of IDS, wrote a post about RCTs and Development on his Development Horizons blog.

    He concludes, “I am not blind to the virtues of RCTs, I just don’t want to be blinded by them….RCTs run the risk of locking in a result, not only within a country but across countries and over time. This is partly because: (a) they are costly to run…and so replication will be seen as costly and not terribly sexy to researchers or funders, (b) they are iconic (e.g. Progresa in Mexico, Orange Flesh Sweet Potato in Mozambique) because they are expensive and donors and researchers want to maximise their investments by publicising results (if they are positive), (c) they are not great at exploring the distribution of effects (sample sizes get too small if there are too many treatment variants) and (d) they are not particularly interested in external validity (understanding the likelihood of an interventions shown to be successful in one area being successful in another) because this requires a different set of skills.

    Reda the full post here: http://www.developmenthorizons.com/2011/05/new-harvest-of-rcts.html

  3. 3

    Development needs personal or country’s straggles,understanding capacity and workerbility


6 Trackbacks/Pingbacks

  1. Don’t Bite The Silver Bullet « Chasing Fat Tails 25 05 11
  2. The development ideas hype cycle « KM on a dollar a day 03 06 11
  3. The British lead . . . and who will follow? « Open The Echo Chamber 14 06 11
  4. Development Digest « What am I doing here? 19 09 11
  5. Measuring impacts of Financial Inclusion « housetheplanet 25 09 11
  6. Four Critiques Of RCTs That Aren’t Really About RCTs | Development Intern 04 03 14

Your Comment