In @Saundra_S’s absence, it’s only fitting that someone step in to compile and share the barrage of articles that have been regarding or related to RCTs in the past month or so due to the release of More Than Good Intentions: How a New Economics is Helping to Solve Global Poverty and Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty. En homage to Good Intentions Are Not Enough, here’s my attempt to do so, in no particular order. Please do share additional posts and articles via the comments section.
I am also including at the bottom the content of Friday’s “tweet debate” regarding RCTs that demonstrates some of the polarities in thinking, to which I hope I’ve done justice to those who participated.
Here’s to the continued conversations…
***
Dean Karlan’s Q&A on the Freakonomics blog and with Jacob Appel in Foreign Policy
World Bank’s Development Impact Portal: “Dean Karlan’s New Book: RCTs – this time it’s personal” and “Stuff You Cannot Randomize” and “Esther Duflo’s refreshing perspective on fighting poverty” and “Is it the program or is it participation? Randomization and placebos”
Boston Review: “Small Changes, Big Results,” a forum on applying behavioral economics to global development.
Tom Murphy in the Huffington Post: Proven Impact: Better International Development Through Rigorous Research
Innovation for Poverty Action’s blog: “Helping Ugandan families save for school fees” and “What should NGOs do when they can’t run an RCT?” and “RCTs: The Backlash”
Nicholas Kristof in the New York Times: “Getting Smart on Aid”
Arvind Subramanian, Center for Global Development: “Nicholas Kristof and Aid”
The Wall Street Journal: Is ‘Nudging’ really enough? and Bill Easterly’s review of More Than Good Intentions
Owen Barder: “Do Economists Have Better Tools?”
Rachel Strohm: “Development Aid in Context”
Ben Goldacre in The Guardian: “How can you tell if a policy is working? Run a trial”
Madeleine Bunting in The Guardian: “Duflo and Banerjee take the guesswork out of policies that help the poor” (on the glaring omission of an understanding of power)
Guardian Editorial: “In praise of…Esther Duflo and Abhijit Banerjee”
Duncan Green of Oxfam GB reviews Poor Economics and guest post by Karl Hughes “Can we demonstrate effectiveness without bankrupting our NGO and/or becoming a randomista?”
Rick Davies on MandE News “Do we need a minimum level of failure?” and “Relative rather than absolute counterfactuals: A more useful alternative?”
Ian Thorpe: “The problem with the big picture”
Tales from the Hood on the necessity of “good vibes”
Chris Blatmann: “Go Short on Randomized Control Trials?” and “Aspiring PhD students: Should you become a field research assistant for an RCT?” and “Development Aid 3.0?”
Development Policy Blog: “Why abundance evidence won’t inevitably lead to abundance”
Lawrence Haddad of IDS: “A New Harvest of RCTs?”
Find What Works’ series on the Limitations of RCTs – Part 1, Part 2, Part 3, Postscript
Good Intentions Are Not Enough: “’More Than Good Intentions’ and ‘Three Cups of Tea’”
OpenAid: “Randomized trials are NOT the hottest thing in the fight against poverty”
Ben Ramalingam at ODI: “When will we learn how to learn?”
Tate Watkins at Short Sentences: “Why most aid won’t be ‘good aid’”
Edward Carr: “On explanation in development research” and “The Qualitative Research Challenge to RCT4D”
Interaction Institute for Social Change: “95% Failure”
Mike Klassen at Doorway to the Universe: “At Second Glance”
GiveWell blog: “Suggestions for the Social Sciences”
Marc Bellemare: “Thoughts on the RCT Debate”
Chasing Fat Tails: “Don’t Bite the Silver Bullet”
David Week: “Who guards the RCT guardians?”
The Monkey Cage: “Why aren’t there institutional review boards for the development industry?”
Centro Studi Luca d’Agliano: “Lentils for Vaccines” and “Dangerous Liaisons: On the ambiguous relationship between development and development economics”
Stuff Expat Aid Workers Like: “#60 Randomized Controlled Trials”
Dennis Whittle: “If not randomized trials, then what?”
Ian Thrope: “The development ideas hype cycle”
Rasmussen, Malchow-Møller & Andersen, University of Southern Denmark: “Walking the talk: the need for a trial registry for development interventions”
Mark Engler: “On Nicholas Kristof’s Boneheaded ‘Paean to Economists‘”
Kate McKee: “Why Poor People Don’t Use Savings Accounts: Can qualitative research (RCT’s homely stepsister) help answer the question?”
Philip Auerswald: “Why Randomized Controlled Trials Work in Public Health…and Not Much Else.”
Tim Ogden: “Revisiting Objections to Randomized Control Trials” and “Making RCTs Better” (love the call for humility)
David Roodman: “The Smartest RCT Critic”
Marc Maxson: “What Aid Experts Could Learn From Fantasy Football”
14 Points: “Randomized Control Trials on Trial: Evaluating the Efficacy of RCTs” and “Vying with Velker: RCTs Reconsidered”
Brendan Rigby: “The decline effect: a storm on the horizon for RCTs?”
VOX, Anders Olofsgård: Why political short-sightedness and randomised control trials can be a deadly mix for aid effectivness
How Matters: “RCTs: A band-aid on a deeper issue? and “RCTs: some ‘how matters’ advice for donors”
***
@Dochasnetwork May 19, 10:29pm @intldogooder @owenbarder I’ve a huge problem with Randomised Trials. Immoral, in my book. Overkill. Measuring Outcomes should suffice.
@intldogooder 4:18am Moral ? is how to justify control group that doesn’t get services & how they are later brought into progs @dochasnetwork @owenbarder #RCTs
@viewfromthecave 4:23am @intldogooder From my understanding of RCTs there are multiple ways of dealing with ethics of control groups. One is lack of funds which…means that only so many can get the services, so some will naturally be without. If enough resources for all, interventions…can be gradually timed to get a comparative, but not deny services. Or all can get different kinds of services, etc…Maybe @poverty_action will be kind enough to post on the ethics of RCTs and determining control groups…
@intldogooder 4:55am: The “how” discussion is important given attn RT @viewfromthecave: Maybe @poverty_action will be kind enough to post on the ethics of #RCTs
@DevEconHealth 4:31am @intldogooder @dochasnetwork @owenbarder Not sure that is a serious moral ?. Same issue with any proj trgt group and med. trial.
@intldogooder 4:55am Counter-balance to Kristof’s article @HumDevNGOs: better ways to measure complex #socialchange http://bit.ly/ilPqfm #complexity #smartaid
@viewfromthecave 5:24am via @intldogooder You read Karlan’s book, right? [Ethics] is addressed a bit in the beginning
@intldogooder 5:33am @viewfromthecave I did and you’re right, they include that & many other caveats. Important for me to keep beating the “how matters” drum…Have seen such poorly-executed evals/research in the field that I have visions of that now for wide-scale #RCTs
@poverty_action 6:32am @intldogooder @dochasnetwork how about the ethics of potentially wasting limited money on unproven projects?
@intldogooder 7:00am Also vital but deficit mentality has its drawbacks RT @poverty_action: @intldogooder @dochasnetwork ethics of potentially wasting limited $?
@poverty_action 7:04am @intldogooder which are..?
@intldogooder 7:19am @poverty_action econ tools “often used unhelpfully due to hubris & a shocking level of ignorance about being studied” http://ow.ly/4Z9UN
@poverty_action 7:19am @txtpablo @viewfromthecave @intldogooder any ethical issues with using control group for medical trials? Or should we just hope for the best
@intldogooder 7:20am @poverty_action thus why sharing more of your experience on HOW to do #RCTs well so important, many of us grounded in assets-based approach
@txtpablo 7:21am @poverty_action @viewfromthecave @intldogooder Well, let us know! For sure there are some discussions around.
@poverty_action 7:27am @intldogooder we run trainings for partner orgs & run exec ed with J-PAL http://bit.ly/mwVNkY this WB book is great http://bit.ly/iP5848…but always grateful for suggestions of how we can do more!
@intldogooder 7:33am Lets acknowledge differences btwn social change & medical science @txtpablo @poverty_action @viewfromthecave re: ethics of control groups
@poverty_action 7:37am @intldogooder @txtpablo @viewfromthecave true, but not an argument against striving for the greatest rigor in evaluation possible
@txtpablo 7:40am @poverty_action: @intldogooder @viewfromthecave agree, but rigor in #evaluation is not only RCT, it is much more
@poverty_action @txtpablo there are other rigorous methods but RCTs generally agreed to be the best way of attributing causal impact
@txtpablo @poverty_action I agree it is good for causal impacts but in simple contexts, where you can manipulate few variables, but in complex ones?
@poverty_action @txtpablo do you have a specific example you are thinking of?
@txtpablo @poverty_action what causes a revolution? what causes development in a region? RCT only tells us part of what we need to know
@poverty_action @txtpablo spot on. Lots of important questions that RCTs can’t answer. But for the ones they can, they are best method going.
@intldogooder 7:41am Does genuine learning on an individual level always come from rigor? RT @poverty_action: @intldogooder @txtpablo @viewfromthecave @itsjina
@BonnieKoenig 7:44am @intldogooder I don’t believe in ‘always’, ‘best’ etc.. See previous tweets @poverty_action @txtpablo @viewfromthecave…Yes these differences r impt 4 diff situations
@viewfromthecave 7:42am @intldogooder Both impact livelihoods of many and known solutions can lead to improvement for many in the future #DevilsAdvocate
@intldogooder 7:47am At times this is like atheism vs. faith debate. What is comfort level with not knowing? @txtpablo @poverty_action @viewfromthecave @itsjina
@sabes_a_arim 7:50am @intldogooder what?
@viewfromthecave 7:50am @intldogooder There has to be comfort in not knowing everything, but that does not mean we cant know more @txtpablo @poverty_action @itsjina
@poverty_action 7:53am @viewfromthecave @intldogooder @txtpablo @itsjina will always be unknowns, but not knowing what works to fight poverty IS uncomfortable
@ithorpe 7:58am @poverty_action @viewfromthecave @intldogooder @txtpablo @itsjina exchange reminded me of “The truth is out there” http://wp.me/p110pY-12
@intldogooder 8:12am What works is investing in people, not just ideas- must pay attn to that 2 @poverty_action @viewfromthecave @intldogooder @txtpablo @itsjina
@viewfromthecave 8:25am @intldogooder no disagreement on that.
@poverty_action 8:15am @intldogooder we should test that theory….
***
Related Posts
Remembering George Harrison on the road to Busan
Aid, Africa, Corruption and Colonialism: An Honest Conversation
Egypt’s Thundering Wave & Questions for Development Aid
Confessions of a Recovering Neocolonialist
I have tried to copy and paste some of the interchange we had with @poverty_action last Friday, which I think it is missing in the post (sorry if it is not the case). I wonder also if there is a SIMPLE method for saving and ordering tweets!
txtpablo @poverty_action: @intldogooder @viewfromthecave agree, but rigor in
#evaluation is not only RCT, it is much more
poverty_action @txtpablo there are other rigorous methods but RCTs generally agreed to be the best way of attributing causal impact
http://bit.ly/…
txtpablo @poverty_action I agree it is good for causal impacts but in simple contexts, where you can manipulate few variables, but in complex ones?
poverty_action @txtpablo do you have a specific example you are thinking of?
txtpablo @poverty_action what causes a revolution? what causes development in a region? RCT only tells us part of what we need to know
poverty_action @txtpablo spot on. Lots of important questions that RCTs can’t answer. But for the ones they can, they are best method going.
Pingback: Don’t Bite The Silver Bullet « Chasing Fat Tails
Pingback: The development ideas hype cycle « KM on a dollar a day
Pingback: The British lead . . . and who will follow? « Open The Echo Chamber
Pingback: Development Digest « What am I doing here?
Pingback: Measuring impacts of Financial Inclusion « housetheplanet
Back in May 2011, Lawrence Haddad, Director of IDS, wrote a post about RCTs and Development on his Development Horizons blog.
He concludes, “I am not blind to the virtues of RCTs, I just don’t want to be blinded by them….RCTs run the risk of locking in a result, not only within a country but across countries and over time. This is partly because: (a) they are costly to run…and so replication will be seen as costly and not terribly sexy to researchers or funders, (b) they are iconic (e.g. Progresa in Mexico, Orange Flesh Sweet Potato in Mozambique) because they are expensive and donors and researchers want to maximise their investments by publicising results (if they are positive), (c) they are not great at exploring the distribution of effects (sample sizes get too small if there are too many treatment variants) and (d) they are not particularly interested in external validity (understanding the likelihood of an interventions shown to be successful in one area being successful in another) because this requires a different set of skills.
Reda the full post here: http://www.developmenthorizons.com/2011/05/new-harvest-of-rcts.html
Development needs personal or country’s straggles,understanding capacity and workerbility
Pingback: Four Critiques Of RCTs That Aren’t Really About RCTs | Development Intern