RCTs: A band-aid on a deeper issue?

I finished the book weeks ago. When I closed Dean Karlan and Jacob Appel’s “More Than Good Intentions: How a New Economics Is Helping to Solve Global Poverty,” I had many notes and reactions jotted down but they hadn’t yet gelled. Something just didn’t sit quite right with me and I was having trouble articulating it.

Of course that’s related to intuition and inspiration, which can’t be measured…but I digress.

After my Friday “tweet debate” with @poverty_action and after reading many other bloggers’ reactions to the book (see related post), I realized that Karlan and Appel have heard and refuted it all by now. As seasoned and skilled randomistas, they have ready responses in support of incorporating randomized control trials (RCTs) into more aid projects.

And this is not something I’m necessarily against. In a development discourse that is still ruled by economic academics, I respect Innovations for Poverty Action’s work, which attempts to bridge this knowledge base with aid practitioners’ experience. In fact, Karlan, Appel and I agree on many aspects of their main arguments. We all agree that aid can be more effective and that well-formed questions and well-executed, applied research can offer many relevant clues about this. We all want to see deeper thinking behind the doing.

Where I think we differ is on some fundamental beliefs about what prevents this and what ails the aid industry overall. Is it a lack of information about “what works”? Or is it a lack of respect for local initiatives and understanding about complex power dynamics that impede authentic relationships among development partners? And if it’s the latter, are RCTs just a band-aid on a deeper issue?

I am, like many others, worried about the implications of donors moving towards making RCTs yet another conditionality of aid, more food to satisfy their seemingly insatiable appetite for evidence. (See my related post providing “how matters” advice for donors on RCTs.) As someone working within the extensive web of local organizations and grassroots movements in the developing world, this is especially troubling for nascent, non-“formal” and under-resourced organizations that are already marginalized from the aid system.

Whether RCTs gain momentum as just the latest fad in aid, or whether they become a part of accepted practice, I am also afraid that now “unsubstantiated claims” such as the example below, which I recently read in a report, will now be considered invalid (and un-fundable), rather than be probed for clearer understanding and viewed as a opportunity for learning.

Example of a statement of long-term impact of a program: “Most notable is the increase in the enrollment rates of girls in school in areas where Org X was focused on girls education. In the District of X, enrollment of girls went from 43% to 46% of total enrollment from 2002-3 to 2008-9. The overall graduation rate from primary schools in District X grew from 45 percent in 2002-3 to 79 percent in 2008-9.

Anyone reading this can ask the obvious questions related to the comparison and attribution. Of course there may be many factors in these increases, but this report was on a grant that was less than US$20,000, a relatively small amount when you consider the scale of most development projects. Let’s always consider what is the appropriate cost and complexity needed for measurement, especially given the size and scope of the program.

Proportional expectations for the applicability of RCTs, as well as the potential consequences of poorly-done RCTs for those who are being studied are also important, especially when people are in the process of organizing at the local level. Rather than an afterthought, let’s talk simultaneously about how local partner organizations become drivers of the use of RCTs, rather than just being consulted or included in them. As a commenter on Owen Barder’s blog shared, “Great tools, we economists undoubtedly do have. In studying development issues, they are often used unhelpfully due to hubris and a shocking level of comfort with ignorance about the phenomenon being studied.”

Despite behavioral economists’ so-called acceptance of the rationality of the poor’s decision-making, I find phrases like “the bizarre thing was that Oti didn’t seem to mind wasting his own time,” “he might have spent his last twenty rupees on [flower garlands],” “people showed they had both the will and desire to save,” “people were learning” (as if these were a surprise) and the most striking, “were [the research subjects] just thickheaded?” contained in the book’s anecdotes to reveal subtle, underlying, and perhaps unexamined judgment, if not contempt, to which I am admittedly very sensitive.

If an assumption is operating that poor people don’t know what’s good for them, then the flip side of this assumption is that someone else must. As Tom Murphy comments on Bottom Up Thinking, “In development, can’t we say that the batch of behavioral economists are exercising some amount of paternalism? They are using ‘nudges’ to encourage behaviors but that inherently comes from a place of knowing.” In my career in the aid sector, I’ve learned the hard way to become comfortable with stepping away from the role of “expert.” In fact, much of my work is focused on encouraging aid workers, donors, and international do-gooders to do the same.

Sixty years of development aid hasn’t reduced poverty using existing methods, yes. But sixty years of development aid that has squashed local initiatives by not giving the due attention to how that aid (and the accompanying monitoring, surveys, etc.) makes people feel, is, I believe, perhaps one of our biggest challenges in making aid more effective. The prevalent, yet not often exposed negative attitudes, behaviors and perceptions towards local people and organizations in the aid world is something that has been under-reported, insufficiently documented, and poorly-studied. Consider—How would programs change if an equal amount of curiosity and energy that we spend in conducting RCTs were invested in well-facilitated listening exercises in which we had to learn about people’s experiences of being on the receiving side of aid?

Let’s not forget our Sen. Freedom, power and poverty are inexorably intertwined and the mechanics of economic transformation just a part of development. In all of the discussions of RCTs and their usefulness, the reality of power relationships within the aid system and the lack of humility that continues to plague us is not something we can escape.

Undoubtedly, soundly-interpreted data provides important new perspectives for us all. There remains, however, quite a lot we cannot know.

And I, for one, am okay with that.

***

See also related how-matters.org posts, RCTs: Much to be said and RCTs: “how matters” advice for donors.

***

Related Posts

 

Got ‘Em: An Evaluation Story

 

Trying to Quit

 

Did I fund Organization X?

More on Why ‘How Matters’

How to build strong relationships with grassroots organizations (Part 1)

9 Comments

  1. Great post! I am a big fan of approaches which seek to empower individuals to make their own development decisions. I think RCT’s can be useful if they form a limited part of a community dialog of development and individual empowerment.

  2. Sasha Rabsey

    This is great Jennifer. I am in the middle of the book and noticed that when talking about solutions the authors never seem to ask anyone how they feel about aid or what they might want or not want regarding assistance.
    My next question is “who is supposed to carry out these RCT’s?” I can’t imagine asking grantees to perform statistical work as a criteria for a grant. When I do site visits I ask pertinent questions but in the end it is almost always my intuition that makes the decision to make a grant or not. I am not saying there is not a place for RCT’s but when looking at small grass roots organizations I can’t see the usefulness. Please set me straight.

  3. Great piece. It really speaks of flaws in the dominant cultures practices – an unchecked feeling of superiority and a lack of reflection.It seems that funders, NGOs, and GO’s could use some interventions themselves!

  4. Pingback: Interesting Links « Bounded Irrationality

  5. I am yet to read about these RCTs, so I do not know much about them. But what I seem to get from this blog entry is that they follow in the tradition of the 60 year old paradigm of aid which as you say has failed to rid this world of poverty. In fact, if anything it has probably made it worse by incapacitating the aid recipients to help themselves following years of drilling into them that they don’t know what is good for them, what they want, and that theirs is to wait for the aider to rescue them. This is obviously inhumane. But then aid and power economics are two sides of the same coin. In that scheme of things, the aider and all the institutions supportive of that paradigm feel duty bound to justify their actions with more actions in that same direction. The RCTs, I think, fit into this framework, and that same dance will continue. The onus is on the do-gooders to up their game, as what you obviously are trying to do by keeping us posted on such latest developments and persuading other aiders to ‘our’ side, and those on the receiving end of aid to re-examine their moves in this ‘dance’ to see if they are not inadvertently colluding if not deliberately asking for more of the same, for encore. Change can only come from the bottom, from us.

  6. marcus catsam

    There is insufficient evidence- outside of the international health field- about what works (and what doesn’t) and why in international aid (or is the word “aid” somehow offensive?). The judicious use of RCTs would be a strong step forward in understanding how those who are committed to social change can do so more effectively. I don’t see anything wrong with promoting rigorous measurement of development projects or relying on experts to provide sound technical advice about proven interventions. We should, of course, also be making a stronger effort to put communities in the driver’s seat and to ensure that projects respond to communities’ needs. I don’t see these as necessarily mutually exclusive to one another. This post, as well as many others here, seems to suggest two diametric realities: that of the big, corporate, self-interested aid industry vs. that of the wholly good, local change movement. Neither of these categorizations is accurate or fair. By reducing whole concepts and groups (“us” vs “them”) to easy, common-denominator stereotypes, I’m not sure we move forward. Should we question our assumptions? Yes, we should. But that is exactly the point. Until we have evidence to show that local is “better,” we demonize and sanctify based on assumptions and anecdote without the benefit of empirical evidence to support our claims. This is the strongest argument for RCTs I can think of.

  7. farm & market

    So much complaining about the attitudes of aid donors and implementers! Perhaps the reason it is not known the extent to which so many projects respect the wishes and priorities of beneficiaries, is that these projects are playing Santa Clause, giving away things for free to people who accept them even though they might not be what is desired, simply because the things are free.

    Try running a project where you don’t choose the participants, they choose you; and where they have to pay for the service or product, it’s not a paternalistic hand-out. When they have to pay, you’ll see quickly enough how much they really want it. If you’re doing the same job of convincing people that most projects do, you won’t have many customers. To have any adoption or acceptance, you’ll first have to have a darn good service to offer, and secondly you’ll have to do a fantastic job of advertising it and persuading people.

    Sound like private-sector product promotion and advertising? What a coincidence – it is! In fact, why not cut the aid and provide some light incentives for companies to go in and provide the products and services missing from poor economies. Craft them so they target those who most need them. Then stand back and let the economy grow.

  8. @Marcus – Supporting local efforts should not replace policy efforts, economic reforms, or programs targeted at other stakeholders still needed to bring about structural change at national and international levels. How-matters.org will remain, however, true to a point of view in which creating a smoother flow of funds to support the web of effective, indigenous, community-level initiatives deserves more recognition, thought, and investment. This requires development practitioners, including donors, to pay more attention to the concept of organization itself and the practice of facilitating the development of authentic and sovereign local organizations and social movements. If you perceive the conversation on how-matters.org at times devolving to us vs. them or good vs. bad, this is merely a reflection of the very real power asymmetries within the aid system. And if exploring these polarities helps us to acknowledge and challenge the policies and practices that marginalize and demotivate people on the receiving end of aid, then how-matters.org as a blog has fulfilled its purpose.

    As Zairah Khan shared on LinkedIn:
    “Hi Jennifer, I enjoyed your post and here is my take on the issue (which has occupied me for some time as well).
    I think indeed there is nothing wrong with deepening the thinking behind the doing, however what matters is where the thinking takes place and who is doing the thinking. I worry that financially speaking a lot of aid is spent on thinking that takes place primarilly at the donors’ or the institutions’ desks, whereas the recipient is then left with the crumbs and having to think her/his way around preexisting perceptions. I think the word ‘micro management’ is not out of place. This in my opinion has everything to do with the continous ‘upscaling’ of aid. The applied research risks becoming a justification for that upscaling, because results can’t be measured in a scientifically sound way when applied to local circumstance alone. The distance between practice and principle is getting increasingly big and this is leading to a lack of trust and understanding between the thinkers and the doers. I think the thinking we should be doing is ‘how do we close that gap?'”

  9. Pingback: Development Digest « What am I doing here?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.