A guest post by Michael Buckler, CEO & General Counsel of VillageX.
In the NGO world, impact is a murky concept. Every organization claims impact; few prove it. In the haze of donors dollars, fewer still recognize that impact is inherently populist.
In international development speak, impact is the consequence of an intervention, often initiated from the outside. The impact can be positive, negative or insignificant, qualitative or quantitative. How impact gets measured is an art and a science. Yet, a sustainable framing of impact from the outset is critical for collecting and analyzing useful impact metrics.
Most NGOs conducting impartial program evaluations make a foundational mistake. They omit impact indicators that matter most—those consecrated by local people. Such NGOs (and their evaluators) view poverty as an engineering problem whose solution merely requires identifying the right outside technologies and incentives to achieve their donors’ desired outcomes (causation of observable variable X, Y or Z). Consequently, they collect data out of context, reducing people to objects of measurement. What these social engineers often measure is desperation, not impact.
This dominant paradigm is not only disempowering for the evaluated, it conflates impact and efficacy, producing misleading results. A finding that girls stay in school longer or children get less malaria is not, by itself, a sufficient showing of impact (although it may be a necessary one). Instead, impact is whether, as compared to control group peers, more girls schlep to schools offering the empowerment program designed for them or increasing numbers of families that purchase and use bed nets. Impact hinges on local enthusiasm—whether recipients vote for interventions with their feet or pocketbooks, preferably both. Impact occurs only when programs treat each recipient as a discerning stakeholder possessing autonomy and dignity.
Take pharmaceutical research. Randomized controlled trials can prove drug efficacy in the clinic; the drug works, for example, to lower the incidence of diabetes in young adults. But trials cannot demonstrate how patients will subjectively tolerate the drug in real-life situations. Impact truth lies in such messy micro-nuances. Maybe young adults will dislike the drug’s side effects and deem them worse than the illness. Perhaps a widespread urban myth will portray the drug as a cause of low libido. The drug may be effective, but it won’t have impact unless patients buy and use it as directed.
Local people, literally and figuratively, aren’t buying a lot of NGO offerings. Perhaps it’s because they alone possess a complete understanding of what’s not only technically achievable, but culturally, politically, and financially feasible. Outside the grasp of well-intentioned NGO coercion, they choose.
Examples abound in Malawi, where I’ve worked for the last decade. NGOs profess that efficient cookstoves deliver cleaner air and cheaper energy, but I haven’t seen any in use or heard of a family spending its precious money on a cookstove. Solar products are en vogue among donors, but Malawian families aren’t purchasing solar lanterns in large numbers, despite demonstrated cost savings over time from deferred kerosene purchases. In addition to protecting against malaria, mosquito bed nets make excellent fencing material for gardens, let alone fishing nets. Nutrient deficient cornmeal fills the stomach better than nutrient rich offerings. Frustrated NGO workers scratch their heads and sometimes utter disparaging remarks about the local population. But who’s really at fault?
Most NGOs fail to heed the single most important indicator of impact—whether target populations spend cash on interventions. Impact is about behavior, not conjecture, and consumption of goods and services for sale is the most efficient way of assessing local value. Of course, when market prices are too high for low income consumers to express their consumption preferences, subsidized pricing is appropriate, as are unconditional cash transfers that permit recipients to purchase items of their choosing at or near full price. Handouts, by contrast, are caustic. Freebies not only damage local markets (and prevent others from forming), they muffle local voices.
Admittedly, those voices aren’t static. Local perspectives can and do change, albeit slowly, and a NGO with hard-earned local credibility can play a meaningful role in that process. An effective intervention that causes little impact in a given year, for example, can become impactful years later after concerted campaigns galvanize authentic local buy-in (perhaps after several rounds of people-centered design improvements). In the marketplace of ideas, the good stuff (as assessed by locals) eventually rises to the top. That’s why, in addition to listening to those on the receiving end of aid and selling them desired goods and services, NGOs should help facilitate affordable, unbiased information access (e.g. mobile Internet) for all people.
Going forward, let’s spend every available dollar on interventions with community-defined and -endorsed impact. Combining targeted outside investment with local know-how is a proven anti-poverty strategy. Imagine the gains of refocusing resources of the NGO industry on deploying folks who possess active listening skills and a deep knowledge of local conditions, intent on making locals financial investors in projects, starting small and lean, failing fast and often, and aligning interventions with local goals for progress.
NGOs need to stop hugging the trees of efficacy – and start assessing the forest of impact. It’s time to measure not what local people could do with outsiders’ interventions, but what they actually do.
***
Related Posts
A New Discipline for Development Practitioners
“Smart Aid”: A cautionary note
RCTs: Some “how matters” advice for donors
Accountability in all the wrong places
The too-often forgotten, critical first half of M&E
Does your organization unleash demand-driven development?
Listening to “beneficiaries” and the more measurement bandwagon
