Preserving the “I don’t know” within big data

My main concern with an increasing reliance on big data is that the space for possibility and the need for control or certainty too often operate in an inverse relationship. In international aid and philanthropy, our work is often focused on unanswered questions. But are some unanswerable questions?

AsimovWe have conditioned tendencies that are a result of education, our training, or organizational processes, that make “I don’t know” an unacceptable answer to a question these days. Yet “I don’t know” is found in imprecise information, in unseen or undetectable outcomes. It’s found in our trust in people, in their innate capacities and energy.

I’ve worked extensively in building the M&E capacity of grassroots organizations in Africa. What I have found is that abstract metrics and “big data” often don’t often help people understand their relationship to improving the well-being of those around them. Rather local leaders, as members of a community, read real-time trends via observation of what’s happening on the ground. This, in turn, drives intuition, much like entrepreneurialism. They seem to know what we have forgotten—that this ephemeral life is governed by a multitude of forces.

Most important to me is that our ability as “thinkers” to gather and use data and high-mindedly question everything about “what works” can insulate us and can greatly remove us from the realities of those we’re serving. In practice, this can mean imposing risk-averse procedures on people who are in the process of organizing at grassroots levels.

Undoubtedly, soundly-gathered and -interpreted data can provide important new perspectives for us all to consider. But now more than ever, having more information at our disposal than ever before in our history means that we will need to employ a rigorous humility to increase our tolerance for the risk of “not knowing.” I could not agree more with Cukier and Mayer-Schoenberger in “The Rise of Big Data” in the May/June issue of Foreign Affairs, that:

“There will be a special need to carve out a place for the human: to reserve space for intuition, common sense, and serendipity.”

Consider the Arab Spring – could big data have predicted that? From where we sit, there remains quite a lot we cannot know about how social change occurs.

And I, for one, am okay with that.


  1. Hi Jennifer. You’re inspiring me to write a post that I’ve long been gestating, about the year 1894, in which a bunch of data analysts of various persuasions collect enormous amounts of data about “heavier than air craft”, and conclude that they are an extremely poor investments, since the only impacts they have are with the ground.

    The problem with depending on data is not, in my mind, that it detracts from the human, insulates us from reality, or devalues intuition and responsiveness. I think the big problem with data is that all data is from the past (by definition), whereas development is (also by definition) about creating a future which does not yet exist.

  2. My experience with managing ‘big data’ is rather sad. Previously as an M&E Manager coordinating multi-aid programmes and now as a consultant in research and evaluations. In most cases, we do not use them or we just think of them later when something of significance arises(bring some back to life), however in the development context organizations sieve what data is supportive of its agenda. I believe that if a re-analysis of big data sets are done, we may be shocked how much we didn’t know and will even make us faint if we know now.

  3. William Finseth

    I totally agree with your thesis. We lose sight of things in the clutter of Big Data and beyond real-time observation, historical insight, intuition, and a gestalt of connecting the dots that we can’t see, we often do not see what is truly hidden.

    I remember viewing a video of a U.S. General who was discussing issues related to 9/11 in relation to the assault and planned overthrow of Iraq. He said that he happened to walk into another general’s office in the Pentagon and saw the plans to bring down the governments of most Arab governments of the North Africa and the Middle East. What may have appeared as an Arab Spring uprising may have had a helping hand from may overt and covert resources. In the case of the overt resources I am referring to the spate of Pro-democracy NGOs funded by the State Department or USAID. These organisation have raised the ire of the governments of Egypt and another nation which seems to be in the sights of America. It was no accident that Vladimir Putin shut down a number of pro-democracy NGOs in Russia. As for the covert organisations…lots of things go on behind the scenes that the public never hears about and I will bet my house that the CIA was playing its part in the change that took place in Egypt, Libya and Tunisia. To what end, I am not sure. So far the new governments appear to be as bad as the old ones. Is Syria far behind. When will the swoop down on Iran.

    The Government of the US believes in big data, as all of its snooping has revealed. It also believes in working behind the scenes. As a foreign observer, it’s hard to feel terrific about the direction of the United States at home or abroad, in the field of development or foreign affairs or military interventions. I used to look up to it as the champion of peace and democracy, human rights, etc. Sadly that is no longer the case.

  4. Pingback: Practicing Rigorous Humility | Spirit in Action International

  5. Pingback: Relying on our network | Spirit in Action International

Leave a Reply

Your email address will not be published. Required fields are marked *