Measuring social change
David Bonbright, founder and chief executive of the UK-based Keystone organization, says social investors need “eco-intelligence” systems in their bid to develop sustainable solutions to important social and environmental problems.
“Understanding, measuring, and communicating social change is not like a market transaction, where you’re buying a bunch of fruit that you can maybe smell and taste before you pass money back and forth,” explains David Bonbright, founder of Keystone, a UK-based organization that advocates accountability for social change. The transaction is far more complex, yet public information for deciding how to engage and support social change processes remains weak.
Bonbright delivered these and other observations on the social investment field during the inaugural lecture of IDRC’s Evaluation Unit’s new speaker series, Innovation in Evaluation: Ideas Worth Sharing.
Launched in January 2008 in Ottawa, the series aims to enhance IDRC’s culture and quality of evaluation by promoting the exchange of ideas on social change, learning, and accountability.
Transparent comparative reporting on development agencies is in its infancy. Evaluation and development research therefore needs to foster a better information ecosystem – “eco-intelligence” – for social change investing, so that investors can base their choices on validated information. These were some of the conclusions Bonbright shared with his audience.
“We don’t have good data about most organizations... and it’s very important that we begin to change that,” he said. Bonbright also touched on the transformative changes in knowledge transfer triggered by advances in technology and communications, such as blogs and other Web communications, and their effect on social investors and development.
Keystone has spent the past three years developing practical ways of planning, measuring, and communicating social change that fosters accountability and learning among a wide range of partners, including donors, non-governmental organizations (NGOs), and other stakeholders. Bonbright outlined Keystone’s belief that accountability for development agencies requires that they publish what they can demonstrate they have achieved plus what their primary constituents (those meant to benefit from their work) have to say about those achievements. In so doing, they create a public, self-correcting reflexive learning, validation, and social auditing process.
According to Bonbright, Keystone “follows shamelessly in the slipstream” of IDRC’s pioneering outcome mapping work, an evaluation methodology that focuses on changes in behaviours, relationships, and activities of the people and organizations with whom an organization works rather than focusing on the outputs (products) of a program.
According to Bonbright, donors and development organizations favour a more rigorous feedback from their constituents, but lack incentives to make it a priority. One method that Keystone has pursued to motivate organizations is a comparative constituency feedback mechanism that lets NGOs benchmark themselves against their peers on the basis of the views of their constituents. “It is one thing to say that one’s constituents rate us at 5.8 on a 7-point scale, but quite another to realize that that 5.8 is only at the 52nd percentile in the peer cohort!”
Bonbright envisions that these comparative constituency assessments can become a standard part of social change practice. Related to this, Keystone is designing core principles for a yet-to-be-determined “‘validated impact reporting”’ methodology that would assess how soundly an organization’s current reporting methods measure social change. The thrust here, said Bonbright, is to give the most weight to the opinions of those most affected.
Critical drivers – the big picture
The need for more transparent evaluation plays out against larger drivers in international development. These include the increasing use of market and business enterprise models in the realm of social change, which in recent years has seen a growing number of business graduates enter the field.
But the most important driver “by a considerable margin” is Web 2.0’s rapidly emerging social networking, community-generated data, and interactivity.
“Today, the best data we have says only 3% of giving is happening through the Internet,” said Bonbright. “But when you break that into smaller, more carefully nuanced segments that are likely to be triggers of things to come, like Hurricane Katrina or the Indian Ocean tsunami of December 2004, you see a much higher proportion of the giving coming through the Internet – 30% or 40%.”
Another recent Internet-driven phenomenon is what Bonbright calls “online giving marketplaces” – websites that allow individual donors to direct donations toward particular causes or projects.
For example, Kiva, an online microcredit organization, and Give India match individual donors with impoverished entrepreneurs in the developing world and Indian charities, respectively. More than US$25 million flowed through the two organizations last year. Bovespa, a social-investment offshoot of the Brazilian Stock Exchange, may spawn similar ventures elsewhere in the South.
Bonbright said such online marketplaces (there are now 30) need to measure and demonstrate the social impacts of their projects to their donors. To date, none has found a methodologically sound way to do so, but the growing demand for evidence-based data increases outcome mapping’s importance, and thus drives calls for better data.
This, in turn, should push demand for more rigorous data on social change. “We, as development professionals, who have a practical view about what’s really important, about what we should be measuring in social change, need to be part of this conversation,” says Bonbright. “We need to be influencing it, and we need to be influencing it in this space, as well as in the big aid agencies in which we’re working.”
These Web-based alternatives to big traditional aid agencies are encroaching on the agencies’ long-held intermediary roles and are forcing donors to re-evaluate “business as usual.” He notes that OXFAM UK, in response, has quietly opened a window on its website through which donors can choose Oxfam projects on the ground and channel money to them directly. (Keystone’s recently published study on online marketplaces – Online Philanthropy Markets: from “Feel-Good” Giving to Effective Social Investing? – is freely available from its website.)
A need for high-level metrics
Bonbright also pointed to the critical need for standards and tools for measuring social change and impact. “At some point, you need some high level metrics that are easy for people to follow,” said Bonbright, because “99.9% of the people who are choosing to get involved and support social change actually don’t want to drill down and find out all the detail. (But) they want to know it’s there, and they want to watch meaningful high-level indicators.”
As an example, he cited a simplified computer dashboard readout, already common in business intelligence software. But development agencies, he said, haven’t been good at representing complexity as simply. Keystone suggests they could adopt a readout with three top-level dials – impact, performance, and voice (the extent to which an organization truly helps empower constituents) – that also give users a feel for the quality of the underlying data.
“Its purpose is to enable benchmarking, management of performance in real time, to develop flexible standards... and to enable third-party investment choices,” he said.
What Bonbright really wants to see is a deeply reflective learning practice in every social development organization. Encouraging this practice is critically urgent, and must cut across the usual ways development organizations now work.
“We’ll never be effective until we build systems that plan, assess, and report with constituents, in an ecosystem for sustainable and lasting change,” he argued. “When we . . . look at where capacity is in the world, and where it’s needed, a critical priority is to build capacity in the global South. This can’t be something that continues to be held here – it has to be decentralized. The Internet is an ally for us because it’s the ultimate decentralizer and democratizer. And I think communities of practice are going to be critical tools for us as we do this.”
“Information spaces” are emerging to rate social change agencies. These, according to David Bonbright of Keystone, are the ones to watch. Though still nascent, they’re quickly developing. Among the first bloomers are the non-profit US-based Guidestar and other similar attempts at ISO-style certification programs or rating agencies.
The US-based Great Nonprofits wants to create a Zagat’s-style guide of social change, where anyone with direct experience with a non-profit group can enter feedback on a public database. Organizations can respond, creating public debate. Although there are huge technical problems, Bonbright insists it’s a trend worth watching.
Bonbright also notes some serious and well-funded efforts to create what he calls “outcome indicator topologies” – standard ways for potential donors to compare particular organizations or projects. Google and Acumen have partnered to develop one such beta “donation portfolio management system,” and Keystone is lobbying to ensure the integrity of the underlying databases.
K.J. Shore is an Ottawa-based writer.