Don’t Let Conventional Measurement Wisdom Fragment Your Impact
Social impact bonds. Evidence-based funding. Investing in what works. To what do we owe this explosion of “pay-or-performance” funding models?…
Social impact bonds. Pay for success. Evidence-based funding. Investing in what works. To what do we owe this explosion of “pay-or-performance” funding models?
Most of the rhetoric is some variant of: “There are too many ineffective and inefficient nonprofits getting funding and not really having any impact.” While this may be too strong a statement, research clearly shows that many nonprofits lack the resources, expertise, and conviction to continuously measure and improve their programs. So, it is not a stretch to imagine that society would be better off if resources were directed to more effective nonprofits.
But a far greater problem lurks beneath the surface, and it has to do with whether all of the individual impacts nonprofits generate actually add up to something meaningful. A recent observation relayed to me by the leader of a community-based organization in Boston could not have summed it up better: “You know what, I’ve been doing this work for over ten years, and while we’re making some headway, the truth is we haven’t done enough to change the trajectories of the young people in our city.”
The sad truth: Conventional wisdom on measurement perpetuates this status quo. It fragments impact into stages that can be evaluated, rather than encouraging organizations to aim for trajectory change.
For example, suppose you run a summer sports program for middle school youth who are stable (physically, mentally, emotionally, etc.) but nonetheless disadvantaged and at risk of dropping out of high school. You measure students’ level of physical activity, involvement in crime, and substance abuse before and at the end of their time in the program, and the data you collect is promising. So you contact an evaluator. “We think our program is working, but we want to know for sure—what should we do?”
Conventional wisdom on performance measurement will take you down the path of one or more rigorous external evaluations to attribute outcomes to your specific program, to show that your program creates more benefits for society than it costs, and to show that it is replicable to other settings. And with this bullet-proof data in hand, voilà, you are now “evidence-based” and ready to attract the resources you need to dramatically scale.
While this pathway through evaluation seems reasonable at first glance, something is missing, and it relates to the youth leader’s observation cited earlier: We don’t have a clue whether this program is changing, or contributes to changing, the actual trajectories of the youth it serves. For some (those who would have committed serious crimes in the middle school years if they had not been in the program), it may have; but with a relatively stable target population, such cases are few. What about the rest? Once they enter high school, do they revert to pre-program levels of crime and substance abuse? Do they graduate, or do they end up dropping out of school at the same rate as they might have without the program?
“Not to worry,” the measurement gurus tell you, “You’re working on an important predictor of high-school graduation, so you can just use the literature to show your program is helping youth down this path.” But such advice makes the fairly big leap that other high-quality programs are addressing the more than dozen other critical predictors of high-school graduation for the specific youth in this program.
The fact is, far too often, we rely on a patchwork of programs, at best evidence-based on one or two dimensions, working with different youth in an uncoordinated fashion. Consider the 76 promising or proven interventions in the Promising Practices Network’s registry. Less than one-third are impacting more than one indicator of child well-being, making it highly unlikely that they are changing the ultimate destiny of any given youth.
At the end of the day, it comes down to accountability. More nonprofits need to explore what it means to hold their organizations accountable for meaningful change in the trajectory of a client, however they define it. Doing so may lead the organization to consider a more multi-service, client-centered approach (like Latin American Youth Center in Washington D.C.) or to collaborate with other organizations that provide the rest of the services needed (like the Strive Partnership in Cincinnati). Or it might lead to a commitment to measure long-term change, which could in turn lead to the discovery of the next great intervention (like Nurse-Family Partnership).
And therein lies the problem with “pay-for-performance” models. To the extent these funding mechanisms reward programs that generate single, short-term outcomes (e.g., keeping a job for six months, improving test scores in one grade), our society won’t really be any better off. Developing and proving an intervention that can produce short-term outcomes is rare and laudable, but impact could be dramatically amplified through a simultaneous (or subsequent) pursuit of longer-term success by those who have perfected these models. Instead of fragmenting impact into measurable bites, we need to purpose measurement to create solid pathways to better, ultimate outcomes for those we serve.
How are you using measurement to ensure your beneficiaries access the supports they need for long-term improvements?
Matthew Forti is the Performance Measurement Capability Area manager at the Bridgespan Group, an advisory firm to mission-driven leaders and organizations. Matt has supported several leading organizations in the design and implementation of performance measurement systems that promote continuous learning and improvement, the primary focus of his blog, Measuring to Improve. Matt is also the founding board chair of One Acre Fund, a nonprofit that assists over 50,000 smallholder farming families in East Africa to triple their crop yields.
Simply putting boxes and lines down on paper doesn’t guarantee that your organization will make better decisions.
How the Rockefeller Foundation is approaching evaluation with developing country partners.
What the Irvine Foundation has learned over the past six years about performance assessment.
How the Ford Foundation is engaging its global staff in building a shared culture of results.
How The William and Flora Hewlett Foundation approaches high-risk philanthropic ventures.
How the Omidyar Network uses a venture capital model to measure and evaluate effectiveness.
It is essential to build direct consumer feedback into funding criteria for government and nonprofit programs serving low-income people.
Lessons from the Community Catalyst Fund in Charlotte, North Carolina.
A new brand of social entrepreneurship is emerging.
COMMENTS
BY Anna Ruman, Harvard University
ON August 22, 2011 11:42 AM
Dear Matthew,
Thank you for your article. I agree that without longitudinal and/or holistic analyses of environmental and social impact, we’re not actually making more informed investment decisions.
In addition, a question: I am a cofounder of a microfinance tourism startup, Investours. I current manage our impact metrics program. Since January, I have been designing and implementing an impact measurement strategy. Our greatest challenge has been establishing a control within the general population, i.e. individuals with similar demographic profiles who will NOT receive microloans as part of our program.
Have you run across any particularly amazing articles or could you provide any advice on establishing an effective control in microfinance lending?
Thanks!
Anna Ruman
.(JavaScript must be enabled to view this email address)
www.investours.org
BY Stephen M. Pratt, Root Cause
ON August 22, 2011 12:51 PM
Thanks for your post, Matt, which I think is valid in the spirit in which it was offered. That said, I think you are holding individual nonprofits to too high a burden given their limited influence and power to effect systemic change. At the end of the day, organizations focus on narrow results for which they can produce evidence because that is what their donors, public and private, will pay for. I can think of one large and successful youth development organization working with middle school students that has dialed back language suggesting a goal of college for its clients because its funders pointed out, correctly, that such a result lay outside of its theory of change, i.e., the results for which it could directly claim influence. While I agree with your point that we should all hold ourselves accountable for the trajectory toward which all of our efforts point, I wonder what options nonprofit leaders really have. While I wish that they had the power to band together and demand a different sort of calculus, the evidence that that tactic would have an effect is elusive. The solutions that you suggest—more collaboration and multiservice approaches—have been championed for some time by practitioners, with only limited enthusiasm from donors.
BY Eric Longo
ON August 22, 2011 06:57 PM
Thx for this post Matt and for your on-going investigation of defining impact. As for the debate over education reform, I don’t think there is a silver bullet that can define what impact means and the metrics to measure it across the board. The nonprofit sector is too vast and too diverse to have a set of metrics that could even attempt to adequately capture what works and what doesn’t for all services. Though far from perfect, I would however support the notion that successful programs need to create long-term social value to be called truly effective (for an important reference point on this debate, I suggest: “The End of Charity: How to Fix the Nonprofit Sector Through Effective Social Investing” by David E. K. Hunter, Philadelphia Social Innovations Journal, October 2009″ http://bit.ly/hC7Rg2). From this perspective, I agree with your conclusion that to gauge whether a particular program adds value, and therefore to measure any significant degree of impact, we need to adopt the long-term view over short-term outcomes.
BY MissionMeasurement
ON August 23, 2011 02:43 PM
I think Matt is spot-on! The biggest issue we face today is that we have too many nonprofits producing “fragments” of an outcome rather than working more intensively to produce a “high-value” outcome. i think it’s time we flip our focus from “doing good” and hoping it adds up to something - to purchasing impacts that we value and finding the best, most effective organization to deliver that impact. see more here: http://www.ssireview.org/articles/entry/supplicants_no_more/
BY Matthew Forti
ON August 23, 2011 02:47 PM
Many thanks for the comments!
Eric reminds us that standard definitions of impact and metrics are challenging to come by in the sector, but that this doesn’t obviate the need for each organization to define for itself and its stakeholders what it means by impact and how to measure it. I wholeheartedly agree! I will add, however, that there are exciting examples of shared measurement where ‘similar enough’ organizations come together, define outcomes and metrics, and agree to share data and learn from one another. See, for example, PPV’s Benchmarking Project for workforce development organizations. These opportunities should be seized and replicated.
Stephen raises two very important points. First, he notes it would be foolish for an organization to hold itself accountable for long-term success if its intervention can only produce short-term outcomes. I very much agree - and the organization Stephen references made the right decision to remove ‘college entry’ from its theory of change. But I would still argue that the status quo - which is often ten different organizations achieving ten different short-term outcomes for ten different clients in a given area - is a recipe for stagnation. Which brings me to Stephen’s second point: when faced with this dilemma of short-termism, nonprofit leaders’ hands are tied if funders won’t support collaborations and multi-service approaches. This is sadly very much the truth. My hope is that, over time, dialogue like this, plus the powerful results that multi-service organizations (like LAYC) and collaboratives (like Strive) will generate, can contribute to changing the status quo.
BY David Pritchard, New Philanthropy Capital
ON August 24, 2011 10:31 AM
Matthew: You are right to point out that many organisations focussing on single, short-term outcomes leads to an inefficient use of resources. Because of weak co-ordination among organisations it is a matter of luck if the right interventions happen at the right time in the right dosage for the right people to change life trajectories. Mergers and more collaboration to provide more coordination are obvious solutions. But they will only come about (and herein lies the rub) if the indivdual charities are persuaded this is good for them and the people they serve. But short of that individual organisations can take their own performance measurement to the next level by developing theories of change that incorporate local conditions and activities of other organisations and using them to develop a richer set of metrics to manage their own activities. This is a third route to better co-ordination and, hence, greater impact.
BY Elizabeth Kronoff, Insaan Group
ON August 24, 2011 10:55 PM
Some great points are raised in the article and especially in the comments section.
To highlight a few:
“The solutions that you suggest—more collaboration and multiservice approaches—have been championed for some time by practitioners, with only limited enthusiasm from donors.” - Anna
This comment hits the nail on the head. Philanthropists and family funds have the greatest flexibility to take the long-term approach and work with NGOs and the end-user to produce meaningful strategies. Larger donors with funds that are essentially taxpayer money are much more limited, so it’s understandable. Nonetheless, political pressure could theoretically be applied. This is hardly on the political barometer right now in most of the donor countries, but if philanthropists and funds stick with it, the best approach will eventually be adopted more widely.
“The biggest issue we face today is that we have too many nonprofits producing “fragments” of an outcome rather than working more intensively to produce a “high-value” outcome.”—MissionMeasurements
This sounds like such a good idea. However, imagine if we suggested the same thing about business start-ups and small businesses. Competition among NGOs, and some level of failure, has to be acceptable if we are to have evolution in the field. Yes, collaboration is necessary especially when we are talking about basic-needs services such as health, housing, etc. where we cannot afford to let people slip through the cracks of a fragmented non-profit sector. But at the same time, we have to ensure we allow for different short-term goals and strategies to achieve the overall end, to keep invention going.
“Because of weak co-ordination among organisations it is a matter of luck if the right interventions happen at the right time in the right dosage for the right people to change life trajectories. Mergers and more collaboration to provide more coordination are obvious solutions.” - David
Another approach, which is in fact practiced overseas, is to use donor coordination and regional and national planning. The Millennium Development Goal project is aimed at this. Unfortunately many NGOs that target populations whose needs do not seem to fall under the broader goals get left out. Still, if we are talking about key indicators, then it is the funders / donors / philanthropists that need to coordinate. Good NGOs will simply translate this into ground-level objectives with the target group.
For my part, I really think you have got a great point about “pay-for-performance” when performance is only measured short-term because of short funding cycles. It’s a systemic problem that must change. We also need to change the way we fund performance management and M&E, and how we look at it. It’s not overhead: it’s a necessary part of service delivery. When we don’t track true impact, it’s like a business not tracking net profit.
BY Joe Kriesberg, President Mass. Assoc of CDCs
ON August 30, 2011 02:25 PM
Great discussion!
The other critical challenge herre is moving from individual level change to population level change. Most programs focus on changing the lives of individuals - appropriately enough! However, this does not address the society level issues that seem to ensure that some percentage of us will be poor and disadvantaged. In an era with 9% unemployment and 16% underemployment; in an era with millions of low paying jobs that need to be filled by someone; I fear that one individual’s success may very well come at the expense of another person. Client A recieved a job; bur four other applicants remain unemployed. Client B recieved a Section 8 housing voucher; but 100 people remain on the waiting list. Are we simply playing musical chairs or are we really moving the needle for the population as a whole?
I don’t raise this to disparage or discourage programs aimed at individuals - not at all. Such programs are vital. And the world is not a zero sum game - indeed I do believe that “we all do better when we all do better.” So these programs do help move us forward - and help mitigage the negative impacts of poverty. But we also need to look at policy and system changes that can impact thousands if not millions of lives. Organizing and advocacy for such policy shifts should be the leading edge of our innovation and investment. The conservative movement has figured that out and dramatically changed our economic system for the worse while liberals are working one person at a time delivering services and programs. Advocacy efforts don’t easily fit under “pay for performance” schemes, but are essential if we are really to transform lives and communities at scale.