Stanford Social Innovation Review : Informing and inspiring leaders of social change

SUBSCRIBE | HELP

Measuring Social Impact

Social Sector Measurement: Down but Not Out

Recently released studies confirm that few nonprofits possess the capacity to measure for continuous improvement, but new initiatives launched in 2012 provide hope.

For those pining to see a revolution in how social sector organizations view and use measurement, this holiday season brings two reports with good insight into where we are and how far we still have to travel. To sum up: Buckle your seatbelts—it’s going to be a long ride!

Innovation Network’s “State of Evaluation 2012” study (October 2012) draws from a representative sample of US nonprofits and provides a bevy of statistics and insight, including comparisons to results from a similar 2010 survey. The good news is that 90 percent of nonprofits report measuring their work, and while funders and boards remain the primary audiences for measurement, more than three quarters also report using their measurement to plan and revise strategies and programs.

That said, the proportion of nonprofits that have at least one full-time employee devoted to measurement remains frighteningly low (18 percent), as does the proportion of them that are spending at least 5 percent of their budget on measurement (27 percent)—figures that have increased only a few, potentially statistically insignificant percentage points since the 2010 survey. Further, the proportion of nonprofits that have revised their theories of change or logic models within the past year has actually declined, with fewer than half of nonprofits using measurement to adjust their programs at least annually. Finally, nonprofits continue to rate research and evaluation as their two lowest organizational priorities.

Not surprisingly, the study paints funders as one of the biggest culprits, both in how they support grantees in using measurement and how they use it themselves. (One particularly telling datapoint: Funders are nearly half as likely as nonprofits to use measurement to plan and revise their programs).

Taken together, these and other findings suggest a lot of measurement is happening, but it’s unlikely the quality required for organizations to truly learn what’s working and continuously improve their programs. With fewer than one-third of nonprofits demonstrating promising capacities and behaviors to meaningfully engage in measurement, the authors conclude that the state of evaluation in the social sector can best be judged only as “fair.”

The disappointing role of funders is echoed in the Center for Effective Philanthropy’s “Room for Improvement” (September 2012). The headlines from this study:  only 32 percent of nonprofits believe foundation funders have been helpful to their ability to measure progress, and only 29 percent have foundation funders that provide financial and/or non-monetary support for measurement efforts. And given that foundations place relatively high importance on nonprofits’ evidence-bases (according to recent Bridgespan research), the figures above likely overstate the support nonprofits are receiving from their typical funders.

Both reports conclude that funders can play a pivotal role by changing their approaches to working with and supporting grantees in measurement. But if 2012 taught us anything, the greater promise may lie in funders supporting initiatives that get nonprofits learning from one other. For instance, Mario Morino’s Leap of Reason, which recently surpassed the threshold of 50,000 books in circulation, is chock-full of practitioner essays and examples that provide rich, often first-person accounts around how measurement has changed the trajectories of nonprofit organizations. PerformWell, which launched earlier this year, is attracting hundreds of nonprofit leaders to its free webinars to hear how their peers are establishing performance cultures and regularly using data to improve. And the “Saving Philanthropy” documentary, with its compelling footage of leaders and staff of Nurse Family Partnership, Roca, and others describing how they use performance measurement, has met with great acclaim from those who have participated in screenings and workshops. 

We believe the biggest returns on investment in educating and inspiring nonprofit leaders around performance measurement will tend to accrue at the earliest stages of a nonprofit’s development, when leaders are still forming their operating philosophies and cultures. To see greater improvement by the time “State of Evaluation 2014” is released, we’ll need many more initiatives like the ones described above—initiatives that help nonprofit leaders (particularly of young organizations) learn directly from their peers about how measurement can first and foremost be a tool to improve outcomes for those they serve and better advance their missions.

What are your views on the future of social sector measurement? What other exciting initiatives have you seen?

Tracker Pixel for Entry
 

COMMENTS

  • BY Phil Buchanan

    ON January 3, 2013 07:46 AM

    Great post and thanks for citing the Center for Effective Philanthropy’s (CEP) research.

    I think the statement introducing this piece (which I assume is the editor’s not the author’s) that “few nonprofits possess” the capacity for improvement is perhaps a bit more sweeping than what is supported by the research I have seen. In our survey, fully 80 percent of nonprofits said they are using data to improve performance on an ongoing basis. What our research suggests is that funders, who in other surveys we have conducted overwhelmingly say they want nonprofits to do more to demonstrate their effectiveness, are not stepping up to adequately support that work (although there are notable exceptions, of course).

    I think the critieria used to evaluate “capacity” in the Innovation Network study is debatable.

    I also would disagree that the lack of a full-time employee devoted to measurement is necessarily a cause for alarm. In our survey, the median nonprofit had expenses of $1.1 million. It’s not realistic for nonprofits of that size to have a staff member whose only job is performance measurement, but that does not mean these organizations can’t do it well. CEP, with a $7 million budget, tracks key performance indicators connected to our strategic plan and logic model, and uses that data to inform improvement, but this is not the sole focus of a full-time person. (Does Bridgespan have such a position? I could not locate it on your Web site.)

    Nor do we at CEP generally spend five percent of our budget on these efforts, which the Innovation Network, I think arbitrarily, declares as a threshold level. Fact is, what should be spent on assessment will vary dramatically based on the context of a nonprofit – and may even vary, for good reason, year-to-year.

    Finally, I would mention one other study that I thought shed a lot of light on this issue – from Johns Hopkins Center for Civil Society Studies. I thought this was an excellent, rigorous look at the reality when it comes to nonprofit performance assessment.

    http://ccss.jhu.edu/wp-content/uploads/downloads/2011/09/LP_Communique17_2010.pdf

  • Matthew Forti's avatar

    BY Matthew Forti

    ON January 19, 2013 08:16 PM

    Phil- thanks so much for a really thoughtful comment!

    For sure the State of Evaluation report is imperfect - and I would argue any report that attempts to chronicle the state of performance measurement through nonprofit self-reporting has to be taken with a grain of salt.  For instance, we have seen time and again in our client work that nonprofits believe they are measuring outcomes, but in truth are measuring outputs, or very short-term outcomes that do not adequately gauge how their constituents’ lives are improving.

    Your points about the ‘5% of budget’ and ‘1 full-time employee’ thresholds do resonate with me - I have indeed worked with nonprofits that are exceptional at performance measurement with one or neither of these conditions met.  (To your question-Bridgespan does have a Director of Impact Assessment, though she is part-time and measurement responsibilities are spread across several other folks as well, much like the approach it sounds like CEP uses).

    Perhaps the byline of the piece (which was the author’s) was overly pessimistic in the hopes of being provocative, though we do believe the sector has a ways to go on measurement for continuous improvement.

    Thank you again for the comment and the important work CEP does in encouraging funders and nonprofits to measure for this purpose.

Leave a Comment

 
 
 
 
 

Please enter the word you see in the image below: