Good Measure
As a new subscriber, I want to thank Alana Conner Snibbe for her arresting “Drowning in Data” (SSIR, vol. 4, no. 3, p. 38). I sent the article to our top management and board of directors at Volunteers of America Los Angeles (VOALA), and now we are reconsidering our ambition to measure the effectiveness of our programs.
VOALA has an annual budget of over $35 million to help people in tough straits with programs like homeless drop-in centers, substance abuse treatments, ex-offender halfway houses, senior nutrition and support centers, family strengthening activities like Head Start and Upward Bound, and even Grameen-style microlending. Our management and board are frustrated by the fact that most of our client populations are growing. We can’t keep up. And yet our funders profess great satisfaction with our work and want us to do more of the same. Our fondest desire would be to work ourselves out of business, but instead, we are a growth business in a growth industry – hence our desire to get brutally honest about our programs via evaluation. We want to modify some, shut down others, and focus on those with the greatest promise for long-term positive impact on social ills. Working on just one program would be OK with us. We are do-gooders who actually want to do long-term good, not just short-term feel-good good. But your article brought us up short. What to do now?
Our increasing frustration comes at a time when we have a substantial financial war chest for creating and supporting new or expanded programs of our own design. We want to challenge and step away from the status quo and conventional wisdom. We are trying to encourage entrepreneurship in our ranks and amongst our collaborators to achieve a different outcome for our clients. To whom can we turn for wisdom? What social service organizations have taken a similar attitude and path toward change and effectiveness? What leaders have done this? Who can help us avoid the land mines embedded in concepts like “results” and “accountability” and “effectiveness?” We don’t lack contacts, but ours tend to sing from the traditional hymn book.
KIM CALDWELL
Board Chair
Volunteers of America Los Angeles
Los Angeles, Calif.
Alana Conner Snibbe replies: Thank you for your letter. I could – and should – write another article about how nonprofits can make better evidence-based decisions without breaking their banks or overburdening their staff members. Until that happens, though, let me recommend to you the person and works of Michael Quinn Patton, whom I briefly mentioned in my article. Patton’s book Utilization-Focused Evaluation: The New Century Text (SAGE Publications, 1997) outlines how to ask and answer useful questions based on easily obtained data. I also suggest that you check out the Center for What Works (http://www.whatworks.org) for road maps on what to measure and how to measure it. And if you want to save yourself the trouble of remaking the wheel every time you start a new program, consult the Campbell Collaboration (http://www.campbellcollaboration.org) for a listing of proven-effective programs in the areas of social welfare, education, and crime and justice.
We appreciate Snibbe’s recognition of our efforts at the Center for What Works. We are committed to establishing research and methods for nonprofit benchmarking – a systematic approach to facilitating performance comparison, learning, and improvement.
To address the limitations described by Snibbe, the center is actively working to strengthen current performance measurement techniques by developing new research and implementation tools for nonprofit grantee use. These techniques, research, and other resources are made available to the sector through the center’s Web site. Our membership network will provide opportunities for nonprofits to compare their best practices with other organizations.
Snibbe emphasized the role of funders for evaluation success. Social sector organizations and their leadership need to remain vigilant about improving their impact, and funders play a critical role in standardizing measurement. The contemporary model for philanthropy is no longer solely as financier, as foundations are repositories for the collected experiences of their grantees. Knowledge is the currency of the 21st century, and funders are in a unique position to compile, process, and distribute the information they receive from grantees. Snibbe pointed to our recently completed two-year research project with the Urban Institute. This project compiled lists of researched outcome indicators for 14 program areas, ranging from advocacy to youth tutoring. Additionally, we developed standardized outcome sequence charts or logic model-like schematics for each program area. Finally, we developed a taxonomy of common outcomes – general outcome indicators that can be used for any program. As the article states, “By using these standard measures … funders will not only lighten their grantees’ evaluation load, but will also be able to compare programs more easily.”
Without leadership from the grantmaking community, these rigorously researched outcomes will be helpful only to those aggressive nonprofits that actively seek them out, rather than providing the opportunity to develop a universal baseline. If foundations used baseline outcomes and indicators for prospective and current grantees, the common language would enable nonprofits to learn from each other, and grantmakers to compare and learn from their grantees as well. By developing a comparable language for performance, we would see many positive changes across the social sector.
DEBRA BRODSKY NATENSHON
CEO
The Center for What Works
Chicago, Ill.
Snibbe has sounded an alarm that foundations, nonprofits, and evaluators all should heed. No one doubts the value of serious self-reflection and thoughtful evaluation. But far too often, nonprofits feel compelled to collect data that they have no resources to compile and analyze, in order to answer questions that seem irrelevant, as well as create reports that no one will read about impacts that are not currently measurable. In the end, little is learned.
As troubling as the situation may be for the kinds of direct service programs Snibbe describes, the problems are compounded in policy and advocacy programs. As more foundations are funding policy and advocacy activities, they are confronting the unique evaluation challenges this work entails. Measuring the effectiveness of any single organization’s contribution to work that is – by its very nature – collaborative, complex, subject to forces beyond the organization’s control, long term, and evolutionary requires a new way of thinking about evaluation. Counting outputs, like the number of alerts faxed to an advocacy network, can’t demonstrate social impact. Trying to figure out which coalition member is really the one to take credit for getting a bill passed is futile and pointless. Measuring policy success at the end of a one-year grant is arbitrary and shortsighted. The metrics and assumptions used to assess service programs simply don’t apply. Experimental and quasi-experimental models don’t make sense in these environments, and as Snibbe points out, the questions an experimental design sets out to answer may not even be the right questions to ask.
Formative evaluations that help nonprofits and their funders learn along the way, modify strategies, set capacity-building benchmarks, and capitalize on opportunities make far more sense for policy and advocacy initiatives. They may not offer funders the satisfaction of tracking clear wins and losses that their colleagues in the business world enjoy, and they may not be as helpful in making a splashy case about impact to their boards or the press, but they can do a great deal to promote nonprofit effectiveness and social change. And they can do it in a way that offers value to the nonprofit partner. Our work on behalf of the California Endowment, “The Challenge of Assessing Policy and Advocacy Activities: Strategies for a Prospective Evaluation Approach” (K. Guthrie, J. Louie, T. David, & C. Crystal Foster, October 2005), and a follow-up paper published last fall offer some tools for addressing these very issues.
To their credit, a growing number of foundations are wrestling with these challenges and are truly seeking to develop evaluation approaches that ease grantees’ burden and promote learning for both the foundation and the grantee. When funders, evaluators, and nonprofits all are willing to take risks, move beyond traditional notions of success and failure, and seek out new strategies for measuring effectiveness, everyone can gain.
CATHERINE CRYSTAL FOSTER
Principal
Policy & Advocacy Consulting
Palo Alto, Calif.
JUSTIN LOUIE
Senior Analyst
Blueprint Research & Design
San Francisco, Calif.
Snibbe has written an excellent summary of the state of nonprofit evaluation. And she has done it with input from some of the best thinkers in the field – Patrizi, Cook, and Patton, among others. Every funder and grantee should be discussing the issues raised in her article. Nevertheless, I feel compelled to disagree with a central point.
It isn’t a matter of one type of evaluation being worse than another. It’s a matter of what questions you are trying to answer, and how confident you need to be of your evidence. If I want to know how much better one intervention is than another, or how much better an intervention is than no intervention – and I need statistically significant results – then I have no choice but to use some sort of experimental or quasi-experimental design letters(control groups, comparison groups, time series, etc.).
But it is quite rare that funders and nonprofits need the answers to those particular questions. Most of the time, what they really want to know is whether an intervention can have positive outcomes under the right conditions, and if those outcomes are worth the investment. They need to know these answers “beyond a reasonable doubt.”
These answers can be found by collecting data from stakeholders that will tell the story of change over time and allow for logical links between the intervention and changes in individuals, groups, organizations, or communities. Usually, this approach doesn’t require a great deal of time or money. It does, however, require being very clear about what you want to know, why you want to know it, and how you are going to use the information.
STEPHEN J. GILL, PH.D.
Owner
Stephen J. Gill Consulting
Ann Arbor, Mich.
Think Like a Start-Up
I largely agree with the thoughts presented in “Capitalizing on Convergence” by Austin, Gutiérrez, Ogliastri, & Reficco (SSIR, vol. 5, no. 1, p. 24). But I feel their point that corporations “must learn from nonprofits how to create and measure social value” is less substantiated. I’d submit that most nonprofits don’t know how to measure the social value they create. Nonprofits have become more adept at defining plans and working toward targets, based on an assumption of the correlation between results and outcomes. The extent to which this assumption is tested appears to be minimal. The business corollary is the creation of shareholder value, or share price, and social and environmental impacts. Many social responsibility reports include a high degree of impact reporting against social and environmental goals.
But there are certainly things that corporations can learn from nonprofits. Corporations may have better “professional processes,” as the authors put it, but they don’t do as well engaging employees in the entrepreneurial spirit that runs many nonprofits and is necessary for innovation and dynamicgrowth. As corporations grow, they become weighed down with “professional processes” and an overriding growth/profit motivation that distances employees from their aspirations to make a difference. Conscious capitalism may be here to stay, but it tends to be externally focused (reputation and risk management). The degree to which corporations engage employees in their social missions can be correlated to business performance. The Body Shop, Ben & Jerry’s, and Timberland are probably the bestknown examples of this. Other corporations are slow to follow.
ELAINE COHEN
Founder and Manager
PotentialOne
Herzliya, Israel
Socially Irresponsible Nonprofits
Mal Warwick’s “Guess Who’s Socially Irresponsible Now?” (SSIR, vol. 5, no. 1, p. 12) hit home. After earning an MBA from a reputed university, I decided to seek “meaningful” work in India’s nonprofit sector. With the surge of overseas donor money in India, I surmised that NGOs would also need skilled and trained management. But I found that they simply weren’t willing to provide salaries and benefits comparable to the corporate sector. In 1998, while managing an $800,000 budget for a three-year project, I worked for $200 per month with no benefits. (Incidentally, while living in Shanghai, I paid the same amount plus benefits to my household help.)
After an M.Phil. in development studies, I hoped that my experience would be better in the United States. In my job search over the past three months, however, I have discovered that to be considered for a job, I will first have to provide pro bono services to NGOs. And after that, my salary would still be only in the range of $30,000 per year. Even entry-level salaries in the corporate world are much higher that that, with bonuses and other benefits thrown in. Consequently, I am tempted to rein in my ambition and look for “less meaningful” work.
MEENAKSHI BHATT
Fairfield, Conn.