Stanford Social Innovation Review

What I believe is flawed is the belief that the use of “outcomes” in the broad social service field is intended to measure numbers. Not so. I go back to the earlier days of “management by objectives” introduced by Peter Drucker. Even MBO was not intended to just measure results in terms of numbers. Whether it is MBO, learning objectives, or outcomes, the use of this type of approach is designed to have the players think about a mutually desired outcome - a change, a learning, a capacity, an accomplishment, etc.; what that will “look” like, and then what will be experienced to help move the “client” - that person, group, community, what-have-you, toward the desired outcome. Yes, for example, group behavior changes are stated in terms like “80% of the group will be able to...,” but it isn’t the measurement per se that’s important, rather it is that a desired result/outcome was planned for that is of benefit to the participant(s), and now we are seeing what works, given all the variables.
The use of this discipline definitely improves client value by focusing on benficial experiences, behaviors, results...and for the external community they can “see” the results in terms that they can “measure.”

»» Posted by: Joe Valentine on January 17, 2004 05:07 PM

Joe,I agree that there is nothing wrong with seeking to ascertain the results of work in the nonprofit sector.  My concern is that some funders, unlike you, are beginning to assume that “outcomes” equates to finding hard data that indisputedly demonstrate those results.  This tendency both narrows the definition of purpose of nonprofit work and makes sweeping assumptions about chains of causality and the impact of single factors among a multitude of variables. And, it makes life miserable for many nonprofits.

»» Posted by: Bruce Sievers on January 20, 2004 08:09 PM

I, too, have great concerns about the current measurement mantra. To take only one example, years of research and thousands of studies on educational achievement have shown beyond any doubt that it is incredibly difficult to establish a causal link between various “outcomes” and factors presumably related to those outcomes, such as schools and teachers. This is especially true if one focuses on higher skills such as logical or creative thinking, as distinguished from simple computation or reading comprehension. In addition to the educational “inputs,” there are many other variables at work, including maturation, family effect, personal interests and abilities, and other non-school factors. And this is in an area of work where the client-service provider interaction typically involves 1,000 hours per year, far more than is the case with most other nonprofits.

Obviously, evaluation and accountability are important, and nonprofits should do everything they reasonably can to learn and divulge the effects of their efforts, but no one should assume, by mindless comparisons with corporate profit reports, that such assessment will be easy or, in some cases, even possible.

»» Posted by: Michael O'Neill on January 26, 2004 04:39 PM

I would agree that ‘metrics’ are not the be all and end all.  However, I would argue that, without some attempts to quantify results, non-profits will miss much of the funding and attention they are due.  Individuals and corporations rightly want to give to those projects that they believe will make the greatest actual difference.  In the world in which we live, ‘real’ and ‘quantifiable’ are close cousins. 

While not every impact can be given a numeric value, we should not give up on attempts to quantify results.  I don’t believe that non-profit managers are (or themselves believe that they are) guided to the right actions by a magical 6th sense.  There is a reason that the corporate world is so guided by numbers - they give strong grounds for appropriate action.  The more reliable the numbers that are generated in this sector, the greater the level of confidence that non-profits can hold in the value of their decisions, which has to be a good thing for us all.

»» Posted by: Alice Woodwark on January 30, 2004 03:26 AM

Alice, your point, which is a good one, illustrates why this issue is so maddeningly complex.  Metrics can be useful as a way of objectively assessing results of any effort, nonprofit or for-profit.  However, the fundmental difference between np and fp work is that there is, ultimately, a single test of the results of fp activity:  Does it make money or not? This is a universally accepted definition of success.  With nonprofits, on the other hand, the criteria for success are open to determination by all those with a stake in the process:  the np organization, clients served, funders, and public at large. (Examples:  educational success, artistic excellence, policy advances, youth development, human rights protection) These are in part subjectively defined and always open to revision.  While numbers can be attached to each of these, they are far from telling the whole story of what is being accomplished.  And this is assuming that the challenges of complexity and causality pointed out by Michael can be overcome, which is doubtful.  My real concern is that numbers can become the tail that wags the dog.

»» Posted by: Bruce Sievers on January 30, 2004 03:29 PM

My view of the debate over “measurable outcomes” in the nonprofit sector is colored by many years of working in the arts, a field in which many of the objectives pertain to intrinsic values that are inherently unmeasurable.  “How much were you inspired by a given painting or performance?” may be a question that can be answered in relative terms, but to apply numeric values would be of dubious usefulness.  Much of the nonprofit sector is pursuing objectives that are a blend of intrinsic and utilitarian values.  Among the intrinsic values are justice, equality, peace, beauty, and salvation, In judging a nonprofit organization’s performance, does anyone seriously believe, for example, that it would be possible to pinpoint a cost/effectiveness ratio that would determine whether the Lutheran faith was superior or inferior to the Hindu religion in delivering a virtuous life?  Or whether the experience of seeing a painting by Goya leads to higher personal income than viewing paintings by Hockney?  It seems evident that the commercial domain is almost exclusively dedicated to the pursuit of utilitarian objectives, that in many cases these objectives are subject to measurement, and that measurement can be a tool for improving results.  To the extent that the nonprofit sector is seeking utilitarian results, I have no quibble with applying measurements for improving effectiveness and accountability.  I would reject, however, the idea that utilitarian outcomes are de facto superior to intrinsic outcomes.  People who hold this view would do well to spend more time in the religious denomination or museum of their choice.  Also, I would recommend Lewis Hyde’s book, “The Gift: Imagination and the Erotic Life of Property”, which is the most elegant essay I’ve encountered on the profound differences between charitable and commercial enterprises.

»» Posted by: John Kreidler on February 5, 2004 03:46 PM

Let me add two points to the helpful discussion on evaluation so far.

1.  The single, bottom-line business metric--does the product/activity make a profit?--has its own shortcomings for BOTH business and nonprofits as an evaluation tool.  The actions and impacts of both types of enterprises are much more complex than one simple metric can measure.
Thus, certainly nonprofits should have good information about what they are doing and measure the degree to which it matches their mission and goals, insofar as it can be reliably and validly measured and proven--
But so too could business’adopt a “3-P” or triple bottom line--measuring not only impact on Profit, but also on People and Planet.  This approach seeks to capture some long-term, as well as short- term, impacts, thus acknowledging the contribution of also more subtle developmental and cumulative effects of business interventions, beyond the mere financial transaction.

2. To do really good evaluation (e.g., experimental field experiments, longitudinal surveys) is very expensive.  At a time when funding is so short for most kinds of capacity building, how would potential funders recommend that nonprofits accomplish this feat?

»» Posted by: Diane Kaplan Vinokur on February 7, 2004 08:00 PM

Dear Bruce,

There have been several broad-based histories on social policy (Alice Connolly, Michael Bates) that castigate the often unknowing role of numbers in undermining the role of social change as an outcome. In fact, my own organization at present, United Way of America, has been a prime example of a pallid social actor seen as progressive by its business supporters because of its focus on measurable outcomes,bureaucratic elegance, and numbers culture. Of course, as the movement’s prime historian, Eleanor Brilliant of Rutgers, points out, this numerology has been a screen for a fiscal “effectiveness” that has historically excluded grass roots women’s and civil rights organizations. As several historians of social thought have pointed out, the eminence grise Donald Rumsfeld left his numerocratic mark on the Peace Corps and the creep of data-based Pentagonese has steadily diluted social criticism with quantitative relativism. It is well to remember that if Victorian sentimentality was the “heart of a heartless world” (Engels) then the Invisible Hand of quantitative thought has guided us far from the passions of Progressism and the New Deal.

John Hanson

»» Posted by: John H. Hanson, Ph.D. on February 16, 2004 03:23 PM

Joe Valentine’s reference to Drucker’s management by objective provides a good basis for discussion regarding appropriate “measurables” for social service providers.  Funders or others who seek statistically valid cause/effect data or cost/benefit analyses are bound to be disappointed.

Moving a client or a group toward self-sufficiency, for instance, involves acquiring skills and learning how to use them over time.  Each individual’s progress toward that goal is the only “statistic” that indicates whether the program is fulfilling its mission.

Identifying the appropriate skills to be mastered, constructing the plan to attain them, motivating and rewarding the client—this is the tough part.  Measuring progress is relatively easy.

If a desired outcome is a sustainable income level for instance, the individualized plan would probably include basic education, life skills, economic literacy, job training, etc.  Once progress in each component is measured for each individual, the program can aggregate the data to evolve and improve their approach overall.  That’s a meaningful measurement.  Perhaps effort needs to be expended in educating the funders.

»» Posted by: Anne K. Lynn on February 16, 2004 03:50 PM

Measurement is an art not a science.  The goal of accounting is to identify the health of an organization not prove the value of its mission.  Businesses don’t have to measure the value of their services to the society.  It is inferred that a profit is a “good thing” in and of itself.  Every nonprofit competes for passion and committment of the current, and potential participants.  As a die-hard volunteer I give of my time and energy to those organizations that I FEEL utilize their intellecutal, spiritual and financial capital most wisely.  Effective community communication is the key.  One of my goals is to prove measurement can be a natural part of effective community dialogue and significantly reduce the cost of evaluation.

Jeff Clark

»» Posted by: Jeffrey B. Clark on February 16, 2004 04:29 PM

Perhaps too much is being made of “measurable outcomes”. Arts people say “tough to measure”, education people say “won’t know for years”, etc.  (I’ve been in the latter group.)

How about a more simple understanding of outcomes:  Make the measurements boolean.  i.e. Did you do what you said you would do?  Yes or no?  Did it accomplish what you hoped?  yes or no?  etc.

Often we get worked up in believing we can’t ‘measure’ or in not being accountble at all.  Most non-profits can do the simple boolean tests. 

Dick Jensen

»» Posted by: Dick Jensen on February 16, 2004 05:22 PM

While funders of all types increasingly demand measurable outcomes, I believe that much of that focus is too often based upon false assumptions, understandings and linear thinking that do not reflect the complex realities of our times (in addition to the differences between the 3rd sector and business). 

I also believe that by working in this manner, we often practice a form of alchemy through the apparent belief that quantifying something makes it real.  In its worst form, this is simply numerology posing as “good business.”

Perhaps of more fundamental concern, I think that the focus on outcomes often masks a basic fear of learning.  Foundations and nonprofit organizations would serve themselves and others far better by developing and utilizing real learning models ? and by doing so, changing the way they work (this, of course, presumes that financial support would still flow in such a scenario, an issue for another discussion).

Until we truly commit to learning- and reality-based approaches, we will too often continue to act “business-like” in a manner that sometimes looks like kids trying to act like grownups.  Creating false numerical truths yields no more reality than wearing an oversized suit and hat makes you your father.

»» Posted by: Lon M. Burns on February 16, 2004 07:56 PM

Much of the discussion on measurement and outcomes appropriately focuses on program impact - educational outcomes, artistic inspiration, etc. Perhaps measurement is better focused on process, i.e., how we deliver programming.

I have had good success in advising non-profits to first change their “service model”. The change is best established by renaming non-profits, Corporations for the Public Benefit. This introduces a critical separation in service functions - process from programming.

Next, we apply measurement (outcomes) primarily to the process side not the programming side. Two key measures are developed. The first is Reach, ie, what % of the “client” base has been reached with our programs and how often? The second measure is the Cost per reach “event”.  This is the total number of times clients have been reached with our programs divided by our budget. This can be applied at the level of the total organization or at the program/division level.

This approach assumes programming effectiveness is high and seeks to improve efficiency. Why this approach? Stretching limited resources through improvements in process assures continued mission delivery while attending to continuous improvements in operations. It is also easily understood by funders.

»» Posted by: Daniel D. Sherr on February 16, 2004 08:00 PM

I wish this weblog has been up when SSIR’s premiere issue carried my article, “The Effectiveness Trap,” which is in the same spirit as many of the comments here.  Right now CompassPoint is working on a related and intriguing question: how to evaluate unrestricted grants for core support?

»» Posted by: Jan Masaoka on February 17, 2004 12:08 AM

This is a great thread, with many valuable comments.  I won’t try to respond to all, except to say that many seem to share the concern with “measurable outcomes” for a variety of conceptual and practical reasons.  One question for John Hanson (great analogy,btw):  I am not familiar with the work of either Alice Connolly or Michael Bates and couldn’t find them through a quick search.  Could you provide a reference?  And to Jan Masaoka:  Jan, I hope you noticed the reference to your good “Effectiveness Trap” piece that helped kick-start this discussion!

»» Posted by: Bruce Sievers on February 17, 2004 12:38 AM

I find results numbers to be a fine supplementary way to compare organizations.  In a period of very limited resources, with more and more proposals to review, we want to choose the organizations that are the most effective for the most people in the areas of the most need.  In our case, we are working with local organizations that we have been supporting for years, so we are quite familiar with the less quantifiable aspects of their work, and we visit them to learn what we can about the reality of their current operations--we think we have those bases covered.  But when we have to reduce or eliminate support, we really need the best numbers on results that we can get.  And there is another need for numbers: sometimes we give grants because an organization is so good and sometimes we give grants because an organization is not so good.  When a grantmaker is trying to help an organization improve its effectiveness and build its ability to serve more people, quantifiable results are a very helpful gauge along the way--helpful to the grantmaker but indispensable to the organization.

»» Posted by: Nancy K. Zimmerman on February 17, 2004 10:42 AM

I take a practical approach to measurement in nonprofits.  In managing nonprofits for 17 years, teaching nonprofit management at the GSB for several years, and consulting on a number of Stanford ACT projects and on my own, I find that almost every nonprofit I’ve been involved with would benefit a great deal from more focus on measured outcomes, as long as they are significant, easily gathered, and replicable for regular comparision of both internal and peer performance.

Too often nonprofits shy aware from quantifiable performance, sometimes hiding behind the declaration that they’re doing good so they don’t need to be held to business standards.  Nonprofits, unlike for-profit companies, have two bottom lines--you could call them units of good and responsible fiscal operations.  I find that most nonprofits would do a better job of meeting their missions if they regularly took a hard look at how they perform in delivering on both bottom lines, and apply good business principles wherever it makes sense.

As the role of nonprofit boards is changing, their members are now seeking greater accountability, including measures of impact.  I note an increased presence of nonprofit managers with MBAs, perhaps as a response to a desire for more sophisticated analysis. And in my own organization, I seek managers with for-profit business experience to increase our effectiveness.  Public TV as a system is now seeing the benefit of a five-year old universal measured outcomes project, initiated by a group of CFOs and COOs, in which we can compare our performance to peer stations in the system on a range of performance ratios and measures.  It’s sharpened our performance, and in some instances, changed the way we work.  It’s so useful that all stations now have to submit their data in order to receive annual federal support from CPB.

»» Posted by: David Hosley, Ph.D. on February 17, 2004 11:24 AM

Dear Bruce:  Thanks for kicking this discussion off. I wonder, though, if we’re setting up somewhat of a straw dog in this debate.  Who truly thinks that a set of metrics can be found that fully define and explain the messiness of humanity and social interaction?  For many of us the issue is of figuring out how to get at the questions:  “How do we know how we’re doing?” and “How and to what do we hold ourselves and our grantees accountable?” Then, we can talk about the role of metrics (or numbers) in that paradigm…

I also find it interesting that the philanthropic sector tails in some strange ways the public sector in this debate.  The governmental sector has always been just about numbers...how many people served, how many shelter beds provided, how many people got jobs, etc.  Ironically, it was the Gingrich Congress in 1994 that pushed many in local government to begin to address what difference the anti-poverty and other programs they managed made ...a process that quickly revealed the inadequacy of the “numbers” to tell the whole story.

The question to me is not whether or not to measure effectiveness...but how to define what appropriate measures are and be clear about what we mean by effectiveness.  Most important to me, as a funder, is that the organizations in which we invest ask themselves the question--often--about how they’re doing, and how they know it--and reflect on that information to adjust and improve their program design and delivery.Are numbers part of that?  Certainly---but so is client and staff feedback, other environmental and economic factors, and so on.

There is a role for measurable meaningful outcomes---and there will always be the few who approach that need from a reductionist or simplistic vantage--but I think the majority of us are struggling with how to better frame the work that we do (or fund) so that we can, in fact, talk about why certain programs work and for whom, what we’re learning, why we think X program or organization is important to support, and so on.

»» Posted by: Pam David on February 17, 2004 01:12 PM

My own research has shown, some of the major factors affecting peoples’ willingness to donate to particular npo’s include the donor feels an affinity to that organization, they feel the npo will use the donation wisely, how much control over the use of the donation the donor will have, and who asks them for the donation.  I also found that a donor profile crafted on a modified Theory of Reasoned Action had a high degree of accurately predicting who most likely would donate.

Further, I found broadbased appeal npo’s, like United Way, can draw from a larger donor base than a narrowly focused npo.  However, the latter has a more staunch supportive donor base than npo’s like United way.  Therefore, basic marketing principles of segmentation and targeting is crucial for the npo.  Surprisingly, of all the things one can donate, e.g., time, money, jewelry, clothing, blood, artwork, etc., it seems clothing tops the list as the item most likely to be donated.

I’ve researched the donor side of npos.  Npo’s like any other kind of organization must be held accountable.  That does not mean they ought to be compared blindly with fpos.  But quantitative measures should be used; but used wisely.  A problem with the modern MBA holders is they tend to overly emphasize numbers crunching and ignore the totality of the organization.  They overlook the “emotional glue” which binds the organization together.

»» Posted by: Ira Teich, Ph.D. on February 17, 2004 02:17 PM

The discussion reminds me of a joke. A group of students took their statistics professor, who refused to consider any interpretation but an empirical one, to see a magician’s act.  Although the magician was pretty good and created all sorts of neat “outcomes,” the professor was non-plussed, he simply refused to be impressed.  Finally came the evening’s best trick: the magician would make an elephant disappear from a cage on the stage!  A cover was lowered, the magician waved his wand, and he said the magic words.  When the cover was removed from the cage the elephant was gone.  The students were astonished,and they all turned to the professor.  One of them asked: “Well, how do you account for that?” The statistics professor, without the slightest self-doubt, replied, “So what?  It’s only an ‘n’ of one.”

»» Posted by: Stephen Mark Dobbs on February 17, 2004 09:02 PM

Actually , numbers are a form of abstraction evolved from words. So to what extent our social fabric can quantify some ‘ measurable’ aspects is doubtful. Moreover “ outcome measures” less quantifiable and more qualitative can give better measurement of non-profits

»» Posted by: Atanu Hazrah on February 18, 2004 09:12 AM

Your observations are 100% on the mark.  Unfortunately, for those of us who must toe the line with higher education accreditation agencies, we must devote a considerable amount of our time, $$$, and staff to “measurable outcomes.”

»» Posted by: Moishe Gonzales on February 19, 2004 10:59 PM

I agree with Pam David’s suggestion that a straw dog is being debated--in this discussion as well as in the sector at large. 

In the first place the profit measure is not the simple and ultimate bottom line we seem to be assuming.  Not taking into account the spectacular legal and illegal ways financial information can be manipulated, stock price is the final “measure” of a public company’s worth.  But stock price is the product of the same process the Vegas bookmakers go through--a number where there are an even number of people who think the “real” number is below and above the current number.  And it changes every minute! 

Thus, while seemingly the private sector should have an easy job of measurement, billions of dollars are spent on gathering all kinds of information and hiring experts to give their opinions on the financial value of a company.  Hardly a simple or inexpensive process--and yet getting closer to the unknowable truth is deemed “worth” it because of the financial returns.

On the other hand, I think we shrink too quickly from measuring what we can measure--not in order to find the ultimate bottom line, but in order to have a better idea about the degree to which we are succeeding in accomplishing our missions.

At CompassPoint we work with hundreds of nonprofits each year.  In the vast majority of engagements a small increase in resources devoted to measurement (of financial performance, board performance, program performance, etc.) results in improved understanding that leads to better practice--in order to do a better job.  This is true even with highly unscientific and subjective measures (e.g. a 30 minute discussion at a board meeting on “how are we doing?") Measurement and the discussion of results leads to new understanding about what measures are meaningful.  The process never ends, just as the stock market never rests.  And that’s okay!

Increasingly I find that funders appreciate and accept a thoughtful discussion of performance and do not insist on single indices, but that funders are impatient (and I believe rightfully so) with no measurement at all.

W. Edwards Demming, father of TQM said “if you can’t measure it, you can’t improve it.” Putting more attention on measurement does not require an oath to submit to numbers as the ultimate determinant of quality or effectiveness.  All it does is help us do a better job--which is the point after all.

»» Posted by: Mike Allison on February 21, 2004 05:13 PM

A fascinating discussion and so timely. I can’t agree that such a discussion is a straw dog though - if it is, it’s one that bites! Of course, MBO can be seen as promoting the subjective, as ‘slavery by snapshot’, as not adequately representing the intrinsic worth of particular activities and, of course, we cannot treat all NFPs as if they were the same kind of thing. The present environment for NFPs, however, is resolutely in the market economy - whether they like it or not - because of competition for resources. It’s clear that Moishe Gonzales, ruefully, knows this. NFPs, particularly in Europe, have become the channel of choice for a healthy proportion of state sponsored social interventions and welfare programmes. In this case, I tend to go with David Hoseley’s view, and that of Nancy Zimmerman, that we just have to handle that. If an NFP agency is working to contract with a state department, or an independent funding body or an individual donor, whose objectives are we working to? The answer is - them all! NFPs have to get smart (but many are already) about co-ordinating their ethical imperatives with their functional objectives and being able to express that in a way which everyone in the organisation understands and is happy with. Everyone outside the organisation has then to be satisfied with what is delivered by way of tackling social problems and social inequity. So it’s not about creating ‘false numerical truths’ as Lon Burns suggests - with a hint of the false tautology - it’s about (amongst many other things, including the monitoring of process which Danile Sher points too) using numerical truth to support good practice in social intervention.

»» Posted by: Alan Foster on February 23, 2004 07:33 AM

This is a great discussion thread. I fully agree that there is an absurdity in the belief that all important outcomes can be quantified. My favorite example these days is in mental health, where it is becoming increasingly necessary to use ?manualized? therapies that have been scientifically evaluated in order to obtain funding. We?ve traveled a far piece down a strange road when we believe that the meaningful human connection that is at the heart of the therapeutic relationship can be captured in manual with a script that could be delivered by a smart robot.

Beyond railing against the prevailing obsession with empirical outcomes, it would be of value to me to have examples of evaluations, writings and documents that address the need for accountability of social programs but that use methodologies that offer alternatives to a narrow use of quantifiable outcomes. I?ve recently written a piece, ?Reflections on Evaluating our Grants,? (its URL is: http://www.tcwf.org/reflections/mar_2004.htm) that describes some of our attempts, successful and unsuccessful, to get to outcomes in ten years of grantmaking at The California Wellness Foundation. In it, I mention an evaluation of grants intended to influence public policies concerning violence prevention that used historical analysis as an evaluation methodology. I would be interested in references others might have. Couldn?t we build a database of evaluations that use imaginative and interesting methodologies to meet the criterion of accountability?

Best, Ruth Brousseau

»» Posted by: Ruth Brousseau on May 10, 2004 03:14 PM

What I believe is flawed is the belief that the use of “outcomes” in the broad social service field is intended to measure numbers. Not so. I go back to the earlier days of “management by objectives” introduced by Peter Drucker. Even MBO was not intended to just measure results in terms of numbers. Whether it is MBO, learning objectives, or outcomes, the use of this type of approach is designed to have the players think about a mutually desired outcome - a change, a learning, a capacity, an accomplishment, etc.; what that will “look” like, and then what will be experienced to help move the “client” - that person, group, community, what-have-you, toward the desired outcome. Yes, for example, group behavior changes are stated in terms like “80% of the group will be able to...,” but it isn’t the measurement per se that’s important, rather it is that a desired result/outcome was planned for that is of benefit to the participant(s), and now we are seeing what works, given all the variables.
The use of this discipline definitely improves client value by focusing on benficial experiences, behaviors, results...and for the external community they can “see” the results in terms that they can “measure.”

»» Posted by: Martin on December 5, 2007 02:18 PM