Stanford Social Innovation Review : Informing and inspiring leaders of social change

SUBSCRIBE | HELP

Nonprofits

Social Good = Scale x Impact (Who Knew?)

Why both nonprofits and academics should focus on scale and impact in this simple formula for good.

Pursuit of social good has never been more prominent than it is today. Funding flows are larger than ever. The number, size, and variety of organizations delivering services are all steadily increasing, as are the amount and quality of research and evaluation that academics and others are carrying out. Our mission, should we choose to accept it, is to make the most of this historic opportunity and maximize impact on people and planet.

However, we are troubled by the widening gap between how those delivering services (particularly nonprofits) and those evaluating interventions (particularly academics) approach this challenge. Nonprofits often focus on scale while evaluators focus on net impact. We need both, and we need nonprofits and evaluators to adapt their approaches in pursuit of maximum social good.

At its worst, nonprofits’ laser-focus on scale manifests as “empire-building”—some raise vast funds to get big fast and assume impact will follow. But more often, this focus is for good reason: Nonprofits are keenly aware of the magnitude of the social problems they address and want to grow beyond the microscopic “market shares” they typically command. At One Acre Fund, for example, we will serve roughly 200,000 African staple-crop, small-scale farm families this year—just 0.4 percent of the potential market of 50 million. Many peers, including several contributors to the recent “SSIR x Bridgespan Achieving Transformative Scale” series, face similar challenges.

Pursuit of scale with little regard for impact is dangerous. When nonprofits aren’t equally concerned with continually improving their programs through measurement and R&D, they are less able to create transformational change.

On the other hand, evaluators (particularly academics) often devote too much mindshare to “net impact” (impact per cost), as they pursue the best “bang for the buck” in a given outcome area among the maddeningly large number of interventions from which donors, policy-makers, and replicators must choose.

But pursuit of high net impact with little regard for scale is also dangerous. Too often, it means evaluators conduct research under idealized conditions, with little regard for whether a given intervention will ever impact the lives of millions. Too much academic research ignores real-world settings (will impact hold up under greater contextual diversity?), implementation factors (how do leader characteristics or the regulatory climate, for instance, influence outcomes?), and scale factors (what is the real cost of the intervention at scale, and what mechanism will allow it to reach lots of people?). The sad result is that few interventions that “prove” effective through high-quality research actually scale.

Bringing Together Scale and Impact

At One Acre Fund, we are working to de-isolate these two critical factors, and achieve both meaningful scale and net impact. We’ve grown more than 75 percent per year in farmers reached in the last five years and nearly doubled our net impact per client (measured as the incremental farm income we generate, less the donor cost to generate that income, on a per-farmer basis). We validate our scale and net impact through robust internal systems, and calculate our impact by physically weighing the harvests of our farm families and comparing them to a well-formed control group of newly enrolled families. Recognizing we still have room to improve, we constantly experiment with new measurement designs and techniques to more definitively attribute our impact.

Behind this progress is an extremely simple visual (see below) we use to guide our decision-making—the “social good box.”

This construct has been extremely helpful to our thinking. To share three examples:

  • We recently segmented our growing R&D operation into two arms: product innovations (testing and rolling-out more impact-enhancing products to clients) and scale innovations (searching for opportunities to increase our penetration in existing areas).
  • Our recent strategic plan includes a mix of initiatives, some focused on higher net impact per client (for example, improving soil health and climate resilience), others on greater scale (such as more rigorous new country scouting operations).
  • We now conduct new product trials by examining both expected adoption (scale), and expected impact per adopter, helping us avoid enticing and very high net impact products that would have relatively modest adoption. Before this shift, for instance, we introduced passion fruit to our Kenyan farmers based on research showing very high profit potential, but we completely underestimated the challenge of convincing farmers to adopt this entirely unfamiliar product. It was a colossal failure, and we soon discontinued the product.

Increasing the Size of the Social Good Box, Sector-wide

Nonprofits must devote much more attention to measuring and managing performance to achieve maximum net impact per client, and academics can be powerful partners. In our own work, we have benefited tremendously from partnerships with IDinsight, which uses rigorous field experiments to help leaders improve their programs and decisions, and UC-Berkeley’s Center for Effective Global Action, which conducts research with our farmers on topics of joint interest—we then rapidly scale those with high net impact through our network.

Meanwhile, evaluators must give greater consideration to client adoption in their research and adjust study design accordingly. They should also devote more time to measuring at-scale work implemented by real-world actors who plan to further scale after the research concludes, or create another mechanism to achieve scale based on successful research. A great example of the latter is Evidence Action, a year-old organization incubated out of the evaluation firm Innovations for Poverty Action (IPA). The organization’s leadership is already hard at work on scaling two interventions—chlorine dispenses and deworming treatments—backed by rigorous evidence from IPA’s research.

Finally, funders—particularly foundations—need to help bridge the gap. They can insist that nonprofits verify scale and net impact for each intervention they support, and do the same in their own due diligence on prospective grantees. The latter means paying deeper attention to the integration of net impact (outcomes relative to a well-constructed control group, full cost to serve a client today and in steady state) and scale (including market size, barriers to entry, resource magnetism of the leader, and leadership bench).

Only by working together, and prioritizing scale with impact, will we maximize the social good we achieve.

Tracker Pixel for Entry
 

COMMENTS

  • BY Kayla at WE THINQ

    ON July 29, 2014 01:51 AM

    Thanks so much for writing this! I’m currently working on an article about measuring social impact and I think you’ve touched on a lot of great stuff! I agree it’s essential for the non profits and other organisations doing the social good, to team up with the researchers and academics to get the most effective metrics!

  • Aaron Cavanaugh's avatar

    BY Aaron Cavanaugh

    ON July 29, 2014 08:28 PM

    Hi,
    If your a neoclassical economist (which is what mainstream economics currently is) then this is true. However if you are a classical, vulgar or political economist this formula is not true. The problem is that only things you can count are important as a social good and obviously that is only true if you are a corporatist who has no value of capitalism (economics plus social well-being).
    Thanks. God bless.
    Aaron

  • David Resetar's avatar

    BY David Resetar

    ON August 1, 2014 01:00 PM

    Great article! I have two questions, both concerned with measuring impact. Should programs that may not see significant outcomes for many years, such as those affecting climate change, be de-funded using this rubric? Also, in what situations is it fair to use income change as the only metric?

    The writers rightly point out how difficult it is to measure local context or local leadership ability. And so the Social Good box cuts the same corners and makes the same mistake of oversimplification that most social scientists do.

    Using “income change per person per cost” as the sole measure for program outcomes will give misleading results in a number of situations. Chlorine dispensers and deworming treatments are examples of important projects that might not necessarily increase gross incomes.

    Sen and Nussbaum’s list of Capabilities may offer some interesting alternatives in addition to income to tell a more complete story.

  • Matthew Forti's avatar

    BY Matthew Forti, One Acre Fund

    ON August 3, 2014 04:25 PM

    Many thanks to Kayla, Aaron, and David for their comments!

    Aaron and David both drive at a critical point which this entry did not address:  namely that the social good box is easiest to use as a construct for direct-service organizations, and especially those that address simple or complicated (as opposed to complex) problems (see Michael Quinn Patton’s description).  Two quick thoughts on this point:

    a) Even for One Acre Fund, income is not the only outcome measure we track or care about.  For instance, we measure hunger severity (for our core model) and diarrheal incidents (as we do in fact offer clean water interventions).  But income is the single most meaningful outcome we produce (in that we can attribute our intervention to its creation, and it is found to be a determinant of long-term quality of life improvement) - and we would highly recommend that, where possible, direct-service NGOs choose their most meaningful outcome (and look at the cost to produce that outcome) for the y-axis of the social good box. 

    b) For those addressing complex problems, we hope at least the spirit of the social good box will resonate.  For instance, if you engage in advocacy or neighborhood revitalization, you can still think about, and try to balance, scale (say, the # potential lives improved) and net impact (say, the depth at which those lives can be improved, per cost).  Progress may not be as linear, or as quick, as in direct service, but we believe the principle still holds.

    - Matt and Andrew

  • BY Veronica Olazabal, Nuru International

    ON August 5, 2014 11:12 AM

    Thanks Matt and Andrew for starting this dialogue. Having been on both sides of these issues as a donor and implementer, I concur with your assessment: we cannot meaningfully separate scale from impact, or impact from scale, since they are two sides of the same coin. I also appreciate the “danger” signs you touched on given that too often, I have seen development actors assume impact will follow scale and vice versa. Yet, in practice, academically defined “net impact” on a small scale means very little if we cannot replicate and scale it effectively; and large scale in terms of numbers reached means nothing if we cannot demonstrate it makes any difference. In fact, in isolation, each may disrupt and harm the very same communities we are looking to affect, essentially skewing your “social good” box toward one axis or another.

    In my work at Nuru International, our single most meaningful outcome is eradication of multidimensional poverty, with the goal of graduating the rural communities we work with out of poverty, sustainably, within a given number of years. That stated, applying your rubric would put extreme poverty (based on a multidimensional definition in support of Sen principals around access to meaningful choices) on the Y-axis. However, like One Acre, Nuru has a host of other strategies we use to assess the performance and impact of each of our four intentionally and sequentially layered interventions.

    While Nuru’s value proposition is complex and not easily measurable, like the One Acre Fund, I do believe there is a time and place for more academic studies. In the same vein, similar to the for-profit sector, I also believe that performance and impact systems can be constructed to deliver real-time ground intelligence that we, implementers, need in order to make rapid evidence-based decisions to effectively scale. Thanks for taking the dialogue around scale and impact to the next level and look forward to further discussion.

  • David Resetar's avatar

    BY David Resetar

    ON August 6, 2014 12:56 PM

    The social good box is definitely a helpful way to visualize program success. Maybe future versions can incorporate a third axis either for time or statistical distribution.

    For time, the current graph is able to either show a snapshot of a particular moment or of the average. If time were incorporated in the y-axis, you could use the volume of the shape to give you a score. Adding time would also allow you to forecast trends using regressions.

    As for distribution, (correct me if I’m wrong) the current model doesn’t take into account the variations between participants. Whether one individual made an extra $1,000 or 1000 individuals made an extra $1, the area would be the same. If standard deviation (over time) was incorporated visually somehow, the volume could be used as a more precise score for success.

  • BY Heath Prince

    ON August 11, 2014 11:44 AM

    Great article!

    The social policy field is littered with examples of organizations that focused on scale and ignored impact. Too often, size or scale is equated with effectiveness, and simple outcomes are pointed to as evidence of the “difference” that an intervention has made on the lives of the people said to benefit. This phenomenon, more often than not, speaks to the power of an influential, but not always good, idea, as well as to the pressure on funders to be seen as supporting the cutting edge. While rarely ever easy, taking a program to scale is both doable and desirable if the funding community is supportive (and the intervention is effective).
    Typically much more difficult, both in technical terms and small “p” political terms, however, is measuring the impact that an intervention actually, not just anecdotally, has.

    Fortunately, there has been an increasing demand on the part of funders, particularly those in the public sphere, for rigorous evaluations that measure impact. This has come about both for reasons having to do with accountability of funds, and out of a genuine impulse to help the people the program or policy is meant to serve.  Scaling up a bad idea is the worst of all possible worlds in the social policy field, to mix a few metaphors, and funders have taken note.

    One Acre Fund’s attention to this issue is greatly welcomed, especially given the organization’s stellar reputation and influence in the development world.

    The question, as the authors note, becomes one of how to measure impact in a way that will contribute to taking effective interventions to scale. Here, it’s difficult to overstate the importance and value of mixed-methods evaluations. A rigorous and robust evaluation might approach an intervention from several angles at once:  econometric modeling to measure impact; qualitative research to understand the context for the intervention and outcomes; as a formative evaluation in order to contribute to the continuous improvement, if desired by the funders, of the intervention; and each of these with the aim to shape policy designed to bring effective interventions to scale. At the Ray Marshall Center, UT Austin, we’ve found that this rubric—measure impact, contextualize impact, and use the evidence to educate on the need for policy change—has been a successful formula for improving well-being for the least well-off.

    One Acre Fund is to be applauded for bringing this issue to light, and its recommendations for funders, particularly related to providing support for non-profits to measure net impact, are important and welcome additions to the development dialogue.

  • Matthew Forti's avatar

    BY Matthew Forti, One Acre Fund

    ON August 15, 2014 12:29 PM

    Thank you to Veronica, David, and Heath for their great comments!

    Veronica correctly points out the need for certain multi-intervention organizations, like Nuru, to evaluate the success of their individual programs (each with their own social good box) in addition to the most meaningful organization-wide measure of success.

    David proposes some excellent advancements to the social good box - by incorporating some measure of outcome variance, along with time dimensionality to see how the organization’s social good has evolved.  A Hans Rosling ‘motion chart’ comes to mind!

    Heath makes an important point about the role mixed-methods evaluation can play in achieving a deeper understanding of both impact and scale potential (the latter encompassing both direct growth and growth through other means such as policy change). 

    Echoing this latter point, a colleague pointed me to a terrific blog entry that reminds us that scale is best conceived not as a ‘how can we grow our organization’ construct, but rather a ‘how can we bring our solution to the greater number of people’ construct, which means accommodating other pathways such as ecosystem building:  http://blogs.hbr.org/2013/01/its-not-all-about-growth-for-s/

    Thanks to all and we look forward to continuing this important dialogue in the field!

    -Matt and Andrew

  • Shawna A. Hoffman's avatar

    BY Shawna A. Hoffman, The MasterCard Foundation

    ON September 17, 2014 08:27 AM

    Many thanks to Andrew and Matt for bringing attention to this timely and relevant issue, and to everyone who’s weighed-in on this dynamic discussion.

    In reading through the blog, it struck me that the terms ‘evaluators’ and ‘academics’ were often used interchangeably. It’s true that evaluators can be academics and academics can evaluate programs; however, in many cases the two are quite distinct, with differing foci, objectives and approaches. For example, satisfying strictly academic standards of rigour vis-a-vis assessing impact often requires the construction of a laboratory-esque experiment.  This approach can tell us a lot about a specific intervention, implemented in a given context for a defined population— but the transferability of lessons learned through this approach is minimal. The challenge here seems to stem from employing a strict notion of impact (usually defined by one primary indicator) and adhering to a rigid measurement approach, devoid of the ‘real world’ considerations that ultimately drive or impede scalability. As Heath said, a mixed-methods approach is key.

    Professional evaluators offer a unique value proposition here.  Unbridled by methodological dogmatism and mandated to provide timely, actionable and practical insights around outcomes, impact and context, program evaluators might be best positioned to help bridge scale-impact chasm. Whereas researchers seek to test a defined hypothesis, with little or no regard for context or unintended consequences, evaluators are mandated and equipped to move past an ‘obsession with net impact,’ focusing more broadly on factors that intersect with and bear on the impact and scalability of an initiative.  So, academic research may be great for telling us whether something worked, but a more developmental, evaluative approach is needed to provide relevant information about the how and why. 

    All this to say that when seeking to maximize social good, organizations should be thoughtful about how they define and seek to measure impact, as well as strategic about when and how to engage professional evaluators in measuring impact and assessing interventions’ efficiency, contextual appropriateness, effectiveness, etc.  Equipped with unique skills that complement those of implementing organizations and researchers/academics, evaluators can help bridge this critical gap and move us toward the ultimate goal of greater social good.

Leave a Comment

 
 
 
 
 

Please enter the word you see in the image below: