Stanford Social Innovation Review : Informing and inspiring leaders of social change

SUBSCRIBE | HELP

Measuring Social Impact

Measuring the Indirect Impacts of Market Innovators

Why measuring indirect impact matters and how on earth we can do it.

Conventional wisdom insists, “If you can’t measure impact, it doesn’t exist.” The pressure has never been higher on nonprofits, donors, aid organizations, and impact investors to demonstrate that we’re achieving what we’ve promised. Anecdotes, imagery, and faith no longer cut it.

Organizations and funders have recently poured plenty of money, time, and intellectual energy into identifying and proving the development impact of our projects and investments. Last year alone, funders spent millions on randomized controlled trials that use experimental models to tease out causal linkages between actions and outcomes. Foundations now routinely employ teams for “monitoring and evaluation.” And an entire industry has sprung up to encourage better tracking, reporting, and standardizing of social performance metrics at the company and organizational levels.

While these efforts are crucial, we worry that such focus on measuring only direct outputs and proving causal certainty might miss something important—specifically, we might miss a constellation of indirect impacts that are very real, even though we don’t yet have the tools to understand or articulate them.

Take the case of “market innovators,” which, as the Omidyar Network articulated in a 2012 report, are those pioneers—usually start-ups—that “contribute meaningfully to a sector by de-risking the generic model of an innovation or product.” These innovators generate positive sector-level externalities—informing, inspiring, and instructing others’ efforts to build robust sectors addressing the needs of the poor—even though not all may individually succeed in scaling up. Start-up pioneers often spend years laboring in the trenches to reach the first handfuls of customers. But the value of market innovators lies not just in the first customers it reaches on its own, but also in the ways that it enables others to serve millions more in the years that follow.

Sounds great in theory, but how can we measure this?

At Accion Venture Lab, we are on the hunt for precisely these externality-generating innovators. We support the world’s most innovative “financial inclusion” start-ups—those companies seeking to help underserved customers access better financial services and tools to improve their lives.

We’ve learned that supporting these innovators is hard and uncertain work, and, unfortunately, there are only a few of us looking at these seed-stage start-ups. Clearly, one big barrier to further investment in market innovators is that folks who do this work can’t readily explain or prove the indirect impacts they’re having. This is a special challenge for a number of reasons. It’s hard to define impact trajectories early in a start-up’s life. It’s also hard to attribute quantifiable impact to the activities of a single company within a sector. Even when these indirect impacts happen, the timeframe over which the impacts become visible often takes years—long enough to outlast the patience and “results frameworks” of most funds and donors. And it’s uniquely difficult to make apples-to-apples comparisons at the level of indirect impacts, even more so than the commensurability challenges of traditional cross-sector impact metrics.

So, bearing all this in mind, how might we start to measure and talk about indirect impact?

For us, this question isn’t one of simple intellectual curiosity—the answer is integral to securing the support of funders and other partners for these start-up pioneers. We don’t claim to have all the answers, but here are a few things we’re trying:

  • Define alternate pathways to scale: Start-up innovators won’t always (or even often) be the ones to achieve scale on their own. Many times, scale comes only when other players get involved, whether as partners, acquirers, or imitators. Let’s tolerate and even celebrate these alternative “pathways to scale”—when we can track a clear line from the initial experiment to a scaled effort, let’s highlight the first mover’s role in enabling success (see this example from Kenya).
  • Focus on the sector, not the firm: To borrow from Omidyar Network, we’re interested in building sectors, not just firms—so let’s start measuring that way. For example, let’s keep track of not only how many customers one of our investments reaches, but also how many customers the sector reaches as a whole. Yes, attribution of general-sector results to specific firm actions will be a significant challenge, but let’s start collecting the data now and hope our tools and understanding of causal linkages can improve with time.
  • Make efforts to raise awareness and monitor “buzz”: A start-up can’t be a demonstration case if no one knows about it, so we are proactive about sharing our companies’ stories, successes, as well as failures and lessons. We track these instances, and the time we spend speaking and writing for broader audiences. We also actively monitor our start-ups’ media hits and general industry buzz as a (very imperfect) proxy for influence. Could we someday have a Klout score for market innovators?
  • Tolerate different output data at different innovation stages: Output measures look very different during the various stages of an innovation’s life cycle. There is a danger in thinking that firms pursuing more-proven, later-stage innovations are having more impact simply because they have more concrete results. Start-ups may generate lower-volume outputs, but these may lay the foundation for steeper growth and impact trajectories in the future—that is, going from 0 to 1,000 customers may suggest a bigger impact than, say, 1,000,000 to 1,001,000 customers.
  • Be patient:  As time goes on, the job gets relatively easier. As companies evolve, we can point to indirect impacts more clearly: the emergence of copycats or competitors; measures of “crowded-in” incremental investment in particular sectors; the relative speed with which subsequent entrants achieve break-even; specific anecdotes or theories about how a given management team might have inspired, mentored, funded, or founded similar enterprises (potentially conveyed through visualization); and actual macro-level indicators suggesting that the company has demonstrably “moved the needle” in a sector or region. We hope for a day when we can construct and measure an “indirect impact multiple” for investing early. If one entrant serves a million customers but lays the groundwork to serve millions of customers, is there a way we can index this?

Is any of this as rigorous, objective, or quantifiable as a randomized controlled trial or firm-level social performance metrics? No—at least not yet. We’re in the early days of making all of this real, though the stakes are already high. Without credible proof of indirect impact, innovators may continue to face a “pioneer gap” of funding and resources. So if you believe these impacts exist and can be game-changing, let’s try to figure it out.

After all, if impact exists, we must be able to measure it.

Tracker Pixel for Entry
 

COMMENTS

  • BY Steve Boland

    ON March 4, 2014 01:49 PM

    Lots of good thoughts, but want to emphasize the importance of getting some measurement, even if it isn’t “randomized controlled trial or firm-level social performance metrics.” The pursuit of the perfect far too often harms the attainment of the “better than nothing” knowledge base so we can course correct as we go. Nice work!

  • BY Paula Goldman

    ON March 6, 2014 09:48 AM

    Great post, Paul and Rishabh.  Agreed that it is crucial for us to begin to outline and track what indirect impact looks like for start-ups.  I personally wonder whether we can come up with something akin to an “impact multiple” to show the exponential indirect impact that accrues to investments in very early stage market innovators.  In any case thanks for laying out these issues so clearly.

  • BY Avishek Gupta

    ON March 8, 2014 08:13 PM

    Nice insightful article!
    Deciding upon metrics (and finding ways to measure them) has always been a difficult thing to do in new businesses and it becomes exceedingly more complicated when it comes to measuring social impact in other words impact that is beyond the financials/operating performance of the business/project itself.

    While I do completely agree with the idea that there is indeed indirect impact of varied sorts, what would be interesting to me is to know where do you draw a line and not claim credit for impact beyond a certain point? Because when you are going beyond the clearly identifiable “boundaries” of the specific business that you have invested into to define your impact, you are exposing yourself to the opportunity to define your achievements across a spectrum which has no identifiable boundary. What I can see from the article is that you have (in some ways) addressed that issue by defining impact by the creation of a conducive ecosystem for a type of business to flourish and hence you are restricting yourself to measuring the “direct” impact of several copycat/allied businesses that you have encouraged by investing in one business/start-up. That was my limited understanding from what I read, would be interesting to know your views on defining the “boundaries” of indirect impact.

  • BY Rishabh Khosla

    ON March 12, 2014 12:31 PM

    @Avishek

    Defining the boundaries of where we expect impact to take place is more likely to be art than science at this point, and will likely vary for each investment. That said, by defining the parameters of potential indirect impact too broadly, we risk over-attributing impact from a given company. That is why it’s important to do two things at the outset:

    1) Rigorously measure all the direct outputs and impact of a given portfolio company – the financial performance, the volume of products or services delivered, and the kind of client base served – so you don’t lose sight of what you can concretely attribute

    2) Generate hypotheses at the outset about what sort of indirect impact an investment might feasibly have. For example: “we believe this startup will catalyze digital payments acceptance by small informal merchants in India.” This generally bounds the intended impact to a given geography, sector, or end-user base.  Of course, the impact might be wider-reaching, so, in this example, we can opportunistically look for examples of copy-cats in far-flung geographies deploying a similar product or approach (whether these copy-cats explicitly acknowledge it or not).

    At the end of the day, transparency and documentation is the key: separate the direct company outputs from the indirect impacts that were hypothesized, and the anecdotal indirect impacts which may have been entirely unexpected.

  • BY Rishabh Khosla

    ON March 12, 2014 12:32 PM

    @Paula
    Developing an impact multiple is a fantastic aspiration, we’d love to work with you on this! Have you done any work to date on this concept?

Leave a Comment

 
 
 
 
 

Please enter the word you see in the image below: