Stanford Social Innovation Review : Informing and inspiring leaders of social change

SUBSCRIBE | HELP

Nonprofit Management

Finding the Right Measurement Director

New research reveals that most organizations prioritize the wrong skills in their search for a measurement director.

Nonprofits on the journey to using measurement as a continuous improvement tool will eventually need to decide whether to create a full-time position to lead their performance measurement function. According to research we’ve just completed with nonprofits of varying sizes and domains, most organizations that do so will prioritize the wrong skills in their search for a measurement director.

Nonprofits often assume the best person for the job will be a “data wonk”—someone who can execute complex analyses that will reveal powerful insights about their programs. While few would argue the value of such a skill set, what nonprofits most need in a first measurement director are consultative, interpersonal, and change management skills. The person in that position must do far more than crunch numbers: their most important job is often to lead a broad-scale organizational change in mindset and behavior so that everyone comes to see the value of using measurement for learning and improvement.

Horizons for Homeless Children, a $10-million early childhood development organization in Boston, hired its first measurement director in 2011, when the organization became more serious about measuring child and family outcomes. Though the need for a director was clear, the role was not easy to fill. “We struggled with the right job description,” said CEO Asa Fanelli. “We thought: ‘Should it be someone who has been doing [measurement] for years?’” However, as the hiring process progressed, it became clearer that strong interpersonal and strategic skills were critical characteristics. Eventual hire Nathan Hutto noted: “Especially when you’re starting up, it is more important to have someone who is good at designing measurement around the vision for the organization and building relationships with the staff, children, and families to ensure measurement is helpful to them. In my role, I bridge different departments and need to be able to speak to people at all different levels.”

Several organizations we spoke with hired the wrong measurement director the first time around. A senior leader in one of them said, “He was a classic researcher and statistician, with a great technical background but an inability to be flexible.” The measurement director wanted to use the most rigorous methods possible to evaluate impact, but the organization needed to focus first on improving its model, which called for a different kind of measurement. Within a year, he was replaced with an internal hire who truly understood the needs of the organization. Another individual provided the technical skills that the director lacked, and the arrangement has proven successful.

Indeed, our research found that nonprofits can successfully use contractors, additional hires, or even pro bono assistance from graduate students to access deep statistical capabilities, so long as measurement directors have the base-level analytical and quantitative skills needed. In our experience, the most important of these analytical skills are an ability to: frame and test hypotheses (for example, do youth who attend the program more frequently do better?), quickly execute basic data analyses (for example, compare test scores by youth attendance rates), and zero in on potential improvements (for example, what are the root causes behind low program attendance, and how do we deal with them)?

To address this full range of necessary skill sets, you need to search for your measurement director in a different way. First, broaden the role to recognize the intersection of measurement with organizational learning, strategic planning, and knowledge management. Second, consider a more diverse set of candidates beyond evaluators and academics, such as those in strategy or program roles inside and outside your organizations, who may possess strong cultural competence and change management experience. Third, during interviews, test a candidate’s experience in working directly with staff to influence the strategic direction of a program or initiative using measurement.

Over time, organizational measurement priorities may evolve. For example, you may want to begin measuring to improve results and to prove that program models work in order to attract the resources for replication. This task puts a much greater premium on statistics experience, writing and speaking skills, and credibility with the evaluation community—qualities which may be missing in your current staff, board, and outside advisors. Your measurement director may be able to evolve as well. But sometimes a new person in that role or another leadership position will be required to take the organization to the next level.

Our research into how nonprofits build internal capacity to measure and manage performance spotlighted some other important lessons—about the vital role of leadership commitment, the importance of incentives in nurturing a culture of continuous improvement, and why organizations may want to engage outside expertise early and often to support internal capacity. These lessons are explored in Bridgespan’s latest white paper "Building the Capacity to Measure and Manage Performance," which profiles the performance measurement practices of organizations such as Roca, Youth Villages, and BRAC, and includes job descriptions, interview questions, and job posting strategies for hiring a measurement director.

What lessons would you share about hiring your first measurement director?

Read more stories by Matthew Forti.

Tracker Pixel for Entry
 

COMMENTS

  • BY Andrew Means

    ON April 24, 2012 04:18 PM

    I think this article is right on. I work for the YMCA of Metro Chicago in their newly formed Performance Improvement Office. While my boss, the VP of Performance Improvement, and I can do and understand the statistical work, our interpersonal, strategic, and change management skills have proven to be much more important.

  • BY Paul Penley

    ON April 27, 2012 08:54 AM

    As the Director of research & Evaluation at philanthropic advisory firm, I’ve observed nonprofits swinging from undermeasuring to overmeasuring once they recognize the internal and external value of measuring performance.  I highly recommend either hiring someone who knows the organization well (be it internal or an external consultant) or making sure that task number one is understanding its current status, values, and capacity.  That allows the measurement goals and ultimately the process to be fitted appropriately rather than forced awkwardly.

  • Gregory Kurth's avatar

    BY Gregory Kurth

    ON April 28, 2012 04:15 PM

    Why not just call a “measurement director” a strategic planning officer?  If data analysis is secondary to change manement and strong interpersonal leadership skills, then I’d avoid the term “measurement” in such a title.

  • BY Dr. Bob Frost

    ON April 29, 2012 08:20 AM

    I completely agree.  Most often, the ability to use simple data sensibly and in service to organizational change is more important than exotic analyses.  As a sidenote, I’ve noticed in my part of the country that only a few of those most directly charged with organizational change and improvement—such as Organizational Development/ Organizational Behavior/Organizational Change practitioners, etc.—are interested in, or rely on, a data-based approach.  They remain far too dedicated to, and reliant upon, implementing humanistic philosophies, etc. rather than newer change approaches.

  • BY William P Fisher Jr

    ON November 27, 2012 02:56 PM

    Wonderful! This really opens the door on a key issue. We absolutely need charismatic change agents who have enough measurement savvy to be able to shape measures to meet the needs of policy, assessment, and outcomes management. The problem is, of course, that even in the usual optimal scenario, neither statistical data wonks nor those oriented to sensible change applications are taking advantage of basic measurement methods and models that have been routinely available for decades. We need leaders and technical support staff capable of thinking in terms of the big picture and the practical value of what being able to:
    • dramatically reduce data volume not only with no loss of information but with the addition of otherwise unavailable information,
    • check the internal consistency of data more rigorously than is usually done,
    • transform the usual ordinal and nonlinear score units into interval and linear measuring units,
    • integrate qualitative and quantitative considerations for enhanced interpretability,
    • use measurement results to map paths to improved performance,
    • compare measures across different item sets via equating and item banking,
    • adjust for variations in rater severity/leniency with multifaceted models,
    • set up common outcome product definitions across service providers,
    • arrive at consensus standards for accountability and research,
    • etc.
    None of this is even on the radar of 99% of the statistical data wonks or the existing change agents. For more information, see http://www.rasch.org.

Leave a Comment

 
 
 
 
 

Please enter the word you see in the image below: