Earlier this year, we conducted a third-party assessment of the nation’s largest philanthropic leadership program—Neighborhood Builders, operated by Bank of America. Now in its eighth year, Neighborhood Builders selects “high performing” community-based nonprofits, and provides them with a meaningful unrestricted grant ($200,000) and multiple three-day sessions of leadership training for the executive director and an emerging leader. It also fosters relationships within a network that has grown to 1,200 nonprofit leaders. The bank invests in this program as part of a broader commitment to strengthen communities where it does business.

Our research found that the program is having a strong, positive impact on participating leaders and organizations, and through them, the individuals and communities they serve. For instance, Chinese American Service League, a Chicago-based social services organization, attributes its ability to grow through the downturn in large part to the fundraising knowledge it gained at the training, the financial management advice it received from a peer organization it met at the training, and fundraising and accounting software it purchased with the unrestricted grant. 

Here are some important take-aways from this assessment that others running or funding leadership development measurement programs might find valuable:

1. Develop a detailed theory of change. It isn’t worth spending a dime on measurement until you’ve carefully defined which leaders you intend to target, what specific training and other programming they need, what they will gain, how those gains will be applied, and what should ultimately result. Ask yourself: Do you want to be accountable for long-term outcomes? At what levels (the leader, the organization, the community, etc.)? Over what time period? Should every participant receive the same set of services and benefit in the same way? For Neighborhood Builders, detailing the theory of change clarified the intermediate outcomes it sought (improvements in aspirations, leadership knowledge and skills, peer relationships, and stature with other funders) and the longer-term outcomes for improved leadership and organizational performance. 

2. Measure with mixed methods. Start with basic data tracking, leader by leader, along the theory of change: characteristics of the leaders, the amount and type of programming they access, their assessment of whether the programming was delivered as intended, etc. Then make sure to gather intermediate and longer-term outcome data after sufficient time has elapsed. While questionnaires and surveys will tell you whether outcomes were achieved, qualitative methods such as interviews, focus groups, and document review are often necessary to determine whether the outcomes resulted from the program, particularly in the absence of a comparison group. Indeed, for Neighborhood Builders, we relied on in-depth interviews using common qualitative evaluation techniques such as most significant change and alternative explanation elimination to assess the program’s influence on outcomes.

3. Continuously measure to improve impact. High-performing nonprofits use measurement to enable innovation and improvement. By gathering and analyzing your data leader by leader, you should be able to determine which types of leaders benefit the most, what programming most drives outcomes, etc.; and then adjust your approaches to achieve greater impact. Our assessment of Neighborhood Builders identified nearly a dozen potential programmatic improvements, including how to better select leaders and which follow-up services (such as ongoing peer networks and skills-based employee volunteerism) to add or strengthen. These improvements can be piloted, measured, and then adopted at scale if they work.

4. Build rigor over time. Leadership programs don’t need to build a full-scale measurement system right from the start. The best programs are intentional about whether and how to improve the rigor of their measurement over time, based partly on what they want to do with the results. Do they want to improve at their current size? Attract more resources to scale it? Inform the field? By engaging evaluators early, these programs can understand their options and implement changes now to prepare for the future. In the case of Neighborhood Builders, those options include building a comparison group, validating self-reported data, and measuring the program’s return on investment.

Measuring the results of leadership development is a lot harder than, say, understanding the impact of a tutoring program on student test scores. Yet, to avoid measuring and funding leadership development is to deprive the social sector of one of its greatest performance improvement tools.

What experiences can you share about measuring leadership development in your organizations?

We have been planning and building websites for social change institutions for nearly two decades, and over that time have worked with some of the most well-known social brands in the world, as well as hundreds of lesser-known groups. What we’ve seen across all organizations, regardless of size, is that digital teams—their structure, leadership, and how they are affected by the culture of the institutions where they work—are the biggest predictor of online effectiveness. Without well-structured teams, strong leadership, appropriate skills, and an aligned internal culture, you simply can’t do all the great things you want to do online, sustainably over the long term.

This summer we set out to learn more about the state of digital teams in the nonprofit sector. Finding few resources on the topic, we decided to create the world’s first digital team structure benchmark for the nonprofit sector. We did this in order to start a conversation about the importance of building better teams and the importance of investing in them.

Senior online leaders from 67 nonprofit organizations contributed to the final report. Here are seven of the most important patterns we observed:

Digital teams are communications teams.

Most digital teams live within the communications department—not surprising, since most of digital work is related to communications. Very few digital teams still report to IT. Interestingly, a small—but, we expect, growing—minority of “digital supergroups” report directly to an executive director. This can be an ideal location to lead new engagement functions, drive innovation, and lead change across silos, and we expect to see this model used more in the future.

Most teams need more full-time help.

Most teams are small and share responsibility for digital with others inside the organization. While the size of many teams concerns us, the collaborative model that most teams use to manage digital is healthy. One interesting finding was that mid-sized teams (6 to 10 full-time staff) have more full-time equivalent contractors than staff (an average of 8 per team). While this model offers obvious flexibility, it can also be financially inefficient and may reflect a lack of success on the part of team leaders in advocating for appropriate skilled staff.

Teams overinvest in social media and underinvest in user experience.

Social media was the most common team responsibility—not surprising, given the enormous attention paid to this field over the last five years. The next most common responsibilities were strategy, content editing, and project management. The least common were research and user-experience design. This is a concerning contrast to the make-up of many best-in-class corporate teams, which use outcome-driven models that invest heavily in both these disciplines. Many teams also appear to not value design, technical development, or analytics work, as these also ranked on the low side.

More and more are moving from reactive to proactive.

Fully 28 percent of teams describe their culture as “strategically led”, with an additional 50 percent reporting a “somewhat proactive” culture. This is important; fully 60 percent of teams are expected to always manage new initiatives, not merely act as a service desk. This reflects an important recent trend across most industries, where the digital function has evolved from a “service shop” function to more of an innovation and leadership role.

Poor structure holds them back.

Team structures are barely adequate. More than half of the online leaders we talked to report that their team structures need improvement, with an additional 15 percent reporting that a poor structure is holding back their performance. Only 32 percent say that their team structures are appropriate for the demands placed on them. We expect this last number to increase as more institutions re-structure their digital teams in the coming years.

Digital programs could have more impact.

On the whole, how effective are our digital programs? Unfortunately, 55 percent of respondents describe their programs as “somewhat effective” at serving the needs of their constituents and organization, with another 9 percent reporting they are either mostly or highly ineffective. For a sector that prides itself on innovation and impact, and where the successes of others are so easy to see, having a majority of the sector self-report lukewarm success is disheartening.

The future is brighter.

Here’s the good news: 57 percent of respondents report plans to increase their digital spending next year, with only 10 percent expecting a drop. This is significant given that overall, nonprofit budgets have flat-lined, while the importance of digital and the expectations that senior managers are placing on digital to support the entire organization are continuously increasing.

All of this research points to the fact that institutions are in transition. Some are still structured and resourced in ways that reflect the organic evolution of the web or its legacy functions (ie. publishing). Others struggle with how to balance the needs of multiple internal customers with driving new forms of technical and engagement innovation—all with teams that have not been able to keep pace with the demands placed on them. In most cases, digital is increasingly important to the overall success of the institution, yet the resources, leadership, and mandate haven’t yet emerged to fully support it.

In the coming years, digital teams likely will continue to mature and shift from reactive service, to proactive leadership and innovation hubs that drive fundamental change in how institutions operate. Many organizations will restructure and reposition their digital teams to meet the increasing internal and external expectations placed on the digital experience. And finally, team leaders will need to better make the business case for appropriate staff and funding in a resource-constrained environment. Only then will these teams receive the positioning and resources they need to be consistently effective.

The Loudspeaker Effect
Donors who contribute actively to social media sites not only position themselves as leaders on certain issues, but also they can also promote the work of their grantees. As William Perrin of the Indigo Trust has observed, “If you see an inspirational beneficiary of one of your grants on a field visit, with their permission, talk about the experience with your supporters on social networks straight away—don’t wait until the annual report.”

Information Dissemination
Foundations can greatly amplify their impact by making their data publicly available through social media. Beth Kanter, an expert in nonprofit technology, has noted that “Foundations are repositories of a great deal of knowledge…they hold vast amounts of information on social problems in particular areas, as well as data from grant evaluations, applications, and impact.” Making that data available can be invaluable for grant-makers working in the same area; they can then make decisions based on a firmer factual footing. Social media platforms such as wiki sites can be effective tools to achieve this dissemination.

Stakeholder Participation
Philanthropists can use social media to invite stakeholders to input into processes that are normally internal and opaque; which, in some cases, can help them to achieve their goals. In 2006, the Case Foundation took the innovative step of designing a grant program that “…would be almost entirely shaped by people outside its doors—from determining the grant guidelines and judging criteria to reviewing applications and voting on the winners.” In opening up the grant-giving process, the foundation funded better-targeted community projects and encouraged direct civic participation—one of the key aims of the Case Foundation.

Promotion of Transparency and Accountability
As donors become more engaged with media platforms, their activities inevitably become more transparent to outsiders. This is a positive development: Greater clarity about a grant-making organisation’s internal processes helps strengthen bonds of trust between funders and grantees. What is more, those who view conversations from the outside could turn out to be potential supporters who spread the message to others, offer suggestions on how to improve programs, or even help deliver services. The David and Lucile Packard Foundation’s OE Wiki is an example of an initiative that has been successful in opening up internal information to a wider audience, making learning, thinking, and processes public.

These, then, are just four reasons—but forceful reasons—why those few philanthropists who are still tentative take the leap should join their peers in exploring the exciting opportunities for leadership, learning and collaboration that are available to them through social media. We believe that this step will be beneficial for them, their grantees, and the wider public. 

So what’s the best way to manage digital at your organization? While no two organizations look the same, there are typically four models we’ve seen in our consulting experience: informal, centralized, independent, and hybrid. The first is typically a legacy of a poorly managed institution, the second and third get closer to what we think institutions need to sustain cultures of innovation, but it is really the fourth that can most consistently produce the integrated, customer-centric digital experience across all channels that is the holy grail of excellence in today’s world.

The most progressive organizations are learning to be like the web—they distribute digital staff across key departments, with a core group of experts that lead key initiatives, set up frameworks, and connect the dots while supporting others to lead. This organized but distributed leadership is known as the hybrid model. Unfortunately, this model is extremely challenging to pull off in strongly hierarchical institutions and requires collaborative leadership—thus, it is rare.

Here’s a look at the four models:

Informal is typically a legacy of how the Internet grew up organically in institutions: Program funding drives needs, and digital work is loosely and randomly dispersed across various functions and departments. The structure would more accurately be called un-managed, and is marked by an extreme lack of brand consistency, inefficiencies, and an overall rudderless strategy. Surprisingly, there are still institutions stuck in this pre-digital leadership stage. It’s not pretty. 

Centralized is a much more common model, as it fits well with how institutions have learned to manage other single-focus channels or functions, such as fundraising. They put digital in a silo (most often in communications), hire a smart person to run it, and serve the institution from this central place. The benefits are consistent messaging and branding, common tools, and, importantly, clear ownership and easy reporting lines. Unfortunately, these teams can be slow to respond to change, and get bogged down in heavy processes that stifle creativity from the fringes. These teams also typically have strong technical and publishing skills, but lack the ability (or, importantly, a mandate) to provide content leadership, design or usability services, and engagement with the constituents of the institution. These positions are typically low paid, low-level positions without much ability to shape how the institution uses digital strategically. This model makes for a cleanly run, professional system, but it does tend to make innovation difficult, especially around adding capacity for increasingly mission-critical functions such as storytelling and engagement.

Independent is a less common model, but it appears to be where many institutions are headed, which is a little concerning. This model sees multiple centers of digital leadership established, with digital roles sprinkled throughout the institution. It is unfortunately somewhat random: Some departments run high-performing digital silos (often with their own brands, sites, and social networks) that are “protected” or isolated from the rest of the institution, while others with less clout struggle to get the attention of an over-taxed and under-resourced central digital team. This model can create a competitive rather than collaborative culture, can end up duplicating resources, and—with no strong digital services group looking after the whole and how it all fits together for the audience—can contribute to the confusing user experiences (with subsequent low actions and conversions) found on most institutional websites. 

Hybrid is the most progressive and the most conducive to producing continuous innovation at the pace of digital change. In this model, different business units continue to build their own capacity based on their specific needs, but all digital staffers are connected to and supported by a central and strong digital experience team that directs the whole system toward long-term strategic goals. With this model, the culture of the central digital team is practicing what we’ll call “open leadership”: service oriented, highly collaborative, hyper-connected listeners, who also have the technical and content expertise to be high-value strategists. They take on leadership of high-leverage or high-risk projects themselves, but leave space for others to lead on their own initiatives.  This may sound ideal, but in practice it is a more organic model than most institutions are comfortable with. It’s actually unclear whether this model can actually exist if the rest of the institution is highly silo-ized, politicized, and competitive. To be sustainable, support for this new type of collaborative leadership needs to come via a larger change initiative from the top that moves toward looser, more adaptive structures overall.

However your institution manages digital today, the best way to improve it is to first bring more awareness to how you currently manage digital. Find out by asking some tough questions: How well do your digital channels (web, email, social media, mobile) perform against your peers? Can you innovate fast enough? Do internal departments feel well served? Does the overall institution have a sense of purpose and direction with where digital is going?

Once you know where you stand, then you can aim to move away from the chaotic and unorganized informal model, or the rigid and often slow centralized model, and instead, move toward something more flexible and collaborative that’s suitable for our networked world. It takes time, but nailing the right structure and leadership for digital at your organization will likely have a much bigger impact on your overall performance than any other digital initiative you can undertake.

Note: The final post in this series will share insights from the world’s first Digital Team Structure survey for the nonprofit sector. It will run in the SSIR blog later this month.

The facilitator of the focus group was taken aback. “How do you have time to come to this two-hour focus group but don’t have the time to go online?”

The leaders in the room saw the return on investment for a well-organized meeting with their peers to be much greater than anything they could do in the digital commons. They were more confident that they would get actionable insight into management techniques. But perhaps more importantly, the meeting provided an emotional and psychological benefit that is rarely met: a space for candid conversation about successes and failures in a safe, supportive environment.

Face-to-face conversations with other leaders—commercial and nonprofit—remind us that failure is, in fact, the norm and does not preclude success.

“If [CEOs’ successes] were graded on a curve, the mean on the test would be 22 out of a 100,” Ben Horowitz of Andreessen Horowitz recently wrote on TechCrunch. “This kind of mean can be psychologically challenging…because nobody tells you that the mean is 22.”

If the buck stops at the CEO, every failure in our organization ends up piled on my desk—whether it’s a typo on the website, a bad hire, or a missed market opportunity. After just a week, the failures stack up so high that it is hard to see past the mountain of complaints.

I remember when I was running product management at a venture in the Bay Area a dozen years ago, my reaction was the same: “Who the hell is running this place?” Now as CEO of my own organization, the disturbing answer is: me.

For Ben and other fellow CEOs from the Hip Hop Generation, these challenges are hard to accept. If, as Eminem said so well, “Success is my only ^&$%ing option and failure’s not,” it’s hard to stay the course as the failures mount every hour. You suspect that your team and board also see the growing list of shortcomings and wonder how you could have let them happen. How can you not feel incompetent?

One of the greatest gifts of my career was being asked to the join the board of Public Architecture. While passionate about its mission, from day one I was deeply concerned with nearly every aspect of its work. It failed to follow the most basic best practices in staffing, fundraising, and strategic planning.

And yet I soon realized that they were achieving astounding things that felt out of reach for my own organization, where we put such pride in doing things “right.” It placed the seriousness of all my daily failures in a new light.

Success, it appears, is about doing a few things right in the presence of countless inevitable failures. This is the difference between management and leadership: The act of management is ultimately about control, while leadership is about letting go and trusting others and the universe. It is about being able to take risks in the face of bad odds and having the resilience to see a 22 percent success rate as a realistic standard.

This, I suspect, is why the heads of nonprofits find so much value in coming together in person. Nonprofit leaders need to be able to get out of their bubbles and see that they are not alone in facing continuous failure. This can’t be done online.

While we need to continue to build formal peer-to-peer support groups for nonprofit leaders, I recommend another path. Nonprofit executive directors should join more nonprofit boards where they can not only gain perspective, but also provide critical support to one of their peers. 

When I am helping nonprofits with strategic planning, I sometimes ask the executive director what past decision made them wish they could take a mulligan. The answer in the vast majority of cases is a bad hiring decision.

You wouldn’t expect this answer if what you knew about nonprofit leadership was gleaned solely from the sector’s journals, blogs, and conferences, where the secret of success seems to be about determining the best theory of change, scaling and funding models, vision, or other, sexier topics.

But for one client of ours—the country’s leading advocacy organization for its issue area—four successive poor hires for one mid-level position had profound consequences on its scorecard: The pattern derailed a key part of the agency’s strategy, drained staff morale, and undermined the agency’s reputation in its community. And when it comes to fixing a bad hire, several executive directors have described to us how draining it is to have to manage a termination process.

Venture capitalists have long known the preeminent importance of having the right personnel. “Bet on the jockey, not the horse” is the phrase they use. The right people can fix wrong theories, models, and visions. But the best theories, models, and visions can’t fix the wrong people.

Hiring the right people is never an easy practice for any organization. But I suspect many sectors are prone to some industry-specific blind spots and bad habits. Think: a sales organization that is easily seduced by smooth talkers, or an academic institution that believes an advanced degree equals leadership ability. 

In that vein, here are three costly hiring mistakes nonprofits tend to make, plus some suggestions for how to avoid them, since in real life, you don’t get to take a mulligan.

1. The Multi-Headed Full Time Employee (FTE)

Mistake: Nonprofits generally have a multitude of needs and a scarcity of affordable FTEs. A common form of coping, especially among smaller nonprofits, is to carve out positions that break down something like this: 1/2 grant writing and donor development; 1/4 website and social media; 1/4 administrative support. The results are invariably disappointing. Few people have a strong set of such disparate skills or enjoy such a dispersed focus. And management easily loses track of what this multi-headed employee is supposed to be doing.

Solution: Discover the joys of outsourcing. Spend money on experts who are actually good at and focused on the competency you need. Yes, it might seem more expensive at first, even when factoring in taxes and benefits. But don’t forget the value of higher-quality work, more flexibility, and not having to manage in-house staff.

2. The Aspirational Hire

Mistake: An executive director probably doesn’t get into the nonprofit game unless they have great expectations for people face great obstacles to success.  They believe a low-income kid can go to college, a homeless man can find a home, and a terminally sick population can get better. They have to be willing to grab hold of even small signs of hope and ignore any number of predictors of failure.

This aspirational thinking is a crucial ingredient for any leader of social change. It also is the recipe for making bad hires. In my experience, the post-mortem on some bad hire often involves the phrase, “She can grow into this role” from the hiring process. While some level of aspirational thinking is good in hiring, the tolerances have to be much, much lower: A 50 percent success rate with your mission population is probably fantastic; a 50 percent success rate in hiring is definitely fatal.

Solution: Get professional HR help. The human resources field has evolved best practices for gauging the tolerances for any role and potential hire, in a sober and clear-eyed manner. Ask your board or other people in your network—it’s very likely they know someone who might be willing to volunteer on a specific hiring project.

3. The Thin Pool Hire

Mistake: It’s axiomatic that the quality of your hires will be only as good as your hiring pool. And for too many nonprofits, that pool is just too thin. 

Most for-profit companies have recognized that for many key positions, the pool needs to include people outside their specific industry. These companies recognize that some skill sets transfer across boundaries: management, marketing, HR, finance, etc.

Nonprofits generally haven’t recognized this truth. Look at the staff bios on most nonprofits, and you’ll see that corporate pedigrees are rare. But while nonprofits don’t have the pay to drive recruiting, they usually offer a better cause and more flexible lifestyle—both significant (and under-leveraged) assets in attracting talent. But the bigger point is that nonprofits just aren’t trying. They’re not looking beyond their own industry, and so their talent pool is always going to be thin.

Solution: While a search firm such as Common Good Careers can be helpful if you can afford it, the reality is that even search firms are most helpful if you already have an established network. And the larger point here is that nonprofits need to broaden their network beyond the nonprofit world.

Building your talent pool can be integrated into the work of building your donor pool. As an executive director, you should be looking for ways to build relationships with people in the corporate world that don’t involve immediately asking for money. Asking for help on talent scouting is a natural way—that happens all the time in the corporate world—to engage people with your cause.

Farfetched? Maybe not. We’ve all been to a fund-raising dinner where the executive director wraps up his speech by boldly saying, “Our mission is to put ourselves out of business!” The emotional audience jumps up and applauds. The leader basks in the praise. But what if a business owner really did that—intentionally?

In his book The 7 Habits of Highly Successfully People, Stephen Covey says to “plan with the end in mind.” When I trained as a FranklinCovey coach, I learned an exercise that carried the point: We all had to write about our own funerals: Who was there? What was said? It sounds morbid to some, but by clearly envisioning how we see the end of our life, we can become more intention-driven in the present. The same could be said of world-changing leaders.

When social entrepreneurs came on the scene in a big way in the ’90s, they were reformers not institutions. They pledged to lessons of the business world to address injustice and change systems. Ironically, the part of corporations they’ve most imitated is the desire to build an organization and keep it alive at all costs.

Businesses are built to last; social entrepreneur efforts should not be—they should shed light on necessary reforms, offer solutions, and change systems.

Social entrepreneurs’ priorities typically shift over time. When they begin, the priorities often look like this:

1. Create awareness of injustice
2. Create a model that remedies injustice
3. Raise the funds to fund the program
4. Build the talent to implement the model

Over time, the mission creeps away from tackling the injustice to institutionalizing the model the company created. The new priorities:

1. Raise money to keep the program going
2. Hire staff members who can raise money
3. Keep the program going sufficiently to raise money

Ultimately, many nonprofits exist purely to fund-raise. The world-changing model has become like the set on an old movie—it’s just propped up to put on a show. The commitment shifts from changing systems to paying the bills. All things related to fund-raising get elevated, while program concerns and changing policy fall by the wayside.

Imagine what would happen if social entrepreneurs announced on their organization’s launch day, “We will increase literacy in this city by 5 percent over the next 10 years. We’ll succeed or fail with our reforms, but either way we’ll shut down in year 10.” There could be four big benefits:

1. Donors would be more likely to invest funds, because they know there’s an end game. 
2. Social entrepreneurs would stay focused on long-term strategic systems change.
3. Funding for the next generation of social entrepreneurs would be freed up when their older, sister organizations shut down.
4. When organizations did shut their doors, we could celebrate their great accomplishments and disruptions instead of shaking our heads in sadness as we often do now.

Keeping the focus on the mission—not the institution—may not be as hard as it sounds. Michigan Corps, founded by Anuja and Rishi Jaitly, launched as a nonprofit in 2010 with a seed grant by Eric Schmidt, the former CEO of Google.

Redefining social entrepreneurship, they announced their mission as purposely bringing local and global Michigander’s together to change their home state. From there, the Jaitlys intentionally moved out of their leadership roles and into advisory roles—they moved to the organization’s board after year one. 

They saw results quickly. Last spring, they were heralded by former President Bill Clinton at his Global Initiative Conference for fostering a first-of-its-kind connection between Michigan social entrepreneurs and Kiva, the world-famous micro lender. The organization has also set in motion countless instances of leaders collaborating in new ways.

When I asked them how they see Michigan Corps in existence, they said: “We built the organization in response to unlocked potential among pro-Michigan leaders everywhere; as the engagement of Michiganders evolves, our organizational role must necessarily shift too. We need to keep our eyes on the mission and off the perpetuation of one entity.”

By creating an organic organization that rises up to meet challenges and goes away after it fosters connections, the Jaitlys are offering a new way for social entrepreneurs to address issues.

Maybe social entrepreneurs don’t need to announce on day one when they plan to shut down. But the sector needs to shift the definition of success from those organizations that survive to those that actually achieve their missions.

So the next time you hear a social entrepreneur say an organization’s mission is to put itself out of business, ask the obvious question: “When?”

Most nonprofit leaders believe that a comparison group study that employs control trials is the most valuable tool for evaluating program effectiveness. These rigorous research methods are used to prove whether a whole program (i.e., an intervention that has been completed and/or has crossed some dosage threshold that deems it fully implemented) led to the achievement of greater results for those who participated, versus those who did not. If the participants’ group average on a measurable outcome was “significantly” higher than the non-participant group’s average, then the program is considered successful. But it is important to note that a comparison study has to prove only that the group outcome average of those receiving the whole program experience is statistically higher than the group outcome average for those who did not participate in the program, and that the difference in the average scores wasn’t due to chance.

Statistical significance in no way means that everyone in the intervention group succeeded! In fact, most statistically significant differences in comparison group studies are unremarkable when you look at the average scores of the intervention and control groups. For example, in a research brief on Early Head Start, published on the US Department of Health and Human Services website makes the following claim: “Early Head Start programs produced statistically significant, positive impacts on standardized measures of children’s cognitive and language development. When children were age 3, program children scored 91.4 on the Bayley Mental Development Index, compared with 89.9 for control group…” This finding may benefit funders and policy makers, but what is a program designer supposed to do with these data points? The statistically significant difference can’t be leveraged from a design standpoint—what is a program designer supposed to do with the knowledge that the program participants scored 1.5 points better?

Comparative group designs do not lend themselves to the real-time learning and fast program adaptations demanded by the complex and tumultuous environment in which nonprofits operate today. This type of evaluation is not required to examine why some program participants do not achieve a desired outcome. To continually refine their programs, nonprofit leaders need to know much more, including which members of the group benefited, which did not, why, and the explicit cause-and-effect relationships. And nonprofit leaders must be involved in interpreting the data. They cannot afford to be on the sidelines, waiting for a professional evaluator to collect data, draw conclusions, and write and deliver a report, while programmatic rigor mortis sets in.

In the private sector, R&D helps product and service designers analyze what is and is not working for different customers, understand the various contributing factors, and continually test new ways to serve more people better. What does an excellent R&D function look like in a nonprofit organization? Research begins by going straight to the source—the student, the theater-goer, the homeless person—to get their direct feedback on how the program has impacted their lives. It looks at which program elements worked and for whom, and designers pay attention to both strong and weak performance to decipher how particular program ingredients cause short-term results for specific sub-groups. After preliminary and often-times sophisticated data analysis are completed, program leaders are deeply engaged in interpreting data and spearheading the innovation or re-design process, with an evaluator in a technically supportive role. And the testing process is ongoing.

Though these R&D practices can benefit nonprofits, few practice them. Recently, TCC Group examined the aggregate results of over 2,500 nonprofit organizations across the country and found that only 5 percent of nonprofits are engaging in R&D practices. The study also discovered that organizations that use R&D practices are almost two and a half times more likely to grow at or above the annual rate of inflation, regardless of the size of the organization’s budget. In particular, the following R&D behaviors are uniquely and significantly correlated with organizational sustainability and growth:

• Gathering data directly from program recipients to determine how to improve services
• Determining outcome metrics by listening to, documenting, and sharing actual client success stories and results
• Engaging key leaders and staff in interpreting the client-derived data
• Evaluating a program to figure out what aspects of it work, rather than whether the program as a whole makes an impact
• Bringing program design leaders together to assess and address the resources needed to deliver programs effectively
• Leveraging R&D insights to inform the program implementation team

Compared to rigid social science methods, R&D can help more nonprofits learn, innovate, and reach goals faster—for less money. It is also a more pragmatic way to assess program design replicability and costs, and to develop better business models that support the realistic expansion of high-impact nonprofit programs. To accelerate the scaling of social innovation over the next few years, nonprofits need to rely less on the rigor of academic experimental design and more on the vigor of R&D.

Peter York is hosting a free webinar on this topic on August 18. Register for “Success by Design: An R&D Approach to Evaluation” here.

Peter York is senior partner and director of research at TCC Group, a national management consulting firm.

This news, of course, comes against the backdrop of numerous other recent scandals involving state-supported nonprofits, including one particularly egregious example wherein a chain of publicly funded nonprofits founded and effectively run by Brooklyn Assemblyman Vito Lopez paid the Assemblyman’s girlfriend $329,910 over 12 months and his campaign manager $659,591 over the same period…for working a 17½-hour week! 

While it is true that there are currently no state rules governing executive and administrative compensation for nonprofits that receive state support, according to the New York State Department of the Budget’s January 2010 preliminary analysis of nonprofit employees contracting with just the state’s three mental hygiene agencies, there were approximately 1,926 employees with annual salaries greater than or equal to $100,000. The total value of their salaries was $324.6 million, with an average salary of $168,555. Nice work if you can get it!

While $160,000 might not seem to some too excessive, at a time when millions of people are out of work, serious questions have to be asked—especially given the example of the Levy brothers and members of Assemblyman Lopez’s posse—regarding whether these levels of pay represent appropriate compensation for the work done, the talents of the recipients, and the responsibilities of the job, or if they are, instead, political rewards or simply outright fraud.

The issue of nonprofit executive compensation has long been the subject of heated debate, irrespective of the source of that funding. Apologists like Dan Palotta suggest that to get “the best and the brightest,” the sector needs to pay its CEOs the same as for-profits corporations do, and if they become millionaires in the process, so be it. Critics such as Ken Berger, president and CEO of Charity Navigator, have argued in contrast that this is an insane approach, and that the best practice is to set nonprofit executive compensation based upon local market conditions, including such considerations as geography, cause area, organizational budget size, and compensation of comparable nonprofit leaders. Nonprofit executive compensation has become such a hot-button issue that the IRS has redesigned Form 990, in part, to force greater transparency around nonprofit compensation. The agency itself has said that it is going to be looking hard at compensation and benefit levels for nonprofit executives, warning that nonprofits that want to avoid unwanted scrutiny may need to be more familiar with such terms as “disqualified persons,” “excess benefit transactions,” and the “rebuttable presumption of reasonableness.” 

Illustrating that this is a sector issue, and not one limited to either just the United States or to the sewer-like political system of New York, last year the Canadian government considered a bill that would have allowed the government to revoke the nonprofit tax-exempt status of any charity paying exorbitant amounts to a single executive or employee. This move was prompted by the revelation that the former president of a Canadian foundation received $2.7 million dollars as a golden parachute upon being dismissed by the organization for poor performance. It is bad enough when for-profit corporations agree to such outrageous compensation packages, but when nonprofits do it, it is unconscionable. Indeed, as the examples of the Levy brothers and the Lopez nonprofit empire demonstrate, it is not compensation alone that needs to be dragged out into the light of public scrutiny, but rather the entire range of unfettered discretion in spending it seems that some nonprofit executives—and their boards—exercise.

For its part, the New York Council of Nonprofits has initially backed Governor Cuomo’s move. The council’s CEO, Doug Sauer, said that the larger nonprofit world must realize that such abuses, where charitable resources are brazenly used for the private and personal gain of executives, are “a stain on the sector.”

Yet while Sauer notes that “public trust is integral to the mission and work of our state’s charities” and Canadian M. P. Guarnieri, the author of the bill introduced in Ottawa last year, stated that its passage could “replace doubt and cynicism about the management of charities with the confidence that the personal financial sacrifices of donors are managed by people [whose compensation packages do not] make a mockery of the concept of charity”—it is clear from the stories that continue to crop up that the sector still has a long way to go when it comes to setting appropriate levels for not only compensation (and executive benefit packages), but also for tuitions, condos, designer suits, and exotic rugs that some nonprofits pay for out of donations and taxpayer dollars. Those bad apples, however many or few they may be, are hurting all of us.

Key points to remember:
• Great visual design matters a lot, but design is about more than just beautiful websites and collateral materials.
• Great design is about telling stories and building a brand.
• Your brand is the way all of your constituents see you, feel about you, and interact with you across all touch points.

So how does an organization with limited resources build a strong brand and support it with great design? Asking questions is a great place to start. Answers to the right questions shine a light on assumptions about an organization’s identity. Leaders should pose these questions to people within their organization and compare responses; they also should examine both internal and external communications with questions like these in mind:

• What is the purpose of our organization?  Why does it need to exist?
• When people think of us, what adjectives come to mind?
• When people think of us, what adjectives do we want to come to mind?
• Who do we exist to serve, and what do they care about?
• What would we like our clients or customers to say about us?
• What do they actually say?
• Why should someone use our services instead of our competitor’s?

Keep in mind that these are not the only questions that matter, nor are they phrased in some sort of magical way. But questions like these help clarify your organization’s purpose and who you serve, with a focus on their needs, rather than your own.

It’s also important to start out by asking some simple questions to inform your strategy before constructing deliverables such as a Twitter account or website. Questions might include:

• Who is the audience for this method of communication?
• Why do we think this is the best way to reach them?
• What is the main thing we want to communicate to them?
• What do we want them to do after seeing this?

Even if you ask all the right questions, there is a chance that the new website, brochure, or program you just created just isn’t as great as you hoped. Silicon Valley has started adopting agile processes—a fancy term for iterating, which is just a fancy term for creating many versions and testing them. Iterating can save a ton of time and money, and produce a much better result. So before designing an entire website, you may want to show a few pages, even in rough form, to some of the people who you are hoping will use the website. 

There will almost certainly be things that you didn’t think about or assumptions that prove incorrect. Maybe no one watches the new video on your site because your clients have slow Internet speeds. Maybe your new brochure contradicts your green image, and it would be better to conduct a digital campaign. The point is, if you are willing to show things at an early stage and are willing to watch how people use them (rather than simply asking them whether they like the product), you are bound to learn a great deal. It will be tempting to keep your great idea to yourself until you perfect it, but that will just make you less willing to hear the feedback your users have about how it needs to change.

Finally, hire professionals. It’s a bad idea to have your administrative assistant, or your development director—even if they have an interest in graphic design—creating all of your materials. Find a designer whose work you love, and talk to them. Many great designers discount their work if it’s for a good cause, and if they can’t, they can often point you in the direction of a designer who can. If you get the strategy right, don’t mess it all up by skimping on the execution. Invest in good design, and it will likely pay off in a big way.