Wouldn’t it be nice, as we are sitting down to write our year-end checks to our chosen causes, to have a ratings system to help us make these difficult choices? Indeed, it has long been a dream of many involved with philanthropy and charitable giving to develop such an objective set of criteria to rationalize what is inevitably a highly competitive funding process. Well, several enterprising nonprofit organizations are trying to do just that. The result? Beware of what you wish for.
Case in point: the “Charity Navigator” (CN). CN is a nonprofit organization with the stated goal “to guide intelligent giving” and with the longer term aim “to revolutionize the entire charitable marketplace.” CN’s website trumpets a star rating system for nonprofits, including “Top Ten Lists” and “10 Slam Dunk Charities.” (www.charitynavigator.org) Given that the methodology is based purely on financial data (described below), one can only hope they do not succeed with their hoped-for revolution. Such ratings, in my view, can be wildly misleading because they ignore more important factors in the assessment of nonprofit work and they become downright dangerous if used as the primary criteria for judging nonprofit organizations. Is anyone else troubled by these ratings systems?
I became interested in all of this recently when I came across CN’s link on Earthlink’s homepage. (Earthlink apparently has chosen to showcase this tool as an aid to member for their holiday charitable decision-making.) CN received the mixed blessing of additional public visibility by being featured in an interview on the “Daily Show” on December 14, although it was the kind of ironic publicity that most organizations try to avoid. As I began to probe the nature and assumptions behind CN’s rating system, I became increasingly troubled by both its underlying theory and its potential impact on the public’s understanding of the workings of the nonprofit sector.
The ratings are actually created by turning financial data from the 990s and into simple ratios, e.g., the ratio of expenditures on fund-raising to funds raised in a given year and the ratio of administrative to total organizational expenses. The combination of these ratios they call “organizational efficiency” and create a series of impressive looking graphs by which organizations’ relative “efficiencies” are compared. These ratings, combined with “organizational capacity” ratings (based on measures of revenue growth and working capital) result in an assignment of stars, four stars for the highest and none for the lowest.
There are at least three fundamental problems with such ratings:
1)It is simply wrong to assume that the lower the ratio of administrative to program costs, the better the organization and the greater its contribution to society. There is no evidence for this, other than in perhaps the most extreme cases, yet CN’s “star” comparisons clearly suggest this ratio is fundamentally important.
2)The data from IRS 990 forms on which the ratings are based are notoriously unreliable for comparative purposes. One simple example is the subjective nature of the judgment involved in classifying a mass mailing that both contains information about current social issues and solicits donations. Is it a fund-raising or a program expense? Such judgment calls are repeated tens of thousands of times by organizations in filling out 990s, resulting in a fog of data about operations.
3)Ultimately, financial data, as CN admits in the fine print of explanatory notes on methodology (read only by a few intrepid researchers I am sure), tell only a small part of the story about the work of nonprofit organizations. What really matters are the organization’s vision, on-the-ground activities, and long-term results, and these are nowhere captured in CN’s charts and graphs. CN does declares an intention to create a methodology to “evaluate the quality of the programs and services a charity provides,” but, given that this is the most difficult challenge of evaluation in the nonprofit world, I am not optimistic about their chances for success.
Although the fine print of the “Methodology” section on the website issues a caveat that its ratings should not be used “as the only factor in deciding whether to support a particular organization,” the star system, “Top Ten Lists,” and “Slam Dunk” labels featured prominently on the website suggest otherwise.
I don’t mean to pick on CN (ratings similar to theirs, such as the American Institute of Philanthropy grading system, have the same problems), but such systems, touted by places like Earthlink to guide year-end donors, reinforce the problem of viewing charities like commodities and using simplistic analyses to “inform” the public. A much better approach is that used by the Wise Giving Alliance which simply indicates that a nonprofit either meets or does not meet a set of widely accepted standards of transparency and governance.