Mismeasuring Impact
The hidden dangers of randomized controlled trials
The need to demonstrate the effectiveness of nonprofit social programs has led to a rapid rise in the use of randomized controlled trials (RCTs), for evaluation. As a result, most nonprofit sector professionals can tell you why nonprofits should do an RCT. This book tells you why they probably shouldn't, and what to do instead.
Mismeasuring Impact explores why RCTs are being embraced as the "gold standard" for nonprofit evaluation, despite the high cost and time investment required and the serious problems with using RCTs in a nonprofit context. (Most RCTs conducted in nonprofits fail to meet required standards for rigor, undercutting their accuracy). The book describes what happens inside nonprofits when they take part in RCTs, the unintended equity issues that arise, and why nonprofits feel pressured participate in RCTs despite the problems.
University of Chicago professors Marwell and Mosley's research is based on extensive interviews with key players: nonprofit managers, professional program evaluators, and program officers in philanthropic foundations. The book argues that, ultimately, RCTs are used to poorly ground nonprofit legitimacy, not to foster nonprofit improvement. RCTs also privilege program and organizational standardization over the key strengths of nonprofit organizations: flexible innovation and responsiveness to community needs.
Nonprofits and funders need forms of evaluation that lift up these strengths. Mismeasuring Impact offers alternative approaches that build strong organizations, not just cookie-cutter programs, and which funders and nonprofits of all sizes can support.
—Johanna Mair, Professor of Organization, Strategy and Leadership at the Hertie School
"Finally, a book that puts RCTs in their proper place: an evaluation method that's useful only under limited conditions. Marwell and Mosley show how the uncritical use of RCTs can harm nonprofits—by undermining relationships with communities, sapping staff morale, and putting organizations at risk. The gold standard is now, more fittingly, just a hammer in the evaluation toolbox."
—Alnoor Ebrahim, Professor of Management at Tufts University
"In an exemplary work of humane and rigorous scholarship, Marwell and Mosley stand athwart the rise of the RCT as the dominant form of evidence for nonprofit impact and yell: 'Stop!' Even if you are a devotee of the RCT—especially if you one—this is an essential book to learn from and wrestle with."
—Benjamin Soskis, Center on Nonprofits and Philanthropy, The Urban Institute
"A courageous, timely, and well-written warning against fashionable methodological assumptions in nonprofit evaluation. Marwell and Mosley demonstrate the not-so-glittering consequences of the so-called gold standard in evaluation."
—Peter Dahler-Larsen, University of Copenhagen. Author of The Evaluation Society.
"Though I did not agree with everything here, this book should be required reading for thoughtful evaluators who want to 'do no harm' and desire to adopt the right tool in judicious partnership with funders, program operators, and community members."
—Matthew Stagner, Former president of the Association for Public Policy Analysis and Management
"Rather than succumbing to pressures to embrace the Randomized Controlled Trial (RCT) 'gold standard,' this book sets the record straight with a clear description of how we got here and guidance for more effective evaluation. Essential reading for social sector leaders who want to 'do good better,' and social scientists who want to help them."
—Jodi Sandfort, Dean at the Evans School of Public Policy and Governance, at the University of Washington
"Accurately measuring the impact of nonprofit organizations remains the 'Holy Grail' for many. Yet, there are many reasons why, all these years on, we are still short of consensus on the best way to do that work. Marwell & Mosley add to our shared understanding of how one such method – Randomized Controlled Trials – may not be the 'gold standard' once advertised. While measuring impact is important, we must not lose sight of the inputs needed to ensure nonprofits are healthy and equipped to deliver the impact we all desire."
—Jeffrey Moore, Chief Strategy Officer at Independent Sector
"Marwell and Mosley show that RCTs cannot capture the complexity of the real world and undermine important nonprofit contributions to communities and society as a result. Mismeasuring Impact is necessary reading for scholars doing policy-relevant research and for policymakers who want to base their decisions on evidence."
—Mary Pattillo, author of Black on the Block: The Politics of Race and Class in the City, Harold Washington Professor of Sociology at Northwestern University
"RCT's have quickly emerged as the preferred evaluation tool for many nonprofit programs. Yet Marwell and Mosley, in their timely, outstanding book, persuasively call attention to the programmatic and organizational drawbacks of RCTs that can undermine nonprofit effectiveness, responsiveness, and capacity for innovation."
—Steven Rathgeb Smith, McCourt School of Public Policy, Georgetown University, past president of the Association for Research on Nonprofit and Voluntary Action (ARNOVA) and the International Society for Third Sector Research (ISTR)
"As experts in sociology, social work, and the nonprofit sector, the authors offer an illuminating and sobering critique of randomized controlled trials (RCTs) for charitable organizations. This book will appeal to academics, policymakers, and practitioners alike."
—David Suárez, Colleen Willoughby Endowed Faculty Fellow in Philanthropy & Civil Society, Evans School of Public Policy and Governance, at the University of Washington
