Finding out what works

One of the more promising advancements in public policy in recent years was the passage in 2018 of the Foundations for Evidence-Based Policymaking Act: a bipartisan measure designed to improve the ability of government agencies to evaluate the success of federal programs and better inform policy decisions.

A team of McCourt researchers — led by Associate Professor Sebastian Jilke — are partnering with the Office of Evaluation Sciences at the General Services Administration, the Evidence Team at the Office of Management and Budget and colleagues in agencies to leverage the Evidence Act as an opportunity to develop a new model for government program evaluation.

With support from Arnold Ventures, Jilke and co-investigators Associate Professor Nada Eissa, McCourt Chair Donald Moynihan and Assistant Professor Andrew Zeitlin, as well as their affiliated project members, are executing a proposal to embed postdoctoral researchers within federal agencies, tasked with developing a pipeline of project proposals to rigorously evaluate federal programs using randomized controlled trials.

This work is just getting off the ground in the summer of 2021. As of July 2021, two postdoctoral fellows have been onboarded: Paul Lagunes, who joined from the School of International and Public Affairs at Columbia University, and John Ternovski, previously a Ph.D. candidate in political science at Yale University. The fellows are working in partner- ship with the Office of Evaluation Sciences (OES), a division currently housed at the General Services Administration, that grew out of a social and behavioral science team based at the White House during the Obama administration.

Jilke explains how the project is advancing from there: “From within OES, they’re working together with the director Kelly Bidwell, myself, and project members to have scoping conversations with various federal agencies, which will suggest potential programs for evaluation.”

Jilke and his McCourt cohorts will sort through the suggested programs and select those for rigorous experiments that meet strict criteria under the Arnold Ventures grant. “The programs must be appropriate for a very strong experimental design,” notes Jilke. “Number two, the evaluation needs to be of an important social program with outcomes that change people’s lives like educational attainment, income, earnings, child maltreatment and government spending. It should be a trial that is either based on highly promising prior evidence or be a program that has significant taxpayer investment for which the outcomes of the program are yet unknown. And that holds true for many of the federal programs, fortunately.”

After this process takes place, the McCourt researchers will shepherd fully developed proposals through Arnold Ventures to fund significant multi-year evaluations spanning three to five years. The ultimate goal? “These evaluations will lead to changes that improve the ways government delivers services to its inhabitants and therefore make real improvements in their quality of life,” concludes Jilke.