August 24, 2021
An Insider’s View on Implementing the Evidence Act
By Matt Soldner, Evaluation Officer, ED
As the Evaluation Officer for the Department of Education (ED) and member of the Evaluation Officer Council, I have been privileged to help make the vision of the Evidence Act a reality. Along the way, I have been struck by the energy, enthusiasm, and expertise of my colleagues across government and deeply grateful for the support from our colleagues at the Office of Management and Budget (OMB). This isn’t to say there aren’t areas of growth—there most certainly are! But I am more convinced today than ever before that the federal evaluation community is well-positioned to make evidence-based policymaking a reality across government and, as a result, a real difference in the lives of those we serve.
The Important Role of the Evaluation Officer Council. As many readers will already know, the Evidence Act creates three new roles: Evaluation Officers (EOs), Statistical Officials, and Chief Data Officers. How it arrays and organizes those roles across government, though, varies. Although every federal agency is required to appoint a Chief Data Officer, for example, only CFO Act agencies are required to appoint a Statistical Official and EO. (Though I should note many non-CFO Act agencies have appointed all three positions.) More variation is seen in how roles organize themselves to complete their work. For whatever reason, the Evidence Act organizes Chief Data Officers and Statistical Officials into formal councils but does not do the same for EOs. Evaluators are not ones to be left out, though, and thanks to OMB we don’t have to be: with their support, the Evaluation Officer Council was codified in guidance (OMB M-19-23) and kicked off in the fall of 2019. And thank goodness for it.
The Evaluation Officer Council—or EOC—is the official community of practice for EOs across government. Our ranks are further augmented by “plus ones,” typically EO’s chief deputy in their home agency. Taken together, the membership of the EOC has hundreds of years’ worth of collective experience in program evaluation, both in and out of government. Importantly, that experience goes well beyond how to design a randomized trial or an implementation evaluation—it includes deep expertise in the competencies required to make evaluation happen in government. This includes skill in procurement, contracts, and grants; leading teams and managing multimillion dollar/multi-year projects; statistical analysis and data visualization; plain writing and effective dissemination of complex information; and coalition building and political savvy. From my perspective, the most critical role of the EOC is ensuring that each EO—and each agency—has access to that deep well of talent whenever they need it. Further, some non-CFO Act agencies, sub-agencies, and bureaus have also recognized the importance of evaluation in evidence-building and use and have appointed an EO when not required to do so. Their expertise, and that of their staffs, is being leveraged through the Interagency Council on Evaluation Policy (ICEP).
The EOC also provides a forum for solving problems that are new to each of us as we seek to implement the Evidence Act and to coordinate with other councils and committees whose work bears upon our own. In the first year of Evidence Act implementation, no EO really knew how to craft the three statutory deliverables that are core to our work—the Learning Agenda, Annual Evaluation Plan, and Capacity Assessment—even if they’d done something similar for their agency in prior years. Luckily, the EOC became a forum for brainstorming, group problem solving, peer mentoring and feedback, and sharing promising practices. EOC meetings and events are also forums for EOs to connect with peers in adjacent functions to ensure our work is aligned. For example, the work of EOs is tightly coupled to the work of agency Performance Improvement Officers (PIOs) due to the statutory connection between Evidence Act deliverables and Agency Strategic Plans. In many agencies, the PIO’s office is a finely oiled machine, and the need to collaborate with EOs could have thrown a wrench in their work. Thanks to OMB, though, the EO and PIO Councils met together early in our first year of Evidence Act implementation, making sure each agency started off on the best possible foot, EOs and PIOs working together to achieve their mutual goals. Similar relationships exist with the Chief Data Officers’ Council and the federal Advisory Committee on Data for Evidence Building.
Challenges, and Looking Ahead. No effort, particularly as one as young and complex as the implementation of the Evidence Act, proceeds without a hitch. It should come as no surprise, then, that there are still areas of growth for individual EOs, our work within our own agencies, the EOC, and the EOC’s collaboration with our peers. From my perspective, three are of note:
- Cross-agency collaboration on Learning Agendas. Agencies will complete their first official multi-year learning agenda in fall of 2020 and share them broadly in early 2021. As they do, there is a natural opportunity for agencies to look across agendas and identify collaborations that can accelerate evidence-building and use. (In truth, this is already beginning—but we should expect more in the years to follow.) Individuals don’t live their lives in silos that mirror the organization of the Executive Branch, and we can’t build evidence in those silos, either. In my own agency, there are natural and obvious synergies with the work of the Department of Labor, the Department of Health and Human Services, the Department of Housing and Urban Development, and several others. Each agency can say the same, and I expect that each of us will come to realize the benefits of more frequent collaboration.
- Reducing barriers to data sharing. It isn’t enough, of course, for two or more agencies to realize that they can accelerate evidence-building by partnering with one another: they have to execute! That’s where things can get tricky. Again, using my own agency as an example, consider the case of trying to more completely understand the outcomes associated with one of our most important programs: the Pell grant. In academic year 2019-2020, nearly 7 million students received Pell grants of up to $6,195 to pursue a postsecondary certificate or degree. The program represents a critical—and significant—investment in the nation’s human capital. But just how large is the return on that investment? By securely linking data on Pell recipients with longitudinal income data from the IRS in a privacy-protecting way, that return could be calculated to the penny. Understandably, the process of linking administrative (or other) data across agencies is both legally and organizationally complex. The Evidence Act seeks to reduce those barriers when appropriate, and charges EOs, Chief Data Officers, and Statistical Officials to do their part to make data more accessible for evidence-building. How effectively we meet that challenge will, in no small measure, define the success of the Evidence Act.
- Ensuring the use of research evidence in policymaking and agency improvement efforts. Finally, the most carefully-built evidence isn’t truly valuable unless it is used. Neither data nor evaluation are ends unto themselves—they are each in service of improving operations and programs so that peoples’ lives are changed for the better. This means evidence must be front and center in agency decision-making processes, using high-quality evidence when it is available and ensuring that it is built when it is not. As I noted above, the Evidence Act is off to a good start in this regard as it has brought together the Performance, Evaluation, Data, and Statistical communities in ways they may not have been traditionally aligned. But that work will need to continue to be sure that the right evidence is available on the right issues at the right time. (Those of you who are particularly interested in this topic may wish to learn more about the academic scholarship surrounding the use of research evidence.)
If EOs and our partners keep working together, continuing along the strong positive trajectory we’ve seen over these past two years, I have every confidence we can meet the challenge the Evidence Act sets for us: policymaking driven by the best available evidence. When we do, it’ll be a technical, managerial, and—yes—bureaucratic feat for which all involved should feel rightly proud. But, as we all know, this work is about much more than new deliverables, processes, and structures. Government isn’t the beneficiary of evidence-based policymaking. Instead, it’s the infant and new mother who benefitted from higher-quality home visits; the middle-schooler who participated in a college access program with a proven track-record of success and was catapulted along the pathway to a career with family-sustaining wages; or the entrepreneur who received evidence-based technical support to open their small business. That’s who really wins when evidence drives policy. Time to get back to work.
Tags: