As we strive to put the COVID-19 pandemic in our rear-view mirror, it is important for career and technical education (CTE) researchers and practitioners to reflect on lessons learned and consider new approaches for building strong partnerships with districts and schools. In this blog post, Margaret Hennessy, a K–12 education research associate at MDRC, discusses the changing education research landscape and explores strategies for recruiting districts and schools to participate in CTE studies.

The K–12 education research landscape is rapidly evolving. As researchers conduct CTE evaluations in the coming years, it will be important to understand how the recruitment of districts and schools for studies has evolved, in large part due to disruptions from the COVID-19 pandemic. We know that CTE program delivery and operations changed significantly during the pandemic, affecting the engagement of students and staff in ongoing research projects.

What has been less frequently reported were the ways in which site recruitment changed and the implications for CTE evaluations. Researchers attempting to launch studies during the pandemic had to shift strategies overnight to account for the realities of virtual site recruitment in an environment where research was justifiably low on educators’ priority lists.

Understanding these shifts and their implications for evaluations is crucial for funders, researchers, and policymakers because there is limited information on evidence-based CTE strategies and relatively few causal studies of CTE.

Lessons learned from the pandemic

To continue to expand the evidence base for CTE, it is important to take stock of what the research community learned during the pandemic about recruiting districts and schools to participate in studies.

Districts and schools are facing many challenges. Identifying districts and schools that have the resources and capabilities to engage in a major research evaluation is challenging. For example, a team conducting recruitment during the 2020–22 school years for a recent evaluation, Exploring the Role and Effects of High School Advising on CTE Students’ Transitions to Postsecondary Education and the Workforce (CTE Advising), found it difficult to engage districts because they were so overwhelmed with addressing their own internal issues related to the pandemic. In addition, many districts at the time were facing staffing shortages and higher-than-normal turnover, both of which continue to be impediments to site recruitment for major evaluations. Looking ahead, recruiters will undoubtedly face the challenge of burnout among those educators who remained and persevered but cannot handle adding another thing to their plate, which already includes recently adopted pandemic-related interventions.

Finding service contrast in a saturated field is difficult. The range and variety of CTE programs that have cropped up and been adopted in recent years has made it more difficult to find a clean “business-as-usual” counterfactual for a randomized controlled trial. For example, the CTE Advising evaluation was testing two specific CTE technology tools. The team found it quite challenging to identify eligible districts that met all the study’s counterfactual criteria because the two tools were already so prevalent across K–12 classrooms. Contributing to this challenge, more states are adopting CTE technologies or creating their own freely accessible platforms for their districts to use.

District research approvals can cause significant delays. The timelines and requirements imposed by district-level institutional review boards (IRBs) are posing challenges for recruitment. Increasingly, district IRBs are restricting or limiting external researchers’ access to student-level data. In addition, it is not uncommon for IRB chairs to limit a recruitment team’s access to school leaders while an application is under review, which delays the relationship-building process in recruitment.

At the height of the pandemic, districts’ research and legal offices were overwhelmed by more pressing issues, such as mask and vaccine mandates. More recently, districts have been navigating a flood of information requests competing for their attention. These issues have contributed to prolonged district approval timelines for external research applications, which in an average large district typically take 6–8 weeks in normal times. Prolonged approval processes may result in delayed site agreements and project startup.

School autonomy in external evaluation decisions has increased. Another reality that has become more evident in recent years is the shift toward greater school-level autonomy in deciding whether to partner on an external research project. Researchers have long relied on a “top-down” approach to K–12 site recruitment, wherein a district partner played a large role in bringing schools on board or simply informed schools about their participation after the decision was made. This shift by many districts to include schools in the decision-making process as to whether to participate in external evaluations is good for all involved. However, it makes planning for site recruitment more difficult. Researchers can no longer assume that a district champion will bring on all the schools needed to reach a study’s sample targets. The consequence of this changing power balance is that evaluations may get far fewer schools participating and only after much more intensive involvement from the recruitment team.

Strategies for building strong partnerships with districts and schools

There has never been a more important time to conduct evaluations of CTE programs. With growing support for CTE, state and district CTE programs have been expanding rapidly, increasing the need to build evidence about which programs and models are most effective for students and most cost-effective for districts. Currently, the field must rely on a patchwork system of data and a few causal studies to assess the efficacy of the short- and long-term growth of CTE programs. Now is the time to foster research partnerships with practitioners to help them build an effective data strategy and learning agenda for continual CTE program improvement. So what can be done to tackle the challenges?

Consider the research-practice partnership (RPP) model. RPPs offer a great model for long-term working relationships between practitioners and researchers that make it possible to develop and sustain research projects that address education needs. Although their structures vary, RPPs ultimately benefit both practitioners and researchers due to the shared commitment to create mutually beneficial research questions and designs. Given the current K–12 landscape, the effort to develop and fund an RPP is worth considering as a long-term strategy to address the challenges associated with getting districts’ attention and energy.

Survey the landscape early. Funders and researchers should build enough time and budget into their project development phase to get into the field early. Conducting early conversations with target participant populations will help establish whether the study’s design and questions will be a fit for a district or school. It also is crucial that researchers go into project development with the community at the forefront to ensure the research is applicable and worthy of practitioners’ valuable time. These conversations can help the recruitment team answer the following key questions to guide their outreach strategy:

• Why is this project worth a district’s or school’s limited time?
• What remains unclear about the research project’s design?
• What research question is most exciting to practitioners?
• When can partners engage with district and school leaders in the recruitment process, and how can they do so most cost-effectively?

To help with this early effort, researchers can draw on the insights of CTE networks, such as the CTE Research Network (CTERN) and the Association for Career and Technical Education (ACTE). Research teams also can hold conversations with state-level CTE offices that can potentially connect them to districts and schools. Although getting in the field early does require additional up-front resources (especially when done in person, which is ideal), the effort may help researchers avoid a “so what” reaction or nonresponse to their later recruitment efforts. Ultimately, surveying the landscape will help the team refine its site recruitment strategy and make its research findings and products resonate with the intended audience.

Expand the target population. If it appears that it will be difficult to find a counterfactual, consider expanding or redefining the target population early in the recruitment effort. Researchers have long relied on achieving the sample size they need for a randomized controlled trial through partnerships with large urban districts. This strategy is no longer as effective as it once was for some of the reasons explained in the previous section.

For example, in the CTE Advising evaluation, the team pursued a strategy that researchers often avoid: recruiting rural one-school districts. Although researchers sometimes assume this strategy will lead to higher data-collection burdens, there was a payoff that made it worthwhile. The CTE Advising recruitment team often heard from these districts that unlike large urban districts, rural one-school districts do not often receive opportunities to participate in U.S. Department of Education projects and to receive the associated benefits of participation. Ultimately, this factor contributed to the districts’ willingness to participate and their high levels of engagement in the research itself. In one of the states, the team was able to secure a data-sharing agreement through the state’s research office, reducing the cost of collecting administrative data district by district. In addition, research teams are increasingly using virtual methods to collect qualitative data, which mitigates the cost concerns associated with conducting in-person site visits to rural districts.

It's a numbers game. To account for these new realities, researchers would be wise to staff, plan, and budget for a site recruitment numbers game. The CTE Advising team reached out to hundreds more districts than it anticipated, which required more “boots on the ground” and staff time to conduct the outreach needed to yield a sufficient sample. Funders should be aware that the recruitment process has changed, and they may need to allow for longer timelines and higher levels of effort to accomplish the same goals that could be done with less time and effort in the past.