Data management and use through research practice partnerships: A literature review

: Through research-practice partnerships (RPPs) researchers and practitioners engage in long-term problem-solving collaborations aimed, in part, at increasing the capacity of personnel in local schools and districts to manage and use educational data for improvement. There has been an increasing number of studies exploring the process and outcomes of RPPs in the United States since 2013. Based on the review of 86 articles published during 2013–2019, we described the data management and use engagement process across various types of RPPs in the field of education and identified key considerations reported by RPPs in the United States. For researchers and practitioners engaged in the initiation and development of RPPs, the findings of this review offer empirically derived data management and use strategies for consideration.


Introduction
To challenge the one-way street approach to the intersection of research and practice in education, with its emphasis on the fidelity of research applications in practice, research-practice partnerships (RPPs) emerged to focus on "two-way streets of engagement" (Tseng, Easton, & Supplee, 2017, p. 3).RPPs offer researchers and practitioners a framework for engaging in longterm problem-solving collaborations aimed, in part, at increasing the capacity of personnel in local schools and districts to manage and use educational data for improvement (Coburn, Penuel, & Geil, 2013a;Coburn & Penuel, 2016).The benefits of RPPs have been well-documented (Nelson, London, & Stroebel, 2015;Tseng, 2012;Walsh & Backe, 2013).Henrick, Cobb, Penuel, Jackson, and Clark (2017) noted that more than merely bridging the gulf between research and practice, RPP teams do so through major contributions: 1) cultivating trusting, productive partnership relationships; 2) producing rigorous research; 3) producing knowledge that can inform educational improvement; 4) supporting the partnership in achieving important Building upon research efforts to explore the process and outcomes of RPPs, we focus our attention on the strategies employed within various RPPs to address data management and use challenges specifically.In this review, we included 86 empirical RPP studies in the United States published during 2013-2019.We used systematic literature review methods in conducting and extending previous reviews of RPPs (e.g., Coburn et al., 2013a;Coburn & Penuel, 2016;Farrell et al., 2017).Differing from previous reviews, we focused on identifying strategies specifically related to RPP data management and use.Specific review questions included: 1) How do researchers and practitioners engage in data management and use across various RPP structures?2) What strategies are used to support RPP data management and use to promote the two-way engagement essential to RPPs?

Overview of RPPs
Drawing on the foundational work of Coburn et al. (2013a), we defined RPPs as "long-term, mutualistic collaborations between practitioners and researchers that are intentionally organized to investigate problems of practice and solutions for improving district outcomes" (p.2).Tseng et al. (2017) pointed out that the key elements of successful RPPs involve the partnership structure including governance, roles, and staffing; shared commitments in terms of research agenda and research projects; the iterative communication and engagement of research production and use; researcher, practitioner, and partnership capacity building; consideration of both project and infrastructure funding; and the partnership identity.
There are various types of RPPs that have long been used by researchers engaged in educational research.Coburn et al. (2013a) conceptualized three general types of RPPs that are not mutually exclusive: 1) research alliances; 2) design research; and 3) networked improvement communities (NICs).Table 1 highlights these three major types of RPPs and sample projects.Research alliances address policy and practice questions school districts face using various data sources often maintained within longitudinal data archives (Tseng et al., 2017).The John W. Gardner Center for Youth and Their Communities (Gardner Center) at Stanford University's Graduate School of Education, for example, is a cross-sector research alliance of youth-serving organizations that connects researchers at the Gardner Center with various community sectors.
The major mission of the center is to support local children, youth, and family well-being through research inquiry into local youth experiences (Biag, 2017).Similarly, research alliances such as the University of Chicago Consortium on Chicago School Research, the Houston Education Research Consortium, and the Los Angeles Education Research Institute have demonstrated the impact of RPPs through their local partnerships.
Design research partnerships focus on the design, testing, and implementation of significant educational interventions and are situated in authentic educational contexts (Anderson & Shattuck, 2012).Design-based research has been a commonly used approach adopted by learning scientists to develop interventions, such as curriculum materials, pedagogical approaches, and instructional tools to address an identified educational problem; refine the intervention through the design-based research process; and refine or generate theories based on the design experiments (Cobb, Jackson, Smith, Sorum, & Henrick, 2013).The Strategic Education Research Partnership (SERP), for example, is a well-established design based RPP that has carried out successful curriculum interventions including Word Generation (Snow, Lawrence, & White, 2009) and AlgebraByExample (Booth et al., 2015;Cooper, 2007).
Finally, a networked improvement community (NIC) allows participating schools or school districts to share their experiences and focus on solving similar problems of practice together through collective knowledge generation (LeMahieu, Grunow, Baker, Nordstrum, & Gomez, 2017).The Carnegie Foundation for the Advancement of Teaching is one of the most wellknown NICs.To improve the graduation rate of community college students, faculty across institutions tested and refined instructional resources using a "Plan-Do-Study-Act" (PDSA) inquiry model to contribute to the larger shared knowledge base for improvement (Bryk, Gomez, Grunow, & LeMahieu, 2015).
There are other forms of RPPs in education as well.The Spencer Foundation, for example, funded 22 different partnerships between 2015 and 2018 with several partnerships involving various types of long-term collaborations among university-based researchers, state and school agencies, non-profit providers, and community partners.The Regional Educational Laboratory (REL) is another example of an RPP that carries its own unique characteristics even though it shares many commonalities with other research alliances (Scher, McCowan, & Castaldo-Walsh, 2018).There are also hybrid RPPs with characteristics of more than one of the three types of RPPs (e.g., hybrids of design research partnerships with NICs; Henrick et al., 2017).

Challenges for RPP data management and use
The characteristics of RPPs challenge the traditional one-way approach to managing research in practice and call for data management and use strategies that support two-way engagement (Tseng et al., 2017  Considering these reported challenges regarding data management and use through the lens of the proposed five-dimension framework of effective RPPs (Henrick et al., 2017), we summarized challenges regarding RPP data management and use along each dimension (see Table 2).

Inclusion and exclusion criteria
We considered publication type, publication date, study context, study methods, and study focus during the initial screening review.Table 3 highlights the inclusion and exclusion criteria used.The inclusion criteria were: 1) studies published as reports, journal articles, and conference proceedings; 2) studies published between 2013 and 2019; 3) studies focusing on problems, issues, and practices in U.S. educational contexts, including early childhood education, K-12 schools, and community colleges and universities; 4) empirical studies using quantitative, qualitative or mixed methods; and 5) articles including discussions of the nature of RPPs and data management and use process.We focused on studies published since 2013 conducted within the U.S. educational context, and we excluded books or dissertation studies in this review.
We deliberatively chose the time span of the review (2013-2019) due to the publication of the seminal work of Coburn et al. (2013a) that conceptualized the definition and characteristics of RPPs.The increasing number of funding opportunities for RPPs in the United States since 2013 (e.g., Spencer Foundation, Institute of Education Sciences) contributed to the growing number of research teams publishing studies reflecting their engagement in RPPs.Searching the key term "research practice partnership" in the Scopus database, for example, we identified a total of 148 results dating from 1994.Among these studies, 132 (89%) have been published since 2013.
While RPPs have also been widely applied in the international education context, we included only studies that were conducted within the U.S. education context.Through legislative measures and funding streams, U.S. policymakers and funders have exhorted researchers and practitioners to work together to reduce the research-to-practice gap and improve educational outcomes (Coburn & Penuel, 2016).As a result, descriptions of U.S. RPP related projects and outcomes have emerged in the literature.As a research team composed of scholars engaged in RPP efforts in the United States, we believe that our examination of the description of RPP data management and use across publications situated in the U.S. education context may offer insights for international researchers and practitioners as well.
To identify articles that met our inclusion criteria, we used both database searches and academic journal searches.For the database search, we turned to academic databases including Academic Search Complete, ERIC, PsycINFO, PsycArticles, and Scopus, and used the subject term "education" and keywords, such as "research practice partnership," "research-practice partnership," and "researcher practitioner partnership.Initially, we identified 357 results through database search and 76 results through journal and reference search (see Fig. 1).After deleting duplicates, we identified 259 records for screening review.These results all meet the first two inclusion criteria in terms of publication type and publication date.During the screening review, we applied the inclusion and exclusion criteria regarding study context, study methods, and study focus (see Table 3).We excluded any articles that focused on fields other than U.S. educational contexts, any articles that were not empirical studies, and any articles that did not include specific discussions of data management and data use (e.g., articles that address overall features and principles of RPPs).The screening review excluded 148 articles and reports and resulted in 111 articles that were included for full-text review.The full-text review excluded 25 results that do not include discussions regarding data management and use.A total of 86 articles were included in the final review.

Review articles and review process
Among the 86 articles included in the review, 73 were journal articles (85%) and 13 were conference proceedings or reports.We used Mendeley to manage review article selection and publication metadata.All review articles were analyzed in two phases.During the first phase, NVivo was used for exploratory analysis to capture the RPP context, theoretical perspectives, data management processes in terms of data collection, database establishment, data analysis and interpretation, and the data use, including internal dissemination, external dissemination and impact of study beyond the RPP project itself.An Excel spreadsheet was extracted based on the initial exploratory analysis and used for the second level synthesis of themes to address the research questions.The second level analysis focused on 1) the RPP engagement process involving data exploration, co-design, implementation and dissemination; and 2) the strategies used and recommended by researchers and practitioners as they reflect on RPP processes specific to data analysis and use.During the second level analysis, the articles were reviewed to highlight specific engagement processes and strategies as specified by the article authors.The research team met and discussed the coding categories based on RPP literature.Twenty articles were then randomly selected to be coded independently by researchers to ensure inter-coder reliability.Each article was reviewed by at least two researchers, and the team reached over 95% agreement based on independent coding.The research team then met and resolved any disagreements.Table 4 includes the general coding categories used in the second level analysis.

RPP data management and use engagement process
When RPPs are carried out in educational settings, key stakeholders generally engage in the iterative process of 1) initial data exploration to establish the focal problems of practice; 2) codesign of interventions/innovations and the accompanying research protocol and measures in educational contexts; 3) implementation of the intervention/innovation while studying its effectiveness; and 4) production and dissemination of research findings to contribute to new understanding (see Fig. 2).In this section, we describe how the different RPP structures engage in this iterative process.

Data exploration
As a core characteristic of all RPPs, the identification of problems of practice involves the collaboration of both researchers and practitioners through initial data exploration.Data at this stage may include evidence-based research, existing empirical data collected by the districts or community partners, and additional data gathered for the purpose of the specific RPPs.
Research alliances tend to have the opportunity to gather or explore large-scale existing empirical data from the districts or community partners.Among the 16 studies that reported RPPs as research alliances, 10 studies discussed the use of large-scale databases.The Youth Data Archive (YDA) project managed by the John W. Gardner Center, for example, "combines administrative records matched at the individual student level from a variety of public and nonprofit institutions to enable local policy makers to look across institutional boundaries" (McLaughlin & London, 2013, p. 198).Linking the variety of databases with data regarding youth allowed researchers to prepare initial data reports to share with stakeholders to generate research questions or identify the problems of practice for further examination (Biag, 2017;McLaughlin & London, 2013;Russell et al., 2013).Similarly, other research alliances such as the research alliance for New York City Schools (Coburn, Penuel, & Geil, 2013c), Baltimore Education Research Consortium (BERC), Shared Solution, the Stanford University and San Francisco Unified School District (SFUSD) partnership, and the various Regional Education Laboratory research alliances also have data agreements with various stakeholders and use databases to archive and link data to explore and identify focused problems of practice (Bryant et al., 2016;Scher et al., 2018;Scholz, Ehrlich, & Roth, 2017;Wolford, Desimone, Hill, Reitano, & Kowalski, 2016).
While research alliances tend to work with relatively large-scale data sets, design partnerships range in scale.Some design partnerships involve university researchers, districts, and community partners just like many research alliances (e.g., McGee & Nutakki, 2017;Penuel, 2014;Penuel, Allen, Coburn, & Farrell, 2015), some partnerships are situated in one particular community/school context (e.g., Pollock, 2013;Rigby, Forman, Fox, & Kazemi, 2018), and others may be focused on one particular instructional setting (e.g., Jung & Brady, 2016).All 46 studies including design partnerships reported the use of data from local contexts, and 36 studies relied solely on data from the local partnership.With most of the design partnerships focusing on instructional intervention in educational contexts, the initial data exploration may involve feedback from district administrators and teachers regarding specific interventions.The Strategic Education Research Partnership (SERP), for example, implemented Word Generation and AlgebraByExample across multiple district sites and included district leaders, research leaders and SERP staff in the implementation and feedback process (Donovan, Snow, & Daro, 2013).The Collaborative Strategic Reading (CSR) Colorado project is another example of a design partnership that focused on an instructional intervention (Borko & Klingner, 2013).The project team designed differentiated professional development for participating teachers and collected ongoing data on participants' responses to the PD prior to the school-based CSR implementation.
In addition to data regarding specific interventions, the initial data exploration may also include student data, teacher surveys, interviews, and classroom observations.Jung and Brady (2016), for example, followed the multi-tiered teaching experiment model where teacher acts as both an investigator and participant and the researcher as both a teacher and learner (Lesh & Kelly, 2000) to start their collaboration based on the exploration of student data.The Middle-school Mathematics and the Institutional Setting of Teaching (MIST) project is another example where the initial step of the project focused on exploration of existing instructional practices rather than specific interventions.The annual data collection, analysis, and feedback cycle started with interviews of approximately 10 leaders from each of the four partnering school districts that were selected based on project criteria (Cobb et al., 2013;Penuel et al., 2015;Rosenquist, Henrick, & Smith, 2015).The research team then analyzed the interviews and prepared District Design Documents where the districts' goals and implementation strategies are summarized (Cobb et al., 2013;Rosenquist et al., 2015).Penuel (2014) used MIST as a sample project to illustrate the application of cultural-historical activity theory (CHAT) and pointed out that the researchers' roles "became more facilitative, like the role of researchers in a Change Laboratory" (p.108).
NICs share many similar features with design partnerships in terms of data exploration.Whether the initial data are retrieved from existing databases, collected in the local contexts, or explored across various contexts, successful RPPs situate the problems of practice within the local historical and cultural contexts.While considering the ideal alternatives to the current challenges based on research, practical considerations are also critical in the initiation of the partnership.

Co-design
Co-design is "a highly-facilitated, team-based process in which teachers, researchers, and developers work together in defined roles to design an educational innovation, realize the design in one or more prototypes, and evaluate each prototype's significance for addressing a concrete educational need" (Penuel, Roschelle, & Shechtman, 2007, p. 51).With the focus on the problems of practice, researchers and practitioners engage in the collaborative design of both the instructional innovations and the research protocol to evaluate the effectiveness of the intervention.
Educational intervention and innovation co-design are the core of design partnerships.The majority of studies that reported engaging in design partnerships described the co-design process (43 out of 46 studies, or 93%).Curriculum interventions typically involve researchers and practitioners co-designing and adapting throughout the implementation process (e.g., Debarger et al., 2017;Jung & Brady, 2016).After the development of a curriculum intervention through the co-design process, researchers and practitioners still need to engage in a collaborative process to contextualize the design when expanding the implementation sites.The SERP institute, for example, uses three structures to ensure co-design: the core group, the ideas team, and the research, development, and implementation (RDI) team.The core group includes SERP staff who offer guidance and support; the ideas team includes researchers and local practitioners who design implementations given the local constraints and resources; and the RDI team executes the design and documents the practices using various data collection tools and protocols.Design partnerships that promote social justice and equity offer additional insights to the co-design process.Using a social design experimentation approach, Gutiérrez and Jurow (2016) reported on the Migrant Student Leadership Institute (MSLI), a month-long residential summer program designed for youth from various migrant regions across California.Through MSLI, youths are encouraged to engage in hybrid language practices, reframe their lived stories through critical testimonios, and participate in expansive forms of literacy learning.As the authors noted, equity in this design process involved seeing ethnographically to attend to the mundane everyday practices of youth so that they could be leveraged toward new ends.It involved seeing ingenuity in students' practices and strategies for negotiating oppression, poverty, environmental racism, and the microaggressions they experienced in everyday life.(p.583).
The researchers were in "constant design and redesign mode" through their interactions with youth to ensure "equality for all participants in ways where they could see themselves learning and growing" (p.584).Engaging participants in the research co-design process, design-based partnerships may adopt a participatory design-based research approach (Bang & Vossoughi, 2016) to engage participants in intervention and research design (Ishimaru & Takahashi, 2017).
Building upon existing data sources, most research alliances (12 out of 16, or 75%) reported engaging researchers and practitioners in the co-design and validation of research protocol and data collection tools (e.g., Baharav & Newman, 2019;Coburn et al., 2013a;Tseng et al., 2017;Wolford et al., 2016) Gomez, Russel, & Bryk, 2013).The driver diagram provides a visual display that highlights the teams' shared aim as well as the primary and secondary drivers to achieve the aim.The driver diagram not only clarifies the shared aim and necessary strategies but also allows the team to test the relationship between the primary and secondary drivers.Accordingly, measures are also co-designed to document the improvement process and impact.Distinct measures for accountability, improvement, and research guide the co-design process not only between researchers and practitioners, but also among practitioners across various sites (Bryk et al., 2015;Heinrich & Good, 2018).In addition to the tools provided by the Carnegie Foundation, the PDSA cycle adopted by NIC stakeholders also engaged researchers and practitioners in the "difficult process of creating instruments and data-collection protocols that struck a balance between empirical rigor and the resource, time, and personnel constraints that practitioners faced in enacting them" (Cannata et al., 2017, p. 582).

Implementation
During the implementation process, the innovations/interventions are implemented based on the design and, at the same time, the designed research protocol is also carried out to gather additional research data to inform practice.During this process, practitioners may be actively engaged in the data production and interpretation and work alongside researchers to carry out research design and analysis.
Almost two-thirds of the RPP studies included in this review (56 out of 86, or 65%) explicitly described how practitioners are data users as well as data producers.As part of the process of the YDA research alliance project, for example, researchers sought input from various stakeholders to establish connected databases and create Internal Factsheets to be shared with district leaders for input.Actionable research questions were generated by practitioners based on the data and then an Issue Brief was distributed internally and externally together with a Snapshot summary to support further discussions and interpretations based on data (Biag, 2017;Coburn et al., 2013c;McLaughlin & London, 2013).Similarly, in design-partnership work, practitioners also engaged in the data collection and interpretation process.Practitioners participated in surveys, interviews, focus groups, and followed specific protocols co-designed with the researchers to document design implementation.Borko and Klingner (2013) reported the professional development and implementation focusing on the Collaborative Strategic Reading (CSR) approach in local schools, where practitioners completed surveys, interviews, reports of activities, coaching logs, implementation logs, and an implementation validity checklist.In addition, researchers conducted classroom observations and used student outcome data.Findings based on the data led to changes and improvements to the professional development provided.
While practitioners mainly served as data producers in this type of design-partnership, the feedback cycle for improvement was critical.
The MIST project's annual data collection, analysis and feedback cycle mentioned earlier provides another example (Cobb et al., 2013;Penuel et al., 2015;Rosenquist et al., 2015).Involving multiple implementation sites, NIC stakeholders used co-designed protocols across sites for data collection and specified additional data sources based on existing educational databases.Heinrich and Good (2018), for example, reported a mixed methods study utilizing qualitative data including observations, semi-structured interviews, focus groups, and documents, together with quantitative data collected from available student records across six school district partners over five years.Practitioners across multiple sites engaged in data production and interpretation throughout the ongoing research process.
RPPs also offer opportunities for researchers and practitioners to engage in collaborative research that transitions practitioners from being not only the impetus and consumer of the research, but also the producers of research (Van Zoest, 2006).Research alliances engaged researchers and practitioners in collective capacity building to sustain the research process (McLaughlin & London, 2013).Baharav and Newman (2019), for example, reported the formation of the Post-Secondary Strengthening Collaborative (PSSC) that enhances students' college readiness and completion rates.Through the engagement in professional learning opportunities and collaborative design and use of research tools and protocols, school partners "build their capacity to conduct start-to-finish inquiry" (p.247).Similarly, through NICs, researchers and practitioners may use shared research protocols and tools to lead investigations in their local contexts and generate "practice-based knowledge" (Thompson et al., 2019, p. 3).
Design-partnerships engaged researchers and practitioners as collaborative partners in the research implementation process.Participatory research, for example, was used as a methodology to empower participants as researchers.Kirshner and Polman (2013) reported two projects where they engaged students as researchers through youth participatory research.
Similarly, Jung and Brady (2016) described their mathematics professional development design in middle school classrooms where "the teacher acts as an investigator (and participant), while the researcher also acts as a teacher/learner (and investigator)" (p.282).NIC stakeholders who followed the PDSA cycle tended to engage researchers and practitioners through research team capacity building.Empowering practitioners to serve as researchers in local contexts is one of the key strategies for scaling up the interventions and research.Barron et al. (2015), for example, reported their implementation of interventions to enhance students' growth mindset and motivation in learning.While researchers were engaged in the early design and analysis, teachers took on the researchers' role and embraced the PDSA cycle through their own classroom implementations.2017) found that "RPP district leaders reported less frequent symbolic use of research and more frequent process use.Within the RPP sample, practitioners in research roles were significantly more likely to report higher levels of process use of research than their peers in non-research roles" (p.4).Similarly, the studies included in this review also revealed that researchers and practitioners used research findings to inform or validate decision making in local settings (instrumental and symbolic use) and to guide classroom, school, or district educational practices (70 out of 86, or 81%).In addition, research findings also informed the continued engagement in RPPs through refined research protocols and research tools as discussed in previous sections (see Fig. 2).

Summary
The three general types of RPP share common characteristics and features in data management and use.Based on the examination of the studies included in this review, we also noted some unique characteristics that bring strengths to the specific design of RPPs.Table 5 provides a summary that highlights key strengths for each type of RPP.PDSA cycle for capacity building (e.g., Barron et al., 2015) Dissemination Internal dissemination to support implementation improvement and refine research process (e.g., Heinrich & Good, 2018;Jesson & Spratt, 2017;Rosenquist et al., 2015) External dissemination to share RPP findings and RPP processes (e.g., publications and project websites)

RPP data management and use strategies
Building upon the description of the RPP stakeholder engagement process, in this section, we highlight three key considerations regarding strategies adopted by a variety of RPP projects for data management and use including: 1) considering time, space, and cultural contexts of research and practice; 2) establishing routines for communication, data collection, analysis, interpretation, and dissemination; and 3) seeking mutual appropriation to cross researcher-practitioner boundaries.These three key considerations were employed across RPP studies included in this review to address specific challenges for data management and use.

Consideration of contexts
It is critical to take educational contexts into consideration when initiating and developing RPPs.The authors of most RPP studies included in this review emphasized the importance of researchers becoming familiar with the local contexts through the RPP engagement process (76 out of 86, or 88%).Staff involved in the RPP project led by the Gardner Center with Redwood City, for example, shared that: "If it's a real partnership, the researchers have to be hanging out in the community.Then, there's a real opportunity for true partnership" (Coburn, Penuel, & Geil, 2013b, p. 4).Researchers and practitioners engaged in these partnership efforts also emphasized the importance of building upon existing partnerships (e.g., Hindman et al., 2015) and starting with small-scale activities that respond to the motivation and actions of the local classrooms, districts, and communities (e.g., Heinrich & Good, 2018;Jung & Brady, 2016;Pollock, 2013).
Even with relatively large interventions, such as those implemented through SERP (Donovan et al., 2013), the importance of building local infrastructure to take district context into consideration when scaling up intervention was emphasized.
The data exploration stage for all RPPs was a process wherein local culture, historical context, practical considerations, and ideal alternatives as solutions -including issues of timing and space -were taken into consideration.In RPPs that involved data archives (e.g., McLaughlin & London, 2013;Russell et al., 2013), stakeholders explored the local context to focus on problems of practice through initial data exploration and discussion with community stakeholders.It is worth noting that several articles included in this review specifically referred to the use of sociocultural theories such as CHAT as researchers and practitioners integrate the cultural and historical contexts in RPP design (Gutiérrez & Jurow, 2016;Ishimaru & Takahashi, 2014;Penuel, 2014;Severance, Penuel, Sumner, & Leary, 2016).Social design experiments (Gutiérrez & Jurow, 2016), formative interventions (Penuel, 2014), and participatory designbased research approaches (Ishimaru and Takahashi, 2014) were used to guide the data management and use through the RPP process.All these approaches positioned members of the community as the center of the initiation, design and implementation process while the external researchers served as facilitators to support the expansive learning process for the development of transformative agency in the wider community.When collectively designed by those embedded in the cultural and historical context and engaged in the analysis of cultural and historical contradictions, the projects achieved generative impact in the community.Penuel (2014) also addressed Engeström's critique of design-based research and provided two examples of emerging forms of formative intervention research, including the MIST project.Additionally, informed by CHAT but different from the formative intervention approach, Severance et al. ( 2016) reported the curricula co-design process for teacher agency development.They started with the presentation of an external framework for science education as the first stimulus, organized teachers in small groups to leverage distributed expertise, and used the storyline tool as a second stimulus to support teacher agency in the development of units that reflect the framework.Regardless of the tools and approaches used, contextual interpretation by community members was critical in the RPP process.

Establishment of routines
Establishing shared routines and protocols for data management and use was another key consideration elaborated in the RPP studies (50 out of 86, or 58%).For large quantitative data access and management, RPP stakeholders developed data sharing agreements to provide access and connect databases (e.g., Bryant et al., 2016).Collaborative development of survey instruments and qualitative data collection tools involving both researchers and practitioners were also reflected in the process of RPPs (e.g., Cannata et al., 2017;Cohen-Vogel et al., 2016;Davidson et al., 2018;Falk et al., 2016;Fonger, 2015;Heinrich & Good, 2018;Rosenquist et al., 2015;Wang et al., 2017a, Wang et al., 2017b;Wentworth, Mazzeo, & Connolly, 2017).
More importantly, protocols and reporting formats were created to routinely share immediate data analysis results among researcher and practitioner teams.For example, the Gardner Center's YDA project included various internal and external reporting formats such as Factsheet, Data Brief, and Snapshot (Biag, 2017).Carnegie Foundation's projects followed a 90-day cycle protocol to engage in data discussions (Dolle et al., 2013).The NCSU RPP involved data sharing following the PDSA cycle among the District Innovation Design Team (DIDTs) and School Innovation Design Teams (SIDTs) composed of administrators, classroom practitioners, and researchers.The establishment of communication routines such as retreats, webinars, PLC discussions also supported ongoing engagement of all stakeholders in the RPP process.As Dolle et al. ( 2013) pointed out, "when done well, common tools and routines make work more standard, promoting the implementation of core practices with integrity-that is, in a manner that remains true to its essential ideas and guiding principles-while remaining responsive to local conditions and context" (p.455).Regardless of the scale, routines for data collection, analysis, ongoing communication, immediate dissemination and reporting were critical to support stakeholder engagement.These routines took time to establish and were based on mutual understanding among researchers and practitioners within the educational contexts.Most of the studies that included clear discussions regarding established routines and protocols were longterm RPPs that took years to build and sustain.

Mutual appropriation
Finally, the process of appropriation where "individuals and groups are encouraged to adapt practices to their particular conditions and capabilities" (Scherrer, Israel, & Resnick, 2013, p. 434) is highlighted in the RPP studies (32 out of 86, or 37%).As Penuel, Coburn, & Gallagher (2013) argued, researchers and practitioners negotiated frame alignment and resonance based on their different perspectives and experiences: Frame resonance requires that participants develop ways of framing problems and solutions that communicate across these different worlds so that the social problems identified and the solutions proposed by teams "make sense" to one another.(p.244) The RPP engagement process involves appropriation where researchers gain new skills to state the problems of practice from practitioners' perspectives, communicate research findings in multiple formats with diverse audiences, and envision research agendas with local community members for actions in local contexts (Tseng et al., 2017).At the same time, practitioners engage in the research process where peripheral participation is legitimized (Dolle et al., 2013), flexible researcher roles can be assumed (Fonger, 2015), and participants are positioned as researchers given their knowledge of the historical and cultural context (e.g., Gutiérrez & Jurow, 2016).
Research and practice capacity building becomes the ultimate goal of RPP engagement.
As Davidson et al. (2018) summarized, RPPs help with knowledge-transfer among experts, which leads to new insights for all partners.In many RPPs, the mutual appropriation, intentional capacity building, and "the process of cross-agency collaboration toward a goal of data sharing and querying is itself an important product" (McLaughlin & London, 2013, p. 208).

Discussion
In the field of education, the use and analysis of RPPs as long-term, problem-solving collaborations between researchers and practitioners has grown rapidly in recent years.In this review, we highlighted the RPP engagement process and strategies RPP teams used to address challenges regarding data management and use in RPPs.We undertook a narrative review of the literature to synthesize and gain a greater understanding of stakeholders' data management and use within and across various RPP structures.In this section, we highlight promising strategies for data management and use as RPP stakeholders initiate, implement, and sustain RPPs and offer implications for RPP teams within and beyond U.S. educational contexts.

Initiating RPPs
RPP stakeholders need to consider 1) thoroughly exploring the cultural and historical contexts, 2) establishing clear data access and use agreements, and 3) specifying mutual appropriation and capacity building as key outcomes of RPPs.
Second, establishing data sharing and data use agreements among stakeholders is essential during the initiation of RPPs to leverage the use of existing data and take historical and cultural contexts into consideration before initiating any additional data collection.In the educational context, large-scale databases exist at international, national, and regional levels.With the advancement in technology, linking across multiple databases has proven to be plausible and useful to enhance our understanding of educational practices and student development (e.g., Bryant et al., 2016).Making explicit the intention for data use and ensuring equitable interpretation and dissemination of the data also needs to be considered at the initiation of the RPP to ensure trust and mutualism among stakeholders.
Third, RPP stakeholders need to embrace capacity building as one of the key outcomes at the initiation of RPPs rather than an after-thought.Putting mutual appropriation and capacity building at the center of the RPP outcomes alongside the specific problems of practice that may have triggered the initiation of RPPs further supports the full engagement of all stakeholders in the RPP process through the RPP implementation.Collaborative efforts to recognize those with knowledge of the historical and cultural context as key researchers (e.g., Gutiérrez & Jurow, 2016), legitimize peripheral research participation (Dolle et al., 2013), and allow for flexible researcher roles (Fonger, 2015) should be made at the initiation of RPPs.

Implementing RPPs
During implementation, RPP teams need to engage in contextualized protocol and instrument development and ongoing communications and internal dissemination for program improvement and protocol/instrument refinement.Data use and management also needs to be strategically situated within the micropolitical context.
Many RPPs included in this review highlighted the importance of co-design.In addition to the co-design of intervention and innovation as commonly implemented in design-based research (e.g., Debarger et al., 2017;Jung & Brady, 2016), it is also important for RPP stakeholders to engage in the co-design of research protocols and data collection instruments as reflected in research alliance RPPs (e.g., Wolford et al., 2016) and process-based scale-up implementations as conducted in NICs (e.g., Bryk et al., 2015;Heinrich & Good, 2018).The process of co-design ensures contextualized RPP implementation and supports ultimate capacity building beyond the RPP team.
Establishing routines for communication and dissemination is essential for successful RPPs.Communication routines should be embedded in the RPP implementation rather than focusing only on disseminating formative or summative outcomes of research (Durham et al., 2015).All practitioners, just as their researcher counterparts, need to be included in key research decision making, such as the selection of research protocols, assessment methods, and the interpretation of research findings (Burk, Dees, & Kalbaugh, 2018).Data sharing protocol and reporting formats shared by successful RPPs (e.g., Biag, 2017;Dolle et al., 2013) offer good examples and templates to follow.
When engaging in internal dissemination and communication, it is critical that the RPP stakeholders are mindful of sensitive micropolitical issues that may arise.Brosky (2011) defined the micropolitics in education as "human behavior, power, and how people use it to influence others and to protect themselves, and how people compete with each other to get what they want" (p.2).In addition to equitable engagement in these discussions, RPP stakeholders also need to consider micropolitical pressures, including but not limited to district or university politics, power differentials, and researchers' and/or practitioners' one-sided interests or ambitions --all of which influence data management and use within RPPs.

Sustaining RPPs
Sustaining RPPs requires RPP stakeholders to learn to become brokers of boundaries, and to share thick descriptions of the inner workings of their engagement within the cultural and historical contexts of the RPPs.Brokers --or boundary spanners --are crucial to developing and maintaining trust during data management and use within and across various RPP structures.Brokers work across partner institutions to link partners; promote exchange of ideas, information, and resources; and support capacity development and information management (Paré-Blagoev & Booth, 2018).When members of the community are positioned at the center of project initiation, co-design, and implementation processes, external researchers may have opportunities to engage in the expansive learning process to further promote the development of transformative agency in the wider community.
Further, this review also highlighted a need for RPP stakeholders to include thick, rich descriptions when documenting RPP data management and use processes.Explaining the inner workings of how they negotiated power differentials to promote and assess social justice and equity throughout the iterative data management and use process would offer insights to other RPP stakeholders as they engage in similar processes.Providing thick, rich description in more abundant and nuanced ways also allows for greater alignment with the extensive reflexivity and fieldwork that is required when engaging ethnographically with legitimacy and authority (Clifford, 1983;Emerson, Fretz, & Shaw, 2011;Ferguson & Gupta, 1997;Geertz, 1983;Jackson, 2010;LeCompte & Schensul, 2010).

Limitations
Rather than providing an exhaustive, comprehensive review of RPPs, we focused this review on synthesizing data management and use within and across various RPP structures among selected review studies.Although we used systematic search strategies and explicitly articulated our inclusion and exclusion criteria, the review process has limitations.
Over the last two decades, a growing number of researchers have engaged in both RPP studies and studies of RPP functioning.Many long-standing RPPs (e.g., Carnegie Foundation for the Advancement of Teaching) and funders supporting RPP initiatives (e.g., Institute of Education Sciences, William T. Grant Foundation) share white papers and RPP reports on their websites.
While these reports reflect the process and product of various RPPs, in this review, we only included publications in which the researchers specifically addressed data management and use.
Further, we acknowledge there are search biases, such as location bias and multiple publication bias inherent in the review article identification process.As described in the Research Method section (see 3.1), we focused this review on RPP scholarship within the United States.In addition to this geographical limitation, we only included published studies in this review.
Although we have attempted to decrease publication bias by including peer reviewed and nonpeer reviewed publications, we acknowledge that we did not include unpublished research on this topic.We also noted that some RPP teams have produced multiple publications that were included in this review.For example, the Middle-school Mathematics and the Institutional Setting of Teaching (MIST) project yielded multiple articles in which the researchers either addressed data management and use issues (e.g., Cobb et al., 2013) or included examples in the studies of RPPs to illustrate the data management and use process (e.g., Rosenquist et al., 2015).We included multiple publications regarding the same project as they provided additional insights into the data management and use process of these projects.The inclusion of multiple publications resulting from the same project would inflate the findings and interpretations if we were to only rely on quantified themes and trends.In this review, therefore, we used projects and publications as examples to identify and substantiate the themes in response to the literature review questions.
Finally, the publication time limit (i.e., 2013-2019) reflects the publication date range inherent in our search procedure rather than the researchers' RPP project timeline.Due to the publication time frame, most journal articles included in this review were RPP studies conducted prior to 2013.These limitations notwithstanding, our findings provide a foundational overview of the iterative data management and use processes in RPPs.

Future research and conclusion
This review is responsive to Coburn and Penuel's (2016) call in that the findings of our review describe how RPP stakeholders managed and used data within and across RPP structures.Specifically, we summarized the RPP engagement process characterized by data exploration, codesign, implementation, dissemination, and use across three general types of RPP structures; highlighted key strategies RPP stakeholders have used to address challenges in terms of data management and use including considerations regarding time, space and cultural contexts, routines for RPP processes, and mutual appropriation for boundary crossing; and discussed recommended approaches for RPP stakeholders as they initiate, implement, and sustain various RPPs.
Through our review, we also revealed several key directions for future research as RPP stakeholders consider external dissemination beyond the specific results and outcomes of the RPP interventions or innovations.First, while equity may be at the center of the problems of practice for many RPPs, few researchers explicitly addressed how equity was established in the iterative data management and use process.There may be assumptions built into the notion of "partnership" that preclude direct discussions of how equity is defined and how RPP stakeholders intentionally design and tend to equity at various levels (e.g., whose voices contribute to data, who has decision making power over data).If RPP stakeholders are to ensure equity is achieved, a need exists for explicit, thick, rich discussions of critical reflexivity in relation to data management and use.Understanding how equity is intentionally negotiated within and across various types of RPPs for data management and use will also offer RPP stakeholders further insights to refine the iterative process and to promote social justice in research and educational practices.
Moreover, our findings also revealed a critical need regarding more in-depth analysis of the inner workings that occur during the iterative RPP process.As such, we echo Coburn and Penuel's (2016) call for more researchers to investigate the dynamics experienced and the outcomes achieved by successful and unsuccessful RPPs.It is incumbent upon current and future RPP teams to identify and describe obstacles regarding data management and use so that RPP researchers and practitioners are more readily able to anticipate and recognize them throughout the iterative process.More importantly, contextualized description of the strategies stakeholders used to overcome these obstacles and to negotiate issues, such as racialization and micropolitical pressures, could offer much needed guidance to more RPP stakeholders engaged in collective reflections aimed at improving the RPP process.
Finally, we focused this review on studies featuring RPPs in U.S. educational context.Future research is needed regarding the unique cultural contexts of RPP data management and use when implemented in other countries.As such, a need exists for conducting more in-depth quantitative, qualitative, and mixed methods meta-analysis of RPP studies across structures and geographic locations beyond a few isolated initiatives.Engaging in crowdsourcing initiatives, such as the Psychological Science Accelerator (https://psysciacc.org/),which serves as a globally distributed research network, may offer pathways for RPP stakeholders to join together through networked research communities in hastening the production and application of evidence about data management and use within and across various RPP structures.Creating a more inclusive, just society depends, in part, on education research that is designed, carried out, and interpreted on the basis of more equitable participation among researchers, practitioners, and community members.RPPs offer one way to do just that, yet much more needs to be investigated, regarding the "hows" inherent in successful RPPs.
In closing, we note that boundary crossing collaborations in RPPs offered an opportunity for us to prioritize and reflect on equity issues inherent in educational research and practice through bridging the oft disparate worlds of researchers and practitioners (Penuel et al., 2015).The identification of theoretical orientations (e.g., CHAT) and specific approaches provided insights as researchers and practitioners continue to explore boundary spanning strategies to enact successful and powerful "long-term, mutualistic collaborations between practitioners and researchers that are intentionally organized to investigate problems of practice and solutions for improving district outcomes" (Coburn et al., 2013a, p. 2).The human and cultural/contextual possibilities of such work, while making data use and management more complicated, simultaneously showcase the infinite potential of partnerships.

Table 1 .
Major types of RPPs.

Table 2 .
Challenges for RPP data management and use.

Table 3 .
Inclusion and exclusion criteria.
" In addition, we searched leading educational journals in the United States such as Educational Researcher, American Educational Research Journal, Journal of the Learning Sciences, Journal of Educational Psychology, Theory into Practice, Contemporary Educational Psychology, Cognition and Instruction, and Review of Educational Research.The reference lists of identified articles were examined to identify additional studies.
NICs are unique in that participating schools, districts, or communities not only learn from data based on the local contexts, but also develop collective knowledge across contexts(Cannata, Cohen- Vogel, & Sorum, 2017; LeMahieu et al., 2017).Twelve studies featuring NICs explicitly commented on data exploration across contexts.The National Center on Scaling Up Effective Schools (NCSU) is an example of a NIC that evolved from design-based partnership work started in 2007(Cannata et al., 2017).Cohen-Vogel et al. (2016)described the NCSU RPP as involving four phases: research, innovation design and development, implementation, and scaling-up.The initial research phase reflects the collaborative data exploration where researchers and practitioners collected and analyzed data to identify "programs and practices that Barron, Hulleman, Inouye, & Hartka, 2015;rformance" (p.7).Surveys and week-long school visits served as data sources during initial data exploration.With the PDSA cycle as a signature form of many NICs, researchers have also reported use of a small-scale pilot with multiple data sources collected throughout the pilot phase to initiate the data sharing cycle across partnering institutions (e.g.,Barron, Hulleman, Inouye, & Hartka, 2015; Wang et al., 2017a, Wang et al.,  2017b).
. Stakeholders engaged in the research alliance at the New York University Steinhardt School of Culture, Education and Human Development, for example, worked with New York City Schools to develop and validate a revised teacher survey instrument to examine school environments(Coburn et al., 2013c).The researchers affiliated with the research alliance provided support for reliability and validity studies and convened measurement expert panels, while administrators and practitioners in the district offered content expertise.Similarly, Shared Solution, a "hybrid of the place-based research alliances and design research teams" between the University of Pennsylvania's Graduate School of Education and the School District of Philadelphia(Wolford et al., 2016, p. 1)formed working groups of researchers and practitioners to develop and pilot instruments used in implementation studies in partner districts alongside the implementation of interventions.
research to influence one's perceptions of the problems of practice or the potential solutions, 3) symbolic or political use of research to validate or legitimate decisions made, and 4) process use of research to integrate the research process into daily practices.Based on their review, Farrell et al. ( The intentional building of research capacity through training practitioners on the use of specific research protocols and tools was integrated in many NICs (e.g.,Cannata et al.,  2017; Coburn, Penuel, & Geil, 2013d; Wilcox, Lawson, & Angelis, 2017).practices in general.As included in this paper, RPP researchers publish findings through journal articles and conference proceedings.Many large-scale, long-term RPP researchers also have established websites through which they share research findings and protocols, such as that for

Table 5 .
Features of data management and use in RPPs.
The key to the success of RPPs lies in the contextualization of research problem identification, intervention design, and research implementation.Instead of focusing only on the transfer and adaptation of existing theory-driven interventions and protocols, initiating RPPs through exploration of localized norms and routines embedded in social and cultural contexts allows RPP stakeholders to engage in role appropriation and boundary crossing at the initiation stage.Approaches including social design experiments(Gutiérrez & Jurow, 2016), formative interventions(Penuel, 2014), and participatory designbased research approaches(Ishimaru & Takahashi, 2017) can be considered.In addition, in contexts where the racialization of educational and research practices may be embedded(Omi & Conceptualization, Data curation, Writing -original draft, Writing -review & editing.Kimberly Kappler Hewitt: Conceptualization, Data curation, Writing -review & editing.Marcia L. Rock: Conceptualization, Data curation, Writing -original draft, Writingreview & editing.Sophia Rodriguez: Conceptualization, Writing -review & editing.Laura M. Gonzalez: Data curation, Writing -review & editing.Amy Vetter: Conceptualization, Project administration, Data curation, Writing -review & editing.