Integrating Mixed Methods Approaches in Counseling Outcome Research

Abstract Mixed methods research has the ability to help explore the complex contexts and experiences of individuals within counseling, which in turn will enhance our understanding of evidence-based treatments. Mixed methods research can answer the what, when, and how questions that are important in implementing effective interventions. Yet, given the complexity of mixed methods research, it can easily be implemented incorrectly, limiting its application. Integral steps to conducting mixed methods research include determining a theoretical lens, selecting methodologies, and determining integration procedures in sampling strategies, data analysis, or interpretation of results. Using mixed methods research to better understand the impact mental health professionals have with the clients and students they serve can surpass using quantitative or qualitative methodologies in isolation.

Plano Clark, & Smith, 2011). This report proclaimed the need for methodological diversity given the disparities among populations, ethnicities, and cultures in health, disabilities, and treatments. The team highlighted best practices for engaging in mixedmethod approaches to understanding complex health problems, as well as the context of the individual, culture, and society on health problems and subsequent treatment. Although the committee generally focused on health problems, mental health needs and treatments are also complex and socioculturally embedded, and MMR is a viable means to develop our understanding of mental health problems and treatment outcomes. Ultimately, it addresses the need for methodological pluralism to better understand the complexity with which our clinical work affected individuals (Lenz, 2018). Although mixed methods can answer many questions related to mental health outcomes, the methodology could be misunderstood and carried out incorrectly (Bryman, 2006). Therefore, this article provides an overview and rationale for MMR, as well as a review of various typologies with consideration for conducting quality studies.

What Is Mixed Methods Research?
MMR refers to the combination of quantitative and qualitative research methodology within the same study. Johnson and Onwuegbuzi (2004) suggested that MMR "combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth and depth of understanding and corroboration" (p. 123). Combining both qualitative and quantitative methodologies helps to explore a research question from more than one angle, allowing the ability to integrate the information for a fuller understanding. Fetters (2018) stressed using quantitative and qualitative methodologies in the same study without integrating the results is no more than using the independent methods alone. Rather, researchers must truly combine quantitative and qualitative methodologies through intentional integration to conduct MMR that "yields a whole greater than the sum of the individual parts" (Fetters, 2018, p. 263). MMR should be used as a way to add value, or to gain a better understanding of a phenomenon of interest, thereby ultimately gaining a more holistic picture of a research question.
This approach is important in mental health research given that presenting concerns by clients originate from complex situations and experiences. Thus, it is imperative to have quantitative methodology to inform the researcher of the factor(s) that explains the most variance or the intervention that is most effective. Likewise, a qualitative approach is important to provide depth of understanding of the client's experience, along with aspects the researcher might not have considered or have the ability to formally measure. Historically, researchers have debated the feasibility of integration because quantitative and qualitative methodologies typically arise out of different research paradigms (Bryman, 2007;Morgan 2007). These differing paradigms have resulted in philosophical and paradigmatic debates about a new paradigm or methodology that stands on its own-one that combines quantitative and qualitative methodology in a systematic way that it results in its own, separate, methodology (Fetters, 2018).

Why Mixed Methods Research?
A researcher would select MMR based on his or her research question(s) (Hanson, Creswell, Plano Clark, Petska, & Creswell, 2005;Tashakkori & Teddlie, 2003). Specifically, researchers might evaluate whether they can answer their research question(s) with one methodology, or a combination of methodologies to gain a complete picture of the phenomenon of interest. Consider the purposes of qualitative and quantitative methodologies. First, qualitative is inductive in its approach, thus grounding the emergent theories or knowledge in the data. The goal with qualitative methodology is depth of understanding, to explain the phenomenon or topic under investigation. This occurs with a smaller, more purposeful sample. Conversely, quantitative methodology is grounded in deductive reasoning, meaning an a priori theory guides the empirical approach and the results (dis)confirm the originally proposed theory. Quantitative methodologies are used to gain breadth of understanding, demonstrate cause and effect, or examine the degree to which one variable (e.g., anxiety) explains another (e.g., seeking mental health treatment).
Consider an example: A counselor is implementing a new intervention in his or her work with children. The counselor might set up a quantitative study (e.g., single-case research design to explore one to a few different cases [Lenz, 2015;Lenz & Callender, 2018], an experimental design with a larger sample size [Barrio Minton, 2018;Ray, 2018]). This quantitative study would answer research questions about the effectiveness of the new intervention and whether this intervention influenced changes in the outcome variable of interest (e.g., symptom reduction, daily functioning). Although this quantitative approach might help the counselor answer questions about whether the treatment was effective or the degree to which the treatment influenced the outcome, the counselor does not have a depth of understanding about why or how the intervention might have worked. Qualitative methodologies support gaining this depth of information by understanding the context of interventions with clients. By blending the findings across methodologies, the counselor gains a more complete picture of the intervention, the clients, the context, and the mechanisms of change.

Rationale for MMR
There are a few justifications for conducting MMR (Bryman, 2006;Hanson et al., 2005): Triangulation: Using two or more methods (or strands) to corroborate or converge on the findings, ultimately to either support (i.e., validate) the findings from both approaches or to gain a larger, more complete understanding. Complementarity: To elaborate, enhance, illustrate, or clarify the results of one method by using a second method. Development: Using one method to develop or inform the other method. Development can be any aspect of the second method, including sample selection, selection of methodology, or implementation of procedures. Initiation: To gain new information or to seek out contradicting information, ultimately to work in formulating new perspectives or frameworks. Expansion: To extend the breadth or range of inquiry through use of various methodologies for different inquiry components. Therefore, understanding the goal behind why the researcher is conducting MMR is important, as it could assist in how MMR is designed. At times MMR is designed a priori (i.e., fixed), whereas other times researchers might already be collecting data to answer a research question and unearth the need to gain additional information to fully answer the questions (i.e., emergent; Creswell & Plano Clark, 2018). In fixed MMR, the researcher plans for the mixing of designs from the outset, knowing that the research questions could not be answered by one method alone. On the other hand, a researcher might engage in one methodology (e.g., quantitative), and based on results, he or she might want a greater understanding of how or why, and thus the need for qualitative methodology to gain depth of understanding to gain a more complete picture of the findings emerges. Fixed or emergent MMR can occur within any of the given justifications, but should be stated in a method section. Bryman (2006) noted that 27% of researchers did not state the reason for using MMR, and when justification was noted, it did not always match in practice. Bryman hypothesized that the lack of, or inaccurately stated, rationales could be due to a lack of sufficient preplanning and thought to the MMR process.
Ultimately, researchers should use MMR when (a) a quantitative or qualitative approach is inadequate to answer a research question by itself, (b) a researcher needs multiple perspectives or forms of data to have a complete understanding about a research problem, or (c) these multiple perspectives will enhance the meaning of a singular perspective (Creswell et al., 2011). Additionally, the research question(s) might lend themselves primarily to quantitative methodologies, but when attempting to gain an understanding of marginalized populations or advocating for social change, MMR can be important to highlight voices and experiences of underrepresented groups (Chan & Henesy, 2019; Fassinger & Morrow, 2013).

Approaches to MMR
There have been many approaches and typologies of MMR noted (Creswell & Plano Clark, 2018;Hanson et al., 2005;Johnson & Onwuegbuzi, 2004); however, four overarching approaches are discussed here. These are the convergent, explanatory sequential, exploratory sequential, and embedded designs (see Figure 1), which are recognized as best practices by the National Institutes of Health (Creswell et al., 2011).

Convergent Designs
Convergent designs, also called parallel, triangulation, or concurrent designs, refer to engaging in qualitative and quantitative methodologies (or strands) simultaneously, yet independently throughout data collection, analysis, and interpretation. A convergent design can be done within the same sample or accomplished with separate samples depending on the research question and the rationale for using the strands concurrently. Researchers collect and analyze quantitative data while they are collecting and analyzing qualitative data, but the data are analyzed independently. After analysis, the researcher integrates the results by comparing and contrasting the findings to gain a more complete understanding of the phenomenon of interest.

Explanatory Sequential Designs
Sequential designs involve one strand of data building on the results from another strand. In explanatory sequential designs, researchers typically employ quantitative methodology first to answer a quantitatively oriented research question. The researcher then uses a qualitative strand to explain the quantitative findings in more depth. These designs are frequently used to learn how factors relate to each other, or whether an intervention decreased symptomology within the quantitative phase, with the qualitative phase helping to explain these results by providing depth with participants' experiences within the treatment, or their understanding of the phenomenon of interest. Explanatory sequential designs can assist in better understanding the relationships found (or not found) statistically, along with highlighting components important to individuals that might not have been assessed in the quantitative phase.

Exploratory Sequential Designs
In an exploratory sequential design, the researcher uses a qualitative methodology first to explore a situation, context, experience, or phenomenon of interest in depth. Then, the researcher uses a quantitative methodology as a follow-up to further test the information gleaned from the qualitative findings. Researchers frequently use these designs to develop instruments, as the qualitative strand can promote understanding of construct nuances that support development of instrument items. Exploratory sequential designs can also be implemented to build a new theoretical framework from individual experiences, followed by the exploration of the statistical relationships within the emergent framework.

Embedded Designs
Embedded designs can be variants of sequential or concurrent designs, but usually feature quantitative and qualitative data collection from the same participants. Embedded designs can also occur when there are nested data and different types of data occurring at different levels. An important consideration in embedded designs is to determine the weight of the type of data; more specifically, understanding whether the quantitative data being collected hold more importance or value in the interpretation of the data than the qualitative data, or vice versa. The importance, or weight, placed on the various methodologies within the embedded design is dependent on the research question and the purpose of the study.
Transformative MMR research designs are typically discussed as a distinct design (e.g., Creswell & Plano Clark, 2018;Mertens, 2010); however, transformative MMR designs can be seen as a paradigm that is applicable across the four previously discussed methodologies. The underlying principle of a transformative MMR design is that the researcher uses a theoretical model within the study, along with a methodological design, such as participatory action research, that will lead to social action and advocacy for the participants or community (Creswell & Plano Clark, 2018;Mertens, 2010). Fassinger and Morrow (2013) highlighted many ways to approach qualitative, quantitative, and MMR from a transformative, social justice perspective. They noted that researchers can do this through being culturally aware of their own impact and privilege as researchers, understanding the perspectives of the participants, being sensitive to the true needs of and the implications that research results can have on underrepresented populations, and how all of these considerations should affect the selection of the research methods and procedures used within a study, and in developing relationships with participants throughout the research process. Therefore, a transformative design is not discussed here in isolation as it can be the paradigm or lens with which a researcher uses any of the other MMR designs.

Designing MMR
To appropriately carry out an MMR design, researchers need to intentionally follow steps in the design process. Many of the steps in designing MMR are similar to those taken with solely qualitative and quantitative methodologies. In practice, researchers engaging in MMR design both a qualitative and a quantitative study and determine how the studies are integrated together meaningfully. Erroneously, some researchers develop one methodological study, supplement the original methodology with a miniscule component of another, and then deem it MMR. For example, a researcher might engage in a quantitative study using survey data with multiple instruments to determine overall relationships between constructs. To gain depth of information about a particular topic, the researcher might add one or two open-ended questions at the end of the survey, resulting in categories or themes from these latter questions. Or, a researcher might engage in semistructured interviews using a phenomenological approach, and he or she might ask each participant to complete the Beck Depression Inventory at the time of the interview to gain an understanding of the participant's level of depression. Neither of these examples truly equates MMR design (Bryman, 2006).
The way to truly consider conducting and designing MMR is to fully design both a quantitative methodology (or study) and a qualitative methodology (or study) in their entirety, and then determine the data collection procedures and the integration procedures a priori (Hanson et al., 2005). This complex process is one potential reason why researchers might infrequently engage in MMR, particularly given the feasibility of designing and carrying out two methods while determining the best strategy for integration so that data from the two strands and studies combine to be powerful and provide a broader picture of the whole phenomenon under study (Fetters, 2018).
Researchers should consider three steps when designing MMR: (a) selecting a paradigmatic or theoretical lens, (b) selecting methodologies, and (c) determining the integration procedures.
Step 1: Paradigmatic or Theoretical Lens Selecting the overall MMR approach is typically grounded in the research question(s), the theoretical lens or paradigm approach of the researcher, and the justification or rationale for the overall MMR design (Creswell et al., 2011;Hanson et al., 2005). This lens tends to shift the potential research question(s) that are answered, as well as the ways researchers interface with participants (Fassinger & Morrow, 2013). Based on the research question(s) being asked, it needs to be determined if the question can be answered using one methodology or if a combination of methodologies is needed.
Step 2: Selecting Methodologies First, the researcher needs to determine individual qualitative and quantitative methodologies. As noted earlier, simply adding one instrument or a few open-ended questions does not equate to mixing methodologies. Consider this more minutely: If a researcher conducted a quantitative study using only one measure at one point in time, this would not be a strong quantitative study. Given the specific research question, what quantitative methodology would best answer that question? Similarly, to answer a research question with a few open-ended inquiries would not provide the depth needed to explain or understand a situation or the experience of an individual or individuals. Therefore, such an approach would not be deemed robust qualitative research because qualitative research involves underlying philosophies that lend to how a study is designed from the outset (Hays & Singh, 2011). Therefore, the researcher first must fully develop and articulate both the quantitative and qualitative strands to accurately design an MMR study.
Once the researcher selects and designs the individual methodologies, he or she determines how data collection will be implemented and prioritized. In this portion of the step, the researcher needs to consider the order of the methodologies, or MMR strands (e.g., concurrent, sequential, or embedded), and the priority or weight given to the two types of data. There is no correct answer to the priority or weight provided to one type of data or methodology versus another, but the researcher should decide at the outset of designing MMR and should relate the decisions to the research questions. Will the quantitative data receive a greater priority, and thus weight, than the qualitative data, or vice versa? Will both be given equal weight? The priority given to one form of data, or strand, over another could result in which MMR approach is selected, and also influences the third step.
Step 3: Integration Procedures Integration procedures refer to when the mixing of data occurs, and might differ based on the MMR approach selected, the methodologies selected, and your research questions. When developing and implementing MMR, researchers need to think through when, why, and how they are going to integrate the two strands of their MMR to provide a comprehensive understanding of their research question(s). For a study to be truly MMR, integration needs to occur; otherwise, there is no difference than simply conducting quantitative and qualitative methodology in isolation (Fetters, 2018). Integration can occur during methodological procedures, data analysis, or interpretation of the results.
Integration during methodological procedures. Integration during methodological procedures could involve collecting data concurrently or when sampling procedures of one methodology are connected to the other. Teddlie and Yu (2007) suggested that this can occur through the blending of probability and purposive sampling strategies across any MMR design (e.g., concurrent, sequential, embedded).
Two examples of purposive sampling include stratified and random sampling. Stratified purposive sampling (Teddlie & Yu, 2007), which involves dividing the group of interest into strata (e.g., clients who are clinically depressed vs. not clinically depressed), and then selecting a small number of cases to study in greater depth within each strata based on the purposive sampling techniques. Another common MMR sampling strategy is purposive random sampling, which involves randomly sampling a small number of individuals from a larger target population that meet a specific criteria (e.g., clients who terminated treatment early). The goal in MMR sampling is to combine probability sampling from the quantitative methodology and integrate that with purposive sampling from the qualitative methodology.
Another MMR sampling procedure involves using probability and purposive sampling techniques at different levels of the study. A researcher might do this when data are nested, such as in embedded designs, and the researcher needs to use multilevel mixed methods sampling. For example, consider a researcher who is interested in how school climate affects academic achievement. This researcher might randomly select students from within a school to participate in the quantitative strand of the study, but purposefully select students at risk of failing academically to participate in the qualitative strand to gain depth on a specific population of interest. Consider another example: As an owner of a private practice, a counselor is curious about the effectiveness of dialectical behavior therapy (DBT) he or she uses with clients. The counselor randomly selects 50 former clients with whom he or she has worked during the past year, but who are no longer in counseling, to respond to a survey with various measures (e.g., posttraumatic stress, depression, anxiety, mindfulness). At the same time, the counselor asks current clients to provide subjective feedback to a few questions about what they like, do not like, and what they believe has worked for them within the sessions of DBT. Although these are examples of how data can be collected from the same or different samples, what is important to note is that data can be collected and never integrated, thus not equating MMR and not allowing the findings to merge to provide meaning greater than the qualitative and quantitative strands in isolation. It is the integration through the data analysis or the interpretation that would truly equate integration.
Integration during data analysis. There are two ways to integrate during data analysis. The more frequently discussed is merging the data or databases of the two methodologies through the transformation of one type of data into another (Creswell et al., 2011;Creswell & Plano Clark, 2018). For example, a researcher might take qualitative data and transform it into numeric data (e.g., 1 ¼ presence of theme; 0 ¼ absence of theme), and then merge the transformed data into the quantitative database to use during analyses. A researcher can also merge quantitative data with qualitative data by taking a construct or scores from a measure to make sense of the qualitative data (e.g., what are the themes of participants with clinical levels of depression vs. those who do not have clinical levels of depression). The goal of merging databases or data is to gain a more complete understanding of the phenomenon of interest.
Although merging or transforming data is more commonly discussed for integrating data analysis during MMR, another integration procedure during data analysis is to follow a thread (O'Cathain, Murphy, & Nicholl, 2010). According to O'Cathain and colleagues, following a thread begins with an initial analysis of each strand to identify the statistical results and themes that emerge. This analysis results in questions that require further exploration, so the researcher selects a question or theme from one strand, based on the initial results, and follows it across the other strand. Next, the researcher engages in a cyclical process, going back and forth from one methodology to the other, to follow the thread across both types of data toward a more complete understanding of the answer to the original question. For example, interviews completed by Adamson, Ben-Shlomo, Chaturvedi, and Donovan (2009) found that participants were self-rationing health care services. The authors explored this emergent theme in their quantitative survey data to find a possible reason for why patients might self-ration. Their initial explorations in the survey data did not offer an explanation, so Adamson et al. went back into the qualitative data to help interpret why no significant differences were found. They found many different reasons for self-rationing, including judging help-seeking of others to be inappropriate and anxiety being considered a good justification for help-seeking. Adamson et al. provided just one example of how a researcher can follow a thread and engage in an iterative or cyclical analysis process to better understand data strands.
Integration during interpretation. One of the most common methods of integrating MMR is to keep data collection and analysis separate, but to integrate during interpretation of the results. Triangulation is one common method of integration during interpretation and a justification for engaging in MMR (Bryman, 2006;O'Cathain et al., 2010). Triangulation involves analyzing data from both strands independently, regardless of MMR design. Results from each strand should be listed, compared, and contrasted on the same page to identify how results from the two methods converge, complement, or contradict each other. Full integration requires discussing findings from both strands of MMR together. Triangulation can be used regardless of MMR design or the sequence in which data were collected.

Conclusion
Despite its infrequent use in counseling outcome research, MMR is a robust methodology with the potential to offer multifaceted answers to complex needs and issues facing clients. The successful integration of quantitative and qualitative data in MMR designs can aid counselors in identifying and understanding evidence-based practices. By thinking through each step of the research process from the outset of a research endeavor, the researcher can bring together and integrate quantitative and qualitative data in such a way that the combination of the two strands is greater than the sum of its parts.