Methods for a Qualitative Systematic Review


Joann Starks: Good afternoon, everyone. I am Joann Starks of SEDL in Austin, Texas, and I will be moderating today’s webinar entitled Methods for Qualitative Evidence Synthesis. It is the second in a series of four webinars that make up an online workshop on Qualitative Research Synthesis. I want to thank my colleague, Ann Williams, for her logistical and technical support for today’s session. The webinar is offered through the Center on Knowledge Translation for Disability and Rehabilitation Research, KTDRR, which is funded by the National Institute on Disability and Rehabilitation Research.

The KTDRR is sponsoring a community of practice on evidence for disability and rehabilitation or D&R research. Evidence in the field of disability and rehabilitation often includes studies that follow a variety of qualitative research paradigms. Such evidence is difficult to summarize using traditional, systematic research review procedures.

The goal of this series of web-based workshops is to introduce D&R researchers to the methodology of qualitative evidence reviews. Participants will be provided a state-of-the-art overview on current approaches and will learn to apply those to the literature base. Ongoing, innovative initiatives at review-producing institutions will be highlighted. Today, our speaker is Michael Saini, Associate Professor at the Factor-Inwentash Faculty of Social Work, University of Toronto, and holds the Endowed Factor-Inwentash Chair of Law and Social Work.

His professional interests include research policy and practice with children and families involved with child welfare, family law, and alternative dispute programs. He is actively involved in the Cochrane collaboration and the Campbell Collaboration as an author of systematic reviews and is a member of Campbell’s Process and Implementation Methods Group. He has over 50 publications, including books, book chapters, government reports, systematic reviews, qualitative reviews and peer-reviewed journal articles. He is the co-author with Dr. Aron Shlonsky of a book entitled “Systematic Synthesis of Qualitative Research.” Welcome and thank you for agreeing to conduct the session today on Methods for Synthesizing Qualitative Evidence. If you’re ready, please go ahead. Michael Saini: Thanks, Joann.

Methods for a Qualitative Systematic Review

I’m really excited to be here to present to you today the methods for qualitative evidence synthesis. So let’s talk about specifically qualitative synthesis. So now we’ve introduced you to the concepts related to qualitative research of the primary studies. Qualitative synthesis is simply taking those primary studies and bringing them together in a way that you tell a larger story about those experiences that was in the individual primary studies. So it’s a method within a family of systematic reviews that soaks us on either aggregation, integration or interpretation of the data. To aggregate primary studies that are qualitative, you would simply look at the number of times that views and comments or themes are presented and you would add them up across studies and that would give you an indication of how pronounced those themes may be.

Integration, you perhaps then would just collect those themes and you would integrate those themes into a larger theme. Then the interpretation, you may look at those themes and look at the context in which these themes were derived and come up with some interpretations about what does this mean, what new knowledge do we have by bringing these islands of primary studies together. What’s the big picture that gets – that emerges from this process? So similar to systematic reviews, qualitative synthesis should continue to strive to be transparent, systematic and rigorous in its methods. Now there is some debate about this because of the qualitative nature of qualitative synthesis. Some would say that those methods need to be fluid and they need to be moved around and changed around depending on the context or the best fit, the qualitative methods. I think you can have some flexibility. I think that flexibility is wanted and needed to be able to develop a qualitative synthesis that still relates to the qualitative primary studies. I think it’s also important that we document, that we’re transparent, that people can follow our logic of how we got to the results, how do we interpret these findings.

They should be able to go back and see what was our research question, which studies do we include and not include and why do we include them and not include them. What methods do we use to integrate or aggregate or to interpret? What methods were included? What methods were not included? All of these decision points, which I’ll talk about in a moment, are all really important to consider. By doing so, we enhance the utilization value of these qualitative studies and the power to be able to tell a rich and compelling story. There are a growing number of examples of qualitative synthesis within the field of disabilities and rehabilitation. So here are a couple of them that I found just doing a quick search of qualitative synthesis within the disability and rehabilitation field. So you see that there is one on disruption, disbelief and resistance: a meta-synthesis of disability in the workplace. There is another one on youth with disabilities. There is another one on students with disabilities and another one with caregivers.

I think the possibilities are endless. Considering how much qualitative research is now out there in the electronic databases and other sources, there is so much rich qualitative research that if it’s just sitting there for us to pick and to be able to use in a way that tells that larger rich story. So questions specifically related to the interventions could be: How do people experience the identified outcomes? Why does an intervention work or not work? For whom? In what circumstances? Who is not at the table? Who has not been included in these quantitative studies? What groups of people have been omitted from having voice to tell us whether or not these interventions would work or not work? What aspects of intervention are valued? Maybe, for instance, it’s the distance to the intervention. Perhaps, it may be the accessibility issues or the lack of accessibility to the interventions. So knowing something about transportation could be very useful for us, knowing something about the uptake of an intervention and ultimately then, the effectiveness of that intervention. What factors facilitate or hinder the successful implementation of the program? How can a particular intervention be adapted for a large-scale rollout? If you were to take a very controlled, random-controlled trial and wanted to roll that out across jurisdictions, what do we need to know before we begin rolling out that program? Let me just show you two examples of the integration of qualitative research in the evidence-based practice framework as it relates to systematic reviews. The first one is a study by Palinkas in 2008.

They wanted to understand the evidence-based treatment plan that was integrated within agencies, within the child mental health centre. They found that the trainers struggled with the implementation of the treatment. So even though they are measuring the effectiveness of the treatment approach, there was a range of experiences about whether or not the actual intervention was working. So then they went further into the analysis by doing qualitative research and they found that there were many barriers to the implementation of implementing this treatment plan. So people said that there was a lag time between the initial training and the use of the treatment in practice.

So they received the treatment and then a year after that, then the treatment was then included and then evaluated. By that time, they forgot about those core elements of the treatment. So they started to create their own ideas on what that treatment should look like. There was a lack of consistency in the competency in the use of administering the treatment. The clinician engagement with the project was not always consistent as well and therefore, the absolute effect of the treatment may show that it is effective, but the relative effect suggests that that relative effect of whether or not it is beneficial may be influenced by these other factors such as the implementation itself of how the treatment was implemented. This example from Sibthorpe, I think, is a great example of looking at which question is the right question. In this example, they wanted to look at hazardous drinking among an Indigenous Australian Community Medical Centre. So they initiated a random-controlled trial and they had good funding to do this. Their goal was to have 200 people in the intervention, 200 in the control, and then they would look at hazardous drinking.

Interestingly though, only 10 participants agreed to participate. None of them indicated they had a drinking problem. So therefore, none of them could be included into this study. The authors had to actually give back the money to the funders saying, “I’m sorry, but we were unable to fulfill our objective of doing this random-controlled trial.” Now what’s interesting is that they had – they’ve done a qualitative study first to understand and to explore the experiences of the participants about their experiences of hazardous drinking. They would have found that the patients themselves were too embarrassed and did not want to approach or deal with their drinking within this study. So there are barriers to the implementation of having folks come into the survey. Those who were screened, although the authors, to give them credit, included aboriginal health workers to help with the inclusion and the assignment, they, because of their connections with the participants, did not want to disclose that certain people had drinking problems. So, therefore, no one had a drinking problem even though the study was about hazardous drinkers.

Now have they done some qualitative interviews or some focus groups with community members and explore how do we go about – this is an issue that’s been identified in their community. We want to figure out how best to proceed had we – had they spent the time and did some interviews and focus groups. They perhaps could have come up within strategies to have – to best implement such an approach. So qualitative synthesis in its various forms should always be faithful to the primary studies that focus on transparency, that focus on the original studies and tracking those experiences from those primary studies right to the synthesis is critical to ensure that the overall quality of synthesis is credible and that it relates back to those questions. So in terms of capacity to do qualitative synthesis, you need to ensure that you have the experience and the advice of the qualitative research synthesis experts to help you ensure that you have a process. I think just going through these webinars, then you also end up with a level of expertise that can help you develop the specific methods that you want to conduct to do these qualitative synthesis. So I think you’ll then be able to share your expertise with your team. Perhaps, you have to look at whether additional training may be required.

Maybe you will want to read some of the books that are out there, some of the other webinars that are being produced in terms of educating the qualitative synthesis. The budget and cost for additional time and resources, it does take – it can take quite a bit of time to conduct a qualitative synthesis properly because you have to do an information retrieval that’s exhaustive, that tries to identify all of the qualitative studies and then you need the resources and the time to be able to sift and sort all of those studies and then begin the analysis, which then leads into the write-up. So you can see that there are many different steps and it does take time and it does cost – it can cost quite a bit, depending on your setup. So if you’re attached to a university, for instance, those costs may not be as high because there are built-in resources like a librarian who can volunteer their time.

If you’re doing this on your own, then you may want to think about how best to connect to those other resources to offset some of those costs. You’ll need to think about whether or not you have access to the appropriate databases and journals. I think anybody in academia, one of the most fortunate things about being in academia is the access to electronic databases. I just need to log into my account at the university and I have access to millions of journals across sectors. So I can go into journals specifically related to disabilities and rehabilitation and have access to most of the research that’s out there currently in the electronic databases.

If you don’t have access to a university library, then that can be much more challenging. There could be a significant cost to you if you need to purchase articles online and so you’ll need to think about how best to get to those databases in the cheapest way. Again, this is why I think having some collaboration with university researchers is really important because then you have an access, both of the expertise, but also of the journals themselves. I used to, when I first started doing qualitative synthesis, do most of my own information retrieval searching, but I’ve recently been hiring librarians. They’re wonderful at being able to track down this information in a systematic way. I’ve been really happy with the work of the librarians. Not only do they give me the literature that I’m looking for, but they also set up updates.

So as new literature becomes available, I get an email, letting me know of those new studies and then I can go get those studies. So I think if you’re going to develop a qualitative synthesis team, you should really think about including a librarian as part of your group. If there are specific challenges of the systematic review process that you’re thinking that you’re going to be confronted by, maybe it’s choosing a method, maybe it’s choosing an approach, maybe it is trying to figure out how best to deal with different types of methods, then think about who you could have on your team that’s going to be able to help you through this process. There are also some resources that you can review. There are PowerPoint presentations of these webinars. There are books.

What else can you bring in to educate yourself on – before starting this process? So I’m going to just quickly now take you through some of the methodological considerations and including and focusing specifically on information retrieval strategy, which is basically the literature review. Then I’ll also talk about the screening process and then the critical appraisal. The critical appraisal basically means how do you appraise the quality of the primary studies? There is lots of debate about how to do that and I’ll talk about that in a moment. Then choosing appropriate method and then whether or not a review should be a stand-alone or mixed-method design.

So a mixed method integration of quantitative and qualitative primary studies and – which provides you both the information from the meta-analysis and information about coming from the qualitative synthesis and that combination provides you a certain picture of the story or there are other times that you just want to do a stand-alone qualitative synthesis. I showed you some of those examples that are just qualitative relevant to the areas of disabilities and rehabilitation. I’ve done just stand-alone because I wanted to look at the views of specific groups. I wanted to look at, for instance, the children’s views of separation divorce and looking at what input they had within the separation process. So that made sense then to go get all of the qualitative studies that it included children’s views and bring them into a stand-alone review. I also looked at this intervention in child welfare called family group conferencing, where we wanted to know both the effectiveness of the interventions or whether it was effective.

Also, we wanted to look at people’s experience as being part of this approach. So in that example, we used an integrated approach, where we did both the quantitative and the qualitative. So we’d look at and we present in our book this notion of the family of systematic reviews, this idea that qualitative synthesis is not better or worse than meta-analysis.

It’s different. It’s for a different research question with a different approach, much like if you’re going to do a rapid evidence assessment or scoping review or if you’re going to look at the level of evidence, so what type of evidence out there on a specific issue, all of these approaches or all reviews and always in the umbrella of a systematic review. So by doing so, by including qualitative synthesis within the umbrella of families – within the umbrella of systematic reviews so that family approach, then we don’t consider qualitative synthesis as being in the shadows of quantitative, that we don’t consider that a systematic review of random-controlled trials is the gold standard and that the qualitative synthesis is its subordinate, much like the debate of quantitative versus qualitative in the primary studies. We’re saying that that integration of the two and the various purposes for doing qualitative or quantitative are important to consider and this gives us a richer understanding of the complete evidence that’s out there. So going back to the definition of evidence-based practice, it is based on the best available evidence. Sometimes, it’s qualitative and sometimes, it’s quantitative. By also putting it within the family of systematic reviews, we’re also suggesting that whether it’s quantitative review or qualitative review, both of them should be transparent. Both of them should be rigorous in their approach and both of them should be consistent.

You should be able to trace back from beginning to end of a review, whether it’s quantitative or qualitative and be able to identify the critical decision points that were made throughout the process. So this next slide, the family of systematic reviews just basically illustrates what I’ve been talking about. So at the very top, the most important part of beginning a systematic review, whether it’s quantitative or qualitative, whether it’s a rapid evidence assessment or a scoping review, it is the research question. The research question will guide the process. If you are looking to explore much like in the research primary studies of qualitative research, if you are looking to explore the views, the cultural setting, the perceptions, the experiences, then a qualitative synthesis then would be perhaps your best option. If you’re looking to look at the effectiveness of an intervention, then perhaps the quantitative way would be the best way to go. So your research question guides process and then you move into the context of the research process.

Is this going to be a comprehensive search of all the literature that’s out there? Do you have the capacity to do this? Do you have the resources to go get all of the studies that had been out there on a specific topic area? Is your intention to be comprehensive in its focus? If so, then you would go through the left side of the screen, which is that comprehensive information retrieval strategy process, which would include the full comprehensive search of electronic databases and other resources to help you figure out which study should be included and not included. If you’re looking at more provisional results, so you’re not looking at a complete comprehensive picture of all of the research that’s out there but you’re trying to give an overview, then perhaps then you will be doing a brief information retrieval strategy. So perhaps, you’ll only look at peer-reviewed studies and you won’t bring in government reports. You won’t bring in dissertations. You won’t bring in other resources that may be also connected to a comprehensive approach. Here, you’re going to be having some specific questions that you wanted to address.

So you may be using this brief approach because it’s more specific to your research question. Here then, you have a rapid evidence assessment, which basically is the same as the systematic review, whether it be quantitative or qualitative, except you are just focusing on the peer-reviewed data and not the non-peer-reviewed, not the fugitive stuff that’s hard to find. If you’re going to use the scoping review, then there, all you’re doing is mapping out the evidence. I also suggest though that a scoping review can be part of a more comprehensive approach. So you see that error of going back to the comprehensive. I think before you even do a quantitative or qualitative review, you may want to do a scoping review of the literature. I’ve now published a couple of scoping reviews and people seem to be interested and I seem to be – I’m finding really interesting information that comes out of this, just knowing about the type of research that’s out there on a specific issue.

So I wanted to look at the issue of separation and divorce for parents with children with autism. I wanted to know. I didn’t know whether I’d be looking at interventions or qualitative information or cross-sectional studies. So I did the scoping review.

I found every single study that was out there and then I mapped it to look at, “What do we know about this area? What studies have been done? What methods do they use?” Then I took the qualitative studies and went further into my analysis and we’re currently now doing a meta analysis specifically focused on those qualitative studies to look at the experiences of parents to the various transitions of diagnosis, treatment and maintenance. So on the left side where it says type of study, my initial screen included all studies. Only when I started to extract this data, I then divided up my quantitative and my qualitative. So I had two bins. Then by doing that, of course, you have to look at those that are on both quantitative and qualitative.

So much like the process of finding the best research question for a qualitative primary study, the same would apply for determine the research question for a systematic review. You first want to identify the research question that’s going to address the overall target population, the phenomenon that you’re looking to explore or the intervention perhaps, if you’re looking at a combination of qualitative and quantitative. So my example of the family group conferencing intervention that was the intervention that I was looking at, but I kept it open to both effectiveness studies and also studies that looked at the views and experience of those who’ve been through the intervention. So the components of the questions help you determine the type of study that you’re looking for. Is this a research question that will only get at qualitative studies? The question around the views of children post separation divorce, I just wanted to know about the reviews.

So I was not interested in cross-sectional or experimental studies. I was specifically looking at only studies where children were included in interviews about their experiences. It also helps you to the databases and the sources to search. So if I’m looking at a psychological question, there are specific databases that I’m going to want to include such as psych info, med line, sociological abstracts. If I have a question that is more education-based, then I would probably want to include ERIC, which is another electronic database which holds mostly education-related titles and abstracts. Again, if these databases are not familiar to you, then again, all the more reason why including a librarian within this process can be really helpful. Also, the research terms. An example, we were doing a systematic review on cyber reviews.

We wanted to look at what interventions had been developed for cyber reviews. We spent a lot of time trying to find studies and we could not find anything, although we were pretty sure that there were studies out there. Going back to the research question and looking at how we are asking the question, we realized that what we were asking for in our research question was cyber reviews, but what we should have been asking about is cyber safety because we were looking at prevention studies and not studies where cyber reviews have actually occurred. So then by changing the search terms to include cyber safety rather than cyber reviews, we were then able to uncover much more research that was – that had been completed and much more in line with what we were expecting to find. So, again, going back to that diagram of whether it’s comprehensive or whether it is more brief and focused, we want to look at the breadth and scope of the information retrieval process. Are we looking to locate all studies or are we looking to find a certain view? If we’re using the methodology such as a grounded theory approach, where we’re looking at just getting a sample, a theoretical sample from the qualitative studies that had been completed, then that’s going to have an implication of how we search the literature. We want to do this in collaboration with the service users, with those expert panels that I talked about to understand what do we need to know about? How do we go about finding it? What should be the expected time frames for completion? What resources do we need to put in place to ensure that we get all this done? This is where the scoping or the mapping of the evidence can be really helpful. So the information retrieval process, if it’s going to be comprehensive, then it should be transparent. It should focus on both published and unpublished studies, if it’s going to be comprehensive. If you’re going to do a rapid evidence assessment, then maybe that’s just going to be peer-reviewed and not the unpublished.

When you’re locating the qualitative studies special attention should be made to be more to trying to find these studies. If you’re doing quantitative searches, then our filters are much more developed in electronic databases to be able to define studies that are RCTs so we can use those type of filters like randomized and controlled and compare. Although some filters have been developed for qualitative research, it’s – they’re not as good and therefore, we really need to be close to the literature review process to ensure that they research them appropriately. Of course, there has been a debate about whether or not qualitative evidence should be exhaustive, comprehensive or whether they should be more flexible. I really do think it depends on the method that we’re choosing. So, again, if we are using a grounded theory approach or something to – that type of approach, then we would be looking at a – more of a theoretical sample or more of a proposal sampling approach. If we’re looking at more of a systematic approach, then we’d be using a comprehensive approach for identifying the search terms, identifying the electronic databases, going through each of those databases with those search terms and making clear decision points about whether studies should be included or excluded.

So here is an example of a qualitative filter that has been created. I’ve used this just to test this out. I’m not convinced that this is the best approach to use, but I do provide this to you in case you want to try this out.

Again, I’d like to look at all of the evidence and not filter by study design and only filter by target population and the experience that I am trying to capture and then make decision based on titles and abstracts because I’m not always confident that this is the best way to filter down to qualitative studies. So I think the qualitative filters can be too limiting and our focus on the word “systematic” within systematic synthesis suggests that we are really suggesting that you’d be more comprehensive in your approach and that that approach is transparent in the sense that you are documenting all of your decision points. So you’re documenting which electronic databases you use, the search terms, when you did it, and that accountability and that transparency provides the details for somebody else to look at how you went about, including some studies and not others. Your inclusion criteria is based on making clear decision points about what studies should be included and not included and we’ve suggested a multiple level screening process where you go from a liberal screening so you include everything that is relevant to your research question and that is a study, whether it be quantitative and qualitative, and then you go into more specific screening based on the type of design. So then at that point, you would then screen and put it in a separate bin all studies that seem to be qualitative. When you do extraction for the qualitative and quantitative, you want to do that separately, and for the qualitative extraction, so bringing in extraction just basically means bringing in the information that is relevant to your qualitative synthesis. So it could be some demographic information. It would be the themes. It would be the quotes.

There is no standard template for doing this. So there are many different examples out there. One of them that I like using is just simply NVivo. That type of approach of bringing in your primary studies into the NVivo software program, which a qualitative management program, allows you then to code and to highlight specific parts of each primary study text and then you then create themes that way which then allows you to code the data to link it to other documents to then search across themes that you’ve created. That organization can be really important, especially if you have a qualitative synthesis with a number of qualitative studies included. So now you have all of the data, let’s say, into NVivo. You have extracted some common themes and you also want to look at the appraisal of each study to see whether or not it adds to the overall interpretation of the findings. This is different than quantitative research because in quantitative research, you may be using the checklist of some form and looking at the overall rating of quality for that quantitative study and then perhaps even using that into a moderator analysis, whether it’s high quality one, moderate quality two or low quality three, and that one, two and three would be added into your analysis. In qualitative research, we are not developing checklists to put numbers around quality.

We’re using the considerations of quality to build our overall interpretation of those primary studies. So looking at the quality of the evidence is important because it does add to our overall understanding of the applicability and relevance of these primary studies in our overall analysis by looking at issues around credibility and transparency and whether or not these primary studies made a contribution to the wider knowledge of the specific issue that you’re interested in. So within the critical appraisal, it’s really important to have a non-hierarchical view of evidence, specifically around the different research designs that may be used for qualitative research, unlike from a quantitative standpoint where the RCT is a gold standard, everything falls from there. When we look at the critical appraisal within qualitative studies, there is no common approach, but there are some key questions to be asked. So why is the research clear? Did it use a clear statement? I focus a lot on the importance of the research question guiding the process. Was the qualitative methodology appropriate? So if they’re looking at the relationship among concepts, did they focus – did they use, let’s say, a more of a grounded theory approach? Or if they’re looking at the culture nuances, did they use an ethnographic view of the information to bring in those cultural understandings? Was the research design appropriate to address the aims of the research? Was there a good nest of fit with the questions and the methodologies? Was the recruitment strategy appropriate for the aims for the research? If they’re wanting to understand the rich experiences of a select few people but then they had a sample of 50, are they able then to really capture those experiences if they only spent 20 minutes with those 50 people? Or would you rather have seen and spent two to three hours with maybe 10 people? Was the data collected in a way that addressed the research question? Were they able to achieve a good understanding of those emerging experiences based on the methods used? Has the relationship between the research and the participants been adequately considered? So, again, we’re not looking at the research as being objective.

We understand that the researcher is subjective. There should be some transparency around that as well. Ethical considerations are important to consider. The data analysis, is it sufficiently rigorous? There’s a common approach of analyzing data and that’s just doing a thematic representation.

You don’t get the nuances. You don’t get the richness of that analysis process. You want to see that the quotes, for instance, support the themes being generated, that tells you that there is sufficient rigor in the analysis process if you, as the reader, can say, “Oh, yes, I read that quote and that makes sense to what the theme is being presented by the researcher.” Is there a clear statement of findings and then the overall value of the research study itself? So our quality checklist is an appraisal tool that we developed. It has 25 dimensions of quality and includes such things as rigor, credibility, dependability, confirmability, the notions that we’ve talked about. We also included four other dimensions to appraise the authenticity, fairness, promotion of justice. When these are the central components of a qualitative study where specifically, participants are brought into research studies that are qualitative to empower their involvement and to bring action. So, for instance, the participatory action research, the reason why one would bring in participants into a participant action research design is so that they can share in the full production of the research process from beginning to the very end to sharing the information, to being part of the dissemination process.

So in those, goals are set out at the very beginning of a research design where promotion of justice is central to the objectives of the research, then there need to be a clear demonstration that the participants were part of the process right from the very beginning. Now, of course, if you are doing a grounded theory study and the objective is not to empower and not to promote action, then perhaps the promotion of justice would not be central to your analysis around quality. That’s why it’s important to use the criteria in a flexible way, depending on the purposes and the overall goals of the research study that you’re assessing. So our QRC appraisal form was piloted here at the Factor-Inwentash Faculty of Social Work in 2008. We looked at the inter-agreement scores, and then we made modifications. We also developed the user guide in our book and chapter, I believe, seven. We have a section on qualitative appraisal and we also include a working example of using the appraisal form, using two qualitative interviews of family group conferencing. So as summary of the challenges, as we’ve talked about, the first is the research question, whether this should be an iterative or a predetermined focus.

I think that perhaps the best way to start off would be, again, focusing on your intention of doing this qualitative synthesis. In some ways, if you’re looking at more of a predetermined research question and you want to know specifically about some experience or some specific perception, then you may want to start off with that predetermined. Even in those cases, because you’re looking at qualitative research, because it’s so rich in complexity, I think you still need to give yourself an option of changing that question or modifying that question as you go through. That’s different than a quantitative study where you would have a predetermined question at the very beginning. So does A costs as B? You should have all of your moderator variables that may impact the A cost as B, clearly set out and be theoretically linked at the very beginning. So you’re not just putting a whole bunch of stuff into SPSS or some statistical software and see what comes out to be statistically significant because that’s considered, in quantitative terms, that seems to be referred to as dredging. So you would just find what is statistically significant and there is no theoretical basis for choosing those variables. Whereas in qualitative, it’s much more iterative. It’s much more contextual.

So you are very much focused then on being flexible in your methods, rather than rigid. The information retrieval process, theoretical or comprehensive, in our approach, we have developed a comprehensive because if you’re going to spend that much time searching for qualitative research, it’s really interesting to find out the – what’s out there and how much information is out there and not limit yourself to a select group of studies. Then the critical appraisal, whether it’s interpretation versus screening, if you’re screening out, I think you need to have a really good rationale for doing so and I can’t think of a good rationale for screening out even based on poor quality because I do think that’s part of the interpretation process. Then choosing appropriate methods, I’ve given you three ways of thinking about this: aggregative, integrative and interpretive and having the sense of what you’re trying to do with this synthesis prior to your start, whether you’re trying to summarize, whether you’re trying to build a story line or whether you’re trying to generate new information or new theories can really help you in making those decisions. Then issues around transferability versus generalizability, so connecting the islands of evidence and being clear even in your write-up because I think you have to educate other people that these 20 studies should not be generalized to the large population, that it really is contextual, that it’s still based on those 20 studies with the participants’ experience within those 20 studies and so it does not allow us to make inferences now because we’ve simply connected these qualitative studies. It really moves into helping us understand the applicability, the relevancy and the lessons learned from these studies to start pointing us in much clear directions.

For your reference, I’ve included the cover of the book that I’ve been talking about throughout the webinar. It’s Systematic Synthesis of Qualitative Research that I published with Aron Shlonsky in 2012. If you want to email me for that information or any other questions you have about this webinar, I’d be happy to respond to you. I really do appreciate you taking the time to listen and to read through these PowerPoints and I am hoping that this creates more discussion.

Again, I don’t have all the answers. I think though moving in this direction of finding ways to include qualitative research is a right set for it and I encourage you to begin this journey or to continue on this journey and I wish you all the best in doing so. Joann Starks: Well, thank you very much, Dr. Saini, for a very informative presentation. I also want to thank everyone for participating today. We hope you found the session to be interesting and helpful and that you’ll join us for the next two webinars.

Here on the final slide, we have a link to a brief evaluation form, and we’d really appreciate your input. We will also be sending an email with the link for the evaluation to everyone who registered. On this final note, I would like to conclude today’s webinar with a big thank you to our speaker, Dr. Michael Saini from myself, Ann Williams and all of the staff at the KTDRR. We appreciate the support from NIDRR to carry out the webinars and other activities.

We look forward to your participation in the following sessions. Thank you.

Views: 3 620 Likes: 15 Dislikes: 0
100% Likes
0% Dislikes
James Martin, Doctoral Dissertation Defense, 2015

Omar: Okay. Alright, well, I think... James, you ready? James: Yep. Omar: Okay, welcome to the dissertation defense of James Martin. James has been working on the general problem of…

Views: 590 By: Charlie Martin
Jaesung Choi Doctoral Dissertation

Hi, my name is Jaesung Choi. I'm a PhD candidate in Transportation and Logistics Program at North Dakota State University. I'm planning to complete my PhD program in summer…

Views: 54 By: Ugptindsu
Dissertation Defense Auriel Fournier #MORailsDefense

[muffled talking in the background] [muffled talking in the background] [muffled talking in the background] [muffled talking in the background] [muffled talking in the background] [muffled…

Views: 470 By: Auriel Fournier
ABD Research Tools for Cohort 25

Hello Cohort 25 ! I wish that I could be with you in-person today, but I’m recovering from a minor surgical procedure. So, I made this video to share a couple of research reminders…

Views: 22 By: Dorinne Banks