#NephJC Chat
Tuesday July 14 2020 9 pm Eastern
Wednesday July 15 2020 9 pm IST
Wednesday July 15 2020 9 pm BST
Semin Nephrol. 2020 May;40(3):249-263.
doi: 10.1016/j.semnephrol.2020.04.003.
To Tweet or Not to Tweet, That Is the Question
Avital Y O'Glasser, Rebecca C Jaffe, Michelle Brooks
PMID: 32560773 Full Text at Seminars
AND
Semin Nephrol. 2020 May;40(3):309-319.
doi: 10.1016/j.semnephrol.2020.04.011.
Quality Appraisal and Assurance Techniques for Free Open Access Medical Education (FOAM) Resources: A Rapid Review
Daniel K Ting, Patrick Boreskie, S Luckett-Gatopoulos, Lisa Gysel, Matthew B Lanktree, Teresa M Chan
PMID: 32560781 Full Text at Seminars
Check out the entire issue, chockablock with all things nephrology and FOAMed. The next NephJC chat will be about the entire issue, but we will anchor it around these two articles discussed in a bit more detail below.
Introduction
In this week’s NephJC, we’ll be discussing two articles from the nephrology social media issue of Seminars in Nephrology. Social media has become an important player in medical education and discussion. An abundance of Free Open Access Medical education (FOAMed) resources exist. If you are new to this space then it’s important to know that of the platforms, Twitter, has emerged as the predominant means by which medical topics are discussed.
The first article, To Tweet or Not to Tweet, That is the Question, is a comprehensive narrative review of the role and value of twitter in medical education. It provides an overview of how twitter can be integrated into medical education and gives the reader a road map for getting started with Twitter for medical education.
The ease with which medical information can be distributed on social media has led to a Cambrian explosion of FOAMed resources. The problem is not finding FOAMed but rather separating the wheat from the chaff. How do users assess quality in this internet fueled abundance. Online posts often stimulate real-time debates which serves as a form of peer-review, but further measures of quality assurance need to occur if online medical education is to continue to evolve into a firstline resource for medical education.
The second article of this discussion, Quality Appraisal and Assurance Techniques for Free Open Access Medical Education (FOAM) Resources: A Rapid Review, addresses this issue.
Table 1 from Ting et al, Seminars in Nephrol 2020
The goal of the authors was to identify quality assurance techniques that can be used to assess the validity of online medical education resources. In order to do so, they performed a review of the literature for potential tools and came up with a list of ten quality appraisal tools.
The Study
Methods
This study utilized a modified rapid review. What is this, you may ask? Let’s start with, “What is a systematic review?” — a comprehensive review for a particular topic or study question. Systematic reviews are exhaustive and time consuming. Rapid reviews attempt to answer the study question, just like a systematic review, but try to do so in a more timely manner. They accomplish this by narrowing the study question, limiting the literature search, and limiting the analysis. The benefit is the production of an article with findings synthesized from various other articles with relatively greater speed.
For this article, the question for the study was “For medical trainees or physicians who use FOAM, do critical appraisal tools or checklists improve the assessment of reliability of the online resource when compared with individual gestalt?” With the help of a medical librarian, they searched Medline and Cumulative Index of Nursing and Allied Health Literature databases using the relevant keywords. Parameters for the review started with a limitation of studies published between 2014 and 2019; studies written only in English. Only full-text studies were included in the final review and results were summarized in a descriptive format.
Studies included in the paper were evaluated for quality utilizing the Medical Education Research Study Quality Instrument (MERSQI score). The MERSQI score is an evaluation tool used for individual articles in education research. It evaluates six domains:
Study design
Sampling
Type of data
Validity of evaluation instrument
Data analysis
Outcomes
Higher scores are associated with a higher acceptance rate to peer-reviewed journals.
Outcomes of the study were simply the identification of tools that can assist a reader in assessing FOAM resources. This is an initial study in this area and is intended as one which lays the groundwork for future studies.
Results
The initial database search yielded 5080 citations. After sifting through the citations, 13 articles were included in the descriptive analysis.
Figure 1 from Ting et al, Seminars in Nephrol 2020
As mentioned, the goal of the study was to identify tools that can assist readers during their evaluation of FOAM resources. Although 13 articles were included in this study, only 10 tools were listed in this paper. This is because some manuscripts evaluated the same tool.
Table 3 from Ting et al, Seminars in Nephrol 2020
Table 4 from Ting et al, Seminars in Nephrol 2020
The authors do make recommendations for the best appraisal tools found in their search. The average MERSQI score between two independent raters helped identify tools with the most merit.
The first tool the authors recommend was the Medical Education Translatinal Resources: Impact and Quality-8 (METRIQ-8) tool for readers to assess individual online medical resources. This tool was created in 2016 with the rationale of streamlining the process of critical appraisal of online educational resources (OERs). The METRIQ-8 score consists of a series of eight questions that the reader should ask when reading an OER. Each question is scored on a seven point scale. Higher cumulative scores indicate a higher quality resource. The METRIQ-8 score was found to correlate well with faculty educator gestalt.
The METRIQ-8 score. Table from Digital Appraisal (PDF link)
The authors also recommended the Academic Life in Emergency Medicine (ALiEM) tool for educators to curate resources. This tool rates online resources in several domains. Posts with at least 30 out of 35 total possible points are considered to be quality resources.
Table 2 from Lin et al, JGME (PDF link)
Other appraisal tools are also mentioned were the METRIQ-5 score -- unsurprisingly a more simplified version of the METRIQ-8 score. The “Gestalt Score” mentioned in table 3 is considered to be a gold standard, but as it is simply the “gut feeling” of the validity of a resource, it has considerable variability amongst users. Social media index serves as an objective, single score. It correlates well with other quality appraisal tools, but may disproportionately value websites with a strong social media influence and underestimate quality websites with a smaller readership.
Discussion
This article lays the groundwork for future investigation to quality assessment of FOAM resources. The natural tendency of information in the world has been one that has flowed from a state of expensive and scarce to free and ubiquitous. Medical educational material will continue to follow the same course. As such, it is our responsibility to ensure that the quality of such resources is apparent to all who read it. The evaluation tools mentioned in this article are a good collection of tools needed to start this process in earnest.
Strengths of this article are that it covers appraisal tools in more depth than earlier literature. Limitations of the article are that it is a rapid review. Because it only included literature in the English language in two traditional scientific databases, publication bias may exist. The authors did try to minimize this bias by performing a manual search of the references in the articles. Lastly, a limitation of the article was the descriptive format of the analysis -- one which may have overlooked inconsistencies or contradictions in the articles.
In the later paragraphs, the article also mentions the next step — strategies for integrating FOAM appraisal tools into academia. These are good ideas. It is incredibly easy to post educational information on Twitter. It is now also easy to create an educational website. Due to this, we can expect the continued proliferation of online medical educational resources. Just like we learned while getting used to Web 1.0, the phrase “just because it’s on the internet doesn’t mean it’s true” is a sentiment that can be extended to online educational resources. Just because something is posted online by a physician or other medical professional doesn’t mean it’s true. As any seasoned physician knows, it takes a lot of effort to correctly understand the published medical literature and communicate this in a clear and true manner. Academic medicine has already built the skeleton on which #medtwitter stands and will continue to mold this landscape. Table 5 outlines strategies to incorporate FOAM appraisal into the workflow of academic medicine. This paper lays important groundwork in this area. We could expect future studies to assess the feasibility of implementing various tools as well as comparing the effectiveness of one tool compared to another. Be on the lookout for more literature in FOAM appraisal.
Summary prepared by Gerren Hobby,
NEA Baptist, Jonesboro, AR;
University of Arkansas for Medical Sciences, Little Rock
NSMC Intern, Class of 2020