Title: Synthesis of Guidelines for Assessing the Quality of Student-Generated Point of Care Ultrasound Images
Authors: Madelyne Green, OMSII; Lauren Kirk, OMSII; Andrew Widman, OMSII; Ryan Voelker, OMSII; Elizabeth McMurtry, DO
Introduction
Point of care ultrasound (POCUS) is emerging as a new staple for bedside evaluation and has been heralded by some as the “‘visual stethoscope” of the 21st century. In the primary care setting, especially in rural areas, POCUS can bridge the gaps caused by a large physical distance to tertiary care centers since its uses are broad. Its integration is hindered by the systemic lack of guidelines and credentialing for its uses in both medical education and practice. As a result, implementation of POCUS into medical education varies widely between institutions, creating a gap in knowledge for both medical students and incoming residents. We endeavored to review existing POCUS curricula and materials to devise POCUS guidelines for medical students at Pacific Northwest University’s (PNWU) College of Osteopathic Medicine.
Methods
A literature review was conducted on the integration of point of care ultrasound into medical school and residency curricula with limited results. PNWU convened a POCUS task force to streamline POCUS initiatives and refine curricular objectives. Results of the literature searches, reviewing curricula for ultrasound technicians, and input from the Task Force pointed towards the immediate first step of developing guidelines, or checklists, to help faculty and students determine whether an image is suitable enough for medical evaluation.
Results
We developed sets of POCUS guidelines for two study sites: the gallbladder and uterus. The gallbladder guidelines include criteria for taking images of the long axis, short axis, neck of the gallbladder, and CBD and portal triad. The uterus guidelines include criteria for the sagittal view and transverse view. All views contain criteria for gain, depth, image centering, and certain measurements when applicable. Most of the criteria were binary, having ‘satisfactory’ and ‘unsatisfactory’ response options. A handful of the criteria contained a third “needs improvement” option.
Discussion
The POCUS Task Force believes the format developed in these guidelines will help students and instructors quantify what elements comprise a quality image and will be suitable for developing criteria for imaging other anatomical sites. Our next steps involve formally testing these guidelines with medical students to ensure their accuracy and utility in teaching with the ultimate goal of developing an Entrustable Professional Activity for medical students across the nation.
A question from a judge: You provide a checklist for 2 common POCUS examinations, but no information about how these points were proposed, evaluated, and included in the final checklist. Was this done by consensus, majority vote, or some ranking system? Were there criteria proposed that didn’t make the list? If so, why not? I think we would like more information about this process to be able to better evaluate the final product.
Dr. Elliott, thank you for your question. The OMS 2 students were exposed to these scans as part of FM 601 this year, and several of the team members on this project spent a few weeks this summer with Dr. Baldwin, Dr. Brady, and Dr. Justice learning how to take quality images of these and other areas so that we could act as TAs for that class. As a result we had a baseline idea of what constitutes a “quality image.” As we completed our literature review it became clear that the existing guidelines for ultrasound are highly targeted towards ultrasound (US) technologists. US techs are required to perform much more in-depth studies than physicians, and dedicated US machines have many more functions than the handheld POCUS machines offer. In compiling these guidelines we used existing guidelines for dedicated ultrasound technologists and drew on our previous training to trim them down to exclude functions that the POCUS machines cannot perform, and to take out parts of the studies that a physician would not do with a POCUS (ie. a full uterus US includes transvaginal views, which would not be done with a POCUS at bedside). Additionally, we aimed to create a checklist of topics that were quantifiable instead of vague (ie. “good image”), for the purposes of grading students. At that point we had PNWU’s newly hired US technologist review the criteria to ensure we did not miss any key exam pieces. We elected to create the guidelines in this way because we found several studies with synthesized guidelines that needed a professional radiologist to evaluate them for correctness. Given that most of the faculty at PNWU who will be expected to evaluate student images are not radiologists, we wanted to create guidelines that were accessible to faculty after several hours of POCUS training.
Dr. Elliott, thank you for your question. The OMS 2 students were exposed to these scans as part of FM 601 this year, and several of the team members on this project spent a few weeks this summer with Dr. Baldwin, Dr. Brady, and Dr. Justice learning how to take quality images of these and other areas so that we could act as TAs for that class. As a result we had a baseline idea of what constitutes a “quality image.” As we completed our literature review it became clear that the existing guidelines for ultrasound are highly targeted towards ultrasound (US) technologists. US techs are required to perform much more in-depth studies than physicians, and dedicated US machines have many more functions than the handheld POCUS machines offer. In compiling these guidelines we used existing guidelines for dedicated ultrasound technologists and drew on our previous training to trim them down to exclude functions that the POCUS machines cannot perform, and to take out parts of the studies that a physician would not do with a POCUS (ie. a full uterus US includes transvaginal views, which would not be done with a POCUS at bedside). Additionally, we aimed to create a checklist of topics that were quantifiable instead of vague (ie. “good image”), for the purposes of grading students. At that point we had PNWU’s newly hired US technologist review the criteria to ensure we did not miss any key exam pieces. We elected to create the guidelines in this way because we found several studies with synthesized guidelines that needed a professional radiologist to evaluate them for correctness. Given that most of the faculty at PNWU who will be expected to evaluate student images are not radiologists, we wanted to create guidelines that were accessible to faculty after several hours of POCUS training.
Thank you for your work. I am NOT an official judge for your project, but I do have a question. As a non-physician, I’d like to ask if you believe there is adequate baseline knowledge to generate these images accurately? Or does this process require someone who has experience, such as an ultrasound tech or ultrasound educator, to move this forward with confidence? Thanks again!
Dr. Briggs-Early, I do think there is baseline knowledge required. We noticed during the course of our literature review that several papers noted that residents and physicians who have not had much exposure to POCUS do not feel confident in their abilities to perform scans on patients. There are many small technical pieces to POCUS (and regular ultrasound) that one needs to be taught in order to get a quality scan, such as the amount of gel to use, the angle of the probe, and how to equate the image on the screen to the direction of the probe head.
As a judge, I would like clarification on the process. Your group conducted a literature review to inform curriculum,? but you also state you created guidelines from multiple sources, which appear to include sources outside of your literature review. Also, once material were compiled, how were items to include in the guidelines decided? For example, did the faculty member make those decisions, others not listed on the presentation etc? Thanks!
Dr. Fritz, I appreciate the question. I think we have answered this more in detail in response to Dr. Elliott’s question, which was similar. Our citations slide includes both citations from the literature review, and the synthesized guidelines. There was quite a bit of overlap. The criteria were based on easily assessed measures and were decided on by us as a group. Several of us spent time this summer with Dr. Brady, Dr. Baldwin, and Dr. Justice learning POCUS in depth in order for us to TA for the FM 601 POCUS course. This gave us a good background idea on what a “quality image” consists of. Additionally, Sarah Sanders, the US technologist hired by PNWU this year, reviewed our guidelines to make sure we did not omit any key exam pieces. Lastly, we submitted these guidelines to the PNWU POCUS Curriculum Task Force, which is composed of several EM physicians (who use POCUS heavily), Sarah Sanders, and other faculty, and received the feedback that these guidelines were of high quality.
Thank you for your presentation. As a judge, I would like to know about your plans for testing accuracy and utility in curriculum? How will you test and at what metrics will you be looking?
Thank you for the question Dr. Salido. We have not ironed out all of the details of this next step, but likely we will have students attend a several hour training where they are shown how to perform these exams. Then, at some point after the training, they will be asked to create scans of both organs and save their scans to the ButterflyIQ app. These scans will be uploaded to a secure database and any identifying info will be removed. Then we will have three people, ideally faculty members, grade each scan separately from one-another. The scores the scans receive will then be evaluated for inter-rater agreement. Depending on the grader’s agreement, we will assess if changes need to be made to the checklist. We will also assess what proportion of the students are successfully creating images rated as quality.