Predicting and Improving NBME Performance: A Collaborative Approach Implemented at Texas A&M University

Clinical rotations represent an exciting phase in medical education, promising hands-on learning, direct patient interactions, and valuable teamwork with residents and faculty.  However, this phase in medical education also brings the challenge of shelf exams, which can be particularly daunting for new clinical students. These exams present unique hurdles. Each shelf exam for the core clinical rotations covers a distinct and extensive body of knowledge, demanding a deep understanding of medicine and its clinical applications.   

Furthermore, the transition to clinical rotations often requires a significant adjustment from the study habits developed during the preclinical years. While preclinical studies may have centered around attending lectures, reviewing notes and flashcards, and taking practice questions during dedicated study periods, clinical rotations involve long hours, demanding schedules that can include overnight and weekend duties. This shift is both physically and mentally taxing, significantly impacting available study time and overall stamina. Adding to the complexity, students face a steep learning curve as they adapt to new clinical responsibilities, protocols, and expectations. Balancing patient care, personal life, and shelf exam preparation creates a considerable challenge.   

Against this backdrop, medical educators face the ongoing challenge of adequately preparing students for rigorous standardized exams like the National Board of Medical Examiners (NBME) subject exams. At Texas A&M University, College of Medicine, a collaborative approach using TrueLearn quizzes has shown promising results in predicting and improving student success on the Family Medicine NBME.

Clinical Rotation Challenges Impacting NBME Outcomes

The impetus for this initiative stemmed from a concerning trend: an average 9.2% failure rate on the Family Medicine NBME between 2019 and 2023. This rate aligned with the national average at the 10th percentile, highlighting a significant need for intervention. 

The breadth of content covered during a family medicine rotation is extensive. Coupled with the relatively short clerkship duration, this presents a challenge for students accustomed to ample time to dedicate to studying in their routine. Additionally, clinical experiences, while valuable, don’t always directly correlate with the specific focus of the NBME exams. This can lead to knowledge gaps that are not discovered until final preparations, or worse, on the actual shelf exam.

Limitations of Ineffective Study Strategies

New clinical students often lack clear and effective preparation strategies, which leads to increased stress among students as they begin preparing for their first shelf exam—if they underperform, the stress amplifies.

Study methods that students would employ during the preclinical years, when there is more protected time for studying, are quickly found to be less effective during clinical years. For example, spending hours reading texts, listening to lectures, and reviewing notes is honestly not an option during rotations. 

These passive forms of learning have their place, but are not sufficient in and of themselves for content mastery and long-term retention, and certainly not efficient. Pairing such methods of learning with active learning strategies that involve the retrieval of information learned increases learning efficiency and has been shown to significantly improve critical thinking, application of knowledge, and lasting retention.

Utilizing question banks and flashcards is an example of active learning. These tools are effective resources, and medical students are known to rely heavily on them throughout medical school. However, during clinical rotations, students don’t have significant chunks of time to take practice questions and work through flashcard decks. The time constraints they face often lead to ineffective utilization of these resources during clinicals—selecting questions randomly without strategy, plowing through questions and flashcards in a cram session, forgoing review of answer explanations in practice questions, or overly relying on these as a sole study resource altogether. 

Structured Active Learning Based on Adult Learning Theory

Students need a robust feedback loop to help guide their learning and study efforts. Quizzes offer continuous feedback and have been shown to aid in the early identification of at-risk students.¹ The implementation of continuous assessment through quizzes can significantly enhance student engagement and retention of knowledge.² Studies have shown that regular quizzing correlates with improved academic performance, as it encourages students to review and consolidate their learning.³ The Assessment for Learning framework positions quizzes as integral to promoting self-regulated learning. Leveraging predictive analytics allows for early detection of students at risk of failing, enabling timely interventions.⁴

To ensure students receive timely feedback on their learning and to model effective learning strategies to students, a structured intervention was implemented using TrueLearn’s learning and assessment platform. 

Clerkship directors cultivated a structured approach to help the students focus their study efforts and establish a feedback loop to better support students. The intervention involved four key components: timed weekly quizzes within the TrueLearn platform to simulate the test environment, NBME-style questions with immediate performance feedback, over 350 additional questions in “tutor mode” for extra practice, and faculty monitoring of student performance to allow for early identification and support of struggling students.

The intervention’s design incorporated key principles of adult learning theory to maximize effectiveness:

• Self-Directed Learning: Students could actively track their progress, fostering a sense of ownership over their learning.
• Relevance: Quiz content was carefully mapped to real clinical cases, enhancing the relevance of the material.
• Feedback: Real-time quiz feedback provided immediate insights for students to adjust their learning strategies.
• Motivation: Both internal motivation and performance-linked motivation were supported through the design of the intervention.

Predictive Power of TrueLearn Quizzes on NBME Success

The results of this intervention at Texas A&M University have been encouraging. A retrospective analysis of one and a half years of Family Medicine Clerkship student data was conducted by collecting pass/fail statuses for four quizzes with a similar format and the NBME exam, coded as binary variables (1 = fail, 0 = pass). Logistic regression modeled the likelihood of NBME failure based on quiz performances, with odds ratios (ORs) and 95% confidence intervals (CIs) assessing association strength and significance of associations. Results suggest that failing two or more TrueLearn quizzes significantly increased odds of NBME failure (OR = 3.54; 95% CI: 1.46–8.39; p < 0.01).

These results underscore the value of formative assessments in guiding instruction, as highlighted by the Assessment for Learning Framework. Predictive analytics enables data-driven strategies to pinpoint and aid at-risk students. By leveraging student quiz performance and implementing targeted support, such as focused test-taking skills, extra resources, and more practice questions, learning outcomes improved, demonstrably reducing the program’s first-time NBME failure rate from 9.2% to 4.8% (surpassing the national average of 6.2%). This approach equips students for enhanced success in their clinical practice through data-supported, targeted interventions.

“Predictive analytics enables data-driven strategies to pinpoint and aid at-risk students…reducing the program’s first-time NBME failure rate from 9.2% to 4.8% (surpassing the national average of 6.2%).”

Beyond academic gains, the TrueLearn quiz intervention positively impacted students psychologically. Regular check-ins and progress tracking reduced uncertainty and anxiety surrounding the NBME exam. Quantitative analytics empowered students and fostered a sense of ownership over their learning. Opportunities for “small failures” within the quizzes provided valuable learning experiences and helped build resilience. Finally, the system encouraged students to seek help when needed and to engage in self-reflection on their learning.

Conclusion

The collaborative use of TrueLearn quizzes, grounded in adult learning theory, has proven to be an effective strategy for predicting and improving student performance on the Family Medicine NBME. This model offers valuable insights for medical educators seeking to enhance student learning and support success in high-stakes examinations.

TrueLearn offers a comprehensive set of practice questions and detailed explanations tailored to exams like the NBME, USMLE, COMLEX, and various medical specialty boards. The platform simulates the actual exam experience, allowing users to get comfortable with the format and types of questions they will encounter on exam day. Additionally, TrueLearn provides performance analytics to track progress and identify content areas and topics that may need extra attention, helping students focus their study efforts more effectively. 

For further information or collaboration, please contact:

Tim Murphy, MBA
Undergraduate Medical Education Program Administrator
Family Medicine, Neurology, and Critical Care
Baylor Scott & White Health, an affiliate of Texas A&M University, College of Medicine

Benjamin McKinney, MD
Physician, Baylor Scott & White – CitySquare Clinic
Associate Program Director for Baylor University Medical Center Family Medicine Residency
Family Medicine Executive Clerkship Director and Adjunct Associate Professor for Texas A&M University, College of Medicine

References

1. DiMeo, Fabrizio & Martí-Ballester, Carmen. (2020). Effects of the perceptions of online quizzes and electronic devices on student performance. DOI: 10.14742/ajet.4888 
2. Katzman SD, Hurst-Kennedy J, Barrera A, Talley J, Javazon E, Diaz M, Anzovino ME.2021.The Effect of Specifications Grading on Students’ Learning and Attitudes in an Undergraduate-Level Cell Biology Course. J Microbiol Biol Educ.22:e00200-21. DOI: 10.1128/jmbe.00200-21
3. Walck-Shannon, E. M., Cahill, M. J., McDaniel, M. A., Frey, R. F. (2019). Participation in voluntary re-quizzing is predictive of increased performance on cumulative assessments in introductory biology. CBE—Life Sciences Education, 18(2), ar15. DOI: 10.1187/cbe.18-08-0163
4. Azzi, A.J., Ramnanan, C.J., Smith, J., Dionne, É., and Jalali, A. (2015), To quiz or not to quiz: Formative tests help detect students at risk of failing the clinical anatomy course. American Association of Anatomists, 8: 413-420. DOI: 10.1002/ase.1488