TrueLearn Data Lab: The Best Way to Use Question Banks

How should I take questions? What type of questions should I take? What testing mode should I use? These are common questions we receive from students, residents and physicians looking to get that edge for their next in-training, certification or MOC (maintenance of certification) exam. Let’s dive into the data we have gathered to discover what our high performers have determined is the best way to use question banks.  

Qbank Test Modes

First, let’s review the three types of tests you can create in TrueLearn and how they work. The TrueLearn SmartBank platform allows you create tests in three types of modes:

1. Timed Mode

This testing mode imposes a time limit for your test based on the maximum average per question time you have to spend to complete the actual exam. If you create a 10 question test in TrueLearn and the maximum average time you are allotted to complete the actual exam is two minutes per question, you will have a maximum of 20 minutes to finish the test.

Also, you will not be able to see correct and incorrect answers and explanations until you complete the test. Timed mode most closely simulates the actual exam and is used for summative assessment testing.

2. Untimed Mode

This testing mode is the opposite of timed mode in that it does not impose any time limit during the test. However, like timed mode, you will not be able to see if you answered the questions correctly or the answer explanations until you complete the test.

3. Tutor Mode

This testing mode moves question-by-question, displaying the correct answer and explanation after each question is answered and before the next question appears. Once an item is answered, its explanation is retained for the duration of the test.

In this mode, the platform does not impose any time limit on the test. This mode is ideal for formative practice testing, in which you can read one question’s explanation and consider the material carefully before moving onto the next question.

Tutor Mode & Timed Mode are the most commonly used test modes, together accounting for over 91% of all tests taken by students preparing for COMLEX Level 1. This pattern holds true when considering students and residents preparing using other TrueLearn practice questions, as well.

TrueLearn Test Creation Methods

• Quick Create – Simulate the Exam: Test using blueprint categories in balance with the actual exam.
• Custom Test Creation: This mode provides the most flexibility in testing and allows you to customize your test type, question type, marked items, and lets you decide which categories to include.
• Quick Create – Improve Weaknesses: Test using questions in your weakest categories, questions you previously answered incorrectly and questions you mark along the way.
• Quick Create – Diagnose Strengths and Weaknesses: Create an exam from categories where you have used the fewest number of questions.
• Retake: Repeat a previously completed test.

5 Ways to Get the Most out of Your Question Bank

This leads to our next question: does using different practice testing modes lead to different outcomes, in terms of improvement or practice test scores?

To analyze this, we created a group of “high performers” and a group of “lower performers,” and evaluated whether there were significantly different study patterns across the two groups. This dataset includes over 2,500 users who finished at least 800 questions in our COMLEX Level 1 2016 QBank.

Here’s what we found to be the best way to use question banks:

1. Higher performing students used Timed Mode tests more often than lower performing students, with high performing students taking 77% of all questions in Timed Mode.
2. Both groups of students transitioned to a higher percentage of Timed Mode questions as they progressed through the time available in their subscriptions, although the higher performing group consistently took a larger proportion of Timed Mode throughout. Make sure to mix at least some Timed Mode practice in early.
3. Before test day, ensure you become comfortable with the timed, simulated exam experience. You want to become familiar with taking questions randomly from all categories, and you will want to become comfortable with taking tests under a realistic time constraint. Our highest performing students take more Simulate Mode questions and less custom created tests.
4. Start your test preparation early (at least six months before your exam) and take all the unique questions, you will maximize your average score performance.
5. The more practice questions you take, the better result you can expect. The average percentage improvement per 1,000 questions taken = 4.1 percent. Therefore, if a student takes all 2,400 unique COMLEX Level 1 practice questions exactly one time, they can expect an average score increase of 9.8 percent.

Our Research and Methodology

To create these groups for comparison, we looked at each student’s score for their first and last 30% of first-time-taken questions. Performance is analyzed in two axes, improvement in score and the final score of last 30% questions taken.

Then users who are simultaneously the top/bottom 12.5% in both axes are classified as being “high performers” (students who significantly improved over the course of their practice questions and ended up with top scores) or “lower performers” (students who showed low or no improvement and ultimately performed at a comparatively low level).  

The next part of analysis is to investigate both groups’ study patterns across a list of attributes. These attributes include factors such as:  

1. The number of questions taken
2. The overall percentage of questions taken in different modes (study pattern)
3. Study pattern employed during different time periods

Wilcoxon Test is carried out in order to test whether the population mean of these factors between the two groups is different. Null hypothesis is that the population mean of these attributes between two groups is the same.

P-value is computed and values below 0.05 would reject the null hypothesis. Rejecting the null hypothesis means that the population mean is different for that attribute at 95% confidence level.

Conclusions

After conducting this comparative analysis, it is clear different students and residents take advantage of practice test modes in different ways. Some TrueLearn users take almost all timed mode questions; others rely heavily on untimed modes.

However, the “high performing” group used a higher proportion of Timed Mode questions in general than students in the “lower performing” group.

Both groups demonstrated an increasing portion of questions taken in Timed Mode as their exam date approached, but higher performing students took a larger portion than lower performing students. Results show the two groups’ population mean of (1) percentage of questions taken in different modes and (2) study pattern employed during different time periods is different at a confidence level of 95%.

Finally, after identifying both groups of users, we compared their proportions of test creation modes. From the summary table below, over-performers take significantly more questions under tests created by Simulation Mode, and they take less questions under tests that are custom created by themselves.

Considering all three “quick create” modes together (Simulate, Improve, Diagnose), the over-performing students attempted more questions than the lower performing group. Upon applying the Wilcoxon test, we can see the group population averages for the over-performers and under-performers for the percentage of questions taken under Simulation Mode and under Custom Mode are significantly different with a confidence level of 95 percent.

Summary Table