TrueLearn Data Lab: Which Test Mode Do Higher Performers Use?

How should I take questions? What type of questions should I take? What testing mode should I use? These are common questions we receive from students, residents and physicians looking to get that edge for their next in-training, certification or MOC (maintenance of certification) exam. Let’s dive into the data we have gathered to discover what works best.  

First, let’s review the three types of tests you can create in TrueLearn and how they work. The TrueLearn platform allows you create tests in three types of modes:

1. Timed Mode

This testing mode imposes a time limit for your test based on the maximum average per question time you have to spend to complete the actual exam. If you create a 10 question test in TrueLearn and the maximum average time you are allotted to complete the actual exam is two minutes per question, you will have a maximum of 20 minutes to finish the test.

Also, you will not be able to see correct and incorrect answers and explanations until you complete the test. Timed mode most closely simulates the actual exam and is used for summative assessment testing.

2. Untimed Mode

This testing mode is the opposite of timed mode in that it does not impose any time limit during the test. However, like timed mode, you will not be able to see if you answered the questions correctly or the answer explanations until you complete the test.

3. Tutor Mode

This testing mode moves question-by-question, displaying the correct answer and explanation after each question is answered and before the next question appears. Once an item is answered, its explanation is retained for the duration of the test.

In this mode, the platform does not impose any time limit on the test. This mode is ideal for formative practice testing, in which you can read one question’s explanation and consider the material carefully before moving onto the next question.

As you can see, Tutor Mode & Timed Mode are the most commonly used test modes, together accounting for over 91% of all tests taken by students preparing for COMLEX Level 1. This pattern holds true when considering students and residents preparing using other TrueLearn practice questions, as well.

In our data set, it is clear different students and residents take advantage of these practice test modes in different ways. Some TrueLearn users take almost all timed mode questions; others rely heavily on untimed modes.

This leads to our next question: does using different practice testing modes lead to different outcomes, in terms of improvement or practice test scores?

To analyze this, we created a group of “high performers” and a group of “lower performers,” and evaluated whether there were significantly different study patterns across the two groups. This dataset includes over 2,500 users who finished at least 800 questions in our COMLEX Level 1 2016 QBank. Generalizing the results of this Data Lab to other products will be left for future articles.

To create these groups for comparison, we looked at each student’s score for their first and last 30% of first-time-taken questions. Performance is analyzed in two axes, improvement in score and the final score of last 30% questions taken.

Then users who are simultaneously the top/bottom 12.5% in both axes are classified as being “high performers” (students who significantly improved over the course of their practice questions and ended up with top scores) or “lower performers” (students who showed low or no improvement and ultimately performed at a comparatively low level).  

The next part of analysis is to investigate both groups’ study patterns across a list of attributes.

These attributes include factors such as:  

1. The number of questions taken
2. The overall percentage of questions taken in different modes (study pattern)
3. Study pattern employed during different time periods

Wilcoxon Test is carried out in order to test whether the population mean of these factors between the two groups is different. Null hypothesis is that the population mean of these attributes between two groups is the same.

P-value is computed and values below 0.05 would reject the null hypothesis. Rejecting the null hypothesis means that the population mean is different for that attribute at 95% confidence level.

After conducting this comparative analysis, it is clear that the “high performing” group used a higher proportion of Timed Mode questions in general than students in the “lower performing” group.

Both groups demonstrated an increasing portion of questions taken in Timed Mode as their exam date approached, but higher performing students took a larger portion than lower performing students. Results show the two groups’ population mean of (1) percentage of questions taken in different modes and (2) study pattern employed during different time periods is different at a confidence level of 95%.

Data Lab Results

Based on the latest COMLEX Level 1 2016 users’ data, we identified significant differences in the study patterns of “high performing” students and “lower performing” students.  To structure your practice testing like higher performing students who showed a high rate of improvement and scored near the top of their peer group, we recommend you keep these things in mind:

1. Higher performing students used Timed Mode tests more often than lower performing students, with high performing students taking 77% of all questions in Timed Mode.
2. Both groups of students transitioned to a higher percentage of Timed Mode questions as they progressed through the time available in their subscriptions, although the higher performing group consistently took a larger proportion of Timed Mode throughout. Make sure to mix at least some Timed Mode practice in early, and ensure you become comfortable with the timed, simulated exam experience before your test day.