The Writer’s Series: Examining Testwiseness and Item Flaws in USMLE Question Items
As you prepare for your upcoming USMLE exam, it’s helpful to go “behind the scenes” to see how the NBME instructs their writers to write question items. Having insight into this process will help you better approach questions on your actual exam.
A good question must satisfy two basic criteria. First, the question item must address important and relevant content. Second, the question must be well structured. It’s nearly impossible to assess one’s proficiency on a high-yield topic if the question item is not well-structured.
Technical item flaws are what cause questions to be considered as not well-structured. These flaws are split into two buckets: issues related to testwiseness and issues related to irrelevant difficulty. Testwiseness is defined as “any skill which allows a student to choose the correct answer on an item without knowing the correct answer.” Test takers with high levels of testwiseness are often able to determine the correct answer by using unintentional clues that are in the question. Let’s take a look some of these technical item flaws that can help you become more testwise.
Grammatical cues: one or more distractors don’t follow grammatically from the stem.
In the example below, the term “intravenous administration of” should trigger the elimination of choices A and C, leaving just three choices to narrow down from.
A 60-year-old man is brought to the emergency department by the police, who found him lying unconscious on the sidewalk. After ascertaining that the airway is open, the first step in management should be intravenous administration of :
A. examination of cerebrospinal fluid
B. glucose with vitamin B1 (thiamine)
C. CT scan of the head
Logical cues: a subset of the options are collectively exhaustive
In the example below, choices A, B and C collectively cover all options (poor, middle class, rich and all), so that effectively eliminates D and E right away.
A. equally distributed among the social classes
B. overrepresented among the poor
C. overrepresented among the middle class and rich
D. primarily an indication of psychosexual maladjustment
E. reaching a plateau of tolerability for the nation
Absolute terms: terms such as “always” or “never” are used in options
In the next example, a testwise student would eliminate choices C and D based on the fact that both use the absolute term “never”. It’s less likely that a choice containing absolute terms would be true versus one that is worded less absolutely.
In patients with advanced dementia, Alzheimer’s type, the memory defect
A. can be treated adequately with phosphatidylcholine (lecithin)
B. could be a sequela of early parkinsonism
C. is never seen in patients with neurofibrillary tangles at autopsy
D. is never severe
E. possibly involves the cholinergic system
Word repeats: a word or phrase is included in the stem and in the correct answer
In the example below, the word “unreal” is mentioned. Subsequently, “derealization” is answer choice C, which would result in a testwise student choosing it as the correct answer. Also be on the lookout for veiled but similar phrases like “cardio” and “heart palpitations”.
A 58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called
D. focal memory deficit
E. signal anxiety
Convergence strategy: the correct answer includes the most elements in common with the other options
This strategy analyzes the words used in question choices. The theory is that the question writer will mention correct elements more than incorrect ones. Below we can see that anionic is mentioned once, and cationic and uncharged form are mentioned twice. Moving to the second element of the answer choices we see that “acting from inside the nerve membrane” is mentioned 3 times whereas outside the membrane is only mentioned twice. A testwise student would look at these combinations and choose between options B and D.
Local anesthetics are most effective in the
A. anionic form, acting from inside the nerve membrane
B. cationic form, acting from inside the nerve membrane
C. cationic form, acting from outside the nerve membrane
D. uncharged form, acting from inside the nerve membrane
E. uncharged form, acting from outside the nerve membrane
If you are aware of all of these signals on exam day, you’ll benefit greatly from increased testwiness.