Whether an applicant receives a high or low score may have more to do with who else was interviewed that day than the overall strength of the applicant pool, according to new research published in Psychological Science, a journal of the Association for Psychological Science.
Drawing on previous research on the gambler fallacy, Uri Simonsohn of The Wharton School of the University of Pennsylvania and Francesca Gino of Harvard Business School hypothesized that admissions interviewers would have a difficult time seeing the forest for the trees. Instead of evaluating applicants in relation to all of the applicants who had been or would be interviewed, interviewers would only consider them in the frame of applicants interviewed on that day. This phenomenon is often referred to as “narrow bracketing.”
Much like gamblers bet on red after the wheel stops at black four times in a row, an interviewer bets on “bad” after she interviews four “goods” in a row; the difference in this case is that the interviewer controls the wheel.
If the interviewer expected that half of the whole pool would be recommended, she would avoid recommending more than half of the applicants she interviewed in a given day.
Simonsohn and Gino analyzed ten years of data from over 9000 MBA interviews to test their hypothesis.
As predicted, interviews earlier in the day had a negative impact on the assessments for the interviews that followed — if the interviewer had already given several high scores, the next score was likely to be lower. This held true even after various applicant characteristics and interview characteristics were taken into account.
As the average score for previous applicants increased by .75 (on a 1-5 scale), the predicted score for the next applicant dropped by about .075. This drop may seem small, but the effect is meaningful. An applicant would need about 30 more points on the GMAT, 23 more months of experience, or .23 more points in the assessment of the written application to make up for the drop. And the impact of previous scores grew stronger as the interviewer progressed through the day.
“People are averse to judging too many applicants high or low on a single day, which creates a bias against people who happen to show up on days with especially strong applicants,” Simonsohn and Gino observe.
Interestingly, they found that the effect was twice as large when a rating followed a set of identical scores (e.g., 4, 4, 4), compared to a set of varied scores (e.g., 4, 3, 5) with the same average.
Simonsohn and Gino were surprised by the overall strength of their findings. “We were able to document this error with experts who have been doing the job for years, day in and day out.”
They point out that these findings are relevant to many different kinds of judgments, from evaluating job candidates to approving loan applications, even choosing contestants trying out for a reality show. And because many jobs in real life involve making these subsets of judgments, the error could be more pervasive than we realize.
So, if you want to get that job, or that loan, or make it onto that reality show, you might want to make sure the strongest contenders stay home that day.