Nature, published online: 11 February 2005
Emma Marris
From song contests to figure-skating, the order of contestants biases decisions.
If you're thinking of going speeddating this Valentine's Day, take note. In certain contests, candidates who take their turn at the end of a sequence are consistently ranked higher than those at the beginning.
So says a researcher who has evaluated the judging of music competitions and figure-skating contests. The bias is evident regardless of whether the judges score each contestant immediately, or rank them all at the end.
Wändi Bruine de Bruin, of Carnegie Mellon University in Pittsburgh, Pennsylvania, noticed that most experiments in decision science (a relatively new, interdisciplinary field that probes the mysteries of how humans make choices) present all of the options to their subjects simultaneously. But in the real world, when choosing an apartment or meeting a stream of suitors, for example, one often sees the alternatives in sequence.
"Just from my own experience of looking for apartments or jobs, I know you don't get all the information at the same time," Bruine de Bruin says. She studied the judging of World and European Figure Skating Championships which are scored step-by-step, with judges awarding points to each skater directly after their routine.
She also looked at the Eurovision Song Contest, an annual jamboree in which European countries pit their finest pop songs against one another. Before 1975, judges handed out all the scores together at the end of the competition, but the event's organizers have since switched to the more televisually dramatic step-by-step method.
Late comers
Bruine de Bruin found that scores climbed as the competitions went on, with late-appearing singers and skaters getting higher marks on average. Even those early Eurovision judges who gave out marks to everyone at the end preferred later acts. Score decisions nominally made at the end of the night may actually have been made step-by-step in each judge's head, Bruine de Bruin concludes. Her results are presented in the journal Acta Psychologica1.
The reasons for this trend are still unclear. Bruine de Bruin suggests that it may be due to 'direction of comparison effects', which arise as judges focus heavily on the differences between performers2. If contestants are all of a high standard, as in the figure-skating championships, any unique features will probably be considered positive, and the contestant will seem better than previous ones. "You ask yourself, did the last one have this neat new thing?" Bruine de Bruin explains.
Negative points
Of course, one might question whether the Eurovision Song Contest is an event in which all the contestants are of an impeccable standard. And Dan Ariely, a decision scientist at the Massachusetts Institute of Technology in Cambridge, Massachusetts, warns that although the positive unique features in each performance may push scores up in skilled competitions, other situations are rife with negative features, and so scores may well slide downhill over the sequence. "It is a complex picture. I expect that her results would not hold in more general circumstances where negative things are more salient," he argues.
Nevertheless, it will be food for thought for those aiming to make rational decisions or arrange fair contests. Scientists routinely present the options in their experiments in various orders to factor out the effects of position, but speed-dating is enough of a logistical effort without attempting to have everyone meet each other again in a different order.
From wine-tastings to the dating game, the standard procedure is to assign positions in the sequence randomly. But this doesn't eliminate bias, it merely applies the bias randomly. Bruine de Bruin doesn't see a way out. "The sad thing is that I don't know if these order effects can be prevented," she says.
References
1. Bruine de Bruin W. Acta Psychologica 118, 245 – 260 (2005).
2. Bruine de Bruin W. & Keren G. Organizational Behavior and Human Decision Processes 92, 91 – 101 (2003).
Emma Marris
From song contests to figure-skating, the order of contestants biases decisions.
If you're thinking of going speeddating this Valentine's Day, take note. In certain contests, candidates who take their turn at the end of a sequence are consistently ranked higher than those at the beginning.
So says a researcher who has evaluated the judging of music competitions and figure-skating contests. The bias is evident regardless of whether the judges score each contestant immediately, or rank them all at the end.
Wändi Bruine de Bruin, of Carnegie Mellon University in Pittsburgh, Pennsylvania, noticed that most experiments in decision science (a relatively new, interdisciplinary field that probes the mysteries of how humans make choices) present all of the options to their subjects simultaneously. But in the real world, when choosing an apartment or meeting a stream of suitors, for example, one often sees the alternatives in sequence.
"Just from my own experience of looking for apartments or jobs, I know you don't get all the information at the same time," Bruine de Bruin says. She studied the judging of World and European Figure Skating Championships which are scored step-by-step, with judges awarding points to each skater directly after their routine.
She also looked at the Eurovision Song Contest, an annual jamboree in which European countries pit their finest pop songs against one another. Before 1975, judges handed out all the scores together at the end of the competition, but the event's organizers have since switched to the more televisually dramatic step-by-step method.
Late comers
Bruine de Bruin found that scores climbed as the competitions went on, with late-appearing singers and skaters getting higher marks on average. Even those early Eurovision judges who gave out marks to everyone at the end preferred later acts. Score decisions nominally made at the end of the night may actually have been made step-by-step in each judge's head, Bruine de Bruin concludes. Her results are presented in the journal Acta Psychologica1.
The reasons for this trend are still unclear. Bruine de Bruin suggests that it may be due to 'direction of comparison effects', which arise as judges focus heavily on the differences between performers2. If contestants are all of a high standard, as in the figure-skating championships, any unique features will probably be considered positive, and the contestant will seem better than previous ones. "You ask yourself, did the last one have this neat new thing?" Bruine de Bruin explains.
Negative points
Of course, one might question whether the Eurovision Song Contest is an event in which all the contestants are of an impeccable standard. And Dan Ariely, a decision scientist at the Massachusetts Institute of Technology in Cambridge, Massachusetts, warns that although the positive unique features in each performance may push scores up in skilled competitions, other situations are rife with negative features, and so scores may well slide downhill over the sequence. "It is a complex picture. I expect that her results would not hold in more general circumstances where negative things are more salient," he argues.
Nevertheless, it will be food for thought for those aiming to make rational decisions or arrange fair contests. Scientists routinely present the options in their experiments in various orders to factor out the effects of position, but speed-dating is enough of a logistical effort without attempting to have everyone meet each other again in a different order.
From wine-tastings to the dating game, the standard procedure is to assign positions in the sequence randomly. But this doesn't eliminate bias, it merely applies the bias randomly. Bruine de Bruin doesn't see a way out. "The sad thing is that I don't know if these order effects can be prevented," she says.
References
1. Bruine de Bruin W. Acta Psychologica 118, 245 – 260 (2005).
2. Bruine de Bruin W. & Keren G. Organizational Behavior and Human Decision Processes 92, 91 – 101 (2003).
