The Stanford and University of California, Berkeley Quiz Bowl teams matched wits Thursday evening against each other and a formidable opponent–the IBM Watson “supercomputer,” capable of answering questions posed in natural language.
The Jeopardy-style competition demonstrated the technology behind the powerful supercomputer “Watson,” named after IBM founder Thomas J. Watson.
Stanford’s team was represented by Nico Martinez ’07, J.D. ’13, Benji Nguyen ’15 and Bill Rowan, a computer science graduate student.
“My assumption is that Watson will beat us,” Rowan said before the competition. According to Rowan, the student teams had an advantage “parsing and understanding the questions,” while Watson held the advantage in buzzer speed, “which turns out to be crucial.”
“It’s David and Goliath,” said Jack Dubie ‘11, a computer science graduate student who helped organize the event.
“The stakes could not be higher,” Rowan joked. “It’s the future of the human race.”
The game, moderated by Todd Crain–who has moderated over one hundred matches between Watson and humans–was preceded by a presentation by Eric Brown, an IBM research scientist.
“It’s not about the game,” Brown said of the long-term goals following Watson’s early success. “It’s about the technology and what we’re going to do with it.”
“We’re all familiar with the expert as a librarian,” Brown said, adding that librarians are not always accessible and do not have the same deep understanding of a wide range of content that Watson achieves.
According to Brown, Watson offers precise answers, a fast response time and “accurate confidences” and “consumable justifications” for those answers.
Brown described future applications of Watson’s automatic open domain question answering, from business and commercial applications to healthcare innovations in differential diagnosis. Overall, he described the goal of the developing technology as striving “to help people make better decisions.”
For now, Watson is just very good at winning at Jeopardy. After testing against past Tournament of Champions contestants during a demonstration stage, Watson played and won on national television this past February against Ken Jennings, the longest championship streak record holder, and Brad Rutter, the show’s biggest money winner.
Brown described the complexity of natural language and the methods Watson, which uses 90 IBM power servers, employs to parse questions, such as determining key words and the desired answer form.
“I’m completely unbiased here,” Brown said before the competition. “I don’t care who comes in second.”
“I smell fear,” Crain said, as he welcomed the teams to the stage.
After a series of questions dominated completely by Watson, Crain added, “You’re off to a bad start.”
After two categories of single Jeopardy and with both human teams expressing slight frustration, the Berkeley team finally managed to beat Watson to the buzzer.
IBM scientists have been able to cut down Watson’s response rate to the average Jeopardy contestant time frame of fewer than three seconds, combined with high precision and number of questions answered. If the computer does not find an answer that crosses its confidence threshold, it will not buzz in. During the game, audience members were able to see Watson’s top three answer choices and corresponding confidence percentages.
While both teams managed to enter the game during the first round, Watson took a strong lead and benefited from the “Daily Double.” Stanford, after a detour in the red, trailed Berkeley.
During the second round, one clear shortcoming of the computer became clear. After the Berkeley team answered a question incorrectly, Watson repeated the same response.
At another point, Watson had a delay in choosing the next category after answering a question correctly.
“Remember, its Watson supercomputer but Microsoft software,” said Bernie Meyerson, vice president of innovation and global university relations for IBM.
Going into final Jeopardy, despite successes for both student teams, the score stood at $8,800 for Stanford, $16,400 for Berkeley and $33,300 for Watson.
All three teams answered the final question on 20th century thinkers correctly (Jean-Paul Sartre was the answer) and the final order remained the same.
Following the competition, Brown responded to questions and described some of Watson’s tools, such as a pun filter and a vulgarity filter. Audience members posed questions about the fairness of the game, particularly about buzzer speed and whether the competition was biased.
“I would say that understanding natural language is rigged in the humans’ favor,” Brown said.