The IBM Jeopardy Challenge, Man vs. Machine

Saturday, 05 March 2011 16:58 Ayman Abouseif
Print PDF

In 2007, IBM scientist David Ferrucci and his team embarked on the challenge of building a computer that could beat the best players of the popular US TV quiz show Jeopardy!, a 25 year old trivia game in which contestants are given clues in categories ranging from academic subjects to pop culture and must ring in with responses that are in the form of questions.

In some sense, the project was a follow-up to Deep Blue, the IBM computer that defeated chess champion Garry Kasparov in 1997. Although a TV quiz show may seem to lack the gravitas of the classic game of chess, the task was in many ways much harder. It wasn’t just that the computer had to master straightforward language, it had to master humor, nuance, puns, allusions, and slang—a verbal complexity well beyond the reach of most computer programs. Meeting that challenge was about much more than just a Jeopardy! championship. The work of Ferrucci and his team illuminates both the great potential and the severe limitations of current computer intelligence—as well as the capacities of the human mind. Although the machine they created was ultimately dubbed “Watson” (in honor of IBM’s founder, Thomas J. Watson), to the team that painstakingly constructed it, the game-playing computer was known as Blue J.

Last month, Watson ultimately proved to be too much for the humans in Jeopardy. Ken Jennings and Brad Rutter, two of the most successful Jeopardy players, put up a spirited fight, but couldn’t handle the IBM supercomputer. The two day totals highlighted the extent of the victory for Watson which had $77,147 while Jennings had $24,000 and Rutter  had $21,600. In the end, Watson was a natural on Jeopardy. It (I am tempted to say he!) was even likeable with his quirks, very precise wagers and a robotic voice. “Unlike us, Watson cannot be intimidated. It never gets cocky or discouraged. It plays its game coldly, implacably, always offering a perfectly timed buzz when it's confident about an answer.” Said Ken Jennings, “But there's no shame in losing to silicon, after all I don't have 2,880 processor cores and 15 terabytes of reference works at my disposal. My puny human brain, just a few bucks worth of water, salts, and proteins, hung in there just fine against a jillion-dollar supercomputer.”

But what does Watson’s landslide victory really mean? On the one hand someone might consider it to be an indication that machine is becoming smarter than man. Of course that is not the case, this is a massive machine with thousands of processes and an unbelievable amount of pre arranged and stored data designed by man. On the other hand some might consider this to be just a purpose built piece of technology that won’t go beyond the purpose it was built for which is winning the Jeopardy quiz show. That too isn’t accurate because Ferrcci’s team had to develop and build complex artificial intelligence programs that can and will be reusable in many commercial applications from healthcare to providing legal advice. "We're moving beyond Jeopardy!" said Ferrucci. "With the Watson technology, we're going to look at creating a commercial offering in the next 24 months that will help empower doctors to do higher quality decision making and diagnoses."

Despite the Jeopardy win and the promise of Watson, Ferrucci is careful to point out that his creation is still no substitute for human decision making."With Jeopardy! these are human questions written for humans, whereas all the computer has are words. It can't rely on human context to determine things, when you deconstruct this, and look at the machine, is any part of this really understanding the question? No."
Blue J’s literal-mindedness posed the greatest challenge. Finding suitable data for this machine was only the first job. Once Blue J was equipped with its source material the IBM team would have to teach the machine to make sense of those texts: to place names and facts into context, and to come to grips with how they were related to each other. Hamlet, just to pick one example, was related not only to his mother, Gertrude, but also to Shakespeare, Denmark, Elizabethan literature, and themes ranging from mortality to self-doubt, just for starters. Preparing Blue J to navigate all of these connections for virtually every entity on earth, factual or fictional, would be the machine’s true education and the team’s true miracle. The process would involve creating, testing, and fine-tuning thousands of algorithms. The final challenge would be to prepare the machine to play the game itself. Eventually, Blue J would have to come up with and formulate answers it could bet on within three to five seconds and then use its robotic arm operate the buzzer.


How does Watson “Think”

Consider one of the thousands of clues the system grappled with. Under the category Diplomatic Relations, one clue was: “Of the four countries the United States does not have diplomatic relations with, the one that’s farthest north.”

First a wave of grammar specialized algorithms analyzed the sentence identifying the nouns, verbs, direct objects, and prepositional phrases. This analysis helped to clear up doubts about specific words. In this clue concluding “the United States” referred to the country, not the Army, the economy, or the Olympic basketball team. Unlike humans, who can instantly understand a question and pursue a single answer, the computer might hedge, launching searches for hundreds of different possibilities at the same time. In this way and many others, Blue J would battle the efficient human mind with spectacular, flamboyant inefficiency.

Another set of algorithms searched for the focus, or answer type concluding that in this clue about diplomacy, “the one” evidently referred to a country. If this was the case, the universe of Blue J’s possible answers was reduced to a mere 194, the number of countries in the world. This, of course, was assuming that “country” didn’t refer to “Marlboro Country” or “country music.” Blue J had to remain flexible, because these types of exceptions often popped up in Jeopardy!

Once the clue was parsed into a question the machine could understand, the hunt commenced. Expert algorithm looked to match strings of words in the clue with similar strings maybe in some stored Wikipedia entry or in articles about diplomacy. The computer had to come up with the four countries with which the United States had no diplomatic ties. Then it had to figure out which of those four was the farthest north. A group of Blue J’s programmers had recently developed an algorithm that focused on these so-called nested clues, in which one answer lay inside another. It broke the clues into two questions, pursued two hunts for answers, and then pieced them together. It found the four countries (Bhutan, Cuba, Iran, and North Korea), checked their geographical coordinates, and came up with and formulated the answer: “What is North Korea?”

At this point, Blue J had the right answer. But the machine did not yet know that North Korea was correct, or that it even merited enough confidence for a bet. The machine would proceed to check how many of the answers matched the question type: “country.” After ascertaining from various lists that North Korea appeared to be a country, confidence in “What is North Korea?” rose further up the list. For an additional test, it would place the words “North Korea” into a simple sentence generated from the clue: “North Korea has no diplomatic relations with the United States.” Then it would see if similar sentences showed up in its data trove. If so, confidence climbs even higher. In the end, it chose North Korea as the answer to bet on.


But that is not all, elsewhere in IBM’s Almaden Research Center, Dharmendra Modha is building a simulated brain equipped with 700 million electronic neurons. Within years, he hopes to map the brain of a cat, and then a monkey, and, eventually, a human. But mapping the human brain, with its 100 billion neurons and trillions or quadrillions of connections among them, is a long-term project. With time, it might result in a bold new architecture for computing, one that could lead to a new level of computer intelligence. Perhaps then, machines would come up with their own ideas, wrestle with concepts, appreciate irony, and think more like humans.


What really differentiates IBM is not its ability to build the fastest computer or to develop the next version of Tivoli but its deep rooted ability and determination to build its leadership on investing in basic research that may not produce commercial results for many years, the type of research you would normally expect from university research centers. For many years IBM has been registering and receiving more US patents than any other company in the world.


Whether it is the hard disk or the barcode or the PC, IBM's innovations are all around us. Now you know who will make our planet smarter!

Add this page to your favorite Social Bookmarking websites
Reddit!! Mixx! Google! Live! Facebook! StumbleUpon! LinkedIn! TwitThis Joomla Free PHP
Last Updated on Saturday, 07 May 2011 11:47


0 #7 Mansour El Banhawi 2011-12-25 22:00
IBM's Watson supercomputer is about to begin work evaluating evidence-based cancer treatment options that can be delivered to the physician in a matter of seconds for assessment.

Read the whole staory here:
0 #6 Noor Ibrahim 2011-04-15 08:59
I had fun watching the quiz show on TV but I like the way this blog post describes the inner workings of Watson. Nice piece thank you
0 #5 Leila Mostafa 2011-03-31 11:43
I really love what IBM is doing here, not only because it is technically challenging and exciting but also because it can and will change what IT can do for the society .. hats off to IBM.
0 #4 Tarron D 2011-03-21 09:39
This says two things:

- IBM is not the technology leader it is for no reason or just by inertia, it is there on the top because of its incredible capabilities that go beyond its competitors.

- Computers cannot be programmed to think but they can be programmed to act like they are thinking!
0 #3 Ahmed Taher 2011-03-09 14:39
IBM has that special thing about it .. it is doing basic research and has a very long term view while retaining technology leadership on a day to day basis. Impressive.

I agree though that a more vibrant field operation in this part of the world could benefit the company immensely.
0 #2 B Khalil 2011-03-08 02:56
Nice piece covers all the angles of the topic. I must admit that it has renewed my respect for IBM research and engineering .. hope their field people get equally aggressive and take more challenges.
0 #1 Sami Matar 2011-03-05 18:49
Interesting piece but you can't portrait IBM as the one and only source of innovation in technology, companies like HP and Microsoft have made major contributions to the IT industry also.

Add comment

By commenting on our blog, you are automatically confirming that you have read and agreed to abide by our editorial policy found under About CIO2CIO - CIO2CIO Editorial Policy

Security code

Anti-spam: complete the task
  • Poll
  • Login
  • Search

Does your organization use big data?

Follow CIO2CIO on: