Hawk-Eye in the Crosshairs at Wimbledon Again

23 June 2008--When Roger Federer and Rafael Nadal took to Centre Court for the Wimbledon men's singles final in July 2007, the last thing they expected was a controversial line call. The tournament organizers had introduced Hawk-Eye, an automated line-call system, which its makers claim can decide whether a ball is in or out of play with an average accuracy of 3.6 millimeters, or about the width of the fuzz on the ball.

In the fourth set, Nadal asked for Hawk-Eye's judgment on a shot that looked to all and sundry as if it had landed beyond the baseline and was out. But Hawk-Eye said it had hit the line and called it in by a single millimeter. That gave Nadal the point, which he went on to convert into a three games to nil lead in the set. It was an angry Federer, however, who went on to win the match and his fifth consecutive Wimbledon title.

The incident raised many questions about the Hawk-Eye technology, installed once again at Wimbledon, for the 2008 championships, which begin today. Television replays seemed to show that the ball had landed beyond the line. And after the game, analysts asked how a machine with an average error of 3.6 mm could determine that the ball was in by 1 mm.

A team led by Harry Collins, a social scientist at Cardiff University, in Wales, is raking up the controversy again with a study into the way Hawk-Eye makes decisions. The team, which received no help from the company behind Hawk-Eye, says it has been unable to fathom how the errors on Hawk-Eye's measurements are calculated nor how they are handled by the system. ”The system should certainly be more transparent. We need to know how it works,” says Collins, who specializes in the public understanding of science.

Hawk-Eye is the brainchild of Paul Hawkins, a British computer scientist, and has been used since 2001 to provide replays of the flight of cricket balls during games (although not yet to adjudicate on umpiring decisions). Its debut in grand-slam tennis was at the 2006 US Open, and it's also used in snooker and is being tested for use in soccer.

The idea is relatively straightforward. At Wimbledon this year, Hawk-Eye will use a system of 10 cameras to photograph the ball in flight. A computer triangulates the ball's position in three-dimensional space and repeats the process for the next frame. ”It then joins the dots,” says Hawkins, who talked to IEEE Spectrum after a day on Centre Court at Wimbledon setting up the system for this year's tournament. The computer then produces an animated picture of the ball's trajectory in a virtual tennis court, showing how the ball moved in the air and where it bounced.

For many fans, players, and sports-governing bodies, Hawk-Eye has been a revelation. It collects huge volumes of data about the way athletes play their games, and this has allowed a new level of statistical analysis to inform players and viewers alike. It also shows in unprecedented detail just how a ball bounces on a tennis court.

”A ball can skid for up to 10 centimeters when it hits the surface of the court,” says Hawkins. ”[Television] replays just don't show this kind of detail.” A skid was the crux of the Federer controversy.

In his own analysis of the Federer-Nadal call, which is posted on the Hawk-Eye Innovations Web site, Hawkins shows the three television frames used by TV pundits to analyze the call. The first shows the ball just before it touches the ground, the second shows it in contact with the ground but beyond the line, and the third shows it after it has bounced.

What these images do not show is how the ball skidded while it was in contact with the ground, says Hawkins. What actually happened according to Hawk-Eye is that the ball made contact with the ground on the line, albeit by a single millimeter, skidded along the surface, and then left the ground. The one TV frame showing the ball on the ground froze the action at a single instant during the skid, after it had moved beyond the line.

But how can Hawk-Eye make such close calls when its average accuracy is only 3.6 mm? Hawkins says that the system is set up so that it is most accurate for calls when the ball hits the back edge of the line, exactly where the Federer-Nadal incident occurred. The farther the ball is from the back of the line, the less accurate the system becomes and the less accurate it needs to be. So while the average error is 3.6 mm, Hawkins says the error for balls that land at the back of the line is much lower.

Hawkins fiercely disagrees with Collins's claim that his system lacks transparency. He says he is happy to explain how the system works and to talk through the way errors are handled, within the bounds of commercial common sense. Releasing too much information might give an advantage to his competitors, he says. Hawkins says he turned down a request from the Cardiff team to explain the technology because he was pressed for time.

In any case, he points out that the International Tennis Federation (ITF), the world governing body of tennis, has carried out thousands of tests on Hawk-Eye, measuring its accuracy with a high-speed TV camera, which records images at the rate of 2000 frames per second. The ITF's conclusion is that Hawk-Eye's accuracy falls well within the 5 mm limit it has set for such systems.

Nevertheless, Collins is adamant that there is a way in which Hawk-Eye is pulling the wool over viewers' eyes. ”Hawk-Eye attempts to take the element of uncertainty out of umpiring decisions, and the truth is that this cannot and should not be done. There will always be errors,” he says. Hawk-Eye presents its decisions as if it were 100 percent certain, but in reality that just isn't possible, he says. The public, says Collins, would be better served by a system that gave its result with error bars so that people could make up their own minds.

But to Hawkins, removing uncertainty is exactly what everybody involved in sports wants. ”An airline pilot doesn't say, ’We have a 99.996 percent chance of landing safely at Heathrow within 30 minutes,'” he says. ”And you couldn't have a trophy that said, ’We are 99.76 percent certain that Roger Federer won.'”

It's hard to imagine sports fans disagreeing.