Computing

Fujitsu Plans to Support Professional Judges With Lidar and AI at Gymnastics Meets

Fujitsu has developed a judging system that can objectively score a routine based on the angles of a gymnast’s joints

A screenshot shows the Fujitsu 'skeleton recognition technology' at work, with an outline of a gymnast during their routine.
Image: Fujitsu

Perhaps more than any other sport, gymnastics is both an art and a science. As an athlete executes their carefully-rehearsed flips and leaps, judges look to award or deduct points based on the precise position of the gymnast’s body.

To ensure a fair competition, organizers typically rely on highly trained human eyes. Now, an international gymnastics association has agreed to incorporate lidar-based technology developed by Fujitsu into future competitions to help judges assess performances. The organization plans to fully automate scoring by 2020.

Gymnastics’ rigorous scoring rules require judges to closely evaluate the angles of gymnasts’ joints, such as their knees and elbows, as athletes move through their routines. But “human eyes cannot accurately measure the bending angle of joints,” said Shoichi Masui, senior expert of Fujitsu’s gymnastics project. As a result, scores can “fluctuate for identical movements,” he says, which may look like bias or favoritism even though judges are trying to be fair.

The Fujitsu judging support system aims to take that variability out of the equation by using lidar technology to track the angles of athletes’ joints during their competitions. A lidar set up on the sidelines bounces light off the athletes and measures the reflection to create 3-D images that show the contours of the gymnast’s body. Then an AI system—what Fujitsu calls “skeleton recognition technology”—uses those images to determine the angles of various joints: elbows, knees, and shoulders, for example.

That joint-angle information can be delivered to judges immediately after each gymnast’s exercise, to help judges determine the athlete’s final score. The new system’s target error is less than ±1 centimeter, Masui said. “Our final goal is to automate judges’ scores by adding skill recognition and deduction judgement to this system,” he added.

Fujitsu’s judging support system combines lidar with artificial intelligence to determine the angles of a gymnast’s joints during their routine.Image: Fujitsu

First, the joint-recognition part of the system will be tested on the international stage. On 20 November, the International Gymnastics Federation (FIG) announced that it would trial the system at an upcoming World Cup event and aim to roll it out during competition at the 2019 World Championships.

By 2020, FIG wants to automate judges’ scores altogether, though Masui says the plan is not to eliminate human judges, but rather to support them with this new system. It would be a major move for the world of gymnastics, adding a technology more complex—and perhaps more significant—than systems like the line-calling system in tennis.

It would also be the culmination of a project inspired by a joke three years ago. After a 2015 meeting with Fujitsu in 2015, FIG president Morinari Watanabe remarked, “In the Tokyo Olympic Games in 2020, robots will be scoring gymnastics!”

Challenge accepted, the Fujitsu team thought. They began toying around with the potential for 3D-sensing technology in sports, connecting lidar with AI systems to see what was possible.

Fujitsu plans to use lidar sensors to record gymnasts’ movements during meets.Photo: Fujitsu

After developing an early version of the system, Fujitsu trained and improved it by setting it up to assess performances of elite Japanese gymnasts through a partnership with the Japan Gymnastics Association. (Masui said the AI is built on “a deep learning and optimization method” but could not disclose further details.)

The team conducted its first field test at 2017 World Championships in Montreal, where challenges quickly became apparent. It was more difficult than expected to find appropriate places to set the sensors, Masui said: The placements not only had to meet specific range and viewing angles required for the Fujitsu system to work, but also had to avoid interfering with the television broadcast and official scoring activities.

But they figured it out in Montreal and completed technical verification a year later at the World Championship in Doha that began in late October 2018.

Overall, Masui concluded, “the judging support system offers more precise and fair judgement than ever before.” And Fujitsu sees a large market in sports business” for such technology, he added.

Beyond the judging support, FIG plans to leverage Fujitsu’s system to help gymnasts and their fans alike. Athletes can use the data to train and track progress. And the images collected during competitions can be displayed on boards in the arena or on smartphone apps, giving gymnastics fans a deeper understanding of every twist and turn.

IEEE Spectrum
FOR THE TECHNOLOGY INSIDER

Follow IEEE Spectrum

Support IEEE Spectrum

IEEE Spectrum is the flagship publication of the IEEE — the world’s largest professional organization devoted to engineering and applied sciences. Our articles, podcasts, and infographics inform our readers about developments in technology, engineering, and science.