Emmy Award Winner’s Algorithms Bring High-Quality Video to Your TV

Alan Conrad Bovik’s passion for science fiction inspired him to pursue a career in engineering. His favorite sci-fi authors when he was young were Arthur C. Clarke, who penned 2001: A Space Odyssey, and Isaac Asimov, author of the Foundation series. Bovik says they wrote from a “very scientific point of view”—which made him want to help develop aerospace technology that would send humans “to other worlds.”

But he decided to study nuclear engineering in school—which then seemed like the future of energy. He discovered, however, that he didn't like the subject because it “required too much chemistry and memorization,” he says with a laugh. When he took a course in computer programming, he fell in love with it and ended up changing his major to computer engineering.

While pursuing a Ph.D. at the University of Illinois Urbana-Champaign, Bovik found his calling: digital image processing. It is a technology, he says, that back then didn’t have many applications and seemed like something out of a sci-fi novel.

The IEEE Fellow has since done groundbreaking work in the field.

He created theoretical models of how people will perceive the quality of an image or video after it has been compressed. He has used the models to develop algorithms that automatically measure picture quality.

Programs he helped develop include the Structural SIMilarity (SSIM) index, the MOtion-tuned Video Integrity Evaluation (MOVIE) index, visual information fidelity algorithms, and the Video Multimethod Assessment Fusion (VMAF) metric. His video quality prediction technology is used by streaming services, broadcasting companies, movie studios, and technology companies worldwide.

Bovik has received several awards for his work. The Academy of Television Arts and Sciences honored him in 2015 with a Primetime Emmy for Outstanding Achievement in Engineering Development and with a 2020 Technology and Engineering Emmy. IEEE recognized Bovik with this year’s Edison Medal “for pioneering high-impact scientific and engineering contributions leading to the perceptually optimized global streaming and sharing of visual media.” The award is sponsored by Samsung Electronics Co.

“The IEEE Edison Medal is the most important award I’ve received,” he says, “because of its history, the amazing past recipients, and because it’s from my peers.”

For nearly 40 years, he has been a professor of electrical and computer engineering at the University of Texas Cockrell School of Engineering, in Austin. He serves as director of Cockrell’s Laboratory for Image and Video Engineering, where graduate students and faculty members develop psychometric databases, theories, and algorithms to advance modern perceptual video processing technologies.

Bovik says he plans to remain at the university for another 40 years because “being an educator and video research is the best.”

“I didn't really know how much I would enjoy the academic experience until I started teaching,” he says.

Developing the SSIM index

At the University of Illinois, Bovik took a class on digital image processing taught by Thomas Huang, a professor of electrical and computer engineering. The IEEE Life Fellow was a pioneer in computer vision, image processing, and image compression.

Bovik enjoyed the course so much that he joined Huang’s Image Formation and Processing Group, part of the university’s Beckman Institute for Advanced Science and Technology. The group conducted research in image and video coding, computer vision, pattern recognition, and machine learning. The professor became one of Bovik’s doctoral advisors. Bovik’s research focused on nonlinear statistical methods for image analysis and enhancement.

Huang suggested that Bovik consider applying for a job at the University of Texas after he graduated because he thought “it had a great future.” Bovik took his mentor’s advice and moved to Texas in 1984 to work for the Austin school as an assistant professor.

Bovik soon became interested in visual neuroscience and how it could be applied to video engineering. He spoke with several visual neuroscientists and visual psychologists to gain a better understanding of the brain’s visual cortex, which receives, integrates, and processes visual information. Bovik and his team of researchers created mathematical models of the visual cortex as well as the retina, which converts light into neural signals and sends them to the brain.

“The IEEE Edison Medal is the most significant award I’ve received because it’s an award from my peers.”

In the early 2000s, he began studying how people perceive distortions—such as blurring, resizing, and noise—in pictures and videos, as video compression was becoming important because of digital television.

Video images take up a lot of space, and because bandwidth is limited, videos generally need to be compressed before being distributed. But the process can distort the images, causing people to perceive them as blurry or jagged. At the time, there was no program that could accurately predict how badly the images would be distorted before they were distributed, Bovik says, because the algorithms in use didn’t account for how people perceive the images.

Using the mathematical models he and his team previously developed, they created computational algorithms that accurately gauged the way humans would see distortions in compressed images.

Bovik with IEEE President Ray Liu [left] and President-Elect Saifur Rahman at the IEEE Honors Ceremony held in San Diego in May.Alan Bovik

Bovik says the key to creating the algorithms was understanding how people process visual information when they first see an image.

“If an image is blurry, you don't have to think about it,” he says. “You sense that it’s blurry instantly.”

In 2001 he and his team developed SSIM. The program predicts how people will perceive a compressed image on a screen by measuring three factors: luminance masking (distortion visibility depending on an image’s brightness), contrast masking (distortion visibility depending on an image’s texture), and structural similarity.

If SSIM predicts an image will appear distorted, the errors can be fixed before the image is shared.

The algorithm has since become part of the International Telecommunication Union’s H.264 video coding reference software standard. The standard allows streaming services, broadcasting companies, and technology companies to include the program in their products and services.

Preparing for the metaverse

Bovik continues to develop algorithms for image processing and video quality assessment. He and his team are working with Meta’s Reality Labs to develop an algorithm to be used in the metaverse, a simulated “embodied internet” experienced in 3D, as defined by Meta founder Mark Zuckerberg. Virtual and augmented reality will let users feel as if they can move through space at will and pick up objects with ease, Zuckerberg says.

“Back in 1991, when we formed Transactions, not many people understood image processing. It seemed like something out of a science-fiction novel. But things have changed, and now it’s everywhere.”

“Nobody can predict how quickly the metaverse will be upon us,” Bovik says, “but it introduces all kinds of new challenges that are related to visual neuroscience.” The metaverse will require high-resolution images, he says, because the viewer’s eyes are centimeters away from the augmented-reality technology displays.

By developing perception-based algorithms, Bovik says, he hopes to “perceptually optimize the way videos are transmitted, compressed, and displayed in VR and AR.”

Image processing at IEEE

Bovik says he joined IEEE as a student member so he could access the organization’s journals. Maintaining that access is why he continues to be a member, he says.

He frequently attends IEEE conferences, where he has met engineers who have become his lifelong friends and mentors, he says. He often brings his students along. “I used to call the opportunity to go to a conference with me the Alan Bovik Motivational System,” he says, laughing.

Because he believed that image processing was going to have “a great feature,” he says, he helped launch the IEEE International Conference on Image Processing and the IEEE Transactions on Image Processing. He is the publication’s longest-serving editor-in-chief.

“When I helped form Transactions in 1991, not many people understood the future applications of image processing,” Bovik says. “The technology seemed like something out of a science-fiction novel, but now it’s everywhere.”