This Lens-less Camera Is Built Specially for AI and Computer Vision Programs

Cameras used to be their own devices with lenses and film and trips to the drug store to get the pictures developed. Then, they disappeared into phones, tablets, laptops, and video game consoles. Now, it appears that cameras could someday become as inconspicuous as a pane of glass.

According to new research, a photodetector pressed up against the edge of a window can detect the reflections that bounce around inside the glass—like light signals traversing a fiber optic cable. And some clever processing of those tiny trickles of detected light enables the pane of glass to act like a giant camera lens.

The resulting grainy images (think of pixelated, somewhat distorted and lower-resolution cousins to shots taken by first-generation smartphones) won’t compete anytime soon with conventional cameras for picture quality. But for the purposes of many computer vision programs, a window pane or a piece of car windshield may provide all the resolution that an image processing algorithm or neural network might need.

A fair number of images captured by cameras today are never seen by the human eye, says Rajesh Menon, associate professor of electrical and computer engineering at the University of Utah. They’re seen only by algorithms processing security camera feeds or videos from a factory floor, or autonomous vehicle image sensors. And the number of images never seen by humans is increasing.

So, Menon asks, “If machines are going to be seeing these images and video more than humans, then why don’t we think about redesigning the cameras purely for machines? Take the human out of the loop entirely, and think of cameras purely from a non-human perspective.”

In other words, computer vision algorithms don’t always need the high resolutions and image fidelity that a discerning human eye demands. Plenty of information might still be extracted (at lower cost and with a smaller gadget footprint) from the lower-quality images taken by Menon and his co-author Ganghun Kim’s “see-through lens-less camera.”

Menon and Kim’s technology, for which they’ve already applied for a patent on behalf of the university, begins with a pane of glass or plastic. Nothing special is required for the visual medium itself, Menon says. They used a sheet of plexiglass because it was easy to work with and cut.

They attached an off-the-shelf photodetector (640 by 480 pixels offering 8 bit resolution) to an edge of the plastic sheet that they’d smoothed and prepared to interface with the imaging device. They, the put reflective tape around the rest of the perimeter of the pane of plexiglass. They can do the imaging without the tape, Menon says, but this trick boosts the signal-to-noise ratio.

They kept their field of view simple for this proof-of-concept implementation. The object they set in front of the pane was an array of 32 by 32 LED lights. Then, they looked at the signal arriving at the photodetector when each of the 1,024 lights was individually illuminated.

So any arbitrary image from the LED array would, at least in a first approximation, just be a linear combination of the signals from each of the individual LED lights that had been illuminated.

Menon says for this project, they developed traditional signal processing algorithms that could reconstruct the image from the signal received at the photodetector. They called this step the “inverse problem,” because their algorithm was taking a complex and jumbled signal and driving it backward to discover the object(s) that could have generated the photons their detector detected.

“We’re detecting a distribution [of photons] in space that corresponds to a particular object,” he says. “As humans, we like to see one-to-one maps. That’s exactly what a lens does. Here we have a one-to-many map. Which is why we have to solve the inverse problem.”

That’s also why these windowpane “cameras” could be particularly well suited for programs that rely on computer vision. The image quality and resolvable information may be good enough for computer vision, but is not yet (and perhaps never will be) ready to replace the traditional lens-based camera for images that humans view.

Menon also says his team is now developing a machine learning algorithm to study more complex images—such as hand-written numbers that could be detected and resolved into the numerical digits they represent.

Menon says one of the first applications for this technology could be in virtual reality and augmented reality goggles. A goggle’s image generation and image display hardware is already bulky enough in these headsets that eye-tracking cameras can be a bothersome addition. So what if a version of this see-through lens-less camera could provide information good enough to track the user’s eye movement?

It’s ironic of course that a breakthrough might come in the form of a technology that suffers a drastic reduction in quality from the state of the art today. But, says Menon, perhaps the most substantial leap forward is the mind-shift in thinking about redesigning technology that’s “good enough” for AI and image processing systems. Because like a fly’s eye, what matters in the AI world isn’t so much the high-quality of a single data source but rather the proliferation of data sources.

Which is why, at least to a computer vision algorithm, perhaps a camera and a pane of glass may start looking a lot more like each other in the years ahead.