The December 2022 issue of IEEE Spectrum is here!

Close bar

Sony Builds AI Into a CMOS Image Sensor

This smart image sensor uses digital signal processing of machine-learning algorithms to decode what it "sees"

2 min read
Sony's new smart image sensor, in bare and packaged form
Photo: Sony

Sony today announced that it has developed and is distributing smart image sensors. These devices use machine learning to process captured images on the sensor itself. They can then select only relevant images, or parts of images, to send on to cloud-based systems or local hubs.

This technology, says Mark Hanson, vice president of technology and business innovation for Sony Corp. of America, means practically zero latency between the image capture and its processing; low power consumption enabling IoT devices to run for months on a single battery; enhanced privacy; and far lower costs than smart cameras that use traditional image sensors and separate processors.

Sony’s San Jose laboratory developed prototype products using these sensors to demonstrate to future customers. The chips themselves were designed at Sony’s technology center in Atsugi, Japan. Hanson says that while other organizations have similar technology in development, Sony is the first to ship devices to customers.

Sony builds these chips by thinning and then bonding two wafers—one containing chips with light-sensing pixels and one containing signal-processing circuitry and memory. This type of design is possible only because Sony is using a back-illuminated image sensor. In standard CMOS image sensors, the electronic traces that gather signals from the photodetectors are laid on top of the detectors. This makes them easy to manufacture but sacrifices efficiency, because the traces block some of the incoming light. Back-illuminated devices put the readout circuitry and the interconnects under the photodetectors, adding to the cost of manufacture.

“We originally went to backside illumination so we could get more pixels on our device,” says Hanson. “That was the catalyst to enable us to add circuitry; then the question was what were the applications you could get by doing that.”

Example of real time object tracking in video frames.Sony’s smart image processor can identify and track objects, sending data on to the cloud only when it spots an anomaly.Images: Sony

Hanson indicates that the initial applications for the technology will be in security, particularly in large retail situations requiring many cameras to cover a store. In this case, the amount of data being collected quickly becomes overwhelming, so processing the images at the edge would simplify that and cost much less, he says. The sensors could, for example, be programmed to spot people, carve out the section of the image containing the person, and  send only that on for further processing. Or, he indicates, they could simply send metadata instead of the image itself—say, the number of people entering a building. The smart sensors can also track objects from frame to frame as a video is captured—for example, packages in a grocery store moving from cart to self-checkout register to bag.

Remote surveillance, with the devices running on battery power, is another application that’s getting a lot of interest, Hanson says, along with manufacturing companies that mix robots and people looking to use image sensors to improve safety. Consumer gadgets that use the technology will come later, but he expects developers to begin experimenting with samples.

The Conversation (0)

Europe Expands Virtual Borders To Thwart Migrants

Our investigation reveals that Europe is turning to remote sensing to detect seafaring migrants so African countries can pull them back

14 min read
A photo of a number of people sitting in a inflatable boat on the water with a patrol ship in the background.

Migrants in a dinghy accompanied by a Frontex vessel at the village of Skala Sikaminias, on the Greek island of Lesbos, after crossing the Aegean sea from Turkey, on 28 February 2020.

ASSOCIATED PRESS

It was after midnight in the Maltese search-and-rescue zone of the Mediterranean when a rubber boat originating from Libya carrying dozens of migrants encountered a hulking cargo ship from Madeira and a European military aircraft. The ship’s captain stopped the engines, and the aircraft flashed its lights at the rubber boat. But neither the ship nor the aircraft came to the rescue. Instead, Maltese authorities told the ship’s captain to wait for vessels from Malta to pick up the migrants. By the time those boats arrived, three migrants had drowned trying to swim to the idle ship.

The private, Malta-based vessels picked up the survivors, steamed about 237 kilometers south, and handed over the migrants to authorities in Libya, which was and is in the midst of a civil war, rather than return to Malta, 160 km away. Five more migrants died on the southward journey. By delivering the migrants there, the masters of the Maltese vessels, and perhaps the European rescue authorities involved, may have violated the international law of the sea, which requires ship masters to return people they rescue to a safe port. Instead, migrants returned to Libya over the last decade have reported enslavement, physical abuse, extortion, and murders while they try to cross the Mediterranean.

If it were legal to deliver rescued migrants to Libya, it would be as cheap as sending rescue boats a few extra kilometers south instead of east. But over the last few years, Europe’s maritime military patrols have conducted fewer and fewer sea rescue operations, while adding crewed and uncrewed aerial patrols and investing in remote-sensing technology to create expanded virtual borders to stop migrants before they get near a physical border.

Keep Reading ↓Show less
{"imageShortcodeIds":["29177566"]}