Photo: Adafruit Industries
I want to find out what happens when we bring machine learning to cheap, robust devices that can have all kinds of sensors and work in all kinds of environments. And I want you to help. The kind of AI we can squeeze into a US $30 or $40 system won’t beat anyone at Go, but it opens the door to applications we might never even imagine otherwise.
Specifically, I want to bring machine learning to the Arduino ecosystem. This has become recently possible thanks to improvements in hardware and software.
On the hardware side, Moore’s Law may be running out of steam when it comes to cutting-edge processors, but the party’s not over when it comes to microcontrollers. Microcontrollers based on 8-bit AVR processors dominated the Arduino ecosystem’s early years, for example, but in more recent years, embedded-chip makers have moved toward more powerful ARM-based chips. We can now put enough processing power into these cheap, robust devices to rival desktop PCs of the mid 1990s.
On the software side, a big step has been the release of Google’s TensorFlow Lite, a framework for running pretrained neural networks—also known as models—on so-called edge devices. Last April, IEEE Spectrum’s Hands On column looked at Google’s Coral Dev Board, a single-board computer that’s based on the Raspberry Pi form factor, designed to run TensorFlow Lite models. The Coral incorporates a dedicated tensor processing unit and is powerful enough to process a live video feed and recognize hundreds of objects. Unfortunately for my plan, it costs $150 and requires a hefty power supply, and its bulky heat sink and fan limit how it can be packaged.
But fortunately for my plan, Pete Warden and his team have done amazing work in bringing TensorFlow Lite to chips based on ARM’s Cortex family of processors. This was great to discover, because at my open-source hardware company, Adafruit Industries, our current favorite processor is the 32-bit SAMD51, which incorporates a Cortex-M4 CPU. We’ve used the SAMD51 as the basis for many of our recent and upcoming Arduino-compatible boards, including the PyGamer, a simple battery-powered gaming handheld. What if we could use it to literally put AI into people’s hands?
Warden had created a speech-recognition model that can identify the words “yes” and “no” in an analog audio feed. I set about seeing if I could bring this to the PyGamer, and what I might do with a model that could recognize only two words. I wanted to create a project that would spark the imagination of makers and encourage them to start exploring machine learning on this kind of hardware.
A Bit of Fun: After she created an Arduino-compatible version of TensorFlow Lite, the author adapted a voice-recognition demo so that pressing a button and speaking into a microphone attached to a SAMD51-based PyGamer would play back different animations.Photo: Adafruit Industries
I decided to make it as playful as possible. The more playful something is, the more forgivable its mistakes—there’s a reason why Sony’s artificial pet Aibo was given the form of a puppy, because real puppies are clumsy and sometimes run into walls, or don’t follow instructions.
I recalled the original Tron movie, where the hero is stuck in cyberspace and picks up a sidekick of sorts, a single bit that can say only “yes” or “no,” with an accompanying change of shape. The PyGamer has a 1.8-inch color display, with 192 kilobytes of RAM and 8 megabytes of flash file storage, enough to display snippets of video from Tron showing the bit’s “yes” and “no” responses. The PyGamer’s SAMD51 processor normally runs at 120 megahertz, which I overclocked to 200 MHz for a performance boost. I connected an electret microphone breakout board to one of the PyGamer’s three JST ports.
Then I turned to the trickiest task: porting the TensorFlow Lite ARM code written by Warden and company into a library that any Arduino programmer can use (although not for every Arduino board! Even as a “lite” framework, the RAM requirements are far beyond the 2 kB of the Arduino Uno, for example).
I found the source code well written, so the biggest challenge became understanding how it handles incoming data. Data is not digested as a simple linear stream but in overlapping chunks. I also wanted to expose the code’s capabilities in a way that would be familiar to Arduino programmers and wouldn’t overwhelm them. I identified the functions most likely to be useful to programmers, so that data could be easily fed into a model from a sensor, such as a microphone, and the results outputted to the rest of the programmer’s code for them to handle as they wish. I then created an Arduino library incorporating these functions, which you can find on Adafruit’s Github repository.
Putting it all together, I wrote a short program using the new library. Now when I press a button and speak into the PyGamer’s attached microphone, the appropriate Tron clip is triggered if I say “yes” or “no,” letting me walk around with my own animated sidekick.
Although this project is a limited (but fun) intro to machine learning, I hope it persuades more makers and engineers to combine AI with hardware explorations. Our next steps at Adafruit will be to make it easier to install different models and create new ones. With 200 kB of RAM, you could have a model capable of recognizing 10 to 20 words. But even more exciting than voice recognition is the prospect of using these cheap boards to gather data and run models built around very different kinds of signals. Can we use data from the PyGamer’s onboard accelerometer to learn how to distinguish the user’s movements in doing different tasks? Could we pool data and train a model to, say, recognize the sound of a failing servo or a switching power supply? What surprises could lie in store? The only way to find out is to try.
This article appears in the August 2019 print issue as “Making Machine Learning Arduino Compatible.”
Editor’s note: Limor Fried is a member of IEEE Spectrum’s editorial board.