How to Turn Music Into Colors With a Wi-Fi Bridge

Several years ago, I wrote an algorithm I called ColorChord, which assigned colors to notes. The neat thing about it was that every time I played a note, the same color was assigned—regardless of the octave. By using these color values to control LEDs, I had a whole new way to visualize chords and melodies. Sadly, the algorithm was a bit clunky. The Pentium-based processors available at the time had difficulty running the algorithm in real time, forcing me to use specialized graphics processors.

Fast-forward a couple of years: A friend of mine dug into the algorithm and found it could be sped up enormously. Based on his feedback, I rewrote it twice. Eventually, it could easily run on a desktop CPU, and further tweaks and changes made it possible to run on a 168-megahertz STM32F407/417 microcontroller. Eventually, I wondered if I could run it on something even simpler—something as simple as a Wi-Fi bridge?

To be fair, I did have a specific Wi-Fi bridge in mind: the remarkable ESP8266 from Espressif Systems, based in Shanghai. The ESP was designed to be little more than a serial-to-Wi-Fi device that you could plug into your favorite controller and operate using AT-style modem commands to get access to wireless networks. However, the hacker community quickly realized that the ESP’s onboard processor could be repurposed for so much more.

The ESP8266 uses a Tensilica processor core, factory-set to 80 MHz but overclockable to 160 MHz. Its features include general-purpose input/output ports (GPIOs), Integrated Interchip Sound (I2S) interfacing with direct memory access (DMA) for speedy data transfers, hardware support for pulse-width modulation, and an analog-to-digital converter (ADC). It’s possible to make Internet of Things devices and servers possessing rich Web-based-configuration graphical-user interfaces with code running on the ESP. And with WebSockets, applications running on the ESP can provide updates to and receive input from remote users at a rate of over 600 hertz.

The community has developed firmware tools, a compiler, demos, and integrated development environments (IDEs) that let you run Arduino and NodeMCU code on the ESP. Fortunately, Espressif has embraced this opportunity. The company has released and updated its software development kit (SDK) and released documentation, rather than holding it behind mountains of nondisclosure agreements. Consequently, the ESP has gone from a US $2 Wi-Fi chip (breakout boards cost around $10) to a fully featured 32-bit wireless microcontroller with a growing following.

I wanted to run ColorChord on the ESP so that it could listen to sounds, determine color values, and then use them to control a string of WS2812B [pdf] LEDs. WS2812Bs are individually controllable color LED modules. Each LED accepts a serial signal that specifies 8-bit RGB (red-green-blue) values for a given color. When arranged in a string, each module takes its color from the first three bytes it sees, strips those bytes off, and passes the rest to the next module.

For many projects, lots of folks will find the Arduino IDE and its Wiring language to be ideal. My go-to, however, is the C language, which I use with the official Espressif SDK to build my own firmware. Though it can be more awkward, the SDK provides all the functionality you’ll need and gives you the ability to optimize your code. You build firmware using a GCC compiler and the SDK, and upload it to the ESP over a serial port.

In writing the software for my project, I knew I would need to sample the onboard ADC to analyze incoming sound from an attached electret microphone. Though the official SDK provides a function that can read the ADC, it wasn’t precise enough to produce the continuous audio wave forms that the ColorChord algorithm needed. After doing some online digging, I found someone who had worked out how to read the ADC using precise processor interrupt requests. Using these interrupts, I found I could read the ADC at a rate of 16 kilohertz and get a suitable wave form without jumps or jitters.

The next problem was controlling the string of WS2812Bs. They require precise timing to operate, and it can take a while to shift out all the color values for a long string. I turned to the ESP’s I2S bus and DMA. The I2S bus is normally used to read or write audio data, but it can be abused as a glorified shift register. Hooking up this bus to the DMA engine allows large buffers of data to be shifted out to the string of LED modules over a single pin, all with minimal use of the CPU.

With most of the low-level input and output being handled by supporting hardware, the ESP core was freed up for the heavy lifting of running the ColorChord algorithm. I took the highly optimized ColorChord C code I used for my desktop version, dropped it on the ESP, and ran it. Much to my shock, the ESP didn’t melt down. It connected to my home network, and when I visited its Web interface, everything seemed to be running. To convince myself it really was working, I hummed into the mic—and the LEDs glowed a bright yellow. And I was running the ESP at 80 MHz, not even tapping half its potential speed! Adding additional features to the Web interface let me configure and tune the algorithm, watch what all the notes were doing, and even see an oscilloscope of the wave forms picked up by the mic. I had done it!

The entire setup was small enough to attach to a saxophone. Then I placed LEDs inside the bell so that when the sax is played by someone—such as my friend David Chapman—it emits colors as well as sounds.

I have been amazed by the ESP8266 time and time again. Espressif is still revealing more and more of what this chip can do, and the community at esp8266.com is exploring possibilities that Espressif hasn’t even considered. I am going to keep searching and programming and learning more and more.

This article appears in the August 2016 print issue as “Turning Music Into Colors.”