Music visualization using an RGB lamp

I wrote Winamp and Audacious music visualization plugins for my RGB lamp. The code is published on GitHub. Pitch is represented by colour, and loudness is represented by intensity. This post explains the method in detail.

The code starts with an array of FFT bin values. This is provided by the music player via its visualization plugin API. If only waveform data was available, the FFT could be computed using the FFTW library.

Each value in that array corresponds to sound intensity in a particular range of frequencies. Those ranges are all of equal width in hertz, but humans perceive pitch in a roughly logarithmic way. This means perceived pitch varies a lot between entries at the start of the array, and a little at the end of the array. This is handled by makeramps.rb. The frequency at the midpoint of each bin is mapped to pitch using the MIDI pitch formula. This allows pitch to be mapped to colour in a sort of linear way. Here's how the various FFT bins sum into the different colours:

The lowest frequency bin sums to red. Increasing frequencies sum less into red and more into green. The pitch midpoint sums into green. Then increasing frequencies sum less into green and more into blue. Finally, the highest frequency sums into blue. The code uses a single table called green_tab, with values corresponding to the fraction of the bin that is to be summed into green. Before the pitch midpoint, one minus the table value times the bin is summed into red, and after the midpoint the same is summed into blue.

Before summing, bins can be scaled based on human ear audio sensitivity at that frequency. This ensures that frequencies where the human ear is less sensitive make less of a contribution. I found an ISO 226 Equal-Loudness-Level Contour Signal script for MATLAB which can run in GNU Octave. This isn't very important though.

When summing bins, it is important to sum power, not amplitude. Summing amplitudes does not make sense, but fortunately they are easy to convert to power: just square them before summing. Afterwards, they are converted back to amplitude via square root, because they are to be used for setting PWM values.

If the output of this algorithm was used to directly set lamp brightness, there would be a lot of rapid brightness changes. I find this unpleasant, so I added some smoothing. The code calculates an exponential moving average for each colour, but with more smoothing for decreases and less smoothing for increases. The faster response to increases ensures the lamp will appear responsive to loud sounds.

After all this, the colours still did not appear to be in balance. To fix this, I calibrated using pink noise, and created scale factors for red and blue which cause white light when playing pink noise. It may be better to calibrate using music, but that would be more difficult and the resulting calibration might be biased toward the music used to calibrate it.

I'm posting about this because of how I'm satisfied with the end result. I haven't done any tweaking of this algorithm in many months, and I still really like the effect.

Recently I have been experimenting with an on-screen visualization plugin inspired by this. Each moment is displayed on a horizontal line with stereo determining horizontal position. Pitch sets colour, intensity sets brightness, and very intense sounds cause a wider area to be coloured.  Old data scrolls down. This part is still a work in progress, and I haven't published that code yet.