spxlAudioToMidi v11.5

Description

A small application made with Processing to produce MIDI control messages from a live audio feed. Auto-levelling frequency band meters are used to obtain "peak" output values across the entire audio spectrum analysed (whereas it is usual to have strong bass response with weaker treble response), and easing is applied to make the levels less erratic.

The auto-levelling algorithm tracks high and low values output by the audio analysis (FFT) for each frequency band:

Determine a lower bound for the high value: a multiple of the current low value, with a hard lower limit.
The high value decays toward the lower bound (to handle the audio levels getting quieter).
The high value jumps up to the current FFT value, if the FFT value is higher.
The low value tracks up toward the high value (to handle the audio levels getting quieter).
The low value jumps down to the current FFT value, if the FFT value is lower.
The new position of the meter is determined as the current FFT value's position between the new low value and high value, normalised to the range [0,1] (zero to one, inclusive).
The output position jumps to the new position if the new position is higer (or easing disabled), else is eased down to the new position.

10 separate monitor outputs are supplied, where each monitor is attached to one of the frequency band meters and that meter's output is scaled to the particular output range for the monitor, allowing for a reduced final output range as well as inverted ranges. Drop-down menus are provided to select the MIDI input and output devices, as well as the MIDI channels to use.

This app has been used to control VJ applications Resolume and Neon v2, allowing simultaneous control from an external MIDI controller on the same channel (using loopMIDI as a virtual MIDI driver and MIDI-OX for routing).

Monitoring audio on Windows 10

The "Stereo Mix" device only exists for some audio hardware. To do monitoring in Windows 10 when there is no "Stereo Mix" requires using a 3rd party driver. See How to Get Stereo Mix in Windows 10.

As of v11.3, spxlAudioToMidi searches for a mixer name starting with strings listed in a text file in the data directory called "mixer-prefs.txt", listed in order of preference., for example:

Microphone
Stereo Mix
VoiceMeeter Output
VoiceMeeter Aux Output
VoiceMeeter VAIO3 Output
CABLE Output

This should match any mixer starting with "Microphone" (eg "Microphone Array ..."); the three outputs from Voicemeeter from VB-Audio Software (tested with Voicemeeter Potato), VB-CABLE Virtual Audio Device, also from VB-Audio Software, or the oldschool "Stereo Mix" available on some systems, if any of those are available. If you are using a different audio device, or want to change the order of preference, simply edit the text file and reload the mixer preferences by pressing F5 (or restarting the app).

Download

spxlAudioToMidi v11.5 (2022-09-13): developed and tested with Processing v3 (3.5.4 Win 64) on Windows 10 Home (64-bit).

NOTE: MIDI input has been disabled. It needs to be rewritten for The MidiBus library.

spxlAudioToMidi-v11.5-source.zip (26 KB) - source only; requires Processing 3 + libraries
spxlAudioToMidi-v11.5-win32.zip (5.2 MB) - Win32 application, source; requires Java 8 (32-bit)
spxlAudioToMidi-v11.5-win64.zip (5.3 MB) - Win64 application, source; requires Java 8 (64-bit)
spxlAudioToMidi-v11.5-win64-java.zip (76 MB) - Win64 application, source, Java 8 (64-bit) included

You can also check the file list.

Libraries used

Library	Purpose	Local copy
The MidiBus	MIDI input/output	themidibus.zip (132KB)
Minim	Audio analysis (FFT)	minim.zip (18MB)

Changes

v11.5

Add spxlMidiManager.outputMuted (key 'o')
Add class MixerManager: handles list of mixers; display mixer-prefs list and (unselectable) other mixers
Introduce class SimpleLock, object menuLock: resolve mouseEvent() clashes (MeterMonitor, MidiiManager, MixerManager)
FN_KEY + 4: always setup the selected mixer (including setting to null, and reset meter levels)
Add try/catch for minim.getLineIn() to fail gracefully instead of crashing (eg on unsupported sample rate, bit depth)
Add FN_KEY + 5 to rerun MixerManager.setupMixers(); pops the selection list if another menu not open
FN_KEY + 3 to select microphone input now redundant; mic mixer name instead specified in the MIXER_PREFS_FILENAME file

v11.4 (unreleased)

setupInputMixers(): Stop searching if microphone mixer found and first preference mixer found

v11.3

setupInputMixers(): Use strings from "mixer-prefs.txt" to choose stereo mixer

v11.2

Change order of operations in meter.update() to ensure current lo/high value will always accommodate current value (possible error in previous code: if value < highValue, highValue decays toward zero, and that could have been to less than the current value; similarly for lowValue calculation).

v11.1

Pass raw (not eased) FFT value to meter.update(value) and allow the meter to eaase its output

v11.0

Change MIDI library from ProMidi (no loger supported) to MidiBus
Change frameRate(30) to frameRate(60) (also update spxlMeter algorithm and constants)
Update Minim input audio mixer handling (ditched JSminim stuff, based on Advanced/setInputMixer example)
Now handles repeated switching between "Microphone" and "Stereo Mix" / "CABLE Output"
Modified meter.update() algorithm

Demo videos

spxlAudioToMidi v11 short demo (16sec) (v11.0, 2020-10-06)
spxlAudioToMidi v11 medium demo (5mins) (v11.0, 2020-10-06)
spxlAudioToMidi v11 long demo (15mins) (v11.0, 2020-10-06)

Operation

The text at the top of the window controls the MIDI output. The left column of 30 bars are audio meters; each meter representing an average for a frequency band produced by FFT analysis with the low frequencies at the top of the display and the high frequencies at the bottom of the display. The meters are auto-levelling and provide normalised output values between 0 and 1 (inclusive). The right column of 10 bars are meter monitors, with each one connected to a particular meter. A monitor's output value is determiend by scaling the input meter value over the monitor's output range between 0 and 127 (inclusive); the lower and upper limits of each range can be adjusted, including to scale the meter input inversely, and the meter it is connected to (thus the frequency band it monitors) can be changed. If a MIDI output device is selected, each time a monitor output value changes, the value is emitted as a MIDI control message on the selected output channel. The control number is in the range 1 to 10, with the top monitor being control 1 and the bottom one being control 10.

Selecting MIDI intput device

Click on the "No MIDI input" text to pop up the "INPUT DEVICE:" menu. Click a displayed device to select. Subsequently, clicking the displayed device name presents the "SELECT DEVICE:" menu again, and clicking the displayed channel number to the left presents the "INPUT CHANNEL:" menu with channels 1 to 16.

Selecting MIDI output device

Click on the "No MIDI output" text to pop up the "OUTPUT DEVICE:" menu. Click a displayed device to select. Subsequently, clicking the displayed device name presents the "OUTPUT DEVICE:" menu again, and clicking the displayed channel number to the left presents the "OUTPUT CHANNEL:" menu with channels 1 to 16.

Selecting audio mixer

The selected audio mixer is displayed just below the MIDI input and output devices on a red background. Click on the selected device text to pop up the "MIXER:" menu. The items listed in the menu begin with the preferred mixers on a red background (as supplied in the "data\mixer-prefs.txt" file; mixer names not matched shown in parentheses), followed other audio mixers detected in the system ("non-preferred mixers") on a grey background. The non-preferred mixers are display-only (not selectable) and are provided to guide what to put into the mixer-prefs.txt file. Click a displayed mixer to select that mixer. Subsequently, clicking the displayed device name presents the "MIXER:" menu again, and pressing F5 reloads the mixer preferences list from the file.

Activating the selected audio mixer

Selecting the mixer does not activate it immediately. You must press the F4 key to activate the selected audio mixer. This also resets the meter levels (like pressing the Z key).

Adjust monitor output ranges

Each monitor ouutputs values in the range 0 to 127 (inclusive) by default. The "minimum" and "maximum" limits can be adjusted by clicking (and dragging) the monitor output bar. Clicking in the top 3/4 of the bar adjusts the maximum; clicking in the top 1/4 of the bar adjusts the minimum. When dragging, if the mouse is dragged above or below the bar, the limit being dragged (min or max) snaps back to the previous value. Right-click the bar to reset to the default full range (0 to 127).

Changing monitor links

When the mouse is just off the left edge of a monitor bar the link to the associated meter is hilighted and "Drag to meter" is displayed. Click and drag the mouse pointer, releasing over to the desired meter; releasing when not over a meter leaves the link as it was previously. Right-clicking when "Drag to meter" is shown resets the link to the meter assigned by default when the application was started.

Keyboard controls

The "selected monitor" refers to the monitor that the mouse is over.

Space / P: [P]ause (and unpause)
M: [M]ute meter inputs; Shift+M for a mute shot (a few values output while muted)
O: toggle MIDI output mute on/off
E: Reset [E]asing amount
Shift+E: Toggle [E]asing of meter values (on by default)
Up / Down: Increase / Decrease easing amount
[ / ]: Decrease / Increase input MIDI CC and note offsets
Z: Reset ("[Z]ero") the meter levels
Q: randomise links (to meters) for all monitors; Shift+Q for selected monitor
A: reset links (to default meters) for all monitors; Shift+A for selected monitor
F: [F]lip range (swap min and max output values) for all monitors; Shift+F for selected monitor
V: in[V]ert range (sets min to 127-min, max to 127-max) for all monitors; Shift+V for selected monitor
R: [R]andom range (sets min and max output values) for all monitors; Shift+R for selected monitor
S: re[S]et range to default (sets min to 0, max to 127) for all monitors; Shift+S for selected monitor
U: Random [U]pper range (sets max output values only) for all monitors; Shift+U for selected monitor
Y: reset range upper value to default (sets max to 127) for all monitors; Shift+Y for selected monitor
L: Random [L]ower range (sets min output values only) for all monitors; Shift+L for selected monitor
K: reset range lower value to default (sets min to 0) for all monitors; Shift+K for selected monitor
1 2 3 4 5 6 7 8 9 0: Set MIDI output channel (channels 1 to 10 only, 0 for channel 10)
! @ # $ % ^ & * ( ): Set the the MIDI input channel (channels 1 to 10 only)
H: Toggle display of values in [H]exadecimal
F1: Toggle display of meters
F2: Toggle display of monitors
F3: Analyse microphone input (defunct)
F4: Analyse selected audio mixer
F5: Reload mixer preferences (and re-select first available preference)