Audio⇔Visual Audio⇔Visual

at Tokyo University of the Arts Class in 2014 “Interactive Music 2”

AMC Archive Page

General Info

The sound takes visual’s infomation, the visuals draw sound’s wave. Each of information are transformed interactively.

Application was developed with Openframeworks (both of sound and visual). GUI was made with Puredata.

Screenshot

Pd patch

System

Audio→Visual

The visual was constructed with 6 waveform, 2*RGB. Actually there are 6ch audio but each RGB waveform was summed and send to Lch and Rch.

Horizontal drawing position, Waveform amplitude, Filling of waveform can be controlled by Puredata. Also Visual can be JPG glitched in any situation.

Visual→Audio

White Holizontal line seeks upper side to bottom side.

In each frame, Holizontal pixels at the position of white line was scanned and its left half, right half and each of RGB amplitude was taken as Audio Spectrum.

Each Spectrum were processed with Inverse FFT and processed waveform is put to audio buffer.

Technical tips

The buffer size and pixel-array size was converted using ofFbo.

The framerate and the buffer rate (sampling rate/buffer size) should be the same. The memory access to audio buffer at the same time from both of visual side and audio side should be avoided, it crashes app.

Actually mutex should be used in this situation but I did’nt used because I don’t know much about that. The framerate was set to (sampling rate/buffer size). It doesn’t crash frequently, Maybe, because visual side accesses to 1-frame delayed buffer from audio side.

本来はupdate()とaudioOut()においてmutexを使用すべきだがよくわかっていなかったのでフレームレート側を無理やり合わせている。多分ビジュアルがオーディオから1フレーム分遅れたやつにアクセスしているので落ちなくて済んでいるような気がする。