Tijana Šušnica (ALUO), Ingrid Gatz (FRI), Luka Krivec (FRI)
Mentor: Borut Batagelj, Franc Solina (FRI)

Multisensorium

Multisensorium is an interactive art project designed to simulate the sensation of synesthesia. It is an intuitive instrument which question human sensory abilities and abilities to adapt to new environment, without having to learn to play the instrument. It captures body movements with Kinect and translates them into sounds and colors seen on a screen. The version of the Multisensorium realized here is the beginning of a bigger project and captures hand movements of the right arm and the position of the body on the ground for the simulation process. The hand movements lead to different tones and colors shown on screen, while the body position on the ground defines one out of four different instruments possible to play.
For easy intuitive playing, the possible hand movements to be detected and expressed as colors and sounds have been set to the following five different positions of the right hand of the player: 90, 45, -45, -135 and 135. These degrees are set accordingly to a normal unit circle, with the fingers on the circle line and the wrist as the center.
The selection of the 12 possible tones was done using the upper right part of the circle of fifths. The five selected tones are C, D, E, G, and A . The five hand positions and notes are matched with corresponding colors, inspired by the Skriabin Color Circle: red, yellow, green, blue, and violet.
There can be four different instruments selected for playing: Vibraphone, (synth) organ, PAD, and bass guitar. This is done according to the position of the player on the ground, the behavior is thus similar to an orchestra projected to the ground.
The visual part of the project includes two different views to be seen on screen. Usually, there will only be the corresponding color displayed on screen. However, there exists also the possibility to select a different version of the screen which will not only display the color but also the player as detected by Kinect, the exact position of the body on the ground, the thereby selected instrument, and the detected hand position.

Video

88_multisenosirum-debugscreen.jpg