I participated in a collaboration between artists and scientists organized by Pavlov E-lab in 2012. The team consisted of Professor of Neuromechanics and Prosthetics Bert Otten, PhD student in Cognitive Neuroscience Barbara Nordhjem, filmmaker Dinanda Luttikhedde from Peergroup, visual artist Michael Horsham from the design collective Tomato, and mediaartist and composer Jan Klug.
Here are some notes I made about using sensory data for art projects.
Here I introduce some projects where data is turned into something that can be sensed and experienced. A lot on this post is inspired by the course Perceptualization I followed at the Media Technology department in Leiden.
Recording human data
There are so many ways of getting data from the human body. For instance, Bert Otten uses a motion tracking system, and in my lab we work with fMRI and the eye-tracker. Both movement of the eyes and the rest of the body can also be tracked with inexpensive web-cams by using OpenCV or other video based tracking systems and commercial systems like the Kinect.
Gazetracker found at http://www.gazegroup.org/
There are also cheap ways of doing EEG to measure the brain’s electrical activity. Electrodes placed on the scalp pick up electrical changes in voltage that correlates with post-synaptic action potentials. The signal is quite weak, typically just a few microvolt, and has to be amplified. Only activity of the neurons closest to the scalp can be detected, and the signal picked up by a single electrode represents the activity of millions of neurons firing synchronously. The advantage of working with EEG compared to for instance fMRI i that it has a very high temporal resolution; changes can be measured down to the millisecond, where other brain imaging techniques are much slower.
Below is an art project where EEG measures are used:
EEG based art evaluation system made byhttp://www.acclair.co.uk/
Instead of plotting graphs, you can listen to, touch, smell, and feel your data.
Sonification is displaying information as sound. Here I’ve included the Grafische Methode Fiets as an example. Here the increase of muscle tension during the performance is turned into sound. The technique used is Electromyography (EMG) which can record the electrical activity produced by the muscles.
You can also think of turning data into odors, haptic feedback etc. In this project the movement of a avatar is transferred into Galvanic Vestibular Stimulation, which affects the sense of balance (Equilibrioception). The wavy movement of the doll is literally felt by the participant.
Brain plasticity is ‘the adaptive capacities of the central nervous system – its ability to modify its own structural organization and functioning’ . Even though there are areas in the brain that are associated with certain specialized functions, the brain can “rewire” itself. For blind people you can often see that visual areas in the brain are being used when they are reading Braille.
Braille is one form of sensory substitution where visual information from reading is acquired by touch. In the sixties physician Bach-y-Rita started developing machines that could compensate senses, for instance in blindness. The Tactile Vision Sensory Substitution device transformed visual information recorded with a camera into tactile stimulation of the back. Later a smaller version for the tongue has also been developed (the BrainPort). People can learn to orient by using these signals because there is consistent relation between the environment and the information they receive.
Bach-y-Rita with the Tactile Vision Sensory Substitution device for the back
Another tactile-vision sensory substitution system. Electrotactile stimuli are delivered to the tongue via ﬂexible electrode arrays.
Animals are sometimes able to sense things in the environment we can’t. For instance birds also have receptors in their eye to see infrared. Providing a consistent feedback from the environment might give us an extra sense with time. Here some electronic hairs that work a bit like whiskers. Another of a new sense is this vibrating belt creating a new compass-like sense.
You can also limit the senses such as done in the project Monocromeye. It is a portable helmet that enhances low resolution vision. You wear three light sensors on a the finger which each pick up color information from the environment. The color information is sent to light diodes that sends ligt straight to the eyes.