Creative self-expression can play an important role in quality of life. But self-expression in drawing may be hard for people with paralysis and/or motor disabilities. Attempts have been made to create computerized substitutes for such users, but for most of these tools, use can be tiring, and the control achieved is not complete, and does not seem flowing enough.
During my MA studies, I took a course on Human Factor research with Hadas Marciano and was inspired by a study we discussed during one of the lessons. The study presented "VoiceDraw" – a hands-free voice-driven drawing application for people with motor impairments that provides a way to generate free-form drawings (Susumu, O. Wobbrock, and Landay, 2007). The tool proposed a flowing control of the brush, by using continuous vowel-sounds for directions, rather than command words . See products of both types of control bellow:
pictures from Susumu, O. Wobbrock, and Landay (2007)
Sample interaction of drawing a stroke using (left) speech-based cursor control, and (right) VoiceDraw. With speech-based control, changes are discrete and incremental. With VoiceDraw, changes are continuous and fluid. Loudness of the utterance can be mapped to stroke thickness or brush speed.
I liked very much the idea of the tool, but felt it wasn't taken far enough: the connection between the vowels and directions wasn't intuitive, and in order to control certain aspects of the line (like color and speed) users had to open word-menus. Also, in VoiceDraw the beginning and end of lines was controlled by the sound "ch", since moving the coursor to a different locations on the screen was controlled by the same vocal joystick. See the vocal joystick, and the menu (controlled by the joystick):
pictures from Susumu, O. Wobbrock, and Landay (2007)
Directional mapping of vowel sounds (in bold red) used in VoiceDraw.
The menu is invoked by issuing the discrete sound “ck”.
My idea
I suggest a voice-drawing tool, in which the gaze will control location, while the voice will control all the characteristics of the lines: thickness, brightness, hue and saturation.
Drawing will be made by continuous voice production, looking at the picture will not produce any change by itself. Focused gaze, when long enough, will be marked by a semi-transparent graphical symbol, to show the place for a possible beginning for a "brush stroke."
Connections between voice characteristics and visual characteristics
Several studies have shown a tendency to associate high pitch sounds with light colors and lower tones with dark colors. This perceptual link between pitch and brightness was found in both synaesthetic and non-synaesthetic populations (Marks, 1974, Hubbard, 1996; Ward Brett and Elias, 2006). Hence, voice pitch may be an intuitive way to control line brightness and should be easy to learn.
There is also evidence for consistent associations between vowel sounds and color hues, beyond a specific person and beyond language. Mapping studies link front vowels to bright colors (i.e. green/yellow), back vowels to dark colors (i.e. brown/black/blue), and central vowels to gray (Wrembel & Rataj, 2008, Wrembel, 2007). So, vowels can be used for both hue and saturation control. See suggested mapping below.
Finally, I suggest to use the loudness ("volume") of the voice to define thickness of the brush strokes.
Suggested vowel-color mapping for the intuitive drawing program, based on research linking these two types of sensory information
Synesthesia, associativness, and creativity The phenomenon of synesthesia is common among artists, and attempts have been made to link creativity with synesthetic perception -(Domino, 1989; Ward et al., 2008). Researchers also associate creativity with associative thinking in general (Mednick, 1962; Dailey et al., 1997) showed stronger associations between color and sounds, and syllables in particular, for creative people (from the general population). So possibly encouraging associative thinking and connectivity can encourage creativity. In any case, it is likely that people with a greater need for artistic expression (which are potential users of a drawing software) will be characterized by associative thinking, at least to a certain extent.
Main Advantage of the proposed tool
The main advantage of this tool is intuitiveness. My belief is that while control with a menu may be more convenient in the short term, the newly proposed tool will enable seamless and continuous work, and after a short adjustment time, will allow for more direct expression, a better control experience, and will encourage more engagement in the drawing activity.
Main Disadvantage of the proposed tool
A significant disadvantage of the tool, for the time being, may be its price (due to the use of eye tracking technology), as well as the need for dedicated hardware.
Possible uses
The tool is meant for people who can't use their hands, and other artists who may want to experiment with it.
The tool can also be used for other purposes, for example as a pedagogical aid for voice-control and pronunciation training - for speech therapy or singing\accent improvement, as the visualization of voice may increase awareness to it.
Future of intuitive control
On our way to intuitiveness of control, direct brain-computer interactions will probably replace other solutions both for people with motor disabilities and the general population.
Human-computer interaction becomes more and more direct, and eventually users will possibly just imagine visual content to transfer it into the screen. Still, even in such a future, I imagine voice-drawing remaining an engaging activity in itself.
References