AudioMostly 2008, Piteå northern Stockholm, notes from the second day of the conference. Notes from the first day are here.

Keynote by Stephen Brewster - Multimodal interaction design

Stephen is THE multimodal guru, professor of human-computer interaction at the University of Glasgow since 2001. Stephen talked a lot about multisensory interfaces, and alternative display techniques especially focusing on auditory and tactile feedback. He is calling for truly mobile interaction – the sort that doesn’t require stopping in your tracks whenever you want to do something with your device. That’s a real challenge, and calls for flexible interfaces not constantly requiring 100% user attention.

Stephen showed us some really good examples, not only in terms of what they did but I also liked the design process and the philosophy, if you like, that went into redesigning the products they worked with. Like the camera phone they worked on to improve the experience of digital photography, adding small and really non-intrusive auditory icons telling the photographer about memory space left, luminance etc. The beautiful decision was to use only sounds that already are in the camera, not introducing new sounds but working with the old and augmenting/tweaking them to convey more information. For example, with the camera the shutter sound changes as memory space goes down and the focus sound is giving you boiled down information about exposure. Other examples included using tactile feedback for battery level; audio, visual and tactile information about stability. Another example was to work with spatial sound in headphones to create audio menus, e.g. giving each application or menu choice its own part of the audio space. The information is organized in pie-menues, and you interact by nodding.

Another interesting thing was the notion of tactons: vibrotactile icons or tactile earcons. They are stuctured, abstract messages that can be used to communicate non-visually. Tactons are designed using musical structures, actually driven via audio signals. Some results on the information structure for cutaneous perception suggests rhythm works well, but waveform does not – you really don’t get a sense of timbre in tactile feedback. If I got it right, roughness and intensity would also work ok. They had tested the crossmodal learning, and the results were really positive. So training with audio would transfer to tactile and vice versa. Finally, some experiments with touchscreen buttons show vibrotactile feedback really enhances performance, in their case added vibrotactile feedback to a touchscreen increased performance to the same level as obtained when testing with real buttons.

Talks

Lars Stockmann, Axel Berndt & Niklas Röber: A Musical Instrument based on 3D Data and Volume

Larst started by showing some earlier work on the sonification of 2D shapes. The method used a sweep line and sonifies the gradients or edges of figures (as pitches). With active exploration the sweep line would give information and you could stay and explore something in space (to me, this sounds a lot like how touching works – does the auditory become a tactile experience?). Using a similar interaction style, they extended this into 3D, exploring sonification of volumetric data. The Tone Wall is this all, turned into a music instrument. Exciting non-visual interface, I expect you could get a grasp of it but this is really something you must try to understand to get the “my-action, my-sound” mapping.

Stuart Cunningham, Stephen Caulder & Vic Grout: Saturday Night or Fever? Context Aware Music Playlists

Automatically generated playlists are good, but one list doesn’t cover it all. People want different music to different situations, and music taste changes depending on context. Traditional measures (genre, rating, BPM) don’t solve all situations – some songs have strong subjective connotations, others just defy classifications. Stuart & co. propose enhancing automated playlists by adding a control parameter regarding the emotional state (e-state) of the user. The e-state could involve both user biometric data (HR, locomotion, GSR, …) and contextual information from environment variables (lighting, temperature, noise environment…). The work is still in progress, at the state where they are developing a fuzzy logic matching system to derive an e-state (0 to 10) and provide music suggestions based on that. I really like their broad approach to music suggestion, it resonates well with the multiple influencing factors on emotion I have worked with. We talked a bit with Stuart, I thought it would be cool to add some game parameters as environment variables (talking now about pervasive, alternate reality games) to influence the music soundtrack.

Anders-Petter Andersson, Birgitta Cappelen & Fredrik Olofsson: Same but Different - composing for interactivity

Anders-Petter was able to speak with some confidence about sounding installations – he has been making those for 10 years! I really appreciate the founding principles he talked about, to include everyone in the design and interaction, use many-to-many mappings and considering interaction in a truly free, unpredictable and open way. I think what was common between all the projects he mentioned was that they all featured soft design categories, presenting the user with objects that changed nature flexibly depending on how they are being used. The design team had also been very sensitive to the social aspect of using and interacting with objects. Especially the ORFI-cushions that Anders-Petter showed us were really designed around communication, both with the object and with other people, using the object as a means of expressing yourself. The ORFI-cusions feature a set of (20+) different size cushions, small to armchair-size, connectible and buildable, with speakers, sensors, microphones. The cushions are wireless and what you could call “child-proof”. Anders-Petter talked about composition, mainly the thing I got most out of was the way communication was embedded in the composition, with sounds responding to different patterns of user interaction such as call and response, simultaneous activity, rhythmic (matching a certain beat), imitation, etc. The cushions feature several different genres to choose between, with slightly different musical strategies. All in all, the presentation was really nice and I kept watching those cushions and thinking how extremely cool it would be to have a set of PaRappa cushions that you play by wiggling their ‘ears’.

Gabriel Gatzsche, Markus Mehnert & David Gatzsche: The Harmony Pad - A new creative tool for analyzing, generating and teaching tonal music

Gabriel presented the Harmony Pad, an electronic musical instrument designed specifically around classical music structures. The user interface for example makes it easier to access harmonically consonant notes and form cadences. Gabriel argued this makes it more fun to play impromptu, bringing the barrier down for people to start improvising at an early stage in their musical training. Playing with the Harmony Pad also implicitly teaches the player about musical structures, learn music theory and harmonization. Very thought out interface layout. Very though out approach. I think it is an interesting step to start designing the layout of an instrument to fit a certain musical style and not even try design it to suit all purposes equally (and end up with instruments that if you play them without really knowing how to, they sound plain out horrible). The risk is, if we have too many of these, does it restrict playing too much? Gabriel’s idea was clearly not to abandon the piano, but play with the Harmony Pad alongside instrument lessons. That is not the easiest way out, so there would always be musicians abandoning the piano for the Harmony Pad.

Antti Jylhä & Cumhur Erkut: Sonic interactions with hand clap sounds

This presentation was about using continuous hand clapping as a way of communicating with the computer, especially looking at the rhythmical pattern of clapping in communication with the machine. Antti presented three demos for clapping, I assume the last one, adjusting the tempo of a song by clapping, most clearly demonstrates the goal of the study. In clap-based tempo modification, the machine implicitly conveys information about the communication by responding to the users claps. However, the clapping rhythm of the person controlling the system is also somehow influenced by the rhythm of the process – it is hard clapping out of sync. Antti made a strong point about this two-way interaction in clapping: you control by clapping, but you also tend to clap in synchrony. In essence, then, the clapping is a negotiation between you and the computer. It is not only control, it is communication. I love the idea! The demo was lagging quite a lot, though; I think really getting into the negotiation would require better time-response. Maybe it would be possible to somehow work with a known lag and synchronize the response with the next beat?

Ulrich Reiter: Toward a Salience Model for Interactive Audiovisual Applications of Moderate Complexity

Ulrich presented work on developing a salience model for quality predicting. His goal is to develop metrics for multimodal quality assessment and guidelines. The work is based on a cognitive approach, looking into concepts like the perceptual cycle, concepts of attention.

Andreas Floros, Nikolaos Grigoriou, Nikolaos Moustakas & Nikolaos Kanellopoulos: dots: and Audio Entertainment Installation using Visual and Spatialbased Interaction

Nikolaos presented dots, an interaction installation using image capture with webcam.

 

Summa summarum: I would say the third AudioMostly had matured into a great event! The couple of days spent in Piteå was like a flowing waltz of extremely well organized sessions, combining interesting papers, demos and presentations and, for once, enough free time in between for passionate exchange between fellow audiophiles. The non-academic activities presented us with a touch of Swedish gourmet (both of the "aah" and "ugh" type), a tour in the acoustically impressive, newly built Acusticum concert hall, some rather civilized barhopping, and even northern lights! All this in exemplary weather. I like!