I notice that our paper Ekman, I. & Kajastila, R.: Spatialization Cues Affect Emotional Judgements – Results from a User Study on Scary Sound has been accepted to the AES 35th conference, running the theme Audio For Games. Looking forward to visiting London next spring.

AudioMostly 2008, Piteå northern Stockholm, notes from the second day of the conference. Notes from the first day are here.

Keynote by Stephen Brewster - Multimodal interaction design

Stephen is THE multimodal guru, professor of human-computer interaction at the University of Glasgow since 2001. Stephen talked a lot about multisensory interfaces, and alternative display techniques especially focusing on auditory and tactile feedback. He is calling for truly mobile interaction – the sort that doesn’t require stopping in your tracks whenever you want to do something with your device. That’s a real challenge, and calls for flexible interfaces not constantly requiring 100% user attention.

Stephen showed us some really good examples, not only in terms of what they did but I also liked the design process and the philosophy, if you like, that went into redesigning the products they worked with. Like the camera phone they worked on to improve the experience of digital photography, adding small and really non-intrusive auditory icons telling the photographer about memory space left, luminance etc. The beautiful decision was to use only sounds that already are in the camera, not introducing new sounds but working with the old and augmenting/tweaking them to convey more information. For example, with the camera the shutter sound changes as memory space goes down and the focus sound is giving you boiled down information about exposure. Other examples included using tactile feedback for battery level; audio, visual and tactile information about stability. Another example was to work with spatial sound in headphones to create audio menus, e.g. giving each application or menu choice its own part of the audio space. The information is organized in pie-menues, and you interact by nodding.

Another interesting thing was the notion of tactons: vibrotactile icons or tactile earcons. They are stuctured, abstract messages that can be used to communicate non-visually. Tactons are designed using musical structures, actually driven via audio signals. Some results on the information structure for cutaneous perception suggests rhythm works well, but waveform does not – you really don’t get a sense of timbre in tactile feedback. If I got it right, roughness and intensity would also work ok. They had tested the crossmodal learning, and the results were really positive. So training with audio would transfer to tactile and vice versa. Finally, some experiments with touchscreen buttons show vibrotactile feedback really enhances performance, in their case added vibrotactile feedback to a touchscreen increased performance to the same level as obtained when testing with real buttons.

Talks

Lars Stockmann, Axel Berndt & Niklas Röber: A Musical Instrument based on 3D Data and Volume

Larst started by showing some earlier work on the sonification of 2D shapes. The method used a sweep line and sonifies the gradients or edges of figures (as pitches). With active exploration the sweep line would give information and you could stay and explore something in space (to me, this sounds a lot like how touching works – does the auditory become a tactile experience?). Using a similar interaction style, they extended this into 3D, exploring sonification of volumetric data. The Tone Wall is this all, turned into a music instrument. Exciting non-visual interface, I expect you could get a grasp of it but this is really something you must try to understand to get the “my-action, my-sound” mapping.

Stuart Cunningham, Stephen Caulder & Vic Grout: Saturday Night or Fever? Context Aware Music Playlists

Automatically generated playlists are good, but one list doesn’t cover it all. People want different music to different situations, and music taste changes depending on context. Traditional measures (genre, rating, BPM) don’t solve all situations – some songs have strong subjective connotations, others just defy classifications. Stuart & co. propose enhancing automated playlists by adding a control parameter regarding the emotional state (e-state) of the user. The e-state could involve both user biometric data (HR, locomotion, GSR, …) and contextual information from environment variables (lighting, temperature, noise environment…). The work is still in progress, at the state where they are developing a fuzzy logic matching system to derive an e-state (0 to 10) and provide music suggestions based on that. I really like their broad approach to music suggestion, it resonates well with the multiple influencing factors on emotion I have worked with. We talked a bit with Stuart, I thought it would be cool to add some game parameters as environment variables (talking now about pervasive, alternate reality games) to influence the music soundtrack.

Anders-Petter Andersson, Birgitta Cappelen & Fredrik Olofsson: Same but Different - composing for interactivity

Anders-Petter was able to speak with some confidence about sounding installations – he has been making those for 10 years! I really appreciate the founding principles he talked about, to include everyone in the design and interaction, use many-to-many mappings and considering interaction in a truly free, unpredictable and open way. I think what was common between all the projects he mentioned was that they all featured soft design categories, presenting the user with objects that changed nature flexibly depending on how they are being used. The design team had also been very sensitive to the social aspect of using and interacting with objects. Especially the ORFI-cushions that Anders-Petter showed us were really designed around communication, both with the object and with other people, using the object as a means of expressing yourself. The ORFI-cusions feature a set of (20+) different size cushions, small to armchair-size, connectible and buildable, with speakers, sensors, microphones. The cushions are wireless and what you could call “child-proof”. Anders-Petter talked about composition, mainly the thing I got most out of was the way communication was embedded in the composition, with sounds responding to different patterns of user interaction such as call and response, simultaneous activity, rhythmic (matching a certain beat), imitation, etc. The cushions feature several different genres to choose between, with slightly different musical strategies. All in all, the presentation was really nice and I kept watching those cushions and thinking how extremely cool it would be to have a set of PaRappa cushions that you play by wiggling their ‘ears’.

Gabriel Gatzsche, Markus Mehnert & David Gatzsche: The Harmony Pad - A new creative tool for analyzing, generating and teaching tonal music

Gabriel presented the Harmony Pad, an electronic musical instrument designed specifically around classical music structures. The user interface for example makes it easier to access harmonically consonant notes and form cadences. Gabriel argued this makes it more fun to play impromptu, bringing the barrier down for people to start improvising at an early stage in their musical training. Playing with the Harmony Pad also implicitly teaches the player about musical structures, learn music theory and harmonization. Very thought out interface layout. Very though out approach. I think it is an interesting step to start designing the layout of an instrument to fit a certain musical style and not even try design it to suit all purposes equally (and end up with instruments that if you play them without really knowing how to, they sound plain out horrible). The risk is, if we have too many of these, does it restrict playing too much? Gabriel’s idea was clearly not to abandon the piano, but play with the Harmony Pad alongside instrument lessons. That is not the easiest way out, so there would always be musicians abandoning the piano for the Harmony Pad.

Antti Jylhä & Cumhur Erkut: Sonic interactions with hand clap sounds

This presentation was about using continuous hand clapping as a way of communicating with the computer, especially looking at the rhythmical pattern of clapping in communication with the machine. Antti presented three demos for clapping, I assume the last one, adjusting the tempo of a song by clapping, most clearly demonstrates the goal of the study. In clap-based tempo modification, the machine implicitly conveys information about the communication by responding to the users claps. However, the clapping rhythm of the person controlling the system is also somehow influenced by the rhythm of the process – it is hard clapping out of sync. Antti made a strong point about this two-way interaction in clapping: you control by clapping, but you also tend to clap in synchrony. In essence, then, the clapping is a negotiation between you and the computer. It is not only control, it is communication. I love the idea! The demo was lagging quite a lot, though; I think really getting into the negotiation would require better time-response. Maybe it would be possible to somehow work with a known lag and synchronize the response with the next beat?

Ulrich Reiter: Toward a Salience Model for Interactive Audiovisual Applications of Moderate Complexity

Ulrich presented work on developing a salience model for quality predicting. His goal is to develop metrics for multimodal quality assessment and guidelines. The work is based on a cognitive approach, looking into concepts like the perceptual cycle, concepts of attention.

Andreas Floros, Nikolaos Grigoriou, Nikolaos Moustakas & Nikolaos Kanellopoulos: dots: and Audio Entertainment Installation using Visual and Spatialbased Interaction

Nikolaos presented dots, an interaction installation using image capture with webcam.

 

Summa summarum: I would say the third AudioMostly had matured into a great event! The couple of days spent in Piteå was like a flowing waltz of extremely well organized sessions, combining interesting papers, demos and presentations and, for once, enough free time in between for passionate exchange between fellow audiophiles. The non-academic activities presented us with a touch of Swedish gourmet (both of the "aah" and "ugh" type), a tour in the acoustically impressive, newly built Acusticum concert hall, some rather civilized barhopping, and even northern lights! All this in exemplary weather. I like!

AudioMostly 2008, Piteå norther Sweden. First day.

Here are some notes from Piteå, day 1. There were no parallel sessions, which was a really good thing, so this is really an overview of the whole conference.

Keynote by Nick Laviers
Nick Laviers is a video game Audio Director, based at EA’s Los Angeles Studio. Nick’s talk was about the music in Red Alert 3 and some of his earlier work such as the Harry Potter games. He took a very broad approach, and in his speech covered the design process both in terms of design decisions as well as the practical aspects of the process, from finding the right composers to organizing orchestral recordings. The talk was really engaging and peppered with useful insights about the realities you face when working on a huge title such as the C&C.
Some strategies I really want to look into:

• Using the memory of play (at restart of level etc.) to influence sounds. Nick’s example was with the background music remembering earlier encounters, I think you could extend that to the acoustic space as well. A room with a ‘worn’ feel.
• "Up yours" music. Basically, playing the other teams music when they really hit you down. That’s almost like the game is taking sides, temporarily abandoning your music to play the other players theme…
• Resting moments, reflection moments. Giving time for the players to absorb the consequences of their actions.

Talks

Mark Grimshaw, Craig A. Lindley & Lennart Nacke: Sound and Immersion in the First-Person Shooter: Objectively Measuring the Player´s Sonic Experience

Mark and colleagues are trying to measure immersion. Their experiment had people playing a Half Life 2 mod with different sound conditions and then measuring their subjective and psychophysiological responses to the sound. Not surprisingly, the subjective reports show people do perceive more ‘immersion’ when the sounds are on. The unexpected result was that they found no sign of difference in the psychophysiological (tonic) data depending on audio condition. Personally, I do think this has something to do with the measurement strategy: using a tonic measurement would probably even out all effects. The thing then is to decide suitable chunks of data for phasic analysis – now that’s a real challenge. It will surely be interesting to see how they proceed.

Daniel Kromand: Sound and the diegesis in survival-horror games

Daniel gave us a fascinating theoretic/analytic investigation into sound in two survival horror game titles: FEAR and SilentHill(2). He’s extending on Joergensen´s notion of the transdiegetic and looks at the tension created through functional use of sound, essentially the very same thing I’ve been working on the last few months. What Daniel is saying is emotional effect exists in the collapse of functional reliability. So, tying it down with the notion of functional fit, if positive emotions come from facilitating gameplay, certain negative emotions (fear/anxiety) are created by selectively making the cues in sound unreliable, this would make it more difficult to use game sounds for building up a strategy.

I had my paper as well on the first day:

Inger Ekman: Psychologically Motivated Techniques for Emotional Sound in Computer Games.

I looked at the complex tangle that is emotion design and what tools a game designer will use to create emotional impact in a game. The crux of my talk were the two competing frameworks for emotion design in games; you could think of them as story-emotions and gameplay-emotions. The two approaches employ partly conflicting strategies in sound, so you should try to find a way for either alternating or going for one of them, but not both at once. I also looked into all the other emotional evaluation that is going on alongside the cognitive evaluation processes and suggested how that maps into the general emotional experience. It seems to me the talk was well received and I was fortunate to have many interesting exchanges about it throughout the rest of the conference. And, how lovely it was to know in the evening of the first day I don’t have to be completely sober presenting the next morning

I’ll be putting up my slides soon and add a link so you can find them.

Louise Valgerður Nickerson & Thomas Hermann: Interactive sonification of grid-based games

Louise presented work on a sonified sudoku game and a four-in-a-row game, basically both really challenging grid-sonifications. The Sudoku had this touch-to-play approach, which would essentially let you use sound as a sort of tactile information and feel your way through the lines. The four-in-a-row game plays constantly, looping all seven piles in two bars of 4/4 beats and pitch representing the columns. I thought it sounded really funky in a cool way. I got to try it out in the evening, and I would say it really is playable and fun. Could be cool to have something like that on a mobile working over a Bluetooth connection!

David Moffat & David Carr: Using audio aids to augment games to be playable for blind people

David presented a sonified Asteroids! game, that is the normal Asteroids! but with added sound. They evaluated the game with sighted players playing either visual-only, audio-only or audio-visual conditions. Interesting results, as the best audio-only player was actually equal with a novice playing in visual-only condition! Adding audio did mostly not affect playing, however there were signs of cognitive owerload for the novice player when playing the audio-visual game condition. Had a very interesting discussion with David afterwards, where we talked about the real challenges for audio gaming and which sonification tasks will be the most challenging to solve as we try to create blind-accessible gaming. He suspected the most difficult tasks will be sonifying abstract things, and that just adding more realistic ‘naturalistic’ representations will get us quite far. I think we have quite a challenge left there, though, because the goal is also really to use ‘naturalistic’ in a suitably non-naturalistic way for other effects, like emotion. So finding a good balance there will be important.

Mats Liljedahl & Nigel Papworth: !BeoWulf - Sound based game

Nigel presented fresh research done with their BeoWulf game, specifically user studies of how people perceive the sounds of BeoWulf. I really enjoy the position they argue – on how sound opens up the creative processes of people and that lack of visual information actually enhances the sense of ‘being there’ instead of the opposite. This research was looking into how people describe the audio-mostly game world (visually they only see a simple map). The results demonstrate the richness of interpretations when using mainly sounds as a stimulus. I wouldn’t myself go so far as to say that the visual information closes down imagination, but certainly sound and visuals work in slightly different ways to open different ‘doors of imagination’.

Scott Beveridge & Don Knox: Control of Sound Environment using Genetic Algorithms

Scott started the presentation by showing an earlier project, a sonified cow (yeah, real one!). We all laughed. Then he went on to describe their current system for creating meaningful sonic representation of crowds. They collect data by webcam, map the crowd actions to sound. The main goal was to strive for musical output, which creates a question of mapping. Too literal mapping breaks up the structure, which will lead to atonality. Too abstract mapping will make the system feel detached and unresponsive.
The system uses direct and indirect translation, directly mapping certain features of an emotion classification engine and mapping them onto an arousal/valence map. The indirect mapping will produce high-level features such as chord progression and melodic contour, using genetic algorithms evaluated for consonance. The system seems interesting and complex enough, it will be interesting to follow their future work, especially as Scott said they are planning for inclusion of suprasegmental features such as melodic contour. That’s a tough one.

Daniel Hug: Genie in a Bottle: Object-Sound Reconfigurations for Interactive Commodities

Daniel is looking at interactive commodities and considering the semantics relating to the sound of objects. Especially interesting are electroacoustically augmented gadgets, intelligent, enhanced commodities that can play with the sound-object relationship. I’ve heard Daniel talk about these things before and I’m always fascinated by the way he ties together information from all these different fields: computer games, film sound, design tradition, sonification… Good overview of different fields with implications for game sound design as well.

In a few weeks, I will be presenting a paper entitled "Psychologically Motivated Techniques for Emotional Sound in Computer Games" at the AudioMostly 2008 conference in Piteå, Sweden. AudioMostly is an interesting conference, both with its focus on interactive sound but especially so because it has been attracting research specifically concentrating on game sound, something you really don’t see much of anywhere. This is the third year of AudioMostly and it will be exciting to see how the event has developed since last year and the year before.

Today I noticed that the program for the conference is finally online. I am happy to see some interesting names there, people whose research I have been following for a long time but haven’t had the chance to meet in person before. Looking forward to the event.