Citation info: Ekman, I. (2009) "Modelling the Emotional Listener: Making Psychological Processes Audible", Proc. AudioMostly 2009, Sepbember, Glasgow, Great Britain.
 
Abstract. There is an increasing tendency to use of procedural strategies for the creation and manipulation of sound in computer games. This development is motivating a shift in the design process; meaning is no longer tied to a specific asset or asset type, but is instead linked to the procedural manipulation of the sound material. Often the design phase for this type of game sound includes modelling sound within the virtual environment in terms of a source, a medium and a listener. This way of thinking about sound is not new; however, how it relates to emotional expression has not been investigated before. Particularly what has not been modelled is the listener as a perceptual entity, whose perception changes to reflect psychological states and processes. This study addresses the lack of research in this area. We identify four psychological processes that influence what and how sounds are heard: attention, emotion, multimodal perception and internal sound. We also provide a detailed investigation of a special case of psychological process: the perceptual distortions people suffer during extreme stress. Drawing from this empirical data, we form a listener model capable of expressing the avatar’s psychological qualities through sound manipulation. The listener model is described, along with examples on how to apply it in practise.
 
Read more…

Horror Video Games: Essays on the Fusion of Fear and Play, is a collection of essays concerning horror video games. The book is edited by Bernard Perron and has a foreword by Clive Barker.

This publication is a first of its kind, as nowhere has the themes and variations of horror games previously been so thoroughly analyzed. The book provides 14 essays on various topics, by the following authors: Clive Barker, Inger Ekman, Ewan Kirkland, Tanya Krzywinska, Petri Lankoski, Christian McCrea, Simon Niedenthal, Michael Nitsche, Martin Picard, Bernard Perron, Dan Pinchbeck, Richard Rouse III, Guillaume Roux-Girard, Laurie N. Taylor, Carl Therrien, and Matthew Weise.

Together with Petri Lankoski, we contributed with an essay on horror sound called Hair-Raising Entertainment: Emotions, Sound, and Structure in Silent Hill 2 and Fatal Frame

The book will be available in October, but you can already pre-order it straight from McFarland, or alternatively from Amazon.

Earlier this year, I was fortunate to be able to attend the 35th conference on Audio for Games held by the Acoustical Engineering Society. This was a very special event since it was the first ever AES conference (in a long history of events) to deal with game sound. I enjoyed the conference very much. Particularly I was delighted with the relatively even representation of both academia and industry, and the fact that the two camps really did share thoughts and views with each other. Academia and industry mixing, you don’t get that so often.

The conference was a huge success, the downside of which was that the organizers were unable to squeezen in all of those who wanted to attend. (!) Well, in addition to organizing great conferences, the AES has a fantastic habit of providing conference reports on their webpage, and now the report for the 35th is also out. So if you weren’t able to attend, you can at least browse through the topics and the whos and whens.

The report is accessible directly from here [pdf] or alternatively you can browse your way through the long list of conference reports on this page. Maybe stumble on some other interesting topics there, too…

Unfortunately, the conference papers are not provided online by the AES. However, for anyone interested in the full story on my paper (with Raine Kajastila) "Localisation Cues Affect Emotional Judgements – Results from a User Study on Scary Sound", I provide it in full length online here.

I’ve always loved flamenco - I think the style matches my temper and moods better than any other music styles I know of. I’ve never had a go at the guitar, but I’ve been dancing for a while now, enough to dare call myself a bailaora, and I’m slowly getting into singing as well. Recently, flamenco has also entered my work interests. A couple of colleagues of mine - Antti Jylhä and Cumhur Erkut - have developed a hand-clapping based interface, that will be presented at CHI2009: "A Hand Clap Interface for Sonic Interaction with the Computer". Recently, I’ve been working together with them on an extension of that work, contributing among other with insights from flamenco. The first part of this work "Exploring self-induced sonic interactions in the context of performing arts" is being presented at the Workshop on Sonic Interaction Design and its relation to Film and Theatre sound design, at the University of York, 2. April 2009.

I was actually searching for info on similar projects, but instead ran across this very well written piece by David Rubin on creating flamenco sounds for your audio project: desktop flamenco. It’s a valuable read for anyone who works on a piece in need for some authentic-sounding Flamenco, as it covers the major principles of the flamenco style, and provides reviews of a bunch of sample libraries containing flamenco effects. Nice work bringing all that info together on one page!

Noticing that the deadline for chapter proposals has been pushed forward to March 16th, I post here a link to the call for chapters to a book on game sound (ed. Mark Grimshaw) http://www.igi-global.com/requests/details.asp?ID=578

Selected copy-paste:

The heart of this book is the relationship between player and game sound; how this is both effected and affected by technology and how this relationship itself impacts upon the design of computer game sound and the development of technology. It deals with both technical and theoretical aspects in a way that that stimulates ideas and broadens the potential readership base beyond computer games.

[…]

Recommended topics include, but are not limited to, the following (which may be combined in chapters):

• Hardware and software
• Sound synthesis and audio samples
• Similarity and difference in practice across game genres
• Sound stimulus
• Psychophysiology, emotion, meaning and sound
• Authenticity, realism and verisimilitude
• Audio icons and earcons in games
• Sonification
• Spatializing sound and localization
• Sound and image
• Sound and no image (audio-only games)
• Player(s) and soundscape(s)
• Sonic environment
• Engagement and flow
• Sonic interaction
• Immersion
• Virtual reality
• Relational agents
• Biofeedback

Citation info: Ekman, I. and Kajastila, R. (2009) "Localisation Cues Affect Emotional Judgements – Results from a User Study on Scary Sound", Proc. AES 35th Conference on Audio for Games, February 2009, London, UK. CD-ROM.
 
Abstract. The current paradigm for creating emotional impact in game sound is to carefully choose which sounds to play. This paper takes an alternative approach, suggesting that emotional impact of sounds can be affected by choosing how to play those sounds. We describe a novel concept for emotional sound design - emotional fine-tuning - and show how it is possible to systematically influence the emotional impact of a single sound sample. A controlled user study with 8 subjects confirmed that changing the reproduction of a sample so that source localization of the sound is challenged will increase its perceived scariness compared to the same sound with clearly detectable source. The work extends experimental research on emotion perception in sound. It has practical implications for sound design in games and other interactive media.
 
Read more…

Citation info: Ekman, I. (2008) "Psychologically Motivated Techniques for Emotional Sound in Computer Games", Proc. AudioMostly 2008, October, Piteå, Sweden, 20-26.
 
Abstract. One main function of sound in games is to create and enhance emotional impact. The expressive model for game sound has its tradition in sound design for linear audiovisual media: animation and cinema. Current theories on emotional responses to fiction are mainly concerned with linear medial, and only partly applicable to interactive systems like games. The interactivity inherent to games introduces new requirements for sound design, and suggests a break in perception compared with linear media. This work reviews work on emotional responses to fiction and applies them to the area of game sound. The synthesis is interdisciplinary, combining information and insights from a number of fields, including psychology of emotion, film sound theory, experimental research on music perception and philosophy. The paper identifies two competing frameworks for explaining fictional emotions, with specific requirements, and signature techniques for sound design. The role of sound is examined in both cases. The result is a psychologically motivated theory of sound perception capable of explaining the emotional impact of sound in film, as well as identifying the similarities and difference in emotional sound design for these two media.
 
Read more…

I notice that our paper Ekman, I. & Kajastila, R.: Spatialization Cues Affect Emotional Judgements – Results from a User Study on Scary Sound has been accepted to the AES 35th conference, running the theme Audio For Games. Looking forward to visiting London next spring.

AudioMostly 2008, Piteå northern Stockholm, notes from the second day of the conference. Notes from the first day are here.

Keynote by Stephen Brewster - Multimodal interaction design

Stephen is THE multimodal guru, professor of human-computer interaction at the University of Glasgow since 2001. Stephen talked a lot about multisensory interfaces, and alternative display techniques especially focusing on auditory and tactile feedback. He is calling for truly mobile interaction – the sort that doesn’t require stopping in your tracks whenever you want to do something with your device. That’s a real challenge, and calls for flexible interfaces not constantly requiring 100% user attention.

Stephen showed us some really good examples, not only in terms of what they did but I also liked the design process and the philosophy, if you like, that went into redesigning the products they worked with. Like the camera phone they worked on to improve the experience of digital photography, adding small and really non-intrusive auditory icons telling the photographer about memory space left, luminance etc. The beautiful decision was to use only sounds that already are in the camera, not introducing new sounds but working with the old and augmenting/tweaking them to convey more information. For example, with the camera the shutter sound changes as memory space goes down and the focus sound is giving you boiled down information about exposure. Other examples included using tactile feedback for battery level; audio, visual and tactile information about stability. Another example was to work with spatial sound in headphones to create audio menus, e.g. giving each application or menu choice its own part of the audio space. The information is organized in pie-menues, and you interact by nodding.

Another interesting thing was the notion of tactons: vibrotactile icons or tactile earcons. They are stuctured, abstract messages that can be used to communicate non-visually. Tactons are designed using musical structures, actually driven via audio signals. Some results on the information structure for cutaneous perception suggests rhythm works well, but waveform does not – you really don’t get a sense of timbre in tactile feedback. If I got it right, roughness and intensity would also work ok. They had tested the crossmodal learning, and the results were really positive. So training with audio would transfer to tactile and vice versa. Finally, some experiments with touchscreen buttons show vibrotactile feedback really enhances performance, in their case added vibrotactile feedback to a touchscreen increased performance to the same level as obtained when testing with real buttons.

Talks

Lars Stockmann, Axel Berndt & Niklas Röber: A Musical Instrument based on 3D Data and Volume

Larst started by showing some earlier work on the sonification of 2D shapes. The method used a sweep line and sonifies the gradients or edges of figures (as pitches). With active exploration the sweep line would give information and you could stay and explore something in space (to me, this sounds a lot like how touching works – does the auditory become a tactile experience?). Using a similar interaction style, they extended this into 3D, exploring sonification of volumetric data. The Tone Wall is this all, turned into a music instrument. Exciting non-visual interface, I expect you could get a grasp of it but this is really something you must try to understand to get the “my-action, my-sound” mapping.

Stuart Cunningham, Stephen Caulder & Vic Grout: Saturday Night or Fever? Context Aware Music Playlists

Automatically generated playlists are good, but one list doesn’t cover it all. People want different music to different situations, and music taste changes depending on context. Traditional measures (genre, rating, BPM) don’t solve all situations – some songs have strong subjective connotations, others just defy classifications. Stuart & co. propose enhancing automated playlists by adding a control parameter regarding the emotional state (e-state) of the user. The e-state could involve both user biometric data (HR, locomotion, GSR, …) and contextual information from environment variables (lighting, temperature, noise environment…). The work is still in progress, at the state where they are developing a fuzzy logic matching system to derive an e-state (0 to 10) and provide music suggestions based on that. I really like their broad approach to music suggestion, it resonates well with the multiple influencing factors on emotion I have worked with. We talked a bit with Stuart, I thought it would be cool to add some game parameters as environment variables (talking now about pervasive, alternate reality games) to influence the music soundtrack.

Anders-Petter Andersson, Birgitta Cappelen & Fredrik Olofsson: Same but Different - composing for interactivity

Anders-Petter was able to speak with some confidence about sounding installations – he has been making those for 10 years! I really appreciate the founding principles he talked about, to include everyone in the design and interaction, use many-to-many mappings and considering interaction in a truly free, unpredictable and open way. I think what was common between all the projects he mentioned was that they all featured soft design categories, presenting the user with objects that changed nature flexibly depending on how they are being used. The design team had also been very sensitive to the social aspect of using and interacting with objects. Especially the ORFI-cushions that Anders-Petter showed us were really designed around communication, both with the object and with other people, using the object as a means of expressing yourself. The ORFI-cusions feature a set of (20+) different size cushions, small to armchair-size, connectible and buildable, with speakers, sensors, microphones. The cushions are wireless and what you could call “child-proof”. Anders-Petter talked about composition, mainly the thing I got most out of was the way communication was embedded in the composition, with sounds responding to different patterns of user interaction such as call and response, simultaneous activity, rhythmic (matching a certain beat), imitation, etc. The cushions feature several different genres to choose between, with slightly different musical strategies. All in all, the presentation was really nice and I kept watching those cushions and thinking how extremely cool it would be to have a set of PaRappa cushions that you play by wiggling their ‘ears’.

Gabriel Gatzsche, Markus Mehnert & David Gatzsche: The Harmony Pad - A new creative tool for analyzing, generating and teaching tonal music

Gabriel presented the Harmony Pad, an electronic musical instrument designed specifically around classical music structures. The user interface for example makes it easier to access harmonically consonant notes and form cadences. Gabriel argued this makes it more fun to play impromptu, bringing the barrier down for people to start improvising at an early stage in their musical training. Playing with the Harmony Pad also implicitly teaches the player about musical structures, learn music theory and harmonization. Very thought out interface layout. Very though out approach. I think it is an interesting step to start designing the layout of an instrument to fit a certain musical style and not even try design it to suit all purposes equally (and end up with instruments that if you play them without really knowing how to, they sound plain out horrible). The risk is, if we have too many of these, does it restrict playing too much? Gabriel’s idea was clearly not to abandon the piano, but play with the Harmony Pad alongside instrument lessons. That is not the easiest way out, so there would always be musicians abandoning the piano for the Harmony Pad.

Antti Jylhä & Cumhur Erkut: Sonic interactions with hand clap sounds

This presentation was about using continuous hand clapping as a way of communicating with the computer, especially looking at the rhythmical pattern of clapping in communication with the machine. Antti presented three demos for clapping, I assume the last one, adjusting the tempo of a song by clapping, most clearly demonstrates the goal of the study. In clap-based tempo modification, the machine implicitly conveys information about the communication by responding to the users claps. However, the clapping rhythm of the person controlling the system is also somehow influenced by the rhythm of the process – it is hard clapping out of sync. Antti made a strong point about this two-way interaction in clapping: you control by clapping, but you also tend to clap in synchrony. In essence, then, the clapping is a negotiation between you and the computer. It is not only control, it is communication. I love the idea! The demo was lagging quite a lot, though; I think really getting into the negotiation would require better time-response. Maybe it would be possible to somehow work with a known lag and synchronize the response with the next beat?

Ulrich Reiter: Toward a Salience Model for Interactive Audiovisual Applications of Moderate Complexity

Ulrich presented work on developing a salience model for quality predicting. His goal is to develop metrics for multimodal quality assessment and guidelines. The work is based on a cognitive approach, looking into concepts like the perceptual cycle, concepts of attention.

Andreas Floros, Nikolaos Grigoriou, Nikolaos Moustakas & Nikolaos Kanellopoulos: dots: and Audio Entertainment Installation using Visual and Spatialbased Interaction

Nikolaos presented dots, an interaction installation using image capture with webcam.

 

Summa summarum: I would say the third AudioMostly had matured into a great event! The couple of days spent in Piteå was like a flowing waltz of extremely well organized sessions, combining interesting papers, demos and presentations and, for once, enough free time in between for passionate exchange between fellow audiophiles. The non-academic activities presented us with a touch of Swedish gourmet (both of the "aah" and "ugh" type), a tour in the acoustically impressive, newly built Acusticum concert hall, some rather civilized barhopping, and even northern lights! All this in exemplary weather. I like!

AudioMostly 2008, Piteå norther Sweden. First day.

Here are some notes from Piteå, day 1. There were no parallel sessions, which was a really good thing, so this is really an overview of the whole conference.

Keynote by Nick Laviers
Nick Laviers is a video game Audio Director, based at EA’s Los Angeles Studio. Nick’s talk was about the music in Red Alert 3 and some of his earlier work such as the Harry Potter games. He took a very broad approach, and in his speech covered the design process both in terms of design decisions as well as the practical aspects of the process, from finding the right composers to organizing orchestral recordings. The talk was really engaging and peppered with useful insights about the realities you face when working on a huge title such as the C&C.
Some strategies I really want to look into:

• Using the memory of play (at restart of level etc.) to influence sounds. Nick’s example was with the background music remembering earlier encounters, I think you could extend that to the acoustic space as well. A room with a ‘worn’ feel.
• "Up yours" music. Basically, playing the other teams music when they really hit you down. That’s almost like the game is taking sides, temporarily abandoning your music to play the other players theme…
• Resting moments, reflection moments. Giving time for the players to absorb the consequences of their actions.

Talks

Mark Grimshaw, Craig A. Lindley & Lennart Nacke: Sound and Immersion in the First-Person Shooter: Objectively Measuring the Player´s Sonic Experience

Mark and colleagues are trying to measure immersion. Their experiment had people playing a Half Life 2 mod with different sound conditions and then measuring their subjective and psychophysiological responses to the sound. Not surprisingly, the subjective reports show people do perceive more ‘immersion’ when the sounds are on. The unexpected result was that they found no sign of difference in the psychophysiological (tonic) data depending on audio condition. Personally, I do think this has something to do with the measurement strategy: using a tonic measurement would probably even out all effects. The thing then is to decide suitable chunks of data for phasic analysis – now that’s a real challenge. It will surely be interesting to see how they proceed.

Daniel Kromand: Sound and the diegesis in survival-horror games

Daniel gave us a fascinating theoretic/analytic investigation into sound in two survival horror game titles: FEAR and SilentHill(2). He’s extending on Joergensen´s notion of the transdiegetic and looks at the tension created through functional use of sound, essentially the very same thing I’ve been working on the last few months. What Daniel is saying is emotional effect exists in the collapse of functional reliability. So, tying it down with the notion of functional fit, if positive emotions come from facilitating gameplay, certain negative emotions (fear/anxiety) are created by selectively making the cues in sound unreliable, this would make it more difficult to use game sounds for building up a strategy.

I had my paper as well on the first day:

Inger Ekman: Psychologically Motivated Techniques for Emotional Sound in Computer Games.

I looked at the complex tangle that is emotion design and what tools a game designer will use to create emotional impact in a game. The crux of my talk were the two competing frameworks for emotion design in games; you could think of them as story-emotions and gameplay-emotions. The two approaches employ partly conflicting strategies in sound, so you should try to find a way for either alternating or going for one of them, but not both at once. I also looked into all the other emotional evaluation that is going on alongside the cognitive evaluation processes and suggested how that maps into the general emotional experience. It seems to me the talk was well received and I was fortunate to have many interesting exchanges about it throughout the rest of the conference. And, how lovely it was to know in the evening of the first day I don’t have to be completely sober presenting the next morning

I’ll be putting up my slides soon and add a link so you can find them.

Louise Valgerður Nickerson & Thomas Hermann: Interactive sonification of grid-based games

Louise presented work on a sonified sudoku game and a four-in-a-row game, basically both really challenging grid-sonifications. The Sudoku had this touch-to-play approach, which would essentially let you use sound as a sort of tactile information and feel your way through the lines. The four-in-a-row game plays constantly, looping all seven piles in two bars of 4/4 beats and pitch representing the columns. I thought it sounded really funky in a cool way. I got to try it out in the evening, and I would say it really is playable and fun. Could be cool to have something like that on a mobile working over a Bluetooth connection!

David Moffat & David Carr: Using audio aids to augment games to be playable for blind people

David presented a sonified Asteroids! game, that is the normal Asteroids! but with added sound. They evaluated the game with sighted players playing either visual-only, audio-only or audio-visual conditions. Interesting results, as the best audio-only player was actually equal with a novice playing in visual-only condition! Adding audio did mostly not affect playing, however there were signs of cognitive owerload for the novice player when playing the audio-visual game condition. Had a very interesting discussion with David afterwards, where we talked about the real challenges for audio gaming and which sonification tasks will be the most challenging to solve as we try to create blind-accessible gaming. He suspected the most difficult tasks will be sonifying abstract things, and that just adding more realistic ‘naturalistic’ representations will get us quite far. I think we have quite a challenge left there, though, because the goal is also really to use ‘naturalistic’ in a suitably non-naturalistic way for other effects, like emotion. So finding a good balance there will be important.

Mats Liljedahl & Nigel Papworth: !BeoWulf - Sound based game

Nigel presented fresh research done with their BeoWulf game, specifically user studies of how people perceive the sounds of BeoWulf. I really enjoy the position they argue – on how sound opens up the creative processes of people and that lack of visual information actually enhances the sense of ‘being there’ instead of the opposite. This research was looking into how people describe the audio-mostly game world (visually they only see a simple map). The results demonstrate the richness of interpretations when using mainly sounds as a stimulus. I wouldn’t myself go so far as to say that the visual information closes down imagination, but certainly sound and visuals work in slightly different ways to open different ‘doors of imagination’.

Scott Beveridge & Don Knox: Control of Sound Environment using Genetic Algorithms

Scott started the presentation by showing an earlier project, a sonified cow (yeah, real one!). We all laughed. Then he went on to describe their current system for creating meaningful sonic representation of crowds. They collect data by webcam, map the crowd actions to sound. The main goal was to strive for musical output, which creates a question of mapping. Too literal mapping breaks up the structure, which will lead to atonality. Too abstract mapping will make the system feel detached and unresponsive.
The system uses direct and indirect translation, directly mapping certain features of an emotion classification engine and mapping them onto an arousal/valence map. The indirect mapping will produce high-level features such as chord progression and melodic contour, using genetic algorithms evaluated for consonance. The system seems interesting and complex enough, it will be interesting to follow their future work, especially as Scott said they are planning for inclusion of suprasegmental features such as melodic contour. That’s a tough one.

Daniel Hug: Genie in a Bottle: Object-Sound Reconfigurations for Interactive Commodities

Daniel is looking at interactive commodities and considering the semantics relating to the sound of objects. Especially interesting are electroacoustically augmented gadgets, intelligent, enhanced commodities that can play with the sound-object relationship. I’ve heard Daniel talk about these things before and I’m always fascinated by the way he ties together information from all these different fields: computer games, film sound, design tradition, sonification… Good overview of different fields with implications for game sound design as well.