Citation info: Ekman, I. (2008) "Psychologically Motivated Techniques for Emotional Sound in Computer Games", Proc. AudioMostly 2008, October, Piteå, Sweden, 20-26.

Psychologically Motivated Techniques for Emotional Sound in Computer Games

Inger Ekman
Department of Media Technology,
Helsinki University of Technology
P.O.Box 5400
FIN-02015 HUT

Abstract. One main function of sound in games is to create and enhance emotional impact. The expressive model for game sound has its tradition in sound design for linear audiovisual media: animation and cinema. Current theories on emotional responses to fiction are mainly concerned with linear medial, and only partly applicable to interactive systems like games. The interactivity inherent to games introduces new requirements for sound design, and suggests a break in perception compared with linear media. This work reviews work on emotional responses to fiction and applies them to the area of game sound. The synthesis is interdisciplinary, combining information and insights from a number of fields, including psychology of emotion, film sound theory, experimental research on music perception and philosophy. The paper identifies two competing frameworks for explaining fictional emotions, with specific requirements, and signature techniques for sound design. The role of sound is examined in both cases. The result is a psychologically motivated theory of sound perception capable of explaining the emotional impact of sound in film, as well as identifying the similarities and difference in emotional sound design for these two media.

1 Introduction

The importance of sound for establishing mood in audiovisual media is well recognized. It should come as no surprise that viewing a movie without sound can strip it of its emotional impact, making the events pictured on screen seemingly distant and of little relevance. The potency of sound is also rather well exploited at least in contemporary gaming. The model for sound design in these games is often that of film sound; game sounds should be “Cinematic” [3] or “Bigger than life” [22].

However, it is a well known fact that games differ from film. Also sound design in games is different from film sound on several points. An obvious disparity resides in the technical realization of game sound compared to film sound. For one, the expected technical setups for game and film consumption are of a vastly different nature. Technical aspects are, however, outside the scope of this paper. Instead, the focus is on the special representational requirements that games, as interactive systems, put on the sound design as compared to those within static linear media. Interactivity creates a fundamental differentiation in what constitutes as the sound design for film, and games, respectively. This is to say that the final mix for a film differs by nature from the end result of game sound design, usually a set of programmatically expressed rules for combining/manipulating sound samples as a player interacts with the system. While technology dictates the alternatives on both sides, the distinction between the two is present by nature of film vs. game, and exists independently of technological issues. This aspect is the focus of this investigation.

These are the challenges game sound has to conform to: 1) Whereas film sound is written to a fixed set of actions, seen through predefined points of view, game sounds must support alternative paths and viewpoints through the story. 2) Film sound is written to sequences of actions with known durations. Game sounds must span action sequences of flexible temporal duration. 3) Games ask players to become active and play. Hence, sounds must support action, respond to player control and often survive high repetitiveness. The question we are concerned with here is how to tackle all these issues while maintaining the emotional functions of sound.

Surprisingly little academic investigations into sound have addressed this issue. Also, while there are many books on game sound design, the majority of literature deals either with the technological tools for creating the sounds or managing the overall process of sound design e.g. [2][25]. The best foundations for game sound design decisions can be found in the fragmental accounts written by designers [22][3][13], sharing glimpses of their intuition and experience in the craft through audio post mortems and project descriptions.

Underlying many successful design decisions is an implicit understanding of what kind of sounds and actions create emotional impact. The aim of this study is to bring together various sources of information, to contribute to an explicated version of this knowledge. Such knowledge may ultimately aid the design process, and it should be of benefit to researchers working to understand the phenomenon of game sound. To this purpose the paper will jointly address two close questions, namely why and how sound works to create emotion. Asking why looks for the reason behind having emotional reactions to sound emanating from fictional worlds in the first place. Answering this question requires looking at the relationship between player and game, and the role of sound in shaping this relation. The second question, how, deals with the mechanisms underlying emotional reactions to sound, seeking cognitive and biological explanations to the emotional experiences of a certain sound used in a certain context. Finally, a preliminary synthesis of these findings sketches out some practical sound design implications for emotional impact in film and games, respectively.

This work serves several purposes. First my hope is that an understanding of the underlying processes of sound will provide useful concepts for practical work, by bringing to light the alternative possibilities and approaches for impacting the player experience. This view is in no way attempting to challenge the value of intuition to design, nor to explain away or trivialize the talent that goes into quality sound design. On the contrary, explanations of why and how masterful sound designs influence emotion are as much a celebration of the skills that went into designing them in the first place as tools for shaping new experiences in the future. I am also positive that a bit of clearly enunciated arguments for why sound holds so much power can be of use, especially in defending sound when it has to compete for attention (or budget) with visual effects. Finally, this paper links into the academic discussion on sound design, and provides a synthesis of relevant literature from several vastly different fields. The topic of sound design is still advanced in many separate camps, partly because it is interdisciplinary in nature and that requires tapping into so many fields all at once. The goal in this paper is not to provide an exhaustive review. Rather it offers a selection of findings from different sources, suggesting both where future connections may be found and directions how some of them may be explored further.

The structure of the paper is as follows: Chapter 2 discusses the nature of emotion in general, and looks at emotion theory applied to film and games. Chapter 3 considers the processes wherein film sounds influence emotional response and consider several sources of emotion in sound. Chapter 4 offers a summary on emotional effect of sound and compares emotional sound in film with sound in the interactive context of games.

2 Emotions and Emotional Responses in Film and Games

In psychology literature, whole chapters have been written about defining emotions. Perhaps it is due to the subjective nature of emotional experiences, as well as culturally negotiated nature of what emotions are suitable and for whom, that it is hard to pin down precisely what constitutes an emotion. Intuitively, however, the struggle to define emotion seems puzzling. Most of us would seem to know, by intuition, what sort of experience the word emotion refers to. For the purpose of this text, let us though consider briefly what it implies. Oatley and Jenkins [20] 96] mention the following three defining features:

Emotions are thus evaluations of a specific (visceral and urgent) type, which signal events of critical importance and relevance to the perceiver. They provide the perceiver with an evaluation of the event, producing (typically) either a positive or negative sensation. This sensation is referred to as valence. They also produce a sense of urgency. Generally, the activation part of emotions is called arousal.

2.1 Fictional Emotions?

From a biological viewpoint the function of emotions seems clear. They provide the foundations for successful behaviour in an environment with promises and perils. Emotional relations to film and games, however, is an example of emotional relation to fictional events. This has been a very controversial subject, and arguments go back and forth about whether all these emotional reactions, which have their cause in completely fictive events, are even real emotions at all. And if they are real, what is their cause? To shed some light on these questions, and as an inventory of the tools available for looking at the effects of sound, let us turn for a moment to psychology. What psychological support exists for emotional reaction to fiction?

The simplest objection to calling emotional responses to fiction real is related to the reality of the events underlying these responses. Most emotion theories require that evaluated events should be of real significance to the evaluating subject. This is precisely the concern underlying Tan’s argument [27]. Investigating the emotional responses in film viewing, he bases the validity of his model in Frijda's [9] theory of emotions and discusses in length the compatibility of his theory with Frijda's basic tenets. In short, he argues that emotional responses can arise to fictive events as long as they are perceived as apparent reality. This is achieved mainly by the diegetic effect, a cognitive and perceptual illusion in the viewer' head. The illusion is maintained with several cinematic techniques, usually by dissolving and eradicating the medium. The witness position, Tan argues, is one of the very means in which the cinema excuses the viewer's passivity and explains, in keeping with the diegetic effect, the viewer's ability to see it all.

Another way to make room for fictional emotions is to allow emotional responses not only to real events, but to the fictions of our imagination. Damasio does just so: he calls the latter type as-if emotions, and describes in detail the processes in which the mind simulates affective responses [7]. According to Damasio, fundamentally affective processes are part of all rational decision-making, and function as a means of making decisions – we reason by simulations on how the outcome might feel. From this point of view, fiction would be just another way of running simulations.

2.2 Emotional Responses to Film

There are two dominant approaches to understanding how film elicits emotional responses. One approach is based on Freudian psychoanalysis, invoking as a key concept the Lacanian mirror stage, which “typifies an essential libidinal relationship with the body-image” [14]. This stage is called forth by identifying in film, a process that links the self with the film experience through subconscious mental responses (the basis of which is rooted in early childhood and formations of the Ego). An alternative approach explains the emotional reaction to film through cognitive appraisal of portrayed fictional events. Instead of identifying subconsciously, the viewer responds emotionally because of their cognitive investment in the fictional events. The viewers' emotional responses are related to motivational processes [27][9][16].

According to the appraisal theory, the key to understanding emotions is to understand the role fictive events play to the viewer. Tan identifies two types of emotion at play in the viewing situation: fiction emotions (F emotions, emphatic emotions) and artefact emotions (A emotions, non-empathic emotions). Empathetic (F) emotions are emotional reactions to the story; they require empathy with the main character. Non-empathetic artefact emotions are linked to sensory pleasures – the viewer enjoys the good looks of the protagonist or the beautiful scenery. [27]

Non-empathetic emotions require no appreciation of events. The mechanism by which emphatic emotions are created is more complicated. Tan suggest the film viewer connects to the fiction is through a witness position. This position of agreed-upon inaction (even in the most anxious of moments, we are content to just watch the film) dictates the relations between viewer and film events. Our commitment to the protagonist's cause gives rise to emphatic relations. However, the inactivity dictated by the viewing position also creates tensions, such as worry or frustration when we know more than the protagonist, but cannot change the situation. On the other hand, inactivity may allow us to more deeply empathize, since no action is asked of us.

2.3 Emotions in Games

The interaction in games usually has the player controlling a main character (or other objects in the game). Whereas cognitive appraisal of game events lends itself to an emotional impact, a key question in games is how the narrative reality is maintained. Especially, games remove the passive viewing position, previously suggested a main contributor of empathic effect in film. Also, the typical structure of games – featuring repetitive action, often only loosely bound together by a story – is challenging to empathetic emotion.

On the surface, it would seem that the active nature of players in games break the passivity that Tan proposes so significant in relation to justification of emotions. However, it turns out the activity of games lends itself rather well to the purposes of explaining emotion and the cognitive appraisal model has also been invoked in relation to computer games and emotions [21][15]. Activity, while challenging the passive ride of empathetic emotion, provides an alternative frame in which to evaluate actions.

Perron [21] explicitly works from Tan’s theory to add gameplay emotions (he calls them G emotions), that arise from the cognitive appraisal of game situations. Perron’s G emotions arise just because, in games, the player is invested in acting. Emotional evaluation is fuelled by care for the progress of the game. Lankoski [15] offers a detailed breakdown of emotions within the gaming context, demonstrating how different goal evaluations give rise to basic emotions and their combinations.

In games with a protagonist, gameplay emotions may equal care for the protagonist, but this care is essentially different from empathetic emotion: From the perspective of gameplay, the protagonist is a means, a tool, for playing the game (and achieving the personal goal of completing the task).

Whereas the protagonist can also provide a vessel for empathetic emotions, to some extent, the two frameworks for emotional evaluation - empathetic and gameplay - are competing. As Perron [21] notes, even in games, both the main story line as well as individual plot elements and narrative turning points are indeed furthered from time to time through filmic means, using stills, predetermined animation sequences/ dialogue or cut scenes. During these moments, the player is stripped of control and, effectively, reduced into a witness position. Empathetic emotion comes with loss of activity. A similar point is made by Lankoski [15] when he suggests the empathetic capability of the player is inversely related to the cognitive challenge of action - in the heat of a battle, there is little time to ponder the protagonist's feelings.

3 Roles of Sound in Emotional Appraisal

The appraisal theory would seem to give little explicit explanation of how sound contributes to emotion. The question is to what extent sound influences the process of appraisal. It seems clear that for both film and games, sound is part of the process enforcing the inherent emotionality of events as appraised within each framework. However, the practical and functional approach to sound design is different. In film, sound makes actions seem real and consequential for the viewer, both factors mentioned as prerequisite to empathetic or F emotions [27]. Within this category of emotions, two specific cases can be identified for sound.

One goal of film (and game) production is to make things on screen seem real. A silent two-dimensional representation has limited apparent reality on its own, but adding sound helps perceiving the pictures on screen as physical bodies. Especially important in this aspect is the use of Foley, or synchronized sound effects.

As another case of supporting realism, an integral role for sound at least in classical (Hollywood) film production has been to hide the medium. An example of such is the common practise of continuing sounds over cuts, making them less apparently noticeable. Many other conventions of film sound also contribute to the invisible medium effect, either directly or indirectly. Thus, a role of sound is to create a sense of immediacy.

In the context of games, the significant emotional investment goes into advancing goal-related progress. Most current games rely on both auditory and visual content to represent the game world. Crucially, in the context of interaction, sound comes to take on a new task: facilitator and confirmatory of action. Historically, technical considerations have long dictated tools for sound expression in interactive context, which has forced (and allowed) game sound to deviate from some film sound conventions. The role for sound in games is at least partly dictated by a functional approach and a sound's impact defined by its capability of supporting and facilitating gameplay.

Finally, in discussing empathic emotions we mentioned that some appraisals require no cognitive investment in the story, but are linked purely to sensory pleasures. Interestingly, there is no explicit mention of negative affect in the context of artefact emotions. The same artefact emotions are present in games as well, where part of the game can be enjoyed (or not) apart from aspects of gameplay-related progress.

Soundtrack CD sales gives strong evidence that artefact emotions functions with regards to both film and game music. Beautiful pieces of music obviously give themselves to being appreciated as such, disregarding of whether the viewer is attending to the story. Further, it is motivated to broaden the spectrum of artefact emotions to include also negative effects. With thought to how visual material is used for shock effects (e.g. displaying blood and entrails), it is easy to imagine a similar process in which unpleasant sounds, regardless of story, could produce negative affect. Both in film and games, sound provokes sensory pleasure and displeasure.

3.1 The Realism Fallacy

The above categories identify the main roles of sound in creating and steering the emotional experience. However, aswe shall soon see, there are unresolved contradictions hiding within these categories. Namely, the effects within the two latter categories of sound (gameplay and non-empathetic) seem to contradict the traditional sound design goal of narrative realism.

Appraisal theory of emotion holds that empathetic emotional processing and value judgement is guided by conscious attention to a story, and this cognitive investment is heightened in the perceived realism of portrayal. On the other hand, it allows emotional experiences that arise from appreciation of the artefact: the pictures and sounds of the film/game as such. Similar appreciation is present in games, a medium where technological artistry is elaborately showcased, often even used in promotion.

Tan’s view is that artefact emotions detach the viewer from the story, drawing attention away from the narrative towards the film as artefact, thus making the actions within the narrative less consequential for the viewer. This position raises a complicated question, namely how to interpret such sounds that appear to transgress the borders of realism, despite experientially supporting narrative.

For example, while discussing the effect of sentiment, Tan and Frijda [28] 62] mention sound, especially orchestral music, as one possible source for the awe-inspiring. Awe requires a sense of overwhelming power, and this role is partly to be played by sound. The effect, according to Tan and Frijda, is the emotional function of total submission, a feeling underlying e.g. crying. Sound is thus a tool in portraying power, and heightening sentiment. The question is, by which channel this non-empathetic heightening of sentiment is capable of influencing the (empathetic) evaluation of narrative events.

The problem of border transgression is most apparent in two sound conventions that would seem to deviate from the purpose of reality. In what seems like a blatant contradiction, they invoke a sense of realism in highly unrealistic sounds. One is the use of musical scoring. The other is the use of sound effects that mismatch what is seen on screen. Similar breaches are omnipresent in games, as well, where sound elements effortlessly transgress borders, allowing objects within the story world (diegesis) to refer to non-diegetic space, and vice versa [8][12].

The above concerns are about where to draw the line of realism and about how emotional effects communicate across different categories of judgement. Upon closer scrutiny, both questions generalize to the way non-empathetic affect influences other sources of emotion (empathetic or gameplay- related appraisal). To proceed further, we need an explanation on how non-empathetic emotions arise, and a way of predicting when and how artefact emotions lend emotional meaning to other evaluative processes.

3.2 Unconscious Affective Processes

The theoretical frameworks dealt with above have considered affect by means of perceived experience. Nevertheless, it seems many of the associations and effects of sound in both film and games are working on an unconscious level.Several findings state to the fact that at least some evaluations of stimuli are made precognitively. Zajonc was among the first to point this out in his essay, famously entitled “Feeling and Thinking: Preferences need noInferences” [31]. This view is further fortified by subsequent results from experimental psychology, leading researchers to suggest that there exists such a thing as unconscious emotion [30]. For example, Öhman [32] has demonstrated fear reactions in people who were presented with spider and snake pictures subconsciously; that is, people became frightened of pictures they never even realized they had seen.

In light of unconscious value judgements, it is easy to read Tan's non-empathic emotions are but one example of these: the sounds of the film are invoking emotion by nature of their perceptual properties, unrelated to story at hand. This also opens up a way to understand and predict what would result in a positive (or negative, for that matter) value judgement. One especially interesting factor is the importance of familiarity and perceptual fluency for eliciting positive value judgements [23]. By this account, beauty is defined by the ease in which a stimulus can be processed.

3.3 Misattribution and Making Sense of Emotions

Affective responses include paraphernalia of bodily responses (pounding heart, sweaty palms) and may also bring with them a certain action tendency (e.g. fight-or-flight). In fact, it has been suggested that one possible function in which we do consciously attend to our affective processes is by appreciation of the abrupt changes in the felt background state, or what Russell [24] has called core affect. This change would lead us seeking for a cause of our altered state, leading us via cognitive process to attribute the jolt to the most plausible event in our environment.

Whether or not we are willing to accept unconscious affect as the source of emotions, we can agree that when consciously attended to, emotions tend to have an object. Emotions are evaluations of something. To be able to function properly, we must be able to determine an object for our emotions, something to be afraid of, or pleased by. This distinguishes emotion from moods, which are long-term affective states without an object. However, and here comes the catch, the events we cognitively allocate, as objects do not necessarily have to be the true cause of our initial affective response. In fact, when it comes to reasoning why we feel like we do, we are prone to make mis-attributions and erroneously appreciate our affects quite differently from their real causes even in everyday life.

A classical example of misattribution is a study by Schachter and Singer [26]. They injected subjects with doses of adrenaline, a hormone associated with an excited body state. Depending on the situation that followed the injection, subjects judged their aroused state as either anger or elation. It can be argued that in most cases, our appreciation of our emotional state is at least partly determined by context.

3.4 Misattribution and Emotion in Film and Games

Misattribution is a process wherein the contextual appraisal of perceived emotional ‘raw material’ lending emotional meaning to an outside cause, irrelevant to that particular emotional stir. Now consider a similar process at work during film viewing or when playing a game, with music, sounds, pictures, actions, all mingling to create emotional impacts. Is it not probable, that at some point, the true causes of our feelings might remain oblivious to us? Is it not possible, that we, stirred by our passions, unwittingly, in deciphering the cause of our emotions take them to be caused by whatever the film serves to us on a silver(screen) plate? Could it be that we just happen to attend to a game event, and assume our emotions are caused by that event, when in fact they are not?

This is, essentially, what Annabel Cohen proposes. Cohen [5][6] has dealt extensively with the difficult question why, and how, something as obviously constructed as the film score, does not completely destroy the sense of realism in a film. On the contrary, as many composers will confirm, a carefully chosen (or composed) piece of music will actually heighten the sense of reality in a film. Music also seems to lend a great deal of emotion to events, in ways other than proposed by the cognitive appraisal theory. Cohen's answer lies in a congruence-associationist model of film viewing [18], whereby music focuses attention on those objects in the film that are congruent with the sound. At the same time, the conscious attention is directed away from non-associated sounds, and attending suppressed for stimuli irrelevant for ongoing cognitive processes.

The emotional impact comes from the fact that sounds, even when unattended to, will nevertheless affect perception of objects in the film. Cohen [6] highlights the importance of temporal unity as a binding factor and predictor of which parts of the sound will draw attention. She calls our attention to animation and the technique called mickey-mousing, whereby sound effects are replaced by short musical motifs. Their temporal matching allows these music snippets to replace the original sounds of the events, at the same imbuing both the events, and the objects part of the action, with specific characteristics.

The account on musical meaning in film provides an equally useful tool for approaching the question about other film sounds as well. Applied to object sound, the theory suddenly appears much less mysterious: Consider how we find out properties of objects in real life. What we do is handle the object – tap it, stroke it, bang it against something. By perceiving synchronic sounds, we find out the normal sound of a chair, a balloon or a mandolin. Now, turn that process around and we have precisely what sounds do in a film (and, to a great extent in games as well): now the temporal unity of event and sound defines the object through what sound it makes. Longer chains of events, if temporally matched, cause similar perceptions. When approached from this angle, it is not so odd that sounds in fiction may deviate somewhat from their real life counterparts without seeming false or unrealistic. What may seem surprising is that a whole sum of temporally congruent sounds may become involved in the same process, from simple Foley through more elaborate layers of sound effects all the way to music.

3.5 Where do The Emotions Come From?

So far we have established that unconscious emotional processes may ‘contaminate’ temporally congruent events through misattribution and shown how this may influence perception of events in film. The big question remains, where do the emotions come from?

The most frequently researched category of emotional sound is music. Huron [10] describes musical emotions in terms of fulfilling expectations: the interplay between anticipated and sounded music progression creates patterns of dynamic tension and relaxation. Musical expectations arise from several sources, most of them cultural, but according to some studies at least part of the functions of musical meaning appear to be universal [19].

The existence of universal function of musical emotion suggests there may be other sources of emotion humans draw from in their interpretations of music. The obvious case is that music invokes memories and connotations awaken by that music. However, those would not be universal, even less so than culturally learned expectations. Van Leuwen [17] turns to the human body, suggesting that the most primitive, and also a common link between sound and emotion for all humans, is the perception of our own bodies. Especially the vocal system sets a reference point through simultaneous experience of how it feels and what it requires to produce a certain sound.

Another suggestion is that evaluation of some sounds has biological motivation. This appears to be the fact with the startle response (the phenomenon in which we jump to someone shouting ‘boo’ at us), which aside from providing for pranks also makes us more alert for dangers and automatically directs our attention toward potentially harmful events. However, there may well be other ways in which our perception of sound is evolutionary determined. For example Huron [11] suggests that the perceived cuteness of sounds may be an evolutionary adaptation that promotes parenting.

Value judgements (which are the raw stuff of emotion) seem to be going on even on the lowest level of perception. A classical example within music (and other perceptual) research is the mere exposure effect, wherein a stimulus is judged as likeable merely as the function of familiarity. Investigations into a phenomenon called perceptual fluency suggest that emotional processes are influenced by the very ease of processing [23]. These findings would imply that such things as differences in perceptual clarity (think signal-to-noise ratio) of audio influence the emotional impact of sound, such as perceived beauty or likeability.

The critical requirement for musical expectations to arise is that it is attended to as music. Further, there appears to be boundaries in our listening schemes that separate different styles of listening. Notably, musical listening, in which sounds are perceived as sounds, is not the only form of attending to events. An illustrative deviation from this frame are listening styles provoked by compositional techniques invoking other listening styles, like musique concrète, where the use of real world sounds provokes listening, not at sounds and patterns, but for causes – Chion aptly refers to this as causal listening [4]. This is a special case of music, perhaps seldom used in film, but appearing more and more in games. In these cases, the framework of listening is perhaps more determined by evolutionary and low-level perceptual processes of meaning-making than musical listening modes.

3.6 Realism Revisited

We should now attempt a new understanding of realism. Within fiction, realism is not an absolute, but a nominator for a certain level of fit, an apparent realism or credibility. On the narrative level, realism allows taking the story seriously enough to allow emotions of empathetic quality. Good fit is determined by whether the sound is credible (or illustrative) of a certain sound source [1], pp 190]. When the Foley artist (the person responsible for creating sounds to on-screen events) smashes pumpkins in his studio, he does so in order to produce such sounds with good fit with on screen events. Many times, the sounds produced have little to do with theactual event seen on the film screen – indeed, often non-realistic sounds are purposefully used to make the action sound better. It is, for example, recognized that walking on cornstarch sounds much 'more real' on film than the actual sounds of walking on snow.

A possible explanation underlying the perceived realism of some Foley sounds is the notion of prototypicality. A prototype is an object that inhabits central perceptual characteristics of a given category. Prototypes do not necessarily exist in reality, they are mental constructs of our perceptual system. The prototypical chair is the average of all chair perceptions of your brain, and by definition, it will be the 'chairest' chair of them all. Experimental psychology has established that people perceive prototypes as more easily recognized [27], also more beautiful, and trustworthy [23] than other category members.

Similarly, narrative reality determines how sound behaves within the diegesis, and how the source sounds should sound when listened to from different Points of Audition (1), such as listening behind a wall or under water. At the core, then, also immediacy is but one way of creating a sense of realism. In film, it serves to reinforce the witness position, being present but out of control. Tan [27], pp 25] considers this in his analysis of the camera point of view, mentioning how even in first person view the camera is often a bit off, making space for someone to ‘look over the shoulder’. In the case of sound, the heightened, focussed sound including over-clear dialogue can be considered realistic if we view it as a portrayal not of the scene, but of the experience of listening to the scene. Consider this: while our environment usually contains a multitude of sounds, we only attend to a select few at a time. We are also exemplary at picking out and following these sounds. A person with normal hearing has no difficulty in following a single conversation in a room filled with people, the phenomenon so aptly named the ‘cocktail-party’ effect. Thus, what would seem presented as the films sounds is not the scene itself from a given point in space, but the scene as heard if listened to by attentive ears.

The narrative realism of a sound is thus not in faithful reproduction of sound sources, nor of their environments. The apparent realism of a sound in the context of narrative is defined by how representative a sound is of a certain event. Sounds that are highly representative have good narrative fit. High narrative fit supports empathetic emotion.

Importantly, however, the evaluation of sounds spans several layers. Below the level of narrative meaning are (partly) unconscious processes whereby sounds are judged emotionally. For example, a sound can have good fit narratively, but poor legibility, because the signal-to-noise ratio is so high. Importantly, breaches in this level are disruptive to the perception of sound. We have seen that perceptual fluency is also capable of influencing affective evaluations [23]. Thus, as with the narrative fit, unconscious processing of sound influences emotional judgements. These affects are unrelated to the narrative content of the sound, but tap into the notion of artefact emotions. Depending on their nature, they can cause pleasure or displeasure, which can then be attributed to other temporally congruent events.

Finally, in interactive systems interpretations of sound take on a new role, conveying functional information [29]. By this task, sounds are evaluated on a third level, in how well they serve a functional value, Jørgensen [12], pp 49] refers to this as a sounds functional fidelity. On this stage, emotional evaluations are no longer determined only by the sound itself, but by the utility of a sound for the higher goal of performing goal-related actions. The value of functional sound depends on how the functional aspect supports game progress, the utility of sound. The utility of sound is connected to goal-related cognitive evaluations. High utility enforces gameplay emotion.

4 Comparison of Emotional Sound in Film and Games

The cognitive appraisal framework provides two alternatives for emotional relation to fictive events: the passive witness position allows empathetic emotion, while the active player draws emotional meaning from goal-related evaluation. These two frameworks ask the viewer/gamer to take on a different attitudes towards the fiction and appear to be competing. They also rely on different strategies for sound design.

For film sound, the effort is usually on heightening narrative reality. This is achieved through detailed attention to narrative fit, often striving for high apparent reality. Focus is on advancing the narrative, heightening and clarifying those specific actions that are necessary for following the story’s progress (usually the top priority is on dialogue).

For games, the focus point is different, as games have to support player action. In games, the task of many sounds is primarily to provide feedback about actions. Hence, narrative fit is often sacrificed for utility. To the extent that auditory cues are used to guide actions, they are treated with utmost respect for legibility. For example, even in the case of instructions with diegetic source (a non-player character, voice mail, etc.) it is common that auditory instructions remain heard even if the character runs away from their diegetic source.

An interesting avenue for sound design in games is to shift focus from music to the emotional impacts of Foley and sound effects. A possible alternative for emotionality in games is in environmental sounds, which is already used in many games, where ambient sounds are beautifully merged with musically suggestive elements and event sounds into a sonic landscape in the spirit of musique concrète. However, for this approach to be systematically explored, there is need for better understanding of how everyday sounds influence emotions. In these investigations, theories of unconscious emotion may prove especially informative.

Footnontes

(1) Similar to Point of View in camera techniques, but using sound.

References

[1] Bordwell, D. and Thompson, K. 1985. Fundamental Aesthetics of Sound in the Cinema. In Weis, e. and Belton, J. (eds.) Film Sound, Theory and Practice. Columbia University Press. 181-199.
[2] Brandon, A. 2005. Audio For Games. Planning, Process and Production. New Riders.
[3] Bridgett, B. 2007. Audio Postmortem: Scarface: The World is Yours. Available at http://www.gamasutra.com/features/20070322/bridgett_pfv.htm [Accessed 29.5.2008]
[4] Chion, M. 1994. Audio-Vision. Sound on Screen. (Translation by Claudia Gorbman). Columbia University Press.
[5] Cohen, A. 1990. Understanding musical soundtracks. Empirical Studies of the Arts 8, 111-124.
[6] Cohen, A. 2001. Music as the source of Emotion in Film. In Juslin, P. and Sloboda, J (eds.) Music and Emotion. Oxford University Press. 249-272.
[7] Damasio, A. 2005. Descartes' Error: Emotion, Reason, and the Human Brain. Penguin, paperback reprint (1994).
[8] Ekman, I. 2005. Understanding Sound Effects in Computer Games In Proc. Digital Arts and Cultures 2005, Kopenhagen, Denmark.
[9] Frijda, N. H. 1986. The emotions. Cambridge University.
[10] Huron, D. 2007. Sweet Anticipation: Music and the Psychology of Expectation. MIT press. Paperback reprint (2006).
[11] Huron, D. 2005. The Plural Pleasures of Music. Proc. 2004 Music and Music Science Conference. Kungliga Musikhögskolan & KTH (Royal Institute of Technology), 1-13.
[12] Jørgensen, K. 2007. ‘What are Those Grunts and Growls Over There?’ Computer Game Audio and Player Action. Ph.D. dissertation, Copenhagen University.
[13] Kutay, S. 2006. Bigger Than Big: The Game Audio Explosion. A Guide to Great Game Sound. Available at http://www.gamedev.net/reference/articles/article2317.asp [Accessed 29.5.2008]
[14] Lacan,J. 1951. Some reflections on the ego. (fut lu par Lacan à la British Psycho-Analytical Society le 2 mai 1951) Available at: http://aejcpp.free.fr/lacan/1951-05-02.htm [Accessed 29.5.2008]
[15] Lankoski, P. 2007. Goals, affects, and empathy in games. Paper presented at Philosophy of Computer Games, Reggio Emilia, Italy. Available at: http://www.mlab.uiah.fi/~plankosk/blog/?p=53. [Accessed 29.5.2008]
[16] Lazarus, R. 1991. Emotion and adaptation. Oxford University Press.
[17] Leewen, T. van. 1999. Speech, Music, Sound. Macmillan.
[18] Marshall, S. and Cohen, A. 1988. Effects of Musical Soundtracks on Attitudes toward Animated Geometric Figures. Music Perception 6, 95-112.
[19] Narmour, E. 1990. The analysis and cognition of basic melodic structures: The implication-realization model. University of Chicago Press.
[20] Oatley, K. and Jenkins, J. 1996. Understanding Emotion. Blackwell Publishing.
[21] Perron, B. 2005. A Cognitive Psychological Approach to Gameplay Emotions. Proc. DiGRA 2005 Conference: Changig Views – Worlds in Play.
[22] Prince, B. Tricks and Techniques for Sound Effect Design. Computer Game Developers Conference 1996. Available at http://www.gamasutra.com/features/sound_and_music/081997/sound_effect.htm [Accessed August 20, 2008.]
[23] Reber, R.; Schwarz, N. and Winielman, P. 2004. Processing Fluency and Aesthetic Pleasure: Is Beauty in the Perceiver's Processing Experience? Personality and Social Psychology Review, 8 (4). 364-382.
[24] Russell, J. 2003. Core Affect and the Psychological Construction of Emotion. Psychological Review 110 (1), 145-172.
[25] Sanger, G. 2003. The Fat Man on Game Audio: Tasty Morsels of Sonic Goodness. New Riders.
[26] Schachter, S., & Singer, J. 1962. Cognitive, Social, and Physiological Determinants of Emotional State. Psychological Review, 69, 379-399.
[27] Tan, E. 1994. Film-induced affect as a witness emotion. Poetics 23, 7-32.
[28] Tan, E. and Frijda, N. 1999. Sentiment in Film Viewing. In Plantinga, C. and Smith, G. (eds.) Passionate Views. Film, Cognition, and Emotion. Johns Hopkins. 48-64.
[29] Tuuri, K.; Mustonen, M.-S.; Pirhonen, A. 2007. "Same sound - Different meanings: A novel scheme for modes of listening." Proc. AudioMostly 2007, Ilmenau, Germany.
[30] Winkielman, P. and Berridge, K. 2004. Unconscious Emotion. Current Directions in Psychological Science, 13 (3). 120-123.
[31] Zajonc, R. B. 1980. Feeling and Thinking: Preferences Need No Inferences. American Psychologist, 35, 151-175.
[32] Öhman, A. 2005. The role of the amygdala in human fear: Automatic detection of threat. Psychoneuroendocrinology 30, 953-958.