Martin Rose's Animation & Media Notes

Sound for Animation

Quick Links
For Animation Program students
Context & Definitions

As animators, we spend an enormous amount of time, and effort, at designing and directing the many aspects of animation for the projects we create. We focus on elements with exacting detail. Everything from what a character looks like, from knowing their backstory and personality, to portraying personality through gestures and telltale silhouette poses, the manipulation of their movement. We apply animation principles with precision and style, timing and pacing, weight, arcs, slow-ins and slow-outs, overlapping actions, anticipation and reaction, and more. With the settings and environments we design, again, much is contemplated: camera angles and scene structure, space and layers, the composition of foreground, middle-ground stage areas, and background regions. We think of light, tone, and shadow to evoke depth and atmosphere, linear and aerial perspective. We carefully engineer numerous qualities of any given frame, exactingly, both to the millimetre and to the pixel. 

We do this to enhance the visual experience. But, overall, we should consider the whole filmic experience, including the aural details, to pair with the visual. Audiences will use both eyes and ears when they take in a film.

To this end, sound design is vitally important and should not be under-considered or overlooked. It’s part of making animation. By applying and manipulating layers of audio, including foreground effects that match with specific character animation and voice / lip synchronization, you will continue to shape stories, situations, and worlds with the same kind of measurement that you employ with visual design.

Without adequate aural details, a film will seem flat and hollow. It will lack in spirit and emotion. Just adding music is not good enough either. Unless it’s a pre-synchronous approach, when you compose visuals and animation to melodic movements, beats, rhythms, and phrases, simply adding music as an afterthought is a mistake. Instead, you should think carefully about what you intend to portray in a film. You need to write and design with sound and music in mind.

With regard to sound designers and editors, an often-used principle is “… we don’t have to see everything but we have to hear everything we see…”

Not involving sound design, on all levels, is an omission, both artistically and technically. The tools and software are at your fingertips and are easily accessible, already part of the programs you already employ at making visual parts of animation production, with Adobe Premiere for instance.

Creatively, so much of the work that you might consider applying visually, with character portrayals and painted, dimensional artwork, can be accomplished, and certainly enhanced, with audio design and effects.

For example, quite simply, along with a visual setting, together with vista details, is sonic ambience, the evidence of a world full of life and movement, situated around the viewer. Sound expands the visual features. It’s beyond the front view. The sonic space will include the two sides, above and behind, panoramic, all around. Breeze, people milling about, footsteps, traffic, wind through leaves and grass, insects and birds. The volume, or intensity, of the sound confirms spatial orientation, how close objects and actions are, sometimes without being seen.

Simultaneous with the visual details, the audience will perceive events in the film and will respond emotionally. We experience settings, events, story, and plot and we are certain to be engaged, intrigued, and convinced.

These are are sound terms that are typically used with sound for animation. 

Acoustic

Concerns and relates to the physical characteristics and behaviour of sound, rather than psychoacoustics or the perception of sound.

Acoustic also covers the manipulation and control of sound in a physical, non-electronic way, such as with an acoustic guitar, which physically amplifies; or with acoustic tiles, which physically attenuates or reduces sound.

Auditory

Concerns and relates to the mechanics of the ear and the organs of hearing, such as the auditory nerve. The term also refers to that which is perceived through, or results from, the sense of hearing.

Aural

The word comes from the Latin “auris” which means “ear.” It is of the ear, what is experienced through, or by means of, the ear. Aural perception involves both hearing and listening abilities.

Audio

Pertains to, or employs, the electronic and digital transmission or reception of sound, such as in radio and television. The term is also used to refer to frequencies or signals in the audible range.

An audio signal refers to a waveform or alternating current, one that represents or is analogous to an acoustic signal. It may be converted into an acoustic signal by means of a loudspeaker.

Electro-acoustic

An adjective that describes any process involving the transfer of a signal from acoustic to electrical form, or inversely. Most commonly, transducers like microphones or loudspeakers are examples of this process.

Although the term more precisely refers to a signal transfer from electrical to acoustic form or inversely, it also refers to any process for the electronic generation and/or manipulation of sound signals, including techniques of sound synthesis for the electronic or digital generation of such signals. When the purpose of such manipulation is artistic, the result is commonly called electroacoustic music.

Loudness

The subjective impression of the intensity or magnitude of a sound. It is dependent on frequency, waveform, and duration, as well as sound intensity or sound pressure. Loudness is expressed quantitatively in units of “sones” and “phons” for “sine tones” or narrow band noise, and in terms of “perceived noise level” (PNdB) for broadband environmental sounds.

Pitch

The subjective impression of frequency, in the same sense that loudness is the subjective sense of the intensity or amplitude of a sound. Pitch is psychoacoustic or perceptual, and the degree of sensitivity varies widely with people. Some individuals have a sense of remembered pitch; that is, they remember it and, compared to others, they do so for some length of time. Other individuals have a sense of absolute pitch, or have “perfect pitch.”

The pitch of a “tone” or “note” allows it to be placed in a musical “scale.” Musical notes of a scale are often called pitches, and are given names like “A, B, C, C#, doh, re, mi,” and so on.

Timbre

Timbre or tone quality is determined by the behaviour in time of the frequency content or spectrum of a sound.

It is the presence and distribution of these frequency components, whether harmonic or inharmonic, and their onset, growth and decay in time. Together, with “phase” relations between them, frequencies combine to give every sound its distinctive tonal quality or timbre.

Often qualities of timbre are described by analogy to colour or texture like “bright, dark, rough, smooth.”

Reverberation or reverb

The persistence of sound after it is produced. It occurs when the signal reflects from acoustic conditions, from things like walls or floor surfaces for example. The sound energy will eventually decay, reduce in amplitude, and be absorbed.

Reverberation is short, in milliseconds. Echo is longer, in longer fractions of a second.

Diegetic and non-diegetic (or extra-diegetic) sound

The term comes from the Greek word that means “to narrate.” 

With diegetic, an audience sees a world and situations of characters in a narrative. A story is shown through their actions.

Combined with non-diegetic, elements of the story are told or recounted.

Diegetic sound is in the story space. The actual sounds, perceived by the characters, are in that space. As audience-observers, we hear the sound as well. 

Such sounds include voices of characters and, what we, animators, call “hard” sound effects, like footsteps, sounds that come from objects and materials that move, and so on.

Music that is heard by both the characters and the audience is diegetic. This includes instruments that figures play in the story space, or from a device like a radio on a kitchen table in a scene.

Non-diegetic sound is outside or beyond the story space and is heard only by the audience. Non-diegetic sound is meant to provide more information, such as the telling from a narrator. Music compositions, like a film score, also “tell,” figuratively, and conjure other senses and emotions. This is not heard by the characters in the space but, obviously, is heard, and is experienced, by the audience.

Conclusion

As animation filmmakers, as sound-and-vision experience creators, we contemplate all the various potential details ahead. We can envision, write, design, produce through recording and effects with software, projects with both diegetic and non-diegetic elements, for strong creative results.

By layering sounds on audio tracks in a program like Adobe Audition, Premiere, or ProTools, you will situate sound by adjusting the amplitude and pitch, and consequently balances the sound to suit the nature of spaces and performances.

With both specific recording techniques, considering acoustics at the source and microphone placement, and/or audio effects that you employ with the software, you can manipulate reverberation. Such qualities also help to define spatial placement and advance visual features of settings or environments. 

Considering the union of ideas, writing, and direction, and, contemplating visual and aural energies, you will design important key synchronization points, when important conceptual strands come together, as meaningful moments, dynamic sequence, and particularly sound-and-image phrases.

By pursing the creative approaches of sound design, you will explore and utilize the medium’s special, distinctive qualities, paired with the visual, to influence the audience’s perception and response to the important directions that you carefully plan and then pursue.

Creative sound design

For directors and practitioners involved with sound design, as an overall motivation, to effectively pair both sonic and visual features together on screen, a goal will be to integrate a compelling aural experience. Projects can evolve in a variety of ways. For instance, it could involve sound effects, where clips are edited precisely to animation in the foreground, and, additionally, background effects and ambience define atmosphere.

Regarding music, you might choose to involve non-diegetic music or use such music only. With the latter, you would carefully and specifically edit movement/animation to enhance the film experience. For example, you would accurately synchronize animation with particular beats, rhythms, and phrases.

In addition to hard effects, ambience, and mixing to augment spatial placement, you might want to apply specific conditions, to emphasize effects and qualities to particular dynamic phrases, like story beats. The intention here is for the events to resonate and be meaningful. Coupling the aural with animation would be distinct and uniquely emphasized.

Depending on the approach you consider, the following notes address further details.

Synchronization of foreground sound effects 

As we know, there is an adage with media production, “… if the audience see actions, they should hear them…” Events on-screen that make sound are in “hard sync.” Visual elements are synchronized with the sonic elements. For instance, footsteps should be matched with appropriate sound effects.

Many, possibly all, of the potential sounds can be located from the Emily Carr Library, Sound Effects & Music by Westar, here.

Lip synchronization with character animation is another example of accurate, to-the-frame matching. In such cases, you would start with a recording and then carefully animate to the track 

To actually situate the effects in visual settings and conditions, you should adjust the intensity or amplitude of the effects, as a way of “mixing” the audio.

See below for further explanation. 

Characteristics of ambience 

If we contemplate the visual scene, with foreground, middle-ground, and background, the setting has depth. At the same time, we should also consider the notion of the aural-scape. Doing so will complement the visual details. 

The audience hears what they see in front of them, where actions make sound. Extending the scene, layers and degrees of sound will provide atmosphere and further experiential detail. This includes the middle-ground and background, but also expands beyond, to involve what may be off-screen — behind, above, and to the sides. There are sounds the audience hears, but does not necessarily see.

Consequently, you should integrate some ambience, such as city sounds (like traffic, people, and other urban sounds), natural qualities (like breeze in trees and grass, birds, insects, waves at the ocean, lake, and riverside), weather (like wind, thunder, and rain), and so on.

Many, possibly all, of the potential sounds can be located from the Emily Carr Library, Sound Effects & Music by Westar, here.

Mixing of clips and tracks / layers 

With animation production, the adjustment of visual elements and layers, of colour, tone, and saturation, is part of compositing. With audio, the adjustment of clips and tracks, of qualities of volume, pitch, and reverberation, is called “mixing.” It’s the sonic equivalent of compositing.

Sound design has similarities to visual design, when spatial qualities are defined by modulating intensity, like amplitude and pitch. Louder, clearer, higher-frequency sounds tend to move forward; subtle, quiet sounds are nearby, close to the viewer, and so on.

Additional creative approaches 

With comprehensive animation filmmaking, you should adjust degrees of amplitude, to focus aspects of the visual scene. For further refinement, you might want to experiment with, and modify, clips and tracks, to apply aspects of pitch and reverberation, to enhance acoustic attributes, and so on. 

In addition to hard effects, ambience, and mixing to augment spatial placement, you might want to apply distinct treatments, to emphasize effects and qualities of particular dynamic phrases, to match specific points in the animation, like with a story beat, to make the events especially resonant and meaningful. Coupling the aural details with animation often emphasizes distinct features of an action or character portrayal.

Regarding music, if you choose non-diegetic music as the primary source of sound, you should work to carefully and specifically edit and synchronize the material, to enhance the movement/animation and film experience. This is particularly so when animation accurately matches particular beats, rhythms, and phrases in the music.

Back to