GB2334867A

GB2334867A - Spatial localisation of sound

Info

Publication number: GB2334867A
Application number: GB9803968A
Authority: GB
Inventors: Adam Dorr
Original assignee: STEELS ELIZABETH ANNE
Current assignee: STEELS ELIZABETH ANNE
Priority date: 1998-02-25
Filing date: 1998-02-25
Publication date: 1999-09-01
Also published as: GB9803968D0

Abstract

A method of processing a digital signal to artificially create a spatial localisation of sound. In particular, a method in which a sound signal is recorded in digital form; the digital signal is processed according to parameters established from a database of differential patterns; the resulting two digital signals corresponding to left and right stereo channels are recorded and the recording is reproduced via headphones or loudspeakers to create the effect of spatially localised sound in three-dimensions.

Description

Spatial Localisation of Sound The present invention relates to the processing of a digitised recording of a sound signal prior to reproduction in standard 2-track stereo to enhance the realism of artificially created spatial localisation of sound.

The existing industry standard of stereo sound is capable of reproducing highly realistic spatial localisation effects. This patent is directed towards the method of recording the digital and analogue signals which are reproduced and amplified by existing equipment.

The result is a further realisation of stereo sound's full potential.

The human brain locates the source of a sound in three-dimensional space by comparing the two slightly different sound signals received by each ear. By a learned process of inference, we are able to unconsciously 'triangulate' the source of a sound in three dimensions based on four major categories, or sonic 'clues': (i) phase-variance (low frequency sound); (ii) intensity difference, or sound-shadow' (high frequency sound); (iii) filtration by the Pinna and resonance in the ear canal; (iv) distance: triangulation, filtration; and (v) movement of the head.

Further information in connection with this can be found in Neurophysiology, 3rd Edition by RHS Carpenter.

It is therefore logical to conclude that if an appropriate sound signal could be supplied to each ear separately, the illusion of a location in three-dimensional space would be created.

It is the object of this invention to provide a method for achieving exactly such an effect.

In order to recreate spatial localisation, each ear must be supplied with the appropriate sound signal corresponding to a specific location in three-dimensional space. It is possible to modify a single digitised sound wave from an existing or new recording such that two new and slightly different signals are produced, corresponding to the left and right ear respectively. These can subsequently be reproduced and amplified in stereo sound.

The invention comprises a method of processing a digital signal to artificially create a spatial localisation of sound. Preferably, the digital signal is processed by means of a differential pattern. The resulting product is two slightly different new digital signals corresponding to standard stereo channels, left and right. The reproduction of these tracks using standard equipment creates the effect of spatially localised sound in three dimensions.

Preferably the differential pattern comprises the combination of wave samples corresponding to the left and right ear. The two slightly different sound signals received by microphones acting as left and right ears of an artificial head would be digitally recorded as wave samples and together would indicate the differential pattern of a particular location in three dimensions of the source of sound.

By compiling a database of many of these differential patterns, a reasonably comprehensive map of sound sources in three dimensions could be created.

In a first embodiment, a digitised sound signal (e.g. any existing or newly digitised sound recording) is compared to a database of differential patterns and altered in exactly the manner in which the two microphones received and altered form of the sample sine wave.

Where sound signal (wave) characteristics (such as amplitude, frequency, wavelength) were affected in the original sample, the same characteristics would be affected in any newly recorded and digitised sample after alteration.

In a second embodiment, a differential pattern in three-dimensional space is calculated from a database of differential patterns. From the map of samples, it would be possible to determine consistent pattems of alteration in the digitised sound wave corresponding to the signal received by the left and right ear, such that any differential pattern in three dimensional space could be calculated.

The invention extends to a method of artificially creating a spatial localisation of sound, in which a sound signal is recorded in digital form; the digital signal is processed according to parameters established from a database of differential patterns; the resulting two digital signals corresponding to left and right stereo channels are recorded; and the recording is reproduced via headphones or loudspeakers to create the effect of spatially localised sound in three dimensions.

It will be appreciated that the two digital signals recorded after processing according to the differential patterns can be treated in a conventional way. For example, they can be mixed as desired, with or without other tracks (which could be similarly processed). They can be mastered onto stereo format (any recording medium, including analogue and digital audio tape, video tape, compact disc, optical disc, magnetic disc or any other analogue or digital data storage medium).

Clearly, it is easier to isolate the sound signal reaching each ear when reproducing stereo sound using headphones than when using loudspeakers. In the latter case, the problem of controlled isolation between stereo left and right signals arises. This phenomenon, known as crosstalk, interferes with the spatial localisation effect described above. However, it is quite possible to isolate the desired signal reaching each ear using a method of wave cancellation By emitting the inverse ofthe sound signal received from the left speaker by the right ear, the right speaker cancels that unwanted sound signal. The left speaker does the same with respect to the sound signal received from the right speaker by the left ear. When operating together, the crosstalk is effectively cancelled out, isolating the signals received from the speakers by each ear such that the left ear effectively hears only the desired signal from the left speaker, and the right ear hears only the desired signal from the right speaker.

Generation of the inverse of the crosstalk sound signals received by each ear would be achieved by using the sample database, and, again, could include the use of the calculating program. Several possible methods of adding the inverse of the sound signals to the primary sound signal of each stereo channel deserve specific attention, each with advantages and limitations.

In a first method of creating the effect of spatial localisation using loudspeakers, the inverse of the crosstalk sound signals are recorded simultaneously with the spatial localisation effect, prior to stereo reproduction. The advantage of this approach is that it would allow the effect to be reproduced on any existing stereo sound reproduction system. However, a limitation of this approach is that the crosstalk location point (i.e. the point in threedimensional space relative to each speaker for which a specific pattern of crosstalk is cancelled using the method described above) is fixed. Therefore, in order to experience the effect, the listener would have to occupy a similarly fixed position in three-dimensional space relative to each of the speakers in order to experience the spatial localisation effect. This method is therefore especially applicable to sound reproduction systems incorporated with personal computer systems, wherein the user remains in a virtually fixed position relative to the speakers during use.

A second method of creating the effect of spatial localisation of sound using loud speakers employs an additional digital signal processor which modifies the recorded sound prior to reproduction through loudspeakers. The additional digital signal processor using the database and possibly a calculating program could supplement any conventional stereo sound reproduction system (e.g. personal, commercial, professional). It would modify existing recorded sound signals before, during or after amplification (i.e. at any step in the reproduction sequence) prior to reproduction through loudspeakers in real time according to specified pararneters. The digital signal processor would be capable of producing the appropriate inverse interference signals for any specific crosstalk location point in three dimensions. In addition, the control of this crosstalk location point in three dimensions would be placed in the hands of the listener for leisurely real time adjustment with a remote control or manually on the digital signal processing unit itself Alternatively, the unit could be programmed or pre-programmed to run specific algorithms of location change (again in real time).

The advantage of this method is that it would allow the user to have a degree of influence over the crosstalk location point, either interactively or in a preordained manner.

Obviously, a limitation of this method is that there is the need for an additional investment in order to enjoy the three dimensional effect through speakers. Further, while there is a possibility for altering the crosstalk location point, this mechanism does not deal with continuous movement of the listener. This method is therefore particularly applicable for users viewing and listening to various forms of stage and screen production, e.g. television, cinema, theatre, musical perforrnance or any other form of entertainment or leisure where there exists an audience e.g. amusement park rides and simulators. Additionally, exotic sonic effects would result from a changing crosstast: location point especially with respect to an audience.

A third method of creating the effect of spatial localisation using either headphones or loudspeakers, employs an additional digital signal processor, capable of adjusting the crosstalk-location-point in real time, which monitors a threedirnensional orientation sensor or tracking device worn by the listener and modifies the crosstalk-location-point on a continuous basis, the recorded sound being modified with respect to this modified crosstalklocation-point prior to reproduction through the headphones or loudspeakers.

The digital signal processor may be incorporated into any conventional sound reproduction system. The orientation sensor or tracking device can be of any kind e.g. signal emission or kinetically based, and is worn preferably on the head or around the neck of the listener. The location of the tracking device would inform the digital signal processor (using any appropriate means, e.g. triangulation from the two receivers connected to the processor) of the location of the listener's head allowing the processor to modify the crosstalk location point accordingly on a continuous basis.

The advantage of this method is that the listener can maintain mobility while still experiencing the spatial localisation effect, corrected by cancellation, while listening to recorded stereo sound reproduced through either headphones or loudspeakers.

Creating this effect using loudspeakers allows for more freedom and comfort than creating this effect using headphones while achieving a similar effect. A limitation of this method is that it will require further investment on behalf of the listener to obtain both the digital signal processor and the signal ernitting device.

Using multiple pairs of speakers and more sophisticated signal processors it may be possible to recreate the spatial localisation effect in more than one location simultaneously using the crosstalk cancellation method described above. In a more complex system containing multiple speakers, the same essential method of producing inverse signals in order to cancel out unwanted crosstalk would be implemented. Each speaker would produce an additional inverse signal in order to cancel the crosstalk from each additional speaker in the system. Therefore, for each additional listener i.e. each additional pair of speakers, it will be possible to create the spatial localisation effect.

The sound recording generated by any of the above mentioned means may be used in combination with orientation sensors or tracking devices worn by the listener (and possibly integrated into headphones) to create spatial localisation relative to the listener's surroundings and thereby give the effect of enhanced realism for the listener. The output of the recording could be modified in real-time such that the illusion of spatial localisation (whether fixed points or movement, or a combination of the two) relative to the listener's surroundings (i.e. a fixed position or movement or any combination of the two relative to the stereo sound reproduction apparatus itself) is created. This is in contrast to the illusion of spatial localisation relative to the listener's ears and head. This effect is, of course, much closer to the manner in which sound is perceived in reality.

As the listener moves their head, they will perceive a change in the spatial localisation of the source of the sound relative to their own head corresponding with maintaining a fixed point relative to their immediate surroundings. As a result, a listener's head movement would provide additional clues as to the location of the sound sources in three-dimensional space in a similar fashion to the way this occurs in reality. Movement of the head is a method of confirmation: after comparing the two slightly different signals received from a sound source by each ear, which creates a differential pattem, the head changes its position and/or orientation in three dimensional space relative to the sound source and generates another differential pattern or patterns. By comparing the differential patterns, the listener's brain can confirm the accuracy of its original assessment of the sound source's spatial localisation (a form of compound differentiation, or compound triangulation). Recreating this method of confirmation as part of the spatial localisation effect would greatly enhance its overall realism, and therefore the enjoyment ofthe listener.

The invention further extends to the use of the method of creating the effect of spatial localisation to create spatially localised tracking of one or more elements through 3dimensional space. This can be any given sound moving from point to point and in any given pattern (i.e. direction or speed). The illusion can be created at any point during the sound recording or reproduction. This tracking effect of spatial localisation offers listeners and recording artists the option of creating the illusion of any pattern of movement through 3dimensional space for a recorded element (for example, a musical instrument or collection of instruments, or a voice or a collection of voices, or any other sound or collection of sounds, or any combination of the three). These patterns could be implemented manually or using pre-programmed algorithms during both the recording and the reproduction process. This creates the possibility of interactive listening, wherein the listener could isolate a particular audio element from the recording (such as a specific instrument) and instruct it to "move" through 3-dimensional space according to any desired pattern in real-time.

As described above, the current industry standard of recording and reproduction of stereo sound fails to make use of differential patterns that exist which define a source of sound in 3-dimensional space. As a result, stereo sound reproductions using conventional methods of recording do not reproduce spatial localisation effects. This invention therefore also comprises the use of a sound recording as created by any of the methods described above, in music, computers, television, cinema or stage production. As stated previously, the spatial localisation effect can be implemented using existing sound recording mediums (analogue and digital audio-tape, video tape, compact disc, magnetic disc, or any other analogue or digital data storage medium) and existing stereo sound reproduction technology.

In the music industry, the capacity to create the illusion of spatial localisation would allow music to be recorded such that each separate track of the recording (e.g. each different instrument, or group of instruments, recorded on a single track) could be affected with the illusion of any specific location or pattern of motion in three < imensional space at a specific time in the recording. This digital signal processing method is no more difficult to incorporate into conventional recordings than other forms of digital signal processing such as flange, chorus, delay, reverb, pitch shifting, distortion, equalisation, phasing, limiting, harmonisation, or excitation.

Equally in computing, the reproduction of sound, including, but not limited to music, is now fully integrated with many sectors of computing. This technology could be particularly useful in the field of virtual reality where the visual media perceived by the user is modified according to the orientation and location of the user's head in two or three dimensions to create the illusion that the visual elements remain fixed relative to the user's surroundings rather than to the user's head. The invention could also be applicable in the area of simulation (recreational, commercial, professional, governmental or otherwise).

Again, this technology is applicable to the mediums of television, cinema and stage production (dramatic, musical or otherwise) in which the use of spatial localisation effects applied to all forms of reproduced sound could provide heightened realism.

The invention may be put into practice in various ways and a general method will be described by way of example with reference to the following drawings, in which: Figure 1 is a block diagram of a method of collecting the data; and Figure 2 is a block diagram of a method of modifying a single recorded and digitised sound in order to generate two slightly different new signals corresponding to the left and right ear.

Referring to figure 1, a single sound, such as a sine wave, produced by a signal generator (such as an oscilloscope or a synthesiser) and a loud speaker (sound source) would generate different response between the left and right microphones on the artificial head. This is due to the factors mentioned above of phase-variance, intensity variance, Pinna filtration and distance. The two slightly different sound signals received by each microphone are digitally recorded as wave samples corresponding to the left and right ear. Their combination would indicate the differential pattem of a particular location in three-dimensional space. In this way, a database of many thousand differential patterns could be generated from which two courses of action are described for example.

Firstly, this map of samples could be used to alter any digitised sound signal (i.e. from any existing or newly digitised sound recording) in exactly the manner in which the left-ear and right-ear microphones received an altered form of the sample sine wave signal. Where sound signal characteristics (amplitudes, frequencies, and wavelengths) were affected in the original samples, these characteristics would be similarly affected in any new recorded and digitised sample.

Alternatively, the rnap of samples could be used to determine consistent patterns of alteration in the digitised sound wave corresponding to the signal received by the left and right ear. Using this method, any differential pattern in three-dimensional space could be calculated using various combinations of the patterns generated. This calculation program would then be used to process recorded sounds.

In figure 2, a sequence for the digital processing of a recorded sound using the sample database is illustrated. The sound source emits a sound signal, such as a previous recording, using conventional means. This sound signal is recorded and converted into digital form (if necessary) using conventional means. The digital sound or track is then processed according to parameters established by the generated database, or the calculating programme.

These parameters are selected for a particular location in three-dimensional space. The product is two slightly different new digital signals or tracks corresponding to standard stereo channels left and right. These tracks are then re-recorded using conventional means (either analogue or digital stereo or multi-track recording) and may be mixed and mastered as desired. The stereo recording is then reproduced in either standard headphones or loudspeakers using one ofthe crosstalk cancellation methods described above.

Claims

CLAIMS 1. A method of processing a digital signal to artificially create a spatial localisation of sound.
2. A method as claimed in claim 1, in which the digital signal is processed by means ofa differential pattern.
3. A method as claimed in Claim 2, in which the differential pattem comprises the combination of wave samples corresponding to the left and right ear.
4. A method as claimed in any preceding claim, in which a digitised sound is altered by comparison with a database of differential pattems.
5. A method as claimed in any one of claims 1 to 3, in which a differential pattern in 3 dimensional space is calculated from a database of differential patterns.
6. A method of artificially creating a spatial localisation of sound, in which a sound signal is recorded in digital form; the digital signal is processed according to parameters established from a database of differential patterns; the resulting two digital signals corresponding to left and right stereo channels are recorded; and the recording is reproduced via headphones or loudspeakers to create the effect of spatially localised sound in three dimensions.
7. A method of creating the effect of spatial localisation using loudspeakers, in which the inverse of the cross talk sound signals are recorded simultaneously with the spatial localisation effect prior to stereo reproduction.
8. A method of creating the effect of spatial localisation using loudspeakers, in which an additional digital signal processor modifies the recorded sound prior to reproduction through said loudspeakers.
9. A method of creating the effect of spatial localisation using loudspeakers, in which an additional digital signal processor, capable of adjusting the crosstalk-location-point in real time, monitors a three-dimensional orientation sensor or tracking device wom by the listener and modifies the crosstalk-location-point on a continuous basis, the recorded sound being modified with respect to this modified crosstalk-location-point prior to reproduction through the loudspeakers.
10. The use of a sound recording as created in any preceding claim in combination with orientation sensors or tracking devices worn by the listener to create spatial localisation relative to the listener's surroundings.
11. The use of the method as claimed in any one of claims 1 to 9 to create spatially localised tracking of one or more elements through 3-dimensional space.
12. The use of a sound recording as created in any one of claims 1 to 9 in music, computers, television, cinema or stage production.
13. A sound recording to artificially create spatial localisation of sound constructed and arranged substantially as herein specifically described.