EP1994526A1 - Joint sound synthesis and spatialization - Google Patents

Joint sound synthesis and spatialization

Info

Publication number
EP1994526A1
EP1994526A1 EP07731685A EP07731685A EP1994526A1 EP 1994526 A1 EP1994526 A1 EP 1994526A1 EP 07731685 A EP07731685 A EP 07731685A EP 07731685 A EP07731685 A EP 07731685A EP 1994526 A1 EP1994526 A1 EP 1994526A1
Authority
EP
European Patent Office
Prior art keywords
spatialization
parameters
source
channel
synthesis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP07731685A
Other languages
German (de)
French (fr)
Other versions
EP1994526B1 (en
Inventor
Grégory PALLONE
Marc Emerit
David Virette
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Orange SA
Original Assignee
France Telecom SA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by France Telecom SA filed Critical France Telecom SA
Priority to PL07731685T priority Critical patent/PL1994526T3/en
Publication of EP1994526A1 publication Critical patent/EP1994526A1/en
Application granted granted Critical
Publication of EP1994526B1 publication Critical patent/EP1994526B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H7/00—Instruments in which the tones are synthesised from a data store, e.g. computer organs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S3/00—Systems employing more than two channels, e.g. quadraphonic
    • H04S3/002—Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
    • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
    • G10H2210/155—Musical effects
    • G10H2210/265—Acoustic effect simulation, i.e. volume, spatial, resonance or reverberation effects added to a musical sound, usually by appropriate filtering or delays
    • G10H2210/295—Spatial effects, musical uses of multiple audio channels, e.g. stereo
    • G10H2210/301—Soundscape or sound field simulation, reproduction or control for musical purposes, e.g. surround or 3D sound; Granular synthesis
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2499/00—Aspects covered by H04R or H04S not otherwise provided for in their subgroups
    • H04R2499/10—General applications
    • H04R2499/11—Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's

Definitions

  • the present invention relates to audio processing and, more particularly, to three-dimensional spatialization of synthetic sound sources.
  • non-parametric methods are known. No particular parameter is used a priori to modify samples previously stored in memory.
  • the best-known representative of these methods is classical wave table synthesis.
  • Parametric synthesis methods which rely on the use of a model that makes it possible to manipulate a small number of parameters, compared to the number of signal samples produced in the sense of the non-parametric methods.
  • Parametric synthesis techniques typically rely on additive, subtractive, source / filter or non-linear models.
  • Some techniques are based on the consideration of HRTFs ("Head Related Transfer Function") transfer functions representing the disturbance of acoustic waves by the morphology of an individual, these HRTFs functions being specific to this individual.
  • HRTFs Head Related Transfer Function
  • the sound reproduction is carried out in a way adapted to the HRTFs of the listener, typically on two remote speakers (“transauraf") or from the two earpieces of a headset (“binaural")
  • Other techniques for example Vamphonic "or the” multichannel "(5.1 to 10.1 or more) provide rather a restitution on more than two speakers.
  • some HRTFs techniques use the separation of the frequency and position variables of the HRTFs, thus giving a set of p basic filters (corresponding to the first p eigenvalues of the covariance matrix of the HRTFs whose variables statistics are the frequencies), these filters being weighted by spatial functions (obtained by projection of the HRTFs on basic filters).
  • the spatial functions can then be interpolated, as described in US-5,500,900.
  • Spatialization of many sound sources can be achieved through a multichannel implementation applied to the signal of each of the sound sources.
  • the gains of the spatialization channels are applied directly to the sound samples of the signal, often described in the time domain (but possibly also in the frequency domain). These samples sound are processed by a spatial isation algorithm (with gain applications that depend on the desired position), irrespective of the origin of these samples.
  • the proposed spatialization could apply to both natural and synthetic sounds.
  • each sound source must be synthesized independently (with a temporal or frequency signal), in order to then be able to apply independent spatialization gains.
  • N sound sources it is therefore necessary to perform N synthesis calculations.
  • the application of the gains to sound samples, whether they come from the time or frequency domain, requires at least as many multiplications as there are samples.
  • Q gains M being the number of intermediate channels (surround channels for example) and N being the number of sources.
  • this technique requires a high calculation cost in the case of the spatialization of many sound sources.
  • the so-called “virtual loudspeakers” method makes it possible to encode the signals to be spatialized by applying them in particular gains, the decoding being done by convolution of the signals encoded by pre-calculated filters (Jérians Daniel, "Representation of acoustic fields, application to the transmission and reproduction of complex sound scenes in a multimedia context", PhD Thesis, 2000).
  • an exemplary embodiment which is referred to in this document WO-05/069272 and in which the sources are synthesized by associating amplitudes at frequencies constituting a "sound tone" (for example a fundamental frequency and its harmonics ), plans to group synthesized signals by identical frequencies, with a view to subsequent spatialization operating on the frequencies.
  • a "sound tone” for example a fundamental frequency and its harmonics
  • FIG. 1 This exemplary embodiment is illustrated in FIG. 1.
  • S N respective amplitudes a 0 1 , ai 1 , ..., a p 1 , ..., a /, ..., a 0 N , a- ⁇ ..., a p N , where, in the general notation al, j is a source index between 1 and N and i is a frequency index between 0 and p.
  • amplitudes of a set ao j , a- ⁇ j , ..., a p j to be assigned to the same source j can be zero if the corresponding frequencies are not represented in the sound signal of this source j .
  • the amplitudes ai 1 , ..., ai N relative to each frequency fi are grouped ("mixed") to be applied, frequency by frequency, to the SPAT spatialization block for encoding operating on the frequencies (in binaural for example, in anticipating interaural delay to be applied to each source).
  • the signals of the channels Ci,..., C k , originating from the spatialization block SPAT, are then intended to be transmitted through one or more networks, or else stored, or other, for the purpose of a subsequent restitution (preceded by where appropriate, a suitable spatialization decoding). This technique, although very promising, still deserves some optimizations.
  • the present invention improves the situation.
  • a method for jointly synthesizing and spatialising a plurality of sound sources in associated positions of the space comprising: a) a step of assigning to each source at least one parameter representative of a amplitude, b) a spatialization step implementing an encoding in a plurality of channels, in which each amplitude parameter is duplicated to multiply by a spatialization gain, each spatialization gain being determined, on the one hand, for an encoding channel and, secondly, for a source to be spatialised, c) a step of grouping the parameters multiplied by the gains, in respective channels, by applying a sum of said multiplied parameters to all the sources for each channel and d) a parametric synthesis step applied to each of the channels.
  • the present invention proposes for this purpose to first apply a spatialization encoding, then a "pseudo-synthesis", the term “pseudo” aiming at the fact that the synthesis applies in particular to the encoded parameters, derived from spatialization and not to usual synthetic sound signals.
  • a feature that the invention proposes is the spatial encoding of some synthesis parameters, rather than performing a spatial encoding of the signals corresponding directly to the sources.
  • This spatial encoding applies more particularly to synthesis parameters which are representative of an amplitude and it advantageously consists in applying to these few synthesis parameters spatialization gains which are calculated as a function of respective desired positions of the sources. It will thus be understood that the parameters multiplied by the gains in step b) and grouped in step c) are not really sound signals, as in the general prior art described above.
  • the present invention uses a mutual parametric synthesis where one of the parameters has the dimension of an amplitude. Unlike techniques of the prior art, it thus takes advantage of the advantages of such a synthesis to perform the spatialization.
  • the combination of synthesis parameter sets obtained for each of the sources advantageously makes it possible to globally control the encoded blocks of mutual parametric synthesis.
  • the present invention then makes it possible to spatialize simultaneously and independently of numerous synthesized sound sources from a parametric synthesis model, the spatialization gains being applied to the synthesis parameters rather than to the samples of the time or frequency domain. This embodiment thus ensures a substantial saving of the computing power required because it implies a low calculation cost.
  • the technique in the sense of the invention requires fewer calculations than the usual techniques in the sense of the prior art. For example, at the surround order 1 and in two dimensions (ie three intermediate channels), the invention already allows a calculation gain for only four sources to spatialize.
  • the present invention also makes it possible to reduce the number of gains to be applied. Indeed, the gains are applied to the synthesis parameters and not to the sound samples. Updating parameters such as the volume is generally less frequent than the sampling frequency of a signal, a calculation economy is thus achieved. For example, for a frequency of updating parameters (such as the volume in particular) of 200 Hz, a substantial saving in multiplication is achieved for a sampling frequency of the signal of 44100 Hz
  • the fields of application of the present invention may concern both the musical field (including polyphonic ringtones of mobiles), the field of multimedia (including video game sound systems), the field of virtual reality (rendering of sound scenes). , simulators (synthesis of engine noise), or others.
  • FIG. 2 illustrates the general processing of spatialization and synthesis provided for in a method according to the invention
  • FIG. 3 illustrates a processing of the spatialized and synthesized signals for spatial decoding with a view to restitution
  • FIG. 4 illustrates a particular embodiment in which several amplitude parameters are assigned to each source, each parameter being associated with a frequency component
  • FIG. 5 illustrates the steps of a method in the sense of the invention, and may correspond to a flowchart of a computer program for the implementation of the invention.
  • At least one parameter Pi is assigned to a source Si from among a plurality of sources Si, ..., S N to be synthesized and spatialized (i being between 1 and N ).
  • Each parameter pi is duplicated in as many spatialization channels provided in the spatialization block SPAT.
  • M encoding channels are provided for the spatialization
  • each parameter pi is duplicated M to apply respective spatialization gains g ⁇ ..., where M (i being, as a reminder, an index of source Si).
  • M i being, as a reminder, an index of source Si
  • new parameters p TM (i varying from 1 to N and m varying from 1 to M) are calculated by multiplying the parameters pi by the encoding gains g TM, obtained from the position of each of the sources.
  • the parameters Pi m are combined (by summation in the example described) to provide the parameters p g m which feed M mutual parametric synthesis blocks.
  • These M blocks constitute the synthesis module SYNTH, which delivers M time or frequency signals ss m (m varying from 1 to M), obtained by synthesis from of the parameters p g m .
  • These signals ss m can then feed a conventional block of spatial decoding, as will be seen below with reference to FIG.
  • the synthesis used is an additive synthesis with application of an inverse Fourier transform (IFFT).
  • IFFT inverse Fourier transform
  • a set of N sources is characterized by a plurality of parameters pi, k representing the amplitude in the frequency domain of the kth frequency component for the ith source Sj.
  • the time signal Si (n) that would correspond to this source If, if it were synthesized independently of the other sources, would be given by:
  • the parameter Pi, k represents the amplitude of a given frequency component k for a given source Si.
  • G gains are predetermined for a desired position for the source Si and according to the selected spatialization encoding.
  • Y m is a spherical harmonic of order m
  • ⁇ i and ⁇ i are respectively the azimuth and the desired site for the Si source.
  • the parameters p m i, k are then combined frequency by frequency, so as to obtain a single global parameter:
  • the value of k 1 is less than ki because common frequencies can characterize several sources at once. In one embodiment, it may be provided to associate the same global set of frequencies to all sources, even if certain amplitude parameters for certain source frequencies are zero.
  • the synthesis step consists of using these parameters p m g , k (m varying from 1 to M) to synthesize each of the M frequency spectra ss m ( ⁇ ) from the SYNTH synthesis module. It may be provided for this purpose to apply the technique described in FR-2,679,689, iteratively adding spectral envelopes corresponding to the Fourier transform of a time window (for example Hanning), these spectral envelopes being previously sampled , tabulated, centered at frequencies fk and then weighted by p m g , k , which is written as:
  • K amplitude parameters p iik are assigned to each source Sj.
  • the index i, of source is between 1 and N.
  • the index k, of frequency is between 1 and K.
  • K parameters are duplicated, M times, to be multiplied each by one gain Spatialization g TM.
  • the index m, of spatialization encoding channel is between 1 and M.
  • the processing then continues by multiplying the global parameter of each subchannel p m g , k associated with a frequency fk by a spectral envelope envk ( ⁇ ) centered at this frequency fk, and this, for all the K subchannels ( k between 1 and K), and globally, for all M channels (m being between 1 and M). Then the sub-channels K are summed in each channel m, according to the following relation:
  • ss m ( ⁇ ") env k ⁇ ), for m ranging from 1 to M channels in total.
  • the signals ss, m m (/ ⁇ ) encoded for their spatialization and synthesized in the sense of the invention are then obtained. They are expressed in the frequency domain.
  • SS m (n) IFFT (ss m ( ⁇ )).
  • successive frames can be achieved by a conventional technique of addition / overlap.
  • SS m (n) can then be supplied to a spatialization decoding block.
  • the processing performed by the spatial decoding DECOD block of FIG. 3 can be of the type:
  • the adaptation filters from the surround format to the binaural format can be applied directly in the frequency domain, thus avoiding convolution in the time domain and a corresponding calculation cost.
  • the spectra are then summed by ear before performing the inverse Fourier transform and the addition / recovery operation, ie:
  • the present invention also relates to a computer program product, whether it is stored in a memory of a central unit or a terminal, or on a removable support adapted to cooperate with a reader of this central unit.
  • This program comprises in particular instructions for the implementation of the method described above and a flowchart can be illustrated by way of example in Figure 5, summarizing the steps of such a method.
  • Step a) is aimed at assigning the parameters representative of an amplitude to each source Sj.
  • a parameter pi, k is assigned by frequency component f k as described above.
  • Step b) aims at the duplication of these parameters and their multiplication by the gains g TM of the encoding channels.
  • Step c) relates to the grouping of the products obtained in step b), with in particular the calculation of their sum on all Si sources.
  • Step d) targets the parametric synthesis with multiplication by a spectral envelope env k as described above, followed by a grouping of the subchannels by applying, in each channel, a sum over all the frequency components (d index k ranging from 1 to K).
  • Step e) aims at decoding the spatialization of the signals ss m originating from the respective channels, synthesized, spatialised and represented in the frequency domain, for a reproduction on two loudspeakers, for example in binaural format.
  • the present invention also provides a device for generating synthetic and spatialized sounds, comprising in particular a processor, and in particular a working memory adapted to store instructions of the computer program product defined above.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Stereophonic System (AREA)
  • Telephone Set Structure (AREA)
  • Telephone Function (AREA)
  • Golf Clubs (AREA)
  • Circuit For Audible Band Transducer (AREA)

Abstract

The invention concerns a process for joint synthesis and spatialization of multiple sound sources in associated spatial positions, including: a) a step of assigning to each source at least one parameter (pi) representing an amplitude; b) a step of spatialization consisting in implementing an encoding into a plurality of channels, wherein each amplitude (pi) is duplicated to be multiplied to a specialization gain (gim), each spatialization gain being determined for one encoding channel (pgm) and for a source to be spatialized (Si); c) a step of grouping (R) the parameters multiplied by the gains (Pim), in respective channels (pg1,..., pgM), by applying a sum of said multiplied parameters (pim) on all the sources (Si) for each channel (pgm), and d) a step of parametric synthesis (SYNTH(I ),..., SYNTH(M)) applied to each of the channels (pgm).

Description

Synthèse et spatialisation sonores conjointes Joint sound synthesis and spatialization
La présente invention concerne un traitement audio et, plus particulièrement, une spatialisation tridimensionnelle de sources sonores synthétiques.The present invention relates to audio processing and, more particularly, to three-dimensional spatialization of synthetic sound sources.
Actuellement, la spatialisation d'une source sonore synthétique est réalisée souvent sans tenir compte du mode de production du son, c'est-à-dire de la manière même dont est synthétisé le son. Ainsi, de nombreux modèles, notamment paramétriques, ont été proposés pour la synthèse. Parallèlement, de nombreuses techniques de spatialisation ont été aussi proposées, sans toutefois proposer un recoupement avec la technique choisie pour une synthèse.Currently, the spatialization of a synthetic sound source is often performed without taking into account the sound production mode, that is to say, the way in which the sound is synthesized. Thus, many models, including parametric, have been proposed for synthesis. At the same time, many spatialization techniques have also been proposed, without proposing a cross-check with the technique chosen for a synthesis.
On connaît, parmi les techniques de synthèse, les méthodes dites "non paramétriques". Aucun paramètre particulier n'est utilisé a priori pour modifier des échantillons précédemment stockés en mémoire. Le représentant le plus connu de ces méthodes est la synthèse par table d'onde classique.Among the synthetic techniques, the so-called "non-parametric" methods are known. No particular parameter is used a priori to modify samples previously stored in memory. The best-known representative of these methods is classical wave table synthesis.
A ce type de technique s'opposent les méthodes de synthèse "paramétrique" qui reposent sur l'utilisation d'un modèle permettant de manipuler un nombre réduit de paramètres, comparé au nombre d'échantillons de signaux produits au sens des méthodes non paramétriques. Les techniques de synthèse paramétriques reposent typiquement sur des modèles additifs, soustractifs, source/filtre ou non-linéaires.This type of technique is opposed by "parametric" synthesis methods, which rely on the use of a model that makes it possible to manipulate a small number of parameters, compared to the number of signal samples produced in the sense of the non-parametric methods. Parametric synthesis techniques typically rely on additive, subtractive, source / filter or non-linear models.
Parmi ces méthodes paramétriques, on qualifie de "mutuelles" celles qui permettent de manipuler en commun des paramètres correspondant à des sources sonores différentes, pour n'utiliser alors qu'un seul processus de synthèse, néanmoins pour la totalité des sources. Dans les méthodes dites "sinusoïdales", typiquement, on construit un spectre en fréquence à partir des paramètres tels que l'amplitude et la fréquence de chaque composante partielle du spectre sonore global des sources. En effet, une implémentation par transformée de Fourier inverse, suivie d'une addition/recouvrement, assure une synthèse extrêmement efficace de plusieurs sources sonores simultanément.Among these parametric methods, those which make it possible to manipulate parameters corresponding to different sound sources, so as to use only one synthesis process, nevertheless for all sources, are called "mutuals". In so-called "sinusoidal" methods, typically, a frequency spectrum is constructed from parameters such as the amplitude and frequency of each component. partial of the overall sound spectrum of the sources. Indeed, an implementation by inverse Fourier transform, followed by an addition / overlap, ensures an extremely efficient synthesis of several sound sources simultaneously.
Pour ce qui concerne la spatial isation de sources sonores, différentes techniques sont connues actuellement. Certaines techniques (comme le "transaural" ou le "binaural") se basent sur la prise en compte de fonctions de transfert HRTFs (pour "Head Related Transfer Function") représentant la perturbation d'ondes acoustiques par la morphologie d'un individu, ces fonctions HRTFs étant propres à cet individu. La restitution sonore s'effectue de façon adaptée aux HRTFs de l'auditeur, typiquement sur deux haut- parleurs distants ("transauraf) ou à partir des deux oreillettes d'un casque ("binaural"). D'autres techniques (par exemple Vambiophonique" ou le "multicanal" (5.1 à 10.1 ou plus) prévoient plutôt une restitution sur plus de deux haut-parleurs.As far as the spatialization of sound sources is concerned, different techniques are currently known. Some techniques (such as "transaural" or "binaural") are based on the consideration of HRTFs ("Head Related Transfer Function") transfer functions representing the disturbance of acoustic waves by the morphology of an individual, these HRTFs functions being specific to this individual. The sound reproduction is carried out in a way adapted to the HRTFs of the listener, typically on two remote speakers ("transauraf") or from the two earpieces of a headset ("binaural") Other techniques (for example Vamphonic "or the" multichannel "(5.1 to 10.1 or more) provide rather a restitution on more than two speakers.
Plus précisément, certaines techniques à base de HRTFs utilisent la séparation des variables "fréquence" et "position" des HRTFs, donnant ainsi un ensemble de p filtres de base (correspondant aux p premières valeurs propres de la matrice de covariance des HRTFs dont les variables statistiques sont les fréquences), ces filtres étant pondérés par des fonctions spatiales (obtenues par projection des HRTFs sur des filtres de base). Les fonctions spatiales peuvent ensuite être interpolées, comme décrit dans le document US-5,500,900.More specifically, some HRTFs techniques use the separation of the frequency and position variables of the HRTFs, thus giving a set of p basic filters (corresponding to the first p eigenvalues of the covariance matrix of the HRTFs whose variables statistics are the frequencies), these filters being weighted by spatial functions (obtained by projection of the HRTFs on basic filters). The spatial functions can then be interpolated, as described in US-5,500,900.
La spatialisation de nombreuses sources sonores peut être réalisée grâce à une implémentation multicanale appliquée au signal de chacune des sources sonores. Les gains des canaux de spatialisation sont appliqués directement aux échantillons sonores du signal, souvent décrits dans le domaine temporel (mais possiblement aussi dans le domaine fréquentiel). Ces échantillons sonores sont traités par un algorithme de spatial isation (avec applications de gains qui sont fonction de la position désirée), et ce, indépendamment de l'origine de ces échantillons. Ainsi, la spatialisation proposée pourrait s'appliquer aussi bien à des sons naturels qu'à des sons synthétiques.Spatialization of many sound sources can be achieved through a multichannel implementation applied to the signal of each of the sound sources. The gains of the spatialization channels are applied directly to the sound samples of the signal, often described in the time domain (but possibly also in the frequency domain). These samples sound are processed by a spatial isation algorithm (with gain applications that depend on the desired position), irrespective of the origin of these samples. Thus, the proposed spatialization could apply to both natural and synthetic sounds.
D'une part, chaque source sonore doit être synthétisée indépendamment (avec obtention d'un signal temporel ou fréquentiel), afin de pouvoir appliquer ensuite des gains de spatialisation indépendants. Pour N sources sonores, il est donc nécessaire de réaliser N calculs de synthèse. D'autre part, l'application des gains à des échantillons sonores, qu'ils soient issus du domaine temporel ou fréquentiel, nécessite au minimum autant de multiplications qu'il y a d'échantillons. Pour un bloc de Q échantillons, il est donc nécessaire d'appliquer au moins N. M. Q gains, M étant le nombre de canaux intermédiaires (canaux ambiophoniques par exemple) et N étant le nombre de sources.On the one hand, each sound source must be synthesized independently (with a temporal or frequency signal), in order to then be able to apply independent spatialization gains. For N sound sources, it is therefore necessary to perform N synthesis calculations. On the other hand, the application of the gains to sound samples, whether they come from the time or frequency domain, requires at least as many multiplications as there are samples. For a block of Q samples, it is therefore necessary to apply at least N. M. Q gains, M being the number of intermediate channels (surround channels for example) and N being the number of sources.
Ainsi, cette technique nécessite un coût de calcul élevé dans le cas de la spatialisation de nombreuses sources sonores.Thus, this technique requires a high calculation cost in the case of the spatialization of many sound sources.
Parmi les techniques ambiophoniques, la méthode dite des "haut-parleurs virtuels" permet d'encoder les signaux à spatialiser en leur appliquant en particulier des gains, le décodage étant réalisé par convolution des signaux encodés par des filtres pré-calculés (Jérôme Daniel, "Représentation de champs acoustiques, application à la transmission et à la reproduction de scènes sonores complexes dans un contexte multimédia", Thèse de doctorat, 2000).Among the ambiophonic techniques, the so-called "virtual loudspeakers" method makes it possible to encode the signals to be spatialized by applying them in particular gains, the decoding being done by convolution of the signals encoded by pre-calculated filters (Jérôme Daniel, "Representation of acoustic fields, application to the transmission and reproduction of complex sound scenes in a multimedia context", PhD Thesis, 2000).
Une technique très prometteuse, combinant synthèse et spatialisation, a été présentée dans le document WO-05/069272.A very promising technique, combining synthesis and spatialization, has been presented in document WO-05/069272.
Elle consiste à déterminer des amplitudes à affecter à des signaux représentant des sources sonores pour définir, à la fois, l'intensité sonore (par exemple un "volume") d'une source à synthétiser et un gain de spatialisation de cette source. Ce document divulgue notamment une spatialisation binaurale avec prise en compte des retards et des gains (ou "fonctions spatiales") et, en particulier, un mixage des sources synthétisées dans la partie encodage de spatialisation.It consists in determining amplitudes to be assigned to signals representing sound sources to define, at the same time, the loudness (by example a "volume") of a source to synthesize and a gain of spatialization of this source. This document notably discloses a binaural spatialization taking into account delays and gains (or "spatial functions") and, in particular, a mixing of the sources synthesized in the encoding part of spatialization.
Plus particulièrement encore, un exemple de réalisation qui est visé dans ce document WO-05/069272 et dans lequel les sources sont synthétisées en associant des amplitudes à des fréquences constitutives d'un "timbre sonore" (par exemple une fréquence fondamentale et ses harmoniques), prévoit de regrouper par fréquences identiques des signaux de synthèse, en vue d'une spatialisation ultérieure opérant sur les fréquences.More particularly still, an exemplary embodiment which is referred to in this document WO-05/069272 and in which the sources are synthesized by associating amplitudes at frequencies constituting a "sound tone" (for example a fundamental frequency and its harmonics ), plans to group synthesized signals by identical frequencies, with a view to subsequent spatialization operating on the frequencies.
Cet exemple de réalisation est illustré sur la figure 1. Dans un bloc de synthèse SYNTH (représenté en traits pointillés), on affecte à des fréquences fo, fi, f2, ..., fP de chaque source à synthétiser S-i, ..., SN des amplitudes respectives a0 1, a-i1, ..., ap 1, ..., a/, ..., a0 N, a-Λ ..., ap N, où, dans la notation générale al, j est un indice de source compris entre 1 et N et i est un indice de fréquence compris entre 0 et p. Bien entendu, certaines amplitudes d'un jeu aoj, a-ιj, ..., ap j à affecter à une même source j peuvent être nulles si les fréquences correspondantes ne sont pas représentées dans le timbre sonore de cette source j.This exemplary embodiment is illustrated in FIG. 1. In a SYNTH synthesis block (represented in dashed lines), the frequencies fo, fi, f2,..., F P of each source to be synthesized Si are assigned to. . S N respective amplitudes a 0 1 , ai 1 , ..., a p 1 , ..., a /, ..., a 0 N , a-Λ ..., a p N , where, in the general notation al, j is a source index between 1 and N and i is a frequency index between 0 and p. Of course, certain amplitudes of a set ao j , a-ι j , ..., a p j to be assigned to the same source j can be zero if the corresponding frequencies are not represented in the sound signal of this source j .
Les amplitudes ai1, ..., aiN relatives à chaque fréquence fi sont regroupées ("mixées") pour être appliquées, fréquence par fréquence, au bloc de spatialisation SPAT pour un encodage opérant sur les fréquences (en binaural par exemple, en prévoyant alors un retard interaural à appliquer à chaque source). Les signaux des canaux Ci, ..., ck, issus du bloc de spatialisation SPAT, sont ensuite destinés à être transmis à travers un ou plusieurs réseaux, ou encore stockés, ou autres, en vue d'une restitution ultérieure (précédée le cas échéant d'un décodage de spatialisation adapté). Cette technique, quoique très prometteuse, mérite encore quelques optimisations.The amplitudes ai 1 , ..., ai N relative to each frequency fi are grouped ("mixed") to be applied, frequency by frequency, to the SPAT spatialization block for encoding operating on the frequencies (in binaural for example, in anticipating interaural delay to be applied to each source). The signals of the channels Ci,..., C k , originating from the spatialization block SPAT, are then intended to be transmitted through one or more networks, or else stored, or other, for the purpose of a subsequent restitution (preceded by where appropriate, a suitable spatialization decoding). This technique, although very promising, still deserves some optimizations.
De manière générale, les procédés actuels requièrent des puissances de calcul notables pour spatialiser de nombreuses sources sonores synthétisées.In general, current methods require significant computing power to spatialize many synthesized sound sources.
La présente invention vient améliorer la situation.The present invention improves the situation.
Elle propose à cet effet un procédé pour synthétiser et spatialiser conjointement une pluralité de sources sonores dans des positions associées de l'espace, le procédé comportant : a) une étape d'affectation à chaque source d'au moins un paramètre représentatif d'une amplitude, b) une étape de spatialisation mettant en œuvre un encodage en une pluralité de canaux, dans laquelle on duplique chaque paramètre d'amplitude pour le multiplier par un gain de spatialisation, chaque gain de spatialisation étant déterminé, d'une part, pour un canal d'encodage et, d'autre part, pour une source à spatialiser, c) une étape de regroupement des paramètres multipliés par les gains, dans des canaux respectifs, en appliquant une somme desdits paramètres multipliés sur toutes les sources pour chaque canal, et d) une étape de synthèse paramétrique appliquée à chacun des canaux.To this end, it proposes a method for jointly synthesizing and spatialising a plurality of sound sources in associated positions of the space, the method comprising: a) a step of assigning to each source at least one parameter representative of a amplitude, b) a spatialization step implementing an encoding in a plurality of channels, in which each amplitude parameter is duplicated to multiply by a spatialization gain, each spatialization gain being determined, on the one hand, for an encoding channel and, secondly, for a source to be spatialised, c) a step of grouping the parameters multiplied by the gains, in respective channels, by applying a sum of said multiplied parameters to all the sources for each channel and d) a parametric synthesis step applied to each of the channels.
Ainsi, la présente invention propose à cet effet d'appliquer d'abord un encodage en spatialisation, puis une "pseudo-synthèse", le terme "pseudo" visant le fait que la synthèse s'applique en particulier aux paramètres encodés, issus de la spatialisation et non à des signaux sonores synthétiques habituels.Thus, the present invention proposes for this purpose to first apply a spatialization encoding, then a "pseudo-synthesis", the term "pseudo" aiming at the fact that the synthesis applies in particular to the encoded parameters, derived from spatialization and not to usual synthetic sound signals.
En effet, une particularité que propose l'invention est l'encodage spatial de quelques paramètres de synthèse, plutôt que de réaliser un encodage spatial des signaux correspondant directement aux sources. Cet encodage spatial s'applique plus particulièrement à des paramètres de synthèse qui sont représentatifs d'une amplitude et il consiste avantageusement à appliquer à ces quelques paramètres de synthèse des gains de spatialisation qui sont calculés en fonction de positions souhaitées respectives des sources. On comprendra ainsi que les paramètres multipliés par les gains à l'étape b) et regroupés à l'étape c) ne sont pas réellement des signaux sonores, comme au sens de l'art antérieur général décrit ci-avant.Indeed, a feature that the invention proposes is the spatial encoding of some synthesis parameters, rather than performing a spatial encoding of the signals corresponding directly to the sources. This spatial encoding applies more particularly to synthesis parameters which are representative of an amplitude and it advantageously consists in applying to these few synthesis parameters spatialization gains which are calculated as a function of respective desired positions of the sources. It will thus be understood that the parameters multiplied by the gains in step b) and grouped in step c) are not really sound signals, as in the general prior art described above.
La présente invention utilise alors une synthèse paramétrique mutuelle où l'un des paramètres possède la dimension d'une amplitude. Contrairement aux techniques de l'art antérieur, elle tire ainsi partie des avantages d'une telle synthèse pour effectuer la spatialisation. La combinaison des jeux de paramètres de synthèse obtenus pour chacune des sources permet avantageusement de contrôler globalement les blocs encodés de synthèse paramétrique mutuelle.The present invention then uses a mutual parametric synthesis where one of the parameters has the dimension of an amplitude. Unlike techniques of the prior art, it thus takes advantage of the advantages of such a synthesis to perform the spatialization. The combination of synthesis parameter sets obtained for each of the sources advantageously makes it possible to globally control the encoded blocks of mutual parametric synthesis.
La présente invention permet alors de spatialiser simultanément et indépendamment de nombreuses sources sonores synthétisées à partir d'un modèle de synthèse paramétrique, les gains de spatialisation étant appliqués aux paramètres de synthèse plutôt qu'aux échantillons du domaine temporel ou fréquentiel. Cette réalisation assure alors une économie substantielle de la puissance de calcul requise car elle implique un faible coût de calcul.The present invention then makes it possible to spatialize simultaneously and independently of numerous synthesized sound sources from a parametric synthesis model, the spatialization gains being applied to the synthesis parameters rather than to the samples of the time or frequency domain. This embodiment thus ensures a substantial saving of the computing power required because it implies a low calculation cost.
Selon l'un des avantages que procure l'invention, comme le nombre d'étapes dans la synthèse est rendu indépendant par rapport au nombre de sources, une seule synthèse par canal intermédiaire peut être appliquée. Quel que soit le nombre de sources sonores, seul un nombre constant M de calculs de synthèse est prévu. Typiquement, dès lors que le nombre de sources N devient plus grand que le nombre M de canaux intermédiaires, la technique au sens de l'invention nécessite moins de calculs que les techniques habituelles au sens de l'art antérieur. Par exemple, à l'ordre ambiophonique 1 et en deux dimensions (soit trois canaux intermédiaires), l'invention permet déjà un gain de calcul pour seulement quatre sources à spatialiser.According to one of the advantages provided by the invention, since the number of steps in the synthesis is made independent with respect to the number of sources, only one intermediate channel synthesis can be applied. Regardless of the number of sound sources, only a constant number M of synthesis calculations is planned. Typically, since the number of sources N becomes larger than the number M of intermediate channels, the technique in the sense of the invention requires fewer calculations than the usual techniques in the sense of the prior art. For example, at the surround order 1 and in two dimensions (ie three intermediate channels), the invention already allows a calculation gain for only four sources to spatialize.
La présente invention permet aussi de diminuer le nombre de gains à appliquer. En effet, les gains sont appliqués aux paramètres de synthèse et non aux échantillons sonores. La mise à jour des paramètres tels que le volume étant généralement moins fréquente que la fréquence d'échantillonnage d'un signal, une économie de calcul est ainsi réalisée. Par exemple, pour une fréquence de mise à jour de paramètres (tel que le volume notamment) de 200Hz, on réalise une économie de multiplications substantielle pour une fréquence d'échantillonnage du signal de 44100HzThe present invention also makes it possible to reduce the number of gains to be applied. Indeed, the gains are applied to the synthesis parameters and not to the sound samples. Updating parameters such as the volume is generally less frequent than the sampling frequency of a signal, a calculation economy is thus achieved. For example, for a frequency of updating parameters (such as the volume in particular) of 200 Hz, a substantial saving in multiplication is achieved for a sampling frequency of the signal of 44100 Hz
(selon un rapport d'environ 200).(according to a ratio of about 200).
Les champs d'application de la présente invention peuvent concerner aussi bien le domaine musical (notamment les sonneries polyphoniques de mobiles), le domaine du multimédia (notamment les sonorisations de jeux vidéo), le domaine de la réalité virtuelle (rendu de scènes sonores), les simulateurs (synthèse de bruits moteurs), ou autres.The fields of application of the present invention may concern both the musical field (including polyphonic ringtones of mobiles), the field of multimedia (including video game sound systems), the field of virtual reality (rendering of sound scenes). , simulators (synthesis of engine noise), or others.
D'autres caractéristiques et avantages de l'invention apparaîtront à l'examen de la description détaillée ci-après, et des dessins annexés sur lesquels, outre la figure 1 relative à l'art antérieur et décrite ci avant :Other features and advantages of the invention will appear on examining the detailed description below, and the accompanying drawings in which, in addition to Figure 1 relating to the prior art and described above:
- la figure 2 illustre le traitement général de spatial isation et synthèse prévu dans un procédé au sens de l'invention, - la figure 3 illustre un traitement des signaux spatialisés et synthétisés, pour un décodage spatial en vue d'une restitution,FIG. 2 illustrates the general processing of spatialization and synthesis provided for in a method according to the invention; FIG. 3 illustrates a processing of the spatialized and synthesized signals for spatial decoding with a view to restitution,
- la figure 4 illustre un mode de réalisation particulier dans lequel on affecte plusieurs paramètres d'amplitude à chaque source, chaque paramètre étant associé à une composante fréquentielle, - la figure 5 illustre les étapes d'un procédé au sens de l'invention, et peut correspondre à un organigramme d'un programme d'ordinateur pour la mise en œuvre de l'invention.FIG. 4 illustrates a particular embodiment in which several amplitude parameters are assigned to each source, each parameter being associated with a frequency component, FIG. 5 illustrates the steps of a method in the sense of the invention, and may correspond to a flowchart of a computer program for the implementation of the invention.
En référence à la figure 2, on affecte au moins un paramètre Pi, représentatif d'une amplitude, à une source Si parmi une pluralité de sources S-i, ..., SN à synthétiser et spatialiser (i étant compris entre 1 et N). On duplique chaque paramètre pi en autant de canaux de spatialisation prévus dans le bloc de spatialisation SPAT. Dans l'exemple représenté où l'on prévoit M canaux d'encodage pour la spatialisation, on duplique M fois chaque paramètre pi pour appliquer des gains de spatialisation respectifs gΛ ..., gιM (i étant, pour rappel, un indice de source Si). On obtient alors N. M paramètres multipliés chacun par un gain : p-igΛ ...,With reference to FIG. 2, at least one parameter Pi, representative of an amplitude, is assigned to a source Si from among a plurality of sources Si, ..., S N to be synthesized and spatialized (i being between 1 and N ). Each parameter pi is duplicated in as many spatialization channels provided in the spatialization block SPAT. In the example shown where M encoding channels are provided for the spatialization, each parameter pi is duplicated M to apply respective spatialization gains gΛ ..., where M (i being, as a reminder, an index of source Si). We then obtain N.M parameters each multiplied by a gain: p-igΛ ...,
Pi9iM, ---, PigΛ ---, Pi9iM, ---, PNgN1, ..., PNgNM. On regroupe ensuite ces paramètres multipliés (référence R de la figure 2) par canaux de spatialisation (M canaux en tout), soit :Pi9i M , ---, PigΛ ---, Pi9i M , ---, PNgN 1 , ..., PNgN M. These multiplied parameters (reference R of FIG. 2) are then grouped by spatialization channels (M channels in all), namely:
- pigΛ ..., pigΛ ..., PNgN1 regroupés dans un premier canal de spatialisation- pigΛ ..., pigΛ ..., PNgN 1 grouped in a first spatialization channel
P9 1. et ce, jusqu'à : - pigiM, ..., PigiM, ..., PNgNM regroupés dans un M'eme canal de spatialisationP 9 1 . and this, up to: - pigi M , ..., Pigi M , ..., PNgN M grouped in a same channel of spatialization
la lettre g de l'indice désignant le terme "globaf.the letter g of the index designating the term "globaf.
Ainsi, de nouveaux paramètres p ™ (i variant de 1 à N et m variant de 1 à M) sont calculés par multiplication des paramètres pi par les gains d'encodage g ™, obtenus à partir de la position de chacune des sources. Les paramètres Pim sont combinés (par sommation dans l'exemple décrit) afin de fournir les paramètres pg m qui alimentent M blocs de synthèse paramétrique mutuelle.Thus, new parameters p ™ (i varying from 1 to N and m varying from 1 to M) are calculated by multiplying the parameters pi by the encoding gains g ™, obtained from the position of each of the sources. The parameters Pi m are combined (by summation in the example described) to provide the parameters p g m which feed M mutual parametric synthesis blocks.
Ces M blocs (référencés SYNTH(I ) à SYNTH(M) sur la figure 2) sont constitutifs du module de synthèse SYNTH, lequel délivre M signaux temporels ou fréquentiels ssm (m variant de 1 à M), obtenus par synthèse à partir des paramètres pg m. Ces signaux ssm peuvent ensuite alimenter un bloc classique de décodage spatial, comme on le verra plus loin en référence à la figure 3.These M blocks (referenced SYNTH (I) to SYNTH (M) in FIG. 2) constitute the synthesis module SYNTH, which delivers M time or frequency signals ss m (m varying from 1 to M), obtained by synthesis from of the parameters p g m . These signals ss m can then feed a conventional block of spatial decoding, as will be seen below with reference to FIG.
Dans un mode de réalisation particulier, la synthèse utilisée est une synthèse additive avec application d'une transformée de Fourier inverse (IFFT).In a particular embodiment, the synthesis used is an additive synthesis with application of an inverse Fourier transform (IFFT).
A cet effet, un ensemble de N sources est caractérisé par une pluralité de paramètres pi,k représentant l'amplitude dans le domaine fréquentiel de la kιeme composante fréquentielle pour la i'eme source Sj. Le signal temporel Si(n) qui correspondrait à cette source Si, s'il était synthétisé indépendamment des autres sources, serait donné par :For this purpose, a set of N sources is characterized by a plurality of parameters pi, k representing the amplitude in the frequency domain of the kth frequency component for the ith source Sj. The time signal Si (n) that would correspond to this source If, if it were synthesized independently of the other sources, would be given by:
si (») = ∑ cα (n) » avec cα (n) = Pa (») cos[2^α ^n j Fe + V1* (n)] k=l où Pi,k est l'amplitude de la composante de fréquence fi,k et dont la phase est donnée par φijk pour la source Si, à l'instant n. II est possible de réaliser la synthèse additive dans le domaine fréquentiel à partir des seuls paramètres pi,k, fi,k et φi,k donnés, en utilisant par exemple la technique exposée dans le document FR-2 679 689. s i (») = Σ c α ( n )" with c α ( n ) = Pa (») cos [ 2 ^ α ^ nj F e + V 1 * ( n )] k = 1 where Pi, k is the amplitude of the component of frequency fi, k and whose phase is given by φ ijk for the source Si, at time n. It is possible to carry out the additive synthesis in the frequency domain from the only given parameters pi, k , fi, k and φi, k , using, for example, the technique described in the document FR-2 679 689.
Le paramètre Pi,k représente l'amplitude d'une composante fréquentielle k donnée pour une source Si donnée. On en déduit donc les paramètres pmi,k pour chaque source et chacun des M canaux grâce à la relation : pmi,k = gmι - Pι,k > rn variant de 1 à M.The parameter Pi, k represents the amplitude of a given frequency component k for a given source Si. We thus deduce the parameters p m i, k for each source and each of the M channels by means of the relation: p m i, k = g m ι - Pι, k > rn varying from 1 to M.
Les gains g™ sont prédéterminés pour une position désirée pour la source Si et en fonction de l'encodage de spatial isation choisi.G gains are predetermined for a desired position for the source Si and according to the selected spatialization encoding.
Dans le cas d'un encodage ambiophonique par exemple, ces gains correspondent aux harmoniques sphériques et peuvent s'écrire g™ = Ym(θi,δi), où :In the case of an ambiophonic encoding for example, these gains correspond to the spherical harmonics and can be written g ™ = Y m (θi, δi), where:
- Ym est un harmonique sphérique d'ordre m,Y m is a spherical harmonic of order m,
- θi et δi sont respectivement l'azimut et le site souhaités pour la source Si. Les paramètres pmi,k sont ensuite combinés fréquence par fréquence, de manière à obtenir un seul paramètre global :θi and δi are respectively the azimuth and the desired site for the Si source. The parameters p m i, k are then combined frequency by frequency, so as to obtain a single global parameter:
NNOT
P1V = ∑Pm'k > ou k< décrit toutes les fréquences fi,k présentes dans toutes lesP 1 V = ΣP m ' k > or k < describes all the frequencies fi, k present in all
1=1 sources Sj.1 = 1 sources Sj.
En pratique, la valeur de k1 est inférieure à k.i car des fréquences communes peuvent caractériser plusieurs sources à la fois. Dans une réalisation, il peut être prévu d'associer un même jeu global de fréquences à toutes les sources, quitte à ce que certains paramètres d'amplitude pour certaines fréquences de sources soient nuls.In practice, the value of k 1 is less than ki because common frequencies can characterize several sources at once. In one embodiment, it may be provided to associate the same global set of frequencies to all sources, even if certain amplitude parameters for certain source frequencies are zero.
Dans ce cas, les valeurs de k et k' sont égales et la relation précédente s'écrit simplement :In this case, the values of k and k 'are equal and the preceding relation is written simply:
NNOT
P S.k = 2^P >-k P S. k = 2 ^ P > - k
1=11 = 1
L'étape de synthèse consiste à utiliser ces paramètres pm g,k (m variant de 1 à M) pour synthétiser chacun des M spectres en fréquence ssm(ω) issus du module de synthèse SYNTH. Il peut être prévu à cet effet d'appliquer la technique décrite dans FR-2 679 689, en ajoutant itérativement des enveloppes spectrales correspondant à la transformée de Fourier d'une fenêtre temporelle (par exemple de Hanning), ces enveloppes spectrales étant précédemment échantillonnées, tabulées, centrées aux fréquences fk et pondérées alors par pm g,k, ce qui s'écrit :The synthesis step consists of using these parameters p m g , k (m varying from 1 to M) to synthesize each of the M frequency spectra ss m (ω) from the SYNTH synthesis module. It may be provided for this purpose to apply the technique described in FR-2,679,689, iteratively adding spectral envelopes corresponding to the Fourier transform of a time window (for example Hanning), these spectral envelopes being previously sampled , tabulated, centered at frequencies fk and then weighted by p m g , k , which is written as:
ssmθ) = envk(ω) , où envk(ω) est l'enveloppe spectrale centrée à la fréquence fk.ss θ m) = approx k (ω), where env k (ω) is the spectral envelope centered at the frequency f k .
Cette réalisation est illustrée sur la figure 4. On affecte K paramètres d'amplitude piik à chaque source Sj. L'indice i, de source, est compris entre 1 et N. L'indice k, de fréquence, est compris entre 1 et K. Pour chaque source Si, on duplique ces K paramètres, M fois, pour être multiplié chacun par un gain de spatialisation g ™. L'indice m, de canal d'encodage de spatialisation, est compris entre 1 et M.This embodiment is illustrated in FIG. 4. K amplitude parameters p iik are assigned to each source Sj. The index i, of source, is between 1 and N. The index k, of frequency, is between 1 and K. For each source Si, these K parameters are duplicated, M times, to be multiplied each by one gain Spatialization g ™. The index m, of spatialization encoding channel, is between 1 and M.
Dans chaque canal m, on regroupe, fréquence par fréquence, les K résultats des produits gΛpi.k, selon l'expression donnée ci-avant :In each channel m, we group, frequency by frequency, the K results of products gΛpi. k , according to the expression given above:
pV = ∑Pmα . avec pmi,k = gmi . pi,k ,pV = ΣP m α. with p m i, k = g m i. pi, k ,
1=1 où k varie de 1 à K dans chaque canal m, et m varie globalement de 1 à M. On comprendra ainsi que dans chaque canal m, il est prévu des sous-canaux pm g,k associés chacun à une composante fréquentielle k, l'indice g désignant, pour rappel, le terme "globaf.1 = 1 where k varies from 1 to K in each channel m, and m varies globally from 1 to M. It will thus be understood that in each channel m, sub-channels p m g , k each associated with a component are provided. frequency k, the index g designating, as a reminder, the term "globaf.
Le traitement se poursuit alors en multipliant le paramètre global de chaque sous-canal pm g,k associé à une fréquence fk par une enveloppe spectrale envk(ω) centrée en cette fréquence fk, et ce, pour tous les K sous-canaux (k compris entre 1 et K), et globalement, pour tous les M canaux (m étant compris entre 1 et M). Ensuite, les K sous-canaux sont sommés dans chaque canal m, conformément à la relation ci-après :The processing then continues by multiplying the global parameter of each subchannel p m g , k associated with a frequency fk by a spectral envelope envk (ω) centered at this frequency fk, and this, for all the K subchannels ( k between 1 and K), and globally, for all M channels (m being between 1 and M). Then the sub-channels K are summed in each channel m, according to the following relation:
ssm(<») = envk{ω) , pour m allant de 1 à M canaux au total. ss m (<") = env k {ω), for m ranging from 1 to M channels in total.
On obtient alors les signaux ss ,mm(/ω) encodés pour leur spatialisation et synthétisés au sens de l'invention. Ils sont exprimés dans le domaine fréquentiel.The signals ss, m m (/ ω) encoded for their spatialization and synthesized in the sense of the invention are then obtained. They are expressed in the frequency domain.
Pour ramener ces M signaux dans le domaine temporel (notés alors SSm(n)), on peut leur appliquer ensuite une transformée de Fourier inverse (IFFT) : SSm(n) = IFFT(ssm(ω)) Le traitement par trames successives peut être réalisé par une technique classique d'addition/recouvrement. Chacun des M signaux temporels SSm(n) peut ensuite être fourni à un bloc de décodage de spatialisation.To bring back these M signals in the time domain (then noted SS m (n)), we can then apply them an inverse Fourier transform (IFFT): SS m (n) = IFFT (ss m (ω)). successive frames can be achieved by a conventional technique of addition / overlap. Each of the M time signals SS m (n) can then be supplied to a spatialization decoding block.
A cet effet, il peut être prévu par exemple une paire de filtres adaptés Fgm(n), Fdm(n) à appliquer, par convolution, à chaque signal SSm(n), comme représenté sur la figure 3, pour une adaptation d'un encodage ambiophonique vers une restitution en binaural à deux voies gauche et droite. Ces filtres pour une telle transition ambiophonique/binaural peuvent être obtenus par application de la technique des haut-parleurs virtuels citée ci-avant.For this purpose, it is possible, for example, to provide a pair of matched filters Fg m (n), Fd m (n) to be convolutionally applied to each signal SS m (n), as shown in FIG. adaptation of an ambiophonic encoding to a two-way left and right binaural rendition. These filters for such an ambiophonic / binaural transition can be obtained by applying the virtual speaker technique mentioned above.
Le traitement réalisé par le bloc DECOD de décodage spatial de la figure 3 peut être du type :The processing performed by the spatial decoding DECOD block of FIG. 3 can be of the type:
SSm g (n) = (SSm*Fgm)(n)SS m g (n) = (SS m * Fg m ) (n)
SSm d (n) = (SSm*Fdm)(n) Après filtrage, tous les signaux destinés aux oreilles gauche et droite sont sommés respectivement, et on obtient ainsi une paire de signaux binauraux :SS m d (n) = (SS m * Fd m ) (n) After filtering, all the signals for the left and right ears are summed respectively, thus obtaining a pair of binaural signals:
MM
MM
Sd (n) = ∑SS™ (n) m=l qui viennent alimenter alors les écouteurs d'un casque à deux oreillettes.S d (n) = ΣSS ™ (n) m = 1 which then supply the headphones of a headset with two earpieces.
On décrit néanmoins une variante plus avantageuse ci-après. Les filtres d'adaptation du format ambiophonique vers le format binaural peuvent être appliqués directement dans le domaine fréquentiel, évitant ainsi une convolution dans le domaine temporel et un coût de calcul correspondant.However, a more advantageous variant is described below. The adaptation filters from the surround format to the binaural format can be applied directly in the frequency domain, thus avoiding convolution in the time domain and a corresponding calculation cost.
A cet effet, chacun des M spectres en fréquence ssm(ω) est directement multiplié par les transformées de Fourier respectives des filtres temporels, notées Fgm(ω) et Fdm(ω) (adaptées le cas échéant pour avoir un nombre de points cohérent), ce qui s'écrit : ssm g (ω) = ssm (ω)Fgm (ω) ssm d (ω) = ssm(ω)Fdm (ω) Les spectres sont ensuite sommés par oreille avant d'effectuer la transformée de Fourier inverse et l'opération d'addition/recouvrement, soit :For this purpose, each of the M frequency spectra ss m (ω) is directly multiplied by the respective Fourier transforms of the temporal filters, noted Fg m (ω) and Fd m (ω) (adapted if necessary to have a coherent number of points), which is written: ss m g (ω) = ss m (ω) Fg m (ω) ss m d (ω) = ss m (ω) Fd m (ω) The spectra are then summed by ear before performing the inverse Fourier transform and the addition / recovery operation, ie:
sg (ω) = ∑smAω) m=l M sd(ω) = ∑sm d (ω)s g (ω) = Σs m Aω) m = l M s d (ω) = Σs m d (ω)
Puis, pour exprimer les signaux alimentant le dispositif de restitution dans le domaine temporel, on applique la transformée de Fourier inverse : Sg (n) = IFFT(S g (ω))Then, to express the signals supplying the rendering device in the time domain, the inverse Fourier transform is applied: S g (n) = IFFT (S g (ω))
Sd (n) = IFFT(sd (ω))S d (n) = IFFT (s d (ω))
La présente invention vise aussi un produit programme d'ordinateur, qu'il soit stocké dans une mémoire d'une unité centrale ou d'un terminal, ou sur un support amovible propre à coopérer avec un lecteur de cette unité centraleThe present invention also relates to a computer program product, whether it is stored in a memory of a central unit or a terminal, or on a removable support adapted to cooperate with a reader of this central unit.
(CD-ROM, disquette ou autre), ou encore téléchargeable via un réseau de télécommunications. Ce programme comporte en particulier des instructions pour la mise en œuvre du procédé décrit ci-avant et dont un organigramme peut être illustré à titre d'exemple sur la figure 5, résumant les étapes d'un tel procédé.(CD-ROM, floppy disk or other), or downloadable via a telecommunications network. This program comprises in particular instructions for the implementation of the method described above and a flowchart can be illustrated by way of example in Figure 5, summarizing the steps of such a method.
L'étape a) vise l'affectation des paramètres représentatifs d'une amplitude à chaque source Sj. Dans l'exemple représenté, on affecte un paramètre pi,k par composante fréquentielle fk comme décrit ci-avant. L'étape b) vise la duplication de ces paramètres et leur multiplication par les gains g ™ des canaux d'encodage.Step a) is aimed at assigning the parameters representative of an amplitude to each source Sj. In the example shown, a parameter pi, k is assigned by frequency component f k as described above. Step b) aims at the duplication of these parameters and their multiplication by the gains g ™ of the encoding channels.
L'étape c) vise le regroupement des produits obtenus à l'étape b), avec en particulier le calcul de leur somme sur toutes les sources Si. L'étape d) vise la synthèse paramétrique avec multiplication par une enveloppe spectrale envk comme décrit ci-avant, suivi d'un regroupement des sous-canaux par application, dans chaque canal, d'une somme sur toutes les composantes fréquentielles (d'indice k allant de 1 à K). L'étape e) vise un décodage de spatialisation des signaux ssm issus des canaux respectifs, synthétisés, spatialisés et représentés dans le domaine fréquentiel, pour une restitution sur deux haut-parleurs par exemple au format binaural.Step c) relates to the grouping of the products obtained in step b), with in particular the calculation of their sum on all Si sources. Step d) targets the parametric synthesis with multiplication by a spectral envelope env k as described above, followed by a grouping of the subchannels by applying, in each channel, a sum over all the frequency components (d index k ranging from 1 to K). Step e) aims at decoding the spatialization of the signals ss m originating from the respective channels, synthesized, spatialised and represented in the frequency domain, for a reproduction on two loudspeakers, for example in binaural format.
La présente invention vise aussi un dispositif de génération de sons synthétiques et spatialisés, comprenant notamment un processeur, et, en particulier, une mémoire de travail propre à stocker des instructions du produit programme d'ordinateur défini ci-avant.The present invention also provides a device for generating synthetic and spatialized sounds, comprising in particular a processor, and in particular a working memory adapted to store instructions of the computer program product defined above.
Bien entendu, la présente invention ne se limite pas à la forme de réalisation décrite ci-avant à titre d'exemple ; elle s'étend à d'autres variantes.Of course, the present invention is not limited to the embodiment described above by way of example; it extends to other variants.
Ainsi, il a été décrit ci-avant à titre d'exemple un encodage de spatialisation en format ambiophonique réalisé par le module SPAT de la figure 2, suivi d'une adaptation du format ambiophonique vers le format binaural. En variante, il peut être prévu par exemple d'appliquer directement un encodage vers le format binaural.Thus, it has been described above as an example encoding space spatialization in surround format made by the SPAT module of Figure 2, followed by an adaptation of the surround format to the binaural format. As a variant, it may be provided, for example, to directly apply an encoding to the binaural format.
Par ailleurs, la multiplication par des enveloppes spectrales de la synthèse paramétrique est décrite ci-avant à titre d'exemple, d'autres modèles pouvant être prévus en variante. Moreover, the spectral envelope multiplication of the parametric synthesis is described above by way of example, other models that can be provided alternatively.

Claims

Revendications claims
1. Procédé pour synthétiser et spatialiser conjointement une pluralité de sources sonores dans des positions associées de l'espace, comportant : a) une étape d'affectation à chaque source d'au moins un paramètre (pi) représentatif d'une amplitude, b) une étape de spatialisation mettant en œuvre un encodage en une pluralité de canaux, dans laquelle on duplique chaque paramètre d'amplitude (pi) pour le multiplier à un gain de spatialisation (g ™), chaque gain de spatialisation étant déterminé, d'une part, pour un canal d'encodage (pg m) et, d'autre part, pour une source à spatialiser (Si), c) une étape de regroupement (R) des paramètres (pim) multipliés par les gains, dans des canaux respectifs (pg 1,..., pg M), en appliquant une somme desdits paramètres multipliés (pim) sur toutes les sources (Si) pour chaque canal (pg m), et d) une étape de synthèse paramétrique (SYNTH(I ), ..., SYNTH(M)) appliquée à chacun des canaux (pg m).A method for synthesizing and spatially co-ordinating a plurality of sound sources in associated positions of the space, comprising: a) a step of assigning to each source at least one parameter (pi) representative of an amplitude, b ) a spatialization step implementing an encoding in a plurality of channels, in which each amplitude parameter (pi) is duplicated to multiply it to a spatialization gain (g ™), each spatialization gain being determined, and on the one hand, for an encoding channel (p g m ) and, on the other hand, for a source to be spatialized (Si), c) a grouping step (R) of the parameters (pi m ) multiplied by the gains, in respective channels (p g 1 ,..., p g M ), by applying a sum of said multiplied parameters (pi m ) to all the sources (Si) for each channel (p g m ), and d) a step of parametric synthesis (SYNTH (I), ..., SYNTH (M)) applied to each of the channels (p g m ).
2. Procédé selon la revendication 1 , dans lequel : a) on affecte à chaque source (Si) une pluralité de paramètres (pijk) représentatifs, chacun, d'une d'amplitude d'une composante fréquentielleThe method according to claim 1, wherein: a) each source (Si) is assigned a plurality of representative parameters (p ijk ), each of an amplitude of a frequency component
(fk), b) on duplique chaque paramètre d'amplitude (pi,k) représentatif d'une composante fréquentielle (fk) pour le multiplier à un gain de spatialisation (gim), chaque gain de spatialisation étant déterminé, d'une part, pour un canal d'encodage (pg m) et, d'autre part, pour une source à spatialiser (S1), c) dans chaque canal, on regroupe, composante fréquentielle par composante fréquentielle, les produits des paramètres (pijk) par les gains (g ™), en sous- canaux (pg,km) associés chacun à une composante fréquentielle (fk). (f k ), b) we duplicate each amplitude parameter (pi, k ) representative of a frequency component (f k ) to multiply it to a spatialization gain (gi m ), each spatialization gain being determined, d on the one hand, for an encoding channel (p g m ) and, on the other hand, for a source to be spatialised (S 1 ), c) in each channel, the frequency component by frequency component is grouped together. parameters (p ijk ) by the gains (g ™), in subchannels (p g , k m ) each associated with a frequency component (fk).
3. Procédé selon la revendication 2, dans lequel la synthèse est menée, dans chaque canal, en : d1 ) multipliant la sortie de chaque sous-canal associé à une composante fréquentielle (fk) par une enveloppe spectrale (envk) centrée sur une fréquence correspondant à ladite composante fréquentielle (fk), d2) et en regroupant, par une somme sur les composantes fréquentielles (fk), les produits résultant de l'opération d1 ), pour obtenir, suite à l'opération d2), un signal (ssm) issu de chaque canal, encodé en spatial isation et synthétisé.3. Method according to claim 2, in which the synthesis is carried out, in each channel, by: d1) multiplying the output of each subchannel associated with a frequency component (fk) by a spectral envelope (env k ) centered on a frequency corresponding to said frequency component (f k ), d2) and grouping, by a sum on the frequency components (f k ), the products resulting from the operation d1), to obtain, following the operation d2), a signal (ss m ) from each channel, encoded in space isation and synthesized.
4. Procédé selon l'une des revendications précédentes, dans lequel la spatialisation est menée par encodage ambiophonique et les paramètres représentatifs d'une amplitude qui sont affectés aux sources correspondent à des amplitudes d'harmoniques sphériques (Ym).4. Method according to one of the preceding claims, wherein the spatialization is conducted by surround encoding and the representative parameters of an amplitude which are assigned to the sources correspond to spherical harmonics amplitudes (Y m ).
5. Procédé selon la revendication 4, prise en combinaison avec la revendication 3, dans lequel, pour passer d'un encodage ambiophonique à un décodage en vue d'une restitution en spatialisation binaurale, on applique un traitement dans le domaine fréquentiel directement aux résultats des produits issus des canaux respectifs après l'opération d2).The method of claim 4, taken in combination with claim 3, wherein, to switch from surround encoding to decoding for binaural spatialization rendering, frequency domain processing is directly applied to the results. products from the respective channels after the operation d2).
6. Produit programme d'ordinateur, stocké dans une mémoire d'une unité centrale ou d'un terminal, et/ou sur un support amovible propre à coopérer avec un lecteur de ladite unité centrale, et/ou téléchargeable via un réseau de télécommunications, caractérisé en ce qu'il comporte des instructions pour la mise en œuvre du procédé selon l'une des revendications 1 à 5.6. Computer program product, stored in a memory of a central unit or a terminal, and / or on a removable support adapted to cooperate with a reader of said central unit, and / or downloadable via a telecommunications network , characterized in that it comprises instructions for implementing the method according to one of claims 1 to 5.
7. Module de génération de sons synthétiques spatialisés, comprenant notamment un processeur, caractérisé en ce qu'il comporte en outre une mémoire de travail stockant des instructions du produit programme d'ordinateur selon la revendication 6. 7. module for generating spatialised synthetic sounds, comprising in particular a processor, characterized in that it further comprises a working memory storing instructions of the computer program product according to claim 6.
EP07731685A 2006-03-13 2007-03-01 Joint sound synthesis and spatialization Active EP1994526B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PL07731685T PL1994526T3 (en) 2006-03-13 2007-03-01 Joint sound synthesis and spatialization

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FR0602170 2006-03-13
PCT/FR2007/050868 WO2007104877A1 (en) 2006-03-13 2007-03-01 Joint sound synthesis and spatialization

Publications (2)

Publication Number Publication Date
EP1994526A1 true EP1994526A1 (en) 2008-11-26
EP1994526B1 EP1994526B1 (en) 2009-10-28

Family

ID=37400911

Family Applications (1)

Application Number Title Priority Date Filing Date
EP07731685A Active EP1994526B1 (en) 2006-03-13 2007-03-01 Joint sound synthesis and spatialization

Country Status (8)

Country Link
US (1) US8059824B2 (en)
EP (1) EP1994526B1 (en)
JP (1) JP5051782B2 (en)
AT (1) ATE447224T1 (en)
DE (1) DE602007002993D1 (en)
ES (1) ES2335246T3 (en)
PL (1) PL1994526T3 (en)
WO (1) WO2007104877A1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8786852B2 (en) 2009-12-02 2014-07-22 Lawrence Livermore National Security, Llc Nanoscale array structures suitable for surface enhanced raman scattering and methods related thereto
US9395304B2 (en) 2012-03-01 2016-07-19 Lawrence Livermore National Security, Llc Nanoscale structures on optical fiber for surface enhanced Raman scattering and methods related thereto
US9788135B2 (en) 2013-12-04 2017-10-10 The United States Of America As Represented By The Secretary Of The Air Force Efficient personalization of head-related transfer functions for improved virtual spatial audio
JP2019530312A (en) * 2016-10-04 2019-10-17 オムニオ、サウンド、リミテッドＯｍｎｉｏ Ｓｏｕｎｄ Ｌｉｍｉｔｅｄ Stereo development technology

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2679689B1 (en) * 1991-07-26 1994-02-25 Etat Francais METHOD FOR SYNTHESIZING SOUNDS.
JPH08502867A (en) 1992-10-29 1996-03-26 ウィスコンシン アラムニ リサーチ ファンデーション Method and device for producing directional sound
US5596644A (en) 1994-10-27 1997-01-21 Aureal Semiconductor Inc. Method and apparatus for efficient presentation of high-quality three-dimensional audio
FR2782228B1 (en) 1998-08-05 2001-05-25 Ct Scient Tech Batiment Cstb SOUND SIMULATION DEVICE AND METHOD FOR PRODUCING SUCH A DEVICE
FR2847376B1 (en) * 2002-11-19 2005-02-04 France Telecom METHOD FOR PROCESSING SOUND DATA AND SOUND ACQUISITION DEVICE USING THE SAME
FI118247B (en) * 2003-02-26 2007-08-31 Fraunhofer Ges Forschung Method for creating a natural or modified space impression in multi-channel listening
FR2851879A1 (en) * 2003-02-27 2004-09-03 France Telecom PROCESS FOR PROCESSING COMPRESSED SOUND DATA FOR SPATIALIZATION.
WO2005069272A1 (en) * 2003-12-15 2005-07-28 France Telecom Method for synthesizing acoustic spatialization
SE0400998D0 (en) * 2004-04-16 2004-04-16 Cooding Technologies Sweden Ab Method for representing multi-channel audio signals
US8204261B2 (en) * 2004-10-20 2012-06-19 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Diffuse sound shaping for BCC schemes and the like
EP2112652B1 (en) * 2006-07-07 2012-11-07 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Apparatus and method for combining multiple parametrically coded audio sources
JP5021809B2 (en) * 2007-06-08 2012-09-12 ドルビー ラボラトリーズ ライセンシング コーポレイション Hybrid derivation of surround sound audio channels by controllably combining ambience signal components and matrix decoded signal components

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2007104877A1 *

Also Published As

Publication number Publication date
US8059824B2 (en) 2011-11-15
ATE447224T1 (en) 2009-11-15
DE602007002993D1 (en) 2009-12-10
US20090097663A1 (en) 2009-04-16
EP1994526B1 (en) 2009-10-28
WO2007104877A1 (en) 2007-09-20
JP2009530883A (en) 2009-08-27
JP5051782B2 (en) 2012-10-17
PL1994526T3 (en) 2010-03-31
ES2335246T3 (en) 2010-03-23

Similar Documents

Publication Publication Date Title
EP1600042B1 (en) Method for the treatment of compressed sound data for spatialization
EP2000002B1 (en) Method and device for efficient binaural sound spatialization in the transformed domain
EP1992198B1 (en) Optimization of binaural sound spatialization based on multichannel encoding
EP1563485B1 (en) Method for processing audio data and sound acquisition device therefor
EP2042001B1 (en) Binaural spatialization of compression-encoded sound data
WO2007110520A1 (en) Method for binaural synthesis taking into account a theater effect
EP1999999A1 (en) Generation of spatial downmixes from parametric representations of multi channel signals
FR2762467A1 (en) MULTI-CHANNEL ACOUSTIC ECHO CANCELLATION METHOD AND MULTI-CHANNEL ACOUSTIC ECHO CANCELLATOR
EP1695335A1 (en) Method for synthesizing acoustic spatialization
EP1886535B1 (en) Method of producing a plurality of time signals
EP2005420A1 (en) Device and method for encoding by principal component analysis a multichannel audio signal
KR102660704B1 (en) Spectrally orthogonal audio component processing
EP1994526B1 (en) Joint sound synthesis and spatialization
EP3400599A1 (en) Improved ambisonic encoder for a sound source having a plurality of reflections
EP3025514B1 (en) Sound spatialization with room effect
WO2023232586A1 (en) Signal processing method
TW202309881A (en) Colorless generation of elevation perceptual cues using all-pass filter networks
CN117678014A (en) Colorless generation of elevation-aware cues using an all-pass filter network
WO2006097633A1 (en) Method and system for spatializing an audio signal based on its intrinsic qualities

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20080828

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC MT NL PL PT RO SE SI SK TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

17Q First examination report despatched

Effective date: 20090616

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC MT NL PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REF Corresponds to:

Ref document number: 602007002993

Country of ref document: DE

Date of ref document: 20091210

Kind code of ref document: P

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2335246

Country of ref document: ES

Kind code of ref document: T3

LTIE Lt: invalidation of european patent or patent extension

Effective date: 20091028

REG Reference to a national code

Ref country code: PL

Ref legal event code: T3

NLV1 Nl: lapsed or annulled due to failure to fulfill the requirements of art. 29p and 29m of the patents act
PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20100301

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20100228

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

REG Reference to a national code

Ref country code: IE

Ref legal event code: FD4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20100128

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

Ref country code: IE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

BERE Be: lapsed

Owner name: FRANCE TELECOM

Effective date: 20100331

26N No opposition filed

Effective date: 20100729

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20100331

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20100129

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20100331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20110331

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20110331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20100429

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20100301

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091028

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 9

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 10

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 11

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 12

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: ES

Payment date: 20230403

Year of fee payment: 17

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240220

Year of fee payment: 18

Ref country code: GB

Payment date: 20240220

Year of fee payment: 18

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: PL

Payment date: 20240226

Year of fee payment: 18

Ref country code: FR

Payment date: 20240220

Year of fee payment: 18