EP1639580B1 - Kodierung von Mehrkanalsignalen - Google Patents

Kodierung von Mehrkanalsignalen Download PDF

Info

Publication number
EP1639580B1
EP1639580B1 EP04809080.7A EP04809080A EP1639580B1 EP 1639580 B1 EP1639580 B1 EP 1639580B1 EP 04809080 A EP04809080 A EP 04809080A EP 1639580 B1 EP1639580 B1 EP 1639580B1
Authority
EP
European Patent Office
Prior art keywords
channel
constraint
signal
filter
gain
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP04809080.7A
Other languages
English (en)
French (fr)
Other versions
EP1639580A1 (de
Inventor
Stefan Bruhn
Ingemar Johansson
Anisse Taleb
Patrik Sandgren
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Telefonaktiebolaget LM Ericsson AB
Original Assignee
Telefonaktiebolaget LM Ericsson AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from SE0303499A external-priority patent/SE0303499D0/xx
Application filed by Telefonaktiebolaget LM Ericsson AB filed Critical Telefonaktiebolaget LM Ericsson AB
Priority to EP12154099A priority Critical patent/EP2456236A1/de
Priority to PL04809080T priority patent/PL1639580T3/pl
Publication of EP1639580A1 publication Critical patent/EP1639580A1/de
Application granted granted Critical
Publication of EP1639580B1 publication Critical patent/EP1639580B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S5/00—Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation 
    • H04S5/02—Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation  of the pseudo four-channel type, e.g. in which rear channel signals are derived from two-channel stereo signals
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing

Definitions

  • the present invention relates in general to encoding of audio signals, and in particular to encoding of multi-channel audio signals.
  • stereophonic or multi-channel coding of audio signals is to encode the signals of the different channels separately as individual and independent signals.
  • Another basic way used in stereo FM radio transmission and which ensures compatibility with legacy mono radio receivers is to transmit a sum and a difference signal of the two involved channels.
  • M/S stereo coding is similar to the described procedure in stereo FM radio, in a sense that it encodes and transmits the sum and difference signals of the channel sub-bands and thereby exploits redundancy between the channel sub-bands.
  • the structure and operation of an encoder based on M/S stereo coding is described, e.g. in US patent 5,285,498 by J.D. Johnston .
  • Intensity stereo on the other hand is able to make use of stereo irrelevancy. It transmits the joint intensity of the channels (of the different sub-bands) along with some location information indicating how the intensity is distributed among the channels. Intensity stereo does only provide spectral magnitude information of the channels. Phase information is not conveyed. For this reason and since the temporal inter-channel information (more specifically the inter-channel time difference) is of major psycho-acoustical relevancy particularly at lower frequencies, intensity stereo can only be used at high frequencies above e.g. 2 kHz.
  • An intensity stereo coding method is described, e.g. in the European patent 0497413 by R. Veldhuis et al , see also "Intensity Stereo Coding", Harre J et al, 96th AES Convention, 26.02.1994.
  • a recently developed stereo coding method is described, e.g. in a conference paper with the title " Binaural cue coding applied to stereo and multi-channel audio compression", 112th AES convention, May 2002, Kunststoff, Germany by C. Faller et al.
  • This method is a parametric multi-channel audio coding method.
  • the basic principle is that at the encoding side, the input signals from N channels c 1 , c 2 , ... c N are combined to one mono signal m.
  • the mono signal is audio encoded using any conventional monophonic audio codec.
  • parameters are derived from the channel signals, which describe the multi-channel image.
  • the parameters are encoded and transmitted to the decoder, along with the audio bit stream.
  • the decoder first decodes the mono signal m' and then regenerates the channel signals c 1 ', c 2 ',..., c N ', based on the parametric description of the multi-channel image.
  • the principle of the Binaural Cue Coding (BCC) method is that it transmits the encoded mono signal and so-called BCC parameters.
  • the BCC parameters comprise coded inter-channel level differences and inter-channel time differences for sub-bands of the original multi-channel input signal.
  • the decoder regenerates the different channel signals by applying sub-band-wise level and phase adjustments of the mono signal based on the BCC parameters.
  • the advantage over e.g. M/S or intensity stereo is that stereo information comprising temporal inter-channel information is transmitted at much lower bit rates.
  • a problem with the state-of-the-art multi-channel coding techniques described above is that they require high bit rates in order to provide good quality. Intensity stereo, if applied at low bit rates as low as e.g. only a few kbps suffers from the fact that it does not provide any temporal inter-channel information. As this information is perceptually important for low frequencies below e.g. 2 kHz, it is unable to provide a stereo impression at such low frequencies.
  • BCC is able to reproduce the multi-channel image even at low frequencies at low bit rates of e.g. 3 kbps since it also transmits temporal inter-channel information.
  • this technique requires computational demanding time-frequency transforms on each of the channels, both at the encoder and the decoder.
  • BCC optimises the mapping in a pure mathematical manner. Characteristic artefacts immanent in the coding method will, however, not disappear.
  • side information consists of predictor filters and optionally a residual signal.
  • the predictor filters estimated by a least-mean-square algorithm, when applied to the mono signal allow the prediction of the multi-channel audio signals.
  • This technique synthesises the right and left channel signals by filtering sound source signals with so-called head-related filters.
  • this technique requires the different sound source signals to be separated and can thus not generally be applied for stereo or multi-channel coding.
  • the predictor filters are known to be optimal in the least-mean-square sense, they do not always fully restore the perceptual characteristics of the original multi-channel signals.
  • stereo image instability may occur, where the sound jumps randomly between left to right.
  • spectral nulls may cause instabilities and lead to a filter whose frequency response at these frequencies is aberrant. This may cause the filter to perform unnecessary amplification in certain regions and lead to very annoying audible artefacts, especially if the signals are low-pass or high-pass filtered.
  • An object of the present invention is to provide a method and device for multi-channel encoding that improves the perceptual quality of the audio signal.
  • a further object of the present invention is to provide such a method and device, which requires low bit rate representation.
  • the signals of the different channels are combined into one main signal.
  • a set of adaptive filters preferably one for each channel, is derived.
  • a filter When a filter is applied to the main signal it reconstructs the signal of the respective channel under a perceptual constraint.
  • the perceptual constraint is a gain and/or shape constraint.
  • the gain constraint allows the preservation of the relative energy between the channels while the shape constraint allows stereo image stability, e.g. by avoiding unnecessary filtering of spectral nulls.
  • the transmitted parameters are the main signal, in encoded form, and the parameters of the adaptive filters, preferably also encoded.
  • the receiver reconstructs the signal of the different channels by applying the adaptive filters and possibly some additional post-processing.
  • An advantage with the present invention is that perceptual artefacts are reduced when decoding audio signals.
  • the required transmission bit rate is at the same time also kept at a very low level.
  • FIG. 1 illustrates a typical system 1, in which the present invention advantageously can be utilised.
  • a transmitter 10 comprises an antenna 12 including associated hardware and software to be able to transmit radio signals 5 to a receiver 20.
  • the transmitter 10 comprises among other parts a multi-channel encoder 14, which transforms signals of a number of input channels 16 into output signals suitable for radio transmission. Examples of suitable multi-channel encoders 14 are described in detail further below.
  • the signals of the input channels 16 can be provided from e.g. an audio signal storage 18, such as a data file of digital representation of audio recordings, magnetic tape or vinyl disc recordings of audio etc.
  • the signals of the input channels 16 can also be provided in "live", e.g. from a set of microphones 19.
  • the audio signals are digitised, if not already in digital form, before entering the multi-channel encoder 14.
  • an antenna 22 with associated hardware and software handles the actual reception of radio signals 5 representing polyphonic audio signals.
  • typical functionalities such as e.g. error correction, are performed.
  • a decoder 24 decodes the received radio signals 5 and transforms the audio data carried thereby into signals of a number of output channels 26.
  • the output signals can be provided to e.g. loudspeakers 29 for immediate presentation, or can be stored in an audio signal storage 28 of any kind.
  • the system 1 can for instance be a phone conference system, a system for supplying audio services or other audio applications.
  • the communication has to be of a duplex type, while e.g. distribution of music from a service provider to a subscriber can be essentially of a one-way type.
  • the transmission of signals from the transmitter 10 to the receiver 20 can also be performed by any other means, e.g. by different kinds of electromagnetic waves, cables or fibres as well as combinations thereof.
  • Fig. 2a illustrates one embodiment of a multi-channel encoder 14 according to the present invention.
  • a number of channel signals c 1 , c 2 , ..., c N are received at separate inputs 16:1-16:N.
  • the channel signals are connected to a linear combination unit 34.
  • all channel signals are summed together to form a mono signal x.
  • any predetermined linear combination of one or more of the channel signals may be used as an alternative, including pure channel signals.
  • a pure sum will simplify most mathematical operations.
  • the mono signal x is provided as an input signal 42 to a channel filter section 130.
  • the mono signal x is provided to, and encoded in, a mono signal encoder 38 to provide encoding parameters p x representing the mono signal x.
  • the mono signal encoder operates according to any suitable mono signal encoding technique. Many such techniques are available in known technology. The actual details of the encoding technique are not of importance for enabling the present invention and is therefore not further discussed.
  • the channel signals are also connected to the channel filter section 130.
  • each channel signal is connected to a respective filter adaptation unit 30:1-30:N.
  • the filter adaptation units perform a reconstruction of a respective channel signal when applied to the mono signal x.
  • Coefficients of the filter adaptation units 30:1-30:N are according to the present invention optimised under a perceptual constraint. However, the optimised coefficients of the filter adaptation units 30:1-30:N may also be obtained at least partly in a joint optimisation of two or more of the channel signals.
  • the output of the channel filter section 130 comprises N sets of filter parameters p 1 -p N .
  • These filter parameters p 1 -p N are typically encoded separately or jointly to be suitable for transmission.
  • the filter parameters p 1 -p N and the mono signal x are sufficient to enable reconstruction of all channels signals.
  • the encoded filter parameters p 1 -p N and the encoding parameters p x representing the mono signal x are in the present embodiment multiplexed in a multiplexor 40 into one output signal 52, ready for transmission.
  • Fig. 2b illustrates a multi-channel decoder 24.
  • the decoder 24 in Fig. 2b is suitable for decoding multi-channel signals encoded by the encoder of Fig. 2a .
  • An input signal 54 is received and provided to a demultiplexor 56, which divides the input signal 54 into encoding parameters p x representing the mono signal x and a number of sets of encoded filter parameters p 1 -p N .
  • the encoding parameters p x representing the mono signal x are provided to a mono signal decoder 64, in which the encoding parameters p x representing the mono signal x are used to generate a decoded mono signal x" according any suitable decoding technique associated with the encoding technique used in Fig. 2a . Many such techniques are available in known technology. The actual details of the encoding technique are not of importance and are therefore not further discussed.
  • the decoded mono signal x" is provided to a channel filter section 160.
  • the encoded filter parameters are also provided to the channel filter section 160, where they are decoded and used to define channel filters 60:1-60:N.
  • the so defined respective channel filters 60:1-60:N are applied to the decoded mono signal x" whereby respective channel signals c" 1 -c" N are reconstructed and provided at outputs 26:1-26:N.
  • a mono signal is used as a main signal for regenerating the channel signals at the encoding or decoding.
  • any predetermined linear combination of signals selected among the channel signals may be used as such a main signal.
  • the optimum choice of predetermined linear combination depends on the actual application and implementation.
  • a single channel signal can also constitute a possible such predetermined linear combination.
  • FIG. 3a Another embodiment of a multi-channel encoder 14 according to the present invention is illustrated in Fig. 3a . Similar parts are denoted by similar reference numbers and only the differences are discussed below.
  • the linear combination unit 34 provides as earlier a predetermined linear combination of the channel signals to the mono signal encoder 38.
  • the signal associated with the mono signal x is instead a decoded version x" of the encoding parameters p x representing the mono signal x.
  • Such an arrangement referred to as a closed loop approach, will allow for certain compensations of mono signal encoding inaccuracies, as described further below.
  • the linear combination unit 34 of the present embodiment also combines the channel signals in N-1 predetermined linear combinations c* 1 -c* N-1 , which serves as actual input signals to the channel filter section 130.
  • the N-1 predetermined linear combinations c* 1 -c* N-1 should be mutually linear independent.
  • the linear combinations c* 1 -c* N-1 do not necessarily comprise any contribution from all channel signals.
  • the term "linear combination" should in this context be used as also comprising the special cases where a factor of a component can be set to zero. In fact, in the most simple set-up, the linear combinations c* 1 -c* N-1 can be identical to the channel signals c 1 -c N-1 .
  • the modified channel signals are also in this embodiment connected to the channel filter section 130, in which N-1 sets of filter coefficients are deduced, now corresponding to the modified channel signals.
  • the coefficients of the filter adaptation units 30:1-30:N are according to the present invention optimised under a perceptual constraint.
  • the output of the channel filter section 130 comprises N-1 sets of filter parameters p* 1 -p* N-1 .
  • These filter parameters p* 1 -p* N-1 are typically encoded separately or jointly to be suitable for transmission.
  • the encoded filter parameters p* 1 -p* N-1 and the encoding parameters p x representing the mono signal x are in the present embodiment transmitted separately.
  • Fig. 3b illustrates another multi-channel decoder 24.
  • the decoder 24 in Fig. 3b is suitable for decoding multi-channel signals encoded by the encoder of Fig. 3a .
  • Encoding parameters p x representing the mono signal x and a set of encoded filter parameters p* 1 -p* N-1 are received.
  • the encoding parameters p x representing the mono signal x are used to generate a decoded mono signal x" in a mono signal decoder 64 in analogy with previous embodiment.
  • the filter parameters p* 1 -p* N-1 are likewise provided to the channel filter section 160 for obtaining N-1 decoded modified channel signals c* 1 -c* N-1 .
  • a linear combination unit 74 is then used to provide reconstructed channel signals c" 1 -c" N from the modified channel signals c* 1 -c* N-1 and the decoded mono signal x".
  • the perceptual characteristics may not completely be determined by a pure mathematical minimisation.
  • the predicted channels may have no frequency content above or below a certain frequency. This occurs if, for instance, the channel is high-pass filtered, or results from a band-splitting procedure. Spectral nulls may cause instabilities and lead to filter responses that produces unnecessary amplification and low frequency audible artefacts. According to the present invention, a shape constraint is therefore advantageously utilised during optimisation procedures.
  • Fig. 4 illustrates the basic ideas of the constrained minimisation procedure at the encoder side according to the present invention in an embodiment having two channels (the stereo case) and a linear filter 31.
  • a filter 31 responsive for reconstruction of channel c 1 having filter coefficients h c 1 is derived according to a constrained error minimisation procedure in an optimising unit 32.
  • the filter h c 1 takes as input the combined channel signal, i.e.
  • the mono signal x ( n ), which in this embodiment is a linear combination of the two channel signals c 1 and c 2: x n ⁇ c ⁇ 1 ⁇ c ⁇ 1 n + ⁇ c ⁇ 2 ⁇ c ⁇ 2 n , and derives from it the output signal ⁇ 1( n ).
  • the factors ⁇ c 1 and ⁇ c 2 determine how the channel signals are combined.
  • One possibility is to set ⁇ c 1 to a factor 2 ⁇ and ⁇ c 2 to 2(1- ⁇ ).
  • the mono signal will be a weighted sum of the channels.
  • the weighted combination of the individual channel signals to form the mono signal can in general even be the combination of filtered versions of the respective channel signals. Such an approach will be called pre-filtering. This can be useful if the approach is implemented in the excitation domain or in general a weighted signal domain.
  • the channels can be pre-filtered by a LPC (Linear Predictive Coding) residual filter of the mono signal.
  • LPC Linear Predictive Coding
  • the mono and left and right channel will be assumed to be in general some pre-filtered versions of the real mono, left and right channels.
  • the step of post-filtering with the mono LPC synthesis filter would be needed in order to get back to the signal domains.
  • the filter parameters p 1 comprise the filter coefficients h c 1 and maybe necessary additional data defining the filter.
  • the difference signal of two channel signals is reproduced by a filter.
  • the right and left signals are illustrated by the curves 301 and 302, respectively.
  • the representation is not ideal, giving a slightly larger difference than the target difference over the entire frame. This will lead to a reproduced right signal 303 at the decoder side that is slightly lower than the original right signal, and a reproduced left signal 304 that is slightly higher than the original left signal.
  • the perception of such an artefact is that the volume of the right channel is decreased and the volume of the left channel is increased. If such artefacts moreover vary in time, the sound will swing back and forth between the right and left channel. A gain constraint may improve such a situation.
  • One possible approach is to have a hard constraint, i.e. exact energy match between the original channel and the estimated channel, or to impose a loose gain constraint such as the output channel has a prescribed energy E c 1 , which is not necessarily equal to the original channel signal energy.
  • a channel signal may look like curve 305 of Fig. 6 . No intensity is present below frequency f 1 or above frequency f 2 . However, a pure mathematical optimisation gives rise to a curve 306, which presents some limited power also below and above the frequencies f 1 and f 2 , respectively. Such artefacts are perceived.
  • constraints In order to impose a certain spectral shape on the filter, a set of linear constraints have to be imposed on the filter. These constraints should in general be of a number less than the number of coefficients of the filter.
  • h ⁇ c sc h ⁇ c uc + R ⁇ ⁇ xx - 1 ⁇ W ⁇ ⁇ c ⁇ W ⁇ ⁇ c T ⁇ R ⁇ ⁇ xx - 1 ⁇ W ⁇ ⁇ c - 1 ⁇ w ⁇ c - W ⁇ ⁇ c T ⁇ h ⁇ c uc .
  • This constraint is especially useful when it is known a priori that the channel has no frequency content in a certain frequency range.
  • the gain and shape constraints can also be combined.
  • This equation is useful for bit rate reduction when encoding the channel filters, since it shows that the channel filters are related by quantities that are available at the decoder side.
  • Fig. 7 an illustration shows that one clout of two channels c1, c2 is reproduced by applying the mono signal x to an unconstrained filter 131.
  • the result of the unconstrained filter is modified depending on shape constraints in a shape constraint section 132.
  • the shape constrained filter of channel c2 can be calculated and provided to separate gain constraint sections 133 for each channel.
  • FIG. 8 A more detailed block scheme of another embodiment using a side signal for applying the shape constraint is illustrated in Fig. 8 .
  • Two channel signals c 1 and c 2 are combined in addition means 55, 57 of a linear combination unit 34 to a mono signal x and a side signal s.
  • a channel filter section 130 comprises an unconstrained parametric filter 131, which applied to the mono signal x reproduces an estimate of the side signal ⁇ .
  • the filter coefficients are adapted to give the minimum difference between s and ⁇ .
  • the filter obtained in this manner h ⁇ s uc is provided to a shape constraint section 132, basically according to the discussions further above.
  • a shape-constrained filter h ⁇ s sc h c 1 for the side signal is created. From the relation (1) between channel filters in a stereo application, a shape-constrained filter for each channel signal is calculated, based on the shape-constrained filter h c 1 for the side signal. These filters, or rather the coefficients thereof, are provided to a respective gain constraint section 133:1, 133:2. A gain factor for each channel signal is calculated, and the two filters are provided to a parameter encoding section 66, where the parameters of the two filters are jointly encoded.
  • the constrained channel filters h c 1 and h c 2 After calculation of the constrained channel filters h c 1 and h c 2 , they are quantized and encoded in a representation, which is suitable for transmission to the receiver. Typically, the coefficients of the filters are quantized using scalar or vector quantizers and the quantizer indexes are transmitted. The quantizers may also implement prediction, which is very beneficial for bit rate reduction especially in this scenario.
  • Making use of the complementarities of the filters may further reduce the bit rate since only one of the filters h c 1 or h c 2 or a linear combination of them is quantized and transmitted while the gains g c 1 and g c 2 are jointly vector quantized and transmitted separately. Such a transmission can be carried out at bit rates as low as, e.g. 1 kbps.
  • the receiver first decodes the transmitted mono signal and channel filters. Then, it regenerates the different channel signals by filtering the mono signal through the respective channel filter. Preferably, in the stereo case, the completeness property is used, and the coefficients are recombined to produce the filters h c 1 and h c 2 .
  • Certain post-processing steps that further improve the quality of the reconstructed multi-channel signal may follow the re-generation of the different channels signals.
  • the gain constraint on the filters assumes previously computed channel energies, i.e. E c 1 , E c 2 . It is important to control the gains of the filters, e.g. g c 1 , g c 2 and to avoid unnecessary amplification by limiting the gains. Depending on the properties of the different channel signals, it may occur that the channels are anti-correlated on the whole frequency range or in certain frequency bands. This leads to a certain cancellation when the mono channel is formed. In this case, since the individual channel information has been lost, at least partially and in some frequency bands, it is often beneficial to limit the channels gains when these are greater than a certain amount, e.g. 0 dB.
  • this factor is less than 0 dB, then we have signal cancellation.
  • g F quantifies how severe this cancellation is.
  • the filters are derived based on the original mono signal. This is e.g. the case in Fig. 2a , where the signal 42 is the original mono signal x.
  • the decoder will use a quantized mono signal as input for the channel filtering.
  • the filter calculations are based on the coded and thus already quantized mono signal. This is e.g. the case in Fig. 3a , where the signal 44 is a decoded mono signal x".
  • This approach has the advantage that the channel filter design does not only aim to match the respective channel signals in a best possible way. It also aims to mitigate coding errors, which are the result of the mono signal encoding.
  • FIG. 9 illustrates the principles of sub-band processing.
  • a number of channels c 1 - c N are each divided in K sub-bands SB1, SB2, SBK.
  • the channel signals in each sub-band is provided to a respective multi-channel encoder unit 80:1-80:K, where the channel signals are encoded.
  • One or several of the multi-channel encoder units 80:1-80:K can be multi-channel encoder units according to the present invention.
  • a bit-stream combiner 82 combines the encoded signals into a common encoded signal 53, that is transmitted.
  • the multi-channel encoding for the different sub-bands can be carried out individually optimised with respect to e.g. assigned bit rate, processing frame sizes and sampling rate.
  • sub-band processing does not carry out multi-channel encoding for very low frequencies, e.g. below 200 Hz. That means that for this very low frequency band, a mere mono signal is transmitted. This principle makes use of the fact that the human stereo perception is less sensitive for very low frequencies. It is known from prior art and called sub-woofing.
  • the band splitting is done using a time-frequency transform such as, e.g. a short term Fourier transform (STFT), which allows decomposing the signal into single frequency components.
  • STFT short term Fourier transform
  • the filtering reduces to a mere multiplication of the individual spectral coefficients of the mono signal with a complex factor.
  • the parametric multi-channel coding method according to the invention will typically involve fixed frame-wise processing of signal samples.
  • parameters describing the multi-channel image are derived and transmitted with a rate corresponding to a coding frame length of, e.g. 20 ms.
  • the parameters may, however, be obtained from signal frames which are much larger than the coding frame length.
  • a suitable choice is to set the length of such analysis frames to values larger than the coding frame length. This implies that the parameter calculation is performed with overlapping analysis frames.
  • Analysis frames 83 at the encoder are slightly longer than encoding frames 84, as shown in the top of the figure.
  • a consequence of such overlapping analysis frames is that the parameters evolve smoothly, which is essential in order to provide a stable multi-channel audio signal impression.
  • the same is performed at the decoder side, shown in the middle of the figure. It is thus essential in the decoder to take account of this and to window and overlap-add synthesis frames 85, with an overlap 86, as shown at the bottom of the figure. This allows a smooth transition between filters associated with each frame.
  • smooth filter parameter evolution can be enforced. It is, e.g. possible to apply low-pass or median filtering to the filter parameters.
  • noise shaping of the coding noise.
  • the purpose of this operation is to move coding noise to frequencies where the signal has high spectral density and thus render the noise less audible.
  • Noise shaping is usually done adaptively, i.e. in response to the audio signal. This implies that, in general, the noise shaping performed on the mono signal will be different from what is required for the various channel signals.
  • the subsequent channel filtering according to the invention may lead to an audible coding noise increase in the reconstructed multi-channel signal when comparing to the audible coding noise in the mono signal.
  • signal-adaptive post-filtering may be applied to the reconstructed channel signals in a post-processing step of the receiver.
  • Any state-of-the-art post-filtering techniques can be deployed here, which essentially emphasise spectral tops or deepen spectral valleys and thereby reduce the audible noise.
  • One example of such a technique is so-called high-resolution post-filtering which is described in the European Patent 0 965 123 B1 by E. Ekudden et. al.
  • Other simple methods are so-called pitch- and formant post-filters, which are known from speech coding.
  • Fig. 11 the main steps of an embodiment of an encoding method according to the present invention are illustrated as a flow diagram.
  • the procedure starts in step 200.
  • a main signal preferably a mono signal, deduced from the multi-channel signals is encoded.
  • filter coefficients are optimised to give an as good representation as possible of a channel signal when applied to the main signal. The optimising takes place under perceptual constraints.
  • the optimal coefficients are then encoded in step 224.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
  • Filters That Use Time-Delay Elements (AREA)

Claims (15)

  1. Verfahren zur Codierung von Mehrkanal-Audiosignalen (c1 - cN), die mindestens einen ersten und einen zweiten Kanal umfassen, umfassend die folgenden Schritte:
    Erzeugen von Codierparametern (px), die ein Hauptsignal (x) darstellen, das eine erste vorbestimmte lineare Kombination von Signalen der Mehrkanalsignale (c1 - cN) ist;
    Ableiten von optimalen Parametern (p1 - pN) eines ersten adaptiven Filters (31; 131, 132, 133:1-2); und
    Codieren der optimalen Parameter (p1 - pN),
    gekennzeichnet durch die folgenden weiteren Schritte:
    Ableiten von optimalen Parametern (p1 - pN) mindestens eines zweiten adaptiven Filters (31; 131, 132, 133:1-2);
    wobei das erste adaptive Filter (31; 131, 132, 133:1-2) abgeleitet wird, um eine Mindestdifferenz zwischen dem Signal des ersten Kanals (c1 - cN) und einem Filterausgangssignal zu ergeben, wenn das erste adaptive Filter (31; 131, 132, 133:1-2) auf die erste vorbestimmte lineare Kombination (x) angewendet wird;
    die Mindestdifferenz gemäß einem ersten Kriterium definiert ist;
    das zweite adaptive Filter (31; 131, 132, 133:1-2) abgeleitet wird, um eine Mindestdifferenz zwischen dem Signal des zweiten Kanals (c1 - cN) und einem Filterausgangssignal zu ergeben, wenn das zweite adaptive Filter (31; 131, 132, 133:1-2) auf die erste vorbestimmte lineare Kombination (x) angewendet wird;
    die Mindestdifferenz gemäß einem zweiten Kriterium definiert ist;
    wobei die Ableitschritte des ersten und des zweiten adaptiven Filters (31; 131, 132, 133:1-2) unter mindestens einer perzeptorischen Beschränkung ausgeführt werden, die aus der Gruppe bestehend aus Verstärkungsbeschränkung und Formbeschränkung ausgewählt ist;
    die Verstärkungsbeschränkung eine von einer harten Verstärkungsbeschränkung, die eine genaue Energieübereinstimmung zwischen einem Originalkanal und einem geschätzten Kanal fordert, und einer lockeren Verstärkungsbeschränkung auferlegt, die fordert, dass ein Ausgangskanal eine vorgegebene Energie aufweist;
    die Formbeschränkung dem ersten bzw. zweiten Filter einen Satz von linearen Beschränkungen über eine Spektralform auferlegt.
  2. Verfahren nach Anspruch 1, dadurch gekennzeichnet, dass mindestens eines vom dem ersten Kriterium und dem zweiten Kriterium ein Kriterium der kleinsten mittleren Quadrate ist.
  3. Verfahren nach Anspruch 1 oder 2, dadurch gekennzeichnet, dass die perzeptorische Beschränkung mindestens eine Verstärkungsbeschränkung ist, die danach strebt, eine Gesamtenergie des Filterausgangssignals gleich einer Gesamtenergie des Signals des ersten Kanals zu ergeben.
  4. Verfahren nach Anspruch 1 oder 3, dadurch gekennzeichnet, dass die perzeptorische Beschränkung eine absolute Beschränkung ist, die verlangt, dass die Gesamtenergie des adaptiven Filterausgangssignals gleich der Gesamtenergie des Signals des entsprechenden Kanals ist.
  5. Verfahren nach Anspruch 1 oder 3, dadurch gekennzeichnet, dass die perzeptorische Beschränkung eine weiche Beschränkung ist, die adaptive Filter begünstigt, welche die Gesamtenergie des adaptiven Filterausgangssignals nahe der Gesamtenergie des Signals des entsprechenden Kanals ergeben.
  6. Verfahren nach Anspruch 1, dadurch gekennzeichnet, dass die Verstärkungsbeschränkung als ein Verstärkungsfaktor (gc1 - gcN) mal einem ohne Verstärkungsbeschränkungen abgeleiteten, adaptiven Filter auferlegt wird.
  7. Verfahren nach Anspruch 6, dadurch gekennzeichnet, dass das verstärkungsbeschränkte Filter h ̲ c gc
    Figure imgb0031
     gegeben ist durch: h ̲ c gc = g c ⁢ h ̲ c uc ,
    Figure imgb0032
     g c = E c ∑ n = frame start frame end c ^ uc ⁢ n 2 ,
    Figure imgb0033
    wobei h ̲ c gc
    Figure imgb0034
     das ohne Verstärkungsbeschränkungen abgeleitete, adaptive Filter ist, Eca die vorgegebene Energie des adaptiven Filterausgangssignals ist; und ĉuc(n) eine adaptive Filterausgabe des Hauptsignals x(n) ohne Verstärkungsbeschränkungen ist.
  8. Verfahren nach einem der Ansprüche 1 bis 7, dadurch gekennzeichnet, dass die perzeptorische Beschränkung mindestens eine Formbeschränkung ist, die dem adaptiven Filter (31; 131, 132, 133:1-2) eine vordefinierte Spektralform auferlegt.
  9. Verfahren nach Anspruch 8, dadurch gekennzeichnet, dass die Formbeschränkung ungültigen Inhalt in einem vordefinierten Frequenzbereich auferlegt.
  10. Verfahren nach einem der Ansprüche 1 bis 9, dadurch gekennzeichnet, dass der Schritt des Codierens der optimalen Parameter (p1 - pN) ein gemeinsames Codieren der optimalen Parameter der ersten und zweiten Filter umfasst.
  11. Verfahren nach einem der Ansprüche 1 bis 10 und Anspruch 8, dadurch gekennzeichnet, dass der Schritt des Ableitens von Parametern seinerseits die folgenden Schritte umfasst:
    Erzeugen einer zweiten vorbestimmten linearen Kombination (s; c*1 - c*N-1) der Signale der Mehrkanalsignale (c1 - cN);
    Ableiten von Parametern eines dritten Filters, um eine Mindestdifferenz zwischen der zweiten vorbestimmten linearen Kombination und dem Filterausgangssignal zu ergeben, wenn das dritte Filter unter der Formbeschränkung auf die erste vorbestimmte lineare Kombination angewendet wird;
    Berechnen der optimalen Parameter der ersten und zweiten Filter als eine Funktion der optimalen Parameter des dritten Filters.
  12. Verfahren nach einem der Ansprüche 1 bis 11, dadurch gekennzeichnet, dass der Schritt des Ableitens basierend auf den Codierparametern (px) ausgeführt wird, die das Hauptsignal (x) darstellen.
  13. Verfahren nach einem der Ansprüche 1 bis 11, dadurch gekennzeichnet, dass der Schritt des Ableitens direkt auf der Basis der ersten vorbestimmten linearen Kombination (x) ausgeführt wird.
  14. Verfahren nach einem der Ansprüche 1 bis 13, dadurch gekennzeichnet, dass die Mehrkanalsignale mehr als zwei Kanäle umfassen, wobei das Hauptsignal auf einer ersten vorbestimmten linearen Kombination (x) aller der mehr als zwei Kanäle basiert, und das Signal jedes Kanals durch ein separates adaptives Filter dargestellt wird, das unter der perzeptorischen Beschränkung optimiert ist.
  15. Codiervorrichtung (14), umfassend:
    Eingang (16:1 - 16:N) für Mehrkanal-Audiosignale (c1-cN), die mindestens einen ersten und einen zweiten Kanal umfassen;
    Mittel (38) zum Erzeugen von Codierparametern (px), die ein Hauptsignal (x) darstellen, das eine erste vorbestimmte lineare Kombination von Signalen der Mehrkanalsignale (c1 - cN) ist, wobei die Mittel (38) zum Erzeugen mit dem Eingang (16:1 - 16:N) verbunden sind;
    Mittel (31; 131, 132, 133:1-2) zum Ableiten von optimalen Parametern eines ersten adaptiven Filters;
    Mittel (66) zum Codieren der optimalen Parameter; und
    Ausgabemittel (52);
    gekennzeichnet durch:
    Mittel (31; 131, 132, 133:1-2) zum Ableiten von optimalen Parametern eines zweiten adaptiven Filters;
    wobei das erste adaptive Filter eine Mindestdifferenz zwischen dem Signal des ersten Kanals (c1 - cN) und dem Filterausgangssignal ergibt, wenn das erste adaptive Filter auf die erste vorbestimmte lineare Kombination (x) angewendet wird;
    die Mindestdifferenz gemäß einem ersten Kriterium definiert ist;
    das zweite adaptive Filter eine Mindestdifferenz zwischen dem Signal des zweiten Kanals (c1 - cN) und dem Filterausgangssignal ergibt, wenn das zweite adaptive Filter auf die erste vorbestimmte lineare Kombination (x) angewendet wird;
    die Mindestdifferenz gemäß einem zweiten Kriterium definiert ist;
    wobei die Mittel (31; 131, 132, 133:1-2) zum Ableiten von optimalen Parametern des ersten und des zweiten adaptiven Filters zum Ableiten der optimalen Parameter unter mindestens einer perzeptorischen Beschränkung, die aus der Gruppe bestehend aus Verstärkungsbeschränkung und Formbeschränkung ausgewählt ist, ausgelegt sind;
    die Verstärkungsbeschränkung eine von einer harten Verstärkungsbeschränkung, die eine genaue Energieübereinstimmung zwischen einem Originalkanal und einem geschätzten Kanal fordert, und einer lockeren Verstärkungsbeschränkung auferlegt, die fordert, dass ein Ausgangskanal eine vorgegebene Energie aufweist;
    die Formbeschränkung dem ersten bzw. zweiten Filter einen Satz von linearen Beschränkungen über eine Spektralform auferlegt.
EP04809080.7A 2003-12-19 2004-12-15 Kodierung von Mehrkanalsignalen Active EP1639580B1 (de)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP12154099A EP2456236A1 (de) 2003-12-19 2004-12-15 Eingeschränkte Filtercodierung polyphoner Signale
PL04809080T PL1639580T3 (pl) 2003-12-19 2004-12-15 Kodowanie sygnałów wielokanałowych

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
SE0303499A SE0303499D0 (sv) 2003-12-19 2003-12-19 Multi-channel coding using gain-shape constrained filters
SE0400415A SE527713C2 (sv) 2003-12-19 2004-02-20 Kodning av polyfoniska signaler med villkorsbegränsade filter
PCT/SE2004/001907 WO2005059901A1 (en) 2003-12-19 2004-12-15 Constrained filter encoding of polyphonic signals

Related Child Applications (1)

Application Number Title Priority Date Filing Date
EP12154099.1 Division-Into 2012-02-06

Publications (2)

Publication Number Publication Date
EP1639580A1 EP1639580A1 (de) 2006-03-29
EP1639580B1 true EP1639580B1 (de) 2013-10-23

Family

ID=31996352

Family Applications (2)

Application Number Title Priority Date Filing Date
EP12154099A Ceased EP2456236A1 (de) 2003-12-19 2004-12-15 Eingeschränkte Filtercodierung polyphoner Signale
EP04809080.7A Active EP1639580B1 (de) 2003-12-19 2004-12-15 Kodierung von Mehrkanalsignalen

Family Applications Before (1)

Application Number Title Priority Date Filing Date
EP12154099A Ceased EP2456236A1 (de) 2003-12-19 2004-12-15 Eingeschränkte Filtercodierung polyphoner Signale

Country Status (8)

Country Link
EP (2) EP2456236A1 (de)
JP (1) JP4323520B2 (de)
DK (1) DK1639580T3 (de)
ES (1) ES2439693T3 (de)
PL (1) PL1639580T3 (de)
PT (1) PT1639580E (de)
SE (1) SE527713C2 (de)
WO (1) WO2005059901A1 (de)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPWO2008016098A1 (ja) * 2006-08-04 2009-12-24 パナソニック株式会社 ステレオ音声符号化装置、ステレオ音声復号装置およびこれらの方法
US9330671B2 (en) 2008-10-10 2016-05-03 Telefonaktiebolaget L M Ericsson (Publ) Energy conservative multi-channel audio coding

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5434948A (en) 1989-06-15 1995-07-18 British Telecommunications Public Limited Company Polyphonic coding
NL9100173A (nl) 1991-02-01 1992-09-01 Philips Nv Subbandkodeerinrichting, en een zender voorzien van de kodeerinrichting.
US5285498A (en) * 1992-03-02 1994-02-08 At&T Bell Laboratories Method and apparatus for coding audio signals based on perceptual model
SE9700772D0 (sv) 1997-03-03 1997-03-03 Ericsson Telefon Ab L M A high resolution post processing method for a speech decoder
WO2003009206A1 (en) * 2001-07-19 2003-01-30 Sungwoo Kim The system and operational method of mobile telecommunication device for electronic cash
WO2003009208A1 (en) * 2001-07-20 2003-01-30 Medical Research Group Method and apparatus for communicating between an ambulatory medical device and a control device via telemetry using randomized data
AU2003216682A1 (en) 2002-04-22 2003-11-03 Koninklijke Philips Electronics N.V. Signal synthesizing

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
HERRE J ET AL: "INTENSITY STEREO CODING", PREPRINTS OF PAPERS PRESENTED AT THE AES CONVENTION, XX, XX, vol. 96, no. 3799, 26 February 1994 (1994-02-26), pages 1 - 10, XP009025131 *

Also Published As

Publication number Publication date
DK1639580T3 (da) 2014-01-13
PT1639580E (pt) 2013-11-19
JP4323520B2 (ja) 2009-09-02
EP1639580A1 (de) 2006-03-29
SE0400415D0 (sv) 2004-02-20
EP2456236A1 (de) 2012-05-23
PL1639580T3 (pl) 2014-04-30
JP2007527543A (ja) 2007-09-27
SE0400415L (sv) 2005-06-20
WO2005059901A1 (en) 2005-06-30
SE527713C2 (sv) 2006-05-23
ES2439693T3 (es) 2014-01-24

Similar Documents

Publication Publication Date Title
JP6740496B2 (ja) ステレオオーディオ信号を出力する装置及び方法
US9812136B2 (en) Audio processing system
CA2527971C (en) Fidelity-optimised variable frame length encoding
AU2005328264B2 (en) Near-transparent or transparent multi-channel encoder/decoder scheme
US8249883B2 (en) Channel extension coding for multi-channel source
US7945055B2 (en) Filter smoothing in multi-channel audio encoding and/or decoding
US7809579B2 (en) Fidelity-optimized variable frame length encoding
US20080319739A1 (en) Low complexity decoder for complex transform coding of multi-channel sound
US20090083045A1 (en) Device and Method for Graduated Encoding of a Multichannel Audio Signal Based on a Principal Component Analysis
US7725324B2 (en) Constrained filter encoding of polyphonic signals
WO2006091150A1 (en) Improved filter smoothing in multi-channel audio encoding and/or decoding
US20190156841A1 (en) Adaptive channel-reduction processing for encoding a multi-channel audio signal
EP1639580B1 (de) Kodierung von Mehrkanalsignalen
AU2007237227B2 (en) Fidelity-optimised pre-echo suppressing encoding
MX2008009186A (en) Complex-transform channel coding with extended-band frequency coding

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20051108

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU MC NL PL PT RO SE SI SK TR

DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20100222

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602004043653

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G10L0019000000

Ipc: H04S0005020000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 19/008 20130101ALN20130614BHEP

Ipc: H04S 5/02 20060101AFI20130614BHEP

INTG Intention to grant announced

Effective date: 20130701

RIN1 Information on inventor provided before grant (corrected)

Inventor name: TALEB, ANISSE

Inventor name: JOHANSSON, INGEMAR

Inventor name: SANDGREN, PATRIK

Inventor name: BRUHN, STEFAN

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU MC NL PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 638126

Country of ref document: AT

Kind code of ref document: T

Effective date: 20131115

REG Reference to a national code

Ref country code: PT

Ref legal event code: SC4A

Free format text: AVAILABILITY OF NATIONAL TRANSLATION

Effective date: 20131113

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602004043653

Country of ref document: DE

Effective date: 20131219

REG Reference to a national code

Ref country code: DK

Ref legal event code: T3

Effective date: 20140107

REG Reference to a national code

Ref country code: SE

Ref legal event code: TRGR

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2439693

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20140124

REG Reference to a national code

Ref country code: NL

Ref legal event code: T3

REG Reference to a national code

Ref country code: NL

Ref legal event code: T3

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 638126

Country of ref document: AT

Kind code of ref document: T

Effective date: 20131023

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140223

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

REG Reference to a national code

Ref country code: PL

Ref legal event code: T3

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602004043653

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

Ref country code: LU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131215

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

26N No opposition filed

Effective date: 20140724

REG Reference to a national code

Ref country code: HU

Ref legal event code: AG4A

Ref document number: E020831

Country of ref document: HU

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20131231

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20131215

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20131231

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602004043653

Country of ref document: DE

Effective date: 20140724

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20131023

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20131023

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 12

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140124

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 13

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20151215

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20151215

PGRI Patent reinstated in contracting state [announced from national office to epo]

Ref country code: IT

Effective date: 20170710

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 14

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: SE

Payment date: 20221227

Year of fee payment: 19

Ref country code: PT

Payment date: 20221128

Year of fee payment: 19

Ref country code: NL

Payment date: 20221226

Year of fee payment: 19

Ref country code: GB

Payment date: 20221227

Year of fee payment: 19

Ref country code: FR

Payment date: 20221227

Year of fee payment: 19

Ref country code: DK

Payment date: 20221227

Year of fee payment: 19

Ref country code: CZ

Payment date: 20221123

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: PL

Payment date: 20221125

Year of fee payment: 19

Ref country code: HU

Payment date: 20221129

Year of fee payment: 19

Ref country code: BE

Payment date: 20221227

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: ES

Payment date: 20230102

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20221221

Year of fee payment: 19

Ref country code: DE

Payment date: 20221228

Year of fee payment: 19