EP2245620B1 - Verfahren und mittel zur enkodierung von hintergrundrauschinformationen - Google Patents

Verfahren und mittel zur enkodierung von hintergrundrauschinformationen Download PDF

Info

Publication number
EP2245620B1
EP2245620B1 EP09711709.7A EP09711709A EP2245620B1 EP 2245620 B1 EP2245620 B1 EP 2245620B1 EP 09711709 A EP09711709 A EP 09711709A EP 2245620 B1 EP2245620 B1 EP 2245620B1
Authority
EP
European Patent Office
Prior art keywords
background noise
frames
period
speech
energy
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP09711709.7A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP2245620A1 (de
Inventor
Stefan Schandl
Panji Setiawan
Herve Taddei
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Unify GmbH and Co KG
Original Assignee
Unify GmbH and Co KG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Unify GmbH and Co KG filed Critical Unify GmbH and Co KG
Publication of EP2245620A1 publication Critical patent/EP2245620A1/de
Application granted granted Critical
Publication of EP2245620B1 publication Critical patent/EP2245620B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/012—Comfort noise or silence coding
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
    • G10L19/16—Vocoder architecture
    • G10L19/18—Vocoders using multiple modes

Definitions

  • the invention relates to methods and means for encoding background noise information in speech signal coding methods.
  • Such a limited frequency range is also provided in many speech signal coding methods for today's digital telecommunications.
  • a bandwidth limitation of the analog signal Prior to a coding process, a bandwidth limitation of the analog signal is performed for this purpose.
  • a codec is used which, due to the described bandwidth limitation in the frequency range between 300 Hz and 3400 Hz, is also referred to below as narrow-band speech codec (Narrow Band Speech Codec).
  • the term codec is understood to mean both the coding rule for the digital coding of audio signals and the decoding rule for the decoding of data with the aim of reconstructing the audio signal.
  • a narrowband speech codec is known, for example, from ITU-T Recommendation G.729.
  • a transmission of a narrowband speech signal with a data rate of 8 kbit / s is provided.
  • broadband speech codecs Wide Band Speech Codec
  • Such an extended frequency range is e.g. between a frequency of 50 Hz and 7000 Hz.
  • a wideband speech codec is known from ITU-T Recommendation G.729.EV.
  • coding methods for broadband speech codecs are made scalable.
  • scalability it is meant here that the transmitted coded data includes various demarcated blocks containing the narrowband portion, the wideband portion and / or the full bandwidth of the coded voice signal.
  • such a scalable design allows for backward compatibility on the receiver side and, on the other hand, offers a simple possibility of adapting the data rate and the size of transmitted data frames in the transmission channel in the case of limited data transmission capacities in the transmission channel.
  • a compression of the data to be transmitted For a reduction of the data transmission rate by a codec is usually provided a compression of the data to be transmitted. Compression is achieved, for example, by coding methods in which parameters for an excitation signal and filter parameters are determined for coding the speech data. The filter parameters and parameters specifying the excitation signal are then transmitted to the receiver. There, a synthetic speech signal is synthesized using the codec, which is as similar as possible to the original speech signal in terms of a subjective hearing impression. With the help of this method, also known as "analysis-by-synthesis", the determined and digitized samples are not transmitted, but determined parameters that allow a receiver-side synthesis of the speech signal.
  • a further measure for reducing the data transmission rate is provided by a method for discontinuous transmission (Discontinuous Transmission), which is also familiar in the art under the term DTX.
  • Discontinuous Transmission a method for discontinuous transmission
  • document US 2008/0027716 A1 discloses a DTX signaling.
  • the basic goal of DTX is to reduce the data transfer rate in the event of a speech break.
  • a pause detection (Voice Activity Detection, VAD) is used on the part of the transmitter, which recognizes when a certain signal level falls below a speech break.
  • VAD Voice Activity Detection
  • Comfort noise is noise that is synthesized to fill silence phases on the receiver's side.
  • the comfort noise serves as a subjective impression of a continuing connection, without claiming the data transmission rate intended for the transmission of speech signals. In other words, less effort is required to code the speech data for the transmitter-side coding of the noise. For a receiver-side still perceived as realistic synthesizing the comfort noise data are transmitted at a much lower data rate.
  • the data transmitted here will be Also referred to in the art as SID (Silence Insertion Description).
  • discontinuous transmission methods provide for transmission of SID frames with updated background noise characterization parameters only when significant changes in background noise energy during an inactive speech period (silence) are detected by the encoder. This applies to both narrowband (50Hz to 4kHz) and wideband speech codecs which support discontinuous transmission techniques.
  • an Energy Threshold specified in the decoder is used. This means that no SID frames are sent if the defined energy limit is not exceeded.
  • a suspension of transmission of SID frames is regarded as idle channel.
  • additional data exchange may be required to indicate that the connection is to be maintained.
  • a well-known additional data exchange currently takes place in such a way that administrative bodies in the network management of the transmission network are the sending node, i. request the sending encoder to retransmit the last transmitted SID frame if the elapsed idle period to the last SID frame sent is considered too long for the corresponding connection. For such retransmission, parameters of the retransmitted SID frame are not updated. The encoder does not perform any additional actions.
  • the object of the invention is to provide an improved implementation of the discontinuous transmission in scalable speech codecs.
  • a basic idea of the invention is to design the encoder of a speech codec in such a way that, after a previously determined idle period, it carries out a new determination or calculation of parameters via the background noise, in particular the averaged energy and the autocorrelation function. Said determination of the background noise parameters in other words corresponds to an encoding of the noise signal.
  • Administrative authorities in the network inform the encoder about the idle time set in the transmission network. The encoder thus determines the idle time, for example by requesting administrative bodies in the transmission network. Such a request is necessary only once if the determined idle time is stored by the encoder.
  • An advantage of the method according to the invention is that in order to decide whether to send updated background noise parameters in the form of an updated SID frame, no comparison of the energy of the background noise signal with an energy limit value is required.
  • the method thus saves computational resources compared with the known methods.
  • Another advantage is that the set period of time between two SID frames complies with the requirements of the respective transmission network.
  • An advantageous embodiment of the invention provides a SID structure (SID Bitstream Structure) in which the narrowband portion of the background noise information is separated from the broadband portion of the background noise information.
  • SID Bitstream Structure SID Bitstream Structure
  • Separate handling of narrowband and broadband background noise information in a SID frame enables separate encoding of the narrowband and wideband portions of the background noise and makes the processing transparent.
  • This embodiment also has the advantage that it can be determined on the receiver side whether comfort noise is based on the broadband component of the transmitted data SID framework or on the basis of the narrowband component. This is of particular advantage for the receiver-side acoustic reception in a situation where the transmission rate for speech information frames has been reduced such that only narrowband speech information is transmitted.
  • An advantageous embodiment of the invention provides that energy and autocorrelation function of the background noise are determined to determine the background noise parameters of the narrowband first portion of the background noise.
  • the narrowband portion requires averaging over a relatively long period of speech break, in practice over a period of e.g. 100 ms.
  • the calculation quantities used according to this embodiment include the energy (not the logarithmic energy) and the autocorrelation function.
  • an additional overhang period (hangover period) is introduced according to a further advantageous embodiment of the invention.
  • the newly introduced overhang period in the following: DTX overhang period serves another previously unknown purpose compared to the previously known VAD overhang period (Voice Activity Detection). While both types of overhanging period aim to identify multiple frames as active speech frames and thus avoid misclassification at the end of a speech signal, the DTX overhang period has the additional purpose of gathering information about the background noise.
  • An advantageous embodiment of the invention provides that the broadband second component is attenuated.
  • the attenuation of the broadband component plays a role in the attenuation of the entire energy component in the broadband component. This measure is necessary due to the fact that the generator for generating (synthesizing) the comfort noise in the decoder is unable to produce the same noise characteristics as the original background noise in the encoder.
  • An advantageous embodiment of the invention provides that is applied to the entire background noise signal, ie the combination of broadband and narrowband portion, a downstream emphasis reduction filter ("De-emphasis Post Filter”).
  • the "De-Emphasis Post Filter” leads to a de-emphasis of the energy and the higher frequency components. Since the averaging deforms the spectral envelope in a certain way, this attenuation can advantageously contribute to reducing the disturbing effect of a disturbed broadband noise on a human receiver.
  • the single FIGURE shows a temporal representation of a transition from a classified as a speech to a classified as background noise input signal to a decoder.
  • the inventive method provides for designing the encoder so that it recalculates the averaged energy and the autocorrelation function after a certain given time. Administrative authorities in the network inform the encoder about the required idle time.
  • SID SID Bitstream Structure
  • the calculation quantities used include the energy (not the logarithmized energy) and the autocorrelation function.
  • the autocorrelation function is used for a spectral envelope presentation.
  • An overall amplification factor can be compensated by a combination of all amplification and averaging methods.
  • the values for the autocorrelation function are normalized by summing or averaging (Equally Weighted). This affects all SID frames.
  • a relatively long averaging of the narrowband portion results in smoothening of the narrowband energy and the spectral envelope, so that a sudden change in energy does not significantly affect the synthesizing of comfort noise in the receiver.
  • the same averaging period is used for both energy and averaging of the spectral envelope after a first SID frame is generated after a voice burst is applied. This measure assures a more consistent estimation of the narrowband background noise during a transition from a speech period to a speech pause.
  • FIG. 1 shows a speech signal (Speech Burst) that at a certain time t a certain signal level, threshold, shown in the drawing as a dashed line, falls below.
  • the ordinate is to be understood as the level or energy value of the signal.
  • a pause detection (Voice Activity Detection, VAD) is used on the part of the transmitter, which detects when the threshold falls below a speech break.
  • VAD Voice Activity Detection
  • the VAD method provides for a known overhang period VAD-HO in which active voice frames are still transmitted and only after typically two frame lengths transitions into a mode which provides for generation of SID frames.
  • an additional overhang period DTX-HO is introduced.
  • the new overhang period DTX-HO follows the hitherto known overhang period VAD-HO, which is used as a black box.
  • VAD-HO hitherto known overhang period
  • the signal processed in the encoder is still classified as a speech signal, while at the same time a determination is already made starts from background noise parameters.
  • the data rate of the speech coding is already reduced, since no high-quality encoding is needed at the beginning of a speech break.
  • a portion of the overhead period is used for averaging the first SID frame.
  • the aforementioned embodiments preferably relate to the last frames FRAMES within an overhang period DTX-HO, VAD-HO.
  • the information of the first frames of the overhang period is preferably not used.
  • the newly introduced overhanging period DTX-HO serves a further hitherto unnoticed purpose compared to the well-known overhang period VAD-HO, which was previously motivated by the needs of voice activity detection. While both types of overhang periods DTX-HO, VAD-HO aim to identify multiple frames as active speech frames and thus avoid misclassification at the end of a speech signal, the DTX-overhang period DTX-HO has the additional purpose of providing information about the speech Raise background noise.
  • the new overhanging period DTX-HO provides additional assurance that after expiration of the overhanging period DTX-HO there will definitely be background noise and no speech signals at the input of the decoder.
  • VAD-HO could not be ruled out that it was the background signal exclusively to the adjacent signal.
  • speech fragments speech bursts
  • the new overhang period DTX-HO is used exclusively for learning the background noise.
  • an advantageous setting is e.g. to choose such that a period of two frames - cf. dashed axis FRAMES - is provided for the known overhang period VAD-HO and a period of five frames for the new overhang period DTX-HO.
  • the attenuation of the broadband component plays a role in the attenuation of the entire energy component in the broadband component. This measure is necessary due to the fact that the generator for generating (synthesizing) the comfort noise in the decoder is unable to produce the same noise characteristics as the original background noise in the encoder.
  • the output wideband speech signal ie the combination of broadband and narrowband components
  • a downstream de-emphasis postfilter This filtering mainly attenuates higher frequency components.
  • the De-Emphasis Post Filter also leads to a de-emphasis of the energy and the higher frequency components. Because the averaging deforms the spectral envelope in some way, this attenuation can help to reduce the disturbing effect of a disturbed wideband noise on a human receiver.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Telephonic Communication Services (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
  • Telephone Function (AREA)
  • Transmission Systems Not Characterized By The Medium Used For Transmission (AREA)
EP09711709.7A 2008-02-19 2009-02-02 Verfahren und mittel zur enkodierung von hintergrundrauschinformationen Active EP2245620B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102008009718A DE102008009718A1 (de) 2008-02-19 2008-02-19 Verfahren und Mittel zur Enkodierung von Hintergrundrauschinformationen
PCT/EP2009/051123 WO2009103610A1 (de) 2008-02-19 2009-02-02 Verfahren und mittel zur enkodierung von hintergrundrauschinformationen

Publications (2)

Publication Number Publication Date
EP2245620A1 EP2245620A1 (de) 2010-11-03
EP2245620B1 true EP2245620B1 (de) 2017-08-30

Family

ID=40568601

Family Applications (1)

Application Number Title Priority Date Filing Date
EP09711709.7A Active EP2245620B1 (de) 2008-02-19 2009-02-02 Verfahren und mittel zur enkodierung von hintergrundrauschinformationen

Country Status (8)

Country Link
US (1) US8949121B2 (ru)
EP (1) EP2245620B1 (ru)
JP (1) JP5415460B2 (ru)
KR (1) KR101216496B1 (ru)
CN (1) CN101952887B (ru)
DE (1) DE102008009718A1 (ru)
RU (1) RU2440674C1 (ru)
WO (1) WO2009103610A1 (ru)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012095924A1 (ja) * 2011-01-14 2012-07-19 パナソニック株式会社 符号化装置、通信処理装置および符号化方法
CN103187065B (zh) 2011-12-30 2015-12-16 华为技术有限公司 音频数据的处理方法、装置和***
US8868415B1 (en) * 2012-05-22 2014-10-21 Sprint Spectrum L.P. Discontinuous transmission control based on vocoder and voice activity
PL3550562T3 (pl) * 2013-02-22 2021-05-31 Telefonaktiebolaget Lm Ericsson (Publ) Sposoby i urządzenia dla zawieszenia DTX w kodowaniu audio
US9572103B2 (en) * 2014-09-24 2017-02-14 Nuance Communications, Inc. System and method for addressing discontinuous transmission in a network device
CN112437957A (zh) 2018-07-27 2021-03-02 杜比实验室特许公司 用于全面收听的强加间隙***

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5881373A (en) 1996-08-28 1999-03-09 Telefonaktiebolaget Lm Ericsson Muting a microphone in radiocommunication systems
US5893056A (en) 1997-04-17 1999-04-06 Northern Telecom Limited Methods and apparatus for generating noise signals from speech signals
AU754698B2 (en) * 1998-06-08 2002-11-21 Telefonaktiebolaget Lm Ericsson (Publ) System for elimination of audible effects of handover
RU2237296C2 (ru) 1998-11-23 2004-09-27 Телефонактиеболагет Лм Эрикссон (Пабл) Кодирование речи с функцией изменения комфортного шума для повышения точности воспроизведения
AR024520A1 (es) * 1998-11-24 2002-10-16 Ericsson Telefon Ab L M Metodo para realizar la transmision discontinua (dtx) en un sistema de comunicaciones, metodo para transmitir mensajes de protocolo a un segundo componente en un sistema de comunicaciones donde datos de habla son transmitidos desde un primer componente a un segungo componente, metodo de efectuar cam
WO2000075919A1 (en) * 1999-06-07 2000-12-14 Ericsson, Inc. Methods and apparatus for generating comfort noise using parametric noise model statistics
US6807525B1 (en) 2000-10-31 2004-10-19 Telogy Networks, Inc. SID frame detection with human auditory perception compensation
CN1617605A (zh) 2003-11-12 2005-05-18 皇家飞利浦电子股份有限公司 一种在语音信道传输非语音数据的方法及装置
BRPI0515453A (pt) * 2004-09-17 2008-07-22 Matsushita Electric Ind Co Ltd aparelho de codificação escalável, aparelho de decodificação escalável, método de codificação escalável método de decodificação escalável, aparelho de terminal de comunicação, e aparelho de estação de base
JP2008546341A (ja) 2005-06-18 2008-12-18 ノキア コーポレイション 非連続音声送信の際の擬似背景ノイズパラメータ適応送信のためのシステム及び方法
US20070136055A1 (en) * 2005-12-13 2007-06-14 Hetherington Phillip A System for data communication over voice band robust to noise
US8725499B2 (en) * 2006-07-31 2014-05-13 Qualcomm Incorporated Systems, methods, and apparatus for signal change detection
US8260609B2 (en) * 2006-07-31 2012-09-04 Qualcomm Incorporated Systems, methods, and apparatus for wideband encoding and decoding of inactive frames
US8032359B2 (en) * 2007-02-14 2011-10-04 Mindspeed Technologies, Inc. Embedded silence and background noise compression

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
JP5415460B2 (ja) 2014-02-12
EP2245620A1 (de) 2010-11-03
RU2440674C1 (ru) 2012-01-20
US20110004471A1 (en) 2011-01-06
US8949121B2 (en) 2015-02-03
DE102008009718A8 (de) 2009-12-17
CN101952887A (zh) 2011-01-19
KR101216496B1 (ko) 2012-12-31
DE102008009718A1 (de) 2009-08-20
CN101952887B (zh) 2013-05-29
WO2009103610A1 (de) 2009-08-27
KR20100123734A (ko) 2010-11-24
JP2011515705A (ja) 2011-05-19

Similar Documents

Publication Publication Date Title
EP2245621B1 (de) Verfahren und mittel zur enkodierung von hintergrundrauschinformationen
DE60117471T2 (de) Breitband-signalübertragungssystem
DE60129327T2 (de) Anordnung und verfahren in bezug auf die vermittlung von sprache
DE60214599T2 (de) Skalierbare audiokodierung
EP2047669B1 (de) Verfahren zum durchführen einer audiokonferenz, audiokonferenzeinrichtung und umschalteverfahren zwischen kodierern
EP2245620B1 (de) Verfahren und mittel zur enkodierung von hintergrundrauschinformationen
DE69925168T2 (de) Erkennung der aktivität komplexer signale für verbesserte sprach-/rauschklassifizierung von einem audiosignal
DE69631318T2 (de) Verfahren und Vorrichtung zur Erzeugung von Hintergrundrauschen in einem digitalen Übertragungssystem
DE69533734T2 (de) Durch Sprachaktivitätsdetektion gesteuerte Rauschunterdrückung
DE60209888T2 (de) Kodieren eines audiosignals
EP1953739B1 (de) Verfahren und Vorrichtung zur Geräuschsunterdrückung bei einem decodierten Signal
DE102007018484B4 (de) Vorrichtung und Verfahren zum Senden einer Folge von Datenpaketen und Decodierer und Vorrichtung zum Decodieren einer Folge von Datenpaketen
DE60122751T2 (de) Verfahren und vorrichtung für die objektive bewertung der sprachqualität ohne referenzsignal
DE60118631T2 (de) Verfahren zum ersetzen verfälschter audiodaten
WO1998048531A1 (de) Verfahren zum verschleiern von fehlern in einem audiodatenstrom
WO1999018673A1 (de) Verfahren und vorrichtung zur erzeugung eines bitratenskalierbaren audio-datenstroms
EP1677286A1 (de) Verfahren zur Anpassung von Comfort Noise Generation Parametern
DE4343366C2 (de) Verfahren und Schaltungsanordnung zur Vergrößerung der Bandbreite von schmalbandigen Sprachsignalen
EP2245622B1 (de) Verfahren und mittel zur dekodierung von hintergrundrauschinformationen
DE10252070B4 (de) Kommunikationsendgerät mit parametrierter Bandbreitenerweiterung und Verfahren zur Bandbreitenerweiterung dafür
DE69921643T2 (de) Av-signalübertragung mit variabler bitrate in einem paketnetz
DE60210597T2 (de) Vorrichtung zur adpcdm sprachkodierung mit spezifischer anpassung der schrittwerte
WO2002084929A1 (de) Verfahren und vorrichtung zur übertragung von digitalen signalen
EP1390947B1 (de) Verfahren zum signalempfang in einem digitalen kommunikationssystem
EP2745510B1 (de) Verfahren zur behandlung von störungen bei der übertragung einer zeitlichen folge digitaler bilder

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20100812

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA RS

RIN1 Information on inventor provided before grant (corrected)

Inventor name: SETIAWAN, PANJI

Inventor name: TADDEI, HERVE

Inventor name: SCHANDL, STEFAN

DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: UNIFY GMBH & CO. KG

17Q First examination report despatched

Effective date: 20140129

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: UNIFY GMBH & CO. KG

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 502009014306

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G10L0019000000

Ipc: G10L0019012000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20170331

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 19/18 20130101ALN20170320BHEP

Ipc: G10L 19/012 20130101AFI20170320BHEP

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 924306

Country of ref document: AT

Kind code of ref document: T

Effective date: 20170915

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: GERMAN

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 502009014306

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20170830

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171130

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 10

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171130

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171201

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20171230

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 502009014306

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20180531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20180228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180202

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180228

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180228

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180202

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180228

REG Reference to a national code

Ref country code: AT

Ref legal event code: MM01

Ref document number: 924306

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180202

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180202

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20090202

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170830

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170830

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 502009014306

Country of ref document: DE

Representative=s name: SCHAAFHAUSEN PATENTANWAELTE PARTNERSCHAFTSGESE, DE

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240216

Year of fee payment: 16

Ref country code: GB

Payment date: 20240222

Year of fee payment: 16

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240221

Year of fee payment: 16