EP3869507B1 - Incorporation de metadonées spatiales dans des signaux audio - Google Patents
Incorporation de metadonées spatiales dans des signaux audio Download PDFInfo
- Publication number
- EP3869507B1 EP3869507B1 EP21156162.6A EP21156162A EP3869507B1 EP 3869507 B1 EP3869507 B1 EP 3869507B1 EP 21156162 A EP21156162 A EP 21156162A EP 3869507 B1 EP3869507 B1 EP 3869507B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- audio signal
- spatial metadata
- bandwidth
- metadata
- spatial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005236 sound signal Effects 0.000 title claims description 115
- 230000003044 adaptive effect Effects 0.000 claims description 40
- 238000000034 method Methods 0.000 claims description 38
- 238000001228 spectrum Methods 0.000 claims description 23
- 239000000872 buffer Substances 0.000 claims description 8
- 238000004458 analytical method Methods 0.000 claims description 5
- 230000015572 biosynthetic process Effects 0.000 claims description 3
- 238000003786 synthesis reaction Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 description 24
- 230000006835 compression Effects 0.000 description 18
- 238000007906 compression Methods 0.000 description 18
- 238000004891 communication Methods 0.000 description 17
- 230000009467 reduction Effects 0.000 description 10
- 238000013459 approach Methods 0.000 description 9
- 238000013461 design Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000004590 computer program Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 238000000605 extraction Methods 0.000 description 6
- 239000004065 semiconductor Substances 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000003595 spectral effect Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 230000008867 communication pathway Effects 0.000 description 1
- 239000004020 conductor Substances 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 239000000243 solution Substances 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 230000003245 working effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/018—Audio watermarking, i.e. embedding inaudible data in the audio signal
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/002—Dynamic bit allocation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
Definitions
- the present invention relates to processing of spatial audio signals.
- Modern data processing devices such as mobile phones have multiple processors.
- a general purpose processor can be used to run an operating system, OS, and a special purpose processor can be used to process audio signals.
- Hardware abstraction layer is a software interface that describes a hardware resource. System software is programmed to use these interfaces. Hardware vendors provide drivers for their products by implementing the HAL interfaces. It is important that the HAL interfaces do not change as that allows updating both system software and hardware independently of each other.
- Spatial audio can be encoded according to an audio encoding format.
- Support for spatial audio features should be added into a HAL in order to process the encoded spatial audio by a processor. If audio capturing is performed by one processor and audio encoding is performed by another processor, HALs of both processors should be added support for the spatial audio features. Since HALs allow updating system software and hardware independently of each other, changes to the HALs should be limited. If applications do not support the spatial audio features, they cannot process the spatial audio and they can break. Geiger et al. "Audio Data Hiding with High Data Rates Based on Intmdct", Acoustics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings .
- the invention provides a method according to claim 1 for embedding spatial metadata to an audio signal, an apparatus according to claim 5.
- Preferable aspects are defined in the dependent claims.
- a bandwidth for embedding spatial metadata to an audio signal is determined on the basis of at least one of an adaptive gain control value for the audio signal and an evaluation of a perceptual spectrum of the audio signal, where the spatial metadata is embedded. Then the spatial metadata is embedded to the audio on the basis of the determined bandwidth. In this way audio processing of the spatial audio signal is enabled even if spatial audio features are not supported.
- the spatial metadata is embedded to the audio signal in a transparent way so that the signal doesn't change perceptually. Since the spatial metadata is embedded to the audio signal, the audio signal may be transferred over legacy HAL interfaces and spatial audio use cases are enabled without breaking the legacy applications.
- the audio signal comprising the embedded metadata may be transferred over a legacy HAL from a digital signal processor (DSP) to a main central processing unit (CPU), where the audio signal may be encoded into a spatial audio format.
- the audio signal comprising the embedded metadata may be transferred over a legacy HAL from a DSP to a legacy application or an application supporting spatial audio processing.
- Spatial audio may be processed by processors and applications executed on system software provide they support spatial audio features that may be in accordance to a spatial audio technology.
- An example of spatial audio technology is Nokia's OZO audio.
- Spatial audio may be encoded according to an audio coding format, for example advanced audio coding (AAC), where spatial metadata is included into Data Stream Elements (DSE).
- AAC advanced audio coding
- DSE Data Stream Elements
- FIG. 1 shows a schematic block diagram of an exemplary apparatus or electronic device 50 depicted in Fig. 2 , which may incorporate a transmitter according to an embodiment of the invention.
- the electronic device 50 may for example be a communications device, wireless device, mobile terminal or user equipment of a wireless communication system. However, it would be appreciated that embodiments of the invention may be implemented within any electronic device or apparatus which may require transmission of radio frequency signals.
- the apparatus 50 may comprise a housing 30 for incorporating and protecting the device.
- the apparatus 50 further may comprise a display 32 in the form of a liquid crystal display.
- the display may be any suitable display technology suitable to display an image or video.
- the apparatus 50 may further comprise a keypad 34.
- any suitable data or user interface mechanism may be employed.
- the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.
- the apparatus may comprise one or more microphones 36 or any suitable audio input which may be a digital or analogue signal input.
- the apparatus 50 may further comprise an audio output device which in embodiments of the invention may be any one of: an earpiece 38, speaker, or an analogue audio or digital audio output connection.
- the apparatus 50 may also comprise a battery 40 (or in other embodiments of the invention the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator).
- the term battery discussed in connection with the embodiments may also be one of these mobile energy devices.
- the apparatus 50 may comprise a combination of different kinds of energy devices, for example a rechargeable battery and a solar cell.
- the apparatus may further comprise an infrared port 41 for short range line of sight communication to other devices.
- the apparatus 50 may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB/firewire wired connection.
- the apparatus 50 may comprise a controller 56 or processor for controlling the apparatus 50.
- the controller 56 may be connected to memory 58 which in embodiments of the invention may store both data and/or may also store instructions for implementation on the controller 56.
- the controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and decoding of audio and/or video data or assisting in coding and decoding carried out by the controller 56.
- the controller 56 may be a general purpose processor for running an operating system (OS).
- the controller may be for example an ARM CPU (Advanced RISC Machine Central Processing Unit).
- Examples of the operating systems comprise at least Android and iOS.
- a special purpose processor for example a digital signal processor (DSP), may be connected to the general purpose processor and dedicated for audio processing.
- DSP digital signal processor
- the apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a universal integrated circuit card (UICC) reader and UICC for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.
- a card reader 48 and a smart card 46 for example a universal integrated circuit card (UICC) reader and UICC for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.
- UICC universal integrated circuit card
- the apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network.
- the apparatus 50 may further comprise an antenna 59 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and for receiving radio frequency signals from other apparatus(es).
- the apparatus 50 comprises a camera 42 capable of recording or detecting imaging.
- the system 10 comprises multiple communication devices which can communicate through one or more networks.
- the system 10 may comprise any combination of wired and/or wireless networks including, but not limited to a wireless cellular telephone network (such as a GSM (2G, 3G, 4G, LTE, 5G), UMTS, CDMA network etc.), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a Bluetooth personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and the Internet.
- a wireless cellular telephone network such as a GSM (2G, 3G, 4G, LTE, 5G), UMTS, CDMA network etc.
- WLAN wireless local area network
- the system shown in Fig. 3 shows a mobile telephone network 11 and a representation of the internet 28.
- Connectivity to the internet 28 may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.
- the example communication devices shown in the system 10 may include, but are not limited to, an electronic device or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile telephone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22, a tablet computer.
- PDA personal digital assistant
- IMD integrated messaging device
- the apparatus 50 may be stationary or mobile when carried by an individual who is moving.
- the apparatus 50 may also be located in a mode of transport including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle or any similar suitable mode of transport.
- Some or further apparatus may send and receive calls and messages and communicate with service providers through a wireless connection 25 to a base station 24.
- the base station 24 may be connected to a network server 26 that allows communication between the mobile telephone network 11 and the internet 28.
- the system may include additional communication devices and communication devices of various types.
- the communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, Long Term Evolution wireless communication technique (LTE), 5G and any similar wireless communication technology.
- CDMA code division multiple access
- GSM global systems for mobile communications
- UMTS universal mobile telecommunications system
- TDMA time divisional multiple access
- FDMA frequency division multiple access
- TCP-IP transmission control protocol-internet protocol
- SMS short messaging service
- MMS multimedia messaging service
- email instant messaging service
- Bluetooth IEEE 802.11, Long Term Evolution wireless communication technique (LTE), 5G and any similar wireless communication technology.
- HSDPA high-speed downlink packet access
- HSUPA high-speed uplink packet access
- a communications device involved in implementing various embodiments of the present invention may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.
- various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.
- Fig. 4 illustrates an example of a method in accordance with at least some embodiments.
- the method provides a spatial audio signal that may be processed even if spatial audio features are not supported.
- the method may be performed by the apparatus described with Fig. 1 ., e.g. by the controller 56.
- Phase 402 comprises determining a bandwidth for embedding spatial metadata to an audio signal, on the basis of an adaptive gain control value for the audio signal, and preferably, additionally on the basis of an evaluation of a perceptual spectrum of the audio signal, where the spatial metadata is embedded.
- Phase 404 comprises embedding the spatial metadata to the audio signal, on the basis of the determined bandwidth.
- the spatial metadata is embedded to the audio signal, support for spatial features from HAL layers and/or applications is not necessarily needed. Accordingly, the spatial metadata embedded to the audio signal is transparent to HAL layers and applications that do not support spatial audio features. Therefore, while the audio signal that is embedded with metadata can be processed using the spatial metadata, processing of the audio signal without the spatial metadata is also possible.
- phase 402 comprises that the audio signal is a two-channel pulse code modulation, PCM, signal.
- the PCM audio signal embedded with metadata can be used to render the audio signal to various output formats and also controlling parameters such as listener orientation.
- Examples of the output formats comprise at least a multichannel speaker signal for headphone listening.
- phase 404 comprises interleaving the determined bandwidth with the least significant bits of samples of the audio signal.
- the spatial metadata may be spread among the lowest bits of the samples in a way that reduces or even minimizes the perceptual effect of embedding the spatial metadata to the audio signal.
- the bits for the spatial metadata can be allocated to every Nth of the samples to have an even distribution in time of modified bits.
- Fig. 5 illustrates an example of a method in accordance with at least some embodiments.
- the method provides determining a bandwidth for embedding spatial metadata to an audio signal based on an adaptive gain control value.
- the method may be performed by the apparatus described with Fig. 1 . e.g. by the controller 56.
- Phase 502 comprises determining an adaptive gain control value for the audio signal.
- Phase 504 comprises determining if the adaptive gain control value indicates a gain increase or a gain decrease.
- Phase 506 comprises determining, if the adaptive gain control value fails to indicate a gain increase in phase 504, the bandwidth to be a minimum bandwidth. In this way perceptual effects of embedding the spatial metadata to the audio signal may be kept acceptable.
- the adaptive gain control value fails to indicate a gain increase, when the adaptive gain control value is negative, whereby adaptive gain control value indicates a gain decrease.
- Phase 508 comprises determining, based on the adaptive gain control value indicating a gain increase in phase 504 and an adaptive gain control of the audio signal using the adaptive gain control value, the bandwidth per a sample of the audio signal to be a number of bits the sample has been shifted towards the most significant bit. In this way, the bandwidth for transparently carrying the spatial metadata embedded to the audio signal may be determined.
- phase 506 comprises that the bandwidth is 0.5 bits per sample.
- phase 508 comprises that the bandwidth is an amount of spatial metadata to be embedded or an upper limit that is pre-determined for samples of the audio signal.
- phase 502 comprises that the adaptive gain control value is determined on the basis of an adaptive gain control of the audio signal.
- the adaptive gain control of the audio signal may output the adaptive gain control value.
- the adaptive gain control may cause a shift of one or more bits of a sample of the audio signal towards the least significant bits or towards the most significant bits.
- the adaptive gain control value indicates a gain increase
- the adaptive gain control has shifted the bits towards the most significant bits.
- the shifted bits make room for embedding the spatial metadata to the least significant bits. Every 6dB of gain increase may correspond to a shift of one bit towards the most significant bits in a sample.
- the adaptive gain control value indicates a gain decrease, the adaptive gain control has shifted the bits towards the least significant bits. In this case information is lost and the bandwidth should be kept small or even at minimum.
- Fig. 6 illustrates an example of a method in accordance with at least some embodiments.
- the method provides embedding spatial metadata to an audio signal.
- the method may be performed by the apparatus described with Fig. 1 . e.g. by the controller 56.
- Phase 602 comprises determining a bandwidth for embedding spatial metadata to an audio signal in accordance with phase 402 of Fig. 402.
- Phase 604 comprises determining whether the determined bandwidth is sufficient for the spatial metadata.
- Phase 606 comprises embedding, if the bandwidth is determined to be sufficient for the spatial metadata, the spatial metadata to the audio signal.
- Phase 608 comprises at least one of phases 610 and 612.
- Phase 610 comprises compressing, if the determined bandwidth on the basis of the adaptive gain control value is insufficient, the spatial metadata. In this way the amount of spatial metadata may be reduced. Reducing the amount of spatial metadata provides that the spatial metadata fits to the determined bandwidth. This means storing less bits of spatial metadata for each value of metadata. Storing zero bits for a value means discarding that value altogether. Storing more than zero and less than the original amount means quantizing the value to a coarser scale.
- the phase 610 comprises applying a lossy compression method to the spatial metadata.
- Reduction of the amount of spatial metadata may be based on rules.
- One approach to reduce the amount of metadata is to store the audio signal into audio buffers for processing.
- the buffers may be time domain buffers or time-frequency tiles.
- Analysis data may be stored for each buffered audio.
- the analysis data may comprise amounts or ratios of direct and ambient sound in a buffer.
- the analysis data may further comprise information indicating a direction of arrival for the most prominent sound source related to that the buffer.
- information indicating a use case of the audio signal may be used for the lossy compression.
- the information of the use case may be received from an application in application metadata.
- An example of the application is an application for capturing spatial audio.
- An example of a use case is a 2D camera capture by a mobile phone, whereby spatial metadata of in-direct sounds may be reduced more than spatial metadata of direct sounds.
- spatial metadata of sounds originating from outside a visible area of the camera may be reduced more than spatial metadata of sounds originating from within the visible area.
- phase 610 comprises applying a lossless compression method to the spatial metadata.
- the lossless compression method may be based on entropy encoding.
- Phase 612 comprises determining, if the determined bandwidth on the basis of the adaptive gain control value is insufficient, the bandwidth on the basis of a perceptual spectrum of the audio signal comprising the embedded spatial metadata.
- the amount of metadata may be reduced to fit the bandwidth while controlling the effect of the reduction of the spatial metadata to the perception of the audio signal.
- the spatial metadata may be embedded using different bandwidths to the audio signal. The bandwidth that provides a perceptual spectrum close to the perceptual spectrum of the original audio signal, i.e. the audio signal without embedded spatial metadata, or at least that perceptual changes to the original audio signal are kept acceptable, may be determined to be the bandwidth for embedding spatial metadata.
- the bandwidth indicated by determined perceptual spectrum may be used for embedding the spatial metadata to the audio signal. Embedding the spatial metadata to the bandwidth determined in phase 612 may be facilitated if the spatial metadata is compressed in accordance with phase 610.
- Fig. 7 illustrates an example of a method in accordance with at least some embodiments (not encompassed by the claimed invention).
- the method provides determining a bandwidth for embedding spatial metadata on the basis of a perceptual spectrum of an audio signal comprising the embedded spatial metadata.
- the method may be used to implement phase 612 in Fig. 6 , for example.
- the method may be performed by the apparatus described with Fig. 1 . e.g. by the controller 56.
- Phase 702 comprises determining a reference perceptual spectrum based on a psychoacoustical model and the audio signal without embedded spatial metadata.
- Phase 704 comprises determining candidate perceptual spectra based on the psychoacoustical model and the audio signal embedded with spatial metadata using candidate bandwidths for the spatial metadata.
- Phase 706 comprises determining the bandwidth for embedding spatial metadata from the one or more candidate bandwidths on the basis of evaluating a perceived quality of the audio signals comprising embedded metadata using the candidate bandwidths.
- the evaluating may comprise comparing powers of the candidate perceptual spectra to a power of the perceptual spectra of the audio signal without embedded metadata.
- the evaluating may comprise calculating powers of the audio signals comprising embedded metadata at the candidate bandwidths at perceptually motivated frequency bands such as the Bark scale. The same is done for the audio signal without embedded metadata. Then, spectral comparison is performed to determine whether at some frequency bands the spatial metadata noise power exceeds the signal power. If this happens, it may be determined that spatial metadata noise power is too high and it may be determined that the bandwidth is the highest candidate bandwidth which did not cause such an exceed of spatial metadata noise signal power above the actual signal power.
- Fig. 8 illustrates an example of a method in accordance with at least some embodiments (according to preferable aspects claimed).
- the method provides processing of an audio signal comprising embedded spatial metadata.
- the method may be performed by the apparatus described with Fig. 1 . e.g. by the controller 56.
- Phase 802 comprises receiving the audio signal embedded with spatial metadata.
- Phase 804 comprises extracting the spatial metadata from the received audio.
- Phase 806 comprises at least one of encoding the received the audio signal according to a lossy audio coding format where the extracted spatial metadata is included in metadata elements of the audio encoding format, and performing spatial synthesis on the basis of the extracted spatial metadata and the received audio signal or the received audio signal, where the spatial metadata has been removed.
- phase 802 comprises receiving the audio signal after phase 404 of Fig. 4 .
- the audio signal may be received at a HAL of a processor.
- phase 804 comprises that the audio signal is left unmodified. In another example phase 804 comprises that the extraction process also removes the metadata from the signal producing the original audio signal to which the metadata was embedded.
- phase 806 comprises that the lossy audio encoding format is Advanced Audio Coding, AAC, and the spatial metadata is included into DSE elements.
- AAC bitstream may be formed based on the encoded audio signal and multiplexed to an MP4 file together with a video track.
- Figs. 9 and 10 illustrate example block diagrams of apparatuses for embedding spatial metadata in accordance with at least some embodiments of the present invention.
- the functional blocks may perform one or more functionalities in accordance with examples of the method in Figs. 4 to 8 .
- Fig. 9 illustrates functional blocks on a DSP 900 for encoding a spatial audio signal from captured raw microphone signals.
- Spatial Encoding 902 takes multiple raw microphone signals as input and produces an output of two channels of PCM audio and spatial metadata which together form a presentation of the captured spatial sound.
- the output PCM audio and spatial metadata can be used to render the audio to various output formats (e.g., multichannel speaker signal for home theatre listening, loudspeaker listening or a two-channel binaural signal for headphone listening) and also controlling parameters such as listener orientation.
- the output PCM audio can be two selected input signals or a an audio signal, e.g. processed downmix such as binaural audio, generated by processing the input signals
- the AGC may adjust the dynamics of the produced output audio so that the perceived level of the audio is comfortable to the user.
- the AGC may add gain to the audio when the input signal amplitude is low and reduce gain when the input signal amplitude is very high.
- the AGC may operate on several separate frequency bands and may also perform as a non-linear limiter in addition to controlling the gains linearly.
- the processing of the audio happens in frames of one or more samples and for each frame the AGC may have an associated overall gain control value.
- the AGC increase the signal amplitude. This means that the information content in the input signal to the AGC didn't use the full range of the sample depth and there is room to add information to the signal without removing any information from the input signal. At least some of that space will be in the lowest bits, i.e. the least significant bits, of the output samples because adding gain, i.e. multiplying a sample, causes a shift of the information towards the high bits, the most significant bits, of the sample. Every 6dB of gain corresponds to a shift of one bit towards the most significant bits in a sample.
- the AGC When the gain is negative, the AGC has caused that the information content in the input signal has shifted towards the least significant bits of the signal and also some information has been lost. Empty bits have been added to the most significant bits of each sample, but that headroom cannot be used to convey information in a perceptually transparent way.
- the approach is to always convey spatial metadata in the least significant bits of the signal. That means that only a minimal amount of bits can be allocated for spatial metadata when the AGC gain value is negative or that the perceptual impact of the added spatial metadata has to be evaluated using a different approach, e.g., using a psychoacoustic model.
- the audio and spatial metadata output from the Spatial Encoding and the gain value from the AGC are used in the Lossy Spatial Metadata Compression 906.
- This Lossy Spatial Metadata Compression may estimate the available bandwidth in the PCM signal for embedding spatial metadata, reduce the amount of spatial metadata to fit the available budget for each frame and embed the metadata to the audio signal.
- the output will be two channels of PCM audio with embedded spatial metadata.
- Application metadata e.g. from an application for capturing spatial sound, may be received by the Lossy Spatial Metadata Compression.
- Fig. 10 illustrates functional blocks of the Lossy Spatial Metadata Compression 906.
- the Lossy Spatial Metadata Compression comprises Available Metadata Bandwidth Estimator 1002 and Psychoacoustic Model 1004 that are used to provide input for the Metadata Reduction 1006 and 1008 Metadata Compression. The output from those is fed to Metadata Signal Embedding 1010.
- the output of the Lossy Spatial Metadata Compression is two channels of PCM audio that is perceptually identical or as close as possible to the input signal and has spatial metadata embedded to it.
- the Psychoacoustic Model 1004 and Metadata Compression 1008 can be considered optional. It is possible to create a satisfactory implementation of the invention without them. The availability of Psychoacoustic Model and Metadata Compression help in providing better quality.
- the Available Metadata Bandwidth Estimator 1002 gets an input of at least a gain value for the current frame from the AGC. If the Available Metadata Bandwidth Estimator is operating without a Psychoacoustic Model, then it can blindly use the AGC gain value as the controlling parameter for the processing. This is the simplest realization of the estimation and the logic is as follows.
- the required bandwidth for spatial metadata There may be a minimum value for the required bandwidth for spatial metadata. It could be, e.g., 0.5 bits per sample. If the gain value from the AGC is negative, the output from the spatial metadata bandwidth estimation will be the minimum. There is also a maximum value for the required bandwidth for spatial metadata. This value can be chosen to be equal or less than the amount of spatial metadata coming from Spatial Encoding 902 in Fig. 9 . If it is known a prior that there is an upper limit to how much spatial metadata bandwidth can be embedded to the signal in a transparent way, that value should be selected as the maximum.
- Available Metadata Bandwidth Estimator 1002 becomes to use the minimum value when the AGC gain control value is negative.
- the estimate will use the formula of 1 bit of extra bandwidth for every 6dB of gain to add to the bandwidth estimation based on the current AGC control value.
- the maximum allowed value for the spatial metadata bandwidth mentioned above will be used as a limit for the output.
- the simple model described above can be extended with use of a Psychoacoustic Model 1004. Now the absolute minimum limit for the required bandwidth can stay unchanged, but the bandwidth above that can be adjusted together with an estimate from the Psychoacoustic Model.
- the process can be, e.g., iterative so that various amounts of spatial metadata are embedded to the signal and the highest one without significant changes to the perceived quality is selected.
- the procedure can be as follows: the method creates first just the spatial metadata signal to the lowest bits at a given amount and estimates its perceptual spectrum, e.g., by calculating powers at perceptually motivated frequency bands such as the Bark scale. The same is done for the content signal. Then, spectral comparison is performed to determine whether at some frequency bands the spatial metadata noise power exceeds the signal power. If this happens, then the system can determine that spatial metadata noise power is too high and it determine to use the previous amount which did not cause such an exceed of spatial metadata noise signal power above the actual signal power.
- Metadata Signal Embedding 1010 does not need the Metadata Reduction 1006 and Compression 1008.
- the data to embed can be selected more randomly, if we assume that the perceptual effect will be mostly dependent on the data amount and not the data contents.
- the Metadata Reduction 1006 is responsible for lossy compression of the spatial metadata.
- the Metadata Reduction reduces the amount of spatial metadata.
- the reduction of spatial metadata means storing less bits for each value. Storing zero bits for a value means discarding that value altogether. Storing more than zero and less than the original amount means quantizing the value to a coarser scale.
- the Metadata Reduction 1006 may be a rule-based system for the reduction of spatial metadata.
- the spatial metadata coming from spatial processing can vary.
- One commonly used approach is to partition the signal into time-frequency (TF) tiles and then store analysis data for each tile.
- the values related to a tile can be, e.g., information about the amounts (or ratios) of direct and ambient sound in that tile. There can also be a value for the direction of arrival for the most prominent sound source related to that TF tile.
- the rules may be used to determine which spatial metadata to trim.
- the application use case can affect this behaviour.
- Application metadata e.g. from an application for capturing spatial sound, may be used for determining the use case.
- spatial metadata e.g. from an application for capturing spatial sound
- Another approach is to choose to favour all spatial metadata that identifies direct sound and preserve bits for them, regardless of the direction of arrival.
- the ambient spatial metadata can be chosen to be quantized more heavily. This will affect the spatial sound image but will preserve the direct sound information better, which will affect the rendering when controlling listener orientation. Yet another approach is to combine spatial metadata across TF tiles, for example, use the same direction-of-arrival estimates for different TF tiles, such as TF tiles having the same timestamp across a range of frequencies.
- the Metadata Reduction 1006 may apply an algorithm for reducing the spatial metadata.
- the algorithm may receive the original spatial metadata, the available budget/bandwidth for spatial metadata size and rules that may define priorities about which aspects to preserve and which to quantize. It can then, e.g., iteratively coarsen the quantization for those elements which the priorities allow until it meets the size budget.
- Metadata Compression is a lossless compression. It can be implemented, e.g., using widely known entropy coding methods.
- the Metadata Signal Embedding 1010 may write the reduced and optionally compressed spatial metadata to the audio signal. It may use the lowest bits of the audio signal. There can be an approach to spread the spatial metadata among the lowest bits in a way that minimizes the perceptual effect on the audio signal.
- Figs. 11 and 12 illustrate example block diagrams of apparatuses for processing an audio signal comprising embedded metadata in accordance with at least some embodiments of preferable aspects of the claimed invention.
- Fig. 11 presents an example of an application processor where an audio signal with embedded spatial metadata is received and encoded into a file format supporting spatial features. The audio passes to the application processor 1100 through one or more HAL layers 1102, which are unaware of the embedded spatial metadata. The audio is received by a Spatial Audio Format Encoder 1104. The Spatial Audio Format Encoder may extract by a Metadata Extractor 1106 the spatial metadata from the received PCM signal. The extraction can leave the signal unmodified, but another approach is that the extraction process also removes the spatial metadata from the signal producing the original signal to which the spatial metadata was embedded. After the spatial metadata extraction, the signal is fed to an AAC Encoder 1108. If the spatial metadata was present in the signal that was fed to the AAC encoder, it is no longer possible to read from the AAC signal - the lossy AAC compression will not preserve it.
- the produced AAC bitstream is injected by AAC Metadata Injector 1110 with the spatial metadata that was extracted from the received PCM signal.
- the injection method is to add spatial metadata to DSE elements of the bitstream.
- the bitstream can then be multiplexed by a Muxer 1112 to an MP4 file together with a video track.
- the resulting file is playable on standard compliant players.
- Fig. 12 presents an example of an application processor where an audio signal with embedded spatial metadata is received and used by applications that support spatial features and applications that do not support spatial features.
- the audio passes to the application processor 1200 through one or more HAL layers 1202, which are unaware of the embedded spatial metadata.
- the audio is received by a Spatial Audio Application that support spatial features and one or more Unmodified Audio Applications 1206 that do not support spatial features.
- the Unmodified Audio Applications are not aware of the embedded spatial metadata in the signal, and the spatial metadata does not affect their operation, whereby they function in the same way as for a signal with no added spatial metadata.
- the Unmodified Audio Applications will not break and since the spatial metadata is not audible in the signal, it will not cause degradation in the user experience.
- the Spatial Audio Application may extract by a Metadata Extractor 1208 the spatial metadata from the received PCM signal.
- the extraction can leave the signal unmodified, but another approach is that the extraction process also removes the spatial metadata from the signal producing the original signal to which the spatial metadata was embedded.
- the audio and spatial metadata can be used as input to Spatial Synthesis 1210.
- An apparatus comprising at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform:
- a memory may be a computer readable medium that may be non-transitory.
- the memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory.
- the data processors may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on multi-core processor architecture, as non-limiting examples.
- Embodiments may be implemented in software, hardware, application logic or a combination of software, hardware and application logic.
- the software, application logic and/or hardware may reside on memory, or any computer media.
- the application logic, software or an instruction set is maintained on any one of various conventional computer-readable media.
- a "memory" or “computer-readable medium” may be any media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.
- references to, where relevant, "computer-readable storage medium”, “computer program product”, “tangibly embodied computer program” etc., or a “processor” or “processing circuitry” etc. should be understood to encompass not only computers having differing architectures such as single/multi-processor architectures and sequencers/parallel architectures, but also specialized circuits such as field programmable gate arrays FPGA, application specify circuits ASIC, signal processing devices and other devices.
- References to computer readable program code means, computer program, computer instructions, computer code etc. should be understood to express software for a programmable processor firmware such as the programmable content of a hardware device as instructions for a processor or configured or configuration settings for a fixed function device, gate array, programmable logic device, etc.
- embodiments of the invention operating within an apparatus or electronic device, it would be appreciated that the invention as described above may be implemented as a part of any apparatus comprising a circuitry for processing audio signals and metadata.
- embodiments of the invention may be implemented in a mobile phone, in a computer such as a desktop computer or a tablet computer.
- the various embodiments of the invention may be implemented in hardware or special purpose circuits or any combination thereof. While various aspects of the invention may be illustrated and described as block diagrams or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
- Embodiments of the inventions may be practiced in various components such as integrated circuit modules, field-programmable gate arrays (FPGA), application specific integrated circuits (ASIC), microcontrollers, microprocessors, a combination of such modules.
- FPGA field-programmable gate arrays
- ASIC application specific integrated circuits
- microcontrollers microcontrollers
- microprocessors a combination of such modules.
- the design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
- Programs such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre stored design modules.
- the resultant design in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.
- circuitry may refer to one or more or all of the following:
- circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and/or firmware.
- circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.
Landscapes
- Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Stereophonic System (AREA)
Claims (15)
- Procédé caractérisé en ce qu'il comprend :- la détermination d'une largeur de bande pour intégrer des métadonnées spatiales à un signal audio sur la base d'une valeur de commande de gain adaptatif pour le signal audio ; et- l'intégration des métadonnées spatiales au signal audio sur la base de la largeur de bande déterminée.
- Procédé selon la revendication 1, comprenant :- la détermination, sur la base de la valeur de commande de gain adaptatif indiquant une augmentation de gain et d'une commande de gain adaptative du signal audio utilisant la valeur de commande de gain adaptatif, de la largeur de bande par échantillon du signal audio comme étant un nombre de bits pour lequel l'échantillon a été décalé vers le bit le plus significatif.
- Procédé selon la revendication 1 ou 2, comprenant :- la détermination, si la valeur de commande de gain adaptatif ne parvient pas à indiquer une augmentation de gain, de la largeur de bande comme étant une largeur de bande minimale.
- Procédé selon l'une des revendications 1 à 3, comprenant :- la compression, si la largeur de bande déterminée sur la base de la valeur de commande de gain adaptatif est insuffisante, des métadonnées spatiales ; et/oula détermination, si la largeur de bande déterminée sur la base de la valeur de commande de gain adaptatif est insuffisante, de la largeur de bande sur la base d'un spectre perceptif du signal audio comprenant les métadonnées spatiales intégrées.
- Appareil caractérisé en ce qu'il comprend :des moyens pour déterminer une largeur de bande pour intégrer des métadonnées spatiales à un signal audio sur la base d'une valeur de commande de gain adaptatif pour le signal audio ; etdes moyens pour intégrer les métadonnées spatiales au signal audio sur la base de la largeur de bande déterminée.
- Appareil selon la revendication 5, comprenant :- des moyens pour déterminer, sur la base de la valeur de commande de gain adaptatif indiquant une augmentation de gain et d'une commande de gain adaptative du signal audio utilisant la valeur de commande de gain adaptatif, la largeur de bande par échantillon du signal audio comme étant un nombre de bits pour lequel l'échantillon a été décalé vers le bit le plus significatif.
- Appareil selon la revendication 5 ou 6, comprenant :- des moyens pour déterminer, si la valeur de commande de gain adaptatif ne parvient pas à indiquer une augmentation de gain, la largeur de bande comme étant une largeur de bande minimale.
- Appareil selon l'une des revendications 5 à 7, comprenant :- des moyens pour compresser, si la largeur de bande déterminée sur la base de la valeur de commande de gain adaptatif est insuffisante, les métadonnées spatiales ; et/ou- des moyens pour déterminer, si la largeur de bande déterminée sur la base de la valeur de commande de gain adaptatif est insuffisante, la largeur de bande sur la base d'un spectre perceptif du signal audio comprenant les métadonnées spatiales intégrées.
- Appareil selon l'une des revendications 5 à 8, comprenant :- des moyens pour stocker le signal audio dans des tampons audio ; et au moins un parmi :- des moyens pour stocker des données d'analyse pour chaque tampon audio,- des moyens pour hiérarchiser des métadonnées spatiales sur la base de métadonnées d'application, et- des moyens pour combiner des métadonnées spatiales dans les tampons audio.
- Appareil selon l'une des revendications 5 à 9, dans lequel la largeur de bande déterminée comprend les bits les moins significatifs d'échantillons du signal audio.
- Appareil selon la revendication 10, comprenant :- des moyens pour entrelacer la largeur de bande déterminée avec les bits les moins significatifs d'échantillons du signal audio.
- Appareil selon l'une des revendications 5 à 11,- des moyens pour déterminer un spectre perceptif de référence sur la base d'un modèle psychoacoustique et du signal audio sans métadonnées spatiales intégrées ;- des moyens pour déterminer des spectres perceptifs candidats sur la base du modèle psychoacoustique et du signal audio intégré avec des métadonnées spatiales en utilisant des largeurs de bande candidates pour les métadonnées spatiales ; et- des moyens pour déterminer la largeur de bande pour intégrer des métadonnées spatiales à partir des une ou plusieurs largeurs de bande candidates en évaluant si les spectres perceptifs candidats dépassent le spectre perceptif de référence.
- Appareil selon la revendication 12, comprenant :- des moyens pour déterminer la largeur de bande pour intégrer des métadonnées spatiales comme étant la largeur de bande candidate la plus élevée pour laquelle un spectre perceptif, sur la base du modèle psychoacoustique et du signal audio intégré avec des métadonnées spatiales, ne parvient pas à dépasser le spectre perceptif de référence.
- Appareil selon l'une des revendications 5 à 13, comprenant :- des moyens pour recevoir le signal audio intégré avec des métadonnées spatiales ;- des moyens pour extraire les métadonnées spatiales du signal audio reçu ; et au moins un parmi :- des moyens pour coder le signal audio reçu selon un format de codage audio avec perte où les métadonnées spatiales extraites sont incluses dans des éléments de métadonnées du format de codage audio ; et- des moyens pour réaliser une synthèse spatiale sur la base des métadonnées spatiales extraites et du signal audio reçu, ou du signal audio reçu, où les métadonnées spatiales ont été supprimées.
- Appareil selon l'une des revendications 5 à 14, dans lequel le signal audio est un signal de modulation de code d'impulsion, PCM, à deux canaux.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
FI20205166 | 2020-02-18 |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3869507A1 EP3869507A1 (fr) | 2021-08-25 |
EP3869507B1 true EP3869507B1 (fr) | 2023-07-12 |
Family
ID=74586825
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP21156162.6A Active EP3869507B1 (fr) | 2020-02-18 | 2021-02-10 | Incorporation de metadonées spatiales dans des signaux audio |
Country Status (1)
Country | Link |
---|---|
EP (1) | EP3869507B1 (fr) |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8059815B2 (en) * | 2001-12-13 | 2011-11-15 | Digimarc Corporation | Transforming data files into logical storage units for auxiliary data through reversible watermarks |
-
2021
- 2021-02-10 EP EP21156162.6A patent/EP3869507B1/fr active Active
Also Published As
Publication number | Publication date |
---|---|
EP3869507A1 (fr) | 2021-08-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3692524B1 (fr) | Codage audio à flux multiples | |
US10885921B2 (en) | Multi-stream audio coding | |
EP2898506B1 (fr) | Approche de codage audio spatial en couches | |
CN110537220B (zh) | 信号处理设备和方法及程序 | |
CN112562696B (zh) | 具有离散对象的音频的分层编码 | |
US20230274748A1 (en) | Coding of multi-channel audio signals | |
CN111819863A (zh) | 用音频信号及相关联元数据表示空间音频 | |
KR20180040716A (ko) | 음질 향상을 위한 신호 처리방법 및 장치 | |
US9449602B2 (en) | Dual uplink pre-processing paths for machine and human listening | |
CN111149157A (zh) | 使用经扩展参数对高阶立体混响系数的空间关系译码 | |
CN114365218A (zh) | 空间音频参数编码和相关联的解码的确定 | |
US20200015028A1 (en) | Energy-ratio signalling and synthesis | |
EP4131261A1 (fr) | Procédé de codage, procédé de décodage, dispositif de codage et dispositif de décodage de signal audio | |
EP3869507B1 (fr) | Incorporation de metadonées spatiales dans des signaux audio | |
WO2019106221A1 (fr) | Traitement de paramètres audio spatiaux | |
EP2786373B1 (fr) | Amélioration de qualité de captures multimédia | |
CN114008705A (zh) | 基于操作条件执行心理声学音频编解码 | |
KR20230153402A (ko) | 다운믹스 신호들의 적응형 이득 제어를 갖는 오디오 코덱 | |
US11380344B2 (en) | Device and method for controlling a speaker according to priority data | |
RU2648632C2 (ru) | Классификатор многоканального звукового сигнала | |
WO2023051367A1 (fr) | Procédé et appareil de décodage, et dispositif, support de stockage et produit programme d'ordinateur | |
WO2023232823A1 (fr) | Titre: codage audio spatialisé avec adaptation d'un traitement de décorrélation | |
JP2014002220A (ja) | 携帯型コンテンツ再生装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20220223 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20230210 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10L 19/008 20130101ALN20230131BHEP Ipc: G10L 19/002 20130101ALI20230131BHEP Ipc: G10L 19/018 20130101AFI20230131BHEP |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602021003334 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20230712 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1587991 Country of ref document: AT Kind code of ref document: T Effective date: 20230712 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231013 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231112 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231113 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231012 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231112 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231013 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602021003334 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20231229 Year of fee payment: 4 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 |
|
26N | No opposition filed |
Effective date: 20240415 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230712 |