EP3005356B1 - Efficient coding of audio scenes comprising audio objects - Google Patents
Efficient coding of audio scenes comprising audio objects Download PDFInfo
- Publication number
- EP3005356B1 EP3005356B1 EP14730451.3A EP14730451A EP3005356B1 EP 3005356 B1 EP3005356 B1 EP 3005356B1 EP 14730451 A EP14730451 A EP 14730451A EP 3005356 B1 EP3005356 B1 EP 3005356B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- audio objects
- audio
- downmix
- metadata
- downmix signals
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000009877 rendering Methods 0.000 claims description 158
- 238000000034 method Methods 0.000 claims description 126
- 238000004590 computer program Methods 0.000 claims description 18
- 230000007704 transition Effects 0.000 description 139
- 230000005236 sound signal Effects 0.000 description 51
- 239000011159 matrix material Substances 0.000 description 48
- 238000012952 Resampling Methods 0.000 description 20
- 230000008569 process Effects 0.000 description 13
- 238000004458 analytical method Methods 0.000 description 12
- 230000008901 benefit Effects 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 230000003068 static effect Effects 0.000 description 8
- 238000013459 approach Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000001419 dependent effect Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000007621 cluster analysis Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000009795 derivation Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007717 exclusion Effects 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 229940050561 matrix product Drugs 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 239000000654 additive Substances 0.000 description 1
- 230000000996 additive effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000001066 destructive effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 238000010561 standard procedure Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/018—Audio watermarking, i.e. embedding inaudible data in the audio signal
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
- H04S2400/03—Aspects of down-mixing multi-channel audio to configurations with lower numbers of playback channels, e.g. 7.1 -> 5.1
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
- H04S2400/11—Positioning of individual sound objects, e.g. moving airplane, within a sound field
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S3/00—Systems employing more than two channels, e.g. quadraphonic
- H04S3/008—Systems employing more than two channels, e.g. quadraphonic in which the audio signals are in digital form, i.e. employing more than two discrete digital channels
Definitions
- the disclosure herein generally relates to coding of an audio scene comprising audio objects.
- it relates to an encoder, a decoder and associated methods for encoding and decoding of audio objects.
- An audio scene may generally comprise audio objects and audio channels.
- An audio object is an audio signal which has an associated spatial position which may vary with time.
- An audio channel is an audio signal which corresponds directly to a channel of a multichannel speaker configuration, such as a so-called 5.1 speaker configuration with three front speakers, two surround speakers, and a low frequency effects speaker.
- a legacy decoder which does not support audio object reconstruction may use the multichannel downmix directly for playback on the multichannel speaker configuration.
- a 5.1 downmix may directly be played on the loudspeakers of a 5.1 configuration.
- a disadvantage with this approach is however that the multichannel downmix may not give a sufficiently good reconstruction of the audio objects at the decoder side. For example, consider two audio objects that have the same horizontal position as the left front speaker of a 5.1. configuration but a different vertical position. These audio objects would typically be combined into the same channel of a 5.1 downmix. This would constitute a challenging situation for the audio object reconstruction at the decoder side which would have to reconstruct approximations of the two audio objects from the same downmix channel, a process that cannot ensure perfect reconstruction and that sometimes even lead to audible artifacts.
- Side information or metadata is often employed during reconstruction of audio objects from e.g. a downmix.
- the form and content of such side information may for example affect the fidelity of the reconstructed audio objects and/or the computational complexity of performing the reconstruction. It would therefore be desirable to provide encoding/decoding methods with a new and alternative side information format which allows for increasing the fidelity of reconstructed audio objects, and/or which allows for reducing the computational complexity of the reconstruction.
- a device for audio signal processing comprises a cluster analysis module configured to group, based on spatial information for each of N audio objects, a plurality of audio objects that includes the N audio objects into L clusters, where L is less than N, wherein the cluster analysis module is configured to receive information from at least one of a transmission channel, a decoder, and a renderer, and wherein a maximum value for L is based on the information received.
- the device also comprises a downmix module configured to mix the plurality of audio objects into L audio streams, and a metadata downmix module configured to produce, based on the spatial information and the grouping, metadata that indicates spatial information for each of the L audio streams
- an encoding method an encoder and a computer program product for encoding audio objects, as recited in claims.
- a method for encoding audio objects into a data stream comprising inter alia:
- the M downmix signals are thus formed from the N audio objects independently of any loudspeaker configuration.
- the M downmix signals are not constrained to audio signals which are suitable for playback on the channels of a speaker configuration with M channels.
- the M downmix signals may be selected more freely according to a criterion such that they for instance adapt to the dynamics of the N audio objects and improve the reconstruction of the audio objects at the decoder side.
- the proposed method allows to put the first audio object in a first downmix signal, and the second audio object in the second downmix signal. This enables perfect reconstruction of the audio objects in the decoder. In general, such perfect reconstruction is possible as long as the number of active audio objects does not exceed the number of downmix signals. If the number of active audio objects is higher, then the proposed method allows for selection of the audio objects that have to be mixed into the same downmix signal such that the possible approximation errors occurring in the reconstructed audio object in the decoder have no or the smallest possible perceptual impact on the reconstructed audio scene.
- a second advantage of the M downmix signals being adaptive is the ability to keep certain audio objects strictly separate from other audio objects. For example, it can be advantageous to keep any dialog object separate from background objects, to ensure that dialog is rendered accurately in terms of spatial attributes, and allows for object processing in the decoder, such as dialog enhancement or increase of dialog loudness for improved intelligibility. In other applications (e.g. karaoke), it may be advantageous to allow complete muting of one or more objects, which also requires that such objects are not mixed with other objects. Conventional methods using a multichannel downmix corresponding to a specific speaker configuration do not allow for complete muting of audio objects present in a mix of other audio objects.
- the word downmix signal reflects that a downmix signal is a mix, i.e. a combination, of other signals.
- the word "down" indicates that the number M of downmix signals typically is lower than the number N of audio objects.
- the method may further comprise associating each downmix signal with a spatial position and including the spatial positions of the downmix signals in the data stream as metadata for the downmix signals.
- the metadata associated with the downmix signals may be used on a decoder side for rendering the downmix signals to the channels of a legacy playback system.
- the N audio objects are associated with metadata including spatial positions of the N audio objects, and the spatial positions associated with the downmix signals are calculated based on the spatial positions of the N audio objects.
- the downmix signals may be interpreted as audio objects having a spatial position which depends on the spatial positions of the N audio objects.
- the side information is also time-varying thereby allowing the parameters governing the reconstruction of the audio objects to vary temporally.
- the encoder may apply different criteria for the calculation of the downmix signals.
- the criterion for calculating the M downmix signals may be based on spatial proximity of the N audio objects. For example, audio objects which are close to each other may be combined into the same downmix signal.
- the criterion for calculating the M downmix signals may further be based on the importance values of the N audio objects. For example, the most important one(s) of the N audio objects may be mapped directly to a downmix signal, while the remaining audio objects are combined to form the remaining downmix signals.
- the step of calculating M downmix signals comprises a first clustering procedure which includes associating the N audio objects with M clusters based on spatial proximity and importance values, if applicable, of the N audio objects, and calculating a downmix signal for each cluster by forming a combination of audio objects associated with the cluster.
- an audio object may form part of at most one cluster.
- an audio object may form part of several clusters. In this way, different groups, i.e. clusters, are formed from the audio objects.
- Each cluster may in turn be represented by a downmix signal which may be thought of as an audio object.
- the clustering approach allows associating each downmix signal with a spatial position which is calculated based on the spatial positions of the audio objects associated with the cluster corresponding to the downmix signal. With this interpretation the first clustering procedure thus reduces the dimensionality of the N audio objects to M audio objects in a flexible manner.
- the spatial position associated with each downmix signal may for example be calculated as a centroid or a weighted centroid of the spatial positions of the audio objects associated with the cluster corresponding to the downmix signal.
- the weights may for example be based on importance values of the audio objects.
- the N audio objects are associated with the M clusters by applying a K-means algorithm having the spatial positions of the N audio objects as input.
- the method may take further measures for reducing the dimensionality of the audio scene, thereby reducing the computational complexity at the decoder side when reconstructing the audio objects.
- the method may further comprise a second clustering procedure for reducing a first plurality of audio objects to a second plurality of audio objects.
- the second clustering procedure is performed prior to the calculation of the M downmix signals.
- the first plurality of audio objects hence correspond to the original audio objects of the audio scene
- the second, reduced, plurality of audio objects corresponds to the N audio objects on the basis of which the M downmix signals are calculated.
- the set of audio objects (to be reconstructed in the decoder) formed on basis of the N audio objects corresponds, i.e. is equal to, to the N audio objects.
- the second clustering procedure is performed in parallel with the calculation of the M downmix signals.
- the N audio objects on the basis of which the M downmix signals are calculated as well as the first plurality of audio objects being input to the second clustering procedure correspond to the original audio objects of the audio scene.
- the set of audio objects (to be reconstructed in the decoder) formed on basis of the N audio objects corresponds to the second plurality of audio objects.
- the second clustering procedure comprises:
- the second clustering procedure exploits spatial redundancy present in the audio scene, such as objects having equal or very similar locations.
- importance values of the audio objects may be taken into account when generating the second plurality of audio objects.
- the audio scene may also comprise audio channels.
- Such audio channels may be thought of as an audio object being associated with a static position, viz. the position of the loudspeaker corresponding to the audio channel.
- the second clustering procedure may further comprise:
- the method allows for encoding of an audio scene comprising audio channels as well as audio objects.
- a computer program product comprising a computer-readable medium with instructions for performing the decoding method according to exemplary embodiments.
- an encoder for encoding audio objects into a data stream comprising:
- the second aspect may generally have the same features and advantages as the first aspect.
- a method in a decoder for decoding a data stream including encoded audio objects comprising inter alia:
- the data stream further comprises metadata for the M downmix signals including spatial positions associated with the M downmix signals, the method further comprising:
- the spatial positions associated with the M downmix signals are time-varying.
- the side information is time-varying.
- the data stream further comprises metadata for the set of audio objects formed on basis of the N audio objects including the spatial positions of the set of audio objects formed on basis of the N audio objects, the method further comprising:
- the set of audio objects formed on basis of the N audio objects is equal to the N audio objects.
- the set of audio objects formed on basis of the N audio objects comprises a plurality of audio objects which are combinations of the N audio objects, and the number of which is lower than N.
- a computer program product comprising a computer-readable medium with instructions for performing the decoding method according to exemplary embodiments.
- a decoder for decoding a data stream including encoded audio objects comprising:
- an encoding method an encoder, and a computer program product for encoding audio objects.
- the methods, encoders and computer program products according to the third aspect may generally have features and advantages in common with the methods, encoders and computer program products according to the first aspect.
- a method for encoding audio objects as a data stream comprises inter alia:
- the method further comprises including, in the data stream:
- the side information is time-variable, e.g. time-varying, allowing for the parameters governing the reconstruction of the audio objects to vary with respect to time, which is reflected by the presence of the side information instances.
- a side information format which includes transition data defining points in time to begin and points in time to complete transitions from current reconstruction settings to respective desired reconstruction settings
- the side information instances are made more independent of each other in the sense that interpolation may be performed based on a current reconstruction setting and a single desired reconstruction setting specified by a single side information instance, i.e. without knowledge of any other side information instances.
- the provided side information format therefore facilitates calculation/introduction of additional side information instances between existing side information instances.
- the provided side information format allows for calculation/introduction of additional side information instances without affecting the playback quality.
- the process of calculating/introducing new side information instances between existing side information instances is referred to as "resampling" of the side information. Resampling of side information is often required during certain audio processing tasks. For example, when audio content is edited, by e.g. cutting/merging/mixing, such edits may occur in between side information instances. In this case, resampling of the side information may be required. Another such case is when audio signals and associated side information are encoded with a frame-based audio codec.
- the audio signals/objects may be part of an audio-visual signal or multimedia signal which includes video content.
- the data stream in which the downmix signal and the side information is included may for example be a bitstream, in particular a stored or transmitted bitstream.
- calculating the M downmix signals by forming combinations of the N audio objects means that each of the M downmix signals is obtained by forming a combination, e.g. a linear combination, of the audio content of one or more of the N audio objects. In other words, each of the N audio objects need not necessarily contribute to each of the M downmix signals.
- the word downmix signal reflects that a downmix signal is a mix, i.e. a combination, of other signals.
- the downmix signal may for example be an additive mix of other signals.
- the word "down" indicates that the number M of downmix signals typically is lower than the number N of audio objects.
- transition data including two independently assignable portions is meant that the two portions are mutually independently assignable, i.e. may be assigned independently of each other.
- the portions of the transition data may for example coincide with portions of transition data for other types of side information of metadata.
- the two independently assignable portions of the transition data in combination, define the point in time to begin the transition and the point in time to complete the transition, i.e. these two points in time are derivable from the two independently assignable portions of the transition data.
- the method may further comprise a clustering procedure for reducing a first plurality of audio objects to a second plurality of audio objects, wherein the N audio objects constitute either the first plurality of audio objects or the second plurality of audio objects, and wherein the set of audio objects formed on the basis of the N audio objects coincides with the second plurality of audio objects.
- the clustering procedure may comprise:
- the method according to the present example embodiment takes further measures for reducing the dimensionality of the audio scene by reducing the first plurality of audio objects to a second plurality of audio objects.
- the set of audio objects which is formed on the basis of the N audio objects and which is to be reconstructed on a decoder side based on the downmix signals and the side information, coincides with the second plurality of audio objects, which corresponds to a simplification and/or lower-dimensional representation of the audio scene represented by the first plurality of audio signals, and the computational complexity for reconstruction on a decoder side is reduced.
- the inclusion of the cluster metadata in the data stream allows for rendering of the second set of audio signals on a decoder side, e.g. after the second set of audio signals has been reconstructed based on the downmix signals and the side information.
- the cluster metadata in the present example embodiment is time-variable, e.g. time-varying, allowing for the parameters governing the rendering of the second plurality of audio objects to vary with respect to time.
- the format for the downmix metadata may be analogous to that of the side formation and may have the same or corresponding advantages.
- the form of the cluster metadata provided in the present example embodiment facilitates resampling of the cluster metadata. Resampling of the cluster metadata may e.g. be employed to provide common points in time to start and complete respective transitions associated with the cluster metadata and the side information, and/or for adjusting the cluster metadata to a frame rate of the associated audio signals.
- the clustering procedure may further comprise:
- the clustering procedure exploits spatial redundancy present in the audio scene, such as objects having equal or very similar locations.
- importance values of the audio objects may be taken into account when generating the second plurality of audio objects, as described with respect to example embodiments within the first aspect.
- Associating the first plurality of audio objects with at least one cluster includes associating each of the first plurality of audio objects with one or more of the at least one cluster.
- an audio object may form part of at most one cluster, while in other cases, an audio object may form part of several clusters. In other words, in some cases, an audio object may be split between several clusters as part of the clustering procedure.
- Spatial proximity of the first plurality of audio objects may be related to distances between, and/or relative positions of, the respective audio objects in the first plurality of audio objects. For example, audio objects which are close to each other may be associated with the same cluster.
- an audio object being a combination of the audio objects associated with the cluster is meant that the audio content/signal associated with the audio object may be formed as a combination of the audio contents/signals associated with the respective audio objects associated with the cluster.
- the respective points in time defined by the transition data for the respective cluster metadata instances may coincide with the respective points in time defined by the transition data for corresponding side information instances.
- joint settings for reconstruction and rendering may be determined for each side information instance and metadata instance and/or interpolation between joint settings for reconstruction and rendering may be employed instead of performing interpolation separately for the respective settings.
- Such joint interpolation may reduce computational complexity at the decoder side as fewer coefficients/parameters need to be interpolated.
- the clustering procedure may be performed prior to the calculation of the M downmix signals.
- the first plurality of audio objects corresponds to the original audio objects of the audio scene
- the N audio objects on the basis of which the M downmix signals are calculated constitute the second, reduced, plurality of audio objects.
- the set of audio objects (to be reconstructed on a decoder side) formed on the basis of the N audio objects coincides with the N audio objects.
- the clustering procedure may be performed in parallel with the calculation of the M downmix signals.
- the N audio objects on the basis of which the M downmix signals are calculated constitute the first plurality of audio objects which correspond to the original audio objects of the audio scene.
- the M downmix signals are hence calculated on basis of the original audio objects of the audio scene and not on basis of a reduced number of audio objects.
- the method may further comprise:
- downmix metadata in the data stream is advantageous in that it allows for low-complexity decoding to be used in case of legacy playback equipment. More precisely, the downmix metadata may be used on a decoder side for rendering the downmix signals to the channels of a legacy playback system, i.e. without reconstructing the plurality of audio objects formed on the basis of the N objects, which typically is a computationally more complex operation.
- the spatial positions associated with the M downmix signals may be time-variable, e.g. time- varying, and the downmix signals may be interpreted as dynamic audio objects having an associated position which may change between time frames or downmix metadata instances.
- the downmix signals correspond to fixed spatial loudspeaker positions. It is recalled that the same data stream may be played in an object oriented fashion in a decoding system with more evolved capabilities.
- the N audio objects may be associated with metadata including spatial positions of the N audio objects, and the spatial positions associated with the downmix signals may for example be calculated based on the spatial positions of the N audio objects.
- the downmix signals may be interpreted as audio objects having spatial positions which depend on the spatial positions of the N audio objects.
- the respective points in time defined by the transition data for the respective downmix metadata instances may coincide with the respective points in time defined by the transition data for corresponding side information instances.
- Employing the same points in time for beginning and for completing transitions associated with the side information and the downmix metadata facilitates joint processing, e.g. resampling, of the side information and the downmix metadata.
- the respective points in time defined by the transition data for the respective downmix metadata instances may coincide with the respective points in time defined by the transition data for corresponding cluster metadata instances.
- Employing the same points in time for beginning and ending transitions associated with the cluster metadata and the downmix metadata facilitates joint processing, e.g. resampling, of the cluster metadata and the downmix metadata.
- the encoder comprises:
- a decoding method for decoding multichannel audio content.
- the methods, decoders and computer program products according to the fourth aspect are intended for cooperation with the methods, encoders and computer program products according to the third aspect, and may have corresponding features and advantages.
- the methods, decoders and computer program products according to the fourth aspect may generally have features and advantages in common with the methods, decoders and computer program products according to the second aspect.
- a method for reconstructing audio objects based on a data stream comprises:
- employing a side information format which includes transition data defining points in time to begin and points in time to complete transitions from current reconstruction settings to respective desired reconstruction settings e.g. facilitates resampling of the side information.
- the data stream may for example be received in the form of a bitstream, e.g. generated on an encoder side.
- Reconstructing, based on the M downmix signals and the side information, the set of audio objects formed on the basis of the N audio objects may for example include forming at least one linear combination of the downmix signals employing coefficients determined based on the side information.
- Reconstructing, based on the M downmix signals and the side information, the set of audio objects formed on the basis of the N audio objects may for example include forming linear combinations of the downmix signals, and, optionally one or more additional (e.g. decorrelated) signal derived from the downmix signals, employing coefficients determined based on the side information.
- the data stream may further comprise time-variable cluster metadata for the set of audio objects formed on the basis of the N audio objects, the cluster metadata including spatial positions for the set of audio objects formed on the basis of the N audio objects.
- the data stream may comprise a plurality of cluster metadata instances, and the data stream may further comprise, for each cluster metadata instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current rendering setting to a desired rendering setting specified by the cluster metadata instance, and a point in time to complete the transition to the desired rendering setting specified by the cluster metadata instance.
- the method may further comprise:
- the predefined channel configuration may for example correspond to a configuration of the output channels compatible with a particular playback system, i.e. suitable for playback on a particular playback system.
- Rendering of the reconstructed set of audio objects formed on the basis of the N audio objects to output channels of a predefined channel configuration may for example include mapping, in a renderer, the reconstructed set of audio signals formed on the basis of the N audio objects to (a predefined configuration of) output channels of the renderer under control of the cluster metadata.
- Rendering of the reconstructed set of audio objects formed on the basis of the N audio objects to output channels of a predefined channel configuration may for example include forming linear combinations of the reconstructed set of audio objects formed on the basis of the N audio objects, employing coefficients determined based on the cluster metadata.
- the respective points in time defined by the transition data for the respective cluster metadata instances may coincide with the respective points in time defined by the transition data for corresponding side information instances.
- the method may further comprise:
- a matrix such as reconstruction matrix or a rendering matrix, as referenced in the present example embodiment, may for example consist of a single row or a single column, and may therefore correspond to a vector.
- Reconstruction of audio objects from downmix signals is often performed by employing different reconstruction matrices in different frequency bands, while rendering is often performed by employing the same rendering matrix for all frequencies.
- a matrix corresponding to a combined operation of reconstruction and rendering e.g. the first and second matrices referenced in the present example embodiment, may typically be frequency-dependent, i.e. different values for the matrix elements may typically be employed for different frequency bands.
- the set of audio objects formed on the basis of the N audio objects may coincide with the N audio objects, i.e. the method may comprise reconstructing the N audio objects based on the M downmix signals and the side information.
- the set of audio objects formed on the basis of the N audio objects may comprise a plurality of audio objects which are combinations of the N audio objects, and whose number is less than N, i.e. the method may comprise reconstructing these combinations of the N audio objects based on the M downmix signals and the side information.
- the data stream may further comprise downmix metadata for the M downmix signals including time-variable spatial positions associated with the M downmix signals.
- the data stream may comprise a plurality of downmix metadata instances, and the data stream may further comprise, for each downmix metadata instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current downmix rendering setting to a desired downmix rendering setting specified by the downmix metadata instance, and a point in time to complete the transition to the desired downmix rendering setting specified by the downmix metadata instance.
- the method may further comprise:
- the decoder may e.g. output the reconstructed set of audio objects the cluster metadata for rendering of the reconstructed set of audio objects.
- the decoder may for example discard the side information and, if applicable, the cluster metadata, and provide the downmix metadata and the M downmix signals as output. Then, the output may be employed by a renderer for rendering the M downmix signals to output channels of the renderer.
- the method may further comprise rendering the M downmix signals to output channels of a predefined output configuration, e.g. to output channels of a renderer, or to output channels of the decoder (in case the decoder has rendering capabilities), based on the downmix metadata.
- a predefined output configuration e.g. to output channels of a renderer, or to output channels of the decoder (in case the decoder has rendering capabilities
- a decoder for reconstructing audio objects based on a data stream.
- the decoder comprises:
- the method within the third or fourth aspect may further comprise generating one or more additional side information instances specifying substantially the same reconstruction setting as a side information instance directly preceding or directly succeeding the one or more additional side information instances.
- Example embodiments are also envisaged in which additional cluster metadata instances and/or downmix metadata instances are generated in an analogous fashion.
- the side information instances provided by an analysis component may e.g. be distributed in time in such a way that they do not match a frame rate of the downmix signals provided by a downmix component, and the side information may therefore advantageously be resampled by introducing new side information instances such that there is at least one side information instance for each frame of the downmix signals.
- the received side information instances may e.g.
- the side information may therefore advantageously be resampled by introducing new side information instances such that there is at least one side information instance for each frame of the downmix signals.
- An additional side information instance may for example be generated for a selected point in time by: copying the side information instance directly succeeding the additional side information instance and determining transition data for the additional side information instance based on the selected point in time and the points in time defined by the transition data for the succeeding side information instance.
- a method, a device, and a computer program product for transcoding side information encoded together with M audio signals in a data stream are provided.
- the methods, devices and computer program products according to the fifth aspect are intended for cooperation with the methods, encoders, decoder and computer program products according to the third and fourth aspect, and may have corresponding features and advantages.
- a method for transcoding side information encoded together with M audio signals in a data stream comprises:
- the one or more additional side information instances may be generated after he side information has been extracted from the received data stream, and the generated one or more additional side information instances may then be included in a data stream together with the M audio signals and the other side information instances.
- resampling of the side information by generating more side information instances may be advantageous in several situations, such as when audio signals/objects and associated side information are encoded using a frame-based audio codec, since then it is desirable to have at least one side information instance for each audio codec frame.
- Embodiments are also envisaged in which the data stream further comprises cluster metadata and/or downmix metadata, as described in relation to the third and fourth aspect, and wherein the method further comprises generating additional downmix metadata instances and/or cluster metadata instances, analogously to how the additional side information instances are generated.
- the M audio signals may be coded in the received data stream according to a first frame rate, and the method may further comprise:
- the third aspect it may be advantageous in several situations to process audio signals so as to change the frame rate employed for coding them, e.g. so that the modified frame rate matches the frame rate of video content of an audio-visual signal to which the audio signals belong.
- the presence of the transition data for each side information instance facilitates resampling of the side information, as described above in relation to the third aspect.
- the side information may be resampled to match the new frame rate e.g. by generating additional side information instances such that there is at least one side information instance for each frame of the processed audio signals.
- a device for transcoding side information encoded together with M audio signals in a data stream comprising:
- the device further comprises:
- the method within the third, fourth or fifth aspect may further comprise: computing a difference between a first desired reconstruction setting specified by a first side information instance and one or more desired reconstruction settings specified by one or more side information instances directly succeeding the first side information instance; and removing the one or more side information instances in response to the computed difference being below a predefined threshold.
- Example embodiments are also envisaged in which cluster metadata instances and/or downmix metadata instances are removed in an analogous fashion.
- side information instances By removing side information instances according to the present example embodiment, unnecessary computations based on these side information instances may be avoided, e.g. during reconstruction at a decoder side.
- the predefined threshold By setting the predefined threshold at an appropriate (e.g. low enough) level, side information instances may be removed while the playback quality and/or the fidelity of the reconstructed audio signals is at least approximately maintained.
- the difference between the desired reconstruction settings may for example be computed based on differences between respective values for a set of coefficients employed as part of the reconstruction.
- the two independently assignable portions of the transition data for each side information instance may be:
- the points in time to start and to end a transition may be defined in the transition data either by two time stamps indicating the respective points in time, or a combination of one of the time stamps and an interpolation duration parameter indicating a duration of the transition.
- the respective time stamps may for example indicate the respective points in time by referring to a time base employed for representing the M downmix signals and/or the N audio objects.
- the two independently assignable portions of the transition data for each cluster metadata instance may be:
- the two independently assignable portions of the transition data for each downmix metadata instance may be:
- a computer program product comprising a computer-readable medium with instructions for performing the method of any of the methods within the third, fourth or fifth aspect.
- Fig. 1 illustrates an encoder 100 for encoding audio objects 120 into a data stream 140 according to an exemplary embodiment.
- the encoder 100 comprises a receiving component (not shown), a downmix component 102, an encoder component 104, an analysis component 106, and a multiplexing component 108.
- the operation of the encoder 100 for encoding one time frame of audio data is described in the following. However, it is understood that the below method is repeated on a time frame basis. The same also applies to the description of Figs 2-5 .
- the receiving component receives a plurality of audio objects (N audio objects) 120 and metadata 122 associated with the audio objects 120.
- An audio object as used herein refers to an audio signal having an associated spatial position which typically is varying with time (between time frames), i.e. the spatial position is dynamic.
- the metadata 122 associated with the audio objects 120 typically comprises information which describes how the audio objects 120 are to be rendered for playback on the decoder side.
- the metadata 122 associated with the audio objects 120 includes information about the spatial position of the audio objects 120 in the three-dimensional space of the audio scene.
- the spatial positions can be represented in Cartesian coordinates or by means of direction angles, such as azimuth and elevation, optionally augmented with distance.
- the metadata 122 associated with the audio objects 120 may further comprise object size, object loudness, object importance, object content type, specific rendering instructions such as application of dialog enhancement or exclusion of certain loudspeakers from rendering (so-called zone masks) and/or other object properties.
- the audio objects 120 may correspond to a simplified representation of an audio scene.
- the N audio objects 120 are input to the downmix component 102.
- the downmix component 102 may further calculate one or more auxiliary audio signals 127, here labeled by L auxiliary audio signals 127.
- the role of the auxiliary audio signals 127 is to improve the reconstruction of the N audio objects 120 at the decoder side.
- the auxiliary audio signals 127 may correspond to one or more of the N audio objects 120, either directly or as a combination of these.
- the auxiliary audio signals 127 may correspond to particularly important ones of the N audio objects 120, such as an audio object 120 corresponding to a dialogue. The importance may be reflected by or derived from the metadata 122 associated with the N audio objects 120.
- the M downmix signals 124, and the L auxiliary signals 127 if present, may subsequently be encoded by the encoder component 104, here labeled core encoder, to generate M encoded downmix signals 126 and L encoded auxiliary signals 129.
- the encoder component 104 may be a perceptual audio codec as known in the art. Examples of known perceptual audio codecs include Dolby Digital and MPEG AAC.
- the downmix component 102 may further associate the M downmix signals 124 with metadata 125.
- downmix component 102 may associate each downmix signal 124 with a spatial position and include the spatial position in the metadata 125.
- the metadata 125 associated with the downmix signals 124 may also comprise parameters related to size, loudness, importance, and/or other properties.
- the spatial positions associated with the downmix signals 124 may be calculated based on the spatial positions of the N audio objects 120. Since the spatial positions of the N audio objects 120 may be dynamic, i.e. time-varying, also the spatial positions associated with the M downmix signals 124 may be dynamic. In other words, the M downmix signals 124 may themselves be interpreted as audio objects.
- the analysis component 106 calculates side information 128 including parameters which allow reconstruction of the N audio objects 120 (or a perceptually suitable approximation of the N audio objects 120) from the M downmix signals 124 and the L auxiliary signals 129 if present.
- the side information 128 may be time-variable.
- the analysis component 106 may calculate the side information 128 by analyzing the M downmix signals 124, the L auxiliary signals 127 if present, and the N audio objects 120 according to any known technique for parametric encoding.
- the analysis component 106 may calculate the side information 128 by analyzing the N audio objects, and information on how the M downmix signals were created from the N audio objects, for example by providing a (time-varying) downmix matrix. In that case, the M downmix signals 124 are not strictly required as an input to the analysis component 106.
- the M encoded downmix signals 126, the L encoded auxiliary signals 129, the side information 128, the metadata 122 associated with the N audio objects, and the metadata 125 associated with the downmix signals are then input to the multiplexing component 108 which includes its input data in a single data stream 140 using multiplexing techniques.
- the data stream 140 may thus include four types of data:
- the M downmix signals are chosen such that they are suitable for playback on the channels of a speaker configuration with M channels, referred to herein as a backwards compatible downmix.
- a backwards compatible downmix constrains the calculation of the downmix signals in that the audio objects may only be combined in a predefined manner. Accordingly, according to prior art, the downmix signals are not selected from the point of view of optimizing the reconstruction of the audio objects at a decoder side.
- the downmix component 102 calculates the M downmix signals 124 in a signal adaptive manner with respect to the N audio objects.
- the downmix component 102 may, for each time frame, calculate the M downmix signals 124 as the combination of the audio objects 120 that currently optimizes some criterion.
- the criterion is typically defined such that it is independent with respect to a any loudspeaker configuration, such as a 5.1 or other loudspeaker configuration. This implies that the M downmix signals 124, or at least one of them, are not constrained to audio signals which are suitable for playback on the channels of a speaker configuration with M channels.
- the downmix component 102 may adapt the M downmix signals 124 to the temporal variation of the N audio objects 120 (including temporal variation of the metadata 122 including spatial positions of the N audio objects), in order to e.g. improve the reconstruction of the audio objects 120 at the decoder side.
- the downmix component 102 may apply different criteria in order to calculate the M downmix signals.
- the M downmix signals may be calculated such that the reconstruction of the N audio objects based on the M downmix signals is optimized.
- the downmix component 102 may minimize a reconstruction error formed from the N audio objects 120 and a reconstruction of the N audio objects based on the M downmix signals 124.
- the criterion is based on the spatial positions, and in particular spatial proximity, of the N audio objects 120.
- the N audio objects 120 have associated metadata 122 which includes the spatial positions of the N audio objects 120. Based on the metadata 122, spatial proximity of the N audio objects 120 may be derived.
- the downmix component 102 may apply a first clustering procedure in order to determine the M downmix signals 124.
- the first clustering procedure may comprise associating the N audio objects 120 with M clusters based on spatial proximity. Further properties of the N audio objects 120 as represented by the associated metadata 122, including object size, object loudness, object importance, may also be taken into account during the association of the audio objects 120 with the M clusters.
- the well-known K-means algorithm with the metadata 122 (spatial positions) of the N audio objects as input, may be used for associating the N audio objects 120 with the M clusters based on spatial proximity.
- the further properties of the N audio objects 120 may be used as weighting factors in the K-means algorithm.
- the first clustering procedure may be based on a selection procedure which uses the importance of the audio objects, as given by the metadata 122, as a selection criterion.
- the downmix component 102 may pass through the most important audio objects 120 such that one or more of the M downmix signals correspond to one or more of the N audio objects 120.
- the remaining, less important, audio objects may be associated with clusters based on spatial proximity as discussed above.
- the first clustering procedure may associate an audio object 120 with more than one of the M clusters.
- an audio object 120 may be distributed over the M clusters, wherein the distribution e.g. depends on the spatial position of the audio object 120 and optionally also further properties of the audio object including object size, object loudness, object importance, etc.
- the distribution may be reflected by percentages, such that an audio object for instance is distributed over three clusters according to the percentages 20%, 30%, 50%.
- the downmix component 102 calculates a downmix signal 124 for each cluster by forming a combination, typically a linear combination, of the audio objects 120 associated with the cluster.
- the downmix component 102 may use parameters comprised in the metadata 122 associated with audio objects 120 as weights when forming the combination.
- the audio objects 120 being associated with a cluster may be weighted according to object size, object loudness, object importance, object position, distance from an object with respect to a spatial position associated with the cluster (see details in the following) etc.
- the percentages reflecting the distribution may be used as weights when forming the combination.
- the first clustering procedure is advantageous in that it easily allows association of each of the M downmix signals 124 with a spatial position.
- the downmix component 120 may calculate a spatial position of a downmix signal 124 corresponding to a cluster based on the spatial positions of the audio objects 120 associated with the cluster.
- the centroid or a weighted centroid of the spatial positions of the audio objects being associated with the cluster may be used for this purpose.
- the same weights may be used as when forming the combination of the audio objects 120 associated with the cluster.
- Fig. 2 illustrates a decoder 200 corresponding to the encoder 100 of Fig. 1 .
- the decoder 200 is of the type that supports audio object reconstruction.
- the decoder 200 comprises a receiving component 208, a decoder component 204, and a reconstruction component 206.
- the decoder 200 may further comprise a renderer 210.
- the decoder 200 may be coupled to a renderer 210 which forms part of a playback system.
- the receiving component 208 is configured to receive a data stream 240 from the encoder 100.
- the receiving component 208 comprises a demultiplexing component configured to demultiplex the received data stream 240 into its components, in this case M encoded downmix signals 226, optionally L encoded auxiliary signals 229, side information 228 for reconstruction of N audio objects from the M downmix signals and the L auxiliary signals, and metadata 222 associated with the N audio objects.
- the decoder component 204 processes the M encoded downmix signals 226 to generate M downmix signals 224, and optionally L auxiliary signals 227.
- the M downmix signals 224 were formed adaptively on the encoder side from the N audio objects, i.e. by forming combinations of the N audio objects according to a criterion which is independent of any loudspeaker configuration.
- the object reconstruction component 206 then reconstructs the N audio objects 220 (or a perceptually suitable approximation of these audio objects) based on the M downmix signals 224 and optionally the L auxiliary signals 227 guided by the side information 228 derived on the encoder side.
- the object reconstruction component 206 may apply any known technique for such parametric reconstruction of the audio objects.
- the reconstructed N audio objects 220 are then processed by the renderer 210 using the metadata 222 associated with the audio objects 222 and knowledge about the channel configuration of the playback system in order to generate an multichannel output signal 230 suitable for playback.
- Typical speaker playback configurations include 22.2 and 11.1. Playback on soundbar speaker systems or headphones (binaural presentation) is also possible with dedicated renderers for such playback systems.
- Fig. 3 illustrates a low-complexity decoder 300 corresponding to the encoder 100 of Fig. 1 .
- the decoder 300 does not support audio object reconstruction.
- the decoder 300 comprises a receiving component 308, and a decoding component 304.
- the decoder 300 may further comprise a renderer 310.
- the decoder is coupled to a renderer 310 which forms part of a playback system.
- a backwards compatible downmix such as a 5.1 downmix
- a downmix comprising M downmix signals which are suitable for direct playback on a playback system with M channels
- Such prior art systems typically decodes the backwards compatible downmix signals themselves and discards additional parts of the data stream such as side information (cf. item 228 of Fig. 2 ) and metadata associated with the audio objects (cf. item 222 of Fig. 2 ).
- the downmix signals are formed adaptively as described above, the downmix signals are generally not suitable for direct playback on a legacy system.
- the decoder 300 is an example of a decoder which allows low-complexity decoding of M downmix signals which are adaptively formed for playback on a legacy playback system which only supports a particular playback configuration.
- the receiving component 308 receives a bit stream 340 from an encoder, such as encoder 100 of Fig. 1 .
- the receiving component 308 demultiplexes the bit stream 340 into its components. In this case, the receiving component 308 will only keep the encoded M downmix signals 326 and the metadata 325 associated with the M downmix signals.
- the other components of the data stream 340 such as the L auxiliary signals (cf. item 229 of Fig. 2 ) metadata associated with the N audio objects (cf. item 222 of Fig. 2 ) and the side information (cf. item 228 of Fig. 2 ) are discarded.
- the decoding component 304 decodes the M encoded downmix signals 326 to generate M downmix signals 324.
- the M downmix signals are then, together with the downmix metadata, input to the renderer 310 which renders the M downmix signals to a multichannel output 330 corresponding to a legacy playback format (which typically has M channels).
- the renderer 310 may typically be similar to the renderer 210 of Fig. 2 , with the only difference that the renderer 310 now takes the M downmix signals 324 and the metadata 325 associated with the M downmix signals 324 as input instead of audio objects 220 and their associated metadata 222.
- the N audio objects 120 may correspond to a simplified representation of an audio scene.
- an audio scene may comprise audio objects and audio channels.
- an audio channel is here meant an audio signal which corresponds to a channel of a multichannel speaker configuration. Examples of such multichannel speaker configurations include a 22.2 configuration, a 11.1 configuration etc.
- An audio channel may be interpreted as a static audio object having a spatial position corresponding to the speaker position of the channel.
- the number of audio objects and audio channels in the audio scene may be vast, such as more than 100 audio objects and 1-24 audio channels. If all of these audio objects/channels are to be reconstructed on the decoder side, a lot of computational power is required. Furthermore, the resulting data rate associated with object metadata and side information will generally be very high if many objects are provided as input. For this reason it is advantageous to simplify the audio scene in order to reduce the number of audio objects to be reconstructed on the decoder side.
- the encoder may comprise a clustering component which reduces the number of audio objects in the audio scene based on a second clustering procedure. The second clustering procedure aims at exploiting the spatial redundancy present in the audio scene, such as audio objects having equal or very similar locations.
- Such a clustering component may be arranged in sequence or in parallel with the downmix component 102 of Fig. 1 .
- the sequential arrangement will be described with reference to Fig. 4 and the parallel arrangement will be described with reference to Fig. 5 .
- Fig. 4 illustrates an encoder 400.
- the encoder 400 comprises a clustering component 409.
- the clustering component 409 is arranged in sequence with the downmix component 102, meaning that the output of the clustering component 409 is input to the downmix component 102.
- the clustering component 409 takes audio objects 421 a and/or audio channels 421 b as input together with associated metadata 423 including spatial positions of the audio objects 421 a.
- the clustering component 409 converts the audio channels 421 b to static audio objects by associating each audio channel 421 b with the spatial position of the speaker position corresponding to the audio channel 421 b.
- the audio objects 421 a and the static audio objects formed from the audio channels 421 b may be seen as a first plurality of audio objects 421.
- the clustering component 409 generally reduces the first plurality of audio objects 421 to a second plurality of audio objects, here corresponding to the N audio objects 120 of Fig. 1 .
- the clustering component 409 may apply a second clustering procedure.
- the second clustering procedure is generally similar to the first clustering procedure described above with respect to the downmix component 102. The description of the first clustering procedure therefore also applies to the second clustering procedure.
- the second clustering procedure involves associating the first plurality of audio objects 121 with at least one cluster, here N clusters, based on spatial proximity of the first plurality of audio objects 121.
- the association with clusters may also be based on other properties of the audio objects as represented by the metadata 423.
- Each cluster is then represented by an object which is a (linear) combination of the audio objects associated with that cluster.
- the clustering component 409 further calculates metadata 122 for the so generated N audio objects 120.
- the metadata 122 includes spatial positions of the N audio objects 120.
- the spatial position of each of the N audio objects 120 may be calculated based on the spatial positions of the audio objects associated with the corresponding cluster.
- the spatial position may be calculated as a centroid or a weighted centroid of the spatial positions of the audio objects associated with the cluster as further explained above with reference to Fig. 1 .
- the N audio objects 120 generated by the clustering component 409 are then input to the downmix component 120 as further described with reference to Fig. 1 .
- Fig. 5 illustrates an encoder 500.
- the encoder 500 comprises a clustering component 509.
- the clustering component 509 is arranged in parallel with the downmix component 102, meaning that the downmix component 102 and the clustering component 509 have the same input.
- the input comprises a first plurality of audio objects, corresponding to the N audio objects 120 of Fig. 1 , together with associated metadata 122 including spatial positions of the first plurality of audio objects.
- the first plurality of audio objects 120 may, similar to the first plurality of audio objects 121 of Fig. 4 , comprise audio objects and audio channels being converted into static audio objects.
- the downmix component 102 of Fig. 5 operates on the full audio content of the audio scene in order to generate M downmix signals 124.
- the clustering component 509 is similar in functionality to the clustering component 409 described with reference to Fig. 4 .
- the clustering component 509 reduces the first plurality of audio objects 120 to a second plurality of audio objects 521, here illustrated by K audio objects where typically M ⁇ K ⁇ N (for high bit applications M ⁇ K ⁇ N), by applying the second clustering procedure described above.
- the second plurality of audio objects 521 is thus a set of audio objects formed on basis of the N audio objects 126.
- the clustering component 509 calculates metadata 522 for the second plurality of audio objects 521 (the K audio objects) including spatial positions of the second plurality of audio objects 521.
- the metadata 522 is included in the data stream 540 by the demultiplexing component 108.
- the analysis component 106 calculates side information 528 which enables reconstruction of second plurality of audio objects 521, i.e. the set of audio objects formed on basis of the N audio objects (here the K audio objects), from the M downmix signals 124.
- the side information 528 is included in the data stream 540 by the multiplexing component 108.
- the analysis component 106 may for example derive the side information 528 by analyzing the second plurality of audio objects 521 and the M downmix signals 124.
- the data stream 540 generated by the encoder 500 may generally be decoded by the decoder 200 of Fig. 2 or the decoder 300 of Fig. 3 .
- the reconstructed audio objects 220 of Fig. 2 now correspond to the second plurality of audio objects 521 (labeled K audio objects) of Fig. 5
- the metadata 222 associated with the audio objects now correspond to the metadata 522 of the second plurality of audio objects (labeled metadata of K audio objects) of Fig. 5 .
- side information or metadata associated with the objects is typically updated relatively infrequently (sparsely) in time to limit the associated data rate.
- Typical update intervals for object positions can range between 10 and 500 milliseconds, depending on the speed of the object, the required position accuracy, the available bandwidth to store or transmit metadata, etc.
- Such sparse, or even irregular metadata updates require interpolation of metadata and/or rendering matrices (i.e. matrices employed in rendering) for audio samples in-between two subsequent metadata instances. Without interpolation, the consequential step-wise changes in the rendering matrix may cause undesirable switching artifacts, clicking sounds, zipper noises, or other undesirable artifacts as a result of spectral splatter introduced by step-wise matrix updates.
- Fig. 6 illustrates a typical known process to compute rendering matrices for rendering of audio signals or audio objects, based on a set of metadata instances.
- a set of metadata instances (m1 to m4) 610 correspond to a set of points in time (t1 to t4) which are indicated by their position along the time axis 620.
- each metadata instance is converted to a respective rendering matrix (c1 to c4) 630, or rendering setting, which is valid at the same time point as the metadata instance.
- metadata instance m1 creates rendering matrix c1 at time t1
- metadata instance m2 creates rendering matrix c2 at time t2, and so on.
- the rendering matrices 630 generally comprise coefficients that represent gain values at different points in time. Metadata instances are defined at certain discrete points in time, and for audio samples in-between the metadata time points, the rendering matrix is interpolated, as indicated by the dashed line 640 connecting the rendering matrices 630. Such interpolation can be performed linearly, but also other interpolation methods can be used (such as band-limited interpolation, sine/cosine interpolation, and etc.).
- interpolation duration The time interval between the metadata instances (and corresponding rendering matrices) is referred to as an "interpolation duration," and such intervals may be uniform or they may be different, such as the longer interpolation duration between times t3 and t4 as compared to the interpolation duration between times t2 and t3.
- the calculation of rendering matrix coefficients from metadata instances is well-defined, but the reverse process of calculating metadata instances given a (interpolated) rendering matrix, is often difficult, or even impossible.
- the process of generating a rendering matrix from metadata can sometimes be regarded as a cryptographic one-way function.
- the process of calculating new metadata instances between existing metadata instances is referred to as "resampling" of the metadata. Resampling of metadata is often required during certain audio processing tasks. For example, when audio content is edited, by cutting/merging/mixing and so on, such edits may occur in between metadata instances. In this case, resampling of the metadata is required. Another such case is when audio and associated metadata are encoded with a frame-based audio codec.
- the metadata 122, 222 associated with the N audio objects 120, 220 and the metadata 522 associated with the K objects 522 originate, at least in some example embodiments, from clustering components 409 and 509, and may be referred to as cluster metadata.
- the metadata 125, 325 associated with the downmix signals 124, 324 may be referred to as downmix metadata.
- the downmix component 102 may calculate the M downmix signals 124 by forming combinations of the N audio objects 120 in a signal-adaptive manner, i.e. according to a criterion which is independent of any loudspeaker configuration. Such operation of the downmix component 102 is characteristic of example embodiments within a first aspect. According to example embodiments within other aspects, the downmix component 102 may e.g. calculate the M downmix signals 124 by forming combinations of the N audio objects 120 in a signal-adaptive manner, or, alternatively, such that the M downmix signals are suitable for playback on the channels of a speaker configuration with M channels, i.e. as a backwards compatible downmix.
- the encoder 400 described with reference to Fig. 4 employs a metadata and side information format particularly suitable for resampling, i.e. for generating additional metadata and side information instances.
- the analysis component 106 calculates the side information 128 in a form which includes a plurality of side information instances specifying respective desired reconstruction settings for reconstructing the N audio objects 120, and, for each side information instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current reconstruction setting to the desired reconstruction setting specified by the side information instance, and a point in time to complete the transition.
- the two independently assignable portions of the transition data for each side information instance are: a time stamp indicating the point in time to begin the transition to the desired reconstruction setting and an interpolation duration parameter indicating a duration for reaching the desired reconstruction setting from the point in time to begin the transition to the desired reconstruction setting.
- the interval during which a transition is to take place is in the present example embodiment uniquely defined by the time at which the transition is to begin and the duration of the transition interval.
- This particular form of the side information 128 will be described below with reference to Figs. 7-11 . It is to be understood that there are several other ways to uniquely define this transition interval.
- a reference point in the form of a start, end or middle point of the interval, accompanied by the duration of the interval may be employed in the transition data to uniquely define the interval.
- the start and end points of the interval may be employed in the transition data to uniquely define the interval.
- the clustering component 409 reduces the first plurality of audio objects 421 to a second plurality of audio objects, here corresponding to the N audio objects 120 of Fig. 1 .
- the clustering component 409 calculates the cluster metadata 122 for the generated N audio objects 120 which enables rendering of the N audio objects 122 in a renderer 210 at a decoder side.
- the clustering component 409 provides the cluster metadata 122 in a form which includes a plurality of cluster metadata instances specifying respective desired rendering settings for rendering the N audio objects 120, and, for each cluster metadata instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current rendering setting to the desired rendering setting specified by the cluster metadata instance, and a point in time to complete the transition to the desired rendering setting.
- the two independently assignable portions of the transition data for each cluster metadata instance are: a time stamp indicating the point in time to begin the transition to the desired rendering setting and an interpolation duration parameter indicating a duration for reaching the desired rendering setting from the point in time to begin the transition to the desired rendering setting.
- the downmix component 102 associates each downmix signal 124 with a spatial position and includes the spatial position in the downmix metadata 125 which allows rendering of the M downmix signals in a renderer 310 at a decoder side.
- the downmix component 102 provides the downmix metadata 125 in a form which includes a plurality of downmix metadata instances specifying respective desired downmix rendering settings for rendering the downmix signals, and, for each downmix metadata instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current downmix rendering setting to the desired downmix rendering setting specified by the downmix metadata instance, and a point in time to complete the transition to the desired downmix rendering setting.
- the two independently assignable portions of the transition data for each downmix metadata instance are: a time stamp indicating the point in time to begin the transition to the desired downmix rendering setting and an interpolation duration parameter indicating a duration for reaching the desired downmix rendering setting from the point in time to begin the transition to the desired downmix rendering setting.
- the same format is employed for the side information 128, the cluster metadata 122 and the downmix metadata 125.
- This format will now be described with reference to Figs. 7-11 in terms of metadata for rendering of audio signals.
- terms or expressions like “metadata for rendering of audio signals” may just as well be replaced by corresponding terms or expressions like "side information for reconstruction of audio objects", “cluster metadata for rendering of audio objects” or “downmix metadata for rendering of downmix signals”.
- Fig. 7 illustrates the derivation, based on metadata, of coefficient curves employed in rendering of audio signals, according to an example embodiment.
- a set of metadata instances m x generated at different points in time t x are converted by a converter 710 into corresponding sets of matrix coefficient values c x .
- These sets of coefficients represent gain values, also referred to as gain factors, to be employed for rendering of the audio signals to various speakers and drivers in a playback system to which the audio content is to be rendered.
- An interpolator 720 then interpolates the gain factors c x to produce a coefficient curve between the discrete times t x .
- the time stamps t x associated with each metadata instance m x may correspond to random points in time, synchronous points in time generated by a clock circuit, time events related to the audio content, such as frame boundaries, or any other appropriate timed event. Note that, as described above, the description provided with reference to Fig. 7 applies analogously to side information for reconstruction of audio objects.
- Fig. 8 illustrates a metadata format according to an embodiment (and as described above, the following description applies analogously to a corresponding side information format), which addresses at least some of the interpolation problems associated with present methods, as described above, by defining a time stamp as the start time of a transition or an interpolation, and augmenting each metadata instance with an interpolation duration parameter that represents the transition duration or interpolation duration (also referred to as "ramp size").
- a set of metadata instances m2 to m4 (810) specifies a set of rendering matrices c2 to c4 (830).
- Each metadata instance is generated at a particular point in time t x , and each metadata instance is defined with respect to its time stamp, m2 to t2, m3 to t3, and so on.
- the associated rendering matrices 830 are generated after performing transitions during respective interpolation durations d2, d3, d4 (830), from the associated time stamp (t1 to t4) of each metadata instance 810.
- the metadata essentially provides a schematic of how to proceed from a current rendering setting (e.g., the current rendering matrix resulting from previous metadata) to a new rendering setting (e.g., the new rendering matrix resulting from the current metadata).
- Each metadata instance is meant to take effect at a specified point in time in the future relative to the moment the metadata instance was received and the coefficient curve is derived from the previous state of the coefficient.
- m2 generates c2 after a duration d2
- m3 generates c3 after a duration d3
- m4 generates c4 after a duration d4.
- the previous metadata need not be known, only the previous rendering matrix or rendering state is required.
- the interpolation employed may be linear or non-linear depending on system constraints and configurations.
- Fig. 9 illustrates a first example of lossless processing of metadata, according to an example embodiment (and as described above, the following description applies analogously to a corresponding side information format).
- Fig. 9 shows metadata instances m2 to m4 that refer to the future rendering matrices c2 to c4, respectively, including interpolation durations d2 to d4.
- the time stamps of the metadata instances m2 to m4 are given as t2 to t4.
- a metadata instance m4a, at time t4a is added.
- time t4a may represent the time that an audio codec employed for coding audio content associated with the metadata starts a new frame.
- the metadata values of m4a are identical to those of m4 (i.e. they both describe a target rendering matrix c4), but the time d4a to reach that point has been reduced by d4-d4a.
- metadata instance m4a is identical to that of the previous metadata instance m4 so that the interpolation curve between c3 and c4 is not changed.
- the new interpolation duration d4a is shorter than the original duration d4. This effectively increases the data rate of the metadata instances, which can be beneficial in certain circumstances, such as error correction.
- FIG. 10 A second example of lossless metadata interpolation is shown in Fig. 10 (and as described above, the following description applies analogously to a corresponding side information format).
- the goal is to include a new set of metadata m3a in between two metadata instances m3 and m4.
- Fig. 10 illustrates a case where the rendering matrix remains unchanged for a period of time. Therefore, in this situation, the values of the new set of metadata m3a are identical to those of the prior metadata m3, except for the interpolation duration d3a.
- the value of the interpolation duration d3a should be set to the value corresponding to t4-t3a, i.e.
- Fig. 10 may for example occur when an audio object is static and an authoring tool stops sending new metadata for the object due to this static nature. In such a case, it may be desirable to insert new metadata instances m3a, e.g. to synchronize the metadata with codec frames.
- FIG. 8 illustrates an interpolation scheme using a sample-and-hold circuit with a low-pass filter, according to an example embodiment (and as described above, the following description applies analogously to a corresponding side information format).
- the metadata instances m2 to m4 are converted to sample-and-hold rendering matrix coefficients c2 and c3.
- the sample-and-hold process causes the coefficient states to jump immediately to the desired state, which results in a step-wise curve 1110, as shown.
- This curve 1110 is then subsequently low-pass filtered to obtain a smooth, interpolated curve 1120.
- the interpolation filter parameters e.g., cut-off frequency or time constant
- the interpolation duration or ramp size can have any practical value, including a value of or substantially close to zero.
- Such small interpolation duration is especially helpful for cases such as initialization in order to enable setting the rendering matrix immediately at the first sample of a file, or allowing for edits, splicing, or concatenation of streams.
- having the possibility to instantaneously change the rendering matrix can be beneficial to maintain the spatial properties of the content after editing.
- the interpolation scheme described herein is compatible with the removal of metadata instances (and analogously with the removal of side information instances, as described above), such as in a decimation scheme that reduces metadata bitrates.
- Removal of metadata instances allows the system to resample at a frame rate that is lower than an initial frame rate.
- metadata instances and their associated interpolation duration data that are provided by an encoder may be removed based on certain characteristics. For example, an analysis component in an encoder may analyze the audio signal to determine if there is a period of significant stasis of the signal, and in such a case remove certain metadata instances already generated to reduce bandwidth requirements for the transmittal of data to a decoder side.
- the removal of metadata instances may alternatively or additionally be performed in a component separate from the encoder, such as in a decoder or in a transcoder.
- a transcoder may remove metadata instances that have been generated or added by the encoder, and may be employed in a data rate converter that re-samples an audio signal from a first rate to a second rate, where the second rate may or may not be an integer multiple of the first rate.
- the encoder, decoder or transcoder may analyze the metadata. For example, with reference to Fig.
- a difference may be computed between a first desired reconstruction setting c3 (or reconstruction matrix), specified by a first metadata instance m3, and desired reconstruction settings c3a and c4 (or reconstruction matrices) specified by metadata instances m3a and m4 directly succeeding the first metadata instance m3.
- the difference may for example be computed by employing a matrix norm to the respective rendering matrices. If the difference is below a predefined threshold, e.g. corresponding to a tolerated distortion of the reconstructed audio signals, the metadata instances m3a and m4 succeeding the first metadata instance m2 may be removed.
- a predefined threshold e.g. corresponding to a tolerated distortion of the reconstructed audio signals
- the object reconstruction component 206 may employ interpolation as part of reconstructing the N audio objects 220 based on the M downmix signals 224 and the side information 228.
- reconstructing the N audio objects 220 may for example include: performing reconstruction according to a current reconstruction setting; beginning, at a point in time defined by the transition data for a side information instance, a transition from the current reconstruction setting to a desired reconstruction setting specified by the side information instance; and completing the transition to the desired reconstruction setting at a point in time defined by the transition data for the side information instance.
- the renderer 210 may employ interpolation as part of rendering the reconstructed N audio objects 220 in order to generate the multichannel output signal 230 suitable for playback.
- the rendering may include: performing rendering according to a current rendering setting; beginning, at a point in time defined by the transition data for a cluster metadata instance, a transition from the current rendering setting to a desired rendering setting specified by the cluster metadata instance; and completing the transition to the desired rendering setting at a point in time defined by the transition data for the cluster metadata instance.
- the object reconstruction section 206 and the renderer 210 may be separate units, and/or may correspond to operations performed as separate processes.
- the object reconstruction section 206 and the renderer 210 may be embodied as a single unit or process in which reconstruction and rendering is performed as a combined operation.
- matrices employed for reconstruction and rendering may be combined into a single matrix which may be interpolated, instead of performing interpolation on a rendering matrix and a reconstruction matrix, separately.
- the renderer 310 may perform interpolation as part of rendering the M downmix signals 324 to the multichannel output 330.
- the rendering may include: performing rendering according to a current downmix rendering setting; beginning, at a point in time defined by the transition data for a downmix metadata instance, a transition from the current downmix rendering setting to a desired downmix rendering setting specified by the downmix metadata instance; and completing the transition to the desired downmix rendering setting at a point in time defined by the transition data for the downmix metadata instance.
- the renderer 310 may be comprised in the decoder 300 or may be a separate device/unit.
- the decoder may output the downmix metadata 325 and the M downmix signals 324 for rendering of the M downmix signals in the renderer 310.
- the systems and methods disclosed hereinabove may be implemented as software, firmware, hardware or a combination thereof.
- the division of tasks between functional units referred to in the above description does not necessarily correspond to the division into physical units; to the contrary, one physical component may have multiple functionalities, and one task may be carried out by several physical components in cooperation.
- Certain components or all components may be implemented as software executed by a digital signal processor or microprocessor, or be implemented as hardware or as an application-specific integrated circuit.
- Such software may be distributed on computer readable media, which may comprise computer storage media (or non-transitory media) and communication media (or transitory media).
- Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
- Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.
- communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Mathematical Physics (AREA)
- Stereophonic System (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Description
- The present application claims the benefit of the filing date of
U.S. Provisional Patent Application No: 61/827,246 filed on May 24 2013 U.S. Provisional Patent Application No: 61/893,770 filed on October 21, 2013 U.S. Provisional Patent Application No: 61/973,623 filed on April 1, 2014 - The disclosure herein generally relates to coding of an audio scene comprising audio objects. In particular, it relates to an encoder, a decoder and associated methods for encoding and decoding of audio objects.
- An audio scene may generally comprise audio objects and audio channels. An audio object is an audio signal which has an associated spatial position which may vary with time. An audio channel is an audio signal which corresponds directly to a channel of a multichannel speaker configuration, such as a so-called 5.1 speaker configuration with three front speakers, two surround speakers, and a low frequency effects speaker.
- Since the number of audio objects typically may be very large, for instance in the order of hundreds of audio objects, there is a need for coding methods which allow the audio objects to be efficiently reconstructed at the decoder side. There have been suggestions to combine the audio objects into a multichannel downmix (i.e. into a plurality of audio channels which corresponds to the channels of a certain multichannel speaker configuration such as a 5.1 configuration) on an encoder side, and to reconstruct the audio objects parametrically from the multichannel downmix on a decoder side.
- An advantage of such an approach is that a legacy decoder which does not support audio object reconstruction may use the multichannel downmix directly for playback on the multichannel speaker configuration. By way of example, a 5.1 downmix may directly be played on the loudspeakers of a 5.1 configuration.
- A disadvantage with this approach is however that the multichannel downmix may not give a sufficiently good reconstruction of the audio objects at the decoder side. For example, consider two audio objects that have the same horizontal position as the left front speaker of a 5.1. configuration but a different vertical position. These audio objects would typically be combined into the same channel of a 5.1 downmix. This would constitute a challenging situation for the audio object reconstruction at the decoder side which would have to reconstruct approximations of the two audio objects from the same downmix channel, a process that cannot ensure perfect reconstruction and that sometimes even lead to audible artifacts.
- There is thus a need for encoding/decoding methods which provide an efficient and improved reconstruction of audio objects.
- Side information or metadata is often employed during reconstruction of audio objects from e.g. a downmix. The form and content of such side information may for example affect the fidelity of the reconstructed audio objects and/or the computational complexity of performing the reconstruction. It would therefore be desirable to provide encoding/decoding methods with a new and alternative side information format which allows for increasing the fidelity of reconstructed audio objects, and/or which allows for reducing the computational complexity of the reconstruction.
- The International Search Report pertaining to the present document cites, inter-alia, the following two documents:
- International (
PCT) Patent Application Publication No. WO 2010/125104 A1 - International (
PCT) Patent Application Publication No. WO 2014/015299 A1 - Example embodiments will now be described with reference to the accompanying drawings, on which:
-
Fig. 1 is a schematic illustration of an encoder according to exemplary embodiments; -
Fig. 2 is a schematic illustration of a decoder which supports reconstruction of audio objects according to exemplary embodiments; -
Fig. 3 is a schematic illustration of a low-complexity decoder which does not support reconstruction of audio objects according to exemplary embodiments; -
Fig. 4 is a schematic illustration of an encoder which comprises a sequentially arranged clustering component for simplification of an audio scene according to exemplary embodiments; -
Fig. 5 is a schematic illustration of an encoder which comprises a clustering component arranged in parallel for simplification of an audio scene according to exemplary embodiments; -
Fig. 6 illustrates a typical known process to compute a rendering matrix for a set of metadata instances; -
Fig. 7 illustrates the derivation of a coefficient curve employed in rendering of audio signals; -
Fig. 8 illustrates a metadata instance interpolation method, according to an example embodiment; -
Figs. 9 and 10 illustrate examples of introduction of additional metadata instances, according to example embodiments; and -
Fig. 11 illustrates an interpolation method using a sample-and-hold circuit with a low-pass filter, according to an example embodiment. - All the figures are schematic and generally only show parts which are necessary in order to elucidate the disclosure, whereas other parts may be omitted or merely suggested. Unless otherwise indicated, like reference numerals refer to like parts in different figures.
- In view of the above it is thus an object to provide encoding and decoding methods, a computer program product, and a decoder, which allow for efficient and improved reconstruction of audio objects, and/or which allows for increasing the fidelity of reconstructed audio objects, and/or which allows for reducing the computational complexity of the reconstruction.
- The object is solved by the independent claims. Specific embodiments are defined in the dependent claims.
- According to a first aspect, there is provided an encoding method, an encoder and a computer program product for encoding audio objects, as recited in claims.
- According to exemplary embodiments there is provided a method for encoding audio objects into a data stream, comprising inter alia:
- receiving N audio objects, wherein N>1;
- calculating M downmix signals, wherein M≤N, by forming combinations of the N audio objects according to a criterion which is independent of any loudspeaker configuration;
- calculating side information including parameters which allow reconstruction of a set of audio objects formed on basis of the N audio objects from the M downmix signals; and
- including the M downmix signals and the side information in a data stream for transmittal to a decoder.
- With the above arrangement, the M downmix signals are thus formed from the N audio objects independently of any loudspeaker configuration. This implies that the M downmix signals are not constrained to audio signals which are suitable for playback on the channels of a speaker configuration with M channels. Instead, the M downmix signals may be selected more freely according to a criterion such that they for instance adapt to the dynamics of the N audio objects and improve the reconstruction of the audio objects at the decoder side.
- Returning to the example with two audio objects that have the same horizontal position as the left front speaker of a 5.1 configuration but a different vertical position, the proposed method allows to put the first audio object in a first downmix signal, and the second audio object in the second downmix signal. This enables perfect reconstruction of the audio objects in the decoder. In general, such perfect reconstruction is possible as long as the number of active audio objects does not exceed the number of downmix signals. If the number of active audio objects is higher, then the proposed method allows for selection of the audio objects that have to be mixed into the same downmix signal such that the possible approximation errors occurring in the reconstructed audio object in the decoder have no or the smallest possible perceptual impact on the reconstructed audio scene.
- A second advantage of the M downmix signals being adaptive is the ability to keep certain audio objects strictly separate from other audio objects. For example, it can be advantageous to keep any dialog object separate from background objects, to ensure that dialog is rendered accurately in terms of spatial attributes, and allows for object processing in the decoder, such as dialog enhancement or increase of dialog loudness for improved intelligibility. In other applications (e.g. karaoke), it may be advantageous to allow complete muting of one or more objects, which also requires that such objects are not mixed with other objects. Conventional methods using a multichannel downmix corresponding to a specific speaker configuration do not allow for complete muting of audio objects present in a mix of other audio objects.
- The word downmix signal reflects that a downmix signal is a mix, i.e. a combination, of other signals. The word "down" indicates that the number M of downmix signals typically is lower than the number N of audio objects.
- According to exemplary embodiments, the method may further comprise associating each downmix signal with a spatial position and including the spatial positions of the downmix signals in the data stream as metadata for the downmix signals. This is advantageous in that it allows for low-complexity decoding to be used in case of a legacy playback system. More precisely, the metadata associated with the downmix signals may be used on a decoder side for rendering the downmix signals to the channels of a legacy playback system.
- According to exemplary embodiments, the N audio objects are associated with metadata including spatial positions of the N audio objects, and the spatial positions associated with the downmix signals are calculated based on the spatial positions of the N audio objects. Thus, the downmix signals may be interpreted as audio objects having a spatial position which depends on the spatial positions of the N audio objects.
- Further, the spatial positions of the N audio objects and the spatial positions associated with the M downmix signals may be time-varying, i.e. they may vary between time frames of audio data. In other words, the downmix signals may be interpreted as dynamic audio objects having an associated position which varies between time frames. This is in contrast to prior art systems where the downmix signals correspond to fixed spatial loudspeaker positions.
- Typically, the side information is also time-varying thereby allowing the parameters governing the reconstruction of the audio objects to vary temporally.
- The encoder may apply different criteria for the calculation of the downmix signals. According to exemplary embodiments in which the N audio objects are associated with metadata including spatial positions of the N audio objects, the criterion for calculating the M downmix signals may be based on spatial proximity of the N audio objects. For example, audio objects which are close to each other may be combined into the same downmix signal.
- According to exemplary embodiments in which the metadata associated with N audio objects further comprises importance values indicating the importance of the N audio objects in relation to each other, the criterion for calculating the M downmix signals may further be based on the importance values of the N audio objects. For example, the most important one(s) of the N audio objects may be mapped directly to a downmix signal, while the remaining audio objects are combined to form the remaining downmix signals.
- In particular, according to exemplary embodiments, the step of calculating M downmix signals comprises a first clustering procedure which includes associating the N audio objects with M clusters based on spatial proximity and importance values, if applicable, of the N audio objects, and calculating a downmix signal for each cluster by forming a combination of audio objects associated with the cluster. In some cases an audio object may form part of at most one cluster. In other cases, an audio object may form part of several clusters. In this way, different groups, i.e. clusters, are formed from the audio objects. Each cluster may in turn be represented by a downmix signal which may be thought of as an audio object. The clustering approach allows associating each downmix signal with a spatial position which is calculated based on the spatial positions of the audio objects associated with the cluster corresponding to the downmix signal. With this interpretation the first clustering procedure thus reduces the dimensionality of the N audio objects to M audio objects in a flexible manner.
- The spatial position associated with each downmix signal may for example be calculated as a centroid or a weighted centroid of the spatial positions of the audio objects associated with the cluster corresponding to the downmix signal. The weights may for example be based on importance values of the audio objects.
- According to exemplary embodiments, the N audio objects are associated with the M clusters by applying a K-means algorithm having the spatial positions of the N audio objects as input.
- Since an audio scene may comprise a vast number of audio objects, the method may take further measures for reducing the dimensionality of the audio scene, thereby reducing the computational complexity at the decoder side when reconstructing the audio objects. In particular, the method may further comprise a second clustering procedure for reducing a first plurality of audio objects to a second plurality of audio objects.
- According to one embodiment, the second clustering procedure is performed prior to the calculation of the M downmix signals. In that embodiment the first plurality of audio objects hence correspond to the original audio objects of the audio scene, and the second, reduced, plurality of audio objects corresponds to the N audio objects on the basis of which the M downmix signals are calculated. Moreover, in such embodiment, the set of audio objects (to be reconstructed in the decoder) formed on basis of the N audio objects corresponds, i.e. is equal to, to the N audio objects.
- According to another embodiment, the second clustering procedure is performed in parallel with the calculation of the M downmix signals. In such embodiment, the N audio objects on the basis of which the M downmix signals are calculated as well as the first plurality of audio objects being input to the second clustering procedure correspond to the original audio objects of the audio scene. Moreover, in such embodiment, the set of audio objects (to be reconstructed in the decoder) formed on basis of the N audio objects corresponds to the second plurality of audio objects. With this approach, the M downmix signals are hence calculated on basis on the original audio objects of the audio scene and not on basis of a reduced number of audio objects.
- According to exemplary embodiments, the second clustering procedure comprises:
- receiving the first plurality of audio objects and their associated spatial positions,
- associating the first plurality of audio objects with at least one cluster based on spatial proximity of the first plurality of audio objects,
- generating the second plurality of audio objects by representing each of the at least one cluster by an audio object being a combination of the audio objects associated with the cluster,
- calculating metadata including spatial positions for the second plurality of audio objects, wherein the spatial position of each audio object of the second plurality of audio objects is calculated based on the spatial positions of the audio objects associated with the corresponding cluster; and
- including the metadata for the second plurality of audio objects in the data stream.
- In other words, the second clustering procedure exploits spatial redundancy present in the audio scene, such as objects having equal or very similar locations. In addition, importance values of the audio objects may be taken into account when generating the second plurality of audio objects.
- As mentioned above, the audio scene may also comprise audio channels. Such audio channels may be thought of as an audio object being associated with a static position, viz. the position of the loudspeaker corresponding to the audio channel. In more detail, the second clustering procedure may further comprise:
- receiving at least one audio channel;
- converting each of the at least one audio channel to an audio object having a static spatial position corresponding to a loudspeaker position of that audio channel; and
- including the converted at least one audio channel in the first plurality of audio objects.
- In this way, the method allows for encoding of an audio scene comprising audio channels as well as audio objects.
- According to exemplary embodiments, there is provided a computer program product comprising a computer-readable medium with instructions for performing the decoding method according to exemplary embodiments.
- According to exemplary embodiments, there is provided an encoder for encoding audio objects into a data stream, comprising:
- a receiving component configured to receive N audio objects, wherein N>1
- a downmix component configured to calculate M downmix signals, wherein M≤N, by forming combinations of the N audio objects according to a criterion which is independent of any loudspeaker configuration;
- an analysis component configured to calculate side information including parameters which allow reconstruction of the set of audio objects formed on basis of the N audio objects from the M downmix signals; and
- a multiplexing component configured to include the M downmix signals and the side information in a data stream for transmittal to a decoder.
- According to a second aspect, there is provided a decoding method, a decoder and a computer program product for decoding multichannel audio content, as recited in the claims.
- The second aspect may generally have the same features and advantages as the first aspect.
- According to exemplary embodiments there is provided a method in a decoder for decoding a data stream including encoded audio objects, comprising inter alia:
- receiving a data stream comprising M downmix signals which are combinations of N audio objects calculated according to a criterion which is independent of any loudspeaker configuration, wherein M≤N, and side information including parameters which allow reconstruction of a set of audio objects formed on basis of the N audio objects from the M downmix signals; and
- reconstructing the set of audio objects formed on basis of the N audio objects from the M downmix signals and the side information.
- According to exemplary embodiments, the data stream further comprises metadata for the M downmix signals including spatial positions associated with the M downmix signals, the method further comprising:
- on a condition that the decoder is configured to support audio object reconstruction, performing the step of reconstructing the set of audio objects formed on basis N audio objects from the M downmix signals and the side information; and
- on a condition that the decoder is not configured to support audio object reconstruction, using the metadata for the M downmix signals for rendering of the M downmix signals to output channels of a playback system.
- According to exemplary embodiments, the spatial positions associated with the M downmix signals are time-varying.
- According to exemplary embodiments, the side information is time-varying.
- According to exemplary embodiments, the data stream further comprises metadata for the set of audio objects formed on basis of the N audio objects including the spatial positions of the set of audio objects formed on basis of the N audio objects, the method further comprising:
- using the metadata for the set of audio objects formed on basis of the N audio objects for rendering of the reconstructed set of audio objects formed on basis of the N audio objects to output channels of a playback system.
- According to exemplary embodiments, the set of audio objects formed on basis of the N audio objects is equal to the N audio objects.
- According to exemplary embodiments, the set of audio objects formed on basis of the N audio objects comprises a plurality of audio objects which are combinations of the N audio objects, and the number of which is lower than N.
- According to exemplary embodiments, there is provided a computer program product comprising a computer-readable medium with instructions for performing the decoding method according to exemplary embodiments.
- According to exemplary embodiments, there is provided a decoder for decoding a data stream including encoded audio objects, comprising:
- a receiving component configured to receive a data stream comprising M downmix signals which are combinations of N audio objects calculated according to a criterion which is independent of any loudspeaker configuration, wherein M≤N, and side information including parameters which allow reconstruction of a set of audio objects formed on basis of the N audio objects from the M downmix signals; and
- a reconstructing component configured to reconstruct the set of audio objects formed on basis of the N audio objects from the M downmix signals and the side information.
- According to a third aspect, there is provided an encoding method, an encoder, and a computer program product for encoding audio objects.
- The methods, encoders and computer program products according to the third aspect may generally have features and advantages in common with the methods, encoders and computer program products according to the first aspect.
- According to example embodiments, there is provided a method for encoding audio objects as a data stream. The method comprises inter alia:
- receiving N audio objects, wherein N>1;
- calculating M downmix signals, wherein M≤N, by forming combinations of the N audio objects;
- calculating time-variable side information including parameters which allow reconstruction of a set of audio objects formed on the basis of the N audio objects from the M downmix signals; and
- including the M downmix signals and the side information in a data stream for transmittal to a decoder.
- In the present example embodiments, the method further comprises including, in the data stream:
- a plurality of side information instances specifying respective desired reconstruction settings for reconstructing the set of audio objects formed on the basis of the N audio objects; and
- for each side information instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current reconstruction setting to the desired reconstruction setting specified by the side information instance, and a point in time to complete the transition.
- In the present example embodiment, the side information is time-variable, e.g. time-varying, allowing for the parameters governing the reconstruction of the audio objects to vary with respect to time, which is reflected by the presence of the side information instances. By employing a side information format which includes transition data defining points in time to begin and points in time to complete transitions from current reconstruction settings to respective desired reconstruction settings, the side information instances are made more independent of each other in the sense that interpolation may be performed based on a current reconstruction setting and a single desired reconstruction setting specified by a single side information instance, i.e. without knowledge of any other side information instances. The provided side information format therefore facilitates calculation/introduction of additional side information instances between existing side information instances. In particular, the provided side information format allows for calculation/introduction of additional side information instances without affecting the playback quality. In this disclosure, the process of calculating/introducing new side information instances between existing side information instances is referred to as "resampling" of the side information. Resampling of side information is often required during certain audio processing tasks. For example, when audio content is edited, by e.g. cutting/merging/mixing, such edits may occur in between side information instances. In this case, resampling of the side information may be required. Another such case is when audio signals and associated side information are encoded with a frame-based audio codec. In this case, it is desirable to have at least one side information instance for each audio codec frame, preferably with a time stamp at the start of that codec frame, to improve resilience of frame losses during transmission. For example, the audio signals/objects may be part of an audio-visual signal or multimedia signal which includes video content. In such applications, it may be desirable to modify the frame rate of the audio content to match a frame rate of the video content, whereby a corresponding resampling of side information may be desirable.
- The data stream in which the downmix signal and the side information is included may for example be a bitstream, in particular a stored or transmitted bitstream.
- It is to be understood that calculating the M downmix signals by forming combinations of the N audio objects means that each of the M downmix signals is obtained by forming a combination, e.g. a linear combination, of the audio content of one or more of the N audio objects. In other words, each of the N audio objects need not necessarily contribute to each of the M downmix signals.
- The word downmix signal reflects that a downmix signal is a mix, i.e. a combination, of other signals. The downmix signal may for example be an additive mix of other signals. The word "down" indicates that the number M of downmix signals typically is lower than the number N of audio objects.
- The downmix signals may for example be calculated by forming combinations of the N audio signals according to a criterion which is independent of any loudspeaker configuration, according to any of the example embodiments within the first aspect. Alternatively, the downmix signals may for example be calculated by forming combinations of the N audio signals such that the downmix signals are suitable for playback on the channels of a speaker configuration with M channels, referred to herein as a backwards compatible downmix.
- By the transition data including two independently assignable portions is meant that the two portions are mutually independently assignable, i.e. may be assigned independently of each other. However, it is to be understood that the portions of the transition data may for example coincide with portions of transition data for other types of side information of metadata.
- In the present example embodiment, the two independently assignable portions of the transition data, in combination, define the point in time to begin the transition and the point in time to complete the transition, i.e. these two points in time are derivable from the two independently assignable portions of the transition data.
- According to an example embodiment, the method may further comprise a clustering procedure for reducing a first plurality of audio objects to a second plurality of audio objects, wherein the N audio objects constitute either the first plurality of audio objects or the second plurality of audio objects, and wherein the set of audio objects formed on the basis of the N audio objects coincides with the second plurality of audio objects. In the present example embodiment, the clustering procedure may comprise:
- calculating time-variable cluster metadata including spatial positions for the second plurality of audio objects; and
- further including, in the data stream, for transmittal to the decoder:
- a plurality of cluster metadata instances specifying respective desired rendering settings for rendering the second set of audio objects; and
- for each cluster metadata instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current rendering setting to the desired rendering setting specified by the cluster metadata instance, and a point in time to complete the transition to the desired rendering setting specified by the cluster metadata instance.
- Since an audio scene may comprise a vast number of audio objects, the method according to the present example embodiment, takes further measures for reducing the dimensionality of the audio scene by reducing the first plurality of audio objects to a second plurality of audio objects. In the present example embodiment, the set of audio objects, which is formed on the basis of the N audio objects and which is to be reconstructed on a decoder side based on the downmix signals and the side information, coincides with the second plurality of audio objects, which corresponds to a simplification and/or lower-dimensional representation of the audio scene represented by the first plurality of audio signals, and the computational complexity for reconstruction on a decoder side is reduced.
- The inclusion of the cluster metadata in the data stream allows for rendering of the second set of audio signals on a decoder side, e.g. after the second set of audio signals has been reconstructed based on the downmix signals and the side information.
- Similar to the side information, the cluster metadata in the present example embodiment is time-variable, e.g. time-varying, allowing for the parameters governing the rendering of the second plurality of audio objects to vary with respect to time. The format for the downmix metadata may be analogous to that of the side formation and may have the same or corresponding advantages. In particular, the form of the cluster metadata provided in the present example embodiment, facilitates resampling of the cluster metadata. Resampling of the cluster metadata may e.g. be employed to provide common points in time to start and complete respective transitions associated with the cluster metadata and the side information, and/or for adjusting the cluster metadata to a frame rate of the associated audio signals.
- According to an example embodiment, the clustering procedure may further comprise:
- receiving the first plurality of audio objects and their associated spatial positions;
- associating the first plurality of audio objects with at least one cluster based on spatial proximity of the first plurality of audio objects;
- generating the second plurality of audio objects by representing each of the at least one cluster by an audio object being a combination of the audio objects associated with the cluster; and
- calculating the spatial position of each audio object of the second plurality of audio objects based on the spatial positions of the audio objects associated with the respective cluster, i.e. with the cluster which the audio object represent.
- In other words, the clustering procedure exploits spatial redundancy present in the audio scene, such as objects having equal or very similar locations. In addition, importance values of the audio objects may be taken into account when generating the second plurality of audio objects, as described with respect to example embodiments within the first aspect.
- Associating the first plurality of audio objects with at least one cluster includes associating each of the first plurality of audio objects with one or more of the at least one cluster. In some cases, an audio object may form part of at most one cluster, while in other cases, an audio object may form part of several clusters. In other words, in some cases, an audio object may be split between several clusters as part of the clustering procedure.
- Spatial proximity of the first plurality of audio objects may be related to distances between, and/or relative positions of, the respective audio objects in the first plurality of audio objects. For example, audio objects which are close to each other may be associated with the same cluster.
- By an audio object being a combination of the audio objects associated with the cluster is meant that the audio content/signal associated with the audio object may be formed as a combination of the audio contents/signals associated with the respective audio objects associated with the cluster.
- According to an example embodiment, the respective points in time defined by the transition data for the respective cluster metadata instances may coincide with the respective points in time defined by the transition data for corresponding side information instances.
- By employing the same points in time to begin and to complete transitions associated with the side information and the cluster metadata, joint processing of the side information and the cluster metadata, such as joint resampling, is facilitated.
- Moreover, the use of common points in time to begin and to complete transitions associated with the side information and the cluster metadata facilitates joint reconstruction and rendering at a decoder side. If for example, reconstruction and rendering is performed as a joint operation on a decoder side, joint settings for reconstruction and rendering may be determined for each side information instance and metadata instance and/or interpolation between joint settings for reconstruction and rendering may be employed instead of performing interpolation separately for the respective settings. Such joint interpolation may reduce computational complexity at the decoder side as fewer coefficients/parameters need to be interpolated.
- According to an example embodiment, the clustering procedure may be performed prior to the calculation of the M downmix signals. In the present example embodiment, the first plurality of audio objects corresponds to the original audio objects of the audio scene, and the N audio objects on the basis of which the M downmix signals are calculated constitute the second, reduced, plurality of audio objects. Hence, in the present example embodiment, the set of audio objects (to be reconstructed on a decoder side) formed on the basis of the N audio objects coincides with the N audio objects.
- Alternatively, the clustering procedure may be performed in parallel with the calculation of the M downmix signals. According to the present alternative, the N audio objects on the basis of which the M downmix signals are calculated constitute the first plurality of audio objects which correspond to the original audio objects of the audio scene. With this approach, the M downmix signals are hence calculated on basis of the original audio objects of the audio scene and not on basis of a reduced number of audio objects.
- According to an example embodiment, the method may further comprise:
- associating each downmix signal with a time-variable spatial position for rendering the downmix signals, and
- further including, in the data stream, downmix metadata including the spatial positions of the downmix signals,
- wherein the method further comprises including, in the data stream:
- a plurality of downmix metadata instances specifying respective desired downmix rendering settings for rendering the downmix signals; and
- for each downmix metadata instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current downmix rendering setting to the desired downmix rendering setting specified by the downmix metadata instance, and a point in time to complete the transition to the desired downmix rendering setting specified by the downmix metadata instance.
- Including downmix metadata in the data stream is advantageous in that it allows for low-complexity decoding to be used in case of legacy playback equipment. More precisely, the downmix metadata may be used on a decoder side for rendering the downmix signals to the channels of a legacy playback system, i.e. without reconstructing the plurality of audio objects formed on the basis of the N objects, which typically is a computationally more complex operation.
- According to the present example embodiment, the spatial positions associated with the M downmix signals may be time-variable, e.g. time- varying, and the downmix signals may be interpreted as dynamic audio objects having an associated position which may change between time frames or downmix metadata instances. This is in contrast to prior art systems where the downmix signals correspond to fixed spatial loudspeaker positions. It is recalled that the same data stream may be played in an object oriented fashion in a decoding system with more evolved capabilities.
- In some example embodiments, the N audio objects may be associated with metadata including spatial positions of the N audio objects, and the spatial positions associated with the downmix signals may for example be calculated based on the spatial positions of the N audio objects. Thus, the downmix signals may be interpreted as audio objects having spatial positions which depend on the spatial positions of the N audio objects.
- According to an example embodiment, the respective points in time defined by the transition data for the respective downmix metadata instances may coincide with the respective points in time defined by the transition data for corresponding side information instances. Employing the same points in time for beginning and for completing transitions associated with the side information and the downmix metadata facilitates joint processing, e.g. resampling, of the side information and the downmix metadata.
- According to an example embodiment, the respective points in time defined by the transition data for the respective downmix metadata instances may coincide with the respective points in time defined by the transition data for corresponding cluster metadata instances. Employing the same points in time for beginning and ending transitions associated with the cluster metadata and the downmix metadata facilitates joint processing, e.g. resampling, of the cluster metadata and the downmix metadata.
- According to example embodiments, there is provided an encoder for encoding N audio objects as a data stream, wherein N>1. The encoder comprises:
- a downmix component configured to calculate M downmix signals, wherein M≤N, by forming combinations of the N audio objects;
- an analysis component configured to calculate time-variable side information including parameters which allow reconstruction of a set of audio objects formed on the basis of the N audio objects from the M downmix signals; and
- a multiplexing component configured to include the M downmix signals and the side information in a data stream for transmittal to a decoder,
- wherein the multiplexing component is further configured to include, in the data stream, for transmittal to the decoder:
- a plurality of side information instances specifying respective desired reconstruction settings for reconstructing the set of audio objects formed on the basis of the N audio objects; and
- for each side information instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current reconstruction setting to the desired reconstruction setting specified by the side information instance, and a point in time to complete the transition.
- According to a fourth aspect, there is provided a decoding method, a decoder, and a computer program product for decoding multichannel audio content.
- The methods, decoders and computer program products according to the fourth aspect are intended for cooperation with the methods, encoders and computer program products according to the third aspect, and may have corresponding features and advantages.
- The methods, decoders and computer program products according to the fourth aspect, may generally have features and advantages in common with the methods, decoders and computer program products according to the second aspect.
- According to example embodiments, there is provided a method for reconstructing audio objects based on a data stream. The method comprises:
- receiving a data stream comprising M downmix signals which are combinations of N audio objects, wherein N>1 and M≤N, and time-variable side information including parameters which allow reconstruction of a set of audio objects formed on the basis of the N audio objects from the M downmix signals; and
- reconstructing, based on the M downmix signals and the side information, the set of audio objects formed on the basis of the N audio objects,
- wherein the data stream comprises a plurality of side information instances, wherein the data stream further comprises, for each side information instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current reconstruction setting to a desired reconstruction setting specified by the side information instance, and a point in time to complete the transition, and wherein reconstructing the set of audio objects formed on the basis of the N audio objects comprises:
- performing reconstruction according to a current reconstruction setting;
- beginning, at a point in time defined by the transition data for a side information instance, a transition from the current reconstruction setting to a desired reconstruction setting specified by the side information instance; and
- completing the transition at a point in time defined by the transition data for the side information instance.
- As described above, employing a side information format which includes transition data defining points in time to begin and points in time to complete transitions from current reconstruction settings to respective desired reconstruction settings e.g. facilitates resampling of the side information.
- The data stream may for example be received in the form of a bitstream, e.g. generated on an encoder side.
- Reconstructing, based on the M downmix signals and the side information, the set of audio objects formed on the basis of the N audio objects, may for example include forming at least one linear combination of the downmix signals employing coefficients determined based on the side information. Reconstructing, based on the M downmix signals and the side information, the set of audio objects formed on the basis of the N audio objects, may for example include forming linear combinations of the downmix signals, and, optionally one or more additional (e.g. decorrelated) signal derived from the downmix signals, employing coefficients determined based on the side information.
- According to an example embodiment, the data stream may further comprise time-variable cluster metadata for the set of audio objects formed on the basis of the N audio objects, the cluster metadata including spatial positions for the set of audio objects formed on the basis of the N audio objects. The data stream may comprise a plurality of cluster metadata instances, and the data stream may further comprise, for each cluster metadata instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current rendering setting to a desired rendering setting specified by the cluster metadata instance, and a point in time to complete the transition to the desired rendering setting specified by the cluster metadata instance. The method may further comprise:
- using the cluster metadata for rendering of the reconstructed set of audio objects formed on the basis of the N audio objects to output channels of a predefined channel configuration, the rendering comprising:
- performing rendering according to a current rendering setting;
- beginning, at a point in time defined by the transition data for a cluster metadata instance, a transition from the current rendering setting to a desired rendering setting specified by the cluster metadata instance; and
- completing the transition to the desired rendering setting at a point in time defined by the transition data for the cluster metadata instance.
- The predefined channel configuration may for example correspond to a configuration of the output channels compatible with a particular playback system, i.e. suitable for playback on a particular playback system.
- Rendering of the reconstructed set of audio objects formed on the basis of the N audio objects to output channels of a predefined channel configuration may for example include mapping, in a renderer, the reconstructed set of audio signals formed on the basis of the N audio objects to (a predefined configuration of) output channels of the renderer under control of the cluster metadata.
- Rendering of the reconstructed set of audio objects formed on the basis of the N audio objects to output channels of a predefined channel configuration may for example include forming linear combinations of the reconstructed set of audio objects formed on the basis of the N audio objects, employing coefficients determined based on the cluster metadata.
- According to an example embodiment, the respective points in time defined by the transition data for the respective cluster metadata instances may coincide with the respective points in time defined by the transition data for corresponding side information instances.
- According to an example embodiment, the method may further comprise:
- performing at least part of the reconstruction and at least part of the rendering as a combined operation corresponding to a first matrix formed as a matrix product of a reconstruction matrix and a rendering matrix associated with a current reconstruction setting and a current rendering setting, respectively;
- beginning, at a point in time defined by the transition data for a side information instance and a cluster metadata instance, a combined transition from the current reconstruction and rendering settings to desired reconstruction and rendering settings specified by the side information instance and the cluster metadata instance, respectively; and
- completing the combined transition at a point in time defined by the transition data for the side information instance and the cluster metadata instance, wherein the combined transition includes interpolating between matrix elements of the first matrix and matrix elements of a second matrix formed as a matrix product of a reconstruction matrix and a rendering matrix associated with the desired reconstruction setting and the desired rendering setting, respectively.
- By performing a combined transition in the above sense, instead of separate transitions of reconstruction settings and rendering settings, fewer parameters/coefficients need to be interpolated, which allows for a reduction of computational complexity.
- It is to be understood that a matrix, such as reconstruction matrix or a rendering matrix, as referenced in the present example embodiment, may for example consist of a single row or a single column, and may therefore correspond to a vector.
- Reconstruction of audio objects from downmix signals is often performed by employing different reconstruction matrices in different frequency bands, while rendering is often performed by employing the same rendering matrix for all frequencies. In such cases, a matrix corresponding to a combined operation of reconstruction and rendering, e.g. the first and second matrices referenced in the present example embodiment, may typically be frequency-dependent, i.e. different values for the matrix elements may typically be employed for different frequency bands.
- According to an example embodiment, the set of audio objects formed on the basis of the N audio objects may coincide with the N audio objects, i.e. the method may comprise reconstructing the N audio objects based on the M downmix signals and the side information.
- Alternatively, the set of audio objects formed on the basis of the N audio objects may comprise a plurality of audio objects which are combinations of the N audio objects, and whose number is less than N, i.e. the method may comprise reconstructing these combinations of the N audio objects based on the M downmix signals and the side information.
- According to an example embodiment, the data stream may further comprise downmix metadata for the M downmix signals including time-variable spatial positions associated with the M downmix signals. The data stream may comprise a plurality of downmix metadata instances, and the data stream may further comprise, for each downmix metadata instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current downmix rendering setting to a desired downmix rendering setting specified by the downmix metadata instance, and a point in time to complete the transition to the desired downmix rendering setting specified by the downmix metadata instance. The method may further comprise:
- on a condition that the decoder is operable (or configured) to support audio object reconstruction, performing the step of reconstructing, based on the M downmix signals and the side information, the set of audio objects formed on the basis of the N audio objects; and
- on a condition that the decoder is not operable (or configured) to support audio object reconstruction, outputting the downmix metadata and the M downmix signals for rendering of the M downmix signals.
- In case the decoder is operable to support audio object reconstruction and the data stream further comprises cluster metadata associated with the set of audio objects formed on the basis of the N audio objects, the decoder may e.g. output the reconstructed set of audio objects the cluster metadata for rendering of the reconstructed set of audio objects.
- In case the decoder is not operable to support audio object reconstruction, it may for example discard the side information and, if applicable, the cluster metadata, and provide the downmix metadata and the M downmix signals as output. Then, the output may be employed by a renderer for rendering the M downmix signals to output channels of the renderer.
- Optionally, the method may further comprise rendering the M downmix signals to output channels of a predefined output configuration, e.g. to output channels of a renderer, or to output channels of the decoder (in case the decoder has rendering capabilities), based on the downmix metadata.
- According to example embodiments, there is provided a decoder for reconstructing audio objects based on a data stream. The decoder comprises:
- a receiving component configured to receive a data stream comprising M downmix signals which are combinations of N audio objects, wherein N>1 and M≤N, and time-variable side information including parameters which allow reconstruction of a set of audio objects formed on the basis of the N audio objects from the M downmix signals; and
- a reconstructing component configured to reconstruct, based on the M downmix signals and the side information, the set of audio objects formed on the basis of the N audio objects,
- wherein the data stream comprises a plurality of side information instances associated, and wherein the data stream further comprises, for each side information instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current reconstruction setting to a desired reconstruction setting specified by the side information instance, and a point in time to complete the transition. The reconstructing component is configured to reconstruct the set of audio objects formed on the basis of the N audio objects by at least:
- performing reconstruction according to a current reconstruction setting;
- beginning, at a point in time defined by the transition data for a side information instance, a transition from the current reconstruction setting to a desired reconstruction setting specified by the side information instance; and
- completing the transition at a point in time defined by the transition data for the side information instance.
- According to an example embodiment, the method within the third or fourth aspect may further comprise generating one or more additional side information instances specifying substantially the same reconstruction setting as a side information instance directly preceding or directly succeeding the one or more additional side information instances. Example embodiments are also envisaged in which additional cluster metadata instances and/or downmix metadata instances are generated in an analogous fashion.
- As described above, resampling of the side information by generating more side information instances may be advantageous in several situations, such as when audio signals/objects and associated side information are encoded using a frame-based audio codec, since then it is desirable to have at least one side information instance for each audio codec frame. At an encoder side, the side information instances provided by an analysis component may e.g. be distributed in time in such a way that they do not match a frame rate of the downmix signals provided by a downmix component, and the side information may therefore advantageously be resampled by introducing new side information instances such that there is at least one side information instance for each frame of the downmix signals. Similarly, at a decoder side, the received side information instances may e.g. be distributed in time in such a way that they do not match a frame rate of the received downmix signals, and the side information may therefore advantageously be resampled by introducing new side information instances such that there is at least one side information instance for each frame of the downmix signals.
- An additional side information instance may for example be generated for a selected point in time by: copying the side information instance directly succeeding the additional side information instance and determining transition data for the additional side information instance based on the selected point in time and the points in time defined by the transition data for the succeeding side information instance.
- According to a fifth aspect, there is provided a method, a device, and a computer program product for transcoding side information encoded together with M audio signals in a data stream.
- The methods, devices and computer program products according to the fifth aspect are intended for cooperation with the methods, encoders, decoder and computer program products according to the third and fourth aspect, and may have corresponding features and advantages.
- According to example embodiments, there is provided a method for transcoding side information encoded together with M audio signals in a data stream. The method comprises:
- receiving a data stream;
- extracting, from the data stream, M audio signals and associated time-variable side information including parameters which allow reconstruction of a set of audio objects from the M audio signals, wherein M≥1, and wherein the extracted side information includes:
- a plurality of side information instances specifying respective desired reconstruction settings for reconstructing the audio objects, and
- for each side information instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current reconstruction setting to the desired reconstruction setting specified by the side information instance, and a point in time to complete the transition;
- generating one or more additional side information instances specifying substantially the same reconstruction setting as a side information instance directly preceding or directly succeeding the one or more additional side information instances; and
- including the M audio signals and the side information in a data stream.
- In the present example embodiment, the one or more additional side information instances may be generated after he side information has been extracted from the received data stream, and the generated one or more additional side information instances may then be included in a data stream together with the M audio signals and the other side information instances.
- As described above in relation to the third aspect, resampling of the side information by generating more side information instances may be advantageous in several situations, such as when audio signals/objects and associated side information are encoded using a frame-based audio codec, since then it is desirable to have at least one side information instance for each audio codec frame.
- Embodiments are also envisaged in which the data stream further comprises cluster metadata and/or downmix metadata, as described in relation to the third and fourth aspect, and wherein the method further comprises generating additional downmix metadata instances and/or cluster metadata instances, analogously to how the additional side information instances are generated.
- According to an example embodiment, the M audio signals may be coded in the received data stream according to a first frame rate, and the method may further comprise:
- processing the M audio signals to change the frame rate according to which the M downmix signals are coded to a second frame rate different than the first frame rate; and
- resampling the side information to match, and/or to be compatible with, the second frame rate by at least generating the one or more additional side information instances.
- As described above in relation to the third aspect, it may be advantageous in several situations to process audio signals so as to change the frame rate employed for coding them, e.g. so that the modified frame rate matches the frame rate of video content of an audio-visual signal to which the audio signals belong. The presence of the transition data for each side information instance facilitates resampling of the side information, as described above in relation to the third aspect. The side information may be resampled to match the new frame rate e.g. by generating additional side information instances such that there is at least one side information instance for each frame of the processed audio signals.
- According to example embodiments, there is provided a device for transcoding side information encoded together with M audio signals in a data stream. The device comprises:
- a receiving component configured to receive a data stream and to extract, from the data stream, M audio signals and associated time-variable side information including parameters which allow reconstruction of a set of audio objects from the M audio signals, wherein M≥1, and wherein the extracted side information includes:
- a plurality of side information instances specifying respective desired reconstruction settings for reconstructing the audio objects, and
- for each side information instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current reconstruction setting to the desired reconstruction setting specified by the side information instance, and a point in time to complete the transition.
- The device further comprises:
- a resampling component configured to generate one or more additional side information instances specifying substantially the same reconstruction setting as a side information instance directly preceding or directly succeeding the one or more additional side information instances; and
- a multiplexing component configured to include the M audio signals and the side information in a data stream.
- According to an example embodiment, the method within the third, fourth or fifth aspect may further comprise: computing a difference between a first desired reconstruction setting specified by a first side information instance and one or more desired reconstruction settings specified by one or more side information instances directly succeeding the first side information instance; and removing the one or more side information instances in response to the computed difference being below a predefined threshold. Example embodiments are also envisaged in which cluster metadata instances and/or downmix metadata instances are removed in an analogous fashion.
- By removing side information instances according to the present example embodiment, unnecessary computations based on these side information instances may be avoided, e.g. during reconstruction at a decoder side. By setting the predefined threshold at an appropriate (e.g. low enough) level, side information instances may be removed while the playback quality and/or the fidelity of the reconstructed audio signals is at least approximately maintained.
- The difference between the desired reconstruction settings may for example be computed based on differences between respective values for a set of coefficients employed as part of the reconstruction.
- According to example embodiments within the third, fourth or fifth aspect, the two independently assignable portions of the transition data for each side information instance may be:
- a time stamp indicating the point in time to begin the transition to the desired reconstruction setting and a time stamp indicating the point in time to complete the transition to the desired reconstruction setting;
- a time stamp indicating the point in time to begin the transition to the desired reconstruction setting and an interpolation duration parameter indicating a duration for reaching the desired reconstruction setting from the point in time to begin the transition to the desired reconstruction setting; or
- a time stamp indicating the point in time to complete the transition to the desired reconstruction setting and an interpolation duration parameter indicating a duration for reaching the desired reconstruction setting from the point in time to begin the transition to the desired reconstruction setting.
- In other words, the points in time to start and to end a transition may be defined in the transition data either by two time stamps indicating the respective points in time, or a combination of one of the time stamps and an interpolation duration parameter indicating a duration of the transition.
- The respective time stamps may for example indicate the respective points in time by referring to a time base employed for representing the M downmix signals and/or the N audio objects.
- According to example embodiments within the third, fourth or fifth aspect, the two independently assignable portions of the transition data for each cluster metadata instance may be:
- a time stamp indicating the point in time to begin the transition to the desired rendering setting and a time stamp indicating the point in time to complete the transition to the desired rendering setting;
- a time stamp indicating the point in time to begin the transition to the desired rendering setting and an interpolation duration parameter indicating a duration for reaching the desired rendering setting from the point in time to begin the transition to the desired rendering setting; or
- a time stamp indicating the point in time to complete the transition to the desired rendering setting and an interpolation duration parameter indicating a duration for reaching the desired rendering setting from the point in time to begin the transition to the desired rendering setting.
- According to example embodiments within the third, fourth or fifth aspect, the two independently assignable portions of the transition data for each downmix metadata instance may be:
- a time stamp indicating the point in time to begin the transition to the desired downmix rendering setting and a time stamp indicating the point in time to complete the transition to the desired downmix rendering setting;
- a time stamp indicating the point in time to begin the transition to the desired downmix rendering setting and an interpolation duration parameter indicating a duration for reaching the desired downmix rendering setting from the point in time to begin the transition to the desired downmix rendering setting; or
- a time stamp indicating the point in time to complete the transition to the desired downmix rendering setting and an interpolation duration parameter indicating a duration for reaching the desired downmix rendering setting from the point in time to begin the transition to the desired downmix rendering setting.
- According to example embodiments, there is provided a computer program product comprising a computer-readable medium with instructions for performing the method of any of the methods within the third, fourth or fifth aspect.
-
Fig. 1 illustrates anencoder 100 for encodingaudio objects 120 into adata stream 140 according to an exemplary embodiment. Theencoder 100 comprises a receiving component (not shown), adownmix component 102, anencoder component 104, ananalysis component 106, and amultiplexing component 108. The operation of theencoder 100 for encoding one time frame of audio data is described in the following. However, it is understood that the below method is repeated on a time frame basis. The same also applies to the description ofFigs 2-5 . - The receiving component receives a plurality of audio objects (N audio objects) 120 and
metadata 122 associated with the audio objects 120. An audio object as used herein refers to an audio signal having an associated spatial position which typically is varying with time (between time frames), i.e. the spatial position is dynamic. Themetadata 122 associated with theaudio objects 120 typically comprises information which describes how theaudio objects 120 are to be rendered for playback on the decoder side. In particular, themetadata 122 associated with the audio objects 120 includes information about the spatial position of theaudio objects 120 in the three-dimensional space of the audio scene. The spatial positions can be represented in Cartesian coordinates or by means of direction angles, such as azimuth and elevation, optionally augmented with distance. Themetadata 122 associated with theaudio objects 120 may further comprise object size, object loudness, object importance, object content type, specific rendering instructions such as application of dialog enhancement or exclusion of certain loudspeakers from rendering (so-called zone masks) and/or other object properties. - As will be described with reference to
Fig. 4 , the audio objects 120 may correspond to a simplified representation of an audio scene. - The N audio objects 120 are input to the
downmix component 102. Thedownmix component 102 calculates a number M of downmix signals 124 by forming combinations, typically linear combinations, of the N audio objects 120. In most cases, the number of downmix signals 124 is lower than the number ofaudio objects 120, i.e. M<N, such that the amount of data that is included in thedata stream 140 is reduced. However, for applications where the target bit rate of thedata stream 140 is high, the number of downmix signals 124 may be equal to the number ofobjects 120, i.e. M=N. - The
downmix component 102 may further calculate one or more auxiliary audio signals 127, here labeled by L auxiliary audio signals 127. The role of the auxiliary audio signals 127 is to improve the reconstruction of the N audio objects 120 at the decoder side. The auxiliary audio signals 127 may correspond to one or more of the N audio objects 120, either directly or as a combination of these. For example, the auxiliary audio signals 127 may correspond to particularly important ones of the N audio objects 120, such as anaudio object 120 corresponding to a dialogue. The importance may be reflected by or derived from themetadata 122 associated with the N audio objects 120. - The M downmix signals 124, and the L auxiliary signals 127 if present, may subsequently be encoded by the
encoder component 104, here labeled core encoder, to generate M encoded downmix signals 126 and L encoded auxiliary signals 129. Theencoder component 104 may be a perceptual audio codec as known in the art. Examples of known perceptual audio codecs include Dolby Digital and MPEG AAC. - In some embodiments, the
downmix component 102 may further associate the M downmix signals 124 withmetadata 125. In particular,downmix component 102 may associate each downmix signal 124 with a spatial position and include the spatial position in themetadata 125. Similar to themetadata 122 associated with theaudio objects 120, themetadata 125 associated with the downmix signals 124 may also comprise parameters related to size, loudness, importance, and/or other properties. - In particular, the spatial positions associated with the downmix signals 124 may be calculated based on the spatial positions of the N audio objects 120. Since the spatial positions of the N audio objects 120 may be dynamic, i.e. time-varying, also the spatial positions associated with the M downmix signals 124 may be dynamic. In other words, the M downmix signals 124 may themselves be interpreted as audio objects.
- The
analysis component 106 calculatesside information 128 including parameters which allow reconstruction of the N audio objects 120 (or a perceptually suitable approximation of the N audio objects 120) from the M downmix signals 124 and the Lauxiliary signals 129 if present. Also theside information 128 may be time-variable. For example, theanalysis component 106 may calculate theside information 128 by analyzing the M downmix signals 124, the L auxiliary signals 127 if present, and the N audio objects 120 according to any known technique for parametric encoding. Alternatively, theanalysis component 106 may calculate theside information 128 by analyzing the N audio objects, and information on how the M downmix signals were created from the N audio objects, for example by providing a (time-varying) downmix matrix. In that case, the M downmix signals 124 are not strictly required as an input to theanalysis component 106. - The M encoded downmix signals 126, the L encoded
auxiliary signals 129, theside information 128, themetadata 122 associated with the N audio objects, and themetadata 125 associated with the downmix signals are then input to themultiplexing component 108 which includes its input data in asingle data stream 140 using multiplexing techniques. Thedata stream 140 may thus include four types of data: - a) M downmix signals 126 (and optionally L auxiliary signals 129)
- b)
metadata 125 associated with the M downmix signals, - c)
side information 128 for reconstruction of the N audio objects from the M downmix signals, and - d)
metadata 122 associated with the N audio objects. - As mentioned above, some prior art systems for coding of audio objects requires that the M downmix signals are chosen such that they are suitable for playback on the channels of a speaker configuration with M channels, referred to herein as a backwards compatible downmix. Such a prior art requirement constrains the calculation of the downmix signals in that the audio objects may only be combined in a predefined manner. Accordingly, according to prior art, the downmix signals are not selected from the point of view of optimizing the reconstruction of the audio objects at a decoder side.
- As opposed to prior art systems, the
downmix component 102 calculates the M downmix signals 124 in a signal adaptive manner with respect to the N audio objects. In particular, thedownmix component 102 may, for each time frame, calculate the M downmix signals 124 as the combination of theaudio objects 120 that currently optimizes some criterion. The criterion is typically defined such that it is independent with respect to a any loudspeaker configuration, such as a 5.1 or other loudspeaker configuration. This implies that the M downmix signals 124, or at least one of them, are not constrained to audio signals which are suitable for playback on the channels of a speaker configuration with M channels. Accordingly, thedownmix component 102 may adapt the M downmix signals 124 to the temporal variation of the N audio objects 120 (including temporal variation of themetadata 122 including spatial positions of the N audio objects), in order to e.g. improve the reconstruction of theaudio objects 120 at the decoder side. - The
downmix component 102 may apply different criteria in order to calculate the M downmix signals. According to one example, the M downmix signals may be calculated such that the reconstruction of the N audio objects based on the M downmix signals is optimized. For example, thedownmix component 102 may minimize a reconstruction error formed from the N audio objects 120 and a reconstruction of the N audio objects based on the M downmix signals 124. - According to another example, the criterion is based on the spatial positions, and in particular spatial proximity, of the N audio objects 120. As discussed above, the N audio objects 120 have associated
metadata 122 which includes the spatial positions of the N audio objects 120. Based on themetadata 122, spatial proximity of the N audio objects 120 may be derived. - In more detail, the
downmix component 102 may apply a first clustering procedure in order to determine the M downmix signals 124. The first clustering procedure may comprise associating the N audio objects 120 with M clusters based on spatial proximity. Further properties of the N audio objects 120 as represented by the associatedmetadata 122, including object size, object loudness, object importance, may also be taken into account during the association of theaudio objects 120 with the M clusters. - According to one example, the well-known K-means algorithm, with the metadata 122 (spatial positions) of the N audio objects as input, may be used for associating the N audio objects 120 with the M clusters based on spatial proximity. The further properties of the N audio objects 120 may be used as weighting factors in the K-means algorithm.
- According to another example, the first clustering procedure may be based on a selection procedure which uses the importance of the audio objects, as given by the
metadata 122, as a selection criterion. In more detail, thedownmix component 102 may pass through the most importantaudio objects 120 such that one or more of the M downmix signals correspond to one or more of the N audio objects 120. The remaining, less important, audio objects may be associated with clusters based on spatial proximity as discussed above. - Further examples of clustering of audio objects are given in
US provisional application with number 61/865,072 - According to yet another example, the first clustering procedure may associate an
audio object 120 with more than one of the M clusters. For example anaudio object 120 may be distributed over the M clusters, wherein the distribution e.g. depends on the spatial position of theaudio object 120 and optionally also further properties of the audio object including object size, object loudness, object importance, etc. The distribution may be reflected by percentages, such that an audio object for instance is distributed over three clusters according to the percentages 20%, 30%, 50%. - Once the N audio objects 120 have been associated with the M clusters, the
downmix component 102 calculates adownmix signal 124 for each cluster by forming a combination, typically a linear combination, of theaudio objects 120 associated with the cluster. Typically, thedownmix component 102 may use parameters comprised in themetadata 122 associated withaudio objects 120 as weights when forming the combination. By way of example, theaudio objects 120 being associated with a cluster may be weighted according to object size, object loudness, object importance, object position, distance from an object with respect to a spatial position associated with the cluster (see details in the following) etc. In the case where theaudio objects 120 are distributed over the M clusters, the percentages reflecting the distribution may be used as weights when forming the combination. - The first clustering procedure is advantageous in that it easily allows association of each of the M downmix signals 124 with a spatial position. For example, the
downmix component 120 may calculate a spatial position of adownmix signal 124 corresponding to a cluster based on the spatial positions of theaudio objects 120 associated with the cluster. The centroid or a weighted centroid of the spatial positions of the audio objects being associated with the cluster may be used for this purpose. In case of a weighted centroid, the same weights may be used as when forming the combination of theaudio objects 120 associated with the cluster. -
Fig. 2 illustrates adecoder 200 corresponding to theencoder 100 ofFig. 1 . Thedecoder 200 is of the type that supports audio object reconstruction. Thedecoder 200 comprises a receivingcomponent 208, adecoder component 204, and areconstruction component 206. Thedecoder 200 may further comprise arenderer 210. Alternatively, thedecoder 200 may be coupled to arenderer 210 which forms part of a playback system. - The receiving
component 208 is configured to receive adata stream 240 from theencoder 100. The receivingcomponent 208 comprises a demultiplexing component configured to demultiplex the receiveddata stream 240 into its components, in this case M encoded downmix signals 226, optionally L encodedauxiliary signals 229,side information 228 for reconstruction of N audio objects from the M downmix signals and the L auxiliary signals, andmetadata 222 associated with the N audio objects. - The
decoder component 204 processes the M encoded downmix signals 226 to generate M downmix signals 224, and optionally L auxiliary signals 227. As further discussed above, the M downmix signals 224 were formed adaptively on the encoder side from the N audio objects, i.e. by forming combinations of the N audio objects according to a criterion which is independent of any loudspeaker configuration. - The
object reconstruction component 206 then reconstructs the N audio objects 220 (or a perceptually suitable approximation of these audio objects) based on the M downmix signals 224 and optionally the Lauxiliary signals 227 guided by theside information 228 derived on the encoder side. Theobject reconstruction component 206 may apply any known technique for such parametric reconstruction of the audio objects. - The reconstructed N audio objects 220 are then processed by the
renderer 210 using themetadata 222 associated with theaudio objects 222 and knowledge about the channel configuration of the playback system in order to generate anmultichannel output signal 230 suitable for playback. Typical speaker playback configurations include 22.2 and 11.1. Playback on soundbar speaker systems or headphones (binaural presentation) is also possible with dedicated renderers for such playback systems. -
Fig. 3 illustrates a low-complexity decoder 300 corresponding to theencoder 100 ofFig. 1 . Thedecoder 300 does not support audio object reconstruction. Thedecoder 300 comprises a receivingcomponent 308, and adecoding component 304. Thedecoder 300 may further comprise arenderer 310. Alternatively, the decoder is coupled to arenderer 310 which forms part of a playback system. - As discussed above, prior art systems which use a backwards compatible downmix (such as a 5.1 downmix), i.e. a downmix comprising M downmix signals which are suitable for direct playback on a playback system with M channels, easily enable low complexity decoding for legacy playback systems (that e.g. only support a 5.1 multichannel loudspeaker setup). Such prior art systems typically decodes the backwards compatible downmix signals themselves and discards additional parts of the data stream such as side information (cf.
item 228 ofFig. 2 ) and metadata associated with the audio objects (cf.item 222 ofFig. 2 ). However, when the downmix signals are formed adaptively as described above, the downmix signals are generally not suitable for direct playback on a legacy system. - The
decoder 300 is an example of a decoder which allows low-complexity decoding of M downmix signals which are adaptively formed for playback on a legacy playback system which only supports a particular playback configuration. - The receiving
component 308 receives abit stream 340 from an encoder, such asencoder 100 ofFig. 1 . The receivingcomponent 308 demultiplexes thebit stream 340 into its components. In this case, the receivingcomponent 308 will only keep the encoded M downmix signals 326 and themetadata 325 associated with the M downmix signals. The other components of thedata stream 340, such as the L auxiliary signals (cf.item 229 ofFig. 2 ) metadata associated with the N audio objects (cf.item 222 ofFig. 2 ) and the side information (cf.item 228 ofFig. 2 ) are discarded. - The
decoding component 304 decodes the M encoded downmix signals 326 to generate M downmix signals 324. The M downmix signals are then, together with the downmix metadata, input to therenderer 310 which renders the M downmix signals to amultichannel output 330 corresponding to a legacy playback format (which typically has M channels). Since thedownmix metadata 325 comprises spatial positions of the M downmix signals 324, therenderer 310 may typically be similar to therenderer 210 ofFig. 2 , with the only difference that therenderer 310 now takes the M downmix signals 324 and themetadata 325 associated with the M downmix signals 324 as input instead ofaudio objects 220 and their associatedmetadata 222. - As mentioned above in connection to
Fig. 1 , the N audio objects 120 may correspond to a simplified representation of an audio scene. - Generally, an audio scene may comprise audio objects and audio channels. By an audio channel is here meant an audio signal which corresponds to a channel of a multichannel speaker configuration. Examples of such multichannel speaker configurations include a 22.2 configuration, a 11.1 configuration etc. An audio channel may be interpreted as a static audio object having a spatial position corresponding to the speaker position of the channel.
- In some cases the number of audio objects and audio channels in the audio scene may be vast, such as more than 100 audio objects and 1-24 audio channels. If all of these audio objects/channels are to be reconstructed on the decoder side, a lot of computational power is required. Furthermore, the resulting data rate associated with object metadata and side information will generally be very high if many objects are provided as input. For this reason it is advantageous to simplify the audio scene in order to reduce the number of audio objects to be reconstructed on the decoder side. For this purpose, the encoder may comprise a clustering component which reduces the number of audio objects in the audio scene based on a second clustering procedure. The second clustering procedure aims at exploiting the spatial redundancy present in the audio scene, such as audio objects having equal or very similar locations. Additionally, perceptual importance of audio objects may be taken into account. Generally, such a clustering component may be arranged in sequence or in parallel with the
downmix component 102 ofFig. 1 . The sequential arrangement will be described with reference toFig. 4 and the parallel arrangement will be described with reference toFig. 5 . -
Fig. 4 illustrates anencoder 400. In addition to the components described with reference toFig. 1 , theencoder 400 comprises a clustering component 409. The clustering component 409 is arranged in sequence with thedownmix component 102, meaning that the output of the clustering component 409 is input to thedownmix component 102. - The clustering component 409 takes
audio objects 421 a and/oraudio channels 421 b as input together with associatedmetadata 423 including spatial positions of theaudio objects 421 a. The clustering component 409 converts theaudio channels 421 b to static audio objects by associating eachaudio channel 421 b with the spatial position of the speaker position corresponding to theaudio channel 421 b. The audio objects 421 a and the static audio objects formed from theaudio channels 421 b may be seen as a first plurality of audio objects 421. - The clustering component 409 generally reduces the first plurality of
audio objects 421 to a second plurality of audio objects, here corresponding to the N audio objects 120 ofFig. 1 . For this purpose the clustering component 409 may apply a second clustering procedure. - The second clustering procedure is generally similar to the first clustering procedure described above with respect to the
downmix component 102. The description of the first clustering procedure therefore also applies to the second clustering procedure. - In particular, the second clustering procedure involves associating the first plurality of audio objects 121 with at least one cluster, here N clusters, based on spatial proximity of the first plurality of audio objects 121. As further described above, the association with clusters may also be based on other properties of the audio objects as represented by the
metadata 423. Each cluster is then represented by an object which is a (linear) combination of the audio objects associated with that cluster. In the illustrated example, there are N clusters and hence N audio objects 120 are generated. The clustering component 409 further calculatesmetadata 122 for the so generated N audio objects 120. Themetadata 122 includes spatial positions of the N audio objects 120. The spatial position of each of the N audio objects 120 may be calculated based on the spatial positions of the audio objects associated with the corresponding cluster. By way of example the spatial position may be calculated as a centroid or a weighted centroid of the spatial positions of the audio objects associated with the cluster as further explained above with reference toFig. 1 . - The N audio objects 120 generated by the clustering component 409 are then input to the
downmix component 120 as further described with reference toFig. 1 . -
Fig. 5 illustrates anencoder 500. In addition to the components described with reference toFig. 1 , theencoder 500 comprises a clustering component 509. The clustering component 509 is arranged in parallel with thedownmix component 102, meaning that thedownmix component 102 and the clustering component 509 have the same input. - The input comprises a first plurality of audio objects, corresponding to the N audio objects 120 of
Fig. 1 , together with associatedmetadata 122 including spatial positions of the first plurality of audio objects. The first plurality ofaudio objects 120 may, similar to the first plurality of audio objects 121 ofFig. 4 , comprise audio objects and audio channels being converted into static audio objects. In contrast to the sequential arrangement ofFig. 4 where thedownmix component 102 operates on a reduced number of audio objects corresponding to a simplified version of the audio scene, thedownmix component 102 ofFig. 5 operates on the full audio content of the audio scene in order to generate M downmix signals 124. - The clustering component 509 is similar in functionality to the clustering component 409 described with reference to
Fig. 4 . In particular, the clustering component 509 reduces the first plurality ofaudio objects 120 to a second plurality ofaudio objects 521, here illustrated by K audio objects where typically M<K<N (for high bit applications M≤K≤N), by applying the second clustering procedure described above. The second plurality ofaudio objects 521 is thus a set of audio objects formed on basis of the N audio objects 126. Moreover the clustering component 509 calculatesmetadata 522 for the second plurality of audio objects 521 (the K audio objects) including spatial positions of the second plurality of audio objects 521. Themetadata 522 is included in thedata stream 540 by thedemultiplexing component 108. Theanalysis component 106 calculatesside information 528 which enables reconstruction of second plurality ofaudio objects 521, i.e. the set of audio objects formed on basis of the N audio objects (here the K audio objects), from the M downmix signals 124. Theside information 528 is included in thedata stream 540 by themultiplexing component 108. As further discussed above, theanalysis component 106 may for example derive theside information 528 by analyzing the second plurality ofaudio objects 521 and the M downmix signals 124. - The
data stream 540 generated by theencoder 500 may generally be decoded by thedecoder 200 ofFig. 2 or thedecoder 300 ofFig. 3 . However, the reconstructedaudio objects 220 ofFig. 2 (labeled N audio objects) now correspond to the second plurality of audio objects 521 (labeled K audio objects) ofFig. 5 , and themetadata 222 associated with the audio objects (labeled metadata of N audio objects) now correspond to themetadata 522 of the second plurality of audio objects (labeled metadata of K audio objects) ofFig. 5 . - In object-based audio encoding/decoding systems, side information or metadata associated with the objects is typically updated relatively infrequently (sparsely) in time to limit the associated data rate. Typical update intervals for object positions can range between 10 and 500 milliseconds, depending on the speed of the object, the required position accuracy, the available bandwidth to store or transmit metadata, etc. Such sparse, or even irregular metadata updates require interpolation of metadata and/or rendering matrices (i.e. matrices employed in rendering) for audio samples in-between two subsequent metadata instances. Without interpolation, the consequential step-wise changes in the rendering matrix may cause undesirable switching artifacts, clicking sounds, zipper noises, or other undesirable artifacts as a result of spectral splatter introduced by step-wise matrix updates.
-
Fig. 6 illustrates a typical known process to compute rendering matrices for rendering of audio signals or audio objects, based on a set of metadata instances. As shown inFig. 6 , a set of metadata instances (m1 to m4) 610 correspond to a set of points in time (t1 to t4) which are indicated by their position along thetime axis 620. Subsequently, each metadata instance is converted to a respective rendering matrix (c1 to c4) 630, or rendering setting, which is valid at the same time point as the metadata instance. Thus, as shown, metadata instance m1 creates rendering matrix c1 at time t1, metadata instance m2 creates rendering matrix c2 at time t2, and so on. For simplicity,Fig. 6 shows only one rendering matrix for each metadata instance m1 to m4. In practical systems, however, a rendering matrix c1 may comprise a set of rendering matrix coefficients or gain coefficients c 1,i,j to be applied to respective audio signals xi (t) to create output signals yj (t): - The
rendering matrices 630 generally comprise coefficients that represent gain values at different points in time. Metadata instances are defined at certain discrete points in time, and for audio samples in-between the metadata time points, the rendering matrix is interpolated, as indicated by the dashedline 640 connecting therendering matrices 630. Such interpolation can be performed linearly, but also other interpolation methods can be used (such as band-limited interpolation, sine/cosine interpolation, and etc.). The time interval between the metadata instances (and corresponding rendering matrices) is referred to as an "interpolation duration," and such intervals may be uniform or they may be different, such as the longer interpolation duration between times t3 and t4 as compared to the interpolation duration between times t2 and t3. - In many cases, the calculation of rendering matrix coefficients from metadata instances is well-defined, but the reverse process of calculating metadata instances given a (interpolated) rendering matrix, is often difficult, or even impossible. In this respect, the process of generating a rendering matrix from metadata can sometimes be regarded as a cryptographic one-way function. The process of calculating new metadata instances between existing metadata instances is referred to as "resampling" of the metadata. Resampling of metadata is often required during certain audio processing tasks. For example, when audio content is edited, by cutting/merging/mixing and so on, such edits may occur in between metadata instances. In this case, resampling of the metadata is required. Another such case is when audio and associated metadata are encoded with a frame-based audio codec. In this case, it is desirable to have at least one metadata instance for each audio codec frame, preferably with a time stamp at the start of that codec frame, to improve resilience of frame losses during transmission. Moreover, interpolation of metadata is also ineffective for certain types of metadata, such as binary-valued metadata, where standard techniques would derive the incorrect value more or less every second time. For example, if binary flags such as zone exclusion masks are used to exclude certain objects from the rendering at certain points in time, it is virtually impossible to estimate a valid set of metadata from the rendering matrix coefficients or from neighboring instances of metadata. This is shown in
Fig. 6 as a failed attempt to extrapolate or derive a metadata instance m3a from the rendering matrix coefficients in the interpolation duration between times t3 and t4. As shown inFig. 6 , metadata instances mx are only definitely defined at certain discrete points in time tx , which in turn produces the associated set of matrix coefficients cx. In between these discrete times tx , the sets of matrix coefficients must be interpolated based on past or future metadata instances. However, as described above, present metadata interpolation schemes suffer from loss of spatial audio quality due to unavoidable inaccuracies in metadata interpolation processes. Alternative interpolation schemes, according to example embodiments, will be described below with reference toFigs. 7-11 . - In the exemplary embodiments described with reference to
Figs. 1-5 , themetadata metadata 522 associated with the K objects 522 originate, at least in some example embodiments, from clustering components 409 and 509, and may be referred to as cluster metadata. Further, themetadata - As described with reference to
Figs. 1 ,4 and5 , thedownmix component 102 may calculate the M downmix signals 124 by forming combinations of the N audio objects 120 in a signal-adaptive manner, i.e. according to a criterion which is independent of any loudspeaker configuration. Such operation of thedownmix component 102 is characteristic of example embodiments within a first aspect. According to example embodiments within other aspects, thedownmix component 102 may e.g. calculate the M downmix signals 124 by forming combinations of the N audio objects 120 in a signal-adaptive manner, or, alternatively, such that the M downmix signals are suitable for playback on the channels of a speaker configuration with M channels, i.e. as a backwards compatible downmix. - In an example embodiment, the
encoder 400 described with reference toFig. 4 employs a metadata and side information format particularly suitable for resampling, i.e. for generating additional metadata and side information instances. In the present example embodiment, theanalysis component 106 calculates theside information 128 in a form which includes a plurality of side information instances specifying respective desired reconstruction settings for reconstructing the N audio objects 120, and, for each side information instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current reconstruction setting to the desired reconstruction setting specified by the side information instance, and a point in time to complete the transition. In the present example embodiment, the two independently assignable portions of the transition data for each side information instance are: a time stamp indicating the point in time to begin the transition to the desired reconstruction setting and an interpolation duration parameter indicating a duration for reaching the desired reconstruction setting from the point in time to begin the transition to the desired reconstruction setting. The interval during which a transition is to take place is in the present example embodiment uniquely defined by the time at which the transition is to begin and the duration of the transition interval. This particular form of theside information 128 will be described below with reference toFigs. 7-11 . It is to be understood that there are several other ways to uniquely define this transition interval. For example, a reference point in the form of a start, end or middle point of the interval, accompanied by the duration of the interval, may be employed in the transition data to uniquely define the interval. Alternatively, the start and end points of the interval may be employed in the transition data to uniquely define the interval. - In the present example embodiment, the clustering component 409 reduces the first plurality of
audio objects 421 to a second plurality of audio objects, here corresponding to the N audio objects 120 ofFig. 1 . The clustering component 409 calculates thecluster metadata 122 for the generated N audio objects 120 which enables rendering of the N audio objects 122 in arenderer 210 at a decoder side. The clustering component 409 provides thecluster metadata 122 in a form which includes a plurality of cluster metadata instances specifying respective desired rendering settings for rendering the N audio objects 120, and, for each cluster metadata instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current rendering setting to the desired rendering setting specified by the cluster metadata instance, and a point in time to complete the transition to the desired rendering setting. In the present example embodiment, the two independently assignable portions of the transition data for each cluster metadata instance are: a time stamp indicating the point in time to begin the transition to the desired rendering setting and an interpolation duration parameter indicating a duration for reaching the desired rendering setting from the point in time to begin the transition to the desired rendering setting. This particular form of thecluster metadata 122 will be described below with reference toFigs. 7-11 . - In the present example embodiment, the
downmix component 102 associates each downmix signal 124 with a spatial position and includes the spatial position in thedownmix metadata 125 which allows rendering of the M downmix signals in arenderer 310 at a decoder side. Thedownmix component 102 provides thedownmix metadata 125 in a form which includes a plurality of downmix metadata instances specifying respective desired downmix rendering settings for rendering the downmix signals, and, for each downmix metadata instance, transition data including two independently assignable portions which in combination define a point in time to begin a transition from a current downmix rendering setting to the desired downmix rendering setting specified by the downmix metadata instance, and a point in time to complete the transition to the desired downmix rendering setting. In the present example embodiment, the two independently assignable portions of the transition data for each downmix metadata instance are: a time stamp indicating the point in time to begin the transition to the desired downmix rendering setting and an interpolation duration parameter indicating a duration for reaching the desired downmix rendering setting from the point in time to begin the transition to the desired downmix rendering setting. - In the present example embodiment, the same format is employed for the
side information 128, thecluster metadata 122 and thedownmix metadata 125. This format will now be described with reference toFigs. 7-11 in terms of metadata for rendering of audio signals. However, it is to be understood that in the following examples described with reference toFigs. 7-11 , terms or expressions like "metadata for rendering of audio signals" may just as well be replaced by corresponding terms or expressions like "side information for reconstruction of audio objects", "cluster metadata for rendering of audio objects" or "downmix metadata for rendering of downmix signals". -
Fig. 7 illustrates the derivation, based on metadata, of coefficient curves employed in rendering of audio signals, according to an example embodiment. As shown inFig.7 , a set of metadata instances mx generated at different points in time tx , e.g. associated with unique time stamps, are converted by aconverter 710 into corresponding sets of matrix coefficient values cx. These sets of coefficients represent gain values, also referred to as gain factors, to be employed for rendering of the audio signals to various speakers and drivers in a playback system to which the audio content is to be rendered. Aninterpolator 720 then interpolates the gain factors cx to produce a coefficient curve between the discrete times tx. In an embodiment, the time stamps tx associated with each metadata instance mx may correspond to random points in time, synchronous points in time generated by a clock circuit, time events related to the audio content, such as frame boundaries, or any other appropriate timed event. Note that, as described above, the description provided with reference toFig. 7 applies analogously to side information for reconstruction of audio objects. -
Fig. 8 illustrates a metadata format according to an embodiment (and as described above, the following description applies analogously to a corresponding side information format), which addresses at least some of the interpolation problems associated with present methods, as described above, by defining a time stamp as the start time of a transition or an interpolation, and augmenting each metadata instance with an interpolation duration parameter that represents the transition duration or interpolation duration (also referred to as "ramp size"). As shown inFig. 8 , a set of metadata instances m2 to m4 (810) specifies a set of rendering matrices c2 to c4 (830). Each metadata instance is generated at a particular point in time tx , and each metadata instance is defined with respect to its time stamp, m2 to t2, m3 to t3, and so on. The associatedrendering matrices 830 are generated after performing transitions during respective interpolation durations d2, d3, d4 (830), from the associated time stamp (t1 to t4) of eachmetadata instance 810. An interpolation duration parameter indicating the interpolation duration (or ramp size) is included with each metadata instance, i.e., metadata instance m2 includes d2, m3 includes d3, and so on. Schematically this can be represented as follows: mx = (metadata(tx ), dx ) → cx. In this manner, the metadata essentially provides a schematic of how to proceed from a current rendering setting (e.g., the current rendering matrix resulting from previous metadata) to a new rendering setting (e.g., the new rendering matrix resulting from the current metadata). Each metadata instance is meant to take effect at a specified point in time in the future relative to the moment the metadata instance was received and the coefficient curve is derived from the previous state of the coefficient. Thus, inFig. 8 , m2 generates c2 after a duration d2, m3 generates c3 after a duration d3 and m4 generates c4 after a duration d4. In this scheme for interpolation, the previous metadata need not be known, only the previous rendering matrix or rendering state is required. The interpolation employed may be linear or non-linear depending on system constraints and configurations. - The metadata format of
Fig. 8 allows for lossless resampling of metadata, as shown inFig. 9. Fig. 9 illustrates a first example of lossless processing of metadata, according to an example embodiment (and as described above, the following description applies analogously to a corresponding side information format).Fig. 9 shows metadata instances m2 to m4 that refer to the future rendering matrices c2 to c4, respectively, including interpolation durations d2 to d4. The time stamps of the metadata instances m2 to m4 are given as t2 to t4. In the example ofFig. 9 , a metadata instance m4a, at time t4a, is added. Such metadata may be added for several reasons, such as to improve error resilience of the system or to synchronize metadata instances with the start/end of an audio frame. For example, time t4a may represent the time that an audio codec employed for coding audio content associated with the metadata starts a new frame. For lossless operation, the metadata values of m4a are identical to those of m4 (i.e. they both describe a target rendering matrix c4), but the time d4a to reach that point has been reduced by d4-d4a. In other words, metadata instance m4a is identical to that of the previous metadata instance m4 so that the interpolation curve between c3 and c4 is not changed. However, the new interpolation duration d4a, is shorter than the original duration d4. This effectively increases the data rate of the metadata instances, which can be beneficial in certain circumstances, such as error correction. - A second example of lossless metadata interpolation is shown in
Fig. 10 (and as described above, the following description applies analogously to a corresponding side information format). In this example, the goal is to include a new set of metadata m3a in between two metadata instances m3 and m4.Fig. 10 illustrates a case where the rendering matrix remains unchanged for a period of time. Therefore, in this situation, the values of the new set of metadata m3a are identical to those of the prior metadata m3, except for the interpolation duration d3a. The value of the interpolation duration d3a should be set to the value corresponding to t4-t3a, i.e. to the difference between time t4 associated with the next metadata instance m4 and the time t3a associated with the new set of metadata m3a. The case illustrated inFig. 10 may for example occur when an audio object is static and an authoring tool stops sending new metadata for the object due to this static nature. In such a case, it may be desirable to insert new metadata instances m3a, e.g. to synchronize the metadata with codec frames. - In the examples illustrated in
Figs. 8 to 10 , the interpolation from a current to a desired rendering matrix or rendering state was performed by linear interpolation. In other example embodiments, different interpolation schemes may also be used. One such alternative interpolation scheme uses a sample-and-hold circuit combined with a subsequent low-pass filter.Fig. 11 illustrates an interpolation scheme using a sample-and-hold circuit with a low-pass filter, according to an example embodiment (and as described above, the following description applies analogously to a corresponding side information format). As shown inFig. 11 , the metadata instances m2 to m4 are converted to sample-and-hold rendering matrix coefficients c2 and c3. The sample-and-hold process causes the coefficient states to jump immediately to the desired state, which results in astep-wise curve 1110, as shown. Thiscurve 1110 is then subsequently low-pass filtered to obtain a smooth, interpolatedcurve 1120. The interpolation filter parameters (e.g., cut-off frequency or time constant) can be signaled as part of the metadata, in addition to the time stamps and the interpolation duration parameters. It is to be understood that different parameters may be used depending on the requirements of the system and the characteristics of the audio signal. - In an example embodiment, the interpolation duration or ramp size can have any practical value, including a value of or substantially close to zero. Such small interpolation duration is especially helpful for cases such as initialization in order to enable setting the rendering matrix immediately at the first sample of a file, or allowing for edits, splicing, or concatenation of streams. With this type of destructive edits, having the possibility to instantaneously change the rendering matrix can be beneficial to maintain the spatial properties of the content after editing.
- In an example embodiment, the interpolation scheme described herein is compatible with the removal of metadata instances (and analogously with the removal of side information instances, as described above), such as in a decimation scheme that reduces metadata bitrates. Removal of metadata instances allows the system to resample at a frame rate that is lower than an initial frame rate. In this case, metadata instances and their associated interpolation duration data that are provided by an encoder may be removed based on certain characteristics. For example, an analysis component in an encoder may analyze the audio signal to determine if there is a period of significant stasis of the signal, and in such a case remove certain metadata instances already generated to reduce bandwidth requirements for the transmittal of data to a decoder side. The removal of metadata instances may alternatively or additionally be performed in a component separate from the encoder, such as in a decoder or in a transcoder. A transcoder may remove metadata instances that have been generated or added by the encoder, and may be employed in a data rate converter that re-samples an audio signal from a first rate to a second rate, where the second rate may or may not be an integer multiple of the first rate. Alternatively to analyzing the audio signal in order to determine which metadata instances to remove, the encoder, decoder or transcoder may analyze the metadata. For example, with reference to
Fig. 10 , a difference may be computed between a first desired reconstruction setting c3 (or reconstruction matrix), specified by a first metadata instance m3, and desired reconstruction settings c3a and c4 (or reconstruction matrices) specified by metadata instances m3a and m4 directly succeeding the first metadata instance m3. The difference may for example be computed by employing a matrix norm to the respective rendering matrices. If the difference is below a predefined threshold, e.g. corresponding to a tolerated distortion of the reconstructed audio signals, the metadata instances m3a and m4 succeeding the first metadata instance m2 may be removed. In the example illustrated inFig. 10 , the metadata instance m3a directly succeeding the first metadata instance m3 specifies the same rendering settings c3=c3a as the first metadata instance m3and will therefore be removed, while the next metadata setting m4 specifies a different rendering setting c4 and may, depending on the threshold employed, be kept as metadata. - In the
decoder 200 described with reference toFig. 2 , theobject reconstruction component 206 may employ interpolation as part of reconstructing the N audio objects 220 based on the M downmix signals 224 and theside information 228. In analogy with the interpolation scheme described with reference toFigs. 7-11 , reconstructing the N audio objects 220 may for example include: performing reconstruction according to a current reconstruction setting; beginning, at a point in time defined by the transition data for a side information instance, a transition from the current reconstruction setting to a desired reconstruction setting specified by the side information instance; and completing the transition to the desired reconstruction setting at a point in time defined by the transition data for the side information instance. - Similarly, the
renderer 210 may employ interpolation as part of rendering the reconstructed N audio objects 220 in order to generate themultichannel output signal 230 suitable for playback. In analogy with the interpolation scheme described with reference toFigs. 7-11 , the rendering may include: performing rendering according to a current rendering setting; beginning, at a point in time defined by the transition data for a cluster metadata instance, a transition from the current rendering setting to a desired rendering setting specified by the cluster metadata instance; and completing the transition to the desired rendering setting at a point in time defined by the transition data for the cluster metadata instance. - In some example embodiments, the
object reconstruction section 206 and therenderer 210 may be separate units, and/or may correspond to operations performed as separate processes. In other example embodiments, theobject reconstruction section 206 and therenderer 210 may be embodied as a single unit or process in which reconstruction and rendering is performed as a combined operation. In such example embodiments, matrices employed for reconstruction and rendering may be combined into a single matrix which may be interpolated, instead of performing interpolation on a rendering matrix and a reconstruction matrix, separately. - In the low-
complexity decoder 300, described with reference toFig. 3 , therenderer 310 may perform interpolation as part of rendering the M downmix signals 324 to themultichannel output 330. In analogy with the interpolation scheme described with reference toFigs. 7-11 , the rendering may include: performing rendering according to a current downmix rendering setting; beginning, at a point in time defined by the transition data for a downmix metadata instance, a transition from the current downmix rendering setting to a desired downmix rendering setting specified by the downmix metadata instance; and completing the transition to the desired downmix rendering setting at a point in time defined by the transition data for the downmix metadata instance. As previously described, therenderer 310 may be comprised in thedecoder 300 or may be a separate device/unit. In example embodiments where therenderer 310 is separate from thedecoder 300, the decoder may output thedownmix metadata 325 and the M downmix signals 324 for rendering of the M downmix signals in therenderer 310. - Further embodiments of the present disclosure will become apparent to a person skilled in the art after studying the description above. Even though the present description and drawings disclose embodiments and examples, the disclosure is not restricted to these specific examples. Numerous modifications and variations can be made without departing from the scope of the present disclosure, which is defined by the accompanying claims. Any reference signs appearing in the claims are not to be understood as limiting their scope.
- Additionally, variations to the disclosed embodiments can be understood and effected by the skilled person in practicing the disclosure, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage.
- The systems and methods disclosed hereinabove may be implemented as software, firmware, hardware or a combination thereof. In a hardware implementation, the division of tasks between functional units referred to in the above description does not necessarily correspond to the division into physical units; to the contrary, one physical component may have multiple functionalities, and one task may be carried out by several physical components in cooperation. Certain components or all components may be implemented as software executed by a digital signal processor or microprocessor, or be implemented as hardware or as an application-specific integrated circuit. Such software may be distributed on computer readable media, which may comprise computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to a person skilled in the art, the term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it is well known to the skilled person that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
- All the figures are schematic and generally only show parts which are necessary in order to elucidate the disclosure, whereas other parts may be omitted or merely suggested. Unless otherwise indicated, like reference numerals refer to like parts in different figures.
Claims (15)
- A method for encoding audio objects into a data stream (140), comprising:receiving N audio objects (120), wherein N>1;calculating M downmix signals (124), wherein M≤N, by forming combinations of the N audio objects according to a criterion which is independent of any M-channel loudspeaker configuration for playback of the M downmix signals, wherein the N audio objects are associated with metadata including spatial positions of the N audio objects and importance values indicating the importance of the N audio objects in relation to each other, wherein the criterion for calculating the M downmix signals is based on spatial proximity of the N audio objects and on the importance values of the N audio objects;calculating side information (128) including parameters which allow reconstruction of a set of audio objects formed on basis of the N audio objects from the M downmix signals; andincluding the M downmix signals and the side information in a data stream (140) for transmittal to a decoder.
- The method of claim 1, wherein one of the M downmix signals corresponds to a single one of the N audio objects, wherein said single one of the N audio objects is the audio object of the N audio objects which is the most important in relation to the other ones of the N audio objects.
- The method of any one of claims 1-2, further comprising associating each downmix signal with a spatial position and including the spatial positions of the downmix signals in the data stream as metadata for the downmix signals.
- The method of claim 3, wherein the N audio objects are associated with metadata including spatial positions of the N audio objects, and the spatial positions associated with the downmix signals are calculated based on the spatial positions of the N audio objects, and
optionally, wherein the spatial positions of the N audio objects and the spatial positions associated with the M downmix signals are time-varying. - The method of any one of the preceding claims, wherein the side information is time-varying.
- The method of any one of the preceding claims, wherein the step of calculating M downmix signals comprises a first clustering procedure which includes associating the N audio objects with M clusters based on spatial proximity and importance values, of the N audio objects, and calculating a downmix signal for each cluster by forming a combination of audio objects associated with the cluster.
- The method of claim 6, wherein each downmix signal is associated with a spatial position which is calculated based on the spatial positions of the audio objects associated with the cluster corresponding to the downmix signal, and
optionally, wherein the spatial position associated with each downmix signal is calculated as a centroid or a weighted centroid of the spatial positions of the audio objects associated with the cluster corresponding to the downmix signal. - A method for decoding a data stream including encoded audio objects, comprising:receiving a data stream comprising M downmix signals which are combinations of N audio objects calculated according to a criterion which is independent of any M-channel loudspeaker configuration for playback of the M downmix signals, wherein M≤N, wherein the criterion for calculating the M downmix signals is based on spatial proximity of the N audio objects and on importance values of the N audio objects indicating the importance of the N audio objects in relation to each other,receiving side information including parameters which allow reconstruction of a set of audio objects formed on basis of the N audio objects from the M downmix signals; andreconstructing the set of audio objects formed on basis of the N audio objects from the M downmix signals and the side information.
- The method of claim 8, wherein one of the M downmix signals corresponds to a single one of the N audio objects, wherein said single one of the N audio objects is the audio object of the N audio objects which is the most important in relation to the other ones of the N audio objects.
- The method of claim 8 or claim 9, wherein the data stream further comprises metadata for the M downmix signals including spatial positions associated with the M downmix signals, the method further comprising:on a condition that the decoder is configured to support audio object reconstruction, performing the step of reconstructing the set of audio objects formed on basis N audio objects from the M downmix signals and the side information; andon a condition that the decoder is not configured to support audio object reconstruction, using the metadata for the M downmix signals for rendering of the M downmix signals to output channels of a playback system, andoptionally, wherein the spatial positions associated with the M downmix signals are time-varying.
- The method of any one of claims 8-10, wherein the side information is time-varying.
- The method of any one of claims 8-11, wherein the data stream further comprises metadata for the set of audio objects formed on basis of the N audio objects including the spatial positions of the set of audio objects formed on basis of the N audio objects, the method further comprising:using the metadata for the set of audio objects formed on basis of the N audio objects for rendering of the reconstructed set of audio objects formed on basis of the N audio objects to output channels of a playback system.
- The method of any one of claims 8-12, wherein the set of audio objects formed on basis of the N audio objects is equal to the N audio objects, or
wherein the set of audio objects formed on basis of the N audio objects comprises a plurality of audio objects which are combinations of the N audio objects, and the number of which is lower than N. - A computer program product comprising a computer-readable medium with instructions for performing the method of any one of claims 1 to 7, or with instructions for performing the method of any one of claims 8 to 13.
- A decoder (200, 300, 400, 500) for decoding a data stream including encoded audio objects, the decoder comprising a receiving component and a reconstructing component, the decoder being configured to perform the method of any one of claims 8 to 13.
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201361827246P | 2013-05-24 | 2013-05-24 | |
US201361893770P | 2013-10-21 | 2013-10-21 | |
US201461973623P | 2014-04-01 | 2014-04-01 | |
PCT/EP2014/060733 WO2014187990A1 (en) | 2013-05-24 | 2014-05-23 | Efficient coding of audio scenes comprising audio objects |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3005356A1 EP3005356A1 (en) | 2016-04-13 |
EP3005356B1 true EP3005356B1 (en) | 2017-08-09 |
Family
ID=50943284
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP14730451.3A Active EP3005356B1 (en) | 2013-05-24 | 2014-05-23 | Efficient coding of audio scenes comprising audio objects |
Country Status (10)
Country | Link |
---|---|
US (1) | US9892737B2 (en) |
EP (1) | EP3005356B1 (en) |
JP (1) | JP6190947B2 (en) |
KR (1) | KR101760248B1 (en) |
CN (1) | CN105229732B (en) |
BR (2) | BR112015029129B1 (en) |
ES (1) | ES2640815T3 (en) |
HK (1) | HK1213685A1 (en) |
RU (1) | RU2630754C2 (en) |
WO (1) | WO2014187990A1 (en) |
Families Citing this family (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3203469A4 (en) * | 2014-09-30 | 2018-06-27 | Sony Corporation | Transmitting device, transmission method, receiving device, and receiving method |
CA2963771A1 (en) * | 2014-10-16 | 2016-04-21 | Sony Corporation | Transmission device, transmission method, reception device, and reception method |
US10475463B2 (en) * | 2015-02-10 | 2019-11-12 | Sony Corporation | Transmission device, transmission method, reception device, and reception method for audio streams |
CN111586533B (en) * | 2015-04-08 | 2023-01-03 | 杜比实验室特许公司 | Presentation of audio content |
JPWO2016194563A1 (en) | 2015-06-02 | 2018-03-22 | ソニー株式会社 | Transmitting apparatus, transmitting method, media processing apparatus, media processing method, and receiving apparatus |
WO2017027308A1 (en) * | 2015-08-07 | 2017-02-16 | Dolby Laboratories Licensing Corporation | Processing object-based audio signals |
US10278000B2 (en) | 2015-12-14 | 2019-04-30 | Dolby Laboratories Licensing Corporation | Audio object clustering with single channel quality preservation |
US10779106B2 (en) | 2016-07-20 | 2020-09-15 | Dolby Laboratories Licensing Corporation | Audio object clustering based on renderer-aware perceptual difference |
EP4054213A1 (en) | 2017-03-06 | 2022-09-07 | Dolby International AB | Rendering in dependence on the number of loudspeaker channels |
WO2019069710A1 (en) * | 2017-10-05 | 2019-04-11 | ソニー株式会社 | Encoding device and method, decoding device and method, and program |
CN108733342B (en) * | 2018-05-22 | 2021-03-26 | Oppo(重庆)智能科技有限公司 | Volume adjusting method, mobile terminal and computer readable storage medium |
JP7504091B2 (en) | 2018-11-02 | 2024-06-21 | ドルビー・インターナショナル・アーベー | Audio Encoders and Decoders |
BR112021009306A2 (en) * | 2018-11-20 | 2021-08-10 | Sony Group Corporation | information processing device and method; and, program. |
CN113490980A (en) * | 2019-01-21 | 2021-10-08 | 弗劳恩霍夫应用研究促进协会 | Apparatus and method for encoding a spatial audio representation and apparatus and method for decoding an encoded audio signal using transmission metadata, and related computer program |
JPWO2021140959A1 (en) * | 2020-01-10 | 2021-07-15 | ||
US20240187807A1 (en) * | 2021-02-20 | 2024-06-06 | Dolby Laboratories Licensing Corporation | Clustering audio objects |
Family Cites Families (44)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7567675B2 (en) | 2002-06-21 | 2009-07-28 | Audyssey Laboratories, Inc. | System and method for automatic multiple listener room acoustic correction with low filter orders |
DE10344638A1 (en) | 2003-08-04 | 2005-03-10 | Fraunhofer Ges Forschung | Generation, storage or processing device and method for representation of audio scene involves use of audio signal processing circuit and display device and may use film soundtrack |
FR2862799B1 (en) * | 2003-11-26 | 2006-02-24 | Inst Nat Rech Inf Automat | IMPROVED DEVICE AND METHOD FOR SPATIALIZING SOUND |
US7394903B2 (en) | 2004-01-20 | 2008-07-01 | Fraunhofer-Gesellschaft Zur Forderung Der Angewandten Forschung E.V. | Apparatus and method for constructing a multi-channel output signal or for generating a downmix signal |
WO2005098824A1 (en) | 2004-04-05 | 2005-10-20 | Koninklijke Philips Electronics N.V. | Multi-channel encoder |
GB2415639B (en) | 2004-06-29 | 2008-09-17 | Sony Comp Entertainment Europe | Control of data processing |
DE602006015294D1 (en) | 2005-03-30 | 2010-08-19 | Dolby Int Ab | MULTI-CHANNEL AUDIO CODING |
WO2007055463A1 (en) * | 2005-08-30 | 2007-05-18 | Lg Electronics Inc. | Apparatus for encoding and decoding audio signal and method thereof |
JP5154538B2 (en) | 2006-03-29 | 2013-02-27 | コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ | Audio decoding |
US8379868B2 (en) | 2006-05-17 | 2013-02-19 | Creative Technology Ltd | Spatial audio coding based on universal spatial cues |
EP2067138B1 (en) * | 2006-09-18 | 2011-02-23 | Koninklijke Philips Electronics N.V. | Encoding and decoding of audio objects |
RU2407072C1 (en) * | 2006-09-29 | 2010-12-20 | ЭлДжи ЭЛЕКТРОНИКС ИНК. | Method and device for encoding and decoding object-oriented audio signals |
KR20090013178A (en) | 2006-09-29 | 2009-02-04 | 엘지전자 주식회사 | Methods and apparatuses for encoding and decoding object-based audio signals |
US8620465B2 (en) | 2006-10-13 | 2013-12-31 | Auro Technologies | Method and encoder for combining digital data sets, a decoding method and decoder for such combined digital data sets and a record carrier for storing such combined digital data set |
EP2068307B1 (en) * | 2006-10-16 | 2011-12-07 | Dolby International AB | Enhanced coding and parameter representation of multichannel downmixed object coding |
RU2431940C2 (en) | 2006-10-16 | 2011-10-20 | Фраунхофер-Гезелльшафт цур Фёрдерунг дер ангевандтен Форшунг Е.Ф. | Apparatus and method for multichannel parametric conversion |
KR101055739B1 (en) | 2006-11-24 | 2011-08-11 | 엘지전자 주식회사 | Object-based audio signal encoding and decoding method and apparatus therefor |
CN101490745B (en) * | 2006-11-24 | 2013-02-27 | Lg电子株式会社 | Method and apparatus for encoding and decoding an audio signal |
US8290167B2 (en) | 2007-03-21 | 2012-10-16 | Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. | Method and apparatus for conversion between multi-channel audio formats |
EP2076900A1 (en) | 2007-10-17 | 2009-07-08 | Fraunhofer-Gesellschaft zur Förderung der Angewandten Forschung e.V. | Audio coding using upmix |
CA2710562C (en) | 2008-01-01 | 2014-07-22 | Lg Electronics Inc. | A method and an apparatus for processing an audio signal |
KR101461685B1 (en) | 2008-03-31 | 2014-11-19 | 한국전자통신연구원 | Method and apparatus for generating side information bitstream of multi object audio signal |
US8311810B2 (en) | 2008-07-29 | 2012-11-13 | Panasonic Corporation | Reduced delay spatial coding and decoding apparatus and teleconferencing system |
MX2011011399A (en) * | 2008-10-17 | 2012-06-27 | Univ Friedrich Alexander Er | Audio coding using downmix. |
EP2214161A1 (en) | 2009-01-28 | 2010-08-04 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Apparatus, method and computer program for upmixing a downmix audio signal |
US9105264B2 (en) | 2009-07-31 | 2015-08-11 | Panasonic Intellectual Property Management Co., Ltd. | Coding apparatus and decoding apparatus |
KR101805212B1 (en) | 2009-08-14 | 2017-12-05 | 디티에스 엘엘씨 | Object-oriented audio streaming system |
US9432790B2 (en) | 2009-10-05 | 2016-08-30 | Microsoft Technology Licensing, Llc | Real-time sound propagation for dynamic sources |
WO2011048067A1 (en) | 2009-10-20 | 2011-04-28 | Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E. V. | Apparatus for providing an upmix signal representation on the basis of a downmix signal representation, apparatus for providing a bitstream representing a multichannel audio signal, methods, computer program and bitstream using a distortion control signaling |
AU2010321013B2 (en) | 2009-11-20 | 2014-05-29 | Dolby International Ab | Apparatus for providing an upmix signal representation on the basis of the downmix signal representation, apparatus for providing a bitstream representing a multi-channel audio signal, methods, computer programs and bitstream representing a multi-channel audio signal using a linear combination parameter |
TWI444989B (en) | 2010-01-22 | 2014-07-11 | Dolby Lab Licensing Corp | Using multichannel decorrelation for improved multichannel upmixing |
DK2556502T3 (en) | 2010-04-09 | 2019-03-04 | Dolby Int Ab | MDCT-BASED COMPLEX PREVIEW Stereo Decoding |
GB2485979A (en) | 2010-11-26 | 2012-06-06 | Univ Surrey | Spatial audio coding |
JP2012151663A (en) | 2011-01-19 | 2012-08-09 | Toshiba Corp | Stereophonic sound generation device and stereophonic sound generation method |
US9026450B2 (en) * | 2011-03-09 | 2015-05-05 | Dts Llc | System for dynamically creating and rendering audio objects |
WO2013142657A1 (en) | 2012-03-23 | 2013-09-26 | Dolby Laboratories Licensing Corporation | System and method of speaker cluster design and rendering |
US9516446B2 (en) | 2012-07-20 | 2016-12-06 | Qualcomm Incorporated | Scalable downmix design for object-based surround codec with cluster analysis by synthesis |
US9761229B2 (en) * | 2012-07-20 | 2017-09-12 | Qualcomm Incorporated | Systems, methods, apparatus, and computer-readable media for audio object clustering |
CN104520924B (en) | 2012-08-07 | 2017-06-23 | 杜比实验室特许公司 | Indicate coding and the presentation of the object-based audio of gaming audio content |
CN104885151B (en) | 2012-12-21 | 2017-12-22 | 杜比实验室特许公司 | For the cluster of objects of object-based audio content to be presented based on perceptual criteria |
WO2014161993A1 (en) | 2013-04-05 | 2014-10-09 | Dolby International Ab | Stereo audio encoder and decoder |
EP3270375B1 (en) | 2013-05-24 | 2020-01-15 | Dolby International AB | Reconstruction of audio scenes from a downmix |
CA2990261C (en) | 2013-05-24 | 2020-06-16 | Dolby International Ab | Audio encoder and decoder |
CN105247611B (en) | 2013-05-24 | 2019-02-15 | 杜比国际公司 | To the coding of audio scene |
-
2014
- 2014-05-23 RU RU2015150055A patent/RU2630754C2/en active
- 2014-05-23 EP EP14730451.3A patent/EP3005356B1/en active Active
- 2014-05-23 KR KR1020157033447A patent/KR101760248B1/en active IP Right Grant
- 2014-05-23 ES ES14730451.3T patent/ES2640815T3/en active Active
- 2014-05-23 CN CN201480029540.0A patent/CN105229732B/en active Active
- 2014-05-23 BR BR112015029129-5A patent/BR112015029129B1/en active IP Right Grant
- 2014-05-23 WO PCT/EP2014/060733 patent/WO2014187990A1/en active Application Filing
- 2014-05-23 BR BR122020017144-8A patent/BR122020017144B1/en active IP Right Grant
- 2014-05-23 JP JP2016513405A patent/JP6190947B2/en active Active
- 2014-05-23 US US14/893,485 patent/US9892737B2/en active Active
-
2016
- 2016-02-03 HK HK16101241.7A patent/HK1213685A1/en unknown
Non-Patent Citations (1)
Title |
---|
None * |
Also Published As
Publication number | Publication date |
---|---|
BR112015029129B1 (en) | 2022-05-31 |
WO2014187990A1 (en) | 2014-11-27 |
BR122020017144B1 (en) | 2022-05-03 |
US9892737B2 (en) | 2018-02-13 |
BR112015029129A2 (en) | 2017-07-25 |
KR20160003058A (en) | 2016-01-08 |
US20160125887A1 (en) | 2016-05-05 |
RU2630754C2 (en) | 2017-09-12 |
JP6190947B2 (en) | 2017-08-30 |
JP2016522911A (en) | 2016-08-04 |
ES2640815T3 (en) | 2017-11-06 |
EP3005356A1 (en) | 2016-04-13 |
CN105229732B (en) | 2018-09-04 |
RU2015150055A (en) | 2017-05-26 |
KR101760248B1 (en) | 2017-07-21 |
HK1213685A1 (en) | 2016-07-08 |
CN105229732A (en) | 2016-01-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11705139B2 (en) | Efficient coding of audio scenes comprising audio objects | |
EP3005356B1 (en) | Efficient coding of audio scenes comprising audio objects | |
EP3127109B1 (en) | Efficient coding of audio scenes comprising audio objects | |
JP7413418B2 (en) | Audio decoder for interleaving signals | |
EP2227804B1 (en) | A method and an apparatus for processing a signal |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20160104 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20170306 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP Ref country code: AT Ref legal event code: REF Ref document number: 917622 Country of ref document: AT Kind code of ref document: T Effective date: 20170815 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: FP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602014012861 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: ES Ref legal event code: FG2A Ref document number: 2640815 Country of ref document: ES Kind code of ref document: T3 Effective date: 20171106 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 917622 Country of ref document: AT Kind code of ref document: T Effective date: 20170809 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171109 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171109 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171110 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20171209 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602014012861 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 5 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed |
Effective date: 20180511 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20180531 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180531 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180531 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180523 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180523 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180531 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180523 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 Ref country code: MK Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170809 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20140523 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170809 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 9 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 602014012861 Country of ref document: DE Owner name: DOLBY INTERNATIONAL AB, IE Free format text: FORMER OWNER: DOLBY INTERNATIONAL AB, AMSTERDAM, NL Ref country code: DE Ref legal event code: R081 Ref document number: 602014012861 Country of ref document: DE Owner name: DOLBY INTERNATIONAL AB, NL Free format text: FORMER OWNER: DOLBY INTERNATIONAL AB, AMSTERDAM, NL |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 602014012861 Country of ref document: DE Owner name: DOLBY INTERNATIONAL AB, IE Free format text: FORMER OWNER: DOLBY INTERNATIONAL AB, DP AMSTERDAM, NL |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230512 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: IT Payment date: 20230420 Year of fee payment: 10 Ref country code: FR Payment date: 20230420 Year of fee payment: 10 Ref country code: ES Payment date: 20230601 Year of fee payment: 10 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: NL Payment date: 20240418 Year of fee payment: 11 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20240419 Year of fee payment: 11 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20240418 Year of fee payment: 11 |