US12014743B2 - Spatial audio parameter merging - Google Patents
Spatial audio parameter merging Download PDFInfo
- Publication number
- US12014743B2 US12014743B2 US17/058,699 US201917058699A US12014743B2 US 12014743 B2 US12014743 B2 US 12014743B2 US 201917058699 A US201917058699 A US 201917058699A US 12014743 B2 US12014743 B2 US 12014743B2
- Authority
- US
- United States
- Prior art keywords
- metadata
- audio signal
- parameter
- signal
- audio
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005236 sound signal Effects 0.000 claims abstract description 390
- 238000000034 method Methods 0.000 claims description 43
- 230000004044 response Effects 0.000 claims 3
- 230000005540 biological transmission Effects 0.000 description 9
- 238000013461 design Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 230000015572 biosynthetic process Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 239000004065 semiconductor Substances 0.000 description 6
- 230000003416 augmentation Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 5
- 238000003786 synthesis reaction Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000008867 communication pathway Effects 0.000 description 1
- 239000004020 conductor Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000007274 generation of a signal involved in cell-cell signaling Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012732 spatial analysis Methods 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
- G10L19/16—Vocoder architecture
- G10L19/167—Audio streaming, i.e. formatting and decoding of an encoded audio signal representation into a data stream for transmission or storage purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R3/00—Circuits for transducers, loudspeakers or microphones
- H04R3/005—Circuits for transducers, loudspeakers or microphones for combining the signals of two or more microphones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
- H04S7/30—Control circuits for electronic adaptation of the sound field
- H04S7/302—Electronic adaptation of stereophonic sound system to listener position or orientation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
- H04S2400/03—Aspects of down-mixing multi-channel audio to configurations with lower numbers of playback channels, e.g. 7.1 -> 5.1
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
- H04S2400/13—Aspects of volume control, not necessarily automatic, in stereophonic sound systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
- H04S2400/15—Aspects of sound capture and related signal processing for recording or reproduction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S2420/00—Techniques used stereophonic systems covered by H04S but not provided for in its groups
- H04S2420/03—Application of parametric coding in stereophonic audio systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04S—STEREOPHONIC SYSTEMS
- H04S3/00—Systems employing more than two channels, e.g. quadraphonic
Definitions
- the present application relates to apparatus and methods for sound-field related parameter estimation in frequency bands, but not exclusively for time-frequency domain sound-field related parameter estimation for an audio encoder and decoder.
- Parametric spatial audio processing is a field of audio signal processing where the spatial aspect of the sound is described using a set of parameters.
- parameters such as directions of the sound in frequency bands, and the ratios between the directional and non-directional parts of the captured sound in frequency bands.
- These parameters are known to well describe the perceptual spatial properties of the captured sound at the position of the microphone array.
- These parameters can be utilized in synthesis of the spatial sound accordingly, for headphones binaurally, for loudspeakers, or to other formats, such as Ambisonics.
- the directions and direct-to-total energy ratios in frequency bands are thus a parameterization that is particularly effective for describing spatial audio.
- an apparatus comprising means for: determining for at least one first audio signal of an audio signal format, at least one metadata parameter; determining for at least one further audio signal of a further audio signal format; at least one further metadata parameter; controlling combining of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one further audio signal in such a way that the combined metadata comprises at least one spatial audio parameter.
- the means for determining for at least one first/further audio signal at least one metadata parameter may be further for at least one of: receiving at least one metadata parameter associated with the at least one first/further audio signal; analysing the at least one first/further audio signal to determine the at least one metadata parameter; and decoding the at least one first/further audio signal to determine the at least one metadata parameter.
- the means for controlling combining of the at least one metadata parameter associated with the first audio signal with the at least one metadata parameter associated with the further audio signal to generate a combined metadata may be for: extracting the at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the at least one first/further audio signal as a metadata block; adding the extracted at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the at least one first or further audio signal as a secondary metadata block within the metadata associated with the at least one further/first audio signal respectively.
- the means for extracting the at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the at least one first/further audio signal as a metadata block may be for extracting at least one of: at least one direction parameter; at least one energy ratio parameter; and at least one coherence parameter associated with the at least one first/further audio signal as the metadata block, and wherein adding the extracted at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the first/further audio signal as a secondary metadata block within the metadata associated with the at least one further/first audio signal respectively may be for: appending the at least one of: the at least one direction parameter; at least one energy ratio parameter; and at least one coherence parameter associated with the at least one first/further audio signal as the secondary metadata block within the metadata associated with the at least one further/first audio signal, the metadata associated with the at least one further/first audio signal may also comprise: at least one primary metadata block comprising at least one spatial audio parameter associated with the at least one further/first audio signal,
- the means for controlling combining of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata may be for: determining at least one first signal ratio associated with at least one metadata parameter associated with the at least one first audio signal; generating at least one first signal weight based on the at least one first signal ratio; determining at least one further ratio associated with at least one metadata parameter associated with the at least one further audio signal; generating at least one further signal weight based on the at least one further signal ratio; comparing the at least one first signal weight and the at least one further signal weight; and generating the combined metadata based on the comparing the at least one first signal weight and the at least one further signal weight.
- the means for generating the combined metadata based on the comparing the at least one first signal weight and the at least one further signal weight may be for: using the at least one metadata parameter associated with the at least one first audio signal as the combined metadata when the comparing indicates the at least one first signal weight is greater than the at least one further signal weight by a determined threshold; using the at least one metadata parameter associated with the at least one further audio signal as the combined metadata when the comparing indicates the at least one further signal weight is greater than the at least one first signal weight by a further determined threshold; generating a weighted average of the at least one metadata parameter associated with the at least one first audio signal and the at least one metadata parameter associated with the at least one further audio signal when the comparing indicates otherwise.
- the means for controlling combining of the at least one metadata parameter associated with the at least one first audio signal with the at least one metadata parameter associated with the at least one further audio signal to generate a combined metadata may be further for: determining at least one first signal energy associated with the at least one metadata parameter associated with the at least one first audio signal; determining at least one further signal energy associated with the at least one metadata parameter associated with the at least one further audio signal, wherein the means for generating at least one first signal weight based on the at least one first signal ratio is further based on the at least one first signal energy, wherein the means for generating at least one further signal weight based on the at least one further signal ratio is further based on the at least one further signal energy.
- the means for controlling combining of the at least one metadata parameter associated with the at least one first audio signal with the at least one metadata parameter associated with the further audio signal to generate a combined metadata may be further for: determining at least one first signal user input associated with the at least one metadata parameter associated with the at least one first audio signal; determining at least one further signal user input associated with the at least one metadata parameter associated with the at least one further audio signal, wherein the means for generating at least one first signal weight based on the at least one first signal ratio may be further based on the at least one first user input energy, wherein the means for generating at least one further signal weight based on the at least one further signal ratio may be further based on the at least one further signal user input.
- the means for controlling combining of the at least one metadata parameter associated with the at least one first audio signal with the at least one metadata parameter associated with the at least one further audio signal to generate a combined metadata may be further for: determining at least one first signal server input associated with the at least one metadata parameter associated with the at least one first audio signal; determining at least one further signal server input associated with the at least one metadata parameter associated with the at least one further audio signal, wherein the means for generating at least one first signal weight based on the at least one first signal ratio is further based on the at least one first server input energy, wherein the means for generating at least one further signal weight based on the at least one further signal ratio may be further based on the at least one further signal server input.
- the means for may be further for combining of the at least one metadata parameter associated with the at least one first audio signal with the at least one at least one metadata parameter associated with the at least one further audio signal to generate a combined metadata.
- the at least one first audio signal of an audio signal format may be at least one of: 2 ⁇ N channels of a spatial microphone array; 2 ⁇ N channels of multi-channel audio signal; a first order ambisonics signal; a higher order ambisonics signal; and a spatial audio signal.
- the at least one further audio signal of a further audio signal format may be at least one of: 2 ⁇ N channels of a spatial microphone array; 2 ⁇ N channels of multi-channel audio signal; a first order ambisonics signal; a higher order ambisonics signal; and a spatial audio signal.
- the means for controlling combining of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata may be such that the combined metadata comprises at least one spatial audio parameter and at least one non-spatial audio related parameter.
- an apparatus comprising at least one processor and at least one memory including a computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to: determine for at least one first audio signal of an audio signal format, at least one metadata parameter; determine for at least one further audio signal of a further audio signal format; at least one further metadata parameter; control a combination of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one further audio signal in such a way that the combined metadata comprises at least one spatial audio parameter.
- the apparatus caused to determine for at least one first/further audio signal at least one metadata parameter may be further caused to perform at least one of: receive at least one metadata parameter associated with the at least one first/further audio signal; analyse the at least one first/further audio signal to determine the at least one metadata parameter; and decode the at least one first/further audio signal to determine the at least one metadata parameter.
- the apparatus caused to control the combination of the at least one metadata parameter associated with the first audio signal with the at least one metadata parameter associated with the further audio signal to generate a combined metadata may be further caused to: extract the at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the at least one first/further audio signal as a metadata block; add the extracted at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the at least one first or further audio signal as a secondary metadata block within the metadata associated with the at least one further/first audio signal respectively.
- the apparatus caused to extract the at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the at least one first/further audio signal as a metadata block may be caused to extract at least one of: at least one direction parameter; at least one energy ratio parameter; and at least one coherence parameter associated with the at least one first/further audio signal as the metadata block, and wherein the apparatus caused to add the extracted at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the first/further audio signal as a secondary metadata block within the metadata associated with the at least one further/first audio signal respectively may be further caused to: append the at least one of: the at least one direction parameter; at least one energy ratio parameter; and at least one coherence parameter associated with the at least one first/further audio signal as the secondary metadata block within the metadata associated with the at least one further/first audio signal, the metadata associated with the at least one further/first audio signal may also comprise: at least one primary metadata block comprising at least one spatial audio parameter associated with the at least one
- the apparatus caused to control the combination of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata may be caused to: determine at least one first signal ratio associated with at least one metadata parameter associated with the at least one first audio signal; generate at least one first signal weight based on the at least one first signal ratio; determine at least one further ratio associated with at least one metadata parameter associated with the at least one further audio signal; generate at least one further signal weight based on the at least one further signal ratio; compare the at least one first signal weight and the at least one further signal weight; and generate the combined metadata based on the comparing the at least one first signal weight and the at least one further signal weight.
- the apparatus caused to generate the combined metadata based on the comparing the at least one first signal weight and the at least one further signal weight may be caused to: use the at least one metadata parameter associated with the at least one first audio signal as the combined metadata when the comparing indicates the at least one first signal weight is greater than the at least one further signal weight by a determined threshold; use the at least one metadata parameter associated with the at least one further audio signal as the combined metadata when the comparing indicates the at least one further signal weight is greater than the at least one first signal weight by a further determined threshold; generate a weighted average of the at least one metadata parameter associated with the at least one first audio signal and the at least one metadata parameter associated with the at least one further audio signal when the comparing indicates otherwise.
- the apparatus caused to control the combination of the at least one metadata parameter associated with the at least one first audio signal with the at least one metadata parameter associated with the at least one further audio signal to generate a combined metadata may be further caused to: determine at least one first signal energy associated with the at least one metadata parameter associated with the at least one first audio signal; determine at least one further signal energy associated with the at least one metadata parameter associated with the at least one further audio signal, wherein the apparatus caused to generate at least one first signal weight based on the at least one first signal ratio may be further based on the at least one first signal energy, wherein the apparatus caused to generate at least one further signal weight based on the at least one further signal ratio may be further based on the at least one further signal energy.
- the apparatus caused to control the combination of the at least one metadata parameter associated with the at least one first audio signal with the at least one metadata parameter associated with the further audio signal to generate a combined metadata may be further caused to: determine at least one first signal user input associated with the at least one metadata parameter associated with the at least one first audio signal; determine at least one further signal user input associated with the at least one metadata parameter associated with the at least one further audio signal, wherein the apparatus caused to generate at least one first signal weight based on the at least one first signal ratio may be further based on the at least one first user input energy, wherein the apparatus caused to generate at least one further signal weight based on the at least one further signal ratio may be further based on the at least one further signal user input.
- the apparatus caused to control the combination of the at least one metadata parameter associated with the at least one first audio signal with the at least one metadata parameter associated with the at least one further audio signal to generate a combined metadata may be caused to: determine at least one first signal server input associated with the at least one metadata parameter associated with the at least one first audio signal; determine at least one further signal server input associated with the at least one metadata parameter associated with the at least one further audio signal, wherein the apparatus caused to generate at least one first signal weight based on the at least one first signal ratio may be further based on the at least one first server input energy, wherein the apparatus caused to generate at least one further signal weight based on the at least one further signal ratio may be further based on the at least one further signal server input.
- the apparatus may be further caused to combine the at least one metadata parameter associated with the at least one first audio signal with the at least one at least one metadata parameter associated with the at least one further audio signal to generate a combined metadata.
- the at least one first audio signal of an audio signal format may be at least one of: 2 ⁇ N channels of a spatial microphone array; 2 ⁇ N channels of multi-channel audio signal; a first order ambisonics signal; a higher order ambisonics signal; and a spatial audio signal.
- the at least one further audio signal of a further audio signal format may be at least one of: 2 ⁇ N channels of a spatial microphone array; 2 ⁇ N channels of multi-channel audio signal; a first order ambisonics signal; a higher order ambisonics signal; and a spatial audio signal.
- the apparatus caused to control the combination of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata may be such that the combined metadata comprises at least one spatial audio parameter and at least one non-spatial audio related parameter.
- a method comprising: determining for at least one first audio signal of an audio signal format, at least one metadata parameter; determining for at least one further audio signal of a further audio signal format; at least one further metadata parameter; controlling combining of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one further audio signal in such a way that the combined metadata comprises at least one spatial audio parameter.
- Determining for at least one first/further audio signal at least one metadata parameter may further comprise at least one of: receiving at least one metadata parameter associated with the at least one first/further audio signal; analysing the at least one first/further audio signal to determine the at least one metadata parameter; and decoding the at least one first/further audio signal to determine the at least one metadata parameter.
- Controlling combining of the at least one metadata parameter associated with the first audio signal with the at least one metadata parameter associated with the further audio signal to generate a combined metadata may further comprise: extracting the at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the at least one first/further audio signal as a metadata block; adding the extracted at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the at least one first or further audio signal as a secondary metadata block within the metadata associated with the at least one further/first audio signal respectively.
- Extracting the at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the at least one first/further audio signal as a metadata block may further comprise extracting at least one of: at least one direction parameter; at least one energy ratio parameter; and at least one coherence parameter associated with the at least one first/further audio signal as the metadata block, and wherein adding the extracted at least one metadata parameter comprising at least one spatial audio parameter, the metadata parameter associated with the first/further audio signal as a secondary metadata block within the metadata associated with the at least one further/first audio signal respectively may further comprise: appending the at least one of: the at least one direction parameter; at least one energy ratio parameter; and at least one coherence parameter associated with the at least one first/further audio signal as the secondary metadata block within the metadata associated with the at least one further/first audio signal, the metadata associated with the at least one further/first audio signal may also comprise: at least one primary metadata block comprising at least one spatial audio parameter associated with the at least one further/first audio signal, the at least
- Controlling combining of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata may further comprise: determining at least one first signal ratio associated with at least one metadata parameter associated with the at least one first audio signal; generating at least one first signal weight based on the at least one first signal ratio; determining at least one further ratio associated with at least one metadata parameter associated with the at least one further audio signal; generating at least one further signal weight based on the at least one further signal ratio; comparing the at least one first signal weight and the at least one further signal weight; and generating the combined metadata based on the comparing the at least one first signal weight and the at least one further signal weight.
- Generating the combined metadata based on the comparing the at least one first signal weight and the at least one further signal weight may further comprise: using the at least one metadata parameter associated with the at least one first audio signal as the combined metadata when the comparing indicates the at least one first signal weight is greater than the at least one further signal weight by a determined threshold; using the at least one metadata parameter associated with the at least one further audio signal as the combined metadata when the comparing indicates the at least one further signal weight is greater than the at least one first signal weight by a further determined threshold; generating a weighted average of the at least one metadata parameter associated with the at least one first audio signal and the at least one metadata parameter associated with the at least one further audio signal when the comparing indicates otherwise.
- Controlling combining of the at least one metadata parameter associated with the at least one first audio signal with the at least one metadata parameter associated with the at least one further audio signal to generate a combined metadata may further comprise: determining at least one first signal energy associated with the at least one metadata parameter associated with the at least one first audio signal; determining at least one further signal energy associated with the at least one metadata parameter associated with the at least one further audio signal, wherein generating at least one first signal weight based on the at least one first signal ratio may be further based on the at least one first signal energy, wherein generating at least one further signal weight based on the at least one further signal ratio may be further based on the at least one further signal energy.
- Controlling combining of the at least one metadata parameter associated with the at least one first audio signal with the at least one metadata parameter associated with the further audio signal to generate a combined metadata may further comprise: determining at least one first signal user input associated with the at least one metadata parameter associated with the at least one first audio signal; determining at least one further signal user input associated with the at least one metadata parameter associated with the at least one further audio signal, wherein generating at least one first signal weight based on the at least one first signal ratio may be further based on the at least one first user input energy, wherein generating at least one further signal weight based on the at least one further signal ratio may be further based on the at least one further signal user input.
- Controlling combining of the at least one metadata parameter associated with the at least one first audio signal with the at least one metadata parameter associated with the at least one further audio signal to generate a combined metadata may further comprise: determining at least one first signal server input associated with the at least one metadata parameter associated with the at least one first audio signal; determining at least one further signal server input associated with the at least one metadata parameter associated with the at least one further audio signal, wherein generating at least one first signal weight based on the at least one first signal ratio may be further based on the at least one first server input energy, wherein generating at least one further signal weight based on the at least one further signal ratio may be further based on the at least one further signal server input.
- the method may further comprise combining of the at least one metadata parameter associated with the at least one first audio signal with the at least one at least one metadata parameter associated with the at least one further audio signal to generate a combined metadata.
- the at least one first audio signal of an audio signal format may be at least one of: 2 ⁇ N channels of a spatial microphone array; 2 ⁇ N channels of multi-channel audio signal; a first order ambisonics signal; a higher order ambisonics signal; and a spatial audio signal.
- the at least one further audio signal of a further audio signal format may be at least one of: 2 ⁇ N channels of a spatial microphone array; 2 ⁇ N channels of multi-channel audio signal; a first order ambisonics signal; a higher order ambisonics signal; and a spatial audio signal.
- Controlling combining of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata may be such that the combined metadata comprises at least one spatial audio parameter and at least one non-spatial audio related parameter.
- a computer program comprising instructions [or a computer readable medium comprising program instructions] for causing an apparatus to perform at least the following: determining for at least one first audio signal of an audio signal format, at least one metadata parameter; determining for at least one further audio signal of a further audio signal format; at least one further metadata parameter; controlling combining of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one further audio signal in such a way that the combined metadata comprises at least one spatial audio parameter.
- a non-transitory computer readable medium comprising program instructions for causing an apparatus to perform at least the following: determining for at least one first audio signal of an audio signal format, at least one metadata parameter; determining for at least one further audio signal of a further audio signal format; at least one further metadata parameter; controlling combining of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one further audio signal in such a way that the combined metadata comprises at least one spatial audio parameter.
- an apparatus comprising: first metadata determining circuitry configured to determine for at least one first audio signal of an audio signal format, at least one metadata parameter; further metadata determining circuitry configured to determine for at least one further audio signal of a further audio signal format; at least one further metadata parameter; controlling circuitry configured to control a combination of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one further audio signal in such a way that the combined metadata comprises at least one spatial audio parameter.
- a computer readable medium comprising program instructions for causing an apparatus to perform at least the following: determining for at least one first audio signal of an audio signal format, at least one metadata parameter; determining for at least one further audio signal of a further audio signal format; at least one further metadata parameter; controlling combining of the at least one metadata parameter with the at least one further metadata parameter to generate a combined metadata, wherein the combined metadata is configured to be associated with a combined audio signal formed from the at least one first audio signal and the at least one further audio signal in such a way that the combined metadata comprises at least one spatial audio parameter.
- An apparatus comprising means for performing the actions of the method as described above.
- An apparatus configured to perform the actions of the method as described above.
- a computer program comprising program instructions for causing a computer to perform the method as described above.
- a computer program product stored on a medium may cause an apparatus to perform the method as described herein.
- An electronic device may comprise apparatus as described herein.
- a chipset may comprise apparatus as described herein.
- Embodiments of the present application aim to address problems associated with the state of the art.
- FIG. 1 shows schematically an example combining scenario according to some embodiments
- FIG. 2 shows schematically an example pre-processor with mixer according to some embodiments
- FIG. 3 shows an example processor following combining according to some embodiments
- FIG. 4 shows schematically a first example of a mixer according to some embodiments
- FIG. 5 shows a flow diagram of the operation the first example of the mixer according to some embodiments
- FIG. 6 shows a flow diagram of the operation of the second example of a mixer according to some embodiments.
- FIG. 7 shows a flow diagram of the operation of the second example of a mixer according to some embodiments.
- FIG. 8 shows schematically the implementation of a mixer within a conference bridge example according to some embodiments
- FIG. 9 shows an example user input device control user interface according to some embodiments.
- FIG. 10 shows schematically an example device suitable for implementing the apparatus shown herein.
- parametric spatial audio can be used in various forms of telecommunications (for example these can include one-to-one immersive calls or telepresence, multi-user teleconferencing, etc.).
- these contexts it can often happen that there are multiple streams of parametric spatial audio.
- sources e.g., spatial microphone grid, premixed 5.1, etc.
- transmission of audio it is often desired that minimal bitrate is used. This is especially true in mobile communication where transmission channel can occasionally become very limited and efficient transmission is therefore important.
- the concept as discussed in the embodiments in further detail hereafter is to merge the streams (by mixing or some other combining method) before transmission.
- the embodiments achieve this without the need to convert the signals into a common non-parametric audio format (e.g., B-format) and thus are computationally simpler and avoid any quality decrease through repeated synthesis and analysis operations.
- the embodiments thus hereafter attempt to produce a direct mixing or combination of the parametric audio formats.
- the embodiments therefore describe apparatus and methods that combines multiple streams (for example, a multi-microphone capture spatial audio and channel-based audio) in the parametric domain.
- the combination can be implemented as part of the capture processing apparatus/operations and therefore create a combined audio representation that can be used, for example, as an audio codec input.
- the combination can be carried out inside the audio codec.
- the apparatus and operations are implemented outside the codec thus producing a single input stream. This has, for example, the advantage of reducing the number of streams to be encoded, which can reduce the associated codec mode selection and compression complexity.
- the audio is first analysed for spatial parameters and transport signals are generated (in other words the audio signal is converted to a suitable parametric format). Then, produced parametric signals are combined. This can be based on associated bitrate requirement.
- An example implementation which may use more bitrate is to combine the transport signals and add a new spatial metadata block into spatial metadata or modify the spatial metadata by adding, e.g., a second analysed direction.
- a bitrate reducing option is to fully combine the spatial metadata from the two sources in addition to combining the transport signals.
- user and/or the mixing (combining) service e.g., on the device, on a conference bridge, or in the network
- FIG. 1 is schematically presented an example combining scenario according to some embodiments.
- the combination is made for an interactive call wherein a call is initialised and then capture is started but in some embodiments the capture and/or audio background signals are started prior to the call being made or in a pure recording scenario the capture and audio signal selection made and the combination made independently of the making of a call.
- some embodiments can be implemented in a capture pre-processing system prior to encoding.
- the apparatus under the control of the user is configured to initiate or negotiate the call as shown in FIG. 1 by step 101 .
- An example of a call being negotiated (or one end of a video or audio-only capture being implemented) is shown in FIG. 1 by the bottom left image part which shows a user 153 operating a mobile device and attempting to capture the other user 151 using the cameras and microphones.
- step 103 The operation of capturing the first audio is shown in FIG. 1 by step 103 .
- the first audio is then processed to generate suitable transport audio signals and spatial metadata as shown in FIG. 1 by step 105 .
- the processed transport audio signals and spatial metadata may then be passed to a ‘direct’ mixer as discussed in further detail hereafter.
- the user 153 may be able to configure the apparatus to initialise an immersive audio augmentation as shown in FIG. 1 by step 111 .
- this may be activated based on interaction with user interface elements such as displayed icons to launch a regular mono call and an immersive call.
- a user can initiate an immersive call.
- the user initiating an immersive sharing may use a defined streaming service, which has at least immersive audio (for example the service may also have traditional video or VR video). This is then followed by the call negotiation 101 .
- audio capture 103 may be captured before the call is initiated, and is not transmitted to anyone prior to the call being established.
- Adding a background music 112 may then be implemented via the immersive audio call/augmentation app anytime during the call.
- the call can begin with background music added or the user may, for example, add the music 5 minutes into the call.
- the user may be able to configure the apparatus to add in a background music audio signal, for example a 5.0 channel audio signal 155 as shown in FIG. 1 by step 112 .
- This selection may in some embodiments be passed to the direct mixer configured to mix the captured audio (in the form of suitable transport audio signals and spatial metadata) and the background music audio signal.
- the background music audio signal (the second audio signal) may be retrieved or opened as shown in FIG. 1 by step 113 .
- the background audio signal (the second audio signal) may be processed to generate suitable second audio signal transport audio signals and spatial metadata and pass this to the mixer.
- the apparatus may be configured to determine whether the mixer is to directly mix the first and second audio signal as shown in FIG. 1 by step 107 .
- the apparatus is configured not to mix the first and second audio signals (including the example where there is only a first audio) then the first audio signal only is passed to the encoder for transmission where the apparatus is configured to mix the first and second audio signals then there is a direct mixing or combining of the transport audio signals and spatial metadata as shown in FIG. 1 by step 121 .
- the apparatus may then in some embodiments convert the mixed (or unmixed) transport audio signals and spatial metadata into a suitable input format for encoding and transmitting as shown in FIG. 1 by step 123 .
- the apparatus may then be configured to transmit (or store) the encoded bit-stream comprising the first and second audio signals as shown in FIG. 1 by step 125 .
- FIGS. 2 and 3 are shown an example mixer implementation according to some embodiments.
- the example implementation as shown in FIG. 2 presents the implementation in a pre-processing of audio and input format generation.
- pre-processing example all the mixing is performed prior to encoding (and prior to an encoder “seeing” the streams). With respect to an encoding viewpoint and implementation this is thus the most straightforward implementation.
- FIG. 2 therefore shows an example first audio input 201 configured to receive the first audio signal.
- the first audio input signal may for example be a captured, in other words microphone or microphone array audio signal comprising one or more channels of audio signals.
- the input signal may be one of any suitable formats of which the following examples describe the input signal being 2 ⁇ N channels of a spatial microphone array that is analysed with microphone array methods.
- suitable input signal formats include 2 ⁇ N channels of premixed multi-channel audio that are analysed with channel-based analysis methods, first order ambisonics (FOA) or higher order ambisonics (HOA) signals that are analysed from FOA using suitable methods or object based audio signals, that comprise direction metadata and may further comprise other spatial parameters.
- FOA first order ambisonics
- HOA higher order ambisonics
- any suitable audio signal may be input, for example a multichannel (surround sound) audio signal, a synthetic multichannel audio signal, an ambisonic audio signal, and/or an object based audio signal.
- the apparatus furthermore comprises a spatial processor 203 configured to receive the first audio signal and generate from the first audio signal a transport audio signal (which may in some embodiments be a downmix of the first audio signal, a selection of one or more channels of the audio signal, or any suitable rationalisation of the first audio signal).
- a spatial processor may be configured to generate suitable spatial metadata, such as direction parameters (which may include elevation, azimuth and/or distance), energy ratio parameters, and other suitable spatial parameters.
- the first audio signal parameters and the transport audio signal may be passed to the mixer 215 .
- FIG. 2 furthermore shows an example second audio input 211 configured to receive the second audio signal.
- the second audio input signal may for example be a multichannel (speaker) audio signal comprising two or more channels of audio signals.
- any suitable audio signal may be input, for example a multichannel microphone captured audio signal, a synthetic multichannel audio signal, an ambisonic audio signal, and/or an object based audio signal.
- the apparatus furthermore comprises a further spatial processor 213 configured to receive the second audio signal and generate from the second audio signal a transport audio signal (which may in some embodiments be a downmix of the second audio signal, a selection of one or more channels of the second audio signal, or any suitable rationalisation of the second audio signal).
- a transport audio signal which may in some embodiments be a downmix of the second audio signal, a selection of one or more channels of the second audio signal, or any suitable rationalisation of the second audio signal.
- the spatial processor may be configured to generate suitable spatial metadata, such as direction parameters (which may include elevation, azimuth and/or distance), energy ratio parameters, and other suitable spatial parameters.
- the second audio signal parameters and the transport audio signal may be passed to the mixer 215 .
- the apparatus comprises a mix controller 205 .
- the mix controller in some embodiments may be configured to control the mixing of the audio signals as described in further detail hereafter.
- the mix controller 205 may be configured to receive user inputs and/or the audio signals and/or parameters (for example energy parameters) based on the audio signals.
- the apparatus may furthermore comprise a mixer 215 .
- the mixer 215 may be configured to receive the first and second audio signals and furthermore the controls from the mix controller 205 and be configured to generate a mix of the two directly using the examples presented herein.
- the output of the mixer 215 may be passed to the encoder input 221 configured to receive the mixed audio signals and in some embodiments generate a suitable format to be processed by the encoder.
- the encoder 223 is configured to receive the mixed audio signals and generate suitable encoded audio signals (comprising the transport audio signal(s) and the combined metadata) and output it to a transmitter.
- the transmitter 225 may receive the encoded audio signal and transmit it or store it for further use.
- FIG. 3 is shown an example implementation wherein the mixing is implemented within the encoder, where at least one spatial metadata based input and at least one channel-based input can be combined into a single spatial metadata representation which may then be encoded for transmission.
- the mixing may be implemented at any suitable apparatus, for example within a server receiving or retrieving the audio signals and/or the decoder configured to receive the audio signals.
- the encoder may be configured in a suitable uplink format to communicate the captured audio signals in a partially or fully encoded form and an indication of the ‘background’ multichannel audio signal to be mixed (and/or the ‘background’ multichannel audio signal to be mixed). These audio signals are them mixed and encoded within the server and a in a suitable downlink format communicated to a renderer/decoder.
- the encoder may be configured to transmit the captured audio signals in a partially or fully encoded form and an indication of the ‘background’ multichannel audio signal to be mixed (and/or the ‘background’ multichannel audio signal to be mixed) to a decoder or renderer which may then mix and store the audio signals for later consumption or render the combined audio signals.
- a channel-based audio signal may require transport signal generation and metadata extraction (in other words a spatial analysis). This may then be followed by audio mixing and metadata mixing stages.
- a partial or full decoding of at least one audio input may be included in the encoder before the mixing and/or before the generation of the suitable transmission format.
- the apparatus in this embodiment comprises an encoder input 371 .
- the encoder input may be configured to receive a first audio signal.
- the first audio input signal may for example be a captured, in other words microphone or microphone array audio signal comprising two or more channels of audio signals.
- any suitable audio signal may be input, for example a multichannel (surround sound) audio signal, a synthetic multichannel audio signal, an ambisonic audio signal, and/or an object based audio signal.
- the encoder input 371 may furthermore be configured to receive a second audio signal.
- the second audio input signal may for example be a multichannel (speaker) audio signal comprising two or more channels of audio signals.
- any suitable audio signal may be input, for example a multichannel microphone captured audio signal, a synthetic multichannel audio signal, an ambisonic audio signal, and/or an object based audio signal.
- the encoder 373 may comprise an input analyser and classifier 301 .
- the input analyser and classifier 301 may be configured to receive the first and second audio signals and determine whether it is required to generate spatial metadata associated with the first and second audio signals, whether the first or second audio signal is to be partially or fully decoded, and furthermore generate controls for the mixing of the first and second audio signals.
- the encoder 373 may furthermore comprise a partial/full decoder 303 configured when controlled to partially or fully decode an audio signal and pass the decoded audio signal to the audio mixer 307 and/or the transport signal generator and metadata generator 305 .
- the encoder may furthermore comprise a transport signal generator and metadata generator 305 .
- the transport signal generator and metadata generator 305 may be configured to receive the first and/or second audio signals and/or the partially/fully decoded first and/or second audio signals and from these generate suitable transport audio signals (which may in some embodiments be a downmix, a selection of one or more channels of the audio signal, or any suitable rationalisation of the audio signal).
- the transport signal generator and metadata generator 305 may be configured to generate suitable spatial metadata, such as direction parameters (which may be elevation, azimuth and distance), energy ratio parameters, and other suitable spatial parameters for the first and/or second audio signals.
- the encoder comprises an audio mixer 307 configured to mix the audio signals and pass these to the core audio encoder 311 .
- the encoder may furthermore comprise a core audio encoder 311 configured to receive the mixed audio signals and generate a suitable mono/stereo or other core audio coding.
- the encoder comprises a metadata mixer 309 configured to mix or merge the metadata associated with the first and second audio signals.
- the encoder may furthermore comprise a metadata compressor 313 the metadata compressor may receive the mixed or merged metadata and perform a suitable metadata compression prior to transmission/storage.
- the apparatus furthermore may comprise a transmitter/storage 375 configured to receive the compressed audio and metadata and transmit and/or store it in the form of payload bitstream 321 or similar.
- a first example may be one in which metadata is combined by adding an additional analyzed direction block to the spatial metadata. This requires that the direction metadata format matches on both directional metadata streams or a conversion has to be done. Transport signals may in some embodiments also be combined but can be transmitted separately. This method may be able to produce high quality and require relatively low complexity at the potential expense of that the bitrate is high if channel signals are sent separately.
- the simplest form of combining two streams, shown in FIG. 4 as metadata stream 1 401 comprising a common metadata part 403 , which comprises a directional metadata block 405 and metadata stream 2 411 comprising common metadata part 413 and directional metadata block 415 is to take the direction metadata block from one spatial metadata stream and add it into the other spatial metadata stream as a secondary direction metadata block.
- the mixer 421 receives the metadata stream 1 401 and metadata stream 2 411 and outputs the combined metadata stream 421 .
- the combined metadata stream comprises the common metadata part 423 (which may be one or other or a combination of the common metadata parts 403 , 413 ) and which further comprises the directional metadata block 405 associated with the metadata stream 1 and the directional metadata block 415 associated with the metadata stream 2.
- This process can be implemented so that it requires no or only partial decoding of the spatial metadata and thus may be computationally very efficient.
- the transport signals can be combined in a separate process or and in some other embodiments the transport signals are transmitted forward or stored without combination.
- step 502 Another operation of receiving stream 2 as shown in FIG. 5 by step 502 .
- the next operation is that of extracting the direction metadata block from stream 1 (or from stream 2 in some embodiments) and then inserting or adding it into the metadata as a secondary directional metadata block into stream 2 (or into stream 1 in the same other embodiments) as shown in FIG. 5 by step 503 .
- next operation is one of outputting the stream 2 with the added directional metadata block from stream 1 as the secondary directional metadata block (or in the same other embodiments outputting the stream 1 with the added directional metadata block from stream 2 as the secondary directional metadata block) as shown in FIG. 5 by step 505 .
- a second example of direct mixing may be one in which metadata is mixed with the help of available channels signals. All underlying transport signals are either decoded or are available prior encoding and are thus known to the metadata mixer.
- the advantage of this method is potentially even higher perceptual quality, and the transport signals can be mixed at the same time for further transmission, which saves bandwidth downstream.
- this method may require higher computational complexity in the mixer. Transport signals are analyzed, and in some scenarios, they have to be decoded as an extra step. This method may also produce additional delays if additional decoding/encoding is done for the spatial metadata mixing, which may happen for example in a conference bridge.
- this method is shown with respect to FIG. 6 .
- stream 1 With respect to the first stream, stream 1, and for all sub-frames and frequency bands the following operations may be performed by suitable means.
- any user preference for example via a suitable user interface or user input for stream 1 as shown in FIG. 6 by step 607 .
- any server preference for example via a suitable server input for stream 1 as shown in FIG. 6 by step 609 .
- step 605 The following operation is then to determine a weight-value w 1 for stream 1 as shown in FIG. 6 by step 605 and which may be determined as: signal energy multiplied with signal energy ratio (and with the option of combination with the user or server mixing weighting).
- stream 2 With respect to the second stream, stream 2, and for all sub-frames and frequency bands the following operations may be performed by suitable means.
- any user preference for example via a suitable user interface or user input for stream 2 as shown in FIG. 6 by step 617 .
- any server preference for example via a suitable server input for stream 2 as shown in FIG. 6 by step 619 .
- step 615 The following operation is then to determine a weight-value w 2 for stream 2 as shown in FIG. 6 by step 615 and which may be determined as: signal energy multiplied with signal energy ratio (and with the option of combination with the user or server mixing weighting).
- N is number of time samples in this time frame
- K b and K t are the current frequency band bottom and top frequency bins, and/is the number of transport channels for this stream.
- S(i,k,n) is the time-frequency domain representation of the transport signal.
- weights as discussed herein are then used to decide how the metadata is combined. If the weight of signal 1 is larger than the weight of signal 2 by the determined threshold, the metadata from the larger weight signal is used for the mixed signal. The same may be applied if signal 2 has a larger weight than signal 1. Furthermore as discussed above where the weights are close, a weighted average of the parameters is used. In some embodiments the weighted average of the parameters may be determined as follows:
- m m ⁇ i ⁇ x ( i , k ) ⁇ m 1 ( i , k ) , w 1 > c ⁇ w 2 m 2 ( i , k ) , w 2 > c ⁇ w 1 m 1 ⁇ w 1 + m 2 ⁇ w 2 w 1 + w 2 , otherwise
- m(i,k) signifies any single parameter in metadata and c is a predefined limit constant (e.g., 1 . 5 ). Note that this limit constant allows transition between strict selection between the two input metadata and weighted average between the two input metadata. With suitable selection of c, it is possible to use just one of these operating modes if desired.
- a third example may be one in which metadata is mixed without access to transport signals.
- the transport signals are either both transmitted forwards, or are mixed totally independently of the spatial metadata mixing.
- the advantage of this is that spatial metadata mixing is computationally very light and requires no additional delay.
- the perceptual quality of this mixing method is reasonable but may not be as good as the previous method.
- This method may be useful in some apparatus types such as a conference bridge.
- the potentially lower combined audio quality may not be crucial, e.g., in a teleconference bridge mixing implementation, since participants generally try to avoid talking over each other in teleconferences.
- the two (or more)N-channel+metadata streams are mixed together with no access to the transport signals and therefore the energy values.
- This method is shown with respect to FIG. 7 .
- stream 1 With respect to the first stream, stream 1, and for all sub-frames and frequency bands the following operations may be performed by suitable means.
- any user preference for example via a suitable user interface or user input for stream 1 as shown in FIG. 7 by step 707 .
- any server preference for example via a suitable server input for stream 1 as shown in FIG. 7 by step 709 .
- step 705 determines a weight-value w 1 for stream 1 as shown in FIG. 7 by step 705 and which may be determined as signal energy ratio (and with the option of combination with the user or server mixing weighting).
- stream 2 With respect to the second stream, stream 2, and for all sub-frames and frequency bands the following operations may be performed by suitable means.
- any user preference for example via a suitable user interface or user input for stream 2 as shown in FIG. 7 by step 717 .
- any server preference for example via a suitable server input for stream 2 as shown in FIG. 7 by step 719 .
- step 715 The following operation is then to determine a weight-value w 2 for stream 2 as shown in FIG. 7 by step 715 and which may be determined as: signal energy multiplied with signal energy ratio (and with the option of combination with the user or server mixing weighting).
- this method applies mixing of metadata similar to the second method but with the difference of the formation of the weight values as energy is not available now.
- the apparatus may be implemented as part of a conference bridge equipment.
- FIG. 8 for example show such an implementation.
- a server 801 which may be a centralised or cloud or distributed entity configured to perform the mixing and then pass the output to the participant apparatus such as user equipment 811 , computer 831 and other apparatus 821 such as virtual reality or augmented reality or mixed reality capture/display apparatus or a conference phone equipment.
- the participants may pass the video and audio signals to the participant inputs 851 .
- the participant inputs may be passed to an uplink interface 853 .
- the participant inputs may furthermore be encoded.
- the uplink interface 853 may then pass the audio signals from the participants and any other audio signals to be mixed (for example selected background audio signals) to an input decoder and audio deriver 855 and to an input activity detector 857 . These may respectively decode and/or retrieve and then output the audio and determine spatial audio parameters and pass this to a stream selector 859 .
- the stream selector may select the streams to mix and pass them to the audio encoder and mixer 861 configured to mix the stream transport (or other suitable format) audio signals and the metadata combiner or mixer 863 which is configured to mix the spatial metadata according to the embodiments described herein.
- the audio output from the audio encoder and mixer and the spatial metadata from the metadata combiner and mixer 863 may be passed to the downlink interface 865 which outputs the combined audio and metadata to the participants via the participant output 867 .
- a user may be able to perform two actions, first initiating an immersive audio augmentation or immersive audio call where the user is recording at least immersive audio (and may also record video) of himself and a friend in an interesting environment.
- the user secondly adds a background music track or any other second audio (e.g., a 5.0 studio mix).
- the experience thus now corresponds to the situation where the capture point is, in addition to recording the real environment, surrounded by a virtual loudspeaker setup providing background music.
- the user may add the background track by triggering the service on the device to open a second audio (file), process the second audio (into the negotiated format), and to combine the processed second audio with the first audio into one common spatial metadata based representation.
- This combined audio is fed to the codec, encoded, and transmitted.
- the addition of the background music does not significantly increase the codec processing overhead, require any re-negotiation, or result in discontinuity or other perceived artefact by the recipient.
- the apparatus may be configured to provide a signal to the encoder about the role of each audio stream.
- Such signals may in some embodiments also be included in the payload and thus allow, e.g., independent volume control (at least to some degree) for the recipient.
- the selection and mixing of the streams may be performed optionally based on user input or preferences.
- FIG. 9 shows an example user interface providing inputs for mixing.
- These position controls 905 may control the orientation and location of the multichannel inputs.
- FIG. 9 shows a plan position 909 showing the difference between the captured content in the centre of the plan position 909 and the multichannel (speaker) audio content surrounding the centre and enable a rotation of one stream relative to the other.
- FIG. 9 shows an elevation view 907 showing the difference between the captured content in the centre of the elevation view 907 and the multichannel (speaker) audio content surrounding the centre and enable a tilt and height adjustment of one stream relative to the other.
- FIG. 9 shows a slider control 903 for adjusting the volume of one stream (the captured audio 911 ) relative to the other stream (the multichannel audio file 913 ).
- examples of user control of the alignment and volume level between the captured content (which can be considered a fixed stream configuration relative to the capture device) and the content from at least one file (which can be considered a virtual stream configuration relative to the capture device) can be implemented using any suitable user input interface or device.
- the relative distance of the audio file virtual loudspeaker sources e.g., a 5.0 or a 7.1 configuration
- This can be of particular interest for any parametric configuration that includes a distance parameter and, of course, for any use cases implementing six degrees of freedom.
- some inputs need not be (at least fully) decoded.
- the mixing of audio for the downlink may not always require audio encoding. For example, this is the case if the uplink audio streams have not been decoded (into waveforms).
- the combination of the independent metadata based streams may take into account the information related to the audio signals, or the mixing of the metadata may be done independently.
- the device may be any suitable electronics device or apparatus.
- the device 1400 is a mobile device, user equipment, tablet computer, computer, audio playback apparatus, etc.
- the device 1400 comprises at least one processor or central processing unit 1407 .
- the processor 1407 can be configured to execute various program codes such as the methods such as described herein.
- the device 1400 comprises a memory 1411 .
- the at least one processor 1407 is coupled to the memory 1411 .
- the memory 1411 can be any suitable storage means.
- the memory 1411 comprises a program code section for storing program codes implementable upon the processor 1407 .
- the memory 1411 can further comprise a stored data section for storing data, for example data that has been processed or to be processed in accordance with the embodiments as described herein. The implemented program code stored within the program code section and the data stored within the stored data section can be retrieved by the processor 1407 whenever needed via the memory-processor coupling.
- the device 1400 comprises a user interface 1405 .
- the user interface 1405 can be coupled in some embodiments to the processor 1407 .
- the processor 1407 can control the operation of the user interface 1405 and receive inputs from the user interface 1405 .
- the user interface 1405 can enable a user to input commands to the device 1400 , for example via a keypad.
- the user interface 1405 can enable the user to obtain information from the device 1400 .
- the user interface 1405 may comprise a display configured to display information from the device 1400 to the user.
- the user interface 1405 can in some embodiments comprise a touch screen or touch interface capable of both enabling information to be entered to the device 1400 and further displaying information to the user of the device 1400 .
- the device 1400 comprises an input/output port 1409 .
- the input/output port 1409 in some embodiments comprises a transceiver.
- the transceiver in such embodiments can be coupled to the processor 1407 and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network.
- the transceiver or any suitable transceiver or transmitter and/or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.
- the transceiver can communicate with further apparatus by any suitable known communications protocol.
- the transceiver or transceiver means can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA).
- UMTS universal mobile telecommunications system
- WLAN wireless local area network
- IRDA infrared data communication pathway
- the transceiver input/output port 1409 may be configured to receive the loudspeaker signals and in some embodiments determine the parameters as described herein by using the processor 1407 executing suitable code. Furthermore the device may generate a suitable transport signal and parameter output to be transmitted to the synthesis device.
- the device 1400 may be employed as at least part of the synthesis device.
- the input/output port 1409 may be configured to receive the transport signals and in some embodiments the parameters determined at the capture device or processing device as described herein, and generate a suitable audio signal format output by using the processor 1407 executing suitable code.
- the input/output port 1409 may be coupled to any suitable audio output for example to a multichannel speaker system and/or headphones or similar.
- circuitry may refer to one or more or all of the following:
- circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and/or firmware.
- circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.
- the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof.
- some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto.
- firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto.
- While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
- the embodiments of this invention may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware.
- any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions.
- the software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD.
- the memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory.
- the data processors may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), gate level circuits and processors based on multi-core processor architecture, as non-limiting examples.
- Embodiments of the inventions may be practiced in various components such as integrated circuit modules.
- the design of integrated circuits is by and large a highly automated process.
- Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
- Programs such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre-stored design modules.
- the resultant design in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or “fab” for fabrication.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Otolaryngology (AREA)
- Stereophonic System (AREA)
Abstract
Description
w=rEavg p user p server
where r is the energy ratio parameter, puser is the user preference weight and and pserver is the server preference weight. As mentioned, user and server preferences are an optional inclusion.
w=rp user p server
-
- (a) hardware-only circuit implementations (such as implementations in only analogue and/or digital circuitry) and
- (b) combinations of hardware circuits and software, such as (as applicable):
- (i) a combination of analogue and/or digital hardware circuit(s) with software/firmware and
- (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and
- (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.
Claims (20)
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
GB1808929 | 2018-05-31 | ||
GB1808929.2A GB2574238A (en) | 2018-05-31 | 2018-05-31 | Spatial audio parameter merging |
GB1808929.2 | 2018-05-31 | ||
PCT/FI2019/050413 WO2019229299A1 (en) | 2018-05-31 | 2019-05-29 | Spatial audio parameter merging |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210210104A1 US20210210104A1 (en) | 2021-07-08 |
US12014743B2 true US12014743B2 (en) | 2024-06-18 |
Family
ID=62872748
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/058,699 Active US12014743B2 (en) | 2018-05-31 | 2019-05-29 | Spatial audio parameter merging |
Country Status (5)
Country | Link |
---|---|
US (1) | US12014743B2 (en) |
EP (1) | EP3803858A4 (en) |
CN (1) | CN112513981A (en) |
GB (1) | GB2574238A (en) |
WO (1) | WO2019229299A1 (en) |
Families Citing this family (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2582749A (en) * | 2019-03-28 | 2020-10-07 | Nokia Technologies Oy | Determination of the significance of spatial audio parameters and associated encoding |
GB2590651A (en) | 2019-12-23 | 2021-07-07 | Nokia Technologies Oy | Combining of spatial audio parameters |
GB2590650A (en) * | 2019-12-23 | 2021-07-07 | Nokia Technologies Oy | The merging of spatial audio parameters |
GB2590913A (en) * | 2019-12-31 | 2021-07-14 | Nokia Technologies Oy | Spatial audio parameter encoding and associated decoding |
GB2598932A (en) * | 2020-09-18 | 2022-03-23 | Nokia Technologies Oy | Spatial audio parameter encoding and associated decoding |
US11662975B2 (en) | 2020-10-06 | 2023-05-30 | Tencent America LLC | Method and apparatus for teleconference |
GB2608406A (en) * | 2021-06-30 | 2023-01-04 | Nokia Technologies Oy | Creating spatial audio stream from audio objects with spatial extent |
WO2023088560A1 (en) | 2021-11-18 | 2023-05-25 | Nokia Technologies Oy | Metadata processing for first order ambisonics |
GB2617055A (en) * | 2021-12-29 | 2023-10-04 | Nokia Technologies Oy | Apparatus, Methods and Computer Programs for Enabling Rendering of Spatial Audio |
GB202215632D0 (en) | 2022-10-21 | 2022-12-07 | Nokia Technologies Oy | Generating parametric spatial audio represntations |
GB202215617D0 (en) | 2022-10-21 | 2022-12-07 | Nokia Technologies Oy | Generating parametric spatial audio representations |
GB2624890A (en) | 2022-11-29 | 2024-06-05 | Nokia Technologies Oy | Parametric spatial audio encoding |
GB2624874A (en) | 2022-11-29 | 2024-06-05 | Nokia Technologies Oy | Parametric spatial audio encoding |
CN117499850B (en) * | 2023-12-26 | 2024-05-28 | 荣耀终端有限公司 | Audio data playing method and electronic equipment |
GB202405792D0 (en) | 2024-04-25 | 2024-06-12 | Nokia Technologies Oy | Signalling of pass-through mode in spatial audio coding |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2111062A1 (en) | 2008-04-16 | 2009-10-21 | LG Electronics, Inc. | A method and an apparatus for processing an audio signal |
US20090262957A1 (en) * | 2008-04-16 | 2009-10-22 | Oh Hyen O | Method and an apparatus for processing an audio signal |
CN102007533A (en) | 2008-04-16 | 2011-04-06 | Lg电子株式会社 | A method and an apparatus for processing an audio signal |
EP2324645A1 (en) | 2008-08-13 | 2011-05-25 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Apparatus for merging spatial audio streams |
CN103119846A (en) | 2010-09-22 | 2013-05-22 | 杜比实验室特许公司 | Audio stream mixing with dialog level normalization |
US20130216070A1 (en) | 2010-11-05 | 2013-08-22 | Florian Keiler | Data structure for higher order ambisonics audio data |
US20150245153A1 (en) | 2014-02-27 | 2015-08-27 | Dts, Inc. | Object-based audio loudness management |
CA2967249A1 (en) * | 2014-11-28 | 2016-06-02 | Sony Corporation | Transmission device, transmission method, reception device, and reception method |
US20170251321A1 (en) * | 2014-09-25 | 2017-08-31 | Dolby Laboratories Licensing Corporation | Insertion of Sound Objects Into a Downmixed Audio Signal |
GB2549532A (en) | 2016-04-22 | 2017-10-25 | Nokia Technologies Oy | Merging audio signals with spatial metadata |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105074818B (en) * | 2013-02-21 | 2019-08-13 | 杜比国际公司 | Audio coding system, the method for generating bit stream and audio decoder |
-
2018
- 2018-05-31 GB GB1808929.2A patent/GB2574238A/en not_active Withdrawn
-
2019
- 2019-05-29 EP EP19812566.8A patent/EP3803858A4/en active Pending
- 2019-05-29 US US17/058,699 patent/US12014743B2/en active Active
- 2019-05-29 CN CN201980050110.XA patent/CN112513981A/en active Pending
- 2019-05-29 WO PCT/FI2019/050413 patent/WO2019229299A1/en unknown
Patent Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2111062A1 (en) | 2008-04-16 | 2009-10-21 | LG Electronics, Inc. | A method and an apparatus for processing an audio signal |
US20090262957A1 (en) * | 2008-04-16 | 2009-10-22 | Oh Hyen O | Method and an apparatus for processing an audio signal |
CN102007533A (en) | 2008-04-16 | 2011-04-06 | Lg电子株式会社 | A method and an apparatus for processing an audio signal |
EP2324645A1 (en) | 2008-08-13 | 2011-05-25 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Apparatus for merging spatial audio streams |
US20110216908A1 (en) * | 2008-08-13 | 2011-09-08 | Giovanni Del Galdo | Apparatus for merging spatial audio streams |
CN103119846A (en) | 2010-09-22 | 2013-05-22 | 杜比实验室特许公司 | Audio stream mixing with dialog level normalization |
US20130170672A1 (en) * | 2010-09-22 | 2013-07-04 | Dolby International Ab | Audio stream mixing with dialog level normalization |
EP2619904A1 (en) | 2010-09-22 | 2013-07-31 | Dolby Laboratories Licensing Corporation | Audio stream mixing with dialog level normalization |
US20130216070A1 (en) | 2010-11-05 | 2013-08-22 | Florian Keiler | Data structure for higher order ambisonics audio data |
US20150245153A1 (en) | 2014-02-27 | 2015-08-27 | Dts, Inc. | Object-based audio loudness management |
US20170251321A1 (en) * | 2014-09-25 | 2017-08-31 | Dolby Laboratories Licensing Corporation | Insertion of Sound Objects Into a Downmixed Audio Signal |
CA2967249A1 (en) * | 2014-11-28 | 2016-06-02 | Sony Corporation | Transmission device, transmission method, reception device, and reception method |
GB2549532A (en) | 2016-04-22 | 2017-10-25 | Nokia Technologies Oy | Merging audio signals with spatial metadata |
WO2017182714A1 (en) | 2016-04-22 | 2017-10-26 | Nokia Technologies Oy | Merging audio signals with spatial metadata |
US20190132674A1 (en) | 2016-04-22 | 2019-05-02 | Nokia Technologies Oy | Merging Audio Signals with Spatial Metadata |
US10477311B2 (en) | 2016-04-22 | 2019-11-12 | Nokia Technologies Oy | Merging audio signals with spatial metadata |
Non-Patent Citations (4)
Title |
---|
ETSI TS 101 154 V2.4.1 (Feb. 2018), "Digital Video Broadcasting (DVB); Specification for the use of Video and Audio Coding in Broadcast and Broadband Applications", EBU Operating Eurovision, 291 pgs. |
Laitinen, et al. "Converting 5.1 Audio Recordings to B-Format for Directional Audio Coding Reproduction", IEEE ICASSP, pp. 61-64, IDS, May, (Year: 2011). * |
Laitinen, Mikko-Ville, et al., "Converting 5.1 Audio Recordings to B-Format for Directional Audio Coding Reproduction", IEEE ICASSP, May 2011, pp. 61-64. |
Office Action dated Jan. 5, 2024 in European Patent Application No. 19 812 566.8, 7 pages. |
Also Published As
Publication number | Publication date |
---|---|
GB201808929D0 (en) | 2018-07-18 |
EP3803858A1 (en) | 2021-04-14 |
EP3803858A4 (en) | 2022-03-16 |
WO2019229299A1 (en) | 2019-12-05 |
CN112513981A (en) | 2021-03-16 |
US20210210104A1 (en) | 2021-07-08 |
GB2574238A (en) | 2019-12-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US12014743B2 (en) | Spatial audio parameter merging | |
US20240007814A1 (en) | Determination Of Targeted Spatial Audio Parameters And Associated Spatial Audio Playback | |
US20110002469A1 (en) | Apparatus for Capturing and Rendering a Plurality of Audio Channels | |
WO2020152394A1 (en) | Audio representation and associated rendering | |
US20230047237A1 (en) | Spatial audio parameter encoding and associated decoding | |
KR20210072736A (en) | Converting audio signals captured in different formats to a reduced number of formats to simplify encoding and decoding operations. | |
US20210250717A1 (en) | Spatial audio Capture, Transmission and Reproduction | |
US20200015028A1 (en) | Energy-ratio signalling and synthesis | |
CN112673649A (en) | Spatial audio enhancement | |
US20230085918A1 (en) | Audio Representation and Associated Rendering | |
US11483669B2 (en) | Spatial audio parameters | |
EP3824464B1 (en) | Controlling audio focus for spatial audio processing | |
US11729574B2 (en) | Spatial audio augmentation and reproduction | |
US20240029745A1 (en) | Spatial audio parameter encoding and associated decoding | |
US20230188924A1 (en) | Spatial Audio Object Positional Distribution within Spatial Audio Communication Systems | |
US20240236611A9 (en) | Generating Parametric Spatial Audio Representations | |
US20230197087A1 (en) | Spatial audio parameter encoding and associated decoding | |
WO2023066456A1 (en) | Metadata generation within spatial audio | |
US20240196158A1 (en) | Surround sound to immersive audio upmixing based on video scene analysis | |
WO2024012805A1 (en) | Transporting audio signals inside spatial audio signal |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: NOKIA TECHNOLOGIES OY, FINLAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LAAKSONEN, LASSE;RAMO, ANSSI;LAITINEN, MIKKO-VILLE;AND OTHERS;REEL/FRAME:054514/0204 Effective date: 20190612 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING TC RESP., ISSUE FEE NOT PAID |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING TC RESP, ISSUE FEE PAYMENT RECEIVED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING TC RESP, ISSUE FEE PAYMENT VERIFIED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |