US20190057694A1 - Speech/Dialog Enhancement Controlled by Pupillometry - Google Patents
Speech/Dialog Enhancement Controlled by Pupillometry Download PDFInfo
- Publication number
- US20190057694A1 US20190057694A1 US15/998,796 US201815998796A US2019057694A1 US 20190057694 A1 US20190057694 A1 US 20190057694A1 US 201815998796 A US201815998796 A US 201815998796A US 2019057694 A1 US2019057694 A1 US 2019057694A1
- Authority
- US
- United States
- Prior art keywords
- audio signal
- speech
- headset
- cognitive load
- eye
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 230000001149 cognitive effect Effects 0.000 claims abstract description 115
- 230000005236 sound signal Effects 0.000 claims abstract description 112
- 238000000034 method Methods 0.000 claims abstract description 107
- 230000001404 mediated effect Effects 0.000 claims abstract description 24
- 238000012545 processing Methods 0.000 claims abstract description 23
- 238000009877 rendering Methods 0.000 claims abstract description 12
- 230000010344 pupil dilation Effects 0.000 claims description 42
- 230000000193 eyeblink Effects 0.000 claims description 13
- 238000007781 pre-processing Methods 0.000 claims description 10
- 210000001747 pupil Anatomy 0.000 description 8
- 238000009499 grossing Methods 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 210000004556 brain Anatomy 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005562 fading Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000036760 body temperature Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 238000010924 continuous production Methods 0.000 description 1
- 238000000537 electroencephalography Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/24—Speech recognition using non-acoustical features
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
- G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0316—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude
- G10L21/0364—Speech enhancement, e.g. noise reduction or echo cancellation by changing the amplitude for improving intelligibility
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/78—Detection of presence or absence of voice signals
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L2015/226—Procedures used during a speech recognition process, e.g. man-machine dialogue using non-speech characteristics
- G10L2015/227—Procedures used during a speech recognition process, e.g. man-machine dialogue using non-speech characteristics of the speaker; Human-factor methodology
Definitions
- the present disclosure relates to methods of processing audio signals.
- the present disclosure relates to methods of processing audio signals that selectively apply speech enhancement and particularly dialog enhancement to (portions of) the audio signals.
- the present disclosure also relates to corresponding apparatus, such as decoders, for example.
- the present disclosure is particularly, though not exclusively, applicable to the AC-4 codec.
- audio codecs such as AC-4
- dialog enhancement can be guided by dialog enhancement parameters that are provided in the encoded bitstream (e.g. AC-4 bitstream).
- dialog enhancement on the overall may also depend on the individual listener (e.g., their hearing abilities), the individual listening conditions, and the complexity of the audio scene in question.
- dialog detector speech detector
- speech detector may not be fully accurate, which might result in un-boosted dialog and boosted non-dialog, at the cost of understandability of dialog and/or overall perceived audio quality.
- the evaluation of the quality of dialog enhancement both subjectively and objectively, is a complicated task without universally applicable solution, so that reliable tuning algorithms for dialog enhancement are still wanting.
- a method of processing an audio signal (e.g., a decoded audio signal, such as an audio signal decoded from a bitstream, for example) is described.
- the method may be said to be a method of selectively applying speech/dialog enhancement to portions of the audio signal.
- the method may include obtaining a measure of a cognitive load of a listener that listens to a rendering (e.g., a rendered representation) of the audio signal.
- the method may further include determining whether speech/dialog enhancement shall be applied based on the obtained measure of the cognitive load.
- the method may yet further include performing speech/dialog enhancement based on (e.g., in accordance with) the determination.
- speech/dialog enhancement may be performed only if it is determined that speech/dialog enhancement shall be applied. For example, it may be determined that speech/dialog enhancement shall be applied if the cognitive load of the listener is found to be above a predefined threshold for the cognitive load.
- Performing speech/dialog enhancement may involve applying speech/dialog enhancement to a portion (e.g., frame) of the audio signal, for example the portion that is currently rendered.
- the proposed method is based on the fact that the cognitive load (i.e., how loaded the brain is) of a listener increases if the listener has problems to understand speech/dialog in a portion of rendered audio.
- speech/dialog enhancement can be applied in a manner that is specifically tailored to the needs of the listener. That is, speech/dialog enhancement can be applied only when the listener has problems understanding speech/dialog.
- speech/dialog enhancement does not have to rely on potentially inaccurate speech/dialog detection. In consequence, unnecessary speech/dialog enhancement, which could degrade the overall perceived quality of the rendered audio, can be avoided, while it can be ensured that the listener is aided in understanding speech/dialog where this is needed.
- obtaining the measure of the cognitive load may be based on information indicative of a pupil dilation (e.g., pupil diameter) of the listener.
- the method may be said to include monitoring the pupil dilation of the listener.
- obtaining the measure of the cognitive load of the listener may be based on information indicative of a pupil dilation of the listener.
- the pupil dilation (e.g., pupil diameter) of a listener is a good indicator of the cognitive load (i.e., how loaded the brain is) of a listener that listens to an audio signal.
- the pupil dilation e.g., pupil diameter
- the measure of the cognitive load can also be based on other techniques such as Electroencephalography, monitoring the body temperature, or a push button that can be actuated by the listener.
- the pupil dilation e.g., pupil diameter
- the pupil dilation is above a given threshold, this indicates high cognitive load of the listener, which in turn can be seen as an indication that speech/dialog enhancement shall be applied. This enables selective application of speech/dialog enhancement, specifically tailored to the listener's needs.
- obtaining the measure of the cognitive load may be based on eye-tracking data for the listener.
- obtaining the measure of the cognitive load may involve analyzing the eye-tracking data.
- the eye-tracking data may be data that is obtained by an eye-tracking device, such as an eye-tracking device of a virtual reality (VR) headset (or augmented reality (AR)/mixed AR+VR reality headset) or a TV set with eye tracking, for example. Since eye tracking data is available for many media applications (e.g., for gesture-controlled TV sets or in AR environments, VR environments, and mixed AR+VR environments), this allows for a particularly simple implementation of the proposed method, without incurring the need for additional hardware.
- VR virtual reality
- AR augmented reality
- obtaining the measure of the cognitive load may involve determining information indicative of a pupil dilation (e.g., pupil diameter) of the listener based on the eye-tracking data, for example by analyzing the eye-tracking data.
- the method may be said to include monitoring the pupil dilation of the listener.
- obtaining the measure of the cognitive load of the listener may be based on information indicative of a pupil dilation of the listener.
- the method may further include pre-processing of the eye-tracking data (i.e., a processing of the eye-tracking data before determining information indicative of the pupil dilation based on the eye-tracking data).
- pre-processing of the eye-tracking data may involve removing portions of the eye-tracking data that relate to eye blinks.
- the pre-processing may further involve interpolation and/or smoothing between timings before (e.g., immediately before or a certain time interval before) and after (e.g., immediately after, or a certain time interval after) eye blinks. Thereby, an accuracy of the determination of the pupil dilation can be improved.
- the method may further include applying a heuristic model to the eye-tracking data for determining the cognitive load of the listener.
- the heuristic model can also be applied to data indicative of the pupil dilation.
- the heuristic model may evolve over time, for example to adapt to a specific user. Evolution of the heuristic model over time may involve machine learning (learning over time).
- the method may further include receiving side-information (e.g., metadata) associated with the audio signal.
- the side-information may include information on whether speech/dialog is present in the audio signal.
- the side-information may be conveyed in a bitstream, i.e., the side-information may be received from the bitstream.
- the determination of whether speech/dialog enhancement shall be applied may be further based on the received information.
- the side-information may have been obtained at an encoder performing speech/dialog detection (e.g., in the context of speech/dialog enhancement).
- speech/dialog enhancement shall be applied for a given portion of the audio signal if the side-information indicates that speech/dialog is present in the given portion of the audio signal and if the measure of the cognitive load for the given portion of the audio signal satisfies a predetermined condition.
- the predetermined condition may relate to whether the cognitive load for the given portion of the audio signal exceeds a predetermined threshold value (e.g., whether the pupil dilation of the listener is found to be above a predefined threshold).
- the determination of whether speech/dialog enhancement shall be applied may be performed for each of a plurality of consecutive portions of the audio signal.
- the method may further involve smoothing a degree of speech/dialog enhancement between consecutive portions of the audio signal, e.g., by fading in and/or fading out speech/dialog enhancement. Thereby, audible artifacts that might result from abrupt starting or ending speech/dialog enhancement can be avoided.
- the determination of whether speech/dialog enhancement shall be applied may be performed for each of a plurality of channels.
- the side-information may be channel-specific and may indicate for each channel whether speech/dialog is present.
- the method may include obtaining a measure of a cognitive load of a listener that listens to a rendering of the audio signal.
- the method may further include determining a degree of speech/dialog enhancement (e.g., a magnitude of a gain for speech/dialog enhancement) that shall be applied based on the obtained measure of the cognitive load.
- the method may yet further include performing speech/dialog enhancement based on the determination.
- the method may additionally include receiving side-information associated with the audio signal.
- the side-information may include information on gains for speech/dialog enhancement of the audio signal.
- the side information may include, for each portion (e.g., frame) of the audio signal, a plurality of gains for speech/dialog enhancement from which a selection can be made. Then, the determination may be further based on the received side-information.
- the decoder may include a processor and a memory coupled to the processor.
- the processor may be adapted to perform the operations outlined in the present disclosure.
- the memory may store instructions for causing the processor to perform the operations outlined in the present disclosure.
- the software program may be adapted for execution on a processor and for performing the method steps outlined in the present disclosure when carried out on a computing device.
- the storage medium may include a program (software program) adapted for execution on a processor and for performing the method steps outlined in the present disclosure when carried out on a computing device.
- the computer program may include executable instructions for performing the operations outlined in the present disclosure when executed on a computer.
- a headset for computer-mediated reality e.g., AR headset, VR headset, or mixed AR+VR headset
- the headset may include a processor and a memory coupled to the processor.
- the processor may be adapted to obtain eye-tracking data of a wearer of the headset.
- the processor may be further adapted to determine a measure of a cognitive load of the wearer of the headset based on the eye-tracking data.
- the processor may be yet further adapted to output an indication of the cognitive load of the wearer of the headset.
- the indication may be output for example to a decoder for processing an audio signal (e.g., a decoded audio signal, such as an audio signal decoded from a bitstream, for example).
- the indication may enable the decoder to determine whether speech/dialog enhancement shall be applied to a current portion of an audio signal that is rendered by the decoder.
- the memory may store instructions for causing the processor to perform the operations according to this aspect.
- a method of operating a headset for computer-mediated reality e.g., AR headset, VR headset, or mixed AR+VR headset
- the method may include obtaining eye-tracking data of a wearer of the headset.
- the method may further include determining a measure of a cognitive load of the wearer of the headset based on the eye-tracking data.
- the method may yet further include outputting an indication of the cognitive load of the wearer of the headset.
- the headset may operate in conjunction (e.g., in cooperation) with the decoder.
- the headset may transmit information, such as an indication of a cognitive load of a wearer of the headset and/or eye-tracking data, to the decoder.
- the headset may comprise a processor and a memory coupled to the processor.
- the processor may be adapted to obtain eye-tracking data of a wearer of the headset.
- the processor may be further adapted to determine a measure of a cognitive load of the wearer of the headset based on the eye-tracking data.
- the processor may be yet further adapted to output an indication of the cognitive load of the wearer of the headset to the decoder.
- the decoder may be adapted to render the audio signal for listening by the wearer of the headset.
- the decoder may be further adapted to receive the indication of the cognitive load of the wearer of the headset from the headset.
- the decoder may be further adapted to determine whether speech/dialog enhancement shall be applied based on the indication of the cognitive load of the wearer of the headset.
- the decoder may be yet further adapted to perform speech/dialog enhancement based on the determination.
- a method of operating a system of a decoder for processing an audio signal e.g., a decoded audio signal, such as an audio signal decoded from a bitstream, for example
- a headset for computer-mediated reality e.g., AR headset, VR headset, or mixed AR+VR headset
- the method may include, at the headset, obtaining eye-tracking data of a wearer of the headset.
- the method may further include, at the headset, determining a measure of a cognitive load of the wearer of the headset based on the eye-tracking data.
- the method may yet further include, at the headset, outputting an indication of the cognitive load of the wearer of the headset to the decoder.
- the method may include, at the decoder, rendering the audio signal for listening by the wearer of the headset.
- the method may further include, at the decoder, receiving the indication of the cognitive load of the wearer of the headset from the headset.
- the method may further include, at the decoder, determining whether speech/dialog enhancement shall be applied based on the indication of the cognitive load of the wearer of the headset.
- the method may yet further include, at the decoder, performing speech/dialog enhancement based on the determination.
- FIG. 1 schematically illustrates an example of an encoding/decoding environment in which speech/dialog enhancement may be applied
- FIG. 2 schematically illustrates an example of an encoding/decoding environment according to embodiments of the disclosure
- FIG. 3 is a flow chart schematically illustrating an example of a method according to embodiments of the disclosure
- FIG. 4 is a flow chart schematically illustrating another example of a method according to embodiments of the disclosure.
- FIG. 5 is a flow chart schematically illustrating yet another example of a method according to embodiments of the disclosure.
- FIG. 1 schematically illustrates an example of an encoding/decoding environment 100 in which speech/dialog enhancement may be applied.
- the environment can be seen as being split into an encoder side (to the left of the dashed vertical line) and a decoder side (to the right of the dashed vertical line).
- an encoder 10 receives input audio (input audio signal) 11 .
- the input audio 11 may be non-enhanced audio, such as audio to which speech/dialog enhancement has not been applied.
- the input audio 11 is also provided to a speech/dialog enhancement analysis block 20 .
- separate speech/dialog tracks may be provided to the speech/dialog enhancement analysis block 20 as well.
- the separate speech/dialog tracks 12 may be in alignment (time alignment) with the input audio 11 .
- the encoder 10 encodes the input audio 11 and outputs encoded audio 13 .
- the encoded audio 13 may be provided to the decoder side. As an intermediate step, the encoded audio 13 may be stored on a storage medium.
- the speech/dialog enhancement analysis block 20 analyzes the input audio 11 and/or the separate speech/dialog tracks 12 . Based on this analysis, the speech/dialog enhancement analysis block 20 determines whether speech/dialog is present for each given portion (e.g., frame) of the input audio 11 . The speech/dialog enhancement analysis block 20 may further determine parameters for speech/dialog enhancement of respective portions of the input audio 11 .
- the speech/dialog enhancement analysis block 20 outputs side-information 14 (e.g., dedicated side-information for speech/dialog enhancement) that includes an indication (e.g., flag) for each portion of the input audio 11 of whether the respective portion includes speech/dialog.
- side-information 14 may further include the parameters for speech/dialog enhancement. These parameters may provide the decoder side with instructions on how to apply the speech/dialog enhancement for a given portion of the audio.
- a decoder e.g., core decoder
- receives the encoded audio 13 On the decoder side, a decoder (e.g., core decoder) 50 receives the encoded audio 13 .
- the decoder 50 decodes the encoded audio 13 and outputs decoded audio 51 .
- a speech/dialog enhancement block (speech/dialog synthesis block) 70 receives the side-information 14 and boosts portions of the decoded audio 51 by means of a gain unit (booster) 60 .
- the speech/dialog enhancement block 70 controls the gain unit 60 in accordance with the side-information 14 , i.e., applies speech/dialog enhancement to those portions of the audio that are flagged as including speech/dialog by the side-information 14 .
- the side-information 14 may further specify the manner in which speech/dialog enhancement is applied, e.g., a gain that is applied to the respective portion of the audio.
- the gain unit 60 outputs output audio 52 that may be rendered to one or more speakers for listening.
- Both the encoder-side processing and the decoder-side processing may be performed on a channel-by-channel basis, i.e., for each of a plurality of channels.
- Speech/dialog enhancement may be only applied to those channels (and to those portions of the audio) that include speech/dialog.
- speech/dialog enhancement is performed (i.e., a boost is applied to the decoded audio 51 ) whenever speech/dialog is detected in the relevant portion of the audio. Whether or not this speech/dialog enhancement improves the perceived audio quality may depend on the individual listener, the listening conditions, or the relevant audio scene.
- Methods and apparatus according to the present disclosure are based on the finding that the pupil dilation of a human's eye is a measure of how loaded the brain is (i.e., a measure of a cognitive load). Methods and apparatus according to the present disclosure are further based on the finding that a listener's cognitive load increases under difficult speech reception conditions, i.e., if the listener has problems understanding speech. In other words, a listener's increase in cognitive effort is reflected in an increased pupil dilation (e.g., pupil diameter). Based on these finding, the present disclosure proposes to use a listener's pupil dilation (which reflects the listener's cognitive load) to control speech/dialog enhancement (e.g., boosting of speech/dialog). Information on the pupil dilation may be obtained from an eye-tracking device that operates in conjunction with an audio decoder, for example.
- FIG. 2 An example of an encoding/decoding environment 200 according to embodiments of the disclosure is schematically illustrated in FIG. 2 .
- the encoder side (to the left of the dashed vertical line) is identical to the encoder side in the encoder/decoder environment 100 of FIG. 1 .
- the decoder side (to the right of the dashed vertical line) differs from the decoder side in the encoder/decoder environment 100 of FIG. 1 in that a gain unit 60 ′ (a modified gain unit) is provided that is controlled by the speech/dialog enhancement block 70 and a cognitive load determination block 80 .
- the gain unit 60 ′ outputs (modified) output audio 52 ′.
- the cognitive load determination block 80 receives eye-tracking data 81 from an eye-tracking device 90 .
- the eye-tracking data may be time series data that indicates, for each of a plurality of consecutive instances of time, one or more characteristics (parameters) of the eye of a user 95 .
- the eye-tracking device 90 may be in visual contact with the user 95 and may be part of, for example, a headset for computer-mediated reality (e.g., AR headset, VR headset, or mixed AR+VR headset) or a TV set having an eye-tracking capability.
- the user 95 may listen to a rendering of the output audio 52 ′.
- the cognitive load determination block 80 determines a cognitive load of the user 95 , by analyzing the eye-tracking data.
- the cognitive load determination block 80 may analyze the eye-tracking data to determine a pupil dilation (e.g., pupil diameter) of the user 95 , and may determine the cognitive load of the user 95 based on the determined pupil dilation.
- the gain unit 60 ′ is controlled by the speech/dialog enhancement block 70 and the cognitive load determination unit 80 .
- speech/dialog enhancement by boosting the respective portion of the audio may be applied if the side-information 14 indicates that speech/dialog is present, and if the determined cognitive load indicates that the user 95 has problems understanding said speech/dialog.
- speech/dialog enhancement may be applied based on the determined cognitive load alone, i.e., without reference to the side-information 14 , or at least without reference to the indication of whether speech/dialog is present in the respective portion of the audio.
- the proposed methods and apparatus selectively apply speech/dialog enhancement (i.e., selectively boost or do not boost speech/dialog) based on a listener's effort in understanding the speech/dialog.
- FIG. 3 illustrates an example of a method 300 of processing an audio signal (e.g., a decoded audio signal, such as an audio signal decoded from a bitstream, for example) and for selectively applying speech/dialog enhancement to (portions of) the audio signal.
- the method 300 may be performed by a decoder, for example.
- the method may comprise an initial step of decoding the audio signal from a bitstream.
- a measure of a cognitive load of a listener that listens to a rendering of the audio signal is obtained.
- the measure of the cognitive load can be obtained (e.g., determined) based on information indicative of a pupil dilation (e.g., pupil diameter) of an eye of the listener.
- step S 310 may be said to be a step of monitoring a pupil dilation of the listener.
- obtaining the measure of the cognitive load may be based on information indicative of a pupil dilation of the listener.
- obtaining the measure of the cognitive load may be based on eye-tracking data for the listener.
- the eye-tracking data may be used for determining information indicative of the pupil dilation of the listener. This may involve analyzing the eye-tracking data.
- the eye-tracking data may be obtained by an eye-tracking device, such as an eye-tracking device of a headset for computer-mediated reality (e.g., AR headset, VR headset, or mixed AR+VR headset) with eye tracking, or a TV set with eye tracking.
- Suitable eye-tracking devices and consumer devices capable of eye tracking e.g., headsets for computer-mediated reality, TV sets
- a headset for computer-mediated reality is understood to be any headset that can present visual information to a wearer of the headset to overlay or replace the wearer's view of a real scene.
- the eye-tracking data may be analyzed using a heuristic model for determining the cognitive load of the listener.
- the heuristic model may be applied to the information indicative of the pupil dilation, to determine the cognitive load of the listener.
- the heuristic model may evolve over time. Evolution over time may involve machine learning (learning over time).
- the heuristic model may be adapted to a specific user (listener), and may evolve over time to improve the adaptation.
- the eye-tracking data may be pre-processed before obtaining the information indicative of the pupil dilation and/or the measure of the cognitive load of the listener (i.e., before analyzing the eye-tracking data).
- Pre-processing of the eye-tracking data may involve removing portions of the eye-tracking data that relate to eye blinks (e.g., removing eye blinks).
- Eye blinks may be detected by an automatic classification algorithm, for example, or using a heuristic measure. When using the heuristic measure, an eye blink may be detected, for example, if the pupil dilation is below a certain threshold.
- a portion of the eye-tracking data relating to the detected eye blink, optionally together with portions relating to a certain interval of time before and after the detected eye blink may be removed from the eye-tracking data.
- the pre-processing may further include interpolation between timings before and after the detected (or removed) eye blink (e.g., between a latest timing before a time interval relating to the removed portion of the eye-tracking data and an earliest timing after this time interval).
- the pre-processing may further include smoothing (e.g., over time) of the eye-tracking data after removing eye blinks (or after subsequent interpolation).
- step S 320 it is determined whether speech/dialog enhancement shall be applied based on the obtained measure of the cognitive load. For example, it may be determined that speech/dialog enhancement shall be applied if the cognitive load of the listener is found to be above a predefined threshold for the cognitive load. Likewise, it may be determined that speech/dialog enhancement shall be applied if the pupil dilation of the listener is found to be above a predefined threshold for the pupil dilation. The rationale behind this determination is that a high cognitive load (which would be indicated by an enlarged pupil dilation) can be indicative of the listener having problems understanding speech or speech/dialog.
- the method 300 may further include receiving side-information (metadata) associated with the audio signal, wherein the side-information includes information on whether speech/dialog is present in the audio signal. That is, the side-information may include, for each of a plurality of consecutive portions (e.g., frames) of the audio signal, and indication (e.g., flag) of whether that portion of the audio signal contains speech/dialog.
- the side-information may have been obtained at an encoder for enabling or facilitating speech/dialog enhancement at the decoder.
- the determination at step S 320 may be further based on the side-information (e.g., based on the aforementioned indication of whether speech/dialog is present).
- speech/dialog enhancement shall be applied for a given portion (e.g., frame) of the audio signal if the side-information indicates that speech/dialog is present in the given portion of the audio signal and if the measure of the cognitive load for the given portion of the audio signal satisfies a predetermined condition.
- the predetermined condition may be satisfied if the cognitive load for the given portion of the audio signal exceeds a predetermined threshold value, for example.
- the predetermined condition may be deemed satisfied if the pupil dilation of the listener is found to be above a predefined threshold.
- speech/dialog enhancement is performed based on the determination (e.g., in accordance with the determination).
- the speech/dialog enhancement may be applied to a current portion (e.g., frame) of the audio signal (a portion that is currently rendered), or to the next portion of the audio signal.
- speech/dialog enhancement may be selectively performed for each of a plurality of successive portions (e.g., frames) of the audio signal.
- the method may further comprise rendering the audio signal.
- the side-information may include parameters for speech/dialog enhancement. These parameters may provide the decoder side with instructions on how to apply the speech/dialog enhancement.
- the side-information may indicate, for each portion of the audio signal that contains speech/dialog, a gain for application to the respective portion of the audio. This gain then could be applied at step S 330 if it is determined at step S 320 that speech/dialog enhancement shall be applied (i.e., the speech/dialog enhancement may be performed in accordance with the parameters for speech/dialog enhancement included in the side-information for the relevant portion of the audio signal).
- the side-information may also contain more than one gain for each portion of the audio signal that contains speech/dialog.
- a first gain may be selected and applied if it is determined at step S 320 that speech/dialog enhancement shall be applied (i.e., if the measure of the cognitive load exceeds the predefined threshold value) but the measure of the cognitive load is still found to be below a predefined second threshold (that is larger than the predefined threshold).
- a second gain (that is larger than the first gain) could be applied if the measure of the cognitive load is found to be above the predefined second threshold. It is understood that this scheme can be extended to an arbitrary number of gains and predefined thresholds.
- a degree of speech/dialog enhancement that is performed at step S 330 may depend on the measure of the cognitive load of the listener.
- the steps of method 300 may be performed for each of a plurality of consecutive portions (e.g., frames) of the audio signal.
- the determination of whether speech/dialog enhancement shall be applied may be performed for each of a plurality of consecutive portions of the audio signal, and the speech/dialog enhancement may be selectively applied to each portion of the audio signal in accordance with a result of the determination for that portion of the audio signal.
- the method may further include a step of smoothing a degree of the speech/dialog enhancement (e.g., a gain that is applied when boosting the speech/dialog) between consecutive portions of the audio signal.
- the steps of method 300 may be performed for each of a plurality of channels that form the audio signal.
- method 300 may be performed by a decoder.
- decoder may comprise a processor and a memory coupled to the processor.
- the processor may be adapted to perform each of steps S 310 , S 320 , and S 330 described above.
- the memory may store respective instructions for causing the processor to perform these steps.
- the decoder may operate in conjunction (e.g., in cooperation) with a headset for computer-mediated reality.
- a headset for computer-mediated reality e.g., AR headset, VR headset, or mixed AR+VR headset
- the headset for computer-mediated reality may be seen as a non-limiting example of a device for obtaining eye-tracking data.
- FIG. 4 illustrates an example of a method 400 of operating such headset. Without intended limitation, the remainder of the present disclosure makes reference to a VR headset. It is understood that any other headset for computer-mediated reality could be used as well in the embodiments described below.
- step S 410 eye-tracking data of a wearer of the VR headset is obtained. This may be done using an eye-tracking device of the VR headset.
- a measure of a cognitive load of the wearer of the VR headset is determined based on the eye-tracking data. This may be done in the manner described with reference to step S 310 above.
- the wearer of the VR headset is assumed to be the listener that listens to the rendering of the audio signal.
- an indication of the cognitive load of the wearer of the VR headset is output.
- the indication of the cognitive load of the wearer of the VR headset may be output to the decoder.
- the output indication may enable the decoder to determine whether speech/dialog enhancement shall be applied to a current portion of an audio signal that is rendered by the decoder, for example in the manner described with reference to step S 320 above.
- the VR headset (as an example of a headset for computer-mediated reality, or more generally, an example of device for obtaining eye-tracking data) may directly output the eye-tracking data to the decoder.
- the decoder may then determine the measure of the cognitive load of the wearer of the VR headset, for example in the manner described with reference to step S 310 above.
- Steps S 410 to S 430 may be continuously performed. For example, each of these steps may be performed once every given time interval. For example, each of these steps may be performed once per portion (e.g., frame) of the audio signal.
- a corresponding VR headset may comprise a processor and a memory coupled to the processor.
- the processor may be adapted to perform each of steps S 410 , S 420 , and S 430 described above.
- the memory may store respective instructions for causing the processor to perform these steps.
- the VR headset may further comprise an optical device (e.g., a camera) for capturing a visual representation of the wearer's eye or eyes, from which the eye-tracking data can be derived.
- a decoder and a VR headset (as an example of a headset for computer-mediated reality, or more generally, an example of device for obtaining eye-tracking data) operating in conjunction with the decoder may form a system for processing the audio signal.
- FIG. 5 illustrates an example of a method 500 of operating such system.
- step S 510 eye-tracking data of a wearer of the VR headset is obtained at the VR headset. This may be performed in the manner described above with reference to step S 410 .
- a measure of a cognitive load of the wearer of the VR headset is determined at the VR headset based on the eye-tracking data. This may be performed in the manner described above with reference to step S 420 (or step S 310 ).
- an indication of the cognitive load of the wearer of the VR headset to the decoder is output by the VR headset. This may be performed in the manner described above with reference to step S 430 .
- Steps S 510 to S 530 may be continuously performed.
- steps S 540 to S 560 described below may be continuously performed. For example, each of these steps may be performed once every given time interval. For example, each of these steps may be performed once per portion (e.g., frame) of the audio signal.
- the audio signal is rendered by the decoder for listening by the wearer of the VR headset. As noted above, this may be a continuous process.
- step S 550 the indication of the cognitive load of the wearer of the VR headset from the VR headset is received at the decoder.
- steps S 540 and S 550 may be performed in any order.
- step S 560 it is determined by the decoder whether speech/dialog enhancement shall be applied based on the indication of the cognitive load of the wearer of the VR headset. This may be performed in the manner described above with reference to step S 320 . The determination may be further based on side-information associated with the audio signal, as described above.
- speech/dialog enhancement is performed by the decoder based on (e.g., in accordance with) the determination. This may be performed in the manner described above with reference to step S 330 .
- the VR headset may directly transmit the eye-tracking data to the decoder.
- step S 520 would be performed at the decoder, i.e., the decoder would determine the measure of the cognitive load of the wearer of the VR headset based on the eye-tracking data, for example in the manner described above with reference to step S 310 .
- a decoder for use in this system may comprise a processor and a memory coupled to the processor.
- the processor may be adapted to perform each of steps S 540 , S 550 , S 560 , and S 570 described above.
- the memory may store respective instructions for causing the processor to perform these steps.
- a VR headset for use in the system may comprise a processor and a memory coupled to the processor.
- the processor may be adapted to perform each of steps S 510 , S 520 , and S 530 described above.
- the memory may store respective instructions for causing the processor to perform these steps.
- the VR headset may further comprise an optical device (e.g., a camera) for capturing a visual representation of the wearer's eye or eyes, from which the eye-tracking data can be derived.
- the methods and apparatus described in the present disclosure may be implemented as software, firmware and/or hardware. Certain components may e.g. be implemented as software running on a digital signal processor or microprocessor. Other components may e.g. be implemented as hardware and or as application specific integrated circuits.
- the signals encountered in the described methods and apparatus may be stored on media such as random-access memory or optical storage media. They may be transferred via networks, such as radio networks, satellite networks, wireless networks or wireline networks, e.g. the Internet.
- EEEs enumerated example embodiments
- EEE 1 A method of processing a decoded audio signal, the method comprising:
- EEE 2 The method of EEE 1, further comprising receiving side-information associated with the audio signal, the side-information including information on whether speech/dialog is present in the audio signal, wherein the determination is further based on the received side-information.
- EEE 3 The method of EEE 2, wherein it is determined that speech/dialog enhancement shall be applied for a given portion of the audio signal if the side-information indicates that speech/dialog is present in the given portion of the audio signal and if the measure of the cognitive load for the given portion of the audio signal satisfies a predetermined condition.
- EEE 4 The method of EEE 1, wherein it is determined that speech/dialog enhancement shall be applied if the cognitive load of the listener is found to be above a predefined threshold for the cognitive load.
- EEE 5 The method of EEE 1, wherein obtaining the measure of the cognitive load is based on information indicative of a pupil dilation of the listener.
- EEE 6 The method of EEE 5, wherein it is determined that speech/dialog enhancement shall be applied if the pupil dilation of the listener is found to be above a predefined threshold for the pupil dilation.
- EEE 7 The method of EEE 1, wherein obtaining the measure of the cognitive load is based on eye-tracking data for the listener.
- EEE 8 The method of EEE 7, wherein obtaining the measure of the cognitive load involves determining information indicative of a pupil dilation of the listener based on the eye-tracking data.
- EEE 9 The method of EEE 7, further comprising pre-processing of the eye-tracking data.
- EEE 10 The method of EEE 9, wherein pre-processing the eye-tracking data involves removing portions of the eye-tracking data that relate to eye blinks.
- EEE 11 The method of EEE 7, further comprising applying a heuristic model to the eye-tracking data for determining the cognitive load of the listener.
- EEE 12 The method of EEE 1, wherein the determination of whether speech/dialog enhancement shall be applied is performed for each of a plurality of consecutive portions of the audio signal.
- EEE 13 The method of EEE 1, wherein the determination of whether speech/dialog enhancement shall be applied is performed for each of a plurality of channels.
- a method of processing a decoded audio signal comprising:
- EEE 15 The method of EEE 14, further comprising receiving side-information associated with the audio signal, the side-information including information on gains for speech/dialog enhancement of the audio signal, wherein the determination is further based on the received information.
- a decoder for processing an audio signal comprising a processor and a memory coupled to the processor, wherein the processor is adapted to perform the operations of EEE 1.
- EEE 17 A program for causing a computer to perform the operations of EEE 1 when performed on the computer.
- EEE 18 A computer-readable storage medium storing a program for causing a computer to perform the operations of EEE 1 when performed on the computer.
- a headset for computer-mediated reality comprising a processor and a memory coupled to the processor, wherein the processor is adapted to:
- EEE 20 A method of operating a headset for computer-mediated reality, the method comprising:
- EEE 21 A system of the decoder of EEE 16 and the headset for computer-mediated reality of EEE 19, the headset operating in conjunction with the decoder.
- EEE 22 A system of a decoder for processing an audio signal and a headset for computer-mediated reality operating in conjunction with the decoder, the headset comprising a processor and a memory coupled to the processor,
- processor is adapted to:
- decoder is adapted to:
- EEE 23 A method of operating a system of a decoder for processing an audio signal and a headset for computer-mediated reality operating in conjunction with the decoder,
- the method comprises, at the headset:
- the method further comprises, at the decoder:
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Theoretical Computer Science (AREA)
- Acoustics & Sound (AREA)
- Computational Linguistics (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Health & Medical Sciences (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The present disclosure relates to methods for processing a decoded audio signal and for selectively applying speech/dialog enhancement to the decoded audio signal. The present disclosure also relates to a method of operating a headset for computer-mediated reality. A method of processing a decoded audio signal comprises obtaining a measure of a cognitive load of a listener that listens to a rendering of the audio signal, determining whether speech/dialog enhancement shall be applied based on the obtained measure of the cognitive load, and performing speech/dialog enhancement based on the determination. A method of operating a headset for computer-mediated reality comprises obtaining eye-tracking data of a wearer of the headset, determining a measure of a cognitive load of the wearer of the headset based on the eye-tracking data, and outputting an indication of the cognitive load of the wearer of the headset. The present disclosure further relates to corresponding apparatus and systems, and to methods of operating such apparatus and systems.
Description
- The present application claims priority to U.S. Provisional Patent Application No. 62/546,640, filed Aug. 17, 2017, which is incorporated herein by reference in its entirety.
- The present disclosure relates to methods of processing audio signals. In particular, the present disclosure relates to methods of processing audio signals that selectively apply speech enhancement and particularly dialog enhancement to (portions of) the audio signals. The present disclosure also relates to corresponding apparatus, such as decoders, for example. The present disclosure is particularly, though not exclusively, applicable to the AC-4 codec.
- State of the art audio codecs, such as AC-4, for example, can use post-processing techniques such as dialog enhancement to boost dialog when it is detected. This dialog enhancement can be guided by dialog enhancement parameters that are provided in the encoded bitstream (e.g. AC-4 bitstream).
- However, in some situations simply boosting any dialog may do more harm to the perceived audio quality than it adds to the understandability or intelligibility of dialog. To complicate matters, whether dialog enhancement on the overall is beneficial may also depend on the individual listener (e.g., their hearing abilities), the individual listening conditions, and the complexity of the audio scene in question. Another issue that state-of-the-art dialog enhancement techniques are facing is that the dialog detector (speech detector) may not be fully accurate, which might result in un-boosted dialog and boosted non-dialog, at the cost of understandability of dialog and/or overall perceived audio quality. Also the evaluation of the quality of dialog enhancement, both subjectively and objectively, is a complicated task without universally applicable solution, so that reliable tuning algorithms for dialog enhancement are still wanting.
- Thus, there is a need for an improved method and apparatus for processing media content and for applying dialog enhancement. There is further need for such method and apparatus that enable selective application of dialog enhancement.
- According to an aspect of the disclosure, a method of processing an audio signal (e.g., a decoded audio signal, such as an audio signal decoded from a bitstream, for example) is described. The method may be said to be a method of selectively applying speech/dialog enhancement to portions of the audio signal. The method may include obtaining a measure of a cognitive load of a listener that listens to a rendering (e.g., a rendered representation) of the audio signal. The method may further include determining whether speech/dialog enhancement shall be applied based on the obtained measure of the cognitive load. The method may yet further include performing speech/dialog enhancement based on (e.g., in accordance with) the determination. That is, speech/dialog enhancement may be performed only if it is determined that speech/dialog enhancement shall be applied. For example, it may be determined that speech/dialog enhancement shall be applied if the cognitive load of the listener is found to be above a predefined threshold for the cognitive load. Performing speech/dialog enhancement may involve applying speech/dialog enhancement to a portion (e.g., frame) of the audio signal, for example the portion that is currently rendered.
- The proposed method is based on the fact that the cognitive load (i.e., how loaded the brain is) of a listener increases if the listener has problems to understand speech/dialog in a portion of rendered audio. Thus, by obtaining a measure of the cognitive load of the listener and selectively applying speech/dialog enhancement in dependence on the measure of the cognitive load, speech/dialog enhancement can be applied in a manner that is specifically tailored to the needs of the listener. That is, speech/dialog enhancement can be applied only when the listener has problems understanding speech/dialog. Moreover, speech/dialog enhancement does not have to rely on potentially inaccurate speech/dialog detection. In consequence, unnecessary speech/dialog enhancement, which could degrade the overall perceived quality of the rendered audio, can be avoided, while it can be ensured that the listener is aided in understanding speech/dialog where this is needed.
- In some embodiments, obtaining the measure of the cognitive load may be based on information indicative of a pupil dilation (e.g., pupil diameter) of the listener. Thus, the method may be said to include monitoring the pupil dilation of the listener. In general, obtaining the measure of the cognitive load of the listener may be based on information indicative of a pupil dilation of the listener. As has been found, the pupil dilation (e.g., pupil diameter) of a listener is a good indicator of the cognitive load (i.e., how loaded the brain is) of a listener that listens to an audio signal. Reference in this regard is made for example to Kramer, S. E., Kapteyn, T. S., Festen, J. M., et al., Assessing aspect of hearing handicap by means of pupil dilation, 1997, Audiology 36, 155-164, and to Zekveld, A. A., Kramer, S. E., and Festen, J. M., Pupil Response as an Indication of Effortful Listening: The Influence of Sentence Intelligibility, 2010, Ear & Hearing, Vol. 31 (4), 480-490. However, the measure of the cognitive load can also be based on other techniques such as Electroencephalography, monitoring the body temperature, or a push button that can be actuated by the listener.
- In some embodiments, it may be determined that speech/dialog enhancement shall be applied if the pupil dilation (e.g., pupil diameter) of the listener is found to be above a predefined threshold for the pupil dilation.
- Typically, if the pupil dilation is above a given threshold, this indicates high cognitive load of the listener, which in turn can be seen as an indication that speech/dialog enhancement shall be applied. This enables selective application of speech/dialog enhancement, specifically tailored to the listener's needs.
- In some embodiments, obtaining the measure of the cognitive load may be based on eye-tracking data for the listener. Thus, obtaining the measure of the cognitive load may involve analyzing the eye-tracking data. The eye-tracking data may be data that is obtained by an eye-tracking device, such as an eye-tracking device of a virtual reality (VR) headset (or augmented reality (AR)/mixed AR+VR reality headset) or a TV set with eye tracking, for example. Since eye tracking data is available for many media applications (e.g., for gesture-controlled TV sets or in AR environments, VR environments, and mixed AR+VR environments), this allows for a particularly simple implementation of the proposed method, without incurring the need for additional hardware.
- In some embodiments, obtaining the measure of the cognitive load may involve determining information indicative of a pupil dilation (e.g., pupil diameter) of the listener based on the eye-tracking data, for example by analyzing the eye-tracking data. Thus, the method may be said to include monitoring the pupil dilation of the listener. In general, obtaining the measure of the cognitive load of the listener may be based on information indicative of a pupil dilation of the listener.
- In some embodiments, the method may further include pre-processing of the eye-tracking data (i.e., a processing of the eye-tracking data before determining information indicative of the pupil dilation based on the eye-tracking data). For example, the pre-processing of the eye-tracking data may involve removing portions of the eye-tracking data that relate to eye blinks. The pre-processing may further involve interpolation and/or smoothing between timings before (e.g., immediately before or a certain time interval before) and after (e.g., immediately after, or a certain time interval after) eye blinks. Thereby, an accuracy of the determination of the pupil dilation can be improved.
- In some embodiments, the method may further include applying a heuristic model to the eye-tracking data for determining the cognitive load of the listener. Alternatively, the heuristic model can also be applied to data indicative of the pupil dilation. The heuristic model may evolve over time, for example to adapt to a specific user. Evolution of the heuristic model over time may involve machine learning (learning over time).
- In some embodiments, the method may further include receiving side-information (e.g., metadata) associated with the audio signal. The side-information may include information on whether speech/dialog is present in the audio signal. The side-information may be conveyed in a bitstream, i.e., the side-information may be received from the bitstream. Then, the determination of whether speech/dialog enhancement shall be applied may be further based on the received information. The side-information may have been obtained at an encoder performing speech/dialog detection (e.g., in the context of speech/dialog enhancement). For example, it may be determined that speech/dialog enhancement shall be applied for a given portion of the audio signal if the side-information indicates that speech/dialog is present in the given portion of the audio signal and if the measure of the cognitive load for the given portion of the audio signal satisfies a predetermined condition. The predetermined condition may relate to whether the cognitive load for the given portion of the audio signal exceeds a predetermined threshold value (e.g., whether the pupil dilation of the listener is found to be above a predefined threshold). Thereby, applying speech/dialog enhancement can be avoided in situations in which the cognitive load and/or pupil dilation of listener is high for other reasons.
- In some embodiments, the determination of whether speech/dialog enhancement shall be applied may be performed for each of a plurality of consecutive portions of the audio signal. The method may further involve smoothing a degree of speech/dialog enhancement between consecutive portions of the audio signal, e.g., by fading in and/or fading out speech/dialog enhancement. Thereby, audible artifacts that might result from abrupt starting or ending speech/dialog enhancement can be avoided.
- In some embodiments, the determination of whether speech/dialog enhancement shall be applied may be performed for each of a plurality of channels. For example, the side-information may be channel-specific and may indicate for each channel whether speech/dialog is present.
- Another aspect relates to a further method of processing an audio signal (e.g., a decoded audio signal). The method may include obtaining a measure of a cognitive load of a listener that listens to a rendering of the audio signal. The method may further include determining a degree of speech/dialog enhancement (e.g., a magnitude of a gain for speech/dialog enhancement) that shall be applied based on the obtained measure of the cognitive load. The method may yet further include performing speech/dialog enhancement based on the determination. The method may additionally include receiving side-information associated with the audio signal. The side-information may include information on gains for speech/dialog enhancement of the audio signal. For example, the side information may include, for each portion (e.g., frame) of the audio signal, a plurality of gains for speech/dialog enhancement from which a selection can be made. Then, the determination may be further based on the received side-information.
- Another aspect relates to a decoder for processing an audio signal (e.g., a decoded audio signal, such as an audio signal decoded from a bitstream, for example). The decoder may include a processor and a memory coupled to the processor. The processor may be adapted to perform the operations outlined in the present disclosure. For example, the memory may store instructions for causing the processor to perform the operations outlined in the present disclosure.
- Another aspect relates to a program (software program). The software program may be adapted for execution on a processor and for performing the method steps outlined in the present disclosure when carried out on a computing device.
- Another aspect relates to a computer-readable storage medium. The storage medium may include a program (software program) adapted for execution on a processor and for performing the method steps outlined in the present disclosure when carried out on a computing device.
- Another aspect relates to a computer program product. The computer program may include executable instructions for performing the operations outlined in the present disclosure when executed on a computer.
- According to another aspect, a headset for computer-mediated reality (e.g., AR headset, VR headset, or mixed AR+VR headset) is described. The headset may include a processor and a memory coupled to the processor. The processor may be adapted to obtain eye-tracking data of a wearer of the headset. The processor may be further adapted to determine a measure of a cognitive load of the wearer of the headset based on the eye-tracking data. The processor may be yet further adapted to output an indication of the cognitive load of the wearer of the headset. The indication may be output for example to a decoder for processing an audio signal (e.g., a decoded audio signal, such as an audio signal decoded from a bitstream, for example). The indication may enable the decoder to determine whether speech/dialog enhancement shall be applied to a current portion of an audio signal that is rendered by the decoder. The memory may store instructions for causing the processor to perform the operations according to this aspect.
- According to another aspect, a method of operating a headset for computer-mediated reality (e.g., AR headset, VR headset, or mixed AR+VR headset) is described. The method may include obtaining eye-tracking data of a wearer of the headset. The method may further include determining a measure of a cognitive load of the wearer of the headset based on the eye-tracking data. The method may yet further include outputting an indication of the cognitive load of the wearer of the headset.
- Another aspect relates to a system of the decoder outlined in the present disclosure and the headset for computer-mediated reality outlined in the present disclosure. In this system, the headset may operate in conjunction (e.g., in cooperation) with the decoder. For example, the headset may transmit information, such as an indication of a cognitive load of a wearer of the headset and/or eye-tracking data, to the decoder.
- Another aspect relates to a system of a decoder for processing an audio signal (e.g., a decoded audio signal, such as an audio signal decoded from a bitstream, for example) and a headset for computer-mediated reality (e.g., AR headset, VR headset, or mixed AR+VR headset) operating in conjunction with the decoder. The headset may comprise a processor and a memory coupled to the processor. The processor may be adapted to obtain eye-tracking data of a wearer of the headset. The processor may be further adapted to determine a measure of a cognitive load of the wearer of the headset based on the eye-tracking data. The processor may be yet further adapted to output an indication of the cognitive load of the wearer of the headset to the decoder. The decoder may be adapted to render the audio signal for listening by the wearer of the headset. The decoder may be further adapted to receive the indication of the cognitive load of the wearer of the headset from the headset. The decoder may be further adapted to determine whether speech/dialog enhancement shall be applied based on the indication of the cognitive load of the wearer of the headset. The decoder may be yet further adapted to perform speech/dialog enhancement based on the determination.
- According to another aspect, a method of operating a system of a decoder for processing an audio signal (e.g., a decoded audio signal, such as an audio signal decoded from a bitstream, for example) and a headset for computer-mediated reality (e.g., AR headset, VR headset, or mixed AR+VR headset) operating in conjunction with the decoder is described. The method may include, at the headset, obtaining eye-tracking data of a wearer of the headset. The method may further include, at the headset, determining a measure of a cognitive load of the wearer of the headset based on the eye-tracking data. The method may yet further include, at the headset, outputting an indication of the cognitive load of the wearer of the headset to the decoder. The method may include, at the decoder, rendering the audio signal for listening by the wearer of the headset. The method may further include, at the decoder, receiving the indication of the cognitive load of the wearer of the headset from the headset. The method may further include, at the decoder, determining whether speech/dialog enhancement shall be applied based on the indication of the cognitive load of the wearer of the headset. The method may yet further include, at the decoder, performing speech/dialog enhancement based on the determination.
- It should be noted that the methods and apparatus including its preferred embodiments as outlined in the present disclosure may be used stand-alone or in combination with the other methods and systems described in this disclosure. Furthermore, all aspects of the methods and apparatus outlined in the present disclosure may be arbitrarily combined. In particular, the features of the claims may be combined with one another in an arbitrary manner. Any statements made with regard to a given method are understood to likewise apply to a corresponding apparatus, and vice versa, and repetitions of such statements may be omitted for reasons of conciseness.
- Example embodiments of the present disclosure are explained below with reference to the accompanying drawings, wherein identical reference numerals indicate like or similar elements, and wherein:
-
FIG. 1 schematically illustrates an example of an encoding/decoding environment in which speech/dialog enhancement may be applied, -
FIG. 2 schematically illustrates an example of an encoding/decoding environment according to embodiments of the disclosure, -
FIG. 3 is a flow chart schematically illustrating an example of a method according to embodiments of the disclosure, -
FIG. 4 is a flow chart schematically illustrating another example of a method according to embodiments of the disclosure, and -
FIG. 5 is a flow chart schematically illustrating yet another example of a method according to embodiments of the disclosure. -
FIG. 1 schematically illustrates an example of an encoding/decoding environment 100 in which speech/dialog enhancement may be applied. The environment can be seen as being split into an encoder side (to the left of the dashed vertical line) and a decoder side (to the right of the dashed vertical line). - On the encoder side, an encoder (e.g., core encoder) 10 receives input audio (input audio signal) 11. The
input audio 11 may be non-enhanced audio, such as audio to which speech/dialog enhancement has not been applied. Theinput audio 11 is also provided to a speech/dialogenhancement analysis block 20. Optionally, separate speech/dialog tracks (separate speech/dialog channels, separate speech/dialog stems) 12 may be provided to the speech/dialogenhancement analysis block 20 as well. The separate speech/dialog tracks 12 may be in alignment (time alignment) with theinput audio 11. Theencoder 10 encodes theinput audio 11 and outputs encodedaudio 13. The encodedaudio 13 may be provided to the decoder side. As an intermediate step, the encodedaudio 13 may be stored on a storage medium. The speech/dialogenhancement analysis block 20 analyzes theinput audio 11 and/or the separate speech/dialog tracks 12. Based on this analysis, the speech/dialogenhancement analysis block 20 determines whether speech/dialog is present for each given portion (e.g., frame) of theinput audio 11. The speech/dialogenhancement analysis block 20 may further determine parameters for speech/dialog enhancement of respective portions of theinput audio 11. Based on these determinations, the speech/dialogenhancement analysis block 20 outputs side-information 14 (e.g., dedicated side-information for speech/dialog enhancement) that includes an indication (e.g., flag) for each portion of theinput audio 11 of whether the respective portion includes speech/dialog. The side-information 14 may further include the parameters for speech/dialog enhancement. These parameters may provide the decoder side with instructions on how to apply the speech/dialog enhancement for a given portion of the audio. - On the decoder side, a decoder (e.g., core decoder) 50 receives the encoded
audio 13. Thedecoder 50 decodes the encodedaudio 13 and outputs decodedaudio 51. A speech/dialog enhancement block (speech/dialog synthesis block) 70 receives the side-information 14 and boosts portions of the decodedaudio 51 by means of a gain unit (booster) 60. The speech/dialog enhancement block 70 controls thegain unit 60 in accordance with the side-information 14, i.e., applies speech/dialog enhancement to those portions of the audio that are flagged as including speech/dialog by the side-information 14. As noted above, the side-information 14 may further specify the manner in which speech/dialog enhancement is applied, e.g., a gain that is applied to the respective portion of the audio. Thegain unit 60outputs output audio 52 that may be rendered to one or more speakers for listening. - Both the encoder-side processing and the decoder-side processing may be performed on a channel-by-channel basis, i.e., for each of a plurality of channels. Speech/dialog enhancement may be only applied to those channels (and to those portions of the audio) that include speech/dialog.
- Thus, in the encoding/
decoding environment 100, speech/dialog enhancement is performed (i.e., a boost is applied to the decoded audio 51) whenever speech/dialog is detected in the relevant portion of the audio. Whether or not this speech/dialog enhancement improves the perceived audio quality may depend on the individual listener, the listening conditions, or the relevant audio scene. - Methods and apparatus according to the present disclosure are based on the finding that the pupil dilation of a human's eye is a measure of how loaded the brain is (i.e., a measure of a cognitive load). Methods and apparatus according to the present disclosure are further based on the finding that a listener's cognitive load increases under difficult speech reception conditions, i.e., if the listener has problems understanding speech. In other words, a listener's increase in cognitive effort is reflected in an increased pupil dilation (e.g., pupil diameter). Based on these finding, the present disclosure proposes to use a listener's pupil dilation (which reflects the listener's cognitive load) to control speech/dialog enhancement (e.g., boosting of speech/dialog). Information on the pupil dilation may be obtained from an eye-tracking device that operates in conjunction with an audio decoder, for example.
- An example of an encoding/
decoding environment 200 according to embodiments of the disclosure is schematically illustrated inFIG. 2 . The encoder side (to the left of the dashed vertical line) is identical to the encoder side in the encoder/decoder environment 100 ofFIG. 1 . The decoder side (to the right of the dashed vertical line) differs from the decoder side in the encoder/decoder environment 100 ofFIG. 1 in that again unit 60′ (a modified gain unit) is provided that is controlled by the speech/dialog enhancement block 70 and a cognitiveload determination block 80. Thegain unit 60′ outputs (modified)output audio 52′. The cognitiveload determination block 80 receives eye-trackingdata 81 from an eye-trackingdevice 90. The eye-tracking data may be time series data that indicates, for each of a plurality of consecutive instances of time, one or more characteristics (parameters) of the eye of auser 95. The eye-trackingdevice 90 may be in visual contact with theuser 95 and may be part of, for example, a headset for computer-mediated reality (e.g., AR headset, VR headset, or mixed AR+VR headset) or a TV set having an eye-tracking capability. Theuser 95 may listen to a rendering of theoutput audio 52′. The cognitiveload determination block 80 determines a cognitive load of theuser 95, by analyzing the eye-tracking data. For example, the cognitiveload determination block 80 may analyze the eye-tracking data to determine a pupil dilation (e.g., pupil diameter) of theuser 95, and may determine the cognitive load of theuser 95 based on the determined pupil dilation. Thegain unit 60′ is controlled by the speech/dialog enhancement block 70 and the cognitiveload determination unit 80. For example, speech/dialog enhancement (by boosting the respective portion of the audio) may be applied if the side-information 14 indicates that speech/dialog is present, and if the determined cognitive load indicates that theuser 95 has problems understanding said speech/dialog. Alternatively, speech/dialog enhancement may be applied based on the determined cognitive load alone, i.e., without reference to the side-information 14, or at least without reference to the indication of whether speech/dialog is present in the respective portion of the audio. - Broadly speaking, the proposed methods and apparatus selectively apply speech/dialog enhancement (i.e., selectively boost or do not boost speech/dialog) based on a listener's effort in understanding the speech/dialog.
-
FIG. 3 illustrates an example of amethod 300 of processing an audio signal (e.g., a decoded audio signal, such as an audio signal decoded from a bitstream, for example) and for selectively applying speech/dialog enhancement to (portions of) the audio signal. Themethod 300 may be performed by a decoder, for example. The method may comprise an initial step of decoding the audio signal from a bitstream. - At step S310, a measure of a cognitive load of a listener that listens to a rendering of the audio signal (e.g., that listens to a rendered representation of the audio signal) is obtained. The measure of the cognitive load can be obtained (e.g., determined) based on information indicative of a pupil dilation (e.g., pupil diameter) of an eye of the listener. Accordingly, step S310 may be said to be a step of monitoring a pupil dilation of the listener. In general, obtaining the measure of the cognitive load may be based on information indicative of a pupil dilation of the listener.
- In certain implementations, obtaining the measure of the cognitive load may be based on eye-tracking data for the listener. For example, the eye-tracking data may be used for determining information indicative of the pupil dilation of the listener. This may involve analyzing the eye-tracking data. The eye-tracking data may be obtained by an eye-tracking device, such as an eye-tracking device of a headset for computer-mediated reality (e.g., AR headset, VR headset, or mixed AR+VR headset) with eye tracking, or a TV set with eye tracking. Suitable eye-tracking devices and consumer devices capable of eye tracking (e.g., headsets for computer-mediated reality, TV sets) are commercially available. In the context of the present disclosure, a headset for computer-mediated reality is understood to be any headset that can present visual information to a wearer of the headset to overlay or replace the wearer's view of a real scene.
- The eye-tracking data may be analyzed using a heuristic model for determining the cognitive load of the listener. For example, the heuristic model may be applied to the information indicative of the pupil dilation, to determine the cognitive load of the listener. The heuristic model may evolve over time. Evolution over time may involve machine learning (learning over time). For example, the heuristic model may be adapted to a specific user (listener), and may evolve over time to improve the adaptation.
- In order to be able to obtain more accurate information on the pupil dilation (and thereby, more accurate information on the cognitive load), the eye-tracking data may be pre-processed before obtaining the information indicative of the pupil dilation and/or the measure of the cognitive load of the listener (i.e., before analyzing the eye-tracking data). Pre-processing of the eye-tracking data may involve removing portions of the eye-tracking data that relate to eye blinks (e.g., removing eye blinks). Eye blinks may be detected by an automatic classification algorithm, for example, or using a heuristic measure. When using the heuristic measure, an eye blink may be detected, for example, if the pupil dilation is below a certain threshold. If an eye blink is detected, a portion of the eye-tracking data relating to the detected eye blink, optionally together with portions relating to a certain interval of time before and after the detected eye blink may be removed from the eye-tracking data. The pre-processing may further include interpolation between timings before and after the detected (or removed) eye blink (e.g., between a latest timing before a time interval relating to the removed portion of the eye-tracking data and an earliest timing after this time interval). The pre-processing may further include smoothing (e.g., over time) of the eye-tracking data after removing eye blinks (or after subsequent interpolation).
- At step S320, it is determined whether speech/dialog enhancement shall be applied based on the obtained measure of the cognitive load. For example, it may be determined that speech/dialog enhancement shall be applied if the cognitive load of the listener is found to be above a predefined threshold for the cognitive load. Likewise, it may be determined that speech/dialog enhancement shall be applied if the pupil dilation of the listener is found to be above a predefined threshold for the pupil dilation. The rationale behind this determination is that a high cognitive load (which would be indicated by an enlarged pupil dilation) can be indicative of the listener having problems understanding speech or speech/dialog.
- In certain implementations, the
method 300 may further include receiving side-information (metadata) associated with the audio signal, wherein the side-information includes information on whether speech/dialog is present in the audio signal. That is, the side-information may include, for each of a plurality of consecutive portions (e.g., frames) of the audio signal, and indication (e.g., flag) of whether that portion of the audio signal contains speech/dialog. The side-information may have been obtained at an encoder for enabling or facilitating speech/dialog enhancement at the decoder. In this case, the determination at step S320 may be further based on the side-information (e.g., based on the aforementioned indication of whether speech/dialog is present). For example, it may be determined that speech/dialog enhancement shall be applied for a given portion (e.g., frame) of the audio signal if the side-information indicates that speech/dialog is present in the given portion of the audio signal and if the measure of the cognitive load for the given portion of the audio signal satisfies a predetermined condition. The predetermined condition may be satisfied if the cognitive load for the given portion of the audio signal exceeds a predetermined threshold value, for example. Likewise, the predetermined condition may be deemed satisfied if the pupil dilation of the listener is found to be above a predefined threshold. - At step S330, speech/dialog enhancement is performed based on the determination (e.g., in accordance with the determination). The speech/dialog enhancement may be applied to a current portion (e.g., frame) of the audio signal (a portion that is currently rendered), or to the next portion of the audio signal. In general, speech/dialog enhancement may be selectively performed for each of a plurality of successive portions (e.g., frames) of the audio signal. The method may further comprise rendering the audio signal.
- As indicated above, the side-information may include parameters for speech/dialog enhancement. These parameters may provide the decoder side with instructions on how to apply the speech/dialog enhancement. In certain implementations, the side-information may indicate, for each portion of the audio signal that contains speech/dialog, a gain for application to the respective portion of the audio. This gain then could be applied at step S330 if it is determined at step S320 that speech/dialog enhancement shall be applied (i.e., the speech/dialog enhancement may be performed in accordance with the parameters for speech/dialog enhancement included in the side-information for the relevant portion of the audio signal). The side-information may also contain more than one gain for each portion of the audio signal that contains speech/dialog. Then, if speech/dialog enhancement is performed at step S330, selection among the more than one gain can be made based on the measure of the cognitive load. For example, a first gain may be selected and applied if it is determined at step S320 that speech/dialog enhancement shall be applied (i.e., if the measure of the cognitive load exceeds the predefined threshold value) but the measure of the cognitive load is still found to be below a predefined second threshold (that is larger than the predefined threshold). A second gain (that is larger than the first gain) could be applied if the measure of the cognitive load is found to be above the predefined second threshold. It is understood that this scheme can be extended to an arbitrary number of gains and predefined thresholds.
- In general, a degree of speech/dialog enhancement that is performed at step S330 (e.g., a magnitude of a gain that is applied at this step) may depend on the measure of the cognitive load of the listener.
- The steps of
method 300 may be performed for each of a plurality of consecutive portions (e.g., frames) of the audio signal. For example, the determination of whether speech/dialog enhancement shall be applied may be performed for each of a plurality of consecutive portions of the audio signal, and the speech/dialog enhancement may be selectively applied to each portion of the audio signal in accordance with a result of the determination for that portion of the audio signal. In this case, the method may further include a step of smoothing a degree of the speech/dialog enhancement (e.g., a gain that is applied when boosting the speech/dialog) between consecutive portions of the audio signal. Further, the steps ofmethod 300 may be performed for each of a plurality of channels that form the audio signal. - As noted above,
method 300 may be performed by a decoder. Such decoder may comprise a processor and a memory coupled to the processor. The processor may be adapted to perform each of steps S310, S320, and S330 described above. The memory may store respective instructions for causing the processor to perform these steps. - In certain implementations, the decoder may operate in conjunction (e.g., in cooperation) with a headset for computer-mediated reality. (e.g., AR headset, VR headset, or mixed AR+VR headset) that obtains eye-tracking data. The headset for computer-mediated reality may be seen as a non-limiting example of a device for obtaining eye-tracking data.
FIG. 4 illustrates an example of amethod 400 of operating such headset. Without intended limitation, the remainder of the present disclosure makes reference to a VR headset. It is understood that any other headset for computer-mediated reality could be used as well in the embodiments described below. - At step S410, eye-tracking data of a wearer of the VR headset is obtained. This may be done using an eye-tracking device of the VR headset.
- At step S420, a measure of a cognitive load of the wearer of the VR headset is determined based on the eye-tracking data. This may be done in the manner described with reference to step S310 above. The wearer of the VR headset is assumed to be the listener that listens to the rendering of the audio signal.
- At step S430, an indication of the cognitive load of the wearer of the VR headset is output. The indication of the cognitive load of the wearer of the VR headset may be output to the decoder. The output indication may enable the decoder to determine whether speech/dialog enhancement shall be applied to a current portion of an audio signal that is rendered by the decoder, for example in the manner described with reference to step S320 above.
- In alternative implementations, the VR headset (as an example of a headset for computer-mediated reality, or more generally, an example of device for obtaining eye-tracking data) may directly output the eye-tracking data to the decoder. The decoder may then determine the measure of the cognitive load of the wearer of the VR headset, for example in the manner described with reference to step S310 above.
- Steps S410 to S430 may be continuously performed. For example, each of these steps may be performed once every given time interval. For example, each of these steps may be performed once per portion (e.g., frame) of the audio signal.
- A corresponding VR headset may comprise a processor and a memory coupled to the processor. The processor may be adapted to perform each of steps S410, S420, and S430 described above. The memory may store respective instructions for causing the processor to perform these steps. The VR headset may further comprise an optical device (e.g., a camera) for capturing a visual representation of the wearer's eye or eyes, from which the eye-tracking data can be derived.
- A decoder and a VR headset (as an example of a headset for computer-mediated reality, or more generally, an example of device for obtaining eye-tracking data) operating in conjunction with the decoder may form a system for processing the audio signal.
FIG. 5 illustrates an example of amethod 500 of operating such system. - At step S510, eye-tracking data of a wearer of the VR headset is obtained at the VR headset. This may be performed in the manner described above with reference to step S410.
- At step S520, a measure of a cognitive load of the wearer of the VR headset is determined at the VR headset based on the eye-tracking data. This may be performed in the manner described above with reference to step S420 (or step S310).
- At step S530, an indication of the cognitive load of the wearer of the VR headset to the decoder is output by the VR headset. This may be performed in the manner described above with reference to step S430.
- Steps S510 to S530 may be continuously performed. Likewise, steps S540 to S560 described below may be continuously performed. For example, each of these steps may be performed once every given time interval. For example, each of these steps may be performed once per portion (e.g., frame) of the audio signal.
- At step S540, the audio signal is rendered by the decoder for listening by the wearer of the VR headset. As noted above, this may be a continuous process.
- At step S550, the indication of the cognitive load of the wearer of the VR headset from the VR headset is received at the decoder. Notably, steps S540 and S550 may be performed in any order.
- At step S560, it is determined by the decoder whether speech/dialog enhancement shall be applied based on the indication of the cognitive load of the wearer of the VR headset. This may be performed in the manner described above with reference to step S320. The determination may be further based on side-information associated with the audio signal, as described above.
- At step S570, speech/dialog enhancement is performed by the decoder based on (e.g., in accordance with) the determination. This may be performed in the manner described above with reference to step S330.
- In an alternative implementation, the VR headset may directly transmit the eye-tracking data to the decoder. In this case, step S520 would be performed at the decoder, i.e., the decoder would determine the measure of the cognitive load of the wearer of the VR headset based on the eye-tracking data, for example in the manner described above with reference to step S310.
- A decoder for use in this system may comprise a processor and a memory coupled to the processor. The processor may be adapted to perform each of steps S540, S550, S560, and S570 described above. The memory may store respective instructions for causing the processor to perform these steps.
- A VR headset for use in the system may comprise a processor and a memory coupled to the processor. The processor may be adapted to perform each of steps S510, S520, and S530 described above. The memory may store respective instructions for causing the processor to perform these steps. The VR headset may further comprise an optical device (e.g., a camera) for capturing a visual representation of the wearer's eye or eyes, from which the eye-tracking data can be derived.
- It should be noted that the description and drawings merely illustrate the principles of the proposed methods and apparatus. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope. Furthermore, all examples recited herein are principally intended expressly to be only for pedagogical purposes to aid the reader in understanding the principles of the proposed methods and apparatus and the concepts contributed by the inventors to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass equivalents thereof.
- The methods and apparatus described in the present disclosure may be implemented as software, firmware and/or hardware. Certain components may e.g. be implemented as software running on a digital signal processor or microprocessor. Other components may e.g. be implemented as hardware and or as application specific integrated circuits. The signals encountered in the described methods and apparatus may be stored on media such as random-access memory or optical storage media. They may be transferred via networks, such as radio networks, satellite networks, wireless networks or wireline networks, e.g. the Internet.
- Various aspects of the present invention may be appreciated from the following enumerated example embodiments (EEEs):
- EEE 1. A method of processing a decoded audio signal, the method comprising:
- obtaining a measure of a cognitive load of a listener that listens to a rendering of the audio signal;
- determining whether speech/dialog enhancement shall be applied based on the obtained measure of the cognitive load; and
- performing speech/dialog enhancement based on the determination.
- EEE 2. The method of EEE 1, further comprising receiving side-information associated with the audio signal, the side-information including information on whether speech/dialog is present in the audio signal, wherein the determination is further based on the received side-information.
- EEE 3. The method of EEE 2, wherein it is determined that speech/dialog enhancement shall be applied for a given portion of the audio signal if the side-information indicates that speech/dialog is present in the given portion of the audio signal and if the measure of the cognitive load for the given portion of the audio signal satisfies a predetermined condition.
- EEE 4. The method of EEE 1, wherein it is determined that speech/dialog enhancement shall be applied if the cognitive load of the listener is found to be above a predefined threshold for the cognitive load.
- EEE 5. The method of EEE 1, wherein obtaining the measure of the cognitive load is based on information indicative of a pupil dilation of the listener.
- EEE 6. The method of EEE 5, wherein it is determined that speech/dialog enhancement shall be applied if the pupil dilation of the listener is found to be above a predefined threshold for the pupil dilation.
- EEE 7. The method of EEE 1, wherein obtaining the measure of the cognitive load is based on eye-tracking data for the listener.
- EEE 8. The method of EEE 7, wherein obtaining the measure of the cognitive load involves determining information indicative of a pupil dilation of the listener based on the eye-tracking data.
- EEE 9. The method of EEE 7, further comprising pre-processing of the eye-tracking data.
-
EEE 10. The method of EEE 9, wherein pre-processing the eye-tracking data involves removing portions of the eye-tracking data that relate to eye blinks. -
EEE 11. The method of EEE 7, further comprising applying a heuristic model to the eye-tracking data for determining the cognitive load of the listener. -
EEE 12. The method of EEE 1, wherein the determination of whether speech/dialog enhancement shall be applied is performed for each of a plurality of consecutive portions of the audio signal. -
EEE 13. The method of EEE 1, wherein the determination of whether speech/dialog enhancement shall be applied is performed for each of a plurality of channels. -
EEE 14. A method of processing a decoded audio signal, the method comprising: - obtaining a measure of a cognitive load of a listener that listens to a rendering of the audio signal;
- determining a degree of speech/dialog enhancement that shall be applied based on the obtained measure of the cognitive load; and
- performing speech/dialog enhancement based on the determination.
- EEE 15. The method of
EEE 14, further comprising receiving side-information associated with the audio signal, the side-information including information on gains for speech/dialog enhancement of the audio signal, wherein the determination is further based on the received information. - EEE 16. A decoder for processing an audio signal, the decoder comprising a processor and a memory coupled to the processor, wherein the processor is adapted to perform the operations of EEE 1.
- EEE 17. A program for causing a computer to perform the operations of EEE 1 when performed on the computer.
- EEE 18. A computer-readable storage medium storing a program for causing a computer to perform the operations of EEE 1 when performed on the computer.
- EEE 19. A headset for computer-mediated reality comprising a processor and a memory coupled to the processor, wherein the processor is adapted to:
- obtain eye-tracking data of a wearer of the headset;
- determine a measure of a cognitive load of the wearer of the headset based on the eye-tracking data; and
- output an indication of the cognitive load of the wearer of the headset.
-
EEE 20. A method of operating a headset for computer-mediated reality, the method comprising: - obtaining eye-tracking data of a wearer of the headset;
- determining a measure of a cognitive load of the wearer of the headset based on the eye-tracking data; and
- outputting an indication of the cognitive load of the wearer of the headset.
- EEE 21. A system of the decoder of EEE 16 and the headset for computer-mediated reality of EEE 19, the headset operating in conjunction with the decoder.
- EEE 22. A system of a decoder for processing an audio signal and a headset for computer-mediated reality operating in conjunction with the decoder, the headset comprising a processor and a memory coupled to the processor,
- wherein the processor is adapted to:
- obtain eye-tracking data of a wearer of the headset;
- determine a measure of a cognitive load of the wearer of the headset based on the eye-tracking data; and
- output an indication of the cognitive load of the wearer of the headset to the decoder; and
- wherein the decoder is adapted to:
- render the audio signal for listening by the wearer of the headset;
- receive the indication of the cognitive load of the wearer of the headset from the headset;
- determine whether speech/dialog enhancement shall be applied based on the indication of the cognitive load of the wearer of the headset; and
- perform speech/dialog enhancement based on the determination.
- EEE 23. A method of operating a system of a decoder for processing an audio signal and a headset for computer-mediated reality operating in conjunction with the decoder,
- wherein the method comprises, at the headset:
- obtaining eye-tracking data of a wearer of the headset;
- determining a measure of a cognitive load of the wearer of the headset based on the eye-tracking data;
- outputting an indication of the cognitive load of the wearer of the headset to the decoder; and
- the method further comprises, at the decoder:
- rendering the audio signal for listening by the wearer of the headset;
- receiving the indication of the cognitive load of the wearer of the headset from the headset;
- determining whether speech/dialog enhancement shall be applied based on the indication of the cognitive load of the wearer of the headset; and
- performing speech/dialog enhancement based on the determination.
Claims (20)
1. A method of processing an audio signal, the method comprising:
obtaining eye-tracking data of a wearer of a headset for computer-mediated reality;
determining a measure of a cognitive load of the wearer of the headset that listens to a rendering of the audio signal, based on the eye-tracking data; and
determining whether speech enhancement shall be applied based on the obtained measure of the cognitive load; and
performing speech enhancement based on the determination.
2. The method of claim 1 , further comprising receiving side-information associated with the audio signal, the side-information including information on whether speech is present in the audio signal,
wherein the determination is further based on the received side-information.
3. The method of claim 2 , wherein it is determined that speech enhancement shall be applied for a given portion of the audio signal if the side-information indicates that speech is present in the given portion of the audio signal and if the measure of the cognitive load for the given portion of the audio signal satisfies a predetermined condition.
4. The method of claim 2 , wherein the side-information includes information on gains for speech enhancement of the audio signal.
5. The method of claim 2 , wherein the determining whether speech enhancement shall be applied may be performed for each of a plurality of channels, wherein the side-information is channel-specific and indicates for each channel whether speech is present.
6. The method of claim 1 , wherein it is determined that speech enhancement shall be applied if the cognitive load of the listener is found to be above a predefined threshold for the cognitive load.
7. The method claim 1 , wherein determining the measure of the cognitive load is based on information indicative of a pupil dilation of the wearer.
8. The method of claim 7 , wherein it is determined that speech enhancement shall be applied if the pupil dilation of the wearer is found to be above a predefined threshold for the pupil dilation.
9. The method of claim 1 , wherein determining the measure of the cognitive load involves determining information indicative of a pupil dilation of the wearer based on the eye-tracking data.
10. The method of claim 1 , further comprising applying a heuristic model to the eye-tracking data for determining the cognitive load of the wearer.
11. The method of claim 1 , further comprising pre-processing of the eye-tracking data.
12. The method of claim 11 , wherein pre-processing the eye-tracking data involves removing portions of the eye-tracking data that relate to eye blinks.
13. The method of claim 1 , wherein the determination of whether dialog enhancement shall be applied is performed for each of a plurality of consecutive portions of the audio signal.
14. The method of claim 1 , wherein the determination of whether dialog enhancement shall be applied is performed for each of a plurality of channels.
15. A headset for computer-mediated reality comprising a processor and a memory coupled to the processor, wherein the processor is adapted to:
obtain eye-tracking data of a wearer of the headset;
determine a measure of a cognitive load of the wearer of the headset based on the eye-tracking data; and
output an indication of the cognitive load of the wearer of the headset.
16. A system of the headset for computer-mediated reality of claim 15 and a decoder for processing an audio signal, wherein the decoder is adapted to:
determine whether speech enhancement shall be applied based on the indication of the cognitive load; and
perform speech enhancement based on the determination.
17. The system of claim 16 , wherein the decoder is adapted to receive side-information associated with the audio signal, the side-information including information on whether speech is present in the audio signal,
wherein the determination is further based on the received side-information.
18. The system of claim 17 , wherein the decoder is adapted to determine that speech enhancement shall be applied for a given portion of the audio signal if the side-information indicates that speech is present in the given portion of the audio signal and if the measure of the cognitive load for the given portion of the audio signal satisfies a predetermined condition.
19. The system of claim 16 , wherein the side-information includes information on gains for speech enhancement of the audio signal.
20. The system of claim 16 , wherein the determination whether speech enhancement shall be applied may be performed for each of a plurality of channels, wherein side-information is channel-specific and indicates for each channel whether speech is present.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/998,796 US20190057694A1 (en) | 2017-08-17 | 2018-08-16 | Speech/Dialog Enhancement Controlled by Pupillometry |
US17/213,088 US20210264928A1 (en) | 2017-08-17 | 2021-03-25 | Speech/Dialog Enhancement Controlled by Pupillometry |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201762546640P | 2017-08-17 | 2017-08-17 | |
US15/998,796 US20190057694A1 (en) | 2017-08-17 | 2018-08-16 | Speech/Dialog Enhancement Controlled by Pupillometry |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/213,088 Continuation US20210264928A1 (en) | 2017-08-17 | 2021-03-25 | Speech/Dialog Enhancement Controlled by Pupillometry |
Publications (1)
Publication Number | Publication Date |
---|---|
US20190057694A1 true US20190057694A1 (en) | 2019-02-21 |
Family
ID=65360695
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/998,796 Abandoned US20190057694A1 (en) | 2017-08-17 | 2018-08-16 | Speech/Dialog Enhancement Controlled by Pupillometry |
US17/213,088 Pending US20210264928A1 (en) | 2017-08-17 | 2021-03-25 | Speech/Dialog Enhancement Controlled by Pupillometry |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/213,088 Pending US20210264928A1 (en) | 2017-08-17 | 2021-03-25 | Speech/Dialog Enhancement Controlled by Pupillometry |
Country Status (1)
Country | Link |
---|---|
US (2) | US20190057694A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2021059745A1 (en) * | 2019-09-25 | 2021-04-01 | 株式会社Jvcケンウッド | Gaze data correction device, evaluation device, gaze data correction method, evaluation method, gaze data correction program, and evaluation program |
US11327636B2 (en) * | 2019-08-20 | 2022-05-10 | Dell Products L.P. | Dynamically scale complexity of a user interface based on cognitive load |
WO2023053557A1 (en) * | 2021-09-30 | 2023-04-06 | 富士フイルム株式会社 | Information processing device, information processing method, and program |
US11929085B2 (en) | 2018-08-30 | 2024-03-12 | Dolby International Ab | Method and apparatus for controlling enhancement of low-bitrate coded audio |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DK2200347T3 (en) * | 2008-12-22 | 2013-04-15 | Oticon As | Method of operating a hearing instrument based on an estimate of the current cognitive load of a user and a hearing aid system and corresponding device |
US20120249797A1 (en) * | 2010-02-28 | 2012-10-04 | Osterhout Group, Inc. | Head-worn adaptive display |
US10405786B2 (en) * | 2013-10-09 | 2019-09-10 | Nedim T. SAHIN | Systems, environment and methods for evaluation and management of autism spectrum disorder using a wearable data collection device |
EP3140780B1 (en) * | 2014-05-09 | 2020-11-04 | Google LLC | Systems and methods for discerning eye signals and continuous biometric identification |
US10129659B2 (en) * | 2015-05-08 | 2018-11-13 | Doly International AB | Dialog enhancement complemented with frequency transposition |
US10242501B1 (en) * | 2016-05-03 | 2019-03-26 | WorldViz, Inc. | Multi-user virtual and augmented reality tracking systems |
US9953650B1 (en) * | 2016-12-08 | 2018-04-24 | Louise M Falevsky | Systems, apparatus and methods for using biofeedback for altering speech |
US10555106B1 (en) * | 2017-01-27 | 2020-02-04 | Facebook Technologies, Llc | Gaze-directed audio enhancement |
US20180357038A1 (en) * | 2017-06-09 | 2018-12-13 | Qualcomm Incorporated | Audio metadata modification at rendering device |
-
2018
- 2018-08-16 US US15/998,796 patent/US20190057694A1/en not_active Abandoned
-
2021
- 2021-03-25 US US17/213,088 patent/US20210264928A1/en active Pending
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11929085B2 (en) | 2018-08-30 | 2024-03-12 | Dolby International Ab | Method and apparatus for controlling enhancement of low-bitrate coded audio |
US11327636B2 (en) * | 2019-08-20 | 2022-05-10 | Dell Products L.P. | Dynamically scale complexity of a user interface based on cognitive load |
WO2021059745A1 (en) * | 2019-09-25 | 2021-04-01 | 株式会社Jvcケンウッド | Gaze data correction device, evaluation device, gaze data correction method, evaluation method, gaze data correction program, and evaluation program |
WO2023053557A1 (en) * | 2021-09-30 | 2023-04-06 | 富士フイルム株式会社 | Information processing device, information processing method, and program |
JP7499976B2 (en) | 2021-09-30 | 2024-06-14 | 富士フイルム株式会社 | Information processing device, information processing method, and program |
Also Published As
Publication number | Publication date |
---|---|
US20210264928A1 (en) | 2021-08-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20210264928A1 (en) | Speech/Dialog Enhancement Controlled by Pupillometry | |
JP7443423B2 (en) | Multichannel signal encoding method and encoder | |
EP3469584B1 (en) | Neural decoding of attentional selection in multi-speaker environments | |
CA2880126C (en) | Improving at least one of intelligibility or loudness of an audio program | |
EP3582514B1 (en) | Sound processing apparatus | |
JP6212567B2 (en) | System, computer-readable storage medium and method for recovering compressed audio signals | |
US9232904B2 (en) | Electroencephalogram recording apparatus, hearing aid, electroencephalogram recording method, and program thereof | |
EP3039675B1 (en) | Parametric speech enhancement | |
US8655656B2 (en) | Method and system for assessing intelligibility of speech represented by a speech signal | |
US8571873B2 (en) | Systems and methods for reconstruction of a smooth speech signal from a stuttered speech signal | |
JP7201721B2 (en) | Method and Apparatus for Adaptive Control of Correlation Separation Filter | |
JP5542206B2 (en) | Method and system for determining perceptual quality of an audio system | |
US11184723B2 (en) | Methods and apparatus for auditory attention tracking through source modification | |
US8793128B2 (en) | Speech signal processing system, speech signal processing method and speech signal processing method program using noise environment and volume of an input speech signal at a time point | |
CN106033671B (en) | Method and apparatus for determining inter-channel time difference parameters | |
CN115348507A (en) | Impulse noise suppression method, system, readable storage medium and computer equipment | |
JP5782402B2 (en) | Voice quality objective evaluation apparatus and method | |
CN114694678A (en) | Sound quality detection model training method, sound quality detection method, electronic device, and medium | |
Raghavan et al. | Distinct neural encoding of glimpsed and masked speech in multitalker situations | |
EP3444820B1 (en) | Speech/dialog enhancement controlled by pupillometry | |
KR20130139074A (en) | Method for processing audio signal and audio signal processing apparatus thereof | |
US11875813B2 (en) | Systems and methods for brain-informed speech separation | |
US20230360662A1 (en) | Method and device for processing a binaural recording | |
EP3662470B1 (en) | Audio object classification based on location metadata | |
US20130202124A1 (en) | Method for testing hearing aids |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: DOLBY INTERNATIONAL AB, NETHERLANDS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BISWAS, ARIJIT;REEL/FRAME:046856/0344 Effective date: 20170915 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |