US20220122623A1 - Real-Time Voice Timbre Style Transform - Google Patents
Real-Time Voice Timbre Style Transform Download PDFInfo
- Publication number
- US20220122623A1 US20220122623A1 US17/071,454 US202017071454A US2022122623A1 US 20220122623 A1 US20220122623 A1 US 20220122623A1 US 202017071454 A US202017071454 A US 202017071454A US 2022122623 A1 US2022122623 A1 US 2022122623A1
- Authority
- US
- United States
- Prior art keywords
- frequency
- bark
- response curve
- source
- domain
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000004044 response Effects 0.000 claims abstract description 84
- 230000001131 transforming effect Effects 0.000 claims abstract description 17
- 238000000034 method Methods 0.000 claims description 90
- 239000013074 reference sample Substances 0.000 claims description 14
- 238000004891 communication Methods 0.000 claims description 13
- 238000013507 mapping Methods 0.000 claims description 4
- 239000000523 sample Substances 0.000 description 26
- 230000015654 memory Effects 0.000 description 14
- 230000008859 change Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008447 perception Effects 0.000 description 3
- 230000005236 sound signal Effects 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 241000282412 Homo Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/003—Changing voice quality, e.g. pitch or formants
- G10L21/007—Changing voice quality, e.g. pitch or formants characterised by the process used
- G10L21/013—Adapting to target pitch
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/003—Changing voice quality, e.g. pitch or formants
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H1/00—Details of electrophonic musical instruments
- G10H1/0091—Means for obtaining special acoustic effects
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/80—Responding to QoS
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/003—Changing voice quality, e.g. pitch or formants
- G10L21/007—Changing voice quality, e.g. pitch or formants characterised by the process used
- G10L21/013—Adapting to target pitch
- G10L2021/0135—Voice conversion or morphing
Definitions
- This disclosure relates generally to speech enhancement and more specifically to voice timbre style transform in, for example, real-time applications.
- RTC real-time communication
- the video can include audio (e.g., speech, voice) and visual content.
- One user i.e., a sending user
- may transmit e.g., the video
- a concert may be live-streamed to many viewers.
- a teacher may live-stream a classroom session to students.
- a few users may hold a live chat session that may include live video.
- a user may wish to add filters, masks, and other visual effects to add an element of fun to the communications.
- a sunglasses filter which the communications application digitally adds to the user's face.
- users may wish to modify their voice. More specifically, users may wish to modify the timbre, or tone color, of their voice in an RTC session.
- a first aspect is a method for transforming a voice of a speaker to a reference timbre.
- the method includes converting a first portion of a source signal of the voice of the speaker into a time-frequency domain to obtain a time-frequency signal; obtaining frequency bin means of magnitudes over time of the time-frequency signal; converting the frequency bin magnitude means into a Bark domain to obtain a source frequency response curve (SR), where SR(i) corresponds to magnitude mean of the i th frequency bin; obtaining respective gains of frequency bins of the Bark domain with respect to a reference frequency response curve (Rf); obtaining equalizer parameters using the respective gains of the frequency bins of the Bark domain; and transforming the first portion to the reference timbre using the equalizer parameters.
- SR source frequency response curve
- a second aspect is an apparatus for transforming a voice of a speaker to a reference timbre.
- the apparatus includes a processor that is configured to convert a first portion of a source signal of the voice of the speaker into a time-frequency domain to obtain a time-frequency signal; obtain frequency bin means of magnitudes over time of the time-frequency signal; convert the frequency bin magnitude means into a Bark domain to obtain a source frequency response curve (SR), where SR(i) corresponds to magnitude mean of the i th frequency bin; obtain respective gains of frequency bins of the Bark domain with respect to a reference frequency response curve (Rf); obtain equalizer parameters using the respective gains of the frequency bins of the Bark domain; and transform the first portion to the reference timbre using the equalizer parameters.
- SR source frequency response curve
- aspects can be implemented in any convenient form.
- aspects may be implemented by appropriate computer programs which may be carried on appropriate carrier media which may be tangible carrier media (e.g. disks) or intangible carrier media (e.g. communications signals).
- aspects may also be implemented using suitable apparatus which may take the form of programmable computers running computer programs arranged to implement the methods and/or techniques disclosed herein. Aspects can be combined such that features described in the context of one aspect may be implemented in another aspect.
- FIG. 1 is a diagram of an example of a technique of a prepare phase of timbre style transform according to implementations of this disclosure.
- FIG. 2 illustrates the Bark filter bank according to implementations of this disclosure.
- FIG. 3 is a diagram of an example of a technique of a real-time phase of timbre style transform according to implementations of this disclosure.
- FIG. 4 is a block diagram of an example of a computing device in accordance with implementations of this disclosure.
- FIG. 5 is an example of a flowchart of a technique for transforming a voice of a speaker to a target timbre according to an implementation of this disclosure.
- Timbre also known as tone color
- tone color is the distinguishing characteristic that differentiates one sound from another. For example, while two instruments (e.g., a piano and a violin) may be playing the same note at the same frequency and at the same frequency amplitude, the note will be heard differently. Words such as sharp, round, reedy, brassy, bright, magnetic, vigorous, light, flat, smooth, smoky, breathy, rough, or fresh can be used to describe timbre.
- timbres Different people or music styles have different timbres. Put more simply, different people and different music styles sound differently. A person may wish to sound different. That is, the person may wish to change his/her timbre, such as during an RTC session.
- the timbre of a voice (or sound) can be understood to be comprised of different energy levels in different frequency bands.
- timbre such as, of a recorded sound
- Professional audio producers such as broadcasters or music makers, often use sophisticated hardware or software equalizers to change the timbre of different voices or instruments in a recording.
- a composer may record, in multiple tracks, all portions of an orchestral composition using one instrument.
- the timbre of each track can be modified to be that of the target instrument of that track.
- Implementations according to this disclosure can be used to transform the timbre of a voice, such as the voice of a user in a real-time communication application.
- a voice such as the voice of a user
- RTC there can be a sending user and a receiving user.
- An audio stream such as the voice of sending user, can be sent from a sending device of the sending user to a receiving device of the receiving user.
- the sending user may wish to change the timbre of his/her voice to a certain (e.g., reference, desired, etc.) style or the receiving user may wish to change the timbre of the sending user's voice to that certain style.
- the techniques described herein can be used at a sending user's device (i.e., a sending device), a receiving user's device (e.g., a receiving device), or both.
- the sending user as used herein, is a person who may be speaking and whose speech is to be transmitted to and heard by the receiving user.
- the techniques described can also be employed by a central server (e.g., a cloud-based server) that may receive an audio signal from a sending user and relay the audio signal to the receiving user.
- the sending user can select a timbre style that the sending user's voice is to be transformed to prior to sending to the receiving user.
- the receiving user via a user interface of an RTC application that may be used by the receiving user using the receiving device, the receiving user can select a timbre style that the sending user's voice is to be transformed to prior to being heard by (i.e., output to) the receiving user.
- the user may wish to transform the timbre to a certain style to fit a certain situation, such as news reporting or music style (e.g., jazz, hip-hop, etc.).
- Transforming the timbre of a speaker (i.e., the voice of the speaker) to a reference (e.g., desired, target, selected, etc.) timbre includes a setup (e.g., prepare, train, etc.) phase and a real-time phase.
- a setup phase e.g., prepare, train, etc.
- a reference frequency response curve for a target (e.g., reference, etc.) voice timbre style is generated.
- a source voice timbre can also be described by a source domain frequency response curve.
- the difference between the source frequency response curve of the source voice timbre and the reference frequency response curve of the reference voice timbre can be used by a mapping technique, as further described below, to obtain parameters of an equalizer that is then applied to the source voice.
- a reference sample of the target timbre can be received and a Bark frequency response curve can be obtained from the reference sample; in the real-time phase, the Bark frequency response curve can be used, in real-time, to transform a source voice sample (e.g., frames of the source voice sample) of the speaker to the target timbre.
- a source voice sample e.g., frames of the source voice sample
- the Bark transform is the result of psychoacoustic experiments and is defined so that the critical bands of human hearing each has a width of one Bark.
- the Bark scale represents the spectral information processing in the human ear.
- the Bark domain reflects the psychoacoustic frequency response thereby providing better information on how humans recognize the power difference in the different frequency bands.
- MEL scale may be used.
- Bark scale reflects a human's subjective loudness perception and energy integration.
- energy distribution in different frequency bands may be more relevant to timbre transform (e.g., change) than pitch.
- a constant-parameter equalizer may not be suitable for long-term use. That is, a constant-parameter equalizer may not be suitable for use for the duration of an RTC session.
- the timbre of the speaker may change five minutes into an RTC session, such as due to emotion or to a changing singing style; or another person, with a different timbre altogether, may start talking instead of the original speaker.
- Such change in timbre may require a dynamic change to the parameters of the equalizer so that the changed timbre style can still be transformed to the target timbre style.
- the speaker's timbre changes during an RTC session the changing timbre can still be changed to the target timbre. Accordingly, parameters of the equalizer can be dynamically updated.
- the disclosure herein mainly describes the transformation of the timbre of a single voice or sound.
- techniques such as voice source separation can be used to separate the voices and apply timbre transform as described herein to each voice separately.
- the source voice may be noisy or reverberant.
- denoising and/or dereverberation techniques can be applied to the source voice prior to transforming the timbre as described herein.
- FIG. 1 is a diagram of an example of a technique 100 of a prepare (e.g., setup) phase of timbre style transform according to implementations of this disclosure.
- the technique 100 receives a reference sample of a target timbre style and generates a reference (e.g., target) frequency response curve of the target timbre.
- the technique 100 can be used off-line to generate the reference frequency response curve.
- a speaker may wish his/her sound to be like that of the singer Justin Bieber; thus, a reference voice sample of the singer can be used as the target timbre style sample.
- the user may wish to sound vigorous during RTC sessions; thus, a recording of a vigorous sound can be used as the reference sample.
- the technique 100 can be repeated for each desired (e.g., reference) timbre style to generate a corresponding reference frequency response curve (Rf).
- Rf reference frequency response curve
- a male reference sample and a female reference sample can be used to obtain two frequency response curves of the desired timbre.
- the lengths of the two samples i.e., the sample of the male voice and the sample of the female voice
- the technique 100 receives a reference voice sample (i.e., a reference signal) of the desired (i.e., target) timbre style.
- the reference voice sample can include at least one period of vocal wave signals.
- the reference voice sample can be in any format.
- the voice sample can be a waveform audio file (wave or way file), an MP3 file, a window media audio (wma), an audio interchange file format (aiff), or the like.
- the reference voice sample can be a few (e.g., 0.5, 1, 2, 5, more, or fewer) minutes in length.
- the technique 100 can receive a longer voice sample from which a shorter reference voice sample is extracted.
- the technique 100 transforms the means of magnitudes in the time dimension of the time-frequency domain signal to the Bark domain to obtain the reference frequency response curve (Rf) 108 , which is psychoacoustic frequency response curve.
- the time-domain results of the STFT can be visualized on a spectrogram, such as the merely illustrative spectrogram 120 of FIG. 1 .
- the spectrogram 120 shows the frequency content of signals when that frequency content varies with time. Time is shown on the x-axis of the spectrogram 120 ; frequency is shown on a y-axis of the spectrogram 120 ; and the frequency magnitudes are typically indicated by color intensities (i.e., gray scale levels in the spectrogram 120 ).
- the mean of magnitudes M j FFT can be, as the name implies, the mean of at least a subset (e.g., all) of the magnitudes of the frequency bin B j over all the time windows (i.e., the time axis, the horizontal dimension).
- each M j FFT represents an average frequency magnitude response of the frequency bin B k .
- the means of magnitudes are converted (i.e., transformed, mapped, etc.) from the STFT domain to the i th Bark domain magnitude M i Bark )) though mapping magnitudes of the FFT frequency bins (M j FFT ) to bark frequency bins using formula (1):
- M i Bark ⁇ j ⁇ B i ⁇ ij *M j FFT (1)
- Formula (1) represents a conversion from the Fourier to the Bark domain.
- the Bark scale can range from 1 to 24, corresponding to the first 24 critical bands of hearing.
- the Bark band edges are given, in Hertz (Hz), as [0, 100, 200, 300, 400, 510, 630, 770, 920, 1080, 1270, 1480, 1720, 2000, 2320, 2700, 3150, 3700, 4400, 5300, 6400, 7700, 9500, 12000, 15500]; and the band centers in Hertz are [50, 150, 250, 350, 450, 570, 700, 840, 1000, 1170, 1370, 1600, 1850, 2150, 2500, 2900, 3400, 4000, 4800, 5800, 7000, 8500, 10500, 13500].
- i ranges from 1 to 24.
- the Bark scale used can contain 109 bins. As such, i can range from 1 to 109 and the whole frequency can range from 0 to 24000 Hz.
- FIG. 2 illustrates the Bark filter bank 200 according to implementations of this disclosure. So as not to not overly clutter FIG. 2 , it is noted that the Bark filter bank 200 illustrates only 29 bins and a frequency range of 0 to 8000 Hz. The number of triangles in FIG. 2 should equal to number of bins.
- the filter bank 200 is used to illustrate how the coefficients ⁇ ij of formula (1) are obtained.
- the coefficient ⁇ ij are the Bark transform coefficients of the STFT.
- the index i corresponds to a Bark frequency bin and the index j corresponds to an FFT frequency bin.
- the index j corresponds to the x-axis in FIG. 2 ; and the index i corresponds to a frequency.
- Each coefficient ⁇ ij is determined in two dimensions: it is determined first by which triangle is to be used and second by which frequency bin the M j FFT corresponds to.
- Each of the Bark filters is a triangular band-pass filter, such as the filter 202 , with certain overlaps.
- the peeks of the Bark filter bank 200 such as a peek 201 , correspond to the center frequencies of the different Bark filters. It is to be noted that, in FIG. 2 , some triangles have are drawn to have thicker lines than other triangles. This is merely so as not to clutter the figure. No meaning should be ascribed to the fact that some triangles are drawn with thinner sides.
- the triangle 204 corresponds, approximately, to the frequency band 4200 Hz to 6300 Hz; and triangle 206 corresponds, approximately, to the frequency band 5300 Hz to 7000 Hz.
- FIG. 3 is a diagram of an example of a technique 300 of a real-time phase of timbre style transform according to implementations of this disclosure.
- the technique 300 can be used in real-time applications, such as audio and/or video conferencing, telephone conversations, and the like, to transform the timbre of a source voice of at least one of the participants.
- the technique 300 receives the source voice in frames, such as a source voice frame 302 .
- the technique 300 itself can partition a received audio signal into the frames.
- a frame can correspond to an m number of milliseconds of audio. In an example, m can be 20 milliseconds. However, other values of m are possible.
- the technique 300 outputs (e.g., generates, obtains, results in, calculates, etc.) a transformed voice frame 306 .
- the source voice frame 302 is in a source timbre style and the transformed voice frame 306 that is in a reference timbre style.
- the technique 300 can be implemented by a computing device, such as the computing device 400 described with respect to FIG. 4 .
- the technique 300 can be implemented by a sending device.
- the timbre style of the speaker can be transformed to a reference timbre on the device of the sending user, before transmission to a receiving user, so that the receiving user can receive the voice of the sending user in the reference timbre.
- the technique 300 can be implemented by a receiving device.
- the voice received at the receiving device of a receiving user can be transformed to a reference timbre that may be selected by the receiving user.
- the technique 300 can be performed on the received speech to produce transformed speech with the reference timbre.
- the transformed speech is then output to the receiving user.
- the technique 300 can be implemented by a central server, which receives a voice sample in a source timbre from a sending device, performs the technique 300 to obtain a voice in a reference (e.g., desired, etc.) timbre, and transmit (e.g., forward, relay, etc.) the transformed speech to one or more receiving devices.
- a central server which receives a voice sample in a source timbre from a sending device, performs the technique 300 to obtain a voice in a reference (e.g., desired, etc.) timbre, and transmit (e.g., forward, relay, etc.) the transformed speech to one or more receiving devices.
- the source voice frame 302 can be processed via an equalizer 304 to produce the transformed voice frame 306 .
- the equalizer 304 transforms the timbre using equalizer parameters, which are initially calculated and later updated, upon detection of large variations, as described below.
- the technique 300 obtains (e.g., calculates, looks up, determines, etc.) the gap between the reference frequency response curve (Rf) of the reference sample and the source frequency response curve (SR) of the source sample.
- the technique 300 can obtain the gap (e.g., difference) in each of the frequency bins. That is, the technique 300 can obtain the gain(s) in amplification between the reference frequency response curve (Rf) of the reference sample and the source frequency response curve (SR) of the source sample.
- the gain can be obtained in the logarithmic scale.
- the gap can be obtained in decibels (dB). As is known, a decibel (dB) is a ratio between two quantities reported on a logarithmic scale and allows for a realistic modelling of human auditory perception.
- the technique 300 can calculate the dB difference, G b (k), between the source and the reference psychoacoustic frequency response curves using formula (2).
- formula (2) can be used to measure the gain in amplification, in each of the Bark frequency bins, between reference frequency response curve (Rf) and the source frequency response curve (SR).
- the set of gains G b (k) for all of the Bark domain frequency bins can constitute (e.g., can be considered to be, can be the basis for obtaining, etc.) the parameters of the equalizer 304 .
- the equalizer 304 uses the equalizer parameters to transform the timbre of the source voice to the reference timbre style.
- the equalizer 304 is a set of filters.
- the equalizer 304 can have a filter for a lower frequency f n (e.g., 0 Hz) to upper frequency f n+1 (e.g., 800 Hz) band, which has a center frequency of (f n +f n+1 )/2 (e.g., 400 Hz).
- the equalizer 304 can use the equalizer parameters (i.e., the gains G b (k)) that determine how much to add to or subtract from the center frequency to adjust the center frequency.
- Interpolations parameters which calculate the adjusted center frequency as an interpolation between the lower and upper frequencies of the frequency band, can then be determined.
- the interpolation parameters can also include (e.g., determine, define, etc.) a shape of the interpolation.
- the interpolation can be a cubic or cubic spline interpolation. Cubic spline interpolation can result in smoother interpolation than, for example, linear interpolation.
- the cubic spline interpolation method used to obtain an interpolation value of the i th gain, G i e can be described by the following equation (3).
- the interpolation parameters a i to d i are determined by the G b (i) near to the i th center frequency of equalizer.
- the equalizer 304 can include (e.g., use) an initial set of equalizer parameters.
- the initial set of equalizer parameters may be obtained from previous executions of the technique 300 .
- a store 322 can include stored reference response curve(s), stored source frequency response curve(s), and/or corresponding equalizer parameters.
- the store 322 can include a reference frequency response curve 322 of the reference the reference timbre style.
- the store 322 can be a permanent storage (e.g., a database, a file, etc.) or non-permanent memory.
- the equalizer 304 may not include equalizer parameters.
- Initial equalizer parameters can be obtained as described below with respect to 314 - 318 .
- the technique 300 can normalize the gains to keep the volumes of voice before and after equalizing by the equalizer 304 at the same (or roughly the same) levels.
- normalizing the gains can mean dividing each of the gains by the sum of all the gains. However, other normalizing techniques may be used.
- the technique 300 can perform the operations 308 - 318 to obtain initial equalizer parameters and when large variations (described below) are detected.
- the source voice frame 302 may be received into a signal buffer 308 , which may store received voice frames until a period of source voice samples is available for further processing.
- the period of the source audio can be 30 seconds, 1 minute, 2 minutes, longer, or shorter period.
- the technique 300 converts the voice sample to the transform domain, such as described with respect to 104 of FIG. 1 .
- the voice sample i.e., the period of the source audio
- the technique 300 converts means of magnitudes in the time dimension of the time-frequency domain signal to the Bark domain to obtain the source frequency response curve (SR).
- the source frequency response curve (SR) can be obtained as described with respect to the reference frequency response curve (Rf) and 106 of FIG. 1 .
- the source frequency response curve (SR) can be a collection of Bark domain magnitudes, M i source Bark , of the source sample.
- the technique 314 determines whether a large difference in source voice timbre occurs.
- the difference can be obtained at 316 .
- the source voice may be that of a first speaker (e.g., a 45-year old male).
- a second speaker e.g., a 7-year old female
- the technique 300 can replace the source frequency response curve (initially obtained for the first speaker) with that of the second speaker.
- the technique 300 can replace the source frequency response curve only when there is a large variation between a stored source frequency response curve and a current source frequency response curve.
- a large variation can be determined at 314 when the equalizer parameters have not yet been obtained (e.g., initialized).
- the technique 300 proceeds to 304 where the previous equalizer parameters are used. However, if there is a large variation, at 314 , then the technique 300 stores the current source frequency response curve in the store 322 so that the current source frequency response curve can be compared to subsequent source frequency response curve to detect any subsequent large variations; the technique 300 also proceeds to 318 to update the equalizer parameters. That is, the technique 300 obtains the interpolations parameters, as described with respect to formula (3).
- a relation threshold can be designed for large variation detection at 314 .
- a relation coefficient can be calculated between the frequency response curve of the current period and stored, such as in the store 322 . If the relation coefficient is larger than a threshold, the stored frequency response curve will be replaced by the current one, and the parameters of an equalizer will be updated. Otherwise, the equalizer and stored frequency response curve will not be updated.
- timbre style transform As the updating of the equalizer parameters can be completed (e.g., performed, finished, etc.) within one frame of the source voice signal (e.g., 10 ms), timbre style transform according to implementations of this disclosure is not interrupted due to the updating of the equalizer parameters. That is, delays or discontinuous are experienced when the equalizer parameters are updated.
- FIG. 4 is a block diagram of an example of a computing device 400 in accordance with implementations of this disclosure.
- the computing device 400 can be in the form of a computing system including multiple computing devices, or in the form of one computing device, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, and the like.
- a processor 402 in the computing device 400 can be a conventional central processing unit.
- the processor 402 can be another type of device, or multiple devices, capable of manipulating or processing information now existing or hereafter developed.
- the disclosed implementations can be practiced with one processor as shown (e.g., the processor 402 ), advantages in speed and efficiency can be achieved by using more than one processor.
- a memory 404 in computing device 400 can be a read only memory (ROM) device or a random access memory (RAM) device in an implementation. However, other suitable types of storage devices can be used as the memory 404 .
- the memory 404 can include code and data 406 that are accessed by the processor 402 using a bus 412 .
- the memory 404 can further include an operating system 408 and application programs 410 , the application programs 410 including at least one program that permits the processor 402 to perform at least some of the techniques described herein.
- the application programs 410 can include applications 1 through N, which further include applications and techniques useful in real-time voice timbre style transform.
- the application programs 410 can include the technique 100 or aspects thereof, to implement a training phase.
- the application programs 410 can include the technique 300 or aspects thereof to implement real-time voice timbre style transform.
- the computing device 400 can also include a secondary storage 414 , which can, for example, be a memory card used with a mobile computing device.
- the computing device 400 can also include one or more output devices, such as a display 418 .
- the display 418 may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs.
- the display 418 can be coupled to the processor 402 via the bus 412 .
- Other output devices that permit a user to program or otherwise use the computing device 400 can be provided in addition to or as an alternative to the display 418 .
- the output device is or includes a display
- the display can be implemented in various ways, including by a liquid crystal display (LCD), a cathode-ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.
- LCD liquid crystal display
- CRT cathode-ray tube
- LED light emitting diode
- OLED organic LED
- the computing device 400 can also include or be in communication with an image-sensing device 420 , for example, a camera, or any other image-sensing device 420 now existing or hereafter developed that can sense an image such as the image of a user operating the computing device 400 .
- the image-sensing device 420 can be positioned such that it is directed toward the user operating the computing device 400 .
- the position and optical axis of the image-sensing device 420 can be configured such that the field of vision includes an area that is directly adjacent to the display 418 and from which the display 418 is visible.
- the computing device 400 can also include or be in communication with a sound-sensing device 422 , for example, a microphone, or any other sound-sensing device now existing or hereafter developed that can sense sounds near the computing device 400 .
- the sound-sensing device 422 can be positioned such that it is directed toward the user operating the computing device 400 and can be configured to receive sounds, for example, speech or other utterances, made by the user while the user operates the computing device 400 .
- the computing device 400 can also include or be in communication with a sound-playing device 424 , for example, a speaker, a headset, or any other sound-playing device now existing or hereafter developed that can play sounds as directed by the computing device 400 .
- FIG. 4 depicts the processor 402 and the memory 404 of the computing device 400 as being integrated into one unit, other configurations can be utilized.
- the operations of the processor 402 can be distributed across multiple machines (wherein individual machines can have one or more processors) that can be coupled directly or across a local area or other network.
- the memory 404 can be distributed across multiple machines such as a network-based memory or memory in multiple machines performing the operations of the computing device 400 .
- the bus 412 of the computing device 400 can be composed of multiple buses.
- the secondary storage 414 can be directly coupled to the other components of the computing device 400 or can be accessed via a network and can comprise an integrated unit such as a memory card or multiple units such as multiple memory cards.
- the computing device 400 can thus be implemented in a wide variety of configurations.
- FIG. 5 is an example of a flowchart of a technique 500 for transforming a voice of a speaker to a reference timbre according to an implementation of this disclosure.
- the technique 500 can receive an audio sample, such as an voice stream.
- the audio stream can be part of a video stream.
- the technique 500 can receive frames of the audio stream for processing.
- the technique 500 can partition the audio sample into frames and process each frame separately as further described below and consistent with the description of the technique 300 of FIG. 3 .
- the technique 500 can be implemented by a computing device (e.g., an apparatus), such as the computing device 400 of FIG. 4 .
- the technique 500 can be implemented, for example, as a software program that may be executed by computing devices, such as the computing device 400 of FIG. 4 .
- the software program can include machine-readable instructions that may be stored in a memory such as the memory 404 or the secondary storage 414 , and that, when executed by a processor, such as CPU 402 , may cause the computing device to perform the technique 500 .
- the technique 500 can be implemented using specialized hardware or firmware. Multiple processors, memories, or both, may be used.
- the reference timbre can be received from the speaker.
- the technique 500 converts a portion of a source signal of the voice of the speaker into a time-frequency domain to obtain a time-frequency signal, as described above.
- the technique 500 obtains frequency bin means of magnitudes over time of the time-frequency signal, as described above with respect to M k FFT .
- the technique 300 converts the frequency bin magnitude means into a bark domain to obtain a source frequency response curve (SR), as described above.
- An SR(i) corresponds to magnitude mean of the i th frequency bin.
- the technique 500 obtains respective gains of frequency bins of the bark domain with respect to a reference frequency response curve (Rf).
- the reference frequency response curve (Rf) can be obtained as described above.
- the technique 300 can include, as described above, receiving a reference sample of the reference timbre; converting the reference sample into the time-frequency domain to obtain a reference time-frequency signal; obtaining reference frequency bin means of magnitudes (M j FFT ) over time of the reference time-frequency signal; and converting the reference frequency bin means of magnitudes (M j FFT ) into the bark domain to obtain the reference frequency response curve (Rf).
- the reference frequency response curve (Rf) includes respective bark domain frequency magnitudes (M i Bark ) for respective bark domain frequency bins, i. As such, an Rf(i) corresponds to a magnitude mean of the i th frequency bin.
- the technique 500 can convert the reference frequency bin means of magnitudes (M j FFT ) into the bark domain to obtain a reference frequency response curve (Rf) using formula (1).
- obtaining respective gains of frequency bins of the bark domain can include calculating a gain G b (k) of a k th frequency bin in the bark domain using a ratio of the reference frequency bin magnitude mean of the k th frequency bin to the source frequency response curve (SR) of the k th frequency bin.
- the gain G b (k) can be calculated using formula (2).
- the technique 500 can obtain equalizer parameters using the respective gains of the frequency bins of the Bark domain.
- obtaining the equalizer parameters using the respective gains of the frequency bins of the bark domain further can include mapping the respective gains to respective center frequencies of the equalizer to obtain values for gains of the equalizer.
- the technique 500 can normalize the respective gains to obtain the equalizer parameters.
- the technique 500 transforms the first portion to the reference timbre using the equalizer parameters.
- the gain for each frequency band of the equalizer can be an interpolated gain G i e , derived using formula (3).
- the technique 500 can further include obtaining a second source frequency response curve for a second portion of the source signal; in response to detecting a difference between the source frequency response curve and the second source frequency response curve exceeding a threshold, obtaining new equalizer parameters and using the new equalizer parameters as the equalizer parameters; and transforming the second portion of the source signal using the equalizer parameters, which may be the new equalizer parameters if a large variation is detected.
- example is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” is not necessarily to be construed as being preferred or advantageous over other aspects or designs. Rather, use of the word “example” is intended to present concepts in a concrete fashion.
- the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise or clearly indicated otherwise by the context, the statement “X includes A or B” is intended to mean any of the natural inclusive permutations thereof. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances.
- Implementations of the computing device 400 can be realized in hardware, software, or any combination thereof.
- the hardware can include, for example, computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuit.
- IP intellectual property
- ASICs application-specific integrated circuits
- programmable logic arrays optical processors
- programmable logic controllers microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuit.
- processor should be understood as encompassing any of the foregoing hardware, either singly or in combination.
- signal and “data” are used interchangeably.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Circuit For Audible Band Transducer (AREA)
Abstract
Description
- None.
- This disclosure relates generally to speech enhancement and more specifically to voice timbre style transform in, for example, real-time applications.
- Many interactions occur online over different communication channels and via many media types. An example of such interactions is real-time communication (RTC) using video conferencing or streaming or a simple telephone voice calls. The video can include audio (e.g., speech, voice) and visual content. One user (i.e., a sending user) may transmit (e.g., the video) to one or more receiving users. For example, a concert may be live-streamed to many viewers. For example, a teacher may live-stream a classroom session to students. For example, a few users may hold a live chat session that may include live video.
- In real-time communications, some users may wish to add filters, masks, and other visual effects to add an element of fun to the communications. To illustrate, a user can select a sunglasses filter, which the communications application digitally adds to the user's face. Similarly, users may wish to modify their voice. More specifically, users may wish to modify the timbre, or tone color, of their voice in an RTC session.
- A first aspect is a method for transforming a voice of a speaker to a reference timbre. The method includes converting a first portion of a source signal of the voice of the speaker into a time-frequency domain to obtain a time-frequency signal; obtaining frequency bin means of magnitudes over time of the time-frequency signal; converting the frequency bin magnitude means into a Bark domain to obtain a source frequency response curve (SR), where SR(i) corresponds to magnitude mean of the ith frequency bin; obtaining respective gains of frequency bins of the Bark domain with respect to a reference frequency response curve (Rf); obtaining equalizer parameters using the respective gains of the frequency bins of the Bark domain; and transforming the first portion to the reference timbre using the equalizer parameters.
- A second aspect is an apparatus for transforming a voice of a speaker to a reference timbre. The apparatus includes a processor that is configured to convert a first portion of a source signal of the voice of the speaker into a time-frequency domain to obtain a time-frequency signal; obtain frequency bin means of magnitudes over time of the time-frequency signal; convert the frequency bin magnitude means into a Bark domain to obtain a source frequency response curve (SR), where SR(i) corresponds to magnitude mean of the ith frequency bin; obtain respective gains of frequency bins of the Bark domain with respect to a reference frequency response curve (Rf); obtain equalizer parameters using the respective gains of the frequency bins of the Bark domain; and transform the first portion to the reference timbre using the equalizer parameters.
- A third aspect is a non-transitory computer-readable storage medium that includes executable instructions that, when executed by a processor, facilitate performance of operations including converting a first portion of a source signal of the voice of the speaker into a time-frequency domain to obtain a time-frequency signal; obtaining frequency bin means of magnitudes over time of the time-frequency signal; converting the frequency bin magnitude means into a Bark domain to obtain a source frequency response curve (SR), where SR(i) corresponds to magnitude mean of the ith frequency bin; obtaining respective gains of frequency bins of the Bark domain with respect to a reference frequency response curve (Rf); obtaining equalizer parameters using the respective gains of the frequency bins of the Bark domain; and transforming the first portion to the reference timbre using the equalizer parameters.
- It will be appreciated that aspects can be implemented in any convenient form. For example, aspects may be implemented by appropriate computer programs which may be carried on appropriate carrier media which may be tangible carrier media (e.g. disks) or intangible carrier media (e.g. communications signals). Aspects may also be implemented using suitable apparatus which may take the form of programmable computers running computer programs arranged to implement the methods and/or techniques disclosed herein. Aspects can be combined such that features described in the context of one aspect may be implemented in another aspect.
- The description herein makes reference to the accompanying drawings wherein like reference numerals refer to like parts throughout the several views.
-
FIG. 1 is a diagram of an example of a technique of a prepare phase of timbre style transform according to implementations of this disclosure. -
FIG. 2 illustrates the Bark filter bank according to implementations of this disclosure. -
FIG. 3 is a diagram of an example of a technique of a real-time phase of timbre style transform according to implementations of this disclosure. -
FIG. 4 is a block diagram of an example of a computing device in accordance with implementations of this disclosure. -
FIG. 5 is an example of a flowchart of a technique for transforming a voice of a speaker to a target timbre according to an implementation of this disclosure. - Timbre, also known as tone color, is the distinguishing characteristic that differentiates one sound from another. For example, while two instruments (e.g., a piano and a violin) may be playing the same note at the same frequency and at the same frequency amplitude, the note will be heard differently. Words such as sharp, round, reedy, brassy, bright, magnetic, vigorous, light, flat, smooth, smoky, breathy, rough, or fresh can be used to describe timbre.
- Different people or music styles have different timbres. Put more simply, different people and different music styles sound differently. A person may wish to sound different. That is, the person may wish to change his/her timbre, such as during an RTC session. The timbre of a voice (or sound) can be understood to be comprised of different energy levels in different frequency bands.
- Changing the timbre, such as, of a recorded sound is possible. Professional audio producers, such as broadcasters or music makers, often use sophisticated hardware or software equalizers to change the timbre of different voices or instruments in a recording. To illustrate, a composer may record, in multiple tracks, all portions of an orchestral composition using one instrument. Using an equalizer, the timbre of each track can be modified to be that of the target instrument of that track.
- Using an equalizer amounts to finding equalizer parameters for tuning different aspects of an audio spectrum. Such parameters include gains (e.g., amplitudes) of certain frequency bands, center frequency (e.g., adjusting the center frequency ranges of selected frequency bands), bandwidth, filter slopes (e.g., steepness of a filter when selecting either a low cut or a high cut filter), tilter types (e.g., filter shapes for the selected frequency bands), and the like. For example, with respect to gain, a center frequency can be cut or boosted by a certain number of decibels (dB). Bandwidth refers to the frequency range located on either side of a center frequency. When a particular frequency is altered, other frequencies that are above and below the particular frequency are typically also affected. The range of affected frequencies is referred as the bandwidth. With respect to filter types, many filter types may be available and can include low cut, high cut, low shelf, high shelf, notch, bell, or other filter types.
- As can be appreciated from the foregoing high level and simplified description, using an equalizer can be complicated, beyond the reach of an average user, and impractical in real-time applications.
- Implementations according to this disclosure can be used to transform the timbre of a voice, such as the voice of a user in a real-time communication application. As is known, in RTC, there can be a sending user and a receiving user. An audio stream, such as the voice of sending user, can be sent from a sending device of the sending user to a receiving device of the receiving user. The sending user may wish to change the timbre of his/her voice to a certain (e.g., reference, desired, etc.) style or the receiving user may wish to change the timbre of the sending user's voice to that certain style.
- The techniques described herein can be used at a sending user's device (i.e., a sending device), a receiving user's device (e.g., a receiving device), or both. The sending user, as used herein, is a person who may be speaking and whose speech is to be transmitted to and heard by the receiving user. The techniques described can also be employed by a central server (e.g., a cloud-based server) that may receive an audio signal from a sending user and relay the audio signal to the receiving user.
- For example, via a user interface of an RTC application that may be used by the sending user using the sending device, the sending user can select a timbre style that the sending user's voice is to be transformed to prior to sending to the receiving user. Similarly, via a user interface of an RTC application that may be used by the receiving user using the receiving device, the receiving user can select a timbre style that the sending user's voice is to be transformed to prior to being heard by (i.e., output to) the receiving user. The user may wish to transform the timbre to a certain style to fit a certain situation, such as news reporting or music style (e.g., jazz, hip-hop, etc.).
- Transforming the timbre of a speaker (i.e., the voice of the speaker) to a reference (e.g., desired, target, selected, etc.) timbre includes a setup (e.g., prepare, train, etc.) phase and a real-time phase. In the setup phase, a reference frequency response curve for a target (e.g., reference, etc.) voice timbre style is generated. In the real-time phase, a source voice timbre can also be described by a source domain frequency response curve. The difference between the source frequency response curve of the source voice timbre and the reference frequency response curve of the reference voice timbre can be used by a mapping technique, as further described below, to obtain parameters of an equalizer that is then applied to the source voice.
- For example, in the setup phase, a reference sample of the target timbre can be received and a Bark frequency response curve can be obtained from the reference sample; in the real-time phase, the Bark frequency response curve can be used, in real-time, to transform a source voice sample (e.g., frames of the source voice sample) of the speaker to the target timbre.
- As is known, the Bark transform is the result of psychoacoustic experiments and is defined so that the critical bands of human hearing each has a width of one Bark. The Bark scale represents the spectral information processing in the human ear. Stated differently, the Bark domain reflects the psychoacoustic frequency response thereby providing better information on how humans recognize the power difference in the different frequency bands.
- Other perceptual transforms, or scales, can also be used. For example, the MEL scale may be used. Whereas the MEL scale reflects a human's perception of pitch, the Bark scale reflects a human's subjective loudness perception and energy integration. However, energy distribution in different frequency bands may be more relevant to timbre transform (e.g., change) than pitch.
- In some situations, a constant-parameter equalizer may not be suitable for long-term use. That is, a constant-parameter equalizer may not be suitable for use for the duration of an RTC session. To illustrate, the timbre of the speaker may change five minutes into an RTC session, such as due to emotion or to a changing singing style; or another person, with a different timbre altogether, may start talking instead of the original speaker. Such change in timbre may require a dynamic change to the parameters of the equalizer so that the changed timbre style can still be transformed to the target timbre style. As such, if the speaker's timbre changes during an RTC session, the changing timbre can still be changed to the target timbre. Accordingly, parameters of the equalizer can be dynamically updated.
- The disclosure herein mainly describes the transformation of the timbre of a single voice or sound. In the case of multiple voices, techniques such as voice source separation can be used to separate the voices and apply timbre transform as described herein to each voice separately. Additionally, the source voice may be noisy or reverberant. In some examples, denoising and/or dereverberation techniques can be applied to the source voice prior to transforming the timbre as described herein.
-
FIG. 1 is a diagram of an example of atechnique 100 of a prepare (e.g., setup) phase of timbre style transform according to implementations of this disclosure. Thetechnique 100 receives a reference sample of a target timbre style and generates a reference (e.g., target) frequency response curve of the target timbre. Thetechnique 100 can be used off-line to generate the reference frequency response curve. To illustrate, and without loss of generality, a speaker may wish his/her sound to be like that of the singer Justin Bieber; thus, a reference voice sample of the singer can be used as the target timbre style sample. As another example, the user may wish to sound vigorous during RTC sessions; thus, a recording of a vigorous sound can be used as the reference sample. - The
technique 100 can be repeated for each desired (e.g., reference) timbre style to generate a corresponding reference frequency response curve (Rf). In an example, as gender differences may have a large influence on the timbre, for the same desired timbre, a male reference sample and a female reference sample can be used to obtain two frequency response curves of the desired timbre. The lengths of the two samples (i.e., the sample of the male voice and the sample of the female voice) can be the same or can be different. - At 102, the
technique 100 receives a reference voice sample (i.e., a reference signal) of the desired (i.e., target) timbre style. The reference voice sample can include at least one period of vocal wave signals. The reference voice sample can be in any format. In an example, the voice sample can be a waveform audio file (wave or way file), an MP3 file, a window media audio (wma), an audio interchange file format (aiff), or the like. The reference voice sample can be a few (e.g., 0.5, 1, 2, 5, more, or fewer) minutes in length. In an example, thetechnique 100 can receive a longer voice sample from which a shorter reference voice sample is extracted. - At 104, the
technique 100 converts the reference voice sample to the transform domain. Thetechnique 100 can use the short-time Fourier transform (STFT) to convert the reference signal to the time-frequency domain. The STFT can be used to obtain the magnitudes of each frequency in the reference voice sample over time. As is known, the STFT calculates the Fast Fourier Transform (FFT) over a defined window length and a hop length, representing a number of samples of the voice sample, and producing both magnitude and phase information over time. - At 106, the
technique 100 transforms the means of magnitudes in the time dimension of the time-frequency domain signal to the Bark domain to obtain the reference frequency response curve (Rf) 108, which is psychoacoustic frequency response curve. - As is known, the time-domain results of the STFT can be visualized on a spectrogram, such as the merely
illustrative spectrogram 120 ofFIG. 1 . Thespectrogram 120 shows the frequency content of signals when that frequency content varies with time. Time is shown on the x-axis of thespectrogram 120; frequency is shown on a y-axis of thespectrogram 120; and the frequency magnitudes are typically indicated by color intensities (i.e., gray scale levels in the spectrogram 120). - The
spectrogram 120 illustrates that there are j frequency bins 122 (Bj, j=0, . . . , j−1 where j is the number of frequency bins). Means ofmagnitudes 124 over time, Mj FFT for j=0, . . . , k−1, can be calculated for the frequency bins, Bj, respectively. The mean of magnitudes Mj FFT can be, as the name implies, the mean of at least a subset (e.g., all) of the magnitudes of the frequency bin Bj over all the time windows (i.e., the time axis, the horizontal dimension). As such, each Mj FFT represents an average frequency magnitude response of the frequency bin Bk. To illustrate, with respect to spoken words for example, the means of magnitudes can represent the average performance in different (types of) words that are pronounced in the reference voice sample. Mj FFT can be calculated as Mj FFT=Σt=1 nmt,j where mt,j is the magnitude of spectrum where t and j are the time and frequency indexes, respectively, and where n is the last time index of the voice sample. - The means of magnitudes are converted (i.e., transformed, mapped, etc.) from the STFT domain to the ith Bark domain magnitude Mi Bark)) though mapping magnitudes of the FFT frequency bins (Mj FFT) to bark frequency bins using formula (1):
-
M i Bark=Σj∈Bi βij *M j FFT (1) - Formula (1) represents a conversion from the Fourier to the Bark domain. The Bark domain magnitudes, Mi Bark, for i=1, . . . , 24, constitute the reference frequency response curve, Rf.
- The Bark scale can range from 1 to 24, corresponding to the first 24 critical bands of hearing. The Bark band edges are given, in Hertz (Hz), as [0, 100, 200, 300, 400, 510, 630, 770, 920, 1080, 1270, 1480, 1720, 2000, 2320, 2700, 3150, 3700, 4400, 5300, 6400, 7700, 9500, 12000, 15500]; and the band centers in Hertz are [50, 150, 250, 350, 450, 570, 700, 840, 1000, 1170, 1370, 1600, 1850, 2150, 2500, 2900, 3400, 4000, 4800, 5800, 7000, 8500, 10500, 13500]. Thus, i ranges from 1 to 24. In another example, the Bark scale used can contain 109 bins. As such, i can range from 1 to 109 and the whole frequency can range from 0 to 24000 Hz.
- As mentioned above, Bis in formula (1) are the FFT frequency bins in ith Bark frequency band; and the coefficients βij are the Bark transform parameters. It is noted that the Bark domain transform can smooth the frequency response curve therewith eliminating any frequency outliers. The Bark transform is an auditory filter bank that can be thought of as calculating a moving average that smoothes the frequency response curve. The coefficients βij are triangle shaped parameters, as described with respect to
FIG. 2 . -
FIG. 2 illustrates theBark filter bank 200 according to implementations of this disclosure. So as not to not overly clutterFIG. 2 , it is noted that theBark filter bank 200 illustrates only 29 bins and a frequency range of 0 to 8000 Hz. The number of triangles inFIG. 2 should equal to number of bins. Thefilter bank 200 is used to illustrate how the coefficients βij of formula (1) are obtained. The coefficient βij are the Bark transform coefficients of the STFT. In the coefficient βij, the index i corresponds to a Bark frequency bin and the index j corresponds to an FFT frequency bin. The index j corresponds to the x-axis inFIG. 2 ; and the index i corresponds to a frequency. Each coefficient βij is determined in two dimensions: it is determined first by which triangle is to be used and second by which frequency bin the Mj FFT corresponds to. - Each of the Bark filters is a triangular band-pass filter, such as the
filter 202, with certain overlaps. The peeks of theBark filter bank 200, such as apeek 201, correspond to the center frequencies of the different Bark filters. It is to be noted that, inFIG. 2 , some triangles have are drawn to have thicker lines than other triangles. This is merely so as not to clutter the figure. No meaning should be ascribed to the fact that some triangles are drawn with thinner sides. - Using j=6 as an example, as can be seen in
FIG. 2 , j=6 is within two triangles (namely,triangles 204 and 206). Thetriangle 204 corresponds, approximately, to the frequency band 4200 Hz to 6300 Hz; andtriangle 206 corresponds, approximately, to the frequency band 5300 Hz to 7000 Hz. Projecting j=6 upward, the right side of thetriangle 204 is intersected at apoint 208, which corresponds to a β=0.2; and the left side of thetriangle 206 is intersected at apoint 210, which corresponds to a β=1.8. - To illustrate further, and without loss of generality, with respect to the
triangle 206, and referring to the formula (1), an index i=28 (Mi Bark) refers to the 28th triangle or the 28th Bark frequency band; the center frequency is the horizontal-axis value (i.e., the x value) of the top of triangle 206 (i.e. the peek 201), which is 6150 Hz; and Bi of formula (1) represent the frequency bins in the range from 5300 (i.e., a point 212) to 7000 Hz (i.e., a point 214), which are determined by the bottom oftriangle 206. Taking one of the Mi Bark for j∈Bi as an example, whose jth frequency bins is at 6000 Hz, then according to the y axis projection of thetriangle 206 at frequency 6000 Hz, the βi=28,j is 1.8. Thus, for the Mi=28 Bark, and assuming, for example, an FFT size of 1024 and a sampling rate of 16 kHz and that there are 109 pairs of Mj FFT and that βij is in range 5300 to 7000 Hz, then Mi=28 Bark can be calculated as Mi=28 Bark=Σj∈Bi βi=6100,j*Mj FFT. -
FIG. 3 is a diagram of an example of atechnique 300 of a real-time phase of timbre style transform according to implementations of this disclosure. Thetechnique 300 can be used in real-time applications, such as audio and/or video conferencing, telephone conversations, and the like, to transform the timbre of a source voice of at least one of the participants. Thetechnique 300 receives the source voice in frames, such as asource voice frame 302. In another example, thetechnique 300 itself can partition a received audio signal into the frames. A frame can correspond to an m number of milliseconds of audio. In an example, m can be 20 milliseconds. However, other values of m are possible. Thetechnique 300 outputs (e.g., generates, obtains, results in, calculates, etc.) a transformedvoice frame 306. Thesource voice frame 302 is in a source timbre style and the transformedvoice frame 306 that is in a reference timbre style. - The
technique 300 can be implemented by a computing device, such as thecomputing device 400 described with respect toFIG. 4 . - The
technique 300 can be implemented by a sending device. Thus, the timbre style of the speaker can be transformed to a reference timbre on the device of the sending user, before transmission to a receiving user, so that the receiving user can receive the voice of the sending user in the reference timbre. Thetechnique 300 can be implemented by a receiving device. Thus, the voice received at the receiving device of a receiving user can be transformed to a reference timbre that may be selected by the receiving user. Thetechnique 300 can be performed on the received speech to produce transformed speech with the reference timbre. The transformed speech is then output to the receiving user. Thetechnique 300 can be implemented by a central server, which receives a voice sample in a source timbre from a sending device, performs thetechnique 300 to obtain a voice in a reference (e.g., desired, etc.) timbre, and transmit (e.g., forward, relay, etc.) the transformed speech to one or more receiving devices. - The
source voice frame 302 can be processed via anequalizer 304 to produce the transformedvoice frame 306. Theequalizer 304 transforms the timbre using equalizer parameters, which are initially calculated and later updated, upon detection of large variations, as described below. - The
technique 300 obtains (e.g., calculates, looks up, determines, etc.) the gap between the reference frequency response curve (Rf) of the reference sample and the source frequency response curve (SR) of the source sample. Thetechnique 300 can obtain the gap (e.g., difference) in each of the frequency bins. That is, thetechnique 300 can obtain the gain(s) in amplification between the reference frequency response curve (Rf) of the reference sample and the source frequency response curve (SR) of the source sample. In an example, the gain can be obtained in the logarithmic scale. The gap can be obtained in decibels (dB). As is known, a decibel (dB) is a ratio between two quantities reported on a logarithmic scale and allows for a realistic modelling of human auditory perception. - For each kth frequency bin of the Bark domain, the
technique 300 can calculate the dB difference, Gb(k), between the source and the reference psychoacoustic frequency response curves using formula (2). -
G b(k)=20*log(Rf(k)/SR(k)) (2) - To reiterate, formula (2) can be used to measure the gain in amplification, in each of the Bark frequency bins, between reference frequency response curve (Rf) and the source frequency response curve (SR). The set of gains Gb(k) for all of the Bark domain frequency bins can constitute (e.g., can be considered to be, can be the basis for obtaining, etc.) the parameters of the
equalizer 304. Again, theequalizer 304 uses the equalizer parameters to transform the timbre of the source voice to the reference timbre style. - The
equalizer 304 is a set of filters. To illustrate, theequalizer 304 can have a filter for a lower frequency fn (e.g., 0 Hz) to upper frequency fn+1 (e.g., 800 Hz) band, which has a center frequency of (fn+fn+1)/2 (e.g., 400 Hz). Theequalizer 304 can use the equalizer parameters (i.e., the gains Gb(k)) that determine how much to add to or subtract from the center frequency to adjust the center frequency. - Interpolations parameters, which calculate the adjusted center frequency as an interpolation between the lower and upper frequencies of the frequency band, can then be determined. The interpolation parameters can also include (e.g., determine, define, etc.) a shape of the interpolation. In an example, the interpolation can be a cubic or cubic spline interpolation. Cubic spline interpolation can result in smoother interpolation than, for example, linear interpolation. The cubic spline interpolation method used to obtain an interpolation value of the ith gain, Gi e, can be described by the following equation (3). In equation (3), the interpolation parameters ai to di are determined by the Gb(i) near to the ith center frequency of equalizer.
-
G i e a i +b i x+c i x 2 +d i x 3 (3) - The
equalizer 304 can include (e.g., use) an initial set of equalizer parameters. In an example, the initial set of equalizer parameters may be obtained from previous executions of thetechnique 300. For example, astore 322 can include stored reference response curve(s), stored source frequency response curve(s), and/or corresponding equalizer parameters. As such, thestore 322 can include a referencefrequency response curve 322 of the reference the reference timbre style. Thestore 322 can be a permanent storage (e.g., a database, a file, etc.) or non-permanent memory. In another example, theequalizer 304 may not include equalizer parameters. As such, Initial equalizer parameters can be obtained as described below with respect to 314-318. - As different gains may be added or subtracted for the different Bark frequency bands by the
equalizer 304, the total energy of the source signal may be changed. In an example, thetechnique 300 can normalize the gains to keep the volumes of voice before and after equalizing by theequalizer 304 at the same (or roughly the same) levels. In an example, normalizing the gains can mean dividing each of the gains by the sum of all the gains. However, other normalizing techniques may be used. - The
technique 300 can perform the operations 308-318 to obtain initial equalizer parameters and when large variations (described below) are detected. - The
source voice frame 302 may be received into asignal buffer 308, which may store received voice frames until a period of source voice samples is available for further processing. In an example, the period of the source audio can be 30 seconds, 1 minute, 2 minutes, longer, or shorter period. - At 310, the
technique 300 converts the voice sample to the transform domain, such as described with respect to 104 ofFIG. 1 . As such, the voice sample (i.e., the period of the source audio) can be transformed to the STFT domain. At 312, thetechnique 300 converts means of magnitudes in the time dimension of the time-frequency domain signal to the Bark domain to obtain the source frequency response curve (SR). The source frequency response curve (SR) can be obtained as described with respect to the reference frequency response curve (Rf) and 106 ofFIG. 1 . Thus, the source frequency response curve (SR) can be a collection of Bark domain magnitudes, Mi source Bark, of the source sample. - At 314, the
technique 314 determines whether a large difference in source voice timbre occurs. The difference can be obtained at 316. To illustrate, and without loss of generality, during an RTC session, the source voice may be that of a first speaker (e.g., a 45-year old male). However, at some point during the RTC session, a second speaker (e.g., a 7-year old female) starts talking. As such, the source voice has changed significantly. Therefore, in an example, thetechnique 300 can replace the source frequency response curve (initially obtained for the first speaker) with that of the second speaker. In an example, thetechnique 300 can replace the source frequency response curve only when there is a large variation between a stored source frequency response curve and a current source frequency response curve. As also mentioned above, a large variation can be determined at 314 when the equalizer parameters have not yet been obtained (e.g., initialized). - At 314, if there is no large variation, then the
technique 300 proceeds to 304 where the previous equalizer parameters are used. However, if there is a large variation, at 314, then thetechnique 300 stores the current source frequency response curve in thestore 322 so that the current source frequency response curve can be compared to subsequent source frequency response curve to detect any subsequent large variations; thetechnique 300 also proceeds to 318 to update the equalizer parameters. That is, thetechnique 300 obtains the interpolations parameters, as described with respect to formula (3). - A relation threshold can be designed for large variation detection at 314. A relation coefficient can be calculated between the frequency response curve of the current period and stored, such as in the
store 322. If the relation coefficient is larger than a threshold, the stored frequency response curve will be replaced by the current one, and the parameters of an equalizer will be updated. Otherwise, the equalizer and stored frequency response curve will not be updated. - As the updating of the equalizer parameters can be completed (e.g., performed, finished, etc.) within one frame of the source voice signal (e.g., 10 ms), timbre style transform according to implementations of this disclosure is not interrupted due to the updating of the equalizer parameters. That is, delays or discontinuous are experienced when the equalizer parameters are updated.
-
FIG. 4 is a block diagram of an example of acomputing device 400 in accordance with implementations of this disclosure. Thecomputing device 400 can be in the form of a computing system including multiple computing devices, or in the form of one computing device, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, and the like. - A
processor 402 in thecomputing device 400 can be a conventional central processing unit. Alternatively, theprocessor 402 can be another type of device, or multiple devices, capable of manipulating or processing information now existing or hereafter developed. For example, although the disclosed implementations can be practiced with one processor as shown (e.g., the processor 402), advantages in speed and efficiency can be achieved by using more than one processor. - A
memory 404 incomputing device 400 can be a read only memory (ROM) device or a random access memory (RAM) device in an implementation. However, other suitable types of storage devices can be used as thememory 404. Thememory 404 can include code anddata 406 that are accessed by theprocessor 402 using abus 412. Thememory 404 can further include anoperating system 408 andapplication programs 410, theapplication programs 410 including at least one program that permits theprocessor 402 to perform at least some of the techniques described herein. For example, theapplication programs 410 can includeapplications 1 through N, which further include applications and techniques useful in real-time voice timbre style transform. For example theapplication programs 410 can include thetechnique 100 or aspects thereof, to implement a training phase. For example, theapplication programs 410 can include thetechnique 300 or aspects thereof to implement real-time voice timbre style transform. Thecomputing device 400 can also include asecondary storage 414, which can, for example, be a memory card used with a mobile computing device. - The
computing device 400 can also include one or more output devices, such as adisplay 418. Thedisplay 418 may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs. Thedisplay 418 can be coupled to theprocessor 402 via thebus 412. Other output devices that permit a user to program or otherwise use thecomputing device 400 can be provided in addition to or as an alternative to thedisplay 418. When the output device is or includes a display, the display can be implemented in various ways, including by a liquid crystal display (LCD), a cathode-ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display. - The
computing device 400 can also include or be in communication with an image-sensingdevice 420, for example, a camera, or any other image-sensingdevice 420 now existing or hereafter developed that can sense an image such as the image of a user operating thecomputing device 400. The image-sensingdevice 420 can be positioned such that it is directed toward the user operating thecomputing device 400. In an example, the position and optical axis of the image-sensingdevice 420 can be configured such that the field of vision includes an area that is directly adjacent to thedisplay 418 and from which thedisplay 418 is visible. - The
computing device 400 can also include or be in communication with a sound-sensing device 422, for example, a microphone, or any other sound-sensing device now existing or hereafter developed that can sense sounds near thecomputing device 400. The sound-sensing device 422 can be positioned such that it is directed toward the user operating thecomputing device 400 and can be configured to receive sounds, for example, speech or other utterances, made by the user while the user operates thecomputing device 400. Thecomputing device 400 can also include or be in communication with a sound-playingdevice 424, for example, a speaker, a headset, or any other sound-playing device now existing or hereafter developed that can play sounds as directed by thecomputing device 400. - Although
FIG. 4 depicts theprocessor 402 and thememory 404 of thecomputing device 400 as being integrated into one unit, other configurations can be utilized. The operations of theprocessor 402 can be distributed across multiple machines (wherein individual machines can have one or more processors) that can be coupled directly or across a local area or other network. Thememory 404 can be distributed across multiple machines such as a network-based memory or memory in multiple machines performing the operations of thecomputing device 400. Although depicted here as one bus, thebus 412 of thecomputing device 400 can be composed of multiple buses. Further, thesecondary storage 414 can be directly coupled to the other components of thecomputing device 400 or can be accessed via a network and can comprise an integrated unit such as a memory card or multiple units such as multiple memory cards. Thecomputing device 400 can thus be implemented in a wide variety of configurations. -
FIG. 5 is an example of a flowchart of atechnique 500 for transforming a voice of a speaker to a reference timbre according to an implementation of this disclosure. In an example, thetechnique 500 can receive an audio sample, such as an voice stream. The audio stream can be part of a video stream. In an example, thetechnique 500 can receive frames of the audio stream for processing. In an example, thetechnique 500 can partition the audio sample into frames and process each frame separately as further described below and consistent with the description of thetechnique 300 ofFIG. 3 . - The
technique 500 can be implemented by a computing device (e.g., an apparatus), such as thecomputing device 400 ofFIG. 4 . Thetechnique 500 can be implemented, for example, as a software program that may be executed by computing devices, such as thecomputing device 400 ofFIG. 4 . The software program can include machine-readable instructions that may be stored in a memory such as thememory 404 or thesecondary storage 414, and that, when executed by a processor, such asCPU 402, may cause the computing device to perform thetechnique 500. Thetechnique 500 can be implemented using specialized hardware or firmware. Multiple processors, memories, or both, may be used. In an example, the reference timbre can be received from the speaker. - At 502, the
technique 500 converts a portion of a source signal of the voice of the speaker into a time-frequency domain to obtain a time-frequency signal, as described above. At 504, thetechnique 500 obtains frequency bin means of magnitudes over time of the time-frequency signal, as described above with respect to Mk FFT. At 506, thetechnique 300 converts the frequency bin magnitude means into a bark domain to obtain a source frequency response curve (SR), as described above. An SR(i) corresponds to magnitude mean of the ith frequency bin. - At 508, the
technique 500 obtains respective gains of frequency bins of the bark domain with respect to a reference frequency response curve (Rf). The reference frequency response curve (Rf) can be obtained as described above. As such, thetechnique 300 can include, as described above, receiving a reference sample of the reference timbre; converting the reference sample into the time-frequency domain to obtain a reference time-frequency signal; obtaining reference frequency bin means of magnitudes (Mj FFT) over time of the reference time-frequency signal; and converting the reference frequency bin means of magnitudes (Mj FFT) into the bark domain to obtain the reference frequency response curve (Rf). The reference frequency response curve (Rf) includes respective bark domain frequency magnitudes (Mi Bark) for respective bark domain frequency bins, i. As such, an Rf(i) corresponds to a magnitude mean of the ith frequency bin. - As described above, the
technique 500 can convert the reference frequency bin means of magnitudes (Mj FFT) into the bark domain to obtain a reference frequency response curve (Rf) using formula (1). As described above, obtaining respective gains of frequency bins of the bark domain can include calculating a gain Gb(k) of a kth frequency bin in the bark domain using a ratio of the reference frequency bin magnitude mean of the kth frequency bin to the source frequency response curve (SR) of the kth frequency bin. The gain Gb(k) can be calculated using formula (2). - At 510, the
technique 500 can obtain equalizer parameters using the respective gains of the frequency bins of the Bark domain. In an example, obtaining the equalizer parameters using the respective gains of the frequency bins of the bark domain further can include mapping the respective gains to respective center frequencies of the equalizer to obtain values for gains of the equalizer. In an example, thetechnique 500 can normalize the respective gains to obtain the equalizer parameters. At 512, thetechnique 500 transforms the first portion to the reference timbre using the equalizer parameters. To illustrate, and without loss of generality, assume that an equalizer with 30 frequency bands is selected where the center frequencies of the frequency bands are fci, from fc1 to fc30, respectively; then the gain for each frequency band of the equalizer can be an interpolated gain Gi e, derived using formula (3). - As described above with respect to detecting large variations, the
technique 500 can further include obtaining a second source frequency response curve for a second portion of the source signal; in response to detecting a difference between the source frequency response curve and the second source frequency response curve exceeding a threshold, obtaining new equalizer parameters and using the new equalizer parameters as the equalizer parameters; and transforming the second portion of the source signal using the equalizer parameters, which may be the new equalizer parameters if a large variation is detected. - For simplicity of explanation, the
techniques FIGS. 1, 3, and 5 , respectively, are each depicted and described as a series of blocks, steps, or operations. However, the blocks, steps, or operations in accordance with this disclosure can occur in various orders and/or concurrently. Additionally, other steps or operations not presented and described herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a technique in accordance with the disclosed subject matter. - The word “example” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” is not necessarily to be construed as being preferred or advantageous over other aspects or designs. Rather, use of the word “example” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise or clearly indicated otherwise by the context, the statement “X includes A or B” is intended to mean any of the natural inclusive permutations thereof. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more,” unless specified otherwise or clearly indicated by the context to be directed to a singular form. Moreover, use of the term “an implementation” or the term “one implementation” throughout this disclosure is not intended to mean the same implementation unless described as such.
- Implementations of the
computing device 400, and/or any of the components therein described with respect toFIG. 4 and/or any of the components therein described with respect to modules or components ofFIG. 1 orFIG. 3 , (and any techniques, algorithms, methods, instructions, etc., stored thereon and/or executed thereby) can be realized in hardware, software, or any combination thereof. The hardware can include, for example, computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuit. In the claims, the term “processor” should be understood as encompassing any of the foregoing hardware, either singly or in combination. The terms “signal” and “data” are used interchangeably. - Further, in one aspect, for example, the techniques described herein can be implemented using a general purpose computer or general purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms, and/or instructions described herein. In addition, or alternatively, for example, a special purpose computer/processor can be utilized which can contain other hardware for carrying out any of the methods, algorithms, or instructions described herein.
- Further, all or a portion of implementations of this disclosure can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be any device that can, for example, tangibly contain, store, communicate, or transport the program for use by or in connection with any processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable mediums are also available.
- While the disclosure has been described in connection with certain embodiments, it is to be understood that the disclosure is not to be limited to the disclosed embodiments but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures as is permitted under the law.
Claims (20)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/071,454 US11380345B2 (en) | 2020-10-15 | 2020-10-15 | Real-time voice timbre style transform |
CN202110311790.5A CN114429763A (en) | 2020-10-15 | 2021-03-24 | Real-time voice tone style conversion technology |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/071,454 US11380345B2 (en) | 2020-10-15 | 2020-10-15 | Real-time voice timbre style transform |
Publications (2)
Publication Number | Publication Date |
---|---|
US20220122623A1 true US20220122623A1 (en) | 2022-04-21 |
US11380345B2 US11380345B2 (en) | 2022-07-05 |
Family
ID=81185161
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/071,454 Active US11380345B2 (en) | 2020-10-15 | 2020-10-15 | Real-time voice timbre style transform |
Country Status (2)
Country | Link |
---|---|
US (1) | US11380345B2 (en) |
CN (1) | CN114429763A (en) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
USRE37864E1 (en) * | 1990-07-13 | 2002-10-01 | Sony Corporation | Quantizing error reducer for audio signal |
US20070192100A1 (en) * | 2004-03-31 | 2007-08-16 | France Telecom | Method and system for the quick conversion of a voice signal |
US20080240282A1 (en) * | 2007-03-29 | 2008-10-02 | Motorola, Inc. | Method and apparatus for quickly detecting a presence of abrupt noise and updating a noise estimate |
US20090281811A1 (en) * | 2005-10-14 | 2009-11-12 | Panasonic Corporation | Transform coder and transform coding method |
US20210217431A1 (en) * | 2020-01-11 | 2021-07-15 | Soundhound, Inc. | Voice morphing apparatus having adjustable parameters |
-
2020
- 2020-10-15 US US17/071,454 patent/US11380345B2/en active Active
-
2021
- 2021-03-24 CN CN202110311790.5A patent/CN114429763A/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
USRE37864E1 (en) * | 1990-07-13 | 2002-10-01 | Sony Corporation | Quantizing error reducer for audio signal |
US20070192100A1 (en) * | 2004-03-31 | 2007-08-16 | France Telecom | Method and system for the quick conversion of a voice signal |
US20090281811A1 (en) * | 2005-10-14 | 2009-11-12 | Panasonic Corporation | Transform coder and transform coding method |
US20080240282A1 (en) * | 2007-03-29 | 2008-10-02 | Motorola, Inc. | Method and apparatus for quickly detecting a presence of abrupt noise and updating a noise estimate |
US20210217431A1 (en) * | 2020-01-11 | 2021-07-15 | Soundhound, Inc. | Voice morphing apparatus having adjustable parameters |
Non-Patent Citations (1)
Title |
---|
Turk, O. (2007). Cross-lingual voice conversion. Bogaziçi University, 3. * |
Also Published As
Publication number | Publication date |
---|---|
CN114429763A (en) | 2022-05-03 |
US11380345B2 (en) | 2022-07-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11749243B2 (en) | Network-based processing and distribution of multimedia content of a live musical performance | |
JP7150939B2 (en) | Volume leveler controller and control method | |
Verfaille et al. | Adaptive digital audio effects (A-DAFx): A new class of sound transformations | |
Ternström | Preferred self-to-other ratios in choir singing | |
US11727949B2 (en) | Methods and apparatus for reducing stuttering | |
US10262677B2 (en) | Systems and methods for removing reverberation from audio signals | |
Kumar | Comparative performance evaluation of MMSE-based speech enhancement techniques through simulation and real-time implementation | |
Tahmasebi et al. | Design and evaluation of a real-time audio source separation algorithm to remix music for cochlear implant users | |
Pelegrín-García et al. | Equal autophonic level curves under different room acoustics conditions | |
Monson et al. | Detection of high-frequency energy level changes in speech and singing | |
Payton et al. | Comparison of a short-time speech-based intelligibility metric to the speech transmission index and intelligibility data | |
JP6482880B2 (en) | Mixing apparatus, signal mixing method, and mixing program | |
Dong et al. | Long-term-average spectrum characteristics of Kunqu Opera singers’ speaking, singing and stage speech | |
US20230186782A1 (en) | Electronic device, method and computer program | |
Li et al. | Multi-metric optimization using generative adversarial networks for near-end speech intelligibility enhancement | |
US11380345B2 (en) | Real-time voice timbre style transform | |
Beerends et al. | Quantifying sound quality in loudspeaker reproduction | |
Wilson | Evaluation and modelling of perceived audio quality in popular music, towards intelligent music production | |
Pulakka et al. | Conversational quality evaluation of artificial bandwidth extension of telephone speech | |
Haeussler et al. | Crispness, speech intelligibility, and coloration of reverberant recordings played back in another reverberant room (Room-In-Room) | |
Czyżewski et al. | Adaptive personal tuning of sound in mobile computers | |
Hermes | Towards Measuring Music Mix Quality: the factors contributing to the spectral clarity of single sounds | |
CN112951265B (en) | Audio processing method and device, electronic equipment and storage medium | |
EP4247011A1 (en) | Apparatus and method for an automated control of a reverberation level using a perceptional model | |
Griesinger | The physics of auditory proximity and its effects on intelligibility and recall |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: AGORA LAB, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:FENG, JIANYUAN;HANG, RUIXIANG;ZHAO, LINSHENG;AND OTHERS;REEL/FRAME:054068/0284 Effective date: 20201015 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |