CN113079327A - Video generation method and device, storage medium and electronic equipment - Google Patents
Video generation method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN113079327A CN113079327A CN202110298211.8A CN202110298211A CN113079327A CN 113079327 A CN113079327 A CN 113079327A CN 202110298211 A CN202110298211 A CN 202110298211A CN 113079327 A CN113079327 A CN 113079327A
- Authority
- CN
- China
- Prior art keywords
- audio
- video
- sample
- processed
- organ
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 210000000056 organ Anatomy 0.000 claims abstract description 210
- 239000013598 vector Substances 0.000 claims abstract description 127
- 230000009471 action Effects 0.000 claims abstract description 88
- 238000012549 training Methods 0.000 claims abstract description 55
- 230000001755 vocal effect Effects 0.000 claims description 43
- 238000004590 computer program Methods 0.000 claims description 13
- 210000002105 tongue Anatomy 0.000 claims description 13
- 210000001847 jaw Anatomy 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 12
- 210000003437 trachea Anatomy 0.000 claims description 8
- 210000000214 mouth Anatomy 0.000 claims description 7
- 210000002409 epiglottis Anatomy 0.000 claims description 6
- 210000003238 esophagus Anatomy 0.000 claims description 6
- 210000003928 nasal cavity Anatomy 0.000 claims description 6
- 210000001260 vocal cord Anatomy 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 210000000867 larynx Anatomy 0.000 claims description 5
- 210000003800 pharynx Anatomy 0.000 claims description 5
- 210000002396 uvula Anatomy 0.000 claims description 5
- 230000008569 process Effects 0.000 abstract description 9
- 230000006870 function Effects 0.000 description 9
- 238000002595 magnetic resonance imaging Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 230000003287 optical effect Effects 0.000 description 7
- 241001465754 Metazoa Species 0.000 description 6
- 238000004891 communication Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 150000001875 compounds Chemical class 0.000 description 2
- 210000003128 head Anatomy 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 210000004072 lung Anatomy 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000003278 mimic effect Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 241001481833 Coryphaena hippurus Species 0.000 description 1
- 238000005481 NMR spectroscopy Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 210000004195 gingiva Anatomy 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000000513 principal component analysis Methods 0.000 description 1
- 210000005181 root of the tongue Anatomy 0.000 description 1
- 210000005182 tip of the tongue Anatomy 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/06—Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice
- G10L15/063—Training
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/03—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the type of extracted parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
- G10L2015/025—Phonemes, fenemes or fenones being the recognition units
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Electrically Operated Instructional Devices (AREA)
Abstract
The present disclosure relates to a video generation method and apparatus, a storage medium, and an electronic device, the method including: acquiring audio to be processed; converting the audio to be processed into audio feature vectors to be processed; inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ video feature sequence; generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video feature sequence; the video generation model is obtained by training in the following way: constructing model training data according to a sample audio feature vector converted from a sample audio and a sample pronunciation organ video feature sequence of a sample pronunciation organ action video corresponding to the sample audio; and training according to the model training data to obtain the video generation model. The method and the device not only improve the generation efficiency of the pronunciation organ action video, but also restore and intuitively display the real pronunciation action process of the pronunciation organ by utilizing the pronunciation organ action video.
Description
Technical Field
The present disclosure relates to the field of videos, and in particular, to a video generation method and apparatus, a storage medium, and an electronic device.
Background
In the scene of pronunciation learning, people are difficult to know the action and the force of the oral pronunciation organs of other people, and it is difficult to judge and simulate how to pronounce according to the voice, so the learning effect is poor and the efficiency is low.
At present, the human face condition during pronunciation can be simulated and displayed in a mode of making demonstration animation, but the mode cannot display the real action picture of the pronunciation organ in the oral cavity, the reference is not high, and time, labor and efficiency are not high when animation is drawn artificially.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
In a first aspect, the present disclosure provides a video generation method, including obtaining audio to be processed; converting the audio to be processed into audio feature vectors to be processed; inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ video feature sequence; generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video feature sequence; the video generation model is obtained by training in the following way: constructing model training data according to a sample audio feature vector converted from a sample audio and a sample pronunciation organ video feature sequence of a sample pronunciation organ action video corresponding to the sample audio; and training according to the model training data to obtain the video generation model.
In a second aspect, the present disclosure provides a video generation apparatus, the apparatus comprising: the acquisition module is used for acquiring audio to be processed; the conversion module is used for converting the audio to be processed into the audio feature vector to be processed; the input module is used for inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ video feature sequence; the generating module is used for generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video feature sequence; the training module is used for constructing model training data according to a sample audio characteristic vector converted from a sample audio and a sample pronunciation organ video characteristic sequence of a sample pronunciation organ action video corresponding to the sample audio; and training according to the model training data to obtain the video generation model.
In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program which, when executed by a processing apparatus, performs the steps of the method of the first aspect of the present disclosure.
In a fourth aspect, the present disclosure provides an electronic device comprising a storage device and a processing device, the storage device having a computer program stored thereon; processing means for executing the computer program in the storage means to implement the steps of the method of the first aspect of the present disclosure.
Through the technical scheme, the following technical effects can be at least achieved:
obtaining a pronunciation organ video characteristic sequence by obtaining the audio to be processed and inputting the audio characteristic vector to be processed converted from the audio to be processed into a video generation model, and generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video characteristic sequence. Therefore, the corresponding pronunciation organ action video can be generated quickly and efficiently on the basis of any audio, the generation efficiency of the pronunciation organ action video is improved, and the real pronunciation action process of the pronunciation organ is restored and visually displayed by utilizing the pronunciation organ action video.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale. In the drawings:
fig. 1 is a flow chart illustrating a video generation method according to an exemplary disclosed embodiment of the present disclosure.
Fig. 2 is a block diagram illustrating a video generation apparatus according to an exemplary disclosed embodiment of the present disclosure.
FIG. 3 is a block diagram illustrating an electronic device according to an exemplary disclosed embodiment of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
Fig. 1 is a flow chart illustrating a video generation method according to an exemplary disclosed embodiment of the present disclosure, as shown in fig. 1, the method comprising the steps of:
and S11, acquiring the audio to be processed.
The audio to be processed is audio containing arbitrary sound. The arbitrary sound may be a sound emitted through a sound-emitting organ. Alternatively, the arbitrary sound may be a sound emitted by a sound-organ simulating apparatus. Further alternatively, the arbitrary sound may be other pure sound, compound sound, noise, or the like that can be simulated by the vocal organs. It should be understood that pure tones, compound tones, noises that can be simulated by the vocal organs are not limited to human sounds, animal sounds, musical instrument sounds, and the like.
Illustratively, the audio to be processed is audio recorded when a user pronounces any text, wherein the text is any length of text such as a phoneme, a word, a sentence, a paragraph, and an article. As another example, the pending audio is audio recorded while the user is speaking. As another example, the pending audio is audio recorded when the user sings. Also illustratively, the audio to be processed may also mimic the sound of an animal (e.g., dolphin's voice), the sound of a musical instrument, and so forth for the user. The present disclosure is not particularly limited.
And S12, converting the audio to be processed into the audio feature vector to be processed.
A possible implementation manner is that the audio to be processed is input into a speech recognition model to obtain the audio feature vector to be processed, where the audio feature vector to be processed includes a phoneme posterior probability vector (PPG for short) of each frame of audio in the audio to be processed, and a dimension of each phoneme posterior probability vector is a phoneme dimension included in a language type corresponding to the audio to be processed.
Phonemes are the smallest units of speech that are divided according to the natural properties of the speech. Each of human voice, animal voice, musical instrument voice can be divided into a limited number of minimum voice units based on attributes.
Each frame of audio in the audio to be processed may be the audio of one phoneme. A phoneme may be characterized by a phoneme posterior probability vector. The dimensionality of each phoneme posterior probability vector is the phoneme dimensionality included by the language type corresponding to the audio to be processed. For example, assuming that the language type corresponding to the audio to be processed is english, since the number of phonemes in english is 48, the dimension of the posterior probability vector of english phonemes is 48. That is, an english phoneme posterior probability vector includes 48 probability values greater than or equal to 0 and less than 1, and the sum of the 48 probability values is 1. The phoneme corresponding to the maximum value of the 48 probability values is the english phoneme represented by the phoneme posterior probability vector. For another example, assuming that the language type corresponding to the audio to be processed is a language type simulating a target musical instrument, if there are 50 phonemes corresponding to the target musical instrument, the dimension of the phoneme posterior probability vector is also 50, and specifically consists of 50 probability values with a sum of 1.
Each frame of audio in the audio to be processed may also be an audio of a word/word. Accordingly, a word/phrase is characterized by a posterior probability vector of the word/phrase. Therefore, it is worth explaining that the audio frame playing time corresponding to each frame of audio in the audio to be processed can be freely set according to the user requirement, so that each frame of audio is the audio of one or more phonemes, characters or words.
A Speech Recognition model (ASR) is a model that converts voice into corresponding text or commands.
Because the number of words or phrases in any language is large and the number of phonemes is small, and the pronunciation of each word or phrase is composed of one or more phonemes, in a preferred embodiment, the speech recognition model can be trained by the following training method: constructing a model training sample according to a sample audio frame and a phoneme corresponding to the sample audio frame; and training according to the model training sample to obtain the voice recognition model.
In detail, signal processing and knowledge mining are carried out on the sample audio frame, the voice characteristic parameters of the sample audio frame are analyzed, and a voice template is manufactured to obtain a voice parameter library. And constructing a mapping table of the speech characteristic parameters and the phonemes according to the sample audio frame and the phonemes corresponding to the sample audio frame.
After the audio to be processed is input into the trained voice recognition model, the voice characteristic parameters to be processed are obtained through the same analysis as that in the training process aiming at each frame of audio in the audio to be processed, and the voice characteristic parameters to be processed are matched with voice templates in a voice parameter library one by one to obtain the matching probability of the voice characteristic parameters to be processed and each voice characteristic parameter in the voice parameter library. Further, a phoneme posterior probability vector of each frame of audio in the audio to be processed is obtained according to the mapping table of the speech characteristic parameters and the phonemes.
Compared with the mode of training the speech recognition model by using the audio of a large number of words and characters/words, the mode of training the speech recognition model by using the audio of a small number of limited phonemes and phonemes can reduce the model training tasks and quickly obtain the trained speech recognition model.
And S13, inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ video feature sequence.
The video generation model is obtained by training in the following way: constructing model training data according to a sample audio feature vector converted from a sample audio and a sample vocal organ video feature sequence of a sample vocal organ action video corresponding to the sample audio; and training according to the model training data to obtain the video generation model.
The loss function of the video generation model is not particularly limited by this disclosure.
Because the type of the word or word (or segment) in any language is huge and the number of phonemes is small, and the pronunciation of each word or word (or segment) is composed of one or more phonemes, in a preferred embodiment, the sample audio is the reading audio of all phonemes corresponding to the target language type. The sample vocal organ motion video may be a vocal organ motion animation demonstration video corresponding to each phoneme and made by using any animation rendering software. The sample vocal organ motion video may be a vocal organ motion video corresponding to each phoneme captured by an anatomical imaging apparatus such as a camera, a nuclear magnetic resonance apparatus, or a CT apparatus. Since the user can not only read characters or words in various human languages but also imitate sounds of animals, musical instruments, and the like. Therefore, in order to facilitate the understanding of the embodiments of the present disclosure by those skilled in the art, it is to be noted that the above-mentioned segments refer to sound segments (e.g., a sound segment corresponding to one key or one string of a musical instrument) in other sounds simulating non-human languages.
Similarly, in another embodiment, the sample audio is the audio of all the words or phrases (or segments) corresponding to the target language type. The sample pronunciation organ motion video may be a pronunciation organ motion animation demonstration video corresponding to each character or word (or sound segment) and made by adopting any animation rendering software. The sample vocal organ motion video may be a vocal organ motion video corresponding to each word or word (or segment) captured by an anatomical imaging apparatus such as a camera, a magnetic resonance apparatus, or a CT apparatus.
An implementation manner of constructing model training data of a video generation model according to the collected sample audio and the collected sample vocal organ action video corresponding to the sample audio may specifically include the following steps:
converting each frame of audio in the sample audio into a sample phoneme posterior probability vector to obtain a sample phoneme posterior probability vector sequence comprising at least one sample phoneme posterior probability vector, and taking the sample phoneme posterior probability vector sequence as a sample audio feature vector; extracting a sample pronunciation organ video characteristic corresponding to each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence based on the sample pronunciation organ action video to obtain a sample pronunciation organ video characteristic sequence; and taking the sample audio feature vector and the sample pronunciation organ video feature sequence as model training data of the video generation model.
Each frame of audio in the sample audio corresponds to each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence one by one, and each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence corresponds to each sample pronunciation organ video feature in the sample pronunciation organ video feature sequence one by one.
It is easily understood that, in the case that one frame of audio corresponds to one phoneme, the pronunciation process of the pronunciation organ corresponding to one phoneme is embodied by one or more frames of video images. Therefore, each sample pronunciation organ video feature is pixel point feature information of at least one frame of video image in the sample pronunciation organ action video; or each sample pronunciation organ video feature is principal component feature information of at least one frame of video image in the sample pronunciation organ action video.
It is worth to be noted that the principal component feature information is principal component coefficient data representing the video image obtained by performing dimensionality reduction on the video image through a principal component analysis algorithm.
An implementable embodiment, before said extracting, based on said sample pronunciation organ motion video, a sample pronunciation organ video feature corresponding to each of said sample phoneme posterior probability vectors in said sequence of sample phoneme posterior probability vectors, may further comprise the steps of:
and adjusting the position of the pronunciation organ in the sample pronunciation organ action video frame by frame so as to enable the same pronunciation organ in each frame of video image to be located at the same image position.
The adjustment may be performed in the form of pixel tracking or optical flow tracking, or may be performed in a manner of feature point extraction and alignment, and the processing of each frame of video image includes, but is not limited to, rotation, translation, enlargement, reduction, and uniform cropping of the size of each frame of video image. The positions of the pronunciation organs in the sample pronunciation organ action video are adjusted frame by frame, so that the same pronunciation organs in each frame of video image are positioned at the same image position, and the interference on the model training effect and the model convergence speed caused by different positions of the same pronunciation organs in each frame of video image is favorably reduced.
In step S13, since the audio feature vector to be processed includes the phoneme posterior probability vector of each frame of audio in the audio to be processed, after inputting the audio feature vector to be processed into the trained video generation model, the pronunciation organ video feature corresponding to the phoneme posterior probability vector of each frame of audio can be obtained. And obtaining the pronunciation organ video feature sequence corresponding to the audio feature vector to be processed.
And S14, generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video feature sequence.
It is easy to understand that when the vocal organ video feature sequence is pixel point feature information of the video image, the vocal organ video image can be generated according to the pixel point feature information, and then the vocal organ action video is obtained. When the pronunciation organ video feature sequence is the principal component feature information of the video image, the pronunciation organ video image can be generated according to the principal component feature information of the video image, and then the pronunciation organ action video is obtained.
By adopting the mode, the audio to be processed is obtained, the characteristic vector of the audio to be processed converted from the audio to be processed is input into the video generation model, the video characteristic sequence of the vocal organs is obtained, and the vocal organ action video corresponding to the audio to be processed is generated according to the video characteristic sequence of the vocal organs. Therefore, the corresponding pronunciation organ action video can be generated quickly and efficiently on the basis of any audio, the generation efficiency of the pronunciation organ action video is improved, and the real pronunciation action process of the pronunciation organ is restored and visually displayed by utilizing the pronunciation organ action video.
In addition, if the to-be-processed audio is converted into the to-be-processed audio feature vector by using the speech recognition model ASR in step S12, since the speech recognition model ASR focuses on recognizing the content in the speech and does not focus on the non-text content information of loudness, pitch, timbre, accent, pause, and the like of the sound in the speech, the to-be-processed audio feature vector (i.e., the vector including the phoneme posterior probability vector of each frame of audio in the to-be-processed audio) is converted into the to-be-processed audio feature vector by using the speech recognition model ASR, regardless of the non-text content information of loudness, pitch, timbre, accent, pause, and the like of the sound in the to-be-processed audio. However, if the video generation model in the present disclosure generates the corresponding pronunciation organ video feature sequence based on the to-be-processed audio feature vector that is not related to the non-text content information such as loudness, pitch, timbre, accent, pause, etc. of the sound in the to-be-processed audio, the pronunciation organ video feature sequence is not affected by the non-text content information such as loudness, pitch, timbre, accent, pause, etc. of the sound in the to-be-processed audio. And furthermore, the pronunciation organ action video generated by the pronunciation organ video characteristic sequence which is irrelevant to the non-text content information such as loudness, tone, accent, pause and the like of the sound in the audio to be processed is more accurate because the pronunciation organ action video is not interfered by the non-text content information such as loudness, tone, accent, pause and the like of the sound in the audio to be processed. Therefore, the above-described method can achieve the purpose of generating a standard vocal organ movement video by voice driving which is independent of a speaker (specifically, independent of non-text content information such as loudness, pitch, tone, accent, and pause of a speaker's voice).
The video generation method can be applied to mobile terminals and servers. Under the condition of saving the memory resource occupation of the mobile terminal of the user, the video generation method can be applied to the server. In the case that the video generating method is applied to the server, in step S11, the to-be-processed audio is obtained, specifically, the to-be-processed audio uploaded by the client is obtained by the server. Further, after the step S14, the video generation method may further include the steps of: and the server sends the pronunciation organ action video to the client so that the client displays the pronunciation organ action video.
By adopting the mode, the corresponding pronunciation organ action video can be generated aiming at the audio frequency of any text read by the user, the audio frequency of any song singing and the audio frequency simulating any animal/musical instrument sound. The pronunciation organ action video is displayed for the user, so that the user can visually see the pronunciation action process of the real pronunciation organ. Under the education-oriented scene, if the user can visually see the pronunciation action of the real pronunciation organ, the pronunciation learning of the user is facilitated. For example, it is advantageous for the user to learn how to speak various languages, how to sing various songs, how to mimic the sounds of various animals/musical instruments, and so on.
The vocal organ action video is a Magnetic Resonance Imaging (MRI) video, and comprises the actions of at least one vocal organ of an upper lip, a lower lip, an upper tooth, a lower tooth, a gum, a hard jaw, a soft jaw, a small tongue, a tongue tip, a tongue surface, a tongue root, a nasal cavity, an oral cavity, a pharynx, an epiglottis, an esophagus, a trachea, a vocal cord and a larynx. Correspondingly, the sample vocal organ motion video used for training the video generation model is also a magnetic resonance imaging MRI video, and the sample vocal organ motion video comprises the motion of at least one vocal organ of the upper lip, the lower lip, the upper teeth, the lower teeth, the gingiva, the hard jaw, the soft jaw, the uvula, the tongue tip, the tongue surface, the tongue root, the nasal cavity, the oral cavity, the pharyngeal head, the epiglottis, the esophagus, the trachea, the vocal cords and the laryngeal head.
In addition, since the sound-producing organ includes a lung, a diaphragm, a trachea, and other sound-producing motive organs, the sound-producing organ motion video and the sample sound-producing organ motion video may include a motion of at least one sound-producing organ of the lung, the diaphragm, and the trachea.
By adopting the method, the client sends the audio to be processed collected by the client to the server, the server generates the pronunciation organ action video according to the audio to be processed and sends the pronunciation organ action video to the client, and the client displays the pronunciation organ action video. Therefore, the user can intuitively see the pronunciation action of any real pronunciation organ through the pronunciation organ action video.
Fig. 2 is a block diagram illustrating a video generation apparatus according to an exemplary disclosed embodiment of the present disclosure. As shown in fig. 2, the apparatus 200 includes:
an obtaining module 210, configured to obtain an audio to be processed;
a conversion module 220, configured to convert the audio to be processed into an audio feature vector to be processed;
the input module 230 is configured to input the audio feature vector to be processed into a video generation model, so as to obtain a pronunciation organ video feature sequence;
a generating module 240, configured to generate a vocal organ motion video corresponding to the audio to be processed according to the vocal organ video feature sequence;
the apparatus 200 further comprises a training module 250 configured to construct model training data according to a sample audio feature vector converted from a sample audio and a sample vocal organ video feature sequence of a sample vocal organ action video corresponding to the sample audio; and training according to the model training data to obtain the video generation model.
In one possible implementation, the sample audio feature vector is obtained by: converting each frame of audio in the sample audio into a sample phoneme posterior probability vector to obtain a sample phoneme posterior probability vector sequence comprising at least one sample phoneme posterior probability vector, and taking the sample phoneme posterior probability vector sequence as the sample audio feature vector; the video characteristic sequence of the sample pronunciation organ is obtained by the following steps: and extracting the video characteristics of the sample pronunciation organ corresponding to each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence based on the sample pronunciation organ action video to obtain the sample pronunciation organ video characteristic sequence.
In a possible implementation manner, the training module 250 is configured to adjust the position of the pronunciation organ in the sample pronunciation organ motion video frame by frame before extracting the sample pronunciation organ video feature corresponding to each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence based on the sample pronunciation organ motion video, so that the same pronunciation organ in each frame of video image is located at the same image position.
In a possible implementation manner, each sample pronunciation organ video feature is pixel point feature information of at least one frame of video image in the sample pronunciation organ action video; or each sample pronunciation organ video feature is principal component feature information of at least one frame of video image in the sample pronunciation organ action video.
In a possible implementation manner, the converting module 220 is configured to input the audio to be processed into a speech recognition model, so as to obtain the audio feature vector to be processed, where the audio feature vector to be processed includes a phoneme posterior probability vector of each frame of audio in the audio to be processed, and a dimension of each phoneme posterior probability vector is a phoneme dimension included in a language type corresponding to the audio to be processed; the speech recognition model is obtained by training in the following training mode: constructing a model training sample according to a sample audio frame and a phoneme corresponding to the sample audio frame; and training according to the model training sample to obtain the voice recognition model.
In a possible implementation manner, the obtaining module 210 is configured to obtain to-be-processed audio uploaded by a client; the apparatus 200 further includes a sending module 260, configured to send the sound organ action video to the client, so that the client displays the sound organ action video.
In one possible embodiment, the vocal organ action video is a magnetic resonance imaging MRI video, and the vocal organ action video includes actions of at least one vocal organ selected from the group consisting of upper lip, lower lip, upper tooth, lower tooth, gum, hard jaw, soft jaw, uvula, tip of tongue, surface of tongue, root of tongue, nasal cavity, oral cavity, pharynx, epiglottis, esophagus, trachea, vocal cords, and larynx.
The steps specifically executed by the modules have been described in detail in some embodiments of the method, and are not described herein again.
Through the technical scheme, the following technical effects can be at least achieved:
obtaining a pronunciation organ video characteristic sequence by obtaining the audio to be processed and inputting the audio characteristic vector to be processed converted from the audio to be processed into a video generation model, and generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video characteristic sequence. Therefore, the corresponding pronunciation organ action video can be generated quickly and efficiently on the basis of any audio, the generation efficiency of the pronunciation organ action video is improved, and the real pronunciation action process of the pronunciation organ is restored and visually displayed by utilizing the pronunciation organ action video.
Referring now to FIG. 3, a block diagram of an electronic device 300 suitable for use in implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 3, the electronic device 300 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 301 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)302 or a program loaded from a storage means 308 into a Random Access Memory (RAM) 303. In the RAM 303, various programs and data necessary for the operation of the electronic apparatus 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.
Generally, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 307 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage devices 308 including, for example, magnetic tape, hard disk, etc.; and a communication device 309. The communication means 309 may allow the electronic device 300 to communicate wirelessly or by wire with other devices to exchange data. While fig. 3 illustrates an electronic device 300 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication means 309, or installed from the storage means 308, or installed from the ROM 302. The computer program, when executed by the processing device 301, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring audio to be processed; converting the audio to be processed into audio feature vectors to be processed; inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ video feature sequence; and generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video feature sequence.
Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present disclosure may be implemented by software or hardware. Wherein the name of a module in some cases does not constitute a limitation on the module itself.
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Example 1 provides a video generation method according to one or more embodiments of the present disclosure, including: acquiring audio to be processed; converting the audio to be processed into audio feature vectors to be processed; inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ video feature sequence; generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video feature sequence; the video generation model is obtained by training in the following way: constructing model training data according to a sample audio feature vector converted from a sample audio and a sample pronunciation organ video feature sequence of a sample pronunciation organ action video corresponding to the sample audio; and training according to the model training data to obtain the video generation model.
Example 2 provides the method of example 1, the sample audio feature vector being obtained by: converting each frame of audio in the sample audio into a sample phoneme posterior probability vector to obtain a sample phoneme posterior probability vector sequence comprising at least one sample phoneme posterior probability vector, and taking the sample phoneme posterior probability vector sequence as the sample audio feature vector; the video characteristic sequence of the sample pronunciation organ is obtained by the following steps: and extracting the video characteristics of the sample pronunciation organ corresponding to each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence based on the sample pronunciation organ action video to obtain the sample pronunciation organ video characteristic sequence.
Example 3 provides the method of example 2, before extracting, based on the sample vocal organ motion video, a sample vocal organ video feature corresponding to each of the sample phoneme posterior probability vectors in the sample phoneme posterior probability vector sequence, according to one or more embodiments of the present disclosure, including: and adjusting the position of the pronunciation organ in the sample pronunciation organ action video frame by frame so as to enable the same pronunciation organ in each frame of video image to be located at the same image position.
Example 4 provides the method of example 2 or 3, each of the sample vocal organ video features being pixel point feature information of at least one frame video image in the sample vocal organ motion video; or each sample pronunciation organ video feature is principal component feature information of at least one frame of video image in the sample pronunciation organ action video.
Example 5 provides the method of example 1, the converting the to-be-processed audio into to-be-processed audio feature vectors, comprising: inputting the audio to be processed into a speech recognition model to obtain the audio feature vector to be processed, wherein the audio feature vector to be processed comprises a phoneme posterior probability vector of each frame of audio in the audio to be processed, and the dimension of each phoneme posterior probability vector is the phoneme dimension included in the language type corresponding to the audio to be processed.
Example 6 provides the method of example 1, in accordance with one or more embodiments of the present disclosure, the obtaining the to-be-processed audio, including: acquiring audio to be processed uploaded by a client; the method further comprises the following steps: and sending the pronunciation organ action video to the client so as to enable the client to display the pronunciation organ action video.
Example 7 provides the method of examples 1-3, the sound-producing organ motion video being a magnetic resonance imaging MRI video, the sound-producing organ motion video including motion of at least one sound-producing organ of an upper lip, a lower lip, an upper tooth, a lower tooth, a gum, a hard jaw, a soft jaw, a uvula, a tongue tip, a lingual surface, a tongue root, a nasal cavity, an oral cavity, a pharynx, an epiglottis, an esophagus, a trachea, a vocal cord, a larynx.
Example 8 provides, in accordance with one or more embodiments of the present disclosure, a video generation apparatus, the apparatus comprising: the acquisition module is used for acquiring audio to be processed; the conversion module is used for converting the audio to be processed into the audio feature vector to be processed; the input module is used for inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ video feature sequence; the generating module is used for generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video feature sequence; the training module is used for constructing model training data according to a sample audio characteristic vector converted from a sample audio and a sample pronunciation organ video characteristic sequence of a sample pronunciation organ action video corresponding to the sample audio; and training according to the model training data to obtain the video generation model.
Example 9 provides the apparatus of example 8, the sample audio feature vector obtained by: converting each frame of audio in the sample audio into a sample phoneme posterior probability vector to obtain a sample phoneme posterior probability vector sequence comprising at least one sample phoneme posterior probability vector, and taking the sample phoneme posterior probability vector sequence as the sample audio feature vector; the video characteristic sequence of the sample pronunciation organ is obtained by the following steps: and extracting the video characteristics of the sample pronunciation organ corresponding to each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence based on the sample pronunciation organ action video to obtain the sample pronunciation organ video characteristic sequence.
Example 10 provides the apparatus of example 9, and a training module, configured to adjust a position of a sound organ in the sample sound organ motion video frame by frame before extracting a sample sound organ video feature corresponding to each of the sample phoneme posterior probability vectors in the sample phoneme posterior probability vector sequence based on the sample sound organ motion video, so that the same sound organ in each frame of video image is located at the same image position.
Example 11 provides the apparatus of example 9 or 10, each of the sample vocal organ video features being pixel point feature information of at least one video image in the sample vocal organ motion video; or each sample pronunciation organ video feature is principal component feature information of at least one frame of video image in the sample pronunciation organ action video.
Example 12 provides the apparatus of example 1, where the conversion module is configured to input the audio to be processed into a speech recognition model to obtain the audio feature vector to be processed, where the audio feature vector to be processed includes a phoneme posterior probability vector of each frame of audio in the audio to be processed, and a dimension of each phoneme posterior probability vector is a phoneme dimension included in a language type corresponding to the audio to be processed.
Example 13 provides the apparatus of example 8, in accordance with one or more embodiments of the present disclosure, the obtaining module is configured to obtain the to-be-processed audio uploaded by the client; the device further comprises a sending module, which is used for sending the pronunciation organ action video to the client so that the client can display the pronunciation organ action video.
Example 14 provides the apparatus of examples 8-10, the sound-organ motion video being a magnetic resonance imaging MRI video, the sound-organ motion video including motion of at least one sound-organ of an upper lip, a lower lip, an upper tooth, a lower tooth, a gum, a hard jaw, a soft jaw, a uvula, a tongue tip, a lingual surface, a tongue root, a nasal cavity, an oral cavity, a pharynx, an epiglottis, an esophagus, a trachea, a vocal cord, a larynx.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Claims (10)
1. A method of video generation, the method comprising:
acquiring audio to be processed;
converting the audio to be processed into audio feature vectors to be processed;
inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ video feature sequence;
generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video feature sequence;
the video generation model is obtained by training in the following way:
constructing model training data according to a sample audio feature vector converted from a sample audio and a sample pronunciation organ video feature sequence of a sample pronunciation organ action video corresponding to the sample audio;
and training according to the model training data to obtain the video generation model.
2. The method of claim 1, wherein the sample audio feature vector is obtained by:
converting each frame of audio in the sample audio into a sample phoneme posterior probability vector to obtain a sample phoneme posterior probability vector sequence comprising at least one sample phoneme posterior probability vector, and taking the sample phoneme posterior probability vector sequence as the sample audio feature vector;
the video characteristic sequence of the sample pronunciation organ is obtained by the following steps:
and extracting the video characteristics of the sample pronunciation organ corresponding to each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence based on the sample pronunciation organ action video to obtain the sample pronunciation organ video characteristic sequence.
3. The method according to claim 2, wherein before said extracting a sample vocal organ video feature corresponding to each of said sample phoneme posterior probability vectors in said sequence of sample phoneme posterior probability vectors based on said sample vocal organ motion video, comprising:
and adjusting the position of the pronunciation organ in the sample pronunciation organ action video frame by frame so as to enable the same pronunciation organ in each frame of video image to be located at the same image position.
4. The method according to claim 2 or 3, wherein each of the sample vocal organ video characteristics is pixel point characteristic information of at least one frame video image in the sample vocal organ action video; alternatively, the first and second electrodes may be,
each sample pronunciation organ video feature is principal component feature information of at least one frame of video image in the sample pronunciation organ action video.
5. The method of claim 1, wherein the converting the to-be-processed audio into to-be-processed audio feature vectors comprises:
inputting the audio to be processed into a speech recognition model to obtain the audio feature vector to be processed, wherein the audio feature vector to be processed comprises a phoneme posterior probability vector of each frame of audio in the audio to be processed, and the dimension of each phoneme posterior probability vector is the phoneme dimension included in the language type corresponding to the audio to be processed.
6. The method of claim 1, wherein the obtaining the audio to be processed comprises:
acquiring audio to be processed uploaded by a client;
the method further comprises the following steps:
and sending the pronunciation organ action video to the client so as to enable the client to display the pronunciation organ action video.
7. The method according to any one of claims 1-3, wherein the vocal organ action video is MRI video, and the vocal organ action video comprises the action of at least one vocal organ selected from the group consisting of upper lip, lower lip, upper teeth, lower teeth, gums, hard jaw, soft jaw, uvula, tongue tip, lingual surface, lingual root, nasal cavity, oral cavity, pharynx, epiglottis, esophagus, trachea, vocal cords, and larynx.
8. A video generation apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring audio to be processed;
the conversion module is used for converting the audio to be processed into the audio feature vector to be processed;
the input module is used for inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ video feature sequence;
the generating module is used for generating a pronunciation organ action video corresponding to the audio to be processed according to the pronunciation organ video feature sequence;
the training module is used for constructing model training data according to a sample audio characteristic vector converted from a sample audio and a sample pronunciation organ video characteristic sequence of a sample pronunciation organ action video corresponding to the sample audio; and training according to the model training data to obtain the video generation model.
9. A computer-readable medium, on which a computer program is stored, characterized in that the program, when being executed by processing means, carries out the steps of the method of any one of claims 1 to 7.
10. An electronic device, comprising:
a storage device having a computer program stored thereon;
processing means for executing the computer program in the storage means to carry out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110298211.8A CN113079327A (en) | 2021-03-19 | 2021-03-19 | Video generation method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110298211.8A CN113079327A (en) | 2021-03-19 | 2021-03-19 | Video generation method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113079327A true CN113079327A (en) | 2021-07-06 |
Family
ID=76613017
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110298211.8A Pending CN113079327A (en) | 2021-03-19 | 2021-03-19 | Video generation method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113079327A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022033327A1 (en) * | 2020-08-12 | 2022-02-17 | 北京字节跳动网络技术有限公司 | Video generation method and apparatus, generation model training method and apparatus, and medium and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013031677A1 (en) * | 2011-08-26 | 2013-03-07 | 国立大学法人豊橋技術科学大学 | Pronunciation movement visualization device and pronunciation learning device |
| CN110189394A (en) * | 2019-05-14 | 2019-08-30 | 北京字节跳动网络技术有限公司 | Shape of the mouth as one speaks generation method, device and electronic equipment |
| CN110880315A (en) * | 2019-10-17 | 2020-03-13 | 深圳市声希科技有限公司 | Personalized voice and video generation system based on phoneme posterior probability |
| CN110910479A (en) * | 2019-11-19 | 2020-03-24 | 中国传媒大学 | Video processing method and device, electronic equipment and readable storage medium |
| CN111370020A (en) * | 2020-02-04 | 2020-07-03 | 清华珠三角研究院 | Method, system, device and storage medium for converting voice into lip shape |
| CN111897976A (en) * | 2020-08-18 | 2020-11-06 | 北京字节跳动网络技术有限公司 | Virtual image synthesis method and device, electronic equipment and storage medium |
| CN111933110A (en) * | 2020-08-12 | 2020-11-13 | 北京字节跳动网络技术有限公司 | Video generation method, generation model training method, device, medium and equipment |
-
2021
- 2021-03-19 CN CN202110298211.8A patent/CN113079327A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013031677A1 (en) * | 2011-08-26 | 2013-03-07 | 国立大学法人豊橋技術科学大学 | Pronunciation movement visualization device and pronunciation learning device |
| CN110189394A (en) * | 2019-05-14 | 2019-08-30 | 北京字节跳动网络技术有限公司 | Shape of the mouth as one speaks generation method, device and electronic equipment |
| CN110880315A (en) * | 2019-10-17 | 2020-03-13 | 深圳市声希科技有限公司 | Personalized voice and video generation system based on phoneme posterior probability |
| CN110910479A (en) * | 2019-11-19 | 2020-03-24 | 中国传媒大学 | Video processing method and device, electronic equipment and readable storage medium |
| CN111370020A (en) * | 2020-02-04 | 2020-07-03 | 清华珠三角研究院 | Method, system, device and storage medium for converting voice into lip shape |
| CN111933110A (en) * | 2020-08-12 | 2020-11-13 | 北京字节跳动网络技术有限公司 | Video generation method, generation model training method, device, medium and equipment |
| CN111897976A (en) * | 2020-08-18 | 2020-11-06 | 北京字节跳动网络技术有限公司 | Virtual image synthesis method and device, electronic equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022033327A1 (en) * | 2020-08-12 | 2022-02-17 | 北京字节跳动网络技术有限公司 | Video generation method and apparatus, generation model training method and apparatus, and medium and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230223010A1 (en) | Video generation method, generation model training method and apparatus, and medium and device | |
| CN111899720B (en) | Method, apparatus, device and medium for generating audio | |
| CN111402842B (en) | Method, apparatus, device and medium for generating audio | |
| CN111899719A (en) | Method, apparatus, device and medium for generating audio | |
| WO2022048405A1 (en) | Text-based virtual object animation generation method, apparatus, storage medium, and terminal | |
| CN111798821B (en) | Sound conversion method, device, readable storage medium and electronic equipment | |
| CN113205793B (en) | Audio generation method and device, storage medium and electronic equipment | |
| CN111883107B (en) | Speech synthesis and feature extraction model training method, device, medium and equipment | |
| CN110880198A (en) | Animation generation method and device | |
| CN111161695B (en) | Song generation method and device | |
| CN113257218B (en) | Speech synthesis method, device, electronic equipment and storage medium | |
| WO2022194044A1 (en) | Pronunciation assessment method and apparatus, storage medium, and electronic device | |
| WO2022042418A1 (en) | Music synthesis method and apparatus, and device and computer-readable medium | |
| CN112185363A (en) | Audio processing method and device | |
| CN113421550A (en) | Speech synthesis method, device, readable medium and electronic equipment | |
| CN112383721B (en) | Method, apparatus, device and medium for generating video | |
| CN111369968A (en) | Sound reproduction method, device, readable medium and electronic equipment | |
| CN113223555A (en) | Video generation method and device, storage medium and electronic equipment | |
| CN113079327A (en) | Video generation method and device, storage medium and electronic equipment | |
| CN112381926A (en) | Method and apparatus for generating video | |
| CN111415662A (en) | Method, apparatus, device and medium for generating video | |
| CN113079328B (en) | Video generation method and device, storage medium and electronic equipment | |
| CN116189652A (en) | Speech synthesis method and device, readable medium and electronic equipment | |
| CN113205797B (en) | Virtual anchor generation method, device, computer equipment and readable storage medium | |
| CN113035235A (en) | Pronunciation evaluation method and apparatus, storage medium, and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210706 |
|
| RJ01 | Rejection of invention patent application after publication |