CN112420014A - Virtual face construction method and device, computer equipment and computer readable medium - Google Patents
Virtual face construction method and device, computer equipment and computer readable medium Download PDFInfo
- Publication number
- CN112420014A CN112420014A CN202011285914.9A CN202011285914A CN112420014A CN 112420014 A CN112420014 A CN 112420014A CN 202011285914 A CN202011285914 A CN 202011285914A CN 112420014 A CN112420014 A CN 112420014A
- Authority
- CN
- China
- Prior art keywords
- phoneme
- text
- preset
- training
- audio
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010276 construction Methods 0.000 title claims abstract description 47
- 238000003062 neural network model Methods 0.000 claims abstract description 47
- 238000000034 method Methods 0.000 claims abstract description 44
- 125000004122 cyclic group Chemical group 0.000 claims abstract description 31
- 230000011218 segmentation Effects 0.000 claims abstract description 29
- 238000012549 training Methods 0.000 claims description 160
- 238000004590 computer program Methods 0.000 claims description 19
- 230000000306 recurrent effect Effects 0.000 claims description 17
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 5
- 230000001815 facial effect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 239000003795 chemical substances by application Substances 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 238000013179 statistical model Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/06—Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids
- G10L21/10—Transforming into visible information
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
- G10L2015/025—Phonemes, fenemes or fenones being the recognition units
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Acoustics & Sound (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Evolutionary Biology (AREA)
- Quality & Reliability (AREA)
- Signal Processing (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Processing Or Creating Images (AREA)
Abstract
The application belongs to the technical field of artificial intelligence and provides a virtual human face construction method and device based on phoneme texts, computer equipment and a computer readable storage medium. According to the method and the device, the target text is obtained, the audio is generated through the target text in a preset TTS mode, the audio is subjected to phoneme segmentation to obtain a phoneme text corresponding to the audio, the phoneme text is input to a preset cyclic neural network model to obtain the face characteristics of a preset human anchor corresponding to the phoneme text, the virtual face of the preset human anchor corresponding to the phoneme text is constructed according to the face characteristics of the preset human anchor, the virtual face corresponding to the preset human anchor can be directly constructed according to the input target text, the generation program for converting the human anchor into the corresponding virtual anchor is simplified, and the generation efficiency and accuracy of the virtual anchor are improved.
Description
Technical Field
The present application relates to the field of artificial intelligence technologies, and in particular, to a virtual face construction method and apparatus based on phoneme text, a computer device, and a computer-readable storage medium.
Background
The virtual anchor is an anchor or customer service which uses an avatar to interact with a customer in a video based on technologies such as voice, NLP, vision and the like. The virtual anchor can solve the problems that the traditional customer service agent is high in cost (for example, the agent needs wage and welfare, the management and training cost is high, the period is long, and the like), the working quality is unstable (easily influenced by emotion, fatigue and the like), and the like, reduces the customer service cost of a company, improves the customer satisfaction degree, and ensures the stability of the working quality.
At present, most of the conventional virtual anchor technologies are based on audio or audio combined with images to construct virtual video images, corresponding audio or audio and images need to be recorded as input, and different virtual anchors need to learn different audio characteristics in a customization process, or collect a large amount of data of video/audio recorded by anchor anchors of different real persons to perform generalized learning, so that the technical problem of low efficiency in constructing virtual anchors exists in the conventional technologies.
Disclosure of Invention
The application provides a virtual face construction method and device based on phoneme texts, computer equipment and a computer readable storage medium, and can solve the problem of low efficiency of constructing a virtual anchor in the prior art.
In a first aspect, the present application provides a virtual face construction method based on phoneme texts, where the method includes: acquiring a target text, and generating audio from the target text in a preset TTS mode; performing phoneme segmentation on the audio to obtain a phoneme text corresponding to the audio; inputting the phoneme text into a preset cyclic neural network model to obtain the human face characteristics of a preset human anchor corresponding to the phoneme text; and constructing a virtual face of the preset human anchor corresponding to the phoneme text according to the face characteristics of the preset human anchor.
In a second aspect, the present application further provides a virtual face constructing apparatus based on a phoneme text, including: the first acquisition unit is used for acquiring a target text and generating an audio from the target text in a preset TTS mode; the first segmentation unit is used for carrying out phoneme segmentation on the audio to obtain a phoneme text corresponding to the audio; the first input unit is used for inputting the phoneme text into a preset recurrent neural network model so as to obtain the human face characteristics of a preset human anchor corresponding to the phoneme text; and the construction unit is used for constructing the virtual face of the preset human anchor corresponding to the phoneme text according to the human face characteristics of the preset human anchor.
In a third aspect, the present application further provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the virtual face construction method based on phoneme text when executing the computer program.
In a fourth aspect, the present application further provides a computer-readable storage medium storing a computer program, which when executed by a processor, causes the processor to execute the steps of the phoneme text based virtual face construction method.
The application provides a virtual face construction method and device based on phoneme texts, computer equipment and a computer readable storage medium. This application is through the target text who obtains one section input, and will the target text is through predetermineeing TTS mode generation audio frequency, will the audio frequency carries out the phoneme segmentation, in order to obtain the phoneme text that the audio frequency corresponds is input to predetermineeing the recurrent neural network model, in order to obtain the facial feature of the preset real man anchor that the phoneme text corresponds, according to the facial feature of the preset real man anchor, the structure the phoneme text corresponds predetermine the virtual human face of real man anchor, can directly construct the virtual human face that the preset real man anchor corresponds according to the target text of input, simplified and converted the real man anchor into the generation procedure of the virtual anchor that corresponds, improved the generation efficiency and the accuracy of virtual anchor.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a virtual face construction method based on phoneme texts according to an embodiment of the present application;
fig. 2 is a schematic view of a first sub-flow of a virtual face construction method based on phoneme texts according to an embodiment of the present application;
fig. 3 is a second sub-flow diagram of a virtual face construction method based on phoneme texts according to an embodiment of the present application;
fig. 4 is a third sub-flow diagram of a virtual face construction method based on phoneme texts according to an embodiment of the present application;
fig. 5 is a fourth sub-flow diagram of a virtual face construction method based on phoneme texts according to an embodiment of the present application;
fig. 6 is a fifth sub-flow diagram of a virtual face construction method based on phoneme texts according to an embodiment of the present application;
fig. 7 is a schematic block diagram of a virtual face construction apparatus based on phoneme texts according to an embodiment of the present application; and
fig. 8 is a schematic block diagram of a computer device provided in an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Referring to fig. 1, fig. 1 is a schematic flowchart of a virtual face construction method based on phoneme texts according to an embodiment of the present application. As shown in FIG. 1, the method includes the following steps S11-S14:
and S11, acquiring a target text, and generating audio from the target text in a preset TTS mode.
Wherein, TTS, english is Text To Speech, and "from Text To Speech" is a process of intelligently converting characters into Speech.
Specifically, in the embodiment of the application, when the virtual face is constructed based on the phoneme text, only the target text needs to be input, and the virtual face can be constructed according to the input target text. After a target text is acquired, generating an audio corresponding to the target text from the target text in a preset TTS mode. The TTS includes a "splicing method" and a "parametric method", where the splicing method prepares a large amount of speech, and the speech is spliced by basic units (such as syllables, phonemes, etc.), and then extracts a synthesis target sound from the prepared speech. The parametric method is to generate speech parameters (including fundamental frequency, formant frequency, etc.) at every moment according to a statistical model and then convert the parameters into waveforms.
S12, carrying out phoneme segmentation on the audio to obtain a phoneme text corresponding to the audio.
Wherein, a phoneme (Phone in english) is the minimum voice unit divided according to the natural attribute of the voice, and is analyzed according to the pronunciation action in the syllable, and one action constitutes one phoneme.
Specifically, after audio is generated according to the obtained target text, the audio is segmented and aligned to obtain factor texts corresponding to the text, virtual faces are constructed according to the factor texts, and when the virtual faces and the audio are used as virtual videos to be played, the factor texts correspond to each frame in the virtual videos.
And S13, inputting the phoneme text into a preset recurrent neural network model to obtain the human face characteristics of the preset human anchor corresponding to the phoneme text.
S14, constructing a virtual face of the preset human anchor corresponding to the phoneme text according to the face features of the preset human anchor.
Wherein, the Recurrent Neural Network is called current Neural Network in English, and abbreviated as RNN.
Specifically, a preset recurrent neural network model is trained first. The training preset cyclic neural network model is characterized in that a training audio sample contained in a training video sample is segmented by recording a training video sample of a human-centered character, so as to obtain a training phoneme text, the training phoneme text and an image of the human-centered character corresponding to the training phoneme text contained in the training video sample are input into the preset cyclic neural network model, so that the cyclic neural network model automatically learns the incidence relation between the training phoneme text and the human face characteristics of the human-centered character image corresponding to the training phoneme text, and the training phoneme text is matched with the human face characteristics of the human-centered character contained in the image of the human-centered character.
After the phoneme text corresponding to the acquired target text is input into a preset cyclic neural network model, the preset cyclic neural network model combines the incidence relation between the training phoneme text and the human face characteristics of the human anchor contained in the image of the human anchor during training according to the phoneme text to obtain the human face characteristics of the preset human anchor contained in each video frame corresponding to the phoneme text, and combines the relationship between the human face characteristics and the human face to construct a virtual human face corresponding to the human face characteristics of the preset human anchor so as to obtain the virtual human face of the preset human anchor corresponding to the phoneme text, render the virtual human face, and then correspond the virtual human face with the audio so as to obtain the virtual anchor video.
In this application embodiment, through obtaining the target text, and will the target text generates the audio frequency through predetermineeing the TTS mode, will the audio frequency carries out the phoneme segmentation, in order to obtain the phoneme text that the audio frequency corresponds, will the phoneme text is input to predetermineeing the recurrent neural network model, in order to obtain the facial feature of the preset real man anchor that the phoneme text corresponds, according to the facial feature of the preset real man anchor, the structure the phoneme text corresponds the virtual human face of preset real man anchor can directly be found the virtual human face that the preset real man anchor corresponds according to the target text of input, has simplified and has been converted into the generation procedure of the virtual anchor that corresponds with real man anchor, has improved the generation efficiency and the accuracy of virtual anchor.
Referring to fig. 2, fig. 2 is a schematic sub-flow chart of a virtual face construction method based on phoneme texts according to an embodiment of the present application. As shown in fig. 2, in this embodiment, before the step of inputting the phoneme text into a preset recurrent neural network model to obtain a face feature of a preset human anchor corresponding to the phoneme text, the method further includes:
s21, acquiring a training video sample recorded by a preset live anchor, wherein the training video sample comprises a training audio sample and the image of the preset live anchor;
s22, performing phoneme segmentation on the training audio sample to obtain a training phoneme text, and extracting the face features of the training human anchor corresponding to the training phoneme text from the image of the preset human anchor;
and S23, inputting the training phoneme text and the human face characteristics of the training human anchor into a preset cyclic neural network model so as to train the preset cyclic neural network model.
Specifically, before the step of inputting the phoneme text into a preset cyclic neural network model to obtain the face features of the preset human anchor corresponding to the phoneme text, training the preset cyclic neural network model by using a training video sample corresponding to the preset human anchor. When the training video sample corresponding to the preset real man anchor is adopted to train the preset cyclic neural network model, recording the training video sample corresponding to the preset real man anchor in a preset time period, requiring the front face of the preset real man anchor to be recorded and exposing the whole face, supplementing light and adopting a microphone to record when recording, for example, recording the training video sample with the time duration of 2-3 hours to meet the training requirement of the preset cyclic neural network model in the embodiment of the application, wherein the training video sample comprises a training audio sample and the image of the preset real man anchor, performing phoneme segmentation on the training audio sample to obtain a training phoneme text, and extracting the face characteristics of the training real man anchor corresponding to the training phoneme text from the training video sample, for example, 3DMM features of a human face (i.e., Morph of 3dmax, 3D deformable model) may be extracted, where each phoneme corresponds to each video frame of the video.
Inputting the training phoneme text and the human face features of the training human anchor into a preset recurrent neural network model to train the preset recurrent neural network model, for example, Embedding the obtained phoneme text to obtain an input vector of the preset recurrent neural network model, and using the pre-extracted 150-dimensional features of the 3DMM as the Label of the preset recurrent neural network model. The training of the preset recurrent neural network model comprises the following two processes:
1) during forward propagation: the Embedding feature vector passes through two layers of bidirectional RNNs to obtain a 150-dimensional feature Predict. 2) When reversely propagating: and calculating the Loss such as MSE/MAE/L1/L2 and the like according to the Predict and the Label, and updating in a reverse mode to perform gradient updating. When a virtual face is constructed by using a preset cyclic neural network model subsequently, only a phoneme text (with an assumed length of N) needs to be input, and N x 150 dimensional 3DMM characteristics of Ppreredict can be obtained according to the forward propagation process, and are respectively corresponding to each frame of N frames of a video, and are used for face reconstruction of each frame in a subsequent virtual anchor, so that a complete and continuous virtual anchor video is obtained.
And repeating the steps 1) and 2) until convergence, so that the preset cyclic neural network model adjusts the parameters of the preset cyclic neural network model through automatic learning, and the preset cyclic neural network model is trained by adopting training video samples corresponding to preset human anchor, wherein different preset RNN models, different Loss, different optimization modes and initialization modes of model parameters can be tried to adapt to different types of data. Because the training video sample within shorter time can be adopted in the embodiment of the application, for example, the training requirement of the preset cyclic neural network model can be met without recording the training audio and video sample data for 2 to 3 hours, the cyclic neural network model with better effect can be trained, and the accurate virtual face can be constructed according to the input target text, compared with the traditional technology which needs a large number of training video samples or training video samples with longer time, the training process of the preset cyclic neural network model is greatly simplified in the embodiment of the application, the problem that a large number of audios and videos need to be played at the same time when the preset cyclic neural network model is trained is avoided, the data usage amount in the training process of the cyclic neural network model is greatly reduced, and the training efficiency and the training accuracy of the preset cyclic neural network model are improved, the training video sample data size required by customizing a virtual anchor is reduced to a great extent, and the time cost for generating the recurrent neural network model is greatly reduced.
Referring to fig. 3, fig. 3 is a schematic view of a second sub-flow of a virtual face construction method based on phoneme texts according to an embodiment of the present application. As shown in fig. 3, in this embodiment, the step of performing phoneme segmentation on the training audio sample to obtain a training phoneme text, and extracting a face feature of the training human anchor corresponding to the training phoneme text from the image of the preset human anchor includes:
s31, obtaining a training audio sample corresponding to the training video sample;
s32, performing phoneme segmentation on the training audio sample according to a preset phoneme according to a time sequence to obtain a training phoneme text sequence arranged according to the time sequence, wherein the training phoneme text sequence comprises a plurality of training phoneme text elements;
s33, acquiring video frames corresponding to the training phoneme text elements from the training video samples according to the time sequence, wherein the video frames comprise the images of the preset human anchor;
and S34, extracting the appearance feature, the expression feature and the posture feature of the preset human anchor contained in the video frame corresponding to the training phoneme text element to obtain the face feature of the training human anchor corresponding to the training phoneme text.
Specifically, the audio contained in the training video sample is a set of phonemes, the video contained in the training video sample is a set of video frames, and the phoneme elements in the phoneme set and the video frame elements in the video frame set have a correspondence in chronological order. Therefore, after the training video sample is obtained, a training audio sample corresponding to the training video sample is obtained, the training audio sample is a set of training phoneme text elements according to a time sequence, and the training audio sample is subjected to phoneme segmentation according to a preset phoneme according to the time sequence so as to obtain a training phoneme text sequence which is arranged according to the time sequence and contains a plurality of training phoneme text elements. For example, in one example, the phoneme segmentation results are as follows:
Intervals[1]:xmin＝0.000 xmax＝0.700 text＝”sil”；
Intervals[2]:xmin＝0.700 xmax＝0.780 text＝”HH”；
Intervals[3]:xmin＝0.780 xmax＝0.920 text＝”IY1”；
Intervals[4]:xmin＝0.920 xmax＝1.070 text＝”HH”；
where intervals [ i ] represents the ith phoneme, xmin, xmax represents the start time and end time, respectively, and so on.
And generating a factor text corresponding to the video frame according to the segmentation result, wherein in an example, the generated text is as follows:
[”sil”,”sil”,”sil”,”HH”,”HH””HH”,”IY1”,”IY1”,”IY1”,”HH”,”HH”,”HH”,”HH”]。
where each factor corresponds to each frame of the video.
According to the time sequence, acquiring video frames corresponding to the training phoneme text elements from the training video sample, wherein the video frames comprise images of the preset human anchor, extracting human face features of the training human anchor contained in the video frames corresponding to the training phoneme text elements according to the phoneme text elements, wherein the human face features of the training human anchor can comprise appearance features, expression features and posture features of a human face, and thus, the human face features of the training human anchor corresponding to the training phoneme text arranged according to the time sequence can be obtained, and the training phoneme text sequence contained in the training audio sample and the human face features of the training human anchor of each video frame corresponding to the training phoneme text sequence are obtained.
When extracting the face features from the training video sample, 3D DMM face features (3D DMM, 3D Morphable Models, i.e., face 3D deformation statistical Models) can be extracted from the training video sample. 3DMM features are extracted from each video frame of a training video sample, and [0-80] dimension Shape features (namely, appearance features), [80-144] dimension expression features, and [224-, [ 227-, [254-, [ 257] Pose features (namely, Pose features) can be selected to obtain the face features for training the real human anchor, so that rich real human anchor face features can be obtained from the aspects of appearance features, expression features, Pose features and the like, and the richness, the vividness and the reality of the constructed virtual face can be improved. The shape feature is to describe the shape of the face by extracting key points of the shape of the face, the expression feature extraction is to position and extract organ features, texture regions and predefined feature points of the face, and the posture feature is a posture feature of the face, such as a posture feature of the face looking up, looking down or looking sideways.
Referring to fig. 4, fig. 4 is a third sub-flow diagram of a virtual face construction method based on phoneme texts according to an embodiment of the present application. As shown in fig. 4, in this embodiment, after the step of constructing the virtual face of the preset human anchor corresponding to the phoneme text according to the human face features of the preset human anchor, the method further includes:
s41, acquiring all the phoneme texts contained in the audio and virtual faces corresponding to the phoneme texts;
s42, combining all the virtual human faces into a video frame sequence according to the time sequence of the phoneme texts in the audio;
s43, combining the audio and the video frame sequence to obtain the virtual video.
Specifically, after the virtual faces corresponding to the phoneme texts are constructed, all the phoneme texts contained in the audio are sequenced according to the time sequence of the phoneme texts in the audio, so that a phoneme text sequencing sequence consistent with the audio can be obtained, the virtual faces corresponding to the phoneme texts are sequenced according to the sequence of the phoneme text sequencing sequence, all the virtual faces are combined into a video frame sequence, and then the audio and the video frame sequence are combined to obtain a virtual video, so that when the virtual main broadcast video is constructed, the corresponding video can be generated by directly using the phoneme texts without recording audio data, the use cost of customers is reduced, and the use efficiency is improved.
Referring to fig. 5, fig. 5 is a fourth sub-flow diagram of a virtual face construction method based on phoneme texts according to an embodiment of the present application. As shown in fig. 5, in this embodiment, after the step of constructing the virtual face of the preset human anchor corresponding to the phoneme text according to the human face features of the preset human anchor, the method further includes:
s51, generating other audios by the target text in other preset TTS modes, wherein the other audios and the audios are audios of different audio style types;
s52, combining the other audio and the video frame sequence to obtain other virtual video.
Specifically, different TTS (Text-To-Speech) tools can be used To generate different styles and types of audio for the same target Text, because the audio of different audio style types generated by the same target text has the same factor text and the same time sequence corresponding to the factor text, the audio of different style types is respectively combined with the same video frame sequence, the video with different audio styles and types and different video styles and types can be obtained, the possibility of changing different sounds for the same video can be realized according to different scene requirements, different sound supports can be provided for the video of the same anchor, the video does not need to be recorded again, data processing, training and model adjustment are not needed to regenerate the model, the efficiency of generating the videos with various styles and types is improved, and the applicability of the generated videos is improved.
Referring to fig. 6, fig. 6 is a fifth sub-flow diagram of a virtual face construction method based on phoneme texts according to an embodiment of the present application. As shown in fig. 6, in this embodiment, before the step of generating audio by using a preset TTS manner, the method further includes:
s61, counting the text length corresponding to the target text;
s62, judging whether the text length is larger than or equal to a preset text length threshold value;
s63, if the text length is larger than or equal to the preset text length threshold value, executing the step of generating audio by the target text in a preset TTS mode;
and S64, if the text length is smaller than the preset text length threshold value, sending an alarm that the target text does not meet the preset requirement.
Specifically, for the target text which is in an extreme case and does not meet the condition of being converted into audio, for example, the target text is an exclamation word or a phrase, in this case, the target text cannot be converted into voice, and then the virtual anchor video is constructed according to the target text. Therefore, before the target text is used to generate audio in a preset TTS manner, the text length corresponding to the target text may be counted, for example, the number of characters corresponding to characters included in the target text is counted, the text length corresponding to the target text is obtained, and then whether the text meets the requirement of converting into speech is determined according to the text length, it is determined whether the text length is greater than or equal to a preset text length threshold, for example, the preset text length threshold may be 2 characters or 5 characters, if the target text is greater than or equal to the preset text length threshold, the step of generating audio from the target text in a preset TTS manner is executed, if the target text is less than the preset text length threshold, that is, the target text is too short, an alarm that the target text does not meet the preset requirement is issued, therefore, the virtual anchor can not be constructed subsequently according to the target text which does not meet the requirement.
It should be noted that, in the virtual face construction method based on the phoneme text described in the above embodiments, the technical features included in different embodiments may be recombined as needed to obtain a combined implementation, but all of the embodiments are within the scope of the present application.
Referring to fig. 7, fig. 7 is a schematic block diagram of a virtual face construction apparatus based on phoneme texts according to an embodiment of the present application. Corresponding to the virtual face construction method based on the phoneme text, the embodiment of the application also provides a virtual face construction device based on the phoneme text. As shown in fig. 7, the phoneme text based virtual face construction apparatus includes a unit for performing the phoneme text based virtual face construction method, and the phoneme text based virtual face construction apparatus may be configured in a computer device. Specifically, referring to fig. 7, the virtual face construction apparatus 70 based on phoneme text includes a first obtaining unit 71, a first dividing unit 72, a first input unit 73, and a construction unit 74.
The first obtaining unit 71 is configured to obtain a target text, and generate an audio from the target text in a preset TTS manner;
a first segmentation unit 72, configured to perform phoneme segmentation on the audio to obtain a phoneme text corresponding to the audio;
a first input unit 73, configured to input the phoneme text into a preset recurrent neural network model, so as to obtain a human face feature of a preset human anchor corresponding to the phoneme text;
and the constructing unit 74 is configured to construct a virtual face of the preset human anchor corresponding to the phoneme text according to the face features of the preset human anchor.
In an embodiment, the virtual face construction apparatus 70 based on phoneme text further includes:
the second acquisition unit is used for acquiring a training video sample recorded by a preset live anchor, wherein the training video sample comprises a training audio sample and the image of the preset live anchor;
the second segmentation unit is used for performing phoneme segmentation on the training audio sample to obtain a training phoneme text, and extracting the face features of the training human anchor corresponding to the training phoneme text from the image of the preset human anchor;
and the training unit is used for inputting the training phoneme text and the human face characteristics of the training human anchor into a preset cyclic neural network model so as to train the preset cyclic neural network model.
In one embodiment, the second slicing unit includes:
the first obtaining subunit is used for obtaining a training audio sample corresponding to the training video sample;
the first molecular unit is used for carrying out phoneme segmentation on the training audio sample according to a preset phoneme according to a time sequence so as to obtain a training phoneme text sequence arranged according to the time sequence, and the training phoneme text sequence comprises a plurality of training phoneme text elements;
a second obtaining subunit, configured to obtain, according to the time sequence, a video frame corresponding to the training phoneme text element from the training video sample, where the video frame includes an image of the preset human anchor;
and the extracting subunit is used for extracting the appearance characteristic, the expression characteristic and the posture characteristic of the preset human anchor contained in the video frame corresponding to the training phoneme text element so as to obtain the face characteristic of the training human anchor corresponding to the training phoneme text.
In an embodiment, the virtual face construction apparatus 70 based on phoneme text further includes:
a third obtaining unit, configured to obtain all the phoneme texts included in the audio and a virtual face corresponding to the phoneme texts;
the first combination unit is used for combining all the virtual human faces into a video frame sequence according to the time sequence of the phoneme texts in the audio;
a second combining unit for combining the audio with the sequence of video frames to obtain a virtual video.
In an embodiment, the virtual face construction apparatus 70 based on phoneme text further includes:
the generating unit is used for generating other audios from the target text in other preset TTS modes, wherein the other audios and the audios are audios of different audio style types;
a third combining unit for combining the further audio with the sequence of video frames to obtain a further virtual video.
In an embodiment, the virtual face construction apparatus 70 based on phoneme text further includes:
the statistical unit is used for counting the text length corresponding to the target text;
the judging unit is used for judging whether the text length is larger than or equal to a preset text length threshold value or not;
and the execution unit is used for executing the step of generating the audio by the target text in a preset TTS mode if the text length is greater than or equal to the preset text length threshold value.
In an embodiment, the virtual face construction apparatus 70 based on phoneme text further includes:
and the sending unit is used for sending an alarm that the target text does not meet the preset requirement if the text length is smaller than the preset text length threshold.
It should be noted that, as can be clearly understood by those skilled in the art, the above-mentioned virtual face construction apparatus based on phoneme text and the specific implementation process of each unit may refer to the corresponding description in the foregoing method embodiment, and for convenience and brevity of description, no further description is provided here.
Meanwhile, the division and connection mode of each unit in the virtual face construction device based on the phoneme text are only used for illustration, in other embodiments, the virtual face construction device based on the phoneme text may be divided into different units as required, and each unit in the virtual face construction device based on the phoneme text may also adopt different connection orders and modes to complete all or part of the functions of the virtual face construction device based on the phoneme text.
The above-described virtual face construction apparatus based on phoneme text may be implemented in the form of a computer program that can be run on a computer device as shown in fig. 8.
Referring to fig. 8, fig. 8 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or part of another device.
Referring to fig. 8, the computer device 500 includes a processor 502, a memory, and a network interface 505 connected by a system bus 501, wherein the memory may include a non-volatile storage medium 503 and an internal memory 504, and the memory may also be a volatile computer-readable storage medium.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, may cause the processor 502 to perform a method for constructing a virtual face based on phoneme text as described above.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.
The internal memory 504 provides an environment for running the computer program 5032 in the non-volatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 may be enabled to execute a virtual face construction method based on the phoneme text.
The network interface 505 is used for network communication with other devices. Those skilled in the art will appreciate that the configuration shown in fig. 8 is a block diagram of only a portion of the configuration relevant to the present teachings and does not constitute a limitation on the computer device 500 to which the present teachings may be applied, and that a particular computer device 500 may include more or less components than those shown, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor, and in such embodiments, the structures and functions of the memory and the processor are consistent with those of the embodiment shown in fig. 8, and are not described herein again.
Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps: acquiring a target text, and generating audio from the target text in a preset TTS mode; performing phoneme segmentation on the audio to obtain a phoneme text corresponding to the audio; inputting the phoneme text into a preset cyclic neural network model to obtain the human face characteristics of a preset human anchor corresponding to the phoneme text; and constructing a virtual face of the preset human anchor corresponding to the phoneme text according to the face characteristics of the preset human anchor.
In an embodiment, before implementing the step of inputting the phoneme text into a preset recurrent neural network model to obtain a face feature of a preset human anchor corresponding to the phoneme text, the processor 502 further implements the following steps:
acquiring a training video sample recorded by a preset live anchor, wherein the training video sample comprises a training audio sample and the image of the preset live anchor;
performing phoneme segmentation on the training audio sample to obtain a training phoneme text, and extracting the face features of the training human anchor corresponding to the training phoneme text from the image of the preset human anchor;
and inputting the training phoneme text and the human face characteristics of the training human anchor into a preset cyclic neural network model so as to train the preset cyclic neural network model.
In an embodiment, when implementing the step of performing phoneme segmentation on the training audio sample to obtain a training phoneme text, and extracting a face feature of the training human anchor corresponding to the training phoneme text from the image of the preset human anchor, the processor 502 specifically implements the following steps:
acquiring a training audio sample corresponding to the training video sample;
according to the time sequence, performing phoneme segmentation on the training audio sample according to preset phonemes to obtain a training phoneme text sequence arranged according to the time sequence, wherein the training phoneme text sequence comprises a plurality of training phoneme text elements;
according to the time sequence, acquiring a video frame corresponding to the training phoneme text element from the training video sample, wherein the video frame comprises an image of the preset human anchor;
and extracting the appearance feature, the expression feature and the posture feature of the preset human anchor contained in the video frame corresponding to the training phoneme text element so as to obtain the face feature of the training human anchor corresponding to the training phoneme text.
In an embodiment, after the step of constructing the virtual face of the preset human anchor corresponding to the phoneme text according to the human face features of the preset human anchor is implemented by the processor 502, the following steps are further implemented:
acquiring all the phoneme texts contained in the audio and virtual faces corresponding to the phoneme texts;
combining all the virtual human faces into a video frame sequence according to the time sequence of the phoneme texts in the audio;
combining the audio with the sequence of video frames to obtain a virtual video.
In an embodiment, after the step of constructing the virtual face of the preset human anchor corresponding to the phoneme text according to the human face features of the preset human anchor is implemented by the processor 502, the following steps are further implemented:
generating other audio frequencies by the target text in other preset TTS modes, wherein the other audio frequencies and the audio frequencies are audio frequencies with different audio style types;
combining the other audio with the sequence of video frames to obtain other virtual video.
In an embodiment, before the step of generating audio by using a preset TTS manner for the target text, the processor 502 further performs the following steps:
counting the text length corresponding to the target text;
judging whether the text length is greater than or equal to a preset text length threshold value or not;
and if the text length is greater than or equal to the preset text length threshold value, executing the step of generating the audio by the target text in a preset TTS mode.
In an embodiment, after implementing the step of generating the audio from the target text in a preset TTS manner if the text length is greater than or equal to the preset text length threshold, the processor 502 further implements the following steps:
and if the text length is smaller than the preset text length threshold value, sending an alarm that the target text does not meet the preset requirement.
It should be understood that in the embodiment of the present Application, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiments may be implemented by a computer program, and the computer program may be stored in a computer readable storage medium. The computer program is executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.
Accordingly, the present application also provides a computer-readable storage medium. The computer readable storage medium may be a non-volatile computer readable storage medium, or may be a volatile computer readable storage medium, and the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program causes the processor to execute the steps of the virtual face construction method based on the phoneme text described in the above embodiments.
The computer readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or a memory of the device. The computer readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), etc. provided on the device. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the apparatus.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses, devices and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The storage medium is an entity and non-transitory storage medium, and may be various entity storage media capable of storing computer programs, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.
The steps in the method of the embodiment of the application can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the application can be combined, divided and deleted according to actual needs. In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing an electronic device (which may be a personal computer, a terminal, or a network device) to perform all or part of the steps of the method according to the embodiments of the present application.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, and these modifications or substitutions should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A virtual human face construction method based on phoneme texts is characterized by comprising the following steps:
acquiring a target text, and generating audio from the target text in a preset TTS mode;
performing phoneme segmentation on the audio to obtain a phoneme text corresponding to the audio;
inputting the phoneme text into a preset cyclic neural network model to obtain the human face characteristics of a preset human anchor corresponding to the phoneme text;
and constructing a virtual face of the preset human anchor corresponding to the phoneme text according to the face characteristics of the preset human anchor.
2. The method for constructing a virtual face based on phoneme texts as claimed in claim 1, wherein before the step of inputting the phoneme texts into a preset recurrent neural network model to obtain the face features of the preset human anchor corresponding to the phoneme texts, the method further comprises:
acquiring a training video sample recorded by a preset live anchor, wherein the training video sample comprises a training audio sample and the image of the preset live anchor;
performing phoneme segmentation on the training audio sample to obtain a training phoneme text, and extracting the face features of the training human anchor corresponding to the training phoneme text from the image of the preset human anchor;
and inputting the training phoneme text and the human face characteristics of the training human anchor into a preset cyclic neural network model so as to train the preset cyclic neural network model.
3. The method for constructing a virtual face based on phoneme texts as claimed in claim 2, wherein the step of performing phoneme segmentation on the training audio samples to obtain training phoneme texts and extracting the face features of the training human anchor corresponding to the training phoneme texts from the image of the preset human anchor comprises:
acquiring a training audio sample corresponding to the training video sample;
according to the time sequence, performing phoneme segmentation on the training audio sample according to preset phonemes to obtain a training phoneme text sequence arranged according to the time sequence, wherein the training phoneme text sequence comprises a plurality of training phoneme text elements;
according to the time sequence, acquiring a video frame corresponding to the training phoneme text element from the training video sample, wherein the video frame comprises an image of the preset human anchor;
and extracting the appearance feature, the expression feature and the posture feature of the preset human anchor contained in the video frame corresponding to the training phoneme text element so as to obtain the face feature of the training human anchor corresponding to the training phoneme text.
4. The method for constructing a virtual face based on a phoneme text as claimed in claim 1, wherein after the step of constructing the virtual face of the preset human anchor corresponding to the phoneme text according to the human face features of the preset human anchor, the method further comprises:
acquiring all the phoneme texts contained in the audio and virtual faces corresponding to the phoneme texts;
combining all the virtual human faces into a video frame sequence according to the time sequence of the phoneme texts in the audio;
combining the audio with the sequence of video frames to obtain a virtual video.
5. The method for constructing a virtual face based on a phoneme text as claimed in claim 4, wherein after the step of constructing the virtual face of the preset human anchor corresponding to the phoneme text according to the human face features of the preset human anchor, the method further comprises:
generating other audio frequencies by the target text in other preset TTS modes, wherein the other audio frequencies and the audio frequencies are audio frequencies with different audio style types;
combining the other audio with the sequence of video frames to obtain other virtual video.
6. The method for constructing a virtual face based on phoneme texts as claimed in claim 1, wherein before the step of generating audio from the target text by a preset TTS manner, the method further comprises:
counting the text length corresponding to the target text;
judging whether the text length is greater than or equal to a preset text length threshold value or not;
and if the text length is greater than or equal to the preset text length threshold value, executing the step of generating the audio by the target text in a preset TTS mode.
7. The method for constructing a virtual face based on phoneme texts as claimed in claim 6, wherein if the text length is greater than or equal to the preset text length threshold, after the step of generating the audio from the target text in a preset TTS manner, the method further comprises:
and if the text length is smaller than the preset text length threshold value, sending an alarm that the target text does not meet the preset requirement.
8. A virtual human face constructing device based on phoneme texts is characterized by comprising:
the first acquisition unit is used for acquiring a target text and generating an audio from the target text in a preset TTS mode;
the first segmentation unit is used for carrying out phoneme segmentation on the audio to obtain a phoneme text corresponding to the audio;
the first input unit is used for inputting the phoneme text into a preset recurrent neural network model so as to obtain the human face characteristics of a preset human anchor corresponding to the phoneme text;
and the construction unit is used for constructing the virtual face of the preset human anchor corresponding to the phoneme text according to the human face characteristics of the preset human anchor.
9. A computer device, comprising a memory and a processor coupled to the memory; the memory is used for storing a computer program; the processor is adapted to run the computer program to perform the steps of the method according to any of claims 1-7.
10. A computer-readable storage medium, characterized in that the storage medium stores a computer program which, when being executed by a processor, realizes the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011285914.9A CN112420014A (en) | 2020-11-17 | 2020-11-17 | Virtual face construction method and device, computer equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011285914.9A CN112420014A (en) | 2020-11-17 | 2020-11-17 | Virtual face construction method and device, computer equipment and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112420014A true CN112420014A (en) | 2021-02-26 |
Family
ID=74831520
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011285914.9A Pending CN112420014A (en) | 2020-11-17 | 2020-11-17 | Virtual face construction method and device, computer equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112420014A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113077537A (en) * | 2021-04-29 | 2021-07-06 | 广州虎牙科技有限公司 | Video generation method, storage medium and equipment |
| CN113096242A (en) * | 2021-04-29 | 2021-07-09 | 平安科技（深圳）有限公司 | Virtual anchor generation method and device, electronic equipment and storage medium |
| CN113205797A (en) * | 2021-04-30 | 2021-08-03 | 平安科技（深圳）有限公司 | Virtual anchor generation method and device, computer equipment and readable storage medium |
| CN113269872A (en) * | 2021-06-01 | 2021-08-17 | 广东工业大学 | Synthetic video generation method based on three-dimensional face reconstruction and video key frame optimization |
| CN113282791A (en) * | 2021-06-29 | 2021-08-20 | 京东科技控股股份有限公司 | Video generation method and device |
| CN113379875A (en) * | 2021-03-22 | 2021-09-10 | 平安科技（深圳）有限公司 | Cartoon character animation generation method, device, equipment and storage medium |
| CN114051105A (en) * | 2021-11-09 | 2022-02-15 | 北京百度网讯科技有限公司 | Multimedia data processing method and device, electronic equipment and storage medium |
| CN114245230A (en) * | 2021-11-29 | 2022-03-25 | 网易（杭州）网络有限公司 | Video generation method and device, electronic equipment and storage medium |
| CN115278293A (en) * | 2022-06-15 | 2022-11-01 | 平安科技（深圳）有限公司 | Virtual anchor generation method and device, storage medium and computer equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109637518A (en) * | 2018-11-07 | 2019-04-16 | 北京搜狗科技发展有限公司 | Virtual newscaster's implementation method and device |
| CN110853614A (en) * | 2018-08-03 | 2020-02-28 | Tcl集团股份有限公司 | Virtual object mouth shape driving method and device and terminal equipment |
| CN110866968A (en) * | 2019-10-18 | 2020-03-06 | 平安科技（深圳）有限公司 | Method for generating virtual character video based on neural network and related equipment |
| CN111243626A (en) * | 2019-12-30 | 2020-06-05 | 清华大学 | Speaking video generation method and system |
| CN111325817A (en) * | 2020-02-04 | 2020-06-23 | 清华珠三角研究院 | Virtual character scene video generation method, terminal device and medium |
-
2020
- 2020-11-17 CN CN202011285914.9A patent/CN112420014A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110853614A (en) * | 2018-08-03 | 2020-02-28 | Tcl集团股份有限公司 | Virtual object mouth shape driving method and device and terminal equipment |
| CN109637518A (en) * | 2018-11-07 | 2019-04-16 | 北京搜狗科技发展有限公司 | Virtual newscaster's implementation method and device |
| CN110866968A (en) * | 2019-10-18 | 2020-03-06 | 平安科技（深圳）有限公司 | Method for generating virtual character video based on neural network and related equipment |
| CN111243626A (en) * | 2019-12-30 | 2020-06-05 | 清华大学 | Speaking video generation method and system |
| CN111325817A (en) * | 2020-02-04 | 2020-06-23 | 清华珠三角研究院 | Virtual character scene video generation method, terminal device and medium |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113379875B (en) * | 2021-03-22 | 2023-09-29 | 平安科技（深圳）有限公司 | Cartoon character animation generation method, device, equipment and storage medium |
| CN113379875A (en) * | 2021-03-22 | 2021-09-10 | 平安科技（深圳）有限公司 | Cartoon character animation generation method, device, equipment and storage medium |
| CN113096242A (en) * | 2021-04-29 | 2021-07-09 | 平安科技（深圳）有限公司 | Virtual anchor generation method and device, electronic equipment and storage medium |
| CN113077537A (en) * | 2021-04-29 | 2021-07-06 | 广州虎牙科技有限公司 | Video generation method, storage medium and equipment |
| CN113205797A (en) * | 2021-04-30 | 2021-08-03 | 平安科技（深圳）有限公司 | Virtual anchor generation method and device, computer equipment and readable storage medium |
| CN113205797B (en) * | 2021-04-30 | 2024-03-05 | 平安科技（深圳）有限公司 | Virtual anchor generation method, device, computer equipment and readable storage medium |
| CN113269872A (en) * | 2021-06-01 | 2021-08-17 | 广东工业大学 | Synthetic video generation method based on three-dimensional face reconstruction and video key frame optimization |
| CN113282791A (en) * | 2021-06-29 | 2021-08-20 | 京东科技控股股份有限公司 | Video generation method and device |
| CN113282791B (en) * | 2021-06-29 | 2024-05-24 | 京东科技控股股份有限公司 | Video generation method and device |
| CN114051105A (en) * | 2021-11-09 | 2022-02-15 | 北京百度网讯科技有限公司 | Multimedia data processing method and device, electronic equipment and storage medium |
| CN114051105B (en) * | 2021-11-09 | 2023-03-10 | 北京百度网讯科技有限公司 | Multimedia data processing method and device, electronic equipment and storage medium |
| CN114245230A (en) * | 2021-11-29 | 2022-03-25 | 网易（杭州）网络有限公司 | Video generation method and device, electronic equipment and storage medium |
| CN115278293A (en) * | 2022-06-15 | 2022-11-01 | 平安科技（深圳）有限公司 | Virtual anchor generation method and device, storage medium and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112420014A (en) | Virtual face construction method and device, computer equipment and computer readable medium | |
| CN109377539B (en) | Method and apparatus for generating animation | |
| US9361722B2 (en) | Synthetic audiovisual storyteller | |
| CN111489424A (en) | Virtual character expression generation method, control method, device and terminal equipment | |
| US10970909B2 (en) | Method and apparatus for eye movement synthesis | |
| CN114895817B (en) | Interactive information processing method, network model training method and device | |
| CN112329451B (en) | Sign language action video generation method, device, equipment and storage medium | |
| CN113077537B (en) | Video generation method, storage medium and device | |
| CN112397056B (en) | Voice evaluation method and computer storage medium | |
| US7257538B2 (en) | Generating animation from visual and audio input | |
| CN113299312A (en) | Image generation method, device, equipment and storage medium | |
| CN115700772A (en) | Face animation generation method and device | |
| JPWO2018147193A1 (en) | Model learning device, estimation device, their methods, and programs | |
| CN113111812A (en) | Mouth action driving model training method and assembly | |
| CN114242033A (en) | Speech synthesis method, apparatus, device, storage medium and program product | |
| US20230298564A1 (en) | Speech synthesis method and apparatus, device, and storage medium | |
| CN114283783A (en) | Speech synthesis method, model training method, device and storage medium | |
| CN114255737B (en) | Voice generation method and device and electronic equipment | |
| CN113903347A (en) | Voice processing method, device, equipment and medium based on machine learning | |
| Filntisis et al. | Video-realistic expressive audio-visual speech synthesis for the Greek language | |
| Deena et al. | Visual speech synthesis using a variable-order switching shared Gaussian process dynamical model | |
| San-Segundo et al. | Proposing a speech to gesture translation architecture for Spanish deaf people | |
| CN114581570B (en) | Three-dimensional face action generation method and system | |
| CN113299270B (en) | Method, device, equipment and storage medium for generating voice synthesis system | |
| CN113205797B (en) | Virtual anchor generation method, device, computer equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |