CN112633947A

CN112633947A - Text generation model generation method, text generation method, device and equipment

Info

Publication number: CN112633947A
Application number: CN202011643231.6A
Authority: CN
Inventors: 宋珍巧; 陈家泽; 周浩; 李磊
Original assignee: Beijing Youzhuju Network Technology Co Ltd
Current assignee: Beijing Youzhuju Network Technology Co Ltd
Priority date: 2020-12-30
Filing date: 2020-12-30
Publication date: 2021-04-09
Anticipated expiration: 2040-12-30
Also published as: WO2022143105A1; CN112633947B

Abstract

The embodiment of the disclosure discloses a text generation model generation method, a text generation device and text generation equipment, wherein the text generation model generation method comprises the following steps: training an initialization language encoder by using text samples of a plurality of languages and parallel corpora among the languages; initializing a text generation model according to parameters of an initialization language encoder to obtain an initial text generation model; performing paradigm training on an initial text generation model based on a text sample, a parallel corpus and a keyword and target text sample pair generated by the construction of the text sample; and performing parameter adjustment on the initial text generation model subjected to the paradigm training through the keyword and target text sample pair to obtain a final text generation model. The embodiment of the disclosure solves the problems that the number of text samples in non-Chinese languages is small, and multi-language rich text resources cannot be fully utilized to generate related target texts, and realizes the generation of the language case corresponding to the keywords in different languages.

Description

Text generation model generation method, text generation method, device and equipment

Technical Field

The embodiment of the disclosure relates to the field of computer application, in particular to a text generation model generation method, a text generation device and text generation equipment.

Background

In the process of product popularization, because the languages used by the popularization target population are different, the corresponding popularization documents need to use different languages, for example, a product is popularized to Chinese to use a Chinese advertisement document, and the product is popularized to Japanese or English advertisement documents. The advertisement platform is expected to provide relevant and various advertisement texts of corresponding language types for the advertisers according to the advertisement keywords of any language provided by the advertisers.

However, in the current neural network model capable of generating an advertisement copy, the advertisement copy is obtained only by training according to a text sample of a single language, and text sample data of multiple languages cannot be utilized, and a corresponding advertisement copy cannot be generated directly according to a keyword to be used for providing services for advertisers in advertisement services.

BRIEF SUMMARY OF THE PRESENT DISCLOSURE

The embodiment of the disclosure provides a text generation model generation method, a text generation device and text generation equipment, so as to generate corresponding texts in the same language according to a query keyword in a certain language.

In a first aspect, an embodiment of the present disclosure provides a text generation model generation method, where the method includes:

training an initialization language encoder by using text samples of a plurality of languages and parallel corpora among the languages;

initializing a text generation model according to the parameters of the initialization language encoder to obtain an initial text generation model;

performing paradigm training on the initial text generation model based on the text samples of the languages, the parallel linguistic data and a keyword and target text sample pair constructed and generated by the text samples of the languages;

and performing parameter adjustment on the initial text generation model subjected to the paradigm training through the keyword and target text sample pair to obtain a final text generation model.

In a second aspect, an embodiment of the present disclosure further provides a text generation method, where the method includes:

acquiring text generation keywords, and generating a text generation model based on the text generation model generation method according to any embodiment of the disclosure;

and inputting the text generation keywords into a text generation model to obtain a target text.

In a third aspect, an embodiment of the present disclosure further provides a text generation model generation apparatus, where the apparatus includes:

the system comprises an encoder pre-training module, a language encoder initialization module and a language encoder initialization module, wherein the encoder pre-training module is used for training an initialization language encoder by using text samples of a plurality of languages and parallel corpora among the languages;

the model initialization module is used for initializing a text generation model according to the parameters of the initialization language encoder to obtain an initial text generation model;

the model normal form training module is used for carrying out normal form training on the initial text generation model based on the text samples of the languages, the parallel linguistic data and a keyword and target text sample pair constructed and generated by the text samples of the languages;

and the model parameter determining module is used for carrying out parameter adjustment on the initial text generation model after the paradigm training through the keyword and target text sample pair to obtain a final text generation model.

In a fourth aspect, an embodiment of the present disclosure further provides a text generating apparatus, where the apparatus includes:

the data acquisition module is used for acquiring text generation keywords and generating a text generation model based on the text generation model generation method of any embodiment of the disclosure;

and the text generation module is used for inputting the text generation keywords into a text generation model to obtain a target text.

In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, where the electronic device includes:

one or more processors;

a memory for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement a text generation model generation method or a text generation method as described in any embodiment of the present disclosure.

In a sixth aspect, the embodiments of the present disclosure further provide a computer storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a text generation model generation method or a text generation method as described in any one of the embodiments of the present disclosure.

The method comprises the steps that text samples of multiple languages and parallel corpora among the multiple languages are used as model training data, an initialization language encoder is obtained through training, and the initialization language encoder learns all the text samples semantically; initializing a text generation model by using parameters of an initialization language encoder to obtain an initial text generation model; further, based on text samples and parallel corpora of a plurality of languages, performing paradigm training on the initial text generation model, so that the text generation model can output a target text according to the input keywords; and finally, performing parameter adjustment on the initial text generation model after the paradigm training through the key words in the text samples of the languages and the target text sample pairs to obtain a final text generation model. The method solves the problems that in the prior art, the number of text samples of non-Chinese languages is small, multi-language rich text resources cannot be fully utilized, and related target advertisements cannot be generated according to keywords of different languages, and realizes generation of target texts of corresponding languages according to the keywords of different languages.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.

FIG. 1 is a flowchart of a text generation model generation method in a first embodiment of the disclosure;

FIG. 2 is a diagram of an initialization language coder model according to a first embodiment of the present disclosure;

FIG. 3 is a diagram of a denoised self-encoder model according to a first embodiment of the disclosure;

FIG. 4 is a diagram of a cross-language self-coder model in a first embodiment of the disclosure;

FIG. 5 is a schematic diagram of a word prediction model in a first embodiment of the disclosure;

FIG. 6 is a flowchart of a text generation model generation method in the second embodiment of the disclosure;

fig. 7 is a flowchart of a text generation method in a third embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of a text-generating model generating apparatus in the fourth embodiment of the present disclosure;

fig. 9 is a schematic structural diagram of a text generation apparatus in the fifth embodiment of the present disclosure

Fig. 10 is a schematic structural diagram of an electronic device in a sixth embodiment of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.

It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.

It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.

The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.

Example one

Fig. 1 is a flowchart illustrating a method for generating a text generative model according to an embodiment of the present disclosure, which is applicable to a case of training a text generative model based on text corpora of different languages.

As shown in fig. 1, the method for generating a text generation model provided in the embodiment of the present disclosure includes the following steps:

and S110, training an initialization language encoder by using text samples of a plurality of languages and parallel corpora among the languages.

The languages can be languages for obtaining a certain number of text samples, more common languages include multiple languages such as chinese, english, japanese, german, french, and chinese, and text contents of a certain number of languages can be selected as samples for model training according to the application range of the model. Furthermore, text content related to the corresponding field can be selected as a text sample according to the field of the model application. For example, in the medical field, text contents of various languages are related contents related to the medical field; or, the application field of the model to be trained is the makeup field, and then the text contents of various languages are related to the makeup field. The parallel corpora among a plurality of languages refer to text contents with the same semantics among different languages. For example, "I'm so happy" in Chinese corresponds to "I'm so happy" in English. In the process of model training, the parallel linguistic data between every two languages can be used as a group of parallel linguistic data samples.

Specifically, in the process of initializing a language encoder by using text samples of multiple languages and parallel corpus pre-training among the multiple languages, model training may be performed by using a Mask Language Model (MLM) paradigm based on the text samples of the languages to obtain a first bidirectional encoder; then, after splicing two language texts in parallel corpora of every two languages, model training is carried out on the basis of a first bidirectional encoder by using a mask language model paradigm, namely cross-linear MLM (XMLM) paradigm, so as to obtain an initial text generation model, namely a cross-language bidirectional encoder.

For example, the process of obtaining the initial text generation model may refer to the schematic diagram shown in fig. 2. In fig. 2, in the first stage, the content of each single-language text sample is input into an Encoder, before the text sample is input into the Encoder (Encoder), words in a certain proportion in the text are randomly replaced with Mask marks [ Mask ], and then the words are input into the Encoder, and the replaced words are predicted by the Encoder and output until a first bidirectional Encoder meeting the requirement of a loss function is obtained by training. In the second stage, on the basis of the first bidirectional encoder, the source sentence (source sensor) and the target sentence (target sensor) of each pair of parallel sentences in the parallel corpus are spliced together, and then the MLM is made. For example, the 'You are' of English (EN) and the 'You are' of Chinese (ZH) are spliced, model training is carried out through an MLM paradigm, and finally an initial text generation model is obtained. Of course, the training sequence of the MLM paradigm and the XMLM paradigm is not critical and may be performed alternately. In fig. 2, Tokens refers to a marker of a character sequence, and in the process of computer language processing, a time character sequence is converted into a marker (token) sequence, and is recorded and used in the form of a marker. Languages then represents the Languages corresponding to the source sentence and the target sentence in the input set of parallel predictions, and Positions represents the position of each character or character string in the text sequence.

The existing text sample can be more fully utilized through the operation of the step, and the text corpus is expanded. For example, in an application field, for a chinese user, more text contents can be obtained, and fewer text data in the same application field can be obtained in other languages (e.g., japanese and english), so that text contents corresponding to chinese and not previously obtained in other languages can be obtained through conversion of parallel corpora, thereby enriching corpora between different languages. The same is true for users using other languages.

And S120, initializing a text generation model according to the parameters of the initialization language encoder to obtain an initial text generation model.

Specifically, in this step, an initial text generation model pre-trained in the first stage (step S110) is used, a cross-language bidirectional Encoder (cross-linear Encoder) is used, and an Encoder encorder and a Decoder in the text generation model are respectively initialized, so as to obtain the initial text generation model. With the training basis of the first stage, in the following model paradigm training process, the initial text generation model can better understand the input text sample semantically, and the model can realize the convergence of the loss function more quickly and complete the model training process.

S130, performing paradigm training on the initial text generation model based on the text samples of the languages, the parallel linguistic data and a keyword and target text sample pair constructed and generated by the text samples of the languages.

And performing paradigm training, namely enabling the model to output a corresponding target text according to the input keywords according to the use requirements of the user. The method can directly use keywords extracted from text samples of various languages and text sample pairs corresponding to the target samples to carry out model paradigm training, and inputs the keywords into the Encoder so that the Decoder outputs the target text.

However, considering that there is a certain interval between the model training paradigm at the first stage and the autoregressive generation mode, in this embodiment, the model is pre-trained by a generation task, and specifically, the following steps may be taken:

firstly, generating a model training for an initial text through text samples of various languages to obtain a denoising autoencoder. De-noising auto-encoding (DAE) means that given an input x, some noise is added to x to make it become x ', the encoder input x ', and the decoder needs to decode the original x based on x '. In this embodiment, the traditional DAE is slightly modified to obtain a generative model with a tighter semantic join. For example, a text sample of each single language is sampled by a preset length using a poisson distribution (λ ═ 3), a sequence whose continuous length from a specified position in x is the preset length is replaced with a MASK symbol [ MASK ] to obtain x ', and then, the original sequence x is decoded by a decoder based on the input signal added with noise at the encoder input x'. The preset length of samples is about 15% of the original sequence length. This process can be referred to the schematic diagram shown in fig. 3. In FIG. 3, x is "Today is a raw day! ", x' is" Today is a [ MASK ]! ".

Then, on the basis of the denoising autoencoder, model training is carried out on the basis of the parallel corpus to obtain the cross-language autoencoder. In this step, similar to the input process of the de-drying self-encoder, the difference is that the text generation model is enabled to output the corresponding parallel corpus text according to the input text. Reference may be made to the schematic diagram shown in fig. 4. In FIG. 4, x is "Today is a raw day! ", x' is" Today is a [ MASK ]! "the output is" today is a rainy day! ".

And finally, taking the keywords extracted from the text samples and the target text samples as sample pairs, and performing model training on the cross-language self-encoder, so that the model predicts the text content associated with the keywords, and the normal form training of the initial text generation model is completed. Specifically, the keyword in the text sample and the target text sample are paired to pair of pair < x, y >, the keyword x is input in the encoder, and the first few words in the target text y are input in the decoder, so that the decoder decodes the subsequent words, which is equivalent to giving a prompt to the decoder, and can decode and learn the content to be output more quickly. Reference may be made to the model training process illustrated in fig. 5. The keyword "Weather" is entered in the encoder, "Today is" is entered in the decoder, and "a raw day! ".

And S140, performing parameter adjustment on the initial text generation model subjected to the paradigm training through the keyword and target text sample pair to obtain a final text generation model.

The initial text generation model after the paradigm training can perform semantic recognition according to the input text content, and output the input text itself, the corresponding parallel corpora or the corresponding target text. And then, utilizing the key words in the text samples of various languages to be in pair with the target text sample, carrying out parameter adjustment on the initial text generation model after the paradigm training, and finishing the process of model training when the similarity between the model prediction output and the corresponding result meets the preset degree.

According to the technical scheme of the embodiment of the disclosure, an initialized language encoder is obtained by training text samples of a plurality of languages and parallel corpora among the languages as model training data, so that the initialized language encoder learns each text sample semantically; initializing a text generation model by using parameters of an initialization language encoder to obtain an initial text generation model; further, based on text samples and parallel corpora of a plurality of languages, performing paradigm training on the initial text generation model, so that the text generation model can output a target text according to the input keywords; and finally, performing parameter adjustment on the initial text generation model after the paradigm training through the key words in the text samples of the languages and the target text sample pairs to obtain a final text generation model. The method solves the problems that in the prior art, the number of text samples of non-Chinese languages is small, multi-language rich text resources cannot be fully utilized, and related target texts cannot be generated according to keywords of different languages, and realizes generation of the target texts of corresponding languages according to the keywords of different languages.

Example two

On the basis of the above embodiment, the present embodiment further describes a text sample data construction process, which belongs to the same inventive concept as the model training method proposed in the above embodiment, and reference may be made to the above embodiment for technical details which are not described in detail in the present embodiment.

Fig. 6 shows a flowchart of a text generation model generation method provided in the second embodiment of the present disclosure, where the text generation model generation method provided in the second embodiment of the present disclosure includes the following steps:

s210, constructing data based on the text samples of multiple languages and the parallel linguistic data among the languages by using keywords and target text samples.

In a specific embodiment, an application scenario of the text generation model is generation of an advertisement scheme, and text samples of multiple languages are advertisement texts of multiple languages. Of course, existing advertisement text data has no existing corresponding keywords (i.e., search term query of an article), and the keywords need to be extracted in advance to construct keyword-target text sample pair data for final model paradigm training.

Firstly, a part-of-speech tagging tool (pos tagging) can be used to tag text samples of various languages, so as to obtain part-of-speech tags of various words in the text samples. Then, all noun phrases including vernoun phrases, adjective nouns and noun phrases are extracted from the text sample based on the part-of-speech tags, and an association relation between each noun phrase and the corresponding text sample is established. Generally, a text sample can be extracted from a plurality of noun phrases, so as to construct a plurality of keyword and corresponding target text sample pairs. Then, semantic filtering is further performed, a keyword and target text sample pair with low semantic correlation between the noun phrase and the target text sample is kept, and a keyword and target text sample pair with semantic correlation larger than a preset threshold is kept. Wherein, text filtering can be carried out through phrase embedding.

On the basis of the above process, the expansion of the keyword and target text sample pair can be further performed. Specifically, vector representations of constructed keywords and keywords in a target text sample pair and vector representations of target text samples are obtained through TextCNN respectively; then, optimizing the semantic relevance of each keyword vector and a target text sample vector in a negative sampling mode; therefore, the keyword and target text sample pairs can be expanded based on embedding. Specifically, the relevance between the vector representation of each keyword and the vector representations of all target texts is calculated respectively, and the keywords with relevance values meeting preset conditions and the target text samples form a keyword and target text sample pair. Illustratively, noun phrases A, B and C are extracted from Chinese text 1, and noun phrases D and E are extracted from Chinese text 2; through the calculation of semantic relevance between each noun phrase (A, B, C, D and E) and the Chinese text 1 and the Chinese text 2, if the semantic relevance between the noun phrase D and the Chinese text 1 meets a preset condition, the corresponding relation between the D and the Chinese text 1, namely the keyword-target text sample pair, can be established. By the method, the data can be expanded by 2-5 times by the keyword-target text sample.

S220, training an initialization language encoder by using text samples of a plurality of languages and parallel corpora among the languages.

And S230, initializing a text generation model according to the parameters of the initialization language encoder to obtain an initial text generation model.

S240, performing paradigm training on the initial text generation model based on the text samples of the languages, the parallel linguistic data and a keyword and target text sample pair constructed and generated by the text samples of the languages.

And S250, classifying the data by the keywords and the target text samples, and respectively carrying out parameter adjustment on the initial text generation model after the paradigm training by the keywords and the target text samples of different categories to obtain a final text generation model.

Specifically, the keyword and target text sample pairs may be divided into at least a first sample set and a second sample set according to sample data quality, where the sample quality in the second sample set is better than the sample quality in the first sample set; and then, initially adjusting parameters of the initial text generation model after the pattern training by using the sample data in the first sample set, and then performing parameter adjustment on the initial text generation model after the pattern training by using the sample data in the second sample set on the basis of the initial parameter adjustment. For example, it is assumed that the advertisement text as the text sample is derived from the advertisement data of different channels, wherein the advertisements of some channels are more suitable for the advertiser's requirements on the advertisement style, etc., and the advertisement documents of the rest channels are not very close to the advertiser's user requirements. The method comprises the steps of firstly using an advertisement case which is not close to the requirement of an advertiser as sample data in a first sample set, carrying out model training to carry out parameter preliminary adjustment, then using an advertisement case which is close to the requirement of an advertiser user as a sample in a second sample set to carry out model training, and further adjusting the preliminarily adjusted model so as to determine a final text generation model.

According to the technical scheme of the embodiment, on the basis of the embodiment, firstly, data construction of a keyword-target text sample pair is carried out on the basis of text samples of various languages, and then, an initialized language encoder is obtained by taking the text samples of the languages and parallel corpora among the languages as model training data; initializing a text generation model by using parameters of an initialization language encoder to obtain an initial text generation model; based on text samples of multiple languages, parallel linguistic data and constructed keyword-target text sample pair data, performing paradigm training on an initial text generation model to enable the text generation model to output a target text according to input keywords; and finally, carrying out parameter adjustment on the initial text generation model after the paradigm training in stages through key words in the text samples of the languages and target text sample pairs to obtain a final optimized text generation model. The method solves the problems that in the prior art, the number of text samples in non-Chinese languages is small, multi-language rich text resources cannot be fully utilized, and related target texts cannot be generated according to keywords in different languages, achieves the quantity amplification of keyword-target text samples, can generate target texts in corresponding languages according to the keywords in different languages, and can generate target advertisement documents based on the keywords if the method is applied to the field of advertisement documents.

EXAMPLE III

Fig. 7 is a flowchart of a text generation method provided by a third embodiment of the present disclosure, where the third embodiment of the present disclosure is applicable to a case where a target text is generated according to a keyword, and the method may be implemented by a text generation apparatus, and may be implemented by software and/or hardware in a mobile terminal.

As shown in fig. 7, the text generation method includes the steps of:

s310, obtaining text generation keywords, and generating a text generation model based on the text model generation method according to any embodiment of the disclosure.

The text generation keyword may be a keyword provided by a user to obtain the target text. When the generated target text is an advertisement case related to the text generation keyword, the sample data of the text generation model generated based on the text model generation method provided by any embodiment of the disclosure is the currently-acquired advertisement case data of different languages and the parallel linguistic data among different languages.

And S320, inputting the text generation keywords into a text generation model to obtain a target text.

And inputting the text generation keywords into a text generation model, and outputting a corresponding target text with highest relevance by the model after calculation. For example, the keyword lace dress is input into the model, and the output advertisement file can be a new style lace dress in autumn, which is slim and slim.

In a preferred embodiment, before the text-generating model is used, the output parameters of the model are set, so that a decoder of the text-generating model performs decoding in a greedy decoding manner, and thus after the text-generating keyword is input into the text-generating model, a plurality of target texts can be obtained. For example, a constraint sampling mode can be used for setting the output process of the text generation model, for the first K steps, a temperature is added to the predicted distribution to make the predicted distribution become more sharp, and sampling is used for decoding; starting from the K +1 st step, the system decodes by greedy decoding to obtain a plurality of outputs. That is, for the first K words in the output target text, only the word with the highest probability corresponding to each word is taken, and for the output K +1 th word, the first few words (e.g. 3 words) with the highest probability can be output according to the setting, so that a plurality of K +1 th words and the first K words are combined to obtain a plurality of target texts. If the subsequent K +2 th word has a plurality of choices, more target texts can be obtained. More target texts can be provided for the user, and the optimal scheme can be selected. Still taking the keyword "lace dress" as an example, the corresponding output target text may further include "new autumn lace dress, high definition of designer" and "new autumn lace dress, body shaping is of quality" and the like.

According to the technical scheme of the embodiment of the disclosure, the keywords are input into the text generation model, the target texts related to the keywords can be directly obtained, and the text generation model can be enabled to decode and output a plurality of target texts, so that a mode for directly generating the advertisement copy is provided. And the text generation model is trained on the basis of multilingual text samples, and can generate target texts in multiple languages. The problem that related target texts cannot be generated according to keywords in different languages in the prior art is solved, the target texts in the corresponding languages are generated according to the keywords in the different languages, and if the method is applied to the field of advertisement documents, the target advertisement documents can be generated based on the keywords.

Example four

Fig. 8 is a schematic structural diagram of a text generation model generation apparatus according to a fourth embodiment of the present disclosure, which is applicable to a case of performing text generation model training based on text corpora of different languages, and the text generation model generation method according to the fourth embodiment of the present disclosure may be implemented through the text generation model generation method according to the fourth embodiment of the present disclosure.

As shown in fig. 8, the text generation model generation apparatus in the embodiment of the present disclosure includes: an encoder pre-training module 410, a model initialization module 420, a model paradigm training module 430, and a model parameter determination module 440.

The encoder pre-training module 410 is configured to train an initialization language encoder by using text samples of multiple languages and parallel corpora between the multiple languages; a model initialization module 420 for initializing a text generation model according to the parameters of the initialization language encoder to obtain an initial text generation model; a model normal form training module 430, configured to perform normal form training on the initial text generation model based on the text samples in the multiple languages, the parallel corpora, and a keyword and target text sample pair constructed and generated by the text samples in the multiple languages; and the model parameter determining module 440 is configured to perform parameter adjustment on the initial text generation model subjected to the paradigm training through the keyword and target text sample pair to obtain a final text generation model.

According to the technical scheme of the embodiment, text samples of multiple languages and parallel corpora among the multiple languages are used as model training data, an initialization language encoder is obtained through training, and the initialization language encoder learns the text samples semantically; initializing a text generation model by using parameters of an initialization language encoder to obtain an initial text generation model; further, based on text samples and parallel corpora of a plurality of languages, performing paradigm training on the initial text generation model, so that the text generation model can output a target text according to the input keywords; and finally, performing parameter adjustment on the initial text generation model after the paradigm training through the key words in the text samples of the languages and the target text sample pairs to obtain a final text generation model. The method solves the problems that in the prior art, the number of text samples of non-Chinese languages is small, multi-language rich text resources cannot be fully utilized, and related target advertisements cannot be generated according to keywords of different languages, and realizes generation of target texts of corresponding languages according to the keywords of different languages.

Optionally, the encoder pre-training module 410 is specifically configured to:

based on each language text sample, performing model training by using a mask language model paradigm to obtain a first bidirectional encoder;

and after splicing two language texts in parallel corpora of every two languages, performing model training on the basis of the first bidirectional encoder by using the paradigm of the mask language model to obtain the initial text generation model.

Optionally, the model paradigm training module 430 includes:

the first model training submodule is used for training the initial text generation model through text samples of various languages to obtain a denoising autoencoder;

the second paradigm training submodule is used for carrying out model training on the basis of the denoising autoencoder and based on the parallel corpus to obtain a cross-language autoencoder;

and the third paradigm training submodule is used for performing model training on the cross-language self-encoder by using the keyword and target text sample pair to finish paradigm training of the initial text generation model.

Optionally, the first paradigm training submodule is specifically configured to:

and replacing the sequence with preset length at the appointed position in each text sample with a mask code, inputting the mask code to an encoder of the initial text generation model for model training, and enabling a decoder of the initial text generation model to output the original text sample without sequence replacement.

Optionally, the second paradigm training submodule is specifically configured to:

and replacing the sequence with preset length at a specified position in each text sample with a mask code, inputting the mask code to an encoder of the denoising self-encoder for model training, and enabling a decoder of the denoising self-encoder to output the parallel corpus of the original text sample without sequence replacement.

Optionally, the third paradigm training submodule is specifically configured to:

and for each keyword and target text sample pair, inputting the keywords to an encoder of the cross-language self-encoder, and inputting partial words in the target text to a decoder of the cross-language self-encoder for model training, so that the decoder of the cross-language self-encoder outputs words which are not input into the decoder in the target text.

Optionally, the model parameter determining module 440 is specifically configured to:

dividing the keyword and target text sample pairs into a first sample set and a second sample set according to the sample data quality, wherein the sample quality in the second sample set is superior to that in the first sample set;

using sample data in the first sample set to perform preliminary parameter adjustment on the initial text generation model subjected to the paradigm training;

and on the basis of the preliminary parameter adjustment, performing parameter adjustment on the initial text generation model after the paradigm training by using the sample data in the second sample set.

Optionally, the text generation model generating apparatus further includes a sample data constructing module:

the system comprises a word tagging tool, a word processing tool and a word processing tool, wherein the word tagging tool is used for carrying out word tagging on text samples of various languages and extracting all noun phrases in the text samples according to tagging results;

filtering out noun phrases of which the semantic relevance with the corresponding text sample is smaller than a preset threshold value from the noun phrases;

and establishing association relations between the filtered noun phrases and the corresponding text samples respectively to form the keyword and target text sample pairs.

Optionally, the sample data constructing module is further configured to:

after the incidence relation between the filtered noun phrases and the corresponding text samples is established respectively to form the keyword and target text sample pairs, optimizing the vector representation of the keywords in the keyword and target text sample pairs and the vector representation of the target text samples;

respectively calculating the correlation between each keyword and all target text samples according to the optimized vector representation of the keywords and the vector representation of the target text samples;

and combining the keywords with the correlation meeting the preset conditions and the target text sample to form a keyword and target text sample pair.

Optionally, the text sample is an advertisement text of multiple languages.

The text generation model generation device provided by the embodiment of the disclosure is the same as the text generation model generation method provided by the embodiment, and technical details which are not described in detail in the embodiment of the disclosure can be referred to the embodiment, and the embodiment of the disclosure has the same beneficial effects as the embodiment.

EXAMPLE five

Fig. 9 is a schematic structural diagram of a text generating apparatus according to a fifth embodiment of the present disclosure, where the fifth embodiment of the present disclosure is applicable to a case where a target text is generated according to a keyword, and the text generating apparatus according to the fifth embodiment of the present disclosure may implement the text generating method according to the foregoing embodiment.

As shown in fig. 9, the text generation apparatus in the embodiment of the present disclosure includes: a data acquisition module 510 and a text generation module 520.

The data obtaining module 510 is configured to obtain a text generation keyword, and generate a text generation model based on a text generation module generation method described in any embodiment of the present disclosure; and the text generation module 520 is configured to input the text generation keywords into a text generation model to obtain a target text.

Optionally, the text generating apparatus further includes a model decoding module, configured to:

before using the text generation model, setting decoder parameters of the text generation model, and enabling the decoder to output a plurality of target texts in a greedy decoding mode.

The text generation device provided by the embodiment of the disclosure belongs to the same inventive concept as the text generation method provided by the embodiment, and technical details which are not described in detail in the embodiment of the disclosure can be referred to the embodiment, and the embodiment of the disclosure has the same beneficial effects as the embodiment.

EXAMPLE six

Referring now to FIG. 10, a block diagram of an electronic device 600 suitable for use in implementing embodiments of the present disclosure is shown. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., car navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 10 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 10, the electronic device 600 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 601 that may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage means 606 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

Generally, the following devices may be connected to the I/O interface 605: input devices 604 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 606 including, for example, magnetic tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 10 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such embodiments, the computer program may be downloaded and installed from a network through the communication device 609, or installed from the storage device 606, or installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of the embodiments of the present disclosure.

It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: training an initialization language encoder by using text samples of a plurality of languages and parallel corpora among the languages; initializing a text generation model according to the parameters of the initialization language encoder to obtain an initial text generation model; performing paradigm training on the initial text generation model based on the text samples of the languages and the parallel corpora; and performing parameter adjustment on the initial text generation model after the paradigm training through the key words in the text samples of the languages and the target text sample pairs to obtain a final text generation model.

Alternatively, the computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:

acquiring text generation keywords, and generating a text generation model based on the text generation model generation method according to any embodiment of the disclosure; and inputting the text generation keywords into a text generation model to obtain a target text.

Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of a unit does not in some cases constitute a limitation of the unit itself, for example, the first retrieving unit may also be described as a "unit for retrieving at least two internet protocol addresses".

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, [ example one ] there is provided a text generative model generation method comprising:

In accordance with one or more embodiments of the present disclosure, [ example two ] there is provided the method of example one, further comprising:

the training initialization language encoder using the text samples of a plurality of languages and the parallel corpora among the languages includes:

In accordance with one or more embodiments of the present disclosure, [ example three ] there is provided the method of example one, further comprising:

constructing a generated keyword and target text sample pair based on the text samples of the languages, the parallel language materials and the text samples of the languages, and performing paradigm training on the initial text generation model:

training the initial text generation model through text samples of various languages to obtain a denoising autoencoder;

on the basis of the denoising autoencoder, performing model training based on the parallel corpus to obtain a cross-language autoencoder;

and performing model training on the cross-language self-encoder by using the keyword and target text sample pairs to finish the paradigm training of the initial text generation model.

According to one or more embodiments of the present disclosure, [ example four ] there is provided the method of example three, further comprising:

the method for training the initial text generation model through the text samples of various languages to obtain the denoising autoencoder comprises the following steps:

In accordance with one or more embodiments of the present disclosure, [ example five ] there is provided the method of example three, further comprising:

on the basis of the denoising autoencoder, model training is carried out based on the parallel corpus to obtain a cross-language autoencoder, and the method comprises the following steps:

In accordance with one or more embodiments of the present disclosure, [ example six ] there is provided the method of example three, further comprising:

performing model training on the cross-language self-encoder by using the keyword and target text sample pair to complete the paradigm training of the initial text generation model, wherein the model training comprises:

In accordance with one or more embodiments of the present disclosure, [ example seven ] there is provided the method of example one, further comprising:

the parameter adjustment of the initial text generation model after the paradigm training is performed through the keyword and target text sample pair, and the parameter adjustment comprises the following steps:

In accordance with one or more embodiments of the present disclosure, [ example eight ] there is provided the method of example one, further comprising:

the construction process of the keyword and target text sample pair comprises the following steps:

performing part-of-speech tagging on the text samples of various languages by using a part-of-speech tagging tool, and extracting all noun phrases in the text samples according to tagging results;

In accordance with one or more embodiments of the present disclosure, [ example nine ] there is provided the method of example eight, further comprising:

after the filtered noun phrases are respectively associated with corresponding text samples to form the keyword and target text sample pairs, the construction process further includes:

optimizing vector representations of the keywords and the target text sample in the keyword and target text sample pair;

In accordance with one or more embodiments of the present disclosure, [ example v ] there is provided the method of example i, further comprising:

the text sample is advertisement text of a plurality of languages.

According to one or more embodiments of the present disclosure, [ example eleven ] there is provided a text generation method including:

In accordance with one or more embodiments of the present disclosure, [ example twelve ] there is provided the method of example eleven, further comprising:

According to one or more embodiments of the present disclosure, [ example thirteen ] there is provided a text-generating model generating apparatus including:

the model initialization module initializes a text generation model according to the parameters of the initialization language encoder to obtain an initial text generation model;

In accordance with one or more embodiments of the present disclosure, [ example fourteen ] there is provided the apparatus of example thirteen, further comprising:

the encoder pre-training module is specifically configured to:

According to one or more embodiments of the present disclosure, [ example fifteen ] there is provided the apparatus of example thirteen, further comprising:

the model paradigm training module includes:

In accordance with one or more embodiments of the present disclosure, [ example sixteen ] there is provided the apparatus of example fifteen, further comprising:

the first paradigm training submodule is specifically configured to:

According to one or more embodiments of the present disclosure, [ example seventeen ] there is provided the apparatus of example fifteen, further comprising:

the second paradigm training submodule is specifically configured to:

In accordance with one or more embodiments of the present disclosure, [ example eighteen ] there is provided the apparatus of example fifteen, further comprising:

the third paradigm training submodule is specifically configured to:

In accordance with one or more embodiments of the present disclosure, [ example nineteen ] provides the apparatus of example thirteen, further comprising:

the model parameter determination module is specifically configured to:

In accordance with one or more embodiments of the present disclosure, [ example twenty ] there is provided the apparatus of example thirteen, further comprising a sample data construction module:

In accordance with one or more embodiments of the present disclosure, [ example twenty-one ] provides the apparatus of example twenty, further comprising:

the sample data construction module is further configured to:

In accordance with one or more embodiments of the present disclosure, [ example twenty-two ] there is provided the apparatus of example thirteen, further comprising:

the text sample is advertisement text of a plurality of languages.

According to one or more embodiments of the present disclosure, [ example twenty-three ] there is provided a text generation apparatus comprising:

In accordance with one or more embodiments of the present disclosure, [ example twenty-four ] there is provided the apparatus of example twenty-three, further comprising a model decoding module to:

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for generating a text generation model, comprising:

2. The method of claim 1, wherein training an initial speech coder using text samples in a plurality of languages and parallel corpora between the plurality of languages comprises:

3. The method according to claim 1, wherein said performing a paradigm training on said initial text generation model based on said plurality of language text samples, said parallel corpus, and said keyword and target text sample pairs generated by constructing said plurality of language text samples comprises:

4. The method of claim 3, wherein said training said initial text generative model through text samples of different languages to obtain a denoising autoencoder comprises:

5. The method according to claim 3, wherein said performing model training based on said parallel corpus on the basis of said denoising autoencoder to obtain a cross-language autoencoder comprises:

6. The method according to claim 3, wherein the model training of the cross-language self-encoder using the keyword and target text sample pairs to complete the paradigm training of the initial text generation model comprises:

7. The method of claim 1, wherein the performing parameter adjustment on the canonical-trained initial text generation model through the keyword and target text sample pair comprises:

8. The method of claim 1, wherein the construction of the keyword and target text sample pair comprises:

9. The method according to claim 8, wherein after associating the filtered noun phrases with the corresponding text samples respectively to form the keyword and target text sample pair, the constructing process further comprises:

10. The method according to any one of claims 1-9, wherein the text sample is advertisement text in a plurality of languages.

11. A text generation method, comprising:

acquiring a text generation keyword, and generating a text generation model based on any one of the methods in claims 1-10;

12. The method of claim 11, further comprising:

13. A text generation model generation apparatus, comprising:

14. A text generation apparatus, comprising:

a data acquisition module, configured to acquire a text generation keyword and generate a text generation model based on the method of any one of claims 1 to 10;

15. An electronic device, characterized in that the electronic device comprises:

one or more processors;

a memory for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement a text generation model generation method as recited in any one of claims 1-10, or a text generation method as recited in any one of claims 11-12.

16. A computer storage medium having stored thereon a computer program, characterized in that the program, when being executed by a processor, implements a text generation model generation method according to any one of claims 1 to 10, or a text generation method according to any one of claims 11 to 12.