CN117708307B - Method and device for fusing micro-tuning and Adapter of large language model - Google Patents
Method and device for fusing micro-tuning and Adapter of large language model Download PDFInfo
- Publication number
- CN117708307B CN117708307B CN202410170139.4A CN202410170139A CN117708307B CN 117708307 B CN117708307 B CN 117708307B CN 202410170139 A CN202410170139 A CN 202410170139A CN 117708307 B CN117708307 B CN 117708307B
- Authority
- CN
- China
- Prior art keywords
- question
- answer
- dialogue
- adapter
- lora
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 230000006870 function Effects 0.000 claims abstract description 114
- 230000004927 fusion Effects 0.000 claims abstract description 90
- 238000007500 overflow downdraw method Methods 0.000 claims abstract description 18
- 238000012549 training Methods 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 8
- 230000006978 adaptation Effects 0.000 claims description 5
- 230000008014 freezing Effects 0.000 claims description 3
- 238000007710 freezing Methods 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 abstract description 2
- 238000010276 construction Methods 0.000 abstract 1
- 238000013480 data collection Methods 0.000 abstract 1
- 238000003058 natural language processing Methods 0.000 description 8
- 238000011160 research Methods 0.000 description 7
- 238000011161 development Methods 0.000 description 5
- 230000002452 interceptive effect Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000013473 artificial intelligence Methods 0.000 description 4
- 238000004140 cleaning Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000013519 translation Methods 0.000 description 4
- 230000015556 catabolic process Effects 0.000 description 3
- 238000006731 degradation reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000001737 promoting effect Effects 0.000 description 2
- 238000012827 research and development Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000001427 coherent effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 231100000331 toxic Toxicity 0.000 description 1
- 230000002588 toxic effect Effects 0.000 description 1
- 238000009966 trimming Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Machine Translation (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a large language model tuning and Adapter fusion method and device, and relates to the field of deep learning. The method is used for solving the problems of large consumption and poor data quality caused by the fact that the data collection is required to be manually carried out in the construction of the existing multi-mode data set. The method comprises the following steps: collecting a plurality of question-answer data sets and dialogue data sets from a set network platform; performing LoRA-adapter fine tuning on the question-answer data set and the dialogue data set respectively to sequentially obtain a question-answer large language model, a question-answer negative log-likelihood loss function, a dialogue large language model and a dialogue negative log-likelihood loss function; obtaining an ideal loss function, ideal fusion weight and a first ideal parameter of the question-answer data set and the dialogue data set in an ideal state; obtaining optimal parameters of the questioning and answering LoRA-adapter, optimal parameters of the dialogue LoRA-adapter and optimal fusion parameters; and obtaining the general LORA-adapter according to the optimal parameters of the questioning and answering LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters.
Description
Technical Field
The invention relates to the field of deep learning, in particular to a large language model fine tuning and Adapter fusion method and device.
Background
The training of the large language model has important scientific research and application value, can promote the performance of natural language processing tasks, improve the interactive experience of a dialogue system, and promote the universality of scientific research, technical innovation and artificial intelligence development. The large language model can learn rich language knowledge and grammar rules by training a massive corpus, so that the large language model has better performance in natural language processing tasks such as machine translation, text generation, text classification and the like. These models are able to understand and generate more accurate and fluent natural language. Large language models can be used to build intelligent dialog systems that provide more natural, accurate, and personalized replies by performing dialogues with users. The trained model can understand and generate human language, so that the requirements of users can be better met, and the interactive experience of a dialogue system is improved. Training a large language model requires processing massive amounts of data and huge computing resources, which is of great significance in promoting scientific research and technological innovation. In the process of training a large language model, many technical challenges, such as data processing, model design, training algorithms, etc., need to be solved, and the solution of these challenges has a positive driving effect on research and development in the related field.
The conventional scheme adopted by the current large language model training is as follows: and collecting a large amount of instruction fine tuning data, fusing the instruction fine tuning data to construct a large-scale data set, and fine tuning an open-source large language model on the data set. However, it seems impossible to fuse multiple data sets to construct one multi-functional data set, on the one hand, there is a possibility that contradictions may exist between different data sets, and it is difficult to evaluate the quality of data; on the other hand these datasets consist of examples of various specific tasks such as mathematics, coding, role playing, authoring, etc. If these datasets are blended and fine-tuned on this blended dataset, the performance of the large language model may be degraded or even severely degraded.
Disclosure of Invention
The embodiment of the invention provides a large language model tuning and Adapter fusion method and device, which can prevent the problem of performance degradation caused by conflict of different data sets in a semantic space.
The embodiment of the invention provides a large language model tuning and Adapter fusion method, which comprises the following steps:
Collecting a plurality of question-answer data sets and dialogue data sets from a set network platform, and respectively performing LoRA-adapter fine tuning on the question-answer data sets and the dialogue data sets to sequentially obtain a question-answer large language model, a question-answer negative log-likelihood loss function, a dialogue large language model and a dialogue negative log-likelihood loss function;
Obtaining an ideal loss function of the question-answer dataset and the dialogue dataset in an ideal state according to the question-answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and initial fusion weights which are included based on fine tuning of each LoRA-adapter, and obtaining an ideal fusion weight and a first ideal parameter which correspond to the ideal loss function according to the minimum value of the ideal loss function; wherein the first ideal parametric representation is added to all LoRA-adapters of the question-answer large language model and the dialogue large language model, respectively;
according to the ideal loss function, fine tuning is carried out on the question-answer LoRA-adapter corresponding to each question-answer data set and the dialogue LoRA-adapter corresponding to each dialogue data set, so as to respectively obtain the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters;
and obtaining the general LoRA-adapter according to the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters.
Preferably, the performing LoRA-adapter fine tuning on the question-answer dataset sequentially obtains a question-answer large language model and a question-answer negative log likelihood loss function, which specifically includes:
Training the question-answer data set to obtain a question-answer LoRA-adapter, and obtaining the question-answer large language model according to the question-answer LoRA-adapter and the question-answer data set;
obtaining the question-answer negative log-likelihood loss function according to the question-answer large language model and the token of the question-answer large language model;
the question-answer dataset, the question-answer large language model, and the question-answer negative log-likelihood loss function are as follows:
Where Q i represents the ith question-answer dataset, s i,j represents the jth system information of the ith question-answer dataset, Q i,j represents the jth question of the ith question-answer dataset, r i,j represents the jth reply of the ith question-answer dataset, |Q i | represents the length of the question-answer dataset Q i, Represents the question and answer LoRA-adapter trained on the question and answer dataset Q i, |r i,j | represents the length of r i,j, r k represents the kth token generated by the large language model, p θ represents the large language model, θ represents the freezing parameters of the large language model,/>Representing a question-answer negative log-likelihood loss function.
Preferably, the performing LoRA-adapter fine tuning on the dialogue data set sequentially obtains a dialogue large language model and a dialogue negative log likelihood loss function, which specifically includes:
Training the dialogue data set to obtain a dialogue LoRA-adapter, and obtaining a dialogue large language model according to the dialogue LoRA-adapter and the dialogue data set;
obtaining the dialogue negative log likelihood loss function according to the dialogue large language model and the token of the dialogue large language model;
The dialogue dataset, dialogue large language model, and the dialogue negative log likelihood loss function are as follows:
Wherein C i denotes the ith session dataset, A j-th query representing an i-th session dataset in a T-th round,The j-th reply representing the i-th dialog data set in the T-th round, |c i | represents the length of dialog data set C i,/>Representing a dialog LoRA-adapter trained on dialog dataset C i, Q j representing all tags belonging to a user query, R j representing a target tag,/>Representing the number of j-th data containing tokens in the dialog data set C i,/>Representing a dialogue negative log-likelihood loss function, p θ representing a large language model, θ representing a freeze parameter of the large language model.
Preferably, the ideal loss function is as follows:
The minimum of the ideal loss function is as follows:
Wherein L represents an ideal loss function, Representing fine-tuning the acquisition/>, on the question-answer dataset Q i Initial fusion weights of,/>Representing fine-tuning of acquisition/>, on dialog dataset C i A * denotes all the first ideal parameters, ω * denotes all the ideal fusion weights, a denotes the first ideal parameters, ω denotes the ideal fusion weights.
Preferably, the first ideal parameter is as follows:
the ideal fusion weights are as follows:
Wherein A represents a first ideal parameter, Representing the first ideal parameters obtained by fine tuning on the question-answer dataset Q M,Represents the first ideal parameter obtained by fine tuning on the dialogue dataset C N, M represents the number of question-answer datasets, N represents the number of dialogue datasets, ω represents the ideal fusion weight,/>Representation/>Is/are the ideal fusion weights of (1)Representation/>Is used for the optimal fusion weight of the (c).
Preferably, the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter, and the optimal fusion parameters are as follows:
Wherein, Optimal parameters representing question-answer LoRA-adapter,/>The best parameters for dialog LoRA-adapter, ω ** the best fusion parameters,/>Representing a question-answer negative log-likelihood loss function,/>Representing a dialogue negative log likelihood loss function,/>Representing initial fusion weights obtained by fine-tuning on the question-answer dataset Q i,/>Representing initial fusion weights obtained by fine-tuning on dialog dataset C i,/>Representing the resulting dialog LoRA-adapter trained on dialog data set C i.
The embodiment of the invention provides a large language model tuning and Adapter fusion device, which comprises the following components:
a first obtaining unit for collecting a plurality of question-answer data sets and dialogue data sets from the set network platform; performing LoRA-adapter fine tuning on the question-answer dataset and the dialogue dataset respectively to sequentially obtain a question-answer large language model, a question-answer negative log-likelihood loss function, a dialogue large language model and a dialogue negative log-likelihood loss function;
The second obtaining unit is used for obtaining an ideal loss function of the question-answer data set and the dialogue data set in an ideal state according to the question-answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and initial fusion weights which are included on the basis of fine adjustment of each LoRA-adapter, and obtaining ideal fusion weights and first ideal parameters which correspond to the ideal loss function according to the minimum value of the ideal loss function; wherein the first ideal parametric representation is added to all LoRA-adapters of the question-answer large language model and the dialogue large language model, respectively;
The third obtaining unit is configured to fine tune the question answer LoRA-adapter corresponding to each question-answer dataset and the dialogue LoRA-adapter corresponding to each dialogue dataset according to the ideal loss function, so as to obtain an optimal parameter of the question answer LoRA-adapter, an optimal parameter of the dialogue LoRA-adapter and an optimal fusion parameter respectively;
And a fourth obtaining unit, configured to obtain a general LoRA-adapter according to the optimal parameter of the question-answer LoRA-adapter, the optimal parameter of the dialogue LoRA-adapter, and the optimal fusion parameter.
The embodiment of the invention provides a computer device, which comprises a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, enables the processor to execute the large language model refinement and Adapter fusion method described in any one of the above.
An embodiment of the present invention provides a computer readable storage medium storing a computer program, where the computer program when executed by a processor causes the processor to execute the large language model refinement and Adapter fusion method described in any one of the above.
The embodiment of the invention provides a large language model tuning and Adapter fusion method and device, wherein the method comprises the following steps: collecting a plurality of question-answer data sets and dialogue data sets from a set network platform; performing LoRA-adapter fine tuning on the question-answer dataset and the dialogue dataset respectively to sequentially obtain a question-answer large language model, a question-answer negative log-likelihood loss function, a dialogue large language model and a dialogue negative log-likelihood loss function; obtaining an ideal loss function of a question-answer data set and a dialogue data set in an ideal state according to the question-answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and initial fusion weights which are included based on fine adjustment of each LoRA-adapter, and obtaining an ideal fusion weight and a first ideal parameter which correspond to the ideal loss function according to the minimum value of the ideal loss function; wherein the first ideal parametric representation is added to all LoRA-adapters of the question-answer large language model and the dialogue large language model, respectively; according to the ideal loss function, fine tuning is carried out on the question-answer LoRA-adapter corresponding to each question-answer data set and the dialogue LoRA-adapter corresponding to each dialogue data set, so as to respectively obtain the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters; and obtaining the general LoRA-adapter according to the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters. According to the method, a plurality of instruction fine-tuning data sets are constructed, consumption of a GPU (English: graphic Process Unit, chinese: graphic processor) is saved by utilizing a quantization technology of QLoRA, a large language model training mode which saves computing resource cost and is high in quality is provided, and meanwhile, a multi-LoRA-adapter fusion mode based on Grid-Search (Chinese: parameter tuning means) optimization is designed to fuse the trained LoRA-adapters. By fusing LoRA-adapters, semantic space conflict caused by data set fusion can be effectively avoided, and generalization performance of a large language model on a plurality of tasks is improved. The problem of performance degradation caused by conflict of different data sets in semantic space in the prior art is solved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a large language model tuning and Adapter fusion method provided by an embodiment of the invention;
Fig. 2 is a schematic structural diagram of a large language model tuning and Adapter fusion device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Training a large language model not only aims at improving the performance of natural language processing tasks and improving the interactive experience of a dialogue system, but also has richer value and significance in scientific research and application fields.
Firstly, a large language model can learn rich language knowledge and grammar rules by training a massive corpus. The models can understand and generate more accurate and smoother natural language, and provide better performance for natural language processing tasks such as machine translation, text generation, text classification and the like. In the machine translation task, the large language model can more accurately understand the meaning of the source language and generate a more natural target language translation result. In a text generation task, a model can generate text content that is more logical and coherent. In the text classification task, the model can judge the type of the text more accurately, and the classification accuracy is improved.
Second, a large language model can be used to build an intelligent dialog system that provides a more natural, accurate and personalized reply by having a dialog with the user. This capability is very useful for chat robots, intelligent customer service, etc. in everyday life. The trained model can understand and generate human language, so that the requirements of users can be better met, and the interactive experience of a dialogue system is improved. The dialogue system can generate personalized replies through the model, so that the user can feel the same interaction experience as human dialogue, and the satisfaction degree of the user is enhanced.
In addition, training a large language model requires processing massive amounts of data and huge computing resources, which is of great significance in promoting scientific research and technological innovation. In training large language models, many technical challenges need to be addressed, such as data processing, model design, training algorithms, etc. The resolution of these challenges not only can drive the development of language models, but also helps in research and development of related fields. For example, by improving and optimizing the model, the efficiency and performance of the model can be improved, and technical support is provided for development and application of other natural language processing tasks.
Finally, training a large language model can provide intelligent natural language processing services and promote the development of the universality of artificial intelligence technology. These models can be applied to various fields such as education, medical treatment, finance, and the like. In the education field, the model can be used for assisting learning, intelligent answering and the like, and personalized learning resources and communication platforms are provided. In the medical field, the model can be used for assisting doctor diagnosis, intelligent medical record and the like, and the quality and efficiency of medical service are improved. In the financial field, the model can be used for intelligent customer service, risk management and the like, and more personalized and efficient financial services are provided.
In summary, training a large language model has important scientific research and application values, not only can improve the performance of natural language processing tasks and improve the interactive experience of a dialogue system, but also can promote the universality of scientific research, technical innovation and artificial intelligence development. By training a large language model, an intelligent natural language processing technology can be applied to various fields, and better intelligent service and solution can be provided for society.
Because of the conventional scheme adopted by the current large language model training, it seems impossible to combine multiple data sets to construct a multifunctional data set, and there is a possibility that contradictions may exist between different data sets, and it is difficult to evaluate the quality of data. Based on the above, the embodiment of the invention provides an efficient training method for constructing a high-quality and strong-capacity large language model. Cleaning and arranging a plurality of open source data on a Huggingface platform to obtain a plurality of different knowledge question-answer data sets and dialogue data sets, then independently training one LoRA-adapter on each data set by QLoRA (Chinese: low-rank adaptation), and finally dynamically optimizing the fusion weights of the LoRA-adapters by using Grid-Search.
FIG. 1 is a schematic flow chart of a large language model tuning and Adapter fusion method provided by an embodiment of the invention; as shown in fig. 1, the method comprises the steps of:
Step 101, collecting a plurality of question and answer data sets and dialogue data sets from a set network platform; performing LoRA-adapter fine tuning on the question-answer dataset and the dialogue dataset respectively to sequentially obtain a question-answer large language model, a question-answer negative log-likelihood loss function, a dialogue large language model and a dialogue negative log-likelihood loss function;
102, obtaining ideal loss functions of a question-answer data set and a dialogue data set in an ideal state according to the question-answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and initial fusion weights which are included on the basis of fine adjustment of each LoRA-adapter, and obtaining ideal fusion weights and first ideal parameters which correspond to the ideal loss functions according to the minimum values of the ideal loss functions; wherein the first ideal parametric representation is added to all LoRA-adapters of the question-answer large language model and the dialogue large language model, respectively;
Step 103, fine tuning the question-answer LoRA-adapter corresponding to each question-answer dataset and the dialogue LoRA-adapter corresponding to each dialogue dataset according to the ideal loss function to respectively obtain the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters;
And 104, obtaining the general LoRA-adapter according to the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters.
It should be noted that, the embodiment of the invention provides a large language model tuning and Adapter fusion method, the execution main body of which is a processor.
In step 101, a plurality of question-answer data sets and dialogue data sets are collected from a setting network platform, where the setting network platform may be a Huggingface community, and in the embodiment of the present invention, the setting network platform is not specifically limited.
Specifically, after collecting a plurality of data sets from a set network platform, cleaning the plurality of data sets is required, and finally obtaining a plurality of question-answer data sets and a plurality of dialogue data sets after cleaning, in the embodiment of the present invention, the cleaning rule of the data sets is as follows:
1) Deletion and ChatGPT (english: CHAT GENERATIVE PRE-trained Transformer) -3.5-Turbo, only the dialogue instance with GPT-4 is reserved; 2) Deleting the GPT-4 refused answer or directly interpreted dialog; 3) The deleted answer is GPT-4 null or GPT-4 missed answer dialogue; 4) Deleting dialogs containing toxic or illegal information; 5) Deleting a dialogue containing a OpenAI or ChatGPT typeface or replacing a dialogue containing a OpenAI or ChatGPT typeface with information that ensures that the model has the correct identity; 6) Deleting user questions with similarity to the reference questions being greater than 85%; 7) Lengthy dialog instances are separated into dialogs that match the model maximum context length.
Through the above situation rules, a plurality of question-answer data sets and a plurality of dialogue data sets required by the embodiment of the invention can be finally obtained.
In an embodiment of the invention, the question-answer dataset may be represented as { Q 1,Q2,…,QM }, and the dialogue dataset may be represented as { C 1,C2,…,CN }.
Specifically, the question-answer dataset may be represented by formula (1):
where s represents system information SYSTEM MESSAGE, Q represents a question query of the user, r represents a response of the artificial intelligence, Q i represents an i-th question-answer dataset, s i,j represents a j-th system information of the i-th question-answer dataset, Q i,j represents a j-th question of the i-th question-answer dataset, r i,j represents a j-th response of the i-th question-answer dataset, and |q i | represents a length of the question-answer dataset Q i.
In the embodiment of the invention, the obtained question-answer data set is given from a specific example when being subjected to LoRA-adapter fine tuningThe large language model should learn to generate the corresponding reply r i,j, the system message s i,j and the query q i,j. The process can obtain a question and answer language model, which is as follows:
Wherein, Representing a question and answer LoRA-adapter trained on a question and answer dataset Q i, p θ representing a large language model, |r i,j | representing the length of r i,j, r k representing the kth token generated by the large language model, θ representing the freeze parameters of the large language model, and r <k representing all r with a small scale less than k.
Further, a question-answer negative log-likelihood loss function is obtained according to the question-answer large language model and the token of the question-answer large language model.
Wherein the question-answer negative log-likelihood loss function is as follows:
Wherein, Representing a question-answer negative log-likelihood loss function, s i,j representing the j-th system information of the i-th question-answer data set, q i,j representing the j-th question of the i-th question-answer data set, and r i,j representing the j-th answer of the i-th question-answer data set.
Accordingly, the dialog data set may be represented by formula (4):
Wherein the session dataset comprises a plurality of session instances having T rounds, C i represents the ith session dataset, Jth query representing ith session dataset in the T-th round,/>Represents the jth reply to the ith session dataset in the T-th round, |c i | represents the length of session dataset C i.
In embodiments of the present invention, when a dialog dataset is subjected to LoRA-adapter fine-tuning, the large language model will learn the dialog history and queries before a given round TPredicting per reply/>This process results in a large dialog language model, which is shown below:
Wherein, Representing a dialog LoRA-adapter trained on dialog dataset C i, Q j representing all tags belonging to a user query, R j representing a target tag,/>Representing the number of j-th data containing tokens in the session dataset C i.
Further, a dialogue negative log likelihood loss function is obtained according to the dialogue large language model and the token of the dialogue large language model. Wherein the dialogue negative log-likelihood loss function is as follows:
Wherein, Representing a dialogue negative log likelihood loss function,/>Representing the number of jth data containing tokens in the dialog data set C i, p θ representing a large language model, θ representing the freeze parameters of the large language model,/>Representing the resulting dialog LoRA-adapter trained on dialog data set C i.
It should be noted that, in the embodiment of the present invention, for LoRA-adapter fusion, the loss function of each fine-tuning LoRA-adapter is given a trainable weight, and the loss functions of all LoRA-adapters with fusion weights are fine-tuned, where the fusion weights can be expressed as:
In step 102, an ideal loss function of the question-answer dataset and the dialogue dataset in an ideal state can be obtained according to the question-answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and the initial fusion weights included based on each LoRA-adapter fine tuning, wherein the ideal loss function is as follows:
Wherein L represents an ideal loss function, Representing a question-answer negative log-likelihood loss function,/>Representing a dialogue negative log likelihood loss function,/>Representing fine-tuning the acquisition/>, on the question-answer dataset Q i Initial fusion weights of,/>Representing fine-tuning of acquisition/>, on dialog dataset C i Is used to determine the initial fusion weights of (a).
Further, an ideal fusion weight and a first ideal parameter corresponding to the ideal loss function are obtained according to the minimum value of the ideal loss function, and the minimum value of the ideal loss function is expressed by the following formula:
Where A * represents all ideal LORA-adaper, ω * represents all ideal fusion weights, argmin represents the values of the corresponding parameters A and ω when the latter formula takes the minimum value.
In an embodiment of the present invention, all ideal LORA-adaper may also be referred to as all first ideal parameters, which represent all LoRA-adapters added to the quiz language model and the dialog language model, respectively, wherein the first ideal parameters are represented by the following formulas:
Wherein A represents a first ideal parameter, Representing the first ideal parameters obtained by fine tuning on the question-answer dataset Q M,Representing the first ideal parameter obtained by trimming on the dialogue dataset C N, M representing the number of question-answer datasets and N representing the number of dialogue datasets.
Specifically, ω represents the ideal fusion weight for all LoRA-adapters, i.e., for question and answer LoRA-adapters and dialogue LoRA-adapters, which can be expressed by the following formula:
Where ω represents the ideal fusion weight, Representation/>Is/are the ideal fusion weights of (1)Representation/>Is used for the optimal fusion weight of the (c).
In step 103, fine tuning is performed on the question and answer LoRA-adapter corresponding to each question and answer data set and the dialogue LoRA-adapter corresponding to each dialogue data set according to the ideal loss function, so as to obtain the optimal parameters of the question and answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters respectively.
In practical applications, the first ideal parameters and ideal fusion weights are trimmed sequentially for efficiency and simplicity. In the first stage, each LoRA-adapter (all first ideal parameters) on each question-answer dataset or dialogue dataset is fine-tuned, namely, for formula (8), the formula (8) is split into two formulas shown below, and then the optimal parameters of the question-answer LoRA-adapter and the optimal parameters of the dialogue LoRA-adapter can be obtained, specifically as follows:
Wherein, Optimal parameters representing question-answer LoRA-adapter,/>Optimal parameters representing dialog LoRA-adapter,/>Representing a dialog LoRA-adapter,/>, trained on dialog dataset C i Representing a question LoRA-adapter, i.e./>, trained on a question dataset Q i Representing a question-answer negative log-likelihood loss function,/>Representing a dialogue negative log-likelihood loss function.
Further, only fine tuning the ideal fusion weight in the second stage, and freezing the basic large language model and the first ideal parameters to obtain the optimal fusion parameters, wherein the optimal fusion parameters are specifically as follows:
Wherein omega ** represents the optimal fusion parameters, Representing fine-tuning the acquisition/>, on the question-answer dataset Q i Initial fusion weights of,/>Representing fine-tuning of acquisition/>, on dialog dataset C i Initial fusion weights of,/>Representing a question-answer negative log-likelihood loss function,/>Representing the dialogue negative log-likelihood loss function, ω represents the ideal fusion weight.
It should be noted that in practical applications, when the number of question-answer datasets and dialogue datasets is small, some simple and fast algorithms may be used to optimize the ideal fusion weights.
And 104, obtaining the general LoRA-adapter according to the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters.
In summary, the embodiment of the invention provides a large language model refinement and adaptation fusion method and device, and the method can effectively avoid semantic space conflict caused by data set fusion by fusing LoRA-adaptation, and simultaneously promote generalization performance of a large language model on a plurality of tasks. The problem of performance degradation caused by conflict of different data sets in semantic space in the prior art is solved.
Based on the same inventive concept, the embodiment of the invention provides a large language model tuning and Adapter fusion device, and because the principle of the device for solving the technical problem is similar to that of a large language model tuning and Adapter fusion method, the implementation of the device can be referred to the implementation of the method, and the repetition is omitted.
As shown in fig. 2, the apparatus mainly includes a first obtaining unit 201, a second obtaining unit 202, a third obtaining unit 203, and a fourth obtaining unit 204.
A first obtaining unit 201, configured to collect a plurality of question-answer data sets and dialogue data sets from a set network platform, and perform LoRA-adapter fine tuning on the question-answer data sets and the dialogue data sets respectively, so as to obtain a question-answer large language model, a question-answer negative log likelihood loss function, a dialogue large language model and a dialogue negative log likelihood loss function in sequence;
A second obtaining unit 202, configured to obtain an ideal loss function of the question-answer dataset and the dialogue dataset in an ideal state according to the question-answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function, and initial fusion weights included based on fine tuning of each LoRA-adapter, and obtain an ideal fusion weight and a first ideal parameter corresponding to the ideal loss function according to a minimum value of the ideal loss function; wherein the first ideal parametric representation is added to all LoRA-adapters of the question-answer large language model and the dialogue large language model, respectively;
A third obtaining unit 203, configured to fine tune a question-answer LoRA-adapter corresponding to each question-answer dataset and a dialogue LoRA-adapter corresponding to each dialogue dataset according to the ideal loss function, to obtain an optimal parameter of the question-answer LoRA-adapter, an optimal parameter of the dialogue LoRA-adapter, and an optimal fusion parameter respectively;
A fourth obtaining unit 204, configured to obtain a general LoRA-adapter according to the optimal parameter of the question-answer LoRA-adapter, the optimal parameter of the dialogue LoRA-adapter, and the optimal fusion parameter.
It should be understood that the above large language model tuning and Adapter fusion device includes units that are logically divided only according to functions implemented by the device, and in practical application, the units may be overlapped or split. The functions implemented by the large language model tuning and Adapter fusion device provided in this embodiment are in one-to-one correspondence with the large language model tuning and Adapter fusion method provided in the foregoing embodiment, and the more detailed process flow implemented by the device is described in detail in the foregoing method embodiment one, which is not described in detail herein.
Another embodiment of the present invention also provides a computer apparatus, including: a processor and a memory; the memory is used for storing computer program codes, and the computer program codes comprise computer instructions; when the processor executes the computer instructions, the electronic device executes the steps of the large language model refinement and Adapter fusion method in the method flow shown in the method embodiment.
Another embodiment of the present invention further provides a computer readable storage medium, where computer instructions are stored, where the computer instructions, when executed on a computer device, cause the computer device to execute the steps of the large language model refinement and Adapter fusion method in the method flow shown in the foregoing method embodiment.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (8)
1. A large language model micro-tuning and Adapter fusion method is characterized by comprising the following steps:
Collecting a plurality of question-answer data sets and dialogue data sets from a set network platform, and respectively performing LoRA-adapter fine tuning on the question-answer data sets and the dialogue data sets to sequentially obtain a question-answer large language model, a question-answer negative log-likelihood loss function, a dialogue large language model and a dialogue negative log-likelihood loss function;
Obtaining an ideal loss function of the question-answer dataset and the dialogue dataset in an ideal state according to the question-answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and initial fusion weights which are included based on fine tuning of each LoRA-adapter, and obtaining an ideal fusion weight and a first ideal parameter which correspond to the ideal loss function according to the minimum value of the ideal loss function; wherein the first ideal parametric representation is added to all LoRA-adapters of the question-answer large language model and the dialogue large language model, respectively;
according to the ideal loss function, fine tuning is carried out on the question-answer LoRA-adapter corresponding to each question-answer data set and the dialogue LoRA-adapter corresponding to each dialogue data set, so as to respectively obtain the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters;
obtaining a general LoRA-adapter according to the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters;
The ideal loss function, the optimal parameters of the question LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter, and the optimal fusion parameters are as follows:
Wherein L represents an ideal loss function, Representing fine-tuning the acquisition/>, on the question-answer dataset Q i Is used to determine the initial fusion weights of (1),Representing fine-tuning of acquisition/>, on dialog dataset C i Initial fusion weights of,/>Optimal parameters representing question-answer LoRA-adapter,/>The best parameters for dialog LoRA-adapter, ω ** the best fusion parameters,/>Representing a question-answer negative log-likelihood loss function,/>Representing a dialogue negative log likelihood loss function,/>Representing a question LoRA-adapter, i.e./>, trained on a question dataset Q i Representing the resulting dialog LoRA-adapter trained on dialog data set C i.
2. The method of claim 1, wherein said performing LoRA-adapter fine-tuning on said question-answer dataset sequentially results in a question-answer large language model, a question-answer negative log-likelihood loss function, comprising:
Training the question-answer data set to obtain a question-answer LoRA-adapter, and obtaining the question-answer large language model according to the question-answer LoRA-adapter and the question-answer data set;
obtaining the question-answer negative log-likelihood loss function according to the question-answer large language model and the token of the question-answer large language model;
the question-answer dataset, the question-answer large language model, and the question-answer negative log-likelihood loss function are as follows:
where Q i represents the ith question-answer dataset, s i,j represents the jth system information of the ith question-answer dataset, Q i,j represents the jth question of the ith question-answer dataset, r i,j represents the jth reply of the ith question-answer dataset, |Q i | represents the length of the question-answer dataset Q i, Representing the question and answer LoRA-adapter trained on the question and answer dataset Q i, p θ representing a large language model, |r i,j | representing the length of r i,j, r k representing the kth token generated by the large language model, θ representing the freezing parameters of the large language model,Representing a question-answer negative log-likelihood loss function.
3. The method of claim 1, wherein performing LoRA-adapter fine-tuning on the dialogue dataset sequentially obtains a dialogue large language model and a dialogue negative log likelihood loss function, specifically comprising:
Training the dialogue data set to obtain a dialogue LoRA-adapter, and obtaining a dialogue large language model according to the dialogue LoRA-adapter and the dialogue data set;
obtaining the dialogue negative log likelihood loss function according to the dialogue large language model and the token of the dialogue large language model;
The dialogue dataset, dialogue large language model, and the dialogue negative log likelihood loss function are as follows:
Wherein C i denotes the ith session dataset, Jth query representing ith session dataset in the T-th round,/>The j-th reply representing the i-th dialog data set in the T-th round, |c i | represents the length of dialog data set C i,/>Representing a dialog LoRA-adapter trained on dialog dataset C i, Q j representing all tags belonging to a user query, R j representing a target tag,/>Representing the number of j-th data containing tokens in the dialog data set C i,/>Representing a dialogue negative log-likelihood loss function, p θ representing a large language model, θ representing a freeze parameter of the large language model.
4. The method of claim 1, wherein the minimum value of the ideal loss function is as follows:
Wherein, Representing fine-tuning the acquisition/>, on the question-answer dataset Q i Initial fusion weights of,/>Representing fine-tuning of acquisition/>, on dialog dataset C i A * denotes all the first ideal parameters, ω * denotes all the ideal fusion weights, a denotes the first ideal parameters, ω denotes the ideal fusion weights.
5. The method of claim 1, wherein the first desired parameter is as follows:
the ideal fusion weights are as follows:
Wherein A represents a first ideal parameter, Representing the first ideal parameter obtained by fine tuning on the question-answer dataset Q M,/>Represents the first ideal parameter obtained by fine tuning on the dialogue dataset C N, M represents the number of question-answer datasets, N represents the number of dialogue datasets, ω represents the ideal fusion weight,/>Representation/>Is/are the ideal fusion weights of (1)Representation/>Is used for the optimal fusion weight of the (c).
6. A large language model refinement and adaptation fusion apparatus, comprising:
The first obtaining unit is used for collecting a plurality of question-answer data sets and dialogue data sets from a set network platform, respectively performing LoRA-adapter fine tuning on the question-answer data sets and the dialogue data sets, and sequentially obtaining a question-answer large language model, a question-answer negative log likelihood loss function, a dialogue large language model and a dialogue negative log likelihood loss function;
the second obtaining unit is used for obtaining an ideal loss function of the question-answer dataset and the dialogue dataset in an ideal state according to the question-answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and initial fusion weights which are included based on fine adjustment of each LoRA-adapter, and obtaining an ideal fusion weight and a first ideal parameter which correspond to the ideal loss function according to the minimum value of the ideal loss function; wherein the first ideal parametric representation is added to all LoRA-adapters of the question-answer large language model and the dialogue large language model, respectively;
The third obtaining unit is configured to fine tune the question answer LoRA-adapter corresponding to each question-answer dataset and the dialogue LoRA-adapter corresponding to each dialogue dataset according to the ideal loss function, so as to obtain an optimal parameter of the question answer LoRA-adapter, an optimal parameter of the dialogue LoRA-adapter and an optimal fusion parameter respectively;
A fourth obtaining unit, configured to obtain a general LoRA-adapter according to the optimal parameter of the question-answer LoRA-adapter, the optimal parameter of the dialogue LoRA-adapter, and the optimal fusion parameter;
The ideal loss function, the optimal parameters of the question LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter, and the optimal fusion parameters are as follows:
Wherein L represents an ideal loss function, Representing fine-tuning the acquisition/>, on the question-answer dataset Q i Is used to determine the initial fusion weights of (1),Representing fine-tuning of acquisition/>, on dialog dataset C i Initial fusion weights of,/>Optimal parameters representing question-answer LoRA-adapter,/>The best parameters for dialog LoRA-adapter, ω ** the best fusion parameters,/>Representing a question-answer negative log-likelihood loss function,/>Representing a dialogue negative log likelihood loss function,/>Representing a question LoRA-adapter, i.e./>, trained on a question dataset Q i Representing the resulting dialog LoRA-adapter trained on dialog data set C i.
7. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the large language model refinement and adaptation fusion method of any one of claims 1-5.
8. A computer readable storage medium, storing a computer program which, when executed by a processor, causes the processor to perform the large language model refinement and adaptation fusion method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410170139.4A CN117708307B (en) | 2024-02-06 | 2024-02-06 | Method and device for fusing micro-tuning and Adapter of large language model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410170139.4A CN117708307B (en) | 2024-02-06 | 2024-02-06 | Method and device for fusing micro-tuning and Adapter of large language model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117708307A CN117708307A (en) | 2024-03-15 |
| CN117708307B true CN117708307B (en) | 2024-05-14 |
Family
ID=90144771
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410170139.4A Active CN117708307B (en) | 2024-02-06 | 2024-02-06 | Method and device for fusing micro-tuning and Adapter of large language model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117708307B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115712709A (en) * | 2022-11-18 | 2023-02-24 | 哈尔滨工业大学 | Multi-modal dialog question-answer generation method based on multi-relationship graph model |
| CN116975241A (en) * | 2023-09-20 | 2023-10-31 | 广东技术师范大学 | Liver cancer auxiliary diagnosis and question-answering method, system and medium based on large language model |
| CN117033608A (en) * | 2023-09-28 | 2023-11-10 | 中国电子科技集团公司第十研究所 | Knowledge graph generation type question-answering method and system based on large language model |
| CN117171326A (en) * | 2023-09-20 | 2023-12-05 | 宜宾电子科技大学研究院 | Rapid construction method of financial question-answering algorithm and life cycle management platform |
| WO2023235346A1 (en) * | 2022-06-03 | 2023-12-07 | Google Llc | Prompting machine-learned models using chains of thought |
| CN117216234A (en) * | 2023-08-18 | 2023-12-12 | 腾讯科技（深圳）有限公司 | Artificial intelligence-based speaking operation rewriting method, device, equipment and storage medium |
| CN117217289A (en) * | 2023-10-09 | 2023-12-12 | 北银金融科技有限责任公司 | Banking industry large language model training method |
| CN117290492A (en) * | 2023-11-27 | 2023-12-26 | 深圳市灵智数字科技有限公司 | Knowledge base question-answering method and device, electronic equipment and storage medium |
| CN117371527A (en) * | 2023-11-01 | 2024-01-09 | 中国科学院计算技术研究所 | Multi-mode entity linking method and system based on large model |
| CN117453925A (en) * | 2023-10-24 | 2024-01-26 | 腾讯科技（深圳）有限公司 | Knowledge migration method, apparatus, device, readable storage medium and program product |
| CN117455009A (en) * | 2023-10-27 | 2024-01-26 | 腾讯科技（深圳）有限公司 | Federal learning method, federal prediction method, apparatus, device, and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110188182B (en) * | 2019-05-31 | 2023-10-27 | 中国科学院深圳先进技术研究院 | Model training method, dialogue generating method, device, equipment and medium |
-
2024
- 2024-02-06 CN CN202410170139.4A patent/CN117708307B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023235346A1 (en) * | 2022-06-03 | 2023-12-07 | Google Llc | Prompting machine-learned models using chains of thought |
| CN115712709A (en) * | 2022-11-18 | 2023-02-24 | 哈尔滨工业大学 | Multi-modal dialog question-answer generation method based on multi-relationship graph model |
| CN117216234A (en) * | 2023-08-18 | 2023-12-12 | 腾讯科技（深圳）有限公司 | Artificial intelligence-based speaking operation rewriting method, device, equipment and storage medium |
| CN116975241A (en) * | 2023-09-20 | 2023-10-31 | 广东技术师范大学 | Liver cancer auxiliary diagnosis and question-answering method, system and medium based on large language model |
| CN117171326A (en) * | 2023-09-20 | 2023-12-05 | 宜宾电子科技大学研究院 | Rapid construction method of financial question-answering algorithm and life cycle management platform |
| CN117033608A (en) * | 2023-09-28 | 2023-11-10 | 中国电子科技集团公司第十研究所 | Knowledge graph generation type question-answering method and system based on large language model |
| CN117217289A (en) * | 2023-10-09 | 2023-12-12 | 北银金融科技有限责任公司 | Banking industry large language model training method |
| CN117453925A (en) * | 2023-10-24 | 2024-01-26 | 腾讯科技（深圳）有限公司 | Knowledge migration method, apparatus, device, readable storage medium and program product |
| CN117455009A (en) * | 2023-10-27 | 2024-01-26 | 腾讯科技（深圳）有限公司 | Federal learning method, federal prediction method, apparatus, device, and storage medium |
| CN117371527A (en) * | 2023-11-01 | 2024-01-09 | 中国科学院计算技术研究所 | Multi-mode entity linking method and system based on large model |
| CN117290492A (en) * | 2023-11-27 | 2023-12-26 | 深圳市灵智数字科技有限公司 | Knowledge base question-answering method and device, electronic equipment and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| Zoie Zhao.More human than human: LLM-generated narratives outperform human-LLM interleaved narratives.C&C '23: Proceedings of the 15th Conference on Creativity and Cognition.2023,全文. * |
| 基于BiLSTM模型的定义抽取方法;阳萍;谢志鹏;;计算机工程;20200331;第46卷(第03期);全文 * |
| 基于深度学习的个性化聊天机器人研究;王乾铭;李吟;;计算机技术与发展;20200430;第30卷(第04期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117708307A (en) | 2024-03-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10740370B2 (en) | Dialog agent for conducting task-oriented computer-based communications | |
| US20200301954A1 (en) | Reply information obtaining method and apparatus | |
| CN111125309A (en) | Natural language processing method and device, computing equipment and storage medium | |
| KR102259390B1 (en) | System and method for ensemble question-answering | |
| CN107741976B (en) | Intelligent response method, device, medium and electronic equipment | |
| CN111897941A (en) | Dialog generation method, network training method, device, storage medium and equipment | |
| CN112214593A (en) | Question and answer processing method and device, electronic equipment and storage medium | |
| CN109543165B (en) | Text generation method and device based on circular convolution attention model | |
| Sheikh et al. | Generative model chatbot for human resource using deep learning | |
| WO2018195459A1 (en) | Processing sequential data using recurrent neural networks | |
| CN114281957A (en) | Natural language data query method and device, electronic equipment and storage medium | |
| CN117077792B (en) | Knowledge graph-based method and device for generating prompt data | |
| CN113761220A (en) | Information acquisition method, device, equipment and storage medium | |
| CN113761156A (en) | Data processing method, device and medium for man-machine interaction conversation and electronic equipment | |
| CN117235215A (en) | Large model and knowledge graph based dialogue generation method, system and medium | |
| CN113987151A (en) | Intelligent customer service reply method and device | |
| CN112307166B (en) | Intelligent question-answering method and device, storage medium and computer equipment | |
| CN110598127B (en) | Group recommendation method and device | |
| CN117708307B (en) | Method and device for fusing micro-tuning and Adapter of large language model | |
| CN116910217A (en) | Natural language question-answering method, device and medium based on small language model cluster | |
| CN111611378A (en) | Behavior training dialogue control method, behavior training dialogue control system, storage medium, program, and terminal | |
| CN108959467B (en) | Method for calculating correlation degree of question sentences and answer sentences based on reinforcement learning | |
| CN116842143A (en) | Dialog simulation method and device based on artificial intelligence, electronic equipment and medium | |
| CN113051388A (en) | Intelligent question and answer method and device, electronic equipment and storage medium | |
| CN112818108B (en) | Text semantic misinterpretation chat robot based on shape and near words and data processing method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |