CN114417889A - Task type dialogue system and method based on seq2seq framework - Google Patents

Task type dialogue system and method based on seq2seq framework Download PDF

Info

Publication number
CN114417889A
CN114417889A CN202210043220.7A CN202210043220A CN114417889A CN 114417889 A CN114417889 A CN 114417889A CN 202210043220 A CN202210043220 A CN 202210043220A CN 114417889 A CN114417889 A CN 114417889A
Authority
CN
China
Prior art keywords
module
task
answer
input
intention
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210043220.7A
Other languages
Chinese (zh)
Inventor
冯卫森
冯落落
李沛
李晓瑜
高明
王建华
尹青山
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong New Generation Information Industry Technology Research Institute Co Ltd
Original Assignee
Shandong New Generation Information Industry Technology Research Institute Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong New Generation Information Industry Technology Research Institute Co Ltd filed Critical Shandong New Generation Information Industry Technology Research Institute Co Ltd
Priority to CN202210043220.7A priority Critical patent/CN114417889A/en
Publication of CN114417889A publication Critical patent/CN114417889A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G06F40/35—Discourse or dialogue representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a task-based dialogue system and method based on a seq2seq architecture, which comprises four parts, namely an intention and slot filling network module, a database operation module, a trust tracking module and an optimization and generation network module; the method comprises the following steps: the bert output of the classification task and the vector of the input label are mapped to the same vector space, and the groove value extraction task is trained through a CRF layer; the database operation module is mainly used for inquiring other slot values according to the identified slot values, trusts a tracking layer, takes the output of the intention and slot value network module and the database operation module as input, takes the output of the layer as input, takes the input of intention and generation network, optimizes and generates the network module to select the answer An with the highest score as the answer of the current round of conversation, and carries out subsequent prediction on the answer. The invention combines the task type dialogue system and the chatting type dialogue system, and combines the module of the task type dialogue system inside, thereby ensuring the function of context and carrying out integral optimization.

Description

Task type dialogue system and method based on seq2seq framework
Technical Field
The invention relates to a task type dialogue system and method based on a seq2seq framework, belonging to the technical field of intelligent question answering.
Background
A traditional task type dialogue system adopts a pipeline structure. Due to the fact that the structure adopts a modular mode, errors of the upstream task parameters are conducted to the downstream. The chatting dialogue system adopts a seq2seq structure and produces corresponding answers through an Encoder module and a Decoder module. But the chatty type dialogue system does not consider the role of context, and the generated NLG answer has uncontrollable property.
Disclosure of Invention
The invention aims to provide a task type dialogue system and method based on a seq2seq framework, which not only consider the context influence and avoid generating uncertain answers, but also can carry out overall optimization.
In order to achieve the purpose, the invention is realized by the following technical scheme:
a task-based dialogue system based on a seq2seq architecture comprises four parts, namely an intention and slot filling network module, a database operation module, a trust tracking module and an optimization and generation network module;
the intention and trough value network module is built by adopting a Bert + CRF structure, wherein the Bert is used for intention classification, the CRF is used for trough value extraction, and the module pre-trains three tasks, namely an intention classification task, a trough value extraction task and a masking word prediction task of the Bert;
the database operation module is mainly used for inquiring other slot values according to the identified slot values;
the trust tracking module (Belief Tracker) adopts a Roberta model, and outputs a vector S containing all information from the beginning to the current roundn+1；
And the optimization and generation network module adopts a classification algorithm, selects the highest score as the answer of the current round of conversation in the existing action, and takes the answer as the input of the trust tracking module for subsequent prediction.
A task-based dialog method based on seq2seq architecture comprises the following steps:
1) bert output __ CLS __ Token a of classification taskclsAnd the vector y of the input labelintentMapping to the same vector space by maximizing the similarity S of the predicted value and the label positive caseI +＝hCLSh+ intentAnd minimizing the similarity S of the predicted value and the label negative caseI -＝hCLSh- intentThe optimization is carried out, and the specific formula is as follows:
Figure BDA0003471124870000021
2) the groove value extraction task is trained through a CRF layer, and the formula is as follows:
LE＝LCRF(a,yentity)
wherein y isentityInputting a hidden vector of a label for an entity, a being a corresponding vector Token, L of a data inputCRF(.) is the negative similarity of the CRF;
3) the overall loss of the intent and slot value fill layers is calculated as:
Ltotal＝LI+LE+LM
in the formula LMMask prediction for Bert self;
4) the database operation module is mainly used for inquiring other slot values according to the identified slot values and searching other slot values through select, where food and drink according to the keywords provided by the system;
5) a trust tracking layer using the output of the intention and slot value network module and the database operation module as input UIOne, with the output Sn of the layer as one of the inputs, one of the inputs An of the intent and generation network, the Belief Tracker, as the final input
Ib＝UI+Sn+An
Wherein U isIRepresenting the user intention and the corresponding entity in the current round, Sn representing all states from the beginning to the previous round, An representing the answer to the user question in the previous round;
6) and the optimizing and generating network module adopts a classification algorithm for a primary final result, selects the answer An with the highest score as the answer of the current conversation in the existing action, and takes the answer as the input of the Belief Tracker for subsequent prediction.
The invention has the advantages that: the present invention combines a task-based dialog system with a chat-based dialog system. The whole adopts a seq2seq structure, and the module of the task dialog system is combined inside, so that the context effect is ensured, and the whole optimization can be carried out.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention.
Fig. 1 is a schematic view of the overall structure of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The entry to this model is the intent and slot value network. The network is built by adopting a Bert + CRF structure. Bert is used for intent classification and CRF is used for bin value extraction. During pre-training, three tasks, namely an intention classification task, a groove value extraction task and a Bert self mask word prediction task, are mainly considered.
Bert output __ CLS __ Token a of classification taskclsAnd the vector y of the input labelintentMapping to the sameVector space by maximizing the similarity S of the predicted value and the label positive caseI +＝hCLSh+ intentAnd minimizing the similarity S of the predicted value and the label negative caseI -＝hCLSh- intentThe optimization is carried out, and the specific formula is as follows:
Figure BDA0003471124870000031
the groove value extraction task is trained through a CRF layer, and the formula is as follows:
LE＝LCRF(a,yentity)
wherein y isentityInputting a hidden vector of a label for an entity, a being a corresponding vector Token, L of a data inputCRF(.) is the negative similarity of the CRF;
the overall loss of the intent and slot value fill layers is calculated as:
Ltotal＝LI+LE+LM
in the formula LMMask prediction for Bert self;
and the database operation module is mainly used for inquiring other slot values according to the identified slot values and is used for answering and generating. For example:
human: giving me a cup of beverage.
The system comprises the following steps: ask what beverage you need?
Human: what do you have?
The system comprises the following steps: we have coffee, milk tea, and fruit juice.
In the above dialog, after the system extracts the "beverage" keyword, it can search through select where food is drink.
A trust tracking layer, which takes the output of the intention and slot value network module and the database operation module as input UIOne, with the output Sn of the layer as one of the inputs, one of the inputs An of the intent and generation network, the Belief Tracker, as the final input
Ib＝UI+Sn+An
Wherein U isIRepresenting the user intention and the corresponding entity in the current round, Sn representing all states from the beginning to the previous round, An representing the answer to the user question in the previous round; the network considers the historical information and the influence of the information of the current round. The network adopts the Roberta model. The output is a vector S containing all information from the beginning to the current roundn+1。
The optimization and generation network module adopts a classification algorithm, and because the model is used for a task-based dialog system, the accuracy of output results is most important. And (4) adopting a classification algorithm as a final result, selecting the answer An with the highest score as the answer of the current conversation in the existing action, and taking the answer as the input of a Belief Tracker for subsequent prediction.

Claims (2)

1. A task-based dialogue system based on a seq2seq architecture is characterized by comprising four parts, namely an intention and slot filling network module, a database operation module, a trust tracking module and an optimization and generation network module;
the intention and trough value network module is built by adopting a Bert + CRF structure, wherein the Bert is used for intention classification, the CRF is used for trough value extraction, and the module pre-trains three tasks, namely an intention classification task, a trough value extraction task and a masking word prediction task of the Bert;
the database operation module is mainly used for inquiring other slot values according to the identified slot values;
the trust tracking module adopts a Roberta model and outputs a vector S containing all information from the beginning to the current roundn+1；
And the optimization and generation network module adopts a classification algorithm, selects the highest score as the answer of the current round of conversation in the existing action, and takes the answer as the input of the trust tracking module for subsequent prediction.
2. A task-based dialog method using the seq2seq architecture as claimed in claim 1, characterized in that it comprises the following steps:
1) of classification tasksbert output __ CLS __ Token aclsAnd the vector y of the input labelintentMapping to the same vector space by maximizing the similarity S of the predicted value and the label positive caseI +＝hCLSh+ intentAnd minimizing the similarity S of the predicted value and the label negative caseI -＝hCLSh- intentThe optimization is carried out, and the specific formula is as follows:
Figure FDA0003471124860000011
2) the groove value extraction task is trained through a CRF layer, and the formula is as follows:
LE＝LCRF(a,yentity)
wherein y isentityInputting a hidden vector of a label for an entity, a being a corresponding vector Token, L of a data inputCRF(.) is the negative similarity of the CRF;
3) the overall loss of the intent and slot value fill layers is calculated as:
Ltotal＝LI+LE+LM
in the formula LMMask prediction for Bert self;
4) the database operation module is mainly used for inquiring other slot values according to the identified slot values and searching other slot values through select, where food and drink according to the keywords provided by the system;
5) a trust tracking layer module using the output of the intention and slot value network module and the database operation module as input UIOne, with the output Sn of the layer as one of the inputs, one of the inputs An of the intent and generation network, the Belief Tracker, as the final input
Ib＝UI+Sn+An
Wherein U isIRepresenting the user intention and the corresponding entity in the current round, Sn representing all states from the beginning to the previous round, An representing the answer to the user question in the previous round;
6) and the optimizing and generating network module adopts a classification algorithm for a primary final result, selects the answer An with the highest score as the answer of the current conversation in the existing action, and takes the answer as the input of the Belief Tracker for subsequent prediction.
CN202210043220.7A 2022-01-14 2022-01-14 Task type dialogue system and method based on seq2seq framework Pending CN114417889A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210043220.7A CN114417889A (en) 2022-01-14 2022-01-14 Task type dialogue system and method based on seq2seq framework

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210043220.7A CN114417889A (en) 2022-01-14 2022-01-14 Task type dialogue system and method based on seq2seq framework

Publications (1)

Publication Number Publication Date
CN114417889A true CN114417889A (en) 2022-04-29

Family

ID=81273088

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210043220.7A Pending CN114417889A (en) 2022-01-14 2022-01-14 Task type dialogue system and method based on seq2seq framework

Country Status (1)

Country Link
CN (1) CN114417889A (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111324744A (en) * 2020-02-17 2020-06-23 中山大学 Data enhancement method based on target emotion analysis data set
US20210201079A1 (en) * 2019-12-27 2021-07-01 Fujitsu Limited Training data generation method and information processing apparatus
CN113705249A (en) * 2021-08-25 2021-11-26 上海云从企业发展有限公司 Dialogue processing method, system, device and computer readable storage medium

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210201079A1 (en) * 2019-12-27 2021-07-01 Fujitsu Limited Training data generation method and information processing apparatus
CN111324744A (en) * 2020-02-17 2020-06-23 中山大学 Data enhancement method based on target emotion analysis data set
CN113705249A (en) * 2021-08-25 2021-11-26 上海云从企业发展有限公司 Dialogue processing method, system, device and computer readable storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李培芸;李茂西;裘白莲;王明文;: "融合BERT语境词向量的译文质量估计方法研究", 中文信息学报, no. 03, 15 March 2020 (2020-03-15) *

Similar Documents

Publication Publication Date Title
CN111931513B (en) Text intention recognition method and device
You et al. Image captioning at will: A versatile scheme for effectively injecting sentiments into image descriptions
US20230244879A1 (en) Language representation model system, pre-training method and apparatus, device, and medium
CN111914085B (en) Text fine granularity emotion classification method, system, device and storage medium
CN109977207A (en) Talk with generation method, dialogue generating means, electronic equipment and storage medium
CN108563779B (en) Neural network-based template-free natural language text answer generation method
US11475225B2 (en) Method, system, electronic device and storage medium for clarification question generation
US20230395075A1 (en) Human-machine dialogue system and method
CN112562669B (en) Method and system for automatically abstracting intelligent digital newspaper and performing voice interaction chat
CN109933773B (en) Multiple semantic statement analysis system and method
CN110472255A (en) Neural network machine interpretation method, model, electric terminal and storage medium
Hani et al. Image caption generation using a deep architecture
Zhu et al. Prompt-based learning for unpaired image captioning
CN111128175B (en) Spoken language dialogue management method and system
Zhu et al. A prompt model with combined semantic refinement for aspect sentiment analysis
CN114328866A (en) Strong anthropomorphic intelligent dialogue robot with smooth and accurate response
Zhang et al. Chatbot design method using hybrid word vector expression model based on real telemarketing data
CN114417889A (en) Task type dialogue system and method based on seq2seq framework
CN116644759B (en) Method and system for extracting aspect category and semantic polarity in sentence
CN116403608A (en) Speech emotion recognition method based on multi-label correction and space-time collaborative fusion
CN115841106A (en) Aspect-level emotion analysis method based on graph convolution network
Lei et al. Zero-shot state tracking and user adoption tracking on schema-guided dialogue
Guo et al. Ernie-bilstm based Chinese text sentiment classification method
CN114330367A (en) Semantic similarity obtaining method, device and equipment based on sentences
Mecik et al. Neural language generation for a Turkish task-oriented dialogue system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination