CN117035078A - Multi-mode knowledge graph unified representation learning framework - Google Patents
Multi-mode knowledge graph unified representation learning framework Download PDFInfo
- Publication number
- CN117035078A CN117035078A CN202310979635.XA CN202310979635A CN117035078A CN 117035078 A CN117035078 A CN 117035078A CN 202310979635 A CN202310979635 A CN 202310979635A CN 117035078 A CN117035078 A CN 117035078A
- Authority
- CN
- China
- Prior art keywords
- knowledge
- model
- entity
- graph
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012549 training Methods 0.000 claims abstract description 83
- 238000000034 method Methods 0.000 claims abstract description 50
- 238000005070 sampling Methods 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 33
- 230000006870 function Effects 0.000 claims abstract description 30
- 230000008569 process Effects 0.000 claims abstract description 23
- 238000011156 evaluation Methods 0.000 claims abstract description 22
- 238000010586 diagram Methods 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 7
- 238000012360 testing method Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 5
- 230000008859 change Effects 0.000 claims description 4
- 238000004458 analytical method Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 3
- 230000005477 standard model Effects 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 abstract description 7
- 238000012827 research and development Methods 0.000 abstract description 3
- 238000004590 computer program Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 7
- 238000001914 filtration Methods 0.000 description 5
- 238000006467 substitution reaction Methods 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000003860 storage Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000009827 uniform distribution Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000002787 reinforcement Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/022—Knowledge engineering; Knowledge acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
- G06F16/367—Ontology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
- G06N5/041—Abduction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- Databases & Information Systems (AREA)
- Machine Translation (AREA)
Abstract
The application relates to a multi-mode knowledge graph unified representation learning framework, which comprises the following structures: the knowledge graph data processing module is used for preprocessing an input knowledge graph data set; the knowledge graph data sampling module is used for generating positive samples and negative samples required by the training model according to the data in the knowledge graph data set; the basic knowledge representation model support library is used for constructing and evaluating a knowledge representation model through the basic knowledge representation model support library and the knowledge graph data set; the basic knowledge representation model support library comprises a basic model library, a loss function library and a model evaluation function library; knowledge representation model training support tools for assisting in building, training and evaluating knowledge representation models. The application supports a plurality of samplers which are convenient to develop knowledge representation learning, and greatly simplifies the research and development process of multi-mode knowledge representation learning.
Description
Technical Field
The application relates to a multi-mode knowledge graph unified representation learning framework, and belongs to the technical field of knowledge graphs.
Background
Knowledge graph is a graph structure data representing relationships between facts, concepts, events, and entities. Knowledge maps integrate real-world information and relationships into a structure that is easy to understand and query in order to quickly find the desired information. Knowledge graph has become an effective knowledge representation, organization, association and display technology, is widely applied to aspects of recommendation systems, automatic question and answer, information retrieval and the like, and has become an infrastructure of artificial intelligence systems. The traditional knowledge graph is constructed by text information such as web pages, documents and the like, and the emerging multi-mode knowledge graph can be fused with data of different modes such as characters, images, audio and video to organize different types of information in a structured form. For each entity, the multimodal knowledge graph can store image, audio, and video information related thereto, as well as relationships between them. Through the multi-mode knowledge graph, the computer can better utilize the data to perform semantic understanding, reasoning and prediction, so that the performance of applications such as intelligent recommendation, searching, information retrieval and the like is improved. Compared with the traditional knowledge graph, the multi-mode knowledge graph can better capture the information rich behind the entity, and is beneficial to realizing more accurate and personalized service.
Knowledge representation learning is a method of automatically learning and extracting useful information from a knowledge graph using computer technology. Its purpose is to present the relationships of facts, concepts, events, and entities in a computer-understandable form. Specifically, knowledge representation learning is based on the idea of a distributed representation, mapping semantic information of entities (or relationships) into a low-dimensional, dense, real-valued vector space, such that the distance between two objects that are semantically similar is also similar. Through knowledge representation learning, a computer can make inferences about things, predict things, and provide more accurate results. Knowledge representation learning is an important technical means for better understanding and utilizing knowledge maps. The traditional single-mode knowledge representation learning has a mature technical process, and a knowledge graph representation learning model can be used for different knowledge representation learning tasks to develop. In contrast, multi-modal knowledge graph representation learning is in an early stage, and based on single-modal knowledge graph representation learning, processing and representation of multi-modal data such as images, texts, videos, audios and the like are additionally considered, and specific challenges brought by multi-modal data, including heterogeneous data processing, multi-modal data alignment, modal data deletion and the like, are additionally considered.
In the prior art, the patent application numbers CN202110305818.4 (a time sequence knowledge graph representation learning method based on collaborative evolution modeling) and CN201911380039.X (a dynamic knowledge graph representation learning method and system based on anchor points) mainly pay attention to time sequence and dynamic knowledge graph representation learning, and the processing involves time evolution and self-adaptive growth of a knowledge base. The limitation of the traditional representation learning method in the time sequence and dynamic knowledge graph scene is solved.
However, services such as multisource heterogeneous data processing, a universal sampler method and an omnibearing support knowledge representation learning model are not considered, so that the system has poor universality and cannot work for multitasking.
In the prior art, patent application numbers CN201911376444.4 (a knowledge representation learning method based on double-agent reinforcement learning path search), CN201910431538.0 (a rapid learning method oriented to a large-scale knowledge base), CN201910629813.X (a knowledge graph representation learning method based on attention mechanism integrated text semantic features), and CN201911300781.5 (a Chinese knowledge graph representation learning method based on feature tensor) use specific technologies (such as reinforcement learning, attention mechanism, etc.) to perform knowledge graph representation learning, and integrate text semantic features of entities and relations to improve the effect of representation learning. The method solves the problems of insufficient semantic features, convergence speed and precision faced by the traditional representation learning method.
In summary, the universality and convenience of the whole framework are not considered in the prior art, and the problems of weak data processing capability and insufficient model training efficiency exist.
Disclosure of Invention
In order to overcome the problems, the application provides a multi-modal knowledge graph unified representation learning framework which supports a plurality of samplers for conveniently developing knowledge representation learning, supports a basic model library, a loss function library and an evaluation index library for training a high-performance domain professional knowledge graph representation model in all directions, efficiently and conveniently trains a supporting tool for the knowledge representation model, and greatly simplifies the research and development process of the multi-modal knowledge representation learning.
The technical scheme of the application is as follows:
first aspect
A multi-modal knowledge graph unified representation learning framework comprises the following structures:
the knowledge graph data processing module is used for preprocessing an input knowledge graph data set and comprises the step of distributing a unique ID for each entity, relation and other modes in the knowledge graph data set;
the knowledge graph data sampling module is used for generating positive samples and negative samples required by the training model according to the data in the knowledge graph data set;
a basic knowledge representation model support library, through which a knowledge representation model is constructed and evaluated; the basic knowledge representation model support library comprises a basic model library, a loss function library and a model evaluation function library; the basic model library provides a knowledge graph representation learning algorithm, the loss function library provides a loss function for optimizing the performance of the model, and the model evaluation function library provides an evaluation index for evaluating the performance of the knowledge representation learning model;
knowledge representation model training support tools for assisting in building, training and evaluating knowledge representation models.
Further, other modes in the knowledge-graph data set include images and audio, and the knowledge-graph data processing module performs preprocessing on the knowledge-graph data set further includes:
reading and processing the multi-mode data, and preprocessing the image data in the knowledge graph data set to enable different image data to have the same basic attribute;
extracting the characteristics of the audio data in the knowledge graph data set;
dividing the data in the knowledge graph data set into a training set, a verification set and a test set according to a preset proportion;
statistical data is generated, and the composition of the knowledge graph dataset is counted from the dimensions of the entity, the attribute, the relation and other modes.
Further, the knowledge-graph data sampling is performed by a basic sampler, and the basic sampler is used for:
establishing a statistical dictionary by counting the training set, wherein the statistical dictionary records entities and corresponding relations thereof in the training set and frequency of the entities and the relations;
when generating a negative sample, searching whether the negative sample to be generated is in the statistical dictionary, if so, eliminating the negative sample to be generated, otherwise, taking the negative sample to be generated as the negative sample;
setting a threshold T 1 If the occurrence frequency of the entity or the relation of the negative sample to be generated in the statistical dictionary is lower than a threshold T 1 Removing;
if the entity degree of the entity to be generated into the negative sample in the statistical dictionary is lower than a threshold T 2 Removing; the entity degree is the number of the relations connected with the entity.
Further, the knowledge-graph data sampling is performed by a negative sampler, and the negative sampler is used for:
randomly selecting a head entity, a tail entity and a relation from the training set, generating a triplet as a negative sample A, and replacing the head entity and the tail entity of the negative sample A according to a preset probability;
and sampling each head entity and each tail entity from the training set according to the same probability to generate a negative sample.
Further, the identification map data sampling is sampled by a map sampler, and the map sampler is used for:
establishing a statistical chart according to the training set, wherein the statistical chart records the association degree and weight among entities;
and carrying out the following operations on all triples in the training set:
for the triplet A, other entities and weights thereof related to the head entity and the tail entity of the triplet A are obtained from the statistical diagram, and a head entity set A and a tail entity set A related to the triplet A are generated;
and according to the weights of the entities in the head entity set A and the tail entity set A, sampling according to the normalized probability to obtain the head entity A in the head entity set A, obtaining the tail entity A in the tail entity set A, replacing the head entity of the triplet A by the head entity A, replacing the tail entity of the triplet A by the tail entity A, and generating a negative sample.
Further, the knowledge representation model training support tool includes:
and (3) super parameter debugging: receiving and managing command line parameters by using argparse, and performing super-parameter searching through the assistance of a super-parameter optimization library so as to automatically find the optimal super-parameter configuration;
graphical training curves: outputting the loss value and the evaluation index obtained in the training process to a TensorBoard X or TensorBoard, supporting the real-time check of the change of the loss value and the evaluation index in the training process by using a web interface of the TensorBoard,
further, the knowledge representation model training support tool further includes:
storing a standardized model, and after model training is completed, storing the model by adopting a standard model storing method of Pytorch;
and recording and checking analysis of the log, and recording training process information of the model through a logging library.
Second aspect
A knowledge representation model construction method based on a multi-modal knowledge graph unified representation learning framework, according to the first aspect, constructs a knowledge representation model comprising:
processing the input knowledge-graph data through the knowledge-graph data processing module;
sampling the knowledge-graph data by the knowledge-graph data sampling module to generate a positive sample and a negative sample;
constructing and evaluating a knowledge representation model through the basic knowledge representation model support library and the knowledge graph dataset;
the knowledge representation model is built, trained and evaluated with the knowledge representation model training support tool.
Third aspect of the application
A knowledge-graph data sampling method based on a multi-modal knowledge-graph unified representation learning framework, according to the first aspect, samples input knowledge-graph data by the multi-modal knowledge-graph unified representation learning framework, comprising:
and processing the input knowledge-graph data through the knowledge-graph data sampling module, and combining the basic sampler, the negative sampler and the graph sampler when generating a positive sample and a negative sample.
Further, the method further comprises the following steps:
when the positive sample and the negative sample are generated, the ratio of the positive sample to the negative sample reaches a preset value.
The application has the following beneficial effects:
1. the framework supports a plurality of samplers for conveniently developing knowledge representation learning, supports a basic model library, a loss function library and an evaluation index library for training a high-performance domain professional knowledge map representation model in all directions, is an efficient and simple knowledge representation model training support tool, and greatly simplifies the research and development process of multi-mode knowledge representation learning.
2. The framework provides reading and processing functions of multi-mode data, such as loading of a large-scale multi-mode knowledge graph data set, processing of image audio data and the like, and realizes efficient data loading and management.
3. The framework provides a variety of common samplers, such as basic samplers, negative samplers, graph samplers, etc., to support convenient development of knowledge representation learning. Different samplers are adopted, so that the proportion of positive and negative samples can be balanced, and the generalization capability, accuracy and learning effect of the model are improved. Meanwhile, the optimization sampling strategy is also beneficial to reducing training time and improving model training efficiency.
Drawings
Fig. 1 is a schematic block diagram of an embodiment of the present application.
Detailed Description
The application will now be described in detail with reference to the drawings and to specific embodiments.
First aspect
Referring to fig. 1, a multi-modal knowledge graph uniformly represents a learning framework, comprising the following structures:
the knowledge graph data processing module is used for preprocessing an input knowledge graph data set, and comprises the step of respectively distributing a unique ID for each entity, relation and other modes in the knowledge graph data set, wherein the unique IDs are helpful for quick positioning, inquiring and operating of elements in the data set, and the unique IDs are used as one of input features in a subsequent model to provide better distinguishability for different entity, relation and mode contents, so that the efficiency of a data processing flow is improved.
The knowledge graph data sampling module is used for generating positive samples and negative samples required by the training model according to the data in the knowledge graph data set;
a basic knowledge representation model support library, through which a knowledge representation model is constructed and evaluated; the basic knowledge representation model support library comprises a basic model library, a loss function library and a model evaluation function library; the basic model library provides a knowledge graph representation learning algorithm, the loss function library provides a loss function for optimizing the performance of the model, and the model evaluation function library provides an evaluation index for evaluating the performance of the knowledge representation learning model;
knowledge representation model training support tools for assisting in building, training and evaluating knowledge representation models.
The framework supports multi-source heterogeneous data processing, distributes unique IDs for each entity, relation and other modal contents in the data set, and realizes efficient data loading and management. The comprehensive knowledge representation learning model is constructed, trained, evaluated and optimized, and the comprehensive knowledge representation learning model comprises a basic model library, a loss function library, an evaluation index library and the like.
In one embodiment of the present application, the other modes in the knowledge-graph data set include images and audio, and the knowledge-graph data processing module is specifically configured to:
reading and processing the multi-mode data, and preprocessing the image data in the knowledge graph data set to enable different image data to have the same basic attribute; image data with the same basic attribute can be processed uniformly by the model;
the method is characterized in that the audio data in the knowledge graph data set are subjected to feature extraction, so that the waveform data of the original audio data are converted into a more representative and easy-to-process form, the audio data after feature extraction can better reflect the content of the audio, the dimension of the data is reduced, and the training efficiency of a model is improved.
Dividing the data in the knowledge graph data set into a training set, a verification set and a test set according to a preset proportion; considering that the data sets for constructing the professional multi-modal knowledge graph are large, the index graph data set is segmented to form a series of data sets which are formed by small batches of data and are easy to split and calculate, so that the subsequent training, verification and test in the cluster distributed environment are facilitated;
generating statistical data, and counting the constitution of a knowledge graph dataset from the dimensions of entities, attributes, relations and other modes; the formation of the statistical knowledge graph dataset provides statistical references for the subsequent processes including selection of the negative sampler, and the like, so that the distribution characteristics of the dataset can be known, and the performance of the model can be optimized.
Through the realization of the capability of the knowledge graph data processing module, the knowledge graph data processing link can effectively preprocess, load and divide a large-scale multi-modal knowledge graph data set into a form suitable for training a multi-modal deep learning model, thereby solving the problems of small batch training, hardware resource utilization rate, cross-modal negative sampling strategy selection and the like, and improving the efficiency and effect of multi-modal knowledge representation learning
In some embodiments of the application, the framework is built based on an open-source deep learning framework Pytorch.
In a specific embodiment of the present application, multi-modal data reading is achieved by expanding the Dataset class of PyTorch.
In a specific embodiment of the present application, the basic properties of the image data include size and scale.
In a specific embodiment of the application, preprocessing of the image data is achieved by cropping, scaling and random flipping.
The clipping scaling and the shrinking scaling can enable the image data to have the same size and proportion, random overturning can increase the diversity of data in the map data set only, and the generalization capability of the model is improved.
In a specific embodiment of the present application, the characteristic representation of the audio data is a mel-frequency cepstral coefficient (MFCC).
In one embodiment of the application, the segmentation of the multimodal dataset is supported by extending the Dataoader class of PyTorch. The specific implementation steps comprise:
splitting a data set into a training set, a verification set and a test set according to a given proportion by using a random_split function of PyTorch;
and loading the segmented sub-data set by using the extended DataLoader.
The sampling link is an important link in knowledge graph representation learning and is used for generating positive samples and negative samples required by a training model. Positive samples are the correct triples extracted from the existing knowledge-graph, typically all or a subset of the samples of the training set, and are relatively simple to obtain. While negative samples refer to triplets that are incorrect or non-existent, but are required to meet characteristics of typically, equidistribution, etc. The generation of negative samples helps to improve the generalization ability of the model and the ability to distinguish between correct and incorrect relationships is a difficulty in constructing a dataset.
In one embodiment of the application, the knowledge-graph data sampling is performed by a base sampler for:
establishing a statistical dictionary by counting the training set, wherein the statistical dictionary records entities and corresponding relations thereof in the training set and frequency of the entities and the relations;
error negative sample filtering, comprising:
when generating a negative sample, searching whether the negative sample to be generated is in the statistical dictionary, if so, eliminating the negative sample to be generated, otherwise, taking the negative sample to be generated as the negative sample; in a specific embodiment, several negative samples can be repeatedly generated by this method
Low quality negative sample filtering, comprising:
frequency rule, set threshold T 1 If the occurrence frequency of the entity or the relation of the negative sample to be generated in the statistical dictionary is lower than a threshold T 1 Then eliminating to improve the quality and diversity of the generated negative samples;
entity degree rule, if entity degree of entity to be generated negative sample is lower than threshold T in the statistical dictionary 2 Removing; the entity degree is the number of the relations connected with the entity. The entities in the remaining negative samples have a higher degree and are more likely to appear in the actual positive samples, thus better helping the model learn the actual information.
In an embodiment of the application, the base sampler generates negative samples by both error negative sample filtering and low quality negative sample filtering in combination or separately.
In embodiments of the present application, the frequency rule and the entity degree rule may be used in combination or separately when low quality negative sample filtering is performed.
In one embodiment of the application, the knowledge-graph data sampling is performed by a negative sampler for:
a random replacement strategy, namely randomly selecting a head entity, a tail entity and a relation from the training set, generating a triplet as a negative sample A, and replacing the head entity and the tail entity of the negative sample A according to a preset probability; the probability distribution of the negative samples is changed by adjusting the replacement proportion of the head entity or/and the tail entity, so that the frequency of different negative samples in training data can be balanced, the diversity of the negative samples is improved, and the model can learn and distinguish the correct and wrong relation better.
And (3) uniformly distributing strategies, sampling each head entity and each tail entity from the training set according to the same probability, and generating a negative sample.
In embodiments of the present application, the random substitution strategy and the uniform distribution strategy may be used in combination or alone when sampling by the negative sampler.
In a specific embodiment of the present application, the probability of replacement of the head entity and the tail entity of the random replacement policy is 0.5.
For example, a triplet represents a person (head entity) in a city (tail entity) and the relationship is "liquids_in". Triplet a is represented as:
(Small piece, lives_in, fuzhou)
Negative samples are generated by replacing either the head entity or the tail entity. Examples of replacement header entities:
(Laowang, lives_in, fuzhou)
Depending on the given probability, sometimes the head entity is replaced, sometimes the tail entity is replaced, sometimes both the head entity and the tail entity are replaced. This approach guarantees the diversity of the negative samples and allows the model to learn the distinction between single-sided substitution (substitution of only head or tail entities) and double-sided substitution (simultaneous substitution of head and tail entities).
The entity which is different from the original entity can be selected from the current entity set randomly as a replacement object during replacement. Thus, a plurality of different negative samples can be obtained, and the generalization capability of the model is enhanced.
In a specific embodiment of the application, when sampling is performed through a uniform distribution strategy, only 5 entities and 3 relations are in the map, each entity and each relation have the same probability of being extracted, the probability of the entity is 1/5, and the probability of the relation is 1/3. By generating the negative samples in this way, the same opportunity for each entity and relationship to be selected can be ensured, thereby improving the diversity of the negative samples.
In one embodiment of the application, the identification map data samples are sampled by a map sampler for:
establishing a statistical chart according to the training set, wherein the statistical chart records the association degree and weight among entities; the degree of association and the weight are used to describe the strength of relationship between entities, and in some embodiments of the present application, the degree of association indicates the relationship between entities in a knowledge graph, and the weight indicates the strength of such relationship, for example, entities B and C appear in 10 triples together in the knowledge graph, and entities B and D appear in 1 triplet only together, and the degree of association and the weight of entities B and C are greater than those of entities B and D. The method comprises the steps of carrying out a first treatment on the surface of the
And carrying out the following operations on all triples in the training set:
for the triplet A, other entities and weights thereof related to the head entity and the tail entity of the triplet A are obtained from the statistical diagram, and a head entity set A and a tail entity set A related to the triplet A are generated;
and according to the weights of the entities in the head entity set A and the tail entity set A, sampling according to the normalized probability to obtain the head entity A in the head entity set A, obtaining the tail entity A in the tail entity set A, replacing the head entity of the triplet A by the head entity A, replacing the tail entity of the triplet A by the tail entity A, and generating a negative sample.
By considering the degree of association and the weight between entities in the statistical map, the negative samples generated will take into account the structural information between the entities, so that negative samples that are more closely associated with the correct triples are more likely to be selected. Therefore, the method can ensure that the generated negative sample has higher structural correlation, and is beneficial to the model to learn more knowledge closely related to the structure, so that the learning effect model which is sparse or easy to ignore is improved to learn the easily ignored or undersampled relation in the knowledge graph.
Obtaining an entity A in the entity set A according to the normalized probability sampling, specifically:
obtaining an entity weight set W= { W according to the entity weight 1 ,w 2 ,...,w n W, where 1 ,w 2 ,...,w n Weights for different entities;
calculate the normalized weight set W' = { W 1 ',w 2 ',...,w n '}；
Wherein w is i '＝w i /ΣW；
The sampling is performed according to the normalized weights in W', and the probability that each entity is selected is equal to its normalized weight.
In the embodiments of the present application, the basic sampler, the negative sampler, and the graph sampler described above may be used in combination or separately.
In knowledge representation learning, there are often a large number of positive samples (i.e., labeled triples), but fewer negative samples (erroneous triples). The negative sampler improves generalization capability, accuracy and learning effect of the model by sampling negative samples. By adopting different samplers, we can generate high-quality negative samples from different angles, balance the proportion of the positive and negative samples, and further improve the generalization capability, accuracy and learning effect of the model. The basic sampler generally does not directly generate negative samples, and the key point is that incorrect and low-quality negative samples are filtered out by providing a convenient rule and strategy mechanism, so that a better negative sample data set is obtained; the negative sampler is specially responsible for generating negative samples, generates the negative samples based on a random replacement or uniform distribution strategy, and can simply sample to obtain a large number of negative samples; the graph sampler considers the sampling problem from the perspective of the network topology structure, and can help the model capture richer data characteristics so as to generate high-quality samples required by specific tasks.
Through the use of different samplers, the generalization capability, accuracy and learning effect of the model can be improved from multiple angles such as positive and negative sample balance, topological structure capture and the like. These samplers may be used alone or in combination to achieve better performance. The practical case can select an appropriate sampling engine according to specific tasks and data sets, and parameter adjustment is performed to realize the best performance.
In one embodiment of the present application, the base model library includes a conventional knowledge graph embedding algorithm, a knowledge graph embedding algorithm based on a graph neural network, and a rule-based knowledge graph embedding algorithm;
each algorithm in the basic model library has a unified base class and interface.
The basic model library provides a highly flexible and modularized framework for knowledge graph representation learning, so that researchers can construct and adjust the domain professional knowledge graph representation model more easily and quickly, the development difficulty is reduced, and the research progress is accelerated.
In one embodiment of the present application, the library of Loss functions includes BCE Loss, sigmod functions.
In one embodiment of the application, the model evaluation function library includes creating independent model evaluation tool classes that are capable of providing different evaluation metrics. Such as Link Prediction (calculating the matching degree of the predicted knowledge-graph entity relationship and the real entity relationship), mean Reciprocal Rank (MRR, measuring the ability of the model to rank the correct entity or relationship in the front of the list), etc., helps researchers and developers to understand the model performance, and further optimizes the model.
In one embodiment of the present application, the knowledge representation model training support tool comprises:
and (3) super parameter debugging: receiving and managing command line parameters by using argparse, and performing super-parameter searching through the assistance of a super-parameter optimization library so as to automatically find the optimal super-parameter configuration; in knowledge graph representation learning, there may be many hyper-parameters for different models. These hyper-parameters can affect the performance and convergence speed of the model. Training the support tool module provides a simplified hyper-parametric tuning process that allows developers to more easily adjust parameters to obtain optimal model performance.
Graphical training curves: outputting the loss value and the evaluation index obtained in the training process to a TensorBoard X or TensorBoard, and supporting the real-time check of the change of the loss value and the evaluation index in the training process by using a web interface of the TensorBoard; intuitive training curves may help developers better understand the training process and performance of the model. The training support tool module provides a function of drawing a training curve, thereby reflecting the loss and the precision change of the model in the training process in real time so as to evaluate the convergence and the stability of the model.
In one embodiment of the present application, the knowledge representation model training support tool further comprises:
storing a standardized model, and after model training is completed, storing the model by adopting a standard model storing method of Pytorch; after model training is completed, the model is usually required to be stored for subsequent verification, test or deployment of the model, and the method simplifies the process of model storage and loading, and is convenient for subsequent direct use.
And recording and checking analysis of the log, and recording training process information of the model through a logging library. In the training process, log information recording is critical to model debugging, problem positioning and performance optimization, and the training support tool module can record rich training process information, such as loss value and precision of each round, super-parameter setting, training duration and the like, so that developers can better know the training process and performance.
Second aspect
A knowledge representation model construction method based on a multi-modal knowledge graph unified representation learning framework, according to the first aspect, constructs a knowledge representation model comprising:
processing the input knowledge-graph data through the knowledge-graph data processing module;
sampling the knowledge-graph data by the knowledge-graph data sampling module to generate a positive sample and a negative sample;
constructing and evaluating a knowledge representation model through the basic knowledge representation model support library and the knowledge graph dataset;
the knowledge representation model is built, trained and evaluated with the knowledge representation model training support tool.
Third aspect of the application
A knowledge-graph data sampling method based on a multi-modal knowledge-graph unified representation learning framework, according to the first aspect, samples input knowledge-graph data by the multi-modal knowledge-graph unified representation learning framework, comprising:
and processing the input knowledge-graph data through the knowledge-graph data sampling module, and combining the basic sampler, the negative sampler and the graph sampler when generating a positive sample and a negative sample.
In one embodiment of the present application, further comprising:
when the positive sample and the negative sample are generated, the ratio of the positive sample to the negative sample reaches a preset value.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The scheme in the embodiment of the application can be realized by adopting various computer languages, such as object-oriented programming language Java, an transliteration script language JavaScript and the like.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (10)
1. The multi-mode knowledge graph unified representation learning framework is characterized by comprising the following structures:
the knowledge graph data processing module is used for respectively distributing a unique ID to each entity, relation and other modes in the input knowledge graph data set;
the knowledge graph data sampling module is used for generating positive samples and negative samples required by the training model according to the data in the knowledge graph data set;
a basic knowledge representation model support library, through which a knowledge representation model is constructed and evaluated; the basic knowledge representation model support library comprises a basic model library, a loss function library and a model evaluation function library; the basic model library provides a knowledge graph representation learning algorithm, the loss function library provides a loss function for optimizing the performance of the model, and the model evaluation function library provides an evaluation index for evaluating the performance of the knowledge representation learning model;
knowledge representation model training support tools for assisting in building, training and evaluating knowledge representation models.
2. The multi-modal knowledge graph unified presentation learning framework of claim 1, wherein other modalities in a knowledge graph dataset include images and audio, the knowledge graph data processing module is specifically configured to read and process multi-modal data, and preprocess image data in the knowledge graph dataset to enable different image data to have the same basic attribute; extracting the characteristics of the audio data in the knowledge graph data set; dividing the data in the knowledge graph data set into a training set, a verification set and a test set according to a preset proportion; statistical data is generated, and the composition of the knowledge graph dataset is counted from the dimensions of the entity, the attribute, the relation and other modes.
3. The multi-modal knowledge graph unified representation learning framework of claim 2, wherein knowledge graph data sampling is performed by a basic sampler, the basic sampler is used for establishing a statistical dictionary by counting the training set, and the statistical dictionary records entities and corresponding relations thereof in the training set and the frequency of the entities and relations;
when generating the negative sample, searching whether the negative sample to be generated is in the statistical dictionary, if so, eliminating the negative sample to be generated, otherwise, taking the negative sample to be generated as the negative sample;
frequency rule, set threshold T 1 If the occurrence frequency of the entity or the relation of the negative sample to be generated in the statistical dictionary is lower than a threshold T 1 Removing;
entity degree rule, set threshold T 2 If the entity degree of the entity to be generated as the negative sample in the statistical dictionary is lower than a threshold T 2 Removing; the entity degree is the number of the relations connected with the entity.
4. A multi-modal knowledge-graph unified presentation learning framework as claimed in claim 3 wherein knowledge-graph data sampling is performed by a negative sampler for:
randomly selecting a head entity, a tail entity and a relation from the training set, generating a triplet as a negative sample A, and replacing the head entity and the tail entity of the negative sample A according to a preset probability;
and sampling each head entity and each tail entity from the training set according to the same probability to generate a negative sample.
5. The multi-modal knowledge-graph unified presentation learning framework of claim 4 wherein the knowledge-graph data samples are sampled by a graph sampler for:
establishing a statistical chart according to the training set, wherein the statistical chart records the association degree and weight among entities;
and carrying out the following operations on all triples in the training set:
for the triplet A, other entities and weights thereof related to the head entity and the tail entity of the triplet A are obtained from the statistical diagram, and a head entity set A and a tail entity set A related to the triplet A are generated;
and according to the weights of the entities in the head entity set A and the tail entity set A, sampling according to the normalized probability to obtain the head entity A in the head entity set A, obtaining the tail entity A in the tail entity set A, replacing the head entity of the triplet A by the head entity A, replacing the tail entity of the triplet A by the tail entity A, and generating a negative sample.
6. The multi-modal knowledge graph unified presentation learning framework of claim 1, wherein the knowledge presentation model training support tool comprises:
and (3) super parameter debugging: receiving and managing command line parameters by using argparse, and performing super-parameter searching through the assistance of a super-parameter optimization library so as to automatically find the optimal super-parameter configuration;
graphical training curves: and outputting the loss value and the evaluation index obtained in the training process to a TensorBoard X or TensorBoard, and supporting the real-time check of the change of the loss value and the evaluation index in the training process by using a web interface of the TensorBoard.
7. The multi-modal knowledge graph unified presentation learning framework of claim 6, wherein the knowledge presentation model training support tool further comprises:
storing a standardized model, and after model training is completed, storing the model by adopting a standard model storing method of Pytorch;
and recording and checking analysis of the log, and recording training process information of the model through a logging library.
8. A knowledge representation model construction method based on a multi-modal knowledge graph unified representation learning framework, according to any one of claims 1 to 7, characterized by comprising:
processing the input knowledge-graph data through the knowledge-graph data processing module;
sampling the knowledge-graph data by the knowledge-graph data sampling module to generate a positive sample and a negative sample;
constructing and evaluating a knowledge representation model through the basic knowledge representation model support library and the knowledge graph dataset;
the knowledge representation model is built, trained and evaluated with the knowledge representation model training support tool.
9. A knowledge-graph data sampling method based on a multi-modal knowledge-graph unified representation learning framework, which samples input knowledge-graph data according to the multi-modal knowledge-graph unified representation learning framework of claim 5, comprising:
and processing the input knowledge-graph data through the knowledge-graph data sampling module, and combining the basic sampler, the negative sampler and the graph sampler when generating a positive sample and a negative sample.
10. The knowledge-graph data sampling method based on the multi-mode knowledge-graph unified representation learning framework of claim 9, further comprising:
when the positive sample and the negative sample are generated, the ratio of the positive sample to the negative sample reaches a preset value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310979635.XA CN117035078A (en) | 2023-08-03 | 2023-08-03 | Multi-mode knowledge graph unified representation learning framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310979635.XA CN117035078A (en) | 2023-08-03 | 2023-08-03 | Multi-mode knowledge graph unified representation learning framework |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117035078A true CN117035078A (en) | 2023-11-10 |
Family
ID=88622004
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310979635.XA Pending CN117035078A (en) | 2023-08-03 | 2023-08-03 | Multi-mode knowledge graph unified representation learning framework |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117035078A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117787244A (en) * | 2023-12-18 | 2024-03-29 | 慧之安信息技术股份有限公司 | Data analysis method and system for Handle identification analysis |
-
2023
- 2023-08-03 CN CN202310979635.XA patent/CN117035078A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117787244A (en) * | 2023-12-18 | 2024-03-29 | 慧之安信息技术股份有限公司 | Data analysis method and system for Handle identification analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110442718B (en) | Statement processing method and device, server and storage medium | |
| WO2021093755A1 (en) | Matching method and apparatus for questions, and reply method and apparatus for questions | |
| US10740678B2 (en) | Concept hierarchies | |
| CN109284363A (en) | A kind of answering method, device, electronic equipment and storage medium | |
| US20220058222A1 (en) | Method and apparatus of processing information, method and apparatus of recommending information, electronic device, and storage medium | |
| CN110427483B (en) | Text abstract evaluation method, device, system and evaluation server | |
| CN111723295B (en) | Content distribution method, device and storage medium | |
| US20220147702A1 (en) | Automatic generation of transformations of formatted templates using deep learning modeling | |
| CN113742488B (en) | Embedded knowledge graph completion method and device based on multitask learning | |
| US11995573B2 (en) | Artificial intelligence system providing interactive model interpretation and enhancement tools | |
| US20230153335A1 (en) | Searchable data structure for electronic documents | |
| CN117035078A (en) | Multi-mode knowledge graph unified representation learning framework | |
| CN114911915A (en) | Knowledge graph-based question and answer searching method, system, equipment and medium | |
| CN112970011A (en) | Recording pedigrees in query optimization | |
| CN117494814A (en) | Prompt word full life cycle management method, system, electronic equipment and storage medium | |
| CN116049376B (en) | Method, device and system for retrieving and replying information and creating knowledge | |
| CN116186297A (en) | Graph manifold learning-based literature relationship discovery method and system | |
| CN115048531A (en) | Knowledge management method, device and system for urban physical examination knowledge | |
| CN109710844A (en) | The method and apparatus for quick and precisely positioning file based on search engine | |
| CN117708308B (en) | RAG natural language intelligent knowledge base management method and system | |
| CN117235237B (en) | Text generation method and related device | |
| CN117216193B (en) | Controllable text generation method and device based on large language model | |
| CN113761196B (en) | Text clustering method and system, electronic equipment and storage medium | |
| US11983244B1 (en) | Artificial intelligence system for media item classification using transfer learning and active learning | |
| CN113392864B (en) | Model generation method, video screening method, related device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |