CN117076494A - Real-time data query method and device, computer equipment and readable storage medium - Google Patents
Real-time data query method and device, computer equipment and readable storage medium Download PDFInfo
- Publication number
- CN117076494A CN117076494A CN202311329470.8A CN202311329470A CN117076494A CN 117076494 A CN117076494 A CN 117076494A CN 202311329470 A CN202311329470 A CN 202311329470A CN 117076494 A CN117076494 A CN 117076494A
- Authority
- CN
- China
- Prior art keywords
- query
- real
- time data
- description
- communication endpoint
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000004891 communication Methods 0.000 claims abstract description 112
- 238000012549 training Methods 0.000 claims abstract description 49
- 230000000875 corresponding effect Effects 0.000 claims description 81
- 239000013598 vector Substances 0.000 claims description 35
- 238000004590 computer program Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 5
- 238000003058 natural language processing Methods 0.000 claims description 4
- 230000002596 correlated effect Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 14
- 230000006870 function Effects 0.000 description 8
- 230000008859 change Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 239000013618 particulate matter Substances 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000002245 particle Substances 0.000 description 2
- 238000007637 random forest analysis Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- CBENFWSGALASAD-UHFFFAOYSA-N Ozone Chemical compound [O-][O+]=O CBENFWSGALASAD-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000000356 contaminant Substances 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 239000010419 fine particle Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2452—Query translation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24549—Run-time optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/211—Selection of the most significant subset of features
- G06F18/2113—Selection of the most significant subset of features by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Remote Sensing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a method, a device, computer equipment and a readable storage medium for inquiring real-time data, which relate to the technical field of data processing, wherein the method comprises the following steps: setting a plurality of communication endpoints, constructing query descriptions corresponding to each communication endpoint, wherein the query descriptions are used for defining data objects and/or regional ranges corresponding to real-time data allowed to be queried by each communication endpoint; receiving an input query request for real-time data, wherein the query request comprises a current query statement of the real-time data; the method comprises the steps that a current query statement is respectively matched with query descriptions corresponding to a plurality of communication endpoints through an intention model, a communication endpoint corresponding to the query description with the highest matching degree is determined, and the intention model is obtained by training a large language model by taking historical query statements and query descriptions as samples; and calling the interface corresponding to the determined communication endpoint to perform real-time data query, and outputting a query result of the query request. The scheme can improve the accuracy and efficiency of real-time data query.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and apparatus for querying real-time data, a computer device, and a readable storage medium.
Background
Large models are widely used in various fields, but conventional large models have certain difficulties in processing real-time data. Since large models typically require offline training and batch inference, changes in real-time data cannot be accommodated in time.
The current method for combining the large model with real-time data is as follows:
(1) Judging and calling a real-time data API by semantic understanding: in a large model, a set of rules or conditions are designed, input is analyzed through semantic understanding, and whether a real-time data API needs to be called is judged according to the rules. These rules may be based on keywords, grammatical structures, or other specific semantic patterns. If the input meets the rule, a trigger is made to call the real-time data API.
However, large models may face difficulties in understanding and interpreting inputs when doing semantic understanding and judgment. Despite the tremendous advances made by modern language models in semantic understanding, there are challenges such as ambiguity, complex contextual understanding, etc., which can result in limited or erroneous understanding of the actual intent by the large model, which in turn invokes the wrong API.
(2) Search engine results generalization: the large model may search and analyze real-time data through a query interface provided by a search engine. According to specific requirements, the large model can utilize the powerful query capability of the search engine to extract key information, find patterns, conduct statistical analysis and the like from real-time data.
However, in the case of a method of summarizing the search engine results by using a large model, under the condition that the update frequency of real-time data is high, the search engine may not be able to process all the real-time data in time, thereby causing lag or incompleteness of information. In addition, real-time data often has high variability and noise, resulting in a tendency to missummarize inaccurate or incomplete data
(3) On-line training of a large model: based on the existing model, new real-time data is gradually introduced to perform incremental training, so that the model can adapt to the change of the new data. Incremental learning may utilize online learning algorithms, such as online gradient descent or random gradient descent, to update the model in real time.
However, with online training, more time and computing resources are required, and the training process becomes slow when the amount of real-time data is large or the update frequency is high. Moreover, online training involves the transmission and use of real-time data, with particular attention to the privacy and security issues of the data.
Thus, the problem of implementing accurate real-time data queries based on large models is currently unsolved.
Disclosure of Invention
In view of the above, the embodiment of the application provides a real-time data query method, so as to solve the technical problem that the real-time data cannot be accurately queried based on a large model in the prior art. The method comprises the following steps:
setting a plurality of communication endpoints, and constructing query descriptions corresponding to each communication endpoint, wherein the query descriptions are used for defining data objects and/or regional ranges corresponding to real-time data allowed to be queried by each communication endpoint;
receiving an input query request for real-time data, wherein the query request comprises a current query statement of the real-time data;
matching the current query statement with query descriptions corresponding to a plurality of communication endpoints through an intention model, and determining the communication endpoint corresponding to the query description with the highest matching degree, wherein the intention model is obtained by training a large language model by taking historical query statements and query descriptions as samples;
and calling the determined interface corresponding to the communication endpoint to perform real-time data query, and outputting a query result of the query request.
The embodiment of the application also provides a real-time data query device, which solves the technical problem that the real-time data cannot be accurately queried based on a large model in the prior art. The device comprises:
the construction module is used for setting a plurality of communication endpoints and constructing query descriptions corresponding to each communication endpoint, wherein the query descriptions are used for defining data objects and/or regional ranges corresponding to real-time data which each communication endpoint is allowed to query;
the receiving module is used for receiving an input query request for the real-time data, wherein the query request comprises a current query statement of the real-time data;
the matching module is used for respectively matching the current query statement with query descriptions corresponding to a plurality of communication endpoints through an intention model, and determining the communication endpoint corresponding to the query description with the highest matching degree, wherein the intention model is obtained by training a large language model by taking historical query statements and query descriptions as samples;
and the query module is used for calling the interface corresponding to the determined communication endpoint to perform real-time data query and outputting the query result of the query request.
The embodiment of the application also provides computer equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the random real-time data query method when executing the computer program so as to solve the technical problem that the prior art can not accurately query real-time data based on a large model.
The embodiment of the application also provides a computer readable storage medium which stores a computer program for executing the random real-time data query method, so as to solve the technical problem that the real-time data can not be accurately queried based on a large model in the prior art.
Compared with the prior art, the beneficial effects that above-mentioned at least one technical scheme that this description embodiment adopted can reach include at least: setting a plurality of communication endpoints, and constructing query descriptions corresponding to each communication endpoint to define data objects and/or regional ranges corresponding to real-time data allowed to be queried by each communication endpoint; further, matching current query sentences of real-time data in the query request with query descriptions corresponding to a plurality of communication endpoints respectively through an intention model, and determining the communication endpoint corresponding to the query description with the highest matching degree; and finally, calling the interface corresponding to the determined communication endpoint to perform real-time data query, and outputting the query result of the query request. The method has the advantages that the query of real-time data is realized based on the intention model in the form of the communication endpoints and the large language model, the intention model can directly match the current query statement of the real-time data with query descriptions corresponding to a plurality of communication endpoints respectively to determine the communication endpoint with the highest matching degree, the query statement in the input query problem is not required to be split into a plurality of terms to be matched, so that misunderstanding or errors caused by semantic understanding and term splitting can be avoided, further, the accuracy of the matching is improved, and the accuracy of the real-time data query is also improved; meanwhile, by constructing a plurality of communication endpoints with a real-time data query function, the real-time data query can be realized based on the communication endpoint which is most matched with a query statement or a query requirement in a query request, the search engine in the prior art is avoided to query the real-time data, the problem that the search engine can not timely process all real-time data of index under the condition that the update frequency of the real-time data is higher is avoided, and therefore the problems of information lag or incompleteness and the like can be avoided, and the accuracy of the real-time data query is further facilitated to be improved; in addition, because the query descriptions corresponding to different communication endpoints are different, the communication endpoints with the best matching of different query requests are different, the data query is carried out by respectively distributing or dispersing the different query requests to the different communication endpoints, the parallel query of the data is indirectly realized, the efficiency of real-time data query is improved, meanwhile, the number of the communication endpoints is not limited, the number of the communication endpoints can be adjusted (added or reduced) according to the dynamic change of the real-time data, the query requirements of different real-time data can be dynamically met, the dynamic adaptability of the real-time data query is improved, and the expansibility and the flexibility of the query method are improved; in addition, the intention model is obtained by training a large language model by taking historical query sentences and query descriptions as samples, real-time data is not required to be introduced into the large language model, and the intention model is prevented from being trained on line by adopting the real-time data, so that the training efficiency of the intention model can be improved, and the occupied computing resources can be reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for querying real-time data according to an embodiment of the present application;
FIG. 2 is a block diagram of a computer device according to an embodiment of the present application;
fig. 3 is a block diagram of a real-time data query device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in detail below with reference to the accompanying drawings.
Other advantages and effects of the present application will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present application with reference to specific examples. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. The application may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present application. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
In an embodiment of the present application, a method for querying real-time data is provided, as shown in fig. 1, where the method includes:
step S101: setting a plurality of communication endpoints, and constructing query descriptions corresponding to each communication endpoint, wherein the query descriptions are used for defining data objects and/or regional ranges corresponding to real-time data allowed to be queried by each communication endpoint;
step S102: receiving an input query request for real-time data, wherein the query request comprises a current query statement of the real-time data;
step S103: matching the current query statement with query descriptions corresponding to a plurality of communication endpoints through an intention model, and determining the communication endpoint corresponding to the query description with the highest matching degree, wherein the intention model is obtained by training a large language model by taking historical query statements and query descriptions as samples;
step S104: and calling the determined interface corresponding to the communication endpoint to perform real-time data query, and outputting a query result of the query request.
As can be seen from the flow shown in fig. 1, in the embodiment of the present application, real-time data query is realized based on the communication endpoints and intent models in the form of large models, the intent models can directly match query sentences with query descriptions corresponding to a plurality of communication endpoints to determine the communication endpoint with the highest matching degree, and the query sentences in the input query questions do not need to be split into a plurality of terms to be matched, so that misunderstanding or errors caused by semantic understanding and term splitting can be avoided, meaning and intent of the query questions can be determined more accurately, further, matching accuracy is improved, and real-time data query accuracy is also improved; meanwhile, by constructing a plurality of communication endpoints with a real-time data query function, the real-time data query can be realized based on the communication endpoint which is most matched with a query statement or a query requirement in a query request, the search engine in the prior art is avoided to query the real-time data, the problem that the search engine can not timely process all real-time data of index under the condition that the update frequency of the real-time data is higher is avoided, and therefore the problems of information lag or incompleteness and the like can be avoided, and the accuracy of the real-time data query is further facilitated to be improved; in addition, because the query descriptions corresponding to different communication endpoints are different, the communication endpoints with the best matching of different query requests are different, the data query is realized by respectively distributing or dispersing the different query requests to the different communication endpoints, the parallel query of the data is indirectly realized, the efficiency of real-time data query is improved, meanwhile, the number of the communication endpoints is not limited, the number of the communication endpoints can be adjusted (added or reduced) according to the dynamic change of the real-time data, the query requirements of different real-time data can be dynamically met, the dynamic adaptability of the real-time data query is improved, and the expansibility and the flexibility of the query method are improved.
In particular, the communication endpoint (endpoint) is an endpoint with a real-time data query function, i.e. an interface for communication interception.
In particular, the real-time data (RTD) refers to information that is transmitted immediately after collection.
In the implementation, in the process of setting a plurality of communication endpoints, the number of the communication endpoints can be determined according to the number of the actual query data objects and/or the number of the query areas, each communication endpoint corresponds to one query description, and the query descriptions among different communication endpoints are different. Specifically, the data object may be a data type or a name of a different substance (or a different object), the region range may be a description of a size of a region (for example, a circular region with a preset length as a radius around a certain center, or a name of a region of a city, town, etc.), or a place name, etc., and the query description may be a query statement including the data object and/or the region range, for example, query information about a certain object of a certain region range. For example, the data object is exemplified by a contaminant, and the query description may be in the form of: real-time PM2.5 concentration of a city (e.g., beijing, tianjin, shanghai, etc.), explaining what is fine particulate matter, or O3 concentration of a street, village, etc.
In a specific implementation, in order to implement a query of real-time data by matching an accurate communication endpoint for each query statement, in this embodiment, it is proposed to construct a query description corresponding to each communication endpoint by:
collecting historical query sentences; for example, historical query statement (or problem of historical query, query input) data may be collected, obtained from various data sources, and the data may be cleaned, annotated, etc., to ensure the quality and accuracy of the collected data. Alternatively, the historical query statement data may be obtained directly from an existing dataset. The obtained historical query statement data should cover the query questions of different question statements of various types of data (or data objects).
Extracting nouns from the historical query sentences and the real-time data related to the query to obtain noun characteristics;
converting the obtained noun features to obtain a plurality of feature vectors;
selecting partial feature vectors from the plurality of feature vectors according to a preset correlation threshold and a preset occurrence frequency to form a feature group;
for each feature group, forming a query description by noun features corresponding to a plurality of feature vectors according to a grammar relationship, so as to obtain a plurality of query descriptions;
mapping each query description with each communication endpoint one by one respectively to construct the query description corresponding to each communication endpoint.
In a specific implementation, in order to improve accuracy of query description, in this embodiment, the real-time data related to the query may include: and at least one item of query response data corresponding to the historical query statement and stored real-time data for query. The noun can be extracted according to the historical query statement and the corresponding query response data (i.e. the expected output or response data of the historical query statement) to obtain noun characteristics, or the noun can be extracted according to the historical query statement and the stored real-time data for query (i.e. the stored real-time data for query in the database) to obtain noun characteristics, and the noun characteristics are selected based on the relevance so as to obtain query description.
In a specific implementation, in order to improve accuracy and effectiveness of query description, in this embodiment, it is proposed to extract nouns with high attention and relevance to construct a query description by:
extracting a plurality of nouns from the historical query statement and the real-time data related to the query, calculating the similarity value between each noun and each other noun to obtain a plurality of similarity values, and accumulating the plurality of similarity values to obtain a sum which is used as a comprehensive similarity value of the noun;
determining the attention weight of each noun according to the comprehensive similarity value of each noun, wherein the magnitude of the attention weight is positively correlated with the magnitude of the comprehensive similarity value, and the larger the attention weight of a noun is, the larger the association degree of the noun and other nouns is;
and extracting nouns with the attention weight larger than a preset threshold as the noun characteristics.
Specifically, the similarity value between different terms can be calculated by a measurement method such as cosine similarity or dot product.
Specifically, after the attention weights of all nouns are normalized, the sum of all normalized attention weights is ensured to be 1. The attention weights may be normalized using a Softmax function.
In the implementation, in the process of selecting part of feature vectors from a plurality of feature vectors to form a feature group according to a preset relevance threshold, the most relevant (for example, the noun feature with the smallest variance or the noun feature with the largest relevance coefficient) or the most important noun feature can be selected by a statistical index (such as variance and relevance coefficient) or a machine learning model (such as random forest and LASSO regression) and other methods, so that the dimension reduction of the feature is realized, and the complexity of query description is reduced under the condition of ensuring the accuracy of query description.
In particular, the different query descriptions may be different data objects and/or different ranges, for example, the query description 1 may be: inquiring the concentration of the particles in the city A; the query description 2 may be: inquiring the concentration of the particles in the city B; the query description 3 may be: query O of A City 3 Concentration.
In the implementation, in the process of training the intention model, the obtained historical query statement and all query descriptions can be formed into a training data set to train a large language model so as to obtain the intention model, for example, the historical query statement and the query descriptions corresponding to all communication endpoints are integrated into the training data set; checking the training data set; and training a first large language model by adopting the training data set to obtain the intention model.
In particular, in order to further improve the accuracy of the intent model, in this embodiment, a method of screening data and constructing a high-accuracy training data set to train a large language model to obtain the intent model is provided, where the method of screening data and constructing a high-accuracy training data set is as follows: inputting the training data set into a second large language model, respectively matching each historical query statement with each query description through a natural language processing algorithm of the second large language model, determining the query description with the highest matching degree with each historical query statement, forming one sample data by each historical query statement and the query description with the highest matching degree with each historical query statement, and integrating all sample data into the training data set.
In specific implementation, for example, the query description has the following 3 terms, query description 1: inquiring the real-time PM2.5 concentration of a certain city; query description 2: explain what is fine particulate matter; query description 3: querying a certain street and town O3 concentration, the large language model is used to determine a query description that best matches each historical query statement from multiple query descriptions to form a training data set, for example, the large language model may implement the following functions:
q (input): "what is the best match for PM2.5 of Nanjing today (i.e., historical query statement? The method comprises the steps of carrying out a first treatment on the surface of the
A (output): query description 1;
q: what is the best match for "new village street SO 2"? The method comprises the steps of carrying out a first treatment on the surface of the
A: query description 3.
In specific implementation, the natural language processing algorithm of the large language model can perform word segmentation on the historical query sentences and the query descriptions, perform word segmentation matching, and determine the matching degree of each historical query sentence and each query description respectively by combining the context understanding of each word segmentation, so as to determine one query description with the highest matching degree with each historical query sentence.
In the implementation, in the process of matching the current query statement and the query description through the intention model, the current query statement and the query description can be directly matched in the form of a statement or a whole sentence; in order to accurately and efficiently match the communication endpoint with the highest matching degree with each query statement, it is proposed to match the current query statement with the query description in the form of a similarity value, for example, the intention model calculates the similarity value between the current query statement and the query description corresponding to a plurality of communication endpoints respectively to obtain a plurality of similarity values; and determining the query description corresponding to the maximum value in the calculated multiple similarity values as the query description with the highest matching degree, and determining the communication endpoint corresponding to the query description with the highest matching degree.
In the implementation, in the process of calculating the similarity value between the current query statement and the query descriptions corresponding to the communication endpoints through the intention model, in order to further improve the precision of the intention model and avoid the precision influence caused by semantic understanding and word splitting, it is proposed that the similarity calculation is performed on the current query statement and the query descriptions in the form of integral vectors, for example, in the intention model, the current query statement is encoded to obtain a query vector; respectively encoding query descriptions corresponding to each communication endpoint to obtain description vectors; and calculating similarity values between each query vector and each description vector respectively to obtain a plurality of similarity values.
In specific implementation, in the process of calculating the similarity value between the current query vector and each description vector, the similarity value between the current query vector and the description vector can be calculated by adopting methods such as cosine similarity or dot product.
In particular implementations, the following describes a process for constructing a communication endpoint and a corresponding query description, the process including the steps of:
step one: data collection and preparation: first, data for creating an endpoint is collected and prepared. Data can be obtained from a variety of data sources and cleaned and annotated to ensure the quality and accuracy of the data. The acquired data should cover various historical query questions (or historical query statements) and corresponding expected output or historical response data.
Step two: a plurality of end points and corresponding query descriptions are constructed. When the query description is constructed, nouns can be extracted from historical query sentences and corresponding historical response data to obtain noun features, the obtained noun features are converted to obtain a plurality of feature vectors, the most relevant or important partial name feature vectors can be selected from the feature vectors to form feature groups based on statistical indexes (such as variance and correlation coefficients), machine learning models (such as random forest and LASSO regression) and other methods to obtain a plurality of feature groups, and for each feature group, noun features corresponding to the feature vectors are formed into a query description according to a grammatical relation to obtain a plurality of query descriptions, so that complexity of the query description is reduced under the condition that the accuracy of the query description is ensured.
Examples: the query corresponding to Endpoint1 is described as: inquiring the ozone concentration of a certain city. The query corresponding to Endpoint2 is described as: inquiring the concentration of fine particles in certain street and villages and towns.
Step three: training an intent large model (i.e., the intent model described above). And (3) constructing a prompt set (namely a model training data set) according to the data obtained in the step one, and inputting the prompt set into the large model for training to obtain the large-scale model. The method comprises the steps of forming a training sample in a prompt set by each historical query question and the most matched query description, wherein the training sample consists of n query descriptions and the historical query questions, and the specific format can be as follows:
query description there are three items:
query description 1: inquiring the real-time PM2.5 concentration of a certain city;
query description 2: explain what is fine particulate matter;
query description 3: inquiring the concentration of O3 in a certain street and village;
the query description that each historical query statement matches best is determined by a large language model, which may be formatted as follows:
q (input): "what is the best match for PM2.5 of Nanjing today (i.e., historical query statement? The method comprises the steps of carrying out a first treatment on the surface of the
A (output): query description 1;
q: what is the best match for "new village street SO 2"? The method comprises the steps of carrying out a first treatment on the surface of the
A: query description 3;
training a large model with the training data set to obtain the intent model, when query question Q "Beijing today's PM2.5" is the best match to the intent model? The Intent big model will give an answer to query description 1.
Step four: in the process of carrying out real-time data query by applying the intention model, the intention model outputs query descriptions which are most matched with each query statement in real time, and then an endpoint function corresponding to the most matched query descriptions is called. And selecting the endpoint with the highest similarity with each query statement, and calling a corresponding interface thereof to realize real-time data query.
In this embodiment, a computer device is provided, as shown in fig. 2, including a memory 201, a processor 202, and a computer program stored on the memory and capable of running on the processor, where the processor implements any of the real-time data query methods described above when executing the computer program.
In particular, the computer device may be a computer terminal, a server or similar computing means.
In the present embodiment, a computer-readable storage medium storing a computer program for executing the above-described arbitrary real-time data query method is provided.
In particular, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, may be used to implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable storage media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
Based on the same inventive concept, the embodiment of the application also provides a real-time data query device, as described in the following embodiment. Because the principle of solving the problem of the real-time data query device is similar to that of the real-time data query method, the implementation of the real-time data query device can refer to the implementation of the real-time data query method, and the repetition is omitted. As used below, the term "unit" or "module" may be a combination of software and/or hardware that implements the intended function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
Fig. 3 is a block diagram of a real-time data query device according to an embodiment of the present application, and as shown in fig. 3, the device includes:
the building module 301 is configured to set a plurality of communication endpoints, and build a query description corresponding to each communication endpoint, where the query description is used to define a data object and/or a region range corresponding to real-time data allowed to be queried by each communication endpoint;
a receiving module 302, configured to receive an input query request for real-time data, where the query request includes a current query statement of the real-time data;
the matching module 303 is configured to match the current query statement with query descriptions corresponding to a plurality of communication endpoints through an intent model, and determine the communication endpoint corresponding to the query description with the highest matching degree, where the intent model is obtained by training a large language model by using historical query statements and query descriptions as samples;
and the query module 304 is configured to invoke the interface corresponding to the determined communication endpoint to perform real-time data query, and output a query result of the query request.
In one embodiment, the matching module includes:
the similarity calculation unit is used for calculating similarity values between the current query statement and query descriptions corresponding to a plurality of communication endpoints respectively through the intention model to obtain a plurality of similarity values;
and the matching unit is used for determining the query description corresponding to the maximum value in the plurality of calculated similarity values as the query description with the highest matching degree, and determining the communication endpoint corresponding to the query description with the highest matching degree.
In one embodiment, the similarity calculation unit is configured to encode the current query sentence in the intent model to obtain a query vector; encoding the query description corresponding to each communication endpoint to obtain a description vector; and calculating similarity values between each query vector and each description vector respectively to obtain a plurality of similarity values.
In one embodiment, a building module is used for collecting historical query sentences; extracting nouns from the historical query sentences and the real-time data related to the query to obtain noun characteristics; converting the obtained noun characteristics to obtain a plurality of characteristic vectors; selecting partial feature vectors from the plurality of feature vectors according to a preset correlation threshold and a preset occurrence frequency to form a feature group; for each feature group, forming a noun feature corresponding to a plurality of feature vectors into one query description according to a grammar relationship so as to obtain a plurality of query descriptions; mapping each query description with each communication endpoint one by one respectively to construct the query description corresponding to each communication endpoint.
In one embodiment, the building module is further configured to extract a plurality of nouns from the historical query statement and the real-time data related to the query, calculate a similarity value between each noun and each other noun, obtain a plurality of similarity values, and accumulate a sum of the plurality of similarity values to be used as a comprehensive similarity value of the noun; determining the attention weight of each noun according to the comprehensive similarity value of each noun, wherein the magnitude of the attention weight is positively correlated with the magnitude of the comprehensive similarity value, and the larger the attention weight of the noun is, the larger the association degree of the noun with other nouns is; and extracting nouns with the attention weight larger than a preset threshold as the noun characteristics.
In one embodiment, the apparatus further comprises:
the data set integration module is used for integrating the historical query statement and the query description corresponding to the communication endpoint into a training data set;
a data checking module for checking the training data set;
and the training module is used for training the first large language model by adopting the training data set to obtain the intention model.
In one embodiment, the data set integrating module is configured to input the training data set into a second large language model, match each of the historical query sentences with each of the query descriptions through a natural language processing algorithm of the second large language model, determine the query description with the highest matching degree with each of the historical query sentences, combine each of the historical query sentences and the query description with the highest matching degree with each of the historical query sentences into one sample data, and integrate all the sample data into the training data set.
The embodiment of the application realizes the following technical effects: the method has the advantages that the query of real-time data is realized based on the intent model in the form of the communication endpoints and the large model, the intent model can directly match query sentences with query descriptions corresponding to a plurality of communication endpoints to determine the communication endpoint with the highest matching degree, the query sentences in the input query problems are not required to be split into a plurality of terms to be matched, so that misunderstanding or errors caused by semantic understanding and term splitting can be avoided, the meaning and intent of the query problems can be determined more accurately, the matching accuracy is improved, and the accuracy of real-time data query is improved; meanwhile, by constructing a plurality of communication endpoints with a real-time data query function, the real-time data query can be realized based on the communication endpoint which is most matched with a query statement or a query requirement in a query request, the search engine in the prior art is avoided to query the real-time data, the problem that the search engine can not timely process all real-time data of index under the condition that the update frequency of the real-time data is higher is avoided, and therefore the problems of information lag or incompleteness and the like can be avoided, and the accuracy of the real-time data query is further facilitated to be improved; in addition, because the query descriptions corresponding to different communication endpoints are different, the communication endpoints with the best matching of different query requests are different, the data query is realized by respectively distributing or dispersing the different query requests to the different communication endpoints, the parallel query of the data is indirectly realized, the efficiency of real-time data query is improved, meanwhile, the number of the communication endpoints is not limited, the number of the communication endpoints can be adjusted (added or reduced) according to the dynamic change of the real-time data, the query requirements of different real-time data can be dynamically met, the dynamic adaptability of the real-time data query is improved, and the expansibility and the flexibility of the query method are improved.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the application described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may alternatively be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than what is shown or described, or they may be separately fabricated into individual integrated circuit modules, or a plurality of modules or steps in them may be fabricated into a single integrated circuit module. Thus, embodiments of the application are not limited to any specific combination of hardware and software.
The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, and various modifications and variations can be made to the embodiments of the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (11)
1. A method for querying real-time data, comprising:
setting a plurality of communication endpoints, and constructing query descriptions corresponding to each communication endpoint, wherein the query descriptions are used for defining data objects and/or regional ranges corresponding to real-time data allowed to be queried by each communication endpoint;
receiving an input query request for real-time data, wherein the query request comprises a current query statement of the real-time data;
matching the current query statement with query descriptions corresponding to a plurality of communication endpoints through an intention model, and determining the communication endpoint corresponding to the query description with the highest matching degree, wherein the intention model is obtained by training a large language model by taking historical query statements and query descriptions as samples;
and calling the determined interface corresponding to the communication endpoint to perform real-time data query, and outputting a query result of the query request.
2. The method for querying real-time data according to claim 1, wherein the determining, by using an intention model, the communication endpoint corresponding to the query description with the highest matching degree by matching the current query statement with the query descriptions corresponding to the plurality of communication endpoints, respectively, includes:
calculating similarity values between the current query statement and query descriptions corresponding to a plurality of communication endpoints respectively through the intention model to obtain a plurality of similarity values;
and determining the query description corresponding to the maximum value in the calculated similarity values as the query description with the highest matching degree, and determining the communication endpoint corresponding to the query description with the highest matching degree.
3. The method of claim 2, wherein calculating, by the intent model, similarity values between the current query statement and query descriptions corresponding to the plurality of communication endpoints, respectively, to obtain a plurality of similarity values, includes:
encoding the current query statement in the intention model to obtain a query vector; encoding the query description corresponding to each communication endpoint to obtain a description vector;
and calculating similarity values between each query vector and each description vector respectively to obtain a plurality of similarity values.
4. The method for querying real-time data according to claim 1, wherein said constructing a query description corresponding to each of said communication endpoints comprises:
collecting historical query sentences;
extracting nouns from the historical query sentences and the real-time data related to the query to obtain noun characteristics;
converting the obtained noun characteristics to obtain a plurality of characteristic vectors;
selecting partial feature vectors from the plurality of feature vectors according to a preset correlation threshold and a preset occurrence frequency to form a feature group;
for each feature group, forming a query description by noun features corresponding to a plurality of feature vectors according to a grammatical relation;
mapping each query description with each communication endpoint one by one respectively to construct the query description corresponding to each communication endpoint.
5. The method for querying real-time data as recited in claim 4, wherein extracting nouns from the historical query statement and query-related real-time data to obtain noun features comprises:
extracting a plurality of nouns from the historical query statement and the real-time data related to the query, calculating the similarity value between each noun and each other noun to obtain a plurality of similarity values, and accumulating the plurality of similarity values to obtain a sum which is used as a comprehensive similarity value of the noun;
determining the attention weight of each noun according to the comprehensive similarity value of each noun, wherein the magnitude of the attention weight is positively correlated with the magnitude of the comprehensive similarity value, and the larger the attention weight of each noun is, the larger the association degree of the noun and other nouns is;
and extracting nouns with the attention weight larger than a preset threshold as the noun characteristics.
6. The method for querying real-time data as recited in claim 4, wherein querying the related real-time data comprises: and at least one item of query response data corresponding to the historical query statement and stored real-time data for query.
7. The method for querying real-time data as recited in claim 6, further comprising:
integrating the historical query statement and the query description corresponding to the communication endpoint into a training data set;
checking the training data set;
and training a first large language model by adopting the training data set to obtain the intention model.
8. The method for querying real-time data according to claim 7, wherein integrating the historical query statement and the query description corresponding to the communication endpoint into a training data set comprises:
inputting the training data set into a second large language model, respectively matching each historical query statement with each query description through a natural language processing algorithm of the second large language model, determining the query description with the highest matching degree with each historical query statement, forming one sample data by each historical query statement and the query description with the highest matching degree with each historical query statement, and integrating all sample data into the training data set;
and training a large language model by adopting the training data set to obtain the intention model.
9. A real-time data query device, comprising:
the construction module is used for setting a plurality of communication endpoints and constructing query descriptions corresponding to each communication endpoint, wherein the query descriptions are used for defining data objects and/or regional ranges corresponding to real-time data which each communication endpoint is allowed to query;
the receiving module is used for receiving an input query request for real-time data, wherein the query request comprises a current query statement for the real-time data;
the matching module is used for respectively matching the current query statement with query descriptions corresponding to a plurality of communication endpoints through an intention model, and determining the communication endpoint corresponding to the query description with the highest matching degree, wherein the intention model is obtained by training a large language model by taking historical query statements and query descriptions as samples;
and the query module is used for calling the interface corresponding to the determined communication endpoint to perform real-time data query and outputting the query result of the query request.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of querying real-time data according to any of claims 1 to 8 when executing the computer program.
11. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program that performs the real-time data query method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311329470.8A CN117076494A (en) | 2023-10-16 | 2023-10-16 | Real-time data query method and device, computer equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311329470.8A CN117076494A (en) | 2023-10-16 | 2023-10-16 | Real-time data query method and device, computer equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117076494A true CN117076494A (en) | 2023-11-17 |
Family
ID=88717486
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311329470.8A Pending CN117076494A (en) | 2023-10-16 | 2023-10-16 | Real-time data query method and device, computer equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117076494A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7120627B1 (en) * | 2000-04-26 | 2006-10-10 | Global Information Research And Technologies, Llc | Method for detecting and fulfilling an information need corresponding to simple queries |
| CN111897836A (en) * | 2020-07-03 | 2020-11-06 | 中国建设银行股份有限公司 | Search system, method and storage medium |
| CN113641796A (en) * | 2021-08-30 | 2021-11-12 | 平安医疗健康管理股份有限公司 | Data searching method, system and storage medium |
| CN116341007A (en) * | 2023-03-29 | 2023-06-27 | 阿里巴巴（中国）有限公司 | Data query method, system, equipment and storage medium |
-
2023
- 2023-10-16 CN CN202311329470.8A patent/CN117076494A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7120627B1 (en) * | 2000-04-26 | 2006-10-10 | Global Information Research And Technologies, Llc | Method for detecting and fulfilling an information need corresponding to simple queries |
| CN111897836A (en) * | 2020-07-03 | 2020-11-06 | 中国建设银行股份有限公司 | Search system, method and storage medium |
| CN113641796A (en) * | 2021-08-30 | 2021-11-12 | 平安医疗健康管理股份有限公司 | Data searching method, system and storage medium |
| CN116341007A (en) * | 2023-03-29 | 2023-06-27 | 阿里巴巴（中国）有限公司 | Data query method, system, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020143184A1 (en) | Knowledge fusion method and apparatus, computer device, and storage medium | |
| CN111581949B (en) | Method and device for disambiguating name of learner, storage medium and terminal | |
| Chatterjee et al. | Intent mining from past conversations for conversational agent | |
| WO2019144728A1 (en) | Data processing | |
| CN111274267A (en) | Database query method and device and computer readable storage medium | |
| EP2634705A1 (en) | Method for discovering relevant concepts in a semantic graph of concepts | |
| CN111694940A (en) | User report generation method and terminal equipment | |
| Wu et al. | Extracting topics based on Word2Vec and improved Jaccard similarity coefficient | |
| CN102123172A (en) | Implementation method of Web service discovery based on neural network clustering optimization | |
| CN116304748B (en) | Text similarity calculation method, system, equipment and medium | |
| CN110727769B (en) | Corpus generation method and device and man-machine interaction processing method and device | |
| CN116306504B (en) | Candidate entity generation method and device, storage medium and electronic equipment | |
| CN110909126A (en) | Information query method and device | |
| CN115017425B (en) | Location search method, location search device, electronic device, and storage medium | |
| CN115795030A (en) | Text classification method and device, computer equipment and storage medium | |
| CN113806492B (en) | Record generation method, device, equipment and storage medium based on semantic recognition | |
| CN113610626A (en) | Bank credit risk identification knowledge graph construction method and device, computer equipment and computer readable storage medium | |
| CN117076494A (en) | Real-time data query method and device, computer equipment and readable storage medium | |
| CN114579883A (en) | Address query method, method for obtaining address vector representation model and corresponding device | |
| CN114595334A (en) | Language analysis method and system based on double-graph-spectrum fusion and terminal equipment | |
| Wang et al. | A graph-based approach for semantic similar word retrieval | |
| CN111898389B (en) | Information determination method, information determination device, computer equipment and storage medium | |
| CN117556050B (en) | Data classification and classification method and device, electronic equipment and storage medium | |
| CN117972113B (en) | Patent authorization prediction evaluation method and system based on attribute knowledge graph | |
| CN117033724B (en) | Multi-mode data retrieval method based on semantic association |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |