CN118012889A - Vector determination method and device, storage medium and electronic device - Google Patents

Abstract

The invention discloses a vector determination method and device, a storage medium and an electronic device, wherein the method comprises the following steps: receiving a vector retrieval request, wherein the vector retrieval request carries: a first vector and first structured data, wherein the first structured data is used for indicating attribute information corresponding to the first vector; splicing the first structured data to obtain spliced first structured data; controlling an open source vector search engine packaged in a database to determine a target vector table partition according to a plurality of vector table partitions of the spliced first structured data in the database; and determining a second vector in the target vector table partition according to the first vector, wherein the similarity between the second vector and the first vector is greater than or equal to a preset threshold value. By adopting the technical scheme, the problem that more index algorithms with better retrieval effect cannot be supported in the existing database is solved.

Description

Vector determination method and device, storage medium and electronic device

Technical Field

The present invention relates to the field of computers, and in particular, to a method and apparatus for determining a vector, a storage medium, and an electronic apparatus.

Background

With the continuous development of artificial intelligence technology, the application of an AI large model has a fierce growth situation. For AI large models, all concepts can be represented by vectors, while vector space has some good mathematical properties that can support the calculation of the "distance" between vectors. This means that the correlation between any two abstractions can be measured in terms of the distance of the corresponding vector. This function can be applied in the search. For example, documents in the knowledge base may be modeled into abstract vectors, and only documents "closest" to the query vector need be found for retrieval as answers and returned to the user. And a vector database is needed to support such a retrieval process.

There are many vector database products currently on the market, for example: commercial pine cones AI (Pinecone AI), pinecone AI are a simple and easy-to-use hosted vector database service that provides a completely hosted, server-free environment for real-time vector similarity search and recommendation systems that reduces the operational burden. Pinecone is used as a commercial vector database, and although the free space provided can meet most of the situations, the cost starting point is higher in the production environment of the outsole warehouse relative to other products. Mi Erwei s (Milvus) of an open source, milvus is an open source vector similarity search engine, supports various similarity measurement standards and has high expandability. Milvus although well known in the domestic market, the ecosystem of Milvus is relatively weak and lacks compatibility with other database products. Based on the vector diagram (pgvector) provided by the relational database in the plug-in form, pgvector selects to provide services on the open source relational database Postgre SQL in the plug-in form, so that the method is simple to operate, excellent in performance and inherits the ecological set of PostgreSQL. pgvector as a plug-in to the database management system (PostgreSQL Database MANAGEMENT SYSTEM, simply PostgreSQL) supports vectors very well, but the indexing algorithm supported by pgvector is limited and does not perform any quantization or storage optimization.

Aiming at the problem that more index algorithms with better retrieval effect cannot be supported in the existing database, no effective solution is proposed at present.

Accordingly, there is a need for improvements in the related art to overcome the drawbacks of the related art.

Disclosure of Invention

The embodiment of the invention provides a vector determining method and device, a storage medium and an electronic device, which aim to solve the problem that more index algorithms with better retrieval effect cannot be supported in at least the existing database.

According to an aspect of an embodiment of the present invention, there is provided a method for determining a vector, including: receiving a vector retrieval request, wherein the vector retrieval request carries: a first vector and first structured data, wherein the first structured data is used for indicating attribute information corresponding to the first vector; splicing the first structured data to obtain spliced first structured data; controlling an open source vector search engine packaged in a database to determine a target vector table partition according to a plurality of vector table partitions of the spliced first structured data in the database; and determining a second vector in the target vector table partition according to the first vector, wherein the similarity between the second vector and the first vector is greater than or equal to a preset threshold value.

In an exemplary embodiment, controlling the open source vector search engine to determine a target vector table partition according to the plurality of vector table partitions in the database of the spliced first structured data includes: controlling the open source vector search engine to compare the spliced first structured data with the identification information of the plurality of vector table partitions so as to obtain a comparison result; determining first identification information consistent with the spliced first structured data according to the comparison result; and determining the vector table partition corresponding to the first identification information as the target vector table partition.

In one exemplary embodiment, controlling an open source vector search engine encapsulated in a database to determine a target vector table partition from a plurality of vector table partitions in the database of the spliced first structured data comprises: packaging the interface of the open source vector search engine into a static library file; packaging the generated static library file and the header file of the interface into a port of a packaging layer of the database; and calling a port of a packaging layer of the database to control an open source vector search engine packaged in the database to determine a target vector table partition according to the plurality of vector table partitions of the spliced first structured data in the database.

In an exemplary embodiment, the method further comprises: receiving a vector insertion request, wherein the vector retrieval request carries: a third vector and second structured data, wherein the second structured data is used for indicating attribute information corresponding to the third vector; parsing the vector insertion request to obtain the third vector and the second structured data; and determining a storage area of the third vector according to the second structured data.

In an exemplary embodiment, determining the storage area of the third vector from the second structured data includes: splicing the second structured data to obtain spliced second structured data; determining whether second identification information consistent with the spliced second structured data exists in the identification information of the vector table partitions; controlling the open source vector search engine to determine a storage area of the third vector according to a determination result, wherein the determination result is used for indicating one of the following: second identification information consistent with the spliced second structured data exists in the identification information of the vector table partitions; and second identification information consistent with the spliced second structured data does not exist in the identification information of the vector table partitions.

In an exemplary embodiment, determining the storage area of the third vector according to the determination result includes: determining that the vector table partition corresponding to the second identification information is a storage area of the third vector when the determination result indicates that the second identification information consistent with the spliced second structured data exists in the identification information of the plurality of vector table partitions; and under the condition that the determining result indicates that second identification information consistent with the spliced second structured data does not exist in the identification information of the plurality of vector table partitions, establishing a target vector table partition according to the spliced second structured data, and determining that the target vector table partition is a storage area of the third vector.

In one exemplary embodiment, after determining the second vector in the target vector table partition from the first vector, the method further comprises: determining a data type identifiable by the database, and converting the second vector into a third vector corresponding to the data type; and sending a vector retrieval result to terminal equipment according to the third vector, wherein the terminal equipment is equipment for sending the vector retrieval request.

According to another aspect of the embodiment of the present invention, there is also provided a vector determination apparatus, including: the receiving module is configured to receive a vector search request, where the vector search request carries: a first vector and first structured data, wherein the first structured data is used for indicating attribute information corresponding to the first vector; the splicing module is used for splicing the first structured data to obtain spliced first structured data; the control module is used for controlling an open source vector search engine packaged in a database to determine a target vector table partition according to a plurality of vector table partitions in the database of the spliced first structured data; and the determining module is used for determining a second vector in the target vector table partition according to the first vector, wherein the similarity between the second vector and the first vector is greater than or equal to a preset threshold value.

According to a further aspect of embodiments of the present invention, there is also provided a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the above-described vector determination method when run.

According to still another aspect of the embodiments of the present invention, there is further provided an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the method for determining the vector by the computer program.

According to the method and the device for searching the vector, a vector search request carrying the first vector and the first structured data for indicating attribute information corresponding to the first vector is received, and an open source vector search engine packaged in a database is controlled to determine a target vector table partition in a plurality of vector table partitions of the database according to the spliced first structured data, wherein the spliced first structured data is obtained by splicing the first structured data. And further determining a second vector with the similarity to the first vector being greater than or equal to a preset threshold value in the target vector table partition according to the first vector. By adopting the technical scheme, the problem that more index algorithms with better retrieval effect cannot be supported in the existing database is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

FIG. 1 is a block diagram of the hardware architecture of a computer device of a method of vector determination according to an embodiment of the present invention;

FIG. 2 is a flow chart of a method of determining a vector according to an embodiment of the invention;

FIG. 3 is a flowchart of inserting vector data according to an exemplary embodiment of the present application;

FIG. 4 is a flowchart of a retrieval process according to an exemplary embodiment of the present application;

FIG. 5 is a schematic diagram of vector joint retrieval time consuming under different structured data according to an exemplary embodiment of the present application;

Fig. 6 is a block diagram of a configuration of a vector determination apparatus according to an embodiment of the present invention.

Detailed Description

In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.

It should be noted that the terms and "first," "second," and the like in the description and claims of the present invention and the above-described drawings are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The method embodiments provided in the embodiments of the present application may be performed in a computer device or similar computing device. Taking a computer device as an example, fig. 1 is a block diagram of a hardware configuration of a computer device of a vector determination method according to an embodiment of the present application. As shown in FIG. 1, the computer apparatus may include one or more processors 102 (only one is shown in FIG. 1) (the processor 102 may include, but is not limited to, a microprocessor (Microprocessor Unit, abbreviated MPU) or programmable logic device (Programmable logic device, abbreviated PLD) and a memory 104 for storing data. In one exemplary embodiment, the computer apparatus may further include a transmission device 106 for communication functions and an input-output device 108. As will be appreciated by those of ordinary skill in the art, the structure shown in FIG. 1 is merely illustrative and is not limiting of the structure of the computer apparatus described above. For example, the computer apparatus may also include more or fewer components than shown in FIG. 1 or have a different configuration than equivalent functions shown in FIG. 1 or more than those shown in FIG. 1.

The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a method for determining a vector in an embodiment of the present invention, and the processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, which corresponds to implementing the above-mentioned method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 104 may further include memory located remotely from processor 102, which may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communications provider of the computer device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as a NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is configured to communicate with the internet wirelessly.

In this embodiment, a method for determining a vector is provided, and fig. 2 is a flowchart of a method for determining a vector according to an embodiment of the present invention, applied to the above-mentioned computer device, where the flowchart includes the following steps:

step S202, receiving a vector search request, where the vector search request carries: the device comprises a first vector and first structured data, wherein the first structured data is used for indicating attribute information corresponding to the first vector.

It should be noted that a thing is represented by a set of numbers, and such a set of numbers is a vector. Vector retrieval refers to finding one or more vectors closest to the target vector in a given set of vectors, i.e., vector retrieval.

Step S204, splicing the first structured data to obtain spliced first structured data.

Step S206, controlling an open source vector search engine packaged in a database to determine a second vector table partition according to a plurality of vector table partitions in the database of the spliced first structured data.

Step S208, determining a second vector in the target vector table partition according to the first vector, wherein the similarity between the second vector and the first vector is greater than or equal to a preset threshold.

Through the steps, a vector retrieval request carrying a first vector and first structured data for indicating attribute information corresponding to the first vector is received, and an open source vector search engine encapsulated in a database is controlled to determine a target vector table partition in a plurality of vector table partitions of the database according to the spliced first structured data, wherein the spliced first structured data is obtained by splicing the first structured data. And further determining a second vector with the similarity to the first vector being greater than or equal to a preset threshold value in the target vector table partition according to the first vector. The method can solve the problem that more index algorithms with better retrieval effect cannot be supported in the existing database.

In order to better understand how to control the open source vector search engine encapsulated in the database in the step S206, the step S206 may be implemented by determining a second vector table partition according to the plurality of vector table partitions of the spliced first structured data in the database: controlling the open source vector search engine to compare the spliced first structured data with the identification information of the plurality of vector table partitions so as to obtain a comparison result; determining first identification information consistent with the spliced first structured data according to the comparison result; and determining the vector table partition corresponding to the first identification information as the target vector table partition.

The vector search engine in the above technical solution may be Milvus. Milvus is an open source vector database for storing and managing large-scale vector data. The method provides efficient vector similarity searching and analyzing functions, and can quickly search similar vectors in a large-scale data set.

Further, in the case that there is no first identification information consistent with the spliced first structured data, a new vector table partition may be created as the target vector table partition with the identification information corresponding to the spliced first structured data.

Optionally, controlling an open source vector search engine encapsulated in a database to determine a target vector table partition according to the plurality of vector table partitions in the database of the spliced first structured data, including: packaging the interface of the open source vector search engine into a static library file; packaging the generated static library file and the header file of the interface into a port of a packaging layer of the database; and calling a port of a packaging layer of the database to control an open source vector search engine packaged in the database to determine a target vector table partition according to the plurality of vector table partitions of the spliced first structured data in the database.

It should be noted that, in the embodiment of the present application, the application programming interfaces (Application Programming Interface, api) provided by Milvus are packaged as a static library, so as to provide operations on vectors and indexes for the development of subsequent open source projects (Multicorn).

Optionally, in the embodiment of the present application, a manner of storing a vector is further provided: receiving a vector insertion request, wherein the vector retrieval request carries: a third vector and second structured data, wherein the second structured data is used for indicating attribute information corresponding to the third vector; parsing the vector insertion request to obtain the third vector and the second structured data; and determining a storage area of the third vector according to the second structured data.

Optionally, determining the storage area of the third vector according to the second structured data includes: splicing the second structured data to obtain spliced second structured data; determining whether second identification information consistent with the spliced second structured data exists in the identification information of the vector table partitions; controlling the open source vector search engine to determine a storage area of the third vector according to a determination result, wherein the determination result is used for indicating one of the following: second identification information consistent with the spliced second structured data exists in the identification information of the vector table partitions; and second identification information consistent with the spliced second structured data does not exist in the identification information of the vector table partitions.

Specific: determining that the vector table partition corresponding to the second identification information is a storage area of the third vector when the determination result indicates that the second identification information consistent with the spliced second structured data exists in the identification information of the plurality of vector table partitions; and under the condition that the determining result indicates that second identification information consistent with the spliced second structured data does not exist in the identification information of the plurality of vector table partitions, establishing a target vector table partition according to the spliced second structured data, and determining that the target vector table partition is a storage area of the third vector.

For example, if a record is inserted into partest tables, the values of each column are respectively: 1, '{1,1}',0, 'hei', 'nan', 'zhong', the code corresponding to the received vector insertion request may be:

insertintopartestvalues(1,'{1,1}',0,'hei','nan','zhong')；

this is an SQL statement that inserts a record into a table named partest.

Wherein, "insert into partest": indicating that data is to be inserted into partest tables; "values": representing the specific value to be inserted; "(1, '{1,1}',0, 'hei', 'nan', 'zhong')": representing the specific value to be inserted, and assigning values according to the column sequence of the table; the first value of 1 will be inserted into the first column of the table; "{1,1}" is an array to be inserted into the second column of the table; 0 will be inserted into the third column of the table; "hei" will be inserted into the fourth column of the table; "nan" will be inserted into the fifth column of the table; "zhong" will be inserted in the sixth column of the table.

insert into partest(myid,sex,race,age,myvec)values(1,'nan','hei','zhong','{1,1}')；

This is an SQL statement that is used to insert a new record into the partest tables. Specific: myid is set to 1, sex is set to "nan", race is set to "hei", age is set to "zhong", myvec is set to "{1,1}".

The technical scheme of receiving the vector insertion request and analyzing the vector insertion request operation can be completed through the two lines of codes.

Optionally, after determining the second vector in the target vector table partition according to the first vector, the method further includes: determining a data type identifiable by the database, and converting the second vector into a third vector corresponding to the data type; and sending a vector retrieval result to terminal equipment according to the third vector, wherein the terminal equipment is equipment for sending the vector retrieval request.

That is, the second vector determined in the embodiment of the present application may be a data type unrecognizable by the database, and thus, once the second vector is determined, the second vector is converted into a third vector according to the data type identifiable by the database, and the third vector is transmitted to the device that transmits the vector retrieval request through the database.

In order to better understand the process of the above-mentioned vector determination method, the implementation flow of the above-mentioned vector determination method is described below in conjunction with the optional embodiments, but is not used to limit the technical solution of the embodiments of the present application.

In the currently mainstream vector databases, most vector databases are independent database systems, and although some vector databases provide an API call interface, additional learning cost is still required when the vector databases are used; the vector database plug-in integrated on the relational database cannot support more index algorithms with better retrieval effect due to the limitation of the existence form.

Therefore, the exemplary embodiment of the application provides a database implementation method supporting vector structured joint search, which not only can realize the operation of a vector database based on a traditional relational database platform, but also can realize the joint search of the vector and structured data, and further improve the search performance.

Before describing exemplary embodiments of the present application, a further explanation of vectors and vector retrieval is first required.

Vector (Vector): a thing is represented by a set of numbers, which are a vector. Illustrating: if a vector consists of n numbers, it is an n-dimensional vector. Further, taking face recognition as an example, a computer extracts an image of a face from a photograph, and then can convert the face image into a vector of 128 dimensions or more.

Vector retrieval: in a given set of vectors, one or more vectors closest to the target vector are found, i.e., vector retrieval. Whether the vectors are close depends on the distance between them, which may be cosine distance, euclidean distance, hamming distance, etc. Illustrating: assuming that a and B are used to represent two normalized n-dimensional vectors, the cosine distance between them is calculated as:

The exemplary embodiment of the present application uses a vector search engine Milvus, which comprises the following steps:

1) Packaging Milvus provided application programming interfaces (Application Programming Interface, api) as a static library, providing operations on vector and index for development of subsequent open source projects (Multicorn), part of the interface code being as follows:

void call_Connect(const char*ip,const char*port,int*code,char**message)；

This code states a function named "call_connect" which accepts two parameters "ip" and "port" of the "const char" type, a pointer parameter "code" of the "int" type, and a pointer parameter "message" of the "char" type.

The call Connect function means for setting up a connection. It may attempt to connect to the specified server based on the incoming "ip" and "port" and return the status code and message of the connection. The state code of the connection is stored in a variable pointed by the code pointer, and the message of the connection is returned by allocating memory and assigning the pointer to the message.

void call_Create_Collection(milvus_CollectionParam cp,int*code,char**message)；

This code defines a function named "call_create_collection" that accepts three parameters: milvus _ CollectionParam type cp, int type pointer code, and char type pointer message.

The call_create_collection function functions to Create a Collection and return the result of the operation.

void call_Create_Index(const char*name,char*indextype,int*code,char**message)；

This is a function declaration, declaring a function named "call_CreateIndex". The function accepts four parameters, name, indextype, code and message, respectively.

Wherein: "name" is a pointer to a constant character, specifying the name of the index; "indextype" is a pointer to a character, specifying the type of index; "code" is a pointer to an integer for returning the result code of the function execution; "message" is a pointer to a pointer for a message that returns execution of a function.

It should be noted that the specific implementation of the call_create_index function needs to be determined according to the specific embodiment, and only the parameters and return type of the function are stated here.

void call_Insert_Entity(constchar*name,const char*tag,milvus_ids ids,milvus_Entities entities,const char*ip,const char*port,int*code,char**message);

This function is a method of invoking an insertion entity. The specific parameter meaning is: "name": the name of the entity; "tag": a tag of the entity; "ids": an ID list of the entity to be inserted; "entries": a list of entities to be inserted; "ip": milvus the IP address of the server; "port": milvus port numbers of servers; "code": a returned status code; "message": the content of the returned message; the function is to insert entity data into Milvus servers and return status codes and messages for the insert operation.

void call_Search(const char*name,milvus_PartitionTagList plist,milvus_Entities entities,int topk,milvus_TopKQueryResult*query_results,int*code,char**message);

This is a function declaration, which defines a function named "call_search" with the following parameters and return values: wherein: "name": a pointer to the constant character, representing the name of the search; "plist": one data type named milvus _ PartitionTagList represents a partition tag list; "entries": one data type named milvus _ Entities, represents an entity; "topk": an integer representing the number of most similar entities returned; "query_results": a pointer to milvus _ TopKQueryResult, representing the query result; "code": a pointer to an integer representing the returned error code; "message": a pointer to the pointer, representing the returned error information; "void": indicating that the function has not returned a value.

2) Development of fdw is implemented in Multicorn, where the ports that need to be implemented are:

1. GetForeignRelSize (a function in PostgreSQL database management system): and obtaining cost estimation of the basic scanning mode of the SQL sentence.

2. GetForeignPaths, getForeignPlan: and planpath, calculating various conditions, including basic scanning and various join methods, and recovering sql statements needed to be executed in an executor from one best path after the best path is selected from a plurality of plan paths.

3. BeginForeignScan, iterateForeignScan, reScanForeignScan, endForeignScan: for performing a query operation. Here, the SQL needs to be parsed, the incoming target vector is acquired, the packaged Milvus Api is called for searching, and the topK query result is returned.

4. BeginForeignModify, execForeignInsert, endForeignModify, execForeignDelete, execForeignUpdate: the method is used for executing the operations of inserting, deleting and updating data, and also needs to analyze SQL sentences, convert the data into corresponding structures and call interfaces to store the data into a vector engine.

3) Packaging after the development of the required functions in Multicorn is completed, and adding the extension in PostgreSQL can be realized by the following codes:

create extension multicorn；

In storing the vector with structured information, the following tabulated approach may be used:

create foreign table partest(myid bigint,myvec float[2],myscore float4,race varchar,sex varchar,age varchar)server pgVector_srv options();

wherein race, sex, age is the structured data column.

With the above code, when creating an external table, some OPTIONS can be specified for the external table using the "OPTIONS ()" clause. In a given statement, the "OPTIONS ()" clause is empty, i.e., no option is specified for the external table.

The options may be used to specify parameters associated with the external table, such as file path, connection strings, data formats, data separators, and the like. Different options may be supported depending on the particular external table type and external table server used. In an exemplary embodiment of the present application, the "OPTIONS ()" clause is empty, possibly because no particular option is specified for the external table "partest".

Further, FIG. 3 is a flowchart of inserting vector data, wherein when vector data is inserted, structured information is extracted within an implemented port and a corresponding partition is created within a vector engine, the partition ID being associated with the extracted structured information. Specifically, as shown in fig. 3:

step S301, inserting an SQL sentence into the PostgreSQL;

Step S302, postgreSQL is input to Multicorn through FDW;

Step S303, extracting vector information and structured data, splicing the structured data into partition IDs, and inputting the spliced partition IDs into a vector engine;

Step S304, judging whether the partition corresponding to the ID exists in the vector table or not through a vector engine; in the case where there has been a partition of the corresponding ID in the vector table, step S305 is performed; in the case where there is no partition corresponding to the ID in the vector table, step S306 is performed;

Step S305, vector is stored in the partition;

Step S306, create a new partition with the ID and store the vector.

The SQL grammar with structured vector data insertion is as follows:

insert into partest values(1,'{1,1}',0,'hei','nan','zhong')；

insert into partest(myid,sex,race,age,myvec)values(1,'nan','hei','zhong','{1,1}')；

In addition, default and field nulling are supported, but are not equivalent at query time.

Insert into partest values (1, '{1,1}',0, 'hei', 'nan', 'zhong') is an SQL statement that inserts a record into a table named partest. Wherein, "insert into partest": indicating that data is to be inserted into partest tables; "values": representing the specific value to be inserted; "(1, '{1,1}',0, 'hei', 'nan', 'zhong')": representing the specific value to be inserted, and assigning values according to the column sequence of the table; the first value of 1 will be inserted into the first column of the table; "{1,1}" is an array to be inserted into the second column of the table; 0 will be inserted into the third column of the table; "hei" will be inserted into the fourth column of the table; "nan" will be inserted into the fifth column of the table; "zhong" will be inserted in the sixth column of the table.

Insert into partest (myid, sex, trace, age, myvec) values (1, 'nan', 'hei', 'zhong', '1, 1') are an SQL statement used to insert a new record into the partest table. Specific: myid is set to 1, sex is set to "nan", race is set to "hei", age is set to "zhong", myvec is set to "{1,1}".

Further, fig. 4 is a flowchart of a retrieval process according to an exemplary embodiment of the present application, specifically, as shown in fig. 4:

Step S401, inserting SQL sentences into PostgreSQL;

Step S402, inputting PostgreSQL to Multicorn through FDW;

step S403, extracting vector information and structured data, splicing the structured data into partition IDs, and inputting the spliced partition IDs into a vector engine;

Step S404, vector retrieval is carried out in the corresponding partition by a vector engine;

Step S405, returning the searched TopK of vector table T area A/a/3, vector table T area A/b/3, vector table T area A/c/3 … …, vector table T area to the vector engine;

step S406, the vector engine returns the search result to Multicorn;

Step S407, converting the search result in Multicorn into a data type which can be identified by PG and sending the data type to PostgreSQL;

step S408, the search result is output in PostgreSQL.

The operation on the vector can be directly completed by using SQL sentences through the exemplary embodiment of the application, so that the business is stored in a set of system, and the system architecture is simplified to a certain extent.

Furthermore, SQL sentences used by the operation vector library are basically the same as those used by the relational database, extra learning cost is not needed, the functions of multi-table inquiry, transaction and the like can be supported, and the original characteristics of the database are maintained.

In addition, because the vector structured joint query is supported, the data volume required to be calculated during query is further reduced, and compared with unstructured search, the performance is greatly improved.

By the embodiment of the application, the following steps can be obtained: vector structured joint search versus non-joint search performance versus experimental result data fig. 5 is a schematic diagram of vector joint search time consumption under different structured data according to an exemplary embodiment of the present application, as shown in fig. 5, in particular:

Two structured columns are in the vector table used in the comparative experiment, and the storage generated partitions i d are: each partition stores 128-dimensional vector data of tens of millions, including one hundred million, of the 128-dimensional vector data of 'A/1', 'A/2', 'A/3', 'A/4', 'B/1', 'B/2', 'B/3', 'C/1', 'C/2', and 'C/3'. And recording the joint retrieval time consumption under different structured data, and taking the average value record of ten retrieval times.

As can be seen from fig. 5, the query efficiency using vector structured joint search is significantly improved compared to non-joint search (i.e., full-volume search).

From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method of the various embodiments of the present invention.

The present embodiment also provides a vector determining device, which is used to implement the foregoing embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.

Fig. 6 is a block diagram of a vector determination apparatus according to an embodiment of the present invention, the apparatus including:

a receiving module 52, configured to receive a vector search request, where the vector search request carries: a first vector and first structured data, wherein the first structured data is used for indicating attribute information corresponding to the first vector;

a stitching module 54, configured to stitch the first structured data to obtain stitched first structured data;

The control module 56 is configured to control an open source vector search engine encapsulated in a database to determine a target vector table partition according to the plurality of vector table partitions of the spliced first structured data in the database;

A determining module 58, configured to determine a second vector in the target vector table partition according to the first vector, where a similarity between the second vector and the first vector is greater than or equal to a preset threshold.

By the device, a vector retrieval request carrying a first vector and first structured data for indicating attribute information corresponding to the first vector is received, and an open source vector search engine packaged in a database is controlled to determine a target vector table partition in a plurality of vector table partitions of the database according to the spliced first structured data, wherein the spliced first structured data is obtained by splicing the first structured data. And further determining a second vector with the similarity to the first vector being greater than or equal to a preset threshold value in the target vector table partition according to the first vector. By adopting the technical scheme, the problem that more index algorithms with better retrieval effect cannot be supported in the existing database is solved.

In an exemplary embodiment, the determining module 58 is configured to control the open source vector search engine to compare the spliced first structured data with the identification information of the plurality of vector table partitions to obtain a comparison result; determining first identification information consistent with the spliced first structured data according to the comparison result; and determining the vector table partition corresponding to the first identification information as the target vector table partition.

In one exemplary embodiment, the determining module 58 is configured to package the interface of the open source vector search engine as a static library file; packaging the generated static library file and the header file of the interface into a port of a packaging layer of the database; and calling a port of a packaging layer of the database to control an open source vector search engine packaged in the database to determine a target vector table partition according to the plurality of vector table partitions of the spliced first structured data in the database.

In an exemplary embodiment, the above apparatus further includes: the insertion module is configured to receive a vector insertion request, where the vector retrieval request carries: a third vector and second structured data, wherein the second structured data is used for indicating attribute information corresponding to the third vector; parsing the vector insertion request to obtain the third vector and the second structured data; and determining a storage area of the third vector according to the second structured data.

In an exemplary embodiment, an inserting module is configured to splice the second structured data to obtain spliced second structured data; determining whether second identification information consistent with the spliced second structured data exists in the identification information of the vector table partitions; controlling the open source vector search engine to determine a storage area of the third vector according to a determination result, wherein the determination result is used for indicating one of the following: second identification information consistent with the spliced second structured data exists in the identification information of the vector table partitions; and second identification information consistent with the spliced second structured data does not exist in the identification information of the vector table partitions.

In an exemplary embodiment, the inserting module is configured to determine, when the determination result indicates that second identification information consistent with the spliced second structured data exists in the identification information of the plurality of vector table partitions, that a vector table partition corresponding to the second identification information is a storage area of the third vector; and under the condition that the determining result indicates that second identification information consistent with the spliced second structured data does not exist in the identification information of the plurality of vector table partitions, establishing a target vector table partition according to the spliced second structured data, and determining that the target vector table partition is a storage area of the third vector.

In an exemplary embodiment, the above apparatus further includes: the sending module is used for determining the data type identifiable by the database and converting the second vector into a third vector corresponding to the data type; and sending a vector retrieval result to terminal equipment according to the third vector, wherein the terminal equipment is equipment for sending the vector retrieval request.

Alternatively, in the present embodiment, the above-described storage medium may be configured to store a computer program for performing the steps of:

s1, receiving a vector retrieval request, wherein the vector retrieval request carries: a first vector and first structured data, wherein the first structured data is used for indicating attribute information corresponding to the first vector;

s2, splicing the first structured data to obtain spliced first structured data;

S3, controlling an open source vector search engine packaged in a database to determine a second vector table partition according to a plurality of vector table partitions of the spliced first structured data in the database;

S4, determining a second vector in the target vector table partition according to the first vector, wherein the similarity between the second vector and the first vector is greater than or equal to a preset threshold value.

In one exemplary embodiment, the computer readable storage medium may include, but is not limited to: a usb disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing a computer program.

Specific examples in this embodiment may refer to the examples described in the foregoing embodiments and the exemplary implementation, and this embodiment is not described herein.

An embodiment of the invention also provides an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.

Alternatively, in the present embodiment, the above-described processor may be configured to execute the following steps by a computer program:

s1, receiving a vector retrieval request, wherein the vector retrieval request carries: a first vector and first structured data, wherein the first structured data is used for indicating attribute information corresponding to the first vector;

s2, splicing the first structured data to obtain spliced first structured data;

S3, controlling an open source vector search engine packaged in a database to determine a second vector table partition according to a plurality of vector table partitions of the spliced first structured data in the database;

S4, determining a second vector in the target vector table partition according to the first vector, wherein the similarity between the second vector and the first vector is greater than or equal to a preset threshold value.

In an exemplary embodiment, the electronic apparatus may further include a transmission device connected to the processor, and an input/output device connected to the processor.

Specific examples in this embodiment may refer to the examples described in the foregoing embodiments and the exemplary implementation, and this embodiment is not described herein.

It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A method of vector determination, comprising:

receiving a vector retrieval request, wherein the vector retrieval request carries: a first vector and first structured data, wherein the first structured data is used for indicating attribute information corresponding to the first vector;

splicing the first structured data to obtain spliced first structured data; controlling an open source vector search engine packaged in a database to determine a second vector table partition according to a plurality of vector table partitions of the spliced first structured data in the database;

And determining a second vector in the target vector table partition according to the first vector, wherein the similarity between the second vector and the first vector is greater than or equal to a preset threshold value.

2. The method of claim 1, wherein controlling an open source vector search engine to determine a target vector table partition based on a plurality of vector table partitions in a database of the spliced first structured data comprises:

controlling the open source vector search engine to compare the spliced first structured data with the identification information of the plurality of vector table partitions so as to obtain a comparison result;

Determining first identification information consistent with the spliced first structured data according to the comparison result;

And determining the vector table partition corresponding to the first identification information as the target vector table partition.

3. The method of claim 1, wherein controlling an open source vector search engine encapsulated in a database to determine a target vector table partition from a plurality of vector table partitions in the database of the spliced first structured data comprises:

packaging the interface of the open source vector search engine into a static library file;

packaging the generated static library file and the header file of the interface into a port of a packaging layer of the database;

And calling a port of a packaging layer of the database to control an open source vector search engine packaged in the database to determine a target vector table partition according to the plurality of vector table partitions of the spliced first structured data in the database.

4. The method of vector determination of claim 1, further comprising:

receiving a vector insertion request, wherein the vector retrieval request carries: a third vector and second structured data, wherein the second structured data is used for indicating attribute information corresponding to the third vector;

parsing the vector insertion request to obtain the third vector and the second structured data;

And determining a storage area of the third vector according to the second structured data.

5. The method of vector determination according to claim 4, wherein determining a storage area of the third vector from the second structured data comprises:

Splicing the second structured data to obtain spliced second structured data; determining whether second identification information consistent with the spliced second structured data exists in the identification information of the vector table partitions;

Controlling the open source vector search engine to determine a storage area of the third vector according to a determination result, wherein the determination result is used for indicating one of the following: second identification information consistent with the spliced second structured data exists in the identification information of the vector table partitions; and second identification information consistent with the spliced second structured data does not exist in the identification information of the vector table partitions.

6. The method of determining a vector according to claim 5, wherein determining the storage area of the third vector based on the determination result comprises:

Determining that the vector table partition corresponding to the second identification information is a storage area of the third vector when the determination result indicates that the second identification information consistent with the spliced second structured data exists in the identification information of the plurality of vector table partitions;

And under the condition that the determining result indicates that second identification information consistent with the spliced second structured data does not exist in the identification information of the plurality of vector table partitions, establishing a target vector table partition according to the spliced second structured data, and determining that the target vector table partition is a storage area of the third vector.

7. The method of vector determination of claim 1, wherein after determining a second vector in a target vector table partition from the first vector, the method further comprises:

Determining a data type identifiable by the database, and converting the second vector into a third vector corresponding to the data type;

And sending a vector retrieval result to terminal equipment according to the third vector, wherein the terminal equipment is equipment for sending the vector retrieval request.

8. A vector determination apparatus, comprising:

the receiving module is configured to receive a vector search request, where the vector search request carries: a first vector and first structured data, wherein the first structured data is used for indicating attribute information corresponding to the first vector;

The splicing module is used for splicing the first structured data to obtain spliced first structured data;

the control module is used for controlling an open source vector search engine packaged in a database to determine a target vector table partition according to a plurality of vector table partitions in the database of the spliced first structured data;

And the determining module is used for determining a second vector in the target vector table partition according to the first vector, wherein the similarity between the second vector and the first vector is greater than or equal to a preset threshold value.

9. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored program, wherein the program when run performs the method of any of the preceding claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method according to any of the claims 1 to 7 by means of the computer program.