CN113868160A - Data query method, device and related equipment - Google Patents
Data query method, device and related equipment Download PDFInfo
- Publication number
- CN113868160A CN113868160A CN202110989367.0A CN202110989367A CN113868160A CN 113868160 A CN113868160 A CN 113868160A CN 202110989367 A CN202110989367 A CN 202110989367A CN 113868160 A CN113868160 A CN 113868160A
- Authority
- CN
- China
- Prior art keywords
- data
- fpga
- data information
- query
- data query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
- G06F13/28—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/38—Information transfer, e.g. on bus
- G06F13/42—Bus transfer protocol, e.g. handshake; Synchronisation
- G06F13/4282—Bus transfer protocol, e.g. handshake; Synchronisation on a serial bus, e.g. I2C bus, SPI bus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2213/00—Indexing scheme relating to interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F2213/0026—PCI express
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data query method, which comprises the steps of starting a preset number of working processes according to a data query instruction; wherein the preset number is greater than 1; reading data information in parallel by utilizing each working process, and sending the data information to the FPGA through a DMA channel corresponding to each working process so that the FPGA queries the data information to obtain target data; receiving the target data returned by the FPGA; the data query method can effectively improve the data query efficiency. The application also discloses a data query device, a system, equipment and a computer readable storage medium, which all have the beneficial effects.
Description
Technical Field
The application relates to the technical field of big data, in particular to a data query method, and further relates to a data query device, a data query system, data query equipment and a computer readable storage medium.
Background
In the current big data era, data are generated all the time, and management of mass data by using a database is a necessary choice, but how to efficiently query and acquire mass data becomes a hotspot direction of research.
In order to improve the database query efficiency and release resources occupying a Central Processing Unit (CPU), a heterogeneous acceleration computing Unit for separately computing data is designed, an FPGA (Field-Programmable Gate Array) is used as an accelerator to implement acceleration computing, and an HOST computer uses an X86 CPU, so that the overall scheme is implemented in a CPU + FPGA manner, the implementation of the method is divided into two aspects of hardware and software, a logic circuit for data acceleration is designed on a hardware layer, data integration is implemented on a software layer, and data is issued by driving software, but the existing scheme has some defects and deficiencies in the efficiency of querying data.
Therefore, how to more effectively improve the data query efficiency is an urgent problem to be solved by those skilled in the art.
Disclosure of Invention
The data query method can effectively improve the data query efficiency; it is another object of the present application to provide a data query device, system, apparatus and computer readable storage medium, all of which have the above advantages.
In a first aspect, the present application provides a data query method, including:
starting a preset number of working processes according to the data query instruction; wherein the preset number is greater than 1;
reading data information in parallel by utilizing each working process, and sending the data information to the FPGA through a DMA channel corresponding to each working process so that the FPGA queries the data information to obtain target data;
and receiving the target data returned by the FPGA.
Preferably, the reading data information in parallel by using each of the work processes includes:
reading the data information from the disk in parallel by utilizing each working process;
and storing the data information to a cache space corresponding to each work process.
Preferably, before the sending the data information to the FPGA through the DMA channel corresponding to each work process, the method further includes:
judging whether each cache space reaches a preset cache upper limit;
and if so, executing the step of sending the data information to the FPGA through the DMA channel corresponding to each working process.
Preferably, the sending the data information to the FPGA through the DMA channel corresponding to each work process includes:
and sending the data information to the FPGA by utilizing each DMA channel through a PCIe protocol.
Preferably, if the data information is stored in a first storage area of the FPGA, the FPGA queries the data information to obtain target data, including:
reading data information with a preset size from the first storage area;
inquiring the data information to obtain an inquiry result;
judging whether the data information in the first storage area is read completely;
if not, returning to the step of reading the data information with the preset size from the first storage area until the data information in the first storage area is completely read;
and storing each query result to a second storage area of the FPGA to generate the target data.
Preferably, the receiving the target data returned by the FPGA includes:
when an interrupt signal fed back by the FPGA is acquired, reading the target data from a DMA (direct memory access) driving layer according to the interrupt signal;
and storing the target data to a shared memory.
In a second aspect, the present application also discloses a data query apparatus, including:
the process starting module is used for starting a preset number of working processes according to the data query instruction; wherein the preset number is greater than 1;
the data query module is used for reading data information in parallel by utilizing each working process and sending the data information to the FPGA through the DMA channel corresponding to each working process so that the FPGA queries the data information to obtain target data;
and the data acquisition module is used for receiving the target data returned by the FPGA.
In a third aspect, the present application also discloses a data query system, including:
the CPU is used for starting a preset number of working processes according to the data query instruction; wherein the preset number is greater than 1; reading data information in parallel by utilizing each working process, and sending the data information to the FPGA through a DMA channel corresponding to each working process;
and the FPGA is used for inquiring the data information, acquiring target data and feeding the target data back to the CPU.
In a fourth aspect, the present application also discloses a data query device, including:
a memory for storing a computer program;
a processor for implementing the steps of any of the data query methods described above when executing the computer program.
In a fifth aspect, the present application further discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the data query methods described above.
The data query method comprises the steps of starting a preset number of working processes according to a data query instruction; wherein the preset number is greater than 1; reading data information in parallel by utilizing each working process, and sending the data information to the FPGA through a DMA channel corresponding to each working process so that the FPGA queries the data information to obtain target data; and receiving the target data returned by the FPGA.
Therefore, according to the data query method provided by the application, while the FPGA is used as an accelerator to realize data query, a plurality of working processes capable of running in parallel are designed at the main control end, and a corresponding number of DMA (Direct Memory Access) channels are developed at the driving layer, so that data can be acquired through the plurality of working processes, data transmission is carried out through the plurality of DMA channels, and the acceleration characteristic of the FPGA is added, so that the data query efficiency is greatly improved.
The data query device, the data query system, the data query device and the computer readable storage medium provided by the application all have the beneficial effects, and are not described again herein.
Drawings
In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the drawings that are needed to be used in the description of the prior art and the embodiments of the present application will be briefly described below. Of course, the following description of the drawings related to the embodiments of the present application is only a part of the embodiments of the present application, and it will be obvious to those skilled in the art that other drawings can be obtained from the provided drawings without any creative effort, and the obtained other drawings also belong to the protection scope of the present application.
FIG. 1 is a schematic flow chart of a data query method provided in the present application;
FIG. 2 is a diagram of a PostgreSQL-based multi-process architecture provided herein;
FIG. 3 is a schematic diagram of a data query system provided in the present application;
FIG. 4 is a flow chart illustrating a design for implementing multi-process custom parallel scanning according to the present application;
FIG. 5 is a schematic flow chart of another data query method provided in the present application;
FIG. 6 is a schematic structural diagram of a data query device provided in the present application;
fig. 7 is a schematic structural diagram of a data query device provided in the present application.
Detailed Description
The core of the application is to provide a data query method, which can effectively improve the data query efficiency; another core of the present application is to provide a data query apparatus, system, device and computer readable storage medium, which also have the above beneficial effects.
In order to more clearly and completely describe the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The embodiment of the application provides a data query method.
Referring to fig. 1, fig. 1 is a schematic flow chart of a data query method provided in the present application, where the data query method may include:
s101: starting a preset number of working processes according to the data query instruction; wherein the preset number is more than 1;
the step aims to realize the starting of a work process, and the work process is used for realizing the reading function of the data information, namely reading and obtaining the data information, so that the target data can be conveniently inquired and obtained from the data information, and the data inquiry is realized. Specifically, when data query is needed, a user can initiate a data query instruction based on corresponding terminal equipment; for the main control end (such as a CPU), a preset number of work processes may be started in response to the data query instruction.
The value of the preset number is an integer greater than 1, that is, the master control end is provided with a plurality of working processes for reading data information, so that the data reading efficiency is effectively improved, and the data query efficiency is further improved. It can be understood that the larger the value of the preset number is, the higher the data query efficiency is, however, in consideration of the actual operation load of the master control end, the specific value of the preset number should be set in combination with the actual characteristics of the master control end.
S102: reading data information in parallel by utilizing each working process, and sending the data information to the FPGA through a DMA channel corresponding to each working process so that the FPGA queries the data information to obtain target data;
the step aims to realize data query by using the FPGA so as to obtain target data. As described above, the master control end performs data information reading by designing a plurality of working processes to improve data query efficiency, and on this basis, a corresponding number of DMA channels may be designed in the DMA drive layer, that is, one working process corresponds to one DMA channel, so as to implement data information transmission, so as to improve data transmission efficiency and further improve data query efficiency.
In a specific implementation process, after the master control end starts each working process, each working process runs in parallel to read and obtain corresponding data information, wherein the data information is stored in a specified storage space; further, the data information obtained by reading each working process is transmitted to the FPGA through the DMA channel corresponding to each working process, and the FPGA performs data query processing; and finally, the FPGA carries out data query on the obtained data information to obtain final target data.
As a preferred embodiment, the reading data information in parallel by using the work processes may include: reading data information from the disk in parallel by using each working process; and storing the data information to the cache space corresponding to each working process.
The preferred embodiment provides an implementation mode for reading data information by a work process. Specifically, when data information is stored in the disk, after each work process is started, each work process can read the data information in parallel in the disk and store the data information into a corresponding cache space, that is, for each work process, a corresponding cache space can be preset for realizing caching of the data information, so that after more data information is stored in the cache space, the data information is sent to the FPGA through the DMA channel, so that more data information can be sent at one time, and the number of data sending times is reduced.
As a preferred embodiment, before sending the data information to the FPGA through the DMA channel corresponding to each work process, the method may further include: judging whether each cache space reaches a preset cache upper limit or not; and if so, executing the step of sending the data information to the FPGA through the DMA channel corresponding to each working process.
Specifically, a cache upper limit may be preset for the cache space corresponding to each work process, so that before sending the data information to the FPGA, it may be determined whether the data information cached in each cache space has reached the preset cache upper limit, if not, the step of reading the data information in parallel by using each work process is returned, the reading of the data information is continued until the data information in each cache space reaches the preset cache upper limit, and finally, the step of sending the data information to the FPGA is executed. It can be understood that the specific value of the preset cache upper limit is not unique, and may be set by a technician according to an actual situation, which is not limited in the present application.
As a preferred embodiment, the sending the data information to the FPGA through the DMA channel corresponding to each work process may include: the data information is sent to the FPGA by using each DMA channel through a PCIe (peripheral component interconnect express) protocol.
The preferred embodiment provides a method for sending data information, which is realized based on a PCIe protocol. Of course, the data communication protocol is only one implementation manner provided in the preferred embodiment, and is not unique, and other types of data communication protocols may be set according to actual requirements, so that transmission of data information may be implemented, which is not limited in this application.
As a preferred embodiment, the data information may be stored in a first storage area of the FPGA, and the querying the data information by the FPGA to obtain the target data may include: reading data information with a preset size from a first storage area; inquiring the data information to obtain an inquiry result; judging whether the data information in the first storage area is read completely; if not, returning to the step of reading the data information with the preset size from the first storage area until the data information in the first storage area is completely read; and storing each query result to a second storage area of the FPGA to generate target data.
The preferred embodiment provides a method for realizing data query based on FPGA. It should be noted that, in the FPGA, the received data information and the target data queried based on the data information may be stored in different storage spaces, that is, the first storage area and the second storage area, where the first storage area is used for storing the data information and the second storage area is used for storing the target data. In a specific implementation process, the FPGA sequentially reads data information with a preset size from the first storage area, and performs data query on the data information based on preset query conditions to obtain a corresponding query result until the data information in the first storage area is completely read; further, the query result of each time is stored in the second storage area, and the target data can be generated.
For example, the above-mentioned reference may be to create a corresponding cache space for each work process, and each cache space is preset with a corresponding cache upper limit, so that the preset size may be equal to the preset cache upper limit, that is, data query of one cache space is implemented for each data query.
It should be noted that, the process of storing each query result in the second storage area may be executed immediately after the query result is obtained in each data query, or may be executed after all data queries are finished, and all query results are stored in the second storage area differently, and a specific execution sequence thereof does not affect implementation of the present technical solution, and the present application does not limit this.
In addition, the specific types of the first storage area and the second storage area do not affect the implementation of the present technical solution, and a technician may select the types according to actual situations, which is not limited in the present application. For example, in the present application, a DDR (Double Data Rate) memory implementation is selected.
S103: and receiving target data returned by the FPGA.
The step aims to realize the receiving of the target data, and after the target data is obtained by the FPGA through inquiring from the data information sent by the main control terminal, the target data can be fed back to the main control terminal, so that the data inquiry is completed. It is understood that, the present application is intended to implement a data query function, and the number of target data obtained by querying may be 1, or may be multiple, and the specific number does not affect the implementation of the present technical solution.
As a preferred embodiment, the receiving target data returned by the FPGA may include: when an interrupt signal fed back by the FPGA is acquired, target data are read from a DMA drive layer according to the interrupt signal; and storing the target data to the shared memory.
As described above, the data information is sent to the FPGA through the DMA channel, so when the target data is fed back, the target data can also be fed back through the DMA channel, and at this time, the target data is inevitably fed back to the DMA drive layer. Therefore, after the FPGA completes data query and feeds the target data back to the main control terminal, the FPGA can further send an interrupt signal to the main control terminal to notify the main control terminal that the data query is completed, so that the main control terminal can obtain a data query result, that is, the target data. In a specific implementation process, for the main control end, when an interrupt signal fed back by the FPGA is received, the target data can be obtained from the DMA drive layer according to the interrupt signal and stored in the shared memory, so that the target data can be obtained in time.
In addition, in the above preferred embodiment, a small number of data query manners are adopted, for example, each data query is performed to realize data query of one cache space, and the number of the cache spaces corresponds to the number of the work processes, so that when an interrupt signal is sent, an interrupt signal can be sent to the master control end once after the data query of one cache space is completed, and then the master control end can obtain a preset number of interrupt signals, and then read and obtain target data from the DMA drive layer, and store the target data into the shared memory.
Therefore, according to the data query method provided by the application, while the FPGA is used as an accelerator to realize data query, a plurality of working processes capable of running in parallel are designed at the main control end, and a corresponding number of DMA channels are developed at the driving layer, so that data can be acquired through the plurality of working processes, data transmission is performed through the plurality of DMA channels, and the data query efficiency is greatly improved due to the acceleration characteristic of the FPGA.
Based on the above embodiments, the present application provides another data query method.
It should be noted that the data query method provided in the embodiment of the present application is implemented based on PostgreSQL, which is an object-relational database management system (ordms) of free software with very complete characteristics, supports multi-process sequential parallel query, does not support custom parallel scan (custom parallel scan) data query, and needs to be implemented by a designer. The design belongs to parallel heterogeneous accelerated query and is one of self-defined scanning.
As shown in fig. 2, fig. 2 is a diagram of a PostgreSQL-based multi-process architecture provided by the present application, where the PostgreSQL is executed in parallel by a mechanism of multiple processes, each process is internally referred to as 1 worker, and the workers can be dynamically created and destroyed. Certainly, PostgreSQL is not parallel in the SQL (Structured Query Language) statement parsing and Query plan generating stage, but in the Executor module, multiple workers concurrently execute fragmented subtasks, and even in the link where the Query plan is executed in parallel, an existing process serves as one worker to complete parallel subtasks, which may be referred to as a main process. Meanwhile, n worker processes can be started to execute other sub-plans according to the number of workers specified by the configuration parameters. In addition, a shared memory mechanism is continued in the PostgreSQL, when each worker is initialized, the shared memory is distributed to each worker and used for the workers to respectively obtain plan data and cache intermediate results, and the workers have no complex communication mechanism and are all started and executed by a main process through simple communication.
Further, referring to fig. 3, fig. 3 is a schematic structural diagram of a data query system provided in the present application, where the HOST program in HOST runs on the CPU on the left side, and the acceleration board on the right side is implemented based on the hardware logic implemented by the FPGA. Firstly, reading file data from a disk to an application layer by a PostgreSQL application program running at a HOST end, starting four (preset number) processes, and distributing a 128MB cache space for each process; then, copying the data to a DMA cache of an operating system kernel space through a DMA driver, starting transmission, and issuing the data to a DDR memory of the FPGA acceleration board card through a PCIe interface protocol; and finally, the parallel speed adding calculation unit of the FPGA reconfigurable area reads data with specified size from a specified DDR memory, then starts calculation, and after the calculation is finished, the FPGA stores the calculation result in another specified address space of the DDR memory and sequentially and circularly carries out the calculation until the data set needing to be inquired is processed, so that the calculation and the inquiry of the data are finished.
The method comprises the following steps of:
1. the design process for realizing multi-process self-defined parallel scanning based on HOOK (HOOK function) comprises the following steps:
the PostgreSQL database provides a HOOK function mechanism for a user to realize multi-process customized parallel scanning query, the function needs technical personnel to design and realize the function according to the self requirements, as shown in fig. 4, and fig. 4 is a design flow chart for realizing multi-process customized parallel scanning provided by the application.
Specifically, a user-defined interface can be adopted to realize multi-process parallel scanning to complete a query plan, the design and implementation of a user-defined parallel scanning scheme are mainly based on a sequential parallel scanning mechanism in a PG source code, and the main implementation flow mainly comprises the following steps:
the method comprises the following steps: registering a custom scanning Methods function;
step two: creating a custom and adding a parallel path, wherein the main purpose is to add some parameters related to parallel such as nworkers and the like, the nworkers represent the number of processes required by the custom parallel scanning, and the number of workers can be specified by a user according to actual needs;
step three: initializing a dynamic shared memory DSM (Distributed shared memory), wherein the DSM is an important mechanism introduced by a PostgreSQL database for realizing multi-process parallel scanning, and is mainly used for storing data results processed by a worker process, and besides, the workers can mutually coordinate and communicate by using the shared memory;
step four: the workload of each worker is controlled and distributed in an atomic operation mode, then the data are issued to the driving layer, the data can be continuously sent to the DDR at the FPGA side through DMA after entering the driving layer, and the data can be returned to the driving layer after being calculated by the FPGA;
step five: the workers work process reads result data from the driver and stores the result data to a DSM (digital document model) of a database application layer;
step six: the results are read from the DSM.
2. The method comprises the following implementation processes of multi-process data query of an application layer and a driver layer of a PostgreSQL database at the HOST HOST side:
referring to fig. 5, fig. 5 is a schematic flow chart of another data query method provided in the present application. When database query begins, 4 workers are started by a PostgreSQL main process, 4 128MB _ buf memory spaces are applied for simultaneously, wherein the 4 workers read data from a disk to the corresponding 128MB _ buf; further, after the 4 worker tasks are distributed, 128MB _ buf data can be issued to the DMA drive layer buf corresponding to the 4 DMA write channels respectively, at this time, the OS kernel starts the DMA drive engine, sends the data to the FPGA through the PCIe protocol, and when the 4 128MB _ buf data are sent, continues to read the data from the disk until the data in the disk are read completely. Meanwhile, the FPGA receives data and stores the data into the DDR _ ADDR1, performs data query to obtain a query result, and stores the query result into the DDR _ ADDR 2; further, after the data query is finished, the FPGA sequentially sends four times of calculation completion interrupts to the DMA. Therefore, the HOST-end PostgreSQL application layer can acquire four interrupt signals from the DMA drive through reading the interrupt event interface, further acquire a query result from the DMA drive through the DMA data interface, and finally, the PostgreSQL application layer assigns values to each tuple according to the query result and outputs a final query result, so that the database query is finished.
In addition, the PostgreSQL11.2 open source database version is used in the scheme, and in the PostgreSQL11.2 source code, a normal data query implementation mode is software sequential scanning, but in the present application, a multi-process Custom parallel scanning mode is used to implement data query, which may be referred to as Custom multi-process parallel scanning (Custom parallel scanning), and adding this scanning mode does not change the source code of PostgreSQL11.2, and only needs to be completed by adding a HOOK function, and the four 128MB _ buf caches applied in the application layer implementation flow may also be implemented in the HOOK function.
In addition, the DMA driver adopts a PCIe protocol-based DMA driver developed by Xilinx corporation, and is used for realizing the transmission and the reception of data between the HOST HOST end and the FPGA acceleration board card. According to the technical scheme, four worker working processes are started in the database application layer, and accordingly four DMA channels are needed for data transmission in the driving process, so that the bandwidth and the efficiency of data transmission are greatly improved. Meanwhile, some functional registers can be added for storing information such as addresses and lengths related to the source data and the result data; some event interrupt status registers may also be provided for recording the operating status of the FPGA.
Therefore, according to the data query method provided by the embodiment of the application, while the FPGA is used as an accelerator to realize data query, a plurality of working processes capable of running in parallel are designed at the main control end, and a corresponding number of DMA channels are developed at the driving layer, so that data can be acquired through the plurality of working processes, data transmission is performed through the plurality of DMA channels, and the acceleration characteristic of the FPGA is added, so that the data query efficiency is greatly improved.
To solve the above technical problem, the present application further provides a data query device, please refer to fig. 6, where fig. 6 is a schematic structural diagram of a data query device provided in the present application, and the data query device may include:
the process starting module 1 is used for starting a preset number of working processes according to the data query instruction; wherein the preset number is more than 1;
the data query module 2 is used for reading data information in parallel by utilizing each working process and sending the data information to the FPGA through the DMA channel corresponding to each working process so that the FPGA queries the data information to obtain target data;
and the data acquisition module 3 is used for receiving the target data returned by the FPGA.
It can be seen that, the data query device provided in the embodiment of the present application, while implementing data query with the FPGA as an accelerator, designs a plurality of parallel running work processes at the main control end, and develops a corresponding number of DMA channels at the driver layer, so that data can be acquired through the plurality of work processes, and data transmission is performed through the plurality of DMA channels, in addition to the acceleration characteristic of the FPGA, so that the data query efficiency is greatly improved by this implementation manner.
As a preferred embodiment, the data query module 2 may be specifically configured to utilize each work process to read data information from a disk in parallel; and storing the data information to the cache space corresponding to each working process.
As a preferred embodiment, the data query module 2 may be further configured to determine whether each cache space reaches a preset cache upper limit before the DMA channel corresponding to each work process sends the data information to the FPGA; and if so, executing the step of sending the data information to the FPGA through the DMA channel corresponding to each working process.
As a preferred embodiment, the data query module 2 may be specifically configured to send the data information to the FPGA through each DMA channel by using a PCIe protocol.
As a preferred embodiment, the data obtaining module 3 may be specifically configured to, when obtaining an interrupt signal fed back by the FPGA, read and obtain target data from the DMA drive layer according to the interrupt signal; and storing the target data to the shared memory.
For the introduction of the apparatus provided in the present application, please refer to the above method embodiments, which are not described herein again.
In order to solve the above technical problem, the present application further provides a data query system, which may include:
the CPU is used for starting a preset number of working processes according to the data query instruction; wherein the preset number is more than 1; reading data information in parallel by utilizing each working process, and sending the data information to the FPGA through a DMA channel corresponding to each working process;
and the FPGA is used for inquiring the data information, acquiring target data and feeding the target data back to the CPU.
It can be seen that, in the data query system provided in the embodiment of the present application, while implementing data query by using the FPGA as an accelerator, a plurality of working processes capable of running in parallel are designed at the master control end, and a corresponding number of DMA channels are developed at the driver layer, so that data can be acquired through the plurality of working processes, and data transmission is performed through the plurality of DMA channels, in addition to the acceleration characteristic of the FPGA, so that the data query efficiency is greatly improved by the implementation manner.
As a preferred embodiment, the data information may be stored in a first storage area of the FPGA, and the FPGA may be specifically configured to read data information of a preset size from the first storage area; inquiring the data information to obtain an inquiry result; judging whether the data information in the first storage area is read completely; if not, returning to the step of reading the data information with the preset size from the first storage area until the data information in the first storage area is completely read; and storing each query result to a second storage area of the FPGA to generate target data.
For the introduction of the system provided by the present application, please refer to the above method embodiment, which is not described herein again.
To solve the above technical problem, the present application further provides a data query device, please refer to fig. 7, where fig. 7 is a schematic structural diagram of a data query device provided in the present application, and the data query device may include:
a memory 10 for storing a computer program;
the processor 20, when executing the computer program, may implement the steps of any of the data query methods described above.
For the introduction of the device provided in the present application, please refer to the above method embodiment, which is not described herein again.
To solve the above problem, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, can implement the steps of any one of the data query methods described above.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
For the introduction of the computer-readable storage medium provided in the present application, please refer to the above method embodiments, which are not described herein again.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The technical solutions provided by the present application are described in detail above. The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall into the protection scope of the present application.
Claims (10)
1. A method for querying data, comprising:
starting a preset number of working processes according to the data query instruction; wherein the preset number is greater than 1;
reading data information in parallel by utilizing each working process, and sending the data information to the FPGA through a DMA channel corresponding to each working process so that the FPGA queries the data information to obtain target data;
and receiving the target data returned by the FPGA.
2. The data query method of claim 1, wherein the reading data information in parallel by using each of the work processes comprises:
reading the data information from the disk in parallel by utilizing each working process;
and storing the data information to a cache space corresponding to each work process.
3. The data query method according to claim 2, wherein before the data information is sent to the FPGA through the DMA channel corresponding to each of the work processes, the method further includes:
judging whether each cache space reaches a preset cache upper limit;
and if so, executing the step of sending the data information to the FPGA through the DMA channel corresponding to each working process.
4. The data query method according to claim 1, wherein the sending the data information to the FPGA through the DMA channel corresponding to each of the work processes includes:
and sending the data information to the FPGA by utilizing each DMA channel through a PCIe protocol.
5. The data query method according to claim 1, wherein if the data information is stored in the first storage area of the FPGA, the FPGA queries the data information to obtain the target data, and the method includes:
reading data information with a preset size from the first storage area;
inquiring the data information to obtain an inquiry result;
judging whether the data information in the first storage area is read completely;
if not, returning to the step of reading the data information with the preset size from the first storage area until the data information in the first storage area is completely read;
and storing each query result to a second storage area of the FPGA to generate the target data.
6. The data query method according to claim 1, wherein the receiving the target data returned by the FPGA comprises:
when an interrupt signal fed back by the FPGA is acquired, reading the target data from a DMA (direct memory access) driving layer according to the interrupt signal;
and storing the target data to a shared memory.
7. A data query apparatus, comprising:
the process starting module is used for starting a preset number of working processes according to the data query instruction; wherein the preset number is greater than 1;
the data query module is used for reading data information in parallel by utilizing each working process and sending the data information to the FPGA through the DMA channel corresponding to each working process so that the FPGA queries the data information to obtain target data;
and the data acquisition module is used for receiving the target data returned by the FPGA.
8. A data query system, comprising:
the CPU is used for starting a preset number of working processes according to the data query instruction; wherein the preset number is greater than 1; reading data information in parallel by utilizing each working process, and sending the data information to the FPGA through a DMA channel corresponding to each working process;
and the FPGA is used for inquiring the data information, acquiring target data and feeding the target data back to the CPU.
9. A data query device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data query method of any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the data query method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110989367.0A CN113868160A (en) | 2021-08-26 | 2021-08-26 | Data query method, device and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110989367.0A CN113868160A (en) | 2021-08-26 | 2021-08-26 | Data query method, device and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113868160A true CN113868160A (en) | 2021-12-31 |
Family
ID=78988323
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110989367.0A Withdrawn CN113868160A (en) | 2021-08-26 | 2021-08-26 | Data query method, device and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113868160A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114465952A (en) * | 2022-01-20 | 2022-05-10 | 绿盟科技集团股份有限公司 | Management method and device for configuration parameters and electronic equipment |
| CN114547612A (en) * | 2022-02-23 | 2022-05-27 | 安天科技集团股份有限公司 | Process query method and device, electronic equipment and storage medium |
| CN115544069A (en) * | 2022-09-26 | 2022-12-30 | 山东浪潮科学研究院有限公司 | Reconfigurable database query acceleration processor and system |
-
2021
- 2021-08-26 CN CN202110989367.0A patent/CN113868160A/en not_active Withdrawn
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114465952A (en) * | 2022-01-20 | 2022-05-10 | 绿盟科技集团股份有限公司 | Management method and device for configuration parameters and electronic equipment |
| CN114465952B (en) * | 2022-01-20 | 2023-12-01 | 绿盟科技集团股份有限公司 | Management method and device for configuration parameters and electronic equipment |
| CN114547612A (en) * | 2022-02-23 | 2022-05-27 | 安天科技集团股份有限公司 | Process query method and device, electronic equipment and storage medium |
| CN115544069A (en) * | 2022-09-26 | 2022-12-30 | 山东浪潮科学研究院有限公司 | Reconfigurable database query acceleration processor and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113868160A (en) | Data query method, device and related equipment | |
| WO2021223356A1 (en) | Server architecture, database query method, and storage medium | |
| US11474712B2 (en) | Method, apparatus, device and storage medium for managing access request | |
| WO2018075182A1 (en) | Gpu remote communication with triggered operations | |
| CN114880346B (en) | Data processing method, related assembly and acceleration processor | |
| CN110119304B (en) | Interrupt processing method and device and server | |
| CN110291503B (en) | Information processing system and information processing method | |
| CN113704301B (en) | Data processing method, device, system, equipment and medium of heterogeneous computing platform | |
| CN111737564B (en) | Information query method, device, equipment and medium | |
| CN112100090A (en) | Data access request processing method, device, medium and memory mapping controller | |
| EP3036648B1 (en) | Enhanced data transfer in multi-cpu systems | |
| CN111736907B (en) | Data analysis method of self-adaptive low-delay memory computing engine | |
| CN112947857A (en) | Data moving method, device, equipment and computer readable storage medium | |
| JP2021022379A (en) | Autonomous job queueing system for hardware accelerators | |
| US20190327342A1 (en) | Methods and electronic devices for data transmission and reception | |
| CN109992539B (en) | Double-host cooperative working device | |
| US20230153153A1 (en) | Task processing method and apparatus | |
| CN109144853B (en) | Software defined radio SoC chip debugging system | |
| CN110781137A (en) | Directory reading method and device for distributed system, server and storage medium | |
| JP7427775B2 (en) | Stored procedure execution method, device, database system, and storage medium | |
| CN114547199A (en) | Database increment synchronous response method and device and computer readable storage medium | |
| CN113220608A (en) | NVMe command processor and processing method thereof | |
| CN112559434A (en) | Multi-core processor and inter-core data forwarding method | |
| CN110543349A (en) | Application starting acceleration method and device and computer readable storage medium | |
| CN116594951B (en) | FPGA-based data transmission system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20211231 |