CN112380169A - Storage device, data processing method, device, apparatus, medium, and system - Google Patents
Storage device, data processing method, device, apparatus, medium, and system Download PDFInfo
- Publication number
- CN112380169A CN112380169A CN202011311360.5A CN202011311360A CN112380169A CN 112380169 A CN112380169 A CN 112380169A CN 202011311360 A CN202011311360 A CN 202011311360A CN 112380169 A CN112380169 A CN 112380169A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- module
- target
- target data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000012545 processing Methods 0.000 claims abstract description 77
- 238000004364 calculation method Methods 0.000 claims abstract description 38
- 238000000034 method Methods 0.000 claims description 28
- 239000013598 vector Substances 0.000 claims description 19
- 230000008569 process Effects 0.000 claims description 14
- 238000000605 extraction Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 230000001815 facial effect Effects 0.000 claims description 8
- 238000012795 verification Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 abstract description 17
- 238000010586 diagram Methods 0.000 description 12
- 230000003287 optical effect Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000012216 screening Methods 0.000 description 5
- 238000012217 deletion Methods 0.000 description 3
- 230000037430 deletion Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 239000000284 extract Substances 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/156—Query results presentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a storage device, a data processing method, a device, equipment, a medium and a system. The storage device includes: the device comprises at least one storage module and a calculation module corresponding to each storage module, wherein the storage module is connected with the corresponding calculation module; the storage module is used for storing data; and the computing module is used for processing data in the connected storage modules. The embodiment of the invention can reduce the data query transmission amount and improve the data query efficiency.
Description
Technical Field
Embodiments of the present invention relate to the field of data processing, and in particular, to a storage device, a data processing method, a device, an apparatus, a medium, and a system.
Background
With the wide application of the internet, a large amount of information interaction exists in various industries, so that the circulating data volume is more and more huge. In data retrieval applications, such as face retrieval and fingerprint retrieval performed by the police, the data to be retrieved and matched is very huge (more than 10G).
In the application of the existing big data retrieval, all retrieval data are stored in a storage device, a sampling module is used for collecting samples, a processor receives the collected samples and extracts the characteristics of the collected samples, and due to the fact that the storage space on the processor is limited, the processor receives comparison characteristics sent by the storage device in batches, and then compares the comparison characteristics received in batches with the extracted characteristics until a final result is determined.
However, when the above method is used, a large amount of data needs to be transmitted from the storage device to the processor, and even more than 10G of data needs to be transmitted to the processor due to the uncertain position of the final result, so that the transmission takes a long time and the processing time is long.
Disclosure of Invention
Embodiments of the present invention provide a storage device, a data processing method, an apparatus, a device, a medium, and a system, which can reduce the amount of query transmission data and improve the data query efficiency.
In a first aspect, an embodiment of the present invention provides a storage apparatus, including:
the device comprises at least one storage module and a calculation module corresponding to each storage module, wherein the storage module is connected with the corresponding calculation module;
the storage module is used for storing data;
and the computing module is used for processing data in the connected storage modules.
In a second aspect, an embodiment of the present invention further provides a data processing method applied to the storage device according to any embodiment of the present invention, including:
acquiring target data to be processed through a computing module;
and processing data according to the target data in the connected storage modules through the computing module.
In a third aspect, an embodiment of the present invention further provides a data processing method, including:
acquiring target data to be inquired;
sending the target data to at least one computing module so that each computing module can perform data query in a corresponding storage module according to the target data;
and receiving the query result fed back by each computing module, and processing the query result to obtain a target query result, wherein the storage device according to any embodiment of the invention comprises each computing module and each storage module, and the computing modules are connected with the corresponding storage modules.
In a fourth aspect, an embodiment of the present invention further provides a data processing apparatus, including:
the target data acquisition module is used for acquiring target data to be inquired;
the data query module is used for sending the target data to at least one computing module so as to enable each computing module to perform data query in a corresponding storage module according to the target data;
and the query result processing module is used for receiving the query results fed back by the computing modules and processing the query results to obtain target query results, wherein the storage device according to any embodiment of the invention comprises the computing modules and the storage modules, and the computing modules are connected with the corresponding storage modules.
In a fifth aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the data processing method according to any one of the embodiments of the present invention when executing the computer program.
In a sixth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data processing method according to any one of the embodiments of the present invention.
In a seventh aspect, an embodiment of the present invention further provides a data processing system, including: a storage apparatus according to any embodiment of the present invention, and a querying device according to any embodiment of the present invention, the storage apparatus being connected to the querying device;
the storage device is used for storing data, inquiring in the stored data according to the target data sent by the inquiring equipment and feeding back an inquiring result to the inquiring equipment;
and the query equipment is used for sending the target data to the storage device, receiving the query result fed back by the storage device and processing the query result to obtain a target query result.
The embodiment of the invention can improve the query efficiency and reduce the data volume of the query result by configuring a plurality of storage modules and the computing modules corresponding to the storage modules in the storage device, wherein the computing modules are used for processing data in the corresponding storage modules, so that the data is preprocessed in the storage device, the problem that in the prior art, the data volume of large data retrieval is large and the processing time is long is solved, the data can be preprocessed in the storage device, the data volume in the query result is reduced, the data volume of transmission is reduced, and the data query efficiency and the transmission efficiency are improved.
Drawings
FIG. 1 is a schematic structural diagram of a memory device according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a data processing method according to a second embodiment of the present invention;
FIG. 3a is a flow chart of a data processing method according to a third embodiment of the present invention;
FIG. 3b is a flowchart of a data processing method according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data processing apparatus according to a fourth embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a computer device according to a fifth embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a computer device according to a fifth embodiment of the present invention;
fig. 7 is a schematic structural diagram of a data processing system in a seventh embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a schematic structural diagram of a storage device according to a first embodiment of the present invention, which is applicable to a case of storing data and processing the stored data, and as shown in fig. 1, the storage device 100 may include a storage module 110 and a calculation module 120.
The storage device 100 includes at least one storage module 110 and a corresponding computing module 120 for each storage module 110, and the storage module 110 is connected to the corresponding computing module 120. The storage module 110 is used for storing data; the calculation module 120 is used for data processing in the connected storage module 110.
The storage module can be obtained by dividing the storage space of the storage device. The storage space formed by the plurality of storage modules can store a large amount of data, for example, 10G data. The storage modules are arranged in a distributed mode to scatter the storage space of the storage device, and the calculation modules are respectively configured corresponding to the storage modules.
The calculation module may refer to a module that performs data processing on data stored in the connected storage module. And the data in different storage modules are processed by different computing modules. Illustratively, a compute module may include at least one compute core. The computing core included in the computing module may be a computing core with a general-purpose computing function, or may not have a general-purpose computing function, for example, a computing core with a targeted configuration query function, which is advantageous in that the cost of the storage device can be reduced.
Optionally, the data processing may include a classification process, a query process, a storage process, and the like. The storage device can process data, replaces an inquirer to execute preliminary inquiry operation in advance, and can reduce the data volume of the inquiry result, thereby reducing the data volume of the transmission inquiry result, and simultaneously reducing the processing operation of the inquirer and improving the data inquiry efficiency of the inquirer. In addition, the data processing may further include modification processing, deletion processing, and the like, and may be specifically set as needed.
The embodiment of the invention can improve the query efficiency and reduce the data volume of the query result by configuring a plurality of storage modules and the computing modules corresponding to the storage modules in the storage device, wherein the computing modules are used for processing data in the corresponding storage modules, so that the data is preprocessed in the storage device, the problem that in the prior art, the data volume of large data retrieval is large and the processing time is long is solved, the data can be preprocessed in the storage device, the data volume in the query result is reduced, the data volume of transmission is reduced, and the data query efficiency and the transmission efficiency are improved.
Example two
Fig. 2 is a schematic diagram of a flowchart of a data processing method in a second embodiment of the present invention, where this embodiment is applicable to a case where data is stored and processed, and the method may be executed by a storage device provided in the second embodiment of the present invention, and the storage device may be implemented in a software and/or hardware manner and may be generally integrated into a storage device. As shown in fig. 2, the method of this embodiment specifically includes:
and S210, acquiring target data to be processed through a calculation module.
The computing module is connected with the external module and receives the target data to be processed sent by the external module. The target data may refer to data to be stored or data to be matched to a query. In addition, the target data can also be data to be deleted and data to be modified, so that the matched data can be inquired in the storage device for deletion and modification. The specific setting may be performed according to actual situations, and the embodiment of the present invention is not particularly limited.
And S220, performing data processing according to the target data in the connected storage modules through the computing module.
And the computing module performs data processing on the connected storage modules in the connected storage modules according to the target data. Data processing may include storage, querying, sorting, modification, deletion, and the like.
Optionally, the performing data processing according to the target data includes: inquiring the target data and feeding back an inquiry result; or acquiring the type of the target data and storing the target data so as to store the target data in a storage module matched with the type.
When there is a query demand, querying the target data may refer to querying data matched with the target data in the corresponding storage module. The query result includes data that matches the target data in the corresponding storage module. And feeding back the query result to the inquirer sending the target data. It should be noted that, the accuracy of the query result is determined according to the computing power of the computing module. The query result can be only the result of the preliminary screening and is fed back to the querying party, the querying party further queries the query result, partial query operation of the querying party is transferred to the computing module of the storage device for execution, the query range of the querying party can be reduced, and the query efficiency of the querying party is improved.
And when the storage requirement exists, classifying and storing the target data. The type of the target data can be determined according to a preset classification rule, and the target data and the data of the same type are stored together according to the type of the target data, for example, in the same storage module, or in storage modules with adjacent storage addresses. A storage module that matches a type may refer to a storage module that stores the type of data or a storage module that is located adjacent to a storage module that stores the type of data. Optionally, different storage modules are used to store different types of data.
The computing module can be connected with any one of the storage modules or only connected with the corresponding storage module. If the computing module is connected with any storage module, the target data is sent to the storage module matched with the type for storage; and if the computing module is connected with the corresponding storage module only, sending the target data to the computing module matched with the type, and forwarding the target data to the storage module matched with the type by the computing module for storage.
The data processing is configured into data query, so that partial query operation of a query party is transferred to a computing module of a storage device for execution, the query range of the query party can be reduced, and the query efficiency of the query party is improved, or the data processing is configured into data classification storage, so that the data classification is realized, the data of the same type can be quickly acquired in query, the query speed is increased, and the query efficiency is improved.
In addition, the computing module can process the stored data in a targeted manner according to the type of the data stored in the connected storage module.
Optionally, the data processing method further includes: and image processing is carried out on the image data in the connected storage modules through the calculation module.
Illustratively, the stored data is image data, and the computing module is further configured to perform image processing on the data stored in the connected storage module. In other cases, the setting may be specifically performed according to a requirement, for example, the stored data is audio data, and the computing module is further configured to perform audio processing on the data stored in the connected storage module.
By configuring the type of the stored data as the image data and performing image processing on the image data in a targeted manner through the computing module, the service module of the storage device in an image query matching application scene can be increased, the application scene of the storage device is enriched, and the diversity of the data stored in the storage device is increased.
Meanwhile, the calculation module can also perform data processing matched with the type on the received target data, for example, the target data is image data, and the calculation module performs image processing on the target data; in another example, the target data is audio data, and the calculation module performs audio processing on the target data.
The embodiment of the invention can improve the query efficiency and reduce the data volume of the query result by configuring a plurality of storage modules and the computing modules corresponding to the storage modules in the storage device, and the computing modules can process data in the corresponding storage modules, so as to realize the preprocessing of the data in the storage device, solve the problems of large data retrieval transmission data volume and long processing time in the prior art, can carry out the preprocessing of the data in the storage device, reduce the data volume in the query result, reduce the data volume of the transmission, and improve the data query efficiency and the transmission efficiency.
Fig. 3a is a schematic diagram of a flowchart of a data processing method in a third embodiment of the present invention, where this embodiment is applicable to a case where query data is sent to a storage device and a query result is received and fed back, and this method may be implemented by a data processing apparatus provided in this embodiment of the present invention, where this apparatus may be implemented in a software and/or hardware manner, and may be generally integrated into a computer device, and specifically executed by a processor in the computer device. As shown in fig. 3a, the method of this embodiment specifically includes:
s310, target data to be inquired are obtained.
The target data is used as target data for matching query in the storage device.
S320, sending the target data to at least one computing module so that each computing module can perform data query in a corresponding storage module according to the target data.
The computing module is connected with a processor in the computer equipment for executing the data processing method provided by the embodiment of the invention. Illustratively, the storage device is configured with an external interface, and the processor is connected to the storage device through the external interface, and specifically, each computing module is connected to the external interface, so that the processor is connected to each computing module through the external interface.
The processor sends the target data to the calculation module, and the calculation module can perform matching query in the data of the corresponding storage module according to the target data.
In addition, the processor can also send the data to each calculation module and store the data in the storage module corresponding to each calculation module. Each computing module can classify the data to be stored and store the data in different storage modules according to type distribution.
The format of the target data can be the same as that of the data in the storage module, and the calculation module can compare the target data with each piece of data in the storage module one by one, query the data matched with the target data, and determine the data as a query result. The format of the target data can be different from that of the data in the storage module, the calculation module can process the target data into the target data matched with the format of the data in the storage module, or process the data in the storage module into the data matched with the format of the target data, and then the target data and each piece of data in the storage module are compared one by one to inquire the data matched with the target data.
In a specific example, the target data is a feature vector of a key point of a face image, and if the data in the storage module is the feature vector of the key point of the face image of different users, the calculation module may compare the target data with the data in the storage module one by one to obtain data matched with the target data as a query result.
For another example, the target data is a face image, if the data in the storage module is key point feature vectors of face images of different users, the calculation module may process the face image in advance to obtain key point feature vectors, which are used as processing results of the target data, compare the processing results corresponding to the target data with the data in the storage module one by one to obtain data matched with the target data, which are used as query results.
For another example, the target data is a feature vector of a key point of a face image, and if the data in the storage module is face images of different users, the calculation module may process the face images in the storage module in advance to obtain the feature vector of the key point, which is used as a processing result of the data in the storage module, and compare the target data with the processing result corresponding to the data in the storage module one by one to obtain data matched with the target data as a query result.
And S330, receiving the query result fed back by each computing module, and processing the query result to obtain a target query result, wherein the storage device according to any embodiment of the invention comprises each computing module and each storage module, and the computing modules are connected with the corresponding storage modules.
Each computing module may obtain a query result. And acquiring a plurality of query results fed back by each calculation module, and counting to obtain a target query result.
In fact, when the computing module has low computing power, the computing module cannot directly obtain an accurate query result. At this time, the calculation module is equivalent to primarily screening the data in the storage module, excluding most irrelevant data, then feeding back the screened data, and finishing the final query by a query party, such as a processor, to finally obtain a comparison result. And when the computing module has high computing capability, the computing module directly queries to obtain an accurate query result.
In a specific example, as shown in fig. 3b, the data processing method may include:
s301, the processor extracts the characteristics of the collected data to obtain target data.
S302, the processor distributes the target data to each calculation module.
S303, each calculation module receives target data.
S304, when the processing capacity of the calculation module is low, the calculation module carries out primary screening.
The calculation module may exclude most of the interference results.
S305, the screened query result is sent back to the processor, and the processor queries to obtain a target query result.
And S306, directly inquiring and matching to obtain a target inquiry result when the processing capacity of the calculation module is high.
Optionally, the processing to obtain the target query result includes: and inquiring data matched with the target data in each inquiry result to obtain a target inquiry result.
No matter whether the query result is accurate or not, the querying party can continue to query in the data range after the range is reduced, and data matched with the target data is determined to obtain an accurate target query result.
The data matched with the target data are continuously inquired in each inquiry result to obtain a target inquiry result, so that an accurate inquiry result can be obtained, the accuracy of the inquiry result is improved, meanwhile, the data inquiry range of an inquirer can be reduced by primarily screening through the storage device and finely screening through the inquirer, and the accuracy of the inquiry data is improved.
Optionally, the processing to obtain the target query result includes: checking the query result fed back by each computing module; if the verification is passed, determining each query result as a target query result; and if the check is not passed, inquiring data matched with the target data in each inquiry result to obtain a target inquiry result.
The query result fed back by the calculation module is dynamically completed in the actual processing process, whether the calculation module can complete accurate query or not is unpredictable, and the query result can only be determined through the query result, so that a query party cannot predetermine whether the received query result is accurate or not, and the query result can be verified and determined.
Checking to judge whether the query result is an accurate query result, if so, determining that the query result meets the query requirement of the query party, and directly determining the query result as a target query result; if not, further query matching is still required to be carried out on the query result to obtain a target query result. The checking can be to judge whether the data volume of the query result exceeds a set number threshold, if so, the checking is passed, otherwise, the checking is not passed; or judging whether the similarity threshold of the n query results and the target data exceeds a set similarity threshold, if so, passing the verification, otherwise, failing to pass the verification, and n is less than or equal to the number of the query results. The setting may be performed as the case may be.
By checking the query result and determining that the computing capability of the computing module is not enough to obtain an accurate result when the check fails, the query result can be continuously subjected to further matching query, the accuracy of the query result is improved, and the query result is directly determined as a target query result when the check passes, so that the query data volume of a query party is reduced, and the query efficiency is improved.
Optionally, the obtaining target data to be queried includes: performing feature extraction on the acquired data to acquire target data; and the format of the data in the storage module is the same as that of the target data.
The collected data may include at least one of: text, images, audio, and video. And performing feature extraction on the acquired data, and extracting feature information from the acquired data. The feature extraction method may include extracting key data from the acquired data by using a pre-trained feature extraction model or using a preset template, and the like, and determining the key data as target data. For example, the collected data is text, and the target data may be set fields and corresponding attribute values in the text. The acquired data is an image, and the target data can be characteristic information of a target object in the image, such as the position of a license plate. The data collected is audio and the target data may be an audio characteristic in the audio, such as frequency. The collected data is a video, and the target data can be motion characteristic data, such as speed, of a moving object in the video. Further, the setting may be made as the case may be.
The format of the data in the storage module is the same as that of the target data, which indicates that the calculation module can directly perform data matching query in the storage module without processing the target data or the data in the storage module.
By extracting the characteristics of the acquired data and determining the target data, the characteristic information can be extracted from the acquired data for matching query, the query accuracy is improved, the query data volume is reduced, and the query efficiency is improved.
In a specific application scenario, optionally, the performing feature extraction on the collected data to obtain the target data includes: acquiring a face image; extracting the features of the face image to obtain a face feature vector; the storage module stores facial feature vectors of a plurality of users.
The collected data is a face image. The feature information of the face image may be a facial feature vector. The facial feature vector is used to describe features such as spatial structure features, texture features, shape features, and color features. The feature extraction method may include a model-based feature extraction method or a feature extraction method based on a machine learning model trained in advance, or the like. The data in the storage module are facial feature vectors of different users. The calculation module may directly compare the facial feature vectors, for example, calculate a vector distance between two target data and the data in the storage module, and determine that the data matches the target data when the vector distance is equal to or less than a set threshold, thereby determining as a query result.
The target data is configured into the facial feature vector of the face image to be applied to an application scene of face matching query, so that the efficiency and the accuracy of the face matching query can be improved.
According to the embodiment of the invention, the target data to be inquired is obtained and sent to each computing module of the storage device, and the inquiry result of each computing module is obtained, so that all unprocessed data can be avoided, the transmitted data amount is reduced, the transmission time of the inquiry result is shortened, the transmission speed of the inquiry result is increased, the inquiry result is quickly obtained, the subsequent processing is quickly carried out, and the operation speed is increased.
Example four
Fig. 4 is a schematic diagram of a data processing apparatus according to a fourth embodiment of the present invention. The fourth embodiment is a corresponding apparatus for implementing the data processing method provided by the third embodiment of the present invention, and the apparatus may be implemented in a software and/or hardware manner, and may be generally integrated into a computer device.
Accordingly, the apparatus of the present embodiment may include:
a target data obtaining module 410, configured to obtain target data to be queried;
a data query module 420, configured to send the target data to at least one computing module, so that each computing module performs data query in a corresponding storage module according to the target data;
a query result processing module 430, configured to receive query results fed back by each computing module, and process the query results to obtain a target query result, where the storage device according to claim 1 includes each computing module and each storage module, and the computing modules are connected to the corresponding storage modules.
According to the embodiment of the invention, the target data to be inquired is obtained and sent to each computing module of the storage device, and the inquiry result of each computing module is obtained, so that all unprocessed data can be avoided, the transmitted data amount is reduced, the transmission time of the inquiry result is shortened, the transmission speed of the inquiry result is increased, the inquiry result is quickly obtained, the subsequent processing is quickly carried out, and the operation speed is increased.
Further, the query result processing module 430 is specifically configured to: and inquiring data matched with the target data in each inquiry result to obtain a target inquiry result.
Further, the query result processing module 430 is specifically configured to: checking the query result fed back by each computing module; if the verification is passed, determining each query result as a target query result; and if the check is not passed, inquiring data matched with the target data in each inquiry result to obtain a target inquiry result.
Further, the target data obtaining module 410 is specifically configured to: performing feature extraction on the acquired data to acquire target data; and the format of the data in the storage module is the same as that of the target data.
Further, the target data obtaining module 410 is specifically configured to: acquiring a face image; extracting the features of the face image to obtain a face feature vector; the storage module stores facial feature vectors of a plurality of users.
The device can execute the data processing method provided by the embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
EXAMPLE five
Fig. 5 is a schematic structural diagram of a computer device according to a fifth embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 5 is only an example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention.
As shown in FIG. 5, computer device 12 is in the form of a general purpose computing device. The components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16. The computer device 12 may be a device that is attached to a bus.
Computer device 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer device 12 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a Compact disk Read-Only Memory (CD-ROM), Digital Video disk (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
Computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with computer device 12, and/or with any devices (e.g., network card, modem, etc.) that enable computer device 12 to communicate with one or more other computing devices. Such communication may be through an Input/Output (I/O) interface 22. Also, computer device 12 may communicate with one or more networks (e.g., Local Area Network (LAN), Wide Area Network (WAN)) via Network adapter 20. As shown, Network adapter 20 communicates with other modules of computer device 12 via bus 18. it should be understood that although not shown in FIG. 5, other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to microcode, device drivers, Redundant processing units, external disk drive Arrays (RAID) systems, tape drives, data backup storage systems, and the like.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, to implement the data processing method provided in the third embodiment of the present invention.
In an alternative embodiment, as shown in fig. 6, the computer device provided in this embodiment includes a processor and a storage apparatus 100, i.e. the storage apparatus is not a stand-alone device.
EXAMPLE six
A sixth embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the methods provided in all the embodiments of the present invention of the present application:
that is, the program when executed by the processor implements: acquiring target data to be inquired; sending the target data to at least one computing module so that each computing module can perform data query in a corresponding storage module according to the target data; and receiving the query result fed back by each computing module, and processing the query result to obtain a target query result, wherein the storage device according to the embodiment of the invention comprises each computing module and each storage module, and the computing modules are connected with the corresponding storage modules.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a RAM, a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, Radio Frequency (RF), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a LAN or a WAN, or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
EXAMPLE seven
Fig. 7 is a schematic structural diagram of a data processing system in a seventh embodiment of the present invention, which is applicable to a case where a computer device sends target data to a storage device for data query, as shown in fig. 7, a data processing system 300 includes the storage device 100 and a computer device 200. The storage device 100 may include a storage module 110 and a calculation module 120.
The storage apparatus 100 is a storage apparatus according to any embodiment of the present invention, the computer device 200 is an inquiry device according to any embodiment of the present invention, and the storage apparatus 100 is connected to the computer device 200;
the storage device 100 is configured to store data, perform query on the stored data according to target data sent by the computer device 200, and feed back a query result to the computer device 200; the computer device 200 is configured to send the target data to the storage apparatus 100, receive the query result fed back by the storage apparatus 100, and process the query result to obtain the target query result.
In the data processing system, the computer device 200 transfers most of the operations of matching queries to the storage device 100 for execution, and ideally (specifically, the query results are directly obtained by the computing module on the storage device 100), only the target data needs to be transmitted, and most of the interference results can be screened out even if the computing module in the storage device 100 has limited computing capability, so that the amount of transmitted data is greatly reduced, the time required for transmission is correspondingly reduced, and the overall operation speed is improved. The multiple computing modules in storage device 100 may begin searching and matching operations synchronously after receiving the same object. To ensure the accuracy of the results, the computer device 200 may perform final verification on each query result to finally determine the target query result.
In the embodiment of the present invention, the storage apparatus 100 does not belong to the computer device 200, and the storage apparatus 100 may be connected to the computer device 200 in a plug-in manner. The storage apparatus 100 is connected to the computer device 200, and specifically, the storage apparatus 100 is connected to a processor in the computer device 200.
According to the embodiment of the invention, most interference results are screened out by the storage device, the data volume transmitted between the storage device and the computer equipment is reduced, the time required by transmission is correspondingly reduced, and the system operation speed is increased.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (13)
1. A memory device, comprising: the device comprises at least one storage module and a calculation module corresponding to each storage module, wherein the storage module is connected with the corresponding calculation module;
the storage module is used for storing data;
and the computing module is used for processing data in the connected storage modules.
2. A data processing method applied to the storage apparatus according to claim 1, comprising:
acquiring target data to be processed through a computing module;
and processing data according to the target data in the connected storage modules through the computing module.
3. The method of claim 2, wherein the processing data according to the target data comprises:
inquiring the target data and feeding back an inquiry result; or
And acquiring the type of the target data, and storing the target data so as to store the target data in a storage module matched with the type.
4. The method of claim 2 or 3, further comprising:
and image processing is carried out on the image data in the connected storage modules through the calculation module.
5. A data processing method, comprising:
acquiring target data to be inquired;
sending the target data to at least one computing module so that each computing module can perform data query in a corresponding storage module according to the target data;
receiving the query result fed back by each computing module, and processing the query result to obtain a target query result, wherein the storage device according to claim 1 comprises each computing module and each storage module, and the computing modules are connected with the corresponding storage modules.
6. The method of claim 5, wherein the performing the processing to obtain the target query result comprises:
and inquiring data matched with the target data in each inquiry result to obtain a target inquiry result.
7. The method of claim 5, wherein the performing the processing to obtain the target query result comprises:
checking the query result fed back by each computing module;
if the verification is passed, determining each query result as a target query result;
and if the check is not passed, inquiring data matched with the target data in each inquiry result to obtain a target inquiry result.
8. The method of claim 5, wherein the obtaining target data to be queried comprises:
performing feature extraction on the acquired data to acquire target data; and the format of the data in the storage module is the same as that of the target data.
9. The method of claim 8, wherein the performing feature extraction on the collected data to obtain target data comprises:
acquiring a face image;
extracting the features of the face image to obtain a face feature vector; the storage module stores facial feature vectors of a plurality of users.
10. A data processing apparatus, comprising:
the target data acquisition module is used for acquiring target data to be inquired;
the data query module is used for sending the target data to at least one computing module so as to enable each computing module to perform data query in a corresponding storage module according to the target data;
the query result processing module is configured to receive the query result fed back by each computing module, and process the query result to obtain a target query result, wherein the storage device according to claim 1 includes each computing module and each storage module, and the computing modules are connected to the corresponding storage modules.
11. A computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor being coupled to the storage device of claim 1, characterized in that the processor implements the data processing method of any of claims 5-9 when executing the program.
12. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the data processing method of any one of claims 5 to 9.
13. A data processing system, comprising: a storage apparatus as claimed in claim 1, and a computer device as claimed in claim 11, the storage apparatus being connected to the computer device;
the storage device is used for storing data, inquiring in the stored data according to the target data sent by the computer equipment, and feeding back an inquiry result to the computer equipment;
and the computer equipment is used for sending the target data to the storage device, receiving the query result fed back by the storage device and processing the query result to obtain a target query result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011311360.5A CN112380169A (en) | 2020-11-20 | 2020-11-20 | Storage device, data processing method, device, apparatus, medium, and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011311360.5A CN112380169A (en) | 2020-11-20 | 2020-11-20 | Storage device, data processing method, device, apparatus, medium, and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112380169A true CN112380169A (en) | 2021-02-19 |
Family
ID=74585921
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011311360.5A Pending CN112380169A (en) | 2020-11-20 | 2020-11-20 | Storage device, data processing method, device, apparatus, medium, and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112380169A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115098548A (en) * | 2022-07-21 | 2022-09-23 | 南宁沃众科技有限公司 | Data decision method, system and cloud platform |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109408508A (en) * | 2018-09-30 | 2019-03-01 | 广州恩业电子科技有限公司 | The storage and querying method of monitoring data, device and computer equipment |
| CN110457256A (en) * | 2019-08-01 | 2019-11-15 | 大众问问(北京)信息科技有限公司 | Date storage method, device, computer equipment and storage medium |
| CN110737678A (en) * | 2019-10-23 | 2020-01-31 | 北京锐安科技有限公司 | data searching method, device, equipment and storage medium |
| CN111078723A (en) * | 2019-11-29 | 2020-04-28 | 支付宝(杭州)信息技术有限公司 | Data processing method and device for block chain browser |
| CN111190991A (en) * | 2019-12-10 | 2020-05-22 | 华能集团技术创新中心有限公司 | Unstructured data transmission system and interaction method |
| CN111190992A (en) * | 2019-12-10 | 2020-05-22 | 华能集团技术创新中心有限公司 | Mass storage method and storage system for unstructured data |
| CN111488114A (en) * | 2019-01-28 | 2020-08-04 | 北京灵汐科技有限公司 | Reconfigurable processor architecture and computing device |
-
2020
- 2020-11-20 CN CN202011311360.5A patent/CN112380169A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109408508A (en) * | 2018-09-30 | 2019-03-01 | 广州恩业电子科技有限公司 | The storage and querying method of monitoring data, device and computer equipment |
| CN111488114A (en) * | 2019-01-28 | 2020-08-04 | 北京灵汐科技有限公司 | Reconfigurable processor architecture and computing device |
| CN110457256A (en) * | 2019-08-01 | 2019-11-15 | 大众问问(北京)信息科技有限公司 | Date storage method, device, computer equipment and storage medium |
| CN110737678A (en) * | 2019-10-23 | 2020-01-31 | 北京锐安科技有限公司 | data searching method, device, equipment and storage medium |
| CN111078723A (en) * | 2019-11-29 | 2020-04-28 | 支付宝(杭州)信息技术有限公司 | Data processing method and device for block chain browser |
| CN111190991A (en) * | 2019-12-10 | 2020-05-22 | 华能集团技术创新中心有限公司 | Unstructured data transmission system and interaction method |
| CN111190992A (en) * | 2019-12-10 | 2020-05-22 | 华能集团技术创新中心有限公司 | Mass storage method and storage system for unstructured data |
Non-Patent Citations (3)
| Title |
|---|
| 钱磊;原昊;赵锦明;李祥;吴东;谢向辉;: "基于蚁群平台的智能存储技术及应用", 计算机工程与设计, no. 05, 31 May 2017 (2017-05-31), pages 1 - 7 * |
| 钱磊;原昊;赵锦明;李祥;吴东;谢向辉;: "基于蚁群平台的智能存储技术及应用", 计算机工程与设计, no. 05, pages 1 - 3 * |
| 高玉平;: "海量图书检索信息的快速查询***优化设计研究", 现代电子技术, no. 06, pages 299 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115098548A (en) * | 2022-07-21 | 2022-09-23 | 南宁沃众科技有限公司 | Data decision method, system and cloud platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109309630B (en) | Network traffic classification method and system and electronic equipment | |
| CN107992596B (en) | Text clustering method, text clustering device, server and storage medium | |
| US11182594B2 (en) | Face image retrieval methods and systems, photographing apparatuses, and computer storage media | |
| CN106951484B (en) | Picture retrieval method and device, computer equipment and computer readable medium | |
| CN112184508B (en) | Student model training method and device for image processing | |
| CN112016638B (en) | Method, device and equipment for identifying steel bar cluster and storage medium | |
| CN111325156B (en) | Face recognition method, device, equipment and storage medium | |
| CN111343161A (en) | Abnormal information processing node analysis method, abnormal information processing node analysis device, abnormal information processing node analysis medium and electronic equipment | |
| CN110610127B (en) | Face recognition method and device, storage medium and electronic equipment | |
| CN109885597B (en) | User grouping processing method and device based on machine learning and electronic terminal | |
| WO2020155790A1 (en) | Method and apparatus for extracting claim settlement information, and electronic device | |
| CN111950279B (en) | Entity relationship processing method, device, equipment and computer readable storage medium | |
| CN111209431A (en) | Video searching method, device, equipment and medium | |
| CA3052846A1 (en) | Character recognition method, device, electronic device and storage medium | |
| CN112085701A (en) | Face ambiguity detection method and device, terminal equipment and storage medium | |
| WO2022134881A1 (en) | Data processing method, data processing apparatus, computer device, and non-transitory storage medium | |
| CN111694866A (en) | Data searching and storing method, data searching system, data searching device, data searching equipment and data searching medium | |
| CN110909817B (en) | Distributed clustering method and system, processor, electronic device and storage medium | |
| CN109800215B (en) | Bidding processing method and device, computer storage medium and terminal | |
| CN117011581A (en) | Image recognition method, medium, device and computing equipment | |
| US10346716B2 (en) | Fast joint template machining | |
| CN112380169A (en) | Storage device, data processing method, device, apparatus, medium, and system | |
| CN111738290B (en) | Image detection method, model construction and training method, device, equipment and medium | |
| US20140172874A1 (en) | Intelligent analysis queue construction | |
| US20230186613A1 (en) | Sample Classification Method and Apparatus, Electronic Device and Storage Medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |