CN107070987B - Data acquisition method and system for distributed object storage system - Google Patents
Data acquisition method and system for distributed object storage system Download PDFInfo
- Publication number
- CN107070987B CN107070987B CN201710118848.8A CN201710118848A CN107070987B CN 107070987 B CN107070987 B CN 107070987B CN 201710118848 A CN201710118848 A CN 201710118848A CN 107070987 B CN107070987 B CN 107070987B
- Authority
- CN
- China
- Prior art keywords
- data
- request
- server
- sub
- metadata sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a data acquisition method and a data acquisition system for a distributed object storage system. Wherein, the method comprises the following steps: sending a first request to a server according to a preset rule, wherein the first request is used for requesting to acquire data of a first metadata sub-table in a distributed object storage system; receiving data of a first metadata sub-table returned by the server; and sequentially sending second requests for requesting data of other metadata sub-tables, and receiving data returned by the server until the server returns an end mark. The invention solves the technical problem of low resource utilization rate when a large amount of data is acquired.
Description
Technical Field
The invention relates to the field of data processing, in particular to a data acquisition method and a data acquisition system for a distributed object storage system.
Background
Distributed object storage is a main scheme for solving the problem of mass data storage at present, and in order to meet the requirement of data management, a distributed object storage system generally provides a traversal function for a user, and metadata of files or objects stored in the system can be acquired through the traversal function. One typical implementation is as follows: establishing a large number of metadata sub-tables in advance, wherein the metadata sub-tables are stored on different nodes (such as storage devices) in the system so as to prevent the data amount stored on a single node from exceeding the upper limit of the single storage device in the system; and dispersing the metadata into different metadata sub-tables according to a specific rule, wherein the data inside the metadata sub-tables are ordered, and the metadata sub-tables are unordered. In the scheme, the perception of the client to the metadata sub-tables is completely shielded, when List operation is carried out, the system pulls data from all the metadata sub-tables, carries out merging and sequencing and sends the pulling result to the client. In the above scheme, because the data size is huge, after the data is pulled, the data is sorted, rather than the full data returned to the client within a single query, under the ordered condition, the client can determine a flag according to the last data of the last query, and then let the system continue to return the data after that flag.
However, when the above method is used for full traversal, a large number of metadata sub-tables are required to store metadata of massive data, it is time-consuming to query and merge sort the large number of metadata sub-tables, and a large number of useless sorts may need to be repeated. For example, there are N metadata sub-tables in the system, and the user client can only receive Y records at a time. When the client needs to traverse all records, the process would be: the client sends out List operation (limited to return Y records) to the system, the system inquires the first Y records to each metadata sub-table in the N metadata sub-tables, sorts the N records by Y records, returns Y records, and then discards the rest (N-1) records by Y records, namely the waste ratio is as high as (N-1)/N. The client would traverse all records in this manner. The metadata is traversed, which is a common operation (such as active-standby file comparison, file cleaning, and the like) in the daily operation and maintenance process, so that such an inefficient operation scheme greatly limits the scale expansion of the cluster. During traversal operation, optionally, a sorting scheme is used, and the global order of the returned data list can be ensured through the scheme, but the scheme greatly consumes internal bandwidth, memory or a CPU, and the efficiency of acquiring data is low.
Aiming at the problem of low resource utilization rate when a large amount of data is obtained, an effective solution is not provided at present.
Disclosure of Invention
The embodiment of the invention provides a data acquisition method and a data acquisition system for a distributed object storage system, which at least solve the technical problem of low resource utilization rate when a large amount of data is acquired.
According to an aspect of an embodiment of the present invention, there is provided a data acquisition method for a distributed object storage system, including: sending a first request to a server according to a preset rule, wherein the first request is used for requesting to acquire data of a first metadata sub-table in a distributed object storage system; receiving data of the first metadata sub-table returned by the server; and sequentially sending second requests for requesting data of other metadata sub-tables, and receiving data returned by the server until the server returns an end mark.
Further, the preset rule includes: when a first request sent to a server does not contain a metadata sub-table number, the first request indicates that the metadata sub-table data needs to be subjected to sub-table crossing sorting; when the first request sent to the server contains the metadata sub-table number, the first request does not need to carry out cross-sub-table sorting on the metadata sub-table data.
Further, sequentially sending a second request for requesting data of other metadata sub-tables and receiving data returned by the server until the server returns an end mark comprises: sending a No. P second request to the server, wherein P is a natural number larger than 1; receiving data of a metadata sub-table returned by the server based on the P second request; and after a predetermined operation is performed on the data of the metadata sub-table, sending a (P +1) th second request to the server until the data in all the metadata sub-tables are acquired.
Further, the server sends a pth second request, where P is a natural number greater than 1; receiving data of the metadata sub-table returned by the server based on the pth second request further comprises: and sending a second request for acquiring the data in the P-th metadata sub-table to the server, and when the server returns data without an end mark of the sub-table, acquiring the last data in the data as a starting mark by the previous request, and continuously requesting the data of the metadata sub-table from the server until the end mark of the metadata sub-table is returned in the acquired data. Further, the data acquisition method is applied to acquisition of full-scale data.
Further, before sending the first request to the server according to the preset rule, the method further includes: the number of sub-tables of the metadata is requested from the server.
According to another aspect of the embodiments of the present invention, there is also provided a data acquisition method for a distributed object storage system, including: receiving a first request sent by a client according to a preset rule, wherein the first request is used for requesting to acquire data of a first metadata sub-table in a distributed object storage system; returning the data of the first metadata sub-table to the client; receiving second requests which are sequentially sent by the client and used for requesting data of other metadata sub-tables; and returning an end mark after returning the data of other metadata sub-tables to the client.
Further, after receiving a second request for requesting data of other metadata sub-tables sequentially sent by the client, the method includes: acquiring an initial mark carried in the second request; and in the metadata sub-table indicated by the second request, acquiring data in the metadata sub-table from the next data corresponding to the starting mark.
According to another aspect of the embodiments of the present invention, there is provided a data acquisition system for a distributed object storage system, the system including: a client; a server; the client sends a first request to the server according to a preset rule, wherein the first request is used for requesting to acquire data of a first metadata sub-table in the distributed object storage system; returning the data of the first metadata sub-table to the client under the condition that the server receives the first request; and the client receives the data of the first metadata sub-table returned by the server, sequentially sends second requests for requesting the data of other metadata sub-tables, and receives the data returned by the server until the server returns an end mark.
Further, after returning the data of the first metadata sub-table to the client, the server is further configured to: receiving the second request; and the server returns the end mark after returning the data of other metadata sub-tables to the client.
In the embodiment of the present invention, a first request may be sent to a server according to a preset rule, where the first request is used to request to acquire data of a first metadata sub-table in a distributed object storage system, and receive data of the first metadata sub-table returned by the server, and then a client sequentially sends a second request for requesting data of other metadata sub-tables, and receives data returned by the server, until the server returns an end mark. According to the embodiment of the invention, the data request for acquiring the metadata sub-tables is sent to the server according to the preset rule, and the cross-sub-table sorting at the server is avoided, so that the data of a plurality of metadata sub-tables is acquired, the data acquisition efficiency and the resource utilization rate are improved, and the technical problem of low resource utilization rate when a large amount of data is acquired is solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a first flowchart of a data acquisition method for a distributed object storage system according to an embodiment of the present invention;
FIG. 2 is a second flowchart of an alternative data acquisition method for a distributed object storage system according to an embodiment of the present invention;
FIG. 3 is a flow chart of a data acquisition method for a client of a distributed object storage system according to an embodiment of the present invention;
FIG. 4 is a flowchart of an alternative data acquisition method for a server of a distributed object storage system according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an alternative data acquisition system for a distributed object storage system in accordance with embodiments of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Some terms in the present invention are explained:
distributed storage system: the data is stored on a plurality of independent devices in a scattered manner. An expandable system structure is adopted, a plurality of storage servers are used for sharing storage load, and a position server is used for positioning storage information.
Object: including file data and related attribute information, can be self-managed.
In accordance with an embodiment of the present invention, there is provided a method embodiment for data acquisition of a distributed object storage system, it should be noted that the steps illustrated in the flowchart of the accompanying drawings may be performed in a computer system such as a set of computer-executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
Fig. 1 is a flowchart of a first data acquisition method for a distributed object storage system according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
step S102, sending a first request to a server according to a preset rule, wherein the first request is used for requesting to acquire data of a first metadata sub-table in a distributed object storage system;
step S104, receiving the data of the first metadata sub-table returned by the server;
and step S106, sequentially sending second requests for requesting data of other metadata sub-tables, and receiving data returned by the server until the server returns an end mark.
Through the embodiment, the client side can send the first request to the server according to the preset rule, wherein the first request is used for requesting to acquire the data of the first metadata sub-table in the distributed object storage system and receive the data of the first metadata sub-table returned by the server, and then the client side sequentially sends the second requests for requesting the data of other metadata sub-tables and receives the data returned by the server until the server returns the end mark. According to the embodiment of the invention, the data for acquiring the metadata sub-tables can be sent to the server according to the preset rule, the data of the plurality of metadata sub-tables in the server can be acquired in sequence, the client can effectively acquire the data of the plurality of metadata sub-tables, the resource utilization rate is improved, and the technical problem of low resource utilization rate when a large amount of data is acquired is solved.
The client of the above embodiment may include a smart terminal, which may include a smart phone, a PC, and the like, and the client may be a device used by a user.
The server of the above embodiments may be a place for storing data (e.g., files, metadata), where a large amount of data may be stored and stored in respective storage devices of the server, and the server may include a plurality of storage devices, each storage device may correspond to a storage node, and each storage node may include one or more metadata sub-tables.
Alternatively, a plurality of data items may be stored in the metadata sub-table, the data items in each metadata sub-table may be arranged in order, and the respective metadata sub-tables may not be ordered. In sorting the metadata sub-table, the metadata sub-table may be labeled, wherein the label may include a number, letter, symbol, and the like. Preferably, the indicia may be a number. Where the numbers of the metadata sub-tables are taken from an ordered set, such as a natural number.
Optionally, before sending the first request to the server according to the preset rule, the method further includes: the number of sub-tables of the metadata is requested from the server. In the embodiment of the present invention, the number of sub-tables of the metadata may be N. N may be a natural number equal to or greater than 1. Wherein, each storage node in the server can comprise a plurality of metadata sub-tables. Optionally, in the technical solution provided in step S102, a first request is sent to the server according to a preset rule, where the first request is used to request to acquire data of a first metadata sub-table in the distributed object storage system.
Optionally, for the first request, the data of the first metadata sub-table in the N metadata sub-tables may be obtained for the request, where the data may be the data item in the foregoing embodiment, when the data in the server needs to be processed, request information may be sent first to obtain the data that can be processed, the data that can be stored in each storage device in each server is limited, the data that is processed by the client is also limited, and when the client sends the first request, the parameter of the number of the data that can be processed may be carried in the first request.
Wherein, the preset rule may include: when the first request sent to the server does not contain the metadata sub-table number, the first request indicates that the metadata sub-table data needs to be subjected to sub-table crossing sorting; when the first request sent to the server contains the metadata sub-table number, the request does not need to sort the metadata sub-table data across sub-tables. That is, after the first request is sent, whether the first request includes the number information of the metadata sub-table is judged, if yes, the metadata sub-table does not need to be sorted, the first request can carry the data of the first metadata sub-table according to the number information, and if not, the metadata sub-tables need to be sorted, so that the client can conveniently obtain the data in the metadata sub-tables.
In the technical solution provided in step S104, data of the first metadata sub-table returned by the server is received. And the server sends the data of the first metadata sub-table in the sorted metadata sub-tables to the client. The server may send the number information of the metadata sub-table to the client, and the client may receive the number information of the metadata sub-table.
In the technical solution provided in step S106, a second request for requesting data of other metadata sub-tables is sequentially sent, and data returned by the server is received until the server returns an end mark.
Optionally, the above embodiment may be that, in a plurality of metadata sub-tables, after receiving the second request, the server may send data of each metadata sub-table to the client. Optionally, the data of each metadata sub-table may be marked in advance, that is, the data in the metadata sub-table may be sorted; after receiving the second request, the data in the metadata sub-table may be sequentially sent to the client.
Alternatively, when transmitting data in the metadata sub-table, a predetermined amount of data is transmitted at a time. The predetermined number may be a predetermined number, and the predetermined number may be determined according to the processing capability of the client, that is, the number of data that can be processed by the client at one time may not exceed the predetermined number when receiving data, the server may send data within the predetermined number each time when sending data, and the client may inform the server of the predetermined number of parameters when sending a request; each time a data sending request is received, a predetermined amount of data may be sent to the client, and after the request is received next time, the predetermined amount of data may be obtained again, and optionally, the amount of data obtained this time and the amount of data obtained last time may be the same.
In another alternative embodiment, sequentially sending second requests for requesting data of other metadata sub-tables and receiving data returned by the server until the server returns an end mark comprises: sending a No. P second request to the server, wherein P is a natural number larger than 1; receiving data of the metadata sub-table returned by the server based on the P second request; and after the predetermined operation is performed on the data of the metadata sub-table, sending a (P +1) th second request to the server until the data in all the metadata sub-tables are acquired.
By the embodiment, the data in the metadata sub-table can be continuously acquired, and after the acquired data is processed, the data of the next metadata sub-table can be acquired. Where P is an unlimited number, when sending the second request, the metadata sub-table after the first request may be started until the data acquiring all the metadata sub-tables is finished.
For the above embodiments, the predetermined operation may include sorting the data, for example, cutting a file or data into fixed pieces of data. The first data may be a predetermined amount of data of the above embodiment, and the data may be obtained from the metadata sub-table of the transmitted second request.
In another optional implementation manner, a pth second request is sent to the server, where P is a natural number greater than 1; receiving the data of the metadata sub-table returned by the server based on the pth second request further comprises: and sending a second request for acquiring the data in the P-th metadata sub-table to the server, and when the server returns the data without the end mark of the sub-table, acquiring the last data in the data as a starting mark by the previous request, and continuously requesting the data of the metadata sub-table from the server until the end mark of the sub-table is returned in the acquired data.
Optionally, the foregoing embodiment may process data of multiple (e.g., N) metadata sub-tables, and after receiving number information of a metadata sub-table sent by the server, sequentially obtain data of other metadata sub-tables according to the number information, in the foregoing embodiment, a P-th second request is sent, all data of the metadata sub-table returned by the server are detected, and when there is no end mark in the returned data (i.e., the data of the metadata sub-table is not completely obtained), the last data in the returned data may be selected as a start mark. Until all the data of the metadata sub-tables are acquired, and according to the above embodiment, the data of other metadata sub-tables are sequentially acquired until all the data of the N metadata sub-tables are acquired.
In another alternative embodiment, the second request further records a flag indicating that the last data in the predetermined amount of data was last received.
Optionally, the mark of the above embodiment may include a plurality of types, such as a separator, a line, a letter, and the like, where the mark is movable, that is, each time the second request is sent, mark information of last data in the predetermined amount of data received last time may be carried to inform the server of a last position of data sent last time in response to the second request, where the position may have mark information, and by using the mark information, the server may quickly query a position of last data in the predetermined amount of data sent last time. Optionally, after the second request is obtained, according to the position of the last data recorded by the marker information, starting from the next data of the last data recorded by the marker information, the data of the other metadata sub-tables may be sent to the client again.
Preferably, the data acquisition method in the above embodiment may be applied to acquisition of full-scale data. The server may store full data, and the client may store the full data in different metadata sub-tables when acquiring the full data stored in the server.
Through the embodiment, the client can acquire the marking information of the plurality of metadata sub-tables and continuously acquire the data with the preset quantity in the metadata sub-tables according to the marking information, so that redundant query and sequencing of the server are avoided, and the data transmission efficiency is improved.
Fig. 2 is a second flowchart of an alternative data acquisition method for a distributed object storage system according to an embodiment of the present invention, as shown in fig. 2, the method includes the following steps:
step S201, receiving a first request sent by a client according to a preset rule, wherein the first request is used for requesting to acquire data of a first metadata sub-table in a distributed object storage system;
step S203, returning the data of the first metadata sub-table to the client;
step S205, receiving a second request for requesting data of other metadata sub-tables sent by the client in sequence;
step S207, after returning the data of other metadata sub-tables to the client, returns an end mark.
Through the above embodiment, the server may receive a first request sent by the client according to a preset rule, where the first request is used to request to acquire data of a first metadata sub-table in the distributed object storage system and return the data of the first metadata sub-table to the client, after the client sends a second request, may receive a second request, which is sent by the client in sequence, to request data of other metadata sub-tables, and then, after returning the data of other metadata sub-tables to the client, return an end mark. According to the embodiment, the server can send the data of the plurality of metadata sub-tables to the client according to the request of the client, and can directly send the data in other metadata sub-tables according to the second request, so that the work of the server is reduced, the data transmission efficiency between the client and the server is improved, and the resource utilization efficiency between the server and the client is improved.
Optionally, after receiving a second request for requesting data of other metadata sub-tables sequentially sent by the client, the method includes: acquiring an initial mark carried in the second request; and in the metadata sub-table indicated by the second request, acquiring data in the metadata sub-table from the next data corresponding to the start mark.
Optionally, after receiving the second request, the data of the metadata sub-table corresponding to the second request may be sent to the client, and in the second request, the client sends data information of the metadata sub-table that can be processed to the server, so that the server sends corresponding data according to the second request.
Optionally, while returning the data of the obtained metadata sub-table to the client, the method further includes: and informing the client whether the data in the metadata sub-table indicated by the second request is acquired completely.
After the predetermined amount of data is acquired, whether all the data in the metadata sub-table is acquired can be queried, so that after the client is informed that all the data in the metadata sub-table is acquired, an end mark can be sent, and the client acquires the data of the next metadata sub-table after receiving the end mark until all the data in the metadata sub-table is acquired. The end mark may be notification information, for example, a notification box, in which the user may be notified that all the data in the data table has been acquired. In this way, the client does not have to send the request information again.
The following are specific embodiments according to the present invention.
Fig. 3 is a flowchart of a data acquisition method for a client of a distributed object storage system according to an embodiment of the present invention, and as shown in fig. 3, the method includes:
in step S301, it is determined whether the metadata sub-table has already been numbered.
If yes, go to step S303, otherwise go to step S317.
In the distributed storage system, a client can receive the number of the metadata sub-table through a parameter to perform traversal operation on the specific metadata sub-table.
In step S303, the client requests to acquire Y data items of the first metadata sub-table in the full amount of data.
Step S305, requesting the server for Y data items from the tag in the metadata sub-table.
When the distributed storage server (i.e. the server in the above embodiment) system receives the request, the number of the metadata sub-table is extracted from the parameter, then the metadata sub-table corresponding to the number is queried (no cross sub-table sorting is performed), and the result is returned to the client.
In step S307, the client processes Y data items.
And after the client finishes processing the Y items, taking the last item of the Y items as a marked item, and continuously requesting the Y items of the metadata sub-table after the marked item from the server.
In step S309, it is determined whether it is the last data item in the metadata sub-table.
If so, go to step S311, otherwise, go to step S313.
Alternatively, if a metadata sub-table has returned all data items, the client continues to perform the correlation operation with the number of the next sub-table.
Step S311, request to acquire the first data item of the next metadata sub-table.
Step 313, marking the last of the Y data items acquired this time.
After the marking, return is made to step S305.
In step S315, it is determined whether the metadata is the last metadata sub-table.
If yes, step S321 is executed, otherwise, the process returns to step S305.
Step S317, sequentially acquiring Y items of data in the full amount of data.
Step S319, determining whether all the data in the full amount of data is completely acquired.
If yes, step S321 is executed, otherwise, the process returns to step S317.
In step S321, the client ends the operation.
The scheme optimizes application scenes insensitive to the global sequence, avoids redundant query and sequencing of a server side by exposing the marking information (such as numbers) of the sub-tables to the client side, and improves the efficiency.
Fig. 4 is a flowchart of an alternative data obtaining method for a server of a distributed object storage system according to an embodiment of the present invention, and as shown in fig. 4, the method includes the following steps:
step S402, receiving a first request operation of a client for acquiring a first metadata sub-table.
Step S404, sending the data of the first metadata sub-table to the client.
Step S406, receiving a second request operation for the client to obtain data of other metadata sub-tables.
The second request operation carries a metadata sub-table corresponding to the acquired data item and mark information, and the mark information records the position of the data to be acquired.
Step S408, sending the data in the other metadata sub-tables corresponding to the second request operation to the client, until the data in all metadata sub-tables are sent.
Optionally, when sending the data, the server (i.e. the server described above) may notify the client whether all the data in the metadata sub-table is acquired.
By the implementation mode, unnecessary query and sequencing are avoided, the data transmission speed is accelerated, resources are saved, waste of a system on a memory, a CPU or an internal bandwidth is reduced, the global sequence of the whole metadata sub-table is not required to be guaranteed, and the client side is allowed to process the data of the metadata sub-table one by one.
FIG. 5 is a schematic diagram of an alternative data acquisition system for a distributed object storage system according to an embodiment of the present invention, as shown in FIG. 5, the system comprising: a client 51; a server 53; the client 51 sends a first request to the server 53 according to a preset rule, wherein the first request is used for requesting to acquire data of a first metadata sub-table in the distributed object storage system; in the case where the server 53 receives the first request, returning data of the first metadata sub-table to the client 51; the client 51 receives the data of the first metadata sub-table returned by the server 53, sequentially sends a second request for requesting the data of other metadata sub-tables, and receives the data returned by the server 53 until the server 53 returns an end mark.
With the above embodiment, a first request for requesting to acquire data of a first metadata sub-table in the distributed object storage system may be sent to the server 53 through the client 51, and in a case where the server 53 receives the first request, the data of the first metadata sub-table is returned to the client 51, and then the client 51 may receive the data of the first metadata sub-table returned by the server 53, sequentially send second requests for requesting data of other metadata sub-tables, and receive the data returned by the server 53 until the server 53 returns an end mark. .
In the embodiment of the invention, a data request for acquiring the metadata sub-tables can be sent to the server according to the preset rule, and cross-sub-table sequencing at the server is avoided, so that the data of a plurality of metadata sub-tables is acquired, the data acquisition efficiency and the resource utilization rate are improved, and the technical problem of low resource utilization rate when a large amount of data is acquired is solved.
Further, after returning the data of the first metadata sub-table to the client 51, the server 53 is further configured to: receiving a second request; the server 53 returns an end flag after returning the data of the other metadata sub-table to the client 51.
Optionally, the preset rule includes: when the first request sent to the server 53 does not contain the metadata sub-table number, it indicates that the first request needs to perform sub-table-crossing sorting on the metadata sub-table data; when the first request sent to the server 53 includes a metadata sub-table number, this indicates that the request does not need to sort the metadata sub-table data across sub-tables.
Preferably, the data acquisition system is adapted for use in the acquisition of a full amount of data.
With the above embodiment, the server 53 may receive the first request sent by the client 51 and return the data of the first metadata sub-table to the client 51, and after the client sends the second request, may receive the second requests sequentially sent by the client 51 to request the data of other metadata sub-tables, and then return the end flag after returning the data of other metadata sub-tables to the client 51. According to the above embodiment, the server 53 may send the data of the plurality of metadata sub-tables to the client 51 according to the request of the client 51, and may send the data in the other metadata sub-tables according to the second request, thereby avoiding sorting across sub-tables, thereby reducing the computational burden of the server 53, improving the efficiency of data transmission between the client 51 and the server 53, and improving the resource utilization efficiency of the server 53 and the client 51.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (9)
1. A data acquisition method for a distributed object storage system, comprising:
sending a first request to a server according to a preset rule, wherein the first request is used for requesting to acquire data of a first metadata sub-table in a distributed object storage system;
receiving data of the first metadata sub-table returned by the server;
sequentially sending second requests for requesting data of other metadata sub-tables, and receiving data returned by the server until the server returns an end mark,
the preset rules include: when a first request sent to a server does not contain a metadata sub-table number, the first request indicates that the metadata sub-table data needs to be subjected to sub-table crossing sorting; when the first request sent to the server contains the metadata sub-table number, the first request does not need to carry out cross-sub-table sorting on the metadata sub-table data.
2. The method of claim 1, wherein sequentially sending second requests for requesting data of other metadata sub-tables and receiving data returned by the server until the server returns an end marker comprises:
sending a No. P second request to the server, wherein P is a natural number larger than 1;
receiving data of a metadata sub-table returned by the server based on the P second request;
and after a predetermined operation is performed on the data of the metadata sub-table, sending a (P +1) th second request to the server until the data in all the metadata sub-tables are acquired.
3. The data acquisition method according to claim 2,
sending a No. P second request to the server, wherein P is a natural number greater than 1;
receiving data of the metadata sub-table returned by the server based on the pth second request further comprises:
sending a second request for obtaining data in the P-th metadata sub-table to the server,
when the server returns the end mark without the metadata sub-table in the data, the last data in the data obtained by the above request is used as the start mark, and the server continues to request the data of the metadata sub-table until the end mark of the sub-table is returned in the obtained data.
4. The data acquisition method as claimed in claim 1 or claim 2, wherein the data acquisition method is applied to acquisition of a full amount of data.
5. The data acquisition method according to claim 1, before sending the first request to the server according to the preset rule, further comprising: the number of sub-tables of the metadata is requested from the server.
6. A data acquisition method for a distributed object storage system, comprising:
receiving a first request sent by a client according to a preset rule, wherein the first request is used for requesting to acquire data of a first metadata sub-table in a distributed object storage system;
returning the data of the first metadata sub-table to the client;
receiving second requests which are sequentially sent by the client and used for requesting data of other metadata sub-tables;
after returning the data of other metadata sub-tables to the client, returning an end mark,
the preset rules include: when the first request does not contain the metadata sub-table number, the first request indicates that the metadata sub-table data needs to be subjected to cross sub-table sorting; when the first request contains the metadata sub-table number, the first request does not need to sort the metadata sub-table data across sub-tables.
7. The data acquisition method according to claim 6, wherein after receiving a second request for requesting data of other metadata sub-tables sequentially sent by the client, the method comprises:
acquiring an initial mark carried in the second request;
and in the metadata sub-table indicated by the second request, acquiring data in the metadata sub-table from the next data corresponding to the starting mark.
8. A data acquisition system for a distributed object storage system, the system comprising:
a client;
a server;
the client sends a first request to the server according to a preset rule, wherein the first request is used for requesting to acquire data of a first metadata sub-table in the distributed object storage system;
returning the data of the first metadata sub-table to the client under the condition that the server receives the first request;
the client receives the data of the first metadata sub-table returned by the server, sequentially sends second requests for requesting the data of other metadata sub-tables, and receives the data returned by the server until the server returns an end mark,
the preset rules include: when a first request sent to a server does not contain a metadata sub-table number, the first request indicates that the metadata sub-table data needs to be subjected to sub-table crossing sorting; when the first request sent to the server contains the metadata sub-table number, the first request does not need to carry out cross-sub-table sorting on the metadata sub-table data.
9. The data acquisition system of claim 8, wherein after returning the data of the first sub-table of metadata to the client, the server is further configured to:
receiving the second request;
and the server returns the end mark after returning the data of other metadata sub-tables to the client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710118848.8A CN107070987B (en) | 2017-03-01 | 2017-03-01 | Data acquisition method and system for distributed object storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710118848.8A CN107070987B (en) | 2017-03-01 | 2017-03-01 | Data acquisition method and system for distributed object storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107070987A CN107070987A (en) | 2017-08-18 |
| CN107070987B true CN107070987B (en) | 2020-02-14 |
Family
ID=59621805
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710118848.8A Active CN107070987B (en) | 2017-03-01 | 2017-03-01 | Data acquisition method and system for distributed object storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107070987B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107566514A (en) * | 2017-09-22 | 2018-01-09 | 郑州云海信息技术有限公司 | A kind of method that metadata request quantity is controlled in distributed memory system |
| CN108735224B (en) * | 2018-04-11 | 2021-04-30 | 北京小唱科技有限公司 | Audio correction method and device based on distributed structure |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105912609A (en) * | 2016-04-06 | 2016-08-31 | 中国农业银行股份有限公司 | Data file processing method and device |
| CN106202261A (en) * | 2016-06-29 | 2016-12-07 | 浪潮(北京)电子信息产业有限公司 | The distributed approach of a kind of data access request and engine |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10095807B2 (en) * | 2015-04-28 | 2018-10-09 | Microsoft Technology Licensing, Llc | Linked data processor for database storage |
-
2017
- 2017-03-01 CN CN201710118848.8A patent/CN107070987B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105912609A (en) * | 2016-04-06 | 2016-08-31 | 中国农业银行股份有限公司 | Data file processing method and device |
| CN106202261A (en) * | 2016-06-29 | 2016-12-07 | 浪潮(北京)电子信息产业有限公司 | The distributed approach of a kind of data access request and engine |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107070987A (en) | 2017-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109741060B (en) | Information inquiry system, method, device, electronic equipment and storage medium | |
| CN105653537B (en) | Paging query method and device for database application system | |
| CN107515878B (en) | Data index management method and device | |
| CN107770229B (en) | Cloud file sharing method and system and cloud server | |
| CN110413845B (en) | Resource storage method and device based on Internet of things operating system | |
| CN107229429B (en) | Storage space management method and device | |
| CN103186666A (en) | Method, device and equipment for searching based on favorites | |
| CN109947759A (en) | A kind of data directory method for building up, indexed search method and device | |
| CN109947729A (en) | A kind of real-time data analysis method and device | |
| CN108154024B (en) | Data retrieval method and device and electronic equipment | |
| CN104462096A (en) | Public opinion monitoring and analysis method and device | |
| CN107070987B (en) | Data acquisition method and system for distributed object storage system | |
| CN104035943B (en) | Store the method and respective server of data | |
| CN111367870A (en) | Method, device and system for sharing picture book | |
| CN104092754B (en) | Document storage system and file memory method | |
| CN104636368A (en) | Data retrieval method and device and server | |
| CN111159131A (en) | Performance optimization method, device, equipment and computer readable storage medium | |
| CN103365892A (en) | Method and device for processing multiple contact objects | |
| CN108897858A (en) | The appraisal procedure and device, electronic equipment of distributed type assemblies index fragment | |
| CN104636384B (en) | A kind of method and device handling document | |
| CN106446080B (en) | Data query method, query service equipment, client equipment and data system | |
| CN110888840A (en) | File query method, device, equipment and medium in distributed file system | |
| CN112650964B (en) | Service processing method, equipment and machine readable storage medium | |
| CN110704481A (en) | Method and device for displaying data | |
| CN105095225A (en) | Method and apparatus for obtaining file data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210809 Address after: 518000 b1-1302, building B, Kexing Science Park, No. 15, Keyuan Road, Science Park community, Yuehai street, Nanshan District, Shenzhen, Guangdong Patentee after: Shenzhen aijieyun Technology Co.,Ltd. Address before: Floor 5, building a, Guangqi Cultural Plaza, No. 2899 Xietu Road, Xuhui District, Shanghai 200030 Patentee before: WANGSU SCIENCE & TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |