CN106713385A - Distributed storage redundant data compression method and system, client and server - Google Patents
Distributed storage redundant data compression method and system, client and server Download PDFInfo
- Publication number
- CN106713385A CN106713385A CN201510777060.9A CN201510777060A CN106713385A CN 106713385 A CN106713385 A CN 106713385A CN 201510777060 A CN201510777060 A CN 201510777060A CN 106713385 A CN106713385 A CN 106713385A
- Authority
- CN
- China
- Prior art keywords
- data block
- data
- client
- server
- block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
- H04L67/5651—Reducing the amount or size of exchanged application data
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a distributed storage redundant data compression method, a client and a server, relating to the technical field of distributed computing. The method comprises the steps that: the client packs upload data into a data block; the client generates the data block ID of the data block; the client sends the data block ID to the server, and the server judges whether the data block corresponding to the data block ID exists or not; the client receives a response of whether the data block corresponding to the data block ID exists returned by the server; according to the response, if the data block corresponding to the data block ID exists in the server, the client starts the transmission of a next data block; and if the data block corresponding to the data block ID does not exist in the server, the client transmits the data block corresponding to the data block ID to the server. While reliability of distributed storage is ensured, the network load is reduced, the server-side data redundancy is reduced from an application level, and the distributed energy storage efficiency is enhanced.
Description
Technical field
The present invention relates to Distributed Calculation field, more particularly to a kind of distributed storage redundant data
Compress technique.
Background technology
With the extensive use at home of cloud computing correlation technique, the distribution based on inexpensive hardware is deposited
Storage is widely applied in enterprise's inside and outside Data processing.
To ensure the global reliability of distributed memory system, typical distributed storage is the bottom of at present
A data storage generally by the way of data redundancy storage, for example, is retained multiple copies by layer,
And storage is on different main frames, with the lifting of space metathesis security.Based on the number that bottom is provided
According to safety guarantee, it is no longer needed for doing data redundancy treatment from application.Due to same text
Part stores many parts, in distributed storage the redundancy of application layer data can not only waste computing resource and
Internet resources, can also increase overall maintenance cost, it is necessary to be carried out to application layer redundant data necessary
Compression is processed.
Current distributed storage is compressed in application layer data, generally using first uploading, after pick the side of weight
Case, although this scheme can also discharge the memory space that redundant data is occupied, needs to take and leads to
Communication network resource and pick again operation concentrate on server end, overall efficiency can be reduced.For substantially
For product file, this defect is especially apparent.
The content of the invention
An embodiment of the present invention technical problem to be solved is：First upload the application layer for picking weight afterwards
The communication network resource that data compression method is present is wasted and overall efficiency is than relatively low problem.
A kind of one side according to embodiments of the present invention, there is provided distributed storage redundant data pressure
Compression method, including：Client is packaged into data block by data are uploaded；Client generates data block
Data block ID；Client sends to server data block ID, server is judged the number
Whether exist according to the corresponding data blocks of block ID；The number that client the reception server is returned
Whether there is response according to the corresponding data blocks of block ID；According to response, if the data block
The corresponding data blocks of ID are present in server, and client starts the transmission of next data block；
If the corresponding data blocks of data block ID are not present in server, client is by the data block
The corresponding transmission of data blocks of ID is to server.
A kind of other side according to embodiments of the present invention, there is provided distributed storage redundant digit
According to compression method, including：The data block ID that received server-side client sends；Server
End judges whether the corresponding data blocks of data block ID have been present；If data block ID correspondences
Data block existed, then server end by the already present feedback of the information of data block ID to visitor
Family end；If the corresponding data blocks of data block ID do not exist, server end notifies client
The corresponding data blocks of data block ID are uploaded, and after the corresponding data blocks of data block ID are received,
Data block ID recorded in having uploaded data block list.
Another aspect according to embodiments of the present invention, there is provided one kind is superfluous for distributed storage
The client of remaining data compression, including：Data block packaged unit, for data packing will to be uploaded
Into data block；Data block ID generation units, the data block ID for generating data block；Data
Block ID transmitting elements, for data block ID to be sent to server, make server judge the number
Whether exist according to the corresponding data blocks of block ID；Response receiving unit, services for receiving
Whether the corresponding data blocks of data block ID that device is returned have had response；Transmission of data blocks
Unit, for according to response, if the corresponding data blocks of data block ID are present in service
Device, then start the transmission of next data block；If the corresponding data blocks of data block ID are not
It is present in server, then by the corresponding transmission of data blocks of data block ID to server.
Another aspect according to embodiments of the present invention, there is provided one kind is superfluous for distributed storage
The server of remaining data compression, including：Data block ID receiving units, for receiving client
The data block ID of transmission；Data block judging unit, for judging the corresponding numbers of data block ID
Whether exist according to block；Information feedback unit, if for the corresponding data of data block ID
Block has been present, then information feedback unit by the already present feedback of the information of data block ID to client
End；If the corresponding data blocks of data block ID do not exist, information feedback unit notifies client
End uploads the corresponding data blocks of data block ID；Data block reception unit, for receiving data block
The corresponding data blocks of ID；Data block ID recording units, for receiving data block ID pairs
After the data block answered, data block ID recorded in having uploaded data block list.
A kind of other side according to embodiments of the present invention, there is provided distributed storage redundant data
Compressibility, including foregoing client and foregoing server.
The present invention at least has advantages below：
By the cooperation at client and server end, first judgement is target data before upload server
It is no to have existed in server end, if there is " precommpression " is then carried out, it is achieved thereby that to application
The compression of layer redundant data, is reduced to communication network resource and the occupancy of server end resource, lifting
The overall efficiency of distributed storage.
By referring to the drawings to the detailed description of exemplary embodiment of the invention, the present invention
Further feature and its advantage will be made apparent from.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will
The accompanying drawing to be used needed for embodiment or description of the prior art is briefly described, it is clear that
Ground, drawings in the following description are only some embodiments of the present invention, for the common skill in this area
For art personnel, without having to pay creative labor, can also be obtained according to these accompanying drawings
Obtain other accompanying drawings.
Fig. 1 is shown according to one of distributed storage redundant data compression method of the present invention implementation
The schematic flow sheet of example.
Fig. 2 shows another reality according to distributed storage redundant data compression method of the present invention
Apply the schematic flow sheet of example.
Fig. 3 shows another reality according to distributed storage redundant data compression method of the present invention
Apply the schematic flow sheet of example.
Fig. 4 shows to compress a reality of client according to distributed storage redundant data of the present invention
Apply the structural representation of example.
Fig. 5 show according to distributed storage redundant data of the present invention compress client another
The structural representation of embodiment.
Fig. 6 shows a reality according to distributed storage redundant data compression service device of the present invention
Apply the structural representation of example.
Fig. 7 show according to distributed storage redundant data compression service device of the present invention another
The structural representation of embodiment.
Specific embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, to the technical scheme in the embodiment of the present invention
It is clearly and completely described, it is clear that described embodiment is only a real part of the invention
Example is applied, rather than whole embodiments.Below to the description reality of at least one exemplary embodiment
It is merely illustrative on border, never as to the present invention and its application or any limitation for using.
Based on the embodiment in the present invention, those of ordinary skill in the art are not before creative work is made
The every other embodiment for being obtained is put, the scope of protection of the invention is belonged to.
The distributed storage redundant data pressure of one embodiment of the invention is described below with reference to Fig. 1
Compression method.
Fig. 1 is shown according to one of distributed storage redundant data compression method of the present invention implementation
The schematic flow sheet of example.As shown in figure 1, the method for the embodiment includes：
Step S102, client is packaged into data block by data are uploaded.
A kind of exemplary method, client is packaged into according to data packing strategy by data are uploaded
Data block.By taking the strategy based on volume as an example, client compares the volume and data for uploading data
The data block threshold volume X set in packing strategy；If the volume for uploading data is less than or equal to
X, then will upload data and be packaged into a data block, and packing data block product is equal to and uploads
Data volume；If the volume for uploading data is more than X, to uploading data in units of volume X
Split, the part after segmentation less than X is filled treatment, each several part after segmentation is uploaded
Data are packaged into a data block respectively.
For example, client preliminary setting data block threshold value volume is 100 megabits.Client ratio
The data block threshold volume set in volume and data the packing strategy for relatively uploading data：If uploading
The volume of data is less than or equal to 100 megabits, then will upload data and be packaged into a data block,
And packing data block product is equal to and uploads data volume；If the volume for uploading data is more than 100
Megabit, then split to uploading data in units of 100 megabits of volume, after segmentation not
The part of 100 megabits of foot is filled treatment, and each several part after segmentation uploads data and beats respectively
It is bundled into a data block.Additionally, in data packing strategy, being filled except splitting including data block
Outside mechanism, data block completeness check mechanism can also be included, to ensure to upload data block
Integrality.
Step S104, client generates the data block of data block according to data block ID generation strategies
ID。
A kind of exemplary method, client generates data block ID according to data block volume.Client
Compare the volume of data block and the data block threshold volume of setting in data block ID generation strategies in end
Y；If the volume of data block is less than Y, after the source filename, file according to where data block
Sew, at least one information generation data block ID in file size and filemodetime；If
The volume of data block is equal to or more than Y, then calculate generation using Message Digest 5 to data block
Data block ID.If additionally, data block volume is less than or equal to the source file where Y, and data block
Name, file suffixes, file size and filemodetime are all identical, then can also be according to matching somebody with somebody
Put and generation data block ID is calculated using Message Digest 5 to data block.
Typical data block ID generation strategies include the data block ID generation plans based on file characteristic
Slightly and based on Message Digest 5 data block ID generation strategies.
Data block ID generation strategies based on file characteristic calculate simple, the system resource of consuming
It is less, but there is the risk of file uniqueness identification mistake.For example, for small size data block,
Can be using " filename+file suffixes name+file size+modification time " as packing data
The data block ID of block.But, filename if multiple different pieces of information blocks, file suffixes name,
File size and modification time all same, then according to the data block ID based on file characteristic
The data block ID that generation strategy is generated is identical.
Data block ID generation strategies based on Message Digest 5 can avoid file uniqueness from knowing
Not wrong generation, but it calculates complicated, and the system resource of consuming is more, can generally be applicable
In large volume data block.It is common based on Message Digest 5 include CRC32, MD5 and
SHA algorithms.For example, by MD5 algorithms, different data blocks can generate unique number
According to block ID.
Distributed storage redundant data compression efficiency, can will be based on file in order to better improve
The data block ID generation strategies of feature and the data block ID generation strategies based on Message Digest 5
It is used in combination.For example, 100 megabits of client preliminary setting data block threshold value volume.Then,
Client compares the volume of data block and the data block threshold value of setting in data block ID generation strategies
Volume：If data block volume is less than 100 megabits, remove the process of generation data block ID from,
Directly using " filename+file suffixes name+file size+modification time " as packing data
The data block ID of block, and carry out data block upload；If the volume of data block is more than 100 megabits,
Generation data block ID then is calculated using MD5 algorithms to data block.
If additionally, multiple data block volumes are less than where 100 megabits, and multiple data blocks
Source filename, file suffixes, file size and filemodetime are all identical, then also to this
A little data blocks calculate generation data block ID using Message Digest 5.
Step S106, client sends to server, accordingly, server data block ID
End receives the data block ID that client sends.
Step S108, server judges whether the corresponding data blocks of data block ID have been present.
A kind of exemplary determination methods, server can be according to having uploaded data block list records
Information, whether the data block ID for having in list request to upload is judged, if it has, then explanation should
Data block is transmitted through on, and can perform step S110 carries out " precommpression ", if it did not,
It is transmitted through on then illustrating the data block also not, can perform step S112 carries out the upload of data block.
Step S110, if server judges data block ID, corresponding data block has been present,
Then server end by the already present feedback of the information of data block ID to client, under client starts
One transmission of data block；
Step S112, if server judges data block ID, corresponding data block does not exist,
Server end notifies the corresponding data block of client upload data block ID, and client is by the data
To server, server is to receive data block ID corresponding for the corresponding transmission of data blocks of block ID
After data block, data block ID recorded in having uploaded data block list.
Wherein, record has the data block ID for having uploaded in having uploaded data block list, can also wrap
Include the contents such as the storage location information that has uploaded data block.
The cooperation that the above method passes through client and server end, target data is before upload server
First judge whether existed in server end, if there is " precommpression " is then carried out, so as to realize
Compression to application layer redundant data, and reduce to communication network resource and server end resource
Occupancy, lifted distributed storage overall efficiency.
Fig. 2 shows another reality according to distributed storage redundant data compression method of the present invention
Apply the schematic flow sheet of example.As shown in Fig. 2 on the basis of embodiment illustrated in fig. 1, this reality
The method for applying example also includes：
Step S201, in server configuration data blocks packing strategy and data block ID generation strategies,
Client before being packed to data block, from server sync data pack strategy and data
Block ID generation strategies, data block packing sum is carried out by each client respectively to local data
According to the generation of block ID.
The parameter of influence data packing strategy can include data volume, data owner, data
File class.Based on synchronization parameter, it may be determined that the packing strategy of data block.Server can be with
According to safety, performance and other individual requirements, data block ID computational algorithms, mark are formulated
Its uniqueness.
Pack tactful by client and server sync data block, server can be processed to be had
The data block of unified specification, further increases the overall efficiency of distributed storage.Further, since
Data block ID is the unique mark for uploading data, and server is made according to data block ID to data block
Go out the judgement for whether needing " precommpression ".If the data block ID generations of server and client side
It is tactful different, then to be possible to two following situations occur：Different clients are according to different pieces of information
Block generation strategy generates identical block ID, the execution precommpression that server end can be wrong；It is different
Client generates different pieces of information block ID according to identical block, and the omission that server end can be wrong is pre-
Compression.Accordingly, as a preferred scheme, the data block ID of server and each client
Generation strategy is identical, it is ensured that different data blocks has its corresponding different pieces of information block ID.
Such that it is able to avoid the generation of said circumstances, precommpression is contributed to efficiently and accurately to carry out.
Fig. 3 shows another implementation according to distributed storage redundant data compression method of the present invention
The schematic flow sheet of example.As shown in figure 3, on the basis of embodiment illustrated in fig. 1, this implementation
The method of example also includes：Step S307, client will upload the file name of the affiliated file of data
Send to server, accordingly, server receives text belonging to the upload data block that client sends
The file name of part.Step S307 can simultaneously be performed with step S106.
After the data block that step S110 or S112 server receives client upload, hold
Row step S314, server sets up the connection of file name and its data block.
In this embodiment, server end safeguards the orderly connection of " filename-data block ID ",
One filename can correspond to one or more data block ID.If multiple data blocks,
Need sequentially to deposit.One data block ID can correspond to one or more filenames, when
When data block ID does not have corresponding filename, data block ID and data block can be deleted in itself
Remove.Based on filename, its corresponding data block is combined according to data block ID in order, then may be used
To obtain file in itself.
The distributed storage redundant data pressure of one embodiment of the invention is described below with reference to Fig. 4
The structure of contracting client.
Fig. 4 shows to compress an implementation of client according to distributed storage redundant data of the present invention
The structural representation of example.As shown in figure 4, the client 40 of the embodiment includes：
Data block packaged unit 402, data block is packaged into for will upload data；
Data block ID generation units 404, the data block ID for generating data block；
Data block ID transmitting elements 406, for data block ID to be sent to server, make clothes
Business device judges whether the corresponding data blocks of data block ID have been present；
Response receiving unit 408, for the corresponding numbers of data block ID that the reception server is returned
Whether there is response according to block；
Transmission of data blocks unit 410, for according to the response, if data block ID correspondences
Data block be present in server, then start the transmission of next data block；If the data
The corresponding data blocks of block ID are not present in server, then by the corresponding data blocks of data block ID
Transmit to server.
Wherein, data block packaged unit 402 is used to that number will to be uploaded according to data packing strategy configuration
According to data block is packaged into, for example, comparing the volume and setting in data packing strategy for uploading data
Data block threshold volume X；According to data pack strategy, if upload data volume be less than or
Equal to X, then will upload data and be packaged into a data block, and packing data block product is equal to
Upload data volume；If the volume for uploading data is more than X, to uploading in units of volume X
Data are split, and the part after segmentation less than X is filled treatment, each several part after segmentation
Upload data and be packaged into a data block respectively.
Wherein, data block ID generation units 404 are used to be configured according to data block ID generation strategies
The data block ID of data block is generated, in comparing the volume of data block and data block ID generation strategies
The data block threshold volume Y of setting；According to data block ID generation strategies, if the body of data block
Product is less than Y, then source filename according to where data block, file suffixes, file size and
At least one information generation data block ID in filemodetime, or data block use is disappeared
Breath digest algorithm calculates generation data block ID；If the volume of data block is equal to or more than Y, right
Data block calculates generation data block ID using Message Digest 5.
Wherein, data block ID generation units 404 are used for according to data block ID generation strategies,
Data block volume is less than or equal to source filename, file suffixes, the file where Y, and data block
In the case of size and filemodetime are the same from, then data block is calculated using eap-message digest
Method calculates generation data block ID, to check whether it has existed in server end.
The distributed storage redundant data of another embodiment of the present invention is described below with reference to Fig. 5
Compress the structure of client.
Fig. 5 shows to compress another reality of client according to distributed storage redundant data of the present invention
Apply the structural representation of example.As shown in figure 5, the client 50 of the embodiment also includes：
Including policy synchronization unit 501, for being packed strategy and data from server sync data
Block ID generation strategies.
Filename transmitting element 509, the file name for will upload the affiliated file of data sends
To server, the connection of file name and its data block is set up so as to server.
Preferably, data block ID transmitting elements 406 can be same with filename transmitting element 509
Step is performed.
The distributed storage redundant data pressure of one embodiment of the invention is described below with reference to Fig. 6
The structure of contracting server.
Fig. 6 is shown according to one of distributed storage redundant data compression service device of the present invention implementation
The structural representation of example.As shown in fig. 6, the server 60 of the embodiment includes：
Data block ID receiving units 602, the data block ID. for receiving client transmission
Data block judging unit 604, for whether judging the corresponding data blocks of data block ID
Exist.
Information feedback unit 606, if existed for the corresponding data blocks of data block ID,
Then information feedback unit by the already present feedback of the information of data block ID to client；If data
The corresponding data blocks of block ID do not exist, then information feedback unit notifies client upload data block
The corresponding data blocks of ID.
Data block reception unit 608, for receiving the corresponding data blocks of data block ID.
Data block ID recording units 610, for receiving the corresponding data blocks of data block ID
Afterwards, data block ID recorded in having uploaded data block list.
The distributed storage redundant data of another embodiment of the present invention is described below with reference to Fig. 7
The structure of compression service device.
Fig. 7 shows another reality according to distributed storage redundant data compression service device of the present invention
Apply the structural representation of example.As shown in fig. 7, the server 70 of the embodiment also includes：
Policy synchronization unit 701, for data packing strategy and data block ID generation strategies is same
Walk each client.
Filename receiving unit 709, for receiving text belonging to the upload data block that client sends
The file name of part, and set up the connection of file name and its data block.
Preferably, data block ID receiving units 602 can be same with filename receiving unit 709
Step is performed.
One of ordinary skill in the art will appreciate that realizing that all or part of step of above-described embodiment can
To be completed by hardware, it is also possible to instruct the hardware of correlation to complete by program, described journey
Sequence can be stored in a kind of computer-readable recording medium, and storage medium mentioned above can be
Read-only storage, disk or CD etc..
Presently preferred embodiments of the present invention is the foregoing is only, is not intended to limit the invention, it is all at this
Within the spirit and principle of invention, any modification, equivalent substitution and improvements made etc. all should be wrapped
It is contained within protection scope of the present invention.
Claims (19)
1. a kind of distributed storage redundant data compression method, including：
Client is packaged into data block by data are uploaded；
Client generates the data block ID of data block；
Client sends to server data block ID, server is judged data block ID pairs
Whether the data block answered has been present；
Whether the corresponding data blocks of data block ID that client the reception server is returned have deposited
In response；
According to the response, if the corresponding data blocks of data block ID are present in server,
Client starts the transmission of next data block；If the corresponding data blocks of data block ID are not
It is present in server, client is by the corresponding transmission of data blocks of data block ID to server.
2. method according to claim 1, it is characterised in that methods described also includes：
Client from server sync data packing strategy and data block ID generation strategies.
3. method according to claim 1, it is characterised in that methods described also includes：
The file name that client will also upload the affiliated file of data is sent to server, with housecoat
Business device sets up the connection of file name and its data block.
4. method according to claim 1 and 2, it is characterised in that wherein, client
Data block is packaged into by data are uploaded according to data packing strategy configuration, including：
Client compares the data block threshold value of the volume with setting in data packing strategy that upload data
Volume X；
Pack tactful according to data, if the volume for uploading data is less than or equal to X, will upload
Data are packaged into a data block, and packing data block product is equal to and uploads data volume；
If the volume for uploading data is more than X, divided uploading data in units of volume X
Cut, the part after segmentation less than X is filled treatment, each several part after segmentation uploads data point
A data block is not packaged into.
5. method according to claim 1 and 2, it is characterised in that wherein, client
According to the data block ID of data block ID generation strategies configuration generation data block, including：
Client compares the volume of data block and the data block of setting in data block ID generation strategies
Threshold volume Y；
According to data block ID generation strategies, if the volume of data block is less than Y, according to data block
In the source filename at place, file suffixes, file size and filemodetime at least one
Item information generation data block ID, or generation data are calculated using Message Digest 5 to data block
Block ID；
If the volume of data block is equal to or more than Y, Message Digest 5 meter is used to data block
Calculate generation data block ID.
6. method according to claim 5, it is characterised in that given birth to according to data block ID
Into strategy, if data block volume is less than or equal to source filename, the file where Y, and data block
Suffix, file size and filemodetime are all identical, then use eap-message digest to data block
Algorithm calculates generation data block ID, to check whether it has existed in server end.
7. a kind of distributed storage redundant data compression method, including：
The data block ID that received server-side client sends；
Server end judges whether the corresponding data blocks of data block ID have been present；
If the corresponding data blocks of data block ID have been present, server end is by data block ID
Already present feedback of the information is to client；If the corresponding data blocks of data block ID do not exist,
Then server end notifies the corresponding data blocks of client upload data block ID, and is receiving number
After according to the corresponding data blocks of block ID, data block ID recorded in having uploaded data block list.
8. method according to claim 7, it is characterised in that also include：
Server receives the file name of the affiliated file of upload data block that client sends, and builds
The connection of vertical file name and its data block.
9. method according to claim 7, it is characterised in that also include：
Data packing strategy and data block ID generation strategies are synchronized to each client by server.
10. it is a kind of for distributed storage redundant data compression client, including：
Data block packaged unit, data block is packaged into for will upload data；
Data block ID generation units, the data block ID for generating data block；
Data block ID transmitting elements, for data block ID to be sent to server, make server
Judge whether the corresponding data blocks of data block ID have been present；
Response receiving unit, for the corresponding data of data block ID that the reception server is returned
Whether block has had response；
Transmission of data blocks unit, for according to the response, if data block ID is corresponding
Data block is present in server, then start the transmission of next data block；If the data block
The corresponding data blocks of ID are not present in server, then pass the corresponding data blocks of data block ID
Transport to server.
11. clients according to claim 10, it is characterised in that the client is also
Including policy synchronization unit, for being given birth to from server sync data packing strategy and data block ID
Into strategy.
12. clients according to claim 10, it is characterised in that the client is also
Including：Filename transmitting element, the file name for the affiliated file of data will to be uploaded send to
Server, the connection of file name and its data block is set up so as to server.
13. client according to claim 10 or 11, it is characterised in that the number
It is used to be packaged into data block by data are uploaded according to data packing strategy configuration according to block packaged unit,
Compare the data block threshold volume X of the volume with setting in data packing strategy that upload data；Root
Pack tactful according to data, if the volume for uploading data is less than or equal to X, data will be uploaded and beaten
A data block is bundled into, and packing data block product is equal to and uploads data volume；If uploading number
According to volume be more than X, then in units of volume X to upload data split, after segmentation not
The part of sufficient X is filled treatment, and each several part after segmentation uploads data and is packaged into one respectively
Data block.
14. client according to claim 10 or 11, it is characterised in that the number
It is used for the data according to data block ID generation strategies configuration generation data block according to block ID generation units
Block ID, compares the volume of data block and the data block threshold value of setting in data block ID generation strategies
Volume Y；According to data block ID generation strategies, if the volume of data block is less than Y, according to number
According in the source filename where block, file suffixes, file size and filemodetime extremely
One item missing information generates data block ID, or calculates generation using Message Digest 5 to data block
Data block ID；If the volume of data block is equal to or more than Y, eap-message digest is used to data block
Algorithm calculates generation data block ID.
15. clients according to claim 14, it is characterised in that the data block
ID generation units are less than or equal to Y according to data block ID generation strategies in data block volume, and
Source filename, file suffixes, file size and filemodetime all phases where data block
In the case of, then generation data block ID is calculated using Message Digest 5 to data block, so as to
Check whether it has existed in server end.
A kind of 16. servers for the compression of distributed storage redundant data, including：
Data block ID receiving units, the data block ID for receiving client transmission；
Data block judging unit, for whether to judge the corresponding data blocks of data block ID
In the presence of；
Information feedback unit, if existed for the corresponding data blocks of data block ID,
Information feedback unit is by the already present feedback of the information of data block ID to client；If data block
The corresponding data blocks of ID do not exist, then information feedback unit notifies client upload data block ID
Corresponding data block；
Data block reception unit, for receiving the corresponding data blocks of data block ID；
Data block ID recording units, for after the corresponding data blocks of data block ID are received,
Data block ID recorded in having uploaded data block list.
17. servers according to claim 16, it is characterised in that also include：
Filename receiving unit, for receiving the affiliated file of upload data block that client sends
File name, and set up the connection of file name and its data block.
18. servers according to claim 16, it is characterised in that also include：
Policy synchronization unit, for by data packing strategy and data block ID generation strategy synchronizations
To each client.
19. a kind of distributed storage redundant data compressibilities, it is characterised in that including right
It is required that the clothes described in client and claim any one of 16-18 described in any one of 10-15
Business device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510777060.9A CN106713385A (en) | 2015-11-13 | 2015-11-13 | Distributed storage redundant data compression method and system, client and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510777060.9A CN106713385A (en) | 2015-11-13 | 2015-11-13 | Distributed storage redundant data compression method and system, client and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106713385A true CN106713385A (en) | 2017-05-24 |
Family
ID=58931781
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510777060.9A Pending CN106713385A (en) | 2015-11-13 | 2015-11-13 | Distributed storage redundant data compression method and system, client and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106713385A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113780450A (en) * | 2021-09-16 | 2021-12-10 | 郑州云智信安安全技术有限公司 | Distributed storage method and system based on self-coding neural network |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101699822A (en) * | 2009-08-06 | 2010-04-28 | 腾讯科技（深圳）有限公司 | File uploading method and device, and mass storage system |
-
2015
- 2015-11-13 CN CN201510777060.9A patent/CN106713385A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101699822A (en) * | 2009-08-06 | 2010-04-28 | 腾讯科技（深圳）有限公司 | File uploading method and device, and mass storage system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113780450A (en) * | 2021-09-16 | 2021-12-10 | 郑州云智信安安全技术有限公司 | Distributed storage method and system based on self-coding neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107193490B (en) | Distributed data storage system and method based on block chain | |
| CN107819828B (en) | Data transmission method and device, computer equipment and storage medium | |
| TWI528191B (en) | File Handling Method Based on Cloud Storage, System and Server Cluster System | |
| US8838968B2 (en) | System and method for virtual machine data protection in a public cloud | |
| CN107846282A (en) | A kind of electronic data distribution keeping method and system based on block chain technology | |
| CN109309730B (en) | Credible file transmission method and system | |
| CN101009516A (en) | A method and system for data synchronization | |
| CN106156359A (en) | A kind of data synchronization updating method under cloud computing platform | |
| US9560010B1 (en) | Network file transfer | |
| CN105025053A (en) | Distributed file upload method based on cloud storage technology and system | |
| CN103581216B (en) | The method, apparatus and system of fragment data storage | |
| CN102761596A (en) | Method for uploading a file in an on-line storage system and corresponding on-line storage system | |
| CN102255974A (en) | Cloud storage method for cloud computing server | |
| CN102571788B (en) | Sample collection method and system | |
| CN104202385A (en) | Data backup and updating method of distributed file system | |
| CN103780615A (en) | Sharing method of client conversation data among multiple servers | |
| CN109067733A (en) | The method and apparatus for sending data, and the method and apparatus for receiving data | |
| CN101771548A (en) | File synchronizing method and system | |
| CN106713385A (en) | Distributed storage redundant data compression method and system, client and server | |
| CN113010474B (en) | File management method, instant messaging method and storage server | |
| CN106033448A (en) | Method and device for compressing Html page | |
| CN111695885A (en) | Digital certificate block chain compression method based on transaction input reduction | |
| CN111736949B (en) | Kubernetes container cloud platform application importing method and system | |
| US20120303588A1 (en) | Data de-duplication processing method for point-to-point transmission and system thereof | |
| CN105389384A (en) | Medical private data exchange file generation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170524 |