CN109361772A - File uploading method and device in distributed object storage system - Google Patents
File uploading method and device in distributed object storage system Download PDFInfo
- Publication number
- CN109361772A CN109361772A CN201811527148.5A CN201811527148A CN109361772A CN 109361772 A CN109361772 A CN 109361772A CN 201811527148 A CN201811527148 A CN 201811527148A CN 109361772 A CN109361772 A CN 109361772A
- Authority
- CN
- China
- Prior art keywords
- big file
- uploaded
- value
- file
- fragment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 239000012634 fragment Substances 0.000 claims abstract description 134
- 238000004590 computer program Methods 0.000 claims description 10
- 239000004744 fabric Substances 0.000 claims 1
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000012546 transfer Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013467 fragmentation Methods 0.000 description 2
- 238000006062 fragmentation reaction Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/32—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials
- H04L9/3236—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions
- H04L9/3239—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a file uploading method in a distributed object storage system, which comprises the following steps: when a large file uploading instruction is received, determining the name of a large file to be uploaded; calculating the MD5 value of each fragment of the large file to be uploaded; acquiring the MD5 value of each fragment stored in the fragment head object corresponding to the uploaded large file with the same name according to the name; respectively judging whether the MD5 value of the fragment of the large file to be uploaded is equal to the MD5 value of the corresponding fragment in the same-name large file; if not, uploading the fragments corresponding to the unequal MD5 values in the large file to be uploaded; when the method is used for uploading the large file, only different contents in the same-name large file existing in the system are uploaded, so that the efficiency of uploading the large file in the storage system is improved; the invention also discloses a file uploading device in the distributed object storage system, the distributed object storage system and a computer readable storage medium, which have the beneficial effects.
Description
Technical field
The present invention relates to distributed objects technical field of memory, in particular to file in a kind of distributed objects storage system
Method for uploading, device, distributed objects storage system and computer readable storage medium.
Background technique
Currently, more and more business scenarios are needed using distributed objects storage system, wherein distributed objects storage
System refers to the distributed storage towards unstructured data object.With the explosive increase of internet data, people couple
The capacity requirement of storage system is increasing, and the number of objects of single container storage is also more and more.Therefore it is directed to big file
Upload be the frequent operation to be performed of user.Wherein, big file uploading detailed process is, when user uploads big file, meeting
Big file is divided into multi-disc to upload, under default situations, every is 15M, that is to say, that the file more than 15M uses default
Fragment upload mode on transmitting file, be more than at this time 15M file just referred to as big file.Wherein, fragment uploads client i.e.
If the big file to be uploaded is held to be divided into stem portion according to fragment size set by user, all fragments are then uploaded respectively and are deposited
Storage.And in face of the big file of tens G easily, how to promote efficiency when uploading big file in storage system just seems especially heavy
It wants.
Summary of the invention
The object of the present invention is to provide file uploading method, device, distribution in a kind of distributed objects storage system are right
As storage system and computer readable storage medium, it can be avoided identical content and repeat to upload, and then promoted in storage system
Pass efficiency when big file.
In order to solve the above technical problems, the present invention provides file uploading method in a kind of distributed objects storage system, packet
It includes:
When receiving big file uploading instruction, the title of big file to be uploaded is determined；
Calculate the MD5 value of each fragment of the big file to be uploaded；
The MD5 of each fragment stored in the corresponding slice header object of big file of the same name uploaded according to the name acquiring
Value；
The MD5 value of the fragment of the big file to be uploaded and corresponding fragment in the big file of the same name are judged respectively
Whether MD5 value is equal；
If unequal, the corresponding fragment of unequal MD5 value in the big file to be uploaded is uploaded.
Optionally, before the MD5 value of each fragment for calculating the big file to be uploaded, further includes:
Calculate the whole MD5 value of the big file to be uploaded；
Obtain the whole MD5 value stored in the corresponding whole head object of the big file of the same name；
Judge whether whole MD5 value and the whole MD5 value of the big file of the same name of the big file to be uploaded are equal；
If equal, the big file to be uploaded is not uploaded；
If unequal, the step of executing the MD5 value of each fragment for calculating the big file to be uploaded.
Optionally, before the whole MD5 value for calculating the big file to be uploaded, further includes:
Judge in distributed objects storage system with the presence or absence of the big file of the same name；
If it exists, then the step of executing the whole MD5 value for calculating the big file to be uploaded.
Optionally, when the big file of the same name is not present in the distributed objects storage system, comprising:
Upload whole fragments of the big file to be uploaded.
The present invention also provides files in a kind of distributed objects storage system to upload device, comprising:
Module is obtained, for determining the title of big file to be uploaded when receiving big file uploading instruction；
First computing module, the MD5 value of each fragment for calculating the big file to be uploaded；
Fragment MD5 value obtains module, the corresponding slice header of big file of the same name for having been uploaded according to the name acquiring
The MD5 value of each fragment stored in object；
First judgment module, for judge respectively the big file to be uploaded fragment MD5 value and the big text of the same name
Whether the MD5 value of corresponding fragment is equal in part；
First execution module, if the MD5 value for corresponding to fragment in the big file to be uploaded and the big file of the same name
It is unequal, then upload the corresponding fragment of unequal MD5 value in the big file to be uploaded.
Optionally, the file uploads device further include:
Second computing module, for calculating the whole MD5 value of the big file to be uploaded；
Whole MD5 value obtains module, for obtaining the entirety stored in the corresponding whole head object of the big file of the same name
MD5 value；
Second judgment module, for judging the whole of the whole MD5 value of the big file to be uploaded and the big file of the same name
Whether body MD5 value is equal；
Second execution module, if equal with the whole MD5 value of the big file of the same name for the big file to be uploaded,
The big file to be uploaded is not uploaded；If the whole MD5 value of the big file to be uploaded and the big file of the same name is unequal,
Trigger first computing module.
Optionally, the file uploads device further include:
Third judgment module, for judging in distributed objects storage system with the presence or absence of the big file of the same name；
Third execution module, if for, there are the big file of the same name, being triggered in the distributed objects storage system
Second computing module.
Optionally, if the third execution module is also used in the distributed objects storage system, there is no described of the same name
Big file then uploads whole fragments of the big file to be uploaded.
The present invention also provides a kind of distributed objects storage systems, comprising:
Memory, for storing computer program；
Processor realizes file in distributed objects storage system described above when for executing the computer program
The step of transmission method.
The present invention also provides a kind of computer readable storage medium, calculating is stored on the computer readable storage medium
Machine program, the computer program realize file upload side in distributed objects storage system described above when being executed by processor
The step of method.
File uploading method in distributed objects storage system provided by the present invention, comprising: when receiving on big file
When teletype command, the title of big file to be uploaded is determined；Calculate the MD5 value of each fragment of the big file to be uploaded；According to described
The MD5 value of each fragment stored in the corresponding slice header object of big file of the same name that name acquiring has uploaded；Respectively described in judgement
Whether the MD5 value of the fragment of big file to be uploaded and the MD5 value of corresponding fragment in the big file of the same name are equal；If not phase
Deng then uploading the corresponding fragment of unequal MD5 value in the big file to be uploaded.
As it can be seen that this method is when uploading big file, by comparing big each fragment MD5 value of file to be uploaded and big file of the same name
Each fragment MD5 value uploads fragment corresponding with MD5 value unequal in big file of the same name in big file to be uploaded, i.e., only uploads
The content different from big file of the same name existing in system, content identical for the two do not need to upload again, Jin Erti
Rise efficiency when uploading big file in storage system；The present invention also provides files in a kind of distributed objects storage system to upload
Device, distributed objects storage system and computer readable storage medium have above-mentioned beneficial effect, and details are not described herein.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of invention for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is the flow chart of file uploading method in distributed objects storage system provided by the embodiment of the present invention；
Fig. 2 is the structural block diagram that file uploads device in distributed objects storage system provided by the embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.
Currently, distributed objects storage system (can be referred to as object storage system) is directed to the upload of big file, execute
Different process is uploaded with plain objects.Entire big file uploading execution process is divided into three phases by it: initialization multi-disc
It uploads, uploads fragment, completes multi-disc upload.Final implementing result is that big file is stored in object storage in the form of fragment
In system.The critically important head object of two classes can be generated in upload procedure, the entire big head object of file and the head of each fragment
Object, respectively to identify the big file and each fragment.Wherein for big file uploading of the same name, big file generated
Head object naming be identical, and in the name of each fragment object due to containing unique identification upload_id and it is different.But
All contain the filename of big file in all object names.Wherein, upload_id: the unique identification of big file uploading is used for
Breakpoint transmission, list fragment, destroy uploaded fragment etc..The different of big file of the same name upload the upload_id difference generated.
In existing distributed objects storage system, in order to support that multi version function, system are to allow to upload text of the same name
Part.The big file that repetition for the upload of big file of the same name, or referred to as big file uploads and modification content is few
It uploads, the implementation procedure of current distributed objects storage system is as follows: executing process same as common big file uploading, i.e.,
After big file fragmentation, all fragments are uploaded to storage system and are saved.Before it was mentioned that the invention that on big file of the same name
Pass, the head object naming of big file generated be it is identical, due to containing unique identification in the name of each fragment object
Upload_id and it is different.Therefore before the head object of newly-generated big file can cover, and for all fragments before,
It then transfers to the refuse disposal system of system to handle, at the appointed time all deletes it.
The big file uploading efficiency of the above process is very poor.The repetition upload of big file and content are changed few
Big file uploading, in fact completely need not by its all upload.For this purpose, the present embodiment provides a kind of promotion big file uploading effects
The method of rate can reduce the computing cost of distributed objects storage system, alleviate I/O pressure, accelerate the upload speed of big file
Degree improves user experience.Specifically referring to FIG. 1, Fig. 1 is distributed objects storage system Chinese provided by the embodiment of the present invention
The flow chart of part method for uploading；This method may include:
S110, when receive big file uploading instruction when, determine the title of big file to be uploaded.
In the present embodiment when receiving big file uploading instruction, it is first determined the title of big file to be uploaded.Due to this
Application scenarios are after having had existed big file of the same name in systems in embodiment, need to upload of the same name to be uploaded big therewith
The scene of file.Therefore big file uploading instruction is being got, it is necessary first to determine the title of big file to be uploaded, and then can be with
According to the title it is determining it is of the same name with big file to be uploaded on the big file of the same name that is transmitted through.Pass through in subsequent the present embodiment
Big file of the same name refers to the big file that has uploaded of the same name with big file to be uploaded.Therefore, have to improve in the present embodiment
The upper transfer efficiency of the file to be uploaded of big file of the same name, can also first determine whether big file to be uploaded whether have uploaded it is same
The big file of name.If it does not exist, then it is directly uploaded in the way of in the related technology, i.e., directly uploads the whole of big file to be uploaded
Fragment.If it exists, then the step in the application is continued to execute.And then transfer efficiency can be improved, avoid idle work.I.e. preferably
, judge in distributed objects storage system with the presence or absence of big file of the same name.Judgment step setting is got in step S110
After the title of big file to be uploaded, and then big file of the same name can be determined whether there is according to the title.Certainly, in order to mention
The upper transfer efficiency of high file to be uploaded, avoids idle work.The judgment step is got executing the step S110 in the present embodiment
After the title of big file to be uploaded, it is immediately performed.
It should be noted that the present embodiment is not defined the specific size of big file.Such as big file can be
Refer to the file greater than 15M of current default.In other words due to current default in the case of, the size of each fragment is 15M, therefore
File more than 15M will default uploaded using fragment by the way of on transmitting file, be more than at this time 15M file be just referred to as it is literary greatly
Part.So it can be appreciated that be more than default fragment size file be exactly big file, that is, need fragment upload file
It is exactly big file.It is understood that since the present embodiment is not defined the specific size of big file, that is, this reality
Example is applied not to be defined the size of fragment.It can be determined according to the actual situation.
S120, calculate big file to be uploaded each fragment MD5 value.
Wherein, MD5 is a kind of Cryptographic Hash Function being widely used, and can produce out one 128 (16 byte)
Hashed value, for ensuring that information transmission is complete consistent.I.e. by compare MD5 value can determine two fragments data whether one
It causes, and then can be according to MD5 comparison result, it is determined whether need to upload corresponding data.
Determining that file to be uploaded there are when big file of the same name, calculates each fragment of big file to be uploaded in the present embodiment
MD5 value does not limit the mode for calculating the MD5 value of each fragment of big file to be uploaded in the present embodiment, for example, if to be uploaded big
The MD5 value of current slice content has been computed and housed in file in slice header object, then step S120 can directly to
Upload the MD5 value for obtaining each fragment in big file in slice header object.If not deposited in slice header object in big file to be uploaded
The MD5 value of current slice content is put, then step S120 needs to calculate the MD5 value of each fragment of big file to be uploaded, at this time preferably
, for the ease of it is subsequent have the to be uploaded big file of the same name with the big file to be uploaded again when, big file to be uploaded at this time is just
Become the big file of the same name uploaded, for more quick execution step S130, each fragment that can will be calculated at this time
MD5 value be stored directly in slice header object.
Each fragment stored in S130, the corresponding slice header object of big file of the same name uploaded according to name acquiring
MD5 value.
Each fragment that can be directly stored from the corresponding slice header object of big file of the same name uploaded in the present embodiment
MD5 value.
The sequence of step S120 and step S130 are not limited in the present embodiment.That is, first carrying out step S130 again
Executing step S120 can also.
S140, judge respectively big file to be uploaded fragment MD5 value and the MD5 value of corresponding fragment in big file of the same name
It is whether equal.
The purpose of this step is to determine between big file to be uploaded and big file of the same name equal with the presence or absence of MD5 value
Fragment.That is, the fragment in the corresponding big file to be uploaded of equal MD5 value does not need to upload, remaining fragment is only uploaded,
It can thus be uploaded to avoid the repetition of identical data, reduce the quantity for the fragment that need to be uploaded, also just improved to be uploaded big
The upper transfer efficiency of file can reduce the computing cost of distributed objects storage system, alleviate I/O pressure, accelerate big file
Uploading speed improves user experience.It since big file is when being divided into fragment, is carried out according to certain rule, therefore two
The identical big file of content after dividing fragment, corresponding fragment sequence be it is identical, improve judgement effect in the present embodiment
Rate, reduction judge number, and can be compared according to fragment corresponding relationship in the present embodiment (can simply be interpreted as, slice header
Fragment object arranges in sequence in object, and same location can will be located in big file to be uploaded and big file of the same name
Two fragment objects regard corresponding fragment object as.Such as the 3rd fragment of big file to be uploaded and big file of the same name
3rd fragment is corresponding fragment), that is, be respectively compared corresponding two fragments MD5 value it is whether equal, if equal, no
The fragment is uploaded, and judges whether the MD5 value of next group of corresponding two fragment is equal, circulation is executed until whole correspondences point
(successively compare, it can be understood as single thread processing) until piece is completeer.It is of course also possible to be performed simultaneously with multiple threads
Comparison procedure, i.e., by improving relative efficiency parallel.The MD5 value of the fragment of big file to be uploaded is judged in the present embodiment respectively
It is whether equal with the MD5 value of corresponding fragment in big file of the same name.
If S150, unequal, the corresponding fragment of unequal MD5 value in big file to be uploaded is uploaded.
If S160, equal, the corresponding fragment of MD5 value equal in big file to be uploaded is not uploaded.
Further, it is identical with big file of the same name already present in system that there is also big files to be uploaded in the present embodiment
The case where, at this time in order to more quickly determine the operation for not needing to upload, can only in big file more to be uploaded and system
Existing big file entirety MD5 of the same name, so as to avoid executing the operation for the MD5 for being respectively compared each fragment.It is i.e. excellent
Choosing, can also include: before calculating the MD5 value of each fragment of big file to be uploaded
Calculate the whole MD5 value of big file to be uploaded；
Obtain the whole MD5 value stored in the corresponding whole head object of big file of the same name；
Whether the whole MD5 value for judging big file to be uploaded and the whole MD5 value of big file of the same name are equal；
If equal, big file to be uploaded is not uploaded；
If unequal, the step of executing the MD5 value for calculating each fragment of big file to be uploaded, that is, execute step
S120。
The mode for calculating entirety MD5 value is not limited in the present embodiment, it can be with reference to the specific side for calculating fragment MD5 value
Formula.It can be in the whole MD5 value and the corresponding MD5 value of big file of the same name for needing to obtain big file to be uploaded in the present embodiment
When, acquisition can be calculated separately；Can certainly add interrelated logic in the code of distributed objects storage system service end,
It for calculating big file entirety MD5 value, and deposits in whole head object, for being obtained when subsequent compare.
At this point, the step of judging in distributed objects storage system with the presence or absence of big file of the same name can be to be uploaded in calculating
It is executed before the whole MD5 value of big file.Namely judge in distributed objects storage system with the presence or absence of big file of the same name；If
There are big file of the same name, then the step of executing the whole MD5 value for calculating big file to be uploaded.
Illustrate the above process below by specific example:
Step 1: after client initiates big file uploading request (namely receiving big file uploading instruction), sentencing first
Whether there is the object of big file of the same name to execute big file uploading, i.e., directly if not having in disconnected object storage system corresponding container
Upload whole fragments of big file to be uploaded；Otherwise, the whole MD5 value of local big file (big file i.e. to be uploaded) is calculated；
Step 2: obtaining the whole MD5 value stored in the whole head object of server-side big file of the same name, and big literary with local
The whole MD5 value of part is compared, if the two MD5 value is identical, is illustrated the identical no change of the two content, is skipped upload；Otherwise,
The MD5 value of each fragment of local big file is calculated before uploading fragment；
Step 3: obtain the MD5 value of each fragment stored in the slice header object of server-side big file of the same name, and with local
The fragment MD5 value of calculating is compared, if the two MD5 value is identical, is illustrated the identical no change of current slice content, is then skipped and work as
The upload of preceding fragment；Otherwise, current slice is uploaded；
Step 4: circulation executes step 3, completes until all fragments upload, terminates this big file uploading task.
The whole MD5 value of big file of the same name in storage system is obtained, the MD5 value come out with local computing is compared,
If the two is identical, it was demonstrated that the file of existing identical content in storage system, it may not be necessary to upload again, significant increase user's body
It tests；If the two is not identical, the MD5 value that big file of the same name in storage system correspond to fragment is obtained, with the fragment of local computing out
The MD5 value of content is compared, and if they are the same, then skips the upload of current slice, only uploads the fragment content that content changes, greatly
Ground improves the uploading speed of big file, improves transfer efficiency.
Based on the above-mentioned technical proposal, file uploading method in distributed objects storage system provided in an embodiment of the present invention,
The existing upload for big file of the same name of distributed objects storage system, using method identical with common big file uploading,
It namely will all upload after big file fragmentation, then transfer to all fragments of the big file of the same name uploaded before at rubbish
Reason cleaned system falls.The repetition of such processing method big file of the same name identical for content uploads and content changes
For the upload of less big file of the same name, both occupy bandwidth, committed memory resource, and increase I/O, at the same to user with
Carry out very poor experience.For this scene, the present embodiment proposes in a kind of distributed objects storage system and promotes big file uploading
The method of efficiency increases the comparison mechanism of big file entirety MD5 value and corresponding fragment MD5 value, reaches duplicate contents and do not upload
Purpose, reduce EMS memory occupation, avoid uploading duplicate content, effectively promote upper transfer efficiency, improve user experience.
Device is uploaded to file in distributed objects storage system provided in an embodiment of the present invention below, distributed objects are deposited
Storage system and computer readable storage medium are introduced, and file uploads dress in distributed objects storage system described below
It sets, file in distributed objects storage system and computer readable storage medium and above-described distributed objects storage system
Method for uploading can correspond to each other reference.
Referring to FIG. 2, Fig. 2 is that file uploads device in distributed objects storage system provided by the embodiment of the present invention
Structural block diagram；The apparatus may include:
Module 100 is obtained, for determining the title of big file to be uploaded when receiving big file uploading instruction；
First computing module 200, the MD5 value of each fragment for calculating big file to be uploaded；
Fragment MD5 value obtains module 300, the corresponding slice header pair of big file of the same name for having been uploaded according to name acquiring
As the MD5 value of each fragment of middle storage；
First judgment module 400, for judge respectively big file to be uploaded fragment MD5 value with it is right in big file of the same name
Whether the MD5 value for the fragment answered is equal；
First execution module 500, if MD5 value for corresponding to fragment in big file to be uploaded and big file of the same name not phase
Deng then uploading the corresponding fragment of unequal MD5 value in big file to be uploaded；If big file to be uploaded with it is right in big file of the same name
It answers the MD5 value of fragment equal, does not then upload the corresponding fragment of MD5 value equal in big file to be uploaded.
Based on the above embodiment, which can also include:
Second computing module, for calculating the whole MD5 value of big file to be uploaded；
Whole MD5 value obtains module, for obtaining the whole MD5 stored in the corresponding whole head object of big file of the same name
Value；
Second judgment module, for judging that whole MD5 value and the whole MD5 value of big file of the same name of big file to be uploaded be
It is no equal；
Second execution module, if equal with the whole MD5 value of big file of the same name for big file to be uploaded, do not upload to
Upload big file；If the whole MD5 value of big file to be uploaded and big file of the same name is unequal, the first computing module is triggered.
Based on the above embodiment, which can also include:
Third judgment module, for judging in distributed objects storage system with the presence or absence of big file of the same name；
Third execution module, if for, there are big file of the same name, triggering second to calculate in distributed objects storage system
Module.
Based on the above embodiment, if third execution module is also used in distributed objects storage system, there is no big texts of the same name
Part then uploads whole fragments of big file to be uploaded.
It should be noted that being based on above-mentioned any embodiment, described device can be to be realized based on programmable logic device
, programmable logic device includes FPGA, CPLD, single-chip microcontroller etc..The programmable logic device can be deposits in distributed objects
In storage system.
The embodiment of the present invention also provides a kind of distributed objects storage system, comprising: memory, for storing computer journey
Sequence；Processor realizes the Chinese of distributed objects storage system described in above-mentioned any embodiment when for executing computer program
The step of part method for uploading.If processor when executing computer program for realizing when receiving big file uploading instruction, really
The title of fixed big file to be uploaded；Calculate the MD5 value of each fragment of the big file to be uploaded；According to the name acquiring on
The MD5 value of each fragment stored in the corresponding slice header object of big file of the same name passed；The big file to be uploaded is judged respectively
Fragment MD5 value and the MD5 value of corresponding fragment in the big file of the same name it is whether equal；If unequal, described in upload
The corresponding fragment of unequal MD5 value in big file to be uploaded.
The present embodiment is not defined the scale of distributed objects storage system.
The present invention also provides a kind of computer readable storage medium, computer journey is stored on computer readable storage medium
Sequence is realized in distributed objects storage system described in above-mentioned any embodiment when computer program is executed by processor on file
The step of transmission method.It is realized when being executed by processor such as computer program when receiving big file uploading instruction, is determined to upper
Pass the title of big file；Calculate the MD5 value of each fragment of the big file to be uploaded；It has been uploaded according to the name acquiring same
The MD5 value of each fragment stored in the corresponding slice header object of the big file of name；The fragment of the big file to be uploaded is judged respectively
MD5 value and the MD5 value of corresponding fragment in the big file of the same name it is whether equal；If unequal, upload described to be uploaded
The corresponding fragment of unequal MD5 value in big file.
The computer readable storage medium may include: USB flash disk, mobile hard disk, read-only memory (Read-Only
Memory, ROM), random access memory (Random Access Memory, RAM), magnetic or disk etc. is various to deposit
Store up the medium of program code.
Each embodiment is described in a progressive manner in specification, the highlights of each of the examples are with other realities
The difference of example is applied, the same or similar parts in each embodiment may refer to each other.For device disclosed in embodiment
Speech, since it is corresponded to the methods disclosed in the examples, so being described relatively simple, related place is referring to method part illustration
?.
Professional further appreciates that, unit described in conjunction with the examples disclosed in the embodiments of the present disclosure
And algorithm steps, can be realized with electronic hardware, computer software, or a combination of the two, in order to clearly demonstrate hardware and
The interchangeability of software generally describes each exemplary composition and step according to function in the above description.These
Function is implemented in hardware or software actually, the specific application and design constraint depending on technical solution.Profession
Technical staff can use different methods to achieve the described function each specific application, but this realization is not answered
Think beyond the scope of this invention.
The step of method described in conjunction with the examples disclosed in this document or algorithm, can directly be held with hardware, processor
The combination of capable software module or the two is implemented.Software module can be placed in random access memory (RAM), memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
Above to file uploading method, device, distribution in a kind of distributed objects storage system provided by the present invention
Object storage system and computer readable storage medium are described in detail.Specific case used herein is to of the invention
Principle and embodiment is expounded, method and its core of the invention that the above embodiments are only used to help understand
Thought.It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, also
Can be with several improvements and modifications are made to the present invention, these improvement and modification also fall into the protection scope of the claims in the present invention
It is interior.
Claims (10)
1. file uploading method in a kind of distributed objects storage system characterized by comprising
When receiving big file uploading instruction, the title of big file to be uploaded is determined；
Calculate the MD5 value of each fragment of the big file to be uploaded；
The MD5 value of each fragment stored in the corresponding slice header object of big file of the same name uploaded according to the name acquiring；
The MD5 value of the fragment of the big file to be uploaded and the MD5 value of corresponding fragment in the big file of the same name are judged respectively
It is whether equal；
If unequal, the corresponding fragment of unequal MD5 value in the big file to be uploaded is uploaded.
2. file uploading method in distributed objects storage system according to claim 1, which is characterized in that the calculating
Before the MD5 value of each fragment of the big file to be uploaded, further includes:
Calculate the whole MD5 value of the big file to be uploaded；
Obtain the whole MD5 value stored in the corresponding whole head object of the big file of the same name；
Judge whether whole MD5 value and the whole MD5 value of the big file of the same name of the big file to be uploaded are equal；
If equal, the big file to be uploaded is not uploaded；
If unequal, the step of executing the MD5 value of each fragment for calculating the big file to be uploaded.
3. file uploading method in distributed objects storage system according to claim 2, which is characterized in that the calculating
Before the whole MD5 value of the big file to be uploaded, further includes:
Judge in distributed objects storage system with the presence or absence of the big file of the same name；
If it exists, then the step of executing the whole MD5 value for calculating the big file to be uploaded.
4. file uploading method in distributed objects storage system according to claim 2, which is characterized in that when described point
When the big file of the same name being not present in cloth object storage system, comprising:
Upload whole fragments of the big file to be uploaded.
5. file uploads device in a kind of distributed objects storage system characterized by comprising
Module is obtained, for determining the title of big file to be uploaded when receiving big file uploading instruction；
First computing module, the MD5 value of each fragment for calculating the big file to be uploaded；
Fragment MD5 value obtains module, the corresponding slice header object of big file of the same name for having been uploaded according to the name acquiring
The MD5 value of each fragment of middle storage；
First judgment module, for judge respectively the big file to be uploaded fragment MD5 value and the big file of the same name in
Whether the MD5 value of corresponding fragment is equal；
First execution module, if MD5 value for corresponding to fragment in the big file to be uploaded and the big file of the same name not phase
Deng then uploading the corresponding fragment of unequal MD5 value in the big file to be uploaded.
6. file uploads device in distributed objects storage system according to claim 5, which is characterized in that further include:
Second computing module, for calculating the whole MD5 value of the big file to be uploaded；
Whole MD5 value obtains module, for obtaining the whole MD5 stored in the corresponding whole head object of the big file of the same name
Value；
Second judgment module, for judging the whole MD5 value of the big file to be uploaded and the whole MD5 of the big file of the same name
Whether equal it is worth；
Second execution module, if equal with the whole MD5 value of the big file of the same name for the big file to be uploaded, not on
Pass the big file to be uploaded；If the whole MD5 value of the big file to be uploaded and the big file of the same name is unequal, trigger
First computing module.
7. file uploads device in distributed objects storage system according to claim 6, which is characterized in that further include:
Third judgment module, for judging in distributed objects storage system with the presence or absence of the big file of the same name；
Third execution module, if in the distributed objects storage system there are the big file of the same name, described in triggering
Second computing module.
8. file uploads device in distributed objects storage system according to claim 7, which is characterized in that the third
If execution module is also used in the distributed objects storage system, there is no the big files of the same name, upload described to be uploaded
Whole fragments of big file.
9. a kind of distributed objects storage system characterized by comprising
Memory, for storing computer program；
Processor realizes that distributed objects store as described in any one of Claims 1-4 when for executing the computer program
In system the step of file uploading method.
10. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium
Program realizes the distributed objects storage system as described in any one of Claims 1-4 when the computer program is executed by processor
In system the step of file uploading method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811527148.5A CN109361772A (en) | 2018-12-13 | 2018-12-13 | File uploading method and device in distributed object storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811527148.5A CN109361772A (en) | 2018-12-13 | 2018-12-13 | File uploading method and device in distributed object storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109361772A true CN109361772A (en) | 2019-02-19 |
Family
ID=65329598
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811527148.5A Pending CN109361772A (en) | 2018-12-13 | 2018-12-13 | File uploading method and device in distributed object storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109361772A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110650207A (en) * | 2019-09-29 | 2020-01-03 | 中电福富信息科技有限公司 | Method for uploading break point of super-large video file |
| CN111209254A (en) * | 2020-01-10 | 2020-05-29 | Oppo（重庆）智能科技有限公司 | File fingerprint acquisition method and device, electronic equipment and storage medium |
| CN111708743A (en) * | 2020-05-28 | 2020-09-25 | 浪潮电子信息产业股份有限公司 | File storage management method, file management client and file storage management system |
| CN112383614A (en) * | 2020-11-11 | 2021-02-19 | 常州微亿智造科技有限公司 | File transmission method and device |
| CN112559464A (en) * | 2020-12-22 | 2021-03-26 | 鲁班(北京)电子商务科技有限公司 | File uploading method and uploading device |
| CN116032913A (en) * | 2022-10-12 | 2023-04-28 | 浪潮云信息技术股份公司 | File uploading method and system based on distributed object storage |
| CN116233120A (en) * | 2023-05-10 | 2023-06-06 | 深圳普菲特信息科技股份有限公司 | Large file fragment transmission method, system and medium based on data processing |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101699822A (en) * | 2009-08-06 | 2010-04-28 | 腾讯科技（深圳）有限公司 | File uploading method and device, and mass storage system |
| US20100248842A1 (en) * | 2008-04-30 | 2010-09-30 | Bally Gaming, Inc | Download and Data Transfer Gaming Method |
| CN103166992A (en) * | 2011-12-14 | 2013-06-19 | 中国电信股份有限公司 | Method and system of fast file updating applied to cloud storage |
| CN103414759A (en) * | 2013-07-22 | 2013-11-27 | 华为技术有限公司 | Network disc file transmission method and network disc file transmission device |
| CN105635324A (en) * | 2016-03-17 | 2016-06-01 | 新浪网技术(中国)有限公司 | Big file uploading and continuous uploading method and device for browser or server |
| CN106202173A (en) * | 2016-06-26 | 2016-12-07 | 厦门天锐科技股份有限公司 | The Intelligent drainage weighing method of a kind of file repository storage and system |
-
2018
- 2018-12-13 CN CN201811527148.5A patent/CN109361772A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100248842A1 (en) * | 2008-04-30 | 2010-09-30 | Bally Gaming, Inc | Download and Data Transfer Gaming Method |
| CN101699822A (en) * | 2009-08-06 | 2010-04-28 | 腾讯科技（深圳）有限公司 | File uploading method and device, and mass storage system |
| CN103166992A (en) * | 2011-12-14 | 2013-06-19 | 中国电信股份有限公司 | Method and system of fast file updating applied to cloud storage |
| CN103414759A (en) * | 2013-07-22 | 2013-11-27 | 华为技术有限公司 | Network disc file transmission method and network disc file transmission device |
| CN105635324A (en) * | 2016-03-17 | 2016-06-01 | 新浪网技术(中国)有限公司 | Big file uploading and continuous uploading method and device for browser or server |
| CN106202173A (en) * | 2016-06-26 | 2016-12-07 | 厦门天锐科技股份有限公司 | The Intelligent drainage weighing method of a kind of file repository storage and system |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110650207A (en) * | 2019-09-29 | 2020-01-03 | 中电福富信息科技有限公司 | Method for uploading break point of super-large video file |
| CN111209254A (en) * | 2020-01-10 | 2020-05-29 | Oppo（重庆）智能科技有限公司 | File fingerprint acquisition method and device, electronic equipment and storage medium |
| CN111708743A (en) * | 2020-05-28 | 2020-09-25 | 浪潮电子信息产业股份有限公司 | File storage management method, file management client and file storage management system |
| CN112383614A (en) * | 2020-11-11 | 2021-02-19 | 常州微亿智造科技有限公司 | File transmission method and device |
| CN112559464A (en) * | 2020-12-22 | 2021-03-26 | 鲁班(北京)电子商务科技有限公司 | File uploading method and uploading device |
| CN116032913A (en) * | 2022-10-12 | 2023-04-28 | 浪潮云信息技术股份公司 | File uploading method and system based on distributed object storage |
| CN116233120A (en) * | 2023-05-10 | 2023-06-06 | 深圳普菲特信息科技股份有限公司 | Large file fragment transmission method, system and medium based on data processing |
| CN116233120B (en) * | 2023-05-10 | 2023-07-14 | 深圳普菲特信息科技股份有限公司 | Large file fragment transmission method, system and medium based on data processing |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109361772A (en) | File uploading method and device in distributed object storage system | |
| CN106991607B (en) | Disordered transaction control method based on block chain account model | |
| CN110703980B (en) | File transmission method and device | |
| CN106354865B (en) | Method, device and system for synchronizing master database and slave database | |
| CN107770229B (en) | Cloud file sharing method and system and cloud server | |
| CN105739971B (en) | Verify generation, application method and the device of file | |
| CN111033487A (en) | Microservice-based deduplication | |
| CN105677754B (en) | Obtain the methods, devices and systems of subitem metadata in file system | |
| CN110457128B (en) | Task allocation method, device and system | |
| CN106776785B (en) | Data writing method and device and data processing system | |
| CN111033459A (en) | Microservice-based deduplication optimized for wide area networks | |
| CN104573064A (en) | Data processing method under big-data environment | |
| WO2019080232A1 (en) | Task information transmission method and apparatus in workflow system, and computer device | |
| CN106775470B (en) | Data storage method and system | |
| CN104980489A (en) | Secure collection synchronization using matched network names | |
| CN101771548A (en) | File synchronizing method and system | |
| CN107204998B (en) | Method and device for processing data | |
| CN105653209A (en) | Object storage data transmitting method and device | |
| CN109491976A (en) | A kind of storing data analytic method, device and relevant device | |
| CN105991282B (en) | Password generated method and device | |
| CN104125294A (en) | Method and system for big data safety management | |
| CN108540510A (en) | A kind of cloud host creation method, device and cloud service system | |
| CN113256291A (en) | Method and device for creating account and distributing transaction in blockchain system | |
| CN104023070B (en) | file compression method based on cloud storage | |
| CN105205011B (en) | A kind of method, normal client end and management client obtaining blocks of files reference count |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190219 |