CN102035881B - Data caching method of cloud storage system - Google Patents
Data caching method of cloud storage system Download PDFInfo
- Publication number
- CN102035881B CN102035881B CN 201010554529 CN201010554529A CN102035881B CN 102035881 B CN102035881 B CN 102035881B CN 201010554529 CN201010554529 CN 201010554529 CN 201010554529 A CN201010554529 A CN 201010554529A CN 102035881 B CN102035881 B CN 102035881B
- Authority
- CN
- China
- Prior art keywords
- data
- file
- cache
- block
- stores service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a data caching method of a cloud storage system, belonging to the technical field of computer data storage. The method comprises the steps of: taking a flash disk as a carrier of the cloud storage system, starting a computer from the flash disk and loading an operation system in a system area of the flash disk into a memory of the computer; scanning data block caching files in a local data caching area by the cloud storage system and storing each obtained data block caching file in the memory; capturing the operation commands of an upper application program by the cloud storage system through a virtual file system layer, wherein the operation commands comprise the steps of creating the files, modifying the files, reading the files and deleting the files, and the new data blocks formed after the operation are cached in the local data caching area in file form; and storing the formed new data block caching files to a cloud storage server by a network communication module and obtaining data block caching files from the cloud storage server when the needed data block caching files do not exist in the local data caching area. The method improves the response speed and realizes the access of the cloud storage data under the situation of off-network.
Description
Technical field
The invention belongs to technical field of computer data storage, specially refer to method data cached in a kind of cloud storage system.
Background technology
Along with the fast development of information technology, the social informatization degree is more and more higher, and the digital information that the individual has also is explosive trend growth, and under such background, memory device has become requisite instrument in people's daily life.But various numerous and diverse memory devices have also brought a lot of problems simultaneously, for example how to guarantee data's consistency between a plurality of memory devices of personal user, how to guarantee the safe and reliable of the data deposited in all memory devices, how to solve memory device limited space etc.Therefore, how to provide a kind of efficiently, be convenient to manage, the stores service of space dynamic growth become the hot issue of research.
The new breakthrough that the fast development of computer networking technology brings to memory technology, network attached storage (NAS), the technology such as storage area network (SAN) are in the ascendant, brought change to a great extent for traditional memory technology, but these technology are very expensive on the one hand, are not suitable on the other hand being applied in the wide area network scope.The concept of the cloud storage (Cloud Storage) that is proposed by companies such as SNIA (SNIA) and Amazons (Amazon) has proposed revolutionary conception for present Development of storage technology trend to the network storage in future: the data storage service can be accomplished just as the service of present power and water, network is routed to every household, various expenses standards are provided, provide different services to different user.Doing so on the one hand can be for the user provides the very stores service of high-quality, and arbitrfary point access in the dynamic growth service of memory space and the network for example can be provided; Accomplished on the other hand user transparent, all given special cloud storage service provider with all technical problems and go to solve, the problems such as the reliability that the user needn't concern of data and fail safe; Simultaneously can also be for the user provides the more cheap stores service of high-quality, so that the user needn't spend a large amount of expense maintenance upgrade storage systems.
The cloud storage generally is divided into two major parts according to framework, is respectively cloud stores service and cloud storage system.The cloud stores service refers to that large-scale company of several families is deployed in data, services in the whole internet environment as what the service provider provided, can use certain interface that the cloud stores service is carried out corresponding data access, the data that leave in the cloud stores service are called cloud storage data.Cloud storage system refers to install in client the storage system of deployment, generally comprise the kernel module that capable of dynamic loads, be used for catching file-system command, the Executive Module that carries out alternately with network communication module that cloud storage data are provided and be used for the order that captures is processed with cloud stores service interface, Executive Module is undertaken alternately by network communication module and cloud stores service, and different cloud storage system differences are embodied in the Executive Module.The file that is stored in the cloud storage system is the same with file on being stored in hard disk, also formed by a lot of data blocks in logic, but different be that the data block in the cloud storage system also is file, is referred to as the data block file.Cloud storage system be responsible for the data block file be stored in the cloud stores service, obtain data block file in the cloud stores service, according to data and management data block file in the file-system command Update Table block file of catching.
The cloud storage service provider of main flow comprises the S3 of Amazon (Amazon) and the mesh of Microsoft at present.For different cloud storage service providers, some corresponding cloud storage systems have also appearred.Representative cloud storage system comprises Dropbox and SugarSync etc., and they all are to utilize in client to deposit a complete data backup, then after each the modification, calculates residual quantity data back cloud stores service; When cloud storage system access cloud storage data, needs at first are stored in client with the download portion of data integrity, and then operate for the data of depositing to client.There is following obvious shortcoming in such strategy: one, the inefficiency of access cloud storage data: during cloud storage system access cloud stores service, need all cloud storage data of indiscriminate download to client, can operate accordingly after all download is over, so that the user need to spend the downloading process of plenty of time waiting time, if and the user only needs to operate a small documents, but have to all data are all downloaded from the cloud stores service; Two, opaque to the user: the user can not accomplish to access cloud storage data as access local file system data, need to open corresponding cloud storage system at every turn, carries out just can having access to cloud storage data after the data simultaneous operation; Three, too responsive to network state, if network failure appears in cloud storage system to cloud stores service request msg the time, then all cloud storage data are all unavailable;
Summary of the invention
Order of the present invention is for overcoming the weak point of prior art, a kind of data cache method of cloud storage system is proposed, the method is utilized the strategy of data cached block file and the advantage of program locality (locality), reach the purpose that improves response speed, can also be implemented in the cloud storage data in the access local data cache district in the suspension situation simultaneously.
The data cache method of a kind of cloud storage system that the present invention proposes, it is characterized in that, the method uses flash disk (or to use SIM card, the storage mediums such as SSD dish) as the carrier of high in the clouds file system, and flash disk is divided into system area and data field, deposit operation system in the system area, computer starts from flash disk; This data field Further Division is local data cache district and metadata database, deposits the data-block cache file that obtains from the cloud stores service in the local data cache district, and metadata database is used for recording the descriptor of cloud storage data;
Cloud storage system in the method comprises the kernel module that capable of dynamic loads, network communication module, and the Executive Module that can carry out cache management; The method may further comprise the steps:
1) adopts flash disk as the carrier of cloud storage system, start computer from flash disk, the operating system in the flash disk system area is loaded in the calculator memory; Cloud storage system moves along with the startup of operating system as background program, and the kernel module that the capable of dynamic of cloud storage system is loaded joins in the operating system nucleus, so that cloud storage system is local file system to user's the form of expression;
2) the data-block cache file in the Executive Module scanning of home data buffer area of the carried out cache management of cloud storage system, data block numbering in the file that this data-block cache file corresponding to each data-block cache file that obtains formed leaves in the ordered list in the internal memory;
3) network communication module is carried out initialization, set up message queue, carrying out alternately with the cloud stores service on the internet by message queue; Upper level applications is passed through the POSIX file system interface to virtual file system (VFS) Transmit message system command;
4) kernel module of the capable of dynamic of cloud storage system loading sees through virtual file system and catches the operational order that upper level applications creates file, revised file, file reading and deleted file, and these orders are redirected to the Executive Module that can carry out cache management;
The Executive Module that 5) can carry out cache management carries out the operational order that captures concrete establishment file operation, writes data to existing file operation, file reading data manipulation and delete file operation, and the new data block that forms after these operations is cached in the local data cache district with the form of file; Data when writing data to existing file and reading existing file in the local data cache district surpass setting threshold, the Executive Module that triggering can be carried out cache management carries out the buffer memory replacement to the data-block cache file in the local data cache district, and when needed data-block cache file during not in the local data cache district, obtain the data-block cache file to the cloud stores service;
6) network communication module is with step 5) in the new data-block cache file that forms deposit back the cloud stores service, and when needed data-block cache file during not in the local data cache district, obtain from the cloud stores service.
Characteristics of the present invention and beneficial effect are:
1, utilizes method of the present invention, can utilize the local data cache district effectively to accelerate the response speed of cloud storage system, bring better user to experience;
2, utilize method of the present invention, can be implemented in the cloud that access is left in the local data cache district under the suspension state and store data, and present existing cloud storage system is too responsive to network state, can't work fully at the suspension state;
3, compare with existing cloud storage system, the inventive method can realize the operation of the random read-write of file, thereby can accomplish to support the operations such as video playback that drag at random, and just can carry out associative operation after not needing pending file to download to this locality;
4, the inventive method has been avoided the beyond the clouds existing data of transmission the other side between the storage system and cloud stores service, decrease network overhead;
The pressure to the cloud stores service that has brought when 5, the inventive method has been avoided a large amount of read-write operation, if the data-block cache file leaves in the local data cache district, then directly data block cache file is operated the pressure that has brought to the cloud stores service when having avoided in a large number to cloud stores service request msg;
6, compare with existing cloud storage system, the inventive method is particularly useful for having the cloud storage environment that network between Bandwidth-Constrained, cloud storage system and the cloud stores service between cloud storage system local memory device finite capacity, cloud storage system and the cloud stores service is prone to fault characteristic.
Description of drawings
Fig. 1 is the inventive method schematic diagram.
Fig. 2 is the process chart that cloud storage system response application program writes data to the existing file order in the inventive method.
Fig. 3 is the process chart that cloud storage system response application program reads the existing file data command in the inventive method.
Fig. 4 is the process chart of cloud storage system response application program deleted file order in the inventive method.
Embodiment
The data cache method of a kind of cloud storage system that the present invention proposes reaches by reference to the accompanying drawings embodiment and is described in detail as follows:
The present invention uses flash disk (or using SIM card, the storage mediums such as SSD dish) as the carrier of high in the clouds file system, and flash disk is divided into system area and data field, deposit operation system in the system area, and computer starts from flash disk; This data field Further Division is local data cache district and metadata database, deposits the data-block cache file that obtains from the cloud stores service in the local data cache district, and metadata database is used for recording the descriptor of cloud storage data;
Cloud storage system in the method comprises the kernel module that capable of dynamic loads, network communication module, and the Executive Module that can carry out cache management; The method may further comprise the steps:
1) adopts flash disk as the carrier of cloud storage system, start computer from flash disk, the operating system in the flash disk system area is loaded in the calculator memory; Cloud storage system moves along with the startup of operating system as background program, and the kernel module that the capable of dynamic of cloud storage system is loaded joins in the operating system nucleus, so that cloud storage system is local file system to user's the form of expression;
2) the data-block cache file in the Executive Module scanning of home data buffer area of the carried out cache management of cloud storage system, data block numbering in the file that this data-block cache file corresponding to each data-block cache file that obtains formed leaves in the ordered list in the internal memory;
3) network communication module is carried out initialization, set up message queue, carrying out alternately with the cloud stores service on the internet by message queue; Upper level applications is passed through the POSIX file system interface to virtual file system (VFS) Transmit message system command;
4) kernel module of the capable of dynamic of cloud storage system loading sees through virtual file system and catches the operational order that upper level applications creates file, revised file, file reading and deleted file, and these orders are redirected to the Executive Module that can carry out cache management;
The Executive Module that 5) can carry out cache management carries out the operational order that captures concrete establishment file operation, writes data to existing file operation, file reading data manipulation and delete file operation, and the new data block that forms after these operations is cached in the local data cache district with the form of file; Data when writing data to existing file and reading existing file in the local data cache district surpass setting threshold, the Executive Module that triggering can be carried out cache management carries out the buffer memory replacement to the data-block cache file in the local data cache district, and when needed data-block cache file during not in the local data cache district, obtain the data-block cache file to the cloud stores service;
6) network communication module is with step 5) in the new data-block cache file that forms deposit back the cloud stores service, and when needed data-block cache file during not in the local data cache district, obtain from the cloud stores service.As shown in Figure 1.
Data-block cache file in the described flash disk local data cache district is used for forming can be for the file of application program; File (can for the file of application program) information table, data block information table and document composition table are set in the metadata database.
Described file information table is as shown in table 1, in this table record all leave file metadata information in the cloud stores service in by cloud storage system, comprise the creation-time (Ctime), modification time (Mtime) of file identification, file size, file type, filename, file parent directory sign, access privilege and file and last access time (Vtime);
Table 1: file information table
Described data block information table is as shown in table 2, data block sign, number of references and the data block size stored in the record cloud stores service in this table, and the data block size maximum of present embodiment can be set to 10MB;
Table 2: data block information table
| The data block sign | Number of references | The data block size |
| A | 2 | 10 |
| B | 1 | 8 |
| C | 3 | 10 |
| ... | ... | ... |
Described document composition table is as shown in table 3, and the data block information of record composing document comprises file identification in this table, data block sign and data block piece number;
Table 3: document composition table
| File identification | The data block sign | The data block numbering |
| 1 | C | 0 |
| 1 | B | 1 |
| ... | ... | ... |
Existing data-block cache file in the Executive Module scanning of home data buffer area of the carried out cache management of cloud storage system, obtain each data-block cache file correspondence data block numbering hereof, leave in the ordered list in the internal memory, ordered list is used for the corresponding data block cache file of quick search data-oriented block identification whether in the local data cache district, if in the local data cache district, then return corresponding data-block cache file filename, if do not exist, then return not information (generally being made as 0);
Described step 5) the establishment file operation in is included in adds the metadata information that is created file in metadata database file information table and the document composition table, the mode of thereafter database file being transmitted with residual quantity backups to the cloud stores service;
Described step 5) the existing file operation that writes data in specifically may further comprise the steps as shown in Figure 2:
The parameter that writes order that (5-11) kernel module will capture passes to the Executive Module that can carry out cache management, and command parameter comprises file identification, writes character array pointer and length to be written; Can carry out the Executive Module of cache management and judge at first whether the character array length that writes comprises a plurality of data blocks, if so, then with data truncation to be written, the data writing after blocking is guaranteed all in a data block, remaining this process of datacycle writes, until write; Then can carry out the Executive Module of cache management according to the filename inquiry file information table in the command parameter, obtain the file identification of this document, form table by file identification and the document misregistration inquiry file in the command parameter that obtains, acquisition data block sign, by data block sign inquiry ordered list, judge data-block cache file corresponding to this document whether in the local data cache district;
If (5-12) judgement draws the data-block cache file not in the local data cache district, then send to the cloud stores service according to the file identification that obtains in (5-11) and data block sign and obtain request of data, the cloud stores service finds corresponding data-block cache file and passes back according to file identification and data block sign and is saved in the local data cache district; If the data-block cache file in the local data cache district, is then skipped this step;
(5-13) data in the data-block cache file are read in the calculator memory, according to the command parameter that writes that obtains in (5-11), character array to be written is write in this region of memory, this region of memory is calculated cryptographic Hash, by cryptographic Hash data query block information table, if the data consistent in the data in this region of memory and certain data-block cache file, then the number of references with this data-block cache file adds 1, otherwise, data in the region of memory are written in the local data buffering area, deposit into a new data-block cache file, and be dirty (dirty) with this new data-block cache file identification, the notice kernel module successfully writes, and kernel module and then notice upper level applications successfully write;
(5-14) according to the size of data writing, revise file information table, data block information table and document composition table in the metadata database, the mode of thereafter database file being transmitted with residual quantity backups to the cloud stores service;
(5-15) when local data cache district amount of capacity surpasses setting threshold (generally be made as total capacity size 2/3rds), begin that the local data cache district is carried out buffer memory and replace; At first will not be labeled as dirty data-block cache file and transfer back to the cloud stores service, and delete these data-block cache files; If local data cache district amount of capacity then stops the buffer memory replacement process less than setting threshold at this moment; If the size in local data cache district then adopts the LRU replace Algorithm to replace being labeled as dirty data-block cache file still greater than setting threshold at this moment; Replace and specifically to comprise according to the last access time data block cache file is sorted, calling successively cloud stores service interface will be stored in the cloud stores service with current and be labeled as the data-block cache file deletion of the same name of dirty data-block cache file, thereafter will be labeled as dirty data-block cache file and be transmitted back to the cloud stores service, then this data-block cache file be deleted from the local data cache district.In case local data cache district amount of capacity then stops the buffer memory replacement process less than setting threshold;
Described step 5) read the existing file data manipulation as shown in Figure 3 in, specifically may further comprise the steps:
(5-21) kernel module passes to the Executive Module that can carry out cache management with the parameter of the reading order that captures, and command parameter comprises file identification, core buffer pointer and length to be read; The Executive Module that can carry out cache management is at first judged whether a plurality of data blocks of include file of data length to be read, if comprise a plurality of data blocks, then length to be read is blocked in a data block, and remaining length this process that circulates is read in, until run through; Then can carry out the Executive Module of cache management according to the filename inquiry file information table in the command parameter, obtain the file identification of this document, form table by file identification and the document misregistration inquiry file in the command parameter that obtains, acquisition data block sign, by data block sign inquiry ordered list, judge data-block cache file corresponding to this document whether in the local data cache district;
If (5-22) judgement draws the data-block cache file not in the local data cache district, then send to the cloud stores service according to the file identification that obtains and data block sign and obtain request of data, the cloud stores service finds corresponding data-block cache file and passes back according to file identification and data block sign and is saved in the local data cache district; If the data-block cache file is in the local data cache district, then this step is skipped;
(5-23) copy to the local data cache district from the cloud stores service when data-block cache file to be read, cache manager is read into internal memory with the data-block cache file, according to the reading order parameter that obtains in (5-21), the corresponding data of read block cache file, and the data that read are returned to kernel module, kernel module and then data are returned to upper layer application;
(5-24) last access time item corresponding to institute's file reading sign in the revised file information table, the mode of thereafter database file being transmitted with residual quantity backups to the cloud stores service;
Described step 5) the deletion existing file order in specifically may further comprise the steps as shown in Figure 4:
(5-31) kernel module passes to the Executive Module that can carry out cache management with the parameter of the deleted file order that captures, and command parameter comprises file identification; Form table according to the file identification inquiry file, obtain the data block sign of composing document;
(5-32) according to the result queries data block information table that obtains, number of references corresponding to the sign of the data block among the result subtracted 1, when number of references is 0, this data block sign is saved in the delete list;
(5-33) according to the data block in delete list sign, inquire about successively ordered list, judge that data-block cache file corresponding to this data block sign whether in the local data cache district, if in the local data cache district, then delete the data-block cache file;
(5-34) according to the sign of the data block in the delete list, call successively the deleted file interface of cloud stores service, the respective data blocks cache file in the cloud stores service is deleted;
(5-35) metadata information of deletion deleted file in metadata database file information table and document composition table, the mode of thereafter database file being transmitted with residual quantity backups to the cloud stores service;
(5-36) identify according to the data block in the delete list, delete successively the entry in the data block information table, then the respective record item in deleted file information table and the document composition table is deleted successfully by return value notice kernel module, kernel module and then the success of notice upper layer application deleted file.
Claims (6)
1. the data cache method of a cloud storage system is characterized in that, the method uses flash disk as the carrier of high in the clouds file system, and flash disk is divided into system area and data field, deposit operation system in the system area, and computer starts from flash disk; This data field Further Division is local data cache district and metadata database, deposits the data-block cache file that obtains from the cloud stores service in the local data cache district, and metadata database is used for recording the descriptor of cloud storage data; Cloud storage system in the method comprises the kernel module that capable of dynamic loads, network communication module, and the Executive Module that can carry out cache management; The method may further comprise the steps:
1) adopts flash disk as the carrier of cloud storage system, start computer from flash disk, the operating system in the flash disk system area is loaded in the calculator memory; Cloud storage system moves along with the startup of operating system as background program, and the kernel module that the capable of dynamic of cloud storage system is loaded joins in the operating system nucleus, so that cloud storage system is local file system to user's the form of expression;
2) the data-block cache file in the Executive Module scanning of home data buffer area of the carried out cache management of cloud storage system, data block numbering in the file that this data-block cache file corresponding to each data-block cache file that obtains formed leaves in the ordered list in the internal memory;
3) network communication module is carried out initialization, set up message queue, carrying out alternately with the cloud stores service on the internet by message queue; Upper level applications is passed through the POSIX file system interface to virtual file system (VFS) Transmit message system command;
4) kernel module of the capable of dynamic of cloud storage system loading sees through virtual file system and catches the operational order that upper level applications creates file, revised file, file reading and deleted file, and these orders are redirected to the Executive Module that can carry out cache management;
The Executive Module that 5) can carry out cache management carries out the operational order that captures concrete establishment file operation, writes data to existing file operation, file reading data manipulation and delete file operation, and the new data block that forms after these operations is cached in the local data cache district with the form of file; Data when writing data to existing file and reading existing file in the local data cache district surpass setting threshold, the Executive Module that triggering can be carried out cache management carries out the buffer memory replacement to the data-block cache file in the local data cache district, and when needed data-block cache file during not in the local data cache district, obtain the data-block cache file to the cloud stores service;
6) network communication module is with step 5) in the new data-block cache file that forms deposit back the cloud stores service, and when needed data-block cache file during not in the local data cache district, obtain from the cloud stores service.
2. method as claimed in claim 1 is characterized in that the data-block cache file in the described flash disk local data cache district is used for forming can be for the file of application program; File information table, data block information table and document composition table are set in the metadata database; In the described file information table record all leave file metadata information in the cloud stores service in by cloud storage system, comprise the creation-time (Ctime), modification time (Mtime) of file identification, file size, file type, filename, file parent directory sign, access privilege and file and last access time (Vtime).
3. method as claimed in claim 2, it is characterized in that, described step 5) the establishment file operation in is included in adds the metadata information that is created file in metadata database file information table and the document composition table, the mode of thereafter database file being transmitted with residual quantity backups to the cloud stores service.
4. method as claimed in claim 2 is characterized in that described step 5) in the existing file operation that writes data to specifically may further comprise the steps:
The parameter that writes order that (5-11) kernel module will capture passes to the Executive Module that can carry out cache management; Can carry out the Executive Module of cache management and judge at first whether the character array length that writes comprises a plurality of data blocks, if so, then with data truncation to be written, the data writing after blocking is guaranteed all in a data block, remaining this process of datacycle writes, until write; Then can carry out the Executive Module of cache management according to the filename inquiry file information table in the command parameter, obtain the file identification of this document, form table by file identification and the document misregistration inquiry file in the command parameter that obtains, acquisition data block sign, by data block sign inquiry ordered list, judge data-block cache file corresponding to this document whether in the local data cache district;
If (5-12) judgement draws the data-block cache file not in the local data cache district, then send to the cloud stores service according to the file identification that obtains in (5-11) and data block sign and obtain request of data, the cloud stores service finds corresponding data-block cache file and passes back according to file identification and data block sign and is saved in the local data cache district; If the data-block cache file in the local data cache district, is then skipped this step;
(5-13) data in the data-block cache file are read in the calculator memory, according to the command parameter that writes that obtains in (5-11), character array to be written is write in this region of memory, this region of memory is calculated cryptographic Hash, by cryptographic Hash data query block information table, if the data consistent in the data in this region of memory and certain data-block cache file, then the number of references with this data-block cache file adds 1, otherwise, data in the region of memory are written in the local data buffering area, deposit into a new data-block cache file, and be dirty (dirty) with this new data-block cache file identification, the notice kernel module successfully writes, and kernel module and then notice upper level applications successfully write;
(5-14) according to the size of data writing, revise file information table, data block information table and document composition table in the metadata database, the mode of thereafter database file being transmitted with residual quantity backups to the cloud stores service;
(5-15) when local data cache district amount of capacity surpasses setting threshold, begin that the local data cache district is carried out buffer memory and replace; At first will not be labeled as dirty data-block cache file and transfer back to the cloud stores service, and delete these data-block cache files; If local data cache district amount of capacity then stops the buffer memory replacement process less than setting threshold at this moment; If the size in local data cache district then adopts the LRU replace Algorithm to replace being labeled as dirty data-block cache file still greater than setting threshold at this moment; Replace and specifically to comprise according to the last access time data block cache file is sorted, calling successively cloud stores service interface will be stored in the cloud stores service with current and be labeled as the data-block cache file deletion of the same name of dirty data-block cache file, thereafter will be labeled as dirty data-block cache file and be transmitted back to the cloud stores service, then this data-block cache file be deleted from the local data cache district; In case local data cache district amount of capacity then stops the buffer memory replacement process less than setting threshold.
5. method as claimed in claim 2 is characterized in that described step 5) in the file reading data manipulation specifically may further comprise the steps:
(5-21) kernel module passes to the Executive Module that can carry out cache management with the parameter of the reading order that captures; The Executive Module that can carry out cache management is at first judged whether a plurality of data blocks of include file of data length to be read, if comprise a plurality of data blocks, then length to be read is blocked in a data block, and remaining length this process that circulates is read in, until run through; Then can carry out the Executive Module of cache management according to the filename inquiry file information table in the command parameter, obtain the file identification of this document, form table by file identification and the document misregistration inquiry file in the command parameter that obtains, acquisition data block sign, by data block sign inquiry ordered list, judge data-block cache file corresponding to this document whether in the local data cache district;
If (5-22) judgement draws the data-block cache file not in the local data cache district, then send to the cloud stores service according to the file identification that obtains and data block sign and obtain request of data, the cloud stores service finds corresponding data-block cache file and passes back according to file identification and data block sign and is saved in the local data cache district; If the data-block cache file is in the local data cache district, then this step is skipped;
(5-23) copy to the local data cache district from the cloud stores service when data-block cache file to be read, cache manager is read into internal memory with the data-block cache file, according to the reading order parameter that obtains in (5-21), the corresponding data of read block cache file, and the data that read are returned to kernel module, kernel module and then data are returned to upper layer application;
(5-24) last access time item corresponding to institute's file reading sign in the revised file information table, the mode of thereafter database file being transmitted with residual quantity backups to the cloud stores service.
6. method as claimed in claim 2 is characterized in that described step 5) in delete file operation specifically may further comprise the steps:
(5-31) kernel module passes to the Executive Module that can carry out cache management with the parameter of the deleted file order that captures, and command parameter comprises file identification; Form table according to the file identification inquiry file, obtain the data block sign of composing document;
(5-32) according to the result queries data block information table that obtains, number of references corresponding to the sign of the data block among the result subtracted 1, when number of references is 0, this data block sign is saved in the delete list;
(5-33) according to the data block in delete list sign, inquire about successively ordered list, judge that data-block cache file corresponding to this data block sign whether in the local data cache district, if in the local data cache district, then delete the data-block cache file;
(5-34) according to the sign of the data block in the delete list, call successively the deleted file interface of cloud stores service, the respective data blocks cache file in the cloud stores service is deleted;
(5-35) metadata information of deletion deleted file in metadata database file information table and document composition table, the mode of thereafter database file being transmitted with residual quantity backups to the cloud stores service;
(5-36) identify according to the data block in the delete list, delete successively the entry in the data block information table, then the respective record item in deleted file information table and the document composition table is deleted successfully by return value notice kernel module, kernel module and then the success of notice upper layer application deleted file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010554529 CN102035881B (en) | 2010-11-19 | 2010-11-19 | Data caching method of cloud storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010554529 CN102035881B (en) | 2010-11-19 | 2010-11-19 | Data caching method of cloud storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102035881A CN102035881A (en) | 2011-04-27 |
| CN102035881B true CN102035881B (en) | 2013-01-16 |
Family
ID=43888198
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010554529 Active CN102035881B (en) | 2010-11-19 | 2010-11-19 | Data caching method of cloud storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102035881B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI576703B (en) * | 2015-03-27 | 2017-04-01 | 宏碁股份有限公司 | Electronic apparatus and method for temporarily storing data thereof |
Families Citing this family (54)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999398B (en) * | 2011-09-15 | 2014-06-11 | 腾讯科技（深圳）有限公司 | Method, system and device for user system recovery |
| CN103023939B (en) * | 2011-09-26 | 2017-10-20 | 中兴通讯股份有限公司 | The method and system of the REST interfaces of cloud caching is realized on Nginx |
| CN102355502B (en) * | 2011-10-13 | 2015-02-04 | 罗立群 | Remote access method for remotely accessing storage system into desktop operation system |
| CN102368737A (en) * | 2011-11-25 | 2012-03-07 | 裘嘉 | Cloud storage system and data access method thereof |
| CN102404411A (en) * | 2011-12-23 | 2012-04-04 | 创新科存储技术有限公司 | Data synchronization method of cloud storage system |
| CN103209195A (en) * | 2012-01-11 | 2013-07-17 | 国家电网公司 | Data acquisition method, terminal and far-end device |
| CN103309818B (en) * | 2012-03-09 | 2015-07-29 | 腾讯科技（深圳）有限公司 | Store method and the device of data |
| CN102662870B (en) * | 2012-03-20 | 2014-08-13 | 武汉噢易科技有限公司 | Android operation system protection method based on input/output request intercepted by VFS (virtual file system) layer |
| CN102710742A (en) * | 2012-04-11 | 2012-10-03 | 佳都新太科技股份有限公司 | Offline file synchronous processing mechanism based on cloud computing virtual application |
| CN102932415B (en) * | 2012-09-26 | 2015-04-15 | 东软集团股份有限公司 | Method and device for storing mirror image document |
| CN102946415B (en) * | 2012-10-18 | 2016-01-27 | 北京华胜鸣天科技有限公司 | A kind of implementation method of mobile terminal this locality application and device |
| CN102970358B (en) * | 2012-11-08 | 2016-06-15 | 百度在线网络技术（北京）有限公司 | The control method of the local cache of mobile client and device in net dish |
| CN103064637A (en) * | 2012-12-24 | 2013-04-24 | 珠海金山办公软件有限公司 | Network disk cache synchronizing method and system |
| CN103078907B (en) * | 2012-12-26 | 2016-03-30 | 华为技术有限公司 | Upload, cloud backs up, search, recover method and the device of data |
| CN103118138B (en) * | 2013-03-04 | 2016-02-10 | 中国信息安全测评中心 | A kind of message queue method for stream processing based on cloud service |
| CN104035814A (en) * | 2013-03-07 | 2014-09-10 | 联想(北京)有限公司 | Data processing method and electronic device |
| CN103324551A (en) * | 2013-05-23 | 2013-09-25 | 广东欧珀移动通信有限公司 | Application data migration method and system |
| AU2013403132B2 (en) | 2013-10-18 | 2016-11-24 | Huawei Technologies Co., Ltd. | Data storage method, data storage apparatus, and storage device |
| CN104717175B (en) * | 2013-12-11 | 2018-07-13 | 中国电信股份有限公司 | The processing method and system of virtual desktop |
| CN103701719A (en) * | 2013-12-27 | 2014-04-02 | 乐视网信息技术（北京）股份有限公司 | Method and system for updating data queues in server |
| CN103984780A (en) * | 2014-06-09 | 2014-08-13 | 王松豪 | Enterprise data storage system |
| CN104158863A (en) * | 2014-08-02 | 2014-11-19 | 成都致云科技有限公司 | Cloud storage mechanism based on transaction-level whole-course high-speed buffer |
| CN105812436A (en) * | 2014-12-31 | 2016-07-27 | ***通信集团公司 | Heterogeneous storage operation method and device |
| CN104537085B (en) * | 2014-12-31 | 2019-03-26 | 广州华多网络科技有限公司 | Image processing method and system based on cloud storage |
| CN106155921B (en) * | 2015-04-03 | 2019-06-25 | 联想(北京)有限公司 | A kind of information processing method and electronic equipment |
| CN106302601A (en) * | 2015-06-04 | 2017-01-04 | 中兴通讯股份有限公司 | Network disk this locality carry processing method, Apparatus and system |
| WO2017031637A1 (en) * | 2015-08-21 | 2017-03-02 | 华为技术有限公司 | Memory access method, apparatus and system |
| CN105788030B (en) * | 2016-03-25 | 2018-07-06 | 航天科技控股集团股份有限公司 | A kind of cycle for saving memory replaces formula data transmission method for uplink |
| CN107294928A (en) * | 2016-04-05 | 2017-10-24 | 北京优朋普乐科技有限公司 | A kind of terminal access CDN method and system, driving and CDN |
| CN106021381A (en) * | 2016-05-11 | 2016-10-12 | 北京搜狐新媒体信息技术有限公司 | Data access/storage method and device for cloud storage service system |
| CN106155934B (en) * | 2016-06-27 | 2019-08-09 | 华中科技大学 | Caching method based on repeated data under a kind of cloud environment |
| CN106161637A (en) * | 2016-07-21 | 2016-11-23 | 广州杰赛科技股份有限公司 | A kind of cloud backup method and device |
| CN107818111B (en) * | 2016-09-13 | 2021-10-15 | 腾讯科技（深圳）有限公司 | Method for caching file data, server and terminal |
| US11074220B2 (en) | 2017-01-06 | 2021-07-27 | Oracle International Corporation | Consistent file system semantics with cloud object storage |
| CN106959860A (en) * | 2017-03-30 | 2017-07-18 | 山东超越数控电子有限公司 | The implementation method and device of a kind of kernel Dynamic Execution network based on acceptance of the bid kylin operating system |
| CN107122265B (en) * | 2017-05-19 | 2020-10-23 | 苏州浪潮智能科技有限公司 | Application program operation control system and control method in cluster environment |
| US10713238B2 (en) * | 2017-11-14 | 2020-07-14 | Snowflake Inc. | Database metadata in immutable storage |
| CN110096354B (en) * | 2018-01-29 | 2021-06-15 | 华为技术有限公司 | Cloning method and device for application |
| CN109669818B (en) * | 2018-12-20 | 2019-11-12 | 广州鼎甲计算机科技有限公司 | Continuous data protection method and system without local cache |
| CN109783455B (en) * | 2018-12-20 | 2024-03-12 | 昂纳科技(深圳)集团股份有限公司 | Large-scale data-based management method, system and storage device |
| CN110008188B (en) * | 2019-03-31 | 2022-11-18 | 华南理工大学 | External storage quota system of application software at file system level |
| US10880583B2 (en) | 2019-04-26 | 2020-12-29 | Advanced New Technologies Co., Ltd. | Method, apparatus, terminal, and readable storage medium for offline caching |
| CN110267077B (en) * | 2019-04-26 | 2020-11-06 | 创新先进技术有限公司 | Offline caching method, device, terminal and readable storage medium |
| CN110581896A (en) * | 2019-09-30 | 2019-12-17 | 恒信东方文化股份有限公司 | Storage method and system thereof |
| CN112988039B (en) * | 2019-12-18 | 2024-01-30 | 中国电信股份有限公司 | Data uploading method and device, cloud storage gateway and system and storage medium |
| CN111800771B (en) * | 2020-06-30 | 2021-09-21 | 深圳市中科蓝讯科技股份有限公司 | Management method and system for message cache of Bluetooth Mesh node network |
| CN112347060B (en) * | 2020-10-19 | 2023-09-26 | 北京天融信网络安全技术有限公司 | Data storage method, device and equipment of desktop cloud system and readable storage medium |
| CN113377723B (en) * | 2021-07-02 | 2024-04-05 | 厦门雅基软件有限公司 | Cache file management method, device and storage medium |
| CN113377722A (en) * | 2021-07-02 | 2021-09-10 | 厦门雅基软件有限公司 | Resource data reading method and device and storage medium |
| CN113407506A (en) * | 2021-07-06 | 2021-09-17 | 中国电信股份有限公司 | Data processing method, device, equipment and storage medium |
| CN114647387A (en) * | 2022-05-23 | 2022-06-21 | 南京道成网络科技有限公司 | Cache optimization method suitable for cloud storage |
| CN117271073A (en) * | 2022-06-15 | 2023-12-22 | 顺丰科技有限公司 | Execution method and device of computing task |
| CN117555733B (en) * | 2024-01-09 | 2024-03-15 | 贵州联广科技股份有限公司 | Data disaster recovery backup method and system |
| CN117708179B (en) * | 2024-02-02 | 2024-05-03 | 成都深瑞同华科技有限公司 | Method, device, equipment and medium for caching measurement point data of electric power comprehensive monitoring system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101809536A (en) * | 2007-09-28 | 2010-08-18 | 微软公司 | Multi-OS (operating system) boot via mobile device |
| WO2010127365A1 (en) * | 2009-05-01 | 2010-11-04 | Citrix Systems, Inc. | Systems and methods for establishing a cloud bridge between virtual storage resources |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8762642B2 (en) * | 2009-01-30 | 2014-06-24 | Twinstrata Inc | System and method for secure and reliable multi-cloud data replication |
| US8560639B2 (en) * | 2009-04-24 | 2013-10-15 | Microsoft Corporation | Dynamic placement of replica data |
-
2010
- 2010-11-19 CN CN 201010554529 patent/CN102035881B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101809536A (en) * | 2007-09-28 | 2010-08-18 | 微软公司 | Multi-OS (operating system) boot via mobile device |
| WO2010127365A1 (en) * | 2009-05-01 | 2010-11-04 | Citrix Systems, Inc. | Systems and methods for establishing a cloud bridge between virtual storage resources |
Non-Patent Citations (2)
| Title |
|---|
| Research on Management of Data Flow in the Cloud Storage Node Based on Data Block;ZHAN Ying;《2010 Third International Conference on Information and Computing》;20100606;全文 * |
| ZHAN Ying.Research on Management of Data Flow in the Cloud Storage Node Based on Data Block.《2010 Third International Conference on Information and Computing》.2010, |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI576703B (en) * | 2015-03-27 | 2017-04-01 | 宏碁股份有限公司 | Electronic apparatus and method for temporarily storing data thereof |
| US9836468B2 (en) | 2015-03-27 | 2017-12-05 | Acer Incorporated | Electronic apparatus and method for temporarily storing data thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102035881A (en) | 2011-04-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102035881B (en) | Data caching method of cloud storage system | |
| CN103179185B (en) | Method and system for creating files in cache of distributed file system client | |
| US8650159B1 (en) | Systems and methods for managing data in cloud storage using deduplication techniques | |
| CN103020315B (en) | A kind of mass small documents storage means based on master-salve distributed file system | |
| CN103858109B (en) | Information processing system and use the file access pattern method of this information processing system | |
| CN101187901B (en) | High speed cache system and method for implementing file access | |
| JP5485866B2 (en) | Information management method and information providing computer | |
| CN103034684A (en) | Optimizing method for storing virtual machine mirror images based on CAS (content addressable storage) | |
| CN103282899B (en) | The storage method of data, access method and device in file system | |
| CN105183839A (en) | Hadoop-based storage optimizing method for small file hierachical indexing | |
| CN102567427B (en) | Method and device for processing object data | |
| CN103338242B (en) | A kind of mixed cloud storage system based on multi-level buffer and method | |
| CN110647497A (en) | HDFS-based high-performance file storage and management system | |
| CN104935648A (en) | High-cost-performance CDN system, and file pre-push and fragment buffer memory methods | |
| CN103279502B (en) | A kind of framework and method with the data de-duplication file system be combined with parallel file system | |
| US11513996B2 (en) | Non-disruptive and efficient migration of data across cloud providers | |
| CN109522283A (en) | A kind of data de-duplication method and system | |
| CN103455577A (en) | Multi-backup nearby storage and reading method and system of cloud host mirror image file | |
| CN104408111A (en) | Method and device for deleting duplicate data | |
| CN106155934B (en) | Caching method based on repeated data under a kind of cloud environment | |
| CN100424699C (en) | Attribute extensible object file system | |
| CN103916467A (en) | Load transfer method and system in metadata cluster | |
| KR20090063733A (en) | Method recovering data server at the applying multiple reproduce dispersion file system and metadata storage and save method thereof | |
| CN107885616A (en) | A kind of mass small documents back-up restoring method based on file system parsing | |
| CN103501319A (en) | Low-delay distributed storage system for small files |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20161206 Address after: 210042 Xuanwu District, Xuanwu District, Jiangsu, Nanjing, No. 699-22, building 18 Patentee after: CERTUSNET CORP. Address before: 100084 Haidian District Tsinghua Yuan Beijing No. 1 Patentee before: Tsinghua University |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20180404 Address after: Tianhe District Jinsui road Guangzhou City, Guangdong province 510630 U.S. Department of Building No. 3 Patentee after: Mdt InfoTech Ltd, Guangdong Address before: 210042 Xuanwu District, Xuanwu District, Jiangsu, Nanjing, No. 699-22, building 18 Patentee before: CERTUSNET CORP. |
|
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: 510630 room 1601, Pearl River West Road, Tianhe District, Guangzhou, Guangdong, 1601 Patentee after: Mdt InfoTech Ltd, Guangdong Address before: Tianhe District Jinsui road Guangzhou City, Guangdong province 510630 U.S. Department of Building No. 3 Patentee before: Mdt InfoTech Ltd, Guangdong |