CN104462148A - Data storage management method and device - Google Patents
Data storage management method and device Download PDFInfo
- Publication number
- CN104462148A CN104462148A CN201310442311.9A CN201310442311A CN104462148A CN 104462148 A CN104462148 A CN 104462148A CN 201310442311 A CN201310442311 A CN 201310442311A CN 104462148 A CN104462148 A CN 104462148A
- Authority
- CN
- China
- Prior art keywords
- cqfs
- data block
- data
- business datum
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage management method and device. The data storage management method and device are used for solving the problems that a disk is fragmented, and files are prone to being lost. The method comprises the steps that in the system configuration stage, the management device sets up at least one CQFS, and the at least one CQFS comprises CQFS data blocks of the preset number and the preset size; in the system running stage, the management device writes generated service data in each CQFS data block of the at least one CQFS one by one, and every time one CQFS data block is used, a corresponding data block recovery timestamp is generated corresponding to the used CQFS data block. The management device removes the service data in each data block in sequence according to the data block recovery timestamp corresponding to each CQFS data block. Every time the service data in one data block are removed, newly-generated service data continue to be written in the corresponding CQFS data block with the service data removed.
Description
Technical field
The present invention relates to data storage technology, particularly a kind of data storage and management method and device.
Background technology
In safety defense monitoring system, video recording memory function is one of most crucial function, audio frequency, video data that camera is recorded all the time all need to store, and the limited storage space in system, system just needs constantly to reclaim audio frequency, video data in the past, stores new audio frequency, the video data recorded.Traditional universal document system, as New Technology File System (New Technology File System, NTFS) file system under Microsoft WindowsNT environment or third generation extended file system (Third Extended File System, EXT3) Linux environment Journaling File System is in the process of process storage space recovery, a large amount of disk fragmentses can be produced, if things go on like this, there will be the problem of disk performance decay.
Digital monitor system (Digital Surveillance System, DSS-2.x) enhancing Virtual File System (the Enhanced Virtual File System based on NTFS is being used in, ENVFS) time, ENVFS can be pre-created the file of fixed size on NTFS, as Video data block and video recording index block, ENVFS uses above-mentioned several small documents for completing the processes such as video retrieval, the distribution of Video data block, video file distribution to come maintaining file system information, data block assignment information, video file assignment information etc.When system malfunctions, NTFS can carry out data consistent check and reparation, sometimes above-mentioned several small documents that ENVFS creates there will be the situation of loss, will cause the loss of data of whole file system and cannot recover, there is serious data security sex chromosome mosaicism.In addition, after ENVFS long-time running, data space is tending towards discrete after constantly reclaiming, concurrent write point of recording a video also can be tending towards discrete, causes disk fragments, causes system storage performance degradation, more easily cause the loss of Video data, there is the problem of memory property poor stability.
Summary of the invention
The embodiment of the present invention provides a kind of data storage and management method and device, and after solving file system long-time running of the prior art, data space is constantly tending towards discrete, causes disk fragments, and then the problem that generation file is easily lost.
The concrete technical scheme that the embodiment of the present invention provides is as follows:
First aspect, a kind of data storage and management method, comprising:
In the system configuration stage, create at least one round-robin queue file system CQFS file system, in this at least one CQFS file system, comprise the CQFS data block of preset number and default size;
In the system cloud gray model stage, the business datum of generation write in each the CQFS data block in this at least one CQFS file system one by one, wherein, often use a CQFS data block, the corresponding CQFS data block used generates corresponding data block recovery time stamp;
The data block recovery time corresponding according to each CQFS data block stabs the business datum removed successively in each data block; Wherein, often remove the business datum in a data block, in the CQFS data block after removing business datum, continue the business datum of the up-to-date generation of write.
By the method that the embodiment of the present invention provides, this data storage and management method is based on round-robin queue, when CQFS file system is run, write is there will not be to put problem that is discrete or disk fragments, therefore, the long-time running of CQFS file system, write performance can not be decayed, also the problem easily can not lost because of disk fragments and then generation file, further ensures the stability of disk.
In conjunction with first aspect, in the implementation that the first is possible, create at least one CQFS file system, comprising:
If only there is a disk in system, then to creating a CQFS file system by a disk; Or,
If there are at least two disks in system, then each disk corresponding creates a CQFS file system respectively, the write of the business datum in each CQFS file system and remove separate; Or,
If there is at least one disk in system, and there are at least two file directorys in this at least one disk, then each file directory corresponding creates a CQFS file system respectively, the write of the business datum in each CQFS file system and remove separate.
By this possible implementation, between the business datum that each CQFS data block in same CQFS file system stores, height is from complete, does not namely all have data dependence between any two CQFS data blocks.The business datum stored between different CQFS file system yet height, from complete, does not namely all have data dependence between any two CQFS file system.
In conjunction with any one implementation above-mentioned of first aspect, in the implementation that the second is possible, create a CQFS file system, comprising:
Create first area, second area, the 3rd region and the 4th region, wherein,
First area, for storing CQFS header, this CQFS header is used to indicate the configuration information of CQFS file system;
Second area, for storing the list of CQFS data block information; This CQFS data block information list, for recording the descriptor of each CQFS data block, at least comprises the data block recovery time stamp of corresponding CQFS data block in this descriptor;
3rd region, is divided into some CQFS data blocks for storage service data, at least comprises the index information of CQFS data block and the descriptor of corresponding business datum in described CQFS data block;
4th region, for storing the Backup Data of first area and second area.
In conjunction with the implementation that the second of first aspect is possible, in the implementation that the third is possible, create a CQFS file system, comprise further:
Create the 5th region, described 5th region is for storing the distribution list of CQFS data block;
And the 4th region is also further for storing the Backup Data in the 5th region.
In conjunction with the second or the third possible implementation of first aspect, in the 4th kind of possible implementation, in any one CQFS data block write business datum, comprising:
Record traffic data in this any one CQFS data block;
After business datum record, in this any one CQFS data block, record corresponding index information.
In conjunction with the second or the third possible implementation of first aspect, in the 5th kind of possible implementation, after system reboot, comprise further:
The consistance of the descriptor of each CQFS data block of comparison and the descriptor of corresponding business datum;
When determining that the descriptor of the descriptor of any one CQFS data block and corresponding business datum is inconsistent, the descriptor of this any one CQFS data block is reverted to the descriptor of corresponding business datum.
By this possible implementation, carry out in the process of data storage management in CQFS file system, the data of loss can be easy to be detected, and system can recover data in time, solve the problem that file is easily lost.
In conjunction with the 5th kind of possible implementation of first aspect, in the 6th kind of possible implementation, after system reboot, comprise further:
Check data block recovery time stamp, obtain the index information that data block recovery time stabs the business datum in a minimum CQFS data block, judge that this index information is as time empty, produces warning message.
By this possible implementation, after system reboot, can Timeliness coverage data loss problem.
Second aspect, a kind of data storage management device, this device comprises:
Configuration module, in the system configuration stage, creates at least one round-robin queue file system CQFS file system, in this at least one CQFS file system, comprises the CQFS data block of preset number and default size;
First processing module, for in the system cloud gray model stage, the business datum of generation is write one by one in each the CQFS data block in this at least one CQFS file system, wherein, processing module often uses a CQFS data block, and the corresponding CQFS data block used generates corresponding data block recovery time stamp;
Second processing module, stabs for the data block recovery time corresponding according to each CQFS data block the business datum removed successively in each data block; Wherein, processing module often removes the business datum in a data block, continues the business datum of the up-to-date generation of write in the CQFS data block of processing module after removing business datum.
By the method that the embodiment of the present invention provides, this data storage and management method is based on round-robin queue, when CQFS file system is run, write is there will not be to put problem that is discrete or disk fragments, therefore, the long-time running of CQFS file system, write performance can not be decayed, also the problem easily can not lost because of disk fragments and then generation file, further ensures the stability of disk.
In conjunction with second aspect, in the implementation that the first is possible, configuration module creates at least one CQFS file system, comprising:
If only there is a disk in system, then to creating a CQFS file system by a disk; Or,
If there are at least two disks in system, then each disk corresponding creates a CQFS file system respectively, the write of the business datum in each CQFS file system and remove separate; Or,
If there is at least one disk in system, and there are at least two file directorys in this at least one disk, then each file directory corresponding creates a CQFS file system respectively, the write of the business datum in each CQFS file system and remove separate.
By this possible implementation, between the business datum that each CQFS data block in same CQFS file system stores, height is from complete, does not namely all have data dependence between any two CQFS data blocks.The business datum stored between different CQFS file system yet height, from complete, does not namely all have data dependence between any two CQFS file system.
In conjunction with any one implementation above-mentioned of second aspect, in the implementation that the second is possible, configuration module creates a CQFS file system, comprising:
Create first area, second area, the 3rd region and the 4th region, wherein,
First area, for storing CQFS header, this CQFS header is used to indicate the configuration information of CQFS file system;
Second area, for storing the list of CQFS data block information; This CQFS data block information list, for recording the descriptor of each CQFS data block, at least comprises the data block recovery time stamp of corresponding CQFS data block in this descriptor;
3rd region, is divided into some CQFS data blocks for storage service data, at least comprises the index information of CQFS data block and the descriptor of corresponding business datum in described CQFS data block;
4th region, for storing the Backup Data of first area and second area;
In conjunction with the implementation that the second of second aspect is possible, in the implementation that the third is possible, configuration module creates a CQFS file system, comprises further:
Create the 5th region, the 5th region is for storing the distribution list of CQFS data block;
And the 4th region is also further for storing the Backup Data in the 5th region.
In conjunction with the second or the third possible implementation of second aspect, in the 4th kind of possible implementation, the first processing module, in any one CQFS data block write business datum, comprising:
Record traffic data in this any one CQFS data block;
After business datum record, processing module records corresponding index information in this any one CQFS data block.
In conjunction with the second or the third possible implementation of second aspect, in the 5th kind of possible implementation, detection module, after system reboot, comprises further:
The consistance of the descriptor of each CQFS data block of comparison and the descriptor of corresponding business datum;
When determining that the descriptor of the descriptor of any one CQFS data block and corresponding business datum is inconsistent, the descriptor of this any one CQFS data block is reverted to the descriptor of corresponding business datum.
By this possible implementation, after system reboot, the data of loss can be easy to be detected, and system can recover data in time, solve the problem that file is easily lost.
In conjunction with the 5th kind of possible implementation of second aspect, in the 6th kind of possible implementation, detection module, after system reboot, comprises further:
Check data block recovery time stamp, obtain the index information that data block recovery time stabs the business datum in a minimum CQFS data block, judge that this index information is as time empty, produces warning message.
By this possible implementation, after system reboot, can Timeliness coverage data loss problem.
Accompanying drawing explanation
Fig. 1 is data storage and management method process flow diagram;
Fig. 2 is CQFS data block assigning process schematic diagram;
Fig. 3 is data storage management device process flow diagram.
Embodiment
After solving the long-time running of prior art file system, data space is constantly tending towards discrete, causes disk fragments, and then the problem that generation file is easily lost.In the embodiment of the present invention, round-robin queue's file system (Circular Queue File System, CQFS) is adopted to carry out storage administration to the file in system.
Below in conjunction with accompanying drawing, the preferred embodiment of the present invention is described in detail.
Consult shown in Fig. 1, in the embodiment of the present invention, the idiographic flow that management devices carries out data storage management is as follows:
Step 100: in the system configuration stage, management devices creates at least one CQFS file system, in this at least one CQFS file system, comprises the CQFS data block of preset number and default size.
In the present embodiment, data administrator can be user terminal, and e.g., PC, notebook etc., can be also server, this is no longer going to repeat them.
Step 110: in the system cloud gray model stage, the business datum of generation writes in each the CQFS data block at least one CQFS file system above-mentioned by management devices one by one, wherein, often use a CQFS data block, the corresponding CQFS data block used generates corresponding data block recovery time stamp.
Step 120: management devices stabs according to the data block recovery time that each CQFS data block is corresponding the business datum removed successively in each data block; Wherein, often remove the business datum in a data block, in the CQFS data block after removing business datum, continue the business datum of the up-to-date generation of write.
Based on above-described embodiment, specifically, in step 100, in the system configuration stage, management devices, when creating at least one CQFS file system, can adopt but be not limited to following several mode:
First kind of way is: if only there is a disk in system, then management devices is to creating a CQFS file system by a disk;
The second way is: if there are at least two disks in system, then management devices each disk corresponding creates a CQFS file system respectively, and the write of the business datum in each CQFS file system and removing are independent mutually;
The third mode is: if there is at least one disk in system, and there are at least two file directorys in this at least one disk, then management devices each file directory corresponding creates a CQFS file system respectively, and write and the removing of the business datum in each CQFS file system are all separate.
Be introduced to create a CQFS file system below.When creating a CQFS file system, management devices creates the CQFS data block of a series of preset number and default size on a storage medium continuously, and each CQFS data block is joined in a round-robin queue.
Table 1A
Such as, consult shown in table 1A, management devices, when a creation CQFS file system, needs to create first area, second area, the 3rd region, the 4th region, and wherein, the position in the 5th region is reserved, but does not store data;
Table 1B
Further, consult shown in table 1B, the 5th region can also be created, and at the 5th area stores CQFS data block distribution list (Block Allocation Table), all store data instance with five regions below and describe in detail:
First area, is called CQFS head (CQFS Header), for storing CQFS header;
CQFS header is used to indicate the configuration information of CQFS file system, the configuration information of CQFS file system can be CQFS version information, the general unique identifier of disk (Universally UniqueIdentifier, UUID), CQFS number of data blocks and CQFS data block size information etc.
Second area, is called CQFS data block information list (Block Info Table), for storing the descriptor of CQFS data block;
Table 2
| Block Info | Block Info | ... | Block Info |
The list of CQFS data block information comprises several fields, and each field is for recording the descriptor of a CQFS data block.
Consult shown in table 2, the list of CQFS data block information comprises some fields, is called Block Info, and each field is for storing the descriptor of a CQFS data block;
Table 3A
Consult shown in table 3A, comprise seven fields at a Block Info, be used for the data block ID(Block ID of the corresponding CQFS data block of sequential storage), data stream ID(Stream ID), data block recovery time stamp (Recycle TS), data start time stamp (Begin TS), ED timestamp (End TS) and data block lock token (Is Locked), service scripts beginning label (Is FileBegin);
Table 3B
Further, consult shown in table 3B, can also comprise the 8th field in a Block Info, the 8th field is used for storing the data block abnormal marking (Is BadBlock) of corresponding CQFS data block; Describe in detail for eight fields below:
Wherein, data block ID is used for the absolute offset values of calculation document start memory location.
Data stream ID is used to indicate data flow path corresponding to the business datum of write, such as, in safety defense monitoring system, which Video data what data stream ID stored in order to instruction is, namely data stream ID can indicate the Video data of storage corresponding be which recording channel.
Data block recovery time stamp is used to indicate the data record time, it is the foundation as round-robin queue's structure, such as, have 10 CQFS data blocks in the round-robin queue of a CQFS file system, starting stage, when writing data one by one in these 10 CQFS data blocks, these 10 CQFS data blocks are corresponding in turn to the data block recovery time stamp generated from 0 to 9, wherein, the recovery time stamp that the CQFS data block of round-robin queue's head is corresponding is 9, the recovery time stamp that the CQFS data block of round-robin queue's afterbody is corresponding is 0, namely the recovery time stamp that CQFS data block is corresponding is larger, data in this CQFS data block are newer, CQFS data block corresponding recovery time stamp less, the data in this CQFS data block are older.When disk once reclaims, management devices selects the oldest data in these 10 CQFS data blocks to remove, namely select corresponding recovery time stamp be 0 CQFS data block, remove data wherein and write new data, after completing, this CQFS data block is joined the head of round-robin queue, simultaneously by its recovery time stamp last time statistics CQFS reclaim queue tail recovery time stamp 9 basis on add 1, save as 10, namely this is this CQFS recovery queue tail recovery time stamp.By that analogy, disk often once reclaims, and CQFS reclaims queue tail recovery time stamp just increases by 1.Namely in the system cloud gray model stage, when starting to perform business datum storage operation, all the time obtain data block recovery time from the afterbody of round-robin queue and stab minimum CQFS data block write business datum, after the action of write business datum completes, the CQFS data block storing business datum is joined the head of round-robin queue.
Data start time stamp is used to indicate data and stores the start time.Such as, in safety defense monitoring system, data start time stamp can when playing back videos be located, for location playback from which CQFS data block.
ED timestamp is used to indicate data and stores the end time.Such as, in safety defense monitoring system, ED timestamp can when playing back videos be located, for location playback from which CQFS data block.Such as, in safety defense monitoring system, data start time stamp and ED timestamp are jointly for locating playback from which CQFS blocks of files.Concrete, user is when knowing Video data start time and Video data end time, want to obtain this section of Video data, need the ED timestamp comparison utilizing Video data start time and Video data end time to stab with the data start time stored and store respectively, when request time window has overlapping with the time window of certain CQFS data block, just can obtain this CQFS data block and start playback.Wherein, the Video data start time is called request time window to the Video data end time, and data block start time of CQFS data block stabs the time window that end of data block timestamp is called CQFS data block.
Whether data block lock token is used to indicate data block and is locked, and when data block is locked, this data block can not be recovered, and the business datum stored in the CQFS data block that namely this data block lock token is corresponding can not be recovered.
Service scripts beginning label is used for identification service file and stores the start time.Such as, in safety defense monitoring system, business datum when camera runs is with the mode Coutinuous store of data stream, continuously data stream is divided into the multiple service scriptss corresponding with each time period, the corresponding service scripts beginning label of each service scripts, the concept of supplementary service file is because service scripts is convenient to identification and the operation of user more relative to data block.
Whether data block abnormal marking is used to indicate the storage medium region that this data block stores abnormal, and namely whether this data block can normally be read and write.
3rd region, is called data field (Data Area), for storage service data.
Table 4
| Block Data | Block Data | ... | Block Data |
Data field is divided into some CQFS data blocks for storage service data.
Table 5
Consult shown in table 5, in each CQFS data block, comprise three fields, be used for business datum (Dahua Frame Data Area) that sequential storage produces, the index information (DahuaIndex) of corresponding service data and the descriptor (Description) of corresponding service data.Index information amount can be estimated according to code stream, such as, size is the CQFS data block of 16MB, when stream rate is between 512Kbps to 8Mbps, the scope of index data amount is between 12 bytes to 2048 bytes, therefore, management devices can reserve a block size is that the field of 64KB is as the region storing index information.
4th region, is called backup area (Backup Area), and backup area is for storing first area, second area backup information, further, to in the 5th area stores CQFS data block distribution list, then backup area is also for storing the 5th region backup information, and regular update.
Table 6A
Consult shown in table 6A, backup area comprises three fields, first character section is empty, and other two fields are used for storing CQFS data block information list Backup Data (Block Info Table backup) and CQFS header Backup Data (CQFS Header backup) successively.
Further, consult shown in table 1B, can also in the 5th area stores CQFS data block distribution list, for storing CQFS data block assignment information.
CQFS data block assignment information is used to indicate the corresponding relation of the round-robin queue that user and CQFS data block are formed.Concrete, if only there is a disk in system, then management devices creates a CQFS file system to this disk.This disk can only have a user to use, and many users also can be had to use; If when this disk only has a user to use, then this disk only exists a round-robin queue, the distribution list of CQFS data block is used to indicate this round-robin queue and distributes to this user; If when this disk has many users to use, then there is many round-robin queues in this disk, wherein, the allocation scheme that round-robin queue is distributed to user by management devices can be that a round-robin queue distributes to a user, also can be that a round-robin queue distributes to multiple user, the distribution list of CQFS data block be used to indicate each round-robin queue and distributes to which user's use.
When using CQFS file system to carry out data storage management, between the business datum that each CQFS data block in same CQFS file system stores, height is from complete, does not namely all have data dependence between any two CQFS data blocks.The business datum stored between different CQFS file system yet height, from complete, does not namely all have data dependence between any two CQFS file system.
Table 6B
Further, consult shown in table 6B, if in the 5th area stores CQFS data block distribution list, then three fields that backup area comprises are used for storing CQFS data block distribution list Backup Data, CQFS data block information list Backup Data and CQFS header Backup Data successively.
There is Backup Data at backup area in CQFS header and the list of CQFS data block information, and in the 5th area stores CQFS data block distribution list, then the 5th region is also backed up at backup area.Therefore, in system exception situation, Backup Data according to disk storage can easily rebuild CQFS file system, namely on the disk of application CQFS file system, when there is the situation of access fault in whichever CQFS data block, all can not cause the loss of data of whole CQFS file system, ensure that the security of data.
Further, in step 110, in the system cloud gray model stage, management devices writes business datum on a storage medium, namely business datum writes in each the CQFS data block at least one CQFS file system by management devices one by one, as, obtain data block recovery time and stab minimum CQFS data block write business datum, namely a CQFS data block write business datum is obtained from the afterbody of round-robin queue, if this CQFS data block is empty, then to write direct business datum, if there is business datum in this CQFS data block, only can remove the business datum in this CQFS data block, also the business datum in this CQFS data block is removed after can removing the index information recorded in this CQFS data block again, the business datum of the up-to-date generation of write is continued in CQFS data block after removing business datum, then data block recovery time stamp and the index information of corresponding service data is generated.After above-mentioned business datum ablation process terminates, this CQFS data block is joined the head of round-robin queue.During each business datum of write on a storage medium, all perform above-mentioned same operation.By that analogy, can ensure that the CQFS data block that each management devices reads from round-robin queue's afterbody can be all the storage time of CQFS data block at most, in this storage time of the new business datum produced of CQFS data block write at most, again the CQFS data block after write business datum is added the head of round-robin queue, operation like this, the CQFS data block in CQFS file system can reclaim in the mode of round-robin queue and write.
Concrete, the business datum of write CQFS data block will be stored in the first character section shown in table 5, after business datum write activity completes, the descriptor of the index information of second field write corresponding service data shown in table 5 and the 3rd field write corresponding service data shown in table 5, represents the release of above-mentioned CQFS data block write business datum; Meanwhile, management devices seven fields shown in table 3A can generate data block ID, the data stream ID of corresponding CQFS data block, data block recovery time stamp, data start time stamp, ED timestamp, data block lock token, service scripts beginning label and generates corresponding Backup Data showing the backup area shown in 1B.
Therefore, during application CQFS file system management storage file, business datum in CQFS data block adopts the mode of round-robin queue to store, the concurrent write point of business datum controls within limits all the time, concrete, as, be applied in safety defense monitoring system, the business datum of multi-path camera is had all to need to be stored in a disk, when multi-path camera while during storage service data, management devices can distribute a storage area for each camera, there is the write point of a business datum in each storage area, multiple concurrent write point can be there is in this disk, management devices is by regularly redistributing the position of each camera storage area, make the write point of each camera on disk more concentrated, thus disk address-time can be reduced.Learn thus, management devices is when multi-path camera write business datum, and address-time is short, write is there will not be to put problem that is discrete or disk fragments, therefore, when the long-time running of CQFS file system, write performance can not be decayed, and further ensures the stability of disk.
On the other hand, after system reboot, management devices can do data in magnetic disk consistency check according to the default configuration of CQFS file system, namely check that whether the descriptor (i.e. Block Info) of same CQFS data block is consistent with the descriptor (i.e. Description) of the business datum recorded in this CQFS data block, if inconsistent, then the information that the descriptor descriptor of this CQFS data block being reverted to the corresponding business datum of this CQFS data block comprises.Now, management devices can recover the data of mistake automatically, does not produce warning message.
Further, after system reboot, management devices also can check data block recovery time stamp according to the default configuration of CQFS file system, obtain the index information of the business datum in a CQFS data block minimum with the mistiming of current time, because after each write service data manipulating, capital generates the index information of corresponding service data, so, if when finding that index information is empty, illustrate before system reboot, the operation of write business datum does not also complete, may be then that fault has appearred in system, business datum in corresponding CQFS data block may be lost, now, management devices produces warning message.
When creating multiple CQFS file system in system, recovery and write that management devices is adopting recycle design in each CQFS data block corresponding, carry out data respectively in each CQFS file system, suppose that above-mentioned multiple CQFS file system is applied in safety defense monitoring system, then management devices realizes between multiple CQFS file system by regularly redistributing recording channel load balancing.As, consult shown in Fig. 2, in CQFS data block assigning process, to choose physical disk D dish, create a CQFS file system, if the CQFS number of data blocks preset is three, then the data field (i.e. DataArea) that D dish distributes is divided into three CQFS data blocks, above-mentioned three CQFS data blocks are joined in a CQFS data block round-robin queue, in the CQFS file system starting stage, the data block ID of first CQFS data block is set to 0, and corresponding data block recovery time stamp is set to 0, and corresponding data stream ID is set to 0.The data block ID of second CQFS data block is set to 1, and corresponding data block recovery time stamp is set to 1, and corresponding data stream ID is set to 0.The data block ID of the 3rd CQFS data block is set to 2, and corresponding data block recovery time stamp is for being set to 2, and corresponding data stream ID is set to 0.When first via business datum starts to write disk, obtain data block recovery time and stab minimum CQFS data block, namely data block recovery time stamp is the CQFS data block of 0, because of the CQFS data block that this CQFS data block is first write business datum, be the storage time of CQFS data block at most, therefore, obtain this CQFS data block write business datum, write is after service data manipulating completes, and upgrading data block recovery time stamp corresponding to this CQFS data block is 3 and to record data stream ID corresponding to this CQFS data block be 1; When the second tunnel business datum starts to write disk, obtain data block recovery time stamp minimum CQFS data block, namely data block recovery time stamp is the CQFS data block of 1, now this CQFS data block is the storage time of CQFS data block at most, therefore, obtain this CQFS data block write business datum, write is after service data manipulating completes, and upgrading data block recovery time stamp corresponding to this CQFS data block is 4 and to record data stream ID corresponding to this CQFS data block be 2; By that analogy, each acquisition storage time, CQFS data block at most, after write operation completes, upgraded data block recovery time stamp corresponding to this CQFS data block and record data stream ID.
Consult shown in Fig. 3, in the embodiment of the present invention, data storage management device comprises configuration module 300, first processing module 310 and the second processing module 320, and further data storage management device also comprises detection module 330, wherein,
Configuration module 300, in the system configuration stage, creates at least one CQFS file system, in this at least one CQFS file system, comprises the CQFS data block of preset number and default size.
First processing module 310 is in the system cloud gray model stage, the business datum of generation is write one by one in each the CQFS data block at least one CQFS file system above-mentioned, wherein, often use a CQFS data block, the corresponding CQFS data block used generates corresponding data block recovery time stamp.
Second processing module 320 stabs according to the data block recovery time that each CQFS data block is corresponding the business datum removed successively in each data block; Wherein, often remove the business datum in a data block, in the CQFS data block after removing business datum, continue the business datum of the up-to-date generation of write.
Configuration module 300 creates at least one CQFS file system, comprising:
If only there is a disk in system, then to creating a CQFS file system by a disk; Or,
If there are at least two disks in system, then each disk corresponding creates a CQFS file system respectively, the write of the business datum in each CQFS file system and remove separate; Or,
If there is at least one disk in system, and there are at least two file directorys in this at least one disk, then each file directory corresponding creates a CQFS file system respectively, the write of the business datum in each CQFS file system and remove separate.
Configuration module 300 creates a CQFS file system, also comprises:
Create first area, second area, the 3rd region and the 4th region, wherein,
First area, for storing CQFS header, this CQFS header is used to indicate the configuration information of CQFS file system;
Second area, for storing the list of CQFS data block information; This CQFS data block information list, for recording the descriptor of each CQFS data block, at least comprises the data block recovery time stamp of corresponding CQFS data block in this descriptor;
3rd region, is divided into some CQFS data blocks for storage service data, at least comprises the index information of CQFS data block and the descriptor of corresponding business datum in described CQFS data block;
4th region, for storing the Backup Data of first area and second area.
Configuration module 300 creates a CQFS file system, comprises further:
Create the 5th region, the 5th region is for storing the distribution list of CQFS data block;
And the 4th region is also further for storing the Backup Data in the 5th region.
First processing module 310, in any one CQFS data block write business datum, comprising:
Record traffic data in this any one CQFS data block;
After business datum record, in this any one CQFS data block, record corresponding index information.
Further, data storage management device also comprises detection module 330,
Detection module 330 is for after system reboot:
The consistance of the descriptor of each CQFS data block of comparison and the descriptor of corresponding business datum; When determining that the descriptor of the descriptor of any one CQFS data block and corresponding business datum is inconsistent, the descriptor of this any one CQFS data block is reverted to the descriptor of corresponding business datum.
Or detection module 330 is for after system reboot:
Check data block recovery time stamp, obtain the index information that data block recovery time stabs the business datum in a minimum CQFS data block, judge that this index information is as time empty, produces warning message.
Therefore, when the CQFS file system that the application embodiment of the present invention proposes carries out storage administration to the file in system, the concurrent write point of its business datum can control within the specific limits all the time, the present embodiment is based on round-robin queue, when CQFS file system is run, write is there will not be to put problem that is discrete or disk fragments, therefore, the long-time running of CQFS file system, write performance can not be decayed, also the problem easily can not lost because of disk fragments and then generation file, further ensures the stability of disk.
Obviously, those skilled in the art can carry out various change and modification to the embodiment of the present invention and not depart from the spirit and scope of the embodiment of the present invention.Like this, if these amendments of the embodiment of the present invention and modification belong within the scope of the claims in the present invention and equivalent technologies thereof, then the present invention is also intended to comprise these change and modification.
Claims (14)
1. a data storage and management method, is characterized in that, described method comprises:
In the system configuration stage, create at least one round-robin queue file system CQFS file system, at least one CQFS file system described, comprise the CQFS data block of preset number and default size;
In the system cloud gray model stage, the business datum of generation is write one by one in each the CQFS data block at least one CQFS file system described, wherein, often use a CQFS data block, the corresponding CQFS data block used generates corresponding data block recovery time stamp;
The data block recovery time corresponding according to each CQFS data block stabs the business datum removed successively in each data block; Wherein, often remove the business datum in a data block, in the CQFS data block after removing business datum, continue the business datum of the up-to-date generation of write.
2. method according to claim 1, is characterized in that, creates at least one CQFS file system, comprising:
If only there is a disk in system, then a corresponding described disk creates a CQFS file system; Or,
If there are at least two disks in system, then each disk corresponding creates a CQFS file system respectively, the write of the business datum in each CQFS file system and remove separate; Or,
If there is at least one disk in system, and at least one disk described, there are at least two file directorys, then each file directory corresponding creates a CQFS file system respectively, the write of the business datum in each CQFS file system and remove separate.
3. method as claimed in claim 1 or 2, it is characterized in that, described establishment CQFS file system, comprising:
Create first area, second area, the 3rd region and the 4th region, wherein,
First area, for storing CQFS header, described CQFS header is used to indicate the configuration information of CQFS file system;
Second area, for storing the list of CQFS data block information; The list of described CQFS data block information, for recording the descriptor of each CQFS data block, at least comprises the data block recovery time stamp of corresponding CQFS data block in described descriptor;
3rd region, is divided into some CQFS data blocks for storage service data, at least comprises the index information of CQFS data block and the descriptor of corresponding business datum in described CQFS data block;
4th region, for storing the Backup Data of first area and second area.
4. method as claimed in claim 3, it is characterized in that, described establishment CQFS file system, comprises further:
Create the 5th region, described 5th region is for storing the distribution list of CQFS data block;
And the 4th region is also further for storing the Backup Data in the 5th region.
5. method as claimed in claim 3, is characterized in that, in any one CQFS data block write business datum, comprising:
Record traffic data in any one CQFS data block described;
After business datum record, in any one CQFS data block described, record corresponding index information.
6. method as claimed in claim 3, is characterized in that, after system reboot, comprise further:
The consistance of the descriptor of each CQFS data block of comparison and the descriptor of corresponding business datum;
When determining that the descriptor of the descriptor of any one CQFS data block and corresponding business datum is inconsistent, the descriptor of any one CQFS data block described is reverted to the descriptor of corresponding business datum.
7. method as claimed in claim 6, is characterized in that, after system reboot, comprise further:
Check data block recovery time stamp, obtain the index information that data block recovery time stabs the business datum in a minimum CQFS data block, judge that described index information is as time empty, produces warning message.
8. a data storage management device, is characterized in that, described device comprises:
Configuration module, in the system configuration stage, creates at least one round-robin queue file system CQFS file system, at least one CQFS file system described, comprises the CQFS data block of preset number and default size;
First processing module, for in the system cloud gray model stage, the business datum of generation is write one by one in each the CQFS data block at least one CQFS file system described, wherein, often use a CQFS data block, the corresponding CQFS data block used generates corresponding data block recovery time stamp;
Second processing module, stabs for the data block recovery time corresponding according to each CQFS data block the business datum removed successively in each data block; Wherein, often remove the business datum in a data block, in the CQFS data block after removing business datum, continue the business datum of the up-to-date generation of write.
9. device according to claim 8, is characterized in that, described configuration module creates at least one CQFS file system, comprising:
If only there is a disk in system, then a corresponding described disk creates a CQFS file system; Or,
If there are at least two disks in system, then each disk corresponding creates a CQFS file system respectively, the write of the business datum in each CQFS file system and remove separate; Or,
If there is at least one disk in system, and at least one disk described, there are at least two file directorys, then each file directory corresponding creates a CQFS file system respectively, the write of the business datum in each CQFS file system and remove separate.
10. device as claimed in claim 8 or 9, it is characterized in that, described configuration module creates a CQFS file system, comprising:
Create first area, second area, the 3rd region and the 4th region, wherein,
First area, for storing CQFS header, described CQFS header is used to indicate the configuration information of CQFS file system;
Second area, for storing the list of CQFS data block information; The list of described CQFS data block information, for recording the descriptor of each CQFS data block, at least comprises the data block recovery time stamp of corresponding CQFS data block in described descriptor;
3rd region, is divided into some CQFS data blocks for storage service data, at least comprises the index information of CQFS data block and the descriptor of corresponding business datum in described CQFS data block;
4th region, for storing the Backup Data of first area and second area.
11. devices as claimed in claim 10, it is characterized in that, described configuration module creates a CQFS file system, comprises further:
Create the 5th region, described 5th region is for storing the distribution list of CQFS data block;
And the 4th region is also further for storing the Backup Data in the 5th region.
12. devices as claimed in claim 10, is characterized in that, described first processing module, in any one CQFS data block write business datum, comprising:
Record traffic data in any one CQFS data block described;
After business datum record, processing module records corresponding index information in any one CQFS data block described.
13. devices as claimed in claim 10, is characterized in that, comprise further:
Detection module, for after system reboot, to the consistance of the descriptor of each CQFS data block and the descriptor of corresponding business datum;
When determining that the descriptor of the descriptor of any one CQFS data block and corresponding business datum is inconsistent, the descriptor of any one CQFS data block described is reverted to the descriptor of corresponding business datum.
14. devices as claimed in claim 13, is characterized in that, comprise further:
Detection module, for after system reboot, looks into data block recovery time stamp, obtains the index information that data block recovery time stabs the business datum in a minimum CQFS data block, judges that described index information is as time empty, produces warning message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310442311.9A CN104462148B (en) | 2013-09-24 | 2013-09-24 | A kind of data storage and management method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310442311.9A CN104462148B (en) | 2013-09-24 | 2013-09-24 | A kind of data storage and management method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104462148A true CN104462148A (en) | 2015-03-25 |
| CN104462148B CN104462148B (en) | 2018-02-16 |
Family
ID=52908203
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310442311.9A Active CN104462148B (en) | 2013-09-24 | 2013-09-24 | A kind of data storage and management method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104462148B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105100716A (en) * | 2015-07-23 | 2015-11-25 | 浙江立元通信技术股份有限公司 | Safe memory cell used for network video monitoring and system thereof |
| CN110647294A (en) * | 2019-09-09 | 2020-01-03 | Oppo（重庆）智能科技有限公司 | Storage block recovery method and device, storage medium and electronic equipment |
| CN111538613A (en) * | 2020-04-28 | 2020-08-14 | 浙江大华技术股份有限公司 | Cluster system exception recovery processing method and device |
| CN112306824A (en) * | 2019-07-31 | 2021-02-02 | 上海哔哩哔哩科技有限公司 | Disk performance evaluation method, system, device and computer readable storage medium |
| CN114385075A (en) * | 2021-12-01 | 2022-04-22 | 浙江大华技术股份有限公司 | Storage method and query method of characteristic value, electronic device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020174102A1 (en) * | 2001-04-27 | 2002-11-21 | Kyler Daniel B. | Filter driver for identifying disk files by analysis of content |
| CN101090345A (en) * | 2007-07-20 | 2007-12-19 | 哈尔滨工程大学 | Performance test method for network storage system |
| US20110167167A1 (en) * | 2010-01-05 | 2011-07-07 | Disney Enterprises, Inc. | Method and system for providing real-time streaming media content |
| CN102789458A (en) * | 2011-05-18 | 2012-11-21 | 腾讯科技（深圳）有限公司 | Image file processing method and system |
-
2013
- 2013-09-24 CN CN201310442311.9A patent/CN104462148B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020174102A1 (en) * | 2001-04-27 | 2002-11-21 | Kyler Daniel B. | Filter driver for identifying disk files by analysis of content |
| CN101090345A (en) * | 2007-07-20 | 2007-12-19 | 哈尔滨工程大学 | Performance test method for network storage system |
| US20110167167A1 (en) * | 2010-01-05 | 2011-07-07 | Disney Enterprises, Inc. | Method and system for providing real-time streaming media content |
| CN102789458A (en) * | 2011-05-18 | 2012-11-21 | 腾讯科技（深圳）有限公司 | Image file processing method and system |
Non-Patent Citations (1)
| Title |
|---|
| 张游杰: "一种有效保护数据的文件***", 《现代电子技术》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105100716A (en) * | 2015-07-23 | 2015-11-25 | 浙江立元通信技术股份有限公司 | Safe memory cell used for network video monitoring and system thereof |
| CN105100716B (en) * | 2015-07-23 | 2018-09-18 | 浙江立元通信技术股份有限公司 | A kind of secure storage unit and its system for Network Video Surveillance |
| CN112306824A (en) * | 2019-07-31 | 2021-02-02 | 上海哔哩哔哩科技有限公司 | Disk performance evaluation method, system, device and computer readable storage medium |
| CN112306824B (en) * | 2019-07-31 | 2022-07-26 | 上海哔哩哔哩科技有限公司 | Disk performance evaluation method, system, device and computer readable storage medium |
| CN110647294A (en) * | 2019-09-09 | 2020-01-03 | Oppo（重庆）智能科技有限公司 | Storage block recovery method and device, storage medium and electronic equipment |
| CN110647294B (en) * | 2019-09-09 | 2022-03-25 | Oppo广东移动通信有限公司 | Storage block recovery method and device, storage medium and electronic equipment |
| CN111538613A (en) * | 2020-04-28 | 2020-08-14 | 浙江大华技术股份有限公司 | Cluster system exception recovery processing method and device |
| CN111538613B (en) * | 2020-04-28 | 2023-06-13 | 浙江大华技术股份有限公司 | Cluster system exception recovery processing method and device |
| CN114385075A (en) * | 2021-12-01 | 2022-04-22 | 浙江大华技术股份有限公司 | Storage method and query method of characteristic value, electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104462148B (en) | 2018-02-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8924664B2 (en) | Logical object deletion | |
| CN103064639B (en) | Date storage method and device | |
| CN106547641B (en) | CDP backup method based on volume | |
| CN101876885B (en) | A kind of method and apparatus of assignment logic drive | |
| CN104636266B (en) | Cover tile magnetic recording hard disk, cover tile magnetic recording hard disk write method and the device of data | |
| CN104461390A (en) | Method and device for writing data into imbricate magnetic recording SMR hard disk | |
| CN104462148A (en) | Data storage management method and device | |
| CN103226965B (en) | Based on the audio/video data access method of time bitmap | |
| CN102667703A (en) | System and method for optimized reclamation processing in a virtual tape library system | |
| CN108268344B (en) | Data processing method and device | |
| JP2017079053A (en) | Methods and systems for improving storage journaling | |
| CN109582213B (en) | Data reconstruction method and device and data storage system | |
| CN102314383A (en) | Failure recovery method and device for data index | |
| CN108399050B (en) | Data processing method and device | |
| CN103761190A (en) | Data processing method and apparatus | |
| WO2016029743A1 (en) | Method and device for generating logical disk of virtual machine | |
| CN110597762A (en) | File processing method, device, equipment and storage medium | |
| CN112115002B (en) | Method and device for recovering file from damaged or untrusted mechanical hard disk | |
| CN106709014B (en) | File system conversion method and device | |
| US9740571B1 (en) | Intelligent continuous data protection snapshot based backups | |
| KR102277731B1 (en) | Method for operating storage system and storage controller | |
| CN105573862B (en) | Method and equipment for recovering file system | |
| CN113886352B (en) | Metadata recovery method, device, equipment and medium of distributed file system | |
| CN107545022B (en) | Disk management method and device | |
| KR101506330B1 (en) | Apparatus for managing data stability, method for storing data and method for recovering data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |