CN106294603A - File memory method and device - Google Patents
File memory method and device Download PDFInfo
- Publication number
- CN106294603A CN106294603A CN201610616789.2A CN201610616789A CN106294603A CN 106294603 A CN106294603 A CN 106294603A CN 201610616789 A CN201610616789 A CN 201610616789A CN 106294603 A CN106294603 A CN 106294603A
- Authority
- CN
- China
- Prior art keywords
- memory space
- small documents
- unit memory
- file
- space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a kind of file memory method and device, be applied to store the storage engines of multiple small documents, wherein, described small documents refers to that memory space shared by file is less than file threshold value.Described method includes: utilizing described storage engines to divide memory space, predistribution goes out unit memory space；The index node of correspondence is set for small documents to be stored in described unit memory space in described storage engines；When described storage engines receives the data write request of new small documents, keep described index node constant, described new small documents is write described unit memory space and preserves.This technical scheme not only reduces the fragment of disk space and produces, and without the content revising in index node when making to write data in memory space, thus the generation that the system that decreases is called, optimize the utilization rate to disk I/O.
Description
Technical field
The present invention relates to file processing technology field, particularly relate to a kind of file memory method and device.
Background technology
After mass small documents storage occurs, the most always a great problem of field of data storage.Based on LSM tree structure
Leveldb data base be relatively more suitable on line frequently scene to the write of certain some keyword, and our small documents storage
The file data aiming at storage user of engine definitions, and the feature of file data is data after write, seldom deposits
After the situation of amendment, such as one pictures storage, or delete, without there is the situation of content inside amendment picture.
Owing to LSM tree has the concept of level, a data can be write frequently and be merged, then based on leveldb data
Storehouse such LSM tree structure exists for the problem that write is amplified.
In correlation technique, storage engines is to distribute a file when in use, and then operating system can give correspondence
File distribution index node inode information, this inode information is stored in inside internal memory.For example, common
Below xfs file system, the size of this index node inode is 536byte, and common file size is 40kb, our line
On use the storage type of 32G memory size, 88T disk space, then under this storage type, 32G/536byte is
Much smaller than 88T/40kb's, say, that we cannot be stored in so many file.Visible, how storing mass small documents becomes
The big problem that field of data storage is urgently to be resolved hurrily.
Summary of the invention
In view of the above problems, it is proposed that the present invention in case provide one overcome the problems referred to above or at least in part solve on
State file memory method and the device of problem.
According to one aspect of the present invention, it is provided that a kind of file memory method, it is applied to store depositing of multiple small documents
Storage engine, wherein, described small documents refers to that memory space shared by file is less than file threshold value, and described method includes:
Utilizing described storage engines to divide memory space, predistribution goes out unit memory space；
The index node of correspondence is set for small documents to be stored in described unit memory space in described storage engines；
When described storage engines receives the data write request of new small documents, keep described index node constant,
Described new small documents is write described unit memory space and preserves.
Alternatively, described described new small documents write described unit memory space and preserves, including:
The Piece file mergence that will store in described new small documents and described unit memory space, generates and merges file；
When memory space shared by described merging file is less than or equal to described unit memory space, store in described unit
Space stores described merging file.
Alternatively, the described Piece file mergence that will store in described new small documents and described unit memory space, including:
Obtain each self-corresponding binary code of file stored in described new small documents and described unit memory space；
Binary code corresponding for described new small documents is corresponding with the file stored in described unit memory space
Binary code combines.
Alternatively, described method also includes:
When memory space shared by described merging file is more than described unit memory space, distribute new unit storage sky
Between；
Described new small documents is write described new unit memory space and preserves.
Alternatively, described described new small documents write described unit memory space and preserves, including:
Search the end position of the last write file in described unit memory space；
After described end position, it is sequentially written in described new small documents and preserves.
Alternatively, described utilizing described storage engines to divide memory space, predistribution goes out described unit storage sky
Between, including:
Taking up room of gross space capacity according to described memory headroom and each index node, determines described memory space
The number of the unit memory space that can divide down；
Gross space capacity according to described number and described memory space, determines that the space of described unit memory space is big
Little；
According to the space size of described unit memory space, predistribution goes out described unit memory space.
Alternatively, described method also includes:
Setting up the index information table of described unit memory space, described index information table includes in described unit memory space
The each small documents stored positional information in described unit memory space.
Alternatively, described method also includes:
When receiving the access request to described small documents, inquire about described small documents in institute according to described index information table
State the positional information in unit memory space；
According to small documents described in described location information access.
According to another aspect of the present invention, it is provided that a kind of file storage device, it is applied to store multiple small documents
Storage engines, wherein, described small documents refers to that memory space shared by file is less than file threshold value, and described device includes:
Forward allocator module, is suitable to utilize described storage engines to divide memory space, and predistribution goes out unit storage sky
Between；
Arranging module, it is right to be suitable to arrange for small documents to be stored in described unit memory space in described storage engines
The index node answered；
First memory module, is suitable to when described storage engines receives the data write request of new small documents, keeps
Described index node is constant, and described new small documents is write described unit memory space and preserved.
Alternatively, described first memory module includes:
Combining unit, is suitable to the Piece file mergence that will store in described new small documents and described unit memory space, raw
Become to merge file；
First memory element, is suitable to when memory space shared by described merging file is less than or equal to described unit memory space
Time, described unit memory space stores described merging file.
Alternatively, described combining unit, be suitable to obtain in described new small documents and described unit memory space and store
The each self-corresponding binary code of file；By in binary code corresponding for described new small documents and described unit memory space
The binary code that the file of storage is corresponding combines.
Alternatively, described device also includes:
Distribution module, is suitable to when memory space shared by described merging file is more than described unit memory space, and distribution is new
Unit memory space；
Second memory module, is suitable to described new small documents is write described new unit memory space and preserved.
Alternatively, described first memory module includes:
Search unit, be suitable to search the end position of the last write file in described unit memory space；
Second memory element, is suitable to be sequentially written in described new small documents after described end position and preserve.
Alternatively, described forward allocator module includes:
First determines unit, and be suitable to the gross space capacity according to described memory headroom and each index node takies sky
Between, determine the number of the unit memory space that can divide under described memory space；
Second determines unit, is suitable to the gross space capacity according to described number and described memory space, determines described unit
The space size of memory space；
Predistribution unit, is suitable to the space size according to described unit memory space, and predistribution goes out described unit storage sky
Between.
Alternatively, described device also includes:
Setting up module, be adapted to set up the index information table of described unit memory space, described index information table includes described
The each small documents stored in unit memory space positional information in described unit memory space.
Alternatively, described device also includes:
Enquiry module, is suitable to when receiving the access request to described small documents, inquires about according to described index information table
Described small documents positional information in described unit memory space；
Access modules, is suitable to according to small documents described in described location information access.
Use the technical scheme that the embodiment of the present invention provides, it is possible to utilize storage engines to allocate in advance out in memory space
Unit memory space, and the index node of correspondence is set for small documents to be stored in this unit memory space, thus decrease
The fragment of disk space produces, and, when receiving the data write request of small documents, keep the situation that index node is constant
Lower small documents write unit memory space and preserves so that without revising in index node when writing data in memory space
Content, thus the generation that the system that decreases is called, optimize the utilization rate to disk I/O, meanwhile, use this writing direct
The method of file can also reduce the problem that write is amplified, and is very suitable for storing the storage engines of large amount of small documents.
Further, in the embodiment of the present invention, by will small documents to be stored and unit memory space store
Piece file mergence, generates and merges file, and storage merges file in this unit memory space, it is achieved that closed by substantial amounts of small documents
And become a big file to carry out the purpose stored, such that it is able to efficiently reduce the size space shared by index node.
Described above is only the general introduction of technical solution of the present invention, in order to better understand the technological means of the present invention,
And can be practiced according to the content of description, and in order to allow above and other objects of the present invention, the feature and advantage can
Become apparent, below especially exemplified by the detailed description of the invention of the present invention.
According to below in conjunction with the accompanying drawing detailed description to the specific embodiment of the invention, those skilled in the art will be brighter
Above-mentioned and other purposes, advantage and the feature of the present invention.
Accompanying drawing explanation
By reading the detailed description of hereafter preferred implementation, various other advantage and benefit common for this area
Technical staff will be clear from understanding.Accompanying drawing is only used for illustrating the purpose of preferred implementation, and is not considered as the present invention
Restriction.And in whole accompanying drawing, it is denoted by the same reference numerals identical parts.In the accompanying drawings:
Fig. 1 is the indicative flowchart of a kind of file memory method；
Fig. 2 is the indicative flowchart of a kind of file memory method；
Fig. 3 is the indicative flowchart of a kind of file memory method；
Fig. 4 is the signal of predistribution unit memory space in a kind of file memory method
Property flow chart；
Fig. 5 is the schematic block diagram of a kind of file storage device；
Fig. 6 is the schematic block diagram of a kind of file storage device；
Fig. 7 is the schematic block diagram of a kind of file storage device.
Detailed description of the invention
It is more fully described the exemplary embodiment of the disclosure below with reference to accompanying drawings.Although accompanying drawing shows the disclosure
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure and should be by embodiments set forth here
Limited.On the contrary, it is provided that these embodiments are able to be best understood from the disclosure, and can be by the scope of the present disclosure
Complete conveys to those skilled in the art.
Fig. 1 is the indicative flowchart of a kind of file memory method.This document stores
Method is applied to store the storage engines of multiple small documents, and wherein, multiple small documents can be the small documents of magnanimity, and small documents refers to
Memory space shared by file is less than file threshold value (such as 1M).As it is shown in figure 1, the method can include step S101-in general manner
S103:
Step S101, utilizes storage engines to divide memory space, and predistribution goes out unit memory space.
Step S102, arranges the index joint of correspondence in storage engines for small documents to be stored in unit memory space
Point.In storage engines, the corresponding unique index node of each unit memory space.
Step S103, when storage engines receives the data write request of new small documents, keeps index node constant,
New small documents is write unit memory space and preserves.
Use the technical scheme that the embodiment of the present invention provides, it is possible to utilize storage engines to allocate in advance out in memory space
Unit memory space, and the index node of correspondence is set for small documents to be stored in this unit memory space, thus decrease
The fragment of disk space produces, and, when receiving the data write request of small documents, keep the situation that index node is constant
Lower small documents write unit memory space and preserves so that without revising in index node when writing data in memory space
Content, thus the generation that the system that decreases is called, optimize the utilization rate to disk I/O, meanwhile, use this writing direct
The method of file can also reduce the problem that write is amplified, and is very suitable for storing the storage engines of large amount of small documents.
In one embodiment, when performing step S103, the mode of new small documents write unit memory space is had two
Kind, based on the following two kinds of different writing modes, above-mentioned file memory method is illustrated respectively.
In one embodiment, new small documents uses and writes with the mode storing Piece file mergence.On as in figure 2 it is shown,
State file memory method and can perform as following steps S201-S207:
Step S201, utilizes storage engines to divide memory space, and predistribution goes out unit memory space.
Step S202, arranges the index joint of correspondence in storage engines for small documents to be stored in unit memory space
Point.
Step S203, when storage engines receives the data write request of new small documents, keeps index node constant,
The Piece file mergence that will store in new small documents and unit memory space, generates and merges file.
In this step, if unit memory space has not stored file, then directly new small documents write unit is deposited
Storage space.By as follows for the step of the Piece file mergence stored in new small documents and unit memory space: first, obtain new little
The each self-corresponding binary code of file stored in file and unit memory space；Secondly, by corresponding for new small documents two
The ary codes binary code corresponding with the file stored in unit memory space combines.
Step S204, it is judged that whether merge memory space shared by file less than or equal to unit memory space；When merging literary composition
When memory space shared by part is less than or equal to unit memory space, perform step S205；When merging, memory space shared by file is big
When unit memory space, perform step S206.
Step S205, in unit memory space, storage merges file.
Step S206, distributes new unit memory space in memory space.After distributing new unit memory space, need
New unit memory space is arranged new index node, i.e. in memory space, each unit memory space is corresponding each
Unique index node, in order to can accurately find required literary composition according to the corresponding relation between index node and unit memory space
Part.
Step S207, writes new unit memory space by new small documents and preserves.
In the present embodiment, by the Piece file mergence that will store in small documents to be stored and unit memory space, generate
Merge file, and deposit in this unit memory space when merging memory space shared by file less than or equal to unit memory space
Storage merges file, it is achieved that substantial amounts of small documents is merged into the purpose that a big file carries out storing, thus without to each
The small documents of individual write all arranges index node, effectively reduces the size space shared by index node.
In another embodiment, new small documents uses the mode being sequentially written in write in unit memory space, i.e. first
First search the end position of the last write file in unit memory space；Then it is sequentially written in new after this end position
Small documents and preserve.Certainly, if the insufficient space in unit memory space is to store new small documents, then still need to again divide
Join new unit memory space.Therefore, as it is shown on figure 3, above-mentioned file memory method can also carry out as following steps S301-
S307:
Step S301, utilizes storage engines to divide memory space, and predistribution goes out unit memory space.
Step S302, arranges the index joint of correspondence in storage engines for small documents to be stored in unit memory space
Point.
Step S303, when storage engines receives the data write request of new small documents, it is judged that unit memory space
In remaining space size whether more than or equal to the size of new small documents.If the remaining space in unit memory space is big
The little size more than or equal to new small documents, then perform step S304；If the remaining space size in unit memory space
Less than the size of new small documents, then perform step S306.
Step S304, keeps index node constant, searches the stop bits of the last write file in unit memory space
Put.
Step S305, is sequentially written in new small documents after the end position of the last time write file and preserves.
Step S306, distributes new unit memory space in memory space.Identical with a upper embodiment, distribute new
After unit memory space, need new unit memory space is arranged new index node, i.e. each unit in memory space
Memory space the most unique all corresponding index node, in order to can be according to the corresponding pass between index node and unit memory space
System accurately finds required file.
Step S307, writes new unit memory space by new small documents and preserves.
In the present embodiment, by new small documents is sequentially written in unit memory space so that during follow-up lookup small documents
Can accurately find out required file according to the order of write, and the program is without arranging the small documents that each is to be stored
Index node, thus decrease disk space fragment produce, and make in memory space write data time without revising rope
Draw the content in node, thus the generation that the system that decreases is called, optimize the utilization rate to disk I/O.
In any of the above-described embodiment, step S101 can perform as step S401-S403 as shown in Figure 4:
Step S401, according to the gross space capacity of memory headroom and taking up room of each index node, determines storage sky
The number of the unit memory space that can divide under between.
Step S402, according to the gross space capacity of this number and memory space, determines the space size of unit memory space.
Step S403, according to the space size of unit memory space, predistribution goes out unit memory space.
In the present embodiment, the gross space capacity of memory headroom is storage divided by the value taken up room of each index node
The number of the unit memory space that can divide under space；The gross space capacity of memory space i.e. stores for unit divided by the value of this number
The space size in space.For example, the gross space capacity of the i.e. disk space of memory space is 88T, the gross space of memory headroom
Capacity is 32G, each index node take up room as 536byte, then the unit memory space that can divide under memory space
Number is: 32G/536byte=32*1024*1024byte/536byte=62602, the therefore sky of unit memory space
Between size be: 88T/62602=88*1024*1024M/62602=1.82M.It is to say, it is pre-in memory space
The space size of the unit memory space distributed is 1.82M.
From the technical scheme of the present embodiment, by the gross space capacity of memory headroom and taking of each index node
SPATIAL CALCULATION goes out the number of the unit memory space that can divide under memory space, then total further according to this number and memory space
Spatial content calculates the space size of unit memory space so that memory space can be according to interior when storing multiple small documents
Deposit the number of open ended index node in space and carry out allocation unit memory space, so that any storage type can basis
Self configuration carrys out dividing unit memory space, and then stores multiple small documents, solves in field of data storage and cannot deposit at present
The problem of storage mass small documents.
In one embodiment, said method is further comprising the steps of: set up the index information table of unit memory space, rope
Draw information table and include each small documents stored in unit memory space positional information in unit memory space.This enforcement
In example, each unit memory space is set up an index information table so that in each unit memory space, storage is multiple
Small documents has each self-corresponding positional information, consequently facilitating utilize the least from unit memory space of this index information table
File accurately finds out required small documents.
In one embodiment, after setting up index information table for unit memory space, said method also includes following step
Rapid: when receiving the access request to small documents, inquire about small documents position in unit memory space according to index information table
Confidence ceases；According to this location information access small documents.Visible, in this embodiment, it is possible to accurately to inquire according to index information table
Each small documents positional information in unit memory space, and then according to the small documents needed for location information access, for data
Index provides a great convenience.
Fig. 5 is the schematic block diagram of a kind of file storage device.This document storage dress
Putting the storage engines being applied to store multiple small documents, wherein, small documents refers to that memory space shared by file is less than file threshold value.As
Shown in Fig. 5, this device includes:
Forward allocator module 510, is suitable to utilize storage engines to divide memory space, and predistribution goes out unit storage sky
Between.
Module 520 is set, is coupled with forward allocator module 510, be suitable to treat in unit memory space in storage engines
The small documents of storage arranges the index node of correspondence.
First memory module 530, is coupled with arranging module 520, is suitable to when storage engines receives new small documents
During data write request, keep index node constant, new small documents is write unit memory space and preserves.
In one embodiment, the first memory module 530 includes:
Combining unit, is suitable to the Piece file mergence that will store in new small documents and unit memory space, generates and merge literary composition
Part.
First memory element, is suitable to when shared by merging file, memory space is less than or equal to unit memory space, at list
In the memory space of position, storage merges file.
In one embodiment, combining unit, be suitable to obtain the literary composition stored in new small documents and unit memory space
The each self-corresponding binary code of part；The file pair that will store in binary code corresponding for new small documents and unit memory space
The binary code answered combines.
In one embodiment, as shown in Figure 6, said apparatus also includes:
Distribution module 540, is coupled with the first memory module 530, is suitable to when merging memory space shared by file more than single
During the memory space of position, distribute new unit memory space.
Second memory module 550, is coupled with distribution module 540, is suitable to unit storage new for new small documents write
Space also preserves.
In one embodiment, the first memory module 530 includes:
Search unit, be suitable to search the end position of the last write file in unit memory space.
Second memory element, is suitable to be sequentially written in new small documents after end position and preserve.
In one embodiment, forward allocator module 510 includes:
First determines unit, is suitable to taking up room, really of the gross space capacity according to memory headroom and each index node
Determine the number of the unit memory space that can divide under memory space.
Second determines unit, is suitable to the gross space capacity according to number and memory space, determines the sky of unit memory space
Between size.
Predistribution unit, is suitable to the space size according to unit memory space, and predistribution goes out unit memory space.
In one embodiment, as it is shown in fig. 7, said apparatus also includes:
Set up module 560, be coupled with the first memory module 530, be adapted to set up the index information table of unit memory space,
Index information table includes each small documents stored in unit memory space positional information in unit memory space.
Enquiry module 570, is coupled with setting up module 560, is suitable to when receiving the access request to small documents, according to
Index information table inquiry small documents positional information in unit memory space.
Access modules 580, is coupled with enquiry module 570, is suitable to according to location information access small documents.
Use the device that the embodiment of the present invention provides, it is possible to utilize storage engines to allocate out unit in memory space in advance
Memory space, and the index node of correspondence is set for small documents to be stored in this unit memory space, thus decrease disk
The fragment in space produces, and, when receiving the data write request of small documents, will in the case of keeping index node constant
Small documents write unit memory space also preserves so that when writing data in memory space without revising in index node in
Hold, thus the generation that the system that decreases is called, optimize the utilization rate to disk I/O, meanwhile, use this file that writes direct
Method can also reduce write amplify problem, be very suitable for store large amount of small documents storage engines.
Should be understood that, the file storage device in Fig. 5 to Fig. 7 can be described previously for realizing
File storage scheme, details therein describe should with method part above describe similar, loaded down with trivial details for avoiding, the most separately go to live in the household of one's in-laws on getting married
State.
In description mentioned herein, illustrate a large amount of detail.It is to be appreciated, however, that the enforcement of the present invention
Example can be put into practice in the case of not having these details.In some instances, it is not shown specifically known method, structure
And technology, in order to do not obscure the understanding of this description.
Similarly, it will be appreciated that one or more in order to simplify that the disclosure helping understands in each inventive aspect, exist
Above in the description of the exemplary embodiment of the present invention, each feature of the present invention is grouped together into single enforcement sometimes
In example, figure or descriptions thereof.But, the method for the disclosure should not be construed to reflect an intention that i.e. required guarantor
The application claims feature more more than the feature being expressly recited in each claim protected.More precisely, as following
Claims reflected as, inventive aspect is all features less than single embodiment disclosed above.Therefore,
The claims following detailed description of the invention are thus expressly incorporated in this detailed description of the invention, the most each claim itself
All as the independent embodiment of the present invention.
Those skilled in the art are appreciated that and can carry out the module in the equipment in embodiment adaptively
Change and they are arranged in one or more equipment different from this embodiment.Can be the module in embodiment or list
Unit or assembly are combined into a module or unit or assembly, and can put them in addition multiple submodule or subelement or
Sub-component.In addition at least some in such feature and/or process or unit excludes each other, can use any
Combine all features disclosed in this specification (including adjoint claim, summary and accompanying drawing) and so disclosed appoint
Where method or all processes of equipment or unit are combined.Unless expressly stated otherwise, this specification (includes adjoint power
Profit requires, summary and accompanying drawing) disclosed in each feature can be carried out generation by providing identical, equivalent or the alternative features of similar purpose
Replace.
Although additionally, it will be appreciated by those of skill in the art that embodiments more described herein include other embodiments
Some feature included by rather than further feature, but the combination of the feature of different embodiment means to be in the present invention's
Within the scope of and form different embodiments.Such as, in detail in the claims, embodiment required for protection one of arbitrarily
Can mode use in any combination.
The all parts embodiment of the present invention can realize with hardware, or to run on one or more processor
Software module realize, or with combinations thereof realize.It will be understood by those of skill in the art that and can use in practice
Microprocessor or digital signal processor (DSP) realize some in file storage device according to embodiments of the present invention or
The some or all functions of the whole parts of person.The present invention is also implemented as performing method as described herein
Point or whole equipment or device program (such as, computer program and computer program).Such realize this
Bright program can store on a computer-readable medium, or can be to have the form of one or more signal.Such
Signal can be downloaded from internet website and obtain, or provides on carrier signal, or provides with any other form.
The present invention will be described rather than limits the invention to it should be noted above-described embodiment, and ability
Field technique personnel can design alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference marks that should not will be located between bracket is configured to limitations on claims.Word " comprises " and does not excludes the presence of not
Arrange element in the claims or step.Word "a" or "an" before being positioned at element does not excludes the presence of multiple such
Element.The present invention and can come real by means of including the hardware of some different elements by means of properly programmed computer
Existing.If in the unit claim listing equipment for drying, several in these devices can be by same hardware branch
Specifically embody.Word first, second and third use do not indicate that any order.These word explanations can be run after fame
Claim.
So far, although those skilled in the art will appreciate that the multiple of the most detailed present invention of illustrate and describing show
Example embodiment, but, without departing from the spirit and scope of the present invention, still can be direct according to present disclosure
Determine or derive other variations or modifications of many meeting the principle of the invention.Therefore, the scope of the present invention is it is understood that and recognize
It is set to and covers other variations or modifications all these.
Embodiments provide A1. file memory method, be applied to store the storage engines of multiple small documents,
Wherein, described small documents refers to that memory space shared by file is less than file threshold value, and described method includes:
Utilizing described storage engines to divide memory space, predistribution goes out unit memory space；
The index node of correspondence is set for small documents to be stored in described unit memory space in described storage engines；
When described storage engines receives the data write request of new small documents, keep described index node constant,
Described new small documents is write described unit memory space and preserves.
A2, according to the method described in A1, wherein, described described new small documents write described unit memory space and protects
Deposit, including:
The Piece file mergence that will store in described new small documents and described unit memory space, generates and merges file；
When memory space shared by described merging file is less than or equal to described unit memory space, store in described unit
Space stores described merging file.
A3, according to the method described in A2, wherein, described described new small documents is deposited in described unit memory space
The Piece file mergence of storage, including:
Obtain each self-corresponding binary code of file stored in described new small documents and described unit memory space；
Binary code corresponding for described new small documents is corresponding with the file stored in described unit memory space
Binary code combines.
A4, according to the method described in A2 or A3, wherein, described method also includes:
When memory space shared by described merging file is more than described unit memory space, distribute new unit storage sky
Between；
Described new small documents is write described new unit memory space and preserves.
A5, according to the method described in A1, wherein, described described new small documents write described unit memory space and protects
Deposit, including:
Search the end position of the last write file in described unit memory space；
After described end position, it is sequentially written in described new small documents and preserves.
A6, according to the method described in any one of A1-A5, wherein, described utilize described storage engines that memory space is carried out
Dividing, predistribution goes out described unit memory space, including:
Taking up room of gross space capacity according to described memory headroom and each index node, determines described memory space
The number of the unit memory space that can divide down；
Gross space capacity according to described number and described memory space, determines that the space of described unit memory space is big
Little；
According to the space size of described unit memory space, predistribution goes out described unit memory space.
A7, according to the method described in any one of A1-A6, wherein, described method also includes:
Setting up the index information table of described unit memory space, described index information table includes in described unit memory space
The each small documents stored positional information in described unit memory space.
A8, according to the method described in A7, wherein, described method also includes:
When receiving the access request to described small documents, inquire about described small documents in institute according to described index information table
State the positional information in unit memory space；
According to small documents described in described location information access.
The embodiment of the present invention additionally provides B9, a kind of file storage device, and the storage being applied to store multiple small documents is drawn
Holding up, wherein, described small documents refers to that memory space shared by file is less than file threshold value, and described device includes:
Forward allocator module, is suitable to utilize described storage engines to divide memory space, and predistribution goes out unit storage sky
Between；
Arranging module, it is right to be suitable to arrange for small documents to be stored in described unit memory space in described storage engines
The index node answered；
First memory module, is suitable to when described storage engines receives the data write request of new small documents, keeps
Described index node is constant, and described new small documents is write described unit memory space and preserved.
B10, according to the device described in B9, wherein, described first memory module includes:
Combining unit, is suitable to the Piece file mergence that will store in described new small documents and described unit memory space, raw
Become to merge file；
First memory element, is suitable to when memory space shared by described merging file is less than or equal to described unit memory space
Time, described unit memory space stores described merging file.
B11, according to the device described in B10, wherein, described combining unit, be suitable to obtain described new small documents and described
The each self-corresponding binary code of file stored in unit memory space；By binary code corresponding for described new small documents with
The binary code that the file that stored in described unit memory space is corresponding combines.
B12, according to the device described in B10 or B11, wherein, described device also includes:
Distribution module, is suitable to when memory space shared by described merging file is more than described unit memory space, and distribution is new
Unit memory space；
Second memory module, is suitable to described new small documents is write described new unit memory space and preserved.
B13, according to the device described in B9, wherein, described first memory module includes:
Search unit, be suitable to search the end position of the last write file in described unit memory space；
Second memory element, is suitable to be sequentially written in described new small documents after described end position and preserve.
B14, according to the device described in any one of B9-B13, wherein, described forward allocator module includes:
First determines unit, and be suitable to the gross space capacity according to described memory headroom and each index node takies sky
Between, determine the number of the unit memory space that can divide under described memory space；
Second determines unit, is suitable to the gross space capacity according to described number and described memory space, determines described unit
The space size of memory space；
Predistribution unit, is suitable to the space size according to described unit memory space, and predistribution goes out described unit storage sky
Between.
B15, according to the device described in any one of B9-B14, wherein, described device also includes:
Setting up module, be adapted to set up the index information table of described unit memory space, described index information table includes described
The each small documents stored in unit memory space positional information in described unit memory space.
B16, according to the device described in B15, wherein, described device also includes:
Enquiry module, is suitable to when receiving the access request to described small documents, inquires about according to described index information table
Described small documents positional information in described unit memory space；
Access modules, is suitable to according to small documents described in described location information access.
Claims (10)
1. a file memory method, is applied to store the storage engines of multiple small documents, and wherein, described small documents refers to file institute
Accounting for memory space and be less than file threshold value, described method includes:
Utilizing described storage engines to divide memory space, predistribution goes out unit memory space；
The index node of correspondence is set for small documents to be stored in described unit memory space in described storage engines；
When described storage engines receives the data write request of new small documents, keep described index node constant, by institute
State new small documents write described unit memory space and preserve.
Method the most according to claim 1, wherein, described writes described unit memory space also by described new small documents
Preserve, including:
The Piece file mergence that will store in described new small documents and described unit memory space, generates and merges file；
When memory space shared by described merging file is less than or equal to described unit memory space, at described unit memory space
Middle storage described merging file.
Method the most according to claim 2, wherein, described by described new small documents and described unit memory space
The Piece file mergence of storage, including:
Obtain each self-corresponding binary code of file stored in described new small documents and described unit memory space；
Two enter binary code corresponding for described new small documents is corresponding with the file stored in described unit memory space
Code processed combines.
The most according to the method in claim 2 or 3, wherein, described method also includes:
When memory space shared by described merging file is more than described unit memory space, distribute new unit memory space；
Described new small documents is write described new unit memory space and preserves.
Method the most according to claim 1, wherein, described writes described unit memory space also by described new small documents
Preserve, including:
Search the end position of the last write file in described unit memory space；
After described end position, it is sequentially written in described new small documents and preserves.
6. according to the method described in any one of claim 1-5, wherein, described utilize described storage engines that memory space is carried out
Dividing, predistribution goes out described unit memory space, including:
Taking up room of gross space capacity according to described memory headroom and each index node, determining can under described memory space
The number of the unit memory space divided；
Gross space capacity according to described number and described memory space, determines the space size of described unit memory space；
According to the space size of described unit memory space, predistribution goes out described unit memory space.
7. according to the method described in any one of claim 1-6, wherein, described method also includes:
Setting up the index information table of described unit memory space, described index information table includes depositing in described unit memory space
The each small documents of storage positional information in described unit memory space.
Method the most according to claim 7, wherein, described method also includes:
When receiving the access request to described small documents, inquire about described small documents at described list according to described index information table
Positional information in the memory space of position；
According to small documents described in described location information access.
9. a file storage device, is applied to store the storage engines of multiple small documents, and wherein, described small documents refers to file institute
Accounting for memory space and be less than file threshold value, described device includes:
Forward allocator module, is suitable to utilize described storage engines to divide memory space, and predistribution goes out unit memory space；
Module is set, is suitable to arrange correspondence for small documents to be stored in described unit memory space in described storage engines
Index node；
First memory module, is suitable to when described storage engines receives the data write request of new small documents, keeps described
Index node is constant, and described new small documents is write described unit memory space and preserved.
Device the most according to claim 9, wherein, described first memory module includes:
Combining unit, is suitable to the Piece file mergence that will store in described new small documents and described unit memory space, generates and close
And file；
First memory element, is suitable to when memory space shared by described merging file is less than or equal to described unit memory space,
Described merging file is stored in described unit memory space.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610616789.2A CN106294603B (en) | 2016-07-29 | 2016-07-29 | File memory method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610616789.2A CN106294603B (en) | 2016-07-29 | 2016-07-29 | File memory method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106294603A true CN106294603A (en) | 2017-01-04 |
| CN106294603B CN106294603B (en) | 2019-07-23 |
Family
ID=57663526
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610616789.2A Active CN106294603B (en) | 2016-07-29 | 2016-07-29 | File memory method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106294603B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106897391A (en) * | 2017-01-24 | 2017-06-27 | 北京百悟科技有限公司 | File memory method and device |
| CN107423321A (en) * | 2017-03-31 | 2017-12-01 | 上海斐讯数据通信技术有限公司 | It is applicable the method and its device of high-volume small documents cloud storage |
| CN107678981A (en) * | 2017-08-24 | 2018-02-09 | 北京盛和大地数据科技有限公司 | Data processing method and device |
| CN109992216A (en) * | 2019-04-11 | 2019-07-09 | 苏州浪潮智能科技有限公司 | A kind of data component method of completing the square, system and electronic equipment and storage medium |
| CN110008179A (en) * | 2019-04-02 | 2019-07-12 | 深圳创维汽车智能有限公司 | File memory method, automobile data recorder and readable storage medium storing program for executing |
| CN110147203A (en) * | 2019-05-16 | 2019-08-20 | 北京金山云网络技术有限公司 | A kind of file management method, device, electronic equipment and storage medium |
| CN110580599A (en) * | 2019-07-24 | 2019-12-17 | 唐颐(湖州)生物科技有限公司 | Intelligent positioning algorithm and device for biological sample storage space planning |
| CN110716696A (en) * | 2019-09-27 | 2020-01-21 | 新华三信息安全技术有限公司 | Object processing method and related device |
| CN110990606A (en) * | 2019-12-11 | 2020-04-10 | Tcl移动通信科技（宁波）有限公司 | Picture storage method and device, storage medium and electronic equipment |
| CN111026890A (en) * | 2019-11-28 | 2020-04-17 | 天脉聚源(杭州)传媒科技有限公司 | Picture data storage method, system, device and storage medium based on index table |
| CN112149026A (en) * | 2020-10-20 | 2020-12-29 | 北京天华星航科技有限公司 | Distributed data storage system based on web end |
| CN113132661A (en) * | 2021-03-11 | 2021-07-16 | 深圳市阿达视高新技术有限公司 | Video data storage method and device, storage medium and camera equipment |
| WO2021223669A1 (en) * | 2020-05-06 | 2021-11-11 | 华为技术有限公司 | Metadata storage method and device |
| CN114116634A (en) * | 2022-01-26 | 2022-03-01 | 苏州浪潮智能科技有限公司 | Caching method and device and readable storage medium |
| CN114168084A (en) * | 2021-12-10 | 2022-03-11 | 中国电信股份有限公司 | File merging method, file merging device, electronic equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101165661A (en) * | 2006-10-19 | 2008-04-23 | 索尼株式会社 | Data storage device and data controlling method |
| CN101644996A (en) * | 2009-09-25 | 2010-02-10 | 杭州华三通信技术有限公司 | Storage method of index data and storage control device |
| CN102375852A (en) * | 2010-08-24 | 2012-03-14 | ***通信集团公司 | Method for building data index as well as method and system using data index for inquiring data |
| CN102662992A (en) * | 2012-03-14 | 2012-09-12 | 北京搜狐新媒体信息技术有限公司 | Method and device for storing and accessing massive small files |
| CN102843396A (en) * | 2011-06-22 | 2012-12-26 | 中兴通讯股份有限公司 | Data writing and reading method and device in distributed caching system |
| CN103140840A (en) * | 2011-09-30 | 2013-06-05 | 华为技术有限公司 | Data management method and device |
| CN105069048A (en) * | 2015-07-23 | 2015-11-18 | 东方网力科技股份有限公司 | Small file storage method, query method and device |
| CN105138571A (en) * | 2015-07-24 | 2015-12-09 | 四川长虹电器股份有限公司 | Distributed file system and method for storing lots of small files |
| CN105404691A (en) * | 2015-12-14 | 2016-03-16 | 曙光信息产业股份有限公司 | File storage method and apparatus |
| CN105468642A (en) * | 2014-09-09 | 2016-04-06 | 杭州海康威视数字技术股份有限公司 | Data storage method and apparatus |
-
2016
- 2016-07-29 CN CN201610616789.2A patent/CN106294603B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101165661A (en) * | 2006-10-19 | 2008-04-23 | 索尼株式会社 | Data storage device and data controlling method |
| CN101644996A (en) * | 2009-09-25 | 2010-02-10 | 杭州华三通信技术有限公司 | Storage method of index data and storage control device |
| CN102375852A (en) * | 2010-08-24 | 2012-03-14 | ***通信集团公司 | Method for building data index as well as method and system using data index for inquiring data |
| CN102843396A (en) * | 2011-06-22 | 2012-12-26 | 中兴通讯股份有限公司 | Data writing and reading method and device in distributed caching system |
| CN103140840A (en) * | 2011-09-30 | 2013-06-05 | 华为技术有限公司 | Data management method and device |
| CN102662992A (en) * | 2012-03-14 | 2012-09-12 | 北京搜狐新媒体信息技术有限公司 | Method and device for storing and accessing massive small files |
| CN105468642A (en) * | 2014-09-09 | 2016-04-06 | 杭州海康威视数字技术股份有限公司 | Data storage method and apparatus |
| CN105069048A (en) * | 2015-07-23 | 2015-11-18 | 东方网力科技股份有限公司 | Small file storage method, query method and device |
| CN105138571A (en) * | 2015-07-24 | 2015-12-09 | 四川长虹电器股份有限公司 | Distributed file system and method for storing lots of small files |
| CN105404691A (en) * | 2015-12-14 | 2016-03-16 | 曙光信息产业股份有限公司 | File storage method and apparatus |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106897391A (en) * | 2017-01-24 | 2017-06-27 | 北京百悟科技有限公司 | File memory method and device |
| CN107423321B (en) * | 2017-03-31 | 2020-12-01 | 北京亿智云科技有限公司 | Method and device suitable for cloud storage of large-batch small files |
| CN107423321A (en) * | 2017-03-31 | 2017-12-01 | 上海斐讯数据通信技术有限公司 | It is applicable the method and its device of high-volume small documents cloud storage |
| CN107678981A (en) * | 2017-08-24 | 2018-02-09 | 北京盛和大地数据科技有限公司 | Data processing method and device |
| CN110008179A (en) * | 2019-04-02 | 2019-07-12 | 深圳创维汽车智能有限公司 | File memory method, automobile data recorder and readable storage medium storing program for executing |
| CN110008179B (en) * | 2019-04-02 | 2023-06-16 | 深圳创维汽车智能有限公司 | File storage method, automobile data recorder and readable storage medium |
| CN109992216A (en) * | 2019-04-11 | 2019-07-09 | 苏州浪潮智能科技有限公司 | A kind of data component method of completing the square, system and electronic equipment and storage medium |
| CN109992216B (en) * | 2019-04-11 | 2023-01-10 | 苏州浪潮智能科技有限公司 | Data set distribution method and system, electronic equipment and storage medium |
| CN110147203A (en) * | 2019-05-16 | 2019-08-20 | 北京金山云网络技术有限公司 | A kind of file management method, device, electronic equipment and storage medium |
| CN110147203B (en) * | 2019-05-16 | 2022-11-04 | 北京金山云网络技术有限公司 | File management method and device, electronic equipment and storage medium |
| CN110580599A (en) * | 2019-07-24 | 2019-12-17 | 唐颐(湖州)生物科技有限公司 | Intelligent positioning algorithm and device for biological sample storage space planning |
| CN110716696A (en) * | 2019-09-27 | 2020-01-21 | 新华三信息安全技术有限公司 | Object processing method and related device |
| CN111026890A (en) * | 2019-11-28 | 2020-04-17 | 天脉聚源(杭州)传媒科技有限公司 | Picture data storage method, system, device and storage medium based on index table |
| CN110990606A (en) * | 2019-12-11 | 2020-04-10 | Tcl移动通信科技（宁波）有限公司 | Picture storage method and device, storage medium and electronic equipment |
| CN110990606B (en) * | 2019-12-11 | 2023-10-03 | Tcl移动通信科技（宁波）有限公司 | Picture storage method and device, storage medium and electronic equipment |
| WO2021223669A1 (en) * | 2020-05-06 | 2021-11-11 | 华为技术有限公司 | Metadata storage method and device |
| CN112149026A (en) * | 2020-10-20 | 2020-12-29 | 北京天华星航科技有限公司 | Distributed data storage system based on web end |
| CN112149026B (en) * | 2020-10-20 | 2021-04-02 | 北京天华星航科技有限公司 | Distributed data storage system based on web end |
| CN113132661A (en) * | 2021-03-11 | 2021-07-16 | 深圳市阿达视高新技术有限公司 | Video data storage method and device, storage medium and camera equipment |
| CN113132661B (en) * | 2021-03-11 | 2022-04-12 | 深圳市阿达视高新技术有限公司 | Video data storage method and device, storage medium and camera equipment |
| CN114168084A (en) * | 2021-12-10 | 2022-03-11 | 中国电信股份有限公司 | File merging method, file merging device, electronic equipment and storage medium |
| CN114116634A (en) * | 2022-01-26 | 2022-03-01 | 苏州浪潮智能科技有限公司 | Caching method and device and readable storage medium |
| CN114116634B (en) * | 2022-01-26 | 2022-04-22 | 苏州浪潮智能科技有限公司 | Caching method and device and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106294603B (en) | 2019-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106294603A (en) | File memory method and device | |
| KR101994021B1 (en) | File manipulation method and apparatus | |
| CN110096227B (en) | Data storage method, data processing device, electronic equipment and computer readable medium | |
| CN108255958A (en) | Data query method, apparatus and storage medium | |
| CN111880746A (en) | Method and device for writing service data into block chain system | |
| US20190121901A1 (en) | Database Sharding | |
| US20150169570A1 (en) | Method and device for managing data | |
| US20180329725A1 (en) | Method and apparatus for loading application | |
| CN108021333B (en) | System, device and method for randomly reading and writing data | |
| CN103559139B (en) | A kind of date storage method and device | |
| CN109076021A (en) | The method and apparatus of data processing | |
| US20170083537A1 (en) | Mapping logical identifiers using multiple identifier spaces | |
| CN105447030A (en) | Index processing method and equipment | |
| CN106227769A (en) | Date storage method and device | |
| US20170293468A1 (en) | Efficient handling of sort payload in a column organized relational database | |
| CN100524314C (en) | Apparatus and method of measuring integrity | |
| CN106599247A (en) | Method and device for merging data file in LSM-tree structure | |
| CN105468644A (en) | Method and device for performing query in database | |
| CN105550220B (en) | A kind of method and device of the access of heterogeneous system | |
| KR102071072B1 (en) | Method for managing of memory address mapping table for data storage device | |
| US9189406B2 (en) | Placement of data in shards on a storage device | |
| US9317420B2 (en) | Computer program installation across multiple memories | |
| US20190354471A1 (en) | Sharing consumed off-heap for parallel data loading | |
| US11249991B2 (en) | System and method for creating storage containers in a data storage system | |
| CN107391610A (en) | Background picture setting method and device, readable storage medium and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220802 Address after: Room 801, 8th floor, No. 104, floors 1-19, building 2, yard 6, Jiuxianqiao Road, Chaoyang District, Beijing 100015 Patentee after: BEIJING QIHOO TECHNOLOGY Co.,Ltd. Address before: 100088 room 112, block D, 28 new street, new street, Xicheng District, Beijing (Desheng Park) Patentee before: BEIJING QIHOO TECHNOLOGY Co.,Ltd. Patentee before: Qizhi software (Beijing) Co.,Ltd. |
|
| TR01 | Transfer of patent right |