CN110287201A - Data access method, device, equipment and storage medium - Google Patents
Data access method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN110287201A CN110287201A CN201910591781.9A CN201910591781A CN110287201A CN 110287201 A CN110287201 A CN 110287201A CN 201910591781 A CN201910591781 A CN 201910591781A CN 110287201 A CN110287201 A CN 110287201A
- Authority
- CN
- China
- Prior art keywords
- catalogue
- data
- index
- index file
- deleted
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a kind of data access method, device, equipment and storage medium, is related to cloud storage technical field.Wherein, this method can be requested by receiving the directory access that client is sent, directory access request includes: the information of catalogue, according to the information of catalogue, the index file of catalogue is loaded onto memory from disk, the index file for the catalogue left in being stored in is to determine the data under catalogue, the index file of catalogue includes: the index of data under catalogue, and to client Returning catalogue access response, realize the directory information indicated just for directory access request, by the index file of the catalogue from being loaded onto disk in the memory, without the index file of all catalogues is loaded onto memory, namely, without will be all index in all databases of load in memory, so as to reduce the occupancy of memory, promote the service performance of memory.
Description
Technical field
This application involves cloud storage technical fields, in particular to a kind of data access method, device, equipment and deposit
Storage media.
Background technique
With the development of internet technology, the index stored in computer is more and more, how by the index in database
It is effectively stored, reduces EMS memory occupation amount, be of great significance.
In the prior art, the index that will acquire is stored in memory database, and real-time loading calls when in use, when depositing
When the index of storage is less, this kind of storage mode, which can be loaded out quickly, to be indexed, to realize the real-time retrieval to index.
But when the index substantial amounts of storage, all indexes are all loaded into database in memory, are consumed very much
With memory, the service performance of memory can be seriously affected.
Summary of the invention
The purpose of the application is, in view of the deficiency of the prior art, provide a kind of data access method, device,
Equipment and storage medium, EMS memory occupation is high when solving to carry out mass data access based on existing data storage scheme, influences
The problem of memory service performance.
To achieve the above object, the embodiment of the present application the technical solution adopted is as follows:
In a first aspect, the embodiment of the present application provides a kind of data access method, comprising:
The directory access request that client is sent is received, directory access request includes: the information of catalogue；
According to the information of catalogue, the index file of catalogue is loaded onto memory from disk, is stored in the interior catalogue left
For index file to determine the data under catalogue, the index file of catalogue includes: the index of data under catalogue；
To client Returning catalogue access response.
Optionally, according to the information of catalogue, the index file of catalogue is loaded onto memory from disk, comprising:
According to the information of catalogue, the index file of catalogue is searched from disk；
Using the corresponding storage format in data in EMS memory library, the database index file of catalogue being loaded onto memory.
Optionally, the data access method further include:
Receive client send catalogue under data delete operation, data delete operation include: catalogue information and to
Delete the information of data；
According to the information of the information of catalogue and data to be deleted, data to be deleted in the index file of catalogue are deleted in memory
Index；
The index of data to be deleted is moved to the index to be deleted of catalogue in disk from the index file of catalogue in disk
In file；
It deletes and responds to client returned data.
Optionally, if data to be deleted be catalogue under first group of data to be deleted, by the index of data to be deleted from
Before the index file of catalogue is moved in disk in the index file to be deleted of catalogue in disk, which is also wrapped
It includes:
The index file to be deleted to create directory in disk.
Optionally, the data access method further include:
Receive the data write request for the catalogue that client is sent, data write request includes: the information of catalogue and to be written
Enter data；
It according to the information of catalogue, is written into data and stores into disk under catalogue, and generate the index of data to be written；
The index for being written into data increases to the index file of catalogue in the index file and memory of catalogue in disk
In；
It is written and responds to client returned data.
Optionally, this method further include:
The data read request for the catalogue that client is sent is received, data read request includes: the information of catalogue, and continues
The identification information for evidence of fetching；
According to the information of catalogue, the index file that catalogue whether is stored in memory determined；
If so, searching access of continuing from the index file of the catalogue in memory according to the identification information of data to be read
According to index, and data to be read are searched under catalogue in disk according to the index of data to be read；
It reads and responds to client returned data.
Second aspect, the embodiment of the present application also provides a kind of data access devices, comprising:
First receiving module, for receiving the directory access request of client transmission, directory access request includes: catalogue
Information；
Index file under catalogue is loaded onto memory from disk, is stored in by loading module for the information according to catalogue
For the index file for the catalogue inside left to determine the data under catalogue, the index file of catalogue includes: the rope of data under catalogue
Draw；
First respond module is used for client Returning catalogue access response.
Optionally, loading module is also used to the information according to catalogue, and the index file of catalogue is searched from disk；Using
The corresponding storage format in data in EMS memory library, the database index file of catalogue being loaded onto memory.
Optionally, the data access device further include:
Second receiving module, the data delete operation under catalogue for receiving client transmission, data delete operation packet
It includes: the information of the information of catalogue and data to be deleted；
Removing module, for deleting the index text of catalogue in memory according to the information of catalogue and the information of data to be deleted
The index of data to be deleted in part；
Mobile module, for the index file of catalogue to be moved to described in disk from disk by the index of data to be deleted
In the index file to be deleted of catalogue；
Second respond module is responded for deleting to client returned data.
Optionally, data access device further include: creation module；
If data to be deleted are first group of data to be deleted under catalogue, creation module is used for will be wait delete in mobile module
Before being moved in disk in the index file to be deleted of catalogue except the index of data from the index file of catalogue in disk, in magnetic
The index file to be deleted to create directory in disk.
Optionally, the data access device further include:
Third receiving module, the data write request of the catalogue for receiving client transmission；Data write request includes:
The information of catalogue and data to be written；
Writing module is written into data and stores into disk under catalogue for the information according to catalogue, and generates to be written
Enter the index of data；
Increase module, the index for being written into data increases in disk mesh in the index file and memory of catalogue
In the index file of record；
Third respond module is responded for being written to client returned data.
Optionally, the data access device further include:
4th receiving module, the data read request of the catalogue for receiving client transmission；Data read request includes:
The identification information of the information of catalogue and data to be read；
The index file that catalogue whether is stored in memory determined for the information according to catalogue for searching module；If so,
Then according to the identification information of data to be read, the index of data to be read, and root are searched from the index file of the catalogue in memory
Data to be read are searched under catalogue in disk according to the index of data to be read；
4th respond module is responded for reading to client returned data.
The third aspect, the embodiment of the present application also provide a kind of data access equipment, comprising: processor and memory, storage
The computer program that can be run in processor is stored in device, processor is realized as described in relation to the first aspect when executing computer program
Data access method.
Fourth aspect, the embodiment of the present application also provide a kind of computer storage medium, are stored thereon with computer program, meter
Data access method as described in relation to the first aspect is realized when calculation machine program is executed by processor.
The beneficial effect of the application is:
Data access method, device, equipment and storage medium provided by the present application can be sent by receiving client
Directory access request, directory access request includes: the information of catalogue, according to the information of catalogue, by the index file of catalogue from magnetic
Disk is loaded onto memory, is stored in the index file of the interior catalogue left to determine the data under catalogue, the index file of catalogue
Include: the index of data under catalogue, and to client Returning catalogue access response, realizes just for indicated by directory access request
Directory information, by the index file of the catalogue from being loaded onto disk in the memory, without by the index text of all catalogues
Part is loaded onto memory, that is, without all loading all indexes in database in memory, in reducing
The occupancy deposited promotes the service performance of memory.
Detailed description of the invention
Technical solution in ord to more clearly illustrate embodiments of the present application, below will be to needed in the embodiment attached
Figure is briefly described, it should be understood that the following drawings illustrates only some embodiments of the application, therefore is not construed as pair
The restriction of range for those of ordinary skill in the art without creative efforts, can also be according to this
A little attached drawings obtain other relevant attached drawings.
Fig. 1 is the flow diagram of data access method provided by the embodiments of the present application；
Fig. 2 is another flow diagram of data access method provided by the embodiments of the present application；
Fig. 3 is index file storage format schematic diagram provided by the embodiments of the present application；
Fig. 4 is the another flow diagram of data access method provided by the embodiments of the present application；
Fig. 5 is bibliographic structure schematic diagram provided by the embodiments of the present application；
Fig. 6 is file memory format schematic diagram to be deleted provided by the embodiments of the present application；
Fig. 7 is the another flow diagram of data access method provided by the embodiments of the present application；
Fig. 8 is the another flow diagram of data access method provided by the embodiments of the present application；
Fig. 9 is the structural schematic diagram of data access device provided by the present application；
Figure 10 is another structural schematic diagram of data access device provided by the embodiments of the present application；
Figure 11 is the another structural schematic diagram of data access device provided by the embodiments of the present application；
Figure 12 is the another structural schematic diagram of data access device provided by the embodiments of the present application；
Figure 13 is the another structural schematic diagram of data access device provided by the embodiments of the present application；
Figure 14 is the structural schematic diagram of data access equipment provided by the embodiments of the present application.
Specific embodiment
To keep the purposes, technical schemes and advantages of the embodiment of the present application clearer, below in conjunction with the embodiment of the present application
In attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is
Some embodiments of the present application, instead of all the embodiments.
In safety-security area, for example, in terms of traffic, to monitor whether the traveling of driving vehicle meets specification, in road or
It is usually provided with candid photograph camera on expressway, to capture the instantaneous photo of vehicle, the picture of capture or other data are sent out
It send to server, picture is further analyzed in server, judges whether the vehicle driving in the picture of capture meets specification.
Since the number of pictures obtained within a certain period of time is huge, how under the premise of not influencing server performance to huge photo
It is effectively accessed, reduces the use of server memory.
Wherein, above-mentioned candid photograph camera can be electronic police, high speed bayonet, illegal behaviour capturing, flow detection, the monitoring of day net
Deng specifically used camera the application is with no restrictions.
It is analyzed for storing 10,000,000,000 grades of image datas, each image data occupies 64 bytes, then needs about 596GB empty
Between, and as the size of image data increases at multiple.In internal storage data library storage technical solution, picture is stored entirely in interior
In depositing, then the server of tens of 16GB memories, higher cost are needed, and the access frequency of most image datas is lower,
Memory-resident has that memory usage is low, and Server Restart load 16GB is indexed in memory, is taken a long time.
Therefore, the embodiment of the present application provides a kind of data access method, to solve the above problems.This method can be by server-side
It realizes, which can be for example application server corresponding to client.Wherein, data can be picture, video recording, recording
Deng, the application for the concrete type and with no restriction of data.
Fig. 1 is the flow diagram of data access method provided by the embodiments of the present application.As shown in Figure 1, the data access
Method can include:
S110, the directory access request that client is sent is received, directory access request includes: the information of catalogue.
The catalogue can be the data storage catalogue on the disk of server-side.On the disk, catalogue is hierarchical structure.The mesh
The hierarchical structure of record is divided according to data information, such as according to the size of time range, the hierarchical structure of catalogue according to
Secondary may include: the catalogue of the levels such as year, month, day, hour.
It include 12 months nodes in the node that catalogue is year, in the node that catalogue is the moon when using the time as foundation
It with the data of capture is on an hourly basis a period in the node that catalogue is day comprising 30 or 31 day node, it may include
24 timing nodes.The data of capture can be subjected to storage classification sequentially in time.
Optionally, when using the time as the division of the hierarchical structure of foundation not with year, month, day, it is small when etc. hierarchical structures draw
It is divided into limitation.Correspondingly, it may also include four or five weeks node under moon node, further include day in week node
Node etc..
It should be noted that data access method provided by the present application, in directory hierarchy, the division of timing node
Mode not to be a period as limitation per hour, can also be that two hours are divided into a period correspondingly,
Or half an hour divides a period, specific zoned format is subject to practical application.
It is illustrated by taking image data as an example as follows, if desired accesses the picture of some vehicle shot in some day, point
The vehicle is analysed with the presence or absence of unlawful practice, catalogue need to be first accessed, can be looked into catalogue directly according to the temporal information of user demand
Find specific day node.For example, it is desired to know that the vehicle that 2019/01/01 license plate is capital A00000 whether there is unlawful practice,
The information in catalogue and catalogue can be accessed directly according to known temporal information.
It should be noted that the hierarchical structure of above-mentioned catalogue is not limitation with basis of time.
Index file under catalogue is loaded onto memory from disk by S120, the information according to catalogue, be stored in leave
For the index file of catalogue to determine the data under catalogue, the index file of catalogue includes: the index of data under catalogue.
It should be noted that above-mentioned index file can include: be stored in the index of each data under the catalogue, every number
According to index may include following at least one information: the titles of each data, mark (ID), format, acquisition time, size,
Storage location etc..
The index of data is illustrated by taking the image identification of image data as an example, following manner can be used in the index of image
It generates: according to the acquisition time of image sequence, the index of image is generated with this, such as 1.jpg, 2.jpg, 3.jpg.With obtaining for image
For taking the time, following manner generation is can be used in the index of image: according to the acquisition time of the image, generating the rope of the image
Draw.If the acquisition time of an image is 01 points of 10 AM on January 1st, 2,019 01 seconds, the index of an image
It can be 2019/01/01/10/01/01.It above are only the specific example of index, herein with no restrictions.
Wherein, jpeg format (Joint Photographic Experts Group, abbreviation JPEG) is common one kind
Image storage format, extension name include .jpg or .jpeg.Certainly, image involved in this programme can also be extended formatting
Image, the application is not to this limitation.
It should be noted that the index file under catalogue is loaded onto memory from disk in this method, without by the mesh
Image under record is loaded onto memory.That is, the index file under the catalogue is only stored on the memory of server-side, without storing the mesh
Image under record.
It, can be according to the information of the catalogue, from the index file obtained in disk under the catalogue, and by the catalogue in this method
Under index file store the load that index file is realized to the memory.Server-side can by the index file under the catalogue, with
The form of database is stored on memory, that is, is stored in the database to the memory.
The index file of the catalogue stored on the memory can include: the index of data, client device can under the catalogue
By sending directory access request to server, to obtain the index file under the catalogue, realize to the catalogue on the disk
Access.
S130, to client Returning catalogue access response.
Directory access response may include such as down toward a kind of parameter: the index under the catalogue under index file and the catalogue
Display mode, display parameters of file etc..Server-side can be sent to client the directory access response so that the client according to
The directory access response received, shows the listed files of the catalogue, such as on the visualization interface of client with list
Form shows the information of the data under the catalogue, to realize the access of catalogue.
The scheme of the application can also be achieved specific data access in the case where meeting directory access.User is if you need to visit
It asks target data, data access can be sent by sending data access instruction to client to trigger the client to server-side
Request, the data access request can include: the information of the target data is as identified, so that the server-side can be somebody's turn to do according to what is received
The information of target data searches the index of the target data from the index file of the catalogue in memory, then according to the mesh
The index of heading record searches the target data from disk.The server-side can be sent out after searching the target data to client
It send the data access including the target data to respond, realizes the data access under the catalogue.
It is pointed out that in an index and disk in the index file in the memory under the catalogue under the catalogue
Data there are corresponding relationship, such as an index can correspond to one or more data under the catalogue catalogue.
From the above mentioned, data access method provided by the embodiments of the present application, the catalogue that can be sent by receiving client
Access request, directory access request include: that the information of catalogue adds the index file of catalogue from disk according to the information of catalogue
It is loaded onto memory, is stored in the index file of the interior catalogue left to determine the data under catalogue, the index file of catalogue includes:
The index of data under catalogue, and to client Returning catalogue access response, realize the mesh indicated just for directory access request
Information is recorded, by the index file of the catalogue from being loaded onto the memory in disk, without the index file of all catalogues is equal
It is loaded onto memory, that is, without all loading all indexes in database in memory, so as to reduce memory
It occupies, promotes the service performance of memory.
In addition, it is necessary to explanation, due to data access method provided by the embodiments of the present application, without by all indexes
All in the database of load in memory, occupied memory source is less, it is thereby achieved that the storage to mass data
And access.
On the basis of above-mentioned data access method shown in FIG. 1, the embodiment of the present application also provides a kind of data access side
Method.The realization of step S120 is illustrated in detail below.
Fig. 2 is another flow diagram of data access method provided by the embodiments of the present application.Optionally, as shown in Fig. 2,
Above-mentioned steps S120 may include:
S121, the information according to catalogue search the index file of catalogue from disk.
The known temporal information for requiring to look up data is searched from disk comprising being somebody's turn to do according to the hierarchy structure information of catalogue
The index file of catalogue.
S122, using the corresponding storage format in data in EMS memory library, the number index file of catalogue being loaded onto memory
According to library.
Database in above-mentioned memory can be key assignments (Key-Value, abbreviation KV) database, relevant database etc.,
The form of concrete database is herein with no restrictions.If the database in the memory is key assignments storing data library, the storage format
It can be key assignments storage format.
It, can will be under the catalogue in the case where receiving the directory access request of client transmission by taking KV database as an example
Index file be saved in the form of KV in the KV database in memory, and receive client transmission the catalogue under number
In the case where according to access request, the KV database in the memory may have access to, with literary from the index under the catalogue in the KV database
In part, the index of the target data is searched, then according to the index of the target data, the access target data from disk.Using
KV storage format stores the index file of the catalogue in the KV database into the memory, can be improved and has indexed in memory
Readwrite performance reduces delay, to improve the access speed of data.
Hereafter, it is illustrated by taking KV storage format as an example.Fig. 3 is index file storage format provided by the embodiments of the present application
Schematic diagram.
As shown in figure 3, index file can include: index (INDEX), version (version), the pairing of keyword value
(key-value pairs), end-of-file mark (EOF), verification and (check_sum).
Wherein, the part INDEX saves " I ", " N ", " D ", " E ", " X " character, to indicate the type of index file, checks
It is loaded into whether file is index file.Version uses a string representation integer in part, is used to indicate the version of index file
This.The part key-value pairs saves the mapping relations under current directory between the title and data id of all data, for example,
It may include: length (len), keyword (key) and value (value) etc..The part the EOF marked index end of file, when loading rope
When encountering the character during quotation part, indicates that all indexes are loaded and finish, to the end indexed in identification index file
Position.What verification and the part check_sum record INDEX, version, key-value pairs and the part EOF were calculated
It verifies and is used to check whether loading file to be wrong or damages.
Fig. 4 is the another flow diagram of data access method provided by the embodiments of the present application.
Optionally, as shown in figure 4, the data access method is on the basis of above-mentioned data access method shown in fig. 1 or fig. 2
On, it may also include that
Data delete operation under the catalogue that S140, reception client are sent, data delete operation includes: the information of catalogue
With the information of data to be deleted.
It, can if the data for including under current directory had accessed, and not will do it secondary access within a preset period of time
Delete operation is carried out to the data.
It is data to be deleted that the application, which can will need the Data Identification that delete, the data information deleted as needed obtain to
The directory information for deleting data, finds the information for the data to be deleted for including in the catalogue in directory information.
S141, according to the information of catalogue and the information of data to be deleted, delete in memory in the index file of catalogue wait delete
Except the index of data.
It, can be by the number to be deleted to be deleted in the case where the delete operation for receiving the data to be deleted of client transmission
According to index from being moved in the index file of the catalogue in the disk in disk in the index file to be deleted, realize magnetic
Disk guarantees the recovery of the subsequent data to be deleted to the index of data to be deleted and the temporary storage of data.
S142, the index of data to be deleted is moved to the to be deleted of catalogue in disk from the index file of catalogue in disk
In index file.
Wherein, during deletion, by the index of the data to be deleted from the disk in the index file of the catalogue
It is moved in disk in the index file to be deleted, refers to the index of the data to be deleted, the catalogue from the disk
Index file in delete, and be stored in the index file to be deleted.
It stores by the data of the index of data to be deleted to file to be deleted, if within a preset period of time, service
Termination receive client transmission the access request for the data to be deleted, server-side can by the index of the data to be deleted from
The index file to be deleted is moved in the disk in the index file of the catalogue, and the index file of the catalogue adds in the disk
It is loaded onto memory, realizes the recovery of the index file of the catalogue in memory, can then be somebody's turn to do according to being obtained from the memory wait delete
Except the index of data, the access of the data to be deleted is then carried out.
In above-mentioned recovery process, the index of the data to be deleted is moved in the disk from the index file to be deleted
In the index file of the catalogue, refer in the catalogue by the index of the data to be deleted, where before restoring into disk
In index file, and save in index file in the original location.
If the access for the data to be deleted that within a preset period of time, server-side does not receive client transmission is asked
It asks, then the server-side can delete the index of the data to be deleted in the index file to be deleted, while can also be to this
The data to be deleted of this in disk are deleted.
Preset time period as shown above for example can be any for two days, three days or one week etc., and the specific time does not do herein
Limitation.
Further, referring to figure 5., Fig. 5 is bibliographic structure schematic diagram provided by the embodiments of the present application.Wherein, suffix is
" index " (" .index ") is expressed as index file, saves the index of all data under current directory, only deposits under current directory
In data, index file can be just created under the catalogue, catalogue one and only one index file.Suffix is that " is deleted
Except " (" .del ") be expressed as index file to be deleted, it saves and indexes delete operation under current directory, efficiently data to be supported to delete
Operation, when only executing delete operation under current directory, can just create index file to be deleted, a mesh under the catalogue
Record one and only one index file to be deleted.
Further, Fig. 6 is please referred to, Fig. 6 is file memory format schematic diagram to be deleted provided by the embodiments of the present application.
Index delete (INDEXD) part save " I ", " N ", " D ", " E ", " X ", " D " character, for check be loaded into file whether be to
Index file is deleted, the type (TYPE) of index operation and keyword take under the record current directory of the part key-value pairs
It is worth (key value), rest part is identical as index file format, does not add to repeat herein.
S143, response is deleted to client returned data.
After the completion of deletion, the information such as " deletion " or " delete and complete " can be responded on the visualization interface of client,
Make user define current delete operation to have completed, the response after the completion of deletion on particular user visualization interface is not done herein
Limitation.
It further, can also be to the index of the catalogue outside the index of data is deleted in the catalogue in memory
File is deleted.
For example, multiple indexes included in a certain index file will not access again within a preset period of time, can incite somebody to action
The index file is deleted, and all indexes of the index file are stored in index file to be deleted.
Further, data access method provided by the present application, if data to be deleted are to be deleted for first group under catalogue
The index of data to be deleted is then moved to the index to be deleted text of catalogue in disk by data from the index file of catalogue in disk
Before in part, the data access method further include:
The index file to be deleted to create directory in disk.
For example, this method can be first in disk if data to be deleted are first data to be deleted under catalogue
The index file to be deleted to create directory, it is then that the index of data to be deleted is mobile from the index file of catalogue described in disk
Into disk in the index file to be deleted of catalogue.
Wherein, it if deleted for the first time index file data under same catalogue, need to be created under catalogue to be deleted
Index file can directly will be to be deleted when needing to delete other index file data under same catalogue herein
Index file data be moved in disk in the index file to be deleted of catalogue.
Optionally, the application also provides a kind of data access method, and Fig. 7 is data access side provided by the embodiments of the present application
The another flow diagram of method.
Optionally, as shown in fig. 7, the data access method is in above-mentioned Fig. 1, Fig. 2 or data access method shown in Fig. 4
On the basis of, can also include:
S160, the data write request for receiving the catalogue that client is sent, data write request includes: the information of catalogue, and
Data to be written.
Data access method provided by the present application, when user needs that data are written into some catalogue, it is necessary first to connect
The write request of user is received, wherein the write request includes: the information and data to be written of catalogue.I.e. known data needs to be written are write
Which enter into catalogue.
S161, the information according to catalogue, are written into data and store into disk under catalogue, and generate data to be written
Index.
According to known directory information, the index file under the catalogue is inquired into memory database first, and will be to be written
Enter under the correspondence catalogue that data are stored in disk, produces the index of data to be written.
S162, the index for being written into data increase to the index of catalogue in the index file and memory of catalogue in disk
In file.
The index of data to be written includes the mapping relations of data ID to be written, which is recorded mesh in memory
In the index file of record.
S163, response is written to client returned data.
After the completion of write-in, the information such as response " storage is completed " or " preservation ", specifically may be used on the visualization interface of user
Depending on changing the response message at interface herein with no restrictions.
In addition to data can be written, data access method provided by the present application can also be read out data.Fig. 8
For the another flow diagram of data access method provided by the embodiments of the present application.
Optionally, as shown in figure 8, the data access method is in above-mentioned Fig. 1, Fig. 2 or data access method shown in Fig. 4
On the basis of, it may also include that
S170, the data read request for receiving the catalogue that client is sent, data read request includes: the information of catalogue,
With the identification information of data to be read.
When user needs to read the read tablet under some catalogue, it is known that directory information and data to be read where data
Identification information.
Wherein, the identification information of data to be read can store for the index of data to be read or data to be read in catalogue
Specific directory information in structure.
S171, the information according to catalogue determine the index file that catalogue whether is stored in memory.
Data to be read are first inquired in memory database with the presence or absence of index file, and if it exists, show in disk with
The index in other index files that data to be read are stored under same catalogue is formerly accessed, can be directly from the mesh
The index that data to be read are read in the lower index file of record, according to the ID of data to be read realization data to be read from disk to
Loading procedure in memory, and realize the reading of data.
If it does not exist, the index file where showing index to be read is never being read in memory database, needs elder generation
From the index file loaded under the catalogue on disk into memory database, then data to be read are inquired from memory database
Index is realized to be read according to the mapping relations of index file ID to be read in the index and disk of obtained data to be read
Data from disk to memory in loading procedure, and realize the reading of data.
S172, if so, according to the identification information of data to be read, continue from the lookup of the index file of the catalogue in memory
The index for evidence of fetching, and data to be read are searched under catalogue in disk according to the index of data to be read.
S173, response is read to client returned data.
By taking image data as an example, reading data response can respond for the reading of image, can be on user's visualization circle
Face shows capture of continuing, and user is made to check the image.
The hierarchical relationship of data access method provided by the present application, only index list structure is stored in memory database
In, and index and be stored entirely in disk, only when needing to access the data in some catalogue, just needing will be under the catalogue
Index file is loaded into memory, and committed memory is less, and loading velocity is fast.Follow-up data access is directly visited from memory database
It asks.
On the other hand, the application is in addition to providing a kind of data access method, accordingly, also provides a kind of data access dress
It sets, Fig. 9 is the structural schematic diagram of data access device provided by the embodiments of the present application.
As shown in figure 9, the apparatus may include: the first receiving module 11, for receiving the directory access of client transmission
Request, directory access request include: the information of catalogue；Loading module 12, for the information according to catalogue, by the index under catalogue
File is loaded onto memory from disk, be stored in the index file of catalogue left to determine the data under catalogue, catalogue
Index file includes: the index of data under catalogue；First respond module 13 is used for client Returning catalogue access response.
Optionally, loading module 12 is also used to the information according to catalogue, and the index file of catalogue is searched from disk；Using
The corresponding storage format in data in EMS memory library, the database index file of catalogue being loaded onto memory.
On the basis of above-mentioned data access device shown in Fig. 9, a kind of data access device is can also be provided in the application, figure
10 be another structural schematic diagram of data access device provided by the embodiments of the present application.
Optionally, as shown in Figure 10, data access device further include: the second receiving module 14 is also used to receive client
The data delete operation under the catalogue sent is held, data delete operation includes: the information of catalogue and the information of data to be deleted；It deletes
Except module 15, for according to the information of catalogue and the information of data to be deleted, deleting in memory in the index file of catalogue wait delete
Except the index of data；Mobile module 16, for the index file of catalogue to be moved to magnetic from disk by the index of data to be deleted
In the index file to be deleted of catalogue described in disk；Second respond module 17 is responded for deleting to client returned data.
Figure 11 is the another structural schematic diagram of data access device provided by the embodiments of the present application.
Optionally, as shown in figure 11, data access device further include: creation module 18；If data to be deleted are catalogue
Under first group of data to be deleted, then creation module 18 be used for mobile module 16 by the index of data to be deleted the mesh from disk
Before the index file of record is moved in disk in the index file to be deleted of catalogue, the rope to be deleted that creaties directory in disk
Quotation part.
On the basis of the data access device shown in Fig. 9, Figure 10 and Figure 11, data access dress is can also be provided in the application
It sets, Figure 12 is the another structural schematic diagram of data access device provided by the embodiments of the present application.
Optionally, as shown in figure 12, the data access device further include: third receiving module 19, for receiving client
The data write request of the catalogue of transmission, data write request include: the information and data to be written of catalogue；Writing module 20,
It for the information according to catalogue, is written into data and stores into disk under catalogue, and generate the index of data to be written；Increase
Module 21, the index for being written into data increase to the index text of catalogue in the index file and memory of catalogue in disk
In part；Third respond module 22 is responded for being written to client returned data.
Figure 13 is the another structural schematic diagram of data access device provided by the embodiments of the present application.
Optionally, as shown in figure 13, data access device provided by the present application can also include: the 4th receiving module 23,
For receiving the data read request of the catalogue of client transmission, data read request includes: the information of catalogue, and access of continuing
According to identification information；The index text that catalogue whether is stored in memory determined for the information according to catalogue for searching module 24
Part, if so, searching the rope of data to be read from the index file of the catalogue in memory according to the identification information of data to be read
Draw, and data to be read are searched under catalogue in disk according to the index of data to be read；4th respond module 25 is used for visitor
Family end returned data reads response.
The method that above-mentioned apparatus is used to execute previous embodiment offer, it is similar that the realization principle and technical effect are similar, herein not
It repeats again.
The above module can be arranged to implement one or more integrated circuits of above method, such as: one
Or multiple specific integrated circuits (Application Specific Integrated Circuit, abbreviation ASIC), or, one
Or multi-microprocessor (digital singnal processor, abbreviation DSP), or, one or more field programmable gate
Array (Field Programmable Gate Array, abbreviation FPGA) etc..For another example, when some above module passes through processing elements
When the form of part scheduler program code is realized, which can be general processor, such as central processing unit (Central
Processing Unit, abbreviation CPU) or it is other can be with the processor of caller code.For another example, these modules can integrate
Together, it is realized in the form of system on chip (system-on-a-chip, abbreviation SOC).
The embodiment of the present application also provides a kind of data access equipment, which can be the calculating with data-handling capacity
Machine, server etc..
Figure 14 is the structural schematic diagram of data access equipment provided by the embodiments of the present application.
As shown in figure 14, which includes: processor 31 and memory 32, and being stored in memory 32 can be
The computer program that processor 31 is run, processor 31 realize the number as described in preceding method embodiment when executing computer program
According to access method.Specific implementation is similar with technical effect, and which is not described herein again.
Optionally, the embodiment of the present application also provides a kind of computer storage medium, is stored thereon with computer program, calculates
The data access method as described in preceding method embodiment is realized when machine program is executed by processor.
In several embodiments provided herein, it should be understood that disclosed device and method can pass through it
Its mode is realized.For example, the apparatus embodiments described above are merely exemplary, for example, the division of the unit, only
Only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or components can be tied
Another system is closed or is desirably integrated into, or some features can be ignored or not executed.Another point, it is shown or discussed
Mutual coupling, direct-coupling or communication connection can be through some interfaces, the INDIRECT COUPLING or logical of device or unit
Letter connection can be electrical property, mechanical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme
's.
It, can also be in addition, each functional unit in each embodiment of the application can integrate in one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of hardware adds SFU software functional unit.
The above-mentioned integrated unit being realized in the form of SFU software functional unit can store and computer-readable deposit at one
In storage media.Above-mentioned SFU software functional unit is stored in a storage medium, including some instructions are used so that a computer
Equipment (can be personal computer, server or the network equipment etc.) or processor (English: processor) execute this Shen
Please each embodiment the method part steps.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory
(English: Read-Only Memory, abbreviation: ROM), random access memory (English: Random Access Memory, letter
Claim: RAM), the various media that can store program code such as magnetic or disk.
Claims (14)
1. a kind of data access method characterized by comprising
The directory access request that client is sent is received, the directory access request includes: the information of catalogue；
According to the information of the catalogue, the index file of the catalogue is loaded onto memory from disk, is stored in described and leaves
The catalogue index file to the data under the determination catalogue, the index file of the catalogue includes: the catalogue
The index of lower data；
To the client Returning catalogue access response.
2. the method according to claim 1, wherein the information according to the catalogue, by the catalogue
Index file is loaded onto memory from disk, comprising:
According to the information of the catalogue, the index file of the catalogue is searched from the disk；
Using the corresponding storage format in the data in EMS memory library, the index file of the catalogue is loaded onto the memory
Database.
3. the method according to claim 1, wherein the method also includes:
The data delete operation under the catalogue that the client is sent is received, the data delete operation includes: the mesh
The information of the information of record and data to be deleted；
According to the information of the information of the catalogue and the data to be deleted, the index file of catalogue described in the memory is deleted
Described in data to be deleted index；
The index file of index catalogue described in the disk of the data to be deleted is moved to described in the disk
In the index file to be deleted of catalogue；
It deletes and responds to the client returned data.
4. according to the method described in claim 3, it is characterized in that, if the data to be deleted are first group under the catalogue
Data to be deleted, then the index file of index catalogue described in the disk by the data to be deleted is moved to institute
Before stating in the index file to be deleted of catalogue described in disk, the method also includes:
The index file to be deleted of the catalogue is created in the disk.
5. the method according to claim 1, wherein the method also includes:
Receive the data write request for the catalogue that the client is sent；The data write request includes: the catalogue
Information and data to be written；
According to the information of the catalogue, the data to be written are stored to catalogue described in the disk, and described in generation
The index of data to be written；
The index of the data to be written is increased into institute in the index file and the memory of catalogue described in the disk
It states in the index file of catalogue；
It is written and responds to the client returned data.
6. the method according to claim 1, wherein the method also includes:
Receive the data read request for the catalogue that the client is sent；The data read request includes: the catalogue
Information and data to be read identification information；
According to the information of the catalogue, the index file that the catalogue whether is stored in the memory determined；
If so, being searched according to the identification information of the data to be read from the index file of the catalogue in the memory
The index of the data to be read, and according to being searched under index catalogue described in the disk of the data to be read
Data to be read；
It reads and responds to the client returned data.
7. a kind of data access device characterized by comprising
First receiving module, for receiving the directory access request of client transmission, the directory access request includes: catalogue
Information；
The index file of the catalogue is loaded onto memory from disk for the information according to the catalogue by loading module, storage
The interior catalogue left index file to the data under the determination catalogue, the index file packet of the catalogue
It includes: the index of data under the catalogue；
First respond module is used for the client Returning catalogue access response.
8. device according to claim 7, which is characterized in that the loading module is also used to the letter according to the catalogue
Breath, searches the index file of the catalogue from the disk；Using the corresponding storage format in the data in EMS memory library, by institute
The index file for stating catalogue is loaded onto database in the memory.
9. device according to claim 7, which is characterized in that described device further include:
Second receiving module, for receiving the data delete operation under the catalogue that the client is sent, the data are deleted
Except operation includes: the information of the catalogue and the information of data to be deleted；
Removing module, for deleting described in the memory according to the information of the catalogue and the information of the data to be deleted
The index of data to be deleted described in the index file of catalogue；
Mobile module, for the index file of index catalogue described in the disk of the data to be deleted to be moved to institute
It states in the index file to be deleted of catalogue described in disk；
Second respond module is responded for deleting to the client returned data.
10. device according to claim 9, which is characterized in that described device further include: creation module；
If the data to be deleted are first group of data to be deleted under the catalogue, the creation module, for described
The index file of index catalogue described in the disk of the data to be deleted is moved in the disk by mobile module
Before in the index file to be deleted of the catalogue, the index file to be deleted of the catalogue is created in the disk.
11. device according to claim 7, which is characterized in that described device further include:
Third receiving module, for receiving the data write request for the catalogue that the client is sent；The data write-in
Request includes: the information and data to be written of the catalogue；
Writing module stores the data to be written to catalogue described in the disk for the information according to the catalogue
Under, and generate the index of the data to be written；
Increase module, for the index of the data to be written is increased to catalogue described in the disk index file and
In the index file of catalogue described in the memory；
Third respond module is responded for being written to the client returned data.
12. device according to claim 7, which is characterized in that described device further include:
4th receiving module, for receiving the data read request for the catalogue that the client is sent；The reading data
Request includes: the information of the catalogue and the identification information of data to be read；
The index text that the catalogue whether is stored in the memory determined for the information according to the catalogue for searching module
Part；If so, searching institute from the index file of the catalogue in the memory according to the identification information of the data to be read
State the index of data to be read, and according to being searched under index catalogue described in the disk of the data to be read to
Read data；
4th respond module is responded for reading to the client returned data.
13. a kind of data access equipment characterized by comprising processor and memory are stored with and can handled in memory
The computer program of device operation, processor realize that data as claimed in any one of claims 1 to 6 are visited when executing computer program
Ask method.
14. a kind of computer storage medium, is stored thereon with computer program, which is characterized in that the computer program is located
Reason device realizes data access method as claimed in any one of claims 1 to 6 when executing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910591781.9A CN110287201A (en) | 2019-07-02 | 2019-07-02 | Data access method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910591781.9A CN110287201A (en) | 2019-07-02 | 2019-07-02 | Data access method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110287201A true CN110287201A (en) | 2019-09-27 |
Family
ID=68021814
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910591781.9A Pending CN110287201A (en) | 2019-07-02 | 2019-07-02 | Data access method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110287201A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111046044A (en) * | 2019-12-13 | 2020-04-21 | 南京富士通南大软件技术有限公司 | High-reliability framework of distributed object storage system based on memory type database |
| CN111414335A (en) * | 2020-03-20 | 2020-07-14 | 深圳市凯沃尔电子有限公司 | Data file searching method and device |
| CN113127479A (en) * | 2019-12-31 | 2021-07-16 | 奇安信科技集团股份有限公司 | Method and device for loading Elasticissearch index, computer equipment and storage medium |
| WO2022036927A1 (en) * | 2020-08-19 | 2022-02-24 | 安徽鸿程光电有限公司 | File storage method and apparatus, device, and medium |
| WO2022141650A1 (en) * | 2021-01-04 | 2022-07-07 | Alibaba Group Holding Limited | Memory-frugal index design in storage engine |
| CN116954745A (en) * | 2023-05-25 | 2023-10-27 | 成都融见软件科技有限公司 | Target file partial loading system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090198902A1 (en) * | 2008-02-04 | 2009-08-06 | Apple Inc. | Memory mapping techniques |
| CN104636464A (en) * | 2015-02-09 | 2015-05-20 | 腾讯科技（深圳）有限公司 | Method and device for having access to file |
| CN105069091A (en) * | 2015-08-05 | 2015-11-18 | 北京思特奇信息技术股份有限公司 | HBASE based virtual file system and realization method therefor |
| CN108319634A (en) * | 2017-12-15 | 2018-07-24 | 创新科存储技术（深圳）有限公司 | The directory access method and apparatus of distributed file system |
-
2019
- 2019-07-02 CN CN201910591781.9A patent/CN110287201A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090198902A1 (en) * | 2008-02-04 | 2009-08-06 | Apple Inc. | Memory mapping techniques |
| CN104636464A (en) * | 2015-02-09 | 2015-05-20 | 腾讯科技（深圳）有限公司 | Method and device for having access to file |
| CN105069091A (en) * | 2015-08-05 | 2015-11-18 | 北京思特奇信息技术股份有限公司 | HBASE based virtual file system and realization method therefor |
| CN108319634A (en) * | 2017-12-15 | 2018-07-24 | 创新科存储技术（深圳）有限公司 | The directory access method and apparatus of distributed file system |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111046044A (en) * | 2019-12-13 | 2020-04-21 | 南京富士通南大软件技术有限公司 | High-reliability framework of distributed object storage system based on memory type database |
| CN113127479A (en) * | 2019-12-31 | 2021-07-16 | 奇安信科技集团股份有限公司 | Method and device for loading Elasticissearch index, computer equipment and storage medium |
| CN111414335A (en) * | 2020-03-20 | 2020-07-14 | 深圳市凯沃尔电子有限公司 | Data file searching method and device |
| WO2022036927A1 (en) * | 2020-08-19 | 2022-02-24 | 安徽鸿程光电有限公司 | File storage method and apparatus, device, and medium |
| WO2022141650A1 (en) * | 2021-01-04 | 2022-07-07 | Alibaba Group Holding Limited | Memory-frugal index design in storage engine |
| CN116954745A (en) * | 2023-05-25 | 2023-10-27 | 成都融见软件科技有限公司 | Target file partial loading system |
| CN116954745B (en) * | 2023-05-25 | 2024-02-09 | 成都融见软件科技有限公司 | Target file partial loading system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110287201A (en) | Data access method, device, equipment and storage medium | |
| CN103548003B (en) | Method and system for improving the client-side fingerprint cache of deduplication system backup performance | |
| CN109739815B (en) | File processing method, system, device, equipment and storage medium | |
| CN105701156B (en) | A kind of distributed file system management method and device | |
| US8321487B1 (en) | Recovery of directory information | |
| JP2006107446A (en) | Batch indexing system and method for network document | |
| CN110321325A (en) | File inode lookup method, terminal, server, system and storage medium | |
| CN108614837B (en) | File storage and retrieval method and device | |
| CN109766318B (en) | File reading method and device | |
| CN109101580A (en) | A kind of hot spot data caching method and device based on Redis | |
| CN109558378A (en) | File management method, device, equipment and storage medium | |
| CN105138275B (en) | A kind of Lustre memory system datas sharing method | |
| CN110399348A (en) | File deletes method, apparatus, system and computer readable storage medium again | |
| CN111198856A (en) | File management method and device, computer equipment and storage medium | |
| CN110505495A (en) | Multimedia resource takes out frame method, device, server and storage medium | |
| CN113448946B (en) | Data migration method and device and electronic equipment | |
| CN105912540A (en) | Method and device for synchronizing file system | |
| WO2020215580A1 (en) | Distributed global data deduplication method and device | |
| CN109947730A (en) | Metadata restoration methods, device, distributed file system and readable storage medium storing program for executing | |
| CN109522305B (en) | Big data deduplication method and device | |
| CN108090186A (en) | A kind of electric power data De-weight method on big data platform | |
| CN112286457B (en) | Object deduplication method and device, electronic equipment and machine-readable storage medium | |
| CN111752941B (en) | Data storage and access method and device, server and storage medium | |
| Nguyen et al. | An efficient similar image search framework for large-scale data on cloud | |
| US8495025B2 (en) | Foldering by stable query |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190927 |