CN103167036A - Raster data access method based on distributed multi-stage cache system - Google Patents
Raster data access method based on distributed multi-stage cache system Download PDFInfo
- Publication number
- CN103167036A CN103167036A CN2013100837242A CN201310083724A CN103167036A CN 103167036 A CN103167036 A CN 103167036A CN 2013100837242 A CN2013100837242 A CN 2013100837242A CN 201310083724 A CN201310083724 A CN 201310083724A CN 103167036 A CN103167036 A CN 103167036A
- Authority
- CN
- China
- Prior art keywords
- data
- raster data
- client
- server end
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to raster data access, in particular to access of a small piece of spatial raster data and discloses a raster data access method based on a distributed multi-stage cache system. According to the raster data access method, a server data storage system, a server cache system and a client are involved, and the client comprises a client cache and a user interface. The raster data access method comprises the following steps that: the user interface receives a raster data request sent by a user, and whether required data exist in the client cache is judged; the server client cache system judges whether data required by the raster data request exist in the server cache system; and the server data storage system returns the data required by the raster data request; and the client cache is updated, and the returned data are stored in the client cache system. The method has the advantages that the spatial raster data storage and cache performance are improved; the reliability and availability are high; and data are optimized and accessed, so that the data request can be responded efficiently.
Description
Technical field
The present invention relates to the raster data access, the particularly access of fritter space lattice data discloses a kind of raster data access method based on the distributed multi-stage buffer system.
Background technology
Raster data is the row and the array data that is listed as arrangement, has different gray scales or color by grid cell.Lattice structure is the Organization of Data that pixel (grid cell) array that equal and opposite in direction is evenly distributed, closely is connected comes representation space atural object or phenomenon to distribute.The simplest, spatial data structure the most intuitively, it with earth surface be divided into size, evenly, tight adjacent grid array.The position of each unit (pixel) is by its ranks number definition, and represented provider location lies in the grid line column position, each the data representation atural object in Organization of Data or the non-geometric attribute of phenomenon or point to the pointer of its attribute.For lattice structure: the some entity is represented by a grid cell, raster cell; Raster data line entity is represented by the adjacent cells pixel that connects bunchiness on certain orientation; Face entity (zone) is represented by the set of blocks of the adjacent cells pixel with same alike result.The spatial data that electronic chart etc. comprise a large amount of geography information tends to store and access with the form of raster data.In prior art, because the data volume of raster data is huge, needed memory space is larger, the system resource that the data access process consumes is larger, the data access performance is not high, cause decline and the response of network transmission efficiency to be delayed, the reliability of the transfer of data of client and server is not high.
Summary of the invention
The present invention is directed to the not high shortcoming of data memory property in prior art, a kind of novel raster data access method based on the distributed multi-stage buffer system towards the fritter spatial data is provided.
For achieving the above object, the present invention can take following technical proposals:
Based on the raster data access method of distributed multi-stage buffer system, comprise servers' data storage system, server end caching system, client, described client comprises client-cache and user interface, concrete steps comprise:
1) described user interface receives the request of Client-initiated raster data, and described client is searched the needed data of described raster data request in described client-cache, if find these data directly the data that find returned to the user; If do not find needed data, described client is sent to described server end caching system with described raster data request;
2) after the service end buffer system receives described raster data request, if the needed data of described raster data request are Already in the service end buffer system, these data are returned to user and execution in step 4, otherwise described raster data request is sent to distributed memory system and execution in step 3;
3) after described servers' data storage system receives described raster data request, the needed data of described raster data request are returned to client and execution in step 4, step 5, if described servers' data storage system does not find the needed data of described raster data request returns to error message;
4) upgrade described client-cache, the data of returning are stored to described client-cache;
5) upgrade described server end caching system, the data of returning are stored to described server end caching system.
As preferably, also comprise buffer update mechanism, described buffer update mechanism comprises the buffer update mechanism of client-cache and the buffer update mechanism of server end caching system, if client-cache inadequate buffer space, trigger the buffer update mechanism of client-cache, if server end caching system inadequate buffer space, the buffer update that triggers the server end caching system is machine-processed; Described buffer update mechanism comprises, uses the data block in the superseded buffer memory of least recently used algorithm.
As preferably, also comprise horizontal prestrain and vertical prestrain, described horizontal prestrain step comprises, in described step 1, the raster data request that described client sends to described server end caching system also comprises the adjacent cells unit of the needed data of described raster data request; Described vertical prestrain step comprises, in described step 1, the raster data request that described client sends to described server end caching system also comprises the grid cell corresponding with the needed data of described raster data request in next figure layer.
As preferably, described user interface be the POSIX interface or with the interface of POSIX compatibility.
As preferably, the described servers' data storage system distributed storage cluster that many number of units form according to storage server of serving as reasons.
As preferably, described data storage server is the data storage server based on the HBase database.
As preferably, the described server end caching system distributed caching cluster that many caching servers consist of of serving as reasons.
As preferably, described caching server is the caching server based on the Redis database.
The present invention has significant technique effect owing to having adopted above technical scheme:
By in conjunction with distributed memory system and distributed cache system, compared to existing raster data access mode, data access method of the present invention is when access fritter spatial data (size is no more than 1M), has more efficient data access efficiency, data-storage system response data request quickly.Data renewal mechanism has guaranteed that caching data block has higher hit rate, and the buffer memory that the prestrain mechanism of data makes server end is resident hot spot data more exactly, has further improved the hit rate of data buffering.The secondary data caching mechanism that lays respectively at the client and server end has further improved the efficient of data storages.In addition, data storing and caching system adopt distributed structure/architecture, have improved the concurrent processing quantity of request, have greatly improved the transmission rate of data.
Generally speaking, the space lattice data that the present invention is directed to fritter are optimized, and have improved the efficient of the space lattice data of system access fritter.
Description of drawings
Fig. 1 is the logical construction schematic diagram of embodiment 1 described distributed system.
Fig. 2 is the deployment architecture schematic diagram of embodiment 1 described distributed system.
Fig. 3 is the method flow schematic diagram of embodiment 1.
Fig. 4 is the schematic diagram data of the prestrain mechanism of embodiment 1.
Fig. 5 is the schematic diagram of the experimental result of contrast test 1.
Embodiment
The present invention is described in further detail below in conjunction with embodiment.
raster data access method based on the distributed multi-stage buffer system, the distributed system of using such as Fig. 1, shown in 2, this distributed system is divided into three layers physically, comprise servers' data storage system 1, server end caching system 2, client 3, the bottom is servers' data storage system 1, server end caching system 2, the superiors, be namely and 3 layers of the clients of user interactions, the distributed storage cluster that the servers' data storage system 1 here is comprised of according to storage server many number of units, data storage server is the data storage server based on the HBase database.Institute's server end caching system 2 distributed caching cluster that many caching servers consist of of serving as reasons, caching server is the server based on the Redis database.This distributed system is from being divided in logic four layers, and the bottom is the physical layer of data storage, and namely the servers' data storage system 1, it is the cache layer of server end on physical layer, be server end caching system 2, then on be client-cache, the superiors are the user interfaces that are exposed to the user.Described client 3 comprises client-cache and user interface.Use POSIX in the present embodiment or with the interface of POSIX compatibility as user interface.Concrete steps comprise, as shown in Figure 3:
1) described user interface receives the request of Client-initiated raster data, and described client 3 is searched the needed data of described raster data request in described client-cache, if find these data directly the data that find returned to the user; If do not find needed data, described client 3 is sent to described server end caching system 2 with described raster data request;
2) after server end caching system 2 receives described raster data request, if the needed data of described raster data request are Already in server end caching system 2, these data are returned to user and execution in step 4, otherwise described raster data request is sent to servers' data storage system 1 and execution in step 3;
3) after described servers' data storage system 1 receives described raster data request, the needed data of described raster data request are returned to client and execution in step 4, step 5, if described servers' data storage system 1 does not find the needed data of described raster data request returns to error message;
4) upgrade described client-cache, the data of returning are stored to described client-cache;
5) upgrade described server end caching system 2, the data of returning are stored to described server end caching system 2.
In addition, raster data access method of the present invention also comprises buffer update mechanism and prestrain mechanism.
Buffer update mechanism comprises the buffer update mechanism of client-cache and the buffer update mechanism of server end caching system 2, if client-cache insufficient space, trigger the buffer update mechanism of client-cache, if server end caching system 2 inadequate buffer spaces, the buffer update that triggers server end caching system 2 is machine-processed; Described buffer update mechanism comprises, uses the data block in the superseded buffer memory of least recently used algorithm.
Prestrain mechanism comprises horizontal prestrain and vertical prestrain, described horizontal prestrain step comprises, in described step 1, the raster data request that described client 3 sends to described server end caching system 2 also comprises the adjacent cells unit of the needed data of described raster data request; Because tending to carry out layering according to different resolution, the space lattice data store, one group of space lattice data tends to include a plurality of figure layers in up and down, exist corresponding relation between the grid cell of these figure layers, as shown in Figure 5, described vertical prestrain step comprises, in step 1, the raster data request that described client 3 sends to described server end caching system 2 also comprises the grid cell corresponding with the needed data of described raster data request in next figure layer.
Due to accumulation layer itself, and have overlapping between accumulation layer and cache layer, this has just determined corresponding to any a image data, many parts of storages are arranged in system, because the data block in client-cache and server end buffer memory is hot spot data, mistake easily appears, by being set, client-cache and server end buffer memory improved the redundancy of data in access procedure here, when mechanical disorder occurring, also can the normal response request of data.
Test data is for being divided into three groups, respectively 900MB data (comprising 3741 pictures), 4GB data (comprising 14180 pictures), 16G picture (comprising 55701 pictures), the technical scheme that Application Example 1 is put down in writing detects respectively response time of data access request.
Contrast test is tested in three kinds of situations respectively, is respectively without cache mode, only opens the server end buffer memory and opens simultaneously server and client side's cache mode, and acquired results as shown in Figure 5.
Can find out by test, use the method for the invention, there has been great reduction in system to the response time of data access request, and the performance of system is enhanced.
In a word, the above is only preferred embodiment of the present invention, and all equalizations of doing according to the present patent application the scope of the claims change and modify, and all should belong to the covering scope of patent of the present invention.
Claims (8)
1. raster data access method based on the distributed multi-stage buffer system, it is characterized in that, comprise servers' data storage system (1), server end caching system (2), client (3), described client (3) comprises client-cache and user interface, and concrete steps comprise:
1) described user interface receives the request of Client-initiated raster data, and described client (3) is searched the needed data of described raster data request in described client-cache, if find these data directly the data that find returned to the user; If do not find needed data, described client (3) is sent to described server end caching system (2) with described raster data request;
2) after server end caching system (2) receives described raster data request, if the needed data of described raster data request are Already in server end caching system (2), these data are returned to user and execution in step 4, otherwise described raster data request is sent to servers' data storage system (1) and execution in step 3;
3) after described servers' data storage system (1) receives described raster data request, the needed data of described raster data request are returned to client and execution in step 4, step 5, if described servers' data storage system (1) does not find the needed data of described raster data request returns to error message;
4) upgrade described client-cache, the data of returning are stored to described client-cache;
5) upgrade described server end caching system (2), the data of returning are stored to described server end caching system (2).
2. the raster data access method based on the distributed multi-stage buffer system according to claim 1, it is characterized in that, also comprise buffer update mechanism, described buffer update mechanism comprises the buffer update mechanism of client-cache and the buffer update mechanism of server end caching system (2), if client-cache insufficient space, trigger the buffer update mechanism of client-cache, if server end caching system (2) inadequate buffer space, the buffer update that triggers server end caching system (2) is machine-processed; Described buffer update mechanism comprises, uses the data block in the superseded buffer memory of least recently used algorithm.
3. the raster data access method based on the distributed multi-stage buffer system according to claim 1, it is characterized in that, also comprise horizontal prestrain and vertical prestrain, described horizontal prestrain step comprises, in described step 1, the raster data request that described client (3) sends to described server end caching system (2) also comprises the adjacent cells unit of the needed data of described raster data request; Described vertical prestrain step comprises, in described step 1, the raster data request that described client (3) sends to described server end caching system (2) also comprises the grid cell corresponding with the needed data of described raster data request in next figure layer.
4. the raster data access method based on the distributed multi-stage buffer system according to claim 1, is characterized in that, described user interface be the POSIX interface or with the interface of POSIX compatibility.
5. the raster data access method based on the distributed multi-stage buffer system according to claim 1, is characterized in that, described servers' data storage system (1) the distributed storage cluster that many number of units form according to storage server of serving as reasons.
6. the raster data access method based on the distributed multi-stage buffer system according to claim 5, is characterized in that, described data storage server is the data storage server based on the HBase database.
7. the raster data access method based on the distributed multi-stage buffer system according to claim 1, is characterized in that, described server end caching system (2) the distributed caching cluster that many caching servers consist of of serving as reasons.
8. the raster data access method based on the distributed multi-stage buffer system according to claim 7, is characterized in that, described caching server is the caching server based on the Redis database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013100837242A CN103167036A (en) | 2013-01-28 | 2013-01-28 | Raster data access method based on distributed multi-stage cache system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013100837242A CN103167036A (en) | 2013-01-28 | 2013-01-28 | Raster data access method based on distributed multi-stage cache system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103167036A true CN103167036A (en) | 2013-06-19 |
Family
ID=48589772
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2013100837242A Pending CN103167036A (en) | 2013-01-28 | 2013-01-28 | Raster data access method based on distributed multi-stage cache system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103167036A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103473267A (en) * | 2013-08-09 | 2013-12-25 | 深圳市中科新业信息科技发展有限公司 | Data storage query method and system |
| CN103810269A (en) * | 2014-01-28 | 2014-05-21 | 浙江大学 | Map searching method based on large memory |
| CN104378452A (en) * | 2013-08-14 | 2015-02-25 | 阿里巴巴集团控股有限公司 | Method, device and system for domain name resolution |
| CN104461929A (en) * | 2013-09-23 | 2015-03-25 | ***股份有限公司 | Distributed type data caching method based on interceptor |
| CN105095282A (en) * | 2014-05-14 | 2015-11-25 | 腾讯科技（深圳）有限公司 | Method, device and system for updating cache data |
| CN105589664A (en) * | 2015-12-29 | 2016-05-18 | 四川中电启明星信息技术有限公司 | Virtual storage high-speed transmission method |
| CN109871367A (en) * | 2019-02-28 | 2019-06-11 | 江苏实达迪美数据处理有限公司 | A kind of distributed cold and heat data separation method based on Redis and HBase |
| CN109935230A (en) * | 2019-04-01 | 2019-06-25 | 北京宇航***工程研究所 | A kind of survey hair password monitoring system and method based on voice driven |
| CN109981740A (en) * | 2019-02-25 | 2019-07-05 | 湖北金拓维信息技术有限公司 | A kind of distribution image data multi-level buffer method |
| CN110427446A (en) * | 2019-08-02 | 2019-11-08 | 武汉中地数码科技有限公司 | A kind of huge image data service release quickly and browsing method and system |
| CN111928862A (en) * | 2020-08-10 | 2020-11-13 | 廊坊和易生活网络科技股份有限公司 | Method for constructing semantic map on line by fusing laser radar and visual sensor |
| CN112905129A (en) * | 2021-05-06 | 2021-06-04 | 蚂蚁金服（杭州）网络技术有限公司 | Method and device for eliminating cache memory block and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1852318A (en) * | 2006-04-19 | 2006-10-25 | 华中科技大学 | Distributed multi-stage buffer storage system suitable for object network storage |
| CN101692229A (en) * | 2009-07-28 | 2010-04-07 | 武汉大学 | Self-adaptive multilevel cache system for three-dimensional spatial data based on data content |
| CN102355490A (en) * | 2011-08-23 | 2012-02-15 | 武汉大学 | Spatial information cluster cache pre-fetching method for network spatial information service system |
-
2013
- 2013-01-28 CN CN2013100837242A patent/CN103167036A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1852318A (en) * | 2006-04-19 | 2006-10-25 | 华中科技大学 | Distributed multi-stage buffer storage system suitable for object network storage |
| CN101692229A (en) * | 2009-07-28 | 2010-04-07 | 武汉大学 | Self-adaptive multilevel cache system for three-dimensional spatial data based on data content |
| CN102355490A (en) * | 2011-08-23 | 2012-02-15 | 武汉大学 | Spatial information cluster cache pre-fetching method for network spatial information service system |
Non-Patent Citations (1)
| Title |
|---|
| 黄振: "基于 WebGIS 的电信资源***性能优化研究与应用", 《中国优秀硕士学位论文全文数据库 信息科技辑》, no. 5, 15 May 2011 (2011-05-15) * |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103473267A (en) * | 2013-08-09 | 2013-12-25 | 深圳市中科新业信息科技发展有限公司 | Data storage query method and system |
| CN103473267B (en) * | 2013-08-09 | 2016-11-16 | 深圳市中科新业信息科技发展有限公司 | Data store query method and system |
| CN104378452A (en) * | 2013-08-14 | 2015-02-25 | 阿里巴巴集团控股有限公司 | Method, device and system for domain name resolution |
| CN104461929A (en) * | 2013-09-23 | 2015-03-25 | ***股份有限公司 | Distributed type data caching method based on interceptor |
| CN104461929B (en) * | 2013-09-23 | 2018-03-23 | ***股份有限公司 | Distributed data cache method based on blocker |
| CN103810269A (en) * | 2014-01-28 | 2014-05-21 | 浙江大学 | Map searching method based on large memory |
| CN105095282A (en) * | 2014-05-14 | 2015-11-25 | 腾讯科技（深圳）有限公司 | Method, device and system for updating cache data |
| CN105095282B (en) * | 2014-05-14 | 2019-12-24 | 腾讯科技（深圳）有限公司 | Cache data updating method, device and system |
| CN105589664B (en) * | 2015-12-29 | 2018-07-31 | 四川中电启明星信息技术有限公司 | Virtual memory high speed transmission method |
| CN105589664A (en) * | 2015-12-29 | 2016-05-18 | 四川中电启明星信息技术有限公司 | Virtual storage high-speed transmission method |
| CN109981740A (en) * | 2019-02-25 | 2019-07-05 | 湖北金拓维信息技术有限公司 | A kind of distribution image data multi-level buffer method |
| CN109871367A (en) * | 2019-02-28 | 2019-06-11 | 江苏实达迪美数据处理有限公司 | A kind of distributed cold and heat data separation method based on Redis and HBase |
| CN109935230A (en) * | 2019-04-01 | 2019-06-25 | 北京宇航***工程研究所 | A kind of survey hair password monitoring system and method based on voice driven |
| CN110427446A (en) * | 2019-08-02 | 2019-11-08 | 武汉中地数码科技有限公司 | A kind of huge image data service release quickly and browsing method and system |
| CN110427446B (en) * | 2019-08-02 | 2023-05-16 | 武汉中地数码科技有限公司 | Method and system for rapidly publishing and browsing mass image services |
| CN111928862A (en) * | 2020-08-10 | 2020-11-13 | 廊坊和易生活网络科技股份有限公司 | Method for constructing semantic map on line by fusing laser radar and visual sensor |
| CN111928862B (en) * | 2020-08-10 | 2023-11-21 | 廊坊和易生活网络科技股份有限公司 | Method for on-line construction of semantic map by fusion of laser radar and visual sensor |
| CN112905129A (en) * | 2021-05-06 | 2021-06-04 | 蚂蚁金服（杭州）网络技术有限公司 | Method and device for eliminating cache memory block and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103167036A (en) | Raster data access method based on distributed multi-stage cache system | |
| US11620187B2 (en) | Distributing data on distributed storage systems | |
| US9514015B2 (en) | Efficient data reads from distributed storage systems | |
| US9535790B2 (en) | Prioritizing data reconstruction in distributed storage systems | |
| US20180285167A1 (en) | Database management system providing local balancing within individual cluster node | |
| CN107168657B (en) | Virtual disk hierarchical cache design method based on distributed block storage | |
| CN104115133B (en) | For method, system and the equipment of the Data Migration for being combined non-volatile memory device | |
| CN101135994B (en) | Method and apparatus for dividing buffer memory space and buffer memory controller thereof | |
| CN104317736B (en) | A kind of distributed file system multi-level buffer implementation method | |
| CN106909644A (en) | A kind of multistage tissue and indexing means towards mass remote sensing image | |
| CN105205126A (en) | Storage method and storage system for map tiles | |
| CN113672175A (en) | Distributed object storage method, device and equipment and computer storage medium | |
| WO2007137133A2 (en) | Database partitioning by virtual partitions | |
| CN101567003A (en) | Method for managing and allocating resource in parallel file system | |
| CN103116552A (en) | Method and device for distributing storage space in distributed type storage system | |
| CN106909317A (en) | The data storage on memory node | |
| CN112380302B (en) | Thermodynamic diagram generation method and device based on track data, electronic equipment and storage medium | |
| CN102982182A (en) | Data storage planning method and device | |
| CN104111898A (en) | Hybrid storage system based on multidimensional data similarity and data management method | |
| US11010301B2 (en) | Method, apparatus, and computer program product for providing cache service | |
| US8799572B2 (en) | Sliding-window multi-class striping | |
| CN102279808A (en) | Method and device for managing video memory of embedded equipment | |
| CN109597903A (en) | Image file processing apparatus and method, document storage system and storage medium | |
| Zhong et al. | A novel method to manage very large raster data on distributed key-value storage system | |
| CN107450860A (en) | A kind of map file pre-head method based on distributed storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20130619 |