CN114372034A - Access method based on remote sensing image map service - Google Patents
Access method based on remote sensing image map service Download PDFInfo
- Publication number
- CN114372034A CN114372034A CN202111607495.0A CN202111607495A CN114372034A CN 114372034 A CN114372034 A CN 114372034A CN 202111607495 A CN202111607495 A CN 202111607495A CN 114372034 A CN114372034 A CN 114372034A
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- sensing image
- key
- image
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/56—Information retrieval; Database structures therefor; File system structures therefor of still image data having vectorial format
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses an access method based on remote sensing image map service, which comprises the following steps: obtaining remote sensing image data, and constructing a remote sensing image pyramid by using a MapReduce parallel model; the remote sensing image pyramid comprises a plurality of hierarchical structures with different resolutions, and the image layers of the same hierarchy are formed by splicing grids cut by a plurality of remote sensing images with the same resolution; partitioning the remote sensing image raster data in each hierarchical structure according to hash codes, and simultaneously performing dimension reduction processing; and performing distributed storage on the grid after the dimension reduction processing in a Key-Value Key Value pair mode. The client requests to obtain the information of the image file to be accessed, and the Block information corresponding to each file is obtained through a Key-Value pair index; and reading Block in parallel, and obtaining the raster image layer required to be read by the client. The invention can effectively improve the warehousing efficiency, can fetch raster data of different levels under a multi-user high-concurrency scene, and provides an efficient index to read and query the raster data.
Description
Technical Field
The invention relates to the technical field of remote sensing, in particular to an access method based on remote sensing image map service.
Background
With the rapid development of the terrestrial satellite remote sensing technology, remote sensing images are generally applied to various fields of earth observation, and good effects are achieved. The earth observation refers to a process of acquiring information, processing and generating a product by taking the earth as an observation object and relying on space platforms such as satellites, space shuttle airplanes and unmanned planes and by using technical means such as visible light, infrared, hyperspectrum and microwave. The remote sensing images obtained by the space platform are wide in source, various and complex in structure, and how to store and manage the remote sensing images is the key for providing online service for earth observation application and releasing the data value of the remote sensing images.
There are three common remote sensing image storage management systems, the first is based on distributed storage system to organize the remote sensing image block, the specific method is: firstly, performing layered block organization on an image so that a user can quickly access image blocks with a specific resolution level and a specific spatial region range according to needs; then, uniformly coding each resolution image block, mapping the multi-dimensional image block space to a one-dimensional space, and quickly searching the obtained one-dimensional code by adopting a secondary index structure; and the second method is to comprehensively use storage environments such as a distributed file system, an online disk array and an offline disk and the like to store and manage remote sensing data resources based on a big data architecture. The third is to use the traditional relational database to store large binary data or directly store the binary data in a magnetic disk form, and the main problem is slow reading and writing of remote sensing data.
The requests of online users for image blocks are mostly concentrated in a small local area of an image coverage space range, the local area requests show that pixels acquired through a remote sensing storage system can be as small as a single pixel, but the existing image storage system forces to access the image blocks, so that a lot of unnecessary image blocks are stored in a redundant manner, huge waste of storage space is caused, extra processing burden is brought, and the cost of network transmission is huge.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide an access method based on remote sensing image map service, which can effectively store remote sensing image data into a distributed data storage database, improve the storage efficiency, can fetch raster data of different levels under a multi-user high-concurrency scene, and provide an efficient index to read and query the raster data.
In order to achieve the purpose, the technical scheme of the invention is as follows:
an access method based on remote sensing image map service comprises the following steps:
1) obtaining remote sensing image data, and constructing a remote sensing image pyramid by using a MapReduce parallel model; the remote sensing image pyramid comprises a plurality of hierarchical structures with different resolutions;
2) partitioning the remote sensing image data in each hierarchical structure according to the codes after hash, simultaneously performing dimensionality reduction, and storing the grids subjected to dimensionality reduction in a distributed storage system (HDFS) in a Key-Value Key Value pair mode;
3) and reading the remote sensing image through a distributed storage system (HDFS) in a multi-client high-concurrency scene.
Step 1) a remote sensing image pyramid construction module for constructing a remote sensing image pyramid by using a MapReduce parallel model; the remote sensing image pyramid comprises a plurality of hierarchical structures with different resolutions; and index calculation is used for partitioning the remote sensing image data in each hierarchical structure according to the generated hash code so as to calculate an index value.
And 2) taking the image Block information corresponding to each raster after storage as a Value attribute to form a Key-Value Key Value pair, extracting the row and column numbers of all raster pixels by using the remote sensing image data storage method, reducing the row and column numbers into one-dimensional character strings by hash coding, and taking the pixel Value corresponding to each raster as the Value attribute to form the Key-Value Key Value pair.
And step 3) after each client accesses for the first time when reading the image data, all file indexes stored in the Block are obtained, and after the first access, the index structure under the Block is also stored in the client and is cached in the memory.
And 3) the equipment concurrently acquires the image files and obtains Block information corresponding to the corresponding files, the equipment calculates the number of blocks which can be read in parallel in the same time period, judges whether the blocks belong to the same DataNode, stores the blocks in a set to be accessed in parallel and reads the blocks in the set in parallel. And judging whether all blocks of the image file are read or not, and if not, continuously reading the next image file until all the image files are completely accessed.
Drawings
Fig. 1 is a flowchart of a method for storing remote sensing image data.
Fig. 2 is a schematic diagram of remote sensing image pyramid construction.
FIG. 3 is a flow chart of metadata Block for multi-client parallel acquisition of remote sensing images.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a flowchart of a method for storing remote sensing image data according to an embodiment of the present invention, including:
obtaining remote sensing image data, and constructing a remote sensing image pyramid by using a MapReduce parallel model; the remote sensing image pyramid comprises a plurality of hierarchical structures with different resolutions;
partitioning the remote sensing image data in each hierarchical structure according to Hash codes to calculate key (index) values;
carrying out dimension reduction processing on the grids of each remote sensing image data by using a space filling curve;
and performing distributed storage on the grid after dimension reduction processing in the form of key-value key value pairs.
Specifically, referring to fig. 2, fig. 2 is a schematic structural diagram of a remote sensing image pyramid in the remote sensing image data storage method according to the embodiment of the present invention; the image pyramid is a multilayer pyramid structure which is constructed by adopting different resolutions and different dimensions to store and display according to user requirements under the same spatial reference and adopting a resampling method according to a certain rule, wherein the multilayer pyramid structure is formed by the steps that the data size is from small to large and the resolution is from coarse to fine. The remote sensing image pyramid structure is mainly used for progressive image transmission and image coding, is a common layered data structure, is very suitable for multilayer resolution organization of remote sensing data or raster data, and is also a lossy compression mode of the remote sensing data or the raster data.
The remote sensing image pyramid is constructed by utilizing a MapReduce parallel model, and the method specifically comprises the following steps:
specifically, the MapReduce parallel model structure uses a MapReduce distributed parallel operation framework technology to automatically decompose complex data on a large-scale cluster into hundreds of thousands of small data sets by using a Map function, each (or several) data set is respectively distributed to cluster nodes for processing by balance to generate an intermediate result, and then the nodes are merged by a Reduce function to form a final result.
The dimension reduction index of the remote sensing image is used for parallel computation, and the parallel computation process comprises the following steps:
reading the remote sensing data after the dimension reduction index from a distributed non-relational database of the raster space data conversion database;
performing parallel computation on the read data;
and writing the data after parallel computation into the distributed non-relational database again by using the raster space data conversion library.
Reading the remote sensing data after the dimension reduction index in a raster space data conversion library (namely a converter between the traditional remote sensing image and the data of the dimension reduction index) distributed non-relational database so as to perform parallel computation on the remote sensing image data. The obtained remote sensing image data still with dimension reduction index is written into the distributed database again by utilizing the raster space data conversion library.
Obtaining image file information to be accessed according to a client request, and obtaining Block information corresponding to each file through a Key-Value pair index;
calculating to obtain the number of blocks which can be read in parallel in the same time period
Judging whether the blocks belong to the same DataNode, and storing the blocks to a set to be accessed in parallel;
reading blocks in the set in parallel;
and judging whether all blocks of the image file are read or not, if not, continuing to the step 3, and if so, continuing to read the next image file until all the image files are completely accessed.
The embodiments in the above description can be further combined or replaced, and the embodiments are only for describing the preferred embodiments of the present invention, and do not limit the concept and scope of the present invention, and those skilled in the art can make various changes and modifications to the technical solution of the present invention without departing from the design idea of the present invention
Furthermore, all fall within the scope of protection of the present invention. The scope of the invention is given by the appended claims and any equivalents thereof.
Claims (5)
1. An access method based on remote sensing image map service is characterized by comprising the following steps:
1) obtaining remote sensing image data, and constructing a remote sensing image pyramid by using a MapReduce parallel model; the remote sensing image pyramid comprises a plurality of hierarchical structures with different resolutions;
2) partitioning the remote sensing image data in each hierarchical structure according to the codes after hash, simultaneously performing dimensionality reduction, and storing the grids subjected to dimensionality reduction in a distributed storage system (HDFS) in a Key-Value Key Value pair mode;
3) and reading the remote sensing image through a distributed storage system (HDFS) in a multi-client high-concurrency scene.
2. The method of claim 1, wherein: step 1) a remote sensing image pyramid construction module for constructing a remote sensing image pyramid by using a MapReduce parallel model; the remote sensing image pyramid comprises a plurality of hierarchical structures with different resolutions; and index calculation is used for partitioning the remote sensing image data in each hierarchical structure according to the generated hash code so as to calculate an index value.
3. The method of claim 1, wherein: and 2) taking the image Block information corresponding to each raster after storage as a Value attribute to form a Key-Value Key Value pair, extracting the row and column numbers of all raster pixels by using the remote sensing image data storage method, reducing the row and column numbers into one-dimensional character strings by hash coding, and taking the pixel Value corresponding to each raster as the Value attribute to form the Key-Value Key Value pair.
4. The method of claim 1, wherein: and step 3) after each client accesses for the first time when reading the image data, all file indexes stored in the Block are obtained, and after the first access, the index structure under the Block is also stored in the client and is cached in the memory.
5. The method of claim 1, wherein: step 3) the equipment concurrently obtains image files and obtains Block information corresponding to the corresponding files, the equipment calculates the number of blocks which can be read in parallel in the same time period, the equipment judges whether the blocks belong to the same DataNode, stores the blocks in a set to be accessed in parallel and reads the blocks in the set in parallel;
and judging whether all blocks of the image file are read or not, and if not, continuously reading the next image file until all the image files are completely accessed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111607495.0A CN114372034A (en) | 2021-12-27 | 2021-12-27 | Access method based on remote sensing image map service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111607495.0A CN114372034A (en) | 2021-12-27 | 2021-12-27 | Access method based on remote sensing image map service |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114372034A true CN114372034A (en) | 2022-04-19 |
Family
ID=81142910
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111607495.0A Pending CN114372034A (en) | 2021-12-27 | 2021-12-27 | Access method based on remote sensing image map service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114372034A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117931810A (en) * | 2024-03-21 | 2024-04-26 | 成都歧明通信息科技有限公司 | Structured management method and system for spatial image data |
-
2021
- 2021-12-27 CN CN202111607495.0A patent/CN114372034A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117931810A (en) * | 2024-03-21 | 2024-04-26 | 成都歧明通信息科技有限公司 | Structured management method and system for spatial image data |
| CN117931810B (en) * | 2024-03-21 | 2024-05-31 | 成都歧明通信息科技有限公司 | Structured management method and system for spatial image data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110442444B (en) | Massive remote sensing image-oriented parallel data access method and system | |
| CN110599490B (en) | Remote sensing image data storage method and system | |
| CN109992636B (en) | Space-time coding method, space-time index and query method and device | |
| CN111291016B (en) | Hierarchical hybrid storage and indexing method for massive remote sensing image data | |
| CN112115198B (en) | Urban remote sensing intelligent service platform | |
| Li et al. | A spatiotemporal indexing approach for efficient processing of big array-based climate data with MapReduce | |
| CN111125392B (en) | Remote sensing image storage and query method based on matrix object storage mechanism | |
| CN104820714B (en) | Magnanimity tile small documents memory management method based on hadoop | |
| CN106909644A (en) | A kind of multistage tissue and indexing means towards mass remote sensing image | |
| CN106933833A (en) | A kind of positional information method for quickly querying based on Spatial Data Index Technology | |
| CN108009265B (en) | Spatial data indexing method in cloud computing environment | |
| Jing et al. | An improved distributed storage and query for remote sensing data | |
| Li et al. | Integration and management of massive remote-sensing data based on GeoSOT subdivision model | |
| CN112380302B (en) | Thermodynamic diagram generation method and device based on track data, electronic equipment and storage medium | |
| CN116860905B (en) | Space unit coding generation method of city information model | |
| CN110968636A (en) | Multi-dimensional big data analysis and processing system for earthquake early warning | |
| Deibe et al. | Big data storage technologies: a case study for web-based LiDAR visualization | |
| CN113742505B (en) | Mass synthetic aperture radar interferometry (InSAR) data online visualization method | |
| CN114372034A (en) | Access method based on remote sensing image map service | |
| Liang et al. | Research on distributed storage of big data based on hbase remote sensing image | |
| Albrecht et al. | Pairs (Re) loaded: system design & benchmarking for scalable geospatial applications | |
| CN114168697A (en) | Tile data publishing method and device | |
| Yao et al. | LandQ v1: A GIS cluster-based management information system for arable land quality big data | |
| Rueda-Ruiz et al. | SPSLiDAR: towards a multi-purpose repository for large scale LiDAR datasets | |
| CN114079496A (en) | Space-time gridding satellite-borne data interaction method and device thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |