CN115080774B - Remote sensing image warehousing system and method based on available domain - Google Patents
Remote sensing image warehousing system and method based on available domain Download PDFInfo
- Publication number
- CN115080774B CN115080774B CN202210852457.XA CN202210852457A CN115080774B CN 115080774 B CN115080774 B CN 115080774B CN 202210852457 A CN202210852457 A CN 202210852457A CN 115080774 B CN115080774 B CN 115080774B
- Authority
- CN
- China
- Prior art keywords
- data
- image data
- remote sensing
- metadata
- sensing image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a remote sensing image warehousing system and method based on an available domain, relating to the technical field of remote sensing data warehousing, wherein the system comprises: the data acquisition module is configured to acquire remote sensing image data; the data classification module is configured to classify the remote sensing image data into a plurality of categories and classify the remote sensing image data according to different file forms; the data uploading module is configured to upload the remote sensing image data meeting the requirements to a database based on a set transmission mode; a data storage module configured to support multi-level storage; and the data retrieval module is configured to perform query retrieval, including fuzzy retrieval and accurate retrieval, on the remote sensing image data according to the indexes. The warehousing system and the warehousing method provided by the invention can greatly increase the utilization rate of the remote sensing image, avoid the resource waste of the remote sensing image data, and have high warehousing speed and high retrieval efficiency.
Description
Technical Field
The invention relates to the technical field of remote sensing data warehousing, in particular to a remote sensing image warehousing system and method based on an available domain.
Background
A plurality of remote sensing satellites which comprise meteorology, land, ocean, environment and the like are emitted in China, a series of industrial satellite systems and combined constellations are combined with various aerial photography technologies, massive multi-source remote sensing data with multiple resolutions, multiple types and high coverage are formed, and a data basis is provided for remote sensing popularization application and industrial service. The remote sensing has the characteristics of fast acquisition and updating, multi-resolution, rich information, multiple data types, strong monitoring capability and the like, and has wide industrial application prospect.
Because the image data is bulky and complex, in the prior art, the image data is generally compressed or divided into different image blocks, and then the image blocks are distributed to different processing nodes for processing. However, the existing remote sensing image storage and warehousing system cannot achieve high data utilization rate, the remote sensing image data resource is wasted, and data organization and storage are not performed from the usability angle of the remote sensing image.
Therefore, there is a need for a remote sensing image warehousing system with high utilization rate to solve the existing problems.
Disclosure of Invention
Based on the technical problems, the invention provides a remote sensing image warehousing system and method based on available domain, which can efficiently and quickly store and query the remote sensing image by storing image data, metadata, available domain data and the like and creating two indexes, and can retrieve and obtain the remote sensing image available for a specific task by utilizing the characteristics of the available domain data, thereby greatly increasing the application efficiency of the remote sensing image and increasing the utilization rate of remote sensing image data resources.
In order to achieve the above technical object, the present invention provides a remote sensing image warehousing system based on an available domain, the system comprising:
the data acquisition module is configured to acquire remote sensing image data and support multiple data acquisition modes, wherein the remote sensing image data comprises image data and metadata;
the data classification module is configured to classify the remote sensing image data into a plurality of categories and classify the remote sensing image data according to different file forms, wherein the categories comprise image data, metadata, quality data and available domain data, and the available domain data are data consisting of pixels which are judged to have availability in the image data;
the data uploading module is configured to upload the remote sensing image data meeting the requirements to a database based on a set transmission mode;
the data storage module is configured to support multilevel storage and store the remote sensing image data according to the categories;
and the data retrieval module is configured to perform query retrieval, including fuzzy retrieval and accurate retrieval, on the remote sensing image data according to the indexes.
In an embodiment of the present invention, the available field data includes tag information, and the tag information is information for marking that one pixel is available.
In an embodiment of the present invention, the data classification module includes:
the data preliminary classification unit is configured to preliminarily divide the acquired remote sensing image data into image data and metadata and store the image data and the metadata in different file forms;
a first data detection unit configured to perform preliminary quality detection on image data in the remote sensing image data based on a quality detection item, form the quality data, and add the quality data into the category;
the second data detection unit is configured to extract metadata of the remote sensing image data, compare the extracted metadata with locally stored metadata, judge the integrity of the metadata, form an evaluation result of the availability of the metadata, and add the evaluation result into the metadata;
and the data evaluation unit is configured to evaluate the quality data based on evaluation criteria, judge available areas according to evaluation results, and arrange the available areas to obtain the available domain data of the remote sensing image data.
In an embodiment of the invention, in the data evaluation unit, the judgment criterion is whether the quality data reaches a preset value of the quality detection item.
In an embodiment of the present invention, the result of the integrity of the metadata includes integrity of the metadata, missing of the metadata, damage of the metadata, and confusion of the metadata.
In a specific embodiment of the present invention, the data storage module is further configured to perform an analysis operation on the remote sensing image data to obtain metadata information and available domain information, determine a storage path according to the metadata information and the available domain information, bind the storage path with the corresponding remote sensing image data, store the storage path in a database, and return a processing result and prompt information to a user after the storage is completed.
In a specific embodiment of the present invention, the data storage module is further configured to store the remote sensing image data in a manner of combining database storage and file storage;
the database storage adopts a relational database, and the file storage adopts shared file storage.
In an embodiment of the present invention, the system further includes:
and the data inspection module is configured to inspect the content, format and file of the remote sensing image data, is arranged between the data acquisition module and the data classification module, and feeds an inspection result back to the data acquisition module to acquire the remote sensing image data again if the remote sensing image data does not accord with the warehousing requirement after inspection.
In a specific embodiment of the present invention, the data retrieval module is further configured to perform fuzzy retrieval according to the first index and perform accurate retrieval according to the second index;
the first index is constructed according to the image data and the metadata, and the second index is constructed according to the metadata, the image data and the available domain data.
In addition, the invention also provides a remote sensing image warehousing method based on the available domain, and the method is applied to any one of the systems.
Compared with the prior art, the beneficial effects of the invention at least comprise the following items:
(1) The data stored in the database comprises image data, metadata, available domain data and the like, different query conditions can be established for different data, and various requirements of a user are met, wherein the available domain data are available regions in the image and are added into the system, when the user conducts retrieval query, rapid retrieval can be conducted through the needed available domain types, more effective data are obtained, when the user conducts subsequent application, the utilization rate of the remote sensing image can be greatly increased, and resource waste of the remote sensing image data is avoided.
(2) And adding the metadata and the available domain into the determining factors of the storage path during storage, storing the metadata and the available domain by using the ternary relationship of the image data-the metadata-the available domain data, and reasonably planning the remote sensing image data so as to avoid the condition of data errors or confusion during storage.
(3) By establishing the two indexes, various retrieval requirements of a user can be met, the retrieval results of the first index are more, the retrieval results of the second index are less, the image required by the user can be retrieved more accurately by utilizing the available domain data to establish the second index, the user can select to use the image data or the available domain data, and the processing speed and the processing efficiency of the available domain data are greatly improved.
(4) And storing according to a mode of combining database storage and file storage during storage. The method can improve the sharing and service level of the remote sensing image data, and is more convenient for managing the remote sensing image data.
(5) The data checking module is arranged, preliminary file and content checking can be carried out on the data, problematic data can be screened out, manual repairing is carried out, and data are obtained again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required in the embodiments will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a remote sensing image warehousing system based on an available domain according to a first embodiment of the present invention;
fig. 2 is a schematic structural diagram of a remote sensing image warehousing system based on available domains according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a remote sensing image warehousing system based on available domains according to a third embodiment of the present invention;
fig. 4 is a schematic flow chart of a remote sensing image warehousing method based on an available domain according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. It should be noted that, as long as there is no conflict, the embodiments and the features of the embodiments in the present invention may be combined with each other, and the technical solutions formed are all within the scope of the present invention.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
Example one
Referring to fig. 1, the present invention provides a remote sensing image warehousing system based on available domains, the system comprising:
and the data acquisition module is configured to acquire remote sensing image data and support multiple data acquisition modes.
Specifically, in an embodiment of the present invention, the data obtaining module includes:
the image data acquisition unit is configured to receive remote sensing image data uploaded by a user, wherein the remote sensing image data comprises image data and metadata.
And the metadata acquisition unit is configured to extract metadata in the remote sensing image data.
In the module, the image data and the metadata are subjected to primary classification so as to form subsequent specific categories, and the storage speed of the subsequent data can be improved to a certain extent.
The data classification module is configured to classify the remote sensing image data into a plurality of categories and classify the remote sensing image data according to different file forms, wherein the categories comprise image data, metadata, quality data and available domain data, and the available domain data are data composed of pixels which are judged to have availability in the image data.
Specifically, the available field data includes tag information, which is information for marking that one pixel is available.
In an embodiment of the present invention, the data classification module includes:
and the data preliminary classification unit is configured to preliminarily divide the acquired remote sensing image data into image data and metadata and store the image data and the metadata in different file forms.
Specifically, the file forms include tif image files, tfw files, xml files, ovr files, xls metadata files, txt files, and table files.
A data first detection unit configured to perform a preliminary quality detection on image data in the remote sensing image data based on a quality detection item, form the quality data, and add the quality data into the category.
Specifically, since the image data also needs to be subjected to quality detection after being subjected to geometric preprocessing, a detection strategy is set, and the preliminary quality detection of the image data is divided into original quality detection and geometric quality detection. The method specifically comprises the following steps: determining original quality detection items of the image data, performing quality detection on the image data by using a detection method corresponding to each original quality detection item to obtain a detection result of each original quality detection item, and taking the detection results of all the original quality detection items as first detection information; performing geometric preprocessing on the image data to obtain preprocessed image data; and determining geometric quality detection items of the preprocessed image data, performing quality detection on the preprocessed image data by using a detection method corresponding to each geometric quality detection item to obtain a detection result of each geometric quality detection item, and taking the detection results of all the geometric quality detection items as second detection information. And integrating the first detection information and the second detection information to form the quality data.
The original quality detection items comprise edge detection, cloud cover detection, strip detection, null value detection and high exposure detection; the geometric quality detection items comprise multispectral dislocation detection and distortion detection. It should be clear that the original quality detection item and the geometric quality detection item may also include another detection item, and the content of the detection item is not limited in this embodiment.
And the second data detection unit is configured to extract metadata of the remote sensing image data, compare the extracted metadata with locally stored metadata, judge the integrity of the metadata, form a judgment result of the availability of the metadata, and add the judgment result into the metadata.
Specifically, the locally stored metadata is a metadata file attached to the remote sensing image data during acquisition, the metadata file is automatically stored in a local server, the operation of extracting the metadata from the remote sensing image data is executed in the metadata acquisition unit, and the extracted metadata information includes: the image area name, time, date, image type, remote sensor, serial number, etc. in the data second detection unit, the determining the integrity of the metadata may be performed by:
since the metadata is one or more files, after the remote sensing image data is decompressed, the integrity of the obtained files needs to be detected, and the obtained integrity result includes the following four conditions: the first is complete metadata, the second is missing metadata, the third is damaged metadata, and the fourth is disordered metadata.
The metadata integrity means that all files of the metadata can be normally opened, the content of the metadata is not lost, the file name of the metadata is matched with the content of the file, and the like. The metadata missing cases include content missing of metadata, file name missing of metadata, information missing in a file of metadata, and the like. The metadata damage condition includes that the file of the metadata can not be opened, the file format of the metadata is wrong, and the like. The case where the metadata is scrambled includes a case where the name of the metadata does not match the name of the corresponding video data, and a case where the file content of the metadata has an error or a scrambling code.
And analyzing the content of the metadata based on an analysis method to obtain a content analysis result of the metadata. And forming the integrity result and the content analysis result into the structural information of the metadata. And judging the structured information based on the integrity result to obtain a judgment result of the availability of the metadata, and adding the judgment result into the metadata.
It should be noted that after the integrity result of the metadata is obtained, if the integrity result is metadata missing, metadata damaged, or metadata scrambled, the metadata needs to be repaired and then stored, that is, when the integrity result is metadata missing, metadata damaged, or metadata scrambled, the structural information of the metadata should include hidden information that the metadata waits to be repaired.
And the data evaluation unit is configured to evaluate the quality data based on evaluation criteria, judge an available area according to the evaluation result, and arrange the available area to obtain the available area data of the remote sensing image data.
Wherein the evaluation criterion is whether the quality data reaches a preset value of the quality detection item.
Specifically, the data evaluation unit performs evaluation in combination with quality data of the first data detection unit during execution, where the quality data is normalized data of the detection result, which represents a quality result of a content item of the preliminary quality detection, and since the preliminary quality detection is performed with a pixel as a basic unit, that is, the quality result is a quality result of each pixel, the data evaluation unit judges whether each pixel is available according to the quality result, and a judgment criterion is that whether the detection result of the pixel reaches a preset value of each quality detection item, and if so, the pixel is judged to be available and included in available domain data. It should be noted that the available domain data is specifically stored in available domains corresponding to different quality detection items, for example, the available domain data includes cloud detection available domain data, shadow detection available domain data, and stripe detection available domain data. The "criterion of evaluation is whether the detection result of the pixel reaches the preset value of each quality detection item" may specifically be: the quality result of the pixel comprises detection results of three quality detection items, the three quality detection items are respectively provided with preset values, when judging whether the quality result is available, the pixel is classified into available domain data of the corresponding quality detection item according to whether the detection result reaches the corresponding preset value or not, and if the quality result reaches the preset value. It should be clear that a pixel may repeatedly include available domain data of multiple quality detection items, for example, if the cloud detection result and the shadow detection result of the pixel both reach corresponding preset values, the pixel is included in both the available domain data of cloud detection and the available domain data of shadow detection.
The data stored in the database comprises image data, metadata, available domain data and the like, different query conditions can be established for different data, and various requirements of a user are met, wherein the available domain data are available regions in the image and are added into the system, when the user conducts retrieval query, rapid retrieval can be conducted through the needed available domain types, more effective data are obtained, when the user conducts subsequent application, the utilization rate of the remote sensing image can be greatly increased, and resource waste of the remote sensing image data is avoided.
And the data uploading module is configured to upload the remote sensing image data meeting the requirements to the database based on the set transmission mode. The transmission mode can be a data transmission protocol, and the data transmission protocol is set according to the data format, the data waveband, the data time phase and the resolution of the current remote sensing data. In practical applications, in order to ensure confidentiality and security of data, a specific uploading operation authority is required to upload data to a server, such as a user name and a password. And before uploading the data, carrying out encryption processing on the remote sensing data and the metadata to be transmitted according to a data transmission protocol.
The uploading and the encryption operations both need a user name and a password to be connected with a server, the data uploading is realized through a network transmission protocol, and meanwhile, the data is encrypted in the data uploading process. In an optional embodiment, because the remote sensing data has huge data volume and security requirement, in a local area network environment, the data transmission protocol of the system adopts an FTP protocol, the FTP protocol ensures the reliable and efficient transmission of massive remote sensing data, the user authentication is realized by connecting the FTP server through a user name and a password, and meanwhile, the security of remote sensing data transmission is enhanced by using FTP transmission encryption.
Furthermore, in the steps of the embodiment, the uploading of the remote sensing data and the metadata is realized by adopting the data transmission protocol corresponding to the data format, the data wave band, the data time phase and the resolution of the remote sensing data, so that the timeliness and the reliability of the uploading operation of the remote sensing data are ensured, and the error probability in the transmission process caused by the special attribute of the remote sensing data is reduced.
And the data storage module is configured to support multilevel storage, and the remote sensing image data is stored according to categories.
The data storage module is also configured to store the remote sensing image data in a manner of combining database storage and file storage. The method can improve the sharing and service level of the remote sensing image data, and is more convenient for managing the remote sensing image data.
And various types of databases and file warehousing systems are comprehensively utilized to efficiently and safely store various data. The method adopts a mode of combining database storage and file storage, wherein the database storage adopts a relational database (spatial database), and the file storage adopts shared file storage.
1) In view of data volume, the remote sensing image data and the product result data volume are large, and the entity data are stored by adopting files.
2) From the perspective of data structure, metadata relates to spatial data and attribute data, and is stored uniformly by adopting a relational database.
3) In view of application requirements, the spatial layer data is mainly used for browsing, extracting and other services, a spatial database is used for storage, and image files and other files requiring high IO operation are stored.
4) From the perspective of data security migration and cost, for data with large access amount, recently accessed data and latest results stored in a business intranet filing disk array, a historical data entity is cleaned locally as required, and only metadata is reserved.
Specifically, the data storage module has a data heat evaluation function and supports evaluation in modes of data import time, image task priority, data access frequency and the like; the system has a data storage access scheduling function and supports access scheduling of image data in a memory, an SSD and an HDD according to the heat; supporting data storage and solidification, and solidifying the processed image data into an SSD and an HDD; the system has a data storage preemption scheduling function and supports various preemption strategies such as temporary emergency tasks, priority high tasks, manual tasks and the like.
The data storage module is further configured to perform analysis operation on the remote sensing image data to obtain metadata information and available domain information, determine a storage path according to the metadata information and the available domain information, bind the storage path and the corresponding remote sensing image data, store the storage path and the corresponding remote sensing image data in a database, and return a processing result and prompt information to a user after the storage is completed.
The data storage module is also configured to analyze a file stored with metadata to obtain metadata information, select a table with fields in a database consistent with the obtained metadata information fields as a storage path of the current metadata, and further determine the storage position of the remote sensing data according to the naming information of the remote sensing data in the metadata, wherein the fields in the metadata information are determined according to the actual working condition requirements of the corresponding remote sensing data, and the naming information of the remote sensing data is set according to a set rule based on the metadata information.
The module adds the metadata and the available domain into the determining factors of the storage path, stores the metadata and the available domain data according to the ternary relationship of the image data, the metadata and the available domain data, and reasonably plans the remote sensing image data so as to avoid the condition of data errors or chaos during storage.
And the data retrieval module is configured to perform query retrieval, including fuzzy retrieval and accurate retrieval, on the remote sensing image data according to the indexes.
The data retrieval module is further configured to perform fuzzy retrieval according to the first index and perform accurate retrieval according to the second index.
The first index is constructed according to the image data and the metadata, and the second index is constructed according to the metadata, the image data and the available domain data.
Specifically, in an embodiment of the present invention, the first index is a spatial index, and the constructing step includes:
generating a spatial coding character string by using the metadata and the image data, then generating a first index according to the spatial coding character string, and correspondingly storing the first index and the image data.
The space encoding character string comprises a plurality of encoding character strings, the encoding character strings are obtained based on a Hilbert curve, and the encoding character strings comprise WKT character strings.
Specifically, in an embodiment of the present invention, the second index is a spatio-temporal index, and the constructing step includes:
step one, establishing a grid model of remote sensing image data, and associating space-time information on the basis of a pyramid grid hierarchical structure.
Firstly, establishing a corresponding relation between image data and a grid model according to information such as image spatial resolution, geographical range and the like in the image data and metadata, copying remote sensing image data to a corresponding position in a grid according to the corresponding relation, and obtaining a grid generated by the remote sensing image data. And then, carrying out step-by-step downsampling processing on the remote sensing image data through a resampling algorithm, wherein the resampling algorithm can be a linear interpolation algorithm, and obtaining grid data under a lower resolution level, so that a complete remote sensing image data pyramid is obtained. And finally, sampling the remote sensing image data on the pyramid hierarchical structure to form a hierarchical structure.
And step two, extracting data characteristics and space-time attributes of each grid for the remote sensing image data grids, quantizing the data characteristics and the space-time attributes, and packaging the quantized data characteristics and the space-time attributes into vectors. Different characteristic weight factors are set for different characteristics, fusion of different characteristics of the remote sensing image data is achieved, and complete description is conducted on the remote sensing image data. And (3) expanding the dimensionality of the clustering algorithm, taking the feature vector with the set weight as the input of the clustering algorithm, and dividing the data with higher feature similarity into the same class to realize the preliminary division of the remote sensing image data grid. When the weight factor is determined, an initial weight value meeting the constraint condition is set for each feature, then the weight is used for dividing and clustering, the evaluation index feedback of the clustering result is used for carrying out weight adjustment on the feature, and the proportional coefficient of each feature when the classification result is optimal can be obtained through multiple iterations by using an empirical learning or machine learning method.
Step three, fusing the available domain and the grid pyramid to construct a cube, and constructing a spatial data set with unified geographic range, spatial resolution, pixel availability and grid size on spatial dimension; in the time dimension, the time effective range of the data set is defined in the form of time stamps, and the time data set is constructed. The characteristics of the available domain data are used for describing the combination range of different time dimensions and space dimensions, the available domain is divided into a plurality of units in the time dimension and the space dimension, and a user is supported to uniformly extract and analyze the remote sensing image data based on the available domain.
And step four, establishing a second index to realize accurate retrieval of the remote sensing image data. Filling the cube by using Hilbert curves respectively under different scales to ensure that the cube can traverse all grids in spatial dimension; space filling curves under different scales are connected end to end according to the sequence of increasing levels, all grids on the space-time pyramid model can be uniquely mapped to be one point on the space filling curves, and dimension reduction of the cube is achieved. And uniformly dividing the grid of the level to which the nodes belong into different regions according to the spatial positions, numbering each region according to the filling sequence of the filling curve, and calculating the grid offset corresponding to the region to obtain the coding value of the grid nodes in the whole grid pyramid. And constructing Rowkey according to the encoding result filled by the space filling curve, introducing the time and space attributes of the metadata and the pixel attribute of the available domain data into the Rowkey as prefixes, providing indexing capability in space and time, and ensuring that the data is consistent with the spatial and time arrangement sequence of the image data when stored, thereby realizing the continuity of the remote sensing image data indexing in multiple dimensions. And the accuracy degree of the second index can be improved by the characteristics of the available domain data, the second index can be used as the index by availability, the validity of the inquired data can be ensured, and redundant invalid data can be screened out.
By establishing two indexes, various retrieval requirements of a user can be met, the retrieval result of the first index is more, the retrieval result of the second index is less, the second index is created by utilizing the available domain data, the image required by the user can be retrieved more accurately, the user can select to use the image data or the available domain data, and if the requirement of the user is to perform image processing aiming at the available domain, the processing speed and the processing efficiency of the available domain data can be greatly improved.
Example two
The detection of the uploaded remote sensing image data is one of effective means for guaranteeing the success rate of data transmission, after the data are checked to meet the storage requirements, the data are classified, sorted and the like and are imported into the database, so that the subsequent redundant operation caused by transmission failure due to lack of data, data errors and the like can be reduced to a great extent, the storage pressure of the database can be relieved to a certain extent, and the validity of the data stored in the system can be effectively guaranteed. Therefore, fig. 2 is a schematic structural diagram of a remote sensing image warehousing system based on an available domain in an embodiment of the present invention, and referring to fig. 2, the system in an embodiment of the present invention includes: the device comprises a data acquisition module, a data inspection module, a data classification module, a data uploading module, a data storage module and a data retrieval module.
The data inspection module is arranged between the data acquisition module and the data classification module, and if the remote sensing image data does not meet the warehousing requirement after inspection, the inspection result is fed back to the data acquisition module to acquire the remote sensing image data again.
Specifically, in the data inspection module, it is configured to perform content, format, and file inspection on the remote sensing image data, including: (1) Checking the file format of the image data to check whether the format is correct, specifically judging whether the image data is in a remote sensing data format or not, and generally judging whether the image data is in an effective remote sensing image file format or not; (2) Checking the file content of the image data to judge whether the content is missing or whether the file can be opened, wherein the file content is checked to determine whether the file content is wrong or not by adopting a manual checking mode; (3) And checking the metadata by combining the structural information of the metadata to determine whether the implicit information exists, if so, manually repairing the metadata, and if not, re-acquiring the metadata.
This embodiment has set up data inspection module, can carry out preliminary file, content inspection to data to select the data that have problems, and go on artifical restoration and acquire data again, this module can effectively improve the utilization ratio of data, and avoids the wasting of resources, and can let the data correctness of putting in storage improve, do benefit to subsequent image and use.
EXAMPLE III
Referring to fig. 3, in this embodiment, the warehousing system may further include a data management module configured to backup remote sensing image data, and perform automatic backup and recovery when the storage function is abnormal.
The data management module comprises a data management unit and a data directory management unit, wherein the data management unit has a function of fast data backup and recovery and supports automatic data backup and recovery according to a copy strategy under the condition that a storage node or data is abnormal; the system has a data copy consistency function, supports automatic generation of copies, supports data locking access of storage nodes, and provides a data verification function; the method has a data ancestry function, and supports image processing calculation of related data blocks again according to an ancestry mechanism under the condition that the processed image data is abnormal, so that the usability of the panoramic image data is guaranteed; the video data import and export system has a data import and export function and supports import and export of video data.
The data directory management unit has a global unified view display function, supports unified view display of original image data, original image data blocks, processed image data and processed image data blocks, supports display according to categories such as types, space geographic ranges and wave bands, supports display of image data states, and provides data retrieval support; the data node local view display function is provided, all data and data blocks on the storage node and corresponding data states are supported to be checked, and data retrieval support is provided; the system has an original data view display function, supports the viewing of the original data and original data blocks of global and data nodes, and provides data retrieval support; the method has the function of original data view display, supports the viewing of the original data and original data blocks of global and data nodes, and provides data retrieval support.
The interface and directory mapping module provides functions of client file interface (providing C/C + +, JAVA secondary development interface) and local directory mapping for image data access of users and external applications, supports Linux and Windows directory mapping, and supports standard Posix file interface access.
The embodiment can effectively support data distributed management by arranging the data management unit and the data directory management unit, and improves the operability and the access friendliness of the system by arranging the directory mapping module.
In the remote sensing image warehousing system based on the available domain provided by the embodiment of the invention, each module or unit structure can be independently operated or combined to operate according to actual requirements so as to realize corresponding technical effects.
Based on other aspects of any one or more of the foregoing embodiments, an embodiment of the present invention further provides a remote sensing image warehousing method based on an available domain, and fig. 4 shows a schematic flow chart of the remote sensing image warehousing method based on the available domain in this embodiment, where the method includes:
step S410, obtaining the remote sensing image data through the data obtaining module, where the remote sensing image data obtained in the step includes image data and metadata.
Step S420, the remote sensing image data are classified according to categories by using a data classification module, wherein the categories comprise image data, metadata, quality data and available domain data, and the available domain data are formed by pixels with availability after the image data perform availability judgment on the pixels according to an evaluation rule.
And S430, uploading the remote sensing image data by using a data uploading module, and transmitting the data to a database.
And S440, performing multilevel storage on the uploaded remote sensing image data according to a data storage module, and storing different data types according to different forms.
S450, performing fuzzy retrieval and accurate retrieval on the stored data according to a data retrieval module, wherein the fuzzy retrieval utilizes a first index to perform retrieval, and the first index is constructed by the metadata and the corresponding image data; and the accurate retrieval is carried out by utilizing a second index, and the second index is constructed by the metadata, the image data and the available domain data.
Specifically, in an embodiment of the present invention, the remote sensing image data is stored by combining database storage and file storage, where the database storage is a relational database, and the file storage is shared file storage.
Specifically, in an embodiment of the present invention, global unified view management is performed on the remote sensing image data according to the data management module, and directory management is established to perform view display management on the remote sensing image data.
Further, in an embodiment of the present invention, the warehousing method further includes the following steps:
and carrying out preliminary detection on the remote sensing image data according to a data inspection module, wherein the step is carried out after the initial remote sensing image data is obtained, if the remote sensing image data does not accord with the warehousing requirement after inspection, the inspection result is fed back to the data acquisition module, and the remote sensing image data is obtained again.
The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (8)
1. A remote sensing image warehousing system based on available domain, the system comprising:
the data acquisition module is configured to acquire remote sensing image data and support multiple data acquisition modes, wherein the remote sensing image data comprises image data and metadata;
the data classification module is configured to classify the remote sensing image data into a plurality of categories and classify the remote sensing image data according to different file forms, wherein the categories comprise image data, metadata, quality data and available domain data, and the available domain data are data consisting of pixels which are judged to have availability in the image data;
the data classification module comprises:
the data preliminary classification unit is configured to preliminarily divide the acquired remote sensing image data into image data and metadata and store the image data and the metadata in different file forms;
a data first detection unit configured to perform preliminary quality detection on image data in the remote sensing image data based on a quality detection item, form the quality data, and add the quality data into the category;
the second data detection unit is configured to extract metadata of the remote sensing image data, compare the extracted metadata with locally stored metadata, judge the integrity of the metadata, form an evaluation result of the availability of the metadata, and add the evaluation result into the metadata;
the data evaluation unit is configured to evaluate the quality data based on evaluation criteria, judge an available area according to the evaluation result, and arrange the available area to obtain the available area data of the remote sensing image data;
the data uploading module is configured to upload the remote sensing image data meeting the requirements to a database based on a set transmission mode;
the data storage module is configured to support multilevel storage and store the remote sensing image data according to the categories;
the data retrieval module is configured to perform query retrieval on the remote sensing image data according to the indexes, wherein the query retrieval comprises fuzzy retrieval and accurate retrieval;
the data retrieval module is further configured to perform fuzzy retrieval according to the first index and perform accurate retrieval according to the second index;
the first index is constructed according to the image data and the metadata, and the second index is constructed according to the metadata, the image data and the available domain data.
2. The system of claim 1, wherein the available field data includes label information, and the label information is information for marking one pixel available.
3. The system of claim 1, wherein in the data evaluation unit, the evaluation criterion is whether the quality data reaches a preset value of the quality detection item.
4. The system of claim 1, wherein the result of the integrity of the metadata comprises metadata integrity, metadata loss, metadata corruption, and metadata confusion.
5. The system of claim 1, wherein the data storage module is further configured to perform parsing on the remote-sensing image data to obtain metadata information and available domain information, determine a storage path according to the metadata information and the available domain information, bind the storage path with the corresponding remote-sensing image data, store the storage path in a database, and return a processing result and prompt information to a user after the storage is completed.
6. The system of claim 1, wherein the data storage module is further configured to store the remotely sensed image data in a combination of database storage and file storage;
the database storage adopts a relational database, and the file storage adopts shared file storage.
7. The system of claim 1, further comprising:
and the data inspection module is configured to inspect the content, format and file of the remote sensing image data, is arranged between the data acquisition module and the data classification module, and feeds an inspection result back to the data acquisition module to acquire the remote sensing image data again if the remote sensing image data does not accord with the warehousing requirement after inspection.
8. A remote sensing image warehousing method based on available domain, characterized in that the method is applied to the system of any of the above claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210852457.XA CN115080774B (en) | 2022-07-20 | 2022-07-20 | Remote sensing image warehousing system and method based on available domain |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210852457.XA CN115080774B (en) | 2022-07-20 | 2022-07-20 | Remote sensing image warehousing system and method based on available domain |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115080774A CN115080774A (en) | 2022-09-20 |
| CN115080774B true CN115080774B (en) | 2022-11-04 |
Family
ID=83260226
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210852457.XA Active CN115080774B (en) | 2022-07-20 | 2022-07-20 | Remote sensing image warehousing system and method based on available domain |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115080774B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292530A (en) * | 2022-09-30 | 2022-11-04 | 北京数慧时空信息技术有限公司 | Remote sensing image overall management system |
| CN116186319A (en) * | 2022-11-23 | 2023-05-30 | 自然资源部国土卫星遥感应用中心 | Satellite image data archiving management method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1698856A2 (en) * | 2005-03-02 | 2006-09-06 | Deutsches Zentrum für Luft- und Raumfahrt e.V. | Method and system for obtaining remote sensing data |
| CN109284338A (en) * | 2018-10-25 | 2019-01-29 | 南京航空航天大学 | A kind of satellite remote sensing big data Optimizing Queries method based on hybrid index |
| CN113590894A (en) * | 2021-07-12 | 2021-11-02 | 浙江大学 | Dynamic and efficient remote sensing image metadata warehousing retrieval method |
| CN114003755A (en) * | 2021-10-25 | 2022-02-01 | 中国自然资源航空物探遥感中心 | Multi-source satellite scene-separating image data organization storage and retrieval method, system and equipment |
| CN114463081A (en) * | 2021-12-14 | 2022-05-10 | 北京航天世景信息技术有限公司 | Mass image online transaction and distribution integrated system and method based on remote sensing cloud |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8484259B1 (en) * | 2009-12-08 | 2013-07-09 | Netapp, Inc. | Metadata subsystem for a distributed object store in a network storage system |
| CN107861983A (en) * | 2017-09-30 | 2018-03-30 | 中国电子科技集团公司第三十二研究所 | Remote sensing image storage system for high-speed remote sensing image processing |
| CN109783665B (en) * | 2018-12-29 | 2022-10-14 | 武汉大学 | Design method for realizing Hbase database remote sensing big data storage model based on Google S2 |
| CN111125392B (en) * | 2019-12-25 | 2023-06-16 | 华中科技大学 | Remote sensing image storage and query method based on matrix object storage mechanism |
-
2022
- 2022-07-20 CN CN202210852457.XA patent/CN115080774B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1698856A2 (en) * | 2005-03-02 | 2006-09-06 | Deutsches Zentrum für Luft- und Raumfahrt e.V. | Method and system for obtaining remote sensing data |
| CN109284338A (en) * | 2018-10-25 | 2019-01-29 | 南京航空航天大学 | A kind of satellite remote sensing big data Optimizing Queries method based on hybrid index |
| CN113590894A (en) * | 2021-07-12 | 2021-11-02 | 浙江大学 | Dynamic and efficient remote sensing image metadata warehousing retrieval method |
| CN114003755A (en) * | 2021-10-25 | 2022-02-01 | 中国自然资源航空物探遥感中心 | Multi-source satellite scene-separating image data organization storage and retrieval method, system and equipment |
| CN114463081A (en) * | 2021-12-14 | 2022-05-10 | 北京航天世景信息技术有限公司 | Mass image online transaction and distribution integrated system and method based on remote sensing cloud |
Non-Patent Citations (1)
| Title |
|---|
| 遥感影像快速入库和元数据提取***的研究;杜晓等;《国土资源遥感》;20060920(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115080774A (en) | 2022-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115080774B (en) | Remote sensing image warehousing system and method based on available domain | |
| CN110781325A (en) | High-resolution remote sensing data grid refined management model and construction method thereof | |
| CN101685539B (en) | On-line ortho-rectification method and system for remote sensing image | |
| CN112559534B (en) | Remote sensing image data filing management system and method | |
| CN110659369B (en) | On-orbit high-precision lightweight global image control point database construction method and system | |
| CN111831856B (en) | Metadata-based automatic holographic digital power grid data storage system and method | |
| CN109741209A (en) | Power distribution network multi-source data fusion method, system and storage medium under typhoon disaster | |
| CN113626437B (en) | Method and system for rapidly inquiring mass vector data | |
| CN112508733A (en) | Big data intelligence service system in electric wire netting space-time based on big dipper | |
| CN104361128A (en) | Data synchronization method of PC (Personnel Computer) end and mobile terminal based on hydraulic polling business | |
| Chen et al. | An efficient method for near-real-time on-demand retrieval of remote sensing observations | |
| Yılmaz et al. | Real time vector database updating system: a case study for Turkish topographic vector database (Topovt) | |
| Stonebraker et al. | The SEQUOIA 2000 architecture and implementation strategy | |
| Caprioli et al. | Rules and standards for spatial data quality in GIS environments | |
| CN114860730A (en) | Land data storage method, system and storage medium based on block chain | |
| CN110716898A (en) | Method and system for gathering field geological survey data in real time | |
| López et al. | Design scheme for spatial database of climatic and environmental variables in Mexico, integrating big data technology | |
| Xie et al. | Implementing next-generation national Earth Observation data infrastructure to integrate distributed big Earth Observation data | |
| CN116737854B (en) | Space-time data lake management system based on multi-source remote sensing data and safety protection method thereof | |
| CN103164417A (en) | Geological information processing method and device | |
| CN116166723A (en) | Heterogeneous data processing method, device, equipment and storage medium | |
| Xie et al. | Distributed geospatial data infrastructure for heterogeneous disaster data integration and application | |
| King et al. | Distributed Gridded Data Delivery for Marine Research | |
| CN116578648A (en) | Multi-source heterogeneous data collaboration method and device | |
| Rustad et al. | Exposing military sensor data using SpatioTemporal Asset Catalog (STAC) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |