CN113254568A - Real-time analysis processing method, device and medium combined with map service - Google Patents
Real-time analysis processing method, device and medium combined with map service Download PDFInfo
- Publication number
- CN113254568A CN113254568A CN202110723784.0A CN202110723784A CN113254568A CN 113254568 A CN113254568 A CN 113254568A CN 202110723784 A CN202110723784 A CN 202110723784A CN 113254568 A CN113254568 A CN 113254568A
- Authority
- CN
- China
- Prior art keywords
- tile
- map service
- workflow
- data
- function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Remote Sensing (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a real-time analysis processing method, a real-time analysis processing device and a real-time analysis processing medium combined with a map service, and belongs to the field of cloud computing. The method comprises the steps of firstly analyzing a computational expression of a map service request to complete the modeling of a workflow; then, calling a function packaged in a computing node in the workflow, and obtaining image data in the map tile from a cloud data source to execute the workflow in a recursive manner to finish the processing of the image tile; and finally, performing visual rendering on the result tiles obtained by processing. The method has the advantages that the cloud image data is rapidly acquired in the form of map service and is analyzed and processed in real time, direct visual conversion from the original data to the data product is realized, and the problems of complexity, hysteresis and the like in the process of remote sensing application from data processing, visualization to sharing service and the like are solved. The method has important practical application value for extraction and sharing and rapid application of the remote sensing information.
Description
Technical Field
The invention belongs to the field of cloud computing, and particularly relates to a real-time computing method combined with a map service.
Background
With the rapid growth of remote sensing image data, more challenges are provided for efficient information processing and value mining of raster data, and the traditional map service focuses on content sharing and visualization and lacks a function of real-time image analysis and processing. At present, the production capacity of remote sensing image data far exceeds the processing capacity of the data, and no matter the image is manually interpreted or digitally analyzed, the problems of complex flow, product lag and the like exist. However, in real life, there are many application scenarios requiring high timeliness, such as emergency response, rescue and disaster relief, and location-based services, which require immediate access to various remote sensing image data or derivative information products processed from the image data to make decisions and take actions quickly, and these application requirements place more efficient, real-time, and dynamic requirements on comprehensive map services. In addition, the remote sensing data is large in data volume, and the traditional local storage scheme is difficult to adapt to a distributed GIS system architecture. Cloud storage is one of the mainstream storage schemes of the current mass data, not only can provide safe and persistent support for remote sensing image data, but also can easily manage hundreds of millions of files.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provides a real-time analysis processing method, a real-time analysis processing device and a real-time analysis processing medium which are combined with a map service.
In order to achieve the above purpose, the invention specifically provides the following technical scheme:
in a first aspect, the present invention provides a real-time analysis processing method combined with a map service, which is used for performing analysis and visual conversion on remote sensing image data stored in a cloud in a map service form, and comprises the following steps:
s1: acquiring a map service request submitted by a client, analyzing a calculation expression of the map service request, and modeling the map service request into a workflow consisting of grid tile metadata nodes and calculation nodes according to the priority of operation in the expression; the grid tile metadata node is a mapping table for recording grid tile information; the computing nodes are packaged with function names and function parameters corresponding to operators and used for executing computing tasks in the computing nodes and participating in the carrying drive of workflow task links, wherein the function parameter type is at least one of grid tile metadata nodes and the computing nodes;
s2: triggering from the last calculation node in the workflow, calling the function name and the function parameter packaged in the calculation node, and executing the workflow recursively to complete the processing of the image tile; in the workflow execution process, for any current calculation node, if all function parameter types required by the current calculation node are grid tile metadata nodes, acquiring grid tile data from a cloud according to grid tile information recorded in the grid tile metadata nodes and obtaining an operation result through function operation, and if the function parameter types required by the current calculation node contain internal calculation nodes, executing the internal calculation node firstly, replacing the internal calculation nodes with the operation result and then continuing to execute the function operation of the current calculation node;
s3: and performing visual rendering on the operation result of the last calculation node in the workflow in a tile format, and returning the operation result to the client.
Based on the technical scheme, the steps are preferably realized in the following specific mode. The preferred implementation manners of each step can be combined correspondingly without conflict, and are not limited.
Preferably, in the step S1, for each map service request, the workflow is modeled according to S11 to S12:
s11: acquiring a computational expression representing the computational requirement of image data from the map service request, wherein the computational expression consists of an operator and image data information participating in operation;
s12: the server analyzes the received computational expression in real time, and carries out workflow modeling by traversing character strings of the computational expression and combining two pre-constructed stack data structures, wherein a stack A is used for storing an operator, and a stack B is used for storing nodes in a workflow; in the traversal process, if the character represents image data information participating in operation, the character is used as a raster tile metadata node and is placed in a stack B, and if the character represents an operator, the character is placed in a stack A according to the priority of the operator; when the elements in the stack B meet the parameters required by the operator with the highest priority in the stack A, combining and packaging the operator and the grid tile metadata nodes corresponding to the required parameters into a new calculation node to be stored in the stack B; after the whole calculation expression is traversed, a workflow consisting of grid tile metadata nodes and calculation nodes is finally obtained, and the last node of the workflow is a calculation node; the grid tile metadata node is a mapping table for recording grid tile information, and comprises a data storage path for acquiring the grid tile, a space triple coordinate and required wave band information.
Preferably, the specific method of step S2 is as follows:
s21: acquiring the last calculation node of the workflow as the current calculation node;
s22: calling function names and function parameters packaged in current calculation nodes, traversing each function parameter, and directly obtaining an operation result through function operation if the function parameters required by the function operation are all raster tile data; if at least one function parameter is not the raster tile data, parameter replacement operation is required to be executed firstly, the raster tile data corresponding to the execution result is replaced to the corresponding function parameter position, and after all the function parameters required by the function operation are replaced by the raster tile data, the operation result is obtained through the function operation;
the parameter replacement operation needs to select A) or B) according to the type of the function parameter:
A) if the function parameter is a grid tile metadata node, acquiring grid tile data from a cloud according to grid tile information recorded in the grid tile metadata node and replacing the grid tile metadata node;
B) if the function parameter is a calculation node, temporarily taking the calculation node corresponding to the function parameter as a new current calculation node, re-executing S22 and returning the operation result to perform parameter replacement on the calculation node.
Preferably, in S2, the specific method for acquiring raster tile data from the cloud according to the raster tile information recorded in the raster tile metadata node is as follows:
a1: reading metadata information of the cloud image, converting a space effective range of the cloud image and a space range of grid tiles required in the map service request into a uniform coordinate system, judging whether the requested grid tiles are located in an area of the cloud image or not through a space intersection relation, if so, executing A2, otherwise, terminating execution of a workflow and returning error information to the client;
a2: mapping the space range of the grid tiles required in the map service request to the cloud image to obtain a corresponding pixel area range, then reading the effective data segment range of the pixel area range in the cloud image according to the organization format of the cloud image, acquiring grid data corresponding to the map service request from the cloud, and normalizing the grid tile data into 256 × 256.
Preferably, the specific method of step S3 is as follows:
s31: performing numerical linear stretching on the grid tile corresponding to the operation result of the last calculation node in the workflow, and mapping the value interval to 0-255 to obtain the stretched grid tile;
s32: rendering and coloring the stretched raster tiles according to the specified color chart style, and returning rendering results to the client in a picture form.
Preferably, the cloud image is in a COG data format.
Preferably, the unified coordinate system is a WGS coordinate system.
Preferably, in S32, a rendering array with a size of 256 and containing RGB triples is generated for the tile according to the specified color map pattern, the stretched raster tile is rendered by the rendering array, and a png format still picture is generated and returned to the client.
In a second aspect, the invention provides a real-time analysis processing device combined with a map service, which is characterized by comprising a memory and a processor;
the memory for storing a computer program;
the processor is configured to, when executing the computer program, implement the real-time analysis processing method combined with the map service according to any one of the above first aspects.
In a third aspect, the present invention provides a computer-readable storage medium, wherein the storage medium stores thereon a computer program, and when the computer program is executed by a processor, the computer program can implement the real-time analysis processing method combined with a map service according to any one of the aspects of the first aspect.
Compared with the prior art, the invention has the following beneficial effects:
aiming at the problem that the map service is lack of real-time calculation and analysis, the method fully considers the characteristics that the data tiles can be obtained and calculated in real time, and combines a remote sensing application model to quickly analyze massive remote sensing data stored in the cloud, so that the direct visual conversion from original data to a data product is realized, and the extraction, sharing and quick application of remote sensing information are accelerated. The invention has the following four advantages: 1) the computing units are map tiles and have distributed expansion capability; 2) the calculation process is carried out in real time, and the calculation result is presented to the user in a map service mode, so that the effect of calculation is achieved; 3) the requirements of calculation and analysis can be flexibly combined through an expression to complete the diversified analysis of the image; 4) the calculation process is interactive on demand, and the calculation of the current area is triggered along with the translation and the zooming of the map interface.
Drawings
FIG. 1 is a flow chart of the steps of a real-time analysis processing method in conjunction with a mapping service.
FIG. 2 is a flow diagram of a computational expression parsing modeling workflow.
FIG. 3 is a grid tile computation workflow execution flow diagram.
Detailed Description
The invention will be further elucidated and described with reference to the drawings and specific embodiments.
The invention provides a real-time analysis processing method combined with a map service, which is used for analyzing and visually converting remote sensing image data stored in a cloud end in the form of the map service. In order to facilitate distributed storage of mass map data, the remote sensing image data is generally converted into Cloud Optimized Geo TIFF (COG) in advance, and is stored in the Cloud (a distributed storage manner such as S3 or HDFS may be adopted). The method can be realized based on a standard grid tile map service, and a calculation expression parameter can be additionally added in a map service request input by a client to transfer the calculation requirement of a user on the grid tile, so that the map service has the real-time calculation capability on the remote sensing image data stored in the cloud.
The calculation expression input at the client side in the invention is composed of an operator and remote sensing image data information participating in operation, wherein the operator represents a flow (such as x, div, + and' -) of data processing, and each operator needs parameters participating in operation, and the parameters in the embodiment are the remote sensing image data. The remote sensing image data has different bands, so the mode in which the band data can be set to be represented in this embodiment is composed of a letter B and a number, for example: b1 represents the first band data of the video data, B2 represents the second band data of the video data, and so on.
Since computational expressions cannot be directly executed, the present invention requires workflow to fulfill the user's computational requirements for grid tiles. The workflow is a series of data operation and calculation steps in the application program, and the flow between the steps can be clearly reflected. Therefore, the workflow needs to be modeled into a workflow according to operator priorities contained in a computational expression, the workflow comprises two types of nodes which are grid tile metadata nodes and computational nodes respectively, the grid tile metadata nodes are responsible for acquiring corresponding grid tile data from a cloud, the computational nodes are responsible for executing a workflow of a current node, the work tasks comprise two types, one type is a function computation task in the node, and the other type is a carried drive needing to participate in a workflow task link, such as parameter replacement, data transmission, recursion and the like, so that the workflow can be executed in sequence to complete an operation flow corresponding to the computational expression.
Referring to fig. 1, in a preferred embodiment of the present invention, the real-time analysis processing method combined with the map service includes the following specific steps:
s1: the server side obtains a map service request submitted by the client side, wherein the map service request comprises a calculation expression and a space range of grid tiles participating in calculation. After the map service request is obtained, the server side firstly analyzes the calculation expression of the map service request, and models the calculation expression into a workflow according to the priority of operation in the expression.
As previously described, the workflow is composed of grid tile metadata nodes and computation nodes, the grid tile metadata nodes are a mapping table for recording grid tile information, wherein the recorded grid tile information should include the data storage path of the grid tile, the spatial triplet coordinates of the grid tile, and the required band information. The tile map is stored in a tile pyramid mode, each grid tile can be uniquely marked by a hierarchy and a row and column number, and therefore the spatial triple coordinate refers to the hierarchy Z and the row and column number (X, Y) of the grid tile.
The calculation node is a special node responsible for executing the current node workflow, wherein function names and function parameters corresponding to operators are packaged, and calculation tasks in the node can be executed and simultaneously participate in the carrying drive of a workflow task link, such as parameter replacement, data transmission, recursion and the like. Because an operator is usually realized through a corresponding function in the algorithm realization process, for the operator in the calculation expression, a function which realizes the same function as the operator is found in a function library, and the function name of the function is packaged in a calculation node. The necessary parameters required by each function to realize the operation are packaged in the calculation nodes in the form of function parameters, and because the calculation expression has operation priority, the parameters required by one function to realize the operation can be certain raster tile data or operation results of other operators with higher priority. Therefore, in the present invention, the type of the function parameter encapsulated in one computation node may be a grid tile metadata node, or may be another computation node, or may be a combination of the two.
To automate the workflow modeling described above, the modeling process for the workflow may be implemented via a stack data structure. A stack (stack), as a data structure, is a special linear table that can perform insert and delete operations only at one end, where the end that allows insert and delete operations is called the top of the stack and the other end is the bottom of the stack.
In the embodiment, for each map service request, the modeling process of the workflow can be realized according to S11-S12 by means of a stack data structure:
s11: the server side obtains a computational expression representing the computational requirement of the image data from the map service request, wherein the computational expression consists of an operator and image data information participating in operation, and the specific form is as described above.
S12: the server analyzes the received computational expression in real time, the analyzing process can be carried out by traversing the character string of the computational expression from left to right and combining two pre-constructed stack data structures, and the workflow modeling can be completed by constructing nodes in the analyzing process. Two pre-constructed stack data structures are denoted as stack a and stack B in the present invention, where stack a is used to store operators and stack B is used to store nodes in the workflow. In the traversal process of the computational expression, if the character represents the image data information participating in the operation, the character is taken as a raster tile metadata node and is placed in a stack B, if the character represents an operator, the character is placed in a stack A according to the priority of the operator, the operator with high priority is positioned at the top of the stack, and the operator with low priority is positioned at the bottom of the stack. In the traversal process, when the elements in the stack B meet all parameters required by an operator with the highest priority in the stack A, grid tile metadata nodes corresponding to the operator and the required parameters are merged and packaged into a new calculation node to be stored in the stack B. After the traversal of the whole computational expression is finished, a workflow consisting of grid tile metadata nodes and computational nodes can be finally obtained, and the last node of the workflow is a computational node. As mentioned above, a grid tile metadata node is a mapping table for recording grid tile information, and includes a data storage path, spatial triple coordinates, and required Band information for obtaining the grid tile, which are denoted as (URL, X, Y, Z, Band). And subsequently, the storage position of the raster tile in the cloud can be found through the data storage path of the raster tile, and then the hierarchy and the row and column number of the raster tile are determined through the spatial triple coordinates (X, Y, Z), so that the required wave band is extracted from the raster tile.
After the modeling process of the workflow is completed, the workflow can be executed according to the following step S2, and the remote sensing image data of the cloud end is calculated and analyzed in real time.
S2: and triggering from the last calculation node in the workflow, calling the function name and the function parameter packaged in the calculation node, and executing the workflow in a recursion way to finish the processing of the image tile.
In the above-described recursive workflow execution process, for convenience of description, the present invention refers to the computing node being executed as the current computing node. For any current calculation node, the function parameter type required by the current calculation node needs to be judged first to perform corresponding operation according to the situation, which is specifically as follows:
if all the function parameter types required by the current calculation nodes are grid tile metadata nodes, acquiring grid tile data from a cloud according to grid tile information recorded in the grid tile metadata nodes, wherein the grid tile data acquired from the cloud can replace corresponding grid tile metadata nodes, so that the function parameters required by function operation are changed into the grid tile data, and an operation result can be obtained by executing the function operation;
if the function parameter type required by the current calculation node contains the internal calculation node, the internal calculation node is executed first, and the operation result is used for replacing the internal calculation node, so that the function operation of the current calculation node can be continuously executed.
Therefore, if a computing node finds that the required function parameter type contains an internal computing node in the execution process, the operation of the internal computing node needs to be executed first, and at this time, the internal computing node needs to be used as the current computing node first. Similarly, if the type of function parameter required in an internal computation node also contains further internal computation nodes, the further internal computation nodes in the internal computation node are executed first. Thus, this calculation is actually a recursive, multiple-iteration calculation.
In the present embodiment, the specific recursive execution method of step S2 may be executed as follows:
s21: acquiring the last calculation node of the workflow as the current calculation node;
s22: calling function names and function parameters packaged in current calculation nodes, traversing each function parameter, and directly obtaining an operation result through function operation if the function parameters required by the function operation are all raster tile data; if at least one function parameter is not the raster tile data, parameter replacement operation is required to be executed firstly, the raster tile data corresponding to the execution result is replaced to the corresponding function parameter position, and after all the function parameters required by the function operation are replaced by the raster tile data, the operation result is obtained through the function operation;
since the function parameter may be a grid tile metadata node or a computation node, the above-mentioned parameter replacement operation actually selects a) or B) according to the type of the function parameter:
A) if the function parameter is a grid tile metadata node, acquiring grid tile data from a cloud according to grid tile information recorded in the grid tile metadata node and replacing the grid tile metadata node;
B) if the function parameter is a calculation node, temporarily taking the calculation node corresponding to the function parameter as a new current calculation node, re-executing S22 and returning the operation result to perform parameter replacement on the calculation node.
It should be noted that, in the recursive execution process, the current computing nodes are continuously changed, and after each new current computing node obtains the computation result and completes the parameter replacement of the computing node, the new current computing node disappears, returns to the previous current computing node, and executes the function computation of the previous current computing node.
In addition, acquiring raster tile data from the cloud according to raster tile information recorded in the raster tile metadata node is actually a process of calling data from the cloud, and the implementation form of the process can refer to any prior art. The specific method adopted for implementing the process in the embodiment is as follows:
a1: metadata information of the cloud image is read, and a space effective range representing the maximum space range of the cloud image can be extracted from the metadata information. Because the spatial coordinate system adopted by the cloud image storage may be inconsistent with the spatial coordinate system adopted by the request range input in the map service request, both the spatial effective range of the cloud image and the spatial range of the grid tile required in the map service request need to be converted into a uniform coordinate system (such as a WGS coordinate system, etc.), then whether the requested grid tile is located in the area of the cloud image is judged through a spatial intersection relationship, if so, a2 is executed, otherwise, the execution workflow is terminated, and an error message is returned to the client to remind the user that the requested data range exceeds the effective range.
A2: mapping the space range of the grid tiles required in the map service request to the cloud image to obtain a corresponding pixel area range, then reading the effective data segment range of the pixel area range in the cloud image according to the organization format of the cloud image, acquiring grid data corresponding to the map service request from the cloud, and normalizing the grid tile data into 256 × 256.
After the whole workflow is executed in the recursive manner, the processing of the image tile can be completed, and the operation result of the last calculation node in the workflow is the final operation result. The operation result needs to be rendered in S3 before being sent to the client.
S3: and performing visual rendering on the operation result of the last calculation node in the workflow in a tile format, and returning the operation result to the client.
In this embodiment, the specific method for performing visual rendering on the operation result may include the following steps:
s31: performing numerical linear stretching on the grid tile corresponding to the operation result of the last calculation node in the workflow, and mapping the value interval to 0-255 to obtain the stretched grid tile;
s32: rendering and coloring the stretched raster tiles according to the color map pattern, and returning rendering results to the client in a picture form. The color chart style adopted by the rendering can be specified by a user when a client submits a service request, or a default specified style can be adopted, and the same returned picture format can be adjusted according to the actual condition. In the invention, a rendering array which comprises RGB triples and is 256 in size is generated for the client according to the specified color map style, the stretched raster tile is rendered by using the rendering array, and a png format static picture is generated and returned to the client.
Additionally, in other embodiments, a real-time analysis processing apparatus in combination with a map service may also be provided, including a memory and a processor;
the memory for storing a computer program;
the processor is configured to implement the real-time analysis processing method combined with the map service as described in the above S1-S3 when executing the computer program.
In addition, in other embodiments, a computer-readable storage medium may be provided, having stored thereon a computer program which, when executed by a processor, is capable of implementing the real-time analysis processing method in combination with a map service as described in the above S1-S3.
It should be noted that the Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. Of course, the device should also have the necessary components to implement the program operation, such as power supply, communication bus, etc.
The real-time analysis processing method combined with the map service is applied to specific examples to show the effects of the real-time analysis processing method based on the real-time analysis processing methods combined with the map service described in the above embodiments S1-S3. As described above, the specific process is not fully described, and specific examples and implementation effects thereof are mainly shown below.
Examples
The following takes the example of calculating NDVI in real time from Landsat 8 images in the hangzhou city in the form of map service, and specifically describes the present invention, and the specific steps are as follows:
step 1) merging a plurality of Landsat 8 single-band images into a single Geo TIFF, converting the image into COG, and storing the COG in an HDFS.
The calculation expression to be calculated in this embodiment is the following NDVI formula:
in the formulaB nir Is in the near-infrared wave band, and the infrared wave band,B red is in the red light wave band.
Thus, a computational expression is added to the request in the map service by the client: expression = (B4-B3)/(B4 + B3).
Wherein the band data is represented by a manner consisting of a letter B and a number, and Bn represents the second of Landsat 8 single-band image datanNumber of wave bands.
And 2) the server carries out real-time analysis modeling on the computational expression, establishes grid tile metadata nodes according to the data set, the waveband information and the map tile parameters, establishes computational nodes according to operators and finally generates a workflow.
In the present embodiment, for the computational expressions added in the map service request, fig. 2 is a flow for analytically modeling the expressions "(B1-B2)/(B3 + B4)", specifically performing modeling of a workflow according to the following processes:
and after receiving the request, the server analyzes the expression in real time, and models the expression into a workflow according to the priority of operation in the expression. The analysis process can be realized by two auxiliary data structure stacks A and B, wherein the stack A is used for storing operators or operation functions in expressions, the stack B is used for storing constructed raster tile metadata nodes or calculation nodes, and a critical pointer is further arranged in the stack B and used for indicating a valid node flag bit in the stack. As shown in fig. 2 (a), stack a and stack B are initially empty, and in the modeling process, the expression string parameters need to be traversed, and when a band number is encountered, a grid tile metadata node is constructed and placed in stack B. When the number of elements in the stack B satisfies the number of parameters required by the top operator of the stack a, Node merging is required, that is, a new computation Node is constructed, as shown in (B) and (c) in fig. 2, in the traversal process, a new computation Node1 is first created, grid tile metadata nodes B1 and B2 are assigned to the computation Node1, and a matched function name "sub" is found from a custom function library by an operator "-" at the top of the stack a, and is assigned to the computation function attribute of the Node1, and the operator "-" is removed from the stack a, and the newly created computation Node1 is placed in the stack B. Then, the expression string is further traversed, the operators "/" and "+" are placed in the stack a in turn, and the grid tile metadata nodes B3 and B4 are placed in the stack B in turn, at which time the function parameters required by the operator "+" at the top of stack a are satisfied, so that the grid tile metadata nodes B3 and B4 are assigned to the compute Node2, and the operator "+" at the top of stack a finds the matched function name "add" from the custom function library to the compute function attribute of Node2, while the operator "+" is removed from stack a, and the newly created Node2 is placed in stack B as shown in fig. 2 (d) and (e). Finally, the whole character string has been traversed, the function parameters required by the operator "/" have been satisfied, so the remaining nodes of stack a and stack B need to be finally merged, that is, the computation Node1 and the computation Node2 are assigned to the computation Node3, and the operator "/" at the top level of stack a finds the matched function name "div" from the customized function library and assigns the matched function name "div" to the computation function attribute of the Node3, as shown in fig. 2 (f), only one computation Node3 is finally left in stack B, that is, the construction of the workflow task information flow is completed.
After the modeling work of the calculation workflow is completed through the step 2), an execution flow among the workflow nodes is defined. Recursion is a widely used algorithm in programming languages, and is characterized by a finite logic to define an infinite set of objects. In the execution process of the workflow, a recursion form is adopted, and the processing is carried out among the task nodes according to the execution sequence of 'from back to front' and then 'from front to back'. Interpolation reconstruction is required depending on step S3. The workflow of the abstract modeling comprises grid metadata nodes and computing nodes, wherein the grid metadata nodes contain information required for acquiring current tiles and are responsible for acquiring grid tile data in a map service request from cloud storage, and the computing nodes have function names and computing parameter attributes, are not only responsible for task execution in the current nodes, but also participate in supporting driving of workflow task links, and comprise data transmission, recursion and other operations. The recursive execution of the workflow described above is described in detail below.
And 3) triggering from the last calculation node in the workflow, calling the function name and the function parameter encapsulated in the calculation node, and executing the workflow in a recursive mode to obtain a calculated tile result after the workflow is executed.
Fig. 3 is a schematic diagram of the workflow executed by converting the above computational expression into three computational nodes, where the workflow can automatically transfer parameters, data or execute tasks according to a preset function, embodies the execution steps and dependency relationships between the computational nodes, and forms a directed acyclic graph between the computational nodes.
A specific execution process of the computing workflow is described with reference to fig. 3, wherein (i) - (ii) represent the first six operations in the execution process, and the specific operation steps are as follows:
s31: obtaining the last calculation Node3 to execute calculation, calculating the parameters Node1 and Node2 by loop iteration, judging the calculation parameter type is the Node type, needing parameter replacement operation, and executing step S32.
S32: performing the calculation of the Node1, performing loop iteration calculation on the parameters MetaData1 and MetaData2, judging that the calculation parameter type is a MetaData (MetaData) type, acquiring the image data in the map tile from a cloud data source according to the raster tile information recorded in the raster tile MetaData nodes B1 and B2, returning the result to give the function parameter of the current Node, completing the parameter replacement operation of the Node1 function, and performing step S33.
S33: at this time, the function parameter (params) of Node1 is the raster tile data, the sub function is called according to the function (func) name to perform the operation, the calculation result is returned to complete the function parameter replacement operation of Node3, at this time, the calculation parameter Node1 in Node3 is replaced with the execution result of Node1, and step S34 is executed.
S34: similarly, referring to step S32 and step S33, the parameter replacement operation of the Node2 in the Node3 is completed in a similar manner, and after all the calculation parameters are replaced, the div function in the Node3 is executed to finally obtain the operation result of the workflow, which is recorded as the grid tile
。
For the grid tile metadata nodes B1, B2, B3, and B4, acquiring image data in map tiles from a cloud data source according to grid tile information recorded in the nodes is mainly realized by the following specific method:
31) reading metadata information of cloud images and defining the effective space range of the imagesBBox1Converting into space range under WGS coordinate systemBBox2。
32) Spatial range corresponding to tile triple coordinate (X, Y, Z) of map service requestBBox3AndBBox2to carry outAnd judging the space intersection, namely judging whether the requested tile belongs to the area of the image, and if the judgment result is that the subsequent steps 33) -35) are selected to be executed, otherwise, stopping executing the workflow and returning error information to the client to remind the user that the requested data range exceeds the effective range.
33) Is calculated by the following formulaBBox3The spatial range is mapped to a pixel Region range Region of the cloud image, namely a corresponding interest Region range is requested:
in the formula (I), the compound is shown in the specification,transform -1 is the inverse of the image affine transformation matrix (
) Is thatBBox3Four to the coordinate of (A), (B), (C
，
) The space range corresponding to the current map service request is at the pixel row number and column number position in the cloud image.
34) According to the organization format of the COG, reading the effective data segment range of the pixel Region range Region in the COG, and acquiring the raster data in the current interest Region range from the cloud.
35) The grid data obtained from the cloud is normalized to 256 × 256 tile data using nearest neighbor interpolation, and returned for replacement grid tile metadata nodes.
And 4) carrying out numerical value stretching on the operation result of the workflow, and rendering by applying a proper pattern to obtain a static picture.
The specific method for grid tile visual rendering in this embodiment is as follows:
s41: the grid tile obtained by calculation
Carrying out numerical linear stretching to obtain the grid tile
And mapping the value interval to 0-255, wherein the linear stretching formula is as follows:
in the formula (I), the compound is shown in the specification,vthe pixel values representing the grid tile data before stretching,v min represents the minimum of the entire grid tile data before stretching,v max representing the maximum value of the entire grid tile data before stretching,wrepresenting pixel values of the stretched grid tile data;
s42: due to the stretched grid tilest 2 The value interval is between 0 and 255, and a coloring array containing RGB triples is generated according to the name of the color chart stylecolormapThe array size is 256, the first element in the array represents the color to be rendered when the tile pixel value is 0, the second element represents the color to be rendered when the tile pixel value is 1, and so on.
S43: according tocolormapIs related to the grid tile
And rendering is carried out, a static picture png is generated and returned to the client, and the real-time analysis and processing of the whole workflow are completed.
Compared with the original image, the vegetation normalization index NDVI calculated by the Landsat 8 image shows that the method provided by the invention models expression conversion into a calculation workflow, realizes real-time processing on raster tiles in a request of a map service, quickly analyzes massive remote sensing data stored in a cloud end, realizes direct visual conversion from the original data to a data product, and has very important practical application value for extraction, sharing and quick application of remote sensing information.
The above-described embodiments are merely preferred embodiments of the present invention, which should not be construed as limiting the invention. Various changes and modifications may be made by one of ordinary skill in the pertinent art without departing from the spirit and scope of the present invention. Therefore, the technical scheme obtained by adopting the mode of equivalent replacement or equivalent transformation is within the protection scope of the invention.
Claims (10)
1. A real-time analysis processing method combined with map service is used for analyzing and visually converting remote sensing image data stored in a cloud in a map service mode, and is characterized by comprising the following steps of:
s1: acquiring a map service request submitted by a client, analyzing a calculation expression of the map service request, and modeling the map service request into a workflow consisting of grid tile metadata nodes and calculation nodes according to the priority of operation in the expression; the grid tile metadata node is a mapping table for recording grid tile information; the computing nodes are packaged with function names and function parameters corresponding to operators and used for executing computing tasks in the computing nodes and participating in the carrying drive of workflow task links, wherein the function parameter type is at least one of grid tile metadata nodes and the computing nodes;
s2: triggering from the last calculation node in the workflow, calling the function name and the function parameter packaged in the calculation node, and executing the workflow recursively to complete the processing of the image tile; in the workflow execution process, for any current calculation node, if all function parameter types required by the current calculation node are grid tile metadata nodes, acquiring grid tile data from a cloud according to grid tile information recorded in the grid tile metadata nodes and obtaining an operation result through function operation, and if the function parameter types required by the current calculation node contain internal calculation nodes, executing the internal calculation node firstly, replacing the internal calculation nodes with the operation result and then continuing to execute the function operation of the current calculation node;
s3: and performing visual rendering on the operation result of the last calculation node in the workflow in a tile format, and returning the operation result to the client.
2. The real-time analysis processing method combined with map service according to claim 1, characterized in that: in the step S1, for each map service request, modeling of a workflow is performed according to S11 to S12:
s11: acquiring a computational expression representing the computational requirement of image data from the map service request, wherein the computational expression consists of an operator and image data information participating in operation;
s12: the server analyzes the received computational expression in real time, and carries out workflow modeling by traversing character strings of the computational expression and combining two pre-constructed stack data structures, wherein a stack A is used for storing an operator, and a stack B is used for storing nodes in a workflow; in the traversal process, if the character represents image data information participating in operation, the character is used as a raster tile metadata node and is placed in a stack B, and if the character represents an operator, the character is placed in a stack A according to the priority of the operator; when the elements in the stack B meet the parameters required by the operator with the highest priority in the stack A, combining and packaging the operator and the grid tile metadata nodes corresponding to the required parameters into a new calculation node to be stored in the stack B; after the whole calculation expression is traversed, a workflow consisting of grid tile metadata nodes and calculation nodes is finally obtained, and the last node of the workflow is a calculation node; the grid tile metadata node is a mapping table for recording grid tile information, and comprises a data storage path for acquiring the grid tile, a space triple coordinate and required wave band information.
3. The real-time analysis processing method combined with map service according to claim 1, characterized in that: the specific method of step S2 is as follows:
s21: acquiring the last calculation node of the workflow as the current calculation node;
s22: calling function names and function parameters packaged in current calculation nodes, traversing each function parameter, and directly obtaining an operation result through function operation if the function parameters required by the function operation are all raster tile data; if at least one function parameter is not the raster tile data, parameter replacement operation is required to be executed firstly, the raster tile data corresponding to the execution result is replaced to the corresponding function parameter position, and after all the function parameters required by the function operation are replaced by the raster tile data, the operation result is obtained through the function operation;
the parameter replacement operation needs to select A) or B) according to the type of the function parameter:
A) if the function parameter is a grid tile metadata node, acquiring grid tile data from a cloud according to grid tile information recorded in the grid tile metadata node and replacing the grid tile metadata node;
B) if the function parameter is a calculation node, temporarily taking the calculation node corresponding to the function parameter as a new current calculation node, re-executing S22 and returning the operation result to perform parameter replacement on the calculation node.
4. The real-time analysis processing method combined with map service according to claim 3, characterized in that: in S2, the specific method for obtaining raster tile data from the cloud according to the raster tile information recorded in the raster tile metadata node is as follows:
a1: reading metadata information of the cloud image, converting a space effective range of the cloud image and a space range of grid tiles required in the map service request into a uniform coordinate system, judging whether the requested grid tiles are located in an area of the cloud image or not through a space intersection relation, if so, executing A2, otherwise, terminating execution of a workflow and returning error information to the client;
a2: mapping the space range of the grid tiles required in the map service request to the cloud image to obtain a corresponding pixel area range, then reading the effective data segment range of the pixel area range in the cloud image according to the organization format of the cloud image, acquiring grid data corresponding to the map service request from the cloud, and normalizing the grid tile data into 256 × 256.
5. The real-time analysis processing method combined with map service according to claim 1, characterized in that: the specific method of step S3 is as follows:
s31: performing numerical linear stretching on the grid tile corresponding to the operation result of the last calculation node in the workflow, and mapping the value interval to 0-255 to obtain the stretched grid tile;
s32: rendering and coloring the stretched raster tiles according to the specified color chart style, and returning rendering results to the client in a picture form.
6. The real-time analysis processing method in combination with a map service according to claim 4, characterized in that: the cloud image is in a COG data format.
7. The real-time analysis processing method in combination with a map service according to claim 4, characterized in that: the unified coordinate system is a WGS coordinate system.
8. The real-time analysis processing method combined with map service according to claim 5, characterized in that: in S32, a rendering array with a size of 256 and containing RGB triples is generated for the tile according to the designated color map pattern, the stretched raster tile is rendered by the rendering array, and a png format still picture is generated and returned to the client.
9. A real-time analysis processing apparatus in combination with a map service, comprising a memory and a processor;
the memory for storing a computer program;
the processor, when executing the computer program, is configured to implement the real-time analysis processing method in combination with the map service according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, is able to carry out a real-time analysis processing method in combination with a map service according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110723784.0A CN113254568B (en) | 2021-06-29 | 2021-06-29 | Real-time analysis processing method, device and medium combined with map service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110723784.0A CN113254568B (en) | 2021-06-29 | 2021-06-29 | Real-time analysis processing method, device and medium combined with map service |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113254568A true CN113254568A (en) | 2021-08-13 |
| CN113254568B CN113254568B (en) | 2021-10-08 |
Family
ID=77190011
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110723784.0A Active CN113254568B (en) | 2021-06-29 | 2021-06-29 | Real-time analysis processing method, device and medium combined with map service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113254568B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115510173A (en) * | 2022-09-27 | 2022-12-23 | 清华大学 | DAS-based intelligent thematic map making method, device and product |
| CN116594977A (en) * | 2023-05-10 | 2023-08-15 | 阿里巴巴达摩院(杭州)科技有限公司 | Distributed processing system for remote sensing data |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502594A (en) * | 2019-07-10 | 2019-11-26 | 武汉数趣信息科技有限公司 | A kind of Remote sensing photomap tile real time fusion and online updating method |
| CN110599490A (en) * | 2019-09-02 | 2019-12-20 | 广州市城市规划勘测设计研究院 | Remote sensing image data storage method and system |
| CN111723221A (en) * | 2020-06-19 | 2020-09-29 | 珠江水利委员会珠江水利科学研究院 | Mass remote sensing data processing method and system based on distributed architecture |
-
2021
- 2021-06-29 CN CN202110723784.0A patent/CN113254568B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502594A (en) * | 2019-07-10 | 2019-11-26 | 武汉数趣信息科技有限公司 | A kind of Remote sensing photomap tile real time fusion and online updating method |
| CN110599490A (en) * | 2019-09-02 | 2019-12-20 | 广州市城市规划勘测设计研究院 | Remote sensing image data storage method and system |
| CN111723221A (en) * | 2020-06-19 | 2020-09-29 | 珠江水利委员会珠江水利科学研究院 | Mass remote sensing data processing method and system based on distributed architecture |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115510173A (en) * | 2022-09-27 | 2022-12-23 | 清华大学 | DAS-based intelligent thematic map making method, device and product |
| CN116594977A (en) * | 2023-05-10 | 2023-08-15 | 阿里巴巴达摩院(杭州)科技有限公司 | Distributed processing system for remote sensing data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113254568B (en) | 2021-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9905012B2 (en) | Segmentation of an image based on color and color differences | |
| CN113254568B (en) | Real-time analysis processing method, device and medium combined with map service | |
| US8959003B2 (en) | Interactive data visualization for trend analysis | |
| Telea et al. | Image‐based edge bundles: Simplified visualization of large graphs | |
| EP4120199A1 (en) | Image rendering method and apparatus, and electronic device and storage medium | |
| CN112287958A (en) | Classifying colors of objects in digital images | |
| JP2004038926A (en) | Texture map editing | |
| WO2021232609A1 (en) | Semantic segmentation method and system for rgb-d image, medium and electronic device | |
| US20220327769A1 (en) | Neural light transport | |
| US20230360285A1 (en) | Generating differentiable procedural materials | |
| CN106611056B (en) | Method and device for cutting three-dimensional simulation map | |
| CN107870949A (en) | Data analysis job dependence relation generation method and system | |
| CN114417064A (en) | Data processing method and device, computer equipment and storage medium | |
| CN115292951A (en) | Intelligent planning design auxiliary system for urban and rural updating planning design | |
| CN109255004A (en) | The online interaction formula method for visualizing of accuracy guarantee towards geographical spatial data | |
| CN112308939A (en) | Image generation method and device | |
| CN113254000B (en) | Method, device, storage medium and equipment for generating 2D scene based on JSON data | |
| US10055811B2 (en) | System and method for generating interactive 2D projection of 3D model | |
| KR101818745B1 (en) | Drawing and product management system, and Controlling method therefor | |
| CN116630139A (en) | Method, device, equipment and storage medium for generating data | |
| JP2002024298A (en) | Three-dimensional shape processor, three-dimensional shape displaying method, and storage medium stored with program for implementing the same method | |
| CN111696180A (en) | Method, system, device and storage medium for generating virtual dummy | |
| CN117271936B (en) | Web-based big data relation map rendering method, system and equipment | |
| CN116883575B (en) | Building group rendering method, device, computer equipment and storage medium | |
| EP2846269A1 (en) | Connector for a BI (Business Intelligence) platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |