CN116486030A - Modeling method and related device of three-dimensional geologic body model based on surface image - Google Patents

Modeling method and related device of three-dimensional geologic body model based on surface image Download PDF

Info

Publication number
CN116486030A
CN116486030A CN202310542068.1A CN202310542068A CN116486030A CN 116486030 A CN116486030 A CN 116486030A CN 202310542068 A CN202310542068 A CN 202310542068A CN 116486030 A CN116486030 A CN 116486030A
Authority
CN
China
Prior art keywords
dimensional
features
feature
modeling
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310542068.1A
Other languages
Chinese (zh)
Inventor
刘镇
兰春晖
周翠英
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sun Yat Sen University
Original Assignee
Sun Yat Sen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sun Yat Sen University filed Critical Sun Yat Sen University
Priority to CN202310542068.1A priority Critical patent/CN116486030A/en
Publication of CN116486030A publication Critical patent/CN116486030A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05Geographic models
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/044Recurrent networks, e.g. Hopfield networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/0464Convolutional networks [CNN, ConvNet]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/0475Generative networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/094Adversarial learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/46Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462Salient features, e.g. scale invariant feature transforms [SIFT]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00Indexing scheme for image data processing or generation, in general
    • G06T2200/04Indexing scheme for image data processing or generation, in general involving 3D image data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00Indexing scheme for image data processing or generation, in general
    • G06T2200/08Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02ATECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/30Assessment of water resources

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Remote Sensing (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the invention discloses a modeling method and a related device of a three-dimensional geologic body model based on a surface image. The method comprises the following steps: constructing a two-dimensional image dataset; extracting features of each scale of the image by using a Gaussian pyramid based on a scale invariant feature transformation algorithm through a two-dimensional encoder to obtain two-dimensional features; the projection thought is utilized to realize the conversion from two-dimensional characteristics to three-dimensional characteristics, and the three-dimensional characteristics are constructed for modeling; based on the three-dimensional features established through conversion, a three-dimensional transcoder is constructed, convolution calculation is carried out on the three-dimensional vertex feature matrixes, and the new position and the new three-dimensional feature of each feature matrix vertex are predicted; and establishing a geologic body grid unit by using an implicit modeling method in combination with the sparse drilling data, and establishing association between the grid unit and the vertex position of the feature matrix and the new three-dimensional feature to obtain the target three-dimensional geologic body model. The modeling difficulty caused by insufficient related drilling or geological data can be solved, and the accuracy of the geological model is improved.

Description

Modeling method and related device of three-dimensional geologic body model based on surface image
Technical Field
The invention belongs to the field of computer vision, and particularly relates to a modeling method and a related device of a three-dimensional geologic body model based on a surface image.
Background
The three-dimensional geological model can intuitively display the information of geological structure, spatial topology, lithology and the like of stratum, not only provides accurate information for the work of analyzing geological structure, fault distribution and the like of researchers, but also provides reliable basis for development and utilization of underground space. The traditional three-dimensional geological modeling technology is mainly based on geological drilling data and other data obtained by geological exploration, is limited by the factors of high geological exploration data acquisition cost, high acquisition difficulty, limited data quantity, lack of integrity and the like, and the traditional three-dimensional geological modeling technology based on drilling data is difficult to meet the requirement of guiding actual geological engineering.Is an interdisciplinary subject related to the fields of artificial intelligence, neurobiology, computer science, image processing, pattern recognition and the like. Computer vision mainly uses a computer to simulate the visual function of a person, extracts information from an image of an objective object, processes and understands the information, and is finally used for actual detection, measurement and control. The computer vision technology has the greatest characteristics of high speed, large information quantity and multiple functions.
The three-dimensional reconstruction technology is based on a computer vision technology, and the real scene is characterized into a mathematical model conforming to the logical expression of a computer through the processes of depth data acquisition, preprocessing, point cloud registration and fusion, surface generation and the like, so that a three-dimensional entity model conforming to the target expectation is finally constructed. The application of the current three-dimensional reconstruction technology in the field of geologic modeling mainly comprises the steps of constructing a three-dimensional geologic model by using an image set as a data source and recovering depth information according to a camera imaging principle. The method directly utilizes the two-dimensional image to reconstruct three-dimensionally, uses common data acquisition equipment such as a camera, a mobile phone and the like, has lower reconstruction cost, and has simple and convenient image data acquisition mode, but the prior technical difficulty is that:
(1) The two-dimensional image loses the information of the depth direction, and the process of converting the two-dimensional image features into the three-dimensional features is difficult to completely retain and extract all the information, so that the modeling accuracy is reduced.
(2) The three-dimensional geologic modeling process based on the two-dimensional surface image only can greatly increase the calculated amount and the related super-parameter amount along with the increase of the depth of the training network, the 3D convolutional neural network and the dense network can further bring about exponential increase of the data amount, and difficulties are brought to the training and the application of the modeling network.
Therefore, a three-dimensional geologic modeling technique is needed that can improve geologic modeling accuracy without placing excessive load on the training network.
Disclosure of Invention
In view of the above, the embodiment of the invention provides a modeling method and a related device for a three-dimensional geologic body model based on a surface image, which are used for solving the technical problems that the traditional three-dimensional geologic modeling technology based on drilling data is difficult to meet the requirement of guiding actual geologic engineering due to the fact that the actual three-dimensional geologic modeling is limited by the factors of high acquisition cost, large acquisition difficulty, limited data quantity, lack of integrity and the like of geologic exploration data.
The first aspect of the invention provides a modeling method of a three-dimensional geologic body model based on a surface image, which comprises the following steps:
and 1, obtaining pseudo two-dimensional surface images by generating two-dimensional surface image data and performing countermeasure network training, and constructing a two-dimensional image data set.
In step 1, a two-dimensional surface image dataset is arranged, which can be an image dataset formed by a single picture or a plurality of pictures, and a multi-component complete two-dimensional image dataset is constructed by generating a countermeasure network (GAN) training to obtain more pseudo two-dimensional surface images.
Since the number of surface images is limited, the angles are relatively single, and it is necessary to expand multiple angles based on machine learning means, a sufficient number of image datasets. The condition countermeasure generating network comprises a generating network G and a judging network D, the generating network G learns the input image data, the judging network D discriminates the learning effect, and the generating network G and the judging network D play a game mutually until the discrimination of the final judging network D is true, so that a new two-dimensional surface image is generated.
Step 2, extracting features of each scale of the image by using a Gaussian pyramid through a two-dimensional encoder based on a scale invariant feature conversion algorithm to obtain two-dimensional features;
in the step 2, the features of each Scale of the image are extracted by a two-dimensional encoder based on a Scale-invariant feature transform algorithm (Scale-invariant feature transform, SIFT) by using a gaussian pyramid. The gaussian pyramid has the following criteria: extracting image features by adopting a downsampling mode; and (3) convolving the image by using a Gaussian kernel (Gaussian filtering), and removing even rows and columns in the convolution result to enable the downsampling result to be 1/4 of the original image size.
Step 3, realizing the conversion from two-dimensional features to three-dimensional features by utilizing a projection thought, and reserving and integrating the extracted image features by utilizing a cyclic neural network to construct three-dimensional features for modeling;
in the step 3, the image extraction features are two-dimensional, so that the dimension is increased to three-dimensional, and the completeness and the authenticity of the two-dimensional features of the original image are reserved. Based on the method, the two-dimensional to three-dimensional feature conversion is realized by utilizing a projection thought, the extracted image features are reserved and integrated by utilizing a cyclic neural network (RNN), and the three-dimensional features are constructed for modeling.
Step 4, constructing a three-dimensional code converter based on the three-dimensional features established through conversion, carrying out convolution calculation on the three-dimensional vertex feature matrixes, and predicting the new position and the new three-dimensional feature of each feature matrix vertex;
and 4, after the conversion of the image features from two dimensions to three dimensions is completed, constructing a three-dimensional transcoder, carrying out convolution calculation on the three-dimensional vertex feature matrixes, and predicting the new positions and the three-dimensional features of the vertexes of each feature matrix. And carrying out preliminary visual construction of the three-dimensional geological model by the new three-dimensional features.
And 5, combining sparse drilling data, establishing a geologic body grid unit by using an implicit modeling method, and establishing association between the grid unit and the vertex position of the feature matrix and new three-dimensional features to obtain a target three-dimensional geologic body model.
In step 5, in view of the difficulty in accurately expressing stratum layering and thickness information of the geological model, a geological body grid unit is established by using an implicit modeling method in combination with limited sparse drilling data, and the association between the grid unit and the vertex position and the three-dimensional feature of the feature matrix is established, so that a final three-dimensional geological body model is obtained.
Optionally, in the step 3, the converting the two-dimensional feature to the three-dimensional feature by using a projection concept specifically includes:
and realizing the feature conversion from two-dimensional features to three-dimensional features by using an Image2 Mesh-RNN joint converter, wherein the joint converter is formed by an Image2Mesh and a circulating neural network.
The invention performs function realization based on Pytorch deep learning framework under Python language, and relates to a conditional antagonism generation neural network (GAN); a two-dimensional and three-dimensional graph convolutional neural network (2D/3D GCN); support vector machine classifier.
The second aspect of the present invention provides a modeling apparatus for a three-dimensional geologic body model based on a surface image, comprising:
the construction unit is used for obtaining pseudo two-dimensional surface images by generating the two-dimensional surface image data and performing countermeasure network training, and constructing a multi-element complete two-dimensional image data set;
the feature extraction unit is used for extracting features of each scale of the image by using a Gaussian pyramid through a two-dimensional encoder based on a scale-invariant feature conversion algorithm;
the feature conversion unit is used for realizing two-dimensional to three-dimensional feature conversion by utilizing a projection thought, and preserving and integrating the extracted image features by utilizing a cyclic neural network to construct three-dimensional features for modeling;
the computing unit is used for constructing a three-dimensional transcoder based on the three-dimensional features established through conversion, carrying out convolution computation on the three-dimensional vertex feature matrixes and predicting the new position and the three-dimensional feature of each feature matrix vertex;
the modeling unit is used for combining sparse drilling data, establishing a geologic body grid unit by using an implicit modeling method, and establishing association between the grid unit and the vertex position and the three-dimensional feature of the feature matrix to obtain a target three-dimensional geologic body model.
Optionally, the two-dimensional surface image data in the building unit includes: a single two-dimensional earth surface image data or an image set made up of a plurality of two-dimensional earth surface image data.
Optionally, the criterion features of the gaussian pyramid in the feature extraction unit specifically include: and extracting image features in a downsampling mode, convoluting the image by using Gaussian filtering, and removing even lines and columns in the convolution result to enable the downsampling result to be 1/4 of the original image size.
Optionally, in the feature conversion unit, the implementation of feature conversion from two-dimensional features to three-dimensional features by using a projection concept specifically includes:
and realizing the feature conversion from two-dimensional features to three-dimensional features by using an Image2 Mesh-RNN joint converter, wherein the joint converter is composed of the Image2Mesh and a circulating neural network.
A third aspect of the present invention provides an electronic device, comprising: including a memory, a processor, and one or more programs;
the one or more programs are stored in the memory;
the processor, when executing the one or more programs, causes the electronic device to implement a method of modeling a three-dimensional geologic volume model based on a surface image as described in any of the first aspects.
A fourth aspect of the invention provides a computer readable storage medium comprising instructions which, when run on an electronic device, cause the electronic device to perform the method of modeling a three-dimensional geological volume model based on a surface image as defined in any one of the first aspects.
The invention has the following advantages:
(1) The conditional challenge-generating network image generation technique is utilized to provide a sufficient two-dimensional image dataset for the creation of a three-dimensional geologic model.
(2) The Gaussian pyramid is adopted to extract the image features in all directions and at multiple angles, so that the problem that the traditional technology cannot extract the image features comprehensively and singly and influences the establishment of a subsequent model is avoided.
(3) By adopting a convolutional neural network-cyclic neural network (CNN-RNN) coupling method, the characteristics of each dimension of the two-dimensional image input into the converter from the two-dimensional encoder can be reserved, the three-dimensional grid characteristics are dynamically updated in real time according to the input, and the characteristic fusion of each dimension is achieved, so that the construction of a precise model is realized.
(4) An implicit modeling method based on sparse drilling data is introduced, the geologic body modeling method based on the surface two-dimensional image is coupled, and conditions are provided for refinement, comprehension and accurate establishment of a three-dimensional geologic model.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:
FIG. 1 illustrates a flow chart of the present invention for generating a two-dimensional surface image based on a conditional antagonism neural network.
FIG. 2 is a schematic diagram of a two-dimensional encoder network model according to an embodiment of the present invention.
FIG. 3 is a schematic diagram of a bilinear interpolation algorithm according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of a projection-based converter according to an embodiment of the invention.
Fig. 5 is a flow chart of a gating loop network according to an embodiment of the present invention.
Fig. 6 is a general flow chart of an embodiment of the present invention.
FIG. 7 is a schematic diagram of an implicit modeling three-dimensional grid generation flow of an embodiment of the present invention.
Detailed Description
For the purposes, technical solutions and advantages of the present application, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The embodiment of the invention provides a modeling method and a related device for a three-dimensional geologic body model based on a surface image, which are used for solving the technical problems that the traditional three-dimensional geologic modeling technology based on drilling data is difficult to meet the requirement of guiding actual geologic engineering due to the factors of high acquisition cost, large acquisition difficulty, limited data and data quantity, lack of integrity and the like of the actual three-dimensional geologic modeling limited by geologic exploration data.
For the purpose of clearly showing the invention, technical solutions and advantages, the following description of specific implementation steps of the invention refers to the accompanying drawings.
The modeling method of the three-dimensional geologic body model based on the surface image in the embodiment comprises the following steps:
step 1, a two-dimensional surface image dataset is arranged, which can be an image dataset formed by a single picture or a plurality of pictures, and a multi-element complete two-dimensional image dataset is constructed by generating a countermeasure network (GAN) training to obtain more pseudo two-dimensional surface images. Since the number of surface images is limited, the angles are relatively single, and it is necessary to expand multiple angles based on machine learning means, a sufficient number of image datasets. The condition countermeasure generating network comprises a generating network G and a judging network D, the generating network G learns the input image data, the judging network D discriminates the learning effect, and the generating network G and the judging network D play a game mutually until the discrimination of the final judging network D is true, so that a new two-dimensional surface image is generated. The specific operation of the step 1 is as follows:
(1) The method comprises the steps of inputting partial image data in the existing image data set as noise data into a generation network in an countermeasure generation network to generate a preliminary pseudo-two-dimensional surface image, randomly inputting partial image data from the existing image data set as a training set, and jointly inputting two types of image data into a discrimination network as a real surface image to perform classification discrimination.
In particular, for two-dimensional image data, convolutional neural networks should be employed in the generator herein to extract image features.
(2) And defining a loss function, providing an optimized target for training the GAN network model, enabling a gradient back propagation algorithm to be realized, and further measuring the difference between the training model output and the true value. It should be noted that the choice of the loss function is related to the convergence degree of the model, and different types of loss functions need to be evaluated. The evaluation was selected as follows:
and evaluating the square loss function, the cross entropy loss function and the BEC loss function respectively, performing loss calculation according to the generated image data characteristics and the real image characteristic labels, and finally determining the BEC loss function as a final loss function to select.
In connection with the challenge-generating network training flow of fig. 1, the operations of this step can be summarized as follows: image data input, neural network output, output and label loss calculation, gradient feedback updating model parameters.
It is particularly noted that the only purpose of this step flow is to generate new surface images using existing limited two-dimensional surface image datasets, providing complete and adequate image samples for subsequent neural network training.
Step 2, extracting features of each Scale of the image by using a Gaussian pyramid based on a Scale-invariant feature transform algorithm (Scale-invariant feature transform, SIFT) through a two-dimensional encoder. The gaussian pyramid has the following criteria: extracting image features by adopting a downsampling mode; the image is convolved using a gaussian kernel (gaussian filter) and the even rows and columns in the convolution result are removed. Downsampling is performed continuously to reduce the dimensions of the image and extract the most useful features. The specific details of step 2 are as follows:
(1) The CNN convolutional neural network is a feedforward neural network which comprises convolutional calculation and has a depth structure, has characteristic learning capability and mainly comprises an input layer, a hidden layer and an output layer. The method is characterized in that a biological nervous system is imitated, convolution calculation is carried out by using a convolution kernel, neuron synapses are simulated by using an activation function, training speed and effect of a neural network are improved, input information can be classified in a translation unchanged mode according to a hierarchical structure of the neural network, image graphic features are extracted, and image reconstruction and analysis are carried out.
(2) The main purpose of a two-dimensional encoder is to acquire image features of a two-dimensional image. The traditional three-dimensional reconstruction algorithm uses a SIFT or monocular camera three-dimensional reconstruction algorithm (Structure from motion, SFM) for feature extraction, whereas the SIFT feature extraction algorithm mainly uses a gaussian pyramid for feature extraction of each scale of the image.
(3) Based on a two-dimensional encoder and a SIFT algorithm, the method obtains characteristic values of all scales by referring to a Gaussian pyramid model of the traditional three-dimensional reconstruction. To achieve back propagation of the network, convolution calculations are used instead of gaussian convolutions in the image gaussian pyramid. As shown in fig. 2, the input two-dimensional surface image is subjected to 6-level feature extraction by using a two-dimensional encoder, 256×256 pixel images are taken as input, feature values are respectively acquired on six scales of 256×256, 128×128, 64×64, 32×32, 16×16 and 8×8, 4 times of convolution calculation are performed on each dimension, and a convolution kernel (filter) is a matrix of 3×3. The extracted dimensions are 16,32,64,128,256, 512, respectively, using the Relu activation function.
And step 3, the two-dimensional features extracted from the image are upscaled to be three-dimensional, and the completeness and the authenticity of the two-dimensional features of the original image are reserved. Specifically, the projection thought is utilized to realize the feature conversion from two dimensions to three dimensions, and the extracted image features are reserved and integrated by using a cyclic neural network (RNN) to construct three-dimensional features for modeling. The specific description and details of step 3 are as follows:
and converting the two-dimensional Image characteristics into three dimensions by using an Image2mesh_RNN joint converter, wherein the joint converter consists of two parts of an Image2Mesh and a Recurrent Neural Network (RNN).
Image2Mesh can realize conversion from two-dimensional features to three-dimensional features of an Image. Initializing and constructing a three-dimensional grid sphere to obtain the relation among points and corresponding three-dimensional coordinate points, and performing dimension conversion by utilizing a projection relation, wherein the relation is represented by the following formula:
wherein: x, Y, Z represent the X-axis, Y-axis, Z-axis coordinates of the three-dimensional grid vertices respectively;andrespectively representing the length and width of the two-dimensional characteristics;andrepresenting the abscissa and ordinate offsets of the two-dimensional feature.
Furthermore, the three-dimensional to two-dimensional feature map in the projective transformation has decimal places, and the two-dimensional feature coordinates are integers, which can cause obvious aliasing effects on the image. In order to improve the accuracy of the image characteristics, the invention adopts bilinear interpolation method to calculate the transformation coordinates with the combination of figures 3 and 4And (3) withThe corresponding eigenvalues are described in detail below with respect to the two-wire interpolation method:
bilinear interpolation is a linear interpolation extension of an interpolation function with two variables, and the core idea is to perform linear interpolation once in two directions respectively. The principle is that the pixel value of the point to be inserted takes the linear interpolation in the horizontal and vertical directions of the adjacent 4 point pixel values in the original image, namely, corresponding weights are determined according to the distances between the point to be sampled and the surrounding 4 adjacent points, so that the pixel value of the point to be sampled is calculated.
Specifically, a known functionAt the pointThe value at point P is further obtained. First, linear interpolation in X direction is performed inAndis inserted intoIn the followingAndis inserted intoThe method comprises the steps of carrying out a first treatment on the surface of the A second step of linear interpolation in the Y direction, calculated by the first stepAndinterpolation calculation to obtain the function value of P point
In particular the number of the elements to be processed,representing 4 pixels of the image that are known. The result of the linear interpolation is independent of the order of interpolation. The result is the same by first performing interpolation in the Y direction and then performing interpolation in the X direction. The result of bilinear interpolation is independent of which direction interpolation was first performed.
Three-dimensional characteristics can be obtained after extractionA matrix of dimensions is provided which,2252 is the number of vertices for the dimension at each scale, i.eThe method comprises the following steps of: 8,16,32,64,128, 256. At each scaleThe dimension matrix is used as an input for each moment of the recurrent neural network.
In particular, since the conventional Convolutional Neural Network (CNN) or Graph Neural Network (GNN) is difficult to fuse multi-scale Image feature data, the invention adds a gating cyclic network (Gated reccurent unit, GRU) in the Image2Mesh converter to circularly retain the extracted Image features. After the two-dimensional features of the images under each scale are expanded into three-dimensional features through Image2Mesh, the three-dimensional features are input as features of the circulating neural network, and finally integrated three-dimensional features are output. Specific details of the cyclic network are described below:
the cyclic neural network (RNN) is used to process sequence data, where a current output of a sequence in the network is associated with a previous output, in that the network will memorize the previous information and apply it to the calculation of the current output, i.e. the nodes between hidden layers are no longer connectionless but connected, and the input of the hidden layers includes not only the output of the input layer but also the output of the hidden layer at the previous time.
In particular, the gated loop network (GRU) is an upgrade optimization of the loop neural network, and can better capture the dependency relationship on the sequence with long time step distance, wherein the reset gate helps to capture the short-term dependency relationship in the sequence, and the update gate helps to capture the long-term dependency relationship in the sequence. When the reset gate is opened, the gate control circulation unit comprises a basic circulation neural network; the gate-controlled loop unit may skip the sub-sequence when the update gate is open. Specifically, the update gate and reset gate are detailed as follows:
the update gate helps the model to decide how much past information to transfer to the future, or how much information from the previous and current time steps needs to be transferred, so that the model can decide to copy all information from the past to reduce the risk of gradient disappearance, for controlling the extent to which state information from the previous moment is brought into the current state, the larger the value of the update gate is to say the more state information from the previous moment is brought into, the right multiplication weight matrix is respectively carried out on the information from the previous moment and the current moment, and then the added data is sent to the update gate, i.e. multiplied by the sigmoid function, to obtain a value between [0,1 ].
The reset gate is used for controlling how much historical information of the previous state is written into the current candidate set, the smaller the reset gate is, the less information of the previous state is written into, the same as the data processing of the update gate is, the information of the previous moment and the current moment are respectively multiplied by the weight matrix to the right, and then the added data are sent into the reset gate, namely multiplied by the sigmoid function, and the obtained value is between [0,1 ]. The value and usefulness of the weight matrix are different only twice.
Specifically, the invention is based on a gate-controlled loop network (GRU), as shown in figure 5, the hidden layer of the gate-controlled loop networkThe initial value of (1) is the spherical three-dimensional grid which is initialized and constructed, and the characteristics of the three-dimensional grid are continuously updated according to the input. In the cyclic neural network part, the three-dimensional network is paired by using a graphic neural network (GCN)The trellis data is convolved. Because the final required data is a three-dimensional grid in the hidden layer, the circulating neural network eliminates redundant output gates and reduces the number of parameters. The hidden layer of the loop network can be obtained as a 2252×3 matrix, namely an initialized three-dimensional grid model, and after 6 time period iterations, each input is respectively as follows: 8,16,32,64,128,256, which ultimately is a matrix of 1101 channels.
And 4, after the conversion of the image features from two dimensions to three dimensions is completed, constructing a three-dimensional transcoder, carrying out convolution calculation on the three-dimensional vertex feature matrixes, and predicting the new position and the three-dimensional feature of each feature matrix vertex. And performing preliminary visual construction of the three-dimensional geological model by the new three-dimensional features and the three-dimensional vertex positions. The specific details and descriptions are as follows:
the three-dimensional encoder is mainly used for carrying out convolution calculation on 1101-dimensional vertex feature matrixes output by the Image2mesh_RNN hidden layer and predicting the new position and three-dimensional feature of each vertex.
Specifically, the invention designs a 100-layer GCN convolution network, in order to effectively improve and accelerate the optimization process of a deep neural network and prevent gradient disappearance or explosion, a residual connection layer is added between every two standard network layers to construct 40 residual units, and all GCN convolution layers in the residual network are 190 channels for output. In order to increase the convergence rate, the invention uses a ReLU activation function. The last layer of output channels is 3, representing the three-dimensional coordinates of the predicted vertices.
In particular, in order to evaluate the deviation condition and the quality degree of the three-dimensional grid in the neural network calculation, the following loss function is used for judging:
(1) The Chamfer loss function is used for ensuring that the three-dimensional grid obtained by the neural network training is similar to the vertex of the real three-dimensional grid. Specifically, for each point in the set, find another point in the set that is closest to the determined neighbor set, and add the sum of squares of its distances, calculate as follows:
wherein:and (3) withRepresenting the vertex distances of the predicted three-dimensional mesh and the real three-dimensional mesh, respectively.
(2) The Normal loss function is used to consider the characteristic loss of the three-dimensional grid surface except the coordinates of the grid in the matrix representation of the three-dimensional grid.
(3) The Edge length loss function is used for preventing model discontinuity and unsmooth caused by local discontinuity of individual points.
And 5, combining limited sparse drilling data, establishing a geologic body grid unit by using an implicit modeling method, and establishing association between the grid unit and the vertex position and the three-dimensional feature of the feature matrix to obtain a final three-dimensional geologic body model. The specific details and descriptions are as follows:
the implicit modeling method converts geologic modeling problems into stratigraphic attribute classification problems of grid cells of the subsurface space. Taking the three-dimensional coordinates of each point in the modeling area as classification characteristic attributes, taking part of limited drilling data as a training set, taking a machine learning process training classifier as an intermediate medium, and classifying the modeling area by the trained classifier to obtain the geological three-dimensional grid of the area. Specifically, in connection with fig. 7, the steps are as follows:
(1) Preprocessing the drilling data, carrying out encryption processing on the drilling data according to the start-stop burial depths of each stratum category, and changing the geological data into a series of points with three-dimensional coordinates and stratum attributes in a unit of meters;
(2) The method comprises the steps of searching optimized classifiers and super-parameters, selecting a Support Vector Machine (SVM), a decision tree and naive Bayes as candidate classifiers, utilizing each classifier to train preprocessed drilling data one by one, optimizing the classifier in the training process, and searching the optimal super-parameters. In particular, in order to save calculation time and reduce calculation memory, the invention selects a Bayesian optimizer to perform super-parameter optimization, the Bayesian optimization observes super-parameters in an iterative mode, collects the super-parameters expected to have good classification results, discards the super-parameters with uncertain results, reduces storage space and accelerates calculation speed.
(3) And establishing a geobody grid unit, and establishing grid units with equal size and uniform distribution according to the coordinates of boundary points of the established geobody by a certain step length to construct a regular geobody three-dimensional space data field.
After the three-dimensional grid unit of the geologic body is built, the corresponding relation between the coordinates of the three-dimensional grid unit and the three-dimensional characteristics and the three-dimensional vertex positions output in the step 4 is built, and finally, the three-dimensional geologic body model is built and generated more accurately.
This embodiment has the following advantages: (1) The conditional challenge-generating network image generation technique is utilized to provide a sufficient two-dimensional image dataset for the creation of a three-dimensional geologic model. (2) The Gaussian pyramid is adopted to extract the image features in all directions and at multiple angles, so that the problem that the traditional technology cannot extract the image features comprehensively and singly and influences the establishment of a subsequent model is avoided. (3) By adopting a convolutional neural network-cyclic neural network (CNN-RNN) coupling method, the characteristics of each dimension of the two-dimensional image input into the converter from the two-dimensional encoder can be reserved, the three-dimensional grid characteristics are dynamically updated in real time according to the input, and the characteristic fusion of each dimension is achieved, so that the construction of a precise model is realized. (4) An implicit modeling method based on sparse drilling data is introduced, the geologic body modeling method based on the surface two-dimensional image is coupled, and conditions are provided for refinement, comprehension and accurate establishment of a three-dimensional geologic model.
The embodiment of the invention provides an embodiment of a modeling device of a three-dimensional geologic body model based on a surface image, which comprises the following components:
the construction unit is used for obtaining pseudo two-dimensional surface images by generating the two-dimensional surface image data and performing countermeasure network training, and constructing a multi-element complete two-dimensional image data set;
the feature extraction unit is used for extracting features of each scale of the image by using a Gaussian pyramid through a two-dimensional encoder based on a scale-invariant feature conversion algorithm;
the feature conversion unit is used for realizing two-dimensional to three-dimensional feature conversion by utilizing a projection thought, and preserving and integrating the extracted image features by utilizing a cyclic neural network to construct three-dimensional features for modeling;
the computing unit is used for constructing a three-dimensional transcoder based on the three-dimensional features established through conversion, carrying out convolution computation on the three-dimensional vertex feature matrixes and predicting the new position and the three-dimensional feature of each feature matrix vertex;
the modeling unit is used for combining sparse drilling data, establishing a geologic body grid unit by using an implicit modeling method, and establishing association between the grid unit and the vertex position and the three-dimensional feature of the feature matrix to obtain a target three-dimensional geologic body model.
Optionally, the two-dimensional surface image data in the building unit includes: a single two-dimensional earth surface image data or an image set made up of a plurality of two-dimensional earth surface image data.
Optionally, the criterion features of the gaussian pyramid in the feature extraction unit specifically include: and extracting image features in a downsampling mode, convoluting the image by using Gaussian filtering, and removing even lines and columns in the convolution result to enable the downsampling result to be 1/4 of the original image size.
Optionally, in the feature conversion unit, the implementation of feature conversion from two-dimensional features to three-dimensional features by using a projection concept specifically includes: and realizing the feature conversion from two-dimensional features to three-dimensional features by using an Image2 Mesh-RNN joint converter, wherein the joint converter is formed by an Image2Mesh and a circulating neural network.
This embodiment has the following advantages: (1) The conditional challenge-generating network image generation technique is utilized to provide a sufficient two-dimensional image dataset for the creation of a three-dimensional geologic model. (2) The Gaussian pyramid is adopted to extract the image features in all directions and at multiple angles, so that the problem that the traditional technology cannot extract the image features comprehensively and singly and influences the establishment of a subsequent model is avoided. (3) By adopting a convolutional neural network-cyclic neural network (CNN-RNN) coupling method, the characteristics of each dimension of the two-dimensional image input into the converter from the two-dimensional encoder can be reserved, the three-dimensional grid characteristics are dynamically updated in real time according to the input, and the characteristic fusion of each dimension is achieved, so that the construction of a precise model is realized. (4) An implicit modeling method based on sparse drilling data is introduced, the geologic body modeling method based on the surface two-dimensional image is coupled, and conditions are provided for refinement, comprehension and accurate establishment of a three-dimensional geologic model.
An embodiment of the present invention provides an embodiment of an electronic device, including: including a memory, a processor, and one or more programs; the one or more programs are stored in the memory; the processor, when executing the one or more programs, causes the electronic device to implement a method of modeling a three-dimensional geologic volume model based on a surface image as described in the previous embodiments. Wherein the memory is for storing a computer program and is configurable to store various other data to support operations on the electronic device. Examples of such data include instructions, messages, pictures, videos, etc. for any application or method operating on an electronic device.
Embodiments of the present invention provide an embodiment of a computer-readable storage medium, the storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform a method of modeling a three-dimensional geologic volume model based on a surface image as described in the previous embodiment.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer readable media, as defined herein, does not include transitory computer readable media (transmission media), such as modulated data signals and batches.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.

Claims (10)

1. The modeling method of the three-dimensional geologic body model based on the surface image is characterized by comprising the following steps of:
step 1, two-dimensional surface image data are trained to obtain pseudo two-dimensional surface images through generation of an countermeasure network, and a two-dimensional image data set is constructed;
step 2, extracting features of each scale of the image by using a Gaussian pyramid through a two-dimensional encoder based on a scale invariant feature conversion algorithm to obtain two-dimensional features;
step 3, realizing the conversion from two-dimensional features to three-dimensional features by utilizing a projection thought, and reserving and integrating the extracted image features by utilizing a cyclic neural network to construct three-dimensional features for modeling;
step 4, constructing a three-dimensional code converter based on the three-dimensional features established through conversion, carrying out convolution calculation on the three-dimensional vertex feature matrixes, and predicting the new position and the new three-dimensional feature of each feature matrix vertex;
and 5, combining sparse drilling data, establishing a geologic body grid unit by using an implicit modeling method, and establishing association between the grid unit and the vertex position of the feature matrix and new three-dimensional features to obtain a target three-dimensional geologic body model.
2. The method for modeling a three-dimensional geologic volume model based on a surface image according to claim 1, wherein the two-dimensional surface image data in step 1 comprises: a single two-dimensional earth surface image data or an image set made up of a plurality of two-dimensional earth surface image data.
3. The modeling method of a three-dimensional geologic volume model based on a surface image according to claim 1, wherein the criterion features of the gaussian pyramid in the step 2 specifically comprise: and extracting image features in a downsampling mode, convoluting the image by using Gaussian filtering, and removing even lines and columns in a convolution result to enable the downsampling result to be 1/4 of the original image size.
4. The method for modeling a three-dimensional geologic volume model based on a surface image according to claim 1, wherein in the step 3, the method for converting two-dimensional features into three-dimensional features by using projection ideas specifically comprises:
and realizing the feature conversion from two-dimensional features to three-dimensional features by using an Image2 Mesh-RNN joint converter, wherein the joint converter is composed of the Image2Mesh and a circulating neural network.
5. A modeling apparatus for a three-dimensional geologic body model based on a surface image, comprising:
the construction unit is used for obtaining pseudo two-dimensional surface images by generating the two-dimensional surface image data and performing countermeasure network training, and constructing a multi-element complete two-dimensional image data set;
the feature extraction unit is used for extracting features of each scale of the image by using a Gaussian pyramid through a two-dimensional encoder based on a scale-invariant feature conversion algorithm;
the feature conversion unit is used for realizing two-dimensional to three-dimensional feature conversion by utilizing a projection thought, and preserving and integrating the extracted image features by utilizing a cyclic neural network to construct three-dimensional features for modeling;
the computing unit is used for constructing a three-dimensional transcoder based on the three-dimensional features established through conversion, carrying out convolution computation on the three-dimensional vertex feature matrixes and predicting the new position and the three-dimensional feature of each feature matrix vertex;
the modeling unit is used for combining sparse drilling data, establishing a geologic body grid unit by using an implicit modeling method, and establishing association between the grid unit and the vertex position and the three-dimensional feature of the feature matrix to obtain a target three-dimensional geologic body model.
6. The apparatus for modeling a three-dimensional geologic volume model based on a surface image according to claim 5, wherein the two-dimensional surface image data in the construction unit comprises: a single two-dimensional earth surface image data or an image set made up of a plurality of two-dimensional earth surface image data.
7. The modeling apparatus of a three-dimensional geologic volume model based on a surface image according to claim 5, wherein the criterion features of the gaussian pyramid in the feature extraction unit specifically comprise: and extracting image features in a downsampling mode, convoluting the image by using Gaussian filtering, and removing even lines and columns in the convolution result to enable the downsampling result to be 1/4 of the original image size.
8. The apparatus for modeling a three-dimensional geologic volume model based on a surface image according to claim 5, wherein in the feature conversion unit, the feature conversion from two-dimensional features to three-dimensional features by using projection ideas specifically comprises:
and realizing the feature conversion from two-dimensional features to three-dimensional features by using an Image2 Mesh-RNN joint converter, wherein the joint converter is formed by an Image2Mesh and a circulating neural network.
9. An electronic device, comprising: including a memory, a processor, and one or more programs;
the one or more programs are stored in the memory;
the processor, when executing the one or more programs, causes the electronic device to implement a method of modeling a three-dimensional geologic volume model based on a surface image as defined in any of claims 1-4.
10. A computer readable storage medium comprising instructions that, when executed on an electronic device, cause the electronic device to perform the method of modeling a three-dimensional geologic volume model based on a surface image of any of claims 1-4.
CN202310542068.1A 2023-05-15 2023-05-15 Modeling method and related device of three-dimensional geologic body model based on surface image Pending CN116486030A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310542068.1A CN116486030A (en) 2023-05-15 2023-05-15 Modeling method and related device of three-dimensional geologic body model based on surface image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310542068.1A CN116486030A (en) 2023-05-15 2023-05-15 Modeling method and related device of three-dimensional geologic body model based on surface image

Publications (1)

Publication Number Publication Date
CN116486030A true CN116486030A (en) 2023-07-25

Family

ID=87223237

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310542068.1A Pending CN116486030A (en) 2023-05-15 2023-05-15 Modeling method and related device of three-dimensional geologic body model based on surface image

Country Status (1)

Country Link
CN (1) CN116486030A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117633140A (en) * 2024-01-25 2024-03-01 中化地质矿山总局山东地质勘查院 Urban geological investigation method based on big data cloud computing technology

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117633140A (en) * 2024-01-25 2024-03-01 中化地质矿山总局山东地质勘查院 Urban geological investigation method based on big data cloud computing technology
CN117633140B (en) * 2024-01-25 2024-04-16 中化地质矿山总局山东地质勘查院 Urban geological investigation method based on big data cloud computing technology

Similar Documents

Publication Publication Date Title
Han et al. Image-based 3D object reconstruction: State-of-the-art and trends in the deep learning era
Huang et al. Shape synthesis from sketches via procedural models and convolutional networks
Fan et al. A point set generation network for 3d object reconstruction from a single image
EP3822910A1 (en) Depth image generation method and device
Jimenez Rezende et al. Unsupervised learning of 3d structure from images
Zeng et al. Pc-nbv: A point cloud based deep network for efficient next best view planning
CN110490917A (en) Three-dimensional rebuilding method and device
Islam et al. Interpretation of deep CNN based on learning feature reconstruction with feedback weights
CN116486030A (en) Modeling method and related device of three-dimensional geologic body model based on surface image
CN117036612A (en) Three-dimensional reconstruction method based on nerve radiation field
Garcia-Garcia et al. A study of the effect of noise and occlusion on the accuracy of convolutional neural networks applied to 3D object recognition
Samavati et al. Deep learning-based 3D reconstruction: a survey
Zhu et al. Nonlocal low-rank point cloud denoising for 3-D measurement surfaces
Chen et al. Circle: Convolutional implicit reconstruction and completion for large-scale indoor scene
Ul Islam et al. Learning typical 3D representation from a single 2D correspondence using 2D-3D transformation network
Muzahid et al. A new volumetric CNN for 3D object classification based on joint multiscale feature and subvolume supervised learning approaches
Lin et al. A-SATMVSNet: An attention-aware multi-view stereo matching network based on satellite imagery
Wang et al. High-resolution point cloud reconstruction from a single image by redescription
CN112802202A (en) Image processing method, image processing device, electronic equipment and computer storage medium
Qian et al. Seismic facies recognition based on prestack data using deep convolutional autoencoder
Yan et al. Three-dimensional rapid registration and reconstruction of multi-view rigid objects based on end-to-end deep surface model
Wang A text image generation model based on deep learning
Steinlechner et al. Adaptive pointcloud segmentation for assisted interactions
Gorbatsevich et al. Enhancing detail of 3D terrain models using GAN
Xiao et al. Multi-dimensional graph interactional network for progressive point cloud completion

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination