CN115938544A - Automatic orthodontic tooth arrangement method and system based on grid feature deep learning - Google Patents
Automatic orthodontic tooth arrangement method and system based on grid feature deep learning Download PDFInfo
- Publication number
- CN115938544A CN115938544A CN202211299421.XA CN202211299421A CN115938544A CN 115938544 A CN115938544 A CN 115938544A CN 202211299421 A CN202211299421 A CN 202211299421A CN 115938544 A CN115938544 A CN 115938544A
- Authority
- CN
- China
- Prior art keywords
- tooth
- feature
- model
- deep learning
- triangular mesh
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000013135 deep learning Methods 0.000 title claims abstract description 24
- 210000004513 dentition Anatomy 0.000 claims abstract description 51
- 230000036346 tooth eruption Effects 0.000 claims abstract description 51
- 238000013136 deep learning model Methods 0.000 claims abstract description 21
- 238000013507 mapping Methods 0.000 claims abstract description 19
- 239000013598 vector Substances 0.000 claims description 42
- 230000006870 function Effects 0.000 claims description 24
- 206010061274 Malocclusion Diseases 0.000 claims description 22
- 238000012549 training Methods 0.000 claims description 18
- 230000033001 locomotion Effects 0.000 claims description 16
- 230000009466 transformation Effects 0.000 claims description 14
- 239000011159 matrix material Substances 0.000 claims description 11
- 238000003860 storage Methods 0.000 claims description 11
- 238000000605 extraction Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 6
- 210000000214 mouth Anatomy 0.000 claims description 5
- 238000013519 translation Methods 0.000 claims description 5
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 238000003491 array Methods 0.000 claims description 3
- 210000000332 tooth crown Anatomy 0.000 claims description 2
- 230000001131 transforming effect Effects 0.000 claims 1
- 238000012545 processing Methods 0.000 description 16
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 7
- 210000000988 bone and bone Anatomy 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 5
- 210000002455 dental arch Anatomy 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000006116 polymerization reaction Methods 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 210000004763 bicuspid Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 238000005336 cracking Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 210000004195 gingiva Anatomy 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 210000004283 incisor Anatomy 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000035515 penetration Effects 0.000 description 1
- 239000011505 plaster Substances 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013334 tissue model Methods 0.000 description 1
Images
Landscapes
- Dental Tools And Instruments Or Auxiliary Dental Instruments (AREA)
Abstract
The invention discloses an automatic orthodontic tooth arrangement method and system based on grid feature deep learning. The method comprises the following steps: acquiring a tooth triangular mesh three-dimensional model for tooth arrangement; and inputting the tooth triangular mesh model into a pre-trained deep learning model to obtain the tooth predicted position of the tooth arrangement result. The deep learning model comprises a first feature encoder, a second feature encoder and a feature decoding and mapping device, wherein the first feature encoder takes the tooth triangular mesh three-dimensional model as input to obtain tooth shape features, the second feature encoder takes dentition point cloud as input to obtain dentition global features, and the feature decoding and mapping device obtains predicted tooth arrangement results based on the tooth shape features and the dentition global features. The invention can automatically and accurately plan the ideal pose of the teeth of the patient.
Description
Technical Field
The invention relates to the technical field of oral medicine, in particular to an orthodontic automatic tooth arrangement method and system based on grid feature deep learning.
Background
The incidence of malocclusion of teeth is high, and the harm to oral health is not negligible. Orthodontics is a subject that specializes in the study of the cause, diagnosis, treatment and prevention of malocclusions. In the clinical orthodontic treatment, the orthodontic appliances are arranged inside and outside the mouth of a patient with malocclusion, and the teeth are applied with orthodontic force to move the teeth, so that the normal arrangement and occlusion relation of the teeth are restored.
Orthodontic tooth arrangement refers to arranging the dentition of a malocclusion patient into an expected ideal post-treatment pose as a reference basis for orthodontic treatment planning, and is a key step of digital orthodontic treatment. But currently in the clinical orthodontic treatment planning stage, physicians often manually list the desired tooth poses through a human-computer interface. This manual tooth arrangement creates excessive labor and time costs.
In recent years, with the development of computer graphics and artificial intelligence technology, the academic community develops the research of artificial intelligence tooth arrangement algorithm, and the artificial intelligence tooth arrangement algorithm is mainly divided into two categories, namely an early method based on dental arch curve fitting and a current method based on deep learning. The deep learning-based method is characterized in that a tooth arrangement task is regarded as a three-dimensional six-degree-of-freedom pose regression problem, dentition pose data before and after treatment are used for training a deep neural network, the mapping relation between the shape characteristics and the space pose of malocclusion teeth and the pose of ideal teeth is learned from a large number of cases, and then the movement amount from the malocclusion teeth to the ideal teeth is solved.
In the prior art, computer-aided tooth arrangement schemes mainly include tooth arrangement methods based on dental arch curve fitting and automatic tooth arrangement methods based on deep learning.
According to the method based on dental arch curve fitting, firstly, an ideal dental arch curve is fitted by using characteristic points on teeth, then the moving amount of each tooth is calculated according to the position relation between the current position of the teeth and the ideal dental arch curve, and the tooth pose after arrangement is further solved. The tooth arrangement effect of the method depends on the precision of input feature points, and the feature points are usually selected manually, so that full-automatic tooth arrangement is difficult to realize.
For the method based on deep learning, the original data used by the method is non-perspective data obtained by scanning, only the information of the tooth crown on the surface is provided, the information of the tooth root is not provided, and the tooth root cannot be considered explicitly, so that the situation that the tooth root of the arranged tooth is too close to or even moves out of the alveolar bone may occur, and the situation is inconsistent with the orthodontic clinical requirement. In addition, the mainstream data types currently used for describing the three-dimensional model include point clouds, voxels, multi-view pictures and a mesh model, wherein the mesh model not only contains the position information of the object, but also extracts the relative topological relation of adjacent units on the surface of the object, and has a strong expression capability on the shape of the object. Accordingly, the neural network for processing the grid has better effect on tasks such as object classification and segmentation compared with the neural network for processing the point cloud. Because the existing automatic tooth arrangement deep neural network uses the point cloud processing network as an encoder to extract the tooth point cloud, compared with the grid processing network, the automatic tooth arrangement deep neural network has weaker capturing and processing capabilities on fine-grained features of teeth, and therefore, the automatic tooth arrangement effect is restricted.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide a novel automatic tooth arrangement method based on grid feature deep learning, which can learn feature representation from a tooth triangular grid model and improve the accuracy of an automatic tooth arrangement result based on the deep learning method.
According to a first aspect of the invention, an orthodontic automatic tooth arrangement method based on grid feature deep learning is provided. The method comprises the following steps:
acquiring a tooth triangular mesh three-dimensional model for tooth arrangement, wherein the tooth triangular mesh three-dimensional model represents shape information and spatial position information of teeth by using vertexes, centroids, middle lines, normal vectors and adjacent surface indexes of triangular surface patches;
inputting the tooth triangular mesh model into a pre-trained deep learning model to obtain a tooth predicted position of a tooth arrangement result;
the deep learning model comprises a first feature encoder, a second feature encoder and a feature decoding and mapping device, wherein the first feature encoder takes the tooth triangular mesh three-dimensional model as input to obtain tooth shape features, the second feature encoder takes dentition point cloud as input to obtain dentition global features, and the feature decoding and mapping device obtains predicted tooth arrangement results based on the tooth shape features and the dentition global features.
According to a second aspect of the invention, an orthodontic automatic tooth arrangement system based on grid feature deep learning is provided. The system comprises:
a tooth model acquisition unit: the tooth triangular mesh three-dimensional model is used for obtaining a tooth triangular mesh three-dimensional model for tooth arrangement, and the tooth triangular mesh three-dimensional model represents shape information and spatial position information of teeth by using vertexes, centroids, middle lines, normal vectors and adjacent surface indexes of triangular surface patches;
a prediction unit: the tooth triangular mesh model is input into a pre-trained deep learning model to obtain tooth predicted positions of tooth arrangement results;
the deep learning model comprises a first feature encoder, a second feature encoder and a feature decoding and mapping device, wherein the first feature encoder takes the tooth triangular mesh three-dimensional model as input to obtain tooth shape features, the second feature encoder takes dentition point cloud as input to obtain dentition global features, and the feature decoding and mapping device obtains predicted tooth arrangement results based on the tooth shape features and the dentition global features.
Compared with the prior art, the method has the advantages that the ideal pose of the teeth of the malocclusion patient can be automatically planned based on the tooth triangular mesh three-dimensional model, the time used in the treatment planning stage of an orthodontist is saved, and the defect of low result accuracy caused by the weak capability of extracting fine-grained features of the teeth of the existing automatic tooth arrangement method based on deep learning is overcome.
Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is a flowchart of an orthodontic automatic tooth arrangement method based on grid feature deep learning according to an embodiment of the invention;
FIG. 2 is a schematic diagram of a tooth triangular mesh model according to one embodiment of the invention;
FIG. 3 is a block diagram of an automatic tooth placement network model according to one embodiment of the invention;
fig. 4 is a schematic view of the tooth arrangement effect according to an embodiment of the present invention.
Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be discussed further in subsequent figures.
Referring to fig. 1, the provided mesh feature deep learning-based automatic tooth arrangement method includes the following steps.
And step S110, constructing a training set by using the tooth triangular mesh three-dimensional model.
In one embodiment, the training set is constructed according to the following steps:
and step S111, segmenting and reconstructing the oral bone tissue model.
For example, a high-precision dental crown model and a three-dimensional model of a complete tooth are obtained by segmentation reconstruction from an intraoral scan image (or a laser scan image of a plaster model) and an oral cavity CT image of a malocclusion patient, and then root parts of the high-precision dental crown model obtained from the dental laser scan image and the root parts of the complete tooth model obtained from the oral cavity CT image are registered and fused to obtain a tooth triangular mesh three-dimensional model for tooth arrangement. The model is simplified, and the number of the triangular faces of all teeth is unified to be N. In one embodiment, N is taken to be 1000.
And step S112, constructing a tooth arrangement data set.
For example, after moving the malocclusion teeth to the desired positions according to the six-jaw criteria, the triangular mesh models of all teeth of the malocclusion and the desired dentition are saved, respectively. The expected dentition position is the expected value of the tooth arrangement task, which is used as a supervision value of network training in the training process and as a gold standard in the testing process. And randomly dividing the data set to obtain a training set, a verification set and a test set.
In another embodiment, a three-dimensional model of the mouth created from post-treatment (teeth aligned) image data may also be used as the expectation value.
Step S113, tooth arrangement data is enhanced.
For example, data enhancement is divided into individual movement and overall movement. For individual movement, randomly moving each tooth in the malocclusion tooth arrays in the training set, wherein the malocclusion tooth arrays before and after movement correspond to the same expected tooth array. And for the overall movement, synchronously and randomly moving all malocclusion dentitions and expected dentitions in the training set to obtain an expanded training set. Through data enhancement, the data set can be expanded, thereby improving the accuracy of subsequent model training.
And step S114, preprocessing data.
And extracting point cloud and grid data from the tooth model to be used as input of a subsequent tooth arrangement network model and a loss function. For example, the point cloud is the original vertices of all triangular patches, and the mesh data is calculated from the normalized tooth model, including the vertices, centroids, median lines, normal vectors, and the indices of the adjacent surface of the triangular patches, as shown in fig. 2.
In one embodiment, normalizing the tooth model comprises: all teeth are zoomed to be respectively circumscribed with the unit ball with the center of mass as the center of the ball. Shape information of the tooth and its spatial absolute position information are decoupled by normalization, and only the relative position information between vertices is retained. The processing mode improves the robustness of the subsequent automatic tooth arrangement network model to teeth with different shapes.
And step S120, constructing a deep learning model as a tooth arrangement network model, wherein the model comprises a dentition global feature encoder, a tooth shape feature encoder and a feature decoding and mapping device.
In this step, a deep learning model is constructed as an automatic tooth arrangement network model, and in the description herein, the deep learning model and the tooth arrangement network model or the automatic tooth arrangement network model have the same meaning unless otherwise indicated according to the context.
Referring to fig. 3, the deep learning model includes a dentition global feature encoder, a tooth shape feature encoder, and a feature decoding and mapper.
(1) Dentition global feature encoder
In one embodiment, the dentition global feature encoder may include multiple convolutional layers, each of which in turn connects the batch regularization and Relu activation functions.
The dentition global feature encoder inputs the point cloud of the original malocclusion teeth
And outputting the malocclusion dentition characteristics with global information after convolution.
(2) Tooth shape feature encoder
In addition, for the sake of the Automatic Tooth Arrangement network design of TANet (Wei G, cui Z, liu Y, et al. TANet: towards rather Automatic Tooth Arrangement [ C ]// European Conference reference on Computer vision. Springer, char, 2020. Different from the existing network, the tooth shape feature extraction module of the embodiment of the invention learns feature representation from the triangular mesh model of the tooth so as to improve the extraction capability of local fine-grained features of the tooth. The tooth shape feature extraction module comprises a space description module, a structure description module and a grid convolution module.
It should be noted that the existing automatic tooth arrangement method based on deep learning uses point cloud data and a network, the embodiment of the invention firstly proposes that a mesh feature learning network is used for feature extraction of a triangular mesh model of teeth, and MeshNet or other deep learning networks for processing the triangular mesh model can be adopted to realize tooth-level feature extraction. By designing the tooth shape feature encoder based on grid feature learning, the extraction capability of tooth fine-grained shape features is improved.
In one embodiment, the spatial description module maps a centroid under the tooth local coordinate system to a local spatial feature using two layers of one-dimensional convolution layers with a kernel size of 1, where the first layer of convolution is used to extract the local spatial feature with a receptive field size of 1 from the local spatial locations of the triangular patch. The second layer of convolution is used for learning the transformation of the characteristics, so as to register the characteristic vectors of the triangular patches with similar shapes and different poses, learn the local space characteristics with more robustness and record the local space characteristics as
In one embodiment, the structure description module includes a triangular patch rotated convolution and a triangular patch kernel correlation. The triangular patch convolution can be described using a symmetric function f, expressed as:
wherein h is 1 (·,·):
h 2 (·,·):/>
Respectively 2 one-dimensional convolution layers, h 1 The combination of the three input groups of central lines is subjected to weight sharing, oa represents a three-dimensional vector formed by the centroid of the triangular patch and a vertex A, ob represents a three-dimensional vector formed by the centroid of the triangular patch and a vertex B, and oc represents a three-dimensional vector formed by the centroid of the triangular patch and a vertex C. In this embodiment, K 1 ＝32，K 2 =64. The triangular patch rotary convolution operation has rotation invariance to eliminate the disorder of the central line for extractionThe influence of the shape characteristics of the uniform triangular face.
The triangular patch kernel correlation is used to expand the receptive field of the structural feature vector to the neighboring patches of the triangular patch, which are input as the normal vector and the adjacency index of the triangular patch. Firstly, the normal vector of each surface and the normal vectors of the corresponding three adjacent surfaces are combined through the adjacent surface indexes to obtain a polymerization vector. And then setting a kernel with learnable parameters to perform related operation with the polymerization vector.
Further, the results of the triangular patch rotary convolution operation and the triangular patch kernel correlation operation are spliced, and then the spliced vectors are mapped into structural features and recorded as structural features by using two layers of one-dimensional convolution with channels of 131
In order to increase the receptive field of local spatial and structural features, a grid convolution module is used to recombine the outputs of the spatial and structural description modules at the dental level
And &>
To extract deeper features.
For the local spatial features, the local spatial features are spliced with corresponding structural feature vectors and then convolved to obtain the local spatial features fused with structural information, which can be expressed as:
wherein
Representing concatenation of vectors, W mc1 The convolution kernel size is 1 and the number of channels is 256.
For the structural features, similar to the normal vector aggregation operation of triangular patches in the structure description module, the structural feature vector of each triangular patch and the structural feature vector of an adjacent triangular patch are aggregated by an index of an adjacent surface, and then the aggregated structural feature vectors are subjected to convolution and maximum pooling operations to obtain the aggregated structural features, which can be expressed as:
where gather represents the aggregation of the structural feature vectors of adjacent triangular patches, W mc2 The convolution kernel size is 1 × 1, and the number of channels is 256.
Next, at the dental level, to
And &>
Splicing to obtain the tooth shape characteristics
Can be expressed as:
(3) Feature decoding and mapper
In the characteristic decoding and mapping device, the dentition global characteristic with the tooth space position information and the shape characteristic of all teeth are fused and dimensionality reduced, and then mapped into the movement amount of the teeth as the output of the tooth arrangement network.
In FIG. 3, the decoder inputs the dentition global feature F dentition And the shape characteristics F of all teeth shape Splicing by XN, decoding the spliced characteristics by using two one-dimensional convolution layers and one full-connection layer with 28 and 64 channels respectively to obtain the movement amount to be converted
Then, a motion representation transformation is performed. The movement representation transformation may include two steps of a constraint transformation and a representation transformation. />
For constrained transformation, in order to make the rotation axis vector satisfy the unit vector and ensure the numerical stability of the transformation process, the rotation axis vector is subjected to softmax function L A axis ＝( L A 0 ， L A 1 ， L A 2 ) Mapping to obtain new rotation axis vector L A axis ′：
Where sign (·) represents a sign function. Considering that the amount of movement of the teeth before and after the treatment does not exceed 20mm in practical cases, the tanh function can be used for the translation vector L t＝( L A 4 ， L A 5 ， L A 6 ) Mapping to obtain limited translation vector Lt ′。
For the representation transformation, the rotation amount and the translation amount are transformed into a homogeneous transformation matrix in a world coordinate system, and the tooth predicted position is calculated. The shaft vector is first mapped using an exponential from SO (3) to SO (3) L A axis ' conversion to rotational matrix
L R＝I+sin(θ) L A axis +(1-cos(θ)) L A axis (6)
Wherein θ = L A 3 ，
Is a vector of the rotation axis L A axis ' of the inverse symmetric matrix (Sew-symmetry matrix). Will rotate the matrix L R and translation vector L the t' combination is a homogeneous transformation matrix under a tooth local coordinate system>
Then, will L T is converted into a homogeneous transformation matrix under a world coordinate system W T, for the original malocclusion teeth
The movement is carried out to obtain the automatic tooth arrangement result, namely the tooth predicted position is recorded as ^ er>
In the characteristic decoding and mapping device, the rotation amount is restricted by using the softmax function, so that the output movement amount is ensured to be a rigid body transformation matrix, and the numerical stability of the calculation process can be met.
And S130, training a deep learning model based on the set loss function.
For example, the distance between corresponding points of the predicted tooth position and the expected tooth position output by using the euclidean distance metric network sets a loss function expressed as:
wherein,
representing the desired tooth point cloud location, avg represents the average.
It should be understood that, instead of directly using the average of the euclidean distance deviations between the predicted positions and the expected positions of the tooth point cloud after arrangement as the loss function, the maximum of the euclidean distance deviations may be weighted and used as a term of the loss function alone. Or the cosine similarity error of the predicted transform matrix and the desired transform matrix as a loss function.
Further, the constructed deep learning model is trained by using a training set. In the training process, an Adam optimizer is used, parameters of the model are optimized by minimizing loss function values, and a series of network models with different parameters are obtained. And then verifying the trained model by using a verification set, saving the network model with the optimal effect as a final tooth arrangement network model, and carrying out tooth arrangement operation on the malocclusion sample without tooth arrangement by using the tooth arrangement network model.
And step S140, obtaining the tooth arrangement result of the target patient by using the trained deep learning model.
After model training and verification are completed, the obtained final tooth arrangement network model can be used for actual tooth arrangement prediction, namely, a tooth triangular mesh three-dimensional model of a target patient to be tooth arranged is input into the trained tooth arrangement network model, and a predicted tooth movement amount is obtained to serve as a tooth arrangement result.
It should be noted that those skilled in the art can appropriately change or modify the above-described embodiments without departing from the spirit and scope of the present invention. For example. In the feature decoding and mapper, in addition to using the softmax function to constrain the rotation amount, other methods may be used so that the three-dimensional rotation axis vector used for representing the rotation is constant as a unit vector and numerical stability is satisfied at the time of calculation. For example:
wherein L A axis K =0,1,2, which are rotation axis vectors in the local coordinate system, respectively represent three-dimensional components of the rotation axis vectors.
Correspondingly, the invention further provides an automatic orthodontic tooth arrangement system based on grid feature deep learning. The system comprises: a tooth model obtaining unit for obtaining a tooth triangular mesh three-dimensional model for tooth arrangement, the tooth triangular mesh three-dimensional model representing shape information and spatial position information of teeth by using vertexes, centroids, central lines, normal vectors and abutment plane indexes of triangular patches; a prediction unit, configured to input the dental triangular mesh model to a pre-trained deep learning model, and obtain a predicted dental position of a dental alignment result, where the deep learning model includes a dentition global feature encoder, a dentition shape feature encoder, and a feature decoding and mapper, the dentition shape feature encoder takes the dental triangular mesh three-dimensional model as input and obtains a dentition shape feature, the dentition global feature encoder takes a dentition point cloud as input and obtains a dentition global feature, and the feature decoding and mapper obtains the predicted dental alignment result based on the dentition shape feature and the dentition global feature. Each unit in the system can be realized by adopting DSP, FPGA, processor or special hardware.
In order to further verify the effect of the invention, on the same data set, a comparison experiment is carried out on the method provided by the invention and the existing similar method TANet based on deep learning. The result shows that the automatic tooth arrangement method can learn the mapping relation between the malocclusion dentition and the ideal dentition pose from the data before and after tooth arrangement, and the accuracy of the obtained automatic tooth arrangement result is higher than that of the similar method. Referring to fig. 4, wherein fig. 4 (a) corresponds to a malocclusion dentition, fig. 4 (b) is a result of a prior art TANet dentition, fig. 4 (c) is a result of a present invention dentition, fig. 4 (d) is a desired dentition, and each row in fig. 4 represents an example. As can be seen from fig. 4, for the first sample, the positions of the middle and side incisors were as expected, but the premolar spacing for the TANet output results was too large; for the second sample, the crowding of the right maxillary premolars was improved; for the third sample, the dentition after tooth arrangement was significantly more regular, but smaller gaps and even model penetration occurred compared to the expected dentition obtained by artificial tooth arrangement.
The model training process can be carried out in a server or a cloud offline mode, and the trained model is embedded into the electronic equipment, so that the automatic tooth arrangement result can be generated in real time. The electronic device can be a terminal device or a server, and the terminal device comprises any terminal device such as a mobile phone, a tablet computer, a Personal Digital Assistant (PDA), a point-of-sale (POS), a vehicle-mounted computer, a smart wearable device (a smart watch, virtual reality glasses, a virtual reality helmet and the like). The server includes but is not limited to an application server or a Web server, and may be a stand-alone server, a cluster server or a cloud server. The invention can be applied to automatic tooth arrangement in the field of orthodontic and can also be popularized to the task of automatic arrangement of false teeth in the field of oral prosthetics.
In conclusion, the invention designs the mesh feature learning network to extract the shape feature of the triangular mesh model of the tooth, and on the basis, the shape feature, the global space feature and the potential rule of the expected position of the tooth are learned to arrange the tooth. Compared with the prior art, the invention has the following advantages:
1) The existing automatic tooth arrangement method based on deep learning only comprises information of the dental crowns but not information of tooth roots and alveolar bones inside the gingiva, so that the pose of the tooth roots cannot be considered explicitly, and the situations of tooth root absorption, bone windowing, bone cracking and the like caused by the fact that the tooth roots are too close to or even move out of the alveolar bones in clinical use can occur, and the method does not meet clinical requirements. The invention considers the pose of the tooth root, inputs the three-dimensional model data of the complete tooth containing the tooth root into the dentition global characteristic encoder and the tooth shape characteristic encoder, correspondingly calculates the distance error of the tooth arrangement result containing the tooth root by the loss function, and further realizes the introduction of the pose of the tooth root into the automatic tooth arrangement algorithm by optimizing the parameters of the network model.
2) In the existing automatic tooth arrangement method based on deep learning, a point cloud processing network is used as a tooth point cloud feature encoder, and the extraction capability of fine-grained features is limited. The invention uses the grid processing network to extract the tooth shape characteristics, can more fully extract the characteristics from the triangular grid data with the object surface topological relation, and designs the tooth shape encoder based on the grid characteristic learning to extract the fine-grained shape characteristics of the tooth triangular grid model, thereby improving the accuracy of the automatic tooth arrangement of the deep learning network.
The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present invention may be assembler instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + +, python, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, by software, and by a combination of software and hardware are equivalent.
Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.
Claims (10)
1. An orthodontic automatic tooth arrangement method based on grid feature deep learning comprises the following steps:
acquiring a tooth triangular mesh three-dimensional model for tooth arrangement, wherein the tooth triangular mesh three-dimensional model represents shape information and spatial position information of teeth by using vertexes, centroids, middle lines, normal vectors and adjacent surface indexes of triangular surface patches;
inputting the tooth triangular mesh model into a pre-trained deep learning model to obtain a tooth predicted position of a tooth arrangement result;
the deep learning model comprises a first feature encoder, a second feature encoder and a feature decoding and mapping device, wherein the first feature encoder takes the tooth triangular mesh three-dimensional model as input to obtain tooth shape features, the second feature encoder takes dentition point cloud as input to obtain dentition global features, and the feature decoding and mapping device obtains predicted tooth arrangement results based on the tooth shape features and the dentition global features.
2. The method of claim 1, wherein the three-dimensional model of the tooth triangular mesh is obtained according to the following steps:
obtaining a crown model and a complete tooth three-dimensional model by segmenting and reconstructing the intra-oral scanning image and the oral cavity image of the target;
and registering and fusing the tooth crown model and the tooth root part of the tooth three-dimensional model to obtain the tooth triangular mesh three-dimensional model for tooth arrangement.
3. The method of claim 1, wherein the first feature encoder comprises a plurality of tooth shape feature extraction modules connected in parallel, each tooth shape feature extraction module corresponding to a single tooth for independently learning a shape representation of the single tooth from data of the single tooth.
4. The method according to claim 3, wherein the tooth shape feature extraction module comprises a spatial description module, a structural description module and a grid convolution module, the spatial description module takes the centroid of the tooth as an input for mapping the centroid under the tooth local coordinate system to a local spatial feature and learning the transformation between the local spatial features; the structure description module takes a central line, a normal vector and an adjacent surface index of a tooth as input, and obtains structure features through related operations of rotary convolution and kernel; the grid convolution module obtains the tooth shape feature by fusing the local spatial feature and the structural feature.
5. The method of claim 4, wherein the tooth shape feature is represented as:
wherein:
wherein,
representing a concatenation of vectors, W mc1 Is a one-dimensional convolution, gather represents the aggregation of the structural feature vectors of adjacent triangular patches, W mc2 Is two-dimensional convolution, <' > based on the convolution>
Is the output of the space description module, and>
is the output of the structure description module.
6. The method of claim 1, wherein the feature decoding and mapper performs:
splicing the dentition global characteristics and the shape characteristics of all teeth, and decoding the spliced characteristics to obtain the movement amount to be converted;
for the movement amount to be converted, mapping the corresponding rotating shaft vector by using a softmax function to obtain a new rotating shaft vector;
and transforming the rotation amount and the translation amount into a homogeneous transformation matrix in a world coordinate system, and calculating the tooth predicted position.
7. The method of claim 1, wherein the loss function for training the deep learning model is expressed as:
wherein,
representing the post-alignment tooth point cloud predicted position, < > or >>
Representing the desired tooth point cloud location, avg represents the average.
8. The method of claim 1, wherein training the training set of deep learning models is obtained according to the following steps:
acquiring a plurality of case samples to construct a tooth arrangement data set, wherein each piece of data in the tooth arrangement data set reflects the corresponding relation between a tooth triangular mesh three-dimensional model of tooth arrangement and an expected tooth triangular mesh three-dimensional model;
performing data enhancement on the tooth arrangement data set to obtain an expanded tooth arrangement data set, wherein the data enhancement comprises:
randomly moving each tooth in the malocclusion tooth array in the tooth arrangement data set, wherein the malocclusion tooth arrays before and after moving correspond to the same expected tooth array;
and synchronously and randomly moving all malocclusion dentitions and expected dentitions in the tooth arrangement data set.
9. An orthodontic automatic tooth arrangement system based on grid feature deep learning comprises:
a tooth model acquisition unit: the tooth triangular mesh three-dimensional model is used for obtaining a tooth triangular mesh three-dimensional model for tooth arrangement, and the tooth triangular mesh three-dimensional model represents shape information and spatial position information of teeth by using vertexes, centroids, middle lines, normal vectors and adjacent surface indexes of triangular surface patches;
a prediction unit: the tooth triangular mesh model is input into a pre-trained deep learning model to obtain tooth predicted positions of tooth arrangement results;
the deep learning model comprises a first feature encoder, a second feature encoder and a feature decoding and mapping device, wherein the first feature encoder takes the tooth triangular mesh three-dimensional model as input to obtain tooth shape features, the second feature encoder takes dentition point cloud as input to obtain dentition global features, and the feature decoding and mapping device obtains predicted tooth arrangement results based on the tooth shape features and the dentition global features.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211299421.XA CN115938544A (en) | 2022-10-21 | 2022-10-21 | Automatic orthodontic tooth arrangement method and system based on grid feature deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211299421.XA CN115938544A (en) | 2022-10-21 | 2022-10-21 | Automatic orthodontic tooth arrangement method and system based on grid feature deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115938544A true CN115938544A (en) | 2023-04-07 |
Family
ID=86551372
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211299421.XA Pending CN115938544A (en) | 2022-10-21 | 2022-10-21 | Automatic orthodontic tooth arrangement method and system based on grid feature deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115938544A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112120810A (en) * | 2020-09-29 | 2020-12-25 | 深圳市深图医学影像设备有限公司 | Three-dimensional data generation method of tooth orthodontic concealed appliance |
| CN112819741A (en) * | 2021-02-03 | 2021-05-18 | 四川大学 | Image fusion method and device, electronic equipment and storage medium |
| CN114187293A (en) * | 2022-02-15 | 2022-03-15 | 四川大学 | Oral cavity palate part soft and hard tissue segmentation method based on attention mechanism and integrated registration |
| CN114612532A (en) * | 2022-02-24 | 2022-06-10 | 西安电子科技大学 | Three-dimensional tooth registration method, system, computer equipment and storage medium |
-
2022
- 2022-10-21 CN CN202211299421.XA patent/CN115938544A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112120810A (en) * | 2020-09-29 | 2020-12-25 | 深圳市深图医学影像设备有限公司 | Three-dimensional data generation method of tooth orthodontic concealed appliance |
| CN112819741A (en) * | 2021-02-03 | 2021-05-18 | 四川大学 | Image fusion method and device, electronic equipment and storage medium |
| CN114187293A (en) * | 2022-02-15 | 2022-03-15 | 四川大学 | Oral cavity palate part soft and hard tissue segmentation method based on attention mechanism and integrated registration |
| CN114612532A (en) * | 2022-02-24 | 2022-06-10 | 西安电子科技大学 | Three-dimensional tooth registration method, system, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 张东霞: ""基于口腔计算机断层扫描图像与激光扫描图 像融合的牙齿三维模型重构"", 《生物医学工程学杂志》, vol. 34, no. 1, 28 February 2017 (2017-02-28) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220218449A1 (en) | Dental cad automation using deep learning | |
| US11735306B2 (en) | Method, system and computer readable storage media for creating three-dimensional dental restorations from two dimensional sketches | |
| JP7489964B2 (en) | Automated Orthodontic Treatment Planning Using Deep Learning | |
| US9418474B2 (en) | Three-dimensional model refinement | |
| CN111784754B (en) | Tooth orthodontic method, device, equipment and storage medium based on computer vision | |
| WO2023242757A1 (en) | Geometry generation for dental restoration appliances, and the validation of that geometry | |
| CN113554607A (en) | Tooth body detection model, generation method and tooth body segmentation method | |
| TW202409874A (en) | Dental restoration automation | |
| CN115938544A (en) | Automatic orthodontic tooth arrangement method and system based on grid feature deep learning | |
| WO2024082284A1 (en) | Orthodontic automatic tooth arrangement method and system based on mesh feature deep learning | |
| WO2023100078A1 (en) | Geometric deep learning for setups and staging in clear tray aligners | |
| Broll et al. | Generative deep learning approaches for the design of dental restorations: A narrative review | |
| US20240029380A1 (en) | Integrated Dental Restoration Design Process and System | |
| US20230087800A1 (en) | Automated tooth administration in a dental restoration workflow | |
| TW202416910A (en) | Integrated dental restoration design process and system | |
| WO2024127308A1 (en) | Classification of 3d oral care representations | |
| WO2024127311A1 (en) | Machine learning models for dental restoration design generation | |
| WO2024127302A1 (en) | Geometric deep learning for final setups and intermediate staging in clear tray aligners | |
| Hosseinimanesh et al. | From Mesh Completion to AI Designed Crown | |
| WO2024127304A1 (en) | Transformers for final setups and intermediate staging in clear tray aligners | |
| WO2024127310A1 (en) | Autoencoders for the validation of 3d oral care representations | |
| WO2024127313A1 (en) | Metrics calculation and visualization in digital oral care | |
| Zhao et al. | Few sampling meshes-based 3D tooth segmentation via region-aware graph convolutional network | |
| WO2024013282A1 (en) | Method and system for tooth pose estimation | |
| WO2024127309A1 (en) | Autoencoders for final setups and intermediate staging in clear tray aligners |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |