CN116129112A - Oral cavity three-dimensional point cloud segmentation method of nucleic acid detection robot and robot - Google Patents
Oral cavity three-dimensional point cloud segmentation method of nucleic acid detection robot and robot Download PDFInfo
- Publication number
- CN116129112A CN116129112A CN202211708393.2A CN202211708393A CN116129112A CN 116129112 A CN116129112 A CN 116129112A CN 202211708393 A CN202211708393 A CN 202211708393A CN 116129112 A CN116129112 A CN 116129112A
- Authority
- CN
- China
- Prior art keywords
- image
- oropharynx
- dimensional
- point cloud
- dimensional point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/0014—Image feed-back for automatic industrial control, e.g. robot with camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
- G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Robotics (AREA)
- Manipulator (AREA)
Abstract
The invention discloses an oral cavity three-dimensional point cloud segmentation method of a nucleic acid detection robot and the robot, wherein the method comprises the following steps: establishing an oropharynx data set according to the acquired image data; wherein the oropharyngeal dataset comprises: data sets of mouth, tongue, posterior pharyngeal wall and tonsils; performing two-dimensional image processing on the oropharynx data set through the optimized Mask R-CNN neural network to obtain a segmented two-dimensional oropharynx image; and mapping the segmented two-dimensional oropharynx image to a three-dimensional point cloud to obtain a three-dimensional point cloud segmented image of the oropharynx. The robot camera can acquire the point cloud image of the part to be sampled of the oropharynx in real time, and can more efficiently and accurately assist the oropharynx swab robot to finish the nucleic acid sampling work.
Description
Technical Field
The invention relates to the technical field of robots, in particular to an oral cavity three-dimensional point cloud segmentation method of a nucleic acid detection robot and the robot.
Background
Pharyngeal swab nucleic acid detection plays an important role in the detection of novel coronaviruses, and during the sampling process, the nucleic acid sampling personnel are at risk of droplet spreading and contact spreading. Quick, efficient and safe pharyngeal swab sampling is a key to finding an infectious agent in advance and cutting off a transmission path, and is a new challenge. The oropharynx swab nucleic acid detection robot can assist medical staff in collecting nucleic acid samples remotely, and can perform nucleic acid collection tasks efficiently, so that pain spots are effectively solved. Medical staff has professional medical knowledge and skills, can sample corresponding positions rapidly and accurately, and for a robot, good visual positioning and guiding are required, so that the accuracy of sampling can be ensured.
To assist the oropharyngeal swab robot in accurately performing nucleic acid sampling, it is necessary to obtain the center of the mouth and the sampling area of the mouth, such as tonsils, posterior pharyngeal walls, uvulae, etc. Because the features of tonsil, pharyngeal wall and the like are not obvious, the edge limit is fuzzy, and the accuracy of the existing detection and segmentation algorithm is limited; as the vision algorithm of the auxiliary oropharyngeal swab robot, the real-time performance is required to be strong, but the conventional Mask R-CNN and other example segmentation algorithms have complex neural network structure and weak real-time performance; the main purpose after division is to provide accurate sampling position information for a robot, and two main methods for acquiring depth information exist: the depth image is directly segmented, so that the data processing capacity is large and the implementation is difficult; the second method is that after the two-dimensional image is divided, a proper sampling point is calculated, then the depth information of the sampling point is obtained according to the position of the sampling point, the fault tolerance rate of the punctiform sampling point is low, and the success rate of the robot sampling is low.
Accordingly, there is a need in the art for improvement.
Disclosure of Invention
The invention aims to solve the technical problems that aiming at the defects in the prior art, the invention provides a method for dividing the oral cavity three-dimensional point cloud of a nucleic acid detection robot and the robot, so as to solve the technical problems of low sampling success rate caused by low accuracy of point-shaped depth information acquired in the existing image dividing mode.
The technical scheme adopted for solving the technical problems is as follows:
in a first aspect, the present invention provides a method for three-dimensional point cloud segmentation of an oral cavity by a nucleic acid detection robot, comprising:
establishing an oropharynx data set according to the acquired image data; wherein the oropharyngeal dataset comprises: data sets of mouth, tongue, posterior pharyngeal wall and tonsils;
performing two-dimensional image processing on the oropharynx data set through the optimized Mask R-CNN neural network to obtain a segmented two-dimensional oropharynx image;
and mapping the segmented two-dimensional oropharynx image to a three-dimensional point cloud to obtain a three-dimensional point cloud segmented image of the oropharynx.
In one implementation, the establishing an oropharyngeal dataset from the acquired image data includes:
collecting a plurality of oral cavity RGB images with different postures;
calibrating the mouth, tongue, back pharyngeal wall and tonsils of each oral cavity RGB image in an image labeling tool according to the oral cavity anatomical model to obtain a calibrated oral cavity RGB image;
and establishing the oropharynx data set according to the calibrated oral cavity RGB image.
In one implementation, the optimized Mask R-CNN neural network includes: an acceptance v2 network, a region candidate network, a full convolution network, and a full connectivity layer.
In one implementation, the two-dimensional image processing of the oropharyngeal dataset through the optimized Mask R-CNN neural network, previously comprising:
and training the optimized Mask R-CNN neural network according to the oropharynx data set to obtain a trained oropharynx partial model.
In one implementation manner, the performing two-dimensional image processing on the oropharynx data set through the optimized Mask R-CNN neural network to obtain a segmented two-dimensional oropharynx image includes:
inputting the oropharynx data set into the optimized Mask R-CNN neural network;
and extracting image features through the optimized Mask R-CNN neural network, generating an image candidate region, and carrying out semantic segmentation to obtain a segmented two-dimensional oropharyngeal image.
In one implementation manner, the extracting the image features through the optimized Mask R-CNN neural network, generating an image candidate region, and performing semantic segmentation includes:
extracting features of the oropharynx data set through an acceptance v2 network to obtain corresponding image features;
the extracted image features are transmitted into a region candidate network, and classified by an ROI classifier to generate an image candidate region;
and inputting the generated image candidate region into a full convolution network to perform semantic segmentation of the oropharynx part, inputting the segmented image into a full connection layer to perform detection and positioning, and outputting a mask image.
In one implementation, the mask image includes: background, mouth, posterior pharyngeal wall, and uvula.
In one implementation, the mapping the segmented two-dimensional oropharynx image to a three-dimensional point cloud to obtain a three-dimensional point cloud segmented image of the oropharynx includes:
marking edge pixels of the two-dimensional segmented image on the depth image corresponding to the segmented two-dimensional oropharyngeal image according to the external reference matrix of the RGB-D camera;
and mapping the pixel points of the segmented two-dimensional oropharynx image and depth image to a three-dimensional coordinate system according to a small-hole imaging model, and obtaining an intra-oral three-dimensional point cloud segmentation result.
In a second aspect, the present invention also provides a robot comprising: the method comprises the steps of a processor and a memory, wherein the memory stores an oral cavity three-dimensional point cloud segmentation program of a nucleic acid detection robot, and the oral cavity three-dimensional point cloud segmentation program of the nucleic acid detection robot is used for realizing the operation of the oral cavity three-dimensional point cloud segmentation method of the nucleic acid detection robot according to the first aspect when being executed by the processor.
In a third aspect, the present invention also provides a storage medium, which is a computer-readable storage medium storing an oral three-dimensional point cloud segmentation program of a nucleic acid detection robot, the oral three-dimensional point cloud segmentation program of the nucleic acid detection robot being for implementing the operations of the oral three-dimensional point cloud segmentation method of the nucleic acid detection robot according to the first aspect when executed by a processor.
The technical scheme adopted by the invention has the following effects:
according to the method, an oropharynx data set comprising a mouth, a tongue, a back pharyngeal wall and tonsils is established according to acquired image data, and two-dimensional image processing is carried out on the oropharynx data set through an optimized Mask R-CNN neural network to obtain a segmented two-dimensional oropharynx image; and mapping the segmented two-dimensional oropharynx image to a three-dimensional point cloud to obtain a three-dimensional point cloud segmented image of the oropharynx. The invention has the advantages of larger sampling position, higher sampling accuracy, higher fault tolerance, lower calculation force requirement and higher speed, and can more efficiently and accurately assist the oropharyngeal swab robot to finish the nucleic acid sampling work.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to the structures shown in these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of an oral three-dimensional point cloud segmentation method of a nucleic acid detection robot in one implementation of the invention.
FIG. 2 is a schematic diagram of a neural network architecture in one implementation of the invention.
Figure 3 is a functional schematic of a robot in one implementation of the invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more clear and clear, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Exemplary method
In the process of accounting for the sampling, it is necessary to obtain the center of the mouth and the sampling area of the mouth, such as tonsils, posterior pharyngeal wall, uvula, etc. Because the features of tonsil, pharyngeal wall and the like are not obvious, the edge limit is fuzzy, and the accuracy of the existing detection and segmentation algorithm is limited; as the vision algorithm of the auxiliary oropharyngeal swab robot, the real-time performance is required to be strong, but the conventional Mask R-CNN and other example segmentation algorithms have complex neural network structure and weak real-time performance; the main purpose after division is to provide accurate sampling position information for a robot, and two main methods for acquiring depth information exist: the depth image is directly segmented, so that the data processing capacity is large and the implementation is difficult; the second method is that after the two-dimensional image is divided, a proper sampling point is calculated, then the depth information of the sampling point is obtained according to the position of the sampling point, the fault tolerance rate of the punctiform sampling point is low, and the success rate of the robot sampling is low.
Aiming at the technical problems, the embodiment of the invention provides an oral cavity three-dimensional point cloud segmentation method of a nucleic acid detection robot, which is characterized in that an optimized Mask R-CNN neural network is used for carrying out two-dimensional image processing on an oropharynx data set to obtain segmented two-dimensional oropharynx images; and mapping the segmented two-dimensional oropharynx image to a three-dimensional point cloud to obtain a three-dimensional point cloud segmented image of the oropharynx. The embodiment of the invention has the advantages of larger sampling position for the robot, higher sampling accuracy, higher fault tolerance, lower calculation force requirement and higher speed, and can more efficiently and accurately assist the oropharyngeal swab robot to finish the nucleic acid sampling work.
As shown in fig. 1, an embodiment of the present invention provides a method for dividing an oral cavity three-dimensional point cloud of a nucleic acid detection robot, including the following steps:
step S100, establishing an oropharynx data set according to the acquired image data.
In this embodiment, the method for dividing the three-dimensional point cloud of the oral cavity of the nucleic acid detection robot is applied to the robot; the robot is a nucleic acid detection robot facing the oropharyngeal swab.
In this embodiment, an oral three-dimensional point cloud segmentation algorithm of a nucleic acid detection robot facing an oropharyngeal swab robot is provided. The algorithm is used for training by establishing a new oropharynx data set and improving a Mask R-CNN neural network to obtain an oropharynx partial model. During nucleic acid detection, the oropharynx part of the two-dimensional plane image is identified and segmented, and the segmentation result is mapped to the three-dimensional point cloud, so that an oropharynx three-dimensional point cloud segmentation image is obtained. According to the oral cavity three-dimensional point cloud segmentation method of the nucleic acid detection robot, a camera on the robot can acquire point cloud images of a part, which is to be sampled, of the oropharynx in real time aiming at any sampler, so that the oropharynx swab robot can be more efficiently and accurately assisted to complete nucleic acid sampling.
Specifically, in one implementation of the present embodiment, step S100 includes the steps of:
step S101, collecting a plurality of oral cavity RGB images with different postures;
step S102, calibrating the mouth, tongue, back pharyngeal wall and tonsils of each oral cavity RGB image in an image labeling tool according to an oral cavity anatomical model to obtain a calibrated oral cavity RGB image;
step S103, the oropharynx data set is established according to the calibrated oral cavity RGB image.
In this embodiment, the three-dimensional point cloud segmentation algorithm of the oral cavity of the nucleic acid detection robot is mainly divided into three parts, which are respectively: 1) Establishing a new oropharynx data set; 2) Two-dimensional image processing; 3) The two-dimensional image is mapped to a three-dimensional point cloud.
In the embodiment, by establishing a new oropharynx data set, the accuracy of the robot to the parts of the posterior pharyngeal wall, tonsils and the like of the collected person can be improved in the accounting and sampling process of the robot, so that the robot can be more accurately segmented in the follow-up detection process. In all public data sets of the mouth, mainly the calibration of teeth, tongue, etc., the calibration of tonsils is very little and inaccurate for the posterior pharyngeal wall.
In the embodiment, 3255 oral RGB images with different postures are shot, and a labelme tool (namely an image marking tool) is used for calibrating the mouth, tongue, pharyngeal rear wall and tonsils in the oral RGB images according to an oral anatomical model; the label can be used for manual calibration in the calibration process, and the automatic calibration can be performed through a specific label.
Because the characteristics of the pharyngeal wall, tonsils and the like are not obvious and the edges are blurred, the accuracy of the positions of the pharyngeal wall, the tonsils and the like is emphasized during calibration to improve the segmentation accuracy. Meanwhile, the posture of the person to be collected will also determine the accuracy of segmentation, so that when the RGB image of the oral cavity is taken, all volunteers take images according to the posture of nucleic acid sampling and collect the images at multiple angles as much as possible, so as to cope with errors caused by angle changes during sampling.
In this embodiment, after the calibrated oral RGB image is obtained, an oropharynx data set may be established according to the calibrated oral RGB image; wherein the oropharyngeal dataset comprises: data sets of the mouth, tongue, posterior pharyngeal wall, tonsils, etc.
In the embodiment, by collecting the RGB images of the oral cavity in different postures, the positions with unobvious characteristics such as the pharyngeal wall and tonsils and the like and the positions with blurred edges can be calibrated, so that the segmentation precision of the positions in the accounting and sampling process is improved.
As shown in fig. 1, an embodiment of the present invention provides a method for dividing an oral cavity three-dimensional point cloud of a nucleic acid detection robot, including the following steps:
and step S200, performing two-dimensional image processing on the oropharynx data set through the optimized Mask R-CNN neural network to obtain a segmented two-dimensional oropharynx image.
In this embodiment, after a new oropharyngeal data set is established, the Mask R-CNN neural network is modified, and the modified Mask R-CNN neural network is trained using the oropharyngeal data set, so as to obtain an oropharyngeal partial model (i.e., the modified Mask R-CNN neural network after training).
Specifically, in one implementation of the present embodiment, the step S200 includes the following steps before:
and step S201a, training the optimized Mask R-CNN neural network according to the oropharynx data set to obtain a trained oropharynx partial model.
In the embodiment, the main work of two-dimensional image processing is to optimize the existing Mask R-CNN neural network so as to adapt to the requirement of the oropharynx swab robot on real-time detection of the oral cavity. The existing Mask R-CNN network has a complex network structure, has poor real-time performance on target (namely oral cavity) detection, and is not suitable for being directly used for detecting and dividing the oropharynx; therefore, by improving the Mask R-CNN neural network, the oral cavity can be rapidly and accurately identified during actual detection.
In this embodiment, in the process of improving the Mask R-CNN neural network, the part of res net/FPN in the original Mask R-CNN may be replaced by an indication v2 network structure in *** net, where the indication v2 network structure uses a separable convolution with a depth multiplier of 8, and the separable convolution greatly improves the processing speed of the network. Such an optimized Mask R-CNN neural network includes: an acceptance v2 network, a regional candidate network (RPN), a Full Convolutional Network (FCN), and a full connectivity layer (FC); the structure of the optimized Mask R-CNN neural network is shown in figure 2.
In the training process of the optimized Mask R-CNN neural network, inputting the preprocessed picture (namely the oropharynx data set) into the pre-trained neural network to obtain a characteristic map; setting a ROI value for each point in the feature map, thereby obtaining a plurality of ROI values; sending the candidate ROI values into a regional candidate network (RPN) for binary classification, and filtering out a part of candidate ROI values; the characteristics of the original image are corresponding to the characteristic patterns, and the ROIs are classified and Mask is generated, so that the training process can be completed; the loss function in the training process is the sum of classification error, detection error and segmentation error.
In this embodiment, after training the optimized Mask R-CNN neural network, the obtained model is used for oropharynx segmentation, and the segmented two-dimensional oropharynx image can be output by inputting the oropharynx data set into the trained optimized Mask R-CNN neural network.
Specifically, in one implementation of the present embodiment, step S200 includes the steps of:
step S201, inputting the oropharynx data set into the optimized Mask R-CNN neural network;
step S202, extracting image features through the optimized Mask R-CNN neural network, generating an image candidate region, and performing semantic segmentation to obtain a segmented two-dimensional oropharyngeal image.
In the embodiment, in the process of performing oropharynx partial segmentation on a model, firstly, image features of an oropharynx data set are extracted through an acceptance v2 network, then, candidate areas are generated through an area candidate network (RPN), semantic segmentation of oropharynx parts is performed through a Full Convolution Network (FCN), and finally, detection and positioning are performed on each area through a full connection layer (FC), so that picture information with masks is output, and mapping of a two-dimensional image to a three-dimensional point cloud is realized subsequently.
Specifically, in one implementation of the present embodiment, step S202 includes the steps of:
step S202a, extracting features of the oropharynx data set through an acceptance v2 network to obtain corresponding image features;
step S202b, the extracted image features are transmitted into a region candidate network, and classified by an ROI classifier to generate an image candidate region;
step S202c, inputting the generated image candidate region into a full convolution network to perform semantic segmentation of the oropharyngeal region, inputting the segmented image into a full connection layer to perform detection and positioning, and outputting a mask image.
In this embodiment, the specific processing procedures of the collected image data in the acceptance v2 network, the region candidate network (RPN), the Full Convolution Network (FCN) and the full connection layer (FC) are as follows:
the preprocessed image (namely, the oropharynx data set) is transmitted into an acceptance v2 network, and the characteristics of the image are extracted; then, the target area candidate network (RPN) is transmitted in, image scanning is carried out through a sliding window, and an area with the target is searched for, so that a boundary box is generated; at this time, the target is divided into specific classes and backgrounds by using an ROI classifier in a region candidate network (RPN), the positions of the frames are further finely adjusted by using a frame regression, the positive region selected by the ROI classifier is taken as input, the operations of a Full Convolution Network (FCN) and a full connection layer (FC) are performed on each ROI, the classification and Mask generation are completed, and picture information with masks is output so as to realize the mapping of the two-dimensional image to the three-dimensional point cloud in the follow-up process.
The mask picture processed in this embodiment includes the following information: background, mouth, back wall of pharynx, uvula, etc.
As shown in fig. 1, an embodiment of the present invention provides a method for dividing an oral cavity three-dimensional point cloud of a nucleic acid detection robot, including the following steps:
and step S300, mapping the segmented two-dimensional oropharynx image to a three-dimensional point cloud to obtain a three-dimensional point cloud segmented image of the oropharynx.
In this embodiment, the three-dimensional point cloud segmented image of the oropharynx can be obtained by mapping the two-dimensional oropharynx image output by the oral segmentation model to the three-dimensional point cloud.
Specifically, in one implementation of the present embodiment, step S300 includes the steps of:
step S301, marking edge pixels of a two-dimensional segmentation image on a depth image corresponding to the segmented two-dimensional oropharynx image according to an external reference matrix of an RGB-D camera;
step S302, mapping the pixel points of the segmented two-dimensional oropharynx image and depth image to a three-dimensional coordinate system according to a small-hole imaging model, and obtaining an intra-oral three-dimensional point cloud segmentation result.
In this embodiment, after the image segmentation result is obtained, edge pixels of the two-dimensional segmented image may be marked on a depth image corresponding to the current two-dimensional image according to an external parameter matrix of the RGB-D camera, that is, the image segmentation result is mapped onto the depth image, and finally, the segmented two-dimensional image and the depth image pixels are mapped onto a three-dimensional coordinate system by using a pinhole imaging model, so as to obtain an intra-oral three-dimensional point cloud segmentation result.
In this embodiment, the extrinsic matrix of the RGB-D camera is calibrated by a general extrinsic calibration method after the camera is assembled into a hardware system, and is a known parameter when the algorithm is running. The calibration method can be an existing calibration method, for example: and calibrating camera external parameters on the ROS system by using the two-dimensional code and the checkerboard.
The difference between this embodiment and the prior art is that the optimized Mask R-CNN neural network added with the acceptance v2 is used, so that the detection speed is faster during oral cavity detection and instance segmentation, and in addition, the two-dimensional image is mapped to the three-dimensional point cloud to obtain the depth information of the oropharyngeal swab part, which is also an improvement in this embodiment. Compared with the mode of calculating the sampling points, the position for the robot to sample is larger, the sampling accuracy is higher, and the fault tolerance is higher; compared with a mode of directly dividing the three-dimensional point cloud, the method has lower calculation force requirement and higher speed, and is more suitable for being used as a vision subsystem of an oropharyngeal swab robot.
In other implementations of the present embodiment, other neural network algorithms may be selected to increase the detection speed of the neural network by reducing accuracy; or the detection speed is reduced to improve the accuracy of detection. In the aspect of acquiring depth information, the purpose of assisting the robot in sampling can be achieved by giving up a two-dimensional to three-dimensional mapping mode and adopting other depth information acquisition modes.
The following technical effects are achieved through the technical scheme:
according to the embodiment, an oropharynx data set comprising a mouth, a tongue, a back pharyngeal wall and tonsils is established according to acquired image data, and two-dimensional image processing is carried out on the oropharynx data set through an optimized Mask R-CNN neural network, so that a segmented two-dimensional oropharynx image is obtained; and mapping the segmented two-dimensional oropharynx image to a three-dimensional point cloud to obtain a three-dimensional point cloud segmented image of the oropharynx. The embodiment can be used for the robot to sample the position more, the sampling accuracy is higher, the fault tolerance is higher, the calculation force requirement is lower, the speed is faster, and the oropharyngeal swab robot can be more efficiently and accurately assisted to finish the nucleic acid sampling work.
Exemplary apparatus
Based on the above embodiment, the present invention further provides a robot including: the system comprises a processor, a memory, an interface, a display screen and a communication module which are connected through a system bus; wherein the processor is configured to provide computing and control capabilities; the memory includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used for connecting external equipment, such as mobile terminals, computers and other equipment; the display screen is used for displaying corresponding information; the communication module is used for communicating with a cloud server or a mobile terminal.
The computer program when executed by the processor is used for realizing the operation of the oral cavity three-dimensional point cloud segmentation method of the nucleic acid detection robot.
It will be appreciated by those skilled in the art that the schematic block diagram shown in fig. 3 is merely a block diagram of a portion of the structure associated with the present inventive arrangements and is not limiting of the robots to which the present inventive arrangements are applied, and that a particular robot may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a robot is provided, comprising: a processor and a memory storing an oral three-dimensional point cloud segmentation program of a nucleic acid detection robot, which when executed by the processor is for implementing the operation of the oral three-dimensional point cloud segmentation method of the nucleic acid detection robot as described above.
In one embodiment, a storage medium is provided, wherein the storage medium stores an oral three-dimensional point cloud segmentation program of a nucleic acid detection robot, which when executed by the processor is for implementing the operations of the oral three-dimensional point cloud segmentation method of the nucleic acid detection robot as described above.
Those skilled in the art will appreciate that implementing all or part of the above-described methods may be accomplished by way of a computer program comprising instructions for the relevant hardware, the computer program being stored on a non-volatile storage medium, the computer program when executed comprising the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory.
In summary, the invention provides an oral cavity three-dimensional point cloud segmentation method of a nucleic acid detection robot and the robot, wherein the method comprises the following steps: establishing an oropharynx data set according to the acquired image data; wherein the oropharyngeal dataset comprises: data sets of mouth, tongue, posterior pharyngeal wall and tonsils; performing two-dimensional image processing on the oropharynx data set through the optimized Mask R-CNN neural network to obtain a segmented two-dimensional oropharynx image; and mapping the segmented two-dimensional oropharynx image to a three-dimensional point cloud to obtain a three-dimensional point cloud segmented image of the oropharynx. The robot camera can acquire the point cloud image of the part to be sampled of the oropharynx in real time, and can more efficiently and accurately assist the oropharynx swab robot to finish the nucleic acid sampling work.
It is to be understood that the invention is not limited in its application to the examples described above, but is capable of modification and variation in light of the above teachings by those skilled in the art, and that all such modifications and variations are intended to be included within the scope of the appended claims.
Claims (10)
1. The method for dividing the three-dimensional point cloud of the oral cavity of the nucleic acid detection robot is characterized by comprising the following steps of:
establishing an oropharynx data set according to the acquired image data; wherein the oropharyngeal dataset comprises: data sets of mouth, tongue, posterior pharyngeal wall and tonsils;
performing two-dimensional image processing on the oropharynx data set through the optimized Mask R-CNN neural network to obtain a segmented two-dimensional oropharynx image;
and mapping the segmented two-dimensional oropharynx image to a three-dimensional point cloud to obtain a three-dimensional point cloud segmented image of the oropharynx.
2. The method of claim 1, wherein the creating an oropharynx dataset from the acquired image data comprises:
collecting a plurality of oral cavity RGB images with different postures;
calibrating the mouth, tongue, back pharyngeal wall and tonsils of each oral cavity RGB image in an image labeling tool according to the oral cavity anatomical model to obtain a calibrated oral cavity RGB image;
and establishing the oropharynx data set according to the calibrated oral cavity RGB image.
3. The method for three-dimensional point cloud segmentation of an oral cavity of a nucleic acid detection robot according to claim 1, wherein the optimized Mask R-CNN neural network comprises: an acceptance v2 network, a region candidate network, a full convolution network, and a full connectivity layer.
4. The method for three-dimensional point cloud segmentation of an oral cavity of a nucleic acid detection robot according to claim 1, wherein the two-dimensional image processing of the oropharynx data set by an optimized Mask R-CNN neural network comprises:
and training the optimized Mask R-CNN neural network according to the oropharynx data set to obtain a trained oropharynx partial model.
5. The method for three-dimensional point cloud segmentation of an oral cavity of a nucleic acid detection robot according to claim 1, wherein the performing two-dimensional image processing on the oropharynx data set through the optimized Mask R-CNN neural network to obtain a segmented two-dimensional oropharynx image comprises:
inputting the oropharynx data set into the optimized Mask R-CNN neural network;
and extracting image features through the optimized Mask R-CNN neural network, generating an image candidate region, and carrying out semantic segmentation to obtain a segmented two-dimensional oropharyngeal image.
6. The method for three-dimensional point cloud segmentation of an oral cavity of a nucleic acid detection robot according to claim 5, wherein the extracting image features by the optimized Mask R-CNN neural network, generating image candidate regions, and performing semantic segmentation, comprises:
extracting features of the oropharynx data set through an acceptance v2 network to obtain corresponding image features;
the extracted image features are transmitted into a region candidate network, and classified by an ROI classifier to generate an image candidate region;
and inputting the generated image candidate region into a full convolution network to perform semantic segmentation of the oropharynx part, inputting the segmented image into a full connection layer to perform detection and positioning, and outputting a mask image.
7. The method of three-dimensional point cloud segmentation of an oral cavity of a nucleic acid detection robot according to claim 6, wherein the mask image comprises: background, mouth, posterior pharyngeal wall, and uvula.
8. The method of claim 1, wherein mapping the segmented two-dimensional oropharynx image to a three-dimensional point cloud to obtain a three-dimensional point cloud segmented image of the oropharynx, comprises:
marking edge pixels of the two-dimensional segmented image on the depth image corresponding to the segmented two-dimensional oropharyngeal image according to the external reference matrix of the RGB-D camera;
and mapping the pixel points of the segmented two-dimensional oropharynx image and depth image to a three-dimensional coordinate system according to a small-hole imaging model, and obtaining an intra-oral three-dimensional point cloud segmentation result.
9. A robot, comprising: a processor and a memory storing an oral three-dimensional point cloud segmentation program of a nucleic acid detection robot, which when executed by the processor is configured to implement the operations of the oral three-dimensional point cloud segmentation method of the nucleic acid detection robot according to any one of claims 1 to 8.
10. A storage medium, characterized in that the storage medium is a computer-readable storage medium, the storage medium storing an oral three-dimensional point cloud segmentation program of a nucleic acid detection robot, which when executed by a processor is for realizing the operations of the oral three-dimensional point cloud segmentation method of the nucleic acid detection robot according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211708393.2A CN116129112A (en) | 2022-12-28 | 2022-12-28 | Oral cavity three-dimensional point cloud segmentation method of nucleic acid detection robot and robot |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211708393.2A CN116129112A (en) | 2022-12-28 | 2022-12-28 | Oral cavity three-dimensional point cloud segmentation method of nucleic acid detection robot and robot |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116129112A true CN116129112A (en) | 2023-05-16 |
Family
ID=86303934
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211708393.2A Pending CN116129112A (en) | 2022-12-28 | 2022-12-28 | Oral cavity three-dimensional point cloud segmentation method of nucleic acid detection robot and robot |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116129112A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200253673A1 (en) * | 2017-06-28 | 2020-08-13 | Intuitive Surgical Operations, Inc, | Systems and methods for projecting an endoscopic image to a three-dimensional volume |
| CN111862029A (en) * | 2020-07-15 | 2020-10-30 | 哈尔滨市科佳通用机电股份有限公司 | Fault detection method for bolt part of vertical shock absorber of railway motor train unit |
| CN111906784A (en) * | 2020-07-23 | 2020-11-10 | 湖南爱米家智能科技有限公司 | Pharyngeal swab double-arm sampling robot based on machine vision guidance and sampling method |
| CN112472148A (en) * | 2020-09-13 | 2021-03-12 | 张新语 | Nucleic acid detection device |
| CN113192080A (en) * | 2021-04-27 | 2021-07-30 | 北京大学 | Portable cell imaging and intelligent analysis system based on edge calculation |
| CN113855087A (en) * | 2021-09-29 | 2021-12-31 | 清华大学 | Portable self-service oropharynx swab sampling robot |
| WO2022027921A1 (en) * | 2020-08-05 | 2022-02-10 | 谈斯聪 | Medical robotic device, system, and method |
| CN114120283A (en) * | 2021-11-29 | 2022-03-01 | 华中光电技术研究所（中国船舶重工集团公司第七一七研究所） | Method for distinguishing unknown obstacles in road scene three-dimensional semantic segmentation |
| CN114841990A (en) * | 2022-05-26 | 2022-08-02 | 长沙云江智科信息技术有限公司 | Self-service nucleic acid collection method and device based on artificial intelligence |
| CN114998230A (en) * | 2022-05-23 | 2022-09-02 | 肇庆学院 | Pharynx swab oral cavity nucleic acid sampling area image identification method |
-
2022
- 2022-12-28 CN CN202211708393.2A patent/CN116129112A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200253673A1 (en) * | 2017-06-28 | 2020-08-13 | Intuitive Surgical Operations, Inc, | Systems and methods for projecting an endoscopic image to a three-dimensional volume |
| CN111862029A (en) * | 2020-07-15 | 2020-10-30 | 哈尔滨市科佳通用机电股份有限公司 | Fault detection method for bolt part of vertical shock absorber of railway motor train unit |
| CN111906784A (en) * | 2020-07-23 | 2020-11-10 | 湖南爱米家智能科技有限公司 | Pharyngeal swab double-arm sampling robot based on machine vision guidance and sampling method |
| WO2022027921A1 (en) * | 2020-08-05 | 2022-02-10 | 谈斯聪 | Medical robotic device, system, and method |
| CN112472148A (en) * | 2020-09-13 | 2021-03-12 | 张新语 | Nucleic acid detection device |
| CN113192080A (en) * | 2021-04-27 | 2021-07-30 | 北京大学 | Portable cell imaging and intelligent analysis system based on edge calculation |
| CN113855087A (en) * | 2021-09-29 | 2021-12-31 | 清华大学 | Portable self-service oropharynx swab sampling robot |
| CN114120283A (en) * | 2021-11-29 | 2022-03-01 | 华中光电技术研究所（中国船舶重工集团公司第七一七研究所） | Method for distinguishing unknown obstacles in road scene three-dimensional semantic segmentation |
| CN114998230A (en) * | 2022-05-23 | 2022-09-02 | 肇庆学院 | Pharynx swab oral cavity nucleic acid sampling area image identification method |
| CN114841990A (en) * | 2022-05-26 | 2022-08-02 | 长沙云江智科信息技术有限公司 | Self-service nucleic acid collection method and device based on artificial intelligence |
Non-Patent Citations (3)
| Title |
|---|
| YONGQUAN CHEN ET AL.: "A collaborative robot for COVID-19 oropharyngeal swabbing", 《ROBOTICS AND AUTONOMOUS SYSTEMS》, pages 5 - 6 * |
| 王盛杰;刘博;李和平;陈臻;吕升林;: "基于面阵三维成像激光雷达的目标点云分割技术", 半导体光电, no. 05 * |
| 胡敦利;张维;: "基于Mask R-CNN的动态物体剔除方法在RGB-D视觉SLAM中的应用", 工业控制计算机, no. 03 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110956635B (en) | Lung segment segmentation method, device, equipment and storage medium | |
| CN109859203B (en) | Defect tooth image identification method based on deep learning | |
| WO2021051885A1 (en) | Target labeling method and apparatus | |
| CN111862044B (en) | Ultrasonic image processing method, ultrasonic image processing device, computer equipment and storage medium | |
| CN110570435B (en) | Method and device for carrying out damage segmentation on vehicle damage image | |
| CN110136153B (en) | Image processing method, device and storage medium | |
| CN108921854B (en) | Method and system for labeling irregular lesion region of gastrointestinal endoscope image | |
| CN111145220B (en) | Tunnel target track tracking method based on visual information | |
| CN114121232A (en) | Classification model training method and device, electronic equipment and storage medium | |
| US20240005494A1 (en) | Methods and systems for image quality assessment | |
| CN113129278B (en) | X-Ray picture femoral shaft fracture detection method based on non-local separation attention mechanism | |
| WO2024087910A1 (en) | Orthodontic treatment monitoring method and apparatus, device, and storage medium | |
| CN111401102A (en) | Deep learning model training method and device, electronic equipment and storage medium | |
| CN113707279A (en) | Auxiliary analysis method and device for medical image picture, computer equipment and medium | |
| CN116129112A (en) | Oral cavity three-dimensional point cloud segmentation method of nucleic acid detection robot and robot | |
| CN113454684A (en) | Key point calibration method and device | |
| CN117635833A (en) | Image processing method, device, equipment and storage medium | |
| CN117496389A (en) | Hand hygiene real-time detection method suitable for Android equipment | |
| CN113140031B (en) | Three-dimensional image modeling system and method and oral cavity scanning equipment applying same | |
| CN115700805A (en) | Plant height detection method, device, equipment and storage medium | |
| CN117152507B (en) | Tooth health state detection method, device, equipment and storage medium | |
| CN112150422A (en) | Modeling method of oral health self-detection model based on multitask learning | |
| CN114205482B (en) | Scanning device, scanning control method, electronic apparatus, and storage medium | |
| WO2024124639A1 (en) | Fault locating method and fault locating apparatus for photovoltaic power station, and electronic device | |
| EP4346221A1 (en) | Photographing method and apparatus, storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |