CN110992410B

CN110992410B - Robot vision guiding method and device based on RGB-D data fusion

Info

Publication number: CN110992410B
Application number: CN201911196215.4A
Authority: CN
Inventors: 刁世普; 郑振兴; 秦磊
Original assignee: Guangdong Polytechnic Normal University
Current assignee: FOSHAN DILE VISION TECHNOLOGY Co.,Ltd.
Priority date: 2019-11-29
Filing date: 2019-11-29
Publication date: 2020-10-30
Anticipated expiration: 2039-11-29
Also published as: CN110992410A; WO2021103558A1

Abstract

The invention discloses a robot vision guiding method and device based on RGB-D data fusion, which are based on an RGB-D composite sensor, and a machining guide point sequence is finally obtained by sequentially carrying out the steps of machining target identification, machining target segmentation, machining path point acquisition and machining guide point conversion from the beginning of machining target data acquisition, so that the calculation time is reduced, the requirement on real-time processing is met, the requirement on the performance of software and hardware is reduced, the cost can be saved, the development difficulty is reduced, and the requirement on a high-speed large-scale production mode is met.

Description

Robot vision guiding method and device based on RGB-D data fusion

Technical Field

The invention relates to the field of robot vision, in particular to a robot vision guiding method and device based on RGB-D data fusion.

Background

Automation equipment (robot system) for manufacturing strong country instruments is required to be accelerated and intelligentized. An important means of intellectualization of automation equipment is to equip the machine with an "eye" and a "brain" able to cooperate with this eye. The "eye" may be a monocular camera, a binocular camera, a multi-view camera, a three-dimensional scanner, or an RGB-D (RGB + Depth) sensor. The core work content of the automation equipment intellectualization comprises the following steps: the image data acquired by the "eye" is analyzed (e.g., image recognition), and the robotic system is guided to perform a particular machining or assembly operation based on the analysis. Therefore, the image data analysis based on two-dimensional images, which is widely adopted at present, is a key basic core technology. However, the method based on two-dimensional image data analysis is easily interfered by external environmental factors such as light and the like, so that the recognition accuracy is low, the robustness is poor, and the precision does not meet the requirement. In order to meet the current requirements for high speed and high precision of image data analysis, the RGB-D sensor, which can acquire the depth information of the corresponding two-dimensional image data that is not easily interfered by external environmental factors such as light while acquiring the conventional two-dimensional image data, is popularized and applied. The 3D data (point cloud data) corresponding to the target object may be obtained by processing the RGB image data and the depth data acquired by the RGB-D sensor. The existing point cloud analysis method has the following disadvantages: the method for segmenting the processing target by the point cloud is time-consuming and cannot meet the requirement of current high-speed production; the 3D processing target positioning method based on pure 3D point cloud and deep learning has poor accuracy and low resolution precision, and does not meet the requirement of current high-precision processing. In summary, the existing point cloud analysis method is not highly universal and is only suitable for a specific automatic processing system. In addition, since no key information based on RGB-D data for a specific machining or assembly operation task is currently available to guide the research application of the robot system. Therefore, the invention discloses a method and a device for determining a target direction of a robot based on RGB-D data, thereby providing a visual guidance method and a visual guidance device which can meet the detection precision requirement of a processing target required by current processing and are suitable for a processing target workpiece.

Disclosure of Invention

The invention mainly aims to provide a robot vision guiding method and device based on RGB-D data fusion, aiming at solving the technical problems that the existing point cloud segmentation processing target method is time-consuming and cannot meet the requirement of current high-speed production, the 3D processing target positioning method based purely on 3D point cloud and deep learning is poor in accuracy, low in resolution precision and not in line with the requirement of current high-precision processing, and the existing point cloud analysis method is not high in universality and only suitable for a specific automatic processing system. In view of these disadvantages, the object of the present invention is to provide a method and an apparatus for recognizing a target orientation, which can satisfy the accuracy of detection of a processing target required for current processing, and which are suitable for a workpiece to be processed, which can satisfy the requirements of detection and processing of a current processing target, and which can satisfy the requirements of accuracy for medium accuracy, and which are suitable for a workpiece to be processed.

In order to solve the above problems, the present invention provides a robot vision guiding method based on RGB-D data fusion, comprising:

step 1, acquiring an RGB two-dimensional image and depth data containing a target to be processed through an RGB-D composite sensor, and registering the RGB two-dimensional image and the depth data according to preset registration parameters of the RGB-D composite sensor to generate a registered two-dimensional image I corresponding to the RGB two-dimensional image_RGBAnd registered depth data I corresponding to said depth data_D；

Step 2, using the image segmentation model trained in advance to obtain the registered two-dimensional image I as input_RGBDividing the region S corresponding to the target to be processed_RGBAccording to said region S_RGBFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D；

Step 3, from the 3D point cloud data S_3DExtracting significant feature point sequence { KX_i}_i＝1->mUsing the significant feature point sequence { KX_i}_i＝1->m(ii) assigning the 3D point cloud data S for a cut-off point_3DDivided into local point cloud sets { S_3D-j}_j＝1->nSaid KX_iIs a significant feature point, i is the corresponding significant feature point KX_iThe value range of i is [1, m ]]M is the significant feature point KX_iTotal number of said S_3D-jIs a local point cloud, j is the local point cloud S_3D-jThe value range of j is [1, n ]]N is the local point cloud S_3D-jThe total number of (c);

step 4, sequentially setting the j to be from 1 to n, and searching and extracting the point cloud S vertical to the local point by using a local search algorithm_3D-jAnd the plane of the bounding box of (A) and the corresponding local point cloud S_3D-jMachining path point SX at the boundary of (1)_jThereby obtaining the machining path point sequence { SX_j}_j＝1->n；

Step 5, setting the j to be from 1 to n in sequence, and using a conversion matrix BET to convert the processing path point SX_jConversion to a machining guide point BX_jSo as to process the path point sequence { SX_j}_j＝1->nConversion into a sequence of machining guide points { BX_j}_j＝1->n。

Preferably, in the step 4, the j is sequentially set to be from 1 to n, and the local point cloud S is obtained through a surface fitting algorithm_3D-jFitting to curved surfaces SS_3D-jThen searching and extracting the point cloud S vertical to the local point by using the local search algorithm_3D-jThe plane of the bounding box and the corresponding curved surface SS_3D-jThe machining path point SX at the junction of_jThereby obtaining the machining path point sequence { SX_j}_j＝1->n。

Preferably, said step 2, registering said two-dimensional images I_RGBAnd corresponding said registration depth data I_DAre merged to generate fused data I_RGB-DUsing said fused data I as input by a pre-trained image segmentation model_RGB-DDividing the region S corresponding to the target to be processed_RGB-DAccording to said region S_RGB-DFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D。

Preferably, the training samples required for the pre-training of the image segmentation model of the step 2 are obtained by acquiring the registered two-dimensional image I containing the object to be processed of the same type using the RGB-D composite sensor_RGBAnd corresponding said registration depth data I_DAnd (4) generating.

Preferably, the conversion matrix BET of step 5 is generated by a previously calibrated Denavit-Hartenberg (D-H) parameter.

The invention further provides a robot vision guiding device based on RGB-D data fusion, which comprises:

the processing target data acquisition module is used for acquiring an RGB two-dimensional image and depth data containing a target to be processed through an RGB-D composite sensor, and registering the RGB two-dimensional image and the depth data according to preset registration parameters of the RGB-D composite sensor so as to generate a registered two-dimensional image I corresponding to the RGB two-dimensional image_RGBAnd registered depth data I corresponding to said depth data_D；

A processing target recognition module for using a pre-trained image segmentation model to extract the registered two-dimensional image I from the image as input_RGBDividing the region S corresponding to the target to be processed_RGBAccording to said region S_RGBFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D；

A processing target segmentation module for segmenting the 3D point cloud data S_3DExtracting significant feature point sequence { KX_i}_i＝1->mUsing the significant feature point sequence { KX_i}_i＝1->m(ii) assigning the 3D point cloud data S for a cut-off point_3DDivided into local point cloud sets { S_3D-j}_j＝1->nSaid KX_iIs a significant feature point, i is the corresponding significant feature point KX_iThe value range of i is [1, m ]]M is the significant feature point KX_iTotal number of said S_3D-jIs a local point cloud, j is the local point cloud S_3D-jThe value range of j is [1, n ]]N is the local point cloud S_3D-jThe total number of (c);

a processing path point acquisition module for sequentially setting the j from 1 to n, searching and extracting the point cloud S perpendicular to the local point by using a local search algorithm_3D-jAnd the plane of the bounding box of (A) and the corresponding local point cloud S_3D-jMachining path point SX at the boundary of (1)_jThereby obtaining the machining path point sequence { SX_j}_j＝1->n；

A processing guide point conversion module for setting the j from 1 to n in sequence, wherein the processing guide point conversion module comprises a BE (least squares element) passing through a conversion matrixT will the machining path point SX_jConversion to a machining guide point BX_jSo as to process the path point sequence { SX_j}_j＝1->nConversion into a sequence of machining guide points { BX_j}_j＝1->n。

Preferably, the processing path point obtaining module is configured to set j to be from 1 to n in sequence, and use a surface fitting algorithm to obtain the local point cloud S_3D-jFitting to curved surfaces SS_3D-jThen searching and extracting the point cloud S vertical to the local point by using the local search algorithm_3D-jThe plane of the bounding box and the corresponding curved surface SS_3D-jThe machining path point SX at the junction of_jThereby obtaining the machining path point sequence { SX_j}_j＝1->n。

Preferably, the processing target identification module is used for registering the two-dimensional image I_RGBAnd corresponding said registration depth data I_DAre merged to generate fused data I_RGB-DUsing said fused data I as input by a pre-trained image segmentation model_RGB-DDividing the region S corresponding to the target to be processed_RGB-DAccording to said region S_RGB-DFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D。

Preferably, training samples required for the pre-training of the image segmentation model among the processing target recognition modules are acquired by using the RGB-D composite sensor to acquire a registered two-dimensional image I containing the target to be processed of the same kind_RGBAnd corresponding said registration depth data I_DAnd (4) generating.

Preferably, the conversion matrix BET among the machining guide point conversion modules is generated by a previously calibrated Denavit-Hartenberg (D-H) parameter.

According to the technical scheme, the method can meet the requirement of the detection precision of the machining target required by the current machining, greatly reduces the calculation amount, reduces the calculation complexity, accelerates the processing speed, reduces the calculation time, meets the requirement of real-time processing, reduces the requirement on the performance of software and hardware, can save the cost, reduces the development difficulty, and meets the requirement on a high-speed large-scale production mode.

Drawings

FIG. 1 is a schematic flow chart of a robot vision guiding method based on RGB-D data fusion according to a first embodiment of the present invention;

FIG. 2 is a functional block diagram of a first embodiment of the robot vision guiding device based on RGB-D data fusion according to the present invention;

fig. 3 is a schematic diagram of an RGB-D composite sensor embodying the present invention.

The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Various embodiments for implementing the present invention will now be described with reference to the accompanying drawings. In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in themselves. Thus, "module" and "component" may be used in a mixture.

Referring to fig. 1, fig. 1 is a flowchart illustrating a robot vision guiding method based on RGB-D data fusion according to a first embodiment of the present invention. In the embodiment shown in fig. 1, the method for guiding the robot vision based on RGB-D data fusion includes the following steps:

and S10, acquiring processing target data.

The method comprises the steps of acquiring an RGB two-dimensional image and depth data containing a target to be processed through an RGB-D composite sensor, and carrying out registration processing on the RGB two-dimensional image and the depth data according to preset registration parameters of the RGB-D composite sensor, so as to generate a registration two-dimensional image I corresponding to the RGB two-dimensional image_RGBAnd registered depth data I corresponding to said depth data_D。

Referring to fig. 3, the RGB-D composite sensor is placed on top of the robot arm D40 with the RGB camera D20 in the middle of the RGB-D composite vision sensor, and the color image data is compressed before being transferred to the computer to ensure speed in analyzing the RGB data. The sensors D10 and D30 on the left and right sides of the RGB-D composite vision sensor are responsible for emitting and receiving infrared rays, respectively: firstly, infrared rays are emitted to an object O10 to be processed through an infrared emitter D10 on the left side, and the infrared rays have high randomness, so that light spots formed by the reflection of the infrared rays at any two different positions in space are different, and a three-dimensional 'light code' is formed for the environment; then, an infrared image in the visual field is collected through the infrared receiver D30 on the right side; finally, a series of complex calculations are carried out on the infrared image by using the parameters of the RGB-D composite vision sensor, and the depth data in the visual field can be obtained.

And S20, identifying the processing target.

I.e. from the registered two-dimensional image I as input using a pre-trained image segmentation model_RGBDividing the region S corresponding to the target to be processed_RGBAccording to said region S_RGBFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D。

The image segmentation model based on the deep learning framework realizes a neural network model for semantic segmentation of the target to be processed, and has the characteristics of high precision, high processing speed and real-time processing.

And S30, dividing the processing target.

I.e. from the 3D point cloud data S_3DExtracting significant feature point sequence { KX_i}_i＝1->mUsing the significant feature point sequence { KX_i}_i＝1->m(ii) assigning the 3D point cloud data S for a cut-off point_3DDivided into local point cloud sets { S_3D-j}_j＝1->nSaid KX_iIs a significant feature point, i is the corresponding significant feature point KX_iThe value range of i is [1, m ]]M is the significant feature point KX_iOf saidS_3D-jIs a local point cloud, j is the local point cloud S_3D-jThe value range of j is [1, n ]]N is the local point cloud S_3D-jThe total number of (a).

The KX_iIs a coordinate vector of the corresponding salient feature point, said S_3D-jA set of all points comprising the local point cloud.

And S40, acquiring machining path points.

Namely, sequentially setting the j from 1 to n, searching and extracting the point cloud S vertical to the local point by using a local search algorithm_3D-jAnd the plane of the bounding box of (A) and the corresponding local point cloud S_3D-jMachining path point SX at the boundary of (1)_jThereby obtaining the machining path point sequence { SX_j}_j＝1->n。

The step S30 is to obtain the 3D point cloud data S of the target to be processed_3DWith a sequence of salient feature points { KX_i}_i＝1->mDividing the boundary point into local point cloud sets (S)_3D-j}_j＝1->nIn step S40, the corresponding machining path point sequence { SX can be obtained simply and effectively_j}_j＝1->n. Especially for very complex said 3D point cloud data S_3DMore beneficial to processing path point sequence { SX_j}_j＝1->nAnd stability and robustness are greatly enhanced.

And S50, converting the machining guide point.

Setting j to from 1 to n in sequence, using a process comprising converting the machining path points SX by means of a transformation matrix BET_jConversion to a machining guide point BX_jSo as to process the path point sequence { SX_j}_j＝1->nConversion into a sequence of machining guide points { BX_j}_j＝1->n。

Machining path point SX_jThe position coordinate information in the coordinate system corresponding to the RGB-D composite vision sensor needs to be converted into the position coordinate information in the corresponding working coordinate system. Finally, the machining guide point sequence { BX_j}_j＝1->nSends the corresponding operation to the robot to execute. By aligning sequences of machining guide points { BX_j}_j＝1->nThe machining path can be obtained by using a curve interpolation algorithm, so that the machining operation is guided.

Therefore, the processing steps can reduce the calculation amount, reduce the calculation complexity, accelerate the processing speed, reduce the calculation time, meet the requirements of real-time processing, reduce the requirements on the performance of software and hardware, save the cost, reduce the development difficulty and meet the requirements on a high-speed large-scale production mode.

Further, in the step S40, setting the j to be from 1 to n in sequence, and performing a surface fitting algorithm to obtain the local point cloud S_3D-jFitting to curved surfaces SS_3D-jThen searching and extracting the point cloud S vertical to the local point by using the local search algorithm_3D-jThe plane of the bounding box and the corresponding curved surface SS_3D-jThe machining path point SX at the junction of_jThereby obtaining the machining path point sequence { SX_j}_j＝1->n。

Using fitted surfaces SS_3D-jCan filter out local point cloud S_3D-jData redundancy of (2) to make local point cloud S_3D-jThe data are homogenized, the measurement deviation caused by a measurement system is reduced, and the fluctuation of the data is eliminated. Particularly, a smooth machining path can be generated by NURBS curve fitting.

Further, the step S20 is to register the two-dimensional image I_RGBAnd corresponding said registration depth data I_DAre merged to generate fused data I_RGB-DUsing said fused data I as input by a pre-trained image segmentation model_RGB-DDividing the region S corresponding to the target to be processed_RGB-DAccording to said region S_RGB-DFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D。

Using said fusion data I_RGB-DThe area S corresponding to the target to be processed can be effectively divided_RGB-DPrecision and accuracy of, and is largeThe robustness and stability of the segmentation are greatly enhanced.

Further, the training samples required for the pre-training of the image segmentation model of the step S20 are obtained by acquiring the registered two-dimensional image I containing the object to be processed of the same kind using the RGB-D composite sensor_RGBAnd corresponding said registration depth data I_DAnd (4) generating.

Using the RGB-D composite sensor with local grid space placement, a large number of the registered two-dimensional images I of the object to be processed can be obtained_RGBAnd corresponding said registration depth data I_DThen, the training sample can be made by labeling; and then training the image segmentation model based on the deep learning framework, and finely adjusting related parameters in the training process until the accuracy of the model reaches an expected value. By the processing steps, a large amount of training sample data can be obtained very efficiently, so that the requirements of the image segmentation model based on the deep learning framework on precision and robustness are ensured.

Further, the conversion matrix BET of the step S50 is generated by a previously calibrated Denavit-Hartenberg (D-H) parameter.

The purpose of calibrating the Denavit-Hartenberg (D-H) parameters by using a laser tracker is to improve the overall accuracy of the robot vision guidance algorithm and to ensure that the machining path point SX is fixed_jConversion to a machining guide point BX_jThe method is accurate, and has the characteristics of high processing speed, maturity, reliability and easy engineering realization.

The robot vision guiding method based on RGB-D data fusion in the first embodiment of the robot vision guiding method based on RGB-D data fusion of the present invention may be implemented by the robot vision guiding device based on RGB-D data fusion provided in the first embodiment of the robot vision guiding device based on RGB-D data fusion of the present invention.

Referring to fig. 2, fig. 2 is a robot vision guiding device 1 based on RGB-D data fusion according to a first embodiment of the robot vision guiding device based on RGB-D data fusion of the present invention, where the device 1 includes:

a processing target data acquisition module 10, configured to acquire an RGB two-dimensional image and depth data including a target to be processed through an RGB-D composite sensor, and perform registration processing on the RGB two-dimensional image and the depth data according to preset registration parameters of the RGB-D composite sensor, so as to generate a registration two-dimensional image I corresponding to the RGB two-dimensional image_RGBAnd registered depth data I corresponding to said depth data_D。

A processing target recognition module 20 for using a pre-trained image segmentation model to extract the registered two-dimensional image I from the image as input_RGBDividing the region S corresponding to the target to be processed_RGBAccording to said region S_RGBFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D。

A processing object segmentation module 30 for segmenting the 3D point cloud data S from_3DExtracting significant feature point sequence { KX_i}_i＝1->mUsing the significant feature point sequence { KX_i}_i＝1->m(ii) assigning the 3D point cloud data S for a cut-off point_3DDivided into local point cloud sets { S_3D-j}_j＝1->nSaid KX_iIs a significant feature point, i is the corresponding significant feature point KX_iThe value range of i is [1, m ]]M is the significant feature point KX_iTotal number of said S_3D-jIs a local point cloud, j is the local point cloud S_3D-jThe value range of j is [1, n ]]N is the local point cloud S_3D-jThe total number of (a).

A processing path point obtaining module 40 for sequentially setting the j from 1 to n, searching and extracting the point cloud S perpendicular to the local point by using a local search algorithm_3D-jAnd the plane of the bounding box of (A) and the corresponding local point cloud S_3D-jMachining path point SX at the boundary of (1)_jThereby obtaining the machining path point sequence { SX_j}_j＝1->n。

The processing target segmentation module 30 divides the 3D point cloud data S of the target to be processed_3DWith a sequence of salient feature points { KX_i}_i＝1->mDividing the boundary point into local point cloud sets (S)_3D-j}_j＝1->nThe machining path point acquiring module 40 may simply and effectively acquire the corresponding machining path point sequence { SX }_j}_j＝1->n. Especially for very complex said 3D point cloud data S_3DMore beneficial to processing path point sequence { SX_j}_j＝1->nAnd stability and robustness are greatly enhanced.

A machining guide point switching module 50 for sequentially setting the j from 1 to n, using a tool comprising switching the machining path point SX by a switching matrix BET_jConversion to a machining guide point BX_jSo as to process the path point sequence { SX_j}_j＝1->nConversion into a sequence of machining guide points { BX_j}_j＝1->n。

Machining path point SX_jThe position coordinate information in the coordinate system corresponding to the RGB-D composite vision sensor needs to be converted into the position coordinate information in the corresponding working coordinate system. Finally, the machining guide point sequence { BX_j}_j＝1->nAnd sending the data to the robot to execute corresponding operation. By aligning sequences of machining guide points { BX_j}_j＝1->nThe machining path can be obtained by using a curve interpolation algorithm, so that the machining operation is guided.

Therefore, the module can reduce the calculation amount, reduce the calculation complexity, accelerate the processing speed, reduce the calculation time, meet the requirements of real-time processing, reduce the requirements on the performance of software and hardware, save the cost, reduce the development difficulty and meet the requirements on a high-speed large-scale production mode.

Further, the processing path point obtaining module 40 is configured to set j to be from 1 to n in sequence, and apply a surface fitting algorithm to the local point cloud S_3D-jFitting to curved surfaces SS_3D-jThen searching and extracting the point cloud S vertical to the local point by using the local search algorithm_3D-jThe plane of the bounding box and the corresponding curved surface SS_3D-jThe machining path point SX at the junction of_jThereby obtaining the machining path point sequence { SX_j}_j＝1->n。

Further, the processing object identification module 20 is used for registering the two-dimensional image I_RGBAnd corresponding said registration depth data I_DAre merged to generate fused data I_RGB-DUsing a pre-trained image segmentation modelFor the input of said fusion data I_RGB-DDividing the region S corresponding to the target to be processed_RGB-DAccording to said region S_RGB-DFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D。

Using said fusion data I_RGB-DThe area S corresponding to the target to be processed can be effectively divided_RGB-DAnd greatly enhances the robustness and stability of the segmentation.

Further, the training samples required for the pre-training of the image segmentation model of the processing target recognition module 20 are obtained by acquiring the registered two-dimensional image I containing the target to be processed of the same kind using the RGB-D composite sensor_RGBAnd corresponding said registration depth data I_DAnd (4) generating.

Further, the conversion matrix BET of the machining guide point conversion module 50 is generated by a previously calibrated Denavit-Hartenberg (D-H) parameter.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

It will be apparent to those skilled in the art that the above-described block units or steps of the present invention may be implemented by a general purpose computing device, or alternatively, they may be implemented by program code executable by a computing device, so that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in a different order than here, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.

The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims

1. The robot vision guiding method based on RGB-D data fusion is characterized by comprising the following steps:

Step 2, using the image segmentation model trained in advance to obtain the registered two-dimensional image I as input_RGBA region S corresponding to the target to be processed is divided from the center_RGBAccording to said region S_RGBFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D；

Step 3, from the 3D point cloud data S_3DExtracting significant feature point sequence { KX_i}_i＝1->mUsing the significant feature point sequence { KX_i}_i＝1->m(ii) assigning the 3D point cloud data S for a cut-off point_3DDivided into local point cloud sets { S_3D-j}_j＝1->n，KX_iIs a significant feature point, i is the corresponding significant feature point KX_iThe value range of i is [1, m ]]M is the significant feature point KX_iTotal number of said S_3D-jIs a local point cloud, j is the local point cloud S_3D-jThe value range of j is [1, n ]]N is the local point cloud S_3D-jThe total number of (c);

step 4, sequentially setting the j from 1 to n, and carrying out a surface fitting algorithm on the local point cloud S_3D-jFitting to curved surfaces SS_3D-jAnd then by makingSearching and extracting the point cloud S perpendicular to the local point by using a local search algorithm_3D-jAnd the plane of the bounding box and the corresponding curved surface SS_3D-jMachining path point SX at the boundary of (1)_jThereby obtaining the machining path point sequence { SX_j}_j＝1->n；

2. The RGB-D data fusion-based robotic visual guidance method of claim 1,

the step 2 of registering the two-dimensional images I_RGBAnd corresponding said registration depth data I_DAre merged to generate fused data I_RGB-DUsing said fused data I as input by a pre-trained image segmentation model_RGB-DDividing the region S corresponding to the target to be processed_RGB-DAccording to said region S_RGB-DFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D。

3. The RGB-D data fusion-based robot vision guiding method as claimed in claim 1, wherein the training samples required for the pre-training of the image segmentation model of the step 2 are registered two-dimensional images I including the objects to be processed of the same kind by using the RGB-D composite sensor_RGBAnd corresponding said registration depth data I_DAnd (4) generating.

4. The RGB-D data fusion-based robot vision guidance method of claim 1, wherein the conversion matrix BET of the step 5 is generated by a previously calibrated Denavit-Hartenberg (D-H) parameter.

5. Robot vision guiding device based on RGB-D data fusion, characterized by includes:

A processing target recognition module for using a pre-trained image segmentation model to extract the registered two-dimensional image I from the image as input_RGBA region S corresponding to the target to be processed is divided from the center_RGBAccording to said region S_RGBFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D；

A processing target segmentation module for segmenting the 3D point cloud data S_3DExtracting significant feature point sequence { KX_i}_i＝1->mUsing the significant feature point sequence { KX_i}_i＝1->m(ii) assigning the 3D point cloud data S for a cut-off point_3DDivided into local point cloud sets { S_3D-j}_j＝1->n，KX_iIs a significant feature point, i is the corresponding significant feature point KX_iThe value range of i is [1, m ]]M is the significant feature point KX_iTotal number of said S_3D-jIs a local point cloud, j is the local point cloud S_3D-jThe value range of j is [1, n ]]N is the local point cloud S_3D-jThe total number of (c);

a processing path point acquisition module used for setting the j from 1 to n in sequence and carrying out local point cloud S through a curved surface fitting algorithm_3D-jFitting to curved surfaces SS_3D-jThen searching and extracting the point cloud S vertical to the local point by using a local search algorithm_3D-jAnd the plane of the bounding box and the corresponding curved surface SS_3D-jThe path of the junction ofPoint SX_jThereby obtaining the machining path point sequence { SX_j}_j＝1->n；

A machining guide point conversion module for setting the j from 1 to n in sequence, the machining path point SX being obtained by converting the matrix BET_jConversion to a machining guide point BX_jSo as to process the path point sequence { SX_j}_j＝1->nConversion into a sequence of machining guide points { BX_j}_j＝1->n。

6. The RGB-D data fusion-based robot vision guiding device of claim 5,

the processing target identification module is used for registering the two-dimensional image I_RGBAnd corresponding said registration depth data I_DAre merged to generate fused data I_RGB-DUsing said fused data I as input by a pre-trained image segmentation model_RGB-DDividing the region S corresponding to the target to be processed_RGB-DAccording to said region S_RGB-DFrom the registered depth data I_DExtracting 3D point cloud data S of the target to be processed_3D。

7. The RGB-D data fusion-based robot vision guiding device according to claim 5, wherein training samples required for the pre-training of the image segmentation model in the processing target recognition module are obtained by acquiring a registered two-dimensional image I containing the objects to be processed of the same kind using the RGB-D composite sensor_RGBAnd corresponding said registration depth data I_DAnd (4) generating.

8. The RGB-D data fusion based robot vision guidance apparatus of claim 5, wherein the conversion matrix BET among the processing guidance point conversion modules is generated by a precalibrated Denavit-Hartenberg (D-H) parameter.