TW201624326A - Method and apparatus for fusing 2D image and 3D point cloud data and the storage medium thereof - Google Patents
Method and apparatus for fusing 2D image and 3D point cloud data and the storage medium thereof Download PDFInfo
- Publication number
- TW201624326A TW201624326A TW103145219A TW103145219A TW201624326A TW 201624326 A TW201624326 A TW 201624326A TW 103145219 A TW103145219 A TW 103145219A TW 103145219 A TW103145219 A TW 103145219A TW 201624326 A TW201624326 A TW 201624326A
- Authority
- TW
- Taiwan
- Prior art keywords
- point cloud
- dimensional
- image
- dimensional point
- workpiece
- Prior art date
Links
Landscapes
- Length Measuring Devices By Optical Means (AREA)
- Image Processing (AREA)
Abstract
Description
本發明是有關於一種姿態估測的方法,其適用於具有對稱性的待測工件以進行姿態估測,且特別是有關於一種三維點雲融合二維影像的方法、裝置與儲存媒體。 The invention relates to a method for estimating a posture, which is suitable for a workpiece with a symmetry to perform a posture estimation, and particularly relates to a method, a device and a storage medium for merging a two-dimensional image by a three-dimensional point cloud.
近年來,三維感測元件被廣泛應用於工業自動化的應用當中,最常見的應用則是對於工件的取放。為了達成彈性且正確地取放工件的目標,一般藉由三維感測元件所擷取的三維點雲進行姿態估測,並分析出工件擺放的位置與角度,進而提供機器手臂進行自動化地上下料與加工。 In recent years, three-dimensional sensing components have been widely used in industrial automation applications, the most common application is the pick and place of the workpiece. In order to achieve the objective of elastically and correctly picking and dropping the workpiece, the attitude estimation is generally performed by the three-dimensional point cloud captured by the three-dimensional sensing component, and the position and angle of the workpiece placement are analyzed, thereby providing the robot arm to automatically move up and down. Materials and processing.
然而,由於三維感測元件所擷取的三維點雲無法辨識物體表面的顏色與圖案資訊,而且其不具有方向性。再者,在加工產業中,許多工件又具有高度對稱的特性,造成無法從三維點雲中得到工件正確的姿態資訊,使得在姿態估測的結果無法建立正確的姿態估測模型,進而造成姿態估測錯誤而影響夾取的正確性。 However, the three-dimensional point cloud captured by the three-dimensional sensing element cannot recognize the color and pattern information of the surface of the object, and it has no directivity. Moreover, in the processing industry, many workpieces have highly symmetrical characteristics, which makes it impossible to obtain correct posture information of the workpiece from the 3D point cloud, so that the result of the attitude estimation cannot establish a correct attitude estimation model, thereby causing a posture. Estimate errors and affect the correctness of the capture.
由上可知,解決對於對稱性工件姿態估測的問題,以提高對工件夾取的準確度,是本領域技術人員所亟欲達成的目標。 It can be seen from the above that solving the problem of symmetrical workpiece attitude estimation to improve the accuracy of the workpiece clamping is a goal that those skilled in the art desire to achieve.
本發明提供一種三維點雲融合二維影像的方法。其中一實施例包括:取得待測工件的二維影像與三維點雲,對此二維影像進行一圖樣特徵分析,取得待測工件的一方向性。依照取得的方向性對三維點雲進行分割,破壞三維點雲的對稱性,藉以根據分割後的三維點雲對待測工件進行一姿態估測。 The invention provides a method for three-dimensional point cloud fusion of two-dimensional images. One embodiment includes: obtaining a two-dimensional image of the workpiece to be tested and a three-dimensional point cloud, performing a pattern feature analysis on the two-dimensional image, and obtaining a directionality of the workpiece to be tested. The three-dimensional point cloud is segmented according to the obtained directionality, and the symmetry of the three-dimensional point cloud is destroyed, so that a posture estimation is performed according to the divided three-dimensional point cloud.
本發明提供一種三維點雲融合二維影像的裝置,適用於對具有對稱性的待測工件進行姿態估測。在其中一實施例中包括:三維感測元件、影像擷取單元與處理單元。三維感測元件用於擷取待測工件的三維點雲。影像擷取單元用於擷取待測工件的二維影像。處理單元用於對二維影像進行一圖樣特徵分析,取得待測工件的一方向性,並依照所得到的方向性對三維點雲進行分割。 The invention provides a device for three-dimensional point cloud fusion two-dimensional image, which is suitable for estimating a posture of a workpiece to be tested with symmetry. In one embodiment, a three-dimensional sensing element, an image capturing unit and a processing unit are included. The three-dimensional sensing element is used to capture a three-dimensional point cloud of the workpiece to be tested. The image capturing unit is used to capture a two-dimensional image of the workpiece to be tested. The processing unit is configured to perform a pattern feature analysis on the two-dimensional image, obtain a directionality of the workpiece to be tested, and segment the three-dimensional point cloud according to the obtained directionality.
本發明提供一種儲存程式的電腦可讀取之儲存媒體,當電腦載入該程式並執行後,以完成本發明一實施例之三維點雲融合二維影像之方法。 The present invention provides a computer readable storage medium for storing a program, and when the computer loads the program and executes it, the method for completing a three-dimensional point cloud fusion two-dimensional image according to an embodiment of the present invention is completed.
基於上述,本發明結合三維點雲與二維影像的方向性,並藉由分割三維點雲以破壞三維點雲的對稱性,產生具有姿態唯一性的三維點雲,以提升三維點雲對待測工件進行姿態估測的正 確性。 Based on the above, the present invention combines the directionality of the three-dimensional point cloud and the two-dimensional image, and breaks the symmetry of the three-dimensional point cloud by dividing the three-dimensional point cloud to generate a three-dimensional point cloud with unique posture to enhance the three-dimensional point cloud to be measured. Positive evaluation of the workpiece Authenticity.
為讓本發明的上述特徵和優點能更明顯易懂,下文特舉實施例,並配合所附圖式作詳細說明如下。 The above described features and advantages of the invention will be apparent from the following description.
10‧‧‧待測工件 10‧‧‧Workpiece to be tested
100‧‧‧三維點雲融合二維影像的裝置 100‧‧‧3D point cloud fusion 2D image device
102‧‧‧移動裝置 102‧‧‧Mobile devices
104‧‧‧資料處理中心 104‧‧‧Data Processing Center
106‧‧‧取像裝置 106‧‧‧Image capture device
108‧‧‧數位相機 108‧‧‧Digital camera
110‧‧‧雷射掃瞄器 110‧‧‧Laser scanner
112‧‧‧處理單元 112‧‧‧Processing unit
114‧‧‧儲存媒體 114‧‧‧Storage media
116‧‧‧通訊單元 116‧‧‧Communication unit
118‧‧‧顯示單元 118‧‧‧Display unit
S210、S230、S250、S270、S290‧‧‧三維點雲融合二維影像的方法步驟 Method steps for S210, S230, S250, S270, S290‧‧‧3D point cloud fusion 2D image
S310、S330、S350、S370、S390‧‧‧圖樣特徵分析方法的步驟 Steps for S310, S330, S350, S370, S390‧‧‧ pattern feature analysis methods
60‧‧‧區域 60‧‧‧ area
S710、S730、S750‧‧‧三維點雲分割方法的步驟 Steps for S710, S730, S750‧‧‧3D point cloud segmentation method
80‧‧‧特徵 80‧‧‧ characteristics
801‧‧‧待測工件 801‧‧‧Workpiece to be tested
802‧‧‧標準三維點雲 802‧‧‧Standard 3D point cloud
804‧‧‧標準三維點雲 804‧‧‧Standard 3D point cloud
805‧‧‧待測工件的三維點雲 805‧‧‧3D point cloud of the workpiece to be tested
900‧‧‧方形工件 900‧‧‧ square workpiece
902‧‧‧二維影像 902‧‧‧2D image
904‧‧‧圖樣特徵 904‧‧‧ pattern features
906‧‧‧三維點雲 906‧‧‧3D point cloud
908‧‧‧具方向性的三維點雲 908‧‧‧Directional 3D point cloud
910‧‧‧具唯一性的三維點雲 910‧‧‧Unique 3D point cloud
912‧‧‧標準三維點雲 912‧‧‧Standard 3D point cloud
圖1a是根據本發明一實施例所繪示的三維點雲融合二維影像的裝置示意圖。 FIG. 1a is a schematic diagram of an apparatus for three-dimensional point cloud fusion two-dimensional images according to an embodiment of the invention.
圖1b是根據本發明一實施例所繪示的三維點雲融合二維影像的裝置的方塊圖。 FIG. 1b is a block diagram of an apparatus for three-dimensional point cloud fusion two-dimensional images according to an embodiment of the invention.
圖2是根據本發明一實施例所繪示的三維點雲融合二維影像的方法流程圖。 2 is a flow chart of a method for merging a two-dimensional image by a three-dimensional point cloud according to an embodiment of the invention.
圖3是根據本發明一實施例所繪示的圖樣特徵分析方法的流程圖。 FIG. 3 is a flow chart of a method for analyzing a feature of a pattern according to an embodiment of the invention.
圖4a~4b是根據本發明一實施例所繪示的對圖樣特徵進行影像處理的結果示意圖。 4a-4b are schematic diagrams showing the results of image processing on a pattern feature according to an embodiment of the invention.
圖5a~5b是根據本發明一實施例所繪示的影像投影結果示意圖。 5a-5b are schematic diagrams showing image projection results according to an embodiment of the invention.
圖6a~6b是根據本發明一實施例所繪示的藉由影像投影結果分析圖像特徵的灰階分布示意圖。 6a-6b are schematic diagrams showing grayscale distribution of image features by image projection results according to an embodiment of the invention.
圖7是根據本發明一實施例所繪示的三維點雲分割方法的流程圖。 FIG. 7 is a flowchart of a three-dimensional point cloud segmentation method according to an embodiment of the invention.
圖8a~8c是根據本發明一實施例所繪示的姿態估測的示意圖。 8a-8c are schematic diagrams of attitude estimation according to an embodiment of the invention.
圖9a~9d是根據本發明一實施例所繪示的姿態估測流程的示意圖。 9a-9d are schematic diagrams of a gesture estimation process according to an embodiment of the invention.
本發明提供一種三維點雲融合二維影像的方法、裝置與儲存媒體,可解決對稱性工件於姿態估測的問題,並提升姿態估測的正確性。 The invention provides a method, a device and a storage medium for a two-dimensional point cloud fusion two-dimensional image, which can solve the problem of the symmetry workpiece in the attitude estimation and improve the correctness of the attitude estimation.
本發明提出一種三維點雲融合二維影像的方法,藉由二維影像分析出待測工件的方向性,再將其方向性融合至三維點雲中,並對三維點雲進行分割。其分割的目的在於破壞三維點雲的對稱性,以提高姿態估測的正確性。藉由本發明所提出的方法,可以使配合的立體視覺導引機械手臂(Vision Guided Robot,VGR)系統能由姿態估測的結果,精準地計算工件的位置與姿態,進而大幅提昇工件抓取、搬運和組裝之能力。 The invention provides a method for three-dimensional point cloud fusion two-dimensional image, wherein the directionality of the workpiece to be tested is analyzed by the two-dimensional image, and then the directionality is merged into the three-dimensional point cloud, and the three-dimensional point cloud is segmented. The purpose of the segmentation is to destroy the symmetry of the three-dimensional point cloud to improve the correctness of the attitude estimation. By the method proposed by the invention, the Vision Guided Robot (VGR) system can accurately calculate the position and posture of the workpiece from the result of the attitude estimation, thereby greatly improving the workpiece grasping, The ability to handle and assemble.
圖1a是根據本發明一實施例所繪示的三維點雲融合二維影像的裝置示意圖;圖1b是根據本發明一實施例所繪示的三維點雲融合二維影像的裝置的方塊圖。 FIG. 1 is a block diagram of a device for merging a two-dimensional image with a three-dimensional point cloud according to an embodiment of the invention; FIG. 1b is a block diagram of an apparatus for merging a two-dimensional image with a three-dimensional point cloud according to an embodiment of the invention.
請同時參照圖1a與圖1b,在此實施例中,三維點雲融合二維影像的裝置100至少包括移動裝置102與資料處理中心104。必須暸解的是,圖1a與圖1b所示的裝置非用以限制可採用三維 點雲融合二維影像的裝置,在其他實施例中,移動裝置102也可以更改為固定裝置。 Referring to FIG. 1a and FIG. 1b simultaneously, in this embodiment, the apparatus 100 for merging two-dimensional images by a three-dimensional point cloud includes at least a mobile device 102 and a data processing center 104. It must be understood that the device shown in Figures 1a and 1b is not limited to three-dimensional use. The device in which the point cloud fuses the two-dimensional image, in other embodiments, the mobile device 102 can also be changed to a fixed device.
移動裝置102包括一取像裝置106,其中取像裝置106在一個實施例中可以為數位相機108與雷射掃瞄器110,但並非以此為限制。 The mobile device 102 includes an image capture device 106, which in one embodiment may be a digital camera 108 and a laser scanner 110, but is not limited thereto.
數位相機108用以擷取待測工件10的二維影像。然而必須了解的是,本發明亦不限於此,在本發明另一實施例中,數位相機108更可以是電荷耦合元件(Charge Coupled Device,CCD)、互補性氧化金屬半導體(Complementary Metal-Oxide Semiconductor,CMOS)、旋轉變焦式(Pan/Tilt/Zoom,PTZ)攝影機或數位監控器(Digital Video Recorder,DVR)等可獲得二維影像的影像擷取單元。 The digital camera 108 is used to capture a two-dimensional image of the workpiece 10 to be tested. However, it should be understood that the present invention is not limited thereto. In another embodiment of the present invention, the digital camera 108 may be a Charge Coupled Device (CCD) or a Complementary Metal-Oxide Semiconductor (Complementary Metal-Oxide Semiconductor). , CMOS), Rotary Zoom (Pan/Tilt/Zoom, PTZ) camera or Digital Video Recorder (DVR) and other image capture units for 2D images.
雷射掃瞄器110是用以擷取待測工件10的三維點雲。然而必須了解的是,本發明不限於此,在本發明另一實施例中,雷射掃瞄器110更可以是雷射測距儀或類似Microsoft Kinect的體感裝置,其為可獲得三維點雲資料的三維感測元件。 The laser scanner 110 is a three-dimensional point cloud for capturing the workpiece 10 to be tested. However, it should be understood that the present invention is not limited thereto. In another embodiment of the present invention, the laser scanner 110 may further be a laser range finder or a somatosensory device similar to Microsoft Kinect, which is capable of obtaining a three-dimensional point. Three-dimensional sensing elements of cloud data.
移動裝置102用以根據資料處理中心104的姿態估測指令,透過取像裝置106對待測工件10擷取三維點雲與二維影像,並將該些資訊傳送至資料處理中心104以進行本發明的三維點雲融合二維影像的方法,並透過資料處理中心104回傳的姿態估測結果與作動指令,驅動移動裝置102或其他設備對待測工件10進行夾取或其他動作。 The mobile device 102 is configured to capture a three-dimensional point cloud and a two-dimensional image from the object to be tested 10 by the image capturing device 106 according to the posture estimation command of the data processing center 104, and transmit the information to the data processing center 104 to perform the present invention. The method of merging the two-dimensional image by the three-dimensional point cloud, and the posture estimation result and the actuation instruction returned by the data processing center 104, drives the mobile device 102 or other device to perform the clamping or other action of the workpiece 10 to be tested.
資料處理中心104包括處理單元112與儲存媒體114。在本範例實施例中,資料處理中心104更包括通訊單元116與顯示單元118。 The data processing center 104 includes a processing unit 112 and a storage medium 114. In the present exemplary embodiment, the data processing center 104 further includes a communication unit 116 and a display unit 118.
處理單元112可以是一個處理器電路或是任何以硬體型式或韌體型式執行的多個邏輯閘或控制指令,以實現本發明所揭露的三維點雲融合二維影像的方法。在另一實施例中,本發明所提供的三維點雲融合二維影像的方法,亦可以採用軟體模組(未繪示)形式儲存於儲存媒體114中,並可透過處理單元112載入此軟體模組執行後,以完成本發明所揭露的三維點雲融合二維影像的方法。 The processing unit 112 can be a processor circuit or any of a plurality of logic gates or control commands executed in a hard or firmware form to implement the method of the present invention for three-dimensional point cloud fusion of two-dimensional images. In another embodiment, the method for merging the two-dimensional image by the three-dimensional point cloud is also stored in the storage medium 114 in the form of a software module (not shown), and can be loaded through the processing unit 112. After the software module is executed, the method for merging the two-dimensional image by the three-dimensional point cloud disclosed in the present invention is completed.
依據本實施例的三維點雲融合二維影像的方法,可執行於硬體、韌體或安裝在硬體如處理器或微處理器的軟體或電腦程式碼。機器可執行的程式碼儲存於記錄媒介如光碟、隨機存取記憶體、軟碟、硬碟或磁性光學碟,或可於網路下載原先被儲存於遠端記錄媒介,或非暫存式機器可讀取媒介且將儲存於區域的記錄媒介,使得此處所敘述的系統可使用一般用途的電腦,或特殊處理器或可程式化的專用硬體,作為儲存於記錄媒介的軟體,則例如特殊功能積體電路(ASIC)或可程式化邏輯閘陣列(FPGA)。如同本實施例技術領域中的通常知識,當電腦、處理器或硬體實施本發明的處理方法以存取及執行的軟體或電腦程式碼時,電腦、處理器、微處理器控制器或可程式化硬體,包括記憶體元件如隨機存取記憶體(RAM)、唯讀記憶體(ROM)、快閃記憶體、拇指碟等, 皆可儲存或接收軟體或電腦程式碼。 The method for merging a two-dimensional image by a three-dimensional point cloud according to the embodiment may be implemented by hardware, firmware or software or computer code installed on a hardware such as a processor or a microprocessor. The executable code of the machine is stored in a recording medium such as a compact disc, a random access memory, a floppy disk, a hard disk or a magnetic optical disk, or can be downloaded from a network to be stored in a remote recording medium or a non-temporary machine. The medium that can be read and stored in the area, so that the system described herein can use a general-purpose computer, or a special processor or a programmable special hardware, as a software stored in the recording medium, for example, special Functional integrated circuit (ASIC) or programmable logic gate array (FPGA). As is conventional knowledge in the technical field of the present embodiments, when a computer, processor or hardware implements the processing method of the present invention to access and execute software or computer code, the computer, the processor, the microprocessor controller or the like Stylized hardware, including memory components such as random access memory (RAM), read only memory (ROM), flash memory, thumb disc, etc. Both software and computer code can be stored or received.
儲存媒體114用以暫存取像裝置106所取得的三維點雲與二維影像,以提供這些資訊給處理單元112進行本發明實施例的三維點雲融合二維影像的方法。在另一實施例中,儲存媒體114更可以儲存一軟體模組(未繪示),該軟體模組用以當處理單元112載入執行後,完成本發明所揭露的三維點雲融合二維影像的方法。此處的儲存媒體114可以是靜態隨機存取記憶體(Static Random Access Memory,SRAM)、動態隨機存取記憶體(Dynamic Random Access Memory,DRAM)或是任何可用來儲存電子訊號或資料的記憶體或儲存裝置。 The storage medium 114 is configured to temporarily access the three-dimensional point cloud and the two-dimensional image acquired by the image device 106 to provide the information to the processing unit 112 for performing the three-dimensional point cloud fusion two-dimensional image according to the embodiment of the present invention. In another embodiment, the storage medium 114 can further store a software module (not shown) for performing the three-dimensional point cloud fusion two-dimensionally disclosed in the present invention after the processing unit 112 is loaded and executed. The method of imagery. The storage medium 114 herein may be a static random access memory (SRAM), a dynamic random access memory (DRAM), or any memory that can be used to store electronic signals or data. Or storage device.
通訊單元116用以傳送或接收連接移動裝置102與資料處理中心104之間的訊息。例如,資料處理中心104透過通訊單元116接收取像裝置106所取得的三維點雲與二維影像,或資料處理中心104透過通訊單元116發送對移動裝置102作動的控制指令。此通訊單元116可以採用無線方式構成通聯,則移動裝置102與資料處理中心104之間分別配置無線或有線信號收發裝置。而無線方式可以是例如採用近距離的無線通信方式,如IEEE 802.11標準的Wi-Fi、ZigBee、超寬頻(Ultra Wide Band)、近距離無線通訊(NFC)或藍牙(Blue Tooth)等。 The communication unit 116 is configured to transmit or receive a message connecting the mobile device 102 and the data processing center 104. For example, the data processing center 104 receives the three-dimensional point cloud and the two-dimensional image acquired by the image capturing device 106 through the communication unit 116, or the data processing center 104 transmits a control command to the mobile device 102 through the communication unit 116. The communication unit 116 can form a communication in a wireless manner, and a wireless or wired signal transceiver is disposed between the mobile device 102 and the data processing center 104, respectively. The wireless method may be, for example, a short-range wireless communication method such as Wi-Fi, ZigBee, Ultra Wide Band, Near Field Communication (NFC), or Bluetooth (Blue Tooth) of the IEEE 802.11 standard.
顯示單元118用以顯示取像裝置106所取得的三維點雲與二維資訊,並可透過本發明所揭露的三維點雲融合二維影像的方法將結果顯示於其上,以供使用者監看。顯示單元118在一實施例中可以是任何形式的顯示器。 The display unit 118 is configured to display the three-dimensional point cloud and the two-dimensional information acquired by the image capturing device 106, and display the result on the three-dimensional point cloud fusion two-dimensional image disclosed by the present invention for monitoring by the user. Look. Display unit 118 can be any form of display in one embodiment.
圖2是根據本發明一實施例所繪示的三維點雲融合二維影像的方法流程示意圖。 FIG. 2 is a schematic flow chart of a method for three-dimensional point cloud fusion two-dimensional images according to an embodiment of the invention.
請參照圖2,當實施此三維點雲融合二維影像的執行裝置,欲對一具有對稱性的待測工件進行姿態估測時,在步驟S210中,利用例如雷射掃瞄器掃描待測工件以擷取待測工件的三維點雲。接著,在步驟S230中,經由例如處理單元對三維點雲進行教導,所謂教導是藉由預先儲存用於姿態估測的一標準工件模型進行比對。在步驟S250中,利用例如數位相機擷取待測工件的二維影像。並在步驟S270中,處理單元對二維影像進行圖樣特徵分析,以取得待測工件的方向性。在本發明另一實施例中,步驟S270中的圖樣特徵分析亦可對二維影像進行一圖樣比對(Pattern Matching)以取得方向性。最終步驟S290再利用方向性將三維點雲分割或捨去部分點雲資料,以破壞其對稱性而取得正確姿態之唯一性。 Referring to FIG. 2, when performing the three-dimensional point cloud fusion two-dimensional image execution device to perform posture estimation on a symmetrical workpiece to be tested, in step S210, for example, a laser scanner is used to scan the object to be tested. The workpiece captures a three-dimensional point cloud of the workpiece to be tested. Next, in step S230, the three-dimensional point cloud is taught via, for example, a processing unit, the so-called teaching is performed by pre-storing a standard workpiece model for attitude estimation. In step S250, a two-dimensional image of the workpiece to be tested is captured using, for example, a digital camera. And in step S270, the processing unit performs pattern feature analysis on the two-dimensional image to obtain the directivity of the workpiece to be tested. In another embodiment of the present invention, the pattern feature analysis in step S270 may also perform a pattern matching on the two-dimensional image to obtain directivity. In the final step S290, the three-dimensional point cloud is segmented or partially decoupled by directionality to destroy the symmetry and obtain the uniqueness of the correct posture.
圖3是根據本發明一實施例所繪示的圖樣特徵分析方法的流程示意圖。 FIG. 3 is a schematic flow chart of a method for analyzing a feature of a pattern according to an embodiment of the invention.
請參照圖3,在步驟S310中,處理單元112首先對待測工件的二維影像進行定位,並產生二維影像至少一個圖樣特徵。在本實施例中,處理單元對待測工件(例如活塞)的二維影像產生二個圖樣特徵。 Referring to FIG. 3, in step S310, the processing unit 112 first positions the two-dimensional image of the workpiece to be measured, and generates at least one pattern feature of the two-dimensional image. In this embodiment, the processing unit produces two pattern features for the two-dimensional image of the workpiece to be tested (eg, a piston).
在步驟S330中,處理單元舉例對這些圖樣特徵進行二值化(Binarization)處理,但不以此為限。 In step S330, the processing unit performs binarization processing on these pattern features, but is not limited thereto.
在步驟S350中,處理單元對所述二值化處理後的圖樣特徵進行閉合(Close)處理,但不以此為限。 In step S350, the processing unit performs a closing process on the binarized pattern feature, but is not limited thereto.
在步驟S370中,處理單元對所述閉合處理後的圖樣特徵進行反向(Invert)處理,但不以此為限。 In step S370, the processing unit performs an inverse processing on the closed pattern feature, but is not limited thereto.
圖4a~4b是根據本發明一實施例所繪示的對圖樣特徵進行影像處理的結果示意圖,待測工件舉例為活塞。 4a-4b are schematic diagrams showing the result of image processing on a pattern feature according to an embodiment of the invention. The workpiece to be tested is exemplified by a piston.
請再參照圖3,在步驟S390中,處理單元對上述步驟S330~S370影像處理完後的圖樣特徵進行影像投影,以產生分別對應該些影像處理完後的圖樣特徵的多個灰階分布,並藉由一門檻值從該些灰階分布中,決定出對應圖樣特徵中的一目標特徵,並藉由此目標特徵來分辨待測工件的方向性。 Referring to FIG. 3 again, in step S390, the processing unit performs image projection on the image features after the image processing in steps S330-S370 to generate a plurality of grayscale distributions corresponding to the image features after the image processing. And a threshold value is used to determine a target feature in the corresponding pattern feature from the gray scale distribution, and the directivity of the workpiece to be tested is distinguished by the target feature.
請參照圖4a~4b,圖4a~4b為本實施例經由例如上述圖3的影像處理步驟S330~S370所產生的結果示意圖,然而必須要暸解的是,本發明並不限定上述圖3的步驟S330~S370的圖樣特徵影像處理方式,影像處理方式更可以包括或另外採用其他本領域技術人員所習知的影像處理或形態學(Morphology)等技術。 4a-4b, FIG. 4a-4b are schematic diagrams showing the results of the image processing steps S330-S370 of FIG. 3, but it is to be understood that the present invention does not limit the steps of FIG. 3 described above. The image feature processing method of S330~S370 may further include or otherwise adopt other techniques such as image processing or Morphology as known to those skilled in the art.
圖5a~5b是根據本發明一實施例所繪示的影像投影結果示意圖。圖6a~6b是根據本發明一實施例所繪示的藉由影像投影結果,分析圖像特徵的灰階分布示意圖。 5a-5b are schematic diagrams showing image projection results according to an embodiment of the invention. 6a-6b are schematic diagrams showing grayscale distribution of image features by image projection results according to an embodiment of the invention.
請同時參照圖5a~5b、圖6a~6b並搭配參照圖4a~4b,圖5a與圖5b分別為對應圖4a與圖4b進行影像投影所得到的結果,其為分別根據圖4a與圖4b中每一單位橫軸的灰階值所產生,並 藉由圖5a與圖5b的影像投影結果產生對應的灰階分布圖,如圖6a與圖6b所示。 Please refer to FIG. 5a to FIG. 5b and FIG. 6a to FIG. 6b together with reference to FIG. 4a to FIG. 4b. FIG. 5a and FIG. 5b respectively show the results obtained by image projection corresponding to FIG. 4a and FIG. 4b, which are respectively according to FIG. 4a and FIG. 4b. The gray scale value of the horizontal axis of each unit is generated, and A corresponding gray scale distribution map is generated by the image projection results of FIGS. 5a and 5b, as shown in FIGS. 6a and 6b.
在一個實施範例中,處理單元根據上述投影結果,以一預設的門檻值對投影結果進行分析此門檻值的數量,並從所取得的圖樣特徵中決定出一目標特徵,用於分辨待測工件的方向性。 In an embodiment, the processing unit analyzes the projection result by a predetermined threshold value according to the projection result, and determines a target feature from the obtained pattern feature, and is used to distinguish the to-be-tested The directionality of the workpiece.
在本實施例中,圖6b的區域60中有一段連續為零的灰階分布(指標22~32),處理單元即可以此分辨出具對稱性的待測工件的方向性。當處理單元已獲得待測工件的方向性時,即可依照其方向性對三維點雲進行分割,以破壞其對稱性。 In the present embodiment, the region 60 of FIG. 6b has a continuous gray scale distribution (indicators 22 to 32), and the processing unit can distinguish the directionality of the workpiece to be tested with symmetry. When the processing unit has obtained the directivity of the workpiece to be tested, the three-dimensional point cloud can be segmented according to its directivity to destroy its symmetry.
圖7是根據本發明一個實施例所繪示的三維點雲分割方法的流程示意圖。 FIG. 7 is a schematic flow chart of a three-dimensional point cloud segmentation method according to an embodiment of the invention.
請參照圖7,在步驟S710中,處理單元112依照上述二維影像所求得的方向性與三維點雲作匹配,以取得三維點雲的方向性。接著在步驟S730中,由上述藉由灰階分布所求得二維影像中的目標特徵與三維點雲進行融合。 Referring to FIG. 7, in step S710, the processing unit 112 matches the three-dimensional point cloud according to the directionality obtained by the two-dimensional image to obtain the directivity of the three-dimensional point cloud. Next, in step S730, the target features in the two-dimensional image obtained by the gray-scale distribution are merged with the three-dimensional point cloud.
最後在步驟S750中,處理單元112對三維點雲進行分割或忽略或刪除部分點雲,以破壞對應該待測工件的三維影像的對稱性。在本實施例中,三維點雲分割的方法為保留具有目標特徵的區域,而忽略或刪除其他的圖樣特徵,藉以產生具有姿態唯一性的三維點雲。然而必須了解的是,本發明並不限定三維點雲的分割方法,三維點雲的分割方法更可以是其他可破壞三維影像的對稱性的分割方法。 Finally, in step S750, the processing unit 112 segments the three-dimensional point cloud or ignores or deletes part of the point cloud to destroy the symmetry of the three-dimensional image corresponding to the workpiece to be tested. In this embodiment, the method of three-dimensional point cloud segmentation is to retain the region with the target feature, and ignore or delete other pattern features, thereby generating a three-dimensional point cloud with posture uniqueness. However, it must be understood that the present invention does not limit the segmentation method of the three-dimensional point cloud, and the segmentation method of the three-dimensional point cloud may be other segmentation methods that can destroy the symmetry of the three-dimensional image.
以下將針對採用本發明實施例所提出三維點雲融合二維影像的方法,運用在分析待測工件的方向性,並將其方向性融合至三維點雲中,以對三維點雲進行分割的實際範例之中。藉由本發明實施例所提出的方法,可以使待測工件的姿態估測的結果,精準地計算工件的位置與姿態,進而大幅提昇工件抓取、搬運和組裝之能力。 In the following, a method for merging a two-dimensional image by using a three-dimensional point cloud in accordance with an embodiment of the present invention is applied to analyze the directivity of a workpiece to be tested, and directionality is integrated into the three-dimensional point cloud to segment the three-dimensional point cloud. In the actual example. By the method proposed by the embodiment of the invention, the result of the posture estimation of the workpiece to be tested can accurately calculate the position and posture of the workpiece, thereby greatly improving the ability of the workpiece to be grasped, handled and assembled.
在運用本發明實施例所提出三維點雲融合二維影像的方法實施例中,如圖8a~8c,為根據本發明一實施例所繪示的姿態估測的示意圖。 FIG. 8 is a schematic diagram of a posture estimation according to an embodiment of the present invention.
請參照圖8a~8c,在傳統方法中,當欲對待測工件801(舉例為活塞)進行姿態估測時,會使用預先儲存於系統中對應此待測工件801的標準三維點雲802(圖中以線條繪出外型是為便於顯示與說明,實際是由多數掃瞄點所組成)進行姿態估測。然而,由於三維點雲不具有方向性,且待測工件801與標準三維點雲802兩者所對應的特徵80又極為微小,不易分辨,導致在傳統方法中,對於具有對稱性的待測工件801的姿態估測容易發生錯誤,如圖8a所示。而正確的比對結果,應如圖8b所示。 Referring to FIGS. 8a-8c, in the conventional method, when the workpiece 801 (for example, a piston) is to be subjected to posture estimation, a standard three-dimensional point cloud 802 (FIG.) stored in the system corresponding to the workpiece 801 to be tested is used. In the middle, the outline is drawn for the convenience of display and description, and it is actually composed of a plurality of scanning points to perform posture estimation. However, since the three-dimensional point cloud does not have directionality, and the feature 80 corresponding to both the workpiece 801 to be tested and the standard three-dimensional point cloud 802 is extremely small and difficult to distinguish, resulting in the symmetry of the workpiece to be tested in the conventional method. The attitude estimation of 801 is prone to errors, as shown in Figure 8a. The correct comparison result should be as shown in Figure 8b.
然而,透過本發明實施例所提出的三維點雲融合二維影像的方法,如圖8c所示,可對標準三維點雲802進行分割,以產生具有姿態唯一性的標準三維點雲804。待測工件801亦透過本發明實施例的方法產生一具有唯一性的待測工件的三維點雲805,並可藉由標準三維點雲804與待測工件的三維點雲805來進行姿態 估測,以提高姿態估測的正確性。 However, the method for merging the two-dimensional image by the three-dimensional point cloud proposed by the embodiment of the present invention, as shown in FIG. 8c, can segment the standard three-dimensional point cloud 802 to generate a standard three-dimensional point cloud 804 having unique pose. The workpiece 801 to be tested also generates a unique three-dimensional point cloud 805 of the workpiece to be tested through the method of the embodiment of the present invention, and can perform the posture by using the standard three-dimensional point cloud 804 and the three-dimensional point cloud 805 of the workpiece to be tested. Estimate to improve the correctness of the posture estimate.
在運用本發明實施例所提出三維點雲融合二維影像的方法實施例中,圖9a~9d是根據本發明實施例所繪示的姿態估測流程的示意圖。 In the embodiment of the method for transmitting a two-dimensional image by using a three-dimensional point cloud in the embodiment of the present invention, FIGS. 9a-9d are schematic diagrams of a posture estimation process according to an embodiment of the present invention.
請參照圖9a~9d,根據本發明三維點雲融合二維影像的方法,當欲對方形工件900進行姿態估測時,首先會擷取其二維影像902,並擷取二維影像902上的圖樣特徵904(箭頭圖樣)。如圖9a所示。並將圖樣特徵904與方形工件900的三維點雲906作匹配,產生一具方向性的三維點雲908,如圖9b所示。 Referring to FIG. 9a to FIG. 9d, according to the method for merging two-dimensional images by a three-dimensional point cloud according to the present invention, when the posture estimation of the square workpiece 900 is to be performed, the two-dimensional image 902 is first captured and captured on the two-dimensional image 902. Pattern feature 904 (arrow pattern). As shown in Figure 9a. The pattern feature 904 is matched to the three-dimensional point cloud 906 of the square workpiece 900 to produce a directional three-dimensional point cloud 908, as shown in Figure 9b.
接著,對三維點雲908進行分割,產生一具唯一性的三維點雲910,如圖9c所示。 Next, the three-dimensional point cloud 908 is segmented to produce a unique three-dimensional point cloud 910, as shown in Figure 9c.
最後,利用上述分割後的三維點雲910與事先建立好的標準三維點雲912進行姿態估測,以得到正確之工件姿態。 Finally, the above-mentioned divided 3D point cloud 910 and the previously established standard 3D point cloud 912 are used for posture estimation to obtain a correct workpiece posture.
綜上所述,本發明多個範例實施例所提出的三維點雲融合二維影像的方法、裝置與儲存媒體,可以藉由破壞三維點雲的對稱性,進而產生具有唯一性或方向性的三維點雲,以進行後續姿態估測,並可提升姿態估測的正確性,使得立體視覺導引機械手臂(Vision Guided Robot,VGR)系統由能更精準地計算工件的位置與姿態,大幅提昇工件抓取、搬運和組裝之能力。 In summary, the method, apparatus, and storage medium for merging a two-dimensional image by a three-dimensional point cloud according to various exemplary embodiments of the present invention can generate a uniqueness or directionality by destroying the symmetry of the three-dimensional point cloud. The 3D point cloud is used for subsequent attitude estimation, and the correctness of the attitude estimation can be improved, so that the Vision Guided Robot (VGR) system can more accurately calculate the position and posture of the workpiece. The ability to grab, handle and assemble workpieces.
雖然本發明已以實施例揭露如上,然其並非用以限定本發明,任何所屬技術領域中具有通常知識者,在不脫離本發明的精神和範圍內,當可作些許的更動與潤飾,故本發明的保護範圍 當視後附的申請專利範圍所界定者為準。 Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of protection of the present invention It is subject to the definition of the scope of the patent application attached.
S210、S230、S250、S270、S290‧‧‧三維點雲融合二維影像的方法步驟 Method steps for S210, S230, S250, S270, S290‧‧‧3D point cloud fusion 2D image
Claims (19)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW103145219A TWI550425B (en) | 2014-12-24 | 2014-12-24 | Method and apparatus for fusing 2d image and 3d point cloud data and the storage medium thereof |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW103145219A TWI550425B (en) | 2014-12-24 | 2014-12-24 | Method and apparatus for fusing 2d image and 3d point cloud data and the storage medium thereof |
Publications (2)
Publication Number | Publication Date |
---|---|
TW201624326A true TW201624326A (en) | 2016-07-01 |
TWI550425B TWI550425B (en) | 2016-09-21 |
Family
ID=56984747
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
TW103145219A TWI550425B (en) | 2014-12-24 | 2014-12-24 | Method and apparatus for fusing 2d image and 3d point cloud data and the storage medium thereof |
Country Status (1)
Country | Link |
---|---|
TW (1) | TWI550425B (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI607412B (en) * | 2016-09-10 | 2017-12-01 | 財團法人工業技術研究院 | Measurement systems and methods for measuring multi-dimensions |
TWI743645B (en) * | 2019-07-29 | 2021-10-21 | 大陸商浙江商湯科技開發有限公司 | Information processing method and device, positioning method and device, electronic equipment and computer readable storage medium |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5665290B2 (en) * | 2009-08-24 | 2015-02-04 | 富士フイルム株式会社 | Deposition equipment |
TWI488069B (en) * | 2012-10-08 | 2015-06-11 | Qisda Corp | Optical painting method |
CN103325143B (en) * | 2013-06-13 | 2016-10-05 | 华南理工大学 | Labelling point automatic registration method based on Model Matching |
-
2014
- 2014-12-24 TW TW103145219A patent/TWI550425B/en active
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI607412B (en) * | 2016-09-10 | 2017-12-01 | 財團法人工業技術研究院 | Measurement systems and methods for measuring multi-dimensions |
US10008005B2 (en) | 2016-09-10 | 2018-06-26 | Industrial Technology Research Institute | Measurement system and method for measuring multi-dimensions |
TWI743645B (en) * | 2019-07-29 | 2021-10-21 | 大陸商浙江商湯科技開發有限公司 | Information processing method and device, positioning method and device, electronic equipment and computer readable storage medium |
US11983820B2 (en) | 2019-07-29 | 2024-05-14 | Zhejiang Sensetime Technology Development Co., Ltd | Information processing method and device, positioning method and device, electronic device and storage medium |
Also Published As
Publication number | Publication date |
---|---|
TWI550425B (en) | 2016-09-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11724400B2 (en) | Information processing apparatus for determining interference between object and grasping unit, information processing method, and storage medium | |
JP5612916B2 (en) | Position / orientation measuring apparatus, processing method thereof, program, robot system | |
US10288418B2 (en) | Information processing apparatus, information processing method, and storage medium | |
TWI566204B (en) | Three dimensional object recognition | |
JP6573354B2 (en) | Image processing apparatus, image processing method, and program | |
US9621793B2 (en) | Information processing apparatus, method therefor, and measurement apparatus | |
US10083512B2 (en) | Information processing apparatus, information processing method, position and orientation estimation apparatus, and robot system | |
CN107860311B (en) | Method of operating a triangulation laser scanner to identify surface characteristics of a workpiece | |
US9710960B2 (en) | Closed-form 3D model generation of non-rigid complex objects from incomplete and noisy scans | |
JP6782046B1 (en) | Object detection system and method based on image data | |
JP2015147256A (en) | Robot, robot system, control device, and control method | |
KR102595787B1 (en) | Electronic device and control method thereof | |
US20170280130A1 (en) | 2d video analysis for 3d modeling | |
TWI550425B (en) | Method and apparatus for fusing 2d image and 3d point cloud data and the storage medium thereof | |
JP6683088B2 (en) | Surface texture inspection device, surface texture inspection method and program | |
US20190313082A1 (en) | Apparatus and method for measuring position of stereo camera | |
Ali et al. | Camera based precision measurement in improving measurement accuracy | |
JP2018063675A (en) | Image processor and control method | |
JP6840589B2 (en) | Information processing equipment and programs | |
JP7424800B2 (en) | Control device, control method, and control system | |
JP2016053481A (en) | Information processing device, information processing system, method for controlling information processing device and program | |
US20220270282A1 (en) | Information processing device, data generation method, and non-transitory computer-readable medium storing program | |
JP2013250604A (en) | Object detecting device and object detecting method | |
JP4351090B2 (en) | Image processing apparatus and image processing method | |
CN111742349A (en) | Information processing apparatus, information processing method, and program |