TWI817847B - Method, computer program and computer readable medium for fast tracking and positioning objects in augmented reality and mixed reality - Google Patents
Method, computer program and computer readable medium for fast tracking and positioning objects in augmented reality and mixed reality Download PDFInfo
- Publication number
- TWI817847B TWI817847B TW111145521A TW111145521A TWI817847B TW I817847 B TWI817847 B TW I817847B TW 111145521 A TW111145521 A TW 111145521A TW 111145521 A TW111145521 A TW 111145521A TW I817847 B TWI817847 B TW I817847B
- Authority
- TW
- Taiwan
- Prior art keywords
- point cloud
- cloud data
- surface point
- augmented reality
- virtual
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 39
- 238000004590 computer program Methods 0.000 title claims abstract description 9
- 239000011159 matrix material Substances 0.000 claims description 23
- 238000012545 processing Methods 0.000 claims description 13
- 238000013519 translation Methods 0.000 claims description 7
- 238000000354 decomposition reaction Methods 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 abstract description 22
- 238000012549 training Methods 0.000 abstract description 8
- 210000004185 liver Anatomy 0.000 description 96
- 230000000875 corresponding effect Effects 0.000 description 21
- 238000010586 diagram Methods 0.000 description 20
- 230000002792 vascular Effects 0.000 description 17
- 239000011521 glass Substances 0.000 description 14
- 238000004364 calculation method Methods 0.000 description 11
- 210000000056 organ Anatomy 0.000 description 10
- 238000001356 surgical procedure Methods 0.000 description 10
- 210000004204 blood vessel Anatomy 0.000 description 5
- 238000005520 cutting process Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 239000003550 marker Substances 0.000 description 4
- 238000002591 computed tomography Methods 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 229920001296 polysiloxane Polymers 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 239000013013 elastic material Substances 0.000 description 2
- 238000012274 Preoperative evaluation Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000004870 electrical engineering Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000000465 moulding Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Landscapes
- Radar Systems Or Details Thereof (AREA)
Abstract
Description
本發明係一種擴增實境與混合實境下物件快速追蹤定位之方法、電腦程式及電腦可讀取媒體，特別是指取得真實物件之虛擬物件於多個視角下的第一表面點雲數據，當攝影機自一觀察角度取得真實物件之第二表面點雲數據時，經比對後根據視角相符或相近之第一表面點雲數據確認真實物件之觀察角度，並將虛擬物件疊合於真實物件的發明。 The present invention is a method, computer program and computer-readable media for fast tracking and positioning of objects in augmented reality and mixed reality. In particular, it refers to obtaining first surface point cloud data of virtual objects of real objects at multiple perspectives. , when the camera obtains the second surface point cloud data of the real object from an observation angle, after comparison, the observation angle of the real object is confirmed based on the first surface point cloud data that matches or is similar in angle of view, and the virtual object is superimposed on the real object. The invention of objects.
虛擬實境(VR)、擴增實境(AR)、混合實境(MR)等技術皆涉及了虛擬物件的應用。 Technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) all involve the application of virtual objects.
以混合實境(Mixed Reality,MR)為例，混合實境(Mixed Reality,MR)是整合現實世界與虛擬世界所建立出的實境環境，在混合實境中，現實世界中的物件能夠與虛擬世界中的物件產生互動，混合實境合成了虛擬實境與擴增實境。 Take Mixed Reality (MR) as an example. Mixed Reality (MR) is a real environment created by integrating the real world and the virtual world. In Mixed Reality, objects in the real world can interact with Objects in the virtual world interact, and mixed reality combines virtual reality and augmented reality.
混合實境應用於外科手術例如有台灣專利第I741536號「基於混合實境的手術導航影像成像方法」，該案係將患者的手術部位三維影像成像至手術部位，使手術部位與手術部位三維影像疊合，並根據所定義的坐標系統及 混合實境眼鏡的紅外線拍攝追蹤裝置追蹤手術部位，透過座標間的轉換來達到手術部位與手術部位三維影像定位追蹤的功效。 Mixed reality is used in surgical operations. For example, there is Taiwan Patent No. I741536 "Surgical Navigation Image Imaging Method Based on Mixed Reality". This case is to image the patient's three-dimensional image of the surgical site to the surgical site, so that the surgical site and the surgical site can be three-dimensionally imaged. superimposed, and based on the defined coordinate system and The infrared shooting and tracking device of the mixed reality glasses tracks the surgical site, and achieves the effect of positioning and tracking the surgical site and the three-dimensional image of the surgical site through conversion between coordinates.
2018年長庚大學電機系與林口長庚紀念醫院神經外科合作，進行了一項關於MR HMD的研究，以改善手術中MR影像與器官疊合的精度。他們的研究將MR HMD的深度攝影機與點雲庫(PCL)相結合來構建頭部表面點雲數據，並使用所提出的改進對齊算法與HoloLens將術前重建的MR影像疊加在頭部體模上以輔助手術。由於點雲數量龐大影響計算速度，該研究在頭部體模附近放置了一個特徵標記板用於定位方向性，當系統檢測到特徵標記板時，進行點雲數據的對齊，加快定位速度。 In 2018, the Department of Electrical Engineering of Chang Gung Memorial University collaborated with the Neurosurgery Department of Linkou Chang Gung Memorial Hospital to conduct a study on MR HMD to improve the accuracy of superposition of MR images and organs during surgery. Their study combined the depth camera of the MR HMD with the Point Cloud Library (PCL) to construct head surface point cloud data, and used the proposed improved alignment algorithm with HoloLens to overlay the preoperatively reconstructed MR images on the head phantom. to assist in surgery. Since the large number of point clouds affects the calculation speed, this study placed a feature marker plate near the head phantom for positioning directionality. When the system detects the feature marker plate, it aligns the point cloud data to speed up positioning.
但是，使用特徵標記板追踪定位方法的缺點是當特徵圖像(該研究使用QR CODE)被覆蓋或角度放置發生偏移時，系統會導致誤判，且若使用於實際手術輔助，真實器官也無法使用特徵標記板作為輔助定位。 However, the disadvantage of using the feature marker board tracking and positioning method is that when the feature image (this study used QR CODE) is covered or the angle placement is offset, the system will cause misjudgment, and if used to assist actual surgery, the real organ cannot be Use the feature marker board as an aid in positioning.
因此，本發明提出一種擴增實境與混合實境下物件快速追蹤定位之方法，包括：建立一真實物件的一虛擬物件，該虛擬物件對應於該真實物件；自上述虛擬物件取得該虛擬物件於至少二視角的複數第一表面點雲數據，每一所述視角各對應一個第一表面點雲數據；利用一擴增實境單元從一觀看角度觀看該真實物件，該擴增實境單元上的一攝影機從該觀看角度拍攝該真實物件，取得該真實物件的一第二表面點雲數據；一處理單元將該第二表面點雲數據與所述複數第一表面點雲數據進行比對，選出一最相近第一表面點雲數據及該最相近第一表面點雲數據所對應的該視角；該處理單元根據 上步驟的該視角，確認該擴增實境單元的該觀看角度，該處理單元將該虛擬物件旋轉該視角以疊合於該真實物件，共同顯示於該擴增實境單元中。 Therefore, the present invention proposes a method for fast tracking and positioning of objects in augmented reality and mixed reality, including: creating a virtual object of a real object, the virtual object corresponding to the real object; and obtaining the virtual object from the above virtual object A plurality of first surface point cloud data from at least two viewing angles, each of the viewing angles corresponding to one first surface point cloud data; using an augmented reality unit to view the real object from a viewing angle, the augmented reality unit A camera on the camera captures the real object from the viewing angle and obtains a second surface point cloud data of the real object; a processing unit compares the second surface point cloud data with the plurality of first surface point cloud data. , select the closest first surface point cloud data and the viewing angle corresponding to the closest first surface point cloud data; the processing unit is based on The viewing angle in the above step confirms the viewing angle of the augmented reality unit, and the processing unit rotates the virtual object through the viewing angle to overlap with the real object and display them together in the augmented reality unit.
進一步，該處理單元根據該最相近第一表面點雲數據所對應的該視角而旋轉或/及位移該虛擬物件，使該虛擬物件的顯示角度與該真實物件之觀看角度相符，之後以RMSE計算誤差而微調該虛擬物件，使該虛擬物件正確疊合於該真實物件。 Further, the processing unit rotates or/and displaces the virtual object according to the viewing angle corresponding to the closest first surface point cloud data, so that the display angle of the virtual object matches the viewing angle of the real object, and then calculates the RMSE The virtual object is fine-tuned according to the error so that the virtual object correctly overlaps the real object.
進一步，係透過一虛擬攝影機從所述視角拍攝該虛擬物件，該虛擬物件設有一基準線，該虛擬攝影機之一視平面法線和上述基準線之間的角度定義為該視角。 Further, the virtual object is photographed from the perspective through a virtual camera. The virtual object is provided with a reference line, and the angle between the normal line of a viewing plane of the virtual camera and the reference line is defined as the perspective.
進一步，所述第一表面點雲數據係該虛擬攝影機在所述視角下，獲取該虛擬物件之一被拍攝表面的複數點雲而建構。或者，所述第一表面點雲數據係先建立該虛擬物件之一全表面點雲數據，再由所述視角擷取該全表面點雲數據的局部點雲而建構。 Further, the first surface point cloud data is constructed by acquiring a plurality of point clouds of a photographed surface of the virtual object from the virtual camera at the viewing angle. Alternatively, the first surface point cloud data is constructed by first establishing full surface point cloud data of the virtual object, and then capturing partial point clouds of the full surface point cloud data from the viewing angle.
進一步，該攝影機係為一深度攝影機，該深度攝影機拍攝該真實物件後，使用統計過濾器(statistical filter)濾除背景噪聲而產生該第二表面點雲數據。 Further, the camera is a depth camera. After taking the real object, the depth camera uses a statistical filter to filter out background noise to generate the second surface point cloud data.
進一步，該攝影機拍攝該真實物件後，係根據所拍攝影像的色階變化，獲得該第二表面點雲數據。 Further, after the camera captures the real object, the second surface point cloud data is obtained based on the color level changes of the captured image.
進一步，在該第二表面點雲數據與所述複數第一表面點雲數據進行比對的過程中，該處理單元進一步對所述第一表面點雲數據進行縮放，以使該攝影機從不同距離拍攝該真實物件時，均能選出該最相近第一表面點雲數據。可使用k-d樹(k-d tree)進行優化來搜尋第二表面點雲數據與第一表面點雲數
據之間的對應點。更進一步，透過一最小平方法獲得該虛擬物件與該真實物件的相對方位以確認該真實物件的視角，所述最小平方法係：E(R,t)=
本發明也是一種電腦程式，係安裝於一電腦後可執行前述擴增實境與混合實境下物件快速追蹤定位之方法。 The present invention is also a computer program that, after being installed on a computer, can execute the aforementioned method for fast tracking and positioning of objects in augmented reality and mixed reality.
本發明也是一種電腦可讀取媒體，係儲存有前述電腦程式。 The present invention is also a computer-readable medium that stores the aforementioned computer program.
根據上述技術特徵可達成以下功效： According to the above technical characteristics, the following effects can be achieved:
1.本發明建構虛擬物件於各視角的第一表面點雲數據，每一第一表面點雲數據都包含了三維的縱深資訊，再以擴增實境單元的攝影機從一觀看角度拍攝真實物件，獲得該觀看角度下的真實物件的第二表面點雲數據，第二表面點雲數據也包含了三維的縱深資訊，藉由將第二表面點雲數據比對於前述各視角的第一表面點雲數據，選出一最相近第一表面點雲數據，並得知該最相近第一表面點雲數據所對應的視角，藉此將該虛擬物件旋轉正確角度以快速地疊合於該真實物件，共同顯示於該擴增實境單元中，同時隨著攝影機變化不同的觀看角度，虛擬物件也能即時地變化視角而正確疊合於真實物件。 1. The present invention constructs the first surface point cloud data of the virtual object at each viewing angle. Each first surface point cloud data contains three-dimensional depth information, and then uses the camera of the augmented reality unit to capture the real object from a viewing angle. , obtain the second surface point cloud data of the real object at the viewing angle. The second surface point cloud data also contains three-dimensional depth information. By comparing the second surface point cloud data with the first surface points of each of the aforementioned viewing angles Cloud data, select the most similar first surface point cloud data, and learn the angle of view corresponding to the most similar first surface point cloud data, thereby rotating the virtual object at the correct angle to quickly overlap with the real object, They are jointly displayed in the augmented reality unit. At the same time, as the camera changes different viewing angles, the virtual objects can also change the viewing angle in real time and correctly overlap with the real objects.
2.根據上述的最相近第一表面點雲數據及其所對應的視角，確認真實物件的觀看角度，將虛擬物件與真實物件進行初步定位，之後再以RMSE計算誤差並微調，將最相近第一表面點雲數據與第二點雲數據點雲數據對齊，完成細部定位，使虛擬物件可以與真實物件正確疊合。例如真實物件為器官假體的實物，使用混合實境眼鏡觀看該器官假體，結合虛擬物件(即虛擬器官影像)進行手術訓練時，在擴增實境的環境下，虛擬物件可以精確疊合在器官假體上，不須使用外部特徵標記板例如二維條碼板，也能夠快速定位虛擬物件的方位。 2. Based on the above-mentioned closest first surface point cloud data and its corresponding viewing angle, confirm the viewing angle of the real object, initially position the virtual object and the real object, and then calculate the error using RMSE and fine-tune the closest third surface point cloud data. The first surface point cloud data is aligned with the second point cloud data to complete detailed positioning so that virtual objects can be correctly superimposed with real objects. For example, the real object is an organ prosthesis, and mixed reality glasses are used to view the organ prosthesis. When combined with virtual objects (i.e., virtual organ images) for surgical training, the virtual objects can be accurately superimposed in an augmented reality environment. On organ prostheses, there is no need to use external feature marking boards such as two-dimensional barcode boards, and the position of virtual objects can be quickly located.
3.本發明的第一表面點雲數據可以縮放，當攝影機從不同距離拍攝該真實物件時，都能經由比對而選出該最相近第一表面點雲數據。 3. The first surface point cloud data of the present invention can be scaled. When the camera captures the real object from different distances, the closest first surface point cloud data can be selected through comparison.
4.本發明之第二表面點雲數據與第一表面點雲數據於搜尋對應點時使用k-d樹(k-d tree)進行優化，透過k-d樹(k-d tree)的使用空間劃分來減少每次計算的搜尋區域，可減少整體計算負擔和時間。 4. The second surface point cloud data and the first surface point cloud data of the present invention are optimized using a k-d tree when searching for corresponding points, and the space division of the k-d tree is used to reduce the time required for each calculation. search area, which reduces the overall computational burden and time.
5.在一加工程序或移位程序中，真實物件可以是待取拿的工件，當以機械手臂在生產線或工作站取拿工件時，取拿前的工件擺放方位即使不一致，藉由攝影機拍攝該真實物件，再透過本發明的快速追蹤定位方法，電腦能快速辨識該工件的實際方位，然後由電腦控制機械手臂旋轉相應的角度，使機械手臂每次都能從正確位置夾取該工件。對於生產線而言，即使工件以不同的方位例如歪斜、倒立、反置等被置放在生產線上，本發明也能對工件執行快速追蹤定位，使生產線上的機械手臂每次都能旋轉相應角度以正確夾取工件。 5. In a processing program or shifting program, the real object can be the workpiece to be picked up. When a robot arm is used to pick up the workpiece on the production line or workstation, even if the position of the workpiece before picking up is inconsistent, it can be captured by the camera. For this real object, through the fast tracking positioning method of the present invention, the computer can quickly identify the actual position of the workpiece, and then the computer controls the robot arm to rotate the corresponding angle, so that the robot arm can pick up the workpiece from the correct position every time. For the production line, even if the workpiece is placed on the production line in different orientations, such as skewed, inverted, reversed, etc., the present invention can also perform fast tracking and positioning of the workpiece, so that the robotic arm on the production line can rotate the corresponding angle every time to correctly grip the workpiece.
6.利用本發明對真實物件的快速追蹤定位方法，只要透過攝影機拍攝不同的真實物件例如不同的工件，取得各種工件的第二表面點雲數據，並 於電腦中預先儲存的各種形狀的虛擬物件(即虛擬工件)，每一虛擬物件都有眾多不同視角的第一表面點雲數據，藉此可以快速辨識工件的形狀，輔助進行工件的檢選分類，或是用於工件瑕疵的快速檢測。 6. Using the fast tracking and positioning method of real objects of the present invention, as long as different real objects such as different workpieces are photographed through the camera, the second surface point cloud data of various workpieces are obtained, and Virtual objects of various shapes (i.e. virtual workpieces) are pre-stored in the computer. Each virtual object has many first surface point cloud data from different perspectives. This can quickly identify the shape of the workpiece and assist in the selection and classification of workpieces. , or for rapid detection of workpiece defects.
1:肝臟假體 1: Liver prosthesis
10:掃描影像 10:Scan image
11:血管組織 11:Vascular tissue
2:虛擬物件 2:Virtual objects
20:全表面點雲數據 20: Full surface point cloud data
21:虛擬血管 21:Virtual blood vessel
3:虛擬攝影機 3:Virtual camera
4:混合實境單元 4: Mixed reality unit
41:容納凹槽 41:accommodating groove
42:平台 42:Platform
5:第一表面點雲數據 5: First surface point cloud data
6:第二表面點雲數據 6: Second surface point cloud data
7:噪聲 7: Noise
81,81A:生產線 81,81A:Production line
82,82A,82B:工件 82,82A,82B: workpiece
83,83A:檢測電腦 83,83A: Detection of computers
84,84A:攝影機 84,84A:Camera
85,85A:機械手臂 85,85A: Robot arm
Npi:基準線 Npi : baseline
Npc:視平面法線 N pc : normal to the viewing plane
Ng:觀察基準線 N g : observation baseline
Nw:觀察視線 N w :observation line of sight
θ:法向角度 θ : normal angle
θ 1:觀察角度 θ 1 :observation angle
[第一圖]係為本發明擴增實境與混合實境下物件快速追蹤定位之方法的流程圖。 [The first picture] is a flow chart of the method of fast tracking and positioning of objects in augmented reality and mixed reality according to the present invention.
[第二圖]係為本發明實施例中，根據電腦斷層掃瞄影像，製作肝真實物件(肝臟假體)與建立虛擬物件(肝臟三維影像)的示意圖。 [The second figure] is a schematic diagram of creating a real liver object (liver prosthesis) and creating a virtual object (liver three-dimensional image) based on computed tomography images in an embodiment of the present invention.
[第三圖]係為本發明實施例中，虛擬攝影機的視平面法線與虛擬物件的基準線之間的視角示意圖。 [The third figure] is a schematic diagram of the angle of view between the normal line of the visual plane of the virtual camera and the reference line of the virtual object in the embodiment of the present invention.
[第三A圖]係為本發明實施例中，利用虛擬攝影機自不同視角朝向虛擬物件，獲取多個第一表面點雲數據的示意圖。 [Figure A] is a schematic diagram of using a virtual camera to face a virtual object from different viewing angles to obtain multiple first surface point cloud data in an embodiment of the present invention.
[第四A圖]係為本發明實施例中，利用虛擬攝影機自虛擬物件的全表面點雲數據取得不同視角下的第一表面點雲數據的示意圖之一。 [Figure 4A] is one of the schematic diagrams of using a virtual camera to obtain first surface point cloud data from different viewing angles from the full surface point cloud data of a virtual object in an embodiment of the present invention.
[第四B圖]係為本發明實施例中，利用虛擬攝影機自虛擬物件的全表面點雲數據取得不同視角下的第一表面點雲數據的示意圖之二。 [Figure 4B] is the second schematic diagram of using a virtual camera to obtain first surface point cloud data from different viewing angles from the full surface point cloud data of a virtual object in an embodiment of the present invention.
[第四C圖]係為本發明實施例中，利用虛擬攝影機自虛擬物件的全表面點雲數據取得不同視角下的第一表面點雲數據的示意圖之三。 [Figure 4C] is the third schematic diagram of using a virtual camera to obtain first surface point cloud data from different viewing angles from the full surface point cloud data of a virtual object in an embodiment of the present invention.
[第四D圖]係為本發明實施例中，利用虛擬攝影機自虛擬物件的全表面點雲數據取得不同視角下的第一表面點雲數據的示意圖之四。 [Figure 4D] is the fourth schematic diagram of using a virtual camera to obtain first surface point cloud data from different viewing angles from the full surface point cloud data of a virtual object in an embodiment of the present invention.
[第四E圖]係為本發明實施例中，利用虛擬攝影機自虛擬物件的全表面點雲數據取得不同視角下的第一表面點雲數據的示意圖之五。 [Figure 4E] is the fifth schematic diagram of using a virtual camera to obtain first surface point cloud data from different viewing angles from the full surface point cloud data of a virtual object in an embodiment of the present invention.
[第五圖]係為本發明實施例中，利用混合實境單元之深度攝影機自一觀察角度取得肝臟假體的第二表面點雲數據的示意圖。 [The fifth figure] is a schematic diagram of using the depth camera of the mixed reality unit to obtain the second surface point cloud data of the liver prosthesis from an observation angle in an embodiment of the present invention.
[第五A圖]係為本發明實施例中，利用混合實境單元之深度攝影機自一觀察角度觀察肝臟假體的示意圖。 [Figure 5A] is a schematic diagram of using the depth camera of the mixed reality unit to observe the liver prosthesis from an observation angle in an embodiment of the present invention.
[第六圖]係為本發明實施例中，根據比對相符的第一表面點雲數據使虛擬物件之視角與肝臟假體之視角相符的示意圖之一。 [Figure 6] is one of the schematic diagrams of making the perspective of the virtual object consistent with the perspective of the liver prosthesis based on the matching first surface point cloud data in an embodiment of the present invention.
[第七圖]係為本發明實施例中，根據比對相符的第一表面點雲數據使虛擬物件之視角與肝臟假體之視角相符的示意圖之二。 [Figure 7] is the second schematic diagram of making the perspective of the virtual object consistent with the perspective of the liver prosthesis based on the matching first surface point cloud data in an embodiment of the present invention.
[第八圖]係為本發明實施例中，將視角相符的虛擬物件疊合於肝臟假體的示意圖。 [Figure 8] is a schematic diagram of superimposing a virtual object with matching viewing angles on a liver prosthesis in an embodiment of the present invention.
[第九圖]係為本發明實施例中，將虛擬物件及肝臟假體之點雲數據進行迭代運算時，不同迭代次數與所需計算時間的曲線圖。 [Figure 9] is a graph showing the number of iterations and the required calculation time when iteratively calculating the point cloud data of virtual objects and liver prostheses in an embodiment of the present invention.
[第十圖]係為本發明實施例中，在0度視角下，將虛擬物件及肝臟假體之點雲數據進行迭代運算時，不同迭代次數與RMSE誤差的曲線圖。 [Figure 10] is a graph showing different iteration times and RMSE errors when iteratively calculating the point cloud data of virtual objects and liver prostheses at a 0-degree viewing angle in an embodiment of the present invention.
[第十一圖]係為本發明實施例中，將虛擬物件及肝臟假體之點雲數據進行迭代運算時，不同視角下的RMSE誤差的示意圖。 [Figure 11] is a schematic diagram of the RMSE error under different viewing angles when iteratively calculating point cloud data of virtual objects and liver prostheses in an embodiment of the present invention.
[第十二A圖]係為本發明實施例中，真實物件以不同方位置放於生產線，機械手臂正確夾取真實物件的示意圖之一。 [Figure 12A] is one of the schematic diagrams of the real object being placed on the production line in different positions and the robot arm correctly grasping the real object in an embodiment of the present invention.
[第十二B圖]係為本發明實施例中，真實物件以不同方位置放於生產線，機械手臂正確夾取真實物件的示意圖之二。 [Figure 12B] is the second schematic diagram of the real object being placed on the production line in different positions and the robot arm correctly grasping the real object in an embodiment of the present invention.
[第十二C圖]係為本發明實施例中，真實物件以不同方位置放於生產線，機械手臂正確夾取真實物件的示意圖之三。 [Figure 12C] is the third schematic diagram of the real object being placed on the production line in different positions and the robot arm correctly grasping the real object in the embodiment of the present invention.
[第十三A圖]係為本發明實施例中，不同形狀的真實物件置放於生產線，機械手臂正確夾取真實物件的示意圖之一。 [Figure 13A] is one of the schematic diagrams of real objects of different shapes placed on the production line and the robot arm correctly gripping the real objects in an embodiment of the present invention.
[第十三B圖]係為本發明實施例中，不同形狀的真實物件置放於生產線，機械手臂正確夾取真實物件的示意圖之二。 [Figure 13B] is the second schematic diagram showing real objects of different shapes being placed on the production line and the robot arm correctly grasping the real objects in an embodiment of the present invention.
[第十三C圖]係為本發明實施例中，不同形狀的真實物件置放於生產線，機械手臂正確夾取真實物件的示意圖之三。 [Figure 13C] is the third schematic diagram showing real objects of different shapes being placed on the production line and the robot arm correctly grasping the real objects in an embodiment of the present invention.
下列所述的實施例，只是輔助說明本發明擴增實境與混合實境下物件快速追蹤定位之方法、電腦程式及電腦可讀取媒體，並非用以限制本發明。其中實施例之真實物件以器官假體為例且使用於混合實境手術訓練，或者真實物件以工件為例，應用於生產線上對工件的快速檢測。由於混合實境涉及擴增實境技術，本發明除了應用於擴增實境，也適用於混合實境。 The following embodiments are only used to assist in illustrating the method, computer program and computer-readable media for fast tracking and positioning of objects in augmented reality and mixed reality according to the present invention, and are not intended to limit the present invention. In the embodiment, the real object is an organ prosthesis as an example and is used for mixed reality surgical training, or the real object is a workpiece as an example and is used for rapid inspection of the workpiece on a production line. Since mixed reality involves augmented reality technology, the present invention is not only applicable to augmented reality but also to mixed reality.
參閱第一圖及第二圖所示，首先建立一器官假體，本實施例該器官假體是以肝臟假體1為例的一虛擬物件2，其中該肝臟假體1內包含了一血管組織11，該虛擬物件2則對應該肝臟假體1，虛擬物件2也包含該了虛擬血管21。在本實施例中，上述肝臟假體1係根據電腦斷層對人體肝臟真實的掃描影像10，將肝臟部位相關的電腦斷層的複數掃描影像10經重建後獲得一肝臟三維影像，該肝臟三維影像即是前述的虛擬物件2，該肝臟三維影像也包含肝臟內部的血管。另外根據該肝臟三維影像，透過脫蠟成型的手段，先以彈性材質例如矽膠製造中空且有彈性的血管組織11，再製作一模具，模具設有對應肝臟形狀的模穴，然後將上述血管組織11定位於模具的模穴中，並以彈性材質例如矽膠注入模穴，製作包覆著血管組織11的肝臟假體1，該肝臟假體1與 血管組織11均具有彈性，擬真度極高。由於肝臟假體1與血管組織11均根據實際的掃描影像10所建立，同時使用矽膠作為製作肝臟假體1的材質，因此肝臟假體1的形狀、尺寸與彈性等均與真實肝臟幾乎相同，且血管組織11的形狀、尺寸、分佈位置與彈性等，也均與真實肝臟內部的血管相同，適合作為手術訓練、術前評估等用途。本實施例中的肝臟假體1進一步添加有染劑，以模擬肝臟的真實顏色，因此肝臟假體1為不透明形態，無法目視被包覆其中的血管組織11。執行肝臟假體手術訓練時，上述的肝臟假體1有助於練習者在訓練過程中學習掌握血管組織11的位置，避免在切劃肝臟假體1時卻意外劃開非必要的肝臟內部的血管組織11。前述的虛擬物件2，也是根據前述複數掃描影像10所建立的一個三維虛擬肝臟影像，該三維虛擬肝臟影像也包含了肝臟血管，所以虛擬物件2完全對應該肝臟假體1。從該虛擬物件2，電腦可以進一步建構該虛擬物件2的全表面點雲數據20。 Referring to the first and second figures, an organ prosthesis is first created. In this embodiment, the organ prosthesis is a virtual object 2 taking a liver prosthesis 1 as an example. The liver prosthesis 1 includes a blood vessel. Tissue 11, the virtual object 2 corresponds to the liver prosthesis 1, and the virtual object 2 also includes the virtual blood vessel 21. In this embodiment, the above-mentioned liver prosthesis 1 is based on a real computer tomography scan image 10 of the human liver, and a three-dimensional liver image is obtained after reconstructing the multiple computer tomography scan images 10 related to the liver part. The three-dimensional liver image is It is the aforementioned virtual object 2. The three-dimensional image of the liver also includes blood vessels inside the liver. In addition, according to the three-dimensional image of the liver, hollow and elastic vascular tissue 11 is first made of elastic material such as silicone through dewaxing molding, and then a mold is made. The mold is equipped with a mold cavity corresponding to the shape of the liver, and then the above-mentioned vascular tissue is 11 is positioned in the mold cavity of the mold, and an elastic material such as silicone is injected into the mold cavity to produce a liver prosthesis 1 covered with vascular tissue 11. The liver prosthesis 1 and The vascular tissue 11 is elastic and highly realistic. Since the liver prosthesis 1 and vascular tissue 11 are both established based on the actual scanned images 10, and silicone is used as the material for making the liver prosthesis 1, the shape, size, elasticity, etc. of the liver prosthesis 1 are almost the same as the real liver. Moreover, the shape, size, distribution position, elasticity, etc. of the vascular tissue 11 are also the same as those of the blood vessels inside the real liver, and are suitable for surgical training, preoperative evaluation, and other purposes. The liver prosthesis 1 in this embodiment is further added with a dye to simulate the true color of the liver. Therefore, the liver prosthesis 1 is in an opaque form and the vascular tissue 11 covered therein cannot be visually viewed. When performing liver prosthesis surgery training, the above-mentioned liver prosthesis 1 helps practitioners learn to grasp the position of the vascular tissue 11 during the training process, and avoid accidentally cutting unnecessary internal parts of the liver when cutting the liver prosthesis 1. Vascular tissue11. The aforementioned virtual object 2 is also a three-dimensional virtual liver image created based on the aforementioned plurality of scanned images 10. The three-dimensional virtual liver image also includes liver blood vessels, so the virtual object 2 completely corresponds to the liver prosthesis 1. From the virtual object 2 , the computer can further construct the full surface point cloud data 20 of the virtual object 2 .
參閱第一圖、第三圖及第三A圖所示，利用電腦影處理軟體中的一虛擬攝影機3取得上述虛擬物件2至少二視角的第一表面點雲數據5。在第三A圖中，該虛擬物件2位於一直角坐標系統中，較佳是直角坐標系統的Z軸對應人體軀幹的身高方向，Z軸的箭頭指向人體頭部，虛擬物件2的方位對應了人體站立時體內肝臟的方位，X軸代表了垂直於人體胸部的指向，所以人體仰臥接受開刀時，X軸將是從仰臥人體的正上方觀看肝臟的方位。以前述X軸為一基準線Npi，從X軸觀看該虛擬物件2設定為是前視角P2，則虛擬攝影機3分別在XY平面的左側視角P1、右側視角P3、以及該ZY平面的上視角P4、下視角P5等五個視角位置拍攝該虛擬物件2，並分別建構前視角P2、左側視角P1、右側視角P3、上視角P4、下視角P5的第一表面點雲數據5，該第一表面點雲數 據5是虛擬物件2分別在該五個視角下的正面點雲集合，隨著視角的不同，每一個第一表面點雲數據5的形狀都不相同。在XY平面上，該左側視角P1的一視平面法線Npc與基準線Npi之間的夾角為-90度，該右側視角P3的一視平面法線Npc與基準線Npi之間的夾角為90度，在在XZ平面上，該上視角P4的一視平面法線Npc與基準線Npi之間的夾角為90度，該下視角P5的一視平面法線Npc與基準線Npi之間的夾角為-90度，上述的各角度為本發明所定義的『視角』。在建構第一表面點雲數據5時，虛擬攝影機3可以從三維中的任一方位建立第一表面點雲數據5，不限於上述五個視角。上述視角是基於各虛擬平面而定義，上述基準線Npi則未必一定是X軸。只要能界定虛擬攝影機3與虛擬物件2之間的三維關係，使用球座標也屬本發明可行的實施方式。 Referring to the first figure, the third figure and the third figure A, a virtual camera 3 in the computer shadow processing software is used to obtain the first surface point cloud data 5 of at least two viewing angles of the virtual object 2. In the third picture A, the virtual object 2 is located in a rectangular coordinate system. Preferably, the Z axis of the rectangular coordinate system corresponds to the height direction of the human body. The arrow on the Z axis points to the human head. The orientation of the virtual object 2 corresponds to The X-axis represents the direction of the liver perpendicular to the human chest when the human body is standing. Therefore, when the human body is supine for surgery, the X-axis will be the direction of the liver viewed from directly above the supine human body. Taking the aforementioned X-axis as a reference line N pi and viewing the virtual object 2 from the The virtual object 2 is photographed at five viewing angles including P4 and bottom viewing angle P5, and the first surface point cloud data 5 of the front viewing angle P2, the left viewing angle P1, the right viewing angle P3, the upper viewing angle P4, and the lower viewing angle P5 are constructed respectively. The surface point cloud data 5 is a set of front point clouds of the virtual object 2 under the five viewing angles. With different viewing angles, the shape of each first surface point cloud data 5 is different. On the XY plane, the angle between the normal line N pc of the first viewing plane of the left perspective P1 and the reference line N pi is -90 degrees, and the angle between the normal line N pc of the first viewing plane of the right perspective P3 and the reference line N pi is -90 degrees. The angle between is 90 degrees. On the XZ plane, the angle between the normal line N pc of the first viewing plane of the upper viewing angle P4 and the reference line N pi is 90 degrees . The angle between the reference lines N pi is -90 degrees, and each of the above angles is the "viewing angle" defined in the present invention. When constructing the first surface point cloud data 5, the virtual camera 3 can establish the first surface point cloud data 5 from any three-dimensional orientation, and is not limited to the above five viewing angles. The above-mentioned angle of view is defined based on each virtual plane, and the above-mentioned reference line N pi is not necessarily the X-axis. As long as the three-dimensional relationship between the virtual camera 3 and the virtual object 2 can be defined, using spherical coordinates is also a feasible implementation method of the present invention.
除了以虛擬攝影機3直接從虛擬物件2建立第一表面點雲數據5，本發明也可以先建立該虛擬物件2之一全表面點雲數據20，再從該全表面點雲數據20中擷取不同的第一表面點雲數據5。請參閱第三圖，該全表面點雲數據20有一基準線Npi，虛擬攝影機3之視平面法線Npc和上述全表面點雲數據20的基準線Npi之間有一法向角度θ，通過設置閾值以確定上述全表面點雲數據20位於哪個視面上，從而自上述全表面點雲數據20進行分割處理獲得該第一表面點雲數據5。其中，可以使用正弦和餘弦公式獲得法向角度θ：cos θpipc=
第三圖與第三A圖所示的二種建構第一表面點雲數據5的方法，都是本發明可以使用的的方法。 The two methods of constructing the first surface point cloud data 5 shown in the third figure and the third figure A are both methods that can be used in the present invention.
本實施例的虛擬物件2是一個虛擬肝臟，由於虛擬肝臟的外圍表面為連續曲面，由任一視角觀察虛擬肝臟，都可以獲得不同的三維曲面點雲，這些三維曲面點雲在前述視平面法線Npc的方向上具有縱深的變化，構成前述的第一表面點雲數據5。由不同的視角所擷取的第一表面點雲數據5將不相同，每一個第一表面點雲數據5除了輪廓不同之外，在視平面法線Npc方向上的每一點的縱深距離也不同。從愈多視角建立愈多的第一表面點雲數據5，代表建構了愈多的比對樣本，有助將第一表面點雲數據5與後述實境所視的肝臟假體1進行比對，並快速完成定位，迅速將完整的虛擬物件2正確疊合於實境的肝臟假體1。參閱第一圖、第五圖及第五A圖所示，在假體手術訓練中，使用者配戴一混合實境單元4觀看前述肝臟假體1，獲得該肝臟假體1的實境影像。該肝臟假體1被置放在一具有容納凹槽41的平台42上，容納凹槽41的形狀完全對應肝臟假體1的下半部，藉此固定肝臟假體1，肝臟假體1置放於平台42後，完整露出上半部，此時的肝臟假體1的方位如同人體仰臥手術床時，當時真實肝臟的方位，由平台42的上方俯視，肝臟假體1將與手術中俯看仰臥人體的肝臟是相同的。本實施例的混合實境單元4是一個混合實境眼鏡(MR眼鏡)，該混合實境眼鏡包含了一個深度攝影機。過程中，使用者配戴該混合實境眼鏡，透過混合實境眼鏡觀看實境中的肝臟假體1，該混台實境眼鏡連結一電腦，該電腦可以是外部的電腦或內建於MR眼鏡中的微型電腦，電腦中儲存有前述所有視角的第一表面點雲數據5。平台42具有水平的上表面，一觀察基準線Ng垂直於該上表面，該觀察基準線Ng對應了前述的基準線Npi。使用者 透過混合實境眼鏡俯看該肝臟假體1，使用者當時的觀察視線Nw與該觀察基準線Ng之間具有一觀察角度θ 1，觀察視線Nw代表了醫師在手術時觀看真實肝臟的可能視線，醫師的觀看位置在三維空間中變動，因此該觀察視線Nw是隨時變動的。使用者翻動或壓按該肝臟假體1，軟質的肝臟假體1會隨之變形或移位，使用者可以立即由混合實境單元4觀看到被翻動或移位的肝臟假體1，猶如真實手術中翻動或壓按該肝臟假體1所見的景像一樣。隨著肝臟假體1的移位，深度攝影機可以隨時獲取該肝臟假體1在當時觀察角度θ 1的表面影像，該表面影像包含了縱深的三維數據，根據該表面影像，電腦可以透過該縱深的三維數據，快速找出最接近該表面影像的第一表面點雲數據5，以及該第一表面點雲數據5所對應的視角，進一步將虛擬物件2迅速根據該視角以疊合於肝臟假體1。透過混合實境單元4拍攝肝臟假體1，並將虛擬物件2對準疊合於實境所視的該肝臟假體1，可以讓使用者從混合實境單元4即MR眼鏡中同時看到肝臟假體1與精準疊合的虛擬物件2，特別是虛擬物件2中的虛擬血管21，該虛擬血管21的大小與位置完全對應實境中肝臟假體1所包覆血管組織11，其中實境的大部分血管組織11因為被包覆而不可視，虛擬血管21則能在混合實境單元中被使用者看見。如此，使用者配戴MR眼鏡拍攝肝臟假體1進行肝臟手術練習時，可以透過虛擬血管21的顯示而預知肝臟假體的血管組織11的位置，在練習劃開或切割肝臟假體1時，可以有效避免意外劃開血管組織11。同時，練習過程中，使用者翻動或壓按該肝臟假體1，被包覆在肝臟假體1內部的血管組織11會隨之移位，而透過MR眼鏡，虛擬物件2與虛擬血管21都能對應肝臟假體1的移位，而快速更新位置並顯示於MR眼鏡中，因此即使被包覆的血 管組織11無法目視，但藉由虛擬血管21的即時更新位置，使用者能快速有效地練習掌握血管組織11的動態位置，精進肝臟手術技巧。 The virtual object 2 in this embodiment is a virtual liver. Since the peripheral surface of the virtual liver is a continuous curved surface, different three-dimensional curved surface point clouds can be obtained by observing the virtual liver from any viewing angle. These three-dimensional curved surface point clouds are calculated using the aforementioned view plane method. The direction of line N pc has changes in depth, forming the aforementioned first surface point cloud data 5 . The first surface point cloud data 5 captured from different viewing angles will be different. In addition to the outline of each first surface point cloud data 5 being different, the depth distance of each point in the direction of the normal line N pc of the viewing plane is also different. different. The more first surface point cloud data 5 are created from more perspectives, which means more comparison samples are constructed, which helps to compare the first surface point cloud data 5 with the liver prosthesis 1 viewed in the real world as described later. , and quickly complete the positioning, quickly and correctly superimpose the complete virtual object 2 on the real liver prosthesis 1. Referring to the first figure, the fifth figure and the fifth figure A, during the prosthetic surgery training, the user wears a mixed reality unit 4 to view the aforementioned liver prosthesis 1 and obtain a real-life image of the liver prosthesis 1 . The liver prosthesis 1 is placed on a platform 42 with a receiving groove 41. The shape of the receiving groove 41 completely corresponds to the lower half of the liver prosthesis 1, thereby fixing the liver prosthesis 1 and placing the liver prosthesis 1. After being placed on the platform 42, the upper half of the liver prosthesis 1 is completely exposed. At this time, the position of the liver prosthesis 1 is the same as when the human body is lying supine on the operating bed. The position of the real liver at that time, when viewed from the top of the platform 42, the liver prosthesis 1 will be in the same position as during the operation. Looking at the liver in a supine human body is the same. The mixed reality unit 4 of this embodiment is a mixed reality glasses (MR glasses), and the mixed reality glasses include a depth camera. During the process, the user wears the mixed reality glasses and views the liver prosthesis 1 in the real world through the mixed reality glasses. The mixed reality glasses are connected to a computer. The computer can be an external computer or built-in MR system. The microcomputer in the glasses stores the first surface point cloud data from all the aforementioned viewing angles5. The platform 42 has a horizontal upper surface, and an observation reference line N g is perpendicular to the upper surface, and the observation reference line N g corresponds to the aforementioned reference line N pi . The user looks down at the liver prosthesis 1 through mixed reality glasses. There is an observation angle θ 1 between the user's current observation line of sight N w and the observation reference line Ng. The observation line of sight N w represents the doctor's view of the real life during surgery. The possible line of sight of the liver and the doctor's viewing position change in the three-dimensional space, so the observation line of sight N w changes at any time. The user flips or presses the liver prosthesis 1, and the soft liver prosthesis 1 will deform or shift accordingly. The user can immediately see the flipped or shifted liver prosthesis 1 through the mixed reality unit 4, as if The scene seen when flipping or pressing the liver prosthesis 1 during real surgery is the same. As the liver prosthesis 1 is displaced, the depth camera can obtain the surface image of the liver prosthesis 1 at the current observation angle θ 1 at any time. The surface image contains depth three-dimensional data. Based on the surface image, the computer can penetrate the depth of the three-dimensional data, quickly find the first surface point cloud data 5 that is closest to the surface image, and the angle of view corresponding to the first surface point cloud data 5, and further quickly superimpose the virtual object 2 on the liver virtual object according to the angle of view. Body 1. The liver prosthesis 1 is photographed through the mixed reality unit 4, and the virtual object 2 is aligned and superimposed on the liver prosthesis 1 viewed in the real world, allowing the user to see the liver prosthesis 1 simultaneously from the mixed reality unit 4, that is, the MR glasses. The liver prosthesis 1 and the accurately superimposed virtual object 2, especially the virtual blood vessel 21 in the virtual object 2, the size and position of the virtual blood vessel 21 completely correspond to the vascular tissue 11 covered by the liver prosthesis 1 in the real world. Most of the blood vessel tissue 11 in the environment is invisible because it is covered, but the virtual blood vessel 21 can be seen by the user in the mixed reality unit. In this way, when the user wears MR glasses to take pictures of the liver prosthesis 1 for liver surgery practice, the user can predict the position of the vascular tissue 11 of the liver prosthesis through the display of the virtual blood vessels 21. When practicing cutting or cutting the liver prosthesis 1, It can effectively avoid accidental cutting of vascular tissue11. At the same time, during the practice, the user flips or presses the liver prosthesis 1, and the vascular tissue 11 covered inside the liver prosthesis 1 will be displaced accordingly, and through the MR glasses, the virtual object 2 and the virtual blood vessel 21 are both It can respond to the displacement of the liver prosthesis 1 and quickly update the position and display it on the MR glasses. Therefore, even if the covered vascular tissue 11 cannot be visually seen, the user can quickly and effectively update the position of the virtual blood vessel 21 Practice to master the dynamic position of vascular tissue 11 and improve liver surgery skills.
上述實施例中，透過深度攝影機可以取得實境中肝臟假體1表面的縱深數據，但除深度攝影機之外，以一般攝影機拍攝肝臟假體1表面，並確認光源的方位，則所拍攝的肝臟假體1影像，其迎光面與背光面的每一像素的色階將不相同，藉由該表面影像中的色階變化，透過運算也能獲得該表面影像的縱深數據，此類的攝影機不限定為單一，多個攝影機同時拍攝能獲得較好的運算結果。類似此類攝影機結合運算方式也屬於本發明的等效技術。 In the above embodiment, the depth data of the surface of the liver prosthesis 1 in the real world can be obtained through the depth camera. However, in addition to the depth camera, if a general camera is used to capture the surface of the liver prosthesis 1 and confirm the direction of the light source, then the captured liver In the image of prosthesis 1, the color level of each pixel on the light-facing surface and the backlight surface will be different. Through the color level changes in the surface image, the depth data of the surface image can also be obtained through calculation. This type of camera It is not limited to a single one. Shooting with multiple cameras at the same time can obtain better calculation results. Similar camera combination calculation methods also belong to the equivalent technology of the present invention.
上述使虛擬物件2與虛擬血管21對應肝臟假體1的移位或是對應混合實境單元4的移位而即時更新顯示位置，具體的技術內容為使用該混合實境單元4之該深度攝影機自一視角取得該肝臟假體1的一第二表面點雲數據6，由於深度攝影的掃描過程並不能有效地避開背景物體例如平台42，生成的點雲數據也包括了這些非必要對象，因此該深度攝影機拍攝該肝臟假體1後，將非該肝臟假體1的點雲數據定義為噪聲7，並使用統計過濾器(statistical filter)濾除該噪聲7而產生該第二表面點雲數據6。具體的，該統計過濾器(statistical filter)能夠計算每個點雲與其相鄰點雲之間的平均距離，然後根據均方差得到距離閾值，當一個點雲與相鄰點的平均距離大於閾值時，該點雲將被定義為噪聲7而去除。 The above-mentioned method enables the virtual object 2 and the virtual blood vessel 21 to update the display position in real time corresponding to the displacement of the liver prosthesis 1 or the displacement of the mixed reality unit 4. The specific technical content is to use the depth camera of the mixed reality unit 4. A second surface point cloud data 6 of the liver prosthesis 1 is obtained from a viewing angle. Since the scanning process of depth photography cannot effectively avoid background objects such as the platform 42, the generated point cloud data also includes these unnecessary objects. Therefore, after the depth camera captures the liver prosthesis 1, the point cloud data other than the liver prosthesis 1 is defined as noise 7, and a statistical filter is used to filter out the noise 7 to generate the second surface point cloud. Data 6. Specifically, the statistical filter can calculate the average distance between each point cloud and its adjacent point clouds, and then obtain the distance threshold based on the mean square error. When the average distance between a point cloud and adjacent points is greater than the threshold , the point cloud will be defined as noise 7 and removed.
參閱第一圖、第三A圖、第五A圖、第六圖至第八圖所示，當使用者透過混合實境單元8以第五A圖的視線Nu觀察該肝臟假體1時，該觀察視線Nw的觀看位置與第三A圖中的前視角P2方位最為接近。過程中，該混合實境單元8將該觀察視線Nw所拍攝到的第二表面點雲數據6與第一表面點雲數據5進行 比對，獲得視角相差最小之第一表面點雲數據5即前視角P2的第一表面點雲數據5，並根據前視角P2的第一表面點雲數據5旋轉或/及位移該虛擬物件2，使該虛擬物件2及該虛擬血管21之角度對應於該肝臟假體1及該血管組織11之觀察角度θ 1，完成初步定位，之後再以RMSE計算誤差並微調，對齊第一表面點雲數據5與第二表面點雲數據6，(完成細部定位使該虛擬物件2與該肝臟假體1的視角一致，並將該虛擬物件2精確疊合在肝臟假體1上。 Referring to the first figure, the third figure A, the fifth figure A, the sixth figure to the eighth figure, when the user observes the liver prosthesis 1 through the mixed reality unit 8 with the line of sight Nu of the fifth figure A , the viewing position of the observation line of sight N w is closest to the direction of the front viewing angle P2 in the third picture A. During the process, the mixed reality unit 8 compares the second surface point cloud data 6 captured by the observation line of sight N w with the first surface point cloud data 5 to obtain the first surface point cloud data 5 with the smallest viewing angle difference. That is, the first surface point cloud data 5 of the front perspective P2 is rotated or/and displaced according to the first surface point cloud data 5 of the front perspective P2, so that the angles of the virtual object 2 and the virtual blood vessel 21 correspond to The observation angle θ 1 of the liver prosthesis 1 and the vascular tissue 11 completes the preliminary positioning, and then calculates the error using RMSE and fine-tunes it to align the first surface point cloud data 5 and the second surface point cloud data 6 (complete the detailed positioning Make the viewing angles of the virtual object 2 and the liver prosthesis 1 consistent, and accurately overlap the virtual object 2 on the liver prosthesis 1 .
具體的，該第二表面點雲數據6與第一表面點雲數據5之比對包括對第一表面點雲數據5進行歸一化，使該虛擬物件2的尺寸縮放到與該器官假體1相同的尺寸，之後搜尋第二表面點雲數據6與第一表面點雲數據5之間的每個點的對應點，形成對應點集。獲得所述對應點集後，再透過最小平方法獲得該虛擬物件2與該肝臟假體1的相對方位並旋轉或/及位移該虛擬物件2。 Specifically, the comparison of the second surface point cloud data 6 and the first surface point cloud data 5 includes normalizing the first surface point cloud data 5 so that the size of the virtual object 2 is scaled to be the same as that of the organ prosthesis. 1 of the same size, and then search for the corresponding point of each point between the second surface point cloud data 6 and the first surface point cloud data 5 to form a corresponding point set. After obtaining the corresponding point set, the relative orientation of the virtual object 2 and the liver prosthesis 1 is obtained through the least squares method and the virtual object 2 is rotated or/and displaced.
其中，第一表面點雲數據5的集合為T1-5={T1,T2...TNx}，第二表面點雲數據6的集合為P={P1,P2...PNx}，對應點的搜尋如下式：
由於搜索對應點是整個算法中最耗時的一步，特別是當點雲數據的數量很大時，所花費的時間可能會更長，本實施例使用k-d樹進行優化來搜尋第二表面點雲數據6與第一表面點雲數據5之間的對應點(k-d樹的計算方法為習知技術在此不贅述)，透過k-d樹不斷地劃分比較空間，重複這個過程，直到到達葉子節點，而透過此使用空間劃分來減少每次計算的搜索區域，藉此減少整體計算負擔和時間。 Since searching for corresponding points is the most time-consuming step in the entire algorithm, especially when the amount of point cloud data is large, it may take longer. This embodiment uses k-d trees for optimization to search for the second surface point cloud. For the corresponding points between the data 6 and the first surface point cloud data 5 (the calculation method of the k-d tree is a conventional technology and will not be described in detail here), the comparison space is continuously divided through the k-d tree, and this process is repeated until the leaf node is reached, and Through this, spatial division is used to reduce the search area for each calculation, thereby reducing the overall computational burden and time.
所述最小平方法係：
該第一表面點雲數據5與第二表面點雲數據6的中心表示為：
該第一表面點雲數據5與第二表面點雲數據6的各點雲各自減去中心後表示為：
該第一表面點雲數據5與第二表面點雲數據6的協方差矩陣表示為：
再以奇異值分解(SVD)來計算該協方差矩陣W以獲得最佳轉換關係，計算式如下：
當W=3時，旋轉矩陣R將有最優唯一解為R=UVT。 When W=3, the rotation matrix R will have the optimal and unique solution R=UV T .
通過反向計算旋轉矩陣R獲得平移向量t為t=μP-RμT。 The translation vector t is obtained by inversely calculating the rotation matrix R as t=μ P -Rμ T .
之後計算RMSE誤差來微調該虛擬物件2的方位，RMSE如下：
參閱第九圖及第十圖所示，為獲取較佳迭代運算次數，本實施例進一步使用機械臂(UR5，Universal Robots，Odense，Denmark)將深度攝影機(D415e，FRAMOS，Munich，Germany)固定在肝臟假體1上方進行掃描，而獲得肝臟假體1表面的第二表面點雲數據6，並將肝臟假體1上方的位置定義為0度視角。根據前述實施例的處理後，將肝臟假體1與虛擬物件2的兩組點雲數據進行不同次數的迭代運算，迭代次數以2次為間隔從2次增加到52次，以比較不同迭代次數的RMSE誤差和計算時間。圖中可見迭代次數與RMSE呈負相關，RMSE隨著迭代次數的增加而減小，當迭代次數大於20時，RMSE的變化逐漸變得平滑，而當迭代次數為22時，RMSE小於2mm，而配準精度高。以往的研究表明，當圖像更新時間>300ms時，會造成圖像的停頓，讓使用者感覺體驗不佳，鑑於此，本發明將計算時間<200ms和RMSE<2mm的迭代次數的中位數設為30作為迭代次數。 Referring to Figures 9 and 10, in order to obtain a better number of iteration operations, this embodiment further uses a robotic arm (UR5, Universal Robots, Odense, Denmark) to fix the depth camera (D415e, FRAMOS, Munich, Germany) on The top of the liver prosthesis 1 is scanned to obtain the second surface point cloud data 6 of the surface of the liver prosthesis 1, and the position above the liver prosthesis 1 is defined as a 0-degree viewing angle. After processing according to the foregoing embodiment, the two sets of point cloud data of the liver prosthesis 1 and the virtual object 2 are subjected to different numbers of iteration operations. The number of iterations is increased from 2 to 52 times at intervals of 2 to compare the different numbers of iterations. RMSE error and calculation time. It can be seen from the figure that the number of iterations is negatively correlated with RMSE. RMSE decreases as the number of iterations increases. When the number of iterations is greater than 20, the changes in RMSE gradually become smoother. When the number of iterations is 22, the RMSE is less than 2mm, and The registration accuracy is high. Previous research has shown that when the image update time is >300ms, it will cause the image to pause, making the user experience a poor experience. In view of this, the present invention will calculate the median number of iterations with time <200ms and RMSE <2mm. Set to 30 as the number of iterations.
參閱第十一圖所示，本實施例進一步將虛擬物件2在不同視角下(±90°、±60°、±45°、±30°、±15°、±10°、±5°、0°)的第一表面點雲數據5與上述0度 視角的肝臟假體1的第二點雲數據6比對後並進行迭代運算而進行定位，視角±90°的誤差範圍為3.85mm至1.74mm，且當第一表面點雲數據5的視角越接近第二點雲數據6的視角，RMSE值會相對較小，反之，RMSE值會比較大。 Referring to Figure 11, this embodiment further displays the virtual object 2 at different viewing angles (±90°, ±60°, ±45°, ±30°, ±15°, ±10°, ±5°, 0 °) first surface point cloud data 5 with the above 0 degrees After comparing the second point cloud data 6 of the liver prosthesis 1 at different viewing angles and performing an iterative operation for positioning, the error range of the viewing angle ±90° is 3.85mm to 1.74mm, and when the viewing angle of the first surface point cloud data 5 exceeds At a viewing angle close to the second point cloud data 6, the RMSE value will be relatively small, and conversely, the RMSE value will be relatively large.
因此，透過上述說明及驗證，本發明不須使用外部特徵標記板也能夠在MR環境中快速定位虛擬物件的方位。 Therefore, through the above description and verification, the present invention can quickly locate the position of virtual objects in an MR environment without using external feature marking boards.
本發明除了使用在上述器官假體的MR手術訓練之外。請參閱第十二A圖、第十二B圖及第十二C圖，本發明之真實物件例如是生產線81上之工件82，該處理單元則為生產線之電腦83，透過工件82及其虛擬物件的比對，可以快速辨識工件的種類或方位，或者可用於工件瑕疵的檢測，其中將工件及其虛擬物件的第一表面點雲數據以第二表面點雲數據為基準進行歸一化而做尺寸的縮放時，並可分辨工件尺寸正確與否。以快速辨識生產線81上工件82的方位為例，直線前進的生產線81上連續擺置有待夾取的工件82，而每一工件82的方位並不一致，經由具備深度攝影功能的攝影機83拍攝每一工件82，電腦83執行本發明的快速追蹤定位之方法，可以快速定位被拍攝的該工件82，並藉此得知該工件82的偏移角度，電腦83進一步控制機械手臂85預先旋轉對應的角度，從生產線81上正確夾取該工件82，機械手臂85夾取該工件82離開生產線81後，迅速將工件82轉回正確方位，以繼續完成下一程序。 The present invention is used in addition to the MR surgical training of the above-mentioned organ prosthesis. Please refer to Figure 12A, Figure 12B and Figure 12C. The real object of the present invention is, for example, the workpiece 82 on the production line 81, and the processing unit is the computer 83 of the production line. Through the workpiece 82 and its virtual The comparison of objects can quickly identify the type or orientation of the workpiece, or can be used to detect workpiece defects. The first surface point cloud data of the workpiece and its virtual object are normalized based on the second surface point cloud data. When doing size scaling, it can also determine whether the workpiece size is correct or not. Take the quick identification of the position of the workpiece 82 on the production line 81 as an example. The workpieces 82 to be clamped are continuously placed on the straight-forward production line 81, and the position of each workpiece 82 is not consistent. Each workpiece 82 is captured by a camera 83 with a depth photography function. For the workpiece 82, the computer 83 executes the fast tracking and positioning method of the present invention, which can quickly locate the photographed workpiece 82 and thereby learn the offset angle of the workpiece 82. The computer 83 further controls the robot arm 85 to pre-rotate the corresponding angle. , the workpiece 82 is correctly clamped from the production line 81. After the robot arm 85 clamps the workpiece 82 and leaves the production line 81, it quickly turns the workpiece 82 back to the correct position to continue to complete the next program.
請參閱第十三A圖、第十三B圖及第十三C圖，以快速辨識生產線81A上的不同工件82A,82B為例，直線前進的生產線81A上連續擺置有待夾取的不同工件82A,82B，工件82A,82B的形狀不同，預設情況為圖中的機械手臂85A只檢選工件82A，棄檢工件82B。經由具備深度攝影功能的攝影機83A拍攝每一工件82A,82B，電腦83A執行本發明的快速追蹤定位之方法，快速定位被拍攝的 該工件82A,82B，電腦83A並藉由與內建虛擬物件的比對，根據比對結果得知該工件82A,82B的種類，電腦83A進一步控制機械手臂85A檢選預設的工件82A，如果工件82A也有方位不一致的情形，則可併用前述第十二圖的實施方式加以修正夾取時的角度。本實施例所棄檢工件82B，可以由另一機械手臂(圖中未示出)負責夾取，已進行工件82A,82B的分類。 Please refer to Figure 13A, Figure 13B and Figure 13C. Taking the quick identification of different workpieces 82A and 82B on the production line 81A as an example, the straight-forward production line 81A continuously places different workpieces to be clamped. 82A, 82B, the shapes of workpieces 82A, 82B are different. The default situation is that the robot arm 85A in the figure only inspects the workpiece 82A, and discards the inspection of the workpiece 82B. Each workpiece 82A, 82B is photographed through a camera 83A with a depth photography function. The computer 83A executes the fast tracking and positioning method of the present invention to quickly locate the photographed object. The computer 83A compares the workpieces 82A and 82B with built-in virtual objects and learns the types of the workpieces 82A and 82B based on the comparison results. The computer 83A further controls the robot arm 85A to select the preset workpiece 82A. If If the orientation of the workpiece 82A is inconsistent, the aforementioned embodiment in Figure 12 can be used to correct the angle during clamping. The workpiece 82B discarded for inspection in this embodiment can be picked up by another robot arm (not shown in the figure), and the workpieces 82A and 82B have been classified.
第十二A圖、第十二B圖、第十二C圖、第十三A圖、第十三B圖及第十三C圖所示的實施例中，工件82A,82B的表面並非曲面，僅根據工件82A,82B的影像輪廓即能將實體物件與虛擬物件進行對位。如果工件82A,82B的表面包含曲面，則藉由前述深度攝影機或一般攝影機結合影像光影的運算，也能將虛擬物件快速對位於實體物件，以對實體物件進行辨識。本發明的快速追蹤定位之方法，對位快速且正確，適合辨識生產線81A上快速移動的工件82A,82B。 In the embodiments shown in Figures 12A, 12B, 12C, 13A, 13B and 13C, the surfaces of the workpieces 82A and 82B are not curved surfaces. , the physical object and the virtual object can be aligned only based on the image contours of the workpieces 82A and 82B. If the surfaces of the workpieces 82A and 82B include curved surfaces, the virtual object can also be quickly aligned with the physical object through the aforementioned depth camera or general camera combined with image light and shadow calculations to identify the physical object. The fast tracking and positioning method of the present invention has fast and accurate positioning, and is suitable for identifying fast-moving workpieces 82A and 82B on the production line 81A.
上述擴增實境與混合實境下物件快速追蹤定位之方法所寫成之電腦程式可儲存於雲端供下載，或可儲存於電腦可讀取媒體中。 The computer program written for the above method of fast tracking and positioning of objects in augmented reality and mixed reality can be stored in the cloud for download, or can be stored in computer-readable media.
綜合上述實施例之說明，當可充分瞭解本發明之操作、使用及本發明產生之功效，惟以上所述實施例僅係為本發明之較佳實施例，當不能以此限定本發明實施之範圍，即依本發明申請專利範圍及發明說明內容所作簡單的等效變化與修飾，皆屬本發明涵蓋之範圍內。 Based on the description of the above embodiments, the operation, use and effects of the present invention can be fully understood. However, the above embodiments are only preferred embodiments of the present invention and should not be used to limit the implementation of the present invention. The scope, that is, simple equivalent changes and modifications based on the patent scope of the present invention and the description of the invention, are all within the scope of the present invention.
Claims (12)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| TW111145521A TWI817847B (en) | 2022-11-28 | 2022-11-28 | Method, computer program and computer readable medium for fast tracking and positioning objects in augmented reality and mixed reality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| TW111145521A TWI817847B (en) | 2022-11-28 | 2022-11-28 | Method, computer program and computer readable medium for fast tracking and positioning objects in augmented reality and mixed reality |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| TWI817847B true TWI817847B (en) | 2023-10-01 |
| TW202422483A TW202422483A (en) | 2024-06-01 |
Family
ID=89857995
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| TW111145521A TWI817847B (en) | 2022-11-28 | 2022-11-28 | Method, computer program and computer readable medium for fast tracking and positioning objects in augmented reality and mixed reality |
Country Status (1)
| Country | Link |
|---|---|
| TW (1) | TWI817847B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111275734A (en) * | 2018-12-04 | 2020-06-12 | 中华电信股份有限公司 | Object identification and tracking system and method thereof |
| TW202034215A (en) * | 2018-08-13 | 2020-09-16 | 英國帝國科技及醫學學院 | Mapping object instances using video data |
| TWI709062B (en) * | 2019-09-20 | 2020-11-01 | 財團法人資訊工業策進會 | Virtuality reality overlapping method and system |
| US20210213973A1 (en) * | 2018-08-29 | 2021-07-15 | Movidius Ltd. | Computer vision system |
| TW202221656A (en) * | 2020-11-23 | 2022-06-01 | 未來市股份有限公司 | Head mounted display system and method for constructing virtual environment of architectural space |
-
2022
- 2022-11-28 TW TW111145521A patent/TWI817847B/en active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW202034215A (en) * | 2018-08-13 | 2020-09-16 | 英國帝國科技及醫學學院 | Mapping object instances using video data |
| US20210213973A1 (en) * | 2018-08-29 | 2021-07-15 | Movidius Ltd. | Computer vision system |
| CN111275734A (en) * | 2018-12-04 | 2020-06-12 | 中华电信股份有限公司 | Object identification and tracking system and method thereof |
| TWI709062B (en) * | 2019-09-20 | 2020-11-01 | 財團法人資訊工業策進會 | Virtuality reality overlapping method and system |
| TW202221656A (en) * | 2020-11-23 | 2022-06-01 | 未來市股份有限公司 | Head mounted display system and method for constructing virtual environment of architectural space |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Wang et al. | Video see‐through augmented reality for oral and maxillofacial surgery | |
| Chen et al. | SLAM-based dense surface reconstruction in monocular minimally invasive surgery and its application to augmented reality | |
| US10432913B2 (en) | Systems and methods for determining three dimensional measurements in telemedicine application | |
| Grasa et al. | Visual SLAM for handheld monocular endoscope | |
| Dey et al. | Automatic fusion of freehand endoscopic brain images to three-dimensional surfaces: creating stereoscopic panoramas | |
| Boctor et al. | A novel closed form solution for ultrasound calibration | |
| Doignon et al. | Segmentation and guidance of multiple rigid objects for intra-operative endoscopic vision | |
| BRPI0919448B1 (en) | method for tracking a follicular unit and system for tracking a follicular unit. | |
| Richa et al. | Vision-based proximity detection in retinal surgery | |
| NL2022371B1 (en) | Method and assembly for spatial mapping of a model of a surgical tool onto a spatial location of the surgical tool, as well as a surgical tool | |
| Lapeer et al. | Image‐enhanced surgical navigation for endoscopic sinus surgery: evaluating calibration, registration and tracking | |
| CN111544037A (en) | Ultrasonic positioning method and system based on binocular vision | |
| JP6493885B2 (en) | Image alignment apparatus, method of operating image alignment apparatus, and image alignment program | |
| Li et al. | A vision-based navigation system with markerless image registration and position-sensing localization for oral and maxillofacial surgery | |
| JP2002008012A (en) | Method for calculating position and attitude of subject and method for calculating position and attitude of observation camera | |
| Piccinelli et al. | Rigid 3D registration of pre-operative information for semi-autonomous surgery | |
| TWI817847B (en) | Method, computer program and computer readable medium for fast tracking and positioning objects in augmented reality and mixed reality | |
| EP2009613A1 (en) | System for simultaing a manual interventional operation | |
| Vogt et al. | Light fields for minimal invasive surgery using an endoscope positioning robot | |
| Wang et al. | Stereo video analysis for instrument tracking in image-guided surgery | |
| TW202422483A (en) | Method, computer program and computer readable medium for fast tracking and positioning objects in augmented reality and mixed reality | |
| Pérez et al. | Video-based tracking of laparoscopic instruments using an orthogonal webcams system | |
| CN115024805A (en) | Method, system and storage medium for assisting puncture of endoscopic surgery | |
| CN110051433B (en) | Method for keeping track of target and application thereof in image-guided surgery | |
| CN111860275A (en) | Gesture recognition data acquisition system and method |