TWI790854B

TWI790854B - Multi-modal image alignment method and system

Info

Publication number: TWI790854B
Application number: TW110146318A
Authority: TW
Inventors: 黃捷; 鄭憲君; 丁文宏; 李家昶
Original assignee: 財團法人工業技術研究院
Priority date: 2021-08-19
Filing date: 2021-12-10
Publication date: 2023-01-21
Also published as: TW202309833A

Abstract

A multi-modal image alignment method includes obtaining a number of first points corresponding to a center vertex of a calibration object and a number of second point groups corresponding to side vertices of the calibration object from a number of two-dimensional images, obtaining a number of third points corresponding to the center vertex from a number of three-dimensional images, performing first optimization using a first coordinate system associated with the two-dimensional images, the first points and the third points to obtain a first transformation matrix, processing the three-dimensional images using the first transformation matrix to generate a number of firstly-transformed images respectively, performing second optimization using the firstly-transformed images, the first points and the second point groups to obtain a second transformation matrix, and transforming an image to be processed from a second coordinate system associated with the three-dimensional images to the first coordinate system using the first transformation matrix and the second transformation.

Description

多模影像對位方法及系統Multi-mode image alignment method and system

本發明係關於一種多模影像對位方法，特別係關於一種三維影像與二維影像的對位方法。The invention relates to a method for alignment of multi-mode images, in particular to a method for alignment of three-dimensional images and two-dimensional images.

在後疫情時代，科技、經濟、文化、公衛等層面在全球都進入一個新常態，各國從抗疫走向與疫共存，民眾的生活習慣亦隨之改變。除了出門戴口罩之外，民眾進入各種室內場所皆須量測體溫。目前市面上常見的體溫偵測系統，多數僅針對溫度量測，而在體溫量測已為生活日常的情況下，應考量系統功能的擴充，例如生理訊號量測、行動偵測等。In the post-epidemic era, technology, economy, culture, public health and other aspects have entered a new normal around the world. Countries have shifted from fighting the epidemic to coexisting with the epidemic, and people's living habits have also changed accordingly. In addition to wearing masks when going out, people must measure their body temperature when entering various indoor places. Most of the body temperature detection systems currently on the market are only for temperature measurement, and when body temperature measurement is already a part of daily life, the expansion of system functions should be considered, such as physiological signal measurement, motion detection, etc.

上述一機多功能之訴求的基礎量測訊號多為體溫、心搏、呼吸及行為等，所需的非接觸式感測來源多為可見光影像、熱影像及三維影像（點雲）。以成本為考量，目前主流方式係以多種平價品牌的影像擷取裝置混搭以嘗試滿足一機多功能的訴求。然而，各影像擷取裝置的座標系統彼此理應不相同，因此無法同時取得某標的物的多模。The basic measurement signals for the above-mentioned one-machine multi-function demands are mostly body temperature, heartbeat, respiration, and behavior, etc., and the required non-contact sensing sources are mostly visible light images, thermal images, and 3D images (point clouds). Taking cost into consideration, the current mainstream method is to mix and match image capture devices of various affordable brands in an attempt to meet the demands of one machine with multiple functions. However, the coordinate systems of the image capture devices are supposed to be different from each other, so it is impossible to obtain multiple modes of a certain object at the same time.

鑒於上述，本發明提供一種多模影像對位方法及系統。In view of the above, the present invention provides a multi-mode image alignment method and system.

依據本發明一實施例的多模影像對位方法，包含以處理裝置執行：取得關聯於校正體的多張二維影像及多張三維影像，校正體具有中央頂點及多個側頂點，二維影像關聯於第一三維座標系統，且三維影像關聯於第二三維座標系統；從所述多張二維影像取得對應於中央頂點的多個第一點及對應於所述多個側頂點的多個第二點群；從所述多張三維影像取得對應於中央頂點的多個第三點；利用待解第一轉換矩陣、所述多個第一點及所述多個第三點，基於第一三維座標系統執行第一最佳化運算，以取得最佳化第一轉換矩陣；利用最佳化第一轉換矩陣處理所述多張三維影像，以分別產生多張一次轉換影像；利用所述多張一次轉換影像、所述多個第一點、所述多個第二點群以及校正體的預設規格參數組，基於第一三維座標系統執行第二最佳化運算，以取得最佳化第二轉換矩陣；以及利用最佳化第一轉換矩陣及最佳化第二轉換矩陣執行轉換運算，以將待處理影像轉換至第二三維座標系統或第一三維座標系統。The multi-mode image alignment method according to an embodiment of the present invention includes executing by a processing device: obtaining multiple 2D images and multiple 3D images associated with a calibration body, the calibration volume has a central vertex and multiple side vertices, and the 2D images are associated with In the first three-dimensional coordinate system, and the three-dimensional image is associated with the second three-dimensional coordinate system; obtaining a plurality of first points corresponding to the central vertex and a plurality of second points corresponding to the plurality of side vertices from the plurality of two-dimensional images group; obtain a plurality of third points corresponding to the central vertex from the plurality of three-dimensional images; use the first transformation matrix to be solved, the plurality of first points and the plurality of third points, based on the first three-dimensional coordinates The system executes a first optimization operation to obtain an optimized first transformation matrix; process the plurality of 3D images by using the optimized first transformation matrix to generate a plurality of primary transformation images respectively; converting the image, the plurality of first points, the plurality of second point groups, and the preset specification parameter set of the calibration body, and performing a second optimization operation based on the first three-dimensional coordinate system to obtain an optimized second transformation matrix; and performing a transformation operation by using the optimized first transformation matrix and the optimized second transformation matrix, so as to transform the image to be processed into the second 3D coordinate system or the first 3D coordinate system.

依據本發明一實施例的多模影像對位系統，包含校正體、二維影像擷取裝置、三維影像擷取裝置及處理裝置，其中處理裝置連接於該二維影像擷取裝置及該三維影像擷取裝置。校正體包含立體本體及多個指示元件，其中立體本體具有一中央頂點及多個側頂點，且指示元件分別設置於該中央頂點及該些側頂點。二維影像擷取裝置具有第一三維座標系統，且用於產生關聯於校正體的多張二維影像。三維影像擷取裝置具有第二三維座標系統，且用於產生關聯於校正體的多張三維影像。處理裝置用於依據所述多張二維影像及所述多張三維影像，取得座標轉換矩陣，且利用座標轉換矩陣將待處理影像轉換至第二三維座標系統或第一三維座標系統。A multi-mode image alignment system according to an embodiment of the present invention includes a calibration body, a 2D image capture device, a 3D image capture device, and a processing device, wherein the processing device is connected to the 2D image capture device and the 3D image Capture device. The correction body includes a three-dimensional body and a plurality of indicating elements, wherein the three-dimensional body has a central vertex and a plurality of side vertices, and the indicating elements are respectively arranged on the central apex and the side vertices. The 2D image capture device has a first 3D coordinate system and is used to generate multiple 2D images associated with the calibration body. The 3D image capture device has a second 3D coordinate system and is used to generate multiple 3D images associated with the calibration body. The processing device is used to obtain a coordinate transformation matrix according to the plurality of 2D images and the plurality of 3D images, and use the coordinate transformation matrix to transform the image to be processed into the second 3D coordinate system or the first 3D coordinate system.

藉由上述架構，本案所揭示的多模影像對位方法可以透過兩次最佳化運算取得不同座標系統之間的轉換矩陣，不須使用複雜的機器學習訓練即可達成高精準度的對位效果，且透過以三維角點特徵為求取轉換矩陣之根據，相較於傳統以平面式棋盤校正板為求取轉換矩陣之根據，所需之取樣資料數量甚少，即所需取樣時間較少。本案所揭示的多模影像對位系統同樣具有所需之取樣資料數量及取樣時間少的效果，且藉由設置有指示元件之特殊立體校正體設計，系統可以實現二維/三維影像中之特徵點的自動擷取。With the above structure, the multi-mode image alignment method disclosed in this case can obtain the transformation matrix between different coordinate systems through two optimization operations, and can achieve high-precision alignment without using complicated machine learning training effect, and by using the three-dimensional corner features as the basis for obtaining the transformation matrix, compared with the traditional planar checkerboard correction board as the basis for obtaining the transformation matrix, the amount of sampling data required is very small, that is, the required sampling time is shorter few. The multi-mode image alignment system disclosed in this case also has the effect of less sampling data and less sampling time, and the system can realize the characteristics of 2D/3D images through the design of a special stereo calibration body equipped with indicating elements Automatic extraction of points.

以上之關於本揭露內容之說明及以下之實施方式之說明係用以示範與解釋本發明之精神與原理，並且提供本發明之專利申請範圍更進一步之解釋。The above description of the disclosure and the following description of the implementation are used to demonstrate and explain the spirit and principle of the present invention, and provide a further explanation of the patent application scope of the present invention.

以下在實施方式中詳細敘述本發明之詳細特徵以及優點，其內容足以使任何熟習相關技藝者了解本發明之技術內容並據以實施，且根據本說明書所揭露之內容、申請專利範圍及圖式，任何熟習相關技藝者可輕易地理解本發明相關之目的及優點。以下之實施例係進一步詳細說明本發明之觀點，但非以任何觀點限制本發明之範疇。The detailed features and advantages of the present invention are described in detail below in the implementation mode, and its content is enough to make any person familiar with the related art understand the technical content of the present invention and implement it accordingly, and according to the content disclosed in this specification, the scope of the patent application and the drawings , anyone skilled in the art can easily understand the purpose and advantages of the present invention. The following examples are to further describe the concept of the present invention in detail, but not to limit the scope of the present invention in any way.

請參考圖1，圖1係依據本發明一實施例所繪示的多模影像對位系統1的功能方塊圖。如圖1所示，多模影像對位系統1包含處理裝置11、二維影像擷取裝置12、三維影像擷取裝置13及校正體14，其中處理裝置14可以有線或無線的方式連接於二維影像擷取裝置12及三維影像擷取裝置13。Please refer to FIG. 1 , which is a functional block diagram of a multi-mode image alignment system 1 according to an embodiment of the present invention. As shown in FIG. 1 , the multi-mode image alignment system 1 includes a processing device 11, a two-dimensional image capture device 12, a three-dimensional image capture device 13 and a calibration body 14, wherein the processing device 14 can be connected to the two devices in a wired or wireless manner. A three-dimensional image capture device 12 and a three-dimensional image capture device 13.

處理裝置11可以包含但不限於單一處理器以及多個微處理器之集成，例如中央處理器（CPU）、繪圖處理器（GPU）等。處理裝置11用於依據二維影像擷取裝置12及三維影像擷取裝置13對校正體14進行拍攝而產生的影像，取得兩影像擷取裝置之座標系統的轉換矩陣，詳細的執行內容將於後描述。處理裝置11可以利用所述轉換矩陣，將三維影像擷取裝置13所產生的影像映射至二維影像擷取裝置12所產生的影像，以產生疊合影像。The processing device 11 may include but not limited to a single processor and an integration of multiple microprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), and the like. The processing device 11 is used to obtain the transformation matrix of the coordinate system of the two image capturing devices 12 and 3D image capturing device 13 based on the images generated by shooting the calibration object 14. The detailed execution content will be described in described later. The processing device 11 can use the transformation matrix to map the image generated by the 3D image capture device 13 to the image generated by the 2D image capture device 12 to generate a superimposed image.

二維影像擷取裝置12例如為可見光攝影機、近紅外光攝影機、熱像儀等。二維影像擷取裝置12用於進行拍攝以產生二維影像，且具有一相機座標系統及一影像平面座標系統。三維影像擷取裝置13例如為三維點雲感測器、深度攝影機等。三維影像擷取裝置13用於進行拍攝以產生三維影像，且具有一相機座標系統。二維影像擷取裝置12及三維影像擷取裝置13可以設置於同一機殼內，或可以分別設置於不同機殼。The two-dimensional image capture device 12 is, for example, a visible light camera, a near-infrared light camera, a thermal imager, and the like. The 2D image capturing device 12 is used for shooting to generate a 2D image, and has a camera coordinate system and an image plane coordinate system. The 3D image capturing device 13 is, for example, a 3D point cloud sensor, a depth camera, and the like. The 3D image capturing device 13 is used for shooting to generate a 3D image, and has a camera coordinate system. The 2D image capture device 12 and the 3D image capture device 13 can be disposed in the same housing, or can be respectively disposed in different housings.

校正體14包含本體140及多個指示元件141a～141d。立體本體140可以具有四個以上的頂點。圖1示例性地繪示立體本體140為六面體，然不限於此。指示元件141a～141d可以分別設置於立體本體140的頂點，其數量至少為四個。指示元件141a～141d可以皆為發光元件、皆為發熱元件，或皆為兼具發光及發熱功能的元件。指示元件141a～141d可以全部具有相同顏色或/及溫度，或可以分別具有不同顏色或/及溫度。The calibration body 14 includes a main body 140 and a plurality of indicating elements 141a-141d. The three-dimensional body 140 may have more than four vertices. FIG. 1 exemplarily shows that the three-dimensional body 140 is a hexahedron, but it is not limited thereto. The indicating elements 141 a - 141 d can be respectively disposed on vertices of the three-dimensional body 140 , and the number thereof is at least four. The indicating elements 141a-141d can all be light-emitting elements, all be heating elements, or all be elements with both light-emitting and heating functions. The indicating elements 141a-141d may all have the same color or/and temperature, or may have different colors or/and temperatures respectively.

進一步來說，指示元件141a～141d的種類可以取決於二維影像擷取裝置12的種類。於可見光攝影機作為二維影像擷取裝置的實施態樣中，指示元件141a～141d以發光元件實現。於近紅外光攝影機或熱像儀作為二維影像擷取裝置12的實施態樣中，指示元件141a～141d以發熱元件實現。於另一實施例中，多模影像對位系統1可以更包含另一二維影像擷取裝置。若兩個二維影像擷取裝置分別為可見光相機及熱像儀，則指示元件141a～141d以兼具發光及發熱功能的元件實現。Further, the types of the indicating elements 141 a - 141 d may depend on the type of the 2D image capture device 12 . In an embodiment in which the visible light camera is used as a two-dimensional image capture device, the indicating elements 141 a - 141 d are realized by light emitting elements. In an embodiment in which a near-infrared camera or a thermal imager is used as the two-dimensional image capture device 12 , the indicating elements 141 a - 141 d are implemented as heating elements. In another embodiment, the multi-mode image alignment system 1 may further include another 2D image capture device. If the two two-dimensional image capture devices are respectively a visible light camera and a thermal imager, the indicating elements 141 a - 141 d are realized by elements having both light emitting and heating functions.

多模影像對位系統1的校正體14、二維影像擷取裝置12及三維影像擷取裝置13可以協同運作以產生供處理裝置11進行多模影像校正的資料。校正體14可以輪流擺放於多個校正位置。二維影像擷取裝置12可以受控以拍攝輪流擺放於不同校正位置的校正體14而產生多張二維影像。三維影像擷取裝置13可以受控以拍攝輪流擺放於不同校正位置的校正體14而產生多張三維影像。處理裝置11可以從二維影像擷取裝置12取得所述多張二維影像且從三維影像擷取裝置13取得所述多張三維影像，並據以進行多模影像對位校正，以取得二維影像擷取裝置12的相機座標系統與三維影像擷取裝置13的相機座標系統之間的轉換矩陣。The calibration body 14 of the multi-mode image alignment system 1 , the 2D image capture device 12 and the 3D image capture device 13 can cooperate to generate data for the processing device 11 to perform multi-mode image correction. The calibration body 14 can be placed in multiple calibration positions in turn. The 2D image capture device 12 can be controlled to capture the calibration objects 14 placed in different calibration positions in turn to generate multiple 2D images. The 3D image capture device 13 can be controlled to capture the calibration objects 14 placed in different calibration positions in turn to generate multiple 3D images. The processing device 11 can obtain the multiple 2D images from the 2D image capture device 12 and the multiple 3D images from the 3D image capture device 13, and perform multi-mode image alignment correction accordingly to obtain a 2D image A conversion matrix between the camera coordinate system of the capture device 12 and the camera coordinate system of the 3D image capture device 13 .

另外，處理裝置11更可以依據此轉換矩陣及二維影像擷取裝置12的焦距及投影中心，取得二維影像擷取裝置12的影像平面座標系統與三維影像擷取裝置13的相機座標系統之間的轉換矩陣，且可以利用此轉換矩陣疊合兩裝置所產生的影像。處理裝置11更可以從疊合影像取得對應於特定標的物的多模訊號（例如包含溫度資訊及空間資訊，或包含顏色資訊及空間資訊）。所述特定標的物可由處理裝置11依特定演算法（例如人臉辨識演算法）來選取或由操作人員選取，本發明不予限制。或者，處理裝置11可以將疊合影像透過顯示器呈現給操作人員，或輸出至其他多模訊號量測應用裝置。In addition, the processing device 11 can further obtain the relationship between the image plane coordinate system of the 2D image capture device 12 and the camera coordinate system of the 3D image capture device 13 according to the transformation matrix and the focal length and projection center of the 2D image capture device 12 The transformation matrix between them can be used to superimpose the images generated by the two devices. The processing device 11 can further obtain a multimode signal corresponding to a specific object (for example, including temperature information and spatial information, or including color information and spatial information) from the superimposed image. The specific target can be selected by the processing device 11 according to a specific algorithm (such as a face recognition algorithm) or by an operator, which is not limited by the present invention. Alternatively, the processing device 11 can present the superimposed image to the operator through a display, or output it to other multi-mode signal measurement application devices.

請一併參考圖1及圖2，其中圖2係依據本發明一實施例所繪示的校正位置示意圖。如圖2所示，校正體14的校正位置P1～P6位於二維影像擷取裝置12的視角v1及三維影像擷取裝置13的視角v2之重疊範圍內，其數量至少為六。進一步來說，在每個校正位置P1～P6上，校正體14的指示元件141a～141d對於二維影像擷取裝置12及三維影像接取裝置13而言皆為可視。三維影像的數量可以相同於校正位置P1～P6的數量，二維影像的數量則取決於校正體14的指示元件141a～141d的控制方法。Please refer to FIG. 1 and FIG. 2 together, wherein FIG. 2 is a schematic diagram of a calibration position according to an embodiment of the present invention. As shown in FIG. 2 , the calibration positions P1 - P6 of the calibration body 14 are located within the overlapping range of the viewing angle v1 of the 2D image capturing device 12 and the viewing angle v2 of the 3D image capturing device 13 , and the number thereof is at least six. Furthermore, at each of the calibration positions P1 - P6 , the indicator elements 141 a - 141 d of the calibration body 14 are visible to both the 2D image capture device 12 and the 3D image capture device 13 . The number of 3D images can be the same as the number of calibration positions P1 - P6 , and the number of 2D images depends on the control method of the indicating elements 141 a - 141 d of the calibration body 14 .

於一實施態樣中，校正體14的指示元件141a～141d分別具有不同顏色或不同溫度，且在拍攝過程中被一同致能。二維影像擷取裝置12受控以拍攝輪流擺放於不同校正位置P1～P6的校正體14以產生同於校正位置P1～P6之數量的二維影像。每張二維影像包含對應於不同顏色或/及不同溫度的指示元件141a～141d的影像區塊。In an embodiment, the indicating elements 141 a - 141 d of the calibration body 14 have different colors or different temperatures respectively, and are enabled together during the shooting process. The 2D image capturing device 12 is controlled to capture the calibration objects 14 placed in different calibration positions P1 - P6 in turn to generate 2D images equal to the number of calibration positions P1 - P6 . Each two-dimensional image includes image blocks corresponding to the indicating elements 141 a - 141 d of different colors or/and different temperatures.

於另一實施態樣中，校正體14的指示元件141a～141d在一次拍攝程序中被依特定次序致能而發光或發熱。二維影像擷取裝置12受控以對輪流擺放於多個校正位置P1～P6的校正體14分別進行多次拍攝程序以產生多張二維影像。舉例來說，校正體14被擺放於校正位置P1～P6中的任一個時，指示元件141a被致能且二維影像擷取裝置12受控以產生包含對應於指示元件141a之影像區塊的二維影像，接著指示元件141b被致能且二維影像擷取裝置12受控以產生包含對應於指示元件141b之影像區塊的二維影像，指示元件141c及141d所對應之二維影像的產生為相同道理，便不予贅述。於此實施態樣中，二維影像的數量為校正位置P1～P6數量的N倍，其中N為校正體14之指示元件141a～141d的數量。In another embodiment, the indicating elements 141 a - 141 d of the calibration body 14 are activated in a specific order to emit light or generate heat during a shooting procedure. The 2D image capture device 12 is controlled to perform multiple shooting procedures on the calibration objects 14 placed in the multiple calibration positions P1 - P6 in turn to generate multiple 2D images. For example, when the calibration body 14 is placed in any one of the calibration positions P1-P6, the indicating element 141a is enabled and the two-dimensional image capturing device 12 is controlled to generate an image block corresponding to the indicating element 141a 2D image, then the indicating element 141b is enabled and the 2D image capture device 12 is controlled to generate a 2D image including the image block corresponding to the indicating element 141b, and the 2D image corresponding to the indicating elements 141c and 141d The generation is the same reason, so it will not be repeated. In this embodiment, the number of two-dimensional images is N times the number of calibration positions P1 - P6 , where N is the number of indicating elements 141 a - 141 d of the calibration body 14 .

上述校正體14的指示元件141a～141b的控制、二維影像擷取裝置12的拍攝及三維影像擷取裝置13的拍攝可以由操作人員所控制，或可以由儲存有相應控制指令的控制器藉由有線或無線的方式來控制，本發明不予限制。The control of the indicating elements 141a-141b of the correction body 14, the shooting of the two-dimensional image capturing device 12 and the shooting of the three-dimensional image capturing device 13 can be controlled by the operator, or can be borrowed by a controller storing corresponding control instructions. It is controlled in a wired or wireless manner, which is not limited in the present invention.

請一併參考圖1及圖3，其中圖3係依據本發明一實施例所繪示的多模影像對位方法的流程圖。如圖3所示，多模影像對位方法包含步驟S101：取得關聯於校正體的多張二維影像及多張三維影像，校正體具有中央頂點及多個側頂點，二維影像對應於第一三維座標系統，且三維影像對應於第二三維座標系統；步驟S102：從所述多張二維影像取得對應於中央頂點的多個第一點及對應於所述多個側頂點的多個第二點群；步驟S103：從所述多張三維影像取得對應於中央頂點的多個第三點；步驟S104：利用待解第一轉換矩陣、所述多個第一點及所述多個第三點，基於第一三維座標系統執行第一最佳化運算，以取得最佳化第一轉換矩陣；步驟S105：利用最佳化第一轉換矩陣處理所述多張三維影像，以分別產生多張一次轉換影像；步驟S106：利用所述多張一次轉換影像、所述多個第一點、所述多個第二點群及校正體的預設規格參數組，基於第一三維座標系統執行第二最佳化運算，以取得最佳化第二轉換矩陣；以及步驟S107：利用最佳化第一轉換矩陣及最佳化第二轉換矩陣執行轉換運算，以將待處理影像轉換至第二三維座標系統或第一三維座標系統。於此要特別說明的是，本發明不限制步驟S102及步驟S103的執行順序。Please refer to FIG. 1 and FIG. 3 together, wherein FIG. 3 is a flowchart of a multi-mode image alignment method according to an embodiment of the present invention. As shown in FIG. 3 , the multi-mode image alignment method includes step S101: obtaining multiple 2D images and multiple 3D images associated with the calibration body, the calibration volume has a central vertex and multiple side vertices, and the 2D images correspond to the first 3D A coordinate system, and the 3D image corresponds to a second 3D coordinate system; Step S102: Obtain a plurality of first points corresponding to the central vertex and a plurality of second point groups corresponding to the plurality of side vertices from the plurality of 2D images ; Step S103: Obtain a plurality of third points corresponding to the central vertex from the plurality of 3D images; Step S104: Using the first transformation matrix to be solved, the plurality of first points and the plurality of third points, Perform a first optimization operation based on the first three-dimensional coordinate system to obtain an optimized first transformation matrix; step S105: process the plurality of three-dimensional images by using the optimized first transformation matrix to generate multiple primary transformations respectively Image; Step S106: Using the plurality of once-transformed images, the plurality of first points, the plurality of second point groups, and the preset specification parameter set of the correction body, perform the second most accurate operation based on the first three-dimensional coordinate system an optimization operation to obtain an optimized second transformation matrix; and step S107: performing a transformation operation using the optimized first transformation matrix and the optimized second transformation matrix to transform the image to be processed into a second three-dimensional coordinate system or the first 3D coordinate system. It should be noted here that, the present invention does not limit the execution order of step S102 and step S103 .

圖3所示的多模影像對位方法可以適用於圖1所示的多模影像對位系統1，特別係由處理裝置11來執行。以下示例性地以多模影像對位系統1的運作來進一步說明步驟S101～S107。The multi-mode image alignment method shown in FIG. 3 can be applied to the multi-mode image alignment system 1 shown in FIG. 1 , especially executed by the processing device 11 . The following exemplifies the operation of the multi-mode image alignment system 1 to further illustrate steps S101 - S107 .

於步驟S101中，處理裝置11取得校正體14的多張二維影像及多張三維影像。所述多張二維影像及所述多張三維影像的取得方法如前所述，於此不予贅述。步驟S101中的第一三維座標系統可以為二維影像擷取裝置12的相機座標系統，第二三維座標系統可以為三維影像擷取裝置13的相機座標系統，中央頂點可以為校正體14設置有指示元件141a的頂點，所述多個側頂點可以為分別設置有指示元件141b～141d的頂點。In step S101 , the processing device 11 acquires multiple 2D images and multiple 3D images of the calibration body 14 . The methods for obtaining the plurality of 2D images and the plurality of 3D images are as described above, and will not be repeated here. The first three-dimensional coordinate system in step S101 may be the camera coordinate system of the two-dimensional image capture device 12, the second three-dimensional coordinate system may be the camera coordinate system of the three-dimensional image capture device 13, and the central vertex may be the calibration body 14 provided with The vertices of the indicator element 141a, the plurality of side vertices may be the vertices respectively provided with the indicator elements 141b-141d.

於步驟S102中，處理裝置11從所述多張二維影像取得對應於中央頂點的多個第一點及對應於所述多個側頂點的多個第二點群。於每張二維影像包含對應於不同顏色或/及不同溫度的指示元件141a～141d的影像區塊的實施態樣中，處理裝置11可以預先儲存查找表，查找表記錄中央頂點及側頂點各自所對應的顏色或/及溫度。處理裝置11可以影像處理演算法找到具有不同顏色或/及溫度的影像區塊，其中影像處理演算法例如包含但不限於二值化及圓偵測（circle detection）。處理裝置11可以依據所述查找表，判斷這些影像區塊分別對應於設置有指示元件141a的中央頂點及設置有指示元件141b～141d的側頂點。對應於中央頂點之影像區塊的中心點可以作為第一點，對應於側頂點之影像區塊的中心點可以組成第二點群。進一步來說，第一點及第二點群中的點各具有一二維座標表示其在二維影像擷取裝置12的影像平面座標系統中的位置。In step S102 , the processing device 11 obtains a plurality of first points corresponding to the central vertices and a plurality of second point groups corresponding to the plurality of side vertices from the plurality of 2D images. In an embodiment where each two-dimensional image includes image blocks corresponding to indicator elements 141a-141d of different colors or/and different temperatures, the processing device 11 may store a lookup table in advance, and the lookup table records the respective correspondences of the central vertex and the side vertex color and/or temperature. The processing device 11 can find image blocks with different colors and/or temperatures using an image processing algorithm, wherein the image processing algorithm includes, but is not limited to, binarization and circle detection. The processing device 11 can determine that these image blocks respectively correspond to the central vertex with the indicator 141 a and the side vertexes with the indicator 141 b - 141 d according to the lookup table. The center point of the image block corresponding to the central vertex can be used as the first point, and the center points of the image blocks corresponding to the side vertices can form the second point group. Furthermore, each of the points in the first point and the second point group has a two-dimensional coordinate representing its position in the image plane coordinate system of the two-dimensional image capture device 12 .

於指示元件141a～141d在每次拍攝程序中被依特定次序致能的實施態樣中，處理裝置11可以預先儲存所述特定次序。處理裝置11可以影像處理演算法找到具有顏色或/及溫度的影像區塊，其中影像處理演算法例如包含但不限於二值化及圓偵測（circle detection）。處理裝置11可以依據二維影像的產生時間及預存特定次序來判斷二維影像中的這些影像區塊係對應於中央頂點及側頂點中的何者。對應於中央頂點之影像區塊的中心點可以作為第一點，對應於側頂點之影像區塊的中心點可以組成第二點群。In an embodiment where the indicating elements 141 a - 141 d are activated in a specific order in each shooting procedure, the processing device 11 may store the specific order in advance. The processing device 11 can find image blocks with color and/or temperature using image processing algorithms, wherein the image processing algorithms include but not limited to binarization and circle detection. The processing device 11 can determine which of the central vertex and the side vertices the image blocks in the 2D image correspond to according to the generation time of the 2D image and the pre-stored specific sequence. The center point of the image block corresponding to the central vertex can be used as the first point, and the center points of the image blocks corresponding to the side vertices can form the second point group.

於步驟S103中，處理裝置11從所述多張三維影像取得對應於中央頂點的多個第三點。進一步來說，所述多個第三點對與所述多張三維影像具有一對一的關係，且各自具有一三維座標表示其在第二三維座標系統中的位置。處理裝置11可以將每張三維影像作為目標影像，執行：從目標影像取得三個平面，所述三個平面彼此相鄰且具有彼此垂直的法向量；以及取得所述三個平面的交點，作為所述多個第三點中的對應者。更進一步來說，處理裝置11可以從目標影像中找出所有平面，找出所有由彼此相鄰的三個平面組成的組合，計算各組合中的平面的法向量，過濾出組合中平面的法向量彼此垂直的組合，並計算該組合中的平面相交之點，作為所述多個第三點中的對應者。In step S103 , the processing device 11 acquires a plurality of third points corresponding to the central vertex from the plurality of 3D images. Further, the plurality of third point pairs has a one-to-one relationship with the plurality of 3D images, and each has a 3D coordinate indicating its position in the second 3D coordinate system. The processing device 11 may use each three-dimensional image as the target image, and execute: obtain three planes from the target image, the three planes are adjacent to each other and have normal vectors perpendicular to each other; and obtain the intersection point of the three planes as A corresponding one of the plurality of third points. Furthermore, the processing device 11 can find all the planes from the target image, find all the combinations of three adjacent planes, calculate the normal vectors of the planes in each combination, and filter out the normal vectors of the planes in the combination. The vectors are combined perpendicular to each other, and the points where the planes in the combination intersect are calculated as the corresponding ones of the plurality of third points.

於步驟S104中，處理裝置11利用待解第一轉換矩陣、所述多個第一點及所述多個第三點，基於第一三維座標系統執行第一最佳化運算，以取得最佳化第一轉換矩陣。進一步來說，所述多個第一點及所述多個第三點分別對應於前述多個校正位置。處理裝置11可以對每一校正位置的對應第一點及對應第三點執行距離計算作業，以取得多個計算結果，所述多個計算結果亦分別對應於所述多個校正位置。In step S104, the processing device 11 uses the first transformation matrix to be solved, the plurality of first points, and the plurality of third points to perform a first optimization operation based on the first three-dimensional coordinate system to obtain an optimal to transform the first transformation matrix. Further, the plurality of first points and the plurality of third points respectively correspond to the aforementioned plurality of calibration positions. The processing device 11 may perform a distance calculation operation on the corresponding first point and the corresponding third point of each correction position to obtain a plurality of calculation results, and the plurality of calculation results are also respectively corresponding to the plurality of correction positions.

請一併參考圖1及圖4以進一步說明距離計算作業，其中圖4係依據本發明一實施例所繪示的多模影像對位方法中的距離計算作業的執行示意圖。如圖4所示，第一點D1為二維影像SD中對應於校正體14的中央頂點的點，二維影像SD對應於第一三維座標系統CY1（二維影像擷取裝置12的相機座標系統），第三點D2為三維影像TD中對應於校正體14的中央頂點的點，三維影像TD對應於第二三維座標系統CY2（三維影像擷取裝置13的相機座標系統）。Please refer to FIG. 1 and FIG. 4 together to further explain the distance calculation operation, wherein FIG. 4 is a schematic diagram illustrating the execution of the distance calculation operation in the multi-mode image alignment method according to an embodiment of the present invention. As shown in FIG. 4 , the first point D1 is a point corresponding to the central vertex of the calibration body 14 in the two-dimensional image SD, and the two-dimensional image SD corresponds to the first three-dimensional coordinate system CY1 (the camera coordinates of the two-dimensional image capture device 12 system), the third point D2 is a point corresponding to the central vertex of the calibration body 14 in the 3D image TD, and the 3D image TD corresponds to the second 3D coordinate system CY2 (the camera coordinate system of the 3D image capture device 13 ).

在距離計算作業的執行過程中，處理裝置11可以取得連接於第一三維座標系統CY1的原點及第一點D1的射線L1，利用待解第一轉換矩陣轉換第三點D2，並計算經轉換的第三點D2與射線L1之間的距離d，以作為距離計算結果。處理裝置11可以透過一收斂函式（cost function）迭代調整該待解第一轉換矩陣，且將經迭代調整後的該待解第一轉換矩陣作為最佳化第一轉換矩陣，其中所述收斂函式指示使對應於所有校正位置的距離計算結果之和為最小值。所述收斂函式可以表示如式(1)：

(1) 其中 M ₁ 指示待解第一轉換矩陣， Pt3D指示第三點D2， Line指示射線L1。 During the execution of the distance calculation operation, the processing device 11 can obtain the ray L1 connected to the origin of the first three-dimensional coordinate system CY1 and the first point D1, use the first transformation matrix to be solved to transform the third point D2, and calculate the The converted distance d between the third point D2 and the ray L1 is used as the distance calculation result. The processing device 11 can iteratively adjust the first conversion matrix to be solved through a convergence function (cost function), and use the iteratively adjusted first conversion matrix to be solved as the optimized first conversion matrix, wherein the convergence The function instructs to minimize the sum of the distance calculation results corresponding to all corrected positions. The convergence function can be expressed as formula (1):

(1) Where M ₁ indicates the first transformation matrix to be solved, Pt3D indicates the third point D2, and Line indicates the ray L1.

特別來說，待解第一轉換矩陣可以包含旋轉矩陣及位移矩陣，其中旋轉矩陣關聯於三個軸向的角度參數，而位移矩陣關聯於三個軸向的位移參數。為了取得上述六個參數的解以取得最佳化第一轉換矩陣，校正位置的數量至少為六。旋轉矩陣及位移矩陣的詳細參數組成係本發明所屬領域中具有通常知識者能夠基於上述六個參數並依所需而設計者，本發明不予限制。Specifically, the first conversion matrix to be solved may include a rotation matrix and a displacement matrix, wherein the rotation matrix is associated with angle parameters of three axes, and the displacement matrix is associated with displacement parameters of three axes. In order to obtain the solutions of the above six parameters to obtain the optimized first transformation matrix, the number of calibration positions is at least six. The detailed parameter composition of the rotation matrix and the displacement matrix can be designed by those with ordinary knowledge in the field of the present invention based on the above six parameters and according to requirements, and the present invention is not limited thereto.

於圖3的步驟S105中，處理裝置11利用最佳化第一轉換矩陣轉換所述多張三維影像，且於步驟S106中，處理裝置11利用經最佳化第一轉換矩陣轉換的多張三維影像（一次轉換影像）、所述多個第一點、所述多個第二點群及校正體的預設規格參數組，基於第一三維座標系統執行第二最佳化運算，以取得最佳化第二轉換矩陣。理想上，步驟S104所得的最佳化第一轉換矩陣應使對應於所有校正位置的距離計算結果之和趨近於0，然實際上最佳化第一轉換矩陣為近似解。因此，經最佳化第一轉換矩陣轉換的三維影像所屬的座標系統與第一三維座標系統尚有所差異。藉由步驟S106所得之最佳化第二轉換矩陣再轉換經最佳化第一轉換矩陣轉換的三維影像，可以使得三維影像所對應的座標系統更接近第一三維座標系統。In step S105 of FIG. 3 , the processing device 11 transforms the multiple 3D images using the optimized first transformation matrix, and in step S106, the processing device 11 transforms the multiple 3D images using the optimized first transformation matrix The image (primarily converted image), the plurality of first points, the plurality of second point groups, and the preset specification parameter set of the calibration body, perform a second optimization operation based on the first three-dimensional coordinate system to obtain the optimum Optimizing the second transformation matrix. Ideally, the optimized first transformation matrix obtained in step S104 should make the sum of distance calculation results corresponding to all corrected positions close to 0, but actually the optimized first transformation matrix is an approximate solution. Therefore, the coordinate system of the 3D image transformed by the optimized first transformation matrix is still different from the first 3D coordinate system. Using the optimized second transformation matrix obtained in step S106 to transform the 3D image converted by the optimized first transformation matrix, the coordinate system corresponding to the 3D image can be closer to the first 3D coordinate system.

請參考圖1及圖5，其中圖5係依據本發明一實施例所繪示的多模影像對位方法中的第二最佳化運算的流程圖。第二最佳化運算可以包含步驟S601：利用待解第二轉換矩陣處理所述多張一次轉換影像，以分別產生多張二次轉換影像；步驟S602：依據所述多個第一點及所述多個第二點群，分別從所述多個二次轉換影像取得多個第四點群；步驟S603：依據所述多個第四點群，分別取得所述多個估算規格參數組；以及步驟S604：透過收斂函式迭代調整待解第二轉換矩陣，且將經迭代調整的待解第二轉換矩陣作為最佳化第二轉換矩陣，其中收斂函式指示使所述多個估算規格參數組與校正體的預設規格參數組間之差異為最小值。Please refer to FIG. 1 and FIG. 5 , wherein FIG. 5 is a flow chart of the second optimization operation in the multi-mode image alignment method according to an embodiment of the present invention. The second optimization operation may include step S601: using the second transformation matrix to be solved to process the multiple primary transformation images to generate multiple secondary transformation images respectively; step S602: according to the multiple first points and the According to the plurality of second point groups, respectively obtain a plurality of fourth point groups from the plurality of secondary transformed images; Step S603: Obtain the plurality of estimation specification parameter groups respectively according to the plurality of fourth point groups; And step S604: Iteratively adjust the second transformation matrix to be solved through the convergence function, and use the iteratively adjusted second transformation matrix to be solved as the optimized second transformation matrix, wherein the convergence function indicates that the plurality of estimation specifications The difference between the parameter set and the preset specification parameter set of the calibration body is minimal.

進一步來說，所述多個二次轉換影像、所述多個第一點、所述多個第二點群、所述多個第四點群及所述多個估算規格參數組可以分別對應於前述多個校正位置。於步驟S602及S603中，處理裝置11可以對每一校正位置的對應第一點、對應第二點群及對應二次轉換影像進行投影作業以取得對應第四點群，再依據對應第四點群進行規格參數估算作業取得對應估算規格參數組。Further, the plurality of reconverted images, the plurality of first points, the plurality of second point groups, the plurality of fourth point groups and the plurality of estimated specification parameter groups may respectively correspond to In the aforementioned multiple calibration positions. In steps S602 and S603, the processing device 11 can perform projection operations on the corresponding first point, the corresponding second point group and the corresponding secondary conversion image of each corrected position to obtain the corresponding fourth point group, and then according to the corresponding fourth point The group performs the specification parameter estimation operation to obtain the corresponding estimated specification parameter group.

請一併參考圖1及圖6以進一步說明投影作業，其中圖6係依據本發明一實施例所繪示的多模影像對位方法中的投影作業的執行示意圖。如圖6所示，第一點D1為二維影像SD中對應於校正體14的中央頂點的點，第二點D31～D33為二維影像SD中對應於校正體14的側頂點的點且可以組成第二點群，二維影像SD對應於第一三維座標系統CY1，經最佳化第一轉換矩陣及待解第二轉換矩陣轉換的三維影像（二次轉換影像TD’）對應於座標系統CY2’。Please refer to FIG. 1 and FIG. 6 together to further explain the projection operation, wherein FIG. 6 is a schematic diagram of execution of the projection operation in the multi-mode image alignment method according to an embodiment of the present invention. As shown in FIG. 6, the first point D1 is a point corresponding to the central vertex of the calibration body 14 in the two-dimensional image SD, and the second points D31-D33 are points corresponding to the side vertices of the calibration body 14 in the two-dimensional image SD. The second point group can be formed, the two-dimensional image SD corresponds to the first three-dimensional coordinate system CY1, and the three-dimensional image converted by the optimized first transformation matrix and the second transformation matrix to be solved (secondary transformation image TD') corresponds to the coordinates System CY2'.

在投影作業的執行過程中，處理裝置11可以將第一點D1投影至二次轉換影像TD’，以取得對應於第一點D1的第五點D1’，且將第二點D1～D33投影至二次轉換影像TD’，以取得分別對應於第二點D31～D33的多個第六點D31’～D33’。以另個角度來說，處理裝置11可以將二次轉換影像TD’中具有第一點D1的x座標及y座標的點作為第五點D1’，且可以將二次轉換影像TD’中具有第二點D31～D33的x座標及y座標的點作為第六點D31’～D33’。其中，第五點D1’及第六點D31’～D33’可以組成第四點群。對應於其他校正位置的第四點群的取得方式皆同理於上述，便不予贅述。During the execution of the projection operation, the processing device 11 may project the first point D1 to the secondary transformation image TD' to obtain the fifth point D1' corresponding to the first point D1, and project the second points D1-D33 Transform the image TD' twice to obtain a plurality of sixth points D31'-D33' respectively corresponding to the second points D31-D33. From another point of view, the processing device 11 may use the point having the x-coordinate and y-coordinate of the first point D1 in the twice-transformed image TD' as the fifth point D1', and may use the point in the twice-transformed image TD' as the fifth point D1'. The points of the x-coordinates and y-coordinates of the second points D31-D33 are used as the sixth points D31'-D33'. Wherein, the fifth point D1' and the sixth points D31'-D33' can form the fourth point group. The methods for obtaining the fourth point group corresponding to other corrected positions are the same as those described above, and will not be repeated here.

請一併參考圖1、圖5及圖7以進一步說明規格參數估算作業，圖7係依據本發明一實施例所繪示的多模影像對位方法中的規格參數估算作業的執行示意圖。在規格參數估算作業的執行過程中，處理裝置11取得第五點D1’與第六點D31’～D33’的多個連線，且計算所述多個連線的估算長度及連線之間的估算夾角，其中估算長度及估算夾角組成估算規格參數組。圖7示例性地標示第五點D1’與第六點D31’之連線的估算長度E及第五點D1’與第六點D31’之連線與第五點D1’與第六點D33’之連線的夾角A。Please refer to FIG. 1 , FIG. 5 , and FIG. 7 to further illustrate the standard parameter estimation operation. FIG. 7 is a schematic diagram of the standard parameter estimation operation in the multi-mode image alignment method according to an embodiment of the present invention. During the execution of the specification parameter estimating operation, the processing device 11 obtains multiple connecting lines between the fifth point D1' and the sixth points D31'-D33', and calculates the estimated lengths of the multiple connecting lines and the distance between the connecting lines. The estimated included angle of , where the estimated length and estimated included angle form an estimated specification parameter group. Fig. 7 exemplarily marks the estimated length E of the line connecting the fifth point D1' and the sixth point D31' and the line connecting the fifth point D1' and the sixth point D31' and the fifth point D1' and the sixth point D33 'The angle A of the connecting line.

以下進一步說明步驟S604。處理裝置11可以透過一收斂函式迭代調整待解第二轉換矩陣，且將經迭代調整的待解第二轉換矩陣作為最佳化第二轉換矩陣，其中收斂函式指示使所述多個估算規格參數組與校正體14的預設規格參數組間之差異為最小值。第二次最佳化運算所用之收斂函式異於第一次最佳化運算所用之收斂函式。校正體14的預設規格參數組可以包含校正體14的多個預設邊長及多個預設夾角。特別來說，圖7示例性地呈現第五點D1’及第六點D31’～D33’的理想位置I1～I4，所述多個預設邊長可以為理想位置I1～I4之間的多個連線的邊長S，所述多個預設夾角可以為理想位置I1～I4之間的連線的夾角R。Step S604 is further described below. The processing device 11 may iteratively adjust the second transformation matrix to be solved through a convergence function, and use the iteratively adjusted second transformation matrix to be solved as the optimized second transformation matrix, wherein the convergence function indicates that the plurality of estimated The difference between the specification parameter set and the preset specification parameter set of the calibration body 14 is a minimum value. The convergence function used for the second optimization operation is different from the convergence function used for the first optimization operation. The preset specification parameter set of the calibration body 14 may include multiple preset side lengths and multiple preset included angles of the calibration body 14 . In particular, FIG. 7 exemplarily presents the ideal positions I1-I4 of the fifth point D1' and the sixth points D31'-D33', and the multiple preset side lengths can be as many as between the ideal positions I1-I4. The side length S of the connecting line, the plurality of preset included angles may be the included angle R of the connecting line between the ideal positions I1-I4.

步驟S604所述的估算規格參數組與預設規格參數組之間的差異可以指示第一數值與第二數值的加權和，其中第一數值指示所述多個估算長度分別與所述多個預設邊長的多個差值之和，且第二數值指示所述多個估算夾角分別與所述多個預設夾角的多個差值之和。步驟S604之收斂函式可以表示如式(2)：

(2) 其中 S ₁ ～ S ₃ 指示預設邊長， R ₁ ～ R ₃ 指示預設夾角， E ₁ ～ E ₃ 指示估算長度， A ₁ ～ A ₃ 指示估算夾角， α及 β指示可依所需調整之權重。 The difference between the estimated specification parameter set and the preset specification parameter set in step S604 may indicate a weighted sum of a first value and a second value, wherein the first value indicates that the multiple estimated lengths are respectively related to the multiple preset lengths. A sum of a plurality of difference values of side lengths is set, and a second value indicates a sum of a plurality of difference values of the plurality of estimated included angles and the plurality of preset included angles respectively. The convergence function of step S604 can be expressed as formula (2):

(2) Among them, S ₁ ~ S ₃ indicate the preset side length, R ₁ ~ R ₃ indicate the preset included angle, E ₁ ~ E ₃ indicate the estimated length, A ₁ ~ A ₃ indicate the estimated included angle, α and β indicate the optional The weight to be adjusted.

特別來說，用於迭代產生最佳化第二轉換矩陣的待解第二轉換矩陣可以包含旋轉矩陣及位移矩陣，或可以僅包含旋轉矩陣。旋轉矩陣關聯於三個軸向的角度參數。位移矩陣關聯於三個軸向的位移參數。旋轉矩陣及位移矩陣的詳細參數組成係本發明所屬領域中具有通常知識者能夠基於上述六個參數並依所需而設計者，本發明不予限制。In particular, the second transformation matrix to be solved for iteratively generating the optimized second transformation matrix may include a rotation matrix and a displacement matrix, or may only include a rotation matrix. The rotation matrix is associated with the angle parameters of the three axes. The displacement matrix is associated with the displacement parameters of the three axes. The detailed parameter composition of the rotation matrix and the displacement matrix can be designed by those with ordinary knowledge in the field of the present invention based on the above six parameters and according to requirements, and the present invention is not limited thereto.

請再次參考圖1及圖3。經上述步驟取得最佳化第一轉換矩陣及最佳化第二轉換矩陣之後，於步驟S107中，處理裝置11可以利用最佳化第一轉換矩陣及最佳化第二轉換矩陣進行轉換，將待處理影像從第二三維座標系統轉換至第一三維座標系統，或從第一三維座標系統轉換至第二三維座標系統。如前所述，第一三維座標系統為二維影像擷取裝置12的相機座標系統。Please refer to Figure 1 and Figure 3 again. After obtaining the optimized first transformation matrix and the optimized second transformation matrix through the above steps, in step S107, the processing device 11 can use the optimized first transformation matrix and the optimized second transformation matrix to perform transformation, and the The image to be processed is converted from the second 3D coordinate system to the first 3D coordinate system, or from the first 3D coordinate system to the second 3D coordinate system. As mentioned above, the first 3D coordinate system is the camera coordinate system of the 2D image capture device 12 .

進一步地，在將代處理影像轉換至第一三維座標系統之後，處理裝置11更可以利用二維影像擷取裝置12的參數矩陣，使待處理影像從第一三維座標系統轉換至二維影像擷取裝置12的影像平面座標系統。參數矩陣包含二維影像擷取裝置12的焦距參數及投影中心參數。處理裝置11可以最佳化第一轉換矩陣、最佳化第二轉換矩陣及參數矩陣組成用於執行二維影像擷取裝置12的影像平面座標系統與三維影像擷取裝置13的相機座標系統（第二三維座標系統）間之轉換的座標轉換矩陣，並將其儲存於內部記憶體。處理裝置11可以利用此座標轉換矩陣，將三維影像擷取裝置13所產生的影像映射至二維影像擷取裝置12所產生的二維影像，以產生疊合影像。Further, after converting the processing image to the first three-dimensional coordinate system, the processing device 11 can use the parameter matrix of the two-dimensional image capture device 12 to convert the image to be processed from the first three-dimensional coordinate system to the two-dimensional image capture Get the image plane coordinate system of the device 12. The parameter matrix includes focal length parameters and projection center parameters of the 2D image capture device 12 . The processing device 11 can optimize the first transformation matrix, optimize the second transformation matrix and parameter matrix to form the coordinate system of the image plane for executing the two-dimensional image capture device 12 and the camera coordinate system of the three-dimensional image capture device 13 ( The coordinate conversion matrix for conversion between the second three-dimensional coordinate system) and store it in the internal memory. The processing device 11 can use the coordinate transformation matrix to map the image generated by the 3D image capture device 13 to the 2D image generated by the 2D image capture device 12 to generate a superimposed image.

舉例來說，上述二維影像擷取裝置12的影像平面座標系統與三維影像擷取裝置13的相機座標系統間之轉換可依式(3)及(4)來執行：

(3)

(4) 其中 X、 Y及 Z表示三維影像上的座標，M ₁表示最佳化第一轉換矩陣，M ₂表示最佳化第二轉換矩陣， f _x 及 f _y 表示二維影像擷取裝置12的焦距參數， c _x 及 c _y 表示二維影像擷取裝置12的投影中心參數，且 x _t 及 y _t 表示二維影像上的座標。 For example, the conversion between the image plane coordinate system of the above-mentioned 2D image capture device 12 and the camera coordinate system of the 3D image capture device 13 can be performed according to equations (3) and (4):

(3)

(4) where X , Y and Z represent the coordinates on the 3D image, M ₁ represents the optimized first transformation matrix, M ₂ represents the optimized second transformation matrix, f _x and f _y represent the two-dimensional image capture device The focal length parameters of 12, c _x and _cy represent the projection center parameters of the two-dimensional image capture device 12, and x _t and y _t represent the coordinates on the two-dimensional image.

上列多個實施例所述之多模影像對位系統及方法可應用於設置有多個影像擷取裝置的使用環境，請一併參考圖1及圖8，圖8係依據本發明一實施例所繪示的多模影像對位方法的使用環境示意圖。圖8中的影像擷取裝置A1～A6的組成可以為二維影像擷取裝置及三維影像擷取裝置間隔設置。影像擷取裝置A1～A6中的任二相鄰者可以作為前述多模影像對位系統1中的二維影像擷取裝置12及三維影像擷取裝置13。於此要特別說明的是，圖8僅示例性地繪示校正體14的擺放位置，其擺放規則如前列實施例所述，於此不予贅述。The multi-mode image alignment system and method described in the above-mentioned embodiments can be applied to the use environment provided with multiple image capture devices. Please refer to FIG. 1 and FIG. 8 together. FIG. 8 is an implementation according to the present invention A schematic diagram of the usage environment of the multi-mode image alignment method shown in the example. The image capture devices A1 - A6 in FIG. 8 can be composed of two-dimensional image capture devices and three-dimensional image capture devices arranged at intervals. Any two adjacent ones of the image capture devices A1 - A6 can serve as the 2D image capture device 12 and the 3D image capture device 13 in the aforementioned multi-mode image alignment system 1 . It should be particularly noted here that FIG. 8 only schematically shows the placement position of the calibration body 14 , and its placement rules are as described in the previous embodiments, and will not be repeated here.

藉由前列多個實施例所述之多模影像對位方法，處理裝置11可以取得影像擷取裝置A1～A6中的任二相鄰者的座標系統之間的轉換矩陣，再透過串疊轉換架構使影像擷取裝置A1～A6中的任一者轉換至與之相隔一個以上的影像擷取裝置的座標系統。By using the multi-mode image alignment method described in the previous embodiments, the processing device 11 can obtain the transformation matrix between the coordinate systems of any two adjacent image capture devices A1-A6, and then perform cascade transformation The structure enables any one of the image capture devices A1-A6 to switch to the coordinate system of the image capture device more than one away from it.

舉例來說，假設處理裝置11藉由多模影像對位方法取得影像擷取裝置A1與影像擷取裝置A2的座標系統之間的轉換矩陣M ₁₂，且取得影像擷取裝置A2與影像擷取裝置A3的座標系統之間的轉換矩陣M ₂₃，處理裝置11可以藉由式(5)來將影像擷取裝置A1所產生之影像座標轉換至影像擷取裝置A3的座標系統： P ₃=M ₂₃*M ₁₂*P ₁(5) 其中P1表示影像擷取裝置A1所產生之影像座標，P3表示所述影像座標轉換至影像擷取裝置A3的座標系統的座標。藉由上述串疊轉換架構，處理裝置11可以使所有影像擷取裝置A1～A6轉換至影像擷取裝置A1～A6中的特定者的座標系統。 For example, assume that the processing device 11 obtains the transformation matrix M ₁₂ between the coordinate systems of the image capture device A1 and the image capture device A2 through the multi-mode image alignment method, and obtains the image capture device A2 and the image capture device A2 The conversion matrix M ₂₃ between the coordinate systems of the device A3, the processing device 11 can convert the image coordinates generated by the image capture device A1 to the coordinate system of the image capture device A3 by using formula (5): P ₃ =M ₂₃ *M ₁₂ *P ₁ (5) wherein P1 represents the image coordinates generated by the image capture device A1, and P3 represents the coordinates of the image coordinates converted to the coordinate system of the image capture device A3. Through the above-mentioned cascade conversion architecture, the processing device 11 can convert all the image capture devices A1 - A6 to the coordinate system of a specific one of the image capture devices A1 - A6 .

於另一實施例中，多模影像對位系統可以包含三個影像擷取裝置且可以取得三個影像擷取裝置之座標系統間的座標轉換矩陣。請參考圖9，圖9示例性地繪示包含三個影像擷取裝置的多模影像對位系統1’的功能方塊圖。如圖9所示，多模影像對位系統1’包含處理裝置11、兩個二維影像擷取裝置12及15、三維影像擷取裝置13及校正體14。In another embodiment, the multi-mode image alignment system may include three image capture devices and may obtain coordinate transformation matrices between the coordinate systems of the three image capture devices. Please refer to FIG. 9 . FIG. 9 exemplarily shows a functional block diagram of a multi-mode image alignment system 1' including three image capture devices. As shown in FIG. 9 , the multi-mode image alignment system 1' includes a processing device 11, two two-dimensional image capture devices 12 and 15, a three-dimensional image capture device 13 and a calibration body 14.

相較於圖1所示的多模影像對位系統1，多模影像對位系統1’更包含二維影像擷取裝置15。二維影像擷取裝置15例如為可見光攝影機、近紅外光攝影機、熱像儀等，可以連接於處理裝置11，具有一相機座標系統及一影像平面座標系統，且可以受控以拍攝輪流擺放於不同校正位置的校正體14以產生多張二維影像。特別來說，二維影像擷取裝置15的相機座標系統及影像平面座標系統異於二維影像擷取裝置12的相機座標系統及影像平面座標系統。Compared with the multi-mode image alignment system 1 shown in FIG. 1 , the multi-mode image alignment system 1' further includes a two-dimensional image capture device 15. The two-dimensional image capture device 15 is, for example, a visible light camera, a near-infrared camera, a thermal imager, etc., which can be connected to the processing device 11, have a camera coordinate system and an image plane coordinate system, and can be controlled to take pictures and place them in turn The calibration body 14 at different calibration positions can generate multiple 2D images. Specifically, the camera coordinate system and the image plane coordinate system of the 2D image capture device 15 are different from the camera coordinate system and the image plane coordinate system of the 2D image capture device 12 .

多模影像對位系統1’的處理裝置11除了前列多個實施例所述的運作，更可以依據二維影像擷取裝置15所產生之關聯於校正體14的多張二維影像及前述三維影像擷取裝置13所產生之關聯於校正體14的多張三維影像，取得二維影像擷取裝置15的相機座標系統與三維影像擷取裝置13的相機座標系統之間的座標轉換矩陣。利用所述座標轉換矩陣及前列實施例所述之最佳化第一及第二轉換矩陣，處理裝置11可以進行二維影像擷取裝置12的相機座標系統與二維影像擷取裝置15的相機座標系統之間的轉換。利用所述座標轉換矩陣、前列實施例所述之最佳化第一及第二轉換矩陣、二維影像擷取裝置12的參數矩陣及二維影像擷取裝置15的參數矩陣，處理裝置11可以進行二維影像擷取裝置12的影像平面座標系統與二維影像擷取裝置15的影像平面座標系統之間的轉換。In addition to the operations described in the preceding embodiments, the processing device 11 of the multi-mode image alignment system 1' can also capture multiple 2D images associated with the calibration body 14 and the aforementioned 3D images generated by the 2D image capture device 15. The plurality of 3D images associated with the calibration body 14 generated by the capture device 13 is used to obtain a coordinate transformation matrix between the camera coordinate system of the 2D image capture device 15 and the camera coordinate system of the 3D image capture device 13 . Using the coordinate transformation matrix and the optimized first and second transformation matrices described in the preceding embodiments, the processing device 11 can perform the coordinate system of the camera of the two-dimensional image capture device 12 and the camera of the two-dimensional image capture device 15 Conversion between coordinate systems. Using the coordinate transformation matrix, the optimized first and second transformation matrices described in the preceding embodiments, the parameter matrix of the two-dimensional image capture device 12 and the parameter matrix of the two-dimensional image capture device 15, the processing device 11 can Perform conversion between the image plane coordinate system of the 2D image capture device 12 and the image plane coordinate system of the 2D image capture device 15 .

以下進一步說明適用於多模影像對位系統1’的多模影像對位方法。請一併參考圖3、圖9及圖10，其中圖10係依據本發明另一實施例所繪示的多模影像對位方法的流程圖。適用於多模影像對位系統1’的多模影像對位方法可以包含圖3所示的步驟S101～S107及圖9所示的步驟S108：取得關聯於校正體的多張第二二維影像，第二二維影像對應於第三三維座標系統；步驟S109：從所述多張第二二維影像取得對應於中央頂點的多個第七點及對應於所述多個側頂點的多個第八點群；步驟S110：利用待解第三轉換矩陣、所述多個第七點及所述多個第三點，基於第三三維座標系統執行第一最佳化運算，以取得最佳化第三轉換矩陣；步驟S111：利用最佳化第三轉換矩陣處理所述多張三維影像；步驟S112：利用經處理的三維影像、所述多個第七點、所述多個第八點群及校正體的預設規格參數組，基於第三三維座標系統執行第二最佳化運算，以取得最佳化第四轉換矩陣；以及步驟S113：利用最佳化第一轉換矩陣、最佳化第二轉換矩陣、最佳化第三轉換矩陣及最佳化第四轉換矩陣，執行第一三維座標系統與第三三維座標系統之間的轉換。於此要特別說明的是，本發明不限制步驟S101與S108的執行順序，不限制步驟S102、步驟S103與步驟S109的執行順序，亦不限制步驟S104～S106之組合與步驟S110～S112之組合的執行順序。The multi-mode image alignment method applicable to the multi-mode image alignment system 1' is further described below. Please refer to FIG. 3 , FIG. 9 and FIG. 10 together, wherein FIG. 10 is a flowchart of a multi-mode image alignment method according to another embodiment of the present invention. The multi-mode image alignment method applicable to the multi-mode image alignment system 1' may include steps S101-S107 shown in FIG. 3 and step S108 shown in FIG. 9: obtaining multiple second two-dimensional images associated with the calibration body , the second two-dimensional image corresponds to the third three-dimensional coordinate system; step S109: Obtain a plurality of seventh points corresponding to the central vertex and a plurality of seventh points corresponding to the plurality of side vertices from the plurality of second two-dimensional images The eighth point group; step S110: using the third transformation matrix to be solved, the plurality of seventh points and the plurality of third points, to perform a first optimization operation based on the third three-dimensional coordinate system to obtain an optimal Optimizing the third transformation matrix; step S111: using the optimized third transformation matrix to process the plurality of three-dimensional images; step S112: using the processed three-dimensional images, the plurality of seventh points, and the plurality of eighth points performing a second optimization operation based on the third three-dimensional coordinate system to obtain an optimized fourth transformation matrix; and step S113: using the optimized first transformation matrix, the optimal The second conversion matrix is optimized, the third conversion matrix is optimized, and the fourth conversion matrix is optimized to perform conversion between the first three-dimensional coordinate system and the third three-dimensional coordinate system. It should be noted here that the present invention does not limit the execution order of steps S101 and S108, does not limit the execution order of steps S102, S103 and step S109, and does not limit the combination of steps S104-S106 and the combination of steps S110-S112 order of execution.

步驟S101～步驟S113可以由多模影像對位系統1’的處理裝置11執行。步驟S108～S112的進一步實施方式分別同理於步驟S101、S102、S104～S106的進一步實施方式，換言之，S101、S102、S104～S106中是取得二維影像擷取裝置12之第一三維座標系統與三維影像擷取裝置13之第二三維座標系統間的轉換關係，而S108～S112則是基於相同的原理來取得二維影像擷取裝置15之第三三維座標系統與第二三維座標系統間的轉換關係，於此便不再贅述。於步驟S113中，處理裝置11可以利用最佳化第一轉換矩陣、最佳化第二轉換矩陣、最佳化第三轉換矩陣及最佳化第四轉換矩陣，進行第一三維座標系統與第三三維座標系統之間的轉換。進一步來說，最佳化第一轉換矩陣及最佳化第二轉換矩陣可以組成用於將待處理影像從第二三維座標系統轉換至第一三維座標系統的座標轉換矩陣M _a，最佳化第三轉換矩陣及最佳化第四轉換矩陣可以組成用於將將待處理影像從第二三維座標系統轉換至第三三維座標系統的座標轉換矩陣M _b。處理裝置11可以透過以座標轉換矩陣M _a及M _b進行代數運算而取得用於將待處理影像從第一三維座標系統轉換至第三三維座標系統的座標轉換矩陣M _c，例如表示為式(6)： M _c=M _b*M _a ^-1(6) Step S101 to step S113 can be executed by the processing device 11 of the multi-mode image alignment system 1 ′. Further implementations of steps S108-S112 are similar to further implementations of steps S101, S102, S104-S106 respectively. In other words, in S101, S102, S104-S106, the first three-dimensional coordinate system of the two-dimensional image capture device 12 is acquired and the conversion relationship between the second three-dimensional coordinate system of the three-dimensional image capture device 13, and S108-S112 are based on the same principle to obtain the relationship between the third three-dimensional coordinate system of the two-dimensional image capture device 15 and the second three-dimensional coordinate system The conversion relationship will not be repeated here. In step S113, the processing device 11 can use the optimized first transformation matrix, the optimized second transformation matrix, the optimized third transformation matrix and the optimized fourth transformation matrix to perform the first three-dimensional coordinate system and the second Conversion between 3D and 3D coordinate systems. Further, the optimized first transformation matrix and the optimized second transformation matrix can form a coordinate transformation matrix M _a for transforming the image to be processed from the second three-dimensional coordinate system to the first three-dimensional coordinate system, and the optimized The third transformation matrix and the optimized fourth transformation matrix can form a coordinate transformation matrix M _b for transforming the image to be processed from the second three-dimensional coordinate system to the third three-dimensional coordinate system. The processing device 11 can obtain the coordinate conversion matrix M _c for converting the image to be processed from the first three-dimensional coordinate system to the third three-dimensional coordinate system by performing algebraic operations with the coordinate conversion matrices M _a and M _b , for example expressed as the formula ( 6): M _c =M _b *M _a ^-1 (6)

更進一步來說，座標轉換矩陣M _a可以更包含二維影像擷取裝置12的參數矩陣，座標轉換矩陣M _b可以更包含二維影像擷取裝置15的參數矩陣。座標轉換矩陣M _a所包含的最佳化第一轉換矩陣、最佳化第二轉換矩陣及參數矩陣之間的關係如前列式(3)及(4)所示，座標轉換矩陣M _b所包含的最佳化第三轉換矩陣、最佳化第四轉換矩陣及參數矩陣之間的關係亦同，於此便不再贅述。於此實施態樣中，經上述代數運算所得之座標轉換矩陣M _c可用於進行二維影像擷取裝置12的影像平面座標系統與二維影像擷取裝置15的影像平面座標系統之間的轉換。 Furthermore, the coordinate transformation matrix M _a may further include a parameter matrix of the 2D image capture device 12 , and the coordinate transformation matrix M _b may further include a parameter matrix of the 2D image capture device 15 . The relationship between the optimized first conversion matrix included in the coordinate conversion matrix M _a , the optimized second conversion matrix and the parameter matrix is shown in the preceding formula (3) and (4), and the coordinate conversion matrix M _b includes The relationship among the optimized third transformation matrix, the optimized fourth transformation matrix and the parameter matrix is also the same, and will not be repeated here. In this embodiment, the coordinate conversion matrix M _c obtained through the above algebraic operations can be used to perform conversion between the image plane coordinate system of the two-dimensional image capture device 12 and the image plane coordinate system of the two-dimensional image capture device 15 .

利用座標轉換矩陣M _a、M _b及M _c，處理裝置11可以將二維影像擷取裝置12及15及三維影像擷取裝置13中任二者所產生的影像映射至剩餘者所產生的影像。舉例來說，利用座標轉換矩陣M _b及座標轉換矩陣M _c，處理裝置11可以將三維影像擷取裝置13所產生的影像及二維影像擷取裝置12所產生的影像映射至二維影像擷取裝置15所產生的影像，以產生疊合影像。如此一來，多模影像對位系統1’可以從疊合影像一次取得對應於特定標的物的三種資訊。舉例來說，若多模影像對位系統1’的三個影像擷取裝置分別為熱像儀、可見光相機及三維點雲感測器，則多模影像對位系統1’的處理裝置11可以從疊合影像上一次取得特定標的物的溫度、色彩及空間位置資訊。 Using the coordinate transformation matrices M _a , M _b and M _c , the processing device 11 can map the images generated by any two of the two-dimensional image capture devices 12 and 15 and the three-dimensional image capture device 13 to the images generated by the rest . For example, using the coordinate transformation matrix M _b and the coordinate transformation matrix M _c , the processing device 11 can map the image generated by the 3D image capture device 13 and the image generated by the 2D image capture device 12 to the 2D image capture The images generated by the device 15 are taken to generate superimposed images. In this way, the multi-mode image alignment system 1 ′ can acquire three kinds of information corresponding to a specific object from the superimposed image at one time. For example, if the three image capture devices of the multi-mode image alignment system 1' are a thermal imager, a visible light camera, and a three-dimensional point cloud sensor, the processing device 11 of the multi-mode image alignment system 1' can The temperature, color and spatial position information of a specific object is obtained from the last time of the stacked image.

雖然本發明以前述之實施例揭露如上，然其並非用以限定本發明。在不脫離本發明之精神和範圍內，所為之更動與潤飾，均屬本發明之專利保護範圍。關於本發明所界定之保護範圍請參考所附之申請專利範圍。Although the present invention is disclosed by the aforementioned embodiments, they are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, all changes and modifications are within the scope of patent protection of the present invention. For the scope of protection defined by the present invention, please refer to the appended scope of patent application.

1,1’:多模影像對位系統 11:處理裝置 12,15:二維影像擷取裝置 13:三維影像擷取裝置 14:校正體 140:立體本體 141a~141:指示元件 P1~P6:校正位置 v1,v2:視角 CY1:第一三維座標系統 CY2:第二三維座標系統 CY2’:座標系統 SD:二維影像 TD:三維影像 TD’:二次轉換影像 D1:第一點 D1’:第五點 D2:第三點 L1:射線 d:距離 D31~D33:第二點 D31’~D33’:第六點 E:估算長度 A:估算夾角 I1~I4:理想位置 S:邊長 R:夾角 A1~A6:影像擷取裝置 S101~S107:步驟 S601~S604:步驟 S108~S113:步驟 1,1': Multi-mode image alignment system 11: Processing device 12,15: Two-dimensional image capture device 13: Three-dimensional image capture device 14: Correction body 140: Three-dimensional ontology 141a~141: indicating elements P1~P6: Calibration position v1, v2: perspective CY1: The first three-dimensional coordinate system CY2: The second three-dimensional coordinate system CY2': coordinate system SD: Two-dimensional image TD: Three-dimensional video TD’: Twice transformed image D1: the first point D1': fifth point D2: The third point L1: ray d: distance D31~D33: The second point D31'~D33': Sixth point E: estimated length A: Estimate the included angle I1~I4: ideal position S: side length R: included angle A1~A6: Image capture device S101~S107: steps S601~S604: steps S108~S113: Steps

圖1係依據本發明一實施例所繪示的多模影像對位系統的功能方塊圖。圖2係依據本發明一實施例所繪示的校正位置示意圖。圖3係依據本發明一實施例所繪示的多模影像對位方法的流程圖。圖4係依據本發明一實施例所繪示的多模影像對位方法中的距離計算作業的執行示意圖。圖5係依據本發明一實施例所繪示的多模影像對位方法中的第二最佳化運算的流程圖。圖6係依據本發明一實施例所繪示的多模影像對位方法中的投影作業的執行示意圖。圖7係依據本發明一實施例所繪示的多模影像對位方法中的規格參數估算作業的執行示意圖。圖8係依據本發明一實施例所繪示的多模影像對位方法的使用環境示意圖。圖9係依據本發明另一實施例所繪示的多模影像對位系統的功能方塊圖。圖10係依據本發明另一實施例所繪示的多模影像對位方法的流程圖。 FIG. 1 is a functional block diagram of a multi-mode image alignment system according to an embodiment of the present invention. FIG. 2 is a schematic diagram of a calibration position according to an embodiment of the present invention. FIG. 3 is a flow chart of a multi-mode image alignment method according to an embodiment of the present invention. FIG. 4 is a schematic diagram illustrating the execution of a distance calculation operation in a multi-mode image alignment method according to an embodiment of the present invention. FIG. 5 is a flow chart of the second optimization operation in the multi-mode image alignment method according to an embodiment of the present invention. FIG. 6 is a schematic diagram illustrating execution of a projection operation in a multi-mode image alignment method according to an embodiment of the present invention. FIG. 7 is a schematic diagram of execution of specification parameter estimation in the multi-mode image alignment method according to an embodiment of the present invention. FIG. 8 is a schematic diagram of the usage environment of the multi-mode image alignment method according to an embodiment of the present invention. FIG. 9 is a functional block diagram of a multi-mode image alignment system according to another embodiment of the present invention. FIG. 10 is a flowchart of a multi-mode image alignment method according to another embodiment of the present invention.

S101~S107:步驟 S101~S107: steps

Claims

一種多模影像對位方法，包含以一處理裝置執行：取得關聯於一校正體的多張二維影像及多張三維影像，該校正體包含一立體本體，具有一中央頂點及多個側頂點，該些二維影像關聯於一第一三維座標系統，且該些三維影像關聯於一第二三維座標系統；從該些二維影像取得對應於該中央頂點的多個第一點及對應於該些側頂點的多個第二點群；從該些三維影像取得對應於該中央頂點的多個第三點；利用一待解第一轉換矩陣、該些第一點及該些第三點，基於該第一三維座標系統執行一第一最佳化運算，以取得一最佳化第一轉換矩陣，其中該第一最佳化運算包含透過一第一收斂函式迭代調整該待解第一轉換矩陣，以取得該最佳化第一轉換矩陣；利用該最佳化第一轉換矩陣處理該些三維影像，以分別產生多張一次轉換影像；利用該些一次轉換影像、該些第一點、該些第二點群以及該校正體的一預設規格參數組，基於該第一三維座標系統執行一第二最佳化運算，以取得一最佳化第二轉換矩陣，其中該第二最佳化運算包含透過一第二收斂函式迭代調整一待解第二轉換矩陣，以取得該最佳化第二轉換矩陣；以及利用該最佳化第一轉換矩陣及該最佳化第二轉換矩陣執行轉換運算，以將一待處理影像轉換至該第二三維座標系統或該第一三維座標系統。 A method for aligning multi-mode images, including executing with a processing device: obtaining multiple two-dimensional images and multiple three-dimensional images associated with a calibration body, the calibration body includes a three-dimensional body with a central vertex and multiple side vertices, the The 2D images are associated with a first 3D coordinate system, and the 3D images are associated with a second 3D coordinate system; a plurality of first points corresponding to the central vertex and corresponding to the 2D images are obtained from the 2D images a plurality of second point groups of side vertices; obtain a plurality of third points corresponding to the central vertex from the three-dimensional images; use a first transformation matrix to be solved, the first points and the third points, based on Performing a first optimization operation on the first three-dimensional coordinate system to obtain an optimized first transformation matrix, wherein the first optimization operation includes iteratively adjusting the first transformation to be solved through a first convergence function matrix to obtain the optimized first transformation matrix; use the optimized first transformation matrix to process the three-dimensional images to generate multiple primary transformation images respectively; use the primary transformation images, the first points, The second point groups and a preset specification parameter set of the calibration body perform a second optimization operation based on the first three-dimensional coordinate system to obtain an optimized second transformation matrix, wherein the second optimal The optimization operation includes iteratively adjusting a second transformation matrix to be solved through a second convergence function to obtain the optimized second transformation matrix; and A transformation operation is performed by using the optimized first transformation matrix and the optimized second transformation matrix to transform an image to be processed into the second three-dimensional coordinate system or the first three-dimensional coordinate system.

如請求項1所述的多模影像對位方法，其中該些第一點與該些第三點分別對應於多個校正位置，該第一最佳化運算更包含：對於每一該些校正位置的對應第一點及對應第三點執行距離計算，以取得多個分別對應於該些校正位置的多個計算結果，其中該距離計算包含：取得連接該第一三維座標系統之原點與該對應第一點的一射線；利用該待解第一轉換矩陣轉換該對應第三點；以及計算經轉換的該對應第三點與該射線之間的距離；以及將經迭代調整的該待解第一轉換矩陣作為該最佳化第一轉換矩陣，其中該第一收斂函式指示使該些計算結果之和為最小值。 The multi-mode image alignment method as described in claim 1, wherein the first points and the third points respectively correspond to a plurality of correction positions, and the first optimization operation further includes: for each of the corrections Performing distance calculations corresponding to the first point and the third point of the position to obtain a plurality of calculation results respectively corresponding to the corrected positions, wherein the distance calculation includes: obtaining the origin and connecting the first three-dimensional coordinate system A ray corresponding to the first point; transforming the corresponding third point by using the first transformation matrix to be solved; and calculating the distance between the transformed corresponding third point and the ray; and iteratively adjusting the to-be-resolved Solving the first transformation matrix as the optimized first transformation matrix, wherein the first convergence function indicates that the sum of the calculation results is a minimum value.

如請求項1所述的多模影像對位方法，其中該第二最佳化運算更包含：利用該待解第二轉換矩陣處理該些一次轉換影像以分別產生多張二次轉換影像；依據該些第一點及該些第二點群，分別從該些二次轉換影像取得多個第四點群；依據該些第四點群，分別取得多個估算規格參數組；以及將經迭代調整的該待解第二轉換矩陣作為該最佳化第二轉換矩陣，其中該第二收斂函式指示使該些估算規格參數組與該校正體的該預設規格參數組間之差異為最小值。 The multi-mode image alignment method as described in claim 1, wherein the second optimization operation further includes: using the second transformation matrix to be solved to process the primary transformation images to generate multiple secondary transformation images respectively; A plurality of fourth point groups are respectively obtained from the secondary transformed images for the first points and the second point groups; a plurality of estimation specification parameter sets are respectively obtained according to the fourth point groups; and The iteratively adjusted second transformation matrix to be solved is used as the optimized second transformation matrix, wherein the second convergence function indicates the relationship between the estimated specification parameter sets and the preset specification parameter set of the calibration body Differences are minimal.

如請求項3所述的多模影像對位方法，其中該些二次轉換影像、該些第一點、該些第二點群分別對應於多個校正位置，且依據該些第一點及該些第二點群，分別從該些二次轉換影像取得多個第四點群包含：對於每一該些校正位置的對應第一點、對應第二點群及對應二次轉換影像執行：將該對應第一點投影至該對應二次轉換影像，以取得一第五點；以及將該第二點群中的多個點投影至該對應二次轉換影像，以取得多個第六點；其中該第五點及該些第六點成該些第四點群之一。 The multi-mode image alignment method as described in claim 3, wherein the secondary converted images, the first points, and the second point groups respectively correspond to multiple correction positions, and are based on the first points and For the second point groups, obtaining a plurality of fourth point groups from the secondary transformed images respectively includes: for each of the corrected positions corresponding to the first point, corresponding to the second point group and corresponding to the secondary transformed image, perform: projecting the corresponding first point to the corresponding twice transformed image to obtain a fifth point; and projecting the points in the second point group to the corresponding twice transformed image to obtain sixth points ; Wherein the fifth point and the sixth points become one of the fourth point groups.

如請求項4所述的多模影像對位方法，其中依據該些第四點群，分別取得該些估算規格參數組包含：對於每一該些第四點群，執行一規格參數估算以取得該些估算規格參數組之一，其中該規格參數估算包含：取得該第五點與該些第六點的多個連線；以及計算該些連線的多個估算長度及該些連線之間的多個估算夾角；其中該預設規格參數包含該校正體的多個預設邊長及多個預設夾角，該些估算規格參數組與該預設規格參數組之間的該差異指示一第一數值與一第二數值的加權和，該第一數值指示該些估算長度分別與該些預設邊長的多個差值之和，且該第二數值指示該些估算夾角分別與該些預設夾角的多個差值之和。 The multi-mode image alignment method as described in claim 4, wherein according to the fourth point groups, obtaining the estimated specification parameter sets respectively includes: for each of the fourth point groups, performing a specification parameter estimation to obtain One of the estimated specification parameter sets, wherein the specification parameter estimation includes: obtaining multiple connecting lines between the fifth point and the sixth points; and calculating multiple estimated lengths of the connecting lines and the multiple estimated angles between; Wherein the preset specification parameters include a plurality of preset side lengths and a plurality of preset included angles of the calibration body, the difference between the estimated specification parameter sets and the preset specification parameter sets indicates a first value and a first value A weighted sum of two values, the first value indicates the sum of the differences between the estimated lengths and the preset side lengths, and the second value indicates the difference between the estimated included angles and the preset included angles sum of the differences.

如請求項1所述的多模影像對位方法，其中從該些三維影像取得對應於該中央頂點的多個第三點包含：將每一該些三維影像作為一目標影像，執行：從該目標影像取得三個平面，該些平面彼此相鄰且分別具有之三個法向量彼此垂直；以及取得該些平面的交點，作為該些第三點中的一者。 The multi-mode image alignment method as described in claim 1, wherein obtaining a plurality of third points corresponding to the central vertex from the three-dimensional images includes: using each of the three-dimensional images as a target image, and executing: from the three-dimensional images The target image obtains three planes, the planes are adjacent to each other and have three normal vectors perpendicular to each other; and the intersection point of the planes is obtained as one of the third points.

如請求項1所述的多模影像對位方法，其中該些二維影像係多張第一二維影像，且該多模影像對位方法更包含以該處理裝置執行：取得關聯於該校正體的多張第二二維影像，該些第二二維影像對應於一第三三維座標系統；從該些第二二維影像取得對應於該中央頂點的多個第七點及對應於該些側頂點的多個第八點群；利用一待解第三轉換矩陣、該些第七點及該些第三點，基於該第三三維座標系統執行該第一最佳化運算，以取得一最佳化第三轉換矩陣；利用該最佳化第三轉換矩陣處理該些三維影像；利用經處理的該些三維影像、該些第七點、該些第八點群及該校正體的該預設規格參數組，基於該第三三維座標系統執行該第二最佳化運算，以取得一最佳化第四轉換矩陣；以及利用該最佳化第一轉換矩陣、該最佳化第二轉換矩陣、該最佳化第三轉換矩陣及該最佳化第四轉換矩陣，進行該第一三維座標系統與該第三三維座標系統之間的轉換。 The multi-mode image alignment method as described in claim 1, wherein the two-dimensional images are a plurality of first two-dimensional images, and the multi-mode image alignment method further includes executing by the processing device: obtaining the correction associated with the A plurality of second two-dimensional images of the volume, the second two-dimensional images correspond to a third three-dimensional coordinate system; from the second two-dimensional images, a plurality of seventh points corresponding to the central vertex and corresponding to the a plurality of eighth point groups of the side vertices; using a third transformation matrix to be solved, the seventh points and the third points, the first optimization operation is performed based on the third three-dimensional coordinate system to obtain an optimized third transformation matrix; using the optimized third transformation matrix to process the 3D images; Using the processed 3D images, the seventh points, the eighth point groups, and the preset specification parameter set of the calibration body, the second optimization operation is performed based on the third 3D coordinate system, so as to obtaining an optimized fourth transformation matrix; and using the optimized first transformation matrix, the optimized second transformation matrix, the optimized third transformation matrix, and the optimized fourth transformation matrix to perform the Transformation between the first three-dimensional coordinate system and the third three-dimensional coordinate system.

如請求項1所述的多模影像對位方法，其中該第一三維座標系統係一二維影像擷取裝置的相機座標系統，且該多模影像對位方法更包含以該處理裝置執行：利用該二維影像擷取裝置的一參數矩陣，使該待處理影像從該第一三維座標系統轉換至該二維影像擷取裝置的影像平面座標系統；其中該參數矩陣包含該二維影像擷取裝置的焦距參數及投影中心參數。 The multi-mode image alignment method as described in claim 1, wherein the first three-dimensional coordinate system is a camera coordinate system of a two-dimensional image capture device, and the multi-mode image alignment method further includes executing by the processing device: Using a parameter matrix of the 2D image capture device, the image to be processed is converted from the first 3D coordinate system to the image plane coordinate system of the 2D image capture device; wherein the parameter matrix includes the 2D image capture Take the focal length parameter and the projection center parameter of the device.

如請求項1所述的多模影像對位方法，其中該校正體包含多個指示元件，分別設置於該中央頂點及該些側頂點，且該多模影像對位方法更包含：以一二維影像擷取裝置對輪流擺放於多個校正位置的該校正體分別進行多次拍攝程序，且於每一該些拍攝程序中輪流致能該些指示元件並拍攝該校正體，以產生該些二維影像；以及以一三維影像擷取裝置拍攝輪流擺放於該些校正位置的該校正體以產生該些三維影像。 The multi-mode image alignment method as described in claim 1, wherein the correction body includes a plurality of indicating elements, which are respectively arranged on the central vertex and the side vertices, and the multi-mode image alignment method further includes: using one or two The three-dimensional image capture device performs multiple shooting procedures on the calibration body placed in multiple calibration positions in turn, and in each of the shooting procedures, the indicating elements are enabled in turn and the calibration body is photographed, so as to generate the the two-dimensional images; and a three-dimensional image capturing device is used to capture the calibration objects placed in the calibration positions in turn to generate the three-dimensional images.

如請求項1所述的多模影像對位方法，其中該校正體更包含多個指示元件，分別設置於該中央頂點及該些側頂點且分別具有不同顏色或不同溫度，且該多模影像對位方法更包含：以一二維影像擷取裝置拍攝輪流擺放於多個校正位置的該校正體以產生該些二維影像；以及以一三維影像擷取裝置拍攝輪流擺放於該些校正位置的該校正體以產生該些三維影像。 The multi-mode image alignment method as described in claim 1, wherein the correction body further includes a plurality of indicating elements, which are respectively arranged on the central vertex and the side vertices and have different colors or different temperatures, and the multi-mode image The alignment method further includes: using a two-dimensional image capture device to photograph the calibration body placed in a plurality of calibration positions in turn to generate the two-dimensional images; and using a three-dimensional image capture device to capture The calibration volume is corrected in position to generate the 3D images.

一種多模影像對位系統，包含：一校正體，包含：一立體本體，具有一中央頂點及多個側頂點；以及多個指示元件，分別設置於該中央頂點及該些側頂點；一二維影像擷取裝置，具有一第一三維座標系統，且用於產生關聯於該校正體的多張二維影像；一三維影像擷取裝置，具有一第二三維座標系統，且用於產生關聯於該校正體的多張三維影像；以及一處理裝置，連接於該二維影像擷取裝置及該三維影像擷取裝置，用於依據該些二維影像及該些三維影像及該校正體的一預設規格參數組執行運算，取得一座標轉換矩陣，且利用該座標轉換矩陣將一待處理影像轉換至該第二三維座標系統或該第一三維座標系統。 A multi-mode image alignment system, comprising: a correction body, including: a three-dimensional body having a central vertex and multiple side vertices; and a plurality of indicating elements respectively arranged on the central vertex and the side vertices; A three-dimensional image capture device has a first three-dimensional coordinate system and is used to generate multiple two-dimensional images associated with the calibration body; a three-dimensional image capture device has a second three-dimensional coordinate system and is used to generate multiple two-dimensional images associated with the calibration body A plurality of three-dimensional images of the calibration body; and a processing device connected to the two-dimensional image capture device and the three-dimensional image capture device, for a prediction based on the two-dimensional images and the three-dimensional images and the calibration volume The specification parameter set is used to perform calculations to obtain a coordinate transformation matrix, and a to-be-processed image is converted to the second three-dimensional coordinate system or the first three-dimensional coordinate system by using the coordinate transformation matrix.

如請求項11所述的多模影像對位系統，更包含：另一二維影像擷取裝置，連接於該處理裝置，具有一第三三維座標系統，且用於產生關聯於該校正體的多張第二二維影像；其中該處理裝置更用於依據該些第二二維影像及該些三維影像，取得另一座標轉換矩陣，且利用該二座標轉換矩陣進行該第一三維座標系統與該第三三維座標系統之間的轉換。 The multi-mode image alignment system as described in claim item 11 further includes: Another two-dimensional image capture device, connected to the processing device, has a third three-dimensional coordinate system, and is used to generate a plurality of second two-dimensional images associated with the calibration body; wherein the processing device is further used for according to the For the second 2D images and the 3D images, another coordinate transformation matrix is obtained, and the conversion between the first 3D coordinate system and the third 3D coordinate system is performed using the two coordinate transformation matrix.

如請求項11所述的多模影像對位系統，其中該些指示元件分別具有不同顏色或不同溫度。 The multi-mode image alignment system according to claim 11, wherein the indicating elements have different colors or different temperatures.

如請求項11所述的多模影像對位系統，其中該處理裝置所執行之取得該座標轉換矩陣包含：從該些二維影像取得對應於該中央頂點的多個第一點及對應於該些側頂點的多個第二點群；從該些三維影像取得對應於該中央頂點的多個第三點；利用一待解第一轉換矩陣、該些第一點及該些第三點，基於該第一三維座標系統執行第一最佳化運算，以取得一最佳化第一轉換矩陣，其中該第一最佳化運算包含透過一第一收斂函式迭代調整該待解第一轉換矩陣，以取得該最佳化第一轉換矩陣；利用該最佳化第一轉換矩陣處理該些三維影像，以分別產生多張一次轉換影像；以及利用該些一次轉換影像、該些第一點、該些第二點群以及該校正體的該預設規格參數組，基於該第一三維座標系統執行第二最佳化運算，以取得一最佳化第二轉換矩陣，其中該第二最佳化運算包含透過一第二收斂函式迭代調整一待解第二轉換矩陣，以取得該最佳化第二轉換矩陣；其中該座標轉換矩陣包含該最佳化第一轉換矩陣及該最佳化第二轉換矩陣。 The multi-mode image alignment system as described in claim 11, wherein obtaining the coordinate transformation matrix performed by the processing device includes: obtaining a plurality of first points corresponding to the central vertex and corresponding to the two-dimensional images from the two-dimensional images a plurality of second point groups of the side vertices; obtaining a plurality of third points corresponding to the central vertex from the three-dimensional images; using a first transformation matrix to be solved, the first points and the third points, Performing a first optimization operation based on the first three-dimensional coordinate system to obtain an optimized first transformation matrix, wherein the first optimization operation includes iteratively adjusting the first transformation to be solved through a first convergence function matrix to obtain the optimized first transformation matrix; use the optimized first transformation matrix to process the three-dimensional images to generate a plurality of primary transformation images respectively; and use the primary transformation images, the first points , the second point groups and the preset specification parameter set of the calibration body, perform a second optimization operation based on the first three-dimensional coordinate system to obtain an optimized second transformation matrix, wherein the second optimal Optimized Computing Package including iteratively adjusting a second transformation matrix to be solved through a second convergence function to obtain the optimized second transformation matrix; wherein the coordinate transformation matrix includes the optimized first transformation matrix and the optimized second transformation matrix transformation matrix.

如請求項14所述的多模影像對位系統，其中該些第一點與該些第三點分別對應於多個校正位置，且該處理裝置所執行之該第一最佳化運算更包含：對於每一該些校正位置的對應第一點及對應第三點執行距離計算，以取得多個分別對應於該些校正位置的多個計算結果，其中該距離計算包含：取得連接該第一三維座標系統之原點與該對應第一點的一射線；利用該待解第一轉換矩陣轉換該對應第三點；以及計算經轉換的該對應第三點與該射線之間的距離；以及將經迭代調整的該待解第一轉換矩陣作為該最佳化第一轉換矩陣，其中該第一收斂函式指示使該些計算結果之和為最小值。 The multi-mode image alignment system as described in claim 14, wherein the first points and the third points respectively correspond to a plurality of correction positions, and the first optimization operation performed by the processing device further includes : Perform distance calculation for the corresponding first point and the corresponding third point of each of the correction positions, so as to obtain a plurality of calculation results respectively corresponding to the correction positions, wherein the distance calculation includes: obtaining the first point connected to the correction position The origin of the three-dimensional coordinate system and a ray corresponding to the first point; transforming the corresponding third point by using the first transformation matrix to be solved; and calculating the distance between the converted corresponding third point and the ray; and The iteratively adjusted first transformation matrix to be solved is used as the optimized first transformation matrix, wherein the first convergence function indicates that the sum of the calculation results is a minimum value.

如請求項14所述的多模影像對位系統，其中該處理裝置所執行之該第二最佳化運算更包含：利用該待解第二轉換矩陣處理該些一次轉換影像以分別產生多張二次轉換影像；依據該些第一點及該些第二點群，分別從該些二次轉換影像取得多個第四點群；依據該些第四點群，分別取得多個估算規格參數組；以及將經迭代調整的該待解第二轉換矩陣作為該最佳化第二轉換矩陣，其中該第二收斂函式指示使該些估算規格參數組與該校正體的該預設規格參數組間之差異為最小值。 The multi-mode image alignment system as described in claim 14, wherein the second optimization operation performed by the processing device further includes: using the second transformation matrix to be solved to process the once-transformed images to generate multiple images respectively secondary transformation images; according to the first points and the second point groups, obtain a plurality of fourth point groups from the secondary transformation images respectively; According to the fourth point groups, a plurality of estimated specification parameter groups are respectively obtained; and the iteratively adjusted second transformation matrix to be solved is used as the optimized second transformation matrix, wherein the second convergence function indicates that the The difference between the estimated specification parameter sets and the preset specification parameter set of the calibration body is the minimum value.

如請求項16所述的多模影像對位系統，其中該些二次轉換影像、該些第一點、該些第二點群分別對應於多個校正位置，且該處理裝置所執行之取得該些第四點群包含：對於每一該些校正位置的對應第一點、對應第二點群及對應二次轉換影像執行：將該對應第一點投影至該對應二次轉換影像，以取得一第五點；以及將該第二點群中的多個點投影至該對應二次轉換影像，以取得多個第六點；其中該第五點及該些第六點成該些第四點群之一。 The multi-mode image alignment system as described in claim 16, wherein the secondary conversion images, the first points, and the second point groups respectively correspond to a plurality of correction positions, and the acquisition performed by the processing device The fourth point groups include: for each of the corrected positions corresponding to the first point, the corresponding second point group and the corresponding secondary transformation image, perform: project the corresponding first point to the corresponding secondary transformation image, to obtain obtaining a fifth point; and projecting a plurality of points in the second point group onto the corresponding secondary transformed image to obtain a plurality of sixth points; wherein the fifth point and the sixth points become the first One of the four point groups.

如請求項17所述的多模影像對位系統，其中該處理裝置所執行之取得該些估算規格參數組包含：對於每一該些第四點群，執行一規格參數估算以取得該些估算規格參數組之一，其中該規格參數估算包含：取得該第五點與該些第六點的多個連線；以及計算該些連線的多個估算長度及該些連線之間的多個估算夾角；其中該預設規格參數包含該校正體的多個預設邊長及多個預設夾角，該些估算規格參數組與該預設規格參數組之間的該差異指示一第一數值與一第二數值的加權和，該第一數值指示該些估算長度分別與該些預設邊長的多個差值之和，且該第二數值指示該些估算夾角分別與該些預設夾角的多個差值之和。 The multi-mode image alignment system as described in claim 17, wherein the obtaining of the estimated standard parameter sets performed by the processing device includes: for each of the fourth point groups, performing a standard parameter estimation to obtain the estimated parameters One of the specification parameter groups, wherein the specification parameter estimation includes: obtaining multiple connecting lines between the fifth point and the sixth points; and calculating multiple estimated lengths of the connecting lines and multiple distances between the connecting lines an estimated included angle; Wherein the preset specification parameters include a plurality of preset side lengths and a plurality of preset included angles of the calibration body, the difference between the estimated specification parameter sets and the preset specification parameter sets indicates a first value and a first value A weighted sum of two values, the first value indicates the sum of the differences between the estimated lengths and the preset side lengths, and the second value indicates the difference between the estimated included angles and the preset included angles sum of the differences.

如請求項14所述的多模影像對位系統，其中該處理裝置所執行之取得該些第三點包含：將每一該些三維影像作為一目標影像，執行：從該目標影像取得三個平面，該些平面彼此相鄰且分別具有之三個法向量彼此垂直；以及取得該些平面的交點，作為該些第三點中的一者。 The multi-mode image alignment system as described in claim 14, wherein the obtaining of the third points performed by the processing device includes: taking each of the three-dimensional images as a target image, and performing: obtaining three from the target image planes, the planes are adjacent to each other and have three normal vectors perpendicular to each other; and the intersection point of the planes is obtained as one of the third points.

如請求項11所述的多模影像對位系統，其中該第一三維座標系統係一二維影像擷取裝置的相機座標系統，該處理裝置更用於利用該二維影像擷取裝置的一參數矩陣，使該待處理影像從該第一三維座標系統轉換至該二維影像擷取裝置的影像平面座標系統，且該參數矩陣包含該二維影像擷取裝置的焦距參數及投影中心參數。 The multi-mode image alignment system as described in claim 11, wherein the first three-dimensional coordinate system is a camera coordinate system of a two-dimensional image capture device, and the processing device is further used to utilize a one-dimensional image capture device of the two-dimensional image capture device A parameter matrix transforms the image to be processed from the first three-dimensional coordinate system to the image plane coordinate system of the two-dimensional image capture device, and the parameter matrix includes focal length parameters and projection center parameters of the two-dimensional image capture device.