1331731 九、發明說明： 月30日替換^ 【發明所屬之技術領域】 本發明係關於-種建構三維擬真貼圖模型的方法，特別是 種解決遮蔽主體影像之三維擬真貼圖模型建構的方法。 【先前技術】 ' 二維擬真模型的建構過程中，常會利用很多影像處理技術來 L補美化原始的輸人影像，財蹟的數位化保存過程中為例，常 用影像來重建^蹟的三維模型，但是取得之影像常常會受到遮 f物的影響，使得幾何_的_减材質_都會遭遇困難。 第1A圖係為所要處理影像之遮蔽問題之示意圖，在重建三維模 型的過程中’會從各個不同角度來擷取欲重建主體之影像，但由 於遮敝物的影響’常常不能制僅有主體之影像，而會得到遮蔽 物破當成前景資訊之影像，在嚴重遮蔽的情況下，應賴統三維 重建的流_特徵#貞取和對應上容易造成許多的錯誤，也因此無 法獲得良好的三維重建結果，所以在遮蔽的情況下，重建擬真的 二維貼圖模型更是一個重大挑戰。 1知技術中’針對類似三維擬真影像重建的技術方案如美國 專利弟 US6571024 號’專利名稱為「Method and apparatus for multi-view three dimensional estimation」，主要技術手段在於利用 影像序列作為輸入’利用電腦視覺技術恢復攝影機拍攝方位和景 物之三維資訊，並將新物件合成至原先輸入之影像序列中。 美國專利第US6373487號，專利名稱為「Methods and apparatus for constructing a 3D model of a scene from calibrated 5 ^_ 99年6月30日替換頁 ges of the scene」，主要技術手段在於利^正過後的多 機’拍攝物體各角度的剪影(silhouette)，進而使用仏心此之原 理重建物體三維資訊。 美國專利第US6643396號，專利名稱為「Acquisiti〇n 〇f 3 D奶邮 w!th a single hand hdd camera」’主要技術手段在於利用單一相機配上 四面鏡子來取得多角度的影像即可建構出拍攝的場景。其中包含 自我校正技術，並可得到絕對尺寸的三維模型。 美國專利第US5511153號，專利名稱為「Meth〇dandapparatus for three_dimensi〇nai，textured models fr〇m plural vide〇 images」，主 要技術手段在於影像序贿作三維錢。齡影像的特徵來 計f出相機的内外部參數’以建構出拍攝的幾何模型。並將影像 依實際物體劃分出不同區域來做貼圖。 美國專利第US6856314號，專利名稱為「Meth〇d and辦咖 for 3D reconstruction of multiple views with altering search path and —nmodeling」，主要技術手段在於從多視角影像中之挑選特 性佳之特徵當作種子向翻之鄰近特徵點傳播，並避免搜尋路徑 往遮蔽的視角影像傳遞。 -般而言，主體的幾何模型會因為遮蔽產生破洞，而遮蔽物 的材質貼關會關主體的表㈣造成無法產生擬真的貼圖模 型。有時這些遮蔽物可以被移除(如車輛或行人），但有些遮蔽物可 能是固定在現場，而無法或不易排除的(如招牌 '標誌、樹木或其 他建築）。在這種情況下，無論從任何岐，都無法拍攝到完整I 乾淨的標地物錄。· ’在許乡古蹟_的數⑽像保存及三 維數位模型製作等，這些不相干的遮蔽物總 擾，使得數位影像或三維模型無法正確呈現標地物的形貌。 【發明内容】 本發明提出-種解決遮蔽主體影像之三維擬真貼圖模型建 構的方法，包括下列步驟：輸人—影像，並區分影像之遮蔽區域 與主體影像區域；满遮域的平面屬性；依據平面屬性建構 -具正確尺寸之三維幾何模型；依據平面屬性，姻多視角影像 修補遮蔽區域之-貼圖影像；以及結合具正確尺寸之三維幾何模 型與無遮蔽之貼圖影像，以完成主體影像之三維擬真貼圖模型。、 區勺忒影像之該遮蔽區域與主體影像區域的步驟更包括(1) 利用深度/視差資訊為基礎’藉以區分深度不同之該遮蔽區域與該 主體影像區域；（2)_該多視角影像間之交互隱，判斷遮蔽情 况，藉以區分出該遮蔽區域與該主體影像區域。 該遮蔽區域屬性定義為平面區域時’係以單應性矩陣投影來 建構具正確尺寸之三維幾何模型和修補祕區域之關影像。 田遮蔽區域屬性定義為非平面區域時，係以影像校正來建構 具正確尺寸之三維幾何模型和修補遮蔽區域之貼圖影像。 以上之關於本發明内容之說明及以下之實施方式之說明係， 用以不範與轉本發明之原理，並且提供本發明之專利中請範圍 更進一步之解釋。 【實施方式】 本發明所提之實施示意圖係以北墙著名古績金廣福之實景 為例來說明本發明所提綠之可行性，第1B _、為本發明之實施 1331731 例中輕贿之μ目，行域謂 時4遇到則述之遮蔽問題，由於金廣福左右兩邊裁種函作 (標示方框的部分），造成了在三維重建上的困難。除了遮Γ裸f t 落、邊 私，大部分的理論均無法處理這樣嚴重的遮播情況「n 使用立體視覺技術所得到之視差圖（右半邊灰階圖示）^ = 造成只能得到片段破碎之視差圖。因此’在遮蔽部分的正 幾何幾乎沒触线，哺域真闕的三_败是_重二重 因此，本發明提出了-個利用多視角影像解決上述遮蔽問題 =二維擬真關模猶構方法。此方法主要分成兩部分來解決遮 敝_ :建構正確尺寸之幾何模型和修補關兩部分。針對遮蔽 問題發生區域的不同，若舰區域落在重建主體之平面區域上， 就利用建築物的基本特性：通常由平面以及簡單幾何(圓錐、圓柱） 形狀所組成。利用共平面特性來建構數學上稱為單應性 (h〇m0graphy)的矩陣’進一步利用單紐(h_卿㈣來求取正確 的對應點’若遮蔽區域落在重建主體之非平面區域上，則利用基 礎矩陣作影像校正(rectificati〇n)輔助求取對應，點；透過這些方式求 得之對應職夠保證得雜為正叙幾何尺寸。在修獅圖的這 個部分，首先在各個不同的角度拍攝未被遮擋或部分遮擋的影 像’求取空間轉換關係’並對多視角影像作影像變形(warping)， 最後擷取相似部分的區域來修補被遮蔽的區域。在確定被遮蔽部 刀的幾何正確和貼圖也完整修補後，就可以使用材質貼圖的技術 建構出無遮擋的三維擬真貼圖模型。 8 1331731 ^ ^ j- ^ I 99年6月30日替換頁 圖糸為本發明所提方法之流程圖，提出解決遮蔽主體 影像之三祕真闕模輯構的方法，包括下列辣：輸入^影 像，並區分影像之遮蔽區域與主體影像區域(步驟2U斷遮蔽 主體影像之區域的平面屬性(步驟220);依據平面屬性建構一具正 確尺寸之二維幾何㈣(步驟现）；依據平面屬性，利舒視角影 像修補遮蔽區域之-貼圖影像(步驟);以及結合具正確尺寸之 -維幾何㈣與無賴之貼圖影像，以完成域影像之三維擬真 貼圖模型(步驟250)。 在上述之區分影像之遮贿域與主體影賴域的步驟中，首 先必須區分舰區域和欲重建之主體影像區域1單的方法如第 2B圖所示，可以手叫方式_出遮蔽區域，或者以自動對應的 方式，先對兩幅晝面比對對應點，再計算整幅晝面的視差 __ty) ’再根據視差圖的相似性作區塊連結，將視差較小(代表 深度較淺)的部分切割出來，即為樹的部分。 本發明狀義之平面屬性係分為平面區域與非平面區域，依 據不同的平面雜料同之纽，若歧區·在銳主體影像 之平面區域上就利用共平面的特性來建構數學上稱為單應性 (h_gmphy)的矩陣’進—步利用單應性(h_卿聯求取正確 的對應點；若區域落在重建域影像之非平面區域上，則作 影像校正(rectification)來求取對應點。 第3A圖係為本發明所提方法中，以單應性矩陣來建構平面 區域具正软寸之三_何模型，包括下列步驟：以輸入影 基準々像纟主體讀之未遮蔽區域尋找落在與紐區域同—平 9 1331731 ^(homograpt^ ° V V’ =// U V Λπ \2 h，、 h \3 u 1 1 21 n22 h23 V L J 1 S\ h32 A33. 人 u’ M h2\u+Ibiv^th·, l^lw+/?32v+^3 張影二為三維空間中共平面的對應 、「苐—張讀上的對應點座標。展開後可得 «II « + Λ,2 v+A13 - A /w，_, 2.«^22ν+Λ23-Α3]„ν，_ξ ^,- 33^=0 J2 ^33 v = 0 若存在n張影像，則對H 之元素移項整理後可得 10 1331731 99年6月30日替換頁 h V1 1 0 0 0 ~u\u\ 一 Yi’ 六i q 「〇l 0 0 0 u) V1 1 -u\v\ -Yi' -y IJ ^21 0 un vn 1 0 0 0 ~urtun ~vnun _v A，q o 0 0 0 un vn I ^unvn ~vnvfi' 一 V nx9 15 hx 0 ^2 -1 」9xl 最後，至少需要四組或四組以上的對應點才能利用奇異值分 解（Singular Value Decomposition ; SVD )解得單應性 (homography) ° 利用共平面建構具正確尺寸之二維幾何模型之詳細步驟如 下： a_在共平面但未遮擋區域擷取特徵點並作對應； b. lk機選取η組對應點，並利用強健式方法(如或 LMedS)估算單應性(h〇m〇graphy); c. 設定-張基準影像為三維重建基底影像，科動或自動的 方式描緣出賴區域巾m线幾何特徵的重要特徵點 Ι = {χ,，Χ2，·.”Χ”}. d. 利用估异之單應性(h〇m〇graphy)作投影基準影像中的重要 特徵點％，以求得在其他非基準影像中的對應，這 些遮蔽區域中的對應點對為{u.U ; e. 結合遮蔽區射對應點hu和非遮蔽區域中對應點對 ，依照—般三維重建流鱗立三維幾何。 第犯圖係為利用單應性(homography健立遮蔽區域之對應 1^31731 ，由於遮蔽區域是落在-平iSS H在共平面的未魏區域解魏對應點，計算此平 面之早應性(h〇mog响）。縣，在遮蔽區域點選代表金廣 之特徵f ’利用單應性(h_卿hy)投影求得遮蔽區域的 、〜。在求㈣敝區域的對應點之後，就可以應用—般三維重 建流程得到三維幾何。 f 4A圖係為本發明所提樹，以影像校正來建構非平面 品正確尺寸之三維幾何模型，包括下列步驟：以輸入影像為 基準影像’在主體職之未親_録概點，並料他非基 ^像中尋找其對應點(步驟);針對對應點，計算影像之間二 土礎矩陣(步驟42〇); _基礎矩_極雜換成絲準影像之 平行(轉);雜正的滅上搜魏準影财遮蔽區域 寺徵點之對應點(步驟440);將賊點座標_基準影像之一 =(步驟彻）；根據轉_準影像之—影像賴之對應點資:來 建構二維幾何模型(步驟46〇)。 現以實際處理手段·如下以支社述方法的可行性，若遮 蔽=在料祕域，上叙單雜(h嶋_y)健法適用，，’·， 於疋尋求糾-種方絲求得被舰區域之對應點，主要是利用 影像校正(祕Ga㈣細㈣錄(epipQlar .細成和掃瞒 線(顏hne)平行’則對應點會座落在此校正過後的極線上，如此 便能求取對應點。 利用影像校正建構具正確尺寸之三維幾何模型之詳細步驟 如下： 12 a.在未遮擋區域擷取特徵點並作對應；Liill·月30日替換頁_ 並利用強健式方法(如RANSAC或 b隨機選取n組對應點 LMedS)估算基礎矩陣； c. 利用基雜_極轉換鑛影像之細線平行； d. 遮蔽區域之特徵點在轉正的極線上搜尋對應點； e. 將對應點座標轉回原始影像座標，這些對應點對為k，rL; f·結合紐輯情應點PL.和非錢區域中對應點對 P ，依照一般三維重建流程建立三維幾何。 、、上述提到之影像校正的方法主要是利用計算出之基礎矩陣 =投域換咖批細tmnsf_atiQn)將影像之極點轉換至無 躬通處其數學式為：㈣对，其巾£為透過基礎矩陣求得之 冬‘j(epip〇le) ’ p則為我們要求之投影轉換矩陣。利用這個投影轉 換矩陣P ’就可以將極轉換成鱗喊平行，並得出一張校正 過後的影像。 第4B圖係為利用影像校正建立遮蔽區域之對應點示意圖， 在，遮蔽區域找尋特徵對應點，計算影像之間的極線幾何關係。 接著’對影像作校正，使其極線變成與御線平行。在遮蔽區域 中代表金廣福重要舰之特徵點便可在轉正上的極線上搜尋對應 點。在求得遮蔽區域的對應點之後，就可以應用一般三維重建流 程得到三維幾何模型。 本發明所提之修補貼圖影像部分，仍然將遮蔽區域區分為平 面區域和非平面區域。在平面區域我們計算上述之單應性 13 (h㈣吻）’根據此轉換關係將多視角影像 把據區域相似度填補遮蔽區域。在非平面區域，透過計算影像之 間的極線幾何關係’並對多視細_彡像校正，最後也根據區 域相似度填補遮蔽區域。 。第A圖係為本發明所提方法中，以單應性矩陣來補平面 區j之遮祕域之關影像，包括τ列麵··選擇欲修補之一基 準影像(步驟_);計算基準影像之空_何關係(步驟520);對 =同角度拍攝之影像做投影轉換(步驟53G);針對投影轉換過後之 多張影像作主體影像被遮蔽之區域相似影像之搜尋(步驟$你依 據區域相似影像填齡準影像__部分(步驟娜）。 :現以貫際處理手段說明如下財持上述方法的可行性，為了 要讓f建的三維模型魏好的貼圖影像，本發明針對貼圖影像中 被=的部分作影像修補(image㈣咖㈣的動作。提& _決_ 式疋攸各個不同的角度來拍攝被遮蔽區域、 之單轉賴辦屬目魏，詳==4 a. 在多視角影像中選擇欲修補之基準影像；在多視角影 像中選擇前述之第1B圖當作欲修補之基準影像，而 其他輸入之多視角影像如第5B圖所示。 b. 計#影像之間的單應性(homogmphy)幾何關係；針對 不同視角所拍攝的影像’求取絲準影像欲修補區域 之對應點’請參考第第5 C圖所示，並以強健式方法 估异單應性(homography)。 C.將不同角度影像做投影轉換，根據上述求出之單應性 1331731 I 99年6月30日替揸百 (homography)作投影轉換重新計請^ 考下列所示。1331731 IX. Description of the invention: Replacement on the 30th of the month ^ Technical Field of the Invention The present invention relates to a method for constructing a three-dimensional immersive map model, and more particularly to a method for constructing a three-dimensional immersive map model for masking a subject image. [Prior Art] In the construction process of two-dimensional immersive models, many image processing techniques are often used to supplement the original input images. The digital image of the treasury is used as an example to reconstruct the three-dimensional images. The model, but the image obtained is often affected by the obscuration, making the geometry__subtraction material_ encounter difficulties. Figure 1A is a schematic diagram of the problem of shadowing of the image to be processed. In the process of reconstructing the 3D model, the image of the subject to be reconstructed will be retrieved from different angles, but the influence of the concealer is often unable to make only the subject. The image, and the image of the foreground is broken into the image of the foreground information. In the case of severe obscuration, it is easy to cause many errors in the stream_features of the three-dimensional reconstruction and the corresponding three-dimensional reconstruction. Reconstructing the results, so reconstructing the immersive two-dimensional map model is a major challenge in the case of shadowing. 1 Known technology's technical solution for similar three-dimensional imaginary image reconstruction, such as US Patent No. US6571024, the patent name is "Method and apparatus for multi-view three dimensional estimation", the main technical means is to use the image sequence as input 'utilize the computer Visual technology restores the camera's three-dimensional information about the orientation and scene, and synthesizes the new object into the original input image sequence. U.S. Patent No. 6,373,487, entitled "Methods and apparatus for constructing a 3D model of a scene from calibrated 5 ^_June 30, 2010 replacement page ges of the scene", the main technical means is that after the positive The machine 'shoots the silhouette of the object at various angles, and then uses the principle of the heart to reconstruct the three-dimensional information of the object. U.S. Patent No. 6,643,396, entitled "Acquisiti〇n 〇f 3 D Milk Mail w!th a single hand hdd camera"' main technical means is to use a single camera with four mirrors to obtain multi-angle images can be constructed The scene taken. It includes self-correcting techniques and a three-dimensional model of absolute size. U.S. Patent No. 5,511,153, entitled "Meth〇dandapparatus for three_dimensi〇nai, textured models fr〇m plural vide〇 images", the main technical means is to image the bribe for three-dimensional money. The characteristics of the age image are taken from the internal and external parameters of the camera to construct a geometric model of the shot. The image is divided into different areas according to the actual object to make a map. US Patent No. US6856314, the patent name is "Meth〇d and coffee for 3D reconstruction of multiple views with altering search path and -nmodeling", the main technical means is to select features from multi-view images as a seed The adjacent feature points propagate and avoid the search path to the shadowed view image. In general, the geometric model of the subject will create holes due to obscuration, and the material of the obscuration will close the main table (4), resulting in the inability to produce a realistic mapping model. Sometimes these shelters can be removed (such as vehicles or pedestrians), but some shelters may be fixed on site and cannot be or are not easily excluded (such as signboards, signs, trees or other buildings). In this case, it is impossible to capture a complete I clean landmark record from any flaw. · The number of (10) image preservation and three-dimensional digital model production in the Xuxiang Monument _, these irrelevant masks, so that the digital image or 3D model can not correctly display the shape of the object. SUMMARY OF THE INVENTION The present invention provides a method for constructing a three-dimensional immersive texture model for obscuring a subject image, comprising the steps of: inputting a human-image, and distinguishing between a shadow region and a subject image region of the image; and a planar property of the full mask region; According to the planar attribute construction - a three-dimensional geometric model with the correct size; according to the planar property, the multi-view image repairs the shadow-image of the masked area; and combines the three-dimensional geometric model with the correct size and the unshielded texture image to complete the main image 3D immersive texture model. The step of the masking area and the main image area of the area scooping image further comprises: (1) using the depth/disparity information as a basis to distinguish the masking area and the subject image area with different depths; (2) _ the multi-view image The interaction between the two is hidden, and the masking condition is judged to distinguish the masked area from the subject image area. When the masked area attribute is defined as a flat area, the image of the three-dimensional geometric model with the correct size and the repaired secret area is constructed by the homography matrix projection. When the field masking area attribute is defined as a non-planar area, the image correction is used to construct a three-dimensional geometric model with the correct size and the texture image of the patched area. The above description of the present invention and the following description of the embodiments of the present invention are intended to provide a further explanation of the scope of the invention. [Embodiment] The schematic diagram of the implementation of the present invention is based on the example of the famous ancient performance of the North Wall, Jin Guangfu, as an example to illustrate the feasibility of the greening of the present invention, and the first embodiment is the implementation of the invention. In the case of the line domain, the problem of obscuration is described in the case of 4, and the problem of the three-dimensional reconstruction is caused by the cutting of the two sides of Jin Guangfu (the part of the box). Except for concealing the nakedness and the private side, most of the theories can't handle such a serious interception situation. “The disparity map obtained by stereoscopic vision technology (the gray half of the right half) ^ = can only be fragmented. The parallax map. Therefore, the positive geometry in the shadow part is almost not touched, and the three-failure of the true domain is _heavy. Therefore, the present invention proposes to solve the above-mentioned shading problem by using multi-view images. This method is mainly divided into two parts to solve the concealer _: constructing the geometric model of the correct size and repairing the two parts. If the area of the shadowing problem occurs, if the ship area falls on the plane area of the reconstructed body The basic characteristics of the building are used: usually composed of planes and simple geometric (conical, cylindrical) shapes. Coplanar properties are used to construct a matrix mathematically called homography (h〇m0graphy). H_Qing (4) to find the correct corresponding point 'If the shadowed area falls on the non-planar area of the reconstructed subject, then use the basic matrix for image correction (rectificati〇n) , the point; the corresponding position obtained through these methods is enough to ensure that the geometrical dimensions are positive. In this part of the lion's diagram, firstly, unobstructed or partially occluded images are taken at different angles. 'Image warping of multi-view images, and finally grab the similar parts to repair the shaded area. After determining the geometric correctness of the masked knife and the texture is also completely repaired, you can use the texture map technology. Constructing an unobstructed three-dimensional immersive texture model. 8 1331731 ^ ^ j- ^ I Replacement page on June 30, 1999 is a flow chart of the method proposed in the present invention, and proposes to solve the three secrets of the masked subject image. The method of the composition includes the following hot: inputting the image, and distinguishing the shadow area of the image from the main image area (step 2U is to cut the plane attribute of the area of the main image (step 220); constructing a correct size according to the plane attribute Dimensional geometry (4) (steps now); according to the planar properties, the image of the slanted angle of view repairs the shadowed area - the texture image (step); and the geometric dimension with the correct size And the rogue map image to complete the three-dimensional immersive map model of the domain image (step 250). In the above steps of distinguishing the image from the bribery field and the subject image, firstly, the ship area and the subject image to be reconstructed must be distinguished. The method of the area 1 is as shown in FIG. 2B, and the masking area can be called by hand, or the corresponding point can be compared with the two sides by the corresponding method, and the parallax __ty of the whole side can be calculated. 'The block is connected according to the similarity of the disparity map, and the part with small parallax (representing the shallow depth) is cut out, which is the part of the tree. The planar attribute of the present invention is divided into a planar area and a non-planar area. According to the different planes and materials, if the area is in the plane area of the sharp subject image, the characteristics of the coplanar plane are used to construct the matrix called the homography (h_gmphy). (h_Qinglian seeks the correct corresponding point; if the area falls on the non-planar area of the reconstructed domain image, then the image is rectified to obtain the corresponding point. FIG. 3A is a method for constructing a planar region with a positive softness in a method according to the present invention, and includes the following steps: searching for an unmasked region of the image read by the input image Same as the New Zealand area - Ping 9 1331731 ^(homograpt^ ° V V' =// UV Λπ \2 h, h \3 u 1 1 21 n22 h23 VLJ 1 S\ h32 A33. Human u' M h2\u +Ibiv^th·, l^lw+/?32v+^3 Zhang Ying 2 is the correspondence of the common plane in the three-dimensional space, "苐--reading the corresponding point coordinates. After the expansion, you can get «II « + Λ, 2 v+A13 - A /w,_, 2.«^22ν+Λ23-Α3]„ν,_ξ ^,- 33^=0 J2 ^33 v = 0 If there are n images, then the element shift of H can be obtained. 10 1331731 Replacement page on June 30, 1999 h V1 1 0 0 0 ~u\u\ One Yi' six iq "〇l 0 0 0 u) V1 1 -u\v\ -Yi' -y IJ ^21 0 Un vn 1 0 0 0 ~urtun ~vnun _v A,qo 0 0 0 un vn I ^unvn ~vnvfi' A V nx9 15 hx 0 ^2 -1 ”9xl Finally, at least four or more corresponding points are required Can use Singular Value Decomposition (SVD) to solve homography The detailed steps of constructing a two-dimensional geometric model with correct dimensions using coplanar are as follows: a_ extract feature points in coplanar but unoccluded regions and correspond; b. lk machine selects n corresponding points and uses robust methods ( For example, or LMedS) to estimate homography (h〇m〇graphy); c. Set-up of the baseline image as a three-dimensional reconstruction of the base image, or a scientific or automatic way to trace the important features of the m-line geometric features of the regional towelΙ = {χ,,Χ2,·."Χ"}. d. Use the homography (h〇m〇graphy) as the important feature point % in the projected reference image to find other non-reference images. Correspondingly, the corresponding point pairs in the occlusion regions are {uU; e. combined with the corresponding points in the occlusion region and the corresponding point pairs in the non-shadow region, and the three-dimensional geometry is reconstructed according to the general three-dimensional reconstruction. The first offense is to use the homography (the corresponding 1^31731 of the homography-shielding area, and the early response of this plane is calculated because the shadowed area is the corresponding point in the un-wei area of the flat iSS H in the coplanar plane. (h〇mog ring). County, in the shadow area, select the representative of Jin Guang's feature f 'Using the homography (h_qing hy) projection to obtain the shaded area, ~. After seeking the corresponding point of the (four) 敝 area, you can Applying the general three-dimensional reconstruction process to obtain three-dimensional geometry. f 4A is a tree proposed by the invention, and uses image correction to construct a three-dimensional geometric model of the correct size of a non-planar product, including the following steps: using the input image as a reference image If you don't kiss the general point, and expect him to find its corresponding point in the non-base image (step); for the corresponding point, calculate the two earth matrix between the images (step 42〇); _ base moment _ extremely mixed The parallel of the silk image (transfer); the miscellaneous positive on the search for the corresponding point of the temple sign in the area of the shadow area (step 440); the thief point coordinate _ reference image = (steps); The corresponding image of the image-based image: to construct a two-dimensional geometric model (step 4 6〇). Now the actual treatment means, as follows, to illustrate the feasibility of the method, if the shadow = in the secret area, the above-mentioned single (h嶋_y) health law applies, '·, Yu Yu seeks to correct - The square wire is used to obtain the corresponding point of the ship area, mainly by using image correction (secret Ga (four) fine (four) record (epipQlar. Fine and broom line (yan hne) parallel' then the corresponding point will be located in this corrected pole On the line, you can find the corresponding point. The detailed steps of constructing the 3D geometric model with the correct size using image correction are as follows: 12 a. Take the feature points in the unoccluded area and make correspondings; Liill·Replace the page on the 30th of the month_ The basic matrix is estimated by a robust method (such as RANSAC or b randomly selecting n sets of corresponding points LMedS); c. The thin lines of the base-to-pole transition image are parallel; d. The feature points of the shadowed area are searched for corresponding points on the positive pole line e. Convert the corresponding point coordinates back to the original image coordinates, these corresponding point pairs are k, rL; f· combined with the new point of interest PL. and the corresponding point pair P in the non-monetary area, according to the general 3D reconstruction process to establish 3D geometry The method of image correction mentioned above It is to use the calculated basic matrix = the domain to change the tmnsf_atiQn) to convert the pole of the image to the innocent pass. The mathematical formula is: (4) the pair, the towel is the winter obtained by the basic matrix 'j (epip〇) Le) 'p is the projection transformation matrix we require. Using this projection transformation matrix P', we can convert the pole into a scale and parallel, and get a corrected image. Figure 4B is to use the image correction to establish the shadow A schematic diagram of the corresponding points of the region, in the shadow region to find the corresponding points of the feature, calculate the geometric relationship between the polar lines of the image. Then 'correct the image so that its polar line becomes parallel with the royal line. In the shaded area, the feature points representing the important ships of Jin Guangfu can be searched for the corresponding points on the polar line on the positive turn. After obtaining the corresponding points of the occlusion area, a general three-dimensional reconstruction process can be applied to obtain a three-dimensional geometric model. The patch image portion of the present invention still distinguishes the masked area into a flat area and a non-planar area. In the plane area, we calculate the homography 13 (h(four) kiss) above. According to this conversion relationship, the multi-view image is filled with the area similarity to fill the masked area. In the non-planar area, by calculating the polar line relationship between the images and correcting the multi-view 彡 image, the masked area is finally filled according to the area similarity. . Figure A is a method for the method of the present invention, which complements the image of the masking domain of the plane j with a homography matrix, including the τ column surface, selects one of the reference images to be repaired (step _); Image space _ what relationship (step 520); for the same angle shooting image projection conversion (step 53G); for the projection of the converted multiple images as the subject image is shaded area similar image search (step $ you rely on Regional similar image age-filled quasi-image __ part (step Na).: The following method is used to illustrate the feasibility of the above method. In order to make the three-dimensional model of F construct a good map image, the present invention is directed to the map. In the image, the part of the image is patched (image) (the action of the image (four) coffee (four). The & _ decision _ 疋攸 疋攸 疋攸 疋攸 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄 拍摄Select the target image to be repaired in the multi-view image; select the first picture BB as the reference image to be repaired in the multi-view image, and the other input multi-view image as shown in Fig. 5B. b. Hoogmphy geometric relationship For images taken at different angles of view, 'see the corresponding points of the area where the silk image is to be repaired', please refer to Figure 5C, and estimate the homography by a robust method. C. Image of different angles Do projection conversion, according to the above-mentioned homography 13331731 I June 30, 1999 for homography for projection conversion re-counting ^ test the following.
V Μ n\ =H j) u J (】JV Μ n\ =H j) u J (]J
Image 轉換結果如第5D圖所示，持續步驟b和e，直到所有視 角的影像投影轉換均已完成。 d.請參考第5 E圖所示，係為從多視角影像填補遮蔽區 域影像示意圖，根據區域間的相似度填補被遮蔽影像 目標區，填補被遮蔽之影像目標區時， 蔽區的小塊區域(通常為一小塊吟在== 視角影像中作相似度的比對，其比對位置為對應到目 標影像之相同位置。依序進行同樣的程序，直到目標 區均修補完成。第5F圖為經過初步修補後的結果， 選定之小塊區域已經成功的修補完成。 第$ _為本判綠巾，㈣正纽補非平面 旦敝區域之貼圖影像’包括下列步驟：選擇欲修補之一基 ==㈣物她刪物線幾何關係 (步驟620)，對柯㈣簡之影縣册校正，使轉盘基料 像之掃_平行(麵63G) ; __她度在極線上搜尋與基準 影像被遮蔽影像目標區最相似之區域(步驟64〇);依據最相似之區 99年6月30日替換頁 ;將修補後之影像轉換回 域填補基準影像被遮蔽的部分(步驟650) 基準影像座標系統(步驟660)。 現以實際處理手觀明如下以支彳社財法的可行性， 蔽發生在非平題域’同樣可以姻前述影像校正的方式來修補 被遮蔽區域。修補貼圖影像之步驟如下： / a.在未遮擋區域擷取特徵點並作對應； b. 計算影像之間的極線幾何關係； c. 將不同角度影像做影像校正； 之區^根顧域她度在極線上搜尋無鑛影像目標區最相似 e·將修補後之影像轉換回原始影像座標系統。 、上述提到之影像校正的方法主要是利用計算出之基礎矩陣 透過-投影轉換(pr0jective transf_ati〇n)將影像之極點轉換至益 窮遠處，其數學式為：〜(1树，其中e為透過基礎矩陣求狀 極點(等)le) ’ P則為我們要求之投影轉換矩陣。利用這個投影轉 換矩陣P，就可以將極線轉換成與掃财平行，並得出一張校正 過後的影像。 /經過上述兩種處理方式，確定所有遮蔽區域均已填補完畢 後’即可將修補過後的影像用作貼圖影像供後續處理。第6B圖則 為金廣福修補後之結果，遮蔽物已經完全被移 除’其細節也被完整的㈣下來’能夠作後續闕模型處理。 結合具正確尺寸之三維幾何模型與無遮蔽之貼圖影像，以完 成主體影像之三維擬真闕模型：_财貼卿咖em_ng) 丄幻1731 們可以將具有正確尺寸之三維模 影像結合，完成無遮擋問題之三維擬真貼圖模型。第7A圖為未The image conversion result is as shown in Figure 5D, and steps b and e are continued until the image projection conversion for all viewing angles is completed. d. Please refer to Figure 5 E for the image of the masked area image from the multi-view image. Fill the shaded image target area according to the similarity between the areas to fill the shaded image target area. The area (usually a small block of similarity in the == perspective image, the alignment position is the same position corresponding to the target image. The same procedure is performed in sequence until the target area is repaired. 5F The picture shows the results of the preliminary repair, the selected patch area has been successfully repaired. The first $ _ the green towel, (4) the map image of the non-planar area of the positive button includes the following steps: Select the patch to be repaired A base == (four) object she deletes the geometric relationship of the object line (step 620), corrects the county record of Ke (four) Jane, so that the turntable base material image sweeps _ parallel (face 63G); __ she searches on the polar line The area most similar to the target image being masked by the target image (step 64〇); replacing the page according to the most similar area on June 30, 1999; converting the repaired image back to the domain to fill the portion of the reference image that is obscured (step 650) ) Reference image coordinate system (step 660). Now, the actual processing hand is used to observe the feasibility of supporting the social accounting method. The masking occurs in the non-level field. The same can be used to repair the masked area. The steps of patching the image are as follows: / a. Extract feature points in the unoccluded area and correspond; b. Calculate the polar line relationship between the images; c. Correct the image at different angles; The target area of the mine image is most similar. e. Convert the repaired image back to the original image coordinate system. The image correction method mentioned above mainly uses the calculated basic matrix to transmit image by pr0jective transf_ati〇n. The pole is transformed to the far side, and the mathematical formula is: ~ (1 tree, where e is the pole through the basic matrix (equal) le) 'P is the projection transformation matrix we require. Using this projection transformation matrix P, It is possible to convert the polar line into parallel with the sweeping, and obtain a corrected image. / After the above two treatments, it is determined that all the masked areas have been filled. The latter image is used as a texture image for subsequent processing. Figure 6B is the result of Jin Guangfu's repair, the mask has been completely removed 'the details are also complete (four) down' can be used for subsequent 阙 model processing. Combined with the correct size The 3D geometric model and the unobscured texture image to complete the 3D simulation of the subject image: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Quasi-real map model. Figure 7A is not
經修補之貼_型，但經過本發_方法處理之後，結果如第7B 圖所示，效果遠勝於未修補的貼圖模型。此貼圖模型將可應用於 文化保存缝位化文物展轉_，具有較絲和文化價值。 雖然本發明以前述之較佳實施例揭露如上，然苴The patched _ type, but after processing by the _ method, the result is as shown in Figure 7B, the effect is much better than the unpatched texture model. This map model will be applied to cultural preservation of seams and cultural relics. It has a relatively silky and cultural value. Although the present invention has been disclosed above in the preferred embodiments described above,
定本發明。在不脫離本發明之精神和範_，所為之更動與潤飾I 均屬本發明之專梅'護範圍。_本發明所界定之倾範圍請參 考所附之申請專利範圍。 ^ 【圖式簡要說明】 第1A圖係為所要處理影像之遮蔽問題之示意圖； 第1B圖係為本發明之實施例中遮蔽問題之示意圖； 第1C圖係為利用立體視覺技術所得到片段料之視差圖 第2A圖係為本發明所提方法之流程圖； ’?、y像之遮敝區域與該 第2B圖係為本發明所提方法之區分該 主體影像區域之示意圖； 第3A圖係為本發明所提方法中，以單應性矩陣來建構平面 &域具正確尺寸之三維幾何模型； 一 f 3B ®係為利用單應性(h〇m〇gmphy)建立遮蔽區域之對應 以影像校正來建構非平面 第4A圖係為本發明所提方法中 £域具正確尺寸之三維幾何模型； 第4B圖係為利用影像校正建立遮蔽區域之對應點示意圖 巴域第5 A _林糾簡找巾，叫 匕埤之遮蔽區域之貼圖影像； 第5B圖係為不同角度擷取影像示意圖； =5 C圖係絲取對應輯算單應性(h_卿吻示意圖； $ 5D圖係為根據單應性(h〇m〇gra㈣作影像變形示意圖； 第5E圖係為好視肖影像填補遮蔽區域影像示意圖； 第5F圖係為填補部分區域後之結果示意圖； 第6A圖係為本發明所提方法中， 區域之遮蔽區域之貼圖影像； 以影像校正來修補非平面 第6B圖係為作完影像修補後的貼圖影像示意圖； 第7A圖係為沒有作遮蔽補償之三維貼圖模型示意圖；以及 第7B圖係為作遮蔽補償之三維貼圖模型示意圖。 圖式符號說明】 步驟210輸入一影像，並區分影像之遮蔽區域與主 體影像區域 步驟220...........判斷遮蔽該主體影像之區域的平面屬性 步驟230...........依據平面屬性建構一具正確尺寸之_ 幾何模型 二、准 步驟細—依據平面屬性，利用多視角影像修補遮蔽 區域之一貼圖影像 步驟250...........結合具正確尺寸之三維幾何模型與無遮 蔽之貼圖影像’以完成主體影像之三維擬 真貼圖模型 1331731 麵 99年6月30曰替換頁 ’ 步驟310...........以輸入影像為基準影像，在主體影像之未 * 遮蔽區域尋找落在與遮蔽區域同一平面 , 上之特徵點，並在其他非基準影像中尋找 •： 其對應點 • 步驟320...........利用對應點，計算影像之間的單應性 步驟330...........在基準影像中，選擇遮蔽區域中能描述幾 何特徵之特徵點，利用估算之單應性投影 算出在其他非基準影像之對應點 步驟340...........根據對應點資訊來建構三維幾何模型 步驟410...........以輸入影像為基準影像，在主體影像之未 遮蔽區域擷取特徵點，並在其他非基準影 像中尋找其對應點 步驟420...........利用對應點，計算影像之間的一基礎矩陣 步驟4 3 0...........利用基礎矩陣將極線轉換成與基準影像 之掃瞄線平行 步驟440...........在轉正的極線上搜尋基準影像中遮蔽區 域特徵點之對應點 步驟450...........將對應點座標轉回基準影像之一影像座 標 ' 步驟460...........根據轉回基準影像之一影像座標之對應 點資訊來建構三維幾何模型 步驟510........選擇欲修補之一基準影像 步驟520...........計算基準影像與其他視角影像之空間幾 19 1331731 99年6月30日替換頁 何關係 步驟530...........對其他視角影像做投影轉換 步驟540...........針對投影轉換過後之多張影像作主體影 像被遮蔽之部分相似影像之搜尋 步驟550...........依據部分相似影像填補基準影像被遮蔽 的部分 步驟610...........選擇欲修補之一基準影像 步驟620...........計算基準影像與其他視角影像之極線幾 何關係 步驟630...........對其他視角影像作影像校正，使極線與基 準影像之掃瞄線平行 步驟640...........根據區域相似度在極線上搜尋與基準影 像被遮蔽影像目標區最相似之區域 步驟650...........依據最相似之區域填補基準影像被遮蔽 的部分 步驟660...........將修補後之影像轉換回基準影像座標系 統 20The invention is defined. Without departing from the spirit and scope of the present invention, both the modifier and the retouching I are within the scope of the invention. _ The range of the scope defined by the present invention is referred to the attached patent application. ^ [Brief Description] Figure 1A is a schematic diagram of the problem of masking of the image to be processed; Figure 1B is a schematic diagram of the masking problem in the embodiment of the present invention; Figure 1C is a fragment obtained by stereoscopic vision technology. 2A is a flow chart of the method of the present invention; the concealing area of the '?, y image and the 2B picture are schematic diagrams for distinguishing the main image area by the method of the present invention; In the method of the present invention, a three-dimensional geometric model with a correct size of the plane & region is constructed by a homography matrix; a f 3B ® system is used to establish a corresponding area of the masked region by using homography (h〇m〇gmphy) The non-planar 4A image is constructed by image correction, which is the 3D geometric model of the correct size of the method in the proposed method. The 4B figure is the corresponding point of the shadow region established by image correction. Correctly find the towel, call the image of the shadow area of the shadow area; Figure 5B is a schematic diagram of the image taken at different angles; =5 C picture line corresponds to the corresponding calculation of homography (h_qing kiss diagram; $ 5D diagram According to homography (h〇m Gra (4) is a schematic diagram of image deformation; 5E is a schematic diagram of the image of the shadow area filled with the image of the good view; the 5F is a schematic diagram of the result after filling the partial area; FIG. 6A is the masked area of the method in the method of the present invention The image of the texture; the image correction is used to repair the non-planar image of Figure 6B as the image of the texture image after the image is repaired; the 7A is a schematic diagram of the 3D texture model without shading compensation; and the 7B is the shading compensation Schematic diagram of the three-dimensional map model. Schematic description of the symbol] Step 210 input an image, and distinguish between the masked area of the image and the subject image area. Step 220..............determine the planar attribute of the area that masks the subject image Step 230...........Constructing a correct size according to the planar attribute _ Geometric model 2. Quasi-step fine--Based on the planar attribute, the multi-view image is used to repair one of the masked areas of the map image step 250.. .... Combine the 3D geometric model with the correct size and the unshielded texture image to complete the 3D immersive texture model of the subject image 13331731 Face to June 30, 1999 Replace page 'Step 310...........Use the input image as the reference image to find the feature points on the same plane as the masked area in the unmasked area of the subject image, and in other non-references Finding in the image •: its corresponding point • Step 320...........Using the corresponding point, calculate the homography between the images. Steps 330........... in the reference image In the middle, select the feature points in the masked area that can describe the geometric features, and use the estimated homography projection to calculate the corresponding points in the other non-reference images. Steps 340...........Construct the three-dimensional according to the corresponding point information. The geometric model step 410.....takes the input image as the reference image, captures the feature points in the unmasked area of the subject image, and searches for the corresponding points in other non-reference images. Step 420... ........Use a corresponding point to calculate a basic matrix between images. Step 4 3 0...........Use the basic matrix to convert the polar line into a scan line with the reference image. Parallel steps 440...........search for corresponding points of the masked area feature points in the reference image on the positive pole line. Step 450...........turn the corresponding point coordinates back One of the reference image image coordinates 'Step 460...........Construct the 3D geometric model according to the corresponding point information of one of the image coordinates of the reference image. Step 510........Selecting the desired Repair one of the reference image steps 520...........calculate the space between the reference image and other perspective images. 19 1331731 Replacement page on June 30, 1999. Step 530........ ...to the projection conversion step 540...........for the plurality of images after the projection conversion, the searching step 550 of the partial similar image that the subject image is masked... ..... Part of the masked portion of the reference image is filled according to the partial similar image. Step 610........ Select one of the reference image steps to be repaired. Step 620........... Polar line relationship between the reference image and other view images Step 630..............Image correction for other view images, so that the polar line is parallel to the scan line of the reference image. Step 640..... ... searching for the region most similar to the target image of the masked image on the polar line according to the regional similarity. Step 650...........filling the reference image with the most similar area unit Steps 660........... Convert the patched image back to the reference image coordinate system 20