TWI527432B - Repairing method of the depth image for establishing the three-dimensional image - Google Patents
Repairing method of the depth image for establishing the three-dimensional image Download PDFInfo
- Publication number
- TWI527432B TWI527432B TW102139800A TW102139800A TWI527432B TW I527432 B TWI527432 B TW I527432B TW 102139800 A TW102139800 A TW 102139800A TW 102139800 A TW102139800 A TW 102139800A TW I527432 B TWI527432 B TW I527432B
- Authority
- TW
- Taiwan
- Prior art keywords
- hole
- depth
- pixel
- value
- image
- Prior art date
Links
Landscapes
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Description
本發明是關於一種立體影像建立技術，特別是一種用以建立立體影像之深度影像的修補方法。 The present invention relates to a stereoscopic image creation technique, and more particularly to a method for repairing a depth image of a stereoscopic image.
立體顯示裝置包含裸眼式的多視角立體顯示裝置與眼鏡式的雙視角立體顯示裝置，其所需要的影像資訊不盡相同。隨著立體顯示裝置不斷推陳出新，因此有相關研究提出DIBR(Depth-Image-Based Rendering)虛擬視角映射技術。DIBR虛擬視角映射技術利用一張深度圖與一張彩色圖透過深度影像繪圖法(DIBR)合成出多視角的映射影像，以供立體顯示器建立立體影像。 The stereoscopic display device includes a naked-eye multi-view stereoscopic display device and a glasses-type dual-view stereoscopic display device, and the required image information is different. With the continuous development of stereoscopic display devices, related research has proposed DIBR (Depth-Image-Based Rendering) virtual perspective mapping technology. The DIBR virtual view mapping technology uses a depth map and a color map to construct a multi-view map image through a depth image plotting method (DIBR) for stereoscopic images to be created.
深度攝影機所拍攝的影像解析度遠不及一般的彩色攝影機，因而於深度影像中存在有無有效的像素值，以致使以此深度影像建立立體影像時，會造成原本完整的物件被切割成不同景深，造成映射影像的錯誤，或者造成物件***或變形等，而影響使用者觀賞的舒適感。 The image resolution of the depth camera is far less than that of a normal color camera. Therefore, there is a valid pixel value in the depth image, so that when the stereo image is created by using the depth image, the original object is cut into different depth of field. Causes errors in mapping images, or causes objects to split or deform, which affects the user's viewing comfort.
另一方面，在利用深度影像建立立體影像的過程中，原始的深度影像的視角下，部分的背景資訊是被前景的物件所遮蔽，而建立立體影像所需的不同視角的映射影像會將原視角中被前景的物件所遮蔽的背景將暴露出來，但此些映射影像是必須顯現被前景的物件所遮蔽的背景資訊，以致於造成虛擬視角中背景資訊不足的情況，而此些影像資訊不足的像素點則導致映射影像的破洞。 On the other hand, in the process of establishing a stereoscopic image using the depth image, part of the background information is obscured by the foreground object from the perspective of the original depth image, and the mapped image of the different perspectives required to establish the stereoscopic image will be The background of the foreground object obscured by the foreground object will be exposed, but the mapped image is the background information that must be obscured by the foreground object, so that the background information in the virtual perspective is insufficient, and the image information is insufficient. The pixels of the pixel result in a hole in the mapped image.
有鑑於此，本發明提出一種用以建立立體影像之深度影像的修補方法包含讀取深度影像、偵測深度影像的輪廓破洞、以輪廓破洞的破洞像素為中心，參考水平方向及垂直方向的複數個鄰近像素的深度值而修正破洞像素的深度值，以得到第一修正影像、偵測第一修正影像的深度破洞，並根據深度破洞周圍複數個正常像素的深度值來填補深度破洞，以得到第二修正影像、轉換第二修正影像成複數個映射影像、偵測各映射影像中的視差破洞，並且將視差破洞的位移值以及視差破洞周圍的複數個物件像素的位移值重新分配予視差破洞及此些物件像素，及檢測每一映射影像之視角破洞，並根據視角破洞的周邊像素填補視角破洞。其中，所述輪廓破洞具有單一或複數個破洞像素。其中，各映射影像具有複數個位移值，並此些位移值分別對應於第二修正影像的深度值。 In view of the above, the present invention provides a method for repairing a depth image of a stereoscopic image, including reading a depth image, detecting a contour hole of a depth image, centering on a hole pixel of a contour hole, and referring to a horizontal direction and a vertical direction. Correcting the depth value of the hole pixel by using the depth values of the plurality of adjacent pixels in the direction to obtain the first corrected image, detecting the depth hole of the first corrected image, and according to the depth values of the plurality of normal pixels around the depth hole Filling the depth hole to obtain the second corrected image, converting the second corrected image into a plurality of mapped images, detecting the parallax hole in each mapped image, and shifting the displacement value of the parallax hole and the plurality of parallax holes The displacement value of the object pixel is redistributed to the parallax hole and the pixel of the object, and the perspective hole of each mapped image is detected, and the hole is filled according to the surrounding pixels of the hole. Wherein, the contour hole has a single or a plurality of hole pixels. Each of the mapped images has a plurality of displacement values, and the displacement values respectively correspond to depth values of the second corrected image.
本發明之實施例利用不同填補方式進行深度 影像的多階段修補以提升深度影像的飽和度、避免因轉換位移值而產生影像模糊或影像物件***等現象，並且提升不同視角的映射影像的正確性，進而完整地建立前景物件及背景物件的立體影像，並提升立體影像的品質。 Embodiments of the present invention utilize different padding methods for depth Multi-stage patching of images to improve the saturation of depth images, avoid image blurring or image object splitting due to shifting displacement values, and improve the correctness of mapped images from different viewing angles, thus completely establishing foreground objects and background objects. Stereoscopic images and enhance the quality of stereoscopic images.
以下在實施方式中詳細敘述本發明之詳細特徵以及優點，其內容足以使任何熟習相關技藝者瞭解本發明之技術內容並據以實施，且根據本說明書所揭露之內容、申請專利範圍及圖式，任何熟習相關技藝者可輕易地理解本發明相關之目的及優點。 The detailed features and advantages of the present invention are set forth in the Detailed Description of the Detailed Description of the <RTIgt; </ RTI> <RTIgt; </ RTI> </ RTI> </ RTI> <RTIgt; The objects and advantages associated with the present invention can be readily understood by those skilled in the art.
100‧‧‧輪廓破洞 100‧‧‧ contour holes
10‧‧‧破洞像素 10‧‧‧ hole pixel
11-14‧‧‧鄰近像素 11-14‧‧‧ neighboring pixels
21-24‧‧‧鄰近像素 21-24‧‧‧Proximity pixels
31-34‧‧‧鄰近像素 31-34‧‧‧ neighboring pixels
41-44‧‧‧鄰近像素 41-44‧‧‧ neighboring pixels
15‧‧‧加總值 15‧‧‧ total value
25‧‧‧加總值 25‧‧‧ total value
35‧‧‧加總值 35‧‧‧ total value
45‧‧‧加總值 45‧‧‧ total value
401‧‧‧局部的第一修正影像 401‧‧‧ Partial first corrected image
402‧‧‧局部的第一修正影像 402‧‧‧Partial first corrected image
501‧‧‧局部的第二修正影像 501‧‧‧Partial second corrected image
502‧‧‧局部的第二修正影像 502‧‧‧Partial second corrected image
503‧‧‧局部的第二修正影像 503‧‧‧Partial second corrected image
61/63‧‧‧局部的位移值 61/63‧‧‧ Local displacement values
62/64‧‧‧局部的映射影像 62/64‧‧‧Partial mapping images
700‧‧‧局部的映射影像 700‧‧‧Partial mapping images
701‧‧‧局部的映射影像 701‧‧‧Partial mapping image
71/73‧‧‧周邊像素 71/73‧‧‧ peripheral pixels
72‧‧‧視角破洞 72‧‧‧ Perspective hole
80‧‧‧映射影像 80‧‧‧Map image
81‧‧‧視角破洞 81‧‧‧ Perspective hole
811‧‧‧邊緣像素 811‧‧‧Edge pixels
82‧‧‧取值範圍 82‧‧‧Value range
1-10‧‧‧差值 1-10‧‧‧ difference
R‧‧‧半徑 R‧‧‧ Radius
S10‧‧‧讀取深度影像 S10‧‧‧Read depth image
S11‧‧‧偵測深度影像的輪廓破洞 S11‧‧‧Detecting contour holes in depth images
S12‧‧‧以破洞像素為中心，參考水平方向及垂直方向的複數個鄰近像素的深度值而修正破洞像素的深度值，以得到第一修正影像 S12‧‧‧Through the hole pixel as the center, the depth value of the hole pixel is corrected by referring to the depth values of the plurality of adjacent pixels in the horizontal direction and the vertical direction to obtain the first corrected image
S13‧‧‧偵測第一修正影像的深度破洞 S13‧‧‧Detecting the depth of the first corrected image
S14‧‧‧根據深度破洞周圍複數個正常像素的深度值來填補深度破洞，以得到第二修正影像 S14‧‧‧ Fill the depth hole according to the depth values of a plurality of normal pixels around the depth hole to obtain the second corrected image
S15‧‧‧轉換第二修正影像成複數個映射影像 S15‧‧‧ Converting the second corrected image into a plurality of mapped images
S16‧‧‧偵測各映射影像中的視差破洞 S16‧‧‧Detecting parallax holes in each mapped image
S17‧‧‧將視差破洞的位移值以及視差破洞周圍的複數個物件像素的位移值重新分配予視差破洞及此些物件像素 S17‧‧‧ Redistribute the displacement value of the parallax hole and the displacement value of the plurality of object pixels around the parallax hole to the parallax hole and the pixel of the object
S18‧‧‧檢測每一映射影像之視角破洞，並根據視角破洞的周邊像素填補視角破洞 S18‧‧‧Detects the hole in the perspective of each mapped image, and fills the hole according to the surrounding pixels of the hole
第1及2圖係本發明一實施例之流程圖。 1 and 2 are flowcharts of an embodiment of the present invention.
第3圖係本發明一實施例之參考鄰近像素的深度值而修正破洞像素的深度值之示意圖。 FIG. 3 is a schematic diagram of correcting a depth value of a hole pixel by referring to a depth value of a neighboring pixel according to an embodiment of the present invention.
第4A及4B圖係本發明一實施例之消除雜訊之示意圖。 4A and 4B are schematic views showing the elimination of noise according to an embodiment of the present invention.
第5A圖係本發明一實施例之局部的第二修正影像之深度值之示意圖。 Figure 5A is a schematic diagram showing the depth values of a portion of the second corrected image of an embodiment of the present invention.
第5B圖係第5A圖之深度值轉換為位移值之示意圖。 Figure 5B is a schematic diagram of the conversion of the depth value of Figure 5A into a displacement value.
第5C圖係第5B圖之位移值量化之示意圖。 Figure 5C is a schematic diagram of the displacement value quantification of Figure 5B.
第6A圖係本發明一實施例中視差破洞調整前之示意圖。 Fig. 6A is a schematic view showing the adjustment of the parallax hole in an embodiment of the present invention.
第6B圖係第6A圖之視差破洞調整後之示意圖。 Figure 6B is a schematic diagram of the parallax hole adjustment of Figure 6A.
第7A圖係本發明一實施例中局部的映射影像之視角破洞之示意圖。 FIG. 7A is a schematic diagram showing a view hole of a partial mapped image in an embodiment of the present invention.
第7B圖係填補第7A圖之視角破洞之示意圖。 Figure 7B is a schematic diagram of filling the hole in the perspective of Figure 7A.
第8圖係本發明一實施例中局部的映射影像之視角破洞之示意圖。 Figure 8 is a schematic diagram showing the perspective hole of a partial mapped image in an embodiment of the present invention.
第1圖及第2圖為本發明一實施例之流程圖。根據本發明之用以建立立體影像之深度影像的修補方法可藉由一處理單元執行韌體或軟體演算法而在一電子裝置上實現。在一些實施例中，此電子裝置可為一電子裝置，如：個人電腦、手機或平板電腦等。其中，由電子裝置利用演算法設計平台，例如：Microsoft Visual C++搭配使用OpenCV函式庫(Open Source Computer Vision Library)，對深度影像做修補的處理。 1 and 2 are flow charts of an embodiment of the present invention. The repair method for establishing a depth image of a stereoscopic image according to the present invention can be implemented on an electronic device by performing a firmware or software algorithm by a processing unit. In some embodiments, the electronic device can be an electronic device such as a personal computer, a mobile phone, or a tablet. Among them, the electronic device uses an algorithm design platform, for example, Microsoft Visual C++ and OpenC Computer Library (Open Source Computer Vision Library) to repair the depth image.
首先，利用二影像擷取裝置擷取目標物體之深度影像及色彩影像。深度影像具有深度值，色彩影像具有色彩資訊。於此，深度影像及色彩影像的成像範圍大致相同。其中，深度影像包含前景物件及背景物件(以下統稱物件)。前景物件為目標物體的成像，而背景物件為不屬於目標物體的成像的其餘部分。在一些實施例中，影像擷取裝置擷取複數個目標物體，則深度影像中包含複數個前景物件。 First, the depth image and the color image of the target object are captured by the second image capturing device. The depth image has a depth value and the color image has color information. Here, the imaging range of the depth image and the color image is substantially the same. The depth image includes a foreground object and a background object (hereinafter collectively referred to as an object). The foreground object is the image of the target object, while the background object is the rest of the image that does not belong to the target object. In some embodiments, the image capturing device captures a plurality of target objects, and the depth image includes a plurality of foreground objects.
請參閱第1圖及第2圖，讀取深度影像(S10)。 其中，深度影像係由複數個深度值所構成。受影像擷取裝置之特性限制(如解析度或感光度等)，深度影像中會有一個或多個像素點的深度值不正確(例如：非零但深度值未對應影像擷取裝置與對應點之間的實際距離或為零)。是以，深度影像中的深度值不正確的部分(其可包括一個或多個像素點)即為深度影像中的破洞(Hole)。為方便說明，以下將深度值不正確的像素點稱之為破洞像素。 Refer to Figures 1 and 2 to read the depth image (S10). Among them, the depth image is composed of a plurality of depth values. Due to the characteristics of the image capture device (such as resolution or sensitivity), the depth value of one or more pixels in the depth image is incorrect (for example, non-zero but the depth value does not correspond to the image capture device and corresponding The actual distance between points is either zero). Therefore, the portion of the depth image in which the depth value is incorrect (which may include one or more pixels) is a hole in the depth image. For convenience of explanation, the pixel points whose depth values are not correct are hereinafter referred to as hole pixels.
接著，偵測讀取到之深度影像的輪廓破洞(S11)。輪廓破洞是深度影像中位在不同物件之間的破洞像素。於此，輪廓破洞是由一個或多個破洞像素所構成。換言之，輪廓破洞之尺寸可能為單一個像素點或複數個像素點。接著，對輪廓破洞進行修正係判斷破洞像素應歸屬的物件，而將輪廓破洞的破洞像素填入所歸屬的物件的深度值。 Then, the contour hole of the read depth image is detected (S11). A contour hole is a hole in a depth image that is positioned between different objects. Here, the contour hole is composed of one or more hole pixels. In other words, the size of the contour hole may be a single pixel or a plurality of pixels. Next, the correction of the contour hole is performed to determine the object to which the hole pixel belongs, and the hole pixel of the contour hole is filled in the depth value of the belonging object.
舉例而言，以輪廓破洞的破洞像素為中心，參考水平方向及垂直方向的複數個鄰近像素的深度值而修正此破洞像素的深度值，以得到第一修正影像(S12)。 For example, the depth value of the plurality of adjacent pixels is corrected with reference to the depth values of the plurality of adjacent pixels in the horizontal direction and the vertical direction, to obtain the first corrected image (S12).
接著，偵測第一修正影像中是否仍有未修正之破洞像素(S13)。於此，第一修正影像中之此些破洞像素通常係因為影像擷取裝置擷取目標物體時，光線不足以反射物體(包含目標物體及其後方之背景物體)之間有陰影的部分，因而在成像後，深度影像中複數個前景物件之間，或前景物件與背景物件之間呈現複數個像素點的深度值為零(或接近於零)的區塊(以下稱為深度影像的深度破洞)。 換言之，於得到第一修正影像後，則接著偵測第一修正影像的深度破洞。 Next, it is detected whether there are still uncorrected hole pixels in the first corrected image (S13). In this case, the holes in the first corrected image are usually caused by insufficient light to reflect the shadow between the object (including the target object and the background object behind it) when the image capturing device captures the target object. Therefore, after imaging, a plurality of foreground objects in the depth image, or between the foreground object and the background object, a block having a depth value of zero (or close to zero) of a plurality of pixels (hereinafter referred to as depth of the depth image) Broken hole). In other words, after the first corrected image is obtained, the depth hole of the first corrected image is detected.
對於第一修正影像的各深度破洞，根據此深度破洞周圍複數個正常像素的深度值來填補此深度破洞，以得到第二修正影像(S14)。其中，正常像素為深度破洞外圍具有正確深度值的像素點，即其深度值對應實際距離。接著，以虛擬視角轉換第二修正影像成複數個映射影像(S15)。於此，各映射影像具有複數個位移值，並且此些位移值分別對應於第二修正影像的深度值。 For each depth hole of the first corrected image, the depth hole is filled according to the depth values of the plurality of normal pixels around the depth hole to obtain a second corrected image (S14). The normal pixel is a pixel having a correct depth value on the periphery of the deep hole, that is, the depth value corresponds to the actual distance. Next, the second corrected image is converted into a plurality of mapped images by a virtual perspective (S15). Here, each map image has a plurality of displacement values, and the displacement values respectively correspond to depth values of the second corrected image.
於修正深度影像的輪廓破洞及深度破洞(S12~S14)後，已填補深度影像中大部分的破洞像素。因此，第二修正影像的像素點的深度值較為飽和。在以第二修正影像映射成複數個不同視角的映射影像時，將得以確切地轉換各像素點的深度值為映射的位移值，而避免轉換後產生影像模糊或影像***等現象。 After correcting the contour hole and depth hole (S12~S14) of the depth image, most of the hole pixels in the depth image have been filled. Therefore, the depth value of the pixel of the second corrected image is relatively saturated. When the second corrected image is mapped into a plurality of mapped images of different views, the depth value of each pixel is accurately converted to the mapped displacement value, thereby avoiding image blur or image splitting after the conversion.
因此，接著進行映射影像的修補程序。於此，偵測各映射影像中的視差破洞(S16)。其中，映射影像的視差破洞是因暴露出原始的深度影像的視角下被遮蔽的像素點而產生。換言之，在以虛擬視角將第二修正影像轉換成不同視角的映射影像時，係可能因虛擬視角而發生映射影像的影像資訊不足，而導致映射影像中有視差破洞。 Therefore, a patch for mapping the image is next performed. Here, the parallax hole in each map image is detected (S16). The parallax hole of the mapped image is generated by the pixel points that are blocked from the perspective of exposing the original depth image. In other words, when the second corrected image is converted into the mapped image of different viewing angles by the virtual perspective, the image information of the mapped image may be insufficient due to the virtual viewing angle, and the parallax is broken in the mapped image.
修補映射影像的視差破洞係將視差破洞的位移值以及視差破洞周圍的複數個物件像素的位移值重新分配予視差破洞及此些物件像素(S17)。其中，物件像素為 鄰近視差破洞之背景物件的像素點。於此，藉由將物件像素的位移值穿***視差破洞，而切割視差破洞為分佈開的視角破洞。換言之，經由視差破洞周圍的物件像素的位移值調整視差破洞為複數個尺寸較小的視角破洞。 The parallax hole of the patch map image redistributes the displacement value of the parallax hole and the displacement value of the plurality of object pixels around the parallax hole to the parallax hole and the object pixels (S17). Where the object pixel is The pixel of the background object adjacent to the parallax hole. Here, by inserting the displacement value of the object pixel into the parallax hole, the cutting parallax hole is a distributed viewing angle hole. In other words, the parallax hole is adjusted by a displacement value of the object pixel around the parallax hole into a plurality of small-diameter view hole holes.
接著，檢測每一映射影像之視角破洞，並根據視角破洞的周邊像素填補視角破洞(S18)。其中，周邊像素為視角破洞周圍有位移值的像素點。 Next, the view hole of each map image is detected, and the view hole is filled according to the surrounding pixels of the view hole (S18). The peripheral pixel is a pixel point having a displacement value around the hole of the viewing angle.
於此，每一映射影像所呈現為不相同的視角下的前景物件及背景物件。於填補視角破洞後，佈置此些不同視角的映射影像及色彩影像，而呈現前景物件及背景物件的立體影像。 Here, each of the mapped images is presented as a foreground object and a background object in different perspectives. After filling the hole in the perspective, the mapped image and the color image of the different viewing angles are arranged, and the stereoscopic image of the foreground object and the background object is presented.
在一些實施例中，於步驟S11偵測輪廓破洞係經由比較相鄰的深度值作為判斷物件相似性而偵測深度影像的輪廓破洞。其中，可以依據相減之相鄰的深度值的差值決定像素點為輪廓破洞。舉例而言，預設邊緣臨界值，並以邊緣臨界值判斷破洞像素。也就是說，若相減之相鄰的深度值的差值大於邊緣臨界值，則認定此兩像素點為輪廓破洞的破洞像素。 In some embodiments, detecting the contour hole in step S11 detects the contour hole of the depth image by comparing the adjacent depth values as determining the object similarity. Wherein, the pixel point may be determined as a contour hole according to the difference between the adjacent depth values of the subtraction. For example, the edge threshold is preset, and the hole pixel is judged by the edge threshold. That is to say, if the difference between the adjacent depth values of the subtraction is greater than the edge critical value, it is determined that the two pixel points are hole pixels of the contour hole.
在一些實施例中，在比較相鄰的深度值作為判斷物件相似性而偵測深度影像的輪廓破洞時，可採用影像的灰階值以降低資料的運算量。將彩色影像轉換成灰階影像之轉換公式如：
將深度影像轉為灰階影像。灰階值範圍為0至255。在深度影像中最近的距離定義為灰階值255，而最遠的距離則定義為灰階值0。藉由對相鄰的灰階值相減而偵測深度影像中細微的輪廓破洞的方式可以公式與條件表示為：Dx=DepthMap[j][i]-DepthMap[j][i+1] (1) Convert a depth image to a grayscale image. Grayscale values range from 0 to 255. The closest distance in the depth image is defined as the grayscale value of 255, and the farthest distance is defined as the grayscale value of zero. The method of detecting the fine contour holes in the depth image by subtracting the adjacent gray scale values can be expressed as: Dx = DepthMap [ j ][ i ]- DepthMap [ j ][ i +1] (1)
Dy=DepthMap[j][i]-DepthMap[j+1][i] (2) Dy = DepthMap [ j ][ i ]- DepthMap [ j +1][ i ] (2)
將水平的相鄰的深度值相減，並記錄為Dx，如公式(1)所示。將垂直的相鄰的深度值相減，並記錄為Dy，如公式(2)所示。其中，公式(1)與公式(2)中的DepthMap為深度圖暫存器，j為列值，i為行值。之後，依照以下的判斷條件做標籤化，其判斷條件如下： The horizontal adjacent depth values are subtracted and recorded as Dx as shown in equation (1). The vertical adjacent depth values are subtracted and recorded as Dy as shown in equation (2). Among them, DepthMap in formula (1) and formula (2) is the depth map register, j is the column value, and i is the row value. After that, the labeling is performed according to the following judgment conditions, and the judgment conditions are as follows:
(1)若Dx值大於設定的邊緣臨界值，則將邊緣標籤暫存器[j][i+1]設為「255」。其中，可預設邊緣臨界值為「10」。 (1) If the Dx value is greater than the set edge threshold, the edge tag register [j][i+1] is set to "255". Among them, the edge threshold value can be preset to be "10".
(2)若Dy值大於設定的邊緣臨界值，則將邊緣標籤暫存器[j+1][i]設為「255」。 (2) If the Dy value is greater than the set edge threshold, the edge tag register [j+1][i] is set to "255".
(3)若Dx值小於設定的邊緣負臨界值，則將邊緣標籤暫存器[j][i]設為「255」。其中，可預設邊緣負臨界值為「-10」。 (3) If the Dx value is less than the set edge negative threshold, set the edge tag register [j][i] to "255". Among them, the negative margin value of the edge can be preset to be "-10".
(4)若Dy值小於設定的邊緣負臨界值，則將邊緣標籤暫存器[j][i]設為「255」。 (4) If the Dy value is less than the set edge negative threshold, the edge tag register [j][i] is set to "255".
(5)若無滿足以上任一條件，則將標籤暫存器[j][i]設為「0」。 (5) If none of the above conditions are met, the tag register [j][i] is set to "0".
因此，經過使用判斷條件而偵測出深度影像中 的輪廓破洞。 Therefore, the depth image is detected after using the judgment condition. The silhouette of the hole.
在一些實施例中，於步驟S11偵測輪廓破洞後，由於深度影像的輪廓破洞可能過於細小，例如破洞像素為一個像素點，而易忽略修正，因而導致修正效果有限。或者，深度影像中係可能有前景物件深度值不正確，而導致不正確地決定破洞像素歸屬之物件，進而無法有效的賦予鄰近像素的深度值修正破洞像素。 In some embodiments, after the contour is broken in step S11, since the contour hole of the depth image may be too small, for example, the hole pixel is a pixel point, and the correction is easily ignored, thereby causing limited correction effect. Alternatively, in the depth image, the depth value of the foreground object may be incorrect, which may result in an incorrect determination of the object to which the hole pixel belongs, and thus the depth value of the adjacent pixel may not be effectively corrected.
因此，偵測深度影像的輪廓破洞後，可記錄(如標籤化)輪廓破洞，並放大(或稱膨脹)輪廓破洞，以致使細微的輪廓破洞可被放大，而利於修正輪廓破洞。舉例而言，偵測深度影像，並記錄深度影像中小於三個像素點的輪廓破洞，並將所記錄的破洞像素整體向外做5×5像素點的膨脹，而基於經膨脹的輪廓破洞修正破洞像素。 Therefore, after detecting the contour of the depth image, the contour hole can be recorded (such as labeling), and the contour hole can be enlarged (or expanded), so that the fine contour hole can be enlarged, and the contour is broken. hole. For example, detecting a depth image, and recording a contour hole of less than three pixels in the depth image, and expanding the recorded hole pixel as a whole 5×5 pixel point, based on the expanded contour Holes correct hole pixels.
其中，標籤化的破洞像素為深度值不正確的部份。於此，藉由相同物件中的深度值理當相近之特性(或稱相同物件的關聯性)，而將破洞像素重新給予鄰近像素的深度值。 Among them, the tagged hole pixel is the part with the incorrect depth value. Here, the hole value is re-given to the depth value of the adjacent pixel by the similarity in the depth value in the same object (or the correlation of the same object).
在一些實施例中，計算出深度影像的像素點所對應的灰階值。並且，比對出與破洞像素的灰階值相近之鄰近像素的灰階值，而以鄰近像素的灰階值填入破洞像素。舉例而言，以破洞像素的灰階值各別與上下左右每一方向的鄰近像素的灰階值相減，其中上下左右各方向的鄰近像素的灰階值與破洞像素的灰階值差值最小者即為相近之鄰近像素的灰階值，並且以相近之鄰近像素的灰階值填 入標籤化的破洞像素。 In some embodiments, the grayscale value corresponding to the pixel of the depth image is calculated. Moreover, the grayscale values of the adjacent pixels close to the grayscale values of the broken pixels are compared, and the broken pixels are filled with the grayscale values of the adjacent pixels. For example, the grayscale values of the broken pixels are respectively subtracted from the grayscale values of the adjacent pixels in each of the upper, lower, left and right directions, wherein the grayscale values of the adjacent pixels in the up, down, left, and right directions and the grayscale values of the broken pixels The smallest difference is the grayscale value of the adjacent neighboring pixels, and is filled with the grayscale values of the adjacent neighboring pixels. Into the tagged hole pixel.
第3圖為本發明一實施態樣之參考鄰近像素的深度值而修正破洞像素的深度值之示意圖。於此，藉由相同物件中的深度值理當相近的特性，並依據破洞像素周圍的鄰近像素進行物件相似度的比對，而決定破洞像素所歸屬的物件。在一些實施例中，以鄰近像素填補破洞像素還包含填補由破洞像素所膨脹之像素。 FIG. 3 is a schematic diagram of correcting a depth value of a hole pixel by referring to a depth value of a neighboring pixel according to an embodiment of the present invention. In this case, the object belonging to the hole pixel is determined by comparing the depth values in the same object to the similar characteristics and comparing the similarities of the objects according to the neighboring pixels around the hole pixel. In some embodiments, filling the hole pixel with adjacent pixels further includes filling the pixels that are expanded by the hole pixels.
也就是說，於步驟S12修正破洞像素的深度值，係藉由以輪廓破洞的其中一像素點(即破洞像素)為中心，並以破洞像素水平方向及垂直方向延伸為鄰近像素。判斷此些鄰近像素的深度值屬於相同物件或不同物件，而選擇此些鄰近像素中最靠近破洞像素的鄰近像素的深度值作為破洞像素的深度值，進而修正輪廓破洞。 That is, the depth value of the hole pixel is corrected in step S12 by centering on one of the pixel points (ie, the hole pixel) of the contour hole, and extending to the adjacent pixel in the horizontal direction and the vertical direction of the hole pixel. . It is determined that the depth values of the neighboring pixels belong to the same object or different objects, and the depth values of the neighboring pixels closest to the hole pixels among the neighboring pixels are selected as the depth values of the hole pixels, thereby correcting the contour holes.
請參閱第3圖，破洞像素10是為輪廓破洞100的其中一個像素點。修正破洞像素10的深度值時，以破洞像素10為中心，經由水平方向及垂直方向展開為鄰近像素11-14/21-24/31-34/41-44。其中，賦予水平方向及垂直方向的鄰近像素11-14/21-24/31-34/41-44複數個權重。接著，依據權重各別地計算水平方向及垂直方向的鄰近像素11-14/21-24/31-34/41-44的深度值。最後，選擇與破洞像素10的深度值最相近的方向，並以此方向的鄰近像素的深度值填補破洞像素10。 Referring to FIG. 3, the hole pixel 10 is one of the pixel points of the contour hole 100. When the depth value of the hole pixel 10 is corrected, the adjacent pixel 11-14/21-24/31-34/41-44 is developed in the horizontal direction and the vertical direction centering on the hole pixel 10. Wherein, the adjacent pixels 11-14/21-24/31-34/41-44 in the horizontal direction and the vertical direction are given a plurality of weights. Next, the depth values of the adjacent pixels 11-14/21-24/31-34/41-44 in the horizontal direction and the vertical direction are separately calculated according to the weights. Finally, the direction closest to the depth value of the hole pixel 10 is selected, and the hole pixel 10 is filled in with the depth value of the adjacent pixel in this direction.
其中，越靠近破洞像素10的鄰近像素11/21/31/41應越可能是此破洞像素10應歸屬的物件，所 以給予最靠近破洞像素10的鄰近像素11/21/31/41最高的權重，並依序遞減。 Wherein, the closer to the neighboring pixel 11/21/31/41 of the hole pixel 10, the more likely it is that the hole pixel 10 should belong to the object. The highest weight is given to the nearest pixel 11/21/31/41 closest to the hole pixel 10, and is sequentially decreased.
舉例而言，鄰近像素11/21/31/41的權重值為「1」，鄰近像素12/22/32/42的權重值為「0.75」、鄰近像素13/23/33/43的權重值為「0.5」及鄰近像素14/24/34/44的權重值為「0.25」，而破洞像素10的權重值為「2.5」。依據權重值重新計算破洞像素10的深度值為「138*2.5=345」。計算上方鄰近像素11/12/13/14的深度值，並將所得的深度值累加，而得到上方鄰近像素11/12/13/14的加總值15為「138*1+150*0.75+151*0.5+151*0.25=363.75」；再依相同方式計算下方鄰近像素21/22/23/24的加總值25為「348.75」；左方鄰近像素31/32/33/34的加總值35為「361.25」；右方鄰近像素41/42/43/44的加總值45為「340.25」。接著，比對上下左右四個方向的鄰近像素的加總值15/25/35/45，而得知破洞像素10的深度值與右方與下方的鄰近像素相近。因此，可以右方或下方的鄰近像素21/41的深度值填補破洞像素10。其中，又以下方的鄰近像素21的深度值最為接近破洞像素10的深度值，而選擇以下方的鄰近像素21的深度值為破洞像素10的深度值。 For example, the weight value of the adjacent pixel 11/21/31/41 is "1", the weight value of the adjacent pixel 12/22/32/42 is "0.75", and the weight value of the adjacent pixel 13/23/33/43. The weight value of "0.5" and the adjacent pixels 14/24/34/44 is "0.25", and the weight value of the hole pixel 10 is "2.5". The depth value of the hole pixel 10 is recalculated based on the weight value as "138*2.5=345". Calculate the depth value of the upper neighboring pixels 11/12/13/14, and accumulate the obtained depth values, and obtain the total value 15 of the upper neighboring pixels 11/12/13/14 as "138*1+150*0.75+ 151*0.5+151*0.25=363.75”; then calculate the total value 25 of the adjacent pixels 21/22/23/24 below as “348.75” in the same way; the sum of the adjacent pixels 31/32/33/34 on the left The value 35 is "361.25"; the sum value 45 of the adjacent pixel 41/42/43/44 on the right is "340.25". Next, the total value of the adjacent pixels in the four directions of up, down, left, and right is compared to 15/25/35/45, and it is known that the depth value of the hole pixel 10 is close to the adjacent pixel on the right and below. Therefore, the hole pixel 10 can be filled with the depth value of the adjacent pixel 21/41 on the right or below. The depth value of the adjacent pixel 21 is closest to the depth value of the hole pixel 10, and the depth value of the adjacent pixel 21 is selected as the depth value of the hole pixel 10.
換言之，在決定破洞像素所歸屬的物件時，可預設臨界值，並且於各別計算破洞像素周圍的鄰近像素的深度值後，若其中一方向的鄰近像素的深度值(或深度值的加總值)與破洞像素的差值小於預設之臨界值，則認定 此方向的物件即可以是破洞像素所歸屬的物件，而選擇以此方向的鄰近像素的深度值填補破洞像素。其中，可預設臨界值為「20」。 In other words, when determining the object to which the hole pixel belongs, the threshold value may be preset, and after each depth value of the neighboring pixel around the hole pixel is calculated, if the depth value (or depth value) of the adjacent pixel in one direction is If the difference between the summed value and the hole pixel is less than the preset threshold, then it is determined The object in this direction can be the object to which the hole pixel belongs, and the depth value of the adjacent pixel in this direction is selected to fill the hole pixel. Among them, the threshold value can be preset to be "20".
在一些實施例中，破洞像素周圍的鄰近像素於三個方向上皆屬於第一物件，然而相鄰此破洞像素的第四方向的鄰近像素是屬於第二物件，以致於第四方向的鄰近像素的深度值的加總值會與其他三個方向的加總值之間有較大的差異。因此，選擇以相反於第二物件的方向之鄰近像素的深度值填補破洞像素。 In some embodiments, adjacent pixels around the hole pixel belong to the first object in three directions, but adjacent pixels in the fourth direction adjacent to the hole pixel belong to the second object, so that the fourth direction The sum of the depth values of adjacent pixels will be significantly different from the sum of the other three directions. Therefore, the hole pixels are selected to be filled with depth values of adjacent pixels in a direction opposite to the direction of the second object.
換言之，可預設臨界值，並且於各別計算破洞像素周圍的鄰近像素的深度值後，若其中一方向的鄰近像素的深度值(或深度值的加總值)與破洞像素的差值大於預設之臨界值，則認定此方向的物件係與破洞像素較無關聯。因此，選擇以相反於此方向的鄰近像素的深度值填補破洞像素。其中，可預設臨界值為「20」。 In other words, the threshold value may be preset, and after separately calculating the depth values of the neighboring pixels around the hole pixel, if the depth value (or the sum of the depth values) of the adjacent pixels in one direction is different from the hole pixel If the value is greater than the preset threshold, then the object in this direction is considered to be less associated with the hole pixel. Therefore, the hole values of the neighboring pixels opposite to this direction are selected to fill the hole pixels. Among them, the threshold value can be preset to be "20".
在一些實施例中，於步驟S12修正破洞像素的深度值後，藉由中值濾波器消除修正輪廓破洞後的深度影像(即第一修正影像)中的雜訊。於此，提高修正輪廓破洞的正確性。 In some embodiments, after the depth value of the hole pixel is corrected in step S12, the noise in the depth image (ie, the first corrected image) after the contour hole is corrected is removed by the median filter. Here, the correctness of correcting the contour hole is improved.
請參閱第4A圖，於深度影像修正輪廓破洞後，採用3×3遮罩的中值濾波器消除雜訊。中值濾波器偵測局部的第一修正影像401，其遮罩內的所有像素值為「1、4、7、3、50、4、5、6及4」。在進行濾波的過程中，中值濾波器偵測到遮罩內的所有像素值中出現與大部分的像素 值差異甚多的像素點，例如：像素值「50」的像素點，則將此像素點視為雜訊。接著，依據遮罩內除了雜訊之外的其他像素值取代雜訊的像素值。 Please refer to Figure 4A. After the depth image is corrected for the contour hole, the 3×3 masked median filter is used to eliminate the noise. The median filter detects the local first corrected image 401, and all pixel values in the mask are "1, 4, 7, 3, 50, 4, 5, 6, and 4". During the filtering process, the median filter detects the presence of most of the pixels in all pixel values within the mask. A pixel with a large difference in value, for example, a pixel with a pixel value of "50", is regarded as a noise. Then, the pixel values of the noise are replaced according to other pixel values in the mask other than the noise.
請參閱第4B圖，局部的第一修正影像402中，中值濾波器係經由排除雜訊(如第4A圖之像素值「50」的像素點)，且以其他像素值排序而挑選出中間值「4」，而取代雜訊的像素值，進而濾除雜訊。中值濾波器為非線性濾波器，由於中值濾波器為所屬技術領域中通常知識者所熟知，在此不再贅述。 Referring to FIG. 4B, in the partial first corrected image 402, the median filter selects the middle by excluding noise (such as the pixel value of the pixel value "50" in FIG. 4A) and sorting by other pixel values. The value is "4", which replaces the pixel value of the noise, thereby filtering out the noise. The median filter is a non-linear filter, and since the median filter is well known to those of ordinary skill in the art, it will not be repeated here.
在一些實施例中，於步驟S12的破洞像素經由填入深度值相近的鄰近像素的深度值而歸屬於物件，並形成第一修正影像。然而，於步驟S13，第一修正影像中，係可能有深度值為零(或接近於零)的深度破洞。於此，以深度破洞之中心點向外擴張擷取範圍。接著，選擇以擷取範圍中正常像素的深度值相差最大之兩深度值計算平均值，並且以平均值填補擷取範圍。因此，填補第一修正影像的深度破洞，以得到第二修正影像。 In some embodiments, the hole pixel in step S12 is attributed to the object by filling in depth values of adjacent pixels having similar depth values, and forms a first corrected image. However, in step S13, in the first corrected image, there may be a depth hole having a depth value of zero (or close to zero). Here, the range of the depth is expanded outward by the center of the depth hole. Next, the average value is calculated by the two depth values having the largest difference in the depth values of the normal pixels in the extraction range, and the extraction range is filled with the average value. Therefore, the depth hole of the first corrected image is filled to obtain the second corrected image.
換言之，於步驟S13至S14，偵測到第一修正影像的深度破洞時，可以深度破洞的單一像素點或者以深度破洞的中心點向外擴張擷取範圍(例如5×5的視窗)。接著，以擷取範圍中的正常像素的深度值的最大值與最小值計算平均值。並且，使用八相鄰的方法，即以深度破洞的中心點或者單一像素點為九宮格的中心，並將平均值填入九宮格的外圍，而填補第一修正影像的深度破洞。因此， 得到第二修正影像。 In other words, in steps S13 to S14, when the depth of the first corrected image is detected, the single pixel point of the hole may be deeply broken or the center point of the depth hole may be expanded outward (for example, a window of 5×5) ). Next, the average value is calculated by the maximum value and the minimum value of the depth values of the normal pixels in the extraction range. Moreover, the eight-adjacent method is used, that is, the center point of the deep hole or the single pixel point is the center of the nine-square grid, and the average value is filled in the periphery of the nine-square grid to fill the depth hole of the first corrected image. therefore, A second corrected image is obtained.
在一些實施例中，於步驟S15中，係可經由線性轉換的方式，以虛擬視角轉換第二修正影像成複數個映射影像。於此，可將影像轉換為灰階以降低運算量。線性轉換主要目的是將深度值轉換成實際的深度距離，再使用實際的深度距離與相似三角形的概念計算出虛擬視角映射的位移值。由於，使用於映射技術之相似三角形的概念為所屬技術領域中通常知識者所熟知，在此不再贅述。 In some embodiments, in step S15, the second corrected image is converted into a plurality of mapped images by a virtual perspective. Here, the image can be converted to gray scale to reduce the amount of calculation. The main purpose of the linear transformation is to convert the depth value into the actual depth distance, and then calculate the displacement value of the virtual perspective map using the actual depth distance and the concept of a similar triangle. Since the concept of similar triangles used in mapping techniques is well known to those of ordinary skill in the art, no further details are provided herein.
線性轉換公式如下所示：
在公式(4)中，D為像素點的深度灰階值；PZ為視覺目標的深度位置；ZF為最遠虛擬視角成像位置；及ZN為最近虛擬視角成像位置。在公式(5)中，XR為第二修正影像的視角中視覺的成像位置；X i 為i視角的映射影像中視覺的成像位置；b為兩眼間的瞳孔的距離，根據統計大約為6.5公分；d為使用者與立體顯示器的距離；及ρ為將顯示器寬度轉換成像素值的比例值。經過以上兩個公式的線性轉換，即可將深度圖中的灰階值轉換為像素的位移值。 In formula (4), D is the depth grayscale value of the pixel point; P Z is the depth position of the visual target; Z F is the farthest virtual perspective imaging position; and Z N is the nearest virtual perspective imaging position. In the formula (5), X R is the imaging position of the vision in the angle of view of the second corrected image; X i is the imaging position of the vision in the mapped image of the i angle of view; b is the distance of the pupil between the eyes, according to statistics 6.5 cm; d is the distance between the user and the stereo display; and ρ is the ratio of the display width to the pixel value. Through the linear transformation of the above two formulas, the grayscale value in the depth map can be converted into the displacement value of the pixel.
在一些實施例中，以第二修正影像的視角當作中間視角。其中，中間視角的i值可以「0」表示。在一些 實施例中，需要產生多張不同視角的映射影像時，則左側虛擬視角i可為負值，右側虛擬視角i可為正值。 In some embodiments, the perspective of the second corrected image is taken as the intermediate viewing angle. Among them, the i value of the intermediate view can be represented by "0". In some In an embodiment, when a plurality of mapped images of different viewing angles need to be generated, the left virtual perspective i may be a negative value, and the right virtual perspective i may be a positive value.
舉例而言，請參閱第5A圖，在局部的第二修正影像501中，深度值所對應的灰階值的範圍為「202」至「203」。使用線性轉換的方法經過公式(4)及公式(5)的運算後，轉換出局部的第二修正影像502中的每一像素點的位移值(如第5B圖所示)。其中，像素點的位移值可稱作視差(Disparity or Parallax)。 For example, referring to FIG. 5A, in the partial second corrected image 501, the grayscale value corresponding to the depth value ranges from "202" to "203". After the operation of the formula (4) and the formula (5) is performed by the linear conversion method, the displacement value of each pixel in the partial second corrected image 502 is converted (as shown in FIG. 5B). Among them, the displacement value of the pixel can be called Disparity or Parallax.
請參閱第5C圖，在一些實施例中，由於映射時針對每一像素點的位移值為整數值，所以於轉換像素點的位移值後，量化位移值(即局部的第二修正影像503)。 Referring to FIG. 5C, in some embodiments, since the displacement value for each pixel point during mapping is an integer value, after shifting the displacement value of the pixel point, the displacement value is quantized (ie, the local second corrected image 503). .
於此，依據位移值將第二修正影像的像素點映射成虛擬視角的映射點，而映射出不同視角的映射影像。 In this case, the pixel points of the second corrected image are mapped to the mapping points of the virtual perspective according to the displacement values, and the mapped images of different viewing angles are mapped.
在一些實施例中，於步驟S15將第二修正影像轉換成映射影像後，修改線性轉換後的位移值來調整各映射影像中的視差破洞。其中，修改位移值是依據位移值較高者調整位移值較低者。由於，視差破洞並不具有有效的位移值(如位移值等於零)，所以背景物件的物件像素的位移值會高於視差破洞的位移值。因此，將背景物件像素的位移值分別穿插於視差破洞的像素點。 In some embodiments, after converting the second corrected image into the mapped image in step S15, the linearly transformed displacement value is modified to adjust the parallax hole in each mapped image. Among them, the modified displacement value is based on the higher displacement value to adjust the lower displacement value. Since the parallax hole does not have a valid displacement value (for example, the displacement value is equal to zero), the displacement value of the object pixel of the background object is higher than the displacement value of the parallax hole. Therefore, the displacement values of the background object pixels are respectively inserted in the pixel points of the parallax hole.
也就是說，於步驟S17中，將視差破洞周圍的物件像素的位移值重新分配予視差破洞及物件像素，而得以將視差破洞經由穿插物件像素調整成較小的區塊。因此，調整視差破洞為視角破洞。 That is to say, in step S17, the displacement values of the object pixels around the parallax hole are redistributed to the parallax hole and the object pixel, and the parallax hole can be adjusted into a smaller block via the interpolated object pixel. Therefore, adjust the parallax hole to make a hole in the perspective.
第6A圖及第6B圖為修改位移值以調整視差破洞為視角破洞之示意圖。 6A and 6B are schematic diagrams of modifying the displacement value to adjust the parallax hole as a viewing angle.
請參閱第6A圖，將第二修正影像轉換為映射的位移值後，偵測局部的位移值61(如以掃描位移值的方式)。接著，將鄰近的位移值做相減，差值1/2/3/4的值為「0」，差值5的值為「5」。其中，差值5將在局部的映射影像62中產生五個像素點的視差破洞。因此，修改鄰近的位移值而調整差值。 Referring to FIG. 6A, after converting the second corrected image into the mapped displacement value, the local displacement value 61 is detected (eg, by scanning the displacement value). Next, the adjacent displacement values are subtracted, the value of the difference 1/2/3/4 is "0", and the value of the difference 5 is "5". The difference 5 will produce a parallax hole of five pixel points in the local mapped image 62. Therefore, the adjacent displacement values are modified to adjust the difference.
請參閱第6B圖，修改鄰近的位移值而形成局部的位移值63係依據位移值較高者調整位移值較低者。因此，依據相鄰的位移值調整視差破洞為視角破洞。也就是說，在偵測出會產生視差破洞的差值5(可見第6A圖)後，將差值平均分配予相鄰之位移值較低的像素點，例如：將連續五個相鄰之位移值較低的像素點依序增加一個位移值(如第6B圖)，而遞增位移值較低者的位移值，進而調整鄰近的位移值之間僅差距一位移值，以致使差值6/7/8/9/10相同。因此，局部的映射影像64之視差破洞得以被切割為較小的視角破洞。 Referring to FIG. 6B, the displacement value is modified to form a local displacement value. 63 is based on the fact that the displacement value is higher, and the displacement value is lower. Therefore, the parallax hole is adjusted according to the adjacent displacement value to make a hole in the viewing angle. That is to say, after detecting the difference 5 (see Fig. 6A) which will cause the parallax hole, the difference is evenly distributed to the adjacent pixel points with lower displacement values, for example, five consecutive neighbors will be present. The pixel points with lower displacement values are sequentially added with a displacement value (such as FIG. 6B), and the displacement values of the lower displacement values are increased, thereby adjusting the difference between the adjacent displacement values and only the displacement value, so that the difference is 6/7/8/9/10 is the same. Therefore, the parallax hole of the local mapped image 64 can be cut into smaller viewing angle holes.
在一些實施例中，於步驟S17中，將視差破洞的位移值以及視差破洞周圍的物件像素的位移值重新分配予視差破洞及物件像素時，使用暫存器記錄鄰近的位移值相減後的差值，並依據差值調整視差破洞的像素點的位移值。 In some embodiments, in step S17, when the displacement value of the parallax hole and the displacement value of the object pixel around the parallax hole are redistributed to the parallax hole and the object pixel, the temporary displacement value is recorded by using the register. The reduced difference, and the displacement value of the pixel of the parallax hole is adjusted according to the difference.
在一些實施例中，於步驟S17將視差破洞的位 移值以及視差破洞周圍的物件像素的位移值重新分配予視差破洞及物件像素後，可偵測映射影像中的像素點之重疊區域，並比較重疊區域的像素點的位移值，以確認所調整之位移值的正確性。 In some embodiments, the position of the parallax hole is broken in step S17. After the shift value and the displacement value of the object pixel around the parallax hole are redistributed to the parallax hole and the object pixel, the overlapping area of the pixel in the mapped image can be detected, and the displacement value of the pixel of the overlapping area is compared to confirm The correctness of the adjusted displacement value.
由於，視差破洞的像素點不具有效的位移值。因此，若所***的物件像素的像素點與視差破洞的像素點重疊，且視差破洞的像素點位在物件像素的像素點前方(於視覺上先看到的方向)，因而物件像素的像素點的位移值被遮蔽而無法顯現，導致於視覺上出現前後景物件顛倒的情形。 Because the pixel of the parallax hole does not have an effective displacement value. Therefore, if the pixel point of the inserted object pixel overlaps with the pixel point of the parallax hole, and the pixel point of the parallax hole is in front of the pixel point of the object pixel (the direction first seen visually), the object pixel The displacement value of the pixel is masked and cannot be visualized, resulting in visually occurring situations in which the front and back scene objects are reversed.
於此，當重疊區域中在前方的位移值小於後方的位移值，表示映射影像會先呈現位移值較小者，也就是映射影像所呈現的是較遠的物件，因而導致前後景物件顛倒情形。因此，應該要以位移值較大者取代位移值較小者。當重疊區域中的像素點是以位移值較大者覆蓋位移值較小者時，像素點的位移值即可依前景至後景的順序，而顯現正確的映射影像。 In this case, when the displacement value in the front of the overlap region is smaller than the displacement value in the rear, it means that the map image first presents a displacement value that is smaller, that is, the map image presents a far object, thereby causing the front and back scene objects to be reversed. . Therefore, it should be replaced by the one with the smaller displacement value. When the pixel point in the overlapping area is smaller than the displacement value, the displacement value of the pixel point can show the correct mapping image according to the order of the foreground to the back scene.
同理可知，如果重疊區域中在前方的位移值大於後方的位移值，表示映射影像會先呈現位移值較大者，也就是映射影像所呈現的是較近的物件，則顯現正確的映射影像。 Similarly, if the displacement value in the front of the overlap region is greater than the displacement value in the rear, it means that the map image will first exhibit a larger displacement value, that is, the map image presents a closer object, and the correct map image appears. .
在一些實施例中，如上述偵測映射影像中的像素點之重疊區域時，可使用深度暫存法(Z-buffer)的原理偵測錯誤的位移值，並置換位移值為正確的順序。舉例而 言，在以虛擬視角映射映射影像時，預備存放映射影像的第一儲存空間，及作為暫存的第二儲存空間。在映射前，把第二儲存空間中所有的值設定成「0」。接著，以第二儲存空間記錄映射影像中所有像素點的位移值，在映射過程中依序將所要映射的像素點比較第二儲存空間中對應此映射的像素點的位移值。如果偵測到第二儲存空間中的像素點的位移值小於目前要映射的位移值，表示目前要映射的是比較近的物件。因此，要以目前的位移值覆蓋此像素點。由於，深度暫存法(Z-buffer)的原理為所屬技術領域中具通常知識者可熟知，在此不再贅述。 In some embodiments, when the overlapping regions of the pixel points in the mapped image are detected as described above, the principle of the depth temporary storage method (Z-buffer) can be used to detect the erroneous displacement value and replace the displacement value with the correct order. For example In other words, when the mapped image is mapped by the virtual perspective, the first storage space of the mapped image is prepared, and the second storage space is temporarily stored. Set all values in the second storage space to "0" before mapping. Then, the displacement values of all the pixels in the mapped image are recorded in the second storage space, and the pixel points to be mapped are sequentially compared in the mapping process to the displacement values of the pixels corresponding to the mapping in the second storage space. If it is detected that the displacement value of the pixel in the second storage space is smaller than the displacement value currently to be mapped, it indicates that the object that is currently being mapped is relatively close. Therefore, this pixel is to be overwritten with the current displacement value. Since the principle of the Z-buffer is well known to those skilled in the art, no further details are provided herein.
在一些實施例中，於步驟S18檢測每一映射影像之視角破洞時，若視角破洞之寬度小於預設之寬度值(例如兩個像素點)，則以視角破洞周圍的周邊像素的位移值填補視角破洞。 In some embodiments, when the viewing angle of each mapped image is detected in step S18, if the width of the viewing hole is smaller than a preset width value (for example, two pixel points), the surrounding pixels around the hole are broken by the viewing angle. The displacement value fills the hole in the perspective.
請參閱第7A圖，在局部的映射影像700中，視角破洞72占每列像素點中的兩個像素點。因此，在填補視角破洞72時，依據左方的周邊像素71與右方的周邊像素73搜尋位移值。 Referring to FIG. 7A, in the partial mapped image 700, the viewing angle hole 72 occupies two pixels in each column of pixels. Therefore, when the viewing angle hole 72 is filled, the displacement value is searched for based on the left peripheral pixel 71 and the right peripheral pixel 73.
請參閱第7B圖，視角破洞72左方的像素點填入左方的周邊像素71的位移值，及視角破洞72右方的像素點填入右方的周邊像素73的位移值，而填補視角破洞72，形成局部的映射影像701。 Referring to FIG. 7B, the pixel value on the left side of the viewing hole 72 fills the displacement value of the peripheral pixel 71 on the left side, and the pixel point on the right side of the viewing angle hole 72 fills in the displacement value of the peripheral pixel 73 on the right side. The view hole 72 is filled to form a partial map image 701.
在一些實施例中，於步驟S18檢測每一映射影像之視角破洞時，若視角破洞之寬度大於預設之寬度值(例 如大於兩個像素點)時，可以線性方式填補視角破洞。 In some embodiments, when the viewing angle of each mapped image is detected in step S18, if the width of the viewing hole is larger than the preset width value (example) If it is larger than two pixels, the hole can be filled in a linear manner.
請參閱第8圖，在局部的映射影像80中，視角破洞81所占像素點大於兩個像素點。填補視角破洞81可藉由以視角破洞81之其中一邊緣像素811為圓心，並以半徑R所圈選的範圍作為取值範圍82來取得周邊像素的位移值。接著，計算取值範圍82內之位移值的平均位移值而填補邊緣像素811。之後，再依相同的方式取得視角破洞81其他的像素點的位移值。 Referring to FIG. 8, in the partial mapped image 80, the viewing angle hole 81 occupies more than two pixel points. The filling of the viewing angle hole 81 can obtain the displacement value of the peripheral pixel by taking one of the edge pixels 811 of the viewing angle hole 81 as the center and the range circled by the radius R as the value range 82. Next, the average displacement value of the displacement value in the range of values 82 is calculated to fill the edge pixel 811. Thereafter, the displacement values of the other pixel points of the viewing angle hole 81 are obtained in the same manner.
其中，以線性方式填補視角破洞81時，係先填補視角破洞81的邊緣部分的像素點，例如：邊緣像素811，之後再依序對內部的像素點做填補。 When the viewing angle hole 81 is filled in a linear manner, the pixel points of the edge portion of the viewing angle hole 81 are first filled, for example, the edge pixel 811, and then the internal pixel points are sequentially filled.
在一些實施例中，可分別賦予取值範圍內的位移值各別的權重來計算取值範圍內之位移值的平均位移值。例如：對於待填補的像素點為圓心，給予距離圓心愈遠的位移值的權重愈低，相反的，距離愈靠近圓心的位移值的權重越高。 In some embodiments, the weights of the displacement values within the range of values can be individually assigned to calculate the average displacement value of the displacement values within the range of values. For example, if the pixel to be filled is the center of the circle, the weight of the displacement value that is further away from the center of the circle is lower. On the contrary, the weight of the displacement value closer to the center of the circle is higher.
其中，線性方式可例如Telea演算法。由於，Telea影像填補技術為所屬技術領域中具通常知識者可熟知，在此不再贅述。 Among them, the linear mode can be, for example, a Telea algorithm. Since the Telea image filling technique is well known to those skilled in the art, no further details are provided herein.
在一些實施例中，於步驟S18後，已填補映射影像的視角破洞。其中，每一映射影像所呈現為不相同的視角下的前景物件及背景物件。此後，因應不同的立體顯示器的規格，將不同視角的映射影像的像素點重新排列以佈置不同視角的映射影像，而得以呈現前景物件及背景物 件的立體影像。 In some embodiments, after step S18, the angle of view of the mapped image has been filled. Each of the mapped images is presented as a foreground object and a background object at different viewing angles. Thereafter, in accordance with the specifications of different stereoscopic displays, the pixels of the mapped images of different viewing angles are rearranged to arrange the mapped images of different viewing angles, thereby presenting foreground objects and background objects. A stereo image of the piece.
其中，不同視角的映射影像的佈置的方式例如：使用時間與空間分工的方式進行雙視角與多視角的合成。舉例而言，在使用眼鏡式立體顯示裝置時，以雙視角格式搭配時間的間隔，或以雙視角格式搭配空間的切割，而可達到立體視覺效果。 The manner in which the mapped images of different viewing angles are arranged is, for example, a combination of a dual view and a multi-view using a time and space division. For example, when a glasses-type stereoscopic display device is used, a stereoscopic effect can be achieved by using a double-view format with time interval or a double-view format with space cutting.
或者，可應用裸眼式立體顯示器(如柱狀透鏡式立體顯示裝置)佈置不同視角的映射影像並顯示立體影像。例如：以九張不同視角的映射影像顯示三維立體影像的物件。其中，將每個像素點分成紅綠藍三個子像素。原本每個像素點都是同一視角的子像素。經過重新合成後的多視角影像，在同一像素點的子像素則包含三個不同視角的子像素點，而產生多視角立體影像。 Alternatively, a naked-eye stereoscopic display (such as a lenticular stereoscopic display device) may be applied to arrange mapped images of different viewing angles and display stereoscopic images. For example, an object that displays a three-dimensional image with nine different views of the mapped image. Among them, each pixel is divided into three sub-pixels of red, green and blue. Originally each pixel is a sub-pixel of the same viewing angle. After the re-synthesized multi-view image, the sub-pixels at the same pixel point include three sub-pixel points of different viewing angles to generate a multi-view stereoscopic image.
本發明之實施例利用不同填補方式進行深度影像的多階段修補以提升深度影像的飽和度、避免因轉換位移值而產生影像模糊或影像物件***等現象，並且提升不同視角的映射影像的正確性，進而完整地建立前景物件及背景物件的立體影像，並提升立體影像的品質。 The embodiment of the present invention uses different padding methods to perform multi-stage patching of depth images to improve the saturation of the depth image, avoid image blurring or image object splitting due to shifting displacement values, and improve the correctness of the mapped images of different viewing angles. , and then complete the stereoscopic image of the foreground object and the background object, and improve the quality of the stereo image.
雖然本發明的技術內容已經以較佳實施例揭露如上，然其並非用以限定本發明，任何熟習此技藝者，在不脫離本發明之精神所作些許之更動與潤飾，皆應涵蓋於本發明的範疇內，因此本發明之保護範圍當視後附之申請專利範圍所界定者為準。 Although the technical content of the present invention has been disclosed in the above preferred embodiments, it is not intended to limit the present invention, and any modifications and refinements made by those skilled in the art without departing from the spirit of the present invention are encompassed by the present invention. The scope of protection of the present invention is therefore defined by the scope of the appended claims.
S10‧‧‧讀取深度影像 S10‧‧‧Read depth image
S11‧‧‧偵測深度影像的輪廓破洞 S11‧‧‧Detecting contour holes in depth images
S12‧‧‧以破洞像素為中心，參考水平方向及垂直方向的複數個鄰近像素的深度值而修正破洞像素的深度值，以得到第一修正影像 S12‧‧‧Through the hole pixel as the center, the depth value of the hole pixel is corrected by referring to the depth values of the plurality of adjacent pixels in the horizontal direction and the vertical direction to obtain the first corrected image
S13‧‧‧偵測第一修正影像的深度破洞 S13‧‧‧Detecting the depth of the first corrected image
S14‧‧‧根據深度破洞周圍複數個正常像素的深度值來填補深度破洞，以得到第二修正影像 S14‧‧‧ Fill the depth hole according to the depth values of a plurality of normal pixels around the depth hole to obtain the second corrected image
S15‧‧‧轉換第二修正影像成複數個映射影像 S15‧‧‧ Converting the second corrected image into a plurality of mapped images
S16‧‧‧偵測各映射影像中的視差破洞 S16‧‧‧Detecting parallax holes in each mapped image
S17‧‧‧將視差破洞的位移值以及視差破洞周圍的複數個物件像素的位移值重新分配予視差破洞及此些物件像素 S17‧‧‧ Redistribute the displacement value of the parallax hole and the displacement value of the plurality of object pixels around the parallax hole to the parallax hole and the pixel of the object
S18‧‧‧檢測每一映射影像之視角破洞，並根據視角破洞的周邊像素填補視角破洞 S18‧‧‧Detects the hole in the perspective of each mapped image, and fills the hole according to the surrounding pixels of the hole
Claims (9)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| TW102139800A TWI527432B (en) | 2013-11-01 | 2013-11-01 | Repairing method of the depth image for establishing the three-dimensional image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| TW102139800A TWI527432B (en) | 2013-11-01 | 2013-11-01 | Repairing method of the depth image for establishing the three-dimensional image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| TW201519634A TW201519634A (en) | 2015-05-16 |
| TWI527432B true TWI527432B (en) | 2016-03-21 |
Family
ID=53721113
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| TW102139800A TWI527432B (en) | 2013-11-01 | 2013-11-01 | Repairing method of the depth image for establishing the three-dimensional image |
Country Status (1)
| Country | Link |
|---|---|
| TW (1) | TWI527432B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108076338B (en) * | 2016-11-14 | 2022-04-08 | 北京三星通信技术研究有限公司 | Image visual processing method, device and equipment |
-
2013
- 2013-11-01 TW TW102139800A patent/TWI527432B/en not_active IP Right Cessation
Also Published As
| Publication number | Publication date |
|---|---|
| TW201519634A (en) | 2015-05-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9407896B2 (en) | Multi-view synthesis in real-time with fallback to 2D from 3D to reduce flicker in low or unstable stereo-matching image regions | |
| US9684964B2 (en) | Image processing apparatus and image processing method for determining disparity | |
| US20120169722A1 (en) | Method and apparatus generating multi-view images for three-dimensional display | |
| CN107750370B (en) | Method and apparatus for determining a depth map for an image | |
| TWI489418B (en) | Parallax Estimation Depth Generation | |
| US20130010073A1 (en) | System and method for generating a depth map and fusing images from a camera array | |
| US20110234765A1 (en) | Image processing apparatus, image processing method, image processing program, and compound eye digital camera | |
| EP2175663A1 (en) | Image processing apparatus and method | |
| WO2012071356A1 (en) | Depth estimation based on global motion | |
| US9462251B2 (en) | Depth map aligning method and system | |
| TWI439961B (en) | Conversion algorithm for voids generated after converting 2D images | |
| JP2011141710A (en) | Device, method and program for estimating depth | |
| JP4296617B2 (en) | Image processing apparatus, image processing method, and recording medium | |
| KR102490985B1 (en) | Apparatus and method for processing depth maps | |
| KR20140001358A (en) | Method and apparatus of processing image based on occlusion area filtering | |
| TWI527432B (en) | Repairing method of the depth image for establishing the three-dimensional image | |
| KR102122523B1 (en) | Device for correcting depth map of three dimensional image and method for correcting the same | |
| US9380285B2 (en) | Stereo image processing method, stereo image processing device and display device | |
| JP2013090272A (en) | Video processing device, video processing method and video display device | |
| CN116569214A (en) | Apparatus and method for processing depth map | |
| US9008413B2 (en) | Processing method for a pair of stereo images | |
| JP5824896B2 (en) | Image processing apparatus and method, and program | |
| JP5459231B2 (en) | Pseudo stereoscopic image generation apparatus, pseudo stereoscopic image generation program, and pseudo stereoscopic image display apparatus | |
| Liu | Improving forward mapping and disocclusion inpainting algorithms for depth-image-based rendering and geomatics applications | |
| JP5395884B2 (en) | Video processing device, video processing method, and video display device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MM4A | Annulment or lapse of patent due to non-payment of fees |