TWI612495B - Process and device for capturing and rendering a panoramic or stereoscopic stream of images - Google Patents

Process and device for capturing and rendering a panoramic or stereoscopic stream of images Download PDF

Info

Publication number
TWI612495B
TWI612495B TW102141393A TW102141393A TWI612495B TW I612495 B TWI612495 B TW I612495B TW 102141393 A TW102141393 A TW 102141393A TW 102141393 A TW102141393 A TW 102141393A TW I612495 B TWI612495 B TW I612495B
Authority
TW
Taiwan
Prior art keywords
image
pixel
panoramic
capture
item
Prior art date
Application number
TW102141393A
Other languages
Chinese (zh)
Other versions
TW201435792A (en
Inventor
李察 歐利爾
Original Assignee
吉羅普提克公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 吉羅普提克公司 filed Critical 吉羅普提克公司
Publication of TW201435792A publication Critical patent/TW201435792A/en
Application granted granted Critical
Publication of TWI612495B publication Critical patent/TWI612495B/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H04N13/211—Image signal generators using stereoscopic image cameras using a single 2D image sensor using temporal multiplexing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/282—Image signal generators for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/296—Synchronisation thereof; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Studio Devices (AREA)
  • Stereoscopic And Panoramic Photography (AREA)
  • Cameras In General (AREA)

Abstract

要擷取並呈顯一場景的全景或立體影像串流，可利用至少一影像擷取裝置(Ci)按照像素格式對一場景之至少兩個不同的影像進行多次接續擷取作業，該等影像可為重疊或不重疊，該等影像擷取作業是依一頻率(F)進行，而此頻率定義在兩次接續擷取作業之起始間的擷取時間(T)。對於各次擷取作業，(a)對所擷取影像的像素進行數位處理，藉此利用該等像素以形成一最終全景或立體影像，而處理時間是短於或等於該擷取時間(T)，以及(b)在短於或等於該擷取時間之時間間隔的過程裡產生一最終且先前形成的全景或立體影像。各個所擷取影像之各個像素的數位處理(a)至少包含保留或拋除該像素，並且當保留該像素時，將其依一對於該最終全景或立體影像上之各個位置的預定義加權因數(W)指配於該最終全景或立體影像上的其一或多個位置。 To capture and present a panoramic or stereoscopic image stream of a scene, at least one image capture device (C i ) can be used to perform multiple consecutive capture operations on at least two different images of a scene in pixel format. The images can be overlapping or non-overlapping. These image acquisition operations are performed at a frequency (F), and this frequency defines the acquisition time (T) between the start of two consecutive acquisition operations. For each capture operation, (a) digitally process the pixels of the captured image, thereby using those pixels to form a final panoramic or stereoscopic image, and the processing time is shorter than or equal to the capture time (T ), And (b) produce a final and previously formed panoramic or stereo image during a time interval shorter than or equal to the acquisition time. Digital processing of each pixel of each captured image (a) At least includes retaining or discarding the pixel, and when the pixel is retained, it is pre-defined weighting factors for each position on the final panoramic or stereo image (W) assigned to one or more positions on the final panoramic or stereoscopic image.

Description

用於擷取與呈顯全景或立體影像串流之方法與裝置 Method and device for capturing and displaying panoramic or stereo image stream

本發明是關於一種用於擷取並呈顯立體或全景影像串流的方法與裝置。此全景或立體影像串流可為按如電影方式所儲存、轉傳或發佈，或者是進行處理以自該全景或立體影像串流取得其一或多張靜態影像。 The invention relates to a method and a device for capturing and displaying a stereo or panoramic image stream. The panoramic or stereoscopic image stream may be stored, retransmitted, or distributed as a movie, or processed to obtain one or more still images from the panoramic or stereoscopic image stream.

在「單拍」式全景影像擷取的領域裡，眾知現有多款影像擷取裝置，例如CCD或CMOS類型的相機，而各個影像擷取裝置含有影像感測器，像是CCD或CMOS類型的影像感測器，這些感測器是耦接於光學裝置(透鏡)藉以將某場景的影像投射至該影像感測器上。影像擷取裝置光軸的指向會在不同的方向上，並且影像擷取的光學視域可針對於該影像的完整全景視域而重疊。國際專利申請案第WO 2012/032236號案文揭示一種光學裝置，此裝置極為精巧，並且含有三個經標註為「光學群組」的影像擷取裝置，同時可供以360°視域進行「單拍」式全景影像擷取。 In the field of "single-shot" panoramic image capture, there are many known image capture devices, such as CCD or CMOS type cameras, and each image capture device contains an image sensor, such as a CCD or CMOS type These image sensors are coupled to an optical device (lens) to project an image of a scene onto the image sensor. The optical axis of the image capture device will be pointed in different directions, and the optical field of view of the image capture may overlap with the full panoramic view of the image. The text of International Patent Application No. WO 2012/032236 discloses an optical device, which is extremely compact and contains three image capturing devices labeled as "optical group", and can be used for "single Capture "panoramic image capture.

在本文中，該詞彙「全景影像」應依其廣泛意義所解讀，不限於擷取單一360°視域的影像，而是更廣泛地適用於依照延伸視域的影像呈顯處理，此延伸視域相較於該全景影像擷取所使用之影像擷取裝置各者所覆蓋的光學視域更為寬廣。 In this article, the term "panoramic image" should be interpreted in its broad sense. It is not limited to capturing images of a single 360 ° field of view, but is more broadly applicable to image rendering processing according to an extended field of view. The field of view is wider than the optical field of view covered by each of the image capture devices used for the panoramic image capture.

利用這項全景影像擷取方法，該等影像擷取裝置各者可在有限視域內依像素矩陣的形式取得一場景的影像，然後將該等影像轉傳至外部的數位處理裝置，此裝置可供在其等之重疊區域的層級處進行數位方式的影像「縫合」，藉此產生最終的全景影像。 Using this panoramic image capture method, each of these image capture devices can obtain an image of a scene in the form of a pixel matrix within a limited field of view, and then transfer the images to an external digital processing device. This device Digitally “stitch” images at the level of their overlapping areas to produce the final panoramic image.

各個像素矩陣代表由一影像擷取裝置所擷取的影像，這是來自於一球形區域之3D表面被該影像擷取裝置所觀看到的二維投射。此二維投射是依據各個影像擷取裝置而定，並且尤其是按照影像擷取透鏡的光學特性，以及影像擷取裝置在影像擷取過程中的空間指向(即偏指(Yaw)、俯仰(Pitch)和橫滾(Roll))，而定。 Each pixel matrix represents an image captured by an image capture device, which is a two-dimensional projection from a spherical area where a 3D surface is viewed by the image capture device. This two-dimensional projection is based on each image capture device, and especially according to the optical characteristics of the image capture lens and the spatial orientation (i.e., Yaw, pitch ( Pitch) and Roll).

在先前技藝裡，要將影像經數位縫合以構成一全景影像，可例如在當併行排置由該等影像感測器所遞交的影像時進行，而且是藉由在其等之重疊區域的層級處進行影像數位縫合，藉此方式來得到最終全景影像。在此情況下，實作數位縫合並不對像素的二維投射進行修改，同時該最終全景影像的像素會保留該等所源生自之影像感測器的二維投射。 In the prior art, digital stitching of the images to form a panoramic image can be performed, for example, when the images submitted by the image sensors are arranged in parallel, and by the level of the overlapping areas thereof Digital stitching of images is performed in this way to obtain the final panoramic image. In this case, the implementation of digital stitching does not modify the two-dimensional projection of the pixels, and the pixels of the final panoramic image will retain the two-dimensional projections of the image sensors derived from them.

此數位縫合可為自動地執行，例如像是在國際專利申請案第WO 2011/037964號案文或美國專利申請案第2009/0058988號案文中所揭示者；或者可藉由手動輔助而以半自動的方式執行，即如在國際專利申請案第WO2010/01476號案文中所揭示者。 This digital stitching may be performed automatically, such as disclosed in International Patent Application No. WO 2011/037964 or US Patent Application No. 2009/0058988; or may be semi-automatic with manual assistance Implementation, that is, as disclosed in the text of International Patent Application No. WO2010 / 01476.

下文中亦提議一種用於全景影像呈顯的數位影像縫合解決方案，其標題為：「Image Alignment and Stitching：A Tutorial」，作者Richard Szeliski，於2005年元月26日。此文中是對所儲存影像靜態地，然非以動態方式，執行數位縫合，因此在該文章中所揭示的數位縫合解決方案確無法 供以呈顯動態的全景影像串流，更遑論在當正進行影像擷取時按照即時方式呈顯動態的全景影像串流。 The following also proposes a digital image stitching solution for rendering panoramic images. The title is: "Image Alignment and Stitching: A Tutorial" by Richard Szeliski, January 26, 2005. In this article, digital stitching is performed statically, but not dynamically, on the stored image, so the digital stitching solution disclosed in this article is indeed impossible. Provides a dynamic panoramic image stream, not to mention a dynamic panoramic image stream that is displayed in real-time during image capture.

而在立體影像擷取的領域裡，眾知該方法可包含擷取某場景的兩個平面影像，隨後是處理這兩張平面影像藉以產生一可供感知深度與廓形的立體3D影像。 In the field of stereo image capture, it is known that the method may include capturing two planar images of a scene, and then processing the two planar images to generate a stereo 3D image that can be perceived depth and profile.

前述的全景或立體影像擷取及呈顯方法會有如下缺點，即利用由具備分別或獨立光學裝置之感測器所取得的影像以呈顯一全景或立體影像會在最終數位影像裡出現均質性的問題(無論是全景或立體影像皆然)，特別是在相對色度、白色均衡、曝光時間與自動增益方面尤甚。 The foregoing methods for capturing and presenting panoramic or stereoscopic images have the following disadvantages: the use of images obtained by sensors with separate or independent optical devices to present a panoramic or stereoscopic image will appear homogeneous in the final digital image Sexual issues (both panoramic and stereoscopic), especially in terms of relative chroma, white balance, exposure time, and automatic gain.

此外，前述的影像數位縫合方法需要計算時間，這會不利於如電影般依即時方式進行全景影像的擷取及呈顯作業。 In addition, the foregoing digital digital stitching method requires calculation time, which is disadvantageous for capturing and displaying panoramic images in a real-time manner like a movie.

在美國專利申請案第2009/0058988號案文中，為改善處理時間並可供依即時地數位縫合來進行全景影像擷取，即提議一種基於低解析度影像對映的數位縫合解決方案。 In the text of US Patent Application No. 2009/0058988, in order to improve the processing time and provide panoramic image capture by real-time digital stitching, a digital stitching solution based on low-resolution image mapping is proposed.

廣義而言，本發明之目的為提議一種利用一或多台影像擷取裝置來進行全景或立體影像串流擷取及呈顯的新穎技術性解決方案。 In a broad sense, the purpose of the present invention is to propose a novel technical solution for capturing and rendering panoramic or stereoscopic image streams using one or more image capturing devices.

尤其，根據本發明的第一特定特點，此新穎解決方案可供加快數位處理速度，並因而有助於即時地擷取並呈顯全景或立體影像串流。 In particular, according to the first specific feature of the present invention, this novel solution can speed up the digital processing speed, and thus help capture and display a panoramic or stereoscopic image stream in real time.

更進一步，根據本發明的另一特定特點，此新穎解決方案可供修補前述肇因於藉分別或獨立光學裝置之感測器實作方式的不便性，並且特別是可供更簡易地獲得高品質的全景或立體影像。 Furthermore, according to another specific feature of the present invention, this novel solution can repair the inconvenience caused by the implementation of the sensor by using separate or independent optical devices, and in particular, it can make it easier to obtain high Quality panorama or stereo image.

在本發明中，全景或立體影像串流可為例如按照電影方式所儲存、轉傳或發佈，或者是可供稍後處理以自該串流靜態地取得一或多個全景或立體影像。 In the present invention, the panoramic or stereoscopic image stream may be stored, retransmitted, or distributed in a movie manner, for example, or may be processed later to statically obtain one or more panoramic or stereoscopic images from the stream.

根據本發明之第一特點，本發明的主要目的為一種用於擷取與呈顯一場景之全景或立體影像串流的方法，在此過程中是利用至少一影像擷取裝置(Ci)，按照像素格式對該場景的至少兩個不同影像進行多次接續擷取作業，該等影像可為重疊或不重疊，該等接續擷取作業是依一頻率(F)進行，而此頻率定義在兩次接續擷取作業之起始間的擷取時間(T)；以及，對於各次擷取作業，(a)對各個影像的像素進行數位處理，藉此利用該等像素以形成一最終全景或立體影像，而處理時間是短於或等於該擷取時間(T)，以及(b)在短於或等於該擷取時間(T)之時間間隔的過程裡產生一最終且先前形成的全景或立體影像；各個所擷取影像之各個像素的數位處理(a)至少包含保留或拋除該像素，並且當保留該像素時，利用對於該最終全景或立體影像上之各個位置的預定義加權因數(W)將其指配於該最終全景或立體影像裡的其一或多個位置。 According to a first feature of the present invention, a main object of the present invention is a method for capturing and rendering a panoramic or stereoscopic image stream of a scene, in which at least one image capturing device (C i ) is used According to the pixel format, multiple consecutive acquisitions of at least two different images of the scene can be performed. These images can be overlapping or non-overlapping. These successive acquisitions are performed according to a frequency (F), and this frequency is defined The acquisition time (T) between the start of two successive acquisition operations; and for each acquisition operation, (a) digitally process the pixels of each image to use these pixels to form a final Panoramic or stereo image, and the processing time is shorter than or equal to the acquisition time (T), and (b) a final and previously formed process is generated during a time interval shorter than or equal to the acquisition time (T) Panoramic or stereo image; digital processing of each pixel of each captured image (a) includes at least retaining or discarding the pixel, and when the pixel is retained, using a predefined definition of each position on the final panoramic or stereo image Weighting factor (W) Which is assigned to the final panoramic or perspective image in the one or more locations.

本發明之另一目的為一種用於擷取與呈顯全景或立體影像串流的裝置。該裝置含有一或更多影像擷取裝置(Ci)，此等可供擷取至少兩個不同影像作為一組像素，以及用於進行處理的電子裝置，此者可供呈顯全景或立體影像，而利用所擷取影像，該等電子處理裝置可供利用一或更多影像裝置以按照像素格式對一場景的至少兩個不同影像進行多次接續擷取作業，該等影像可為重疊或不重疊，該等接續擷取作業是依一頻率(F)進行，而此頻率定義在兩次接續擷取作業之起始間的擷取時間(T)；該等電子 處理裝置係經調適以，對於各次擷取作業，(a)對各個所擷取影像的像素進行數位處理，藉此利用該等像素以形成一最終全景或立體影像，而處理時間是短於或等於該擷取時間(T)，以及(b)在短於或等於該擷取時間(T)之時間間隔上產生一最終且先前形成的全景或立體影像；利用該等電子處理裝置對各個影像之各個像素的數位處理至少包含保留或拋除該像素，並且當保留該像素時，利用對於該最終全景或立體影像上之各個位置的預定義加權因數(W)將其指配於該最終全景或立體影像裡的其一或多個不同位置。 Another object of the present invention is a device for capturing and presenting a panoramic or stereo image stream. The device contains one or more image capturing devices (C i ), which can capture at least two different images as a group of pixels, and an electronic device for processing, which can be used to display a panoramic or stereoscopic The electronic processing device can use one or more image devices to perform multiple consecutive capture operations on at least two different images of a scene in pixel format using the captured images. The images can be overlapped. Or non-overlapping, these successive acquisition operations are performed according to a frequency (F), and this frequency defines the acquisition time (T) between the start of two successive acquisition operations; these electronic processing devices are adapted Therefore, for each capture operation, (a) digitally process the pixels of each captured image, thereby using these pixels to form a final panoramic or stereo image, and the processing time is shorter than or equal to the capture Time (T), and (b) producing a final and previously formed panoramic or stereo image at a time interval shorter than or equal to the acquisition time (T); the use of such electronic processing devices for each pixel of each image Digitally process at least packets The pixel is retained or discarded, and when the pixel is retained, it is assigned to one or more of the final panoramic or stereo image using a predefined weighting factor (W) for each position on the final panoramic or stereo image Multiple different locations.

根據本發明的第二特點，本發明之目的亦在於一種用於擷取與呈顯一場景之全景或立體影像串流的方法，其特徵在於利用至少一影像擷取裝置(Ci)，按照像素格式對該場景的至少兩個不同影像進行多次接續擷取作業，該等影像可為重疊或不重疊，並且在於，在該等影像擷取作業的過程中，該等所擷取影像的像素係經數位處理以形成全景或立體影像，同時產生一全景或立體影像串流，並且在於各個所擷取影像之各個像素的數位處理至少包含保留或拋除該像素，並且當保留該像素時，利用對於該最終全景或立體影像上之各個位置的預定義加權因數(W)將其指配於該最終全景或立體影像上的其一或多個位置。 According to a second feature of the present invention, an object of the present invention is also a method for capturing and rendering a panoramic or stereoscopic image stream of a scene, which is characterized by using at least one image capturing device (C i ) according to The pixel format performs multiple consecutive capture operations on at least two different images of the scene. The images may be overlapping or non-overlapping, and in the process of the image capturing operations, the captured images are The pixels are digitally processed to form a panoramic or stereoscopic image, while generating a panoramic or stereoscopic image stream, and the digital processing of each pixel in each captured image includes at least retaining or discarding the pixel, and when the pixel is retained , Using a predefined weighting factor (W) for each position on the final panoramic or stereo image to assign it to one or more positions on the final panoramic or stereo image.

根據本發明的該項第二特點，本發明之目的亦在於一種用於擷取與呈顯全景或立體影像串流的裝置，其特徵在於該裝置包含一或更多影像擷取裝置(Ci)，此等可供按照像素集組格式擷取至少兩個不同影像，以及電子處理裝置，此等可供該等一或更多影像擷取裝置按照像素格式對一場景的至少兩個不同影像進行多次接續擷取作業，該等影像可為重疊或不重疊，而該等係經調適以在影像擷取作業的過程中對所擷取影像的像素進 行數位處理藉以形成全景或立體影像，同時產生一全景或立體影像串流，並且在於對各個所擷取影像之各個像素的數位處理至少包含保留或拋除該像素，而且當保留該像素時，藉由對於該最終全景或立體影像上之各個位置的預定義加權因數(W)將其指配於該最終全景或立體影像上的其一或多個位置。 According to the second feature of the present invention, the object of the present invention is also a device for capturing and displaying a panoramic or stereo image stream, which is characterized in that the device includes one or more image capturing devices (C i ), Which can be used to capture at least two different images according to the pixel set group format, and an electronic processing device, which can be used by the one or more image capture devices to at least two different images of a scene according to the pixel format Perform multiple successive capture operations, these images can be overlapping or non-overlapping, and these are adapted to digitally process the pixels of the captured image during the image capture operation to form a panoramic or stereo image, A panoramic or stereo image stream is generated at the same time, and the digital processing of each pixel of each captured image includes at least retaining or discarding the pixel, and when the pixel is retained, the final panoramic or stereo image is A predefined weighting factor (W) for each location assigns it to one or more locations on the final panoramic or stereo image.

根據本發明的第三特點，本發明之目的亦在於一種用於擷取與呈顯一場景之全景或立體影像串流的方法，其特徵在於，在此過程中，利用至少一影像擷取裝置，按照像素格式對該場景的至少兩個不同影像進行多次接續擷取作業，該等影像可為重疊或不重疊，而各個影像擷取裝置可供對按照像素集組格式的影像進行擷取，同時對於各個所擷取影像的輸出遞交至少根據一第一時脈信號(H_Sensor)所同步化的像素串流。各個所擷取影像的各個像素係經數位處理，藉以利用該等像素來產生一最終全景或立體影像而作為至少根據一第二時脈信號(H)所同步化的像素串流。 According to a third feature of the present invention, an object of the present invention is also a method for capturing and rendering a panoramic or stereoscopic image stream of a scene, characterized in that, in this process, at least one image capturing device is used According to the pixel format, multiple consecutive capture operations of at least two different images of the scene can be performed. These images can be overlapping or non-overlapping, and each image capturing device can capture images according to the pixel set group format. At the same time, for the output of each captured image, a pixel stream synchronized at least according to a first clock signal (H_Sensor) is submitted. Each pixel of each captured image is digitally processed to use the pixels to generate a final panoramic or stereoscopic image as a pixel stream synchronized at least according to a second clock signal (H).

根據本發明的該項第三特點，本發明之目的亦在於一種用於擷取與呈顯全景或立體影像串流的裝置，該裝置包含一或更多影像擷取裝置，該等可供按照像素格式對一場景的至少兩個不同影像進行多次接續擷取作業，該等影像可為重疊或不重疊，並且該等電子處理裝置可供利用所擷取影像以呈顯一全景或立體影像串流。各個影像擷取裝置係經調適以對於該等所擷取影像之各個像素的輸出遞交至少根據一第一時脈信號(H_Sensor)所同步化的像素串流。該等電子處理裝置係經設計以數位處理該等所擷取影像的各個像素，藉以利用該等像素來產生一最終全景或立體影像而作為至少根據一第二時脈信號(H)所同步化的像素串流。 According to the third feature of the present invention, an object of the present invention is also a device for capturing and displaying a panoramic or stereo image stream. The device includes one or more image capturing devices. The pixel format performs multiple consecutive capture operations on at least two different images of a scene. The images can be overlapping or non-overlapping, and the electronic processing devices can use the captured images to display a panoramic or stereoscopic image. Streaming. Each image capture device is adapted to submit a pixel stream synchronized at least according to a first clock signal (H_Sensor) to the output of each pixel of the captured images. The electronic processing devices are designed to digitally process each pixel of the captured images, thereby utilizing the pixels to generate a final panoramic or stereoscopic image as synchronized at least according to a second clock signal (H) Pixel stream.

根據本發明的第四特點，本發明之目的亦在於擷取與呈顯一場景之至少一全景或立體影像，而在此過程中，利用至少一影像擷取裝置(Ci)以擷取該場景的至少兩個不同影像，該等影像可為重疊或不重疊，各個影像擷取裝置可供對按照像素集組格式的影像進行擷取，並且對於各個所擷取影像的輸出遞交一像素串流；各個所擷取影像的像素串流係經數位處理，藉以利用該等像素呈顯至少一最終全景或立體影像，同時對應於各個所擷取影像的像素串流之各個像素的數位處理至少包含保留或拋除該像素，而且當保留該像素時，利用對於該最終全景或立體影像上之各個位置的預定義加權因數(W)將其指配於該最終全景或立體影像上的其一或多個位置。 According to a fourth feature of the present invention, the object of the present invention is also to capture and display at least one panoramic or stereoscopic image of a scene. In this process, at least one image capturing device (C i ) is used to capture the scene. At least two different images of the scene. The images can be overlapping or non-overlapping. Each image capturing device can capture images in the pixel set format, and submit a pixel string for the output of each captured image. Stream; the pixel stream of each captured image is digitally processed so that the pixels are used to render at least one final panoramic or stereo image, while the digital processing of each pixel corresponding to the pixel stream of each captured image is at least Includes retaining or discarding the pixel, and when retaining the pixel, assigning it to one of the final panoramic or stereo images using a predefined weighting factor (W) for each position on the final panoramic or stereo image Or multiple locations.

根據本發明的該項第四特點，本發明之目的亦在於一種用於擷取與呈顯至少一全景或立體影像的裝置，該裝置含有一或更多影像擷取裝置(Ci)，該等可供擷取至少兩個不同影像，該等影像可為重疊或不重疊，而各個影像擷取裝置(Ci)係經調適以對於各個所擷取影像遞交一像素串流，以及電子處理裝置，該等可供在影像擷取作業的過程中利用各個所擷取影像的像素串流呈顯一全景或立體影像。該等電子處理裝置係經設計以處理該所擷取影像像素串流的各個像素而包含保留或拋除該像素，並且當保留該像素時，將其依一對於該最終全景或立體影像上之各個位置的預定義加權因數(W)指配於該最終全景或立體影像上的其一或多個不同位置。 According to the fourth feature of the present invention, the object of the present invention is also a device for capturing and presenting at least one panoramic or stereoscopic image. The device includes one or more image capturing devices (C i ). It can be used to capture at least two different images. These images can be overlapping or non-overlapping. Each image capturing device (C i ) is adapted to submit a pixel stream for each captured image, and electronic processing. Device, which can be used to display a panoramic or stereoscopic image using the pixel stream of each captured image during the image capture operation. The electronic processing devices are designed to process each pixel of the captured image pixel stream, including retaining or discarding the pixel, and when the pixel is retained, it is applied to the final panoramic or stereo image one by one. A predefined weighting factor (W) for each position is assigned to one or more different positions on the final panoramic or stereo image.

1‧‧‧本發明之全景影像擷取與呈顯裝置 1‧‧‧ The panoramic image capture and display device of the present invention

10‧‧‧電子處理裝置 10‧‧‧ electronic processing device

11‧‧‧外部電子處理裝置 11‧‧‧External electronic processing device

Ci‧‧‧影像擷取裝置 C i ‧‧‧ image capture device

F‧‧‧擷取頻率 F‧‧‧ Acquisition frequency

H‧‧‧第二時脈信號 H‧‧‧Second Clock Signal

H_Sensor‧‧‧第一時脈信號 H_Sensor‧‧‧First clock signal

H10‧‧‧基本時脈 H10‧‧‧ basic clock

T‧‧‧擷取時間 T‧‧‧Retrieval time

t‧‧‧時間間隔 t‧‧‧time interval

W‧‧‧加權因數 W‧‧‧ weighting factor

經閱覽後文中本發明之一或更多較佳具體實施例的詳細說明，並參照於隨附圖式，將隨能知曉本發明的特徵和優點，此等說明係經 提供作為本發明的非限制或窮舉性範例，而其中：圖1為一根據本發明之裝置的電子架構範例略圖。 After reading the detailed description of one or more preferred embodiments of the present invention, and referring to the accompanying drawings, the characteristics and advantages of the present invention will be known as soon as possible. It is provided as a non-limiting or exhaustive example of the present invention, wherein: FIG. 1 is a schematic diagram of an example of an electronic architecture of a device according to the present invention.

圖2為圖1裝置之主要電子信號的計時器範例。 FIG. 2 is an example of a timer of the main electronic signals of the device of FIG. 1.

圖3表示一「魚眼」透鏡的擷取區域之光學/像素視域間的對應性範例。 FIG. 3 shows an example of the correspondence between the optical / pixel field of view of the capture area of a “fisheye” lens.

圖4為在一最終全景影像之局部中利用影像感測器所擷取像素矩陣的重對映範例。 FIG. 4 is a re-mapping example of a pixel matrix captured by an image sensor in a part of a final panoramic image.

圖5說明在該最終全景影像之像素Pi,j與利用影像感測器所擷取之像素矩陣間的幾何對應性範例。 FIG. 5 illustrates an example of the geometric correspondence between the pixels P i, j of the final panoramic image and the pixel matrix captured by the image sensor.

圖6A至6I表示對於RAW類型影像之特定情況的不同重對映圖式。 Figures 6A to 6I show different anti-mapping patterns for specific cases of RAW type images.

圖7A至7D說明一感測器線段至一全景影像上之重對映的不同範例。 7A to 7D illustrate different examples of re-mapping on a sensor line segment to a panoramic image.

圖8說明對於三張影像為形成一最終全景影像之重對映的特定範例。 FIG. 8 illustrates a specific example of re-mapping for three images to form a final panoramic image.

圖1表示一種可供擷取與呈顯全景影像之本發明裝置1的特定範例。 FIG. 1 shows a specific example of a device 1 according to the present invention for capturing and displaying panoramic images.

在本特定範例裡，該裝置1含有三個例如CCD或CMOS類型的影像擷取裝置C1、C2、C3，而各者可供按像素矩陣格式擷取影像，以及電子處理裝置10，此者可供利用該等影像感測器C1、C2、C3遞交的像素以呈顯一全景影像。一般說來，該等影像擷取裝置C1、C2、C3各者含有像是CCD或CMOS類型的影像感測器，該感測器係經耦接於光學裝置(透鏡)，其中含有一或更多透鏡，而此等係經校準於該影像感測器並可供將光線聚焦於該影像感測器上。 In this particular example, the device 1 contains three image capture devices C 1 , C 2 , C 3 , such as a CCD or CMOS type, each of which can capture images in a pixel matrix format, and an electronic processing device 10, This allows the pixels submitted by the image sensors C 1 , C 2 , C 3 to be used to present a panoramic image. Generally speaking, each of these image capturing devices C 1 , C 2 , and C 3 contains an image sensor such as a CCD or CMOS type. The sensor is coupled to an optical device (lens) and contains One or more lenses, and these are calibrated to the image sensor and can be used to focus light onto the image sensor.

該等影像擷取裝置C1、C2、C3的光軸是按不同方向所指向，同時其等的光學視域會涵蓋整個最終全景影像場域，且該等光學視域最好 是出現重疊。 The optical axes of the image capturing devices C 1 , C 2 , and C 3 are pointed in different directions, and their optical fields of view will cover the entire final panoramic image field, and these optical fields of view should preferably appear overlapping.

在本文中，該詞彙「全景影像」應依其廣泛意義所解讀，不限於根據360°視域所呈顯的全景影像，而是更一般性地依照延伸視域所呈顯的影像，此延伸視域相較於該全景影像擷取所使用之影像擷取裝置各者所涵蓋的光學視域更為寬廣。 In this article, the term "panoramic image" should be interpreted in its broadest sense, not limited to the panoramic image presented according to a 360 ° field of view, but more generally according to the image presented in an extended field of view. This extension The field of view is wider than the optical field of view covered by each of the image capture devices used for the panoramic image capture.

僅為示範之目的，該等影像擷取裝置C1、C2、C3可例如含有三個精巧光學裝置的光學群組，這可如國際專利申請案第WO 2012/03223號案文中所揭示者並且可供單拍擷取全景影像。 For illustration purposes only, the image capture devices C 1 , C 2 , C 3 may, for example, contain an optical group of three delicate optical devices, as disclosed in the text of International Patent Application No. WO 2012/03223 It also allows you to capture panoramic images in a single shot.

最好，然非必要，本發明裝置1含有可攜式設備，藉以達到簡易搬送並且運用於各種位置之目的。 Preferably, but not necessarily, the device 1 of the present invention contains a portable device, so as to achieve easy transportation and use in various locations.

現參照圖2，該等數位處理裝置10遞交一基本時脈H10，這是利用例如石英所產生，並且可用以對該等影像擷取裝置C1、C2、C3各者之影像感測器的作業進行計時。 Referring now to FIG. 2, the digital processing devices 10 submit a basic clock H10, which is generated using, for example, quartz, and can be used for image sensing of each of the image capturing devices C 1 , C 2 , and C 3 . Device's operation is timed.

在輸出方面，該等影像擷取裝置C1、C2、C3各者之影像感測器會在「Pixels」資料匯流排上對於各個所擷取影像遞交一像素串流，此串流是依照一第一時脈信號(H_sensor)所同步化，而該時脈信號則是由該等影像擷取感測器各者利用該基本時脈H10，以及兩個信號「Line Valid」和「Frame Valid」，所產生。由該等影像擷取感測器各者所產生的時脈信號(H_sensor)會更特別地擁有相同的頻率。 In terms of output, the image sensors of each of the image capture devices C 1 , C 2 , and C 3 will submit a pixel stream for each captured image on the “Pixels” data bus. This stream is Synchronized according to a first clock signal (H_sensor), and the clock signal is used by each of the image capture sensors to use the basic clock H10, and two signals "Line Valid" and "Frame Valid ". The clock signal (H_sensor) generated by each of these image capture sensors will more particularly have the same frequency.

該等電子處理裝置10可供利用由該等影像擷取裝置C1、C2、C3之影像感測器所遞交的像素，並且以一可相較於該等影像擷取裝置C1、C2、C3在該「Pixels」資料匯流排上遞交表示該最終全景影像之像素串流者的方式，來呈顯一全景影像。 The electronic processing devices 10 can use the pixels submitted by the image sensors of the image capturing devices C 1 , C 2 , C 3 , and can be compared with the image capturing devices C 1 , C 2 and C 3 submit pixel streamers representing the final panoramic image on the "Pixels" data bus to present a panoramic image.

這些電子處理裝置10之「Pixels」資料匯流排的大小與該等影像擷取裝置C1、C2、C3之「Pixels」資料匯流排的大小可為相同或者互異， 而且最好是較大。例如，然非限制本發明範疇，該等影像擷取裝置C1、C2、C3的「Pixels」資料匯流排可為八位元，而該等電子處理裝置10的「Pixels」資料匯流排則是16位元。 The electronic processing device 10 of the 'Pixels' and the data bus size of such image capturing device C 1, C 2, C 3 of the 'Pixels' data bus size may be the same or different from each, and more preferably Big. For example, without limiting the scope of the present invention, the “Pixels” data bus of the image capture devices C 1 , C 2 , C 3 may be eight bits, and the “Pixels” data bus of the electronic processing device 10 It is 16 bits.

由該等電子處理裝置10所產生的像素串流是同步於一第二時脈信號(H)，此信號是由該等電子處理裝置10利用該基本時脈信號以及兩個由該等電子處理裝置10所產生的「Line Valid」和「Frame Valid」信號所產生。 The pixel stream generated by the electronic processing devices 10 is synchronized with a second clock signal (H). The signal is processed by the electronic processing devices 10 using the basic clock signal and two by the electronic processing devices. The "Line Valid" and "Frame Valid" signals generated by the device 10 are generated.

圖2說明前述之本發明信號同步作業的一特定且非限制性範例。然此圖式中並未顯示在該「Pixels」資料匯流排所傳送的資料。 FIG. 2 illustrates a specific and non-limiting example of the aforementioned signal synchronization operation of the present invention. However, the data sent by this "Pixels" data bus is not shown in this illustration.

現參照圖2，該等接續擷取作業為週期性並且是依一頻率F計時，而此頻率定義等於在兩次接續擷取作業起始間之時間間隔(t)長度的擷取時間T(T=1/F)。 Referring now to FIG. 2, the successive acquisition operations are periodic and timed at a frequency F, and this frequency is defined to be equal to the acquisition time T (length) of the time interval (t) between the start of two successive acquisition operations. T = 1 / F).

更特定說，在圖2上，該等影像擷取裝置C1、C2、C3各者之信號「Frame Valid」的揚升邊緣是同步於該等影像擷取裝置C1、C2、C3所擷取影像之像素在該等影像擷取裝置C1、C2、C3各者的「Pixels」資料匯流排上之傳輸作業的起始處。而該等影像擷取裝置C1、C2、C3各者之信號「Frame Valid」的落降邊緣則是表示在該「Pixels」資料匯流排上該等影像擷取裝置C1、C2、C3所擷取影像之像素傳輸作業的結束處。這些由該等影像擷取裝置C1、C2、C3所遞交之「Frame Valid」信號的揚升邊緣(分別地落降)會在時間線上略微地移位。 More specifically, on FIG. 2, the rising edge of the signal “Frame Valid” of each of the image capturing devices C 1 , C 2 , and C 3 is synchronized with the image capturing devices C 1 , C 2 , C 3-pixel image captured at the start of the capture device C 1, C 2, transmission operations on the "pixels" data bus C 3 each person in those images. The drop off the edge of one of the image capturing device C 1, C 2, C 3 signals of each person's "Frame Valid" is the representation of such image-capturing device C on the "Pixels" data bus 1, C 2 , C 3 a pixel at the end of transmission of image capture operations. The image capturing device by those C 1, C 2, rose edge (falling down respectively) "Frame Valid" signal is submitted by the C 3 will be slightly displaced in the timeline.

該等影像擷取裝置C1、C2、C3的「Line Valid」信號為同步於該「Frame Valid」信號的各個揚升邊緣，並且表示一串影像像素之傳輸作業的起始處。而該「Frame Valid」信號的落降邊緣則表示一串影像像素之傳輸作業的結束處。該等三個影像擷取裝置C1、C2、C3裡各個「Pixels」資料匯流排上之各個所傳影像的像素是分別地利用該等影像擷取裝置C1、C2、C3 各者所遞交之各個時脈信號「H_sensor」由該等電子處理裝置10依平行方式所取樣。 Such image capturing device C 1, C "Line Valid" signal 2, C 3 is in synchronization with the respective Ascension "Frame Valid" signal edge, and indicates the start of the transmission operation of the image pixel string. The falling edge of the "Frame Valid" signal indicates the end of the transmission operation of a series of image pixels. These three image capturing device C 1, C 2, each pixel on the image transfer C 3 in each of the "Pixels" data bus, respectively, use such image capturing device C 1, C 2, C 3 Each of the clock signals “H_sensor” submitted by each of them is sampled by the electronic processing devices 10 in a parallel manner.

現參照圖2，由該等電子處理裝置10所遞交之「Frame Valid」信號的揚升邊緣為同步於在電子處理裝置之「Pixels」資料匯流排上運用該等影像擷取裝置C1、C2、C3所遞交像素而呈顯的最終全景影像之傳輸作業的起始。此揚升邊緣是由該等電子處理裝置10利用該等影像擷取裝置C1、C2、C3所遞交之「Frame Valid」信號的揚升邊緣自動地產生，並且特別是在當偵測到最近發生的揚升邊緣時所產生，這也就是在圖2所示特定範例裡由該影像擷取裝置Ci所遞交之「Frame Valid」信號的揚升邊緣。 Referring now to FIG. 2, the rising edge of the “Frame Valid” signal submitted by the electronic processing devices 10 is synchronized with the use of the image capturing devices C 1 , C on the “Pixels” data bus of the electronic processing device. 2, C 3 pixels was submitted by the initial transmission of significant job of the final panoramic image. This edge is Ascension by those electronic processing device 10 using one of the image capturing device C 1, rose edge "Frame Valid" signal C 2, C 3 submitted by the automatically generated, and in particular when it is detected in the It is generated when the rising edge occurs recently, which is also the rising edge of the “Frame Valid” signal submitted by the image capturing device C i in the specific example shown in FIG. 2.

而由該等電子處理裝置10所遞交之「Frame Valid」信號的落降邊緣則是同步於在電子處理裝置10之「Pixels」資料匯流排上運用該等影像擷取裝置C1、C2、C3所遞交像素而呈顯的最終全景影像之傳輸作業的結束。 The falling edge of the "Frame Valid" signal submitted by the electronic processing devices 10 is synchronized with the use of the image capturing devices C 1 , C 2 , The end of the transmission operation of the final panoramic image presented by C 3 submitted pixels.

由該等電子處理裝置10所遞交的「Line Valid」信號為同步於由該等電子處理裝置10所遞交之「Frame Valid」信號的各個揚升邊緣，並且表示一串全景影像像素之傳輸作業的起始。而由該等電子處理裝置10所遞交之「Line Valid」信號的各個落降邊緣則是表示一串全景影像像素之傳輸作業的結束。 The "Line Valid" signal submitted by the electronic processing devices 10 is synchronized with each rising edge of the "Frame Valid" signal submitted by the electronic processing devices 10, and represents the transmission of a series of panoramic image pixels. Start. The falling edges of the “Line Valid” signals submitted by the electronic processing devices 10 indicate the end of the transmission of a series of panoramic image pixels.

在該等電子處理裝置10的「Pixels」資料匯流排上之各個全景影像的像素寫入處理是按照時脈信號「H」所同步，此信號是由該等電子處理裝置10所產生，並且可由另一外部電子裝置(例如該裝置11)運用以在該資料匯流排上讀取像素。 The pixel writing processing of each panoramic image on the "Pixels" data bus of the electronic processing devices 10 is synchronized according to a clock signal "H", which is generated by the electronic processing devices 10 and can be Another external electronic device (such as the device 11) is used to read pixels on the data bus.

根據本發明的替代性具體實施例，這個由該等電子處理裝置10所遞交的時脈信號「H」可為同步或非同步於由該等影像擷取裝置C1、C2、C3所遞交的「H_sensor」時脈信號。該「H」時脈信號的頻率可為等於 或不同於由該等影像擷取裝置C1、C2、C3所遞交的「H_sensor」時脈信號。最好，該「H」時脈信號的頻率大於由該等影像擷取裝置C1、C2、C3所遞交的「H_sensor」信號，即如圖2所示者。 According to an alternative embodiment of the present invention, the clock signal “H” submitted by the electronic processing devices 10 may be synchronized or asynchronous to the clock signals “H” provided by the image capture devices C 1 , C 2 , and C 3 . Submitted "H_sensor" clock signal. When the "H" clock signal frequency may be equal to or different 1, C 2, C 3 submitted by "H_sensor" by the clock signal C such image capturing apparatus. Preferably, when the "H" clock signal frequency is greater than 1, C 2, C 3 submitted by "H_sensor" signal, i.e. those shown in FIG. 2 by those image capturing apparatus C.

在圖2所示的特定情況下，對於各次擷取作業將會利用該等影像擷取裝置C1、C2、C3平行地進行三次影像擷取，同時在此特定情況下，該時間間隔(t)即為分隔該影像擷取裝置Ci，亦即首先在其「Pixels」資料匯流排上傳送像素的影像擷取裝置，之「Frame Valid」信號中兩次接續揚升邊緣的時間間隔。 In the specific case shown in FIG. 2, for each capture operation, the image capture devices C 1 , C 2 , and C 3 are used to perform three image captures in parallel, and in this specific case, the time The interval (t) is the time that separates the image capture device C i , that is, the image capture device that first sends pixels on its "Pixels" data bus, and the "Frame Valid" signal continues to rise twice. interval.

在分隔兩次接續影像擷取作業起始之時間間隔(t)的過程裡，該等電子處理裝置10會進行：(a)對各個所擷取影像的像素進行數位處理，藉以利用該等像素來呈顯最終全景影像；對於圖1所示架構與圖2所示信號，這些就是在該等影像擷取裝置C1、C2、C3之「Pixels」資料匯流排上傳送至該等電子處理裝置10的像素；並且(b)產生最終全景影像；對於圖1所示架構與圖2所示信號，這些就是該等電子處理裝置10藉由該等電子處理裝置10在該時間間隔(t)過程中遞交之「Frame Valid」信號的揚升與落降邊緣於其等之「Pixels」資料匯流排上所遞交而作為輸出的像素。 In the process of separating the time interval (t) from the start of two consecutive image capture operations, the electronic processing devices 10 will: (a) digitally process the pixels of each captured image to utilize the pixels was significant to the final panoramic image; architecture and the signal shown in Figure 2. Figure 1, which is in such image capturing device C 1, C 2, "Pixels" C 3 data uploaded to the bus of the electrons Pixels of the processing device 10; and (b) generate a final panoramic image; for the architecture shown in FIG. 1 and the signals shown in FIG. 2, these are the electronic processing devices 10 by the electronic processing devices 10 at the time interval (t The rising and falling edges of the "Frame Valid" signal submitted in the process of) are the pixels that are submitted on the "Pixels" data bus as output pixels.

故而能夠藉由該等電子處理裝置按照與接續影像擷取作業相同的速率以即時方式產生接續的全景影像串流。例如，若該等影像擷取裝置C1、C2、C3係經設計以每秒遞交25個影像，則於兩次接續影像擷取作業間之各個時間間隔(t)的擷取時間T是等於40ms，此值對應於25Hz的擷取頻率F，並且該等電子處理裝置也是每秒產生25個全景影像(亦即每40ms有一個全景影像)。 Therefore, these electronic processing devices can generate a continuous panoramic image stream in real time at the same rate as the continuous image capture operation. For example, if the image capture devices C 1 , C 2 , and C 3 are designed to deliver 25 images per second, the capture time T at each time interval (t) between two consecutive image capture operations It is equal to 40ms, and this value corresponds to an acquisition frequency F of 25Hz, and these electronic processing devices also generate 25 panoramic images per second (that is, one panoramic image every 40ms).

該擷取時間T(即兩次接續影像擷取作業間之各個時間間隔 (t)的長度)將是根據該等影像擷取裝置C1、C2、C3的技術而定。實作上，該擷取時間T將最好是短於1秒，並且甚至更佳地為短於或者等於100ms。 The capturing time T (ie, the length of each time interval (t) between two consecutive image capturing operations) will be determined according to the technologies of the image capturing devices C 1 , C 2 , and C 3 . In practice, the acquisition time T will preferably be shorter than 1 second, and even more preferably shorter than or equal to 100 ms.

最好，在各個分隔兩次接續影像擷取作業起始之時間間隔(t)過程中所產生的最終全景影像是源生自該等像素在此相同時間間隔(t)過程中的數位處理(a)。在此情況下，各個接續全景影像可為即時性地產生，並且是與為呈顯該特定全景影像的影像擷取作業幾乎同時地進行，而在將用於呈顯後續全景影像的後續影像擷取作業之前。 Preferably, the final panoramic image generated during the time interval (t) at which the two consecutive image capture operations are started is separated from the digital processing of these pixels during this same time interval (t) ( a). In this case, each successive panoramic image can be generated in real time, and is performed almost simultaneously with the image capture operation to display the specific panoramic image, and the subsequent image capture will be used to display the subsequent panoramic image. Before taking homework.

在另一替代性具體實施例裡，在各個分隔兩次接續影像擷取作業起始之時間間隔(t)過程中所產生的最終影像是源生自這些像素在先前時間間隔(t)過程裡，例如前行的時間間隔(t)，所進行的的數位處理(a)。在此情況下，各個接續全景影像可為即時性地產生，並且是相對於為呈顯該全景影像之時間擷取作業而具略微的時間移位。 In another alternative embodiment, the final image generated during the time interval (t) at which each successive two image capture operations are started is derived from these pixels during the previous time interval (t) , Such as the previous time interval (t), the digital processing (a). In this case, each successive panoramic image can be generated in real time, and it has a slight time shift relative to the time capture operation for rendering the panoramic image.

在另一替代性具體實施例裡，可在一給定擷取週期(N)的過程中開始(即由該等電子處理裝置10所遞交之「Frame Valid」信號的揚升邊緣)產生各個全景影像，並且可在後續的擷取週期(N+1)過程中結束(即由該等電子處理裝置10所遞交之「Frame Valid」信號的落降邊緣)。最好，然非必要，由該等電子處理裝置10所遞交的「Frame Valid」信號之揚升邊緣與落降邊緣間的時間間隔是短於或等於該擷取時間T。 In another alternative embodiment, each panorama may be started during a given acquisition cycle (N) (that is, the rising edge of the "Frame Valid" signal submitted by the electronic processing device 10) to generate each panorama The image can be ended during the subsequent acquisition cycle (N + 1) (that is, the falling edge of the "Frame Valid" signal submitted by the electronic processing device 10). Preferably, but not necessarily, the time interval between the rising edge and the falling edge of the "Frame Valid" signal submitted by the electronic processing devices 10 is shorter than or equal to the acquisition time T.

針對各個影像擷取作業所進行的像素處理(a)可相對於影像擷取週期而在時間線上移位。最好，然非必要，在影像擷取作業過程中來自所有擷取影像而待用以呈顯最終全景影像之像素的處理時間會短於或等於該擷取時間T。例如，為形成一最終全景影像，利用在第N個擷取週期過程中所擷取影像之像素的處理(a)可由該等電子處理裝置10在後續的影像擷取週期過程中，例如在第N+1個擷取週期的過程裡，進行。 The pixel processing (a) performed for each image capture operation can be shifted on the time line with respect to the image capture cycle. Preferably, but not necessarily, the processing time of pixels from all captured images to be used to render the final panoramic image during the image capture operation will be shorter than or equal to the capture time T. For example, in order to form a final panoramic image, the processing using pixels of the captured image during the Nth capture cycle (a) may be performed by the electronic processing devices 10 during subsequent image capture cycles, such as During N + 1 acquisition cycles.

該等電子處理裝置10可含有電子、數位程式設計資料處理 單元，這可根據本發明無分地運用任何已知的電路裝置所實作，例如像是一或多個FPGA類型的可程式設計電路，及/或一或多個ASIC類型的特定電路，或者是可程式設計處理單元，而其等的電子架構可供具體實作一微控制器或微處理器。 These electronic processing devices 10 may include electronic, digital programming data processing Unit, which can be implemented in accordance with the present invention without using any known circuit device, such as, for example, one or more FPGA-type programmable circuits and / or one or more ASIC-type specific circuits, or It is a programmable processing unit, and other electronic architectures can be implemented as a microcontroller or microprocessor.

在如圖1所示的本發明特定變化項目中，按如像素集組而由該等電子處理裝置10所遞交的接續全景影像串流會由額外的電子處理裝置11進行處理，這些裝置可包含例如DPS類型電路，並且可供例如儲存在記憶體內，並且/或是將該全景影像串流以電影方式即時地顯示在螢幕上。 In a specific variation of the present invention as shown in FIG. 1, the continuous panoramic image stream submitted by the electronic processing devices 10 according to the pixel set group will be processed by an additional electronic processing device 11. These devices may include For example, it is a DPS type circuit and can be stored in a memory, for example, and / or the panoramic image stream can be displayed on the screen in a movie mode in real time.

在本發明的另一變化項目裡，可將該等額外電子處理裝置11設計為處理由該等電子處理裝置10所遞交的接續全景影像串流以作為自該串流取得一或多個全景影像的擷取裝置。 In another variation of the present invention, the additional electronic processing devices 11 may be designed to process successive panoramic image streams submitted by the electronic processing devices 10 to obtain one or more panoramic images from the stream. Capture device.

一般說來，在特定的替代性具體實施例裡，各個影像擷取裝置C1、C2、C3含有「魚眼」透鏡類型的光學裝置，此者連接至一擷取矩陣，並且各個所擷取影像的特徵在於三組空間指向資訊，這些資訊通常稱為偏指(Yaw)、俯仰(Pitch)和橫滾(Roll)，而且在影像擷取的過程中是特定於該影像擷取裝置的空間指向。 Generally speaking, in a specific alternative embodiment, each of the image capture devices C 1 , C 2 , and C 3 contains a “fish-eye” lens-type optical device, which is connected to an capture matrix, and each The captured image is characterized by three sets of spatial pointing information, which are commonly referred to as Yaw, Pitch, and Roll, and are specific to the image capture device during image capture Space pointing.

現參照圖3，一「魚眼」透鏡顯現一有效球形中心偵測表面(即圖3上的灰色表面及白色表面)，而且由影像感測器所擷取之影像的有效像素已知為獲自於該影像擷取裝置偵測表面之僅一部份的二維投射(即圖3中的864像素乘900像素)。 Referring now to FIG. 3, a “fisheye” lens shows an effective spherical center detection surface (ie, the gray surface and the white surface on FIG. 3), and the effective pixels of the image captured by the image sensor are known as Only a part of the two-dimensional projection from the detection surface of the image capture device (ie, 864 pixels by 900 pixels in FIG. 3).

因此，通常表示由一影像擷取裝置C1、C2或C3所擷取影像的各個像素矩陣是源生自一球形3D表面部份裡被該影像擷取裝置C1、C2或C3所觀看到的二維投射。此二維投射是依據各個影像擷取裝置C1、C2或C3而定，並且尤其是按照該影像擷取裝置C1、C2或C3的光學裝置，以及該影像擷取裝置C1、C2或C3在影像擷取過程中的空間指向(偏指(Yaw)、俯仰 (Pitch)和橫滾(Roll))，而定。 Therefore, it generally means that each pixel matrix of an image captured by an image capture device C 1 , C 2 or C 3 is derived from a spherical 3D surface portion and is captured by the image capture device C 1 , C 2 or C 3 2D projection as viewed. This two-dimensional projection is based on the respective image capturing devices C 1 , C 2 or C 3 , and in particular according to the optical devices of the image capturing device C 1 , C 2 or C 3 and the image capturing device C 1 , C 2 or C 3 spatial orientation (Yaw, Pitch and Roll) during image capture, depending on.

為提供範例，在圖4中顯示一對應於由影像擷取裝置Ci(例如圖1範例裡的影像擷取裝置C1、C2或C3)所擷取之影像的像素矩陣。在該圖中，黑色像素是對應於位在影像擷取裝置Ci「魚眼」透鏡的有效中心圓形部份之外部的像素。利用該影像擷取裝置Ci所擷取之影像的各個像素是源生自一稱為「對映」的作業，此作業對應於如前文所述該球形3D表面部份中被該影像擷取裝置Ci之「魚眼」透鏡所觀看到並且特定於該影像擷取裝置Ci的二維投射。 To provide an example, a pixel matrix corresponding to an image captured by an image capturing device C i (for example, the image capturing devices C 1 , C 2, or C 3 in the example of FIG. 1) is shown in FIG. 4. In the figure, the black pixels correspond to the pixels located outside the effective center circle portion of the "fish-eye" lens of the image capturing device C i . Each pixel of the image captured by the image capturing device C i is derived from an operation called "map", which corresponds to the image captured by the image in the spherical 3D surface part as described above the device C i "fish-eye" lens to watch and are specific to the image capturing device C i a two-dimensional projection.

在本發明之前，為利用各個影像擷取裝置Ci所擷取的影像以呈顯一全景影像，這些影像通常是會在其等的重疊區域層級處經由影像數位「縫合」處理併行排置，藉此得到最終的連續全景影像。然重點是須瞭解在先前技藝中所引用的這種數位縫合處理並不會修改像素的二維投射，故而保留在最終全景影像上。 Prior to the present invention, in order to use the images captured by each image capture device C i to present a panoramic image, these images are usually arranged in parallel at the level of their overlapping regions through digital digital “stitching” processing. In this way, the final continuous panoramic image is obtained. However, it is important to understand that the digital stitching process cited in the prior art does not modify the two-dimensional projection of the pixels, and therefore remains on the final panoramic image.

在本發明中，不同於前述的先前技藝數位縫合，若要呈顯最終全景影像，由各台感測器Ci所擷取之各個影像的有效像素會被重對映在該最終全景影像上，並且該等像素會重對映至該最終全景影像上的至少一部份，最好是當遞交至一新二維投射時，此投射是不同於來自該等影像擷取裝置Ci並由此導出該等像素之影像上的二維投射。所以是利用該等影像擷取裝置C1、C2或C3來呈顯單一台虛擬全景影像擷取裝置。此像素重對映會透過各個所擷取影像之各個像素的方法自動地進行，這些至少包含保留或拋除該像素，而且當保留該像素時，將藉由對於該最終全景影像上之各個位置的加權因數以指配於該最終全景影像上的一或更多位置。 In the present invention, unlike the previous prior art digital stitching, if a final panoramic image is to be displayed, the effective pixels of each image captured by each sensor C i will be re-mapped on the final panoramic image. And the pixels will be re-reflected to at least a part of the final panoramic image, preferably when submitted to a new two-dimensional projection, this projection is different from the image capture device C i and is This leads to a two-dimensional projection on the image of the pixels. Therefore, these image capturing devices C 1 , C 2 or C 3 are used to present a single virtual panoramic image capturing device. This pixel re-mapping will be performed automatically through the method of each pixel of each captured image. These at least include retaining or discarding the pixel, and when the pixel is retained, the position of the final panoramic image will be determined by Weighting factor to assign one or more positions on the final panoramic image.

圖4中僅顯示該最終全景影像其一局部，此局部對應至該全景影像中源生於由單一影像擷取裝置Ci所擷取影像之像素重對映的部份。 FIG. 4 shows only a part of the final panoramic image, and this part corresponds to a part of the panoramic image originating from the re-mapping of pixels of the image captured by the single image capturing device C i .

現參照圖4，位在由影像擷取裝置Ci所擷取影像之第一條直 線上的像素P1,8會被例如以四個位於該最終全景影像第一條直線上之四個不同相鄰位置的像素P1,9、P1,10、P1,11、P1,12重對映到該最終全景影像上，這可轉譯為從該原始影像至該最終全景影像將此像素拉開。所以，此像素P1,8在該最終全景影像上的對映是對應於此像素在該最終全景影像上的二維投射，這是不同於此像素在由影像處理裝置所擷取之原始影像上的二維投射。這個像素在該最終全景影像上被拉開可例如有利地具體實作為部份地或整體地補償影像擷取裝置之「魚眼」透鏡在靠近上方邊緣處的光學扭曲。對於位在下方邊緣處的像素可有利地具體實作相同的像素拉開。 Referring now to FIG. 4, the pixels P 1,8 located on the first straight line of the image captured by the image capturing device C i will be, for example, four different on the first straight line of the final panoramic image. Pixels P 1,9 , P 1,10 , P 1,11 , P 1,12 in adjacent positions are re-mapped to the final panoramic image, which can be translated into the pixel from the original image to the final panoramic image Pull away. Therefore, the mapping of this pixel P 1,8 on the final panoramic image corresponds to the two-dimensional projection of this pixel on the final panoramic image, which is different from the original image captured by the image processing device of this pixel Two-dimensional projection. This pixel being pulled apart on the final panoramic image can, for example, advantageously be used as a partial or overall compensation for the optical distortion of the "fisheye" lens of the image capture device near the upper edge. For pixels located at the lower edge, it may be advantageous to implement the same pixels apart.

為供比較，由影像擷取裝置Ci所擷取之影像的中心像素P8,8會以唯一像素P11,11等同地重對映至該最終全景影像上，這是因為影像擷取裝置的「魚眼」透鏡在透鏡的中心處並不會或幾乎不會產生光學扭曲。 For comparison, the central pixel P 8,8 of the image captured by the image capture device C i will be re-equivalently mapped to the final panoramic image with the unique pixel P 11,11 , because the image capture device The "fish-eye" lens has no or almost no optical distortion at the center of the lens.

位在由影像擷取裝置Ci所擷取影像之左下方區域上的像素P10,3是以例如三個在該最終全景影像的兩條相鄰直線上之三個相鄰且不同位置處的像素P17,4、P18,4、P18,5重對映在該最終全景影像上，這對於該原始影像的此像素P10,3而言會轉譯為在兩個方向上放大到該最終全景影像內。從而，此像素P10,3在該最終全景影像上的對映是對應於此像素在該最終全景影像上的二維投射，這是不同於此像素在由影像擷取裝置所擷取之原始影像上的二維投射。 The pixels P 10,3 located on the lower left area of the image captured by the image capturing device C i are, for example, three adjacent and different positions on three adjacent straight lines of the final panoramic image. The pixels P 17,4 , P 18,4 , P 18,5 are re-reflected on the final panoramic image, which for this pixel P 10,3 of the original image will be translated to be enlarged in two directions to Within the final panoramic image. Therefore, the mapping of the pixel P 10,3 on the final panoramic image corresponds to the two-dimensional projection of the pixel on the final panoramic image, which is different from the original of the pixel captured by the image capturing device. Two-dimensional projection on the image.

在該原始影像之各個像素的重對映作業過程中，亦即自該影像擷取裝置Ci到該最終全景影像上，像素是可以無需在該最終全景影像上加以保留或復原。這會出現在例如位於由至少兩台影像擷取裝置所擷取影像之重疊區域內的像素。在影像擷取裝置的重疊區域裡，將只會保留來自該等感測器之其一者的單一像素，而對應於其他感測器的其他像素則不會予以保留。在本發明的其他變化項目中，至少兩台影像擷取裝置的重疊區域內，是可以利用原始影像像素的平均值或組合以呈顯最終全景影像。 During the re-mapping operation of each pixel of the original image, that is, from the image capturing device C i to the final panoramic image, the pixels need not be retained or restored on the final panoramic image. This may occur, for example, for pixels located in an overlapping area of an image captured by at least two image capture devices. In the overlapping area of the image capture device, only a single pixel from one of these sensors will be retained, and the other pixels corresponding to other sensors will not be retained. In other variations of the present invention, within the overlapping area of at least two image capturing devices, the average or combination of pixels of the original image can be used to present the final panoramic image.

在像素的重對映作業過程中，當保留像素時並且既已指配該最終全景影像上的一或更多不同位置後，最好是對於該最終全景影像上之各個位置，亦即對於該最終全景影像的各個像素，利用一範圍自0至100%的加權因數以進行該指配作業。參照圖5可更佳地瞭解該加權因數方法以及底層的理由。 During the remapping operation of the pixels, when the pixels are reserved and one or more different positions on the final panoramic image have been assigned, it is best to use the positions on the final panoramic image, that is, the Each pixel of the final panoramic image uses a weighting factor ranging from 0 to 100% to perform the assignment operation. The weighting factor method and the underlying reasons can be better understood with reference to FIG. 5.

現參照圖5，該最終全景影像之像素Pi,j的中心C在實作上並非對應於由一影像擷取裝置Ci所擷取影像之像素的中心，而是幾何地對應於由影像擷取裝置Ci所擷取影像上的一特定真實位置P，此位置在如圖4所表示的特例中是離開中心，位在下方角落的鄰近範圍內，並且對應於由影像擷取裝置Ci所擷取影像之像素P1的左側。因此，在此特定範例裡，將不僅會利用到像素P2，而亦利用其相鄰像素P1、P3、P4並同時藉由各個像素P1、P2、P3、P4的加權因數，例如考量到該位置P相對於各個像素P1、P2、P3、P4之中心的質心，以呈顯該像素Pi,j。在本特定範例裡，該像素Pi,j含有例如25%的像素P1，35%的像素P2，15%的像素P3以及5%的像素P4。 Referring now to FIG. 5, the center C of the pixels P i, j of the final panoramic image does not correspond to the center of the pixels of the image captured by an image capturing device C i in practice, but corresponds geometrically to the image A specific real position P on the image captured by the capturing device C i . This position is away from the center in the special case shown in FIG. 4 and is located in the vicinity of the lower corner and corresponds to the image capturing device C. the pixel P i on the left side of an image capturing. Therefore, in this particular example, not only pixel P 2 will be used, but also its neighboring pixels P 1 , P 3 , and P 4 and at the same time, each pixel P 1 , P 2 , P 3 , and P 4 will be used . The weighting factor, for example, considers the centroid of the position P relative to the center of each pixel P 1 , P 2 , P 3 , and P 4 to show the pixel P i, j . In this specific example, the pixels P i, j include, for example, 25% of pixels P 1 , 35% of pixels P 2 , 15% of pixels P 3, and 5% of pixels P 4 .

本發明適用於所有類型的影像格式，包含：RAW、YUV及RGB衍生項目。對於RGB影像來說，在進行過色彩呈顯後(稱為對於各個影像像素的R、G、B資訊)，將可利用相鄰像素以實作前述的加權因數。 The invention is applicable to all types of image formats, including: RAW, YUV, and RGB-derived items. For RGB images, after color rendering (referred to as R, G, B information for each image pixel), adjacent pixels can be used to implement the aforementioned weighting factor.

然對於RAW影像而言，由於各個像素僅表現一個色度成份，因此將會利用具有與該最終全景影像的像素相同之色彩的鄰近像素來實作前述的加權因數。可參照圖6A至6I以更佳地瞭解這種對於RAW格式之加權因數的特定情形。 However, for a RAW image, since each pixel represents only one chrominance component, neighboring pixels having the same color as the pixels of the final panoramic image will be used to implement the aforementioned weighting factor. Reference can be made to FIGS. 6A to 6I to better understand this particular case of the weighting factor for the RAW format.

圖6A至6I說明，對於依RAW類型格式編碼之像素的情況，最終全景影像之像素Pi,j與由影像擷取裝置Ci所擷取影像之像素矩陣間的各種對應方式。在該等圖式中，字母R、G、B分別地對應於「紅光」、「綠光」及「藍光」像素。Wi為由該影像擷取裝置所擷取原始影像的像素Ri、Gi或 Bi之最終影像上的加權因數。 6A to 6I illustrate various correspondences between the pixels P i, j of the final panoramic image and the pixel matrix of the image captured by the image capturing device C i for the case of pixels encoded in the RAW type format. In these drawings, the letters R, G, and B correspond to the "red", "green", and "blue" pixels, respectively. W i is a weighting factor on the final image of the pixels R i , G i or B i of the original image captured by the image capturing device.

圖6A對應於如下情況，即該最終全景影像之紅光像素Pi,j的中心對應於由影像擷取裝置Ci所擷取之影像裡的真實位置P，而這是位在由該影像擷取裝置Ci所擷取之影像的藍光像素(B)上。在此情況下，會利用鄰近該藍光像素B的紅光像素R1、R2、R3、R4藉由個別地施用加權因數W1、W2、W3、W4以呈顯該最終全景影像的此紅光像素Pi,j。而這些加權因數W1、W2、W3、W4的數值將會是例如依照位置P相對於各個像素R1、R2、R3、R4之中心的質心而定。例如，若該位置P位在該像素P的中心處，在此時所有的加權因數W1、W2、W3、W4都將會是25%。 FIG. 6A corresponds to the case where the center of the red pixel P i, j of the final panoramic image corresponds to the real position P in the image captured by the image capturing device C i , and this is located in the image The blue light pixel (B) of the image captured by the capturing device C i . In this case, the red light pixels R 1 , R 2 , R 3 , R 4 adjacent to the blue light pixel B are used to individually apply the weighting factors W 1 , W 2 , W 3 , W 4 to show the final result. This red light pixel P i, j of the panoramic image. The values of these weighting factors W 1 , W 2 , W 3 , and W 4 will be determined according to the centroid of the position P relative to the centers of the respective pixels R 1 , R 2 , R 3 , and R 4 . For example, if the position P is located at the center of the pixel P, all weighting factors W 1 , W 2 , W 3 , and W 4 will be 25% at this time.

圖6B對應於如下情況，即該最終全景影像之藍光像素Pi,j的中心對應於由一感測器Ci所擷取之影像裡的真實位置P，而這是位在由一影像擷取裝置Ci所擷取之影像的紅光像素(R)上。 FIG. 6B corresponds to the case where the center of the blue light pixel P i, j of the final panoramic image corresponds to the real position P in the image captured by a sensor C i , and this is located in an image capture Take the red light pixel (R) of the image captured by the device C i .

圖6C對應於如下情況，即該最終全景影像之綠光像素Pi,j的中心對應於由一感測器Ci所擷取之影像裡的真實位置P，而這是位在由一影像擷取裝置Ci所擷取之影像的藍光像素(B)上。 FIG. 6C corresponds to the case where the center of the green pixel P i, j of the final panoramic image corresponds to the real position P in the image captured by a sensor C i , and this is located in an image The blue light pixel (B) of the image captured by the capturing device C i .

圖6D對應於如下情況，即該最終全景影像之綠光像素Pi,j的中心對應於由一感測器Ci所擷取之影像裡的真實位置P，而這是位在由一影像擷取裝置Ci所擷取之影像的紅光像素(R)上。 FIG. 6D corresponds to the case where the center of the green pixel P i, j of the final panoramic image corresponds to the real position P in the image captured by a sensor C i , and this is located in an image The red light pixel (R) of the image captured by the capturing device C i .

圖6E對應於如下情況，即該最終全景影像之綠光像素Pi,j的中心對應於由一感測器Ci所擷取之影像裡的真實位置P，而這是位在由一影像擷取裝置Ci所擷取之影像的綠光像素(G5)上。 FIG. 6E corresponds to the case where the center of the green pixel P i, j of the final panoramic image corresponds to the real position P in the image captured by a sensor C i , and this is located in an image The green light pixel (G 5 ) of the image captured by the capturing device C i .

圖6F對應於如下情況，即該最終全景影像之紅光像素Pi,j的中心對應於由一感測器Ci所擷取之影像裡的真實位置P，而這是位在由一影像擷取裝置Ci所擷取之影像的綠光像素(G)上。 FIG. 6F corresponds to the case where the center of the red pixel P i, j of the final panoramic image corresponds to the real position P in the image captured by a sensor C i , and this is located in an image On the green pixel (G) of the image captured by the capturing device C i .

圖6G對應於如下情況，即該最終全景影像之藍光像素Pi,j 的中心對應於由一感測器Ci所擷取之影像裡的真實位置P，而這是位在由一影像擷取裝置Ci所擷取之影像的綠光像素(G)上。 FIG. 6G corresponds to the case where the center of the blue light pixel P i, j of the final panoramic image corresponds to the real position P in the image captured by a sensor C i , and this is located in an image capture Take the green pixel (G) of the image captured by the device C i .

圖6H對應於如下情況，即該最終全景影像之紅光像素Pi,j的中心對應於由一影像擷取裝置Ci所擷取之影像裡的真實位置P，而這是位在由一影像擷取裝置Ci所擷取之影像的紅光像素(R5)上。 FIG. 6H corresponds to the case where the center of the red light pixel P i, j of the final panoramic image corresponds to the real position P in the image captured by an image capturing device C i , and this is located in a The red light pixel (R 5 ) of the image captured by the image capturing device C i .

圖6I對應於如下情況，即該最終全景影像之藍光像素Pi,j的中心對應於由一感測器Ci所擷取之影像裡的真實位置P，而這是位在由一影像擷取裝置Ci所擷取之影像的藍光像素(B5)上。 FIG. 6I corresponds to the case where the center of the blue light pixel P i, j of the final panoramic image corresponds to the real position P in the image captured by a sensor C i , and this is located in an image capture Take the blue light pixel (B 5 ) of the image captured by the device C i .

最後，無論影像的編碼格式如何，對由一影像擷取裝置Ci所擷取影像中各個像素之最終全景影像的重對映方法皆至少包含保留或拋除該像素，並且當保留該像素時，即藉一對於該最終全景影像之各個位置(亦即對於各個像素)的預定義加權因數以指配於該最終全景或立體影像上的一或更多不同位置。在本文中，最終全景影像上之「位置」的概念是融合於最終全景影像上之「像素」的概念。 Finally, regardless of the encoding format of the image, the re-mapping method for the final panoramic image of each pixel in the image captured by an image capture device C i includes at least retaining or discarding the pixel, and when the pixel is retained That is, a predefined weighting factor for each position (ie, for each pixel) of the final panoramic image is borrowed to assign one or more different positions on the final panoramic or stereo image. In this article, the concept of "position" on the final panoramic image is a concept of "pixels" that is integrated into the final panoramic image.

根據本發明，當有智慧地進行像素重對映時，確可至少部份地校正例如在各個影像擷取裝置Ci中各個透鏡之最終影像上的扭曲情況。 According to the present invention, when the pixel remapping is performed intelligently, it is possible to at least partially correct, for example, the distortion on the final image of each lens in each image capturing device C i .

同時亦根據本發明，可例如由額外電子處理裝置11將該等影像擷取裝置C1、C2、C3以及該等電子處理裝置10視為是對於全景影像的唯一虛擬感測器。因此，該額外電子處理裝置11可對於由該等電子處理裝置10所遞交的最終全景影像具體實作例如已知的影像處理演算法(尤其是用於白色均衡、曝光時間與增益管理的演算法)，而每當適用時，相較於在呈顯該全景影像之前實作這些演算法以對於由該等影像擷取裝置C1,C2,C3遞交的各個影像進行影像處理來說，如此可供獲得更為均勻的最終影像，特別是關於色度、白色均衡及曝光時間與增益方面尤甚。 At the same time, according to the present invention, the image capture devices C 1 , C 2 , C 3 and the electronic processing devices 10 can be regarded as the only virtual sensors for panoramic images by the additional electronic processing device 11. Therefore, the additional electronic processing device 11 can implement, for example, known image processing algorithms (especially algorithms for white equalization, exposure time and gain management) on the final panoramic images submitted by the electronic processing devices 10. ), And whenever applicable, rather than implementing the algorithms to perform image processing on each image submitted by the image capture devices C 1 , C 2 , C 3 before displaying the panoramic image, This allows for a more uniform final image, especially with regard to chroma, white balance, and exposure time and gain.

僅為簡便之目的，然非限制本發明範疇，在圖7A至7D中 提供來自一「魚眼」透鏡的原始影像之直線L的像素重對映特定範例，藉以分解該「魚眼」透鏡的光學扭曲以及其空間指向(偏指(Yaw)、俯仰(Pitch)和橫滾(Roll))。重對映是依照該直線L相較於該「魚眼」透鏡之中心以及下方與上方邊緣的位置而定(參見圖7A、7B、7C)，或者是依照該「魚眼」透鏡的空間指向而定(如圖7D)。 For simplification purposes only, without limiting the scope of the invention, in FIGS. 7A to 7D Provides a specific example of pixel re-mapping of the straight line L of the original image from a "fisheye" lens to resolve the optical distortion of the "fisheye" lens and its spatial orientation (Yaw, Pitch, and horizontal Roll). Double reflection is based on the position of the straight line L relative to the center of the "fisheye" lens and the lower and upper edges (see Figures 7A, 7B, 7C), or according to the spatial orientation of the "fisheye" lens It depends (as shown in Figure 7D).

圖8中提供一特定範例，此圖說明分別地由三個影像擷取裝置C1、C2、C3所擷取之三個影像I1、I2、I3，以及自該等影像I1、I2、I3之像素重對映所獲得的最終全景影像(I)。 Providing a particular example of FIG. 8, this figure illustrates three respectively by the image capturing device C 1, C 2, C 3 of the three captured images I 1, I 2, I 3 , and from these image I 1, I 2, the final panoramic image (I) I 3 pixels weight of the obtained enantiomers.

在本發明的框架裡，是可經由實作不同於該等影像擷取裝置C1、C2、C3之二維投射的任何類型二維投射，例如為在一最終全景影像上自動地併入特殊效果之目的，利用像素重對映來呈顯最終全景影像。尤其可實作下列的已知投射方式：- 平面性或直線性投射；- 圓柱形投射；- 墨卡托投射；- 球形或等長方形投射。 In the framework of the present invention, it is possible to implement any type of two-dimensional projection different from the two-dimensional projection of the image capturing devices C 1 , C 2 , C 3 , for example, to automatically combine the two final projections on a final panoramic image. For the purpose of special effects, pixel re-mapping is used to display the final panoramic image. In particular, the following known projection methods can be implemented:-flat or linear projection;-cylindrical projection;-Mercator projection;-spherical or equal rectangular projection.

為進行重對映作業，熟諳本項技藝之人士必須依逐一個案的方式預先定義各台影像擷取裝置Ci之各個像素的重對映，針對各台影像擷取裝置Ci的各個像素決定是否保留此像素，而在此情況下對應於該最終全景影像的像素或多個像素，以及此原始像素對於該最終全景影像之各個像素的加權因數。 For heavy work enantiomers, familiar to skilled persons must be by way of the cases one by a predefined weight of each pixel in each stage of the image capturing device C i enantiomers, determined for each pixel in each stage of the image capturing device C i Whether to keep this pixel, and in this case the pixel or pixels corresponding to the final panoramic image, and the weighting factor of this original pixel to each pixel of the final panoramic image.

此重對映可為例如依照下列類型的「對應表」所實作，即對各個影像擷取裝置Ci中獲保留在該最終全景影像上的各個像素PX,Y指配於該最終全景影像上的其一或多個像素(PXpano,Ypano)，而在該最終全景影像的像素(PXpano,Ypano)上具有該像素PX,Y的加權因數。在下列表格裡，為便於簡化，僅按 示範性目的納入圖4中所舉例列出的特定像素。 This re-mapping can be implemented, for example, in accordance with the following type of "correspondence table", that is, each pixel P X, Y in each image capturing device C i that is retained on the final panoramic image is assigned to the final panoramic image. One or more pixels (P Xpano, Ypano ) on the image, and a pixel (P Xpano, Ypano ) of the final panoramic image has a weighting factor of the pixel P X, Y. In the following table, for simplicity, the specific pixels listed in Figure 4 are included for exemplary purposes only.

Figure TWI612495BD00001
Figure TWI612495BD00001

對於圖1架構的特定情況，會按照儲存在該等記憶體其一者內的「對應表」，利用該等電子處理裝置10以自動地進行對於各台影像擷取裝置C1、C2、C3中各個像素之最終全景影像的重對映作業。在本發明的另一變化項目裡，亦可藉由該等電子處理裝置10利用儲存在該記憶體內的校準及動態計算演算法以自動地進行對於各台影像擷取裝置C1、C2、C3中各個像素之最終全景影像的重對映計算作業。 For the specific case of FIG. 1 architecture, will "correspondence table" in accordance with those stored in one's memory, to perform image capturing device for each stage C 1, C 2 automatically use such electronic processing device 10, the final weight of each pixel in the panoramic image of the job C 3 enantiomers. In another variation of the present invention, the electronic processing devices 10 can also use the calibration and dynamic calculation algorithms stored in the memory to automatically perform the image capture devices C 1 , C 2 , the final weight of each pixel in the panoramic image of the job C 3 enantiomers calculation.

在圖1的範例裡，獲自於該重對映作業之全景影像的各個像素(PXpano,Ypano)會被遞交作為該等電子處理裝置10的輸出(「Pixels」)，此者是按照由該等電子處理裝置10遞交的「H」時脈信號所同步化。根據替代性具體實施例，這個由該等電子處理裝置10所遞交的「H」時脈信號可為同步或非同步於由該等影像感測器C1、C2、C3所遞交的「H_sensor」時脈信號。 In the example of FIG. 1, each pixel (P Xpano, Ypano ) of the panoramic image obtained from the re-mapping operation is submitted as the output ("Pixels") of the electronic processing device 10, which is based on the The "H" clock signals transmitted by the electronic processing devices 10 are synchronized. According to an alternative embodiment, the “H” clock signal submitted by the electronic processing devices 10 may be synchronized or asynchronous to the “H” clock signal submitted by the image sensors C 1 , C 2 , and C 3 . H_sensor "clock signal.

圖1架構的其一優點為可供額外的電子處理裝置11「觀看」到影像感測器C1、C2、C3，同時該等電子處理裝置10則是作為單一虛擬全景感測器。 One advantage of the architecture of FIG. 1 is that additional electronic processing devices 11 can “view” the image sensors C 1 , C 2 , and C 3 , and the electronic processing devices 10 are used as a single virtual panoramic sensor.

圖1的裝置可有利地運用在當該等電子處理裝置10取得該等像素時進行即時性重對映。 The device of FIG. 1 can be advantageously used for instant re-mapping when the electronic processing devices 10 obtain the pixels.

本發明不限於實作三台固定的影像擷取裝置C1、C2、C3，而是可更廣泛地以至少兩台固定的影像擷取裝置C1、C2所實作。 The present invention is not limited to implementing three fixed image capturing devices C 1 , C 2 , and C 3 , but can be more widely implemented with at least two fixed image capturing devices C 1 , C 2 .

在本發明框架裡亦可預期運用單一台行動影像擷取裝置，而各次影像擷取則是對應於該行動影像擷取裝置C1、C2、C3的其一不同指向及/或位置。 It is also conceivable to use a single mobile image capture device in the framework of the present invention, and each image capture corresponds to a different orientation and / or position of the mobile image capture device C 1 , C 2 , C 3 .

在前述具體實施例的特定變化項目中，擷取頻率F是等於該等行動影像擷取裝置C1、C2、C3的擷取頻率。然在其他變化項目裡，該擷取頻率F可為小於該等行動影像擷取裝置C1、C2、C3的擷取頻率，例如該等電子處理裝置僅處理由該等感測器各者所遞交之m個影像(m

Figure TWI612495BD00002
2)中的其一影像，這是對應於該等接續擷取作業的頻率，而此作業頻率是小於由該等行動影像擷取裝置C1、C2、C3遞交影像的頻率。 In a specific variation of the foregoing specific embodiment, the capturing frequency F is equal to the capturing frequencies of the mobile image capturing devices C 1 , C 2 , and C 3 . However, in other changes, the capturing frequency F may be smaller than the capturing frequencies of the mobile image capturing devices C 1 , C 2 , and C 3. For example, the electronic processing device only processes the signals from the sensors. M images (m
Figure TWI612495BD00002
One of the images in 2) corresponds to the frequency of the successive capturing operations, and the operating frequency is less than the frequency of the images submitted by the mobile image capturing devices C 1 , C 2 , and C 3 .

本發明並不受限於呈顯全景影像。本發明亦可適用於呈顯立體影像。 The invention is not limited to displaying panoramic images. The invention can also be applied to display stereoscopic images.

Claims (31)

一種用於擷取與形成一場景之全景或立體影像串流的方法，其中，利用至少一影像擷取裝置進行多次接續擷取作業，每一次擷取作業包含按照像素格式對該場景的至少兩個不同影像進行擷取，該等影像可為重疊或不重疊，其中，該等接續擷取作業是按一頻率所計時，其中，在該等影像擷取作業的過程中，所擷取影像的像素係經數位處理而從每一次擷取作業來形成一全景或立體影像，同時在該等影像擷取作業的過程中產生該全景或立體影像的串流，其中，對各個所擷取影像之各個像素的數位處理至少包含保留或拋除該像素，並且當保留該像素時，藉由對於最終全景或立體影像上之各個位置的預定義加權因數將該像素指配於最終全景或立體影像上的其一或多個位置。 A method for capturing and forming a panoramic or stereoscopic image stream of a scene, wherein at least one image capturing device is used to perform multiple consecutive capturing operations, and each capturing operation includes at least the scene in a pixel format. Two different images are captured. The images may be overlapping or non-overlapping. Among them, the successive capturing operations are timed according to a frequency, and during the image capturing operations, the captured images are captured. The pixels are digitally processed to form a panoramic or stereoscopic image from each capture operation, and a stream of the panoramic or stereoscopic images is generated during the image capture operations, where each captured image is Digital processing of each pixel at least includes retaining or discarding the pixel, and when the pixel is retained, assigning the pixel to the final panoramic or stereo image by a predefined weighting factor for each position on the final panoramic or stereo image One or more positions on it. 如申請專利範圍第1項所述的方法，其中，該等接續擷取作業的頻率定義兩次接續擷取作業之起始間的擷取時間，其中，對於各次擷取作業，每一次所擷取的影像的像素係經數位處理藉以利用該等像素以形成最終全景或立體影像，而處理時間是短於或等於該擷取時間，以及在短於或等於該擷取時間之時間間隔的過程裡產生最終全景或立體影像。 The method according to item 1 of the scope of patent application, wherein the frequency of the successive acquisition operations defines the acquisition time between the start of two successive acquisition operations, and for each acquisition operation, each time The pixels of the captured image are digitally processed to use the pixels to form the final panoramic or stereo image, and the processing time is shorter than or equal to the capture time, and at time intervals shorter than or equal to the capture time. The process produces the final panoramic or stereo image. 如申請專利範圍第1項所述的方法，其中，以與該等接續擷取作業的頻率相同的頻率產生全景或立體影像的串流。 The method according to item 1 of the scope of patent application, wherein a stream of panoramic or stereoscopic images is generated at the same frequency as the frequency of these successive acquisition operations. 如申請專利範圍第1項所述的方法，其中該等接續擷取作業的頻率定義兩次接續影像擷取作業之起始間的擷取時間，並且該擷取時間為短於或等於1秒。 The method according to item 1 of the scope of patent application, wherein the frequency of the successive acquisition operations defines the acquisition time between the start of two successive image acquisition operations, and the acquisition time is shorter than or equal to 1 second . 如申請專利範圍第1項所述的方法，其中各個最終全景或立體影像 是在分隔兩次接續影像擷取作業之起始的各個時間間隔過程中接續地產生。 Method according to item 1 of the patent application, wherein each final panoramic or stereo image It is successively generated during the time intervals that separate the beginning of two consecutive image capture operations. 如申請專利範圍第5項所述的方法，其中在分隔兩次接續擷取作業之起始的時間間隔過程中所產生的最終全景或立體影像是源生自在相同的時間間隔過程中所進行的像素數位處理。 The method described in item 5 of the scope of patent application, wherein the final panoramic or stereo image generated during the time interval separating the two successive acquisition operations is originated from the same time interval Pixel digital processing. 如申請專利範圍第5項所述的方法，其中在分隔兩次接續擷取作業之起始的時間間隔過程中所產生的最終全景或立體影像是源生自在前行的時間間隔過程中所進行的像素數位處理。 The method as described in item 5 of the scope of patent application, wherein the final panoramic or stereo image generated during the time interval that separates two successive acquisition operations is generated during the previous time interval Pixel digital processing. 如申請專利範圍第1項所述的方法，其中進行各個像素的數位處理，使得所擷取影像之像素的至少一部份在經遞交至一不同於該等相同像素之二維投射而映至該等所源自於的影像擷取裝置之影像上的二維投射後會被對映到最終全景或立體影像上。 The method according to item 1 of the scope of patent application, wherein digital processing is performed on each pixel, so that at least a part of the captured image is reflected to a two-dimensional projection different from the same pixels. The two-dimensional projection on the images from these image capture devices will be projected onto the final panoramic or stereo image. 如申請專利範圍第1項所述的方法，其中所擷取影像的多個像素各者會被指配最終全景或立體影像上的多個不同位置。 The method according to item 1 of the scope of patent application, wherein each of the plurality of pixels of the captured image is assigned to a plurality of different positions on the final panoramic or stereo image. 如申請專利範圍第1項所述的方法，其中所擷取影像的多個像素各者會藉不為零、且嚴格地小於100%的加權因數指配至最終全景或立體影像上的一位置。 The method according to item 1 of the scope of patent application, wherein each pixel of the captured image is assigned to a position on the final panoramic or stereo image by a weighting factor that is not zero and strictly less than 100% . 如申請專利範圍第1項所述的方法，其中，利用至少兩個不同的影像擷取裝置以擷取該場景的至少兩個不同影像。 The method of claim 1, wherein at least two different image capturing devices are used to capture at least two different images of the scene. 如申請專利範圍第1項所述的方法，其中，利用至少三個影像擷取裝置以擷取該場景的至少三個不同影像。 The method of claim 1, wherein at least three image capturing devices are used to capture at least three different images of the scene. 如申請專利範圍第1項所述的方法，其中該等接續擷取作業的頻率 定義兩次接續影像擷取作業之起始間的擷取時間，並且該擷取時間為短於或等於100ms。 The method as described in item 1 of the patent application scope, wherein the frequency of these successive acquisition operations Define the capture time between the start of two consecutive image capture operations, and the capture time is shorter than or equal to 100ms. 一種用於擷取與形成全景或立體影像串流的裝置，其中，該裝置包含一或更多影像擷取裝置，按照像素集組格式擷取至少兩個不同影像，以及電子處理裝置，利用該等影像擷取裝置進行多次接續擷取作業，每一次擷取作業包含按照像素格式對一場景的至少兩個不同影像進行擷取，該等影像可為重疊或不重疊，而該等接續擷取作業係經調適以在擷取作業的過程中對所擷取影像的像素進行數位處理藉以形成全景或立體影像，同時在擷取作業的過程中產生該等全景或立體影像的串流，其中，該電子處理裝置利用該等影像擷取裝置按一頻率進行該等接續影像擷取作業，其中，對各個所擷取影像之各個像素的數位處理至少包含保留或拋除該像素，而且當保留該像素時，藉由對於最終全景或立體影像上之各個位置的預定義加權因數將該像素指配於最終全景或立體影像上的其一或多個位置。 A device for capturing and forming a panoramic or stereo image stream, wherein the device includes one or more image capturing devices, capturing at least two different images according to a pixel set group format, and an electronic processing device using the Wait until the image capture device performs multiple consecutive capture operations. Each capture operation includes capturing at least two different images of a scene according to the pixel format. These images can be overlapping or non-overlapping, and these consecutive captures The fetching operation is adapted to digitally process the pixels of the captured image during the capturing operation to form a panoramic or stereoscopic image, while generating a stream of such panoramic or stereoscopic images during the capturing operation, where The electronic processing device uses the image capture devices to perform the subsequent image capture operations at a frequency, wherein the digital processing of each pixel of each captured image includes at least retaining or discarding the pixel, and when retaining The pixel, the pixel is assigned to the final panorama or stereo image by a predefined weighting factor for each position on the final panorama or stereo image One or more locations on the image. 如申請專利範圍第14項所述的裝置，其中，該等接續擷取作業的頻率定義兩次接續擷取作業之起始間的擷取時間，其中，對於各次擷取作業，該電子處理裝置係經調適以數位處理各個所擷取影像的像素藉以利用該等像素以形成最終全景或立體影像，而處理時間是短於或等於該擷取時間，以及在短於或等於該擷取時間之時間間隔的過程裡產生先前形成的最終全景或立體影像。 The device according to item 14 of the scope of the patent application, wherein the frequency of the successive acquisition operations defines the acquisition time between the start of two successive acquisition operations, and for each acquisition operation, the electronic processing The device is adapted to digitally process the pixels of each captured image to utilize the pixels to form the final panoramic or stereoscopic image, and the processing time is shorter than or equal to the acquisition time, and shorter than or equal to the acquisition time. During the time interval, a previously formed final panoramic or stereo image is generated. 如申請專利範圍第14項所述的裝置，其中，該等接續擷取作業的頻率定義兩次接續擷取作業之起始間的擷取時間，且其中該電子處理裝置係經調適以按與該擷取頻率相同的頻率來產生該等最終全景或立體影像。 The device according to item 14 of the scope of patent application, wherein the frequency of the successive acquisition operations defines the acquisition time between the start of two successive acquisition operations, and wherein the electronic processing device is adapted to The acquisition frequency is the same frequency to generate the final panoramic or stereoscopic images. 如申請專利範圍第14項所述的裝置，其中該電子處理裝置利用該(等)影像擷取裝置以按該等接續影像擷取作業的頻率進行該等接續擷取作業，而此頻率定義兩次接續擷取作業之起始間的擷取時間，並且該擷取時間為短於或等於1秒(s)。 The device according to item 14 of the scope of patent application, wherein the electronic processing device uses the image capture device (s) to perform the consecutive capture operations at the frequency of the consecutive image capture operations, and this frequency defines two The acquisition time between the beginnings of successive acquisition operations, and the acquisition time is shorter than or equal to 1 second (s). 如申請專利範圍第14項所述的裝置，其中該電子處理裝置係經設計以在分隔兩次接續影像擷取作業之起始的各個時間間隔過程中接續地產生各個最終全景或立體影像。 The device according to item 14 of the scope of patent application, wherein the electronic processing device is designed to successively generate each final panoramic or stereoscopic image during each time interval that separates the start of two consecutive image capture operations. 如申請專利範圍第18項所述的裝置，其中在分隔兩次接續影像擷取作業之起始的時間間隔過程中所產生的最終全景或立體影像是源生自在相同的時間間隔過程中所進行的像素數位處理。 The device according to item 18 of the scope of patent application, wherein the final panoramic or stereoscopic image generated during the time interval separating the start of two consecutive image capture operations is originated from the same time interval Pixel digital processing. 如申請專利範圍第18項所述的裝置，其中在分隔兩次接續影像擷取作業之起始的時間間隔過程中所產生的最終全景或立體影像是源生自在前行的時間間隔過程中所進行的像素數位處理。 The device according to item 18 of the scope of patent application, wherein the final panoramic or stereoscopic image generated during the time interval separating the start of two consecutive image capturing operations is generated during the time interval that originated in the previous time. Digital pixel processing. 如申請專利範圍第14項所述的裝置，其中該電子處理裝置係經設計以對各個像素進行數位處理，使得來自所擷取影像之像素的至少一部份在經遞交至一不同於該等相同像素之二維投射而映至該等所源自於的影像擷取裝置之影像上的二維投射後會被對映到最終全景或立體影像上。 The device according to item 14 of the scope of patent application, wherein the electronic processing device is designed to digitally process each pixel so that at least a part of the pixels from the captured image is submitted to a The two-dimensional projection of the same pixels onto the two-dimensional projection of the images from the image capturing devices will be mapped onto the final panoramic or stereo image. 如申請專利範圍第14項所述的裝置，其中該電子處理裝置係經設計以藉由指配至最終全景或立體影像上之其一或多個不同位置各者以對來自所擷取影像的多個像素進行處理。 The device according to item 14 of the scope of patent application, wherein the electronic processing device is designed to assign to one or more different positions on the final panoramic or stereoscopic image each of the images from the captured image. Multiple pixels are processed. 如申請專利範圍第14項所述的裝置，其中該電子處理裝置係經設計以藉由利用不為零、且嚴格地小於100%的加權因數，指配各個像素至最終 全景或立體影像上之至少一位置以對來自所擷取影像的多個像素進行處理。 The device according to item 14 of the patent application scope, wherein the electronic processing device is designed to assign each pixel to the final one by using a weighting factor that is not zero and strictly less than 100% At least one position on the panoramic or stereo image to process multiple pixels from the captured image. 如申請專利範圍第14項所述的裝置，其中含有至少兩個影像擷取裝置。 The device according to item 14 of the patent application scope, which comprises at least two image capturing devices. 如申請專利範圍第14項所述的裝置，其中含有至少三個影像擷取裝置。 The device according to item 14 of the patent application scope, which comprises at least three image capturing devices. 如申請專利範圍第14項所述的裝置，其中該等影像擷取裝置各者係經設計以遞交一至少依照一第一時脈信號所同步化的像素串流作為對於各個所擷取影像的輸出，並且該電子處理裝置係經調適以遞交各個最終全景或立體影像作為按照至少一第二時脈信號所同步化的像素串流。 The device according to item 14 of the scope of the patent application, wherein each of the image capturing devices is designed to submit a pixel stream synchronized at least according to a first clock signal as each of the captured images. Output, and the electronic processing device is adapted to submit each final panoramic or stereo image as a pixel stream synchronized according to at least a second clock signal. 如申請專利範圍第26項所述的裝置，其中該第二時脈信號相較於各個第一時脈信號為非同步。 The device according to item 26 of the patent application, wherein the second clock signal is asynchronous compared to each of the first clock signals. 如申請專利範圍第26項所述的裝置，其中該第二時脈信號相較於該(等)第一時脈信號為同步。 The device of claim 26, wherein the second clock signal is synchronized compared to the first clock signal. 如申請專利範圍第14項所述的裝置，其中該電子裝置含有一預先儲存的「對應表」，該表為針對於利用該等至少一影像擷取裝置所擷取之影像的各個像素，而該像素在最終全景或立體影像上的(多個)相對應位置，所編碼；以及針對於該像素在最終全景或立體影像上的各個位置，而經該像素在最終全景或立體影像上的加權因數，所編碼。 The device according to item 14 of the scope of patent application, wherein the electronic device contains a pre-stored "correspondence table" for each pixel of an image captured by the at least one image capture device, and The corresponding position (s) of the pixel on the final panoramic or stereoscopic image, encoded; and for each position of the pixel on the final panoramic or stereoscopic image, weighted by the pixel on the final panoramic or stereoscopic image Factor, encoded. 如申請專利範圍第14項所述的裝置，其中，該裝置為可攜式。 The device according to item 14 of the scope of patent application, wherein the device is portable. 如申請專利範圍第14項所述的裝置，其中該等接續擷取作業的頻率定義兩次接續影像擷取作業之起始間的擷取時間，並且該擷取時間為短於 或等於100ms。 The device according to item 14 of the scope of the patent application, wherein the frequency of these successive acquisition operations defines the acquisition time between the start of two successive image acquisition operations, and the acquisition time is shorter than Or equal to 100ms.
TW102141393A 2012-11-15 2013-11-14 Process and device for capturing and rendering a panoramic or stereoscopic stream of images TWI612495B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FR1260880A FR2998126B1 (en) 2012-11-15 2012-11-15 METHOD AND DEVICE FOR CAPTURING AND CONSTRUCTING A FLOW OF PANORAMIC OR STEREOSCOPIC IMAGES
??1260880 2012-11-15

Publications (2)

Publication Number Publication Date
TW201435792A TW201435792A (en) 2014-09-16
TWI612495B true TWI612495B (en) 2018-01-21

Family

ID=47754666

Family Applications (2)

Application Number Title Priority Date Filing Date
TW102141393A TWI612495B (en) 2012-11-15 2013-11-14 Process and device for capturing and rendering a panoramic or stereoscopic stream of images
TW106135936A TW201804432A (en) 2012-11-15 2013-11-14 Process and device for capturing and rendering a panoramic or stereoscopic stream of images

Family Applications After (1)

Application Number Title Priority Date Filing Date
TW106135936A TW201804432A (en) 2012-11-15 2013-11-14 Process and device for capturing and rendering a panoramic or stereoscopic stream of images

Country Status (15)

Country Link
US (1) US20150288864A1 (en)
EP (1) EP2920956A1 (en)
JP (2) JP2016503618A (en)
KR (1) KR20150084807A (en)
CN (1) CN104782114B (en)
AU (1) AU2013346603B2 (en)
BR (1) BR112015010788A8 (en)
CA (1) CA2889811A1 (en)
FR (4) FR2998126B1 (en)
HK (1) HK1212835A1 (en)
IL (1) IL238622A0 (en)
IN (1) IN2015DN03812A (en)
MX (1) MX355297B (en)
TW (2) TWI612495B (en)
WO (1) WO2014076402A1 (en)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3190780A1 (en) * 2016-01-05 2017-07-12 Giroptic Two-lens spherical camera
US10943340B2 (en) 2016-04-18 2021-03-09 Avago Technologies International Sales Pte. Limited Blending images
EP3249929A1 (en) * 2016-05-25 2017-11-29 Thomson Licensing Method and network equipment for establishing a manifest
US20180018807A1 (en) * 2016-07-15 2018-01-18 Aspeed Technology Inc. Method and apparatus for generating panoramic image with texture mapping
CN108513119A (en) * 2017-02-27 2018-09-07 阿里巴巴集团控股有限公司 Mapping, processing method, device and the machine readable media of image
KR101925011B1 (en) * 2017-03-14 2019-02-27 한국과학기술원 Method and apparatus for inserting and detecting wartermark
TWI775869B (en) * 2017-06-29 2022-09-01 佳能企業股份有限公司 Image capture apparatus and image processing method
TWI642301B (en) * 2017-11-07 2018-11-21 宏碁股份有限公司 Image processing method and electronic system
KR102620783B1 (en) * 2019-03-10 2024-01-04 구글 엘엘씨 360-degree wide-angle camera with baseball stitching
KR102294071B1 (en) * 2020-07-21 2021-08-26 금오공과대학교 산학협력단 Method of object location display conversion and ID assignment in augmented reality based O2O camera solution
KR102555534B1 (en) * 2021-06-28 2023-07-17 한국과학기술원 Method and apparatus for detecting wartermark for spherical panoramic image

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030135675A1 (en) * 2002-01-17 2003-07-17 Koninklijke Philips Electronics N.V. Configurable synchronous or asynchronous bus interface
US20040061787A1 (en) * 2002-09-30 2004-04-01 Zicheng Liu Foveated wide-angle imaging system and method for capturing and viewing wide-angle images in real time
US6885392B1 (en) * 1999-12-31 2005-04-26 Stmicroelectronics, Inc. Perspective correction for preview area of panoramic digital camera
TW200529795A (en) * 2004-03-09 2005-09-16 Chuin-Mu Wang Method and system for examining fitness by photography
CN101119482A (en) * 2007-09-28 2008-02-06 北京智安邦科技有限公司 Overall view monitoring method and apparatus
CN201118859Y (en) * 2007-07-27 2008-09-17 浙江大学 Monocase real time panorama seamless no distortion video camera
US20090201361A1 (en) * 2008-02-08 2009-08-13 Google Inc. Panoramic Camera With Multiple Image Sensors Using Timed Shutters
TW201039246A (en) * 2009-04-27 2010-11-01 Asustek Comp Inc Method for continuously outputting character by video-recording

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4499319B2 (en) * 2001-08-24 2010-07-07 パナソニック株式会社 Driving support device, driving support method, and driving guide data creation method
US7782357B2 (en) * 2002-06-21 2010-08-24 Microsoft Corporation Minimizing dead zones in panoramic images
JP2004159014A (en) * 2002-11-05 2004-06-03 Nec Corp Digital camera attached mobile communication terminal
ES2633742T3 (en) 2007-03-16 2017-09-25 L-3 Communications Corporation System for panoramic image processing
JP4555399B2 (en) 2008-07-04 2010-09-29 清正工業株式会社 Medical waste treatment equipment
JP2010252015A (en) * 2009-04-15 2010-11-04 Panasonic Corp Image composition device, image composition method and program
WO2011037964A1 (en) 2009-09-22 2011-03-31 Tenebraex Corporation Systems and methods for correcting images in a multi-sensor system
JP2011119974A (en) * 2009-12-03 2011-06-16 Sony Corp Panoramic image synthesizer, panoramic image synthesis method, and program
FR2964757B1 (en) 2010-09-09 2013-04-05 Giroptic OPTICAL DEVICE FOR CAPTURING IMAGES ACCORDING TO A 360 ° FIELD
CN102480622A (en) * 2010-11-30 2012-05-30 比亚迪股份有限公司 Three-dimensional image acquisition method and system as well as mobile terminal
US9247133B2 (en) * 2011-06-01 2016-01-26 Apple Inc. Image registration using sliding registration windows
US20130321573A1 (en) * 2012-05-30 2013-12-05 Texas Instruments Incorporated Identification and display of time coincident views in video imaging

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6885392B1 (en) * 1999-12-31 2005-04-26 Stmicroelectronics, Inc. Perspective correction for preview area of panoramic digital camera
US20030135675A1 (en) * 2002-01-17 2003-07-17 Koninklijke Philips Electronics N.V. Configurable synchronous or asynchronous bus interface
US20040061787A1 (en) * 2002-09-30 2004-04-01 Zicheng Liu Foveated wide-angle imaging system and method for capturing and viewing wide-angle images in real time
TW200529795A (en) * 2004-03-09 2005-09-16 Chuin-Mu Wang Method and system for examining fitness by photography
CN201118859Y (en) * 2007-07-27 2008-09-17 浙江大学 Monocase real time panorama seamless no distortion video camera
CN101119482A (en) * 2007-09-28 2008-02-06 北京智安邦科技有限公司 Overall view monitoring method and apparatus
US20090201361A1 (en) * 2008-02-08 2009-08-13 Google Inc. Panoramic Camera With Multiple Image Sensors Using Timed Shutters
TW201039246A (en) * 2009-04-27 2010-11-01 Asustek Comp Inc Method for continuously outputting character by video-recording

Also Published As

Publication number Publication date
FR2998126B1 (en) 2014-12-26
FR3012001A1 (en) 2015-04-17
TW201435792A (en) 2014-09-16
CA2889811A1 (en) 2014-05-22
FR3011968B1 (en) 2016-05-06
HK1212835A1 (en) 2016-06-17
FR3011968A1 (en) 2015-04-17
CN104782114B (en) 2019-05-07
FR3012000A1 (en) 2015-04-17
TW201804432A (en) 2018-02-01
AU2013346603B2 (en) 2017-09-07
IN2015DN03812A (en) 2015-10-02
MX2015006121A (en) 2015-08-06
BR112015010788A8 (en) 2019-10-01
US20150288864A1 (en) 2015-10-08
IL238622A0 (en) 2015-06-30
MX355297B (en) 2018-04-12
WO2014076402A1 (en) 2014-05-22
JP2016503618A (en) 2016-02-04
FR2998126A1 (en) 2014-05-16
JP2019041389A (en) 2019-03-14
FR3012001B1 (en) 2016-05-06
AU2013346603A1 (en) 2015-05-14
BR112015010788A2 (en) 2017-07-11
FR3012000B1 (en) 2016-05-06
KR20150084807A (en) 2015-07-22
EP2920956A1 (en) 2015-09-23
CN104782114A (en) 2015-07-15

Similar Documents

Publication Publication Date Title
TWI612495B (en) Process and device for capturing and rendering a panoramic or stereoscopic stream of images
EP3906663B1 (en) High dynamic range processing based on angular rate measurements
TWI549519B (en) High resolution array camera
JP4661922B2 (en) Image processing apparatus, imaging apparatus, solid-state imaging device, image processing method, and program
JP5353393B2 (en) Image processing apparatus and image processing method
US9961272B2 (en) Image capturing apparatus and method of controlling the same
TW201618531A (en) Image capturing device and digital zooming method thereof
JP2010183316A (en) Image pickup apparatus
EP2720455B1 (en) Image pickup device imaging three-dimensional moving image and two-dimensional moving image, and image pickup apparatus mounting image pickup device
US11908111B2 (en) Image processing including noise reduction
US11563925B2 (en) Multiple tone control
TWI599809B (en) Lens module array, image sensing device and fusing method for digital zoomed images
KR20160137289A (en) Photographing apparatus and method for controlling the same
JP5820720B2 (en) Imaging device
JP5959194B2 (en) Imaging device
US9743007B2 (en) Lens module array, image sensing device and fusing method for digital zoomed images
CN109218602A (en) Image capture unit, image treatment method and electronic device
US20220021852A1 (en) Color fringing processing independent of tone mapping
JP6978720B2 (en) Image processing device
JP6425314B2 (en) Imaging method and imaging apparatus
Byun An efficient Color Edge Fuzzy interpolation Method for improving a Chromatic Aberration
JP2016213747A (en) Image processing apparatus, image processing method, and program

Legal Events

Date Code Title Description
MM4A Annulment or lapse of patent due to non-payment of fees