TW202406327A

TW202406327A - Omni-directional image processing method

Info

Publication number: TW202406327A
Application number: TW111126574A
Authority: TW
Inventors: 簡達益; 郭佳瑜
Original assignee: 達星影像股份有限公司
Priority date: 2022-07-15
Filing date: 2022-07-15
Publication date: 2024-02-01

Abstract

An algorithm stored in a non-transitory computer-readable medium is able to perform "image shift" correction of a wide-angle field of view camera element from a first image acquired from the wide-angle field of view camera element and a main vibration vector from the motion vector detection circuit. The algorithm includes the following steps: a distribution function operation, calculating a vector component of each position vector of the first image in the main vibration vector and determining the corresponding distribution value; and a displacement conversion process, calculating the corresponding correction displacement with the distribution value and the main vibration vector for converting the first image into a second image, wherein the first image and the second image both have the larger image taking magnification ratio at a central area thereof than that at an edge area.

Description

全景影像處理方法 Panoramic image processing method

本創作係關於一種全景影像處理方法；特別關於一種影像偏移的校正，特別在於它的處理速度。 This creation is about a panoramic image processing method; specifically about the correction of image offset, especially its processing speed.

攝像系統被大量應用於監視、檢查、安全和遙控感測，現今這些系統為了節省裝置空間與簡化影像處理與控制步驟，已開始使用具有大角度視野(FOV

120°)的攝像元件，例如：魚眼鏡頭(fish-eye lens)或反射式環狀鏡頭(panoramic annular lens)。僅需單顆鏡頭就能完成全景(omni-directional)攝影，不用移動部件，就能在半球形視場(hemispherical field-of-view)內提供平移、傾斜方向、旋轉和放大的影像。基於魚眼鏡頭收集光線的投影效果，在CCD或CMOS陣列上以圓形形分布的二維影像，可以使用高速電子電路進行數學校正，回歸至以投影角度表示的方形分布的二維影像。如「第1圖」所示，光線前進方向可以球座標(spherical coordinate)的餘緯角(colatitude)θ以及方位角(azimuth)Φ表示，這些光線經鏡片組101集光後，成像在距離f(有效焦距)處的成像平面，藉由二維影像感測器102光電轉換(photoelectrically conversion)為球形影像。 Camera systems are widely used in surveillance, inspection, security and remote sensing. Nowadays, in order to save installation space and simplify image processing and control steps, these systems have begun to use large-angle field of view (FOV).

120°) imaging element, such as a fish-eye lens or a panoramic annular lens. Only a single lens is needed to complete omni-directional photography, and without moving parts, it can provide pan, tilt, rotation and magnified images within a hemispherical field-of-view. Based on the projection effect of light collected by a fisheye lens, a two-dimensional image distributed in a circular shape on a CCD or CMOS array can be mathematically corrected using high-speed electronic circuits and returned to a two-dimensional image distributed in a square shape expressed by a projection angle. As shown in "Figure 1", the forward direction of light can be expressed by the colatitude θ and azimuth Φ of the spherical coordinate. After these light rays are collected by the lens group 101, they are imaged at a distance f The imaging plane at (effective focal length) is photoelectrically converted into a spherical image by the two-dimensional image sensor 102.

一般來說，魚眼鏡頭僅對光線入射角曲折，也就是入射光與魚眼鏡頭光軸100的夾角，以下以餘緯角θ表示。入射光從球座標(θ,Φ)入射後成像在二維影像感測器102，經由光電轉換的橫向讀取Hrd第i個以及縱向讀取Vrd第j個得到此像素(pixel)的光度，可以矩陣(i,j)來表示這些像素的位置；而以成像平面位置向量表示即(rcosΦ,rsinΦ)，其中，π為180度的弳度(radian)，且魚眼鏡頭光軸100穿過成像平面的原點O，其與(i,j)的距離以下稱為半徑r。 Generally speaking, a fisheye lens only bends with respect to the incident angle of light, which is the angle between the incident light and the optical axis 100 of the fisheye lens, which is represented by the co-latitude angle θ below. The incident light is imaged on the two-dimensional image sensor 102 after being incident from the spherical coordinates (θ, Φ), and the i-th horizontal reading Hrd and the longitudinal reading are carried out through photoelectric conversion. Take the jth Vrd to get the luminosity of this pixel (pixel). The positions of these pixels can be represented by the matrix (i, j); and the position vector of the imaging plane is expressed as (rcosΦ, rsinΦ), where π is 180 degrees. degrees (radian), and the optical axis 100 of the fisheye lens passes through the origin O of the imaging plane, and its distance from (i, j) is hereinafter referred to as the radius r.

因應不同的拍攝需求，魚眼鏡頭的鏡片組合也會有所不同，所述餘緯角θ與所述半徑r的關係大致可分為三種模型，其中，第I型為等距投影(equidistance projection)r=f．θ；第II型為平射投影(stereographic projection)r=f．2．tan(θ/2)；以及第III型為等立體角投影(equisolid angle projection)r=f．2．sin(θ/2)。在第I、II和III型投影關係的鏡頭前方放置一方格紙，則該二維影像感測器102上即分別呈現如「第2圖」由左至右的成像。與一般鏡頭視野(FOV)介於40°到60°不同，魚眼鏡頭視野可達180°，上圖方格紙寬度約在視野140°之處。傳統的小角度視野鏡頭稱透視投影(perspective projection)，成像為方形影像且其對角線長度約43mm；而符合第I、II和III型投影的鏡頭，成像為球形影像，其對角線長度僅25-28mm左右。 In response to different shooting needs, the lens combination of the fisheye lens will also be different. The relationship between the co-latitude angle θ and the radius r can be roughly divided into three models, among which type I is the equidistance projection. )r=f． θ; type II is stereographic projection r=f. 2. tan(θ/2); and type III is equisolid angle projection r=f. 2. sin(θ/2). When a square of graph paper is placed in front of the lens of the type I, II and III projection relationships, the two-dimensional image sensor 102 will respectively display images from left to right as shown in "Figure 2". Unlike ordinary lenses whose field of view (FOV) is between 40° and 60°, the field of view of a fisheye lens can reach 180°. The width of the graph paper in the picture above is approximately 140°. The traditional small-angle field of view lens is called perspective projection, and the image is a square image with a diagonal length of about 43mm; while the lens that conforms to Type I, II, and III projections is a spherical image with a diagonal length. Only about 25-28mm.

一般手機附加的魚眼鏡頭多為第I型(最左圖)，轉換回方形透視影像最為簡單，中心與邊緣亮度也不會差異過大，但在視野超過84°後，影像失真(distortion)就十分明顯；視訊會議或全景影像常用的魚眼鏡頭則為第II型(中圖)，透過組合高差異性鏡片或環狀反射鏡，保有視野中各物件的原始形狀(conformal mapping)，雖有邊緣過亮與轉換函數複雜等問題，但能確保影像的正確樣態；第III型(最右圖)則是能任意切換至各角度光軸的魚眼鏡頭，特別適合第一人稱的立體攝影，但由於影像失真嚴重並不適合轉換回方形的透視影像。 Most of the fisheye lenses attached to mobile phones are Type I (far left picture). It is easiest to convert back to a square perspective image, and the brightness between the center and the edge will not differ too much. However, when the field of view exceeds 84°, image distortion will occur. It is very obvious; the fisheye lens commonly used for video conferencing or panoramic imaging is type II (middle picture), which retains the original shape (conformal mapping) of each object in the field of view by combining high-difference lenses or ring reflectors. There are problems such as over-bright edges and complex transfer functions, but it can ensure the correct appearance of the image; Type III (far right picture) is a fisheye lens that can switch the optical axis at any angle, especially suitable for first-person stereoscopic photography , but due to severe image distortion, it is not suitable to convert back to a square perspective image.

上述模型與實際鏡片成像仍有些許差異，而且除了第I型之外，其他模型都有求數值解(numerical solution)的問題。在J.Kannala & S.S.Brandt揭示的多項式(polynomial)轉換法中，顯示至少需到九階才能將所述半徑r從小角度到大角度完整轉換r(θ)

k₁．θ+k₂．θ³+k₃．θ⁵+k₄．θ⁷+k₅．θ⁹。這五個轉換參數k₁,k₂,…,k₅，加上光軸偏移與球面偏差的兩個校正參數，一共有七個變數與實際結果作比對，才能找出(i,j)至所述餘緯角θ以及所述方位角Φ的轉換矩陣(matrix)。然而，七個變數構成的轉換矩陣，在實際應用時極難取得穩定數值解，最後多以三階近似法，也就是r(θ)

k₁．θ+k₂．θ³，在出廠前驗算完成並設定好轉換矩陣在IC中。 There are still some differences between the above-mentioned models and the actual lens imaging, and except for Type I, other models have problems with numerical solutions. In the polynomial conversion method disclosed by J.Kannala & SSBrandt, it is shown that it takes at least the ninth order to completely convert the radius r from a small angle to a large angle r(θ)

k ₁ ． θ+k ₂ ． θ ³ +k ₃ ． θ ⁵ +k ₄ ． θ ⁷ +k ₅ ． θ ⁹ . These five conversion parameters k ₁ , k ₂ ,..., k ₅ , plus the two correction parameters of optical axis offset and spherical deviation, there are a total of seven variables that can be compared with the actual results to find out (i, j ) to the co-latitude angle θ and the azimuth angle Φ. However, it is extremely difficult to obtain a stable numerical solution for the transformation matrix composed of seven variables in practical applications. In the end, the third-order approximation method is often used, that is, r(θ)

k ₁ ． θ+k ₂ ． θ ³ , the calculation is completed and the conversion matrix is set in the IC before leaving the factory.

由此可知，大角度視野攝像元件遇到手震或其他震動(shake)造成的「影像偏移」，在每個像素的偏移量與方向都不相同，例如：US9,800,780的第二圖到第四圖。透視投影的鏡頭，僅需幾個採樣點就可以知道整個「影像偏移」的方向與幅度；魚眼鏡頭的採樣範圍除了要廣要密外，還要應付被攝體非靜止狀態的「運動影像」，所導致的錯誤採樣。因此，US9,800,780負責影像擷取與處理的影像訊號處理器(Image Signal Processor，ISP)除有一塊專門計算被攝體運動向量的電路外，還有一塊用來統計這些運動向量的發散點與聚合點的電路。 It can be seen from this that when the large-angle field of view camera element encounters "image offset" caused by hand shake or other vibrations, the offset amount and direction of each pixel are different. For example: the second picture of US9,800,780 The fourth picture. For perspective projection lenses, it only takes a few sampling points to know the direction and magnitude of the entire "image offset"; in addition to having a wide and dense sampling range, the fisheye lens also has to cope with the non-stationary "movement" of the subject. Image", resulting in erroneous sampling. Therefore, the Image Signal Processor (ISP) in US9,800,780, which is responsible for image acquisition and processing, in addition to a circuit specifically for calculating subject motion vectors, also has a circuit for counting the divergence points and divergence points of these motion vectors. Assemblage point circuit.

魚眼鏡頭的振動(vibration)若不修正，引起的後果並非只有晃動的影像畫質而已。例如：WO98/27718A1利用外圍影像壓縮程度比中央高的特性，在不降低解析度(resolution)的情況下進行電子變焦；然而，隨著變焦比高，魚眼鏡頭的振動將導致圖像失真，引起解析度劣化。又例如：US5,185,667對通過魚眼鏡頭獲得的半球視場中的圖像切割，進行平移、傾斜、旋轉和放大；鏡頭振動產生的「影像偏移」將會使切割後的畫面忽暗忽明，乍暖(偏黃)還寒(偏藍)，甚至是出現扭曲線條。 If the vibration of the fisheye lens is not corrected, the consequences are not just shaky image quality. For example: WO98/27718A1 utilizes the characteristic that the peripheral image is compressed to a higher degree than the center to perform electronic zoom without reducing the resolution; however, as the zoom ratio is high, the vibration of the fisheye lens will cause image distortion. Causes resolution degradation. Another example: US5,185,667 cuts the image in the hemispheric field of view obtained through the fisheye lens and performs translation, tilt, rotation and magnification; the "image offset" caused by the vibration of the lens will make the cut image dim. It suddenly becomes brighter, warmer (yellowish), colder (bluer), or even distorted lines appear.

隨著電動車與車聯網的風潮興起，駕駛員輔助系統之一的環景監控(AVM)也獲得積極的研究，透過安裝在車聯網的多顆鏡頭直接向駕駛員提供車輛周圍環境，甚至可以分析出周圍的停車格、車道或行車異物並加以示警。以四顆鏡頭為例，如「第3圖」所示，車前鏡頭Lf會涵蓋部分左邊和部分右邊的影像，剛好補足車左鏡頭Lt和車右鏡頭Lr沒照到的路面；同理，車後鏡頭Lb也能補足車左鏡頭Lt和車右鏡頭Lr被車輪擋住的部分，只要對影像做些角度的修正，就能營造出在車輛正上方拍攝車輛與周圍環境的感覺。若是要進一步分析車道狀況甚至啟動人工智慧停車程序，這四顆鏡頭重疊的影像就需先行縫合拼貼程序。由於車體在引擎發動中或因路面凹凸會產生震動，且每顆鏡頭與車體的連動的關係不同，各自產生的「影像偏移」當然也就不同；在進行縫合拼貼程序時容易發生拼接不良，導致提供給駕駛員的鳥瞰圖(如第4圖)不時發生中斷，造成駕駛員的困擾。 With the rise of electric vehicles and the Internet of Vehicles, around-view monitoring (AVM), one of the driver assistance systems, has also received active research. It directly provides the driver with the surrounding environment of the vehicle through multiple lenses installed in the Internet of Vehicles, and can even Analyze surrounding parking spaces, lanes or foreign objects on the road and issue warnings. Taking four lenses as an example, as shown in "Picture 3", the front lens Lf will cover part of the left and part of the right image, just making up for the road surface not illuminated by the left lens Lt and the right lens Lr of the car; similarly, The rear lens Lb can also make up for the parts of the left lens Lt and the right lens Lr that are blocked by the wheels. As long as you make some angle corrections to the image, you can create the feeling of shooting the vehicle and the surrounding environment directly above the vehicle. If you want to further analyze the lane conditions or even start the artificial intelligence parking process, the overlapping images of these four lenses need to be stitched together first. Since the car body will vibrate when the engine is running or due to uneven road surfaces, and the linkage relationship between each lens and the car body is different, the "image offset" produced by each lens will of course be different; it is easy to occur during the stitching and collage process. Poor splicing causes the bird's-eye view provided to the driver (such as Figure 4) to be interrupted from time to time, causing trouble to the driver.

鏡頭振動也會影響影像訊號處理器(ISP)內3A(自動白平衡/自動曝光/自動對焦)的處理，尤其是自動白平衡(Auto White Balance，AWB)演算法。雖說各大品牌皆有其獨特的AWB演算法，但大抵不離統計像素裡RBG的色溫分布以調整RGB增益(gain)的邏輯。若鏡頭視野中有不同色溫被攝體，例如：停車場照明燈與各式車燈鏡頭色溫不同，ISP可以選擇多數像素的色溫，也能選擇偏向正中央被攝體的色溫。然而，鏡頭振動造成的「影像偏移」，有可能加速色溫統計的不穩定性，導致AWB演算法不只被反覆驗算拖慢ISP的整體速度，最後還可能呈現忽黃忽藍的影像。因此，像US9,105,105就提出了同時監測亮度直方圖(histogram)的方法，以選擇在亮度過低的情況下關閉AWB校正，避免色溫校正失真。 Lens vibration will also affect the 3A (auto white balance/auto exposure/auto focus) processing in the image signal processor (ISP), especially the Auto White Balance (AWB) algorithm. Although each major brand has its own unique AWB algorithm, it is generally based on the logic of counting the RBG color temperature distribution in pixels to adjust the RGB gain. If there are subjects with different color temperatures in the field of view of the lens, for example, parking lot lights and various car lights have different color temperatures, the ISP can select the color temperature of most pixels, or select a color temperature biased towards the central subject. However, the "image shift" caused by lens vibration may accelerate the instability of color temperature statistics, causing the AWB algorithm to be repeatedly checked and slowing down the overall speed of the ISP. In the end, the image may appear to flicker between yellow and blue. Therefore, US9,105,105 proposes a method of simultaneously monitoring the brightness histogram (histogram) to select the brightness Turn off AWB correction when it is too low to avoid color temperature correction distortion.

由上可知，鏡頭振動造成的影響不小，尤其對中心區域與邊緣區域放大比差異大的全景影像來說，很難用單一平移量來解決問題，例如：US7,511,756。用這種方式轉換回的方形影像，中心區域到邊緣區域變化太大，反而造成邊緣區域因校正而更加模糊。也因此，US7,834,907揭示了一種平移且略小的球形影像範圍，企圖藉由縮小尺寸，來減低中央區域與邊緣區域巨大的落差。 It can be seen from the above that the impact caused by lens vibration is not small. Especially for panoramic images with a large difference in magnification ratio between the central area and the edge area, it is difficult to solve the problem with a single translation amount, for example: US7,511,756. In the square image converted back in this way, the change from the center area to the edge area is too great, which causes the edge area to be blurred even more due to correction. Therefore, US7,834,907 reveals a translational and slightly smaller spherical image range, in an attempt to reduce the huge gap between the central area and the edge area by reducing the size.

本說明書中的「影像偏移」(image shift)係指被攝體在影像中的移動，乃受手持、風吹或機件連動等引起被攝體成像位置的變化；而「運動影像」(movement image)係指被攝體在影像中的移動，乃由被攝體相對於鏡頭的運動引起被攝體成像位置的變化。本說明書中的震動(shake)係指短時間內交替的作動(act)，例如：人體為維持平衡所作的來回移動、受強風吹撫的擺動以及人車行經路面所引發的抖動；而振動(vibration)係指週期性運動，其大致存在一平衡點，例如：因受突發外力導致鏡頭組件或與鏡頭連動的固定部件產生來回移動。 "Image shift" in this manual refers to the movement of the subject in the image, which is caused by changes in the imaging position of the subject caused by hand-holding, wind blowing or mechanical linkage; and "movement" (movement) image) refers to the movement of the subject in the image, which is the change in the imaging position of the subject caused by the movement of the subject relative to the lens. The vibration (shake) in this manual refers to the alternating action (act) in a short period of time, such as: the back and forth movement of the human body to maintain balance, the swing caused by strong wind, and the shaking caused by people and vehicles passing on the road; and vibration ( Vibration refers to periodic motion, which roughly has a balance point, for example: a lens assembly or a fixed part linked to the lens moves back and forth due to sudden external force.

一種全景影像處理方法，儲存於一非暫態電腦可讀取媒體(non-transitory computer-readable medium)，利用一攝像設備提供之主振動向量以及一第一影像，轉換一第二影像，其中，該第一影像與該第二影像的中心區域皆比邊緣區域具有更大的影像放大率。該方法包含以下步驟：一分布函數運算，計算該第一影像每個像素對該主振動向量的投影值，以決定該投影值對應的分布值；以及一位移轉換程序，獲得該第一影像，並根據該分布值以及該主振動向量之數值計算對應的校正位移以轉換該第二影像。該主振動向量可表示為一歐幾里得向量(euclidean vector)，落在所述攝像設備以光電轉換該第一影像之成像平面上，與所述攝像設備因振動造成的「影像偏移」有關，且所述攝像設備之光軸係通過所述成像平面的原點。該投影值與該分布值大致成一高斯分布(Gauss Distribution)，可以利用數學公式求得該分布值，也可以利用查找表(look-up table)輸入該投影值求得該分布值。 A panoramic image processing method, stored in a non-transitory computer-readable medium, uses a main vibration vector provided by a camera device and a first image to convert a second image, wherein, The central area of the first image and the second image both have greater image magnification than the edge area. The method includes the following steps: a distribution function operation to calculate the projection value of each pixel of the first image to the main vibration vector to determine the distribution value corresponding to the projection value; and a displacement conversion procedure to obtain the first image, and based on The distribution value and the value of the main vibration vector calculate the corresponding correction displacement to convert the second image. The main vibration vector can be expressed as a Euclidean vector, falling on the imaging plane where the camera device photoelectrically converts the first image, and the "image offset" caused by the vibration of the camera device Related, and the optical axis of the imaging device passes through the origin of the imaging plane. The projection value and the distribution value roughly form a Gaussian distribution (Gauss Distribution). The distribution value can be obtained by using mathematical formulas, or the distribution value can be obtained by inputting the projection value using a look-up table.

又，一種全景影像處理方法，儲存於一非暫態電腦可讀取媒體(non-transitory computer-readable medium)，利用一攝像設備提供之主振動向量以及一第一影像，轉換一第三影像，其中，該第一影像的中心區域比邊緣區域具有更大的影像放大率，而該第三影像的中心區域與邊緣區域的影像放大率大致相同。該方法包含以下步驟：一分布函數運算，計算該第一影像每個像素對該主振動向量的投影值，以決定該投影值對應的分布值；一位移轉換程序，獲得該第一影像，並根據該分布值以及該主振動向量之數值計算對應的校正位移以轉換一第二影像，且該第二影像的中心區域比邊緣區域具有更大的影像放大率；以及一投影轉換程序，依該第一影像轉換至透視投影影像(perspective projection image)之方式，將該第一影像或該第二影像轉換為該第三影像。該主振動向量可表示為一歐幾里得向量(euclidean vector)，落在所述攝像設備以光電轉換該第一影像之成像平面上，與所述攝像設備因振動造成的「影像偏移」有關，而所述攝像設備之光軸係通過所述成像平面之原點。該投影值與該分布值大致成一高斯分布(Gauss Distribution)，可以利用數學公式求得該分布值，也可以利用查找表(look-up table)輸入該投影值求得該分布值。 Also, a panoramic image processing method is stored in a non-transitory computer-readable medium, using a main vibration vector provided by a camera device and a first image to convert a third image, Wherein, the central area of the first image has a greater image magnification than the edge area, and the image magnification of the central area and the edge area of the third image is approximately the same. The method includes the following steps: a distribution function operation to calculate the projection value of each pixel of the first image to the main vibration vector to determine the distribution value corresponding to the projection value; a displacement conversion program to obtain the first image, and Calculate the corresponding correction displacement according to the distribution value and the value of the main vibration vector to convert a second image, and the central area of the second image has a greater image magnification than the edge area; and a projection conversion process, according to the The first image is converted into a perspective projection image by converting the first image or the second image into the third image. The main vibration vector can be expressed as a Euclidean vector, falling on the imaging plane where the camera device photoelectrically converts the first image, and the "image offset" caused by the vibration of the camera device Relevantly, the optical axis of the camera device passes through the origin of the imaging plane. The projection value and the distribution value roughly form a Gaussian distribution (Gauss Distribution). You can use mathematical formulas to obtain the distribution value, or you can use a look-up table. table) input the projection value to obtain the distribution value.

該投影轉換程序係根據該第一分量決定使用該第一影像或該第二影像轉換該第三影像。 The projection conversion program determines to use the first image or the second image to convert the third image based on the first component.

一種全景攝像設備，用以抓取視野內魚眼影像，其係包含：一全景影像擷取元件，利用光電轉換(photoelectrically conversion)該魚眼影像，其中，該魚眼影像的中心區域比邊緣區域具有更大的影像放大率；一振動檢測區塊，獲得關於所述全景攝像設備的振動資訊，並分析出一主振動向量；一位移運算區塊，從該主振動向量之方向計算每個像素的投影值，並根據該主振動向量之數值與該投影值決定每個像素的校正位移；以及一轉換區塊，得參考該等校正位移，將該魚眼影像轉換為一輸出影像。該主振動向量與所述全景攝像設備因振動造成的「影像偏移」有關，而該投影值與該校正位移大致呈現一高斯分布(Gauss Distribution)。 A panoramic camera device used to capture fish-eye images within a field of view, which includes: a panoramic image capture element that uses photoelectrically conversion to convert the fish-eye image, wherein the center area of the fish-eye image is larger than the edge area With greater image magnification; a vibration detection block to obtain vibration information about the panoramic camera equipment and analyze a main vibration vector; a displacement operation block to calculate each pixel from the direction of the main vibration vector The projection value, and determines the correction displacement of each pixel according to the value of the main vibration vector and the projection value; and a conversion block can convert the fisheye image into an output image with reference to the correction displacement. The main vibration vector is related to the "image offset" caused by vibration of the panoramic camera equipment, and the projection value and the correction displacement generally present a Gaussian distribution (Gauss Distribution).

Gb:車後地板影像 Gb: car rear floor image

Gf:車前地板影像 Gf: car front floor image

Gr:車右地板影像 Gr: Car right floor image

Gt:車左地板影像 Gt: car left floor image

Hrd:橫向讀取 Hrd: read horizontally

f:鏡片組有效焦距 f: effective focal length of lens group

i:橫向讀取序號 i: read serial number horizontally

j:縱向讀取序號 j: read serial number vertically

k₁:一階項次係數 k ₁ : first-order term coefficient

k₂:三階項次係數 k ₂ : third-order term coefficient

k₃:五階項次係數 k ₃ : fifth-order term coefficient

k₄:七階項次係數 k ₄ : seventh-order term coefficient

k₅:九階項次係數 k ₅ : ninth-order term coefficient

Lb:車後鏡頭 Lb: car rear shot

Lf:車前鏡頭 Lf: car front lens

Lr:車右鏡頭 Lr: Car right lens

Lt:車左鏡頭 Lt: Car left lens

O:原點 O: origin

r:半徑 r:radius

Vrd:縱向讀取 Vrd: vertical reading

β:折射角 β: refraction angle

Φ:方位角 Φ: azimuth angle

θ:餘緯角 θ: co-latitude angle

100:光軸 100:Optical axis

101:鏡片組 101: Lens set

102:二維影像感測器 102: Two-dimensional image sensor

103:全景攝像元件 103: Panoramic camera element

104:振動檢測區塊 104: Vibration detection block

105:轉換區塊 105:Conversion block

106:位移運算區塊 106: Displacement operation block

107:記憶體控制介面 107:Memory control interface

108:第一記憶區 108: First memory area

109:第二記憶區 109: Second memory area

110:視訊輸出區塊 110: Video output block

62:投影值 62: Projection value

63:分布值 63: Distribution value

69:高斯分布函數 69:Gaussian distribution function

81:往右偏1.6°「影像偏移」的第一影像 81: The first image with an "image shift" of 1.6° to the right

82:往右偏16°「影像偏移」的第一影像 82: The first image of "Image Shift" 16° to the right

91:往右偏1.6°「影像偏移」的第二影像 91: The second image shifted to the right by 1.6° "image offset"

92:往右偏16°「影像偏移」的第二影像 92: The second image with an "image shift" of 16° to the right

第1圖係攝像設備成像平面與鏡片組之立體示意圖 Figure 1 is a three-dimensional schematic diagram of the imaging plane and lens group of the camera equipment

第2圖係等距投影、平射投影與立體角投影之球形影像示意圖 Figure 2 is a schematic diagram of a spherical image using equidistant projection, planar projection and solid angle projection.

第3圖係本創作應用於AVM系統配置圖 Figure 3 is the configuration diagram of the AVM system used in this creation.

第4圖係本創作應用於車輛之鳥瞰圖 Picture 4 is a bird’s eye view of this creation applied to vehicles

第5圖係習知採用等校正位移在橫向視野80°內之結果 Figure 5 is the result of conventionally using equal correction displacement within a lateral field of view of 80°.

第6圖係本創作往右偏「影像偏移」的應校正程度分布 Figure 6 shows the distribution of the degree of correction that should be corrected for the "image offset" of this creation to the right.

第7圖係本創作查找表使用示範 Figure 7 is a demonstration of the use of this creative lookup table.

第8圖係本創作往右偏1.6°「影像偏移」的第一影像與第二影像 Picture 8 is the first and second images of this creation with an "image shift" of 1.6° to the right.

第9圖係本創作往右偏16°「影像偏移」的第一影像與第二影像 Picture 9 is the first and second images of this creation with an "image shift" of 16° to the right.

第10圖係本創作往右偏「影像偏移」的第三影像 Picture 10 is the third image of this creation that is "image shifted" to the right.

第11圖係本創作之全景攝像設備電路區塊示意圖 Figure 11 is a schematic diagram of the circuit block of the panoramic camera equipment of this creation

本創作為一種校正大角度視野(FOV

120°)攝像元件「影像偏移」的演算法，所述攝像元件係由鏡片組101與二維影像感測器102組合，再由一影像訊號處理器(ISP)轉換該二維影像感測器102上的類比訊號為數位訊號。本創作得以儲存於非暫態電腦可讀取媒體中，並從ISP取得數位訊號，以及從運動向量檢測電路取得所述攝像元件的振動資訊，以計算「影像偏移」的校正位移。所述攝像元件的ISP以及所述運動向量檢測電路常整合成一塊電路板或單晶片，並與所述攝像元件組合成一攝像設備。本創作一樣可整合至所述電路板或單晶片上，或是建置在與所述攝像設備連接之電腦或手機的應用程式記憶體裡。 This product is a corrected wide-angle field of view (FOV)

120°) algorithm for "image offset" of the imaging element. The imaging element is composed of a lens group 101 and a two-dimensional image sensor 102, and then an image signal processor (ISP) converts the two-dimensional image sensing The analog signal on the device 102 is a digital signal. This creation can be stored in a non-transitory computer-readable medium, obtain digital signals from the ISP, and obtain the vibration information of the imaging element from the motion vector detection circuit to calculate the correction displacement of the "image offset". The ISP of the imaging element and the motion vector detection circuit are often integrated into a circuit board or a single chip, and combined with the imaging element to form an imaging device. The invention can also be integrated into the circuit board or single chip, or built into the application memory of a computer or mobile phone connected to the camera device.

請參照「第5圖」，習知採用等校正位移在橫向視野80°內之校正前(左)與校正後(右)之結果(實線)。上方兩圖為約往右偏1.6°之「影像偏移」結果；下方兩圖為約往右側偏16°之「影像偏移」結果；以及圖中虛線係用以標示無振動之結果。很明顯地，習知等校正位移的電子防手震方式，在FOV60°處(點線框處)已達到校正的極限。就如先前技術所提，大角度攝像元件所擷取的影像在中心區域與邊緣區域放大率差異甚大，故「影像偏移」會在中心區域被放大，而在邊緣區域被縮小，整體而言中心區域的影像尺寸會變小。這也是為什麼US7,834,907去計算中心區域縮小後的尺寸，然後再放大回前一張的尺寸。 Please refer to "Figure 5", the conventionally used equal correction displacement within 80° of the lateral field of view is the results before correction (left) and after correction (right) (solid line). The upper two pictures show the "image shift" results of about 1.6° to the right; the lower two pictures show the "image shift" results of about 16° to the right; and the dotted lines in the pictures are used to mark the results without vibration. Obviously, the conventional electronic anti-shake method that corrects displacement has reached the correction limit at FOV 60° (dotted line frame). As mentioned in the previous technology, the magnification ratio of the image captured by the large-angle camera element is very different between the center area and the edge area, so the "image offset" will be enlarged in the center area and reduced in the edge area. Overall The image size in the center area will become smaller. This is why US7,834,907 calculates the reduced size of the central area, and then enlarges it back to the previous size.

請參照「第6圖」，本創作往右偏「影像偏移」的應校正程度分布，所在像素愈往右側或愈往左側所需校正程度愈小，反之，愈往中間則所需校正程度愈須往左移(負值)。上方圖為約往右偏1.6°之「影像偏移」的應校正程度分布；下方圖為約往右偏16°之「影像偏移」的應校正程度分布，其應校正程度之尺度已分別用1.6°與16°歸一。本創作揭示實際應用時，「影像偏移」並非如US7,834,907所設想的線性關係，而是大致成高斯分布(Gauss Distribution)，而且以該主振動方向之數值歸一後，各自差異不大。 Please refer to "Picture 6". In this creation, the distribution of the degree of correction required for "image offset" skewed to the right. The farther the pixel is to the right or the left, the smaller the degree of correction required. On the contrary, the farther to the center, the smaller the degree of correction required. The more you have to move to the left (negative value). The upper picture shows the distribution of the degree of correction for an "image shift" of about 1.6° to the right; the lower picture shows the distribution of the degree of correction for an "image shift" of about 16° to the right. The scales of the degree of correction have been separated. Use 1.6° and 16° to normalize. This creation reveals that when used in practical applications, the "image offset" is not a linear relationship as envisioned in US7,834,907, but is roughly a Gaussian distribution (Gauss Distribution), and after normalizing by the value of the main vibration direction, there is not much difference between them. .

實施時，本創作儲存於非暫態電腦可讀取媒體中，並從所述攝像元件獲得第一影像與主振動向量，其係包含以下步驟：一分布函數運算，計算該第一影像每個像素對該主振動向量的投影值，以決定該投影值對應的分布值；以及一位移轉換程序，根據該分布值以及該主振動向量之數值計算對應的校正位移以轉換該第二影像。該主振動向量之方向可表示成在該第一影像上的兩個分量，與該第一影像位置向量(rcosΦ,rsinΦ)做內積便能求出該投影值62；接著使用已設定好的查找表(look-up table)，請參照「第7圖」，找出該投影值62對應的該分布值63。實際應用的查找表(LUT)為電晶體邏輯電路，可視為一種記憶元件，只要事先燒錄對應的數值就能快速取用複雜計算後的數值。或是直接將該投影值62帶入高斯分布函數69，例如：先計算該投影值62除以一設定參數後的平方再將其帶入指數函數表(Exponential Function table)，求得該分布值63。「第6圖」亦顯示該校正位移與該主振動向量之數值大致成正比。 During implementation, the invention is stored in a non-transitory computer-readable medium, and obtains the first image and the main vibration vector from the camera element, which includes the following steps: a distribution function operation to calculate each of the first images The projection value of the pixel to the main vibration vector is used to determine the distribution value corresponding to the projection value; and a displacement conversion program calculates the corresponding correction displacement according to the distribution value and the value of the main vibration vector to convert the second image. The direction of the main vibration vector can be expressed as two components on the first image. By doing the inner product with the first image position vector (rcosΦ, rsinΦ), the projection value 62 can be obtained; then use the set For the look-up table, please refer to "Figure 7" to find the distribution value 63 corresponding to the projection value 62. The look-up table (LUT) used in practical applications is a transistor logic circuit, which can be regarded as a memory element. As long as the corresponding value is burned in advance, the value after complex calculation can be quickly accessed. Or directly bring the projection value 62 into the Gaussian distribution function 69. For example, first calculate the square of the projection value 62 divided by a set parameter and then bring it into the exponential function table (Exponential Function table) to obtain the distribution value. 63. "Figure 6" also shows that the correction displacement is roughly proportional to the value of the main vibration vector.

該主振動向量之數值係與所述攝像設備因振動造成的「影像偏移」有關，所述攝像設備多具有運動向量檢測電路，例如：三軸加速度器與其分析電路、螺旋加速度器與其分析電路以及複眼檢測器(請參閱US7,511,756之FIG.10)。前兩者因為直接量測所述攝像設備的加速度訊號，特別適合低光度時使用，但需要經過座標轉算；後者雖受光度影響成效，但卻是最直觀的結果，能直接分析出該主振動向量。前兩者的換算主要係包含：從量測訊號得出往左移動一單位長度，再除以所述被攝體至所述攝像設備距離或其反正切函數解，得出原來光軸正前方的被攝體往右偏移的角度；再將該往右偏移的角度帶入所述攝像設備投影關係(第1、2圖)得出該半徑r，即該主振動向量的數值；而該主振動向量的方向可以直接由三軸加速度器等量測訊號得出。該主振動向量只與光軸正前方「影像偏移」程度有關，不需要再計算其他角度求平均結果。 The value of the main vibration vector is related to the "image offset" caused by vibration of the camera equipment. Most of the camera equipment has a motion vector detection circuit, such as three-axis acceleration. accelerator and its analysis circuit, a helical accelerator and its analysis circuit, and a compound eye detector (please refer to FIG.10 of US7,511,756). The former two directly measure the acceleration signal of the camera equipment, so they are particularly suitable for use in low light conditions, but require coordinate conversion; although the latter is affected by light, it is the most intuitive result and can directly analyze the subject. Vibration vector. The conversion of the first two mainly includes: moving one unit length to the left from the measurement signal, and then dividing it by the distance from the subject to the camera device or its arctangent function solution to obtain the original optical axis directly in front The angle at which the subject shifts to the right; and then the angle of the right shift is brought into the projection relationship of the camera equipment (Figures 1 and 2) to obtain the radius r, which is the value of the main vibration vector; and The direction of the main vibration vector can be directly obtained from measurement signals such as a three-axis accelerometer. The main vibration vector is only related to the degree of "image offset" directly in front of the optical axis, and there is no need to calculate the average result from other angles.

請參照「第8圖」，本創作往右偏1.6°「影像偏移」的第一影像81與第二影像91。在該位移轉換程序中，會根據該第一影像每個像素對應的該校正位移，轉換該第二影像。在低「影像偏移」幅度裡，本創作不只可以校正FOV超過60°的鏡頭，甚至應用在FOV140°的鏡頭也沒甚麼問題。 Please refer to "Picture 8", the first image 81 and the second image 91 of this creation are "image shifted" 1.6° to the right. In the displacement conversion process, the second image is converted according to the corrected displacement corresponding to each pixel of the first image. At a low "image shift" range, this creation can not only correct lenses with an FOV of more than 60°, but can even be applied to lenses with an FOV of 140° without any problem.

請參照「第9圖」，本創作往右偏16°「影像偏移」的第一影像82與第二影像92。在該位移轉換程序中，會根據該第一影像每個像素對應的該校正位移，轉換該第二影像。由於16°「影像偏移」已經達到劇烈的振動幅度猶如在奔跑的過程中，影像失真的程度相當大，但本創作僅用單軸參數來擬合，仍可適用在縱向FOV60°及橫向FOV140°的鏡頭，也就是沿該主振動向量的校正擬合程度相當高。 Please refer to "Picture 9". This creation shows the first image 82 and the second image 92 with an "image offset" of 16° to the right. In the displacement conversion process, the second image is converted according to the corrected displacement corresponding to each pixel of the first image. Since the 16° "image offset" has reached a severe vibration amplitude, just like running, the degree of image distortion is quite large. However, this creation only uses single-axis parameters to fit, and it can still be applied to longitudinal FOV60° and horizontal FOV140 ° lens, that is, the degree of correction along the main vibration vector is quite high.

本創作另一實施例，更包含：一投影轉換程序，依該第一影像轉換至透視投影影像(perspective projection image)之方式，將該第二影像轉換為該第三影像。請參照「第10圖」，該第三影像的中心區域與邊緣區域的影像放大率大致相同，左上圖為往右偏1.6°「影像偏移」的該第三影像，左下圖為往右偏16°「影像偏移」的該第三影像，虛線為無振動的影像(只取FOV約82°)。顯示本創作「影像偏移」校正即使是面對激烈的振動，也能保持主視野的畫面穩定度，即使是轉換回透視影像上。面對常見小振動，例如：手震、風吹抖動或是車體振動，這類擺幅在數度以內的狀況，本創作仍然可以維持大角度的畫面穩定度。如右圖所示，往右偏1.6°「影像偏移」的該第三影像(FOV140°)，與無振動的影像(虛線)幾乎重疊。 Another embodiment of the invention further includes: a projection conversion program that converts the second image into a perspective projection image in a manner that converts the first image into a perspective projection image. Replace it with the third image. Please refer to "Picture 10". The image magnifications of the central area and the edge area of the third image are roughly the same. The upper left picture shows the third image with an "image shift" of 1.6° to the right, and the lower left picture shows the third image with an "image shift" of 1.6° to the right. For this third image with 16° "image offset", the dotted line is the image without vibration (only the FOV is about 82°). It shows that the "image offset" correction of this creation can maintain the stability of the main field of view even in the face of intense vibration, even when it is converted back to a perspective image. In the face of common small vibrations, such as hand shake, wind shake or car body vibration, this kind of swing can still maintain the stability of the picture at large angles. As shown in the picture on the right, the third image (FOV140°) with an "image shift" of 1.6° to the right almost overlaps with the image without vibration (dashed line).

請參照「第11圖」，為本創作之全景攝像設備電路區塊示意圖。所述全景攝像設備透過一全景影像擷取元件103，將該鏡片組101收集的光線成像在該二維影像感測器102上，藉由影像訊號處理器光電轉換(photoelectrically conversion)的類比訊號為數位訊號(A/D)、自動增益校正(AGC)、自動白平衡(AWB)以及RGB色光切換等，處理後的數位訊號即前方視野內的魚眼影像，藉由一記憶體控制介面107儲存於第一記憶區108。該魚眼影像的中心區域比邊緣區域具有更大的影像放大率，無法以光學或非光學的傳統影像穩定電路(image stabilization)處理，但仍可以組合習知的運動檢測電路以及一影像偏移分析電路成一振動檢測區塊104，計算出一主振動向量供一位移運算區塊106決定該魚眼影像的校正位移。其係利用一內積電路計算每個像素對該主振動向量的投影值，透過一高斯查找表找出該校正位移。一轉換區塊105，得參考該校正位移，將該魚眼影像轉換為一輸出影像儲存於一第二記憶區109。最後由一視訊輸出區塊110，根據該第一記憶區108與該第二記憶區109產生穩定的視訊源。 Please refer to "Figure 11" for a schematic diagram of the circuit block of the panoramic camera equipment of this creation. The panoramic camera device images the light collected by the lens group 101 on the two-dimensional image sensor 102 through a panoramic image capturing element 103. The analog signal through photoelectrically conversion by the image signal processor is: Digital signal (A/D), automatic gain correction (AGC), automatic white balance (AWB), RGB color light switching, etc. The processed digital signal, which is the fisheye image in the front field of view, is stored through a memory control interface 107 in the first memory area 108. The central area of the fisheye image has a greater image magnification than the edge area and cannot be processed by optical or non-optical traditional image stabilization circuits (image stabilization). However, it can still be combined with a conventional motion detection circuit and an image offset. The analysis circuit forms a vibration detection block 104 and calculates a main vibration vector for a displacement operation block 106 to determine the correction displacement of the fisheye image. It uses an inner product circuit to calculate the projection value of each pixel to the main vibration vector, and finds the correction displacement through a Gaussian lookup table. A conversion block 105 may refer to the correction displacement to convert the fisheye image into an output image and store it in a second memory area 109. Finally, a video output block 110 generates a stable video source according to the first memory area 108 and the second memory area 109 .

該輸出影像，係該轉換區塊105根據該校正位移轉換該魚眼影像的像素位置而來，其中心區域比邊緣區域具有更大的影像放大率，僅針對「影像偏移」作校正；也可以依該魚眼影像轉換至透視投影影像(perspective projection image)之方式，在該轉換區塊105內事先燒入無失真查找表，再據此轉換該魚眼影像的像素位置；更可以將該無失真查找表再加上該校正位移的疊加電路，以此轉換該魚眼影像的像素位置。 The output image is obtained by the conversion block 105 converting the pixel positions of the fisheye image according to the correction displacement. The central area has a greater image magnification than the edge area, and is only corrected for "image offset"; also According to the method of converting the fisheye image into a perspective projection image, a distortion-free lookup table can be burned in the conversion block 105 in advance, and then the pixel position of the fisheye image can be converted accordingly; the pixel position of the fisheye image can also be converted into a perspective projection image. The distortion-free lookup table is coupled with the correction displacement superposition circuit to convert the pixel position of the fisheye image.

在其一實施例中，該主振動向量係以該二維影像感測器102座標的分量表示，且該影像偏移分析電路可以設定一閥值以輸出數值為0之該主振動向量。該視訊輸出區塊110可使用多工器決定在該主振動向量為0時，輸出該第一記憶區108的該魚眼影像或是該第二記憶區109的該輸出影像。同理，該位移運算區塊106也可使用多工器決定在該主振動向量為0時，不執行該內積電路以及查找的動作。本創作更可以再包含：一中央處理器(CPU)，控制該振動檢測區塊104、該位移運算區塊106以及該轉換區塊105的作動以及匯流排控制，例如：接收使用者切換該位移運算區塊106與該轉換區塊105透視投影的作動，以及根據系統統計值設定該振動檢測區塊104的閥值等。 In one embodiment, the main vibration vector is represented by a component of the coordinates of the two-dimensional image sensor 102, and the image offset analysis circuit can set a threshold to output the main vibration vector with a value of 0. The video output block 110 may use a multiplexer to determine whether to output the fisheye image of the first memory area 108 or the output image of the second memory area 109 when the main vibration vector is 0. Similarly, the displacement operation block 106 may also use a multiplexer to decide not to execute the inner product circuit and search operations when the main vibration vector is 0. The invention may further include: a central processing unit (CPU) to control the operation and bus control of the vibration detection block 104, the displacement calculation block 106 and the conversion block 105, for example: receiving the user to switch the displacement. The operations of the perspective projection of the computing block 106 and the conversion block 105, and setting the threshold of the vibration detection block 104 according to the system statistics, etc.

除此之外，該全景影像擷取元件103也能根據該振動檢測區塊104是否檢測出「影像偏移」或「運動影像」來調整或開關自動曝光、自動白平衡或自動增益校正等參數；也能根據是否執行該位移運算區塊106以及該轉換區塊105來決定是否回復前張影像的參數。 In addition, the panoramic image capturing element 103 can also adjust or switch parameters such as automatic exposure, automatic white balance or automatic gain correction according to whether the vibration detection block 104 detects "image offset" or "moving image". ; It can also be determined whether to restore the parameters of the previous image based on whether the displacement operation block 106 and the conversion block 105 are executed.

請參照「第3圖」與「第4圖」，本創作應用於AVM系統配置圖與鳥瞰圖，由於車體結構的關係，大部分是用車前、車後、車右以及車左鏡頭(Lf、Lb、Lr、Lt)來完成。為了盡可能拍攝到全景畫面多使用大角度鏡頭，經該轉換區塊105轉換成透視投影的車前、車後、車右及車左地板影像(Gf、Gb、Gr、Gt)後，在AVM系統的處理器進行拼貼。因車身抖動的關係，不時會受到其他車燈或是鏡子反射對AWB的或大或小的干擾，若不處理「影像偏移」，畫面便會而偏黃時而偏藍。應用本創作之AVM系統，能自動校正車身抖動的「影像偏移」，並在確定該校正位移確實作用在該轉換區塊105時，關閉該魚眼影像的AWB或是使用固定的白平衡參數，提供清晰穩定的該輸出影像。 Please refer to "Picture 3" and "Picture 4". This creation is applied to the AVM system configuration diagram and bird's-eye view. Due to the structure of the car body, most of them use the front, rear, right and right sides of the car. Car left lens (Lf, Lb, Lr, Lt) to complete. In order to capture a panoramic picture as much as possible, large-angle lenses are often used. After the conversion block 105 converts the perspective projection of the front, rear, right and left floor images (Gf, Gb, Gr, Gt) of the car, the AVM The system's processor performs the tiling. Due to the vibration of the car body, the AWB will be interfered with by other car lights or mirror reflections from time to time. If the "image offset" is not processed, the picture will become yellowish and sometimes bluish. Applying the AVM system of this invention, the "image offset" of vehicle body shake can be automatically corrected, and when it is determined that the correction displacement is indeed applied to the conversion block 105, the AWB of the fisheye image can be turned off or a fixed white balance parameter can be used. , providing clear and stable output images.

魚眼鏡頭的集光範圍太廣，無法使用傳統的光學防手震方式，其係根據振動量測的訊號來控制該鏡片組101其一鏡片的位置。魚眼鏡頭的光學防手震方式，不能僅調整其一鏡片的位置，必需整個該鏡片組101一起移動，如第11圖該全景攝像元件103內所示之虛線。然而，要移動整個該鏡片組101的馬達是十分佔空間的，而且能調整的範圍也有限。使用本創作之全景攝像設備或方法，可以節省所述馬達的空間使該全景攝像元件完全貼合薄型外殼；與光學防手震一起搭配時，更能補充光學防手震無法處理的大動作振動校正。 The light collection range of the fisheye lens is too wide to use the traditional optical anti-shake method, which controls the position of one lens of the lens group 101 based on the vibration measurement signal. The optical anti-shake method of the fisheye lens cannot only adjust the position of one lens, but the entire lens group 101 must be moved together, as shown by the dotted line in the panoramic camera element 103 in Figure 11. However, the motor required to move the entire lens assembly 101 takes up a lot of space, and its adjustable range is also limited. Using the panoramic camera device or method of this invention, the space of the motor can be saved so that the panoramic camera element can completely fit into the thin housing; when paired with the optical anti-shake, it can supplement the large-movement vibration that the optical anti-shake cannot handle. Correction.

本創作提供一種快速且節省查找表空間之「影像偏移」校正，該高斯查找表輸入輸出端數目即球形影像的寬度(單位：像素)，也就是兩千至四千左右，不論該主振動向量的大小為何皆可適用。由於使用的LUT空間夠小，該內積電路又簡單，使用現場可編成陣列(FPGA)即可完成該位移運算區塊106；即便是更換該全景攝像元件的投影方式，例如：從第I型換至第II型，也能簡單更換FPGA裡的可變動參數後繼續使用。本說明書雖以水平方向的振動舉例，但任何平行於成像平面上的振動或是光軸傾向造成的在成像平面的振動，亦不脫本創作之範疇。 This creation provides a fast and space-saving lookup table "image offset" correction. The number of input and output terminals of the Gaussian lookup table is the width of the spherical image (unit: pixels), which is about two thousand to four thousand, regardless of the main vibration. Any vector size is applicable. Since the LUT space used is small enough and the inner product circuit is simple, the displacement calculation block 106 can be completed using a field programmable array (FPGA); even if the projection method of the panoramic camera element is changed, for example: from type I If you switch to Type II, you can simply replace the variable parameters in the FPGA and continue to use it. Although this manual is based on horizontal Directional vibration is an example, but any vibration parallel to the imaging plane or vibration in the imaging plane caused by the inclination of the optical axis does not deviate from the scope of this creation.

綜上所述，本創作之全景影像處理方法，確已符合專利申請之要件，爰依法提出專利申請。惟以上所述者，僅為本創作之較佳實施例，當不能以此限定本創作實施之範圍；故，凡依本創作申請專利範圍及說明書內容所作之簡單的等效變化與修飾，皆應仍屬本創作專利涵蓋之範圍內。 To sum up, the panoramic image processing method of this creation has indeed met the requirements for patent application, and the patent application can be filed in accordance with the law. However, the above are only preferred embodiments of this invention and should not be used to limit the scope of implementation of this invention; therefore, any simple equivalent changes and modifications made based on the patent scope of this invention and the content of the specification are It should still be within the scope of this creative patent.

101:鏡片組 101: Lens set

102:二維影像感測器 102: Two-dimensional image sensor

103:全景攝像元件 103: Panoramic camera element

104:振動檢測區塊 104: Vibration detection block

105:轉換區塊 105:Conversion block

106:位移運算區塊 106: Displacement operation block

107:記憶體控制介面 107:Memory control interface

108:第一記憶區 108: First memory area

109:第二記憶區 109: Second memory area

110:視訊輸出區塊 110: Video output block

Claims

一種全景影像處理方法，儲存於一非暫態電腦可讀取媒體(non-transitory computer-readable medium)，利用一攝像設備提供之主振動向量以及一第一影像轉換一第二影像，其中，該第一影像與該第二影像的中心區域皆比邊緣區域具有更大的影像放大率，其係包含以下步驟： A panoramic image processing method, stored in a non-transitory computer-readable medium, using a main vibration vector provided by a camera device and a first image to convert a second image, wherein the Both the central area of the first image and the second image have greater image magnification than the edge area, which includes the following steps:

一分布函數運算，計算該第一影像每個像素對該主振動向量的投影值，以決定該投影值對應的分布值，且該投影值與該分布值大致成一高斯分布(Gauss Distribution)；以及 A distribution function operation calculates the projection value of each pixel of the first image to the main vibration vector to determine the distribution value corresponding to the projection value, and the projection value and the distribution value approximately form a Gaussian distribution (Gauss Distribution); and

一位移轉換程序，獲得該第一影像，並根據該分布值以及該主振動向量之數值計算對應的校正位移，以轉換該第二影像，其中，該主振動向量可表示為一歐幾里得向量(euclidean vector)，落在所述攝像設備以光電轉換該第一影像之成像平面上，與所述攝像設備因振動造成的「影像偏移」有關，且所述攝像設備之光軸係通過所述成像平面之原點。 A displacement conversion program obtains the first image, and calculates the corresponding corrected displacement according to the distribution value and the value of the main vibration vector to convert the second image, where the main vibration vector can be expressed as a Euclidean The vector (euclidean vector) falls on the imaging plane where the camera device photoelectrically converts the first image, and is related to the "image offset" caused by vibration of the camera device, and the optical axis of the camera device passes through The origin of the imaging plane.

如請求項1所述之分布函數運算中，所述計算之意義，係根據該第一影像每個像素位置之於所述成像平面之原點與該主振動向量之單位方向的內積；以及所述決定之意義，係指利用一查找表(look-up table)根據該投影值求得該分布值。 In the distribution function operation described in claim 1, the meaning of the calculation is based on the inner product of the origin of each pixel position of the first image on the imaging plane and the unit direction of the main vibration vector; and The meaning of the determination refers to using a look-up table to obtain the distribution value based on the projection value.

一種全景影像處理方法，儲存於一非暫態電腦可讀取媒體(non-transitory computer-readable medium)，利用一攝像設備提供之主振動向量以及一第一影像轉換一第三影像，其中，該第一影像的中心區域比邊緣區域具有更大的影像放大率，其係包含以下步驟： A panoramic image processing method, stored in a non-transitory computer-readable medium, using a main vibration vector provided by a camera device and a first image to convert a third image, wherein the The central area of the first image has greater image magnification than the edge area, which includes the following steps:

一分布函數運算，計算該第一影像每個像素對該主振動向量的投影值，以決定該投影值對應的分布值，且該投影值與該分布值大致成一高斯分布(Gauss Distribution)； A distribution function operation to calculate the projection value of each pixel of the first image to the main vibration vector to determine the distribution value corresponding to the projection value, and the projection value and the distribution value approximately form a Gaussian distribution (Gauss Distribution);

一位移轉換程序，獲得該第一影像，並根據該分布值以及該主振動向量之數值計算對應的校正位移，以轉換該第二影像，且該第二影像的中心區域比邊緣區域具有更大的影像放大率；以及 A displacement conversion program obtains the first image, and calculates the corresponding correction displacement according to the distribution value and the value of the main vibration vector to convert the second image, and the central area of the second image has a larger value than the edge area. image magnification; and

一投影轉換程序，依該第一影像轉換至透視投影影像(perspective projection image)之方式，將該第一影像或該第二影像轉換為該第三影像，且該第三影像的中心區域與邊緣區域的影像放大率大致相同，其中，該主振動向量可表示為一歐幾里得向量(euclidean vector)，落在所述攝像設備以光電轉換該第一影像之成像平面上，與所述攝像設備因振動造成的「影像偏移」有關，且所述攝像設備之光軸係通過所述成像平面之原點。 A projection conversion program that converts the first image or the second image into the third image in a manner that converts the first image into a perspective projection image, and the center area and edge of the third image The image magnification of the area is approximately the same, where the main vibration vector can be expressed as a Euclidean vector, falling on the imaging plane where the camera device photoelectrically converts the first image, and the same as the camera It is related to the "image offset" caused by vibration of the equipment, and the optical axis of the imaging equipment passes through the origin of the imaging plane.

如請求項3所述之分布函數運算中，所述計算之意義，係根據該第一影像每個像素位置之於所述成像平面之原點與該主振動向量之單位方向的內積；以及所述決定之意義，係指利用一查找表(look-up table)根據該投影值求得該分布值。 In the distribution function operation described in claim 3, the meaning of the calculation is based on the inner product of the origin of each pixel position of the first image on the imaging plane and the unit direction of the main vibration vector; and The meaning of the determination refers to using a look-up table to obtain the distribution value based on the projection value.

一種全景攝像設備，用以抓取視野內魚眼影像，其係包含： A panoramic camera device used to capture fisheye images within the field of view, which includes:

一全景影像擷取元件，利用光電轉換(photoelectrically conversion)該魚眼影像，其中，該魚眼影像的中心區域比邊緣區域具有更大的影像放大率； A panoramic image capturing element uses photoelectrically conversion to convert the fisheye image, wherein the central area of the fisheye image has a greater image magnification than the edge area;

一振動檢測區塊，獲得關於所述全景攝像設備的振動資訊，並分析出一主振動向量； A vibration detection block to obtain vibration information about the panoramic camera equipment and analyze it Get a main vibration vector;

一位移運算區塊，從該主振動向量之方向計算每個像素的投影值，並根據該主振動向量之數值與該投影值決定每個像素的校正位移；以及 A displacement operation block calculates the projection value of each pixel from the direction of the main vibration vector, and determines the corrected displacement of each pixel based on the value of the main vibration vector and the projection value; and

一轉換區塊，根據該等校正位移將該魚眼影像轉換為一輸出影像，其中，該主振動向量可表示為一歐幾里得向量(euclidean vector)，落在所述全景影像擷取元件以光電轉換該魚眼影像之成像平面上，與所述全景攝像設備因振動造成的「影像偏移」有關，且該全景影像擷取元件之光軸係通過所述成像平面之原點。 A conversion block converts the fisheye image into an output image according to the corrected displacements, wherein the main vibration vector can be expressed as a Euclidean vector falling on the panoramic image capturing element The photoelectric conversion of the fisheye image on the imaging plane is related to the "image offset" caused by vibration of the panoramic camera equipment, and the optical axis of the panoramic image capturing element passes through the origin of the imaging plane.

如請求項5所述之轉換區塊，更包含：根據該魚眼影像轉換至透視投影影像(perspective projection image)之轉換方式，配合該等校正位移轉換該輸出影像。 The conversion block as described in claim 5 further includes: converting the output image in conjunction with the correction displacements according to a conversion method of converting the fisheye image into a perspective projection image.

如請求項5所述之位移運算區塊，更包含：一高斯查找表，具有該投影值輸入端以及該主振動向量數值輸入端，供以輸出該校正位移。 The displacement calculation block as described in claim 5 further includes: a Gaussian lookup table having the projection value input terminal and the main vibration vector value input terminal for outputting the corrected displacement.

如請求項6所述之位移運算區塊，更包含：一無失真查找表，具有該校正位移輸入端以及像素輸入端，供以輸出無失真轉換位移，並據此轉換該輸出影像。 The displacement calculation block of claim 6 further includes: a distortion-free lookup table having the correction displacement input terminal and the pixel input terminal for outputting the distortion-free conversion displacement and converting the output image accordingly.

如請求項5所述之全景攝像設備，更包含：一中央控制器，控制該振動檢測區塊以及該位移運算區塊，供以在該振動資訊不足以構成影像偏移時，令該主振動向量之數值為0，以及令該位移運算區塊不作動以輸出該校正位移為0 The panoramic camera device as described in claim 5 further includes: a central controller that controls the vibration detection block and the displacement calculation block to cause the main vibration when the vibration information is insufficient to constitute an image offset. The value of the vector is 0, and the displacement operation block is inactive to output the corrected displacement as 0.

如請求項6所述之全景攝像設備，更包含：一中央控制器，控制該轉換區塊，供以接受外部訊號切換該輸出影像為球形影像或方形影像，其中，當外部訊號係指定輸出球形影像，該中央控制器即令該轉換區塊僅根據該等校正位移將該魚眼影像轉換為該輸出影像；以及當外部訊號係指定輸出方形影像，該中央控制器即令該轉換區塊根據該等校正位移以及該透視投影轉換方式將該魚眼影像轉換為一輸出影像。 The panoramic camera device as described in claim 6 further includes: a central controller that controls the conversion block for receiving an external signal to switch the output image into a spherical image or a square image, wherein, When the external signal is designated to output a spherical image, the central controller instructs the conversion block to convert the fisheye image to the output image only based on the correction displacements; and when the external signal is designated to output a square image, the central controller instructs the The conversion block converts the fisheye image into an output image according to the correction displacements and the perspective projection conversion method.