TWI512351B - Multi-camera imaging system, and compensation method for image reconstruction - Google Patents
Multi-camera imaging system, and compensation method for image reconstruction Download PDFInfo
- Publication number
- TWI512351B TWI512351B TW103126022A TW103126022A TWI512351B TW I512351 B TWI512351 B TW I512351B TW 103126022 A TW103126022 A TW 103126022A TW 103126022 A TW103126022 A TW 103126022A TW I512351 B TWI512351 B TW I512351B
- Authority
- TW
- Taiwan
- Prior art keywords
- image
- module
- lens
- sensor
- spacing
- Prior art date
Links
Landscapes
- Studio Devices (AREA)
- Lens Barrels (AREA)
Description
本發明為一種多相機影像擷取系統與影像重組補償方法，特別是指根據系統內多鏡頭模組與多感測器模組間位移或間距變化產生影像重組參數的多相機影像擷取系統與影像重組補償方法。The invention relates to a multi-camera image capturing system and a image recombination compensation method, in particular to a multi-camera image capturing system for generating image recombination parameters according to displacement or spacing change between a multi-lens module and a multi-sensor module in the system and Image recombination compensation method.
圖1示意顯示習知技術陣列式相機，陣列式相機由多個相機組成，每個相機為一個鏡頭與一個對應的影像感測器的組成，陣列式相機即包括多鏡頭模組與多感測模組。此類陣列式相機設有如圖中顯示陣列型式排列的鏡頭101，下方設有對應各鏡頭101的多個陣列排列的影像感測器(圖未示)，以及外部承載機構10。1 is a schematic diagram showing a conventional array camera. The array camera is composed of a plurality of cameras, each camera is composed of a lens and a corresponding image sensor. The array camera includes a multi-lens module and multi-sensing. Module. Such an array camera is provided with a lens 101 arranged in an array pattern as shown in the figure, and an image sensor (not shown) corresponding to a plurality of arrays of the respective lenses 101, and an external carrier mechanism 10 are disposed below.
從圖1顯示的多鏡頭模組可知，各個鏡頭101之間有一定的間距，因此拍攝同一個場景時，多個相機取得複數個影像，場景可具有至少一個景物，同一景物於複數個感測影像中存在著視差，將透過後端軟體或硬體執行重組複數個影像，獲得融合影像，重組影像時會利用到校正多個景物成像位置與成像範圍的重組參數，使得融合影像具有正確而更良好的影像品質，此外此類相機可以具有更小的體積。It can be seen from the multi-lens module shown in FIG. 1 that there is a certain distance between the respective lenses 101. Therefore, when shooting the same scene, multiple cameras acquire a plurality of images, and the scene can have at least one scene, and the same scene is in multiple sensing. There is parallax in the image, and the reconstructed plurality of images will be reconstructed through the back end software or hardware to obtain the fused image. The reconstructed image will be used to correct the recombination parameters of the imaging position and imaging range of multiple scenes, so that the fused image has correct and more Good image quality, and this type of camera can have a smaller size.
在相機對焦系統的領域中，鏡頭與影像感測器之間具有一個可以清楚成像的間距，根據光學成像公式(1/u+1/v=1/f)，其中物距(u)為鏡頭與景物的距離，像距(v)為鏡頭與成像位置的距 離，公式表示焦距(f)的倒數等於物距的倒數加上像距的倒數。在一般情況下，物距相對於像距為極大，根據成像公式，像距因此近似焦距。In the field of camera focusing systems, there is a clearly imageable spacing between the lens and the image sensor, according to the optical imaging formula (1/u+1/v=1/f), where the object distance (u) is the lens The distance from the scene, the image distance (v) is the distance between the lens and the imaging position. Off, the formula indicates that the reciprocal of the focal length (f) is equal to the reciprocal of the object distance plus the reciprocal of the image distance. In general, the object distance is extremely large with respect to the image distance, and according to the imaging formula, the image distance is thus approximately the focal length.
對焦正確的相機可以取得清楚成像的相片，也就是在適當的像距上呈現清楚的影像。然而，鏡頭可能因為外在因素造成焦距產生變化。比如溫度變化對鏡頭熱脹冷縮的效應產生了變化，或是溫度改變對鏡頭產生折射率的變化等，鏡頭焦距因此產生改變，使得成像有誤差。於是習知技術提出一些溫度補償的技術，應用在相機中，可以補償相機之焦距因為溫度所產生的變化，根據焦距與像距的關係，因此透過修正像距讓透過鏡頭的影像仍可以清楚成像於影像感測器上。A camera with the correct focus can take a clearly imaged image, that is, a clear image at the appropriate image distance. However, the lens may cause a change in focal length due to external factors. For example, the effect of temperature changes on the thermal expansion and contraction of the lens changes, or the change in the refractive index of the lens due to temperature changes, etc., and the focal length of the lens is thus changed, resulting in an error in imaging. Therefore, the prior art proposes some temperature compensation techniques, which can be applied to the camera to compensate for the change of the focal length of the camera due to the temperature. According to the relationship between the focal length and the image distance, the image transmitted through the lens can still be clearly imaged by correcting the image distance. On the image sensor.
此溫度補償的機制包括一種可以隨著溫度改變而有外觀型態變化的物體。若是相機載有此物體，透過機構的設計，可以藉此物體補償因為溫度改變而產生鏡頭與影像感測器焦距變化的問題。This mechanism of temperature compensation includes an object that can change in appearance as the temperature changes. If the camera carries the object, the design of the transmission mechanism can compensate the problem that the focal length of the lens and the image sensor changes due to the temperature change.
此類物體材料有隨著熱而變化外觀型態的特性，比如因為溫度上升產生外觀曲線改變，又因為溫度下降而產生反方向曲線改變，因此可以藉此特性帶動機構，作為修正鏡頭與影像感測器的間距的機制。Such material materials have the characteristics of changing the appearance type with heat, for example, the appearance curve changes due to the temperature rise, and the reverse direction curve changes due to the temperature drop, so that the mechanism can be driven by the characteristic as the correction lens and the image sense. The mechanism of the spacing of the detectors.
舉例來說，此具有溫度補償特性的物體如以複合材料置做的裝置，其中有至少兩種材料，兩種材料因為溫度改變而有不同的熱脹冷縮效應，因此材料產生一個方向的曲度，藉此可以作為一種補償機構。For example, such an object having a temperature-compensating property, such as a device made of a composite material, has at least two materials, and the two materials have different thermal expansion and contraction effects due to temperature changes, so that the material produces a curve in one direction. Degree, which can be used as a compensation mechanism.
習知技術中採用溫度補償機構的鏡頭模組應用中，在鏡筒外且連接於所述鏡筒與鏡座之間設有多個可以隨著溫度變化而彎曲的補償片，藉此可以帶動鏡筒沿光軸方向移動，使鏡頭模組的成像面保持在預定平面。此為一種利用材料特性而補償溫度所造成改變的技術之一。In a lens module application using a temperature compensation mechanism in the prior art, a compensation piece that can be bent along with a temperature change is provided outside the lens barrel and connected between the lens barrel and the lens holder, thereby driving The lens barrel moves in the optical axis direction to maintain the imaging surface of the lens module in a predetermined plane. This is one of the techniques for utilizing material properties to compensate for changes in temperature.
為了提出一種解決在陣列式相機內針對影像擷取模組因焦距改變造成景物於影像感測器上的成像位置改變的問題，揭露書揭示一種多相機影像擷取系統與影像重組補償方法，能夠根據鏡頭模組位移的距離或與影像感測器的位移或間距變化而提出對應的影像重組參數。In order to solve the problem that the image capturing module in the array camera changes the imaging position of the scene on the image sensor due to the focal length change, the disclosure discloses a multi-camera image capturing system and image recombination compensation method, which can Corresponding image recombination parameters are proposed according to the distance of the lens module displacement or the displacement or spacing of the image sensor.
陣列式相機內的影像擷取模組焦距改變，主因包括因溫度改變使得鏡頭折射率變化，或因鏡頭熱脹冷縮效應而影響焦距改變，習知方案中已提出了補償此溫度造成對焦失誤的問題，補償即造成鏡頭模組位移，影響陣列式相機中多感測器模組與多鏡頭模組間的間距(關聯到鏡頭成像的像距)。但此焦距與像距一旦改變也隨之造成景物於影像感測器中的成像位置或是成像範圍產生改變，因此若沿用陣列式相機原本的重組參數，將無法正確地重組影像，於是本揭露書提出的影像重組補償方法即提出對應鏡頭模組位移或間距的影像重組參數，提供陣列式相機的影像重組之用。The focal length of the image capture module in the array camera changes, including the change of the refractive index of the lens due to the temperature change, or the change of the focal length due to the thermal expansion and contraction effect of the lens. In the conventional scheme, compensation for this temperature has been proposed to cause focus error. The problem is that the compensation causes the displacement of the lens module, which affects the distance between the multi-sensor module and the multi-lens module in the array camera (associated with the image distance of the lens imaging). However, if the focal length and the image distance are changed, the imaging position or the imaging range of the scene in the image sensor is changed. Therefore, if the original recombination parameters of the array camera are used, the image cannot be reconstructed correctly, and thus the disclosure is disclosed. The image recombination compensation method proposed in the book proposes image recombination parameters corresponding to the displacement or spacing of the lens module, and provides image reconstruction of the array camera.
揭露書所揭示的多相機影像擷取系統為了達成因為焦距補償而修正重組參數的目的，其中主要元件為影像擷取模組，設有多鏡頭模組與多感測器模組，其中多鏡頭模組由多個鏡頭排列組成，多感測器模組由多個影像感測器排列組成，多個影像感測器的位置係一對一對應到多個鏡頭的排列位置。系統再有一間距調整單元，是為針對溫度影響造成焦距改變的一溫度補償機構，比如溫度變化產生鏡頭的折射率變化或熱漲冷縮而改變了焦距，因應此變化，間距調整單元即用以調整多鏡頭模組與多感測器模組之間距；系統再包括一位置感測模組，用以感測多鏡頭模組與多感測器模組之位移或間距變化，因此系統能根據位置感測模組所感測到多鏡頭模組與多感測器模組的位移或間距變化而對應產生 一組影像重組參數，以進行重組影像。The multi-camera image capture system disclosed in the disclosure aims to achieve the purpose of correcting the recombination parameters due to the focus compensation, wherein the main component is an image capture module, and a multi-lens module and a multi-sensor module are provided, wherein the multi-lens The module is composed of a plurality of lens arrays, and the multi-sensor module is composed of a plurality of image sensor arrays, and the positions of the plurality of image sensors are one-to-one corresponding to the arrangement positions of the plurality of lenses. The system further has a spacing adjustment unit, which is a temperature compensation mechanism for causing a change in the focal length for the temperature influence. For example, the temperature change causes the refractive index change of the lens or the thermal expansion and contraction to change the focal length. In response to the change, the spacing adjustment unit is used. Adjusting the distance between the multi-lens module and the multi-sensor module; the system further includes a position sensing module for sensing displacement or spacing change of the multi-lens module and the multi-sensor module, so the system can The position sensing module senses a change in displacement or spacing of the multi-lens module and the multi-sensor module. A set of image recombination parameters for recombination images.
在多相機影像擷取系統的影像重組補償方法的實施例之一，方法包括取得多相機影像擷取系統中多鏡頭模組與多感測器模組之間的位移或間距變化，當中若偵測到位移或間距改變，比如補償因為溫度改變造成焦距改變所調整的間距，即取得一組影像重組參數，用以進行影像重組。在一實施例中，系統提供一查找表，查找表記載多鏡頭模組與多感測器模組之間的位移或間距與影像重組參數的對應關係，可以據此取得某位移或間距所對應到的影像重組參數。One of the embodiments of the image recombination compensation method of the multi-camera image capture system includes obtaining a displacement or a change in the spacing between the multi-lens module and the multi-sensor module in the multi-camera image capture system, wherein A change in displacement or spacing is detected, such as compensating for the pitch adjusted by the change in focus due to temperature changes, that is, obtaining a set of image recombination parameters for image recombination. In an embodiment, the system provides a lookup table, and the lookup table records the correspondence between the displacement or the spacing between the multi-lens module and the multi-sensor module and the image recombination parameter, and can obtain a corresponding displacement or spacing according to the method. Image reorganization parameters.
為了能更進一步瞭解本發明為達成既定目的所採取之技術、方法及功效，請參閱以下有關本發明之詳細說明、圖式，相信本發明之目的、特徵與特點，當可由此得以深入且具體之瞭解，然而所附圖式與附件僅提供參考與說明用，並非用來對本發明加以限制者。In order to further understand the technology, method and effect of the present invention in order to achieve the intended purpose, reference should be made to the detailed description and drawings of the present invention. The drawings and the annexed drawings are intended to be illustrative and not to limit the invention.
10‧‧‧承載機構10‧‧‧Loading mechanism
101‧‧‧鏡頭101‧‧‧ lens
22,22’‧‧‧鏡頭單元22,22’‧‧‧ lens unit
20‧‧‧影像感測器20‧‧‧Image Sensor
201‧‧‧入射光線201‧‧‧ incident light
203‧‧‧折射光線203‧‧‧Refracted light
201’‧‧‧入射光線201’‧‧‧ incident light
203’‧‧‧折射光線203'‧‧‧ Refracted light
201”‧‧‧入射光線201”‧‧‧ incident light
203”‧‧‧折射光線203”‧‧‧ Refracted light
24‧‧‧間距調整單元24‧‧‧pitch adjustment unit
30‧‧‧影像擷取模組承載機構30‧‧‧Image capture module carrier
32‧‧‧多感測器模組32‧‧‧Multiple sensor modules
302‧‧‧影像感測器302‧‧‧Image Sensor
34‧‧‧多鏡頭模組34‧‧‧Multi-lens module
304‧‧‧鏡頭304‧‧‧ lens
36‧‧‧溫度補償機構36‧‧‧ Temperature compensation agency
303‧‧‧霍爾效應感測器303‧‧‧Hall Effect Sensor
301‧‧‧磁性元件301‧‧‧Magnetic components
35‧‧‧移動部位35‧‧‧moving parts
401,401’‧‧‧人臉401,401’‧‧‧ faces
402,402’‧‧‧愛心402,402’‧‧‧Love
50‧‧‧影像擷取模組50‧‧‧Image capture module
501‧‧‧多鏡頭模組501‧‧‧Multi-lens module
502‧‧‧多感測器模組502‧‧‧Multiple sensor modules
52‧‧‧間距調整單元52‧‧‧pitch adjustment unit
54‧‧‧位置感測模組54‧‧‧ position sensing module
56‧‧‧影像重組參數產生單元56‧‧‧Image Recombination Parameter Generation Unit
58‧‧‧影像重組模組58‧‧‧Image Reassembly Module
503‧‧‧承載機構503‧‧‧Loading mechanism
步驟S601~S609‧‧‧影像重組補償步驟Step S601~S609‧‧‧Image Recombination Compensation Step
圖1顯示習知技術陣列式相機的多鏡頭模組示意圖；圖2A顯示多相機影像擷取系統聚焦正確的狀態示意圖；圖2B顯示多相機影像擷取系統對焦不正確的狀態示意圖；圖2C顯示多相機影像擷取系統調整間距後的狀態示意圖；圖3A所示為多相機影像擷取系統中採用溫度補償機構的結構示意圖之一；圖3B所示為多相機影像擷取系統中採用溫度補償機構的結構示意圖之二；圖4A與圖4B所示為多相機影像擷取系統因為焦距變化致使成像改變的示意圖；圖5顯示本發明多相機影像擷取系統實施例的功能方塊圖；圖6顯示之流程描述本發明多相機影像擷取系統之影像重組 補償步驟實施例。1 is a schematic diagram of a multi-lens module of a conventional array camera; FIG. 2A is a schematic diagram showing a state in which a multi-camera image capturing system is in focus; FIG. 2B is a schematic diagram showing a state in which a multi-camera image capturing system is not in focus; FIG. 2C shows The state of the multi-camera image capture system after adjusting the spacing; FIG. 3A shows one of the structural diagrams of the temperature compensation mechanism used in the multi-camera image capture system; FIG. 3B shows the temperature compensation in the multi-camera image capture system. FIG. 4A and FIG. 4B are schematic diagrams showing changes in imaging of a multi-camera image capturing system due to focal length changes; FIG. 5 is a functional block diagram of an embodiment of the multi-camera image capturing system of the present invention; FIG. Flow of display describes image recombination of the multi-camera image capture system of the present invention Compensation step embodiment.
陣列式相機(camera array)是採用多個相機合併而成的相機裝置，其中多個相機中每一個鏡頭單元對應到一個影像感應元件(如CCD,CMOS)，使得形成多顆小型相機合併組成一個相機，陣列式相機也倚賴後端透過韌體或是軟體方法重組影像，將多個相機個別所擷取到的影像訊號重組獲得一融合影像。A camera array is a camera device that combines multiple cameras. Each of the plurality of cameras corresponds to an image sensing component (such as CCD, CMOS), so that a plurality of small cameras are combined to form a single camera. Cameras, array cameras also rely on the back end to recombine images through firmware or software, and recombine the image signals captured by multiple cameras to obtain a fused image.
揭露書所載之多相機影像擷取系統的影像擷取模組採用陣列式鏡頭(lens array)，其中多鏡頭模組中設有多個以特定陣列型式排列的鏡頭，對應多鏡頭模組的位置上的多感測器模組中也設有位置一對一對應各個鏡頭的多個影像感測器。The image capturing module of the multi-camera image capturing system disclosed in the disclosure uses an array lens, wherein the multi-lens module is provided with a plurality of lenses arranged in a specific array pattern, corresponding to the multi-lens module. A plurality of image sensors corresponding to the respective lenses are also disposed in the multi-sensor module at the position.
多相機影像擷取系統中多鏡頭模組可參考圖1顯示的結構示意圖，在原始的設計中，其中多鏡頭模組與多感測器模組之間距定義出一能清楚成像的像距。根據成像公式，焦距的倒數等於像距的倒數加上物距的倒數，因此在物距相對於像距為極大的一般情況下，焦距的改變將主要導致像距的變化。The multi-lens module of the multi-camera image capturing system can refer to the structural diagram shown in FIG. 1. In the original design, the distance between the multi-lens module and the multi-sensor module defines a clear image distance. According to the imaging formula, the reciprocal of the focal length is equal to the reciprocal of the image distance plus the reciprocal of the object distance. Therefore, in the general case where the object distance is extremely large with respect to the image distance, the change of the focal length will mainly cause the change of the image distance.
圖2A顯示多相機影像擷取系統正常聚焦的狀態，示意以多鏡頭模組中之一個鏡頭單元22對應多感測器模組之一影像感測器20來說明。鏡頭單元22與影像感測器20間具有一像距，入射光線201經此鏡頭單元22後預設能夠正確對焦在對應位置上的影像感測器20上，如所示的折射光線203，影像因此可以清楚成像在影像感測器20上。2A shows a state in which the multi-camera image capturing system is normally focused, and illustrates that one of the multi-lens modules corresponds to one of the multi-sensor modules, the image sensor 20. The lens unit 22 has an image distance between the lens unit 22 and the image sensor 20, and the incident light ray 201 is preset to be correctly focused on the image sensor 20 at the corresponding position through the lens unit 22, as shown by the refracted light 203, the image Therefore, it can be clearly imaged on the image sensor 20.
當圖示中的多相機影像擷取系統的多鏡頭模組(以一鏡頭單元22’為例)的焦距因為溫度的影響下產生變化，比如因為鏡頭熱脹冷縮效應或鏡頭隨著外界溫度產生折射率改變，於是產生對焦誤差的現象，如圖2B所示之示意圖。When the multi-lens module of the multi-camera image capture system shown in the figure (taking a lens unit 22' as an example), the focal length changes due to the influence of temperature, for example, due to the thermal expansion and contraction effect of the lens or the external temperature of the lens. A refractive index change is generated, and a phenomenon of focusing error is generated, as shown in the schematic diagram of FIG. 2B.
在圖2B中，同樣以多鏡頭模組內的單一鏡頭單元22’來說 明，入射光線201’經多鏡頭模組鏡頭單元22’折射後，形成折射光線203’，此例係因溫度上升導致折射率下降，故鏡頭焦距拉長而產生成像對焦不正確的情況，使得焦點偏離影像感測器20。In Figure 2B, also in the single lens unit 22' in the multi-lens module The incident light ray 201 ′ is refracted by the multi-lens module lens unit 22 ′ to form a refracted ray 203 ′. In this case, the refractive index decreases due to the temperature rise, so that the focal length of the lens is elongated and the imaging focus is not correct, so that the imaging focus is incorrect. The focus is off the image sensor 20.
圖2C顯示為調整間距後的狀態。此例之鏡頭單元22’上設有間距調整單元24，間距調整單元24主要帶動了整個多鏡頭模組進行溫度補償而位移，透過改變像距而使得鏡頭單元22’正確對焦在影像感測器20上。如此圖顯示進入系統的入射光線201”，折射後形成折射光線203”，因為間距調整單元24而能使得光線能正確聚焦於影像感測器20上。Fig. 2C shows the state after the pitch is adjusted. The lens unit 22' of this example is provided with a spacing adjusting unit 24, and the spacing adjusting unit 24 mainly drives the entire multi-lens module to perform temperature compensation and displacement, and the lens unit 22' is correctly focused on the image sensor by changing the image distance. 20 on. This figure shows the incident ray 201" entering the system, which is refracted to form refracted ray 203" because the spacing adjustment unit 24 enables the light to be properly focused on the image sensor 20.
舉例來說，針對溫度變化造成對焦誤差的問題，上述的間距調整單元24在習知技術曾採用特定溫度補償機構，透過材料隨著溫度變化而產生形變的特性，修正系統中的多鏡頭模組(此例以鏡頭單元22’為例)與多感測器模組20之間距，使得經多鏡頭模組之鏡頭單元22’的景物光線可以清楚成像於多感測器模組20上，如圖2C所示。For example, for the problem of the focus error caused by the temperature change, the above-mentioned pitch adjustment unit 24 has adopted a specific temperature compensation mechanism in the prior art, and the deformation characteristic of the material is changed with temperature, and the multi-lens module in the system is modified. (This example takes the lens unit 22' as an example) and the distance between the multi-sensor modules 20, so that the scene light of the lens unit 22' of the multi-lens module can be clearly imaged on the multi-sensor module 20, such as Figure 2C shows.
根據本揭露書所載實施例，經前述間距調整後，影像擷取模組中多鏡頭模組將產生位移，此位移顯示多鏡頭模組與多感測器模組之間像距已經改變，因此景物經多鏡頭模組於多感測器模組上成像的位置與/或範圍已經改變，示意圖可參閱圖4A與圖4B，於是本發明提出了校正因為修正焦距變化而牽動多鏡頭模組於多感測器模組之間距(可對應到所述之像距)變化的影像重組技術。According to the embodiment of the present disclosure, after the pitch adjustment, the multi-lens module in the image capturing module will be displaced, and the displacement shows that the image distance between the multi-lens module and the multi-sensor module has changed. Therefore, the position and/or range of the scene imaged by the multi-lens module on the multi-sensor module has been changed. Referring to FIG. 4A and FIG. 4B, the present invention proposes to correct the multi-lens module by correcting the focal length change. Image recombination technology that varies between multiple sensor modules (corresponding to the image distance).
而取得像距或多鏡頭模組與多感測器模組之位移或間距變化的資訊前，本發明揭露書提出的多相機影像擷取系統先偵測出多鏡頭模組的相對位移。可繼續參閱圖3A與圖3B所揭示的實施態樣。The multi-camera image capturing system proposed by the present invention first detects the relative displacement of the multi-lens module before obtaining the information of the displacement or the change of the spacing or the multi-lens module and the multi-sensor module. Reference may be continued to the embodiment disclosed in Figures 3A and 3B.
在圖3A所示的多相機影像擷取系統的結構示意圖中，影像擷取模組承載機構30如陣列式相機模組的主體外框結構，為一固定整個裝置的機構，此固定部位內部設置多相機影像擷取系統的影 像擷取模組，包括多感測器模組32，多感測器模組32由多個影像感測器302排列組成。圖3A更示意表示承載機構30另以移動部位35承載並帶動多鏡頭模組34，多鏡頭模組34由多個鏡頭304排列組成，其中多個影像感測器302的位置係一對一對應到多個鏡頭304的排列位置，其中鏡頭304具有至少一鏡片。In the structure diagram of the multi-camera image capturing system shown in FIG. 3A, the image capturing module carrying mechanism 30, such as the main frame structure of the array camera module, is a mechanism for fixing the entire device, and the fixing portion is internally disposed. Multi-camera image capture system The capture module includes a multi-sensor module 32, and the multi-sensor module 32 is composed of a plurality of image sensors 302. FIG. 3A further shows that the carrying mechanism 30 further carries and drives the multi-lens module 34 with the moving part 35. The multi-lens module 34 is composed of a plurality of lenses 304, wherein the positions of the plurality of image sensors 302 are one-to-one. To the arrangement of the plurality of lenses 304, wherein the lens 304 has at least one lens.
系統內焦距之改變係因溫度改變造成鏡頭熱脹冷縮造成，或因溫度改變直接造成多個鏡頭本身折射率改變所造成。當多鏡頭模組34內多個鏡頭304因為溫度造成焦距改變，因此藉由影像擷取模組中所設的溫度補償機構36帶動移動部位35上的多鏡頭模組34進行補償位移，使得改變多鏡頭模組34與多感測器模組32之間的間距，目的是讓光線能夠清楚聚焦於多感測器模組32上，運作方式亦如前述圖2C的間距調整單元(24)。The change of the focal length in the system is caused by the thermal expansion and contraction of the lens due to the temperature change, or the refractive index of the lens itself is directly changed due to the temperature change. When the plurality of lenses 304 in the multi-lens module 34 change the focal length due to the temperature, the temperature compensation mechanism 36 provided in the image capturing module drives the multi-lens module 34 on the moving portion 35 to compensate for the displacement, so that the change is made. The spacing between the multi-lens module 34 and the multi-sensor module 32 is such that the light can be clearly focused on the multi-sensor module 32 in a manner similar to the pitch adjustment unit (24) of FIG. 2C above.
此類溫度補償機構36實施方式之一可由多種材質疊構而成，各材質具有不同的膨脹係數，因此結合的機構將可以根據溫度改變而有一定曲度的變化，藉此作為機構上的補償用途。溫度補償機構36的初始狀態示意顯示如圖3A，一端連結著承載機構30的固定部位，另一端則是連結承載機構的移動部位35。多鏡頭模組34與多感測器模組32有一初始間距，之後經過溫度補償之後，溫度補償機構36可能有曲率的物理變化，因此帶動移動部位35連同多鏡頭模組34移動，使得與多感測器模組32之間有了另一間距，如圖3B所示。One of the embodiments of the temperature compensation mechanism 36 can be formed by stacking a plurality of materials, each material having a different expansion coefficient, so that the combined mechanism can have a certain curvature change according to the temperature change, thereby serving as a mechanism compensation. use. The initial state of the temperature compensating mechanism 36 is schematically shown in Fig. 3A, the fixed portion of the carrying mechanism 30 is coupled to one end, and the moving portion 35 of the supporting mechanism is coupled to the other end. The multi-lens module 34 has an initial spacing from the multi-sensor module 32. After the temperature compensation, the temperature compensating mechanism 36 may have a physical change in curvature, thereby driving the moving portion 35 together with the multi-lens module 34 to move There is another spacing between the sensor modules 32, as shown in Figure 3B.
然而，經間距調整後，影像可以正確地成像於感測器上，同時也造成成像在感測器上的成像位置或範圍也產生了變化，因此原本用於多相機影像擷取系統的影像重組參數同樣需要修正，本發明系統即先取得多鏡頭模組34相對於多感測器模組32的位移距離，實施例之一將於此多相機影像擷取系統中設置有一位置感測模組(301,303)，用以感測多鏡頭模組34與多感測器模組32之間距變化。之後再根據此位移或間距提出對應的影像重組參數。However, after the pitch adjustment, the image can be correctly imaged on the sensor, and the imaging position or range of the image on the sensor is also changed, so the image recombination originally used for the multi-camera image capture system The parameter also needs to be corrected. The system of the present invention first obtains the displacement distance of the multi-lens module 34 relative to the multi-sensor module 32. One of the embodiments will provide a position sensing module in the multi-camera image capturing system. (301, 303) for sensing a change in the distance between the multi-lens module 34 and the multi-sensor module 32. Then, corresponding image recombination parameters are proposed according to the displacement or the spacing.
實施例同樣可參閱圖3A與圖3B，位置感測模組之實施例之一可以在多相機影像擷取系統中的固定結構上，如影像擷取模組承載機構30，設有一組霍爾效應感測器(Hall effect sensor)303，以及在可被溫度補償機構36帶動的多鏡頭模組34的相關結構上設有一個作為指標用的磁性元件301。根據實施例之一，霍爾效應感測器303可設於承載機構30的固定部位，磁性元件301則可設於承載機構30的移動部位35，使得霍爾效應感測器303可以根據偵測到因為磁性元件301移動所產生的磁場變化，進而判斷出位移的量。The embodiment can also refer to FIG. 3A and FIG. 3B. One of the embodiments of the position sensing module can be provided on a fixed structure in the multi-camera image capturing system, such as the image capturing module carrying mechanism 30, and a set of Halls. A Hall effect sensor 303, and a related structure of the multi-lens module 34 that can be driven by the temperature compensating mechanism 36, are provided with a magnetic element 301 as an index. According to one embodiment, the Hall effect sensor 303 can be disposed at a fixed portion of the carrier mechanism 30, and the magnetic component 301 can be disposed at the moving portion 35 of the carrier mechanism 30, so that the Hall effect sensor 303 can be detected according to the detection. The amount of displacement is determined by the change in the magnetic field generated by the movement of the magnetic element 301.
在另一實施例中，位置感測模組可為一應變感測器(strain gauge)，其可以直接設於溫度補償結構上，用以量測溫度補償機構的應變量，藉此判斷出位移量。In another embodiment, the position sensing module can be a strain gauge, which can be directly disposed on the temperature compensation structure for measuring the strain of the temperature compensation mechanism, thereby determining the displacement. the amount.
圖4A與圖4B所示之狀態示意圖用以表示上述系統在感測器上的成像因為溫度補償產生的位移而變化。The state diagrams shown in Figures 4A and 4B are used to indicate that the imaging of the above system on the sensor varies due to the displacement caused by temperature compensation.
圖4A顯示多相機影像擷取系統拍攝一個場景時，場景有兩個景物，景物於系統內之一相機的感測器成像，如人臉401與愛心402。接著如圖4B所示，此例顯示溫度上升造成焦距變長的情況，因為溫度造成焦距改變後的補償措施，使得多鏡頭模組與多感測器模組之間距隨之變化，補償措施可使景物影像清楚的地成像於感測器上，但鏡頭與感測器間距變化與焦距變化讓景象於某一感測器的成像位置與/或範圍也產生改變，如圖所示的人臉401’與愛心402’。示意圖如圖4A與圖4B相較之下，其中物體的成像位置與範圍都產生了改變。FIG. 4A shows that when the multi-camera image capturing system captures a scene, the scene has two scenes, and the scene is imaged by a sensor of one of the cameras in the system, such as face 401 and love 402. Then, as shown in FIG. 4B, this example shows that the temperature rises and the focal length becomes longer. Because the temperature compensates for the focal length change, the distance between the multi-lens module and the multi-sensor module changes accordingly, and the compensation measure can be The image of the scene is clearly imaged on the sensor, but the change of the distance between the lens and the sensor and the change of the focal length also change the imaging position and/or range of the scene in a certain sensor, as shown in the figure. 401 'and love 402'. The schematic is shown in Figures 4A and 4B, in which the imaging position and range of the object are changed.
圖5顯示本發明多相機影像擷取系統實施例的功能方塊圖。Figure 5 is a functional block diagram showing an embodiment of the multi-camera image capture system of the present invention.
多相機影像擷取系統主要元件為影像擷取模組50，承載機構503上設有主要元件包括多鏡頭模組501與多感測器模組502，其中鏡頭與影像感測器的位置為相互對應，因此經過各個鏡頭的光線將成像於對應之影像感測器上。根據實施例之一，多鏡頭模組 501設於承載機構503的移動部位上，多感測器模組502則相對設於承載機構503的固定部位上，如整個系統的承載機構503。The main component of the multi-camera image capturing system is an image capturing module 50. The main components of the carrying mechanism 503 include a multi-lens module 501 and a multi-sensor module 502, wherein the positions of the lens and the image sensor are mutual Correspondingly, the light passing through each lens will be imaged on the corresponding image sensor. According to one of the embodiments, the multi-lens module The 501 is disposed on the moving portion of the carrying mechanism 503, and the multi-sensor module 502 is oppositely disposed on the fixed portion of the supporting mechanism 503, such as the carrying mechanism 503 of the entire system.
影像擷取模組50上設有補償對焦錯誤的間距調整單元52，間距調整單元52連接多鏡頭模組501，進行補償時，當間距調整單元52帶動多鏡頭模組501移動時。位置感測模組54可以感測多鏡頭模組501的位置改變，系統透過位置感測模組54取得承載機構503上多鏡頭模組501與多感測器模組502之間的位移量或間距，位置感測模組54實施例可參閱圖3A與圖3B之描述。The image capturing module 50 is provided with a pitch adjusting unit 52 for compensating for the focus error. The spacing adjusting unit 52 is connected to the multi-lens module 501 for compensation. When the spacing adjusting unit 52 drives the multi-lens module 501 to move. The position sensing module 54 can sense the position change of the multi-lens module 501, and the system obtains the displacement between the multi-lens module 501 and the multi-sensor module 502 on the carrying mechanism 503 through the position sensing module 54 or The spacing, position sensing module 54 embodiment can be described with reference to Figures 3A and 3B.
接著，本發明所揭示的多相機影像擷取系統更根據此位移量或間距提出用來重組影像的影像重組參數，圖中影像重組參數產生單元56將根據多鏡頭模組501與多感測器模組502之間的位移量或間距調整而對應判斷出校正影像重組參數的資訊，最後將對應的重組參數提供給影像重組模組58進行影像重組。值得一提的是本發明提出對應位移或間距變化的影像重組參數，由影像重組模組58校正因為多鏡頭模組501與多感測器模組502之間因為補償焦距改變所產生的重組誤差問題，產生品質良好的影像。Then, the multi-camera image capturing system disclosed by the present invention further proposes image recombination parameters for recombining images according to the displacement amount or spacing. The image recombination parameter generating unit 56 in the figure will be based on the multi-lens module 501 and the multi-sensor. The displacement amount or the spacing between the modules 502 is adjusted to determine the information for correcting the image recombination parameters, and finally the corresponding recombination parameters are provided to the image recombination module 58 for image recombination. It is worth mentioning that the present invention proposes image recombination parameters corresponding to changes in displacement or spacing, and the image recombination module 58 corrects the recombination error caused by the compensation of the focal length change between the multi-lens module 501 and the multi-sensor module 502. The problem is to produce good quality images.
根據實施例之一，根據多鏡頭模組501與多感測器模組502的位移或間距變化而提供影像重組參數的方法之一是建立一個查找表(lookup table,CUT)。對應的影像重組參數可利用對照多相機影像擷取系統內建的查找表獲得，查找表的建立是根據實驗獲得多鏡頭模組501與多感測器模組502的間距(像距改變牽動焦距變化)與影像重組參數的關係。舉例來說，查找表記載多鏡頭模組501與多感測器模組502之間的位移或間距與影像重組參數的對應關係，也就是記載多鏡頭模組501的位置變化所對應的影像重組參數。According to one of the embodiments, one of the methods for providing image recombination parameters according to the displacement or spacing change of the multi-lens module 501 and the multi-sensor module 502 is to establish a lookup table (CUT). Corresponding image recombination parameters can be obtained by using a look-up table built in the multi-camera image capture system. The search table is created according to the experiment. The distance between the multi-lens module 501 and the multi-sensor module 502 is obtained. Change) Relationship with image recombination parameters. For example, the lookup table records the correspondence between the displacement or the spacing between the multi-lens module 501 and the multi-sensor module 502 and the image recombination parameters, that is, the image recombination corresponding to the position change of the multi-lens module 501. parameter.
接著如圖6所示，其中流程描述本發明多相機影像擷取系統之影像重組補償步驟實施例。Next, as shown in FIG. 6, the flow describes an embodiment of the image recombination compensation step of the multi-camera image capturing system of the present invention.
在本發明所揭示的多相機影像擷取系統中，如步驟S601所 述，溫度造成系統的焦距改變，於是可透過溫度補償機制補償因為溫度造成系統焦距產生變化的問題。如步驟S603所述，然而，補償機制同時造成系統內多鏡頭模組與多感測器模組的間距改變，也造成影像於感測器上的成像位置或範圍產生改變，於是多相機影像擷取系統將修正影像重組參數。如步驟S605，系統透過位置感測的功能取得多鏡頭模組與多感測器模組的位移或間距變化，再根據此位移或間距取得對應的影像重組參數。如步驟S607，實施例如上述，可以先建立位移或間距與影像重組參數的查找表，因此可以隨著不同的位移或間距變化而透過查找表得到影像重組參數。In the multi-camera image capturing system disclosed by the present invention, as in step S601 As a result, the temperature causes the focal length of the system to change, so the temperature compensation mechanism can be used to compensate for the problem that the focal length of the system changes due to temperature. As described in step S603, however, the compensation mechanism causes a change in the spacing between the multi-lens module and the multi-sensor module in the system, and also causes a change in the imaging position or range of the image on the sensor, so that the multi-camera image is defective. The system will correct the image recombination parameters. In step S605, the system obtains the displacement or the spacing change of the multi-lens module and the multi-sensor module through the position sensing function, and then obtains corresponding image recombination parameters according to the displacement or the spacing. In step S607, as described above, the lookup table of the displacement or spacing and the image recombination parameters may be established first, so that the image recombination parameters may be obtained through the lookup table as the displacement or spacing changes.
系統接著根據對應的影像重組參數進行多相機影像擷取系統的影像重組，如步驟S609，以獲得多相機影像擷取系統融合後的影像。The system then performs image recombination of the multi-camera image capturing system according to the corresponding image recombination parameters, as in step S609, to obtain an image of the multi-camera image capturing system.
綜上所述，習知技術為了補償因為溫度改變造成鏡頭光學特性改變，包括折射率改變，而產生了焦距變化，於是提出了調整多鏡頭模組與多感測器模組之間距的溫度補償機制，但卻忽略了因為多鏡頭模組與多感測器模組之間距改變造成景物影像成像位置或範圍的變化，使得原有影像重組的參數失效，於是為了讓多相機影像擷取系統正確重組影像，本發明實施例提出了一種影像重組補償方法，取得上述多鏡頭模組與多感測器模組之位移或間距而提出對應的影像重組參數，進而調整影像重組參數。In summary, the conventional technique is to compensate for the change of the optical characteristics of the lens due to the temperature change, including the change of the refractive index, and the focal length change is generated. Therefore, the temperature compensation for adjusting the distance between the multi-lens module and the multi-sensor module is proposed. Mechanism, but it ignores the change of the position or range of the image of the scene due to the change of the distance between the multi-lens module and the multi-sensor module, so that the parameters of the original image reorganization are invalid, so that the multi-camera image capture system is correct. In the embodiment of the present invention, an image recombination compensation method is proposed, and the corresponding image recombination parameters are obtained by obtaining the displacement or spacing of the multi-lens module and the multi-sensor module, and then the image recombination parameters are adjusted.
惟以上所述僅為本發明之較佳可行實施例，非因此即侷限本發明之專利範圍，故舉凡運用本發明說明書及圖示內容所為之等效結構變化，均同理包含於本發明之範圍內，合予陳明。However, the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Therefore, equivalent structural changes that are made by using the specification and the contents of the present invention are equally included in the present invention. Within the scope, it is combined with Chen Ming.
S601‧‧‧多相機影像擷取系統由於溫度影響，焦距產生變化S601‧‧‧Multi-camera image capture system changes in focal length due to temperature effects
S603‧‧‧補償機制補償焦距變化S603‧‧‧Compensation mechanism to compensate for focal length changes
S605‧‧‧偵測間距S605‧‧‧Detection spacing
S607‧‧‧獲得影像重組參數S607‧‧‧ Obtained image recombination parameters
S609‧‧‧獲得融合後影像S609‧‧‧Get the merged image
Claims (9)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| TW103126022A TWI512351B (en) | 2014-07-30 | 2014-07-30 | Multi-camera imaging system, and compensation method for image reconstruction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| TW103126022A TWI512351B (en) | 2014-07-30 | 2014-07-30 | Multi-camera imaging system, and compensation method for image reconstruction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| TWI512351B true TWI512351B (en) | 2015-12-11 |
| TW201604608A TW201604608A (en) | 2016-02-01 |
Family
ID=55407809
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| TW103126022A TWI512351B (en) | 2014-07-30 | 2014-07-30 | Multi-camera imaging system, and compensation method for image reconstruction |
Country Status (1)
| Country | Link |
|---|---|
| TW (1) | TWI512351B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW200848779A (en) * | 2007-06-01 | 2008-12-16 | Hon Hai Prec Ind Co Ltd | Lens module |
| WO2013055960A1 (en) * | 2011-10-11 | 2013-04-18 | Pelican Imaging Corporation | Lens stack arrays including adaptive optical elements |
| TW201409150A (en) * | 2012-09-24 | 2014-03-01 | Wah Hong Ind Corp | Camera module |
-
2014
- 2014-07-30 TW TW103126022A patent/TWI512351B/en active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW200848779A (en) * | 2007-06-01 | 2008-12-16 | Hon Hai Prec Ind Co Ltd | Lens module |
| WO2013055960A1 (en) * | 2011-10-11 | 2013-04-18 | Pelican Imaging Corporation | Lens stack arrays including adaptive optical elements |
| TW201409150A (en) * | 2012-09-24 | 2014-03-01 | Wah Hong Ind Corp | Camera module |
Also Published As
| Publication number | Publication date |
|---|---|
| TW201604608A (en) | 2016-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9681043B2 (en) | Multi-camera imaging system, and compensation method for image reconstruction | |
| JP5549230B2 (en) | Ranging device, ranging module, and imaging device using the same | |
| JP5094068B2 (en) | Imaging apparatus and focus control method | |
| US20110169918A1 (en) | 3d image sensor and stereoscopic camera having the same | |
| US8854510B2 (en) | Image pickup apparatus, focus detection method, image generation method, and storage medium | |
| JP2010114731A (en) | Method for manufacturing camera module | |
| JP6053347B2 (en) | Imaging apparatus, control method therefor, and program | |
| US10795112B2 (en) | Focal plane shift measurement and adjustment in a lens assembly | |
| US9383540B1 (en) | System and method for controlling the focus of a lens | |
| CN105635564A (en) | Multiple camera apparatus and method for synchronized autofocus | |
| JP2018004691A5 (en) | ||
| JP2009250785A (en) | Imaging device | |
| CN107147858A (en) | Image processing apparatus and its control method | |
| US11233961B2 (en) | Image processing system for measuring depth and operating method of the same | |
| JP2015212772A (en) | Range-finding device, imaging apparatus, range-finding method, and range-finding parameter calculation method | |
| TWI512351B (en) | Multi-camera imaging system, and compensation method for image reconstruction | |
| JP6220986B2 (en) | Infrared image acquisition apparatus and infrared image acquisition method | |
| JP3940010B2 (en) | Image input device having automatic focusing function | |
| JPS6162011A (en) | Focus detector | |
| JP6504825B2 (en) | Imaging device, control method therefor, program, storage medium | |
| JP5953325B2 (en) | CCD camera device | |
| KR102641250B1 (en) | Calibration automation apparatus for thermal imaging camera | |
| TWI843475B (en) | Method and camera for correcting geometric aberrations in image recording | |
| US20100241385A1 (en) | Method and system for measurement and correction of thermally induced changes of boresight, effective focal length, and focus | |
| JP2023083980A (en) | Distance measuring device, distance detection art and imaging device |