WO2012164339A1 - Image stitching - Google Patents
Image stitching Download PDFInfo
- Publication number
- WO2012164339A1 WO2012164339A1 PCT/IB2011/052318 IB2011052318W WO2012164339A1 WO 2012164339 A1 WO2012164339 A1 WO 2012164339A1 IB 2011052318 W IB2011052318 W IB 2011052318W WO 2012164339 A1 WO2012164339 A1 WO 2012164339A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image sensor
- sensor portion
- images
- image
- capture region
- Prior art date
Links
- 238000004458 analytical method Methods 0.000 claims abstract description 75
- 238000004590 computer program Methods 0.000 claims abstract description 28
- 238000000034 method Methods 0.000 claims abstract description 28
- 230000003287 optical effect Effects 0.000 description 13
- 230000008859 change Effects 0.000 description 7
- 230000006641 stabilisation Effects 0.000 description 6
- 238000011105 stabilization Methods 0.000 description 6
- 230000007246 mechanism Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2624—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects for obtaining an image which is composed of whole input images, e.g. splitscreen
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B13/00—Optical objectives specially designed for the purposes specified below
- G02B13/06—Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B17/00—Details of cameras or camera bodies; Accessories therefor
- G03B17/02—Bodies
- G03B17/17—Bodies with reflectors arranged in beam forming the photographic image, e.g. for reducing dimensions of camera
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B37/00—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
- G03B37/04—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe with cameras or projectors providing touching or overlapping fields of view
Definitions
- Embodiments of the present invention relate to image stitching. In particular, they relate to processing images for stitching.
- Image stitching is a process of combining multiple images with overlapping fields of view to produce a larger image (for example, a panoramic image).
- a method comprising: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic; determining, from the analysis of the first and second images, an overlapping capture region for the first image portion and the second image portion; and stitching the first and second images together using the overlapping capture region.
- an apparatus comprising: at least one processor; and at least one memory storing a computer program comprising instructions configured to, working with the at least one processor, cause the apparatus to perform at least the following: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic; determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
- an apparatus comprising: means for analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic; means for determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and means for stitching the first and second images together using the overlapping capture region.
- a non-transitory computer readable medium storing a computer program comprising instructions configured to, working with at least one processor, cause at least the following to be performed: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated depends upon at least one contextual characteristic; determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
- a method comprising: analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion; determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion; analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion; determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the third and fourth images together using the further overlapping capture region.
- an apparatus comprising: means for analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion; means for determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion; means for analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion; means for determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and means for stitching the third and fourth images together using the further overlapping capture region.
- an apparatus comprising: at least one processor; and at least one memory storing a computer program comprising instructions configured to, working with the at least one processor, cause the apparatus to perform at least the following: analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion; determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion; analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion; determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the third and fourth images together using the further overlapping capture region.
- a non-transitory computer readable medium storing a computer program comprising instructions configured to, working with at least one processor, cause at least the following to be performed: analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion; determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion; analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion; determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the third and fourth images together using the further overlapping capture region.
- a method comprising: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion; determining, from the analysis of the first and second images, an overlapping capture region for the first image portion and the second image portion; and stitching the first and second images together using the overlapping capture region.
- an apparatus comprising: at least one processor; and at least one memory storing a computer program comprising instructions configured to, working with the at least one processor, cause the apparatus to perform at least the following: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion; determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
- an apparatus comprising: means for analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion; means for determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and means for stitching the first and second images together using the overlapping capture region.
- a non-transitory computer readable medium storing a computer program comprising instructions configured to, working with at least one processor, cause at least the following to be performed: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion; determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
- Fig. 1 illustrates a schematic of an apparatus
- Fig. 2 illustrates a schematic of a further apparatus
- Fig. 3A illustrates a perspective cross-sectional view of a first example of the further apparatus
- Fig. 3B illustrates a first and second image sensor portions for use in a second example of the further apparatus
- Figs 3C and 3D illustrate the second example of the further apparatus
- Fig. 4 illustrates first and second image sensors
- Fig. 5 illustrates a flow chart of a first method
- Fig. 6 illustrates a flow chart of a second method.
- Embodiments of the invention relate to image stitching.
- Embodiments of the invention may, for example, enable relatively fast image stitching. This may be particularly advantageously for a camera in viewfinder or video recording mode that is providing/recording a panoramic video image which comprises multiple frames per second (for instance, 30 frames per second).
- the figures illustrate an apparatus 10/20, comprising: at least one processor 12; and at least one memory 14 storing a computer program 16 comprising instructions 18 configured to, working with the at least one processor 12, cause the apparatus 10/20 to perform at least the following: analyzing first and second images, the first image being captured by a first image sensor portion 21 and the second image being captured by a second image sensor portion 22, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic; determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion 21 and the second image sensor portion 22; and stitching the first and second images together using the overlapping capture region.
- Fig. 1 illustrates an apparatus 10 comprising at least one processor 12 and at least one memory 14.
- the apparatus 10 may, for example, be a chip or a chipset. Although a single processor 12 and a single memory 14 are illustrated in Fig. 1 , in some implementations of the invention more than one processor 12 and/or more than one memory 14 is provided.
- the processor 12 is configured to read from and write to the memory 14.
- the processor 12 may also comprise an output interface via which data and/or commands are output by the processor 12 and an input interface via which data and/or commands are input to the processor 12.
- the memory 14 stores a computer program 16 comprising computer program instructions 18 that control the operation of the apparatus 10 when loaded into the processor 12.
- the computer program instructions 18 provide the logic and routines that enables the apparatus 10/20 to perform the methods illustrated in Fig. 5 and 6.
- the processor 12 by reading the memory 14 is able to load and execute the computer program 16.
- the computer program 16 may arrive at the apparatus 10/20 via any suitable delivery mechanism 40.
- the delivery mechanism 40 may be, for example, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a compact disc read-only memory (CD-ROM) or digital versatile disc (DVD), an article of manufacture that tangibly embodies the computer program 16.
- the delivery mechanism 40 may be a signal configured to reliably transfer the computer program 16.
- the apparatus 10/20 may propagate or transmit the computer program 16 as a computer data signal.
- Fig. 2 illustrates a further apparatus 20.
- the apparatus 20 may, for example, be a camera.
- the apparatus 20 may be hand portable and may have further functionality.
- the apparatus 20 may be configured to operate as a mobile telephone, a tablet computer, a games console and/or a portable music player.
- the apparatus 20 illustrated in Fig. 2 comprises a first image sensor portion 21 , a second image sensor portion 22, a display 24 and the apparatus 10 illustrated in Fig. 1 .
- the elements 12, 14, 21 , 22 and 24 are operationally coupled and any number or combination of intervening elements can exist (including no intervening elements)
- first image sensor portion 21 and second image sensor portion 22 are separate image sensors. In other embodiments, the first image sensor portion 21 and the second image sensor portion 22 are different portions of a single image sensor.
- Each of the image sensor portions 21 , 22 is configured to convert an optical image into an electrical signal.
- the image sensor portions 21 , 22 may be any type of image sensor portions. For example, in some implementations, they may be charge coupled devices (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor portions.
- CCD charge coupled devices
- CMOS complementary metal-oxide-semiconductor
- the apparatus 20 may further comprise at least one optical arrangement that is configured to convey light to the image sensor portions 21 , 22 to enable them to capture images.
- the image sensor portions 21 , 22 and/or the optical arrangement(s) are arranged such that the image sensor portions 21 , 22 have a partially overlapping field of view. That is, when the image sensor portions 21 , 22 capture images substantially simultaneously, some but not all of the content in the image captured by the first image sensor portion 21 will be present in the image captured by the second image sensor portion 22, and vice-versa.
- the display 24 is configured to display images captured by the first and second image sensor portions 21 , 22.
- the display 24 may be any type of display.
- the display 24 may be a liquid crystal display (LCD) or an organic light emitting diode (OLED) display.
- the display 24 is a touch sensitive display that is configured to receive user input.
- the memory 14 is illustrated as storing overlap data 24, environment data 28 and mode data 29. This will be described in further detail below.
- Fig. 3A illustrates a perspective cross sectional view of one example of a portion of the apparatus 20.
- the first and second image sensor portions 21 , 22 are separate image sensors.
- the apparatus 20 includes a housing 109 that houses the first and second image sensor portions 21 , 22 and two optical arrangements 31 , 32.
- the portion of the apparatus 20 illustrated in Fig. 3 can be considered to be in two parts a, b.
- the first part a includes the first image sensor portion 21 and the first optical arrangement 31 .
- the first optical arrangement 31 comprises a prism 1 13a and a plurality of lenses 1 14a.
- the second part b includes the second image sensor portion 22 and the second optical arrangement 32.
- the second optical arrangement 32 comprises a prism 1 13b and a plurality of lenses 1 14b. In operation, light is bent by the prism 1 13a and conveyed to the first image sensor portion 21 by the lenses 1 14b. Similarly, light is bent by the prism 1 13b and conveyed to the second image sensor portion 22 by the lenses 1 14b. Fig.
- FIG. 3A illustrates a first viewing cone 1 10a that represents the field of view of the first image sensor portion 21 and a second viewing cone 1 10b that represents the field of view of the second image sensor portion 22.
- the fields of view of the first and second image sensor portions 21 , 22 overlap.
- the volume of overlap is designated by the reference numeral 70 in Fig. 3, and means that when images are captured substantially simultaneously by the first and second image sensor portions 21 , 22, some of the same image data (that is, the same image content) will be captured by both the first image sensor portion 21 and the second image sensor portion 22.
- Figs 3B to 3D illustrate a different example of the apparatus 20 to that shown in Fig. 3A.
- Fig. 3B illustrates a first image sensor portion 21 and a second image sensor portion 22 mounted on a printed wiring board 48.
- the first and second image sensor portions 21 , 22 are different image sensors.
- the first and second image sensor portions 21 , 22 are different portions of a single image sensor.
- Figs. 3C and 3D illustrate the first and second image sensor portions 21 and 22 in position within the apparatus 20.
- Light enters the apparatus 20 through one or more apertures in a housing of the apparatus 20 and is directed by a prism/mirror 50 to first and second optical devices 52, 54.
- the optical devices 52, 54 are substantially shaped as pentaprisms 52, 54.
- Each optical device 52, 54 has an upper surface, a lower surface and five sides.
- Each optical device 52, 54 changes the direction of incident light through substantially 90° and directs it towards its respective image sensor portion 21 , 22. This is illustrated by the light rays 62, 64 in Fig. 3D.
- Each optical device 52, 54 also focuses light upon the image sensor portions 21 , 22.
- the image sensor portions 21 , 22 and the printed wiring board 48 may be movable by a mechanism 49 for image focusing.
- the arrows 68, 69 indicate the extent of movement of the image sensor portions 21 , 22 and the printed wiring board 48.
- the image sensor portions 21 , 22 in the Fig. 3B-3D example have the same overlapping field of view as that described above in relation to the Fig. 3A example.
- the first and second image sensor portions 21 , 22 can be considered to have an "overlapping capture region" which includes a subset of the pixels of the first image sensor portion 21 and a subset of the pixels of the second image sensor portion 22.
- the overlapping capture region may include: i) some, but not necessarily all, of the pixels of the first image sensor portion 21 that are used to capture image content which is also captured by the second image sensor portion 22, and ii) some, but not necessarily all, of the pixels of the second image sensor portion 22 which are used to capture image content that is also captured by the first image sensor portion 21.
- Fig. 4 illustrates an example of a schematic of the first and second image sensor portions 21 , 22.
- each of the first and second image sensor portions 21 , 22 comprises ten columns C1 - C10 and ten rows R1 - R10 of pixels.
- An overlapping capture region could, by way of example, comprise columns C1 and C2 of the first image sensor portion 21 and columns C9 and C10 of the second image sensor portion 22.
- Data indicating a theoretical overlapping capture region may be stored in overlap data 26 in the memory 14 of the apparatus 10/20.
- the theoretical overlapping capture region may indicate, for particular theoretical circumstances, pixels of the first image sensor portion 21 and the second image sensor portion 22 that are expected to be used to capture to image content that is present in both the first and second images.
- the actual overlapping capture region may, however, vary from the theoretical overlapping capture region depending upon one or more contextual characteristics. That is, the actual overlapping capture region can vary in dependence upon the context in which images are captured by the apparatus 20.
- the one or more contextual characteristic(s) may be time variable.
- the contextual characteristic(s) could relate to the surrounding environment of the apparatus 20.
- the contextual characteristic(s) could include ambient temperature and ambient pressure. A change in ambient temperature may cause thermal expansion of components of the apparatus 20 and a change in the overlapping capture region. A change in pressure could also cause a change in the overlapping capture region.
- the contextual characteristic(s) could include or relate to the operational mode of the apparatus 20. For example, changes to capture resolution, focal length and image stabilization could cause the overlapping capture region to change.
- the apparatus 20 may have a fixed capture resolution and/or a fixed focal length and may not have image stabilization. Nevertheless, changes to one or more contextual characteristics such as ambient temperature and/or pressure may cause the overlapping capture region to change.
- FIGs. 5 and 6 illustrate flow charts in accordance with embodiments of the invention. A first group of embodiments of the invention will now be described with reference to Fig. 5.
- the memory 14 stores a look up table that indicates how the overlapping capture region is expected to vary in dependence upon one or more contextual characteristics. For example, multiple different overlapping capture regions may be stored in the overlap data 26, each which is associated with different environmental characteristics (for example, different temperatures and/or pressures) stored in the environment data 28 and/or different operational modes (for example, capture resolutions, focal lengths and image stabilization positions/states) in the mode data 29.
- the look up table might, for example, be based on theoretical calculations or empirical data determined post-manufacture.
- the different overlapping capture regions stored in the look up table can be considered to be data that varies with the contextual characteristic(s).
- the processor 12 analyzes a first image captured by the first image sensor portion 21 and a second image captured by the second image sensor portion 22. At least one position on the first and second images at which the analysis of the first and second images is initiated depends upon at least one contextual characteristic.
- the analysis performed by the processor 12 is an image registration process in which the processor 12 compares the first image with the second image to determine content that is present in both images.
- the analysis performed by the processor 12 may, for example, comprise comparing the sum of the absolute differences between data in the first image and data in the second image.
- the purpose of the analysis is to enable the processor 12 to stitch the first and second images together accurately to create a larger image (such as a panoramic image).
- the processor 12 may, for example, determine one or more current contextual characteristics (using inputs from an appropriate sensor) such as the current temperature, the current atmospheric pressure, the current capture resolution, the current focal length and/or the current state/position of the image stabilization. The processor 12 may then compare that information with the stored environment data 28 and/or the stored mode data 29 to retrieve an associated stored overlapping capture region from the overlap data 26.
- current contextual characteristics such as the current temperature, the current atmospheric pressure, the current capture resolution, the current focal length and/or the current state/position of the image stabilization.
- the processor 12 may then compare that information with the stored environment data 28 and/or the stored mode data 29 to retrieve an associated stored overlapping capture region from the overlap data 26.
- the processor 12 analyzes the first and second images in block 43 of Fig. 5 using the stored overlapping capture region retrieved from the overlap data 26 in the memory 14.
- the stored overlapping capture region indicates where the actual overlapping capture region is expected to be, given the apparatus characteristic(s) that has/have been sensed.
- the processor 12 may, for example, initiate its analysis at a position on each of the first and second images that corresponds with a pixel regions which, from the stored overlapping capture region, is expected to include content that is also present in the other image. That is, the processor 12 may use the stored overlapping capture region as a "starting point" for analyzing the first and second images to determine an empirical/actual overlapping capture region for use in stitching the first and second images together.
- the processor 12 determines an actual/empirical overlapping capture region for the first image sensor portion 21 and the second image sensor portion 22, from its analysis of the first and second images.
- the empirical overlapping capture region may comprise a first plurality of pixels of the first image sensor portion 21 and a second plurality of pixels of the second image sensor portion 22. It will be appreciated by those skilled in the art that an empirically determined capture region may not be rectilinear or may not have a rectilinear edge. It could, for example, be a jagged line on an image sensor portion 21 , 22 or be a larger area on an image sensor portion 21 , 22 with a jagged edge.
- the processor 12 stitches the first and second images together using the actual/empirical overlapping capture region determined in block 44 of Fig. 5 to create a larger image (such as a panoramic image).
- the processor 12 may, for example, control the display 24 to display the created image.
- the embodiments of the invention described above enable the processor 12 to align and stitch images together quickly. This is particularly advantageous when the apparatus 20 is a camera in viewfinder or video recording mode, where multiple images are being stitched and displayed every second.
- a user provides user input that causes the processor 12 to control the first image sensor portion 21 to capture a first image and the second image sensor 22 to capture a second image.
- the first and second images may be captured substantially simultaneously, and can be considered to be a "set of images".
- the processor 12 analyzes the set of images captured by the first and second image sensor portions 21 , 22 in order to determine an "overlapping capture region" for the first and second image sensor portions 21 , 22, in respect of the first and second images, to enable the processor 12 to stitch the first and second images together accurately to create a larger image (such as a panoramic image).
- the analysis performed by the processor 12 is an image registration process in which the processor 12 compares the first image with the second image to determine content that is present in both images.
- the analysis performed by the processor 12 may, for example, comprise comparing the sum of the absolute differences between data in the first image and data in the second image.
- the processor 12 may use the overlap data 26 stored in the memory to perform the analysis.
- the overlap data 26 stored in the memory 14 may comprise data that indicates the theoretical overlapping capture region for the first image sensor portion 21 and the second image sensor portion 22.
- the processor 12 may use this information by initiating its analysis in block 41 of Fig. 6 at positions on the first and second images that correspond with pixels in the theoretical overlapping capture region, to determine content that is present in both the first image and the second image. That is, the processor 12 uses the theoretical overlapping capture region as a "starting point" for analyzing the first and second images to determine an empirical overlapping capture region for use in stitching the first and second images together.
- the processor 12 determines, from its analysis of the first and second images, an empirical overlapping capture region for the first image sensor portion 21 and the second image sensor portion 22.
- the overlapping capture region may be stored, at least temporarily, as part of the overlap data 26 in the memory 14.
- an overlapping capture region that has been empirically determined at block 42 of Fig. 6 as described above can be considered to depend upon at least one contextual characteristic.
- an empirically determined overlapping capture region may vary in dependence upon current environmental characteristics (for example, temperature and/or pressure) and/or the current operational mode (for example, capture resolution, focal length and image stabilization state/position) of the apparatus 20 when the first and second images are captured, for the reasons given above.
- the overlapping capture region determined in block 42 of Fig. 6 enables the processor 12 to align the first image with the second image and stitch them together accurately to create a larger image (such as a panoramic image).
- the processor 12 may, for example, control the display 24 to display the created image.
- the first image sensor portion 21 and the second image sensor portion 22 may be used to capture third and fourth images respectively.
- the processor 12 analyzes the third and fourth images in dependence upon the overlapping capture region determined in the analysis at block 42 of Fig. 6 and stored in the overlap data 26.
- the analysis is performed in order to determine a further overlapping capture region for first and second image sensor portions 21 , 22, in respect of the third and fourth images, to enable the processor 12 to stitch the third and fourth images together accurately to create a larger image (such as a panoramic image).
- the analysis performed by the processor 12 may be the same image registration process described above in relation to the first and second images, but in this instance for the third and fourth images.
- the processor 12 may compare the third and fourth images to determine content that is present in both images.
- the processor 12 may, for example, initiate its analysis at a position that depends upon the overlapping capture region determined in block 42 of Fig. 6. That is, the processor 12 may begin by analyzing areas of the third and fourth images that correspond with pixels in the previously determined empirical overlapping capture region, to determine content that is present in the both the third image and the fourth image.
- the previously determined empirical overlapping capture region therefore provides a "starting point" for analyzing the first and second images to determine a further empirical overlapping capture region for use in stitching the third and fourth images together.
- the analysis of the third and fourth images in block 43 of Fig. 6 can, in some circumstances, be considered to depend upon at least one contextual characteristic. This is because the previously determined empirical overlapping capture region that is used to analyze the third and fourth images may depend on at least one contextual characteristic.
- the processor 12 determines a further empirical overlapping capture region and, in block 45, it uses the further empirically determined capture region to stitch the third and fourth images together to create a larger image.
- the processor 12 may control the display 24 to display the larger image.
- the embodiments of the invention described above enable the processor 12 to align and stitch images together quickly. This is particularly advantageous when the apparatus 20 is a camera in viewfinder or video recording mode, where multiple images are being stitched and displayed every second.
- a previously determined empirical overlapping capture region forms the starting point for determining a further empirical overlapping capture region in block 44 of Fig. 6.
- the further empirical overlapping capture region could depend upon a previously determined empirical overlapping capture region and a theoretical overlapping capture region (for example, it could be an average of the two).
- the overlapping capture region could be or depend upon an average overlapping capture region that has been determined from the analysis of a plurality of sets of images, where each set of images comprises an image captured by the first image sensor portion 21 and an image captured by the second image sensor portion 22.
- the processor 12 may be configured to store an empirical overlapping capture region in the overlap data 26 and associate it with a particular temperature or temperature range stored in environment data 28 in the memory 14.
- the apparatus 20 may also be provided with a temperature sensor and/or a pressure sensor that provide(s) inputs to the processor 12.
- the processor 12 may be configured to retrieve a previously determined empirical overlapping capture region that corresponds with a sensed temperature and/or a sensed pressure, for use in analyzing images captured by the first and second images sensor portions 21 , 22.
- the processor 12 may also be configured to store an empirical overlapping capture region in the overlap data 26 and associate it with an operational mode stored in mode data 29 in the memory 14.
- the stored operational mode may, for example, identify an image resolution, focal length and/or an image stabilization state/position.
- the processor 12 may retrieve a previously determined empirical overlapping capture region that corresponds with that operational mode, for use in analyzing images captured by the first and second image sensor portions 21 , 22.
- references to 'computer-readable storage medium', 'computer program product', 'tangibly embodied computer program' etc. or a 'computer', 'processor' etc. should be understood to encompass not only computers having different architectures such as single/multi-processor architectures and sequential (Von Neumann)/parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGA), application specific circuits (ASIC), signal processing devices and other processing circuitry.
- References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device whether instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device etc.
- the term 'circuitry' refers to all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and/or digital circuitry) and
- circuits such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present.
- circuitry would also cover an implementation of merely a processor (or multiple processors) or portion of a processor and its (or their) accompanying software and/or firmware.
- circuitry would also cover, for example and if applicable to the particular claim element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in server, a cellular network device, or other network device.
- the blocks illustrated in Figs. 5 and 6 may represent steps in a method and/or sections of code in the computer program 16.
- the illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some blocks to be omitted.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
- Image Input (AREA)
Abstract
An apparatus, a method and a computer program are provided. The method comprises: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic; determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
Description
TITLE
Image Stitching TECHNOLOGICAL FIELD
Embodiments of the present invention relate to image stitching. In particular, they relate to processing images for stitching.
BACKGROUND
Image stitching is a process of combining multiple images with overlapping fields of view to produce a larger image (for example, a panoramic image).
BRIEF SUMMARY
According to some, but not necessarily all, embodiments of the invention, there is provided a method, comprising: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic; determining, from the analysis of the first and second images, an overlapping capture region for the first image portion and the second image portion; and stitching the first and second images together using the overlapping capture region.
According to some, but not necessarily all, embodiments of the invention, there is provided an apparatus, comprising: at least one processor; and at least one memory storing a computer program comprising instructions configured to, working with the at least one processor, cause the apparatus to perform at least the following: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic;
determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region. According to some, but not necessarily all, embodiments of the invention, there is provided an apparatus, comprising: means for analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic; means for determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and means for stitching the first and second images together using the overlapping capture region. According to some, but not necessarily all, embodiments of the invention, there is provided a non-transitory computer readable medium storing a computer program comprising instructions configured to, working with at least one processor, cause at least the following to be performed: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated depends upon at least one contextual characteristic; determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
According to some, but not necessarily all, embodiments of the invention, there is provided a method, comprising: analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion; determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion; analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion;
determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the third and fourth images together using the further overlapping capture region.
According to some, but not necessarily all, embodiments of the invention, there is provided an apparatus, comprising: means for analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion; means for determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion; means for analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion; means for determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and means for stitching the third and fourth images together using the further overlapping capture region. According to some, but not necessarily all, embodiments of the invention, there is provided an apparatus, comprising: at least one processor; and at least one memory storing a computer program comprising instructions configured to, working with the at least one processor, cause the apparatus to perform at least the following: analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion; determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion; analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion; determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the third and fourth images together using the further overlapping capture region.
According to some, but not necessarily all, embodiments of the invention, there is provided a non-transitory computer readable medium storing a computer program comprising instructions configured to, working with at least one processor, cause at least the following to be performed: analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion; determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion; analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion; determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the third and fourth images together using the further overlapping capture region.
According to some, but not necessarily all, embodiments of the invention, there is provided a method, comprising: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion; determining, from the analysis of the first and second images, an overlapping capture region for the first image portion and the second image portion; and stitching the first and second images together using the overlapping capture region.
According to some, but not necessarily all, embodiments of the invention, there is provided an apparatus, comprising: at least one processor; and at least one memory storing a computer program comprising instructions configured to, working with the at least one processor, cause the apparatus to perform at least the following: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion; determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
According to some, but not necessarily all, embodiments of the invention, there is provided an apparatus, comprising: means for analyzing first and second images, the
first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion; means for determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and means for stitching the first and second images together using the overlapping capture region.
According to some, but not necessarily all, embodiments of the invention, there is provided a non-transitory computer readable medium storing a computer program comprising instructions configured to, working with at least one processor, cause at least the following to be performed: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion; determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
BRIEF DESCRIPTION
For a better understanding of various examples of embodiments of the present invention reference will now be made by way of example only to the accompanying drawings in which:
Fig. 1 illustrates a schematic of an apparatus;
Fig. 2 illustrates a schematic of a further apparatus;
Fig. 3A illustrates a perspective cross-sectional view of a first example of the further apparatus;
Fig. 3B illustrates a first and second image sensor portions for use in a second example of the further apparatus;
Figs 3C and 3D illustrate the second example of the further apparatus;
Fig. 4 illustrates first and second image sensors;
Fig. 5 illustrates a flow chart of a first method; and
Fig. 6 illustrates a flow chart of a second method.
DETAILED DESCRIPTION
Embodiments of the invention relate to image stitching. Embodiments of the invention may, for example, enable relatively fast image stitching. This may be particularly advantageously for a camera in viewfinder or video recording mode that is providing/recording a panoramic video image which comprises multiple frames per second (for instance, 30 frames per second).
The figures illustrate an apparatus 10/20, comprising: at least one processor 12; and at least one memory 14 storing a computer program 16 comprising instructions 18 configured to, working with the at least one processor 12, cause the apparatus 10/20 to perform at least the following: analyzing first and second images, the first image being captured by a first image sensor portion 21 and the second image being captured by a second image sensor portion 22, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic; determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion 21 and the second image sensor portion 22; and stitching the first and second images together using the overlapping capture region.
Fig. 1 illustrates an apparatus 10 comprising at least one processor 12 and at least one memory 14. The apparatus 10 may, for example, be a chip or a chipset. Although a single processor 12 and a single memory 14 are illustrated in Fig. 1 , in some implementations of the invention more than one processor 12 and/or more than one memory 14 is provided. The processor 12 is configured to read from and write to the memory 14. The processor 12 may also comprise an output interface via which data and/or commands are output by the processor 12 and an input interface via which data and/or commands are input to the processor 12. Although the memory 14 is illustrated as a single component it may be implemented as one or more separate components some or all of which may be integrated/removable and/or may provide permanent/semi-permanent/ dynamic/cached storage.
The memory 14 stores a computer program 16 comprising computer program instructions 18 that control the operation of the apparatus 10 when loaded into the processor 12. The computer program instructions 18 provide the logic and routines that enables the apparatus 10/20 to perform the methods illustrated in Fig. 5 and 6. The processor 12 by reading the memory 14 is able to load and execute the computer program 16.
The computer program 16 may arrive at the apparatus 10/20 via any suitable delivery mechanism 40. The delivery mechanism 40 may be, for example, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a compact disc read-only memory (CD-ROM) or digital versatile disc (DVD), an article of manufacture that tangibly embodies the computer program 16. The delivery mechanism 40 may be a signal configured to reliably transfer the computer program 16. The apparatus 10/20 may propagate or transmit the computer program 16 as a computer data signal.
Fig. 2 illustrates a further apparatus 20. The apparatus 20 may, for example, be a camera. In some embodiments of the invention, the apparatus 20 may be hand portable and may have further functionality. For example, the apparatus 20 may be configured to operate as a mobile telephone, a tablet computer, a games console and/or a portable music player.
The apparatus 20 illustrated in Fig. 2 comprises a first image sensor portion 21 , a second image sensor portion 22, a display 24 and the apparatus 10 illustrated in Fig. 1 . The elements 12, 14, 21 , 22 and 24 are operationally coupled and any number or combination of intervening elements can exist (including no intervening elements)
In some embodiments, the first image sensor portion 21 and second image sensor portion 22 are separate image sensors. In other embodiments, the first image sensor portion 21 and the second image sensor portion 22 are different portions of a single image sensor.
Each of the image sensor portions 21 , 22 is configured to convert an optical image into an electrical signal. The image sensor portions 21 , 22 may be any type of image sensor portions. For example, in some implementations, they may be charge coupled
devices (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor portions.
The apparatus 20 may further comprise at least one optical arrangement that is configured to convey light to the image sensor portions 21 , 22 to enable them to capture images. The image sensor portions 21 , 22 and/or the optical arrangement(s) are arranged such that the image sensor portions 21 , 22 have a partially overlapping field of view. That is, when the image sensor portions 21 , 22 capture images substantially simultaneously, some but not all of the content in the image captured by the first image sensor portion 21 will be present in the image captured by the second image sensor portion 22, and vice-versa.
The display 24 is configured to display images captured by the first and second image sensor portions 21 , 22. The display 24 may be any type of display. For example, the display 24 may be a liquid crystal display (LCD) or an organic light emitting diode (OLED) display. In some embodiments, the display 24 is a touch sensitive display that is configured to receive user input.
In Fig. 2, the memory 14 is illustrated as storing overlap data 24, environment data 28 and mode data 29. This will be described in further detail below.
Fig. 3A illustrates a perspective cross sectional view of one example of a portion of the apparatus 20. In this example, the first and second image sensor portions 21 , 22 are separate image sensors. The apparatus 20 includes a housing 109 that houses the first and second image sensor portions 21 , 22 and two optical arrangements 31 , 32. The portion of the apparatus 20 illustrated in Fig. 3 can be considered to be in two parts a, b.
The first part a includes the first image sensor portion 21 and the first optical arrangement 31 . In this example, the first optical arrangement 31 comprises a prism 1 13a and a plurality of lenses 1 14a. The second part b includes the second image sensor portion 22 and the second optical arrangement 32. In this example, the second optical arrangement 32 comprises a prism 1 13b and a plurality of lenses 1 14b.
In operation, light is bent by the prism 1 13a and conveyed to the first image sensor portion 21 by the lenses 1 14b. Similarly, light is bent by the prism 1 13b and conveyed to the second image sensor portion 22 by the lenses 1 14b. Fig. 3A illustrates a first viewing cone 1 10a that represents the field of view of the first image sensor portion 21 and a second viewing cone 1 10b that represents the field of view of the second image sensor portion 22. The fields of view of the first and second image sensor portions 21 , 22 overlap. The volume of overlap is designated by the reference numeral 70 in Fig. 3, and means that when images are captured substantially simultaneously by the first and second image sensor portions 21 , 22, some of the same image data (that is, the same image content) will be captured by both the first image sensor portion 21 and the second image sensor portion 22.
Figs 3B to 3D illustrate a different example of the apparatus 20 to that shown in Fig. 3A. Fig. 3B illustrates a first image sensor portion 21 and a second image sensor portion 22 mounted on a printed wiring board 48. In the illustrated example, the first and second image sensor portions 21 , 22 are different image sensors. However, in some embodiments, the first and second image sensor portions 21 , 22 are different portions of a single image sensor.
Figs. 3C and 3D illustrate the first and second image sensor portions 21 and 22 in position within the apparatus 20. Light enters the apparatus 20 through one or more apertures in a housing of the apparatus 20 and is directed by a prism/mirror 50 to first and second optical devices 52, 54. The optical devices 52, 54 are substantially shaped as pentaprisms 52, 54. Each optical device 52, 54 has an upper surface, a lower surface and five sides. Each optical device 52, 54 changes the direction of incident light through substantially 90° and directs it towards its respective image sensor portion 21 , 22. This is illustrated by the light rays 62, 64 in Fig. 3D. Each optical device 52, 54 also focuses light upon the image sensor portions 21 , 22.
The image sensor portions 21 , 22 and the printed wiring board 48 may be movable by a mechanism 49 for image focusing. The arrows 68, 69 indicate the extent of movement of the image sensor portions 21 , 22 and the printed wiring board 48.
Although not explicitly shown in Figs. 3B to 3D, the image sensor portions 21 , 22 in the Fig. 3B-3D example have the same overlapping field of view as that described above in relation to the Fig. 3A example. In both the Fig. 3A and the Fig. 3B-3D examples, the first and second image sensor portions 21 , 22 can be considered to have an "overlapping capture region" which includes a subset of the pixels of the first image sensor portion 21 and a subset of the pixels of the second image sensor portion 22. For example, the overlapping capture region may include: i) some, but not necessarily all, of the pixels of the first image sensor portion 21 that are used to capture image content which is also captured by the second image sensor portion 22, and ii) some, but not necessarily all, of the pixels of the second image sensor portion 22 which are used to capture image content that is also captured by the first image sensor portion 21. Fig. 4 illustrates an example of a schematic of the first and second image sensor portions 21 , 22. In this example, each of the first and second image sensor portions 21 , 22 comprises ten columns C1 - C10 and ten rows R1 - R10 of pixels. An overlapping capture region could, by way of example, comprise columns C1 and C2 of the first image sensor portion 21 and columns C9 and C10 of the second image sensor portion 22.
Data indicating a theoretical overlapping capture region may be stored in overlap data 26 in the memory 14 of the apparatus 10/20. The theoretical overlapping capture region may indicate, for particular theoretical circumstances, pixels of the first image sensor portion 21 and the second image sensor portion 22 that are expected to be used to capture to image content that is present in both the first and second images.
The actual overlapping capture region may, however, vary from the theoretical overlapping capture region depending upon one or more contextual characteristics. That is, the actual overlapping capture region can vary in dependence upon the context in which images are captured by the apparatus 20. The one or more contextual characteristic(s) may be time variable.
The contextual characteristic(s) could relate to the surrounding environment of the apparatus 20. For example, the contextual characteristic(s) could include ambient temperature and ambient pressure. A change in ambient temperature may cause thermal expansion of components of the apparatus 20 and a change in the overlapping capture region. A change in pressure could also cause a change in the overlapping capture region.
The contextual characteristic(s) could include or relate to the operational mode of the apparatus 20. For example, changes to capture resolution, focal length and image stabilization could cause the overlapping capture region to change.
In some embodiments, the apparatus 20 may have a fixed capture resolution and/or a fixed focal length and may not have image stabilization. Nevertheless, changes to one or more contextual characteristics such as ambient temperature and/or pressure may cause the overlapping capture region to change.
Figs. 5 and 6 illustrate flow charts in accordance with embodiments of the invention. A first group of embodiments of the invention will now be described with reference to Fig. 5.
In this group of embodiments, the memory 14 stores a look up table that indicates how the overlapping capture region is expected to vary in dependence upon one or more contextual characteristics. For example, multiple different overlapping capture regions may be stored in the overlap data 26, each which is associated with different environmental characteristics (for example, different temperatures and/or pressures) stored in the environment data 28 and/or different operational modes (for example, capture resolutions, focal lengths and image stabilization positions/states) in the mode data 29. The look up table might, for example, be based on theoretical calculations or empirical data determined post-manufacture. The different overlapping capture regions stored in the look up table can be considered to be data that varies with the contextual characteristic(s).
At block 43 of Fig. 5, the processor 12 analyzes a first image captured by the first image sensor portion 21 and a second image captured by the second image sensor portion 22. At least one position on the first and second images at which the analysis of the first and second images is initiated depends upon at least one contextual characteristic.
The analysis performed by the processor 12 is an image registration process in which the processor 12 compares the first image with the second image to determine content that is present in both images. The analysis performed by the processor 12 may, for example, comprise comparing the sum of the absolute differences between data in the first image and data in the second image.
The purpose of the analysis is to enable the processor 12 to stitch the first and second images together accurately to create a larger image (such as a panoramic image).
Before, after or during the capture of the first and second images, the processor 12 may, for example, determine one or more current contextual characteristics (using inputs from an appropriate sensor) such as the current temperature, the current atmospheric pressure, the current capture resolution, the current focal length and/or the current state/position of the image stabilization. The processor 12 may then compare that information with the stored environment data 28 and/or the stored mode data 29 to retrieve an associated stored overlapping capture region from the overlap data 26.
The processor 12 analyzes the first and second images in block 43 of Fig. 5 using the stored overlapping capture region retrieved from the overlap data 26 in the memory 14. The stored overlapping capture region indicates where the actual overlapping capture region is expected to be, given the apparatus characteristic(s) that has/have been sensed.
The processor 12 may, for example, initiate its analysis at a position on each of the first and second images that corresponds with a pixel regions which, from the stored overlapping capture region, is expected to include content that is also present in the other image. That is, the processor 12 may use the stored overlapping capture region
as a "starting point" for analyzing the first and second images to determine an empirical/actual overlapping capture region for use in stitching the first and second images together. At block 44 of Fig. 5, the processor 12 determines an actual/empirical overlapping capture region for the first image sensor portion 21 and the second image sensor portion 22, from its analysis of the first and second images.
The empirical overlapping capture region may comprise a first plurality of pixels of the first image sensor portion 21 and a second plurality of pixels of the second image sensor portion 22. It will be appreciated by those skilled in the art that an empirically determined capture region may not be rectilinear or may not have a rectilinear edge. It could, for example, be a jagged line on an image sensor portion 21 , 22 or be a larger area on an image sensor portion 21 , 22 with a jagged edge.
At block 45 of Fig. 5, the processor 12 stitches the first and second images together using the actual/empirical overlapping capture region determined in block 44 of Fig. 5 to create a larger image (such as a panoramic image). The processor 12 may, for example, control the display 24 to display the created image.
The embodiments of the invention described above enable the processor 12 to align and stitch images together quickly. This is particularly advantageous when the apparatus 20 is a camera in viewfinder or video recording mode, where multiple images are being stitched and displayed every second.
A second group of embodiments of the invention will now be described with reference to Fig. 6.
In this example, a user provides user input that causes the processor 12 to control the first image sensor portion 21 to capture a first image and the second image sensor 22 to capture a second image. The first and second images may be captured substantially simultaneously, and can be considered to be a "set of images".
At block 41 of Fig. 6, the processor 12 analyzes the set of images captured by the first and second image sensor portions 21 , 22 in order to determine an "overlapping
capture region" for the first and second image sensor portions 21 , 22, in respect of the first and second images, to enable the processor 12 to stitch the first and second images together accurately to create a larger image (such as a panoramic image). The analysis performed by the processor 12 is an image registration process in which the processor 12 compares the first image with the second image to determine content that is present in both images. The analysis performed by the processor 12 may, for example, comprise comparing the sum of the absolute differences between data in the first image and data in the second image.
The processor 12 may use the overlap data 26 stored in the memory to perform the analysis. For example, the overlap data 26 stored in the memory 14 may comprise data that indicates the theoretical overlapping capture region for the first image sensor portion 21 and the second image sensor portion 22.
The processor 12 may use this information by initiating its analysis in block 41 of Fig. 6 at positions on the first and second images that correspond with pixels in the theoretical overlapping capture region, to determine content that is present in both the first image and the second image. That is, the processor 12 uses the theoretical overlapping capture region as a "starting point" for analyzing the first and second images to determine an empirical overlapping capture region for use in stitching the first and second images together.
At block 42 of Fig. 6, the processor 12 determines, from its analysis of the first and second images, an empirical overlapping capture region for the first image sensor portion 21 and the second image sensor portion 22. The overlapping capture region may be stored, at least temporarily, as part of the overlap data 26 in the memory 14.
In some circumstances, an overlapping capture region that has been empirically determined at block 42 of Fig. 6 as described above can be considered to depend upon at least one contextual characteristic. For instance, an empirically determined overlapping capture region may vary in dependence upon current environmental characteristics (for example, temperature and/or pressure) and/or the current operational mode (for example, capture resolution, focal length and image
stabilization state/position) of the apparatus 20 when the first and second images are captured, for the reasons given above.
The overlapping capture region determined in block 42 of Fig. 6 enables the processor 12 to align the first image with the second image and stitch them together accurately to create a larger image (such as a panoramic image). The processor 12 may, for example, control the display 24 to display the created image.
Before, during or after the overlapping capture region for the first and second image sensor portions 21 , 22 has been determined in block 42 of Fig. 6, the first image sensor portion 21 and the second image sensor portion 22 may be used to capture third and fourth images respectively.
At block 43 of Fig. 6, the processor 12 analyzes the third and fourth images in dependence upon the overlapping capture region determined in the analysis at block 42 of Fig. 6 and stored in the overlap data 26.
The analysis is performed in order to determine a further overlapping capture region for first and second image sensor portions 21 , 22, in respect of the third and fourth images, to enable the processor 12 to stitch the third and fourth images together accurately to create a larger image (such as a panoramic image).
The analysis performed by the processor 12 may be the same image registration process described above in relation to the first and second images, but in this instance for the third and fourth images. For example, the processor 12 may compare the third and fourth images to determine content that is present in both images.
The processor 12 may, for example, initiate its analysis at a position that depends upon the overlapping capture region determined in block 42 of Fig. 6. That is, the processor 12 may begin by analyzing areas of the third and fourth images that correspond with pixels in the previously determined empirical overlapping capture region, to determine content that is present in the both the third image and the fourth image. The previously determined empirical overlapping capture region therefore provides a "starting point" for analyzing the first and second images to determine a
further empirical overlapping capture region for use in stitching the third and fourth images together.
Use of the previously determined empirical overlapping capture region in analyzing the third and fourth images advantageously enables the processor 12 to determine the further empirical overlapping capture region quickly, because any change from the overlapping capture region is likely to be relatively small.
It will be appreciated by those skilled in the art that the analysis of the third and fourth images in block 43 of Fig. 6 can, in some circumstances, be considered to depend upon at least one contextual characteristic. This is because the previously determined empirical overlapping capture region that is used to analyze the third and fourth images may depend on at least one contextual characteristic. At block 44 of Fig. 6, the processor 12 determines a further empirical overlapping capture region and, in block 45, it uses the further empirically determined capture region to stitch the third and fourth images together to create a larger image. The processor 12 may control the display 24 to display the larger image. The embodiments of the invention described above enable the processor 12 to align and stitch images together quickly. This is particularly advantageous when the apparatus 20 is a camera in viewfinder or video recording mode, where multiple images are being stitched and displayed every second. In the example described above in relation to Fig 6, a previously determined empirical overlapping capture region forms the starting point for determining a further empirical overlapping capture region in block 44 of Fig. 6. Alternatively, the further empirical overlapping capture region could depend upon a previously determined empirical overlapping capture region and a theoretical overlapping capture region (for example, it could be an average of the two).
Alternatively, the overlapping capture region could be or depend upon an average overlapping capture region that has been determined from the analysis of a plurality of sets of images, where each set of images comprises an image captured by the
first image sensor portion 21 and an image captured by the second image sensor portion 22.
The processor 12 may be configured to store an empirical overlapping capture region in the overlap data 26 and associate it with a particular temperature or temperature range stored in environment data 28 in the memory 14. The apparatus 20 may also be provided with a temperature sensor and/or a pressure sensor that provide(s) inputs to the processor 12. The processor 12 may be configured to retrieve a previously determined empirical overlapping capture region that corresponds with a sensed temperature and/or a sensed pressure, for use in analyzing images captured by the first and second images sensor portions 21 , 22.
The processor 12 may also be configured to store an empirical overlapping capture region in the overlap data 26 and associate it with an operational mode stored in mode data 29 in the memory 14. The stored operational mode may, for example, identify an image resolution, focal length and/or an image stabilization state/position.
When the processor 12 subsequently determines that the apparatus 20 is in a particular operational mode, it may retrieve a previously determined empirical overlapping capture region that corresponds with that operational mode, for use in analyzing images captured by the first and second image sensor portions 21 , 22.
References to 'computer-readable storage medium', 'computer program product', 'tangibly embodied computer program' etc. or a 'computer', 'processor' etc. should be understood to encompass not only computers having different architectures such as single/multi-processor architectures and sequential (Von Neumann)/parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGA), application specific circuits (ASIC), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device whether instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device etc. As used in this application, the term 'circuitry' refers to all of the following:
(a) hardware-only circuit implementations (such as implementations in only analog and/or digital circuitry) and
(b) to combinations of circuits and software (and/or firmware), such as (as applicable): (i) to a combination of processor(s) or (ii) to portions of processor(s)/software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and
(c) to circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present.
This definition of 'circuitry' applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term "circuitry" would also cover an implementation of merely a processor (or multiple processors) or portion of a processor and its (or their) accompanying software and/or firmware. The term "circuitry" would also cover, for example and if applicable to the particular claim element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in server, a cellular network device, or other network device.
The blocks illustrated in Figs. 5 and 6 may represent steps in a method and/or sections of code in the computer program 16. The illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some blocks to be omitted.
Although embodiments of the present invention have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the invention as claimed. For example, it will be appreciated that it is not necessary for the apparatus 20 to have the configuration illustrated in Fig. 3A or Figs 3B to 3D.
It will, of course, be appreciated by those skilled in the art that the explanation of the image sensor portions 21 , 22 shown in Fig. 4 is provided merely for illustrative
purposes and that, in practice, the number of pixels provided in the first and second image sensor portions 21 , 22 may be far greater and may be of the order of millions.
Features described in the preceding description may be used in combinations other than the combinations explicitly described.
Although functions have been described with reference to certain features, those functions may be performable by other features whether described or not. Although features have been described with reference to certain embodiments, those features may also be present in other embodiments whether described or not.
Whilst endeavoring in the foregoing specification to draw attention to those features of the invention believed to be of particular importance it should be understood that the Applicant claims protection in respect of any patentable feature or combination of features hereinbefore referred to and/or shown in the drawings whether or not particular emphasis has been placed thereon.
l/we claim:
Claims
1 . A method, comprising:
analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic;
determining, from the analysis of the first and second images, an overlapping capture region for the first image portion and the second image portion; and
stitching the first and second images together using the overlapping capture region.
2. A method as claimed in claim 1 , wherein the first and second images are captured substantially simultaneously.
3. A method as claimed in claim 1 or 2, wherein the first image sensor portion comprises a first set of pixels, the second image sensor portion comprises a second set of pixels, and the overlapping capture region comprises: a subset of the first set of pixels, and a subset of the second set of pixels.
4. A method as claimed in claim 3, wherein the subset of the first set of pixels captured first image content as part of the first image, and the subset of the second set of pixels captured the first image content as part of the second image.
5. A method as claimed in any of the preceding claims, wherein the image sensor portion and the second image sensor portion are separate.
6. A method as claimed in any of the preceding claims, wherein the at least one contextual characteristic relates to an operational mode of an apparatus and/or a surrounding environment of an apparatus.
7. A method as claimed in any of the preceding claims, wherein the analysis is performed using data that varies with the at least one contextual characteristic.
8. A method as claimed in claim 7, wherein the data is an overlapping capture region for the first image sensor portion and the second image sensor portion that was previously determined by analyzing one or more sets of images, wherein a set of images comprises an image captured by the first image sensor portion and an image captured by the second image sensor portion.
9. A method as claimed in claim 8, wherein the previously determined overlapping capture region is or depends upon an average overlapping capture region determined from an analysis of a plurality of sets of images.
10. A method as claimed in any of the preceding claims, wherein the overlapping capture region depends upon a theoretical overlapping capture region for the first image sensor portion and the second image sensor portion.
1 1 . A method as claimed in any of the preceding claims, wherein the first and second images are captured using a camera in viewfinder mode or video recording mode.
12. A computer program comprising instructions that, when performed by at least one processor, cause the method as claimed in any of the preceding claims to be performed.
13. An apparatus comprising means for performing the method as claimed in one or more of claims 1 to 1 1 .
14. An apparatus, comprising:
at least one processor; and
at least one memory storing a computer program comprising instructions configured to, working with the at least one processor, cause the apparatus to perform at least the following:
analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic;
determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
15. An apparatus as claimed in claim 14, wherein the first and second images are captured substantially simultaneously.
16. An apparatus as claimed in claim 14 or 15, wherein the first image sensor portion comprises a first set of pixels, the second image sensor portion comprises a second set of pixels, and the overlapping capture region comprises: a subset of the first set of pixels, and a subset of the second set of pixels.
17. An apparatus as claimed in claim 16, wherein the subset of the first set of pixels captured first image content as part of the first image, and the subset of the second set of pixels captured the first image content as part of the second image.
18. An apparatus as claimed in any of claims 14 to 17, wherein the image sensor portion and the second image sensor portion are separate.
19. An apparatus as claimed in any of claims 14 to 18, wherein the at least one contextual characteristic relates to an operational mode of the apparatus and/or a surrounding environment of the apparatus.
20. An apparatus as claimed in any of claims 14 to 19, wherein the analysis is performed using data that varies with the at least one contextual characteristic.
21 . An apparatus as claimed in claim 20, wherein the data is an overlapping capture region for the first image sensor portion and the second image sensor portion that was previously determined by analyzing one or more sets of images, wherein a set of images comprises an image captured by the first image sensor portion and an image captured by the second image sensor portion.
22. An apparatus as claimed in claim 21 , wherein the previously determined overlapping capture region is or depends upon an average overlapping capture region determined from an analysis of a plurality of sets of images.
23. An apparatus as claimed in claim 14 to 22, wherein the overlapping capture region depends upon a theoretical overlapping capture region for the first image sensor and the second image sensor.
24. An apparatus as claimed in any of claims 14 to 23, wherein the first and second images are captured in viewfinder mode or video recording mode.
25. An apparatus as claimed in any of claims 14 to 24, wherein the apparatus is a camera that comprises the first image sensor and the second image sensor.
26. An apparatus, comprising:
means for analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated, depends upon at least one contextual characteristic;
means for determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and
means for stitching the first and second images together using the overlapping capture region.
27. A non-transitory computer readable medium storing a computer program comprising instructions configured to, working with at least one processor, cause at least the following to be performed:
analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion, wherein at least one position on the first and second images, at which the analysis of the first and second images is initiated depends upon at least one contextual characteristic;
determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
28. A non-transitory computer readable medium as claimed in claim 27, wherein the first and second images are captured substantially simultaneously.
29. A non-transitory computer readable medium as claimed in claim 27 or 28, wherein the first image sensor portion comprises a first set of pixels, the second image sensor portion comprises a second set of pixels, and the overlapping capture region comprises: a subset of the first set of pixels, and a subset of the second set of pixels.
30. A non-transitory computer readable medium as claimed in claim 29, wherein the subset of the first set of pixels captured first image content as part of the first image, and the subset of the second set of pixels captured the first image content as part of the second image.
31 . A non-transitory computer readable medium as claimed in any of claims 27 to 30, wherein the image sensor portion and the second image sensor portion are separate.
32. A non-transitory computer readable medium as claimed in any of claims 27 to 31 , wherein the at least one contextual characteristic relates to an operational mode of an apparatus and/or a surrounding environment of an apparatus.
33. A non-transitory computer readable medium as claimed in any of claims 27 to 32, wherein the analysis is performed using data that varies with the at least one contextual characteristic.
34. A non-transitory computer readable medium as claimed in claim 33, wherein the data is an overlapping capture region for the first image sensor portion and the second image sensor portion that was previously determined by analyzing one or more sets of images, wherein a set of images comprises an image captured by the first image sensor portion and an image captured by the second image sensor portion.
35. A non-transitory computer readable medium as claimed in claim 34, wherein the previously determined overlapping capture region is or depends upon an average overlapping capture region determined from an analysis of a plurality of sets of images.
36. A non-transitory computer readable medium as claimed in any of claims 27 to 35, wherein the overlapping capture region depends upon a theoretical overlapping capture region for the first image sensor portion and the second image sensor portion.
37. A method, comprising: analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion;
determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion;
analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion;
determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and
stitching the third and fourth images together using the further overlapping capture region.
38. An apparatus, comprising:
means for analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion;
means for determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion;
means for analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion;
means for determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and
means for stitching the third and fourth images together using the further overlapping capture region.
39. An apparatus, comprising:
at least one processor; and at least one memory storing a computer program comprising instructions configured to, working with the at least one processor, cause the apparatus to perform at least the following:
analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion;
determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion;
analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion;
determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and
stitching the third and fourth images together using the further overlapping capture region.
40. A non-transitory computer readable medium storing a computer program comprising instructions configured to, working with at least one processor, cause at least the following to be performed:
analyzing one or more sets of images, wherein a set of images comprises a first image captured by a first image sensor portion and a second image captured by a second image sensor portion;
determining, from the analysis, an overlapping capture region for the first image sensor portion and the second image sensor portion;
analyzing third and fourth images in dependence upon the overlapping capture region determined from the analysis of the one or more sets of images, the third image being captured by the first image sensor portion and the fourth image being captured by the second image sensor portion;
determining, from the analysis of the third and fourth images, a further overlapping capture region for the first image sensor portion and the second image sensor portion; and
stitching the third and fourth images together using the further overlapping capture region.
41 . A method, comprising:
analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion;
determining, from the analysis of the first and second images, an overlapping capture region for the first image portion and the second image portion; and
stitching the first and second images together using the overlapping capture region.
42. An apparatus, comprising:
at least one processor; and
at least one memory storing a computer program comprising instructions configured to, working with the at least one processor, cause the apparatus to perform at least the following:
analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion;
determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
43. An apparatus, comprising:
means for analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion;
means for determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and
means for stitching the first and second images together using the overlapping capture region.
44. A non-transitory computer readable medium storing a computer program comprising instructions configured to, working with at least one processor, cause at least the following to be performed: analyzing first and second images, the first image being captured by a first image sensor portion and the second image being captured by a second image sensor portion;
determining, from the analysis of the first and second images, an overlapping capture region for the first image sensor portion and the second image sensor portion; and stitching the first and second images together using the overlapping capture region.
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/IB2011/052318 WO2012164339A1 (en) | 2011-05-27 | 2011-05-27 | Image stitching |
EP11866497.8A EP2716033B1 (en) | 2011-05-27 | 2011-05-27 | Image stitching |
CN201180071175.6A CN103597810B (en) | 2011-05-27 | 2011-05-27 | Image stitching |
US14/119,469 US9390530B2 (en) | 2011-05-27 | 2011-05-27 | Image stitching |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/IB2011/052318 WO2012164339A1 (en) | 2011-05-27 | 2011-05-27 | Image stitching |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012164339A1 true WO2012164339A1 (en) | 2012-12-06 |
Family
ID=47258437
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2011/052318 WO2012164339A1 (en) | 2011-05-27 | 2011-05-27 | Image stitching |
Country Status (4)
Country | Link |
---|---|
US (1) | US9390530B2 (en) |
EP (1) | EP2716033B1 (en) |
CN (1) | CN103597810B (en) |
WO (1) | WO2012164339A1 (en) |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015195905A1 (en) * | 2014-06-20 | 2015-12-23 | Qualcomm Incorporated | Folded optic array camera using refractive prisms |
US9374516B2 (en) | 2014-04-04 | 2016-06-21 | Qualcomm Incorporated | Auto-focus in low-profile folded optics multi-camera system |
US9386222B2 (en) | 2014-06-20 | 2016-07-05 | Qualcomm Incorporated | Multi-camera system using folded optics free from parallax artifacts |
US9383550B2 (en) | 2014-04-04 | 2016-07-05 | Qualcomm Incorporated | Auto-focus in low-profile folded optics multi-camera system |
US9398264B2 (en) | 2012-10-19 | 2016-07-19 | Qualcomm Incorporated | Multi-camera system using folded optics |
US9438889B2 (en) | 2011-09-21 | 2016-09-06 | Qualcomm Incorporated | System and method for improving methods of manufacturing stereoscopic image sensors |
US9485495B2 (en) | 2010-08-09 | 2016-11-01 | Qualcomm Incorporated | Autofocus for stereo images |
US9549107B2 (en) | 2014-06-20 | 2017-01-17 | Qualcomm Incorporated | Autofocus for folded optic array cameras |
US9819863B2 (en) | 2014-06-20 | 2017-11-14 | Qualcomm Incorporated | Wide field of view array camera for hemispheric and spherical imaging |
US9832381B2 (en) | 2014-10-31 | 2017-11-28 | Qualcomm Incorporated | Optical image stabilization for thin cameras |
US10013764B2 (en) | 2014-06-19 | 2018-07-03 | Qualcomm Incorporated | Local adaptive histogram equalization |
US10084958B2 (en) | 2014-06-20 | 2018-09-25 | Qualcomm Incorporated | Multi-camera system using folded optics free from parallax and tilt artifacts |
US10178373B2 (en) | 2013-08-16 | 2019-01-08 | Qualcomm Incorporated | Stereo yaw correction using autofocus feedback |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI554976B (en) | 2014-11-17 | 2016-10-21 | 財團法人工業技術研究院 | Surveillance systems and image processing methods thereof |
WO2016173815A1 (en) * | 2015-04-28 | 2016-11-03 | Philips Lighting Holding B.V. | Metadata in multi image scenes |
CN105334692B (en) * | 2015-09-28 | 2018-06-29 | 联想(北京)有限公司 | Information processing method and electronic equipment |
CN105338244B (en) * | 2015-10-30 | 2019-04-16 | 努比亚技术有限公司 | A kind of information processing method and mobile terminal |
US9721371B2 (en) * | 2015-12-02 | 2017-08-01 | Caterpillar Inc. | Systems and methods for stitching metallographic and stereoscopic images |
TWI617195B (en) * | 2017-06-22 | 2018-03-01 | 宏碁股份有限公司 | Image capturing apparatus and image stitching method thereof |
CN109214983B (en) * | 2017-06-30 | 2022-12-13 | 宏碁股份有限公司 | Image acquisition device and image splicing method thereof |
CN109272876B (en) * | 2018-11-21 | 2020-12-11 | 威创集团股份有限公司 | Splicing display system |
US10628698B1 (en) * | 2019-07-02 | 2020-04-21 | Grundium Oy | Method for image stitching |
CN111007693B (en) * | 2019-12-17 | 2022-05-31 | 广州立景创新科技有限公司 | Wide-angle image pickup device |
CN112987149B (en) * | 2021-04-21 | 2023-01-20 | 广州立景创新科技有限公司 | Composite prism module and image acquisition module |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2005050567A1 (en) * | 2003-11-18 | 2005-06-02 | Scalado Ab | Method for processing a digital image and image representation format |
US20080170803A1 (en) * | 2007-01-12 | 2008-07-17 | Babak Forutanpour | Panoramic imaging techniques |
US20080247667A1 (en) * | 2007-04-05 | 2008-10-09 | Hailin Jin | Laying Out Multiple Images |
EP2017783A2 (en) * | 2007-07-18 | 2009-01-21 | Samsung Electronics Co., Ltd. | Method for constructing a composite image |
US20090051778A1 (en) * | 2007-08-21 | 2009-02-26 | Patrick Pan | Advanced dynamic stitching method for multi-lens camera system |
US20100194851A1 (en) * | 2009-02-03 | 2010-08-05 | Aricent Inc. | Panorama image stitching |
US20100289922A1 (en) * | 2006-05-29 | 2010-11-18 | Bit-Side Gmbh | Method and system for processing data sets of image sensors, a corresponding computer program, and a corresponding computer-readable storage medium |
US20100321470A1 (en) | 2009-06-22 | 2010-12-23 | Fujifilm Corporation | Imaging apparatus and control method therefor |
US20110141300A1 (en) * | 2009-12-11 | 2011-06-16 | Fotonation Ireland Limited | Panorama Imaging Using a Blending Map |
US20110317937A1 (en) * | 2010-06-28 | 2011-12-29 | Sony Corporation | Information processing apparatus, information processing method, and program therefor |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7006111B1 (en) * | 1999-11-09 | 2006-02-28 | Intel Corporation | Digital image stitching |
US6930703B1 (en) * | 2000-04-29 | 2005-08-16 | Hewlett-Packard Development Company, L.P. | Method and apparatus for automatically capturing a plurality of images during a pan |
US6788333B1 (en) | 2000-07-07 | 2004-09-07 | Microsoft Corporation | Panoramic video |
JP4297111B2 (en) | 2005-12-14 | 2009-07-15 | ソニー株式会社 | Imaging apparatus, image processing method and program thereof |
TW200740212A (en) * | 2006-04-10 | 2007-10-16 | Sony Taiwan Ltd | A stitching accuracy improvement method with lens distortion correction |
EP2016759A1 (en) * | 2006-05-05 | 2009-01-21 | Nokia Corporation | Optical image recording device with small height and high resolution |
CN101656840B (en) | 2008-08-22 | 2011-09-28 | 原相科技股份有限公司 | Wide-angle sensor array module and image correcting method, operating method and application thereof |
-
2011
- 2011-05-27 EP EP11866497.8A patent/EP2716033B1/en active Active
- 2011-05-27 WO PCT/IB2011/052318 patent/WO2012164339A1/en active Application Filing
- 2011-05-27 CN CN201180071175.6A patent/CN103597810B/en active Active
- 2011-05-27 US US14/119,469 patent/US9390530B2/en active Active
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2005050567A1 (en) * | 2003-11-18 | 2005-06-02 | Scalado Ab | Method for processing a digital image and image representation format |
US20100289922A1 (en) * | 2006-05-29 | 2010-11-18 | Bit-Side Gmbh | Method and system for processing data sets of image sensors, a corresponding computer program, and a corresponding computer-readable storage medium |
US20080170803A1 (en) * | 2007-01-12 | 2008-07-17 | Babak Forutanpour | Panoramic imaging techniques |
US20080247667A1 (en) * | 2007-04-05 | 2008-10-09 | Hailin Jin | Laying Out Multiple Images |
EP2017783A2 (en) * | 2007-07-18 | 2009-01-21 | Samsung Electronics Co., Ltd. | Method for constructing a composite image |
US20090051778A1 (en) * | 2007-08-21 | 2009-02-26 | Patrick Pan | Advanced dynamic stitching method for multi-lens camera system |
US20100194851A1 (en) * | 2009-02-03 | 2010-08-05 | Aricent Inc. | Panorama image stitching |
US20100321470A1 (en) | 2009-06-22 | 2010-12-23 | Fujifilm Corporation | Imaging apparatus and control method therefor |
US20110141300A1 (en) * | 2009-12-11 | 2011-06-16 | Fotonation Ireland Limited | Panorama Imaging Using a Blending Map |
US20110317937A1 (en) * | 2010-06-28 | 2011-12-29 | Sony Corporation | Information processing apparatus, information processing method, and program therefor |
Non-Patent Citations (1)
Title |
---|
See also references of EP2716033A4 |
Cited By (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9485495B2 (en) | 2010-08-09 | 2016-11-01 | Qualcomm Incorporated | Autofocus for stereo images |
US9438889B2 (en) | 2011-09-21 | 2016-09-06 | Qualcomm Incorporated | System and method for improving methods of manufacturing stereoscopic image sensors |
US10165183B2 (en) | 2012-10-19 | 2018-12-25 | Qualcomm Incorporated | Multi-camera system using folded optics |
US9838601B2 (en) | 2012-10-19 | 2017-12-05 | Qualcomm Incorporated | Multi-camera system using folded optics |
US9398264B2 (en) | 2012-10-19 | 2016-07-19 | Qualcomm Incorporated | Multi-camera system using folded optics |
US10178373B2 (en) | 2013-08-16 | 2019-01-08 | Qualcomm Incorporated | Stereo yaw correction using autofocus feedback |
US9860434B2 (en) | 2014-04-04 | 2018-01-02 | Qualcomm Incorporated | Auto-focus in low-profile folded optics multi-camera system |
US9383550B2 (en) | 2014-04-04 | 2016-07-05 | Qualcomm Incorporated | Auto-focus in low-profile folded optics multi-camera system |
US9374516B2 (en) | 2014-04-04 | 2016-06-21 | Qualcomm Incorporated | Auto-focus in low-profile folded optics multi-camera system |
US9973680B2 (en) | 2014-04-04 | 2018-05-15 | Qualcomm Incorporated | Auto-focus in low-profile folded optics multi-camera system |
US10013764B2 (en) | 2014-06-19 | 2018-07-03 | Qualcomm Incorporated | Local adaptive histogram equalization |
US9386222B2 (en) | 2014-06-20 | 2016-07-05 | Qualcomm Incorporated | Multi-camera system using folded optics free from parallax artifacts |
US9843723B2 (en) | 2014-06-20 | 2017-12-12 | Qualcomm Incorporated | Parallax free multi-camera system capable of capturing full spherical images |
US9854182B2 (en) | 2014-06-20 | 2017-12-26 | Qualcomm Incorporated | Folded optic array camera using refractive prisms |
WO2015195905A1 (en) * | 2014-06-20 | 2015-12-23 | Qualcomm Incorporated | Folded optic array camera using refractive prisms |
US9819863B2 (en) | 2014-06-20 | 2017-11-14 | Qualcomm Incorporated | Wide field of view array camera for hemispheric and spherical imaging |
US9733458B2 (en) | 2014-06-20 | 2017-08-15 | Qualcomm Incorporated | Multi-camera system using folded optics free from parallax artifacts |
US10084958B2 (en) | 2014-06-20 | 2018-09-25 | Qualcomm Incorporated | Multi-camera system using folded optics free from parallax and tilt artifacts |
US9541740B2 (en) | 2014-06-20 | 2017-01-10 | Qualcomm Incorporated | Folded optic array camera using refractive prisms |
US9549107B2 (en) | 2014-06-20 | 2017-01-17 | Qualcomm Incorporated | Autofocus for folded optic array cameras |
CN110430345A (en) * | 2014-06-20 | 2019-11-08 | 高通股份有限公司 | Use the folding optical array camera of refracting prisms |
CN110430345B (en) * | 2014-06-20 | 2021-07-06 | 高通股份有限公司 | Folded optical array camera using refractive prisms |
US9832381B2 (en) | 2014-10-31 | 2017-11-28 | Qualcomm Incorporated | Optical image stabilization for thin cameras |
Also Published As
Publication number | Publication date |
---|---|
EP2716033A1 (en) | 2014-04-09 |
US9390530B2 (en) | 2016-07-12 |
EP2716033B1 (en) | 2019-06-26 |
EP2716033A4 (en) | 2015-03-04 |
US20140111607A1 (en) | 2014-04-24 |
CN103597810B (en) | 2017-02-15 |
CN103597810A (en) | 2014-02-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9390530B2 (en) | Image stitching | |
US9473702B2 (en) | Controlling image capture and/or controlling image processing | |
EP2887642A2 (en) | Method, apparatus and computer program product for image refocusing for light-field images | |
US9563966B2 (en) | Image control method for defining images for waypoints along a trajectory | |
US9219857B2 (en) | Image capture | |
US9509902B1 (en) | Methods and devices for correction of camera module sensitivity and flash color variation | |
CN104361070B (en) | Save continuous photo storage method, photo restoring method and the system of memory space | |
US8228394B2 (en) | Apparatus and method for adjusting the display direction of an image captured by a camera system | |
US10063825B2 (en) | Method of operating mobile device and mobile system | |
CN110192385B (en) | Electronic device and control method thereof | |
US20140379757A1 (en) | Methods, apparatus and non-transitory computer readable storage mediums for organising and accessing image databases | |
US8274581B2 (en) | Digital image capture device and digital image processing method thereof | |
US20150015771A1 (en) | Image-capturing devices and methods | |
US10965877B2 (en) | Image generating method and electronic apparatus | |
US10782868B2 (en) | Image navigation | |
US20120162473A1 (en) | Electronic apparatus, image capturing apparatus and method thereof | |
US11257208B2 (en) | Defect inspection system for specimen and defect inspection method for specimen | |
US20150271413A1 (en) | Electronic device and method for viewing images thereof | |
US8928764B2 (en) | Method and device for correcting user's hand tremor in imaging device | |
US20170094190A1 (en) | Processing display of digital camera readout with minimal latency | |
US20110149066A1 (en) | Method and system for evaluating light uniformity through an optical lens | |
CN104427240A (en) | Electronic device and image adjustment method thereof | |
EP3059669A1 (en) | Controlling display of video content | |
WO2014122506A1 (en) | Image processing of sub -images of a plenoptic image | |
WO2015150619A1 (en) | Apparatus, method and computer program for obtaining images |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11866497 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14119469 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |