WO2018078222A1 - Reconstitution de couleur à vues multiples - Google Patents

Reconstitution de couleur à vues multiples Download PDF

Info

Publication number
WO2018078222A1
WO2018078222A1 PCT/FI2017/050744 FI2017050744W WO2018078222A1 WO 2018078222 A1 WO2018078222 A1 WO 2018078222A1 FI 2017050744 W FI2017050744 W FI 2017050744W WO 2018078222 A1 WO2018078222 A1 WO 2018078222A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixel
output
image data
input image
pixels
Prior art date
Application number
PCT/FI2017/050744
Other languages
English (en)
Inventor
Kimmo Roimela
Original Assignee
Nokia Technologies Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Technologies Oy filed Critical Nokia Technologies Oy
Publication of WO2018078222A1 publication Critical patent/WO2018078222A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/15—Processing image signals for colour aspects of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/257—Colour aspects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
    • H04N25/41—Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/74—Circuits for processing colour signals for obtaining special effects
    • H04N9/76—Circuits for processing colour signals for obtaining special effects for mixing of colour signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0077—Colour aspects

Definitions

  • This specification relates to the field of image processing, particularly that of colour reconstruction. More specifically, the specification relates to the colour reconstruction of image data describing multiple different perspectives of a scene.
  • Digital cameras typically utilise a monochrome image sensor overlaid with a colour filter array (e.g. RGB) to enable the capture of colour information.
  • the colour filter array is commonly arranged in a so-called Bayer pattern, where each group of 2x2 pixels on the image sensor receives one red, one blue, and two green pixels. This pattern is then reconstructed into a full-colour image at the resolution of the image sensor by estimating the intensity of the two missing colour channels for each pixel from the pixels in the
  • Colour reconstruction (e.g. that involved in "debayering") is non-trivial and sometimes results in colour artefacts, such as Moire patterns, in high-frequency details or aliasing at sharp edges.
  • an optical low-pass filter is typically placed in front of the colour filter array and the imaging sensor. However, while this may reduce the presence and extent of colour artefacts, imaging resolution is also reduced.
  • Multiple capture systems may differ from common digital cameras in that they have several imaging units (each comprising an image sensor overlaid with a colour filter array), each of which captures a separate colour image. These images can then be combined in post-processing into panoramic and/or stereo images.
  • this specification describes a method comprising, for each of plural constituent portions of input image data captured by at least one image sensor overlaid with a colour filter array, determining a depth value for each input image pixel in the constituent portion, wherein each constituent portion of the input image data has a different associated direction of capture and/or a different associated location of capture and represents a different view of a scene, and wherein each pixel of input image data is associated with a respective one of the colours of the colour filter array.
  • the method further comprises projecting pixels of the input image data into an output image perspective based on the determined depth values, the associated directions and/or locations of capture and information describing the output image perspective.
  • the method further comprises performing colour reconstruction based on the pixels of the input image data projected into the output image perspective.
  • Projecting the pixels of the input image data into the output image perspective may comprise, for each output pixel in the output image perspective, estimating an orientation in three- dimensions of a plane of a surface of an imaged object at the location of the output pixel based on the depth values associated with locations of neighbouring output pixels in the output image perspective, and projecting one or more non-occluded pixels of the input image data which contribute to the output pixel onto the plane.
  • Performing colour reconstruction based on the pixels of the input image data projected into the output image perspective may comprise colour-reconstructing each output pixel in the output image perspective based on the contributing pixels of the input image data which have been projected onto the plane estimated for that output pixel.
  • Colour-reconstructing each output pixel in the output image perspective may comprise, for each output pixel, computing a weighted average of the projected contributing pixels associated with each colour of the colour filter array.
  • Each contributing pixel may be given a weight that has an inverse relationship with a distance between the location of projection onto the plane of the contributing pixel and the location of the output pixel.
  • At least two of the plural constituent portions of input image data may have been captured with different rotational orientations relative to their respective direction of capture.
  • the method may be performed at the image capture system that captures the input image data.
  • this specification describes apparatus configured to perform any method as described with reference to the first aspect.
  • this specification describes an image capture system configured to perform any method as described with reference to the first aspect.
  • this specification describes computer-readable instructions which, when executed by computing apparatus, cause the computing apparatus to perform any method as described with reference to the first aspect.
  • this specification describes apparatus comprising at least one processor; and at least one memory including computer program code which, when executed by the at least one processor, causes the apparatus: for each of plural constituent portions of input image data captured by at least one image sensor overlaid with a colour filter array, to determine a depth value for each input image pixel in the constituent portion, wherein each constituent portion of the input image data has a different associated direction of capture and/or a different associated location of capture and represents a different view of a scene, and wherein each pixel of input image data is associated with a respective one of the colours of the colour filter array; to project pixels of the input image data into an output image perspective based on the determined depth values, the associated directions and/ or locations of capture and information describing the output image perspective; and to perform colour reconstruction based on the pixels of the input image data projected into the output image perspective.
  • Causing the apparatus to project pixels of the input image data into the output image perspective may comprise causing the apparatus to associate each pixel in the output image perspective with one of the determined depth values.
  • Causing the apparatus to associate each pixel in the output image perspective with one of the determined depth value may comprise causing the apparatus to project the determined depth values into three- dimensional space based on the associated directions and/or locations of capture and to project the projected depth values into the output image perspective such that each pixel location in the output image perspective is associated with a single depth value.
  • Causing the apparatus to project the projected depth values into the output image perspective may comprise causing the apparatus to respond to an instance of more than one of the projected depth values corresponding with a pixel location in the output image perspective by associating only the depth value indicating the shortest depth with the pixel location.
  • Causing the apparatus to project the pixels of the input image data into the output image perspective may comprise causing the apparatus, for each output pixel in the output image perspective, to estimate an orientation in three-dimensions of a plane of a surface of an imaged object at the location of the output pixel based on the depth values associated with locations of neighbouring output pixels in the output image perspective, and to project one or more non-occluded pixels of the input image data which contribute to the output pixel onto the plane.
  • Causing the apparatus to perform colour reconstruction based on the pixels of the input image data projected into the output image perspective may comprise causing the apparatus to colour-reconstruct each output pixel in the output image perspective based on the contributing pixels of the input image data which have been projected onto the plane estimated for that output pixel.
  • Causing the apparatus to colour-reconstruct each output pixel in the output image perspective may comprise causing the apparatus, for each output pixel, to compute a weighted average of the projected contributing pixels associated with each colour of the colour filter array.
  • Causing the apparatus to compute a weighted average of the projected contributing pixels associated with each colour of the colour filter array may comprise causing the apparatus to give each contributing pixel a weight that has an inverse relationship with a distance between the location of projection onto the plane of the contributing pixel and the location of the output pixel.
  • At least two of the plural constituent portions of input image data may have been captured with different rotational orientations relative to their respective direction of capture.
  • the apparatus may be an image capture system.
  • this specification describes computer-readable medium having computer- readable code stored thereon, the computer-readable code, when executed by at least one processor, causing performance of at least, for each of plural constituent portions of input image data captured by at least one image sensor overlaid with a colour filter array, determining a depth value for each input image pixel in the constituent portion, wherein each constituent portion of the input image data has a different associated direction of capture and/or a different associated location of capture and represents a different view of a scene, and wherein each pixel of input image data is associated with a respective one of the colours of the colour filter array; projecting pixels of the input image data into an output image perspective based on the determined depth values, the associated directions and/or locations of capture and information describing the output image perspective; and performing colour reconstruction based on the pixels of the input image data projected into the output image perspective.
  • the computer-readable code stored on the medium of the sixth aspect may further cause performance of any of the operations described with reference to the method of the first aspect.
  • this specification describes apparatus comprising means for
  • the apparatus further comprises means for projecting pixels of the input image data into an output image perspective based on the determined depth values, the associated directions and/or locations of capture, and information describing the output image perspective.
  • the apparatus further comprises means for performing colour reconstruction based on the pixels of the input image data projected into the output image perspective.
  • the apparatus of the seventh aspect may further comprise means for causing performance of any of the operations described with reference to the method of the first aspect.
  • Figure ⁇ is an example of a multiple image capture system configured to capture multiple images with various different perspectives of a scene
  • Figure 2 is a schematic illustration of part of an image capture device, specifically an image sensor overlaid with a colour filter array;
  • Figure 3 is a functional block diagram of image processing apparatus for performing colour reconstruction based on input image data describing multiple different perspectives of a scene
  • Figure 4 is a flow chart illustrating various operations which may be performed by the image processing apparatus of Figure 3;
  • FIG. 5 is a schematic illustration of an example hardware configuration of the image processing apparatus of Figure 3.
  • Figure 6 is an illustration of a computer-readable medium upon which computer readable code may be stored.
  • Figure 1 is an example of a multiple image capture system 100 configured to capture multiple images with various different perspectives of a scene.
  • the multiple image capture system 100 is configured to capture image data from a plurality of directions, e.g. simultaneously, by way of plural image sensors.
  • the multiple image capture system 100 may be configured to capture one image at a time, with the perspective of the capturing image sensor being changed between each image capture by changing the capture direction and/ or the location of the sensor.
  • Multiple image capture systems 100 may be used, for instance, to generate panoramic or immersive images or video.
  • the multi-capture system loo comprises a plurality of image capture devices no at least one some of which are arranged to capture image data from different perspectives of a scene.
  • each image capture device no has a different orientation (or, put another way, faces in a different direction of capture) relative to the other capture devices.
  • some of the capture devices no may have the same or a similar orientation but their different location results in a different perspective being captured.
  • Each of the image capture devices no may comprise a camera module.
  • each capture device 110 comprises an image sensor 111 and a colour filter array (CFA) 112.
  • the image sensor 111 comprises an array of light sensitive regions (photo-sites) 113, each of which is configured to output a sample (or pixel), the value of which is indicative of the intensity of light falling on the region.
  • photo-sites photo-sites
  • 113a to 113d reference numerals
  • the CFA 112 overlies the image sensor 111.
  • the CFA includes a plurality of colour filtering regions 114 configured to filter light passing through the colour filtering region and falling on the corresponding light sensitive region below.
  • Each colour filtering region 114 permits light of a specific colour (in the example of Figure 2, one of red, green and blue).
  • the colour filtering regions 114 are arranged in a particular pattern, depending on the type of the CFA 112.
  • the CFA 112 is a Bayer filter so the colour filtering regions are arranged accordingly (each 2x2 group of regions including two green regions and one each of blue and red).
  • the concepts described herein are not limited to use with Bayer filters; they are also applicable to CFAs of different types.
  • the output from each image sensing region is associated with a particular colour of the colour filter arrangement.
  • the output from each image sensing region is indicative of an intensity of light, which is incident on the image sensing region, of the specific colour of the corresponding region of the CFA.
  • the output of sensing region 113a is associated with the colour red R
  • the outputs from sensing region 113b and 113c are associated with the colour green G
  • the output from sensing region 113d is associated with the colour B.
  • Image data derived from the image sensor 111 as a whole indicates the intensity of light incident on each of the image sensing regions. Based on knowledge of the configuration of the CFA 112 and the image data, it is possible to reconstruct a full colour image at the resolution of the image sensor 111. This is called colour reconstruction or demosaicing and, in the specific case in which a Bayer filter has been used, may be referred to as "debayering". In the example of a Bayer filter, reconstruction may include estimating the intensity of the two missing colour channels for each pixel from the samples in the neighbourhood. This specification describes processing methods for performing colour reconstruction on image data derived from a multiple capture (or multi-view) system.
  • the colour reconstruction processing described herein may result in the number of image data samples available per output image pixel and/or the resolution of the output image being increased, without increasing the resolution of the individual capturing image sensor(s).
  • the resolution of the output image may be increased compared to the output image resolution obtainable for each lens/sensor combination and also compared to the resolution of panoramas or other virtual reality renderings stitched from the individual output images.
  • the benefits of a higher number of samples and/or a higher resolution are more pronounced the more overlapping views are included in the input image data.
  • multi-view colour-reconstruction as described herein may also decouple the input images from the output images. This may enable, for instance, less redundancy when coding the output image(s).
  • input image data representing multiple different views of a scene can be processed using the multi-view colour reconstruction process described herein with the output being, for instance, a single panoramic image. This may be particularly beneficial if the multi-view colour-reconstruction is performed at the multi-capture system, in which case the input image data may not need to be permanently stored or transmitted.
  • FIG. 3 is a schematic illustration of image processing apparatus 300 comprising functional blocks (in the form of hardware and/or software) which may function together to output at least one reconstructed colour image from input image data representing multiple views of a scene, for instance derived from a multiple capture system 100, such as that illustrated in Figure 1.
  • Figure 4 is a flow chart illustrating various operations which may be performed so as to process input image data derived from a multiple capture system in such a way as to arrive at a colour-reconstructed image.
  • the image processing apparatus 300 of Figure 3 comprises a depth-mapping function 310.
  • the depth mapping function is configured to receive input image data.
  • the input image data is derived from at least one image sensor overlaid with a colour filter array.
  • the input image data is formed of constituent portions of image data each having a different associated direction of image capture (put another way, the direction in which the sensor is facing when capturing the constituent portion of image data) and/or a different location of capture (put another way, the location of the sensor when capturing the constituent portion of image data).
  • each constituent portion of input image data represents a different view or perspective of a scene.
  • Each constituent portion includes pixels or samples representative of intensity of light incident at each image sensing region of the image sensor which captures it.
  • Each pixel or sample is associated with a particular colour of the CFA 112 overlying the image sensor during image capture.
  • each constituent portion of image data may be derived from a different image sensor.
  • each constituent portion of image data may be captured using the same image sensor (or group of image sensors) but at different time.
  • the depth mapping function 310 is configured to compute depth map data 312.
  • the depth map data 312 includes depth values corresponding to each pixel in input image data 311. The depth value is indicative of the distance of the object represented by the pixel from the multiple capture system 100.
  • the depth map data 312 may be generated based on the input image data 311 and/ or depth sensor data 314.
  • Depth sensor data 314 may be generated by one or more depth sensor 315 ⁇
  • the depth map data 312 is registered relative to the multiple capture system 100. In this way, it is possible to determine which portion of depth map data 312 is associated with which part of the scene surrounding the multiple capture system 100. More specifically, it is possible to determine which depth value is associated with which specification location within the scene.
  • the depth map data 312 that is generated by the depth mapping function 310 is passed to the pixel projection function 320.
  • the pixel projection function 320 is configured to project pixels of the input image data 311 into an output image perspective (or view).
  • the pixel projection is performed based on the input image data 311, the depth map data 312, and information descriptive of the output image 321.
  • the information descriptive of the output image 321 describes the characteristics of the one or more output images which are to be reconstructed. Such characteristics may include one or more of the number, geometry, virtual location (that is the location of the output image relative to the capture system in terms of distance and/or direction), resolution of the output images and orientation of the output image(s).
  • the pixel projection function 320 may be configured to project pixels of the input image data 311 into one or more output image view by associating each pixel in the output image view ("output pixel") with a depth value from the depth map data 312. Subsequently, the pixel projection module 320 projects the input image pixels which correspond with the output image view into the output image view based on the depth value associated with each output pixel.
  • Either or both of the depth mapping function 310 and the pixel projection function 320 may be configured to utilise camera configuration information 313 which describes the
  • Such configuration information 313 may describe the intrinsic and extrinsic parameters of each captured view.
  • the configuration information 313 may specify one or more of the number of captured views, the direction of capture associated with each view, the field of view associated with each of the different views, the location of the sensor(s) when capturing each view, the rotational orientation of the sensor relative to the lens axis, lens distortion, and the image centre-point on sensor.
  • the depth mapping module 310 may use the camera configuration information 313 for registration to the camera system and/or to generate a three-dimensional depth map. Alternatively, this projection of the depth map data into three dimensions may be performed by the pixel projection function 320.
  • the pixel projection function 320 may also (or alternatively) utilise the camera configuration information 313 for performing the projection of the input image pixels into the output image view.
  • the projected input pixel data 322 generated by the pixel projection function 320 is passed to a colour reconstruction function 330.
  • the colour reconstruction function is configured to colour-reconstruct (for instance, to "debayer") the projected input pixel data 322 thereby to produce output image data 331 representing one or more output images.
  • the output image data 331 may then be passed to memory 350 for storage or to a display device 340 for display.
  • the input image data is received by the image processing apparatus 300.
  • the method of Figure 4 may be performed post-capture in which case the input image data is captured and stored and subsequently, the input image data is retrieved from storage and provided (in operation S4.1) for processing.
  • the image processing apparatus 300 for instance as illustrated in Figure 3, may be located remotely from the multiple capture system 100.
  • the input image data may be received from the multiple capture system 100 in substantially real-time.
  • the image processing apparatus 300 may be communicatively coupled with the multiple capture system 100.
  • the functionality of the image processing apparatus 300 may be performed by the multiple capture system 100.
  • the image processing apparatus 30 determines a depth value associated with each pixel of input image data. Put another way, the image processing apparatus 300 creates depth map data 312.
  • Generation of the depth map data 312 can be performed in a number of ways. For instance, it may be generated by visual depth mapping analysis performed on the input image data.
  • the depth map data 312 comprises a specific depth value per input image pixel. This may be achieved using purely visual analysis or, alternatively, may be achieved using depth sensor data with refinement based on visual analysis. However, as will be appreciated, the process may be implemented without a specific depth value associated with each input image pixel. Instead, groups of input image pixels (e.g. a 2x2 group) may be associated with a specific depth value.
  • this may be performed using the green image channel only, thereby to reduce the amount of data processing required.
  • the depths based on all colour channels e.g. the red, green, and blue
  • the best one of the channels may then be selected by voting between the three based on the saliency of features in each of the channels.
  • the image processing apparatus 300 projects the data values associated with the input image pixels into three-dimensions relative to a location of the multiple- capture system 100. This is performed using the camera configuration information 313. For instance, in examples in which the capture devices of the system 100 are arranged in a spherical array (such as that illustrated in Figure 1), the depth map values are projected into a three-dimensional space surrounding an origin corresponding to the position of the multiple capture system 100.
  • the image processing apparatus 300 "re-projects" the projected depth values into one more output image perspective based on output image specification.
  • the output image specification may define the number of output images that are to be generated based on the input image data as well as their geometry, virtual location, and resolution.
  • the output image specification may specify plural output images each corresponding 1:1 with the fields of view of the individual image sensors of the multiple capture system 100.
  • the output image specification may specify a monoscopic panorama or a pair of stereoscopic panoramas depending on the configuration of the multiple capture system (e.g. whether or not it is configured to capture stereoscopic images).
  • the three-dimensional depth values are re-projected onto a virtual cylindrical surface, with characteristics (resolution, geometry, orientation etc.) corresponding to those of the output image.
  • fields of view of the different image sensors typically overlap with one another and, as such, there may be pixels in more than one constituent input image data portion which correspond to a single point in three-dimensional space. Consequently, the depth map data may include multiple depth values which correspond to same location in three-dimensional space (particularly at the overlapping regions). At least because of this, when the projected depth values are projected into the output image perspective, more than one depth value may be projected onto a single output image pixel. This situation (of multiple depth values corresponding to a single output pixel) may occur even more frequently (not just at pixels corresponding to the overlapping regions) when the resolution of the output image(s) is less than that of the image sensors.
  • the image processing apparatus 300 is configured to resolve any conflicts by selecting the nearest depth value (that is, the depth value representing the distance that is closest to the multiple camera system 100).
  • the result of this operation is projected depth map data including plural depth values each associated one to one with a different output pixel location.
  • the image processing apparatus 300 estimates, for each output pixel location, based on the depth values associated with output pixels surrounding the output pixel location, a local plane.
  • This local plane is local to the output pixel and represents an orientation in three dimensions of a surface of an imaged object at the location on the object represented by the output pixel.
  • the image processing apparatus 300 projects the input image pixels which contribute to a particular output pixel onto the estimated local plane for that output pixel. Those input image pixels which contribute to the output pixel may be determined based on a distance between the location of the output pixel and location of the input image pixels when projected onto the output image perspective.
  • the contributing input pixels may be those input pixels which, when projected into the output image perspective, overlap with the output pixel.
  • a maximum number of contributing pixels per colour channel may be defined.
  • the projected input image pixels of each colour, up to that defined limit, which are closest to (and, optionally, are within a defined distance from) the output pixel location could be selected as contributing pixels.
  • the result of the projection of operation S4.7 is, for each of the output pixels, plural input image pixels projected onto the local plane in an area surrounding and encompassing the output pixel location.
  • occluded input pixels may be omitted. That is, input image pixels which are occluded by other pixels from a different constituent portion of the image data (which represents a different view of the scene) may be omitted from projection. As such, occluded pixels may not be taken into account during the subsequent operation of determining the colour for each output pixel.
  • the image processing apparatus 300 may therefore 300 be configured to identify (and omit from further processing) the occluded pixels for each output pixel location.
  • Occluded input image pixels for a particular output pixel location P may be those input pixels in the direction of P that, for a given constituent image portion captured by a sensor that is a distance A from the output pixel location P, have an associated depth map value that is less than the distance A.
  • the location P is occluded from the capturing sensor if the depth map value for input image data derived from the sensor at the location X in the direction of P is less than the distance from X to P.
  • the image processing apparatus 300 performs colour reconstruction based on the input image pixels which have been projected onto the output image perspective.
  • the image processing apparatus 300 determines the colour of each output pixel based on the colours of the input image pixels that have been projected onto the local plane for that output pixel.
  • the distance in the plane between the input image pixels and the output pixel location may also be taken into account when determining the reconstructed colour for an output image pixel. For instance, a weighted average per colour of all input image pixels which contribute to the output image pixel may be computed, with the weighting being based on the distance (in the plane) between the output image pixel location and the projected input pixel.
  • the weights may be inversely proportional to the distance.
  • the output image data 331 may be provided.
  • the output image data may indicate an intensity/brightness and a colour for each pixel in the output image(s).
  • the output image data may be stored on memory 350 for later retrieval and/or may be sent for display on a display device 340.
  • the display device and/or storage device may be in wired or wireless connection with the image processing apparatus 300.
  • the multi-capture system 100 may be configured to utilise different orientations in the "roll" direction (i.e. the multiple views of the scene are rotated differently around the lens axis). Put another way, some of the multiple views may be captured with different rotational orientations relative to their respective direction of capture. For instance, in a system comprising multiple image sensors, at least some of the image sensors may be rotated differently about their lens axis. Having a different orientation in the "roll” direction causes pixels of the different views of the scene to be to be oriented differently to one another. Consequently, the distribution of the colours in the input image data (which includes constituent portions derived using the different orientations in the "roll” direction) due to the CFA may be less regular. This may serve to reduce the occurrence and/or extent of aliasing artefacts.
  • debayering (e.g. one view at a time).
  • the results may also be dependent on the quality of the depth information available.
  • benefits derived from performing the reconstruction as a post-process after capture of the raw data, when issues from, for instance, incorrect depth estimates or poor camera calibration can be corrected.
  • FIG. 5 is a schematic illustration of an example hardware configuration with which the image processing apparatus 300 described with reference to Figures 3 and 4 may be implemented;
  • the image processing apparatus 300 comprises processing apparatus 50.
  • the processing apparatus 50 is configured to receive the input image data and to perform colour
  • the input image data may be received at the processing apparatus 50 via an input interface 53.
  • the input image data is received directly from the image sensors 110 of the multi-capture system.
  • the image processing apparatus 300 may form part of the multi-capture system 100.
  • the input image data may be received at the processing apparatus 50 via wired communication (e.g. via the input interface 53) or wireless communication (via transceiver 54 and antenna 55) from the multi-capture system 100 or from a storage medium.
  • the input image data may be pre-stored in the memory 51 which forms part of the processing apparatus 50.
  • the processing apparatus 50 may provide the colour-reconstructed output image data via an output interface 56.
  • the output image data may be provided for display via a display device 340 or to a storage device for storage and later retrieval.
  • the output image data may be transmitted wirelessly via the transceiver 54 and antenna 55 to a display device or a storage device as appropriate. Additionally or alternatively, the output image data may be stored in local storage 51 at the processing apparatus 50 for later retrieval
  • the processing apparatus 50 may comprise processing circuitry 52 and memory 51.
  • Computer-readable code 512A may be stored on the memory 51, which when executed by the processing circuitry 52, causes the processing apparatus 50 to perform any of the operations described herein. Example configurations of the memory 51 and processing circuitry 52 will be discussed in more detail below.
  • the user may control the operation of the image processing apparatus 300 by means of a suitable user input interface UII (not shown) such as key pad, voice commands, touch sensitive screen or pad, combinations thereof or the like.
  • UII user input interface
  • a speaker and a microphone may also be provided, for instance in conjunction with the display 340.
  • the image processing apparatus 300 may comprise appropriate connectors (either wired or wireless) to other devices and/or for connecting external accessories thereto.
  • the memory 51 has computer readable instructions 512A stored thereon, which when executed by the processing circuitry 52 causes the processing apparatus 50 to cause performance of various ones of the operations described with reference to Figures 1 to 4.
  • the processing apparatus 50 may in some instances be referred to, in general terms, as “apparatus”, “computing apparatus” or “processing means”.
  • the processing circuitry 52 may be of any suitable composition and may include one or more processors 52A of any suitable type or suitable combination of types. Indeed, the term "processing circuitry" should be understood to encompass computers having differing architectures such as single/multi-processor architectures and sequencers/parallel architectures.
  • the processing circuitry 52 may be a programmable processor that interprets computer program instructions 512A and processes data.
  • the processing circuitry 52 may include plural programmable processors.
  • the processing circuitry 52 may be, for example, programmable hardware with embedded firmware.
  • the processing circuitry 52 may alternatively or additionally include one or more specialised circuit such as field programmable gate arrays FPGA, Application Specific Integrated Circuits (ASICs), signal processing devices etc.
  • the processing circuitry 52 is coupled to the memory 51 and is operable to read/write data to/from the memory 51.
  • the memory 51 may comprise a single memory unit or a plurality of memory units, upon which the computer readable instructions (or code) 512A is stored.
  • the memory 51 may comprise both volatile memory 511 and non-volatile memory 512.
  • the computer readable instructions/program code 52A may be stored in the non-volatile memory 512A and may be executed by the processing circuitry 52 using the volatile memory 511 for temporary storage of data or data and instructions.
  • volatile memory include RAM, DRAM, and SDRAM etc.
  • Examples of non-volatile memory include ROM, PROM, EEPROM, flash memory, optical storage, magnetic storage, etc.
  • the memory 51 may be referred to as one or more non-transitory computer readable memory medium or one or more storage devices. Further, the term 'memory', in addition to covering memory comprising both one or more non-volatile memory and one or more volatile memory, may also cover one or more volatile memories only, one or more non-volatile memories only.
  • a "memory" or “computer-readable medium” may be any media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.
  • the computer readable instructions/program code 512A may be pre-programmed into the processing apparatus 50.
  • the computer readable instructions 512A A may arrive at the control apparatus via an electromagnetic carrier signal or may be copied from a physical entity 60 such as a computer program product, a memory device or a record medium such as a CD-ROM or DVD an example of which is illustrated in Figure 6.
  • the computer readable instructions 512A A may provide the logic and routines that enables the apparatus 300 to perform the functionality described above.
  • the combination of computer- readable instructions stored on memory (of any of the types described above) may be referred to as a computer program product.
  • the transceiver and antenna 54, 55 may be adapted for any suitable type of wireless communication including but not limited to a Bluetooth protocol, a cellular data protocol or a protocol in accordance with IEEE 802.11.
  • the input and/or output interface 53, 56 may be of any suitable type of wired interface.
  • the interfaces when one or both of the interfaces is configured for wired connection with another device, they may be, for instance but not limited to, physical Ethernet or USB interfaces.
  • the different functions discussed herein may be performed in a different order and/or concurrently with each other. Furthermore, if desired, one or more of the above- described functions may be optional or may be combined.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)

Abstract

La présente invention concerne un procédé consistant à : pour chacune de plusieurs parties constitutives de données d'image d'entrée (311) capturées par au moins un capteur d'image superposé à un réseau de filtres colorés, déterminer une valeur de profondeur (315) pour chaque pixel d'image d'entrée dans la partie constitutive, chaque partie constitutive des données d'image d'entrée ayant une direction de capture associée différente et/ou une position de capture associée différente (313) et représentant une vue différente d'une scène, chaque pixel de données d'image d'entrée étant associé à une couleur respective des couleurs du réseau de filtres colorés. Le procédé consiste en outre à projeter des pixels des données d'image d'entrée dans une perspective d'image de sortie (321) sur la base des valeurs de profondeur déterminées (312), des directions de capture associées et/ou des positions de capture associées, et d'informations décrivant la perspective d'image de sortie. Le procédé consiste en outre à exécuter une reconstitution de couleur (330) sur la base des pixels des données d'image d'entrée projetées dans la perspective d'image de sortie.
PCT/FI2017/050744 2016-10-31 2017-10-30 Reconstitution de couleur à vues multiples WO2018078222A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB1618343.6A GB2555585A (en) 2016-10-31 2016-10-31 Multiple view colour reconstruction
GB1618343.6 2016-10-31

Publications (1)

Publication Number Publication Date
WO2018078222A1 true WO2018078222A1 (fr) 2018-05-03

Family

ID=57963750

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2017/050744 WO2018078222A1 (fr) 2016-10-31 2017-10-30 Reconstitution de couleur à vues multiples

Country Status (2)

Country Link
GB (1) GB2555585A (fr)
WO (1) WO2018078222A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113538318B (zh) * 2021-08-24 2023-12-15 北京奇艺世纪科技有限公司 图像处理方法、装置、终端设备以及可读存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100097444A1 (en) * 2008-10-16 2010-04-22 Peter Lablans Camera System for Creating an Image From a Plurality of Images
US20100259595A1 (en) * 2009-04-10 2010-10-14 Nokia Corporation Methods and Apparatuses for Efficient Streaming of Free View Point Video
EP2327059B1 (fr) * 2008-10-02 2014-08-27 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Synthèse de vue intermédiaire et extraction de signal de données multivue

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6084979A (en) * 1996-06-20 2000-07-04 Carnegie Mellon University Method for creating virtual reality
IL139995A (en) * 2000-11-29 2007-07-24 Rvc Llc System and method for spherical stereoscopic photographing
WO2008068456A2 (fr) * 2006-12-06 2008-06-12 Sony United Kingdom Limited Procédé et dispositif de génération de contenu d'image
KR101467509B1 (ko) * 2008-07-25 2014-12-01 삼성전자주식회사 이미지 센서 및 이미지 센서 동작 방법
KR100968378B1 (ko) * 2009-03-05 2010-07-09 주식회사 코아로직 파노라마 이미지를 생성하는 장치, 방법 및 그 방법을 실행하는 프로그램이 기록된 기록 매체
US9739914B2 (en) * 2010-07-13 2017-08-22 Nokia Technologies Oy Color image sensing
BR112012027306A2 (pt) * 2011-02-28 2016-08-02 Fujifilm Corp aparelho de geração de imagem colorida
US9191646B2 (en) * 2011-08-29 2015-11-17 Nokia Technologies Oy Apparatus, a method and a computer program for video coding and decoding
NL2010463C2 (nl) * 2013-03-15 2014-09-16 Cyclomedia Technology B V Werkwijze voor het genereren van een panoramabeeld.
KR102214193B1 (ko) * 2014-03-25 2021-02-09 삼성전자 주식회사 깊이 카메라 장치, 그것을 구비한 3d 영상 디스플레이 시스템 및 그 제어방법
CN106796390A (zh) * 2014-10-07 2017-05-31 诺基亚技术有限公司 用于立体成像的具有大视场的相机设备
US20160198131A1 (en) * 2015-01-06 2016-07-07 Samsung Electronics Co., Ltd. Rgb/rwb sensor with independent integration time control for improvement of snr and color accuracy
WO2016137238A1 (fr) * 2015-02-26 2016-09-01 Dual Aperture International Co., Ltd. Traitement de données d'image à ouvertures multiples

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2327059B1 (fr) * 2008-10-02 2014-08-27 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Synthèse de vue intermédiaire et extraction de signal de données multivue
US20100097444A1 (en) * 2008-10-16 2010-04-22 Peter Lablans Camera System for Creating an Image From a Plurality of Images
US20100259595A1 (en) * 2009-04-10 2010-10-14 Nokia Corporation Methods and Apparatuses for Efficient Streaming of Free View Point Video

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ZITNICK, CL. ET AL.: "High-quality video view interpolation using a layered representation", ACM TRANSACTIONS ON GRAPHICS (TOG) - PROCEEDINGS OF ACM SIGGRAPH 2004, vol. 23, no. 3, 12 August 2004 (2004-08-12), pages 600 - 608, XP002354522, Retrieved from the Internet <URL:https://dl.acm.org/citation.cfm?id=1015766> [retrieved on 20180212] *

Also Published As

Publication number Publication date
GB2555585A (en) 2018-05-09
GB201618343D0 (en) 2016-12-14

Similar Documents

Publication Publication Date Title
US10897609B2 (en) Systems and methods for multiscopic noise reduction and high-dynamic range
JP5929553B2 (ja) 画像処理装置、撮像装置、画像処理方法およびプログラム
JP6021541B2 (ja) 画像処理装置及び方法
US10762655B1 (en) Disparity estimation using sparsely-distributed phase detection pixels
KR102565277B1 (ko) 영상 복원 장치 및 방법
WO2016164166A1 (fr) Génération automatisée de prises de vue panoramiques
WO2015048694A2 (fr) Systèmes et procédés destinés à la correction de la distorsion de la perspective utilisant la profondeur
JP5984493B2 (ja) 画像処理装置、画像処理方法、撮像装置およびプログラム
JP2015197745A (ja) 画像処理装置、撮像装置、画像処理方法及びプログラム
JP5766077B2 (ja) ノイズ低減のための画像処理装置及び画像処理方法
JP2011166264A (ja) 画像処理装置、撮像装置、および画像処理方法、並びにプログラム
US20130162786A1 (en) Image processing apparatus, imaging apparatus, image processing method, and program
WO2017051407A1 (fr) Stockage de données extraites à partir de différents capteurs pour générer une image tridimensionnelle
US20130083169A1 (en) Image capturing apparatus, image processing apparatus, image processing method and program
KR20220121533A (ko) 어레이 카메라를 통해 획득된 영상을 복원하는 영상 복원 방법 및 영상 복원 장치
CN110995982B (zh) 图像处理装置及其控制方法、摄像装置、以及记录介质
WO2018078222A1 (fr) Reconstitution de couleur à vues multiples
JP2013150071A (ja) 符号化装置、符号化方法、プログラム及び記憶媒体
CN115170554A (zh) 图像检测方法和电子设备
JP2013200840A (ja) 映像処理装置、映像処理方法、映像処理プログラム、及び映像表示装置
WO2019167571A1 (fr) Dispositif et procédé de traitement d&#39;image
US10425594B2 (en) Video processing apparatus and method and computer program for executing the video processing method
KR20200043778A (ko) 영상 처리 장치 및 방법
JP2013110669A (ja) 画像処理方法および装置、プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17865982

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17865982

Country of ref document: EP

Kind code of ref document: A1