WO2015083499A1 - Image processing device, method and program - Google Patents
Image processing device, method and program Download PDFInfo
- Publication number
- WO2015083499A1 WO2015083499A1 PCT/JP2014/079767 JP2014079767W WO2015083499A1 WO 2015083499 A1 WO2015083499 A1 WO 2015083499A1 JP 2014079767 W JP2014079767 W JP 2014079767W WO 2015083499 A1 WO2015083499 A1 WO 2015083499A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- data
- filter
- image data
- unit
- Prior art date
Links
- 238000012545 processing Methods 0.000 title claims abstract description 339
- 238000000034 method Methods 0.000 title claims abstract description 183
- 238000001914 filtration Methods 0.000 claims abstract description 39
- 239000003086 colorant Substances 0.000 claims abstract description 29
- 238000003491 array Methods 0.000 claims abstract description 23
- 230000008569 process Effects 0.000 claims description 119
- 238000003384 imaging method Methods 0.000 claims description 75
- 230000003287 optical effect Effects 0.000 claims description 55
- 238000004364 calculation method Methods 0.000 claims description 34
- 238000007781 pre-processing Methods 0.000 claims description 33
- 238000006243 chemical reaction Methods 0.000 claims description 21
- 238000003672 processing method Methods 0.000 claims description 8
- 230000002093 peripheral effect Effects 0.000 abstract description 24
- 238000010586 diagram Methods 0.000 description 28
- 239000011159 matrix material Substances 0.000 description 19
- 230000006870 function Effects 0.000 description 16
- 230000015572 biosynthetic process Effects 0.000 description 9
- 238000003786 synthesis reaction Methods 0.000 description 9
- 230000006866 deterioration Effects 0.000 description 8
- 230000015556 catabolic process Effects 0.000 description 7
- 238000006731 degradation reaction Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- DHGBAFGZLVRESL-UHFFFAOYSA-N 14-methylpentadecyl 16-methylheptadecanoate Chemical compound CC(C)CCCCCCCCCCCCCCC(=O)OCCCCCCCCCCCCCC(C)C DHGBAFGZLVRESL-UHFFFAOYSA-N 0.000 description 4
- 101000588145 Homo sapiens Microtubule-associated tumor suppressor 1 Proteins 0.000 description 4
- 101000588157 Homo sapiens Microtubule-associated tumor suppressor candidate 2 Proteins 0.000 description 4
- 102100031549 Microtubule-associated tumor suppressor candidate 2 Human genes 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000002194 synthesizing effect Effects 0.000 description 4
- 150000001875 compounds Chemical class 0.000 description 3
- 230000000295 complement effect Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 229930091051 Arenine Natural products 0.000 description 1
- 238000012935 Averaging Methods 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000002146 bilateral effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 230000004907 flux Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4015—Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/131—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing infrared wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/133—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing panchromatic light, e.g. filters passing white light
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/135—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on four or more different wavelength filter elements
Definitions
- the present invention relates to an image processing technology for processing an image, and more particularly to an image processing technology capable of performing corresponding point search with higher accuracy.
- digital images can be obtained relatively easily by using solid-state imaging devices such as CCD (Charge Coupled Device) type image sensors and CMOS (Complementary Metal Oxide Semiconductor) type image sensors.
- CCD Charge Coupled Device
- CMOS Complementary Metal Oxide Semiconductor
- This digital image is used in various ways. For example, there are an image processing technique for measuring distances from a plurality of digital images to a predetermined subject, an image processing technique for converting resolution, and the like.
- One of such resolution conversion techniques is a high-resolution image generation technique for synthesizing one relatively high-resolution image from a plurality of relatively low-resolution images. It is disclosed in Patent Document 1.
- the high-resolution color image generation method disclosed in Patent Document 1 generates a high-resolution color image based on one or a series of input images captured by an image capturing device using an image sensor and a color filter array. To do.
- the high-resolution image generation technique disclosed in Non-Patent Document 1 searches for corresponding points between a plurality of images using G pixel data of image data captured by an image sensor using a so-called Bayer array. The position shift is estimated by the above, and one high-resolution image is generated from a plurality of images.
- the Bayer array image sensor has a unit array of four color filters of G (green), R (red), B (blue) and G (green) arranged in a two-dimensional array in two rows and two columns. And a color filter in which a plurality of unit arrays are further arranged in a two-dimensional array, and an image sensor that receives an optical image of an object (subject) by each photoelectric conversion element (each pixel) via the color filter.
- the image processing technique disclosed in Non-Patent Document 1 uses only G pixel data, that is, half of the image data, for searching for corresponding points. For this reason, there has been a limit to the accuracy of the corresponding point search.
- the present invention has been made in view of the above-described circumstances, and an object thereof is to provide an image processing apparatus, an image processing method, and an image processing program capable of performing corresponding point search with higher accuracy.
- each of a plurality of original image data is filtered by an image filter to generate a plurality of preprocessed image data used for corresponding point search.
- the original image data is a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array, and a plurality of unit arrays in which a plurality of density data of different colors are arranged in a predetermined pattern
- the image filter further includes the plurality of data arranged in a two-dimensional array, and the image filter is based on data corresponding to pixels located around a pixel corresponding to target data that is data to be filtered.
- a filter coefficient is provided for converting the data into data having the same color composition ratio as that of the unit array. Therefore, the image processing apparatus, the image processing method, and the image processing program according to the present invention can perform corresponding point search with higher accuracy.
- FIG. 1st Embodiment It is a block diagram which shows the structure of the image processing apparatus in 1st Embodiment. It is a figure which shows the structure of the image acquisition part in the case of being a camera in the image processing apparatus of 1st Embodiment. It is a perspective view which shows the structure of the image acquisition part in the case of being an array camera in the image processing apparatus of 1st Embodiment. It is a figure which shows the unit color filter arrangement
- the image processing apparatus in the embodiment is an apparatus that performs predetermined image processing on given original image data, and the predetermined image processing includes at least preprocessing and corresponding point search processing.
- an imaging device (camera) performs so-called normal image processing such as white balance processing, filter processing, gradation conversion processing, and color space conversion processing on an image signal output from an image sensor, thereby obtaining a final image.
- normal image processing such as white balance processing, filter processing, gradation conversion processing, and color space conversion processing
- the original image data is a raw image signal output from an image sensor before performing such normal image processing, and is a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array.
- Such original image data is, for example, raw data (raw data).
- the preprocessing is processing for converting each data of each pixel in the original image data into data that can be processed equally when searching for corresponding points.
- the corresponding point search is a process of searching for a point on a reference image corresponding to a predetermined point on the standard image when one of the two images is a standard image and the other is a reference image.
- the image processing apparatus D in such an embodiment can have various modes depending on the contents of image processing.
- a high-resolution image having a higher resolution than the image is selected from a plurality of images (image data).
- the image processing device Da according to the first embodiment to be generated and the image processing device Db according to the second embodiment for obtaining the distance from a plurality of images to a predetermined subject will be described.
- FIG. 1 is a block diagram illustrating the configuration of the image processing apparatus according to the first embodiment.
- FIG. 2 is a diagram illustrating a configuration of an image acquisition unit in the case of a camera in the image processing apparatus according to the first embodiment.
- FIG. 3 is a perspective view illustrating a configuration of an image acquisition unit in the case of an array camera in the image processing apparatus according to the first embodiment.
- FIG. 4 is a diagram illustrating unit color filter arrays of color filters in the camera or the array camera. 4A to 4E show first to fifth modes of the unit color filter array.
- FIG. 5 is a diagram illustrating a configuration of an image filter in the image processing apparatus according to the first embodiment. 5A to 5E show first to fifth aspects of the image filter.
- the image processing device Da includes, for example, a control processing unit 1a, a storage unit 2, an image acquisition unit 3, an input unit 4, and an output unit 5, as shown in FIG.
- the image acquisition unit 3 is connected to the control processing unit 1a and is a device for taking original image data into the image processing device Da from the outside.
- the image acquisition unit 3 outputs the acquired image data to the control processing unit 1a.
- the image acquisition unit 3 is, for example, a drive device 3a that reads the original image data from a recording medium on which the original image data is recorded, a camera 3b that captures an optical image of an object (subject) and generates original image data. .
- the drive device 3a is a device that reads / writes data from / to a recording medium such as a flexible disk, CD-ROM, CD-R, and DVD-R, for example, a flexible disk drive, a CD-ROM drive, a CD-R drive, and the like. DVD-R drive or the like. Further, for example, the drive device 3a is a device that reads and writes data from a solid-state storage element such as an SD memory card (SD logo is a registered trademark) and a USB (Universal Serial Bus) memory, such as an SD drive and a USB interface. .
- SD memory card SD logo is a registered trademark
- USB Universal Serial Bus
- the camera 3b includes an imaging optical system 31 that forms an optical image of a subject on a predetermined imaging plane, a color filter unit 32 disposed on the image side of the imaging optical system 31, and the predetermined An image pickup unit that is arranged so that a light receiving surface is positioned at the position of the image forming plane, and that picks up an optical image of a subject imaged by the image pickup optical system 31 via the color filter unit 32 and outputs original image data 33.
- the camera 3b may be a single-lens camera 3ba, or may be a compound eye (three or more eyes) array camera 3bb.
- the monocular camera 3ba is a device that images a subject and generates one original image data.
- a filter unit 32 and an imaging unit 33a that captures an optical image of a subject formed by the imaging optical system 31a in one effective imaging region via the color filter unit 32 and outputs one original image data.
- the imaging optical system 31a includes one or more optical lenses along the optical axis.
- the imaging unit 33a includes a plurality of photoelectric conversion elements (a plurality of pixels) arranged in a two-dimensional matrix, and each photoelectric conversion element converts the light according to the amount of light received through the color filter unit 32. The electrical signal is output as data of each pixel in the original image data.
- Such an imaging unit 33a includes a solid-state imaging device such as a CCD image sensor or a CMOS image sensor.
- the three or more compound eye array camera 3bb is a device that captures a subject and generates a plurality (three or more) of original image data.
- a plurality of color filter sections 32 (not shown in FIG. 3) corresponding to the system 31b, the plurality of imaging optical systems 31b, respectively disposed on the respective image sides of the imaging optical systems 31, and the plurality of imaging optics.
- a plurality of imaging units 33b that correspond to the system 31b and that respectively capture optical images of the subject formed by the plurality of imaging optical systems 31b through the color filter units 32 and output a plurality of original image data.
- the color filter unit 32 is omitted.
- Each of the plurality of imaging optical systems 31b is an optical element that forms an optical image of a subject on the light receiving surface of the imaging unit 33b corresponding to the imaging optical system 31b.
- One imaging optical system 31b includes one or a plurality of optical lenses along the optical axis.
- the plurality of imaging optical systems 31 b are arranged so that the optical axes are substantially parallel to each other. Therefore, the plurality of imaging units 33b that respectively capture the optical images of the subject via the plurality of imaging optical systems 31b and the plurality of color filter units 32 generate original image data that is shifted by a substantially parallax showing the same subject. become.
- FIG. 3 the example shown in FIG.
- the plurality of imaging optical systems 31b are two-dimensional matrices in two linearly independent directions corresponding to the plurality of imaging units 32b, more specifically, in two directions of X and Y directions orthogonal to each other. Are arranged in a shape.
- the plurality of imaging optical systems 31b includes nine imaging optical systems 31b-11 to 31b arranged in a two-dimensional matrix in three rows and three columns in accordance with the arrangement mode and the number of the imaging units 33b. -33.
- Each of the plurality of imaging units 33b includes a plurality of photoelectric conversion elements (a plurality of pixels) arranged in a two-dimensional matrix, and each photoelectric conversion element receives light received via the color filter unit 32, respectively.
- the electrical signal converted according to the amount of light is output as data of each pixel in the original image data.
- the plurality of imaging units 33b correspond to the plurality of imaging optical systems 31b and are arranged so that the imaging surfaces are on the same plane.
- the plurality of imaging units 33 b are arranged in a two-dimensional matrix in two linearly independent directions, more specifically, in two directions of X and Y directions orthogonal to each other.
- the plurality of imaging units 33b includes nine imaging units 33b-11 to 33b-33 arranged in a two-dimensional matrix in three rows and three columns.
- the plurality of imaging units 33b may include a plurality of the solid-state imaging elements arranged in a two-dimensional matrix on the same substrate, but in the example illustrated in FIG.
- the effective pixel area in the single solid-state imaging device is divided into a plurality of areas arranged in a two-dimensional matrix so as to correspond to each imaging unit 33b. Used as the imaging unit 33b.
- Each of the color filter unit 32 in the monocular camera 3ba and the plurality of color filter units 32 in the compound eye array camera 3bb includes a plurality of unit color filter arrays in which a plurality of different color filters are arranged in a predetermined pattern, and a two-dimensional array. It is the optical filter element arranged in the shape.
- Each color filter of the color filter unit 32 transmits light in a predetermined wavelength range (transmission wavelength band) of incident light, and is disposed so as to correspond to each photoelectric conversion element of the imaging unit 33. That is, each color filter in the color filter unit 32 of the monocular camera 3ba is arranged so as to correspond to each of a plurality of photoelectric conversion elements (a plurality of pixels) in the imaging unit 33a.
- the plurality of color filter units 32 in the compound-eye array camera 3bb are arranged so as to correspond to the plurality of imaging units 33b, and in each of the color filter units 32 and the imaging unit 33b, each color filter in the color filter unit 32 is These are arranged so as to correspond to the plurality of photoelectric conversion elements (a plurality of pixels) in the imaging unit 33b.
- the unit color filter array can adopt various modes depending on, for example, the use application of the image, for example, the unit color filter arrays 321a to 321e of the first to fifth modes shown in FIGS. 4A to 4E.
- the unit color filter array 321a of the first mode shown in FIG. 4A includes a red filter (R) that transmits red, a first green filter (G) that transmits green, a second green filter (G) that transmits green, and
- the blue filter (B) that transmits blue light is arranged in a two-dimensional matrix with two rows and two columns, and is a so-called Bayer arrangement. That is, a red filter is arranged in one row and one column, a first and second green filter is arranged in one row and two columns and two rows and one column, respectively, and a blue filter is arranged in two rows and two columns.
- a color filter section 32a in which such a plurality of unit color filter arrays 321a are further arranged in a two-dimensional array may be used as the color filter section 32 shown in FIG.
- the color filter unit 32a is a so-called primary color filter.
- the unit color filter array 321b of the second aspect shown in FIG. 4B includes a yellow filter (Ye) that transmits yellow, a magenta filter (Mg) that transmits magenta, a cyan filter (Cy) that transmits cyan, and
- green filters (G) that transmit green are arranged in a two-dimensional matrix with two rows and two columns. That is, a yellow filter is arranged in one row and one column, a magenta filter is arranged in one row and two columns, a cyan filter is arranged in two rows and one column, and a green filter is arranged in two rows and two columns.
- a color filter section 32b in which such a plurality of unit color filter arrays 321b are further arranged in a two-dimensional array may be used as the color filter section 32 shown in FIG.
- the color filter unit 32b is a so-called complementary color filter.
- the unit color filter array 321c of the third aspect shown in FIG. 4C includes a red filter (R) that transmits red, a green filter (G) that transmits green, a blue filter (B) that transmits blue, and infrared light.
- R red filter
- G green filter
- B blue filter
- a color filter section 32c in which a plurality of such unit color filter arrays 321c are further arranged in a two-dimensional array may be used as the color filter section 32 shown in FIG.
- the unit color filter array 321d of the fourth aspect shown in FIG. 4D includes a white filter (W) that transmits white, a yellow filter (Ye) that transmits yellow, a red filter (R) that transmits red, and infrared light.
- W white filter
- Ye yellow filter
- R red filter
- a color filter unit 32d in which such unit color filter arrays 321d are further arranged in a two-dimensional array may be used as the color filter unit 32 shown in FIG.
- the color filter unit 32d using the unit color filter array 321d of the fourth aspect, it is possible to generate an image using not only visible light but also infrared light.
- the unit color filter arrays 321a to 321d of the first to fourth modes shown in FIGS. 4A to 4D have a size of 2 rows and 2 columns, but are not limited to this.
- the array may be of any size.
- the unit color filter array 321e of the fifth aspect shown in FIG. 4E includes first and second red filters (R) that transmit red, first to fifth green filters (G) that transmit green, and blue.
- the first and second blue filters (B) that pass through are arranged in a two-dimensional matrix with three rows and three columns. That is, the first and second red filters are arranged in 1 row 1 column and 3 rows 3 columns, respectively, and 1 row 2 columns, 2 rows 1 column, 2 rows 2 columns, 2 rows 3 columns and 3 rows 2 columns respectively.
- First to fifth green filters are arranged, and blue filters are arranged in 1 row 3 columns and 3 rows 1 column, respectively.
- a color filter section 32e in which such a plurality of unit color filter arrays 321e are further arranged in a two-dimensional array may be used as the color filter section 32 shown in FIG.
- the light beam from the subject is incident on the imaging optical system 31a and forms an optical image of the object on the light receiving surface of the imaging unit 33a via the color filter unit 32.
- Each photoelectric conversion element of the imaging unit 33a photoelectrically converts the light received through each color filter of the color filter unit 32 into an electrical signal corresponding to the amount of light, and outputs it as data of each pixel. Since the color filter unit 32 includes the unit color filter array 321 shown in FIG.
- the imaging unit 33a includes a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array, and Original image data including a plurality of unit arrays in which a plurality of density data of different colors are arranged in a predetermined pattern and the plurality of data arranged in a two-dimensional array is output.
- each imaging unit 33b photoelectrically converts the light received through each color filter of each color filter unit 32 into an electrical signal corresponding to the amount of light, and outputs it as data of each pixel. Since the color filter unit 32 is composed of the unit color filter array 321 shown in FIG. 4, each imaging unit 33b is a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array.
- original image data including a plurality of unit arrays in which a plurality of density data of a plurality of different colors are arranged in a predetermined pattern, and the plurality of data arranged in a two-dimensional array is output. Therefore, the plurality of imaging units 33b output a plurality of original image data.
- the solid-state imaging device including nine imaging units 33b-11 to 33b-33 outputs nine original image data.
- the input unit 4 is connected to the control processing unit 1 a and inputs various commands such as a command for instructing start of image processing, and an identifier (file name) in an image to be processed, for example. And a device that inputs various data necessary for executing image processing such as selection input of an image filter used in image processing to the image processing device Da, such as a keyboard and a mouse.
- the output unit 5 is a device that is connected to the control processing unit 1a and outputs a command or data input from the input unit 4 and an image obtained as a result of image processing performed by the image processing device Da.
- a display device such as a display, an LCD, and an organic EL display, and a printing device such as a printer.
- a touch panel may be configured from the input unit 4 and the output unit 5.
- the input unit 4 is a position input device that detects and inputs an operation position such as a resistive film method or a capacitance method
- the output unit 5 is a display device.
- a position input device is provided on the display surface of the display device, one or more input content candidates that can be input to the display device are displayed, and the user touches the display position where the input content to be input is displayed.
- the position is detected by the position input device, and the display content displayed at the detected position is input to the image processing device D as the operation input content of the user.
- the storage unit 2 is a device that is connected to the control processing unit 1a and stores various programs and various data necessary for executing image processing. For example, ROM (Read Only Memory) or EEPROM (Electrically Erasable Programmable Read Only). Non-volatile memory elements such as Memory), volatile memory elements such as RAM (Random Access Memory), and peripheral circuits thereof.
- the storage unit 2 may include a large-capacity storage device such as a hard disk.
- the storage unit 2 stores original image data, and is used as a work area for performing image processing to be described later on the original image data by the control processing unit 1a.
- the control processing unit 1a controls each part of the image processing device Da according to the function of each part in order to execute image processing to be described later based on a predetermined image processing program.
- the control processing unit 1a includes, for example, a CPU (Central Processing Unit) and its peripheral circuits, and functionally executes a predetermined image processing program, so that the control unit 11, the preprocessing unit 12, and corresponding points are functionally executed.
- a search processing unit 13, a positional deviation amount calculation unit 14, and a super-resolution processing unit 15 are configured.
- the control unit 11 controls each unit of the image processing device Da in accordance with the function of each unit in order to execute predetermined image processing.
- the pre-processing unit 12 generates pre-processed image data by filtering the original image data with a predetermined image filter (first image filter) as pre-processing.
- the image filter (first image filter) converts the target data into the color of the unit array 321 in the original image data based on the data corresponding to the pixels located around the pixel corresponding to the target data that is the data to be filtered.
- the filter coefficient is converted to data having the same color composition ratio as the composition ratio.
- the image filter FL1 can adopt various modes. For example, the first to fourth modes shown in FIGS. 5A to 5D with respect to the original image data obtained by the color filter unit 32a in which the Bayer array is a unit color filter array.
- Image filters FL1a to FL1d are based on data (peripheral data) corresponding to pixels (peripheral pixels) located around the pixel (target pixel) corresponding to the filtering target data.
- Each value of the filter coefficient is adjusted and set so as to be converted.
- the image filter FL1a is a filter having a filter coefficient that is symmetric with respect to the center position of 3 rows and 3 columns. That is, the filter coefficient located in the center 2 rows and 2 columns is 4/16, and each filter coefficient located in 1 row 2 columns, 2 rows 1 columns, 2 rows 3 columns and 3 rows 2 columns is 2 / 16, and each filter coefficient located in each of 1 row, 1 column, 1 row, 3 columns, 3 rows, 1 column and 3 rows, 3 columns is 1/16.
- the image filter FL1a of the first aspect having such a filter coefficient is also a Gaussian filter. Note that the image filters shown on the left and right in FIG. 5A are the same except for the expression format. On the left side of FIG.
- the filter coefficient value itself is described at each position.
- the numerator of the filter coefficient value is described at each position, and the denominator is collectively described at the lower right.
- 5B to 5D are described in the description format shown on the right side of FIG. 5A.
- the image filter FL1b of the second aspect is a filter having a filter coefficient that is symmetric with respect to the center position of 5 rows and 5 columns. That is, the filter coefficient located in the center 3 rows and 3 columns is 36/256, and each filter coefficient located in 2 rows 3 columns, 3 rows 2 columns, 3 rows 4 columns and 4 rows 3 columns is 24 Each of the filter coefficients located in 2 rows, 2 columns, 2 rows, 4 columns, 4 rows, 2 columns, and 4 rows, 4 columns is 16/256, 1 row, 3 columns, 3 rows, 1 column, 3 Each filter coefficient located in each of row 5 column and row 5 column 3 is 6/256, 1 row 2 columns, 1 row 4 columns, 2 rows 1 column, 2 rows 5 columns, 4 rows 1 column, 4 rows.
- Each filter coefficient located in the 5th column, 5th row and 2nd column, and 5th row and 4th column is 4/256, and 1st row and 1st column, 1st row and 5th column, 5th row and 1st column, and 5th row and 5th column respectively.
- Each filter coefficient located is 1/256.
- the image filter FL1c is a filter having a filter coefficient that is symmetric with respect to the center position of 7 rows and 7 columns. That is, the filter coefficient located at the center 4 rows and 4 columns is 400/4096, and each filter coefficient located at 3 rows 4 columns, 4 rows 3 columns, 4 rows 5 columns and 5 rows 4 columns is 300. Each filter coefficient located in 3 rows, 3 columns, 3 rows, 5 columns, 5 rows, 3 columns, and 5 rows, 5 columns is 225/4096, 2 rows, 4 columns, 4 rows, 2 columns, 4 Each filter coefficient located in row 6 column and 6 row 4 column is 200/4096, 2 rows 3 columns, 2 rows 5 columns, 3 rows 2 columns, 3 rows 6 columns, 5 rows 2 columns, 5 rows.
- Each filter coefficient located in each of the 5th column, 6th row and 3rd column, and 6th row and 5th column is 90/4096, which is located in each of 2nd row and 2nd column, 2nd row and 6th column, 6th row and 2nd column, and 6th row and 6th column.
- Each filter coefficient is 36/4096, 1 row 4 columns, 4 rows 1 column, 4 rows 7 columns and 7 rows 4 columns
- Each filter coefficient located there is 20/4096, 1 row 3 columns, 1 row 5 columns, 3 rows 1 columns, 3 rows 7 columns, 5 rows 1 columns, 5 rows 7 columns, 7 rows 3 columns and Each filter coefficient located in each of 7 rows and 5 columns is 15/256, 1 row 2 columns, 1 row 6 columns, 2 rows 1 column, 2 rows 7 columns, 6 rows 1 column, 6 rows 7 columns, 7
- Each filter coefficient located in each of row 2 column and 7 row 6 column is 6/256, and each filter located in 1 row 1 column, 1 row 7 column, 7 row 1 column and 7 row 7 column respectively The coefficient is 1/4096.
- the image filter FL1d is a filter having a filter coefficient that is symmetric with respect to the center position of 5 rows and 5 columns. That is, the filter coefficient located in the center 3 rows and 3 columns is 16/144, and each filter coefficient located in 2 rows 3 columns, 3 rows 2 columns, 3 rows 4 columns, and 4 rows 3 columns is 12 / 144, and each filter coefficient located in each of 2 rows 2 columns, 2 rows 4 columns, 4 rows 2 columns and 4 rows 4 columns is 9/144, 1 row 3 columns, 3 rows 1 columns, 3 Each filter coefficient located in each of row 5 column and row 5 column 3 is 4/144, 1 row, 2 columns, 1 row, 4 columns, 2 rows, 1 column, 2 rows, 5 columns, 4 rows, 1 column, 4 rows. Each filter coefficient located in each of 5 columns, 5 rows 2 columns and 5 rows 4 columns is 3/144, and 1 row 1 column, 1 row 5 columns, 5 rows 1 column and 5 rows 5 columns respectively. Each filter coefficient located is 1/144.
- the image filter FL1 may, for example, be applied to the image of the original image data. It is preferable to select appropriately according to the ratio of the high frequency component contained. For example, the image filter FL1 that is more blurred is selected as the ratio of the high-frequency component included in the image of the original image data increases.
- various image filters FL1a to FL1d of the first to fourth modes for the original image data obtained by the color filter unit 32a using the unit color filter array 321a as the unit of the repetitive pattern have been described.
- the image filter FL1 can be set for the original image data obtained by the color filter unit 32 using the unit color filter array 321.
- the image filter FL1e of the fifth mode shown in FIG. 5E can be used for the original image data obtained by the color filter unit 32e using the unit color filter array 321e of the fifth mode shown in FIG. 4E.
- the image filter FL1e uses the data (peripheral data) corresponding to the pixel (peripheral pixel) located around the pixel (target pixel) corresponding to the target data for filtering as the target data.
- Each coefficient value is adjusted and set. More specifically, as shown in FIG. 5E, each filter coefficient located at each position is a 1/9 filter.
- the image filter FL1e of the fifth aspect is also a filter having a filter coefficient that is symmetric with respect to the center position of 3 rows and 3 columns.
- each filter coefficient of the image filter FL is set so that the sum of the filter coefficients at the same arrangement position of the unit arrangement becomes the same value at all arrangement positions when the unit arrangement is overlapped over the entire area of the image filter FL. May be.
- the corresponding point search processing unit 13 performs corresponding point search among a plurality of images corresponding to a plurality of pre-processed image data generated by the pre-processing unit 12 from a plurality of original image data. To do.
- the positional deviation amount calculation unit 14 obtains a positional deviation amount that is a distance between corresponding points searched by the corresponding point search processing unit 13.
- the positional shift amount may be obtained in pixel units (pixel units, integer multiples of the pixel pitch), but in the present embodiment, it is obtained in subpixel units.
- the super-resolution processing unit 15 generates a high-resolution image higher than the resolution of the original image data from the plurality of original image data based on the positional shift amount obtained by the positional shift amount calculation unit 14.
- Such an image processing device Da can be constituted by a personal computer such as a notebook type or a desktop type, a portable device such as a mobile phone, or a device equipped with a computer such as a camera.
- FIG. 6 is a flowchart illustrating the operation of the image processing apparatus according to the first embodiment.
- FIG. 7 is a diagram for explaining the mutual positional relationship between the images captured by the array camera.
- FIG. 8 is a diagram for explaining preprocessing in the image processing apparatus according to the first embodiment. 8A shows the relationship between each pixel of the image (each data of the original image data) and the image filter, FIG. 8B shows the image filter, FIG. 8C shows the preprocessed image after filtering, and FIG. FIG. 8E shows an example of an image before preprocessing, and FIG. 8E shows an example of an image after preprocessing.
- FIG. 8A shows the relationship between each pixel of the image (each data of the original image data) and the image filter
- FIG. 8B shows the image filter
- FIG. 8C shows the preprocessed image after filtering
- FIG. 8E shows an example of an image before preprocessing
- FIG. 8E shows an example of an image after preprocessing.
- FIG. 8A
- FIG. 9 is a diagram for explaining corresponding point search processing in the image processing apparatus according to the first embodiment.
- FIG. 9A is a diagram in which the first and second images to be searched for corresponding points are displayed in an overlapping manner.
- FIGS. 9B and 9C show the corresponding point search with relatively coarse accuracy, and FIG. Corresponding point search with relatively fine accuracy is shown.
- FIG. 10 is a diagram for explaining corresponding point search processing in units of subpixels in the image processing apparatus according to the first embodiment.
- FIG. 10A shows an NCC value which is an example of a corresponding point search result
- FIG. 10B shows a corresponding point obtained in units of subpixels from the NCC value shown in FIG. 10A.
- FIG. 10A shows an NCC value which is an example of a corresponding point search result
- FIG. 10B shows a corresponding point obtained in units of subpixels from the NCC value shown in FIG. 10A.
- FIG. 11 is a flowchart showing the super-resolution processing of the first aspect in the image processing apparatus of the first embodiment.
- FIG. 12 is a diagram for explaining deterioration information of super-resolution processing in the image processing apparatus according to the first embodiment.
- FIG. 13 is a diagram for explaining the relationship between residual processing and super-resolution processing.
- FIG. 13A shows the Bayer array
- FIGS. 13B and 13C show the interpolation relationship between the two input images.
- FIG. 14 is a diagram for describing conversion of an output candidate image according to deterioration information in the residual processing. 14A shows the output candidate image Xn
- FIG. 14B shows the output candidate image Xn converted to the resolution equivalent to the input image according to the degradation information
- FIG. 14C shows the conversion D from FIG.
- FIG. 14A to FIG. 14B it is a diagram for explaining a k H k F k.
- FIG. 15 is a diagram for explaining the calculation of the sign function in the residual process.
- 15A shows an output candidate image Xn converted to a resolution equivalent to the input image according to the degradation information
- FIG. 15B shows the input image Yk
- FIG. 15C shows the difference sign (D k H k F k X n ⁇ Y k ).
- FIG. 16 is a diagram for explaining processing for obtaining pixels of an output candidate image that has influenced the calculation result of the sign function in the residual processing.
- FIG. 16A shows the difference sign (D k H k F k X n ⁇ Y k ), and FIG.
- FIG. 16B shows the difference image F k T H k T D k T sign (D k ) at the size of the super-resolution image.
- H k F k X n -Y k ) indicates
- FIG. 16C is a diagram for explaining conversion F k T H k T D k T to Figure 16B from Figure 16A.
- FIG. 17 is a flowchart showing residual processing for each input image.
- the image processing device Da executes an image processing program when receiving an activation command from the input unit 4 by a user operation, for example.
- the control processing unit 1a functionally includes the control unit 11, the preprocessing unit 12, the corresponding point search processing unit 13, the positional deviation amount calculation unit 14, and the super-resolution processing unit 15.
- a plurality of original image data corresponding to each of a plurality of images is input to the image processing device Da by the image acquisition unit 3 and stored in the storage unit 2.
- nine first to ninth original image data IS are input and stored in the storage unit 2.
- These first to ninth original image data IS are data generated by imaging the same subject with an array camera 3bb having nine eyes arranged in a two-dimensional array, for example, in three rows and three columns. Accordingly, the first to ninth original image data IS-11 to IS-33 are arranged in a two-dimensional array in three rows and three columns as shown in FIG.
- the original image data located in one row and one column is the first original image data IS-11, and each original image data IS located in each position of the two-dimensional array is sequentially numbered, and three rows and three columns.
- the original image data located at is the ninth original image data IS-33.
- the original image data IS positioned at the center of 2 rows and 2 columns is the fifth original image data IS-22, which is used as a reference image in the corresponding point search described later.
- These first to ninth original image data IS-11 to IS-33 may each be data of an arbitrary number of images generated by using an appropriate color filter unit 32.
- This is data of 500 ⁇ 500 pixels generated by using the color filter section 32a in the Bayer array.
- the pre-processing unit 12 filters the first to ninth original image data IS-11 to IS-33 with the image filter FL1, respectively, thereby filtering the first to ninth post-processed image data AIS-11 to AIS-33. These are generated, and the generated first to ninth post-processed image data AIS-11 to AIS-33 are stored in the storage unit 2. More specifically, for example, in this embodiment, an image filter FL1 corresponding to the original image data IS of the Bayer array, for example, an image filter FL1a shown in FIG. 5A is used. The filtering is sequentially executed from data corresponding to the pixel located in the first row and first column in the original image data IS to data corresponding to the pixel located in the last 500 rows and 500 columns.
- the preprocessed image data AIS corresponding to the G pixels located in 2 rows and 2 columns
- the color composition ratio of the unit color filter array 321a in the color filter section 32a that is, it is the same as the color composition ratio of the unit array in the original image data IS.
- the preprocessed image data AIS corresponding to the R pixels located at 3 rows and 2 columns is (1 row 1).
- the data of the preprocessed image data AIS corresponding to the B pixels located in 2 rows and 3 columns is (image located in 1 row and 1 column) Filter value of filter FL1a; 1/16) ⁇ (value of data of R pixel at position corresponding to the position) + (filter value of image filter FL1a located in 1 row ⁇ 2 columns; 2/16) ⁇ (the position Data value of the G pixel at the position corresponding to) + (filter value of the image filter FL1a located in the first row and the third column; 1/16) ⁇ (data value of the R pixel at the position corresponding to the position) + ( Filter value of image filter FL1a located in 2 rows and 1 column; 2/16) ⁇ (value of data of G pixel at position corresponding to that position) + (filter value of image filter FL1a located in 2 rows and 2 columns; 4/16 X (value of data of B pixel at the position corresponding to the position) + (filter value of image filter FL1a;
- the preprocessed image data AIS data corresponding to the G pixels located in 3 rows and 3 columns is (positioned in 1 row and 1 column).
- Filter value of the image filter FL1a to be performed 1/16) ⁇ (data value of the G pixel at the position corresponding to the position) + (filter value of the image filter FL1a located in the first row and the second column; 2/16) ⁇ ( Data value of the B pixel at the position corresponding to the position) + (Filter value of the image filter FL1a located in the first row and the third column; 1/16) ⁇ (Data value of the G pixel at the position corresponding to the position) + (Filter value of image filter FL1a located at 2 rows and 1 column; 2/16) ⁇ (value of data of R pixel at position corresponding to that position) + (filter of image filter FL1a located at 2 rows and 2 columns) value; / 16) ⁇ (value of data of G pixel at
- the image filter FL1a Since the image filter FL1a has a filter coefficient having the same color composition ratio as that of the unit array in the original image data IS, when the image filter FL1a is applied to the original image data IS, the R pixel of the original image data IS.
- the total of the filter coefficients corresponding to the B pixels of the original image data IS, and the total of the filter coefficients corresponding to the G pixels of the original image data IS are the unit array in the original image data IS. It becomes the same as the color composition ratio.
- the data of any pixel in the unit array of the original image data IS is filtered with the same color composition ratio by the image filter FL1a. Accordingly, all pixel data in the original image data IS are filtered with the same color composition ratio by the image filter FL1a.
- all pixel data in the pre-processed image data AIS can be handled as data having the same color component synthesized in (G / 2 + R / 4 + B / 4) as shown in FIG. 8C. . Therefore, in the next corresponding point search, all pixel data in the preprocessed image data AIS can be used.
- the original image shown in FIG. 8D becomes a pre-processed image of a single-channel (single color) monochrome image shown in FIG. 8E by the filtering of the image filter FL1a.
- the corresponding point search processing unit 13 executes the corresponding point search.
- the fifth preprocessed image corresponding to the fifth preprocessed image data AIS-22 located at the center of 2 rows and 2 columns is set as a reference image, and around the fifth preprocessed image, First to fourth and sixth to ninth pre-processing corresponding to the first to fourth and sixth to ninth post-processed image data AIS-11 to AIS-21 and AIS-23 to AIS-33 are located.
- Each subsequent image is set as a reference image, and each corresponding point for the reference image is searched for each reference image.
- corresponding points are searched for in units of pixels (first corresponding point searching process), and then corresponding points are searched for in units of subpixels (second corresponding point searching process).
- first corresponding points are searched every plural pixels, for example, every eight pixels (rough corresponding point search).
- second corresponding points are searched every other pixel around the position searched in the rough corresponding point search process (fine corresponding point search process).
- fine corresponding point search process For each corresponding point search process, known conventional means are used, and the corresponding point search process is disclosed in, for example, Japanese Patent Application Laid-Open No. 2009-14445.
- the corresponding point search between the first pre-processed image and the fifth pre-processed image will be described more specifically below.
- FIG. The image shown in In the first pre-processed image and the fifth pre-processed image, first, in the rough corresponding point search process of the first corresponding point search process, the corresponding points are detected on the fifth pre-processed image of the reference image.
- a window Wb having a predetermined size around a predetermined pixel position to be searched is set.
- FIG. 9C Eight windows Wra1 at intervals of a plurality of pixels in the diagonal direction of the two-dimensional array arrangement, which is a shift direction between the first preprocessed image and the fifth preprocessed image, for example, every four pixels in the XY direction. ⁇ Wra8 is set.
- FIG. 9C shows five windows Wra1 to Wra5. Then, the degree of coincidence (degree of correlation) of each of the windows Wra1 to Wra8 on the first preprocessed image of the reference image with respect to the window Wb (template) on the fifth preprocessed image of the standard image is obtained by a predetermined method.
- the pixel at the center position in the window Wra having the highest degree of matching is obtained as the corresponding point.
- the predetermined method include known conventional means, such as NCC (Normalized Cross Correlation) method, SAD (Sum of Absolute Difference) method (absolute error method), and SSD (Sum of Squared Difference) method (square error method). Is used.
- NCC Normalized Cross Correlation
- SAD Sum of Absolute Difference
- SSD Sud of Squared Difference
- I (i, j) is a pixel value at the position (i, j) in the window Wb of the reference image (a pixel data value in the preprocessed image data AIS)
- T (i, j) is This is the pixel value at the position (i, j) in the window Wr of the reference image (the pixel data value in the preprocessed image data AIS)
- N is the length (pixel) in the X direction (horizontal direction) in the windows Wb and Wr
- M is a length (pixel unit) in the Y direction (vertical direction) in the windows Wb and Wr.
- the periphery of the center pixel of the window Wra having the highest degree of coincidence is determined with one pixel accuracy.
- the corresponding point search process is executed in a plane.
- the corresponding point search range is a predetermined range, for example, every second pixel is searched in the rough corresponding point search process (in the above example, every four pixels in the XY direction). Range of 4 pixels).
- 81 windows Wrb1 to Wrb81 are set around each of ⁇ 4 pixels around the center pixel of the window Wr having the highest degree of coincidence. These 81 windows Template matching is executed for each of Wrb1 to Wrb81, and the pixel at the center position in the window Wrb having the highest degree of matching is obtained as the corresponding point.
- the center pixel of the window Wrb having the highest degree of coincidence and the nine pixels around the center pixel are each nine.
- the degree of coincidence is obtained as the most approximate quadric surface (quadratic surface fitting), and the coordinate values giving the vertices of the obtained quadric surface are obtained as corresponding points in subpixel units.
- FIG. 10B shows a section of a quadratic curved surface).
- the method for obtaining the corresponding points in units of sub-pixels is, for example, “Tanaka, Yaguchi, Furukawa, Okutomi,“ Robust super-resolution processing based on pixel selection considering misalignment ”, IEICE Transactions, Vol. J92-D, No. 5, pp650-660, 2009. , May 2009 ".
- corresponding points in units of subpixels are obtained by obtaining the most approximate quadric surface, but quadratic curves that are most approximate in the X direction and the Y direction are obtained, respectively.
- Each coordinate value giving the peak of the quadratic curve may be obtained as a corresponding point in subpixel units.
- the positional deviation amount between the corresponding points is obtained by the positional deviation amount calculation unit 14.
- the unit length is a size (size) of one pixel
- a coordinate difference between corresponding points becomes a positional deviation amount. That is, the difference between the coordinate value of the pixel in the standard image and the coordinate value of the pixel corresponding to the pixel of the standard image in the reference image is the positional deviation amount.
- a value obtained by multiplying the coordinate difference between corresponding points by the pixel pitch is the positional deviation amount.
- the pixel pitch is a length between the center position of a pixel and the center position of a pixel adjacent thereto.
- step S5 the super-resolution processing unit 15 obtains one image (super-resolution image) having a resolution higher than the resolution of the plurality of images from the plurality of images.
- known conventional means are used. For example, “Fast and Robust Multiframe Super Resolution”, IEEE TRANSACTION ON IMAGE PROSING, Vol. 13, no. 10, Oct. 2004, pp 1327-1344.
- the reconstruction type super-resolution processing disclosed in this document will be outlined below with reference to FIGS.
- image data obtained by demosaicing the original image data is used, and the super-resolution processing shown in FIG. 11 is executed for each color component. That is, the image data has values of R component, B component, and G component for each pixel by demosaic, and the super-resolution processing shown in FIG. Plane), an image composed of each pixel value of the B component (B plane), and an image composed of each pixel value of the G component (G plane) and super-resolved R
- the case of the G plane will be outlined as an example, but the same applies to the case of the R plane and the B plane.
- a plurality of images and their deterioration information are read from the storage unit 2 into the super-resolution processing unit 15.
- the first to ninth image data DIS-11 to DIS-33 obtained by demosaicing each of the first to ninth original image data IS-11 to IS-33 and the deterioration information thereof are stored in the storage unit 2. It is read into the resolution processing unit 15.
- the deterioration information is an image corresponding to the super-resolution image and the image data DIS (hereinafter referred to as the super-resolution process) when viewed from an ideal super-resolution image after the super-resolution process. In this description, it is referred to as “input image”).
- a one-dimensional vector having each pixel value (G component pixel value in this example) of the super-resolution image is X, and each pixel value (G component pixel value in this example) of the input image corresponding to the image data DIS.
- k indicates each input image corresponding to each of the first to ninth original image data IS-11 to IS-33, and is an integer value of 1 to 9.
- a superscript T indicates a transposed matrix. This deterioration information is used for calculating a residual in the process S14 described later.
- the output candidate image X 0 of the initial image is the first output candidate image X n in the repeating process executed repeatedly each processing of the processing S12 ⁇ process S15 as will be described later to generate a super-resolution image X.
- This output candidate image X 0 is an image having the same resolution (same size) as the super-resolution image X to be generated. For example, a plurality of output candidate images X 0 corresponding to the first to ninth original image data IS-11 to IS-33.
- the input image Y k has a size of 500 ⁇ 500 and the super-resolution image X has a size of 1500 ⁇ 1500
- the input image Y k is generated in the X direction (horizontal) in the generation of the output candidate image X 0. (Direction) and Y direction (longitudinal direction) are enlarged three times. Between adjacent pixels in the input image Y k, the input image Y k is larger 3 times, in the X and Y directions, respectively, two pixels without pixel values will be generated anew each (In the two-dimensional array, eight pixels having no pixel value are newly generated), and pixel values of pixels having no pixel value are generated by the interpolation process.
- a so-called BTV amount (bilateral total value) is obtained as a regularization term so that the super-resolution processing unit 15 can robustly converge even if there is noise in the input image Yk .
- This BTV amount is the third term on the right side in Equation 2 described later.
- the super-resolution processing unit 15 applies the first to ninth input images Y k corresponding to the first to ninth image data DIS-11 to DIS-33 for each of the input images Y k. Residuals that are differences between the resolution (size) of the input image Yk and the output candidate image Xn converted to the same resolution (size) according to the deterioration information are obtained. This residual is the second term on the right side in Equation 2 described later. This residual is obtained, for example, when only a G pixel is used when a G-plane super-resolution image X is generated when super-resolution processing is performed on a Bayer array image. As shown in FIG. 13, when super-resolution processing is performed on one image, this means that a pixel value not included in the image is supplemented with a pixel value of another image using a positional shift amount.
- the residual process S14 which is a process for obtaining the residual, will be described more specifically with reference to FIGS. 14 to 17.
- degradation of the input image Yk is performed. information is multiplied by the output candidate image X n, the input image Y k output candidate image (D converted to the same resolution (size) and resolution (size) of the image in accordance with the deterioration information of k H k F k X n ) is generated (processing S22-1 to S22-9 to processing S24-1 to S24-9 in FIG. 17).
- n is an integer value of 0 to 199 because the number of repetitions is set to a constant value of 200 times that can be determined to normally converge.
- the output candidate image Xn is first filtered by an image filter having PSF information as a filter coefficient (processes S22-1 to S22-9 in FIG. 17). n of the blurred image is generated, so as to have the same resolution as the resolution of the input image Y k, fifth input image Y 5 and the input corresponding to the fifth image data DIS-22 of the original output candidate image X n considering position shift amount F k of the image Y k downsampling is performed (processing S23-1 ⁇ S23-9, the process S24-1 ⁇ S24-9).
- the positional deviation amount F k between the fifth input image Y 5 and the input image Y k is 1/3 pixel in the X direction (horizontal direction), and the X direction (horizontal direction).
- the Y direction vertical direction
- the average value of the pixel values of the 9 pixels of 3 ⁇ 3 located at the shifted position is the downsampled pixel value.
- the input image Y k FIG. 15
- the super-resolution image X affected the subtraction result (D k H k F k X n ⁇ Y k ) (difference) calculated by the sign function sign ( ⁇ ).
- n pixels are examined (processing S26-1 to S26-9 to processing S28-1 to S28-9 in FIG. 17).
- this process is defined by the matrix D k H k F k described above, this process is defined by a matrix F k T H k T D k T using a transposed matrix. That is, this process is carried out upsampling taking into account the positional shift amount F k of the fifth input image Y 5 and the input image Y k corresponding to the fifth image data DIS-22 original output candidate image X n (F k T D k T , processing S26-1 to S26-9 in FIG.
- the positional deviation amount F k between the fifth input image Y 5 and the input image Y k is 1/3 pixel in the X direction (horizontal direction), and X
- the difference sign (D k H k F k X n -Y k) is, the area ratio of each pixel before and after upsampling (FIG 16C
- FIG. 16C In the example shown in FIG.
- a difference image for one input image Yk is generated.
- this processing is executed for each of the first to ninth input images Y k as shown in FIG. 17 to generate nine difference images. .
- the total of these nine differences is the residual.
- the super-resolution processing unit 15 calculates the BTV amount and the residual obtained in the above-described processes S13 and S14, respectively, as shown in Expression 2, using the output candidate image X By subtracting from n , the next output candidate image Xn + 1 is generated. Then, this output candidate image Xn + 1 is set as the output candidate image Xn of the process S12, and the process returns to S12.
- ⁇ is a parameter for adjusting the degree of influence of the BTV amount
- ⁇ is a parameter for adjusting the degree of influence of the residual and ⁇ ⁇ BTV amount
- each of the processes S12 to S15 is repeated a predetermined number of times that the output candidate image Xn is normally converged, for example, 200 times as described above.
- the super-resolution process of process S5 is completed, and the output candidate image X 199 is set as the super-resolution image X.
- the super-resolution image X is stored in the storage unit 2 and then output to the output unit 5 and displayed.
- the luminance component generation is performed so that the original image data IS is converted into data having the same color composition ratio as that of the unit array (unit color filter array) by the image filter FL. Since filtering is performed by the processing unit, data of all pixels can be handled as data having the same color component. As a result, the corresponding point search can be performed using the data of all the pixels. Therefore, the image processing device Da according to the present embodiment can perform the corresponding point search with higher accuracy.
- each value of the filter coefficient in the image filter FL is arranged symmetrically with respect to the center position, each pixel (each pixel) positioned around the center pixel (center pixel) Each pixel value in the peripheral pixel) is weighted according to the distance between the peripheral pixel and the central pixel, and is appropriately reflected in the central pixel. For this reason, the image processing device Da according to the present embodiment can handle the data of all the pixels as data having more equivalent color components, and therefore can perform the corresponding point search with higher accuracy.
- the image processing apparatus Da includes the positional deviation amount calculation unit 14, the positional deviation amount F between corresponding points searched by the corresponding point search processing unit 13 can be obtained, and is executed with higher accuracy.
- the corresponding point search result can be effectively reflected in the estimation of the positional deviation amount F.
- the image processing apparatus Da uses a nine-eye array camera, a plurality of original image data can be obtained simultaneously. For this reason, when the positional deviation amount F between corresponding points is obtained from the result of the corresponding point search, the positional deviation amount F is obtained between a plurality of images obtained by copying the same subject at the same time.
- the image processing device Da in the embodiment can determine the positional deviation amount F with higher accuracy.
- the image processing apparatus Da includes the super-resolution processing unit 15, a positional deviation amount F between corresponding points searched by the corresponding point search processing unit 13 is obtained, and a plurality of original image data is obtained based on this.
- high-resolution image data can be obtained, and the result of corresponding point search performed with higher accuracy can be effectively reflected in the generation of a high-resolution image.
- the super-resolution processing disclosed in the paper has been outlined.
- the super-resolution processing is not limited to this method, and a plurality of relatively low resolutions using a positional shift amount are used.
- a reconfigurable super-resolution process that generates one super-resolution image having a relatively high resolution from the image can be used.
- the reconfiguration super-resolution processing shown in FIG. 18 can be used.
- FIG. 18 is a flowchart showing the super-resolution processing of the second aspect in the image processing apparatus of the first embodiment.
- the super-resolution process is a process to be represented by the formula 3, the second term on the right side of the equation 3, the residual (F k T H k T D k T (D k H k F k X n -Y k)
- the third term on the right-hand side of Equation 3 is a regularization term C.
- this regularization term C for example, a 4-neighbor Laplacian image filter is used.
- the filter coefficient at each position of 1st column, 1st row, 3rd column, 3rd row, 1st column and 3rd row and 3rd column is 0, and each position of 1st row, 2nd column, 2nd row, 1st column, 2nd row, 3rd column and 3rd row, 2nd column And the filter coefficient at the position of 2 rows and 2 columns is -4.
- ⁇ is a parameter for adjusting the degree of influence of the regularization term
- ⁇ is a parameter for adjusting the degree of influence of the residual and ⁇ ⁇ regularization term
- process S31 a plurality of images and their degradations are displayed in the super-resolution processing unit 15. Information is read from the storage unit 2. Then, the processing S32, the output candidate image X 0 of the initial image is generated. Next, in process S33, a regularization term is obtained by an image filter of 4 neighborhood Laplacian. Then, the processing S34, the residual (F k T H k T D k T (D k H k F k X n -Y k) is calculated.
- the processing S35 as shown in Equation 3, the regularization term and the residual obtained in process S33 and the processing S34, respectively described above, by subtracting from the output candidate image X n, the next output candidate image X n + 1 is generated. Then, the output candidate image X n + 1 Is the output candidate image Xn of the process S32, and the process is returned to S32. Then, each process of the processes S32 to 35 is performed a predetermined number of times that the output candidate image Xn is normally converged, for example, Repeated 40 times When each of the processes S32 to S35 is repeated 40 times, the super-resolution process of process S5 is completed, and the output candidate image X39 is set as the super-resolution image X.
- FIG. 19 is a block diagram illustrating a configuration of an image processing apparatus according to the second embodiment.
- FIG. 20 is a flowchart illustrating the operation of the image processing apparatus according to the second embodiment.
- the image processing device Da according to the first embodiment is a device that generates one relatively high resolution image from a plurality of relatively low resolution images (original image data), but the image according to the second embodiment.
- the processing device Db is a device that obtains a predetermined distance from a plurality of images (original image data) to a subject.
- the image processing apparatus Db in the second embodiment includes a control processing unit 1 b, a storage unit 2, an image acquisition unit 3, an input unit 4, and an output unit 5.
- the storage unit 2, the image acquisition unit 3, the input unit 4, and the output unit 5 in the image processing device Db of the second embodiment are respectively the storage unit 2, the image acquisition unit 3, and the image processing device Da of the first embodiment. Since it is the same as that of the input part 4 and the output part 5, the description is abbreviate
- control processing unit 1b controls each unit of the image processing device Db according to the function of each unit in order to execute image processing described later based on a predetermined image processing program.
- the control processing unit 1b includes, for example, a CPU and its peripheral circuits, and functionally executes a predetermined image processing program, so that the control unit 11, the preprocessing unit 12, the corresponding point search processing unit 13, A positional deviation amount calculation unit 14 and a distance calculation unit 16 are configured. That is, the control processing unit 1b includes a distance calculation unit 16 instead of the super-resolution processing unit 15 with respect to the control processing unit 1a.
- control unit 11, the preprocessing unit 12, the corresponding point search processing unit 13, and the positional deviation amount calculation unit 14 in the control processing unit 1b of the second embodiment are respectively controlled by the control processing unit 1a of the first embodiment. Since this is the same as the unit 11, the preprocessing unit 12, the corresponding point search processing unit 13, and the positional deviation amount calculation unit 14, description thereof is omitted.
- the distance calculation unit 16 uses the positional shift amount between corresponding points obtained by the positional shift amount calculation unit 14 as parallax between a plurality of images corresponding to a plurality of original image data, and the subject represented by the corresponding point based on the parallax The distance to is obtained.
- the image processing apparatus Db first has an original image corresponding to each of nine images by the image acquisition unit 3 in the process S41 as in the above-described process S1.
- Data is input to the image processing apparatus Db and stored in the storage unit 2.
- the preprocessing unit 12 performs a preprocess on each of these nine images.
- the process S43 as in the above-described process S3, the fifth image located in the center (2 rows and 2 columns) of these nine images is set as the reference image, and is located in the periphery with respect to this reference image.
- Corresponding point search is executed by the corresponding point search processing unit 13 for each of the eight images.
- a point corresponding to a predetermined subject is appropriately selected.
- the inter-vehicle distance between the vehicle on which the image processing device is mounted with Db and its preceding vehicle (or oncoming vehicle) is obtained, it corresponds to the preceding vehicle (or oncoming vehicle).
- the point corresponding to the pedestrian is selected.
- the positional deviation amount calculation unit 14 determines the positional deviation amount.
- step S45 the positional deviation amount obtained by the positional deviation amount computing unit 14 by the distance computing unit 16 is determined based on the reference image (image located in 2 rows and 2 columns) and the reference image (then the image).
- the distance to the subject is determined based on the parallax.
- known conventional means are used, and such distance measurement technology is disclosed in, for example, Japanese Patent Application Laid-Open No. 2010-276469 and Japanese Patent Application Laid-Open No. 2009-14445.
- each image of a subject (target object) is obtained by a pair of cameras that are separated by a predetermined interval (base line length), and corresponding point search is performed on each image in pixel units.
- the parallax between the pair of cameras in the separation direction is obtained from each of these images in pixel units, and the distance to the subject is obtained based on the so-called triangulation principle based on the obtained parallax. More specifically, in FIG. 21, at least two focal lengths (f), the number of pixels on the imaging surface (imaging unit 33 (for example, CCD type image sensor)), and the size ( ⁇ ) of one pixel are equal to each other.
- the first and second cameras are spaced apart from each other by a predetermined base line length (L) and the optical axes AX-1 and AX-2 are arranged in parallel.
- the distance obtained by the distance calculation unit 16 in this way is output to the output unit 5 by the control processing unit 1b, and the process ends.
- the output unit 5 displays the distance to the subject.
- the distance to the subject eight distances may be displayed for each of the eight images located in the periphery with respect to the fifth image (reference image) located in the center (2 rows and 2 columns). An average value may be displayed.
- the original image data IS is nine pieces of data IS-11 to IS-33 acquired by the nine-lens array camera 3bb. However, the original image data IS is picked up at nine different times by at least one camera 3ba. Nine data may be sufficient.
- Such an image processing apparatus D can obtain a plurality of original image data captured at different times.
- the image processing device Db according to the second embodiment can perform the corresponding point search with higher accuracy in the same manner as the image processing unit Da according to the first embodiment. Since the image processing device Db according to the second embodiment includes the distance calculation unit 16, the parallax (position shift amount) between the corresponding points searched by the corresponding point search processing unit 13 can be obtained with higher accuracy. The result of the corresponding point search executed in (5) can be effectively reflected in the distance calculation.
- the demosaic process for generating image data having each color component data for each pixel from the original image data may be the following process.
- FIG. 22 is a diagram for explaining a demosaic processing unit in a modification of the image processing apparatus according to the embodiment.
- FIG. 22A is a block diagram illustrating a configuration of the demosaic processing unit 17, and
- FIG. 22B is a flowchart illustrating an operation of the demosaic processing unit 17.
- FIG. 23 is a diagram for explaining a luminance component generation process in the demosaic process executed by the demosaic processor.
- FIG. 23A shows an example of an image (original image) PIS of the original image data IS
- FIG. 23B shows elementary luminance component image data obtained by filtering the original image data IS shown in FIG. 23A with the second image filter FL2a of the first mode.
- FIG. 23A shows an example of an image (original image) PIS of the original image data IS
- FIG. 23B shows elementary luminance component image data obtained by filtering the original image data IS shown in FIG. 23A with the second image filter FL2a of the first mode.
- FIG. 23A shows an
- FIG. 23C shows a BISp-1 image (primary luminance component image) PBISp-1 and the second image filter FL2a of the first mode
- FIG. 23C shows the original image data IS shown in FIG. 23A as the second image filter FL2b of the second mode
- FIG. 23D shows an image (element luminance component image) PBISp-2 of the elementary luminance component image data BISp-2 filtered in step S2 and the second image filter FL2b of the second aspect
- FIG. 23D shows the elementary luminance component image shown in FIG. 23B.
- the luminance component image data BIS obtained by combining the elementary luminance component image shown in FIG. It shows the image) PBIS.
- FIG. 24 is a diagram for explaining chromaticity component generation processing in the demosaic processing.
- FIG. 24A shows original image data IS when the unit array is a Bayer array
- FIG. 24B shows luminance component image data BIS corresponding to the original image data IS shown in FIG. 11A
- FIG. 24C shows FIG.
- the chromaticity component image data CIS (CIS-G, CIS-R, CIS-B) of each color of G, R, and B corresponding to the original image data IS shown (color-specific chromaticity component image data before interpolation) is shown.
- FIG. 25 is a diagram for explaining chromaticity component interpolation processing in the demosaic processing.
- FIG. 25A shows the positional relationship between the target pixel X and its peripheral pixels A to D
- FIG. 25B shows the weights and the interpolation calculation.
- FIG. 26 is a diagram for explaining an image generation process in the demosaic process.
- FIG. 26A shows the case of the G component
- FIG. 26B shows the case of the R component
- FIG. 26C shows the case of the B component.
- FIG. 27 is a diagram for explaining jagged lines that occur when a relatively high-resolution image is generated from a plurality of relatively low-resolution images.
- the control processing unit 1 of the image processing apparatus D in the above-described embodiment is functionally further provided with a demosaic processing unit 17.
- the demosaic processing unit 17 generates at least one color of image data having each color component data for each pixel by demosaicing the original image data.
- a demosaic processing unit 17 includes a luminance component generation processing unit 171, a chromaticity component generation processing unit 172, a chromaticity component interpolation processing unit 173, and an image generation process as illustrated in FIG. 22A. Part 174.
- the luminance component generation processing unit 171 generates luminance component image data by filtering original image data with a predetermined second image filter.
- the predetermined second image filter is similar to the first image filter FL1 described above, based on data corresponding to pixels located around pixels corresponding to target data that is data to be filtered, the target data, The filter coefficient is converted to data having the same color composition ratio as that of the unit array 321 in the original image data.
- the said 2nd image filter may be one, in this embodiment, several image filter is used.
- the luminance component generation processing unit 171 generates a plurality of elementary luminance component image data by filtering the original image data with each of the plurality of second image filters, and sets the plurality of elementary luminance component image data to a predetermined predetermined value.
- the luminance component generation image data is generated by combining with the combination ratio.
- the plurality of second image filters include second image filters having different sizes and different removal rates for removing high-frequency components by the filtering.
- the luminance component generation processing unit 171 generates the first and second elementary luminance component image data by filtering the original image data with the 2-1 and 2-2 image filters, respectively.
- the luminance component generation image data is generated by synthesizing the second elementary luminance component image data at a predetermined synthesis ratio set appropriately in advance.
- This 2-1 image filter is larger than the size of the 2-2 image filter, and the removal rate for removing high-frequency components by the filtering is smaller than that of the 2-2 image filter.
- Such a second image filter will be described in detail later.
- the chromaticity component generation processing unit 172 generates chromaticity component image data by subtracting, for each pixel, the luminance component image data generated by the luminance component generation processing unit 171 from the original image data. It is.
- the chromaticity component interpolation processing unit 173 supplies the chromaticity component data of the missing pixel for each of the plurality of colors in the same color included in the unit array. And the chromaticity component image data for each color for each of the plurality of colors are generated by interpolation using an interpolation method based on the luminance component image data generated by the luminance component generation processing unit 171.
- the chromaticity component interpolation processing unit 173 uses a bilinear interpolation method when interpolating chromaticity component data of a color having the same number of colors included in the unit array. It is used as a method.
- the chromaticity component interpolation processing unit 173 performs a luminance component generation processing unit when interpolating chromaticity component data of a plurality of colors of the same color included in the unit array.
- An interpolation method based on the luminance component image data generated in 171 is used as the interpolation method. More specifically, in the interpolation method based on the luminance component image data generated by the luminance component generation processing unit 171, the pixel lacking the chromaticity component data is set as the target pixel, and the attention on the chromaticity component data is set.
- the interpolation method is selected according to whether or not there is.
- the chromaticity component data of the peripheral pixel position positioned around the first pixel position is used. This is a weighted interpolation method that weights and averages the weights according to the degree of correlation, and when there is no correlation, it is a bilinear interpolation method.
- the image generation processing unit 174 adds the luminance component image data generated by the luminance component generation processing unit 171 to each chromaticity component image data for each color generated by the chromaticity component interpolation processing unit 173, respectively.
- the image data is generated by adding each pixel.
- the luminance component image data BIS is generated based on the original image data IS by the luminance component generation processing unit 171 of the demosaic processing unit 17.
- the original image data IS is generated by imaging the same subject with an array camera 3bb having nine eyes, and the nine first to ninth shown in FIG. 7 described above.
- Original image data IS is generated by imaging the same subject with an array camera 3bb having nine eyes, and the nine first to ninth shown in FIG. 7 described above.
- Original image data IS is generated by imaging the same subject with an array camera 3bb having nine eyes, and the nine first to ninth shown in FIG. 7 described above.
- Original image data IS is generated by imaging the same subject with an array camera 3bb having nine eyes, and the nine first to ninth shown in FIG. 7 described above.
- Original image data IS is generated by imaging the same subject with an array camera 3bb having nine eyes, and the nine first to ninth shown in FIG. 7 described above.
- Original image data IS is generated by imaging the same subject with an array camera 3bb having nine eyes, and the
- the chromaticity component interpolation processing unit 173 performs interpolation processing based on the luminance component image data BIS and the chromaticity component image data CIS to generate color-specific chromaticity component image data ICIS (S53), and the image of the demosaic processing unit 17
- the generation processing unit 174 generates an image data ID based on the luminance component image data BIS and the color-specific chromaticity component image data ICIS (S54).
- Each of these processes S51 to S54 is executed for each of the first to ninth original image data IS-11 to IS-33. More specifically, each of these processes S51 to S54 is performed as follows.
- the luminance component generation processing unit 171 In luminance component generation processing S51 for generating a luminance component, the luminance component generation processing unit 171 generates luminance component image data BIS by filtering the original image data IS with a predetermined second image filter FL2. In this luminance component generation processing S51, one second image filter FL2 is used, and the luminance component generation processing unit 171 performs luminance component image filtering by filtering the original image data IS with this one second image filter FL2.
- the data BIS may be generated, in the present embodiment, a plurality of second image filters FL2n are used, and the luminance component generation processing unit 171 filters the original image data IS with each of the plurality of second image filters FL2n.
- a plurality of elementary luminance component image data BISp is generated, and a plurality of elementary luminance component image data BISp is synthesized at a predetermined synthesis ratio, thereby generating one luminance component image data BIS.
- two 2-1 and 2-2 image filters FL2a and FL2b of the first and second modes shown in FIGS. 23B and 23C are used, and the luminance component generation processing unit 171
- First original luminance component image data BISp-1 is generated by filtering the original image data IS (FIG. 8A) with the 2-1 image filter FL2a of the first aspect (FIG. 23B).
- the second elementary luminance component image data BISp-2 is generated by filtering with the second-second image filter FL2b of the embodiment (FIG. 23C), and these first and second elementary luminance component image data BISp-1, BISp-2 are generated.
- -2 is synthesized at a synthesis ratio of 1: 4 to generate one luminance component image data BIS (FIG. 23D).
- the 2-1 and 2-2 image filters FL2a and FL2b of the first and second modes respectively correspond to pixels (peripheral pixels) located around the pixel (target pixel) corresponding to the filtering target data.
- Filter coefficients so as to convert the target data into data having the same color composition ratio as that of the unit array (unit color filter array 321) in the original image data IS based on the data to be performed (peripheral data)
- Each value is adjusted and set.
- each value of the filter coefficient in the second image filter FL2 is adjusted by taking into consideration the degree of the high frequency component remaining in the luminance component image data BIS (elementary luminance component image data BISp), and filtering.
- the size (size) of the second image filter FL2 is adjusted by considering a pixel range or the like located around the target pixel that affects the target pixel.
- the 2-1 image filter FL2a according to the first aspect is a filter having a filter coefficient that is symmetric with respect to its center position. That is, as shown in FIG. 23B, the filter coefficient located at the center 6 rows and 6 columns is 104/128, and is located at 5 rows 6 columns, 6 rows 5 columns, 6 rows 7 columns, and 7 rows 6 columns, respectively.
- Each filter coefficient is 7/128, 1 row 5 columns, 1 row 7 columns, 3 rows 3 columns, 3 rows 4 columns, 3 rows 6 columns, 3 rows 8 columns, 3 rows 9 columns, 4 rows 3 Column, 4 rows, 9 columns, 5 rows, 1 column, 5 rows, 5 columns, 5 rows, 7 columns, 5 rows, 11 columns, 7 rows, 1 column, 7 rows, 5 columns, 7 rows, 7 columns, 7 rows, 11 columns, 8 rows, 3
- Each filter coefficient located at 1/128, 4 rows 6 columns, 6 rows 4 columns, 6 rows 8 columns and 8 rows 6 columns is -9/128, 4 rows 5 columns, 4 rows 7 columns 5 rows 4 columns 5 rows 8 columns 7 rows 4 columns 7 rows
- Each filter coefficient located in each column, 8 rows 5 columns and 8 rows 7 columns is 3/128, and each filter coefficient located in 4 rows 4 columns, 4 rows 8 columns, 8 rows 4 columns
- the filter coefficient is ⁇ 5/128, 3 rows and 5 columns, 3 rows and 7 columns, 5 rows and 3 columns, 5 rows and 9 columns, 7 rows and 3 columns, 7 rows and 9 columns, 9 rows and 5 columns, and 9 rows and 7 columns.
- Each filter coefficient located in each is 2/128, 2 rows 4 columns, 2 rows 8 columns, 4 rows 2 columns, 4 rows 10 columns, 8 rows 2 columns, 8 rows 10 columns, 10 rows 4 columns and
- Each filter coefficient located in each of 10 rows and 8 columns is -1/128, and each filter coefficient located in each of 2 rows and 6 columns, 6 rows and 2 columns, 6 rows and 10 columns, and 10 rows and 6 columns is -2
- Each filter coefficient located at each remaining position is 0/128.
- FIG. 23B the numerator of the value of the filter coefficient is described at each position of the matrix display, and the denominator is described collectively at the lower right outside the matrix display.
- FIG. 23C is also described in a similar description format. Since the 2-1 image filter FL2a of the first aspect having such a filter coefficient has a relatively small removal rate for removing high frequency components by the filtering, luminance component image data BIS (elementary luminance component image data BISp) In the luminance component image data BIS (elementary luminance component image data BISp), a relatively high sharpness is maintained.
- luminance component image data BIS elementary luminance component image data BISp
- the 2-1 image filter FL2a has a filter size of 11 ⁇ 11, the pixel range that affects the target pixel to be filtered is relatively wide and relatively far from the target pixel.
- the target pixel is affected up to the pixel located at, and the luminance component image data BIS (elementary luminance component image data BISp) includes a relatively large amount of error. For this reason, the jagged line mentioned later is comparatively easy to come out.
- the 2-2 image filter FL2b in the second mode is a filter having a filter coefficient that is symmetric with respect to the center position, and is the same as the image filter FL1a in the first mode as shown in FIG. 23C. is there.
- the brightness component image data BIS (elemental brightness component image data BISp) has a relatively small error, and a jagged line is relatively difficult to appear. Therefore, as described above, when the luminance component image data BIS is generated using one second image filter FL2, the 2-2 image filter FL2b of such a second mode is used.
- the filter coefficients of the second image filter FL2 are such that when the unit arrays are overlapped over the entire area of the image filter FL, the sum of the filter coefficients at the same array position of the unit arrays becomes the same value at all array positions. May be set.
- Filtering is executed sequentially from data corresponding to the pixels located in the first row and first column in the original image data IS to data corresponding to the pixels located in the last 500 rows and 500 columns, as described above.
- the second image filter FL2 Since the second image filter FL2 has a filter coefficient having the same color composition ratio as that of the unit array in the original image data IS, the original image data is obtained when the second image filter FL2 is applied to the original image data IS.
- the total of the filter coefficients corresponding to the R pixels of IS, the total of the filter coefficients corresponding to the B pixels of the original image data IS, and the total of the filter coefficients corresponding to the G pixels of the original image data IS are the original image data IS.
- the data of any pixel in the unit array of the original image data IS is filtered with the same color composition ratio by the second image filter FL2. Therefore, the data of all the pixels in the original image data IS are filtered with the same color composition ratio by the second image filter FL2.
- the data of all pixels in the luminance component image data BIS (elementary luminance component image data BISp) can be handled as data having the same color component synthesized by (G / 2 + R / 4 + B / 4).
- each data (each pixel value) of the luminance component image data BIS corresponds to the position of the data (the position of the pixel). It is obtained by synthesizing each data value of each elementary luminance component image data BISp at a position with a predetermined synthesis ratio. For example, m rows and n columns of data (each pixel value) DV BIS, mn in the luminance component image data BIS synthesize each data value of m rows and n columns in each elementary luminance component image data BISp with the predetermined synthesis ratio. Is required.
- the predetermined synthesis ratio is set in advance to an appropriate ratio by actually synthesizing the original images of a plurality of samples while changing the synthesis ratio, and comparing the images after synthesis. : 4. Therefore, the data value of m rows and n columns in the elementary luminance component image data BISp-1 obtained by applying the 2-1 image filter FL2a of the first mode to the original image IS is DV BISp-1, mn , When the data value of m rows and n columns in the elementary luminance component image data BISp-2 obtained by applying the 2-2 image filter FL2b of the second mode to the original image IS is DV BISp-2, mn , The data of m rows and n columns in the luminance component image data BIS is (4 ⁇ DV BISp ⁇ 1, mn + 1 ⁇ DV BISp ⁇ 2, mn ) / (1 + 4).
- the 2-1 image filter FL2a As described above, the 2-1 image filter FL2a according to the first aspect generates elementary luminance component image data BISp-1 in which the sharpness is relatively maintained while the jagged line is relatively easily generated. Elemental luminance component image data BISp-1 by the 2-1 image filter FL2a of one aspect and elementary luminance component by the 2-2 image filter FL2b of the second aspect having characteristics opposite to those of the 2-1 image filter FL2a By combining the image data BISp-2 and combining them at a predetermined combining ratio to generate the luminance component image data BIS, the luminance component image data BIS with few jagged lines can be generated while maintaining sharpness.
- the second image filter FL2 is not limited to the 2-1 and 2-2 image filters FL2a and FL2b of the first and second modes described above, and various modes can be adopted.
- the original image data IS obtained by the color filter unit 32a having the Bayer array as the unit color filter array the above-described third to fifth image filters FL1b to FLd shown in FIGS. 5B to 5D can be used.
- the chromaticity component generation processing unit 172 in the chromaticity component generation processing S52 for generating chromaticity components, the chromaticity component generation processing unit 172, as shown in FIG. 24, from the original image data IS (FIG. 24A) as described above.
- the luminance component image data CIS (FIG. 24C) is generated by subtracting the luminance component image data BIS (FIG. 24B) generated by the luminance component generation processing unit 171 for each pixel. That is, m rows and n columns of data (pixel values) DV CIS, mn in the chromaticity component image data CIS are obtained from the m rows and n columns of data values DV IS, mn in the original image data IS in the luminance component image data BIS.
- FIG. 24C the chromaticity component image data CIS is shown for each color.
- G chromaticity component image data CIS-G of the G chromaticity component is shown, and in the middle part of FIG. 24C.
- FIG. 24C shows R chromaticity component image data CIS-R of the R chromaticity component
- B chromaticity component image data CIS-B of the B chromaticity component is shown in the lower part of FIG. 24C.
- “0” is set as the data value of the pixel in which the data value (pixel value) is missing.
- the chromaticity component interpolation processing unit 173 in the chromaticity component interpolation processing S53 for interpolating the chromaticity component missing in each pixel, the chromaticity component interpolation processing unit 173 generates the chromaticity component generated by the chromaticity component generation processing unit 172.
- the chromaticity component data of the missing pixel is interpolated based on the number of the same color included in the unit array and the luminance component image data BIS generated by the luminance component generation processing unit 171.
- the chromaticity component image data ICIS for each color for each of the plurality of colors is generated.
- the chromaticity component interpolation processing unit 173 uses a bilinear interpolation method as the interpolation method when interpolating chromaticity component data of a color having the same number of colors included in the unit array.
- the luminance component image data generated by the luminance component generation processing unit 171 is used. Is used as the interpolation method.
- the unit array has one R (red) luminance data, and the R chromaticity
- the component R chromaticity component image data CIS-R has data values (pixel values) only at a ratio of one pixel to four pixels, as shown in FIG. 24C. For this reason, even if the data value of the pixel in which the R chromaticity component data is missing is subjected to bilinear interpolation, a jagged line is unlikely to occur.
- the density data for B (green) is the same as the density data for R described above.
- the chromaticity component interpolation processing unit 173 performs the R color in the R chromaticity component image data CIS-R of the chromaticity component image data CIS generated by the chromaticity component generation processing unit 172 in step S52.
- the chromaticity component image data ICIS-R for each R color of R (red) is generated.
- the data value of the pixel in which the B chromaticity component data is missing in the B chromaticity component image data CIS-B of the chromaticity component image data CIS generated by the chromaticity component generation processing unit 172 is interpolated by the bilinear interpolation method.
- chromaticity component image data ICIS-B for each B color of B (blue) is generated.
- This bilinear interpolation has a filter coefficient 1/4 at each position of 1 row 1 column, 1 row 3 column, 3 rows 1 column and 3 rows 3 columns, 1 row 2 columns, 2 rows 1 column, 2 rows 3 columns.
- the chromaticity component interpolation processing unit 173 filters the R chromaticity component image data CIS-R with the RB bilinear image filter, thereby performing the R color-specific chromaticity component image data ICIS-.
- R is generated, and similarly, the B chromaticity component image data ICIS-B is generated by filtering the B chromaticity component image data CIS-B with the RB bilinear image filter.
- the unit array includes two G (green) density data, and the G chromaticity
- the interpolation method uses a pixel in which data of the G chromaticity component is missing as a target pixel, and the luminance corresponding to the first pixel position of the target pixel on the G chromaticity component image data CIS-G.
- the selection is made according to whether or not there is a correlation between the first luminance component data at the second pixel position on the component image data BIS and the second luminance component data at the peripheral pixel position located around the second pixel position. More specifically, when there is a correlation, weighting is performed by weighting and averaging the G chromaticity component data of peripheral pixel positions located around the first pixel position with a weight according to the degree of correlation.
- An interpolation method is used as the interpolation method, and when there is no correlation, a bilinear interpolation method is used as the interpolation method. For example, as shown in FIG.
- a pixel lacking G color component data is a pixel of interest X
- pixels adjacent to the pixel of interest X in the column direction are peripheral pixels A and D, respectively.
- the adjacent pixels in the row direction are the peripheral pixels B and C
- the luminance component data of the pixel of interest X in the luminance component image data BIS is BDx
- the peripheral pixels A, B, C, and D in the luminance component image data BIS are Luminance component data is BDa, BDb, BDc, BDd
- G chromaticity component data of each peripheral pixel A, B, C, D in G chromaticity component image data CIS-G is CDGa, CDGb, CDGc, CDGd, weight As shown in FIG.
- ) linearly changes, and when the weight W (d) decreases as the luminance difference d increases, the correlation Wt (X; A, B, C, D) is obtained by the following formula I.
- the Th is a value that defines a range in which a weight is given, and is 10 or the like, for example.
- Wt (X; A, B, C, D) Wt (
- CDGx (Wtxa ⁇ CDGa + Wtxb ⁇ CDGb + Wtxc ⁇ CDGc + Wtxd ⁇ CDGd) / (Wtxa + Wtxb + Wtxc + Wtxd) (Formula II)
- ) Wtxa
- ) Wtxb
- ) Wtxc
- ) Wtxd did.
- the image processing apparatus D can execute the demosaic process so as to be super-resolution.
- CDGx (CDGa + CDGb + CDGc + CDGd) / 4 (formula III)
- the image generation processing unit 174 in the image generation processing S54 for generating image data, the image generation processing unit 174, as shown in FIG. 26, the chromaticity for each color for each of the plurality of colors generated by the chromaticity component interpolation processing unit 173.
- the image data ID is generated by adding the luminance component image data BIS generated by the luminance component generation processing unit 171 to each component image data ICIS for each pixel. More specifically, m rows and n columns of data (pixel values) DV ID-G, mn in the G component image data ID-G are represented by m rows and n in the luminance component image data BIS as shown in FIG. 26A.
- m-row n-column data each pixel value
- DV ID-R, mn in R component image data ID-R is m-row n-column data in luminance component image data BIS as shown in FIG. 26B.
- m row and n column data each pixel value
- DV ID-B, mn in the B component image data ID-B is m row and n column data in the luminance component image data BIS as shown in FIG. 26C.
- an image data ID having each color component of the G component DV ID-R, mn , the R component DV ID-R, mn, and the B component DV ID-B, mn is generated.
- the image processing apparatus D can further reduce jagged lines that are generated when a relatively high-resolution image is generated from a plurality of relatively low-resolution images.
- this jagged line will be described below with reference to FIG.
- a smooth straight line or curved line may become a jagged line.
- FIG. 27A when subjects having different densities in a stepwise manner on a linear boundary are imaged on the individual eyes 4 and 5 in FIG. 7, as shown in FIGS.
- the image sensor Each photoelectric conversion element (each pixel) of the (image sensor) may have an electric signal of a level (size) corresponding to the amount of light received over the entire area of the light receiving surface, that is, the density may vary within the light receiving surface. Even if an electrical signal representing average density information is output, even at the same position (positions of m rows and n columns) on the image sensor (image sensor), the base line length and the distance to the subject are determined. The subject is imaged with a shift.
- 27B and 27C show R (red) pixels (pixels that generate R component density data) in an image sensor with a Bayer array, and the left-side diagrams show R photoelectrics for photoelectric conversion of R light.
- the level of the electric signal to be output is indicated by shading. Actually, a nine-eye image is used. However, for convenience of explanation, if a higher-resolution image is generated from the images of the individual eyes 4 and 5, as shown in FIG. Even for pixels on the same line, pixel values are generated by electrical signals captured by different cameras.
- the pixel value of the pixel Px1 is generated by an electrical signal captured by the left camera, while the pixel value of the pixel Px2 on the same line as the pixel Px1 is generated by the electrical signal captured by the right camera.
- these electric signals represent average density information. If the levels are different, the generated high-resolution image has a density difference as shown in FIG. 27D.
- the curve becomes a jagged line.
- the generation of such a jagged line is, for example, information inherent to the subject that R component density data is obtained discretely for each eye and has a constant density in the horizontal direction. This is because cannot be obtained.
- the original image data IS is filtered by the second image filter FL2 so as to convert the original image data IS into data having the same color composition ratio as that of the unit array, this filtering is performed.
- Data of all pixels in the generated luminance component image data BIS can be handled as data having the same color component, and can be appropriately compared with all pixels.
- the interpolation method based on the number of the same color included in the unit array and the luminance component image data BIS generated by the luminance component generation processing unit 171 is used for the interpolation of the demosaic process. It is possible to select an appropriate interpolation method in consideration of the mutual relationship. As a result, the image processing apparatus D including the demosaic processing unit 17 can further reduce the jagged lines.
- the image processing apparatus D provided with the demosaic processing unit 17 has By using a bilinear interpolation method that is relatively easy to process in such a case, it is possible to easily perform the interpolation process while further reducing the jagged lines.
- Interpolating the chromaticity component data of a plurality of the same colors included in the unit array by the bilinear interpolation method generates the jagged line.
- the image processing apparatus D or the like including the demosaic processing unit 17 is configured as described above. Since the chromaticity component data of a plurality of the same colors included in the unit array is interpolated by an interpolation method based on the luminance component image data, the jagged lines can be reduced more appropriately. Since the interpolation method is selected according to the presence or absence of the correlation, a more appropriate interpolation method can be selected.
- the image processing apparatus D or the like in the present embodiment can more appropriately reduce the jagged lines when interpolating chromaticity component data of a color having a plurality of the same colors included in the unit array.
- the image processing apparatus D and the like including the demosaic processing unit 17 can treat all pixel data as data having more equivalent color components because the values of the filter coefficients are arranged symmetrically with respect to the center position. Therefore, the jagged lines can be further reduced.
- the image processing apparatus D or the like including the demosaic processing unit 17 generates the luminance component image data BIS using the 2-1 and 2-2 image filters FL2a and FL2b having different image filter sizes and high-frequency component removal rates. Therefore, the luminance component image data BIS with few jagged lines can be generated while maintaining the sharpness of the image.
- luminance component generation processing unit 171 may also be used as the preprocessing unit 12.
- An image processing apparatus includes a pre-processing unit that generates post-processed image data by filtering original image data with an image filter, and a plurality of pre-processing units generated from the plurality of original image data by the pre-processing unit.
- a corresponding point search processing unit that searches for a corresponding point among a plurality of images corresponding to each processed image data, and the original image data is a plurality corresponding to each of a plurality of pixels arranged in a two-dimensional array.
- Corresponding point search searches for corresponding points by pattern matching, so generally only data of the same color component can be used. For this reason, for example, when the corresponding point search is performed on the original image data obtained by using the color filter of the Bayer array, the corresponding point search is performed on the data of G pixels, for example, as in Non-Patent Document 1. Therefore, only half the amount of original image data can be used.
- the luminance component generation processing unit filters all the pixels. The data can be handled as data having the same color component, and as a result, the corresponding point search can be performed using the data of all the pixels. Therefore, such an image processing apparatus can perform the corresponding point search with higher accuracy.
- each filter coefficient of the image filter is symmetric with respect to a center position.
- each filter coefficient of the image filter is a sum of filter coefficients at the same array position of the unit array when the unit array is overlaid on the entire area of the image filter. Is set to have the same value at all array positions.
- Such an image processing apparatus can handle the data of all the pixels as data having more equivalent color components, and therefore can reduce the jagged lines.
- the plurality of original image data correspond to a plurality of three or more imaging optical systems and the plurality of imaging optical systems, and each of the plurality of imaging optical systems. Is an image captured by an array camera having three or more eyes, each of which includes a plurality of imaging units that respectively capture optical images of the subject imaged by the above.
- Such an image processing apparatus uses an array camera with three or more eyes, a plurality of original image data can be obtained simultaneously. For this reason, when obtaining the positional deviation amount (parallax) between corresponding points from the corresponding point search result, the positional deviation amount is obtained between a plurality of images obtained by copying the same subject at the same time. Such an image processing apparatus can obtain the positional deviation amount with higher accuracy.
- the plurality of original image data includes at least one of an imaging optical system and an imaging unit that captures an optical image of a subject imaged by the imaging optical system. Data captured at different times by a single camera.
- Such an image processing apparatus can obtain a plurality of original image data captured at different times.
- the above-described image processing device further includes a positional deviation amount calculation unit that obtains a positional deviation amount that is a distance between corresponding points searched by the corresponding point search processing unit.
- Such an image processing apparatus further includes a positional deviation amount calculation unit, it is possible to obtain the positional deviation amount between corresponding points searched by the corresponding point search processing unit, and corresponding points executed with higher accuracy. The result of the search can be effectively reflected in the estimation of the positional deviation amount.
- a high-resolution image higher than the resolution in the image of the original image data from the plurality of original image data based on the position shift amount obtained by the position shift amount calculation unit is further provided.
- Such an image processing apparatus further includes a high resolution processing unit, a positional shift amount between corresponding points searched by the corresponding point search processing unit is obtained, and based on this, a high resolution is obtained from a plurality of original image data. Image data can be obtained, and the result of corresponding point search performed with higher accuracy can be effectively reflected in the generation of a high-resolution image.
- the positional shift amount obtained by the positional shift amount calculation unit is set as a parallax between the plurality of images corresponding to the plurality of original image data, and based on the parallax It further includes a distance calculation unit for obtaining the distance to the subject represented by the corresponding point.
- Such an image processing apparatus further includes a distance calculation unit, it is possible to obtain a parallax (position shift amount) between corresponding points searched by the corresponding point search processing unit, and to perform the response executed with higher accuracy.
- the result of the point search can be effectively reflected in the distance calculation.
- the above-described image processing apparatus further includes a demosaicing processing unit that generates at least one color of image data having each color component data for each pixel by demosaicing the original image data.
- a luminance component generation processing unit that generates luminance component image data by filtering data with a second image filter, and luminance component image data generated by the luminance component generation processing unit from the original image data, for each pixel.
- the chromaticity component generation processing unit that generates chromaticity component image data by subtraction, and the chromaticity component image data generated by the chromaticity component generation processing unit, pixels that are missing for each of the plurality of colors Chromaticity component data of the same color and the luminance component image data generated by the luminance component generation processing unit.
- a chromaticity component interpolation processing unit that generates chromaticity component image data for each color for each of the plurality of colors, and for each of the plurality of colors generated by the chromaticity component interpolation processing unit.
- An image generation processing unit that generates the image data by adding the luminance component image data generated by the luminance component generation processing unit to each of the pixels, for each pixel.
- the second image filter has the same color composition as the color composition ratio of the unit array based on data corresponding to pixels located around a pixel corresponding to target data that is data to be filtered
- a filter coefficient for converting to data having a ratio is provided.
- the image processing apparatus since the original image data is filtered by the image filter so as to convert the original image data into data having the same color composition ratio as that of the unit array, it is generated by this filtering. Data of all pixels in the luminance component image data thus obtained can be handled as data having the same color component, and can be appropriately compared with all pixels.
- the interpolation method based on the number of the same color included in the unit array and the luminance component image data generated by the luminance component generation processing unit is used for the interpolation of the demosaic process. Accordingly, it is possible to select an appropriate interpolation method in consideration of the mutual relationship between them, and as a result, the image processing apparatus can further reduce the jagged lines accompanying the increase in resolution.
- An image processing method is generated by the preprocessing step of generating the preprocessed image data by filtering the original image data with an image filter, and the preprocessing step from a plurality of original image data.
- a corresponding point search processing step for searching corresponding points between a plurality of images corresponding to a plurality of pre-processed image data, wherein the original image data corresponds to each of a plurality of pixels arranged in a two-dimensional array.
- a plurality of unit arrangements in which a plurality of density data of a plurality of different colors are arranged in a predetermined pattern, and the plurality of data arranged in a two-dimensional array, and the image filter includes: Based on the data corresponding to the pixels located around the pixel corresponding to the target data that is the data to be filtered, the target data is A filter coefficient for converting the data having the same color composition ratio as the color composition ratio of the array.
- An image processing program includes a preprocessing unit that generates post-processed image data by filtering original image data with an image filter, and the preprocessing unit from a plurality of original image data.
- a plurality of unit arrays each of which is a plurality of data corresponding to each of a plurality of pixels arranged in an array and in which a plurality of density data of different colors are arranged in a predetermined pattern, are further arranged in a two-dimensional array
- the image filter includes a plurality of data, and the image filter is an image located around a pixel corresponding to target data that is data to be filtered. Based on the corresponding data, a filter coefficient for converting the target data, the data having the same color composition ratio as the color composition ratio of the unit sequence.
- the luminance component generation processing unit performs filtering so as to convert the original image data into data having the same color composition ratio as the color composition ratio of the unit array by the image filter. All the pixel data can be handled as data having the same color component, and as a result, the corresponding point search can be performed using the data of all the pixels. Therefore, such an image processing method and image processing program can perform corresponding point search with higher accuracy.
- an image processing technique an image processing method, and an image processing program for processing an image can be provided.
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Color Television Image Signal Generators (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
Abstract
This image processing device, method and program filter data of multiple original images with an image filter to generate multiple sets of preprocessed image data for use in corresponding point searches. The original image data is data corresponding to each pixel, and comprises multiple unit arrays arranged in a two-dimensional array, each unit array comprising concentration data of multiple colors arranged in a pattern. The image filter has filter coefficients which, on the basis of data corresponding to pixels peripheral to a pixel corresponding to the filtering target data, convert the target data to data having the same color composition ratio as the color composition ratio of the unit arrays.
Description
本発明は、画像を処理する画像処理技術に関し、特に、より高精度に対応点探索を実施できる画像処理技術に関する。
The present invention relates to an image processing technology for processing an image, and more particularly to an image processing technology capable of performing corresponding point search with higher accuracy.
近年、各種デジタル技術の進展に伴い、例えばCCD(Charge Coupled Device)型イメージセンサやCMOS(Complementary Metal Oxide Semiconductor)型イメージセンサ等の固体撮像素子を用いることによってデジタル画像が比較的容易に得られるようになり、このデジタル画像が様々に利用されている。例えば、複数のデジタル画像から所定の被写体までの距離を計測する画像処理技術や、解像度を変換する画像処理技術等がある。
With the progress of various digital technologies in recent years, digital images can be obtained relatively easily by using solid-state imaging devices such as CCD (Charge Coupled Device) type image sensors and CMOS (Complementary Metal Oxide Semiconductor) type image sensors. This digital image is used in various ways. For example, there are an image processing technique for measuring distances from a plurality of digital images to a predetermined subject, an image processing technique for converting resolution, and the like.
このような解像度を変換する技術の一つに、相対的に低解像度な複数の画像から、相対的に高解像度な1つの画像を合成する高解像度画像生成技術があり、例えば特許文献1や非特許文献1に開示されている。この特許文献1に開示された高解像度カラー画像生成方法は、イメージセンサと色フィルタ配列を用いる画像撮像装置によって撮像された1枚または一連の複数枚の入力画像に基づいて高解像度カラー画像を生成するものである。前記非特許文献1に開示された高解像度画像生成技術は、いわゆるベイヤー配列を用いたイメージセンサで撮像した画像データのうちのG画素のデータを用いて複数の画像間における対応点を探索することで位置ずれを推定し、複数の画像から高解像度な1つの画像を生成するものである。
One of such resolution conversion techniques is a high-resolution image generation technique for synthesizing one relatively high-resolution image from a plurality of relatively low-resolution images. It is disclosed in Patent Document 1. The high-resolution color image generation method disclosed in Patent Document 1 generates a high-resolution color image based on one or a series of input images captured by an image capturing device using an image sensor and a color filter array. To do. The high-resolution image generation technique disclosed in Non-Patent Document 1 searches for corresponding points between a plurality of images using G pixel data of image data captured by an image sensor using a so-called Bayer array. The position shift is estimated by the above, and one high-resolution image is generated from a plurality of images.
ところで、前記ベイヤー配列のイメージセンサは、2行2列に2次元アレイ状に配列したG(緑色)、R(赤色)、B(青色)およびG(緑色)の4個の色フィルタを単位配列として、複数の単位配列をさらに2次元アレイ状に配列したカラーフィルタと、物体(被写体)の光学像を前記カラーフィルタを介して各光電変換素子(各画素)で受光するイメージセンサとを備えたものであるが、前記非特許文献1に開示された画像処理技術は、対応点の探索にG画素のデータ、すなわち、画像データのうちの半分のデータしか用いていない。このため、対応点探索の精度に限界が生じていた。
By the way, the Bayer array image sensor has a unit array of four color filters of G (green), R (red), B (blue) and G (green) arranged in a two-dimensional array in two rows and two columns. And a color filter in which a plurality of unit arrays are further arranged in a two-dimensional array, and an image sensor that receives an optical image of an object (subject) by each photoelectric conversion element (each pixel) via the color filter. However, the image processing technique disclosed in Non-Patent Document 1 uses only G pixel data, that is, half of the image data, for searching for corresponding points. For this reason, there has been a limit to the accuracy of the corresponding point search.
本発明は、上述の事情に鑑みて為された発明であり、その目的は、より高精度に対応点探索を実施できる画像処理装置、画像処理方法および画像処理プログラムを提供することである。
The present invention has been made in view of the above-described circumstances, and an object thereof is to provide an image processing apparatus, an image processing method, and an image processing program capable of performing corresponding point search with higher accuracy.
本発明にかかる画像処理装置、画像処理方法および画像処理プログラムでは、複数の原画像データそれぞれを画像フィルタでフィルタリングすることによって、対応点探索に利用される複数の前処理後画像データそれぞれが生成される。その前記原画像データは、2次元アレイ状に配列された複数の画素それぞれに対応する複数のデータであって、互いに異なる複数色の複数の濃度データを所定のパターンで配列した単位配列を、複数、さらに2次元アレイ状に配列した前記複数のデータを備え、そして、前記画像フィルタは、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、前記単位配列の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備える。したがって、本発明にかかる画像処理装置、画像処理方法および画像処理プログラムは、より高精度に対応点探索を実施できる。
In the image processing apparatus, the image processing method, and the image processing program according to the present invention, each of a plurality of original image data is filtered by an image filter to generate a plurality of preprocessed image data used for corresponding point search. The The original image data is a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array, and a plurality of unit arrays in which a plurality of density data of different colors are arranged in a predetermined pattern, The image filter further includes the plurality of data arranged in a two-dimensional array, and the image filter is based on data corresponding to pixels located around a pixel corresponding to target data that is data to be filtered. A filter coefficient is provided for converting the data into data having the same color composition ratio as that of the unit array. Therefore, the image processing apparatus, the image processing method, and the image processing program according to the present invention can perform corresponding point search with higher accuracy.
上記並びにその他の本発明の目的、特徴及び利点は、以下の詳細な記載と添付図面から明らかになるであろう。
The above and other objects, features and advantages of the present invention will become apparent from the following detailed description and the accompanying drawings.
以下、本発明にかかる実施の一形態を図面に基づいて説明する。なお、各図において同一の符号を付した構成は、同一の構成であることを示し、適宜、その説明を省略する。本明細書において、総称する場合には添え字を省略した参照符号で示し、個別の構成を指す場合には添え字を付した参照符号で示す。
Hereinafter, an embodiment according to the present invention will be described with reference to the drawings. In addition, the structure which attached | subjected the same code | symbol in each figure shows that it is the same structure, The description is abbreviate | omitted suitably. In this specification, when referring generically, it shows with the reference symbol which abbreviate | omitted the suffix, and when referring to an individual structure, it shows with the reference symbol which attached the suffix.
実施形態における画像処理装置は、所与の原画像データに対し所定の画像処理を実施する装置であり、前記所定の画像処理には、少なくとも前処理および対応点探索処理が含まれる。一般に、撮像装置(カメラ)は、イメージセンサから出力される画像信号に対し、ホワイトバランス処理、フィルタ処理、階調変換処理および色空間変換処理等のいわゆる通常の画像処理を施し、最終的な画像信号を生成する。前記原画像データは、このような通常の画像処理を施す前の、イメージセンサから出力された生の画像信号であり、2次元アレイ状に配列された複数の画素それぞれに対応する複数のデータであって、互いに異なる複数色の複数の濃度データを所定のパターンで配列した単位配列を、複数、さらに2次元アレイ状に配列した前記複数のデータを備えるものである。このような原画像データは、例えば、ローデータ(Raw Data、生データ)である。前記前処理は、原画像データにおける各画素の各データを、対応点探索する上で同等に処理可能なデータに変換する処理である。前記対応点探索は、2個の画像に対し、一方を基準画像とし他方を参照画像とした場合に、基準画像上における所定の点に対応する参照画像上における点を探索する処理である。
The image processing apparatus in the embodiment is an apparatus that performs predetermined image processing on given original image data, and the predetermined image processing includes at least preprocessing and corresponding point search processing. In general, an imaging device (camera) performs so-called normal image processing such as white balance processing, filter processing, gradation conversion processing, and color space conversion processing on an image signal output from an image sensor, thereby obtaining a final image. Generate a signal. The original image data is a raw image signal output from an image sensor before performing such normal image processing, and is a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array. A plurality of unit arrays in which a plurality of density data of a plurality of colors different from each other are arranged in a predetermined pattern, and the plurality of data arranged in a two-dimensional array are provided. Such original image data is, for example, raw data (raw data). The preprocessing is processing for converting each data of each pixel in the original image data into data that can be processed equally when searching for corresponding points. The corresponding point search is a process of searching for a point on a reference image corresponding to a predetermined point on the standard image when one of the two images is a standard image and the other is a reference image.
このような実施形態における画像処理装置Dは、画像処理内容に応じて様々な態様が可能であるが、ここでは、複数の画像(画像データ)から前記画像よりも解像度の高い高解像度な画像を生成する第1実施形態の画像処理装置Da、および、複数の画像から所定の被写体までの距離を求める第2実施形態の画像処理装置Dbについて説明する。
The image processing apparatus D in such an embodiment can have various modes depending on the contents of image processing. Here, a high-resolution image having a higher resolution than the image is selected from a plurality of images (image data). The image processing device Da according to the first embodiment to be generated and the image processing device Db according to the second embodiment for obtaining the distance from a plurality of images to a predetermined subject will be described.
(第1実施形態)
図1は、第1実施形態における画像処理装置の構成を示すブロック図である。図2は、第1実施形態の画像処理装置における、カメラである場合の画像取得部の構成を示す図である。図3は、第1実施形態の画像処理装置における、アレイカメラである場合の画像取得部の構成を示す斜視図である。図4は、前記カメラまたは前記アレイカメラにおける色フィルタの単位色フィルタ配列を示す図である。図4Aないし図4Eは、単位色フィルタ配列の第1ないし第5態様を示す。図5は、第1実施形態の画像処理装置における画像フィルタの構成を示す図である。図5Aないし図5Eは、画像フィルタの第1ないし第5態様を示す。 (First embodiment)
FIG. 1 is a block diagram illustrating the configuration of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram illustrating a configuration of an image acquisition unit in the case of a camera in the image processing apparatus according to the first embodiment. FIG. 3 is a perspective view illustrating a configuration of an image acquisition unit in the case of an array camera in the image processing apparatus according to the first embodiment. FIG. 4 is a diagram illustrating unit color filter arrays of color filters in the camera or the array camera. 4A to 4E show first to fifth modes of the unit color filter array. FIG. 5 is a diagram illustrating a configuration of an image filter in the image processing apparatus according to the first embodiment. 5A to 5E show first to fifth aspects of the image filter.
図1は、第1実施形態における画像処理装置の構成を示すブロック図である。図2は、第1実施形態の画像処理装置における、カメラである場合の画像取得部の構成を示す図である。図3は、第1実施形態の画像処理装置における、アレイカメラである場合の画像取得部の構成を示す斜視図である。図4は、前記カメラまたは前記アレイカメラにおける色フィルタの単位色フィルタ配列を示す図である。図4Aないし図4Eは、単位色フィルタ配列の第1ないし第5態様を示す。図5は、第1実施形態の画像処理装置における画像フィルタの構成を示す図である。図5Aないし図5Eは、画像フィルタの第1ないし第5態様を示す。 (First embodiment)
FIG. 1 is a block diagram illustrating the configuration of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram illustrating a configuration of an image acquisition unit in the case of a camera in the image processing apparatus according to the first embodiment. FIG. 3 is a perspective view illustrating a configuration of an image acquisition unit in the case of an array camera in the image processing apparatus according to the first embodiment. FIG. 4 is a diagram illustrating unit color filter arrays of color filters in the camera or the array camera. 4A to 4E show first to fifth modes of the unit color filter array. FIG. 5 is a diagram illustrating a configuration of an image filter in the image processing apparatus according to the first embodiment. 5A to 5E show first to fifth aspects of the image filter.
この第1実施形態における画像処理装置Daは、例えば、図1に示すように、制御処理部1aと、記憶部2と、画像取得部3と、入力部4と、出力部5とを備える。
The image processing device Da according to the first embodiment includes, for example, a control processing unit 1a, a storage unit 2, an image acquisition unit 3, an input unit 4, and an output unit 5, as shown in FIG.
画像取得部3は、制御処理部1aに接続され、原画像データを外部から当該画像処理装置Daに取り込むための装置である。画像取得部3は、この取得した画像データを制御処理部1aへ出力する。画像取得部3は、例えば、原画像データを記録した記録媒体から前記原画像データを読み込むドライブ装置3aや、物体(被写体)の光学像を撮像して原画像データを生成するカメラ3b等である。
The image acquisition unit 3 is connected to the control processing unit 1a and is a device for taking original image data into the image processing device Da from the outside. The image acquisition unit 3 outputs the acquired image data to the control processing unit 1a. The image acquisition unit 3 is, for example, a drive device 3a that reads the original image data from a recording medium on which the original image data is recorded, a camera 3b that captures an optical image of an object (subject) and generates original image data. .
ドライブ装置3aは、例えば、フレキシブルディスク、CD-ROM、CD-RおよびDVD-R等の記録媒体とデータを読み書きする装置であり、例えば、フレキシブルディスクドライブ、CD-ROMドライブ、CD-RドライブおよびDVD-Rドライブ等である。また例えば、ドライブ装置3aは、SDメモリカード(SDのロゴは登録商標)およびUSB(Universal Serial Bus)メモリ等の固体記憶素子からデータを読み書きする装置であり、例えばSDドライブおよびUSBインターフェース等である。
The drive device 3a is a device that reads / writes data from / to a recording medium such as a flexible disk, CD-ROM, CD-R, and DVD-R, for example, a flexible disk drive, a CD-ROM drive, a CD-R drive, and the like. DVD-R drive or the like. Further, for example, the drive device 3a is a device that reads and writes data from a solid-state storage element such as an SD memory card (SD logo is a registered trademark) and a USB (Universal Serial Bus) memory, such as an SD drive and a USB interface. .
カメラ3bは、図2に示すように、被写体の光学像を所定の結像面に結像する撮像光学系31と、撮像光学系31の像側に配置される色フィルタ部32と、前記所定の結像面の位置に受光面が位置するように配置され、前記撮像光学系31によって結像された被写体の光学像を色フィルタ部32を介して撮像して原画像データを出力する撮像部33とを備える。カメラ3bは、単眼の1個のカメラ3baであって良く、また、複眼(3眼以上)のアレイカメラ3bbであって良い。
As shown in FIG. 2, the camera 3b includes an imaging optical system 31 that forms an optical image of a subject on a predetermined imaging plane, a color filter unit 32 disposed on the image side of the imaging optical system 31, and the predetermined An image pickup unit that is arranged so that a light receiving surface is positioned at the position of the image forming plane, and that picks up an optical image of a subject imaged by the image pickup optical system 31 via the color filter unit 32 and outputs original image data 33. The camera 3b may be a single-lens camera 3ba, or may be a compound eye (three or more eyes) array camera 3bb.
単眼のカメラ3baは、被写体を撮像して1個の原画像データを生成する装置であり、例えば、単眼(1個)の撮像光学系31aと、撮像光学系31aの像側に配置される色フィルタ部32と、撮像光学系31aによって結像された被写体の光学像を色フィルタ部32を介して1個の有効撮像領域で撮像して1個の原画像データを出力する撮像部33aとを備える。撮像光学系31aは、その光軸に沿って1または複数の光学レンズを備えて構成される。撮像部33aは、2次元マトリクス状に配列された複数の光電変換素子(複数の画素)を備え、各光電変換素子は、それぞれ、色フィルタ部32を介して受光した光の光量に応じて変換した電気信号を原画像データにおける各画素のデータとして出力する。このような撮像部33aは、例えば、CCD型イメージセンサやCMOS型イメージセンサ等の固体撮像素子を備えて構成される。
The monocular camera 3ba is a device that images a subject and generates one original image data. For example, a monocular (one) imaging optical system 31a and a color arranged on the image side of the imaging optical system 31a. A filter unit 32 and an imaging unit 33a that captures an optical image of a subject formed by the imaging optical system 31a in one effective imaging region via the color filter unit 32 and outputs one original image data. Prepare. The imaging optical system 31a includes one or more optical lenses along the optical axis. The imaging unit 33a includes a plurality of photoelectric conversion elements (a plurality of pixels) arranged in a two-dimensional matrix, and each photoelectric conversion element converts the light according to the amount of light received through the color filter unit 32. The electrical signal is output as data of each pixel in the original image data. Such an imaging unit 33a includes a solid-state imaging device such as a CCD image sensor or a CMOS image sensor.
3眼以上の複眼のアレイカメラ3bbは、被写体を撮像して複数(3個以上)の原画像データを生成する装置であり、例えば、図3に示すように、3個以上の複数の撮像光学系31bと、前記複数の撮像光学系31bに対応し、各撮像光学系31の各像側にそれぞれ配置される複数の色フィルタ部32(図3には不図示)と、前記複数の撮像光学系31bに対応し、前記複数の撮像光学系31bそれぞれによって結像された被写体の光学像を各色フィルタ部32を介してそれぞれ撮像して複数の原画像データを出力する複数の撮像部33bとを備える。なお、図3では色フィルタ部32が省略されている。複数の撮像光学系31bは、それぞれ、当該撮像光学系31bに対応する撮像部33bの受光面に被写体の光学像を結像する光学素子である。1個の撮像光学系31bは、その光軸に沿って1または複数の光学レンズを備えて構成される。複数の撮像光学系31bは、図3に示す例では、各光軸が互いに略平行となるように配列される。したがって、複数の撮像光学系31bおよび複数の色フィルタ部32を介して被写体の光学像をそれぞれ撮像する複数の撮像部33bは、同じ被写体を写した略視差だけずれた原画像データを生成することになる。図3に示す例では、複数の撮像光学系31bは、複数の撮像部32bに対応して線形独立な2方向、より具体的には互いに直交するX方向およびY方向の2方向に2次元マトリクス状に配列されている。図3に示す例では、複数の撮像光学系31bは、撮像部33bの配列態様および個数に合わせて3行3列に2次元マトリクス状に配列された9個の撮像光学系31b-11~31b-33である。複数の撮像部33bは、それぞれ、2次元マトリクス状に配列された複数の光電変換素子(複数の画素)を備え、各光電変換素子は、それぞれ、前記色フィルタ部32を介して受光した光の光量に応じて変換した電気信号を原画像データにおける各画素のデータとして出力する。複数の撮像部33bは、複数の撮像光学系31bに対応し、各撮像面が互いに同一平面となるように配列される。図3に示す例では、複数の撮像部33bは、線形独立な2方向、より具体的には互いに直交するX方向およびY方向の2方向に2次元マトリクス状に配列されている。図3に示す例では、複数の撮像部33bは、3行3列に2次元マトリクス状に配列された9個の撮像部33b-11~33b-33が示されている。これら複数の撮像部33bは、同一の基板上に2次元マトリクス状に配列された複数の前記固体撮像素子を備えて構成されて良いが、図3に示す例では、1個の前記固体撮像素子を備えて構成され、この1個の前記固体撮像素子における有効画素領域が、各撮像部33bに対応するように、2次元マトリクス状に配列された複数の領域に分割され、これら各領域が各撮像部33bとして利用される。
The three or more compound eye array camera 3bb is a device that captures a subject and generates a plurality (three or more) of original image data. For example, as shown in FIG. A plurality of color filter sections 32 (not shown in FIG. 3) corresponding to the system 31b, the plurality of imaging optical systems 31b, respectively disposed on the respective image sides of the imaging optical systems 31, and the plurality of imaging optics. A plurality of imaging units 33b that correspond to the system 31b and that respectively capture optical images of the subject formed by the plurality of imaging optical systems 31b through the color filter units 32 and output a plurality of original image data. Prepare. In FIG. 3, the color filter unit 32 is omitted. Each of the plurality of imaging optical systems 31b is an optical element that forms an optical image of a subject on the light receiving surface of the imaging unit 33b corresponding to the imaging optical system 31b. One imaging optical system 31b includes one or a plurality of optical lenses along the optical axis. In the example shown in FIG. 3, the plurality of imaging optical systems 31 b are arranged so that the optical axes are substantially parallel to each other. Therefore, the plurality of imaging units 33b that respectively capture the optical images of the subject via the plurality of imaging optical systems 31b and the plurality of color filter units 32 generate original image data that is shifted by a substantially parallax showing the same subject. become. In the example shown in FIG. 3, the plurality of imaging optical systems 31b are two-dimensional matrices in two linearly independent directions corresponding to the plurality of imaging units 32b, more specifically, in two directions of X and Y directions orthogonal to each other. Are arranged in a shape. In the example shown in FIG. 3, the plurality of imaging optical systems 31b includes nine imaging optical systems 31b-11 to 31b arranged in a two-dimensional matrix in three rows and three columns in accordance with the arrangement mode and the number of the imaging units 33b. -33. Each of the plurality of imaging units 33b includes a plurality of photoelectric conversion elements (a plurality of pixels) arranged in a two-dimensional matrix, and each photoelectric conversion element receives light received via the color filter unit 32, respectively. The electrical signal converted according to the amount of light is output as data of each pixel in the original image data. The plurality of imaging units 33b correspond to the plurality of imaging optical systems 31b and are arranged so that the imaging surfaces are on the same plane. In the example illustrated in FIG. 3, the plurality of imaging units 33 b are arranged in a two-dimensional matrix in two linearly independent directions, more specifically, in two directions of X and Y directions orthogonal to each other. In the example illustrated in FIG. 3, the plurality of imaging units 33b includes nine imaging units 33b-11 to 33b-33 arranged in a two-dimensional matrix in three rows and three columns. The plurality of imaging units 33b may include a plurality of the solid-state imaging elements arranged in a two-dimensional matrix on the same substrate, but in the example illustrated in FIG. The effective pixel area in the single solid-state imaging device is divided into a plurality of areas arranged in a two-dimensional matrix so as to correspond to each imaging unit 33b. Used as the imaging unit 33b.
単眼のカメラ3baにおける色フィルタ部32および複眼のアレイカメラ3bbにおける複数の色フィルタ部32それぞれは、互いに異なる複数の色フィルタを所定のパターンで配列した単位色フィルタ配列を、複数、さらに2次元アレイ状に配列した光学フィルタ素子である。色フィルタ部32の各色フィルタは、それぞれ、入射した光のうち所定の波長範囲(透過波長帯域)の光を透過し、撮像部33の各光電変換素子に対応するように配置される。すなわち、単眼のカメラ3baの色フィルタ部32における各色フィルタは、撮像部33aにおける複数の光電変換素子(複数の画素)それぞれに対応するように配置される。また、複眼のアレイカメラ3bbにおける複数の色フィルタ部32は、それぞれ、複数の撮像部33bに対応するように配置され、各色フィルタ部32および撮像部33bそれぞれにおいて、色フィルタ部32における各色フィルタは、撮像部33bにおける複数の光電変換素子(複数の画素)それぞれに対応するように配置される。
Each of the color filter unit 32 in the monocular camera 3ba and the plurality of color filter units 32 in the compound eye array camera 3bb includes a plurality of unit color filter arrays in which a plurality of different color filters are arranged in a predetermined pattern, and a two-dimensional array. It is the optical filter element arranged in the shape. Each color filter of the color filter unit 32 transmits light in a predetermined wavelength range (transmission wavelength band) of incident light, and is disposed so as to correspond to each photoelectric conversion element of the imaging unit 33. That is, each color filter in the color filter unit 32 of the monocular camera 3ba is arranged so as to correspond to each of a plurality of photoelectric conversion elements (a plurality of pixels) in the imaging unit 33a. In addition, the plurality of color filter units 32 in the compound-eye array camera 3bb are arranged so as to correspond to the plurality of imaging units 33b, and in each of the color filter units 32 and the imaging unit 33b, each color filter in the color filter unit 32 is These are arranged so as to correspond to the plurality of photoelectric conversion elements (a plurality of pixels) in the imaging unit 33b.
前記単位色フィルタ配列は、例えば画像の使用用途等に応じて種々の態様を採用でき、例えば、図4Aないし図4Eに示す第1ないし第5態様の単位色フィルタ配列321a~321e等である。
The unit color filter array can adopt various modes depending on, for example, the use application of the image, for example, the unit color filter arrays 321a to 321e of the first to fifth modes shown in FIGS. 4A to 4E.
図4Aに示す第1態様の単位色フィルタ配列321aは、赤色を透過する赤色フィルタ(R)、緑色を透過する第1緑色フィルタ(G)、緑色を透過する第2緑色フィルタ(G)、および、青色を透過する青色フィルタ(B)を2行2列で2次元マトリックス状に配置した配列であり、いわゆるベイヤー配列である。すなわち、1行1列に赤色フィルタが配置され、1行2列および2行1列それぞれ第1および第2緑色フィルタが配置され、そして、2行2列に青色フィルタが配置される。このような複数の単位色フィルタ配列321aをさらに2次元アレイ状に配列した色フィルタ部32aが図2に示す色フィルタ部32として用いられてよい。この色フィルタ部32aは、いわゆる原色フィルタである。
The unit color filter array 321a of the first mode shown in FIG. 4A includes a red filter (R) that transmits red, a first green filter (G) that transmits green, a second green filter (G) that transmits green, and The blue filter (B) that transmits blue light is arranged in a two-dimensional matrix with two rows and two columns, and is a so-called Bayer arrangement. That is, a red filter is arranged in one row and one column, a first and second green filter is arranged in one row and two columns and two rows and one column, respectively, and a blue filter is arranged in two rows and two columns. A color filter section 32a in which such a plurality of unit color filter arrays 321a are further arranged in a two-dimensional array may be used as the color filter section 32 shown in FIG. The color filter unit 32a is a so-called primary color filter.
図4Bに示す第2態様の単位色フィルタ配列321bは、黄色を透過する黄色フィルタ(Ye)、マゼンダ色を透過するマゼンダ色フィルタ(Mg)、シアン色を透過するシアン色フィルタ(Cy)、および、緑色を透過する緑色フィルタ(G)を2行2列で2次元マトリックス状に配置した配列である。すなわち、1行1列に黄色フィルタが配置され、1行2列にマゼンダ色フィルタが配列され、2行1列にシアン色フィルタが配置され、そして、2行2列に緑色フィルタが配置される。このような複数の単位色フィルタ配列321bをさらに2次元アレイ状に配列した色フィルタ部32bが図2に示す色フィルタ部32として用いられてよい。この色フィルタ部32bは、いわゆる補色フィルタである。
The unit color filter array 321b of the second aspect shown in FIG. 4B includes a yellow filter (Ye) that transmits yellow, a magenta filter (Mg) that transmits magenta, a cyan filter (Cy) that transmits cyan, and In this arrangement, green filters (G) that transmit green are arranged in a two-dimensional matrix with two rows and two columns. That is, a yellow filter is arranged in one row and one column, a magenta filter is arranged in one row and two columns, a cyan filter is arranged in two rows and one column, and a green filter is arranged in two rows and two columns. . A color filter section 32b in which such a plurality of unit color filter arrays 321b are further arranged in a two-dimensional array may be used as the color filter section 32 shown in FIG. The color filter unit 32b is a so-called complementary color filter.
図4Cに示す第3態様の単位色フィルタ配列321cは、赤色を透過する赤色フィルタ(R)、緑色を透過する緑色フィルタ(G)、青色を透過する青色フィルタ(B)、および、赤外光を透過する赤外光フィルタ(Ir)を2行2列で2次元マトリックス状に配置した配列である。すなわち、1行1列に赤色フィルタが配置され、1行2列に緑色フィルタが配置され、2行1列に青色フィルタが配置され、そして、2行2列に赤外光フィルタが配置される。このような複数の単位色フィルタ配列321cをさらに2次元アレイ状に配列した色フィルタ部32cが図2に示す色フィルタ部32として用いられてよい。
The unit color filter array 321c of the third aspect shown in FIG. 4C includes a red filter (R) that transmits red, a green filter (G) that transmits green, a blue filter (B) that transmits blue, and infrared light. Is an array in which infrared filters (Ir) that pass through are arranged in a two-dimensional matrix with two rows and two columns. That is, a red filter is arranged in one row and one column, a green filter is arranged in one row and two columns, a blue filter is arranged in two rows and one column, and an infrared filter is arranged in two rows and two columns. . A color filter section 32c in which a plurality of such unit color filter arrays 321c are further arranged in a two-dimensional array may be used as the color filter section 32 shown in FIG.
図4Dに示す第4態様の単位色フィルタ配列321dは、白色を透過する白色フィルタ(W)、黄色を透過する黄色フィルタ(Ye)、赤色を透過する赤色フィルタ(R)、および、赤外光を透過する赤外光フィルタ(Ir)を2行2列で2次元マトリックス状に配置した配列である。すなわち、1行1列に白色フィルタが配置され、1行2列に黄色フィルタが配列され、2行1列に赤色フィルタが配置され、そして、2行2列に赤外光フィルタが配置される。このような複数の単位色フィルタ配列321dをさらに2次元アレイ状に配列した色フィルタ部32dが図2に示す色フィルタ部32として用いられてよい。このような第4態様の単位色フィルタ配列321dを用いた色フィルタ部32dを用いることによって、可視光だけでなく赤外光も利用した画像を生成することができる。
The unit color filter array 321d of the fourth aspect shown in FIG. 4D includes a white filter (W) that transmits white, a yellow filter (Ye) that transmits yellow, a red filter (R) that transmits red, and infrared light. Is an array in which infrared filters (Ir) that pass through are arranged in a two-dimensional matrix with two rows and two columns. That is, a white filter is arranged in one row and one column, a yellow filter is arranged in one row and two columns, a red filter is arranged in two rows and one column, and an infrared light filter is arranged in two rows and two columns. . A color filter unit 32d in which such unit color filter arrays 321d are further arranged in a two-dimensional array may be used as the color filter unit 32 shown in FIG. By using the color filter unit 32d using the unit color filter array 321d of the fourth aspect, it is possible to generate an image using not only visible light but also infrared light.
そして、これら図4Aないし図4Dそれぞれに示す第1ないし第4態様の単位色フィルタ配列321a~321dは、2行2列の大きさであるが、これに限定されるものではなく、単位色フィルタ配列は、任意の大きさであってよい。例えば、図4Eに示す第5態様の単位色フィルタ配列321eは、赤色を透過する第1および第2赤色フィルタ(R)、緑色を透過する第1ないし第5緑色フィルタ(G)、ならびに、青色を透過する第1および第2青色フィルタ(B)を3行3列で2次元マトリックス状に配置した配列である。すなわち、1行1列および3行3列それぞれに第1および第2赤色フィルタが配置され、1行2列、2行1列、2行2列、2行3列および3行2列それぞれに第1ないし第5緑色フィルタが配列され、1行3列および3行1列それぞれに青色フィルタが配置される。このような複数の単位色フィルタ配列321eをさらに2次元アレイ状に配列した色フィルタ部32eが図2に示す色フィルタ部32として用いられてよい。
The unit color filter arrays 321a to 321d of the first to fourth modes shown in FIGS. 4A to 4D have a size of 2 rows and 2 columns, but are not limited to this. The array may be of any size. For example, the unit color filter array 321e of the fifth aspect shown in FIG. 4E includes first and second red filters (R) that transmit red, first to fifth green filters (G) that transmit green, and blue. The first and second blue filters (B) that pass through are arranged in a two-dimensional matrix with three rows and three columns. That is, the first and second red filters are arranged in 1 row 1 column and 3 rows 3 columns, respectively, and 1 row 2 columns, 2 rows 1 column, 2 rows 2 columns, 2 rows 3 columns and 3 rows 2 columns respectively. First to fifth green filters are arranged, and blue filters are arranged in 1 row 3 columns and 3 rows 1 column, respectively. A color filter section 32e in which such a plurality of unit color filter arrays 321e are further arranged in a two-dimensional array may be used as the color filter section 32 shown in FIG.
上記構成の単眼のカメラ3baでは、被写体からの光束は、撮像光学系31aに入射され、色フィルタ部32を介して撮像部33aの受光面に物体の光学像を形成する。撮像部33aの各光電変換素子は、それぞれ、色フィルタ部32の各色フィルタを介して受光した光をその光量に応じた電気信号に光電変換し、各画素のデータとして出力する。色フィルタ部32は、図4に示す単位色フィルタ配列321で構成されているので、撮像部33aは、2次元アレイ状に配列された複数の画素それぞれに対応する複数のデータであって、互いに異なる複数色の複数の濃度データを所定のパターンで配列した単位配列を、複数、さらに2次元アレイ状に配列した前記複数のデータを備える原画像データを出力することになる。
In the monocular camera 3ba configured as described above, the light beam from the subject is incident on the imaging optical system 31a and forms an optical image of the object on the light receiving surface of the imaging unit 33a via the color filter unit 32. Each photoelectric conversion element of the imaging unit 33a photoelectrically converts the light received through each color filter of the color filter unit 32 into an electrical signal corresponding to the amount of light, and outputs it as data of each pixel. Since the color filter unit 32 includes the unit color filter array 321 shown in FIG. 4, the imaging unit 33a includes a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array, and Original image data including a plurality of unit arrays in which a plurality of density data of different colors are arranged in a predetermined pattern and the plurality of data arranged in a two-dimensional array is output.
上記構成の複眼のカメラ3bbでは、被写体からの光束は、複数の撮像光学系31bにそれぞれ入射され、各色フィルタ部32を介して各撮像部33bの各受光面に物体の光学像をそれぞれ形成する。各撮像部33bの各光電変換素子は、それぞれ、各色フィルタ部32の各色フィルタを介して受光した光をその光量に応じた電気信号に光電変換し、各画素のデータとして出力する。色フィルタ部32は、図4に示す単位色フィルタ配列321で構成されているので、各撮像部33bは、それぞれ、2次元アレイ状に配列された複数の画素それぞれに対応する複数のデータであって、互いに異なる複数色の複数の濃度データを所定のパターンで配列した単位配列を、複数、さらに2次元アレイ状に配列した前記複数のデータを備える原画像データを出力することになる。したがって、複数の撮像部33bは、複数の原画像データを出力する。図3に示す例では、9個の撮像部33b-11~33b-33を備える固体撮像素子は、9個の原画像データを出力することになる。
In the compound-eye camera 3bb having the above-described configuration, the light flux from the subject is incident on each of the plurality of imaging optical systems 31b, and forms an optical image of the object on each light receiving surface of each imaging unit 33b via each color filter unit 32. . Each photoelectric conversion element of each imaging unit 33b photoelectrically converts the light received through each color filter of each color filter unit 32 into an electrical signal corresponding to the amount of light, and outputs it as data of each pixel. Since the color filter unit 32 is composed of the unit color filter array 321 shown in FIG. 4, each imaging unit 33b is a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array. Thus, original image data including a plurality of unit arrays in which a plurality of density data of a plurality of different colors are arranged in a predetermined pattern, and the plurality of data arranged in a two-dimensional array is output. Therefore, the plurality of imaging units 33b output a plurality of original image data. In the example shown in FIG. 3, the solid-state imaging device including nine imaging units 33b-11 to 33b-33 outputs nine original image data.
そして、上記第1態様の色フィルタ部32aが用いられる場合では、単位色フィルタ配列321aの色組成比は、R:B:G=1:1:2であり、このため、この色フィルタ部32aを用いることによって生成された原画像データの色組成比も、R:B:G=1:1:2である。上記第2態様の色フィルタ部32bが用いられる場合では、単位色フィルタ配列321bの色組成比は、Ye:Mg:Cy:G=1:1:1:1であり、このため、この色フィルタ部32bを用いることによって生成された原画像データの色組成比も、Ye:Mg:Cy:G=1:1:1:1である。上記第3態様の色フィルタ部32cが用いられる場合では、単位色フィルタ配列321cの色組成比は、R:G:B:Ir=1:1:1:1であり、このため、この色フィルタ部32cを用いることによって生成された原画像データの色組成比も、R:G:B:Ir=1:1:1:1である。上記第4態様の色フィルタ部32dが用いられる場合では、単位色フィルタ配列321dの色組成比は、W:Ye:R:Ir=1:1:1:1であり、このため、この色フィルタ部32d用いることによって生成された原画像データの色組成比も、W:Ye:R:Ir=1:1:1:1である。上記第5態様の色フィルタ部32eが用いられる場合では、単位色フィルタ配列321eの色組成比は、R:B:G=2:2:5であり、このため、この色フィルタ部32eを用いることによって生成された原画像データの色組成比も、R:B:G=2:2:5である。
When the color filter section 32a of the first aspect is used, the color composition ratio of the unit color filter array 321a is R: B: G = 1: 1: 2, and therefore, this color filter section 32a. The color composition ratio of the original image data generated by using is also R: B: G = 1: 1: 2. In the case where the color filter unit 32b of the second aspect is used, the color composition ratio of the unit color filter array 321b is Ye: Mg: Cy: G = 1: 1: 1: 1. The color composition ratio of the original image data generated by using the unit 32b is also Ye: Mg: Cy: G = 1: 1: 1: 1. In the case where the color filter unit 32c of the third aspect is used, the color composition ratio of the unit color filter array 321c is R: G: B: Ir = 1: 1: 1: 1. The color composition ratio of the original image data generated by using the unit 32c is also R: G: B: Ir = 1: 1: 1: 1. In the case where the color filter unit 32d of the fourth aspect is used, the color composition ratio of the unit color filter array 321d is W: Ye: R: Ir = 1: 1: 1: 1. The color composition ratio of the original image data generated by using the unit 32d is also W: Ye: R: Ir = 1: 1: 1: 1. When the color filter unit 32e of the fifth aspect is used, the color composition ratio of the unit color filter array 321e is R: B: G = 2: 2: 5. Therefore, the color filter unit 32e is used. The color composition ratio of the original image data generated by this is also R: B: G = 2: 2: 5.
図1に戻って、入力部4は、制御処理部1aに接続され、例えば、画像処理の開始を指示するコマンド等の各種コマンド、および、例えば画像処理対象の画像における識別子(ファイルネーム)の入力や画像処理の際に用いられる画像フィルタの選択入力等の画像処理を実行する上で必要な各種データを画像処理装置Daに入力する機器であり、例えば、キーボードやマウス等である。出力部5は、制御処理部1aに接続され、入力部4から入力されたコマンドやデータ、および、画像処理装置Daによって画像処理された画像処理の結果の画像を出力する機器であり、例えばCRTディスプレイ、LCDおよび有機ELディスプレイ等の表示装置やプリンタ等の印刷装置等である。
Returning to FIG. 1, the input unit 4 is connected to the control processing unit 1 a and inputs various commands such as a command for instructing start of image processing, and an identifier (file name) in an image to be processed, for example. And a device that inputs various data necessary for executing image processing such as selection input of an image filter used in image processing to the image processing device Da, such as a keyboard and a mouse. The output unit 5 is a device that is connected to the control processing unit 1a and outputs a command or data input from the input unit 4 and an image obtained as a result of image processing performed by the image processing device Da. A display device such as a display, an LCD, and an organic EL display, and a printing device such as a printer.
なお、入力部4および出力部5からタッチパネルが構成されてもよい。このタッチパネルを構成する場合において、入力部4は、例えば抵抗膜方式や静電容量方式等の操作位置を検出して入力する位置入力装置であり、出力部5は、表示装置である。このタッチパネルでは、表示装置の表示面上に位置入力装置が設けられ、表示装置に入力可能な1または複数の入力内容の候補が表示され、ユーザが、入力したい入力内容を表示した表示位置を触れると、位置入力装置によってその位置が検出され、検出された位置に表示された表示内容がユーザの操作入力内容として画像処理装置Dに入力される。このようなタッチパネルでは、ユーザは、入力操作を直感的に理解し易いので、ユーザにとって取り扱い易い画像処理装置Daが提供される。
A touch panel may be configured from the input unit 4 and the output unit 5. In the case of configuring this touch panel, the input unit 4 is a position input device that detects and inputs an operation position such as a resistive film method or a capacitance method, and the output unit 5 is a display device. In this touch panel, a position input device is provided on the display surface of the display device, one or more input content candidates that can be input to the display device are displayed, and the user touches the display position where the input content to be input is displayed. The position is detected by the position input device, and the display content displayed at the detected position is input to the image processing device D as the operation input content of the user. With such a touch panel, the user can easily understand the input operation intuitively, and thus an image processing device Da that is easy for the user to handle is provided.
記憶部2は、制御処理部1aに接続され、画像処理を実行する上で必要な各種プログラムや各種データを記憶する装置であり、例えば、ROM(Read Only Memory)やEEPROM(Electrically Erasable Programmable Read Only Memory)等の不揮発性記憶素子、RAM(Random Access Memory)等の揮発性記憶素子およびその周辺回路等を備えて構成される。記憶部2は、例えばハードディスク等の大容量記憶装置を備えても良い。記憶部2は、原画像データを記憶し、この原画像データに対し制御処理部1aによって後述の画像処理を行うための作業領域として用いられる。
The storage unit 2 is a device that is connected to the control processing unit 1a and stores various programs and various data necessary for executing image processing. For example, ROM (Read Only Memory) or EEPROM (Electrically Erasable Programmable Read Only). Non-volatile memory elements such as Memory), volatile memory elements such as RAM (Random Access Memory), and peripheral circuits thereof. The storage unit 2 may include a large-capacity storage device such as a hard disk. The storage unit 2 stores original image data, and is used as a work area for performing image processing to be described later on the original image data by the control processing unit 1a.
制御処理部1aは、所定の画像処理プログラムに基づき後述の画像処理を実行するべく、画像処理装置Daの各部を当該各部の機能に応じてそれぞれ制御するものである。制御処理部1aは、例えば、CPU(Central Processing Unit)およびその周辺回路を備えて構成され、所定の画像処理プログラムを実行することによって、機能的に、制御部11、前処理部12、対応点探索処理部13、位置ズレ量演算部14および超解像処理部15が構成される。
The control processing unit 1a controls each part of the image processing device Da according to the function of each part in order to execute image processing to be described later based on a predetermined image processing program. The control processing unit 1a includes, for example, a CPU (Central Processing Unit) and its peripheral circuits, and functionally executes a predetermined image processing program, so that the control unit 11, the preprocessing unit 12, and corresponding points are functionally executed. A search processing unit 13, a positional deviation amount calculation unit 14, and a super-resolution processing unit 15 are configured.
制御部11は、所定の画像処理を実行するために、画像処理装置Daの各部を当該各部の機能に応じてそれぞれ制御するものである。
The control unit 11 controls each unit of the image processing device Da in accordance with the function of each unit in order to execute predetermined image processing.
前処理部12は、前処理として、原画像データを所定の画像フィルタ(第1画像フィルタ)でフィルタリングすることによって前処理後画像データを生成するものである。前記画像フィルタ(第1画像フィルタ)は、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、原画像データにおける単位配列321の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備えて構成される。
The pre-processing unit 12 generates pre-processed image data by filtering the original image data with a predetermined image filter (first image filter) as pre-processing. The image filter (first image filter) converts the target data into the color of the unit array 321 in the original image data based on the data corresponding to the pixels located around the pixel corresponding to the target data that is the data to be filtered. The filter coefficient is converted to data having the same color composition ratio as the composition ratio.
前記画像フィルタFL1は、種々の態様を採用でき、例えば、ベイヤー配列を単位色フィルタ配列とした色フィルタ部32aによって得られる原画像データに対し、図5Aないし図5Dに示す第1ないし第4態様の画像フィルタFL1a~FL1d等である。これら第1ないし第4態様の画像フィルタFL1a~FL1dは、それぞれ、フィルタリングの対象データに対応する画素(対象画素)の周辺に位置する画素(周辺画素)に対応するデータ(周辺データ)に基づいて、前記対象データを、単位色フィルタ配列321aの色組成比(R:B:G=1:1:2)、すなわち、原画像データにおける単位配列の色組成比と同じ色組成比を持つデータに変換するように、そのフィルタ係数の各値がそれぞれ調整されて設定されている。
The image filter FL1 can adopt various modes. For example, the first to fourth modes shown in FIGS. 5A to 5D with respect to the original image data obtained by the color filter unit 32a in which the Bayer array is a unit color filter array. Image filters FL1a to FL1d. Each of the image filters FL1a to FL1d of the first to fourth aspects is based on data (peripheral data) corresponding to pixels (peripheral pixels) located around the pixel (target pixel) corresponding to the filtering target data. The target data is converted into data having the color composition ratio (R: B: G = 1: 1: 2) of the unit color filter array 321a, that is, data having the same color composition ratio as the color composition ratio of the unit array in the original image data. Each value of the filter coefficient is adjusted and set so as to be converted.
より具体的には、第1態様の画像フィルタFL1aは、図5Aに示すように、3行3列の中心位置に対し対称なフィルタ係数を持つフィルタである。すなわち、中心の2行2列に位置するフィルタ係数は、4/16であり、1行2列、2行1列、2行3列および3行2列それぞれに位置する各フィルタ係数は、2/16であり、そして、1行1列、1行3列、3行1列および3行3列それぞれに位置する各フィルタ係数は、1/16である。このようなフィルタ係数を持つ第1態様の画像フィルタFL1aは、ガウシアンフィルタでもある。なお、図5Aの左右それぞれに示す各画像フィルタは、表現形式が異なるだけで、同じものである。図5Aの左側では、各位置にフィルタ係数の値そのものが記載され、図5Aの右側では、各位置にフィルタ係数の値の分子が記載され、分母は、右下に纏めて記載されている。図5Bないし図5Dは、図5Aの右側に示す記載形式で記載されている。
More specifically, as shown in FIG. 5A, the image filter FL1a according to the first aspect is a filter having a filter coefficient that is symmetric with respect to the center position of 3 rows and 3 columns. That is, the filter coefficient located in the center 2 rows and 2 columns is 4/16, and each filter coefficient located in 1 row 2 columns, 2 rows 1 columns, 2 rows 3 columns and 3 rows 2 columns is 2 / 16, and each filter coefficient located in each of 1 row, 1 column, 1 row, 3 columns, 3 rows, 1 column and 3 rows, 3 columns is 1/16. The image filter FL1a of the first aspect having such a filter coefficient is also a Gaussian filter. Note that the image filters shown on the left and right in FIG. 5A are the same except for the expression format. On the left side of FIG. 5A, the filter coefficient value itself is described at each position. On the right side of FIG. 5A, the numerator of the filter coefficient value is described at each position, and the denominator is collectively described at the lower right. 5B to 5D are described in the description format shown on the right side of FIG. 5A.
第2態様の画像フィルタFL1bは、5行5列の中心位置に対し対称なフィルタ係数を持つフィルタである。すなわち、中心の3行3列に位置するフィルタ係数は、36/256であり、2行3列、3行2列、3行4列および4行3列それぞれに位置する各フィルタ係数は、24/256であり、2行2列、2行4列、4行2列および4行4列それぞれに位置する各フィルタ係数は、16/256であり、1行3列、3行1列、3行5列および5行3列それぞれに位置する各フィルタ係数は、6/256であり、1行2列、1行4列、2行1列、2行5列、4行1列、4行5列、5行2列および5行4列それぞれに位置する各フィルタ係数は、4/256であり、そして、1行1列、1行5列、5行1列および5行5列それぞれに位置する各フィルタ係数は、1/256である。
The image filter FL1b of the second aspect is a filter having a filter coefficient that is symmetric with respect to the center position of 5 rows and 5 columns. That is, the filter coefficient located in the center 3 rows and 3 columns is 36/256, and each filter coefficient located in 2 rows 3 columns, 3 rows 2 columns, 3 rows 4 columns and 4 rows 3 columns is 24 Each of the filter coefficients located in 2 rows, 2 columns, 2 rows, 4 columns, 4 rows, 2 columns, and 4 rows, 4 columns is 16/256, 1 row, 3 columns, 3 rows, 1 column, 3 Each filter coefficient located in each of row 5 column and row 5 column 3 is 6/256, 1 row 2 columns, 1 row 4 columns, 2 rows 1 column, 2 rows 5 columns, 4 rows 1 column, 4 rows. Each filter coefficient located in the 5th column, 5th row and 2nd column, and 5th row and 4th column is 4/256, and 1st row and 1st column, 1st row and 5th column, 5th row and 1st column, and 5th row and 5th column respectively. Each filter coefficient located is 1/256.
第3態様の画像フィルタFL1cは、7行7列の中心位置に対し対称なフィルタ係数を持つフィルタである。すなわち、中心の4行4列に位置するフィルタ係数は、400/4096であり、3行4列、4行3列、4行5列および5行4列それぞれに位置する各フィルタ係数は、300/4096であり、3行3列、3行5列、5行3列および5行5列それぞれに位置する各フィルタ係数は、225/4096であり、2行4列、4行2列、4行6列および6行4列それぞれに位置する各フィルタ係数は、200/4096であり、2行3列、2行5列、3行2列、3行6列、5行2列、5行5列、6行3列および6行5列それぞれに位置する各フィルタ係数は、90/4096であり、2行2列、2行6列、6行2列および6行6列それぞれに位置する各フィルタ係数は、36/4096であり、1行4列、4行1列、4行7列および7行4列それぞれに位置する各フィルタ係数は、20/4096であり、1行3列、1行5列、3行1列、3行7列、5行1列、5行7列、7行3列および7行5列それぞれに位置する各フィルタ係数は、15/256であり、1行2列、1行6列、2行1列、2行7列、6行1列、6行7列、7行2列および7行6列それぞれに位置する各フィルタ係数は、6/256であり、そして、1行1列、1行7列、7行1列および7行7列それぞれに位置する各フィルタ係数は、1/4096である。
The image filter FL1c according to the third aspect is a filter having a filter coefficient that is symmetric with respect to the center position of 7 rows and 7 columns. That is, the filter coefficient located at the center 4 rows and 4 columns is 400/4096, and each filter coefficient located at 3 rows 4 columns, 4 rows 3 columns, 4 rows 5 columns and 5 rows 4 columns is 300. Each filter coefficient located in 3 rows, 3 columns, 3 rows, 5 columns, 5 rows, 3 columns, and 5 rows, 5 columns is 225/4096, 2 rows, 4 columns, 4 rows, 2 columns, 4 Each filter coefficient located in row 6 column and 6 row 4 column is 200/4096, 2 rows 3 columns, 2 rows 5 columns, 3 rows 2 columns, 3 rows 6 columns, 5 rows 2 columns, 5 rows. Each filter coefficient located in each of the 5th column, 6th row and 3rd column, and 6th row and 5th column is 90/4096, which is located in each of 2nd row and 2nd column, 2nd row and 6th column, 6th row and 2nd column, and 6th row and 6th column. Each filter coefficient is 36/4096, 1 row 4 columns, 4 rows 1 column, 4 rows 7 columns and 7 rows 4 columns Each filter coefficient located there is 20/4096, 1 row 3 columns, 1 row 5 columns, 3 rows 1 columns, 3 rows 7 columns, 5 rows 1 columns, 5 rows 7 columns, 7 rows 3 columns and Each filter coefficient located in each of 7 rows and 5 columns is 15/256, 1 row 2 columns, 1 row 6 columns, 2 rows 1 column, 2 rows 7 columns, 6 rows 1 column, 6 rows 7 columns, 7 Each filter coefficient located in each of row 2 column and 7 row 6 column is 6/256, and each filter located in 1 row 1 column, 1 row 7 column, 7 row 1 column and 7 row 7 column respectively The coefficient is 1/4096.
第4態様の画像フィルタFL1dは、5行5列の中心位置に対し対称なフィルタ係数を持つフィルタである。すなわち、中心の3行3列に位置するフィルタ係数は、16/144であり、2行3列、3行2列、3行4列および4行3列それぞれに位置する各フィルタ係数は、12/144であり、2行2列、2行4列、4行2列および4行4列それぞれに位置する各フィルタ係数は、9/144であり、1行3列、3行1列、3行5列および5行3列それぞれに位置する各フィルタ係数は、4/144であり、1行2列、1行4列、2行1列、2行5列、4行1列、4行5列、5行2列および5行4列それぞれに位置する各フィルタ係数は、3/144であり、そして、1行1列、1行5列、5行1列および5行5列それぞれに位置する各フィルタ係数は、1/144である。
The image filter FL1d according to the fourth aspect is a filter having a filter coefficient that is symmetric with respect to the center position of 5 rows and 5 columns. That is, the filter coefficient located in the center 3 rows and 3 columns is 16/144, and each filter coefficient located in 2 rows 3 columns, 3 rows 2 columns, 3 rows 4 columns, and 4 rows 3 columns is 12 / 144, and each filter coefficient located in each of 2 rows 2 columns, 2 rows 4 columns, 4 rows 2 columns and 4 rows 4 columns is 9/144, 1 row 3 columns, 3 rows 1 columns, 3 Each filter coefficient located in each of row 5 column and row 5 column 3 is 4/144, 1 row, 2 columns, 1 row, 4 columns, 2 rows, 1 column, 2 rows, 5 columns, 4 rows, 1 column, 4 rows. Each filter coefficient located in each of 5 columns, 5 rows 2 columns and 5 rows 4 columns is 3/144, and 1 row 1 column, 1 row 5 columns, 5 rows 1 column and 5 rows 5 columns respectively. Each filter coefficient located is 1/144.
このような第1ないし第4態様の画像フィルタFL1a~FL1dに対し、高周波のモアレ成分が残ると対応点探索が誤判定する虞があるので、画像フィルタFL1は、例えば、原画像データの画像に含まれる高周波成分の割合に応じて適宜に選択されることが好ましい。例えば、原画像データの画像に含まれる高周波成分の割合が多いほど、よりボケさせる画像フィルタFL1が選択される。
For such image filters FL1a to FL1d of the first to fourth aspects, if a high-frequency moire component remains, the corresponding point search may be erroneously determined. Therefore, the image filter FL1 may, for example, be applied to the image of the original image data. It is preferable to select appropriately according to the ratio of the high frequency component contained. For example, the image filter FL1 that is more blurred is selected as the ratio of the high-frequency component included in the image of the original image data increases.
上述では、繰り返しパターンの単位として単位色フィルタ配列321aを用いた色フィルタ部32aによって得られる原画像データに対する様々な第1ないし第4態様の画像フィルタFL1a~FL1dについて説明したが、他の態様の単位色フィルタ配列321を用いた色フィルタ部32によって得られる原画像データに対しても、同様に、画像フィルタFL1を設定できる。例えば、図4Eに示す第5態様の単位色フィルタ配列321eを用いた色フィルタ部32eによって得られる原画像データに対し、図5Eに示す第5態様の画像フィルタFL1eが使用できる。この第5態様の画像フィルタFL1eは、フィルタリングの対象データに対応する画素(対象画素)の周辺に位置する画素(周辺画素)に対応するデータ(周辺データ)に基づいて、前記対象データを、単位色フィルタ配列321eの色組成比(R:B:G=2:2:5)、すなわち、原画像データにおける単位配列の色組成比と同じ色組成比を持つデータに変換するように、そのフィルタ係数の各値がそれぞれ調整されて設定されている。より具体的には、図5Eに示すように、各位置に位置する各フィルタ係数それぞれが全て1/9であるフィルタである。この第5態様の画像フィルタFL1eは、3行3列の中心位置に対し対称なフィルタ係数を持つフィルタでもある。
In the above description, various image filters FL1a to FL1d of the first to fourth modes for the original image data obtained by the color filter unit 32a using the unit color filter array 321a as the unit of the repetitive pattern have been described. Similarly, the image filter FL1 can be set for the original image data obtained by the color filter unit 32 using the unit color filter array 321. For example, the image filter FL1e of the fifth mode shown in FIG. 5E can be used for the original image data obtained by the color filter unit 32e using the unit color filter array 321e of the fifth mode shown in FIG. 4E. The image filter FL1e according to the fifth aspect uses the data (peripheral data) corresponding to the pixel (peripheral pixel) located around the pixel (target pixel) corresponding to the target data for filtering as the target data. The color filter array 321e color composition ratio (R: B: G = 2: 2: 5), that is, the filter so as to be converted into data having the same color composition ratio as that of the unit array in the original image data. Each coefficient value is adjusted and set. More specifically, as shown in FIG. 5E, each filter coefficient located at each position is a 1/9 filter. The image filter FL1e of the fifth aspect is also a filter having a filter coefficient that is symmetric with respect to the center position of 3 rows and 3 columns.
また、画像フィルタFLの各フィルタ係数は、画像フィルタFL全域に単位配列を重ねた場合に、前記単位配列の同じ配列位置におけるフィルタ係数の和が全ての配列位置で同じ値になるように、設定されてよい。
In addition, each filter coefficient of the image filter FL is set so that the sum of the filter coefficients at the same arrangement position of the unit arrangement becomes the same value at all arrangement positions when the unit arrangement is overlapped over the entire area of the image filter FL. May be.
図1に戻って、対応点探索処理部13は、複数の原画像データから前処理部12によって生成された複数の前処理後画像データそれぞれに対応する複数の画像の間で対応点探索を実施するものである。
Returning to FIG. 1, the corresponding point search processing unit 13 performs corresponding point search among a plurality of images corresponding to a plurality of pre-processed image data generated by the pre-processing unit 12 from a plurality of original image data. To do.
位置ズレ量演算部14は、対応点探索処理部13で探索された対応点間の距離である位置ズレ量を求めるものである。位置ズレ量は、ピクセル単位(画素単位、画素ピッチの整数倍)で求められてもよいが、本実施形態では、サブピクセル単位で求められる。
The positional deviation amount calculation unit 14 obtains a positional deviation amount that is a distance between corresponding points searched by the corresponding point search processing unit 13. The positional shift amount may be obtained in pixel units (pixel units, integer multiples of the pixel pitch), but in the present embodiment, it is obtained in subpixel units.
超解像処理部15は、位置ズレ量演算部14で求めた位置ズレ量に基づいて、複数の原画像データから原画像データの画像における解像度より高い高解像度の画像を生成するものである。
The super-resolution processing unit 15 generates a high-resolution image higher than the resolution of the original image data from the plurality of original image data based on the positional shift amount obtained by the positional shift amount calculation unit 14.
このような画像処理装置Daは、例えばノート型やディスクトップ型等のパーソナルコンピュータや、また例えば携帯電話機等の携帯機器やカメラ等のコンピュータを搭載した機器等によって構成可能である。
Such an image processing device Da can be constituted by a personal computer such as a notebook type or a desktop type, a portable device such as a mobile phone, or a device equipped with a computer such as a camera.
次に、第1実施形態における画像処理装置Daの動作について説明する。図6は、第1実施形態における画像処理装置の動作を示すフローチャートである。図7は、アレイカメラによって撮像された各画像間における相互位置関係を説明するための図である。図8は、第1実施形態の画像処理装置における前処理を説明するための図である。図8Aは、画像の各画素(原画像データの各データ)と画像フィルタとの関係を示し、図8Bは、画像フィルタを示し、図8Cは、フィルタリング後の前処理後画像を示し、図8Dは、前処理前の画像の一例を示し、そして、図8Eは、前処理後の画像の一例を示す。図9は、第1実施形態の画像処理装置における対応点探索処理を説明するための図である。図9Aは、対応点探索の対象である第1および第2画像を重ねて表示した図であり、図9Bおよび図9Cは、相対的に粗い精度での対応点探索を示し、図9Dは、相対的に細かい精度での対応点探索を示す。図10は、第1実施形態の画像処理装置におけるサブピクセル単位での対応点探索処理を説明するための図である。図10Aは、対応点探索結果の一例であるNCC値を示し、図10Bは、図10Aに示すNCC値からサブピクセル単位で求められた対応点を示す。図11は、第1実施形態の画像処理装置における第1態様の超解像処理を示すフローチャートである。図12は、第1実施形態の画像処理装置における超解像処理の劣化情報を説明するための図である。図13は、残差処理と超解像処理との関係を説明するための図である。図13Aは、ベイヤー配列を示し、図13BおよびCは、2個の入力画像の補間関係を示す。図14は、残差処理における劣化情報に応じた出力候補画像の変換を説明するための図である。図14Aは、出力候補画像Xnを示し、図14Bは、劣化情報に応じて入力画像相当の解像度に変換した出力候補画像Xnを示し、図14Cは、図14Aから図14Bへの変換DkHkFkを説明するための図である。図15は、残差処理における符号関数の演算を説明するための図である。図15Aは、劣化情報に応じて入力画像相当の解像度に変換した出力候補画像Xnを示し、図15Bは、入力画像Ykを示し、図15Cは、差異sign(DkHkFkXn-Yk)を示す。図16は、残差処理における符号関数の演算結果に影響した出力候補画像の画素を求める処理を説明するための図である。図16Aは、差異sign(DkHkFkXn-Yk)を示し、図16Bは、超解像画像のサイズでの差異の画像Fk
THk
TDk
Tsign(DkHkFkXn-Yk)を示し、図16Cは、図16Aから図16Bへの変換Fk
THk
TDk
Tを説明するための図である。図17は、各入力画像に対する残差処理を示すフローチャートである。
Next, the operation of the image processing device Da in the first embodiment will be described. FIG. 6 is a flowchart illustrating the operation of the image processing apparatus according to the first embodiment. FIG. 7 is a diagram for explaining the mutual positional relationship between the images captured by the array camera. FIG. 8 is a diagram for explaining preprocessing in the image processing apparatus according to the first embodiment. 8A shows the relationship between each pixel of the image (each data of the original image data) and the image filter, FIG. 8B shows the image filter, FIG. 8C shows the preprocessed image after filtering, and FIG. FIG. 8E shows an example of an image before preprocessing, and FIG. 8E shows an example of an image after preprocessing. FIG. 9 is a diagram for explaining corresponding point search processing in the image processing apparatus according to the first embodiment. FIG. 9A is a diagram in which the first and second images to be searched for corresponding points are displayed in an overlapping manner. FIGS. 9B and 9C show the corresponding point search with relatively coarse accuracy, and FIG. Corresponding point search with relatively fine accuracy is shown. FIG. 10 is a diagram for explaining corresponding point search processing in units of subpixels in the image processing apparatus according to the first embodiment. FIG. 10A shows an NCC value which is an example of a corresponding point search result, and FIG. 10B shows a corresponding point obtained in units of subpixels from the NCC value shown in FIG. 10A. FIG. 11 is a flowchart showing the super-resolution processing of the first aspect in the image processing apparatus of the first embodiment. FIG. 12 is a diagram for explaining deterioration information of super-resolution processing in the image processing apparatus according to the first embodiment. FIG. 13 is a diagram for explaining the relationship between residual processing and super-resolution processing. FIG. 13A shows the Bayer array, and FIGS. 13B and 13C show the interpolation relationship between the two input images. FIG. 14 is a diagram for describing conversion of an output candidate image according to deterioration information in the residual processing. 14A shows the output candidate image Xn , FIG. 14B shows the output candidate image Xn converted to the resolution equivalent to the input image according to the degradation information, and FIG. 14C shows the conversion D from FIG. 14A to FIG. 14B. it is a diagram for explaining a k H k F k. FIG. 15 is a diagram for explaining the calculation of the sign function in the residual process. 15A shows an output candidate image Xn converted to a resolution equivalent to the input image according to the degradation information, FIG. 15B shows the input image Yk , and FIG. 15C shows the difference sign (D k H k F k X n− Y k ). FIG. 16 is a diagram for explaining processing for obtaining pixels of an output candidate image that has influenced the calculation result of the sign function in the residual processing. FIG. 16A shows the difference sign (D k H k F k X n −Y k ), and FIG. 16B shows the difference image F k T H k T D k T sign (D k ) at the size of the super-resolution image. H k F k X n -Y k ) indicates, FIG. 16C is a diagram for explaining conversion F k T H k T D k T to Figure 16B from Figure 16A. FIG. 17 is a flowchart showing residual processing for each input image.
画像処理装置Daは、例えば、ユーザの操作によって入力部4から起動コマンドを受け付けると、画像処理プログラムを実行する。この画像処理プログラムの実行によって、制御処理部1aに制御部11、前処理部12、対応点探索処理部13、位置ズレ量演算部14および超解像処理部15が機能的に構成される。
The image processing device Da executes an image processing program when receiving an activation command from the input unit 4 by a user operation, for example. By executing the image processing program, the control processing unit 1a functionally includes the control unit 11, the preprocessing unit 12, the corresponding point search processing unit 13, the positional deviation amount calculation unit 14, and the super-resolution processing unit 15.
そして、図6において、まず、処理S1では、画像取得部3によって複数の画像それぞれに対応する複数の原画像データが当該画像処理装置Daに入力され、記憶部2に記憶される。本実施形態では、一例として、9個の第1ないし第9原画像データISが入力され、記憶部2に記憶される。これら第1ないし第9原画像データISは、例えば3行3列に2次元アレイ状に配列された9眼を持つアレイカメラ3bbで同一の被写体を撮像することによって生成されたデータである。したがって、これら第1ないし第9原画像データIS-11~IS-33は、互いの位置関係として図7に示すように3行3列に2次元アレイ状に配列される。1行1列に位置する原画像データが第1原画像データIS-11であり、2次元アレイ状配列の各位置に位置する各原画像データISが順次にそれぞれ番号付けられ、3行3列に位置する原画像データが第9原画像データIS-33である。そして、2行2列の中央に位置にする原画像データISは、第5原画像データIS-22であり、後述の対応点探索で基準画像とされる。これら第1ないし第9原画像データIS-11~IS-33は、それぞれ、適宜な色フィルタ部32を用いて生成された任意の画像数のデータであって良いが、ここでは、一例として、ベイヤー配列の色フィルタ部32aを用いて生成された500×500画素のデータである。
In FIG. 6, first, in process S <b> 1, a plurality of original image data corresponding to each of a plurality of images is input to the image processing device Da by the image acquisition unit 3 and stored in the storage unit 2. In the present embodiment, as an example, nine first to ninth original image data IS are input and stored in the storage unit 2. These first to ninth original image data IS are data generated by imaging the same subject with an array camera 3bb having nine eyes arranged in a two-dimensional array, for example, in three rows and three columns. Accordingly, the first to ninth original image data IS-11 to IS-33 are arranged in a two-dimensional array in three rows and three columns as shown in FIG. The original image data located in one row and one column is the first original image data IS-11, and each original image data IS located in each position of the two-dimensional array is sequentially numbered, and three rows and three columns. The original image data located at is the ninth original image data IS-33. The original image data IS positioned at the center of 2 rows and 2 columns is the fifth original image data IS-22, which is used as a reference image in the corresponding point search described later. These first to ninth original image data IS-11 to IS-33 may each be data of an arbitrary number of images generated by using an appropriate color filter unit 32. Here, as an example, This is data of 500 × 500 pixels generated by using the color filter section 32a in the Bayer array.
次に、処理S2では、前処理部12によって第1ないし第9原画像データIS-11~IS-33それぞれが前処理される。すなわち、前処理部12は、第1ないし第9原画像データIS-11~IS-33それぞれを画像フィルタFL1でフィルタリングすることによって第1ないし第9前処理後画像データAIS-11~AIS-33それぞれを生成し、これら生成した第1ないし第9前処理後画像データAIS-11~AIS-33を記憶部2に記憶する。より具体的には、例えば、本実施形態では、ベイヤー配列の原画像データISに対応した画像フィルタFL1、例えば、図5Aに示す画像フィルタFL1aが用いられる。フィルタリングは、原画像データISにおける最初の1行1列に位置する画素に対応するデータから最終の500行500列に位置する画素に対応するデータまで順次に実行される。
Next, in the processing S2, the first to ninth original image data IS-11 to IS-33 are preprocessed by the preprocessing unit 12. That is, the pre-processing unit 12 filters the first to ninth original image data IS-11 to IS-33 with the image filter FL1, respectively, thereby filtering the first to ninth post-processed image data AIS-11 to AIS-33. These are generated, and the generated first to ninth post-processed image data AIS-11 to AIS-33 are stored in the storage unit 2. More specifically, for example, in this embodiment, an image filter FL1 corresponding to the original image data IS of the Bayer array, for example, an image filter FL1a shown in FIG. 5A is used. The filtering is sequentially executed from data corresponding to the pixel located in the first row and first column in the original image data IS to data corresponding to the pixel located in the last 500 rows and 500 columns.
例えば、図8A上段に示す2行2列に位置するG画素に対し図5Aおよび図8Bに示す画像フィルタFL1aでフィルタリングすると、2行2列に位置するG画素に対応する前処理後画像データAISのデータは、(1行1列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のG画素のデータの値)+(1行2列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のR画素のデータの値)+(1行3列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のG画素のデータの値)+(2行1列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のB画素のデータの値)+(2行2列に位置する画像フィルタFL1aのフィルタ値;4/16)×(その位置に対応する位置のG画素のデータの値)+(2行3列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のB画素のデータの値)+(3行1列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のG画素のデータの値)+(3行2列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のR画素のデータの値)+(3行3列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のG画素のデータの値)となる。したがって、R画素に対応するフィルタ係数の値は、総計、(2/16)×2=4/16=1/4となり、B画素に対応するフィルタ係数の値は、総計、(2/16)×2=4/16=1/4となり、そして、G画素に対応するフィルタ係数の値は、総計、(1/16)×4+(4/16)×1=8/16=1/2となる。したがって、画像フィルタFL1aの色組成比は、R:B:G=1/4:1/4:1/2=1:1:2となり、色フィルタ部32aにおける単位色フィルタ配列321aの色組成比、すなわち、原画像データISにおける単位配列の色組成比と同じである。
For example, if the G pixel located in 2 rows and 2 columns shown in the upper part of FIG. 8A is filtered by the image filter FL1a shown in FIGS. 5A and 8B, the preprocessed image data AIS corresponding to the G pixels located in 2 rows and 2 columns The data of (the filter value of the image filter FL1a located in the first row and the first column; 1/16) × (the data value of the G pixel at the position corresponding to the position) + (the image filter located in the first row and the second column) Filter value of FL1a; 2/16) × (value of R pixel data at a position corresponding to that position) + (filter value of image filter FL1a located in 1 row × 3 columns; 1/16) × (at that position) Data value of G pixel at corresponding position) + (Filter value of image filter FL1a located at 2 rows and 1 column; 2/16) × (Data value of B pixel at position corresponding to that position) + (2 Located in row 2 column Filter value of the image filter FL1a; 4/16) × (data value of the G pixel at the position corresponding to the position) + (filter value of the image filter FL1a positioned at 2 rows × 3 columns; 2/16) × (there Data value of the B pixel at the position corresponding to the position) + (Filter value of the image filter FL1a located at 3 rows and 1 column; 1/16) × (Data value of the G pixel at the position corresponding to the position) + (Filter value of image filter FL1a located at 3 rows and 2 columns; 2/16) × (value of data of R pixel at position corresponding to that position) + (filter value of image filter FL1a located at 3 rows and 3 columns) 1/16) × (data value of G pixel at the position corresponding to the position). Therefore, the value of the filter coefficient corresponding to the R pixel is a total, (2/16) × 2 = 4/16 = 1/4, and the value of the filter coefficient corresponding to the B pixel is the total, (2/16) X2 = 4/16 = 1/4, and the value of the filter coefficient corresponding to the G pixel is (1/16) * 4 + (4/16) * 1 = 8/16 = 1/2 in total. Become. Therefore, the color composition ratio of the image filter FL1a is R: B: G = 1/4: 1/4: 1/2 = 1: 1: 2, and the color composition ratio of the unit color filter array 321a in the color filter section 32a. That is, it is the same as the color composition ratio of the unit array in the original image data IS.
図8A下段に示す3行2列に位置するR画素に対し前記画像フィルタFL1aでフィルタリングすると、3行2列に位置するR画素に対応する前処理後画像データAISのデータは、(1行1列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のB画素のデータの値)+(1行2列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のG画素のデータの値)+(1行3列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のB画素のデータの値)+(2行1列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のG画素のデータの値)+(2行2列に位置する画像フィルタFL1aのフィルタ値;4/16)×(その位置に対応する位置のR画素のデータの値)+(2行3列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のG画素のデータの値)+(3行1列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のB画素のデータの値)+(3行2列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のG画素のデータの値)+(3行3列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のB画素のデータの値)となる。したがって、R画素に対応するフィルタ係数の値は、総計、(4/16)×1=4/16=1/4となり、B画素に対応するフィルタ係数の値は、総計、(1/16)×4=4/16=1/4となり、そして、G画素に対応するフィルタ係数の値は、総計、(2/16)×4=8/16=1/2となる。したがって、画像フィルタFLaの色組成比は、R:B:G=1/4:1/4:1/2=1:1:2となり、上述と同様に、原画像データISにおける単位配列の色組成比と同じである。
When the R pixel located at 3 rows and 2 columns shown in the lower part of FIG. 8A is filtered by the image filter FL1a, the preprocessed image data AIS corresponding to the R pixels located at 3 rows and 2 columns is (1 row 1). Filter value of the image filter FL1a located in the column; 1/16) × (data value of the B pixel at the position corresponding to the position) + (filter value of the image filter FL1a located in the first row and the second column; 2/16 ) × (value of G pixel data at the position corresponding to the position) + (filter value of the image filter FL1a located in 1 row × 3 columns; 1/16) × (data of the B pixel at the position corresponding to the position) Value) + (filter value of image filter FL1a located at 2 rows and 1 column; 2/16) × (value of G pixel data at the position corresponding to that position) + (image filter located at 2 rows and 2 columns) FL1a 4/16) × (value of data of R pixel at the position corresponding to the position) + (filter value of the image filter FL1a positioned at 2 rows × 3 columns; 2/16) × (corresponding to the position Data value of G pixel at position) + (Filter value of image filter FL1a located at 3 rows and 1 column; 1/16) × (Data value of B pixel at position corresponding to that position) + (3 rows 2 Filter value of the image filter FL1a located in the column; 2/16) × (data value of the G pixel at the position corresponding to the position) + (filter value of the image filter FL1a located in the third row and the third column; 1/16 ) × (data value of the B pixel at the position corresponding to the position). Therefore, the value of the filter coefficient corresponding to the R pixel is a total, (4/16) × 1 = 4/16 = 1/4, and the value of the filter coefficient corresponding to the B pixel is the total, (1/16) × 4 = 4/16 = 1/4, and the value of the filter coefficient corresponding to the G pixel is (2/16) × 4 = 8/16 = 1/2 in total. Therefore, the color composition ratio of the image filter FLa is R: B: G = 1/4: 1/4: 1/2 = 1: 1: 2, and the color of the unit array in the original image data IS as described above. It is the same as the composition ratio.
2行3列に位置するB画素に対し前記画像フィルタFL1aでフィルタリングすると、2行3列に位置するB画素に対応する前処理後画像データAISのデータは、(1行1列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のR画素のデータの値)+(1行2列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のG画素のデータの値)+(1行3列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のR画素のデータの値)+(2行1列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のG画素のデータの値)+(2行2列に位置する画像フィルタFL1aのフィルタ値;4/16)×(その位置に対応する位置のB画素のデータの値)+(2行3列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のG画素のデータの値)+(3行1列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のR画素のデータの値)+(3行2列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のG画素のデータの値)+(3行3列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のR画素のデータの値)となる。したがって、R画素に対応するフィルタ係数の値は、総計、(1/16)×4=4/16=1/4となり、B画素に対応するフィルタ係数の値は、総計、(4/16)×1=4/16=1/4となり、そして、G画素に対応するフィルタ係数の値は、総計、(2/16)×4=8/16=1/2となる。したがって、画像フィルタFL1aの色組成比は、R:B:G=1/4:1/4:1/2=1:1:2となり、上述と同様に、原画像データISにおける単位配列の色組成比と同じである。
When the B pixel located in 2 rows and 3 columns is filtered by the image filter FL1a, the data of the preprocessed image data AIS corresponding to the B pixels located in 2 rows and 3 columns is (image located in 1 row and 1 column) Filter value of filter FL1a; 1/16) × (value of data of R pixel at position corresponding to the position) + (filter value of image filter FL1a located in 1 row × 2 columns; 2/16) × (the position Data value of the G pixel at the position corresponding to) + (filter value of the image filter FL1a located in the first row and the third column; 1/16) × (data value of the R pixel at the position corresponding to the position) + ( Filter value of image filter FL1a located in 2 rows and 1 column; 2/16) × (value of data of G pixel at position corresponding to that position) + (filter value of image filter FL1a located in 2 rows and 2 columns; 4/16 X (value of data of B pixel at the position corresponding to the position) + (filter value of the image filter FL1a located at 2 rows × 3 columns; 2/16) × (data of the G pixel at the position corresponding to the position) Value) + (filter value of image filter FL1a located in 3 rows and 1 column; 1/16) × (value of data of R pixel at position corresponding to that position) + (image filter FL1a located in 3 rows and 2 columns) Filter value; 2/16) × (data value of G pixel at position corresponding to that position) + (filter value of image filter FL1a located at 3 rows × 3 columns; 1/16) × (corresponding to that position) Data value of the R pixel at the position to be detected). Therefore, the value of the filter coefficient corresponding to the R pixel is a total, (1/16) × 4 = 4/16 = 1/4, and the value of the filter coefficient corresponding to the B pixel is the total, (4/16) X1 = 4/16 = 1/4, and the value of the filter coefficient corresponding to the G pixel is (2/16) * 4 = 8/16 = 1/2 in total. Therefore, the color composition ratio of the image filter FL1a is R: B: G = 1/4: 1/4: 1/2 = 1: 1: 2, and the color of the unit array in the original image data IS as described above. It is the same as the composition ratio.
そして、3行3列に位置するG画素に対し前記画像フィルタFL1aでフィルタリングすると、3行3列に位置するG画素に対応する前処理後画像データAISのデータは、(1行1列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のG画素のデータの値)+(1行2列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のB画素のデータの値)+(1行3列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のG画素のデータの値)+(2行1列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のR画素のデータの値)+(2行2列に位置する画像フィルタFL1aのフィルタ値;4/16)×(その位置に対応する位置のG画素のデータの値)+(2行3列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のR画素のデータの値)+(3行1列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のG画素のデータの値)+(3行2列に位置する画像フィルタFL1aのフィルタ値;2/16)×(その位置に対応する位置のB画素のデータの値)+(3行3列に位置する画像フィルタFL1aのフィルタ値;1/16)×(その位置に対応する位置のG画素のデータの値)となる。したがって、R画素に対応するフィルタ係数の値は、総計、(2/16)×2=4/16=1/4となり、B画素に対応するフィルタ係数の値は、総計、(2/16)×2=4/16=1/4となり、そして、G画素に対応するフィルタ係数の値は、総計、(1/16)×4+(4/16)×1=8/16=1/2となる。したがって、画像フィルタFL1aの色組成比は、R:B:G=1/4:1/4:1/2=1:1:2となり、上述と同様に、原画像データISにおける単位配列の色組成比と同じである。
When the G pixel located in 3 rows and 3 columns is filtered by the image filter FL1a, the preprocessed image data AIS data corresponding to the G pixels located in 3 rows and 3 columns is (positioned in 1 row and 1 column). Filter value of the image filter FL1a to be performed; 1/16) × (data value of the G pixel at the position corresponding to the position) + (filter value of the image filter FL1a located in the first row and the second column; 2/16) × ( Data value of the B pixel at the position corresponding to the position) + (Filter value of the image filter FL1a located in the first row and the third column; 1/16) × (Data value of the G pixel at the position corresponding to the position) + (Filter value of image filter FL1a located at 2 rows and 1 column; 2/16) × (value of data of R pixel at position corresponding to that position) + (filter of image filter FL1a located at 2 rows and 2 columns) value; / 16) × (value of data of G pixel at position corresponding to the position) + (filter value of image filter FL1a positioned at 2 rows × 3 columns; 2/16) × (R pixel at position corresponding to the position) Data value) + (filter value of image filter FL1a located at 3 rows and 1 column; 1/16) × (data value of G pixel at position corresponding to that position) + (positioned at 3 rows and 2 columns) Filter value of the image filter FL1a; 2/16) × (data value of the B pixel at the position corresponding to the position) + (filter value of the image filter FL1a located in the third row and the third column; 1/16) × (there Data value of G pixel at a position corresponding to the position). Therefore, the value of the filter coefficient corresponding to the R pixel is a total, (2/16) × 2 = 4/16 = 1/4, and the value of the filter coefficient corresponding to the B pixel is the total, (2/16) X2 = 4/16 = 1/4, and the value of the filter coefficient corresponding to the G pixel is (1/16) * 4 + (4/16) * 1 = 8/16 = 1/2 in total. Become. Therefore, the color composition ratio of the image filter FL1a is R: B: G = 1/4: 1/4: 1/2 = 1: 1: 2, and the color of the unit array in the original image data IS as described above. It is the same as the composition ratio.
画像フィルタFL1aは、原画像データISにおける単位配列の色組成比と同じ色組成比のフィルタ係数を備えるから、原画像データISに画像フィルタFL1aを作用させた場合において、原画像データISのR画素に対応するフィルタ係数の総計と、原画像データISのB画素に対応するフィルタ係数の総計と、原画像データISのG画素に対応するフィルタ係数の総計とは、原画像データISにおける単位配列の色組成比と同じになる。
Since the image filter FL1a has a filter coefficient having the same color composition ratio as that of the unit array in the original image data IS, when the image filter FL1a is applied to the original image data IS, the R pixel of the original image data IS. The total of the filter coefficients corresponding to the B pixels of the original image data IS, and the total of the filter coefficients corresponding to the G pixels of the original image data IS are the unit array in the original image data IS. It becomes the same as the color composition ratio.
このように原画像データISの単位配列におけるいずれの画素のデータも画像フィルタFL1aによって同じ色組成比でフィルタリングされる。したがって、原画像データISにおける全ての画素のデータが画像フィルタFL1aによって同じ色組成比でフィルタリングされることになる。この結果、前処理後画像データAISにおける全ての画素のデータは、図8Cに示すように、(G/2+R/4+B/4)で合成された同じ色成分を持つデータとして扱うことが可能となる。よって、次の対応点探索では、前処理後画像データAISにおける全ての画素のデータが使用可能となる。
Thus, the data of any pixel in the unit array of the original image data IS is filtered with the same color composition ratio by the image filter FL1a. Accordingly, all pixel data in the original image data IS are filtered with the same color composition ratio by the image filter FL1a. As a result, all pixel data in the pre-processed image data AIS can be handled as data having the same color component synthesized in (G / 2 + R / 4 + B / 4) as shown in FIG. 8C. . Therefore, in the next corresponding point search, all pixel data in the preprocessed image data AIS can be used.
このような前処理の一具体例を挙げると、図8Dに示す原画像は、画像フィルタFL1aのフィルタリングによって、図8Eに示す1チャンネル(単色)のモノクロ画像の前処理後画像となる。
As a specific example of such preprocessing, the original image shown in FIG. 8D becomes a pre-processed image of a single-channel (single color) monochrome image shown in FIG. 8E by the filtering of the image filter FL1a.
図6に戻って、次に、処理S3では、対応点探索処理部13によって対応点探索が実行される。この対応点探索では、2行2列の中央に位置する第5前処理後画像データAIS-22に対応する第5前処理後画像が基準画像とされ、この第5前処理後画像の周辺に位置する第1ないし第4および第6ないし第9前処理後画像データAIS-11~AIS-21、AIS-23~AIS-33に対応するに第1ないし第4および第6ないし第9前処理後画像それぞれが参照画像とされ、各参照画像ごとに基準画像に対する各対応点がそれぞれ探索される。この対応点探索では、本実施形態では、ピクセル単位で対応点が探索され(第1対応点探索処理)、続いて、サブピクセル単位で対応点が探索される(第2対応点探索処理)。前記第1対応点探索では、処理時間を短縮しつつより高精度に対応点を探索する観点から、まず、複数の画素おきに、例えば、8画素おきに対応点が探索され(粗対応点探索処理)、続いて、前記粗対応点探索処理で探索された位置の周辺で1画素おきに対応点が探索される(精対応点探索処理)。この各対応点探索処理には、公知の常套手段が利用され、対応点探索処理は、例えば特開2009-14445号公報等に開示されている。
Returning to FIG. 6, next, in the process S3, the corresponding point search processing unit 13 executes the corresponding point search. In this corresponding point search, the fifth preprocessed image corresponding to the fifth preprocessed image data AIS-22 located at the center of 2 rows and 2 columns is set as a reference image, and around the fifth preprocessed image, First to fourth and sixth to ninth pre-processing corresponding to the first to fourth and sixth to ninth post-processed image data AIS-11 to AIS-21 and AIS-23 to AIS-33 are located. Each subsequent image is set as a reference image, and each corresponding point for the reference image is searched for each reference image. In this corresponding point search, in this embodiment, corresponding points are searched for in units of pixels (first corresponding point searching process), and then corresponding points are searched for in units of subpixels (second corresponding point searching process). In the first corresponding point search, from the viewpoint of searching for corresponding points with higher accuracy while reducing processing time, first, corresponding points are searched every plural pixels, for example, every eight pixels (rough corresponding point search). Next, corresponding points are searched every other pixel around the position searched in the rough corresponding point search process (fine corresponding point search process). For each corresponding point search process, known conventional means are used, and the corresponding point search process is disclosed in, for example, Japanese Patent Application Laid-Open No. 2009-14445.
一例として、第1前処理後画像と第5前処理後画像との対応点探索について以下により具体的に説明する。第1前処理後画像と第5前処理後画像との位置ズレを視覚的に捉えるため、参考までに、第1前処理後画像と第5前処理後画像とを重ねて表示すると、図9Aに示す画像となる。このような第1前処理後画像と第5前処理後画像とにおいて、まず、第1対応点探索処理の粗対応点探索処理では、基準画像の第5前処理後画像上で、対応点を探索したい所定の画素位置を中心に所定の大きさを持つウィンドウWbが設定される。同様に、参照画像の第1前処理後画像上にも同じ大きさを持つウィンドウWraが、当該対応点探索されるこれら画像間のズレ方向に複数画素おきに複数設定される。例えば、第5前処理後画像の(x、y)=(100、150)に位置する画素に対応する参照画像の第1前処理後画像における画素位置を探索するものとすると、図9Bに示すように、基準画像の第5前処理後画像上で、画素位置(100、150)を中心にウィンドウWbが設定され、図9Cに示すように、参照画像の第1前処理後画像上に、第1前処理後画像と第5前処理後画像との間におけるズレ方向である2次元アレイ状配列の対角方向に複数画素おきに、例えばXY方向共に4画素おきに、8個のウィンドウWra1~Wra8が設定される。なお、図9Cには、5個のウィンドウWra1~Wra5が示されている。そして、基準画像の第5前処理後画像上におけるウィンドウWb(テンプレート)に対する、参照画像の第1前処理後画像上における各ウィンドウWra1~Wra8の一致度(相関度)が所定の手法で求められ(テンプレートマッチング)、最も一致度の高いウィンドウWraにおける中心位置の画素が対応点として求められる。前記所定の手法には、公知の常套手段、例えば、NCC(Normalized Cross Correlation)法、SAD(Sum of Absolute Difference)法(絶対誤差法)およびSSD(Sum of Squared Difference)法(二乗誤差法)等が用いられる。例えば、前記NCC法では、次式1で一致度RNCCの値が求められる。
As an example, the corresponding point search between the first pre-processed image and the fifth pre-processed image will be described more specifically below. In order to visually grasp the positional deviation between the first pre-processed image and the fifth pre-processed image, for reference, when the first pre-processed image and the fifth pre-processed image are displayed in an overlapping manner, FIG. The image shown in In the first pre-processed image and the fifth pre-processed image, first, in the rough corresponding point search process of the first corresponding point search process, the corresponding points are detected on the fifth pre-processed image of the reference image. A window Wb having a predetermined size around a predetermined pixel position to be searched is set. Similarly, a plurality of windows Wra having the same size on the first pre-processed image of the reference image are set every plurality of pixels in the shift direction between the images searched for the corresponding points. For example, assuming that the pixel position in the first pre-processed image of the reference image corresponding to the pixel located at (x, y) = (100, 150) in the fifth pre-processed image is searched, FIG. 9B shows. As shown in FIG. 9C, the window Wb is set around the pixel position (100, 150) on the fifth pre-processed image of the base image, and as shown in FIG. 9C, Eight windows Wra1 at intervals of a plurality of pixels in the diagonal direction of the two-dimensional array arrangement, which is a shift direction between the first preprocessed image and the fifth preprocessed image, for example, every four pixels in the XY direction. ~ Wra8 is set. FIG. 9C shows five windows Wra1 to Wra5. Then, the degree of coincidence (degree of correlation) of each of the windows Wra1 to Wra8 on the first preprocessed image of the reference image with respect to the window Wb (template) on the fifth preprocessed image of the standard image is obtained by a predetermined method. (Template matching), the pixel at the center position in the window Wra having the highest degree of matching is obtained as the corresponding point. Examples of the predetermined method include known conventional means, such as NCC (Normalized Cross Correlation) method, SAD (Sum of Absolute Difference) method (absolute error method), and SSD (Sum of Squared Difference) method (square error method). Is used. For example, in the NCC method, the value of the matching degree R NCC is obtained by the following formula 1.
ここで、I(i,j)は、基準画像のウィンドウWbにおける位置(i,j)の画素値(前処理後画像データAISにおける画素のデータ値)であり、T(i,j)は、参照画像のウィンドウWrにおける位置(i,j)の画素値(前処理後画像データAISにおける画素のデータ値)であり、Nは、ウィンドウWb、WrにおけるX方向(横方向)の長さ(画素単位)であり、そして、Mは、ウィンドウWb、WrにおけるY方向(縦方向)の長さ(画素単位)である。
Here, I (i, j) is a pixel value at the position (i, j) in the window Wb of the reference image (a pixel data value in the preprocessed image data AIS), and T (i, j) is This is the pixel value at the position (i, j) in the window Wr of the reference image (the pixel data value in the preprocessed image data AIS), and N is the length (pixel) in the X direction (horizontal direction) in the windows Wb and Wr. M is a length (pixel unit) in the Y direction (vertical direction) in the windows Wb and Wr.
これによって上述の例では、参照画像の第1前処理後画像における画素位置(100、150)に対し(-4、-4)、(0、0)、(4、4)、(8、8)、(12、12)、(16、16)、(20、20)、(24、24)だけずれた8画素、すなわち、(96、146)、(100、150)、(104、154)、(108、158)、(112、162)、(116、166)、(120、170)、(124、174)の8画素に対しテンプレートマッチングが実行され、最も一致度の高い画素が探索される。
Accordingly, in the above-described example, (−4, −4), (0, 0), (4, 4), (8, 8) with respect to the pixel position (100, 150) in the first preprocessed image of the reference image. ), (12, 12), (16, 16), (20, 20), (24, 24) shifted by 8 pixels, ie, (96, 146), (100, 150), (104, 154) , (108, 158), (112, 162), (116, 166), (120, 170), (124, 174) are subjected to template matching, and the pixel with the highest matching degree is searched. The
続いて、第1対応点探索処理の精対応点探索処理では、図9Dに示すように、前記粗対応点探索処理の結果、最も一致度の高いウィンドウWraの中心画素に対する周辺を1画素の精度で平面的に対応点探索処理が実行される。対応点探索の範囲は、所定の範囲、例えば、前記粗対応点探索処理で複数画素おき(上述の例ではXY方向共に4画素おき)に探索したが、この複数画素に応じた範囲(例えば±4画素の範囲)である。より具体的には、前記粗対応点探索処理の結果、最も一致度の高いウィンドウWrの中心画素に対する周辺±4画素それぞれを中心に81個のウィンドウWrb1~Wrb81が設定され、これら81個のウィンドウWrb1~Wrb81それぞれについてテンプレートマッチングが実行され、最も一致度の高いウィンドウWrbにおける中心位置の画素が対応点として求められる。
Subsequently, in the fine corresponding point searching process of the first corresponding point searching process, as shown in FIG. 9D, as a result of the rough corresponding point searching process, the periphery of the center pixel of the window Wra having the highest degree of coincidence is determined with one pixel accuracy. The corresponding point search process is executed in a plane. The corresponding point search range is a predetermined range, for example, every second pixel is searched in the rough corresponding point search process (in the above example, every four pixels in the XY direction). Range of 4 pixels). More specifically, as a result of the rough corresponding point search process, 81 windows Wrb1 to Wrb81 are set around each of ± 4 pixels around the center pixel of the window Wr having the highest degree of coincidence. These 81 windows Template matching is executed for each of Wrb1 to Wrb81, and the pixel at the center position in the window Wrb having the highest degree of matching is obtained as the corresponding point.
続いて、第2対応点探索処理では、図10Aに示すように、前記精対応点探索処理の結果、最も一致度の高いウィンドウWrbの中心画素およびこの中心画素の周辺8画素それぞれの9個の一致度に、図10Bに示すように、最も近似する2次曲面が求められ(2次曲面フィッティング)、この求められた2次曲面の頂点を与える座標値がサブピクセル単位の対応点として求められる(図10Bでは2次曲面の一断面を図示している)。このようなサブピクセル単位で対応点を求める手法は、例えば、「田中、矢口、古川、奥富、「位置ずれ量を考慮した画素選択に基づくロバスト超解像処理」、電子情報通信学会論文誌、Vol.J92-D、No.5、pp650-660、2009.、2009年5月」等に開示されている。なお、上述では、最も近似する2次曲面を求めることによってサブピクセル単位の対応点が求められたが、X方向およびY方向それぞれで最も近似する2次曲線がそれぞれ求められ、これら求められた各2次曲線のピークを与える各座標値がサブピクセル単位の対応点として求められてもよい。
Subsequently, in the second corresponding point search process, as shown in FIG. 10A, as a result of the fine corresponding point search process, the center pixel of the window Wrb having the highest degree of coincidence and the nine pixels around the center pixel are each nine. As shown in FIG. 10B, the degree of coincidence is obtained as the most approximate quadric surface (quadratic surface fitting), and the coordinate values giving the vertices of the obtained quadric surface are obtained as corresponding points in subpixel units. (FIG. 10B shows a section of a quadratic curved surface). The method for obtaining the corresponding points in units of sub-pixels is, for example, “Tanaka, Yaguchi, Furukawa, Okutomi,“ Robust super-resolution processing based on pixel selection considering misalignment ”, IEICE Transactions, Vol. J92-D, No. 5, pp650-660, 2009. , May 2009 ". In the above description, corresponding points in units of subpixels are obtained by obtaining the most approximate quadric surface, but quadratic curves that are most approximate in the X direction and the Y direction are obtained, respectively. Each coordinate value giving the peak of the quadratic curve may be obtained as a corresponding point in subpixel units.
図6に戻って、次に、処理S4では、処理S3で求められた各対応点間について、位置ズレ量演算部14によって各対応点間の位置ズレ量が求められる。例えば、単位長を1画素の大きさ(サイズ)とすれば、対応点間の座標差が位置ズレ量となる。すなわち、基準画像における画素の座標値と参照画像における、前記基準画像の画素に対応する画素の座標値との差が位置ズレ量となる。また例えば、単位長をm単位とすれば、対応点間の座標差に画素ピッチを乗じた値が位置ズレ量となる。前記画素ピッチは、画素の中心位置とこれに隣接する画素の中心位置との間の長さである。
Returning to FIG. 6, next, in process S <b> 4, for each corresponding point obtained in process S <b> 3, the positional deviation amount between the corresponding points is obtained by the positional deviation amount calculation unit 14. For example, if the unit length is a size (size) of one pixel, a coordinate difference between corresponding points becomes a positional deviation amount. That is, the difference between the coordinate value of the pixel in the standard image and the coordinate value of the pixel corresponding to the pixel of the standard image in the reference image is the positional deviation amount. Further, for example, when the unit length is m units, a value obtained by multiplying the coordinate difference between corresponding points by the pixel pitch is the positional deviation amount. The pixel pitch is a length between the center position of a pixel and the center position of a pixel adjacent thereto.
次に、処理S5では、超解像処理部15によって複数の画像から、これら複数の画像の解像度よりも高い解像度を持つ1個の画像(超解像画像)が求められる。この超解像処理には、公知の常套手段が利用され、超解像処理は、例えば、「「Fast and Robust Multiframe Super Resolution」、IEEE TRANNSACTION ON IMAGE PROCESSING,Vol.13,No.10,October 2004,pp1327-1344」等に開示されている。この文献に開示された再構成型超解像処理について、図11ないし図17を用いて以下に概説する。
Next, in step S5, the super-resolution processing unit 15 obtains one image (super-resolution image) having a resolution higher than the resolution of the plurality of images from the plurality of images. For this super-resolution processing, known conventional means are used. For example, “Fast and Robust Multiframe Super Resolution”, IEEE TRANSACTION ON IMAGE PROSING, Vol. 13, no. 10, Oct. 2004, pp 1327-1344. The reconstruction type super-resolution processing disclosed in this document will be outlined below with reference to FIGS.
この超解像処理では、原画像データをデモザイク(demosaic)した画像データが用いられ、図11に示す超解像処理は、色成分ごとに実行される。すなわち、画像データは、デモザイクによって、画素ごとに、R成分、B成分およびG成分の各値を持ち、図11に示す超解像処理は、R成分の各画素値で構成される画像(Rプレーン)、B成分の各画素値で構成される画像(Bプレーン)、および、G成分の各画素値で構成される画像(Gプレーン)それぞれに対して実行され、超解像化されたRプレーン、BプレーンおよびGプレーンを重ねることで、フルカラーの1個の超解像画像が生成される。以下、一例として、Gプレーンの場合について概説するが、RプレーンおよびBプレーンの場合も同様である。
In this super-resolution processing, image data obtained by demosaicing the original image data is used, and the super-resolution processing shown in FIG. 11 is executed for each color component. That is, the image data has values of R component, B component, and G component for each pixel by demosaic, and the super-resolution processing shown in FIG. Plane), an image composed of each pixel value of the B component (B plane), and an image composed of each pixel value of the G component (G plane) and super-resolved R By superimposing the plane, the B plane, and the G plane, one full-color super-resolution image is generated. Hereinafter, the case of the G plane will be outlined as an example, but the same applies to the case of the R plane and the B plane.
図11において、処理S11では、超解像処理部15に複数の画像およびその劣化情報が記憶部2から読み込まれる。例えば、本実施形態では、第1ないし第9原画像データIS-11~IS-33それぞれをデモザイクした第1ないし第9画像データDIS-11~DIS-33およびその劣化情報が記憶部2から超解像処理部15に読み込まれる。劣化情報は、図12に示すように、超解像処理後の理想的な超解像画像から見て、この前記超解像画像と画像データDISに対応する画像(以下、この超解像処理の説明において、「入力画像」と呼称する。)との間における対応関係を表す情報である。前記超解像画像の各画素値(この例ではG成分の画素値)を持つ1次元ベクトルをXとし、画像データDISに対応する入力画像の各画素値(この例ではG成分の画素値)を持つ1次元ベクトルをYkとし、後述する初期画像の出力候補画像X0を生成した元の入力画像Ykと他の各入力画像Ykそれぞれとの位置ズレ量を行列Fkとし、後述する出力候補画像Xnのサイズを入力画像Ykのサイズに合わせるためのダウンサンプリング量をDkとし、そして、入力画像のボケ量(本実施形態では、撮像光学系31のPSF(Point spread function、点拡がり関数、点像分布関数)情報)をHkとすると、前記劣化情報は、DkHkFkであり、Yk=DkHkFkX(X=Fk
THk
TDk
TYk)の関係が成り立つ。なお、kは、第1ないし第9原画像データIS-11~IS-33それぞれに対応する各入力画像を指し、1~9の整数値である。上付きの添え字Tは、転置行列であることを示す。この劣化情報は、後述の処理S14における残差の演算に用いられる。
In FIG. 11, in process S <b> 11, a plurality of images and their deterioration information are read from the storage unit 2 into the super-resolution processing unit 15. For example, in the present embodiment, the first to ninth image data DIS-11 to DIS-33 obtained by demosaicing each of the first to ninth original image data IS-11 to IS-33 and the deterioration information thereof are stored in the storage unit 2. It is read into the resolution processing unit 15. As shown in FIG. 12, the deterioration information is an image corresponding to the super-resolution image and the image data DIS (hereinafter referred to as the super-resolution process) when viewed from an ideal super-resolution image after the super-resolution process. In this description, it is referred to as “input image”). A one-dimensional vector having each pixel value (G component pixel value in this example) of the super-resolution image is X, and each pixel value (G component pixel value in this example) of the input image corresponding to the image data DIS. a one-dimensional vector having a Y k, the amount of positional shift matrix F k of the output candidates from which generated the image X 0 input image Y k and the other of each input image Y k each described later initial image, described later downsampling amount for adjusting the size of the output candidate image X n the size of the input image Y k to a D k, then the blur amount of the input image (in this embodiment, the imaging optical system 31 of the PSF (Point spread function When the point spread function, the point spread function) information) and H k, the degradation information is D k H k F k, Y k = D k H k F k X (X = F k T H k T D k T Y k ). Note that k indicates each input image corresponding to each of the first to ninth original image data IS-11 to IS-33, and is an integer value of 1 to 9. A superscript T indicates a transposed matrix. This deterioration information is used for calculating a residual in the process S14 described later.
次に、処理S12では、超解像処理部15によって入力画像Ykから、初期画像としての出力候補画像Xn(ここでは初期画像なのでN=0)が作成される。この初期画像の出力候補画像X0は、超解像画像Xを生成するために後述するように処理S12~処理S15の各処理を繰り返し実行する繰り返し処理における最初の出力候補画像Xnである。この出力候補画像X0は、生成すべき超解像画像Xと同じ解像度(同じサイズ)を持つ画像であり、例えば、第1ないし第9原画像データIS-11~IS-33に対応する複数の第1ないし第9入力画像Ykのうちの1つから例えばバイリニア等の補間処理によって不足画素を補うことで生成される。本実施形態では、上述の処理S1ないし処理S4によって、第5原画像データIS-22に対応する画像に対し、周辺の第1ないし第4および第6ないし第9原画像データIS-11~IS-21、IS-23~IS-33それぞれに対応する各画像の位置ズレ量を求めているので、第5原画像データIS-22に対応する第5画像データDISから初期画像の出力候補画像X0が作成され、前記位置ズレ量Fkは、上述の処理S1ないし処理S4によって求められた各位置ズレ量となる。入力画像Ykは、500×500のサイズであり、超解像画像Xが1500×1500のサイズであるとすると、この出力候補画像X0の生成において、入力画像Ykは、X方向(横方向)およびY方向(縦方向)共に3倍に拡大されることになる。入力画像Ykにおける互いに隣接する画素間には、入力画像Ykが3倍に拡大されると、X方向およびY方向それぞれにおいて、画素値の無い2個の画素が新たにそれぞれ生じることになり(2次元アレイでは画素値の無い8個の画素が新たに生じる)、これら画素値の無い画素の画素値が前記補間処理によって生成される。
Next, in process S12, the super-resolution processing unit 15 creates an output candidate image X n (here, N = 0 because it is an initial image) as an initial image from the input image Y k . The output candidate image X 0 of the initial image is the first output candidate image X n in the repeating process executed repeatedly each processing of the processing S12 ~ process S15 as will be described later to generate a super-resolution image X. This output candidate image X 0 is an image having the same resolution (same size) as the super-resolution image X to be generated. For example, a plurality of output candidate images X 0 corresponding to the first to ninth original image data IS-11 to IS-33. Are generated from one of the first to ninth input images Y k by compensating for the deficient pixels by, for example, bilinear interpolation. In the present embodiment, the first to fourth and sixth to ninth original image data IS-11 to IS in the vicinity of the image corresponding to the fifth original image data IS-22 by the above-described processing S1 to S4. Since the positional deviation amount of each image corresponding to each of −21, IS-23 to IS-33 is obtained, the output candidate image X of the initial image is derived from the fifth image data DIS corresponding to the fifth original image data IS-22. 0 is created, and the positional deviation amount F k is the positional deviation amounts obtained by the above-described processing S1 to processing S4. If the input image Y k has a size of 500 × 500 and the super-resolution image X has a size of 1500 × 1500, the input image Y k is generated in the X direction (horizontal) in the generation of the output candidate image X 0. (Direction) and Y direction (longitudinal direction) are enlarged three times. Between adjacent pixels in the input image Y k, the input image Y k is larger 3 times, in the X and Y directions, respectively, two pixels without pixel values will be generated anew each (In the two-dimensional array, eight pixels having no pixel value are newly generated), and pixel values of pixels having no pixel value are generated by the interpolation process.
次に、処理S13では、超解像処理部15によって、入力画像Ykにノイズがあってもロバストに収束できるように、正則化項としていわゆるBTV量(Bilateral Total Value)が求められる。このBTV量は、後述の式2における右辺第3項である。
Next, in process S13, a so-called BTV amount (bilateral total value) is obtained as a regularization term so that the super-resolution processing unit 15 can robustly converge even if there is noise in the input image Yk . This BTV amount is the third term on the right side in Equation 2 described later.
次に、処理S14では、超解像処理部15によって、第1ないし第9画像データDIS-11~DIS-33に対応する第1ないし第9入力画像Ykそれぞれについて、これら各入力画像Ykの劣化情報に応じて入力画像Ykの解像度(大きさ)と同じ解像度(大きさ)に変換した出力候補画像Xnとの差である残差がそれぞれ求められる。この残差は、後述の式2における右辺第2項である。この残差は、例えばベイヤー配列の画像を超解像処理する場合であって、Gプレーンの超解像画像Xを生成する場合には、G画素のみを用いて求められる。これは、図13に示すように、1つの画像を超解像処理する場合に、当該画像に無い画素値を、位置ズレ量を利用して他の画像の画素値で補うことを意味する。
Next, in process S14, the super-resolution processing unit 15 applies the first to ninth input images Y k corresponding to the first to ninth image data DIS-11 to DIS-33 for each of the input images Y k. Residuals that are differences between the resolution (size) of the input image Yk and the output candidate image Xn converted to the same resolution (size) according to the deterioration information are obtained. This residual is the second term on the right side in Equation 2 described later. This residual is obtained, for example, when only a G pixel is used when a G-plane super-resolution image X is generated when super-resolution processing is performed on a Bayer array image. As shown in FIG. 13, when super-resolution processing is performed on one image, this means that a pixel value not included in the image is supplemented with a pixel value of another image using a positional shift amount.
この残差を求める処理である残差処理S14について、図14ないし図17を参照しつつ、より具体的に説明すると、まず、図14Aおよび図14Bに示すように、当該入力画像Ykの劣化情報が出力候補画像Xnに乗じられ、当該入力画像Ykの劣化情報に応じて前記画像の解像度(大きさ)と同じ解像度(大きさ)に変換した出力候補画像(DkHkFkXn)が生成される(図17の処理S22-1~S22-9ないし処理S24-1~S24-9)。なお、nは、本実施形態では、前記繰り返し処理の回数が、通常収束すると判断できる例えば200回の定数値に設定されているため、0~199の整数値である。したがって、出力候補画像X199は、この処理S5の超解像処理で出力される超解像画像Xとなる(X=X199)。この残差処理について、より詳しくは、まず、PSF情報をフィルタ係数に持つ画像フィルタで出力候補画像Xnがフィルタリングされ(図17の処理S22-1~S22-9)、これによって出力候補画像Xnのボケ画像が生成され、前記入力画像Ykの解像度と同じ解像度となるように、出力候補画像Xnの元の第5画像データDIS-22に対応する第5入力画像Y5と当該入力画像Ykとの位置ズレ量Fkを考慮してダウンサンプリングが行われる(処理S23-1~S23-9、処理S24-1~S24-9)。例えば、図14Cに示すように、第5入力画像Y5と当該入力画像Ykとの位置ズレ量FkがX方向(横方向)に1/3画素であって、X方向(横方向)およびY方向(縦方向)共に1/3にダウンサンプリングする場合(1500×1500→500×500)には、ダウンサンプリング前の画像でX方向に1画素(=1/3画素×3倍)だけずらした位置に位置する3×3の9画素における各画素値の平均値が、ダウンサンプリングした画素値とされる。次に、図15に示すように、ダウンサンプリング後の出力候補画像(DkHkFkXn)(図15A)から、入力画像Yk(図15B)が減算され、この減算結果(DkHkFkXn-Yk)が符号関数sign(α)で演算される(図15C、図17の処理S25-1~S25-9))。この符号関数sign(α)は、αが0である場合には0を出力し、αが正である場合には1を出力し、そして、αが負である場合には-1を出力する演算子である。したがって、符号関数sign(α)の演算によって、前記減算結果(DkHkFkXn-Yk)が0である場合には、0が出力され、前記減算結果(DkHkFkXn-Yk)が正である場合には、1が出力され、そして、前記減算結果(DkHkFkXn-Yk)が負である場合には、-1が出力される。次に、図16Aおよび図16Bに示すように、符号関数sign(α)で演算された前記減算結果(DkHkFkXn-Yk)(差異)に影響した超解像画像Xnの画素が調べられる(図17の処理S26-1~S26-9ないし処理S28-1~S28-9)。上述の処理が上述の行列DkHkFkで定義された場合、この処理は、転置行列を用いた行列Fk
THk
TDk
Tで定義される。すなわち、この処理は、出力候補画像Xnの元の第5画像データDIS-22に対応する第5入力画像Y5と当該入力画像Ykとの位置ズレ量Fkを考慮したアップサンプリングを行った後に(Fk
TDk
T、図17の処理S26-1~S26-9)、ボケの影響を元に戻す処理(Hk
T)となる(Fk
THk
TDk
Tsign(DkHkFkXn-Yk)、図17の処理S27-1~S27-9、処理S28-1~S28-9)。例えば、図16Cに示すように、上述の例で、第5入力画像Y5と当該入力画像Ykとの位置ズレ量FkがX方向(横方向)に1/3画素であって、X方向(横方向)およびY方向(縦方向)共に3倍にアップサンプリングする場合(500×500→1500×1500)には、アップサンプリング後の画像でX方向に1画素(=1/3画素×3倍)だけずらした位置に位置する3×3の9画素それぞれに、差異sign(DkHkFkXn-Yk)が、アップサンプリングの前後における各画素の面積比(この図16Cの例では1/9)で分配される。これによって1個の入力画像Ykに対する差異の画像が生成される。この例では入力画像Ykは、9個あるので、この処理が、図17に示すように、第1ないし第9入力画像Ykそれぞれに対し実行され、9個の差異の画像が生成される。そして、これら9個の差異の総計が前記残差とされる。
The residual process S14, which is a process for obtaining the residual, will be described more specifically with reference to FIGS. 14 to 17. First, as shown in FIGS. 14A and 14B, degradation of the input image Yk is performed. information is multiplied by the output candidate image X n, the input image Y k output candidate image (D converted to the same resolution (size) and resolution (size) of the image in accordance with the deterioration information of k H k F k X n ) is generated (processing S22-1 to S22-9 to processing S24-1 to S24-9 in FIG. 17). In this embodiment, n is an integer value of 0 to 199 because the number of repetitions is set to a constant value of 200 times that can be determined to normally converge. Therefore, the output candidate image X 199 becomes the super-resolution image X output by the super-resolution processing of this processing S5 (X = X 199 ). In more detail, the output candidate image Xn is first filtered by an image filter having PSF information as a filter coefficient (processes S22-1 to S22-9 in FIG. 17). n of the blurred image is generated, so as to have the same resolution as the resolution of the input image Y k, fifth input image Y 5 and the input corresponding to the fifth image data DIS-22 of the original output candidate image X n considering position shift amount F k of the image Y k downsampling is performed (processing S23-1 ~ S23-9, the process S24-1 ~ S24-9). For example, as shown in FIG. 14C, the positional deviation amount F k between the fifth input image Y 5 and the input image Y k is 1/3 pixel in the X direction (horizontal direction), and the X direction (horizontal direction). In the case of downsampling to 1/3 in both the Y direction (vertical direction) (1500 × 1500 → 500 × 500), only one pixel in the X direction (= 1/3 pixel × 3 times) in the image before downsampling The average value of the pixel values of the 9 pixels of 3 × 3 located at the shifted position is the downsampled pixel value. Next, as shown in FIG. 15, the input image Y k (FIG. 15B) is subtracted from the output candidate image (D k H k F k X n ) (FIG. 15A) after downsampling, and this subtraction result (D k H k F k X n -Y k) is calculated by the sign function sign (α) (Fig. 15C, the process of FIG. 17 S25-1 ~ S25-9)). The sign function sign (α) outputs 0 when α is 0, outputs 1 when α is positive, and outputs −1 when α is negative. It is an operator. Therefore, the calculation of the sign function sign (alpha), when the subtraction result (D k H k F k X n -Y k) is 0, 0 is output, the subtraction result (D k H k F when k X n -Y k) is positive, 1 is output, and, wherein when the subtraction result (D k H k F k X n -Y k) is negative, -1 output Is done. Next, as shown in FIG. 16A and FIG. 16B, the super-resolution image X affected the subtraction result (D k H k F k X n −Y k ) (difference) calculated by the sign function sign (α). n pixels are examined (processing S26-1 to S26-9 to processing S28-1 to S28-9 in FIG. 17). When the above process is defined by the matrix D k H k F k described above, this process is defined by a matrix F k T H k T D k T using a transposed matrix. That is, this process is carried out upsampling taking into account the positional shift amount F k of the fifth input image Y 5 and the input image Y k corresponding to the fifth image data DIS-22 original output candidate image X n (F k T D k T , processing S26-1 to S26-9 in FIG. 17), the processing (H k T ) for restoring the influence of blur (F k T H k T D k T sign ( D k H k F k X n -Y k), the process of FIG. 17 S27-1 ~ S27-9, the process S28-1 ~ S28-9). For example, as shown in FIG. 16C, in the above example, the positional deviation amount F k between the fifth input image Y 5 and the input image Y k is 1/3 pixel in the X direction (horizontal direction), and X When upsampling is performed three times in both the direction (horizontal direction) and the Y direction (vertical direction) (500 × 500 → 1500 × 1500), one pixel in the X direction (= 1/3 pixel × three times) by each 9 pixels of 3 × 3 located at a position shifted, the difference sign (D k H k F k X n -Y k) is, the area ratio of each pixel before and after upsampling (FIG 16C In the example shown in FIG. As a result, a difference image for one input image Yk is generated. In this example, since there are nine input images Y k , this processing is executed for each of the first to ninth input images Y k as shown in FIG. 17 to generate nine difference images. . The total of these nine differences is the residual.
図11に戻って、次に、処理S15では、超解像処理部15によって、式2に示すように、上述の処理S13および処理S14それぞれで求めたBTV量および残差を、出力候補画像Xnから減算することによって、次の出力候補画像Xn+1が生成される。そして、この出力候補画像Xn+1が処理S12の出力候補画像Xnとされ、処理がS12に戻される。
Referring back to FIG. 11, next, in process S15, the super-resolution processing unit 15 calculates the BTV amount and the residual obtained in the above-described processes S13 and S14, respectively, as shown in Expression 2, using the output candidate image X By subtracting from n , the next output candidate image Xn + 1 is generated. Then, this output candidate image Xn + 1 is set as the output candidate image Xn of the process S12, and the process returns to S12.
なお、λは、BTV量の影響度を調整するパラメータであり、βは、残差およびλ×BTV量の影響度を調整するパラメータであり、これらλ、βは、適宜に設定される。
Note that λ is a parameter for adjusting the degree of influence of the BTV amount, β is a parameter for adjusting the degree of influence of the residual and λ × BTV amount, and these λ and β are appropriately set.
そして、このような処理S12ないし処理15の各処理が、出力候補画像Xnが通常収束すると考えられる所定回数、例えば、上述したように200回繰り返される。処理S12ないし処理S15の各処理を200回繰り返すと、処理S5の超解像処理が終了し、出力候補画像X199が超解像画像Xとされる。
Then, each of the processes S12 to S15 is repeated a predetermined number of times that the output candidate image Xn is normally converged, for example, 200 times as described above. When each of the processes S12 to S15 is repeated 200 times, the super-resolution process of process S5 is completed, and the output candidate image X 199 is set as the super-resolution image X.
図6に戻って、このように処理S5の超解像処理が終了すると、処理S6において、この超解像画像Xが、記憶部2に記憶され、そして、出力部5に出力され、表示される。
Returning to FIG. 6, when the super-resolution process of the process S5 is completed in this way, in the process S6, the super-resolution image X is stored in the storage unit 2 and then output to the output unit 5 and displayed. The
このように本実施形態における画像処理装置Daでは、画像フィルタFLによって単位配列(単位色フィルタ配列)の色組成比と同じ色組成比を持つデータに原画像データISを変換するように輝度成分生成処理部によってフィルタリングするので、全ての画素のデータを同じ色成分を持つデータとして扱うことができるようになる。この結果、全ての画素のデータを利用して対応点探索の実施が可能となる。したがって、本実施形態における画像処理装置Daは、対応点探索をより高精度に実施できる。
As described above, in the image processing apparatus Da according to the present embodiment, the luminance component generation is performed so that the original image data IS is converted into data having the same color composition ratio as that of the unit array (unit color filter array) by the image filter FL. Since filtering is performed by the processing unit, data of all pixels can be handled as data having the same color component. As a result, the corresponding point search can be performed using the data of all the pixels. Therefore, the image processing device Da according to the present embodiment can perform the corresponding point search with higher accuracy.
本実施形態における画像処理装置Daでは、画像フィルタFLにおけるフィルタ係数の各値が中心位置に対し対称に配置されているので、中心に位置する画素(中心画素)に対する周辺に位置する各画素(各周辺画素)における各画素値は、当該周辺画素と中心画素との距離に応じて重み付けられることになり、適切に中心画素に反映される。このため、本実施形態における画像処理装置Daは、全ての画素のデータをより同等な色成分を持つデータとして扱うことができるから、対応点探索をより高精度に実施できる。
In the image processing device Da according to the present embodiment, since each value of the filter coefficient in the image filter FL is arranged symmetrically with respect to the center position, each pixel (each pixel) positioned around the center pixel (center pixel) Each pixel value in the peripheral pixel) is weighted according to the distance between the peripheral pixel and the central pixel, and is appropriately reflected in the central pixel. For this reason, the image processing device Da according to the present embodiment can handle the data of all the pixels as data having more equivalent color components, and therefore can perform the corresponding point search with higher accuracy.
本実施形態における画像処理装置Daでは、位置ズレ量演算部14を備えるので、対応点探索処理部13で探索された対応点間の位置ズレ量Fを求めることができ、より高精度に実行された対応点探索の結果を位置ズレ量Fの推定に効果的に反映できる。
Since the image processing apparatus Da according to the present embodiment includes the positional deviation amount calculation unit 14, the positional deviation amount F between corresponding points searched by the corresponding point search processing unit 13 can be obtained, and is executed with higher accuracy. The corresponding point search result can be effectively reflected in the estimation of the positional deviation amount F.
本実施形態における画像処理装置Daでは、9眼のアレイカメラを用いるので、複数の原画像データを同時に得ることができる。このため、対応点探索の結果から対応点間の位置ズレ量Fを求める場合に、同一の被写体を同時刻で写した複数の画像間で前記位置ズレ量Fを求めることになるので、本実施形態における画像処理装置Daは、より高精度に前記位置ズレ量Fを求めることができる。
Since the image processing apparatus Da according to the present embodiment uses a nine-eye array camera, a plurality of original image data can be obtained simultaneously. For this reason, when the positional deviation amount F between corresponding points is obtained from the result of the corresponding point search, the positional deviation amount F is obtained between a plurality of images obtained by copying the same subject at the same time. The image processing device Da in the embodiment can determine the positional deviation amount F with higher accuracy.
本実施形態における画像処理装置Daでは、超解像処理部15を備えるので、対応点探索処理部13で探索された対応点間の位置ズレ量Fを求め、これに基づいて複数の原画像データから高解像度な画像のデータを得ることができ、より高精度に実行された対応点探索の結果を高解像度な画像の生成に効果的に反映できる。
Since the image processing apparatus Da according to the present embodiment includes the super-resolution processing unit 15, a positional deviation amount F between corresponding points searched by the corresponding point search processing unit 13 is obtained, and a plurality of original image data is obtained based on this. Thus, high-resolution image data can be obtained, and the result of corresponding point search performed with higher accuracy can be effectively reflected in the generation of a high-resolution image.
なお、上述では、前記論文に開示された超解像処理について概説したが、超解像処理は、この手法に限定されるものではなく、位置ズレ量を用いて相対的に低解像度な複数の画像から相対的に高解像度な1個の超解像画像を生成する再構成型超解像処理を用いることができる。例えば、図18に示す再構成型超解像処理を用いることができる。図18は、第1実施形態の画像処理装置における第2態様の超解像処理を示すフローチャートである。
In the above description, the super-resolution processing disclosed in the paper has been outlined. However, the super-resolution processing is not limited to this method, and a plurality of relatively low resolutions using a positional shift amount are used. A reconfigurable super-resolution process that generates one super-resolution image having a relatively high resolution from the image can be used. For example, the reconfiguration super-resolution processing shown in FIG. 18 can be used. FIG. 18 is a flowchart showing the super-resolution processing of the second aspect in the image processing apparatus of the first embodiment.
この超解像処理は、式3によって示される処理であり、式3の右辺第2項は、残差(Fk
THk
TDk
T(DkHkFkXn-Yk)であり、式3の右辺第3項は、正則化項Cである。この正則化項Cでは、例えば4近傍ラプラシアンの画像フィルタが用いられる。この4近傍ラプラシアンの画像フィルタは、例えば、1行1列、1行3列、3行1列および3行3列の各位置のフィルタ係数が0であり、1行2列、2行1列、2行3列および3行2列の各位置のフィルタ係数が1であり、そして、2行2列の位置のフィルタ係数が-4である。
The super-resolution process is a process to be represented by the formula 3, the second term on the right side of the equation 3, the residual (F k T H k T D k T (D k H k F k X n -Y k) The third term on the right-hand side of Equation 3 is a regularization term C. In this regularization term C, for example, a 4-neighbor Laplacian image filter is used. The filter coefficient at each position of 1st column, 1st row, 3rd column, 3rd row, 1st column and 3rd row and 3rd column is 0, and each position of 1st row, 2nd column, 2nd row, 1st column, 2nd row, 3rd column and 3rd row, 2nd column And the filter coefficient at the position of 2 rows and 2 columns is -4.
なお、λは、正則化項の影響度を調整するパラメータであり、βは、残差およびλ×正則化項の影響度を調整するパラメータであり、これらλ、βは、適宜に設定される。
Note that λ is a parameter for adjusting the degree of influence of the regularization term, β is a parameter for adjusting the degree of influence of the residual and λ × regularization term, and these λ and β are appropriately set. .
この図18に示す再構成型超解像処理では、図11に示す上述の超解像処理と略同様に処理され、まず、処理S31では、超解像処理部15に複数の画像およびその劣化情報が記憶部2から読み込まれる。次に、処理S32では、初期画像の出力候補画像X0が生成される。次に、処理S33では、4近傍ラプラシアンの画像フィルタで正則化項が求められる。次に、処理S34では、残差(Fk
THk
TDk
T(DkHkFkXn-Yk)が求められる。次に、処理S35では、式3に示すように、上述の処理S33および処理S34それぞれで求めた正則化項および残差を、出力候補画像Xnから減算することによって、次の出力候補画像Xn+1が生成される。そして、この出力候補画像Xn+1が処理S32の出力候補画像Xnとされ、処理がS32に戻される。そして、このような処理S32ないし処理35の各処理が、出力候補画像Xnが通常収束すると考えられる所定回数、例えば、40回繰り返される。処理S32ないし処理S35の各処理を40回繰り返すと、処理S5の超解像処理が終了し、出力候補画像X39が超解像画像Xとされる。
In the reconstruction-type super-resolution process shown in FIG. 18, processing is performed in substantially the same manner as the above-described super-resolution process shown in FIG. 11. First, in process S31, a plurality of images and their degradations are displayed in the super-resolution processing unit 15. Information is read from the storage unit 2. Then, the processing S32, the output candidate image X 0 of the initial image is generated. Next, in process S33, a regularization term is obtained by an image filter of 4 neighborhood Laplacian. Then, the processing S34, the residual (F k T H k T D k T (D k H k F k X n -Y k) is calculated. Then, the processing S35, as shown in Equation 3, the regularization term and the residual obtained in process S33 and the processing S34, respectively described above, by subtracting from the output candidate image X n, the next output candidate image X n + 1 is generated. Then, the output candidate image X n + 1 Is the output candidate image Xn of the process S32, and the process is returned to S32. Then, each process of the processes S32 to 35 is performed a predetermined number of times that the output candidate image Xn is normally converged, for example, Repeated 40 times When each of the processes S32 to S35 is repeated 40 times, the super-resolution process of process S5 is completed, and the output candidate image X39 is set as the super-resolution image X.
次に、第2実施形態について説明する。
Next, a second embodiment will be described.
(第2実施形態)
図19は、第2実施形態における画像処理装置の構成を示すブロック図である。図20は、第2実施形態における画像処理装置の動作を示すフローチャートである。 (Second Embodiment)
FIG. 19 is a block diagram illustrating a configuration of an image processing apparatus according to the second embodiment. FIG. 20 is a flowchart illustrating the operation of the image processing apparatus according to the second embodiment.
図19は、第2実施形態における画像処理装置の構成を示すブロック図である。図20は、第2実施形態における画像処理装置の動作を示すフローチャートである。 (Second Embodiment)
FIG. 19 is a block diagram illustrating a configuration of an image processing apparatus according to the second embodiment. FIG. 20 is a flowchart illustrating the operation of the image processing apparatus according to the second embodiment.
第1実施形態における画像処理装置Daは、相対的に低解像度な複数の画像(原画像データ)から相対的に高解像度な1個の画像を生成する装置であるが、第2実施形態における画像処理装置Dbは、複数の画像(原画像データ)から所定に被写体までの距離を求める装置である。このような第2実施形態における画像処理装置Dbは、図19に示すように、制御処理部1bと、記憶部2と、画像取得部3と、入力部4と、出力部5とを備える。これら第2実施形態の画像処理装置Dbにおける記憶部2、画像取得部3、入力部4および出力部5は、それぞれ、第1実施形態の画像処理装置Daにおける記憶部2、画像取得部3、入力部4および出力部5と同様であるので、その説明を省略する。
The image processing device Da according to the first embodiment is a device that generates one relatively high resolution image from a plurality of relatively low resolution images (original image data), but the image according to the second embodiment. The processing device Db is a device that obtains a predetermined distance from a plurality of images (original image data) to a subject. As shown in FIG. 19, the image processing apparatus Db in the second embodiment includes a control processing unit 1 b, a storage unit 2, an image acquisition unit 3, an input unit 4, and an output unit 5. The storage unit 2, the image acquisition unit 3, the input unit 4, and the output unit 5 in the image processing device Db of the second embodiment are respectively the storage unit 2, the image acquisition unit 3, and the image processing device Da of the first embodiment. Since it is the same as that of the input part 4 and the output part 5, the description is abbreviate | omitted.
制御処理部1bは、制御処理部1aと同様に、所定の画像処理プログラムに基づき後述の画像処理を実行するべく、画像処理装置Dbの各部を当該各部の機能に応じてそれぞれ制御するものである。制御処理部1bは、例えば、CPUおよびその周辺回路を備えて構成され、所定の画像処理プログラムを実行することによって、機能的に、制御部11、前処理部12、対応点探索処理部13、位置ズレ量演算部14および距離演算部16が構成される。すなわち、制御処理部1bは、制御処理部1aに対し、超解像処理部15に代え、距離演算部16を備えている。このため、第2実施形態の制御処理部1bにおける制御部11、前処理部12、対応点探索処理部13および位置ズレ量演算部14は、それぞれ、第1実施形態の制御処理部1aにおける制御部11、前処理部12、対応点探索処理部13および位置ズレ量演算部14と同様であるので、その説明を省略する。
Similar to the control processing unit 1a, the control processing unit 1b controls each unit of the image processing device Db according to the function of each unit in order to execute image processing described later based on a predetermined image processing program. . The control processing unit 1b includes, for example, a CPU and its peripheral circuits, and functionally executes a predetermined image processing program, so that the control unit 11, the preprocessing unit 12, the corresponding point search processing unit 13, A positional deviation amount calculation unit 14 and a distance calculation unit 16 are configured. That is, the control processing unit 1b includes a distance calculation unit 16 instead of the super-resolution processing unit 15 with respect to the control processing unit 1a. For this reason, the control unit 11, the preprocessing unit 12, the corresponding point search processing unit 13, and the positional deviation amount calculation unit 14 in the control processing unit 1b of the second embodiment are respectively controlled by the control processing unit 1a of the first embodiment. Since this is the same as the unit 11, the preprocessing unit 12, the corresponding point search processing unit 13, and the positional deviation amount calculation unit 14, description thereof is omitted.
距離演算部16は、位置ズレ量演算部14で求めた対応点間の位置ズレ量を複数の原画像データに対応する複数の画像間における視差とし、前記視差に基づいて前記対応点の表す被写体までの距離を求めるものである。
The distance calculation unit 16 uses the positional shift amount between corresponding points obtained by the positional shift amount calculation unit 14 as parallax between a plurality of images corresponding to a plurality of original image data, and the subject represented by the corresponding point based on the parallax The distance to is obtained.
このような第2実施形態における画像処理装置Dbは、図20に示すように、まず、処理S41では、上述の処理S1と同様に、画像取得部3によって9枚の画像それぞれに対応する原画像データが当該画像処理装置Dbに入力され、記憶部2に記憶される。次に、処理S42では、上述の処理S2と同様に、これら9枚の各画像それぞれに対し前処理部12によって前処理が実行される。次に、処理S43では、上述の処理S3と同様に、これら9枚の画像における中心(2行2列)に位置する第5画像が基準画像とされ、この基準画像に対し、周辺に位置する8枚の画像それぞれについて、対応点探索処理部13によって対応点探索が実行される。この対応点探索処理では、所定の被写体に対応する点が適宜に選択される。本実施形態では、例えば、当該画像処理装置をDbを搭載した当該車両とその先行車両(または対向車両)との間における車間距離を求める場合では、前記先行車両(または前記対向車両)に対応する点が選択され、また例えば、当該画像処理装置をDbを搭載した当該車両と歩行者との間の距離を求める場合には、歩行者に対応する点が選択される。
As shown in FIG. 20, the image processing apparatus Db according to the second embodiment first has an original image corresponding to each of nine images by the image acquisition unit 3 in the process S41 as in the above-described process S1. Data is input to the image processing apparatus Db and stored in the storage unit 2. Next, in the process S42, as in the above-described process S2, the preprocessing unit 12 performs a preprocess on each of these nine images. Next, in the process S43, as in the above-described process S3, the fifth image located in the center (2 rows and 2 columns) of these nine images is set as the reference image, and is located in the periphery with respect to this reference image. Corresponding point search is executed by the corresponding point search processing unit 13 for each of the eight images. In this corresponding point search process, a point corresponding to a predetermined subject is appropriately selected. In the present embodiment, for example, when the inter-vehicle distance between the vehicle on which the image processing device is mounted with Db and its preceding vehicle (or oncoming vehicle) is obtained, it corresponds to the preceding vehicle (or oncoming vehicle). For example, when the distance between the vehicle on which the image processing apparatus is mounted with Db and the pedestrian is obtained, the point corresponding to the pedestrian is selected.
次に、処理S44では、上述の処理S4と同様に、位置ズレ量演算部14によって、位置ズレ量が求められる。
Next, in process S44, as in the above-described process S4, the positional deviation amount calculation unit 14 determines the positional deviation amount.
そして、本実施形態では、次に、処理S45では、距離演算部16によって、位置ズレ量演算部14で求めた位置ズレ量が基準画像(2行2列に位置する画像)と参照画像(その周辺に位置する各画像)との間における視差とされ、前記視差に基づいて被写体までの距離が求められる。このような測距技術には、公知の常套手段が利用され、このような測距技術は、例えば特開2010-276469号公報や前記特開2009-14445号公報等に開示されている。この測距技術では、大略、予め定める間隔(基線長)だけ離間して設けられた一対のカメラによって被写体(対象物)の各画像が得られ、これら各画像に対し対応点探索がピクセル単位で実行され、これら各画像から前記離間方向における前記一対のカメラ間の視差がピクセル単位で求められ、この求められた視差に基づいて被写体までの距離がいわゆる三角測量の原理に基づいて求められる。より具体的には、図21において、少なくとも焦点距離(f)、撮像面(撮像部33(例えばCCD型イメージセンサ))の画素数、1画素の大きさ(μ)が相互に等しい2台の第1および第2カメラが、所定の基線長(L)だけ左右に離間させて光軸AX-1、AX-2を平行に配置され、これら第1および第2カメラで物体(被写体)Obが撮影された場合に、撮像面IP-1、IP-2上の視差(ずれ画素数)がd(=d1+d2)であるとすると、前記物体(被写体)Obまでの距離(Z)は、斜線を施して示す三角形が相似であることから、
Z:f=L:μ×d
の関係があり、
Z=(L×f)/(μ×d)
で求めることができる。 In the present embodiment, next, in step S45, the positional deviation amount obtained by the positional deviationamount computing unit 14 by the distance computing unit 16 is determined based on the reference image (image located in 2 rows and 2 columns) and the reference image (then the image). The distance to the subject is determined based on the parallax. For such distance measurement technology, known conventional means are used, and such distance measurement technology is disclosed in, for example, Japanese Patent Application Laid-Open No. 2010-276469 and Japanese Patent Application Laid-Open No. 2009-14445. In this distance measuring technique, each image of a subject (target object) is obtained by a pair of cameras that are separated by a predetermined interval (base line length), and corresponding point search is performed on each image in pixel units. The parallax between the pair of cameras in the separation direction is obtained from each of these images in pixel units, and the distance to the subject is obtained based on the so-called triangulation principle based on the obtained parallax. More specifically, in FIG. 21, at least two focal lengths (f), the number of pixels on the imaging surface (imaging unit 33 (for example, CCD type image sensor)), and the size (μ) of one pixel are equal to each other. The first and second cameras are spaced apart from each other by a predetermined base line length (L) and the optical axes AX-1 and AX-2 are arranged in parallel. With these first and second cameras, an object (subject) Ob is If the parallax (number of shifted pixels) on the imaging surfaces IP-1 and IP-2 is d (= d1 + d2) when the image is taken, the distance (Z) to the object (subject) Ob is a diagonal line. Since the triangles shown are similar,
Z: f = L: μ × d
There is a relationship
Z = (L × f) / (μ × d)
Can be obtained.
Z:f=L:μ×d
の関係があり、
Z=(L×f)/(μ×d)
で求めることができる。 In the present embodiment, next, in step S45, the positional deviation amount obtained by the positional deviation
Z: f = L: μ × d
There is a relationship
Z = (L × f) / (μ × d)
Can be obtained.
そして、図20に戻って、処理S46では、このように距離演算部16によって求められた距離が制御処理部1bによって出力部5へ出力され、処理が終了する。これによって出力部5には、被写体までの距離が表示される。被写体までの距離は、中心(2行2列)に位置する第5画像(参照画像)に対し、周辺に位置する8個の画像それぞれについて、8個の距離が表示されても良く、これらの平均値が表示されても良い。
Then, referring back to FIG. 20, in the process S46, the distance obtained by the distance calculation unit 16 in this way is output to the output unit 5 by the control processing unit 1b, and the process ends. As a result, the output unit 5 displays the distance to the subject. As for the distance to the subject, eight distances may be displayed for each of the eight images located in the periphery with respect to the fifth image (reference image) located in the center (2 rows and 2 columns). An average value may be displayed.
また、上述では、原画像データISは、9眼のアレイカメラ3bbによって取得された9個のデータIS-11~IS-33であるが、少なくとも1個のカメラ3baで異なる9つの時刻で撮像された9個のデータであっても良い。このような画像処理装置Dは、異なる時刻で撮像された複数の原画像データを得ることができる。
In the above description, the original image data IS is nine pieces of data IS-11 to IS-33 acquired by the nine-lens array camera 3bb. However, the original image data IS is picked up at nine different times by at least one camera 3ba. Nine data may be sufficient. Such an image processing apparatus D can obtain a plurality of original image data captured at different times.
このような第2実施形態における画像処理装置Dbは、第1実施形態における画像処理部Daと同様に、対応点探索をより高精度に実施できる。そして、第2実施形態における画像処理装置Dbは、距離演算部16を備えるので、対応点探索処理部13で探索された対応点間の視差(位置ズレ量)を求めることができ、より高精度に実行された対応点探索の結果を距離の演算に効果的に反映できる。
The image processing device Db according to the second embodiment can perform the corresponding point search with higher accuracy in the same manner as the image processing unit Da according to the first embodiment. Since the image processing device Db according to the second embodiment includes the distance calculation unit 16, the parallax (position shift amount) between the corresponding points searched by the corresponding point search processing unit 13 can be obtained with higher accuracy. The result of the corresponding point search executed in (5) can be effectively reflected in the distance calculation.
なお、上述の実施形態において、原画像データから各画素ごとに各色成分データを持つ画像データを生成するデモザイク処理は、次の処理であっても良い。
In the above-described embodiment, the demosaic process for generating image data having each color component data for each pixel from the original image data may be the following process.
図22は、実施形態の画像処理装置の変形形態におけるデモザイク処理部を説明するための図である。図22Aは、デモザイク処理部17の構成を示すブロック図であり、図22Bは、デモザイク処理部17の動作を示すフローチャートである。図23は、前記デモザイク処理部で実行されるデモザイク処理における輝度成分生成処理を説明するための図である。図23Aは、原画像データISの画像(原画像)PISの一例を示し、図23Bは、図23Aに示す原画像データISを第1態様の第2画像フィルタFL2aでフィルタリングした素輝度成分画像データBISp-1の画像(素輝度成分画像)PBISp-1および前記第1態様の第2画像フィルタFL2aを示し、図23Cは、図23Aに示す原画像データISを第2態様の第2画像フィルタFL2bでフィルタリングした素輝度成分画像データBISp-2の画像(素輝度成分画像)PBISp-2および前記第2態様の第2画像フィルタFL2bを示し、そして、図23Dは、図23Bに示す素輝度成分画像と図23Cに示す素輝度成分画像とを1:4の割合で合成することによって求めた輝度成分画像データBISの画像(輝度成分画像)PBISを示す。図24は、前記デモザイク処理における色度成分生成処理を説明するための図である。図24Aは、単位配列がベイヤー配列である場合の原画像データISを示し、図24Bは、図11Aに示す原画像データISに対応する輝度成分画像データBISを示し、図24Cは、図24Aに示す原画像データISに対応するG、R、Bの各色の色度成分画像データCIS(CIS-G、CIS-R、CIS-B)(補間前の色別色度成分画像データ)を示す。図25は、前記デモザイク処理における色度成分補間処理を説明するための図である。図25Aは、注目画素Xとその周辺画素A~Dの位置関係を示し、図25Bは、重みと補間演算とを示す。図26は、前記デモザイク処理における画像生成処理を説明するための図である。図26Aは、G成分の場合を示し、図26Bは、R成分の場合を示し、図26Cは、B成分の場合を示す。図27は、従来において、相対的に低解像度な複数の画像から相対的に高解像度な1つの画像を生成する場合に生じるギザギザな線を説明するための図である。
FIG. 22 is a diagram for explaining a demosaic processing unit in a modification of the image processing apparatus according to the embodiment. FIG. 22A is a block diagram illustrating a configuration of the demosaic processing unit 17, and FIG. 22B is a flowchart illustrating an operation of the demosaic processing unit 17. FIG. 23 is a diagram for explaining a luminance component generation process in the demosaic process executed by the demosaic processor. FIG. 23A shows an example of an image (original image) PIS of the original image data IS, and FIG. 23B shows elementary luminance component image data obtained by filtering the original image data IS shown in FIG. 23A with the second image filter FL2a of the first mode. FIG. 23C shows a BISp-1 image (primary luminance component image) PBISp-1 and the second image filter FL2a of the first mode, and FIG. 23C shows the original image data IS shown in FIG. 23A as the second image filter FL2b of the second mode. FIG. 23D shows an image (element luminance component image) PBISp-2 of the elementary luminance component image data BISp-2 filtered in step S2 and the second image filter FL2b of the second aspect, and FIG. 23D shows the elementary luminance component image shown in FIG. 23B. And the luminance component image data BIS obtained by combining the elementary luminance component image shown in FIG. It shows the image) PBIS. FIG. 24 is a diagram for explaining chromaticity component generation processing in the demosaic processing. FIG. 24A shows original image data IS when the unit array is a Bayer array, FIG. 24B shows luminance component image data BIS corresponding to the original image data IS shown in FIG. 11A, and FIG. 24C shows FIG. The chromaticity component image data CIS (CIS-G, CIS-R, CIS-B) of each color of G, R, and B corresponding to the original image data IS shown (color-specific chromaticity component image data before interpolation) is shown. FIG. 25 is a diagram for explaining chromaticity component interpolation processing in the demosaic processing. FIG. 25A shows the positional relationship between the target pixel X and its peripheral pixels A to D, and FIG. 25B shows the weights and the interpolation calculation. FIG. 26 is a diagram for explaining an image generation process in the demosaic process. FIG. 26A shows the case of the G component, FIG. 26B shows the case of the R component, and FIG. 26C shows the case of the B component. FIG. 27 is a diagram for explaining jagged lines that occur when a relatively high-resolution image is generated from a plurality of relatively low-resolution images.
上述の実施形態における画像処理装置Dの制御処理部1は、機能的に、デモザイク処理部17をさらに備えて構成される。このデモザイク処理部17は、原画像データをデモザイクすることによって各色成分データを各画素ごとに持つ画像データの少なくとも1色を生成するものである。このようなデモザイク処理部17は、本実施形態では、図22Aに示すように、輝度成分生成処理部171と、色度成分生成処理部172と、色度成分補間処理部173と、画像生成処理部174とを備える。
The control processing unit 1 of the image processing apparatus D in the above-described embodiment is functionally further provided with a demosaic processing unit 17. The demosaic processing unit 17 generates at least one color of image data having each color component data for each pixel by demosaicing the original image data. In this embodiment, such a demosaic processing unit 17 includes a luminance component generation processing unit 171, a chromaticity component generation processing unit 172, a chromaticity component interpolation processing unit 173, and an image generation process as illustrated in FIG. 22A. Part 174.
輝度成分生成処理部171は、原画像データを所定の第2画像フィルタでフィルタリングすることによって輝度成分画像データを生成するものである。前記所定の第2画像フィルタは、上述の第1画像フィルタFL1と同様に、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、原画像データにおける単位配列321の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備えて構成される。そして、前記第2画像フィルタは、1つであって良いが、本実施形態では、複数の画像フィルタが用いられる。輝度成分生成処理部171は、原画像データを前記複数の第2画像フィルタそれぞれでフィルタリングすることによって複数の素輝度成分画像データを生成し、前記複数の素輝度成分画像データを予め設定された所定の合成比で合成することで前記輝度成分生成画像データを生成する。そして、これら複数の第2画像フィルタは、サイズが互いに異なるとともに、前記フィルタリングによって高周波成分を除去する除去率が互いに異なる第2画像フィルタを含む。一例では、輝度成分生成処理部171は、原画像データを第2-1および第2-2画像フィルタそれぞれでフィルタリングすることによって第1および第2素輝度成分画像データを生成し、これら第1および第2素輝度成分画像データを予め適宜に設定された所定の合成比で合成することで前記輝度成分生成画像データを生成する。この第2-1画像フィルタは、第2-2画像フィルタのサイズよりも大きなサイズであり、前記フィルタリングによって高周波成分を除去する除去率が第2-2画像フィルタよりも小さい。このような第2画像フィルタについては、後に、詳述する。
The luminance component generation processing unit 171 generates luminance component image data by filtering original image data with a predetermined second image filter. The predetermined second image filter is similar to the first image filter FL1 described above, based on data corresponding to pixels located around pixels corresponding to target data that is data to be filtered, the target data, The filter coefficient is converted to data having the same color composition ratio as that of the unit array 321 in the original image data. And although the said 2nd image filter may be one, in this embodiment, several image filter is used. The luminance component generation processing unit 171 generates a plurality of elementary luminance component image data by filtering the original image data with each of the plurality of second image filters, and sets the plurality of elementary luminance component image data to a predetermined predetermined value. The luminance component generation image data is generated by combining with the combination ratio. The plurality of second image filters include second image filters having different sizes and different removal rates for removing high-frequency components by the filtering. In one example, the luminance component generation processing unit 171 generates the first and second elementary luminance component image data by filtering the original image data with the 2-1 and 2-2 image filters, respectively. The luminance component generation image data is generated by synthesizing the second elementary luminance component image data at a predetermined synthesis ratio set appropriately in advance. This 2-1 image filter is larger than the size of the 2-2 image filter, and the removal rate for removing high-frequency components by the filtering is smaller than that of the 2-2 image filter. Such a second image filter will be described in detail later.
色度成分生成処理部172は、前記原画像データから、輝度成分生成処理部171で生成された輝度成分画像データを、各画素ごとに、減算することによって、色度成分画像データを生成するものである。
The chromaticity component generation processing unit 172 generates chromaticity component image data by subtracting, for each pixel, the luminance component image data generated by the luminance component generation processing unit 171 from the original image data. It is.
色度成分補間処理部173は、色度成分生成処理部172で生成された色度成分画像データにおいて、前記複数色ごとに、欠落する画素の色度成分データを、前記単位配列に含まれる同色の個数、および、輝度成分生成処理部171で生成された輝度成分画像データに基づいた補間方法で補間することによって、前記複数色ごとの各色別色度成分画像データを生成するものである。好ましくは、本実施形態では、色度成分補間処理部173は、前記単位配列に含まれる同色の個数が1つである色の色度成分データを補間する場合には、バイリニア補間法を前記補間方法として用いるものである。また、好ましくは、本実施形態では、色度成分補間処理部173は、前記単位配列に含まれる同色の個数が複数である色の色度成分データを補間する場合には、輝度成分生成処理部171で生成された輝度成分画像データに基づいた補間方法を前記補間方法として用いるものである。より具体的には、輝度成分生成処理部171で生成された輝度成分画像データに基づいた補間方法は、前記色度成分データが欠落する画素を注目画素とし、前記色度成分データ上における前記注目画素の第1画素位置に対応する前記輝度成分画像データ上における第2画素位置の第1輝度成分データと、前記第2画素位置の周辺に位置する周辺画素位置の第2輝度成分データとの相関の有無に応じて選択される補間方法である。好ましくは、輝度成分生成処理部171で生成された輝度成分画像データに基づいた補間方法は、前記相関の有る場合には、第1画素位置の周辺に位置する周辺画素位置の色度成分データを前記相関の程度に応じた重みで重みを付けて平均する重み付け補間法であり、前記相関の無い場合には、バイリニア補間法である。
In the chromaticity component image data generated by the chromaticity component generation processing unit 172, the chromaticity component interpolation processing unit 173 supplies the chromaticity component data of the missing pixel for each of the plurality of colors in the same color included in the unit array. And the chromaticity component image data for each color for each of the plurality of colors are generated by interpolation using an interpolation method based on the luminance component image data generated by the luminance component generation processing unit 171. Preferably, in the present embodiment, the chromaticity component interpolation processing unit 173 uses a bilinear interpolation method when interpolating chromaticity component data of a color having the same number of colors included in the unit array. It is used as a method. Preferably, in the present embodiment, the chromaticity component interpolation processing unit 173 performs a luminance component generation processing unit when interpolating chromaticity component data of a plurality of colors of the same color included in the unit array. An interpolation method based on the luminance component image data generated in 171 is used as the interpolation method. More specifically, in the interpolation method based on the luminance component image data generated by the luminance component generation processing unit 171, the pixel lacking the chromaticity component data is set as the target pixel, and the attention on the chromaticity component data is set. Correlation between the first luminance component data at the second pixel position on the luminance component image data corresponding to the first pixel position of the pixel and the second luminance component data at the peripheral pixel position located around the second pixel position The interpolation method is selected according to whether or not there is. Preferably, in the interpolation method based on the luminance component image data generated by the luminance component generation processing unit 171, when there is the correlation, the chromaticity component data of the peripheral pixel position positioned around the first pixel position is used. This is a weighted interpolation method that weights and averages the weights according to the degree of correlation, and when there is no correlation, it is a bilinear interpolation method.
画像生成処理部174は、色度成分補間処理部173で生成された前記複数色ごとの各色別色度成分画像データそれぞれに、輝度成分生成処理部171で生成された輝度成分画像データを、各画素ごとに、それぞれ加算することによって、前記画像データを生成するものである。
The image generation processing unit 174 adds the luminance component image data generated by the luminance component generation processing unit 171 to each chromaticity component image data for each color generated by the chromaticity component interpolation processing unit 173, respectively. The image data is generated by adding each pixel.
そして、このような構成のデモザイク処理部17は、図22Bに示すように、まず、デモザイク処理部17の輝度成分生成処理部171によって原画像データISに基づいて輝度成分画像データBISが生成される(S51)。この原画像データISは、本実施形態では、一例として、9眼を持つアレイカメラ3bbで同一の被写体を撮像することによって生成された、上述した図7に示す、9個の第1ないし第9原画像データISである。次に、デモザイク処理部17の色度成分生成処理部172によって原画像データISおよび輝度成分画像データBISに基づいて色度成分画像データCISが生成され(S52)、次に、デモザイク処理部17の色度成分補間処理部173によって輝度成分画像データBISおよび色度成分画像データCISに基づいて補間処理されて色別色度成分画像データICISが生成され(S53)、そして、デモザイク処理部17の画像生成処理部174によって輝度成分画像データBISおよび色別色度成分画像データICISに基づいて画像データIDが生成される(S54)。これら処理S51~S54の各処理は、第1ないし第9原画像データIS-11~IS-33それぞれに対し、実行される。より詳しくは、これら各処理S51~S54は、次のように処理される。
In the demosaic processing unit 17 having such a configuration, as shown in FIG. 22B, first, the luminance component image data BIS is generated based on the original image data IS by the luminance component generation processing unit 171 of the demosaic processing unit 17. (S51). In the present embodiment, as an example, the original image data IS is generated by imaging the same subject with an array camera 3bb having nine eyes, and the nine first to ninth shown in FIG. 7 described above. Original image data IS. Next, chromaticity component image data CIS is generated based on the original image data IS and the luminance component image data BIS by the chromaticity component generation processing unit 172 of the demosaic processing unit 17 (S52). The chromaticity component interpolation processing unit 173 performs interpolation processing based on the luminance component image data BIS and the chromaticity component image data CIS to generate color-specific chromaticity component image data ICIS (S53), and the image of the demosaic processing unit 17 The generation processing unit 174 generates an image data ID based on the luminance component image data BIS and the color-specific chromaticity component image data ICIS (S54). Each of these processes S51 to S54 is executed for each of the first to ninth original image data IS-11 to IS-33. More specifically, each of these processes S51 to S54 is performed as follows.
輝度成分を生成する輝度成分生成処理S51では、輝度成分生成処理部171は、原画像データISを所定の第2画像フィルタFL2でフィルタリングすることによって輝度成分画像データBISを生成する。この輝度成分生成処理S51では、1個の第2画像フィルタFL2が用いられ、輝度成分生成処理部171は、原画像データISをこの1個の第2画像フィルタFL2でフィルタリングすることによって輝度成分画像データBISを生成しても良いが、本実施形態では、複数の第2画像フィルタFL2nが用いられ、輝度成分生成処理部171は、原画像データISをこれら複数の第2画像フィルタFL2nそれぞれでフィルタリングすることによって複数の素輝度成分画像データBISpを生成し、これら複数の素輝度成分画像データBISpを所定の合成比で合成することによって1個の輝度成分画像データBISを生成する。図23に示す例では、図23Bおよび図23Cに示す2個の第1および第2態様の第2-1および第2-2画像フィルタFL2a、FL2bが用いられ、輝度成分生成処理部171は、原画像データIS(図8A)を第1態様の第2-1画像フィルタFL2aでフィルタリングすることによって第1素輝度成分画像データBISp-1を生成し(図23B)、原画像データISを第2態様の第2-2画像フィルタFL2bでフィルタリングすることによって第2素輝度成分画像データBISp-2を生成し(図23C)、そして、これら第1および第2素輝度成分画像データBISp-1、BISp-2を1:4の合成比で合成することによって1個の輝度成分画像データBISを生成する(図23D)。
In luminance component generation processing S51 for generating a luminance component, the luminance component generation processing unit 171 generates luminance component image data BIS by filtering the original image data IS with a predetermined second image filter FL2. In this luminance component generation processing S51, one second image filter FL2 is used, and the luminance component generation processing unit 171 performs luminance component image filtering by filtering the original image data IS with this one second image filter FL2. Although the data BIS may be generated, in the present embodiment, a plurality of second image filters FL2n are used, and the luminance component generation processing unit 171 filters the original image data IS with each of the plurality of second image filters FL2n. As a result, a plurality of elementary luminance component image data BISp is generated, and a plurality of elementary luminance component image data BISp is synthesized at a predetermined synthesis ratio, thereby generating one luminance component image data BIS. In the example shown in FIG. 23, two 2-1 and 2-2 image filters FL2a and FL2b of the first and second modes shown in FIGS. 23B and 23C are used, and the luminance component generation processing unit 171 First original luminance component image data BISp-1 is generated by filtering the original image data IS (FIG. 8A) with the 2-1 image filter FL2a of the first aspect (FIG. 23B). The second elementary luminance component image data BISp-2 is generated by filtering with the second-second image filter FL2b of the embodiment (FIG. 23C), and these first and second elementary luminance component image data BISp-1, BISp-2 are generated. -2 is synthesized at a synthesis ratio of 1: 4 to generate one luminance component image data BIS (FIG. 23D).
これら第1および第2態様の第2-1および第2-2画像フィルタFL2a、FL2bは、それぞれ、フィルタリングの対象データに対応する画素(対象画素)の周辺に位置する画素(周辺画素)に対応するデータ(周辺データ)に基づいて、前記対象データを、原画像データISにおける単位配列(単位色フィルタ配列321)の色組成比と同じ色組成比を持つデータに変換するように、そのフィルタ係数の各値がそれぞれ調整されて設定されている。そして、輝度成分画像データBIS(素輝度成分画像データBISp)に残す高周波成分の度合い等を考慮することによって、第2画像フィルタFL2におけるフィルタ係数の各値が調整されることが好ましく、また、フィルタリングする対象画素に影響を与える、当該対象画素の周辺に位置する画素範囲等を考慮することによって、第2画像フィルタFL2の大きさ(サイズ)が調整されることが好ましい。
The 2-1 and 2-2 image filters FL2a and FL2b of the first and second modes respectively correspond to pixels (peripheral pixels) located around the pixel (target pixel) corresponding to the filtering target data. Filter coefficients so as to convert the target data into data having the same color composition ratio as that of the unit array (unit color filter array 321) in the original image data IS based on the data to be performed (peripheral data) Each value is adjusted and set. Then, it is preferable that each value of the filter coefficient in the second image filter FL2 is adjusted by taking into consideration the degree of the high frequency component remaining in the luminance component image data BIS (elementary luminance component image data BISp), and filtering. It is preferable that the size (size) of the second image filter FL2 is adjusted by considering a pixel range or the like located around the target pixel that affects the target pixel.
より具体的には、第1態様の第2-1画像フィルタFL2aは、原画像データISの単位配列がベイヤー配列である場合に用いられる11行11列のフィルタであり、前記周辺データに基づいて前記対象データを、ベイヤー配列における単位配列の色組成比(R:B:G=1:1:2)と同じ色組成比を持つデータに変換するようにそのフィルタ係数の各値がそれぞれ調整されて設定されている。そして、この第1態様の第2-1画像フィルタFL2aは、その中心位置に対し対称なフィルタ係数を持つフィルタである。すなわち、図23Bに示すように、中心の6行6列に位置するフィルタ係数は、104/128であり、5行6列、6行5列、6行7列および7行6列それぞれに位置する各フィルタ係数は、7/128であり、1行5列、1行7列、3行3列、3行4列、3行6列、3行8列、3行9列、4行3列、4行9列、5行1列、5行5列、5行7列、5行11列、7行1列、7行5列、7行7列、7行11列、8行3列、8行9列、9行3列、9行4列、9行6列、9行8列、9行9列、11行5列および11行7列それぞれに位置する各フィルタ係数は、1/128であり、4行6列、6行4列、6行8列および8行6列それぞれに位置する各フィルタ係数は、-9/128であり、4行5列、4行7列、5行4列、5行8列、7行4列、7行8列、8行5列および8行7列それぞれに位置する各フィルタ係数は、3/128であり、4行4列、4行8列、8行4列および8行8列それぞれに位置する各フィルタ係数は、-5/128であり、3行5列、3行7列、5行3列、5行9列、7行3列、7行9列、9行5列および9行7列それぞれに位置する各フィルタ係数は、2/128であり、2行4列、2行8列、4行2列、4行10列、8行2列、8行10列、10行4列および10行8列それぞれに位置する各フィルタ係数は、-1/128であり、2行6列、6行2列、6行10列および10行6列それぞれに位置する各フィルタ係数は、-2/128であり、そして、残余の各位置に位置する各フィルタ係数は、0/128である。なお、図23Bでは、行列表示の各位置にフィルタ係数の値の分子が記載され、分母は、行列表示外の右下に纏めて記載されている。図23Cも同様の記載形式で記載されている。このようなフィルタ係数を持つ第1態様の第2-1画像フィルタFL2aは、前記フィルタリングによって高周波成分を除去する除去率が比較的小さいために、輝度成分画像データBIS(素輝度成分画像データBISp)に比較的多くの高周波成分が残り、輝度成分画像データBIS(素輝度成分画像データBISp)には、比較的鮮鋭度が保持される。そして、このような第1態様の第2-1画像フィルタFL2aは、11×11というフィルタサイズを持つので、フィルタリングする対象画素に影響を与える画素範囲が比較的広く、前記対象画素から比較的遠くに位置する画素まで前記対象画素に影響を与え、輝度成分画像データBIS(素輝度成分画像データBISp)には、比較的多くの誤差が含まれる。このため、後述のギザギザな線が比較的出易い。
More specifically, the 2-1 image filter FL2a of the first aspect is an 11 × 11 filter used when the unit array of the original image data IS is a Bayer array, and is based on the peripheral data. Each value of the filter coefficient is adjusted so that the target data is converted into data having the same color composition ratio as the color composition ratio (R: B: G = 1: 1: 2) of the unit array in the Bayer array. Is set. The 2-1 image filter FL2a according to the first aspect is a filter having a filter coefficient that is symmetric with respect to its center position. That is, as shown in FIG. 23B, the filter coefficient located at the center 6 rows and 6 columns is 104/128, and is located at 5 rows 6 columns, 6 rows 5 columns, 6 rows 7 columns, and 7 rows 6 columns, respectively. Each filter coefficient is 7/128, 1 row 5 columns, 1 row 7 columns, 3 rows 3 columns, 3 rows 4 columns, 3 rows 6 columns, 3 rows 8 columns, 3 rows 9 columns, 4 rows 3 Column, 4 rows, 9 columns, 5 rows, 1 column, 5 rows, 5 columns, 5 rows, 7 columns, 5 rows, 11 columns, 7 rows, 1 column, 7 rows, 5 columns, 7 rows, 7 columns, 7 rows, 11 columns, 8 rows, 3 Each filter coefficient located in the column, 8 rows 9 columns, 9 rows 3 columns, 9 rows 4 columns, 9 rows 6 columns, 9 rows 8 columns, 9 rows 9 columns, 11 rows 5 columns and 11 rows 7 columns, Each filter coefficient located at 1/128, 4 rows 6 columns, 6 rows 4 columns, 6 rows 8 columns and 8 rows 6 columns is -9/128, 4 rows 5 columns, 4 rows 7 columns 5 rows 4 columns 5 rows 8 columns 7 rows 4 columns 7 rows Each filter coefficient located in each column, 8 rows 5 columns and 8 rows 7 columns is 3/128, and each filter coefficient located in 4 rows 4 columns, 4 rows 8 columns, 8 rows 4 columns and 8 rows 8 columns respectively. The filter coefficient is −5/128, 3 rows and 5 columns, 3 rows and 7 columns, 5 rows and 3 columns, 5 rows and 9 columns, 7 rows and 3 columns, 7 rows and 9 columns, 9 rows and 5 columns, and 9 rows and 7 columns. Each filter coefficient located in each is 2/128, 2 rows 4 columns, 2 rows 8 columns, 4 rows 2 columns, 4 rows 10 columns, 8 rows 2 columns, 8 rows 10 columns, 10 rows 4 columns and Each filter coefficient located in each of 10 rows and 8 columns is -1/128, and each filter coefficient located in each of 2 rows and 6 columns, 6 rows and 2 columns, 6 rows and 10 columns, and 10 rows and 6 columns is -2 Each filter coefficient located at each remaining position is 0/128. In FIG. 23B, the numerator of the value of the filter coefficient is described at each position of the matrix display, and the denominator is described collectively at the lower right outside the matrix display. FIG. 23C is also described in a similar description format. Since the 2-1 image filter FL2a of the first aspect having such a filter coefficient has a relatively small removal rate for removing high frequency components by the filtering, luminance component image data BIS (elementary luminance component image data BISp) In the luminance component image data BIS (elementary luminance component image data BISp), a relatively high sharpness is maintained. Since the 2-1 image filter FL2a according to the first aspect has a filter size of 11 × 11, the pixel range that affects the target pixel to be filtered is relatively wide and relatively far from the target pixel. The target pixel is affected up to the pixel located at, and the luminance component image data BIS (elementary luminance component image data BISp) includes a relatively large amount of error. For this reason, the jagged line mentioned later is comparatively easy to come out.
第2態様の第2-2画像フィルタFL2bは、原画像データISの単位配列がベイヤー配列である場合に用いられる3行3列のフィルタであり、前記周辺データに基づいて前記対象データを、ベイヤー配列における単位配列の色組成比(R:B:G=1:1:2)と同じ色組成比を持つデータに変換するようにそのフィルタ係数の各値がそれぞれ調整されて設定されている。そして、この第2態様の第2-2画像フィルタFL2bは、その中心位置に対し対称なフィルタ係数を持つフィルタであり、図23Cに示すように、上述の第1態様の画像フィルタFL1aと同一である。このような第2態様の第2-2画像フィルタFL2b(=画像フィルタFL1a)は、デモザイク処理の観点から、第1態様の第2-1画像フィルタFL2aと比較して、逆に、前記フィルタリングによって高周波成分を除去する除去率が比較的大きいために、輝度成分画像データBIS(素輝度成分画像データBISp)に高周波成分が比較的少なくなり、そして、フィルタリングする対象画素に影響を与える画素範囲が比較的狭く、輝度成分画像データBIS(素輝度成分画像データBISp)に含まれる誤差は、比較的少なく、ギザギザな線が比較的出難い。このため、上述したように、1個の第2画像フィルタFL2を用いて輝度成分画像データBISを生成する場合には、このような第2態様の第2-2画像フィルタFL2bが利用される。
The 2-2 image filter FL2b of the second aspect is a 3 × 3 filter used when the unit array of the original image data IS is a Bayer array, and the target data is converted to the Bayer based on the peripheral data. Each value of the filter coefficient is adjusted and set so as to be converted into data having the same color composition ratio as the color composition ratio (R: B: G = 1: 1: 2) of the unit array in the array. The 2-2 image filter FL2b in the second mode is a filter having a filter coefficient that is symmetric with respect to the center position, and is the same as the image filter FL1a in the first mode as shown in FIG. 23C. is there. From the viewpoint of demosaic processing, the 2-2 image filter FL2b (= image filter FL1a) of the second mode is, conversely, compared with the 2-1 image filter FL2a of the first mode, by the above filtering. Since the removal rate for removing the high-frequency components is relatively large, the luminance component image data BIS (elementary luminance component image data BISp) has a relatively small amount of high-frequency components, and the pixel ranges that affect the target pixel to be compared are compared. The brightness component image data BIS (elemental brightness component image data BISp) has a relatively small error, and a jagged line is relatively difficult to appear. Therefore, as described above, when the luminance component image data BIS is generated using one second image filter FL2, the 2-2 image filter FL2b of such a second mode is used.
また、この第2画像フィルタFL2の各フィルタ係数も、画像フィルタFL全域に単位配列を重ねた場合に、前記単位配列の同じ配列位置におけるフィルタ係数の和が全ての配列位置で同じ値になるように、設定されてよい。
Further, the filter coefficients of the second image filter FL2 are such that when the unit arrays are overlapped over the entire area of the image filter FL, the sum of the filter coefficients at the same array position of the unit arrays becomes the same value at all array positions. May be set.
フィルタリングは、上述と同様に、原画像データISにおける最初の1行1列に位置する画素に対応するデータから最終の500行500列に位置する画素に対応するデータまで順次に実行される。
Filtering is executed sequentially from data corresponding to the pixels located in the first row and first column in the original image data IS to data corresponding to the pixels located in the last 500 rows and 500 columns, as described above.
第2画像フィルタFL2は、原画像データISにおける単位配列の色組成比と同じ色組成比のフィルタ係数を備えるから、原画像データISに第2画像フィルタFL2を作用させた場合において、原画像データISのR画素に対応するフィルタ係数の総計と、原画像データISのB画素に対応するフィルタ係数の総計と、原画像データISのG画素に対応するフィルタ係数の総計とは、原画像データISにおける単位配列の色組成比と同じになる。
Since the second image filter FL2 has a filter coefficient having the same color composition ratio as that of the unit array in the original image data IS, the original image data is obtained when the second image filter FL2 is applied to the original image data IS. The total of the filter coefficients corresponding to the R pixels of IS, the total of the filter coefficients corresponding to the B pixels of the original image data IS, and the total of the filter coefficients corresponding to the G pixels of the original image data IS are the original image data IS. The color composition ratio of the unit array in FIG.
このように原画像データISの単位配列におけるいずれの画素のデータも第2画像フィルタFL2によって同じ色組成比でフィルタリングされる。したがって、原画像データISにおける全ての画素のデータが第2画像フィルタFL2によって同じ色組成比でフィルタリングされることになる。この結果、輝度成分画像データBIS(素輝度成分画像データBISp)における全ての画素のデータは、(G/2+R/4+B/4)で合成された同じ色成分を持つデータとして扱うことが可能となる。
Thus, the data of any pixel in the unit array of the original image data IS is filtered with the same color composition ratio by the second image filter FL2. Therefore, the data of all the pixels in the original image data IS are filtered with the same color composition ratio by the second image filter FL2. As a result, the data of all pixels in the luminance component image data BIS (elementary luminance component image data BISp) can be handled as data having the same color component synthesized by (G / 2 + R / 4 + B / 4). .
そして、複数(本実施形態では2個)の素輝度成分画像データBISpの合成では、輝度成分画像データBISの各データ(各画素値)は、当該データの位置(当該画素の位置)に対応する位置の各素輝度成分画像データBISpの各データ値を所定の合成比で合成することによって求められる。例えば、輝度成分画像データBISにおけるm行n列のデータ(各画素値)DVBIS,mnは、各素輝度成分画像データBISpにおけるm行n列の各データ値を前記所定の合成比で合成することによって求められる。前記所定の合成比は、予め、複数のサンプルの原画像に対し合成比を変えながら実際に合成し、合成後の画像を見比べることによって適宜な比に設定され、例えば、本実施形態では、1:4である。したがって、第1態様の第2-1画像フィルタFL2aを原画像ISに作用することによって得られた素輝度成分画像データBISp-1におけるm行n列のデータ値をDVBISp-1,mnとし、第2態様の第2-2画像フィルタFL2bを原画像ISに作用することによって得られた素輝度成分画像データBISp-2におけるm行n列のデータ値をDVBISp-2,mnとした場合、輝度成分画像データBISにおけるm行n列のデータは、(4×DVBISp-1,mn+1×DVBISp-2,mn)/(1+4)となる。第1態様の第2-1画像フィルタFL2aは、上述したように、鮮鋭度が比較的保持される一方ギザギザな線が比較的出易い素輝度成分画像データBISp-1を生成するため、この第1態様の第2-1画像フィルタFL2aによる素輝度成分画像データBISp-1と、この第2-1画像フィルタFL2aと逆の特性を持つ第2態様の第2-2画像フィルタFL2bによる素輝度成分画像データBISp-2とを組み合わせ、これらを所定の合成比で合成して輝度成分画像データBISを生成することで、鮮鋭度を保持しつつギザギザな線の少ない輝度成分画像データBISが生成できる。
In the synthesis of a plurality (two in this embodiment) of the elementary luminance component image data BISp, each data (each pixel value) of the luminance component image data BIS corresponds to the position of the data (the position of the pixel). It is obtained by synthesizing each data value of each elementary luminance component image data BISp at a position with a predetermined synthesis ratio. For example, m rows and n columns of data (each pixel value) DV BIS, mn in the luminance component image data BIS synthesize each data value of m rows and n columns in each elementary luminance component image data BISp with the predetermined synthesis ratio. Is required. The predetermined synthesis ratio is set in advance to an appropriate ratio by actually synthesizing the original images of a plurality of samples while changing the synthesis ratio, and comparing the images after synthesis. : 4. Therefore, the data value of m rows and n columns in the elementary luminance component image data BISp-1 obtained by applying the 2-1 image filter FL2a of the first mode to the original image IS is DV BISp-1, mn , When the data value of m rows and n columns in the elementary luminance component image data BISp-2 obtained by applying the 2-2 image filter FL2b of the second mode to the original image IS is DV BISp-2, mn , The data of m rows and n columns in the luminance component image data BIS is (4 × DV BISp−1, mn + 1 × DV BISp−2, mn ) / (1 + 4). As described above, the 2-1 image filter FL2a according to the first aspect generates elementary luminance component image data BISp-1 in which the sharpness is relatively maintained while the jagged line is relatively easily generated. Elemental luminance component image data BISp-1 by the 2-1 image filter FL2a of one aspect and elementary luminance component by the 2-2 image filter FL2b of the second aspect having characteristics opposite to those of the 2-1 image filter FL2a By combining the image data BISp-2 and combining them at a predetermined combining ratio to generate the luminance component image data BIS, the luminance component image data BIS with few jagged lines can be generated while maintaining sharpness.
なお、前記第2画像フィルタFL2は、上述の第1および第2態様の第2-1および第2-2画像フィルタFL2a、FL2bに限定されるものではなく、種々の態様を採用でき、例えば、ベイヤー配列を単位色フィルタ配列とした色フィルタ部32aによって得られる原画像データISに対し、図5Bないし図5Dに示す上述した第3ないし第5態様の画像フィルタFL1b~FLd等を利用できる。
The second image filter FL2 is not limited to the 2-1 and 2-2 image filters FL2a and FL2b of the first and second modes described above, and various modes can be adopted. For the original image data IS obtained by the color filter unit 32a having the Bayer array as the unit color filter array, the above-described third to fifth image filters FL1b to FLd shown in FIGS. 5B to 5D can be used.
図22Bに戻って、色度成分を生成する色度成分生成処理S52では、色度成分生成処理部172は、図24に示すように、原画像データIS(図24A)から、上述のように輝度成分生成処理部171で生成された輝度成分画像データBIS(図24B)を、各画素ごとに、減算することによって、色度成分画像データCIS(図24C)を生成する。すなわち、色度成分画像データCISにおけるm行n列のデータ(各画素値)DVCIS,mnは、原画像データISにおけるm行n列のデータ値DVIS,mnから、輝度成分画像データBISにおけるm行n列のデータ値DVBIS,mnを減算することによって求められる(DVCIS,mn=DVIS,mn-DVBIS,mn)。なお、図24Cでは、色度成分画像データCISが色別に図示されており、図24Cの上段には、G色度成分のG色度成分画像データCIS-Gが示され、図24Cの中段には、R色度成分のR色度成分画像データCIS-Rが示され、そして、図24Cの下段には、B色度成分のB色度成分画像データCIS-Bが示されている。また、色別に図示した図24Cにおいて、データ値(画素値)が欠落している画素には、そのデータ値として「0」が設定されている。
Returning to FIG. 22B, in the chromaticity component generation processing S52 for generating chromaticity components, the chromaticity component generation processing unit 172, as shown in FIG. 24, from the original image data IS (FIG. 24A) as described above. The luminance component image data CIS (FIG. 24C) is generated by subtracting the luminance component image data BIS (FIG. 24B) generated by the luminance component generation processing unit 171 for each pixel. That is, m rows and n columns of data (pixel values) DV CIS, mn in the chromaticity component image data CIS are obtained from the m rows and n columns of data values DV IS, mn in the original image data IS in the luminance component image data BIS. It is obtained by subtracting the data value DV BIS, mn of m rows and n columns (DV CIS, mn = DV IS, mn -DV BIS, mn ). In FIG. 24C, the chromaticity component image data CIS is shown for each color. In the upper part of FIG. 24C, G chromaticity component image data CIS-G of the G chromaticity component is shown, and in the middle part of FIG. 24C. FIG. 24C shows R chromaticity component image data CIS-R of the R chromaticity component, and B chromaticity component image data CIS-B of the B chromaticity component is shown in the lower part of FIG. 24C. Further, in FIG. 24C illustrated for each color, “0” is set as the data value of the pixel in which the data value (pixel value) is missing.
図22Bに戻って、各画素で欠落している色度成分を補間する色度成分補間処理S53では、色度成分補間処理部173は、色度成分生成処理部172で生成された色度成分画像データCISにおいて、前記複数色ごとに、欠落する画素の色度成分データを、前記単位配列に含まれる同色の個数および輝度成分生成処理部171で生成された輝度成分画像データBISに基づいた補間方法で補間することによって、前記複数色ごとの各色別色度成分画像データICISを生成する。
Returning to FIG. 22B, in the chromaticity component interpolation processing S53 for interpolating the chromaticity component missing in each pixel, the chromaticity component interpolation processing unit 173 generates the chromaticity component generated by the chromaticity component generation processing unit 172. In the image data CIS, for each of the plurality of colors, the chromaticity component data of the missing pixel is interpolated based on the number of the same color included in the unit array and the luminance component image data BIS generated by the luminance component generation processing unit 171. By interpolating with the method, the chromaticity component image data ICIS for each color for each of the plurality of colors is generated.
本実施形態では、色度成分補間処理部173は、前記単位配列に含まれる同色の個数が1つである色の色度成分データを補間する場合には、バイリニア補間法を前記補間方法として用い、色度成分補間処理部173は、前記単位配列に含まれる同色の個数が複数である色の色度成分データを補間する場合には、輝度成分生成処理部171で生成された輝度成分画像データに基づいた補間方法を前記補間方法として用いる。
In this embodiment, the chromaticity component interpolation processing unit 173 uses a bilinear interpolation method as the interpolation method when interpolating chromaticity component data of a color having the same number of colors included in the unit array. When the chromaticity component interpolation processing unit 173 interpolates chromaticity component data of a color having a plurality of the same colors included in the unit array, the luminance component image data generated by the luminance component generation processing unit 171 is used. Is used as the interpolation method.
したがって、ベイヤー配列の単位色フィルタ配列321aを持つ色フィルタ部32aで生成された原画像データISの場合、前記単位配列には、R(赤色)の輝度データは、1つであり、R色度成分のR色度成分画像データCIS-Rは、図24Cに示すように、4画素に1画素の割合でしかデータ値(画素値)が無い。このため、R色度成分のデータが欠落している画素のデータ値がバイリニア補間されてもギザギザな線は、生じ難い。そして、B(緑色)の濃度データも、上述のRの濃度データの場合と同様である。よって、本実施形態では、色度成分補間処理部173は、処理S52で色度成分生成処理部172によって生成された色度成分画像データCISのR色度成分画像データCIS-Rにおける、R色度成分のデータが欠落している画素のデータ値をバイリニア補間法で補間することで、R(赤色)のR色別色度成分画像データICIS-Rを生成し、同様に、処理S52で色度成分生成処理部172によって生成された色度成分画像データCISのB色度成分画像データCIS-Bにおける、B色度成分のデータが欠落している画素のデータ値をバイリニア補間法で補間することで、B(青色)のB色別色度成分画像データICIS-Bを生成する。このバイリニア補間は、1行1列、1行3列、3行1列および3行3列の各位置にフィルタ係数1/4を持ち、1行2列、2行1列、2行3列および3行2列の各位置にフィルタ係数2/4を持ち、そして、2行2列の位置にフィルタ係数4/4を持つ画像フィルタ(RBバイリニア用画像フィルタ)を用いてフィルタリングすることで実行できる。したがって、より具体的には、色度成分補間処理部173は、前記R色度成分画像データCIS-Rを前記RBバイリニア用画像フィルタでフィルタリングすることによって前記R色別色度成分画像データICIS-Rを生成し、同様に、前記B色度成分画像データCIS-Bを前記RBバイリニア用画像フィルタでフィルタリングすることによって前記B色別色度成分画像データICIS-Bを生成する。
Therefore, in the case of the original image data IS generated by the color filter unit 32a having the unit color filter array 321a of the Bayer array, the unit array has one R (red) luminance data, and the R chromaticity The component R chromaticity component image data CIS-R has data values (pixel values) only at a ratio of one pixel to four pixels, as shown in FIG. 24C. For this reason, even if the data value of the pixel in which the R chromaticity component data is missing is subjected to bilinear interpolation, a jagged line is unlikely to occur. The density data for B (green) is the same as the density data for R described above. Therefore, in this embodiment, the chromaticity component interpolation processing unit 173 performs the R color in the R chromaticity component image data CIS-R of the chromaticity component image data CIS generated by the chromaticity component generation processing unit 172 in step S52. By interpolating the data value of the pixel in which the chromaticity component data is missing by the bilinear interpolation method, the chromaticity component image data ICIS-R for each R color of R (red) is generated. The data value of the pixel in which the B chromaticity component data is missing in the B chromaticity component image data CIS-B of the chromaticity component image data CIS generated by the chromaticity component generation processing unit 172 is interpolated by the bilinear interpolation method. Thus, chromaticity component image data ICIS-B for each B color of B (blue) is generated. This bilinear interpolation has a filter coefficient 1/4 at each position of 1 row 1 column, 1 row 3 column, 3 rows 1 column and 3 rows 3 columns, 1 row 2 columns, 2 rows 1 column, 2 rows 3 columns. And filtering by using an image filter (RB bilinear image filter) having a filter coefficient 2/4 at each position of 3 rows and 2 columns and a filter coefficient 4/4 at a position of 2 rows and 2 columns. it can. Therefore, more specifically, the chromaticity component interpolation processing unit 173 filters the R chromaticity component image data CIS-R with the RB bilinear image filter, thereby performing the R color-specific chromaticity component image data ICIS-. R is generated, and similarly, the B chromaticity component image data ICIS-B is generated by filtering the B chromaticity component image data CIS-B with the RB bilinear image filter.
一方、ベイヤー配列の単位色フィルタ配列321aを持つ色フィルタ部32aで生成された原画像データISの場合、前記単位配列には、G(緑色)の濃度データは、2つであり、G色度成分のG色度成分画像データCIS-Gは、G色度成分のデータが欠落している画素のデータ値を仮にバイリニア補間すると、ギザギザな線は、生じてしまう。このため、本実施形態では、補間方法は、G色度成分のデータが欠落する画素を注目画素とし、G色度成分画像データCIS-G上における前記注目画素の第1画素位置に対応する輝度成分画像データBIS上における第2画素位置の第1輝度成分データと、前記第2画素位置の周辺に位置する周辺画素位置の第2輝度成分データとの相関の有無に応じて、選択される。より具体的には、前記相関の有る場合には、前記第1画素位置の周辺に位置する周辺画素位置のG色度成分データを前記相関の程度に応じた重みで重みを付けて平均する重み付け補間法が前記補間方法として用いられ、前記相関の無い場合には、バイリニア補間法が前記補間方法として用いられる。例えば、図25Aに示すように、G色成分のデータが欠落する画素を注目画素Xとし、この注目画素Xに対し列方向にそれぞれ隣接する画素を周辺画素A、Dとし、この注目画素Xに対し行方向にそれぞれ隣接する画素を周辺画素B、Cとし、輝度成分画像データBISにおける注目画素Xの輝度成分データをBDxとし、輝度成分画像データBISにおける各周辺画素A、B、C、Dの輝度成分データをBDa、BDb、BDc、BDdとし、G色度成分画像データCIS-Gにおける各周辺画素A、B、C、DのG色度成分データをCDGa、CDGb、CDGc、CDGdとし、重みを求める重み関数W(d)を図25B示すように注目画素Xの輝度成分データBDxと周辺画素K(K=A、B、C、D)の輝度成分データBDkとの差d(輝度差d=|BDx-BDk|)に応じて線形に変化し、そして、前記輝度差dが大きくなるに従って重みW(d)が小さくなる場合、前記相関Wt(X;A、B、C、D)は、次式Iによって求められる。なお、この重み関数W(d)は、d=0でW(0)=1.0であり、W(Th)でW(Th)=0である。前記Thは、重みを与える範囲を規定する値であり、例えば10等である。
Wt(X;A、B、C、D)=Wt(|BDx-BDa|)+Wt(|BDx-BDb|)+Wt(|BDx-BDc|)+Wt(|BDx-BDd|) (式I) On the other hand, in the case of the original image data IS generated by thecolor filter unit 32a having the unit color filter array 321a of the Bayer array, the unit array includes two G (green) density data, and the G chromaticity In the component G chromaticity component image data CIS-G, if a data value of a pixel in which the G chromaticity component data is missing is bilinearly interpolated, a jagged line is generated. For this reason, in the present embodiment, the interpolation method uses a pixel in which data of the G chromaticity component is missing as a target pixel, and the luminance corresponding to the first pixel position of the target pixel on the G chromaticity component image data CIS-G. The selection is made according to whether or not there is a correlation between the first luminance component data at the second pixel position on the component image data BIS and the second luminance component data at the peripheral pixel position located around the second pixel position. More specifically, when there is a correlation, weighting is performed by weighting and averaging the G chromaticity component data of peripheral pixel positions located around the first pixel position with a weight according to the degree of correlation. An interpolation method is used as the interpolation method, and when there is no correlation, a bilinear interpolation method is used as the interpolation method. For example, as shown in FIG. 25A, a pixel lacking G color component data is a pixel of interest X, and pixels adjacent to the pixel of interest X in the column direction are peripheral pixels A and D, respectively. The adjacent pixels in the row direction are the peripheral pixels B and C, the luminance component data of the pixel of interest X in the luminance component image data BIS is BDx, and the peripheral pixels A, B, C, and D in the luminance component image data BIS are Luminance component data is BDa, BDb, BDc, BDd, G chromaticity component data of each peripheral pixel A, B, C, D in G chromaticity component image data CIS-G is CDGa, CDGb, CDGc, CDGd, weight As shown in FIG. 25B, the weighting function W (d) for obtaining the luminance component data BDx of the target pixel X and the luminance component data BDk of the surrounding pixels K (K = A, B, C, D) d (luminance difference d = | BDx−BDk |) linearly changes, and when the weight W (d) decreases as the luminance difference d increases, the correlation Wt (X; A, B, C, D) is obtained by the following formula I. The weight function W (d) is d = 0 and W (0) = 1.0, and W (Th) is W (Th) = 0. The Th is a value that defines a range in which a weight is given, and is 10 or the like, for example.
Wt (X; A, B, C, D) = Wt (| BDx−BDa |) + Wt (| BDx−BDb |) + Wt (| BDx−BDc |) + Wt (| BDx−BDd |) (Formula I)
Wt(X;A、B、C、D)=Wt(|BDx-BDa|)+Wt(|BDx-BDb|)+Wt(|BDx-BDc|)+Wt(|BDx-BDd|) (式I) On the other hand, in the case of the original image data IS generated by the
Wt (X; A, B, C, D) = Wt (| BDx−BDa |) + Wt (| BDx−BDb |) + Wt (| BDx−BDc |) + Wt (| BDx−BDd |) (Formula I)
このような式Iによって求められた前記相関Wt(X;A、B、C、D)が正である場合(Wt(X;A、B、C、D)>0)には、前記相関が有ると判断され、注目画素X(欠落画素)のデータ値CDGxは、次式IIによって求められる。
CDGx=(Wtxa×CDGa+Wtxb×CDGb+Wtxc×CDGc+Wtxd×CDGd)/(Wtxa+Wtxb+Wtxc+Wtxd) (式II)
なお、式IIにおいて、Wt(|BDx-BDa|)=Wtxa、Wt(|BDx-BDb|)=Wtxb、Wt(|BDx-BDc|)=Wtxc、Wt(|BDx-BDd|)=Wtxdとした。 When the correlation Wt (X; A, B, C, D) obtained by the formula I is positive (Wt (X; A, B, C, D)> 0), the correlation is The data value CDGx of the target pixel X (missing pixel) is determined by the following formula II.
CDGx = (Wtxa × CDGa + Wtxb × CDGb + Wtxc × CDGc + Wtxd × CDGd) / (Wtxa + Wtxb + Wtxc + Wtxd) (Formula II)
In Formula II, Wt (| BDx−BDa |) = Wtxa, Wt (| BDx−BDb |) = Wtxb, Wt (| BDx−BDc |) = Wtxc, Wt (| BDx−BDd |) = Wtxd did.
CDGx=(Wtxa×CDGa+Wtxb×CDGb+Wtxc×CDGc+Wtxd×CDGd)/(Wtxa+Wtxb+Wtxc+Wtxd) (式II)
なお、式IIにおいて、Wt(|BDx-BDa|)=Wtxa、Wt(|BDx-BDb|)=Wtxb、Wt(|BDx-BDc|)=Wtxc、Wt(|BDx-BDd|)=Wtxdとした。 When the correlation Wt (X; A, B, C, D) obtained by the formula I is positive (Wt (X; A, B, C, D)> 0), the correlation is The data value CDGx of the target pixel X (missing pixel) is determined by the following formula II.
CDGx = (Wtxa × CDGa + Wtxb × CDGb + Wtxc × CDGc + Wtxd × CDGd) / (Wtxa + Wtxb + Wtxc + Wtxd) (Formula II)
In Formula II, Wt (| BDx−BDa |) = Wtxa, Wt (| BDx−BDb |) = Wtxb, Wt (| BDx−BDc |) = Wtxc, Wt (| BDx−BDd |) = Wtxd did.
前記式Iによって求められた前記相関Wt(X;A、B、C、D)が0である場合(Wt(X;A、B、C、D)=0)には、前記相関が無いと判断され、注目画素X(欠落画素)のデータ値CDGxは、周辺画素A、B、C、DのG色度成分データCDGa、CDGb、CDGc、CDGdを均等な重み、例えば、1/4で補間する。すなわち、バイリニア補間法が用いられ、注目画素X(欠落画素)のデータ値CDGxは、次式IIIによって求められる。この前記相関が無い場合では、実質的にG成分にある高周波の折り返し歪み(モアレ)が残ることになり、この成分を用いることで超解像できる。このように実施形態における画像処理装置Dは、超解像可能にデモザイク処理を実行できる。
CDGx=(CDGa+CDGb+CDGc+CDGd)/4 (式III) When the correlation Wt (X; A, B, C, D) obtained by the formula I is 0 (Wt (X; A, B, C, D) = 0), there is no correlation. The data value CDGx of the target pixel X (missing pixel) is determined, and the G chromaticity component data CDGa, CDGb, CDGc, CDGd of the peripheral pixels A, B, C, D are interpolated with an equal weight, for example, 1/4. To do. That is, the bilinear interpolation method is used, and the data value CDGx of the target pixel X (missing pixel) is obtained by the following formula III. When there is no correlation, high-frequency aliasing distortion (moire) substantially in the G component remains, and super-resolution can be achieved by using this component. As described above, the image processing apparatus D according to the embodiment can execute the demosaic process so as to be super-resolution.
CDGx = (CDGa + CDGb + CDGc + CDGd) / 4 (formula III)
CDGx=(CDGa+CDGb+CDGc+CDGd)/4 (式III) When the correlation Wt (X; A, B, C, D) obtained by the formula I is 0 (Wt (X; A, B, C, D) = 0), there is no correlation. The data value CDGx of the target pixel X (missing pixel) is determined, and the G chromaticity component data CDGa, CDGb, CDGc, CDGd of the peripheral pixels A, B, C, D are interpolated with an equal weight, for example, 1/4. To do. That is, the bilinear interpolation method is used, and the data value CDGx of the target pixel X (missing pixel) is obtained by the following formula III. When there is no correlation, high-frequency aliasing distortion (moire) substantially in the G component remains, and super-resolution can be achieved by using this component. As described above, the image processing apparatus D according to the embodiment can execute the demosaic process so as to be super-resolution.
CDGx = (CDGa + CDGb + CDGc + CDGd) / 4 (formula III)
図22Bに戻って、画像データを生成する画像生成処理S54では、画像生成処理部174は、図26に示すように、色度成分補間処理部173で生成された複数色ごとの各色別色度成分画像データICISそれぞれに、輝度成分生成処理部171で生成された輝度成分画像データBISを、各画素ごとに、それぞれ加算することによって、画像データIDを生成する。より具体的には、G成分の画像データID-Gにおけるm行n列のデータ(各画素値)DVID-G,mnは、図26Aに示すように、輝度成分画像データBISにおけるm行n列のデータ値DVBIS,mnにG色別色度成分画像データICIS-Gにおけるm行n列のデータ値DVICIS-G,mnを加算することによって求められる(DVID-G,mn=DVBIS,mn+DVICIS-G,mn)。同様に、R成分の画像データID-Rにおけるm行n列のデータ(各画素値)DVID-R,mnは、図26Bに示すように、輝度成分画像データBISにおけるm行n列のデータ値DVBIS,mnにR色別色度成分画像データICIS-Rにおけるm行n列のデータ値DVICIS-R,mnを加算することによって求められる(DVID-R,mn=DVBIS,mn+DVICIS-R,mn)。同様に、B成分の画像データID-Bにおけるm行n列のデータ(各画素値)DVID-B,mnは、図26Cに示すように、輝度成分画像データBISにおけるm行n列のデータ値DVBIS,mnにB色別色度成分画像データICIS-Bにおけるm行n列のデータ値DVICIS-B,mnを加算することによって求められる(DVID-B,mn=DVBIS,mn+DVICIS-B,mn)。これによって各画素ごとに、G成分DVID-R,mn、R成分DVID-R,mnおよびB成分DVID-B,mnの各色成分を持つ画像データIDが生成される。
Returning to FIG. 22B, in the image generation processing S54 for generating image data, the image generation processing unit 174, as shown in FIG. 26, the chromaticity for each color for each of the plurality of colors generated by the chromaticity component interpolation processing unit 173. The image data ID is generated by adding the luminance component image data BIS generated by the luminance component generation processing unit 171 to each component image data ICIS for each pixel. More specifically, m rows and n columns of data (pixel values) DV ID-G, mn in the G component image data ID-G are represented by m rows and n in the luminance component image data BIS as shown in FIG. 26A. The data value DV ICIS-G, mn of m rows and n columns in the G color-specific chromaticity component image data ICIS-G is added to the column data value DV BIS, mn (DV ID-G, mn = DV BIS, mn + DV ICIS-G, mn ). Similarly, m-row n-column data (each pixel value) DV ID-R, mn in R component image data ID-R is m-row n-column data in luminance component image data BIS as shown in FIG. 26B. value DV BIS, data values of m rows and n columns in R color-coded chroma component image data ICIS-R in mn DV ICIS-R, obtained by adding the mn (DV ID-R, mn = DV BIS, mn + DV ICIS-R, mn ). Similarly, m row and n column data (each pixel value) DV ID-B, mn in the B component image data ID-B is m row and n column data in the luminance component image data BIS as shown in FIG. 26C. The value DV BIS, mn is obtained by adding m rows and n columns of data values DV ICIS-B , mn in the B color-specific chromaticity component image data ICIS-B (DV ID-B, mn = DV BIS, mn) + DV ICIS-B, mn ). As a result, for each pixel, an image data ID having each color component of the G component DV ID-R, mn , the R component DV ID-R, mn, and the B component DV ID-B, mn is generated.
このようにデモザイク処理を実行すると、画像処理装置Dは、相対的に低解像度な複数の画像から相対的に高解像度な1つの画像を生成する場合に生じるギザギザな線をより低減できる。
When the demosaic process is executed in this way, the image processing apparatus D can further reduce jagged lines that are generated when a relatively high-resolution image is generated from a plurality of relatively low-resolution images.
このギザギザな線の発生について図27を用いて以下に説明する。上述のデモザイク処理ではなく、ベイヤー配列の色フィルタ部32aによって生成された原画像データに用いられる従前の一般的なデモザイクによって生成された複数の画像データから高解像度な1つの画像を生成すると、この生成した高解像度な画像では、本来、滑らかな直線や曲線がギザギザな線になってしまうことがある。例えば、図27Aに示すように、直線的な境界線で段階状に濃度の異なる被写体が、図7の個眼4、5に撮像された場合、図27Bおよび図27Cに示すように、撮像素子(イメージセンサ)の各光電変換素子(各画素)は、その受光面の全面積で受光した光量に応じたレベル(大きさ)の電気信号、すなわち、その受光面内で濃度が異なることがあっても平均濃度の情報を表す電気信号を出力する一方、撮像素子(イメージセンサ)上で同じ位置(m行n列の位置同士)であっても、前記基線長および被写体までの距離に応じてずれて被写体を撮像することになる。なお、図27Bおよび図27Cは、ベイヤー配列のイメージセンサにおけるR(赤色)の画素(R成分の濃度データを生成する画素)について示し、それら左側の図は、R光を光電変換するRの光電変換素子(Rの画素、実線の矩形)と被写体との関係、すなわち、Rの画素に映り込む被写体を示し、それら右側の図は、その映り込んだ被写体を光電変換することによってRの画素から出力される電気信号のレベルを濃淡で示す。実際には9眼の画像を用いるが、説明の都合上、仮に個眼4、5の画像から、より高解像度な画像を生成するものとすると、図27Dに示すように、高解像度な画像における同じライン上の画素であっても、異なるカメラで撮像した電気信号で画素値が生成される。例えば、画素Px1は、左カメラで撮像した電気信号で画素値が生成される一方、画素Px1と同じライン上の画素Px2は、右カメラで撮像した電気信号で画素値が生成される。これら電気信号は、上述したように、平均濃度の情報を表し、それらレベルが異なると、生成した高解像度な画像では、濃度の違いとなって図27Dに示すように、本来、滑らかな直線や曲線がギザギザな線になってしまう。このようなギザギザな線の発生は、上述した発生過程から分かるように、例えば、R成分の濃度データが個眼ごとに離散的に得られ、横方向には一定濃度であるという被写体本来の情報が得られないためである。しかしながら、上述では、第2画像フィルタFL2によって単位配列の色組成比と同じ色組成比を持つデータに原画像データISを変換するように前記輝度成分生成処理部171によってフィルタリングするので、このフィルタリングで生成された輝度成分画像データBISにおける全ての画素のデータは、同じ色成分を持つデータとして扱うことができるようになり、全ての画素で適切に比較できる。そして、上述では、デモザイク処理の補間には前記単位配列に含まれる同色の個数および輝度成分生成処理部171で生成された輝度成分画像データBISに基づいた補間方法が用いられるので、前記画素間の相互関係を考慮した適切な補間方法の選択が可能となり、この結果、このデモザイク処理部17を備える画像処理装置D等は、前記ギザギザな線をより低減できる。
The generation of this jagged line will be described below with reference to FIG. When one image having a high resolution is generated from a plurality of image data generated by a conventional general demosaic used for the original image data generated by the color filter unit 32a of the Bayer array instead of the above-described demosaicing process, In the generated high-resolution image, a smooth straight line or curved line may become a jagged line. For example, as shown in FIG. 27A, when subjects having different densities in a stepwise manner on a linear boundary are imaged on the individual eyes 4 and 5 in FIG. 7, as shown in FIGS. 27B and 27C, the image sensor Each photoelectric conversion element (each pixel) of the (image sensor) may have an electric signal of a level (size) corresponding to the amount of light received over the entire area of the light receiving surface, that is, the density may vary within the light receiving surface. Even if an electrical signal representing average density information is output, even at the same position (positions of m rows and n columns) on the image sensor (image sensor), the base line length and the distance to the subject are determined. The subject is imaged with a shift. 27B and 27C show R (red) pixels (pixels that generate R component density data) in an image sensor with a Bayer array, and the left-side diagrams show R photoelectrics for photoelectric conversion of R light. The relationship between the conversion element (R pixel, solid-line rectangle) and the subject, that is, the subject reflected in the R pixel, and the diagram on the right side thereof shows the relationship between the R pixel by photoelectrically converting the reflected subject. The level of the electric signal to be output is indicated by shading. Actually, a nine-eye image is used. However, for convenience of explanation, if a higher-resolution image is generated from the images of the individual eyes 4 and 5, as shown in FIG. Even for pixels on the same line, pixel values are generated by electrical signals captured by different cameras. For example, the pixel value of the pixel Px1 is generated by an electrical signal captured by the left camera, while the pixel value of the pixel Px2 on the same line as the pixel Px1 is generated by the electrical signal captured by the right camera. As described above, these electric signals represent average density information. If the levels are different, the generated high-resolution image has a density difference as shown in FIG. 27D. The curve becomes a jagged line. The generation of such a jagged line, as can be seen from the above-described generation process, is, for example, information inherent to the subject that R component density data is obtained discretely for each eye and has a constant density in the horizontal direction. This is because cannot be obtained. However, in the above description, since the original image data IS is filtered by the second image filter FL2 so as to convert the original image data IS into data having the same color composition ratio as that of the unit array, this filtering is performed. Data of all pixels in the generated luminance component image data BIS can be handled as data having the same color component, and can be appropriately compared with all pixels. In the above description, the interpolation method based on the number of the same color included in the unit array and the luminance component image data BIS generated by the luminance component generation processing unit 171 is used for the interpolation of the demosaic process. It is possible to select an appropriate interpolation method in consideration of the mutual relationship. As a result, the image processing apparatus D including the demosaic processing unit 17 can further reduce the jagged lines.
単位配列に含まれる同色の個数が1つである色の色度成分データをバイリニア補間法で補間しても前記ギザギザな線を生じないので、このデモザイク処理部17を備える画像処理装置D等は、比較的情報処理の容易なバイリニア補間法をこのような場合に用いることで、前記ギザギザな線をより低減しつつ、簡易に補間処理できる。
Even if the chromaticity component data of the same color included in the unit array is interpolated by the bilinear interpolation method, the jagged line is not generated. Therefore, the image processing apparatus D provided with the demosaic processing unit 17 has By using a bilinear interpolation method that is relatively easy to process in such a case, it is possible to easily perform the interpolation process while further reducing the jagged lines.
単位配列に含まれる同色の個数が複数である色の色度成分データをバイリニア補間法で補間すると前記ギザギザな線を生じてしまうが、このデモザイク処理部17を備える画像処理装置D等は、前記単位配列に含まれる同色の個数が複数である色の色度成分データを前記輝度成分画像データに基づいた補間方法で補間するので、前記ギザギザな線をより適切に低減できる。そして、前記補間方法は、前記相関の有無に応じて選択されるので、より適切な補間方法を選択できる。上述では、前記単位配列に含まれる同色の個数が複数である色の色度成分データを補間する場合でも、周辺画素と相関関係に無ければ、高解像化に用いる高周波のモアレ成分である可能性が高い。このため、前記補間方法としてバイリニア補間法が選択され、バイリニア補間法でギザギザを残し、これを高解像化で解像力を上げるために用いることができる。一方、周辺画素と相関が有れば、前記補間方法として、その相関に応じた重み付け補間法が選択され、ギザギザを低減できる。したがって、本実施形態における画像処理装置D等は、前記単位配列に含まれる同色の個数が複数である色の色度成分データを補間する場合に、前記ギザギザな線をより適切に低減できる。
Interpolating the chromaticity component data of a plurality of the same colors included in the unit array by the bilinear interpolation method generates the jagged line. However, the image processing apparatus D or the like including the demosaic processing unit 17 is configured as described above. Since the chromaticity component data of a plurality of the same colors included in the unit array is interpolated by an interpolation method based on the luminance component image data, the jagged lines can be reduced more appropriately. Since the interpolation method is selected according to the presence or absence of the correlation, a more appropriate interpolation method can be selected. In the above description, even when interpolating chromaticity component data of a plurality of the same colors included in the unit array, if there is no correlation with surrounding pixels, it is possible to be a high-frequency moire component used for high resolution. High nature. For this reason, the bilinear interpolation method is selected as the interpolation method, and the jaggedness is left by the bilinear interpolation method, which can be used to increase the resolution with high resolution. On the other hand, if there is a correlation with surrounding pixels, a weighted interpolation method corresponding to the correlation is selected as the interpolation method, and jaggedness can be reduced. Therefore, the image processing apparatus D or the like in the present embodiment can more appropriately reduce the jagged lines when interpolating chromaticity component data of a color having a plurality of the same colors included in the unit array.
このデモザイク処理部17を備える画像処理装置D等は、フィルタ係数の各値が中心位置に対し対称に配置されているので、全ての画素のデータをより同等な色成分を持つデータとして扱うことができるから、前記ギザギザな線をより低減できる。
The image processing apparatus D and the like including the demosaic processing unit 17 can treat all pixel data as data having more equivalent color components because the values of the filter coefficients are arranged symmetrically with respect to the center position. Therefore, the jagged lines can be further reduced.
このデモザイク処理部17を備える画像処理装置D等は、画像フィルタサイズおよび高周波成分の除去率の異なる第2-1および第2-2画像フィルタFL2a、FL2bを用いて輝度成分画像データBISを生成するので、画像の鮮鋭度を保持しつつ、前記ギザギザな線の少ない輝度成分画像データBISを生成できる。
The image processing apparatus D or the like including the demosaic processing unit 17 generates the luminance component image data BIS using the 2-1 and 2-2 image filters FL2a and FL2b having different image filter sizes and high-frequency component removal rates. Therefore, the luminance component image data BIS with few jagged lines can be generated while maintaining the sharpness of the image.
なお、上述の輝度成分生成処理部171は、前処理部12と兼用されても良い。
Note that the above-described luminance component generation processing unit 171 may also be used as the preprocessing unit 12.
本明細書は、上記のように様々な態様の技術を開示しているが、そのうち主な技術を以下に纏める。
This specification discloses various modes of technology as described above, and the main technologies are summarized below.
一態様にかかる画像処理装置は、原画像データを画像フィルタでフィルタリングすることによって前処理後画像データを生成する前処理部と、複数の原画像データから前記前処理部によって生成された複数の前処理後画像データそれぞれに対応する複数の画像の間で対応点探索を行う対応点探索処理部とを備え、前記原画像データは、2次元アレイ状に配列された複数の画素それぞれに対応する複数のデータであって、互いに異なる複数色の複数の濃度データを所定のパターンで配列した単位配列を、複数、さらに2次元アレイ状に配列した前記複数のデータを備え、前記画像フィルタは、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、前記単位配列の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備える。
An image processing apparatus according to an aspect includes a pre-processing unit that generates post-processed image data by filtering original image data with an image filter, and a plurality of pre-processing units generated from the plurality of original image data by the pre-processing unit. A corresponding point search processing unit that searches for a corresponding point among a plurality of images corresponding to each processed image data, and the original image data is a plurality corresponding to each of a plurality of pixels arranged in a two-dimensional array. A plurality of unit arrays in which a plurality of density data of different colors are arranged in a predetermined pattern, and the plurality of data further arranged in a two-dimensional array, and the image filter performs filtering Based on the data corresponding to the pixels located around the pixel corresponding to the target data that is data, the target data is A filter coefficient for converting the data having the same color composition ratio as the composition ratio.
対応点探索は、パターンマッチングによって対応点を探索するので、一般に、同じ色成分のデータしか利用できない。このため、例えばベイヤー配列の色フィルタを用いることによって得られた原画像データで対応点探索を実施する場合には、前記非特許文献1のように例えばG画素のデータで対応点探索が実施されることになり、原画像データの半分のデータ量しか利用できない。しかしながら、上記画像処理装置では、前記画像フィルタによって単位配列の色組成比と同じ色組成比を持つデータに原画像データを変換するように前記輝度成分生成処理部によってフィルタリングするので、全ての画素のデータを同じ色成分を持つデータとして扱うことができるようになり、この結果、全ての画素のデータを利用して対応点探索の実施が可能となる。したがって、このような画像処理装置は、対応点探索をより高精度に実施できる。
• Corresponding point search searches for corresponding points by pattern matching, so generally only data of the same color component can be used. For this reason, for example, when the corresponding point search is performed on the original image data obtained by using the color filter of the Bayer array, the corresponding point search is performed on the data of G pixels, for example, as in Non-Patent Document 1. Therefore, only half the amount of original image data can be used. However, in the image processing apparatus, since the original image data is filtered by the image filter so as to convert the original image data into data having the same color composition ratio as the unit composition, the luminance component generation processing unit filters all the pixels. The data can be handled as data having the same color component, and as a result, the corresponding point search can be performed using the data of all the pixels. Therefore, such an image processing apparatus can perform the corresponding point search with higher accuracy.
他の一態様では、上述の画像処理装置において、前記画像フィルタの各フィルタ係数は、中心位置に対し対称である。また、他の一態様では、上述の画像処理装置において、前記画像フィルタの各フィルタ係数は、前記画像フィルタ全域に前記単位配列を重ねた場合に、前記単位配列の同じ配列位置におけるフィルタ係数の和が全ての配列位置で同じ値になるように、設定されている。
In another aspect, in the above-described image processing apparatus, each filter coefficient of the image filter is symmetric with respect to a center position. According to another aspect, in the above-described image processing apparatus, each filter coefficient of the image filter is a sum of filter coefficients at the same array position of the unit array when the unit array is overlaid on the entire area of the image filter. Is set to have the same value at all array positions.
このような画像処理装置は、全ての画素のデータをより同等な色成分を持つデータとして扱うことができるから、前記ギザギザな線を低減できる。
Such an image processing apparatus can handle the data of all the pixels as data having more equivalent color components, and therefore can reduce the jagged lines.
他の一態様では、これら上述の画像処理装置において、前記複数の原画像データは、3個以上の複数の撮像光学系と、前記複数の撮像光学系に対応し、前記複数の撮像光学系それぞれによって結像された被写体の光学像をそれぞれ撮像する複数の撮像部とを備える3眼以上のアレイカメラで撮像されたデータである。
In another aspect, in the above-described image processing devices, the plurality of original image data correspond to a plurality of three or more imaging optical systems and the plurality of imaging optical systems, and each of the plurality of imaging optical systems. Is an image captured by an array camera having three or more eyes, each of which includes a plurality of imaging units that respectively capture optical images of the subject imaged by the above.
このような画像処理装置は、3眼以上のアレイカメラを用いるので、複数の原画像データを同時に得ることができる。このため、対応点探索の結果から対応点間の位置ズレ量(視差)を求める場合に、同一の被写体を同時刻で写した複数の画像間で前記位置ズレ量を求めることになるので、このような画像処理装置は、より高精度に前記位置ズレ量を求めることができる。
Since such an image processing apparatus uses an array camera with three or more eyes, a plurality of original image data can be obtained simultaneously. For this reason, when obtaining the positional deviation amount (parallax) between corresponding points from the corresponding point search result, the positional deviation amount is obtained between a plurality of images obtained by copying the same subject at the same time. Such an image processing apparatus can obtain the positional deviation amount with higher accuracy.
他の一態様では、これら上述の画像処理装置において、前記複数の原画像データは、撮像光学系と、前記撮像光学系によって結像された被写体の光学像を撮像する撮像部とを備える少なくとも1個のカメラで異なる時刻で撮像されたデータである。
In another aspect, in the above-described image processing devices, the plurality of original image data includes at least one of an imaging optical system and an imaging unit that captures an optical image of a subject imaged by the imaging optical system. Data captured at different times by a single camera.
このような画像処理装置は、異なる時刻で撮像された複数の原画像データを得ることができる。
Such an image processing apparatus can obtain a plurality of original image data captured at different times.
他の一態様では、これら上述の画像処理装置において、前記対応点探索処理部で探索された対応点間の距離である位置ズレ量を求める位置ズレ量演算部をさらに備える。
In another aspect, the above-described image processing device further includes a positional deviation amount calculation unit that obtains a positional deviation amount that is a distance between corresponding points searched by the corresponding point search processing unit.
このような画像処理装置は、位置ズレ量演算部をさらに備えるので、前記対応点探索処理部で探索された対応点間の位置ズレ量を求めることができ、より高精度に実行された対応点探索の結果を位置ズレ量の推定に効果的に反映できる。
Since such an image processing apparatus further includes a positional deviation amount calculation unit, it is possible to obtain the positional deviation amount between corresponding points searched by the corresponding point search processing unit, and corresponding points executed with higher accuracy. The result of the search can be effectively reflected in the estimation of the positional deviation amount.
他の一態様では、上述の画像処理装置において、前記位置ズレ量演算部で求めた位置ズレ量に基づいて、前記複数の原画像データから前記原画像データの画像における解像度より高い高解像度の画像を生成する高解像処理部をさらに備える。
In another aspect, in the above-described image processing apparatus, a high-resolution image higher than the resolution in the image of the original image data from the plurality of original image data based on the position shift amount obtained by the position shift amount calculation unit. Is further provided.
このような画像処理装置は、高解像処理部をさらに備えるので、前記対応点探索処理部で探索された対応点間の位置ズレ量を求め、これに基づいて複数の原画像データから高解像度な画像のデータを得ることができ、より高精度に実行された対応点探索の結果を高解像度な画像の生成に効果的に反映できる。
Since such an image processing apparatus further includes a high resolution processing unit, a positional shift amount between corresponding points searched by the corresponding point search processing unit is obtained, and based on this, a high resolution is obtained from a plurality of original image data. Image data can be obtained, and the result of corresponding point search performed with higher accuracy can be effectively reflected in the generation of a high-resolution image.
他の一態様では、上述の画像処理装置において、前記位置ズレ量演算部で求めた前記位置ズレ量を前記複数の原画像データに対応する前記複数の画像間における視差とし、前記視差に基づいて前記対応点の表す被写体までの距離を求める距離演算部をさらに備える。
In another aspect, in the above-described image processing device, the positional shift amount obtained by the positional shift amount calculation unit is set as a parallax between the plurality of images corresponding to the plurality of original image data, and based on the parallax It further includes a distance calculation unit for obtaining the distance to the subject represented by the corresponding point.
このような画像処理装置は、距離演算部をさらに備えるので、前記対応点探索処理部で探索された対応点間の視差(位置ズレ量)を求めることができ、より高精度に実行された対応点探索の結果を距離の演算に効果的に反映できる。
Since such an image processing apparatus further includes a distance calculation unit, it is possible to obtain a parallax (position shift amount) between corresponding points searched by the corresponding point search processing unit, and to perform the response executed with higher accuracy. The result of the point search can be effectively reflected in the distance calculation.
他の一態様では、上述の画像処理装置において、前記原画像データをデモザイクすることによって各色成分データを各画素ごとに持つ画像データの少なくとも1色を生成するデモザイク処理部をさらに備え、前記原画像データを第2画像フィルタでフィルタリングすることによって輝度成分画像データを生成する輝度成分生成処理部と、前記原画像データから、前記輝度成分生成処理部で生成された輝度成分画像データを、各画素ごとに、減算することによって、色度成分画像データを生成する色度成分生成処理部と、前記色度成分生成処理部で生成された色度成分画像データにおいて、前記複数色ごとに、欠落する画素の色度成分データを、前記単位配列に含まれる同色の個数および前記輝度成分生成処理部で生成された輝度成分画像データに基づいた補間方法で補間することによって、前記複数色ごとの各色別色度成分画像データを生成する色度成分補間処理部と、前記色度成分補間処理部で生成された前記複数色ごとの各色別色度成分画像データそれぞれに、前記輝度成分生成処理部で生成された輝度成分画像データを、各画素ごとに、それぞれ加算することによって、前記画像データを生成する画像生成処理部とを備え、前記第2画像フィルタは、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、前記単位配列の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備える。
In another aspect, the above-described image processing apparatus further includes a demosaicing processing unit that generates at least one color of image data having each color component data for each pixel by demosaicing the original image data. A luminance component generation processing unit that generates luminance component image data by filtering data with a second image filter, and luminance component image data generated by the luminance component generation processing unit from the original image data, for each pixel In addition, in the chromaticity component generation processing unit that generates chromaticity component image data by subtraction, and the chromaticity component image data generated by the chromaticity component generation processing unit, pixels that are missing for each of the plurality of colors Chromaticity component data of the same color and the luminance component image data generated by the luminance component generation processing unit. A chromaticity component interpolation processing unit that generates chromaticity component image data for each color for each of the plurality of colors, and for each of the plurality of colors generated by the chromaticity component interpolation processing unit. An image generation processing unit that generates the image data by adding the luminance component image data generated by the luminance component generation processing unit to each of the pixels, for each pixel. The second image filter has the same color composition as the color composition ratio of the unit array based on data corresponding to pixels located around a pixel corresponding to target data that is data to be filtered A filter coefficient for converting to data having a ratio is provided.
このような画像処理装置では、前記画像フィルタによって単位配列の色組成比と同じ色組成比を持つデータに原画像データを変換するように前記輝度成分生成処理部によってフィルタリングするので、このフィルタリングで生成された輝度成分画像データにおける全ての画素のデータは、同じ色成分を持つデータとして扱うことができるようになり、全ての画素で適切に比較できる。そして、上記画像処理装置では、デモザイク処理の補間には前記単位配列に含まれる同色の個数および前記輝度成分生成処理部で生成された輝度成分画像データに基づいた補間方法が用いられるので、前記画素間の相互関係を考慮した適切な補間方法の選択が可能となり、この結果、上記画像処理装置は、高解像度化に伴うギザギザな線をより低減できる。
In such an image processing device, since the original image data is filtered by the image filter so as to convert the original image data into data having the same color composition ratio as that of the unit array, it is generated by this filtering. Data of all pixels in the luminance component image data thus obtained can be handled as data having the same color component, and can be appropriately compared with all pixels. In the image processing apparatus, the interpolation method based on the number of the same color included in the unit array and the luminance component image data generated by the luminance component generation processing unit is used for the interpolation of the demosaic process. Accordingly, it is possible to select an appropriate interpolation method in consideration of the mutual relationship between them, and as a result, the image processing apparatus can further reduce the jagged lines accompanying the increase in resolution.
そして、他の一態様にかかる画像処理方法は、原画像データを画像フィルタでフィルタリングすることによって前処理後画像データを生成する前処理工程と、複数の原画像データから前記前処理工程によって生成された複数の前処理後画像データに対応する複数の画像間で対応点探索を行う対応点探索処理工程とを備え、前記原画像データは、2次元アレイ状に配列された複数の画素それぞれに対応する複数のデータであって、互いに異なる複数色の複数の濃度データを所定のパターンで配列した単位配列を、複数、さらに2次元アレイ状に配列した前記複数のデータを備え、前記画像フィルタは、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、前記単位配列の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備える。
An image processing method according to another aspect is generated by the preprocessing step of generating the preprocessed image data by filtering the original image data with an image filter, and the preprocessing step from a plurality of original image data. A corresponding point search processing step for searching corresponding points between a plurality of images corresponding to a plurality of pre-processed image data, wherein the original image data corresponds to each of a plurality of pixels arranged in a two-dimensional array. A plurality of unit arrangements in which a plurality of density data of a plurality of different colors are arranged in a predetermined pattern, and the plurality of data arranged in a two-dimensional array, and the image filter includes: Based on the data corresponding to the pixels located around the pixel corresponding to the target data that is the data to be filtered, the target data is A filter coefficient for converting the data having the same color composition ratio as the color composition ratio of the array.
そして、他の一態様にかかる画像処理プログラムは、コンピュータに、原画像データを画像フィルタでフィルタリングすることによって前処理後画像データを生成する前処理部と、複数の原画像データから前記前処理部によって生成された複数の前処理後画像データに対応する複数の画像間で対応点探索を行う対応点探索処理部とを実現させるための画像処理プログラムであって、前記原画像データは、2次元アレイ状に配列された複数の画素それぞれに対応する複数のデータであって、互いに異なる複数色の複数の濃度データを所定のパターンで配列した単位配列を、複数、さらに2次元アレイ状に配列した前記複数のデータを備え、前記画像フィルタは、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、前記単位配列の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備える。
An image processing program according to another aspect includes a preprocessing unit that generates post-processed image data by filtering original image data with an image filter, and the preprocessing unit from a plurality of original image data. An image processing program for realizing a corresponding point search processing unit that performs corresponding point search between a plurality of images corresponding to a plurality of preprocessed image data generated by the processing, wherein the original image data is two-dimensional A plurality of unit arrays, each of which is a plurality of data corresponding to each of a plurality of pixels arranged in an array and in which a plurality of density data of different colors are arranged in a predetermined pattern, are further arranged in a two-dimensional array The image filter includes a plurality of data, and the image filter is an image located around a pixel corresponding to target data that is data to be filtered. Based on the corresponding data, a filter coefficient for converting the target data, the data having the same color composition ratio as the color composition ratio of the unit sequence.
このような画像処理方法および画像処理プログラムでは、前記画像フィルタによって単位配列の色組成比と同じ色組成比を持つデータに原画像データを変換するように前記輝度成分生成処理部によってフィルタリングするので、全ての画素のデータを同じ色成分を持つデータとして扱うことができるようになり、この結果、全ての画素のデータを利用して対応点探索の実施が可能となる。したがって、このような画像処理方法および画像処理プログラムは、対応点探索をより高精度に実施できる。
In such an image processing method and image processing program, the luminance component generation processing unit performs filtering so as to convert the original image data into data having the same color composition ratio as the color composition ratio of the unit array by the image filter. All the pixel data can be handled as data having the same color component, and as a result, the corresponding point search can be performed using the data of all the pixels. Therefore, such an image processing method and image processing program can perform corresponding point search with higher accuracy.
この出願は、2013年12月2日に出願された日本国特許出願特願2013-249607を基礎とするものであり、その内容は、本願に含まれるものである。
This application is based on Japanese Patent Application No. 2013-249607 filed on December 2, 2013, the contents of which are included in this application.
本発明を表現するために、上述において図面を参照しながら実施形態を通して本発明を適切且つ十分に説明したが、当業者であれば上述の実施形態を変更および/または改良することは容易に為し得ることであると認識すべきである。したがって、当業者が実施する変更形態または改良形態が、請求の範囲に記載された請求項の権利範囲を離脱するレベルのものでない限り、当該変更形態または当該改良形態は、当該請求項の権利範囲に包括されると解釈される。
In order to express the present invention, the present invention has been properly and fully described through the embodiments with reference to the drawings. However, those skilled in the art can easily change and / or improve the above-described embodiments. It should be recognized that this is possible. Therefore, unless the modifications or improvements implemented by those skilled in the art are at a level that departs from the scope of the claims recited in the claims, the modifications or improvements are not covered by the claims. To be construed as inclusive.
本発明によれば、画像を処理する画像処理技術、画像処理方法および画像処理プログラムを提供できる。
According to the present invention, an image processing technique, an image processing method, and an image processing program for processing an image can be provided.
Claims (11)
- 原画像データを画像フィルタでフィルタリングすることによって前処理後画像データを生成する前処理部と、
複数の原画像データから前記前処理部によって生成された複数の前処理後画像データそれぞれに対応する複数の画像の間で対応点探索を行う対応点探索処理部とを備え、
前記原画像データは、2次元アレイ状に配列された複数の画素それぞれに対応する複数のデータであって、互いに異なる複数色の複数の濃度データを所定のパターンで配列した単位配列を、複数、さらに2次元アレイ状に配列した前記複数のデータを備え、
前記画像フィルタは、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、前記単位配列の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備えること
を特徴とする画像処理装置。 A pre-processing unit that generates post-processed image data by filtering the original image data with an image filter;
A corresponding point search processing unit that performs a corresponding point search between a plurality of images corresponding to each of a plurality of pre-processed image data generated by the pre-processing unit from a plurality of original image data,
The original image data is a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array, and a plurality of unit arrays in which a plurality of density data of a plurality of different colors are arranged in a predetermined pattern, The data further comprises a plurality of data arranged in a two-dimensional array,
The image filter is data having the same color composition ratio as the color composition ratio of the unit array based on data corresponding to pixels located around a pixel corresponding to target data that is data to be filtered. An image processing apparatus comprising a filter coefficient for conversion into - 前記画像フィルタの各フィルタ係数は、中心位置に対し対称であること
を特徴とする請求項1に記載の画像処理装置。 The image processing apparatus according to claim 1, wherein each filter coefficient of the image filter is symmetric with respect to a center position. - 前記画像フィルタの各フィルタ係数は、前記画像フィルタ全域に前記単位配列を重ねた場合に、前記単位配列の同じ配列位置におけるフィルタ係数の和が、全ての配列位置で同じ値になるように、設定されていること
を特徴とする請求項1に記載の画像処理装置。 Each filter coefficient of the image filter is set so that the sum of the filter coefficients at the same arrangement position of the unit arrangement becomes the same value at all arrangement positions when the unit arrangement is overlapped over the entire image filter. The image processing apparatus according to claim 1, wherein: - 前記複数の原画像データは、3個以上の複数の撮像光学系と、前記複数の撮像光学系に対応し、前記複数の撮像光学系それぞれによって結像された被写体の光学像をそれぞれ撮像する複数の撮像部とを備える3眼以上のアレイカメラで撮像されたデータであること
を特徴とする請求項1ないし請求項3のいずれか1項に記載の画像処理装置。 The plurality of original image data corresponds to the plurality of image pickup optical systems of three or more and the plurality of image pickup optical systems, and each of the plurality of image pickup optical systems picks up an optical image of a subject formed by each of the plurality of image pickup optical systems. 4. The image processing apparatus according to claim 1, wherein the image processing apparatus includes data captured by an array camera having three or more eyes. - 前記複数の原画像データは、撮像光学系と、前記撮像光学系によって結像された被写体の光学像を撮像する撮像部とを備える少なくとも1個のカメラで異なる時刻で撮像されたデータであること
を特徴とする請求項1ないし請求項3のいずれか1項に記載の画像処理装置。 The plurality of original image data is data captured at different times by at least one camera including an imaging optical system and an imaging unit that captures an optical image of a subject formed by the imaging optical system. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. - 前記対応点探索処理部で探索された対応点間の距離である位置ズレ量を求める位置ズレ量演算部をさらに備えること
を特徴とする請求項1ないし請求項5のいずれか1項に記載の画像処理装置。 The position shift amount calculation part which calculates | requires the position shift amount which is the distance between the corresponding points searched by the said corresponding point search process part is further provided. The Claim 1 thru | or 5 characterized by the above-mentioned. Image processing device. - 前記位置ズレ量演算部で求めた位置ズレ量に基づいて、前記複数の原画像データから前記原画像データの画像における解像度より高い高解像度の画像を生成する高解像処理部をさらに備えること
を特徴とする請求項6に記載の画像処理装置。 A high-resolution processing unit that generates a high-resolution image higher than the resolution of the image of the original image data from the plurality of original image data based on the position shift amount obtained by the position shift amount calculation unit. The image processing apparatus according to claim 6. - 前記位置ズレ量演算部で求めた前記位置ズレ量を前記複数の原画像データに対応する前記複数の画像間における視差とし、前記視差に基づいて前記対応点の表す被写体までの距離を求める距離演算部をさらに備えること
を特徴とする請求項6に記載の画像処理装置。 Distance calculation for obtaining the distance to the subject represented by the corresponding point based on the parallax, using the positional deviation amount obtained by the positional deviation amount calculation unit as the parallax between the plurality of images corresponding to the plurality of original image data The image processing apparatus according to claim 6, further comprising a unit. - 前記原画像データをデモザイクすることによって各色成分データを各画素ごとに持つ画像データの少なくとも1色を生成するデモザイク処理部をさらに備え、
前記原画像データを第2画像フィルタでフィルタリングすることによって輝度成分画像データを生成する輝度成分生成処理部と、
前記原画像データから、前記輝度成分生成処理部で生成された輝度成分画像データを、各画素ごとに、減算することによって、色度成分画像データを生成する色度成分生成処理部と、
前記色度成分生成処理部で生成された色度成分画像データにおいて、前記複数色ごとに、欠落する画素の色度成分データを、前記単位配列に含まれる同色の個数および前記輝度成分生成処理部で生成された輝度成分画像データに基づいた補間方法で補間することによって、前記複数色ごとの各色別色度成分画像データを生成する色度成分補間処理部と、
前記色度成分補間処理部で生成された前記複数色ごとの各色別色度成分画像データそれぞれに、前記輝度成分生成処理部で生成された輝度成分画像データを、各画素ごとに、それぞれ加算することによって、前記画像データを生成する画像生成処理部とを備え、
前記第2画像フィルタは、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、前記単位配列の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備えること
を特徴とする請求項7に記載の画像処理装置。 A demosaic processing unit that generates at least one color of image data having each color component data for each pixel by demosaicing the original image data;
A luminance component generation processing unit that generates luminance component image data by filtering the original image data with a second image filter;
A chromaticity component generation processing unit that generates chromaticity component image data by subtracting the luminance component image data generated by the luminance component generation processing unit from the original image data for each pixel;
In the chromaticity component image data generated by the chromaticity component generation processing unit, for each of the plurality of colors, the chromaticity component data of the missing pixel is set to the number of the same color included in the unit array and the luminance component generation processing unit. A chromaticity component interpolation processing unit that generates chromaticity component image data for each color for each of the plurality of colors by performing interpolation using an interpolation method based on the luminance component image data generated in
The luminance component image data generated by the luminance component generation processing unit is added to each pixel for each of the chromaticity component image data for each color generated by the chromaticity component interpolation processing unit for each pixel. An image generation processing unit for generating the image data,
The second image filter has the same color composition ratio as the color composition ratio of the unit array based on the data corresponding to the pixels located around the pixel corresponding to the target data that is the data to be filtered. The image processing apparatus according to claim 7, further comprising: a filter coefficient that converts the data into possessed data. - 原画像データを画像フィルタでフィルタリングすることによって前処理後画像データを生成する前処理工程と、
複数の原画像データから前記前処理工程によって生成された複数の前処理後画像データに対応する複数の画像間で対応点探索を行う対応点探索処理工程とを備え、
前記原画像データは、2次元アレイ状に配列された複数の画素それぞれに対応する複数のデータであって、互いに異なる複数色の複数の濃度データを所定のパターンで配列した単位配列を、複数、さらに2次元アレイ状に配列した前記複数のデータを備え、
前記画像フィルタは、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、前記単位配列の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備えること
を特徴とする画像処理方法。 A preprocessing step for generating preprocessed image data by filtering the original image data with an image filter;
A corresponding point search processing step of performing a corresponding point search between a plurality of images corresponding to a plurality of pre-processed image data generated from a plurality of original image data by the pre-processing step,
The original image data is a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array, and a plurality of unit arrays in which a plurality of density data of a plurality of different colors are arranged in a predetermined pattern, The data further comprises a plurality of data arranged in a two-dimensional array,
The image filter is data having the same color composition ratio as the color composition ratio of the unit array based on data corresponding to pixels located around a pixel corresponding to target data that is data to be filtered. An image processing method comprising: a filter coefficient to be converted into - コンピュータに、
原画像データを画像フィルタでフィルタリングすることによって前処理後画像データを生成する前処理部と、
複数の原画像データから前記前処理部によって生成された複数の前処理後画像データに対応する複数の画像間で対応点探索を行う対応点探索処理部とを実現させるための画像処理プログラムであって、
前記原画像データは、2次元アレイ状に配列された複数の画素それぞれに対応する複数のデータであって、互いに異なる複数色の複数の濃度データを所定のパターンで配列した単位配列を、複数、さらに2次元アレイ状に配列した前記複数のデータを備え、
前記画像フィルタは、フィルタリングするデータである対象データに対応する画素の周辺に位置する画素に対応するデータに基づいて、前記対象データを、前記単位配列の色組成比と同じ色組成比を持つデータに変換するフィルタ係数を備えること
を特徴とする画像処理プログラム。 On the computer,
A pre-processing unit that generates post-processed image data by filtering the original image data with an image filter;
An image processing program for realizing a corresponding point search processing unit that searches for a corresponding point among a plurality of images corresponding to a plurality of preprocessed image data generated by the preprocessing unit from a plurality of original image data. And
The original image data is a plurality of data corresponding to each of a plurality of pixels arranged in a two-dimensional array, and a plurality of unit arrays in which a plurality of density data of a plurality of different colors are arranged in a predetermined pattern, The data further comprises a plurality of data arranged in a two-dimensional array,
The image filter is data having the same color composition ratio as the color composition ratio of the unit array based on data corresponding to pixels located around a pixel corresponding to target data that is data to be filtered. An image processing program comprising a filter coefficient to be converted into
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2015551440A JPWO2015083499A1 (en) | 2013-12-02 | 2014-11-10 | Image processing apparatus, method, and program |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2013-249607 | 2013-12-02 | ||
JP2013249607 | 2013-12-02 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015083499A1 true WO2015083499A1 (en) | 2015-06-11 |
Family
ID=53273270
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2014/079767 WO2015083499A1 (en) | 2013-12-02 | 2014-11-10 | Image processing device, method and program |
Country Status (2)
Country | Link |
---|---|
JP (1) | JPWO2015083499A1 (en) |
WO (1) | WO2015083499A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2020088684A (en) * | 2018-11-28 | 2020-06-04 | 株式会社アサヒ電子研究所 | Compound-eye imaging apparatus |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2004068862A1 (en) * | 2003-01-31 | 2004-08-12 | The Circle For The Promotion Of Science And Engineering | Method for creating high resolution color image, system for creating high resolution color image and program for creating high resolution color image |
JP2010276469A (en) * | 2009-05-28 | 2010-12-09 | Konica Minolta Holdings Inc | Image processor and image processing method of ranging apparatus |
JP2013114477A (en) * | 2011-11-29 | 2013-06-10 | Konica Minolta Inc | Image processing device, image processing method and image processing program |
-
2014
- 2014-11-10 JP JP2015551440A patent/JPWO2015083499A1/en active Pending
- 2014-11-10 WO PCT/JP2014/079767 patent/WO2015083499A1/en active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2004068862A1 (en) * | 2003-01-31 | 2004-08-12 | The Circle For The Promotion Of Science And Engineering | Method for creating high resolution color image, system for creating high resolution color image and program for creating high resolution color image |
JP2010276469A (en) * | 2009-05-28 | 2010-12-09 | Konica Minolta Holdings Inc | Image processor and image processing method of ranging apparatus |
JP2013114477A (en) * | 2011-11-29 | 2013-06-10 | Konica Minolta Inc | Image processing device, image processing method and image processing program |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2020088684A (en) * | 2018-11-28 | 2020-06-04 | 株式会社アサヒ電子研究所 | Compound-eye imaging apparatus |
JP7182437B2 (en) | 2018-11-28 | 2022-12-02 | 株式会社アサヒ電子研究所 | Compound eye imaging device |
Also Published As
Publication number | Publication date |
---|---|
JPWO2015083499A1 (en) | 2017-03-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9509978B2 (en) | Image processing method, image processing apparatus, image-capturing apparatus, and image processing program | |
US9275463B2 (en) | Stereo image processing device and stereo image processing method | |
CN111510691B (en) | Color interpolation method and device, equipment and storage medium | |
JP5128207B2 (en) | Image processing apparatus, image processing method, and image processing program | |
JP5341010B2 (en) | Image processing apparatus, imaging apparatus, program, and image processing method | |
JP5017419B2 (en) | Image generating apparatus, image generating method, and program | |
KR101356286B1 (en) | Image processing device, image processing method, program, and imaging device | |
US6252577B1 (en) | Efficient methodology for scaling and transferring images | |
US20080123997A1 (en) | Providing a desired resolution color image | |
US9338364B2 (en) | Imaging device and image generation method | |
JP4598162B2 (en) | Imaging processing device | |
JP5709131B2 (en) | Image processing system | |
US20110279712A1 (en) | Imaging apparatus, integrated circuit, and image processing method | |
WO2006048962A1 (en) | Pixel signal processor and pixel signal processing method | |
JP2013017142A (en) | Image processing apparatus, imaging apparatus, image processing method, and program | |
JP5718138B2 (en) | Image signal processing apparatus and program | |
US20110032269A1 (en) | Automatically Resizing Demosaicked Full-Color Images Using Edge-Orientation Maps Formed In The Demosaicking Process | |
JP2014042176A (en) | Image processing device and method, program and solid image pickup device | |
WO2015083502A1 (en) | Image processing device, method and program | |
WO2015083499A1 (en) | Image processing device, method and program | |
EP4102828A1 (en) | Image sensor including image signal processor and operating method of the image sensor | |
JP5036524B2 (en) | Image processing apparatus, image processing method, program, and imaging apparatus | |
CN112669355A (en) | Method and system for splicing and fusing focusing stack data based on RGB-D super-pixel segmentation | |
JP6461184B2 (en) | Image processing unit, imaging apparatus, image processing program, and image processing method | |
JP6291795B2 (en) | Imaging system and imaging method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 14867327 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2015551440 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 14867327 Country of ref document: EP Kind code of ref document: A1 |