WO2011099626A1 - 焦点調節装置、及び焦点調節プログラム - Google Patents
焦点調節装置、及び焦点調節プログラム Download PDFInfo
- Publication number
- WO2011099626A1 WO2011099626A1 PCT/JP2011/053156 JP2011053156W WO2011099626A1 WO 2011099626 A1 WO2011099626 A1 WO 2011099626A1 JP 2011053156 W JP2011053156 W JP 2011053156W WO 2011099626 A1 WO2011099626 A1 WO 2011099626A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- edge
- detection unit
- focus
- image
- color component
- Prior art date
Links
- 238000009826 distribution Methods 0.000 claims abstract description 187
- 238000001514 detection method Methods 0.000 claims abstract description 147
- 238000003708 edge detection Methods 0.000 claims abstract description 95
- 230000003287 optical effect Effects 0.000 claims abstract description 23
- 238000000034 method Methods 0.000 claims description 53
- 239000003086 colorant Substances 0.000 claims description 12
- 108091006146 Channels Proteins 0.000 description 225
- 238000003384 imaging method Methods 0.000 description 103
- 238000011156 evaluation Methods 0.000 description 57
- 238000012545 processing Methods 0.000 description 56
- 238000010586 diagram Methods 0.000 description 50
- 230000006870 function Effects 0.000 description 32
- 230000033001 locomotion Effects 0.000 description 25
- 230000008859 change Effects 0.000 description 22
- 230000000875 corresponding effect Effects 0.000 description 16
- 238000006243 chemical reaction Methods 0.000 description 13
- 230000004075 alteration Effects 0.000 description 12
- 230000008569 process Effects 0.000 description 12
- 239000000284 extract Substances 0.000 description 11
- 230000014509 gene expression Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 9
- 230000009467 reduction Effects 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000015572 biosynthetic process Effects 0.000 description 3
- 238000012937 correction Methods 0.000 description 3
- 230000003595 spectral effect Effects 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000007480 spreading Effects 0.000 description 1
- 230000008719 thickening Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
- G02B7/28—Systems for automatic generation of focusing signals
- G02B7/36—Systems for automatic generation of focusing signals using image sharpness techniques, e.g. image processing techniques for generating autofocus signals
- G02B7/38—Systems for automatic generation of focusing signals using image sharpness techniques, e.g. image processing techniques for generating autofocus signals measured at different points on the optical axis, e.g. focussing on two or more planes and comparing image data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
- H04N23/673—Focus control based on electronic image sensor signals based on contrast or high frequency components of image signals, e.g. hill climbing method
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
- G02B7/28—Systems for automatic generation of focusing signals
- G02B7/36—Systems for automatic generation of focusing signals using image sharpness techniques, e.g. image processing techniques for generating autofocus signals
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B13/00—Viewfinders; Focusing aids for cameras; Means for focusing for cameras; Autofocus systems for cameras
- G03B13/32—Means for focusing
- G03B13/34—Power focusing
- G03B13/36—Autofocus systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
- H04N25/61—Noise processing, e.g. detecting, correcting, reducing or removing noise the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
- H04N25/611—Correction of chromatic aberration
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B3/00—Focusing arrangements of general interest for cameras, projectors or printers
- G03B3/10—Power-operated focusing
Definitions
- the present invention relates to a focus adjustment device that adjusts a focus by applying chromatic aberration, and a focus adjustment program.
- Patent Document 1 A video camera that applies chromatic aberration when focusing on a subject image obtained through a lens is known (see Patent Document 1).
- the video camera disclosed in Patent Document 1 can quickly focus on a subject image that has been focused once, but can quickly focus on a subject image that has never been focused. I could not. That is, the video camera disclosed in Patent Document 1 has a problem that it cannot quickly focus on the subject image.
- the focus adjustment apparatus includes an edge detection unit that detects an edge of the subject image for each color component constituting the image including the subject image; and based on the edge detected by the edge detection unit.
- a distribution detection unit that detects a distribution of the in-focus state and the out-of-focus state of the image; and a control unit that moves the lens based on the distribution detected by the distribution detection unit.
- the subject image is incident from an optical system having the lens. Further, as a result of the control unit moving the lens, the subject image is focused.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit detects the distribution based on a gradient of the color component amount of the edge detected by the edge detection unit.
- the focus adjustment device may be configured as follows: the distribution detection unit is configured to be nearer to the subject based on a ratio or difference between the color component amounts of the edges detected by the edge detection unit. And a defocus amount indicating a direction index indicating which of the two points is in focus.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit is configured to perform the defocusing based on a peak-to-peak distance of a color component amount ratio or a difference detected by the edge detection unit. Detect the amount.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit is configured to detect the near point side and the far side of the subject based on a line spread function corresponding to the edge detected by the edge detection unit. A direction index indicating which point is in focus and a defocus amount are detected.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit is configured to perform the defocusing based on a standard deviation or a half value width of a line spread function corresponding to the edge detected by the edge detection unit. Detect the amount.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit selects, from the edges detected by the edge detection unit, an upper edge in descending order of color component amount, and selects the selected edge Based on the edge, the distribution of the focused state and the out-of-focus state of the image is detected.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit selects, from the edges detected by the edge detection unit, a higher-order edge in descending order of contrast of the color component; Based on the selected edge, the distribution of the in-focus state and the out-of-focus state of the image is detected.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit includes a mixture of edges having the same contrast of the color components and edges having different signal-to-noise ratios of the color components.
- the upper edge is selected in descending order of the signal-to-noise ratio of the color component, and the distribution of the in-focus state and the out-of-focus state of the image is detected based on the selected edge.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit has a relatively low signal-to-noise ratio when edges having different signal-to-noise ratios of the color components are mixed. And selecting at least one of an edge having a relatively high contrast and an edge having a relatively high signal-to-noise ratio and a relatively low contrast, and based on the selected edge, The distribution of the in-focus state and the out-of-focus state is detected.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit includes edges in which two or more primary colors are included and the color components change in phase among the edges detected by the edge detection unit And the distribution of the in-focus state and the out-of-focus state of the image is detected based on the selected edge.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit selects an edge including a green component when the edge includes two primary colors.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit detects an edge whose color component is flat more than a predetermined width among the edges detected by the edge detection unit. Based on the selected edge, the distribution of the in-focus state and the out-of-focus state of the image is detected.
- the focus adjustment apparatus may be configured as follows: the distribution detection unit has a length greater than or equal to a length determined according to a signal-to-noise ratio of the color component among the edges detected by the edge detection unit. An edge is selected, and the distribution of the in-focus state and the out-of-focus state of the image is detected based on the selected edge.
- the focus adjustment apparatus includes an edge detection unit that detects an edge of the subject image for each color component constituting the image including the subject image; and a line spread of the edge detected by the edge detection unit A distribution detection unit that calculates a function; and a control unit that moves the lens based on the line spread function.
- the subject image is incident from an optical system having the lens. Further, as a result of the control unit moving the lens, the subject image is focused.
- the computer is a computer that executes a focus adjustment program.
- the focus adjustment program detects an edge of the subject image for each color component constituting an image including a subject image incident from an optical system having a lens for performing focus adjustment; and based on the edge And a procedure for detecting the distribution of the in-focus state and the out-of-focus state of the image; and a step of moving the lens to focus on the subject image based on the distribution.
- the computer is a computer that executes a focus adjustment program.
- the focus adjustment program includes a procedure for detecting an edge of the subject image for each color component constituting an image including the subject image incident from an optical system having a lens for performing focus adjustment; A procedure for calculating a function; and a procedure for moving the lens to focus on the subject image based on the line spread function.
- the focus adjustment device and the focus adjustment program according to the present invention have an effect that a subject image can be focused quickly.
- FIG. 1 is a block diagram illustrating a configuration of an imaging device 100 including a lens barrel 111 and a focus adjustment device 191, and a storage medium 200.
- FIG. 2 is a diagram illustrating the relationship between the subject at a position away from the AF lens 112 by the subject distance, the AF lens 112, the imaging surface of the imaging device 119, and the circle of confusion.
- FIG. 3 is a diagram illustrating a positional relationship on the optical axis between the focal point of the red light incident on the AF lens 112, the focal point of the green light, and the focal point of the blue light.
- FIG. 4 is a diagram showing the relationship between the wavelength of light incident on the AF lens 112 and the position of the image plane on the optical axis.
- FIG. 1 is a block diagram illustrating a configuration of an imaging device 100 including a lens barrel 111 and a focus adjustment device 191, and a storage medium 200.
- FIG. 2 is a diagram illustrating the relationship between the subject at a position away from the
- FIG. 5A is a diagram illustrating the relationship between the AF lens 112, the imaging surface of the imaging device 119, and the circle of confusion in the front pin state.
- FIG. 5B is a diagram illustrating a circle of confusion in the front pin state formed on the imaging surface.
- FIG. 5C is a diagram illustrating a relationship between a color component amount and a radial position from the center of the circle of confusion in the front pin state toward the outer periphery.
- FIG. 6A is a diagram illustrating the relationship between the AF lens 112, the imaging surface of the imaging device 119, and the circle of confusion in the rear pin state.
- FIG. 6B is a diagram illustrating a circle of confusion in the rear pin state formed on the imaging surface.
- FIG. 7 is a diagram illustrating an example of a captured image.
- FIG. 8A is a diagram showing the amount of color components in the vicinity of the edge position 13, and is a gradation sectional view in a state in which the focus is on the G channel.
- FIG. 8B is a diagram showing the amount of color components in the vicinity of the edge position 13, and is a gradation cross-sectional view in a state in which the R channel is in focus.
- FIG. 8C is a diagram showing the amount of color components in the vicinity of the edge position 13, and is a gradation cross-sectional view in a state in which focusing is performed on the B channel.
- FIG. 8A is a diagram showing the amount of color components in the vicinity of the edge position 13, and is a gradation sectional view in a state in which the focus is on the G channel.
- FIG. 8B is a diagram showing the amount of color components in the vicinity of the edge position 13, and is a gradation cross-sectional view in a state in
- FIG. 9A is a diagram showing the amount of color components in the vicinity of the edge position 13, and shows the relationship between the edge of the R channel in the front pin state, the edge of the R channel in the rear pin state, and the edge of the G channel in the focused state.
- FIG. 9B is a diagram showing the amount of color components in the vicinity of the edge position 13, and shows the relationship between the edge of the B channel in the front pin state, the edge of the B channel in the rear pin state, and the edge of the G channel in the focused state.
- FIG. 10A is a diagram showing a “color component amount difference” in the vicinity of the edge position 13, and shows a relationship between the position on the axis D and the difference between the R channel color component amount and the G channel color component amount.
- FIG. 10B is a diagram showing a “color component amount difference” in the vicinity of the edge position 13, and shows a waveform when approaching the in-focus state as compared with FIG. 10A.
- FIG. 11A is a diagram showing LSF in the focused state in the G channel.
- FIG. 11B is a diagram showing LSF in the out-of-focus state (small blur) in the G channel.
- FIG. 11C is a diagram showing LSF in the out-of-focus state (large blur) in the G channel.
- FIG. 12A is a diagram showing the LSF in the focused state for each color channel (R, G, B).
- FIG. 12B is a diagram illustrating the LSF in the front pin state for each color channel (R, G, B).
- FIG. 12C is a diagram illustrating LSF in the rear pin state for each color channel (R, G, B).
- FIG. 13 is a diagram showing the relationship between the LSF profile (data) for each color channel (R, G, B) and the degree of blur.
- FIG. 14 is a diagram illustrating a relationship between a function value using a difference in standard deviation of LSF as a parameter and a subject distance.
- FIG. 15A shows an example of a part (partial image) of an image captured by the imaging unit.
- FIG. 15B shows an R-channel edge image extracted from the captured image.
- FIG. 15C shows an edge image of the G channel extracted from the captured image.
- FIG. 15D is an image showing a result of calculating a logical product (AND) of the edge image of the R channel and the edge image of the G channel.
- FIG. 16 is a flowchart showing the operation of the edge detection unit 192.
- FIG. 17 is a diagram illustrating a color component amount in the vicinity of the edge position 13 and a calculation example using a Laplacian filter.
- FIG. 18 is a diagram illustrating an example of a depth map.
- FIG. 19 is a flowchart showing a focus drive procedure of the focus adjustment device 191 in a tracking operation for tracking a subject.
- FIG. 20 is a flowchart showing an edge characteristic analysis procedure.
- FIG. 21 is a diagram illustrating a scan area when it is determined to be in the front pin state and a scan area when it is determined to be in the back pin state.
- FIG. 22A is a diagram showing the movement of the lens position in the hill-climbing contrast scan in the shooting AF, and shows an example of the relationship between the position of the AF lens 112 in the contrast scan and the contrast value.
- FIG. 22B shows the movement of the position of the AF lens 112 in the normal contrast scan (see the middle part of FIG. 21).
- FIG. 22C shows the movement of the position of the AF lens 112 in a contrast scan (see the lower part of FIG. 21) based on determination of the front pin state and the rear pin state based on the evaluation value.
- FIG. 22A is a diagram showing the movement of the lens position in the hill-climbing contrast scan in the shooting AF, and shows an example of the relationship between the position of the AF lens 112 in the contrast scan and the contrast value.
- FIG. 22B shows the
- FIG. 23 is a flowchart showing the procedure of focusing driving based on LSF.
- FIG. 24 is a diagram illustrating an example of a “defocus-drive pulse table”.
- FIG. 25 is a diagram illustrating a contrast scan area when the front pin and the degree thereof are determined, and a contrast scan area when the rear pin and the degree are determined.
- FIG. 26 is a flowchart showing the procedure of focusing drive of the focus adjustment device 191 in the tracking operation for tracking the subject.
- FIG. 27 is a diagram showing an example of focusing driving in contrast scanning based on the “defocus-driving pulse table”.
- FIG. 28 is an enlarged view of an example of focus drive in contrast scan based on “defocus-drive pulse table”.
- FIG. 29 is a flowchart showing the operation of the focus adjustment apparatus 191 having the moving image capturing mode.
- FIG. 30A is a diagram showing a histogram of difference data in a focused state.
- FIG. 30B is a diagram illustrating a histogram of difference data in an out-of-focus state.
- FIG. 31 is a flowchart showing the operation of the focus adjustment apparatus 191 for determining spot light.
- FIG. 32 is a diagram for explaining the switching determination operation to the macro imaging mode.
- FIG. 33A is a diagram for explaining the priority order of edges selected when edges having different signal-to-noise ratios of color components are mixed, and the signal-to-noise ratio is relatively high and the contrast is relative. The case of a low edge is shown.
- FIG. 30A is a diagram showing a histogram of difference data in a focused state.
- FIG. 30B is a diagram illustrating a histogram of difference data in an out-of-focus state.
- FIG. 31
- FIG. 33B is a diagram for explaining the priority order of edges selected when edges having different signal-to-noise ratios of color components are mixed, and the signal-to-noise ratio is relatively low and the contrast is relative.
- FIG. 34A is a diagram showing the priority order of edges selected when two primary colors are included, and shows the case of an edge constituted by an R channel and a G channel that change in the same phase.
- FIG. 34B is a diagram showing the priority order of edges selected when two primary colors are included.
- the edge having the priority shown in FIG. 34B is configured by the R channel and the G channel that change in the same phase, and the contrast of the R channel is lower than the contrast of the R channel shown in FIG. 34A.
- FIG. 34C is a diagram illustrating the priority order of edges selected when two primary colors are included.
- FIG. 34C shows the case of an edge composed of an R channel and a B channel that change in phase.
- FIG. 35A shows a change in the amount of color component in accordance with the position crossing one edge where the color component is flat.
- FIG. 35B shows a change in the amount of color component depending on the position crossing a plurality of edges.
- FIG. 36 is a flowchart illustrating a procedure for selecting an upper edge in accordance with the priority order.
- FIG. 1 is a block diagram illustrating a configuration of an imaging device 100 including a lens barrel 111 and a focus adjustment device 191, and a storage medium 200.
- the imaging apparatus 100 captures a subject image incident from the lens barrel 111 and stores the obtained image in the storage medium 200 as a still image or a moving image.
- the lens barrel 111 includes a focus adjustment lens (hereinafter referred to as “AF (Auto-Focus) lens”) 112, a lens driving unit 116, an AF encoder 117, and a lens barrel control unit 118.
- the lens barrel 111 may be detachably connected to the imaging device 100 or may be integrated with the imaging device 100.
- the AF lens 112 is driven by the lens driving unit 116.
- the AF lens 112 guides a subject image to a light receiving surface (photoelectric conversion surface) of an imaging element 119 of the imaging unit 110 described later.
- the AF encoder 117 detects the movement of the AF lens 112 and outputs a signal corresponding to the movement amount of the AF lens 112 to the lens barrel control unit 118.
- the signal corresponding to the movement amount of the AF lens 112 may be, for example, a sine wave signal whose phase changes according to the movement amount of the AF lens 112.
- the lens barrel control unit 118 controls the lens driving unit 116 in accordance with a drive control signal input from the focus adjustment device 191 (described later) of the imaging device 100.
- the drive control signal is a control signal for driving the AF lens 112 in the optical axis direction.
- the lens barrel control unit 118 changes the number of steps of the pulse voltage output to the lens driving unit 116, for example, according to the drive control signal.
- the lens barrel control unit 118 outputs the position (focus position) of the AF lens 112 in the lens barrel 111 to a focus adjusting device 191 described later based on a signal corresponding to the movement amount of the AF lens 112.
- the lens barrel control unit 118 calculates the movement amount (position) of the AF lens 112 in the lens barrel 111 by, for example, integrating signals corresponding to the movement amount of the AF lens 112 according to the movement direction. May be.
- the lens driving unit 116 drives the AF lens 112 according to the control of the lens barrel control unit 118.
- the lens driving unit 116 moves the AF lens 112 in the lens barrel 111 in the optical axis direction.
- the imaging apparatus 100 includes an imaging unit 110, an image processing unit 140, a display unit 150, a buffer memory unit 130, an operation unit 180, a storage unit 160, a CPU 190, a communication unit 170, and a focus adjustment device 191. Is provided.
- the imaging unit 110 includes an imaging element 119 and an A / D (Analog / Digital) conversion unit 120.
- the imaging unit 110 is controlled by the CPU 190 in accordance with the set imaging conditions (for example, aperture value, exposure value, etc.).
- the image sensor 119 includes a photoelectric conversion surface.
- the image sensor 119 converts an optical image formed on the photoelectric conversion surface by the lens barrel 111 (optical system) into an electrical signal and outputs the electrical signal to the A / D converter 120.
- the image sensor 119 may be configured by, for example, a CMOS (Complementary Metal Oxide Semiconductor). Further, the image sensor 119 may convert an optical image into an electric signal (cut out) for a partial region of the photoelectric conversion surface.
- CMOS Complementary Metal Oxide Semiconductor
- the image sensor 119 stores an image obtained when a shooting instruction is received via the operation unit 180 in the storage medium 200 via the A / D conversion unit 120.
- the A / D conversion unit 120 is connected to the focus adjustment device 191 and the display unit 150 using continuously obtained images as through images. Output via.
- the A / D converter 120 digitizes the electrical signal converted by the image sensor 119. Then, the A / D conversion unit 120 outputs the image that has become a digital signal to the buffer memory unit 130.
- the operation unit 180 includes, for example, a power switch, a shutter button, a multi-selector (cross key), or other operation keys.
- the operation unit 180 receives a user operation input when operated by the user. Then, the operation unit 180 outputs a signal corresponding to the operation input to the CPU 190.
- the image processing unit 140 refers to the image processing conditions stored in the storage unit 160 and performs image processing on the image temporarily stored in the buffer memory unit 130.
- the processed image is stored in the storage medium 200 via the communication unit 170.
- the image obtained by the imaging unit 110, the operation screen, and the like are displayed.
- An example of the display unit 150 is a liquid crystal display.
- the buffer memory unit 130 temporarily stores the image captured by the imaging unit 110.
- the CPU 190 stores the determination condition referred to when the scene is determined. Furthermore, the storage unit 160 stores imaging conditions associated with each scene determined by the scene determination.
- the CPU 190 controls the imaging unit 110 according to the set imaging conditions (for example, aperture value, exposure value, etc.). Further, the CPU 190 causes the image processing unit 140 to perform image processing as a still image or a moving image based on the “signal according to the operation input” input from the operation unit 180.
- the set imaging conditions for example, aperture value, exposure value, etc.
- the communication unit 170 is connected to a removable storage medium 200 such as a card memory.
- the communication unit 170 writes, reads, or erases information (image data or the like) on the storage medium 200.
- the storage medium 200 is a storage unit that is detachably connected to the imaging apparatus 100 and stores information (image data and the like). Note that the storage medium 200 may be integrated with the imaging device 100.
- the focus adjustment device 191 detects the edge of the subject image from the image generated based on the electrical signal output from the image sensor 119. Then, the focus adjusting device 191 analyzes the color shift caused by the longitudinal chromatic aberration that occurs at the detected edge.
- the axial chromatic aberration is a characteristic that the focal length of the lens differs depending on the wavelength (color) of incident light.
- the focus adjustment device 191 detects a defocus (defocus) feature quantity based on the color shift analysis result. Further, the focus adjustment device 191 generates a drive control signal based on the detected defocus feature value so as to focus on the subject image, and sends this drive control signal to the lens barrel control unit 118 of the lens barrel 111. Output (focus drive).
- the defocus feature amount is a direction index, a defocus amount, a movement amount necessary for focus adjustment of the AF lens 112 (hereinafter referred to as “focus shift amount”), and a history thereof.
- the direction index means that the subject is focused on the near point side (hereinafter referred to as “front pin”), or that the subject is focused on the far point side (hereinafter referred to as “rear pin”). ).
- the amount of focus shift increases as the defocus amount increases.
- the focus shift amount may be expressed by, for example, the number of steps of the pulse voltage output from the lens barrel control unit 118 to the lens driving unit 116 in response to the drive control signal.
- the evaluation value for detecting the direction index and the defocus amount includes Edge (Difference (hereinafter referred to as “Ed”), a defocus amount reference value (Width of Subtraction, hereinafter referred to as “Wd”), and a line spreading function. (Line Spread Function, hereinafter referred to as “LSF”).
- FIG. 2 the subject at a position away from the AF lens 112 by the subject distance, the AF lens 112, the imaging surface of the image sensor 119 (shown as “imaging surface” in FIG. 2), and the circle of confusion.
- the AF lens 112 collects light incident from the subject and forms a subject image on the photoelectric conversion surface (imaging surface) of the image sensor 119.
- a circle of confusion is formed on the imaging surface of the imaging device 119 by the point image included in the subject image.
- FIG. 3 shows the focus of red light (hereinafter referred to as “R channel” or “R”), which is a component of light incident on the AF lens 112, and green light (hereinafter referred to as “G channel” or “G”). ) And the position of the blue light (hereinafter referred to as “B channel” or “B”) on the optical axis.
- FIG. 3 shows that the position where the light is imaged (imaging plane) varies depending on the wavelength of the light.
- imaging plane The reason why the image forming plane varies depending on the wavelength of light is that, for example, the refractive index of the lens varies depending on the wavelength of light.
- the refractive index of a lens may differ for every lens as a characteristic intrinsic
- FIG. 4 shows the relationship between the wavelength of light incident on the AF lens 112 and the position of the image plane on the optical axis.
- the horizontal axis indicates the wavelength of light incident on the AF lens 112.
- the vertical axis indicates the position of the image plane, and the larger the value, the farther away from the AF lens 112. As described above, the position of the image formation plane becomes farther from the AF lens 112 as the wavelength of the incident light is longer.
- the AF lens 112 is optically designed so that the color shift on the imaging surface (imaging surface) is minimized in a focused state. For this reason, in the out-of-focus state, the color misregistration becomes larger than in the in-focus state.
- the color shift is known to change depending on the lens position (focus position, zoom position) or other factors.
- the radius of confusion circle increases as the amount of defocus (out of focus) increases.
- the color arrangement in the circle of confusion also changes depending on the lens position (focus position).
- FIG. 5A, 5B, and 5C show circles of confusion in the front pin state.
- FIG. 5A shows the relationship between the AF lens 112, the imaging surface of the imaging device 119, and the circle of confusion.
- the light incident on the AF lens 112 is refracted (spectral) from the outside in the order of R, G, and B due to axial chromatic aberration, guided to the imaging surface of the image sensor 119, and a circle of confusion on the imaging surface.
- the circle of confusion may be expressed by a point spread function (Point Spread Function, hereinafter referred to as “PSF”).
- PSF Point Spread Function
- FIG. 5B shows a circle of confusion formed on the imaging surface.
- the circle of confusion in the front pin state is formed in the order of R, G, and B from the outside due to axial chromatic aberration, and thus is formed in the order of red, cyan, and white from the outside.
- FIG. 5C is a diagram in which the relationship between the radial position from the center of the circle of confusion toward the outer periphery and the color component amount (color intensity) is represented by the color component amount on the vertical axis (hereinafter referred to as “gradation sectional view”). It is indicated by).
- FIG. 5C shows that the gradient (gradient) of the color component amount in the radial direction from the center of the circle of confusion to the outer circumference is the gentlest R gradient and the steep B gradient in the front pin state. Yes.
- the circle of confusion in the front pin state is formed in the order of white, cyan, blue, yellow, yellow-red, and red in the radial direction from the center toward the outer periphery.
- FIG. 6A and 6B show a circle of confusion in the rear pin state.
- FIG. 6A shows the relationship between the AF lens 112, the imaging surface of the imaging device 119, and the circle of confusion.
- the light incident on the AF lens 112 is refracted (spectral) from the outside in the order of B, G, and R between the focal point on the optical axis and the image sensor 119 due to axial chromatic aberration, and is incident on the image pickup surface of the image sensor 119. Guided to form a circle of confusion on the imaging surface.
- FIG. 6B shows a circle of confusion formed on the imaging surface.
- the circle of confusion in the rear pin state is formed in the order of white, yellow, and blue (border) in the radial direction from the center to the outer periphery because of the axial chromatic aberration in the order of B, G, and R from the outer periphery.
- the focus adjustment device 191 can detect the in-focus state and the out-of-focus state based on the formation of the color of the circle of confusion.
- FIG. 7 shows an example of a captured image.
- a state where a black and white edge chart is captured will be described.
- the following description will be made without considering the lateral chromatic aberration and lens flare, but the gist of the present invention remains unchanged even if these are not taken into consideration.
- the image 10 (monochrome edge chart) imaged by the imaging unit 110 is composed of a black area 11 and a white area 12.
- the black area 11 is rectangular and is in the center of the white area 12.
- an axis that crosses both the black region 11 and the white region 12 in the horizontal direction is defined as an axis D.
- the axis D is an axis orthogonal to the boundary (knife edge) between the black region 11 and the white region 12.
- a boundary where the color changes from white to black on the axis D is defined as an edge position 14.
- a boundary where the color changes from black to white on the axis D is defined as an edge position 13.
- the position on the axis D may be expressed in units of pixels.
- the color component amount of each color channel is expressed by a value “0 to 250” (8 bits). The larger the value of the color component amount of each color channel, the darker the color.
- the color component amount of each color channel may be expressed by a value “0 to 255” (8 bits).
- the color is white at the “position on axis D” where the color component amounts of R, G, and B are all “250”. On the other hand, the color is black at the “position on axis D” where the color component amounts of R, G, and B are all “0”.
- FIG. 8A shows a gradation cross-sectional view in a state where the G channel is in focus (in focus).
- the R channel is the front pin and the B channel is the rear pin.
- the white region 12 side of the edge position 13 becomes green.
- the black region 11 side of the edge position 13 is magenta.
- FIG. 8B shows a gradation cross-sectional view in a state in which focusing is performed on the R channel.
- the G channel is the rear pin, and the gradient (gradient) of the G channel is steeper than the gradient of the B channel.
- FIG. 8C shows a gradation cross-sectional view in a state in which focusing is performed on the B channel. In this state, the G channel is the front pin, and the gradient (gradient) of the G channel is steeper than the gradient of the R channel.
- the focus adjustment device 191 calculates an evaluation value from the gradient of the color channel in the gradation cross-sectional view. In this way, the focus adjustment device 191 can detect the in-focus state and the out-of-focus state.
- 9A and 9B show the color component amounts in the vicinity of the edge position 13 (see FIG. 7) as gradation cross-sectional views.
- the vertical axis indicates the amount of color components for each of R, G, and B (color channel).
- the horizontal axis indicates the position on the axis D (see FIG. 7). Since human eyes have high sensitivity to the G channel, here, as an example, a case where the subject image is focused (focused) on the G channel will be described.
- FIG. 9A shows the relationship between the edge of the R channel in the front pin state, the edge of the R channel in the rear pin state, and the edge of the G channel in the focused state.
- the gradient (tilt) of the R channel in the front pin state is gentler than the gradient of the G channel in the focused state.
- the gradient (gradient) of the R channel in the rear pin state is steeper than the gradient of the G channel in the focused state.
- a section on the right side of the intersection of lines indicating the color component amounts of the R channel and the G channel and having a gradient in the line indicating the G channel is referred to as a section “ ⁇ 1”.
- a section on the left side of the intersection of the lines indicating the color component amount and having a gradient in the line indicating the G channel is defined as a section “ ⁇ 2”.
- the length of these sections may be expressed by the number of pixels.
- the edge of the R channel is in the front pin state or the back pin state according to the comparison result of Ed based on the R channel color component amount and the G channel color component amount (hereinafter referred to as “EdRG”) and a threshold value. Is detected (determined) as follows.
- the value “1” in equations 1 and 2 is a threshold value.
- ( ⁇ (R / G)) in Equations 1 and 2 is a sum obtained by adding the value obtained by dividing the R channel color component amount by the G channel color component amount at the same position in the image over the section “ ⁇ 1”. is there.
- a value obtained by dividing the sum by the length of the section “ ⁇ 1” is “EdRG”.
- Formula 1 is materialized, EdRG shows that the edge of R channel is in a back pin state.
- Expression 2 is satisfied, EdRG indicates that the edge of the R channel is in the front pin state (direction index). The difference between EdRG and the threshold “1” indicates the defocus amount.
- the front pin and the rear pin may be detected by using the “ratio” of the color component amounts as in Expressions 1 and 2.
- the SN ratio Signal to Noise ratio
- the value “0” in equations 3 and 4 is a threshold value.
- ( ⁇ (GR)) in Equations 3 and 4 is a sum obtained by adding a value obtained by subtracting the color component amount of the G channel at the same position in the image from the color component amount of the R channel over the section “ ⁇ 2”. is there. Further, the value obtained by dividing the sum by the length of the section “ ⁇ 2” is “EdRG”. And when Formula 3 is materialized, EdRG shows that the edge of R channel is in a back pin state. On the other hand, when Expression 4 is satisfied, EdRG indicates that the edge of the R channel is in the front pin state (direction index). The difference between EdRG and the threshold “0” indicates the defocus amount.
- FIG. 9B shows the relationship between the edge of the B channel in the front pin state, the edge of the B channel in the rear pin state, and the edge of the G channel in the focused state.
- the gradient (tilt) of the B channel in the front pin state is gentler than the gradient of the G channel in the focused state.
- the gradient (gradient) of the B channel in the rear pin state is steeper than the gradient of the G channel in the focused state.
- a section on the right side of the intersection of the lines indicating the color component amounts of the B channel and the G channel and having a gradient in the line indicating the G channel is referred to as a section “ ⁇ 3”.
- a section on the left side of the intersection of the lines indicating the color component amounts and having a gradient in the line indicating the G channel is defined as a section “ ⁇ 4”.
- EdBG Ed based on the color component amount of the B channel and the color component amount of the G channel and a predetermined threshold
- the edge of the B channel is in the front pin state or the rear pin. The state is detected as follows.
- EdBG ( ⁇ (B / G)) / ( ⁇ 3)> 1 Equation 5
- EdBG ( ⁇ (B / G)) / ( ⁇ 3) ⁇ 1 Equation 6
- the value “1” in Equations 5 and 6 is a threshold value.
- ( ⁇ (B / G)) in Equations 5 and 6 is a sum obtained by adding the value obtained by dividing the B channel color component amount by the G channel color component amount at the same position in the image over the section “ ⁇ 3”.
- a value obtained by dividing the sum by the length of the section “ ⁇ 3” is “EdBG”.
- the front pin and the rear pin may be detected by using the “ratio” of the color component amounts as in equations 5 and 6.
- the SN ratio between the color component amount and noise is small. Therefore, in this case, it is more advantageous for detecting the in-focus state to detect the front pin and the rear pin by the “difference” of the color component amount instead of the “ratio” of the color component amount.
- the value “0” in Equations 7 and 8 is a threshold value.
- ( ⁇ (GB)) in Expressions 7 and 8 is a sum obtained by adding the value obtained by subtracting the G channel color component amount at the same position in the image from the B channel color component amount over the section “ ⁇ 4”.
- a value obtained by dividing the sum by the length of the section “ ⁇ 4” is “EdBG”. If Equation 7 holds, EdBG indicates that the edge of the B channel is in the rear pin state. On the other hand, when Expression 8 is satisfied, EdBG indicates that the edge of the B channel is in the front pin state (direction index). The difference between EdBG and the threshold “0” indicates the defocus amount.
- the focus adjustment device 191 (see FIG. 1) can detect the in-focus state and the out-of-focus state based on the evaluation value Ed.
- the focus adjustment device 191 may detect the front pin and the rear pin based on the “ratio” of the color component amounts in a section where the color component amount is small. Further, the focus adjustment device 191 may detect the front pin and the rear pin based on the “difference” in the color component amount for the section where the color component amount is large.
- the subject image is focused on the G channel, but the focus adjustment device 191 detects (determines) the front pin state and the rear pin state by focusing on the subject image on the R channel or the B channel. May be.
- FIG. 10A and 10B show a “color component amount difference” in the vicinity of the edge position 13 (see FIG. 7).
- FIG. 10A shows the relationship between the position on the axis D and the difference between the R channel color component amount and the G channel color component amount.
- the horizontal axis indicates the position on the axis D (see FIG. 7).
- Wd is a value indicating the defocus amount (defocus amount reference value). As shown in FIG. 10A, Wd is indicated by the distance between peaks of the waveform of the color component amount difference. Wd is expressed by Equation 9 using the positions X1 and X2 on the axis D.
- the positions X1 and X2 are positions where the waveform of the color component amount difference (color component amount ratio) shows a peak in FIG. 10A. And the intersection of a solid line and a broken line respond
- the color component amount difference shows the maximum value “max (RG)”, and at position X1, the color component amount difference shows the minimum value “min (RG)”.
- the polarity (positive / negative) of the waveform indicates that the R channel is in the rear pin state (direction indicator). This is because when the R channel is in the back pin state, as shown in FIG. 9A, the color component amount of the R channel is equal to or greater than the color component amount of the G channel on the right side of FIG. 9A, and on the left side of FIG. This is because the color component amount of the R channel is equal to or less than the color component amount of the G channel.
- the color component amount difference shows the maximum value “max (RG)” at the position X1 in FIG. 10A and the color component amount difference shows the minimum value “min (RG)” at the position X2.
- the polarity (positive / negative) of the waveform indicates that the R channel is in the front pin state (direction indicator). This is because when the R channel is in the front pin state, as shown in FIG. 9A, on the right side of FIG. 9A, the color component amount of the R channel is less than the color component amount of the G channel, and on the left side of FIG. This is because the R channel color component amount is equal to or greater than the G channel color component amount.
- FIG. 10B shows a waveform when approaching the in-focus state as compared with FIG. 10A.
- the value of Wd decreases. Note that the difference between the maximum value “max (RG)” of the color component amount difference and the minimum value “min (RG)” of the color component amount difference varies depending on the “blur degree”. Also good.
- the focus adjustment device 191 (see FIG. 1) can detect the in-focus state and the out-of-focus state based on the evaluation value Wd. Further, the focus adjustment device 191 can detect the front pin state or the rear pin state based on the polarity of the color component amount difference.
- MTF ModulationMoTransfer Function
- the MTF of the system is the product of the MTF of the imaging system (optical system) and the MTF of the sensor (imaging device) system.
- the focus adjustment device can detect (determine) the in-focus state based on the MTF of the system. For example, the focus adjustment device may detect the in-focus state by evaluating the MTF of the edge of the subject image. Note that the focus adjustment device detects the in-focus state by further considering image processing processes (for example, circuit response characteristics, demosaic, noise reduction, edge enhancement, etc.) subsequent to the imaging system and the sensor system. Also good.
- MTF and LSF are reversible via Fourier transform when the light incident on the optical system is incoherent light. Also, the LSF can be easily calculated from the edges included in the image (see, for example, the “ISO12233” standard).
- the LSF may be calculated by differentiating an ESF (Edge Spread Function).
- the LSF may be calculated based on a difference in pixel values (for example, color component amounts) between adjacent pixels.
- the MTF may be calculated by subjecting the LSF to discrete Fourier transform (Discrete Fourier Transform, DFT).
- the focus adjusting device 191 may detect (determine) the in-focus state based on LSF as an evaluation value in addition to the above-described Ed and Wd.
- the focus adjustment device 191 is based on “LSF profile (data) for each color channel” created in advance according to a known image processing process. ) May be detected as follows.
- 11A, 11B and 11C show the LSF of the G channel according to the “blurring degree” of the focused state.
- the LSF is expressed by a standard deviation ⁇ (statistic) that is one of the feature quantities of the LSF.
- ⁇ statistic
- the LSF may be expressed by a half-value width or a peak value, which is one of the feature amounts of the LSF.
- FIG. 11A shows the LSF in the focused state.
- FIG. 11B shows LSF in an out-of-focus (small blur) state.
- FIG. 11C shows the LSF in an out-of-focus (large blurred) state.
- FIG. 12A, 12B, and 12C show the LSF for each color channel and the “blurred degree” in the focused state.
- FIG. 12A shows the LSF in the focused state for each color channel (R, G, B).
- the magnitude relationship of the standard deviation ⁇ for each color channel may be different for each lens as a characteristic characteristic of the lens.
- the standard deviation “ ⁇ R” of the R channel LSF in the focused state the standard deviation “ ⁇ G” of the L channel LSF in the focused state
- ⁇ B standard deviation “ ⁇ B” of the B channel LSF in the focused state.
- FIG. 12B shows the LSF in the front pin state for each color channel (R, G, B).
- the rightmost side of FIG. 12B shows the relationship between the position on the axis D and the color component amount in the front pin state.
- the G channel is also shown in the front pin state, and therefore the gradients of all R, G, and B are gentle.
- the rightmost diagram in FIG. 12B shows that in the front pin state, the gradient (gradient) of the color component amount at the edge has the slowest gradient of R and the steep gradient of B. .
- the slope of R is the slowest and the slope of B is the steepest. Therefore, in the front pin state, the standard deviation “ ⁇ Rdf” of the R channel LSF and the standard of the G channel LSF
- the deviation “ ⁇ Gdf” and the standard deviation “ ⁇ Bdf” of the B channel have a magnitude relationship of “ ⁇ Rdf> ⁇ Gdf> ⁇ Bdf”. More specifically, there is a magnitude relationship of “ ⁇ Rdf> K1 ⁇ ⁇ Gdf” and “ ⁇ Bdf ⁇ K2 ⁇ ⁇ Gdf”.
- FIG. 12C shows the LSF in the rear pin state for each color channel (R, G, B). Also, the rightmost side of FIG. 12C shows the relationship between the position on the axis D and the color component amount in the rear pin state.
- the G channel is also shown in the rear pin state, and therefore the gradients of all R, G, and B are gentle.
- the rightmost diagram in FIG. 12C shows that in the rear pin state, the gradient (gradient) of the color component amount at the edge is that the gradient of B is the gentlest and the gradient of R is the steepest. Yes.
- the standard deviation “ ⁇ Rdb” of the R channel LSF and the standard of the G channel LSF have a magnitude relationship of “ ⁇ Bdb> ⁇ Gdb> ⁇ Rdb”. More specifically, there is a magnitude relationship of “ ⁇ Rdb ⁇ K1 ⁇ ⁇ Gdb” and “ ⁇ Bdb> K2 ⁇ ⁇ Gdb”.
- FIG. 13 shows the relationship between the LSF profile (data) for each color channel and the degree of blur.
- the horizontal axis indicates the degree of blur.
- LSF profiles are shown in the order of G channel, R channel, and B channel.
- the number of the subscript of the standard deviation is larger as the LSF has a larger absolute value (defocus amount) of the degree of blur.
- the LSF profile in the focused state is shown at the center position “0” on the horizontal axis.
- the right side from the center position “0” on the horizontal axis is an area where the edge is in the front pin state (front pin area), and the absolute value of the degree of blur increases as the distance from the center position “0” increases.
- the left side from the center position “0” on the horizontal axis is an area where the edge is in the rear pin state (rear pin area), and the absolute value of the degree of blur increases as the distance from the center position “0” increases.
- the LSF profile for each color channel is created before the imaging apparatus 100 captures an image of a subject, and is stored in advance in the storage unit 160 for each degree of blur.
- the LSF profile for each color channel is expressed in standard deviation, for example, and stored in advance in the storage unit 160.
- the LSF profile may be expressed in half width or peak value and stored in advance in the storage unit 160.
- the “blurring degree” on the horizontal axis may be expressed as a subject distance (depth).
- the LSF profile for each color channel includes, in addition to the “blurring degree”, information indicating the zoom magnification, subject distance, position with respect to the angle of view, and the horizontal (H) or vertical (V) direction in the captured image.
- Information indicating the noise reduction processing method, or information indicating the edge enhancement method may be stored in advance in the storage unit 160.
- FIG. 14 shows the relationship between the function value using the difference in standard deviation of LSF as a parameter and the subject distance (relative distance map, distance function).
- the vertical axis indicates the difference in standard deviation of LSF.
- the horizontal axis indicates the subject distance.
- the subject distance (Depth) is expressed by Expression 10.
- ⁇ Rd is the standard deviation of LSF LSF.
- ⁇ Gd is a standard deviation of the LSF of the G channel.
- ⁇ Bd is the standard deviation of the B channel LSF.
- F ( ⁇ Rd ⁇ Gd) is a function having a difference of the standard deviation “ ⁇ Rd ⁇ Gd” as a parameter, which is a negative value in the rear pin state and a positive value in the front pin state.
- F1 ( ⁇ Bd ⁇ Gd) is a function having a standard deviation difference “ ⁇ Bd ⁇ Gd” as a parameter, and is a function having a positive value in the rear pin state and a negative value in the front pin state.
- the function G is a function having “F ( ⁇ Rd ⁇ Gd) ⁇ F1 ( ⁇ Bd ⁇ Gd)” as a parameter, and is a function that selects at least one of the value of the function F or the value of the function F1.
- the function G is a function used to calculate the subject distance (Depth) by selecting the value of the function F in the front pin state and selecting the value of the function F1 in the rear pin state.
- the LSF may be calculated for only one color channel (single color).
- the focus adjustment device 191 may detect the in-focus state and the out-of-focus state by matching the LSF of the imaged color channel with the LSF profile of one color channel. For example, when the captured LSF matches (matches) the LSF profile indicating the degree of blur “0”, the focus adjustment device 191 may determine that the edge is in focus.
- the focus adjustment device 191 (see FIG. 1) can detect the in-focus state and the out-of-focus state based on the standard deviation or half-value width of the evaluation value LSF. Further, the focus adjustment device 191 can detect the front pin state or the rear pin state by comparing the standard deviation or the half value width of the evaluation value LSF for each color channel.
- the focus adjustment device 191 includes an edge detection unit 192, a distribution detection unit 193, and a control unit 194.
- the edge detection unit 192 detects the edge of the subject image for each color component (R channel, G channel, B channel) constituting the image output by the imaging unit 110.
- FIG. 15A, 15B, 15C, and 15D each show an example of a captured image, an edge image extracted from the captured image, and an image that shows the result of the logical product of the two edge images.
- FIG. 15A illustrates an example of a part (partial image) of an image captured by the imaging unit 110.
- a frame indicated by a broken line indicates a detection area for the edge detection unit 192 to detect an edge.
- the detection region may be a region that extends only on one side of the edge. Further, the position of the detection region in the partial image may be determined in advance.
- the edge detection unit 192 extracts (detects) an edge image of the subject image captured in the detection area (a mask image masked other than the edges) for each color component from the partial image.
- FIG. 15B shows an extracted R channel edge image.
- FIG. 15C shows an extracted edge image of the G channel.
- FIG. 15D shows an image obtained as a result of calculating a logical product (AND) of the R channel edge image and the G channel edge image.
- the procedure for extracting (generating) these images will be described below.
- FIG. 16 is a flowchart showing the operation of the edge detection unit 192.
- the edge detection unit 192 extracts (generates) an edge image for each color channel using a Laplacian filter or the like from a rough image (for example, QVGA resolution) captured by the imaging unit 110 (step Sa1).
- a rough image for example, QVGA resolution
- FIG. 17 shows an example of a gradation sectional view of the color component amount in the vicinity of the edge position 13 and an example of calculation using a Laplacian filter.
- a case where an edge extending in the vertical direction of an image is extracted (generated) will be described.
- the edge detection unit 192 adds a value obtained by multiplying each color component amount by a coefficient of a Laplacian filter for each color component. Then, the edge detection unit 192 compares the addition result with a predetermined threshold value Ek1 for each color component.
- the edge detection unit 192 compares the addition result “( ⁇ 1) ⁇ R1 + 8 ⁇ R2 + ( ⁇ 1) ⁇ R3” for the R channel with the threshold value Ek1. Since the addition result for the R channel is larger than the threshold value Ek1, the edge detection unit 192 generates an R channel edge image using the R channel edge as an effective edge.
- the edge detection unit 192 compares the addition result “( ⁇ 1) ⁇ G1 + 8 ⁇ G2 + ( ⁇ 1) ⁇ G3” for the G channel with the threshold value Ek1. Since the addition result for the G channel is larger than the threshold value Ek1, the edge detection unit 192 generates a G channel edge image using the G channel edge as an effective edge. Similarly, as a result of comparing the addition result for the B channel with the threshold value Ek1, the addition result for the B channel is equal to or less than the threshold value Ek1, so that the edge detection unit 192 sets the edge of the B channel as an effective edge, Do not generate channel edge images.
- the edge detection unit 192 sets the R channel color component amount in the edge image to the same level as the G channel color component amount. For example, the edge detection unit 192 uses a value obtained by dividing the color component amount of the G channel by the color component amount of the R channel as a gain (correction magnification). The edge detection unit 192 multiplies the R channel color component amount by a gain. In this way, the edge detection unit 192 generates an edge image (see FIGS. 15B and 15C).
- the edge detection unit 192 extracts edges (edge pairs) of two or more colors that are correlated with respect to the position. Specifically, the edge detection unit 192 extracts the color component amount of the R channel edge and the color component amount of the G channel edge for the same pixel. Then, the edge detection unit 192 sets the color component amount logical product (AND) of these edge pairs as a common edge image (step Sa2).
- the edge detection unit 192 extracts, as a common edge, the logical product of the color component amounts of edges (edge pairs, color pairs) of adjacent R channels and G channels within “ ⁇ 1 pixel”, even if they are not edges of the same pixel. It may be (thickening). Further, the edge detection unit 192 may extract a common edge based on the logical product of the color component amounts of edges that are equal to or longer than a predetermined length L.
- the edge detection unit 192 causes the storage unit 160 to store the common edges of the R channel and the G channel that constitute the common edge (step Sa3). In this manner, the edge detection unit 192 generates an edge image (see FIG. 15D) from which common edges between the R channel edge and the G channel edge are extracted.
- the image captured by the imaging unit 110 is divided into grid-like blocks.
- the distribution detection unit 193 detects the in-focus state and the out-of-focus state (direction index and defocus amount) of the common edge for each of the divided blocks.
- FIG. 18 shows an example of a depth map.
- the depth map is a map showing the distribution of the in-focus state and the out-of-focus state (the degree of blur) in the captured image in units of blocks (partial images).
- an image captured by the imaging unit 110 is divided into “4 ⁇ 4” blocks, and an edge image in one of the blocks is described as an edge image illustrated in FIG. 15D.
- the distribution detector 193 detects the distribution of the in-focus state and the out-of-focus state of the common edge based on at least one of Ed, Wd, or LSF of the common edge extracted from the edge image. Based on Ed of the three common edges extracted from the edge image of FIG. 15D, the distribution detection unit 193 detects the in-focus state and the out-of-focus state (degree of blur) of the common edge as follows.
- the lengths of the three common edges extracted in the edge image of FIG. 15D are L1, L2, and L3, respectively.
- the distribution detection unit 193 calculates Expression 11.
- ⁇ is the length of the section orthogonal to the edge.
- ⁇ is the length of the section ⁇ 1 or ⁇ 2.
- the distribution detection unit 193 sets EdRG as a value obtained by dividing Equation 11 “ ⁇ (L (( ⁇ (R / G)) / ⁇ ))” by “L1 + L2 + L3”.
- the distribution detection unit 193 determines that the common edge is in the rear pin state.
- Expression 2 the distribution detection unit 193 determines that the common edge is in the front pin state.
- a value surrounded by a square frame indicates a “blurring degree” in the block.
- a block having a negative value indicates that the edge imaged in the block is in the back pin state.
- a block having a positive value indicates that the edge imaged in that block is in the front pin state.
- the distribution detection unit 193 may further subdivide one block divided into “4 ⁇ 4” into small blocks (segments) of “2 ⁇ 2”. Then, the distribution detection unit 193 may detect the in-focus state and the out-of-focus state of the common edge for those small blocks based on the blur degree histogram. For example, the distribution detecting unit 193 may interpolate the “blurring degree” of the small block between the value “2” and the value “8” with a histogram to obtain the value “4”. This interpolated value is an example. When an effective edge having a sufficient color component amount (power) is not detected, an undetectable block in which the direction index and the defocus amount cannot be detected may remain in the depth map.
- the distribution detection unit 193 detects the distribution of the in-focus state and the out-of-focus state of the common edge based on Ed, and creates a depth map. Similarly, the distribution detection unit 193 may detect the distribution of the in-focus state and the out-of-focus state of the common edge based on the above-described Wd, and create a depth map.
- the distribution detection unit 193 may detect the distribution of the in-focus state and the out-of-focus state of the common edge based on the above-described LSF, and create a depth map. Specifically, the distribution detection unit 193 divides the image captured by the imaging unit 110 into grid-like blocks, and extracts an edge image for each block (see FIG. 15D). Then, the distribution detection unit 193 calculates the LSF of the edge extracted from the edge image by the procedure described above in the description of the evaluation value LSF. Further, the distribution detection unit 193 detects the “blurring degree” for each block according to the procedure described in the description of the evaluation value LSF based on the LSF profile and the LSF of the imaged edge, thereby obtaining a depth map. Create
- the distribution detection unit 193 may detect the distribution of the in-focus state and the out-of-focus state of the common edge by combining the evaluation values Ed, Wd, and LSF. For example, the distribution detection unit 193 detects the distribution of the in-focus state and the out-of-focus state of the common edge based on Ed, and then detects the common edge based on the LSF for the block in which the effective edge is not detected. The distribution of the focused state and the out-of-focus state of the edge may be detected complementarily.
- the control unit 194 selects a block in which the subject image is captured from the distribution based on the in-focus state distribution and the out-of-focus state distribution detected by the distribution detection unit 193. Then, the control unit 194 moves the AF lens 112 so as to focus on the edge of the subject image. In addition, the control unit 194 determines whether a shooting instruction for storing the image captured by the imaging unit 110 in the storage medium 200 is output from the operation unit 180 to the CPU 190. Further, the control unit 194 determines whether or not a focus instruction for focusing on the subject image is output from the operation unit 180 to the CPU 190. Based on these determinations, the control unit 194 changes the focus drive procedure as described later.
- FIG. 19 is a flowchart showing the procedure of focusing drive of the focus adjustment device 191 in the tracking operation for tracking the subject image.
- the CPU 190 of the imaging device 100 sets the focus adjustment device 191 to the tracking mode so as to execute a tracking operation for tracking the subject image (step S1). Further, the CPU 190 designates (sets) the subject image to be tracked in the focus adjustment device 191 (step S2). These settings may be executed based on a user operation input received by the operation unit 180.
- the control unit 194 of the focus adjustment device 191 executes the first focusing drive (step S3).
- the focusing method in this focusing drive may be selected as appropriate.
- the focusing method may be a contrast AF (Auto Focus) method or another focusing method (such as a phase difference detection method).
- the image sensor 119 further converts an optical image formed on the photoelectric conversion surface into an electric signal.
- the image pick-up element 119 outputs the electrical signal obtained here to the A / D conversion part 120 as a through image (step S4).
- the resolution of the through image output from the image sensor 119 may be low because a part of the data is thinned out.
- the imaging apparatus 100 and the focus adjustment apparatus 191 perform steps S5 to S7 and steps S8 to S16 described below in parallel. Further, the imaging device 100 and the focus adjustment device 191 may repeatedly execute steps S4 to S17.
- the CPU 190 controls the imaging unit 110 to execute automatic exposure (Automatic Exposure, AE) processing and debayer (color interpolation) processing. Further, the image processing unit 140 acquires a through image and executes image processing (step S5). For example, the image processing unit 140 performs, as image processing, white balance (White Balance, WB) adjustment processing, noise reduction (Noise) Reduction, NR) processing, magnification chromatic aberration correction processing, and the like on the acquired through image.
- white Balance White Balance, WB
- WB noise reduction
- NR noise reduction
- the image processing unit 140 searches for a tracking target imaged in the tracking region of the acquired through image by image matching processing or the like (step S6). Further, the image processing unit 140 superimposes the tracking frame indicating the tracking region and the through image, and causes the display unit 150 to display the superimposed image (step S7).
- the edge detection unit 192 of the focus adjustment device 191 extracts an imaged edge within a tracking region having a predetermined initial window size (see each of FIGS. 15A, 15B, 15C, and 15D) (step S8). ). Then, the edge detection unit 192 analyzes the edge characteristics in the procedure described above with reference to FIG. 16 (step S9).
- the edge detection unit 192 determines whether there is an edge to be tracked (step S10). If no edge exists (step S10—NO), the processing of the edge detection unit 192 returns to step S8. On the other hand, when an edge exists (step S10—YES), the edge detection unit 192 sets an edge to be tracked (step S11).
- the edge detection unit 192 may set, for example, an edge having a strong power (a large amount of color component) as a tracking target.
- the CPU 190 controls the imaging unit 110 to increase the resolution of the through image output from the imaging element 119.
- the resolution of the through image may be increased.
- the CPU 190 controls the image capturing unit 110 to cut out a part of the optical image formed on the photoelectric conversion surface of the image sensor 119 and convert the image into an electrical signal.
- the data is output to the conversion unit 120 (step S12).
- the edge detection unit 192 analyzes the edge characteristics and stores the evaluation value in the storage unit 160 (step 13). In step 13, the edge detection unit 192 executes the procedure shown in FIG. FIG. 20 shows a flowchart showing an edge characteristic analysis procedure.
- the edge detection unit 192 calculates a gain (magnification) based on the R channel edge image and the G channel edge image, and multiplies the R channel edge by the gain (see FIG. 17 and the like) (steps). Sb1).
- the edge detection unit 192 performs affine transformation or cosine correction, thereby transforming the edge extending obliquely with respect to the image so as to be horizontal with respect to the image (step Sb2). Furthermore, the distribution detection unit 193 of the focus adjustment device 191 calculates an evaluation value for each grid-like block in the image captured by the imaging unit 110 based on the interval ⁇ around the edge (see, for example, Equation 1). .
- the distribution detection unit 193 calculates each evaluation value for each edge (see FIG. 15D) extracted by the edge detection unit 192, and averages the calculated evaluation value with the length of the edge to be Ed ( For example, see Equation 11) (step Sb4).
- the distribution detection unit 193 calculates at least one of Ed, Wd, and LSF as the evaluation value.
- the distribution detection unit 193 causes the storage unit 160 to store an evaluation value, an edge pair (color pair) for each color channel, a color component amount (color intensity), and a gain as edge characteristics. In this way, the distribution detection unit 193 creates a depth map (see FIG. 18).
- the distribution detection unit 193 determines whether the edge is in the focused state, the front pin state, or the rear pin state. (See, for example, equations 1 to 4) (step S14).
- the distribution detection unit 193 determines whether or not the evaluation value Ed exceeds a predetermined first threshold value. Further, the distribution detection unit 193 determines whether or not the difference between the previous Ed and the current Ed is larger than a predetermined second threshold value (step S15).
- control unit 194 executes the contrast scan by limiting the moving direction of the AF lens 112.
- FIG. 21 shows a scan area when the front pin state is determined and a scan area when the rear pin state is determined.
- the middle part of FIG. 21 shows that in “normal contrast scanning”, the AF lens 112 moves in the lens barrel 111 within a predetermined lens movable range (nearest to infinity) while performing contrast scanning. Has been.
- This “normal contrast scan” is executed without being based on the determination result of the front pin state and the rear pin state.
- the lower part of FIG. 21 shows the movement in the contrast scan executed based on the determination results of the front pin state and the rear pin state.
- the AF lens 112 moves in a part of a predetermined lens movable range (nearest to infinity) in the lens barrel 111 while performing contrast scanning.
- the control unit 194 restricts the moving direction of the AF lens 112 from the current position of the AF lens 112 to the region on the closest side. To perform a contrast scan.
- the control unit 194 changes the moving direction of the AF lens 112 from the current position of the AF lens 112 to the infinite region. Limited, perform contrast scan.
- the focus adjustment device 191 can focus on the subject image more quickly than the normal contrast scan.
- the control unit 194 moves the AF lens 112 to a focus position detected by a contrast scan executed based on the determination result of the front pin state and the rear pin state in order to focus on the edge of the subject image (Step S194). S16).
- step S15 when the evaluation value Ed does not exceed the predetermined first threshold and the difference between the previous Ed and the current Ed is not greater than the predetermined second threshold (step S15-NO) )
- step S17 the process of the control unit 194 proceeds to step S17 after executing step S16.
- the control unit 194 determines whether or not an imaging instruction for storing the image captured by the imaging unit 110 in the storage medium 200 is output from the operation unit 180 to the CPU 190 (step S17). For example, when the user operates the shutter button of the operation unit 180, a shooting instruction is output from the operation unit 180 to the CPU 190.
- step S17—NO If no shooting instruction is output from the operation unit 180 to the CPU 190 (step S17—NO), the processing of the control unit 194 returns to step S4. On the other hand, when a shooting instruction is output from the operation unit 180 to the CPU 190 (step S17—YES), the process of the control unit 194 proceeds to step S18.
- FIG. 22A shows an example of the relationship between the position of the AF lens 112 in contrast scanning and the contrast value.
- a region on the near side from the current position of the lens is defined as a near side region.
- the region on the infinite side from the current lens position is defined as the infinite side region.
- the AF lens 112 is controlled by the control unit 194 from a start position (for example, a position in the near side region) to an end position (for example, a position in the infinite side region) in a lens movable range determined in advance in the lens barrel 111.
- a start position for example, a position in the near side region
- an end position for example, a position in the infinite side region
- the lens position at which the contrast value shows a peak is set as the in-focus position.
- the AF lens 112 is in the focus position, the edge of the subject image is in focus.
- FIG. 22B shows the movement of the position of the AF lens 112 in the normal contrast scan (see the middle part of FIG. 21).
- the control unit 194 moves the AF lens 112 while performing a contrast scan over the entire lens movable range in the lens barrel 111.
- the control unit 194 moves the AF lens 112 from the lens position at time t1 to the contrast scan start position from time t1 to time t2 (initial position driving). Then, the control unit 194 moves the AF lens 112 from time t2 to time t3a while performing contrast scan from the start position to the end position (scan driving). Further, the control unit 194 moves the AF lens 112 to the in-focus position where the contrast value shows a peak from time t3a to time t4a (focus position drive).
- control unit 194 causes the entire range of lens movement to be contrast scanned. For this reason, it takes time to move the AF lens 112 to the in-focus position.
- FIG. 22C shows the movement of the position of the AF lens 112 in the contrast scan (see the lower part of FIG. 21) based on the determination of the front pin state and the rear pin state based on the evaluation value.
- the control unit 194 moves the AF lens 112 while performing a contrast scan of a predetermined region of the lens movable range in the lens barrel 111.
- the distribution detection unit 193 detects the distribution of the in-focus state and the out-of-focus state of the common edge as described above.
- the distribution detection unit 193 determines that the edge of the subject image to be tracked is in the front pin state by time t1.
- the control unit 194 moves the AF lens 112 from the lens position at time t1 to the contrast scan start position from time t1 to time t2 (initial position driving).
- the control unit 194 predicts that the focus position is in the near side region. Then, from time t2 to t3b, the control unit 194 moves the AF lens 112 while performing contrast scan from the start position to “lens position at time t1” (scan driving). Further, the control unit 194 moves the AF lens 112 to the in-focus position where the contrast value shows a peak from time t3b to t4b (focus position drive).
- control unit 194 may end the focus drive when the evaluation value Ed converges within a predetermined range in the scan drive. Further, in the scan drive, the control unit 194 may detect the focus state based on the LSF as described above, and end the focus drive.
- control unit 194 calculates the in-focus position by interpolation by temporarily passing through the position where the contrast value showed a peak while performing contrast scanning in scan driving.
- the control unit 194 moves the AF lens 112 to the focus position calculated by interpolation (focus position drive).
- the CPU 190 controls the imaging unit 110 so as to expose the edge that is brought into focus by the shooting AF. Then, the CPU 190 stores (records) the image captured by the imaging unit 110 in the storage medium 200 via the communication unit 170 (step S19).
- the focus adjustment device 191 detects the edge of the subject image for each color component constituting the image including the subject image incident from the lens barrel 111 having the AF lens 112 for performing focus adjustment.
- An edge detection unit 192 is provided.
- the focus adjustment device 191 further includes a distribution detection unit 193 that detects the distribution of the in-focus state and the out-of-focus state in the image based on the edge detected by the edge detection unit 192 for each color component.
- the focus adjustment device 191 further includes a control unit 194 that moves the AF lens 112 to focus on the subject image based on the distribution (depth map) detected by the distribution detection unit 193.
- the control unit 194 moves the AF lens 112 only in the near side region.
- the focusing drive is completed in a short time compared to the case where the normal contrast scan is executed.
- the control unit 194 moves the AF lens 112 only in the infinite side region.
- the focusing drive is completed in a short time compared to the case where the normal contrast scan is executed.
- the focus adjustment device 191 can quickly focus on a subject image that has never been focused.
- the distribution detection unit 193 detects the distribution of the in-focus state and the out-of-focus state based on the gradient of the color component amount of the edge detected for each color component by the edge detection unit 192. Thereby, the focus adjustment device 191 can quickly focus on the subject image based on the gradient of the color component amount of the edge.
- the distribution detection unit 193 is focused on the near point side of the subject based on the ratio or difference of the color component amounts of the edges detected by the edge detection unit 192 for each color component, or is far from the subject. A direction indicator indicating that the camera is focused on the side is detected. At the same time, the distribution detection unit 193 also detects the defocus amount. Accordingly, the focus adjustment device 191 can quickly focus on the subject image without increasing the calculation load based on the ratio or difference of the color component amounts of the edges.
- the distribution detection unit 193 detects the defocus amount based on the edge color component amount ratio or the difference between peaks detected by the edge detection unit 192 for each color component. Accordingly, the focus adjustment device 191 can quickly focus on the subject image without increasing the calculation load based on the ratio of the color component amounts of the edges or the distance between peaks of the difference.
- FIG. 23 shows a flowchart showing the procedure of focusing drive based on LSF.
- the CPU 190 of the imaging apparatus 100 sets an area (central area area) in the center of the through image as an area to be focused (AF) on the edge (step Sd1). This setting may be executed based on a user operation input received by the operation unit 180.
- CPU 190 controls imaging unit 110 to execute exposure control (step Sd2).
- the image processing unit 140 acquires a through image (step Sd3).
- the CPU 190 controls the imaging unit 110 to execute a debayer (color interpolation) process.
- the image processing unit 140 performs image processing on the acquired through image.
- the image processing unit 140 performs white balance (White Balance, WB) adjustment processing and noise reduction (Noise Reduction, NR) processing on the acquired through image (step Sd4).
- the through image acquired by the image processing unit 140 may be an image in which the resolution is reduced by thinning out the electrical signal output by the image sensor 119. In this way, the amount of image processing executed by the image processing unit 140 can be reduced.
- the image processing unit 140 displays the through image subjected to the image processing on the display unit 150 (step Sd5).
- CPU 190 determines whether or not a focus instruction is input to CPU 190 (step Sd6). When the focus instruction is not input (step Sd6-NO), the process of the CPU 190 returns to step Sd3. If a focus instruction has been input (step Sd6-YES), the process of the CPU 190 proceeds to step Sd7.
- the image processing unit 140 acquires a through image at full resolution (step Sd7). Then, the CPU 190 controls the imaging unit 110 to execute a debayer (color interpolation) process. In addition, the image processing unit 140 performs image processing on the acquired through image. For example, the image processing unit 140 performs white balance adjustment processing and noise reduction processing on the acquired through image (step Sd8).
- the edge detection unit 192 extracts an edge image for each color channel with a Laplacian filter or the like in the central area region (step Sb9). Then, the edge detection unit 192 extracts an effective edge whose color component amount (edge power) is a certain amount or more (step Sb10). Further, the edge detection unit 192 calculates the LSF of the extracted edge with respect to the horizontal direction of the image (step Sd11).
- the edge detection unit 192 detects the positional correlation for the edge of each color channel (step Sb12). Then, the edge detection unit 192 selects the edges of two or more color channels having a positional correlation, and extracts an edge image (step Sb13).
- an edge image of the R channel and an edge image of the G channel are extracted will be described as an example.
- the edge detection unit 192 thickens the edges of the extracted edge image, and creates a mask image (see FIG. 15D) that masks other than the edges (step Sd14).
- the distribution detection unit 193 divides the mask image into “8 ⁇ 8” blocks. Then, the distribution detection unit 193 refers to the LSF of each block and calculates the average value of the LSF per pixel (step Sd15). Furthermore, the distribution detection unit 193 may perform labeling of the subject image based on the calculated average value. When there are a plurality of labels in the central area, the distribution detection unit 193 determines that there are a plurality of subject distances in the central area (step Sd16).
- the control unit 194 calculates the driving direction of the AF lens 112 and the defocus amount (contrast scan start position) based on the R channel edge and the G channel edge (step Sd17).
- the control unit 194 moves the AF lens 112 to a start position (for example, a position in the near side region in the front pin state).
- the control unit 194 executes contrast scanning while calculating the LSF. While executing the contrast scan, the control unit 194 once passes through the position where the LSF indicates the in-focus state (see FIG. 13), and stops the scan drive (step Sd18).
- control unit 194 calculates a focus position by interpolation, and moves the AF lens 112 to the calculated focus position (step Sd19).
- image processing unit 140 repeatedly acquires through images (step Sd20).
- the image processing unit 140 displays the through image on the display unit 150 (step Sd21).
- the control unit 194 may set a region wider than the central area region as a region to be focused on the edge (AF region).
- the distribution detection unit 193 is focused on the near point side from the subject based on the line spread function (LSF) corresponding to the edge detected for each color component by the edge detection unit 192, or A direction index indicating that the subject is focused on the far point side from the subject is detected. Furthermore, the distribution detection unit 193 also detects the defocus amount. Thereby, the focus adjustment device 191 can quickly focus even when the subject image is a single color, when the area of the subject image is small, or when the spatial frequency of the subject image is low (content tolerance).
- LSF line spread function
- the distribution detection unit 193 detects the defocus amount based on the standard deviation or half-value width of the line spread function (LSF) corresponding to the edge detected for each color component by the edge detection unit 192. Accordingly, the focus adjustment device 191 can quickly focus on the subject image based on the standard deviation or the half-value width of the edge line spread function.
- LSF line spread function
- the focus adjustment device 191 includes an edge detection unit 192 that detects the edge of the subject image for each color component constituting the image including the subject image incident from the lens barrel 111 having the AF lens 112 for performing focus adjustment. Prepare.
- the focus adjustment device 191 further includes a distribution detection unit 193 that calculates an edge line spread function (LSF) detected by the edge detection unit 192 for each color component.
- the focus adjustment device 191 further includes a control unit 194 that moves the AF lens 112 to focus on the subject image based on the line spread function.
- the focus adjustment device 191 can quickly focus even when the subject image is a single color, when the area of the subject image is small, or when the spatial frequency of the subject image is low.
- a second embodiment of the present invention will be described in detail with reference to the drawings.
- the second embodiment is different from the first embodiment in that the distribution detection unit 193 predicts an in-focus position (defocus amount) based on a “defocus-drive pulse table” described later. Only the differences between the second embodiment and the first embodiment will be described below.
- FIG. 24 shows an example of “defocus-drive pulse table”.
- the items (reference) of the defocus-drive pulse table include the lens position, the evaluation value, the number of steps of the drive pulse in the front pin state (hereinafter referred to as “the number of steps in the front pin state”), the rear pin The number of steps of the driving pulse in the state (hereinafter referred to as “the number of steps in the rear pin state”).
- the evaluation value may be any of defocus amounts based on Ed, Wd, and LSF.
- the number of steps of the driving pulse is a driving pulse output from the lens barrel control unit 118 to the lens driving unit 116 in response to the driving control signal.
- the number of steps of the drive pulse is determined in advance based on the structure of the lens barrel control unit 118.
- the number of steps in the front pin state indicates the number of steps necessary to move from the current lens position to the in-focus position in the front pin state.
- the number of steps in the rear pin state indicates the number of steps necessary to move from the current lens position to the in-focus position in the rear pin state.
- the defocus-drive pulse table may be stored in the storage unit 160.
- the distribution detector 193 periodically detects an evaluation value (defocus amount). Then, the distribution detection unit 193 registers the detected evaluation value and the lens position at the time of detection in the item “evaluation value” of the defocus-drive pulse table. Furthermore, the distribution detection unit 193 registers “the number of steps in the front pin state” and “the number of steps in the rear pin state” in the defocus-drive pulse table. These “number of steps in the front pin state” and “number of steps in the rear pin state” are determined in advance based on the structure of the lens barrel control unit 118. The distribution detection unit 193 creates a history by repeating these registration operations.
- control unit 194 limits the moving direction of the AF lens 112 and the contrast scan area based on the defocus-drive pulse table.
- FIG. 25 shows the contrast scan area when the front pin and the degree thereof are determined. At the same time, FIG. 25 also shows the contrast scan area when the rear pin and the degree thereof are determined.
- the control unit 194 focuses on the closest region based on the defocus-drive pulse table (see FIG. 24). Define the forecast range.
- the lens position (focus position) output from the lens barrel control unit 118 is the value “1”
- the evaluation value (defocus amount) detected by the distribution detection unit 193 is the value “1”
- the distribution detection unit It is assumed that the direction indicator detected by 193 indicates the front pin state.
- the control unit 194 determines whether or not the number of steps at the front pin corresponding to the lens position “1” and the evaluation value “1” is registered in the defocus-drive pulse table (see FIG. 24). Refer to In the example shown in FIG. 24, the lens position “1” and the “number of steps at the front pin” corresponding to the evaluation value “1” are registered. Therefore, the control unit 194 obtains the step number “23” at the previous pin from the defocus-drive pulse table.
- control unit 194 predicts that the in-focus position is at the position where the AF lens 112 has moved the movement amount corresponding to the step number “23” of the drive pulse in the closest direction from the current lens position. And the control part 194 determines the prediction range of the predetermined length centering on the estimated position. Furthermore, the control unit 194 executes contrast scanning limited to the prediction range. The length of the prediction range may be further increased by adding a safety coefficient.
- control unit 194 when it is determined that the edge of the subject image set as the tracking target is in the “rear pin state”, the control unit 194 also uses the defocus-driving pulse table (see FIG. 24) to determine the infinity side. The prediction range of the focus position centering on the focus position in the area is determined. Then, the control unit 194 executes the contrast scan only within the focus position prediction range.
- FIG. 26 shows a flowchart showing the focus drive procedure of the focus adjustment device 191 in the tracking operation for tracking the subject image.
- Steps Sc1 to Sc15 in FIG. 26 are the same as steps S1 to S15 in FIG.
- Steps Sc18 to Sc20 in FIG. 26 are the same as steps S17 to S19 in FIG.
- Steps Sc16 and Sc17 in FIG. 26 indicate parallel processing.
- Reference numeral 194 limits the moving direction of the AF lens 112. Then, the control unit 194 refers to the defocus-driving pulse table with reference to FIG. 24 to further determine the focus position prediction range. (Step Sc16).
- the control unit 194 executes the contrast scan (small scan) by limiting the movement direction of the AF lens 112 and further limiting the focus position prediction range. Further, the control unit 194 moves the AF lens 112 to the in-focus position so as to focus on the edge of the subject image (step Sc17). Then, the process of the control unit 194 proceeds to step Sc18.
- the subsequent steps are the same as in the first embodiment.
- the range in which the contrast scan is executed is limited to the in-focus position prediction range.
- the focus adjustment device 191 can focus on the subject image more quickly than the normal contrast scan.
- FIG. 27 shows an example of in-focus driving in contrast scanning based on “defocus-driving pulse table”.
- the control unit 194 executes the focusing drive based on the evaluation value Ed will be described as an example.
- Each horizontal axis in FIG. 27 indicates the number of frames of a through image captured by the imaging unit 110.
- the vertical axis in the upper part of FIG. 27 indicates Ed of the common edge.
- Ed in the frame number f1 is set as “Ed value (initial)”.
- the vertical axis in the lower part of FIG. 27 indicates the lens position.
- the control unit 194 executes the contrast scan and focuses on the edge of the subject image.
- the distribution detection unit 193 calculates Ed at a constant period. If the distribution detection unit 193 determines that the motion per unit time of the subject image is large based on the motion vector of the subject image included in the through image, the distribution detection unit 193 increases the rate of calculating Ed. The distribution detection unit 193 may increase the rate of calculating Ed when it is determined that the relative difference between the previously calculated evaluation value Ed and the currently calculated evaluation value Ed is large.
- control unit 194 executes contrast scan and scans the AF lens 112 in the number of frames f1 to f2.
- the distribution detection unit 193 determines that the movement of the subject image per unit time is large during scan driving, the distribution detection unit 193 increases the rate at which Ed is calculated.
- the control unit 194 drives the AF lens 112 to the in-focus position and focuses on the common edge of the subject image in the frame numbers f2 to f3.
- the common edge of the subject image is in focus in the frame numbers f3 to f4.
- the in-focus position in the number of frames f3 to f4 may be different from the in-focus position (initial).
- the control unit 194 limits the moving direction and moving amount of the AF lens 112 based on the defocus-driving pulse table.
- the lens position (focus position) output from the lens barrel control unit 118 is the value “1”
- the evaluation value (defocus amount) detected by the distribution detection unit 193 is the value “1”
- the distribution detection unit It is assumed that the direction indicator detected by 193 indicates the front pin state.
- the control unit 194 registers “the number of steps at the front pin” corresponding to the lens position “1” and the evaluation value “1” in the “defocus-drive pulse table (see FIG. 24)”. It is determined whether or not. In the example shown in FIG. 24, since “the number of steps at the front pin” corresponding to the lens position “1” and the evaluation value “1” is registered in the table, the control unit 194 performs defocusing. -Get the number of steps "23" at the previous pin from the drive pulse table.
- control unit 194 predicts that the in-focus position is located at the position where the AF lens 112 has moved by the amount of movement corresponding to the step number “23” of the drive pulse from the current lens position toward the closest direction. Then, the control unit 194 outputs a drive control signal to the lens barrel control unit 118. Then, the control unit 194 drives the AF lens 112 in the closest direction by the amount of movement corresponding to the step number “23” of the drive pulse. Accordingly, it is assumed that the control unit 194 focuses on the edge at the frame number f5.
- control unit 194 repeats the same operation as the frame numbers f3 to f5 in the “frame numbers f5 to f7” and the “frame numbers f7 to f9”. Then, it is assumed that a focus instruction or a shooting instruction is input to the CPU 190 at the number of frames f10 by operating the operation unit 160 by the user.
- FIG. 28 shows an enlarged view of the frame number f10 to f12 in the example of the focusing drive shown in FIG.
- the vertical axis indicates the lens position.
- the horizontal axis indicates the number of frames.
- the control unit 194 determines an in-focus position prediction range centered on the in-focus position based on the defocus-drive pulse table (see FIG. 24) at the frame number f10. And the control part 194 performs a contrast scan about a prediction range. At the same time, the control unit 194 temporarily passes through the in-focus position where the contrast value shows a peak, and calculates the evaluation value Ed by interpolation at the frame number f11.
- control unit 194 returns the AF lens 112 to the focus position calculated by interpolation (focus position drive).
- the focus adjustment device 191 detects the edge of the subject image for each color component constituting the image including the subject image incident from the lens barrel 111 having the AF lens 112 for performing focus adjustment.
- An edge detection unit 192 is provided.
- the focus adjustment device 191 further includes a distribution detection unit 193 that detects the distribution of the in-focus state and the out-of-focus state in the image based on the edge detected by the edge detection unit 192 for each color component. Then, the focus adjustment device 191 creates a “defocus-driving pulse table” including the history of the distribution detected by the distribution detection unit 193, and based on the history, the AF lens 112 is focused on the subject image.
- a control unit 194 to be moved is further provided.
- the focus adjustment device 191 can focus the subject image that has never been focused more quickly than when the defocus-drive pulse table is not referred to.
- a third embodiment of the present invention will be described in detail with reference to the drawings.
- the third embodiment is different from the first and second embodiments in that the distribution detection unit 193 calculates a defocus amount using pixel value difference data (autocorrelation) by pixel shift as an evaluation value. Only differences from the first and second embodiments will be described below.
- FIG. 29 shows a flowchart showing the operation of the focus adjustment device 191 having the moving image capturing mode.
- the CPU 190 sets the imaging device 100 to the moving image imaging mode (step Se1).
- the CPU 190 may set the imaging device 100 to the moving image imaging mode based on a user operation input received by the operation unit 180.
- the CPU 190 controls the image processing unit 140 so as to perform image processing on the image captured by the imaging device 110 as a moving image.
- the image processing unit 140 acquires 1/30 thinned data (step Se2).
- 1/30 thinned-out data is a through image output from the image sensor 119 every 1/30 second, and a part of the thinned-out image has a reduced resolution.
- the CPU 190 controls the imaging unit 110 to execute an automatic exposure (AE) process (step Se3).
- AE automatic exposure
- the control unit 194 of the focus adjustment device 191 executes the first focusing drive.
- the focusing method in this focusing drive may be selected as appropriate.
- the focusing method may be a contrast AF method or another focusing method (phase difference detection method or the like).
- the edge detection unit 192 selects a region having a high color component amount contrast from the through image as an evaluation area for evaluating the evaluation value (step Se4).
- the control unit 194 executes contrast scanning and scans the AF lens 112 (step Se5). Then, the edge detection unit 192 shifts the through image by one pixel in the horizontal direction within the evaluation area. Further, the edge detection unit 192 calculates (collects) difference data of pixel values (for example, color component amounts) between the through image shifted by one pixel in the horizontal direction and the original through image for each color channel (step) Se6).
- pixel values for example, color component amounts
- the distribution detection unit 193 creates a histogram of difference data for 10% of the difference data number of the G channel, and calculates an average value of the difference data (Step Se7). Note that the distribution detection unit 193 may create a histogram of difference data for the total number of difference data for the G channel.
- FIG. 30A shows a histogram of difference data in a focused state.
- FIG. 30B shows a histogram of difference data in the out-of-focus state.
- the horizontal axis of Drawing 30A and Drawing 30B shows difference data.
- the vertical axis indicates the frequency of difference data.
- the distribution detection unit 193 determines that the through image is out of focus.
- the image processing unit 140 newly acquires 1/30 thinning data (step Se8). Then, the edge detection unit 192 shifts the through image by one pixel in the horizontal direction within the evaluation area. Furthermore, the edge detection unit 192 calculates (collects) difference data of pixel values (for example, color component amounts) between the through image shifted by one pixel in the horizontal direction and the original through image for each color channel. Further, the distribution detection unit 193 creates a histogram of difference data for 10% of the number of difference data, and calculates an average value of the difference data for each color channel (step Se9). Note that the distribution detection unit 193 may create a histogram of difference data for each color channel with respect to the total number of difference data.
- the distribution detection unit 193 determines that the through image is in a defocused state (Step Se10). Further, the distribution detection unit 193 determines whether the edge of the subject image is in the front pin state or the back pin state based on the change in the average value of the R channel difference data and the change in the average value of the B channel difference data. Is determined (step Se11).
- the distribution detection unit 193 determines that the edge of the subject image is in the front pin state when the change in the average value of the R channel difference data is greater than the change in the average value of the B channel difference data. May be.
- control unit 194 executes a contrast scan by limiting the scan area (step Se12).
- the focus adjustment device 191 shifts the image by one pixel in the horizontal direction and calculates difference data from the original image, thereby executing a contrast scan by limiting the scan region. Accordingly, the control unit 194 ends the focusing drive in a short time with a light processing load as compared with the case where the normal contrast scan is executed. Therefore, the focus adjustment device 191 can quickly focus on a subject image that has never been focused.
- the focus adjustment device 191 may improve the performance of the contrast AF of the spotlight subject based on the evaluation value described above.
- FIG. 31 shows a flowchart showing the operation of the focus adjustment device 191 for determining spot light.
- the image processing unit 140 acquires a through image (decimated VGA resolution) from the imaging unit 110 (step Sf1).
- the cycle in which the image processing unit 140 acquires the through image may be a cycle (interval rate) shorter than usual.
- the CPU 190 controls the imaging unit 110 to execute a debayer (color interpolation) process.
- the image processing unit 140 performs image processing on the acquired through image. For example, the image processing unit 140 performs white balance adjustment processing and noise reduction processing on the acquired through image (step Sf2).
- the distribution detection unit 193 creates a histogram of color component amounts and determines whether or not there is a clip channel in the histogram (step Sf3).
- the clip channel is a channel in which the color component amount is saturated.
- the distribution detection unit 193 may create a histogram of luminance values.
- the distribution detection unit 193 determines that there is spot light around a pixel where clipping (saturation) occurs, and searches for an effective edge in the vicinity (step Sf4). Then, in order to search for an effective edge using a high-resolution image (an image that has not been thinned out), the image processing unit 140 acquires a high-resolution image from the imaging unit 110 (multi-scale window). The distribution detection unit 193 searches for a valid edge around a pixel where clipping (saturation) occurs in a high-resolution image (step Sf5).
- the distribution detection unit 193 calculates the LSF based on the edge information and compares the LSF for each color channel. Then, as described above, the distribution detection unit 193 calculates the direction index and the defocus amount (step Sf6). Then, the control unit 194 drives the AF lens 112 based on the direction indicator and the defocus amount (step Sf7).
- the focus adjustment device 191 searches for a valid edge around the pixel where clipping (saturation) occurs. As a result, it is possible to improve the performance of the contrast AF of the spotlight subject. Further, the imaging apparatus 100 may change imaging conditions (for example, an aperture value, an exposure value, etc.) based on a result of the spot adjustment device 191 detecting spot light (scene detection).
- imaging conditions for example, an aperture value, an exposure value, etc.
- the imaging apparatus 100 may switch the imaging mode to the “macro (close-up) imaging mode” based on the direction index and the defocus amount calculated by the focus adjustment apparatus 191.
- FIG. 32 is a diagram for explaining the switching determination operation to the macro imaging mode.
- the distribution detection unit 193 determines that the subject image is in the front pin state when the position of the AF lens 112 is closest.
- the CPU 190 of the imaging apparatus 100 switches the imaging mode to the “macro (close-up) imaging mode” based on the direction index (front pin state) calculated by the focus adjustment apparatus 191 and the defocus amount.
- the control unit 194 executes the contrast scan by limiting the scan range to the “close range of macro (close-up)” that is further closer than the close range in the normal contrast scan.
- the imaging apparatus 100 sets the imaging mode to “macro (close-up) imaging based on the direction index (front pin state) calculated by the focus adjustment apparatus 191 and the defocus amount, regardless of the user's operation input. Can be switched to "mode".
- the color channel may be expressed in a color space expression format other than RGB.
- the color channel may be expressed in a color space expression format by color difference (YCbCr, YUV, etc.).
- the distribution detection unit 193 may subdivide the blocks based on the result of grouping the subject images with the same color and label the subject images.
- the distribution detection unit 193 may change the resolution of the image when a valid edge is not detected.
- the distribution detection unit 193 may perform super-resolution processing.
- each profile, each parameter, and each threshold value described above may be changed according to the zoom position, the image height, and the focus range position.
- the distribution detection unit 193 determines that the change in the evaluation value is small based on the evaluation value history calculated at a certain period, the distribution detection unit 193 determines the period for calculating the evaluation value as the movement of the subject is small. It may be longer. In this way, the distribution detection unit 193 can reduce the calculation load for calculating the evaluation value.
- SFR Spatial Frequency Response
- the focus adjustment device selects N upper edges (N is an integer equal to or greater than 1) as upper edges from the plurality of edges according to the priority order. Then, the focus adjustment device performs focus adjustment based on these selected edges (see, for example, step S11 in FIG. 19). A method of selecting an edge for performing focus adjustment by the focus adjustment apparatus when a plurality of edges exist in the image will be described below.
- the distribution detection unit 193 selects the upper edge in the descending order of power (the amount of color component is large) among the edges detected by the edge detection unit 192 for each color component. In addition, the distribution detection unit 193 selects an upper edge in descending order of color component contrast among the edges detected by the edge detection unit 192 for each color component.
- the distribution detection unit 193 determines the color component signal pair.
- the upper edge is selected in descending order of noise ratio.
- the distribution detection unit 193 selects the upper edge in descending order of the color component contrast. To do.
- the distribution detection unit 193 determines that the edge having a relatively low signal-to-noise ratio and a relatively high contrast and the signal-to-noise ratio are relatively At least one of the edges having high and relatively low contrast is selected.
- FIG. 33A and 33B are diagrams for explaining the priority order of edges selected when edges having different signal-to-noise ratios of color components are mixed.
- the vertical axis represents the amount of color components.
- the horizontal axis indicates the position on the axis D (see FIG. 7).
- FIG. 33A shows an edge with a relatively high signal-to-noise ratio and a relatively low contrast.
- FIG. 33B shows an edge having a relatively low signal-to-noise ratio and a relatively high contrast.
- the distribution detection unit 193 selects an edge composed of white and black color components. When there is no edge composed of white and black color components, the distribution detection unit 193 includes two or more primary colors among the edges detected by the edge detection unit 192 for each color component, and the color components have the same phase. Select a changing edge. When two primary colors are included in the edge, the distribution detection unit 193 selects an edge including a green component (G channel). When the green component is not included in the edge, the distribution detection unit 193 selects an edge including the blue component (B channel).
- G channel green component
- B channel blue component
- FIG. 34A, 34B, and 34C are diagrams showing the priority order of edges selected when two primary colors are included.
- the vertical axis represents the amount of color components.
- the horizontal axis indicates the position on the axis D (see FIG. 7).
- FIG. 34A shows an edge constituted by an R channel and a G channel that change in phase.
- FIG. 34B shows an edge composed of the R channel and the G channel that change in phase, and the edge of the R channel is lower than the contrast of the R channel shown in FIG. 34A. Yes.
- FIG. 34C shows an edge constituted by an R channel and a B channel that change in phase.
- the edge with the highest priority is the edge shown in FIG. 34A that includes a green component (G channel) and has high contrast.
- the edge with the lowest priority among these edges is the edge shown in FIG. 34C that does not contain a green component.
- the edge containing the green component is detected with high accuracy by the image sensor 119 (see FIG. 1).
- FIGS. 35A and 35B are diagrams for explaining the change in the color component amount according to the position crossing one edge where the color component is flat and the change in the color component amount according to the position crossing a plurality of edges.
- shaft of FIG. 35A and 35B shows the amount of color components.
- the horizontal axis in FIG. 35A indicates a position crossing one edge, that is, a position on the axis D (see FIG. 7).
- the horizontal axis of FIG. 35B indicates a position crossing a plurality of edges.
- FIG. 35A shows a change in the amount of the color component according to the position crossing one edge where the color component is flat.
- the edge has a flat color component amount greater than or equal to predetermined widths W1 and W2.
- widths W3 and W4 including a range in which the color component amount has a gradient may be determined instead of the widths W1 and W2.
- FIG. 35B shows a change in the color component amount according to the position crossing the plurality of edges.
- the color component amount of each edge is not flat. Therefore, the edge with the highest priority among these edges is the edge shown in FIG. 35A, in which the color component is flat beyond a predetermined width.
- the distribution detection unit 193 selects an edge having a length equal to or longer than the length determined according to the signal-to-noise ratio of the color component from the edges detected by the edge detection unit 192 for each color component. For example, the distribution detection unit 193 selects a longer edge as the signal-to-noise ratio of the color component is lower.
- FIG. 36 is a flowchart illustrating a procedure for selecting an upper edge in accordance with the priority order.
- the edge detection unit 192 detects a plurality of edges (step Sg1).
- the distribution detection unit 193 detects the intensity (power, color component amount) of the edge detected by the edge detection unit 192 (step Sg2).
- the edge detection unit 192 selects the top N edges with the highest priority based on the detected edge strength. For example, the edge detection unit 192 sets an edge having a large amount of color component as a tracking target (step Sg3).
- the distribution detection unit 193 selects an edge having a higher priority based on the color components constituting the selected edge (see FIGS. 34A, 34B, and 34C). For example, when two primary colors are included in the edge, the distribution detection unit 193 selects an edge including a green component (G channel) (step Sg4).
- G channel green component
- the distribution detection unit 193 selects an edge having a higher priority based on the contrast of the selected edge (see FIG. 33). For example, the distribution detection unit 193 selects an upper edge in descending order of color component amount difference (adjacent difference) between adjacent pixels (step Sg5).
- the distribution detection unit 193 selects an edge with a higher priority. (Step Sg6).
- the distribution detection unit 193 selects an edge having a length longer than that determined according to the signal-to-noise ratio of the color component. For example, the distribution detection unit 193 selects a longer edge as the signal-to-noise ratio of the color component is lower (step Sg7).
- the distribution detection unit 193 registers an edge address which is edge identification information. That is, the distribution detection unit 193 sets the finally selected upper edge as a tracking target (step Sg8).
- the control unit 194 determines whether the AF lens 112 is in the front pin state or the rear pin state based on the edge evaluation value. That is, the control unit 194 moves the AF lens 112 to focus on the edge of the subject image based on the distribution of the in-focus state and the out-of-focus state detected by the distribution detection unit 193.
- the control unit 194 executes a contrast scan by limiting the moving direction of the AF lens 112 (step Sg9).
- the distribution detection unit 193 may select an edge having a higher priority based on the luminance of the edge instead of the color component of the edge.
- a program for realizing the procedure described with reference to FIGS. 16, 19, 20, 23, 26, 29, 31 and 36 is recorded on a computer-readable recording medium, and the program recorded on the recording medium May be executed by causing the computer system to read and execute.
- the “computer system” may include an OS and hardware such as peripheral devices.
- “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
- the “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.
- the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic) in a computer system which becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)) is also assumed to include a recording medium that holds a program for a certain period of time.
- the program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium.
- the “transmission medium” for transmitting the program refers to a medium having a characteristic point for transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
- the program may be a program for realizing a part of the utility described above.
- the program may be a so-called difference file (difference program) that can realize the above-described utility in combination with a program already recorded in the computer system.
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Optics & Photonics (AREA)
- Automatic Focus Adjustment (AREA)
- Studio Devices (AREA)
- Focusing (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
Description
本願は、2010年2月15日に日本に出願された、特願2010-030481号に基づき優先権を主張し、その内容をここに援用する。
本発明の第1実施形態について図面を参照して詳細に説明する。図1は、レンズ鏡筒111と、焦点調節装置191とを備える撮像装置100と、記憶媒体200との構成を示すブロック図である。撮像装置100は、レンズ鏡筒111から入射される被写体像を撮像し、得られた画像を静止画又は動画の画像として、記憶媒体200に記憶させる。
EdRG=(Σ(R/G))/(Δ1)<1 … 式2
EdRG=(Σ(G-R))/(Δ2)<0 … 式4
EdBG=(Σ(B/G))/(Δ3)<1 … 式6
EdBG=(Σ(G-B))/(Δ4)<0 … 式8
= L1((Σ(R1/G1))/Δ)
+L2((Σ(R2/G2))/Δ)
+L3((Σ(R3/G3))/Δ) … 式11
図23には、LSFに基づく合焦駆動の手順を示すフローチャートが示されている。撮像装置100のCPU190は、エッジに合焦(AF)させる領域として、スルー画像の中央に領域(中央エリア領域)を設定する(ステップSd1)。この設定は、操作部180が受け付けたユーザの操作入力に基づいて、実行されてもよい。
本発明の第2実施形態について図面を参照して詳細に説明する。第2実施形態は、後述する「デフォーカス-駆動パルステーブル」に基づいて分布検出部193が合焦位置(デフォーカス量)を予測する点が、第1実施形態と異なる。以下、第2実施形態と第1実施形態との相違点についてのみ説明する。
本発明の第3実施形態について図面を参照して詳細に説明する。第3実施形態が第1及び2実施形態と異なる点は、分布検出部193が、画素ずらしによる画素値の差分データ(自己相関)を評価値として、デフォーカス量を算出する点である。以下、第1及び2実施形態との相違点についてのみ説明する。
複数のエッジが画像に存在する場合、焦点調節装置は、複数のエッジから優先順位に応じて上位のエッジを、予め定められたN個(Nは、1以上の整数)選択する。そして、焦点調節装置は、選択したこれらのエッジに基づいて焦点調節を行う(例えば、図19のステップS11を参照)。このように複数のエッジが画像に存在する場合に焦点調節装置が焦点調節を行うためのエッジを選択する方法について、以下に説明する。
また、上記プログラムは、このプログラムを記憶装置等に格納したコンピュータシステムから、伝送媒体を介して、あるいは、伝送媒体中の伝送波により他のコンピュータシステムに伝送されてもよい。ここで、プログラムを伝送する「伝送媒体」は、インターネット等のネットワーク(通信網)や電話回線等の通信回線(通信線)のように情報を伝送する特徴点を有する媒体のことをいう。
また、上記プログラムは、前述した効用の一部を実現するためのプログラムであっても良い。さらに、上記プログラムは、前述した効用をコンピュータシステムにすでに記録されているプログラムとの組み合わせで実現できる、いわゆる差分ファイル(差分プログラム)であっても良い。
110…撮像部
111…レンズ鏡筒
190…CPU
191…焦点調節装置
192…エッジ検出部
193…分布検出部
194…制御部
Claims (17)
- 焦点調節装置であって、
被写体像を含む画像を構成する色成分毎に、前記被写体像のエッジを検出するエッジ検出部と;
前記エッジ検出部が検出した前記エッジに基づいて、前記画像の合焦状態及び非合焦状態の分布を検出する分布検出部と;
前記分布検出部が検出した前記分布に基づいてレンズを移動させる制御部と;
を備え、
前記被写体像は、前記レンズを有する光学系から入射され、
前記制御部が前記レンズを移動させる結果、前記被写体像に焦点が合うことを特徴とする焦点調節装置。 - 前記分布検出部は、前記エッジ検出部が検出した前記エッジの色成分量の勾配に基づいて、前記分布を検出することを特徴とする請求項1に記載の焦点調節装置。
- 前記分布検出部は、前記エッジ検出部が検出した前記エッジの色成分量の比又は差に基づいて、被写体に対して近点側と遠点側のどちらに合焦しているかを示す方向指標と、デフォーカス量と、を検出することを特徴とする請求項1に記載の焦点調節装置。
- 前記分布検出部は、前記エッジ検出部が検出した前記エッジの色成分量の比又は差のピーク間距離に基づいて、前記デフォーカス量を検出することを特徴とする請求項3に記載の焦点調節装置。
- 前記分布検出部は、前記エッジ検出部が検出した前記エッジに対応する線広がり関数に基づいて、被写体に対して近点側と遠点側のどちらに合焦しているかを示す方向指標と、デフォーカス量と、を検出することを特徴とする請求項1に記載の焦点調節装置。
- 前記分布検出部は、前記エッジ検出部が検出した前記エッジに対応する線広がり関数の標準偏差又は半値幅に基づいて、前記デフォーカス量を検出することを特徴とする請求項5に記載の焦点調節装置。
- 前記分布検出部は、前記エッジ検出部が検出した前記エッジのうち、色成分量が多い順に上位のエッジを選択し、この選択したエッジに基づいて、前記画像の合焦状態及び非合焦状態の分布を検出することを特徴とする請求項1に記載の焦点調節装置。
- 前記分布検出部は、前記エッジ検出部が検出した前記エッジのうち、前記色成分のコントラストが高い順に上位のエッジを選択し、この選択したエッジに基づいて、前記画像の合焦状態及び非合焦状態の分布を検出することを特徴とする請求項1に記載の焦点調節装置。
- 前記分布検出部は、前記色成分の前記コントラストが同じエッジと、前記色成分の信号対雑音比が異なるエッジとが混在している場合、前記色成分の前記信号対雑音比が高い順に上位のエッジを選択し、この選択したエッジに基づいて、前記画像の合焦状態及び非合焦状態の分布を検出することを特徴とする請求項8に記載の焦点調節装置。
- 前記分布検出部は、前記色成分の信号対雑音比が異なるエッジが混在している場合、前記信号対雑音比が相対的に低く且つ前記コントラストが相対的に高いエッジと、前記信号対雑音比が相対的に高く且つ前記コントラストが相対的に低いエッジとのうち少なくとも一方を選択し、この選択したエッジに基づいて、前記画像の合焦状態及び非合焦状態の分布を検出することを特徴とする請求項8に記載の焦点調節装置。
- 前記分布検出部は、前記エッジ検出部が検出した前記エッジのうち、2原色以上含まれ且つ前記色成分が同位相で変化するエッジを選択し、この選択したエッジに基づいて、前記画像の合焦状態及び非合焦状態の分布を検出することを特徴とする請求項1に記載の焦点調節装置。
- 前記分布検出部は、前記エッジに2原色含まれている場合、緑色成分を含むエッジを選択することを特徴とする請求項11に記載の焦点調節装置。
- 前記分布検出部は、前記エッジ検出部が検出した前記エッジのうち、予め定められた幅以上に前記色成分が平坦であるエッジを選択し、この選択したエッジに基づいて、前記画像の合焦状態及び非合焦状態の分布を検出することを特徴とする請求項1に記載の焦点調節装置。
- 前記分布検出部は、前記エッジ検出部が検出した前記エッジのうち、前記色成分の信号対雑音比に応じて定まる長さ以上のエッジを選択し、この選択したエッジに基づいて、前記画像の合焦状態及び非合焦状態の分布を検出することを特徴とする請求項1に記載の焦点調節装置。
- 焦点調節装置であって、
被写体像を含む画像を構成する色成分毎に、前記被写体像のエッジを検出するエッジ検出部と;
前記エッジ検出部が検出した前記エッジの線広がり関数を算出する分布検出部と;
前記線広がり関数に基づいてレンズを移動させる制御部と;
を備え、
前記被写体像は、前記レンズを有する光学系から入射され、
前記制御部が前記レンズを移動させる結果、前記被写体像に焦点が合うことを特徴とする焦点調節装置。 - 焦点調節プログラムを実行するコンピュータであって、前記焦点調節プログラムは、
焦点調節を行うためのレンズを有する光学系から入射される被写体像を含む画像を構成する色成分毎に、前記被写体像のエッジを検出する手順と;
前記エッジに基づいて、前記画像の合焦状態及び非合焦状態の分布を検出する手順と;
前記分布に基づいて、前記被写体像に合焦させるように前記レンズを移動させる手順と;
を含むことを特徴とするコンピュータ。 - 焦点調節プログラムを実行するコンピュータであって、前記焦点調節プログラムは、
焦点調節を行うためのレンズを有する光学系から入射される被写体像を含む画像を構成する色成分毎に、前記被写体像のエッジを検出する手順と;
前記エッジの線広がり関数を算出する手順と;
前記線広がり関数に基づいて、前記被写体像に合焦させるように前記レンズを移動させる手順と;
を含むことを特徴とするコンピュータ。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/578,927 US9066001B2 (en) | 2010-02-15 | 2011-02-15 | Focus adjusting device and focus adjusting program with distribution detection of focalized and unfocused state |
CN201180015740.7A CN102844705B (zh) | 2010-02-15 | 2011-02-15 | 焦点调节装置和焦点调节方法 |
US14/727,572 US9883095B2 (en) | 2010-02-15 | 2015-06-01 | Focus adjusting device and focus adjusting program with control unit to guide a light image based upon detected distributions |
US15/848,893 US20180115698A1 (en) | 2010-02-15 | 2017-12-20 | Focus adjusting device and focus adjusting program with control unit to guide a light image based upon detected distributions |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2010030481 | 2010-02-15 | ||
JP2010-030481 | 2010-02-15 |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/578,927 A-371-Of-International US9066001B2 (en) | 2010-02-15 | 2011-02-15 | Focus adjusting device and focus adjusting program with distribution detection of focalized and unfocused state |
US14/727,572 Continuation US9883095B2 (en) | 2010-02-15 | 2015-06-01 | Focus adjusting device and focus adjusting program with control unit to guide a light image based upon detected distributions |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2011099626A1 true WO2011099626A1 (ja) | 2011-08-18 |
Family
ID=44367889
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2011/053156 WO2011099626A1 (ja) | 2010-02-15 | 2011-02-15 | 焦点調節装置、及び焦点調節プログラム |
Country Status (4)
Country | Link |
---|---|
US (3) | US9066001B2 (ja) |
JP (2) | JP5757099B2 (ja) |
CN (1) | CN102844705B (ja) |
WO (1) | WO2011099626A1 (ja) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106504279A (zh) * | 2016-10-18 | 2017-03-15 | 安庆师范大学 | 彩色图像自动聚焦方法 |
Families Citing this family (29)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5757099B2 (ja) * | 2010-02-15 | 2015-07-29 | 株式会社ニコン | 焦点調節装置、及び焦点調節プログラム |
JP2011223562A (ja) * | 2010-03-23 | 2011-11-04 | Fujifilm Corp | 撮像装置 |
CN102158731B (zh) * | 2011-05-26 | 2014-03-12 | 威盛电子股份有限公司 | 影像处理***及方法 |
CN102158730B (zh) * | 2011-05-26 | 2014-04-02 | 威盛电子股份有限公司 | 影像处理***及方法 |
US9407809B2 (en) | 2013-11-27 | 2016-08-02 | Qualcomm Incorporated | Strategies for triggering depth sensors and transmitting RGBD images in a cloud-based object recognition system |
JP6464553B2 (ja) * | 2013-11-28 | 2019-02-06 | 株式会社ニコン | レンズ駆動制御装置、電子カメラ及びレンズ駆動制御プログラム |
JP6349703B2 (ja) * | 2013-12-03 | 2018-07-04 | 株式会社ニコン | 電子カメラ及び画像処理プログラム |
JP5874753B2 (ja) * | 2014-01-28 | 2016-03-02 | カシオ計算機株式会社 | 撮像装置、撮像方法及びプログラム |
JP6478496B2 (ja) * | 2014-07-03 | 2019-03-06 | キヤノン株式会社 | 撮像装置およびその制御方法 |
WO2016185952A1 (ja) * | 2015-05-18 | 2016-11-24 | ソニー株式会社 | 制御装置及び医療用撮像システム |
FR3046256B1 (fr) * | 2015-12-23 | 2018-01-05 | Thales | Zoom plenoptique a portee optimisee |
CN106937107B (zh) * | 2015-12-29 | 2019-03-12 | 宁波舜宇光电信息有限公司 | 基于色差的摄像模组调焦方法 |
US20170321987A1 (en) * | 2016-05-05 | 2017-11-09 | Coriolis Games Corporation | Simulated firearm with target accuracy detection, and related methods and systems |
JP6742178B2 (ja) * | 2016-07-19 | 2020-08-19 | 日本放送協会 | Mtf測定装置およびそのプログラム |
JP6742180B2 (ja) * | 2016-07-21 | 2020-08-19 | 日本放送協会 | Mtf測定装置及びそのプログラム |
CN106331499B (zh) * | 2016-09-13 | 2019-10-29 | 努比亚技术有限公司 | 对焦方法及拍照设备 |
JP6967382B2 (ja) * | 2017-01-13 | 2021-11-17 | 日本放送協会 | Mtf測定装置およびそのプログラム |
US10798316B2 (en) * | 2017-04-04 | 2020-10-06 | Hand Held Products, Inc. | Multi-spectral imaging using longitudinal chromatic aberrations |
WO2018207400A1 (ja) * | 2017-05-09 | 2018-11-15 | 三菱電機株式会社 | Mtf測定装置およびmtf測定方法 |
US11074677B2 (en) | 2018-03-29 | 2021-07-27 | Dolby Laboratories Licensing Corporation | Dynamic range extension of partially clipped pixels in captured images |
JP2018163365A (ja) * | 2018-06-07 | 2018-10-18 | 株式会社ニコン | 画像処理装置 |
CN109078342B (zh) * | 2018-08-17 | 2020-03-27 | 浙江大丰实业股份有限公司 | 舞台吊杆高度维护*** |
KR102668212B1 (ko) | 2018-09-28 | 2024-05-23 | 삼성전자 주식회사 | 자동 초점 방법 및 이를 수행하는 전자 장치 |
DE102019117293B3 (de) | 2019-06-27 | 2020-11-05 | Carl Zeiss Smt Gmbh | Vorrichtung zur Vermessung von Masken für die Mikrolithographie und Autofokussierverfahren |
CN110807745B (zh) * | 2019-10-25 | 2022-09-16 | 北京小米智能科技有限公司 | 图像处理方法及装置、电子设备 |
JP7405678B2 (ja) * | 2020-04-01 | 2023-12-26 | 株式会社Screenホールディングス | 合焦位置検出方法、合焦位置検出装置および合焦位置検出用プログラム |
TWI790758B (zh) * | 2021-09-24 | 2023-01-21 | 致茂電子股份有限公司 | 應用軸向色差的對焦方法及使用該方法的光學檢測架構 |
WO2023150671A1 (en) * | 2022-02-04 | 2023-08-10 | Arizona Board Of Regents On Behalf Of The University Of Arizona | Methods, systems and devices for increasing image resolution and dynamic range |
US11852895B2 (en) | 2022-02-28 | 2023-12-26 | e-con Systems India Private Limited | System and method for obtaining optimal focus of an image capturing device |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2003262783A (ja) * | 2002-03-08 | 2003-09-19 | Minolta Co Ltd | オートフォーカス装置および撮像装置 |
JP2007139894A (ja) * | 2005-11-15 | 2007-06-07 | Olympus Corp | 撮像装置 |
Family Cites Families (34)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS62115408A (ja) * | 1985-11-15 | 1987-05-27 | Tamuron:Kk | 固体撮像素子を有するカメラの合焦検出装置 |
JPS6394212A (ja) * | 1986-10-08 | 1988-04-25 | Canon Inc | 合焦検出装置 |
JPH01246516A (ja) * | 1988-03-29 | 1989-10-02 | Sony Corp | 焦点位置検出装置 |
JPH03136012A (ja) * | 1989-10-23 | 1991-06-10 | Olympus Optical Co Ltd | 合焦検出装置 |
JPH03204277A (ja) * | 1990-01-05 | 1991-09-05 | Canon Inc | 合焦検出方法 |
JPH0545574A (ja) | 1991-08-14 | 1993-02-23 | Sony Corp | ビデオカメラ |
JP3482013B2 (ja) * | 1993-09-22 | 2003-12-22 | ペンタックス株式会社 | 光学系のピント評価方法、調整方法、調整装置、およびチャート装置 |
JPH08248302A (ja) * | 1995-03-14 | 1996-09-27 | Nikon Corp | 自動焦点検出装置 |
KR100197609B1 (ko) | 1996-07-27 | 1999-06-15 | 윤종용 | 비데오카메라의 줌(zoom)기능 수행방법 및 그 장치 |
US6088539A (en) | 1997-12-24 | 2000-07-11 | Canon Kabushiki Kaisha | Optical apparatus with focus adjusting function and focus adjustment control circuit |
JP3630965B2 (ja) | 1997-12-24 | 2005-03-23 | キヤノン株式会社 | 焦点検出装置、カメラ及び焦点検出方法 |
CN1815359A (zh) * | 1998-01-27 | 2006-08-09 | 诺日士钢机株式会社 | 底片扫描器、缺陷识别方法和图像修复方法 |
JP2000216978A (ja) * | 1999-01-22 | 2000-08-04 | Canon Inc | 画像抽出方法及び装置並びに記憶媒体 |
JP2000338385A (ja) * | 1999-05-28 | 2000-12-08 | Ricoh Co Ltd | 自動合焦装置およびその合焦方法 |
JP2001141988A (ja) * | 1999-08-31 | 2001-05-25 | Canon Inc | 焦点調節装置及びその制御方法及びその制御プログラムを供給する媒体 |
JP2002122774A (ja) * | 2000-10-16 | 2002-04-26 | Nec Corp | 焦点位置決定システムおよび焦点位置決定方法 |
JP2002214513A (ja) * | 2001-01-16 | 2002-07-31 | Minolta Co Ltd | 光学系制御装置、光学系制御方法および記録媒体 |
JP3748446B2 (ja) * | 2003-12-19 | 2006-02-22 | 株式会社メガチップス | 画像入力装置の画像処理回路 |
JP4706635B2 (ja) * | 2004-04-12 | 2011-06-22 | 株式会社ニコン | 色ずれ補正機能を有する画像処理装置、画像処理プログラム、および電子カメラ |
US20060078217A1 (en) * | 2004-05-20 | 2006-04-13 | Seiko Epson Corporation | Out-of-focus detection method and imaging device control method |
JP4661084B2 (ja) * | 2004-05-20 | 2011-03-30 | セイコーエプソン株式会社 | ミスフォーカス判定方法およびこれに関するプログラム |
KR100866490B1 (ko) * | 2007-01-17 | 2008-11-03 | 삼성전자주식회사 | 영상의 색 수차를 보정하기 위한 장치 및 방법 |
JP4332566B2 (ja) * | 2007-03-23 | 2009-09-16 | キヤノン株式会社 | 撮像装置 |
JP5036421B2 (ja) * | 2007-06-25 | 2012-09-26 | シリコン ヒフェ ベー.フェー. | 画像処理装置、画像処理方法、プログラムおよび撮像装置 |
US8494260B2 (en) | 2007-06-25 | 2013-07-23 | Silicon Hive B.V. | Image processing device, image processing method, program, and imaging device |
KR101399012B1 (ko) * | 2007-09-12 | 2014-05-26 | 삼성전기주식회사 | 영상 복원 장치 및 방법 |
US8264591B2 (en) * | 2007-10-22 | 2012-09-11 | Candela Microsystems (S) Pte. Ltd. | Method and system for generating focus signal |
EP2147854B1 (de) * | 2008-07-16 | 2011-02-16 | Torqeedo GmbH | Halterung für einen Außenbordmotor |
WO2010061352A2 (en) * | 2008-11-26 | 2010-06-03 | Hiok Nam Tay | Auto-focus image system |
US8698905B2 (en) * | 2009-03-11 | 2014-04-15 | Csr Technology Inc. | Estimation of point spread functions from motion-blurred images |
US9131142B2 (en) * | 2009-07-17 | 2015-09-08 | Nikon Corporation | Focusing device and camera |
US8958009B2 (en) * | 2010-01-12 | 2015-02-17 | Nikon Corporation | Image-capturing device |
JP5757099B2 (ja) * | 2010-02-15 | 2015-07-29 | 株式会社ニコン | 焦点調節装置、及び焦点調節プログラム |
US8724009B2 (en) * | 2010-05-05 | 2014-05-13 | Hiok Nam Tay | Auto-focus image system |
-
2011
- 2011-02-09 JP JP2011026263A patent/JP5757099B2/ja active Active
- 2011-02-15 CN CN201180015740.7A patent/CN102844705B/zh active Active
- 2011-02-15 US US13/578,927 patent/US9066001B2/en active Active
- 2011-02-15 WO PCT/JP2011/053156 patent/WO2011099626A1/ja active Application Filing
-
2015
- 2015-06-01 US US14/727,572 patent/US9883095B2/en active Active
- 2015-06-05 JP JP2015115246A patent/JP5994904B2/ja active Active
-
2017
- 2017-12-20 US US15/848,893 patent/US20180115698A1/en not_active Abandoned
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2003262783A (ja) * | 2002-03-08 | 2003-09-19 | Minolta Co Ltd | オートフォーカス装置および撮像装置 |
JP2007139894A (ja) * | 2005-11-15 | 2007-06-07 | Olympus Corp | 撮像装置 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106504279A (zh) * | 2016-10-18 | 2017-03-15 | 安庆师范大学 | 彩色图像自动聚焦方法 |
CN106504279B (zh) * | 2016-10-18 | 2019-02-19 | 安庆师范大学 | 彩色图像自动聚焦方法 |
Also Published As
Publication number | Publication date |
---|---|
JP5757099B2 (ja) | 2015-07-29 |
JP2011186452A (ja) | 2011-09-22 |
CN102844705A (zh) | 2012-12-26 |
US9066001B2 (en) | 2015-06-23 |
US20180115698A1 (en) | 2018-04-26 |
JP2015200903A (ja) | 2015-11-12 |
US20130016275A1 (en) | 2013-01-17 |
US9883095B2 (en) | 2018-01-30 |
US20150264252A1 (en) | 2015-09-17 |
CN102844705B (zh) | 2017-04-12 |
JP5994904B2 (ja) | 2016-09-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5994904B2 (ja) | 焦点調節装置 | |
US9936122B2 (en) | Control apparatus, control method, and non-transitory computer-readable storage medium for performing focus control | |
US10326927B2 (en) | Distance information producing apparatus, image capturing apparatus, distance information producing method and storage medium storing distance information producing program | |
US10395348B2 (en) | Image pickup apparatus, image processing apparatus, and control method of image pickup apparatus | |
US9635280B2 (en) | Image pickup apparatus, method of controlling image pickup apparatus, and non-transitory computer-readable storage medium | |
EP2963492A1 (en) | Image capture apparatus and method for controlling the same | |
US20170230567A1 (en) | Image pickup apparatus, control method, and non-transitory computer-readable storage medium | |
JP4122865B2 (ja) | オートフォーカス装置 | |
US20140185870A1 (en) | Image processing apparatus and image processing method | |
JP6614783B2 (ja) | 自動焦点調節装置およびその制御方法 | |
JP5417827B2 (ja) | 焦点検出装置及び撮像装置 | |
JP4701111B2 (ja) | パターンマッチングシステム及び被写体追尾システム | |
JP6156532B2 (ja) | 領域抽出装置、撮像装置、及び領域抽出プログラム | |
US11838628B2 (en) | Image capturing apparatus, control method therefor, and storage medium | |
JP5673454B2 (ja) | 画像処理装置、画像処理方法及びプログラム | |
US9710897B2 (en) | Image processing apparatus, image processing method, and recording medium | |
JP6624789B2 (ja) | 合焦制御装置、その制御方法、および制御プログラム、並びに撮像装置 | |
JP7023673B2 (ja) | 焦点調節装置、焦点調節方法、及び撮像装置 | |
JP2016201600A (ja) | 画像処理装置、撮像装置、画像処理方法、画像処理プログラム、および、記憶媒体 | |
JP6003974B2 (ja) | 画像処理装置、画像処理方法及びプログラム | |
JP6464553B2 (ja) | レンズ駆動制御装置、電子カメラ及びレンズ駆動制御プログラム | |
JP6238673B2 (ja) | 画像処理装置、撮像装置、撮像システム、画像処理方法、画像処理プログラム、および、記憶媒体 | |
JP6508259B2 (ja) | 撮像装置、及びプログラム | |
JP2018054815A (ja) | 領域判別装置、その制御方法、および制御プログラム、並びに撮像装置 | |
JP2004219461A (ja) | 撮像装置およびオートフォーカス装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201180015740.7 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11742365 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13578927 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 11742365 Country of ref document: EP Kind code of ref document: A1 |