WO2012105222A1 - 画像復元装置、撮像装置及び画像復元方法 - Google Patents
画像復元装置、撮像装置及び画像復元方法 Download PDFInfo
- Publication number
- WO2012105222A1 WO2012105222A1 PCT/JP2012/000582 JP2012000582W WO2012105222A1 WO 2012105222 A1 WO2012105222 A1 WO 2012105222A1 JP 2012000582 W JP2012000582 W JP 2012000582W WO 2012105222 A1 WO2012105222 A1 WO 2012105222A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- range
- sweep
- image
- psf
- restoration
- Prior art date
Links
- 238000003384 imaging method Methods 0.000 title claims description 94
- 238000000034 method Methods 0.000 title claims description 62
- 238000004364 calculation method Methods 0.000 claims abstract description 27
- 229920013655 poly(bisphenol-A sulfone) Polymers 0.000 claims abstract description 20
- 230000008569 process Effects 0.000 claims description 36
- 238000006073 displacement reaction Methods 0.000 claims description 11
- 238000010408 sweeping Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 description 16
- 230000006870 function Effects 0.000 description 10
- 239000000470 constituent Substances 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- ORQBXQOJMQIAOY-UHFFFAOYSA-N nobelium Chemical compound [No] ORQBXQOJMQIAOY-UHFFFAOYSA-N 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/958—Computational photography systems, e.g. light-field imaging systems for extended depth of field imaging
- H04N23/959—Computational photography systems, e.g. light-field imaging systems for extended depth of field imaging by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
Definitions
- the present invention relates to an image restoration apparatus, an imaging apparatus, and an image restoration method, and in particular, a sweep range including a desired distance range to be focused by driving an imaging element or a lens that collects light on the imaging element.
- a sweep range including a desired distance range to be focused by driving an imaging element or a lens that collects light on the imaging element.
- EDOF depth of field expansion
- a blur in the depth direction is made uniform by inserting an optical element called a phase plate into the optical system.
- an image restoration process is performed on the obtained image using a blur pattern measured in advance or a blur pattern calculated by simulation. Thereby, the method generates an EDOF image.
- WFC Wavefront Coding
- the second method measures the distance with high accuracy for each partial range of the image by devising the aperture shape.
- image restoration processing is performed on each partial range using a blur pattern corresponding to each distance measured in advance. Thereby, the method generates an EDOF image.
- This method is referred to as coded aperture (hereinafter referred to as CA).
- the third method is to convolve an image that is uniformly focused in the depth direction by moving the focus lens or image sensor during the exposure time (that is, synonymous with equalizing blur at each depth).
- an image restoration process is performed on the obtained image using a blur pattern measured in advance or a blur pattern calculated by simulation. Thereby, the method generates an EDOF image.
- This method is called “Flexible DOF” (hereinafter referred to as “F-DOF”) (see, for example, Patent Document 1).
- the third F-DOF is a method that can obtain the best image quality, and has a high EDOF effect.
- the off-axis characteristics also depend on the lens characteristics themselves, so it is easy to improve performance.
- it is necessary to use an image side telecentric lens because the same subject needs to be folded on the same image position even if the focus position is moved during exposure.
- the oldest history is the microscope application.
- the EDOF effect can provide an omnifocal image (an image in which all subjects are in focus) without having an autofocus mechanism.
- the present invention solves the above-described conventional problems, and an object thereof is to provide an image restoration device, an imaging device, and an image restoration method capable of reducing the capacity of a memory for storing PSF data.
- an image restoration device includes a desired distance range to be focused by driving an imaging element or a lens that focuses light on the imaging element.
- a restored image with an extended depth of field is obtained by performing restoration processing on the swept image captured by the image sensor that is focus swept during exposure between the focal points on the image plane side corresponding to the sweep range.
- a first spread function (PSF) for performing a restoration process on the first sweep image that is the sweep image when the sweep range is the first range
- Restoration processing is performed on the second sweep image that is the sweep image when the sweep range is the second range that does not include the first range.
- a third sweep image that is the sweep image when the sweep range is a third range that is a range obtained by combining the first range and the second range.
- a PSF storage unit that stores any two PSFs out of the third PSFs for performing the restoration process
- a PSF calculation unit that calculates the remaining one and the first PSF are used to generate the restored image by performing a restoration process on the first sweep image, and the second PSF is used to generate the second
- the restored image is generated by performing restoration processing on the sweep image, and the restoration is performed by performing restoration processing on the third sweep image using the third PSF.
- an image restoring unit which generates an image.
- the image restoration apparatus can calculate the remaining PSF from two PSFs out of the first PSF, the second PSF, and the third PSF, so that the remaining PSF is stored in the PSF storage unit. There is no need to remember. Accordingly, the image restoration apparatus according to an aspect of the present invention can reduce the capacity of PSF data stored in the PSF storage unit.
- the PSF calculation unit may calculate the third PSF by adding the first PSF and the second PSF.
- the image restoration apparatus can calculate the third PSF from the first PSF and the second PSF, and thus it is not necessary to store the third PSF in the PSF storage unit. Accordingly, the image restoration apparatus according to an aspect of the present invention can reduce the capacity of PSF data stored in the PSF storage unit.
- An imaging apparatus includes the image restoration apparatus, the imaging element, the lens, and any one of the first range, the second range, and the third range as the sweep range.
- the lens or the imaging element so as to obtain the image plane side focal point corresponding to the sweep range and obtain an image obtained by performing a focus sweep between the image plane side focal points during exposure.
- a sweep imaging unit that causes the imaging device to capture the sweep image.
- the imaging apparatus can reduce the capacity of PSF data stored in the PSF storage unit.
- the sweep range determining unit may determine any one of the first range, the second range, and the third range as the sweep range based on a specified range specified by a user.
- the imaging apparatus can realize an imaging apparatus capable of sweeping the focusing range designated by the user.
- the sweep range determination unit when the specified range does not match any of the first range, the second range, and the third range, the first range, the second range, and the third range.
- the specified range may be included and the narrowest range may be determined as the sweep range.
- the imaging apparatus can reduce the capacity of PSF data stored in the PSF storage unit.
- the sweep imaging unit displaces a focal position on the subject side of the imaging element by displacing an image plane side distance that is a distance between the imaging element and the lens, and determines a focal position on the subject side.
- the displacement amount of the image plane side distance for sweeping in the first range may be equal to the displacement amount of the image plane side distance for sweeping the focal position on the subject side in the second range.
- the similarity between the first PSF and the second PSF can be increased. That is, if similarities are recognized within an allowable range, it is only necessary to store one type of PSF instead of two types of PSFs, so that the capacity of PSF data can be reduced.
- the present invention can be realized not only as such an image restoration device and an image pickup apparatus, but also as an image restoration method or an image pickup apparatus control method in which characteristic means included in the image restoration device or the image pickup apparatus are used as steps. Or, it can be realized as a program for causing a computer to execute such characteristic steps. Needless to say, such a program can be distributed via a non-transitory computer-readable recording medium such as a CD-ROM and a transmission medium such as the Internet.
- the present invention can be realized as a semiconductor integrated circuit (LSI) that realizes part or all of the functions of such an image restoration device or imaging device.
- LSI semiconductor integrated circuit
- the present invention can provide an image restoration device, an imaging device, and an image restoration method capable of reducing the capacity of a memory for storing PSF data.
- FIG. 1 is a block diagram of an imaging apparatus according to Embodiment 1 of the present invention.
- FIG. 2 is a flowchart of processing by the imaging apparatus according to Embodiment 1 of the present invention.
- FIG. 3 is a flowchart of the PSF calculation process according to Embodiment 1 of the present invention.
- FIG. 4 is a diagram showing an example of a PSF calculation process according to Embodiment 1 of the present invention.
- FIG. 5 is a diagram showing the subject distance and the image plane side distance according to Embodiment 1 of the present invention.
- FIG. 6 is a diagram showing a PSF model according to Embodiment 1 of the present invention.
- FIG. 1 is a block diagram of an imaging apparatus according to Embodiment 1 of the present invention.
- FIG. 2 is a flowchart of processing by the imaging apparatus according to Embodiment 1 of the present invention.
- FIG. 3 is a flowchart of the PSF calculation process according to Embodiment 1 of the present invention
- FIG. 7 is a diagram showing an example of a displacement pattern of the image plane side distance according to Embodiment 1 of the present invention.
- FIG. 8 is a graph showing the relationship between the subject distance and the image plane side distance according to Embodiment 1 of the present invention.
- FIG. 9 is a diagram showing the amount of displacement of the image plane side distance according to Embodiment 1 of the present invention.
- FIG. 10 is a block diagram of the image restoration apparatus according to Embodiment 1 of the present invention.
- FIG. 11 is a flowchart of processing by the image restoration apparatus according to Embodiment 1 of the present invention.
- FIG. 12 is a flowchart of the sweep range determination process according to Embodiment 2 of the present invention.
- FIG. 13 is a diagram showing an example of a sweep range determination process according to Embodiment 2 of the present invention.
- Embodiment 1 The imaging apparatus according to Embodiment 1 of the present invention calculates the PSF of the third range by adding the PSF of the first range and the PSF of the second range. As a result, the imaging apparatus does not need to store the third range of PSF in the memory, so the capacity of the PSF data stored in the memory can be reduced.
- FIG. 1 is a block diagram of an imaging apparatus 100 according to Embodiment 1 of the present invention.
- the imaging device 100 is, for example, a digital still camera or a digital video camera.
- the imaging apparatus 100 has an F-DOF function capable of sweeping an arbitrary focus range. Specifically, the imaging apparatus 100 generates a sweep image 131 in which the focus position is swept within a specified range 130 that is a focus range specified by the user. Then, the imaging apparatus 100 generates a restored image 132 that is in focus in the entire designated range 130 by performing restoration processing on the generated sweep image 131 using the PSF.
- the imaging apparatus 100 includes an imaging element 101, a lens 102, a sweep range determination unit 103, a sweep imaging unit 104, and an image restoration device 110.
- the imaging device 101 generates image data (sweep image 131) by photoelectrically converting incident light.
- the lens 102 condenses light on the image sensor 101.
- the sweep range determination unit 103 determines the sweep range 133 based on the specified range 130 specified by the user.
- the sweep range indicates a concept of a range on the subject side, not on the image plane side.
- the sweep range determining unit 103 determines the range indicated by the specified range 130 among the plurality of predetermined ranges as the sweep range 133.
- the sweep imaging unit 104 causes the image sensor 101 to capture the sweep image 131 while sweeping the focal position of the image sensor 101 within the sweep range 133. Specifically, the sweep imaging unit 104 obtains two image plane side in-focus positions corresponding to the sweep range 133, and an image obtained by performing a focus sweep between the two image plane side in-focus positions during exposure is obtained. By driving the lens 102 or the image sensor 102, the sweep image is captured by the image sensor. That is, the sweep image 131 is an image captured by the image sensor 101 during a period in which the focus position on the subject side of the image sensor 101 is sweeping the sweep range 133.
- the sweep image 131 is focus swept during exposure between the focal points on the image plane side corresponding to the sweep range including the desired distance range to be focused by driving the image sensor 101 or the lens 102. It is an image.
- the whole lens 102 may be moved, or a focus lens that constitutes a part of the lens may be driven.
- the image restoration apparatus 110 generates a restored image 132 with an extended depth of field by performing a restoration process on the sweep image 131. Specifically, the image restoration device 110 performs restoration processing on the sweep image 131 using the PSF corresponding to the sweep range 133.
- the restored image 132 is an image that is in focus in the entire sweep range 133.
- the image restoration apparatus 110 includes a PSF storage unit 111, a PSF calculation unit 112, and an image restoration unit 113.
- the PSF storage unit 111 stores a plurality of PSF data corresponding to a plurality of sweep ranges 133.
- the PSF storage unit 111 stores four PSF data (first range PSF 121, second range PSF 122, third range PSF 123, and fourth range PSF 124) will be described.
- the sweep range that can be specified by the user is larger than the number of PSF data stored in the PSF storage unit 111.
- the PSF calculation unit 112 calculates new PSF data corresponding to the sweep range 133 by adding and normalizing two or more of the plurality of PSF data stored in the PSF storage unit 111.
- the image restoration unit 113 performs a restoration process on the sweep image 131 using any one of the plurality of PSF data stored in the PSF storage unit 111 or the PSF data calculated by the PSF calculation unit 112. Thus, the restored image 132 is generated.
- FIG. 2 is a flowchart of the imaging process performed by the imaging apparatus 100.
- the sweep range determination unit 103 determines the sweep range 133 based on the specified range 130 specified by the user (S101). For example, the user selects one of a plurality of predetermined sweep ranges. The sweep range determination unit 103 determines the range designated by the user as the sweep range 133.
- the image sensor 101 is caused to capture the sweep image 131 while the focus position of the image sensor 101 is swept within the sweep range 133 (S102).
- the PSF calculation unit 112 calculates PSF data corresponding to the sweep range 133 (S103).
- step S103 the process of step S103 will be described in detail.
- FIG. 3 is a flowchart of the PSF calculation process performed by the PSF calculation unit 112.
- FIG. 4 is a diagram illustrating an example of a PSF calculation process performed by the PSF calculation unit 112.
- the PSF calculation unit 112 acquires the sweep range 133 of the sweep image 131 determined by the sweep range determination unit 103 (S111).
- the first range PSF 121 stored in the PSF storage unit 111 corresponds to a sweep range of 1 m to 3 m
- the second range PSF 122 corresponds to a sweep range of 3 m to 5 m
- the third range PSF 123 is 5 m to 10 m
- the fourth range PSF 124 corresponds to a sweep range from 10 m to infinity.
- the PSF storage unit 111 has only PSF data corresponding to each partial range obtained by dividing the sweepable range (here, 1 m to infinity).
- the sweep ranges that can be specified by the user are the plurality of partial ranges and a plurality of types of ranges obtained by combining two or more consecutive partial ranges of the plurality of partial ranges.
- the PSF calculation unit 112 determines whether or not PSF data corresponding to the sweep range 133 acquired in step S111 is stored in the PSF storage unit 111 (S112).
- the sweep range 133 is any one of 1 m to 3 m, 3 m to 5 m, 5 m to 10 m, and 10 m to infinity
- the PSF data corresponding to the sweep range 133 is stored in the PSF. It is determined that the data is stored in the unit 111 (Yes in S112).
- the PSF calculation unit 112 determines that the PSF data corresponding to the sweep range 133 is not stored in the PSF storage unit 111 (No in S112). ).
- the PSF calculation unit 112 acquires PSF data corresponding to the sweep range 133 from the PSF storage unit 111 (S113). ). Then, the PSF calculation unit 112 passes the acquired PSF data to the image restoration unit 113.
- the PSF calculation unit 112 converts the PSF data corresponding to a plurality of partial ranges included in the sweep range 133 to the PSF. Obtained from the storage unit 111 (S114).
- the PSF calculation unit 112 calculates PSF data corresponding to the sweep range 133 by adding and normalizing the PSF data corresponding to the acquired plurality of partial ranges (S115 and S116). Then, the PSF calculation unit 112 passes the calculated PSF data to the image restoration unit 113.
- the PSF calculation unit 112 corresponds to 1 m to 5 m by adding and normalizing the first range PSF 121 and the second range PSF 122. PSF data is calculated.
- the PSF calculation unit 112 adds and normalizes the first range PSF 121, the second range PSF 122, and the third range PSF 123, and thereby PSF data corresponding to 1 m to 10 m. Is calculated.
- the image restoration unit 113 After the PSF calculation process (S103), the image restoration unit 113 performs the restoration process on the sweep image 131 using the PSF data corresponding to the sweep range 133 passed from the PSF calculation unit 112, thereby restoring the restored image. 132 is generated (S104). Then, the image restoration unit 113 outputs the generated restored image 132 to the outside of the imaging device 100 or stores it in a storage unit (not shown) included in the imaging device 100.
- the imaging apparatus 100 according to Embodiment 1 of the present invention can calculate PSF data corresponding to a new range from a plurality of PSF data stored in the PSF storage unit 111. Thereby, the PSF storage unit 111 does not need to store PSF data corresponding to the new range. Thereby, the imaging device 100 according to Embodiment 1 of the present invention can reduce the capacity of the PSF data stored in the PSF storage unit 111.
- steps S102 to S104 shown in FIG. 2 is performed every time one still image is taken in the case of still image shooting, and every frame in the case of moving image shooting.
- the process of step S101 may be performed every time one still image is shot, or the process of step S101 is performed in advance, and then the sweep range 133 set in step S101. A plurality of still images may be taken using.
- the process of step S101 is performed every time one or more times of moving image shooting.
- the sweep range determination unit 103 determines the sweep range 133 based on the specified range 130 specified by the user.
- the sweep range determination unit 103 corresponds to the detection result of the sensor included in the imaging apparatus 100.
- the sweep range 133 may be automatically determined.
- the sweep range determination unit 103 may determine the sweep range 133 according to the distance to the subject used in the autofocus function or the like, or the state of the field (for example, brightness). Further, the distance to the subject and the state of the object scene may be calculated from the captured image. Further, the sweep range determination unit 103 may determine the sweep range 133 based on these information and the specified range 130, or may adjust the sweep range 133 according to these information.
- the determination or adjustment processing of the sweep range 133 is performed every one or a plurality of frames.
- the number of PSF data stored in the PSF storage unit 111 may be any number as long as it is two or more.
- the number of the plurality of partial ranges obtained by dividing the sweepable range may be two or more. That is, the imaging apparatus 100 has, as the sweep range 133, at least a first range, a second range that is continuous with the first range, and a third range that is a range obtained by combining the first range and the second range. It is only necessary to be able to select. In this case, the PSF storage unit 111 only needs to store PSF data corresponding to the first range and the second range.
- the second range may be a range that does not include the first range.
- the number of partial ranges is preferably 2 to the power of n (n is an integer of 1 or more).
- n is an integer of 1 or more.
- the number of the partial ranges is 4, 8, 16, 32, or 64.
- FIG. 5 is a diagram illustrating a relationship among the subject, the lens 102, and the image sensor 101.
- Equation 1 the distance between the subject M and the lens 102 is a subject distance u, and the distance between the lens 102 and the image sensor 101 is an image plane side distance v.
- equation 1 the following equation (Equation 1) is generally established from the lens formula for the subject distance u, the image plane side distance v, and the focal length f.
- the diameter b of the blur is expressed by the following (formula 2).
- P that is PSF is expressed by the following (formula 3).
- ⁇ (x) is a Rectangle function, which is 1 when
- FIG. 6 is a diagram showing PSF in the lens model of this Pillbox.
- IPSF the PSF used for the sweep image
- the IPSF when the focus position is swept between time 0 and time T is expressed by the following (formula 4).
- IPSF is expressed by integration. That is, the sum of the IPSF from time T1 to time T2 and the IPSF from time T2 to time T3 is the IPSF from time T1 to time T3.
- the integral term function itself is unchanged under the rule that the sweep operation p (t) is constant speed. Therefore, the IPSF from time T1 to time T3 is equivalent to setting the integration start time and end time to time T1 and time T3.
- the PSF data of the range combining the plurality of partial ranges can be calculated.
- FIG. 7 is a diagram showing the state of displacement of the image plane side distance v during exposure and the exposure amount.
- the sweep imaging unit 104 displaces the subject distance u, which is the focal position of the image sensor 101 on the subject side, by displacing the image plane side distance v at a constant speed.
- the sweep imaging unit 104 may displace the image plane side distance v by displacing the position of the lens 102, or it may displace the image plane side distance v by displacing the position of the image sensor 101.
- the image plane side distance v may be displaced by displacing both the lens 102 and the image sensor 101.
- FIG. 8 is a graph showing the relationship between the image plane side distance v and the subject distance u when the focal length f is 18 [mm]. As shown in FIG. 8, even if the image plane side distance v is displaced at a constant speed, the subject distance u is not displaced at a constant speed.
- the partial range that is, the range of the subject distance u is not equally spaced.
- the plurality of partial ranges are determined so that the displacement amount of the image plane side distance v corresponding to the plurality of partial ranges is constant.
- FIG. 9 is a diagram showing a displacement amount of the image plane side distance v according to Embodiment 1 of the present invention.
- the displacement amount dv of the image plane side distance v for sweeping each partial range is constant.
- the range of the image plane side distance v0 to v1 shown in FIG. 9 corresponds to a sweep range from 10 m to infinity
- the range of the image plane side distance v1 to v2 corresponds to a sweep range of 5 m to 10 m.
- the range from the distance v2 to v3 corresponds to the sweep range from 3 m to 5 m
- the range from the image plane side distance v3 to v4 corresponds to the sweep range from 1 m to 3 m.
- the S / N ratio in each partial range can be made the same level. Therefore, the PSF similarity corresponding to each partial range can be increased. Thereby, since the processing amount etc. with respect to each partial range can be made comparable, various control can be performed efficiently.
- FIG. 10 is a block diagram of the image restoration apparatus 110 according to Embodiment 1 of the present invention.
- the image restoration device 110 can be realized as a semiconductor integrated circuit used in the imaging device 100 described above, for example. Further, it is also possible to realize a device independent of the imaging device that performs the restoration process on the sweep image 131 captured by the imaging device.
- the image restoration apparatus 110 may be mounted on an apparatus such as a personal computer.
- FIG. 11 is a flowchart of the imaging process performed by the image restoration device 110. Note that the processing in steps S103 and S104 shown in FIG. 11 is the same as the processing in steps S103 and S104 shown in FIG.
- the imaging apparatus 100 and the image restoration apparatus 110 according to Embodiment 1 of the present invention calculate the PSF data of a certain sweep range 133 from the PSF data of a plurality of partial ranges constituting the sweep range 133. Thereby, the imaging device 100 and the image restoration device 110 according to Embodiment 1 of the present invention can reduce the capacity of the PSF data stored in the PSF storage unit 111.
- the present invention to portable cameras such as digital still cameras, digital video cameras, and mobile phone cameras, the memory capacity of portable cameras with limited memory capacity can be reduced. is there.
- the user selects any one of a plurality of predetermined sweep ranges, but in the second embodiment, the user can designate an arbitrary sweep range.
- the imaging apparatus 100 according to Embodiment 2 of the present invention is different from Embodiment 1 in the function of the sweep range determination unit 103.
- FIG. 12 is a flowchart of the sweep range determination process (step S101 in FIG. 2) by the sweep range determination unit 103 according to the second embodiment.
- FIG. 13 is a diagram illustrating an example of the sweep range determination process.
- the sweep range determination unit 103 acquires a specified range 130 from the user (S121).
- the designated range 130 is, for example, an arbitrary sweep range designated by the user.
- the sweep range determining unit 103 determines whether or not the designated range 130 matches any of a plurality of pre-registered ranges (hereinafter, registered ranges) (S122).
- the plurality of registration ranges are, for example, the ten types of ranges illustrated in FIG. 4 described in the first embodiment.
- the sweep range determining unit 103 determines the designated range 130 as the sweep range 133 (S123).
- the sweep range determining unit 103 determines whether the entire designated range 130 is included in the sweepable range (for example, 1 m to infinity). It is determined whether or not (S124).
- the sweep range determining unit 103 includes all of the designated ranges 130 among the plurality of registered ranges and the narrowest range as the sweep range 133. Determine (S125).
- the sweep range determining unit 103 determines from 3 m to infinity as the sweep range 133.
- the sweep range determining unit 103 determines the range closest to the designated range 130 among the plurality of registered ranges as the sweep range 133 (S126). .
- the sweep range determining unit 103 determines 1 m to 3 m as the sweep range 133.
- the sweep range determining unit 103 determines the nearest end of the sweep range 133 as the nearest end of the sweepable range. Set to the edge.
- the sweep range determination unit 103 can sweep the farthest end of the sweep range 133 when the farthest end of the specified range 130 (the end on the side far from the imaging apparatus 100) is farther than the farthest end of the sweepable range. Set to the farthest end of the range. In the example of FIG.
- the farthest end of the designated range 130 cannot be farther than the farthest end of the sweepable range, but for example, the farthest of the sweepable range Such a case occurs when the edges are finite.
- the sweep range determination unit 103 may determine, as the sweep range 133, the range closest to the specified range 130 among the plurality of registered ranges.
- the sweep range determination unit 103 sets a threshold value in each partial range, and when the nearest end of the designated range 130 is less than the threshold value, the nearest end of the partial range is set as the nearest end of the sweep range 133 and is designated.
- the nearest end of the range 130 is equal to or greater than the threshold, the farthest end of the partial range may be the nearest end of the sweep range 133.
- the sweep range determining unit 103 sets the nearest end of the partial range as the farthest end of the sweep range 133, and the farthest end of the designated range 130 is the relevant end.
- the farthest end of the partial range may be the closest end of the sweep range 133.
- the threshold For example, 4.5 m is set as the threshold for a partial range of 3 m to 5 m. In this case, as shown in FIG. 13, when the designated range 130 is from 4 m to infinity, the nearest end (4 m) of the designated range 130 is less than the threshold (4.5 m). Is determined as the closest end of the sweep range 133.
- the method of determining a range wider than the designated range 130 as the sweep range 133 has an advantage that a restored image 132 that is in focus with respect to at least the range designated by the user can be generated.
- the imaging apparatus 100 determines the narrowest range as the sweep range 133 including all the specified ranges 130 among the plurality of registered ranges. Accordingly, the imaging apparatus 100 can reduce the types of sweep ranges that are actually subjected to sweep imaging, and can therefore reduce the types of necessary PSF data. Thereby, the imaging apparatus 100 can reduce the capacity of the PSF data stored in the PSF storage unit 111.
- the imaging apparatus 100 can generate the restored image 132 that is focused on at least the range designated by the user by determining the sweep range 133 as a range wider than the designated range 130. In this way, the imaging apparatus 100 can capture an image that meets the user's request and can reduce the capacity of PSF data stored in the PSF storage unit 111.
- the imaging device according to the embodiment of the present invention has been described above, but the present invention is not limited to this embodiment.
- new PSF data is generated by adding and normalizing two PSF data.
- new PSF data is obtained. It may be generated.
- the imaging apparatus 100 stores PSF data in the range of 1 m to 3 m and PSF data in the range of 1 m to 5 m, and subtracts PSF data in the range of 1 m to 3 m from the PSF data in the range of 1 m to 5 m.
- PSF data of 3 to 5 m may be generated.
- the imaging apparatus 100 includes the first range of PSF data, the second range of PSF data that does not include the first range, and the third range of PSF data that is a combination of the first range and the second range.
- One of the two PSF data may be stored, and the remaining one of the three PSF data may be calculated by adding or subtracting the two PSF data.
- each processing unit included in the imaging device and the image restoration device according to the above embodiment is typically realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them.
- circuits are not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor.
- An FPGA Field Programmable Gate Array
- reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.
- imaging apparatus or the image restoration apparatus may be realized by a processor such as a CPU executing a program.
- the present invention may be the above program or a non-transitory computer-readable recording medium on which the above program is recorded.
- the program can be distributed via a transmission medium such as the Internet.
- connection relationship between the components is exemplified for specifically explaining the present invention, and the connection relationship for realizing the function of the present invention is not limited to this.
- the present invention can be applied to an imaging apparatus such as a digital still camera and a digital video camera.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Studio Devices (AREA)
Abstract
Description
本発明の実施の形態1に係る撮像装置は、第1範囲のPSFと第2範囲のPSFとを加算することにより、第3範囲のPSFを算出する。これにより、当該撮像装置は、第3範囲のPSFをメモリに記憶しておく必要がないのでメモリに記憶しておくPSFデータの容量を削減できる。
本発明の実施の形態2では、上述した実施の形態1に係る撮像装置100の変形例について説明する。なお、以下では、実施の形態1との相違点を主に説明し、重複する説明は省略する。
101 撮像素子
102 レンズ
103 スイープ範囲決定部
104 スイープ撮像部
110 画像復元装置
111 PSF記憶部
112 PSF算出部
113 画像復元部
121 第1範囲PSF
122 第2範囲PSF
123 第3範囲PSF
124 第4範囲PSF
130 指定範囲
131 スイープ画像
132 復元画像
133 スイープ範囲
Claims (9)
- 撮像素子又は前記撮像素子に光を集光するレンズが駆動されることにより、合焦させたい所望の距離範囲を含むスイープ範囲に対応する像面側合焦点間を露光中にフォーカススイープされた、当該撮像素子により撮像されたスイープ画像に対して復元処理を施すことにより、被写界深度が拡張された復元画像を生成する画像復元装置であって、
(1)前記スイープ範囲が第1範囲である場合の前記スイープ画像である第1スイープ画像に対して復元処理を施すための第1PSF(Point Spread Function)と、(2)前記スイープ範囲が、前記第1範囲を含まない第2範囲である場合の前記スイープ画像である第2スイープ画像に対して復元処理を施すための第2PSFと、(3)前記スイープ範囲が、前記第1範囲と前記第2範囲とを合わせた範囲である第3範囲である場合の前記スイープ画像である第3スイープ画像に対して復元処理を施すための第3PSFとのうち、いずれか2つのPSFを記憶するPSF記憶部と、
前記2つのPSFを加算又は減算することにより、前記第1PSF、前記第2PSF及び前記第3PSFのうち残りの一つを算出するPSF算出部と、
前記第1PSFを用いて、前記第1スイープ画像に対して復元処理を施すことにより前記復元画像を生成し、前記第2PSFを用いて、前記第2スイープ画像に対して復元処理を施すことにより前記復元画像を生成し、前記第3PSFを用いて、前記第3スイープ画像に対して復元処理を施すことにより前記復元画像を生成する画像復元部とを備える
画像復元装置。 - 前記PSF算出部は、前記第1PSFと前記第2PSFとを加算することにより、前記第3PSFを算出する
請求項1記載の画像復元装置。 - 請求項1又は2記載の画像復元装置と、
前記撮像素子と、
前記レンズと、
前記第1範囲、前記第2範囲及び前記第3範囲のいずれかを、前記スイープ範囲に決定するスイープ範囲決定部と、
前記スイープ範囲に対応する前記像面側合焦点を求め、露光中に前記像面側合焦点間をフォーカススイープさせた画像が得られるよう前記レンズ又は前記撮像素子を駆動させることにより、前記撮像素子に前記スイープ画像を撮像させるスイープ撮像部とを備える
撮像装置。 - 前記スイープ範囲決定部は、ユーザにより指定された指定範囲に基づき、前記第1範囲、前記第2範囲及び前記第3範囲のいずれかを、前記スイープ範囲に決定する
請求項3記載の撮像装置。 - 前記スイープ範囲決定部は、前記指定範囲が、前記第1範囲、前記第2範囲及び前記第3範囲のいずれにも一致しない場合、前記第1範囲、前記第2範囲及び前記第3範囲のうち、前記指定範囲を全て含み、かつ最も狭い範囲を前記スイープ範囲に決定する
請求項4記載の撮像装置。 - 前記スイープ撮像部は、前記撮像素子と前記レンズとの距離である像面側距離を変位させることにより、前記撮像素子の前記被写体側の焦点位置を変位させ、
前記被写体側の焦点位置を前記第1範囲でスイープさせるための前記像面側距離の変位量は、前記被写体側の焦点位置を前記第2範囲でスイープさせるための前記像面側距離の変位量と等しい
請求項3記載の撮像装置。 - 撮像素子又は前記撮像素子に光を集光するレンズが駆動されることにより、合焦させたい所望の距離範囲を含むスイープ範囲に対応する像面側合焦点間を露光中にフォーカススイープされた、当該撮像素子により撮像されたスイープ画像に対して復元処理を施すことにより、被写界深度が拡張された復元画像を生成する画像復元方法であって、
(1)前記スイープ範囲が第1範囲である場合の前記スイープ画像である第1スイープ画像に対して復元処理を施すための第1PSF(Point Spread Function)と、(2)前記スイープ範囲が、前記第1範囲を含まない第2範囲である場合の前記スイープ画像である第2スイープ画像に対して復元処理を施すための第2PSFと、(3)前記スイープ範囲が、前記第1範囲と前記第2範囲とを合わせた範囲である第3範囲である場合の前記スイープ画像である第3スイープ画像に対して復元処理を施すための第3PSFとのうち、いずれか2つのPSFを加算又は減算することにより、前記第1PSF、前記第2PSF及び前記第3PSFのうち残りの一つを算出するPSF算出ステップと、
前記第1PSFを用いて、前記第1スイープ画像に対して復元処理を施すことにより前記復元画像を生成し、前記第2PSFを用いて、前記第2スイープ画像に対して復元処理を施すことにより前記復元画像を生成し、前記第3PSFを用いて、前記第3スイープ画像に対して復元処理を施すことにより前記復元画像を生成する画像復元ステップとを含む
画像復元方法。 - 請求項7記載の画像復元方法をコンピュータに実行させるための
プログラム。 - 撮像素子又は前記撮像素子に光を集光するレンズが駆動されることにより、合焦させたい所望の距離範囲を含むスイープ範囲に対応する像面側合焦点間を露光中にフォーカススイープされた、当該撮像素子により撮像されたスイープ画像に対して復元処理を施すことにより、被写界深度が拡張された復元画像を生成する集積回路であって、
(1)前記スイープ範囲が第1範囲である場合の前記スイープ画像である第1スイープ画像に対して復元処理を施すための第1PSF(Point Spread Function)と、(2)前記スイープ範囲が、前記第1範囲を含まない第2範囲である場合の前記スイープ画像である第2スイープ画像に対して復元処理を施すための第2PSFと、(3)前記スイープ範囲が、前記第1範囲と前記第2範囲とを合わせた範囲である第3範囲である場合の前記スイープ画像である第3スイープ画像に対して復元処理を施すための第3PSFとのうち、いずれか2つのPSFを記憶するPSF記憶部と、
前記2つのPSFを加算又は減算することにより、前記第1PSF、前記第2PSF及び前記第3PSFのうち残りの一つを算出するPSF算出部と、
前記第1PSFを用いて、前記第1スイープ画像に対して復元処理を施すことにより前記復元画像を生成し、前記第2PSFを用いて、前記第2スイープ画像に対して復元処理を施すことにより前記復元画像を生成し、前記第3PSFを用いて、前記第3スイープ画像に対して復元処理を施すことにより前記復元画像を生成する画像復元部とを備える
集積回路。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP12742400.0A EP2672696B1 (en) | 2011-01-31 | 2012-01-30 | Image restoration device, imaging device, and image restoration method |
US13/636,385 US8767092B2 (en) | 2011-01-31 | 2012-01-30 | Image restoration device, imaging apparatus, and image restoration method |
CN201280000943.3A CN102804751B (zh) | 2011-01-31 | 2012-01-30 | 图像恢复装置、摄像装置以及图像恢复方法 |
JP2012527551A JP5832433B2 (ja) | 2011-01-31 | 2012-01-30 | 画像復元装置、撮像装置及び画像復元方法 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011018949 | 2011-01-31 | ||
JP2011-018949 | 2011-01-31 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012105222A1 true WO2012105222A1 (ja) | 2012-08-09 |
Family
ID=46602447
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2012/000582 WO2012105222A1 (ja) | 2011-01-31 | 2012-01-30 | 画像復元装置、撮像装置及び画像復元方法 |
Country Status (5)
Country | Link |
---|---|
US (1) | US8767092B2 (ja) |
EP (1) | EP2672696B1 (ja) |
JP (1) | JP5832433B2 (ja) |
CN (1) | CN102804751B (ja) |
WO (1) | WO2012105222A1 (ja) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2020016922A1 (ja) * | 2018-07-17 | 2020-01-23 | オリンパス株式会社 | 撮像システム、処理装置および内視鏡 |
US10718715B2 (en) | 2016-03-25 | 2020-07-21 | Olympus Corporation | Microscopy system, microscopy method, and computer-readable storage medium |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2954371A4 (en) * | 2013-03-15 | 2016-11-02 | Nokia Technologies Oy | APPARATUS, METHOD AND COMPUTER PROGRAM FOR IMAGE CAPTURE |
JP6236908B2 (ja) * | 2013-06-21 | 2017-11-29 | 株式会社リコー | 撮像装置、撮像システムおよび撮像方法 |
JP6318520B2 (ja) | 2013-09-27 | 2018-05-09 | 株式会社リコー | 撮像装置、撮像システムおよび撮像方法 |
JP6317635B2 (ja) * | 2014-06-30 | 2018-04-25 | 株式会社東芝 | 画像処理装置、画像処理方法及び画像処理プログラム |
JP6908039B2 (ja) * | 2016-06-17 | 2021-07-21 | ソニーグループ株式会社 | 画像処理装置、画像処理方法、プログラム、及び画像処理システム |
US11422355B2 (en) | 2020-07-22 | 2022-08-23 | Sartorius Bioanalytical Instruments, Inc. | Method and system for acquisition of fluorescence images of live-cell biological samples |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2007121896A (ja) * | 2005-10-31 | 2007-05-17 | Nikon Corp | 焦点検出装置および光学システム |
US20080013941A1 (en) | 2006-07-14 | 2008-01-17 | Micron Technology, Inc. | Method and apparatus for increasing depth of field for an imager |
JP2012005063A (ja) * | 2010-06-21 | 2012-01-05 | Sony Corp | 撮像装置および方法、並びにプログラム |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3191928B2 (ja) | 1988-02-23 | 2001-07-23 | オリンパス光学工業株式会社 | 画像入出力装置 |
DE3905619C2 (de) | 1988-02-23 | 2000-04-13 | Olympus Optical Co | Bildeingabe-/Ausgabevorrichtung |
DE3931934C2 (de) | 1988-10-03 | 1994-11-10 | Olympus Optical Co | Bild-Ein/Ausgabevorrichtung |
GB8917224D0 (en) | 1989-07-27 | 1989-09-13 | Unilever Plc | Dispensing system |
JPH03191928A (ja) | 1989-12-20 | 1991-08-21 | Matsushita Electric Ind Co Ltd | 電気湯沸かし器 |
JP3084130B2 (ja) | 1992-05-12 | 2000-09-04 | オリンパス光学工業株式会社 | 画像入力装置 |
US8005314B2 (en) * | 2005-12-09 | 2011-08-23 | Amnis Corporation | Extended depth of field imaging for high speed object analysis |
JP2004153497A (ja) | 2002-10-30 | 2004-05-27 | Kyocera Corp | ディジタルカメラの自動露出制御システム |
JP2006003803A (ja) * | 2004-06-21 | 2006-01-05 | Olympus Corp | 合焦情報取得装置及び合焦情報取得方法 |
JP5012135B2 (ja) * | 2007-03-28 | 2012-08-29 | コニカミノルタアドバンストレイヤー株式会社 | 超深度画像生成装置 |
WO2009120718A1 (en) * | 2008-03-24 | 2009-10-01 | The Trustees Of Columbia University In The City Of New York | Methods, systems, and media for controlling depth of field in images |
US8131097B2 (en) * | 2008-05-28 | 2012-03-06 | Aptina Imaging Corporation | Method and apparatus for extended depth-of-field image restoration |
CN101373271B (zh) * | 2008-08-15 | 2011-08-10 | 北京泰邦天地科技有限公司 | 一种摄像***和摄像方法 |
US8289438B2 (en) * | 2008-09-24 | 2012-10-16 | Apple Inc. | Using distance/proximity information when applying a point spread function in a portable media device |
CN102884802B (zh) * | 2010-03-24 | 2014-03-12 | 富士胶片株式会社 | 三维成像装置和视点图像恢复方法 |
US8773570B2 (en) * | 2010-06-17 | 2014-07-08 | Panasonic Corporation | Image processing apparatus and image processing method |
CN101937561A (zh) * | 2010-09-19 | 2011-01-05 | 华中科技大学 | 一种天基大视场成像点源/斑状目标的图像恢复增强方法 |
JP5976676B2 (ja) * | 2011-01-14 | 2016-08-24 | ソニー株式会社 | レンズ部の縦の色収差を利用したイメージングシステム及びその操作方法 |
JP5882898B2 (ja) * | 2011-03-14 | 2016-03-09 | パナソニック株式会社 | 撮像装置、撮像方法、集積回路、コンピュータプログラム |
TWI467226B (zh) * | 2011-11-15 | 2015-01-01 | Ind Tech Res Inst | 相位物體顯微系統 |
-
2012
- 2012-01-30 CN CN201280000943.3A patent/CN102804751B/zh active Active
- 2012-01-30 JP JP2012527551A patent/JP5832433B2/ja active Active
- 2012-01-30 WO PCT/JP2012/000582 patent/WO2012105222A1/ja active Application Filing
- 2012-01-30 EP EP12742400.0A patent/EP2672696B1/en active Active
- 2012-01-30 US US13/636,385 patent/US8767092B2/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2007121896A (ja) * | 2005-10-31 | 2007-05-17 | Nikon Corp | 焦点検出装置および光学システム |
US20080013941A1 (en) | 2006-07-14 | 2008-01-17 | Micron Technology, Inc. | Method and apparatus for increasing depth of field for an imager |
JP2012005063A (ja) * | 2010-06-21 | 2012-01-05 | Sony Corp | 撮像装置および方法、並びにプログラム |
Non-Patent Citations (1)
Title |
---|
See also references of EP2672696A4 |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10718715B2 (en) | 2016-03-25 | 2020-07-21 | Olympus Corporation | Microscopy system, microscopy method, and computer-readable storage medium |
WO2020016922A1 (ja) * | 2018-07-17 | 2020-01-23 | オリンパス株式会社 | 撮像システム、処理装置および内視鏡 |
JPWO2020016922A1 (ja) * | 2018-07-17 | 2021-08-26 | オリンパス株式会社 | 撮像システム、処理装置および内視鏡 |
US11258941B2 (en) | 2018-07-17 | 2022-02-22 | Olympus Corporation | Imaging system adapted to perform focal sweep imaging, processing apparatus, and endoscope |
JP7036923B2 (ja) | 2018-07-17 | 2022-03-15 | オリンパス株式会社 | 撮像システム、処理装置および内視鏡 |
Also Published As
Publication number | Publication date |
---|---|
EP2672696A1 (en) | 2013-12-11 |
EP2672696B1 (en) | 2018-06-13 |
US20130010160A1 (en) | 2013-01-10 |
EP2672696A4 (en) | 2015-07-08 |
CN102804751B (zh) | 2016-08-03 |
JP5832433B2 (ja) | 2015-12-16 |
CN102804751A (zh) | 2012-11-28 |
JPWO2012105222A1 (ja) | 2014-07-03 |
US8767092B2 (en) | 2014-07-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5832433B2 (ja) | 画像復元装置、撮像装置及び画像復元方法 | |
JP5367094B2 (ja) | 撮像装置及びその制御方法 | |
KR101233013B1 (ko) | 화상 촬영 장치 및 그 거리 연산 방법과 합초 화상 취득 방법 | |
JP5588461B2 (ja) | 撮像装置および撮像方法 | |
JP6210333B2 (ja) | 距離測定装置、及び、距離測定方法 | |
JP5882898B2 (ja) | 撮像装置、撮像方法、集積回路、コンピュータプログラム | |
WO2007058100A1 (ja) | 合焦検出装置 | |
US9674420B2 (en) | Lens apparatus and image capturing apparatus | |
US10536624B2 (en) | Image pickup apparatus and image pickup method | |
JP2011217368A (ja) | 撮像装置、カメラ本体及び交換レンズ | |
JP2009111774A (ja) | 撮像装置および画像処理方法 | |
JP6642998B2 (ja) | 像ズレ量算出装置、撮像装置、および像ズレ量算出方法 | |
JP2012217090A (ja) | 撮像装置、撮影レンズユニット、および撮像ユニット | |
JP2011053255A (ja) | オートフォーカス装置、オートフォーカス方法及びプログラム | |
WO2013172030A1 (ja) | 撮像装置、撮影レンズユニット、および撮像ユニット | |
JP6645690B2 (ja) | 自動焦点調節装置、撮像装置、および自動焦点調節方法 | |
JP5581177B2 (ja) | 撮像位置調整装置および撮像装置 | |
JP2018045101A (ja) | 焦点調節装置、焦点調節方法、プログラム | |
WO2020054684A1 (en) | Lens apparatus, image pickup apparatus, processing apparatus, and camera apparatus | |
JP2013246315A (ja) | オートフォーカスシステムおよびプログラム | |
JP5741671B2 (ja) | 撮像装置及び撮像方法 | |
JP2020122923A (ja) | レンズ装置および撮像装置 | |
JP2019207350A (ja) | 撮像装置およびその制御方法 | |
JP2018110287A (ja) | 撮像装置、その制御方法、プログラム、及び記録媒体 | |
JP2009094662A (ja) | 固体撮像装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201280000943.3 Country of ref document: CN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2012527551 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2012742400 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13636385 Country of ref document: US |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12742400 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |