WO2012063469A1 - 画像処理装置、画像処理方法およびプログラム - Google Patents
画像処理装置、画像処理方法およびプログラム Download PDFInfo
- Publication number
- WO2012063469A1 WO2012063469A1 PCT/JP2011/006230 JP2011006230W WO2012063469A1 WO 2012063469 A1 WO2012063469 A1 WO 2012063469A1 JP 2011006230 W JP2011006230 W JP 2011006230W WO 2012063469 A1 WO2012063469 A1 WO 2012063469A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- feature point
- movement amount
- unit
- correction
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
- H04N23/6811—Motion detection based on the image signal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
- H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20164—Salient point detection; Corner detection
Definitions
- the present invention relates to a correction technique of an image captured by a digital still camera, a video camera, a wearable camera or the like.
- a method of correcting, by image processing, the shake (position shift between images) of an image captured using a super wide-angle optical system such as a fisheye optical system It is a method of detecting a motion vector represented by the MPEG technology or the like by using information of an object commonly shown between images, using two images captured consecutively in time, This is a method of estimating and correcting camera shake between images).
- the method using such a motion vector has limitations in terms of accuracy and computational cost due to the nature of the algorithm that utilizes the region of the image to detect the motion vector.
- the upper limit of the detectable magnitude of camera shake is set in advance, and for example, detection is performed for large shakes such as shakes included in a video shot during walking or a video shot with a finderless shot. I could not That is, there is a sway of a size that can not be handled by the method using the motion vector, and there is a problem that such sway can not be corrected.
- a feature point-based matching method as a method capable of correcting the magnitude of fluctuation that can not be handled by the method using motion vectors. This uses some characteristic points on the subject that exist in common between two images taken consecutively in time.
- FIGS. 1A to 1D are diagrams for explaining a feature point based matching method.
- an image captured temporally earlier is referred to as an image t-1
- an image captured temporally later is referred to as an image t.
- FIG. 1A is a diagram illustrating an image t-1 and an image t captured temporally after the image t-1.
- FIG. 1B is a diagram showing feature points extracted from the image t-1 and the image t shown in FIG. 1A.
- FIG. 1C is a diagram showing types of characteristics of feature points extracted from the image t-1 and the image t shown in FIG. 1B.
- FIG. 1D is a diagram showing matching of feature points extracted from the image t-1 and the image t shown in FIG. 1B.
- the feature point is a feature point on an image that can be detected by image processing.
- the image t-1 shown in FIG. 1A and the pixel with high contrast on the image t are selected as the feature points shown in FIG. 1B.
- feature points present at corners where the contrast is extremely high are easily extracted in common between images (image t-1 and image t), but feature points such that the contrast is not so high are images It is difficult to extract in common between images (image t-1 and image t).
- the feature points shown in FIG. 1B include feature points (feature points indicated by .smallcircle. In FIG. 1C) obtained from an area commonly shown between images (image t-1 and image t) and between images (images Feature points obtained from an area commonly captured in t-1 and image t) but whose positions are changed between images (image t-1 and image t) (feature points indicated by ⁇ in FIG. 1C) There is. Further, the feature points shown in FIG. 1B also include feature points (feature points indicated by x in FIG. 1C) obtained from an area which is not common between images (image t-1 and image t). The feature points which can be matched among the feature points shown in FIG. 1B are the feature points obtained from the area commonly shown between the images (image t-1 and image t) (features indicated by .smallcircle. In FIG. 1C) point).
- a rotation matrix is generated from a combination of two sets of feature points selected from the feature points extracted in the image t-1 and the feature points extracted in the image t. Then, in order to check whether the generated rotation matrix is correct, the feature points of the image t-1 other than the combination of selected feature points are rotated by the generated rotation matrix and the rotated image t-1 It is confirmed whether the feature point matches the feature point of the image t. If the feature point of the image t-1 after rotation matches the feature point of the image t, the generated rotation matrix is likely to indicate the amount of shaking (the degree of positional deviation) between the correct images Since things are understood, an evaluation function with this degree of coincidence as an evaluation value is set.
- this inlier is a feature point that exists in common between images, such as a feature point indicated by ⁇ in FIG. 1C, and is a feature point obtained mainly from a distant view area in a captured image. Then, the rotation of the image is corrected using the rotation matrix estimated from the distant feature point inlier.
- feature point matching is performed by such processing. That is, feature point matching is performed on an image t-1 obtained from an area in which the image t-1 and the image t are commonly captured. It is a method that searches iteratively so that the distribution of feature points and the distribution of feature points of image t match as closely as possible, and the feature point distribution obtained from the area commonly shown in image t-1 and image t Is the method of estimating the amount of movement when the two are most coincident with the amount of movement between the images (image t-1 and image t). Then, by performing this feature point matching and continuously estimating the shaking amount generated between the images (between frames) in each image, it is possible to correct the shaking of the image (every image) using the estimated shaking amount. .
- feature point matching in order to use the similarity between images (between frames) in feature point distribution as a characteristic of a general algorithm, comparing with a method using a motion vector that handles partial region information of an image It has the feature that the calculation cost is low. Furthermore, in feature point matching, since it is possible to perform matching using feature points of the entire image, it is also possible to estimate a large amount of fluctuation. Therefore, by applying feature point matching, it is possible to estimate even a large shake included in an image captured during walking or an image captured without a finder. That is, in the feature point matching, it is possible to correct the fluctuation which can not be handled by the method using the motion vector.
- the path in which light from the outside enters the lens changes according to the projection method employed in the fisheye optical system.
- Patent Document 1 discloses a method of calculating using a motion vector of image processing as a method of estimating the amount of shake of an image captured by a fisheye optical system.
- the estimation of the shaking amount using image processing may correct the image for which the shaking amount can not be estimated in the image processing, and the image quality is not good. It may deteriorate.
- FIG. 2A is a diagram showing an example in the case where blurring occurs in the image
- FIG. 2B is a diagram showing an example in the case where the image has no feature
- FIG. 2C is a diagram showing an example of the case where the image has a periodic pattern.
- the present invention has been made in view of the above-described circumstances, and even if the image processing includes an image for which estimation of the amount of shake can not be performed, the amount of shake between a plurality of images captured continuously in time is
- An object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of correcting with high accuracy.
- one aspect of the image processing device of the present invention is an image processing device that corrects a positional deviation between a plurality of images captured continuously in time, which is a first image
- the movement amount indicating the positional deviation amount of the second image captured later in time with respect to the first image is estimated using feature points extracted from each of the first image and the second image
- a moving amount estimation unit that determines whether to perform correction using the movement amount estimated by the movement amount estimation unit based on the feature points;
- an image processing apparatus capable of correcting the amount of shaking between a plurality of images captured continuously in time with high accuracy even if the image processing includes an image for which estimation of the amount of shaking can not be performed can do.
- the present invention can be realized not only as an apparatus but also as an integrated circuit including processing means included in such an apparatus, or as a method in which the processing means constituting the apparatus are steps. May be implemented as a program that causes a computer to execute, or as information, data, or a signal that indicates the program.
- the programs, information, data and signals may be distributed via a recording medium such as a CD-ROM or a communication medium such as the Internet.
- the image processing includes an image for which estimation of the amount of shake can not be performed, it is possible to accurately correct the amount of shake between a plurality of images captured continuously in time.
- a processing device, an image processing method, a program, and an imaging device can be realized.
- FIG. 1A is a diagram for explaining a feature point based matching method.
- FIG. 1B is a diagram for explaining a feature point based matching method.
- FIG. 1C is a diagram for explaining a feature point based matching method.
- FIG. 1D is a diagram for explaining a feature point based matching method.
- FIG. 2A is a diagram showing an example when blur occurs in an image.
- FIG. 2B is a diagram showing an example in which there is no feature in the image.
- FIG. 2C is a diagram showing an example of the case where the image has a periodic pattern.
- FIG. 3 is a block diagram showing the entire configuration in the first embodiment of the present invention.
- FIG. 4 is a block diagram showing a configuration of a movement amount estimation unit in Embodiment 1 of the present invention.
- FIG. 5 is a block diagram showing a configuration of determination unit 23 in the first embodiment of the present invention.
- FIG. 6 is a diagram in the case of projecting an image captured by a wide-angle optical system from two dimensions to three dimensions.
- FIG. 7 is a diagram showing an example of the feature points of the image t-1 and the image t projected in three dimensions.
- FIG. 8 is a view in the case where the feature points of the image t-1 and the image t projected in three dimensions in the case of no blur are superimposed.
- FIG. 9 is a diagram showing an example of positional deviation of feature points projected in a three-dimensional manner with and without blurring.
- FIG. 6 is a diagram in the case of projecting an image captured by a wide-angle optical system from two dimensions to three dimensions.
- FIG. 7 is a diagram showing an example of the feature points of the image t-1 and the image t projected in three dimensions.
- FIG. 8 is a view in the case where the feature points of the image t-1 and the image
- FIG. 10 is a diagram showing an example in which the feature points of the image t-1 and the image t projected in three dimensions in the case of blurring are superimposed.
- FIG. 11 is a diagram showing feature points in the texture region.
- FIG. 12 is a diagram for explaining that it is difficult to select a common feature point for the image t-1 and the image t in the texture region.
- FIG. 13 is a diagram showing that the degree of matching of the feature point distribution is different for each feature point when there is a texture region.
- FIG. 14 is a diagram showing a search area of texture.
- FIG. 15 is a flowchart for explaining the flow of processing of the image processing apparatus 20 according to the first embodiment of the present invention.
- FIG. 15 is a flowchart for explaining the flow of processing of the image processing apparatus 20 according to the first embodiment of the present invention.
- FIG. 16 is a flowchart for explaining the flow of processing of the image processing apparatus 20 according to the first embodiment of the present invention.
- FIG. 17 is a block diagram showing an entire configuration in the second embodiment of the present invention.
- FIG. 18 is a flowchart for explaining the flow of processing of the image processing apparatus 30 according to the second embodiment of the present invention.
- FIG. 3 is a block diagram showing the entire configuration in the first embodiment of the present invention.
- the imaging unit 10 is a camera including an imaging element such as a CCD or a CMOS sensor, such as a digital still camera or a digital video camera, for example, and captures an image and outputs it as an electric signal.
- an imaging element such as a CCD or a CMOS sensor, such as a digital still camera or a digital video camera, for example, and captures an image and outputs it as an electric signal.
- the image processing apparatus 20 is an image processing apparatus for correcting positional deviation between a plurality of images captured continuously in time, and includes an image acquisition unit 21 and an image processing unit 20a.
- the image acquisition unit 21 acquires image data to be processed. Specifically, the image acquisition unit 21 selects a first image (image t-1) and a first image (image t-1) from among a plurality of images captured continuously in time by the imaging unit 10. Two pieces of image data of the second image (image t) captured after the time of) are acquired.
- the image t-1 is an example of a first image
- the image t is an example of a second image.
- the image t-1 (first image) is an image captured immediately before the image t (second image) in time series
- the image t is the image t-1 It is assumed that the image was taken immediately after it was taken.
- the image data of the image t-1 and the image t may be compressed and encoded in a general JPEG format, or may be recorded in a moving image format such as MPEG4.
- the image processing unit 20 a includes a movement amount estimation unit 22, a determination unit 23, and an image correction unit 24, and processes the image data acquired by the image acquisition unit 21.
- the movement amount estimation unit 22 includes a feature point extraction unit 221, a feature point coordinate conversion unit 222, a feature point matching unit 223, and a memory 224, as shown in FIG.
- the movement amount indicating the positional deviation amount of the second image captured later with respect to the first image is estimated using feature points extracted from each of the first image and the second image.
- FIG. 4 is a block diagram showing a configuration of movement amount estimating unit 22 in the first embodiment of the present invention.
- the feature point extraction unit 221 extracts a first feature point from the first image, and extracts a second feature point from the second image. Specifically, the feature point extraction unit 221 receives the image t-1 and the image t acquired by the image acquisition unit 21 and extracts the feature points of the input image t-1 and the image t, respectively. Feature point data t-1 and feature point data t are generated.
- the feature point is, as described above, a feature point on the image that can be detected by image processing, and for example, a point at which an edge in which both longitudinal and lateral edges in the image are strongly intersected , A point near the local where there are strong edges with two different directions.
- the feature points are preferably inliers that can be stably detected (estimated) from the point that they are commonly shown between two images of the image t-1 and the image t continuous in time series.
- the feature point extraction unit 221 extracts feature points, the exact positional relationship between the image t-1 and the image t is unknown. Therefore, it is necessary to extract commonly existing feature points using some criteria.
- corner points are extracted such that vertical and horizontal edges intersect with the image edge.
- the feature point extraction unit 221 extracts the above-mentioned corner points by calculating a feature point score representing the degree of edge intersection for each pixel.
- the feature point score is calculated and present for each pixel by the feature point extraction unit 221.
- edges used in Harris reflect changes in the contrast (brightness value) of the image, so if the lighting conditions change to some extent, the contrast of the image is preserved and the edges do not disappear. That is, the edge is unlikely to disappear between images (between frames) unless the edge is concealed by an obstruction and the edge itself disappears. Therefore, a point having a high feature point score based on edge information is highly likely to be present as a feature point in the image t-1 and the image t in common.
- the feature point extraction part 221 when using a feature point score as a standard used to extract a feature point which exists in common, the feature point extraction part 221 will extract the feature point whose score is higher than a certain threshold.
- the above-mentioned specific threshold may use the value of the average score in the image, or the average of the score on the time series of a plurality of images, etc. You may use the value decided on the basis.
- the specific threshold described above does not need to use one threshold for the entire image, and may use a threshold generated for each area.
- the threshold value generated for each region may be determined based on a single image, or may be determined using a plurality of images in time series.
- the feature point score may be determined based on the score of an object when recognition of an object is performed.
- the feature point coordinate conversion unit 222 sets the coordinates of the first feature point of the first image extracted by the feature point extraction unit 221 and the second feature point of the second image according to the projection method of the fisheye optical system. Convert to coordinates. Specifically, the feature point coordinate conversion unit 222 determines the coordinates of feature points of a plurality of captured images when the plurality of images captured continuously in time are captured using a fisheye optical system. Coordinate conversion is performed to convert the coordinates according to the projection method adopted in the fisheye optical system.
- Such coordinate conversion is performed because, for example, the position of feature point coordinates obtained from the input image and the position in the external world differ depending on the projection method employed in the super wide angle optical system of the fisheye optical system, and the camera shake amount is correct. It is necessary to match the position of feature point coordinates obtained from the input image with the position in the external world in order to estimate from the image. Therefore, the feature point coordinate conversion unit 222 performs inverse transformation of projective transformation on the coordinates of the feature point obtained from the input image, and calculates the position of each feature point in the outside world. Although this coordinate conversion is performed on at least the coordinates of the distant feature point (inlier) obtained from the feature point matching unit 223, the coordinate conversion is not limited to only the coordinate conversion for the coordinates of the distant feature point (inlier), and Coordinate conversion may be performed.
- the feature point coordinate conversion unit 222 does not perform processing when a plurality of images captured continuously in time are not captured using a wide-angle optical system including a fisheye optical system.
- the first feature point of the first image and the second feature point of the second image extracted by the feature point extraction unit 221 may be directly input to the feature point matching unit 223. .
- the feature point matching unit 223 performs a matching on the first feature point of the first image extracted by the feature point extraction unit 221 and the second feature point of the second image to obtain the second feature of the first image.
- the displacement amount indicating the displacement amount of the image of is estimated.
- the feature point matching unit 223 performs the first conversion of the coordinates converted by the feature point coordinate conversion unit 222. By performing matching between the feature point and the second feature point, the movement amount indicating the displacement amount of the second image with respect to the first image is estimated.
- the feature point matching unit 223 estimates the matching, that is, the correspondence between the feature point data t-1 of the image t-1 and the feature point data t of the image t.
- the feature point data t-1 it is assumed that the feature point extracting unit 221 extracts from the image t-1 in the previous frame period and uses, for example, the data stored in the memory 224.
- feature point data t feature point data t extracted by the feature point extracting unit 221 from the image t in the current frame period is used.
- the feature point matching unit 223 uses the feature point data t-1 and the feature point data t to estimate a rotation matrix indicating the motion of the camera generated between the image t-1 and the image t.
- the rotation matrix is calculated using a method such as RANSAC (RANdom SAmple Consensus). From this rotation matrix, rotation components of roll, pitch and yaw representing the amount of camera shake generated between frames, that is, between images are obtained. When the estimation of the rotation matrix fails, 0 is set to roll, pitch, and yaw, and it is treated that there is no rotation between images.
- the movement amount estimation unit 22 is configured as described above.
- the determination unit 23 includes a feature point determination unit 231, a blur determination unit 232, and a texture determination unit 233, and is estimated by the movement amount estimation unit 22 based on the extracted feature points. It is determined whether or not to make correction using the movement amount.
- FIG. 5 is a block diagram showing a configuration of determination unit 23 in the first embodiment of the present invention.
- the determination unit 23 determines that the movement amount estimated by the movement amount estimation unit 22 indicates the displacement amount (swinging amount) of the second image with respect to the first image based on the extracted feature points. In this case, it is determined that the correction is performed using the movement amount estimated by the movement amount estimation unit 22. Specifically, based on the information obtained from the image, the determination unit 23 determines a scene that is not good at image processing, that is, a scene including an image for which the amount of camera shake can not be estimated by the image processing. Then, the determination unit 23 controls the image correction unit 24 so as not to perform image processing when it is determined that the scene includes an image for which the amount of camera shake can not be estimated in image processing.
- FIG. 2A As an example of a scene including an image whose amount of camera shake can not be estimated by image processing, as shown in FIG. 2A, when a blur is included in the image, as shown in FIG. 2B, a characteristic subject appears in the image. In such a case, as shown in FIG. 2C, for example, there may be a periodic pattern (texture) such as a tile on the ground.
- a periodic pattern such as a tile on the ground.
- FIG. 6 is a diagram in the case of projecting an image captured by a wide-angle optical system from two dimensions to three dimensions.
- FIG. 7 is a diagram showing an example of the feature points of the image t-1 and the image t projected in three dimensions.
- FIG. 8 is a view in the case where the feature points of the image t-1 and the image t projected in three dimensions in the case of no blur are superimposed.
- FIG. 9 is a diagram showing an example of positional deviation of feature points projected in a three-dimensional manner with and without blurring.
- FIG. 10 is a diagram showing an example in which the feature points of the image t-1 and the image t projected in three dimensions in the case of blurring are superimposed.
- the blur determination unit 232 determines an image including blur from the framework of feature point matching. The method will be described below.
- FIG. 6 a two-dimensional image from which feature points are extracted is three-dimensionally projected.
- FIG. 6A shows that the image is an image photographed using a fisheye optical system as a typical example, and is an image from which feature points are extracted, and FIG. It shows the distribution of feature points projected to dimensional coordinates.
- Such coordinate conversion is performed to obtain rotational motion (roll, pitch, yaw) which is three-dimensional information that can not be obtained on a two-dimensional (x, y) image plane, as described above. Furthermore, when the position of the feature point coordinates obtained from the input image differs from the position in the external world by the projection method adopted in the fisheye optical system, correction is performed in consideration of the positional deviation in the projection method. That is, such coordinate conversion is performed to estimate the rotational motion of the correct camera shake amount from the image.
- projective transformation is performed to match the position of feature point coordinates obtained from the input image with the position in the external world.
- such coordinate conversion is useful not only for the fisheye optical system but also for correcting distortion of the optical system (lens). Then, by projecting the feature points in three dimensions using the calculated rotational motion, the relationship between the distribution of the feature points can be determined from the three-dimensionally projected feature points.
- the blur determination unit 232 projects the feature points to three-dimensional coordinates using such a method, and then the camera shake amount (move amount) generated between the images (frames) estimated by the move amount estimation unit 22. ) And the relationship of the distribution of feature points between images (between frames).
- the contrast of the image between the images (between frames) is maintained, and thus the image t-1 and the image as shown in FIG. At t, the distribution of feature points between images (between frames) is similar. Therefore, when the movement amount estimation unit 22 estimates the correct amount of movement (movement amount), the image t-1 and the image t are such that the state of FIG. 8 (a) becomes the state of FIG. 8 (b). The distribution of the feature points of is roughly consistent. On the other hand, when the image to be subjected to the shake correction process includes blur, the contrast of the image between the images (between frames) can not be maintained.
- the contrast of the image is lowered due to the influence of blurring, so that the distribution of feature points different from the distribution of feature points without blurring, specifically, as shown in FIG. It is possible to obtain a distribution having a low degree of similarity of the distribution of feature points of
- the state of FIG. 10A is in the state of FIG. 10B.
- the correct amount of shaking can not be estimated. That is, the degree of coincidence of the distribution of the feature points of the image t-1 and the image t is low.
- the blur determination unit 232 can roughly determine whether blur has occurred in an image between images (between frames) using the degree of matching of the distribution of feature points.
- the blur determination unit 232 roughly determines whether blur has occurred in the image. Specifically, the blur determination unit 232 counts, for example, the number of feature points at which the degree of coincidence of each feature point is within a certain distance, and if the number of feature points is smaller than a threshold, blur is present. You may judge. Alternatively, the blur determination unit 232 may measure the degree of coincidence of each feature point and determine that a blur is present if the sum is greater than a threshold. Alternatively, the moving direction of each feature point may be measured, and the determination may be made using the variance of the moving direction.
- the blur determination unit 232 projects the coordinates of the first feature point extracted from the first image and the second feature point extracted from the second image into three-dimensional coordinates, and 3 It is determined whether the degree of agreement between the feature point distribution of the first feature point and the feature point distribution of the second feature point on the dimensional coordinates is higher than a predetermined degree of coincidence, and the degree of coincidence is higher than the predetermined degree of coincidence If it is determined that the correction is made, it is determined that the correction is performed using the moving amount estimated by the moving amount estimating unit 22.
- the blur determination unit 232 may determine whether or not there is blur according to the speed of the shutter speed.
- the feature point determination unit 231 determines whether the number of feature points extracted in the first image or the second image is larger than a predetermined number. When the feature point determination unit 231 determines that the number of feature points extracted in the first image or the second image is larger than a predetermined number, the feature point determination unit 231 corrects using the movement amount estimated by the movement amount estimation unit. Is determined to Specifically, the feature point determination unit 231 calculates edge information from the input image, and determines that a characteristic subject is not captured when the number of pixels having edge strengths greater than or equal to a specific value does not satisfy the threshold.
- the feature point determination unit 231 may detect whether or not a specific object is captured, and determine whether a characteristic subject is captured based on the presence or absence of the specific object, or the image may be divided into a plurality of areas. Then, the variance of the luminance value for each area may be examined, and it may be determined whether or not the characteristic subject is shifted based on the magnitude of the variance.
- FIG. 11 is a diagram showing feature points in the texture region.
- FIG. 12 is a diagram for explaining that it is difficult to select a common feature point for the image t-1 and the image t in the texture region.
- FIG. 13 is a diagram showing that the degree of matching of the feature point distribution is different for each feature point when there is a texture region.
- FIG. 14 is a diagram showing a search area of texture.
- the texture determination unit 233 determines from the framework of feature point matching as in the case where a blur is included in the image.
- a blur is included in the image.
- processing is performed by the feature point matching unit 223 using all the extracted feature points regardless of whether or not the image contains a periodic pattern (texture), the calculation cost becomes extremely high as described above. I will. Therefore, in practice, processing is performed by narrowing down to some representative feature points from these extracted feature points.
- the feature points narrowed in this way have ⁇ , ⁇ , and x feature points as shown in FIG. 1C, and if there are many common feature points (inliers) between images (between frames) indicated by ⁇ ⁇ ⁇ ⁇ .
- the number is larger, it is possible to stably estimate the amount of movement (amount of movement) of the camera between images (between frames).
- the more the feature points of ⁇ and ⁇ the higher the possibility that the estimation of the camera shake amount (movement amount) will fail. Therefore, in order to stably estimate the amount of movement (movement amount) of the camera, it is important to leave many inliers of ⁇ in the process of narrowing the number of feature points.
- the problem in the process of narrowing down the number of feature points in this way is the characteristic of the texture area in which pixels having similar high contrast are periodically present.
- FIG. 11 (a) for example, pixels having similar high contrast are periodically arranged in a texture area such as a tile on the ground. Therefore, a feature point using this contrast information is shown in FIG. 11 (b). As shown, many are extracted from the texture area.
- the feature point matching unit 223 can estimate the amount of movement (amount of movement) of the camera generated between the images (between frames).
- the feature point matching unit 223 performs matching using the extracted feature points, the camera shake amount (movement amount) between the images (frames) is erroneously estimated. May.
- this mis-estimation depends on how to define the degree of coincidence of the feature points, in general, for example, in FIG. 8B, the features of the image t-1 after estimation of the amount of movement (movement amount) The number of points at which the distance between the point and the feature point of the image t is less than or equal to a certain value is set as the degree of coincidence.
- the texture determination unit 233 succeeds in matching, that is, the movement estimated by the movement amount estimation unit 22. It may be determined that the correction is performed using the amount.
- FIG. 8 shows a case where no error factor such as blur occurs between images (frames), and in fact, some blur and some error exist. Therefore, the minimum distance is set to satisfy the distance between the feature point of the image t-1 and the feature point of the image t, and the feature point satisfying the set minimum distance represents the amount of movement (movement amount) of the camera. It is considered as a feature point.
- FIG. 13A when there are a large number of feature points ⁇ whose positions are changed between images, the feature points that do not actually represent the amount of movement (the amount of movement) of the camera are shown in FIG.
- the minimum distance it may be regarded as a feature point representing the amount of movement (amount of movement) of the camera. That is, as described above, when there is a texture area in the image, it causes the camera shake amount (movement amount) to be estimated incorrectly.
- the texture determination unit 233 determines the presence or absence of the texture that causes the erroneous estimation, so that the movement amount estimated by the movement amount estimation unit 22 corresponds to the second image of the first image. It is determined whether the positional deviation amount (swinging amount) is indicated.
- the texture determination unit 233 has a tendency that a texture is biased to a specific region in an image (image) captured using a wide-angle optical system such as a fisheye optical system (part of the tile illustrated in FIG. 2C) Use
- the texture determination unit 233 can acquire the shaking amount (movement amount) of the camera after matching is performed by the feature point matching unit 223, the distribution of feature points between images (between frames) can be obtained. You can get the degree of match.
- the degree of coincidence of the acquired feature point distribution is high, there are cases where the correct amount of shaking can be estimated (when many ⁇ in FIG. 1C are included) and in the case of misestimation due to texture (when many ⁇ in FIG. 1C). Conceivable. Therefore, when the degree of coincidence of the acquired distribution of feature points is high, the distribution of characteristic points with high degree of coincidence (inlier distribution) is further divided into areas as shown in FIG. Check.
- the contrast value of the peripheral area is determined, and the contrast similarity of the peripheral area is measured. If the measured similarity is high, it can be seen that texture is present in the corresponding area (localized area).
- the texture determination unit 233 projects the coordinates of the first feature point extracted from the first image and the second feature point extracted from the second image to three-dimensional coordinates, and projects the coordinates.
- the degree of coincidence between the characteristic point distribution of the first characteristic point and the characteristic point distribution of the second characteristic point on the three-dimensional coordinate is higher than a predetermined degree of coincidence
- the first image and the second image are further divided into areas Check the feature point distribution (inlier distribution) of the first feature point and the second feature point higher than the predetermined degree of coincidence for each of the divided regions, and the region in which the feature point distribution (inlier distribution) is divided In the case of uneven distribution in a part of the regions, it is determined that the correction is not performed using the movement amount estimated by the movement amount estimation unit 22.
- the amount of movement is separately estimated using a sensor, and the amount of movement (amount of movement) is corrected.
- the control may be performed, or the area in which the texture exists is masked, the texture area is excluded, and the feature point matching unit 223 performs matching again to estimate the amount of movement (amount of movement) of the camera. It may be Alternatively, a control method may be used in which this result is used on a time series, a texture region is continuously masked, and the influence of texture is excluded.
- the determination unit 23 configured as described above determines whether or not to perform correction using the movement amount estimated by the movement amount estimation unit 22.
- the blur determination unit 232 and the texture determination unit 233 are coordinates that project the coordinates of the first feature point extracted from the first image and the second feature point extracted from the second image into three-dimensional coordinates. Although the degree of coincidence between the feature point distribution of the first feature point and the feature point distribution of the second feature point is confirmed after conversion, the present invention is not limited thereto. When not correcting the distortion of the optical system (lens) instead of the image captured using the fisheye optical system, the subsequent processing may be performed without performing this coordinate conversion.
- the image correction unit 24 corrects a plurality of positional deviations of the second image with respect to the first image using the movement amount. Correct the misalignment between the images of. Specifically, the image correction unit 24 uses the correction value (movement amount) calculated by the feature point matching unit 223 to determine the camera shake generated between the frames of the image t-1 and the image t (between the images). to correct. The image correction unit 24 corrects the shake of the image t with respect to the image t-1 by performing, for example, affine transformation using the parameters roll, pitch, and yaw indicating the correction amount, that is, the movement amount.
- FIG. 15 and FIG. 16 are flowcharts for explaining the flow of processing of the image processing apparatus 20 according to the first embodiment of the present invention.
- FIG. 16 shows the process of FIG. 15 in detail.
- the image processing apparatus 20 causes the image acquisition unit 21 to acquire image data to be processed. Specifically, the image acquisition unit 21 reads the image data of the image t and the image t-1 obtained from the imaging unit 10, respectively.
- the movement amount estimation unit 22 determines the movement amount indicating the positional deviation amount of the second image captured temporally after the first image with respect to the first image, the first image and the second movement amount. It estimates using the feature point extracted from each of 2 images (S10). Specifically, the feature point extraction unit 221 extracts a first feature point from the first image, and extracts a second feature point from the second image (S101). Next, the feature point coordinate conversion unit 222 converts the coordinates of the first feature point of the first image extracted by the feature point extraction unit 221 and the second feature point of the second image into the projection method of the fisheye optical system. It converts to the coordinate according to (S102). As a result, the position of the external world of each feature point is calculated.
- the feature point matching unit 223 performs matching between the first feature point of the first image extracted by the feature point extraction unit 221 and the second feature point of the second image, whereby the first image is extracted.
- the movement amount indicating the displacement amount of the second image is estimated (S103).
- the determination unit 23 determines whether to perform correction using the movement amount estimated by the movement amount estimation unit 22 (S20). Specifically, first, the feature point determination unit is performed. That is, the feature point determination unit 231 determines whether the number of feature points extracted in the first image or the second image is larger than a predetermined number (S201). When the feature point determination unit 231 determines that the number of feature points extracted in the first image or the second image is larger than a predetermined number (pass in S201), the process proceeds to S202.
- the feature point determination unit 231 determines that the number of feature points extracted in the first image or the second image is not greater than (that is, less than) the predetermined number (fail in S201), the movement It is determined that the correction is not performed using the movement amount estimated by the amount estimation unit 22, and the process of the image processing device 20 is ended.
- blur determination is performed. That is, the blur determination unit 232 projects the coordinates of the first feature point extracted from the first image and the second feature point extracted from the second image into three-dimensional coordinates, and It is determined whether the matching degree between the feature point distribution of one feature point and the feature point distribution of the second feature point is higher than a predetermined matching degree (S202).
- the blur determination unit 232 determines that the degree of coincidence is higher than a predetermined degree of coincidence, the blur determination unit 232 determines that the correction is performed using the movement amount estimated by the movement amount estimation unit 22 (pass of S201), and proceeds to S203. And proceed.
- the blur determination unit 232 determines that the degree of coincidence is not higher (that is, lower) than the predetermined degree of coincidence (fail in S202)
- correction is performed using the movement amount estimated by the movement amount estimation unit 22. It is determined that the process is not performed, and the process of the image processing apparatus 20 is ended. Next, texture determination is performed.
- the texture determination unit 233 projects the coordinates of the first feature point extracted from the first image and the second feature point extracted from the second image into three-dimensional coordinates, and When the degree of coincidence between the characteristic point distribution of the first characteristic point and the characteristic point distribution of the second characteristic point is higher than a predetermined degree of coincidence, the first image and the second image are further divided into areas, and divided areas Each time, the feature point distribution (inlier distribution) of the first feature point and the second feature point higher than the predetermined matching degree is confirmed (S203). When the feature point distribution (inlier distribution) is not unevenly distributed in a part of the divided regions (the pass of S203), the texture determination unit 233 performs the movement estimated by the movement amount estimation unit 22.
- the texture determination unit 233 is estimated by the movement amount estimation unit 22 (fail in S203) when the feature point distribution (inlier distribution) is unevenly distributed in a part of the divided regions (fail in S203). It is determined that the correction is not performed using the movement amount, and the processing of the image processing device 20 is ended.
- the image correction unit 24 corrects the positional deviation of the second image with respect to the first image using the movement amount.
- the positional deviation between the plurality of images is corrected (S30). That is, when all the determinations are cleared in the determination unit 23, the image correction unit 24 corrects the shake of the image using the movement amount (swinging amount) estimated by the movement amount estimation unit 22.
- the image processing apparatus 20 performs processing.
- the image processing device 20 when the image processing apparatus 20 estimates the amount of movement of the image captured by the fisheye optical system by image processing, the image processing device 20 erroneously estimates the image for which estimation of the amount of movement can not be performed by image processing. By performing the correction using the amount of shake, it is possible to prevent the deterioration of the video quality. That is, even if the image processing apparatus 20 includes an image that can not estimate the shake amount in image processing, the image processing device 20 corrects the shake amount between a plurality of images captured continuously in time with high accuracy. It is possible to improve the final image quality by avoiding correcting with the wrong value or the wrong value.
- the shake amount between a plurality of images captured continuously in time is corrected with high accuracy.
- the image processing apparatus has a remarkable effect on ultra-wide-angle images such as fish-eye images.
- the present invention is also applicable to a normal angle-of-view image having a field angle of about 70 degrees or less.
- the image processing apparatus 20 includes the image acquisition unit 21 and the image processing unit 20a in the above description, the present invention is not limited thereto.
- the image processing unit 20a may be provided.
- the image processing apparatus 20 shakes between a plurality of images captured continuously in time, even if the image processing includes an image for which estimation of the shake amount can not be performed in image processing. The amount can be corrected with high accuracy.
- the amount of movement (the amount of movement) is calculated by performing image processing by the movement amount estimation unit 22, but the invention is not limited thereto.
- the shaking amount (moving amount) may be simultaneously estimated by the sensor.
- FIG. 17 is a block diagram showing an entire configuration in the second embodiment of the present invention.
- the same components as those in FIGS. 3 and 4 are denoted by the same reference numerals and descriptions thereof are omitted, and only different components will be described.
- the overall configuration of the present embodiment shown in FIG. 17 differs from the overall configuration of the first embodiment shown in FIG. 3 in that it further includes a sensor estimation unit 31 and a correction means determination unit 32.
- the sensor estimation unit 31 measures and measures the rotation angle around the optical axis of the optical system used for imaging or the rotation angle around at least one of two axes perpendicular to the optical axis of the optical system.
- the rotation angle is estimated as a movement amount that indicates the displacement amount of the second image with respect to the first image.
- the sensor estimation unit 31 estimates the amount of movement (amount of movement) of the camera between images (between frames) using a sensor.
- the sensor estimation unit 31 is configured by at least one sensor of an angular acceleration sensor, an angular velocity sensor, an acceleration sensor, a gyro sensor, and an orientation sensor.
- the sensor measures the amount of movement (amount of movement) of the camera generated between the images (between frames) using any or some of the above-described sensors.
- the sensor estimation unit 31 may estimate the amount of movement (amount of movement) of the image (frame) by processing the amount of movement (amount of movement) measured by the sensor in time series.
- the correction unit determination unit 32 performs the correction using the movement amount estimated by the sensor estimation unit 31. Decide whether or not. Specifically, when the determination unit 23 determines that the result of the image processing is not applicable, the correction unit determination unit 32 determines the amount of movement (movement between frames) of the images estimated by the sensor estimation unit 31 (movement) Determine if the amount is applicable. Specifically, the correction means determination unit 32 uses the sensor such as an acceleration sensor or a gyro sensor to observe the behavior between images (between frames) or in a plurality of images (plural frames) to obtain a sensor estimation unit 31. It is determined whether to use or not to use the estimated amount of movement (moving amount).
- observation of behavior between images (between frames) or multiple images (multiple frames) means that, for example, when the sensor value fluctuates sharply between multiple images (between frames), the camera shake amount (movement amount) is It can be determined to be large. Therefore, it is assumed that the correction means determination unit 32 does not perform correction using the shaking amount (moving amount) estimated by the sensor estimation unit 31 because the estimation accuracy of the sensor is lowered when the value of the sensor is violently shaken. It may be decided. Conversely, if the sensor behavior is stable, it is determined that the camera is stationary and correction is performed using the amount of movement (amount of movement) estimated by the sensor estimation unit 31. Good.
- FIG. 18 is a flowchart for explaining the flow of processing of the image processing apparatus 30 according to the second embodiment of the present invention.
- symbol is attached
- the movement amount estimation unit 22 sets the movement amount indicating the displacement amount of the second image captured after the time of the first image with respect to the first image to the first image and the movement amount.
- the estimation is performed using feature points extracted from each of the second images.
- the sensor estimation unit 31 measures the rotation angle around the optical axis of the optical system used for imaging or the rotation angle around at least one of two axes perpendicular to the optical axis of the optical system.
- the measured rotation angle is estimated as a movement amount indicating the positional deviation amount of the second image with respect to the first image (S15).
- the determination unit 23 determines whether to perform correction using the movement amount estimated by the movement amount estimation unit 22 (S20).
- S20 when it is determined that the correction is not performed using the movement amount estimated by the movement amount estimation unit 22 by the determination unit 23 (fail in S20), the process of the image processing apparatus 30 is not ended. It progresses to the process of the sensor reliability determination of S25.
- the correction means determination unit 32 uses the movement amount estimated by the sensor estimation unit 31 when the determination unit 23 determines that the correction is not performed using the movement amount estimated by the movement amount estimation unit 22. And decide whether to make corrections.
- the correction means determination unit 32 proceeds to S30 when it is determined that the correction is to be performed using the swing amount (movement amount) estimated by the sensor estimation unit 31 (pass in S25).
- the correction means determination unit 32 performs the process of the image processing device 20. End.
- the image correction unit 24 performs the first process of the second image.
- the positional deviation between the plurality of images is corrected by correcting the positional deviation with respect to the image using the movement amount estimated by the movement amount estimation unit 22 (S30).
- the image correction unit 24 performs the first process of the second image.
- the positional deviation between the plurality of images is corrected by correcting the positional deviation with respect to the image using the movement amount estimated by the sensor estimation unit 31 (S30).
- the image processing apparatus 30 performs processing.
- the shake amount between a plurality of images captured continuously in time is corrected with high accuracy.
- An image processing apparatus and an image processing method that can be implemented. Specifically, according to the image processing apparatus and the image processing method of the present embodiment, a scene including an image which is not good for image processing is determined from image information that can be acquired from the feature point based matching process, By switching the adoption or rejection of the shake amount estimated by the sensor or the image processing according to the scene, it is possible to correct the shake of the photographed image with high accuracy.
- the image processing apparatus does not know when the image to be corrected is taken.
- the above-described image processing apparatus is incorporated in a digital still camera or a digital video camera, and is configured to correct a captured image on the spot, but is not limited thereto.
- an image processing apparatus may be prepared separately from the imaging apparatus such as being implemented as an application on a personal computer, and connected directly to a recording device such as a camera with a cable or the like to input a captured image.
- the image data may be read via a recording medium such as an SD memory card or a network.
- Each of the above-described devices is specifically a computer system including a microprocessor, a ROM, a RAM, a hard disk unit, a display unit, a keyboard, a mouse and the like.
- a computer program is stored in the RAM or the hard disk unit.
- Each device achieves its function by the microprocessor operating according to the computer program.
- the computer program is configured by combining a plurality of instruction codes indicating instructions to the computer in order to achieve a predetermined function.
- the system LSI is a super-multifunctional LSI manufactured by integrating a plurality of components on one chip, and more specifically, a computer system including a microprocessor, a ROM, a RAM, and the like. . A computer program is stored in the RAM. The system LSI achieves its functions as the microprocessor operates in accordance with the computer program.
- the IC card or the module is a computer system including a microprocessor, a ROM, a RAM, and the like.
- the IC card or the module may include the super multifunctional LSI described above.
- the IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may be tamper resistant.
- the present invention may be the method shown above. Further, the present invention may be a computer program that realizes these methods by a computer, or may be a digital signal composed of the computer program.
- the present invention is a computer readable recording medium that can read the computer program or the digital signal, such as a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray Disc ), And may be recorded in a semiconductor memory or the like. Further, the present invention may be the digital signal recorded on these recording media.
- the computer program or the digital signal may be transmitted via a telecommunication line, a wireless or wired communication line, a network represented by the Internet, data broadcasting, and the like.
- the present invention may be a computer system comprising a microprocessor and a memory, wherein the memory stores the computer program, and the microprocessor operates according to the computer program.
- the image processing apparatus determines a scene which is not good at image processing from image information, and switches the processing of a sensor and an image according to the scene to obtain an image photographed by a wide-angle optical system such as a fisheye optical system. It is useful as an apparatus etc. which correct
- Imaging unit 20 30 image processing device 20a image processing unit 21 image acquisition unit 22 movement amount estimation unit 23 determination unit 24 image correction unit 31 sensor estimation unit 32 correction means determination unit 221 feature point extraction unit 222 feature point coordinate conversion unit 223 Feature point matching unit 224 Memory 231 Feature point determination unit 232 Blur determination unit 233 Texture determination unit
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Studio Devices (AREA)
- Image Analysis (AREA)
Abstract
Description
図3は、本発明の実施の形態1における全体構成を示すブロック図である。
実施の形態1では、揺れ量(移動量)を移動量推定部22で画像処理を行うことにより算出していたが、それに限らない。揺れ量(移動量)をセンサでも同時に推定するとしてもよい。以下、本発明の実施の形態2における画像処理装置および画像処理方法について、図を参照しながら説明する。
20、30 画像処理装置
20a 画像処理部
21 画像取得部
22 移動量推定部
23 判定部
24 画像補正部
31 センサ推定部
32 補正手段決定部
221 特徴点抽出部
222 特徴点座標変換部
223 特徴点マッチング部
224 メモリ
231 特徴点判定部
232 ブラー判定部
233 テクスチャ判定部
Claims (13)
- 時間的に連続して撮影された複数の画像間の位置ずれを補正する画像処理装置であって、
第1の画像の時間的に後に撮影された第2の画像の前記第1の画像に対する位置ずれ量を示す移動量を、前記第1の画像および前記第2の画像それぞれから抽出される特徴点を用いて、推定する移動量推定部と、
前記移動量推定部により推定された前記移動量を用いて補正を行うか否かの判定を、前記特徴点に基づいて行う判定部と、
前記判定部により前記移動量を用いて補正を行うと判定された場合に、前記第2の画像の前記第1の画像に対する位置ずれを、前記移動量を用いて補正することにより、前記複数の画像間の位置ずれを補正する画像補正部とを備える
画像処理装置。 - 前記移動量推定部は、
前記第1の画像から第1特徴点を抽出し、前記第2の画像から第2特徴点を抽出する特徴点抽出部と、
前記特徴点抽出部により抽出された前記第1特徴点と前記第2特徴点とでマッチングを行うことにより、前記第1の画像に対する前記第2の画像の位置ずれ量を示す移動量を推定する特徴点マッチング部とを備える
請求項1に記載の画像処理装置。 - 前記複数の画像は、魚眼光学系を用いて撮影されており、
前記移動量推定部は、さらに、前記特徴点抽出部により抽出された前記第1特徴点と前記第2特徴点との座標を魚眼光学系の射影方式に応じた座標に変換する特徴点座標変換部を備え、
前記特徴点マッチング部は、前記特徴点座標変換部により変換された前記第1特徴点の座標と前記第2特徴点の座標とでマッチングを行うことにより、前記第1の画像に対する前記第2の画像の位置ずれ量を示す移動量を推定する
請求項2に記載の画像処理装置。 - 前記判定部は、前記特徴点に基づいて、前記移動量推定部により推定された前記移動量が前記第1の画像に対する前記第2の画像の位置ずれ量を示していると判定した場合に、前記移動量推定部により推定された前記移動量を用いて補正を行うと判定する
請求項1~3のいずれか1項に記載の画像処理装置。 - 前記判定部は、前記第1の画像または前記第2の画像で抽出される特徴点の数が所定の数より多いと判定した場合に、前記移動量推定部により推定された前記移動量を用いて補正を行うと判定する
請求項1~4のいずれか1項に記載の画像処理装置。 - 前記判定部は、前記第1の画像から抽出される第1特徴点と前記第2の画像から抽出される第2特徴点との座標が3次元投影された3次元座標上における前記第1特徴点の特徴点分布と前記第2特徴点の特徴点分布との一致度が所定の一致度よりも高いか否かを判定し、当該一致度が所定の一致度より高いと判定した場合に、前記移動量推定部により推定された前記移動量を用いて補正を行うと判定する
請求項1~5のいずれか1項に記載の画像処理装置。 - 前記判定部は、前記第1の画像から抽出される第1特徴点と前記第2の画像から抽出される第2特徴点との座標が投影された3次元座標上における前記第1特徴点の特徴点分布と前記第2特徴点の特徴点分布の一致度が所定の一致度より高い場合に、
さらに、前記第1の画像および前記第2の画像を領域分割し、分割した領域ごとに前記所定の一致度より高い前記第1特徴点と前記第2特徴点との特徴点分布を確認し、当該特徴点分布が前記分割した領域のうちの一部の領域に偏在する場合には、前記移動量推定部により推定された前記移動量を用いて補正を行わないと判定する
請求項1~6のいずれか1項に記載の画像処理装置。 - 前記画像処理装置は、さらに、撮影に用いた光学系の光軸周りの回転角度または前記光学系の光軸に対して互いに垂直な2軸のうち少なくとも1つの軸周りの回転角度を計測し、計測した前記回転角度を、前記第2の画像の前記第1の画像に対する移動量と推定するセンサ推定部を備える
請求項1に記載の画像処理装置。 - 前記センサ推定部は、角加速度センサ、角速度センサ、加速度センサおよび方位センサのうち少なくとも1種類のセンサによって構成される
請求項8に記載の画像処理装置。 - 前記画像処理装置は、さらに、前記判定部が前記移動量推定部により推定された移動量用いて補正を行わないと判定した場合に、前記センサ推定部により推定された移動量を用いて補正を行うか否かを決定する補正手段決定部を備える
請求項8に記載の画像処理装置。 - 時間的に連続して撮影された複数の画像間の位置ずれを補正する画像処理方法であって、
第1の画像の時間的に後に撮影された第2の画像の前記第1の画像に対する位置ずれ量を示す移動量を、前記第1の画像および前記第2の画像それぞれから抽出される特徴点を用いて、推定する移動量推定ステップと、
前記移動量推定ステップにおいて推定された前記移動量を用いて補正を行うか否かの判定を、前記特徴点に基づいて行う判定ステップと、
前記判定ステップにおいて前記移動量を用いて補正を行うと判定された場合に、前記第2の画像の前記第1の画像に対する位置ずれを、前記移動量を用いて補正することにより、前記複数の画像間の位置ずれを補正する画像補正ステップとを含む
画像処理方法。 - 時間的に連続して撮影された複数の画像間の位置ずれを補正するためのプログラムであって、
第1の画像の時間的に後に撮影された第2の画像の前記第1の画像に対する位置ずれ量を示す移動量を、前記第1の画像および前記第2の画像それぞれから抽出される特徴点を用いて、推定する移動量推定ステップと、
前記移動量推定ステップにおいて推定された前記移動量を用いて補正を行うか否かの判定を、前記特徴点に基づいて行う判定ステップと、
前記判定ステップにおいて前記移動量を用いて補正を行うと判定された場合に、前記第2の画像の前記第1の画像に対する位置ずれを、前記移動量を用いて補正することにより、前記複数の画像間の位置ずれを補正する画像補正ステップと
をコンピュータに実行させるプログラム。 - 時間的に連続して撮影された複数の画像間の位置ずれを補正する集積回路であって、
第1の画像の時間的に後に撮影された第2の画像の前記第1の画像に対する位置ずれ量を示す移動量を、前記第1の画像および前記第2の画像それぞれから抽出される特徴点を用いて、推定する移動量推定部と、
前記移動量推定部により推定された前記移動量を用いて補正を行うか否かの判定を、前記特徴点に基づいて行う判定部と、
前記判定部により前記移動量を用いて補正を行うと判定された場合に、前記第2の画像の前記第1の画像に対する位置ずれを、前記移動量を用いて補正することにより、前記複数の画像間の位置ずれを補正する画像補正部とを備える
集積回路。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201180005738.1A CN102714697B (zh) | 2010-11-11 | 2011-11-08 | 图像处理装置、图像处理方法及程序 |
US13/521,133 US9001222B2 (en) | 2010-11-11 | 2011-11-08 | Image processing device, image processing method, and program for image processing for correcting displacement between pictures obtained by temporally-continuous capturing |
JP2012512146A JP5694300B2 (ja) | 2010-11-11 | 2011-11-08 | 画像処理装置、画像処理方法およびプログラム |
EP11840319.5A EP2640059B1 (en) | 2010-11-11 | 2011-11-08 | Image processing device, image processing method and program |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2010-253259 | 2010-11-11 | ||
JP2010253259 | 2010-11-11 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012063469A1 true WO2012063469A1 (ja) | 2012-05-18 |
Family
ID=46050634
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2011/006230 WO2012063469A1 (ja) | 2010-11-11 | 2011-11-08 | 画像処理装置、画像処理方法およびプログラム |
Country Status (5)
Country | Link |
---|---|
US (1) | US9001222B2 (ja) |
EP (1) | EP2640059B1 (ja) |
JP (1) | JP5694300B2 (ja) |
CN (1) | CN102714697B (ja) |
WO (1) | WO2012063469A1 (ja) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019133318A (ja) * | 2018-01-30 | 2019-08-08 | トヨタ自動車株式会社 | 位置推定システム |
JP2020031393A (ja) * | 2018-08-24 | 2020-02-27 | セコム株式会社 | 対象検知装置、画像処理装置、対象検知方法、画像処理方法、及びプログラム |
US11245849B2 (en) | 2017-05-18 | 2022-02-08 | Sony Corporation | Information processing apparatus and information processing method |
JP2022051761A (ja) * | 2016-06-30 | 2022-04-01 | マジック リープ, インコーポレイテッド | 3次元空間内の姿勢推定 |
Families Citing this family (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104123716B (zh) * | 2013-04-28 | 2016-08-10 | 腾讯科技(深圳)有限公司 | 检测图像稳定性的方法、装置和终端 |
CN103325109B (zh) * | 2013-05-29 | 2016-02-17 | 湖南源信光电科技有限公司 | 适用于墙装式全景摄像机的鱼眼图像的畸变校正方法 |
JP6376873B2 (ja) * | 2014-07-16 | 2018-08-22 | キヤノン株式会社 | 画像処理装置、画像処理方法及びプログラム |
CN105635552B (zh) * | 2014-10-30 | 2019-10-11 | 宇龙计算机通信科技(深圳)有限公司 | 一种防抖拍照方法、装置及终端 |
KR102281184B1 (ko) * | 2014-11-20 | 2021-07-23 | 삼성전자주식회사 | 영상 보정 방법 및 장치 |
CN104700400A (zh) * | 2015-01-30 | 2015-06-10 | 天津科技大学 | 一种基于特征点训练的高精度图像匹配方法 |
JP6435049B2 (ja) * | 2015-07-15 | 2018-12-05 | 日本電信電話株式会社 | 画像検索装置及び方法、撮影時刻推定装置及び方法、反復構造抽出装置及び方法、並びにプログラム |
CN107113376B (zh) * | 2015-07-31 | 2019-07-19 | 深圳市大疆创新科技有限公司 | 一种图像处理方法、装置及摄像机 |
JP6504036B2 (ja) * | 2015-11-27 | 2019-04-24 | カシオ計算機株式会社 | 移動検出装置、移動検出方法、及び、プログラム |
KR20180005528A (ko) * | 2016-07-06 | 2018-01-16 | 삼성전자주식회사 | 영상 처리를 위한 디스플레이 장치 및 방법 |
US10453204B2 (en) * | 2016-12-06 | 2019-10-22 | Adobe Inc. | Image alignment for burst mode images |
CN107292849B (zh) * | 2017-06-30 | 2020-01-03 | 维沃移动通信有限公司 | 一种图像处理方法及移动终端 |
CN109672818B (zh) * | 2017-10-16 | 2020-12-22 | 华为技术有限公司 | 一种调整图像质量的方法及装置 |
CN109194878B (zh) * | 2018-11-08 | 2021-02-19 | 深圳市闻耀电子科技有限公司 | 视频图像防抖方法、装置、设备和存储介质 |
CN110475067B (zh) * | 2019-08-26 | 2022-01-18 | Oppo广东移动通信有限公司 | 图像处理方法和装置、电子设备、计算机可读存储介质 |
CN111698421B (zh) * | 2020-06-09 | 2021-12-17 | 展讯通信(上海)有限公司 | 一种图像处理方法、装置及计算机可读存储介质 |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000341582A (ja) * | 1999-05-31 | 2000-12-08 | Sony Corp | 撮像装置及びその方法 |
JP2006295626A (ja) | 2005-04-12 | 2006-10-26 | Canon Inc | 魚眼像処理装置及びその方法並びに魚眼像撮像装置 |
JP2007129587A (ja) * | 2005-11-04 | 2007-05-24 | Canon Inc | 撮像装置、交換式レンズ装置、撮像制御方法、歪曲収差補正用データ処理方法、及びコンピュータプログラム |
JP2008219124A (ja) * | 2007-02-28 | 2008-09-18 | Sanyo Electric Co Ltd | 撮像装置 |
JP2008236645A (ja) * | 2007-03-23 | 2008-10-02 | Fujifilm Corp | 撮影装置 |
JP2009021929A (ja) * | 2007-07-13 | 2009-01-29 | Sanyo Electric Co Ltd | 撮像装置及び撮像方法 |
JP2009171341A (ja) * | 2008-01-17 | 2009-07-30 | Sanyo Electric Co Ltd | ぶれ補正装置及び撮像装置 |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4566591B2 (ja) * | 2004-03-19 | 2010-10-20 | キヤノン株式会社 | 画像変形推定方法および画像変形推定装置 |
JP4418317B2 (ja) * | 2004-07-21 | 2010-02-17 | オリンパス株式会社 | 撮像装置 |
JP4958610B2 (ja) | 2007-04-06 | 2012-06-20 | キヤノン株式会社 | 画像防振装置、撮像装置及び画像防振方法 |
US7916177B2 (en) * | 2007-08-03 | 2011-03-29 | Panasonic Corporation | Image-capturing apparatus, image-capturing method and program for detecting and correcting image blur |
JP4926920B2 (ja) * | 2007-11-16 | 2012-05-09 | キヤノン株式会社 | 防振画像処理装置及び防振画像処理方法 |
US20100079605A1 (en) * | 2008-09-29 | 2010-04-01 | William Marsh Rice University | Sensor-Assisted Motion Estimation for Efficient Video Encoding |
-
2011
- 2011-11-08 US US13/521,133 patent/US9001222B2/en not_active Expired - Fee Related
- 2011-11-08 JP JP2012512146A patent/JP5694300B2/ja not_active Expired - Fee Related
- 2011-11-08 EP EP11840319.5A patent/EP2640059B1/en not_active Not-in-force
- 2011-11-08 WO PCT/JP2011/006230 patent/WO2012063469A1/ja active Application Filing
- 2011-11-08 CN CN201180005738.1A patent/CN102714697B/zh not_active Expired - Fee Related
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000341582A (ja) * | 1999-05-31 | 2000-12-08 | Sony Corp | 撮像装置及びその方法 |
JP2006295626A (ja) | 2005-04-12 | 2006-10-26 | Canon Inc | 魚眼像処理装置及びその方法並びに魚眼像撮像装置 |
JP2007129587A (ja) * | 2005-11-04 | 2007-05-24 | Canon Inc | 撮像装置、交換式レンズ装置、撮像制御方法、歪曲収差補正用データ処理方法、及びコンピュータプログラム |
JP2008219124A (ja) * | 2007-02-28 | 2008-09-18 | Sanyo Electric Co Ltd | 撮像装置 |
JP2008236645A (ja) * | 2007-03-23 | 2008-10-02 | Fujifilm Corp | 撮影装置 |
JP2009021929A (ja) * | 2007-07-13 | 2009-01-29 | Sanyo Electric Co Ltd | 撮像装置及び撮像方法 |
JP2009171341A (ja) * | 2008-01-17 | 2009-07-30 | Sanyo Electric Co Ltd | ぶれ補正装置及び撮像装置 |
Non-Patent Citations (1)
Title |
---|
See also references of EP2640059A4 |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2022051761A (ja) * | 2016-06-30 | 2022-04-01 | マジック リープ, インコーポレイテッド | 3次元空間内の姿勢推定 |
US11245849B2 (en) | 2017-05-18 | 2022-02-08 | Sony Corporation | Information processing apparatus and information processing method |
JP2019133318A (ja) * | 2018-01-30 | 2019-08-08 | トヨタ自動車株式会社 | 位置推定システム |
JP2020031393A (ja) * | 2018-08-24 | 2020-02-27 | セコム株式会社 | 対象検知装置、画像処理装置、対象検知方法、画像処理方法、及びプログラム |
JP7194534B2 (ja) | 2018-08-24 | 2022-12-22 | セコム株式会社 | 対象検知装置、画像処理装置、対象検知方法、画像処理方法、及びプログラム |
Also Published As
Publication number | Publication date |
---|---|
US20120281146A1 (en) | 2012-11-08 |
EP2640059A4 (en) | 2014-04-09 |
EP2640059A1 (en) | 2013-09-18 |
JP5694300B2 (ja) | 2015-04-01 |
CN102714697B (zh) | 2016-06-22 |
JPWO2012063469A1 (ja) | 2014-05-12 |
CN102714697A (zh) | 2012-10-03 |
US9001222B2 (en) | 2015-04-07 |
EP2640059B1 (en) | 2018-08-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2012063469A1 (ja) | 画像処理装置、画像処理方法およびプログラム | |
US8798387B2 (en) | Image processing device, image processing method, and program for image processing | |
US10438360B2 (en) | Video processing apparatus, video processing method, and storage medium | |
EP1578116B1 (en) | Image processor | |
JP6395506B2 (ja) | 画像処理装置および方法、プログラム、並びに撮像装置 | |
KR101071352B1 (ko) | 좌표맵을 이용한 팬틸트줌 카메라 기반의 객체 추적 장치 및 방법 | |
US20090290809A1 (en) | Image processing device, image processing method, and program | |
US8774550B2 (en) | Picture processing device, picture processing method, integrated circuit, and program | |
CN110008795B (zh) | 图像目标追踪方法及其***与计算机可读取记录介质 | |
JP2010118862A (ja) | 画像処理装置、撮像装置及び画像処理方法 | |
JPWO2012063467A1 (ja) | 画像処理装置、画像処理方法、プログラム及び撮影装置 | |
US20200099854A1 (en) | Image capturing apparatus and image recording method | |
JP5449980B2 (ja) | 動き補正装置およびその方法 | |
JP5539565B2 (ja) | 撮像装置及び被写体追跡方法 | |
JP5247419B2 (ja) | 撮像装置および被写体追跡方法 | |
CN111489376A (zh) | 跟踪交互设备的方法、装置、终端设备及存储介质 | |
JP2011259044A (ja) | 画像処理装置、方法 | |
JP6188369B2 (ja) | 撮像装置 | |
JP6525693B2 (ja) | 画像処理装置及び画像処理方法 | |
JP2012199814A (ja) | 撮像装置、ぶれ補正方法、制御プログラム、及び制御プログラムを記録する記録媒体 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201180005738.1 Country of ref document: CN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2012512146 Country of ref document: JP |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11840319 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13521133 Country of ref document: US Ref document number: 2011840319 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |