WO2015026002A1 - Image matching apparatus and image matching method using same - Google Patents

Image matching apparatus and image matching method using same Download PDF

Info

Publication number
WO2015026002A1
WO2015026002A1 PCT/KR2013/008936 KR2013008936W WO2015026002A1 WO 2015026002 A1 WO2015026002 A1 WO 2015026002A1 KR 2013008936 W KR2013008936 W KR 2013008936W WO 2015026002 A1 WO2015026002 A1 WO 2015026002A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
transform function
feature points
image sensor
estimating
Prior art date
Application number
PCT/KR2013/008936
Other languages
French (fr)
Korean (ko)
Inventor
이준성
오재윤
김곤수
Original Assignee
삼성테크윈 주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 삼성테크윈 주식회사 filed Critical 삼성테크윈 주식회사
Publication of WO2015026002A1 publication Critical patent/WO2015026002A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/757—Matching configurations of points or features

Definitions

  • An embodiment of the present invention relates to an image matching device and an image matching method using the same.
  • An embodiment of the present invention provides an image matching device for real-time image registration and an image matching method using the same.
  • An image matching device may include: a first estimation function of estimating a first transform function based on feature point information extracted from a first image photographed by a first image sensor and a second image photographed by a second image sensor; 1 transform function estimator; And a zoom transform function estimator configured to estimate a third transform function of adjusting the first transform function based on zoom information of the first image sensor and the second image sensor.
  • the zoom conversion function estimator may estimate the third conversion function when the zoom state of at least one of the first image sensor and the second image sensor is changed.
  • the first transform function estimator may set a region of interest in the first image and the second image, and estimate the first transform function based on feature point information extracted from the set region of interest.
  • the apparatus may further include a transform function selector configured to select the first transform function or the third transform function as a final transform function according to whether the first transform function is estimated.
  • a transform function selector configured to select the first transform function or the third transform function as a final transform function according to whether the first transform function is estimated.
  • the transform function selector may select the third transform function as a final transform function until a new first transform function is estimated.
  • the first transform function estimator may include a feature point detector configured to detect feature points of the first image and the second image; A feature point selector for selecting corresponding feature points between the detected feature points of the first image and the second image; And a first estimator estimating the first transform function based on the selected corresponding feature points.
  • the feature point selector may include: a patch image acquisition unit configured to acquire a patch image centering on feature points of the first image and the second image; A candidate selector that selects candidate feature points corresponding to the remaining images from each feature point of the reference image among the first image and the second image; A similarity determination unit that determines similarity between patch images of feature points of the reference image and patch images of candidate feature points of the remaining images; And a corresponding feature point selector for selecting a corresponding feature point corresponding to the feature point of the reference image among the candidate feature points based on the similarity determination result.
  • the zoom transform function estimator may include: a scale determiner configured to determine a scale transform coefficient corresponding to the zoom information; A second transform function estimator for estimating a second transform function by adjusting the first transform function based on the scale transform coefficients; And a third transform function estimator configured to estimate the third transform function from the second transform function based on a center offset value between the first image and the second image matched by the second transform function. have.
  • the scale determiner may determine the scale conversion coefficient from a relationship between previously stored zoom information for each image sensor and the scale conversion coefficient.
  • the apparatus may further include a matching unit that matches the first image and the second image using the selected first transform function or third transform function.
  • the estimating of the third transform function may include estimating the third transform function when the zoom state of at least one of the first image sensor and the second image sensor is changed.
  • the estimating of the first transform function may include setting a region of interest in the first image and the second image and estimating the first transform function based on feature point information extracted from the set region of interest. have.
  • the method may further include selecting the first transform function or the third transform function as a final transform function according to whether the first transform function is estimated.
  • the third transform function is selected as the final transform function until a new first transform function is estimated. It may include;
  • the estimating of the first transform function may include detecting feature points of the first image and the second image; Selecting corresponding feature points corresponding to the feature points of the detected first and second images; And estimating the first transform function based on the selected corresponding feature points.
  • the feature point selecting step may include obtaining a patch image centering on feature points of the first image and the second image; Selecting candidate feature points corresponding to other feature points of the reference image among the first image and the second image; Determining similarity between patch images of feature points of the reference image and patch images of candidate feature points of the remaining images; And selecting a corresponding feature point corresponding to the feature point of the reference image among the candidate feature points based on the similarity determination result.
  • the estimating of the third transform function may include: determining a scale transform coefficient corresponding to the zoom information; Estimating a second transform function by adjusting the first transform function based on the scale transform coefficients; And estimating the third transform function from the second transform function based on the center offset value between the first image and the second image matched by the second transform function.
  • the determining of the scale conversion coefficient may include determining the scale conversion coefficient from a relationship between previously stored zoom information for each image sensor and the scale conversion coefficient.
  • the method may further include registering the first image and the second image by using the selected first transform function or third transform function.
  • the image registration device enables real-time image registration when the zoom information is changed.
  • FIG. 1 is a block diagram schematically illustrating an image fusion system according to an embodiment of the present invention.
  • FIG. 2 is a block diagram schematically illustrating an image matching device according to an embodiment of the present invention.
  • FIG. 3 is a block diagram schematically illustrating a first transform function estimator according to an embodiment of the present invention.
  • FIG. 4 is an exemplary view illustrating feature point selection according to an embodiment of the present invention.
  • FIG. 5 is a block diagram schematically illustrating a zoom transform function estimating unit according to an embodiment of the present invention.
  • FIG. 8 is a flowchart illustrating an image registration method according to an embodiment of the present invention.
  • FIG. 9 is a flowchart for explaining a method of selecting corresponding feature points of FIG. 8.
  • An image matching device may include: a first estimation function of estimating a first transform function based on feature point information extracted from a first image photographed by a first image sensor and a second image photographed by a second image sensor; 1 transform function estimator; And a zoom transform function estimator configured to estimate a third transform function of adjusting the first transform function based on zoom information of the first image sensor and the second image sensor.
  • first and second may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from another.
  • Embodiments of the present invention can be represented by functional block configurations and various processing steps. Such functional blocks may be implemented in various numbers of hardware or / and software configurations that perform particular functions. For example, embodiments of the invention may be implemented directly, such as memory, processing, logic, look-up table, etc., capable of executing various functions by the control of one or more microprocessors or other control devices. Circuit configurations can be employed. Similar to the components of an embodiment of the present invention may be implemented in software programming or software elements, embodiments of the present invention include various algorithms implemented in combinations of data structures, processes, routines or other programming constructs. It may be implemented in a programming or scripting language such as C, C ++, Java, assembler, or the like.
  • inventions may be implemented with an algorithm running on one or more processors.
  • embodiments of the present invention may employ the prior art for electronic configuration, signal processing, and / or data processing.
  • Terms such as mechanism, element, means, configuration can be used broadly and are not limited to mechanical and physical configurations. The term may include the meaning of a series of routines of software in conjunction with a processor or the like.
  • FIG. 1 is a block diagram schematically illustrating an image fusion system according to an embodiment of the present invention.
  • the image fusion system 1 of the present invention includes a first image sensor 10, a second image sensor 20, an image matching device 30, an image fusion device 40, and a display device 50. ).
  • the first image sensor 10 and the second image sensor 20 may be cameras having different characteristics of photographing the same scene and providing image information.
  • the first image sensor 10 and the second image sensor 20 may have a pan tilt zoom (PTZ) function, and may be panned and tilted together to acquire an image of the same point at each zoom magnification.
  • the first image sensor 10 and the second image sensor 20 are integrally installed inside and outside offices, houses, hospitals, as well as public buildings requiring banks and security, and are used for access control or crime prevention. Depending on the location and purpose of use, it can have various shapes such as straight and dome shaped.
  • the first image sensor 10 is a visible light camera, and acquires image information by detecting light to generate a first image, which is a visible image according to a luminance distribution of an object.
  • the visible camera may be a camera using a CCD or a CMOS as an image pickup device.
  • the second image sensor 20 is an infrared light camera (or thermal camera), which detects radiant energy (thermal energy) emitted by an object, detects it as an infrared wavelength form of electromagnetic waves, and measures the intensity of thermal energy.
  • the second image may be a thermal image having different colors according to the intensity.
  • the image matching device 30 performs image registration by matching the positional relationship of two or more images obtained from different sensors with the same scene in one coordinate system. In the field of surveillance system and medical imaging, image registration using two or more sensors to generate a single fusion image should be performed.
  • the image matching device 30 registers the first image photographed by the first image sensor 10 and the second image photographed by the second image sensor 20. To this end, the image matching device 30 may extract feature point information extracted from the first image and the second image, and zoom information (eg, a zoom ratio) of the first image sensor 10 and the second image sensor 20. Estimate the transform function as the basis.
  • the transform function is a matrix representing a correspondence relationship between feature points of each of the first and second images.
  • the image matching device 30 matches the first image and the second image by applying the estimated conversion function.
  • the image fusion device 40 performs signal processing to output the received image signal as a signal conforming to the display standard.
  • the image fusion device 40 fuses the matched first image and the second image.
  • an infrared camera may well indicate the thermal distribution of an object, but the shape of the measured object is not clear, and a visible light camera may clearly indicate the shape of the object but not the thermal distribution of the object.
  • the image fusion device 40 may display an image of an object using the mutual strengths and weaknesses of the visible light camera and the infrared light camera, and may clearly display the thermal distribution of the object.
  • the image fusion device 40 reduces noise with respect to the first image and the second image, and performs gamma correction, color filter array interpolation, color matrix, and color correction.
  • Image signal processing may be performed to improve image quality such as correction and color enhancement.
  • the image fusion device 40 may generate an image file by compressing the data of the fusion image by processing an image signal for improving image quality, or may restore the image data from the image file.
  • the compressed format of the image may include a reversible format or an irreversible format.
  • the image fusion device 40 can also perform color processing, blur processing, edge enhancement processing, image analysis processing, image recognition processing, image effect processing, and the like. Face recognition, scene recognition processing, and the like can be performed by the image recognition processing.
  • the display device 50 provides the user with a fusion image output from the image fusion device 40, so that the user can monitor the displayed image.
  • the display apparatus 50 may display a fusion image in which the first image and the second image are overlapped.
  • the display device 50 may be formed of a liquid crystal display panel (LCD), an organic light emitting display panel (OLED), an electrophoretic display panel (EPD), and the like.
  • the display device 50 may be provided in the form of a touch screen to receive an input through a user's touch and operate as a user input interface.
  • FIG. 2 is a block diagram schematically illustrating an image matching device according to an embodiment of the present invention.
  • the image matching device 30 may include a first transform function estimator 301, a zoom transform function estimator 303, a transform function selector 305, and a matcher 307. .
  • the first transform function estimator 301 uses the first transform function based on the feature point information extracted from the first image photographed by the first image sensor 10 and the second image photographed by the second image sensor 20. H1) can be estimated.
  • the first transform function estimator 301 may newly estimate the first transform function H1 whenever the zoom state of the first image sensor 10 and the second image sensor 20 changes.
  • the first transform function estimator 301 sets the region of interest without performing feature point detection for the entire first image and the second image, and performs a feature point detection only on the set region of interest, thereby reducing the amount of computation and thus converting the first transform function.
  • H1 The estimation time can be shortened.
  • the ROI may be a region where the photographing region overlaps between the first image and the second image.
  • the first transform function estimator 301 estimates the first transform function H1 through a process of extracting feature points from the first image and the second image and selecting a corresponding pair of feature points, the matching error rate is small. Since the new first transform function H1 must be estimated each time the zoom state is changed, it is difficult to perform real-time matching.
  • the first conversion function estimator 301 when the zoom state of at least one of the first image sensor 10 and the second image sensor 20 is changed, the first conversion function estimator 301 newly converts the first state according to the change of the zoom state. Until the function (H1) estimation is completed, matching is performed with the transform function estimated quickly with a small amount of computation through a simpler estimation process, so that real-time matching is possible even when there is a zoom state change.
  • the zoom conversion function estimator 303 estimates a third conversion function H3 that adjusts the first conversion function H1 based on the zoom information of the first image sensor 10 and the second image sensor 20. Can be. When at least one of the first image sensor 10 and the second image sensor 20 is changed while the image is captured while the first image sensor 10 and the second image sensor 20 are fixed, The zoom transform function estimator 303 may quickly estimate the third transform function H3 by adjusting the first transform function H1 estimated in the fixed zoom state. Since the zoom transform function estimator 303 estimates the third transform function H3 that adjusts the first transform function H1 estimated with only zoom information without the feature point extraction process, the first transform function estimator 301 Real-time matching is possible while E estimates the new first transform function H1.
  • the first transform function estimating unit 301 sets an image of which the photographing area is reduced by changing the zoom state among the first image and the second image as the reference image, sets the photographing region of the reference image as the region of interest,
  • the first transform function H1 may be estimated by extracting feature points only for the ROI of each of the second images. Accordingly, it is not necessary to extract a feature point in an area that is not a common area between the first image and the second image, thereby reducing computation time.
  • the transform function selector 305 may select the first transform function H1 or the third transform function H3 as the final transform function H, depending on whether the first transform function H1 is estimated.
  • the third transform function H3 estimated by 303 is selected as the final transform function H.
  • the transform function selector 305 calculates the first estimated by the first transform function estimator 301.
  • the conversion function H1 is selected as the final conversion function H.
  • the selection method of the conversion function selection unit 305 is shown in Equation 1 below.
  • the matching unit 307 matches the first image and the second image by using the transform function H selected from the first transform function H1 or the third transform function H3.
  • FIG. 3 is a block diagram schematically illustrating a first transform function estimator according to an embodiment of the present invention.
  • 4 is an exemplary view illustrating feature point selection according to an embodiment of the present invention.
  • the first transform function estimator 301 may include a feature point detector 311, a feature point selector 341, and an estimator 391.
  • the feature point detector 311 may detect the feature point F1 of the first image captured by the first image sensor 10, and the feature point of the second image captured by the second image sensor 20. And a second feature point detector 331 for detecting (F2).
  • the first feature point detector 321 and the second feature point detector 331 may be separately or integrally implemented to perform feature point detection sequentially or in parallel.
  • the feature point detector 311 uses a SIFT algorithm, a HARRIS corner algorithm, a SUSAN algorithm, and the like to determine corners, edges, contours, and line intersections from the first and second images, respectively. Can be extracted as feature points.
  • the feature point detection algorithm is not particularly limited, and various feature point extraction algorithms may be used.
  • the feature point selector 341 may select corresponding feature points corresponding to feature points of the first image and the second image.
  • the feature point selector 341 may include a patch image acquirer 351, a candidate selector 361, a similarity determiner 371, and a corresponding feature point selector 381.
  • the patch image acquisition unit 351 may acquire patch images of the feature points of the first image and the second image.
  • the patch image may have an N ⁇ N size around the feature point.
  • the candidate selector 361 may use one of the first image and the second image as a reference image, and select candidate feature points corresponding to the remaining images from each of the feature points of the reference image.
  • the feature points of the two images acquired for the same scene indicate localization.
  • the candidate selector 361 may select, as candidate feature points, feature points within a block having a predetermined size based on feature points of the reference image in the remaining images.
  • the block size can be flexibly optimized according to the field of view (FOV) and viewing direction of the two image sensors. For example, the closer the viewing angle and the viewing direction of the two image sensors are, the smaller the block size can be, and the further the larger the block size can be.
  • the candidate selector 361 may select, as candidate feature points, feature points of the remaining image having a distance from the feature point of the reference image within a threshold based on the block size.
  • the similarity determiner 371 may determine similarity between the patch image of the feature point of the reference image and the patch images of the candidate feature points of the remaining images.
  • the similarity determination may use normalized mutual information and gradient direction information as parameters.
  • Normal mutual information is information that normalizes mutual information representing a statistical correlation between two random variables.
  • the method of calculating the normal mutual information and the gradient direction information is a well-known algorithm and method, and detailed description thereof will be omitted in the detailed description of this embodiment.
  • the corresponding feature point selector 381 may select a corresponding feature point among candidate feature points based on the similarity determination result of the similarity determiner 371.
  • the corresponding feature point selector 381 may select a pair of corresponding feature points using the feature point having the greatest similarity among the candidate feature points of the remaining images as the corresponding feature point corresponding to the feature point of the reference image.
  • FIG. 4 illustrates an example in which feature points are respectively detected from a first image I1 and a second image I2, and a pair of corresponding feature points is selected based on the first image I1 as a reference image.
  • the candidate feature points f21, f22, and f23 of the second image I2 are selected with respect to the feature point f1, which is one of the plurality of feature points of the first image I1.
  • the candidate feature points f21, f22, and f23 are feature points positioned in an area CFA within a predetermined distance from a position corresponding to the feature point f1 of the second image I2.
  • the estimator 391 may estimate the first transform function H1 based on the selected corresponding feature points.
  • the estimator 391 may estimate the first transform function H1 using a random sample consensus (RANSAC) or a locally optimized RANSAC (LO-RANSAC) algorithm.
  • the first transform function H1 may be expressed as Equation 2 below.
  • Each component h11 to h33 of the first transform function H1 is rotation information indicating at which rotation angle to rotate, translation information indicating how much to move in the x, y, and z directions, and x and scaling information indicating how much to change the size in the y and z directions.
  • FIG. 5 is a block diagram schematically illustrating a zoom transform function estimating unit according to an embodiment of the present invention.
  • the zoom conversion function estimator 303 adjusts the first conversion function H1 based on the zoom information Z1 and Z2 of the first image sensor 10 and the second image sensor 20.
  • One third transform function H3 can be estimated.
  • the zoom information Z1 and Z2 may be parameters representing a zoom magnification.
  • the zoom transform function estimator 303 may include a scale determiner 313, a second transform function estimator 353, and a third transform function estimator 373.
  • the scale determiner 313 may determine the scale conversion factor S corresponding to the zoom information Z1 and Z2.
  • the scale conversion coefficient S is a coefficient representing the degree of conversion of the image size. Since the size and the focal length of each zoom section of the zoom lens are different for each image sensor, the image size conversion ratio corresponding to the zoom magnification may be different for each image sensor. Accordingly, the scale determiner 313 calculates the scale conversion coefficient S corresponding to the zoom magnification of the corresponding image sensor by using a graph or a look-up table indicating the relationship between the zoom magnification and the scale conversion coefficient for each image sensor previously stored in a memory or the like. You can decide.
  • the second transform function estimator 353 may estimate the second transform function H2 by adjusting the first transform function H1 based on the scale transform coefficient S.
  • the h11 and h22 components of the first transform function H1 are size conversion information in the x and y directions, respectively, when matching the remaining images with the coordinates of the reference image using either the first image or the second image as the reference image. It includes. Accordingly, the second transform function H2 may be expressed by Equation 3 by dividing the h11 and h22 components of the first transform function H1 by the scale transform coefficient S.
  • the third transform function estimator 373 extracts an offset value O between the first image and the second image matched by the second transform function H2 and based on the offset value O, the second transform function.
  • the offset value O is the degree of center misalignment between the first image and the second image that are aligned and aligned with the second transform function H2.
  • the h13 and h23 components of the first transform function H1 are parallel shifts that move the x and y coordinates when the remaining images are aligned with the coordinates of the reference image using either the first image or the second image as the reference image. Contains information.
  • Equation 4 An offset value O of (tx, ty) from the center coordinates x1 and y1 of the first image and the center coordinates x2 and y2 of the second image may be extracted as in Equation 4 below. Accordingly, the third transform function H3 may be expressed as shown in Equation 5 by adding offset values tx and ty to the h13 and h23 components of the second transform function H2, respectively.
  • FIG. 6 illustrates a first image (a) photographed by the first image sensor 10 of 1x zoom and a second image b photographed by the second image sensor 20 whose zoom state is changed from 1x zoom to 2x zoom.
  • the aligned result image c is illustrated.
  • the second transform function H2 since the second transform function H2 includes only image size conversion information, the centers x1 and y1 of the first image a as the reference image and the second image b 'that are size-converted are used. Of centers (x2, y2) are shifted.
  • FIG. 7 illustrates a first image (a) photographed by the first image sensor 10 of 1x zoom and a second image b photographed by the second image sensor 20 whose zoom state is changed from 1x zoom to 2x zoom.
  • the aligned result image d is illustrated.
  • the third conversion function H3 includes image size conversion information and parallel movement information, the center of the first image a, which is the reference image, and the size of the second image b 'that has been size converted. The centers are aligned.
  • FIG. 8 is a flowchart illustrating an image registration method according to an embodiment of the present invention.
  • FIG. 9 is a flowchart for explaining a method of selecting corresponding feature points of FIG. 8.
  • a first conversion is performed based on feature point information extracted from a first image photographed by a first image sensor and a second image photographed by a second image sensor.
  • the function H1 can be estimated (S80A).
  • the image matching device may detect feature points F1 and F2 of the first image and the second image (S81).
  • Feature points may include corners, edges, contours, line intersections, and the like.
  • the image matching apparatus may select corresponding feature points between the detected feature points F1 and F2 of the first image and the second image (S82).
  • the image registration device may acquire a patch image centering on the feature points F1 and F2 of each of the first and second images (S821).
  • the image matching apparatus may select candidate feature points of the remaining images that may correspond to the feature points of the reference image which is one of the first image and the second image. For example, when the first image is a reference image, candidate feature points of the second image that can correspond to the feature points of the first image may be selected.
  • candidate feature points of the second image that can correspond to the feature points of the first image may be selected.
  • Candidate feature points may be selected based on locality, eg, distance between feature points.
  • the image matching apparatus may determine the similarity between the patch image of the feature point of the reference image and the patch images of the candidate feature points of the remaining image.
  • the degree of similarity may be determined using normal mutual information and gradient direction information between patch images.
  • the image matching apparatus may select a corresponding feature point corresponding to the feature point of the reference image from among the candidate feature points based on the similarity determination result (S827). For example, the image matching apparatus may select candidate feature points having the greatest similarity with the reference image feature points as corresponding feature points of the reference image feature points.
  • the image matching apparatus may estimate the first transform function H1 based on the selected corresponding feature points (S83).
  • the image matching device may select the first transform function H1 as the final transform function H (S87).
  • the image matching device may match the first image and the second image with the final transform function H (S88).
  • the image matching device is based on the zoom information (Z1, Z2) of the first image sensor and the second image sensor, before changing the zoom state.
  • the third transform function H3 may be estimated by adjusting the estimated first transform function H1 (80B).
  • the image matching device may determine the scale conversion factor S corresponding to the change of the zoom state.
  • the image matching device may previously store a graph or a look-up table indicating a relationship between the zoom magnification and the scale conversion coefficient for each image sensor, and determine the scale conversion factor S corresponding to the zoom magnification of the corresponding image sensor by using the same.
  • the image matching device may estimate the second transform function H2 by adjusting the first transform function H1 based on the scale transform coefficient S (S85).
  • the image matching device may estimate the second transform function H2 by applying the scale transform coefficient S to a component including the size transform information in the x and y directions among the components of the first transform function H1.
  • the image matching device estimates the third transform function H3 by adjusting the second transform function H2 based on the offset values between the first and second images matched and aligned with the second transform function H2. Can be (S86).
  • the image matching device applies an offset value to a component including parallel movement information in the x and y directions among the components of the first transform function H1, thereby converting the third transform function H3 from the second transform function H2. It can be estimated.
  • the image matching device may select the third transform function H3 as the final transform function H until a new first transform function H1 is estimated (S87).
  • the image matching device may select the newly estimated first transform function H1 as the final transform function H.
  • the image matching device may match the first image and the second image with the final transform function H (S88).
  • the image matcher estimates the first transform function H1 through the detection of the feature points on the ROI rather than the entire first image and the second image. can do.
  • the ROI may be a region where the photographing region overlaps between the first image and the second image. As a result, the image matching device may reduce the calculation function estimation amount and the calculation time.
  • the first image is described as a visible image and the second image as an example of a thermal image.
  • embodiments of the present invention are not limited thereto, and the first image and the second image are different from each other.
  • Embodiments of the present invention may be equally applicable to images obtained from sensors having different characteristics other than a time or visible light camera and an infrared light camera.
  • the image matching method according to the present invention can be embodied as computer readable codes on a computer readable recording medium.
  • Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like.
  • the computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.
  • functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the art to which the present invention belongs.
  • the above-described embodiments are applicable to boundary area surveillance such as GOP, surveillance requiring 24-hour real-time monitoring such as forest fire monitoring, detection of building and residential intrusion in a no-light or low light environment, tracking of missing and criminals in places such as mountains, medical imaging, etc. Can be.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The present invention discloses an image matching apparatus and an image matching method using the same. An image matching apparatus according to an embodiment may include a first transform function estimation unit for estimating a first transform function on the basis of feature point information extracted from a first image photographed by a first image sensor and a second image photographed by a second image sensor; and a zoom transform function estimation unit for estimating a third transform function adjusted from the first transform function on the basis of zoom information of the first image sensor and the second image sensor.

Description

영상 정합 장치 및 이를 이용한 영상 정합 방법Image Matching Device and Image Matching Method Using the Same
본 발명의 실시예는 영상 정합 장치 및 이를 이용한 영상 정합 방법에 관한 것이다. An embodiment of the present invention relates to an image matching device and an image matching method using the same.
최근에 감시 시스템과 의료 영상 등의 분야에서 변화 감지, 움직임 검출, 초해상도 영상 복원 및 물체 인식과 추적 등의 알고리즘의 정확도와 신뢰도를 향상시키기 위해 두 개 이상의 센서를 이용한 상호 보완적인 정보 융합 연구가 활발히 이루어지고 있다. Recently, complementary information fusion research using two or more sensors to improve the accuracy and reliability of algorithms such as change detection, motion detection, super resolution image reconstruction, and object recognition and tracking in surveillance systems and medical imaging. Actively done.
본 발명의 실시예는 실시간 영상 정합을 위한 영상 정합 장치 및 이를 이용한 영상 정합 방법을 제공한다. An embodiment of the present invention provides an image matching device for real-time image registration and an image matching method using the same.
본 발명의 일 실시예에 따른 영상 정합 장치는, 제1영상센서로 촬영한 제1영상 및 제2영상센서로 촬영한 제2영상으로부터 추출된 특징점 정보를 기초로 제1변환함수를 추정하는 제1 변환함수 추정부; 및 상기 제1영상센서와 제2영상센서의 줌 정보를 기초로 상기 제1변환함수를 조정한 제3변환함수를 추정하는 줌 변환함수 추정부;를 포함할 수 있다. An image matching device according to an embodiment of the present invention may include: a first estimation function of estimating a first transform function based on feature point information extracted from a first image photographed by a first image sensor and a second image photographed by a second image sensor; 1 transform function estimator; And a zoom transform function estimator configured to estimate a third transform function of adjusting the first transform function based on zoom information of the first image sensor and the second image sensor.
상기 줌 변환함수 추정부는, 상기 제1영상센서와 제2영상센서 중 적어도 하나의 줌 상태가 변경된 경우 상기 제3변환함수를 추정할 수 있다. The zoom conversion function estimator may estimate the third conversion function when the zoom state of at least one of the first image sensor and the second image sensor is changed.
상기 제1 변환함수 추정부는, 상기 제1영상 및 제2영상에서 관심영역을 설정하고, 상기 설정된 관심영역으로부터 추출된 특징점 정보를 기초로 상기 제1변환함수를 추정할 수 있다.The first transform function estimator may set a region of interest in the first image and the second image, and estimate the first transform function based on feature point information extracted from the set region of interest.
상기 장치는, 상기 제1변환함수의 추정 여부에 따라, 상기 제1변환함수 또는 제3변환함수를 최종 변환함수로 선택하는 변환함수 선택부;를 더 포함할 수 있다. The apparatus may further include a transform function selector configured to select the first transform function or the third transform function as a final transform function according to whether the first transform function is estimated.
상기 변환함수 선택부는, 상기 제1영상센서와 제2영상센서 중 적어도 하나의 줌 상태가 변경된 경우, 새로운 제1변환함수가 추정될 때까지, 상기 제3변환함수를 최종 변환함수로 선택할 수 있다 When the zoom state of at least one of the first image sensor and the second image sensor is changed, the transform function selector may select the third transform function as a final transform function until a new first transform function is estimated.
상기 제1 변환함수 추정부는, 상기 제1영상 및 제2영상의 특징점들을 검출하는 특징점 검출부; 상기 검출된 제1영상 및 제2영상의 특징점들 간에 대응하는 대응 특징점들을 선별하는 특징점 선별부; 및 상기 선별된 대응 특징점들을 기초로 상기 제1변환함수를 추정하는 제1추정부;를 포함할 수 있다. The first transform function estimator may include a feature point detector configured to detect feature points of the first image and the second image; A feature point selector for selecting corresponding feature points between the detected feature points of the first image and the second image; And a first estimator estimating the first transform function based on the selected corresponding feature points.
상기 특징점 선별부는, 상기 제1영상 및 제2영상의 특징점을 중심으로 하는 패치 영상을 획득하는 패치영상 획득부; 상기 제1영상 및 제2영상 중 기준영상의 각 특징점에 대해, 나머지 영상에서 대응 가능한 후보 특징점들을 선별하는 후보 선별부; 상기 기준영상의 특징점의 패치 영상과 상기 나머지 영상의 후보 특징점들의 패치 영상들 간에 유사성을 판단하는 유사성 판단부; 및 상기 유사성 판단 결과를 기초로, 상기 후보 특징점들 중 기준영상의 특징점에 대응하는 대응 특징점을 선별하는 대응 특징점 선별부;를 포함할 수 있다. The feature point selector may include: a patch image acquisition unit configured to acquire a patch image centering on feature points of the first image and the second image; A candidate selector that selects candidate feature points corresponding to the remaining images from each feature point of the reference image among the first image and the second image; A similarity determination unit that determines similarity between patch images of feature points of the reference image and patch images of candidate feature points of the remaining images; And a corresponding feature point selector for selecting a corresponding feature point corresponding to the feature point of the reference image among the candidate feature points based on the similarity determination result.
상기 줌 변환함수 추정부는, 상기 줌 정보에 대응하는 스케일 변환 계수를 결정하는 스케일 결정부; 상기 스케일 변환 계수를 기초로 상기 제1변환함수를 조정하여 제2변환함수를 추정하는 제2 변환함수 추정부; 및 상기 제2변환함수에 의해 정합된 상기 제1영상 및 제2영상 간의 센터 오프셋 값을 기초로 상기 제2변환함수로부터 상기 제3변환함수를 추정하는 제3 변환함수 추정부;를 포함할 수 있다. The zoom transform function estimator may include: a scale determiner configured to determine a scale transform coefficient corresponding to the zoom information; A second transform function estimator for estimating a second transform function by adjusting the first transform function based on the scale transform coefficients; And a third transform function estimator configured to estimate the third transform function from the second transform function based on a center offset value between the first image and the second image matched by the second transform function. have.
상기 스케일 결정부는, 기 저장된 영상센서별 줌 정보와 스케일 변환 계수 간의 관계로부터 상기 스케일 변환 계수를 결정할 수 있다.The scale determiner may determine the scale conversion coefficient from a relationship between previously stored zoom information for each image sensor and the scale conversion coefficient.
상기 장치는, 상기 선택된 제1변환함수 또는 제3변환함수를 이용하여 상기 제1영상 및 제2영상을 정합하는 정합부;를 더 포함할 수 있다. The apparatus may further include a matching unit that matches the first image and the second image using the selected first transform function or third transform function.
본 발명의 일 실시예에 따른 영상 정합 방법은, 제1영상센서로 촬영한 제1영상 및 제2영상센서로 촬영한 제2영상으로부터 추출된 특징점 정보를 기초로 제1변환함수를 추정하는 단계; 및 상기 제1영상센서와 제2영상센서의 줌 정보를 기초로 상기 제1변환함수를 조정하여 제3변환함수를 추정하는 단계;를 포함할 수 있다.In the image matching method according to an embodiment of the present invention, estimating a first transform function based on feature point information extracted from a first image photographed by a first image sensor and a second image photographed by a second image sensor. ; And estimating a third conversion function by adjusting the first conversion function based on the zoom information of the first image sensor and the second image sensor.
상기 제3변환함수 추정 단계는, 상기 제1영상센서와 제2영상센서 중 적어도 하나의 줌 상태가 변경된 경우 상기 제3변환함수를 추정하는 단계;를 포함할 수 있다. The estimating of the third transform function may include estimating the third transform function when the zoom state of at least one of the first image sensor and the second image sensor is changed.
상기 제1 변환함수 추정 단계는, 상기 제1영상 및 제2영상에서 관심영역을 설정하고, 상기 설정된 관심영역으로부터 추출된 특징점 정보를 기초로 상기 제1변환함수를 추정하는 단계;를 포함할 수 있다. The estimating of the first transform function may include setting a region of interest in the first image and the second image and estimating the first transform function based on feature point information extracted from the set region of interest. have.
상기 방법은, 상기 제1변환함수의 추정 여부에 따라, 상기 제1변환함수 또는 제3변환함수를 최종 변환함수로 선택하는 단계;를 더 포함할 수 있다. The method may further include selecting the first transform function or the third transform function as a final transform function according to whether the first transform function is estimated.
상기 최종 변환함수 선택 단계는, 상기 제1영상센서와 제2영상센서 중 적어도 하나의 줌 상태가 변경된 경우, 새로운 제1변환함수가 추정될 때까지, 상기 제3변환함수를 최종 변환함수로 선택하는 단계;를 포함할 수 있다. In the selecting of the final transform function, when the zoom state of at least one of the first image sensor and the second image sensor is changed, the third transform function is selected as the final transform function until a new first transform function is estimated. It may include;
상기 제1변환함수 추정 단계는, 상기 제1영상 및 제2영상의 특징점들을 검출하는 단계; 상기 검출된 제1영상 및 제2영상의 특징점들 간에 대응하는 대응 특징점들을 선별하는 단계; 및 상기 선별된 대응 특징점들을 기초로 상기 제1변환함수를 추정하는 단계;를 포함할 수 있다. The estimating of the first transform function may include detecting feature points of the first image and the second image; Selecting corresponding feature points corresponding to the feature points of the detected first and second images; And estimating the first transform function based on the selected corresponding feature points.
상기 특징점 선별 단계는, 상기 제1영상 및 제2영상의 특징점을 중심으로 하는 패치 영상을 획득하는 단계; 상기 제1영상 및 제2영상 중 기준영상의 각 특징점에 대해, 나머지 영상에서 대응 가능한 후보 특징점들을 선별하는 단계; 상기 기준영상의 특징점의 패치 영상과 상기 나머지 영상의 후보 특징점들의 패치 영상들 간에 유사성을 판단하는 단계; 및 상기 유사성 판단 결과를 기초로, 상기 후보 특징점들 중 기준영상의 특징점에 대응하는 대응 특징점을 선별하는 단계;를 포함할 수 있다. The feature point selecting step may include obtaining a patch image centering on feature points of the first image and the second image; Selecting candidate feature points corresponding to other feature points of the reference image among the first image and the second image; Determining similarity between patch images of feature points of the reference image and patch images of candidate feature points of the remaining images; And selecting a corresponding feature point corresponding to the feature point of the reference image among the candidate feature points based on the similarity determination result.
상기 제3변환함수 추정 단계는, 상기 줌 정보에 대응하는 스케일 변환 계수를 결정하는 단계; 상기 스케일 변환 계수를 기초로 상기 제1변환함수를 조정하여 제2변환함수를 추정하는 단계; 및 상기 제2변환함수에 의해 정합된 상기 제1영상 및 제2영상 간의 센터 오프셋 값을 기초로 상기 제2변환함수로부터 상기 제3변환함수를 추정하는 단계;를 포함할 수 있다. The estimating of the third transform function may include: determining a scale transform coefficient corresponding to the zoom information; Estimating a second transform function by adjusting the first transform function based on the scale transform coefficients; And estimating the third transform function from the second transform function based on the center offset value between the first image and the second image matched by the second transform function.
상기 스케일 변환 계수 결정 단계는, 기 저장된 영상센서별 줌 정보와 스케일 변환 계수 간의 관계로부터 상기 스케일 변환 계수를 결정하는 단계;를 포함할 수 있다.The determining of the scale conversion coefficient may include determining the scale conversion coefficient from a relationship between previously stored zoom information for each image sensor and the scale conversion coefficient.
상기 방법은, 상기 선택된 제1변환함수 또는 제3변환함수를 이용하여 상기 제1영상 및 제2영상을 정합하는 단계;를 더 포함할 수 있다. The method may further include registering the first image and the second image by using the selected first transform function or third transform function.
본 발명의 실시예에 따른 영상 정합 장치는 줌 정보 변경시에 실시간 영상 정합이 가능하다. The image registration device according to the embodiment of the present invention enables real-time image registration when the zoom information is changed.
도 1은 본 발명의 일 실시예에 따른 영상 융합 시스템을 개략적으로 도시한 블록도이다. 1 is a block diagram schematically illustrating an image fusion system according to an embodiment of the present invention.
도 2는 본 발명의 일 실시예에 따른 영상 정합 장치를 개략적으로 도시한 블록도이다. 2 is a block diagram schematically illustrating an image matching device according to an embodiment of the present invention.
도 3은 본 발명의 일 실시예에 따른 제1 변환함수 추정부를 개략적으로 도시한 블록도이다. 3 is a block diagram schematically illustrating a first transform function estimator according to an embodiment of the present invention.
도 4는 본 발명의 일 실시예에 따른 특징점 선별을 설명하는 예시도이다.4 is an exemplary view illustrating feature point selection according to an embodiment of the present invention.
도 5는 본 발명의 일 실시예에 따른 줌 변환함수 추정부를 개략적으로 도시한 블록도이다. 5 is a block diagram schematically illustrating a zoom transform function estimating unit according to an embodiment of the present invention.
도 6 및 도 7은 본 발명의 일 실시예에 따른 정합을 예시적으로 나타낸 도면이다. 6 and 7 exemplarily show matching according to an embodiment of the present invention.
도 8은 본 발명의 일 실시예에 따른 영상 정합 방법을 설명하는 흐름도이다. 8 is a flowchart illustrating an image registration method according to an embodiment of the present invention.
도 9는 도 8의 대응 특징점을 선별하는 방법을 설명하는 흐름도이다. FIG. 9 is a flowchart for explaining a method of selecting corresponding feature points of FIG. 8.
본 발명의 일 실시예에 따른 영상 정합 장치는, 제1영상센서로 촬영한 제1영상 및 제2영상센서로 촬영한 제2영상으로부터 추출된 특징점 정보를 기초로 제1변환함수를 추정하는 제1 변환함수 추정부; 및 상기 제1영상센서와 제2영상센서의 줌 정보를 기초로 상기 제1변환함수를 조정한 제3변환함수를 추정하는 줌 변환함수 추정부;를 포함할 수 있다. An image matching device according to an embodiment of the present invention may include: a first estimation function of estimating a first transform function based on feature point information extracted from a first image photographed by a first image sensor and a second image photographed by a second image sensor; 1 transform function estimator; And a zoom transform function estimator configured to estimate a third transform function of adjusting the first transform function based on zoom information of the first image sensor and the second image sensor.
본 발명은 다양한 변환을 가할 수 있고 여러 가지 실시 예를 가질 수 있는 바, 특정 실시 예들을 도면에 예시하고 상세한 설명에 상세하게 설명하고자 한다. 그러나, 이는 본 발명을 특정한 실시 형태에 대해 한정하려는 것이 아니며, 본 발명의 사상 및 기술 범위에 포함되는 모든 변환, 균등물 내지 대체물을 포함하는 것으로 이해되어야 한다. 본 발명을 설명함에 있어서 관련된 공지 기술에 대한 구체적인 설명이 본 발명의 요지를 흐릴 수 있다고 판단되는 경우 그 상세한 설명을 생략한다. As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.
이하의 실시예에서, 제1, 제2 등의 용어는 다양한 구성요소들을 설명하는데 사용될 수 있지만, 구성요소들은 용어들에 의해 한정되어서는 안 된다. 용어들은 하나의 구성요소를 다른 구성요소로부터 구별하는 목적으로만 사용된다. In the following embodiments, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from another.
이하의 실시예에서 사용한 용어는 단지 특정한 실시 예를 설명하기 위해 사용된 것으로, 본 발명을 한정하려는 의도가 아니다. 단수의 표현은 문맥상 명백하게 다르게 뜻하지 않는 한, 복수의 표현을 포함한다. 이하의 실시예에서, "포함하다" 또는 "가지다" 등의 용어는 명세서상에 기재된 특징, 숫자, 단계, 동작, 구성요소, 부품 또는 이들을 조합한 것이 존재함을 지정하려는 것이지, 하나 또는 그 이상의 다른 특징들이나 숫자, 단계, 동작, 구성요소, 부품 또는 이들을 조합한 것들의 존재 또는 부가 가능성을 미리 배제하지 않는 것으로 이해되어야 한다.The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the following embodiments, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, one or more It is to be understood that it does not exclude in advance the possibility of the presence or addition of other features or numbers, steps, operations, components, components or combinations thereof.
본 발명의 실시예들은 기능적인 블록 구성들 및 다양한 처리 단계들로 나타내어질 수 있다. 이러한 기능 블록들은 특정 기능들을 실행하는 다양한 개수의 하드웨어 또는/및 소프트웨어 구성들로 구현될 수 있다. 예를 들어, 본 발명의 실시예들은 하나 이상의 마이크로프로세서들의 제어 또는 다른 제어 장치들에 의해서 다양한 기능들을 실행할 수 있는, 메모리, 프로세싱, 로직(logic), 룩업 테이블(look-up table) 등과 같은 직접 회로 구성들을 채용할 수 있다. 본 발명의 실시예의 구성 요소들이 소프트웨어 프로그래밍 또는 소프트웨어 요소들로 실행될 수 잇는 것과 유사하게, 본 발명의 실시예는 데이터 구조, 프로세스들, 루틴들 또는 다른 프로그래밍 구성들의 조합으로 구현되는 다양한 알고리즘을 포함하여, C, C++, 자바(Java), 어셈블러(assembler) 등과 같은 프로그래밍 또는 스크립팅 언어로 구현될 수 있다. 기능적인 측면들은 하나 이상의 프로세서들에서 실행되는 알고리즘으로 구현될 수 있다. 또한, 본 발명의 실시예들은 전자적인 환경 설정, 신호 처리, 및/또는 데이터 처리 등을 위하여 종래 기술을 채용할 수 있다. 매커니즘, 요소, 수단, 구성과 같은 용어는 넓게 사용될 수 있으며, 기계적이고 물리적인 구성들로서 한정되는 것은 아니다. 상기 용어는 프로세서 등과 연계하여 소프트웨어의 일련의 처리들(routines)의 의미를 포함할 수 있다.Embodiments of the present invention can be represented by functional block configurations and various processing steps. Such functional blocks may be implemented in various numbers of hardware or / and software configurations that perform particular functions. For example, embodiments of the invention may be implemented directly, such as memory, processing, logic, look-up table, etc., capable of executing various functions by the control of one or more microprocessors or other control devices. Circuit configurations can be employed. Similar to the components of an embodiment of the present invention may be implemented in software programming or software elements, embodiments of the present invention include various algorithms implemented in combinations of data structures, processes, routines or other programming constructs. It may be implemented in a programming or scripting language such as C, C ++, Java, assembler, or the like. The functional aspects may be implemented with an algorithm running on one or more processors. In addition, embodiments of the present invention may employ the prior art for electronic configuration, signal processing, and / or data processing. Terms such as mechanism, element, means, configuration can be used broadly and are not limited to mechanical and physical configurations. The term may include the meaning of a series of routines of software in conjunction with a processor or the like.
도 1은 본 발명의 일 실시예에 따른 영상 융합 시스템을 개략적으로 도시한 블록도이다. 1 is a block diagram schematically illustrating an image fusion system according to an embodiment of the present invention.
도 1을 참조하면, 본 발명의 영상 융합 시스템(1)은 제1영상센서(10), 제2영상센서(20), 영상 정합 장치(30), 영상 융합 장치(40) 및 디스플레이 장치(50)를 포함한다. Referring to FIG. 1, the image fusion system 1 of the present invention includes a first image sensor 10, a second image sensor 20, an image matching device 30, an image fusion device 40, and a display device 50. ).
제1영상센서(10) 및 제2영상센서(20)는 동일 장면을 촬영하여 영상 정보를 제공하는 서로 다른 특성의 카메라일 수 있다. 제1영상센서(10) 및 제2영상센서(20)는 팬틸트줌(PTZ) 기능을 구비하고, 함께 패닝 및 틸팅되면서 각각의 줌 배율로 동일 지점의 영상을 획득할 수 있다. 제1영상센서(10) 및 제2영상센서(20)는 사무실, 주택, 병원은 물론 은행이나 보안이 요구되는 공공건물 등의 내외에 일체로 설치되어 출입관리나 방범용으로 사용되며, 그 설치 장소 및 사용목적에 따라 일자형, 돔형 등 다양한 형태를 가질 수 있다.The first image sensor 10 and the second image sensor 20 may be cameras having different characteristics of photographing the same scene and providing image information. The first image sensor 10 and the second image sensor 20 may have a pan tilt zoom (PTZ) function, and may be panned and tilted together to acquire an image of the same point at each zoom magnification. The first image sensor 10 and the second image sensor 20 are integrally installed inside and outside offices, houses, hospitals, as well as public buildings requiring banks and security, and are used for access control or crime prevention. Depending on the location and purpose of use, it can have various shapes such as straight and dome shaped.
본 실시예에서 제1영상센서(10)는 가시광 카메라로서, 빛(light)을 검출하는 방식으로 영상 정보를 획득하여 물체의 휘도 분포에 따른 가시 영상인 제1영상을 생성한다. 예를 들어, 가시 카메라는 CCD 또는 CMOS를 촬상소자로 사용하는 카메라일 수 있다. In the present embodiment, the first image sensor 10 is a visible light camera, and acquires image information by detecting light to generate a first image, which is a visible image according to a luminance distribution of an object. For example, the visible camera may be a camera using a CCD or a CMOS as an image pickup device.
본 실시예에서 제2영상센서(20)는 적외광 카메라(또는 열상 카메라)로서, 물체가 발산하는 복사 에너지(열에너지)를 감지하여 전자파의 일종인 적외선 파장 형태로 검출하고, 열에너지의 강도를 측정하여 강도에 따라 각각 다른 색상을 나타내는 열 영상인 제2영상을 생성할 수 있다. In the present embodiment, the second image sensor 20 is an infrared light camera (or thermal camera), which detects radiant energy (thermal energy) emitted by an object, detects it as an infrared wavelength form of electromagnetic waves, and measures the intensity of thermal energy. The second image may be a thermal image having different colors according to the intensity.
영상 정합 장치(30)는 동일한 장면을 다른 센서로부터 얻은 두 개 이상의 영상들의 위치 관계를 대응시켜 하나의 좌표계로 정렬시키는 영상 정합을 수행한다. 감시시스템과 의료 영상 등의 분야에서 두 개 이상의 센서를 이용한 영상을 획득하여 하나의 융합 영상을 생성하는 시스템에서는 영상 정합을 수행하여야 한다. The image matching device 30 performs image registration by matching the positional relationship of two or more images obtained from different sensors with the same scene in one coordinate system. In the field of surveillance system and medical imaging, image registration using two or more sensors to generate a single fusion image should be performed.
영상 정합 장치(30)는 제1영상센서(10)로 촬영한 제1영상 및 제2영상센서(20)로 촬영한 제2영상을 정합한다. 이를 위해, 영상 정합 장치(30)는 제1영상 및 제2영상으로부터 추출된 특징점 정보 및 제1영상센서(10)와 제2영상센서(20)의 줌 정보(예를 들어, 줌 배율)를 기초로 변환함수를 추정한다. 변환함수는 제1영상 및 제2영상 각각의 특징점들 간의 대응 관계를 나타내는 행렬이다. 영상 정합 장치(30)는 추정된 변환함수를 적용하여 제1영상 및 제2영상을 정합한다. The image matching device 30 registers the first image photographed by the first image sensor 10 and the second image photographed by the second image sensor 20. To this end, the image matching device 30 may extract feature point information extracted from the first image and the second image, and zoom information (eg, a zoom ratio) of the first image sensor 10 and the second image sensor 20. Estimate the transform function as the basis. The transform function is a matrix representing a correspondence relationship between feature points of each of the first and second images. The image matching device 30 matches the first image and the second image by applying the estimated conversion function.
영상 융합 장치(40)는 수신된 영상 신호를 디스플레이 규격에 맞는 신호로 출력하는 신호 처리를 수행한다. 영상 융합 장치(40)는 정합된 제1영상과 제2영상을 융합한다. 예를 들어, 적외광 카메라는 물체의 열적 분포는 잘 나타낼 수 있으나 측정된 물체의 형상이 명확하지 않으며, 가시광 카메라는 물체의 형상은 명확하게 나타낼 수 있으나 물체의 열적 분포는 나타낼 수 없다. 영상 융합 장치(40)는 가시광 카메라와 적외광 카메라의 상호 장단점을 적절히 이용하여 물체의 영상을 표시함과 동시에 그 물체의 열적 분포 상태를 명확하게 나타낼 수 있다. The image fusion device 40 performs signal processing to output the received image signal as a signal conforming to the display standard. The image fusion device 40 fuses the matched first image and the second image. For example, an infrared camera may well indicate the thermal distribution of an object, but the shape of the measured object is not clear, and a visible light camera may clearly indicate the shape of the object but not the thermal distribution of the object. The image fusion device 40 may display an image of an object using the mutual strengths and weaknesses of the visible light camera and the infrared light camera, and may clearly display the thermal distribution of the object.
영상 융합 장치(40)는 제1영상 및 제2영상에 대해 노이즈를 저감하고, 감마 보정(Gamma Correction), 색필터 배열보간(color filter array interpolation), 색 매트릭스(color matrix), 색보정(color correction), 색 향상(color enhancement) 등의 화질 개선을 위한 영상 신호 처리를 수행할 수 있다. 또한, 영상 융합 장치(40)는 화질 개선을 위한 영상 신호 처리를 하여 융합 영상의 데이터를 압축 처리하여 영상 파일을 생성할 수 있으며, 또는 영상 파일로부터 영상 데이터를 복원할 수 있다. 영상의 압축형식은 가역 형식 또는 비가역 형식을 포함할 수 있다. The image fusion device 40 reduces noise with respect to the first image and the second image, and performs gamma correction, color filter array interpolation, color matrix, and color correction. Image signal processing may be performed to improve image quality such as correction and color enhancement. In addition, the image fusion device 40 may generate an image file by compressing the data of the fusion image by processing an image signal for improving image quality, or may restore the image data from the image file. The compressed format of the image may include a reversible format or an irreversible format.
또한, 영상 융합 장치(40)는 기능적으로 색채 처리, 블러 처리, 에지 강조 처리, 영상 해석 처리, 영상 인식 처리, 영상 이펙트 처리 등도 행할 수 있다. 영상 인식 처리로 얼굴 인식, 장면 인식 처리 등을 행할 수 있다. In addition, the image fusion device 40 can also perform color processing, blur processing, edge enhancement processing, image analysis processing, image recognition processing, image effect processing, and the like. Face recognition, scene recognition processing, and the like can be performed by the image recognition processing.
디스플레이 장치(50)는 영상 융합 장치(40)로부터 출력되는 융합 영상을 사용자에게 제공함으로써, 사용자가 디스플레이되는 영상을 모니터링할 수 있도록 한다. 디스플레이 장치(50)는 제1영상과 제2영상이 중첩된 융합 영상을 디스플레이할 수 있다. 디스플레이 장치(50)는 액정 디스플레이 패널(LCD), 유기 발광 디스플레이 패널(OLED), 전기 영동 디스플레이 패널(EPD) 등으로 이루어질 수 있다. 디스플레이 장치(50)는 사용자의 터치를 통하여 입력을 받을 수 있도록 터치스크린 형태로 구비되어, 사용자 입력 인터페이스로서 동작할 수 있다. The display device 50 provides the user with a fusion image output from the image fusion device 40, so that the user can monitor the displayed image. The display apparatus 50 may display a fusion image in which the first image and the second image are overlapped. The display device 50 may be formed of a liquid crystal display panel (LCD), an organic light emitting display panel (OLED), an electrophoretic display panel (EPD), and the like. The display device 50 may be provided in the form of a touch screen to receive an input through a user's touch and operate as a user input interface.
도 2는 본 발명의 일 실시예에 따른 영상 정합 장치를 개략적으로 도시한 블록도이다. 2 is a block diagram schematically illustrating an image matching device according to an embodiment of the present invention.
도 2를 참조하면, 영상 정합 장치(30)는 제1 변환함수 추정부(301), 줌 변환함수 추정부(303), 변환함수 선택부(305) 및 정합부(307)를 포함할 수 있다. Referring to FIG. 2, the image matching device 30 may include a first transform function estimator 301, a zoom transform function estimator 303, a transform function selector 305, and a matcher 307. .
제1 변환함수 추정부(301)는 제1영상센서(10)로 촬영한 제1영상 및 제2영상센서(20)로 촬영한 제2영상으로부터 추출된 특징점 정보를 기초로 제1변환함수(H1)를 추정할 수 있다. 제1 변환함수 추정부(301)는 제1영상센서(10)와 제2영상센서(20)의 줌 상태가 바뀔 때마다 새롭게 제1변환함수(H1)를 추정할 수 있다. The first transform function estimator 301 uses the first transform function based on the feature point information extracted from the first image photographed by the first image sensor 10 and the second image photographed by the second image sensor 20. H1) can be estimated. The first transform function estimator 301 may newly estimate the first transform function H1 whenever the zoom state of the first image sensor 10 and the second image sensor 20 changes.
제1 변환함수 추정부(301)는 제1영상과 제2영상 전체에 대한 특징점 검출을 수행하지 않고, 관심 영역을 설정하고, 설정된 관심 영역에 대해서만 특징점 검출을 수행함으로써 연산량을 줄여 제1변환함수(H1) 추정 시간을 단축할 수 있다. 관심 영역은 제1영상과 제2영상 간에 촬영 영역이 겹치는 영역일 수 있다. The first transform function estimator 301 sets the region of interest without performing feature point detection for the entire first image and the second image, and performs a feature point detection only on the set region of interest, thereby reducing the amount of computation and thus converting the first transform function. (H1) The estimation time can be shortened. The ROI may be a region where the photographing region overlaps between the first image and the second image.
제1 변환함수 추정부(301)는 제1영상과 제2영상으로부터 특징점을 추출하고 대응하는 특징점 쌍을 선별하는 과정을 통해 제1변환함수(H1)를 추정하기 때문에, 정합 오차율이 적으나, 줌 상태가 바뀔 때마다 새로운 제1변환함수(H1)를 추정해야 하므로 실시간 정합 수행이 어렵다. Since the first transform function estimator 301 estimates the first transform function H1 through a process of extracting feature points from the first image and the second image and selecting a corresponding pair of feature points, the matching error rate is small. Since the new first transform function H1 must be estimated each time the zoom state is changed, it is difficult to perform real-time matching.
따라서, 본 실시예에서는 제1영상센서(10)와 제2영상센서(20) 중 적어도 하나의 줌 상태가 변경된 경우, 제1 변환함수 추정부(301)가 줌 상태 변경에 따라 새롭게 제1변환함수(H1) 추정을 완료할 때까지, 보다 간단한 추정 과정을 통해 적은 연산량으로 빠르게 추정된 변환함수로 정합을 수행하여 줌 상태 변경이 있는 경우에도 실시간 정합이 가능하도록 한다. Therefore, in the present embodiment, when the zoom state of at least one of the first image sensor 10 and the second image sensor 20 is changed, the first conversion function estimator 301 newly converts the first state according to the change of the zoom state. Until the function (H1) estimation is completed, matching is performed with the transform function estimated quickly with a small amount of computation through a simpler estimation process, so that real-time matching is possible even when there is a zoom state change.
줌 변환함수 추정부(303)는 제1영상센서(10)와 제2영상센서(20)의 줌 정보를 기초로 제1변환함수(H1)를 조정한 제3변환함수(H3)를 추정할 수 있다. 제1영상센서(10)와 제2영상센서(20)가 고정된 줌 상태로 영상 촬영 중, 제1영상센서(10)와 제2영상센서(20) 중 적어도 하나의 줌 상태가 변경된 경우, 줌 변환함수 추정부(303)는 고정된 줌 상태에서 추정된 제1변환함수(H1)를 조정하여 신속하게 제3변환함수(H3)를 추정할 수 있다. 줌 변환함수 추정부(303)는 특징점 추출 과정 없이 줌 정보만으로 기 추정된 제1변환함수(H1)를 조정한 제3변환함수(H3)를 추정하기 때문에, 제1 변환함수 추정부(301)가 새로운 제1변환함수(H1)를 추정하는 동안에도 실시간 정합이 가능하다. The zoom conversion function estimator 303 estimates a third conversion function H3 that adjusts the first conversion function H1 based on the zoom information of the first image sensor 10 and the second image sensor 20. Can be. When at least one of the first image sensor 10 and the second image sensor 20 is changed while the image is captured while the first image sensor 10 and the second image sensor 20 are fixed, The zoom transform function estimator 303 may quickly estimate the third transform function H3 by adjusting the first transform function H1 estimated in the fixed zoom state. Since the zoom transform function estimator 303 estimates the third transform function H3 that adjusts the first transform function H1 estimated with only zoom information without the feature point extraction process, the first transform function estimator 301 Real-time matching is possible while E estimates the new first transform function H1.
제1영상센서(10)와 제2영상센서(20) 중 적어도 하나의 줌 상태가 변경된 경우, 예를 들어, 줌 배율 변경에 의해 줌인 또는 줌아웃된 경우, 해당 영상센서의 촬영 영역이 감소하거나 증가할 수 있다. 제1 변환함수 추정부(301)는 제1영상 및 제2영상 중 줌 상태 변경에 의해 촬영 영역이 감소한 영상을 기준영상으로 하고, 기준영상의 촬영 영역을 관심 영역으로 설정하고, 제1영상 및 제2영상 각각의 관심 영역에 대해서만 특징점을 추출하여 제1변환함수(H1)를 추정할 수 있다. 이에 따라 제1영상과 제2영상 간에 공통 영역이 아닌 영역에 대해서는 특징점을 추출할 필요가 없어 연산 시간을 줄일 수 있다. When the zoom state of at least one of the first image sensor 10 and the second image sensor 20 is changed, for example, when zooming in or out by a zoom magnification change, the photographing area of the image sensor decreases or increases. can do. The first transform function estimating unit 301 sets an image of which the photographing area is reduced by changing the zoom state among the first image and the second image as the reference image, sets the photographing region of the reference image as the region of interest, The first transform function H1 may be estimated by extracting feature points only for the ROI of each of the second images. Accordingly, it is not necessary to extract a feature point in an area that is not a common area between the first image and the second image, thereby reducing computation time.
변환함수 선택부(305)는 제1변환함수(H1)의 추정 여부에 따라, 제1변환함수(H1) 또는 제3변환함수(H3)를 최종 변환함수(H)로 선택할 수 있다. 줌 상태가 고정된 경우, 변환함수 선택부(305)는 제1 변환함수 추정부(301)가 추정한 제1변환함수(H1)를 최종 변환함수(H)로 선택한다. 줌 상태 변경이 있는 경우, 제1 변환함수 추정부(301)가 새로운 제1변환함수(H1) 추정을 완료하기 전(H1_done=0)이면, 변환함수 선택부(305)는 줌 변환함수 추정부(303)가 추정한 제3변환함수(H3)를 최종 변환함수(H)로 선택한다. 제1 변환함수 추정부(301)가 새로운 제1변환함수(H1) 추정을 완료(H1_done=1)하면, 변환함수 선택부(305)는 제1 변환함수 추정부(301)가 추정한 제1변환함수(H1)를 최종 변환함수(H)로 선택한다. 줌 상태 변경이 있는 경우, 변환함수 선택부(305)의 선택 방법은 하기 수학식 1과 같다. The transform function selector 305 may select the first transform function H1 or the third transform function H3 as the final transform function H, depending on whether the first transform function H1 is estimated. When the zoom state is fixed, the transform function selector 305 selects the first transform function H1 estimated by the first transform function estimator 301 as the final transform function H. If there is a zoom state change, and if the first transform function estimator 301 completes the estimation of the new first transform function H1 (H1_done = 0), the transform function selector 305 zooms in. The third transform function H3 estimated by 303 is selected as the final transform function H. When the first transform function estimator 301 completes the estimation of the new first transform function H1 (H1_done = 1), the transform function selector 305 calculates the first estimated by the first transform function estimator 301. The conversion function H1 is selected as the final conversion function H. When there is a change in the zoom state, the selection method of the conversion function selection unit 305 is shown in Equation 1 below.
수학식 1
Figure PCTKR2013008936-appb-M000001
 Equation 1
Figure PCTKR2013008936-appb-M000001
정합부(307)는 제1변환함수(H1) 또는 제3변환함수(H3) 중 선택된 변환함수(H)를 이용하여 제1영상 및 제2영상을 정합한다. The matching unit 307 matches the first image and the second image by using the transform function H selected from the first transform function H1 or the third transform function H3.
도 3은 본 발명의 일 실시예에 따른 제1 변환함수 추정부를 개략적으로 도시한 블록도이다. 도 4는 본 발명의 일 실시예에 따른 특징점 선별을 설명하는 예시도이다. 3 is a block diagram schematically illustrating a first transform function estimator according to an embodiment of the present invention. 4 is an exemplary view illustrating feature point selection according to an embodiment of the present invention.
도 3을 참조하면, 제1 변환함수 추정부(301)는 특징점 검출부(311), 특징점 선별부(341) 및 추정부(391)를 포함할 수 있다. Referring to FIG. 3, the first transform function estimator 301 may include a feature point detector 311, a feature point selector 341, and an estimator 391.
특징점 검출부(311)는 제1영상센서(10)로 촬영된 제1영상의 특징점(F1)을 검출하는 제1 특징점 검출부(321) 및 제2영상센서(20)로 촬영된 제2영상의 특징점(F2)을 검출하는 제2 특징점 검출부(331)를 포함할 수 있다. 제1 특징점 검출부(321) 및 제2 특징점 검출부(331)는 각각 별개로 또는 일체로 구현되어 순차로 또는 병렬로 특징점 검출을 수행할 수 있다. The feature point detector 311 may detect the feature point F1 of the first image captured by the first image sensor 10, and the feature point of the second image captured by the second image sensor 20. And a second feature point detector 331 for detecting (F2). The first feature point detector 321 and the second feature point detector 331 may be separately or integrally implemented to perform feature point detection sequentially or in parallel.
특징점 검출부(311)는 SIFT 알고리즘, HARRIS 코너 알고리즘, SUSAN 알고리즘 등을 이용하여 제1영상 및 제2영상 각각으로부터 코너(corners), 에지(edges), 외곽선(contours), 교차점(line intersections) 등을 특징점으로 추출할 수 있다. 본 발명의 실시예에서는 특징점 검출 알고리즘을 특별히 제한하지 않으며, 다양한 특징점 추출 알고리즘을 이용할 수 있다. The feature point detector 311 uses a SIFT algorithm, a HARRIS corner algorithm, a SUSAN algorithm, and the like to determine corners, edges, contours, and line intersections from the first and second images, respectively. Can be extracted as feature points. In the embodiment of the present invention, the feature point detection algorithm is not particularly limited, and various feature point extraction algorithms may be used.
특징점 선별부(341)는 제1영상 및 제2영상의 특징점들 간에 대응하는 대응 특징점들을 선별할 수 있다. 특징점 선별부(341)는 패치 영상 획득부(351), 후보 선별부(361), 유사성 판단부(371) 및 대응 특징점 선별부(381)를 포함할 수 있다. The feature point selector 341 may select corresponding feature points corresponding to feature points of the first image and the second image. The feature point selector 341 may include a patch image acquirer 351, a candidate selector 361, a similarity determiner 371, and a corresponding feature point selector 381.
패치 영상 획득부(351)는 제1영상 및 제2영상의 특징점 각각의 패치 영상을 획득할 수 있다. 패치 영상은 특징점을 중심으로 NxN 크기를 가질 수 있다. The patch image acquisition unit 351 may acquire patch images of the feature points of the first image and the second image. The patch image may have an N × N size around the feature point.
후보 선별부(361)는 제1영상 및 제2영상 중 하나를 기준영상으로 하고, 기준영상의 특징점 각각에 대해, 나머지 영상에서 대응 가능한 후보 특징점들을 선별할 수 있다. 동일 장면에 대해 획득한 두 영상의 특징점들은 국지성(localization)을 나타낸다. 후보 선별부(361)는 나머지 영상에서 기준영상의 특징점을 기준으로 소정의 사이즈를 갖는 블럭 내의 특징점들을 후보 특징점들로 선별할 수 있다. 블럭 사이즈는 두 영상센서의 시야각(FOV, Field of view) 및 주시 방향에 따라 유연하게 최적화할 수 있다. 예를 들어, 두 영상센서의 시야각과 바라보는 방향이 가까울수록 블럭 사이즈를 감소시킬수 있고, 멀어질수록 블럭 사이즈를 증가시킬 수 있다. 후보 선별부(361)는 블럭 사이즈를 기초로 기준영상의 특징점과의 거리가 임계치 내인 나머지 영상의 특징점들을 후보 특징점들로 선별할 수 있다. The candidate selector 361 may use one of the first image and the second image as a reference image, and select candidate feature points corresponding to the remaining images from each of the feature points of the reference image. The feature points of the two images acquired for the same scene indicate localization. The candidate selector 361 may select, as candidate feature points, feature points within a block having a predetermined size based on feature points of the reference image in the remaining images. The block size can be flexibly optimized according to the field of view (FOV) and viewing direction of the two image sensors. For example, the closer the viewing angle and the viewing direction of the two image sensors are, the smaller the block size can be, and the further the larger the block size can be. The candidate selector 361 may select, as candidate feature points, feature points of the remaining image having a distance from the feature point of the reference image within a threshold based on the block size.
유사성 판단부(371)는 기준영상의 특징점의 패치 영상과 나머지 영상의 후보 특징점들의 패치 영상들 간의 유사성을 판단할 수 있다. 유사성 판단은 정규상호정보(Normalized Mutual Information) 및 그래디언트(gradient) 방향 정보를 파라미터로 이용할 수 있다. 정규상호정보는 두 확률변수의 통계적 상관성을 나타내는 상호정보를 정규화한 정보이다. 정규상호정보 및 그래디언트 방향 정보의 산출 방식은 이미 공지되어 있는 알고리즘 및 방법으로 본 실시예의 상세한 설명에서는 이에 관한 자세한 설명을 생략한다. The similarity determiner 371 may determine similarity between the patch image of the feature point of the reference image and the patch images of the candidate feature points of the remaining images. The similarity determination may use normalized mutual information and gradient direction information as parameters. Normal mutual information is information that normalizes mutual information representing a statistical correlation between two random variables. The method of calculating the normal mutual information and the gradient direction information is a well-known algorithm and method, and detailed description thereof will be omitted in the detailed description of this embodiment.
대응 특징점 선별부(381)는 유사성 판단부(371)의 유사성 판단 결과를 기초로 후보 특징점들 중 대응 특징점을 선별할 수 있다. 대응 특징점 선별부(381)는 나머지 영상의 후보 특징점들 중 유사성 정도가 가장 큰 특징점을 기준영상의 특징점과 대응하는 대응 특징점으로 하여 대응 특징점 쌍을 선별할 수 있다.The corresponding feature point selector 381 may select a corresponding feature point among candidate feature points based on the similarity determination result of the similarity determiner 371. The corresponding feature point selector 381 may select a pair of corresponding feature points using the feature point having the greatest similarity among the candidate feature points of the remaining images as the corresponding feature point corresponding to the feature point of the reference image.
도 4는 제1영상(I1)과 제2영상(I2)으로부터 각각 특징점들이 검출되고, 제1영상(I1)을 기준영상으로 대응 특징점 쌍을 선별하는 예를 도시한다. 제1영상(I1)의 복수의 특징점들 중 하나인 특징점(f1)에 대해, 제2영상(I2)의 후보 특징점들(f21, f22, f23)이 선별되었다. 후보 특징점들(f21, f22, f23)은 제2영상(I2)의 특징점(f1)과 대응하는 위치에서 일정 거리 내의 영역(CFA)에 위치하는 특징점들이다. 제1영상(I1)의 특징점(f1)을 중심으로 하는 패치 영상(P1)과 제2영상(I2)의 후보 특징점들(f21, f22, f23) 각각을 중심으로 하는 패치 영상들(P21, P22, P23) 간에 유사성 정도가 판단된다. 4 illustrates an example in which feature points are respectively detected from a first image I1 and a second image I2, and a pair of corresponding feature points is selected based on the first image I1 as a reference image. The candidate feature points f21, f22, and f23 of the second image I2 are selected with respect to the feature point f1, which is one of the plurality of feature points of the first image I1. The candidate feature points f21, f22, and f23 are feature points positioned in an area CFA within a predetermined distance from a position corresponding to the feature point f1 of the second image I2. Patch images P1 centering on the feature point f1 of the first image I1 and patch images P21 and P22 centering on the candidate feature points f21, f22, and f23 of the second image I2, respectively. , P23), the degree of similarity is determined.
다시 도 3을 참조하면, 추정부(391)는 선별된 대응 특징점들을 기초로 제1변환함수(H1)를 추정할 수 있다. 추정부(391)는 RANSAC(random sample consensus) 또는 LO-RANSAC(Locally Optimized RANSAC) 알고리즘을 이용하여 제1변환함수(H1)를 추정할 수 있다. 제1변환함수(H1)는 하기 수학식 2와 같이 표현할 수 있다. Referring back to FIG. 3, the estimator 391 may estimate the first transform function H1 based on the selected corresponding feature points. The estimator 391 may estimate the first transform function H1 using a random sample consensus (RANSAC) or a locally optimized RANSAC (LO-RANSAC) algorithm. The first transform function H1 may be expressed as Equation 2 below.
수학식 2
Figure PCTKR2013008936-appb-M000002
 Equation 2
Figure PCTKR2013008936-appb-M000002
제1변환함수(H1)의 각 성분(h11 내지 h33)은 어떤 회전각으로 회전할지를 나타내는 회전(rotation) 정보, x, y, z 방향으로 얼마만큼 이동할지를 나타내는 평행이동(translation) 정보, 및 x, y, z 방향으로 얼마만큼 크기를 변화시킬지를 나타내는 크기 변환(scaling) 정보를 포함한다.Each component h11 to h33 of the first transform function H1 is rotation information indicating at which rotation angle to rotate, translation information indicating how much to move in the x, y, and z directions, and x and scaling information indicating how much to change the size in the y and z directions.
도 5는 본 발명의 일 실시예에 따른 줌 변환함수 추정부를 개략적으로 도시한 블록도이다. 5 is a block diagram schematically illustrating a zoom transform function estimating unit according to an embodiment of the present invention.
도 5를 참조하면, 줌 변환함수 추정부(303)는 제1영상센서(10)와 제2영상센서(20)의 줌 정보(Z1, Z2)를 기초로 제1변환함수(H1)를 조정한 제3변환함수(H3)를 추정할 수 있다. 줌 정보(Z1, Z2)는 줌 배율을 나타내는 파라미터일 수 있다. 줌 변환함수 추정부(303)는 스케일 결정부(313), 제2 변환함수 추정부(353) 및 제3 변환함수 추정부(373)를 포함할 수 있다.Referring to FIG. 5, the zoom conversion function estimator 303 adjusts the first conversion function H1 based on the zoom information Z1 and Z2 of the first image sensor 10 and the second image sensor 20. One third transform function H3 can be estimated. The zoom information Z1 and Z2 may be parameters representing a zoom magnification. The zoom transform function estimator 303 may include a scale determiner 313, a second transform function estimator 353, and a third transform function estimator 373.
스케일 결정부(313)는 줌 정보(Z1, Z2)에 대응하는 스케일 변환 계수(S)를 결정할 수 있다. 스케일 변환 계수(S)는 영상 크기의 변환 정도를 나타내는 계수이다. 영상 센서마다 크기와 줌 렌즈의 줌 구간별 초점 거리가 상이하므로, 영상 센서마다 줌 배율 변환에 대응하는 영상 크기 변환 비율이 상이할 수 있다. 따라서, 스케일 결정부(313)는 메모리 등에 미리 저장된 영상 센서별로 줌 배율과 스케일 변환 계수 간의 관계를 나타내는 그래프 또는 룩업 테이블을 이용하여 해당하는 영상 센서의 줌 배율에 대응하는 스케일 변환 계수(S)를 결정할 수 있다. The scale determiner 313 may determine the scale conversion factor S corresponding to the zoom information Z1 and Z2. The scale conversion coefficient S is a coefficient representing the degree of conversion of the image size. Since the size and the focal length of each zoom section of the zoom lens are different for each image sensor, the image size conversion ratio corresponding to the zoom magnification may be different for each image sensor. Accordingly, the scale determiner 313 calculates the scale conversion coefficient S corresponding to the zoom magnification of the corresponding image sensor by using a graph or a look-up table indicating the relationship between the zoom magnification and the scale conversion coefficient for each image sensor previously stored in a memory or the like. You can decide.
제2 변환함수 추정부(353)는 스케일 변환 계수(S)를 기초로 제1변환함수(H1)를 조정하여 제2변환함수(H2)를 추정할 수 있다. 제1변환함수(H1)의 h11 및 h22 성분은 제1영상 또는 제2영상 중 어느 한 영상을 기준영상으로 하여 기준영상의 좌표로 나머지 영상을 정합 시에 각각 x, y 방향에서의 크기 변환 정보를 포함한다. 따라서, 제2변환함수(H2)는 제1변환함수(H1)의 h11 및 h22 성분을 스케일 변환 계수(S)로 나누어줌으로써, 하기 수학식 3과 같이 표현될 수 있다. The second transform function estimator 353 may estimate the second transform function H2 by adjusting the first transform function H1 based on the scale transform coefficient S. The h11 and h22 components of the first transform function H1 are size conversion information in the x and y directions, respectively, when matching the remaining images with the coordinates of the reference image using either the first image or the second image as the reference image. It includes. Accordingly, the second transform function H2 may be expressed by Equation 3 by dividing the h11 and h22 components of the first transform function H1 by the scale transform coefficient S. FIG.
수학식 3
Figure PCTKR2013008936-appb-M000003
 Equation 3
Figure PCTKR2013008936-appb-M000003
제3 변환함수 추정부(373)는 제2변환함수(H2)에 의해 정합된 제1영상 및 제2영상 간의 오프셋 값(O)을 추출하고, 오프셋 값(O)을 기초로 제2변환함수(H2)를 조정하여 제3변환함수(H3)를 추정할 수 있다. 오프셋 값(O)은 제2변환함수(H2)로 정합하여 정렬된 제1영상 및 제2영상 간의 센터 틀어짐 정도이다. 제1변환함수(H1)의 h13 및 h23 성분은 제1영상 또는 제2영상 중 어느 한 영상을 기준영상으로 하여 기준영상의 좌표로 나머지 영상을 정렬 시에 각각 x, y 좌표를 이동시키는 평행이동 정보를 포함한다. 제1영상의 센터 좌표(x1, y1)와 제2영상의 센터 좌표(x2, y2)로부터 오프셋 값(O)인 (tx, ty)는 하기 수학식 4와 같이 추출될 수 있다. 따라서, 제3변환함수(H3)는 제2변환함수(H2)의 h13 및 h23 성분에 각각 오프셋 값(tx, ty)을 가산함으로써, 하기 수학식 5와 같이 표현될 수 있다. The third transform function estimator 373 extracts an offset value O between the first image and the second image matched by the second transform function H2 and based on the offset value O, the second transform function. By adjusting (H2), the third transform function (H3) can be estimated. The offset value O is the degree of center misalignment between the first image and the second image that are aligned and aligned with the second transform function H2. The h13 and h23 components of the first transform function H1 are parallel shifts that move the x and y coordinates when the remaining images are aligned with the coordinates of the reference image using either the first image or the second image as the reference image. Contains information. An offset value O of (tx, ty) from the center coordinates x1 and y1 of the first image and the center coordinates x2 and y2 of the second image may be extracted as in Equation 4 below. Accordingly, the third transform function H3 may be expressed as shown in Equation 5 by adding offset values tx and ty to the h13 and h23 components of the second transform function H2, respectively.
수학식 4
Figure PCTKR2013008936-appb-M000004
 Equation 4
Figure PCTKR2013008936-appb-M000004
수학식 5
Figure PCTKR2013008936-appb-M000005
 Equation 5
Figure PCTKR2013008936-appb-M000005
도 6 및 도 7은 본 발명의 일 실시예에 따른 정합을 예시적으로 나타낸 도면이다. 6 and 7 exemplarily show matching according to an embodiment of the present invention.
도 6은 1배줌의 제1영상센서(10)로 촬영된 제1영상(a)과 1배줌에서 2배줌으로 줌 상태가 변경된 제2영상센서(20)로 촬영된 제2영상(b)을 제2변환함수(H2)로 정합한 경우 정렬된 결과 영상(c)을 도시한 예이다. 도 6을 참조하면, 제2변환함수(H2)는 영상 크기 변환 정보만을 포함하고 있으므로, 기준영상인 제1영상(a)의 센터(x1, y1)와 크기 변환된 제2영상(b')의 센터(x2, y2)가 어긋나 있다. 6 illustrates a first image (a) photographed by the first image sensor 10 of 1x zoom and a second image b photographed by the second image sensor 20 whose zoom state is changed from 1x zoom to 2x zoom. In the case of matching with the second transform function H2, the aligned result image c is illustrated. Referring to FIG. 6, since the second transform function H2 includes only image size conversion information, the centers x1 and y1 of the first image a as the reference image and the second image b 'that are size-converted are used. Of centers (x2, y2) are shifted.
도 7은 1배줌의 제1영상센서(10)로 촬영된 제1영상(a)과 1배줌에서 2배줌으로 줌 상태가 변경된 제2영상센서(20)로 촬영된 제2영상(b)을 제3변환함수(H3)로 정합한 경우 정렬된 결과 영상(d)을 도시한 예이다. 도 7을 참조하면, 제3변환함수(H3)는 영상 크기 변환 정보 및 평행 이동 정보를 포함하고 있으므로, 기준영상인 제1영상(a)의 센터와 크기 변환된 제2영상(b')의 센터가 일치되어 있다. 7 illustrates a first image (a) photographed by the first image sensor 10 of 1x zoom and a second image b photographed by the second image sensor 20 whose zoom state is changed from 1x zoom to 2x zoom. In the case of matching with the third transform function H3, the aligned result image d is illustrated. Referring to FIG. 7, since the third conversion function H3 includes image size conversion information and parallel movement information, the center of the first image a, which is the reference image, and the size of the second image b 'that has been size converted. The centers are aligned.
도 8은 본 발명의 일 실시예에 따른 영상 정합 방법을 설명하는 흐름도이다. 도 9는 도 8의 대응 특징점을 선별하는 방법을 설명하는 흐름도이다. 8 is a flowchart illustrating an image registration method according to an embodiment of the present invention. FIG. 9 is a flowchart for explaining a method of selecting corresponding feature points of FIG. 8.
도 8을 참조하면, 본 발명의 일 실시예에 따른 영상 정합 장치는 제1영상센서로 촬영한 제1영상 및 제2영상센서로 촬영한 제2영상으로부터 추출된 특징점 정보를 기초로 제1변환함수(H1)를 추정할 수 있다(S80A). Referring to FIG. 8, in the image matching device according to an embodiment of the present invention, a first conversion is performed based on feature point information extracted from a first image photographed by a first image sensor and a second image photographed by a second image sensor. The function H1 can be estimated (S80A).
구체적으로, 영상 정합 장치는 제1영상 및 제2영상의 특징점들(F1, F2)을 검출할 수 있다(S81). 특징점은 코너(corners), 에지(edges), 외곽선(contours), 교차점(line intersections) 등을 포함할 수 있다. In detail, the image matching device may detect feature points F1 and F2 of the first image and the second image (S81). Feature points may include corners, edges, contours, line intersections, and the like.
다음으로, 영상 정합 장치는 검출된 제1영상 및 제2영상의 특징점들(F1, F2) 간에 대응하는 대응 특징점들을 선별할 수 있다(S82). 이를 위해, 영상 정합 장치는 제1영상 및 제2영상 각각의 특징점(F1, F2)을 중심으로 하는 패치 영상을 획득할 수 있다(S821). 그리고, 영상 정합 장치는 제1영상 및 제2영상 중 하나인 기준영상의 특징점에 대응 가능한 나머지 영상의 후보 특징점들을 선별할 수 있다(S823). 예를 들어, 제1영상을 기준영상으로 한 경우, 제1영상의 특징점에 대응 가능한 제2영상의 후보 특징점들을 선별할 수 있다. 후보 특징점들은 국지성, 예를 들어, 특징점 간의 거리를 기초로 선별될 수 있다. 그리고, 영상 정합 장치는 기준영상의 특징점의 패치 영상과 나머지 영상의 후보 특징점들의 패치 영상들 간의 유사성을 판단할 수 있다(S825). 유사성 정도는 패치 영상 간의 정규상호정보 및 그래디언트 방향 정보를 이용하여 결정될 수 있다. 영상 정합 장치는 유사성 판단 결과를 기초로, 후보 특징점들 중에서 기준영상의 특징점에 대응하는 대응 특징점을 선별할 수 있다(S827). 예를 들어, 영상 정합 장치는 기준영상 특징점과 유사성 정도가 가장 큰 후보 특징점을 기준영상 특징점의 대응 특징점으로 선별할 수 있다. Next, the image matching apparatus may select corresponding feature points between the detected feature points F1 and F2 of the first image and the second image (S82). To this end, the image registration device may acquire a patch image centering on the feature points F1 and F2 of each of the first and second images (S821). In operation S823, the image matching apparatus may select candidate feature points of the remaining images that may correspond to the feature points of the reference image which is one of the first image and the second image. For example, when the first image is a reference image, candidate feature points of the second image that can correspond to the feature points of the first image may be selected. Candidate feature points may be selected based on locality, eg, distance between feature points. In operation S825, the image matching apparatus may determine the similarity between the patch image of the feature point of the reference image and the patch images of the candidate feature points of the remaining image. The degree of similarity may be determined using normal mutual information and gradient direction information between patch images. The image matching apparatus may select a corresponding feature point corresponding to the feature point of the reference image from among the candidate feature points based on the similarity determination result (S827). For example, the image matching apparatus may select candidate feature points having the greatest similarity with the reference image feature points as corresponding feature points of the reference image feature points.
다음으로, 영상 정합 장치는 선별된 대응 특징점들을 기초로 제1변환함수(H1)를 추정할 수 있다(S83).Next, the image matching apparatus may estimate the first transform function H1 based on the selected corresponding feature points (S83).
영상 정합 장치는 제1변환함수(H1)가 추정되면, 제1변환함수(H1)를 최종 변환함수(H)로 선택할 수 있다(S87).When the first transform function H1 is estimated, the image matching device may select the first transform function H1 as the final transform function H (S87).
영상 정합 장치는 최종 변환함수(H)로 제1영상 및 제2영상을 정합할 수 있다(S88).The image matching device may match the first image and the second image with the final transform function H (S88).
한편, 제1영상센서와 제2영상센서 중 적어도 하나의 줌 상태가 변경된 경우, 영상 정합 장치는 제1영상센서와 제2영상센서의 줌 정보(Z1, Z2)를 기초로, 줌 상태 변경 전에 추정된 제1변환함수(H1)를 조정하여 제3변환함수(H3)를 추정할 수 있다(80B). On the other hand, when the zoom state of at least one of the first image sensor and the second image sensor is changed, the image matching device is based on the zoom information (Z1, Z2) of the first image sensor and the second image sensor, before changing the zoom state. The third transform function H3 may be estimated by adjusting the estimated first transform function H1 (80B).
구체적으로, 영상 정합 장치는 제1영상센서와 제2영상센서 중 적어도 하나의 줌 상태(예를 들어, 줌 배율) 변경이 있는 경우, 줌 상태 변경에 대응하는 스케일 변환 계수(S)를 결정할 수 있다(S84). 영상 정합 장치는 영상 센서별로 줌 배율과 스케일 변환 계수 간의 관계를 나타내는 그래프 또는 룩업 테이블을 미리 저장하고, 이를 이용하여 해당하는 영상 센서의 줌 배율에 대응하는 스케일 변환 계수(S)를 결정할 수 있다. In detail, when there is a change in the zoom state (eg, the zoom ratio) of at least one of the first image sensor and the second image sensor, the image matching device may determine the scale conversion factor S corresponding to the change of the zoom state. There is (S84). The image matching device may previously store a graph or a look-up table indicating a relationship between the zoom magnification and the scale conversion coefficient for each image sensor, and determine the scale conversion factor S corresponding to the zoom magnification of the corresponding image sensor by using the same.
다음으로, 영상 정합 장치는 스케일 변환 계수(S)를 기초로 제1변환함수(H1)를 조정하여 제2변환함수(H2)를 추정할 수 있다(S85). 영상 정합 장치는 제1변환함수(H1)의 성분 중 x, y 방향에서의 크기 변환 정보를 포함하는 성분에 스케일 변환 계수(S)를 적용하여 제2변환함수(H2)를 추정할 수 있다.Next, the image matching device may estimate the second transform function H2 by adjusting the first transform function H1 based on the scale transform coefficient S (S85). The image matching device may estimate the second transform function H2 by applying the scale transform coefficient S to a component including the size transform information in the x and y directions among the components of the first transform function H1.
그리고, 영상 정합 장치는 제2변환함수(H2)로 정합되어 정렬된 제1영상 및 제2영상 간의 오프셋 값을 기초로 제2변환함수(H2)를 조정하여 제3변환함수(H3)를 추정할 수 있다(S86). 영상 정합 장치는 제1변환함수(H1)의 성분 중 x, y 방향에서의 평행이동 정보를 포함하는 성분에 오프셋 값을 적용하여, 제2변환함수(H2)로부터 제3변환함수(H3)를 추정할 수 있다.The image matching device estimates the third transform function H3 by adjusting the second transform function H2 based on the offset values between the first and second images matched and aligned with the second transform function H2. Can be (S86). The image matching device applies an offset value to a component including parallel movement information in the x and y directions among the components of the first transform function H1, thereby converting the third transform function H3 from the second transform function H2. It can be estimated.
영상 정합 장치는 줌 상태 변경이 있는 경우, 새로운 제1변환함수(H1)가 추정될 때까지, 제3변환함수(H3)를 최종 변환함수(H)로 선택할 수 있다(S87). 영상 정합 장치는 줌 상태 변경에 따라 새로운 제1변환함수(H1)가 추정되면, 새로 추정된 제1변환함수(H1)를 최종 변환함수(H)로 선택할 수 있다.When there is a change in the zoom state, the image matching device may select the third transform function H3 as the final transform function H until a new first transform function H1 is estimated (S87). When the new first transform function H1 is estimated according to the zoom state change, the image matching device may select the newly estimated first transform function H1 as the final transform function H.
그리고, 영상 정합 장치는 최종 변환함수(H)로 제1영상 및 제2영상을 정합할 수 있다(S88).The image matching device may match the first image and the second image with the final transform function H (S88).
한편, 영상 정합 장치는 줌 상태 변경에 따라 새로운 제1변환함수(H1)를 추정할 때 제1영상 및 제2영상 전체가 아닌 관심 영역에 대해서 특징점 검출을 통해 제1변환함수(H1)를 추정할 수 있다. 관심 영역은 제1영상 및 제2영상 간에 촬영 영역이 겹치는 영역일 수 있다. 이로써 영상 정합 장치는 변환함수 추정 연산량 및 연산 시간을 줄일 수 있다. Meanwhile, when the image matching device estimates the new first transform function H1 according to the change of the zoom state, the image matcher estimates the first transform function H1 through the detection of the feature points on the ROI rather than the entire first image and the second image. can do. The ROI may be a region where the photographing region overlaps between the first image and the second image. As a result, the image matching device may reduce the calculation function estimation amount and the calculation time.
전술된 실시예에서는 제1영상을 가시 영상, 제2영상을 열 영상으로 예를 들어 설명하였지만, 본 발명의 실시예는 이에 한정되지 않고, 제1영상과 제2영상은 서로 다른 시점, 서로 다른 시간 또는 가시광 카메라와 적외광 카메라 외의 서로 다른 특성의 센서로부터 획득한 영상인 경우에도 본 발명의 실시예가 동일하게 적용될 수 있다. In the above-described embodiment, the first image is described as a visible image and the second image as an example of a thermal image. However, embodiments of the present invention are not limited thereto, and the first image and the second image are different from each other. Embodiments of the present invention may be equally applicable to images obtained from sensors having different characteristics other than a time or visible light camera and an infrared light camera.
본 발명에 따른 영상 정합 방법은 컴퓨터로 읽을 수 있는 기록매체에 컴퓨터가 읽을 수 있는 코드로서 구현하는 것이 가능하다. 컴퓨터가 읽을 수 있는 기록매체는 컴퓨터 시스템에 의해 읽혀질 수 있는 데이터가 저장되는 모든 종류의 기록장치를 포함한다. 컴퓨터가 읽을 수 있는 기록매체의 예로는 ROM, RAM, CD-ROM, 자기 테이프, 플로피 디스크, 광데이터 저장장치 등이 있다. 또한, 컴퓨터가 읽을 수 있는 기록매체는 네트워크로 연결된 컴퓨터 시스템에 분산되어, 분산방식으로 컴퓨터가 읽을 수 있는 코드가 저장되고 실행될 수 있다. 그리고, 본 발명을 구현하기 위한 기능적인(functional) 프로그램, 코드 및 코드 세그먼트들은 본 발명이 속하는 기술분야의 프로그래머들에 의해 용이하게 추론될 수 있다.The image matching method according to the present invention can be embodied as computer readable codes on a computer readable recording medium. Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the art to which the present invention belongs.
본 발명은 첨부된 도면에 도시된 일 실시예를 참고로 설명되었으나, 이는 예시적인 것에 불과하며, 당해 기술분야에서 통상의 지식을 가진 자라면 이로부터 다양한 변형 및 균등한 타 실시예가 가능하다는 점을 이해할 수 있을 것이다. Although the present invention has been described with reference to one embodiment shown in the accompanying drawings, it is merely an example, and those skilled in the art may realize various modifications and equivalent other embodiments therefrom. I can understand.
전술된 실시예는 GOP와 같은 경계 지역 감시, 산불 감시와 같이 24시간 실시간 감시가 필요한 감시, 무광원 혹은 저조도 환경에서 빌딩 및 주거지 침입 감지, 산과 같은 곳에서 실종자 및 범죄자 추적, 의료 영상분야 등에 적용될 수 있다. The above-described embodiments are applicable to boundary area surveillance such as GOP, surveillance requiring 24-hour real-time monitoring such as forest fire monitoring, detection of building and residential intrusion in a no-light or low light environment, tracking of missing and criminals in places such as mountains, medical imaging, etc. Can be.

Claims (20)

  1. 제1영상센서로 촬영한 제1영상 및 제2영상센서로 촬영한 제2영상으로부터 추출된 특징점 정보를 기초로 제1변환함수를 추정하는 제1 변환함수 추정부; 및A first transform function estimator estimating a first transform function based on feature point information extracted from the first image photographed by the first image sensor and the second image photographed by the second image sensor; And
    상기 제1영상센서와 제2영상센서의 줌 정보를 기초로 상기 제1변환함수를 조정한 제3변환함수를 추정하는 줌 변환함수 추정부;를 포함하는 영상 정합 장치.And a zoom transform function estimator estimating a third transform function of adjusting the first transform function based on zoom information of the first image sensor and the second image sensor.
  2. 제1항에 있어서, 상기 줌 변환함수 추정부는, The method of claim 1, wherein the zoom conversion function estimator,
    상기 제1영상센서와 제2영상센서 중 적어도 하나의 줌 상태가 변경된 경우 상기 제3변환함수를 추정하는 영상 정합 장치. And an image matching device estimating the third transform function when the zoom state of at least one of the first image sensor and the second image sensor is changed.
  3. 제1항에 있어서, 상기 제1 변환함수 추정부는, The method of claim 1, wherein the first transform function estimator,
    상기 제1영상 및 제2영상에서 관심영역을 설정하고, 상기 설정된 관심영역으로부터 추출된 특징점 정보를 기초로 상기 제1변환함수를 추정하는 영상 정합 장치. And an ROI set in the first image and the second image, and estimating the first transform function based on feature point information extracted from the set ROI.
  4. 제2항에 있어서,The method of claim 2,
    상기 제1변환함수의 추정 여부에 따라, 상기 제1변환함수 또는 제3변환함수를 최종 변환함수로 선택하는 변환함수 선택부;를 더 포함하는 영상 정합 장치.And a transform function selector for selecting the first transform function or the third transform function as a final transform function according to whether the first transform function is estimated or not.
  5. 제4항에 있어서, 상기 변환함수 선택부는,The method of claim 4, wherein the conversion function selection unit,
    상기 제1영상센서와 제2영상센서 중 적어도 하나의 줌 상태가 변경된 경우, 새로운 제1변환함수가 추정될 때까지, 상기 제3변환함수를 최종 변환함수로 선택하는 영상 정합 장치.And when the zoom state of at least one of the first image sensor and the second image sensor is changed, selecting the third transform function as a final transform function until a new first transform function is estimated.
  6. 제1항에 있어서, 상기 제1 변환함수 추정부는,The method of claim 1, wherein the first transform function estimator,
    상기 제1영상 및 제2영상의 특징점들을 검출하는 특징점 검출부;A feature point detector for detecting feature points of the first and second images;
    상기 검출된 제1영상 및 제2영상의 특징점들 간에 대응하는 대응 특징점들을 선별하는 특징점 선별부; 및A feature point selector for selecting corresponding feature points between the detected feature points of the first image and the second image; And
    상기 선별된 대응 특징점들을 기초로 상기 제1변환함수를 추정하는 제1추정부;를 포함하는 영상 정합 장치.And a first estimator estimating the first transform function based on the selected corresponding feature points.
  7. 제6항에 있어서, 상기 특징점 선별부는,The method of claim 6, wherein the feature point selection unit,
    상기 제1영상 및 제2영상의 특징점을 중심으로 하는 패치 영상을 획득하는 패치영상 획득부;A patch image obtaining unit obtaining a patch image centering on feature points of the first image and the second image;
    상기 제1영상 및 제2영상 중 기준영상의 각 특징점에 대해, 나머지 영상에서 대응 가능한 후보 특징점들을 선별하는 후보 선별부;A candidate selector that selects candidate feature points corresponding to the remaining images from each feature point of the reference image among the first image and the second image;
    상기 기준영상의 특징점의 패치 영상과 상기 나머지 영상의 후보 특징점들의 패치 영상들 간에 유사성을 판단하는 유사성 판단부; 및A similarity determination unit that determines similarity between patch images of feature points of the reference image and patch images of candidate feature points of the remaining images; And
    상기 유사성 판단 결과를 기초로, 상기 후보 특징점들 중 기준영상의 특징점에 대응하는 대응 특징점을 선별하는 대응 특징점 선별부;를 포함하는 영상 정합 장치.And a corresponding feature point selector for selecting a corresponding feature point corresponding to the feature point of the reference image among the candidate feature points based on the similarity determination result.
  8. 제1항에 있어서, 상기 줌 변환함수 추정부는,The method of claim 1, wherein the zoom conversion function estimator,
    상기 줌 정보에 대응하는 스케일 변환 계수를 결정하는 스케일 결정부;A scale determination unit that determines a scale conversion coefficient corresponding to the zoom information;
    상기 스케일 변환 계수를 기초로 상기 제1변환함수를 조정하여 제2변환함수를 추정하는 제2 변환함수 추정부; 및A second transform function estimator for estimating a second transform function by adjusting the first transform function based on the scale transform coefficients; And
    상기 제2변환함수에 의해 정합된 상기 제1영상 및 제2영상 간의 센터 오프셋 값을 기초로 상기 제2변환함수로부터 상기 제3변환함수를 추정하는 제3 변환함수 추정부;를 포함하는 영상 정합 장치.And a third transform function estimator configured to estimate the third transform function from the second transform function based on the center offset value between the first image and the second image matched by the second transform function. Device.
  9. 제8항에 있어서, 상기 스케일 결정부는,The method of claim 8, wherein the scale determination unit,
    기 저장된 영상센서별 줌 정보와 스케일 변환 계수 간의 관계로부터 상기 스케일 변환 계수를 결정하는 영상 정합 장치. And an image matching device configured to determine the scale conversion coefficient from a relationship between previously stored zoom information of each image sensor and a scale conversion coefficient.
  10. 제4항에 있어서, The method of claim 4, wherein
    상기 선택된 제1변환함수 또는 제3변환함수를 이용하여 상기 제1영상 및 제2영상을 정합하는 정합부;를 더 포함하는 영상 정합 장치. And a matching unit which matches the first image and the second image using the selected first transform function or the third transform function.
  11. 제1영상센서로 촬영한 제1영상 및 제2영상센서로 촬영한 제2영상으로부터 추출된 특징점 정보를 기초로 제1변환함수를 추정하는 단계; 및Estimating a first transform function based on feature point information extracted from the first image photographed by the first image sensor and the second image photographed by the second image sensor; And
    상기 제1영상센서와 제2영상센서의 줌 정보를 기초로 상기 제1변환함수를 조정하여 제3변환함수를 추정하는 단계;를 포함하는 영상 정합 방법.And estimating a third transform function by adjusting the first transform function based on the zoom information of the first image sensor and the second image sensor.
  12. 제11항에 있어서, 상기 제3변환함수 추정 단계는,The method of claim 11, wherein the estimating of the third transform function comprises:
    상기 제1영상센서와 제2영상센서 중 적어도 하나의 줌 상태가 변경된 경우 상기 제3변환함수를 추정하는 단계;를 포함하는 영상 정합 장치. And estimating the third transform function when the zoom state of at least one of the first image sensor and the second image sensor is changed.
  13. 제11항에 있어서, 상기 제1 변환함수 추정 단계는,The method of claim 11, wherein the first transform function estimating step,
    상기 제1영상 및 제2영상에서 관심영역을 설정하고, 상기 설정된 관심영역으로부터 추출된 특징점 정보를 기초로 상기 제1변환함수를 추정하는 단계;를 포함하는 영상 정합 방법.And setting a region of interest in the first image and the second image, and estimating the first transform function based on the feature point information extracted from the set region of interest.
  14. 제13항에 있어서, The method of claim 13,
    상기 제1변환함수의 추정 여부에 따라, 상기 제1변환함수 또는 제3변환함수를 최종 변환함수로 선택하는 단계;를 더 포함하는 영상 정합 방법.And selecting the first transform function or the third transform function as a final transform function according to whether the first transform function is estimated.
  15. 제14항에 있어서, 상기 최종 변환함수 선택 단계는,The method of claim 14, wherein the final transform function selection step,
    상기 제1영상센서와 제2영상센서 중 적어도 하나의 줌 상태가 변경된 경우, 새로운 제1변환함수가 추정될 때까지, 상기 제3변환함수를 최종 변환함수로 선택하는 단계;를 포함하는 영상 정합 방법.And when the zoom state of at least one of the first image sensor and the second image sensor is changed, selecting the third transform function as a final transform function until a new first transform function is estimated. Way.
  16. 제11항에 있어서, 상기 제1변환함수 추정 단계는,The method of claim 11, wherein the estimating of the first transform function comprises:
    상기 제1영상 및 제2영상의 특징점들을 검출하는 단계;Detecting feature points of the first image and the second image;
    상기 검출된 제1영상 및 제2영상의 특징점들 간에 대응하는 대응 특징점들을 선별하는 단계; 및Selecting corresponding feature points corresponding to the feature points of the detected first and second images; And
    상기 선별된 대응 특징점들을 기초로 상기 제1변환함수를 추정하는 단계;를 포함하는 영상 정합 방법.And estimating the first transform function based on the selected corresponding feature points.
  17. 제16항에 있어서, 상기 특징점 선별 단계는,The method of claim 16, wherein the feature point selection step,
    상기 제1영상 및 제2영상의 특징점을 중심으로 하는 패치 영상을 획득하는 단계;Acquiring a patch image centering on feature points of the first image and the second image;
    상기 제1영상 및 제2영상 중 기준영상의 각 특징점에 대해, 나머지 영상에서 대응 가능한 후보 특징점들을 선별하는 단계;Selecting candidate feature points corresponding to other feature points of the reference image among the first image and the second image;
    상기 기준영상의 특징점의 패치 영상과 상기 나머지 영상의 후보 특징점들의 패치 영상들 간에 유사성을 판단하는 단계; 및Determining similarity between patch images of feature points of the reference image and patch images of candidate feature points of the remaining images; And
    상기 유사성 판단 결과를 기초로, 상기 후보 특징점들 중 기준영상의 특징점에 대응하는 대응 특징점을 선별하는 단계;를 포함하는 영상 정합 방법.Selecting a corresponding feature point corresponding to the feature point of the reference image among the candidate feature points based on the similarity determination result.
  18. 제11항에 있어서, 상기 제3변환함수 추정 단계는,The method of claim 11, wherein the estimating of the third transform function comprises:
    상기 줌 정보에 대응하는 스케일 변환 계수를 결정하는 단계;Determining a scale conversion coefficient corresponding to the zoom information;
    상기 스케일 변환 계수를 기초로 상기 제1변환함수를 조정하여 제2변환함수를 추정하는 단계; 및Estimating a second transform function by adjusting the first transform function based on the scale transform coefficients; And
    상기 제2변환함수에 의해 정합된 상기 제1영상 및 제2영상 간의 센터 오프셋 값을 기초로 상기 제2변환함수로부터 상기 제3변환함수를 추정하는 단계;를 포함하는 영상 정합 방법.Estimating the third transform function from the second transform function based on a center offset value between the first image and the second image matched by the second transform function.
  19. 제18항에 있어서, 스케일 변환 계수 결정 단계는,The method of claim 18, wherein the step of determining the scale conversion coefficient,
    기 저장된 영상센서별 줌 정보와 스케일 변환 계수 간의 관계로부터 상기 스케일 변환 계수를 결정하는 단계;를 포함하는 영상 정합 방법. And determining the scale conversion coefficient from the relationship between the previously stored zoom information for each image sensor and the scale conversion coefficient.
  20. 제14항에 있어서, The method of claim 14,
    상기 선택된 제1변환함수 또는 제3변환함수를 이용하여 상기 제1영상 및 제2영상을 정합하는 단계;를 더 포함하는 영상 정합 방법. And matching the first image and the second image using the selected first transform function or third transform function.
PCT/KR2013/008936 2013-08-20 2013-10-08 Image matching apparatus and image matching method using same WO2015026002A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020130098608A KR102050418B1 (en) 2013-08-20 2013-08-20 Apparatus and method for alignment of images
KR10-2013-0098608 2013-08-20

Publications (1)

Publication Number Publication Date
WO2015026002A1 true WO2015026002A1 (en) 2015-02-26

Family

ID=52483768

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2013/008936 WO2015026002A1 (en) 2013-08-20 2013-10-08 Image matching apparatus and image matching method using same

Country Status (2)

Country Link
KR (1) KR102050418B1 (en)
WO (1) WO2015026002A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110246100A (en) * 2019-06-11 2019-09-17 山东师范大学 A kind of image repair method and system based on angle perception Block- matching

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101885562B1 (en) * 2017-12-18 2018-08-06 주식회사 뷰노 Method for mapping region of interest in first medical image onto second medical image and apparatus using the same
KR102667740B1 (en) * 2018-02-12 2024-05-22 삼성전자주식회사 Device and method for matching image

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20100138202A (en) * 2009-06-24 2010-12-31 전자부품연구원 System and method of tracking object using different kind camera
KR20110116777A (en) * 2010-04-20 2011-10-26 국방과학연구소 Apparatus for fusing a visible and an infrared image signal, and method thereof
KR101109695B1 (en) * 2010-10-20 2012-01-31 주식회사 아이닉스 Apparatus and method for control speed auto focus
JP2012165333A (en) * 2011-02-09 2012-08-30 Sony Corp Imaging apparatus, imaging apparatus control method, and program
KR101279484B1 (en) * 2012-04-18 2013-06-27 한국항공우주연구원 Apparatus and method for processing image

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20100138202A (en) * 2009-06-24 2010-12-31 전자부품연구원 System and method of tracking object using different kind camera
KR20110116777A (en) * 2010-04-20 2011-10-26 국방과학연구소 Apparatus for fusing a visible and an infrared image signal, and method thereof
KR101109695B1 (en) * 2010-10-20 2012-01-31 주식회사 아이닉스 Apparatus and method for control speed auto focus
JP2012165333A (en) * 2011-02-09 2012-08-30 Sony Corp Imaging apparatus, imaging apparatus control method, and program
KR101279484B1 (en) * 2012-04-18 2013-06-27 한국항공우주연구원 Apparatus and method for processing image

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110246100A (en) * 2019-06-11 2019-09-17 山东师范大学 A kind of image repair method and system based on angle perception Block- matching
CN110246100B (en) * 2019-06-11 2021-06-25 山东师范大学 Image restoration method and system based on angle sensing block matching

Also Published As

Publication number Publication date
KR102050418B1 (en) 2019-12-02
KR20150021352A (en) 2015-03-02

Similar Documents

Publication Publication Date Title
KR102144394B1 (en) Apparatus and method for alignment of images
CN109691989B (en) Human body temperature measuring method based on infrared human face detection technology
WO2018128355A1 (en) Robot and electronic device for performing hand-eye calibration
US7554575B2 (en) Fast imaging system calibration
WO2012124852A1 (en) Stereo camera device capable of tracking path of object in monitored area, and monitoring system and method using same
WO2019164379A1 (en) Method and system for facial recognition
WO2016171341A1 (en) Cloud-based pathology analysis system and method
WO2013151270A1 (en) Apparatus and method for reconstructing high density three-dimensional image
EP4004878A1 (en) Electronic apparatus and method for controlling thereof
WO2018135906A1 (en) Camera and image processing method of camera
JP2001324305A (en) Image correspondent position detector and range finder equipped with the same
WO2016072625A1 (en) Vehicle location checking system for a parking lot using imaging technique, and method of controlling same
JP2016187162A (en) Information processing device, information processing method, and program
JP2019159739A (en) Image processing device, image processing method, and program
WO2015026002A1 (en) Image matching apparatus and image matching method using same
KR20130043305A (en) Position estimation apparatus of mobile robot using indoor ceiling image
JP2010217984A (en) Image detector and image detection method
CN111696143A (en) Event data registration method and system
WO2023149603A1 (en) Thermal-image-monitoring system using plurality of cameras
WO2019194561A1 (en) Location recognition method and system for providing augmented reality in mobile terminal
JP2002027449A (en) Method and apparatus for identifying moving object
WO2019083068A1 (en) Three-dimensional information acquisition system using pitching practice, and method for calculating camera parameters
WO2012074174A1 (en) Augmented reality implementation system using original identification information
KR20220115223A (en) Method and apparatus for multi-camera calibration
KR20220114820A (en) System and method for removing camera movement in video

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13891942

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 13891942

Country of ref document: EP

Kind code of ref document: A1