CN112085771A - Image registration method and device, terminal equipment and computer readable storage medium - Google Patents

Image registration method and device, terminal equipment and computer readable storage medium Download PDF

Info

Publication number
CN112085771A
CN112085771A CN202010784953.7A CN202010784953A CN112085771A CN 112085771 A CN112085771 A CN 112085771A CN 202010784953 A CN202010784953 A CN 202010784953A CN 112085771 A CN112085771 A CN 112085771A
Authority
CN
China
Prior art keywords
target
shooting
distance
image
thermal infrared
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010784953.7A
Other languages
Chinese (zh)
Other versions
CN112085771B (en
Inventor
张惊涛
程骏
胡淑萍
顾在旺
王东
郭渺辰
庞建新
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ubtech Robotics Corp
Original Assignee
Ubtech Robotics Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ubtech Robotics Corp filed Critical Ubtech Robotics Corp
Priority to CN202010784953.7A priority Critical patent/CN112085771B/en
Publication of CN112085771A publication Critical patent/CN112085771A/en
Application granted granted Critical
Publication of CN112085771B publication Critical patent/CN112085771B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J5/00—Radiation pyrometry, e.g. infrared or optical thermometry
    • G01J5/0022—Radiation pyrometry, e.g. infrared or optical thermometry for sensing the radiation of moving bodies
    • G01J5/0025—Living bodies
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G06T2207/30208—Marker matrix
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The application is applicable to the technical field of image processing, and provides an image registration method, an image registration device, a terminal device and a computer-readable storage medium, wherein the image registration method comprises the following steps: acquiring a target RGB image and a target thermal infrared image including a first shooting target through a shooting device; calculating a first area of a detection frame of the first shooting target in the target RGB image, and calculating a target shooting distance between the first shooting target and the shooting device according to the first area; acquiring a mapping matrix corresponding to the target shooting distance; and mapping a first pixel point in the target RGB image to the target thermal infrared image according to the mapping matrix to obtain a second pixel point corresponding to the first pixel point in the target thermal infrared image. By the method, the time for image registration can be saved, the robustness of the image registration method is improved, and the accuracy of the image registration result is improved.

Description

Image registration method and device, terminal equipment and computer readable storage medium
Technical Field
The present application belongs to the field of image processing technologies, and in particular, to an image registration method, an image registration apparatus, a terminal device, and a computer-readable storage medium.
Background
In recent years, the diseases are easily infected by influenza viruses, and particularly, the treatment, prevention and control difficulties are high in public places with dense population and high mobility. The main pathogenesis characteristic of the influenza virus is the rise of body temperature, so that the rapid and accurate measurement of the body temperature of a human body in a densely mobile population is an important means for prevention and control. At present, people are mainly subjected to rapid temperature measurement through a non-contact temperature measuring device, namely, an RGB image and a thermal infrared image of a human body are obtained through the non-contact temperature measuring device, and then the RGB image and the thermal infrared image are subjected to information fusion to obtain the body temperature of the human body.
In information fusion of an RGB image and a thermal infrared image, a common method is to perform image registration on the RGB image and the thermal infrared image. The existing image registration method is usually a feature-based registration algorithm, that is, features (such as SIFT features or ORB features) are respectively extracted from an RGB image and a thermal infrared image, then matching points in the two images are calculated according to the extracted features, and finally a mapping matrix between the two images is calculated through the matching points. However, the mapping matrix calculated by the existing image registration method generally corresponds to only one shooting distance, and when the shooting distance between the shooting device and the shooting target changes, the mapping matrix needs to be recalculated. Therefore, the existing image registration method is long in time consumption and poor in robustness.
Disclosure of Invention
The embodiment of the application provides an image registration method, an image registration device, terminal equipment and a computer readable storage medium, and can solve the problems of long time consumption and poor robustness of the existing image registration method.
In a first aspect, an embodiment of the present application provides an image registration method, including:
acquiring a target RGB image and a target thermal infrared image including a first shooting target through a shooting device;
calculating a first area of a detection frame of the first shooting target in the target RGB image, and calculating a target shooting distance between the first shooting target and the shooting device according to the first area;
acquiring a mapping matrix corresponding to the target shooting distance, wherein the mapping matrix is used for representing a coordinate mapping relation between the target RGB image and the target thermal infrared image when the first shooting target is spaced from the shooting device by the target shooting distance;
and mapping a first pixel point in the target RGB image to the target thermal infrared image according to the mapping matrix to obtain a second pixel point corresponding to the first pixel point in the target thermal infrared image.
In a possible implementation manner of the first aspect, the calculating a target shooting distance between the first shooting target and the shooting device according to the first region area includes:
acquiring a proportionality coefficient, wherein the proportionality coefficient is used for representing the proportionality relation between the first region area and the target shooting distance;
and calculating the target shooting distance according to the scale coefficient and the first region area.
In a possible implementation manner of the first aspect, the obtaining the scaling factor includes:
when the shooting device is spaced from a second shooting target by a first target distance, obtaining a first sample RGB image including the second shooting target through the shooting device, wherein the first target distance is any shooting distance in a shooting range of the shooting device;
and acquiring a second area of the detection frame of the second shooting target in the first sample RGB image, and calculating the proportionality coefficient according to the second area and the first target distance.
In a possible implementation manner of the first aspect, the obtaining a mapping matrix corresponding to the target shooting distance includes:
acquiring an affine transformation model of the shooting device, wherein the affine transformation model comprises homography matrixes corresponding to a plurality of shooting distances in a shooting range of the shooting device, and the homography matrixes are used for expressing a coordinate mapping relation between the target RGB image and the target thermal infrared image when the shooting device is separated from the first shooting target by the shooting distance;
and searching a homography matrix corresponding to the target shooting distance in the affine transformation model, and recording the homography matrix as the mapping matrix.
In a possible implementation manner of the first aspect, the obtaining an affine transformation model of the camera includes:
for each second target distance, when the shooting device is separated from a third shooting target by the second target distance, a second sample RGB image and a sample thermal infrared image which comprise the third shooting target are obtained through the shooting device, wherein the second target distance is any shooting distance in the shooting range of the shooting device;
acquiring a first target point in the second sample RGB image, and acquiring a second target point matched with the first target point in the sample thermal infrared image, wherein the first target point is any pixel point in an image area corresponding to the third shooting target in the second sample RGB image;
and calculating a homography matrix corresponding to the second target distance according to the first target point and the second target point.
In a possible implementation manner of the first aspect, the number of the first target points is at least 3, and correspondingly, the number of the second target points is the same as the number of the first target points;
the calculating the homography matrix corresponding to the second target distance according to the first target point and the second target point comprises:
for each first target point and a second target point corresponding to the first target point, calculating the target point by formula
Figure BDA0002621598920000031
Calculating a homography matrix corresponding to the second target distance;
wherein (u)c，vc) (u) is the coordinate of the first target pointd，vd) And H is a homography matrix corresponding to the second target distance.
In a second aspect, an embodiment of the present application provides an image registration apparatus, including:
an image acquisition unit for acquiring a target RGB image including a first photographic target and a target thermal infrared image by a photographic device;
the distance calculation unit is used for calculating a first area of a detection frame of the first shooting target in the target RGB image and calculating a target shooting distance between the first shooting target and the shooting device according to the first area;
the matrix acquisition unit is used for acquiring a mapping matrix corresponding to the target shooting distance, wherein the mapping matrix is used for representing a coordinate mapping relation between the target RGB image and the target thermal infrared image when the first shooting target is spaced from the shooting device by the target shooting distance;
and the image registration unit is used for mapping a first pixel point in the target RGB image to the target thermal infrared image according to the mapping matrix to obtain a second pixel point corresponding to the first pixel point in the target thermal infrared image.
In a third aspect, an embodiment of the present application provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor, when executing the computer program, implements the image registration method according to any one of the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium, and the present application provides a computer-readable storage medium, which stores a computer program, where the computer program is executed by a processor to implement the image registration method according to any one of the above first aspects.
In a fifth aspect, the present application provides a computer program product, which when run on a terminal device, causes the terminal device to execute the image registration method according to any one of the first aspect.
It is understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description of the first aspect, and are not described herein again.
Compared with the prior art, the embodiment of the application has the advantages that:
according to the method and the device, a target RGB image and a target thermal infrared image of a first shooting target are obtained through a shooting device; then calculating a first area of a detection frame of the first shooting target in the target RGB image, and calculating a target shooting distance between the first shooting target and the shooting device according to the first area; and finally, mapping a first pixel point in the target RGB image to the target thermal infrared image according to the mapping matrix to obtain a second pixel point in the target thermal infrared image corresponding to the first pixel point. Because the mapping matrix is pre-established, the image registration can be carried out according to the mapping matrix corresponding to the target shooting distance only by acquiring the target shooting distance, so that the image registration time is greatly saved; in addition, by the method, the corresponding matrix can be directly acquired no matter how the shooting distance of the target changes, and the robustness of the image registration method is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic flowchart of an image registration method provided in an embodiment of the present application;
FIG. 2 is a schematic diagram of a detection block provided in an embodiment of the present application;
FIG. 3 is a schematic diagram of matching points provided by an embodiment of the present application;
fig. 4 is a block diagram of an image registration apparatus provided in an embodiment of the present application;
fig. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise.
Referring to fig. 1, which is a schematic flowchart of an image registration method provided in an embodiment of the present application, by way of example and not limitation, the method may include the following steps:
s101, acquiring a target RGB image and a target thermal infrared image of a first shooting target through a shooting device.
In the embodiment of the present application, the photographing device may be a non-contact temperature measuring device. In an application scenario, the process of measuring the temperature of the surface of the human body by using the non-contact temperature measuring device comprises the following steps: the method comprises the steps of acquiring an RGB image and a thermal infrared image of a human body surface (such as a face) through a non-contact temperature measuring device, then carrying out image registration on the RGB image and the thermal infrared image, then acquiring a temperature measuring point (such as an eyebrow position) in the RGB image, mapping the temperature measuring point to the thermal infrared image through an image registration result, acquiring a point (namely a point corresponding to the eyebrow position in the thermal infrared image) in the thermal infrared image, and finally recording a temperature value of the point in the thermal infrared image, which corresponds to the temperature measuring point, as a temperature value of the human body.
The non-contact temperature measuring device can be regarded as a pair of binocular cameras consisting of an RGB camera and a thermal infrared camera. The images (i.e., RGB images and thermal infrared images) captured by the two cameras have a scale, rotation and translation relationship. The image registration method provided by the embodiment of the application is substantially used for registering two images acquired by the binocular camera.
The image registration is a process of matching and superimposing two or more images acquired at different times and under different imaging devices or under different conditions (weather, illuminance, camera position and angle, etc.). Typically, the image registration procedure is: firstly, feature extraction is carried out on two images to obtain feature points, matched feature points are found through similarity measurement, then image space coordinate transformation parameters are obtained through the matched feature points, and finally image registration is carried out through the coordinate transformation parameters. In other words, the essence of image registration is to find a coordinate mapping relationship between the two images. However, as can be seen from the above description, the existing image registration method needs to calculate the coordinate mapping relationship in real time, which results in a long time for image registration.
In order to solve the above problem, in the embodiment of the present application, the following steps are adopted for image registration.
S102, calculating a first area of a detection frame of the first shooting target in the target RGB image, and calculating a target shooting distance between the first shooting target and the shooting device according to the first area.
The detection frame of the first photographic target refers to a mark frame corresponding to an image area occupied by the first photographic target in the target RGB image, such as a rectangular frame, a circular frame, and the like. Referring to fig. 2, a schematic diagram of a detection block provided in the embodiment of the present application is shown. As shown in fig. 2, the target RGB image includes a human body image, where the first photographic target is a human face portion, and correspondingly, the detection frame 201 is a detection frame of the first photographic target.
According to the perspective principle of the camera, the camera is small and large. The size and distance of the same object imaged in the camera is approximately proportional. Therefore, alternatively, one way to calculate the target shooting distance is to:
acquiring a proportionality coefficient, wherein the proportionality coefficient is used for representing the proportionality relation between the first region area and the target shooting distance; and calculating the target shooting distance according to the scale coefficient and the first region area.
Wherein, the proportionality coefficient needs to be calibrated in advance. In the embodiment of the present application, the manner of obtaining the proportionality coefficient includes:
when the shooting device is spaced from a second shooting target by a first target distance, obtaining a first sample RGB image including the second shooting target through the shooting device, wherein the first target distance is any shooting distance in a shooting range of the shooting device; and acquiring a second area of the detection frame of the second shooting target in the first sample RGB image, and calculating the proportionality coefficient according to the second area and the first target distance.
Illustratively, assume that the first target distance is 3m, that is, when the second photographic target is 3m away from the camera, a first sample RGB image of the second photographic target is acquired by the camera. And assuming that the second shooting target is a face part, detecting the face in the RGB image of the first sample by using the existing face detection method to obtain a second region area of a detection frame of the face part, and finally calculating a ratio of the second region area to the first target distance of 3m, wherein the ratio is recorded as a proportionality coefficient.
In the embodiment of the present application, the first photographic target refers to a photographic target in an actual temperature measurement process, and the second photographic target refers to a photographic target used when a scale factor is calculated in advance. The scale factor is fixed for the same photographing apparatus regardless of whether the photographing target is the same. In other words, when the same imaging device images different imaging targets, the scaling factor of the imaging device can be used for calculation.
Wherein, the shooting target can be a whole target, such as a human body surface; or local objects such as human face parts. The sample RGB image and the sample thermal infrared image may include the photographic subject, and may not include only the photographic subject. For example, if the imaging target is a human face part, the upper half of the human body may be imaged, and the obtained sample RGB image and the sample thermal infrared image include not only the human face part but also other parts of the upper half of the human body.
And S103, acquiring a mapping matrix corresponding to the target shooting distance.
The mapping matrix is used for representing a coordinate mapping relation between the target RGB image and the target thermal infrared image when the first shooting target is spaced from the shooting device by the target shooting distance.
The coordinate mapping relation between the two images meets the homography relation, namely one shooting distance corresponds to one coordinate mapping relation. Therefore, after the target shooting distance is obtained, in order to directly and quickly obtain the mapping matrix corresponding to the target shooting distance, the mapping matrices corresponding to different shooting distances need to be calibrated in advance.
Optionally, the manner of obtaining the mapping matrix corresponding to the target shooting distance may include:
I. and obtaining an affine transformation model of the shooting device.
The affine transformation model comprises homography matrixes corresponding to a plurality of shooting distances in a shooting range of the shooting device, and the homography matrixes are used for representing coordinate mapping relations between the target RGB image and the target thermal infrared image when the shooting device is separated from the first shooting target by the shooting distance.
The shooting distance in the embodiment of the present application may be a specific certain distance value, or may be a certain distance segment. For example, assuming that the shooting range of the shooting device is 3m-5m, distance segments may be divided at intervals of 50cm (e.g., 3m-3.5m is a distance segment, 3.5m-4m is a distance segment, etc.), and then the homography matrix corresponding to each distance segment is calculated.
II. And searching a homography matrix corresponding to the target shooting distance in the affine transformation model, and recording the homography matrix as the mapping matrix.
When each homography matrix in the affine transformation model corresponds to a distance segment, the distance segment corresponding to the target shooting distance may be determined first, and then the homography matrix corresponding to the distance segment is used as the mapping matrix corresponding to the target shooting distance.
When each homography matrix in the affine transformation model corresponds to a specific distance value, if the target shooting distance is not equal to the distance value corresponding to any homography matrix in the affine transformation model, the homography matrix corresponding to the distance value with the minimum difference value of the target shooting distances can be recorded as the mapping matrix of the target shooting distance. For example: the assumed affine transformation model includes a homography matrix corresponding to a shooting distance of 3m and a homography matrix corresponding to a shooting distance of 5 m. When the target shooting distance is 3.5m, since the difference between 3.5m and 3m is minimum, the homography matrix corresponding to the shooting distance of 3m is recorded as the mapping matrix of the target shooting distance of 3.5 m.
Further, the manner of obtaining the affine transformation model may include:
1) for each second target distance, when the shooting device is separated from a third shooting target by the second target distance, a second sample RGB image and a sample thermal infrared image which comprise the third shooting target are obtained through the shooting device; the second target distance is any shooting distance within the shooting range of the shooting device.
In the embodiment of the present application, the third photographic target refers to a photographic target used for calibration at a stage of establishing an affine transformation model in advance. The affine transformation model is fixed for the same photographing apparatus regardless of whether the photographing target is the same. In other words, when the same imaging device images different imaging targets, the affine transformation model of the imaging device can be used for image registration. Therefore, at the stage of establishing the affine transformation model in advance, the third photographic target can be arbitrarily selected.
In order to ensure that the affine transformation model is more accurate, generally, which kind of object is photographed in the actual temperature measurement process, and which kind of object is photographed in the process of establishing the affine transformation model. For example, in the actual temperature measurement process, the temperature of the human body surface needs to be measured, and in the corresponding affine transformation model establishment process, the third shooting target is the human body surface; in the actual temperature measurement process, the temperature of the face part needs to be measured, and the third shooting target is the face part in the corresponding affine transformation model establishment process.
Fig. 3 is a schematic diagram of a matching point provided in the embodiment of the present application. Fig. 3 (a) shows a second sample RGB image, and fig. 3 (b) shows a sample thermal infrared image.
2) And acquiring a first target point in the second sample RGB image, and acquiring a second target point matched with the first target point in the sample thermal infrared image, wherein the first target point is any pixel point in an image area corresponding to the third shooting target in the second sample RGB image.
The first target point may be manually selected and the second target point may be manually located. The first and second target points are then manually marked. Acquiring the first target point and the second target point herein means acquiring coordinates of the first target point in the second sample RGB image and coordinates of the second target point in the sample thermal infrared image.
As shown in fig. 3, two points connected by a dashed line segment are a pair of matching points.
3) And calculating a homography matrix corresponding to the second target distance according to the first target point and the second target point.
Correspondingly, the way of calculating the homography matrix corresponding to the second target distance in step 3) is as follows:
for each first target point and a second target point corresponding to the first target point, calculating the target point by formula
Figure BDA0002621598920000111
Calculating a homography matrix corresponding to the second target distance;
wherein (u)c，vc) (u) is the coordinate of the first target pointd，vd) And H is a homography matrix corresponding to the second target distance.
Figure BDA0002621598920000112
sxAnd syDenotes a scale between the second sample RGB image and the sample thermal infrared image, theta denotes a rotation angle between the second sample RGB image and the sample thermal infrared image, txAnd tyRepresenting the translation distance between the second sample RGB image and the sample thermal infrared image.
As can be seen from the above expression of H, there are 5 parameters in H. Since a pair of matching points can provide two equations, at least three pairs of matching points are required to solve the above equations.
Further, the number of the first target points is at least 3, and correspondingly, the number of the second target points is the same as that of the first target points.
When the logarithm of the matching points is more than 3, the solution H can be optimized using a least squares method and a random sampling algorithm.
In the process of selecting the first target point, the first target point is ensured to be uniformly distributed in the image area corresponding to the second shooting target as much as possible.
In the embodiment of the application, the affine transformation model is established in advance, and when image registration is needed, image registration is directly carried out according to the established affine transformation model, recalculation is not needed, and the time for image registration is greatly saved.
S104, mapping a first pixel point in the target RGB image to the target thermal infrared image according to the mapping matrix, and obtaining a second pixel point corresponding to the first pixel point in the target thermal infrared image.
In practical application, the first pixel point is usually a temperature measurement point in the target RGB image (as described in the application scenario in step S101). For example: the temperature of the eyebrow position of the human face is usually selected as the measured temperature of the human body. Then the first pixel point in the target RGB image is the pixel point corresponding to the eyebrow position of the human face.
By the method, the corresponding matrix can be directly acquired no matter how the shooting distance of the target changes, and the robustness of the image registration method is improved.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Fig. 4 is a structural block diagram of an image registration apparatus provided in an embodiment of the present application, corresponding to the image registration method described in the foregoing embodiment, and only a part related to the embodiment of the present application is shown for convenience of description.
Referring to fig. 4, the apparatus includes:
an image acquisition unit 41 for acquiring a target RGB image including the first photographic target and a target thermal infrared image by the photographic device.
A distance calculating unit 42, configured to calculate a first area of a detection frame of the first photographic target in the target RGB image, and calculate a target photographic distance between the first photographic target and the photographic apparatus according to the first area.
A matrix obtaining unit 43, configured to obtain a mapping matrix corresponding to the target shooting distance, where the mapping matrix is used to represent a coordinate mapping relationship between the target RGB image and the target thermal infrared image when the first shooting target is spaced from the shooting device by the target shooting distance.
And the image registration unit 44 is configured to map a first pixel point in the target RGB image to the target thermal infrared image according to the mapping matrix, and obtain a second pixel point corresponding to the first pixel point in the target thermal infrared image.
Optionally, the distance calculating unit 42 includes:
and the coefficient acquisition module is used for acquiring a proportionality coefficient, and the proportionality coefficient is used for expressing the proportionality relation between the first area and the target shooting distance.
And the distance calculation module is used for calculating the target shooting distance according to the proportional coefficient and the first area.
Optionally, the coefficient obtaining module is further configured to:
when the shooting device is spaced from a second shooting target by a first target distance, obtaining a first sample RGB image including the second shooting target through the shooting device, wherein the first target distance is any shooting distance in a shooting range of the shooting device;
and acquiring a second area of the detection frame of the second shooting target in the first sample RGB image, and calculating the proportionality coefficient according to the second area and the first target distance.
Optionally, the matrix obtaining unit 43 includes:
the model obtaining module is used for obtaining an affine transformation model of the shooting device, wherein the affine transformation model comprises homography matrixes corresponding to a plurality of shooting distances in a shooting range of the shooting device, and the homography matrixes are used for representing a coordinate mapping relation between the target RGB image and the target thermal infrared image when the shooting device is separated from the first shooting target by the shooting distance.
And the searching module is used for searching a homography matrix corresponding to the target shooting distance in the affine transformation model and recording the homography matrix as the mapping matrix.
Optionally, the model obtaining module is further configured to:
for each second target distance, when the shooting device is separated from a third shooting target by the second target distance, a second sample RGB image and a sample thermal infrared image which comprise the third shooting target are obtained through the shooting device, wherein the second target distance is any shooting distance in the shooting range of the shooting device;
acquiring a first target point in the second sample RGB image, and acquiring a second target point matched with the first target point in the sample thermal infrared image, wherein the first target point is any pixel point in an image area corresponding to the third shooting target in the second sample RGB image;
and calculating a homography matrix corresponding to the second target distance according to the first target point and the second target point.
Optionally, the number of the first target points is at least 3, and correspondingly, the number of the second target points is the same as that of the first target points.
Optionally, the model obtaining module is further configured to:
for each first target point and a second target point corresponding to the first target point, calculating the target point by formula
Figure BDA0002621598920000131
Calculating a homography matrix corresponding to the second target distance; wherein (u)c，vc) (u) is the coordinate of the first target pointd，vd) And H is a homography matrix corresponding to the second target distance.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
In addition, the image registration apparatus shown in fig. 4 may be a software unit, a hardware unit, or a combination of software and hardware unit that is built in the existing terminal device, may be integrated into the terminal device as an independent pendant, and may also exist as an independent terminal device.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
Fig. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present application. As shown in fig. 5, the terminal device 5 of this embodiment includes: at least one processor 50 (only one shown in fig. 5), a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, the processor 50 implementing the steps in any of the various image registration method embodiments described above when executing the computer program 52.
The terminal device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The terminal device may include, but is not limited to, a processor, a memory. Those skilled in the art will appreciate that fig. 5 is only an example of the terminal device 5, and does not constitute a limitation to the terminal device 5, and may include more or less components than those shown, or combine some components, or different components, such as an input-output device, a network access device, and the like.
The Processor 50 may be a Central Processing Unit (CPU), and the Processor 50 may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 51 may in some embodiments be an internal storage unit of the terminal device 5, such as a hard disk or a memory of the terminal device 5. The memory 51 may also be an external storage device of the terminal device 5 in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the terminal device 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the terminal device 5. The memory 51 is used for storing an operating system, an application program, a Boot Loader (Boot Loader), data, and other programs, such as program codes of the computer programs. The memory 51 may also be used to temporarily store data that has been output or is to be output.
The embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps in the above-mentioned method embodiments.
The embodiments of the present application provide a computer program product, which when running on a terminal device, enables the terminal device to implement the steps in the above method embodiments when executed.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to an apparatus/terminal device, recording medium, computer Memory, Read-Only Memory (ROM), Random-Access Memory (RAM), electrical carrier wave signals, telecommunications signals, and software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims (10)

1. An image registration method, comprising:
acquiring a target RGB image and a target thermal infrared image including a first shooting target through a shooting device;
calculating a first area of a detection frame of the first shooting target in the target RGB image, and calculating a target shooting distance between the first shooting target and the shooting device according to the first area;
acquiring a mapping matrix corresponding to the target shooting distance, wherein the mapping matrix is used for representing a coordinate mapping relation between the target RGB image and the target thermal infrared image when the first shooting target is spaced from the shooting device by the target shooting distance;
and mapping a first pixel point in the target RGB image to the target thermal infrared image according to the mapping matrix to obtain a second pixel point corresponding to the first pixel point in the target thermal infrared image.
2. The image registration method according to claim 1, wherein the calculating of the target photographing distance between the first photographing target and the photographing device according to the first region area includes:
acquiring a proportionality coefficient, wherein the proportionality coefficient is used for representing the proportionality relation between the first region area and the target shooting distance;
and calculating the target shooting distance according to the scale coefficient and the first region area.
3. The image registration method of claim 2, wherein the obtaining the scaling factor comprises:
when the shooting device is spaced from a second shooting target by a first target distance, obtaining a first sample RGB image including the second shooting target through the shooting device, wherein the first target distance is any shooting distance in a shooting range of the shooting device;
and acquiring a second area of the detection frame of the second shooting target in the first sample RGB image, and calculating the proportionality coefficient according to the second area and the first target distance.
4. The image registration method according to claim 1, wherein the obtaining a mapping matrix corresponding to the target shooting distance includes:
acquiring an affine transformation model of the shooting device, wherein the affine transformation model comprises homography matrixes corresponding to a plurality of shooting distances in a shooting range of the shooting device, and the homography matrixes are used for expressing a coordinate mapping relation between the target RGB image and the target thermal infrared image when the shooting device is separated from the first shooting target by the shooting distance;
and searching a homography matrix corresponding to the target shooting distance in the affine transformation model, and recording the homography matrix as the mapping matrix.
5. The image registration method of claim 4, wherein the obtaining an affine transformation model of the camera comprises:
for each second target distance, when the shooting device is separated from a third shooting target by the second target distance, a second sample RGB image and a sample thermal infrared image which comprise the third shooting target are obtained through the shooting device, wherein the second target distance is any shooting distance in the shooting range of the shooting device;
acquiring a first target point in the second sample RGB image, and acquiring a second target point matched with the first target point in the sample thermal infrared image, wherein the first target point is any pixel point in an image area corresponding to the third shooting target in the second sample RGB image;
and calculating a homography matrix corresponding to the second target distance according to the first target point and the second target point.
6. The image registration method of claim 5, wherein the number of the first target points is at least 3, and accordingly, the number of the second target points is the same as the number of the first target points;
the calculating the homography matrix corresponding to the second target distance according to the first target point and the second target point comprises:
for each first target point and a second target point corresponding to the first target point, calculating the target point by formula
Figure FDA0002621598910000021
Calculating a homography matrix corresponding to the second target distance;
wherein (u)c，vc) (u) is the coordinate of the first target pointd，vd) And H is a homography matrix corresponding to the second target distance.
7. An image registration apparatus, comprising:
an image acquisition unit for acquiring a target RGB image including a first photographic target and a target thermal infrared image by a photographic device;
the distance calculation unit is used for calculating a first area of a detection frame of the first shooting target in the target RGB image and calculating a target shooting distance between the first shooting target and the shooting device according to the first area;
the matrix acquisition unit is used for acquiring a mapping matrix corresponding to the target shooting distance, wherein the mapping matrix is used for representing a coordinate mapping relation between the target RGB image and the target thermal infrared image when the first shooting target is spaced from the shooting device by the target shooting distance;
and the image registration unit is used for mapping a first pixel point in the target RGB image to the target thermal infrared image according to the mapping matrix to obtain a second pixel point corresponding to the first pixel point in the target thermal infrared image.
8. The image registration apparatus according to claim 7, wherein the distance calculation unit includes:
the coefficient acquisition module is used for acquiring a proportionality coefficient, and the proportionality coefficient is used for expressing the proportionality relation between the first area and the target shooting distance;
and the distance calculation module is used for calculating the target shooting distance according to the proportional coefficient and the first area.
9. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 6.
CN202010784953.7A 2020-08-06 2020-08-06 Image registration method, device, terminal equipment and computer readable storage medium Active CN112085771B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010784953.7A CN112085771B (en) 2020-08-06 2020-08-06 Image registration method, device, terminal equipment and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010784953.7A CN112085771B (en) 2020-08-06 2020-08-06 Image registration method, device, terminal equipment and computer readable storage medium

Publications (2)

Publication Number Publication Date
CN112085771A true CN112085771A (en) 2020-12-15
CN112085771B CN112085771B (en) 2023-12-05

Family

ID=73736173

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010784953.7A Active CN112085771B (en) 2020-08-06 2020-08-06 Image registration method, device, terminal equipment and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN112085771B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113923317A (en) * 2021-08-30 2022-01-11 珠海视熙科技有限公司 Camera frame synchronization test method, device and storage medium
CN114862923A (en) * 2022-07-06 2022-08-05 武汉市聚芯微电子有限责任公司 Image registration method and device and storage medium
CN115797426A (en) * 2023-02-13 2023-03-14 合肥的卢深视科技有限公司 Image alignment method, electronic device and storage medium
CN116152051A (en) * 2023-02-27 2023-05-23 上海福柯斯智能科技有限公司 Method and device for blocking registration image silhouette of X-ray image

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102930525A (en) * 2012-09-14 2013-02-13 武汉大学 Line matching method based on affine invariant feature and homography
CN104501779A (en) * 2015-01-09 2015-04-08 中国人民解放军63961部队 High-accuracy target positioning method of unmanned plane on basis of multi-station measurement
US20160117820A1 (en) * 2014-10-23 2016-04-28 Hanwha Techwin Co., Ltd. Image registration method
CN110110629A (en) * 2019-04-25 2019-08-09 北京工业大学 Personal information detection method and system towards indoor environmental condition control
CN110163898A (en) * 2019-05-07 2019-08-23 腾讯科技（深圳）有限公司 A kind of depth information method for registering and device
CN110529186A (en) * 2019-09-11 2019-12-03 上海同岩土木工程科技股份有限公司 Tunnel structure percolating water based on infrared thermal imaging accurately identifies device and method
CN110691193A (en) * 2019-09-03 2020-01-14 RealMe重庆移动通信有限公司 Camera switching method and device, storage medium and electronic equipment
CN111199198A (en) * 2019-12-27 2020-05-26 深圳市优必选科技股份有限公司 Image target positioning method, image target positioning device and mobile robot
CN111242991A (en) * 2020-01-10 2020-06-05 大连理工大学 Method for quickly registering visible light and infrared camera
CN111383264A (en) * 2018-12-29 2020-07-07 深圳市优必选科技有限公司 Positioning method, positioning device, terminal and computer storage medium

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102930525A (en) * 2012-09-14 2013-02-13 武汉大学 Line matching method based on affine invariant feature and homography
US20160117820A1 (en) * 2014-10-23 2016-04-28 Hanwha Techwin Co., Ltd. Image registration method
CN104501779A (en) * 2015-01-09 2015-04-08 中国人民解放军63961部队 High-accuracy target positioning method of unmanned plane on basis of multi-station measurement
CN111383264A (en) * 2018-12-29 2020-07-07 深圳市优必选科技有限公司 Positioning method, positioning device, terminal and computer storage medium
CN110110629A (en) * 2019-04-25 2019-08-09 北京工业大学 Personal information detection method and system towards indoor environmental condition control
CN110163898A (en) * 2019-05-07 2019-08-23 腾讯科技（深圳）有限公司 A kind of depth information method for registering and device
CN110691193A (en) * 2019-09-03 2020-01-14 RealMe重庆移动通信有限公司 Camera switching method and device, storage medium and electronic equipment
CN110529186A (en) * 2019-09-11 2019-12-03 上海同岩土木工程科技股份有限公司 Tunnel structure percolating water based on infrared thermal imaging accurately identifies device and method
CN111199198A (en) * 2019-12-27 2020-05-26 深圳市优必选科技股份有限公司 Image target positioning method, image target positioning device and mobile robot
CN111242991A (en) * 2020-01-10 2020-06-05 大连理工大学 Method for quickly registering visible light and infrared camera

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
王俊影;李扬;袁浩期;简单;郝敏;: "可见光与红外图像自动配准算法的设计与实现", 《机电工程技术》, vol. 48, no. 11, pages 134 - 137 *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113923317A (en) * 2021-08-30 2022-01-11 珠海视熙科技有限公司 Camera frame synchronization test method, device and storage medium
CN113923317B (en) * 2021-08-30 2022-11-15 珠海视熙科技有限公司 Camera frame synchronization test method, device and storage medium
CN114862923A (en) * 2022-07-06 2022-08-05 武汉市聚芯微电子有限责任公司 Image registration method and device and storage medium
CN114862923B (en) * 2022-07-06 2022-09-09 武汉市聚芯微电子有限责任公司 Image registration method and device and storage medium
CN115797426A (en) * 2023-02-13 2023-03-14 合肥的卢深视科技有限公司 Image alignment method, electronic device and storage medium
CN115797426B (en) * 2023-02-13 2023-05-12 合肥的卢深视科技有限公司 Image alignment method, electronic device and storage medium
CN116152051A (en) * 2023-02-27 2023-05-23 上海福柯斯智能科技有限公司 Method and device for blocking registration image silhouette of X-ray image
CN116152051B (en) * 2023-02-27 2023-12-01 上海福柯斯智能科技有限公司 Method and device for blocking registration image silhouette of X-ray image

Also Published As

Publication number Publication date
CN112085771B (en) 2023-12-05

Similar Documents

Publication Publication Date Title
WO2021115071A1 (en) Three-dimensional reconstruction method and apparatus for monocular endoscope image, and terminal device
CN112085771B (en) Image registration method, device, terminal equipment and computer readable storage medium
CN109559349B (en) Method and device for calibration
CN111080526B (en) Method, device, equipment and medium for measuring and calculating farmland area of aerial image
CN112348863B (en) Image alignment method, image alignment device and terminal equipment
CN112927306B (en) Calibration method and device of shooting device and terminal equipment
CN112686950B (en) Pose estimation method, pose estimation device, terminal equipment and computer readable storage medium
CN113609907B (en) Multispectral data acquisition method, device and equipment
CN111383264B (en) Positioning method, positioning device, terminal and computer storage medium
CN112102378B (en) Image registration method, device, terminal equipment and computer readable storage medium
CN113838151B (en) Camera calibration method, device, equipment and medium
CN113362445B (en) Method and device for reconstructing object based on point cloud data
Santana-Cedrés et al. Estimation of the lens distortion model by minimizing a line reprojection error
CN112200002B (en) Body temperature measuring method, device, terminal equipment and storage medium
Jin et al. Registration of UAV images using improved structural shape similarity based on mathematical morphology and phase congruency
CN113191189A (en) Face living body detection method, terminal device and computer readable storage medium
CN107734324B (en) Method and system for measuring illumination uniformity of flash lamp and terminal equipment
CN112200842B (en) Image registration method, device, terminal equipment and storage medium
CN111336938A (en) Robot and object distance detection method and device thereof
CN112991411B (en) Image registration method and apparatus, and storage medium
CN110930455B (en) Positioning method, positioning device, terminal equipment and storage medium
CN111062984B (en) Method, device, equipment and storage medium for measuring area of video image area
CN114415129A (en) Visual and millimeter wave radar combined calibration method and device based on polynomial model
CN111223139B (en) Target positioning method and terminal equipment
CN113223033A (en) Poultry body temperature detection method, device and medium based on image fusion

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant