CN112184544A - Image splicing method and device - Google Patents
Image splicing method and device Download PDFInfo
- Publication number
- CN112184544A CN112184544A CN201910600184.8A CN201910600184A CN112184544A CN 112184544 A CN112184544 A CN 112184544A CN 201910600184 A CN201910600184 A CN 201910600184A CN 112184544 A CN112184544 A CN 112184544A
- Authority
- CN
- China
- Prior art keywords
- image frame
- frame
- image
- determining
- sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000009466 transformation Effects 0.000 claims abstract description 63
- 239000011159 matrix material Substances 0.000 claims abstract description 62
- 238000006073 displacement reaction Methods 0.000 claims description 61
- 230000001133 acceleration Effects 0.000 claims description 17
- 238000001514 detection method Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 230000015654 memory Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/14—Transformations for image registration, e.g. adjusting or mapping for alignment of images
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
Abstract
The disclosure provides a method and a device for image splicing, and belongs to the technical field of videos. The method comprises the following steps: acquiring an image frame sequence shot by a target camera, determining a registration result of an mth image frame and an m + k image frame according to image contents of the mth image frame and the m + k image frame in the image frame sequence, wherein m and k are positive integers, determining a transformation matrix between the mth image frame and the m + k image frame according to the registration result, then determining a starting frame and an ending frame where a moving target is located in the image frame sequence, and splicing the image frames from the starting frame to the ending frame in the image frame sequence according to the transformation matrix. By adopting the method and the device, the image splicing efficiency can be improved.
Description
Technical Field
The present disclosure relates to the field of video technologies, and in particular, to a method and an apparatus for image stitching.
Background
The image stitching is to stitch a plurality of partial images with overlapped parts into a seamless panoramic image. For example, on a road or a container port gate, since a truck is generally long, the image frames captured by the gate camera generally do not include the full view of the truck, but during the process that the truck passes through the gate, multiple image frames at different times include images of all parts of the whole truck, and the images of all parts are spliced to obtain a complete truck image.
In the related art, when image stitching is performed, the following processes are generally performed on images captured by a single fixed camera in sequence: detecting a moving target, correcting distortion, extracting and registering features, and splicing images.
Since image stitching can be performed only after moving object detection is performed, image stitching efficiency is low.
Disclosure of Invention
In order to solve the problem of low image stitching efficiency, the embodiment of the disclosure provides an image stitching method and device. The technical scheme is as follows:
in a first aspect, a method for image stitching is provided, where the method includes:
acquiring an image frame sequence shot by a target camera;
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are successfully registered, determining a transformation matrix between the m-th image frame and the m + k-th image frame according to the registration result of the m-th image frame and the m + k-th image frame, wherein m and k are positive integers;
according to the registration result, determining a starting frame and an ending frame where a moving object is located in the image frame sequence;
and splicing each image frame from the starting frame to the ending frame in the image frame sequence according to the transformation matrix.
Optionally, the determining, according to the registration result, a start frame and an end frame where the moving object is located in the image frame sequence includes:
determining a first pixel displacement of a first feature point in the mth image frame and a second feature point in the m +1 image frame in a motion direction according to the registration result; if the first pixel displacement is larger than or equal to a first numerical value, determining the mth image frame as a starting frame of a moving target in the image frame sequence, and if the first pixel displacement is smaller than a second numerical value, determining the mth image frame as a starting frame of the moving target in the image frame sequence, wherein the first characteristic point and the second characteristic point belong to matched characteristic point pairs; alternatively, the first and second electrodes may be,
determining second pixel displacement of a first image block in the mth image frame and a second image block in the (m + 1) th image frame in the motion direction according to the registration result; if the second pixel displacement is larger than or equal to the first numerical value, determining the m-th image frame as a starting frame of a moving object in the image frame sequence; and if the second pixel displacement is smaller than the second value, determining that the mth image frame is an end frame where a moving object is located in the image frame sequence, wherein the first image block and the second image block are matched image block pairs.
Optionally, the splicing, according to the transformation matrix, each image frame from the starting frame to the ending frame in the image frame sequence includes:
converting each image frame from the initial frame to the end frame in the image frame sequence to the same coordinate system according to the transformation matrix;
and splicing the image frames from the starting frame to the ending frame in the image frame sequence based on the coordinates of the image frames from the starting frame to the ending frame in the same coordinate system.
Optionally, the determining a transformation matrix between the mth image frame and the m + k image frame according to the registration result of the mth image frame and the m + k image frame includes:
and determining a transformation matrix between the mth image frame and the m + k image frame according to the matched feature point pairs or matched image block pairs of the mth image frame and the m + k image frame.
Optionally, the method further includes:
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are determined to be failed in registration, motion estimation is carried out;
determining a transformation matrix between the m-th image frame and the m + k-th image frame based on a result of the motion estimation.
Optionally, the performing motion estimation includes:
determining an average acceleration of a moving object from the m-th image frame to the m + k-th image frame;
the determining a transformation matrix between the m-th image frame and the m + k-th image frame based on the result of the motion estimation includes:
determining a transformation matrix between the m-th image frame and the m + k-th image frame based on the average acceleration.
Optionally, in the image frame sequence, determining a start frame and an end frame where the moving object is located includes:
and acquiring detection data detected by a sensor, and determining a starting frame and an ending frame of the moving object in the image frame sequence according to the detection data.
In a second aspect, an apparatus for image stitching is provided, the apparatus comprising:
the acquisition module is used for acquiring an image frame sequence shot by the target camera;
a determination module to:
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are successfully registered, determining a transformation matrix between the m-th image frame and the m + k-th image frame according to the registration result of the m-th image frame and the m + k-th image frame, wherein m and k are positive integers;
determining a starting frame and an ending frame where a moving object is located in the image frame sequence;
and the splicing module is used for splicing each image frame from the starting frame to the ending frame in the image frame sequence according to the transformation matrix.
8. The apparatus of claim 7, wherein the determining module is configured to:
determining a first pixel displacement of a first feature point in the mth image frame and a second feature point in the m +1 image frame in a motion direction according to the registration result; if the first pixel displacement is larger than or equal to a first numerical value, determining the mth image frame as a starting frame of a moving target in the image frame sequence, and if the first pixel displacement is smaller than a second numerical value, determining the mth image frame as a starting frame of the moving target in the image frame sequence, wherein the first characteristic point and the second characteristic point belong to matched characteristic point pairs; alternatively, the first and second electrodes may be,
determining second pixel displacement of a first image block in the mth image frame and a second image block in the (m + 1) th image frame in the motion direction according to the registration result; if the second pixel displacement is larger than or equal to the first numerical value, determining the m-th image frame as a starting frame of a moving object in the image frame sequence; and if the second pixel displacement is smaller than the second value, determining that the mth image frame is an end frame where a moving object is located in the image frame sequence, wherein the first image block and the second image block are matched image block pairs.
Optionally, the splicing module is configured to:
converting each image frame from the initial frame to the end frame in the image frame sequence to the same coordinate system according to the transformation matrix;
and splicing the image frames from the starting frame to the ending frame in the image frame sequence based on the coordinates of the image frames from the starting frame to the ending frame in the same coordinate system.
Optionally, the determining module is configured to:
and determining a transformation matrix between the mth image frame and the m + k image frame according to the matched feature point pairs or matched image block pairs of the mth image frame and the m + k image frame.
Optionally, the determining module is further configured to:
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are determined to be failed in registration, motion estimation is carried out;
determining a transformation matrix between the m-th image frame and the m + k-th image frame based on a result of the motion estimation.
Optionally, the determining module is configured to:
determining an average acceleration of a moving object from the m-th image frame to the m + k-th image frame;
the determining module is configured to:
determining a transformation matrix between the m-th image frame and the m + k-th image frame based on the average acceleration.
Optionally, the determining module is configured to:
and acquiring detection data detected by a sensor, and determining a starting frame and an ending frame of the moving object in the image frame sequence according to the detection data.
In a third aspect, a stitching device is provided, the stitching device comprising a processor and a memory, the memory having stored therein at least one instruction, the instruction being loaded and executed by the processor to implement the method of image stitching as described in the first aspect above.
In a fourth aspect, there is provided a computer readable storage medium having stored therein at least one instruction which is loaded and executed by a processor to implement the method of image stitching as described in the first aspect above.
The beneficial effects brought by the technical scheme provided by the embodiment of the disclosure at least comprise:
in the embodiment of the disclosure, when image splicing is performed, an image frame sequence shot by a target camera may be obtained, according to image contents of image frames in the image frame sequence, if it is determined that registration of an mth image frame and an m + k image frame is successful, a transformation matrix between the mth image frame and the m + k image frame is determined according to a registration result of the mth image frame and the m + k image frame, where m and k are positive integers, then according to the registration result, a start frame and an end frame where a moving target is located are determined in the image frame sequence, and according to the transformation matrix, each image frame from the start frame to the end frame in the image frame sequence is spliced. Therefore, image splicing can be performed without target detection, and the efficiency of image splicing can be improved.
Drawings
Fig. 1 is a schematic diagram of a camera provided by an embodiment of the present disclosure;
FIG. 2 is a flowchart of a method for image stitching according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of an image frame provided by an embodiment of the present disclosure;
fig. 4 is a schematic diagram of a matched feature point pair provided by the embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a pair of matched image blocks provided by an embodiment of the present disclosure;
fig. 6 is a schematic diagram of determining a start frame and an end frame according to an embodiment of the disclosure;
FIG. 7 is a schematic diagram of an image frame in a unified coordinate system provided by an embodiment of the present disclosure;
FIG. 8 is a perspective view of a moving object provided by an embodiment of the present disclosure;
FIG. 9 is a flowchart of a method for image stitching according to an embodiment of the present disclosure;
FIG. 10 is a schematic structural diagram of an apparatus for image stitching according to an embodiment of the present disclosure;
fig. 11 is a schematic structural diagram of a splicing apparatus provided in an embodiment of the present disclosure.
Detailed Description
To make the objects, technical solutions and advantages of the present disclosure more apparent, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.
The embodiment of the disclosure provides an image splicing method, wherein an execution main body of the method can be splicing equipment, and the splicing equipment can be a server or a terminal.
The splicing device can be provided with a processor, a memory, a transceiver and the like, wherein the processor can be used for processing in the image splicing process, the memory can be used for storing data required in the image splicing process and generated data, and the transceiver can be used for receiving and transmitting data.
Before implementation, an application scenario of the embodiment of the present disclosure is first introduced:
as shown in fig. 1, a camera having a certain shooting range is provided at a road gate or a container port gate, and the camera can shoot a passing vehicle or the like, and a panoramic image of the passing vehicle can be obtained by shooting an image of the passing vehicle with the camera.
It should be noted that, in the embodiment of the present disclosure, it is assumed that the moving object moves perpendicular to the shooting direction of the camera, or the moving object moves parallel to the shooting direction of the camera.
As shown in fig. 2, an embodiment of the present disclosure provides an image stitching method, where an execution flow of the method may be as follows:
Wherein the target camera is any monocular camera, and takes successive image frames.
In implementation, when the stitching device is to stitch image frames captured by the target camera, an image frame sequence captured by the target camera may be acquired, where the image frame sequence includes a plurality of image frames arranged in the capturing time sequence, as shown in fig. 3, and each image frame includes a part of a moving object (the moving object is, for example, a vehicle) mentioned later.
The m-th image frame is the m-th image frame counted from the first image frame in the image frame sequence, and the m + k-th image frame is the m + k-th image frame counted from the first image frame in the image sequence. For example, m is 5, k is 3, 20 image frames are sequentially arranged in the image frame sequence, the mth image frame is the fifth image frame, and the kth image frame is the eighth image frame.
In implementation, after acquiring the image frame sequence, the stitching device may extract feature points, or image blocks, of each image frame in the image frame sequence using the image content of each image frame. And then matching the feature points in the mth image frame with the feature points in the (m + k) th image frame, if the matching is successful, determining that the registration of the mth image frame and the (m + k) th image frame is successful, determining matched feature point pairs (namely, matching results, shown in fig. 4, are matched feature point pairs), and if the matching is not successful, determining that the registration is failed.
Or matching the image block in the mth image frame with the image block in the m + k image frame, if the matching is successful, determining that the registration of the mth image frame and the m + k image frame is successful, and determining a matched image block pair (i.e., a registration result, which is a matched image block pair as shown in fig. 5), and if the matching is not successful, determining that the registration is failed.
Then, if the registration of the mth image frame and the (m + k) th image frame is successful, a transformation matrix between the mth image frame and the (m + k) th image frame can be determined by directly using a plurality of pairs of matched feature points or pairs of matched image blocks.
It should be noted that, the above-mentioned feature point extraction may be performed by using a surf algorithm and performing registration, and the processing procedure may be as follows:
firstly, detecting the extreme value of the scale space of two image frames to obtain a characteristic point, then refining the position of the characteristic point, calculating the description information of the characteristic point, and generating a characteristic vector for describing the characteristic point, thus obtaining the characteristic point. Then, the registration is obtained by calculating the Euclidean distance between two feature point descriptors, namely finding out two neighbor feature points p2 and p3 which are nearest and next to the Euclidean distance of the feature point p1, then calculating the ratio of the Euclidean distances between the two groups of feature points p1 and p2 and between the two groups of feature points p1 and p3, and if the ratio is smaller than a certain value, determining that the feature point p1 and the feature point p2 are successfully matched, namely that the p1 and the p2 are matched feature point pairs.
In the above description, p1 belongs to the m-th image frame, and p2 and p3 belong to the m + k-th image frame.
In addition, distortion correction and projective transformation can be performed on the image frames to eliminate distortion and perspective before image registration.
Optionally, when the registration is successful, the transformation matrix may be determined as follows:
and determining a transformation matrix between the mth image frame and the m + k image frame according to the matched feature point pairs or matched image block pairs of the mth image frame and the m + k image frame.
In implementation, if the registration is successful and the m-th image frame and the m + k-th image frame match with each other in step 202, the coordinates of the matching feature point pair may be determined, where the coordinate of the feature point i in the m-th image frame in the matching feature point pair is (x)i，yi) The coordinate of the characteristic point j of the m + k th image frame in the matched characteristic point pair is (x)i′，yi'). Then calculate
In equation (1), n represents the number of matched pairs of feature points.
The transformation matrix H of the mth image frame and the m + k image frame may be expressed as:
if the registration result is that the registration is successful, and the image block pair matched with the mth image frame and the mth + k image frame is obtained in step 202, the average value of the abscissa of each pixel point in the image block of the mth image frame and the average value of the ordinate of each pixel point in the image block of the mth image frame may be determined, and the average value of the abscissa of each pixel point in the image block of the mth + k image frame and the average value of the ordinate of each pixel point may be determined. Then, the average value of the abscissa of each pixel point in the image block i and the average value of the abscissa of each pixel point in the image block j are subjected to difference value obtaining
Substituted into the above formula (2) H. And taking the difference value between the average value of the vertical coordinates of all the pixel points in the image block i and the average value of the vertical coordinates of all the pixel points in the image block j to obtain the difference value
Substituting the above H, a transformation matrix of the mth image frame and the m + k image frame can also be obtained.
It should be noted that, in the embodiment of the present disclosure, a rectangular coordinate system is established with the shooting direction of the camera as the y direction, and the direction perpendicular to the y direction and toward the right as the x direction, and the moving object moves perpendicular to the shooting direction of the camera (in this case, the moving object moves perpendicular to the shooting direction of the camera)
0) or parallel to the shooting direction of the camera (in this case)
0) so no rotation is taken into account, so H11＝1,H12＝0,H21＝0,H22＝1。
Alternatively, the transformation matrix may be determined in the following manner if the registration fails:
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are determined to be failed in registration, motion estimation is carried out; based on the result of the motion estimation, a transformation matrix between the m-th image frame and the m + k-th image frame is determined.
In implementation, if the registration fails, motion estimation may be performed to obtain a result of the motion estimation, and then a transformation matrix between the mth image frame and the m + k image frame is determined using the result of the motion estimation.
Optionally, the process of performing motion estimation may be as follows:
and determining the average acceleration of the moving object from the m image frame to the m + k image frame.
In implementation, it is assumed that the moving object is in uniform velocity motion from the m-th image frame to the m + k-th image frame, i.e., the acceleration is constant. The relationship between acceleration and displacement is:
wherein in the formula (3), s is displacement, t is movement time length of the moving object, a is acceleration of the moving object, and v0Is the initial speed of the moving object when the moving object starts to move.
The time length from the beginning of the movement of the moving object to the m + k image frame shot by the object camera is t1Then the displacement from the start of the movement to the m + k th image frame captured by the target camera is
Wherein, in formula (4), a1Is the acceleration from the start of the movement to the m + k image frame captured by the target camera. Thus, can obtain
A is to1An average acceleration of the moving object between the m-th image frame and the m + k-th image frame is determined.
Since the displacement can be determined between the image frames before the mth image frame (the displacement between the ith image frame and the (i + k) th image frame can be directly used
Add to obtain s1I is less than m), so s1Can be obtained, that is, can be obtained1。
The relationship between acceleration and displacement may then be used to determine the displacement of the moving object between the mth image frame and the m + k image frame, i.e., the displacement of the moving object between the mth image frame and the m + k image frame
Wherein, in the formula (6), v1＝a1t2，t2The time length from the beginning of moving of the moving object to the m image frame shot by the object camera is defined.
Then if the moving object is moving along the x direction, the transformation matrix H of the m-th image frame and the m + k-th image frame can be expressed as:
if the moving object is moving along the y direction, the transformation matrix H of the m-th image frame and the m + k-th image frame may be expressed as:
thus, a transformation matrix H of the mth image frame and the m + k image frame is obtained.
In step 203, in the image frame sequence, a start frame and an end frame where the moving object is located are determined.
Wherein, the moving target refers to a target to be subjected to splicing processing. For example, a container vehicle drives into a bayonet, and the container vehicle is a moving target. When the vehicle enters the bayonet, the vehicle is the moving object.
In an implementation, in order to make the number of image frames subjected to the stitching processing smaller, before the stitching processing, a start frame and an end frame where a moving object to be subjected to the stitching processing is located may be first determined, and the corresponding processing may be: the splicing device can determine a starting frame and an ending frame where the moving object is located in the image frame sequence, and the image frames are arranged according to the time sequence, so that the starting frame and the ending frame are determined, and all the image frames containing the moving object can be determined.
And step 204, splicing each image frame from the initial frame to the end frame in the image frame sequence according to the transformation matrix.
In implementation, after the splicing device determines the start frame and the end frame of the moving object, the start frame and the end frame may be used to determine the image frames from the start frame to the end frame, and then the transformation matrix may be used to splice the image frames from the start frame to the end frame to obtain a complete panoramic image.
It should be noted that, in order to complete the panoramic image of the moving object, a start frame and an end frame are included in the stitching process.
Optionally, there are several ways to determine the start frame and the end frame, and two possible ways are given below:
the first method is as follows: and according to the registration result, determining a starting frame and an ending frame where the moving object is located in the image frame sequence.
In implementation, if the registration is successful as a result of the registration, the matching pairs of feature points, or the matching pairs of image blocks, may be used to determine a starting frame and an ending frame in which the moving object is located in the sequence of image frames. If the registration result is registration failure, the result of motion estimation can be used to determine the starting frame and the ending frame where the moving object is located.
The second method comprises the following steps: and acquiring detection data detected by a sensor, and determining a starting frame and an ending frame where the moving target is located in the image frame sequence according to the detection data.
In implementation, a sensor may be further provided within the view angle range of the target camera, and the sensor may be a radar, a laser, an infrared sensor, or the like. The sensor may determine whether a moving object passes. For example, the sensor is an infrared sensor, the infrared sensor emits infrared light, and when the moving target passes through, the infrared light is reflected back, so that as long as the infrared sensor detects the reflected infrared light, the moving target can be determined to pass through, a time point when the infrared sensor detects that the infrared light can be reflected back from the time point when the infrared sensor cannot be reflected back is a time point of starting a frame, and a time point when the infrared sensor detects that the infrared light can be reflected back from the time point when the infrared sensor cannot be reflected back is a time point of ending the frame.
For the infrared sensor, the splicing device may obtain detection data detected by the sensor, determine a time point from when the infrared light cannot be reflected back to when the infrared light can be reflected back, that is, a time point of a start frame, determine a time point from when the infrared light can be reflected back to when the infrared light cannot be reflected back, that is, a time point of an end frame, and then determine a start frame and an end frame where the moving target is located in the image frame sequence by using the time point of the start frame and the time point of the end frame.
Alternatively, using the registration result, the manner of determining the start frame and the end frame may be as follows:
the first method is as follows: determining a first pixel displacement of a first characteristic point in the mth image frame and a second characteristic point in the (m + 1) th image frame in the motion direction according to the registration result; and if the first pixel displacement is larger than or equal to a first numerical value, determining that the mth image frame is the initial frame where the moving target is located in the image frame sequence, and if the first pixel displacement is smaller than a second numerical value, determining that the mth image frame is the initial frame where the moving target is located in the image frame sequence.
The first value and the second value can be preset and stored in the splicing equipment, the first value is larger than the second value, and the second value is generally close to 0. The first characteristic point and the second characteristic point belong to matched characteristic point pairs, the first characteristic point is any characteristic point in the mth image frame, and the second characteristic point is any characteristic point in the m + k image frame.
In implementation, if the registration result is that the registration is successful, the matched feature point pair may be used to determine a first pixel displacement of a first feature point and a second feature point in the matched feature point pair in the motion direction (if the motion direction is the x direction, an abscissa of the first feature point and an abscissa of the second feature point are directly subtracted to obtain an absolute value, that is, the first pixel displacement, if the motion direction is the y direction, the first feature point and an ordinate of the second feature point are directly subtracted to obtain an absolute value, that is, the first pixel displacement), and if the first pixel displacement is greater than or equal to a first numerical value, the mth image frame is determined as a starting frame where the motion target is located in the image frame sequence. And if the first pixel displacement is smaller than the second value, determining the mth image frame as an end frame where the moving object is located in the image frame sequence.
The second method comprises the following steps: determining second pixel displacement of a first image block in the mth image frame and a second image block in the (m + 1) th image frame in the motion direction according to the registration result; if the second pixel displacement is larger than or equal to the first numerical value, determining the mth image frame as a starting frame of the moving object in the image frame sequence; and if the second pixel displacement is smaller than the second numerical value, determining that the mth image frame is an end frame where the moving target is located in the image frame sequence, wherein the first image block and the second image block are matched image block pairs.
In implementation, if the registration result is that the registration is successful, the matched image block pair may be used to determine a second pixel displacement of the first image block and the second image block in the matched image block pair in the motion direction, and if the second pixel displacement is greater than or equal to a first value, the mth image frame is determined to be the starting frame of the moving object in the image frame sequence. And if the second pixel displacement is smaller than the second numerical value, determining the mth image frame as an end frame where the moving object is located in the image frame sequence.
It should be noted that, the determination manner of the second pixel displacement of the first image block and the second image block in the motion direction may be: and if the motion direction is the y direction, directly subtracting the ordinate of a certain pixel point in the first image block from the ordinate of a pixel point corresponding to the pixel point in the second image block to obtain an absolute value, namely the second pixel displacement.
It should be further noted that, a certain pixel point in the first image block corresponds to a certain pixel point in the second image block, which refers to a pixel point describing the same content, for example, if the first image block is a pixel point at the upper left corner, the pixel point corresponding to the pixel point is a pixel point at the upper left corner in the second image block.
In addition, when the pixel displacement mentioned above is greater than or equal to the second value and less than the first value, it is indicated that the m-th image frame is neither the start frame nor the end frame.
It should be noted that, since the stitching process is not performed when the moving object is stationary, when the pixel displacement of the moving object is relatively small (i.e., smaller than the second value), the stitching process is not performed, and therefore the frame is ended, whereas when the pixel displacement of the moving object is relatively large (i.e., larger than the first value), the stitching process is performed, and therefore the frame is started.
In addition, the first and second values may be the same.
In addition, as shown in fig. 6, a flow chart for determining the start frame and the end frame is also provided:
step s1, registering successfully, and acquiring an image frame sequence;
a step S2 of determining a pixel displacement between the mth image frame and the m +1 th image frame;
step S3, judging the relationship between the pixel displacement and the first and second values;
step s4, when the pixel displacement is larger than or equal to the first value, the m image frame is the initial frame;
at step s5, the mth image frame is the end frame when the pixel displacement is less than the first value.
Optionally, when performing the stitching processing, the image frames may be first converted into the same coordinate system, and then the stitching processing is performed, where the corresponding processing may be as follows:
converting each image frame from a starting frame to an ending frame in the image frame sequence to the same coordinate system according to the transformation matrix; and splicing the image frames from the starting frame to the ending frame in the image frame sequence based on the coordinates of the image frames from the starting frame to the ending frame in the same coordinate system.
In an implementation, the stitching device may convert each image frame from the start frame to the end frame in the sequence of image frames into the same coordinate system using the transformation matrix, where the coordinate system may be a rectangular coordinate system established on the surface of the moving object. As shown in fig. 7, the image frames are in the same rectangular coordinate system.
And then splicing the image frames together by using the coordinates of the image frames from the starting frame to the ending frame in the same coordinate system. As shown in fig. 8, a panoramic image is provided after stitching together the image frames shown in fig. 7.
In addition, if the splicing device is a device with a display screen, the spliced panoramic image can be displayed on the display screen.
In addition, as shown in fig. 9, for ease of understanding, a flowchart of the entire scheme is also provided:
step a1, carrying out registration processing;
step a2, judging whether the registration is successful;
step a3, when the registration is successful, using the registration result to determine a transformation matrix;
step a4, when the registration is unsuccessful, carrying out motion estimation, and determining a transformation matrix by using the result of the motion estimation;
and step a5, performing image splicing processing.
In the embodiment of the disclosure, when image splicing is performed, an image frame sequence shot by a target camera may be obtained, according to image contents of image frames in the image frame sequence, if it is determined that registration of an mth image frame and an m + k image frame is successful, a transformation matrix between the mth image frame and the m + k image frame is determined according to a registration result of the mth image frame and the m + k image frame, where m and k are positive integers, then according to the registration result, a start frame and an end frame where a moving target is located are determined in the image frame sequence, and according to the transformation matrix, each image frame from the start frame to the end frame in the image frame sequence is spliced. Therefore, image splicing can be performed without target detection, and the efficiency of image splicing can be improved.
Based on the same technical concept, the embodiment of the present disclosure further provides an image stitching apparatus, as shown in fig. 10, the apparatus including:
an obtaining module 1010, configured to obtain an image frame sequence captured by a target camera;
a determining module 1020 for:
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are successfully registered, determining a transformation matrix between the m-th image frame and the m + k-th image frame according to the registration result of the m-th image frame and the m + k-th image frame, wherein m and k are positive integers; according to the registration result, determining a starting frame and an ending frame where a moving object is located in the image frame sequence
A splicing module 1030, configured to splice, according to the transformation matrix, image frames from the start frame to the end frame in the image frame sequence.
Optionally, the determining module 1020 is configured to:
determining a first pixel displacement of a first feature point in the mth image frame and a second feature point in the m +1 image frame in a motion direction according to the registration result; if the first pixel displacement is larger than or equal to a first numerical value, determining the mth image frame as a starting frame of a moving target in the image frame sequence, and if the first pixel displacement is smaller than a second numerical value, determining the mth image frame as a starting frame of the moving target in the image frame sequence, wherein the first characteristic point and the second characteristic point belong to matched characteristic point pairs; alternatively, the first and second electrodes may be,
determining second pixel displacement of a first image block in the mth image frame and a second image block in the (m + 1) th image frame in the motion direction according to the registration result; if the second pixel displacement is larger than or equal to the first numerical value, determining the m-th image frame as a starting frame of a moving object in the image frame sequence; and if the second pixel displacement is smaller than the second value, determining that the mth image frame is an end frame where a moving object is located in the image frame sequence, wherein the first image block and the second image block are matched image block pairs.
Optionally, the splicing module 1030 is configured to:
converting each image frame from the initial frame to the end frame in the image frame sequence to the same coordinate system according to the transformation matrix;
and splicing the image frames from the starting frame to the ending frame in the image frame sequence based on the coordinates of the image frames from the starting frame to the ending frame in the same coordinate system.
Optionally, the determining module 1020 is configured to:
and determining a transformation matrix between the mth image frame and the m + k image frame according to the matched feature point pairs or matched image block pairs of the mth image frame and the m + k image frame.
Optionally, the determining module 1020 is further configured to:
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are determined to be failed in registration, motion estimation is carried out;
determining a transformation matrix between the m-th image frame and the m + k-th image frame based on a result of the motion estimation.
Optionally, the determining module 1020 is configured to:
determining an average acceleration of a moving object from the m-th image frame to the m + k-th image frame;
the determining module is configured to:
determining a transformation matrix between the m-th image frame and the m + k-th image frame based on the average acceleration.
In the embodiment of the disclosure, when image splicing is performed, an image frame sequence shot by a target camera may be obtained, according to image contents of image frames in the image frame sequence, if it is determined that registration of an mth image frame and an m + k image frame is successful, a transformation matrix between the mth image frame and the m + k image frame is determined according to a registration result of the mth image frame and the m + k image frame, where m and k are positive integers, then according to the registration result, a start frame and an end frame where a moving target is located are determined in the image frame sequence, and according to the transformation matrix, each image frame from the start frame to the end frame in the image frame sequence is spliced. Therefore, image splicing can be performed without target detection, and the efficiency of image splicing can be improved.
It should be noted that: in the image stitching apparatus provided in the above embodiment, when stitching images, only the division of the above functional modules is used as an example, in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules, so as to complete all or part of the above described functions. In addition, the image stitching device and the image stitching method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments in detail and are not described herein again.
Fig. 11 is a schematic structural diagram of a splicing apparatus 1100 according to an embodiment of the present invention, where the splicing apparatus 1100 may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 1101 and one or more memories 1102, where the memory 1102 stores at least one instruction, and the at least one instruction is loaded and executed by the processors 1101 to implement the method steps for determining image splicing described above.
The embodiment of the present disclosure further provides a splicing apparatus, where the splicing apparatus includes a processor and a memory, where the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the image splicing method described above.
The disclosed embodiments also provide a computer-readable storage medium having stored therein at least one instruction, which is loaded and executed by a processor to implement the method of image stitching as described above.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only exemplary of the present disclosure and is not intended to limit the present disclosure, so that any modification, equivalent replacement, or improvement made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (12)
1. A method of image stitching, the method comprising:
acquiring an image frame sequence shot by a target camera;
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are successfully registered, determining a transformation matrix between the m-th image frame and the m + k-th image frame according to the registration result of the m-th image frame and the m + k-th image frame, wherein m and k are positive integers;
according to the registration result, determining a starting frame and an ending frame where a moving object is located in the image frame sequence;
and splicing each image frame from the starting frame to the ending frame in the image frame sequence according to the transformation matrix.
2. The method according to claim 1, wherein determining a start frame and an end frame of a moving object in the sequence of image frames according to the registration result comprises:
determining a first pixel displacement of a first feature point in the mth image frame and a second feature point in the m +1 image frame in a motion direction according to the registration result; if the first pixel displacement is larger than or equal to a first numerical value, determining the mth image frame as a starting frame of a moving target in the image frame sequence, and if the first pixel displacement is smaller than a second numerical value, determining the mth image frame as a starting frame of the moving target in the image frame sequence, wherein the first characteristic point and the second characteristic point belong to matched characteristic point pairs; alternatively, the first and second electrodes may be,
determining second pixel displacement of a first image block in the mth image frame and a second image block in the (m + 1) th image frame in the motion direction according to the registration result; if the second pixel displacement is larger than or equal to the first numerical value, determining the m-th image frame as a starting frame of a moving object in the image frame sequence; and if the second pixel displacement is smaller than the second value, determining that the mth image frame is an end frame where a moving object is located in the image frame sequence, wherein the first image block and the second image block are matched image block pairs.
3. The method of claim 1, wherein said stitching image frames from said start frame to said end frame in said sequence of image frames according to said transformation matrix comprises:
converting each image frame from the initial frame to the end frame in the image frame sequence to the same coordinate system according to the transformation matrix;
and splicing the image frames from the starting frame to the ending frame in the image frame sequence based on the coordinates of the image frames from the starting frame to the ending frame in the same coordinate system.
4. The method of claim 1, wherein the determining a transformation matrix between the mth image frame and the m + k image frame according to the registration result of the mth image frame and the m + k image frame comprises:
and determining a transformation matrix between the mth image frame and the m + k image frame according to the matched feature point pairs or matched image block pairs of the mth image frame and the m + k image frame.
5. The method of any of claims 1 to 4, further comprising:
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are determined to be failed in registration, motion estimation is carried out;
determining a transformation matrix between the m-th image frame and the m + k-th image frame based on a result of the motion estimation.
6. The method of claim 5, wherein the performing motion estimation comprises:
determining an average acceleration of a moving object from the m-th image frame to the m + k-th image frame;
the determining a transformation matrix between the m-th image frame and the m + k-th image frame based on the result of the motion estimation includes:
determining a transformation matrix between the m-th image frame and the m + k-th image frame based on the average acceleration.
7. An apparatus for image stitching, the apparatus comprising:
the acquisition module is used for acquiring an image frame sequence shot by the target camera;
a determination module to:
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are successfully registered, determining a transformation matrix between the m-th image frame and the m + k-th image frame according to the registration result of the m-th image frame and the m + k-th image frame, wherein m and k are positive integers; determining a starting frame and an ending frame where a moving object is located in the image frame sequence;
and the splicing module is used for splicing each image frame from the starting frame to the ending frame in the image frame sequence according to the transformation matrix.
8. The apparatus of claim 7, wherein the determining module is configured to:
determining a first pixel displacement of a first feature point in the mth image frame and a second feature point in the m +1 image frame in a motion direction according to the registration result; if the first pixel displacement is larger than or equal to a first numerical value, determining the mth image frame as a starting frame of a moving target in the image frame sequence, and if the first pixel displacement is smaller than a second numerical value, determining the mth image frame as a starting frame of the moving target in the image frame sequence, wherein the first characteristic point and the second characteristic point belong to matched characteristic point pairs; alternatively, the first and second electrodes may be,
determining second pixel displacement of a first image block in the mth image frame and a second image block in the (m + 1) th image frame in the motion direction according to the registration result; if the second pixel displacement is larger than or equal to the first numerical value, determining the m-th image frame as a starting frame of a moving object in the image frame sequence; and if the second pixel displacement is smaller than the second value, determining that the mth image frame is an end frame where a moving object is located in the image frame sequence, wherein the first image block and the second image block are matched image block pairs.
9. The apparatus of claim 7, wherein the splicing module is configured to:
converting each image frame from the initial frame to the end frame in the image frame sequence to the same coordinate system according to the transformation matrix;
and splicing the image frames from the starting frame to the ending frame in the image frame sequence based on the coordinates of the image frames from the starting frame to the ending frame in the same coordinate system.
10. The apparatus of claim 7, wherein the determining module is configured to:
and determining a transformation matrix between the mth image frame and the m + k image frame according to the matched feature point pairs or matched image block pairs of the mth image frame and the m + k image frame.
11. The apparatus of any of claims 7 to 10, wherein the determining module is further configured to:
according to the image content of each image frame in the image frame sequence, if the m-th image frame and the m + k-th image frame are determined to be failed in registration, motion estimation is carried out;
determining a transformation matrix between the m-th image frame and the m + k-th image frame based on a result of the motion estimation.
12. The apparatus of claim 11, wherein the determining module is configured to:
determining an average acceleration of a moving object from the m-th image frame to the m + k-th image frame;
the determining module is configured to:
determining a transformation matrix between the m-th image frame and the m + k-th image frame based on the average acceleration.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910600184.8A CN112184544B (en) | 2019-07-04 | 2019-07-04 | Image stitching method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910600184.8A CN112184544B (en) | 2019-07-04 | 2019-07-04 | Image stitching method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112184544A true CN112184544A (en) | 2021-01-05 |
| CN112184544B CN112184544B (en) | 2023-08-04 |
Family
ID=73915145
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910600184.8A Active CN112184544B (en) | 2019-07-04 | 2019-07-04 | Image stitching method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112184544B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112991180A (en) * | 2021-03-25 | 2021-06-18 | 北京百度网讯科技有限公司 | Image splicing method, device, equipment and storage medium |
| CN115272085A (en) * | 2022-09-28 | 2022-11-01 | 北京闪马智建科技有限公司 | Panoramic image determination method and device, storage medium and electronic device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060239571A1 (en) * | 2005-03-29 | 2006-10-26 | Shenzhen Mindray Bio-Medical Electronics Co., Ltd. | Method of volume-panorama imaging processing |
| US20070031062A1 (en) * | 2005-08-04 | 2007-02-08 | Microsoft Corporation | Video registration and image sequence stitching |
| CN108447022A (en) * | 2018-03-20 | 2018-08-24 | 北京天睿空间科技股份有限公司 | Moving target joining method based on single fixing camera image sequence |
-
2019
- 2019-07-04 CN CN201910600184.8A patent/CN112184544B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060239571A1 (en) * | 2005-03-29 | 2006-10-26 | Shenzhen Mindray Bio-Medical Electronics Co., Ltd. | Method of volume-panorama imaging processing |
| US20070031062A1 (en) * | 2005-08-04 | 2007-02-08 | Microsoft Corporation | Video registration and image sequence stitching |
| CN108447022A (en) * | 2018-03-20 | 2018-08-24 | 北京天睿空间科技股份有限公司 | Moving target joining method based on single fixing camera image sequence |
Non-Patent Citations (1)
| Title |
|---|
| 雷飞等: "基于改进SURF的实时视频拼接方法", 计算机技术与发展 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112991180A (en) * | 2021-03-25 | 2021-06-18 | 北京百度网讯科技有限公司 | Image splicing method, device, equipment and storage medium |
| CN112991180B (en) * | 2021-03-25 | 2023-10-31 | 北京百度网讯科技有限公司 | Image stitching method, device, equipment and storage medium |
| CN115272085A (en) * | 2022-09-28 | 2022-11-01 | 北京闪马智建科技有限公司 | Panoramic image determination method and device, storage medium and electronic device |
| CN115272085B (en) * | 2022-09-28 | 2023-09-22 | 北京闪马智建科技有限公司 | Panoramic image determining method and device, storage medium and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112184544B (en) | 2023-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111145238B (en) | Three-dimensional reconstruction method and device for monocular endoscopic image and terminal equipment | |
| EP3028252B1 (en) | Rolling sequential bundle adjustment | |
| EP3680808A1 (en) | Augmented reality scene processing method and apparatus, and computer storage medium | |
| US10410089B2 (en) | Training assistance using synthetic images | |
| JP6338595B2 (en) | Mobile device based text detection and tracking | |
| US8792727B2 (en) | Image processing device, image processing method, and program | |
| CN108345821B (en) | Face tracking method and device | |
| CN111383252B (en) | Multi-camera target tracking method, system, device and storage medium | |
| US9838572B2 (en) | Method and device for determining movement between successive video images | |
| CN110909750A (en) | Image difference detection method and device, storage medium and terminal | |
| CN113190120B (en) | Pose acquisition method and device, electronic equipment and storage medium | |
| CN109635639B (en) | Method, device, equipment and storage medium for detecting position of traffic sign | |
| CN112184544B (en) | Image stitching method and device | |
| CN114742866A (en) | Image registration method and device, storage medium and electronic equipment | |
| CN111712857A (en) | Image processing method, device, holder and storage medium | |
| CN113592706A (en) | Method and device for adjusting homography matrix parameters | |
| CN112396634A (en) | Moving object detection method, moving object detection device, vehicle and storage medium | |
| CN110706257B (en) | Identification method of effective characteristic point pair, and camera state determination method and device | |
| CN113763466A (en) | Loop detection method and device, electronic equipment and storage medium | |
| CN109598675B (en) | Splicing method of multiple repeated texture images | |
| JP4578653B2 (en) | Depth image generation apparatus, depth image generation method, and computer-readable recording medium storing a program for causing a computer to execute the method | |
| CN115546312A (en) | Method and device for correcting external parameters of camera | |
| WO2018100230A1 (en) | Method and apparatuses for determining positions of multi-directional image capture apparatuses | |
| JP3452188B2 (en) | Tracking method of feature points in 2D video | |
| CN111860051A (en) | Vehicle-based loop detection method and device and vehicle-mounted terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |