CN117952826A

CN117952826A - Image stitching method, device, equipment, medium and program product

Info

Publication number: CN117952826A
Application number: CN202310489232.7A
Authority: CN
Inventors: 唐宗福; 王华君; 黄�良; 刘雨; 刘琼; 曾勇松
Original assignee: Shenzhen Yuto Packaging Technology Co Ltd
Current assignee: Shenzhen Yuto Packaging Technology Co Ltd
Priority date: 2023-04-28
Filing date: 2023-04-28
Publication date: 2024-04-30

Abstract

The embodiment of the application provides an image stitching method, an image stitching device, a medium and a program product, wherein the image stitching method comprises the following steps: extracting characteristic points of a reference image and an image to be spliced aiming at a video frame acquired by a camera array; determining world coordinates of each feature point based on the internal reference matrix and the external reference matrix calibrated by the cameras in the camera array; determining a homography transformation matrix of the image to be spliced relative to the reference image based on world coordinates of the feature points, the internal reference matrix and the calibrated external reference matrix; based on the homography transformation matrix, carrying out coordinate transformation on the images to be spliced; and finishing image stitching operation based on the optimal stitching seam of the target image to be stitched and the reference image to obtain a frame of video panoramic image. The method for determining the homography transformation matrix based on the internal and external parameters of the camera is realized, the dependence of image stitching on image characteristics is weakened, and the method is suitable for low-texture scenes.

Description

Image stitching method, device, equipment, medium and program product

Technical Field

The embodiment of the application relates to the technical field of image processing, in particular to an image stitching method, an image stitching device, a medium and a program product.

Background

The image stitching technique is a technique of stitching a plurality of images having overlapping areas into one large high-resolution image. By image stitching, a common camera can acquire a wider view angle, such as generating a 360-degree panoramic image.

Image stitching generally depends on extraction and matching of image features, and based on the image features of the overlapping area, a homography matrix between cameras is obtained through calculation, so that images acquired by other cameras are converted to an image coordinate system of a selected reference camera, and image stitching is completed.

However, the image stitching method depending on the image features cannot be applied to low-texture application scenes, and the application scenes are limited.

Disclosure of Invention

The embodiment of the application provides an image stitching method, an image stitching device, image stitching equipment, an image stitching medium and a program product, which realize the image stitching method based on camera parameters, weaken dependence on image characteristics and have wide application scenes.

In a first aspect, an embodiment of the present application provides an image stitching method, including:

Extracting characteristic points of the reference image and the image to be spliced according to video frames acquired by a camera array, wherein the camera array comprises a plurality of cameras with the same internal reference matrix; determining world coordinates of the feature points based on the internal reference matrix and an external reference matrix calibrated by cameras in the camera array; determining a homography transformation matrix of the image to be spliced relative to the reference image based on world coordinates of the feature points, the reference and the calibrated external parameter matrix; based on the homography transformation matrix, carrying out coordinate transformation on the images to be spliced to obtain a target image to be spliced; and finishing image stitching operation based on the optimal stitching joint of the target image to be stitched and the reference image.

Optionally, the reference image is acquired by a reference camera in the camera array, the stitched image is acquired by a target camera in the camera array, and the target camera is other cameras except the reference camera in the camera array;

The determining world coordinates of each feature point based on the internal reference matrix and the external reference matrix calibrated by the cameras in the camera array comprises the following steps:

And determining the world coordinates of each feature point based on the internal reference matrix, the external reference matrix calibrated by the reference camera and the external reference matrix calibrated by the target camera.

Optionally, the determining the homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the internal reference matrix and the calibrated external reference matrix includes:

determining feature point pairs matched with world coordinates in the image to be spliced and the reference image based on a nearest neighbor matching algorithm; and determining a homography transformation matrix of the image to be spliced relative to the reference image based on the image coordinates of each characteristic point in the characteristic point pair on the corresponding image and the internal reference matrix and the external reference matrix of the reference camera.

Optionally, the determining the homography transformation matrix of the image to be stitched relative to the reference image based on the image coordinates of each feature point in the pair of feature points on the corresponding image and the reference matrix and the external matrix of the reference camera includes:

screening the matched characteristic point pairs based on a random sampling consistency algorithm; determining a first conversion matrix based on the image coordinates of each feature point in the screened feature point pairs on the corresponding image; the homography transformation matrix is determined based on the first transformation matrix and the reference camera's reference matrix and reference matrix.

Optionally, after determining the homography transformation matrix of the image to be stitched relative to the reference image based on the world coordinates of the feature points, the internal reference matrix and the calibrated external reference matrix, the method includes:

For a video frame acquired subsequently by the camera array, carrying out coordinate transformation on the image to be spliced subsequently based on the homography transformation matrix to obtain a target image to be spliced subsequently; aiming at a last video frame acquired by the camera array, acquiring a first optimal stitching seam of a last target image to be stitched and a last reference image; judging whether a moving object exists in the splicing seam of the subsequent reference image and the subsequent target splicing image; if yes, executing the optimal splice joint updating operation; if not, taking the first optimal stitching seam as an optimal stitching seam to finish the image stitching operation of the subsequent reference image and the subsequent target stitching image.

Optionally, the determining whether the moving object exists in the stitching seam of the subsequent reference image and the subsequent target stitching image includes:

Acquiring gradient values of pixels of the last reference image and the last target spliced image at a spliced seam, and marking the gradient values as first gradient values; acquiring gradient values of pixels of a subsequent reference image and a subsequent target spliced image at a spliced seam, and marking the gradient values as second gradient values; and judging whether a moving object exists in the splicing seam of the subsequent reference image and the subsequent target splicing image according to the first gradient value and the second gradient value.

Optionally, determining whether a moving object exists in the stitching seam of the subsequent reference image and the subsequent target stitching image according to the first gradient value and the second gradient value includes:

Calculating gradient change values of pixels of the subsequent reference image and the subsequent target spliced image at the splicing seam based on the first gradient value and the second gradient value to obtain a gradient change set; ; selecting two gradient change values from the gradient change set as gradient change centers; calculating the distance between the gradient change value in the gradient change set and each gradient change center; dividing gradient change values in the gradient change set into two groups based on the distance; updating the gradient change center to be the average value of each group of gradient change values; judging whether gradient change centers before and after updating are the same or not; if the gradient change centers before and after updating are the same, judging whether the number of gradient change values in a group with larger average gradient change values is larger than a preset number or not; if the number of the gradient change values in the group with larger average gradient change value is larger than the preset number, determining that a moving object exists in the splicing seam of the subsequent reference image and the subsequent target splicing image; and if the gradient change centers before and after updating are different, returning to the step of calculating the distance between the gradient change value in the gradient change set and each gradient change center.

Optionally, the performing the best splice seam updating operation includes:

Searching the optimal splicing seam from the splicing seams of the subsequent reference image and the subsequent target splicing image again to obtain a second optimal splicing seam, and updating the optimal splicing seam into the second optimal splicing seam; and splicing the subsequent reference image and the subsequent target spliced image based on the second optimal spliced seam.

Optionally, after the image stitching operation is completed, the method further includes:

and displaying the video panorama obtained by the image stitching operation.

In a second aspect, an embodiment of the present application further provides an image stitching apparatus, including:

The characteristic point extraction module is used for extracting characteristic points of the reference image and the image to be spliced aiming at a video frame acquired by a camera array, wherein the camera array comprises a plurality of cameras with the same internal reference matrix; the world coordinate determining module is used for determining world coordinates of the feature points based on the internal reference matrix and the external reference matrix calibrated by the cameras in the camera array; the transformation matrix determining module is used for determining a homography transformation matrix of the image to be spliced relative to the reference image based on world coordinates of the feature points, the internal reference matrix and the calibrated external reference matrix; the image transformation module is used for carrying out coordinate transformation on the images to be spliced based on the homography transformation matrix to obtain target images to be spliced; and the splicing module is used for finishing image splicing operation based on the optimal splicing seam of the target image to be spliced and the reference image.

Optionally, the reference image is acquired by a reference camera in the camera array, the stitched image is acquired by a target camera in the camera array, and the target camera is other cameras except the reference camera in the camera array; the feature point extraction module is specifically configured to:

Optionally, the transformation matrix determining module includes:

The characteristic point pair determining unit is used for determining characteristic point pairs matched with world coordinates in the image to be spliced and the reference image based on a nearest neighbor matching algorithm; the transformation matrix determining unit is used for determining a homography transformation matrix of the image to be spliced relative to the reference image based on the image coordinates of each characteristic point in the characteristic point pair on the corresponding image and the reference matrix and the external reference matrix of the reference camera.

Optionally, the transformation matrix determining unit is specifically configured to:

Optionally, the apparatus further includes:

The subsequent image transformation module is used for carrying out coordinate transformation on the subsequent images to be spliced according to the homography transformation matrix aiming at the video frames acquired by the camera array subsequently to obtain the subsequent target images to be spliced; the history optimal stitching joint acquisition module is used for acquiring a first optimal stitching joint of a last target image to be stitched and a last reference image aiming at a last video frame acquired by the camera array; the splicing seam judging module is used for judging whether a moving object exists at the splicing seam of the subsequent reference image and the subsequent target splicing image; the first subsequent image stitching module is used for taking the first optimal stitching seam as an optimal stitching seam to finish the image stitching operation of the subsequent reference image and the subsequent target stitching image if the subsequent reference image and the subsequent target stitching image have no moving object at the stitching seam; and the second subsequent image stitching module is used for executing the optimal stitching seam updating operation if the subsequent reference image and the subsequent target stitching image have moving objects at the stitching seam.

Optionally, the splice joint judging module is specifically configured to:

calculating gradient change values of pixels of the subsequent reference image and the subsequent target spliced image at the splicing seam based on the first gradient value and the second gradient value to obtain a gradient change set; selecting two gradient change values from the gradient change set as gradient change centers; calculating the distance between the gradient change value in the gradient change set and each gradient change center; dividing gradient change values in the gradient change set into two groups based on the distance; updating the gradient change center to be the average value of each group of gradient change values; judging whether gradient change centers before and after updating are the same or not; if the gradient change centers before and after updating are the same, judging whether the number of gradient change values in a group with larger gradient change value average value is larger than a preset number or not, and determining that a moving object exists in a splicing seam of a subsequent reference image and a subsequent target splicing image when the number of gradient change values in a group with larger gradient change value average value is larger than the preset number; and if the gradient change centers before and after updating are different, returning to the step of calculating the distance between the gradient change value in the gradient change set and each gradient change center.

Optionally, the second subsequent image stitching module is specifically configured to:

If the subsequent reference image and the subsequent target spliced image have moving objects at the spliced seams, searching the optimal spliced seams from the spliced seams of the subsequent reference image and the subsequent target spliced image again to obtain second optimal spliced seams, and updating the optimal spliced seams into the second optimal spliced seams; and splicing the subsequent reference image and the subsequent target spliced image based on the second optimal spliced seam.

Optionally, the apparatus further includes:

and the panorama display module is used for displaying the video panorama obtained by the image stitching operation after the image stitching operation is completed.

In a third aspect, an embodiment of the present application further provides an image stitching device, including a memory and at least one processor; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the image stitching method provided in any embodiment of the present application.

In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, where computer executable instructions are stored, and when a processor executes the computer executable instructions, the image stitching method provided in any embodiment of the present application is implemented.

In a fifth aspect, embodiments of the present application further provide a computer program product, including a computer program, which when executed by a processor implements an image stitching method as provided in any embodiment of the present application.

The embodiment of the application provides an image stitching method, a device, equipment, a medium and a program product, which aim at stitching a panoramic image of a video frame acquired by a camera array, and provide a method for determining a homography transformation matrix based on internal parameters and external parameters of the camera, in particular to determine world coordinates of feature points on an extracted reference image and an image to be stitched based on an internal reference matrix and a calibrated external reference matrix of the camera, and further determine the homography transformation matrix between the camera acquiring the image to be stitched and the camera acquiring the reference image based on the world coordinates of the feature points, the internal reference matrix and the external reference matrix of the camera so as to transform the image to be stitched into the same image coordinate system with the reference image through the homography transformation matrix, thereby providing a foundation for image stitching; and then, the best stitching seam is searched to realize the stitching of the reference image and the image to be stitched, and after the images of the same video frame acquired by each camera in the camera array are stitched, a frame of video panorama is obtained. Compared with the mode of carrying out homography transformation matrix calculation by relying on image feature matching, the mode of combining camera internal and external parameters, provided by the application, has high accuracy, can be better suitable for scenes with low textures or fewer feature points, improves the quality of image stitching in low-texture scenes, and reduces ghosts or artifacts in the stitched panoramic images.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

Fig. 1 is a schematic diagram of an application scenario provided in an embodiment of the present application;

Fig. 2 is a flowchart of an image stitching method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a camera array according to an embodiment of the present application;

FIG. 4 is a flowchart illustrating step S202 in the embodiment of FIG. 2 according to the present application;

fig. 5 is a flowchart of an image stitching method according to another embodiment of the present application;

FIG. 6 is a flowchart illustrating step S502 in the embodiment of FIG. 5 according to the present application;

Fig. 7 is a flowchart of an image stitching method according to another embodiment of the present application;

Fig. 8 is a schematic structural diagram of an image stitching device according to an embodiment of the present application;

Fig. 9 is a schematic structural diagram of an image stitching device according to an embodiment of the present application.

Specific embodiments of the present application have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.

First, the terms related to the present application will be explained:

World coordinate system: also referred to as a reference coordinate system, is a three-dimensional coordinate system used to represent the position of an object in the real world.

Camera coordinate system: is a three-dimensional coordinate system for describing the position of an object from a camera angle, the coordinate system taking the optical center of the camera as the origin of the coordinate system and the optical axis of the camera as one of the coordinate axes.

Image coordinate system: is a two-dimensional coordinate system used to describe the coordinate system of an object imaged at the focal length of the camera, with the intersection of the camera optical axis and the imaging surface as the origin, typically the center of the image.

Splice joint: overlapping area when the two pictures are newly spliced.

Optimal splice joint: and determining a curve with the smallest difference in the new overlapping area of the two images according to a set rule, and taking the curve as the new splicing position of the two images.

The following describes the technical scheme of the present application and how the technical scheme of the present application solves the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

The following explains the application scenario of the embodiment of the present application:

In practical applications, high resolution images with a wide field of view and even panoramic are often used, such as monitoring applications, medical imaging fields, etc. Because wide-angle cameras are expensive, the panoramic image is generally acquired by combining a camera array with an image stitching technology.

Fig. 1 is a schematic diagram of an application scenario provided in an embodiment of the present application, where, as shown in fig. 1, a camera array includes N cameras, and an arrangement manner of the N cameras is a static array arrangement, that is, a relative positional relationship between the cameras is fixed. The camera array collects video through cameras of the cameras to obtain a sequence of video frames, wherein each video frame comprises an image collected by N cameras at corresponding moments.

In some embodiments, images or video frames acquired by different cameras in the camera array may be presented by different displays.

In other embodiments, N images of the same video frame corresponding to the camera array may be stitched into a panoramic image by using an image stitching technique, and recorded as a video panoramic image, and the video panoramic image is displayed by using a display, so as to perform environmental monitoring, image analysis, and the like through the panoramic image.

When image stitching is carried out, the method is generally divided into two stages, wherein the first stage is mainly responsible for calculating a homography transformation matrix to obtain a stitching template; and in the second stage, splicing of the subsequent video frames and updating of the splicing template are carried out based on the splicing template.

In the related art, in the first stage, the transformation of the images of the remaining cameras of the camera array onto the image coordinate system of the selected reference camera is typically achieved by extracting features of the overlapping areas of the images, and calculating a homography transformation matrix by means of feature matching. And then the panorama splicing is realized by searching the optimal splicing seam and the multiband fusion mode.

The image stitching mode is suitable for scenes with easily extracted image features and more image details, and for scenes with difficult extraction of image features, such as low-texture scenes, the accuracy of homography transformation matrix calculation is low, so that double images or artifacts appear in the panoramic image.

Aiming at panorama stitching in a low-texture application scene, the application provides an image stitching method, which realizes a homography transformation matrix calculation method based on internal and external parameters of a camera, weakens dependence on image characteristics, improves accuracy of homography transformation matrix calculation, and improves quality of a panorama in the low-texture scene.

Fig. 2 is a flowchart of an image stitching method according to an embodiment of the present application. The image stitching method may be executed by a device having a corresponding data processing capability, such as an image stitching device, as shown in fig. 2, and the image stitching method provided in this embodiment includes the following steps:

step S201, extracting feature points of the reference image and the image to be stitched for the video frame collected by the camera array.

The reference image and the image to be spliced are video frames acquired by a camera array, such as a first video frame or a designated video frame, the camera array comprises a plurality of cameras with the same internal reference matrix, the reference image is acquired by a reference camera in the camera array, the image to be spliced is acquired by a target camera in the camera array, and the target camera is other cameras except the reference camera of the camera array.

In some embodiments, the camera array may be arranged in a matrix.

In other embodiments, the camera array may be arranged in an annular configuration to surround the photographed target scene, enabling 360 ° panoramic photographing.

The reference camera can be any one camera in the camera array, the image collected by the reference camera is recorded as a reference image, and the images collected by the rest cameras in the camera array are images to be spliced.

Adjacent cameras in the camera array have areas of overlapping fields of view such that images acquired by the adjacent cameras include overlapping areas. The reference image and the image to be stitched have an image of an overlapping area, such as may be acquired for an adjacent camera.

In some embodiments, the cameras in the camera array may be of the same model and have the same internal parameters, including parameters related to the characteristics of the camera itself, such as focal length, pixel size, etc.

For each video frame collected by the camera array, an image collected by a reference camera in the video frame is taken as a reference image, an image collected by an adjacent camera of the reference camera is taken as an image to be spliced, or an image collected by a camera with an overlapping range with the visual field range of the reference camera is taken as an image to be spliced.

For example, fig. 3 is a schematic diagram of a camera array according to an embodiment of the present application, as shown in fig. 3, in which in fig. 3, the camera array includes 4 cameras, and the first image stitching may be performed by using the camera C1 as a reference camera and using the cameras C2 and C3 as target cameras, and the images collected by the cameras C1 to C3 are stitched together, then the second image stitching may be performed by using the camera C3 as a reference camera and using the camera C4 as a target camera, so as to stitch the images collected by the cameras C1 to C4 together. If the visual field ranges of the cameras are wider, and the overlapping ranges exist in the visual field ranges of the cameras C1 to C4, one of the cameras C1 to C4 can be used as a reference camera, and the other cameras can be used as target cameras for performing image stitching once.

Specifically, feature points of images (including a reference image and each image to be stitched) acquired by a camera in the camera array may be extracted based on a SURF (Speeded Up Robust Features, accelerated robust feature) algorithm, a SIFT (Scale-INVARIANT FEATURES TRANSFORM, scale-invariant feature transform) algorithm, or other feature extraction algorithms.

Step S202, determining world coordinates of each feature point based on the internal reference matrix and the external reference matrix calibrated by the cameras in the camera array.

The camera's internal matrix is used to represent the mapping of the camera coordinate system to the image coordinate system, and the external matrix includes a rotation matrix and a translation matrix used to represent the mapping of the camera coordinate system to the world coordinate system. Through the internal reference matrix and the external reference matrix of the camera, the feature points on the image (under the image coordinate system) acquired by the camera can be converted into points under the world coordinate system, so as to obtain the world coordinates of the feature points.

The external matrix of cameras in the camera array may be calibrated in any manner, and the application is not limited in this regard. And calibrating the external parameter matrix of the camera based on the position of the characteristic points on the calibration plate in the image, such as by shooting the calibration plate.

Optionally, the determining the world coordinates of each feature point based on the internal reference matrix and the external reference matrix calibrated by the cameras in the camera array includes:

And step S203, determining a homography transformation matrix of the image to be spliced relative to the reference image based on the world coordinates of the feature points, the internal reference matrix and the calibrated external reference matrix.

Specifically, the mapping relationship of the two images under the world coordinate system can be obtained based on the world coordinates of the feature points in the two images (the reference image and one of the images to be spliced), and then the mapping relationship under the world coordinate system is converted into the mapping relationship under the image coordinate system by combining the reference matrix and the external reference matrix of the reference camera, so that the mapping relationship between the two images is obtained, and the homography transformation matrix is obtained.

Further, matching feature point pairs in the two images can be found based on world coordinates, and the feature points are overlapped as the feature points of the same world coordinates; and calculating the mapping relation of the two images in the world coordinate system based on the coordinates of the feature point pairs in the image coordinate system.

The feature point pair comprises two feature points, one is located on the reference image, and the other is located on the image to be spliced.

Optionally, determining a homography transformation matrix of the image to be stitched relative to the reference image based on world coordinates of the feature points, the internal reference matrix and the calibrated external reference matrix includes:

Searching feature points matched with feature points of the reference image from the feature points of the image to be spliced through a Nearest Neighbor matching algorithm (KNN, K-Nearest Neighbor), obtaining a plurality of feature point pairs, and determining matched feature point pairs from the plurality of feature point pairs, such as feature point pairs with the same world coordinates or feature point pairs with close world coordinates. And determining a homography transformation matrix of the target camera relative to the reference camera based on the image coordinates of the matched feature points on the corresponding image of each feature point, and the internal reference matrix and the external reference matrix of the reference camera.

The characteristic points are matched through the KNN algorithm, so that the matching accuracy is high, the searching accuracy of the matched characteristic points is improved, and a reliable data base is provided for calculation of the homography transformation matrix.

And step S204, carrying out coordinate transformation on the images to be spliced based on the homography transformation matrix to obtain target images to be spliced.

And carrying out coordinate transformation on pixel points on each image to be spliced based on the calculated homography transformation matrix to obtain a target image to be spliced, wherein the target image to be spliced and the reference image are in the same image coordinate system.

Step S205, completing the image stitching operation based on the optimal stitching seam between the target image to be stitched and the reference image.

And splicing a group of reference images and the target images to be spliced into a frame of video panorama through image splicing operation.

Specifically, searching an optimal stitching seam between the target image to be stitched and the reference image, and stitching the reference image and the target image to be stitched based on the optimal stitching seam to obtain a frame of video panorama.

In order to optimize the image stitching effect, the stitching seam needs to be processed. The method specifically comprises the steps of searching an optimal stitching seam, segmenting a reference image and a target image to be stitched along the optimal stitching seam, stitching the segmented reference image and the segmented image to be stitched, and obtaining a video panorama corresponding to a video frame after images of the video frame acquired by each camera are stitched.

Since the reference camera and the target camera have overlapping visual fields, a stitching seam exists on the stitched image, and the stitching seam corresponds to the overlapping area of the two images and can be a polygonal area or an irregular area.

Specifically, an overlapping area of the two images can be determined based on pixel points corresponding to the same image coordinates on the target image to be spliced and the reference image, and an optimal splicing seam is searched in the overlapping area.

By way of example, the best splice seam may be found within the overlap region by defining an image energy function, by minimizing the image energy function. The image energy function may be a function of parameters related to color, gradient, etc.

Any algorithm may be used to search for and update the optimal splice, as the application is not limited in this regard.

The optimal stitching seam is a curve with the smallest difference between the reference image and the target image to be stitched in the overlapping area. By searching the optimal splicing seam, the phenomenon of the splicing seam caused by gray level difference and geometric dislocation of the two images can be reduced, and the image splicing quality is improved.

Optionally, after the image stitching operation is completed and a frame of video panorama is obtained, the method further includes:

And displaying the video panorama.

and displaying the video panorama obtained by the image stitching operation.

And sending the video panoramic images of the frames obtained by the background processing to a display for displaying so as to show the panoramic images to a user.

The embodiment of the application provides an image stitching method, which is used for stitching a panoramic image of a video frame acquired by a camera array, and provides a method for determining a homography transformation matrix based on internal parameters and external parameters of the camera, in particular to a method for determining world coordinates of feature points on an extracted reference image and an image to be stitched based on an internal parameter matrix and a calibrated external parameter matrix of the camera, and further determining the homography transformation matrix between the camera for acquiring the image to be stitched and the camera for acquiring the reference image based on the world coordinates of the feature points, the internal parameter matrix and the external parameter matrix of the camera, so as to transform the image to be stitched into a coordinate system of the same image as the reference image through the homography transformation matrix, thereby providing a foundation for image stitching; and then, the best stitching seam is searched to realize the stitching of the reference image and the image to be stitched, and after the images of the same video frame acquired by each camera in the camera array are stitched, a frame of video panorama is obtained. Compared with the mode of carrying out homography transformation matrix calculation by relying on image feature matching, the mode of combining camera internal and external parameters, provided by the application, has high accuracy, can be better suitable for scenes with low textures or fewer feature points, improves the quality of image stitching in low-texture scenes, and reduces ghosts or artifacts in the stitched panoramic images.

Optionally, fig. 4 is a schematic flow chart of step S202 in the embodiment of fig. 2, and as shown in fig. 4, step S202 specifically includes the following steps:

And step S401, determining feature point pairs matched with world coordinates in the image to be spliced and the reference image based on a nearest neighbor matching algorithm.

Step S402, screening the matched characteristic point pairs based on a random sampling coincidence algorithm.

The random sample consensus algorithm (RANdom SAmple Consensus, RANSAC) is an iterative algorithm that filters outliers or outliers (outliers) from a set of data. And obtaining the inner points (inliers) in the matched characteristic point pairs through screening based on a random sampling coincidence algorithm.

The RANSAC is used for screening the matched characteristic point pairs, so that abnormal values in the characteristic point pairs are filtered, the accuracy of the characteristic point pairs for calculating the mapping relation is improved, and the accuracy of the homography transformation matrix is further improved.

Step S403, determining a first transformation matrix based on the image coordinates of each feature point in the filtered feature point pair on the corresponding image.

And obtaining a first conversion matrix based on the mapping relation between the image coordinates of each characteristic point, namely the inner point, on the corresponding image in the screened characteristic point pairs.

Specifically, the mapping relation between the reference image and the image to be spliced under the world coordinate system can be obtained through least square calculation based on the image coordinates of the plurality of groups of screened inner points, and the first conversion matrix is obtained. One group of interior points corresponds to one feature point pair after screening.

Step S404, determining the homography transformation matrix based on the first transformation matrix, and the reference camera' S internal and external matrices.

Because the internal parameters of each camera in the camera array are the same, only translation and rotation transformation exists between the images acquired by the reference camera and the target camera in the camera array, and then the relation between the image to be spliced and the characteristic point pairs on the reference image under the camera coordinate system is as follows:

Wherein, (X _c1,Y_c1,Z_c1) is the coordinates of the feature point p1 on the reference image in the camera coordinate system; (X _c2,Y_c2,Z_c2) is the coordinates of the feature point p2 on the image to be spliced under the camera coordinate system; r _c is the rotation matrix between the reference camera and the target camera; m is a first conversion matrix; t _c is the translation matrix between the reference camera and the target camera; alpha and beta are angles at which the target camera rotates in the left-right and up-down directions relative to the reference camera, respectively; t ₁、t₂、t₃ is the distance the target camera moves relative to the reference camera in three directions of the world coordinate system XYZ, respectively. The feature point p1 and the feature point p2 are a set of feature pairs.

The relationship between the camera coordinate system and the world coordinate system is:

Wherein, (X _c1,Y_c1,Z_c1) is the coordinate of the feature point p1 on the reference image in the world coordinate system; r and T are rotation matrices and offset vectors between the camera coordinate system and the world coordinate system, respectively, and may be determined based on the reference camera's extrinsic matrices.

Substituting formula (4.4) into formula (4.1),

Substituting the coordinates of a plurality of groups of characteristic points corresponding to the reference image and the image to be spliced under the camera coordinate system and the world coordinate system into a matrix M' obtained by solving (4.5), and realizing the conversion from the camera coordinate system of the image to be spliced to the world coordinate system of the reference image.

The coordinates of the feature points in the camera coordinate system can be obtained by solving based on the internal reference matrix of the camera and the coordinates of the feature points in the image coordinate system. Based on the matrix M' and the internal reference matrix and the external reference matrix of the reference camera, a homography transformation matrix H can be obtained, and the transformation between the image to be spliced and the reference image is realized.

After the image stitching method provided by the embodiment of the invention is used for stitching the images of the target video frames acquired by the camera array to obtain a frame of video panorama, other video frames acquired by the camera array, such as subsequent videos, can be stitched according to the stitching templates obtained when the target video frames are stitched, including the homography transformation matrix and the optimal stitching seam, so as to stitch the images of other video frames.

The target video frame should preferably be one of the images that is more distinct in character, such as a video frame that contains an image of a calibration plate or other object that is more distinct.

In some embodiments, the target video frame may be the first frame acquired by the camera array, or a user-specified first frame.

In the test stage, images containing additional feature objects (such as calibration plates) can be captured by cameras in the camera array to obtain a plurality of test video frames, and one of the plurality of test video frames is designated as a target video frame. And obtaining a splicing template based on the target video frame, wherein the splicing template comprises a homography transformation matrix and an optimal splicing seam so as to facilitate the image splicing of the video frame acquired by the camera array subsequently.

For the video frames collected later, when in splicing, whether the optimal splicing seam needs to be updated or not needs to be judged. Specifically, whether the best splice joint needs to be updated is judged by judging whether a moving object exists in the splice joint. If no moving object exists, splicing according to the previous splicing template; if a moving object exists, the optimal splicing seam is required to be updated, and image splicing is carried out based on the homography transformation matrix and the updated optimal splicing seam.

Fig. 5 is a flowchart of an image stitching method according to another embodiment of the present application, where the image stitching method is directed to image stitching processing of a subsequently acquired video frame, that is, stitching processing of a subsequently acquired reference image and a subsequently to-be-stitched image, where the subsequently reference image is an image of a subsequently acquired frame acquired by a camera corresponding to the reference image, and the subsequently to-be-stitched image is an image of a subsequently acquired frame acquired by a camera corresponding to the to-be-stitched image.

As shown in fig. 5, the image stitching method provided in this embodiment may further include the following steps:

Step S501, performing coordinate transformation on the subsequent image to be stitched based on the homography transformation matrix for the video frame acquired subsequently by the camera array, to obtain the subsequent target image to be stitched.

The homography transformation matrix is obtained through step S203 and related steps, that is, is determined based on world coordinates of feature points of the reference image and the image to be spliced, an internal reference matrix of the camera in the camera array, and a calibrated external reference matrix.

The video frames collected later are the video frames collected by the camera array after the video frames adopted when the homography transformation matrix is determined, and the reference image, the image to be spliced and the target image to be spliced in the video frames collected later are recorded as the subsequent reference image, the subsequent image to be spliced and the subsequent target image to be spliced.

Step S502, for the last video frame acquired by the camera array, a first optimal stitching seam between the last target image to be stitched and the last reference image is acquired.

The last video frame is a video frame before the video frame collected later, and the reference image, the image to be spliced and the target image to be spliced in the last video frame are recorded as the last reference image, the last image to be spliced and the last target image to be spliced.

The last reference image is the last frame image acquired by the camera corresponding to the subsequent reference image, and the last target image to be spliced is the image acquired by the camera corresponding to the subsequent image to be spliced after coordinate transformation is performed on the last frame image based on the homography transformation matrix.

The first optimal stitching seam is the optimal stitching seam adopted when the last target image to be stitched is stitched with the last reference image.

Step S503, judging whether a moving object exists in the splicing seam of the subsequent reference image and the subsequent target splicing image. If not, executing step S504; if yes, step S505 is executed.

And step S504, taking the first optimal stitching seam as an optimal stitching seam, and completing the image stitching operation of the subsequent reference image and the subsequent target stitching image.

If no moving object exists at the splice joint, splicing the subsequent video frames directly based on the determined optimal splice joint.

Step S505, performing an optimal splice seam updating operation.

For a video frame acquired subsequently by the camera array, when a plurality of images corresponding to the video frame are spliced, the images to be spliced are subjected to coordinate transformation based on a homography transformation matrix determined previously, so that an overlapping area between the images to be spliced and a reference image, namely a splicing seam, is obtained, and the specific mode is similar to that of target frame processing and is not repeated here.

For the video frames collected later, judging whether a moving object exists at the splicing seam, if so, the previous optimal splicing seam cannot be used, and the optimal splicing seam needs to be searched again.

It is possible to detect whether or not a moving object is present at the splice line based on the condition of a change in the gradient of a pixel in an image at the splice line (determined by the difference in the gradient value of the pixel at the same position in the two images of the preceding and following frames). Specifically, whether a moving object exists or not can be judged based on the number of pixels with larger gradient change values at the positions of the splicing seams, and if the number of pixels with larger gradient change values at the positions of the splicing seams exceeds a set threshold value, the moving object is determined to exist, and the optimal splicing seams need to be updated.

In some embodiments, in order to improve accuracy of gradient change value determination, the gradient change value may be classified based on a K-means clustering algorithm to obtain a pixel set with fast gradient change, and if the number of pixels in the pixel set exceeds a set threshold, determining that a moving object exists at the splice seam.

Specifically, the optimal stitching seam between the subsequent reference image and the subsequent target image to be stitched can be searched again based on the position of the moving object and the characteristics of the color, texture, optical flow track and the like of the pixels at the stitching seam so as to avoid dislocation of the moving object.

Optionally, the performing the best splice seam updating operation includes:

For example, for camera a and camera B in the camera array, after the first frame images acquired by camera a and camera B obtain a homography transformation matrix and an optimal splice joint, the first frame images acquired by camera a and camera B are spliced; for the images acquired by the camera A and the camera B, such as a second frame image, performing image transformation on the homography-based transformation matrix to obtain an overlapping area or a splicing seam, and judging whether a moving object exists at the splicing seam; if not, splicing the second frame images acquired by the camera A and the camera B based on the optimal splicing seam obtained in the first frame; if yes, the best stitching operation is executed, and the second frame images acquired by the camera A and the camera B are stitched based on the updated best stitching operation.

In this embodiment, for the subsequent video frames, the previously determined stitching template can be directly used in image stitching, and the overlapping area is not required to be determined based on the image features, so that the method and the device can be better adapted to the low-texture application scene. When a moving object exists at the splicing seam of two pictures of the subsequent video frame, updating of the optimal splicing seam is dynamically carried out, so that the image splicing quality is improved, and the phenomenon of splicing dislocation of the moving object is reduced.

Optionally, fig. 6 is a schematic flow chart of step S503 in the embodiment of fig. 5, and as shown in fig. 6, step S502 specifically includes the following steps:

Step S601, obtaining gradient values of pixels at a stitching seam of the previous reference image and the previous target stitching image, and marking the gradient values as first gradient values.

Step S602, obtaining gradient values of pixels at a splicing seam of a subsequent reference image and a subsequent target splicing image, and marking the gradient values as second gradient values.

Step S603, calculating gradient change values of each pixel of the subsequent reference image and the subsequent target stitched image at the stitching seam based on the first gradient value and the second gradient value, so as to obtain a gradient change set.

In step S604, two gradient change values are selected from the gradient change set as gradient change centers.

2 Gradient change values can be arbitrarily selected from the gradient change set S _t as initial gradient change centers. The gradient change set S _t represents a set of gradient change values of the video frame acquired at time t.

The gradient change value S _it of the pixel p _i in the gradient change set S _t can be defined as:

Wherein g _i0 is the gradient of pixel p _i in the first video frame (first frame) or the target video frame; g _it is the gradient of the middle pixel p _i of the video frame acquired at time t.

When there is a moving object in the gradient change set S _t, the gradient change values in the gradient change set S _t may be classified into two types, i.e., a fast gradient change type and a slow gradient change type, in which the average value of the gradient change values of the fast gradient change type is large.

Step S605 calculates the distance between the gradient change value in the gradient change set and each gradient change center.

For the first round of calculation distance, the step specifically comprises the step of calculating the distances between the rest gradient change values except the gradient change center in the gradient change set and each gradient change center.

The gradient includes two dimensions, such as gradients in the x and y directions. The corresponding gradient change values also correspond to two dimensions, and the distance between the gradient change values can be represented by the difference value of each dimension, such as Euclidean distance.

Step S606, based on the distance, the gradient change values in the gradient change set are divided into two groups.

Each gradient change value corresponds to two distances, which are the distances between the gradient change value and two gradient change centers, and the gradient change values can be divided into a group corresponding to the gradient center with the smallest distance.

Taking the gradient change centers as c1 and c2 as an example, for the gradient change value s _it of the pixel p _i, if the distance between the gradient change value s _it and the gradient change center c1 is smaller than that between the gradient change value s _it and the gradient change center c2, the gradient change value s _it is divided into a group where the gradient change center c1 is located.

The gradient change values in the gradient change set S _t are divided into two groups based on the smaller value in the distance.

In step S607, the gradient change center is updated as the average value of the gradient change values of each group.

Step S608, judging whether the gradient change centers before and after the update are the same; if yes, go to step S609; returning to step S605, calculating the distance between the gradient change value in the gradient change set and the updated gradient change center, and re-grouping the gradient change values and updating the gradient change center.

In step S609, it is determined whether the number of gradient change values in the group with larger average gradient change value is greater than the preset number. If yes, go to step S610; if not, step S611 is performed.

Wherein the preset number may be determined based on the total number of pixels at the splice line or a default value may be employed.

In some embodiments, the predetermined number is the product of the total number of pixels at the splice joint and a predetermined coefficient, the predetermined coefficient being located in the (0, 1) interval.

The preset coefficient may be, for example, 0.3.

Step S610, determining that a moving object exists in the splicing seam of the subsequent reference image and the subsequent target splicing image.

Step S611, determining that the moving object does not exist in the splicing seam of the subsequent reference image and the subsequent target splicing image.

In this embodiment, based on the distribution condition of gradient change values of each pixel point at the joint of the image, the pixel with a larger gradient change value is dynamically determined, so that when the moving object is too small in size, the moving object can still be effectively detected, and the accuracy of detecting the moving object is improved.

Fig. 7 is a flow chart of an image stitching method according to another embodiment of the present application, as shown in fig. 7, in which the image stitching method includes the following steps:

in step S701, a target frame acquired by the camera array is selected.

The target frame may be a first frame, or a user-specified video frame.

Step S702, executing an image stitching algorithm based on camera parameters to obtain a homography transformation matrix.

The method for determining the homography transformation matrix provided in the foregoing embodiment may be performed, for example, the method corresponding to step S201 to step S203 is not described herein.

Step S703, finding the optimal splice seam.

Step S704, transforming the image corresponding to the subsequent video frame acquired by the camera array based on the homography transformation matrix.

Step S705, classifying the gradient change values according to the gradient change values of the pixels at the splice points based on the K-means clustering algorithm.

Step S706, judging whether a moving object exists at the splice joint or not based on the classification result; if yes, go to step S707; if not, step S708 is performed.

Specifically, based on the number of gradient change values or the number of pixels in the class with faster gradient change in the classification result, whether a moving object exists at the splice joint is judged. If the number is larger than the preset number, determining that a moving object exists at the splice joint, otherwise, the moving object does not exist.

Step S707, updating the optimal splice seam.

Step S708, a video panorama is generated.

And splicing the images based on the optimal splicing seam or the updated optimal splicing seam to generate the video panoramic image.

Step S709, synthesizing the panoramic video.

And synthesizing the video panoramic images of each frame according to the time sequence to obtain the panoramic video.

Fig. 8 is a schematic structural diagram of an image stitching device according to an embodiment of the present application, as shown in fig. 8, the image stitching device includes: the feature point extraction module 810, the world coordinate determination module 820, the transformation matrix determination module 830, the image transformation module 840, and the stitching module 850.

The feature point extraction module 810 is configured to extract feature points of a reference image and an image to be stitched for a video frame acquired by a camera array, where the camera array includes a plurality of cameras with the same internal reference matrix; the world coordinate determining module 820 is configured to determine world coordinates of each of the feature points based on the internal reference matrix and the external reference matrix calibrated by the cameras in the camera array; the transformation matrix determining module 830 is configured to determine a homography transformation matrix of the image to be stitched relative to the reference image based on world coordinates of the feature points, the internal reference matrix and the calibrated external reference matrix; the image transformation module 840 is configured to perform coordinate transformation on the image to be stitched based on the homography transformation matrix; the stitching module 850 is configured to complete an image stitching operation based on an optimal stitching seam between the target image to be stitched and the reference image.

Optionally, the reference image is acquired by a reference camera in the camera array, the stitched image is acquired by a target camera in the camera array, and the target camera is other cameras except the reference camera in the camera array; the feature point extraction module 810 is specifically configured to:

Optionally, the transformation matrix determining module 830 includes:

Optionally, the apparatus further includes:

Optionally, the splice joint judging module is specifically configured to:

Optionally, the apparatus further includes:

The image splicing device provided by the embodiment of the application can execute the image splicing method provided by any embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method.

Fig. 9 is a schematic structural diagram of an image stitching device according to an embodiment of the present application, as shown in fig. 9, where the image stitching device includes: a memory 910 and at least one processor 920.

Wherein the memory 910 stores computer-executable instructions; at least one processor 920 executes computer-executable instructions stored in the memory, such that the at least one processor executes to implement the image stitching method provided by any embodiment of the present application.

Wherein the memory 910 and the processor 920 are connected by a bus 930.

The related description may be correspondingly understood by referring to the related description and effect corresponding to the steps provided by the related embodiment of the image stitching method of the present application, which are not repeated herein.

An embodiment of the present application provides a computer readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, implement the image stitching method provided by any of the embodiments of the present application.

The computer readable storage medium may be, among other things, ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

An embodiment of the present application provides a computer program product, including a computer program, which when executed by a processor implements the image stitching method provided in any embodiment of the present application.

The processor may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, abbreviated as CPU), a network processor (Network Processor, abbreviated as NP), and the like. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of modules is merely a logical function division, and there may be additional divisions of actual implementation, e.g., multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.

Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. An image stitching method, comprising:

Extracting characteristic points of a reference image and an image to be spliced aiming at a video frame acquired by a camera array, wherein the camera array comprises a plurality of cameras with the same internal reference matrix;

Determining world coordinates of the feature points based on the internal reference matrix and an external reference matrix calibrated by cameras in the camera array;

determining a homography transformation matrix of the image to be spliced relative to the reference image based on world coordinates of the feature points, the internal reference matrix and the calibrated external reference matrix;

based on the homography transformation matrix, carrying out coordinate transformation on the images to be spliced to obtain a target image to be spliced;

and finishing image stitching operation based on the optimal stitching joint of the target image to be stitched and the reference image.

2. The method of claim 1, wherein the reference image is acquired by a reference camera in the camera array, the stitched image is acquired by a target camera in the camera array, the target camera being a camera in the camera array other than the reference camera;

3. The method of claim 2, wherein determining a homography transformation matrix for the image to be stitched relative to the reference image based on the world coordinates of the feature points, the internal reference matrix, and the calibrated external reference matrix comprises:

Determining feature point pairs matched with world coordinates in the image to be spliced and the reference image based on a nearest neighbor matching algorithm;

And determining a homography transformation matrix of the image to be spliced relative to the reference image based on the image coordinates of each characteristic point in the characteristic point pair on the corresponding image and the internal reference matrix and the external reference matrix of the reference camera.

4. A method according to claim 3, wherein the determining a homography transformation matrix of the image to be stitched relative to the reference image based on the image coordinates of each feature point in the pair of feature points on the corresponding image, and the reference and external matrices of the reference camera, comprises:

Screening the matched characteristic point pairs based on a random sampling consistency algorithm;

Determining a first conversion matrix based on the image coordinates of each feature point in the screened feature point pairs on the corresponding image;

the homography transformation matrix is determined based on the first transformation matrix and the reference camera's reference matrix and reference matrix.

5. The method according to claim 1, characterized in that after determining a homography transformation matrix of the image to be stitched with respect to the reference image based on world coordinates of feature points, the internal reference matrix and the calibrated external reference matrix, it comprises:

For a video frame acquired subsequently by the camera array, carrying out coordinate transformation on the image to be spliced subsequently based on the homography transformation matrix to obtain a target image to be spliced subsequently;

Aiming at a last video frame acquired by the camera array, acquiring a first optimal stitching seam of a last target image to be stitched and a last reference image;

Judging whether a moving object exists in the splicing seam of the subsequent reference image and the subsequent target splicing image;

If yes, executing the optimal splice joint updating operation;

If not, taking the first optimal stitching seam as an optimal stitching seam to finish the image stitching operation of the subsequent reference image and the subsequent target stitching image.

6. The method of claim 5, wherein determining whether a moving object is present at the stitching joint for the subsequent reference image and the subsequent target stitched image comprises:

acquiring gradient values of pixels of the last reference image and the last target spliced image at a spliced seam, and marking the gradient values as first gradient values;

acquiring gradient values of pixels of a subsequent reference image and a subsequent target spliced image at a spliced seam, and marking the gradient values as second gradient values;

And judging whether a moving object exists in the splicing seam of the subsequent reference image and the subsequent target splicing image according to the first gradient value and the second gradient value.

7. The method of claim 6, wherein determining whether a moving object is present at the stitching joint for the subsequent reference image and the subsequent target stitched image based on the first gradient value and the second gradient value comprises:

Calculating gradient change values of pixels of the subsequent reference image and the subsequent target spliced image at the splicing seam based on the first gradient value and the second gradient value to obtain a gradient change set;

selecting two gradient change values from the gradient change set as gradient change centers;

calculating the distance between the gradient change value in the gradient change set and each gradient change center;

Dividing gradient change values in the gradient change set into two groups based on the distance;

Updating the gradient change center to be the average value of each group of gradient change values;

Judging whether gradient change centers before and after updating are the same or not;

If the gradient change centers before and after updating are the same, judging whether the number of gradient change values in a group with larger gradient change value average value is larger than a preset number or not, and determining that a moving object exists in a splicing seam of a subsequent reference image and a subsequent target splicing image when the number of gradient change values in a group with larger gradient change value average value is larger than the preset number;

And if the gradient change centers before and after updating are different, returning to the step of calculating the distance between the gradient change value in the gradient change set and each gradient change center.

8. The method of claim 5, wherein performing an optimal splice seam update operation comprises:

Searching the optimal splicing seam from the splicing seams of the subsequent reference image and the subsequent target splicing image again to obtain a second optimal splicing seam, and updating the optimal splicing seam into the second optimal splicing seam;

and splicing the subsequent reference image and the subsequent target spliced image based on the second optimal spliced seam.

9. The method of any of claims 1-8, wherein after completing the image stitching operation, the method further comprises:

and displaying the video panorama obtained by the image stitching operation.

10. An image stitching device, comprising:

the characteristic point extraction module is used for extracting characteristic points of the reference image and the image to be spliced aiming at a video frame acquired by the camera array, wherein the camera array comprises a plurality of cameras with the same internal reference matrix;

The world coordinate determining module is used for determining world coordinates of the feature points based on the internal reference matrix and the external reference matrix calibrated by the cameras in the camera array;

The transformation matrix determining module is used for determining a homography transformation matrix of the image to be spliced relative to the reference image based on world coordinates of the feature points, the internal reference matrix and the calibrated external reference matrix;

the image transformation module is used for carrying out coordinate transformation on the images to be spliced based on the homography transformation matrix to obtain target images to be spliced;

And the splicing module is used for finishing image splicing operation based on the optimal splicing seam of the target image to be spliced and the reference image.

11. An image stitching device, comprising: a memory and at least one processor;

The memory stores computer-executable instructions;

The at least one processor executing computer-executable instructions stored in the memory, causing the at least one processor to perform the image stitching method of any of claims 1-9.

12. A computer readable storage medium having stored therein computer executable instructions which, when executed by a processor, implement the image stitching method of any of claims 1-9.

13. A computer program product comprising a computer program which, when executed by a processor, implements the image stitching method according to any one of claims 1-9.