CN113744133A - Image splicing method, device and equipment and computer readable storage medium - Google Patents
Image splicing method, device and equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN113744133A CN113744133A CN202111069395.7A CN202111069395A CN113744133A CN 113744133 A CN113744133 A CN 113744133A CN 202111069395 A CN202111069395 A CN 202111069395A CN 113744133 A CN113744133 A CN 113744133A
- Authority
- CN
- China
- Prior art keywords
- image
- similarity
- window
- determining
- spliced
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000003860 storage Methods 0.000 title claims abstract description 12
- 238000004364 calculation method Methods 0.000 claims abstract description 20
- 230000004927 fusion Effects 0.000 claims description 20
- 238000004422 calculation algorithm Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 10
- 238000005520 cutting process Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 claims description 9
- 239000011159 matrix material Substances 0.000 abstract description 13
- 230000009466 transformation Effects 0.000 abstract description 7
- 238000012937 correction Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 7
- 238000001514 detection method Methods 0.000 description 4
- 230000010365 information processing Effects 0.000 description 4
- 230000007704 transition Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000003331 infrared imaging Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/14—Transformations for image registration, e.g. adjusting or mapping for alignment of images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The application discloses an image splicing method, an image splicing device, image splicing equipment and a computer readable storage medium, wherein the method comprises the following steps: acquiring a plurality of images, calculating the structural similarity of two adjacent images, and determining the size of an ROI (region of interest) according to the structural similarity; determining a search window in the ROI area of the spliced image, and determining a template window in the ROI area of the spliced image; calculating the similarity of the template window at the corresponding position in the search window, and determining the matching position of the template window in the search window according to the similarity; and determining the overlapping area of the spliced image and the spliced image according to the matching position, and fusing the overlapping area of the spliced image and the spliced image. According to the technical scheme, the ROI area is determined based on the image structure similarity, the similarity calculation of the template window and the search window is carried out based on the area so as to determine the matching position and determine the overlapping area, and operations such as transformation matrix calculation and the like are not needed, so that the complexity of image splicing and the proportion of the overlapping area can be reduced.
Description
Technical Field
The present application relates to the field of image stitching technologies, and in particular, to an image stitching method, an image stitching device, an image stitching apparatus, and a computer-readable storage medium.
Background
Image stitching refers to stitching two or more image series with overlapping regions into an image with a larger field of view by a certain method.
At present, a single camera is often adopted to realize image splicing based on a feature point method, and the method comprises the steps of feature point detection and description, feature point matching, transformation matrix calculation, image projection, image fusion and the like, wherein in order to ensure the accuracy of the transformation matrix calculation, the overlapping range between images is within the range of 30% -50%, and an overlarge overlapping rate causes more images to be needed during image splicing, and the information processing amount is increased.
In summary, how to reduce the complexity of image stitching and the proportion of the overlapping area between the images is a technical problem to be solved urgently by those skilled in the art.
Disclosure of Invention
In view of the above, an object of the present application is to provide an image stitching method, an image stitching device, an image stitching apparatus and a computer-readable storage medium, which are used to reduce the complexity of image stitching and the proportion of overlapping areas between images.
In order to achieve the above purpose, the present application provides the following technical solutions:
an image stitching method, comprising:
acquiring a plurality of images, calculating the structural similarity of two adjacent images, and determining the size of an ROI (region of interest) according to the structural similarity; wherein an overlapping area exists between two adjacent images;
determining a search window in the ROI area of the spliced image, and determining a template window in the ROI area of the spliced image;
calculating the similarity of the template window at the corresponding position in the search window, and determining the matching position of the template window in the search window according to the similarity;
and determining the overlapping area of the spliced image and the spliced image according to the matching position, and fusing the overlapping area of the spliced image and the spliced image.
Preferably, after determining the overlapping region of the stitched image and the stitched image according to the matching position, the method further includes:
calculating a first brightness mean value of an overlapping region in the spliced image, and calculating a second brightness mean value of the overlapping region in the spliced image;
and correcting the brightness of the spliced image according to the first brightness mean value and the second brightness mean value.
Preferably, the fusing the overlapped regions of the stitched image and the stitched image includes:
calculating an angle value corresponding to each pixel point according to the number of columns of each pixel point in the overlapping area;
and correspondingly calculating the pixel value of each pixel point in the fused overlapping area by using the pixel value of each pixel point in the overlapping area in the spliced image, the pixel value of each pixel point in the overlapping area in the spliced image and the trigonometric function value of the angle value corresponding to each pixel point.
Preferably, the calculating the similarity of the template window at the corresponding position in the search window includes:
and calculating the similarity of the template window at the corresponding position in the search window by utilizing a normalized product correlation template matching algorithm.
Preferably, the calculating the similarity of the template window at the corresponding position in the search window, and determining the matching position of the template window in the search window according to the similarity includes:
moving the template window in the search window according to a preset path and a preset step length by taking a preset point of the search window as an original point, and calculating the similarity of the corresponding position of the template window in the search window after the template window is moved each time;
and taking the corresponding position of the template window when the similarity pair is maximum as the matching position of the template window in the search window.
Preferably, a plurality of images are acquired, including:
and acquiring the image shot by the monocular camera rotating to each preset fixed point around the shaft.
Preferably, after fusing the overlapped region of the stitched image and the stitched image, the method further includes:
and cutting and/or scaling the size of the final splicing image obtained by fusion according to the size of the display screen, and displaying the final splicing image after cutting and/or scaling in the display screen.
An image stitching device, comprising:
the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a plurality of images, calculating the structural similarity of two adjacent images and determining the size of an ROI (region of interest) according to the structural similarity; wherein an overlapping area exists between two adjacent images;
the first determining module is used for determining a search window in the ROI area of the spliced image and determining a template window in the ROI area of the spliced image;
the first calculation module is used for calculating the similarity of the template window at the corresponding position in the search window and determining the matching position of the template window in the search window according to the similarity;
and the fusion module is used for determining the overlapped area of the spliced image and the spliced image according to the matching position and fusing the overlapped area of the spliced image and the spliced image.
An image stitching device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the image stitching method as claimed in any one of the above when executing the computer program.
A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the image stitching method as defined in any one of the preceding claims.
The application provides an image splicing method, an image splicing device, image splicing equipment and a computer readable storage medium, wherein the method comprises the following steps: acquiring a plurality of images, calculating the structural similarity of two adjacent images, and determining the size of an ROI (region of interest) according to the structural similarity; wherein, an overlapping area exists between two adjacent images; determining a search window in the ROI area of the spliced image, and determining a template window in the ROI area of the spliced image; calculating the similarity of the template window at the corresponding position in the search window, and determining the matching position of the template window in the search window according to the similarity; and determining the overlapping area of the spliced image and the spliced image according to the matching position, and fusing the overlapping area of the spliced image and the spliced image.
Compared with the existing image splicing based on the characteristic point method, the technical scheme disclosed by the application only needs to determine the size of the ROI according to the structural similarity of two adjacent images, determine the search window of the spliced image and the template window of the spliced image according to the ROI, calculate the similarity between the template window and the search window to determine the matching position of the template window in the search window, determine the overlapping area of the two images according to the matching position and fuse the overlapping area, and does not need to perform operations such as characteristic point detection, transformation matrix calculation and the like, so that the complexity of image splicing can be reduced, the requirement on the proportion of the overlapping area is reduced, the proportion of the overlapping area between the two adjacent images can be reduced to 6%, the number of the images required during image splicing and the information processing amount of image splicing are reduced, so as to improve the image splicing efficiency.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of an image stitching method according to an embodiment of the present application;
fig. 2 is a schematic diagram of two adjacent images before brightness correction according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram of two adjacent images obtained after brightness correction is performed on the two adjacent images in fig. 2;
fig. 4 is a schematic diagram of an infrared mosaic generated before luminance correction according to an embodiment of the present disclosure;
fig. 5 is a schematic diagram of an infrared mosaic generated after luminance correction according to an embodiment of the present application;
fig. 6 is a flowchart of an infrared panoramic image stitching method according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an image stitching apparatus according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an image stitching device according to an embodiment of the present application.
Detailed Description
At present, image splicing of a single camera based on a feature point method needs two adjacent images to have a large overlapping area to obtain a large amount of feature matching, reliability of directional parameter calculation is improved by utilizing a large amount of redundant observation, and errors are provided to guarantee quality of image splicing.
Therefore, the application provides an image stitching method, an image stitching device, image stitching equipment and a computer readable storage medium, which are used for reducing the proportion of overlapping areas between images required during image stitching, reducing the complexity of image stitching and improving the image stitching efficiency.
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, which shows a flowchart of an image stitching method provided in an embodiment of the present application, an image stitching method provided in an embodiment of the present application includes:
s11: acquiring a plurality of images, calculating the structural similarity of two adjacent images, and determining the size of an ROI (region of interest) according to the structural similarity; wherein, an overlapping area exists between two adjacent images.
In the application, a monocular camera with a holder can be used for shooting N images in a target area (specifically, an area needing image observation), and the N images are sent to a back-end server, wherein in the N images, an overlapping area exists between two adjacent images (the proportion of the overlapping area can be as small as 6%), so that image splicing is realized on the N images, and the reliability of image splicing is improved. If a panoramic image is to be obtained by image stitching, a monocular camera with a pan-tilt is used to capture a circle around a target area to obtain a plurality of images. The number of required cameras can be reduced by shooting images through the monocular camera with the holder, the image splicing cost is reduced, the complexity of the image splicing structure design is reduced, and the method is convenient to be applied to large-scale industrial processes. In addition, it should be noted that the image mentioned in the present application may specifically be an infrared image, that is, the infrared image may be spliced, so as to solve the contradiction between the resolution of the infrared image and the limitations of the infrared imaging device, such as narrow imaging range and low resolution, by splicing the infrared image. Of course, the image mentioned in the present application may also be a visible light image, and the present application does not limit the type of the image.
After acquiring the N images acquired by the monocular camera with the pan/tilt head, the back-end server may calculate Structural Similarity (SSIM) between all two adjacent images in the N images, and determine the size of a region of interest (ROI) according to the structural similarity between all two adjacent images, where the determined size of the ROI needs to be larger than the size of an overlapping region of the adjacent images, for example: if the size of the overlapping region is 50 pixels, the size of the ROI may be 60-120 pixels, and meanwhile, due to the influence of the structural accuracy, the size of the overlapping region of different images acquired by the camera at different positions may vary, specifically, assuming that the size of the overlapping region between the first image and the second image is 50 pixels, the size of the overlapping region between the second image and the third image may be 55 pixels, and the size of the overlapping region between the third image and the fourth image is 45 pixels, at this time, the size of the ROI may be selected to be 60-120 pixels, which ensures that even if the size of the overlapping region varies, the ROI still exists within the ROI, so as to ensure that subsequent template matching can be performed within the ROI, thereby facilitating to improve the speed and accuracy of image stitching.
The specific process of determining the size of the ROI region may be: after the structural similarity between two adjacent images is calculated, the ratio of the size of the corresponding ROI area determined based on the structural similarity of the two adjacent images to the whole image is larger than the structural similarity, if the structural precision of the monocular camera is higher, the degree that the ratio of the size of the corresponding ROI area to the whole image is larger than the structural similarity can be smaller, and if the structural precision of the monocular camera is lower, the degree that the ratio of the size of the corresponding ROI area to the whole image is larger than the structural similarity can be larger, so that the size change of an overlapped area caused by the structural deviation of the rotation of the camera is tolerated; after calculating the size of the ROI between all the two adjacent images, the sizes of the ROI corresponding to all the two adjacent images may be averaged to finally determine the size of the ROI, or the maximum ROI may be selected as the size of the finally determined ROI.
The determination of the ROI area can enable the subsequent template matching to be implemented in the ROI area, compared with the implementation in the whole image, the operation amount in the template matching process can be reduced, and meanwhile, the interference of other image contents outside the ROI area in the image on the template matching is convenient to reduce, so that the accuracy of image registration is convenient to increase.
S12: and determining a search window in the ROI area of the spliced image, and determining a template window in the ROI area of the spliced image.
After the size of the ROI area is determined, the ROI area may be selected from the stitched image (the previous image is used as the stitched image) and the stitched image (the next image is used as the stitched image), specifically, a portion of the stitched image and the stitched image, which includes the corresponding overlapping area, is selected as the ROI area, taking the case that the camera shoots from left to right and the back-end server stitches in a sequence from left to right, selecting a right portion of the stitched image as the ROI area, and selecting a left portion of the stitched image as the ROI area.
Then, a search window is determined in the ROI area of the stitched image, and a template window is determined in the ROI area of the stitched image. For the determination of the template window, a region with obvious features and high matching performance may be selected as the template window in the ROI region of the stitched image, so as to make the result of template matching more accurate, where the region with obvious features and high matching performance mentioned herein specifically refers to a region with high distinguishing performance compared to other regions in the ROI region of the stitched image, for example: compared with the sky, the buildings have high distinguishability, and for the spliced images, the whole ROI area can be used as a search window so as to improve the accuracy of template matching. Of course, the region with obvious features and high matching performance in the spliced image can be selected as a search window, so that the calculation amount of template matching is reduced, and the template matching efficiency is improved.
It should be noted that, for step S11 and step S12, after acquiring the multiple images, the back-end server may determine the size of the ROI, then determine the search window in the ROI of the stitched image in the multiple acquired images, and determine the template window in the ROI of the stitched image, so that the monocular camera only needs to shoot at the same position once, that is, the back-end server may perform image stitching on the multiple images acquired in step S11 based on the size of the ROI determined in step S11. Of course, the monocular camera may scan a circle in advance to capture a plurality of images, and send the images to the back-end server, the back-end server obtains the plurality of images captured by the monocular camera in advance, calculates the structural similarity between the two images, determines the size of the ROI area according to the structural similarity, and then captures the images again at the position where the monocular camera captures the images during scanning to obtain corresponding images and send the corresponding images to the back-end server, the back-end server obtains the images captured by the single-camera, determines the search window in the ROI area of the spliced images, and determines the template window in the ROI area of the spliced images to realize real-time splicing of the images, that is, the back-end server may perform image splicing on the images newly captured by the monocular camera based on the size of the ROI area determined in step S11.
S13: and calculating the similarity of the template window at the corresponding position in the search window, and determining the matching position of the template window in the search window according to the similarity.
On the basis of step S12, an image coordinate system may be established with the top left vertex of the ROI in the stitched image as the origin, and the similarity of the template window at the corresponding position in the search window is calculated by using a similarity calculation method, specifically, the similarity of the template window at the corresponding position in the search window may be calculated once every time the template window moves in the search window, and then, for the template window, the matching position in the search window may be determined according to all the calculated similarities corresponding to the template window. Specifically, each time the template window moves in the search window, the similarity of the corresponding position of the template window in the search window obtained through corresponding calculation may be stored in the corresponding position in the similarity matrix (wherein the coordinate value of the similarity matrix is the coordinate information of the ROI, and the similarity matrix may store the similarity calculation result obtained through similarity calculation each time the template window is moved), after the search is completed, the position with the maximum similarity in the similarity matrix is the matching coordinate of the template window in the search window, and the coordinate is the matching position of the template window in the search window.
S14: and determining the overlapping area of the spliced image and the spliced image according to the matching position, and fusing the overlapping area of the spliced image and the spliced image.
Based on step S13, an accurate overlapping region between the stitched image and the stitched image may be determined according to the matching position of the template window in the search window, the size information of the ROI region, and the obtained coordinates of the matching position, that is, the position of the matching position in the stitched image may be determined according to the matching position of the template window in the search window, the offset between the image coordinate system established with the upper left vertex of the ROI region in the stitched image as the origin and the image coordinate system established with the upper left vertex of the stitched image as the origin, and the accurate overlapping region between the stitched image and the stitched image may be determined according to the position of the matching position in the stitched image to complete image registration.
After the image registration is completed, the determined overlapping area between the stitched image and the stitched image may be fused to realize image stitching between the stitched image and the stitched image, at this time, it may be determined whether the stitched image is the last image, if not, the stitched image is taken as a new stitched image, and the above steps S12-S14 are repeated, if so, it is determined that the image stitching is completed to obtain a final stitched image.
Through the process, the ROI area is determined based on the structural similarity of adjacent images, the similarity calculation of the template window and the search window is carried out according to the ROI area to carry out template matching, the matching position of the template window in the search window is determined, image registration and image splicing are realized, and feature point detection and transformation matrix calculation are not needed, so that the image splicing processing steps are far less than the processing steps of image splicing based on a feature point method in the prior art, the complexity is low, the calculated amount is small, the splicing speed is high, the requirements of real-time panoramic splicing can be met, the requirements of the proportion of an overlapping area are reduced by determining the ROI area through the structural similarity of the adjacent images and calculating the similarity of the corresponding position of the template window in the search window, the proportion of the overlapping area between the adjacent images can be reduced to 6 percent, and the image splicing in the same area can be finished by needing a small number of images, further, the information processing amount is further reduced, and the speed and the efficiency of image splicing are improved.
Compared with the existing image splicing based on the characteristic point method, the technical scheme disclosed by the application only needs to determine the size of the ROI according to the structural similarity of two adjacent images, determine the search window of the spliced image and the template window of the spliced image according to the ROI, calculate the similarity between the template window and the search window to determine the matching position of the template window in the search window, determine the overlapping area of the two images according to the matching position and fuse the overlapping area, and does not need to perform operations such as characteristic point detection, transformation matrix calculation and the like, so that the complexity of image splicing can be reduced, the requirement on the proportion of the overlapping area is reduced, the proportion of the overlapping area between the two adjacent images can be reduced to 6%, the number of the images required during image splicing and the information processing amount of image splicing are reduced, so as to improve the image splicing efficiency.
The image stitching method provided by the embodiment of the application, after determining the overlapping area between the stitched image and the stitched image according to the matching position, may further include:
calculating a first brightness mean value of an overlapping region in the spliced image, and calculating a second brightness mean value of the overlapping region in the spliced image;
and correcting the brightness of the spliced image according to the first brightness average value and the second brightness average value.
In this application, consider that the image of camera in the different scenes of different positions shooting inevitably has the inconsistent phenomenon of luminance, avoid luminance inconsistent to bring not good influence for final concatenation picture to make the luminance transition of final concatenation picture more natural, the visual effect uniformity is good, accords with the subjective effect of people's eye more, then can carry out the luminance correction to the concatenation picture.
Specifically, after determining the overlapping region of the stitched image and the stitched image according to the matching position, the number of pixels included in the overlapping region in the stitched image and the brightness values of all pixels included in the overlapping region may be obtained, and the brightness values of all pixels included in the overlapping region in the stitched imageAccumulating the values, and dividing the accumulated sum of the brightness values by the number of pixel points contained in the overlapping area in the spliced image to obtain a first brightness mean value mean of the overlapping area in the spliced imageoverlap1(ii) a In addition, the number of pixels included in the overlapping region in the stitched image and the brightness values of all pixels included in the overlapping region in the stitched image can be obtained, and the first brightness mean value mean calculated by the above method is adoptedoverlap1Similar method to calculate the second mean luminance mean of the overlapped region in the stitched imageoverlap2And the number of pixel points contained in the whole spliced image and the brightness values of all the pixel points can be obtained, and the first brightness mean value mean calculated by the method is adoptedoverlap1Calculating to obtain the mean brightness mean value mean before integral correction of the spliced image by a similar methodbeforecorrectionThen, utilize
Calculating to obtain the brightness mean value mean after the integral correction of the spliced imageaftercorrectionThen, the luminance mean value mean after the whole correction is carried out according to the spliced imageaftercorrectionBrightness correction is carried out on the whole spliced image (specifically, mean is carried out on the whole spliced image)aftercorrection-meanbeforecorrectionBrightness adjustment of the size), for example: referring to fig. 2 to 5, fig. 2 shows schematic diagrams of two adjacent images before brightness correction provided in the embodiment of the present application, fig. 3 shows schematic diagrams of two adjacent images obtained after brightness correction is performed on the two adjacent images in fig. 2, fig. 4 shows a schematic diagram of an infrared mosaic generated before brightness correction provided in the embodiment of the present application, and fig. 5 shows a schematic diagram of an infrared mosaic generated after brightness correction provided in the embodiment of the present application.
The image stitching method provided by the embodiment of the application fuses the overlapped area of the stitched image and the stitched image, and may include:
calculating an angle value corresponding to each pixel point according to the number of columns of each pixel point in the overlapping area;
and correspondingly calculating the pixel value of each pixel point in the overlapped area after fusion by utilizing the pixel value of each pixel point in the overlapped area in the spliced image, the pixel value of each pixel point in the overlapped area in the spliced image and the trigonometric function value of the angle value corresponding to each pixel point.
When the overlapped area of the spliced image and the spliced image is fused, the splicing gap exists in consideration of the fact that the traditional gradual-in and gradual-out fusion algorithm adopts the weight based on the distance transformation to carry out fusion, therefore, the method uses a new fusion algorithm based on trigonometric function weight to perform image fusion on the premise of gradually entering and gradually exiting the fusion algorithm, each column is represented by an angle value by calculating the relation of each column to the angle, and the square of its sine value and the square of its cosine value are calculated as weights, the calculation complexity is low, and meanwhile, the weight of the trigonometric function is very gradually changed when approaching from two boundaries (namely splicing traces) of the overlapping area to the center, while image fusion algorithms generally work as slow as possible as the weights shift with pixel position, which represents the nature of the image transition, therefore, the image fusion based on the trigonometric function weight can make the transition of the spliced image natural.
The specific process of image fusion based on trigonometric function weight is as follows: taking the upper left vertex of the overlapping area as an origin, and taking the right vertex as the positive direction of the pixel column number to establish an image coordinate system, and then calculating an angle value q corresponding to each pixel point by using q (pi x j)/(2 x w) according to the column number of each pixel point in the overlapping area, wherein j is the column number of the pixel point in the overlapping area, j is 0,1, w-1, and w is the width of the overlapping area; then, weight1 of the pixel points in the overlapped region in the spliced image is determined to be cos2(q) determining the weight of pixel points in the overlapping region in the stitched imageHeavy weight2 is sin2(q); then, the pixel value P (x, y) of each pixel in the fused overlapping region can be calculated by using P (x, y) ═ weight 1(x, y) + weight 2(x, y) P2(x, y), where P1(x, y) is the pixel value of the pixel in the overlapping region in the stitched image, and P2(x, y) is the pixel value of the pixel in the overlapping region in the stitched image.
The effect of eliminating the splicing trace at the splicing boundary when the image fusion is carried out based on the trigonometric function weight is good, the image transition is natural, the subjective effect of human eyes is met, and the visual consistency of the generated splicing image is good.
The image stitching method provided by the embodiment of the application calculates the similarity of the template window at the corresponding position in the search window, and may include:
and calculating the similarity of the template window at the corresponding position in the search window by utilizing a normalized product correlation template matching algorithm.
In the application, the similarity of the template window at the corresponding position in the search window can be calculated by utilizing a normalized product correlation template matching algorithm, and the template window has the capabilities of being free from the influence of scale factor errors, resisting white noise interference and the like, so that the accuracy of image registration and image splicing can be improved.
The calculation formula of the normalized product correlation template matching algorithm is as follows:
t represents a template window, S represents a search window, R represents a similarity matrix, T (x ', y') represents the size of a pixel value at a position (x ', y') in the template window T, S (x + x ', y + y') represents the size of a pixel value at a position (x + x ', y + y') in the search window S, R (x, y) represents a similarity value (i.e. similarity) at a position (x, y) in the similarity matrix, the closer to 1 the value of R (x, y), or the larger the similarity value, the higher the matching degree is, in an accumulation value formula, the range of x 'is the width of the template window, and the range of y' is the height of the template window.
The image stitching method provided by the embodiment of the application calculates the similarity of the template window at the corresponding position in the search window, and determines the matching position of the template window in the search window according to the similarity, and may include:
moving the template window in the search window according to a preset path and a preset step length by taking a preset point of the search window as an original point, and calculating the similarity of the corresponding position of the template window in the search window after the template window is moved each time;
and taking the corresponding position when the similarity pair is maximum as the matching position of the template window in the search window.
In the application, when the similarity of the template window at the corresponding position in the search window is calculated, and the matching position of the template window in the search window is determined according to the similarity, the template window can be moved in the search window according to a preset path and a preset step length by taking a preset point of the search window as an origin, and the similarity of the corresponding position of the template window in the search window after the template window is moved once is calculated every time the template window is moved once, then, the calculated similarity is stored in the corresponding position in the similarity matrix, specifically, the top left vertex of the search window can be used as the origin, the template window is moved from left to right and from top to bottom, searching and matching are carried out in a mode that the step length of each movement is 1 pixel point (of course, the moving step can be adjusted according to the requirement), the method and the device realize ordered movement and search, avoid omission and improve the accuracy and reliability of the determination of the matching position.
The image stitching method provided by the embodiment of the application obtains a plurality of images, and may include:
and acquiring an image shot by the monocular camera rotating to each preset fixed point around the shaft.
In the application, a monocular camera with a holder can be used for shooting an image at each preset fixed point, and the shot image is sent to a rear-end server in real time, so that the rear-end server can obtain the image shot by the monocular camera rotating around a shaft to each preset fixed point, wherein the number of the fixed points is N, the fixed points are uniformly distributed, and the number of the N can be specifically adjusted according to the resolution of the camera, the deployment scene of the camera and the overlapping percentage between the images.
In addition, because the fixed points are uniformly distributed, the size of the ROI area of each image can be fixed to the size of the finally determined ROI area, namely, the ROI area is determined only once after the fixed points are determined, and the influence of the change of the shooting scene is avoided. And the fixed point setting can ensure that the monocular camera directly shoots according to the fixed point when shooting every time, so that the complexity of shooting position determination is reduced.
The monocular camera is used for realizing image splicing, so that the image splicing cost can be reduced, and the image splicing can be realized without calibrating the internal and external parameters of the camera in advance, so that the labor and material cost in the process production flow can be reduced in a large procedure, and the monocular camera is suitable for large-scale industrial production of products.
The image stitching method provided by the embodiment of the application can further include, after fusing the overlapped areas of the stitched image and the stitched image:
and cutting and/or scaling the size of the final splicing image obtained by fusion according to the size of the display screen, and displaying the final splicing image after cutting and/or scaling in the display screen.
In the application, the overlapped area of the spliced image and the spliced image is fused to obtain the final spliced image, the final spliced image obtained by fusion can be cut and/or scaled according to the size of the display screen to fit the display size of the display screen and meet the display requirement, and then the final spliced image after cutting and/or scaling can be displayed in the display screen to facilitate the viewing and acquisition of the final spliced image by related personnel.
Taking panoramic stitching of infrared images as an example, specifically referring to fig. 6, which shows a flowchart of a method for stitching infrared panoramic images provided in an embodiment of the present application, and for specific description of relevant steps, reference may be made to the detailed description of the corresponding parts above, which is not repeated herein.
An embodiment of the present application further provides an image stitching device, see fig. 7, which shows a schematic structural diagram of an image stitching device provided in an embodiment of the present application, and the image stitching device may include:
an obtaining module 71, configured to obtain multiple images, calculate a structural similarity between two adjacent images, and determine a size of the ROI according to the structural similarity; wherein, an overlapping area exists between two adjacent images;
a first determining module 72, configured to determine a search window in the ROI region of the stitched image, and determine a template window in the ROI region of the stitched image;
the first calculating module 73 is configured to calculate similarity of corresponding positions of the template window in the search window, and determine a matching position of the template window in the search window according to the similarity;
and the fusion module 74 is configured to determine an overlapping region between the stitched image and the stitched image according to the matching position, and fuse the overlapping region between the stitched image and the stitched image.
The image stitching device provided by the embodiment of the application can further comprise:
the second calculation module is used for calculating a first brightness mean value of an overlapping area in the spliced image and calculating a second brightness mean value of the overlapping area in the spliced image after the overlapping area of the spliced image and the spliced image is determined according to the matching position;
and the brightness correction module is used for correcting the brightness of the spliced image according to the first brightness mean value and the second brightness mean value.
In an image stitching apparatus provided in an embodiment of the present application, the fusion module 74 may include:
the first calculating unit is used for calculating the angle value corresponding to each pixel point according to the number of the columns of the pixel points in the overlapping area;
and the second calculating unit is used for correspondingly calculating the pixel value of each pixel point in the fused overlapping area by utilizing the pixel value of each pixel point in the overlapping area in the spliced image, the pixel value of each pixel point in the overlapping area in the spliced image and the trigonometric function value of the angle value corresponding to each pixel point.
In an image stitching apparatus provided in an embodiment of the present application, the first calculating module 73 may include:
and the third calculating unit is used for calculating the similarity of the template window at the corresponding position in the search window by utilizing a normalized product correlation template matching algorithm.
In an image stitching apparatus provided in an embodiment of the present application, the first calculating module 73 may include:
the fourth calculation unit is used for moving the template window in the search window according to a preset path and a preset step length by taking a preset point of the search window as an original point, and calculating the similarity of the corresponding position of the template window in the search window after the template window is moved each time;
and the corresponding position when the similarity pair is maximum is used as the matching position of the template window in the search window.
In an image stitching apparatus provided in an embodiment of the present application, the obtaining module 71 may include:
and the acquisition unit is used for acquiring the image shot by the monocular camera rotating to each preset fixed point around the shaft.
The image stitching device provided by the embodiment of the application can further comprise:
and the display module is used for cutting and/or scaling the final splicing image obtained by fusion according to the size of the display screen after the overlapped area of the spliced image and the spliced image is fused, and displaying the final splicing image after cutting and/or scaling in the display screen.
An embodiment of the present application further provides an image stitching device, see fig. 8, which shows a schematic structural diagram of the image stitching device provided in the embodiment of the present application, and the image stitching device may include:
a memory 81 for storing a computer program;
the processor 82, when executing the computer program stored in the memory 81, may implement the following steps:
acquiring a plurality of images, calculating the structural similarity of two adjacent images, and determining the size of an ROI (region of interest) according to the structural similarity; wherein, an overlapping area exists between two adjacent images; determining a search window in the ROI area of the spliced image, and determining a template window in the ROI area of the spliced image; calculating the similarity of the template window at the corresponding position in the search window, and determining the matching position of the template window in the search window according to the similarity; and determining the overlapping area of the spliced image and the spliced image according to the matching position, and fusing the overlapping area of the spliced image and the spliced image.
An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the following steps may be implemented:
acquiring a plurality of images, calculating the structural similarity of two adjacent images, and determining the size of an ROI (region of interest) according to the structural similarity; wherein, an overlapping area exists between two adjacent images; determining a search window in the ROI area of the spliced image, and determining a template window in the ROI area of the spliced image; calculating the similarity of the template window at the corresponding position in the search window, and determining the matching position of the template window in the search window according to the similarity; and determining the overlapping area of the spliced image and the spliced image according to the matching position, and fusing the overlapping area of the spliced image and the spliced image.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
For a description of a relevant part in an image stitching device, and a computer-readable storage medium provided by the present application, reference may be made to a detailed description of a corresponding part in an image stitching method provided by the present application, and details are not repeated herein.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include elements inherent in the list. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element. In addition, parts of the above technical solutions provided in the embodiments of the present application, which are consistent with the implementation principles of corresponding technical solutions in the prior art, are not described in detail so as to avoid redundant description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. An image stitching method, comprising:
acquiring a plurality of images, calculating the structural similarity of two adjacent images, and determining the size of an ROI (region of interest) according to the structural similarity; wherein an overlapping area exists between two adjacent images;
determining a search window in the ROI area of the spliced image, and determining a template window in the ROI area of the spliced image;
calculating the similarity of the template window at the corresponding position in the search window, and determining the matching position of the template window in the search window according to the similarity;
and determining the overlapping area of the spliced image and the spliced image according to the matching position, and fusing the overlapping area of the spliced image and the spliced image.
2. The image stitching method according to claim 1, further comprising, after determining an overlapping region of the stitched image and the stitched image according to the matching position:
calculating a first brightness mean value of an overlapping region in the spliced image, and calculating a second brightness mean value of the overlapping region in the spliced image;
and correcting the brightness of the spliced image according to the first brightness mean value and the second brightness mean value.
3. The image stitching method according to claim 2, wherein fusing the overlapped regions of the stitched image and the stitched image comprises:
calculating an angle value corresponding to each pixel point according to the number of columns of each pixel point in the overlapping area;
and correspondingly calculating the pixel value of each pixel point in the fused overlapping area by using the pixel value of each pixel point in the overlapping area in the spliced image, the pixel value of each pixel point in the overlapping area in the spliced image and the trigonometric function value of the angle value corresponding to each pixel point.
4. The image stitching method according to claim 1, wherein calculating the similarity of the template window at the corresponding position in the search window comprises:
and calculating the similarity of the template window at the corresponding position in the search window by utilizing a normalized product correlation template matching algorithm.
5. The image stitching method according to claim 1, wherein calculating the similarity of the template window at the corresponding position in the search window, and determining the matching position of the template window in the search window according to the similarity comprises:
moving the template window in the search window according to a preset path and a preset step length by taking a preset point of the search window as an original point, and calculating the similarity of the corresponding position of the template window in the search window after the template window is moved each time;
and taking the corresponding position of the template window when the similarity pair is maximum as the matching position of the template window in the search window.
6. The image stitching method of claim 1, wherein acquiring a plurality of images comprises:
and acquiring the image shot by the monocular camera rotating to each preset fixed point around the shaft.
7. The image stitching method according to claim 1, further comprising, after fusing the overlapped regions of the stitched image and the stitched image:
and cutting and/or scaling the size of the final splicing image obtained by fusion according to the size of the display screen, and displaying the final splicing image after cutting and/or scaling in the display screen.
8. An image stitching device, comprising:
the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a plurality of images, calculating the structural similarity of two adjacent images and determining the size of an ROI (region of interest) according to the structural similarity; wherein an overlapping area exists between two adjacent images;
the first determining module is used for determining a search window in the ROI area of the spliced image and determining a template window in the ROI area of the spliced image;
the first calculation module is used for calculating the similarity of the template window at the corresponding position in the search window and determining the matching position of the template window in the search window according to the similarity;
and the fusion module is used for determining the overlapped area of the spliced image and the spliced image according to the matching position and fusing the overlapped area of the spliced image and the spliced image.
9. An image stitching device, characterized by comprising:
a memory for storing a computer program;
a processor for implementing the steps of the image stitching method as claimed in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the image stitching method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111069395.7A CN113744133A (en) | 2021-09-13 | 2021-09-13 | Image splicing method, device and equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111069395.7A CN113744133A (en) | 2021-09-13 | 2021-09-13 | Image splicing method, device and equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113744133A true CN113744133A (en) | 2021-12-03 |
Family
ID=78738424
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111069395.7A Pending CN113744133A (en) | 2021-09-13 | 2021-09-13 | Image splicing method, device and equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113744133A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116229297A (en) * | 2023-03-09 | 2023-06-06 | 广东精益空间信息技术股份有限公司 | Mapping data processing method, mapping data processing system, mapping data processing medium and mapping data processing computer |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2009289078A (en) * | 2008-05-29 | 2009-12-10 | Dainippon Printing Co Ltd | Target detection system |
| CN103559704A (en) * | 2013-10-09 | 2014-02-05 | 哈尔滨工程大学 | Method for visually positioning tank mouth of railway oil tank truck |
| CN107239780A (en) * | 2017-04-29 | 2017-10-10 | 安徽慧视金瞳科技有限公司 | A kind of image matching method of multiple features fusion |
| CN107590502A (en) * | 2017-09-18 | 2018-01-16 | 西安交通大学 | A kind of whole audience dense point fast matching method |
| JP2018148513A (en) * | 2017-03-09 | 2018-09-20 | キヤノン株式会社 | Image processing system, image processing method, and program |
| JP2019009643A (en) * | 2017-06-26 | 2019-01-17 | 株式会社リコー | Image processing apparatus, image processing method and program |
| CN109767447A (en) * | 2019-01-04 | 2019-05-17 | 腾讯科技（深圳）有限公司 | A kind of template matching method, device, equipment and medium |
| CN110838086A (en) * | 2019-11-07 | 2020-02-25 | 上海大学 | Outdoor image splicing method based on correlation template matching |
| CN111260561A (en) * | 2020-02-18 | 2020-06-09 | 中国科学院光电技术研究所 | Rapid multi-graph splicing method for mask defect detection |
| CN111598177A (en) * | 2020-05-19 | 2020-08-28 | 中国科学院空天信息创新研究院 | Self-adaptive maximum sliding window matching method facing low-overlapping image matching |
| CN112837257A (en) * | 2019-11-06 | 2021-05-25 | 广州达普绅智能设备有限公司 | Curved surface label splicing detection method based on machine vision |
| CN113111212A (en) * | 2021-04-01 | 2021-07-13 | 广东拓斯达科技股份有限公司 | Image matching method, device, equipment and storage medium |
| CN113362362A (en) * | 2021-06-17 | 2021-09-07 | 易普森智慧健康科技（深圳）有限公司 | Bright field microscope panoramic image alignment algorithm based on total variation area selection |
-
2021
- 2021-09-13 CN CN202111069395.7A patent/CN113744133A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2009289078A (en) * | 2008-05-29 | 2009-12-10 | Dainippon Printing Co Ltd | Target detection system |
| CN103559704A (en) * | 2013-10-09 | 2014-02-05 | 哈尔滨工程大学 | Method for visually positioning tank mouth of railway oil tank truck |
| JP2018148513A (en) * | 2017-03-09 | 2018-09-20 | キヤノン株式会社 | Image processing system, image processing method, and program |
| CN107239780A (en) * | 2017-04-29 | 2017-10-10 | 安徽慧视金瞳科技有限公司 | A kind of image matching method of multiple features fusion |
| JP2019009643A (en) * | 2017-06-26 | 2019-01-17 | 株式会社リコー | Image processing apparatus, image processing method and program |
| CN107590502A (en) * | 2017-09-18 | 2018-01-16 | 西安交通大学 | A kind of whole audience dense point fast matching method |
| CN109767447A (en) * | 2019-01-04 | 2019-05-17 | 腾讯科技（深圳）有限公司 | A kind of template matching method, device, equipment and medium |
| CN112837257A (en) * | 2019-11-06 | 2021-05-25 | 广州达普绅智能设备有限公司 | Curved surface label splicing detection method based on machine vision |
| CN110838086A (en) * | 2019-11-07 | 2020-02-25 | 上海大学 | Outdoor image splicing method based on correlation template matching |
| CN111260561A (en) * | 2020-02-18 | 2020-06-09 | 中国科学院光电技术研究所 | Rapid multi-graph splicing method for mask defect detection |
| CN111598177A (en) * | 2020-05-19 | 2020-08-28 | 中国科学院空天信息创新研究院 | Self-adaptive maximum sliding window matching method facing low-overlapping image matching |
| CN113111212A (en) * | 2021-04-01 | 2021-07-13 | 广东拓斯达科技股份有限公司 | Image matching method, device, equipment and storage medium |
| CN113362362A (en) * | 2021-06-17 | 2021-09-07 | 易普森智慧健康科技（深圳）有限公司 | Bright field microscope panoramic image alignment algorithm based on total variation area selection |
Non-Patent Citations (3)
| Title |
|---|
| 王勇;何晓川;刘清华;许录平;: "一种感兴趣区域寻优搜索的全自动图像拼接算法", 电子与信息学报, no. 02, 15 February 2009 (2009-02-15) * |
| 王诚;李琳;: "基于模板匹配的全景图像拼接", 福建电脑, no. 04, 1 April 2008 (2008-04-01), pages 104 - 105 * |
| 陈天婷: "基于环视***的停车位检测与跟踪算法", 《汽车技术》, 30 September 2020 (2020-09-30), pages 1 - 2 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116229297A (en) * | 2023-03-09 | 2023-06-06 | 广东精益空间信息技术股份有限公司 | Mapping data processing method, mapping data processing system, mapping data processing medium and mapping data processing computer |
| CN116229297B (en) * | 2023-03-09 | 2023-10-13 | 广东精益空间信息技术股份有限公司 | Mapping data processing method, mapping data processing system, mapping data processing medium and mapping data processing computer |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11044458B2 (en) | Methods, systems, and computer-readable storage media for generating three-dimensional (3D) images of a scene | |
| CN111641775B (en) | Multi-shooting zoom control method, device and electronic system | |
| CN106462944B (en) | High-resolution panorama VR generator and method | |
| US9325899B1 (en) | Image capturing device and digital zooming method thereof | |
| US8508580B2 (en) | Methods, systems, and computer-readable storage media for creating three-dimensional (3D) images of a scene | |
| CN103905792B (en) | A kind of 3D localization methods and device based on PTZ CCTV cameras | |
| CN111750820B (en) | Image positioning method and system | |
| EP3163535A1 (en) | Wide-area image acquisition method and device | |
| CN107665483B (en) | Calibration-free convenient monocular head fisheye image distortion correction method | |
| CN101853524A (en) | Method for generating corn ear panoramic image by using image sequence | |
| CN108717704B (en) | Target tracking method based on fisheye image, computer device and computer readable storage medium | |
| JP3907008B2 (en) | Method and means for increasing the depth of field for a photograph | |
| CN112399033B (en) | Camera assembly and monitoring camera | |
| CN103971375A (en) | Panoramic gaze camera space calibration method based on image splicing | |
| CN111935410B (en) | Quick view finding method and system for multi-camera shooting | |
| CN110602376B (en) | Snapshot method and device and camera | |
| CN105227948B (en) | The method and device of distorted region in a kind of lookup image | |
| JP2003179800A (en) | Device for generating multi-viewpoint image, image processor, method and computer program | |
| CN113744133A (en) | Image splicing method, device and equipment and computer readable storage medium | |
| US20090059018A1 (en) | Navigation assisted mosaic photography | |
| CN114022562A (en) | Panoramic video stitching method and device capable of keeping integrity of pedestrians | |
| JP4198536B2 (en) | Object photographing apparatus, object photographing method and object photographing program | |
| CN109543496B (en) | Image acquisition method and device, electronic equipment and system | |
| CN110796690B (en) | Image matching method and image matching device | |
| CN115065782B (en) | Scene acquisition method, acquisition device, image pickup equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |