CN113160053A - Pose information-based underwater video image restoration and splicing method - Google Patents
Pose information-based underwater video image restoration and splicing method Download PDFInfo
- Publication number
- CN113160053A CN113160053A CN202110354434.1A CN202110354434A CN113160053A CN 113160053 A CN113160053 A CN 113160053A CN 202110354434 A CN202110354434 A CN 202110354434A CN 113160053 A CN113160053 A CN 113160053A
- Authority
- CN
- China
- Prior art keywords
- pose information
- frame
- key frame
- camera
- channel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims abstract description 36
- 230000001360 synchronised effect Effects 0.000 claims abstract description 8
- 239000011159 matrix material Substances 0.000 claims description 35
- 238000004364 calculation method Methods 0.000 claims description 24
- 238000005286 illumination Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 15
- 230000008859 change Effects 0.000 claims description 12
- 230000004927 fusion Effects 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 10
- 230000003287 optical effect Effects 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 8
- 238000013519 translation Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 239000002994 raw material Substances 0.000 claims description 3
- 238000011084 recovery Methods 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 230000002194 synthesizing effect Effects 0.000 abstract 1
- 238000000605 extraction Methods 0.000 description 7
- 238000005070 sampling Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000750 progressive effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 241000195493 Cryptophyta Species 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000012535 impurity Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000009364 mariculture Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 230000004083 survival effect Effects 0.000 description 1
- 238000012876 topography Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01D—MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE; ARRANGEMENTS FOR MEASURING TWO OR MORE VARIABLES NOT COVERED IN A SINGLE OTHER SUBCLASS; TARIFF METERING APPARATUS; MEASURING OR TESTING NOT OTHERWISE PROVIDED FOR
- G01D21/00—Measuring or testing not otherwise provided for
- G01D21/02—Measuring two or more variables by means not covered by a single other subclass
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Geometry (AREA)
- Studio Devices (AREA)
Abstract
The invention discloses an underwater video image restoration and splicing method based on pose information, which comprises the following steps: acquiring an underwater video image with a synchronous calibration signal and pose information of a camera through an underwater video image acquisition device; carrying out one-to-one correspondence on video frames and pose information of the underwater video image; calculating homography matrixes among the video frames by using the pose information, further solving the overlapping area among the video frames, extracting key frames according to the proportion of the overlapping area, and performing time calibration on the video frames and the pose information; an improved shallow water image enhancement algorithm which is obtained based on self-adaptive parameters and is stretched relative to a global histogram is adopted to restore the underwater video image; dividing the restored key frame into a plurality of sub-modules by combining pose information; and after the sub-modules are spliced into sub-module images, synthesizing the sub-module images into a final panoramic image. The invention combines the pose information and can quickly obtain a clear underwater panorama.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to an underwater video image restoration and splicing method based on pose information.
Background
The water bottom substrate, water level, water flow, the abundance of floating algae and the like of the cage all influence the capacity of cage culture. A good net cage site selection can provide superior survival and growth conditions for mariculture, and improves the culture capacity of the water area to a certain extent. An unmanned ship is used for carrying an underwater camera in an offshore shallow water area (the water depth is generally 3 to 10 meters), and panoramic splicing is carried out after underwater video shooting is carried out, so that underwater topography and landform can be obtained, and the auxiliary net cage site selection is facilitated, and scientific layout is carried out.
The underwater video image is acquired by an underwater camera, and before video restoration and splicing, a key frame is required to be extracted from the video to reduce information redundancy generated in the video acquisition process and reduce data processing amount. Common key frame extraction methods include methods based on sampling, inter-frame difference, accumulated inter-frame difference, clustering, motion information and the like. The sampling-based key frame extraction method is simple in speed block and implementation, but depends on the setting of a sampling interval, when the interval is too small, the obtained key frames have larger redundancy, and when the interval is too large, the adjacent key frames possibly have no overlapping area. The key frame extraction method based on clustering has the defects of sensitivity to initial parameters and high calculation complexity. These methods directly extract key frames from the video without fully considering the pose and geographic location information of the underwater photography camera. If the video can be combined with position (GPS) information and attitude information, namely pose information for short (which can be acquired by a pose information recording module on the unmanned ship), the problem can be well solved.
Because of strong attenuation caused by scattering of particles and impurities in water and absorption of a water medium, the underwater image generally has the problems of color cast, blurring and small field range; in addition, due to the influence of artificial light sources, the underwater image also has the problem of uneven illumination. The image splicing directly performed by using the acquired underwater image is easy to cause splicing failure, so that the underwater image needs to be subjected to light homogenizing and defogging treatment. The traditional underwater image restoration method comprises a defogging algorithm based on a dark channel, an algorithm based on histogram equalization, an algorithm based on homomorphic filtering, an algorithm based on wavelet transformation and an algorithm based on image fusion. These methods are not ideal for the case of uneven illumination due to artificial light sources.
Disclosure of Invention
The invention aims to solve the defects in the prior art and provides an underwater video image restoration and splicing method based on pose information. Firstly, combining video frames and pose information of an underwater video image together to enable each video frame of the underwater video image to correspond to a piece of pose information, and carrying out time calibration on the pose information; calculating homography matrixes among the video frames according to the pose information, further solving the overlapping area among the video frames, and extracting key frames according to the proportion of the overlapping area; an improved shallow water image enhancement algorithm which is obtained based on self-adaptive parameters and is stretched relative to a global histogram is provided for restoring an underwater video image; dividing the restored key frame into a plurality of sub-modules by combining pose information; and after the sub-modules are spliced into sub-module images, splicing the sub-module images into a final panoramic image.
The purpose of the invention can be achieved by adopting the following technical scheme:
an underwater video image restoration and splicing method based on pose information comprises the following steps:
s1, acquiring an underwater video image with a synchronous calibration signal and pose information of a camera through an underwater video image acquisition device;
s2, carrying out one-to-one correspondence on the video frames and the pose information of the underwater video images;
s3, calculating homography matrixes among the video frames by using the pose information, further solving the overlapping area among the video frames, extracting key frames according to the proportion of the overlapping area, and performing time calibration on the video frames and the pose information;
s4, performing underwater video image restoration by adopting an improved shallow water image enhancement algorithm which is obtained based on self-adaptive parameters and is stretched relative to the global histogram, and obtaining a restored key frame;
s5, dividing the restored key frame into a plurality of sub-modules by combining pose information;
and S6, splicing the sub-modules into sub-module images, and splicing the sub-module images into a final panoramic image.
Further, the underwater video image acquisition device comprises a camera module and a pose information recording module, wherein the camera module consists of a camera and a first SD card, the camera is used for shooting underwater video images, and the first SD card is used for storing the underwater video images with synchronous calibration signals; the pose information recording module consists of a main controller, a gyroscope, a GPS (global positioning system), an underwater ultrasonic distance measuring sensor, a signal lamp and a second SD (secure digital) card, wherein the main controller is used for controlling the work of other units, the gyroscope is used for acquiring the pose information of a camera, the GPS is used for acquiring the position information of the camera, the underwater ultrasonic distance measuring sensor is used for acquiring the height of the camera from the water bottom, the signal lamp is used for sending an optical signal to the camera module, and the second SD card is used for storing the pose information of the camera.
Furthermore, because the underwater video image and the pose information are not acquired by the same system, a communication mode is needed for combining the underwater video image and the pose information; common communication modes include acoustic, optical, electrical and other signals; wherein, the optical signal propagation speed is fast, the resulting error is small, and the implementation is simple, so the optical signal is used as the start and time calibration signal of the video frame and the pose information, and the step S1 is as follows:
the method comprises the steps of firstly starting a camera module to start shooting underwater video images, then starting a pose information recording module, controlling a signal lamp to give a light signal 1 to the camera before the pose information recording module starts recording pose information, then starting recording the pose information, controlling the signal lamp to give a light signal 2 to the camera at intervals, and setting Flag of current pose information to be 1.
The lighting signal 1 is used as a starting signal of a video frame and pose information during post data processing; the light signal 2 is used as a time calibration signal of video frames and pose information during post data processing.
Further, the step S2 process is as follows:
reading the underwater video image obtained in the step S1, taking the video frames detected to contain light as signal frames, taking a certain number of continuously detected signal frames as start signals, taking the last frame of all the detected signal frames as a video frame corresponding to the first piece of pose information, and then sequentially corresponding each frame of video frame to each piece of pose information one by one;
the process of judging whether the current video frame is a signal frame is as follows: uniformly dividing a current video frame into an upper region, a lower region, a left region and a right region, respectively counting whether the proportion of the number of pixels of which the numerical value of a channel R is greater than a threshold value alpha1 in the total pixels of the whole region is greater than a threshold value alpha2 or not, and whether the proportion of the number of pixels of which the numerical value of a channel G, B is less than a threshold value alpha1 in the total pixels of the whole region is greater than a threshold value alpha2 or not; if the value distribution of R, G, B three channels in one area meets the above condition, the current video frame is a signal frame; otherwise the current video frame is not a signal frame.
Further, when extracting key frames, a homography matrix between video frames is obtained by using pose information and a camera imaging principle, so that the overlapping area between the video frames is obtained, and the key frames are extracted according to the proportion of the overlapping area; compared with a key frame extraction method based on sampling, the method can quickly extract key frames and ensure that the overlapping area between the key frames is within a reasonable range; compared with a key frame extraction method based on clustering, the method has small calculation complexity, and the step S3 includes the following steps:
s301, taking the video frame corresponding to the first piece of pose information as a current frame, and taking the first piece of pose information as pose information of the current frame;
s302, judging whether the Flag of the pose information of the current frame is 1, if so, executing a step S303, and if not, executing a step S305;
s303, judging whether the current frame is a signal frame, if so, indicating that no time deviation occurs, executing a step S309, otherwise, indicating that the time deviation occurs, and executing a step S304;
s304, detecting N frames forwards and backwards respectively, setting the signal frame as a current frame if the signal frame is detected, executing the step S309, and ending the step S3 if the signal frame is not detected;
s305, judging whether absolute values of a roll angle and a pitch angle of the current frame are smaller than a threshold beta1, if not, executing a step S309, and if so, executing a step S306;
s306, judging whether the key frame buffer queue is empty, if so, executing a step S308, and if not, executing a step S307;
s307, calculating the overlapping area of the current frame and the last frame in the key frame buffer queue by combining the pose information, judging whether the ratio of the overlapping area to the area of the current frame is in a beta2 range, executing the step S308 if the ratio is in the beta2 range, and executing the step S309 if the ratio is not in the beta2 range;
s308, extracting the current frame as a key frame, storing the key frame into a key frame buffer queue, and executing the step S309;
s309, judging whether a next video frame and next pose information exist, if so, reading the next video frame as a current frame, reading the next pose information as the pose information of the current frame, and repeatedly executing the step S302-the step S309, otherwise, ending the step S3.
Further, the process of calculating the overlap area in step S307 is as follows:
s3071, taking the last frame in the key frame buffer queue as the previous key frame, taking the shooting position of the previous key frame as the origin of coordinates, taking the true west as the positive direction of an X axis, the true north as the positive direction of a Y axis, and the true ground as the positive direction of a Z axis to establish a coordinate system, and calculating a homography matrix H from the current frame to the previous key frame by combining pose information, wherein the H is a matrix of 3X 3, and the calculation formula is as follows:
wherein H1The homography matrix for the last key frame to the projection plane,is H1Inverse matrix of, H2A homography matrix from the current frame to the projection plane;
wherein, because the shooting position of the above key frame is used as the origin of coordinates, the translation vector is zero vector, so H1The calculation formula of (a) is as follows:
H1=KR1K-1 (2)
wherein K is an internal parameter matrix of the camera and is obtained by calibrating the camera, R1A rotation matrix of 3 × 3;
wherein f represents the focal length of the camera; dx and dy respectively represent the physical size of each pixel on the key frame in the coordinate axis direction, and the unit is mm/pixel; (u)0,v0) Coordinates of an intersection point of the optical axis of the camera and the key frame plane are positioned at the center of the key frame;
wherein,θ1、ψ1are respectively the last oneThe roll angle, the pitch angle and the course angle of the camera during key frame shooting;
H2the calculation formula of (a) is as follows:
wherein R is2Is a rotation matrix of 3 multiplied by 3, I is a unit matrix, t is a three-dimensional translation vector, d is the distance from the origin of coordinates to the water bottom plane, and n is a unit normal vector [0, 0, -1 ] of the water bottom plane],nTRepresents a transpose of the vector n;
wherein,θ2、ψ2respectively a roll angle, a pitch angle and a course angle of the camera when the current frame is shot;
t=[x2-x1,y2-y1,z2-z1]T (7)
x2-x1=(lon2-lon1)×111110×cos(lat1×π/180) (8)
y2-y1=(lat2-lat1)×111110 (9)
z2-z1=h2-h1 (10)
wherein (x)1,y1,z1) Coordinates representing the last key frame shot position, (x)2,y2,z2) Coordinates representing the shooting position of the current frame, lon1、lat1、h1Longitude, latitude and height from the bottom of the water, lon, for the last key frame shooting position2、lat2、h2Respectively representing the longitude and latitude of the shooting position of the current frame and the height from the water bottom;
s3072, calculating coordinate positions of four vertexes of the current frame after homography transformation, wherein a calculation formula is as follows:
position=vertex./vertex(3,:) (12)
wherein, width represents the width of the current frame, height represents the height of the current frame, vertex/vertex (3: 3) represents the normalization processing of the homogeneous coordinate to obtain the coordinate of the two-dimensional plane;
s3073, counting the vertexes of the quadrangle of the current frame after homography transformation in the rectangle represented by the last key frame;
s3074, calculating the intersection point of each side of the quadrangle of the current frame after homographic transformation and each side of the previous key frame;
s3075, sequencing the obtained vertexes and intersection points clockwise or anticlockwise;
s3076, the area of the polygon in the overlapping area is obtained, any vertex of the polygon is taken as a fixed point, the polygon is divided into a plurality of triangles, the area of each triangle is calculated respectively and summed, and the size of the overlapping area is obtained.
Further, when the underwater image restoration is performed on the keyframe, the gamma function is used for correcting the bright channel, instead of performing linear sliding stretching on the bright channel in the original algorithm, so that a better dodging processing effect is achieved on the underwater video image with uneven illumination, and the step S4 is as follows:
s401, converting the key frame from an RGB color model into an HSV color model;
s402, convolving and weighting the bright channel V by Gaussian functions with different scales to obtain an estimated value I (x, y) of the illumination component;
wherein (x, y) represents the coordinates of the bright channel, V (x, y) represents the bright channel, G (x, y) represents the luminance of the bright channelg,yg) Representing a Gaussian function, ωlRepresenting a weighting coefficient of an illumination component obtained by convolution of the ith scale Gaussian function on the bright channel V; n is a radical oflIs the total number of Gaussian functions with different scales, i 1, 2lσ is the standard deviation of the Gaussian function, (x)g,yg) Representing the template coordinates of the Gaussian function, wherein the central position of the template is an origin;
s403, correcting the bright channel V by using a two-dimensional gamma function, wherein the formula of the two-dimensional gamma function is as follows:
wherein, VO(x, y) are the values of the corrected bright channel, γ is an index value for luminance enhancement, the index value contains the illumination component characteristics of the key frame, and m is the luminance mean of the illumination components;
s404, converting the key frame processed in the steps S402 and S403 from an HSV color model to an RGB color model, and solving a color balance coefficient theta of a channel G, Bg、θbAnd the pixel values of the channel G, B are multiplied by color equalization coefficients theta, respectivelyg、θbCarrying out color balance to obtain a balanced RGB color model; wherein theta isg、θbThe calculation formula of (a) is as follows:
wherein G isavg、BavgNormalized average values after recovery of channel G, B, M is the key frame height, N is the key frame width, Ig(I, j) is the value of channel G at (I, j), Ib(i, j) is the value of channel B at (i, j), i represents the ith column of channel G, B, j represents the jth row of channel G, B;
s405, carrying out global histogram stretching on the R, G, B three channels after equalization, wherein a calculation formula is as follows:
wherein, PinFor input of pixel values, PoutFor the image values after histogram stretching, IminIs the value of the pixel value of the channel at 0.5% position after being sorted from small to large, ImaxIs the value of the pixel at 99.5% position, O, after the pixel values of the channel are sorted from small to largemaxIs 255, OminIs 0;
s406, converting the keyframe processed in the step S405 from the RGB color model to the CIELab color model, and modifying the color grades of the channels a and b in the CIELab color model, wherein the calculation formula is as follows:
wherein,a color value representing a color value of an input channel,a value of the output color is represented,the optimal experimental values are shown in the figure,taking the empirical value of the raw material to be tested,taking the value as channel a or channel b;
and S407, converting the key frame processed in the step S406 from the CIELab color model to the RGB color model to obtain a restored key frame.
Further, the step S5 process is as follows:
traversing all the key frames, calculating the absolute value change from the pose information of any key frame to the pose information of other key frames, and taking the key frame with the minimum value accumulated by the absolute value changes of the pose information as a reference image; finding a globally optimal key frame, and taking the globally optimal key frame as a globally reference image; continuously recursively finding all locally optimal key frames on two sides of the globally optimal key frame, and taking the locally optimal key frames as local reference images; and dividing all the key frames into a plurality of sub-modules according to a binary tree structure by taking the globally optimal key frame as a root node and the locally optimal key frame as a child node.
Further, the formula for calculating the absolute value change size from the pose information of any one key frame to the pose information of other key frames is as follows:
wherein iτ∈[1,N],Respectively represent the ithτRoll angle, pitch angle and height from the water bottom of the Zhang key frame,respectively represent the j thτRoll angle, pitch angle and height from water bottom of Zhang key frame, alpha, beta respectively represent weight occupied by influence brought by angle change and height change, NτRepresenting the total number of key frames.
Further, the step S6 process is as follows: firstly, splicing submodules at the bottommost layer of the binary tree, and upwards splicing layer by layer to obtain a final panoramic image; in the sub-module, the local optimal key frame is used as a reference image to complete the splicing and fusion of all the two adjacent images;
the process of splicing the two adjacent images is as follows: calculating a roughly overlapped area of two adjacent images by using pose information, obtaining a Feature point by using a Scale Invariant Feature Transform (SIFT) algorithm in the overlapped area, roughly matching the Feature point by using a Best-First search (BBF) algorithm, finely matching the Feature point by using a Progressive Sample Consensus (PROSAC) algorithm, calculating a homography matrix between the two adjacent images by using a Direct Linear Transform (DLT) algorithm, taking one image as a reference image and the other image as a target image, projecting the target image to the reference image by using the homography matrix, and completing splicing of the two adjacent images;
the process of fusing two adjacent images is as follows: obtaining an optimal suture line in an overlapping area of the two images to be spliced by using an optimal suture line algorithm based on dynamic programming; taking the whole overlapping area part in the reference image as a background image, taking the overlapping area part close to one side of the suture line of the target image part in the target image as an image area to be cloned, and carrying out Poisson fusion on the overlapping area part to complete final image fusion.
Compared with the prior art, the invention has the following advantages and effects:
(1) the invention combines the video frames acquired from different systems with the pose information, so that each video frame of the underwater video image corresponds to a piece of pose information, and the video frames and the pose information are subjected to time calibration.
(2) According to the invention, the homography matrix between the video frames is calculated by using the pose information, so that the overlapping area between the video frames is obtained, the key frames are extracted according to the proportion of the overlapping area, the splicing failure caused by the undersize of the overlapping area between the extracted adjacent key frames is avoided, or the operation time is increased due to the oversized overlapping area, and the key frames can be extracted quickly.
(3) The invention provides an improved shallow water image enhancement method based on self-adaptive parameter acquisition and relative global histogram stretching, which has good light homogenizing and defogging effects on an underwater video image with uneven illumination.
(4) The invention selects global and local projection planes from the key frame by combining the pose information, divides the key frame into a plurality of sub-modules according to the structure of the binary tree, and adopts the method of firstly splicing locally and then splicing integrally, thereby reducing the accumulated error generated by splicing multiple images.
(5) In the feature point extraction process, the overlapping area is calculated by using the pose information, the feature points are extracted in the overlapping area, the feature point extraction range is narrowed, and the time for extracting the feature points and matching the feature points is effectively shortened.
Drawings
FIG. 1 is a block diagram of an underwater video image restoration and splicing device based on pose information disclosed in the present invention;
FIG. 2 is a flowchart of the operation of a pose information-based underwater video image restoration and stitching method disclosed in the present invention;
FIG. 3 is a flow chart of the present invention for acquiring an underwater video image with a synchronized calibration signal and pose information of a camera;
FIG. 4 is a flow chart of the present invention for mapping video frames and pose information collected from different systems;
FIG. 5 is a flow chart of the present invention for obtaining keyframes in conjunction with pose information;
FIG. 6 is a flow chart of the present invention for calculating the overlap area of two images using pose information;
fig. 7 is a flow chart of underwater image restoration in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Examples
As shown in fig. 2, the embodiment provides a pose information-based underwater video image restoration and stitching method.
S1, acquiring the underwater video image with the synchronous calibration signal and the pose information of the camera through the underwater video image acquisition device,
the underwater video image acquisition device is structurally shown in fig. 1 and comprises a camera module and a pose information recording module, wherein the camera module consists of a camera and a first SD card, the camera is used for shooting underwater video images, and the first SD card is used for storing video files with synchronous calibration signals; the pose information recording module consists of a main controller, a gyroscope, a GPS (global positioning system), an underwater ultrasonic distance measuring sensor, a signal lamp and a second SD (secure digital) card, wherein the main controller is used for controlling the work of other units, the gyroscope is used for acquiring the pose information of the camera, the GPS is used for acquiring the position information of the camera, the underwater ultrasonic distance measuring sensor is used for acquiring the height of the camera from the water bottom, the signal lamp is used for sending an optical signal to the camera module, and the second SD card is used for storing pose information
The flow of step S1 is shown in fig. 3;
s101, starting a camera module to start shooting underwater video images;
s102, starting a pose information recording module, controlling the time of 6 periods (the camera and the pose information recording module adopt the same frequency) of red (considering that an underwater image presents blue-green color cast and the color of signal light is red) signal lamp lighting, and then extinguishing the time of 1 period;
s103, controlling the time of 1 period of lighting of the red signal lamp by the pose information recording module, simultaneously recording pose information, setting Flag of the current pose information to be 1, and setting the period number n to be 0;
s104, setting the periodicity n as n + 1;
s105, judging whether 3000 periods pass or not, if so, executing a step S103, and if not, executing a step S106;
s106, the pose information recording module records the current pose information and executes the step S107;
and S107, whether the data collection is stopped, if not, the step S104 is executed, and if so, the step S1 is ended.
S2, carrying out one-to-one correspondence on the video frames and the pose information of the underwater video images, wherein the flow of the step S2 is shown in FIG. 4;
s201, reading the underwater video image obtained in the step S1, and taking a first frame of the video as a current video frame;
s202, judging whether the current video frame is a signal frame or not, and if so, executing a step S203; if not, executing step S204;
s203, respectively detecting 5 frames forwards and backwards, calculating the total number of the continuously detected signal frames, if the total number is greater than or equal to 6, executing a step S204, otherwise executing a step S205;
s204, taking the 7 th frame from the 1 st frame to the next frame after the continuous detection of the signal frames as a current video frame, judging whether the current video frame is the signal frame, if not, executing the step S205, if so, the current video frame is the video frame corresponding to the first piece of pose information, and then, each video frame corresponds to each piece of pose information one by one, and ending the step S2;
s205, judging whether a 6 th frame behind the current video frame exists, if so, reading the 6 th frame behind the current video frame as the current video frame, repeatedly executing the steps S202 to S205, and if not, ending the step S2;
the process of judging whether the current video frame is a signal frame is as follows:
uniformly dividing a current video frame into an upper region, a lower region, a left region and a right region, and respectively calculating whether the proportion of the number of pixels with values larger than a threshold value alpha1 in a channel R in each region in the total pixels of the whole region is larger than a threshold value alpha2, and whether the proportion of the number of pixels with values smaller than a threshold value alpha1 in a channel G, B in the total pixels of the whole region is larger than a threshold value alpha 2; if the value distribution of R, G, B three channels in one area meets the above condition, the current video frame is a signal frame; otherwise, the current video frame is not a signal frame;
in this embodiment, the value of the threshold value alpha1 is 150, and the value of the threshold value alpha2 is 0.8.
S3, calculating homography matrixes among the video frames by using the pose information, further solving the overlapping area among the video frames, extracting key frames according to the proportion of the overlapping area, and performing time calibration on the video frames and the pose information; the flow of this step is shown in FIG. 5;
s301, taking the video frame corresponding to the first piece of pose information as a current frame, and taking the first piece of pose information as pose information of the current frame;
s302, judging whether the Flag of the pose information of the current frame is 1, if so, executing a step S303, and if not, executing a step S305;
s303, judging whether the current frame is a signal frame, if so, indicating that no time deviation occurs, executing a step S309, otherwise, indicating that the time deviation occurs, and executing a step S304;
s304, detecting N frames forwards and backwards respectively, setting the signal frame as a current frame if the signal frame is detected, executing the step S309, and ending the step S3 if the signal frame is not detected;
s305, judging whether absolute values of a roll angle and a pitch angle of the current frame are smaller than a threshold beta1, if not, executing a step S309, and if so, executing a step S306;
s306, judging whether the key frame buffer queue is empty, if so, executing a step S308, and if not, executing a step S307;
s307, calculating the overlapping area of the current frame and the last frame in the key frame buffer queue by combining the pose information, judging whether the ratio of the overlapping area to the area of the current frame is in a beta2 range, executing the step S308 if the ratio is in the beta2 range, and executing the step S309 if the ratio is not in the beta2 range;
s308, extracting the current frame as a key frame, storing the key frame into a key frame buffer queue, and executing the step S309;
s309, judging whether a next video frame and next pose information exist, if so, reading the next video frame as a current frame, reading the next pose information as the pose information of the current frame, and repeatedly executing the step S302-the step S309, otherwise, ending the step S3.
In this embodiment, the value of the threshold beta1 is 3, the value of beta2 ranges from 0.3 to 0.8, and N is 5.
The flow of calculating the overlap area in step S307 is shown in fig. 6, and the procedure is as follows:
s3071, taking the last frame in the key frame buffer queue as the previous key frame, taking the shooting position of the previous key frame as the origin of coordinates, taking the true west as the positive direction of an X axis, the true north as the positive direction of a Y axis, and the true ground as the positive direction of a Z axis to establish a coordinate system, and calculating a homography matrix H from the current frame to the previous key frame by combining pose information, wherein the H is a matrix of 3X 3, and the calculation formula is as follows:
wherein H1The homography matrix for the last key frame to the projection plane,is H1Inverse matrix of, H2A homography matrix from the current frame to the projection plane;
H1the calculation formula of (a) is as follows:
H1=KR1K-1 (2)
wherein K is an internal parameter matrix of the camera and is obtained by calibrating the camera, R1A rotation matrix of 3 × 3;
wherein f represents the focal length of the camera; dx and dy respectively represent the physical size of each pixel on the key frame in the coordinate axis direction, and the unit is mm/pixel; (u)0,v0) Coordinates of an intersection point of the optical axis of the camera and the key frame plane are positioned at the center of the key frame;
wherein,θ1、ψ1respectively representing the roll angle, the pitch angle and the course angle of the camera during the shooting of the previous key frame;
H2the calculation formula of (a) is as follows:
wherein R is2Is a rotation matrix of 3 multiplied by 3, I is a unit matrix, t is a three-dimensional translation vector, d is the distance from the origin of coordinates to the water bottom plane, and n is a unit normal vector [0, 0, -1 ] of the water bottom plane],nTRepresents a transpose of the vector n;
wherein,θ2、ψ2respectively a roll angle, a pitch angle and a course angle of the camera when the current frame is shot;
t=[x2-x1,y2-y1,z2-z1]T (7)
x2-x1=(lon2-lon1)×111110×cos(lat1×π/180) (8)
y2-y1=(lat2-lat1)×111110 (9)
z2-z1=h2-h1 (10)
wherein (x)1,y1,z1) Coordinates representing the last key frame shot position, (x)2,y2,z2) Coordinates representing the shooting position of the current frame, lon1、lat1、h1Longitude, latitude and height from the bottom of the water, lon, for the last key frame shooting position2、lat2、h2Respectively representing the longitude and latitude of the shooting position of the current frame and the height from the water bottom;
s3072, calculating coordinate positions of four vertexes of the current frame after homography transformation, wherein a calculation formula is as follows:
position=vertex./vertex(3,:) (12)
wherein, width represents the width of the current frame, height represents the height of the current frame, vertex/vertex (3: 3) represents the normalization processing of the homogeneous coordinate to obtain the coordinate of the two-dimensional plane;
s3073, counting the vertexes of the quadrangle of the current frame after homography transformation in the rectangle represented by the last key frame;
s3074, calculating the intersection point of each side of the quadrangle of the current frame after homographic transformation and each side of the previous key frame;
s3075, sequencing the obtained vertexes and intersection points clockwise or anticlockwise;
s3076, calculating the area of the polygon in the overlapping area, dividing the polygon into a plurality of triangles by taking any vertex of the polygon as a fixed point, respectively calculating the area of each triangle and summing the areas to obtain the size of the overlapping area, wherein the calculation formula for calculating the area of any triangle is as follows:
S4, restoring the underwater video image by adopting an improved shallow water image enhancement algorithm which is obtained based on self-adaptive parameters and is stretched relative to the global histogram to obtain a restored key frame, wherein the flow of the step S4 is shown in FIG. 7;
s401, converting the key frame from an RGB color model into an HSV color model;
s402, convolving and weighting the bright channel V by Gaussian functions with different scales to obtain an estimated value I (x, y) of the illumination component;
wherein (x, y) represents the coordinates of the bright channel, V (x, y) represents the bright channel, G (x, y) represents the luminance of the bright channelg,yg) Representing a Gaussian function, ωlRepresenting the illumination component resulting from convolution of the bright channel V with the ith scale Gaussian functionA weighting coefficient; n is a radical oflIs the total number of Gaussian functions with different scales, i 1, 2lσ is the standard deviation of the Gaussian function, (x)g,yg) Representing the template coordinates of the Gaussian function, wherein the central position of the template is an origin;
s403, correcting the bright channel V by using a two-dimensional gamma function, wherein the formula of the two-dimensional gamma function is as follows:
wherein, VO(x, y) are the values of the corrected bright channel, γ is an index value for luminance enhancement, the index value contains the illumination component characteristics of the key frame, and m is the luminance mean of the illumination components;
s404, converting the key frame processed in the steps S402 and S403 from an HSV color model to an RGB color model, and solving a color balance coefficient theta of a channel G, Bg、θbAnd the pixel values of the channel G, B are multiplied by color equalization coefficients theta, respectivelyg、θbCarrying out color balance to obtain a balanced RGB color model; wherein theta isg、θbThe calculation formula of (a) is as follows:
wherein G isavg,BavgNormalized average values after recovery of channel G, B, M is the key frame height, N is the key frame width, Ig(I, j) is the value of channel G at (I, j), Ib(i, j) is the value of channel B at (i, j), i represents the ith column of channel G, B, j represents the jth row of channel G, B;
s405, carrying out global histogram stretching on the R, G, B three channels after equalization, wherein a calculation formula is as follows:
wherein, PinFor input of pixel values, PoutFor the image values after histogram stretching, IminIs the value of the pixel value of the channel at 0.5% position after being sorted from small to large, ImaxIs the value of the pixel at 99.5% position, O, after the pixel values of the channel are sorted from small to largemaxIs 255, OminIs 0;
s406, converting the keyframe processed in the step S405 from the RGB color model to the CIELab color model, and modifying the color grades of the channels a and b in the CIELab color model, wherein the calculation formula is as follows:
wherein,a color value representing a color value of an input channel,a value of the output color is represented,the optimal experimental values are shown in the figure,taking the empirical value of the raw material to be tested,taking the value as channel a or channel b;
and S407, converting the key frame processed in the step S406 from the CIELab color model to the RGB color model to obtain a restored key frame.
S5, dividing the restored key frame into a plurality of sub-modules by combining pose information;
traversing all the key frames, calculating the absolute value change from the pose information of any key frame to the pose information of other key frames, and taking the key frame with the minimum value accumulated by the absolute value changes of the pose information as a reference image; finding a globally optimal key frame, and taking the globally optimal key frame as a globally reference image; continuously recursively finding all locally optimal key frames on two sides of the globally optimal key frame, and taking the locally optimal key frames as local reference images; and dividing all the key frames into a plurality of sub-modules according to a binary tree structure by taking the globally optimal key frame as a root node and the locally optimal key frame as a child node.
The formula for calculating the absolute value change from the pose information of any key frame to the pose information of other key frames is as follows:
wherein iτ∈[1,N],Respectively represent the ithτRoll angle, pitch angle and height from the water bottom of the Zhang key frame,respectively represent the j thτRoll angle, pitch angle and height from water bottom of Zhang key frame, alpha, beta respectively represent weight occupied by influence brought by angle change and height change, NτRepresenting the total number of key frames.
S6, splicing the sub-modules into sub-module images, and then splicing the sub-module images into a final panoramic image;
firstly, splicing submodules at the bottommost layer of the binary tree, and upwards splicing layer by layer to obtain a final panoramic image; in the sub-module, the local optimal key frame is used as a reference image to complete the splicing and fusion of all the two adjacent images;
the process of splicing the two adjacent images is as follows: calculating a roughly overlapped area of two adjacent images by using pose information, obtaining a Feature point by using a Scale Invariant Feature Transform (SIFT) algorithm in the overlapped area, roughly matching the Feature point by using a Best-First search (BBF) algorithm, finely matching the Feature point by using a Progressive Sample Consensus (PROSAC) algorithm, calculating a homography matrix between the two adjacent images by using a Direct Linear Transform (DLT) algorithm, taking one image as a reference image and the other image as a target image, projecting the target image to the reference image by using the homography matrix, and completing splicing of the two adjacent images;
the process of fusing two adjacent images is as follows: obtaining an optimal suture line in an overlapping area of the two images to be spliced by using an optimal suture line algorithm based on dynamic programming; taking the whole overlapping area part in the reference image as a background image, taking the overlapping area part close to one side of the suture line of the target image part in the target image as an image area to be cloned, and carrying out Poisson fusion on the overlapping area part to complete final image fusion.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (10)
1. An underwater video image restoration and splicing method based on pose information is characterized by comprising the following steps:
s1, acquiring an underwater video image with a synchronous calibration signal and pose information of a camera through an underwater video image acquisition device;
s2, carrying out one-to-one correspondence on the video frames and the pose information of the underwater video images;
s3, calculating homography matrixes among the video frames by using the pose information, further solving the overlapping area among the video frames, extracting key frames according to the proportion of the overlapping area, and performing time calibration on the video frames and the pose information;
s4, performing underwater video image restoration by adopting an improved shallow water image enhancement algorithm which is obtained based on self-adaptive parameters and is stretched relative to the global histogram, and obtaining a restored key frame;
s5, dividing the restored key frame into a plurality of sub-modules by combining pose information;
and S6, splicing the sub-modules into sub-module images, and splicing the sub-module images into a final panoramic image.
2. The pose information-based underwater video image restoration and splicing method according to claim 1, wherein the underwater video image acquisition device comprises a camera module and a pose information recording module, wherein the camera module comprises a camera and a first SD card, the camera is used for shooting underwater video images, and the first SD card is used for storing the underwater video images with synchronous calibration signals; the pose information recording module consists of a main controller, a gyroscope, a GPS (global positioning system), an underwater ultrasonic distance measuring sensor, a signal lamp and a second SD (secure digital) card, wherein the main controller is used for controlling the work of other units, the gyroscope is used for acquiring the pose information of a camera, the GPS is used for acquiring the position information of the camera, the underwater ultrasonic distance measuring sensor is used for acquiring the height of the camera from the water bottom, the signal lamp is used for sending an optical signal to the camera module, and the second SD card is used for storing the pose information of the camera.
3. The pose information-based underwater video image restoration and splicing method according to claim 2, wherein the step S1 is as follows:
the method comprises the steps of firstly starting a camera module to start shooting underwater video images, then starting a pose information recording module, controlling a signal lamp to give a light signal 1 to the camera before the pose information recording module starts recording pose information, then starting recording the pose information, controlling the signal lamp to give a light signal 2 to the camera at intervals, and setting Flag of current pose information to be 1.
4. The pose information-based underwater video image restoration and splicing method according to claim 1, wherein the step S2 is as follows:
reading the underwater video image obtained in the step S1, taking the video frames detected to contain light as signal frames, taking a certain number of continuously detected signal frames as start signals, taking the last frame of all the detected signal frames as a video frame corresponding to the first piece of pose information, and then sequentially corresponding each frame of video frame to each piece of pose information one by one;
the process of judging whether the current video frame is a signal frame is as follows: uniformly dividing a current video frame into an upper region, a lower region, a left region and a right region, respectively counting whether the proportion of the number of pixels of which the numerical value of a channel R is greater than a threshold value alpha1 in the total pixels of the whole region is greater than a threshold value alpha2 or not, and whether the proportion of the number of pixels of which the numerical value of a channel G, B is less than a threshold value alpha1 in the total pixels of the whole region is greater than a threshold value alpha2 or not; if the value distribution of R, G, B three channels in one area meets the above condition, the current video frame is a signal frame; otherwise the current video frame is not a signal frame.
5. The pose information-based underwater video image restoration and splicing method according to claim 4, wherein the step S3 is as follows:
s301, taking the video frame corresponding to the first piece of pose information as a current frame, and taking the first piece of pose information as pose information of the current frame;
s302, judging whether the Flag of the pose information of the current frame is 1, if so, executing a step S303, and if not, executing a step S305;
s303, judging whether the current frame is a signal frame, if so, indicating that no time deviation occurs, executing a step S309, otherwise, indicating that the time deviation occurs, and executing a step S304;
s304, detecting N frames forwards and backwards respectively, setting the signal frame as a current frame if the signal frame is detected, executing the step S309, and ending the step S3 if the signal frame is not detected;
s305, judging whether absolute values of a roll angle and a pitch angle of the current frame are smaller than a threshold beta1, if not, executing a step S309, and if so, executing a step S306;
s306, judging whether the key frame buffer queue is empty, if so, executing a step S308, and if not, executing a step S307;
s307, calculating the overlapping area of the current frame and the last frame in the key frame buffer queue by combining the pose information, judging whether the ratio of the overlapping area to the area of the current frame is in a beta2 range, executing the step S308 if the ratio is in the beta2 range, and executing the step S309 if the ratio is not in the beta2 range;
s308, extracting the current frame as a key frame, storing the key frame into a key frame buffer queue, and executing the step S309;
s309, judging whether a next video frame and next pose information exist, if so, reading the next video frame as a current frame, reading the next pose information as the pose information of the current frame, and repeatedly executing the step S302-the step S309, otherwise, ending the step S3.
6. The pose information-based underwater video image restoration and splicing method according to claim 5, wherein the process of calculating the overlapping area in step S307 is as follows:
s3071, taking the last frame in the key frame buffer queue as the previous key frame, taking the shooting position of the previous key frame as the origin of coordinates, taking the true west as the positive direction of an X axis, the true north as the positive direction of a Y axis, and the true ground as the positive direction of a Z axis to establish a coordinate system, taking a water bottom plane as a projection plane, and calculating a homography matrix H from the current frame to the previous key frame by combining pose information, wherein H is a matrix of 3 multiplied by 3, and the calculation formula is as follows:
wherein H1The homography matrix for the last key frame to the projection plane,is H1Inverse matrix of, H2A homography matrix from the current frame to the projection plane;
H1the calculation formula of (a) is as follows:
H1=KR1K-1 (2)
wherein K is an internal parameter matrix of the camera and is obtained by calibrating the camera, R1A rotation matrix of 3 × 3;
wherein f represents the focal length of the camera; dx and dy respectively represent the physical size of each pixel on the key frame in the coordinate axis direction, and the unit is mm/pixel; (u)0,v0) Coordinates of an intersection point of the optical axis of the camera and the key frame plane are positioned at the center of the key frame;
wherein,θ1、ψ1respectively representing the roll angle, the pitch angle and the course angle of the camera during the shooting of the previous key frame;
H2the calculation formula of (a) is as follows:
wherein R is2Is a rotation matrix of 3 multiplied by 3, I is a unit matrix, t is a three-dimensional translation vector, d is the distance from the coordinate origin to the water bottom plane, n is a unit normal vector of the water bottom planeTRepresents a transpose of the vector n;
wherein,θ2、ψ2respectively a roll angle, a pitch angle and a course angle of the camera when the current frame is shot;
t=[x2-x1,y2-y1,z2-z1]T (7)
x2-x1=(lon2-lon1)×111110×cos(lat1×π/180) (8)
y2-y1=(lat2-lat1)×111110 (9)
z2-z1=h2-h1 (10)
wherein (x)1,y1,z1) Coordinates representing the last key frame shot position, (x)2,y2,z2) Coordinates representing the shooting position of the current frame,lon1、lat1、h1longitude, latitude and height from the bottom of the water, lon, for the last key frame shooting position2、lat2、h2Respectively representing the longitude and latitude of the shooting position of the current frame and the height from the water bottom;
s3072, calculating coordinate positions of four vertexes of the current frame after homography transformation, wherein a calculation formula is as follows:
position=vertex./vertex(3,:) (12)
wherein, width represents the width of the current frame, height represents the height of the current frame, vertex/vertex (3: 3) represents the normalization processing of the homogeneous coordinate to obtain the coordinate of the two-dimensional plane;
s3073, counting the vertexes of the quadrangle of the current frame after homography transformation in the rectangle represented by the last key frame;
s3074, calculating the intersection point of each side of the quadrangle of the current frame after homographic transformation and each side of the previous key frame;
s3075, sequencing the obtained vertexes and intersection points clockwise or anticlockwise;
s3076, the area of the polygon in the overlapping area is obtained, any vertex of the polygon is taken as a fixed point, the polygon is divided into a plurality of triangles, the area of each triangle is calculated respectively and summed, and the size of the overlapping area is obtained.
7. The pose information-based underwater video image restoration and splicing method according to claim 1, wherein the step S4 is as follows:
s401, converting the key frame from an RGB color model into an HSV color model;
s402, convolving and weighting the bright channel V by Gaussian functions with different scales to obtain an estimated value I (x, y) of the illumination component;
wherein (x, y) represents the coordinates of the bright channel, V (x, y) represents the bright channel, G (x, y) represents the luminance of the bright channelg,yg) Representing a Gaussian function, ωlRepresenting a weighting coefficient of an illumination component obtained by convolution of the ith scale Gaussian function on the bright channel V; n is a radical oflIs the total number of Gaussian functions with different scales, i 1, 2lσ is the standard deviation of the Gaussian function, (x)g,yg) Representing the template coordinates of the Gaussian function, wherein the central position of the template is an origin;
s403, correcting the bright channel V by using a two-dimensional gamma function, wherein the formula of the two-dimensional gamma function is as follows:
wherein, VO(x, y) are the values of the corrected bright channel, γ is an index value for luminance enhancement, the index value contains the illumination component characteristics of the key frame, and m is the luminance mean of the illumination components;
s404, converting the key frame processed in the steps S402 and S403 from an HSV color model to an RGB color model, and solving a color balance coefficient theta of a channel G, Bg、θbAnd the pixel values of the channel G, B are multiplied by color equalization coefficients theta, respectivelyg、θbCarrying out color balance to obtain a balanced RGB color model; wherein theta isg、θbThe calculation formula of (a) is as follows:
wherein G isavg、BavgNormalized average values after recovery of channel G, B, M is the key frame height, N is the key frame width, Ig(I, j) is the value of channel G at (I, j), Ib(i, j) is the value of channel B at (i, j), i represents the ith column of channel G, B, j represents the jth row of channel G, B;
s405, carrying out global histogram stretching on the R, G, B three channels after equalization, wherein a calculation formula is as follows:
wherein, PinFor input of pixel values, PoutFor the image values after histogram stretching, IminIs the value of the pixel value of the channel at 0.5% position after being sorted from small to large, ImaxIs the value of the pixel at 99.5% position, O, after the pixel values of the channel are sorted from small to largemaxIs 255, OminIs 0;
s406, converting the keyframe processed in the step S405 from the RGB color model to the CIELab color model, and modifying the color grades of the channels a and b in the CIELab color model, wherein the calculation formula is as follows:
wherein,a color value representing a color value of an input channel,a value of the output color is represented,the optimal experimental values are shown in the figure,taking the empirical value of the raw material to be tested,taking the value as channel a or channel b;
and S407, converting the key frame processed in the step S406 from the CIELab color model to the RGB color model to obtain a restored key frame.
8. The pose information-based underwater video image restoration and splicing method according to claim 1, wherein the step S5 is as follows:
traversing all the key frames, calculating the absolute value change from the pose information of any key frame to the pose information of other key frames, and taking the key frame with the minimum value accumulated by the absolute value changes of the pose information as a reference image; finding a globally optimal key frame, and taking the globally optimal key frame as a globally reference image; continuously recursively finding all locally optimal key frames on two sides of the globally optimal key frame, and taking the locally optimal key frames as local reference images; and dividing all the key frames into a plurality of sub-modules according to a binary tree structure by taking the globally optimal key frame as a root node and the locally optimal key frame as a child node.
9. The method for restoring and splicing the underwater video images based on the pose information according to claim 8, wherein the formula for calculating the absolute value change size from the pose information of any one key frame to the pose information of other key frames is as follows:
wherein iτ∈[1,N],Respectively represent the ithτRoll angle, pitch angle and height from the water bottom of the Zhang key frame,respectively represent the j thτRoll angle, pitch angle and height from water bottom of Zhang key frame, alpha, beta respectively represent weight occupied by influence brought by angle change and height change, NτRepresenting the total number of key frames.
10. The pose information-based underwater video image restoration and splicing method according to claim 8, wherein the step S6 is as follows: firstly, splicing submodules at the bottommost layer of the binary tree, and upwards splicing layer by layer to obtain a final panoramic image; in the sub-module, the local optimal key frame is used as a reference image to complete the splicing and fusion of all the two adjacent images;
the process of splicing the two adjacent images is as follows: calculating a rough overlapping area of the two adjacent images by using the pose information, solving feature points by using an SIFT algorithm in the overlapping area, performing rough matching of the feature points by using a BBF algorithm, performing fine matching of the feature points by using a PROSAC algorithm, calculating a homography matrix between the two adjacent images by using a DLT algorithm, taking one image as a reference image and the other image as a target image, projecting the target image to the reference image by using the homography matrix, and completing splicing of the two adjacent images;
the process of fusing two adjacent images is as follows: obtaining an optimal suture line in an overlapping area of the two images to be spliced by using an optimal suture line algorithm based on dynamic programming; taking the whole overlapping area part in the reference image as a background image, taking the overlapping area part close to one side of the suture line of the target image part in the target image as an image area to be cloned, and carrying out Poisson fusion on the overlapping area part to complete final image fusion.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110354434.1A CN113160053B (en) | 2021-04-01 | 2021-04-01 | Pose information-based underwater video image restoration and splicing method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110354434.1A CN113160053B (en) | 2021-04-01 | 2021-04-01 | Pose information-based underwater video image restoration and splicing method |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113160053A true CN113160053A (en) | 2021-07-23 |
CN113160053B CN113160053B (en) | 2022-06-14 |
Family
ID=76885917
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110354434.1A Active CN113160053B (en) | 2021-04-01 | 2021-04-01 | Pose information-based underwater video image restoration and splicing method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113160053B (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114449130A (en) * | 2022-03-07 | 2022-05-06 | 北京拙河科技有限公司 | Multi-camera video fusion method and system |
CN115131213A (en) * | 2022-07-27 | 2022-09-30 | 成都市晶林科技有限公司 | Real-time infrared binocular image splicing method and system |
CN115861079A (en) * | 2023-02-24 | 2023-03-28 | 和普威视光电股份有限公司 | Panoramic image splicing method and system without overlapping area and splicing terminal |
CN117132728A (en) * | 2023-10-26 | 2023-11-28 | 毫末智行科技有限公司 | Method and device for constructing map, electronic equipment and storage medium |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107067386A (en) * | 2017-04-24 | 2017-08-18 | 上海海洋大学 | A kind of shallow sea underwater picture Enhancement Method stretched based on relative color histogram |
CN108257089A (en) * | 2018-01-12 | 2018-07-06 | 北京航空航天大学 | A kind of method of the big visual field video panorama splicing based on iteration closest approach |
US20190221003A1 (en) * | 2015-12-30 | 2019-07-18 | Tsinghua University | Method and device for interactive calibration based on 3d reconstruction in 3d surveillance system |
CN110727009A (en) * | 2019-10-10 | 2020-01-24 | 武汉理工大学 | High-precision visual map construction and positioning method based on vehicle-mounted all-around image |
CN110736472A (en) * | 2019-10-10 | 2020-01-31 | 武汉理工大学 | indoor high-precision map representation method based on fusion of vehicle-mounted all-around images and millimeter wave radar |
CN111161154A (en) * | 2019-12-31 | 2020-05-15 | 航天图景(北京)科技有限公司 | Real-time and rapid orthoscopic splicing system and method for videos of unmanned aerial vehicle |
CN111951201A (en) * | 2019-05-16 | 2020-11-17 | 杭州海康机器人技术有限公司 | Unmanned aerial vehicle aerial image splicing method and device and storage medium |
CN112399188A (en) * | 2020-11-04 | 2021-02-23 | 贝壳技术有限公司 | Image frame splicing method and device, readable storage medium and electronic equipment |
-
2021
- 2021-04-01 CN CN202110354434.1A patent/CN113160053B/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190221003A1 (en) * | 2015-12-30 | 2019-07-18 | Tsinghua University | Method and device for interactive calibration based on 3d reconstruction in 3d surveillance system |
CN107067386A (en) * | 2017-04-24 | 2017-08-18 | 上海海洋大学 | A kind of shallow sea underwater picture Enhancement Method stretched based on relative color histogram |
CN108257089A (en) * | 2018-01-12 | 2018-07-06 | 北京航空航天大学 | A kind of method of the big visual field video panorama splicing based on iteration closest approach |
CN111951201A (en) * | 2019-05-16 | 2020-11-17 | 杭州海康机器人技术有限公司 | Unmanned aerial vehicle aerial image splicing method and device and storage medium |
CN110727009A (en) * | 2019-10-10 | 2020-01-24 | 武汉理工大学 | High-precision visual map construction and positioning method based on vehicle-mounted all-around image |
CN110736472A (en) * | 2019-10-10 | 2020-01-31 | 武汉理工大学 | indoor high-precision map representation method based on fusion of vehicle-mounted all-around images and millimeter wave radar |
CN111161154A (en) * | 2019-12-31 | 2020-05-15 | 航天图景(北京)科技有限公司 | Real-time and rapid orthoscopic splicing system and method for videos of unmanned aerial vehicle |
CN112399188A (en) * | 2020-11-04 | 2021-02-23 | 贝壳技术有限公司 | Image frame splicing method and device, readable storage medium and electronic equipment |
Non-Patent Citations (1)
Title |
---|
林森等: "水下光学图像中目标探测关键技术研究综述", 《激光与光电子学进展》, no. 06 * |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114449130A (en) * | 2022-03-07 | 2022-05-06 | 北京拙河科技有限公司 | Multi-camera video fusion method and system |
CN115131213A (en) * | 2022-07-27 | 2022-09-30 | 成都市晶林科技有限公司 | Real-time infrared binocular image splicing method and system |
CN115861079A (en) * | 2023-02-24 | 2023-03-28 | 和普威视光电股份有限公司 | Panoramic image splicing method and system without overlapping area and splicing terminal |
CN115861079B (en) * | 2023-02-24 | 2023-05-12 | 和普威视光电股份有限公司 | Panoramic image stitching method and system without overlapping area and stitching terminal |
CN117132728A (en) * | 2023-10-26 | 2023-11-28 | 毫末智行科技有限公司 | Method and device for constructing map, electronic equipment and storage medium |
CN117132728B (en) * | 2023-10-26 | 2024-02-23 | 毫末智行科技有限公司 | Method and device for constructing map, electronic equipment and storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN113160053B (en) | 2022-06-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN113160053B (en) | Pose information-based underwater video image restoration and splicing method | |
CN110211043B (en) | Registration method based on grid optimization for panoramic image stitching | |
CN110570371B (en) | Image defogging method based on multi-scale residual error learning | |
CN115439424B (en) | Intelligent detection method for aerial video images of unmanned aerial vehicle | |
CN110363116B (en) | Irregular human face correction method, system and medium based on GLD-GAN | |
CN113065558A (en) | Lightweight small target detection method combined with attention mechanism | |
CA2662355C (en) | Mosaic oblique images and methods of making and using same | |
US20060078214A1 (en) | Image processing based on direction of gravity | |
CN110473221B (en) | Automatic target object scanning system and method | |
CN110458877B (en) | Navigation method based on bionic vision for fusing infrared and visible light information | |
CN110473185A (en) | Image processing method and device, electronic equipment, computer readable storage medium | |
CN109711268B (en) | Face image screening method and device | |
CN110276831B (en) | Method and device for constructing three-dimensional model, equipment and computer-readable storage medium | |
CN114973028B (en) | Aerial video image real-time change detection method and system | |
CN113298810A (en) | Trace detection method combining image enhancement and depth convolution neural network | |
CN112184604A (en) | Color image enhancement method based on image fusion | |
CN112348775B (en) | Vehicle-mounted looking-around-based pavement pit detection system and method | |
CN102982524B (en) | Splicing method for corn ear order images | |
CN111582074A (en) | Monitoring video leaf occlusion detection method based on scene depth information perception | |
CN112633274A (en) | Sonar image target detection method and device and electronic equipment | |
CN116091314A (en) | Infrared image stitching method based on multi-scale depth homography | |
CN115619623A (en) | Parallel fisheye camera image splicing method based on moving least square transformation | |
CN117456371B (en) | Group string hot spot detection method, device, equipment and medium | |
CN112884795A (en) | Power transmission line inspection foreground and background segmentation method based on multi-feature significance fusion | |
CN116152442B (en) | Three-dimensional point cloud model generation method and device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |