WO2023130842A1 - 一种相机位姿确定方法和装置 - Google Patents

一种相机位姿确定方法和装置 Download PDF

Info

Publication number
WO2023130842A1
WO2023130842A1 PCT/CN2022/132927 CN2022132927W WO2023130842A1 WO 2023130842 A1 WO2023130842 A1 WO 2023130842A1 CN 2022132927 W CN2022132927 W CN 2022132927W WO 2023130842 A1 WO2023130842 A1 WO 2023130842A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
key frames
feature
pose
adjacent
Prior art date
Application number
PCT/CN2022/132927
Other languages
English (en)
French (fr)
Inventor
赵德力
彭登
陶永康
傅志刚
曾阳露
Original Assignee
广东汇天航空航天科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广东汇天航空航天科技有限公司 filed Critical 广东汇天航空航天科技有限公司
Publication of WO2023130842A1 publication Critical patent/WO2023130842A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows

Definitions

  • the present application relates to the technical field of image processing, in particular to a camera pose determining method and a camera pose determining device.
  • Visual odometry estimates mileage based on visual images, and is widely used in the positioning of mobile devices in unknown environments.
  • the visual odometry architecture is usually divided into two parts: the front end and the back end.
  • the front end is responsible for constructing multi-view geometric constraints, and the back end performs nonlinear least squares optimization based on the reprojection error of the front end.
  • the feature point matching method uses the matched feature points to construct the common-view constraint between multiple image frames.
  • the disadvantages of matching The optical flow method does not calculate and match descriptors, and directly tracks feature points through optical flow, which greatly reduces the computational overhead, but also has the problems of relatively low accuracy and easy tracking.
  • both the feature point method and the optical flow method need to rely on the stability of the feature points in the scene. Therefore, in some weakly textured scenes with sparse feature points, the two front-end constraint methods have more or less stability problems.
  • the embodiments of the present application are proposed to provide a method for determining a camera pose and a corresponding device for determining a camera pose that overcome the above problems or at least partially solve the above problems.
  • a camera pose determination method including:
  • multiple image frames collected by the camera multiple image frames include key frames and non-key frames;
  • an optimized camera pose corresponding to the current key frame is determined.
  • the method also includes:
  • determine the optimal camera pose corresponding to the current keyframe including:
  • the second camera relative pose between adjacent keyframes and the third camera pose between adjacent keyframes determine the current keyframe corresponding The optimized camera pose of .
  • the second camera relative pose between adjacent key frames and the third camera pose between adjacent key frames to determine the optimized camera pose corresponding to the current keyframe, including:
  • the optimal camera pose corresponding to the current key frame is determined.
  • extracting corner points satisfying preset feature point conditions as feature points from multiple image blocks includes:
  • Random sampling consensus algorithm to screen target corners from candidate corners.
  • determining the degree of dispersion of initial corner points in each image block includes:
  • the degree of dispersion is the pixel distance and value of each initial corner point in the image block, and according to the degree of dispersion of the initial corner points in each image block and the corner response value, a corresponding screening ratio is set for each image block, include:
  • the screening ratio of each image block is determined.
  • feature points are extracted from key frames, and the relative pose of the second camera is determined according to feature points matched between adjacent key frames, including:
  • the relative pose of the second camera between the adjacent key frames is determined according to the matching feature points between the adjacent key frames.
  • extracting a line segment satisfying a preset feature line condition from a key frame as a feature line includes:
  • the preset feature line conditions include: the length of the line segment is greater than or equal to the preset length threshold, and the distance between the intersection point of the line segment pair and the line segment pair is less than or equal to the preset Distance threshold, the intersection of line segment pairs is within the image.
  • determining the third camera pose between adjacent key frames includes:
  • the third camera pose between adjacent keyframes is determined.
  • the embodiment of the present application also discloses a camera pose determining device, including:
  • the image acquisition module is used to acquire multiple image frames collected by the camera; the multiple image frames include key frames and non-key frames;
  • the first pose determination module is used to divide non-key frames into multiple image blocks, extract corner points satisfying preset feature point conditions from multiple image blocks as feature points, and The matched feature points use the optical flow method to determine the relative pose of the first camera;
  • the second pose determination module is used to extract feature points from key frames, and determine the relative pose of the second camera by using feature point matching method according to feature points matched between adjacent key frames;
  • the optimized pose determination module is used to determine the optimized camera pose corresponding to the current key frame according to the relative pose of the first camera between multiple adjacent non-key frames and the relative pose of the second camera between adjacent key frames .
  • the embodiment of the present application also discloses an electronic device, including: a processor, a memory, and a computer program stored on the memory and capable of running on the processor.
  • a computer program stored on the memory and capable of running on the processor.
  • the embodiment of the present application also discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above steps of the method for determining the camera pose are realized.
  • multiple image frames collected by the camera can be acquired, including key frames and non-key frames; the non-key frames are divided into multiple image blocks, and corner points that meet the preset feature point conditions are extracted from the multiple image blocks
  • the optical flow method is used to determine the relative pose of the first camera; the feature points are extracted from the key frames, and the matching feature points between adjacent key frames are used.
  • the feature point matching method determines the relative pose of the second camera; according to the relative pose of the first camera between multiple adjacent non-key frames and the relative pose of the second camera between adjacent key frames, determine the relative pose of the current key frame Optimize camera pose.
  • the embodiment of the present application integrates the respective advantages of the optical flow tracking algorithm and the feature point matching algorithm, and can achieve a balance between pose accuracy and processing efficiency; and by using the optical flow method to determine uniformly distributed feature points for non-key frames, Determining the relative pose of the first camera between adjacent non-key frames can improve the calculation accuracy of the relative pose of the first camera and improve the calculation efficiency and stability of the optical flow method.
  • FIG. 1 is a flow chart of the steps of a camera pose determination method provided in an embodiment of the present application
  • Fig. 2 is the flowchart of Kalman filtering combined with feature point matching method and optical flow method in the embodiment of the present application;
  • FIG. 3 is a flow chart of steps of another camera pose determination method provided in an embodiment of the present application.
  • FIG. 4 is a schematic diagram of uniformly screening feature points from each image block in the embodiment of the present application.
  • Fig. 5 is a schematic diagram of determining the reprojection error of the characteristic line
  • FIG. 6 is a flow chart of a visual odometry optimization method based on camera pose optimization in an embodiment of the present application
  • Fig. 7 is a structural block diagram of an apparatus for determining a camera pose provided by an embodiment of the present application.
  • Visual odometer VIO Vehicle Inertial Odometry
  • IMU Inertial Measurement Unit
  • the IMU can accurately measure the motion in a short time.
  • the IMU measurement can detect the motion between frames and provide constraints between frames to ensure that the system continues to run.
  • the fusion of vision and IMU will also make the pose estimation more accurate.
  • the core idea of the embodiment of the present application is to integrate the advantages of the optical flow tracking algorithm and the feature point matching algorithm to achieve a balance between pose accuracy and processing efficiency; and to determine the uniform distribution of feature points for non-key frames
  • the optical flow method is used to determine the relative pose of the first camera between adjacent non-key frames, which can provide the calculation accuracy of the relative pose of the first camera and improve the calculation efficiency and stability of the optical flow method.
  • FIG. 1 shows a flow chart of the steps of a method for determining a camera pose provided by an embodiment of the present application.
  • the method may specifically include the following steps:
  • Step 101 acquiring a plurality of image frames collected by a camera; the plurality of image frames include key frames and non-key frames.
  • cameras can be installed on mobile devices such as vehicles and aircraft, and image sequences of the surrounding environment can be collected through the cameras.
  • the image sequence may include multiple image frames, the image frames may be key frames and non-key frames, and the key frames are representative image frames.
  • a certain threshold such as 0.02
  • FIG. 2 it is a flow chart of Kalman filtering combined with the feature point matching method and the optical flow method in the embodiment of the present application.
  • the key frame image is optimized by the feature point matching algorithm to estimate the camera pose
  • the non-key frame is determined by the optical flow method and accumulated
  • the camera pose obtained by the feature point matching algorithm is compared with the camera pose obtained by the optical flow method
  • the pose is fused with a Kalman filter to obtain an optimized camera pose.
  • Step 102 divide the non-key frame into multiple image blocks, extract the corner points satisfying the preset feature point conditions from the multiple image blocks as feature points, and use light to match feature points between adjacent non-key frames
  • the flow method determines the relative pose of the first camera.
  • the movement of the object in the real scene will not only cause the corresponding point on the image to move, but also cause the brightness mode of the corresponding pixel on the image to change.
  • the so-called optical flow is the speed of the movement of the pixel brightness pattern on the image, or the pixel brightness change caused by the movement of the light-emitting part of the object and projected onto the image plane.
  • the optical flow method can determine the relative camera pose between adjacent image frames based on the optical flow constraints of pixels between adjacent image frames.
  • the optical flow method uses corner points for optical flow tracking, and corner points refer to points in the image where the gray value changes greatly. Corner detection methods may include: a Harris corner detection algorithm, an ORB corner detection algorithm, a FAST detection algorithm, and the like.
  • the optical flow method is used to track the feature points in the non-key frames, and the optical flow tracking tends to have the problem that the tracking points are concentrated in a small image area.
  • the improved corner point detection method can be used to uniformly screen and extract feature points in the image frame, and then the random sample concensum algorithm RANSAC (random sample concensum) can be used to further improve the homogenized corner points. Handling to reduce possible pairing errors.
  • non-key frames can be divided into multiple image blocks, and the size of each image block is the same, and a preset number of feature points are extracted from each image block, which can ensure that the feature points in the entire image Evenly distributed.
  • the image frame may be divided into 9 image blocks of the same size, and the corner points satisfying the preset feature point conditions are separately extracted as feature points for each image block.
  • the preset feature point condition can be used to evenly filter corner points from the image block as feature points.
  • the optical flow method is used to determine the relative pose of the first camera between adjacent non-key frames, which can provide the calculation accuracy of the relative pose of the first camera and improve the optical flow. computational efficiency and stability of the method.
  • Step 103 extract feature points from the key frames, and determine the relative pose of the second camera by using the feature point matching method according to the feature points matched between adjacent key frames.
  • the feature point-based matching method needs to extract feature points and determine descriptors, and determine the matching feature points based on the feature point descriptors. Since the matching takes a long time, only the feature point matching method is used for key frames. Based on the feature point matching method, the feature points in the two key frames are used to determine the relative pose of the second camera between the two key frames.
  • Step 104 Determine an optimized camera pose corresponding to the current key frame according to the first camera relative pose between multiple adjacent non-key frames and the second camera relative pose between adjacent key frames.
  • the optimal camera position corresponding to the current key frame can be determined according to the relative pose of the first camera between multiple adjacent non-key frames between adjacent key frames and the relative pose of the second camera between adjacent key frames posture.
  • multiple image frames collected by the camera can be acquired, including key frames and non-key frames; the non-key frames are divided into multiple image blocks, and corner points that meet the preset feature point conditions are extracted from the multiple image blocks
  • the optical flow method is used to determine the relative pose of the first camera; the feature points are extracted from the key frames, and the matching feature points between adjacent key frames are used.
  • the feature point matching method determines the relative pose of the second camera; according to the relative pose of the first camera between multiple adjacent non-key frames and the relative pose of the second camera between adjacent key frames, determine the relative pose of the current key frame Optimize camera pose.
  • the embodiment of the present application integrates the respective advantages of the optical flow tracking algorithm and the feature point matching algorithm, and can achieve a balance between pose accuracy and processing efficiency; and by using the optical flow method to determine uniformly distributed feature points for non-key frames, Determining the relative pose of the first camera between adjacent non-key frames can improve the calculation accuracy of the relative pose of the first camera and improve the calculation efficiency and stability of the optical flow method.
  • FIG. 3 shows a flow chart of the steps of another camera pose determination method provided in the embodiment of the present application.
  • the method may specifically include the following steps:
  • Step 301 acquiring a plurality of image frames collected by a camera; the plurality of image frames include key frames and non-key frames.
  • Step 302 divide the non-key frame into multiple image blocks, extract the corner points satisfying the preset feature point conditions from the multiple image blocks as feature points, and use light to match feature points between adjacent non-key frames
  • the flow method determines the relative pose of the first camera.
  • the improved Harris corner detection algorithm can be used to divide the non-key frame into multiple image blocks, and extract the corner points satisfying the preset feature point conditions as feature points from the multiple image blocks.
  • the step of extracting corner points satisfying preset feature point conditions as feature points from multiple image blocks may include the following sub-steps:
  • Sub-step S11 performing low-pass filtering processing on multiple image blocks.
  • the traditional Harris corner detection algorithm When using the Harris corner detection algorithm, the traditional Harris corner detection algorithm first performs Gaussian smoothing on the image, that is, uses Gaussian Gaussian to filter the image.
  • Gaussian smoothing will greatly weaken the high-frequency part (the edge of the object), the edge will slow down, and the image histogram will be compressed. It leads to the loss of corner points when doing non-maximum suppression.
  • the improved Harris corner detection algorithm of the present application can perform low-pass filtering on multiple image blocks first.
  • a cubic B-spline function with a low-pass characteristic can be used instead of a Gaussian function for smoothing.
  • Sub-step S12 calculating the cornering response values for the pixels in each image block processed by the low-pass filter.
  • M is the matrix representation of the autocorrelation function
  • ⁇ 1 and ⁇ 2 are the two eigenvalues of the matrix M, expressed as the first-order curvature of the autocorrelation function.
  • sub-step S13 respectively sort the cornerization response values corresponding to each pixel in each image block, and select a preset number of pixels as initial corners according to the sorting result.
  • the value of k is different in different image blocks, and the value of k should ensure that each image block has a considerable number of initial corner points.
  • Sub-step S14 determining the degree of dispersion of the initial corner points in each image block.
  • substep S14 may further include:
  • Sub-step S141 clustering the initial corner points of the image block to obtain the cluster center
  • Sub-step S142 determine the pixel distance from each initial corner point to the cluster center, and determine the dispersion degree according to the pixel distance from each initial corner point to the cluster center.
  • the sum of pixel distances from each initial corner point to the cluster center may be used as the degree of dispersion of the initial corner points in the image block.
  • the average value of pixel distances from each initial corner point to the cluster center may be used as the degree of dispersion of the initial corner points in the image block.
  • Sub-step S15 according to the degree of dispersion of the initial corner points in each image block and the corner point response value, set a corresponding screening ratio for each image block.
  • Screening ratio is the ratio of screening and retaining corner points from the initial corner points. In order to ensure that corner points of each image block are retained and utilized, and the more scattered and the higher the response value, the corner points should be retained.
  • a corresponding screening ratio can be set for each image block according to the degree of dispersion of the initial corner points in each image block and the corner point response value.
  • substep S15 may further include:
  • Sub-step S151 calculate the mean square error sum value for the corner point response value of the initial corner point of the image block.
  • the evaluation parameters are calculated according to the sum of pixel distances of each initial corner point in the image block and the sum of mean square errors of cornerization response values.
  • the screening ratio of each image block is determined according to the evaluation parameters of each image block.
  • the image blocks may be sorted according to the evaluation parameters of each image block, and the screening ratio of each image block is determined according to the sorting result. Among them, the higher the ranking, the greater the screening ratio.
  • FIG. 4 it is a schematic diagram of uniformly screening feature points from each image block in the embodiment of the present application. For image blocks with relatively concentrated initial corner points, more initial corner points are deleted. For image blocks with scattered initial corner points, more initial corner points are retained, and each image block can retain a considerable number of corner points after screening.
  • Sub-step S16 according to the screening ratio of each image block, select candidate corner points from the corresponding initial corner points.
  • sub-step S17 a random sampling consensus algorithm is used to select target corners from candidate corners.
  • the consensus algorithm can be randomly sampled to remove outliers from the candidate corners and use the retained candidate corners as the target corners.
  • Random sampling consensus algorithm in a given set P composed of N data points, it is assumed that most of the points in the set can be generated by a model, and at least n points (n ⁇ N) can be fitted.
  • the parameters of the model, the points that do not conform to the fitting model are outliers.
  • the following iterative method can be used for fitting, and the specific steps are: a. randomly select n data points from P; b. use these n data points to fit a model M; c. use P For the remaining data points in , calculate the distance between each point and the model M. If the distance exceeds the threshold, it is considered as an outboard point, otherwise it is considered as an internal point, and the tree amount m of the internal point corresponding to the model is recorded; d. Iteration After k times, the model M with the largest m is selected as the fitting result.
  • Step 303 extract feature points from the key frames, and determine the relative pose of the second camera by using a feature point matching method according to the feature points matched between adjacent key frames.
  • step 303 may include the following sub-steps:
  • Sub-step S21 setting a window with a preset pixel size centered on the feature point in the key frame.
  • Sub-step S22 determining the absolute value of the difference between the gray value of the feature point and the gray value of other pixels in the window.
  • an adjacency matrix is generated according to the absolute value of the gray difference between the feature point and other pixel points in the window.
  • Sub-step S24 according to the adjacency matrix, generate a description vector as a descriptor of the feature point.
  • Sub-step S25 according to the location information and descriptors of the feature points of the adjacent key frames, determine the matching feature points between the adjacent key frames.
  • a 3 ⁇ 3 pixel-sized window can be set with the feature point as the center, and the difference between the window center and the 8-field pixel points and the special gray value can be calculated.
  • the absolute value of the difference. I p represents the center point, that is, the feature point, and I xi is the gray value of the 8-field.
  • Sub-step S26 determining the relative pose of the second camera between the adjacent key frames according to the matching feature points between the adjacent key frames.
  • Step 304 Perform Kalman filter fusion according to the relative pose of the first camera between multiple adjacent non-key frames and the relative pose of the second camera between adjacent key frames, and determine the estimated camera pose corresponding to the current key frame .
  • the results of optical flow tracking are fused with the matching results of optimized feature points.
  • Kalman filtering is performed at each key frame, and the cumulative error of non-key frame optical flow tracking between the two key frames can be corrected by using the high-precision feature point matching results of the two key frames.
  • Kalman filtering is divided into two steps: prediction and update:
  • the relative pose between two key frames is accumulated by the optical flow tracking method as an estimate, and the relative pose of the camera obtained by optimal matching of feature points is used as an observation.
  • a k is the state transition matrix, which represents the camera relative pose transformation accumulated by the optical flow method from the previous key frame to the current key frame; ⁇ k ⁇ (0, R) represents the Gaussian noise of the motion equation; is the corrected state estimation and covariance corresponding to the last key frame; R is the covariance of the noise, set to a constant; and is the state estimate and covariance of the current keyframe.
  • z k is the camera pose obtained by the feature point method at the current moment, C is set as the identity matrix, and ⁇ k ⁇ N(0,Q) represents the observation noise. Since the error of the state estimation equation (optical flow method to estimate the pose) is larger than that of the feature point method, the Q value is generally smaller than the R value.
  • the Kalman gain K is calculated first, and then the state estimation and covariance of the current key frame are corrected to obtain the fused estimated camera pose and covariance
  • Step 305 according to the estimated camera pose corresponding to the current key frame, determine the feature point reprojection error for the current key frame.
  • the position information corresponding to the feature point P1′ at the moment of the current key frame can be calculated; the feature point P2 matching the feature point P1 can be determined from the current key According to the position information of P1′ and P2, the feature point reprojection error can be calculated.
  • feature point reprojection errors can be calculated based on multiple feature points between adjacent keyframes.
  • Step 306 extracting a line segment satisfying a preset feature line condition from the key frame as a feature line.
  • step 306 may include the following sub-steps:
  • Sub-step S31 detecting the line segment in the key frame, using two non-parallel line segments as a line segment pair.
  • Sub-step S32 for the line segment pair, select the line segment pair that meets the preset feature line condition as the feature line;
  • the preset feature line condition includes: the length of the line segment is greater than or equal to the preset length threshold, and the distance between the intersection point of the line segment pair and the line segment pair is less than Or equal to the preset distance threshold, the intersection point of the pair of line segments is within the image.
  • the non-parallel line segments (line segment extension lines) in the image there must be an intersection point, and usually the number of internal elements of such a line segment set will be very large.
  • do the following screening remove the line segment pair whose distance from the intersection point to the line segment pair is greater than or equal to the preset distance threshold, and keep the line segment pair whose distance from the intersection point to the line segment pair is less than or equal to the preset distance threshold value; remove the intersection point For the line segment pairs outside the image, keep the line segment pairs whose intersection points are within the image; remove the line segment pairs whose length is less than the preset length threshold, and keep the line segment pairs whose length is greater than or equal to the preset length threshold; the obtained line segment after the above screening is a valid feature line segment.
  • Step 307 Determine the pose of the third camera between the adjacent key frames according to the matching feature lines between the adjacent key frames.
  • step 307 may include the following sub-steps:
  • Sub-step S42 using the acute angle bisector of the intersection point of the feature line segment pair as the direction quantity of the descriptor, and taking the pixel size of the pixel block at the center of the intersection point as the scale quantity of the descriptor, determine the descriptor of the feature line of the feature line segment pair.
  • the description of line segment features may be based on the intersection points of feature line segment pairs.
  • the multi-scale rotation BRIEF descriptor can be calculated based on the intersection point.
  • the acute angle bisector of the intersection point of the feature line segment pair is used as the direction quantity of the descriptor, and the pixel size of the pixel block in the center of the intersection point is used as the scale of the descriptor Quantity, determine the descriptor of the characteristic line of the characteristic line segment pair.
  • Sub-step S43 according to the location information and descriptors of the feature lines of adjacent key frames, determine the matching feature lines between adjacent key frames.
  • Sub-step S44 according to the matching feature lines between adjacent key frames, determine the third camera pose between adjacent key frames.
  • e represents the reprojection error of a certain line segment
  • d(a, L) represents the distance from the endpoint a of the projected line segment to the observed straight line L.
  • Step 308 according to the third camera pose between adjacent key frames, determine the feature line reprojection error for the current key frame.
  • Step 309 based on the feature point re-projection error and feature line re-projection error for the current key frame, determine the optimal camera pose corresponding to the current key frame.
  • the nonlinear least squares optimization can be performed on the feature point reprojection error and the feature line reprojection error, so as to determine the optimized camera pose corresponding to the current key frame.
  • multiple image frames collected by the camera can be obtained, including key frames and non-key frames; the non-key frames are divided into multiple image blocks, and angles satisfying preset feature point conditions are extracted from multiple image blocks.
  • Points are used as feature points, and the optical flow method is used to determine the relative pose of the first camera according to the matching feature points between adjacent non-key frames; the feature points are extracted from the key frames, and the matching feature points between adjacent key frames Using the feature point matching method to determine the relative pose of the second camera; performing Kalman filter fusion according to the relative pose of the first camera between multiple adjacent non-key frames and the relative pose of the second camera between adjacent key frames, Determine the estimated camera pose corresponding to the current key frame; determine the feature point reprojection error for the current key frame according to the estimated camera pose corresponding to the current key frame; extract a line segment that meets the preset feature line condition from the key frame as a feature line; According to the matching feature lines between adjacent key frames, determine the third camera pose between adjacent key frames; according to the third camera pose between adjacent key frames.
  • the relative pose of the first camera between adjacent non-key frames can be determined by using the optical flow method to process the evenly distributed feature points between adjacent non-key frames, and the relative pose of the first camera can be provided.
  • Calculation accuracy improving the calculation efficiency and stability of the optical flow method.
  • Kalman filtering it combines the advantages of optical flow tracking algorithm and feature point matching algorithm to achieve a balance between accuracy and efficiency.
  • line features can increase the robustness of the system in urban scenes, and the stability of pose estimation can be improved by fusing feature points and optical flow and adding line feature constraints.
  • a binocular camera is installed in the vehicle to collect images, and an inertial navigation unit is used to collect motion information.
  • an inertial navigation unit is used to collect motion information.
  • the motion information (including acceleration and angular velocity) is measured by the inertial navigation unit, and the multiple motion information between image frames is combined into one observation value output by the method of pre-integration, and the camera motion accumulation is obtained; according to the camera motion accumulation , to determine the pre-integration error; finally, the pre-integration error is sent to the backend server for optimization.
  • the visual part first initialize to obtain the image frame; determine whether the image frame is a key frame or a non-key frame; if it is a non-key frame, you can use the improved corner detection method in the embodiment of the present application to detect corner points and use them as feature points ; Use the tube optical flow method to track the feature points in the adjacent non-key frames, and obtain the cumulative change of the camera pose.
  • the key frame For the key frame, on the one hand, feature points are extracted from the key frame, and the descriptor of the feature point is calculated; the feature point matching method is used to determine the relative pose of the camera according to the feature points matched between adjacent key frames; the camera relative pose determined by the optical flow method is used The pose changes are accumulated and the camera relative pose determined according to the feature point matching method is subjected to Kalman filtering to obtain the camera pose corresponding to the current key frame.
  • the feature line is detected from the key frame, the descriptor of the feature line is calculated, the matching is performed according to the matching feature line between adjacent key frames, and the camera pose corresponding to the current key frame is solved.
  • the reprojection square error of the feature points is determined.
  • the camera pose based on the feature line and the IMU motion accumulation between two key frames detected by the inertial navigation unit are used as camera constraints to determine the feature line reprojection square error.
  • FIG. 7 shows a structural block diagram of a camera pose determination device provided in an embodiment of the present application, which may specifically include the following modules:
  • An image acquisition module 701, configured to acquire multiple image frames collected by the camera; multiple image frames include key frames and non-key frames;
  • the first pose determination module 702 is configured to divide non-key frames into multiple image blocks, extract corner points satisfying preset feature point conditions from multiple image blocks as feature points, and The matching feature points are determined by the optical flow method to determine the relative pose of the first camera;
  • the second pose determination module 703 is used to extract feature points from key frames, and determine the relative pose of the second camera by feature point matching method according to feature points matched between adjacent key frames;
  • An optimized pose determination module 704 configured to determine an optimized camera position corresponding to the current key frame according to the first camera relative pose between a plurality of adjacent non-key frames and the second camera relative pose between adjacent key frames posture.
  • multiple image frames collected by the camera can be acquired, including key frames and non-key frames; the non-key frames are divided into multiple image blocks, and corner points that meet the preset feature point conditions are extracted from the multiple image blocks
  • the optical flow method is used to determine the relative pose of the first camera; the feature points are extracted from the key frames, and the matching feature points between adjacent key frames are used.
  • the feature point matching method determines the relative pose of the second camera; according to the relative pose of the first camera between multiple adjacent non-key frames and the relative pose of the second camera between adjacent key frames, determine the relative pose of the current key frame Optimize the camera pose.
  • the embodiment of the present application integrates the respective advantages of the optical flow tracking algorithm and the feature point matching algorithm, and can achieve a balance between pose accuracy and processing efficiency; and by using the optical flow method to determine uniformly distributed feature points for non-key frames, Determining the relative pose of the first camera between adjacent non-key frames can improve the calculation accuracy of the relative pose of the first camera and improve the calculation efficiency and stability of the optical flow method.
  • the device for determining the camera pose may also include:
  • a feature line extraction module is used to extract a line segment satisfying a preset feature line condition from a key frame as a feature line;
  • the third pose determination module is used to determine the third camera pose between adjacent key frames according to the matching feature lines between adjacent key frames;
  • the optimized pose determination module includes:
  • Optimizing the pose determination sub-module which is used to determine the relative pose of the first camera between a plurality of adjacent non-key frames, the relative pose of the second camera between adjacent key frames and the third relative pose of the camera between adjacent key frames Camera pose, which determines the optimized camera pose corresponding to the current keyframe.
  • the optimal pose determination submodule may include:
  • the estimated pose determination unit is used to perform Kalman filter fusion according to the relative pose of the first camera between a plurality of adjacent non-key frames and the relative pose of the second camera between adjacent key frames, so as to determine that the current key frame corresponds to The estimated camera pose of ;
  • the first error determination unit is used to determine the feature point reprojection error for the current key frame according to the estimated camera pose corresponding to the current key frame;
  • the second error determination unit is used to determine the feature line reprojection error for the current key frame according to the third camera pose between adjacent key frames;
  • An optimized pose determining unit is configured to determine an optimized camera pose corresponding to the current key frame based on the feature point re-projection error and the feature line re-projection error for the current key frame.
  • the first pose determining module may include:
  • a filtering submodule is used to perform low-pass filtering processing on multiple image blocks
  • the response value calculation sub-module is used to calculate the corner response value for the pixels in each image block processed by the low-pass filter
  • the initial corner selection submodule is used to sort the corner response values corresponding to each pixel in each image block, and select a preset number of pixels as the initial corner according to the sorting result;
  • the degree of dispersion determination submodule is used to determine the degree of dispersion of the initial corner points in each image block
  • the screening ratio determination sub-module is used to set the corresponding screening ratio for each image block according to the degree of dispersion of the initial corner points in each image block and the corner point response value;
  • the candidate corner point determination submodule is used for screening candidate corner points from corresponding initial corner points according to the screening ratio of each image block;
  • the target corner screening sub-module is used for the random sampling consensus algorithm to screen target corners from candidate corners.
  • the sub-module for determining the degree of dispersion may include:
  • the clustering unit is used to cluster the initial corner points of the image block to obtain the cluster center;
  • the degree of dispersion determination unit is configured to determine the pixel distance from each initial corner point to the cluster center, and determine the degree of dispersion according to the pixel distance from each initial corner point to the cluster center.
  • the degree of dispersion is the pixel distance and value of each initial corner point in the image block
  • the screening ratio determination submodule may include:
  • the response value processing unit is used for calculating the mean square error sum value for the corner point response value of the initial corner point of the image block;
  • An evaluation parameter calculation unit is used to calculate the evaluation parameter according to the pixel distance and value of each initial corner point in the image block and the mean square error and value of the corner response value;
  • the screening ratio determining unit is configured to determine the screening ratio of each image block according to the evaluation parameters of each image block.
  • the second pose determination module may include:
  • the window setting submodule is used to set a window with a preset pixel size centered on the feature point in the key frame;
  • the difference determination submodule is used to determine the absolute value of the difference between the gray value of the feature point and the gray value of other pixels in the window;
  • the matrix generation module is used to generate an adjacency matrix according to the absolute value of the gray difference between the feature point and other pixels in the window;
  • the first descriptor determining submodule is used to generate a description vector as a descriptor of a feature point according to an adjacency matrix
  • the feature point matching submodule is used to determine the matching feature points between adjacent key frames according to the position information and descriptors of the feature points of adjacent key frames;
  • the second pose determination sub-module is used to determine the relative pose of the second camera between adjacent key frames according to the matching feature points between adjacent key frames.
  • the feature line extraction module may include:
  • the line segment pair selection sub-module is used to detect the line segment in the key frame, and two non-parallel line segments are used as the line segment pair;
  • the line segment pair screening sub-module is used to filter the line segment pairs and select the line segment pairs that meet the preset feature line conditions as feature lines;
  • the preset feature line conditions include: the length of the line segment is greater than or equal to the preset length threshold, and the intersection point of the line segment pair is relative to the line segment The distance of the pair is less than or equal to the preset distance threshold, and the intersection point of the pair of line segments is within the image.
  • the third pose determining module may include:
  • a feature line segment pair determination submodule is used to use two non-parallel feature lines in the key frame as a feature line segment pair;
  • the second descriptor determination sub-module is used to determine the feature of the feature line segment pair by taking the acute angle bisector of the intersection point of the feature line segment pair as the direction quantity of the descriptor, and taking the pixel size of the pixel block at the center of the intersection point as the scale quantity of the descriptor line descriptor;
  • the characteristic line matching submodule is used to determine the matching characteristic lines between adjacent key frames according to the position information and descriptors of the characteristic lines of adjacent key frames;
  • the third pose determination sub-module is configured to determine the pose of the third camera between adjacent key frames according to the matching feature lines between adjacent key frames.
  • the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment.
  • the embodiment of the present application also provides an electronic device, including:
  • It includes a processor, a memory, and a computer program stored on the memory and capable of running on the processor.
  • the computer program When the computer program is executed by the processor, it realizes the various processes of the above-mentioned camera pose determination method embodiment, and can achieve the same technical effect. To avoid repetition, details are not repeated here.
  • the embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored.
  • a computer program is stored.
  • the computer program is executed by a processor, each process of the above-mentioned embodiment of the camera pose determination method can be realized, and the same can be achieved.
  • Technical effects, in order to avoid repetition, will not be repeated here.
  • embodiments of the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiment of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
  • computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • Embodiments of the present application are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor or processor of other programmable data processing terminal equipment to produce a machine such that instructions executed by the computer or processor of other programmable data processing terminal equipment Produce means for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.
  • These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the The instruction means implements the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例提供了一种相机位姿确定方法和装置，所述方法包括：获取相机采集的多个图像帧；将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿；从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿；根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿。本申请实施例可以提高位姿精度，在位姿精度和处理效率上达到平衡。

Description

一种相机位姿确定方法和装置
本申请要求在2022年1月06日提交中国专利局、申请号202210012434.8、发明名称为“一种相机位姿确定方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，特别是涉及一种相机位姿确定方法和一种相机位姿确定装置。
背景技术
视觉里程计基于视觉图像对里程进行估计，被广泛应用于移动设备在未知环境的定位实现。视觉里程计架构通常分为前端和后端两大部分，前端负责构建多视角几何约束，后端基于前端的重投影误差做非线性最小二乘优化。
常用的前端分为特征点匹配法和光流法两种视觉方法。特征点匹配法采用匹配的特征点构建多个图像帧之间的共视约束，其特点是精度高，但是由于涉及到特征点提取以及特征描述子计算和匹配，所以存在计算量大，容易无匹配的缺点。光流法不做描述子的计算和匹配，直接通过光流来跟踪特征点，大大的减少了计算开销，但也存在精度相对较低和跟踪容易跟丢的问题。本质上，无论特征点法还是光流法，都需要依赖与场景中的特征点的稳定性。所以在一些特征点比较稀疏的弱纹理场景下，两种前端约束方法或多或少都存在稳定性问题。
发明内容
鉴于上述问题，提出了本申请实施例以便提供一种克服上述问题或者至少部分地解决上述问题的一种相机位姿确定方法和相应的一种相机位姿确定装置。
为了解决上述问题，本申请实施例公开了一种相机位姿确定方法，包括：
获取相机采集的多个图像帧；多个图像帧包括关键帧和非关键帧；
将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流 法确定第一相机相对位姿；
从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿；
根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿。
在一些实施例中，方法还包括：
从关键帧提取满足预设特征线条件的线段作为特征线；
根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿；
根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿，包括：
根据多个相邻非关键帧之间的第一相机相对位姿、相邻关键帧之间的第二相机相对位姿和相邻关键帧之间的第三相机位姿，确定当前关键帧对应的优化相机位姿。
在一些实施例中，根据多个相邻非关键帧之间的第一相机相对位姿、相邻关键帧之间的第二相机相对位姿和相邻关键帧之间的第三相机位姿，确定当前关键帧对应的优化相机位姿，包括：
根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿进行卡尔曼滤波融合，确定当前关键帧对应的估计相机位姿；
根据当前关键帧对应的估计相机位姿，确定针对当前关键帧的特征点重投影误差；
根据相邻关键帧之间的第三相机位姿，确定针对当前关键帧的特征线重投影误差；
基于针对当前关键帧的特征点重投影误差和特征线重投影误差，确定当前关键帧对应的优化相机位姿。
在一些实施例中，从多个图像块中提取满足预设特征点条件的角点作为特征点，包括：
对多个图像块进行低通滤波处理；
分别对经过低通滤波处理的各个图像块中的像素点，计算角点化响应值；
分别对各个图像块中各个像素点对应的角点化响应值进行排序，根据排序结果选取预设个数的像素点作为初始角点；
确定各个图像块中初始角点的分散程度；
根据各个图像块中初始角点的分散程度和角点化响应值，为各个图像块设置对应的筛选比例；
根据各个图像块的筛选比例，从对应的初始角点中筛选候选角点；
随机采样一致性算法，从候选角点中筛选目标角点。
在一些实施例中，确定各个图像块中初始角点的分散程度，包括：
对图像块的初始角点进行聚类，得到聚类中心；
确定各个初始角点到聚类中心的像素距离，并根据各个初始角点到聚类中心的像素距离确定分散程度。
在一些实施例中，分散程度为图像块中各个初始角点的像素距离和值，根据各个图像块中初始角点的分散程度和角点化响应值，为各个图像块设置对应的筛选比例，包括：
对图像块的初始角点的角点化响应值，求均方误差和值；
根据图像块中各个初始角点的像素距离和值以及角点化响应值的均方误差和值，计算评价参数；
根据各个图像块的评价参数，确定各个图像块的筛选比例。
在一些实施例中，从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点确定第二相机相对位姿，包括：
以关键帧中的特征点为中心设置预设像素大小的窗口；
确定特征点的灰度值与窗口中其他像素点的灰度值的差值的绝对值；
根据特征点与窗口中的其他像素点之间的灰度差值的绝对值，生成邻接矩阵；
根据邻接矩阵，生成描述向量作为特征点的描述子；
根据相邻关键帧的特征点的位置信息和描述子，确定相邻关键帧之间匹 配的特征点；
根据相邻关键帧之间匹配的特征点确定相邻关键帧之间的第二相机相对位姿。
在一些实施例中，从关键帧提取满足预设特征线条件的线段作为特征线，包括：
检测关键帧中的线段，以不平行的两线段作为线段对；
对线段对，筛选出满足预设特征线条件的线段对作为特征线；预设特征线条件包括：线段长度大于或等于预设长度阈值，线段对的交点相对线段对的距离小于或等于预设距离阈值，线段对的交点在图像之内。
在一些实施例中，根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿，包括：
以关键帧中不平行的两特征线作为特征线段对；
以特征线段对的交点的锐角角平分线为描述子的方向量，以交点中心像素块的像素大下作为描述子的尺度量，确定特征线段对的特征线的描述子；
根据相邻关键帧的特征线的位置信息和描述子，确定相邻关键帧之间匹配的特征线；
根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿。
本申请实施例还公开了一种相机位姿确定装置，包括：
图像获取模块，用于获取相机采集的多个图像帧；多个图像帧包括关键帧和非关键帧；
第一位姿确定模块，用于将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿；
第二位姿确定模块，用于从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿；
优化位姿确定模块，用于根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机 位姿。
本申请实施例还公开了一种电子设备，包括：处理器、存储器及存储在存储器上并能够在处理器上运行的计算机程序，计算机程序被处理器执行时实现如上的相机位姿确定方法的步骤。
本申请实施例还公开了一种计算机可读存储介质，计算机可读存储介质上存储计算机程序，计算机程序被处理器执行时实现如上的相机位姿确定方法的步骤。
本申请实施例包括以下优点：
本申请实施例，可以获取相机采集的多个图像帧，包括关键帧和非关键帧；将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿；从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿；根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿。本申请实施例，融合了光流跟踪算法和特征点匹配算法各自的优点，能够在位姿精度和处理效率上达到平衡；并且通过对非关键帧确定均匀分布特征点采用光流法进行处理，确定相邻非关键帧之间的第一相机相对位姿，可以提供第一相机相对位姿的计算精度，提升光流法的计算效率和稳定性。
附图说明
图1是本申请实施例提供的一种相机位姿确定方法的步骤流程图；
图2是本申请实施例中结合特征点匹配法和光流法进行卡尔曼滤波的流程图；
图3是本申请实施例提供的另一种相机位姿确定方法的步骤流程图；
图4是本申请实施例中从各个图像块中均匀筛选特征点的示意图；
图5是确定特征线重投影误差的示意图；
图6是本申请实施例中基于相机位姿优化的视觉里程计优化方法的流程图；
图7是本申请实施例提供的一种相机位姿确定装置的结构框图。
具体实施方式
为使本申请的上述目的、特征和优点能够更加明显易懂，下面结合附图和具体实施方式对本申请作进一步详细的说明。
视觉里程计VIO(Visual Inertial Odometry)利用相机采集的图像和惯性导航单元IMU(Inertial Measurement Unit)检测的运动信息(包括加速度和角速度等)进行优势互补，IMU能够准确测量短时间内的运动，当相机在相邻帧间前端跟踪质量较差时，IMU的测量可以测出帧间的运动并提供帧间约束，保证***继续运行，同时视觉和IMU融合的方法也会使位姿估计更加精确。
针对视觉部分，本申请实施例的核心构思在于，融合光流跟踪算法和特征点匹配算法各自的优点，能够在位姿精度和处理效率上达到平衡；并且通过对非关键帧确定均匀分布特征点采用光流法进行处理，确定相邻非关键帧之间的第一相机相对位姿，可以提供第一相机相对位姿的计算精度，提升光流法的计算效率和稳定性。
参照图1，示出了本申请实施例提供的一种相机位姿确定方法的步骤流程图，方法具体可以包括如下步骤：
步骤101，获取相机采集的多个图像帧；多个图像帧包括关键帧和非关键帧。
在实际应用场景中，可以在车辆、飞行器等移动设备上设置相机，通过相机采集周围环境的图像序列。图像序列可以包括多个图像帧，图像帧可以关键帧和非关键帧，关键帧是具有代表性的图像帧。在一些实施例中，关键帧选定原则可以为：1、如果当前图像帧跟踪到的特征点少于20个，则新的图像帧计为关键帧。2、如果当前图像帧与历史前N和前N+1个关键帧(N的取值可自行设定，例如N＝10)的平均视差大于某一阈值(例如0.02)，则计当前帧为关键帧。具体的关键帧选取方式本申请实施例不做限定。
参照图2所示为本申请实施例中结合特征点匹配法和光流法进行卡尔曼滤波的流程图。对关键帧图像进行优化的特征点匹配算法估计相机位姿，对 非关键帧进行光流法确定相机位姿并累计，对采用特征点匹配算法得到的相机位姿与采用光流法得到的相机位姿进行卡尔曼滤波融合，得到优化相机位姿。
步骤102，将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿。
真实场景中物体的运动会导致图像上对应点运动之外，也会导致图像上对应像素点的亮度模式发生改变。所谓光流就是图像上像素点亮度模式运动的速度或者说是物体所带的发光部位的移动投影到图像平面所引起的像素点亮度变化。光流法可以基于相邻图像帧之间像素点的光流约束，确定相邻图像帧之间的相机相对位姿。
光流法利用角点进行光流跟踪，角点是指图像中灰度值变化较大的点。角点检测方法可以包括：Harris角点检测算法、ORB角点检测算法和FAST检测算法等。
在本申请实施例中，对非关键帧中的特征点采用光流法进行跟踪，光流跟踪容易存在跟踪点集中分布在一小片图像区域的问题。对此，本申请实施例中可以采用改进的角点检测方法中图像帧中均匀筛选和提取特征点，然后可以用随机采样一致性算法RANSAC(random sample concensum)对均匀化的角点做进一步的处理来减少可能的配对错误。
在本申请实施例中，可以将非关键帧划分为多个图像块，且每个图像块的尺寸相同，从每个图像块分别提取预设数量的特征点，可以保证整个图像中的特征点均匀分布。例如，可以将图像帧划分为9个尺寸相同的图像块，对每个图像块单独提取满足预设特征点条件的角点作为特征点。预设特征点条件可以用于从图像块中均匀筛选出角点作为特征点。
根据相邻非关键帧之间均匀分布特征点采用光流法进行处理，确定相邻非关键帧之间的第一相机相对位姿，可以提供第一相机相对位姿的计算精度，提升光流法的计算效率和稳定性。
步骤103，从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点 采用特征点匹配法确定第二相机相对位姿。
基于特征点匹配法需要提取特征点以及确定描述子，基于特征点的描述子确定匹配的特征点，由于匹配耗时较大，所以只对关键帧采用特征点匹配法。基于特征点匹配法，采用两个关键帧中的特征点确定两个关键帧之间的第二相机相对位姿。
步骤104，根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿。
可以根据相邻关键帧之间的多个相邻非关键帧之间的第一相机相对位姿，和相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿。
本申请实施例，可以获取相机采集的多个图像帧，包括关键帧和非关键帧；将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿；从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿；根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿。本申请实施例，融合了光流跟踪算法和特征点匹配算法各自的优点，能够在位姿精度和处理效率上达到平衡；并且通过对非关键帧确定均匀分布特征点采用光流法进行处理，确定相邻非关键帧之间的第一相机相对位姿，可以提供第一相机相对位姿的计算精度，提升光流法的计算效率和稳定性。
参照图3，示出了本申请实施例提供的另一种相机位姿确定方法的步骤流程图，方法具体可以包括如下步骤：
步骤301，获取相机采集的多个图像帧；多个图像帧包括关键帧和非关键帧。
步骤302，将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿。
在本申请实施例中可以采用改进的Harris角点检测算法，将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点。
在本申请实施例中，从多个图像块中提取满足预设特征点条件的角点作为特征点的步骤可以包括如下子步骤：
子步骤S11，对多个图像块进行低通滤波处理。
在采用Harris角点检测算法时，传统的Harris角点检测算法先对图像进行高斯平滑，即采用高斯高斯对图像进行滤波。但是高斯平滑会将作为高频部分(物体的边缘)大大削弱，边缘变缓，图像直方图表现上受到压缩。导致在做非极大值抑制的时候角点丢失。
对此，本申请改进的Harris角点检测算法，可以先对多个图像块进行低通滤波处理。在一些实施例中，可以采用具有低通特性的三次B样条函数代替高斯函数做平滑滤波。
子步骤S12，分别对经过低通滤波处理的各个图像块中的像素点，计算角点化响应值。
假设，图像窗口微小移动产生的灰度变化为E(u,v)＝(u,v)M(u,v) T。
这种区域变化形式与局部自相关函数类似，M为自相关函数的矩阵表示，λ 1，λ 2为矩阵M的两个特征值，表示为自相关函数的一阶曲率。
在像素点(x,y)处，角点化响应函数如下：C(x,y)＝λ 1λ 2-α(λ 1+λ 2) 2，其中α为经验证值，一般设为0.04-0.06。
子步骤S13，分别对各个图像块中各个像素点对应的角点化响应值进行排序，根据排序结果选取预设个数的像素点作为初始角点。
例如，可以对各个图像块中N个角点化响应值排序并选取其中值相对较大的前B＝k*N个角点作为被检查出来的初角点(k取值0～1)，k值在不同图像块取值不同，k的取值要保障每个图像块都要有相当数量的初角点。
子步骤S14，确定各个图像块中初始角点的分散程度。
分散程度可以表示图像块中初始角点分散情况。在一些实施例中，子步 骤S14可以进一步包括：
子步骤S141，对图像块的初始角点进行聚类，得到聚类中心；
子步骤S142，确定各个初始角点到聚类中心的像素距离，并根据各个初始角点到聚类中心的像素距离确定分散程度。
在一些示例中，可以根据将各个初始角点到聚类中心的像素距离求和，作为图像块中初始角点的分散程度。在另一些示例中，可以根据将各个初始角点到聚类中心的像素距离的平均值，作为图像块中初始角点的分散程度。
子步骤S15，根据各个图像块中初始角点的分散程度和角点化响应值，为各个图像块设置对应的筛选比例。
筛选比例是从初始角点中筛选并保留角点的比例，为了保证各个图像块均有角点被保留利用，且越是分散、响应值越高的地方，其角点就越应该保留。可以根据各个图像块中初始角点的分散程度和角点化响应值，为各个图像块设置对应的筛选比例。
在一些实施例中，子步骤S15可以进一步包括：
子步骤S151，对图像块的初始角点的角点化响应值，求均方误差和值。
子步骤S152，根据图像块中各个初始角点的像素距离和值以及角点化响应值的均方误差和值，计算评价参数。
在一些实施例中，评价参数
Figure PCTCN2022132927-appb-000001
Ω表示图像块；i表示初角点索引；
Figure PCTCN2022132927-appb-000002
表示块内初角点响应值的均值；d i表示初角点i到聚类中心的像素距离。
子步骤S153，根据各个图像块的评价参数，确定各个图像块的筛选比例。
在一些实施例中，可以根据各个图像块的评价参数，对图像块进行排序，根据排序结果确定各个图像块的筛选比例。其中排序越靠前，筛选比例越多。
例如，根据基于评价参数W的根据排序结果，确定筛选比例ηW，其中(ηW∈(0，1)，最终从单个图像块中提取出ηW·B个候选角点。
参照图4所示为本申请实施例中从各个图像块中均匀筛选特征点的示意 图。对于初始角点比较集中的图像块，删除较多的初始角点。对于初始角点比较分散的图像块，保留较多的初始角点，并且筛选后各个图像块都能保留有数量相当的角点。
子步骤S16，根据各个图像块的筛选比例，从对应的初始角点中筛选候选角点。
子步骤S17，随机采样一致性算法，从候选角点中筛选目标角点。
可以随机采样一致性算法，从候选角点中删除局外点，将保留的候选角点作为目标角点。
随机采样一致性算法，在给定的N个数据点组成的集合P里面，假设集合中大多数的点都可以通过一个模型来产生，且最少通过n个点(n<N)可以拟合出模型的参数，不符合拟合模型的点则为局外点。
在一些实施例中，可以通过以下的迭代方式拟合，具体步骤为：a.从P中随机选择n个数据点；b.用这n个数据点拟合出一个模型M；c.用P中剩余的数据点，计算每个点与模型M的距离，距离超过阈值的则认为是局外点，反之则认为是局内点，并记录模型所对应的局内点的树量m；d.迭代k次后选择m最大的模型M作为拟合结果。
步骤303，从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿。
在本申请实施例中，步骤303可以包括如下子步骤：
子步骤S21，以关键帧中的特征点为中心设置预设像素大小的窗口。
子步骤S22，确定特征点的灰度值与窗口中其他像素点的灰度值的差值的绝对值。
子步骤S23，根据特征点与窗口中的其他像素点之间的灰度差值的绝对值，生成邻接矩阵。
子步骤S24，根据邻接矩阵，生成描述向量作为特征点的描述子。
子步骤S25，根据相邻关键帧的特征点的位置信息和描述子，确定相邻关键帧之间匹配的特征点。
在一些实施例中，为了在特征点匹过程中减少误匹配的概率，可以以特 征点为中心，设置一个3×3像素大小的窗口，计算窗口中心与8领域像素点与特灰度值之差的绝对值。I p表示中心点也就是特征点，I xi是8领域的灰度值。
I i＝|I p-I xi|,i＝1,2,...8
生成邻接矩阵F，
Figure PCTCN2022132927-appb-000003
用一个描述向量H表示特征点p，
Figure PCTCN2022132927-appb-000004
描述向量，其中λ i(p)表示邻接矩阵F分解后的特征值，||λ p||表示特征值的第二范数。同理可得特征点p的配对点q的描述向量
Figure PCTCN2022132927-appb-000005
设置
Figure PCTCN2022132927-appb-000006
|H|和|G|分别是两个向量的模长，并设置阈值t,当v>t时则保留特征点对，否者删除。
子步骤S26，根据相邻关键帧之间匹配的特征点确定相邻关键帧之间的第二相机相对位姿。
步骤304，根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿进行卡尔曼滤波融合，确定当前关键帧对应的估计相机位姿。
通过卡尔曼滤波算法，将光流跟踪的结果与优化特征点匹配结果进行融合。在每个关键帧处进行卡尔曼滤波，可以用两个关键帧的高精确特征点匹配结果来修正这两个关键帧之间非关键帧光流跟踪的累计误差。采用卡尔曼滤波的思想，融合光流跟踪算法和特征点匹配算法各自的优点，可以达到精度和效率的平衡。
卡尔曼滤波分为预测和更新两个步骤：
预测阶段，由光流跟踪法累计得到2个关键帧之间的相对位姿作为估计，而有特征点优化匹配得到的相机相对位姿作为观测。
Figure PCTCN2022132927-appb-000007
Figure PCTCN2022132927-appb-000008
Figure PCTCN2022132927-appb-000009
其中A k为状态转移矩阵，表示从上一个关键帧开始到当前关键帧，光流法累计得到的相机相对位姿变换；ε k～(0,R)表示运动方程的高斯噪声；
Figure PCTCN2022132927-appb-000010
为上一个关键帧对应的修正后的状态估计和协方差；R为噪声的协方差，设为常数；
Figure PCTCN2022132927-appb-000011
和
Figure PCTCN2022132927-appb-000012
是当前关键帧的状态估计和协方差。z k是当前时刻特征点法得到的相机位姿，C设为单位矩阵，δ k～N(0,Q)表示观测噪声。由于状态估计方程(光流法估算位姿)误差较特征点法大，所以Q值一般比R值要小。
更新阶段，先计算卡尔曼增益K，然后再对当前关键帧的状态估计和协方差做修正得到融合后的估计相机位姿
Figure PCTCN2022132927-appb-000013
和协方差
Figure PCTCN2022132927-appb-000014
Figure PCTCN2022132927-appb-000015
Figure PCTCN2022132927-appb-000016
Figure PCTCN2022132927-appb-000017
步骤305，根据当前关键帧对应的估计相机位姿，确定针对当前关键帧的特征点重投影误差。
可以基于前一帧关键帧中特征点P1的位置信息和估计相机位姿，计算在当前关键帧的时刻对应特征点P1′的位置信息；从当前关键中确定与特征点P1匹配的特征点P2的位置信息，根据P1′的位置信息和P2的位置信息，可以计算特征点重投影误差。在实际中，可以根据相邻关键帧之间的多个特征点，计算特征点重投影误差。
步骤306，从关键帧提取满足预设特征线条件的线段作为特征线。
在城市上空弱纹理区域和光线剧烈变化场景中下，视相机的视觉里程计如果单靠点特征约束容易出现定位失效和输出不稳定的问题。在以市区上空对地俯视为场景时，地面建筑存在很强的结构特性。建筑的几何边界所构成的线特征，能提供特征点所没法提供的方向信息。所以，引入线特征可以增加***在城市场景下的鲁棒性，通过特征点和光流融合以及加入线特征约束，能很好的解决定位失效和输出不稳定的问题。
在本申请实施例中，步骤306可以包括如下子步骤：
子步骤S31，检测关键帧中的线段，以不平行的两线段作为线段对。
子步骤S32，对线段对，筛选出满足预设特征线条件的线段对作为特征线；预设特征线条件包括：线段长度大于或等于预设长度阈值，线段对的交点相对线段对的距离小于或等于预设距离阈值，线段对的交点在图像之内。
在一些实施例中，对于图像中不平行的线段(线段延长线)，必定存在交点，通常这样的线段集合的内部元素的数量会非常多。为了更好的提取出特征直线，做以下筛选：去除交点到线段对距离大于或等于预设距离阈值的线段对，保留交点相对线段对的距离小于或等于预设距离阈值的线段对；去除交点在图像外的线段对，保留交点在图像之内的线段对；去除长度小于预设长度阈值的线段对，保留线段长度大于或等于预设长度阈值的线段对；经过上述筛选之后的到的线段即为有效特征线段。
步骤307，根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿。
在本申请实施例中，步骤307可以包括如下子步骤：
子步骤S41，以关键帧中不平行的两特征线作为特征线段对。
子步骤S42，以特征线段对的交点的锐角角平分线为描述子的方向量，以交点中心像素块的像素大下作为描述子的尺度量，确定特征线段对的特征线的描述子。
在本申请实施例中，对于线段特征的描述，可以基于特征线段对的相交点来描述。可以基于交点，计算多尺度旋转BRIEF描述子，在一些实施例中，以特征线段对的交点的锐角角平分线为描述子的方向量，以交点中心像素块的像素大下作为描述子的尺度量，确定特征线段对的特征线的描述子。
子步骤S43，根据相邻关键帧的特征线的位置信息和描述子，确定相邻关键帧之间匹配的特征线。
子步骤S44，根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿。
参照图5所示为确定特征线重投影误差的示意图。假设i时刻图像帧中看到的线段l，匹配到j时刻图像中的观测线段L。但是，通过相机位姿(包括旋转矩阵R，平移矩阵T)将i时刻的线段l投影到j时刻的图像下得到线段ab。线段的重投影误差用投影线段的两个端口a，b到观测直线L(Ax+By+C＝0)的距离来表示。
Figure PCTCN2022132927-appb-000018
Figure PCTCN2022132927-appb-000019
e表示某一线段的重投影误差，d(a，L)表示投影线段端点a到观测直线L的距离。通过优化线段的重投影误差，能够优化相机位姿RT。
步骤308，根据相邻关键帧之间的第三相机位姿，确定针对当前关键帧 的特征线重投影误差。
步骤309，基于针对当前关键帧的特征点重投影误差和特征线重投影误差，确定当前关键帧对应的优化相机位姿。
可以对特征点重投影误差和特征线重投影误差进行做非线性最小二乘优化，从而确定当前关键帧对应的优化相机位姿。
本申请实施例中，可以获取相机采集的多个图像帧，包括关键帧和非关键帧；将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿；从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿；根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿进行卡尔曼滤波融合，确定当前关键帧对应的估计相机位姿；根据当前关键帧对应的估计相机位姿，确定针对当前关键帧的特征点重投影误差；从关键帧提取满足预设特征线条件的线段作为特征线；根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿；根据相邻关键帧之间的第三相机位姿，确定针对当前关键帧的特征线重投影误差；基于针对关键帧的特征点重投影误差和特征线重投影误差，确定关键帧对应的优化相机位姿。本申请实施例，通过根据相邻非关键帧之间均匀分布特征点采用光流法进行处理，确定相邻非关键帧之间的第一相机相对位姿，可以提供第一相机相对位姿的计算精度，提升光流法的计算效率和稳定性。采用卡尔曼滤波的思想，融合了光流跟踪算法和特征点匹配算法各自的优点，达到精度和效率的平衡。并且引入线特征可以增加***在城市场景下的鲁棒性，通过特征点和光流融合以及加入线特征约束，可以提高位姿估计的稳定性。
为了使本领域技术人员能够更好地理解本申请实施例，下面通过一个例子对本申请实施例加以说明：参照图6所示为本申请实施例中基于相机位姿优化的视觉里程计优化方法的流程图。
在该示例中，在车辆设置双目相机采集图像，通过惯性导航单元采集运动信息。相机与惯性导航单元之间可以刚性连接。
对于惯性导航单元部分，通过惯性导航单元测量运动信息(包括加速度和角速度)，采用预积分的方法将图像帧间的多个运动信息合并成一个观测值输出，得到相机运动累计；根据相机运动累计，确定预积分误差；最后将预积分误差发送到后端服务器以进行优化。
对于视觉部分，首先进行初始化以获取图像帧；判断图像帧为关键帧或非关键帧；若是非关键帧，则可以采用本申请实施例中改进的角点检测方法检测角点，并作为特征点；对相邻非关键帧中的特征点采用管光流法跟踪，得到相机位姿变化累计。对于关键帧，一方面从关键帧提取特征点，计算特征点的描述子；根据相邻关键帧之间匹配的特征点采用特征点匹配法确定相机相对位姿；采用根据光流法确定的相机位姿变化累计和根据特征点匹配法确定的相机相对位姿进行卡尔曼滤波，得到当前关键帧对应的相机位姿。另一方面，从关键帧检测特征线，计算特征线的描述子，根据相邻关键帧之间匹配的特征线进行匹配，并求解出当前关键帧对应的相机位姿。
以基于特征点求解出的相机位姿和惯性导航单元检测得到的两关键帧之间的IMU运动累计作为相机约束，确定特征点重投影方误差。以基于特征线求解出的相机位姿和惯性导航单元检测得到的两关键帧之间的IMU运动累计作为相机约束，确定特征线重投影方误差。在后端服务器根据特征点重投影方误差、特征线重投影方误差和预积分误差进行优化，最终得到当前关键帧对应的优化相机位姿。
需要说明的是，对于方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本申请实施例并不受所描述的动作顺序的限制，因为依据本申请实施例，某些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的一些实施例所涉及的动作并不一定是本申请实施例所必须的。
参照图7，示出了本申请实施例提供的一种相机位姿确定装置的结构框图，具体可以包括如下模块：
图像获取模块701，用于获取相机采集的多个图像帧；多个图像帧包括关键帧和非关键帧；
第一位姿确定模块702，用于将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿；
第二位姿确定模块703，用于从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿；
优化位姿确定模块704，用于根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿。
本申请实施例，可以获取相机采集的多个图像帧，包括关键帧和非关键帧；将非关键帧划分为多个图像块，从多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿；从关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿；根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿。本申请实施例，融合了光流跟踪算法和特征点匹配算法各自的优点，能够在位姿精度和处理效率上达到平衡；并且通过对非关键帧确定均匀分布特征点采用光流法进行处理，确定相邻非关键帧之间的第一相机相对位姿，可以提供第一相机相对位姿的计算精度，提升光流法的计算效率和稳定性。
在本申请实施例中，相机位姿确定装置还可以包括：
特征线提取模块，用于从关键帧提取满足预设特征线条件的线段作为特征线；
第三位姿确定模块，用于根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿；
优化位姿确定模块包括：
优化位姿确定子模块，用于根据多个相邻非关键帧之间的第一相机相对位姿、相邻关键帧之间的第二相机相对位姿和相邻关键帧之间的第三相机位姿，确定当前关键帧对应的优化相机位姿。
在本申请实施例中，优化位姿确定子模块可以包括：
估计位姿确定单元，用于根据多个相邻非关键帧之间的第一相机相对位姿和相邻关键帧之间的第二相机相对位姿进行卡尔曼滤波融合，确定当前关键帧对应的估计相机位姿；
第一误差确定单元，用于根据当前关键帧对应的估计相机位姿，确定针对当前关键帧的特征点重投影误差；
第二误差确定单元，用于根据相邻关键帧之间的第三相机位姿，确定针对当前关键帧的特征线重投影误差；
优化位姿确定单元，用于基于针对当前关键帧的特征点重投影误差和特征线重投影误差，确定当前关键帧对应的优化相机位姿。
在本申请实施例中，第一位姿确定模块可以包括：
滤波子模块，用于对多个图像块进行低通滤波处理；
响应值计算子模块，用于分别对经过低通滤波处理的各个图像块中的像素点，计算角点化响应值；
初始角点选取子模块，用于分别对各个图像块中各个像素点对应的角点化响应值进行排序，根据排序结果选取预设个数的像素点作为初始角点；
分散程度确定子模块，用于确定各个图像块中初始角点的分散程度；
筛选比例确定子模块，用于根据各个图像块中初始角点的分散程度和角点化响应值，为各个图像块设置对应的筛选比例；
候选角点确定子模块，用于根据各个图像块的筛选比例，从对应的初始角点中筛选候选角点；
目标角点筛选子模块，用于随机采样一致性算法，从候选角点中筛选目标角点。
在本申请实施例中，分散程度确定子模块可以包括：
聚类单元，用于对图像块的初始角点进行聚类，得到聚类中心；
分散程度确定单元，用于确定各个初始角点到聚类中心的像素距离，并根据各个初始角点到聚类中心的像素距离确定分散程度。
在本申请实施例中，分散程度为图像块中各个初始角点的像素距离和值， 筛选比例确定子模块可以包括：
响应值处理单元，用于对图像块的初始角点的角点化响应值，求均方误差和值；
评价参数计算单元，用于根据图像块中各个初始角点的像素距离和值以及角点化响应值的均方误差和值，计算评价参数；
筛选比例确定单元，用于根据各个图像块的评价参数，确定各个图像块的筛选比例。
在本申请实施例中，第二位姿确定模块可以包括：
窗口设置子模块，用于以关键帧中的特征点为中心设置预设像素大小的窗口；
差值确定子模块，用于确定特征点的灰度值与窗口中其他像素点的灰度值的差值的绝对值；
矩阵生成模块，用于根据特征点与窗口中的其他像素点之间的灰度差值的绝对值，生成邻接矩阵；
第一描述子确定子模块，用于根据邻接矩阵，生成描述向量作为特征点的描述子；
特征点匹配子模块，用于根据相邻关键帧的特征点的位置信息和描述子，确定相邻关键帧之间匹配的特征点；
第二位姿确定子模块，用于根据相邻关键帧之间匹配的特征点确定相邻关键帧之间的第二相机相对位姿。
在本申请实施例中，特征线提取模块可以包括：
线段对选取子模块，用于检测关键帧中的线段，以不平行的两线段作为线段对；
线段对筛选子模块，用于对线段对，筛选出满足预设特征线条件的线段对作为特征线；预设特征线条件包括：线段长度大于或等于预设长度阈值，线段对的交点相对线段对的距离小于或等于预设距离阈值，线段对的交点在图像之内。
在本申请实施例中，第三位姿确定模块可以包括：
特征线段对确定子模块，用于以关键帧中不平行的两特征线作为特征线段对；
第二描述子确定子模块，用于以特征线段对的交点的锐角角平分线为描述子的方向量，以交点中心像素块的像素大下作为描述子的尺度量，确定特征线段对的特征线的描述子；
特征线匹配子模块，用于根据相邻关键帧的特征线的位置信息和描述子，确定相邻关键帧之间匹配的特征线；
第三位姿确定子模块，用于根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿。
对于装置实施例而言，由于其与方法实施例基本相似，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
本申请实施例还提供了一种电子设备，包括：
包括处理器、存储器及存储在存储器上并能够在处理器上运行的计算机程序，该计算机程序被处理器执行时实现上述相机位姿确定方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。
本申请实施例还提供了一种计算机可读存储介质，计算机可读存储介质上存储计算机程序，计算机程序被处理器执行时实现上述相机位姿确定方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。
本说明书中的各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似的部分互相参见即可。
本领域内的技术人员应明白，本申请实施例的实施例可提供为方法、装置、或计算机程序产品。因此，本申请实施例可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请实施例可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本申请实施例是参照根据本申请实施例的方法、终端设备(***)、和计 算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理终端设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理终端设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理终端设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理终端设备上，使得在计算机或其他可编程终端设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程终端设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
尽管已描述了本申请实施例的一些实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例做出另外的变更和修改。所以，所附权利要求意欲解释为包括一些实施例以及落入本申请实施例范围的所有变更和修改。
最后，还需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者终端设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者终端设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括要素的过程、方法、物品或者终端设备中还存 在另外的相同要素。
以上对本申请所提供的一种相机位姿确定方法和一种相机位姿确定装置，进行了详细介绍，本文中应用一些个例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其核心思想；同时，对于本领域的一般技术人员，依据本申请的思想，在具体实施方式及应用范围上均会有改变之处，综上，本说明书内容不应理解为对本申请的限制。

Claims (12)

  1. 一种相机位姿确定方法，其特征在于，包括：
    获取相机采集的多个图像帧；所述多个图像帧包括关键帧和非关键帧；
    将所述非关键帧划分为多个图像块，从所述多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿；
    从所述关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿；
    根据多个所述相邻非关键帧之间的第一相机相对位姿和所述相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    从所述关键帧提取满足预设特征线条件的线段作为特征线；
    根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿；
    所述根据多个所述相邻非关键帧之间的第一相机相对位姿和所述相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿，包括：
    根据多个所述相邻非关键帧之间的第一相机相对位姿、所述相邻关键帧之间的第二相机相对位姿和相邻关键帧之间的第三相机位姿，确定当前关键帧对应的优化相机位姿。
  3. 根据权利要求2所述的方法，其特征在于，所述根据多个所述相邻非关键帧之间的第一相机相对位姿、所述相邻关键帧之间的第二相机相对位姿和相邻关键帧之间的第三相机位姿，确定当前关键帧对应的优化相机位姿，包括：
    根据多个所述相邻非关键帧之间的第一相机相对位姿和所述相邻关键帧之间的第二相机相对位姿进行卡尔曼滤波融合，确定当前关键帧对应的估计相机位姿；
    根据所述当前关键帧对应的估计相机位姿，确定针对所述当前关键帧的特征点重投影误差；
    根据所述相邻关键帧之间的第三相机位姿，确定针对所述当前关键帧的特征线重投影误差；
    基于针对所述当前关键帧的特征点重投影误差和特征线重投影误差，确定所述当前关键帧对应的优化相机位姿。
  4. 根据权利要求1所述的方法，其特征在于，所述从所述多个图像块中提取满足预设特征点条件的角点作为特征点，包括：
    对所述多个图像块进行低通滤波处理；
    分别对经过低通滤波处理的各个图像块中的像素点，计算角点化响应值；
    分别对各个图像块中各个像素点对应的角点化响应值进行排序，根据排序结果选取预设个数的像素点作为初始角点；
    确定各个图像块中初始角点的分散程度；
    根据各个图像块中初始角点的分散程度和角点化响应值，为各个图像块设置对应的筛选比例；
    根据各个图像块的筛选比例，从对应的初始角点中筛选候选角点；
    随机采样一致性算法，从所述候选角点中筛选目标角点。
  5. 根据权利要求4所述的方法，其特征在于，所述确定各个图像块中初始角点的分散程度，包括：
    对所述图像块的初始角点进行聚类，得到聚类中心；
    确定各个初始角点到所述聚类中心的像素距离，并根据各个初始角点到所述聚类中心的像素距离确定分散程度。
  6. 根据权利要求5所述的方法，其特征在于，所述分散程度为图像块中各个初始角点的像素距离和值，所述根据各个图像块中初始角点的分散程度和角点化响应值，为各个图像块设置对应的筛选比例，包括：
    对所述图像块的初始角点的角点化响应值，求均方误差和值；
    根据所述图像块中各个初始角点的像素距离和值以及角点化响应值的均方误差和值，计算评价参数；
    根据各个图像块的评价参数，确定各个图像块的筛选比例。
  7. 根据权利要求1所述的方法，其特征在于，所述从所述关键帧中提 取特征点，根据相邻关键帧之间匹配的特征点确定第二相机相对位姿，包括：
    以关键帧中的特征点为中心设置预设像素大小的窗口；
    确定所述特征点的灰度值与窗口中其他像素点的灰度值的差值的绝对值；
    根据所述特征点与窗口中的其他像素点之间的灰度差值的绝对值，生成邻接矩阵；
    根据所述邻接矩阵，生成描述向量作为特征点的描述子；
    根据相邻关键帧的特征点的位置信息和描述子，确定相邻关键帧之间匹配的特征点；
    根据所述相邻关键帧之间匹配的特征点确定相邻关键帧之间的第二相机相对位姿。
  8. 根据权利要求2所述的方法，其特征在于，所述从所述关键帧提取满足预设特征线条件的线段作为特征线，包括：
    检测所述关键帧中的线段，以不平行的两线段作为线段对；
    对所述线段对，筛选出满足预设特征线条件的线段对作为特征线；所述预设特征线条件包括：线段长度大于或等于预设长度阈值，线段对的交点相对线段对的距离小于或等于预设距离阈值，线段对的交点在图像之内。
  9. 根据权利要求8所述的方法，其特征在于，所述根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿，包括：
    以关键帧中不平行的两特征线作为特征线段对；
    以特征线段对的交点的锐角角平分线为描述子的方向量，以交点中心像素块的像素大下作为描述子的尺度量，确定所述特征线段对的特征线的描述子；
    根据相邻关键帧的特征线的位置信息和描述子，确定相邻关键帧之间匹配的特征线；
    根据相邻关键帧之间匹配的特征线，确定相邻关键帧之间的第三相机位姿。
  10. 一种相机位姿确定装置，其特征在于，包括：
    图像获取模块，用于获取相机采集的多个图像帧；所述多个图像帧包括关键帧和非关键帧；
    第一位姿确定模块，用于将所述非关键帧划分为多个图像块，从所述多个图像块中提取满足预设特征点条件的角点作为特征点，以及根据相邻的非关键帧之间匹配的特征点采用光流法确定第一相机相对位姿；
    第二位姿确定模块，用于从所述关键帧中提取特征点，根据相邻关键帧之间匹配的特征点采用特征点匹配法确定第二相机相对位姿；
    优化位姿确定模块，用于根据多个所述相邻非关键帧之间的第一相机相对位姿和所述相邻关键帧之间的第二相机相对位姿，确定当前关键帧对应的优化相机位姿。
  11. 一种电子设备，其特征在于，包括：处理器、存储器及存储在所述存储器上并能够在所述处理器上运行的计算机程序，所述计算机程序被所述处理器执行时实现如权利要求1-9中任一项所述的相机位姿确定方法的步骤。
  12. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储计算机程序，所述计算机程序被处理器执行时实现如权利要求1-9中任一项所述的相机位姿确定方法的步骤。
PCT/CN2022/132927 2022-01-06 2022-11-18 一种相机位姿确定方法和装置 WO2023130842A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210012434.8 2022-01-06
CN202210012434.8A CN114399532A (zh) 2022-01-06 2022-01-06 一种相机位姿确定方法和装置

Publications (1)

Publication Number Publication Date
WO2023130842A1 true WO2023130842A1 (zh) 2023-07-13

Family

ID=81228722

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/132927 WO2023130842A1 (zh) 2022-01-06 2022-11-18 一种相机位姿确定方法和装置

Country Status (2)

Country Link
CN (1) CN114399532A (zh)
WO (1) WO2023130842A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114399532A (zh) * 2022-01-06 2022-04-26 广东汇天航空航天科技有限公司 一种相机位姿确定方法和装置

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170278231A1 (en) * 2016-03-25 2017-09-28 Samsung Electronics Co., Ltd. Device for and method of determining a pose of a camera
CN108537845A (zh) * 2018-04-27 2018-09-14 腾讯科技（深圳）有限公司 位姿确定方法、装置及存储介质
CN108648215A (zh) * 2018-06-22 2018-10-12 南京邮电大学 基于imu的slam运动模糊位姿跟踪算法
CN109558879A (zh) * 2017-09-22 2019-04-02 华为技术有限公司 一种基于点线特征的视觉slam方法和装置
CN112097742A (zh) * 2019-06-17 2020-12-18 北京地平线机器人技术研发有限公司 一种位姿确定方法及装置
CN112734797A (zh) * 2019-10-29 2021-04-30 浙江商汤科技开发有限公司 图像特征跟踪方法、装置及电子设备
CN113112542A (zh) * 2021-03-25 2021-07-13 北京达佳互联信息技术有限公司 一种视觉定位方法、装置、电子设备及存储介质
CN113744315A (zh) * 2021-09-07 2021-12-03 北京航空航天大学 一种基于双目视觉的半直接视觉里程计
CN114399532A (zh) * 2022-01-06 2022-04-26 广东汇天航空航天科技有限公司 一种相机位姿确定方法和装置

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170278231A1 (en) * 2016-03-25 2017-09-28 Samsung Electronics Co., Ltd. Device for and method of determining a pose of a camera
CN109558879A (zh) * 2017-09-22 2019-04-02 华为技术有限公司 一种基于点线特征的视觉slam方法和装置
CN108537845A (zh) * 2018-04-27 2018-09-14 腾讯科技（深圳）有限公司 位姿确定方法、装置及存储介质
CN108648215A (zh) * 2018-06-22 2018-10-12 南京邮电大学 基于imu的slam运动模糊位姿跟踪算法
CN112097742A (zh) * 2019-06-17 2020-12-18 北京地平线机器人技术研发有限公司 一种位姿确定方法及装置
CN112734797A (zh) * 2019-10-29 2021-04-30 浙江商汤科技开发有限公司 图像特征跟踪方法、装置及电子设备
CN113112542A (zh) * 2021-03-25 2021-07-13 北京达佳互联信息技术有限公司 一种视觉定位方法、装置、电子设备及存储介质
CN113744315A (zh) * 2021-09-07 2021-12-03 北京航空航天大学 一种基于双目视觉的半直接视觉里程计
CN114399532A (zh) * 2022-01-06 2022-04-26 广东汇天航空航天科技有限公司 一种相机位姿确定方法和装置

Also Published As

Publication number Publication date
CN114399532A (zh) 2022-04-26

Similar Documents

Publication Publication Date Title
CN110322500B (zh) 即时定位与地图构建的优化方法及装置、介质和电子设备
CN111060115B (zh) 一种基于图像边缘特征的视觉slam方法及***
US9330471B2 (en) Camera aided motion direction and speed estimation
CN110807809B (zh) 基于点线特征和深度滤波器的轻量级单目视觉定位方法
KR101532864B1 (ko) 모바일 디바이스들에 대한 평면 맵핑 및 트래킹
CN109961506A (zh) 一种融合改进Census图的局部场景三维重建方法
CN112115980A (zh) 基于光流跟踪和点线特征匹配的双目视觉里程计设计方法
CN112419497A (zh) 基于单目视觉的特征法与直接法相融合的slam方法
Gedik et al. 3-D rigid body tracking using vision and depth sensors
Li et al. Review of vision-based Simultaneous Localization and Mapping
CN112752028A (zh) 移动平台的位姿确定方法、装置、设备和存储介质
CN111681275B (zh) 一种双特征融合的半全局立体匹配方法
CN116878501A (zh) 一种基于多传感器融合的高精度定位与建图***及方法
CN112785705A (zh) 一种位姿获取方法、装置及移动设备
WO2023130842A1 (zh) 一种相机位姿确定方法和装置
Rückert et al. Snake-SLAM: Efficient global visual inertial SLAM using decoupled nonlinear optimization
CN116468786A (zh) 一种面向动态环境的基于点线联合的语义slam方法
Lim et al. A review of visual odometry methods and its applications for autonomous driving
WO2022079258A1 (en) Visual-inertial localisation in an existing map
Goldman et al. Robust epipolar geometry estimation using noisy pose priors
CN112731503A (zh) 一种基于前端紧耦合的位姿估计方法及***
CN116894876A (zh) 基于实时图像的6-dof的定位方法
CN111829522A (zh) 即时定位与地图构建方法、计算机设备以及装置
CN115861352A (zh) 单目视觉、imu和激光雷达的数据融合和边缘提取方法
Svedman et al. Structure from stereo vision using unsynchronized cameras for simultaneous localization and mapping

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22918294

Country of ref document: EP

Kind code of ref document: A1