WO2022267257A1 - Image registration method, visual positioning method, apparatus, device, medium, and program - Google Patents

Image registration method, visual positioning method, apparatus, device, medium, and program Download PDF

Info

Publication number
WO2022267257A1
WO2022267257A1 PCT/CN2021/121049 CN2021121049W WO2022267257A1 WO 2022267257 A1 WO2022267257 A1 WO 2022267257A1 CN 2021121049 W CN2021121049 W CN 2021121049W WO 2022267257 A1 WO2022267257 A1 WO 2022267257A1
Authority
WO
WIPO (PCT)
Prior art keywords
pose
image
image frame
current
transformation parameter
Prior art date
Application number
PCT/CN2021/121049
Other languages
French (fr)
Chinese (zh)
Inventor
王求元
Original Assignee
浙江商汤科技开发有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 浙江商汤科技开发有限公司 filed Critical 浙江商汤科技开发有限公司
Publication of WO2022267257A1 publication Critical patent/WO2022267257A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Definitions

  • the present disclosure relates to the technical field of artificial intelligence, and in particular to an image registration method, a visual positioning method, a device, a device, a medium and a program.
  • augmented reality Augmented Reality
  • virtual reality Virtual Reality
  • the computing assistance function of electronic equipment can digitize the surrounding environment, so that users can obtain the experience of interacting with the real environment.
  • Image registration is the focus of research in the field of computer vision such as AR and VR.
  • image registration technology the transformation parameters between the current image captured by the camera and the target image can be obtained, so that the target image can be determined in the current image through the transformation parameters. position in .
  • each frame of image is usually registered with the target image to obtain transformation parameters.
  • Such a registration method has a large amount of computation and is not very accurate.
  • Embodiments of the present disclosure provide an image registration method, a visual positioning method, a device, a device, a medium, and a program.
  • An embodiment of the present disclosure provides an image registration method, the method is executed by an electronic device, and the method includes:
  • the current image frame Acquiring the current image frame; based on the target image information in the current image frame and other image frames, determining a first transformation parameter between the current image frame and other image frames; wherein, the target image information is image information about the target image; the The first transformation parameter includes a homography matrix between the current image frame and the other image frames;
  • a third transformation parameter between the current image frame and the target image is obtained; wherein, the second transformation parameter includes the current image The homography matrix between the frame and the target image.
  • the current image frame and other image frames can be directly used
  • the target image information in the frame realizes the registration of the current image frame and the target image.
  • this method has a small amount of calculation and can improve Registration speed and accuracy of image registration.
  • An embodiment of the present disclosure also provides a visual positioning method, the method is executed by an electronic device, and the method includes: acquiring the current transformation parameter between the current image frame and the target image, wherein the current transformation parameter is obtained by any of the previous The third transformation parameter obtained by the image registration method; using the current transformation parameter, the first pose of the current image frame in the world coordinate system is obtained, wherein the world coordinate system is established based on the plane where the target image is located.
  • the first pose of the current image frame in the world coordinate system can be obtained, and the shooting Visual positioning of the device.
  • An embodiment of the present disclosure also provides an image registration device, including: an image registration device, a first parameter acquisition module, and a second parameter acquisition module, wherein:
  • the image acquisition module is configured to acquire the current image frame
  • the first parameter acquisition module is configured to determine a first transformation parameter between the current image frame and other image frames based on target image information in the current image frame and other image frames; wherein the target image information is information about the target image Image information; the first transformation parameter includes a homography matrix between the current image frame and the other image frames;
  • the second parameter acquisition module is configured to obtain a third transformation parameter between the current image frame and the target image based on the first transformation parameter and second transformation parameters between other image frames and the target image; the second The transformation parameters include a homography matrix between the current image frame and the target image.
  • An embodiment of the present disclosure also provides a visual positioning device, including: a parameter acquisition module and a first pose acquisition module, wherein:
  • the parameter acquisition module is configured to acquire a current transformation parameter between the current image frame and the target image, wherein the current transformation parameter is the third transformation parameter obtained by any previous image registration method;
  • the first pose acquisition module is configured to use the current transformation parameters to obtain the first pose of the current image frame in the world coordinate system, wherein the world coordinate system is established based on the plane where the target image is located.
  • An embodiment of the present disclosure also provides an electronic device, including a memory and a processor coupled to each other, and the processor is used to execute the program instructions stored in the memory, so as to realize the image registration method in the first aspect and the image registration method in the second aspect.
  • Visual positioning method is used to realize the image registration method in the first aspect and the image registration method in the second aspect.
  • An embodiment of the present disclosure also provides a computer-readable storage medium, on which program instructions are stored.
  • program instructions When the program instructions are executed by a processor, the aforementioned image registration method and the aforementioned visual positioning method are implemented.
  • An embodiment of the present disclosure also provides a computer program, where the computer program includes computer readable codes, and when the computer readable codes run in an electronic device, the processor of the electronic device executes the following steps: The image registration method described in any one of the preceding items, and the visual positioning method described in any one of the preceding items.
  • the image registration method obtains the first transformation parameter between the current image frame and other image frames, and then combines the first transformation parameter with the second transformation parameter between other image frames and the target image.
  • the transformation parameters can directly use the target image information in the current image frame and other image frames, and realize the registration of the current image frame and the target image, compared to using all the image information in the current image frame and other image frames for image registration , the calculation amount of this method is small, and it can also improve the registration speed and the accuracy of image registration.
  • FIG. 1 is a schematic flowchart of an image registration method provided by an embodiment of the present disclosure
  • Fig. 2 is a schematic flow chart of determining the first transformation parameter in the image registration method provided by the embodiment of the present disclosure
  • Fig. 3 is a schematic flowchart of a first visual positioning method provided by an embodiment of the present disclosure
  • Fig. 4 is a schematic flow diagram of determining the first pose in the visual positioning method provided by an embodiment of the present disclosure
  • Fig. 5 is a second schematic flowchart of a visual positioning method provided by an embodiment of the present disclosure.
  • FIG. 6 is a schematic flowchart of offset acquisition in a visual positioning method provided by an embodiment of the present disclosure
  • FIG. 7 is a schematic flowchart of a third visual positioning method provided by an embodiment of the present disclosure.
  • Fig. 8 is a schematic structural diagram of an image registration device provided by an embodiment of the present disclosure.
  • FIG. 9 is a schematic structural diagram of a visual positioning device provided by an embodiment of the present disclosure.
  • FIG. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • Fig. 11 is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present disclosure.
  • system and “network” are often used interchangeably herein.
  • the term “and/or” in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations.
  • the character "/” in this article generally indicates that the contextual objects are an “or” relationship.
  • “many” herein means two or more than two.
  • Fig. 1 is a schematic flowchart of an image registration method provided by an embodiment of the present disclosure.
  • the image registration method provided by the embodiments of the present disclosure is executed by an electronic device; wherein, the image registration method may be executed by a processor of the electronic device.
  • the processor of an electronic device may be an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), Programmable Logic Device (Programmable Logic Device, PLD), Field Programmable Logic Gate Array (Field Programmable Gate Array, FPGA), Central Processing Unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor in at least one.
  • ASIC Application Specific Integrated Circuit
  • DSP Digital Signal Processor
  • DSPD Digital Signal Processing Device
  • PLD Programmable Logic Device
  • Field Programmable Logic Gate Array Field Programmable Gate Array
  • FPGA Field Programmable Gate Array
  • CPU Central Processing Unit
  • controller microcontroller, microprocessor in at least one.
  • the process may include step S11 to step S13:
  • Step S11 Obtain the current image frame.
  • the current image frame may be an image captured by an image acquisition device of the electronic device.
  • the current image frame may be an image captured by an electronic device such as a mobile phone, a tablet computer, or smart glasses; or, in a video surveillance scenario, the current image frame may be a surveillance camera
  • the captured images are not limited here. Other scenarios can be deduced in the same way, and examples are not given here.
  • the current image frame may include the target image; for example, if the current image frame includes the target image, registration between the current image frame and the target image may be implemented.
  • Step S12 Based on the target image information in the current image frame and other image frames, determine a first transformation parameter between the current image frame and other image frames.
  • the first transformation parameter includes a homography matrix between the current image frame and other image frames.
  • the target image may be an image on a plane.
  • the plane may be a flat ground or a flat wall;
  • the target image may be pre-acquired, that is, the target image may be It is predetermined before executing the image registration method provided by the embodiment of the present disclosure.
  • the target image may be set according to actual application conditions.
  • the image of building A can be acquired in advance; or, when the position of person B in the current image frame needs to be determined, the person B can be acquired in advance
  • the image of B other situations can be deduced in the same way, so we will not give examples one by one here.
  • the target image can be determined from the acquired images.
  • the interior of the building can be photographed in advance to obtain a certain number of images of the interior of the building, and then in these images, an image area containing a specific object is selected as the target image, for example, a certain image includes a painting, the painting can be used as the target image.
  • the target image information may be image information about the target image.
  • the target image information may be feature points and corresponding feature representations obtained by performing feature extraction on target images in the current image frame and other image frames.
  • the feature extraction algorithm can be a feature point detection (Features From Accelerated Segment Test, FAST) algorithm, a scale-invariant feature transformation (Scale-invariant Feature Transform, SIFT) algorithm, a fast feature point extraction (Oriented FAST and Rotated BRIEF, ORB) algorithm, etc. Wait.
  • FAST Features From Accelerated Segment Test
  • SIFT Scale-invariant Feature Transform
  • ORB fast feature point extraction
  • the feature extraction algorithm is the ORB algorithm
  • the descriptor obtained by the ORB algorithm may be used as a feature representation.
  • the feature points extracted based on the image frame may be considered to be on the same plane as the target image.
  • the feature points obtained by feature extraction through the feature extraction algorithm mentioned in the above embodiments can be considered to be on the same plane as the target image.
  • both the current image frame and other frame images may include target images, so as to implement the image registration method provided by the embodiments of the present disclosure.
  • the other frame images are similar to the current image frame, that is, they can be captured or captured by a video surveillance machine.
  • other image frames and the current image frame are sequentially captured by the image acquisition device of the electronic device. That is to say, the shooting time of other image frames is earlier than the shooting time of the current image frame.
  • the position of the target image in other image frames can be known, and then the feature point tracking method, such as optical flow algorithm, can be used to track the position of the point on the target image in other image frames in the current image frame, In this way, the position of the target image in the current image frame is determined, and then target image information in the current image frame and other image frames can be obtained.
  • the feature point tracking method such as optical flow algorithm
  • the current image frame after obtaining the target image information in the current image frame and other image frames, can be registered with other image frames based on a general image registration method; exemplary, image registration Quasi-algorithms can be based on grayscale and templates, or feature-based matching methods. Exemplary, through feature-based matching method, can obtain a certain number of matching point pairs about current image frame and target image, just can utilize random consistent sampling algorithm (Random Sample Consensus, RANSAC) to calculate current image frame and target image Between the transformation parameters, in order to achieve the registration of the two images.
  • RANSAC Random Sample Consensus
  • Step S13 Obtain a third transformation parameter between the current image frame and the target image based on the first transformation parameter and the second transformation parameter between other image frames and the target image.
  • the second transformation parameter includes a homography matrix between other image frames and the target image.
  • other image frames when other image frames include the target image, other image frames may be used to perform registration with the target image.
  • the second transformation parameters between other image frames and the target image may be obtained by using a common image registration algorithm.
  • the second transformation parameter may be acquired through a template matching method, or through a feature-based matching method, and the like.
  • feature points on other image frames and feature points on the target image may be obtained first, then matching calculations are performed, and second transformation parameters are finally obtained.
  • the previous image frame B such as the previous image frame
  • the second Transformation parameters that is, using the transformation parameters between other image frames A and the previous image frame B, and the transformation parameters between the previous image frame B and the target image, to obtain the transformation parameters between other image frames A and the target image, that is, the above Second transformation parameter.
  • the first transformation parameter between the current image frame and other image frames, and the second transformation between other image frames and the target image can be parameter, using other image frames as connection points to establish a registration relationship between the current image frame and the target image, that is, based on the first transformation parameter and the second transformation parameter, the third transformation parameter can be obtained.
  • the product of the first transformation parameter and the second transformation parameter can be used as the third transformation parameter, as shown in formula (1):
  • the image registration method after obtaining the first transformation parameters between the current image frame and other image frames, combining the first transformation parameters and the transformation parameters between other image frames and the target image
  • the second transformation parameter can directly use the target image information in the current image frame and other image frames to realize the registration of the current image frame and the target image, compared to using all the image information in the current image frame and every other image frame
  • this method has a small amount of calculation, improves the registration speed and the accuracy of image registration.
  • Fig. 2 is a schematic flowchart of determining a first transformation parameter in an image registration method provided by an embodiment of the present disclosure. As shown in Figure 2, step S12 can be realized through steps S121 to S123:
  • Step S121 Find at least one first feature point of the target image from other image frames.
  • the feature points extracted from the image frame may include feature points obtained by performing feature extraction on a series of image frames in the image pyramid established based on the image frame.
  • step S121 can be implemented through steps S1211 to S1212:
  • Step S1211 Determine the target area of the target image in other image frames based on the second transformation parameter
  • the second transformation parameter between other image frames and the target image can be used to represent the corresponding relationship between points on the target image and points on other image frames, so it can be determined that the points of the target image are in other image frames Corresponding points, from which the target area can be determined.
  • Step S1212 Extract at least one first feature point from the target area.
  • At least one first feature point may be extracted from the target area, for example, at least one first feature point may be obtained by using a feature extraction algorithm.
  • the target area on other image frames can be determined by using the second transformation parameters, thereby obtaining at least one first feature point for subsequent image registration.
  • Step S122 Find at least one second feature point on the target image from the current image frame.
  • finding out at least one second feature point about the target image in the current image frame may be to use a feature extraction algorithm to perform feature point extraction in the current image frame, so that a series of feature points can be obtained, and then These feature points are matched with the first feature points, and finally the feature points whose matching degree meets the requirements are used as the second feature points.
  • the feature points extracted from the current image frame may be matched with the feature points obtained from the target image, and then the feature points whose matching degree meets the requirements are used as the second feature points.
  • the second feature point may be obtained by using a feature point tracking method.
  • the feature point tracking method may be an optical flow algorithm.
  • at least one first feature point can be tracked respectively to obtain at least one second feature point about the target image in the current image frame, that is, each first feature point can be tracked to obtain each A first feature point corresponds to points in the current image frame, and then feature extraction is performed on these corresponding points to obtain a second feature point.
  • the feature points corresponding to the first feature points in the current image frame can be obtained, and then at least one second feature point can be obtained for subsequent image registration.
  • Step S123 Based on the first feature point and the second feature point, determine a first transformation parameter.
  • the first transformation parameter may be obtained based on feature information of the first feature point and the second feature point.
  • the first feature point and the second feature point may be processed based on a feature point matching method, so as to determine the first transformation parameter.
  • RANSAC may be used to process the first feature point and the second feature point to determine the first transformation parameter.
  • the third transformation parameter after obtaining the third transformation parameter between the current image frame and the target image frame, the third transformation parameter may be further optimized to obtain a more accurate third transformation parameter.
  • a preset optimization manner may be used to optimize the third transformation parameter.
  • the preset optimization method may be iterative optimization; for example, the iterative optimization method may be used to calculate the similarity between the target image and the target image in the current frame image, and optimize the first frame based on this. Three transformation parameters.
  • optimizing the third transformation parameter can be realized by formula (2):
  • Score represents the similarity score, and the higher the score, the more similar the target image is to the current frame image;
  • F(H -1 ) represents the result of the transformation of the current frame image F by the third candidate transformation parameter H;
  • f The (T,F(H -1 ) function is used to calculate the similarity between the target image T and F(H -1 ), that is, the f(T,F(H -1 ) function is used to calculate the target image and the current frame image
  • this function can be the error sum of squares (Sum of Squared Differences, SSD) function, or normalized cross-correlation (Normalized Cross Correlation, NCC ) function etc.; H, to increase the similarity between the target image and the current frame image as much as possible.
  • the iterative optimization method is, for example, Gauss-Newton (Gauss-Newton) iterative method or Levenberg-Marquard algorithm, etc.
  • the SSD function can be shown as formula (3):
  • the SSD function can also be shown in formula (4):
  • the optimization of the third transformation parameter can be realized, so that a more accurate third transformation parameter can be obtained, and the effect of image registration can be improved.
  • Fig. 3 is a schematic flowchart of a first visual positioning method provided by an embodiment of the present disclosure.
  • the visual positioning method provided by the embodiments of the present disclosure is executed by an electronic device, wherein the visual positioning may be executed by a processor of the electronic device.
  • the processor of the electronic device may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.
  • the method includes step S21 to step S22:
  • Step S21 Obtain the current transformation parameters between the current image frame and the target image.
  • the current transformation parameter is the third transformation parameter obtained by using the image registration method provided in the foregoing embodiments.
  • Step S22 Using the current transformation parameters, obtain the first pose of the current image frame in the world coordinate system.
  • the world coordinate system may be established based on the plane where the target image is located.
  • the plane where the target image is located may be the preset plane of the world coordinate system, for example, the plane where the target image is located is the XOY plane, or the XOZ plane or the YOZ plane of the world coordinate system.
  • the center of the target image can be at the origin of the world coordinate system
  • the horizontal axis of the target image can be parallel to the X axis of the world coordinate system
  • the vertical axis of the target image can be parallel to the Y axis of the world coordinate system
  • the The Z axis can be perpendicular to the target image plane.
  • the current transformation parameters between the current image frame and the target image have been obtained, and the world coordinate system is established based on the plane where the target image is located, the current transformation parameters can be converted to obtain the current image frame at The first pose in the world coordinate system.
  • the first pose may be a pose in the world coordinate system corresponding to the current image frame captured by the image acquisition device of the electronic device.
  • the algorithm for converting the current transformation parameters to obtain the first pose may be a pose estimation (Perspective-n-Point, PnP) algorithm.
  • the current image frame by obtaining the current transformation parameters between the current image frame and the target image, and establishing a world coordinate system based on the plane where the target image is located, the current image frame can be obtained at The first pose in the world coordinate system, thus realizing the visual positioning of the shooting device.
  • Fig. 4 is a schematic flowchart of determining a first pose in the visual positioning method provided by an embodiment of the present disclosure. As shown in FIG. 4 , in the visual positioning method provided by the embodiment of the present disclosure, before step S22 , step S31 to step S33 may also be executed.
  • Step S31 Judging whether the current transformation parameters meet the preset requirements.
  • judging whether the transformation parameters meet the preset requirements may include judging whether the accuracy of the current transformation parameters meets the requirements. For example, if the accuracy of the current transformation parameters meets the preset requirements, it can be considered that the current transformation parameters The accuracy of the parameters is high; if the accuracy of the current transformation parameters does not meet the preset requirements, it can be considered that the accuracy of the current transformation parameters is low, and the current transformation parameters cannot be used to obtain the first pose.
  • judging whether the transformation parameters meet the preset requirements may be judging whether the similarity between the current image frame calculated using the current transformation parameters and the target image meets the preset requirements.
  • the transformation parameters meet the preset requirements, which may include that the Score according to the formula (2) in the aforementioned image registration method meets the requirements.
  • step S32 when the transformation parameters meet the preset requirements, step S32 may be executed; when the transformation parameters do not meet the preset requirements, step S33 may be executed.
  • Step S32 In response to the current transformation parameters satisfying the preset requirements, execute using the current transformation parameters to obtain the first pose of the current image frame in the world coordinate system.
  • the current transformation parameters meet the preset requirements, it means that the accuracy of the current transformation parameters is relatively high.
  • execute using the current transformation parameters to obtain the current image Manipulation of the frame's first pose in world coordinates.
  • Step S33 In response to the fact that the current transformation parameters do not meet the preset requirements, determine the first pose by using the second pose of other images in the world coordinate system and the photometric error between the current image frame and other image frames.
  • the current transformation parameters do not meet the preset requirements, it means that the accuracy of the current transformation parameters is not high, that is, the accuracy of the first pose obtained through the current transformation parameters is not high, and at this time, the response to the current The transformation parameters do not meet the preset requirements, and the second pose of other images in the world coordinate system and the photometric error between the current image frame and other image frames are used to determine the operation of the first pose.
  • the first pose can be obtained through the current transformation parameters when the preset requirements are met, and if the preset requirements are not met , the first pose can be obtained by other methods, so that a more accurate first pose can be flexibly obtained through various ways.
  • the operation of obtaining the pose acquisition mode obtained in the previous image frame may be performed first, so as to determine the transformation parameters in the previous image frame
  • the pose of the last image frame may be obtained by image tracking, or by other methods. Exemplarily, other methods may be methods such as visual navigation.
  • the current transformation parameter between the current image frame and the target image is acquired.
  • the image tracking method is to use the transformation parameters between the previous image frame and the target image to determine the pose of the previous image frame in the world coordinate system.
  • the pose of the last image frame in the world coordinate system can be acquired through the homography matrix between the last image frame and the target image.
  • the pose of the previous image frame is obtained by image tracking, this means that there is a target image in the previous image frame, then there may also be a target image in the current image frame, so you can choose to pass the current image frame
  • the current transformation parameters between the target image and the first pose of the current image frame in the world coordinate system can be obtained through the homography matrix between the last image frame and the target image.
  • the pose acquisition method of the previous image frame is not obtained by image tracking, it means that there may not be a target image in the previous image frame, so other methods can be selected to obtain the current image frame The first pose in world coordinates.
  • the positioning state of the electronic device may include two states, one is an image tracking state, and the other is a visual navigation state.
  • the operation of judging the positioning state of the electronic device may also be performed first, so as to determine whether to acquire the current transformation parameters between the current image frame and the target image.
  • the positioning state may be determined according to an acquisition method of the pose of the last image frame in the world coordinate system.
  • the image registration method may include the image registration method provided in the foregoing embodiments.
  • the positioning state may be determined as the visual navigation state.
  • step S33 may be executed in the case of the current visual navigation state.
  • the visual positioning method After judging whether the current transformation parameters meet the preset requirements, it can be indicated that the operation of obtaining the current transformation parameters between the current image frame and the target image has been executed. At this time, it can be Confirm that the positioning status is image tracking status.
  • the positioning state may be determined again according to the judgment result of whether the current transformation parameter meets the preset requirement. For example, if the current transformation parameters meet the preset requirements, keep in the image tracking state, and the current transformation parameters meet the preset requirements, which means that the first pose of the current image frame can be obtained through the current transformation parameters, so you can keep in the image tracking state state.
  • the current transformation parameters do not meet the preset requirements, it means that the first pose of the current image frame cannot be obtained through the current transformation parameters, at this time, it may switch to the visual navigation state, and perform step S33.
  • the positioning state of the electronic device can be determined according to whether the current transformation parameters meet the preset requirements, and then the specific method for obtaining the first pose can be determined.
  • Fig. 5 is a second schematic flowchart of a visual positioning method provided by an embodiment of the present disclosure.
  • the visual positioning method provided by the embodiments of the present disclosure other image frames and the current image frame are sequentially captured by an image acquisition device of an electronic device.
  • using the second pose of other images in the world coordinate system and the photometric error between the current image frame and other image frames to determine the first pose can be achieved through steps S331 to S333:
  • Step S331 Obtain a first reference pose.
  • the first reference posture is the posture of the image acquisition device corresponding to the shooting moment of the current image frame and relative to the reference plane.
  • the first reference pose may be rotation information of the electronic device, that is, rotation information of the electronic device relative to a reference plane.
  • the first reference posture may be detected by a sensing device of the electronic device.
  • the sensing device may be a gyroscope.
  • the difference between the detection moment of the first reference pose and the shooting moment of the current image frame does not exceed a first preset time difference.
  • the first preset difference may be a short period of time, such as 20 milliseconds or 15 milliseconds, etc., and the first preset difference may be adjusted and set according to actual needs.
  • the detection moment closest to the shooting moment of the current image frame can be selected. time, and the attitude of the corresponding image acquisition device relative to the reference plane to obtain the first reference attitude; for example, the difference between the detection moment and the shooting moment of the current image frame does not exceed the first preset time difference, that is, the detection moment It is very close to the shooting moment of the current image frame, and at this time, it can be considered that the first reference pose is the pose information of the device at the shooting moment of the current image frame.
  • the reference plane is, for example, a certain plane determined based on detection parameters of the sensing device.
  • Step S332 Using the offset between the reference plane and the preset plane in the world coordinate system, the first reference attitude is adjusted to obtain the second reference attitude.
  • the preset plane in the world coordinate system is, for example, the XOY plane, or the XOZ plane, the YOZ plane, etc. of the world coordinate system.
  • the preset plane may be the XOY plane of the world coordinate system.
  • the plane where the target image is located may be a preset plane.
  • the first reference pose after the first reference pose is obtained, it may indicate that the rotation information of the electronic device relative to the reference plane has been obtained. At this time, the offset between the reference plane and other planes can be obtained, and the offset can be used to adjust the first reference attitude, so as to obtain the second reference attitude of the electronic device relative to other planes.
  • the first The second reference posture may include rotation information of the electronic device relative to other planes.
  • other planes may be preset planes in the world coordinate system, so the second reference pose may be regarded as rotation information of the electronic device relative to the preset plane of the world coordinate system.
  • the first reference attitude can be detected by the gyroscope, so the reference plane can be a certain plane determined based on the gyroscope.
  • the reference plane and the preset plane in the world coordinate system The offset between the first reference pose is adjusted, and the obtained second reference pose can also be considered as the rotation amount required to transform the reference plane to the preset plane.
  • the first reference attitude information can be adjusted based on the offset to obtain the second reference attitude information, which can be obtained relative to
  • the reference attitude information of a plane other than the reference plane such as the preset plane of the world coordinate system
  • the reference attitude information can be used to optimize the final position pose information, which improves the accuracy of the final pose information.
  • Fig. 6 is a schematic flowchart of obtaining an offset in the visual positioning method provided by an embodiment of the present disclosure. As shown in FIG. 6, in the visual positioning method provided by the embodiment of the present disclosure, before step S332, steps S41 to S42 may also be performed:
  • Step S41 Obtain the third pose of the first historical image frame in the world coordinate system, and obtain the third reference pose.
  • the third reference pose is the pose of the image acquisition device corresponding to the shooting moment of the first historical image frame and relative to the reference plane; the third pose is determined based on the target image; the preset plane is the plane where the target image is located.
  • the preset plane may be the plane where the target image is located.
  • the third pose may be determined based on the target image; for example, the third pose may be based on the target image, using an image registration algorithm to detect the first historical image frame and the target image, to obtain The fourth transformation parameter between them is obtained after converting the fourth transformation parameter.
  • the third reference posture may be the posture of the image acquisition device of the electronic device corresponding to the shooting moment of the second historical image frame relative to the reference plane; for example, the second historical image frame is located at the first Before the history image frame;
  • the third reference pose may be electronically detected by the sensing device of the device; for example, the difference between the detection moment of the third reference pose and the shooting moment of the first historical image frame It does not exceed the second preset time difference. At this time, it can be considered that the attitude information of the third reference attitude is the same as that of the third attitude.
  • first feature points corresponding to the first historical image frame and second feature points corresponding to the target image can be obtained.
  • the embodiments of the present disclosure do not specifically limit the number of feature points.
  • the feature extraction of the first historical image frame and the target image may be implemented by a feature extraction algorithm, and the feature extraction algorithm may be any algorithm among the FAST algorithm, the SIFT algorithm and the ORB algorithm.
  • a feature representation corresponding to each feature point can also be obtained.
  • the feature representation can be a feature vector.
  • Each feature point may have a corresponding feature representation.
  • calculating the matching degree between the first feature point and the second feature point may be calculating the distance between the feature representation of the first feature point and the feature representation of the second feature point, the closer the distance, the first feature can be determined The more the point matches the second feature point.
  • obtaining the third pose of the first historical image frame in the world coordinate system can be achieved by the following operations:
  • the image registration algorithm may be RANSAC.
  • obtaining the third pose of the first historical image frame in the world coordinate system can be achieved by the following operations:
  • the second historical image frame is located before the first historical image frame.
  • the visual positioning method provided by the embodiments of the present disclosure, by obtaining the fourth transformation parameter between the first historical image frame and the target image, or by using the difference between the first historical image frame and the second historical image frame
  • the fifth transformation parameter between and the sixth transformation parameter between the second historical image frame and the target image to obtain the fourth transformation parameter can obtain the third pose of the first historical image frame to realize visual positioning, so that The way of visual positioning is more flexible.
  • using the fourth transformation parameter to obtain the third pose can be achieved by the following operations:
  • the fourth transformation parameter In response to the fourth transformation parameter meeting the preset requirement, it is determined to be in the image tracking state, and the fourth transformation parameter is used to obtain the third pose.
  • the fourth transformation parameter before using the fourth transformation parameter to obtain the third pose, it may also be judged first whether the fourth transformation parameter satisfies the preset requirement.
  • the fourth transformation parameter does not meet the preset requirements, it may be determined that the electronic device is in the visual navigation state, and the second pose in the world coordinate system using other images provided in the foregoing embodiments, and the current image
  • the photometric error between the frame and other image frames determines the operation of the first pose.
  • the process of obtaining the fourth transformation parameter between the first historical image frame and the target image, or obtaining the sixth transformation parameter between the second historical image frame and the target image can be performed through steps A1 to Step A2 is implemented:
  • Step A1 Select one of the first matching point pairs as a target matching point pair.
  • the feature points obtained by feature extraction of the target image can be defined as the third feature point; the feature points obtained by feature extraction based on the first historical image frame or the second historical image frame can be defined as the third feature point Four feature points; for example, by calculating the matching degree between the third feature point and the fourth feature point, the first matching point pair can be obtained.
  • a group of first matching point pairs is selected as the target matching point pair, which can be selected from the most matching point pair.
  • the third feature point can be the first matching point
  • the fourth feature point The point can be the second matching point.
  • Step A2 Obtain the homography matrix corresponding to the target matching point pair based on the direction information of the target matching point pair.
  • the direction information of the target matching point pair may represent the rotation angle of the first historical frame image relative to the target image, or may represent the rotation angle of the second historical frame image relative to the target image.
  • first the first image area centered on the first matching point can be extracted in the target image, and the second image area centered on the second matching point can be extracted in the first historical image frame or the second historical image frame ; Then, determine the first deflection angle of the first image area and the second deflection angle of the second image area; then obtain transformation parameters based on the first deflection angle and the second deflection angle.
  • the transformation parameter may be obtained based on the direction information of the target matching point pair and the pixel coordinate information of the first matching point and the second matching point in the target matching point pair.
  • the first deflection angle may be the distance between the line connecting the centroid of the first image area and the center of the first image area and a preset direction (for example, the X axis of the world coordinate system).
  • the second deflection angle may be the directional angle between the line connecting the centroid of the second image area and the center of the second image area and the preset direction.
  • the first deflection angle It can be directly obtained by formula (5):
  • (x, y) represents the offset of a certain pixel point in the first image area relative to the center of the first image area
  • I(x, y) represents the pixel value of the pixel point
  • ⁇ represents the calculation and, the summation range is the pixels in the first image area.
  • the second deflection angle can also be calculated by the same method.
  • the conversion parameters between the first historical frame image or the second historical frame image and the target image can be obtained through steps B1 to B2:
  • Step B1 Obtain the angle difference between the first deflection angle and the second deflection angle.
  • ⁇ is the angle difference
  • T represents the target image
  • F represents the first historical frame image or the second historical frame image.
  • Step B2 Obtain a first candidate transformation parameter based on the angle difference and the scale corresponding to the first matching point pair.
  • the first candidate transformation parameter may be a corresponding homography matrix between the first historical frame image or the second historical frame image and the target image.
  • H is the corresponding homography matrix between the target image and the first historical frame image or the second historical frame image, that is, the first candidate transformation parameter;
  • H r indicates that the first historical frame image or the second historical frame image is relative to The translation amount of the target image;
  • H S represents the scale corresponding to the first matching point pair, for example, the scale can be the ratio information when scaling the target image;
  • HR represents the first historical frame image or the second historical frame The amount of rotation of the image relative to the target image, H l represents the translation amount reset after translation.
  • formula (7) can be transformed to obtain formula (8):
  • the rotation angle of the first historical frame image or the second historical frame image relative to the target image can be obtained, so that the rotation angle of the first historical frame image or the second historical frame image can be used.
  • the rotation angle information obtains the transformation parameters between the first historical frame image or the second historical frame image and the target image, and realizes the calculation of transformation parameters by using matching point pairs.
  • Step S42 Using the pose in the third pose and the third reference pose to obtain the offset.
  • the attitude in the third pose is the attitude relative to the preset plane.
  • the third pose may be rotation amount information relative to a preset plane.
  • the ratio between the pose in the third pose and the third reference pose can be used as the offset, that is, the offset can be obtained by calculating the ratio.
  • the distance between the reference plane and the preset plane in the world coordinate system can be obtained. offset between.
  • the feature points obtained by feature extraction through the feature extraction algorithm may be considered to be located on the same plane as the target image.
  • Step S333 Determine the first pose based on the second reference pose, the second pose, and the photometric error between the current image frame and the historical image frame.
  • the relative pose change between the current image frame and other image frames can be obtained first, the photometric error between the current image frame and other image frames can be calculated, and the final result of the current image frame can be obtained by using the relative pose change pose, and then use the second reference pose as a constraint to optimize and reduce the photometric error to determine the first pose.
  • the pose information of the current image frame in the world coordinate system can be obtained as the initial pose, and then the pose information of other image frames in the world coordinate system can be used to obtain the current image frame and other images
  • step S333 can be implemented in the following manner:
  • At least one first candidate pose is obtained, and a first candidate pose is selected as the first pose based on the second reference pose, the second pose, and the first pixel value difference between the current image frame and other image frames.
  • the first candidate pose may be the pose information of the current image frame in the world coordinate system; exemplary, the first candidate pose may be calculated based on an image processing algorithm; exemplary, The first candidate pose can also be obtained by calculating the relative pose changes of the current image frame and other image frames and the second pose of other image frames, and can also directly select the image with pose information closest to the current image frame The pose of the frame is used as the first candidate pose, and then, a plurality of first candidate poses can be generated by using an iterative optimization method.
  • the corresponding to each first candidate pose can be obtained The first pixel value difference, and then select a first candidate pose as the final pose.
  • the first pixel value difference between the current image frame and other image frames may be a pixel value difference between a pixel on the current image frame and a corresponding pixel on the other image frames.
  • the pixel point of A on the current image frame is a1
  • the pixel point of A on other image frames is a2
  • a1 can be the pixel corresponding to point a2 of A on other image frames point.
  • the pose difference between the second reference pose and the pose in the first candidate pose can also be used to optimize the first pixel value difference.
  • formula (10) can be used to select a first candidate pose as the final pose.
  • C1 is the final error information
  • is the first candidate pose is the rotation amount information (also called rotation amount or orientation), is the translation information
  • spatial 3D point Determining spatial points corresponding to feature points in other image frames based on the first candidate pose
  • is a three-dimensional point in space Feature points projected on the current image frame, is a three-dimensional point in space
  • K is the memory matrix of the image acquisition device of the electronic equipment
  • ⁇ , ⁇ are the adjustment parameters of the two constraint items.
  • the ratio can be set by actual use. Indicates that multiple first candidate poses are generated using an iterative optimization method, and the corresponding first candidate pose is selected when the final error information
  • the first candidate pose corresponding to the second feature point whose final error information meets the first preset requirement can be selected as the final pose; exemplary , the first preset requirement can be set as required, which is not limited here.
  • the first pixel value difference and pose difference are calculated by formula (10), and the first candidate pose information corresponding to C1 that meets the preset requirements can be selected as the final pose. Therefore, relatively accurate pose information can be obtained by screening the first candidate poses that meet the preset requirements.
  • At least one first candidate pose is obtained, and based on the second reference pose, and the second pose and the first pixel value difference between the current image frame and other image frames, Selecting a first candidate pose as the first pose can be achieved through steps D1 to D2:
  • Step D1 Using the second pose, determine the spatial point corresponding to the fifth feature point in other image frames.
  • the fifth feature point is the first feature point in the foregoing embodiments.
  • the second poses of other image frames in the world coordinate system may be calculated based on an image registration algorithm, or may be obtained by using a visual tracking algorithm.
  • the depth value of the spatial point corresponding to the fifth feature point can be calculated, and then the three-dimensional coordinates of the spatial point can be calculated, so as to determine the position of the spatial point.
  • the spatial points corresponding to the fifth feature points in a certain number of other image frames can be determined.
  • Step D2 Based on each first candidate pose and space point, determine the sixth feature point corresponding to the first candidate pose from the current image frame, based on the first pixel value difference and the second reference pose and the first candidate pose The difference between the poses, select a first candidate pose as the first pose.
  • the sixth feature point may be the second feature point in the foregoing embodiments.
  • the sixth feature point corresponding to the spatial point can be determined in the current image frame by means of projection; for example, the sixth feature point is the fifth feature point on other image frames in The corresponding point in the current image.
  • the first pixel value difference may be obtained based on the fifth feature point and the sixth feature point, for example, the first pixel value difference may be obtained based on the pixel value of the fifth feature point and the pixel value of the sixth feature point. Then, based on the first pixel value difference and the pose difference between the second reference pose and the first candidate pose, a first candidate pose can be selected as the final pose.
  • the specific calculation method can refer to formula (10).
  • a relatively accurate first candidate pose can be obtained by calculating the difference of pixel values.
  • the first candidate pose is determined based on the initial pose of the current image frame in the world coordinate system. That is to say, based on the initial pose, a series of first candidate poses can be obtained through an iterative optimization method, and then a final pose can be selected from the series of first candidate poses.
  • the initial pose may be determined based on photometric errors between the current image frame and other image frames. That is to say, the photometric error equation can be used in combination with the method of iterative optimization to obtain the initial pose.
  • the initial pose can be obtained in the following manner.
  • At least one second candidate pose is acquired, and based on the second pixel value difference between the current image frame and other image frames, a second candidate pose is selected as the initial pose.
  • the second candidate pose can be the pose information of other image frames relative to the world coordinate system; the second candidate pose can be calculated based on an image processing algorithm, and its number can be several; it can also be directly Selecting the pose of the image frame with pose information closest to the current image frame as a second candidate pose; or directly using the second pose as the second candidate pose.
  • an iterative optimization method may be used to generate multiple second candidate poses.
  • multiple second candidate poses may be generated by using an iterative optimization method based on the second pose. On this basis, based on each second candidate pose and based on the second pixel value difference between the current image frame and other image frames, a second candidate pose can be selected as the initial pose.
  • the second pixel value difference between the current image frame and other image frames may be a pixel value difference between a pixel on the current image frame and a corresponding pixel on the other image frames.
  • the pixel point of B on the current image frame is B1
  • the pixel point of B on other image frames is B2
  • B1 is the pixel point corresponding to point B2 of B on other image frames.
  • formula (12) can be used to select a second candidate pose as the initial change pose between the current image frame and other image frames.
  • C2 is the second pixel value difference; is the second candidate pose, is the rotation amount information; is translation information; three-dimensional point in space Determining a spatial point corresponding to the fifth feature point in other image frames based on the second candidate pose; is a three-dimensional point in space
  • a second candidate pose corresponding to a second feature point whose second pixel value difference satisfies the second preset requirement may also be selected as the initial change pose.
  • the second preset requirement can be set as required, which is not limited here.
  • the second pixel value difference is calculated by formula (12)
  • the second candidate pose information corresponding to C2 that meets the preset requirements is selected as the initial pose. Therefore, relatively accurate pose information can be obtained by screening the second candidate poses that meet the preset requirements.
  • the initial pose that meets the requirements is obtained by calculating the difference of the second pixel value; then based on the initial pose, combined with the detection data (second reference pose) and the photometric error to obtain the final error information, and then can obtain the conforming The final pose required.
  • the correction of the second reference pose By using the correction of the second reference pose, a final pose with higher accuracy can be obtained.
  • Fig. 7 is a schematic flowchart of a third visual positioning method provided by an embodiment of the present disclosure. As shown in Figure 7, the method may include steps S51 to S55:
  • Step S51 Initialization: Perform image registration to obtain the initial image pose.
  • the image registration is to use the current image frame captured by the electronic device and the target image to perform image registration detection. If the detection is successful, the initial image of the current image frame in the world coordinate system established based on the target image can be obtained Pose, that is, the pose of the electronic device in the world coordinate system.
  • the transformation parameters between the current image frame and the target image meet the preset requirements. If the transformation parameters meet the preset requirements, it can be considered that the target image is detected in the current image frame, and the image registration detection is successful. The initial image pose can be obtained; if the transformation parameters do not meet the preset requirements, it can be considered that the target image is not detected in the current image frame, that is, the image registration detection fails.
  • step S51 is repeatedly executed until the initial image pose is obtained.
  • Step S52 Obtain image transformation parameters corresponding to the second image frame and the first image frame by using an image registration method.
  • the corresponding image frame when the initial image pose is obtained may be defined as the first image frame; after the initial image pose is obtained, the electronic device may also obtain the second image frame.
  • the first image frame is an initial pose obtained by using an image registration method
  • the device will execute the image registration method mentioned in the foregoing embodiments, so as to obtain image transformation parameters corresponding to the second image frame and the first image frame.
  • Step S53 judging whether the image transformation parameters meet the preset requirements.
  • the image transformation parameters may include the first transformation parameters; for the method of judging whether the image transformation parameters meet the preset requirements, reference may be made to the foregoing embodiments, and details are not repeated here.
  • step S54 If the image transformation parameters meet the preset requirements, execute step S54; if the image transformation parameters do not meet the preset requirements, execute step S55.
  • Step S54 Enter the image tracking state, and use the image transformation parameters to obtain the second pose.
  • the image transformation parameters meet the preset requirements, it can be confirmed that the electronic device is currently in the image tracking state, and at this time, the image transformation parameters can be used to obtain the second pose corresponding to the second image frame.
  • Step S55 Enter the visual navigation state, and use the second pose of other images in the world coordinate system and the photometric error between the current image frame and other image frames to determine the first pose.
  • the electronic device enters the visual navigation state, and executes "use the second pose of other images in the world coordinate system, and the current image frame and other image frames The photometric error between, determine the first pose" step.
  • the electronic device enters the visual navigation state, and executes "use the second pose of other images in the world coordinate system, and the current image frame and other image frames The photometric error between, determine the first pose" step.
  • this step please refer to the foregoing embodiments, and details will not be repeated here.
  • step S54 if step S54 is executed, step S52 will be re-executed to obtain the image transformation parameters of the third image frame and the second image frame, and the subsequent steps will continue to be executed. If step S55 is executed, step S55 will be executed repeatedly, and the device will continue to be in the visual navigation state.
  • step S51 when the electronic device re-executes step S51, it may continue to perform subsequent steps.
  • Fig. 8 is a schematic structural diagram of an image registration device provided by an embodiment of the present disclosure.
  • the image registration device 80 includes an image acquisition module 81 , a first parameter acquisition module 82 and a second parameter acquisition module 83 . in:
  • An image acquisition module 81 configured to acquire the current image frame
  • the first parameter acquisition module 82 is configured to determine a first transformation parameter between the current image frame and other image frames based on target image information in the current image frame and other image frames, wherein the target image information is an image about the target image Information: the first transformation parameter, including the homography matrix between the current image frame and other image frames.
  • the second parameter acquisition module 83 is configured to obtain a third transformation parameter between the current image frame and the target image based on the first transformation parameter and the second transformation parameters between other image frames and the target image; wherein, the second transformation parameters, including the homography matrix between the current image frame and the target image.
  • the first parameter acquisition module 82 is configured to find out at least one first feature point about the target image from other image frames; after finding out at least one second feature point about the target image from the current image frame A feature point; based on the first feature point and the second feature point, determine a first transformation parameter.
  • the first parameter acquisition module 82 is configured to determine the target area of the target image in other image frames based on the second transformation parameter; extract at least one first feature point from the target area;
  • the first parameter acquiring module 82 is further configured to respectively track at least one first feature point to obtain at least one second feature point related to the target image in the current image frame.
  • the second parameter obtaining module 83 is configured to use the product of the first transformation parameter and the second transformation parameter as the third transformation parameter.
  • the image registration device 80 also includes an optimization module.
  • the second parameter acquisition module is configured to obtain the current image frame and the target image based on the first transformation parameter and the second transformation parameter between other image frames and the target image.
  • the optimization module is configured to optimize the third transformation parameter by using a preset optimization method.
  • FIG. 9 is a schematic structural diagram of a visual positioning device 90 provided by the implementation of the present disclosure.
  • the visual positioning device 90 includes a parameter acquisition module 91 and a first pose acquisition module 92, wherein:
  • the parameter acquisition module 91 is configured to acquire the current transformation parameter between the current image frame and the target image; wherein, the current transformation parameter is the third transformation parameter obtained by the image registration method provided in the foregoing embodiment;
  • the first pose obtaining module 92 is configured to use the current transformation parameters to obtain the first pose of the current image frame in the world coordinate system, wherein the world coordinate system is established based on the plane where the target image is located.
  • the visual positioning device 90 further includes a judgment module and a second pose acquisition module, wherein:
  • the first pose acquisition module is configured to use the current transformation parameters to obtain the first pose of the current image frame in the world coordinate system, and the judging module is configured to judge whether the current transformation parameters meet the preset requirements.
  • the first pose acquisition module is further configured to obtain the first pose of the current image frame in the world coordinate system by using the current transformation parameters in response to the current transformation parameters meeting the preset requirements.
  • the second pose acquisition module is configured to determine the second pose by using the second pose of other images in the world coordinate system and the photometric error between the current image frame and other image frames in response to the fact that the current transformation parameters do not meet the preset requirements. a pose.
  • the visual positioning device 90 further includes a state determination module, the state determination module is configured to respond to the previous image before the parameter acquisition module is configured to acquire the current transformation parameters between the current image frame and the target image
  • the pose acquisition method of the frame is the image tracking method, which executes to obtain the current transformation parameters between the current image frame and the target image, where the image tracking method is to use the transformation parameters between the previous image frame and the target image to determine the previous image frame pose in the world coordinate system.
  • the above-mentioned other image frames and the current image frame are sequentially captured by the image acquisition device of the device.
  • the second pose acquisition module is configured to obtain the first reference pose, and adjust the first reference pose to obtain the second reference pose; based on the second reference pose, the second pose, and the relationship between the current image frame and the historical image frame
  • the photometric error determines the first pose; wherein, the first reference pose is the pose of the image acquisition device corresponding to the shooting moment of the current image frame and relative to the reference plane; using the reference plane and the preset plane in the world coordinate system. Offset,.
  • the visual positioning device 90 further includes an offset acquisition module configured to adjust the first reference pose by using the offset between the reference plane and the preset plane in the world coordinate system to obtain Before the second reference pose, obtain the third pose of the first historical image frame in the world coordinate system, and obtain the third reference pose, and use the pose in the third pose and the third reference pose to obtain the offset, where , the third reference pose is the pose of the image acquisition device corresponding to the shooting moment of the first historical image frame and relative to the reference plane, the third pose is determined based on the target image, and the preset plane is the plane where the target image is located; the third A pose in a pose is a pose relative to a preset plane.
  • the second pose acquisition module is configured to acquire at least one first candidate pose based on the second reference pose, the second pose and the first pixel value between the current image frame and other image frames difference, select a first candidate pose as the first pose.
  • the above-mentioned first candidate pose is determined based on the initial pose of the current image frame in the world coordinate system, and the initial pose is determined based on the photometric error between the current image frame and other image frames; and/or,
  • the above-mentioned second pose acquisition module is configured to use the second pose to determine the spatial points corresponding to the first feature points in other image frames; determine from the current image frame based on each first candidate pose and the spatial point The second feature point corresponding to the first candidate pose, obtain the first pixel value difference between the first feature point and the second feature point, and based on the first pixel value difference and the difference between the second reference pose and the first candidate pose The pose difference of , select a first candidate pose as the first pose.
  • the visual positioning device 90 further includes a historical image frame pose acquisition module configured to determine the first historical image frame and the target image based on the first matching point pair between the first historical image frame and the target image Between the fourth transformation parameter, use the fourth transformation parameter to obtain the second pose; or, based on the second matching point pair between the first historical image frame and the second historical image frame, determine the first historical image frame and the second historical image frame
  • the second historical image frame is located before the first historical image frame.
  • the historical image frame pose acquisition module is configured to determine whether the fourth transformation parameter meets the preset requirement before using the fourth transformation parameter to obtain the second pose; in response to the fourth transformation parameter meeting the preset requirement , a historical image frame pose acquisition module, configured to determine that it is in an image tracking state, and execute using a fourth transformation parameter to obtain a second pose.
  • Fig. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • the electronic device 100 includes a memory 101 and a processor 102 coupled to each other, and the processor 102 is configured to execute program instructions stored in the memory 101, so as to realize the steps of any of the above-mentioned image registration method embodiments, or any of the above-mentioned image registration methods. Steps of a method embodiment.
  • the electronic device 100 may include, but not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include mobile devices such as notebook computers and tablet computers, which are not limited herein.
  • the processor 102 is configured to control itself and the memory 101 to implement the steps of any of the above embodiments of the image registration method, or the steps of any of the above embodiments of the visual positioning method.
  • the processor 102 may also be a CPU, a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the processor 102 may be jointly implemented by integrated circuit chips.
  • Fig. 11 is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present disclosure.
  • the computer-readable storage medium 111 stores program instructions 1111 that can be executed by the processor, and the program instructions 1111 are used to implement the steps of any of the above-mentioned image registration method embodiments, or any of the above-mentioned visual positioning methods Example steps.
  • An embodiment of the present disclosure also provides a computer program, where the computer program includes computer readable code.
  • the computer readable code runs in an electronic device
  • the processor of the electronic device is configured to implement the computer program described in the foregoing embodiments.
  • the image registration method and the visual positioning method described in the foregoing embodiments are implemented.
  • the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the method embodiments above, and its specific implementation can refer to the description of the method embodiments above. For brevity, here No longer.
  • the disclosed methods and devices may be implemented in other ways.
  • the device implementations described above are only illustrative.
  • the division of modules or units is only a logical function division. In actual implementation, there may be other division methods.
  • units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the electronic device involved in the embodiments of the present disclosure may be at least one of a system, a method, and a computer program product.
  • a computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present disclosure.
  • a computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device.
  • a computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Examples of computer-readable storage media include: portable computer disks, hard disks, Random Access Memory (RAM), Read-Only Memory (ROM), erasable Electrical Programmable Read Only Memory (EPROM) or flash memory, Static Random-Access Memory (Static Random-Access Memory, SRAM), Portable Compact Disc Read-Only Memory (CD-ROM), Digital Video Discs (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or raised structures in grooves with instructions stored thereon, and any suitable combination of the foregoing.
  • RAM Random Access Memory
  • ROM Read-Only Memory
  • EPROM erasable Electrical Programmable Read Only Memory
  • flash memory Static Random-Access Memory
  • SRAM Static Random-Access Memory
  • CD-ROM Portable Compact Disc Read-Only Memory
  • DVDs Digital Video Discs
  • memory sticks floppy disks, mechanically encoded devices such as punched cards or raised structures in grooves with instructions stored thereon, and any suitable combination of the foregoing.
  • computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.
  • the computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device over at least one of a network, such as the Internet, a local area network, a wide area network, and a wireless network.
  • the network may include at least one of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.
  • a network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • Computer program instructions for performing the operations of the present disclosure may be assembly instructions, Industry Standard Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the “C” language or similar programming languages.
  • Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it may be connected to an external computer (for example, using Internet Service Provider to connect via the Internet).
  • LAN Local Area Network
  • WAN Wide Area Network
  • electronic circuits such as programmable logic circuits, FPGAs, or programmable logic arrays (Programmable Logic Arrays, PLAs), can be customized by using state information of computer-readable program instructions, which can execute computer-readable Read program instructions, thereby implementing various aspects of the present disclosure.
  • a unit described as a separate component may or may not be physically separated, and a component shown as a unit may or may not be a physical unit, that is, it may be located in one place, or may also be distributed to network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the technical solution of the present disclosure is essentially or part of the contribution to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods in various embodiments of the present disclosure.
  • the aforementioned storage medium includes: various media capable of storing program codes such as U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk.
  • Embodiments of the present disclosure provide an image registration method, a visual positioning method, a device, a device, a medium, and a program, wherein the image registration method is performed by an electronic device, and the method includes: acquiring a current image frame; The target image information in the image frame and other image frames determines the first transformation parameters between the current image frame and other image frames; wherein, the target image information is image information about the target image; the first transformation parameters include the The homography matrix between the current image frame and the other image frames; based on the first transformation parameter and the second transformation parameter between other image frames and the target image, the second transformation parameter between the current image frame and the target image is obtained Three transformation parameters: the second transformation parameter includes a homography matrix between the current image frame and the target image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Analysis (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)

Abstract

Embodiments of the present disclosure provide an image registration method, a visual positioning method, an apparatus, a device, a medium, and a program, the image registration method being executed by an electronic device and comprising: acquiring a current image frame; determining a first transformation parameter between the current image frame and other image frames on the basis of target image information in the current image frame and other image frames, wherein the target image information is image information about a target image, and the first transformation parameter comprises a homography matrix between the current image frame and the other image frames; and obtaining a third transformation parameter between the current image frame and the target image on the basis of the first transformation parameter and a second transformation parameter between the other image frames and the target image, the second transformation parameter comprising a homography matrix between the current image frame and the target image. By means of said method, registration speed and the accuracy of image registration can be increased.

Description

图像配准方法、视觉定位方法、装置、设备、介质及程序Image registration method, visual positioning method, device, equipment, medium and program
相关申请的交叉引用Cross References to Related Applications
本专利申请要求2021年6月25日提交的中国专利申请号为202110713164.9、申请人为浙江商汤科技开发有限公司，申请名称为“图像配准方法、视觉定位方法及相关装置、设备”的优先权，该申请的全文以引用的方式并入本公开中。This patent application claims the priority of the Chinese patent application number 202110713164.9 submitted on June 25, 2021, the applicant is Zhejiang Shangtang Technology Development Co., Ltd., and the application name is "image registration method, visual positioning method and related devices and equipment" , which application is incorporated by reference in its entirety into this disclosure.
技术领域technical field
本公开涉及人工智能技术领域，尤其涉及一种图像配准方法、视觉定位方法、装置、设备、介质及程序。The present disclosure relates to the technical field of artificial intelligence, and in particular to an image registration method, a visual positioning method, a device, a device, a medium and a program.
背景技术Background technique
随着电子信息技术的发展，增强现实(Augmented Reality，AR)、虚拟现实(Virtual Reality，VR)等成为计算机视觉领域中的应用热点，这些技术以相机作为输入设备，借助于图像算法处理，通过电子设备的计算辅助功能，就可以数字化周围环境，从而使用户获得与真实环境交互的使用体验。图像配准是AR、VR等计算机视觉领域中的研究重点，通过图像配准技术可以获取相机拍摄到的当前图像与目标图像之间的变换参数，从而可以通过变换参数，确定目标图像在当前图像中的位置。With the development of electronic information technology, augmented reality (Augmented Reality, AR) and virtual reality (Virtual Reality, VR) have become application hotspots in the field of computer vision. The computing assistance function of electronic equipment can digitize the surrounding environment, so that users can obtain the experience of interacting with the real environment. Image registration is the focus of research in the field of computer vision such as AR and VR. Through image registration technology, the transformation parameters between the current image captured by the camera and the target image can be obtained, so that the target image can be determined in the current image through the transformation parameters. position in .
然而，在实际的图像配准时，通常是将每一帧图像都与目标图像进行配准，以此获得变换参数，这样的配准方法，运算量大且准确度不高。However, in actual image registration, each frame of image is usually registered with the target image to obtain transformation parameters. Such a registration method has a large amount of computation and is not very accurate.
因此，如何提高图像的配准准确度已成为亟待解决的问题。Therefore, how to improve the registration accuracy of images has become an urgent problem to be solved.
发明内容Contents of the invention
本公开实施例提供了一种图像配准方法、视觉定位方法、装置、设备、介质及程序。Embodiments of the present disclosure provide an image registration method, a visual positioning method, a device, a device, a medium, and a program.
本公开实施例提供了一种图像配准方法，所述方法由电子设备执行，所述方法包括：An embodiment of the present disclosure provides an image registration method, the method is executed by an electronic device, and the method includes:
获取当前图像帧；基于当前图像帧与其它图像帧中的目标图像信息，确定当前图像帧与其它图像帧之间的第一变换参数；其中，目标图像信息为关于目标图像的图像信息；所述第一变换参数，包括所述当前图像帧与所述其它图像帧之间的单应性矩阵；Acquiring the current image frame; based on the target image information in the current image frame and other image frames, determining a first transformation parameter between the current image frame and other image frames; wherein, the target image information is image information about the target image; the The first transformation parameter includes a homography matrix between the current image frame and the other image frames;
基于第一变换参数、以及其它图像帧与目标图像之间的第二变换参数，得到当前图像帧与目标图像之间的第三变换参数；其中，所述第二变换参数，包括所述当前图像帧与所述目标图像之间的单应性矩阵。Based on the first transformation parameter and the second transformation parameter between other image frames and the target image, a third transformation parameter between the current image frame and the target image is obtained; wherein, the second transformation parameter includes the current image The homography matrix between the frame and the target image.
如此，通过获得当前图像帧与其它图像帧之间的第一变换参数，再结合第一变换参数和其它图像帧与目标图像之间的第二变换参数，就可以直接利用当前图像帧与其它图像帧中的目标图像信息，实现了当前图像帧与目标图像的配准，相比于利用当前图像帧与每一其它图像帧中所有图像信息进行图像配准，该方法计算量较小，能够提高配准速度以及图像配准的准确度。In this way, by obtaining the first transformation parameters between the current image frame and other image frames, and then combining the first transformation parameters with the second transformation parameters between other image frames and the target image, the current image frame and other image frames can be directly used The target image information in the frame realizes the registration of the current image frame and the target image. Compared with using the current image frame and all the image information in each other image frame for image registration, this method has a small amount of calculation and can improve Registration speed and accuracy of image registration.
本公开实施例还提供了一种视觉定位方法，所述方法由电子设备执行，所述方法包括：获取当前图像帧与目标图像之间的当前变换参数，其中，当前变换参数为通过如前任一所述的图像配准方法获得的第三变换参数；利用当前变换参数，得到当前图像帧在世界坐标系中的第一位姿，其中，世界坐标系是基于目标图像所在的平面建立的。An embodiment of the present disclosure also provides a visual positioning method, the method is executed by an electronic device, and the method includes: acquiring the current transformation parameter between the current image frame and the target image, wherein the current transformation parameter is obtained by any of the previous The third transformation parameter obtained by the image registration method; using the current transformation parameter, the first pose of the current image frame in the world coordinate system is obtained, wherein the world coordinate system is established based on the plane where the target image is located.
由此，通过获取当前图像帧与目标图像之间的当前变换参数，并且基于目标图像所在的平面建立世界坐标系，可以得到当前图像帧在世界坐标系中的第一位姿，实现了对拍摄设备的视觉定位。Thus, by obtaining the current transformation parameters between the current image frame and the target image, and establishing the world coordinate system based on the plane where the target image is located, the first pose of the current image frame in the world coordinate system can be obtained, and the shooting Visual positioning of the device.
本公开实施例还提供了一种图像配准装置，包括：图像配准装置、第一参数获取模块和第二参数获取模块，其中：An embodiment of the present disclosure also provides an image registration device, including: an image registration device, a first parameter acquisition module, and a second parameter acquisition module, wherein:
所述图像获取模块，配置为获取当前图像帧；The image acquisition module is configured to acquire the current image frame;
所述第一参数获取模块，配置为基于当前图像帧与其它图像帧中的目标图像信息，确定当前图像帧与其它图像帧之间的第一变换参数；其中，目标图像信息为关于目标图像的图像信息；所述第一变换参数，包括所述当前图像帧与所述其它图像帧之间的单应性矩阵；The first parameter acquisition module is configured to determine a first transformation parameter between the current image frame and other image frames based on target image information in the current image frame and other image frames; wherein the target image information is information about the target image Image information; the first transformation parameter includes a homography matrix between the current image frame and the other image frames;
所述第二参数获取模块，配置为基于第一变换参数、以及其它图像帧与目标图像之间的第二变换 参数，得到当前图像帧与目标图像之间的第三变换参数；所述第二变换参数，包括所述当前图像帧与所述目标图像之间的单应性矩阵。The second parameter acquisition module is configured to obtain a third transformation parameter between the current image frame and the target image based on the first transformation parameter and second transformation parameters between other image frames and the target image; the second The transformation parameters include a homography matrix between the current image frame and the target image.
本公开实施例还提供了一种视觉定位装置，包括：参数获取模块和第一位姿获取模块，其中：An embodiment of the present disclosure also provides a visual positioning device, including: a parameter acquisition module and a first pose acquisition module, wherein:
所述参数获取模块，配置为获取当前图像帧与目标图像之间的当前变换参数，其中，当前变换参数为如前任一图像配准方法获得的第三变换参数；The parameter acquisition module is configured to acquire a current transformation parameter between the current image frame and the target image, wherein the current transformation parameter is the third transformation parameter obtained by any previous image registration method;
所述第一位姿获取模块，配置为利用当前变换参数，得到当前图像帧在世界坐标系中的第一位姿，其中，世界坐标系是基于目标图像所在的平面建立的。The first pose acquisition module is configured to use the current transformation parameters to obtain the first pose of the current image frame in the world coordinate system, wherein the world coordinate system is established based on the plane where the target image is located.
本公开实施例还提供了一种电子设备，包括相互耦接的存储器和处理器，处理器用于执行存储器中存储的程序指令，以实现上述第一方面中的图像配准方法和第二方面的视觉定位方法。An embodiment of the present disclosure also provides an electronic device, including a memory and a processor coupled to each other, and the processor is used to execute the program instructions stored in the memory, so as to realize the image registration method in the first aspect and the image registration method in the second aspect. Visual positioning method.
本公开实施例还提供了一种计算机可读存储介质，其上存储有程序指令，程序指令被处理器执行时实现如前所述的图像配准方法和如前所述的视觉定位方法。An embodiment of the present disclosure also provides a computer-readable storage medium, on which program instructions are stored. When the program instructions are executed by a processor, the aforementioned image registration method and the aforementioned visual positioning method are implemented.
本公开实施例还提供了一种计算机程序，所述计算机程序包括计算机可读代码，在所述计算机可读代码在电子设备中运行的情况下，所述电子设备的处理器执行用于实现如前任一项所述的图像配准方法、以及如前任一项所述的视觉定位方法。An embodiment of the present disclosure also provides a computer program, where the computer program includes computer readable codes, and when the computer readable codes run in an electronic device, the processor of the electronic device executes the following steps: The image registration method described in any one of the preceding items, and the visual positioning method described in any one of the preceding items.
由以上可知，本公开实施例提供的图像配准方法，通过获得当前图像帧与其它图像帧之间的第一变换参数，再结合第一变换参数和其它图像帧与目标图像之间的第二变换参数，可以直接利用当前图像帧与其它图像帧中的目标图像信息，实现了当前图像帧与目标图像的配准，相比于利用当前图像帧与其它图像帧中所有图像信息进行图像配准，该方法计算量较小，还能提高配准速度以及图像配准的准确度。As can be seen from the above, the image registration method provided by the embodiments of the present disclosure obtains the first transformation parameter between the current image frame and other image frames, and then combines the first transformation parameter with the second transformation parameter between other image frames and the target image. The transformation parameters can directly use the target image information in the current image frame and other image frames, and realize the registration of the current image frame and the target image, compared to using all the image information in the current image frame and other image frames for image registration , the calculation amount of this method is small, and it can also improve the registration speed and the accuracy of image registration.
为使本公开的上述目的、特征和优点能更明显易懂，下文特举较佳实施例，并配合所附附图，作详细说明如下。In order to make the above-mentioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.
附图说明Description of drawings
为了更清楚地说明本公开实施例的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，此处的附图被并入说明书中并构成本说明书中的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开实施例的技术方案。应当理解，以下附图仅示出了本公开的某些实施例，因此不应被看作是对范围的限定，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他相关的附图。In order to illustrate the technical solutions of the embodiments of the present disclosure more clearly, the following will briefly introduce the accompanying drawings used in the embodiments. The accompanying drawings here are incorporated into the specification and constitute a part of the specification. The drawings show embodiments consistent with the present disclosure, and are used together with the specification to illustrate the technical solutions of the embodiments of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. For those skilled in the art, they can also make From these drawings other related drawings are obtained.
图1是本公开实施例提供的图像配准方法的流程示意图；FIG. 1 is a schematic flowchart of an image registration method provided by an embodiment of the present disclosure;
图2是本公开实施例提供的图像配准方法中确定第一变换参数的流程示意图；Fig. 2 is a schematic flow chart of determining the first transformation parameter in the image registration method provided by the embodiment of the present disclosure;
图3是本公开实施例提供的视觉定位方法的第一流程示意图；Fig. 3 is a schematic flowchart of a first visual positioning method provided by an embodiment of the present disclosure;
图4是本公开实施例提供的视觉定位方法中第一位姿的确定流程示意图；Fig. 4 is a schematic flow diagram of determining the first pose in the visual positioning method provided by an embodiment of the present disclosure;
图5是本公开实施例提供的视觉定位方法的第二流程示意图；Fig. 5 is a second schematic flowchart of a visual positioning method provided by an embodiment of the present disclosure;
图6是本公开实施例提供的视觉定位方法中偏移量获取的流程示意图；FIG. 6 is a schematic flowchart of offset acquisition in a visual positioning method provided by an embodiment of the present disclosure;
图7是本公开实施例提供的视觉定位方法的第三流程示意图；FIG. 7 is a schematic flowchart of a third visual positioning method provided by an embodiment of the present disclosure;
图8是本公开实施例提供的图像配准装置的结构示意图；Fig. 8 is a schematic structural diagram of an image registration device provided by an embodiment of the present disclosure;
图9是本公开实施例提供的视觉定位装置的结构示意图；FIG. 9 is a schematic structural diagram of a visual positioning device provided by an embodiment of the present disclosure;
图10是本公开实施例提供的电子设备的结构示意图；FIG. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;
图11为本公开实施例提供的计算机可读存储介质的结构示意图。Fig. 11 is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present disclosure.
具体实施方式detailed description
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合本公开实施例中附图，对本公开实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本公开一部分实施例，而不是全部的实施例。通常在此处附图中描述和示出的本公开实施例的组件可以以各种不同的配置来布置和设计。因此，以下对在附图中提供的本公开的实施例的详细描述并非旨在限制要求保护的本公开的范围，而是仅仅表示本公开的选定实施例。基于本公开的实施例，本领域技术人员在没有做出创 造性劳动的前提下所获得的所有其它实施例，都属于本公开保护的范围。In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only It is a part of the embodiments of the present disclosure, but not all of them. The components of the disclosed embodiments generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present disclosure.
下面结合说明书附图，对本公开实施例的方案进行详细说明。The solutions of the embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings.
以下描述中，为了说明而不是为了限定，提出了诸如特定***结构、接口、技术之类的具体细节，以便透彻理解本公开。In the following description, for purposes of illustration rather than limitation, specific details, such as specific system architectures, interfaces, techniques, are set forth in order to provide a thorough understanding of the present disclosure.
本文中术语“***”和“网络”在本文中常被可互换使用。本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。此外，本文中的“多”表示两个或者多于两个。The terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship. In addition, "many" herein means two or more than two.
图1是本公开实施例提供的图像配准方法的流程示意图。Fig. 1 is a schematic flowchart of an image registration method provided by an embodiment of the present disclosure.
需要说明的是，本公开实施例提供的图像配准方法，由电子设备执行；其中，图像配准方法，可以通过电子设备的处理器执行。It should be noted that the image registration method provided by the embodiments of the present disclosure is executed by an electronic device; wherein, the image registration method may be executed by a processor of the electronic device.
示例性的，电子设备的处理器，可以为特定用途集成电路(Application Specific Integrated Circuit，ASIC)、数字信号处理器(Digital Signal Processor，DSP)、数字信号处理装置(Digital Signal Processing Device，DSPD)、可编程逻辑装置(Programmable Logic Device，PLD)、现场可编程逻辑门阵列(Field Programmable Gate Array，FPGA)、中央处理器(Central Processing Unit，CPU)、控制器、微控制器、微处理器中的至少一种。Exemplarily, the processor of an electronic device may be an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), Programmable Logic Device (Programmable Logic Device, PLD), Field Programmable Logic Gate Array (Field Programmable Gate Array, FPGA), Central Processing Unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor in at least one.
如图1所示，该流程可以包括S11步骤至步骤S13：As shown in Figure 1, the process may include step S11 to step S13:
步骤S11：获取当前图像帧。Step S11: Obtain the current image frame.
在一种实施方式中，当前图像帧可以是电子设备的图像采集装置拍摄到的图像。示例性的，在AR、VR等应用场景中，当前图像帧可以是诸如手机、平板电脑、智能眼镜等电子设备所拍摄到的图像；或者，在视频监控场景中，当前图像帧可以是监控相机所拍摄到的图像，在此不做限定。其它场景可以以此类推，在此不再一一举例。In an implementation manner, the current image frame may be an image captured by an image acquisition device of the electronic device. Exemplarily, in application scenarios such as AR and VR, the current image frame may be an image captured by an electronic device such as a mobile phone, a tablet computer, or smart glasses; or, in a video surveillance scenario, the current image frame may be a surveillance camera The captured images are not limited here. Other scenarios can be deduced in the same way, and examples are not given here.
在一种实施方式中，当前图像帧中可以包括目标图像；示例性的，若当前图像帧中包括目标图像，可以实现当前图像帧与目标图像的配准。In an implementation manner, the current image frame may include the target image; for example, if the current image frame includes the target image, registration between the current image frame and the target image may be implemented.
步骤S12：基于当前图像帧与其它图像帧中的目标图像信息，确定当前图像帧与其它图像帧之间的第一变换参数。Step S12: Based on the target image information in the current image frame and other image frames, determine a first transformation parameter between the current image frame and other image frames.
其中，第一变换参数，包括当前图像帧与其它图像帧之间的单应性矩阵。Wherein, the first transformation parameter includes a homography matrix between the current image frame and other image frames.
在一种实施方式中，目标图像可以是平面上的图像，示例性的，平面可以是平整的地面，或者平整的墙面；示例性的，目标图像可以是预先获取的，也即目标图像可以在执行本公开实施例提供的图像配准方法之前预先确定。In one embodiment, the target image may be an image on a plane. Exemplarily, the plane may be a flat ground or a flat wall; Exemplarily, the target image may be pre-acquired, that is, the target image may be It is predetermined before executing the image registration method provided by the embodiment of the present disclosure.
在一种实施方式中，目标图像可以根据实际应用情况进行设置。示例性的，在需要确定当前图像帧中建筑物A的位置的情况下，可以预先获取建筑物A的图像；或者，在需要确定当前图像帧中人物B的位置的情况下，可以预先获取人物B的图像，其它情况可以以此类推，在此不再一一举例。In an implementation manner, the target image may be set according to actual application conditions. Exemplarily, when the position of building A in the current image frame needs to be determined, the image of building A can be acquired in advance; or, when the position of person B in the current image frame needs to be determined, the person B can be acquired in advance For the image of B, other situations can be deduced in the same way, so we will not give examples one by one here.
在一种实施方式中，可以从已经获取的图像中，确定目标图像。示例性的，可以预先对建筑物的内部情况进行拍照，以得到一定数量的建筑物内部图像，然后在这些图像中，选择包含特定对象的图像区域作为目标图像，如某一图像中包括一幅画，则可以将这幅画作为目标图像。In an implementation manner, the target image can be determined from the acquired images. Exemplarily, the interior of the building can be photographed in advance to obtain a certain number of images of the interior of the building, and then in these images, an image area containing a specific object is selected as the target image, for example, a certain image includes a painting, the painting can be used as the target image.
在一种实施方式中，目标图像信息可以为关于目标图像的图像信息。示例性的，目标图像信息，可以为对当前图像帧与其它图像帧中的目标图像进行特征提取，得到的特征点以及相应的特征表示。特征提取算法可以是特征点检测(Features From Accelerated Segment Test，FAST)算法，尺度不变特征转换(Scale-invariant Feature Transform，SIFT)算法，快速特征点提取(Oriented FAST and Rotated BRIEF，ORB)算法等等。示例性的，在特征提取算法为ORB算法的情况下，在利用ORB算法提取特征点时，通过ORB算法得到的描述子可以作为特征表示。在本实施例中，基于图像帧进行特征提取的特征点，可以认为与目标图像处于同一个平面。In one embodiment, the target image information may be image information about the target image. Exemplarily, the target image information may be feature points and corresponding feature representations obtained by performing feature extraction on target images in the current image frame and other image frames. The feature extraction algorithm can be a feature point detection (Features From Accelerated Segment Test, FAST) algorithm, a scale-invariant feature transformation (Scale-invariant Feature Transform, SIFT) algorithm, a fast feature point extraction (Oriented FAST and Rotated BRIEF, ORB) algorithm, etc. Wait. Exemplarily, when the feature extraction algorithm is the ORB algorithm, when feature points are extracted using the ORB algorithm, the descriptor obtained by the ORB algorithm may be used as a feature representation. In this embodiment, the feature points extracted based on the image frame may be considered to be on the same plane as the target image.
以上实施例提及的通过特征提取算法进行特征提取得到的特征点，都可以认为与目标图像处于同一平面。The feature points obtained by feature extraction through the feature extraction algorithm mentioned in the above embodiments can be considered to be on the same plane as the target image.
在一种实施方式中，在当前图像帧和其它帧图像中，均可以包括目标图像，用于实现本公开实施例提供的图像配准方法。其它帧图像与当前图像帧类似，即它们可以是拍摄得到，或是视频监控机拍摄得到的。In an implementation manner, both the current image frame and other frame images may include target images, so as to implement the image registration method provided by the embodiments of the present disclosure. The other frame images are similar to the current image frame, that is, they can be captured or captured by a video surveillance machine.
在一种实施方式中，其它图像帧和当前图像帧是由电子设备的图像采集装置先后拍摄得到。也就是说，其它图像帧的拍摄时间早于当前图像帧的拍摄时间。In an implementation manner, other image frames and the current image frame are sequentially captured by the image acquisition device of the electronic device. That is to say, the shooting time of other image frames is earlier than the shooting time of the current image frame.
在一种实施方式中，可以已知目标图像在其它图像帧的位置，然后利用特征点跟踪的方法，例如通过光流算法，跟踪其它图像帧中目标图像上的点在当前图像帧的位置，以此确定目标图像在当前图像帧的位置，进而可以获得当前图像帧与其它图像帧中的目标图像信息。In one embodiment, the position of the target image in other image frames can be known, and then the feature point tracking method, such as optical flow algorithm, can be used to track the position of the point on the target image in other image frames in the current image frame, In this way, the position of the target image in the current image frame is determined, and then target image information in the current image frame and other image frames can be obtained.
在一种实施方式中，在得到当前图像帧与其它图像帧中的目标图像信息后，可以基于通用的图像配准方法，对当前图像帧与其它图像帧进行配准；示例性的，图像配准算法可以是基于灰度和模板的算法，或者是基于特征的匹配方法。示例性的，通过基于特征的匹配方法，可以获得一定数量的关于当前图像帧与目标图像的匹配点对，就可以利用随机一致性采样算法(Random Sample Consensus，RANSAC)计算当前图像帧与目标图像之间的变换参数，以此实现两图像的配准。In one embodiment, after obtaining the target image information in the current image frame and other image frames, the current image frame can be registered with other image frames based on a general image registration method; exemplary, image registration Quasi-algorithms can be based on grayscale and templates, or feature-based matching methods. Exemplary, through feature-based matching method, can obtain a certain number of matching point pairs about current image frame and target image, just can utilize random consistent sampling algorithm (Random Sample Consensus, RANSAC) to calculate current image frame and target image Between the transformation parameters, in order to achieve the registration of the two images.
步骤S13：基于第一变换参数、以及其它图像帧与目标图像之间的第二变换参数，得到当前图像帧与目标图像之间的第三变换参数。Step S13: Obtain a third transformation parameter between the current image frame and the target image based on the first transformation parameter and the second transformation parameter between other image frames and the target image.
其中，第二变换参数，包括其它图像帧与目标图像之间的单应性矩阵。Wherein, the second transformation parameter includes a homography matrix between other image frames and the target image.
在一种实施方式中，在其它图像帧中包括目标图像的情况下，可以利用其它图像帧与目标图像进行配准。示例性的，其它图像帧与目标图像之间的第二变换参数，可以是利用通用的图像配准算法得到的。In an implementation manner, when other image frames include the target image, other image frames may be used to perform registration with the target image. Exemplarily, the second transformation parameters between other image frames and the target image may be obtained by using a common image registration algorithm.
在一些实施例中，第二变换参数，可以是通过模板匹配的方法获取，或者是基于特征的匹配方法获取等。示例性的，在基于特征的匹配方法中，可以首先获取其它图像帧上的特征点与目标图像上的特征点，然后进行匹配计算，最后获得第二变换参数。示例性的，若其它图像帧A的之前图像帧B(如前一图像帧)已确定之前变换帧B与目标图像之间的变换参数，则可以采用类似本公开实施例提供的方法得到第二变换参数，即利用其它图像帧A与其之前图像帧B之间的变换参数、以及之前图像帧B与目标图像之间的变换参数，得到其它图像帧A与目标图像之间的变换参数，即上述第二变换参数。In some embodiments, the second transformation parameter may be acquired through a template matching method, or through a feature-based matching method, and the like. Exemplarily, in the feature-based matching method, feature points on other image frames and feature points on the target image may be obtained first, then matching calculations are performed, and second transformation parameters are finally obtained. Exemplarily, if the previous image frame B (such as the previous image frame) of other image frame A has determined the transformation parameters between the previous transformation frame B and the target image, the second Transformation parameters, that is, using the transformation parameters between other image frames A and the previous image frame B, and the transformation parameters between the previous image frame B and the target image, to obtain the transformation parameters between other image frames A and the target image, that is, the above Second transformation parameter.
在一种实施方式中，在得到第一变换参数和第二变换参数以后，可以根据当前图像帧与其它图像帧之间的第一变换参数，以及其它图像帧与目标图像之间的第二变换参数，将其它图像帧作为联结点，建立当前图像帧与目标图像的配准关系，即基于第一变换参数以及第二变换参数，可以得到第三变换参数。In one embodiment, after the first transformation parameter and the second transformation parameter are obtained, the first transformation parameter between the current image frame and other image frames, and the second transformation between other image frames and the target image can be parameter, using other image frames as connection points to establish a registration relationship between the current image frame and the target image, that is, based on the first transformation parameter and the second transformation parameter, the third transformation parameter can be obtained.
在本公开实施例提供的图像配准方法中，可以将第一变换参数与第二变换参数的乘积，作为第三变换参数，如式(1)所示：In the image registration method provided by the embodiment of the present disclosure, the product of the first transformation parameter and the second transformation parameter can be used as the third transformation parameter, as shown in formula (1):
Figure PCTCN2021121049-appb-000001
Figure PCTCN2021121049-appb-000001
其中，
Figure PCTCN2021121049-appb-000002
为当前图像帧与目标图像之间的第三变换参数；
Figure PCTCN2021121049-appb-000003
为其它图像帧与目标图像之间的第二变换参数；
Figure PCTCN2021121049-appb-000004
为当前图像帧与其它图像帧之间的第一变换参数。 in,
Figure PCTCN2021121049-appb-000002
is the third transformation parameter between the current image frame and the target image;
Figure PCTCN2021121049-appb-000003
is the second transformation parameter between other image frames and the target image;
Figure PCTCN2021121049-appb-000004
is the first transformation parameter between the current image frame and other image frames.
由此，在本公开实施例提供的图像配准方法中，通过获得当前图像帧与其它图像帧之间的第一变换参数之后，结合第一变换参数、以及其它图像帧与目标图像之间的第二变换参数，就可以直接利用当前图像帧与其它图像帧中的目标图像信息，实现当前图像帧与目标图像的配准，相比于利用当前图像帧与每一其它图像帧中所有图像信息进行图像配准，该方法计算量较小，提高了配准速度以且图像配准的准确度。Therefore, in the image registration method provided by the embodiment of the present disclosure, after obtaining the first transformation parameters between the current image frame and other image frames, combining the first transformation parameters and the transformation parameters between other image frames and the target image The second transformation parameter can directly use the target image information in the current image frame and other image frames to realize the registration of the current image frame and the target image, compared to using all the image information in the current image frame and every other image frame For image registration, this method has a small amount of calculation, improves the registration speed and the accuracy of image registration.
图2是本公开实施例提供的图像配准方法中确定第一变换参数的流程示意图。如图2所示，步骤S12，可以通过步骤S121至步骤S123实现：Fig. 2 is a schematic flowchart of determining a first transformation parameter in an image registration method provided by an embodiment of the present disclosure. As shown in Figure 2, step S12 can be realized through steps S121 to S123:
步骤S121：从其它图像帧中查找出关于目标图像的至少一个第一特征点。Step S121: Find at least one first feature point of the target image from other image frames.
在一种实施方式中，在其它图像中包括目标图像的条件下，就可以在其它图像帧中查找出关于目标图像的至少一个第一特征点；示例性的，第一特征点可以是利用特征提取算法得到，特征提取算法可以是ORB算法。在本公开实施例提供的图像配准方法中，从图像帧中提取的特征点，可以包括基于该图像帧建立的图像金字塔中的一系列的图像帧进行特征提取得到的特征点。In one embodiment, under the condition that other images include the target image, at least one first feature point about the target image can be found in other image frames; The extraction algorithm is obtained, and the feature extraction algorithm may be an ORB algorithm. In the image registration method provided by the embodiments of the present disclosure, the feature points extracted from the image frame may include feature points obtained by performing feature extraction on a series of image frames in the image pyramid established based on the image frame.
在本公开实施例提供的图像配准方法中，步骤S121可以通过步骤S1211至步骤S1212实现：In the image registration method provided by the embodiment of the present disclosure, step S121 can be implemented through steps S1211 to S1212:
步骤S1211：基于第二变换参数确定目标图像在其它图像帧中的目标区域；Step S1211: Determine the target area of the target image in other image frames based on the second transformation parameter;
示例性的，其它图像帧与目标图像之间的第二变换参数，可以用于表示目标图像上的点与其它图像帧上的点的对应关系，因此可以确定目标图像的点在其它图像帧中对应的点，由此可以确定目标区域。Exemplarily, the second transformation parameter between other image frames and the target image can be used to represent the corresponding relationship between points on the target image and points on other image frames, so it can be determined that the points of the target image are in other image frames Corresponding points, from which the target area can be determined.
步骤S1212：从目标区域中提取至少一个第一特征点。Step S1212: Extract at least one first feature point from the target area.
在一种实施方式中，确定目标区域以后，可以从目标区域中提取至少一个第一特征点，例如可以利用特征提取算法得到至少一个第一特征点。In an implementation manner, after the target area is determined, at least one first feature point may be extracted from the target area, for example, at least one first feature point may be obtained by using a feature extraction algorithm.
在一种实施方式中，利用第二变换参数，可以确定其它图像帧上的目标区域，由此可以获得至少一个第一特征点，用于后续的图像配准。In one implementation manner, the target area on other image frames can be determined by using the second transformation parameters, thereby obtaining at least one first feature point for subsequent image registration.
步骤S122：从当前图像帧中查找出关于目标图像的至少一个第二特征点。Step S122: Find at least one second feature point on the target image from the current image frame.
在一种实施方式中，当前图像帧中查找出关于目标图像的至少一个第二特征点，可以是利用特征提取算法在当前图像帧中进行特征点提取，从而可以得到一系列的特征点，然后将这些特征点与第一特征点进行匹配，最后将匹配程度满足要求的特征点作为第二特征点。In one embodiment, finding out at least one second feature point about the target image in the current image frame may be to use a feature extraction algorithm to perform feature point extraction in the current image frame, so that a series of feature points can be obtained, and then These feature points are matched with the first feature points, and finally the feature points whose matching degree meets the requirements are used as the second feature points.
在一种实施方式中，可以将从当前图像帧中提取得到的特征点与从目标图像上得到的特征点进行匹配，然后将匹配程度满足要求的特征点作为第二特征点。In one embodiment, the feature points extracted from the current image frame may be matched with the feature points obtained from the target image, and then the feature points whose matching degree meets the requirements are used as the second feature points.
在一种实施方式中，可以利用特征点跟踪的方法获得第二特征点。其中，特征点跟踪的方法可以是光流算法。示例性的，可以分别对至少一个第一特征点进行跟踪，得到当前图像帧中关于目标图像的至少一个第二特征点，也就是说，可以对每一个第一特征点进行跟踪，以获得每一个第一特征点在当前图像帧中对应的点，然后对这些对应的点进行特征提取获得第二特征点。由此，通过利用特征点跟踪的方法，可以获得当前图像帧中与第一特征点对应的特征点，继而得到至少一个第二特征点，用于后续的图像配准。In an implementation manner, the second feature point may be obtained by using a feature point tracking method. Wherein, the feature point tracking method may be an optical flow algorithm. Exemplarily, at least one first feature point can be tracked respectively to obtain at least one second feature point about the target image in the current image frame, that is, each first feature point can be tracked to obtain each A first feature point corresponds to points in the current image frame, and then feature extraction is performed on these corresponding points to obtain a second feature point. Thus, by using the feature point tracking method, the feature points corresponding to the first feature points in the current image frame can be obtained, and then at least one second feature point can be obtained for subsequent image registration.
步骤S123：基于第一特征点和第二特征点，确定第一变换参数。Step S123: Based on the first feature point and the second feature point, determine a first transformation parameter.
示例性的，在得到第一特征点和第二特征点以后，可以基于第一特征点和第二特征点的特征信息，得到第一变换参数。例如，可以基于特征点匹配的方法对第一特征点以及第二特征点进行处理，从而确定第一变换参数。示例性的，可以利用RANSAC对第一特征点以及第二特征点进行处理，确定第一变换参数。Exemplarily, after the first feature point and the second feature point are obtained, the first transformation parameter may be obtained based on feature information of the first feature point and the second feature point. For example, the first feature point and the second feature point may be processed based on a feature point matching method, so as to determine the first transformation parameter. Exemplarily, RANSAC may be used to process the first feature point and the second feature point to determine the first transformation parameter.
由此，通过得到其它图像帧上的第一特征点，以及前图像帧中上的第二特征点，可以实现当前图像帧和其它图像帧的配准。Thus, by obtaining the first feature points on other image frames and the second feature points on the previous image frame, registration between the current image frame and other image frames can be realized.
在一种实施方式中，在得到当前图像帧与目标图像帧之间的第三变换参数之后，还可以进一步对第三变换参数进行优化，以得到更加准确的第三变换参数。示例性的，可以利用预设优化方式，对第三变换参数进行优化。In an implementation manner, after obtaining the third transformation parameter between the current image frame and the target image frame, the third transformation parameter may be further optimized to obtain a more accurate third transformation parameter. Exemplarily, a preset optimization manner may be used to optimize the third transformation parameter.
在一种实施方式中，预设优化方式，可以是迭代优化；示例性的，可以通过迭代优化的方法，计算目标图像与当前帧图像中的目标图像的相似度，并以此为依据优化第三变换参数。In one embodiment, the preset optimization method may be iterative optimization; for example, the iterative optimization method may be used to calculate the similarity between the target image and the target image in the current frame image, and optimize the first frame based on this. Three transformation parameters.
示例性的，优化第三变换参数可以通过式(2)实现：Exemplarily, optimizing the third transformation parameter can be realized by formula (2):
Score＝max H f(T,F(H -1))      (2) Score＝max H f(T,F(H -1 )) (2)
在式(2)中，Score代表相似度得分，得分越高，代表目标图像与当前帧图像越相似；F(H -1) 表示当前帧图像F经第三候选变换参数H变换的结果；f(T,F(H -1)函数用于计算目标图像T与F(H -1)之间的相似度，即f(T,F(H -1)函数用于计算目标图像与当前帧图像的相似程度，示例性的，该函数可以为误差平方和(Sum of Squared Differences，SSD)函数，或者归一化互相关(Normalized Cross Correlation，NCC)函数等；max H表示利用迭代优化的方法优化H，使得目标图像与当前帧图像的相似程度尽可能的提高。迭代优化的方法例如是高斯—牛顿(Gauss-Newton)迭代法或是Levenberg-Marquard算法等等。 In formula (2), Score represents the similarity score, and the higher the score, the more similar the target image is to the current frame image; F(H -1 ) represents the result of the transformation of the current frame image F by the third candidate transformation parameter H; f The (T,F(H -1 ) function is used to calculate the similarity between the target image T and F(H -1 ), that is, the f(T,F(H -1 ) function is used to calculate the target image and the current frame image Exemplary, this function can be the error sum of squares (Sum of Squared Differences, SSD) function, or normalized cross-correlation (Normalized Cross Correlation, NCC ) function etc.; H, to increase the similarity between the target image and the current frame image as much as possible. The iterative optimization method is, for example, Gauss-Newton (Gauss-Newton) iterative method or Levenberg-Marquard algorithm, etc.
示例性的，SSD函数可以如式(3)所示：Exemplarily, the SSD function can be shown as formula (3):
Figure PCTCN2021121049-appb-000005
Figure PCTCN2021121049-appb-000005
其中，
Figure PCTCN2021121049-appb-000006
表示对目标图像T中像素点(x,y)以及由第三变换参数H在当前图像帧F中确定的与其对应的像素点(x,y)所组成的对应的点对的像素值进行误差平方求和。由此可见，相似度SSD(T,F)越小，目标图像与当前图像帧之间的相似度越高，反之，相似度SSD(T,F)越大，目标图像与待配准图像之间的相似度越低。 in,
Figure PCTCN2021121049-appb-000006
Indicates that the pixel value of the corresponding point pair composed of the pixel point (x, y) in the target image T and the corresponding pixel point (x, y) determined by the third transformation parameter H in the current image frame F Sum of squares. It can be seen that the smaller the similarity SSD(T, F), the higher the similarity between the target image and the current image frame; on the contrary, the larger the similarity SSD(T, F), the higher the similarity between the target image and the image to be registered The lower the similarity between.
示例性的，SSD函数还可以如式(4)所示：Exemplarily, the SSD function can also be shown in formula (4):
Figure PCTCN2021121049-appb-000007
Figure PCTCN2021121049-appb-000007
其中，
Figure PCTCN2021121049-appb-000008
表示对目标图像T中像素点(x,y)以及由第三变换参数H在当前图像帧F中确定的与其对应的像素点(x',y')所组成的对应的点对的像素值进行归一化互相关处理。此外，
Figure PCTCN2021121049-appb-000009
表示目标图像中像素点(x,y)像素值的平均值，
Figure PCTCN2021121049-appb-000010
表示当前图像帧中像素点(x',y')像素值的平均值。需要说明的是，NCC(T,F)的值域为-1至1，且NCC(T,F)越接近于1，表示目标图像与当前图像帧之间的相似度越高。 in,
Figure PCTCN2021121049-appb-000008
Represents the pixel value of the corresponding point pair composed of the pixel point (x, y) in the target image T and the corresponding pixel point (x', y') determined by the third transformation parameter H in the current image frame F Perform normalized cross-correlation processing. also,
Figure PCTCN2021121049-appb-000009
Represents the average value of the pixel (x, y) pixel values in the target image,
Figure PCTCN2021121049-appb-000010
Indicates the average value of the pixel value of the pixel point (x', y') in the current image frame. It should be noted that the value range of NCC(T,F) is from -1 to 1, and the closer the NCC(T,F) is to 1, the higher the similarity between the target image and the current image frame is.
通过以上方式，可以实现对第三变换参数的优化，从而可以得到更加准确的第三变换参数，进而能够提高图像配准的效果。Through the above method, the optimization of the third transformation parameter can be realized, so that a more accurate third transformation parameter can be obtained, and the effect of image registration can be improved.
图3是本公开实施例提供的视觉定位方法的第一流程示意图。Fig. 3 is a schematic flowchart of a first visual positioning method provided by an embodiment of the present disclosure.
需要说明的是，本公开实施例提供的视觉定位方法，由电子设备执行，其中，视觉定位，可以通过电子设备的处理器执行。示例性的，电子设备的处理器，可以为ASIC、DSP、DSPD、PLD、FPGA、CPU、控制器、微控制器、微处理器中的至少一种。It should be noted that the visual positioning method provided by the embodiments of the present disclosure is executed by an electronic device, wherein the visual positioning may be executed by a processor of the electronic device. Exemplarily, the processor of the electronic device may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.
如图3所示，该方法包括步骤S21至步骤S22：As shown in Figure 3, the method includes step S21 to step S22:
步骤S21：获取当前图像帧与目标图像之间的当前变换参数。Step S21: Obtain the current transformation parameters between the current image frame and the target image.
在本公开实施中，当前变换参数为利用前述实施例提供的的图像配准方法得到的第三变换参数。In the implementation of the present disclosure, the current transformation parameter is the third transformation parameter obtained by using the image registration method provided in the foregoing embodiments.
步骤S22：利用当前变换参数，得到当前图像帧在世界坐标系中的第一位姿。Step S22: Using the current transformation parameters, obtain the first pose of the current image frame in the world coordinate system.
在一种实施方式中，世界坐标系可以是基于目标图像所在的平面建立的。示例性的，可以以目标图像所在的平面为世界坐标系的预设平面，例如以目标图像所在的平面为世界坐标系的XOY面，或者是XOZ面、YOZ面。示例性的，目标图像的中心可以在世界坐标系的原点，目标图像的横轴可以平行于世界坐标系的X轴，目标图像的纵轴可以平行于世界坐标系的Y轴，世界坐标系的Z轴可以垂直于目标图像平面。In an implementation manner, the world coordinate system may be established based on the plane where the target image is located. Exemplarily, the plane where the target image is located may be the preset plane of the world coordinate system, for example, the plane where the target image is located is the XOY plane, or the XOZ plane or the YOZ plane of the world coordinate system. Exemplarily, the center of the target image can be at the origin of the world coordinate system, the horizontal axis of the target image can be parallel to the X axis of the world coordinate system, the vertical axis of the target image can be parallel to the Y axis of the world coordinate system, and the The Z axis can be perpendicular to the target image plane.
示例性的，在已经获取当前图像帧与目标图像之间的当前变换参数、并且世界坐标系也是基于目标图像所在的平面建立的的情况下，可以对当前变换参数进行换算，得到当前图像帧在世界坐标系中的第一位姿，示例性的，第一位姿可以是相对于电子设备的图像采集装置在拍摄的当前图像帧时对应的世界坐标系时的位姿。对当前变换参数进行换算得到第一位姿的算法可以是位姿估计(Perspective-n-Point，PnP)算法。Exemplarily, when the current transformation parameters between the current image frame and the target image have been obtained, and the world coordinate system is established based on the plane where the target image is located, the current transformation parameters can be converted to obtain the current image frame at The first pose in the world coordinate system. Exemplarily, the first pose may be a pose in the world coordinate system corresponding to the current image frame captured by the image acquisition device of the electronic device. The algorithm for converting the current transformation parameters to obtain the first pose may be a pose estimation (Perspective-n-Point, PnP) algorithm.
由以上可知，在本公开实施例提供的视觉定位方法中，通过获取当前图像帧与目标图像之间的当前变换参数，并且基于目标图像所在的平面建立世界坐标系，就能够得到当前图像帧在世界坐标系中的第一位姿，从而实现了对拍摄设备的视觉定位。As can be seen from the above, in the visual positioning method provided by the embodiments of the present disclosure, by obtaining the current transformation parameters between the current image frame and the target image, and establishing a world coordinate system based on the plane where the target image is located, the current image frame can be obtained at The first pose in the world coordinate system, thus realizing the visual positioning of the shooting device.
图4是本公开实施例提供的视觉定位方法中第一位姿的确定流程示意图。如图4所示，在本公开实施例提供的视觉定位方法中，在步骤S22之前，还可以执行步骤S31至步骤S33。Fig. 4 is a schematic flowchart of determining a first pose in the visual positioning method provided by an embodiment of the present disclosure. As shown in FIG. 4 , in the visual positioning method provided by the embodiment of the present disclosure, before step S22 , step S31 to step S33 may also be executed.
步骤S31：判断当前变换参数是否满足预设要求。Step S31: Judging whether the current transformation parameters meet the preset requirements.
在一种实施方式中，判断变换参数是否满足预设要求，可以包括判断当前变换参数的准确度是否满足要求，示例性的，若当前变换参数的准确度满足预设要求，则可以认为当前变换参数的准确度较高；若当前变换参数的准确度不满足预设要求，则可以认为当前变换参数的准确度较低，不能利用当前变换参数来得到第一位姿。In one embodiment, judging whether the transformation parameters meet the preset requirements may include judging whether the accuracy of the current transformation parameters meets the requirements. For example, if the accuracy of the current transformation parameters meets the preset requirements, it can be considered that the current transformation parameters The accuracy of the parameters is high; if the accuracy of the current transformation parameters does not meet the preset requirements, it can be considered that the accuracy of the current transformation parameters is low, and the current transformation parameters cannot be used to obtain the first pose.
在一种实施方式中，判断变换参数是否满足预设要求，可以是对利用当前变换参数计算得到的当前图像帧与目标图像之间的相似度是否满足预设要求进行判断。In one embodiment, judging whether the transformation parameters meet the preset requirements may be judging whether the similarity between the current image frame calculated using the current transformation parameters and the target image meets the preset requirements.
在一种实施方式中，变换参数满足预设要求，可以包括前述的图像配准方法中按照式(2)的得分Score满足要求。In one embodiment, the transformation parameters meet the preset requirements, which may include that the Score according to the formula (2) in the aforementioned image registration method meets the requirements.
示例性的，在变换参数满足预设要求时，可以执行步骤S32；在变换参数不满足预设要求时，可以执行步骤S33。Exemplarily, when the transformation parameters meet the preset requirements, step S32 may be executed; when the transformation parameters do not meet the preset requirements, step S33 may be executed.
步骤S32：响应于当前变换参数满足预设要求，执行利用当前变换参数，得到当前图像帧在世界坐标系中的第一位姿。Step S32: In response to the current transformation parameters satisfying the preset requirements, execute using the current transformation parameters to obtain the first pose of the current image frame in the world coordinate system.
示例性的，在当前变换参数满足预设要求的情况下，意味着当前变换参数的准确度较高，此时可以执行响应于当前变换参数满足预设要求，执行利用当前变换参数，得到当前图像帧在世界坐标系中的第一位姿的操作。Exemplarily, when the current transformation parameters meet the preset requirements, it means that the accuracy of the current transformation parameters is relatively high. At this time, in response to the current transformation parameters meeting the preset requirements, execute using the current transformation parameters to obtain the current image Manipulation of the frame's first pose in world coordinates.
步骤S33：响应于当前变换参数不满足预设要求，利用其它图像在世界坐标系中的第二位姿、以及当前图像帧与其它图像帧之间的光度误差，确定第一位姿。Step S33: In response to the fact that the current transformation parameters do not meet the preset requirements, determine the first pose by using the second pose of other images in the world coordinate system and the photometric error between the current image frame and other image frames.
示例性的，若当前变换参数不满足预设要求，意味着当前变换参数的准确度不高，即通过当前变换参数得到的第一位姿的准确度也不高，此时可以执行响应于当前变换参数不满足预设要求，利用其它图像在世界坐标系中的第二位姿、以及当前图像帧与其它图像帧之间的光度误差，确定第一位姿的操作。Exemplarily, if the current transformation parameters do not meet the preset requirements, it means that the accuracy of the current transformation parameters is not high, that is, the accuracy of the first pose obtained through the current transformation parameters is not high, and at this time, the response to the current The transformation parameters do not meet the preset requirements, and the second pose of other images in the world coordinate system and the photometric error between the current image frame and other image frames are used to determine the operation of the first pose.
由以上可知，在本公开实施例提供的视觉定位方法中，通过判断当前变换参数是否满足预设要求，在满足预设要求时可以通过当前变换参数得到第一位姿，在不满足预设要求时，通过其它方法得到第一位姿，由此能够通过多种途径灵活的得到更加准确的第一位姿。As can be seen from the above, in the visual positioning method provided by the embodiments of the present disclosure, by judging whether the current transformation parameters meet the preset requirements, the first pose can be obtained through the current transformation parameters when the preset requirements are met, and if the preset requirements are not met , the first pose can be obtained by other methods, so that a more accurate first pose can be flexibly obtained through various ways.
在本公开实施例中，在执行获取当前图像帧与目标图像之间的当前变换参数之前，还可以先执行获取于上一图像帧的位姿获取方式的操作，以此来确定于上一图像帧的位姿获取方式。在本实施例中，上一图像帧的位姿可以是通过图像跟踪的方式来获取，也可以是通过其它方式来获取的，示例性的，其它方式可以是视觉导航等方法。In the embodiment of the present disclosure, before performing the acquisition of the current transformation parameters between the current image frame and the target image, the operation of obtaining the pose acquisition mode obtained in the previous image frame may be performed first, so as to determine the transformation parameters in the previous image frame The frame pose acquisition method. In this embodiment, the pose of the last image frame may be obtained by image tracking, or by other methods. Exemplarily, other methods may be methods such as visual navigation.
在本公开实施例提供的视觉定位方法中，在执行获取当前图像帧与目标图像之间的当前变换参数 之前，还可以执行以下操作：In the visual positioning method provided by the embodiments of the present disclosure, before performing the acquisition of the current transformation parameters between the current image frame and the target image, the following operations may also be performed:
响应于上一图像帧的位姿获取方式为图像跟踪方式，执行获取当前图像帧与目标图像之间的当前变换参数。In response to the pose acquisition mode of the last image frame being the image tracking mode, the current transformation parameter between the current image frame and the target image is acquired.
其中，图像跟踪方式为利用上一图像帧与目标图像之间的变换参数确定上一图像帧在世界坐标系中的位姿。Wherein, the image tracking method is to use the transformation parameters between the previous image frame and the target image to determine the pose of the previous image frame in the world coordinate system.
在一种实施方式中，可以通过上一图像帧与目标图像之间的单应性矩阵，获取上一图像帧在世界坐标系中的位姿。示例性的，由于上一图像帧的位姿是通过图像跟踪方式获取的，这就意味着上一图像帧中存在目标图像，那么当前图像帧也可能存在目标图像，因此可以选择通过当前图像帧与目标图像之间的当前变换参数，得到当前图像帧在世界坐标系中的第一位姿。In an implementation manner, the pose of the last image frame in the world coordinate system can be acquired through the homography matrix between the last image frame and the target image. Exemplarily, since the pose of the previous image frame is obtained by image tracking, this means that there is a target image in the previous image frame, then there may also be a target image in the current image frame, so you can choose to pass the current image frame The current transformation parameters between the target image and the first pose of the current image frame in the world coordinate system.
在一种实施方式中，若上一图像帧的位姿获取方式不是通过图像跟踪方式获取的，则意味着上一图像帧中可能不存在目标图像，因此可以选择通过其它方法来获得当前图像帧在世界坐标系中的第一位姿。In one embodiment, if the pose acquisition method of the previous image frame is not obtained by image tracking, it means that there may not be a target image in the previous image frame, so other methods can be selected to obtain the current image frame The first pose in world coordinates.
在本公开实施例提供的视觉定位方法中，视觉定位方法实现时电子设备的定位状态可以包括两种状态，一种是图像跟踪状态，一种是视觉导航状态。In the visual positioning method provided by the embodiments of the present disclosure, when the visual positioning method is implemented, the positioning state of the electronic device may include two states, one is an image tracking state, and the other is a visual navigation state.
示例性的，在执行获取当前图像帧与目标图像之间的当前变换参数之前，还可以先执行判断电子设备定位状态的操作，以确定是否获取当前图像帧与目标图像之间的当前变换参数。示例性的，定位状态，可以是根据上一图像帧在世界坐标系中的位姿的获取方式确定的。示例性的，若上一图像帧在世界坐标系中的位姿，是通过图像配准方法，对目标图像进行检测得到的，再得到变换参数来确定的，则可以确定定位状态为图像跟踪状态；示例性的，图像配准方法，可以包括前述实施例提供的图像配准方法。Exemplarily, before executing the acquisition of the current transformation parameters between the current image frame and the target image, the operation of judging the positioning state of the electronic device may also be performed first, so as to determine whether to acquire the current transformation parameters between the current image frame and the target image. Exemplarily, the positioning state may be determined according to an acquisition method of the pose of the last image frame in the world coordinate system. Exemplarily, if the pose of the last image frame in the world coordinate system is obtained by detecting the target image through the image registration method, and then obtaining the transformation parameters to determine, then the positioning state can be determined as the image tracking state Exemplarily, the image registration method may include the image registration method provided in the foregoing embodiments.
示例性的，若上一图像帧在世界坐标系中的位姿不是通过图像配准方法得到的，则可以确定定位状态为视觉导航状态。Exemplarily, if the pose of the last image frame in the world coordinate system is not obtained through an image registration method, the positioning state may be determined as the visual navigation state.
示例性的，若电子设备当前不处于图像跟踪状态，那么当前可以处于视觉导航状态，此时可以执行获取当前图像帧与目标图像之间的当前变换参数，使得后续可以获得第一位姿。示例性的，在当前处于视觉导航状态的情况下，可以执行步骤S33。Exemplarily, if the electronic device is not currently in the image tracking state, it may be currently in the visual navigation state, and at this time, the current transformation parameters between the current image frame and the target image may be obtained, so that the first pose can be obtained subsequently. Exemplarily, step S33 may be executed in the case of the current visual navigation state.
在本公开实施例提供的视觉定位方法中，在判断当前变换参数是否满足预设要求执行之后，可以表示已经执行了获取当前图像帧与目标图像之间的当前变换参数这一操作，此时可以确认定位状态为图像跟踪状态。In the visual positioning method provided by the embodiments of the present disclosure, after judging whether the current transformation parameters meet the preset requirements, it can be indicated that the operation of obtaining the current transformation parameters between the current image frame and the target image has been executed. At this time, it can be Confirm that the positioning status is image tracking status.
示例性的，在后续，可以根据当前变换参数是否满足预设要求的判断结果，再次确定定位状态。比如，在当前变换参数满足预设要求的情况下，保持处于图像跟踪状态，当前变换参数满足预设要求，表示可以通过当前变换参数获得当前图像帧的第一位姿，因此可以保持处于图像跟踪状态。示例性的，在当前变换参数不满足预设要求的情况下，表示不能通过当前变换参数来获得当前图像帧的第一位姿，此时可以切换至视觉导航状态，并执行步骤S33。Exemplarily, subsequently, the positioning state may be determined again according to the judgment result of whether the current transformation parameter meets the preset requirement. For example, if the current transformation parameters meet the preset requirements, keep in the image tracking state, and the current transformation parameters meet the preset requirements, which means that the first pose of the current image frame can be obtained through the current transformation parameters, so you can keep in the image tracking state state. Exemplarily, in the case that the current transformation parameters do not meet the preset requirements, it means that the first pose of the current image frame cannot be obtained through the current transformation parameters, at this time, it may switch to the visual navigation state, and perform step S33.
因此，在本公开实施例提供的视觉定位方法中，通过当前变换参数是否满足预设要求，就可以确定电子设备的定位状态，进而确定获取第一位姿的具体方法。Therefore, in the visual positioning method provided by the embodiments of the present disclosure, the positioning state of the electronic device can be determined according to whether the current transformation parameters meet the preset requirements, and then the specific method for obtaining the first pose can be determined.
图5是本公开实施例提供的视觉定位方法的第二流程示意图。在本公开实施例提供的视觉定位方法中，其它图像帧和当前图像帧是由电子设备的图像采集装置先后拍摄得到的。在这种情况下，利用其它图像在世界坐标系中的第二位姿、以及当前图像帧与其它图像帧之间的光度误差，确定第一位姿，可以通过步骤S331至步骤S333实现：Fig. 5 is a second schematic flowchart of a visual positioning method provided by an embodiment of the present disclosure. In the visual positioning method provided by the embodiments of the present disclosure, other image frames and the current image frame are sequentially captured by an image acquisition device of an electronic device. In this case, using the second pose of other images in the world coordinate system and the photometric error between the current image frame and other image frames to determine the first pose can be achieved through steps S331 to S333:
步骤S331：获取第一参考姿态。Step S331: Obtain a first reference pose.
其中，第一参考姿态是图像采集装置对应于当前图像帧的拍摄时刻且相对于参考平面的姿态。Wherein, the first reference posture is the posture of the image acquisition device corresponding to the shooting moment of the current image frame and relative to the reference plane.
在一种实施方式中，第一参考姿态，可以是电子设备的旋转信息，即电子设备相对于参考平面的旋转信息。In an implementation manner, the first reference pose may be rotation information of the electronic device, that is, rotation information of the electronic device relative to a reference plane.
在一种实施方式中，第一参考姿态，可以是由电子设备的传感装置检测得到的。示例性的，传感装置可以是陀螺仪。In an implementation manner, the first reference posture may be detected by a sensing device of the electronic device. Exemplarily, the sensing device may be a gyroscope.
在一种实施方式中，第一参考姿态的检测时刻与当前图像帧的拍摄时刻之间的差值不超过第一预设时间差。示例性的，第一预设差值可以是较短的时间长度，比如可以是20毫秒或者15毫秒等等，第一预设差值可以根据实际需要调整和设置。In one embodiment, the difference between the detection moment of the first reference pose and the shooting moment of the current image frame does not exceed a first preset time difference. Exemplarily, the first preset difference may be a short period of time, such as 20 milliseconds or 15 milliseconds, etc., and the first preset difference may be adjusted and set according to actual needs.
在一种实施方式中，在多个检测时刻与当前图像帧的拍摄时刻之间的差值，均不超过第一预设时间差的情况下，可以选择与当前图像帧的拍摄时刻最接近的检测时刻、所对应的图像采集装置相对于参考平面的姿态来获取第一参考姿态；示例性的，检测时刻与当前图像帧的拍摄时刻之间的差值不超过第一预设时间差，即检测时刻与当前图像帧的拍摄时刻非常接近，此时可以认为第一参考姿态是当前图像帧的拍摄时刻的设备的姿态信息。示例性的，参考平面例如是基于传感装置的检测参数确定的某一平面。In one embodiment, when the difference between multiple detection moments and the shooting moment of the current image frame does not exceed the first preset time difference, the detection moment closest to the shooting moment of the current image frame can be selected. time, and the attitude of the corresponding image acquisition device relative to the reference plane to obtain the first reference attitude; for example, the difference between the detection moment and the shooting moment of the current image frame does not exceed the first preset time difference, that is, the detection moment It is very close to the shooting moment of the current image frame, and at this time, it can be considered that the first reference pose is the pose information of the device at the shooting moment of the current image frame. Exemplarily, the reference plane is, for example, a certain plane determined based on detection parameters of the sensing device.
步骤S332：利用参考平面与世界坐标系中的预设平面之间的偏移量，对第一参考姿态进行调整，得到第二参考姿态。Step S332: Using the offset between the reference plane and the preset plane in the world coordinate system, the first reference attitude is adjusted to obtain the second reference attitude.
示例性的，世界坐标系中的预设平面，例如是世界坐标系的XOY平面，或是XOZ平面，YOZ平面等等。在一种实施方式中，预设平面可以是世界坐标系的XOY平面。在一种实施方式中，目标图像所在的平面可以为预设平面。Exemplarily, the preset plane in the world coordinate system is, for example, the XOY plane, or the XOZ plane, the YOZ plane, etc. of the world coordinate system. In an implementation manner, the preset plane may be the XOY plane of the world coordinate system. In an implementation manner, the plane where the target image is located may be a preset plane.
示例性的，在得到第一参考姿态以后，可以表示已经获得电子设备相对于参考平面的旋转信息。此时，可以获取参考平面与其它平面之间的偏移量，并利用偏移量对第一参考姿态进行调整，以此得到电子设备相对于其它平面的第二参考姿态，示例性的，第二参考姿态，可以包括电子设备相对其它平面的旋转信息。Exemplarily, after the first reference pose is obtained, it may indicate that the rotation information of the electronic device relative to the reference plane has been obtained. At this time, the offset between the reference plane and other planes can be obtained, and the offset can be used to adjust the first reference attitude, so as to obtain the second reference attitude of the electronic device relative to other planes. Exemplarily, the first The second reference posture may include rotation information of the electronic device relative to other planes.
在本公开实施例提供的视觉定位方法中，其它平面可以是世界坐标系中的预设平面，因此第二参考姿态可以认为是电子设备相对于世界坐标系的预设平面的旋转信息。In the visual positioning method provided by the embodiments of the present disclosure, other planes may be preset planes in the world coordinate system, so the second reference pose may be regarded as rotation information of the electronic device relative to the preset plane of the world coordinate system.
在一种实施方式中，第一参考姿态可以是由陀螺仪检测得到的，因此参考平面可以是基于陀螺仪确定的某一平面，此时，通过利用参考平面与世界坐标系中的预设平面之间的偏移量，对第一参考姿态进行调整，而得到的第二参考姿态，也可以认为是将参考平面变换到预设平面需要的旋转量。In one embodiment, the first reference attitude can be detected by the gyroscope, so the reference plane can be a certain plane determined based on the gyroscope. At this time, by using the reference plane and the preset plane in the world coordinate system The offset between the first reference pose is adjusted, and the obtained second reference pose can also be considered as the rotation amount required to transform the reference plane to the preset plane.
由此，通过获取参考平面与世界坐标系中的预设平面之间的偏移量，可以基于该偏移量对第一参考姿态信息进行调整，得到第二参考姿态信息，就能够获得相对于参考平面以外的平面(例如是世界坐标系的预设平面)的参考姿态信息，使得世界坐标系的预设平面可以位于参考平面以外的任意平面，并且也同时可以利用参考姿态信息来优化最终位姿信息，提高了最终位姿信息的准确度。Thus, by obtaining the offset between the reference plane and the preset plane in the world coordinate system, the first reference attitude information can be adjusted based on the offset to obtain the second reference attitude information, which can be obtained relative to The reference attitude information of a plane other than the reference plane (such as the preset plane of the world coordinate system), so that the preset plane of the world coordinate system can be located on any plane other than the reference plane, and at the same time, the reference attitude information can be used to optimize the final position pose information, which improves the accuracy of the final pose information.
图6为本公开实施例提供的视觉定位方法中偏移量获取的流程示意图。如图6所示，在本公开实施例提供的视觉定位方法中，在步骤S332之前，还可以执行步骤S41至步骤S42：Fig. 6 is a schematic flowchart of obtaining an offset in the visual positioning method provided by an embodiment of the present disclosure. As shown in FIG. 6, in the visual positioning method provided by the embodiment of the present disclosure, before step S332, steps S41 to S42 may also be performed:
步骤S41：获取第一历史图像帧在世界坐标系中的第三位姿，获取第三参考姿态。Step S41: Obtain the third pose of the first historical image frame in the world coordinate system, and obtain the third reference pose.
其中，第三参考姿态，是图像采集装置对应于第一历史图像帧的拍摄时刻且相对于参考平面的姿态；第三位姿是基于目标图像确定的；预设平面为目标图像所在的平面。Wherein, the third reference pose is the pose of the image acquisition device corresponding to the shooting moment of the first historical image frame and relative to the reference plane; the third pose is determined based on the target image; the preset plane is the plane where the target image is located.
在一种实施方式中，预设平面可以为目标图像所在的平面。In an implementation manner, the preset plane may be the plane where the target image is located.
在一种实施方式中，第三位姿可以是基于目标图像确定的；示例性的，第三位姿，可以是基于目标图像，利用图像配准算法检测第一历史图像帧与目标图像，得到它们之间的第四变换参数后再对第四变换参数进行换算后得到的。In one embodiment, the third pose may be determined based on the target image; for example, the third pose may be based on the target image, using an image registration algorithm to detect the first historical image frame and the target image, to obtain The fourth transformation parameter between them is obtained after converting the fourth transformation parameter.
在一种实施方式中，第三参考姿态，可以是电子设备的图像采集装置对应于第二历史图像帧的拍摄时刻且相对于参考平面的姿态；示例性的，第二历史图像帧位于第一历史图像帧之前；In one embodiment, the third reference posture may be the posture of the image acquisition device of the electronic device corresponding to the shooting moment of the second historical image frame relative to the reference plane; for example, the second historical image frame is located at the first Before the history image frame;
在一种实施方式中，第三参考姿态，可以是电子由设备的传感装置检测得到的；示例性的，第三参考姿态的检测时刻与第一历史图像帧的拍摄时刻之间的差值不超过第二预设时间差，此时，可以认为第三参考姿态与第三位姿的姿态信息相同。In one embodiment, the third reference pose may be electronically detected by the sensing device of the device; for example, the difference between the detection moment of the third reference pose and the shooting moment of the first historical image frame It does not exceed the second preset time difference. At this time, it can be considered that the attitude information of the third reference attitude is the same as that of the third attitude.
在一种实施方式中，通过对第一历史图像帧和目标图像分别进行特征提取，可以得到与第一历史图像帧对应的第一特征点，以及与目标图像对应的第二特征点。本公开实施例对特征点的数量不做具体限制。示例性的，对第一历史图像帧以及目标图像进行特征提取可以是通过特征提取算法实现的，特征提取算法可以是FAST算法、SIFT算法以及ORB算法中的任一算法。In an implementation manner, by performing feature extraction on the first historical image frame and the target image respectively, first feature points corresponding to the first historical image frame and second feature points corresponding to the target image can be obtained. The embodiments of the present disclosure do not specifically limit the number of feature points. Exemplarily, the feature extraction of the first historical image frame and the target image may be implemented by a feature extraction algorithm, and the feature extraction algorithm may be any algorithm among the FAST algorithm, the SIFT algorithm and the ORB algorithm.
在一种实施方式中，在得到第一特征点以及第二特征点以后，还可以得到与每个特征点对应的特征表示，示例性的，特征表示可以是特征向量，在这种情况下，每一个特征点，可以均有一个与其对应的特征表示。In one embodiment, after obtaining the first feature point and the second feature point, a feature representation corresponding to each feature point can also be obtained. Exemplarily, the feature representation can be a feature vector. In this case, Each feature point may have a corresponding feature representation.
在本公开实施例提供的视觉定位方法中，通过计算每一个第一特征点与每一个第二特征点的匹配程度，可以得到一系列的匹配点对，然后可以选择匹配程度高的匹配点作为第一匹配点对。示例性的，计算第一特征点和第二特征点的匹配程度，可以是计算第一特征点的特征表示与第二特征点的特征表示之间的距离，距离越近，可以确定第一特征点与第二特征点越匹配。In the visual positioning method provided by the embodiments of the present disclosure, by calculating the degree of matching between each first feature point and each second feature point, a series of matching point pairs can be obtained, and then a matching point with a high degree of matching can be selected as the The first matching point pair. Exemplarily, calculating the matching degree between the first feature point and the second feature point may be calculating the distance between the feature representation of the first feature point and the feature representation of the second feature point, the closer the distance, the first feature can be determined The more the point matches the second feature point.
在本公开实施例提供的视觉定位方法中，获取第一历史图像帧在世界坐标系中的第三位姿，可以通过以下操作实现：In the visual positioning method provided by the embodiments of the present disclosure, obtaining the third pose of the first historical image frame in the world coordinate system can be achieved by the following operations:
利用基于上述匹配操作得到的一系列第一匹配点对，利用图像配准算法，确定第一历史图像帧与目标图像之间的第四变换参数，并利用第四变换参数得到第三位姿。示例性的，图像配准算法可以是RANSAC。Using a series of first matching point pairs obtained based on the above matching operation, using an image registration algorithm, determine a fourth transformation parameter between the first historical image frame and the target image, and use the fourth transformation parameter to obtain a third pose. Exemplarily, the image registration algorithm may be RANSAC.
在本公开实施例提供的视觉定位方法中，获取第一历史图像帧在世界坐标系中的第三位姿，可以通过以下操作实现：In the visual positioning method provided by the embodiments of the present disclosure, obtaining the third pose of the first historical image frame in the world coordinate system can be achieved by the following operations:
基于第一历史图像帧与第二历史图像帧之间的第二匹配点对，确定第一历史图像帧与第二历史图像帧之间的第五变换参数；利用第五变换参数和第二历史图像帧与目标图像之间的第六变换参数，得到第四变换参数，最后再基于第四变换参数得到第三位姿。Based on the second matching point pair between the first historical image frame and the second historical image frame, determine the fifth transformation parameter between the first historical image frame and the second historical image frame; using the fifth transformation parameter and the second history The sixth transformation parameter between the image frame and the target image is used to obtain the fourth transformation parameter, and finally the third pose is obtained based on the fourth transformation parameter.
其中，第二历史图像帧位于第一历史图像帧之前。Wherein, the second historical image frame is located before the first historical image frame.
示例性的，获得第二匹配点对的过程，可以参照上述获得第一匹配点对的具体描述，此处不再赘述；得到第五变换参数的可以参照上述的图像配准方法实施例，此处不再赘述；第二历史图像帧与目标图像之间的第六变换参数可以基于图像配准算法得到，此处不再赘述。Exemplarily, for the process of obtaining the second matching point pair, refer to the above-mentioned specific description of obtaining the first matching point pair, which will not be repeated here; for obtaining the fifth transformation parameter, refer to the above-mentioned image registration method embodiment, here No more details here; the sixth transformation parameter between the second historical image frame and the target image can be obtained based on an image registration algorithm, so no more details are given here.
由以上可知，在本公开实施例提供的视觉定位方法中，通过获得第一历史图像帧与目标图像之间的第四变换参数，或是通过利用第一历史图像帧与第二历史图像帧之间的第五变换参数、以及第二历史图像帧与目标图像之间的第六变换参数来得到第四变换参数，均可以获得第一历史图像帧的第三位姿，实现视觉定位，从而使得视觉定位的方式更加灵活。As can be seen from the above, in the visual positioning method provided by the embodiments of the present disclosure, by obtaining the fourth transformation parameter between the first historical image frame and the target image, or by using the difference between the first historical image frame and the second historical image frame The fifth transformation parameter between and the sixth transformation parameter between the second historical image frame and the target image to obtain the fourth transformation parameter can obtain the third pose of the first historical image frame to realize visual positioning, so that The way of visual positioning is more flexible.
在本公开实施例提供的视觉定位方法中，利用第四变换参数得到第三位姿，可以通过以下操作实现：In the visual positioning method provided by the embodiment of the present disclosure, using the fourth transformation parameter to obtain the third pose can be achieved by the following operations:
响应于第四变换参数满足预设要求，确定处于图像跟踪状态，利用第四变换参数得到第三位姿。In response to the fourth transformation parameter meeting the preset requirement, it is determined to be in the image tracking state, and the fourth transformation parameter is used to obtain the third pose.
也就是说，在利用第四变换参数得到第三位姿之前，还可以先判断第四变换参数是否满足预设要求。That is to say, before using the fourth transformation parameter to obtain the third pose, it may also be judged first whether the fourth transformation parameter satisfies the preset requirement.
示例性的，在第四变换参数不满足预设要求的情况下，可以确定电子设备处于视觉导航状态，执行前述实施例提供的利用其它图像在世界坐标系中的第二位姿、以及当前图像帧与其它图像帧之间的光度误差，确定第一位姿的操作。Exemplarily, in the case that the fourth transformation parameter does not meet the preset requirements, it may be determined that the electronic device is in the visual navigation state, and the second pose in the world coordinate system using other images provided in the foregoing embodiments, and the current image The photometric error between the frame and other image frames determines the operation of the first pose.
由以上可知，在本公开实施例提供的视觉定位方法中，通过判断第四变换参数是否满足预设要求，可以利用准确度较高的第四变换参数来获得更加准确的第三位姿。It can be seen from the above that, in the visual positioning method provided by the embodiments of the present disclosure, by judging whether the fourth transformation parameter meets the preset requirement, a more accurate third pose can be obtained by using the fourth transformation parameter with higher accuracy.
在本公开实施例提供的视觉定位方法中，获得第一历史图像帧与目标图像的第四变换参数，或者获得第二历史图像帧与目标图像的第六变换参数的过程，可以通过步骤A1至步骤A2实现：In the visual positioning method provided by the embodiment of the present disclosure, the process of obtaining the fourth transformation parameter between the first historical image frame and the target image, or obtaining the sixth transformation parameter between the second historical image frame and the target image can be performed through steps A1 to Step A2 is implemented:
步骤A1：选择其中一组第一匹配点对作为目标匹配点对。Step A1: Select one of the first matching point pairs as a target matching point pair.
在一种实施方式中，对目标图像进行特征提取得到的特征点，可以定义为第三特征点；基于第一历史图像帧或者第二历史图像帧进行特征提取得到的特征点，可以定义为第四特征点；示例性的，通过计算第三特征点与第四特征点的匹配程度，可以得到第一匹配点对。In one embodiment, the feature points obtained by feature extraction of the target image can be defined as the third feature point; the feature points obtained by feature extraction based on the first historical image frame or the second historical image frame can be defined as the third feature point Four feature points; for example, by calculating the matching degree between the third feature point and the fourth feature point, the first matching point pair can be obtained.
示例性的，选择一组第一匹配点对作为目标匹配点对，可以是从最匹配的点对开始选起，在目标匹配点中，第三特征点可以为第一匹配点，第四特征点可以为第二匹配点。Exemplarily, a group of first matching point pairs is selected as the target matching point pair, which can be selected from the most matching point pair. In the target matching point, the third feature point can be the first matching point, and the fourth feature point The point can be the second matching point.
步骤A2：基于目标匹配点对的方向信息，得到与目标匹配点对相对应的单应性矩阵。Step A2: Obtain the homography matrix corresponding to the target matching point pair based on the direction information of the target matching point pair.
示例性的，目标匹配点对的方向信息可以表示第一历史帧图像相对于目标图像的旋转角度，或者 可以表示第二历史帧图像相对于目标图像的旋转角度。示例性的，首先可以在目标图像中提取以第一匹配点为中心的第一图像区域，在第一历史图像帧或者第二历史图像帧中提取以第二匹配点为中心的第二图像区域；然后，确定第一图像区域的第一偏转角度以及第二图像区域的第二偏转角度；再基于第一偏转角度和第二偏转角度，得到变换参数。示例性的，变换参数，可以是基于目标匹配点对的方向信息，以及目标匹配点对中的第一匹配点与第二匹配点的像素坐标信息得到的。Exemplarily, the direction information of the target matching point pair may represent the rotation angle of the first historical frame image relative to the target image, or may represent the rotation angle of the second historical frame image relative to the target image. Exemplarily, first, the first image area centered on the first matching point can be extracted in the target image, and the second image area centered on the second matching point can be extracted in the first historical image frame or the second historical image frame ; Then, determine the first deflection angle of the first image area and the second deflection angle of the second image area; then obtain transformation parameters based on the first deflection angle and the second deflection angle. Exemplarily, the transformation parameter may be obtained based on the direction information of the target matching point pair and the pixel coordinate information of the first matching point and the second matching point in the target matching point pair.
在一种实施方式中，第一偏转角度，可以为第一图像区域的形心与第一图像区域的中心的连线、与预设方向(例如是世界坐标系的X轴)之间的有向夹角；第二偏转角度，可以为第二图像区域的形心与第二图像区域的中心的连线、与预设方向之间的有向夹角。In one embodiment, the first deflection angle may be the distance between the line connecting the centroid of the first image area and the center of the first image area and a preset direction (for example, the X axis of the world coordinate system). directional angle; the second deflection angle may be the directional angle between the line connecting the centroid of the second image area and the center of the second image area and the preset direction.
示例性的，第一偏转角度
Figure PCTCN2021121049-appb-000011
可以直接通过式(5)得到： Exemplary, the first deflection angle
Figure PCTCN2021121049-appb-000011
It can be directly obtained by formula (5):
Figure PCTCN2021121049-appb-000012
Figure PCTCN2021121049-appb-000012
在式(5)中，(x,y)表示第一图像区域中某一像素点相对第一图像区域中心的偏移量，I(x,y)表示该像素点的像素值，∑表示求和，其求和范围为第一图像区域中的像素点。同理，第二偏转角度也可以按照相同的方法计算得到。In formula (5), (x, y) represents the offset of a certain pixel point in the first image area relative to the center of the first image area, I(x, y) represents the pixel value of the pixel point, and ∑ represents the calculation and, the summation range is the pixels in the first image area. Similarly, the second deflection angle can also be calculated by the same method.
在一种实施方式中，可以通过步骤B1至步骤B2得到第一历史帧图像或者第二历史帧图像与目标图像之间的变换参数：In one embodiment, the conversion parameters between the first historical frame image or the second historical frame image and the target image can be obtained through steps B1 to B2:
步骤B1：获取第一偏转角度与第二偏转角度之间的角度差。Step B1: Obtain the angle difference between the first deflection angle and the second deflection angle.
示例性的，计算角度差的计算可以通过式(6)实现：Exemplarily, the calculation of calculating the angle difference can be realized by formula (6):
Figure PCTCN2021121049-appb-000013
Figure PCTCN2021121049-appb-000013
其中，θ为角度差，
Figure PCTCN2021121049-appb-000014
为第一偏转角度，T表示目标图像，
Figure PCTCN2021121049-appb-000015
为第二偏转角度，F表示第一历史帧图像或者第二历史帧图像。 Among them, θ is the angle difference,
Figure PCTCN2021121049-appb-000014
is the first deflection angle, T represents the target image,
Figure PCTCN2021121049-appb-000015
is the second deflection angle, and F represents the first historical frame image or the second historical frame image.
步骤B2：基于角度差和第一匹配点对所对应的尺度，得到第一候选变换参数。Step B2: Obtain a first candidate transformation parameter based on the angle difference and the scale corresponding to the first matching point pair.
示例性的，第一候选变换参数，可以是第一历史帧图像或者第二历史帧图像与目标图像之间对应的单应性矩阵。Exemplarily, the first candidate transformation parameter may be a corresponding homography matrix between the first historical frame image or the second historical frame image and the target image.
其中，单应性矩阵的计算可以通过式(7)实现：Among them, the calculation of the homography matrix can be realized by formula (7):
H＝H lH SH RH r     (7) H＝H l H S H R H r (7)
其中，H为目标图像与第一历史帧图像或者第二历史帧图像之间对应的单应性矩阵，即第一候选变换参数；H r表示第一历史帧图像或者第二历史帧图像相对于目标图像的平移量；H S表示第一匹配点对所对应的尺度，示例性的，尺度可以为在对目标图像进行缩放时的比例信息；H R表示第一历史帧图像或者第二历史帧图像相对于目标图像的旋转量，H l表示平移之后复位的平移量。 Among them, H is the corresponding homography matrix between the target image and the first historical frame image or the second historical frame image, that is, the first candidate transformation parameter; H r indicates that the first historical frame image or the second historical frame image is relative to The translation amount of the target image; H S represents the scale corresponding to the first matching point pair, for example, the scale can be the ratio information when scaling the target image; HR represents the first historical frame image or the second historical frame The amount of rotation of the image relative to the target image, H l represents the translation amount reset after translation.
为了求得角度差，可以对式(7)进行变换，得到式(8)：In order to obtain the angle difference, formula (7) can be transformed to obtain formula (8):
Figure PCTCN2021121049-appb-000016
Figure PCTCN2021121049-appb-000016
在式(8)中，
Figure PCTCN2021121049-appb-000017
为第一匹配点在目标图像上的像素坐标；
Figure PCTCN2021121049-appb-000018
为第二匹配点在第一历史帧图像或者第二历史帧图像上的像素坐标；s为第一匹配点对所对应的尺度，即点
Figure PCTCN2021121049-appb-000019
对应的尺度；θ为角度差。 In formula (8),
Figure PCTCN2021121049-appb-000017
is the pixel coordinate of the first matching point on the target image;
Figure PCTCN2021121049-appb-000018
is the pixel coordinate of the second matching point on the first historical frame image or the second historical frame image; s is the scale corresponding to the first matching point pair, that is, point
Figure PCTCN2021121049-appb-000019
Corresponding scale; θ is the angle difference.
由以上可知，在本公开实施例提供的视觉定位方法中，通过计算目标匹配点对的方向信息，获得第一历史帧图像或者第二历史帧图像相对于目标图像的旋转角度，从而可以利用该旋转角度信息得到第一历史帧图像或者第二历史帧图像与目标图像之间的变换参数，实现了利用匹配点对计算变换参数。As can be seen from the above, in the visual positioning method provided by the embodiment of the present disclosure, by calculating the direction information of the target matching point pair, the rotation angle of the first historical frame image or the second historical frame image relative to the target image can be obtained, so that the rotation angle of the first historical frame image or the second historical frame image can be used. The rotation angle information obtains the transformation parameters between the first historical frame image or the second historical frame image and the target image, and realizes the calculation of transformation parameters by using matching point pairs.
步骤S42：利用第三位姿中的姿态和第三参考姿态，得到偏移量。Step S42: Using the pose in the third pose and the third reference pose to obtain the offset.
其中，第三位姿中的姿态是相对于预设平面的姿态。Wherein, the attitude in the third pose is the attitude relative to the preset plane.
示例性的，第三位姿，可以为相对于预设平面的旋转量信息。Exemplarily, the third pose may be rotation amount information relative to a preset plane.
在一种实施方式中，可以将第三位姿中的姿态与第三参考姿态之间的比值，作为偏移量，即可以通过求取比值的方式，获得偏移量。In an implementation manner, the ratio between the pose in the third pose and the third reference pose can be used as the offset, that is, the offset can be obtained by calculating the ratio.
在一种实施方式中，在第三位姿中的姿态为R 1、第二参考姿态位R 2、δ表示偏移量的情况下，偏移量的计算可以通过式(9)实现： In one embodiment, when the attitude in the third pose is R 1 , the second reference pose is R 2 , and δ represents the offset, the calculation of the offset can be realized by formula (9):
δ＝R 1(R 2) -1       (9) δ = R 1 (R 2 ) -1 (9)
由以上可知，通过第三位姿中的姿态信息，以及可以视为与第三位姿信息拍摄时刻同时刻获取的第三参考姿态，即可以获得参考平面与世界坐标系中的预设平面之间的偏移量。As can be seen from the above, through the attitude information in the third pose and the third reference attitude that can be regarded as being acquired at the same time as the shooting time of the third pose information, the distance between the reference plane and the preset plane in the world coordinate system can be obtained. offset between.
在本公开实施例中，通过特征提取算法进行特征提取得到的特征点，可以认为与目标图像位于同一平面。In the embodiments of the present disclosure, the feature points obtained by feature extraction through the feature extraction algorithm may be considered to be located on the same plane as the target image.
步骤S333：基于第二参考姿态、第二位姿以及当前图像帧与历史图像帧之间的光度误差，确定第一位姿。Step S333: Determine the first pose based on the second reference pose, the second pose, and the photometric error between the current image frame and the historical image frame.
在一种实施方式中，可以首先得到当前图像帧与其它图像帧之间的相对位姿变化，计算当前图像帧与其它图像帧之间的光度误差，利用相对位姿变化得到当前图像帧的最终位姿，然后利用第二参考姿态作为约束因素，优化并减小光度误差，从而确定第一位姿。In one embodiment, the relative pose change between the current image frame and other image frames can be obtained first, the photometric error between the current image frame and other image frames can be calculated, and the final result of the current image frame can be obtained by using the relative pose change pose, and then use the second reference pose as a constraint to optimize and reduce the photometric error to determine the first pose.
在一种实施方式中，可以首先得到当前图像帧在世界坐标系中的位姿信息作为初始位姿，然后利用其它图像帧在世界坐标系中的的位姿信息，得到当前图像帧与其它图像帧之间的光度误差，再利用第二参考姿态作为约束因素，优化并减小光度误差，从而确定第一位姿。In one embodiment, the pose information of the current image frame in the world coordinate system can be obtained as the initial pose, and then the pose information of other image frames in the world coordinate system can be used to obtain the current image frame and other images The photometric error between frames, and then use the second reference pose as a constraint factor to optimize and reduce the photometric error, so as to determine the first pose.
由以上可知，通过获取参考平面与世界坐标系中预设平面之间的偏移量，再基于该偏移量对第一参考姿态信息进行调整，得到第二参考姿态信息，就可以获得相对于参考平面以外的平面(例如是世界坐标系的预设平面)的参考姿态信息，从而使得世界坐标系的预设平面可以位于参考平面以外的任意平面，并且也同时可以利用参考姿态信息来优化最终位姿信息，提高了第一位姿的准确度。It can be seen from the above that by obtaining the offset between the reference plane and the preset plane in the world coordinate system, and then adjusting the first reference attitude information based on the offset to obtain the second reference attitude information, it is possible to obtain relative to The reference attitude information of a plane other than the reference plane (such as the preset plane of the world coordinate system), so that the preset plane of the world coordinate system can be located on any plane other than the reference plane, and at the same time, the reference attitude information can be used to optimize the final Pose information, which improves the accuracy of the first pose.
在本公开实施例提供的视觉定位方法中，步骤S333可以通过以下方式实现：In the visual positioning method provided by the embodiment of the present disclosure, step S333 can be implemented in the following manner:
获取至少一个第一候选位姿，基于第二参考姿态、第二位姿以及当前图像帧和其它图像帧之间的第一像素值差异，选择一第一候选位姿作为第一位姿。At least one first candidate pose is obtained, and a first candidate pose is selected as the first pose based on the second reference pose, the second pose, and the first pixel value difference between the current image frame and other image frames.
在一种实施方式中，第一候选位姿可以是当前图像帧在世界坐标系中的位姿信息；示例性的，第一候选位姿可以是基于图像处理算法计算得到的；示例性的，第一候选位姿，也可以通过计算当前图像帧和其它图像帧的相对位姿变化和其它图像帧的第二位姿得到的，还可以直接选择距离当前图像帧最近的具有位姿信息的图像帧的位姿作为第一候选位姿，然后，可以利用迭代优化的方法，生成多个第一候选位姿。In one embodiment, the first candidate pose may be the pose information of the current image frame in the world coordinate system; exemplary, the first candidate pose may be calculated based on an image processing algorithm; exemplary, The first candidate pose can also be obtained by calculating the relative pose changes of the current image frame and other image frames and the second pose of other image frames, and can also directly select the image with pose information closest to the current image frame The pose of the frame is used as the first candidate pose, and then, a plurality of first candidate poses can be generated by using an iterative optimization method.
在此基础上，可以基于每一个第一候选位姿，并基于第二参考姿态、以及当前图像帧与其它图像帧之间的第一像素值差异，得到对应于每一个第一候选位姿的第一像素值差异，然后选择一个第一候 选位姿作为最终位姿。On this basis, based on each first candidate pose, and based on the second reference pose, and the first pixel value difference between the current image frame and other image frames, the corresponding to each first candidate pose can be obtained The first pixel value difference, and then select a first candidate pose as the final pose.
示例性的，当前图像帧与其它图像帧之间的第一像素值差异，可以是当前图像帧上的像素点在其它图像帧上对应的像素点的像素值差异。例如，空间中有三维点A，A在当前图像帧上像素点的为a1，A在其它图像帧上的像素点为a2，那么，a1可以为A在其它图像帧上的点a2对应的像素点。示例性的，还可以利用第二参考姿态与第一候选位姿中的姿态之间的姿态差异来优化第一像素值差异。Exemplarily, the first pixel value difference between the current image frame and other image frames may be a pixel value difference between a pixel on the current image frame and a corresponding pixel on the other image frames. For example, there is a three-dimensional point A in space, the pixel point of A on the current image frame is a1, and the pixel point of A on other image frames is a2, then, a1 can be the pixel corresponding to point a2 of A on other image frames point. Exemplarily, the pose difference between the second reference pose and the pose in the first candidate pose can also be used to optimize the first pixel value difference.
因此，通过利用第二参考姿态、第二位姿、以及当前图像帧和其它图像帧之间的第一像素值差异来选择第一候选位姿，可以得到更加准确的第一候选位姿。Therefore, by using the second reference pose, the second pose, and the first pixel value difference between the current image frame and other image frames to select the first candidate pose, a more accurate first candidate pose can be obtained.
在一种实施方式中，可以利用式(10)来选择一第一候选位姿作为最终位姿。In one embodiment, formula (10) can be used to select a first candidate pose as the final pose.
Figure PCTCN2021121049-appb-000020
Figure PCTCN2021121049-appb-000020
其中，C1为最终的误差信息；
Figure PCTCN2021121049-appb-000021
为第一候选位姿，
Figure PCTCN2021121049-appb-000022
为旋转量信息(也可以称为旋转量或朝向)，
Figure PCTCN2021121049-appb-000023
为平移量信息；
Figure PCTCN2021121049-appb-000024
是第二参考姿
Figure PCTCN2021121049-appb-000025
与第一候选位姿中的姿态
Figure PCTCN2021121049-appb-000026
之间的姿态差异；空间三维点
Figure PCTCN2021121049-appb-000027
是基于第一候选位姿确定与其它图像帧中的特征点对应的空间点；
Figure PCTCN2021121049-appb-000028
为空间三维点
Figure PCTCN2021121049-appb-000029
投影在当前图像帧上的特征点，
Figure PCTCN2021121049-appb-000030
为空间三维点
Figure PCTCN2021121049-appb-000031
在当前图像帧上对应的特征点的像素值；K为电子设备的图像采集装置的内存矩阵；
Figure PCTCN2021121049-appb-000032
为空间三维点
Figure PCTCN2021121049-appb-000033
在其它图像帧上对应的特征点的像素值；
Figure PCTCN2021121049-appb-000034
为第一像素值差异；
Figure PCTCN2021121049-appb-000035
表示对当前图像帧与其它图像帧上对应的特征点计算第一像素值差异求和；α,β为两个约束项的调整参数。可通过实际使用进行比例设定。
Figure PCTCN2021121049-appb-000036
表示利用迭代优化的方法，生成多个第一候选位姿，并从中选择出最终的误差信息C1最小的时候，对应的第一候选位姿。 Among them, C1 is the final error information;
Figure PCTCN2021121049-appb-000021
is the first candidate pose,
Figure PCTCN2021121049-appb-000022
is the rotation amount information (also called rotation amount or orientation),
Figure PCTCN2021121049-appb-000023
is the translation information;
Figure PCTCN2021121049-appb-000024
is the second reference pose
Figure PCTCN2021121049-appb-000025
with the pose in the first candidate pose
Figure PCTCN2021121049-appb-000026
Pose difference between; spatial 3D point
Figure PCTCN2021121049-appb-000027
Determining spatial points corresponding to feature points in other image frames based on the first candidate pose;
Figure PCTCN2021121049-appb-000028
is a three-dimensional point in space
Figure PCTCN2021121049-appb-000029
Feature points projected on the current image frame,
Figure PCTCN2021121049-appb-000030
is a three-dimensional point in space
Figure PCTCN2021121049-appb-000031
The pixel value of the corresponding feature point on the current image frame; K is the memory matrix of the image acquisition device of the electronic equipment;
Figure PCTCN2021121049-appb-000032
is a three-dimensional point in space
Figure PCTCN2021121049-appb-000033
Pixel values of corresponding feature points on other image frames;
Figure PCTCN2021121049-appb-000034
is the first pixel value difference;
Figure PCTCN2021121049-appb-000035
Indicates to calculate the sum of the first pixel value differences between the current image frame and the corresponding feature points on other image frames; α, β are the adjustment parameters of the two constraint items. The ratio can be set by actual use.
Figure PCTCN2021121049-appb-000036
Indicates that multiple first candidate poses are generated using an iterative optimization method, and the corresponding first candidate pose is selected when the final error information C1 is the smallest.
示例性的，
Figure PCTCN2021121049-appb-000037
可以通过式(11)计算得到： Exemplary,
Figure PCTCN2021121049-appb-000037
It can be calculated by formula (11):
Figure PCTCN2021121049-appb-000038
Figure PCTCN2021121049-appb-000038
其中，
Figure PCTCN2021121049-appb-000039
表示相对于预设平面的旋转量信息，δ是式(9)求得偏移量。 in,
Figure PCTCN2021121049-appb-000039
Represents the rotation amount information relative to the preset plane, and δ is the offset obtained by formula (9).
在式(11)中，
Figure PCTCN2021121049-appb-000040
是传感装置检测，例如是通过陀螺仪检测得到的数据，然后结合偏移量δ求得的相对于预设平面的旋转量信息，而
Figure PCTCN2021121049-appb-000041
为通过计算得到的第一候选位姿中的旋转量信息，也是相对于预设平面的旋转量信息，二者理论上应当是相同的。因此，它们可以用来作为优化的第一候选位姿的约束信息。 In formula (11),
Figure PCTCN2021121049-appb-000040
It is the detection of the sensing device, for example, the data obtained by the detection of the gyroscope, and then combined with the offset δ to obtain the rotation amount information relative to the preset plane, and
Figure PCTCN2021121049-appb-000041
The rotation amount information in the first candidate pose obtained through calculation is also the rotation amount information relative to the preset plane, and the two should be the same in theory. Therefore, they can be used as constraint information for the optimized first candidate pose.
由以上可知，通过利用陀螺仪的数据来对第一候选位姿进行约束，可以在对第一候选位姿进行迭代优化时，求得更加准确的第一候选位姿。It can be seen from the above that by using the data of the gyroscope to constrain the first candidate pose, a more accurate first candidate pose can be obtained when iteratively optimizing the first candidate pose.
在一种实施方式中，在求得最终的误差信息以后，可以选择最终的误差信息满足第一预设要求的第二特征点所对应的第一候选位姿，作为最终位姿；示例性的，第一预设要求可以根据需要设置，此处不做限定。In one embodiment, after obtaining the final error information, the first candidate pose corresponding to the second feature point whose final error information meets the first preset requirement can be selected as the final pose; exemplary , the first preset requirement can be set as required, which is not limited here.
在一种实施方式中，通过式(10)计算第一像素值差异和姿态差异，则可以选择满足预设要求的C1对应的第一候选位姿信息作为最终位姿。由此，通过筛选满足预设要求的第一候选位姿，可以获得相对准确的位姿信息。In one embodiment, the first pixel value difference and pose difference are calculated by formula (10), and the first candidate pose information corresponding to C1 that meets the preset requirements can be selected as the final pose. Therefore, relatively accurate pose information can be obtained by screening the first candidate poses that meet the preset requirements.
在本公开实施例提供的视觉定位方法中，获取至少一个第一候选位姿，并基于第二参考姿态、以及第二位姿和当前图像帧和其它图像帧之间的第一像素值差异，选择一第一候选位姿作为第一位姿，可以通过步骤D1至步骤D2实现：In the visual positioning method provided by the embodiments of the present disclosure, at least one first candidate pose is obtained, and based on the second reference pose, and the second pose and the first pixel value difference between the current image frame and other image frames, Selecting a first candidate pose as the first pose can be achieved through steps D1 to D2:
步骤D1：利用第二位姿，确定与其它图像帧中的第五特征点对应的空间点。Step D1: Using the second pose, determine the spatial point corresponding to the fifth feature point in other image frames.
在本公开实施例提供的视觉定位方法中，第五特征点即为前述实施例中的第一特征点。In the visual positioning method provided by the embodiments of the present disclosure, the fifth feature point is the first feature point in the foregoing embodiments.
在一种实施方式中，其它图像帧在世界坐标系中的第二位姿，可以是基于图像配准算法计算得到的，也可以是利用视觉跟踪算法得到的。In an implementation manner, the second poses of other image frames in the world coordinate system may be calculated based on an image registration algorithm, or may be obtained by using a visual tracking algorithm.
在一种实施方式中，在得到第二位姿以后，可以计算第五特征点对应的空间点的深度值，然后就可以计算该空间点的三维坐标，以此可以确定该空间点的位置。由此，可以确定一定数量的其它图像帧中的第五特征点对应的空间点。In one embodiment, after obtaining the second pose, the depth value of the spatial point corresponding to the fifth feature point can be calculated, and then the three-dimensional coordinates of the spatial point can be calculated, so as to determine the position of the spatial point. Thus, the spatial points corresponding to the fifth feature points in a certain number of other image frames can be determined.
步骤D2：基于每个第一候选位姿和空间点从当前图像帧中确定与第一候选位姿对应的第六特征点，基于第一像素值差异以及第二参考姿态与第一候选位姿之间的位姿差异，选择一第一候选位姿作为第一位姿。Step D2: Based on each first candidate pose and space point, determine the sixth feature point corresponding to the first candidate pose from the current image frame, based on the first pixel value difference and the second reference pose and the first candidate pose The difference between the poses, select a first candidate pose as the first pose.
在本公开实施例提供的视觉定位方法中，第六特征点可以为前述实施例中的第二特征点。In the visual positioning method provided in the embodiments of the present disclosure, the sixth feature point may be the second feature point in the foregoing embodiments.
在一种实施方式中，在得到其它图像帧中的第五特征点对应的空间点的三维坐标、其它图像帧在世界坐标系中的第二位姿、以及当前帧图像在世界坐标系中的第一候选位姿以后，可以利用投影的方式，在当前图像帧中确定与空间点对应的第六特征点；示例性的，该第六特征点即为其它图像帧上的第五特征点在当前图像中上对应的点。In one embodiment, after obtaining the three-dimensional coordinates of the space point corresponding to the fifth feature point in other image frames, the second pose of other image frames in the world coordinate system, and the current frame image in the world coordinate system After the first candidate pose, the sixth feature point corresponding to the spatial point can be determined in the current image frame by means of projection; for example, the sixth feature point is the fifth feature point on other image frames in The corresponding point in the current image.
示例性的，可以基于第五特征点和第六特征点得到第一像素值差异，比如基于第五特征点的像素值和第六特征点的像素值得到第一像素值差异。然后，就可以基于第一像素值差异以及第二参考姿态与第一候选位姿之间的姿态差异，选择一第一候选位姿作为最终位姿。具体的计算方法可以参考式(10)。Exemplarily, the first pixel value difference may be obtained based on the fifth feature point and the sixth feature point, for example, the first pixel value difference may be obtained based on the pixel value of the fifth feature point and the pixel value of the sixth feature point. Then, based on the first pixel value difference and the pose difference between the second reference pose and the first candidate pose, a first candidate pose can be selected as the final pose. The specific calculation method can refer to formula (10).
由此，通过利用确定空间中的三维点在其它图像帧中以及在当前图像帧中的对应的点，就可以通过计算像素值的差异方法，求得较为准确的第一候选位姿。Therefore, by using the corresponding points in other image frames and in the current image frame of the three-dimensional point in the determined space, a relatively accurate first candidate pose can be obtained by calculating the difference of pixel values.
在一种实施方式中，第一候选位姿为基于当前图像帧在世界坐标系中的初始位姿确定的。也就是说，可以基于初始位姿，通过迭代优化的方法，得到一系列的第一候选位姿，再从这一系列的第一候选位姿中选择一个最终位姿。In one embodiment, the first candidate pose is determined based on the initial pose of the current image frame in the world coordinate system. That is to say, based on the initial pose, a series of first candidate poses can be obtained through an iterative optimization method, and then a final pose can be selected from the series of first candidate poses.
在一种实施方式中，初始位姿，可以是基于当前图像帧和其它图像帧之间的光度误差确定的。也就是说，可以利用光度误差方程，结合迭代优化的方法，求得初始位姿。In an implementation manner, the initial pose may be determined based on photometric errors between the current image frame and other image frames. That is to say, the photometric error equation can be used in combination with the method of iterative optimization to obtain the initial pose.
在一种实施方式中，可以执行以下方式得到初始位姿。In one embodiment, the initial pose can be obtained in the following manner.
获取至少一个第二候选位姿，并基于当前图像帧和其它图像帧之间的第二像素值差异，选择一第二候选位姿作为初始位姿。At least one second candidate pose is acquired, and based on the second pixel value difference between the current image frame and other image frames, a second candidate pose is selected as the initial pose.
示例性的，第二候选位姿，可以是其它图像帧相对于世界坐标系的位姿信息；第二候选位姿可以是基于图像处理算法计算得到的，其数量可以为若干个；也可以直接选择距离当前图像帧最近的具有位姿信息的图像帧的位姿作为一个第二候选位姿；或者直接将第二位姿作为第二候选位姿。示例性的，可以利用迭代优化的方法，生成多个第二候选位姿。示例性的，可以以第二位姿为基础，利用迭代优化的方法，生成多个第二候选位姿。在此基础上，可以基于每一个第二候选位姿，并基于当前图像帧和其它图像帧之间的第二像素值差异，然后选择一个第二候选位姿作为初始位姿。示例性的，当前图像帧和其它图像帧之间的第二像素值差异，可以是当前图像帧上的像素点在其它图像帧上对应的像素点的像素值差异。例如。空间中有三维点B，B在当前图像帧上的像素点为B1，B在其它图像帧上的像素点为B2，B1为B在其它图像帧上点B2对应的像素点。Exemplarily, the second candidate pose can be the pose information of other image frames relative to the world coordinate system; the second candidate pose can be calculated based on an image processing algorithm, and its number can be several; it can also be directly Selecting the pose of the image frame with pose information closest to the current image frame as a second candidate pose; or directly using the second pose as the second candidate pose. Exemplarily, an iterative optimization method may be used to generate multiple second candidate poses. Exemplarily, multiple second candidate poses may be generated by using an iterative optimization method based on the second pose. On this basis, based on each second candidate pose and based on the second pixel value difference between the current image frame and other image frames, a second candidate pose can be selected as the initial pose. Exemplarily, the second pixel value difference between the current image frame and other image frames may be a pixel value difference between a pixel on the current image frame and a corresponding pixel on the other image frames. E.g. There is a three-dimensional point B in space, the pixel point of B on the current image frame is B1, the pixel point of B on other image frames is B2, and B1 is the pixel point corresponding to point B2 of B on other image frames.
在一种实施方式中，可以通过式(12)来选择一个第二候选位姿作为当前图像帧与其它图像帧之 间的初始变化位姿。In one embodiment, formula (12) can be used to select a second candidate pose as the initial change pose between the current image frame and other image frames.
Figure PCTCN2021121049-appb-000042
Figure PCTCN2021121049-appb-000042
在式(10)中，C2为第二像素值差异；
Figure PCTCN2021121049-appb-000043
为第二候选位姿，
Figure PCTCN2021121049-appb-000044
为旋转量信息；
Figure PCTCN2021121049-appb-000045
为平移量信息；空间三维点
Figure PCTCN2021121049-appb-000046
是基于第二候选位姿确定与其它图像帧中的第五特征点对应的空间点；
Figure PCTCN2021121049-appb-000047
为空间三维点
Figure PCTCN2021121049-appb-000048
投影在当前图像帧上的第六特征点；
Figure PCTCN2021121049-appb-000049
为当前图像帧上第六特征点的像素值；K为电子设备的图像采集装置的内存矩阵；
Figure PCTCN2021121049-appb-000050
为其它图像帧上对应的第五特征点的像素值；
Figure PCTCN2021121049-appb-000051
为第二像素值差异；
Figure PCTCN2021121049-appb-000052
表示对当前图像帧与其它图像帧上存在对应的点(第五特征点和第六特征点)计算第二像素值差异并求和；
Figure PCTCN2021121049-appb-000053
表示利用迭代优化的方法，生成多个第二候选位姿，并从中选择出第二像素值差异C2最小时对应的第二候选位姿，作为初始位姿。 In formula (10), C2 is the second pixel value difference;
Figure PCTCN2021121049-appb-000043
is the second candidate pose,
Figure PCTCN2021121049-appb-000044
is the rotation amount information;
Figure PCTCN2021121049-appb-000045
is translation information; three-dimensional point in space
Figure PCTCN2021121049-appb-000046
Determining a spatial point corresponding to the fifth feature point in other image frames based on the second candidate pose;
Figure PCTCN2021121049-appb-000047
is a three-dimensional point in space
Figure PCTCN2021121049-appb-000048
The sixth feature point projected on the current image frame;
Figure PCTCN2021121049-appb-000049
Be the pixel value of the sixth feature point on the current image frame; K is the memory matrix of the image acquisition device of the electronic equipment;
Figure PCTCN2021121049-appb-000050
Be the pixel value of the fifth feature point corresponding to other image frames;
Figure PCTCN2021121049-appb-000051
is the second pixel value difference;
Figure PCTCN2021121049-appb-000052
Indicates that there are corresponding points (the fifth feature point and the sixth feature point) on the current image frame and other image frames to calculate and sum the second pixel value difference;
Figure PCTCN2021121049-appb-000053
Indicates that the iterative optimization method is used to generate a plurality of second candidate poses, and the second candidate pose corresponding to the smallest second pixel value difference C2 is selected as the initial pose.
在一种实施方式中，在得到第二像素值差异之后，还可以选择第二像素值差异满足第二预设要求的第二特征点所对应的第二候选位姿，作为初始变化位姿。示例性的，第二预设要求可以根据需要设置，此处不做限定。示例性的，若通过式(12)计算第二像素值差异，则是选择满足预设要求的C2对应的第二候选位姿信息作为初始位姿。由此，通过筛选满足预设要求的第二候选位姿，可以获得相对准确的位姿信息。In one embodiment, after obtaining the second pixel value difference, a second candidate pose corresponding to a second feature point whose second pixel value difference satisfies the second preset requirement may also be selected as the initial change pose. Exemplarily, the second preset requirement can be set as required, which is not limited here. Exemplarily, if the second pixel value difference is calculated by formula (12), the second candidate pose information corresponding to C2 that meets the preset requirements is selected as the initial pose. Therefore, relatively accurate pose information can be obtained by screening the second candidate poses that meet the preset requirements.
由此，通过计算第二像素值的差异求得符合要求的初始位姿；然后再基于初始位姿，结合检测数据(第二参考姿态)和光度误差来求最终的误差信息，继而能够得到符合要求的最终位姿。通过利用第二参考姿态的校正，可以得到准确度更高的最终位姿。Therefore, the initial pose that meets the requirements is obtained by calculating the difference of the second pixel value; then based on the initial pose, combined with the detection data (second reference pose) and the photometric error to obtain the final error information, and then can obtain the conforming The final pose required. By using the correction of the second reference pose, a final pose with higher accuracy can be obtained.
图7是本公开实施例提供的视觉定位方法的第三流程示意图。如图7所示，该方法可以包括步骤S51至步骤S55：Fig. 7 is a schematic flowchart of a third visual positioning method provided by an embodiment of the present disclosure. As shown in Figure 7, the method may include steps S51 to S55:
步骤S51：初始化：进行图像配准，获取初始图像位姿。Step S51: Initialization: Perform image registration to obtain the initial image pose.
示例性的，图像配准是用利用电子设备拍摄到的当前图像帧与目标图像进行图像配准检测，若检测成功，则可以获得当前图像帧在基于目标图像建立的世界坐标系中的初始图像位姿，即电子设备在世界坐标系中的位姿。Exemplarily, the image registration is to use the current image frame captured by the electronic device and the target image to perform image registration detection. If the detection is successful, the initial image of the current image frame in the world coordinate system established based on the target image can be obtained Pose, that is, the pose of the electronic device in the world coordinate system.
示例性的，还可以判断当前图像帧与目标图像之间的变换参数是否满足预设要求，若变换参数满足预设要求，则可以认为在当前图像帧检测到目标图像，图像配准检测成功，可以获取初始图像位姿；若变换参数不满足预设要求，则可以认为在当前图像帧中未检测到目标图像，即图像配准检测失败。Exemplarily, it can also be judged whether the transformation parameters between the current image frame and the target image meet the preset requirements. If the transformation parameters meet the preset requirements, it can be considered that the target image is detected in the current image frame, and the image registration detection is successful. The initial image pose can be obtained; if the transformation parameters do not meet the preset requirements, it can be considered that the target image is not detected in the current image frame, that is, the image registration detection fails.
示例性的，若图像配准检测成功，则重复执行步骤S51，直至获得初始图像位姿为止。Exemplarily, if the image registration detection is successful, step S51 is repeatedly executed until the initial image pose is obtained.
步骤S52：利用图像配准方法，获得第二图像帧与第一图像帧对应的图像变换参数。Step S52: Obtain image transformation parameters corresponding to the second image frame and the first image frame by using an image registration method.
示例性的，可以将获得初始图像位姿时对应的图像帧定义为第一图像帧；在获得初始图像位姿以后，电子设备还可以获取第二图像帧。Exemplarily, the corresponding image frame when the initial image pose is obtained may be defined as the first image frame; after the initial image pose is obtained, the electronic device may also obtain the second image frame.
示例性的，若第一图像帧是利用图像配准方法得到的初始位姿，可以确定此时电子设备处于图像跟踪状态。此时，设备会执行前述实施例提及的图像配准方法，以此可以获得与第二图像帧与第一图像帧对应的图像变换参数。Exemplarily, if the first image frame is an initial pose obtained by using an image registration method, it may be determined that the electronic device is in an image tracking state at this time. At this point, the device will execute the image registration method mentioned in the foregoing embodiments, so as to obtain image transformation parameters corresponding to the second image frame and the first image frame.
步骤S53：判断图像变换参数是否满足预设要求。Step S53: judging whether the image transformation parameters meet the preset requirements.
示例性的，图像变换参数，可以包括第一变换参数；判断图像变换参数是否满足预设要求的方法，可以参照前述实施例，此处不再赘述。Exemplarily, the image transformation parameters may include the first transformation parameters; for the method of judging whether the image transformation parameters meet the preset requirements, reference may be made to the foregoing embodiments, and details are not repeated here.
若图像变换参数满足预设要求，则执行步骤S54；若图像变换参数不满足预设要求，则执行步骤S55。If the image transformation parameters meet the preset requirements, execute step S54; if the image transformation parameters do not meet the preset requirements, execute step S55.
步骤S54：进入图像跟踪状态，利用图像变换参数获取第二位姿。Step S54: Enter the image tracking state, and use the image transformation parameters to obtain the second pose.
示例性的，若图像变换参数满足预设要求，则可以确认电子设备当前处于图像跟踪状态，此时可以利用该图像变换参数获得第二图像帧对应的第二位姿。Exemplarily, if the image transformation parameters meet the preset requirements, it can be confirmed that the electronic device is currently in the image tracking state, and at this time, the image transformation parameters can be used to obtain the second pose corresponding to the second image frame.
步骤S55：进入视觉导航状态，利用其它图像在世界坐标系中的第二位姿、以及当前图像帧与其它图像帧之间的光度误差，确定第一位姿。Step S55: Enter the visual navigation state, and use the second pose of other images in the world coordinate system and the photometric error between the current image frame and other image frames to determine the first pose.
示例性的，若图像变换参数不满足预设要求，则可以确认电子设备进入视觉导航状态，并执行“利用其它图像在世界坐标系中的第二位姿、以及当前图像帧与其它图像帧之间的光度误差，确定第一位姿”步骤。关于本步骤的具体描述，请参照前述实施例，此处不再赘述。Exemplarily, if the image transformation parameters do not meet the preset requirements, it can be confirmed that the electronic device enters the visual navigation state, and executes "use the second pose of other images in the world coordinate system, and the current image frame and other image frames The photometric error between, determine the first pose" step. For the specific description of this step, please refer to the foregoing embodiments, and details will not be repeated here.
在后续的过程中，若执行了步骤S54，则会重新执行步骤S52获得第三图像帧与第二图像帧的图像变换参数，并继续执行后续步骤。若执行了步骤S55，则会重复执行步骤S55，设备持续处于视觉导航状态。In the subsequent process, if step S54 is executed, step S52 will be re-executed to obtain the image transformation parameters of the third image frame and the second image frame, and the subsequent steps will continue to be executed. If step S55 is executed, step S55 will be executed repeatedly, and the device will continue to be in the visual navigation state.
在一种实施方式中，当电子设备重新执行了步骤S51时，则可以继续执行后续的步骤。In one implementation manner, when the electronic device re-executes step S51, it may continue to perform subsequent steps.
图8是本公开实施例提供的图像配准装置的结构示意图。图像配准装置80包括图像获取模块81、第一参数获取模块82和第二参数获取模块83。其中：Fig. 8 is a schematic structural diagram of an image registration device provided by an embodiment of the present disclosure. The image registration device 80 includes an image acquisition module 81 , a first parameter acquisition module 82 and a second parameter acquisition module 83 . in:
图像获取模块81，配置为获取当前图像帧；An image acquisition module 81 configured to acquire the current image frame;
第一参数获取模块82，配置为基于当前图像帧与其它图像帧中的目标图像信息，确定当前图像帧与其它图像帧之间的第一变换参数，其中，目标图像信息为关于目标图像的图像信息；第一变换参数，包括当前图像帧与其它图像帧之间的单应性矩阵。The first parameter acquisition module 82 is configured to determine a first transformation parameter between the current image frame and other image frames based on target image information in the current image frame and other image frames, wherein the target image information is an image about the target image Information: the first transformation parameter, including the homography matrix between the current image frame and other image frames.
第二参数获取模块83，配置为基于第一变换参数、以及其它图像帧与目标图像之间的第二变换参数，得到当前图像帧与目标图像之间的第三变换参数；其中，第二变换参数，包括当前图像帧与目标图像之间的单应性矩阵。The second parameter acquisition module 83 is configured to obtain a third transformation parameter between the current image frame and the target image based on the first transformation parameter and the second transformation parameters between other image frames and the target image; wherein, the second transformation parameters, including the homography matrix between the current image frame and the target image.
在一种实施方式中，第一参数获取模块82，配置为从其它图像帧中查找出关于目标图像的至少一个第一特征点；在从当前图像帧中查找出关于目标图像的至少一个第二特征点；基于第一特征点和第二特征点，确定第一变换参数。In one embodiment, the first parameter acquisition module 82 is configured to find out at least one first feature point about the target image from other image frames; after finding out at least one second feature point about the target image from the current image frame A feature point; based on the first feature point and the second feature point, determine a first transformation parameter.
在一种实施方式中，第一参数获取模块82，配置为基于第二变换参数确定目标图像在其它图像帧中的目标区域；从目标区域中提取至少一个第一特征点；In one embodiment, the first parameter acquisition module 82 is configured to determine the target area of the target image in other image frames based on the second transformation parameter; extract at least one first feature point from the target area;
第一参数获取模块82，还配置为分别对至少一个第一特征点进行跟踪，得到当前图像帧中关于目标图像的至少一个第二特征点。The first parameter acquiring module 82 is further configured to respectively track at least one first feature point to obtain at least one second feature point related to the target image in the current image frame.
在一种实施方式中，第二参数获取模块83，配置为将第一变换参数与第二变换参数的乘积，作为第三变换参数。In one embodiment, the second parameter obtaining module 83 is configured to use the product of the first transformation parameter and the second transformation parameter as the third transformation parameter.
图像配准装置80还包括优化模块，在第二参数获取模块，配置为基于第一变换参数、以及其它图像帧与目标图像之间的第二变换参数，得到当前图像帧与目标图像之间的第三变换参数之后，优化模块，配置为利用预设优化方式，对第三变换参数进行优化。The image registration device 80 also includes an optimization module. The second parameter acquisition module is configured to obtain the current image frame and the target image based on the first transformation parameter and the second transformation parameter between other image frames and the target image. After the third transformation parameter, the optimization module is configured to optimize the third transformation parameter by using a preset optimization method.
图9是本公开实施提供的视觉定位装置90的结构示意图。视觉定位装置90包括参数获取模块91和第一位姿获取模块92，其中：FIG. 9 is a schematic structural diagram of a visual positioning device 90 provided by the implementation of the present disclosure. The visual positioning device 90 includes a parameter acquisition module 91 and a first pose acquisition module 92, wherein:
参数获取模块91，配置为获取当前图像帧与目标图像之间的当前变换参数；其中，当前变换参数为如前述实施例提供的图像配准方法获得的第三变换参数；The parameter acquisition module 91 is configured to acquire the current transformation parameter between the current image frame and the target image; wherein, the current transformation parameter is the third transformation parameter obtained by the image registration method provided in the foregoing embodiment;
第一位姿获取模块92，配置为利用当前变换参数，得到当前图像帧在世界坐标系中的第一位姿，其中，世界坐标系是基于目标图像所在的平面建立的。The first pose obtaining module 92 is configured to use the current transformation parameters to obtain the first pose of the current image frame in the world coordinate system, wherein the world coordinate system is established based on the plane where the target image is located.
在一种实施方式中，视觉定位装置90还包括判断模块和第二位姿获取模块，其中：In one embodiment, the visual positioning device 90 further includes a judgment module and a second pose acquisition module, wherein:
第一位姿获取模块，配置为利用当前变换参数，得到当前图像帧在世界坐标系中的第一位姿之前，判断模块，配置为判断当前变换参数满足是否预设要求。The first pose acquisition module is configured to use the current transformation parameters to obtain the first pose of the current image frame in the world coordinate system, and the judging module is configured to judge whether the current transformation parameters meet the preset requirements.
第一位姿获取模块，还配置为响应于当前变换参数满足预设要求，利用当前变换参数，得到当前 图像帧在世界坐标系中的第一位姿。The first pose acquisition module is further configured to obtain the first pose of the current image frame in the world coordinate system by using the current transformation parameters in response to the current transformation parameters meeting the preset requirements.
第二位姿获取模块，配置为响应于当前变换参数不满足预设要求，利用其它图像在世界坐标系中的第二位姿、以及当前图像帧与其它图像帧之间的光度误差，确定第一位姿。The second pose acquisition module is configured to determine the second pose by using the second pose of other images in the world coordinate system and the photometric error between the current image frame and other image frames in response to the fact that the current transformation parameters do not meet the preset requirements. a pose.
在一种实施方式中，视觉定位装置90还包括状态确定模块，状态确定模块，配置为在参数获取模块，配置为获取当前图像帧与目标图像之间的当前变换参数之前，响应于上一图像帧的位姿获取方式为图像跟踪方式，执行获取当前图像帧与目标图像之间的当前变换参数，其中，图像跟踪方式为利用上一图像帧与目标图像之间的变换参数确定上一图像帧在世界坐标系中的位姿。In one embodiment, the visual positioning device 90 further includes a state determination module, the state determination module is configured to respond to the previous image before the parameter acquisition module is configured to acquire the current transformation parameters between the current image frame and the target image The pose acquisition method of the frame is the image tracking method, which executes to obtain the current transformation parameters between the current image frame and the target image, where the image tracking method is to use the transformation parameters between the previous image frame and the target image to determine the previous image frame pose in the world coordinate system.
其中，上述的其它图像帧和当前图像帧是由设备的图像采集装置先后拍摄得到。Wherein, the above-mentioned other image frames and the current image frame are sequentially captured by the image acquisition device of the device.
第二位姿获取模块，配置为获取第一参考姿态，对第一参考姿态进行调整，得到第二参考姿态；基于第二参考姿态、第二位姿和当前图像帧与历史图像帧之间的光度误差，确定第一位姿；其中，第一参考姿态是图像采集装置对应于当前图像帧的拍摄时刻且相对于参考平面的姿态；利用参考平面与世界坐标系中的预设平面之间的偏移量，。The second pose acquisition module is configured to obtain the first reference pose, and adjust the first reference pose to obtain the second reference pose; based on the second reference pose, the second pose, and the relationship between the current image frame and the historical image frame The photometric error determines the first pose; wherein, the first reference pose is the pose of the image acquisition device corresponding to the shooting moment of the current image frame and relative to the reference plane; using the reference plane and the preset plane in the world coordinate system. Offset,.
在一种实施方式中，视觉定位装置90还包括偏移量获取模块，配置为在利用参考平面与世界坐标系中的预设平面之间的偏移量，对第一参考姿态进行调整，得到第二参考姿态之前，获取第一历史图像帧在世界坐标系中的第三位姿，以及获取第三参考姿态，利用第三位姿中的姿态和第三参考姿态，得到偏移量，其中，第三参考姿态是图像采集装置对应于第一历史图像帧的拍摄时刻且相对于参考平面的姿态，第三位姿是基于目标图像确定的，预设平面为目标图像所在的平面；第三位姿中的姿态是相对于预设平面的姿态。In one embodiment, the visual positioning device 90 further includes an offset acquisition module configured to adjust the first reference pose by using the offset between the reference plane and the preset plane in the world coordinate system to obtain Before the second reference pose, obtain the third pose of the first historical image frame in the world coordinate system, and obtain the third reference pose, and use the pose in the third pose and the third reference pose to obtain the offset, where , the third reference pose is the pose of the image acquisition device corresponding to the shooting moment of the first historical image frame and relative to the reference plane, the third pose is determined based on the target image, and the preset plane is the plane where the target image is located; the third A pose in a pose is a pose relative to a preset plane.
在一种实施方式中，第二位姿获取模块，配置为获取至少一个第一候选位姿，基于第二参考姿态、第二位姿和当前图像帧和其它图像帧之间的第一像素值差异，选择一第一候选位姿作为第一位姿。In one embodiment, the second pose acquisition module is configured to acquire at least one first candidate pose based on the second reference pose, the second pose and the first pixel value between the current image frame and other image frames difference, select a first candidate pose as the first pose.
其中，上述的第一候选位姿是基于当前图像帧在世界坐标系中的初始位姿确定的，初始位姿是基于当前图像帧和其它图像帧之间的光度误差确定的；和/或，上述的第二位姿获取模块，配置为利用第二位姿，确定与其它图像帧中的第一特征点对应的空间点；基于每个第一候选位姿和空间点从当前图像帧中确定与第一候选位姿对应的第二特征点，获取第一特征点与第二特征点的第一像素值差异，并基于第一像素值差异以及第二参考姿态与第一候选位姿之间的位姿差异，选择一第一候选位姿作为第一位姿。Wherein, the above-mentioned first candidate pose is determined based on the initial pose of the current image frame in the world coordinate system, and the initial pose is determined based on the photometric error between the current image frame and other image frames; and/or, The above-mentioned second pose acquisition module is configured to use the second pose to determine the spatial points corresponding to the first feature points in other image frames; determine from the current image frame based on each first candidate pose and the spatial point The second feature point corresponding to the first candidate pose, obtain the first pixel value difference between the first feature point and the second feature point, and based on the first pixel value difference and the difference between the second reference pose and the first candidate pose The pose difference of , select a first candidate pose as the first pose.
在一种实施方式中，视觉定位装置90还包括历史图像帧位姿获取模块，配置为基于第一历史图像帧与目标图像之间的第一匹配点对，确定第一历史图像帧与目标图像之间的第四变换参数，利用第四变换参数得到第二位姿；或者，基于第一历史图像帧与第二历史图像帧之间的第二匹配点对，确定第一历史图像帧与第二历史图像帧之间的第五变换参数，利用第五变换参数和第二历史图像帧与目标图像之间的第六变换参数得到第四变换参数，利用第四变换参数得到第二位姿，其中，第二历史图像帧位于第一历史图像帧之前。In one embodiment, the visual positioning device 90 further includes a historical image frame pose acquisition module configured to determine the first historical image frame and the target image based on the first matching point pair between the first historical image frame and the target image Between the fourth transformation parameter, use the fourth transformation parameter to obtain the second pose; or, based on the second matching point pair between the first historical image frame and the second historical image frame, determine the first historical image frame and the second historical image frame The fifth transformation parameter between the two historical image frames, using the fifth transformation parameter and the sixth transformation parameter between the second historical image frame and the target image to obtain the fourth transformation parameter, using the fourth transformation parameter to obtain the second pose, Wherein, the second historical image frame is located before the first historical image frame.
在一种实施方式中，历史图像帧位姿获取模块，配置为利用第四变换参数得到第二位姿之前，判断第四变换参数是否满足预设要求；响应于第四变换参数满足预设要求，历史图像帧位姿获取模块，配置为确定处于图像跟踪状态，并执行利用第四变换参数得到第二位姿。In one embodiment, the historical image frame pose acquisition module is configured to determine whether the fourth transformation parameter meets the preset requirement before using the fourth transformation parameter to obtain the second pose; in response to the fourth transformation parameter meeting the preset requirement , a historical image frame pose acquisition module, configured to determine that it is in an image tracking state, and execute using a fourth transformation parameter to obtain a second pose.
图10是本公开实施例提供的电子设备的结构示意图。电子设备100包括相互耦接的存储器101和处理器102，处理器102用于执行存储器101中存储的程序指令，以实现上述任一图像配准方法实施例的步骤，或者是上述任一觉定位方法实施例的步骤。Fig. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. The electronic device 100 includes a memory 101 and a processor 102 coupled to each other, and the processor 102 is configured to execute program instructions stored in the memory 101, so as to realize the steps of any of the above-mentioned image registration method embodiments, or any of the above-mentioned image registration methods. Steps of a method embodiment.
示例性的，电子设备100可以包括但不限于：微型计算机、服务器，此外，电子设备100还可以包括笔记本电脑、平板电脑等移动设备，在此不做限定。Exemplarily, the electronic device 100 may include, but not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include mobile devices such as notebook computers and tablet computers, which are not limited herein.
示例性的，处理器102用于控制其自身以及存储器101以实现上述任一图像配准方法实施例的步骤，或者是上述任一觉定位方法实施例的步骤。处理器102还可以为CPU、通用处理器、DSP、ASIC、FPGA或者其它可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。另外，处理器102可以由集成电路芯片共同实现。Exemplarily, the processor 102 is configured to control itself and the memory 101 to implement the steps of any of the above embodiments of the image registration method, or the steps of any of the above embodiments of the visual positioning method. The processor 102 may also be a CPU, a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. In addition, the processor 102 may be jointly implemented by integrated circuit chips.
图11为本公开实施例提供的计算机可读存储介质的结构示意图。如图11所示，计算机可读存储 介质111存储有能够被处理器运行的程序指令1111，程序指令1111用于实现上述任一图像配准方法实施例的步骤，或者是上述任一觉定位方法实施例的步骤。Fig. 11 is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present disclosure. As shown in FIG. 11 , the computer-readable storage medium 111 stores program instructions 1111 that can be executed by the processor, and the program instructions 1111 are used to implement the steps of any of the above-mentioned image registration method embodiments, or any of the above-mentioned visual positioning methods Example steps.
本公开实施例还提供了一种计算机程序，该计算机程序包括计算机可读代码，在该计算机可读代码在电子设备中运行的情况下，电子设备的处理器，配置为实现前述实施例所述的图像配准方法、以及实现前述实施例所述的视觉定位方法。An embodiment of the present disclosure also provides a computer program, where the computer program includes computer readable code. When the computer readable code runs in an electronic device, the processor of the electronic device is configured to implement the computer program described in the foregoing embodiments. The image registration method and the visual positioning method described in the foregoing embodiments are implemented.
在一些实施例中，本公开实施例提供的装置具有的功能或包含的模块可以用于执行上文方法实施例描述的方法，其具体实现可以参照上文方法实施例的描述，为了简洁，这里不再赘述。In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the method embodiments above, and its specific implementation can refer to the description of the method embodiments above. For brevity, here No longer.
上文对各个实施例的描述倾向于强调各个实施例之间的不同之处，其相同或相似之处可以互相参考，为了简洁，本文不再赘述。The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, the same or similar points can be referred to each other, and for the sake of brevity, details are not repeated herein.
在本公开所提供的几个实施例中，应该理解到，所揭露的方法和装置，可以通过其它的方式实现。例如，以上所描述的装置实施方式仅仅是示意性的，例如，模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如单元或组件可以结合或者可以集成到另一个***，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性、机械或其它的形式。In the several embodiments provided in the present disclosure, it should be understood that the disclosed methods and devices may be implemented in other ways. For example, the device implementations described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
本公开实施例中涉及的电子设备可以是***、方法和计算机程序产品中的至少之一。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本公开的各个方面的计算机可读程序指令。The electronic device involved in the embodiments of the present disclosure may be at least one of a system, a method, and a computer program product. A computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present disclosure.
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是但不限于电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(Random Access Memory，RAM)、只读存储器(Read-Only Memory，ROM)、可擦除可编程只读存储器(Electrical Programmable Read Only Memory，EPROM)或闪存、静态随机存取存储器(Static Random-Access Memory，SRAM)、便携式压缩盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、数字多功能盘(Digital Video Disc，DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Examples of computer-readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, Random Access Memory (RAM), Read-Only Memory (ROM), erasable Electrical Programmable Read Only Memory (EPROM) or flash memory, Static Random-Access Memory (Static Random-Access Memory, SRAM), Portable Compact Disc Read-Only Memory (CD-ROM), Digital Video Discs (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or raised structures in grooves with instructions stored thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和无线网中的至少之一下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和边缘服务器中的至少之一。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device over at least one of a network, such as the Internet, a local area network, a wide area network, and a wireless network. . The network may include at least one of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(Industry Standard Architecture，ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言，诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络，包括局域网(Local Area Network，LAN)或广域网(Wide Area Network，WAN)连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、FPGA或可编程逻辑阵列(Programmable Logic Arrays，PLA)，该电子电路可以执行计算机可读程序指令，从而实现本公开的各个方面。Computer program instructions for performing the operations of the present disclosure may be assembly instructions, Industry Standard Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the “C” language or similar programming languages. Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In cases involving a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it may be connected to an external computer (for example, using Internet Service Provider to connect via the Internet). In some embodiments, electronic circuits, such as programmable logic circuits, FPGAs, or programmable logic arrays (Programmable Logic Arrays, PLAs), can be customized by using state information of computer-readable program instructions, which can execute computer-readable Read program instructions, thereby implementing various aspects of the present disclosure.
作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到网络单元上。可以根据实际的需要选 择其中的部分或者全部单元来实现本实施方式方案的目的。A unit described as a separate component may or may not be physically separated, and a component shown as a unit may or may not be a physical unit, that is, it may be located in one place, or may also be distributed to network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
另外，在本公开各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本公开各个实施方式方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or part of the contribution to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods in various embodiments of the present disclosure. The aforementioned storage medium includes: various media capable of storing program codes such as U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk.
工业实用性Industrial Applicability
本公开实施例提供了一种图像配准方法、视觉定位方法、装置、设备、介质及程序，其中，所述图像配准方法由电子设备执行，所述方法包括：获取当前图像帧；基于当前图像帧与其它图像帧中的目标图像信息，确定当前图像帧与其它图像帧之间的第一变换参数；其中，目标图像信息为关于目标图像的图像信息；所述第一变换参数，包括所述当前图像帧与所述其它图像帧之间的单应性矩阵；基于第一变换参数、以及其它图像帧与目标图像之间的第二变换参数，得到当前图像帧与目标图像之间的第三变换参数；所述第二变换参数，包括所述当前图像帧与所述目标图像之间的单应性矩阵。通过该方法，能够提高配准速度以及图像配准的准确度。Embodiments of the present disclosure provide an image registration method, a visual positioning method, a device, a device, a medium, and a program, wherein the image registration method is performed by an electronic device, and the method includes: acquiring a current image frame; The target image information in the image frame and other image frames determines the first transformation parameters between the current image frame and other image frames; wherein, the target image information is image information about the target image; the first transformation parameters include the The homography matrix between the current image frame and the other image frames; based on the first transformation parameter and the second transformation parameter between other image frames and the target image, the second transformation parameter between the current image frame and the target image is obtained Three transformation parameters: the second transformation parameter includes a homography matrix between the current image frame and the target image. Through this method, the registration speed and the accuracy of image registration can be improved.

Claims (29)

  1. 一种图像配准方法，所述方法由电子设备执行，所述方法包括：An image registration method, the method is performed by an electronic device, the method comprising:
    获取当前图像帧；Get the current image frame;
    基于所述当前图像帧与其它图像帧中的目标图像信息，确定所述当前图像帧与所述其它图像帧之间的第一变换参数；其中，所述目标图像信息为关于目标图像的图像信息；所述第一变换参数，包括所述当前图像帧与所述其它图像帧之间的单应性矩阵；Based on target image information in the current image frame and other image frames, determine a first transformation parameter between the current image frame and the other image frames; wherein the target image information is image information about a target image ; The first transformation parameter includes a homography matrix between the current image frame and the other image frames;
    基于所述第一变换参数、以及所述其它图像帧与所述目标图像之间的第二变换参数，得到所述当前图像帧与所述目标图像之间的第三变换参数；其中，所述第二变换参数，包括所述当前图像帧与所述目标图像之间的单应性矩阵。Based on the first transformation parameter and the second transformation parameter between the other image frame and the target image, a third transformation parameter between the current image frame and the target image is obtained; wherein, the The second transformation parameter includes a homography matrix between the current image frame and the target image.
  2. 根据权利要求1所述的方法，其中，所述基于所述当前图像帧与其它图像帧中的目标图像信息，确定所述当前图像帧与所述其它图像帧之间的第一变换参数，包括：The method according to claim 1, wherein said determining the first transformation parameter between the current image frame and the other image frames based on the target image information in the current image frame and other image frames comprises :
    从所述其它图像帧中查找出关于所述目标图像的至少一个第一特征点；finding at least one first feature point on the target image from the other image frames;
    从所述当前图像帧中查找出关于所述目标图像的至少一个第二特征点；finding at least one second feature point on the target image from the current image frame;
    基于所述第一特征点和第二特征点，确定所述第一变换参数。Based on the first feature point and the second feature point, the first transformation parameter is determined.
  3. 根据权利要求2所述的方法，其中，所述从所述其它图像帧中查找出关于所述目标图像的至少一个第一特征点，包括：The method according to claim 2, wherein said finding out at least one first feature point about said target image from said other image frames comprises:
    基于所述第二变换参数确定所述目标图像在所述其它图像帧中的目标区域；determining a target area of the target image in the other image frame based on the second transformation parameter;
    从所述目标区域中提取至少一个所述第一特征点；extracting at least one of the first feature points from the target area;
    所述从所述当前图像帧中查找出关于所述目标图像的至少一个第二特征点，包括：The finding at least one second feature point about the target image from the current image frame includes:
    分别对所述至少一个第一特征点进行跟踪，得到所述当前图像帧中关于所述目标图像的至少一个第二特征点。Tracking the at least one first feature point respectively to obtain at least one second feature point in the current image frame related to the target image.
  4. 根据权利要求1至3任一所述的方法，其中，所述基于所述第一变换参数、以及所述其它图像帧与所述目标图像之间的第二变换参数，得到所述当前图像帧与所述目标图像帧之间的第三变换参数，包括：The method according to any one of claims 1 to 3, wherein said current image frame is obtained based on said first transformation parameter and a second transformation parameter between said other image frame and said target image The third transformation parameter between the target image frame includes:
    将所述第一变换参数与所述第二变换参数的乘积，作为所述第三变换参数；using the product of the first transformation parameter and the second transformation parameter as the third transformation parameter;
    和/或，and / or,
    在所述基于所述第一变换参数、以及所述其它图像帧与所述目标图像之间的第二变换参数，得到所述当前图像帧与所述目标图像帧之间的第三变换参数之后，所述方法还包括：After the third transformation parameter between the current image frame and the target image frame is obtained based on the first transformation parameter and the second transformation parameter between the other image frame and the target image , the method also includes:
    利用预设优化方式，对所述第三变换参数进行优化。The third transformation parameter is optimized by using a preset optimization manner.
  5. 一种视觉定位方法，所述方法由电子设备执行，所述方法包括：A visual positioning method, the method is performed by an electronic device, and the method includes:
    获取当前图像帧与目标图像之间的当前变换参数；其中，所述当前变换参数为利用权利要求1至4任一所述的图像配准方法获得的第三变换参数；Obtaining a current transformation parameter between the current image frame and the target image; wherein, the current transformation parameter is a third transformation parameter obtained by using the image registration method described in any one of claims 1 to 4;
    利用所述当前变换参数，得到所述当前图像帧在世界坐标系中的第一位姿；其中，所述世界坐标系是基于所述目标图像所在的平面建立的。Using the current transformation parameters, obtain the first pose of the current image frame in the world coordinate system; wherein the world coordinate system is established based on the plane where the target image is located.
  6. 根据权利要求5所述的方法，其中，在所述利用所述当前变换参数，得到所述当前图像帧在世界坐标系中的第一位姿之前，所述方法还包括：The method according to claim 5, wherein, before obtaining the first pose of the current image frame in the world coordinate system by using the current transformation parameters, the method further comprises:
    响应于所述当前变换参数满足预设要求，执行所述利用所述当前变换参数，得到所述当前图像帧在所述世界坐标系中的第一位姿；In response to the current transformation parameters satisfying the preset requirements, performing the use of the current transformation parameters to obtain a first pose of the current image frame in the world coordinate system;
    响应于所述当前变换参数不满足所述预设要求，利用其它图像在所述世界坐标系中的第二位姿、以及所述当前图像帧与所述其它图像帧之间的光度误差，确定所述第一位姿。In response to the current transformation parameters not meeting the preset requirements, using the second pose of other images in the world coordinate system and the photometric error between the current image frame and the other image frames, determine The first pose.
  7. 根据权利要求6所述的方法，其中，在所述获取当前图像帧与目标图像之间的当前变换参数之前，所述方法还包括：The method according to claim 6, wherein, before the acquisition of the current transformation parameters between the current image frame and the target image, the method further comprises:
    响应于上一图像帧的位姿获取方式为图像跟踪方式，执行所述获取当前图像帧与目标图像之间的当前变换参数；其中，所述图像跟踪方式，为利用所述上一图像帧与目标图像之间的变换参数确定所述上一图像帧在所述世界坐标系中的位姿。In response to the pose acquisition method of the previous image frame being an image tracking method, performing the acquisition of the current transformation parameters between the current image frame and the target image; wherein, the image tracking method is to use the previous image frame and the target image Transformation parameters between target images determine the pose of the last image frame in the world coordinate system.
  8. 根据权利要求6或7所述的方法，其中，所述其它图像帧和当前图像帧是由所述电子设备的图像采集装置先后拍摄得到；所述利用其它图像在所述世界坐标系中的第二位姿、以及所述当前图像帧与所述其它图像帧之间的光度误差，确定所述第一位姿，包括：The method according to claim 6 or 7, wherein the other image frames and the current image frame are sequentially photographed by the image acquisition device of the electronic device; Two poses, and photometric errors between the current image frame and the other image frames, determining the first pose, including:
    获取第一参考姿态；其中，第一参考姿态是所述图像采集装置对应于所述当前图像帧的拍摄时刻且相对于参考平面的姿态；Acquiring a first reference pose; wherein, the first reference pose is the pose of the image acquisition device corresponding to the shooting moment of the current image frame and relative to the reference plane;
    利用所述参考平面与世界坐标系中的预设平面之间的偏移量，对所述第一参考姿态进行调整，得到第二参考姿态；adjusting the first reference attitude by using an offset between the reference plane and a preset plane in the world coordinate system to obtain a second reference attitude;
    基于所述第二参考姿态、所述第二位姿以及所述当前图像帧与历史图像帧之间的光度误差，确定所述第一位姿。The first pose is determined based on the second reference pose, the second pose, and a photometric error between the current image frame and a historical image frame.
  9. 根据权利要求8所述的方法，其中，在所述利用所述参考平面与世界坐标系中的预设平面之间的偏移量，对所述第一参考姿态进行调整，得到第二参考姿态之前，所述方法还包括：The method according to claim 8, wherein the first reference attitude is adjusted by using the offset between the reference plane and a preset plane in the world coordinate system to obtain a second reference attitude Previously, the method further included:
    获取第一历史图像帧在所述世界坐标系中的第三位姿，获取第三参考姿态；其中，所述第三参考姿态，是所述图像采集装置对应于所述第一历史图像帧的拍摄时刻且相对于参考平面的姿态；所述第三位姿是基于所述目标图像确定的；所述预设平面为所述目标图像所在的平面；Obtain the third pose of the first historical image frame in the world coordinate system, and obtain a third reference pose; wherein, the third reference pose is the image acquisition device corresponding to the first historical image frame The pose at the shooting moment and relative to the reference plane; the third pose is determined based on the target image; the preset plane is the plane where the target image is located;
    利用所述第三位姿中的姿态和第三参考姿态，得到所述偏移量；其中，所述第三位姿中的姿态是相对于所述预设平面的姿态。The offset is obtained by using the posture in the third pose and a third reference pose; wherein, the pose in the third pose is a pose relative to the preset plane.
  10. 根据权利要求8或9所述的方法，其中，所述基于所述第二参考姿态、所述第二位姿以及所述当前图像帧与历史图像帧之间的光度误差，确定所述第一位姿，包括：The method according to claim 8 or 9, wherein said first determination is based on said second reference pose, said second pose, and a photometric error between said current image frame and a historical image frame. pose, including:
    获取至少一个第一候选位姿；其中，所述第一候选位姿为所述当前图像帧在所述世界坐标系中的位姿信息；Acquiring at least one first candidate pose; wherein, the first candidate pose is pose information of the current image frame in the world coordinate system;
    基于所述第二参考姿态、所述第二位姿以及所述当前图像帧和其它图像帧之间的第一像素值差异，选择一所述第一候选位姿作为所述第一位姿。Selecting a first candidate pose as the first pose based on the second reference pose, the second pose, and a first pixel value difference between the current image frame and other image frames.
  11. 根据权利要求10所述的方法，其中，所述第一候选位姿是基于所述当前图像帧在世界坐标系中的初始位姿确定的，所述初始位姿是基于所述当前图像帧和所述其它图像帧之间的光度误差确定的；所述基于所述第二参考姿态、所述第二位姿以及所述当前图像帧和其它图像帧之间的第一像素值差异，选择一所述第一候选位姿作为第一位姿，包括：The method according to claim 10, wherein the first candidate pose is determined based on the initial pose of the current image frame in the world coordinate system, and the initial pose is determined based on the current image frame and The photometric error between the other image frames is determined; based on the second reference pose, the second pose, and the first pixel value difference between the current image frame and other image frames, select a The first candidate pose as the first pose includes:
    利用所述第二位姿，确定与所述其它图像帧中的第一特征点对应的空间点；Using the second pose, determine a spatial point corresponding to the first feature point in the other image frame;
    基于每个所述第一候选位姿和所述空间点，从所述当前图像帧中确定与所述第一候选位姿对应的第二特征点；determining a second feature point corresponding to the first candidate pose from the current image frame based on each of the first candidate poses and the spatial point;
    获取所述第一特征点与第二特征点的第一像素值差异，并基于所述第一像素值差异以及所述第二参考姿态与第一候选位姿之间的位姿差异，选择一所述第一候选位姿作为第一位姿。Obtain the first pixel value difference between the first feature point and the second feature point, and based on the first pixel value difference and the pose difference between the second reference pose and the first candidate pose, select a The first candidate pose is used as the first pose.
  12. 根据权利要求9至11任一项所述的方法，其中，所述获取第一历史图像帧在所述世界坐标系中的第三位姿，包括：The method according to any one of claims 9 to 11, wherein said obtaining the third pose of the first historical image frame in the world coordinate system comprises:
    基于所述第一历史图像帧与所述目标图像之间的第一匹配点对，确定所述第一历史图像帧与所述目标图像之间的第四变换参数，利用所述第四变换参数得到所述第三位姿；Based on the first matching point pair between the first historical image frame and the target image, determine a fourth transformation parameter between the first historical image frame and the target image, and use the fourth transformation parameter obtaining the third pose;
    或者，or,
    基于所述第一历史图像帧与第二历史图像帧之间的第二匹配点对，确定所述第一历史图像帧与第二历史图像帧之间的第五变换参数，利用所述第五变换参数、以及所述第二历史图像帧与所述目标图像之间的第六变换参数得到所述第四变换参数，利用所述第四变换参数得到所述第三位姿；其中，所述第二历史图像帧位于所述第一历史图像帧之前。Based on the second matching point pair between the first historical image frame and the second historical image frame, determine a fifth transformation parameter between the first historical image frame and the second historical image frame, using the fifth transformation parameters, and a sixth transformation parameter between the second historical image frame and the target image to obtain the fourth transformation parameters, and use the fourth transformation parameters to obtain the third pose; wherein, the The second historical image frame is located before the first historical image frame.
  13. 根据权利要求12所述的方法，其中，所述利用所述第四变换参数得到所述第三位姿，包括：The method according to claim 12, wherein said obtaining said third pose using said fourth transformation parameter comprises:
    响应于所述第四变换参数满足预设要求，确定处于图像跟踪状态，利用所述第四变换参数得到所 述第三位姿。In response to the fourth transformation parameter meeting the preset requirement, it is determined to be in an image tracking state, and the third pose is obtained by using the fourth transformation parameter.
  14. 一种图像配准装置，包括：An image registration device, comprising:
    图像获取模块，配置为获取当前图像帧；An image acquisition module configured to acquire the current image frame;
    第一参数获取模块，配置为基于当前图像帧与其它图像帧中的目标图像信息，确定所述当前图像帧与所述其它图像帧之间的第一变换参数，其中，所述目标图像信息为关于目标图像的图像信息；所述第一变换参数，包括所述当前图像帧与所述其它图像帧之间的单应性矩阵；The first parameter acquisition module is configured to determine a first transformation parameter between the current image frame and the other image frames based on target image information in the current image frame and other image frames, wherein the target image information is Image information about the target image; the first transformation parameters include a homography matrix between the current image frame and the other image frames;
    第二参数获取模块，配置为基于所述第一变换参数、以及所述其它图像帧与所述目标图像之间的第二变换参数，得到所述当前图像帧与所述目标图像之间的第三变换参数；所述第二变换参数，包括所述当前图像帧与所述目标图像之间的单应性矩阵。The second parameter acquisition module is configured to obtain a second transformation parameter between the current image frame and the target image based on the first transformation parameter and a second transformation parameter between the other image frame and the target image Three transformation parameters: the second transformation parameter includes a homography matrix between the current image frame and the target image.
  15. 根据权利要求14所述的装置，其中，所述第一参数获取模块，配置为从所述当前图像帧中查找出关于所述目标图像的至少一个第二特征点；基于所述第一特征点和第二特征点，确定所述第一变换参数。The device according to claim 14, wherein the first parameter acquisition module is configured to find at least one second feature point about the target image from the current image frame; based on the first feature point and the second feature point, determine the first transformation parameter.
  16. 根据权利要求15所述的装置，其中，所述第一参数获取模块，配置为基于所述第二变换参数确定所述目标图像在所述其它图像帧中的目标区域；从所述目标区域中提取至少一个所述第一特征点；The device according to claim 15, wherein the first parameter acquisition module is configured to determine a target area of the target image in the other image frame based on the second transformation parameter; from the target area extracting at least one of the first feature points;
    所述第一参数获取模块，还配置为分别对所述至少一个第一特征点进行跟踪，得到所述当前图像帧中关于所述目标图像的至少一个第二特征点。The first parameter acquisition module is further configured to respectively track the at least one first feature point to obtain at least one second feature point related to the target image in the current image frame.
  17. 根据权利要求14至16任一所述的装置，其中，所述第二参数获取模块，配置为将所述第一变换参数与所述第二变换参数的乘积，作为所述第三变换参数；和/或，在所述基于所述第一变换参数、以及所述其它图像帧与所述目标图像之间的第二变换参数；The device according to any one of claims 14 to 16, wherein the second parameter acquisition module is configured to use the product of the first transformation parameter and the second transformation parameter as the third transformation parameter; And/or, based on the first transformation parameter and the second transformation parameter between the other image frame and the target image;
    所述图像配置装置还包括优化模块，配置为在得到所述当前图像帧与所述目标图像之间的所述第三变换参数之后，利用预设优化方式，对所述第三变换参数进行优化。The image configuration device further includes an optimization module configured to optimize the third transformation parameter by using a preset optimization method after obtaining the third transformation parameter between the current image frame and the target image .
  18. 一种视觉定位装置，包括：A visual positioning device, comprising:
    参数获取模块，配置为获取当前图像帧与目标图像之间的当前变换参数；其中，所述当前变换参数为利用权利要求1至4任一所述的图像配准方法获得的第三变换参数；A parameter acquisition module configured to acquire a current transformation parameter between the current image frame and the target image; wherein the current transformation parameter is a third transformation parameter obtained by using the image registration method described in any one of claims 1 to 4;
    第一位姿获取模块，配置为利用所述当前变换参数，得到所述当前图像帧在世界坐标系中的第一位姿，其中，所述世界坐标系是基于所述目标图像所在的平面建立的。The first pose acquisition module is configured to use the current transformation parameters to obtain the first pose of the current image frame in the world coordinate system, wherein the world coordinate system is established based on the plane where the target image is located of.
  19. 根据权利要求18所述的装置，其中，所述装置还包括第二位姿获取模块；The device according to claim 18, wherein the device further comprises a second pose acquisition module;
    所述第一位姿获取模块，配置为在所述利用所述当前变换参数，得到所述当前图像帧在世界坐标系中的第一位姿之前，响应于所述当前变换参数满足预设要求，执行所述利用所述当前变换参数，得到所述当前图像帧在所述世界坐标系中的第一位姿；The first pose acquisition module is configured to respond to the current transformation parameters meeting preset requirements before obtaining the first pose of the current image frame in the world coordinate system by using the current transformation parameters , performing the use of the current transformation parameters to obtain the first pose of the current image frame in the world coordinate system;
    所述第二位姿获取模块，配置为响应于所述当前变换参数不满足所述预设要求，利用其它图像在所述世界坐标系中的第二位姿、以及所述当前图像帧与所述其它图像帧之间的光度误差，确定所述第一位姿。The second pose acquisition module is configured to use the second pose of other images in the world coordinate system, and the relationship between the current image frame and the The photometric error between the other image frames is used to determine the first pose.
  20. 根据权利要求19所述的装置，其中，所述装置还包括状态确定模块，配置为在所述获取当前图像帧与目标图像之间的当前变换参数之前，响应于上一图像帧的位姿获取方式为图像跟踪方式，执行所述获取当前图像帧与目标图像之间的当前变换参数；其中，所述图像跟踪方式，为利用所述上一图像帧与目标图像之间的变换参数确定所述上一图像帧在所述世界坐标系中的位姿。The device according to claim 19, wherein the device further comprises a state determination module, configured to acquire the pose of the previous image frame in response to the acquisition of the current transformation parameters between the current image frame and the target image The method is an image tracking method, executing the acquisition of the current transformation parameters between the current image frame and the target image; wherein, the image tracking method is to use the transformation parameters between the last image frame and the target image to determine the The pose of the last image frame in the world coordinate system.
  21. 根据权利要求19或20所述的装置，其中，所述其它图像帧和当前图像帧是由所述电子设备的图像采集装置先后拍摄得到；所述第二位姿获取模块，配置为获取第一参考姿态；利用所述参考平面与世界坐标系中的预设平面之间的偏移量，对所述第一参考姿态进行调整，得到第二参考姿态；基于所述第二参考姿态、所述第二位姿以及所述当前图像帧与历史图像帧之间的光度误差，确定所述第一位姿；其中，第一参考姿态是所述图像采集装置对应于所述当前图像帧的拍摄时刻且相对于参考平面的姿态。The device according to claim 19 or 20, wherein the other image frames and the current image frame are sequentially captured by the image acquisition device of the electronic device; the second pose acquisition module is configured to acquire the first Reference posture; using the offset between the reference plane and a preset plane in the world coordinate system to adjust the first reference posture to obtain a second reference posture; based on the second reference posture, the The second pose and the photometric error between the current image frame and the historical image frame determine the first pose; wherein the first reference pose is the shooting moment of the image acquisition device corresponding to the current image frame and the attitude relative to the reference plane.
  22. 根据权利要求21所述的装置，其中，所述装置还包括偏移量获取模块，配置为在所述利用所 述参考平面与世界坐标系中的预设平面之间的偏移量，对所述第一参考姿态进行调整，得到第二参考姿态之前，获取第一历史图像帧在所述世界坐标系中的第三位姿，获取第三参考姿态；利用所述第三位姿中的姿态和第三参考姿态，得到所述偏移量；其中，所述第三参考姿态，是所述图像采集装置对应于所述第一历史图像帧的拍摄时刻且相对于参考平面的姿态；所述第三位姿是基于所述目标图像确定的；所述预设平面为所述目标图像所在的平面；所述第三位姿中的姿态是相对于所述预设平面的姿态。The device according to claim 21, wherein the device further comprises an offset acquisition module, configured to use the offset between the reference plane and a preset plane in the world coordinate system to calculate the Adjust the first reference pose, obtain the third pose of the first historical image frame in the world coordinate system before obtaining the second reference pose, and obtain the third reference pose; use the pose in the third pose and a third reference posture to obtain the offset; wherein, the third reference posture is the posture of the image acquisition device corresponding to the shooting moment of the first historical image frame and relative to the reference plane; the The third pose is determined based on the target image; the preset plane is the plane where the target image is located; the pose in the third pose is a pose relative to the preset plane.
  23. 根据权利要求21或22所述的装置，其中，所述第二位姿获取模块，配置为获取至少一个第一候选位姿；基于所述第二参考姿态、所述第二位姿以及所述当前图像帧和其它图像帧之间的第一像素值差异，选择一所述第一候选位姿作为所述第一位姿；其中，所述第一候选位姿为所述当前图像帧在所述世界坐标系中的位姿信息。The device according to claim 21 or 22, wherein the second pose acquisition module is configured to acquire at least one first candidate pose; based on the second reference pose, the second pose and the The first pixel value difference between the current image frame and other image frames, select one of the first candidate poses as the first pose; wherein, the first candidate pose is the current image frame at the Describe the pose information in the world coordinate system.
  24. 根据权利要求23所述的装置，其中，所述第一候选位姿是基于所述当前图像帧在世界坐标系中的初始位姿确定的，所述初始位姿是基于所述当前图像帧和所述其它图像帧之间的光度误差确定的；所述第二位姿获取模块，配置为利用所述第二位姿，确定与所述其它图像帧中的第一特征点对应的空间点；基于每个所述第一候选位姿和所述空间点，从所述当前图像帧中确定与所述第一候选位姿对应的第二特征点；获取所述第一特征点与第二特征点的第一像素值差异，并基于所述第一像素值差异以及所述第二参考姿态与第一候选位姿之间的位姿差异，选择一所述第一候选位姿作为第一位姿。The apparatus according to claim 23, wherein the first candidate pose is determined based on the initial pose of the current image frame in the world coordinate system, and the initial pose is determined based on the current image frame and The photometric error between the other image frames is determined; the second pose acquisition module is configured to use the second pose to determine a spatial point corresponding to the first feature point in the other image frames; Based on each of the first candidate poses and the space point, determine a second feature point corresponding to the first candidate pose from the current image frame; obtain the first feature point and the second feature point The first pixel value difference of the point, and based on the first pixel value difference and the pose difference between the second reference pose and the first candidate pose, select one of the first candidate poses as the first position posture.
  25. 根据权利要求22至24任一项所述的装置，其中，所述装置还包括历史图像帧位姿获取模块，配置为：基于所述第一历史图像帧与所述目标图像之间的第一匹配点对，确定所述第一历史图像帧与所述目标图像之间的第四变换参数，利用所述第四变换参数得到所述第三位姿；The device according to any one of claims 22 to 24, wherein the device further comprises a historical image frame pose acquisition module configured to: based on the first matching point pairs, determining a fourth transformation parameter between the first historical image frame and the target image, and using the fourth transformation parameter to obtain the third pose;
    或者，or,
    基于所述第一历史图像帧与第二历史图像帧之间的第二匹配点对，确定所述第一历史图像帧与第二历史图像帧之间的第五变换参数，利用所述第五变换参数、以及所述第二历史图像帧与所述目标图像之间的第六变换参数得到所述第四变换参数，利用所述第四变换参数得到所述第三位姿；其中，所述第二历史图像帧位于所述第一历史图像帧之前。Based on the second matching point pair between the first historical image frame and the second historical image frame, determine a fifth transformation parameter between the first historical image frame and the second historical image frame, using the fifth transformation parameters, and a sixth transformation parameter between the second historical image frame and the target image to obtain the fourth transformation parameters, and use the fourth transformation parameters to obtain the third pose; wherein, the The second historical image frame is located before the first historical image frame.
  26. 根据权利要求25所述的装置，其中，所述历史图像帧位姿获取模块，配置为利用所述第四变换参数得到所述第二位姿之前，响应于所述第四变换参数满足预设要求，确定处于图像跟踪状态，利用所述第四变换参数得到所述第三位姿。The device according to claim 25, wherein the historical image frame pose acquisition module is configured to respond to the fourth transformation parameter satisfying a preset before obtaining the second pose using the fourth transformation parameter It is required to determine that it is in an image tracking state, and to obtain the third pose by using the fourth transformation parameter.
  27. 一种电子设备，包括相互耦接的存储器和处理器，所述处理器用于执行所述存储器中存储的程序指令，以实现权利要求1至4任一项所述的图像配准方法，以及权利要求5至13任一项所述的视觉定位方法。An electronic device, comprising a memory and a processor coupled to each other, the processor is used to execute the program instructions stored in the memory, so as to realize the image registration method described in any one of claims 1 to 4, and the right The visual positioning method described in any one of 5 to 13 is required.
  28. 一种计算机可读存储介质，其上存储有程序指令，所述程序指令被处理器执行时实现权利要求1至4任一项所述的图像配准方法，以及权利要求5至13任一项所述的视觉定位方法。A computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the image registration method according to any one of claims 1 to 4 is implemented, and any one of claims 5 to 13 The described visual positioning method.
  29. 一种计算机程序，所述计算机程序包括计算机可读代码，在所述计算机可读代码在电子设备中运行的情况下，所述电子设备的处理器执行用于实现如权利要求1至4任一项所述的图像配准方法、以及权利要求5至13任一项所述的视觉定位方法。A computer program, the computer program comprising computer readable code, in the case of the computer readable code running in an electronic device, the processor of the electronic device executes to implement any one of claims 1 to 4 The image registration method described in claim 1, and the visual positioning method described in any one of claims 5 to 13.
PCT/CN2021/121049 2021-06-25 2021-09-27 Image registration method, visual positioning method, apparatus, device, medium, and program WO2022267257A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110713164.9A CN113393505B (en) 2021-06-25 2021-06-25 Image registration method, visual positioning method, related device and equipment
CN202110713164.9 2021-06-25

Publications (1)

Publication Number Publication Date
WO2022267257A1 true WO2022267257A1 (en) 2022-12-29

Family

ID=77624135

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/121049 WO2022267257A1 (en) 2021-06-25 2021-09-27 Image registration method, visual positioning method, apparatus, device, medium, and program

Country Status (3)

Country Link
CN (1) CN113393505B (en)
TW (1) TW202301273A (en)
WO (1) WO2022267257A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116740651A (en) * 2023-08-11 2023-09-12 南京吾悦农业科技有限公司 Edible fungus cultivation monitoring method and system based on intelligent decision

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113393505B (en) * 2021-06-25 2023-11-03 浙江商汤科技开发有限公司 Image registration method, visual positioning method, related device and equipment
CN113763437B (en) * 2021-09-16 2023-12-05 Oppo广东移动通信有限公司 Image processing method, device, electronic equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190204084A1 (en) * 2017-09-29 2019-07-04 Goertek Inc. Binocular vision localization method, device and system
CN110728705A (en) * 2019-09-24 2020-01-24 Oppo广东移动通信有限公司 Image processing method, image processing device, storage medium and electronic equipment
CN111932587A (en) * 2020-08-03 2020-11-13 Oppo广东移动通信有限公司 Image processing method and device, electronic equipment and computer readable storage medium
CN113393505A (en) * 2021-06-25 2021-09-14 浙江商汤科技开发有限公司 Image registration method, visual positioning method, related device and equipment

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101339658B (en) * 2008-08-12 2010-09-01 北京航空航天大学 Aerial photography traffic video rapid robust registration method
US9858681B2 (en) * 2014-10-27 2018-01-02 Digimarc Corporation Signal detection, recognition and tracking with feature vector transforms
CN105046644B (en) * 2015-07-06 2021-08-13 嘉恒医疗科技（上海）有限公司 Ultrasonic and CT image registration method and system based on linear correlation
CN106874949B (en) * 2017-02-10 2019-10-11 华中科技大学 Movement imaging platform moving target detecting method and system based on infrared image
CN108629843B (en) * 2017-03-24 2021-07-13 成都理想境界科技有限公司 Method and equipment for realizing augmented reality
CN108550104B (en) * 2018-02-28 2022-06-10 北京集光通达科技股份有限公司 Image registration method and device
CN108682038B (en) * 2018-04-27 2021-12-14 腾讯科技（深圳）有限公司 Pose determination method, pose determination device and storage medium
CN109242951A (en) * 2018-08-06 2019-01-18 宁波盈芯信息科技有限公司 A kind of face's real-time three-dimensional method for reconstructing
CN110310338B (en) * 2019-06-24 2022-09-06 西北工业大学 Light field camera calibration method based on multi-center projection model
CN110335317B (en) * 2019-07-02 2022-03-25 百度在线网络技术（北京）有限公司 Image processing method, device, equipment and medium based on terminal equipment positioning
CN111327840A (en) * 2020-02-27 2020-06-23 努比亚技术有限公司 Multi-frame special-effect video acquisition method, terminal and computer readable storage medium
CN111402221B (en) * 2020-03-11 2023-02-17 联想(北京)有限公司 Image processing method and device and electronic equipment
CN111445526B (en) * 2020-04-22 2023-08-04 清华大学 Method, device and storage medium for estimating pose of image frame
CN111932595A (en) * 2020-09-24 2020-11-13 平安科技（深圳）有限公司 Image registration method and device, electronic equipment and storage medium
CN112435223B (en) * 2020-11-11 2021-11-23 马鞍山市瀚海云星科技有限责任公司 Target detection method, device and storage medium
CN112907620B (en) * 2021-01-25 2024-06-28 北京地平线机器人技术研发有限公司 Camera pose estimation method and device, readable storage medium and electronic equipment
CN112991411B (en) * 2021-03-05 2024-04-12 浙江华感科技有限公司 Image registration method and apparatus, and storage medium
CN113012226B (en) * 2021-03-22 2024-07-12 浙江商汤科技开发有限公司 Method and device for estimating pose of camera, electronic equipment and computer storage medium
CN112884840A (en) * 2021-03-29 2021-06-01 湖南国科微电子股份有限公司 Visual positioning method, device, equipment and storage medium

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190204084A1 (en) * 2017-09-29 2019-07-04 Goertek Inc. Binocular vision localization method, device and system
CN110728705A (en) * 2019-09-24 2020-01-24 Oppo广东移动通信有限公司 Image processing method, image processing device, storage medium and electronic equipment
CN111932587A (en) * 2020-08-03 2020-11-13 Oppo广东移动通信有限公司 Image processing method and device, electronic equipment and computer readable storage medium
CN113393505A (en) * 2021-06-25 2021-09-14 浙江商汤科技开发有限公司 Image registration method, visual positioning method, related device and equipment

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116740651A (en) * 2023-08-11 2023-09-12 南京吾悦农业科技有限公司 Edible fungus cultivation monitoring method and system based on intelligent decision
CN116740651B (en) * 2023-08-11 2023-10-17 南京吾悦农业科技有限公司 Edible fungus cultivation monitoring method and system based on intelligent decision

Also Published As

Publication number Publication date
CN113393505A (en) 2021-09-14
CN113393505B (en) 2023-11-03
TW202301273A (en) 2023-01-01

Similar Documents

Publication Publication Date Title
WO2022267257A1 (en) Image registration method, visual positioning method, apparatus, device, medium, and program
CN110322500B (en) Optimization method and device for instant positioning and map construction, medium and electronic equipment
WO2019205852A1 (en) Method and apparatus for determining pose of image capture device, and storage medium therefor
Skrypnyk et al. Scene modelling, recognition and tracking with invariant image features
Gordon et al. What and where: 3D object recognition with accurate pose
WO2021057742A1 (en) Positioning method and apparatus, device, and storage medium
US20220282993A1 (en) Map fusion method, device and storage medium
Muñoz et al. Fast 6D pose estimation for texture-less objects from a single RGB image
US20180249144A1 (en) System and Method for Virtually-Augmented Visual Simultaneous Localization and Mapping
WO2023060964A1 (en) Calibration method and related apparatus, device, storage medium and computer program product
CN113409391B (en) Visual positioning method and related device, equipment and storage medium
WO2022267287A1 (en) Image registration method and related apparatus, and device and storage medium
TWI686771B (en) 3d model reconstruction method, electronic device, and non-transitory computer readable storage medium
CN110349212B (en) Optimization method and device for instant positioning and map construction, medium and electronic equipment
JP5833507B2 (en) Image processing device
WO2023151251A1 (en) Map construction method and apparatus, pose determination method and apparatus, and device and computer program product
WO2019157922A1 (en) Image processing method and device and ar apparatus
WO2022247286A1 (en) Positioning method, apparatus, device, and storage medium
JP6662382B2 (en) Information processing apparatus and method, and program
CN114565668A (en) Instant positioning and mapping method and device
JP2023056466A (en) Global positioning device and method for global positioning
CN113407030B (en) Visual positioning method, related device, equipment and storage medium
Xu et al. Real-time camera tracking for marker-less and unprepared augmented reality environments
KR20220004604A (en) Method for detecting obstacle, electronic device, roadside device and cloud control platform
CN112085842B (en) Depth value determining method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21946740

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21946740

Country of ref document: EP

Kind code of ref document: A1