WO2023216918A1 - 渲染图像的方法、装置、电子设备及存储介质 - Google Patents
渲染图像的方法、装置、电子设备及存储介质 Download PDFInfo
- Publication number
- WO2023216918A1 WO2023216918A1 PCT/CN2023/091479 CN2023091479W WO2023216918A1 WO 2023216918 A1 WO2023216918 A1 WO 2023216918A1 CN 2023091479 W CN2023091479 W CN 2023091479W WO 2023216918 A1 WO2023216918 A1 WO 2023216918A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- key frame
- updated
- frame
- key
- current frame
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 103
- 238000009877 rendering Methods 0.000 title claims abstract description 58
- 238000013507 mapping Methods 0.000 claims abstract description 41
- 230000004044 response Effects 0.000 claims abstract description 12
- 238000006073 displacement reaction Methods 0.000 claims description 59
- 230000008569 process Effects 0.000 claims description 33
- 238000004422 calculation algorithm Methods 0.000 claims description 15
- 238000005457 optimization Methods 0.000 claims description 14
- 238000001514 detection method Methods 0.000 claims description 11
- 238000007781 pre-processing Methods 0.000 claims description 10
- 230000004807 localization Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 description 30
- 230000001360 synchronised effect Effects 0.000 description 16
- 230000006870 function Effects 0.000 description 11
- 230000003287 optical effect Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 9
- 230000000694 effects Effects 0.000 description 9
- 238000004590 computer program Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000033001 locomotion Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 238000005070 sampling Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 101000822695 Clostridium perfringens (strain 13 / Type A) Small, acid-soluble spore protein C1 Proteins 0.000 description 1
- 101000655262 Clostridium perfringens (strain 13 / Type A) Small, acid-soluble spore protein C2 Proteins 0.000 description 1
- 101000655256 Paraclostridium bifermentans Small, acid-soluble spore protein alpha Proteins 0.000 description 1
- 101000655264 Paraclostridium bifermentans Small, acid-soluble spore protein beta Proteins 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000002945 steepest descent method Methods 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
Definitions
- Embodiments of the present disclosure relate to the technical field of image processing, for example, to a method, device, electronic device, and storage medium for rendering an image.
- SLAM Simultaneous Localization and Mapping
- the filter-based SLAM system cannot provide more accurate camera pose information and captured spatial information for a long time, which results in poor image effects rendered by the system, while the feature point-based SLAM system It is also necessary to extract the feature points in the image and match the feature points in each frame.
- the disadvantage of this method is that it not only increases the computational overhead in the image processing process, but also makes it difficult to process the images captured on the mobile terminal. Images are processed in real time, affecting the user experience.
- the present disclosure provides a method, device, electronic equipment and storage medium for rendering images, which improves the positioning accuracy of the SLAM space and optimizes the rendering effect of the image. At the same time, it improves the image rendering efficiency and ensures the processing of images captured by mobile terminals. real-time.
- an embodiment of the present disclosure provides a method for rendering an image, including:
- the key frame group to be updated Based on the key frame group to be updated based on the synchronization positioning and mapping system positioning, determine whether the current frame received is No, it is a key frame; wherein, the key frame group to be updated includes at least one key frame to be applied;
- embodiments of the present disclosure also provide a device for rendering images, including:
- the key frame determination module is configured to determine whether the received current frame is a key frame based on the key frame group to be updated based on the synchronization positioning and mapping system positioning; wherein the key frame group to be updated includes at least one key frame to be applied. ;
- An update module configured to respond to determining that the received current frame is a key frame, update the key frame group to be updated according to the preset number of frames and the current frame, and obtain an updated key frame group to be updated;
- the keyframe optimization module to be applied is configured to optimize the keyframes to be applied in the keyframe group to be updated, update the relative pose of the keyframes to be applied, and perform image rendering based on the updated relative pose.
- embodiments of the present disclosure also provide an electronic device, where the electronic device includes:
- a storage device arranged to store at least one program
- the at least one processor When the at least one program is executed by the at least one processor, the at least one processor is caused to implement the method for rendering an image as described in any embodiment of the present disclosure.
- embodiments of the disclosure further provide a storage medium containing computer-executable instructions, which when executed by a computer processor are used to perform rendering images as described in any embodiment of the disclosure. Methods.
- Figure 1 is a schematic flowchart of a method for rendering images provided by an embodiment of the present disclosure
- Figure 2 is a schematic structural diagram of a device for rendering images provided by an embodiment of the present disclosure
- FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
- the term “include” and its variations are open-ended, ie, “including but not limited to.”
- the term “based on” means “based at least in part on.”
- the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; and the term “some embodiments” means “at least some embodiments”. Relevant definitions of other terms will be given in the description below.
- the application scenarios of the embodiments of the present disclosure may be exemplified.
- the user uses a mobile camera device to shoot a video and uploads the captured video to a system based on the SLAM algorithm, or selects a target video in the database and actively uploads the video to the system based on the SLAM algorithm
- the system can analyze and process the video.
- the SLAM system in related technologies is difficult to provide more accurate camera pose information and spatial information for a long time.
- the final image rendering effect is poor, or the SLAM system needs to process the video frames. Click on the features in Line extraction and feature matching are performed. In this process, the large computational overhead makes it difficult for the system to process the video captured on the mobile terminal in real time.
- the key frame group to be updated can be directly determined in the video, and the key frame group to be updated can be updated according to the preset number of frames and the current key frame, and the key frames therein can be optimized. After that, the relative pose of the key frame can be obtained, thereby improving the positioning accuracy of the SLAM space and obtaining better rendering results.
- the SLAM system of the embodiment of the present disclosure does not need to extract and match feature points in the image, which reduces It reduces the computational overhead and facilitates real-time processing of images uploaded by the mobile terminal.
- Figure 1 is a schematic flowchart of a method for rendering images provided by an embodiment of the present disclosure.
- the embodiment of the present disclosure is suitable for processing video based on a SLAM system to render corresponding multi-frame images in real time on a display interface.
- the method may be executed by a device that renders images, which may be implemented in the form of software and/or hardware, or optionally, by an electronic device, which may be a mobile terminal, a PC, or a server.
- the method includes:
- SLAM Simultaneous Localization and Mapping
- the system is a system integrated with SLAM-related algorithms. Its algorithm usually includes several parts such as feature extraction, data association, state estimation, state update, and feature update. For each part, there are various The processing method is not limited in the embodiments of the present disclosure.
- the synchronized positioning and mapping system that executes the method of rendering images provided by the embodiment of the present disclosure can be integrated into application software that supports special effects video processing functions, and the software can be installed in electronic devices.
- the electronic device can be a mobile terminal or PC, etc.
- the application software may be a type of software for image/video processing.
- the specific application software will not be described in detail here, as long as it can realize image/video processing. It can also be a specially developed application to add special effects and The special effects can be displayed in the software or integrated into the corresponding page. Users can process the special effects video through the page integrated in the PC.
- the technical solution of this embodiment can be executed during the process of real-time photography based on the mobile terminal, or can be executed after the system receives the video data actively uploaded by the user.
- the solution of the disclosed embodiment can be applied to enhance In various application scenarios such as reality (Augumented Reality, AR), virtual reality (Virtual Reality, VR), and autonomous driving.
- the key frame group to be updated is a set containing multiple key frames. Based on the SLAM system in the embodiment of the present disclosure, the images in the key frame group to be updated can also be updated. At the same time, the key frame group to be updated includes at least one key frame to be applied.
- a key frame is used to represent multiple frames adjacent to it, which is equivalent to the skeleton of SLAM. It selects one frame from a series of local ordinary frames as the representative of the local frame. Therefore, at least the local information of the video picture is recorded in the key frame.
- using key frames to perform subsequent image rendering processing can also effectively reduce the number of video frames that need to be optimized, thereby improving the image processing efficiency of the system.
- the SLAM system can store the video in a preset sequence.
- the preset sequence can store the above video in the order of each frame in the above video, for example, the sequence of each frame in the above video.
- the order is first frame 1, then frame 2...then frame n-1, and finally frame n.
- the above preset sequence stores the above video in the order of frame 1, frame 2...frame n-1 and frame n.
- the above-mentioned multiple video frames together constitute a key frame group to be updated, in which frame 1, frame 10, frame 20... frame n respectively represent the key frames to be applied to its adjacent frames.
- the SLAM system before the SLAM system determines whether the current frame received is a key frame, it can also preprocess the multiple continuous frame images when receiving the multiple continuous frame images for the first time to determine at least one initial key frame. ; Use at least one initialization keyframe as at least one keyframe to be applied in the keyframe group to be updated.
- multiple consecutive frame images can be images parsed by the system from the received video data, For example, in the above example, frame 1, frame 2...frame n-1 and frame n, those skilled in the art will understand that multiple consecutive frame images can be determined according to the actual situation.
- the system can pre-construct an adaptive-sized sliding window, so that after receiving the above-mentioned multiple continuous frame images, it can preprocess the image and use the sliding window to filter out at least one initial key frame.
- preprocessing includes the operation of removing the influence of rotation.
- the reason for the above preprocessing is: in multiple consecutive video frames, the picture may rotate, and the rotation will affect the pixel distance difference of the frame.
- only rotation cannot perform simultaneous positioning and mapping initialization. Therefore, in order to solve this problem, the embodiment of the present disclosure performs the above preprocessing and uses the pixel distance difference to remove the influence of rotation to filter at least one initial key frame in the window, thereby ensuring that there is sufficient visibility between frames in the window. There is sufficient parallax for simultaneous positioning and mapping initialization. It can be understood that by removing the rotation operation, the impact of rotation on synchronized positioning and mapping initialization is reduced, and the accuracy of synchronized positioning and mapping initialization is improved.
- the system can obtain the above-mentioned rotation information from the inertial measurement unit, thereby determining the pixel distance difference of the rotation-affected frames based on the acquired information, and perform processing to remove the rotation influence on the above-mentioned multiple consecutive frame images, and use the removal
- the pixel distance difference affected by the rotation filters out at least one initial keyframe within the above sliding window.
- the system can use a pre-built sliding window of adaptive size to filter out at least one initial key frame from the above-mentioned multiple consecutive frame images that remove the influence of rotation. This process is explained below.
- the relative pose of the first key frame and the last key frame in multiple key frames is determined; based on the relative pose of the first key frame and the last key frame, the relative pose of each key frame in the multiple key frames is obtained.
- Three-dimensional space points determine the relative pose of each key frame in multiple key frames based on the relative poses of the first key frame and the last key frame, as well as the three-dimensional space points of each key frame in multiple key frames; based on multiple key frames
- the three-dimensional space points of each key frame in the key frame and the relative pose of each key frame in multiple key frames are used to establish an initial map. After the initial map is established, preprocessing operations on multiple consecutive frame images can be performed.
- the system pre-constructs a sliding window with adjustable size, for example, a sliding window with a size of 5 to 10 image frames. Using this sliding window, multiple consecutive frame images can be removed while removing the influence of rotation. Filter out at least one initial key frame in the sliding window. For example, the current length of the sliding window is 5 frames.
- the system uses the pixel distance difference that removes the influence of rotation to filter out the initial key frame in the sliding window. For example, parsing from the received video Frame 1, Frame 2... Frame 25 are filtered out, and Frame 6, Frame 7, Frame 10, Frame 12 and Frame 13 are selected as the above-mentioned initial key frames.
- the obtained at least one initialization key frame is at least one key frame to be applied in the key frame group to be updated.
- the system performs synchronized positioning and mapping initialization based on initial key frames filtered out from multiple consecutive frame images, which reduces the time for synchronized positioning and mapping initialization.
- the system uses pixels that remove the influence of rotation
- the initial keyframes in the distance difference filtering window have enough parallax for synchronized positioning and mapping initialization on the premise of ensuring that there is enough public view between the frames in the window. At the same time, it reduces the rotational impact on synchronized positioning and mapping initialization. Impact, improving the accuracy of simultaneous positioning and mapping initialization.
- determining whether the received current frame is a key frame it also includes determining the point cloud data to be processed in the current frame based on a corner point detection algorithm to perform processing on the point cloud data to be processed based on at least one key frame to be applied. Process to obtain the optimized pose of the current frame to determine whether the current frame is a key frame.
- the system when the system receives the current frame, it first needs to determine the point cloud data (PCD) in the current frame based on the corner detection algorithm.
- point cloud data is usually used in reverse engineering. It is a kind of data recorded in the form of points. These points can be coordinates in three-dimensional space, or information such as color or light intensity.
- Point cloud data In the actual application process, Point cloud data generally also includes point coordinate accuracy, spatial resolution, surface normal vector, etc., and is generally saved in PCD format. In this format, point cloud data is highly operable and can be used to improve the point cloud data in the subsequent process. Cloud registration and integration The speed of closing will not be described again in the embodiment of this disclosure. It can be understood that in this embodiment, the point cloud data in the current frame is the point cloud data to be processed.
- the corner detection algorithm used by the system can be the KLT corner detection method, also known as the KLT optical flow tracking method.
- the KLT corner detection method is used to meet the needs of the Lucas-Kanade optical flow method to select appropriate feature points.
- the Lucas-Kanade optical flow method first establishes a fixed-size window in the two frames of images, and then determines the two windows. The displacement that is the smallest sum of the squared pixel intensity differences between pixels, and the movement of pixels within the window is approximated as such a displacement vector.
- pixel movement is more complicated. At the same time, the pixels in the window do not all move in the same way. This approximate method will inevitably bring errors.
- the KLT corner detection method is to select a suitable tracking Feature points can be understood as, good feature points are points that can be better tracked by the system.
- the process of using the KLT corner point detection method to determine the point cloud data to be processed includes determining the pixel point light intensity function, adjusting the deviation energy within the window to the minimum, corner point selection, feature point selection, and setting the threshold of the energy deviation function to exclude Multiple steps such as blocked points will not be described again in this embodiment of the disclosure.
- the KLT corner detection method is used to determine the point cloud data in the current frame. There is no need to extract the descriptor in the current frame, and there is no need to perform feature point matching operations, thereby enhancing the real-time data processing of the system.
- the performance and robustness enable the system to achieve efficient corner tracking in the process of corner tracking and determining the point cloud data to be processed.
- the point cloud data after obtaining the point cloud data to be processed in the current frame, the point cloud data can be processed based on the key frames to be applied, thereby obtaining the optimized pose of the current frame.
- BA graph optimization with camera poses and spatial points
- the optimization problem of feature points accounts for a large part. After several iterations, the feature points will converge, and there is no greater significance in optimizing at this time. Therefore, in the actual process, after optimizing several times, the feature points can be fixed and regarded as constraints for pose estimation, that is, the pose of the feature points is no longer optimized.
- the optimized pose graph is a graph optimization with only trajectories constructed by considering only the pose, and the edges between pose nodes are formed by special passes between two key frames.
- the initial value is given by the motion estimate obtained after feature matching. Once the initial value is determined, the position of the landmark point is no longer optimized, and only the connection between the camera poses is concerned.
- the optimized pose is information determined based on the pose graph of the current frame. Based on this information, the system can determine whether the current frame is a key frame.
- the above-mentioned incremental BA problem construction method is used to determine the optimized pose of the current frame, so that the synchronized positioning and mapping system can provide a higher BA speed, thereby ensuring that the system processes video frames. real-time.
- the target feature point is the point determined from the object in each frame of image. For example, there are multiple steps on a certain frame of image.
- the system determines the algorithm based on the pre-trained feature points, and then determines the algorithm from each step in the image. Corresponding multiple feature points are determined, and these feature points are the target feature points.
- the system can use the determined multiple feature points as a kind of identification to calculate the change of the camera pose.
- the target feature points determined by the system can be of various types, such as scale-invariant feature transform (SIFT) feature points, accelerated robust features (Speeded Up Robust Features, SURF) feature points and ORB (Oriented FAST and Rotated BRIEF) feature points, etc.
- SIFT scale-invariant feature transform
- SURF accelerated robust features
- ORB Oriented FAST and Rotated BRIEF
- the system can also preset a threshold for the parameter of the number of target feature points, which is the first preset quantity threshold.
- a threshold can also be preset for the parameter of displacement parallax, which The threshold is the first preset displacement parallax threshold.
- the first preset quantity threshold of the system is 100 and the first preset displacement parallax threshold is 100 pixels
- the system can determine that the current frame is a key frame. This field Technicians should understand that if any one of the above two parameters is less than or equal to its corresponding preset threshold, or both parameters are less than or equal to its corresponding preset threshold, the current frame will not be determined as a key frame. , and after discarding the current frame, continue to judge the subsequent received frames one by one in the above manner, which will not be described again in the embodiment of the present disclosure.
- determine the common view feature points of the current frame and at least one key frame to be applied and perform downsampling processing in the current frame based on the common view feature points to determine the target feature points; and determine the current frame and at least one key frame to be applied.
- the displacement deviation of the frame if the number of target feature points is less than the number of feature points to be processed in the current frame, and the displacement deviation is less than the second preset displacement deviation, the current frame is determined to be a key frame.
- the system can also compare these feature points with at least one feature point in the picture corresponding to the key frame to be applied, thereby determining these feature points.
- common-view feature points in images for example, key points and descriptors associated with feature points in multiple video frame images are matched and compared to determine common-view feature points.
- the common view feature point is the common view point of the current frame and the key frame to be applied.
- the system determines the feature points corresponding to multiple steps in the picture from the received current frame, These feature points need to be compared with the feature points in other key frames to be applied.
- a picture of a key frame to be applied also contains this multi-level ladder, that is, it also contains the above feature points
- the system can The feature points corresponding to the multi-level ladder in the two video frames are determined as common view feature points.
- downsampling is A multi-rate digital signal processing technology, which is also a process of reducing the signal sampling rate. It is usually used to reduce the data transmission rate or data size. For example, after 4 times downsampling of the 160 common view feature points in the current frame, that is 40 feature points can be screened out as target feature points. It can be understood that the 4 times parameter used in the above example is the downsampling rate. This parameter is used to express that the sampling period becomes M times the original, or the sampling rate becomes It is 1/M times the original value. At the same time, the down-sampling rate can be preset manually or automatically, which is not limited in the embodiment of the present disclosure.
- the system when the system determines the target feature points, it also needs to determine the displacement deviation between the current frame and the key frame to be applied, where the displacement deviation is information that characterizes changes in camera pose.
- the current frame is taken when the camera is at point A in the scene
- a certain key frame to be applied is taken when the camera is at point B in the same scene.
- the camera moves from point B to point A.
- the change in pose produced by the point is the displacement deviation determined by the system from the two frames of images.
- the system can also preset a threshold value for the displacement deviation parameter, and the threshold value is the second preset displacement deviation. Based on this, when the system determines the target feature points from the current frame and determines the displacement deviation between the current frame and at least one key frame to be applied, the number of target feature points can be compared with the features to be processed in the current frame. The number of points is compared, and the displacement deviation between the current frame and the key frame to be applied is compared with the second preset displacement deviation. When the above two parameters are smaller than their corresponding comparison objects, the current Frames are keyframes.
- the current frame downsample the point cloud data to be processed in the current frame to obtain the target feature points; determine the displacement deviation between the current frame and at least one key frame to be applied; if the number of target feature points is less than or equal to the number of common view feature points , and the displacement deviation is less than the third preset displacement deviation, then the current frame is determined to be a key frame.
- the system can downsample the point cloud data to be processed in the current frame.
- the point cloud data can be downsampled through voxel grids.
- the system uses this method to When downsampling point cloud data, while reducing the number of points in the point cloud data, the shape of the point cloud can still be maintained. It can also improve the speed of registration, surface reconstruction, shape recognition and other algorithms, and ensure the downsampling process. accuracy.
- the system can also be configured according to the embodiments of the present disclosure. In this way, the displacement deviation between the current frame and at least one key frame to be applied is determined, and the embodiments of the present disclosure will not be repeated here.
- the system can pre-set a threshold for the displacement deviation parameter.
- This threshold is the third preset displacement deviation.
- the number of target feature points is compared with the number of common view feature points between multiple video frames.
- a comparison is performed, and the displacement deviation is compared with the third preset displacement deviation.
- the key frame when the current received frame is determined to be a key frame based on the synchronized positioning and mapping system, the key frame needs to be added to the key frame group to be updated, thereby updating the key frame group to be updated.
- This process can be understood as updating and optimizing local map information.
- a sliding window structure is used to maintain local map information, where the sliding window can include multiple adjacent frames, and based on these adjacent frames Frame observed space point. Based on this, in the process of local optimization of map information, historical information needs to be used to constrain the current frame, thereby improving the accuracy of the local map information optimization results.
- a frame can be preset for the sliding window structure. At the same time, the number of frames also determines the number of key frames in the key frame group to be updated.
- the number of local map points is also indirectly controlled, thereby It effectively controls the subsequent calculation efficiency and facilitates the real-time processing of images captured by the mobile terminal by the simultaneous positioning and mapping system.
- the current frame is updated to the key frame group to be updated to obtain an updated key frame group to be updated.
- the sliding window can include at most 10 adjacent key frames as key frames to be applied
- the preset number of frames is 10.
- the number of key frames to be applied included in the current key frame group to be updated is The number is 6, that is, the number of key frames to be applied is less than the preset number of frames.
- the synchronized positioning and mapping system adds the current frame to the key frame group to be updated, the current frame becomes a new key frame to be applied.
- the keyframe group containing 7 keyframes is the updated keyframe group to be updated. In this process, the system There is no need to do any processing on the original 6 keyframes to be applied in the keyframe group to be updated.
- the current frame will be updated to the key frame group to be updated, and the key frame to be applied with the longest distance from the current time will be removed from the key frame group to be updated. Remove to get the updated keyframe group to be updated.
- the sliding window can include at most 10 adjacent key frames as key frames to be applied
- the preset number of frames is 10.
- the number of key frames to be applied included in the current key frame group to be updated is The number is also 10, that is, the number of key frames to be updated is equal to the preset number of frames.
- the synchronized positioning and mapping system adds the current frame to the key frame group to be updated, in order to ensure that the number of key frames to be updated is
- the number of keyframes to be applied is equal to the number of preset frames. It is necessary to select the keyframe to be applied that carries the highest timestamp among the 10 keyframes to be applied as historical keyframes, and add the keyframe to be applied. Remove it from the keyframe group to be updated to ensure the consistency of the sliding window size.
- the key frames to be applied can be optimized to update the relative pose of each frame.
- the Bundle Adjustment method can extract the 3D point coordinates, relative motion parameters and camera optical parameters that describe the scene structure based on the projection of all points in the image as a standard. It can be understood that for any three-dimensional point P in the scene, the light emitted from the optical center of the camera corresponding to each view and passing through the pixel corresponding to P in the image will intersect at point P. For all three-dimensional points , then a considerable number of light beams are formed. In the actual application process, due to the existence of noise and other factors, it is almost impossible for each light beam to converge at one point. Therefore, in the process of solving, it is necessary to continuously adjust the information to be sought, so that the final The ray can intersect at point P.
- the ultimate purpose of the beam adjustment method is to reduce the error in the positional projection transformation between the key frame to be applied as the observation image and the reference image or predicted image point, so as to obtain the best three-dimensional structure and motion (such as camera matrix )Parameter Estimation.
- the beam adjustment method usually uses the sparsity of the BA model to perform calculations.
- the steepest descent method, Newton type method, LM method, etc. may be involved, and the embodiments of the present disclosure are not limited to this.
- the relative pose of the key frame to be applied can be updated.
- the updated relative pose and the updated key frame to be applied in the key frame group to be updated can be
- the optimized pose of the keyframe is sent to the graphics processor (Graphics Processing Unit, GPU), and the map data is updated based on the graphics processor.
- the synchronized positioning and mapping system determines the updated relative pose and optimized pose of each key frame to be applied, this information can be written to the rendering engine, so that the rendering engine can display the The corresponding picture is rendered in, where the rendering engine is the program that controls the GPU to render the relevant images, that is, it allows the computer to complete the mapping task of the scene captured by the camera.
- the relative pose information of the key frames to be applied will be used based on the received As well as optimizing the pose information, you can continue to draw the multi-level stairs between the two floors and the map of the second floor in the display interface, thereby updating the map data.
- the technical solution of the embodiment of the present disclosure locates a key frame group to be updated including at least one key frame to be applied based on the synchronized positioning and mapping system, and determines whether the current frame received is a key frame. If it is determined that the current frame is a key frame , then the key frame group to be updated is updated according to the preset number of frames and the current frame, thereby obtaining the updated key frame group to be updated, and the key frames to be applied in the updated key frame group to be updated are optimized, so as to update the key frame group to be updated.
- Applying the relative pose of key frames to perform image rendering based on the updated relative pose not only improves the positioning accuracy of the SLAM space, optimizes the rendering effect of the image, but also avoids the inconvenience of extracting and matching feature points in the image. It reduces the computational overhead, improves image rendering efficiency, and ensures real-time processing of images captured by mobile devices.
- Figure 2 is a schematic structural diagram of a device for rendering images provided by an embodiment of the present disclosure. As shown in Figure 2, the device includes: a key frame determination module 210, an update module 220, and a key frame optimization module to be applied 230.
- the key frame determination module 210 is configured to determine whether the received current frame is a key frame based on the key frame group to be updated based on the synchronization positioning and mapping system positioning; wherein the key frame group to be updated includes at least one key to be applied. frame;
- the update module 220 is configured to, in response to determining that the current frame received is a key frame, update the key frame group to be updated according to the preset number of frames and the current frame, and obtain an updated key frame group to be updated;
- the key frame to be applied optimization module 230 is configured to optimize the key frames to be applied in the updated key frame group to be updated, update the relative pose of the key frame to be applied, and perform image processing based on the updated relative pose. render.
- the device for rendering images also includes an initial key frame determination module and a key frame to be applied determination module.
- An initial key frame determination module configured to perform preprocessing on the multiple continuous frame images when receiving the multiple continuous frame images for the first time, and determine the at least one initial key frame; wherein the preprocessing includes removing the influence of rotation operation.
- the key frame to be applied determining module is configured to use the at least one initialization key frame as each key frame to be applied in the key frame group to be updated.
- the device for rendering images also includes an optimization pose determination module.
- the optimization pose determination module is configured to determine the point cloud data to be processed in the current frame based on the corner detection algorithm, to process the point cloud data to be processed based on the at least one key frame to be applied, and obtain the optimization of the current frame. pose to determine whether the current frame is a keyframe.
- the key frame determination module 210 includes a target feature point determination unit and a key frame determination unit.
- the target feature point determination unit is configured to determine the target feature point of the current frame and the displacement disparity between the current frame and the at least one key frame to be applied.
- the key frame determination unit is configured to determine the current frame as a key frame when the number of the target feature points reaches a first preset quantity threshold and the displacement disparity is greater than the first preset displacement disparity threshold.
- the target feature point determination unit is further configured to determine common view feature points of the current frame and the at least one key frame to be applied, and perform downsampling processing in the current frame based on the common view feature points to determine Target feature points; and, determine the displacement deviation between the current frame and the key frame to be applied.
- the key frame determination unit is also configured to determine if the number of target feature points is less than the number of feature points to be processed in the current frame, and the displacement deviation is less than the second preset displacement deviation.
- the current frame is a keyframe.
- the target feature point determination unit is also configured to downsample the point cloud data to be processed of the current frame to obtain the target feature point.
- the key frame determination unit is also configured to determine the displacement deviation between the current frame and the at least one key frame to be applied; if the number of target feature points is less than or equal to the number of common view feature points, and the If the displacement deviation is less than the third preset displacement deviation, the current frame is determined to be a key frame; wherein the common view feature point is the common view point of the current frame and the key frame to be applied.
- the update module 220 is also configured to update the current frame to the key frame group to be updated if the number of the at least one key frame to be applied is less than the preset number of frames, and obtain the updated The key frame group to be updated; if the number of the at least one key frame to be applied is greater than or equal to the preset number of frames, the current frame will be updated to the key frame group to be updated, and will be compared with the current moment The key frames to be applied with the longest interval are removed from the key frame group to be updated to obtain an updated key frame group to be updated.
- the key frame to be applied optimization module 230 is also configured to optimize each key frame to be applied based on the beam adjustment method and update the relative pose of each key frame to be applied.
- the device for rendering images also includes a map data update module.
- the map data update module is set to optimize each key frame to be applied based on the beam adjustment method, and update the relative pose of each key frame to be applied.
- the technical solution provided by this embodiment locates a key frame group to be updated including at least one key frame to be applied based on the synchronized positioning and mapping system, and determines whether the current frame received is a key frame, such as If it is determined that the current frame is a key frame, the key frame group to be updated is updated according to the preset number of frames and the current frame, thereby obtaining the updated key frame group to be updated, and each application in the updated key frame group to be updated is
- the key frames are optimized to update the relative pose of each key frame to be applied, and the image is rendered based on the updated relative pose. This not only improves the positioning accuracy of the SLAM space, optimizes the rendering effect of the image, but also avoids the need to modify the image. It reduces the computational overhead caused by extracting and matching feature points, improves image rendering efficiency, and ensures real-time processing of images captured by mobile devices.
- the image rendering device provided by the embodiments of the present disclosure can execute the image rendering method provided by any embodiment of the present disclosure, and has corresponding functional modules for executing the method.
- FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
- Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, PDA), PAD (tablet computers), portable multimedia players (Portable Media Player , PMP), mobile terminals such as vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital televisions (Television, TV), desktop computers, etc.
- PDA Personal Digital Assistant
- PAD tablet computers
- PMP portable multimedia players
- mobile terminals such as vehicle-mounted terminals (such as vehicle-mounted navigation terminals)
- fixed terminals such as digital televisions (Television, TV), desktop computers, etc.
- the electronic device shown in FIG. 3 is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.
- the electronic device 300 may include a processing device (such as a central processing unit, a pattern processor, etc.) 301 , which may process data according to a program stored in a read-only memory (Read-Only Memory, ROM) 302 or from a storage device.
- a processing device such as a central processing unit, a pattern processor, etc.
- ROM read-only memory
- 306 loads the program in the random access memory (Random Access Memory, RAM) 303 to perform various appropriate actions and processes.
- RAM 303 Random Access Memory
- various programs and data required for the operation of the electronic device 300 are also stored.
- Processing device 301, ROM 302 and RAM 303 are connected to each other via bus 304.
- An input/output (I/O) interface 305 is also connected to bus 304.
- an editing device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; including, for example, a Liquid Crystal Display (LCD) , an output device 307 such as a speaker, a vibrator, etc.; a storage device 308 including a magnetic tape, a hard disk, etc.; and a communication device 309.
- the communication device 309 may allow the electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data.
- FIG. 3 illustrates electronic device 300 with various means, it should be understood that implementation or availability of all illustrated means is not required. More or fewer means may alternatively be implemented or provided.
- embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
- the computer program may be downloaded and installed from the network via communication device 309, or from storage device 306, or from ROM 302.
- the processing device 301 When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
- Embodiments of the present disclosure provide a computer storage medium on which a computer program is stored.
- the program is executed by a processor, the method for rendering an image provided in the above embodiments is implemented.
- the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
- the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system or device. device or device, or any combination of the above.
- Computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), removable Programmable read-only memory ((Erasable Programmable Read-Only Memory, EPROM) or flash memory), optical fiber, portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or the above any suitable combination.
- a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.
- a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
- a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
- Program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (Radio Frequency, RF), etc., or any suitable combination of the above.
- the client and server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium.
- Communications e.g., communications network
- Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any current network for knowledge or future research and development.
- LANs Local Area Networks
- WANs Wide Area Networks
- the Internet e.g., the Internet
- end-to-end networks e.g., ad hoc end-to-end networks
- the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.
- the above-mentioned computer-readable medium carries at least one program.
- the electronic device executes the above-mentioned at least one program.
- the key frame group to be updated Based on the key frame group to be updated based on the synchronization positioning and mapping system positioning, determine whether the current frame received is No, it is a key frame; wherein, the key frame group to be updated includes at least one key frame to be applied;
- Each key frame to be applied in the updated key frame group to be updated is optimized, and the relative pose of each key frame to be applied is updated to perform image rendering based on the updated relative pose.
- Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages—such as "C” or similar programming languages.
- the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through Internet connection).
- LAN local area network
- WAN wide area network
- Internet service provider such as an Internet service provider through Internet connection
- each block in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions.
- the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved.
- each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of specialized hardware and computer instructions.
- the units involved in the embodiments of the present disclosure can be implemented in software or hardware. Among them, the name of the unit does not constitute a reference to the unit itself under certain circumstances.
- the first acquisition unit may also be described as "a unit that acquires at least two Internet Protocol addresses.”
- exemplary types of hardware logic components include: field programmable gate array (Field Programmable Gate Array, FPGA), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), application specific standard product (Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), etc.
- a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
- the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
- Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing.
- machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
- RAM random access memory
- ROM read only memory
- EPROM or flash memory erasable programmable read only memory
- CD-ROM portable compact disk read-only memory
- magnetic storage device or any suitable combination of the above.
- Example 1 provides a method for rendering an image, which method includes:
- Each key frame to be applied in the updated key frame group to be updated is optimized, and the relative pose of each key frame to be applied is updated to perform image rendering based on the updated relative pose.
- Example 2 provides a method for rendering an image.
- the method further includes:
- preprocessing is performed on the multiple continuous frame images to determine the at least one initial key frame; wherein the preprocessing includes an operation of removing the influence of rotation;
- the at least one initialization key frame is used as each key frame to be applied in the key frame group to be updated.
- Example 3 provides a method for rendering an image. The method further includes:
- Example 4 provides a method for rendering an image, and the method further includes:
- the current frame is determined to be a key frame.
- Example 5 provides a method for rendering an image.
- the method further includes:
- determine the common view feature points of the current frame and the at least one key frame to be applied and perform downsampling processing in the current frame based on the common view feature points to determine the target feature points; and, determine the current frame The displacement deviation from the key frame to be applied;
- the current frame is determined to be a key frame.
- Example 6 provides a method of rendering an image, the Methods also include:
- the current frame is determined to be a key frame
- the common view feature point is the common view point of the current frame and the key frame to be applied.
- Example 7 provides a method for rendering an image, and the method further includes:
- the current frame is updated to the key frame group to be updated, and the key to be applied with the longest distance from the current time is Frames are removed from the key frame group to be updated to obtain an updated key frame group to be updated.
- Example 8 provides a method for rendering an image. The method further includes:
- each key frame to be applied based on the beam adjustment method, and update the relative pose of each key frame to be applied.
- Example 9 provides a method for rendering an image, the method further includes:
- the updated relative pose and the optimized pose are sent to a graphics processor to update map data based on the graphics processor.
- Example 10 provides a device for rendering images, the device including:
- the key frame determination module is configured to determine whether the received current frame is a key frame based on the key frame group to be updated based on the synchronization positioning and mapping system positioning; wherein the key frame group to be updated includes at least one key frame to be applied. ;
- An update module configured to respond to determining that the received current frame is a key frame, update the key frame group to be updated according to the preset number of frames and the current frame, and obtain an updated key frame group to be updated;
- the key frame optimization module to be applied is configured to optimize each key frame to be applied in the updated key frame group to be updated, and update the relative pose of each key frame to be applied, so as to perform the operation based on the updated relative pose.
- Image rendering is configured to optimize each key frame to be applied in the updated key frame group to be updated, and update the relative pose of each key frame to be applied, so as to perform the operation based on the updated relative pose.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
本公开实施例提供了一种渲染图像的方法、装置、电子设备及存储介质,该方法包括:基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧;其中,所述待更新关键帧组中包括至少一个待应用关键帧;响应于确定接收到的当前帧为关键帧,根据预设帧数和所述当前帧对所述待更新关键帧组进行更新,得到更新后的待更新关键帧组;对所述更新后的待更新关键帧组中的待应用关键帧进行优化,更新待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染。
Description
本申请要求在2022年5月9日提交中国专利局、申请号为202210501160.9的中国专利申请的优先权,该申请的全部内容通过引用结合在本申请中。
本公开实施例涉及图像处理技术领域,例如涉及一种渲染图像的方法、装置、电子设备及存储介质。
随着计算机视觉技术的发展,同步定位与建图(Simultaneous Localization and Mapping,SLAM)算法被广泛应用于如增强现实、虚拟现实、自动驾驶以及机器人或者无人机的定位导航等领域。
根据SLAM算法,可以构建出多种类型的***以执行相应的渲染任务,例如,基于滤波的SLAM***以及基于特征点的SLAM***等。然而在实际应用过程中,基于滤波的SLAM***无法长时间提供较为精确的相机位姿信息、以及拍摄得到的空间信息,这就导致***渲染得到的图像效果较差,而基于特征点的SLAM***又需要对图像中的特征点进行提取,并对各帧画面中的特征点进行匹配,这种方式的缺点在于,不仅增大了图像处理过程中的计算开销,也很难对移动端拍摄的图像进行实时处理,影响用户的使用体验。
发明内容
本公开提供一种渲染图像的方法、装置、电子设备及存储介质,提高了SLAM空间的定位精度,优化了图像的渲染效果,同时,提高了图像渲染效率,保证了对移动端所拍摄图像处理的实时性。
第一方面,本公开实施例提供了一种渲染图像的方法,包括:
基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是
否为关键帧;其中,所述待更新关键帧组中包括至少一个待应用关键帧;
响应于确定接收到的当前帧为关键帧,根据预设帧数和所述当前帧对所述待更新关键帧组进行更新,得到更新后的待更新关键帧组;
对所述更新后的待更新关键帧组中的待应用关键帧进行优化,更新待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染。
第二方面,本公开实施例还提供了一种渲染图像的装置,包括:
关键帧确定模块,设置为基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧;其中,所述待更新关键帧组中包括至少一个待应用关键帧;
更新模块,设置为响应于确定接收到的当前帧为关键帧,根据预设帧数和所述当前帧对所述待更新关键帧组进行更新,得到更新后的待更新关键帧组;
待应用关键帧优化模块,设置为对所述待更新关键帧组中的待应用关键帧进行优化,更新待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染。
第三方面,本公开实施例还提供了一种电子设备,所述电子设备包括:
至少一个处理器;
存储装置,设置为存储至少一个程序,
当所述至少一个程序被所述至少一个处理器执行,使得所述至少一个处理器实现如本公开实施例任一所述的渲染图像的方法。
第四方面,本公开实施例还提供了一种包含计算机可执行指令的存储介质,所述计算机可执行指令在由计算机处理器执行时用于执行如本公开实施例任一所述的渲染图像的方法。
贯穿附图中,相同或相似的附图标记表示相同或相似的元素。应当理解附图是示意性的,原件和元素不一定按照比例绘制。
图1为本公开实施例所提供的一种渲染图像的方法流程示意图;
图2为本公开实施例所提供的一种渲染图像的装置结构示意图;
图3为本公开实施例所提供的一种电子设备的结构示意图。
下面将参照附图更详细地描述本公开的实施例。
应当理解,本公开的方法实施方式中记载的各个步骤可以按照不同的顺序执行,和/或并行执行。此外,方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括,即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”;术语“另一实施例”表示“至少一个另外的实施例”;术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
需要注意,本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分,并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。需要注意,本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的,本领域技术人员应当理解,除非在上下文另有明确指出,否则应该理解为“至少一个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的,而并不是用于对这些消息或信息的范围进行限制。
在介绍本技术方案之前,可以先对本公开实施例的应用场景进行示例性说明。示例性的,当用户利用移动端的摄像装置拍摄视频,并将拍摄得到的视频上传至基于SLAM算法的***,或者,在数据库中选择目标视频,并将视频主动上传至基于SLAM算法的***后,***即可对视频进行解析处理,然而,相关技术中的SLAM***很难长时间提供较为精确的相机位姿信息以及空间信息,最终的图像渲染效果较差,或者,SLAM***需要对视频帧画面中的特征点进
行提取,并进行特征匹配,在这一过程中,较大的计算开销导致***很难对移动端拍摄的视频进行实时处理。此时,可以基于本公开实施例的方案,直接在视频中确定出待更新关键帧组,并根据预设帧数以及当前的关键帧对待更新关键帧组进行更新,对其中的关键帧进行优化后,即可得到关键帧的相对位姿,从而提高了SLAM空间的定位精度,得到更加优异的渲染结果,同时,本公开实施例的SLAM***也无需对图像中特征点进行提取与匹配,减少了计算开销,便于对移动端上传的图像进行实时处理。
图1为本公开实施例所提供的一种渲染图像的方法流程示意图,本公开实施例适用于基于SLAM***对视频进行处理,从而在显示界面上实时渲染出相应的多帧图像的情形,该方法可以由渲染图像的装置来执行,该装置可以通过软件和/或硬件的形式实现,可选的,通过电子设备来实现,该电子设备可以是移动终端、PC端或服务器等。
如图1所示,所述方法包括:
S110、基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧。
其中,同步定位与建图(Simultaneous Localization and Mapping,SLAM)技术主要用于解决移动机器人在未知环境中运行时的定位导航与地图构建问题,可以理解,本公开实施例中的同步定位与建图***即是集成有SLAM相关算法的***,在其算法中通常包括特征提取、数据关联、状态估计、状态更新以及特征更新等几个部分,而对于其中每个部分来说,又均存在多种处理方法,本公开实施例对此不做限定。
在本实施例中,执行本公开实施例提供的渲染图像的方法的同步定位与建图***,可以集成在支持特效视频处理功能的应用软件中,且该软件可以安装至电子设备中,可选的,电子设备可以是移动终端或者PC端等。应用软件可以是对图像/视频处理的一类软件,其具体的应用软件在此不再一一赘述,只要可以实现图像/视频处理即可。还可以是专门研发的应用程序,来实现添加特效并
将特效进行展示的软件中,亦或是集成在相应的页面中,用户可以通过PC端中集成的页面来实现对特效视频的处理。
需要说明的是,本实施例的技术方案可以在基于移动端实时摄像的过程中执行,也可以在***接收到用户主动上传的视频数据后执行,同时,本公开实施例的方案可以应用于增强现实(Augumented Reality,AR)、虚拟现实(Virtual Reality,VR)以及自动驾驶等多种应用场景内。
在本实施例中,基于SLAM***对图像进行渲染之前,首先需要在接收或获取的视频数据中定位出待更新关键帧组。其中,待更新关键帧组即是包含多个关键帧的集合,基于本公开实施例中的SLAM***,还可以对待更新关键帧组中的图像进行更新。同时,在待更新关键帧组中包括至少一个待应用关键帧,本领域技术人员应当理解,在计算机视觉技术领域内,关键帧用于代表与其相邻的多个帧,相当于SLAM的骨架,是在局部一系列普通帧中选出一帧作为局部帧的代表,因此,在关键帧中至少记录着视频画面的局部信息。同时,利用关键帧执行后续的图像渲染处理过程,也可以有效减少需要优化的视频帧数,从而提升***的图像处理效率。
示例性的,SLAM***接收视频数据后,可以将视频存储至预设序列中,例如,该预设序列可以按照上述视频中每一帧的顺序存储上述视频,如,上述视频中每一帧的顺序是先帧1,然后帧2…再是帧n-1,最后是帧n,上述预设序列按照上述顺序,即以帧1、帧2…帧n-1和帧n的顺序存储上述视频。同时,上述多个视频帧共同构成一个待更新关键帧组,其中帧1、帧10、帧20…帧n即是分别代表其相邻的多个帧的待应用关键帧。
在本实施例中,在SLAM***确定接收到的当前帧是否为关键帧之前,还可以在首次接收到多个连续帧图像时,对多个连续帧图像进行预处理,确定至少一个初始关键帧;将至少一个初始化关键帧作为待更新关键帧组中的至少一个待应用关键帧。
其中,多个连续帧图像可以是***从接收的视频数据中解析出来的图像,
如,上述示例中的帧1、帧2…帧n-1和帧n,本领域技术人员应当理解,多个连续帧图像可以根据实际情况确定。同时,***可以预先构建出一个自适应大小的滑动窗口,从而在接收到上述多个连续帧图像后,对图像进行预处理并利用该滑动窗口从中筛选出至少一个初始关键帧。
在本实施例中,预处理包括去除旋转影响的操作,这里,上述预处理的原因是:在多帧连续的视频帧中,画面可能会出现旋转的情况,旋转会影响帧的像素距离差,但是仅有旋转无法进行同步定位与建图初始化。因此,本公开实施例为了解决该问题,进行了上述预处理,并利用去除旋转影响的像素距离差筛选窗口内的至少一个初始关键帧,从而保证窗口中的帧之间有足够公视的前提下拥有足够的视差进行同步定位与建图初始化,可以理解为,通过去除旋转操作,减少了旋转对同步定位与建图初始化的影响,提高了同步定位与建图初始化的精度。
在实际应用过程中,***可以从惯性测量单元中获取上述旋转的信息,从而基于获取的信息确定旋转影响帧的像素距离差,对上述多个连续帧图像进行去除旋转影响的处理,并利用去除旋转影响的像素距离差筛选出上述滑动窗口内的至少一个初始关键帧。
可选的,***利用预先构建的自适应大小的滑动窗口,即可在去除旋转影响的上述多个连续帧图像中筛选出至少一个初始关键帧,下面对这一过程进行说明。
示例性的,确定多个关键帧中的第一关键帧和最后一关键帧的相对位姿;根据第一关键帧和最后一关键帧的相对位姿,获得多个关键帧中各个关键帧的三维空间点;根据第一关键帧和最后一关键帧的相对位姿,以及多个关键帧中各个关键帧的三维空间点,确定多个关键帧中各个关键帧的相对位姿;根据多个关键帧中各个关键帧的三维空间点和多个关键帧中各个关键帧的相对位姿,建立初始地图。在建立初始地图后,即可执行针对于多个连续帧图像的预处理操作。
示例性的,***预先构建一个大小可调的滑动窗口,如,5帧图像帧大小的~10帧图像帧大小的滑动窗口,利用该滑动窗口,即可在去除旋转影响的多个连续帧图像中筛选出至少一个初始关键帧,例如,当前上述滑动窗口的长度为5帧,***利用去除旋转影响的像素距离差筛选出上述滑动窗口内的初始关键帧,例如,对从接收的视频中解析出来的帧1、帧2…帧25进行筛选,筛选出帧6、帧7、帧10、帧12以及帧13作为上述初始关键帧。基于此,若通过这5帧图像帧的大小无法正确进行同步定位与建图初始化,则增加滑动窗口大小为6帧,并按照上述方式继续筛选去除旋转影响后的初始关键帧,进行初始化计算以及滑动窗口,直至同步定位与建图初始化完成。可以理解,所得到的至少一个初始化关键帧即是待更新关键帧组中的至少一个待应用关键帧。
在本实施例中,***基于从多个连续帧图像中筛选出来的初始关键帧进行同步定位与建图初始化,减少了同步定位与建图初始化的时间,而且,***通过利用去除旋转影响的像素距离差筛选窗口内的初始关键帧,在保证窗口中的帧之间有足够公视的前提下拥有足够的视差进行同步定位与建图初始化,同时,减少了旋转对同步定位与建图初始化的影响,提高了同步定位与建图初始化的精度。
需要说明的是,在确定接收到的当前帧是否为关键帧之前,还包括基于角点检测算法确定当前帧中的待处理点云数据,以基于至少一个待应用关键帧对待处理点云数据进行处理,得到当前帧的优化位姿,以确定当前帧是否为关键帧。
在本实施例中,当***接收到当前帧时,首先需要基于角点检测算法确定出当前帧内的点云数据(Point Cloud Data,PCD)。其中,点云数据通常用于逆向工程中,是一种以点的形式记录的数据,这些点既可以是三维空间中的坐标,也可以是颜色或者光照强度等信息,在实际应用过程中,点云数据一般还包括点坐标精度、空间分辨率和表面法向量等内容,一般以PCD格式进行保存,在这种格式下,点云数据的可操作性较强,能够在后续过程中提高点云配准和融
合的速度,本公开实施例对此不再赘述。可以理解,在本实施例中,当前帧中的点云数据即是待处理点云数据。
在实际应用过程中,***所采用的角点检测算法可以是KLT角点检测法,也称为KLT光流跟踪法。KLT角点检测法用于满足Lucas-Kanade光流法选择合适特征点的需求,Lucas-Kanade光流法是通过先在前后两帧图像中分别建立一个固定大小的窗口,然后确定出让两个窗口间像素强度差的平方和最小的位移,将窗口内像素的移动近似为这样的位移向量。然而在实际应用过程中,像素移动较为复杂,同时,窗口内的像素并非都按照同样的方式移动,这种近似的方式必然带来误差,因此,KLT角点检测方法即为了选择一个适合跟踪的特征点,可以理解为,好的特征点即是能够被***更好地跟踪的点。在利用KLT角点检测方法确定待处理点云数据的过程中,包括确定像素点光强度函数、将窗口内偏差能量调整为最小、角点选择、特征点选择以及设置能量偏差函数的阈值以排除被阻挡的点等多个步骤,本公开实施例在此不再赘述。
在本实施例中,利用KLT角点检测法确定出当前帧中的点云数据,无需提取当前帧中的描述子,也无需再执行特征点匹配的操作,从而增强了***对数据处理的实时性与鲁棒性,使***在角点跟踪并确定待处理点云数据的过程中,实现了高效的角点跟踪。
在本实施例中,当得到当前帧中的待处理点云数据后,即可基于待应用关键帧对这些点云数据进行处理,从而得到当前帧的优化位姿。本领域技术人员应当理解,带有相机位姿和空间点的图优化称为BA,能够有效地求解大范围的定位与建图问题,然而,随着规模的不断扩大,计算效率会大幅下降,在这一过程中,特征点的优化问题占了较多的部分,经过若干次迭代之后,特征点就会收敛,此时再进行优化没有更大的意义。因此,在实际过程中,在优化几次后,便可以把特征点固定住,将其看做位姿估计的约束,即,不再优化特征点的位姿。基于此可以理解,优化后的位姿图即是在只考虑位姿的情况下,所构建的一个只有轨迹的图优化,而位姿节点之间的边,由两个关键帧之间通过特
征匹配后得到的运动估计来给定初始值,一旦初始值确定,则不再优化路标点的位置,只关心相机位姿之间的联系。在本实施例中,优化位姿即是基于当前帧位姿图所确定的信息,根据这些信息,***即可判断当前帧是否为关键帧。
在本实施例中,利用上述增量式的BA问题构造方法来确定当前帧的优化位姿,可以使同步定位与建图***提供较高的BA速度,从而保证了***对视频帧进行处理的实时性。
在本实施例中,基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧的方式有多种,下面对各种方式逐一进行说明。
可选的,确定当前帧的目标特征点,以及当前帧与至少一个待应用关键帧的位移视差;在目标特征点的数量大于第一预设数量阈值,且位移视差大于第一预设位移视差阈值,则确定当前帧为关键帧。
其中,由于相机处于不断运动的状态,因此,所拍摄物体在图像上产生运动,从而形成位移视差,可以理解,通过位移视差至少可以判断各帧图像中物体的远近。目标特征点即是从各帧图像中的物体上所确定出来的点,例如,某一帧图像上存在多级阶梯,***根据预先训练好的特征点确定算法,即可从其中每一节阶梯上确定出相应的多个特征点,这些特征点即是目标特征点,***可以将所确定的多个特征点作为一种标识,从而计算出相机位姿的变化。本领域技术人员应当理解,在实际应用过程中,***所确定的目标特征点可以是多种类型,如尺度不变特征变换(Scale-invariant feature transform,SIFT)特征点、加速稳健特征(Speeded Up Robust Features,SURF)特征点以及ORB(Oriented FAST and Rotated BRIEF)特征点等,目标特征点的类型可以根据实际情况进行选择,本公开实施例对此不作限定。
在本实施例中,***还可以针对目标特征点的数量这一参数预先设置一个阈值,该阈值即是第一预设数量阈值,同样地,针对位移视差这一参数也预先设置一个阈值,该阈值即是第一预设位移视差阈值。基于此,当***从当前帧中确定出目标特征点,并确定当前帧与至少一个待应用关键帧之间的位移视差
后,即可对目标特征点的数量以及位移视差值进行判断,当两者均大于相对应的预设阈值时,则确定当前帧为关键帧。
示例性的,当***的第一预设数量阈值为100且第一预设位移视差阈值为100个像素时,若确定出当前帧中的目标特征点个数为300,且确定当前帧与至少一个待应用关键帧之间的位移视差也达到300个像素点的长度后,即可确定上述两个参数均大于其相应的预设阈值,此时,***可以确定当前帧为关键帧,本领域技术人员应当理解,若上述两个参数中有任意一个小于或等于其相应的预设阈值,或者,两个参数均小于或等于其相应的预设阈值,当前帧则不会被判定为关键帧,并在抛弃当前帧后,按照上述方式继续对后续接收的多个帧逐一进行判断,本公开实施例对此不再赘述。
可选的,确定当前帧与至少一个待应用关键帧的共视特征点,并基于共视特征点在当前帧中降采样处理,确定目标特征点;以及,确定当前帧与至少一个待应用关键帧的位移偏差;若目标特征点的数量小于当前帧中待处理特征点的数量,且位移偏差小于第二预设位移偏差,则确定当前帧为关键帧。
示例性的,***在接收到当前帧并确定出其画面中的多个特征点后,还可以将这些特征点与至少一个待应用关键帧对应画面中的特征点进行比对,从而确定出这些图像中的共视特征点,例如,对多个视频帧图像中特征点关联的关键点以及描述子进行匹配比对,从而确定出共视特征点。可以理解,共视特征点即是当前帧与待应用关键帧的共视点,继续以上述示例进行说明,当***从接收的当前帧中,确定出与画面中多个阶梯对应的特征点后,需要将这些特征点与其他待应用关键帧中的特征点进行比对,当某一个待应用关键帧的画面中也包含这多级阶梯,即,也包含上述特征点时,***即可将这两幅视频帧画面中多级阶梯对应的特征点确定为共视特征点。
在本实施例中,当***确定出当前帧与待应用关键帧之间的共视特征点后,即可对当前帧中的共视特征点进行降采样处理,进而从这些共视特征点中筛选出目标特征点。本领域技术人员应当理解,在数字信号处理领域中,降采样是
一种多速率数字信号处理技术,同时也是降低信号采样率的过程,通常用于降低数据传输速率或数据大小,例如,对当前帧中160个共视特征点进行4倍降采样处理后,即可从中筛选出40个特征点作为目标特征点,可以理解,上述示例中采用的4倍这一参数即是降采样率,该参数用于表达采样周期变为原来的M倍,或者采样率变为原来的1/M倍,同时,降采样率可以以手动或自动的方式预先设置,本公开实施例对此不作限定。
在本实施例中,在***确定目标特征点的过程中,还需要确定当前帧与待应用关键帧的位移偏差,其中,位移偏差即是表征相机位姿变化的信息。例如,当前帧是相机处于场景内A点时拍摄得到的,某一待应用关键帧是相机处于相同场景内B点时拍摄得到的,对于这两帧图像来说,相机从B点移动至A点所产生的位姿的变化,即是***从两帧图像中确定出来的位移偏差。
在本实施例中,***同样可以针对位移偏差这一参数预设设置一个阈值,该阈值即是第二预设位移偏差。基于此,当***从当前帧中确定出目标特征点,并确定出确定当前帧与至少一个待应用关键帧之间的位移偏差后,即可将目标特征点的数量与当前帧中待处理特征点的数量进行比对,同时将当前帧与待应用关键帧之间的位移偏差与第二预设位移偏差进行比对,当上述两个参数均小于其对应的比对对象时,则确定当前帧为关键帧。
可选的,对当前帧的待处理点云数据降采样处理,得到目标特征点;确定当前帧与至少一个待应用关键帧的位移偏差;若目标特征点的数量小于等于共视特征点的数量,且位移偏差小于第三预设位移偏差,则确定当前帧为关键帧。
示例性的,当***接收到当前帧后,可以对当前帧中的待处理点云数据进行降采样处理,如,通过体素网格实现点云数据的降采样,当***采用这种方式对点云数据进行降采样处理时,在减少点云数据中点的数量的同时,依然可以保证点云的形状,还可以提高配准、曲面重建、形状识别等算法的速度,并保证降采样处理的准确性。通过对点云数据降采样处理,从点云数据中筛选得到的特征点同样可以作为目标特征点。同时,***还可以按照本公开实施例上
述方式,确定出当前帧与至少一个待应用关键帧之间的位移偏差,本公开实施例在此不再赘述。
在本实施例中,***可以针对位移偏差这一参数预先设置一个阈值,该阈值即是第三预设位移偏差,将目标特征点的数量,与多个视频帧之间共视特征点的数量进行比对,并将位移偏差与第三预设位移偏差进行比对,当上述两个参数均小于其相应的比对对象时,则确定当前帧为关键帧。
S120、响应于确定接收到的当前帧为关键帧,根据预设帧数和当前帧对待更新关键帧组进行更新,得到更新后的待更新关键帧组。
在本实施例中,当基于同步定位与建图***确定接收到的当前帧为关键帧后,则需要将该关键帧添加至待更新关键帧组中,从而实现对待更新关键帧组的更新。这一过程可以理解为,对局部地图信息的更新优化。
同时,在本实施例的方案中,为了保证焦点跟踪的一致性与准确性,采用滑窗结构维护局部的地图信息,其中,滑窗中可以包括相邻的多个帧,以及基于这些相邻帧所观测到的空间点。基于此,在对地图信息进行局部优化的过程中,需要利用历史信息对当前帧进行约束,从而提高局部地图信息优化结果的准确性,在实际应用过程中,可以针对滑窗结构预设一个帧数,同时,该帧数也决定了待更新关键帧组中关键帧的数量,可以理解,通过严格控制待更新关键帧组中关键帧的数量,也间接地控制了局部地图点的数量,从而有效地控制后续的计算效率,便于同步定位与建图***对移动端所拍摄图像的实时处理。
可选的,若至少一个待应用关键帧的数量小于预设帧数,则将当前帧更新至待更新关键帧组中,得到更新后的待更新关键帧组。
示例性的,当滑动窗口中最多可以包括相邻的10个关键帧作为待应用关键帧时,预设帧数即是10,同时,当前待更新关键帧组中包含的待应用关键帧的个数为6,即待应用关键帧的数量小于预设帧数,此时,同步定位与建图***将当前帧添加至待更新关键帧组中后,当前帧便成为一个新的待应用关键帧,包含7个关键帧的关键帧组即是更新后的待更新关键帧组,在这一过程中,***
无需针对待更新关键帧组中原有的6个待应用关键帧做任何处理。
若至少一个待应用关键帧的数量大于或等于预设帧数,则将当前帧更新至待更新关键帧组中,并将与当前时刻间隔最长的待应用关键帧从待更新关键帧组中移除,得到更新后的待更新关键帧组。
示例性的,当滑动窗口中最多可以包括相邻的10个关键帧作为待应用关键帧时,预设帧数即是10,同时,当前待更新关键帧组中包含的待应用关键帧的个数同样为10,即待更新关键帧的数量与预设帧数相等,此时,同步定位与建图***将当前帧添加至待更新关键帧组中后,为了保证待更新关键帧组中的待应用关键帧的数量与预设帧数相等,需要在作为历史关键帧的10个待应用关键帧中,选择出所携带的时间戳处于最前的一个待应用关键帧,并将该待应用关键帧从待更新关键帧组中剔除,从而保证滑窗大小的一致性。
S130、对更新后的待更新关键帧组中的待应用关键帧进行优化,更新待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染。
在本实施例中,当同步定位与建图***对待更新关键帧组更新完毕后,即可对其中的待应用关键帧进行优化,从而更新每帧的相对位姿。可选的,基于光束平差法对待应用关键帧优化,更新待应用关键帧的相对位姿。
其中,光束平差法(Bundle Adjustment)可以根据所有点在图像中的投影作为标准,同时提炼出描述场景结构的3D点坐标、相对运动参数和相机的光学参数。可以理解为,对场景中任意三维点P,由从每个视图所对应的摄像机的光心发射出来并经过图像中P对应的像素后的光线,都将交于P这一点,对于所有三维点,则形成相当多的光束,在实际应用过程中,由于噪声等因素的存在,每条光线几乎不可能汇聚于一点,因此,在求解的过程中,需要不断对待求信息进行调整,从而使最终光线能够交于点P。可以理解,光束平差法最终的目的即是减少作为观测图像的待应用关键帧与参考图像或预测图像的点之间位置投影变换的误差,从而获得最佳的三维结构和运动(如相机矩阵)参数估计。
在本实施例中,光束平差法通常利用BA模型的稀疏性进行计算,在计算过
程中,可以涉及最速下降法、Newton型方法、LM方法等,本公开实施例对此不作限定。利用光束平差法对待应用关键帧进行优化后,即可实现对待应用关键帧相对位姿的更新。
在本实施例中,对更新后的待更新关键帧组中的待应用关键帧相对位姿进行更新后,即可将更新后的相对位姿和更新后的待更新关键帧组中的待应用关键帧的优化位姿发送至图形处理器(Graphics Processing Unit,GPU),并基于图形处理器更新地图数据。
示例性的,当同步定位与建图***确定出每个待应用关键帧更新后的的相对位姿以及优化位姿后,可以将这些信息写入至渲染引擎中,从而使渲染引擎在显示界面中渲染出对应的画面,其中,渲染引擎即是控制GPU对相关图像进行渲染的程序,即,可以使计算机完成对摄像机所拍摄场景的地图绘制任务。例如,当前场景包括一栋房屋内的一楼楼层以及二楼楼层,且GPU已基于多个历史关键帧绘制出一楼楼层的地图画面时,根据接收到的待应用关键帧的相对位姿信息以及优化位姿信息,可以继续在显示界面中绘制出两个楼层之间的多级楼梯以及二楼的地图,从而实现对地图数据的更新。
当然,在实际应用过程中,根据本公开实施例的方案,还可以对自动驾驶等诸多场景内的地图进行更新,本公开实施例对此不作限定。
本公开实施例的技术方案,基于同步定位与建图***定位出包括至少一个待应用关键帧的待更新关键帧组,并确定接收到的当前帧是否为关键帧,如果确定当前帧为关键帧,则根据预设帧数和当前帧对待更新关键帧组进行更新,从而得到更新后的待更新关键帧组,对更新后的待更新关键帧组中的待应用关键帧进行优化,从而更新待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染,不仅提高了SLAM空间的定位精度,优化了图像的渲染效果,还避免了对图像中特征点进行提取与匹配所带来的计算开销,提高了图像渲染效率,保证了对移动端所拍摄图像处理的实时性。
图2为本公开实施例所提供的一种渲染图像的装置结构示意图,如图2所示,所述装置包括:关键帧确定模块210、更新模块220以及待应用关键帧优化模块230。
关键帧确定模块210,设置为基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧;其中,所述待更新关键帧组中包括至少一个待应用关键帧;
更新模块220,设置为响应于确定接收到的当前帧为关键帧,根据预设帧数和所述当前帧对所述待更新关键帧组进行更新,得到更新后的待更新关键帧组;
待应用关键帧优化模块230,设置为对所述更新后的待更新关键帧组中的待应用关键帧进行优化,更新待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染。
在上述各技术方案的基础上,渲染图像的装置还包括初始关键帧确定模块以及待应用关键帧确定模块。
初始关键帧确定模块,设置为在首次接收到多个连续帧图像时,对所述多个连续帧图像进行预处理,确定所述至少一个初始关键帧;其中,所述预处理包括去除旋转影响的操作。
待应用关键帧确定模块,设置为将所述至少一个初始化关键帧作为所述待更新关键帧组中的各待应用关键帧。
在上述各技术方案的基础上,渲染图像的装置还包括优化位姿确定模块。
优化位姿确定模块,设置为基于角点检测算法确定当前帧中的待处理点云数据,以基于所述至少一个待应用关键帧对所述待处理点云数据进行处理,得到当前帧的优化位姿,以确定当前帧是否为关键帧。
在上述各技术方案的基础上,关键帧确定模块210包括目标特征点确定单元以及关键帧确定单元。
目标特征点确定单元,设置为确定当前帧的目标特征点,以及所述当前帧与所述至少一个待应用关键帧的位移视差。
关键帧确定单元,设置为在所述目标特征点的数量达到第一预设数量阈值,且所述位移视差大于第一预设位移视差阈值,则确定所述当前帧为关键帧。
可选的,目标特征点确定单元,还设置为确定当前帧与所述至少一个待应用关键帧的共视特征点,并基于所述共视特征点在所述当前帧中降采样处理,确定目标特征点;以及,确定当前帧与所述待应用关键帧的位移偏差。
可选的,关键帧确定单元,还设置为若所述目标特征点的数量小于所述当前帧中待处理特征点的数量,且所述位移偏差小于第二预设位移偏差,则确定所述当前帧为关键帧。
可选的,目标特征点确定单元,还设置为对所述当前帧的待处理点云数据降采样处理,得到目标特征点。
可选的,关键帧确定单元,还设置为确定所述当前帧与所述至少一个待应用关键帧的位移偏差;若所述目标特征点的数量小于等于共视特征点的数量,且所述位移偏差小于第三预设位移偏差,则确定所述当前帧为关键帧;其中,所述共视特征点为所述当前帧与所述待应用关键帧的共视点。
可选的,更新模块220,还设置为若所述至少一个待应用关键帧的数量小于所述预设帧数,则将所述当前帧更新至所述待更新关键帧组中,得到更新后的待更新关键帧组;若所述至少一个待应用关键帧的数量大于或等于所述预设帧数,则将所述当前帧更新至所述待更新关键帧组中,并将与当前时刻间隔最长的待应用关键帧从所述待更新关键帧组中移除,得到更新后的待更新关键帧组。
可选的,待应用关键帧优化模块230,还设置为基于光束平差法对各待应用关键帧优化,更新各待应用关键帧的相对位姿。
在上述各技术方案的基础上,渲染图像的装置还包括地图数据更新模块。
地图数据更新模块,设置为基于光束平差法对各待应用关键帧优化,更新各待应用关键帧的相对位姿。
本实施例所提供的技术方案,基于同步定位与建图***定位出包括至少一个待应用关键帧的待更新关键帧组,并确定接收到的当前帧是否为关键帧,如
果确定当前帧为关键帧,则根据预设帧数和当前帧对待更新关键帧组进行更新,从而得到更新后的待更新关键帧组,对更新后的待更新关键帧组中的各待应用关键帧进行优化,从而更新各待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染,不仅提高了SLAM空间的定位精度,优化了图像的渲染效果,还避免了对图像中特征点进行提取与匹配所带来的计算开销,提高了图像渲染效率,保证了对移动端所拍摄图像处理的实时性。
本公开实施例所提供的渲染图像的装置可执行本公开任意实施例所提供的渲染图像的方法,具备执行方法相应的功能模块。
值得注意的是,上述装置所包括的各个单元和模块只是按照功能逻辑进行划分的,但并不局限于上述的划分,只要能够实现相应的功能即可;另外,各功能单元的具体名称也只是为了便于相互区分,并不用于限制本公开实施例的保护范围。
图3为本公开实施例所提供的一种电子设备的结构示意图。下面参考图3,其示出了适于用来实现本公开实施例的电子设备(例如图3中的终端设备或服务器)300的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、个人数字助理(Personal Digital Assistant,PDA)、PAD(平板电脑)、便携式多媒体播放器(Portable Media Player,PMP)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字电视(Television,TV)、台式计算机等等的固定终端。图3示出的电子设备仅仅是一个示例,不应对本公开实施例的功能和使用范围带来任何限制。
如图3所示,电子设备300可以包括处理装置(例如中央处理器、图案处理器等)301,其可以根据存储在只读存储器(Read-Only Memory,ROM)302中的程序或者从存储装置306加载到随机访问存储器(Random Access Memory,RAM)303中的程序而执行各种适当的动作和处理。在RAM 303中,还存储有电子设备300操作所需的各种程序和数据。处理装置301、ROM 302以及RAM
303通过总线304彼此相连。输入/输出(Input/Output,I/O)接口305也连接至总线304。
通常,以下装置可以连接至I/O接口305:包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的编辑装置306;包括例如液晶显示器(Liquid Crystal Display,LCD)、扬声器、振动器等的输出装置307;包括例如磁带、硬盘等的存储装置308;以及通信装置309。通信装置309可以允许电子设备300与其他设备进行无线或有线通信以交换数据。虽然图3示出了具有各种装置的电子设备300,但是应理解的是,并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地,根据本公开的实施例,上文参考流程图描述的过程可以被实现为计算机软件程序。例如,本公开的实施例包括一种计算机程序产品,其包括承载在非暂态计算机可读介质上的计算机程序,该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中,该计算机程序可以通过通信装置309从网络上被下载和安装,或者从存储装置306被安装,或者从ROM 302被安装。在该计算机程序被处理装置301执行时,执行本公开实施例的方法中限定的上述功能。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的,而并不是用于对这些消息或信息的范围进行限制。
本公开实施例提供的电子设备与上述实施例提供的渲染图像的方法属于同一发明构思,未在本实施例中详尽描述的技术细节可参见上述实施例。
本公开实施例提供了一种计算机存储介质,其上存储有计算机程序,该程序被处理器执行时实现上述实施例所提供的渲染图像的方法。
需要说明的是,本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的***、装
置或器件,或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于:具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器((Erasable Programmable Read-Only Memory,EPROM)或闪存)、光纤、便携式紧凑磁盘只读存储器(Compact Disc Read-Only Memory,CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中,计算机可读存储介质可以是任何包含或存储程序的有形介质,该程序可以被指令执行***、装置或者器件使用或者与其结合使用。而在本公开中,计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号,其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式,包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质,该计算机可读信号介质可以发送、传播或者传输用于由指令执行***、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输,包括但不限于:电线、光缆、射频(Radio Frequency,RF)等等,或者上述的任意合适的组合。
在一些实施方式中,客户端、服务器可以利用诸如HTTP(HyperText Transfer Protocol,超文本传输协议)之类的任何当前已知或未来研发的网络协议进行通信,并且可以与任意形式或介质的数字数据通信(例如,通信网络)互连。通信网络的示例包括局域网(Local Area Network,LAN),广域网(Wide Area Network,WAN),网际网(例如,互联网)以及端对端网络(例如,ad hoc端对端网络),以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的;也可以是单独存在,而未装配入该电子设备中。
上述计算机可读介质承载有至少一个程序,当上述至少一个程序被该电子设备执行时,使得该电子设备:
基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是
否为关键帧;其中,所述待更新关键帧组中包括至少一个待应用关键帧;
响应于确定接收到的当前帧为关键帧,根据预设帧数和所述当前帧对所述待更新关键帧组进行更新,得到更新后的待更新关键帧组;
对所述更新后的待更新关键帧组中的各待应用关键帧进行优化,更新各待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码,上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++,还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中,远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机,或者,可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图,图示了按照本公开各种实施例的***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上,流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分,该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意,在有些作为替换的实现中,方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如,两个接连地表示的方框实际上可以基本并行地执行,它们有时也可以按相反的顺序执行,这依所涉及的功能而定。也要注意的是,框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合,可以用执行规定的功能或操作的专用的基于硬件的***来实现,或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现,也可以通过硬件的方式来实现。其中,单元的名称在某种情况下并不构成对该单元本
身的限定,例如,第一获取单元还可以被描述为“获取至少两个网际协议地址的单元”。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如,非限制性地,可以使用的示范类型的硬件逻辑部件包括:现场可编程门阵列(Field Programmable Gate Array,FPGA)、专用集成电路(Application Specific Integrated Circuit,ASIC)、专用标准产品(Application Specific Standard Parts,ASSP)、片上***(System on Chip,SOC)、复杂可编程逻辑设备(Complex Programmable Logic Device,CPLD)等等。
在本公开的上下文中,机器可读介质可以是有形的介质,其可以包含或存储以供指令执行***、装置或设备使用或与指令执行***、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体***、装置或设备,或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开的至少一个实施例,【示例一】提供了一种渲染图像的方法,该方法包括:
基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧;其中,所述待更新关键帧组中包括至少一个待应用关键帧;
响应于确定接收到的当前帧为关键帧,根据预设帧数和所述当前帧对所述待更新关键帧组进行更新,得到更新后的待更新关键帧组;
对所述更新后的待更新关键帧组中的各待应用关键帧进行优化,更新各待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染。
根据本公开的至少一个实施例,【示例二】提供了一种渲染图像的方法,该方法,还包括:
可选的,在首次接收到多个连续帧图像时,对所述多个连续帧图像进行预处理,确定所述至少一个初始关键帧;其中,所述预处理包括去除旋转影响的操作;
将所述至少一个初始化关键帧作为所述待更新关键帧组中的各待应用关键帧。
根据本公开的至少一个实施例,【示例三】提供了一种渲染图像的方法,该方法,还包括:
可选的,基于角点检测算法确定当前帧中的待处理点云数据,以基于所述至少一个待应用关键帧对所述待处理点云数据进行处理,得到当前帧的优化位姿,以确定当前帧是否为关键帧。
根据本公开的至少一个实施例,【示例四】提供了一种渲染图像的方法,该方法,还包括:
可选的,确定当前帧的目标特征点,以及所述当前帧与所述至少一个待应用关键帧的位移视差;
在所述目标特征点的数量达到第一预设数量阈值,且所述位移视差大于第一预设位移视差阈值,则确定所述当前帧为关键帧。
根据本公开的至少一个实施例,【示例五】提供了一种渲染图像的方法,该方法,还包括:
可选的,确定当前帧与所述至少一个待应用关键帧的共视特征点,并基于所述共视特征点在所述当前帧中降采样处理,确定目标特征点;以及,确定当前帧与所述待应用关键帧的位移偏差;
若所述目标特征点的数量小于所述当前帧中待处理特征点的数量,且所述位移偏差小于第二预设位移偏差,则确定所述当前帧为关键帧。
根据本公开的至少一个实施例,【示例六】提供了一种渲染图像的方法,该
方法,还包括:
可选的,对所述当前帧的待处理点云数据降采样处理,得到目标特征点;
确定所述当前帧与所述至少一个待应用关键帧的位移偏差;
若所述目标特征点的数量小于等于共视特征点的数量,且所述位移偏差小于第三预设位移偏差,则确定所述当前帧为关键帧;
其中,所述共视特征点为所述当前帧与所述待应用关键帧的共视点。
根据本公开的至少一个实施例,【示例七】提供了一种渲染图像的方法,该方法,还包括:
可选的,若所述至少一个待应用关键帧的数量小于所述预设帧数,则将所述当前帧更新至所述待更新关键帧组中,得到更新后的待更新关键帧组;
若所述至少一个待应用关键帧的数量大于或等于所述预设帧数,则将所述当前帧更新至所述待更新关键帧组中,并将与当前时刻间隔最长的待应用关键帧从所述待更新关键帧组中移除,得到更新后的待更新关键帧组。
根据本公开的至少一个实施例,【示例八】提供了一种渲染图像的方法,该方法,还包括:
可选的,基于光束平差法对各待应用关键帧优化,更新各待应用关键帧的相对位姿。
根据本公开的至少一个实施例,【示例九】提供了一种渲染图像的方法,该方法,还包括:
可选的,将所述更新后的相对位姿和所述优化位姿发送至图形处理器,以基于所述图形处理器更新地图数据。
根据本公开的至少一个实施例,【示例十】提供了一种渲染图像的装置,该装置包括:
关键帧确定模块,设置为基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧;其中,所述待更新关键帧组中包括至少一个待应用关键帧;
更新模块,设置为响应于确定接收到的当前帧为关键帧,根据预设帧数和所述当前帧对所述待更新关键帧组进行更新,得到更新后的待更新关键帧组;
待应用关键帧优化模块,设置为对所述更新后的待更新关键帧组中的各待应用关键帧进行优化,更新各待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染。
此外,虽然采用特定次序描绘了各操作,但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下,多任务和并行处理可能是有利的。同样地,虽然在上面论述中包含了若干具体实现细节,但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地,在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
Claims (12)
- 一种渲染图像的方法,包括:基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧;其中,所述待更新关键帧组中包括至少一个待应用关键帧;响应于确定接收到的当前帧为关键帧,根据预设帧数和所述当前帧对所述待更新关键帧组进行更新,得到更新后的待更新关键帧组;对所述更新后的待更新关键帧组中的待应用关键帧进行优化,更新待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染。
- 根据权利要求1所述的方法,在所述基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧之前,还包括:在首次接收到多个连续帧图像时,对所述多个连续帧图像进行预处理,确定所述至少一个初始关键帧;其中,所述预处理包括去除旋转影响的操作;将所述至少一个初始化关键帧作为所述待更新关键帧组中的至少一个待应用关键帧。
- 根据权利要求2所述的方法,在确定当前帧是否为关键帧之前,还包括:基于角点检测算法确定当前帧中的待处理点云数据,以基于所述至少一个待应用关键帧对所述待处理点云数据进行处理,得到当前帧的优化位姿,以确定当前帧是否为关键帧。
- 根据权利要求1所述的方法,其中,所述基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧,包括:确定当前帧的目标特征点,以及所述当前帧与所述至少一个待应用关键帧的位移视差;响应于所述目标特征点的数量达到第一预设数量阈值,且所述位移视差大于第一预设位移视差阈值,确定所述当前帧为关键帧。
- 根据权利要求1所述的方法,其中,所述基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧,包括:确定当前帧与所述至少一个待应用关键帧的共视特征点,并基于所述共视 特征点在所述当前帧中降采样处理,确定目标特征点;以及,确定当前帧与所述至少一个待应用关键帧的位移偏差;响应于所述目标特征点的数量小于所述当前帧中待处理特征点的数量,且所述位移偏差小于第二预设位移偏差,确定所述当前帧为关键帧。
- 根据权利要求1所述的方法,其中,所述基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧,包括:对所述当前帧的待处理点云数据降采样处理,得到目标特征点;确定所述当前帧与所述至少一个待应用关键帧的位移偏差;响应于所述目标特征点的数量小于等于共视特征点的数量,且所述位移偏差小于第三预设位移偏差,确定所述当前帧为关键帧;其中,所述共视特征点为所述当前帧与所述至少一个待应用关键帧的共视点。
- 根据权利要求1所述的方法,其中,所述根据预设帧数和所述当前帧对所述待更新关键帧组进行更新,得到更新后的待更新关键帧组,包括:响应于所述至少一个待应用关键帧的数量小于所述预设帧数,将所述当前帧更新至所述待更新关键帧组中,得到更新后的待更新关键帧组;响应于所述至少一个待应用关键帧的数量大于或等于所述预设帧数,将所述当前帧更新至所述待更新关键帧组中,并将与当前时刻间隔最长的待应用关键帧从所述待更新关键帧组中移除,得到更新后的待更新关键帧组。
- 根据权利要求1所述的方法,其中,所述对所述更新后的待更新关键帧组中的待应用关键帧进行优化,更新待应用关键帧的相对位姿,包括:基于光束平差法对待应用关键帧优化,更新待应用关键帧的相对位姿。
- 根据权利要求3所述的方法,还包括:将所述更新后的相对位姿和所述更新后的待更新关键帧组中的待应用关键帧的优化位姿发送至图形处理器,以基于所述图形处理器更新地图数据。
- 一种渲染图像的装置,包括:关键帧确定模块,设置为基于同步定位与建图***定位的待更新关键帧组,确定接收到的当前帧是否为关键帧;其中,所述待更新关键帧组中包括至少一个待应用关键帧;更新模块,设置为响应于确定接收到的当前帧为关键帧,根据预设帧数和所述当前帧对所述待更新关键帧组进行更新,得到更新后的待更新关键帧组;待应用关键帧优化模块,设置为对所述更新后的待更新关键帧组中的待应用关键帧进行优化,更新待应用关键帧的相对位姿,以基于更新后的相对位姿进行图像渲染。
- 一种电子设备,包括:至少一个处理器;存储装置,设置为存储至少一个程序,当所述至少一个程序被所述至少一个处理器执行,使得所述至少一个处理器实现如权利要求1-9中任一所述的渲染图像的方法。
- 一种包含计算机可执行指令的存储介质,所述计算机可执行指令在由计算机处理器执行时用于执行如权利要求1-9中任一所述的渲染图像的方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210501160.9A CN117079172A (zh) | 2022-05-09 | 2022-05-09 | 渲染图像的方法、装置、电子设备及存储介质 |
CN202210501160.9 | 2022-05-09 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023216918A1 true WO2023216918A1 (zh) | 2023-11-16 |
Family
ID=88710250
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2023/091479 WO2023216918A1 (zh) | 2022-05-09 | 2023-04-28 | 渲染图像的方法、装置、电子设备及存储介质 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN117079172A (zh) |
WO (1) | WO2023216918A1 (zh) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107369183A (zh) * | 2017-07-17 | 2017-11-21 | 广东工业大学 | 面向mar的基于图优化slam的跟踪注册方法及*** |
CN109387204A (zh) * | 2018-09-26 | 2019-02-26 | 东北大学 | 面向室内动态环境的移动机器人同步定位与构图方法 |
US20200111233A1 (en) * | 2019-12-06 | 2020-04-09 | Intel Corporation | Adaptive virtual camera for indirect-sparse simultaneous localization and mapping systems |
CN111429517A (zh) * | 2020-03-23 | 2020-07-17 | Oppo广东移动通信有限公司 | 重定位方法、重定位装置、存储介质与电子设备 |
-
2022
- 2022-05-09 CN CN202210501160.9A patent/CN117079172A/zh active Pending
-
2023
- 2023-04-28 WO PCT/CN2023/091479 patent/WO2023216918A1/zh unknown
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107369183A (zh) * | 2017-07-17 | 2017-11-21 | 广东工业大学 | 面向mar的基于图优化slam的跟踪注册方法及*** |
CN109387204A (zh) * | 2018-09-26 | 2019-02-26 | 东北大学 | 面向室内动态环境的移动机器人同步定位与构图方法 |
US20200111233A1 (en) * | 2019-12-06 | 2020-04-09 | Intel Corporation | Adaptive virtual camera for indirect-sparse simultaneous localization and mapping systems |
CN111429517A (zh) * | 2020-03-23 | 2020-07-17 | Oppo广东移动通信有限公司 | 重定位方法、重定位装置、存储介质与电子设备 |
Also Published As
Publication number | Publication date |
---|---|
CN117079172A (zh) | 2023-11-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3786890B1 (en) | Method and apparatus for determining pose of image capture device, and storage medium therefor | |
US11928800B2 (en) | Image coordinate system transformation method and apparatus, device, and storage medium | |
US9083960B2 (en) | Real-time 3D reconstruction with power efficient depth sensor usage | |
CN113811920A (zh) | 分布式姿势估计 | |
US11557083B2 (en) | Photography-based 3D modeling system and method, and automatic 3D modeling apparatus and method | |
WO2019238114A1 (zh) | 动态模型三维重建方法、装置、设备和存储介质 | |
JP7150917B2 (ja) | 地図作成のためのコンピュータ実施方法及び装置、電子機器、記憶媒体並びにコンピュータプログラム | |
CN115147558B (zh) | 三维重建模型的训练方法、三维重建方法及装置 | |
US20200410688A1 (en) | Image Segmentation Method, Image Segmentation Apparatus, Image Segmentation Device | |
CN115690382B (zh) | 深度学习模型的训练方法、生成全景图的方法和装置 | |
US20210319234A1 (en) | Systems and methods for video surveillance | |
WO2023207522A1 (zh) | 视频合成方法、装置、设备、介质及产品 | |
CN115578515B (zh) | 三维重建模型的训练方法、三维场景渲染方法及装置 | |
WO2022052782A1 (zh) | 图像的处理方法及相关设备 | |
JP2023525462A (ja) | 特徴を抽出するための方法、装置、電子機器、記憶媒体およびコンピュータプログラム | |
CN114677422A (zh) | 深度信息生成方法、图像虚化方法和视频虚化方法 | |
TW202244680A (zh) | 位置姿勢獲取方法、電子設備及電腦可讀儲存媒體 | |
Zhu et al. | PairCon-SLAM: Distributed, online, and real-time RGBD-SLAM in large scenarios | |
WO2024056030A1 (zh) | 一种图像深度估计方法、装置、电子设备及存储介质 | |
CN115578432B (zh) | 图像处理方法、装置、电子设备及存储介质 | |
WO2023216918A1 (zh) | 渲染图像的方法、装置、电子设备及存储介质 | |
WO2023025085A1 (zh) | 视频处理方法、装置、设备、介质及程序产品 | |
CN113703704B (zh) | 界面显示方法、头戴式显示设备和计算机可读介质 | |
CN114419189A (zh) | 地图构建方法及装置、电子设备、存储介质 | |
CN115205325A (zh) | 目标追踪方法及装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23802690 Country of ref document: EP Kind code of ref document: A1 |