CN115240043A - Data processing method and device, electronic equipment and readable storage medium - Google Patents
Data processing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN115240043A CN115240043A CN202210860829.3A CN202210860829A CN115240043A CN 115240043 A CN115240043 A CN 115240043A CN 202210860829 A CN202210860829 A CN 202210860829A CN 115240043 A CN115240043 A CN 115240043A
- Authority
- CN
- China
- Prior art keywords
- processing
- data
- processing result
- image frame
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/803—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of input or preprocessed data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/62—Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Optical Radar Systems And Details Thereof (AREA)
Abstract
The application provides a data processing method, a data processing device, electronic equipment and a readable storage medium. The data processing method comprises the following steps: the method comprises the steps of obtaining data to be processed, wherein the data to be processed comprises a first image frame and a first data frame, the first image frame comprises an image, and the first data frame comprises a plurality of point clouds. The electronic equipment processes the first image frame through the first NPU unit to obtain a first processing result, and processes the first data frame through the second NPU unit in parallel to obtain a second processing result. And fusing and processing the first processing result and the second processing result to obtain and output a third processing result. The NPU with lower cost is adopted to process data in a parallel mode, so that the computational power is improved, and the data processing efficiency is effectively improved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a readable storage medium.
Background
Autopilot technology is becoming more and more widely used. The perception algorithm in the automatic driving system comprises a 2D target inspection algorithm, a laser radar (Lidar) 3D target detection algorithm, a traffic light inspection algorithm, a garbage detection algorithm, a multi-target tracking algorithm and the like. The data processing of the perception algorithm needs to be realized based on an industrial control computer platform (abbreviated as an industrial personal computer platform).
However, the existing industrial personal computer platform has limited computing power, which results in long time consumption of the operation of the sensing algorithm, and the industrial personal computer platform has high cost for increasing the computing power, cannot provide enough computing power at low cost, and has low processing efficiency.
Disclosure of Invention
The application provides a data processing method, a data processing device, electronic equipment and a readable storage medium, and aims to solve the problems that an existing industrial personal computer platform is limited in computing power, cannot provide enough computing power at low cost, and is low in processing efficiency.
In a first aspect, an embodiment of the present application provides a data processing method, including:
the method comprises the steps of obtaining data to be processed, wherein the data to be processed comprises a first image frame and a first data frame, the first image frame comprises an image, and the first data frame comprises a plurality of point clouds. And processing the first image frame through the first NPU unit to obtain a first processing result, and processing the first data frame in parallel through the second NPU unit to obtain a second processing result. And fusing and processing the first processing result and the second processing result to obtain and output a third processing result.
In some embodiments, obtaining data to be processed includes: image data is acquired by the camera, and the image data comprises a plurality of image frames. The method comprises the steps of obtaining point cloud data through a laser radar, wherein the point cloud data comprise a plurality of data frames, and each data frame comprises a plurality of point clouds collected by the laser radar within a preset time length. And acquiring a first data frame and a first image frame with the same or similar acquisition time from the image data and the point cloud data.
In some embodiments, the image frames and the data frames each include an acquisition timestamp at the time of the respective acquisition.
Acquiring a first data frame and a first image frame with the same or similar acquisition time from image data and point cloud data, wherein the acquisition time comprises the following steps: and determining a first image frame and a first data frame with the same or similar acquisition time according to the acquisition time stamp of the image frame and the acquisition time stamp of the data frame.
In some embodiments, processing the first image frame by the first NPU unit to obtain a first processing result, and processing the first data frame in parallel by the second NPU unit to obtain a second processing result includes: and processing the first image frame through the first NPU unit to obtain a plane target detection result. And processing the first data frame through a second NPU unit to obtain a three-dimensional target detection result.
In some embodiments, fusing and processing the first processing result and the second processing result to obtain and output a third processing result, including: and performing data fusion on the first image frame, the first processing result and the second processing result to obtain a three-dimensional target detection result based on the first image frame. And performing multi-target tracking processing on the three-dimensional target detection result based on the first image frame to obtain a third processing result.
In some embodiments, the data to be processed further comprises a second image frame and a second data frame, the method further comprising:
and processing the second image frame through the third NPU unit to obtain a fourth processing result, and processing the second data frame in parallel through the fourth NPU unit to obtain a fifth processing result. And fusing and processing the fourth processing result and the fifth processing result to obtain and output a sixth processing result.
In some embodiments, the processing the second image frame by the third NPU unit to obtain a fourth processing result, and the processing the second data frame in parallel by the fourth NPU unit to obtain a fifth processing result includes: and processing the second image frame through a third NPU unit to obtain a plane target detection result. And processing the second data frame through a fourth NPU unit to obtain a three-dimensional target detection result.
In some embodiments, the fusing and processing the fourth processing result and the fifth processing result to obtain a sixth processing result, including: and performing data fusion on the second image frame, the fourth processing result and the fifth processing result to obtain a three-dimensional target detection result based on the second image frame. And performing multi-target tracking processing on the stereo target detection result based on the second image frame to obtain a sixth processing result.
In the first aspect, two NPU units respectively process an image frame and a data frame to obtain a first processing result and a second processing result, and then fuse the first processing result and the second processing result and perform corresponding post-processing to obtain a third result. Because the cost of each NPU unit is very low, the computational power and the processing efficiency can be improved through parallel processing at lower cost.
In a second aspect, an embodiment of the present application provides a data processing apparatus, including:
the device comprises an acquisition module and a processing module, wherein the acquisition module is used for acquiring data to be processed, the data to be processed comprises a first image frame and a first data frame, the first image frame comprises an image, and the first data frame comprises a plurality of point clouds.
And the processing module is used for processing the first image frame through the first NPU unit to obtain a first processing result, and processing the first data frame in parallel through the second NPU unit to obtain a second processing result.
And the fusion output module is used for fusing and processing the first processing result and the second processing result to obtain and output a third processing result.
In some embodiments, the acquisition module is specifically configured to acquire image data through a camera, where the image data includes a plurality of image frames. The method comprises the steps of obtaining point cloud data through a laser radar, wherein the point cloud data comprise a plurality of data frames, and each data frame comprises a plurality of point clouds collected by the laser radar within a preset time length. And acquiring a first data frame and a first image frame with the same or similar acquisition time from the image data and the point cloud data.
In some embodiments, the image frames and the data frames each include an acquisition timestamp at the time of the respective acquisition.
And the acquisition module is specifically used for determining a first image frame and a first data frame with the same or similar acquisition time according to the acquisition time stamp of the image frame and the acquisition time stamp of the data frame.
In some embodiments, the processing module is specifically configured to process the first image frame by using a first NPU unit, so as to obtain a planar target detection result. And processing the first data frame through a second NPU unit to obtain a three-dimensional target detection result.
In some embodiments, the processing module is specifically configured to perform data fusion on the first image frame, the first processing result, and the second processing result to obtain a stereoscopic object detection result based on the first image frame. And performing multi-target tracking processing on the three-dimensional target detection result based on the first image frame to obtain a third processing result.
In some embodiments, the data to be processed further includes a second image frame and a second data frame, and the processing module is further configured to process the second image frame through a third NPU unit to obtain a fourth processing result, and process the second data frame in parallel through a fourth NPU unit to obtain a fifth processing result.
And the fusion output module is further used for fusing and processing the fourth processing result and the fifth processing result to obtain and output a sixth processing result.
In some embodiments, the processing module is specifically configured to process the second image frame by using a third NPU unit, so as to obtain a planar target detection result. And processing the second data frame through a fourth NPU unit to obtain a three-dimensional target detection result.
In some embodiments, the fusion output module is specifically configured to perform data fusion on the second image frame, the fourth processing result, and the fifth processing result to obtain a stereoscopic object detection result based on the second image frame. And performing multi-target tracking processing on the stereo target detection result based on the second image frame to obtain a sixth processing result.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the method provided in the first aspect is implemented.
In a fourth aspect, the present application provides a computer-readable storage medium, where a computer program is stored, and when executed by a processor, the computer program implements the method provided in the first aspect.
In a fifth aspect, embodiments of the present application provide a computer program product, which, when run on an electronic device, causes the electronic device to perform the method provided in the first aspect.
It is understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description of the first aspect, and are not described herein again.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of an implementation of S104 in the data processing method according to an embodiment of the present application;
FIG. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it should be understood that the drawings in the present invention are for illustrative and descriptive purposes only and are not used to limit the scope of the present invention. Additionally, it should be understood that the schematic drawings are not necessarily drawn to scale. The flowcharts used in this invention illustrate operations performed in accordance with some embodiments of the present invention. It should be understood that the operations of the flow diagrams may be performed out of order, and that steps without logical context may be reversed in order or performed concurrently. In addition, one skilled in the art, under the direction of the present disclosure, may add one or more other operations to the flowchart, or may remove one or more operations from the flowchart.
In addition, the described embodiments of the present invention are only a part of the embodiments of the present invention, and not all of the embodiments. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the term "comprising" will be used in the embodiments of the invention to indicate the presence of the features stated hereinafter, but does not exclude the addition of further features. It should also be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. In the description of the present invention, it should also be noted that the terms "first", "second", "third", and the like are used for distinguishing the description, and are not intended to indicate or imply relative importance.
In the fields of unmanned aerial vehicles, automatic driving or robots and the like, a perception algorithm is a common algorithm and comprises a 2D target inspection algorithm, a laser radar (Lidar) 3D target detection algorithm, a traffic light inspection algorithm, a garbage detection algorithm, a multi-target tracking algorithm and the like. Through a perception algorithm, data such as a target detection frame can be acquired according to the data acquired by the equipment and output, and the next calculation is carried out.
The existing perception algorithm can be realized through an industrial personal computer platform, and the industrial personal computer platform carries out the operation of the perception algorithm through a display card. However, often, one industrial personal computer platform only has one display card, and due to the limited computing power of the display card, a plurality of algorithms can only multiplex the display card in a time-sharing manner for computing. And the cost of the display card is high, and the scheme of improving the computing power by adding the display card is expensive.
Based on this, the present application provides a data processing method applied to an electronic device, including: the method comprises the steps of obtaining data to be processed, wherein the data to be processed comprises a first image frame and a first data frame, the first image frame comprises an image, and the first data frame comprises a plurality of point clouds. And processing the first image frame through the first NPU unit to obtain a first processing result, and processing the first data frame in parallel through the second NPU unit to obtain a second processing result. And fusing and processing the first processing result and the second processing result to obtain and output a third processing result.
In the application, two NPU units are used for respectively processing image frames and data frames to obtain corresponding first processing results and second processing results, then the first processing results and the second processing results are fused, and corresponding post-processing is carried out to obtain third results. Because the cost of each NPU unit is very low, the computing power of the electronic equipment can be improved through parallel processing at lower cost, and the processing efficiency is improved.
In the present application, the electronic device may be an industrial personal computer, for example, when the electronic device is applied to automatic driving, the electronic device may be implemented in the form of an in-vehicle computer.
As an example, the electronic device may be an MDC300F platform from hua corporation. The MDC300F platform includes one processor, 4 embedded Neural-Network Processing Unit (NPU) units. The four NPU units can be combined into two groups of NPU units in pairs, and the two NPU units in each NPU unit group can respectively and synchronously process a visual perception algorithm and a laser radar (Lidar) point cloud processing algorithm.
For example, one of the NPU units is used in one NPU unit group to process the acquired image frame through a visual perception algorithm, and the other NPU unit is used to process the acquired data frame through a Lidar point cloud processing algorithm. And then, carrying out data fusion on intermediate results of the two algorithms, carrying out processing such as tracking algorithm and the like, and finally outputting the processing results to other algorithm modules for use.
Fig. 1 is a schematic flowchart of a data processing method according to an embodiment of the present application.
Referring to fig. 1, the data processing method includes:
s101, acquiring data to be processed.
The data to be processed comprises a first image frame and a first data frame, wherein the first image frame comprises an image, and the first data frame comprises a plurality of point clouds.
In some embodiments, an electronic device includes at least one camera and at least one lidar.
The camera may be used to acquire image data, which includes a plurality of image frames. For example, when the electronic device is applied to automatic driving, a plurality of cameras may be disposed on the head, the tail, the side mirrors, and the like to collect image data of the surroundings of the vehicle. The image frame acquisition can be realized by shooting a video with a certain frame rate. For example, a video of the front of a vehicle with 60 frames per second (fps) is shot by a camera arranged at the head of the vehicle, that is, 60 photos of the front of the vehicle are shot at intervals of 1/60 second within 1 second, and each photo is an image frame.
The laser radar can be used for acquiring point cloud data, the point cloud data comprises a plurality of data frames, and each data frame comprises a plurality of point clouds collected by the laser radar within a preset time length. For example, when the electronic device is applied to automatic driving, at least one laser radar may be respectively arranged at the head and the tail of the vehicle to collect point cloud data in front of and behind the vehicle. When the laser radar collects point cloud data, the collected point cloud can be packed into a data frame every 50 ms.
In this embodiment, acquiring the data to be processed includes: and acquiring a first data frame and a first image frame with the same or similar acquisition time.
In some embodiments, the first image frame and the first data frame each include an acquisition timestamp at the time of the respective acquisition. Acquiring a first data frame and a first image frame with the same or similar acquisition time, wherein the acquiring comprises the following steps: and determining the image frames and the data frames with the same or similar acquisition time according to the acquisition time stamp of the first image frame and the acquisition time stamp of the first data frame.
As an example, when the camera acquires an image frame, an acquisition time stamp of the acquired image frame may be recorded at the same time, for example, the time when the data frame is acquired is 2022-02-22', then the time stamp may be recorded as 2022022222222222222222.
For example, when the laser radar collects point cloud data, recording the middle time point of collection, and as the collection time stamp, for example, if the collection is performed at intervals of 60ms, the collection time period is 2022-02-22' to 2022-02-22.
In some embodiments, the image frames and the data frames with the same or similar acquisition time are determined according to the acquisition time stamp of the first image frame and the acquisition time stamp of the data frame, one frame of data may be taken out from the data frames as the first data frame, and then one frame of image closest to the acquisition time of the first data frame is matched and searched as the first image frame based on the acquisition time stamp of the first data frame.
Or, a frame may be taken out from the image frames as a first image frame, and then a data frame closest to the acquisition time of the first image frame is matched and searched as the first data frame with reference to the acquisition time stamp of the first image frame.
And S102, processing the first image frame through the first NPU unit to obtain a first processing result.
And S103, processing the first data frame through a second NPU unit to obtain a second processing result.
In some embodiments, referring to the above example, the NPU unit may be a Sheng 310 chip in an MDC300F platform. The NPU unit can be used for finishing algorithm processing of 2D target detection, 3D target detection, instance segmentation and the like of visual image frames and Lidar point cloud data.
For example, in the present embodiment, the first NPU unit performs prediction by a deep learning algorithm of a visual image frame according to the first image frame to obtain a 2D target detection result (first processing result) of the first image frame. And the second NPU unit predicts through a deep learning algorithm of the Lidar point cloud according to the first data frame to obtain a 3D target detection result (second processing result) of the first data frame.
And S104, fusing and processing the first processing result and the second processing result to obtain a third processing result, and outputting the third processing result.
Fig. 2 is a schematic flowchart of the implementation of S104 in the data processing method according to an embodiment of the present application.
In some embodiments, referring to the first processing result and the second processing result shown in S103, and referring to fig. 2, fusing and processing the first processing result and the second processing result to obtain a third processing result, including:
s1041, performing data fusion on the first image frame, the first processing result and the second processing result to obtain a three-dimensional target detection result based on the first image frame.
And S1042, performing multi-target tracking processing on the three-dimensional target detection result based on the first image frame to obtain a third processing result.
In some embodiments, the first image frame, the first processing result, and the second processing result may be fused and a partial post-processing operation may be performed by the visual and Lidar data fusion and post-processing module.
The first image frame, the first processing result and the second processing result are fused, so that data fusion can be performed on the first image frame and the second processing result, and a three-dimensional target detection result based on the first image frame is obtained.
Performing a partial post-processing operation may perform multi-target tracking algorithm processing on the stereoscopic target detection result (e.g., may be a 3D detection frame) based on the first image frame, and buffer the tracking result (third processing result).
In some embodiments, after the third processing result is obtained, the sensing algorithm result output module is responsible for outputting the third processing result to other algorithm modules for further processing.
It should be noted that, since the MDC300F platform includes 4 NPU units, the remaining two NPU units may also be utilized. That is, the electronic device further includes a third NPU unit and a fourth NPU unit.
In some embodiments, processing the data further includes a second image frame and a second data frame, the method further comprising: and processing the second image frame through the third NPU unit to obtain a fourth processing result, and processing the second data frame in parallel through the fourth NPU unit to obtain a fifth processing result. And fusing and processing the fourth processing result and the fifth processing result to obtain and output a sixth processing result.
The manner of acquiring the second image frame and the second data frame is similar to that of acquiring the first image frame and the first data frame, and the second image frame and the second data frame may be acquired after the first image frame and the first data frame or before the first image frame and the first data frame, which is not limited in this application.
It should be further noted that the fourth processing result is a planar target detection result obtained from the second image frame, and the fifth processing result is a stereoscopic target detection result obtained from the second data frame.
Based on this, the method of fusing and processing the fourth processing result and the fifth processing result to obtain a sixth processing result is the same as the method of fusing and processing the first processing result and the second processing result to obtain a third processing result, that is: and performing data fusion on the second image frame, the fourth processing result and the fifth processing result to obtain a three-dimensional target detection result based on the second image frame. And performing multi-target tracking processing on the stereo target detection result based on the second image frame to obtain a sixth processing result. The specific implementation is already described in S104, and is not described herein.
In this embodiment, by processing the image frames and the data frames in parallel using more NPU units, higher computational power can be provided, resulting in higher processing efficiency.
It should be understood that, although the steps in the flowcharts related to the embodiments are shown in sequence as indicated by the arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least a part of the steps in the flowcharts related to the above embodiments may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.
Fig. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.
An embodiment of the present application provides a data processing apparatus, including:
the acquisition module 301 is configured to acquire data to be processed, where the data to be processed includes a first image frame and a first data frame, the first image frame includes an image, and the first data frame includes a plurality of point clouds.
The processing module 302 is configured to process the first image frame through the first NPU unit to obtain a first processing result, and process the first data frame through the second NPU unit in parallel to obtain a second processing result.
And the fusion output module 303 is configured to fuse and process the first processing result and the second processing result to obtain and output a third processing result.
In some embodiments, the acquisition module 301 is specifically configured to acquire image data through a camera, where the image data includes a plurality of image frames. The method comprises the steps of obtaining point cloud data through a laser radar, wherein the point cloud data comprise a plurality of data frames, and each data frame comprises a plurality of point clouds collected by the laser radar within a preset time length. And acquiring a first data frame and a first image frame with the same or similar acquisition time from the image data and the point cloud data.
In some embodiments, the image frames and the data frames each include an acquisition timestamp at the time of the respective acquisition.
The acquisition module 301 is specifically configured to determine a first image frame and a first data frame with the same or similar acquisition times according to the acquisition time stamp of the image frame and the acquisition time stamp of the data frame.
In some embodiments, the processing module 302 is specifically configured to process the first image frame by using a first NPU unit, so as to obtain a planar target detection result. And processing the first data frame through the second NPU unit to obtain a three-dimensional target detection result.
In some embodiments, the processing module 302 is specifically configured to perform data fusion on the first image frame, the first processing result, and the second processing result to obtain a stereoscopic object detection result based on the first image frame. And performing multi-target tracking processing on the three-dimensional target detection result based on the first image frame to obtain a third processing result.
In some embodiments, the data to be processed further includes a second image frame and a second data frame, and the processing module 302 is further configured to process the second image frame through a third NPU unit to obtain a fourth processing result, and process the second data frame through a fourth NPU unit in parallel to obtain a fifth processing result.
The fusion output module 303 is further configured to fuse and process the fourth processing result and the fifth processing result to obtain and output a sixth processing result.
In some embodiments, the processing module 302 is specifically configured to process the second image frame by using a third NPU unit, so as to obtain a planar target detection result. And processing the second data frame through a fourth NPU unit to obtain a three-dimensional target detection result.
In some embodiments, the fusion output module 303 is specifically configured to perform data fusion on the second image frame, the fourth processing result, and the fifth processing result to obtain a stereoscopic object detection result based on the second image frame. And performing multi-target tracking processing on the stereo target detection result based on the second image frame to obtain a sixth processing result.
The various modules in the above-described apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent of a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
It should be understood that the above-described embodiments of the apparatus are merely exemplary, and that the apparatus and method disclosed in the embodiments of the present invention may be implemented in other ways. For example, the division of the modules into only one logical functional division may be implemented in other ways, and for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or modules through some communication interfaces, and may be in an electrical, mechanical or other form. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a user terminal or a driver terminal to perform all or part of the steps of the method according to the embodiments of the present invention.
That is, those skilled in the art will appreciate that embodiments of the present invention may be implemented in any form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects.
Based on this, the embodiment of the present invention further provides a program product, which may be a storage medium such as a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and a computer program may be stored on the storage medium, and when the computer program is executed by a processor, the steps of the object detection method as described in the foregoing method embodiment are performed. The specific implementation and technical effects are similar, and are not described herein again.
It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, the computer program can include the processes of the embodiments of the methods described above. Any reference to memory, databases, or other media used in the embodiments provided herein can include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), magnetic Random Access Memory (MRAM), ferroelectric Random Access Memory (FRAM), phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.
Optionally, an embodiment of the present invention further provides an electronic device, where the electronic device may be a server, a computer, an industrial personal computer, and the like, and fig. 4 is a schematic structural diagram of the electronic device provided in the embodiment of the present application.
As shown in fig. 4, the electronic device may include: a processor 401, a storage medium 402 and a bus 403, the storage medium 402 storing machine-readable instructions executable by the processor 401, the processor 2401 communicating with the storage medium 402 via the bus 403 when the object detection apparatus is operating, the processor 401 executing the machine-readable instructions to perform the steps of the object detection method as described in the previous embodiments. The specific implementation and technical effects are similar, and are not described herein again.
For ease of explanation, only one processor is described in the above object detection apparatus. However, it should be noted that in some embodiments, the object detection device in the present invention may further include a plurality of processors, and thus, the steps performed by one processor described in the present invention may also be performed by a plurality of processors in combination or individually.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of the changes or substitutions within the technical scope of the present invention, and shall cover the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (11)
1. A data processing method, comprising:
acquiring data to be processed, wherein the data to be processed comprises a first image frame and a first data frame, the first image frame comprises an image, and the first data frame comprises a plurality of point clouds;
processing the first image frame through a first NPU unit to obtain a first processing result, and processing the first data frame in parallel through a second NPU unit to obtain a second processing result;
and fusing and processing the first processing result and the second processing result to obtain and output a third processing result.
2. The method of claim 1, wherein the obtaining the data to be processed comprises:
acquiring and acquiring image data through a camera, wherein the image data comprises a plurality of image frames;
acquiring point cloud data through a laser radar, wherein the point cloud data comprises a plurality of data frames, and each data frame comprises a plurality of point clouds collected by the laser radar within a preset time length;
and acquiring the first data frame and the first image frame with the same or similar acquisition time from the image data and the point cloud data.
3. The method of claim 2, wherein the image frame and the data frame each include an acquisition timestamp at the time of the respective acquisition;
the acquiring the first data frame and the first image frame with the same or similar acquisition time from the image data and the point cloud data comprises:
and determining the first image frame and the first data frame with the same or similar acquisition time according to the acquisition time stamp of the image frame and the acquisition time stamp of the data frame.
4. The method of any of claims 1-3, wherein processing the first image frame by a first NPU unit to obtain a first processing result and processing the first data frame in parallel by a second NPU unit to obtain a second processing result comprises:
processing the first image frame through a first NPU unit to obtain a plane target detection result;
and processing the first data frame through a second NPU unit to obtain a three-dimensional target detection result.
5. The method according to claim 4, wherein the fusing and processing the first processing result and the second processing result to obtain and output a third processing result comprises:
performing data fusion on the first image frame, the first processing result and the second processing result to obtain a three-dimensional target detection result based on the first image frame;
and performing multi-target tracking processing on the three-dimensional target detection result based on the first image frame to obtain a third processing result.
6. The method of any of claims 1-3, wherein the data to be processed further comprises a second image frame and a second data frame, the method further comprising:
processing the second image frame through a third NPU unit to obtain a fourth processing result, and processing the second data frame through a fourth NPU unit in parallel to obtain a fifth processing result;
and fusing and processing the fourth processing result and the fifth processing result to obtain and output a sixth processing result.
7. The method of claim 6, wherein the processing the second image frame by a third NPU unit to obtain a fourth processing result, and the processing the second data frame in parallel by a fourth NPU unit to obtain a fifth processing result comprises:
processing the second image frame through a third NPU unit to obtain a plane target detection result;
and processing the second data frame through a fourth NPU unit to obtain a three-dimensional target detection result.
8. The method according to claim 7, wherein the fusing and processing the fourth processing result and the fifth processing result to obtain a sixth processing result includes:
performing data fusion on the second image frame, the fourth processing result and the fifth processing result to obtain a three-dimensional target detection result based on the second image frame;
and performing multi-target tracking processing on the stereo target detection result based on the second image frame to obtain a sixth processing result.
9. A data processing apparatus, comprising:
the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring data to be processed, the data to be processed comprises a first image frame and a first data frame, the first image frame comprises an image, and the first data frame comprises a plurality of point clouds;
the processing module is used for processing the first image frame through a first NPU unit to obtain a first processing result, and processing the first data frame in parallel through a second NPU unit to obtain a second processing result;
and the fusion output module is used for fusing and processing the first processing result and the second processing result to obtain and output a third processing result.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any of claims 1 to 8 when executing the computer program.
11. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210860829.3A CN115240043A (en) | 2022-07-21 | 2022-07-21 | Data processing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210860829.3A CN115240043A (en) | 2022-07-21 | 2022-07-21 | Data processing method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115240043A true CN115240043A (en) | 2022-10-25 |
Family
ID=83675148
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210860829.3A Pending CN115240043A (en) | 2022-07-21 | 2022-07-21 | Data processing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115240043A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115883564A (en) * | 2023-02-21 | 2023-03-31 | 青岛创新奇智科技集团股份有限公司 | Media stream processing method and device, electronic equipment and storage medium |
| CN116540872A (en) * | 2023-04-28 | 2023-08-04 | 中广电广播电影电视设计研究院有限公司 | VR data processing method, device, equipment, medium and product |
-
2022
- 2022-07-21 CN CN202210860829.3A patent/CN115240043A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115883564A (en) * | 2023-02-21 | 2023-03-31 | 青岛创新奇智科技集团股份有限公司 | Media stream processing method and device, electronic equipment and storage medium |
| CN116540872A (en) * | 2023-04-28 | 2023-08-04 | 中广电广播电影电视设计研究院有限公司 | VR data processing method, device, equipment, medium and product |
| CN116540872B (en) * | 2023-04-28 | 2024-06-04 | 中广电广播电影电视设计研究院有限公司 | VR data processing method, device, equipment, medium and product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7106665B2 (en) | MONOCULAR DEPTH ESTIMATION METHOD AND DEVICE, DEVICE AND STORAGE MEDIUM THEREOF | |
| CN115240043A (en) | Data processing method and device, electronic equipment and readable storage medium | |
| CN113160068B (en) | Point cloud completion method and system based on image | |
| Panek et al. | Meshloc: Mesh-based visual localization | |
| CN111401205A (en) | Action recognition method and device, electronic equipment and computer readable storage medium | |
| US20140093122A1 (en) | Image identifiers and methods and systems of presenting image identifiers | |
| US11494975B2 (en) | Method for analyzing three-dimensional model and device for analyzing three-dimensional model | |
| AliAkbarpour et al. | Fast structure from motion for sequential and wide area motion imagery | |
| CN111798505A (en) | Monocular vision-based dense point cloud reconstruction method and system for triangularized measurement depth | |
| EP4163873A1 (en) | Method and apparatus with global localization | |
| CN111931679A (en) | Action recognition method, device, equipment and storage medium | |
| CN113610918A (en) | Pose calculation method and device, electronic equipment and readable storage medium | |
| EP3971781A1 (en) | Method and apparatus with neural network operation | |
| US20180357784A1 (en) | Method for characterising a scene by computing 3d orientation | |
| Hyun et al. | Hardware-friendly architecture for a pseudo 2D weighted median filter based on sparse-window approach | |
| Roszkowski et al. | FPGA design of the computation unit for the semi-global stereo matching algorithm | |
| CN104637043B (en) | Pixel selecting method, device, parallax value is supported to determine method | |
| CN116051448A (en) | Flat surface detection device and method | |
| CN114913470A (en) | Event detection method and device | |
| CN105491277B (en) | Image processing method and electronic equipment | |
| CN114119678A (en) | Optical flow estimation method, computer program product, storage medium, and electronic device | |
| CN113920733A (en) | Traffic volume estimation method and system based on deep network | |
| CN111753766A (en) | Image processing method, device, equipment and medium | |
| CN117557615B (en) | Data processing method and device for light field depth estimation | |
| CN116665177B (en) | Data processing method, device, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |