CN115359097A - Dense optical flow generation method and device, electronic equipment and readable storage medium - Google Patents
Dense optical flow generation method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN115359097A CN115359097A CN202211284018.XA CN202211284018A CN115359097A CN 115359097 A CN115359097 A CN 115359097A CN 202211284018 A CN202211284018 A CN 202211284018A CN 115359097 A CN115359097 A CN 115359097A
- Authority
- CN
- China
- Prior art keywords
- image
- pyramid
- layer
- dma
- optical flow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000003287 optical effect Effects 0.000 title claims abstract description 124
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000015654 memory Effects 0.000 claims abstract description 182
- 239000011159 matrix material Substances 0.000 claims abstract description 96
- 238000004590 computer program Methods 0.000 claims description 16
- 238000004422 calculation algorithm Methods 0.000 abstract description 20
- 238000004364 calculation method Methods 0.000 abstract description 16
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000000903 blocking effect Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/269—Analysis of motion using gradient-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
- G06F13/28—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the invention discloses a dense optical flow generation method and device, electronic equipment and a readable storage medium. The method comprises the following steps: writing two continuous frames of images into a shared memory according to the DMA, and generating a pyramid image layer of each image; writing the pyramid image layer into a shared memory according to the DMA, and generating an image gradient of the pyramid image layer; writing the image gradient into a shared memory according to the DMA, and generating a Hessian matrix of the image in the pyramid image layer; and writing the pyramid image layer, the image gradient and the hessian matrix into a shared memory according to the DMA, and generating dense optical flow between two continuous frames of images according to the pyramid image layer, the image gradient and the hessian matrix. According to the invention, the calculated data required by the dense optical flow algorithm is written into the shared memory in advance through the DMA, so that SRAM resources occupied by the dense optical flow algorithm are reduced, the cost of the dense optical flow algorithm is greatly reduced, and the calculation efficiency is improved.
Description
Technical Field
The present invention relates to the field of computer vision technologies, and in particular, to a dense optical flow generation method and apparatus, an electronic device, and a readable storage medium.
Background
Motion can be described in terms of motion fields, which reflect the three-dimensional motion of objects in the real world; the optical flow field is a projection of the motion field on a two-dimensional image plane, that is, a projection of a three-dimensional velocity vector of the moving object on a two-dimensional imaging plane, and represents an instantaneous change of displacement of the object in the image, that is, an instantaneous velocity, so the optical flow field can also be called a velocity field.
Dense Optical Flow (Dense Optical Flow) is an image registration method for point-by-point matching of images, which is different from sparse Optical Flow that only aims at a plurality of feature points on an image, and the Dense Optical Flow calculates the offset of all points on the image, so that a Dense Optical Flow field is formed. The dense optical flow field can be used for carrying out image registration at a pixel level, and the effect after registration is obviously better than that of sparse optical flow registration.
At present, the dense optical flow algorithm may be implemented by using a DSP (Digital Signal Processing), may be implemented on a GPU, and may also be implemented by using a neural network algorithm. However, when a dense optical flow algorithm is implemented by using a DSP, deep optimization needs to be performed on the DSP, and the performance is difficult to meet the application requirements; when the calculation of the dense optical flow is implemented on the GPU, a large amount of GPU resources are needed, the power consumption and the area of the GPU are large, and the cost for calculating the GPU in the system on chip is higher; when the neural network algorithm is adopted to implement the dense optical flow algorithm, a large amount of computing resources are required to be occupied, and the efficiency is low.
Therefore, how to reduce the resource occupation of the dense optical flow algorithm and improve the computational efficiency is a technical problem to be solved urgently at present.
Disclosure of Invention
Aiming at the defects of the prior art, the embodiment of the invention provides a dense optical flow generation method, a dense optical flow generation device, electronic equipment and a readable storage medium, and aims to solve the technical problems of high resource occupation and low efficiency of a dense optical flow algorithm.
In a first aspect, an embodiment of the present invention provides a dense optical flow generation method, which includes:
writing two continuous frames of images into a shared memory according to a preset DMA (direct memory access), and generating a pyramid image layer of each image;
writing the pyramid layer into the shared memory according to the DMA, and generating an image gradient of the pyramid layer;
writing the image gradients into the shared memory according to the DMA, and generating a Hessian matrix of the image in the pyramid layer;
and writing the pyramid image layer, the image gradient and the hessian matrix into the shared memory according to the DMA, and generating dense optical flow between the two continuous frames of images according to the pyramid image layer, the image gradient and the hessian matrix.
In a second aspect, an embodiment of the present invention provides a dense optical flow generation apparatus, including:
the device comprises a first generation unit, a second generation unit and a third generation unit, wherein the first generation unit is used for writing two frames of continuous images into a shared memory according to a preset DMA (direct memory access) and generating a pyramid layer of each image;
a second generation unit configured to write the pyramid layer into the shared memory according to the DMA, and generate an image gradient of the pyramid layer;
a third generation unit configured to write the image gradient into the shared memory according to the DMA, and generate a hessian matrix of an image in the pyramid layer;
a fourth generating unit, configured to write the pyramid layer, the image gradient, and the hessian matrix into the shared memory according to the DMA, and generate a dense optical flow between the two consecutive frames of images according to the pyramid layer, the image gradient, and the hessian matrix.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the dense optical flow generation method according to the first aspect.
In a fourth aspect, the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the dense optical flow generation method according to the first aspect.
The embodiment of the invention provides a dense optical flow generation method and device, electronic equipment and a readable storage medium. The method comprises the steps of writing two continuous frames of images into a shared memory through a DMA in advance, generating a pyramid image layer of the images, then writing the images in the pyramid image layer into the shared memory according to the DMA, generating image gradients of the pyramid image layer, then writing the image gradients into the shared memory according to the DMA and generating hessian matrixes corresponding to the images according to the image gradients, and finally writing the pyramid image layer, the image gradients and the hessian matrixes into the shared memory according to the DMA and generating dense optical flows between the two continuous frames of images according to the pyramid image layer, the image gradients and the hessian matrixes. The invention writes the calculation data required by the dense optical flow algorithm into the shared Memory in advance through the DMA so as to reduce SRAM (Static Random-Access Memory) resources occupied by the dense optical flow algorithm, greatly reduce the cost of the dense optical flow algorithm and improve the calculation efficiency.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
FIG. 1 is a diagram of a hardware implementation architecture of a dense optical flow generation method provided by an embodiment of the present invention;
FIG. 2 is a flow chart of a dense optical flow generation method provided by an embodiment of the invention;
FIG. 3 is a flow chart of a dense optical flow generation method according to an embodiment of the present invention;
FIG. 4 is another schematic flow chart of a dense optical flow generation method provided by the embodiment of the invention;
FIG. 5 is another schematic flow chart of a dense optical flow generation method provided by an embodiment of the invention;
FIG. 6 is another schematic flow chart of a dense optical flow generation method provided by an embodiment of the invention;
FIG. 7 is another flow chart of a dense optical flow generation method according to an embodiment of the present invention;
FIG. 8 is a schematic block diagram of a dense optical flow generation apparatus provided by an embodiment of the present invention;
fig. 9 is a schematic block diagram of an electronic device provided in an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1, fig. 1 is a diagram illustrating a hardware implementation architecture of a dense optical flow generation method according to an embodiment of the present invention. The dense optical flow generation method of the embodiment of the invention is applied to the terminal equipment, and is executed by application software installed in the terminal equipment to generate the dense optical flow of the image. The terminal device may be a terminal device with an internet access function, such as a smart band, a desktop computer, a notebook computer, a tablet computer, or a mobile phone.
The optical flow is an instantaneous speed of pixel motion of a spatial moving object on an observation imaging plane, and is a method for finding a corresponding relation between a previous frame and a current frame by using the change of pixels in an image sequence in a time domain and the correlation between adjacent frames so as to calculate motion information of an object between the adjacent frames. While the optical flow is generated mainly due to the movement of the foreground object itself in the scene, the motion of the camera, or the joint motion of both.
Specifically, the application software is provided with a Flow Ctrl (Flow control) mechanism for controlling the whole process of the dense optical Flow algorithm, after the application software is configured in the terminal device, the first generation unit, the second generation unit, the third generation unit, and the fourth generation unit are sequentially started by the Flow Ctrl, and after all the generation units complete the calculation, the Flow Ctrl feeds back the calculation result and the calculation state to the application software to complete the dense optical Flow algorithm.
In the hardware implementation architecture diagram shown in fig. 1, the MAC Array is a floating-point multiplication and addition Array unit, and is used for implementing parallel multiply-add calculation of a plurality of floating-point numbers, and the third generation unit and the fourth generation unit share the MAC Array to implement floating-point operation, so that required hardware circuits can be reduced. An AXI3 protocol may be used between the DMA (Direct Memory Access) and the external Memory to implement reading and writing of data, but is not limited to the AXI3 protocol. Intermediate data required in the dense optical flow algorithm and data read from an external memory can be stored in the shared memory LMEM for use by the First generation unit, the second generation unit, the third generation unit, and the fourth generation unit, and the shared memory manages the address space in a FIFO (First Input First Output) and direct read manner.
It should be noted that the application scenario of the foregoing embodiment is only an example, and the service and scenario described in the embodiment of the present invention are for more clearly illustrating the technical solution of the embodiment of the present application, and do not constitute a limitation to the technical solution provided in the embodiment of the present application, and it is known by a person of ordinary skill in the art that the technical solution provided in the embodiment of the present application is also applicable to similar technical problems with the evolution of a system and the occurrence of a new service scenario. The following are detailed below. It should be noted that the following description of the embodiments is not intended to limit the preferred order of the embodiments.
The dense optical flow generation method described below is explained in detail. Referring to fig. 2, fig. 2 is a schematic flowchart of a dense optical flow generation method according to an embodiment of the invention.
As shown in FIG. 2, the method includes the following steps S110 to S140.
And S110, writing two continuous images into a shared memory according to a preset DMA, and generating a pyramid layer of each image.
Specifically, after two consecutive images are written into the shared memory by the DMA, the two consecutive images can be directly read from the shared memory to generate the pyramid layer of each image. The image in the pyramid image layer has gradually increased resolution and image size from top to bottom, and each image in the pyramid image layer has size twice that of the image in the upper layer.
In this embodiment, the pyramid image layer may be a gaussian pyramid, where the gaussian pyramid is a group of images with multiple resolutions, and the pyramid includes a group of images with multiple resolutions.
In another embodiment, before step S110, the method further includes the steps of: and controlling the DMA to read the image in a first external memory according to a first instruction.
In this embodiment, the first instruction is that the terminal sends to the DMA instruction information for reading and writing two consecutive frames of images from and into the first external memory, which is a storage device for storing two adjacent frames of images in the video stream, to the shared memory. After receiving the first command, the DMA can read two continuous frames of images from the first external memory and directly write the images into the shared memory.
And S120, writing the pyramid image layer into the shared memory according to the DMA, and generating the image gradient of the pyramid image layer.
Specifically, the image gradient is derived by taking an image of the pyramid layer as a two-dimensional discrete function, after the image in the pyramid layer is written into the shared memory by the DMA, the written image can be read from the shared memory, and the pixel value of the image is calculated to obtain the gradient of each image in the X direction and the Y direction, so as to obtain the image gradient of the pyramid layer, and after the gradient calculation of one image in the pyramid layer is completed, the calculation result can be fed back to the Flow Ctrl, so that the Flow Ctrl can perform the next operation.
In another embodiment, as shown in fig. 3, before step S120, steps S120a and S120b are further included.
S120a, writing the intermediate data for generating the pyramid graph layer into the shared memory, and writing the pyramid graph layer into a second external memory according to the DMA;
and S120b, controlling the DMA to read one layer of image in the pyramid image layer in the second external memory according to a second instruction.
In this embodiment, the second external memory is a storage device for storing the pyramid layers, the intermediate data of the pyramid layers is data generated in the pyramid layers for constructing the two frames of continuous images, and the data is data required in the process of generating the dense optical flow of the images, so the intermediate data for generating the pyramid layers needs to be written into the shared memory in advance, so that the intermediate data of the pyramid layers can be obtained quickly in the subsequent process. And after receiving the second instruction, the DMA can read the image of a certain layer in the pyramid image layer from the second external memory to calculate the gradient of the image.
And S130, writing the image gradient into the shared memory according to the DMA, and generating a Hessian matrix of the image in the pyramid image layer.
Specifically, the hessian matrix is a square matrix formed by taking an image of a pyramid layer as a second-order partial derivative of a two-dimensional discrete function, the local curvature of the function is described, after the DMA writes the gradient of the image of a certain layer into the shared memory, because the shared memory stores intermediate data for calculating the image in advance, the gradient of the image and the intermediate data for generating the gradient of the image can be directly read from the shared memory to generate the hessian matrix of the image, and after the hessian matrix of the image is calculated, the calculation result can be fed back to the Flow Ctrl so that the Flow Ctrl can perform the next operation.
In another embodiment, as shown in fig. 4, before step S130, steps S130a and S130b are further included.
S130a, writing intermediate data for generating the image gradient into the shared memory, and writing the image gradient into a third external memory according to the DMA;
and S130b, controlling the DMA to read the image gradient in the third external memory according to a third instruction.
In this embodiment, the third external memory is a memory device for storing image gradients, the intermediate data of the image gradients is data generated by calculating the gradients of the images corresponding to the image gradients, the third instruction is that the terminal sends instruction information for reading the gradients of the images in a certain layer of the pyramid layer to the DMA, and after receiving the third instruction, the DMA can read the image gradients in the certain layer of the pyramid layer from the third external memory to calculate the hessian matrix of the corresponding images.
In another embodiment, as shown in fig. 5, step S130 includes steps S131 and S132.
S131, acquiring block information of the image in the pyramid image layer;
s132, generating a Hessian matrix of the image block in the pyramid image layer and an inverse matrix of the Hessian matrix according to the block information and the image gradient.
Specifically, since the size of each layer of image in the pyramid layer is large, and the amount of calculation for directly calculating the hessian matrix of each layer of image is large, in this embodiment, each layer of image in the pyramid layer is divided into a plurality of image blocks, and the amount of calculation is reduced by calculating the hessian matrix and the inverse matrix of the image blocks. The blocking information is size information of each image block in each layer of image of the pyramid image layer, and the gradient corresponding to each image block can be determined through the blocking information, so that a hessian matrix and an inverse matrix of the hessian matrix of each image block can be generated.
S140, writing the pyramid image layer, the image gradient and the Hessian matrix into the shared memory according to the DMA, and generating dense optical flow between the two continuous frames of images according to the pyramid image layer, the image gradient and the Hessian matrix.
Specifically, each time the DMA writes the pyramid layer, the image gradient, and the hessian matrix into the shared memory, only one image in the pyramid layer of one of the two continuous frames of images, the gradient of the image, the hessian matrix, and the corresponding same-layer image in the pyramid layer of the other of the two continuous frames of images are written into the shared memory, so that the optical flow information between the images of one of the two pyramid layers can be generated, then the optical flow information is used as an initial value, and the steps S120 to S140 are repeated, so that the optical flow between the images of the bottom layers in the two pyramid layers can be generated, and the images of the bottom layers in the pyramid layers are both two continuous frames of images, so that the dense optical flow between the images of the bottom layers is the optical flow between the two continuous frames of images.
In another embodiment, as shown in fig. 6, before step S140, steps S140a and S140b are further included.
S140a, writing the intermediate data for generating the Hessian matrix into the shared memory, and writing the Hessian matrix into a fourth external memory according to the DMA;
and S140b, controlling the DMA to read the pyramid image layer, the image gradient and the Hessian matrix in the first external memory, the second external memory, the third external memory and the fourth external memory respectively according to a fourth instruction.
In this embodiment, the intermediate data of the hessian matrix is data generated during calculation of the hessian matrix corresponding to the image, the fourth instruction is instruction information for the terminal to send and read an image of a certain layer in the pyramid layer, a gradient of the image, the hessian matrix, and an image of the same layer in the other pyramid layer to the DMA, and after receiving the fourth instruction, the DMA can read an image of the same layer in the two pyramid layers, a gradient of one of the images, and the hessian matrix in the first external memory, the second external memory, the third external memory, and the fourth external memory, respectively.
It can be understood that the first external memory, the second external memory, the third external memory, and the fourth external memory may be the same memory or different memories, and the pyramid layer, the image gradient, and the hessian matrix may be stored in the same external memory or in a plurality of external memories, which may be specifically selected according to practical applications, and this embodiment is not limited specifically.
In another embodiment, as shown in fig. 7, step S140 includes steps S141 and S142.
S141, carrying out iterative search matching on image blocks of a top layer image in the pyramid image layer according to the image gradient and the Hessian matrix to obtain dense optical flow between the top layer images;
and S142, taking the optical flow information of the top layer image as an initial quantity, and performing iterative search and matching on image blocks of the images of the rest layers in the pyramid image layer from top to bottom to obtain dense optical flows between the two continuous frames of images.
Specifically, in the whole process of generating the dense optical flow between two continuous images, the dense optical flow between two continuous images is optimized layer by layer from coarse to fine, that is, the dense optical flow between top images in two pyramid image layers is generated in advance, then the dense optical flow between the current images is used as an initial value, the steps S120 to S140 are executed again to generate the dense optical flow between the next images, the dense optical flow between the next images is used as the initial value again, and the steps S120 to S140 are executed again at the same time, so that the dense optical flow between the two continuous images can be obtained finally.
In the embodiment, each image block of one top-level image in two pyramid image layers, the gradient of the corresponding image block and the hessian matrix are subjected to iterative search and matching in the other top-level image, so that the dense optical flow between the top-level images can be obtained, then the dense optical flow between the top-level images is used as an initial value, and the steps S120-S140 are executed in a circulating mode, so that the dense optical flow between two continuous frames of images can be obtained finally.
In the dense optical flow generation method provided by the embodiment of the invention, two continuous frames of images are written into a shared memory according to a preset DMA (direct memory access), and a pyramid graph layer of each image is generated; writing the pyramid layer into the shared memory according to the DMA, and generating an image gradient of the pyramid layer; writing the image gradients into the shared memory according to the DMA, and generating a Hessian matrix of the images in the pyramid layer; writing the pyramid layer, the image gradient, and the hessian matrix into the shared memory according to the DMA, and generating a dense optical flow between the two successive frames of images according to the pyramid layer, the image gradient, and the hessian matrix. According to the invention, the calculation data required by the dense optical flow algorithm is written into the shared memory in advance through the DMA so as to reduce SRAM resources occupied by the dense optical flow algorithm, thereby greatly reducing the cost of the dense optical flow algorithm and improving the calculation efficiency.
Embodiments of the present invention also provide a dense optical flow generation apparatus 100 for performing any of the foregoing dense optical flow generation methods.
Specifically, referring to fig. 8, fig. 8 is a schematic block diagram of a dense optical flow generating apparatus 100 according to an embodiment of the present invention.
As shown in fig. 8, the dense optical flow generation apparatus 100 includes: a first generation unit 110, a second generation unit 120, a third generation unit 130, and a fourth generation unit 140.
A first generating unit 110 for writing two frames of consecutive images into the shared memory according to a preset DMA and generating a pyramid layer for each of the images.
In other embodiments of the invention, the dense optical flow generating apparatus 100 further comprises: a first reading unit.
And the first reading unit is used for controlling the DMA to read the image in the first external memory according to a first instruction.
A second generating unit 120, configured to write the pyramid layer into the shared memory according to the DMA, and generate an image gradient of the pyramid layer.
In other embodiments of the present invention, the dense optical flow generating apparatus 100 further comprises: a first writing unit and a second reading unit.
A first writing unit, configured to write intermediate data for generating the pyramid map layer into the shared memory, and write the pyramid map layer into a second external memory according to the DMA; and the second reading unit is used for controlling the DMA to read one layer of image in the pyramid image layer in the second external memory according to a second instruction.
A third generating unit 130, configured to write the image gradient into the shared memory according to the DMA, and generate a hessian matrix of the image in the pyramid layer.
In other embodiments of the present invention, the dense optical flow generating apparatus 100 further comprises: a second write unit and a third read unit.
A second writing unit configured to write intermediate data for generating the image gradient into the shared memory, and write the image gradient into a third external memory according to the DMA; and the third reading unit is used for controlling the DMA to read the image gradient in the third external memory according to a third instruction.
In other inventive embodiments, the third generating unit 130 includes: an acquisition unit and a fifth generation unit.
The acquisition unit is used for acquiring the blocking information of the image in the pyramid image layer; and the fifth generating unit is used for generating a Hessian matrix of the image block in the pyramid image layer and an inverse matrix of the Hessian matrix according to the blocking information and the image gradient.
A fourth generating unit 140, configured to write the pyramid layer, the image gradient, and the hessian matrix into the shared memory according to the DMA, and generate a dense optical flow between the two consecutive frames of images according to the pyramid layer, the image gradient, and the hessian matrix.
In other embodiments of the present invention, the dense optical flow generating apparatus 100 further comprises: a third write unit and a fourth read unit.
A third writing unit, configured to write intermediate data for generating the hessian matrix into the shared memory, and write the hessian matrix into a fourth external memory according to the DMA; a fourth reading unit, configured to control the DMA to read the pyramid layer, the image gradient, and the hessian matrix in the first external memory, the second external memory, the third external memory, and the fourth external memory, respectively, according to a fourth instruction.
In other inventive embodiments, the fourth generating unit 140 includes: a first matching unit and a second matching unit.
The first matching unit is used for carrying out iterative search matching on image blocks of a top layer image in the pyramid image layer according to the image gradient and the Hessian matrix to obtain dense optical flows between the top layer images; and the second matching unit is used for taking the optical flow information of the top layer image as an initial quantity, and performing iterative search matching on image blocks of the images of the rest layers in the pyramid image layer from top to bottom to obtain dense optical flows between the two continuous frames of images.
The dense optical flow generating device 100 provided by the embodiment of the invention is used for writing two continuous frames of images into a shared memory according to a preset DMA and generating a pyramid image layer of each image; writing the pyramid map layer into the shared memory according to the DMA, and generating an image gradient of the pyramid map layer; writing the image gradients into the shared memory according to the DMA, and generating a Hessian matrix of the images in the pyramid layer; writing the pyramid layer, the image gradient, and the hessian matrix into the shared memory according to the DMA, and generating a dense optical flow between the two successive frames of images according to the pyramid layer, the image gradient, and the hessian matrix.
It should be noted that, as can be clearly understood by those skilled in the art, the detailed implementation process of the above-mentioned dense optical flow generating apparatus 100 and each unit may refer to the corresponding description in the foregoing method embodiment, and for convenience and conciseness of description, detailed description is omitted here.
The dense optical flow generation apparatus described above may be implemented in the form of a computer program that can be run on an electronic device as shown in fig. 9.
Referring to fig. 9, fig. 9 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 500 may be a terminal, wherein the terminal may be an electronic device having a communication function, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, and a wearable device.
Referring to fig. 9, the electronic device 500 includes a processor 502, a memory, and a network interface 505 connected by a system bus 501, wherein the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer programs 5032 comprise program instructions that, when executed, cause the processor 502 to perform a dense optical flow generation method.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall electronic device 500.
The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503, which computer program 5032, when executed by the processor 502, causes the processor 502 to perform a dense optical flow generation method.
The network interface 505 is used for network communication with other devices. It will be understood by those skilled in the art that the structure shown in fig. 9 is a block diagram of only a portion of the structure related to the present application, and does not constitute a limitation to the electronic device 500 to which the present application is applied, and a particular electronic device 500 may include more or less components than those shown in the drawings, or may combine certain components, or have a different arrangement of components.
Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps: writing two continuous frames of images into a shared memory according to a preset DMA (direct memory access), and generating a pyramid image layer of each image; writing the pyramid layer into the shared memory according to the DMA, and generating an image gradient of the pyramid layer; writing the image gradients into the shared memory according to the DMA, and generating a Hessian matrix of the images in the pyramid layer; writing the pyramid layer, the image gradient, and the hessian matrix into the shared memory according to the DMA, and generating a dense optical flow between the two successive frames of images according to the pyramid layer, the image gradient, and the hessian matrix.
In an embodiment, before the processor 502 writes two consecutive images into the shared memory according to the preset DMA, the following steps are specifically implemented: and controlling the DMA to read the image in a first external memory according to a first instruction.
In an embodiment, before the processor 502 implements the writing of the pyramid graph layer into the shared memory according to the DMA, the following steps are specifically implemented: writing the intermediate data for generating the pyramid graph layer into the shared memory, and writing the pyramid graph layer into a second external memory according to the DMA; and controlling the DMA to read one layer of image in the pyramid image layer in the second external memory according to a second instruction.
In an embodiment, before implementing the writing of the image gradient into the shared memory according to the DMA, the processor 502 specifically implements the following steps: writing intermediate data for generating the image gradient into the shared memory, and writing the image gradient into a third external memory according to the DMA; and controlling the DMA to read the image gradient in the third external memory according to a third instruction.
In an embodiment, the processor 502 specifically implements the following steps in generating the hessian matrix of the image in the pyramid map layer: acquiring block information of the image in the pyramid image layer; and generating a Hessian matrix of the image block in the pyramid image layer and an inverse matrix of the Hessian matrix according to the block information and the image gradient.
In an embodiment, before the processor 502 implements the writing of the pyramid map layer, the image gradient, and the hessian matrix into the shared memory according to the DMA, the following steps are implemented: writing intermediate data for generating the Hessian matrix into the shared memory, and writing the Hessian matrix into a fourth external memory according to the DMA; and controlling the DMA to read the pyramid image layer, the image gradient and the Hessian matrix in the first external memory, the second external memory, the third external memory and the fourth external memory respectively according to a fourth instruction.
In an embodiment, the processor 502 implements the following steps in generating the dense optical flow between the two consecutive images according to the pyramid map layer, the image gradient, and the hessian matrix: performing iterative search matching on image blocks of top images in the pyramid image layer according to the image gradient and the Hessian matrix to obtain dense optical flows between the top images; and taking the optical flow information of the top layer image as an initial quantity, and performing iterative search matching on image blocks of the images of the rest layers in the pyramid image layer from top to bottom to obtain a dense optical flow between the two continuous frames of images.
It should be understood that, in the embodiment of the present Application, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general-purpose processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It will be understood by those skilled in the art that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program instructing relevant hardware. The computer program includes program instructions, and the computer program may be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.
Accordingly, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program comprises program instructions. The program instructions, when executed by the processor, cause the processor to perform the steps of: writing two continuous frames of images into a shared memory according to a preset DMA (direct memory access), and generating a pyramid image layer of each image; writing the pyramid map layer into the shared memory according to the DMA, and generating an image gradient of the pyramid map layer; writing the image gradients into the shared memory according to the DMA, and generating a Hessian matrix of the image in the pyramid layer; writing the pyramid layer, the image gradient, and the hessian matrix into the shared memory according to the DMA, and generating a dense optical flow between the two successive frames of images according to the pyramid layer, the image gradient, and the hessian matrix.
In an embodiment, before the processor executes the program instructions to implement the writing of two consecutive images into the shared memory according to a preset DMA, the following steps are specifically implemented: and controlling the DMA to read the image in a first external memory according to a first instruction.
In an embodiment, before the processor executes the program instructions to implement the writing of the pyramid graph layer into the shared memory according to the DMA, the following steps are specifically implemented: writing intermediate data for generating the pyramid graph layer into the shared memory, and writing the pyramid graph layer into a second external memory according to the DMA; and controlling the DMA to read one layer of image in the pyramid image layer in the second external memory according to a second instruction.
In one embodiment, before executing the program instructions to implement the writing of the image gradients according to the DMA into the shared memory, the processor implements the following steps: writing intermediate data for generating the image gradient into the shared memory, and writing the image gradient into a third external memory according to the DMA; and controlling the DMA to read the image gradient in the third external memory according to a third instruction.
In an embodiment, the processor, when executing the program instructions, implements the generating of the hessian matrix of the image in the pyramid layer by specifically implementing the following steps: acquiring block information of the image in the pyramid image layer; and generating a Hessian matrix of the image block in the pyramid image layer and an inverse matrix of the Hessian matrix according to the block information and the image gradient.
In an embodiment, before executing the program instructions to implement the writing of the pyramid layer, the image gradients, and the hessian matrix into the shared memory according to the DMA, the processor specifically implements the following steps: writing intermediate data for generating the Hessian matrix into the shared memory, and writing the Hessian matrix into a fourth external memory according to the DMA; and controlling the DMA to read the pyramid image layer, the image gradient and the Hessian matrix in the first external memory, the second external memory, the third external memory and the fourth external memory respectively according to a fourth instruction.
In an embodiment, the processor, when executing the program instructions, implements the generating of the dense optical flow between the two consecutive images according to the pyramid image layer, the image gradient, and the hessian matrix by specifically implementing the following steps: performing iterative search matching on image blocks of top images in the pyramid image layer according to the image gradient and the Hessian matrix to obtain dense optical flows between the top images; and taking the optical flow information of the top layer image as an initial quantity, and performing iterative search matching on image blocks of the images of the rest layers in the pyramid image layer from top to bottom to obtain a dense optical flow between the two continuous frames of images.
The storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, which can store various computer readable storage media of program codes.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the technical solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.
The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be combined, divided and deleted according to actual needs. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing an electronic device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A dense optical flow generation method, comprising:
writing two continuous frames of images into a shared memory according to a preset DMA (direct memory access), and generating a pyramid image layer of each image;
writing the pyramid layer into the shared memory according to the DMA, and generating an image gradient of the pyramid layer;
writing the image gradients into the shared memory according to the DMA, and generating a Hessian matrix of the image in the pyramid layer;
and writing the pyramid image layer, the image gradient and the hessian matrix into the shared memory according to the DMA, and generating dense optical flow between the two continuous frames of images according to the pyramid image layer, the image gradient and the hessian matrix.
2. The method of dense optical flow generation according to claim 1, further comprising, before said writing two consecutive frames of images into a shared memory according to a preset DMA:
and controlling the DMA to read the image in a first external memory according to a first instruction.
3. The dense optical flow generation method of claim 1, further comprising, prior to said writing the pyramid graph layer to the shared memory according to the DMA:
writing intermediate data for generating the pyramid graph layer into the shared memory, and writing the pyramid graph layer into a second external memory according to the DMA;
and controlling the DMA to read one layer of image in the pyramid image layer in the second external memory according to a second instruction.
4. The dense optical flow generation method of claim 1, further comprising, prior to said writing the image gradients to the shared memory according to the DMA:
writing intermediate data for generating the image gradient into the shared memory, and writing the image gradient into a third external memory according to the DMA;
and controlling the DMA to read the image gradient in the third external memory according to a third instruction.
5. The method of dense optical flow generation of claim 1, wherein said generating a hessian matrix of images in the pyramid layer comprises:
acquiring block information of the image in the pyramid image layer;
and generating a Hessian matrix of the image block in the pyramid image layer and an inverse matrix of the Hessian matrix according to the block information and the image gradient.
6. The dense optical flow generation method of claim 1, prior to said writing the pyramid graph layer, the image gradients, the hessian matrix into the shared memory according to the DMA, further comprising:
writing intermediate data for generating the Hessian matrix into the shared memory, and writing the Hessian matrix into a fourth external memory according to the DMA;
and controlling the DMA to read the pyramid image layer, the image gradient and the Hessian matrix in the first external memory, the second external memory, the third external memory and the fourth external memory respectively according to a fourth instruction.
7. The dense optical flow generation method according to claim 1, wherein said generating dense optical flow between said two successive images from said pyramid layers, said image gradients, said hessian matrix comprises:
performing iterative search matching on image blocks of top images in the pyramid image layer according to the image gradient and the Hessian matrix to obtain dense optical flows between the top images;
and taking the optical flow information of the top layer image as an initial quantity, and performing iterative search matching on image blocks of the images of the rest layers in the pyramid image layer from top to bottom to obtain the dense optical flow between the two continuous frames of images.
8. A dense optical flow generation apparatus, comprising:
the device comprises a first generation unit, a second generation unit and a third generation unit, wherein the first generation unit is used for writing two frames of continuous images into a shared memory according to a preset DMA (direct memory access) and generating a pyramid layer of each image;
a second generation unit configured to write the pyramid image layer into the shared memory according to the DMA, and generate an image gradient of the pyramid image layer;
a third generation unit configured to write the image gradient into the shared memory according to the DMA, and generate a hessian matrix of an image in the pyramid layer;
a fourth generating unit, configured to write the pyramid layer, the image gradient, and the hessian matrix into the shared memory according to the DMA, and generate a dense optical flow between the two consecutive frames of images according to the pyramid layer, the image gradient, and the hessian matrix.
9. An electronic device comprising a memory and a processor; the memory stores an application program, and the processor is configured to execute the application program in the memory to perform the operations of the dense optical flow generation method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that a computer program is stored thereon, which is executed by a processor to implement the dense optical flow generation method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211284018.XA CN115359097A (en) | 2022-10-20 | 2022-10-20 | Dense optical flow generation method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211284018.XA CN115359097A (en) | 2022-10-20 | 2022-10-20 | Dense optical flow generation method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115359097A true CN115359097A (en) | 2022-11-18 |
Family
ID=84008018
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211284018.XA Pending CN115359097A (en) | 2022-10-20 | 2022-10-20 | Dense optical flow generation method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115359097A (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102509071A (en) * | 2011-10-14 | 2012-06-20 | 江南大学 | Light stream calculation system and method |
| US20180181816A1 (en) * | 2016-12-22 | 2018-06-28 | Texas Instruments Incorporated | Handling Perspective Magnification in Optical Flow Proessing |
| US20190005656A1 (en) * | 2017-06-29 | 2019-01-03 | Texas Instruments Incorporated | Dense Optical Flow Processing in a Computer Vision System |
| US20190005335A1 (en) * | 2017-06-29 | 2019-01-03 | Texas Instruments Incorporated | Hierarchical Data Organization for Dense Optical Flow Processing in a Computer Vision System |
| US20190325561A1 (en) * | 2018-04-24 | 2019-10-24 | Snap Inc. | Efficient parallel optical flow algorithm and gpu implementation |
| CN111275746A (en) * | 2020-01-19 | 2020-06-12 | 浙江大学 | Dense optical flow computing system and method based on FPGA |
| CN113066001A (en) * | 2021-02-26 | 2021-07-02 | 华为技术有限公司 | Image processing method and related equipment |
| CN113673545A (en) * | 2020-05-13 | 2021-11-19 | 华为技术有限公司 | Optical flow estimation method, related device, equipment and computer readable storage medium |
| CN113780064A (en) * | 2021-07-27 | 2021-12-10 | 华为技术有限公司 | Target tracking method and device |
| CN114119661A (en) * | 2021-11-23 | 2022-03-01 | 上海商汤科技开发有限公司 | Target tracking processor, target tracking method and device |
| CN114511596A (en) * | 2020-10-23 | 2022-05-17 | 华为技术有限公司 | Data processing method and related equipment |
-
2022
- 2022-10-20 CN CN202211284018.XA patent/CN115359097A/en active Pending
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102509071A (en) * | 2011-10-14 | 2012-06-20 | 江南大学 | Light stream calculation system and method |
| US20180181816A1 (en) * | 2016-12-22 | 2018-06-28 | Texas Instruments Incorporated | Handling Perspective Magnification in Optical Flow Proessing |
| CN110651475A (en) * | 2017-06-29 | 2020-01-03 | 德州仪器公司 | Hierarchical data organization for dense optical flows |
| US20190005335A1 (en) * | 2017-06-29 | 2019-01-03 | Texas Instruments Incorporated | Hierarchical Data Organization for Dense Optical Flow Processing in a Computer Vision System |
| CN110637461A (en) * | 2017-06-29 | 2019-12-31 | 德州仪器公司 | Densified optical flow processing in computer vision systems |
| US20190005656A1 (en) * | 2017-06-29 | 2019-01-03 | Texas Instruments Incorporated | Dense Optical Flow Processing in a Computer Vision System |
| US20190325561A1 (en) * | 2018-04-24 | 2019-10-24 | Snap Inc. | Efficient parallel optical flow algorithm and gpu implementation |
| CN112041887A (en) * | 2018-04-24 | 2020-12-04 | 斯纳普公司 | Efficient parallel optical flow algorithm and GPU implementation |
| CN111275746A (en) * | 2020-01-19 | 2020-06-12 | 浙江大学 | Dense optical flow computing system and method based on FPGA |
| CN113673545A (en) * | 2020-05-13 | 2021-11-19 | 华为技术有限公司 | Optical flow estimation method, related device, equipment and computer readable storage medium |
| CN114511596A (en) * | 2020-10-23 | 2022-05-17 | 华为技术有限公司 | Data processing method and related equipment |
| CN113066001A (en) * | 2021-02-26 | 2021-07-02 | 华为技术有限公司 | Image processing method and related equipment |
| CN113780064A (en) * | 2021-07-27 | 2021-12-10 | 华为技术有限公司 | Target tracking method and device |
| CN114119661A (en) * | 2021-11-23 | 2022-03-01 | 上海商汤科技开发有限公司 | Target tracking processor, target tracking method and device |
Non-Patent Citations (3)
| Title |
|---|
| GOOITZEN VAN DER WAL 等: "FPGA Acceleration for Feature Based Processing Applications", 《2015 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW)》 * |
| 吴进等: "一种基于OpenCL的Lukas-Kanade光流并行加速算法", 《电讯技术》 * |
| 周艳等: "基于视频场景多因素的人群运动状态分析", 《地理信息世界》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112967381B (en) | Three-dimensional reconstruction method, apparatus and medium | |
| US11822900B2 (en) | Filter processing device and method of performing convolution operation at filter processing device | |
| CN111709879B (en) | Image processing method, image processing device and terminal equipment | |
| CN113643414A (en) | Three-dimensional image generation method and device, electronic equipment and storage medium | |
| CN114584785A (en) | Real-time image stabilizing method and device for video image | |
| US11055820B2 (en) | Methods, apparatus and processor for producing a higher resolution frame | |
| US10212406B2 (en) | Image generation of a three-dimensional scene using multiple focal lengths | |
| CN111583329B (en) | Augmented reality glasses display method and device, electronic equipment and storage medium | |
| CN116739901A (en) | Video super-processing method and device, electronic equipment and storage medium | |
| CN115359097A (en) | Dense optical flow generation method and device, electronic equipment and readable storage medium | |
| CN114022518B (en) | Method, device, equipment and medium for acquiring optical flow information of image | |
| CN115797194A (en) | Image denoising method, image denoising device, electronic device, storage medium, and program product | |
| JP2004280157A (en) | Image processor | |
| US11704546B2 (en) | Operation processing apparatus that calculates addresses of feature planes in layers of a neutral network and operation processing method | |
| CN115049572A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN115914497A (en) | Video processing method, device, equipment, medium and program product | |
| KR20120043209A (en) | Method and apparatus of executing pixel calculation within window area at high speed in window-based image processing | |
| CN115660935B (en) | Method and system for processing graphics in partitioned rendering mode | |
| CN115564644B (en) | Image data processing method, related device and computer storage medium | |
| CN116310227B (en) | Three-dimensional dense reconstruction method, three-dimensional dense reconstruction device, electronic equipment and medium | |
| US20230140932A1 (en) | Method and device for outputting an image | |
| CN117974441A (en) | Method, apparatus and computer program product for image processing | |
| CN115690176A (en) | Image processing method, apparatus, device and medium | |
| CN118379200A (en) | Image filtering processing method, device, electronic equipment and storage medium | |
| CN118134944A (en) | Image processing method, device, terminal equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20221118 |
|
| RJ01 | Rejection of invention patent application after publication |