CN115952139B

CN115952139B - Multi-frame three-dimensional image processing method and system for mobile equipment

Info

Publication number: CN115952139B
Application number: CN202310243477.1A
Authority: CN
Inventors: 杨涛
Original assignee: Wuhan Xinyundao Data Technology Co ltd
Current assignee: Wuhan Xinyundao Data Technology Co ltd
Priority date: 2023-03-14
Filing date: 2023-03-14
Publication date: 2023-06-30
Anticipated expiration: 2043-03-14
Also published as: CN115952139A

Abstract

The invention relates to the field of computers, in particular to a multi-frame three-dimensional image processing method and system of mobile equipment, which comprises the steps of firstly reading and analyzing a three-dimensional multi-frame file, and storing the three-dimensional multi-frame file into a memory according to a pixel coordinate mode to obtain first data; then, an arm register is called to load first data, the first data is operated by using a Neon mathematical instruction set and then stored into a memory, and second data is obtained; and finally, calling an arm register to load second data, and performing rasterization superposition on the second data based on a ray projection method by using a Neon mathematical instruction set to obtain final data and storing the final data into a memory. By adopting the analysis method of the invention, the performance of accessing the memory data is greatly enhanced.

Description

Multi-frame three-dimensional image processing method and system for mobile equipment

Technical Field

The invention relates to the field of computers, in particular to a method and a system for processing a multi-frame three-dimensional image of a mobile device, which are used for rendering and displaying after three-dimensional reconstruction, by the mobile device (a tablet mobile phone) of an Arm platform for processing the ultra-large multi-frame three-dimensional image stored on a cloud server.

Background

The 3D reconstruction and rendering of multi-frame three-dimensional images has been realized and displayed mainly by high-performance PCs in the past, and with the popularization of mobile devices (mobile phone tablet), more and more processing is performed on the mobile devices. The hardware resources and processing power of the mobile device are weaker than the high performance pc.

Conventionally, a multi-frame three-dimensional image is loaded from a cloud server or a local disk, multi-frame data is read and buffered into a memory frame by frame to form a memory array taking a frame as a unit, a light ray projection method is used for traversing and operating a pixel area corresponding to each frame in each memory array, and finally two-dimensional rasterized data is formed and displayed (refer to fig. 1).

For example: the prior art discloses a method for generating a front-view image map based on a live-action three-dimensional model (publication number: CN 114627237A), which illustrates a method for three-dimensional reconstruction of TIFF, but does not have an image to an arm platform, and the method is not optimized for an arm end, so that the actual operation efficiency on the arm platform is not high. The prior art also discloses methods for creating, storing and providing access to three-dimensional scanned images (publication No. 103038780 a), storing and indexing data of the three-dimensional reconstruction stage through a database, which only allows for application logic storage and access simplicity and does not optimize low latency high performance data processing for arm memory physical layout and TIFF three-dimensional reconstruction.

Disclosure of Invention

The technical problems of the invention are mainly solved by the following technical proposal:

a multi-frame three-dimensional image processing method for mobile equipment based on an arm platform comprises

Reading and analyzing a three-dimensional multi-frame file, and storing the three-dimensional multi-frame file into a memory according to a pixel coordinate mode to obtain first data;

calling an arm register to load first data, performing operation on the first data by using a Neon mathematical instruction set, and storing the first data into a memory to obtain second data;

and calling an arm register to load second data, and performing rasterization superposition on the second data based on a ray projection method by using a Neon mathematical instruction set to obtain final data and storing the final data into a memory.

In the above-mentioned multi-frame three-dimensional image processing method for mobile equipment,

reading a file header of a three-dimensional multi-frame file and applying for a preprocessing memory;

and reading pixel data of the three-dimensional multi-frame file in parallel, and storing the pixel data into a preprocessing memory.

calculating the size of a memory block to be allocated according to the multi-frame three-dimensional file information, the memory size of the ARM system and the number of light rays projected by the light rays;

dividing the applied memory block into memory sub-blocks according to the number of rays, and obtaining a memory block composed of a plurality of memory sub-blocks.

defining the number of file frames of a multi-frame file as nFrame, the number of occupied bytes of each pixel point as nBitCount, the number of rays projected by rays as nRay, the number of pixels sampled by each ray as nPixel, and the memory size of an ARM system as nDDR, and applying for the memory size of nMEm=nframe, nBitCount, nDDR, which is aligned by 4 bytes, in advance.

In the above method for processing multi-frame three-dimensional images of mobile equipment, the number nmemcount=nmem/nRay of a plurality of memory subblocks, and the head address of each memory subblock pre-allocated is stored in the cache.

the method comprises the steps of parallelly reading pixel data of a multi-frame file according to an openmp module of an ARM platform, specifically comprising first-stage pixel data processing and second-stage pixel data processing, wherein the first data is pixel data after secondary processing;

and storing the first data into the pre-allocated memory block.

processing the first-level pixel data into cyclic traversal of each frame of data;

the second-stage pixel data processing is to circularly store the sampling rectangular pixels of the light projection method into the pre-allocation memory in the direction of increasing the abscissa and the ordinate in each frame of data.

In the above method for processing a multi-frame three-dimensional image of a mobile device, the number of rays of a ray projection is defined as nray=4, the resolution of a single frame is 2×2, the number of pixels sampled by each ray is npixel=1, the number of frames of a multi-frame three-dimensional file is nframe=2, the number of occupied bytes of each pixel is nBitCount, the starting address of a memory block is nrddradrinit, the storage address of an nth pixel of an mth frame in a memory is nrddraddriit+ (npixel×nray+m) ×nbitcount=nrddradrinit+ (4n+m) ×nbitcount, wherein 0< m < nFrame, and 0< n < 4).

loading the first data into an ARM Neon double-word vector register group;

using a Neon statics instruction set to carry out mathematical operation in a ray projection algorithm, and storing processed data of each frame rectangle into a memory to obtain second data; loading the second data into an ARM Neon double-word vector register group, rasterizing the ARM Neon double-word vector register group, uniformly superposing the data with the total nFrame frame number once by using a Neon instruction, and writing the superposed data into a memory to obtain final three-dimensional reconstructed image data.

A mobile device multi-frame three-dimensional image processing system, comprising:

a first module: the system is configured to read and analyze three-dimensional multi-frame files in parallel, and store the three-dimensional multi-frame files into a memory according to a pixel coordinate mode to obtain first data;

a second module: the method comprises the steps of being configured to call an arm register to load first data, operating the arm register by using a Neon mathematical instruction set, and then storing the arm register into a memory to obtain second data;

and a third module: and the array register is configured to be called to load second data, and the second data is subjected to rasterization superposition based on a ray projection method by using a Neon mathematical instruction set to obtain final data, and the final data is stored into a memory.

Therefore, the invention has the following advantages: 1. and in the three-dimensional multi-frame file reading stage, parallel processing is performed by using an arm openmp technology, so that the reading and analyzing performances of the cloud server file are accelerated. 2. The read multi-frame data is stored in the memory according to pixel coordinates instead of being stored in frame units, so that the processing performance of a subsequent light projection method can be accelerated, and the performance of accessing the memory data is greatly enhanced. 3. The pipeline of ray projection rasterization operations is accelerated in parallel using the armneon technique. 4. The zero instruction is used to increase the processing bandwidth and reduce the occupancy rate of the multi-core cpu. 5. After the memory access layout optimization process, the total amount of physical memory usage is reduced. After cpu and memory resource usage is reduced, ecological compatibility to fragmented mobile devices is better.

Drawings

FIG. 1 is a schematic flow diagram of a prior art process.

FIG. 2 is a schematic flow chart of the method of the present invention.

Fig. 3 is a diagram of a prior art multi-frame data memory layout.

Fig. 4 is a diagram of a multi-frame data memory layout according to the present invention.

FIG. 5 is a schematic diagram of a TIFF file loading process.

FIG. 6 is a schematic diagram of a prior art ray-casting sequential process.

FIG. 7 is a schematic diagram of the acceleration flow of the ray projection algorithm using ARM Neon technology.

Detailed Description

The technical scheme of the invention is further specifically described below through examples and with reference to the accompanying drawings.

Examples:

the following specifically expands description with reference to a TIFF file as an example of a multi-frame three-dimensional image.

1. Firstly, the whole process of resolving the TIFF file in the prior art is described and comprises the following steps:

1. reading the TIFF file head, analyzing DE (Directory Entry) directory entry, and obtaining offset of each frame relative to the file head in the multi-frame data according to the directory entry;

2. sequentially reading single-frame file data from a first frame, analyzing pixels (RGB) of each coordinate, storing the single-frame file data in a memory, and directly applying from a pile before the memory is used, wherein the final memory layout is sequentially X Y nBit n, x=image width, y=image height, nbit=the number of bytes (generally 3 or 4) stored in each pixel RGB, and n=frame number;

3. and carrying out light projection operation combination on the same pixel area of the multi-frame images.

2. The invention improves the data processing links, and comprises the following detailed steps:

1. and reading the file header of the TIFF image file, and applying for a preprocessing memory for the ray projection method of the multi-frame image. Because the ray projection operation process needs to access a large amount of memory, the memory buffer design needs to improve the access performance as much as possible. The execution steps are as follows: A. according to TIFF file information, ARM system memory size, ray number of ray projection and memory block size to be allocated, assuming that the TIFF file frame number is nFrame, the occupied byte number of each pixel is nBitCount, the ray number of ray projection is nRay, the pixel number of each ray sample is nPixel, and the ARM system memory size is nDDR, then the memory size nMEM=nFrame nBitCount nDDR nPixel aligned by 4 bytes is applied in advance. In general, nMem should be at most 50% of the nDDR, i.e., ARM system memory, and too much application may degrade other system service performance.

B. The applied memory block is divided into sub-blocks according to the number of rays, the number of sub-blocks nmemcount=nmem/nRay, the first address of each sub-block needs to be aligned with 64 bytes, and an ARM PLD command is pre-stored in a cache (64-byte alignment is needed to enable PLD command to execute correctly, because the access requirement of the cache line is address access with 64-byte alignment). Therefore, when the memory data operation is carried out subsequently, the head address of the memory block can be directly obtained from the cache, and the read-write operation is faster.

The TIFF multi-frame file analysis comprises the following steps:

A. the pixel data of the multi-frame file is read in parallel by using the openmp technology of the ARM platform, and the parallel access is supported on a mechanism because the TIFF oversized file is stored in the cloud. Because cloud data is distributed storage, parallel accesses to different data segments of the same file are distributed to different processing network storage areas to perform reading and writing. The parallel processing design is as follows (see fig. 5):

a. opening a cloud file;

b. the first-stage circulation traverses each frame of data, and the second-stage circulation stores sampling rectangular pixels of the light projection method into a pre-allocation memory in each frame of data according to the increasing direction of the abscissa;

c. because there is no correlation between frames and pixel data correlation within a single frame, parallel processing of nested 2-level openmp is used, with the first level processing single frame data manipulation and the second level processing intra-frame pixel data processing.

B. And (3) saving the pixel data after each frame of data analysis to a memory pre-allocated in the step (1). According to 2 frames of images, 4 pixels of each frame of images are illustrated by a simple model that the ray projection ray processing granularity is one pixel, the image data in the prior art is organized according to a frame sequence of a text TIFF (refer to fig. 3), the memory data is organized according to the ray penetration sequence of a ray projection method (refer to fig. 4), and the memory access efficiency of the ray projection by the memory layout is higher because the algorithm processing is operated and overlapped according to the same pixel area of each frame, that is, the same area of all frames needs to be processed at one time, and then the processing is switched to the next area for processing.

3. The arm neon technique is used to optimize the ray-projection process flow.

Even though the data can be accessed efficiently by being buffered in the pre-allocated memory, the algorithm can be improved to optimize the data using the arm neon because the calculation amount is large and the access processing of the data is high-frequency.

In the prior art, the ray projection method processing flow sequentially performs operation and reading and writing on the memory data (refer to fig. 6).

The invention uses the arm neon instruction set to reduce parallel memory read-write, because the 128-bit arm register is used to improve the performance by more than 5 times than 128-bit memory access.

The specific implementation steps are as follows (refer to fig. 7):

A. loading frame rectangular block pixel set data in a memory into an ARM Neon double-word vector register group;

B. mathematical operations in the ray-casting algorithm are performed using the Neon statics instruction set, and are directly performed using the Neon mathematical instruction set, without the conventional ARMRISC instruction set. After the processing is finished, the rectangular processed data of each frame needs to be stored into a memory, and all data cannot be stored because the ARM Neon register space is limited and must be buffered into the memory;

C. the final step of the projection method is to rasterize the rectangular calculation data of all frames, that is, to uniformly superimpose the data of the total nFrame number once. The processing is that the data in the memory is loaded into the Neon register group, and the Neon instruction is directly used for superposition, and the superposed data is written into the memory.

Overall, although using Neon registers increases 2 times of memory and copies among registers, this overhead is far greater than accessing the memory multiple times and using the traditional ARM risc instruction set to perform mathematical operations, the former only transfers data with the first and last registers and memory, other instruction operations are directly at the register level and not completed, and each operation of the latter is performed sequentially: memory fetch instruction- > memory data load register- > register uses risc instruction set to calculate- > register data write to memory.

In this embodiment, the system further relates to a mobile device multi-frame three-dimensional image processing system, including:

3. The analysis method is applied to the specific configuration: in ARM Cortex-A76+linux 5.10 edition and memory 8G equipment, the parameters of TIFF files stored on a cloud server are as follows: epfl_tracking. Tif/file size: 123.90 MB/resolution: 1024 726 x 165/165 frames; opening a file on an ARM system by using the prior art scheme and rendering by a light projection method of a VTK, wherein the total time is 12 seconds; with the optimization scheme of the invention, the final rendering completion total market is 5 seconds, and the improvement is nearly 120%.

The specific embodiments described herein are offered by way of example only to illustrate the spirit of the invention. Those skilled in the art may make various modifications or additions to the described embodiments or substitutions thereof without departing from the spirit of the invention or exceeding the scope of the invention as defined in the accompanying claims.

Claims

1. A multi-frame three-dimensional image processing method for mobile equipment is characterized by comprising the following steps based on an arm platform

calling an arm register to load second data, and performing rasterization superposition on the second data based on a ray projection method by using a Neon mathematical instruction set to obtain final data and storing the final data into a memory;

the method for reading and analyzing the three-dimensional multi-frame file specifically comprises the following steps: reading a file header of a three-dimensional multi-frame file and applying for a preprocessing memory;

the pixel data of the three-dimensional multi-frame file are read in parallel and stored into a preprocessing memory;

calculating the size of a memory block to be allocated according to the multi-frame three-dimensional file information, the ARM system memory size and the number of light rays projected by the light rays;

dividing the applied memory block into memory sub-blocks according to the number of rays to obtain a memory block composed of a plurality of memory sub-blocks;

defining the number of file frames of a multi-frame file as nFrame, the number of occupied bytes of each pixel point as nBitCount, the number of rays projected by rays as nRay, the number of pixels sampled by each ray as nPixel, and the memory size of an ARM system as nDDR, and applying for the memory size of 4 bytes aligned nmem=nframe nBitcount nDDR;

the number nmemcount=nmem/nRay of the memory subblocks, and storing the head address of each memory subblock pre-allocated into the cache;

the method for carrying out rasterization superposition on the optical fiber based on the ray projection method by using a Neon mathematical instruction set specifically comprises the following steps:

loading the first data into an ARM Neon double-word vector register group;

2. A mobile device multi-frame three-dimensional image processing method according to claim 1, wherein,

and storing the first data into the pre-allocated memory block.

3. A mobile device multi-frame three-dimensional image processing method according to claim 2, wherein,

4. A multi-frame three-dimensional image processing method of a mobile device according to claim 3, wherein the number of rays defining the ray projection is nray=4, the single frame resolution is 2×2, the number of pixels sampled by each ray is npixel=1, the number of frames of multi-frame three-dimensional file frames is nframe=2, the number of occupied bytes of each pixel is nBitCount, the memory block start address is nrddraddrinit, the memory address of the nth pixel of the mth frame in the memory is nrddraddr=nrddraddrinit+ (npixel+nray+m) ×nbitcount=nrddradrint+ (4n+m) ×nbitcount, wherein 0< m < nFrame,0< n <4.

5. A mobile device multi-frame three-dimensional image processing system, comprising:

and a third module: the method comprises the steps of being configured to call an arm register to load second data, carrying out rasterization superposition on the second data based on a ray projection method by using a Neon mathematical instruction set, and then obtaining final data and storing the final data into a memory;

loading the first data into an ARM Neon double-word vector register group;