CN115937150A - Point Cloud Quality Calculation Method Based on Point Structured Information Network - Google Patents
Point Cloud Quality Calculation Method Based on Point Structured Information Network Download PDFInfo
- Publication number
- CN115937150A CN115937150A CN202211589728.3A CN202211589728A CN115937150A CN 115937150 A CN115937150 A CN 115937150A CN 202211589728 A CN202211589728 A CN 202211589728A CN 115937150 A CN115937150 A CN 115937150A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- point
- layer
- structured information
- feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a point cloud quality calculation method based on a point structured information network, which is characterized in that the position vector characteristic, the distance characteristic, the brightness characteristic and the brightness difference characteristic of a point cloud block are jointly input into a point structured information network model to extract the structured information characteristic of the point cloud block; inputting the point structured information characteristics into a distortion perception stream network to obtain distortion classification characteristics; and inputting the structural information characteristics into a basic quality perception flow network to obtain the basic quality characteristics of the point cloud blocks. Fusing the basic quality characteristics and the distortion classification characteristics of the point cloud blocks and inputting the fused basic quality characteristics and the distortion classification characteristics into two third full-connection layers to obtain a predicted quality score; and carrying out average calculation on the predicted quality scores of a plurality of point cloud blocks belonging to the same integral point cloud to obtain the final score of the integral point cloud. The influence of information such as point cloud brightness, distance, relative position and the like on the point cloud quality is considered, and the point cloud subjective quality is comprehensively evaluated by introducing brightness difference characteristics and structural information characteristics.
Description
Technical Field
The invention relates to a point cloud quality calculation method based on a point structured information network, and belongs to the technical field of 3D point cloud non-reference quality evaluation.
Background
A point cloud is defined as a set of three-dimensional points, where each point is represented as a three-dimensional coordinate and a particular attribute (e.g., color). With the development of three-dimensional information capturing technology, point clouds are widely applied to applications such as virtual reality, immersive telepresence, moving maps and three-dimensional information printing. One typical use of point clouds is to represent holographic images of humans in virtual reality and immersive presence. However, in order to represent visual information realistically, a model may be composed of millions or even hundreds of millions of points, and in the process of transmission, a lossy compression scheme is usually adopted, which can save much transmission resources and increase the transmission rate compared with a lossless compression scheme, but has a negative effect of generating compressed perceptual distortion. In addition, interference may also occur in the acquisition and transmission processes, so that down-sampling perception distortion, gaussian filtering distortion and the like are generated, and the perception of human eyes is reduced. In order to better manage and control the subjective quality of the point cloud, it is important to provide high-performance color point cloud quality evaluation conforming to the perception of human eyes.
To quantify this mechanism of visual perception, people often conduct research from the perspective of both subjective and objective quality assessment. Subjective quality assessment relies on subjective scoring by the person to provide a true visual perception score for different levels of distortion, and although these methods are accurate, they are expensive in both time and labor costs. The objective quality index refers to the visual quality of the point cloud evaluated by using a model, and the existing objective quality evaluation models can be roughly divided into three types, namely full-reference point cloud quality evaluation, half-reference point cloud quality evaluation and no-reference point cloud quality evaluation. However, in most scenes, it is difficult to obtain the information of the original point cloud, and the amount of data required in the storage and transmission processes is too large, which makes the fully-referenced point cloud quality evaluation method difficult to be applied in actual reality, so the non-referenced point cloud quality evaluation method is becoming the focus of research of people. The existing quality evaluation of the reference-free point cloud can be mainly classified into two directions of measurement based on manual features and measurement based on deep learning. The manual feature-based method comprises the steps of projecting a 3D point cloud to a geometric and color feature domain, extracting quality perception features by utilizing 3D natural scene statistics and entropy, and returning the features to a visual quality score (3D-NSS) by utilizing a support vector machine. However, the handcrafted features are highly dependent on domain knowledge, and thus the performance of the feature representation is often limited. The deep learning-based method comprises the steps of rotating three specific tracks around a point cloud through a camera to obtain three video sequences, and using ResNet3D as a feature extraction model to learn the correlation (VS-ResNET) between a captured video and a corresponding subjective quality score; extracting hierarchical features from the 3D point cloud, considering geometric and texture information, adopting sparse tensor expression, and inputting tensor into CNN to predict and obtain a quality score (ResSCNN); deducing point cloud quality (IT-PCQA) by using unsupervised adversarial adaptation by using a natural image as a source domain and a point cloud as a target domain; and predicting a final score (PQA-NET) through a multi-view joint feature extraction and fusion module, a distortion type identification module and a quality vector prediction module. However, the above-mentioned method based on deep learning projects the 3D point cloud into an image or video because the direct application of convolution operation is hindered by the unstructured nature of the point cloud. These methods all consider only color information and ignore geometric information that is highly relevant to human perception.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a point cloud quality calculation method based on a point structured information network.
In order to achieve the above object, the present invention provides a point cloud quality calculation method based on a point structured information network, comprising:
the method comprises the steps of jointly inputting position vector features of point cloud blocks, distance features of the point cloud blocks, brightness features of the point cloud blocks and brightness difference features of the point cloud blocks, which are acquired in advance, into a point structured information network model, and extracting structured information features of the point cloud blocks;
inputting the point structured information characteristics into a distortion perception flow network after iterative training is completed to obtain distortion classification characteristics;
and inputting the structural information characteristics into a basic quality perception flow network to obtain the basic quality characteristics of the point cloud blocks.
Preferentially, fusing the basic quality characteristics and the distortion classification characteristics of the point cloud blocks and inputting the fused basic quality characteristics and the distortion classification characteristics into two third full-connection layers to obtain a predicted quality score;
and carrying out average calculation on the predicted mass scores of a plurality of point cloud blocks belonging to the same integral point cloud to obtain the final score of the integral point cloud.
Preferentially, the position vector feature, the distance feature, the brightness feature and the brightness difference feature which are acquired in advance are jointly input into the point structured information network model, the structured information feature of the point cloud block is extracted, and the method is realized through the following steps:
the point structured information network model comprises a first convolution layer, a second convolution layer, a third convolution layer and a fourth convolution layer, wherein the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer are connected in sequence;
inputting the position vector features into the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer for processing to obtain structured feature weights;
and carrying out feature weighting on the position vector feature, the distance feature, the brightness feature and the brightness difference feature and the structural feature weight to obtain the structural information feature of the point cloud block.
Preferentially, the point structured information features are input into a distortion perception stream network after iterative training is finished, so that distortion classification features are obtained, and the method is realized through the following steps:
the distortion perception flow network before iterative training comprises a fifth convolution layer, a sixth convolution layer, a first maximum pooling layer, a first global average pooling layer, a first full-connection layer and a linear regression layer, wherein the fifth convolution layer, the sixth convolution layer, the first maximum pooling layer, the first global average pooling layer, the first full-connection layer and the linear regression layer are sequentially connected;
freezing the whole distortion perception flow network after the iterative training is finished, and removing a linear regression layer;
and inputting the point structured information features into a distortion perception flow network which is subjected to iterative training and is free of a linear regression layer, and obtaining distortion classification features.
Preferably, the basic quality-aware flow network includes a plurality of seventh convolutional layers, a second maximum pooling layer, a second global average pooling layer, and a second fully-connected layer, and the plurality of seventh convolutional layers, the second maximum pooling layer, the second global average pooling layer, and the second fully-connected layer are connected in sequence.
Preferentially, the position vector feature, the distance feature, the brightness feature and the brightness difference feature are acquired in advance, and the method is realized by the following steps: sampling the original point cloud according to an FPS (field programmable gate array) farthest point sampling algorithm principle to obtain sampling points;
selecting 1024 nearest distance points of each sampling point to form a point cloud block through a KNN nearest neighbor algorithm;
calculating to obtain the position vector characteristic and the distance characteristic of each point cloud block;
and calculating to obtain the brightness characteristic and the brightness difference characteristic of each point cloud block.
Preferentially, the position vector characteristic and the distance characteristic of each cloud block are obtained through calculation, and the method is realized through the following steps:
calculating the position vector characteristics (delta x) of each sampling point in the point cloud block j ,Δy j ,Δz j }：
{Δx j ,Δy j ,Δz j }＝{x j -x 0 ,y j -y 0 ,z j -z 0 }，
In the formula, p j ＝{x j ,y j ,z j Denotes the three-dimensional coordinates of each sample point, j =1,2, \ 8230;, K, p 0 ＝{x 0 ,y 0 ,z 0 }，p 0 Three-dimensional coordinates of the centroid points;
calculating the distance characteristic of each sampling point in the point cloud block:
preferentially, the luminance characteristic and the luminance difference characteristic of the point cloud block are obtained through calculation, and the method is realized through the following steps:
calculating to obtain the brightness characteristic l of each sampling point in the point cloud block j ：
l j ＝r j ×0.229+g j ×0.587+b j ×0.114，
In the formula, c j ＝{r j ,g j ,b j Denotes a three-dimensional coordinate p j Color of the sample point of (1), r j RGB value, g, representing red j RGB value representing green, b j RGB values representing blue;
calculating the brightness difference characteristic delta l of each sampling point in the point cloud block j ：
Δl j ＝l 0 -l j In the formula I 0 Representing the luminance value of the centroid point.
The invention achieves the following beneficial effects:
k nearest neighbor points of each sampling point are obtained through a KNN nearest neighbor algorithm to form a point cloud block, and four characteristics of a position vector characteristic, a distance characteristic, a brightness characteristic and a brightness difference characteristic are calculated according to the nearest neighbor points;
the method comprises the steps of jointly inputting position vector characteristics, distance characteristics, brightness characteristics and brightness difference characteristics into a point structured information network model, mapping position vectors of adjacent points to weights, and then carrying out matrix multiplication on the weights and the four characteristics to obtain point cloud structured information;
the extracted structural information features are input into a distortion perception flow network, and distortion classification features are obtained through pre-training;
the method comprises the steps of inputting structural information characteristics into a basic quality perception flow network to obtain basic quality characteristics of point cloud blocks;
fusing basic quality characteristics and distortion classification characteristics and inputting the fused basic quality characteristics and distortion classification characteristics into two third full-connection layers to obtain a predicted quality score;
the method combines the multidimensional characteristics and the point structured information network model, and introduces the position vector characteristics, the distance characteristics, the brightness characteristics and the brightness difference characteristics into a double-current quality evaluation network;
the influence of information such as point cloud brightness, distance, relative position and the like on the point cloud quality is considered, and the point cloud subjective quality is comprehensively evaluated by introducing brightness difference characteristics and structural information characteristics; in the aspect of model construction, a double-current branch network is constructed, a pre-trained distortion classification model is used as a supplement for a quality evaluation model, the point cloud quality score is more accurately regressed, and the method has important significance for improving the accuracy of point cloud quality evaluation.
Drawings
FIG. 1 is an architecture diagram of a point structured information network model of the present invention;
fig. 2 is a framework diagram of the present invention.
Detailed Description
The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.
Example one
The invention provides a point cloud quality calculation method based on a point structured information network, which comprises the following steps:
in the model application stage, the position vector characteristics of the point cloud blocks, the distance characteristics of the point cloud blocks, the brightness characteristics of the point cloud blocks and the brightness difference characteristics of the point cloud blocks which are acquired in advance are jointly input into a point structured information network model, and the structured information characteristics of the point cloud blocks are extracted;
inputting the point structured information characteristics into a distortion perception flow network after iterative training is completed to obtain distortion classification characteristics;
and inputting the structural information characteristics into a basic quality perception flow network to obtain the basic quality characteristics of the point cloud blocks.
Further, in the application stage of the model in this embodiment, the basic quality features and the distortion classification features of the point cloud blocks are fused and input into the two third full-connected layers to obtain the predicted quality scores;
and carrying out average calculation on the predicted quality scores of a plurality of point cloud blocks belonging to the same integral point cloud to obtain the final score of the integral point cloud.
Further, in this embodiment, the position vector feature, the distance feature, the luminance feature, and the luminance difference feature that are obtained in advance are jointly input into the point structured information network model, and the structured information feature of the point cloud block is extracted, which is implemented by the following steps: the point structured information network model comprises a first convolution layer, a second convolution layer, a third convolution layer and a fourth convolution layer, wherein the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer are connected in sequence;
inputting the position vector features into a first convolution layer, a second convolution layer, a third convolution layer and a fourth convolution layer for processing to obtain structured feature weights;
and carrying out feature weighting on the position vector feature, the distance feature, the brightness feature and the brightness difference feature and the structural feature weight to obtain the structural information feature of the point cloud block.
Further, in this embodiment, the point structured information features are input into the distortion perception stream network after the iterative training is completed, so as to obtain the distortion classification features, and the method is implemented through the following steps:
the distortion perception flow network before iterative training comprises a fifth convolution layer, a sixth convolution layer, a first maximum pooling layer, a first global average pooling layer, a first full-connection layer and a linear regression layer, wherein the fifth convolution layer, the sixth convolution layer, the first maximum pooling layer, the first global average pooling layer, the first full-connection layer and the linear regression layer are sequentially connected;
freezing the whole distortion perception flow network after the iterative training is completed, and removing a linear regression layer;
and inputting the point structured information features into a distortion perception flow network which is subjected to iterative training and is free of a linear regression layer, and obtaining distortion classification features.
Further, in this embodiment, the basic quality-aware flow network includes a plurality of seventh convolutional layers, a second maximum pooling layer, a second global average pooling layer, and a second fully-connected layer, where the plurality of seventh convolutional layers, the second maximum pooling layer, the second global average pooling layer, and the second fully-connected layer are connected in sequence.
Further, in this embodiment, the position vector feature, the distance feature, the luminance feature, and the luminance difference feature are obtained in advance, and the method includes the following steps:
sampling the original point cloud according to an FPS (field programmable gate array) farthest point sampling algorithm principle to obtain sampling points;
selecting 1024 nearest distance points of each sampling point to form a point cloud block through a KNN nearest neighbor algorithm;
calculating to obtain the position vector characteristic and the distance characteristic of each point cloud block;
and calculating to obtain the brightness characteristic and the brightness difference characteristic of each point cloud block.
Further, in this embodiment, the position vector feature and the distance feature of each cloud block are obtained through calculation, and the method is implemented through the following steps: calculating the position vector characteristics (delta x) of each sampling point in the point cloud block j ,Δy j ,Δz j }：
{Δx j ,Δy j ,Δz j }＝{x j -x 0 ,y j -y 0 ,z j -z 0 }，
In the formula, p j ＝{x j ,y j ,z j Denotes the three-dimensional coordinates of each sample point, j =1,2, \ 8230;, K, p 0 ＝{x 0 ,y 0 ,z 0 }，p 0 Three-dimensional coordinates of the centroid points;
calculating the distance characteristic of each sampling point in the point cloud block:
further, the luminance characteristic and the luminance difference characteristic of the point cloud block are obtained through calculation in this embodiment, and the method includes the following steps:
calculating to obtain the brightness characteristic l of each sampling point in the point cloud block j ：
l j ＝r j ×0.229+g j ×0.587+b j ×0.114，
In the formula, c j ＝{r j ,g j ,b j Denotes a three-dimensional coordinate p j Color of the sample point of (1), r j RGB value, g, representing red j RGB value representing green, b j RGB values representing blue;
calculating the brightness difference characteristic delta l of each sampling point in the point cloud block j ：
Δl j ＝l 0 -l j ，
In the formula I 0 Representing the luminance value of the centroid point.
Example two
The technical scheme of the invention comprises the following parts:
1) Local sampling composition point cloud block
On one hand, considering that the number of points in a point cloud sample is huge (usually, tens of thousands to millions of points exist), the existing computing resources (such as memory and video memory) are difficult to analyze and process the whole point cloud at the same time; on the other hand, the existing point cloud quality evaluation database has a small number of samples, and cannot meet the requirements of a machine learning-based method on large scale, diversity and the like of training samples. Therefore, the invention uses the idea of local sampling in image quality evaluation, namely the idea of local representing the whole situation. Specifically, N points of each distorted point cloud are selected by using a farthest point sampling method, and K nearest neighbor points of each sampling point are selected to form a point cloud block. In the model training stage, expressing the mass fraction of the whole point cloud as the real fraction of the cloud blocks of the training points; when the model is applied to quality evaluation, the final quality score of the whole point cloud is the average value of the scores of all local point cloud blocks.
The sampling method of the local point cloud block comprises the following steps: (1) Firstly, selecting sampling points by utilizing a farthest point sampling method, and continuously and iteratively selecting farthest points from an existing sampling point set so as to cover the whole point cloud as much as possible; (2) And then, constructing a local point cloud block by using a K Nearest Neighbor (KNN) search method, calculating the distances between the sampling points and adjacent points, sequencing the distance calculation from small to large, and selecting K samples closest to the sampling points to form a point cloud block.
2) Point structured information extraction module (PSI)
The prior image quality evaluation method proves that the human visual system is highly suitable for extracting structural information such as gradient, contrast and the like from an observation field. For point clouds, however, the structural information refers to changes in the local 3D block, including changes in color and geometry. However, irregularities in the point cloud pose challenges to the efficient extraction of structural information in three-dimensional space. In the present invention, a point structured information extraction module (PSI) module can simultaneously extract geometric and color structure information from the point cloud by fitting the feature weights as a nonlinear function to the three-dimensional relative coordinates. The framework of the PSI module is shown in figure 2.
Feature preprocessing
As shown in fig. 2, a block of samples with an adjacency point is input to the PSI module. In order to better extract the structure information of the irregular points, the input block needs to be preprocessed. Specifically, we denote the patch as P = { P 0 ,p 1 ,p 2 ,…,p K In which p 0 ＝{x 0 ,y 0 ,z 0 Is the sampling point (or centroid point), p j ＝{x j ,y j ,z j J =1,2, \8230;, K is p 0 And (5) adjacent to the neighbor point. Vector c j ＝{r j ,g j ,b j Denotes p j The color of (c).
Since the structural information refers to the change of the color and geometric information of the local three-dimensional block, in the present study, it is calculated as the change of the geometry and color from the neighboring point to the central point, such as the position vector, the distance, and the luminance difference.
Calculate each point p j Position vector feature of (1), p j ＝{x j ,y j ,z j J =1,2, \ 8230;, K, the corresponding position vector feature is calculated as the centroid point p 0 To p j Vector between:
{Δx j ,Δy j ,Δz j }＝{x j -x 0 ,y j -y 0 ,z j -z 0 }(1)
calculating the corresponding distance features:
wherein d is j Represents p 0 To p j The distance of (c).
Since the human visual system is sensitive to relative changes in brightness, for each p j ＝{x j ,y j ,z j J =1,2, \ 8230;, K, the color of which is converted into luminance values, the luminance characteristic of each dot cloud block is calculated:
l j ＝r j ×0.229+g j ×0.587+b j ×0.114,(3)
wherein l j Represents p j The luminance difference characteristic of (1). p is a radical of j And p 0 The luminance difference of (a) is calculated as follows,
Δl j ＝l 0 -l j . (4)
thus, for each adjacency point p j ＝{x j ,y j ,z j K, we can get a preprocessed input feature, i.e., i =1,2, \ 8230;, K j ＝{Δx j ,Δy j ,Δz j ,d j ,l j ,Δl j } T . That is to say, the position of the nozzle is, the input of the PSI module is I = { I = { I = } 1 ,i 2 ,…,i K } T I.e. by
Location vector based weighting
The neighboring points have different relative positions with respect to the same center point, and thus the local change between them has different weights for the calculation of the structural information of the three-dimensional block. CNNs show a strong ability in learning representations of image features, where the convolution weights are treated as a discrete function of relative position. Whereas on three-dimensional point clouds, the convolution weights are treated as a non-linear continuous function of the position vector, i.e.
W＝f(Δx,Δy,Δz), (5)
Wherein Δ x = { Δ x 1 ,Δx 2 ,…,Δx K } T ，Δy＝{Δy 1 ,Δy 2 ,…,Δy K } T And Δ z = { Δ z 1 ,Δz 2 ,…,Δz K } T As shown in fig. 2, the nonlinear function f (-) is implemented as 4 convolutional layers. Then, the structural information characteristic of the three-dimensional block is calculated by matrix multiplication with the non-quantization weight W and the input I, i.e., the non-quantization weight W is calculated.
Wherein
The structural information feature, also called ψ feature, represents a three-dimensional point cloud block.
3) Point structured information network architecture
As shown in figure 1, in the point structured information network, the PSI module outputs a structural information characteristic F ψ Is fed into two computation streams, DPS and EQPS.
First, DPS is pre-training based on a point cloud distortion classification task. Specifically, F ψ Is reshape of
After the dimension, it is fed into the Conv1 module with 2 convolutional layers and one maximum aggregation layer.
To prevent overfitting, the output of the Conv1 module is passed through a Global Average Pooling (GAP) layer of size 2 × 2 and then mapped to distortion-related features through 2 fully-connected (FC) layers, denoted as distortion-related features
Second, quality regression requires more complex features than distortion classification, so EQPS uses two more convolution modules than DPS, i.e., conv2 and Conv3. Specifically, the Conv2 and Conv3 modules are each composed of 3 convolutional layers with 256 and 512 cores in each convolutional layer, respectively, and one max pooling layer. This is because more convolutional layers and nuclei can enlarge the receptive field and extract more abundant characteristic information.
In general, the structural information features from the PSI module sequentially pass through the Conv1, conv2 and Conv3 modules, a GAP layer with the size of 1 × 1 and 2 FC layers to obtain a feature, which is expressed as
Finally, these two features, i.e. f dps And f eqps The fusion is performed by dot multiplication. Then, the predicted quality scores of the three-dimensional patches were regressed with 2 FC layers according to the fused features.
3) The DPS is pre-trained by a distortion classification task, so the point structured information network provided by the invention is trained in two stages. The cross entropy loss between the predicted distortion type and the true distortion type is used to pre-train the DPS. Note that the true distortion type of each three-dimensional point cloud block is the same as the distortion type of the entire point cloud to which it belongs. The entire network is then trained by freezing the weights of the DPS.
The Mean Square Error (MSE) loss between the predicted quality score and the actual quality score is used to supervise the training of the entire network, i.e. the MSE loss is used.
Where M is the size of the minimum batch. Symbol Q i And
respectively expressed as a prediction score and a true score.
Likewise, the true mass score of each patch is equal to the mass score of the entire point cloud.
4) Application of point structured information network to point cloud quality assessment
And taking the scores of the prediction scores of a plurality of point cloud blocks belonging to the same integral point cloud, and carrying out average calculation to obtain the score serving as the final score of the integral point cloud. The concrete formula is as follows:
the iterative training phase of the invention comprises the following steps:
step 1, during iterative training, sampling an original point cloud according to an FPS (field programmable gate array) farthest point sampling algorithm principle to obtain sampling points, wherein the sampling points form a training data set to obtain the training data set, and historical sampling points comprise historical three-dimensional coordinates and historical specific attributes (red, green and blue colors);
selecting 1024 nearest distance points of each sampling point to form a point cloud block through a KNN nearest neighbor algorithm, and executing the step 2; step 2: calculating to obtain the historical position vector characteristic and the historical distance characteristic of each point cloud block according to the formula (1) and the formula (2), and executing the step 3;
and 3, step 3: and (5) calculating to obtain the historical brightness characteristic and the historical brightness difference characteristic of each point cloud block according to the formula (3) and the formula (4), and executing the step 4.
And 4, step 4: jointly inputting the position vector feature, the distance feature, the brightness feature and the brightness difference feature which are obtained in advance into a point structured information network model (PSI Module), extracting the structured information feature of the historical point cloud block, and executing the step 5;
the point structured information network model PSI Module comprises a first convolution layer, a second convolution layer, a third convolution layer and a fourth convolution layer, wherein the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer are sequentially connected, the first convolution layer, the second convolution layer and the third convolution layer are convolution layers with 3 convolution kernels and 8 output channels, and the fourth convolution layer is a convolution layer with 1 convolution kernel of 3 and 16 output channels;
specifically, the historical position vector features (K × 3 dimensions) are input into the point structured information network model, and are processed by a first convolutional layer, a second convolutional layer, a third convolutional layer and a fourth convolutional layer to obtain historical structured feature weights (K × 16 dimensions). Wherein 4 convolutional layers are regarded as nonlinear functions, and the position vector features are fitted into feature weights;
and carrying out feature weighting on the historical position vector feature, the historical distance feature, the historical brightness feature and the historical brightness difference feature (K multiplied by 6 dimension) and the historical structural feature weight, namely carrying out matrix multiplication on the historical position vector feature, the historical distance feature, the historical brightness feature and the historical brightness difference feature to obtain the structural information feature of the historical point cloud block. The concrete formula is as follows: w = f (Δ x, Δ y, Δ z), where f is a nonlinear function of the four convolutional layers of the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer.
And 5, step 5: inputting the structural information characteristics of the historical points into a distortion perception flow network, and pre-training the distortion perception flow network by using a cross entropy loss formula to obtain a trained distortion perception flow network;
freezing the trained whole distortion perception flow network, removing a linear regression layer to obtain distortion classification characteristics, and executing the step 6;
the distortion perception flow network comprises a fifth convolution layer, a sixth convolution layer, a first maximum pooling layer, a first global average pooling layer, a first full-connection layer and a linear regression layer, wherein the fifth convolution layer, the sixth convolution layer, the first maximum pooling layer, the first global average pooling layer, the first full-connection layer and the linear regression layer are sequentially connected; the fifth convolutional layer and the sixth convolutional layer are convolutional layers with 2 convolutional kernels, 3 output channels and 128 dimensions, and the first maximum pooling layer step size is 2 and the sliding window size is 2.
And (4) regressing the historical point structured information characteristics into distortion types through a linear regression layer in the distortion perception flow network. And 6, step 6: inputting the historical structured information characteristics into a basic quality perception flow network to obtain the basic quality characteristics of the historical point cloud blocks;
the basic quality perception flow network comprises a plurality of layers of a seventh convolutional layer, a second maximum pooling layer, a second global average pooling layer and a second full-connection layer, wherein the plurality of layers of the seventh convolutional layer, the second maximum pooling layer, the second global average pooling layer and the second full-connection layer are connected in sequence; and 7, step 7: fusing the basic quality characteristics and the distortion classification characteristics and inputting the fused basic quality characteristics and the distortion classification characteristics into two third full-connection layers to obtain a predicted quality score;
and 8, step 8: iterating the training point structured network by minimizing the sum of the mean square errors of the predicted mass fraction and the real mass fraction; the input of the point structured network is vector characteristics, distance characteristics, brightness characteristics and brightness difference characteristics of points in K point clouds of M batches, and the output of the point structured network is the quality scores of the M batches.
Inputting historical position vector characteristics, historical distance characteristics, historical brightness characteristics and historical brightness difference characteristics into a point structured information network model, a distortion perception flow network and a basic quality perception flow network for processing, and iteratively training the point structured information network model, the distortion perception flow network and the basic quality perception flow network by using a mean square error, wherein the expression of the mean square error is as follows:
where M is the total number of historical samples in the training data set,
iteratively updating the distortion perception flow network before training by using a cross entropy loss function, wherein the expression of the cross entropy loss function is as follows:
wherein LOSS is a LOSS value, M is the number of all sampling points in the training data set, class is a distortion classification number, y is a real label of the sampling points,
and outputting the prediction label of the sampling point for the distortion perception stream network.
In fig. 2, PSI Module is a point structured information network model, and reshape is an operation for transforming an output feature into a shape suitable for convolutional neural network learning, and transforming a feature of K × 96 into a feature of √ K × × √ K × 96 dimensions. The meaning of Pool-2 is that the step length is 2, the sliding window is the largest pooling layer of 2, and the method is used for reducing dimension, reducing parameter quantity and removing redundant information. GAP is a global mean pooling layer that acts to reduce dimensions and prevent overfitting. The meaning of Flatten is to Flatten the input tensor so that it can be processed by the fully connected layer. FC is a fully connected layer whose function is to integrate the features extracted previously. The meaning of the symbol is a dot product, which means that the features of the two branches are fused. The Quality Score is the predicted Quality Score of the network output.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (8)
1. The point cloud quality calculation method based on the point structured information network is characterized by comprising the following steps:
the method comprises the steps of jointly inputting position vector features of point cloud blocks, distance features of the point cloud blocks, brightness features of the point cloud blocks and brightness difference features of the point cloud blocks, which are acquired in advance, into a point structured information network model, and extracting structured information features of the point cloud blocks;
inputting the point structured information characteristics into a distortion perception flow network after iterative training is completed to obtain distortion classification characteristics;
and inputting the structural information characteristics into a basic quality perception flow network to obtain the basic quality characteristics of the point cloud blocks.
2. The point cloud quality calculation method based on the point structured information network according to claim 1,
fusing the basic quality characteristics and the distortion classification characteristics of the point cloud blocks and inputting the fused basic quality characteristics and the distortion classification characteristics into two third full-connection layers to obtain predicted quality scores;
and carrying out average calculation on the predicted quality scores of a plurality of point cloud blocks belonging to the same integral point cloud to obtain the final score of the integral point cloud.
3. The point cloud quality calculation method based on the point structured information network according to claim 1,
the method comprises the following steps of jointly inputting position vector features, distance features, brightness features and brightness difference features which are obtained in advance into a point structured information network model, extracting the structured information features of point cloud blocks, and realizing the following steps:
the point structured information network model comprises a first convolution layer, a second convolution layer, a third convolution layer and a fourth convolution layer, wherein the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer are connected in sequence;
inputting the position vector features into a first convolution layer, a second convolution layer, a third convolution layer and a fourth convolution layer for processing to obtain structured feature weights;
and carrying out feature weighting on the position vector feature, the distance feature, the brightness feature and the brightness difference feature and the structural feature weight to obtain the structural information feature of the point cloud block.
4. The point cloud quality calculation method based on the point structured information network according to claim 1,
inputting the point structured information characteristics into a distortion perception flow network finished by iterative training to obtain distortion classification characteristics, and realizing the method by the following steps:
the distortion perception flow network before iterative training comprises a fifth convolution layer, a sixth convolution layer, a first maximum pooling layer, a first global average pooling layer, a first full-connection layer and a linear regression layer, wherein the fifth convolution layer, the sixth convolution layer, the first maximum pooling layer, the first global average pooling layer, the first full-connection layer and the linear regression layer are sequentially connected;
freezing the whole distortion perception flow network after the iterative training is completed, and removing a linear regression layer;
and inputting the point structured information features into a distortion perception flow network which is subjected to iterative training and is free of a linear regression layer, and obtaining distortion classification features.
5. The point cloud quality calculation method based on the point structured information network according to claim 4,
the basic quality perception flow network comprises a plurality of layers of a seventh convolutional layer, a second maximum pooling layer, a second global average pooling layer and a second full-connection layer, wherein the plurality of layers of the seventh convolutional layer, the second maximum pooling layer, the second global average pooling layer and the second full-connection layer are connected in sequence.
6. The point cloud quality calculation method based on the point structured information network according to claim 1,
the method comprises the following steps of obtaining a position vector feature, a distance feature, a brightness feature and a brightness difference feature in advance, and realizing the following steps:
sampling the original point cloud according to an FPS (field programmable gate array) farthest point sampling algorithm principle to obtain sampling points;
selecting 1024 nearest distance points of each sampling point to form a point cloud block through a KNN nearest neighbor algorithm;
calculating to obtain the position vector characteristic and the distance characteristic of each point cloud block;
and calculating to obtain the brightness characteristic and the brightness difference characteristic of each point cloud block.
7. The point cloud quality calculation method based on the point structured information network according to claim 6,
calculating to obtain the position vector characteristic and the distance characteristic of each cloud block, and realizing the method by the following steps:
calculating the position vector characteristics (delta x) of each sampling point in the point cloud block j ,Δy j ,Δz j }：
{Δx j ,Δy j ,Δz j }＝{x j -x 0 ,y j -y 0 ,z j -z 0 }，
In the formula, p j ＝{x j ,y j ,z j Denotes the three-dimensional coordinates of each sample point, j =1,2, \ 8230;, K, p 0 ＝{x 0 ,y 0 ,z 0 }，p 0 Three-dimensional coordinates of the centroid point;
calculating the distance characteristic of each sampling point in the point cloud block:
8. the point cloud quality calculation method based on the point structured information network according to claim 6,
calculating to obtain the brightness characteristic and the brightness difference characteristic of the point cloud block, and realizing the following steps:
calculating to obtain the brightness characteristic l of each sampling point in the point cloud block j ：
l j ＝r j ×0.229+g j ×0.587+b j ×0.114，
In the formula, c j ＝{r j ,g j ,b j Denotes a three-dimensional coordinate p j Color of the sample point of (1), r j RGB value, g, representing red j RGB value representing green, b j RGB values representing blue;
calculating the brightness difference characteristic delta l of each sampling point in the point cloud block j ：
Δl j ＝l 0 -l j ，
In the formula I 0 Representing the luminance value of the centroid point.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211589728.3A CN115937150A (en) | 2022-12-12 | 2022-12-12 | Point Cloud Quality Calculation Method Based on Point Structured Information Network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211589728.3A CN115937150A (en) | 2022-12-12 | 2022-12-12 | Point Cloud Quality Calculation Method Based on Point Structured Information Network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115937150A true CN115937150A (en) | 2023-04-07 |
Family
ID=86698878
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211589728.3A Pending CN115937150A (en) | 2022-12-12 | 2022-12-12 | Point Cloud Quality Calculation Method Based on Point Structured Information Network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115937150A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116137059A (en) * | 2023-04-17 | 2023-05-19 | 宁波大学科学技术学院 | Three-dimensional point cloud quality evaluation method based on multi-level feature extraction network model |
-
2022
- 2022-12-12 CN CN202211589728.3A patent/CN115937150A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116137059A (en) * | 2023-04-17 | 2023-05-19 | 宁波大学科学技术学院 | Three-dimensional point cloud quality evaluation method based on multi-level feature extraction network model |
| CN116137059B (en) * | 2023-04-17 | 2024-04-26 | 宁波大学科学技术学院 | Three-dimensional point cloud quality evaluation method based on multi-level feature extraction network model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109816725B (en) | Monocular camera object pose estimation method and device based on deep learning | |
| CN108510485B (en) | Non-reference image quality evaluation method based on convolutional neural network | |
| WO2022252272A1 (en) | Transfer learning-based method for improved vgg16 network pig identity recognition | |
| CN111080511B (en) | End-to-end face exchange method for high-resolution multi-feature extraction | |
| US11182644B2 (en) | Method and apparatus for pose planar constraining on the basis of planar feature extraction | |
| CN108416266B (en) | Method for rapidly identifying video behaviors by extracting moving object through optical flow | |
| CN113065546B (en) | Target pose estimation method and system based on attention mechanism and Hough voting | |
| CN109977757B (en) | Multi-modal head posture estimation method based on mixed depth regression network | |
| CN114255238A (en) | Three-dimensional point cloud scene segmentation method and system fusing image features | |
| CN112489164B (en) | Image coloring method based on improved depth separable convolutional neural network | |
| CN111507357B (en) | Defect detection semantic segmentation model modeling method, device, medium and equipment | |
| CN108121931A (en) | two-dimensional code data processing method, device and mobile terminal | |
| CN110910339B (en) | Logo defect detection method and device | |
| CN110751195B (en) | Fine-grained image classification method based on improved YOLOv3 | |
| CN110969171A (en) | Image classification model, method and application based on improved convolutional neural network | |
| CN115937626B (en) | Automatic generation method of paravirtual data set based on instance segmentation | |
| CN114972216A (en) | Construction method and application of texture surface defect detection model | |
| CN115937150A (en) | Point Cloud Quality Calculation Method Based on Point Structured Information Network | |
| CN116188509A (en) | High-efficiency three-dimensional image segmentation method | |
| CN116342536A (en) | Aluminum strip surface defect detection method, system and equipment based on lightweight model | |
| CN113011506A (en) | Texture image classification method based on depth re-fractal spectrum network | |
| CN116993760A (en) | Gesture segmentation method, system, device and medium based on graph convolution and attention mechanism | |
| CN108765384B (en) | Significance detection method for joint manifold sequencing and improved convex hull | |
| CN114882545A (en) | Multi-angle face recognition method based on three-dimensional intelligent reconstruction | |
| CN110188706B (en) | Neural network training method and detection method based on character expression in video for generating confrontation network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |