CN113115077A - Code rate self-adaptive transmission method and system for static point cloud server - Google Patents
Code rate self-adaptive transmission method and system for static point cloud server Download PDFInfo
- Publication number
- CN113115077A CN113115077A CN202110269265.1A CN202110269265A CN113115077A CN 113115077 A CN113115077 A CN 113115077A CN 202110269265 A CN202110269265 A CN 202110269265A CN 113115077 A CN113115077 A CN 113115077A
- Authority
- CN
- China
- Prior art keywords
- code rate
- space
- point cloud
- slice
- static point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000003068 static effect Effects 0.000 title claims abstract description 79
- 230000005540 biological transmission Effects 0.000 title claims abstract description 44
- 238000000034 method Methods 0.000 title claims abstract description 42
- 230000003044 adaptive effect Effects 0.000 claims abstract description 27
- 238000005457 optimization Methods 0.000 claims abstract description 19
- 238000011426 transformation method Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 7
- 230000000007 visual effect Effects 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 6
- 230000004886 head movement Effects 0.000 claims description 5
- 230000001131 transforming effect Effects 0.000 claims description 3
- 238000004422 calculation algorithm Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000001303 quality assessment method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
- H04N21/23805—Controlling the feeding rate to the network, e.g. by controlling the video pump
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a code rate self-adaptive transmission method and a code rate self-adaptive transmission system for a static point cloud server, wherein the method comprises the following steps: the server divides the static point cloud into 3D space slices; the user side feeds back user viewing information and display screen resolution to the server side in real time, and downloads a code rate version complete set which can be distributed to each 3D space slice from the server side; the server side calculates the QoE contribution of each space slice according to the user viewing information and the resolution of the display screen; and establishing an optimization problem based on code rate adaptive allocation of the space slices by adopting a complete set consisting of the space slices, a code rate version complete set of the space slices, network resource limitation and QoE contribution of each space slice to obtain an optimal space slice code rate version subset. The system comprises: and the user side and the server side correspondingly realize the functions. By the method and the device, the bandwidth utilization rate of static point cloud transmission is improved, and better quality experience is provided for users.
Description
Technical Field
The invention relates to the technical field of video communication, in particular to a code rate self-adaptive transmission method and system for a static point cloud server.
Background
With the rapid development of 3D sensing and acquisition systems, the rise and popularization of a series of emerging technologies such as Virtual Reality (VR), Augmented Reality (AR), and the like, point clouds which can provide users with very immersive experience due to the support of 6 degrees of freedom are becoming the focus of scientific research gradually as a novel media data. When a user watches a point cloud scene, the watching direction can be changed at will, and the spatial position can be changed freely, so that point cloud transmission has great significance for enhancing the immersive experience of the user. However, the point cloud is often huge in data volume, and consumes bandwidth resources, which brings huge challenges to transmission in the current network environment.
In order to solve the challenge brought by point cloud transmission to network bandwidth, code rates are adaptively allocated according to the contribution of a point cloud object to QoE of a user, a larger code rate is allocated to a point cloud with a large QoE contribution, and a smaller code rate is allocated to a point cloud with a small QoE contribution, so that as high quality experience as possible is provided for the user under the limited bandwidth resources. The factors influencing the contribution of the point cloud QoE mainly include the following five points: (1) whether the point cloud is in the field of view of the user; (2) the distance between the point cloud and the user; (3) occlusion between point clouds; (4) code rate versions allocated to the point clouds; (5) the screen resolution is displayed. Specifically, for invisible point clouds (point clouds are not in the user field of view volume or are completely occluded), the QoE contribution is zero; for visible point clouds, the QoE contribution decreases with increasing distance of the point cloud to the user, and is limited by the display screen resolution. Furthermore, the higher the bitrate version of the point cloud, the higher the quality representation and thus the greater the QoE contribution.
The search of the prior art shows that an article entitled "forward 6DoF HTTP Adaptive Streaming Through Point Cloud Compression" is published by Hooft et al in an ACM Multimedia Conference of 2019, and a scene consisting of a plurality of Point clouds is transmitted by using a single Point Cloud as a basic unit for code rate allocation. And respectively providing four indexes to judge the priority of each point cloud according to whether the point cloud is in a view port range of a user and the distance between the point cloud and the user, and providing three code rate self-adaptive heuristic algorithms on the basis. However, the application scene related to the article does not include a single point cloud, the influence of shielding between the point clouds and the resolution of a display screen is not considered when the priority of the point clouds is judged, indexes have no universality, and different indexes are required to be selected according to different scenes.
It is found through search that an article entitled "Rate-utilization Optimized Streaming of volume Media for Augmented Reality" is published in the Journal of IEEE Journal on organizing and Selected Topics in Circuits and Systems of 2019 by Park et al, and the article is transmitted by using a 3D space slice as a basic unit for code Rate allocation, and is applicable to both a single point cloud and a scene composed of a plurality of point clouds. A code rate allocation mechanism based on a greedy algorithm is provided on the basis that the utility of each slice is defined according to whether the spatial slice is in a user visual field, the distance between the spatial slice and a user, the code rate version of the spatial slice and the resolution of a display screen. However, this article does not take into account occlusion between slices when defining slice utility and does not make quality assessments in terms of the user's vision.
In addition, the working code rate self-adaptive algorithms are all heuristic algorithms, and the theoretical guarantee for the performance of the algorithms is lacked.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a code rate self-adaptive transmission method and a code rate self-adaptive transmission system for a static point cloud server, so that the bandwidth utilization rate of static point cloud transmission is improved, and better quality experience is provided for users.
In order to solve the technical problems, the invention is realized by the following technical scheme:
the invention provides a code rate self-adaptive transmission method of a static point cloud server, which comprises the following steps:
s11: the server divides the static point cloud into a plurality of 3D space slices;
s12: feeding back user viewing information and display screen resolution to a server in real time, and downloading a code rate version complete set which can be allocated to each 3D space slice obtained by S11 division from the server;
s13: calculating the QoE contribution of each 3D space slice according to the user viewing information and the resolution of a display screen;
s14: and establishing an optimization problem of code rate self-adaptive allocation of a static point cloud server end based on the space slices by adopting a complete set consisting of the 3D space slices obtained by the division of S11, a complete set of code rate versions of the space slices in S12, QoE contribution and network resource limitation of each 3D space slice obtained in S13, and obtaining an optimal code rate version subset of the space slices by adopting a code rate allocation method.
Preferably, the dividing the static point cloud into 3D spatial slices in S11 is specifically: a point cloud encoding technique is used to divide the static point cloud into a preset number of 3D spatial slices.
Preferably, the user viewing information includes: the head movement trajectory of the user.
Preferably, in S13, the QoE contribution of each 3D spatial slice is calculated by a coordinate transformation method based on perspective projection.
Preferably, the coordinate transformation method of perspective projection includes:
according to the geometric relation between the 3D point cloud and the two-dimensional projection plane thereof, and according to the user viewport information, the distance between the user and the point cloud, and the shielding and display screen resolution between the 3D space slices, carrying out perspective projection transformation which is close to a human eye visual system and is suitable for the 3D space slices;
and according to whether the 3D space slice can be projected on a display screen or not, calculating the QoE contribution of the 3D space slice according to the relation between the number of voxels contained in the 3D space slice and the number of pixels corresponding to the number of pixels which can be seen by a user on the display screen.
Preferably, the QoE contribution calculation for the 3D spatial slice includes:
setting static point clouds to be divided into NT3D spatial slices, each spatial slice being encoded into q quality representations, corresponding to q code rate versions, the ith spatial slice T with quality represented as qiThe QoE contribution of (a) is expressed as:
Qi,q＝UPBi·Bi,q
wherein, Bi,qIs to mix TiCoding the number of bits required for expression as q, UPBiRepresents the utility of each bit, expressed as:
wherein N isv,iSlicing T into uncompressed spaceiNumber of voxels in the user field of view, Np,iIs to project T by perspectiveiMapping to the corresponding number of pixels on the display screen, Bi,maxTo normalize the parameters, represent TiThe coded representation is the number of bits required for the highest quality representation.
Preferably, the optimization problem of establishing the spatial slice-based static point cloud server-side code rate adaptive allocation in S14 is specifically: modeling an optimization problem of code rate self-adaptive distribution into a multi-choice knapsack problem by adopting a complete set formed by 3D space slices obtained by dividing the static point cloud, a complete set of code rate versions of the 3D space slices, network resource limitation and QoE contribution of each 3D space slice; further, the multi-choice knapsack problem is equivalently converted into a submodular function maximization problem under the knapsack constraint.
The invention also provides a code rate self-adaptive transmission system of the static point cloud server, which comprises the following steps:
the user side feeds back the user viewing information and the resolution of the display screen to the server side in real time, and downloads a code rate version complete set which can be distributed to each 3D space slice from the server side;
and the server end divides the static point cloud into a plurality of 3D space slices, calculates QoE contribution of each 3D space slice according to the user viewing information and the resolution of a display screen, establishes an optimization problem of code rate self-adaptive distribution of the static point cloud server end based on the space slices by adopting a full set formed by the 3D space slices obtained by dividing the static point cloud, a full set of code rate versions of the space slices, the QoE contribution of each 3D space slice and network resource limitation, and obtains an optimal space slice code rate version subset by adopting a code rate distribution method.
Preferably, the dividing, by the server side, the static point cloud into 3D spatial slices specifically includes: a point cloud encoding technique is used to divide the static point cloud into a preset number of 3D spatial slices.
Preferably, the user viewing information includes: the head movement trajectory of the user.
Preferably, the server calculates the QoE contribution of each 3D spatial slice by a coordinate transformation method based on perspective projection.
Preferably, the coordinate transformation method of perspective projection further includes: according to the geometric relation between the 3D point cloud and the two-dimensional projection plane thereof, and according to the user viewport information, the distance between the user and the point cloud, and the shielding and display screen resolution between the 3D space slices, carrying out perspective projection transformation which is close to a human eye visual system and is suitable for the 3D space slices;
and according to whether the 3D space slice can be projected on a display screen or not, calculating the QoE contribution of the 3D space slice according to the relation between the number of voxels contained in the 3D space slice and the number of pixels corresponding to the number of pixels which can be seen by a user on the display screen.
Preferably, the QoE contribution calculation for the 3D spatial slice includes:
setting static point clouds to be divided into NT3D spatial slices, each spatial slice being encoded into q quality representations, corresponding to q code rate versions, the ith spatial slice T with quality represented as qiThe QoE contribution of (a) is expressed as:
Qi,q＝UPBi·Bi,q
wherein, Bi,qIs to mix TiCoding the number of bits required for expression as q, UPBiRepresents the utility of each bit, expressed as:
wherein N isv,iSlicing T into uncompressed spaceiNumber of voxels in the user field of view, Np,iBy means of perspectiveProjection will TiMapping to the corresponding number of pixels on the display screen, Bi,maxTo normalize the parameters, represent TiThe coded representation is the number of bits required for the highest quality representation.
Compared with the prior art, the embodiment of the invention has at least one of the following advantages:
(1) according to the code rate self-adaptive transmission method and system for the static point cloud server, each space slice is independently coded into different quality representations, corresponding to different code rate versions, and each point cloud is cut into individual 3D space slices in space according to the characteristic that a user can only view a part of point cloud, so that the user requirements are met;
(2) according to the code rate self-adaptive transmission method and system for the static point cloud server, the code rate allocation optimization problem of a plurality of space slices is modeled into a multi-choice knapsack problem, so that the QoE of a user is maximized, the bandwidth utilization rate of static point cloud transmission is improved, and better quality experience is provided for the user;
(3) according to the method and the system for the adaptive transmission of the code rate of the static point cloud server, provided by the invention, the 3D space slices are projected onto a 2D screen watched by a user by utilizing perspective projection according to whether the space slices are in a visual field of the user, the distance between the space slices and the user, the shielding among the space slices, the code rate version of the space slices and the resolution of the display screen, so that the QoE contribution of each space slice is calculated, the QoE of the user is further improved, the bandwidth utilization rate of static point cloud transmission is improved, and better quality experience is provided for the user.
Drawings
Embodiments of the invention are further described below with reference to the accompanying drawings:
fig. 1 is a flowchart of a code rate adaptive transmission method for a static point cloud server according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a spatial slice and a view field range when a user views a point cloud according to an embodiment of the invention;
fig. 3 is a schematic diagram of a code rate adaptive transmission system of a static point cloud server according to an embodiment of the invention.
Description of reference numerals: 1-user side, 2-server side;
21-space slice codec, 22-space slice QoE contribution calculating device and 23-code rate allocating device.
Detailed Description
The following examples are given for the detailed implementation and specific operation of the present invention, but the scope of the present invention is not limited to the following examples.
Fig. 1 is a flowchart of a code rate adaptive transmission method for a static point cloud server according to an embodiment of the present invention.
Referring to fig. 1, the code rate adaptive transmission method for a static point cloud server of the present embodiment includes:
s11: the server divides the static point cloud into a plurality of 3D space slices;
s12: feeding back user viewing information and display screen resolution to a server in real time, and downloading a code rate version complete set which can be allocated to each 3D space slice obtained by S11 division from the server;
s13: calculating the QoE contribution of each spatial slice according to the user viewing information and the resolution of a display screen;
s14: the method comprises the steps of establishing an optimization problem of code rate self-adaptive distribution of a static point cloud server end based on the space slices by adopting a complete set formed by the 3D space slices obtained by division in S11, a complete set of code rate versions of the 3D space slices in S12, QoE contribution of each 3D space slice obtained by S13 and network resource limitation, and obtaining an optimal space slice version subset by adopting a code rate distribution method to maximize viewing code rate experience of a user.
The following example analysis is performed on the segmentation and encoding of the static point cloud at the server side, in this embodiment, assuming that the network bandwidth capacity of the server is BW, the static point cloud file stored in the server is segmented into NTSame size spatial slice (N)TAny positive integer greater than 2) as a set of spatial slices
In the preferred embodiment, the dividing of the static point cloud into 3D spatial slices in S11 is specifically: the static point cloud is divided into a preset number of 3D space slices by using a point cloud coding technology, and each space slice has the same or different coding code rate. Specifically, any one spatial slice is coded as NQRepresentation of several different quality levels (N)QAny positive integer greater than 2), and is denoted as {1, …, q, …, NQCorresponding to different coding bit numbers, the coding bit number is recorded as a space slice coding bit number set
And the sets are arranged in descending order of the number of coded bits, i.e.
The number of coded bits for the ith spatial slice, quality of which is denoted q, may be denoted Bi,q。
In a preferred embodiment, the user viewing information comprises: the head movement trajectory of the user.
In a preferred embodiment, the step of calculating the QoE contribution of each spatial slice according to the user viewing information and the resolution of the display screen in S13 is specifically as follows: the QoE contribution of each 3D spatial slice is calculated by a coordinate transformation method based on perspective projection.
In a preferred embodiment, the method for transforming coordinates of perspective projection further comprises: according to the geometric relationship between the 3D point cloud and the two-dimensional projection plane thereof, as perspective projection is closest to a human eye vision system, perspective projection transformation which is close to the human eye vision system and is suitable for 3D space slices is performed according to user viewport information, the distance between a user and the point cloud, and shielding and display screen resolution between the 3D space slices, and each 3D space slice is perspectively projected onto a 2D display screen, as shown in FIG. 2. Uncompressed original spatial slice TiThe number of voxels contained therein is denoted by Nv,iAnd the number of pixels projected onto the 2D screen is represented as Np,i。
And according to whether the 3D space slice can be projected onto a two-dimensional display screen or not, calculating the QoE contribution of the 3D space slice according to the relation between the number of voxels contained in the 3D space slice and the number of pixels corresponding to the number of pixels which can be seen by a user on the display screen.
In a preferred embodiment, the QoE contribution of a spatial slice is determined by the user viewport information, the distance of the user to the spatial slice, the occlusion between spatial slices, the display screen resolution, and the number of coded bits of the spatial slice. For invisible spatial slices (spatial slices are not in the user field of view or are completely occluded), the QoE contribution is 0; for a visible slice, the QoE contribution size is inversely proportional to the distance to the user and may be limited by the display screen resolution. Furthermore, the higher the number of coded bits of a spatial slice, the higher the quality representation and thus the greater the QoE contribution.
Further, QoE contribution computation for 3D spatial slices, comprising:
setting static point clouds to be divided into NT3D spatial slices, each spatial slice being encoded into q quality representations, corresponding to q code rate versions, the ith spatial slice T with quality represented as qiThe QoE contribution of (a) is expressed as:
Qi,q＝UPBi·Bi,q
wherein, Bi,qIs to mix TiCoding the number of bits required for expression as q, UPBiRepresents the utility of each bit, expressed as:
wherein N isv,iSlicing T into uncompressed spaceiNumber of voxels in the user field of view, Np,iIs to project T by perspectiveiMapping to the corresponding number of pixels on the display screen, Bi,maxTo normalize the parameters, represent TiThe coded representation is the number of bits required for the highest quality representation.
User viewport information, spatial slice-to-user distance, occlusion between spatial slices, andand the influence of the resolution of the display screen on the QoE contribution is implicitly reflected in the UPB through perspective projectioniIn (2), the influence of the number of spatial slice coding bits on the QoE contribution is explicitly reflected in Bi,qIn (1).
In the preferred embodiment, the target optimization problem is:
the constraint conditions are as follows: sigmai∑qBi,qxi,q≤BW
∑qxi,q＝1,i＝1,…,NT
xi,q∈{0,1},i＝1,…,NT,q＝1,…,NQ,
Wherein,
defining an ith spatial slice with quality denoted q as T for a set of spatial slices visible to a useri,qDefinition of Ti,qQoE contribution of (2) is Qi,qWill TiThe number of bits required to encode the quality representation q is Bi,q，NQThe number of quality representations corresponding to different code rate versions;
the optimization variables are: x is the number ofi,qIndicating whether to transmit T or noti,qTo the user end, the transmission value is 1, otherwise the transmission value is 0;
defining the network bandwidth capacity of a server as BW;
the optimization target is as follows: maximizing the sum of QoE contributions for all visible spatial slices;
the constraint conditions are as follows: 1) network bandwidth limiting conditions, namely server-side bandwidth constraints; 2) code rate version limiting conditions, namely the code rate versions are discrete variables, each visible space slice can only be allocated with one code rate version, 3D space slices with low QoE contribution to a user keep lower code rate versions, and 3D space slices with high QoE contribution to the user should be allocated with high code rate versions as much as possible;
in a preferred embodiment, the code rate allocation method specifically includes:
when a specific code rate version is allocated to each space slice of a static point cloud, an original Non-deterministic Polynomial problem (NP-hard) multi-choice knapsack problem is equivalently converted into a submodel function maximization problem under knapsack constraint, a code rate self-adaptive optimization algorithm based on a greedy algorithm with Polynomial time complexity and high approximate optimization performance is provided, and the worst case (worst-case) performance of the algorithm can be proved to be guaranteed to be a mode function maximization problem
The algorithm is adopted to finally obtain the optimal spatial slice code rate version subset allocated to the user quickly and efficiently.
Fig. 3 is a schematic diagram of a code rate adaptive transmission system of a static point cloud server according to an embodiment of the invention.
Referring to fig. 3, the code rate adaptive transmission system of the static point cloud server of the present embodiment includes: a user terminal 1 and a server terminal 2; the server side 2 includes: a spatial slice codec 21, a spatial slice QoE contribution calculation means 22 and a code rate allocation means 23. Wherein,
user end 1: and feeding back the user viewing information and the resolution of the display screen to the server end 2 in real time, and downloading the code rate version distributed to each 3D space slice from the server end.
The server side 2: the spatial slice codec 21 divides the static point cloud into 3D spatial slices; the spatial slice QoE contribution calculating device 22 calculates QoE contribution of each 3D spatial slice according to the user viewing information provided by the user terminal 1 and the display screen resolution; the code rate allocation device 23 adopts the complete set composed of the 3D space slices obtained by the static point cloud division, the complete set of the code rate versions of the 3D space slices, the QoE contribution of each 3D space slice and the network resource limitation, establishes an optimization problem of the code rate adaptive allocation of the static point cloud server based on the space slices, and obtains an optimal code rate version subset of the space slices by adopting a code rate allocation method.
In a preferred embodiment, when the user terminal requests the server terminal to download the point cloud space slice, the request information sent includes: viewing information for the user at the current time (user viewport information, distance between the user and the spatial slice), display screen resolution, etc. After the above information is processed by the spatial slice QoE contribution device, the code rate allocation device 23 takes the network status at the current time of the server, the complete set composed of spatial slices of different code rate versions, the QoE contribution of each spatial slice, and other information as input, and then optimally allocates the number of coding bits of the visible 3D spatial slice requested by the user, so as to maximize the viewing experience when the user views the point cloud.
In a preferred embodiment, the step of dividing the static point cloud into 3D spatial slices by the server is as follows: a point cloud encoding technique is used to divide the static point cloud into a preset number of 3D spatial slices.
In a preferred embodiment, the user viewing information comprises: the head movement trajectory of the user.
In a preferred embodiment, the calculating, by the server side, the QoE contribution of each spatial slice according to the user viewing information and the display screen resolution specifically includes: the QoE contribution of each spatial slice is calculated by a coordinate transformation method based on perspective projection.
In a preferred embodiment, the method for transforming coordinates of perspective projection further comprises: according to the geometric relation between the 3D point cloud and the two-dimensional projection plane thereof, and according to the user viewport information, the distance between the user and the point cloud, and the shielding and display screen resolution between the 3D space slices, carrying out perspective projection transformation which is close to a human eye visual system and is suitable for the 3D space slices; and according to whether the 3D space slice can be projected on a display screen or not, calculating the QoE contribution of the 3D space slice according to the relation between the number of voxels contained in the 3D space slice and the number of pixels corresponding to the number of pixels which can be seen by a user on the display screen.
In a preferred embodiment, the QoE contribution calculation for a 3D spatial slice includes:
setting static point clouds to be divided into NT3D spatial slices, each spatial slice being encoded into q quality representations, corresponding to q code rate versions, the ith spatial slice T with quality represented as qiThe QoE contribution of (a) is expressed as:
Qi,q＝UPBi·Bi,q
wherein, Bi,qIs to mix TiCoding the number of bits required for expression as q, UPBiRepresents the utility of each bit, expressed as:
wherein N isv,iSlicing T into uncompressed spaceiNumber of voxels in the user field of view, Np,iIs to project T by perspectiveiMapping to the corresponding number of pixels on the display screen, Bi,maxTo normalize the parameters, represent TiThe coded representation is the number of bits required for the highest quality representation.
The method and the system for adaptive transmission of the code rate of the static point cloud server end provided by the embodiment are based on 3D spatial slice QoE contribution calculation by perspective projection, and are combined with a spatial slice encoding and decoding technology at the server to divide the static point cloud into a plurality of spatial slices, wherein each spatial slice is encoded into a plurality of versions with different code rates and cached. And then, the code rate self-adaptive distribution device at the server end determines code rate versions of all 3D space slices visible to the user, and finally, the maximization of the viewing experience of the user is realized. The invention improves the bandwidth utilization rate of static point cloud transmission and provides better service quality for users.
It should be noted that the steps in the server-side code rate adaptive transmission method for static point clouds provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the server-side code rate adaptive transmission system for static point clouds, and those skilled in the art can refer to the technical scheme of the system to implement the steps of the method, that is, the embodiments in the system can be understood as preferred examples for implementing the method, and are not described herein again.
Those skilled in the art will appreciate that, in addition to implementing the system and its various devices provided by the present invention in purely computer readable program code means, the method steps can be fully programmed to implement the same functions by implementing the system and its various devices in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices thereof provided by the present invention can be regarded as a hardware component, and the devices included in the system and various devices thereof for realizing various functions can also be regarded as structures in the hardware component; means for performing the functions may also be regarded as structures within both software modules and hardware components for performing the methods.
The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, and not to limit the invention. Any modifications and variations within the scope of the description, which may occur to those skilled in the art, are intended to be within the scope of the invention.
Claims (14)
1. A code rate self-adaptive transmission method for a static point cloud server is characterized by comprising the following steps:
s11: the server divides the static point cloud into a plurality of 3D space slices;
s12: feeding back user viewing information and display screen resolution to a server in real time, and downloading a code rate version complete set which can be allocated to each 3D space slice obtained by S11 division from the server;
s13: calculating QoE contribution of each 3D space slice obtained by S11 division according to the user viewing information and the resolution of the display screen;
s14: and establishing an optimization problem of code rate self-adaptive allocation of the static point cloud server-side code rate based on the space slices by adopting a complete set consisting of the 3D space slices obtained by the division of S11, a complete set of the code rate versions of the 3D space slices in S12, QoE contribution and network resource limitation of each 3D space slice obtained in S13, and obtaining an optimal code rate version subset of the space slices by adopting a code rate allocation method.
2. The method for adaptive transmission of static point cloud server-side code rate according to claim 1, wherein the dividing of the static point cloud into 3D spatial slices specifically comprises: a point cloud encoding technique is used to divide the static point cloud into a preset number of 3D spatial slices.
3. The adaptive transmission method for the static point cloud server-side code rate according to claim 1, wherein the user viewing information comprises: the head movement trajectory of the user.
4. The adaptive transmission method for the bitrate of the server side of the static point cloud of claim 1, wherein the QoE contribution of each 3D spatial slice is calculated according to the user viewing information and the resolution of the display screen, specifically: the QoE contribution of each 3D spatial slice is calculated by a coordinate transformation method based on perspective projection.
5. The adaptive transmission method for the static point cloud server-side code rate according to claim 4, wherein the coordinate transformation method for perspective projection comprises:
according to the geometric relation between the 3D point cloud and the two-dimensional projection plane thereof, and according to the user viewport information, the distance between the user and the point cloud, and the shielding and display screen resolution between the 3D space slices, carrying out perspective projection transformation which is close to a human eye visual system and is suitable for the 3D space slices;
and according to whether the 3D space slice can be projected on a display screen or not, calculating the QoE contribution of the 3D space slice according to the relation between the number of voxels contained in the 3D space slice and the number of pixels corresponding to the number of pixels which can be seen by a user on the display screen.
6. The method for adaptive transmission of static point cloud server-side code rate according to claim 5, wherein the QoE contribution calculation of the 3D spatial slice comprises:
setting static point clouds to be divided into NTA 3D spatial slice, each spatial slice being encoded intoq quality representations, corresponding to q rate-versions, then the quality is represented as the ith spatial slice T of qiThe QoE contribution of (a) is expressed as:
Qi,q＝UPBi·Bi,q
wherein, Bi,qIs to mix TiCoding the number of bits required for expression as q, UPBiRepresents the utility of each bit, expressed as:
wherein N isv,iSlicing T into uncompressed spaceiNumber of voxels in the user field of view, Np,iIs to project T by perspectiveiMapping to the corresponding number of pixels on the display screen, Bi,maxTo normalize the parameters, represent TiThe coded representation is the number of bits required for the highest quality representation.
7. The adaptive transmission method for the code rate of the static point cloud server end according to claim 1, wherein the optimization problem of establishing the adaptive allocation of the code rate of the static point cloud server end based on the spatial slice specifically comprises: and modeling the optimization problem of code rate self-adaptive distribution into a multi-choice knapsack problem by adopting a complete set formed by 3D space slices obtained by dividing the static point cloud, a complete set of code rate versions of the 3D space slices, network resource limitation and QoE contribution of each 3D space slice.
8. The adaptive transmission method for code rate of static point cloud server end of claim 7, characterized by equivalently transforming the multi-choice knapsack problem into a sub-modular function maximization problem under knapsack constraint.
9. A code rate self-adaptive transmission system of a static point cloud server end is characterized by comprising the following steps:
the user side feeds back the user viewing information and the resolution of the display screen to the server side in real time, and downloads a code rate version complete set which can be distributed to each 3D space slice from the server side;
and the server end divides the static point cloud into a plurality of 3D space slices, calculates QoE contribution of each 3D space slice according to the user viewing information and the resolution of a display screen, establishes an optimization problem of code rate self-adaptive distribution of the static point cloud server end based on the space slices by adopting a full set formed by the 3D space slices obtained by dividing the static point cloud, a full set of code rate versions of the space slices, the QoE contribution of each 3D space slice and network resource limitation, and obtains an optimal space slice code rate version subset by adopting a code rate distribution method.
10. The system of claim 9, wherein the QoE contribution of each 3D spatial slice is calculated by a coordinate transformation method based on perspective projection.
11. The adaptive transmission system for the static point cloud server-side code rate according to claim 10, wherein the coordinate transformation method for perspective projection comprises:
according to the geometric relation between the 3D point cloud and the two-dimensional projection plane thereof, and according to the user viewport information, the distance between the user and the point cloud, and the shielding and display screen resolution between the 3D space slices, carrying out perspective projection transformation which is close to a human eye visual system and is suitable for the 3D space slices;
and according to whether the 3D space slice can be projected on a display screen or not, calculating the QoE contribution of the 3D space slice according to the relation between the number of voxels contained in the 3D space slice and the number of pixels corresponding to the number of pixels which can be seen by a user on the display screen.
12. The system for adaptive transmission of static point cloud server-side bitrate according to claim 11, wherein the QoE contribution calculation of the 3D spatial slice comprises:
setting static point clouds to be divided into NTA 3D space slice eachThe spatial slices are encoded into q quality representations, corresponding to q rate versions, so that the ith spatial slice T, whose quality is represented as qiThe QoE contribution of (a) is expressed as:
Qi,q＝UPBi·Bi,q
wherein, Bi,qIs to mix TiCoding the number of bits required for expression as q, UPBiRepresents the utility of each bit, expressed as:
wherein N isv,iSlicing T into uncompressed spaceiNumber of voxels in the user field of view, Np,iIs to project T by perspectiveiMapping to the corresponding number of pixels on the display screen, Bi,maxTo normalize the parameters, represent TiThe coded representation is the number of bits required for the highest quality representation.
13. The adaptive transmission system for code rate of the static point cloud server end according to claim 8, wherein the server end establishes an optimization problem of adaptive allocation of code rate of the static point cloud server end based on spatial slicing, specifically: and modeling the optimization problem of code rate self-adaptive distribution into a multi-choice knapsack problem by adopting a complete set formed by 3D space slices obtained by dividing the static point cloud, a complete set of code rate versions of the 3D space slices, network resource limitation and QoE contribution of each 3D space slice.
14. The adaptive transmission system for static point cloud server-side code rate according to claim 13, wherein the multi-choice knapsack problem is equivalently transformed into a submodular function maximization problem under knapsack constraint.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110269265.1A CN113115077B (en) | 2021-03-12 | 2021-03-12 | Code rate self-adaptive transmission method and system for static point cloud server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110269265.1A CN113115077B (en) | 2021-03-12 | 2021-03-12 | Code rate self-adaptive transmission method and system for static point cloud server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113115077A true CN113115077A (en) | 2021-07-13 |
| CN113115077B CN113115077B (en) | 2022-04-26 |
Family
ID=76711164
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110269265.1A Active CN113115077B (en) | 2021-03-12 | 2021-03-12 | Code rate self-adaptive transmission method and system for static point cloud server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113115077B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113395603A (en) * | 2021-06-25 | 2021-09-14 | 合肥工业大学 | Point cloud video stream self-adaptive transmission method based on model predictive control |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180041788A1 (en) * | 2015-02-07 | 2018-02-08 | Zhou Wang | Method and system for smart adaptive video streaming driven by perceptual quality-of-experience estimations |
| CN108551586A (en) * | 2018-03-14 | 2018-09-18 | 上海交通大学 | 360 degree of video stream server end code check self-adapting distribution methods of multi-user and system |
| CN109451293A (en) * | 2018-08-09 | 2019-03-08 | 北京邮电大学 | A kind of adaptive three-dimensional video transmission method of open base station computing capability |
| US20190158564A1 (en) * | 2016-03-06 | 2019-05-23 | Ssimwave Inc. | Method and system for automatic user quality-of-experience measurement of streaming video |
| CN110012279A (en) * | 2018-01-05 | 2019-07-12 | 上海交通大学 | Divide visual angle compression and transmission method and system based on 3D point cloud data |
| CN110266714A (en) * | 2019-06-28 | 2019-09-20 | 合肥工业大学 | VR video under a kind of QoE driving adaptively acquires and transmission method |
| WO2020049181A1 (en) * | 2018-09-07 | 2020-03-12 | NEC Laboratories Europe GmbH | System and method for network automation in slice-based network using reinforcement learning |
| US20200151915A1 (en) * | 2018-05-09 | 2020-05-14 | Peking University Shenzhen Graduate School | Hierarchical division-based point cloud attribute compression method |
| EP3754993A1 (en) * | 2019-06-19 | 2020-12-23 | Koninklijke KPN N.V. | Rendering video stream in sub-area of visible display area |
| US20200404241A1 (en) * | 2019-06-20 | 2020-12-24 | At&T Intellectual Property I, L.P. | Processing system for streaming volumetric video to a client device |
-
2021
- 2021-03-12 CN CN202110269265.1A patent/CN113115077B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180041788A1 (en) * | 2015-02-07 | 2018-02-08 | Zhou Wang | Method and system for smart adaptive video streaming driven by perceptual quality-of-experience estimations |
| US20190158564A1 (en) * | 2016-03-06 | 2019-05-23 | Ssimwave Inc. | Method and system for automatic user quality-of-experience measurement of streaming video |
| CN110012279A (en) * | 2018-01-05 | 2019-07-12 | 上海交通大学 | Divide visual angle compression and transmission method and system based on 3D point cloud data |
| CN108551586A (en) * | 2018-03-14 | 2018-09-18 | 上海交通大学 | 360 degree of video stream server end code check self-adapting distribution methods of multi-user and system |
| US20200151915A1 (en) * | 2018-05-09 | 2020-05-14 | Peking University Shenzhen Graduate School | Hierarchical division-based point cloud attribute compression method |
| CN109451293A (en) * | 2018-08-09 | 2019-03-08 | 北京邮电大学 | A kind of adaptive three-dimensional video transmission method of open base station computing capability |
| WO2020049181A1 (en) * | 2018-09-07 | 2020-03-12 | NEC Laboratories Europe GmbH | System and method for network automation in slice-based network using reinforcement learning |
| EP3754993A1 (en) * | 2019-06-19 | 2020-12-23 | Koninklijke KPN N.V. | Rendering video stream in sub-area of visible display area |
| US20200404241A1 (en) * | 2019-06-20 | 2020-12-24 | At&T Intellectual Property I, L.P. | Processing system for streaming volumetric video to a client device |
| CN110266714A (en) * | 2019-06-28 | 2019-09-20 | 合肥工业大学 | VR video under a kind of QoE driving adaptively acquires and transmission method |
Non-Patent Citations (3)
| Title |
|---|
| JEROEN VAN DER HOOFT等: "Towards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression", 《MM "19: PROCEEDINGS OF THE 27TH ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA》 * |
| JOUNSUP PARK等: "Rate-Utility Optimized Streaming of Volumetric Media for Augmented Reality", 《IEEE JOURNAL ON EMERGING AND SELECTED TOPICS IN CIRCUITS AND SYSTEMS》 * |
| 张克新等: "基于MPEG-DASH的视频自适应共享***", 《电视技术》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113395603A (en) * | 2021-06-25 | 2021-09-14 | 合肥工业大学 | Point cloud video stream self-adaptive transmission method based on model predictive control |
| CN113395603B (en) * | 2021-06-25 | 2022-04-01 | 合肥工业大学 | Point cloud video stream self-adaptive transmission method based on model predictive control |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113115077B (en) | 2022-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3669547B1 (en) | Method and apparatus for point-cloud streaming | |
| CN108551586B (en) | Multi-user 360-degree video stream server-side code rate self-adaptive distribution method and system | |
| US20180241988A1 (en) | Multi-View Video Transmission Method and Apparatus | |
| CN108156484B (en) | Prioritizing tile-based virtual reality video streams with adaptive rate allocation | |
| CN110248212B (en) | Multi-user 360-degree video stream server-side code rate self-adaptive transmission method and system | |
| US10242462B2 (en) | Rate control bit allocation for video streaming based on an attention area of a gamer | |
| Park et al. | Volumetric media streaming for augmented reality | |
| WO2018090774A1 (en) | Method and system for bit rate control and version selection for dynamic adaptive video streaming media | |
| CN110248210B (en) | Video transmission optimization method | |
| WO2018126702A1 (en) | Streaming media transmission method applied to virtual reality technology and client | |
| US11159823B2 (en) | Multi-viewport transcoding for volumetric video streaming | |
| CN112995636B (en) | 360-degree virtual reality video transmission system based on edge calculation and active cache and parameter optimization method | |
| US20200404241A1 (en) | Processing system for streaming volumetric video to a client device | |
| CN113115077B (en) | Code rate self-adaptive transmission method and system for static point cloud server | |
| Pazzi et al. | Propane: A progressive panorama streaming protocol to support interactive 3d virtual environment exploration on graphics-constrained devices | |
| Yahia et al. | Http/2-based streaming solutions for tiled omnidirectional videos | |
| Li et al. | Toward Optimal Real-Time Volumetric Video Streaming: A Rolling Optimization and Deep Reinforcement Learning Based Approach | |
| Ramanathan et al. | Rate-distortion optimized interactive light field streaming | |
| Alkhalili et al. | A survey of volumetric content streaming approaches | |
| CN117115326A (en) | Rendering method and device of three-dimensional scene model and distributed rendering server | |
| CN117280680A (en) | Parallel mode of dynamic grid alignment | |
| Shafi et al. | MTC360: A multi-tiles configuration for viewport-dependent 360-degree video streaming | |
| Liu et al. | Point cloud video streaming in 5G systems and beyond: challenges and solutions | |
| JP2019047249A (en) | Client for determining distribution profile based on displayable resolution, program and method | |
| CN112188235B (en) | Media processing mode selection method and media processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |