CN110708562A - Panoramic video transmission method and system based on node calculation - Google Patents
Panoramic video transmission method and system based on node calculation Download PDFInfo
- Publication number
- CN110708562A CN110708562A CN201811178241.XA CN201811178241A CN110708562A CN 110708562 A CN110708562 A CN 110708562A CN 201811178241 A CN201811178241 A CN 201811178241A CN 110708562 A CN110708562 A CN 110708562A
- Authority
- CN
- China
- Prior art keywords
- information
- transmission
- user
- visual field
- redundant
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 138
- 238000000034 method Methods 0.000 title claims abstract description 28
- 230000000007 visual effect Effects 0.000 claims abstract description 71
- 238000012545 processing Methods 0.000 claims abstract description 27
- 239000011159 matrix material Substances 0.000 claims description 49
- 238000013519 translation Methods 0.000 claims description 19
- 238000003384 imaging method Methods 0.000 claims description 7
- 230000004044 response Effects 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 6
- 230000008569 process Effects 0.000 description 4
- 238000009877 rendering Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 208000002173 dizziness Diseases 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000004886 head movement Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23424—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Business, Economics & Management (AREA)
- Marketing (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Studio Devices (AREA)
Abstract
The invention provides a node calculation-based panoramic video transmission method and a node calculation-based panoramic video transmission system, which comprise an intermediate node, wherein feedback information of a user terminal is acquired in real time, a user visual field area is calculated according to user visual field information, and video streams containing the user visual field area are selected for splicing to obtain processed video streams; selecting a transmission scheme according to the user network condition information, extracting a transmission area aiming at the processed video stream, and transmitting the transmission area code to a corresponding user terminal; the transmission scheme comprises non-redundant transmission and redundant transmission, wherein in the non-redundant transmission, a transmission area is a user visual field area, in the redundant transmission, the transmission area is a user visual field area and a redundant area, and the redundant area is obtained by calculation according to user network condition information and user terminal processing capacity information. The invention can effectively reduce the processing time delay and the bandwidth overhead by combining the feedback information of the user terminal, thereby integrally improving the consumption experience of the user.
Description
Technical Field
The invention relates to the field of node calculation and video processing, in particular to a panoramic video transmission method and system based on node calculation.
Background
With the rapid development of immersive media, Virtual Reality (VR) is applied to more and more consumption scenes, such as games, live games, and the like. In these scenarios, it is often required to have a low response delay to ensure that the user does not feel dizzy, which puts higher demands on the transmission delay of data. However, as a basic data format in VR, panoramic video has a huge data size and is complex to process. The generation of a general panoramic video requires multiple cameras to cover a 360-degree range for data collection, as shown in fig. 1, and then multiple videos are spliced into spherical data through splicing and rendering, and then the spherical data is consumed by a user. The process of splicing is complex, and a certain time delay is often generated under the condition that the capacity of a local processor is limited. In a live scene, such a delay may have a certain impact on the user experience. In fact, when a user watches the panoramic video, only part of content in the panoramic video is concerned, complete splicing is not necessary, node calculation is developed gradually, and partial processing of the panoramic video is put into an intermediate node.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a panoramic video transmission method and system based on node calculation.
The node calculation-based panoramic video transmission method provided by the invention comprises an intermediate node, wherein the intermediate node executes the following steps:
an information acquisition step: obtaining feedback information of a user terminal in real time, wherein the feedback information at least comprises: user visual field information, user network condition information and user terminal processing capacity information;
splicing: calculating a user visual field region according to the user visual field information, and selecting a video stream containing the user visual field region for splicing to obtain a processed video stream;
a transmission decision step: selecting a transmission scheme according to the user network condition information, extracting a transmission area aiming at the processed video stream, and transmitting a transmission area code to a corresponding user terminal;
the transmission scheme comprises non-redundant transmission and redundant transmission, wherein in the non-redundant transmission, the transmission region is the user visual field region, in the redundant transmission, the transmission region is the user visual field region plus a redundant region, and the redundant region is obtained by calculation according to the user network condition information and the user terminal processing capacity information.
Preferably, in the redundant transmission, the transmission decision step further includes:
and carrying out unequal quality coding on the user visual field region and the redundant region and transmitting the unequal quality coding to a corresponding user terminal, namely reducing the coding quality of the redundant region.
Preferably, the feedback information at least includes:
user visual field information: describing the position information of the video content currently watched by the user;
user network status information: describing the transmission condition of the current network of the user, including bandwidth and delay;
user terminal processing capability information: the computing power of the user terminal is described, including throughput, response time and cpu occupancy.
Preferably, the video stream before splicing is transmitted to the intermediate node by a camera, the camera transmits the assisted splicing information while transmitting the video stream, and the assisted splicing information includes:
stream coverage information: describing the position of the area covered by the current video stream in the whole panorama when the current video stream is finally imaged, wherein the angle range information comprises horizontal angle information and vertical angle information;
time stamp information: describing a time at which the current video stream was captured;
camera matrix information: the method comprises the following steps that external parameters and internal parameters are included, the external parameters comprise a rotation matrix and a translation matrix of a camera, the rotation matrix and the translation matrix describe how to convert points from a world coordinate system to the camera coordinate system together, the rotation matrix describes the directions of coordinate axes of the world coordinate system relative to the coordinate axes of the camera, and the translation matrix describes the position of a space origin under the camera coordinate system; the intrinsic parameters include focal length and the transformation of the imaging plane coordinate system to the pixel coordinate system.
Preferably, the video stream transmitted by each camera is transmitted as a separate video stream.
The invention provides a panoramic video transmission system based on node calculation, which comprises an intermediate node, wherein the intermediate node comprises:
an information acquisition module: obtaining feedback information of a user terminal in real time, wherein the feedback information at least comprises: user visual field information, user network condition information and user terminal processing capacity information;
splicing modules: calculating a user visual field region according to the user visual field information, and selecting a video stream containing the user visual field region for splicing to obtain a processed video stream;
a transmission decision module: selecting a transmission scheme according to the user network condition information, extracting a transmission area aiming at the processed video stream, and transmitting a transmission area code to a corresponding user terminal;
the transmission scheme comprises non-redundant transmission and redundant transmission, wherein in the non-redundant transmission, the transmission region is the user visual field region, in the redundant transmission, the transmission region is the user visual field region plus a redundant region, and the redundant region is obtained by calculation according to the user network condition information and the user terminal processing capacity information.
Preferably, in the redundant transmission, the transmission decision module further includes:
and carrying out unequal quality coding on the user visual field region and the redundant region and transmitting the unequal quality coding to a corresponding user terminal, namely reducing the coding quality of the redundant region.
Preferably, the feedback information at least includes:
user visual field information: describing the position information of the video content currently watched by the user;
user network status information: describing the transmission condition of the current network of the user, including bandwidth and delay;
user terminal processing capability information: the computing power of the user terminal is described, including throughput, response time and cpu occupancy.
Preferably, the video stream before splicing is transmitted to the intermediate node by a camera, the camera transmits the assisted splicing information while transmitting the video stream, and the assisted splicing information includes:
stream coverage information: describing the position of the area covered by the current video stream in the whole panorama when the current video stream is finally imaged, wherein the angle range information comprises horizontal angle information and vertical angle information;
time stamp information: describing a time at which the current video stream was captured;
camera matrix information: the method comprises the following steps that external parameters and internal parameters are included, the external parameters comprise a rotation matrix and a translation matrix of a camera, the rotation matrix and the translation matrix describe how to convert points from a world coordinate system to the camera coordinate system together, the rotation matrix describes the directions of coordinate axes of the world coordinate system relative to the coordinate axes of the camera, and the translation matrix describes the position of a space origin under the camera coordinate system; the intrinsic parameters include focal length and the transformation of the imaging plane coordinate system to the pixel coordinate system.
Preferably, the video stream transmitted by each camera is transmitted as a separate video stream.
Compared with the prior art, the invention has the following beneficial effects:
according to the invention, by combining the computing power of the intermediate node and the user side feedback information such as the user visual field information, the processing time delay can be effectively reduced, the bandwidth overhead can be reduced, and the consumption experience of the user can be integrally improved.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a schematic diagram of a conventional multi-camera based panoramic video capture;
FIG. 2 is a flow chart of the present invention fused with a conventional transmission method;
fig. 3 is a transmission schematic diagram of the present invention fused with a conventional transmission method.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
As shown in fig. 2 and fig. 3, the panoramic video transmission method based on node calculation provided by the present invention includes an intermediate node, and the intermediate node executes:
an information acquisition step: obtaining feedback information of a user terminal in real time, wherein the feedback information at least comprises: user visual field information, user network condition information and user terminal processing capacity information;
splicing: calculating a user visual field region according to the user visual field information, and selecting a video stream containing the user visual field region for splicing to obtain a processed video stream;
a transmission decision step: selecting a transmission scheme according to the user network condition information, extracting a transmission area aiming at the processed video stream, and transmitting the transmission area code to a corresponding user terminal;
the transmission scheme comprises non-redundant transmission and redundant transmission, wherein in the non-redundant transmission, a transmission area is a user visual field area, in the redundant transmission, the transmission area is a user visual field area and a redundant area, and the redundant area is obtained by calculation according to user network condition information and user terminal processing capacity information.
The transmission scheme of the invention comprises two transmission states of non-redundant transmission and redundant transmission, does not conflict with the traditional full-splicing method, and can coexist.
Non-redundant transmission state: when the bandwidth condition is good, namely the transmission environment can quickly react to the change of the user visual field, the intermediate node calculates the user visual field area as a transmission area according to the user visual field information, then selects the video stream containing the user visual field area for splicing, intercepts the transmission area, and transmits the coding to the user terminal for decoding and presentation, namely a non-redundant transmission scheme.
Redundant transmission state: when the bandwidth condition is poor, namely the transmission environment cannot react to the user head movement in time, a redundant transmission scheme based on the user visual field is adopted, namely the feedback information of the user terminal such as the network bandwidth condition, the user visual field information, the user terminal processing capacity and the like is comprehensively considered, the optimal area to be transmitted is calculated, the corresponding video streams are selected according to the optimal area to be spliced, and the optimal area is used as the transmission area to be transmitted. Specifically, a user visual field region is calculated according to user visual field information, a redundant region is calculated according to user network condition information and user terminal processing capacity information, the user visual field region and the redundant region jointly form a transmission region (namely, the optimal region), and video streams including the user visual field region and the redundant region are selected to be spliced to obtain a processed video stream.
On the basis of the redundant transmission state, the intermediate node can transmit the optimal region of the unified coding, and can also perform unequal quality coding on the user view region and the redundant region to transmit to the corresponding user terminal, namely, the coding quality of the redundant region is reduced. The range of the optimal area is generally larger than the visual field area of the user, so that the condition that the visual field of the user is lost due to network delay can be effectively avoided. After receiving the optimal area, the user terminal needs to further perform mapping and rendering, but the amount of data is small, so that the time is short.
In order to implement the above two splicing schemes, corresponding assisted splicing information needs to be transmitted simultaneously to assist the splicing of the intermediate node when the video camera transmits the video stream to the intermediate node, and the assisted splicing information needs to include the following information:
A. flow coverage CoverageRange information. The position of the area covered by the current video stream when finally imaged in the overall panorama is described as angle range information including horizontal and vertical angle information. Through the initial comparison of the angle coverage and the user visual field information, the camera stream related to the user visual field information can be quickly found for partial splicing.
B. Timestamp information. And describing the time for shooting the current video stream, and guiding the splicing of the cloud server. Since there may be differences in the time of arrival of multiple video streams at the cloud server, but the spliced content must be recorded at the same time, a timestamp is needed to ensure synchronicity.
C. Camera matrix CameraMatrix information: when splicing is carried out, the contents shot by different cameras need to be mapped to a translation coordinate system, so that the relevant parameters of a camera matrix need to be utilized. The camera matrix is divided into two parts, an internal parameter matrix and an external parameter matrix of the camera. When the panorama stitching is carried out, the internal and external parameters of the camera can be calculated through the specific contents of the previous frames of images, but in order to reduce the time delay as much as possible, the internal and external parameters of the camera are recommended to be stored in the information to assist the stitching. The external parameters comprise a rotation matrix and a translation matrix of the camera, wherein the rotation matrix and the translation matrix jointly describe how to convert the points from a world coordinate system to a camera coordinate system, the rotation matrix describes the directions of coordinate axes of the world coordinate system relative to coordinate axes of the camera, and the translation matrix describes the position of a space origin under the camera coordinate system; the intrinsic parameters comprise focal length and transformation from an imaging plane coordinate system to a pixel coordinate system; the internal and external parameters are combined together to obtain a camera matrix, and the image shot by the camera can be projected and transformed to the same coordinate system for splicing by using the matrix.
It should be noted that the above information is necessary information for assisting the intermediate node to perform the splicing, but is not limited to the above information.
In order to realize the judgment of the redundant and non-redundant transmission schemes, the user terminal needs to send feedback information to the intermediate node, where the feedback information includes:
A. user visual field information: describing the position information of the video content currently watched by the user;
B. user network status information: describing the transmission conditions of the current network of the user, including bandwidth and delay;
C. user terminal processing capability information: the computing power of the user's local processor is described, including throughput, response time and cpu occupancy.
It should be noted that the above information is necessary information for assisting the intermediate node in selecting the scheme, but is not limited to the above information.
Based on the above panoramic video transmission method based on node calculation, the present invention further provides a panoramic video transmission system based on node calculation, which includes an intermediate node, where the intermediate node includes:
an information acquisition module: obtaining feedback information of a user terminal in real time, wherein the feedback information at least comprises: user visual field information, user network condition information and user terminal processing capacity information;
splicing modules: calculating a user visual field region according to the user visual field information, and selecting a video stream containing the user visual field region for splicing to obtain a processed video stream;
a transmission decision module: selecting a transmission scheme according to the user network condition information, extracting a transmission area aiming at the processed video stream, and transmitting the transmission area code to a corresponding user terminal;
the transmission scheme comprises non-redundant transmission and redundant transmission, wherein in the non-redundant transmission, a transmission area is a user visual field area, in the redundant transmission, the transmission area is a user visual field area and a redundant area, and the redundant area is obtained by calculation according to user network condition information and user terminal processing capacity information.
In the redundant transmission, the transmission decision module further includes: and carrying out unequal quality coding on the user visual field area and the redundant area and transmitting the unequal quality coding to the corresponding user terminal, namely reducing the coding quality of the redundant area.
In the feedback information, the user visual field information describes the position information of the video content currently watched by the user; the user network condition information describes the transmission condition of the current network of the user, including bandwidth and delay; the user terminal processing capability information describes the computing capability of the user terminal, including throughput, response time and cpu occupancy.
The video stream before splicing is transmitted to the intermediate node by the cameras, the video stream transmitted by each camera is transmitted as an independent video stream, the cameras transmit the assisted splicing information while transmitting the video stream, and the assisted splicing information comprises:
stream coverage: describing the position of the area covered by the current video stream in the whole panorama when the current video stream is finally imaged, wherein the angle range information comprises horizontal angle information and vertical angle information;
time stamping: describing a time at which the current video stream was captured;
camera matrix: the method comprises the following steps that external parameters and internal parameters are included, the external parameters comprise a rotation matrix and a translation matrix of a camera, the rotation matrix and the translation matrix describe how to convert points from a world coordinate system to a camera coordinate system together, the rotation matrix describes the direction of coordinate axes of the world coordinate system relative to the coordinate axes of the camera, and the translation matrix describes the position of a space origin under the camera coordinate system; the intrinsic parameters include focal length and the transformation of the imaging plane coordinate system to the pixel coordinate system.
Taking a watching ball game as an example, a transmission method of mmt (mpeg Media transport) is adopted. The panoramic video comprises all information of a court, five cameras are used for shooting, one path of video stream is generated by each camera, the user visual field information is collected by the intermediate node, under the condition of non-redundant transmission, the intermediate node selects the video streams shot by the related cameras according to the user visual field information, splicing and rendering are carried out, then a rectangular area with the size just right corresponding to the user visual field information is extracted from the formed curved surface video and reflected to a plane, then coding is carried out for streaming transmission, when the user receives the corresponding video streams, the user can directly watch the video streams, and only decoding operation needs to be carried out locally. If the data is transmitted redundantly, the intermediate node needs to collect feedback information such as visual field information of a user, local processing capacity of the user, network performance and the like, then calculates an optimal region for transmission, then selects a video stream related to the optimal region for splicing and rendering, then extracts the optimal region for coding and transmitting, and the user needs to further process after receiving the data locally.
In the case of MMT transmission, the corresponding information description sent with the camera stream is shown in table one:
TABLE-MMT-based Multi-source stream descriptor
descriptor _ tag: indicating the type of descriptor;
descriptor _ length: specifying the number of bytes from the next byte after the field to the byte of the last byte of the descriptor;
timestamp, time information used for stream synchronization, recording the shooting time;
the CoverageRange indicates the angle range covered by the stream in the final imaging process and is used for judging whether the user view information area relates to the camera shooting content or not;
CameraMatrix () contains matrix information of a camera that photographs the stream;
for camera intrinsic and camera extrinsic parameters, it should be noted that the first table is only described for the splicing information by taking the above fields as examples, and is not limited to the above fields and their sizes.
The information fed back by the user is shown in table two:
TABLE II MMT-based User feedback descriptor
descriptor _ tag: indicating the type of descriptor;
descriptor _ length: specifying the number of bytes from the next byte after the field to the byte of the last byte of the descriptor;
UserFov indicating user field of view information
UserNet indicating user network condition parameters
UserCal indicating user terminal processing capability parameter
It should be noted that the second table is only used for describing the splicing information by taking the above fields as an example, and is not limited to the above fields and the sizes thereof.
Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, units provided by the present invention as pure computer readable program code, the system and its various devices, modules, units provided by the present invention can be fully implemented by logically programming method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices, modules and units thereof provided by the invention can be regarded as a hardware component, and the devices, modules and units included in the system for realizing various functions can also be regarded as structures in the hardware component; means, modules, units for performing the various functions may also be regarded as structures within both software modules and hardware components for performing the method.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (10)
1. A node computation-based panoramic video transmission method is characterized by comprising an intermediate node, wherein the intermediate node executes:
an information acquisition step: obtaining feedback information of a user terminal in real time, wherein the feedback information at least comprises: user visual field information, user network condition information and user terminal processing capacity information;
splicing: calculating a user visual field region according to the user visual field information, and selecting a video stream containing the user visual field region for splicing to obtain a processed video stream;
a transmission decision step: selecting a transmission scheme according to the user network condition information, extracting a transmission area aiming at the processed video stream, and transmitting a transmission area code to a corresponding user terminal;
the transmission scheme comprises non-redundant transmission and redundant transmission, wherein in the non-redundant transmission, the transmission region is the user visual field region, in the redundant transmission, the transmission region is the user visual field region plus a redundant region, and the redundant region is obtained by calculation according to the user network condition information and the user terminal processing capacity information.
2. The node-computing-based panoramic video transmission method according to claim 1, wherein in the redundant transmission, the transmission decision step further comprises:
and carrying out unequal quality coding on the user visual field region and the redundant region and transmitting the unequal quality coding to a corresponding user terminal, namely reducing the coding quality of the redundant region.
3. The node-computing-based panoramic video transmission method according to claim 1, wherein the feedback information at least comprises:
user visual field information: describing the position information of the video content currently watched by the user;
user network status information: describing the transmission condition of the current network of the user, including bandwidth and delay;
user terminal processing capability information: the computing power of the user terminal is described, including throughput, response time and cpu occupancy.
4. The node-computing-based panoramic video transmission method according to claim 1, wherein a video stream before splicing is transmitted to the intermediate node by a camera, the camera transmits auxiliary splicing information while transmitting the video stream, and the auxiliary splicing information includes:
stream coverage information: describing the position of the area covered by the current video stream in the whole panorama when the current video stream is finally imaged, wherein the angle range information comprises horizontal angle information and vertical angle information;
time stamp information: describing a time at which the current video stream was captured;
camera matrix information: the method comprises the following steps that external parameters and internal parameters are included, the external parameters comprise a rotation matrix and a translation matrix of a camera, the rotation matrix and the translation matrix describe how to convert points from a world coordinate system to the camera coordinate system together, the rotation matrix describes the directions of coordinate axes of the world coordinate system relative to the coordinate axes of the camera, and the translation matrix describes the position of a space origin under the camera coordinate system; the intrinsic parameters include focal length and the transformation of the imaging plane coordinate system to the pixel coordinate system.
5. The node-computing-based panoramic video transmission method according to claim 4, characterized in that the video stream transmitted by each camera is transmitted as an independent video stream.
6. A node-computation-based panoramic video transmission system, comprising an intermediate node, the intermediate node comprising:
an information acquisition module: obtaining feedback information of a user terminal in real time, wherein the feedback information at least comprises: user visual field information, user network condition information and user terminal processing capacity information;
splicing modules: calculating a user visual field region according to the user visual field information, and selecting a video stream containing the user visual field region for splicing to obtain a processed video stream;
a transmission decision module: selecting a transmission scheme according to the user network condition information, extracting a transmission area aiming at the processed video stream, and transmitting a transmission area code to a corresponding user terminal;
the transmission scheme comprises non-redundant transmission and redundant transmission, wherein in the non-redundant transmission, the transmission region is the user visual field region, in the redundant transmission, the transmission region is the user visual field region plus a redundant region, and the redundant region is obtained by calculation according to the user network condition information and the user terminal processing capacity information.
7. The node-computing based panoramic video transmission system of claim 6, wherein in the redundant transmission, the transmission decision module further comprises:
and carrying out unequal quality coding on the user visual field region and the redundant region and transmitting the unequal quality coding to a corresponding user terminal, namely reducing the coding quality of the redundant region.
8. The node-computing based panoramic video transmission system of claim 6, wherein the feedback information comprises at least:
user visual field information: describing the position information of the video content currently watched by the user;
user network status information: describing the transmission condition of the current network of the user, including bandwidth and delay;
user terminal processing capability information: the computing power of the user terminal is described, including throughput, response time and cpu occupancy.
9. The node-computing-based panoramic video transmission system of claim 6, wherein a video stream before splicing is transmitted to the intermediate node by a camera, the camera transmits assistance splicing information while transmitting the video stream, and the assistance splicing information comprises:
stream coverage information: describing the position of the area covered by the current video stream in the whole panorama when the current video stream is finally imaged, wherein the angle range information comprises horizontal angle information and vertical angle information;
time stamp information: describing a time at which the current video stream was captured;
camera matrix information: the method comprises the following steps that external parameters and internal parameters are included, the external parameters comprise a rotation matrix and a translation matrix of a camera, the rotation matrix and the translation matrix describe how to convert points from a world coordinate system to the camera coordinate system together, the rotation matrix describes the directions of coordinate axes of the world coordinate system relative to the coordinate axes of the camera, and the translation matrix describes the position of a space origin under the camera coordinate system; the intrinsic parameters include focal length and the transformation of the imaging plane coordinate system to the pixel coordinate system.
10. The node-computing-based panoramic video transmission system of claim 9, wherein the video stream transmitted by each camera is transmitted as a separate video stream.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810745690 | 2018-07-09 | ||
| CN2018107456901 | 2018-07-09 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110708562A true CN110708562A (en) | 2020-01-17 |
Family
ID=69192601
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810760053.1A Withdrawn CN110708566A (en) | 2018-07-09 | 2018-07-11 | Panoramic video transmission method and system based on node calculation |
| CN201811178241.XA Pending CN110708562A (en) | 2018-07-09 | 2018-10-10 | Panoramic video transmission method and system based on node calculation |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810760053.1A Withdrawn CN110708566A (en) | 2018-07-09 | 2018-07-11 | Panoramic video transmission method and system based on node calculation |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN110708566A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114827617A (en) * | 2022-06-27 | 2022-07-29 | 致讯科技(天津)有限公司 | Video coding and decoding method and system based on perception model |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN204090039U (en) * | 2014-10-16 | 2015-01-07 | 北京十方慧通科技有限公司 | Integration large scene panoramic video monitoring device |
| CN105915937A (en) * | 2016-05-10 | 2016-08-31 | 上海乐相科技有限公司 | Panoramic video playing method and device |
| CN106101847A (en) * | 2016-07-12 | 2016-11-09 | 三星电子（中国）研发中心 | The method and system of panoramic video alternating transmission |
| CN106130988A (en) * | 2016-06-30 | 2016-11-16 | 深圳市虚拟现实科技有限公司 | Remotely panorama phonotape and videotape real-time Transmission and player method |
| CN106604042A (en) * | 2016-12-22 | 2017-04-26 | Tcl集团股份有限公司 | Panorama webcasting system and panorama webcasting method based on cloud server |
| CN107040794A (en) * | 2017-04-26 | 2017-08-11 | 盯盯拍（深圳）技术股份有限公司 | Video broadcasting method, server, virtual reality device and panoramic virtual reality play system |
| US9824723B1 (en) * | 2015-08-27 | 2017-11-21 | Amazon Technologies, Inc. | Direction indicators for panoramic images |
-
2018
- 2018-07-11 CN CN201810760053.1A patent/CN110708566A/en not_active Withdrawn
- 2018-10-10 CN CN201811178241.XA patent/CN110708562A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN204090039U (en) * | 2014-10-16 | 2015-01-07 | 北京十方慧通科技有限公司 | Integration large scene panoramic video monitoring device |
| US9824723B1 (en) * | 2015-08-27 | 2017-11-21 | Amazon Technologies, Inc. | Direction indicators for panoramic images |
| CN105915937A (en) * | 2016-05-10 | 2016-08-31 | 上海乐相科技有限公司 | Panoramic video playing method and device |
| CN106130988A (en) * | 2016-06-30 | 2016-11-16 | 深圳市虚拟现实科技有限公司 | Remotely panorama phonotape and videotape real-time Transmission and player method |
| CN106101847A (en) * | 2016-07-12 | 2016-11-09 | 三星电子（中国）研发中心 | The method and system of panoramic video alternating transmission |
| CN106604042A (en) * | 2016-12-22 | 2017-04-26 | Tcl集团股份有限公司 | Panorama webcasting system and panorama webcasting method based on cloud server |
| CN107040794A (en) * | 2017-04-26 | 2017-08-11 | 盯盯拍（深圳）技术股份有限公司 | Video broadcasting method, server, virtual reality device and panoramic virtual reality play system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114827617A (en) * | 2022-06-27 | 2022-07-29 | 致讯科技(天津)有限公司 | Video coding and decoding method and system based on perception model |
| CN114827617B (en) * | 2022-06-27 | 2022-10-18 | 致讯科技(天津)有限公司 | Video coding and decoding method and system based on perception model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110708566A (en) | 2020-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11019259B2 (en) | Real-time generation method for 360-degree VR panoramic graphic image and video | |
| EP3896980A1 (en) | Special effect processing method for live broadcasting, same apparatus, and related server | |
| WO2021083176A1 (en) | Data interaction method and system, interaction terminal and readable storage medium | |
| US10116915B2 (en) | Cleaning of depth data by elimination of artifacts caused by shadows and parallax | |
| US10306254B2 (en) | Encoding free view point data in movie data container | |
| CN110675506B (en) | System, method and equipment for realizing three-dimensional augmented reality of multi-channel video fusion | |
| CN112738534B (en) | Data processing method and system, server and storage medium | |
| US11095871B2 (en) | System that generates virtual viewpoint image, method and storage medium | |
| CN112738495B (en) | Virtual viewpoint image generation method, system, electronic device and storage medium | |
| CN109218755B (en) | Media data processing method and device | |
| CN107071358B (en) | Video splicing server and panoramic live broadcast system in mobile state | |
| US20230421810A1 (en) | Encapsulation and decapsulation methods and apparatuses for point cloud media file, and storage medium | |
| WO2022022501A1 (en) | Video processing method, apparatus, electronic device, and storage medium | |
| KR101680545B1 (en) | Method and apparatus for providing panorama moving picture generation service | |
| CN110708562A (en) | Panoramic video transmission method and system based on node calculation | |
| Zheng et al. | Research on panoramic stereo live streaming based on the virtual reality | |
| WO2021032105A1 (en) | Code stream processing method and device, first terminal, second terminal and storage medium | |
| WO2021083175A1 (en) | Data processing method, device and system, readable storage medium and server | |
| CN112738009B (en) | Data synchronization method, device, synchronization system, medium and server | |
| WO2018120857A1 (en) | Streaming media technology-based method and apparatus for processing video data | |
| EP3470974A1 (en) | Selection of animated viewing angle in an immersive virtual environment | |
| WO2021083177A1 (en) | Method for generating depth map, computing nodes, computing node cluster, and storage medium | |
| WO2023024839A1 (en) | Media file encapsulation method and apparatus, media file decapsulation method and apparatus, device and storage medium | |
| EP4387238A1 (en) | File encapsulation method and apparatus for free-viewpoint video, device and storage medium | |
| CN113038262A (en) | Panoramic live broadcast method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200117 |