WO2019157803A1 - Transmission control method - Google Patents

Transmission control method Download PDF

Info

Publication number
WO2019157803A1
WO2019157803A1 PCT/CN2018/100670 CN2018100670W WO2019157803A1 WO 2019157803 A1 WO2019157803 A1 WO 2019157803A1 CN 2018100670 W CN2018100670 W CN 2018100670W WO 2019157803 A1 WO2019157803 A1 WO 2019157803A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
information
client
panoramic
user
Prior art date
Application number
PCT/CN2018/100670
Other languages
French (fr)
Chinese (zh)
Inventor
范宇群
邸佩云
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2019157803A1 publication Critical patent/WO2019157803A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
    • H04N21/2387—Stream processing in response to a playback request from an end-user, e.g. for trick-play
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/26616—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel for merging a unicast channel into a multicast channel, e.g. in a VOD application, when a client served by unicast channel catches up a multicast channel to save bandwidth
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
    • H04N21/4728—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video

Definitions

  • the present invention relates to the technical field of video playback, and in particular, to a method, device and system for VR panoramic video playback.
  • VR Virtual Reality
  • the panoramic video is composed of a series of panoramic images, and the panoramic image is usually photographed by multiple cameras at multiple angles at the same time, and then the stitching algorithm is used to splicing multiple angle images at the same time. Since the panoramic image shows the spherical content in three-dimensional space, and the image storage is stored in two-dimensional coordinates, the panoramic image usually needs to be converted into two-dimensional spatial coordinates by a certain projection method, such as latitude and longitude projection. Further storage.
  • the panoramic video allows multiple users to view the video from different angles. At any moment, each user's given perspective center point and range of view will cover a part of the video, which is called the user's viewing angle range (FOV, Field of View). The content of the panoramic video falling in this part of the FOV will be presented to the user.
  • FOV viewing angle range
  • the panoramic video can be transmitted from the server to the client through some media transmission technologies, such as DASH (Dynamic Adaptive Streaming through HTTP) or RTP (Real-Time Transmit Protocol). Since the client only displays the FOV area content to the user at any time, in order to ensure the viewing quality of the FOV area and reduce the amount of data transmitted by the network, the panoramic video transmission usually transmits the image of the FOV area with high quality and transmits other areas in a low quality manner. image.
  • DASH Dynamic Adaptive Streaming through HTTP
  • RTP Real-Time Transmit Protocol
  • the role of low-quality video in other areas is that when the viewing angle of the user changes, there is a delay in the response and transmission of the high-definition video, and the low-definition video is presented to the user before receiving the high-definition video, thereby avoiding interruption of the picture and affecting the immersion.
  • the server prepares the blocked panoramic video content and the related description file in advance, and the plurality of clients calculate the video content of the sub-blocks corresponding to the respective FOV regions according to the description file and the user's perspective, and request corresponding The data.
  • the server responds to the corresponding data according to the request of the client, and returns the HD video of the corresponding sub-block and the low-definition video of other areas to each client in a unicast manner.
  • the network bandwidth is limited.
  • the transmission bandwidth load is large, and problems such as network congestion and client data transmission delay are prone to occur, which ultimately leads to delay or interruption of the picture and affects the user experience.
  • the processing resources of the current VR client device are very limited. It is relatively expensive to calculate the high-definition video sub-block corresponding to the user's perspective. Especially when the video area is divided, the computing burden of the client is more obvious, which may lead to the client. The device was stuck and even crashed.
  • the present application provides a VR video transmission method and an apparatus using the same, which reduces the transmission bandwidth load by using a multicast method.
  • the calculation of the sub-block corresponding to the user FOV is performed on the server side, thereby saving the computing resources of the client.
  • the application provides a VR panoramic video playback client.
  • the client includes a main function module and a display module, wherein the main function module is configured to acquire and send user video information to the server, where the video information is used to determine video content corresponding to the user's perspective range area.
  • the main function module is further configured to receive the unicast first video code stream sent by the server, and decode the first video data stream to obtain the first video data.
  • the first video data is video content corresponding to the user view range area.
  • the main function module is further configured to receive the second video code stream of the multicast sent by the server, and decode the second video data, where the second video data includes the panoramic video content.
  • the main function module replaces the portion of the second video data corresponding to the video content in the FOV by the first video data, and obtains image information and sends the image information to the display module.
  • the display module receives and displays the image information, and presents the image information to the user.
  • the VR panoramic video playback client receives the panoramic video in a multicast manner, which saves the transmission bandwidth and reduces the bandwidth load. Meanwhile, the VR panoramic video playback client no longer bears the corresponding video of the FOV area. The content calculation process saves valuable client computing resources.
  • the main function module includes a user view information acquiring unit, a data sending unit, a first data receiving unit, a first data decoding unit, a second data receiving unit, a second data decoding unit, and an image processing unit.
  • the user view information acquiring unit is configured to collect user view information, and the user view information may be implemented as a user's visual center point information and visual coverage angle information.
  • the data sending unit is configured to send the user view information to the server, so that the server determines the first video data by using the user view information.
  • the first data receiving unit is configured to receive the unicast first video code stream sent by the server, and send the first video data stream to the first data decoding unit, where the second data receiving unit is configured to receive the multicast second video code stream sent by the server, and Send to the second data decoding unit.
  • the first data decoding unit is configured to decode the first video code stream to obtain first video data
  • the second data decoding unit is configured to decode the second video code stream to obtain second video data.
  • the first data receiving unit and the second data receiving unit may be implemented as one data receiving module, and the first data decoding unit and the second data decoding unit may also be implemented as one data decoding unit.
  • the image processing unit is configured to replace the video information corresponding to the video content in the FOV in the second video data to obtain an image that needs to be presented to the user.
  • the image quality of the first video data is higher than the image quality of the second video data.
  • the compression code rate of the first video data image is higher than the compression code rate of the second video data image, or the signal to noise ratio of the first video data image is higher than the signal to noise ratio of the second video data image.
  • the panoramic video content is divided into a plurality of sub-blocks, and the video content in the FOV refers to video content of a sub-block covered by the FOV.
  • the covered sub-block includes a sub-block to which only a partial area is covered by the FOV.
  • the user view information includes visual center point information and visual coverage angle information
  • the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user.
  • the range determines the first video data.
  • the present application provides a VR panoramic video playing server.
  • the server includes an image generation module, a data receiving module, a sub-block information calculation module, a first video data acquisition module, a first data transmission module, a second video data acquisition module, and a second data transmission module.
  • the image generation module is configured to generate VR panoramic video content, and the panoramic video content is divided into several sub-blocks.
  • the data receiving module is configured to receive user view information.
  • the first video data acquiring module is configured to extract a corresponding sub-block from the panoramic video according to the FOV corresponding sub-block information and encode the first video code stream.
  • the first data sending module is configured to send the first video code stream to the client in a unicast manner.
  • the second video data acquiring module is configured to extract a panoramic video and encode the second video code stream.
  • the second data sending module is configured to send the second video code stream to the client in a multicast form.
  • the VR panoramic video playing server transmits the panoramic video in the form of multicast, which saves the transmission bandwidth and reduces the bandwidth load. Meanwhile, the VR panoramic video playing server undertakes the calculation of the corresponding video content of the FOV area. The process saves valuable client computing resources.
  • the first data sending module and the second data sending module may be implemented as one module.
  • the first video data acquiring module and the second video data acquiring module may also be implemented as one module.
  • the image quality of the first video code stream is higher than the image quality of the second video code stream.
  • the compression code rate of the first video code stream image is higher than the compression code rate of the second video code stream image, or the signal to noise ratio of the first video code stream image is higher than the second video code stream image. Noise ratio.
  • the user view information includes visual center point information and visual coverage angle information
  • the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user.
  • the range and panoramic video blocking mode determines the FOV corresponding sub-block information.
  • the application provides a VR panoramic video playback client.
  • the client includes a sensor, a processor, a communication port, and a display.
  • the sensor is configured to collect user view information and send it to the processor.
  • the processor is configured to send the user view information through the port.
  • the user view information is used to determine first video data, and the first video data corresponds to video content within a user view range area FOV.
  • the client is further configured to receive, by using the communication port, the unicast first video code stream and the multicast second video code stream sent by the server, and respectively decode the two to obtain the first video data and the second Video data.
  • the first video data is replaced with a portion of the second video data corresponding to the video content in the FOV, and image information is obtained and sent to the display.
  • the communication port is for transmitting and receiving information
  • the display is for displaying the image information.
  • the image quality of the first video data is higher than the image quality of the second video data.
  • the compression code rate of the first video data image is higher than the compression code rate of the second video data image, or the signal to noise ratio of the first video data image is higher than the signal to noise ratio of the second video data image.
  • the user view information includes visual center point information and visual coverage angle information
  • the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user.
  • the range determines the first video data.
  • the panoramic video content is divided into a plurality of sub-blocks, and the video content in the FOV refers to video content of a sub-block covered by the FOV.
  • the covered sub-block includes a sub-block to which only a partial area is covered by the FOV.
  • the application provides a VR panoramic video playing server.
  • the server includes a communication port and a processor.
  • the communication port is configured to send and receive information to and communicate with the client.
  • the processor is configured to generate VR panoramic video content that is divided into sub-blocks.
  • the processor receives the user view information sent by the client through the communication port, and determines the video sub-block covered by the user view in combination with the block mode of the panoramic video, that is, determines the FOV corresponding sub-block information.
  • the processor extracts a corresponding sub-block from the panoramic video according to the FOV corresponding sub-block information and encodes it into a first video code stream, and extracts the panoramic video information and encodes the second video code stream.
  • the processor sends the first video code stream to the client in a unicast manner through the communication port, and sends the second video code stream to the client in a multicast manner.
  • the VR panoramic video playing server transmits the panoramic video in the form of multicast, which saves the transmission bandwidth and reduces the bandwidth load.
  • the VR panoramic video playing server undertakes the calculation of the corresponding video content of the FOV area. The process saves valuable client computing resources.
  • the image quality of the first video code stream is higher than the image quality of the second video code stream.
  • the compression code rate of the first video code stream image is higher than the compression code rate of the second video code stream image, or the signal to noise ratio of the first video code stream image is higher than the second video code stream image. Noise ratio.
  • the user view information includes visual center point information and visual coverage angle information
  • the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user.
  • the range and panoramic video blocking mode determines the FOV corresponding sub-block information.
  • the present application provides a VR panoramic video playing method.
  • the client collects and sends user view information to the server, and the user view information is used to determine video content within the user's view range area FOV, that is, the first video data.
  • the client receives and decodes the unicast first video code stream to obtain the first video data, and receives and decodes the multicast second video code stream to obtain second video data including the panoramic video content.
  • the client replaces a portion of the second video data corresponding to the video content in the FOV with the first video data, obtains image information, and displays the image information to the user.
  • the panoramic video is transmitted between the client and the server in a multicast manner, which saves the transmission bandwidth and reduces the bandwidth load; meanwhile, the calculation of the video content corresponding to the FOV region is no longer performed by The VR panoramic video playback client no longer bears, saving valuable client computing resources.
  • the image quality of the first video data is higher than the image quality of the second video data.
  • the compression code rate of the first video data image is higher than the compression code rate of the second video data image, or the signal to noise ratio of the first video data image is higher than the signal to noise ratio of the second video data image.
  • the panoramic video content is divided into a plurality of sub-blocks, and the video content in the FOV refers to video content of a sub-block covered by the FOV.
  • the covered sub-block includes a sub-block to which only a partial area is covered by the FOV.
  • the user view information includes visual center point information and visual coverage angle information
  • the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user.
  • the range determines the first video data.
  • the application provides a VR panoramic video playing method.
  • the server generates chunked VR panoramic video content.
  • the server receives the user view information sent by the client, and determines the corresponding sub-block covered by the user view, that is, the FOV corresponding sub-block information, in combination with the block mode of the panoramic video.
  • the server extracts the corresponding sub-block from the panoramic video according to the FOV corresponding sub-block information, encodes the first video code stream and sends the information in a unicast manner; extracts the panoramic video information, encodes the second video code stream, and uses the multicast
  • the form is sent.
  • the client receives, decodes, and plays content in the first video code stream and the second video code stream.
  • the image quality of the first video code stream is higher than the image quality of the second video code stream.
  • the compression code rate of the first video code stream image is higher than the compression code rate of the second video code stream image, or the signal to noise ratio of the first video code stream image is higher than the second video code stream image. Noise ratio.
  • the user view information includes visual center point information and visual coverage angle information
  • the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user.
  • the range and panoramic video blocking mode determines the FOV corresponding sub-block information.
  • FIG. 1 is a schematic diagram of device interaction of a VR panoramic video playing system
  • FIG. 2 is a schematic diagram of an interaction process of a VR panoramic video playing system
  • FIG. 3 is a schematic diagram of device interaction of another VR panoramic video playing system
  • FIG. 4 is a schematic flow chart of a VR panoramic video playing method
  • FIG. 5 is a schematic diagram of a logical structure of a VR panoramic video playing client
  • FIG. 6 is a schematic diagram of a logical structure of a VR panoramic video playing server
  • FIG. 7 is a schematic diagram of a hardware structure of a VR panoramic video playing client
  • FIG. 8 is a schematic diagram of a hardware structure of a VR panoramic video playing server.
  • the VR (Virtual Reality) video transmission method involved in the present application is mainly applied to VR panoramic video playback.
  • the VR panoramic video is mainly a 360 degree or 180 degree panoramic video.
  • the data transmission between the client and the server usually adopts the RTP (Real-Time Transmit Protocol) communication protocol.
  • RTP Real-Time Transmit Protocol
  • the number of the client-side client devices in the embodiment of the present application may be any number.
  • the specific number of the client in the embodiment is only for convenience of description, and does not constitute a limitation on the application scenario of the present application.
  • the FOV video code stream can be understood as the first video code stream described in the claims, wherein the included video data can be understood as the first video described in the claims.
  • Data; a panoramic video stream can be understood as a second video stream as described in the claims, and the video data contained therein can be understood as the second video data recited in the claims.
  • the video data image quality in the FOV video stream is higher than the video data image quality in the panoramic video stream.
  • the server side uses a higher compression code rate when generating video data of the FOV video stream than the video data in the panoramic video stream, or the signal-to-noise ratio of the video data in the FOV video stream is higher.
  • the signal-to-noise ratio of the video data in the panoramic video stream is higher.
  • the composition of the VR panoramic video playback system in the prior art is as shown in FIG. 1.
  • the VR panoramic video playback system 100 includes a server side 110, a client side 120, and a network device side 130, wherein the server side 110 can be any device that provides media services, such as a server group, a computer, or even a mobile phone; the client side 120 can be a device.
  • the number of users that can view the same panoramic video content at the same time in this embodiment, the two devices of the client 121 and the client 122; the network device side 130 can be implemented as a router or a switch.
  • the devices on the client side communicate with the server side through the network side device.
  • the server side 110 sends the client side to the client side 120 through the network device side 130.
  • the client 121 and the client 122 respectively send request data to the network side 130, where the request data includes the FOV corresponding sub-block information of the client, and the FOV corresponding sub-block information is segmented by the client according to the panoramic video. Information and local user perspective information are calculated.
  • the network side 130 transmits the request data to the server side 110. After receiving the request data, the server side 110 returns response data to the network side 130, which are two FOV video code streams and two panoramic video code streams.
  • the network side 130 sends the two FOV video streams to the client 121 and the client 122, and sends the two panoramic video streams to the client 121 and the client 122, respectively. After receiving the response data, the two clients display the video content contained therein and present it to the user.
  • the communication interaction of the system is completed at three levels, namely, the server side 110, the client side 120, and the network device side 130.
  • the client side 120 can be a device or a plurality of devices, and the number of users depends on the number of users who simultaneously view the same panoramic video content.
  • the client 121 and the client 122 are used.
  • the server side 110 generates the chunked panoramic video content 111 and then transmits the video chunking information to the client side 120 through the network device side 130.
  • the panoramic video content 111 in this example is divided into 1 to 8 sub-video blocks of the same size.
  • the tiled manner of the panoramic video content 111 can be any size and number of rectangles.
  • the client 121 of the client side 120 sends the request information 1 to the network device side, where the FOV corresponding sub-block information of the client 121 is included, and the FOV corresponding sub-block information is determined by the client according to the video segmentation information and the current user view information.
  • the current view range of the client 121 covers the videos of which sub-blocks, and in this embodiment, the sub-blocks 2, 3, 6, and 7.
  • the client 122 sends the request information 2 to the network device side, where the FOV corresponding sub-block information of the client 122 is included, and the FOV corresponding sub-block information describes the video of the specific viewing angle range of the client 122 covering the specific sub-blocks.
  • the embodiment are sub-blocks 1, 2, 5 and 6.
  • the request information 1 and the request information 2 are transmitted to the server side 110 through the network device side.
  • the server side 110 then sends the four video streams in the unicast form to the client side 120 through the network device side 130 according to the request information 1 and the request information 2, which are respectively the FOV video code stream 1 sent by the client 121.
  • the content is FOV video data 1, that is, the video sub-block content requested by the client 121; the panoramic video stream 1 is the panoramic video data 112; and the FOV video stream 2 sent to the client 122 is the FOV video.
  • the client 121 After receiving the FOV video stream 1 and the panoramic video stream 1, the client 121 combines the video content in the two, and the sub-block video content in the sub-blocks 2, 3, 6, and 7.
  • the client video is prone to jams and the like.
  • the client FOV corresponding sub-block information is calculated by the client according to the video description information and the current user user perspective information. Frequently calculating the FOV corresponding sub-block information will occupy a large amount of computing resources of the client, especially when the video partitioning is complicated.
  • the bandwidth between the network side device and the server side is tight.
  • FIG. 3 is an embodiment of the present invention, and is a VR panoramic video playback system 200.
  • the VR panoramic video playback system 200 includes a server side 210, a client side 220, and a network device side 230, wherein the server side 210 can be any device that provides media services, such as a server group, a computer, or even a mobile phone.
  • the client side 220 can be a device or a plurality of devices, and the number of users depends on the number of users who simultaneously view the same panoramic video content.
  • the client 221 and the client 222 are two devices. It should be noted that the number of clients in this embodiment is only for convenience of description, and is not limited to the present application, and the number of clients may be any number.
  • Network device side 230 can be implemented as a router or switch that supports multicast functionality.
  • the devices on each client side communicate with the server side through the network side device.
  • the server side 210 first generates the blocked panoramic video content, and the specific details of the divided panoramic video are referred to the related description in FIG. 2, and details are not described herein again.
  • the client 221 and the client 222 respectively send request data to the server side 210 through the network side 230, where the data includes user view information of the current user of the client, and the user view information describes the current view coverage of the client. Contains visual center point information and visual coverage angle information to determine the specific coverage of the client's user perspective.
  • the server side 210 receives the request data, and determines the FOV corresponding sub-block information of the client 221 through the panoramic video blocking information and the user perspective information of the client 221 in the request data, and the panoramic video blocking information and the request.
  • the user view information of the client 222 in the data determines the FOV corresponding sub-block information of the client 222.
  • the server side 210 extracts corresponding sub-blocks from the blocked panoramic video content according to the FOV corresponding sub-block information of the client 221 and the client 222, respectively, to generate the FOV video stream 1 and the FOV video stream 2, the FOV.
  • the video stream 1 contains the sub-block video required by the client 221, and the FOV video stream 2 contains the sub-block video required by the client 222. Then, the server side 210 returns the response data to the network side 230 in a combination of unicast and multicast, and the FOV video code stream 1 sent to the client 221 in unicast mode is sent to the client 222 in unicast form.
  • the FOV video stream 2 and the panoramic video stream sent to all clients in multicast form.
  • the panoramic video stream includes the entire panoramic video content.
  • the network side 230 sends the FOV video code stream in the response data to the corresponding client, that is, the FOV video stream 1 is sent to the client 221, and the FOV video stream 2 is sent to the client 222, and Two clients send a panoramic video stream.
  • the client side 220 receives the response data. Specifically, the client 221 receives the FOV video stream 1 and the multicast panoramic video stream; the client 222 receives the FOV video stream 2 and the group. Broadcast panoramic video stream. After receiving the response data, the client 221 and the client 222 respectively display the video content contained therein. For the specific presentation manner, refer to the detailed description in FIG. 2, and details are not described herein again.
  • FIG. 4 it is another embodiment of the present invention, which is an information transmission method S300 in a VR panoramic video playing system, and specifically includes:
  • the client side sends request data to the server side, where the request data includes current user perspective information.
  • the client side client obtains current user perspective information, and the user perspective information may include visual center point information and visual coverage angle information. Visual center point information and visual coverage angle information are used to determine the specific coverage of the client's perspective.
  • the client side sends request data to the server side through the network device side.
  • the server side receives the request data, and determines FOV corresponding sub-block information according to the current user view information in the request data.
  • the server side determines, by using the current user view information and the panoramic video block mode, a specific video sub-block that should be displayed in the coverage of the user's perspective, that is, FOV corresponding sub-block information.
  • FOV corresponding sub-block information please refer to the description in the related content of FIG.
  • the server side sends response information to the client side, where the response information includes a unicast FOV video code stream and a multicast panoramic video code stream.
  • the server side sends the FOV video code stream to each client on the client side in a unicast manner, where each FOV video code stream includes a corresponding FOV sub-block video required by each client;
  • the server side transmits the panoramic video code stream to each client on the client side in a multicast manner, and the panoramic video code stream includes the panoramic video content.
  • the client side receives the response data, that is, each client separately receives the FOV video code stream and the panoramic video code stream sent to the local machine.
  • the client receives the FOV video stream sent to the local machine, parses and acquires the sub-block video therein, and also receives the panoramic video code stream sent to the local machine, parses and acquires the panoramic video therein.
  • the information transmission method 300 in the VR panoramic video playing system can be implemented in the RTSP protocol, as follows:
  • the client sends user perspective information to the server. Specifically, the client may send request data to the server, where the request data includes the user view information.
  • the request data may be an RTCP source description report.
  • the RTCP source description report may be sent according to a response sent by the server according to a connection establishment command sent by the client.
  • the establishing a connection command is used to establish a session connection for transmitting a video code stream of a sub-area, where the session is connected to a code stream for unicast sub-area video, and the sub-area may be a tile obtained by dividing the panoramic video.
  • the client may also send a session connection for establishing a code stream for transmitting the panoramic video, the session connecting the code stream for the multicast panoramic video.
  • the image quality corresponding to the code stream of the panoramic video may be lower than the image quality corresponding to the code stream of the sub-region video, wherein the image quality may be indicated by a code rate or a resolution.
  • the sub-area is a sub-area in the panoramic video area, and the content of the sub-area video is a part of the content in the panoramic video.
  • the content of the panoramic video can be obtained by splicing the content of the sub-area video, but only the code of the sub-area video.
  • the image corresponding to the stream is of higher quality.
  • the user view information may include at least one of location information of the user's perspective and size information of the user's perspective.
  • the location information of the user's perspective may be location information of a center point of the user's perspective, or an upper left corner of the user's perspective.
  • the position information, wherein the position can respectively correspond to the yaw, pitch, and roll values of the VR spherical coordinates, or only two (X and Y) corresponding to the central point two-dimensional coordinate value of the viewing angle region or the upper left corner coordinate value of the viewing angle region.
  • the size information of the user's perspective may be the coverage angle information of the user's perspective, specifically, the height and width of the user's perspective, wherein the width and height may be the horizontal coverage angle and the vertical coverage angle of the VR spherical coordinates, It can be the width and height pixel values of a two-dimensional image.
  • the server prepares the video content as shown in Figure 2, which includes low quality panoramic video content and high quality sub-regional video content, and publishes the address of the content.
  • the address is in the RTSP protocol format as follows: rtsp://server.example.com/video
  • the client sends a description command to the above address, and the server responds to the description command, and the response can also describe the video content, wherein the session information of the code stream for the high quality sub-area video for unicast needs to be described and used for Session information for the stream of multicast low quality panoramic video.
  • the description command sent by the client can be as follows:
  • the DESCRIBE field in the above table represents the description command.
  • the server's response to the description command can be as follows:
  • a total of two video sessions are described in the SDP description file of the above response species. All the way is the video stream of the tiles, the code rate is 5000kbps. All the way is a panoramic video stream with a code rate of 1000 kbps. Both sessions are transmitted using the RTP protocol.
  • the client establishes two-way session connection with the server according to the above response, wherein one session connection is used for unicast sub-area code stream, and one session connection is used for multicast panoramic code stream.
  • the specific process can be as follows:
  • the client can send the following two connection establishment commands:
  • the SETUP field in the above command indicates that the command is a connection establishment command.
  • the first command indicates that a connection is established with track1 in the description of the SDP (Session Description Protocol) in the response, wherein the Transport field indicates that the sub-area video code stream is transmitted in a unicast manner through the RTP protocol for receiving the data stream.
  • the RTP port number is 20000, and the RTCP port number used to receive the control flow is 20001.
  • the second command indicates that a connection is established with track2 in the SDP description in the response.
  • the Transport field indicates that the panoramic video stream is transmitted in multicast mode through the RTP protocol, and the RTP port number used to receive the data stream is 20010, which is used for receiving control.
  • the RTCP port number of the stream is 20011.
  • the server can respond to the connection establishment command sent by the client as follows:
  • the above response is a response to the first command of the client, indicating that the client is configured to establish a connection command for unicast, wherein the Transport field indicates that the unicast address is 10.70.144.123, and the RTP receiving port of the client is 20000, and the RTCP receiving port For 20001, the RTP receiving port on the server side is 50000, and the RTCP receiving port is 50001.
  • the session number of the connection is 12345678.
  • the above response is a response from the client's second command, indicating that the client's connection establishment command for multicast is accepted.
  • the Transport field indicates that the multicast address is 238.225.225.225, the RTP receiving port of the client is 20010, and the RTCP receiving port is In 20011, the RTP receiving port on the server side is 50010, and the RTCP receiving port is 50011.
  • the session number of the connection is 22345678.
  • the above content is a process for establishing a session connection disclosed in the implementation of the present invention. It should be noted that, based on the establishment method of the session connection, the client may not send the user view information like the server, and the client may view the user according to the user. Information, determine the sub-area, and apply to the server for the corresponding sub-area code stream.
  • the RTCP source description report format for describing the coverage angle information of the user perspective of the client may be as follows:
  • the COVERAGE value is 9 (the implementation of the present invention is not limited to other fixed values), which may indicate that the RTCP source description report describes the coverage angle information of the user's user perspective, and H Angle represents the horizontal coverage angle (may be called The width of the user's perspective), V Angle represents the vertical coverage angle (which may be referred to as the height of the user's perspective).
  • An example of an RTCP source description report for describing a user's perspective of the client with a horizontal coverage angle of 110 degrees and a vertical coverage angle of 90 degrees may be as follows:
  • the format of the RTCP source description report used to describe the location information of the user's perspective can be as follows:
  • the COVERAGE value is 10 (not limited to other fixed values in the implementation of the present invention), which may indicate that the RTCP source description report describes the central point position information of the user's perspective, and X, Y, and Z respectively represent the yaw of the center point. Pitch, roll value.
  • An example of an RTCP source description report with a center point at -45, -45, 0 for describing the user's perspective can be as follows:
  • the server receives user view information.
  • the server sends a response message to the client, and the response message may include a video code stream of the sub-area.
  • the response information may further include a code stream of the panoramic video.
  • the video code stream of the sub-area may be a code stream of the sub-area video for unicast
  • the code stream of the panoramic video may be a code stream of the panoramic video for multicast.
  • the video stream of the high quality sub-area and the code stream of the low-quality panoramic video may be separately sent to the client through the two-way session connection.
  • the code stream may be carried by the RTP protocol packet, where the video code stream of the sub-area needs to carry the center point coordinates of the sub-area (one or more) and the horizontal and vertical coverage angles.
  • the sub-region corresponding to the video code stream of the sub-area is calculated by the server according to the user view information and the sub-area division manner.
  • the user view information is used to determine an area corresponding to the user's perspective, and the area and the sub-area corresponding to the user's perspective are combined (the division mode indicates how the area of the panoramic video is divided to obtain multiple sub-areas), and may be determined for sending to the client.
  • the sub-area corresponding to the video code stream of the sub-area is used to send the sub-area corresponding to the video code stream of the sub-area of the client (if the sub-area is at least two, the sub-area needs to be spliced to obtain a new sub-area) Covers the area corresponding to the user's perspective.
  • the sub-area division mode is divided into 4x2 and 8 sub-areas as shown in FIG. 2, the sub-area to be transmitted can be calculated. Includes a total of 4 sub-areas of 2, 3, 6, and 7.
  • the RTP header format used to carry the code stream can be as follows:
  • the RTP header can be added with the following extended format:
  • an RTP header with a sub-area video data with a center point at (-45, -45, 0), a horizontal coverage angle of 90 degrees, and a vertical coverage angle of 90 degrees can be as follows:
  • the transmission method 300 may further include: the client plays a video of an area corresponding to a user perspective in a video corresponding to the sub-area video code stream, based on the sub-area video code stream in the response information.
  • the client may also play the video corresponding to the sub-region video stream based on the sub-area video stream in the response message.
  • the client may also play the area corresponding to the sub-area video stream corresponding to the sub-area video stream corresponding to the sub-area video stream and the panoramic video stream based on the sub-area video stream and the panoramic video stream in the response information. Video outside the area.
  • the transmission method 300 may further include:
  • the client sends a new RTCP source description report to the server, updating the user perspective information (similar to step 1).
  • the server calculates the sub-area based on the new user view information and sends a new sub-area video stream to the client.
  • the client 400 may include two parts: a main function module 410 and a display module 420.
  • the main function module 410 includes a user view information acquiring unit 411, a data transmitting unit 412, a first data receiving unit 413, a first data decoding unit 414, a second data receiving unit 415, a second data decoding unit 416, and an image processing unit. 417.
  • the user view information obtaining unit 411 is configured to collect current user view information.
  • the user perspective information describes a current perspective coverage of the client, and may include visual center point information and visual coverage angle information.
  • the user view information unit 411 then transmits the user view information to the data transmitting unit 412.
  • the data sending unit 412 is configured to generate and send request information, where the request information includes the user perspective information.
  • the first data receiving unit 413 is configured to receive the unicast FOV video code stream and transmit it to the first data decoding unit 414.
  • the first data decoding unit 414 is configured to decode the received FOV video code stream, and send the decoded sub-block video information to the image processing unit 417.
  • the second data receiving unit 415 is configured to receive the multicast panoramic video code stream and transmit it to the second data decoding unit 416.
  • the panoramic video code stream includes panoramic video information.
  • the data decoding unit 416 is configured to decode the received panoramic video code stream, and send the decoded panoramic video information to the image processing unit 417.
  • the image processing unit 417 is configured to combine the content in the received sub-block video information with the content in the panoramic video information to obtain the image information. For the specific combination, refer to the description in the related content in FIG. 1 , and details are not described herein again. Thereafter, the image processing unit 417 transmits the image information to the display module 420.
  • the display module 420 is configured to display the received image information and output an image.
  • FIG. 6 another embodiment of the present invention is a VR panoramic video playing server 500, which is used to implement a method performed by a server in the foregoing method.
  • the server 500 may include an image generation module 510, a data receiving module 520, a sub-block information calculation module 530, a first video data acquisition module 540, a first data transmission module 550, a second video data acquisition module 560, and a second data transmission module 570. .
  • the image generation module 510 is configured to generate chunked panoramic video content.
  • the tiled panoramic video can be divided into rectangular sub-blocks of any size and number.
  • the image generation module also transmits panoramic video segmentation information to the sub-block information calculation module 530, which describes a specific blocking mode of the panoramic video.
  • the data receiving module 520 is configured to receive request information, where the request information includes user perspective information.
  • the user perspective information describes a current perspective coverage of the client, and may include visual center point information and visual coverage angle information.
  • the data receiving module 520 sends the user view information in the request information to the sub-block information calculation module 530.
  • the sub-block information calculation module 530 is configured to calculate FOV corresponding sub-block information according to the received user view information and the panoramic video block information.
  • the FOV corresponding sub-block information describes which sub-block videos are specifically needed by the client that sends the request information.
  • the sub-block information calculation module 530 then sends the FOV corresponding sub-block information to the first video data acquisition module 540.
  • the first video data obtaining module 540 is configured to extract, according to the FOV corresponding sub-block information, the video information of the corresponding sub-block from the block-divided panoramic video, and encode the video information into an FOV video code stream and send the data to the first data sending module 550.
  • the first data sending module 550 is configured to send the received FOV video code stream to the client in a unicast manner.
  • the second video data acquiring module 560 is configured to extract the panoramic video information, encode the panoramic video code stream, and send the data to the second data sending module 570.
  • the second data sending module 570 is configured to send the received panoramic video code stream to the client in a multicast manner.
  • FIG. 7 another embodiment of the present invention is a VR panoramic video playback client 600, which is used to implement a method performed by a client in the foregoing method.
  • Client 600 includes sensor 610, communication port 620, processor 630, and display 640.
  • the sensor 610 is used to collect current user perspective information.
  • the user perspective information describes a current perspective coverage of the client, and may include visual center point information and visual coverage angle information. Sensor 610 then transmits the user view information to processor 630.
  • the communication port 620 is configured to send and receive information to and from the server, and specifically, to send request information and receive a video code stream.
  • the processor 630 is configured to generate and send request information through the communication port 620, where the request information includes the user view information.
  • the processor 630 is further configured to receive the multicast panoramic video code stream and the FOV video code stream through the communication port 620, where the panoramic video code stream includes panoramic video information, and the FOV video code stream includes sub-block video information.
  • the processor 630 separately decodes the two video code streams to obtain the panoramic video data and the sub-block video data, and combines the sub-block video data with the panoramic video data to obtain the image information. For the specific combination manner, refer to the related content in FIG. Description, no longer repeat here.
  • Processor 630 then transmits the image information to display 640.
  • the display 640 is for displaying the received image information and outputting an image.
  • FIG. 8 is a VR panoramic video playing server 700 for implementing the method performed by the server in the foregoing method.
  • Server 700 includes a communication port 810 and a processor 820.
  • the communication port 810 is configured to send and receive information to and from the client, and specifically, to receive the request information and send the video code stream.
  • the processor 820 is configured to generate chunked panoramic video content.
  • the tiled panoramic video can be divided into rectangular sub-blocks of any size and number.
  • the processor 820 receives the request information sent by the client through the communication port 810, where the request information includes user perspective information.
  • the user perspective information describes a current perspective coverage of the client, and may include visual center point information and visual coverage angle information.
  • the processor 820 calculates FOV corresponding sub-block information according to the user view information and the panoramic video block information, and the FOV corresponding sub-block information describes which sub-block videos are specifically required by the client that sends the request information.
  • the video information of the corresponding sub-block is extracted according to the FOV corresponding sub-block information and encoded into a FOV video code stream, and the panoramic video code is extracted into a panoramic video code stream, and sent to the client in a unicast manner through the communication port.
  • the FOV video code stream transmits the panoramic video code stream to a client in a multicast manner.
  • the high quality tiled video has a higher code rate than the low quality panoramic video when the resolution is the same as the high quality tiled video and the low quality panoramic video.
  • the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted as one or more instructions or code via a computer-readable medium and executed by a hardware-based processing unit.
  • the computer readable medium can comprise a computer readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium comprising, for example, any medium that facilitates transfer of the computer program from one place to another in accordance with a communication protocol. .
  • computer readable media generally may correspond to (1) a non-transitory tangible computer readable storage medium, or (2) a communication medium such as a signal or carrier wave.
  • Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for use in carrying out the techniques described herein.
  • the computer program product can comprise a computer readable medium.
  • processors such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits
  • DSPs digital signal processors
  • ASICs application specific integrated circuits
  • FPGAs field programmable logic arrays
  • processors may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein.
  • B corresponding to A means that B is associated with A, and B can be determined from A.
  • determining B from A does not mean that B is only determined based on A, and that B can also be determined based on A and/or other information.
  • the disclosed systems, devices, and methods may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Human Computer Interaction (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

Provided in the present application are a method, device and system for playing back a VR panoramic video. The method comprises: a server generates a high-quality block-segmented video and a low-quality panoramic video; a client acquires user viewpoint information, and adds the viewpoint information to request information to send to the server; the server calculates field of view (FOV) corresponding sub-block information of a user according to the viewpoint information, determines a high-quality video sub-block required by the client according to the FOV corresponding sub-block information, sends the high-quality video sub-block to the client in a unicast mode and sends the low-quality panoramic video to the client in a multicast manner; and the client combines the received video information and plays back the same to the user.

Description

传输控制方法Transmission control method
本申请要求于2018年02月13日提交中国专利局、申请号为201810148373.1、申请名称为“传输控制方法”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。The present application claims priority to Chinese Patent Application No. 201101148373.1, filed on Jan.
技术领域Technical field
本发明涉及一种视频播放的技术领域，特别是涉及一种VR全景视频播放的方法、设备和***。The present invention relates to the technical field of video playback, and in particular, to a method, device and system for VR panoramic video playback.
背景技术Background technique
随着数字图像技术的发展，全景视频的VR(Virtual Reality,虚拟现实)技术以真实感强，提供全景浸入式体验等特点得到了日益广泛的关注。With the development of digital image technology, VR (Virtual Reality) technology of panoramic video has gained more and more attention with its strong sense of reality and the ability to provide panoramic immersion experience.
全景视频由一系列全景图像组成，而全景图像通常由多个相机同时对多个角度进行拍摄，进而通过拼接算法将同一时间的多个角度图像拼接而成。由于全景图像展现的是三维空间的球面内容，而图像存储是以二维坐标的方式存储，所以全景图像通常需要通过一定的投影方式，如经纬图投影，将三维空间坐标转换为二维空间坐标进而进行存储。全景视频允许多个用户从不同角度去观看视频，在任意时刻，每一个用户的给定视角中心点和视角范围都会覆盖视频中的一部分区域，该区域被称为该用户视角范围区域(FOV,Field of View)。全景视频落在FOV这部分区域的内容将被呈现给用户。The panoramic video is composed of a series of panoramic images, and the panoramic image is usually photographed by multiple cameras at multiple angles at the same time, and then the stitching algorithm is used to splicing multiple angle images at the same time. Since the panoramic image shows the spherical content in three-dimensional space, and the image storage is stored in two-dimensional coordinates, the panoramic image usually needs to be converted into two-dimensional spatial coordinates by a certain projection method, such as latitude and longitude projection. Further storage. The panoramic video allows multiple users to view the video from different angles. At any moment, each user's given perspective center point and range of view will cover a part of the video, which is called the user's viewing angle range (FOV, Field of View). The content of the panoramic video falling in this part of the FOV will be presented to the user.
全景视频可通过一些媒体传输技术由服务器端传输至客户端，如DASH(Dynamic Adaptive Streaming through HTTP)或者是RTP(Real-Time Transmit Protocol)。因为任意时刻客户端仅展现FOV区域内容给用户，为保障FOV区域的观看质量，同时减少网络传输数据量，全景视频传输通常以高质量传输FOV区域的图像，以低质量的方式传输其它区域的图像。其中，其他区域低质量视频的作用在于，当用户的观看视角改变时，由于高清视频响应和传输会存在延迟，在接收高清视频之前先以低清视频呈现给用户，避免画面中断，影响沉浸感。在现有技术中，服务器会预先准备好分块的全景视频内容以及相关的描述文件，多个客户端会根据描述文件和用户的视角计算各自FOV区域对应哪些子块的视频内容，并请求对应的数据。服务器根据客户端的请求应答相应的数据，以单播的形式分别向每一个客户端返回对应子块的高清视频和其他区域的低清视频。The panoramic video can be transmitted from the server to the client through some media transmission technologies, such as DASH (Dynamic Adaptive Streaming through HTTP) or RTP (Real-Time Transmit Protocol). Since the client only displays the FOV area content to the user at any time, in order to ensure the viewing quality of the FOV area and reduce the amount of data transmitted by the network, the panoramic video transmission usually transmits the image of the FOV area with high quality and transmits other areas in a low quality manner. image. Among them, the role of low-quality video in other areas is that when the viewing angle of the user changes, there is a delay in the response and transmission of the high-definition video, and the low-definition video is presented to the user before receiving the high-definition video, thereby avoiding interruption of the picture and affecting the immersion. . In the prior art, the server prepares the blocked panoramic video content and the related description file in advance, and the plurality of clients calculate the video content of the sub-blocks corresponding to the respective FOV regions according to the description file and the user's perspective, and request corresponding The data. The server responds to the corresponding data according to the request of the client, and returns the HD video of the corresponding sub-block and the low-definition video of other areas to each client in a unicast manner.
然而，网络带宽有限，当较多用户同时观看同一VR全景视频时，传输带宽负荷较大，容易出现网络拥挤、客户端数据传输延迟等问题，最终导致画面延迟或中断，影响用户体验。此外，目前的VR客户端设备处理资源非常有限，频繁计算用户视角对应的高清视频子块比较耗费资源，尤其是当视频区域划分比较复杂时，客户端运算负担更为明显，可能会导致客户端设备卡顿甚至死机。However, the network bandwidth is limited. When more users watch the same VR panoramic video at the same time, the transmission bandwidth load is large, and problems such as network congestion and client data transmission delay are prone to occur, which ultimately leads to delay or interruption of the picture and affects the user experience. In addition, the processing resources of the current VR client device are very limited. It is relatively expensive to calculate the high-definition video sub-block corresponding to the user's perspective. Especially when the video area is divided, the computing burden of the client is more obvious, which may lead to the client. The device was stuck and even crashed.
发明内容Summary of the invention
本申请提供了一种VR视频传输方法以及应用该方法的装置，通过运用组播手段减少传输带宽负荷。此外，还将用户FOV对应子块的计算在服务器端进行，节省客户端的运算资源。The present application provides a VR video transmission method and an apparatus using the same, which reduces the transmission bandwidth load by using a multicast method. In addition, the calculation of the sub-block corresponding to the user FOV is performed on the server side, thereby saving the computing resources of the client.
第一方面，本申请提供一种VR全景视频播放客户端。所述客户端包括主功能模块和显示模块，其中，主功能模块用于获取并向服务器发送用户视频信息，所述视频信息被用来确定用户视角范围区域对应的视频内容。主功能模块还用于接收所述服务器发送的单播的第一视频码流，将其解码得到第一视频数据。所述第一视频数据为所述用户视角范围区域对应的视 频内容。此外，主功能模块还用于接收所述服务器发送的组播的第二视频码流，解码得到第二视频数据，所述第二视频数据包括全景视频内容。主功能模块将所述第一视频数据替代所述第二视频数据中与FOV内视频内容对应的部分，得到图像信息并发送给显示模块。所述显示模块接收并显示所述图像信息，将图像信息呈现给用户。与现有技术相比，所述VR全景视频播放客户端以组播形式接收全景视频，节省了传输带宽，降低了带宽负荷；同时，所述VR全景视频播放客户端不再承担FOV区域对应视频内容的计算过程，节省了宝贵的客户端运算资源。In a first aspect, the application provides a VR panoramic video playback client. The client includes a main function module and a display module, wherein the main function module is configured to acquire and send user video information to the server, where the video information is used to determine video content corresponding to the user's perspective range area. The main function module is further configured to receive the unicast first video code stream sent by the server, and decode the first video data stream to obtain the first video data. The first video data is video content corresponding to the user view range area. In addition, the main function module is further configured to receive the second video code stream of the multicast sent by the server, and decode the second video data, where the second video data includes the panoramic video content. The main function module replaces the portion of the second video data corresponding to the video content in the FOV by the first video data, and obtains image information and sends the image information to the display module. The display module receives and displays the image information, and presents the image information to the user. Compared with the prior art, the VR panoramic video playback client receives the panoramic video in a multicast manner, which saves the transmission bandwidth and reduces the bandwidth load. Meanwhile, the VR panoramic video playback client no longer bears the corresponding video of the FOV area. The content calculation process saves valuable client computing resources.
可选的，所述主功能模块包括用户视角信息获取单元、数据发送单元、第一数据接收单元、第一数据解码单元、第二数据接收单元、第二数据解码单元和图像处理单元。其中，用户视角信息获取单元用于收集用户视角信息，所述用户视角信息可以实现为用户的视觉中心点信息和视觉覆盖角度信息。数据发送单元用于将所述用户视角信息发送给服务器，以使服务器通过所述用户视角信息确定第一视频数据。第一数据接收单元用于接收服务器发送的单播的第一视频码流，并发送给第一数据解码单元；第二数据接收单元用于接收服务器发送的组播的第二视频码流，并发送给第二数据解码单元。所述第一数据解码单元用于解码所述第一视频码流得到第一视频数据，所述第二数据解码单元用于解码所述第二视频码流得到第二视频数据。所述第一数据接收单元和所述第二数据接收单元可以被实现为一个数据接收模块，所述第一数据解码单元和所述第二数据解码单元也可以被实现为一个数据解码单元。所述图像处理单元用于将所述第一视频数据替代所述第二视频数据中与FOV内视频内容相应的视频信息，得到最终需要呈现给用户的图像。Optionally, the main function module includes a user view information acquiring unit, a data sending unit, a first data receiving unit, a first data decoding unit, a second data receiving unit, a second data decoding unit, and an image processing unit. The user view information acquiring unit is configured to collect user view information, and the user view information may be implemented as a user's visual center point information and visual coverage angle information. The data sending unit is configured to send the user view information to the server, so that the server determines the first video data by using the user view information. The first data receiving unit is configured to receive the unicast first video code stream sent by the server, and send the first video data stream to the first data decoding unit, where the second data receiving unit is configured to receive the multicast second video code stream sent by the server, and Send to the second data decoding unit. The first data decoding unit is configured to decode the first video code stream to obtain first video data, and the second data decoding unit is configured to decode the second video code stream to obtain second video data. The first data receiving unit and the second data receiving unit may be implemented as one data receiving module, and the first data decoding unit and the second data decoding unit may also be implemented as one data decoding unit. The image processing unit is configured to replace the video information corresponding to the video content in the FOV in the second video data to obtain an image that needs to be presented to the user.
可选的，所述第一视频数据的图像质量要高于所述第二视频数据的图像质量。具体的，可以体现为第一视频数据图像的压缩码率高于第二视频数据图像的压缩码率，或者第一视频数据图像的信噪比高于第二视频数据图像的信噪比。Optionally, the image quality of the first video data is higher than the image quality of the second video data. Specifically, it may be embodied that the compression code rate of the first video data image is higher than the compression code rate of the second video data image, or the signal to noise ratio of the first video data image is higher than the signal to noise ratio of the second video data image.
可选的，所述全景视频内容被分为多个子块，所述FOV内视频内容是指FOV所覆盖到的子块的视频内容。所述被覆盖到的子块包括仅有部分区域被FOV覆盖到的子块。Optionally, the panoramic video content is divided into a plurality of sub-blocks, and the video content in the FOV refers to video content of a sub-block covered by the FOV. The covered sub-block includes a sub-block to which only a partial area is covered by the FOV.
可选的，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息，所述服务器通过所述视觉中心点信息和视觉覆盖角度信息确定用户视角的具体覆盖范围，进而通过用户视角的具体覆盖范围确定第一视频数据。Optionally, the user view information includes visual center point information and visual coverage angle information, and the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user. The range determines the first video data.
第二方面，本申请提供一种VR全景视频播放服务器。所述服务器包括图像生成模块，数据接收模块，子块信息计算模块，第一视频数据获取模块，第一数据发送模块，第二视频数据获取模块和第二数据发送模块。其中，图像生成模块用于生成VR全景视频内容，且所述全景视频内容被分为若干个子块。所述数据接收模块用于接收用户视角信息。所述第一视频数据获取模块用于根据所述FOV对应子块信息从所述全景视频中提取对应子块并编码为第一视频码流。所述第一数据发送模块用于以单播形式向客户端发送第一视频码流。所述第二视频数据获取模块用于提取全景视频并编码为第二视频码流。第二数据发送模块用于以组播形式向客户端发送所述第二视频码流。与现有技术相比，所述VR全景视频播放服务器以组播形式发送全景视频，节省了传输带宽，降低了带宽负荷；同时，所述VR全景视频播放服务器承担了FOV区域对应视频内容的计算过程，节省了宝贵的客户端运算资源。In a second aspect, the present application provides a VR panoramic video playing server. The server includes an image generation module, a data receiving module, a sub-block information calculation module, a first video data acquisition module, a first data transmission module, a second video data acquisition module, and a second data transmission module. The image generation module is configured to generate VR panoramic video content, and the panoramic video content is divided into several sub-blocks. The data receiving module is configured to receive user view information. The first video data acquiring module is configured to extract a corresponding sub-block from the panoramic video according to the FOV corresponding sub-block information and encode the first video code stream. The first data sending module is configured to send the first video code stream to the client in a unicast manner. The second video data acquiring module is configured to extract a panoramic video and encode the second video code stream. The second data sending module is configured to send the second video code stream to the client in a multicast form. Compared with the prior art, the VR panoramic video playing server transmits the panoramic video in the form of multicast, which saves the transmission bandwidth and reduces the bandwidth load. Meanwhile, the VR panoramic video playing server undertakes the calculation of the corresponding video content of the FOV area. The process saves valuable client computing resources.
可选的，所述第一数据发送模块和所述第二数据发送模块可以实现为一个模块。所述第一视频数据获取模块和所述第二视频数据获取模块也可以实现为一个模块。Optionally, the first data sending module and the second data sending module may be implemented as one module. The first video data acquiring module and the second video data acquiring module may also be implemented as one module.
可选的，所述第一视频码流的图像质量要高于所述第二视频码流的图像质量。具体的，可以体现为第一视频码流图像的压缩码率高于第二视频码流图像的压缩码率，或者第一视频 码流图像的信噪比高于第二视频码流图像的信噪比。Optionally, the image quality of the first video code stream is higher than the image quality of the second video code stream. Specifically, the compression code rate of the first video code stream image is higher than the compression code rate of the second video code stream image, or the signal to noise ratio of the first video code stream image is higher than the second video code stream image. Noise ratio.
可选的，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息，所述服务器通过所述视觉中心点信息和视觉覆盖角度信息确定用户视角的具体覆盖范围，进而通过用户视角的具体覆盖范围和全景视频分块方式确定FOV对应子块信息。Optionally, the user view information includes visual center point information and visual coverage angle information, and the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user. The range and panoramic video blocking mode determines the FOV corresponding sub-block information.
第三方面，本申请提供一种VR全景视频播放客户端。所述客户端包括传感器、处理器、通信端口和显示器。所述传感器用于收集用户视角信息并发送给所述处理器。所述处理器用于通过所述端口发送所述用户视角信息。所述用户视角信息用于确定第一视频数据，所述第一视频数据对应用户视角范围区域FOV内的视频内容。此外，所述客户端还用于通过所述通信端口接收服务器发送的单播的第一视频码流和组播的第二视频码流，并将二者分别解码得到第一视频数据和第二视频数据。将所述第一视频数据替代所述第二视频数据中与FOV内视频内容对应的部分，得到图像信息，并发送给所述显示器。所述通信端口用于收发信息，所述显示器用于显示所述图像信息。与现有技术相比，所述VR全景视频播放客户端以组播形式接收全景视频，节省了传输带宽，降低了带宽负荷；同时，所述VR全景视频播放客户端不再承担FOV区域对应视频内容的计算过程，节省了宝贵的客户端运算资源。In a third aspect, the application provides a VR panoramic video playback client. The client includes a sensor, a processor, a communication port, and a display. The sensor is configured to collect user view information and send it to the processor. The processor is configured to send the user view information through the port. The user view information is used to determine first video data, and the first video data corresponds to video content within a user view range area FOV. In addition, the client is further configured to receive, by using the communication port, the unicast first video code stream and the multicast second video code stream sent by the server, and respectively decode the two to obtain the first video data and the second Video data. The first video data is replaced with a portion of the second video data corresponding to the video content in the FOV, and image information is obtained and sent to the display. The communication port is for transmitting and receiving information, and the display is for displaying the image information. Compared with the prior art, the VR panoramic video playback client receives the panoramic video in a multicast manner, which saves the transmission bandwidth and reduces the bandwidth load. Meanwhile, the VR panoramic video playback client no longer bears the corresponding video of the FOV area. The content calculation process saves valuable client computing resources.
可选的，所述第一视频数据的图像质量要高于所述第二视频数据的图像质量。具体的，可以体现为第一视频数据图像的压缩码率高于第二视频数据图像的压缩码率，或者第一视频数据图像的信噪比高于第二视频数据图像的信噪比。Optionally, the image quality of the first video data is higher than the image quality of the second video data. Specifically, it may be embodied that the compression code rate of the first video data image is higher than the compression code rate of the second video data image, or the signal to noise ratio of the first video data image is higher than the signal to noise ratio of the second video data image.
可选的，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息，所述服务器通过所述视觉中心点信息和视觉覆盖角度信息确定用户视角的具体覆盖范围，进而通过用户视角的具体覆盖范围确定第一视频数据。Optionally, the user view information includes visual center point information and visual coverage angle information, and the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user. The range determines the first video data.
可选的，所述全景视频内容被分为多个子块，所述FOV内视频内容是指FOV所覆盖到的子块的视频内容。所述被覆盖到的子块包括仅有部分区域被FOV覆盖到的子块。Optionally, the panoramic video content is divided into a plurality of sub-blocks, and the video content in the FOV refers to video content of a sub-block covered by the FOV. The covered sub-block includes a sub-block to which only a partial area is covered by the FOV.
第四方面，本申请提供一种VR全景视频播放服务器。所述服务器包括通信端口和处理器。所述通信端口用于同客户端收发信息，进行通信。所述处理器用于生成被分为若干子块的VR全景视频内容。所述处理器通过所述通信端口接收客户端发送的用户视角信息，并结合全景视频的分块方式确定用户视角所覆盖的视频子块，即确定FOV对应子块信息。所述处理器根据所述FOV对应子块信息从所述全景视频中提取对应的子块并编码为第一视频码流，提取全景视频信息并编码为第二视频码流。所述处理器通过所述通信端口以单播形式向客户端发送所述第一视频码流，以组播形式向客户端发送所述第二视频码流。与现有技术相比，所述VR全景视频播放服务器以组播形式发送全景视频，节省了传输带宽，降低了带宽负荷；同时，所述VR全景视频播放服务器承担了FOV区域对应视频内容的计算过程，节省了宝贵的客户端运算资源。In a fourth aspect, the application provides a VR panoramic video playing server. The server includes a communication port and a processor. The communication port is configured to send and receive information to and communicate with the client. The processor is configured to generate VR panoramic video content that is divided into sub-blocks. The processor receives the user view information sent by the client through the communication port, and determines the video sub-block covered by the user view in combination with the block mode of the panoramic video, that is, determines the FOV corresponding sub-block information. The processor extracts a corresponding sub-block from the panoramic video according to the FOV corresponding sub-block information and encodes it into a first video code stream, and extracts the panoramic video information and encodes the second video code stream. The processor sends the first video code stream to the client in a unicast manner through the communication port, and sends the second video code stream to the client in a multicast manner. Compared with the prior art, the VR panoramic video playing server transmits the panoramic video in the form of multicast, which saves the transmission bandwidth and reduces the bandwidth load. Meanwhile, the VR panoramic video playing server undertakes the calculation of the corresponding video content of the FOV area. The process saves valuable client computing resources.
可选的，所述第一视频码流的图像质量要高于所述第二视频码流的图像质量。具体的，可以体现为第一视频码流图像的压缩码率高于第二视频码流图像的压缩码率，或者第一视频码流图像的信噪比高于第二视频码流图像的信噪比。Optionally, the image quality of the first video code stream is higher than the image quality of the second video code stream. Specifically, the compression code rate of the first video code stream image is higher than the compression code rate of the second video code stream image, or the signal to noise ratio of the first video code stream image is higher than the second video code stream image. Noise ratio.
可选的，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息，所述服务器通过所述视觉中心点信息和视觉覆盖角度信息确定用户视角的具体覆盖范围，进而通过用户视角的具体覆盖范围和全景视频分块方式确定FOV对应子块信息。Optionally, the user view information includes visual center point information and visual coverage angle information, and the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user. The range and panoramic video blocking mode determines the FOV corresponding sub-block information.
第五方面，本申请提供一种VR全景视频播放方法。客户端收集并向服务器发送用户视角信息，所述用户视角信息用于确定用户视角范围区域FOV内的视频内容，即第一视频数据。所述客户端接收并解码单播的第一视频码流，得到所述第一视频数据；接收并解码组播的第 二视频码流，得到包括全景视频内容的第二视频数据。所述客户端将第一视频数据替代所述第二视频数据中与FOV内视频内容对应的部分，得到图像信息，并将所述图像信息显示给用户。与现有技术相比，所述VR全景视频播放方法中客户端与服务器间以组播形式传输全景视频，节省了传输带宽，降低了带宽负荷；同时，FOV区域对应视频内容的计算不再由VR全景视频播放客户端不再承担，节省了宝贵的客户端运算资源。In a fifth aspect, the present application provides a VR panoramic video playing method. The client collects and sends user view information to the server, and the user view information is used to determine video content within the user's view range area FOV, that is, the first video data. The client receives and decodes the unicast first video code stream to obtain the first video data, and receives and decodes the multicast second video code stream to obtain second video data including the panoramic video content. The client replaces a portion of the second video data corresponding to the video content in the FOV with the first video data, obtains image information, and displays the image information to the user. Compared with the prior art, in the VR panoramic video playing method, the panoramic video is transmitted between the client and the server in a multicast manner, which saves the transmission bandwidth and reduces the bandwidth load; meanwhile, the calculation of the video content corresponding to the FOV region is no longer performed by The VR panoramic video playback client no longer bears, saving valuable client computing resources.
可选的，所述第一视频数据的图像质量要高于所述第二视频数据的图像质量。具体的，可以体现为第一视频数据图像的压缩码率高于第二视频数据图像的压缩码率，或者第一视频数据图像的信噪比高于第二视频数据图像的信噪比。Optionally, the image quality of the first video data is higher than the image quality of the second video data. Specifically, it may be embodied that the compression code rate of the first video data image is higher than the compression code rate of the second video data image, or the signal to noise ratio of the first video data image is higher than the signal to noise ratio of the second video data image.
可选的，所述全景视频内容被分为多个子块，所述FOV内视频内容是指FOV所覆盖到的子块的视频内容。所述被覆盖到的子块包括仅有部分区域被FOV覆盖到的子块。Optionally, the panoramic video content is divided into a plurality of sub-blocks, and the video content in the FOV refers to video content of a sub-block covered by the FOV. The covered sub-block includes a sub-block to which only a partial area is covered by the FOV.
可选的，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息，所述服务器通过所述视觉中心点信息和视觉覆盖角度信息确定用户视角的具体覆盖范围，进而通过用户视角的具体覆盖范围确定第一视频数据。Optionally, the user view information includes visual center point information and visual coverage angle information, and the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user. The range determines the first video data.
第六方面，本申请提供一种VR全景视频播放方法。服务器生成分块的VR全景视频内容。服务器接收客户端发送的用户视角信息，结合全景视频的分块方式确定用户视角所覆盖的对应子块，即FOV对应子块信息。服务器根据所述FOV对应子块信息从所述全景视频中提取对应子块，编码为第一视频码流并以单播形式发送；提取全景视频信息，编码为第二视频码流并以组播的形式发送。所述客户端接收、解码并播放所述第一视频码流和所述第二视频码流中的内容。In a sixth aspect, the application provides a VR panoramic video playing method. The server generates chunked VR panoramic video content. The server receives the user view information sent by the client, and determines the corresponding sub-block covered by the user view, that is, the FOV corresponding sub-block information, in combination with the block mode of the panoramic video. And the server extracts the corresponding sub-block from the panoramic video according to the FOV corresponding sub-block information, encodes the first video code stream and sends the information in a unicast manner; extracts the panoramic video information, encodes the second video code stream, and uses the multicast The form is sent. The client receives, decodes, and plays content in the first video code stream and the second video code stream.
可选的，所述第一视频码流的图像质量要高于所述第二视频码流的图像质量。具体的，可以体现为第一视频码流图像的压缩码率高于第二视频码流图像的压缩码率，或者第一视频码流图像的信噪比高于第二视频码流图像的信噪比。Optionally, the image quality of the first video code stream is higher than the image quality of the second video code stream. Specifically, the compression code rate of the first video code stream image is higher than the compression code rate of the second video code stream image, or the signal to noise ratio of the first video code stream image is higher than the second video code stream image. Noise ratio.
可选的，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息，所述服务器通过所述视觉中心点信息和视觉覆盖角度信息确定用户视角的具体覆盖范围，进而通过用户视角的具体覆盖范围和全景视频分块方式确定FOV对应子块信息。Optionally, the user view information includes visual center point information and visual coverage angle information, and the server determines a specific coverage of the user's perspective through the visual center point information and the visual coverage angle information, and further covers the specific coverage of the user. The range and panoramic video blocking mode determines the FOV corresponding sub-block information.
附图说明DRAWINGS
图1为一种VR全景视频播放***的设备交互示意图；FIG. 1 is a schematic diagram of device interaction of a VR panoramic video playing system;
图2为一种VR全景视频播放***的交互过程示意图；2 is a schematic diagram of an interaction process of a VR panoramic video playing system;
图3为一种又一种VR全景视频播放***的设备交互示意图；FIG. 3 is a schematic diagram of device interaction of another VR panoramic video playing system;
图4为一种VR全景视频播放方法的流程示意图；4 is a schematic flow chart of a VR panoramic video playing method;
图5为一种VR全景视频播放客户端的逻辑结构示意图；FIG. 5 is a schematic diagram of a logical structure of a VR panoramic video playing client;
图6为一种VR全景视频播放服务器的逻辑结构示意图；6 is a schematic diagram of a logical structure of a VR panoramic video playing server;
图7为一种VR全景视频播放客户端的硬件结构示意图；7 is a schematic diagram of a hardware structure of a VR panoramic video playing client;
图8为一种VR全景视频播放服务器的硬件结构示意图。FIG. 8 is a schematic diagram of a hardware structure of a VR panoramic video playing server.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述。The technical solutions in the embodiments of the present invention will be clearly and completely described in the following with reference to the accompanying drawings.
本申请所涉及的VR(Virtual Reality)视频传输方法，主要应用于VR全景视频播放中。所述的VR全景视频，主要为360度或180度全景视频。客户端和服务器间数据传输通常采用RTP(Real-Time Transmit Protocol)通信协议。但须知的是，当应用场景变换至其他任何 可观看范围大于用户一个视角范围区域(FOV,Field of View)的视频播放情形，或视频传输采用任何其他支持组播的通信协议时，本申请所记载的技术方案同样适用。The VR (Virtual Reality) video transmission method involved in the present application is mainly applied to VR panoramic video playback. The VR panoramic video is mainly a 360 degree or 180 degree panoramic video. The data transmission between the client and the server usually adopts the RTP (Real-Time Transmit Protocol) communication protocol. However, it should be noted that when the application scenario is changed to any other video playback situation in which the viewable range is larger than the user's Field of View (FOV) field, or the video transmission uses any other communication protocol supporting multicast, the present application The technical solutions described are equally applicable.
须知，本申请实施例中客户端侧客户端设备的数量可以为任意个，实施例中的客户端的具体数量仅为方便说明，并不构成对本申请应用场景的限定。在本实施例部分的一个或多个实施例中，FOV视频码流可以理解为权利要求中所述的第一视频码流，其中包含的视频数据可以理解为权利要求中所述的第一视频数据；全景视频码流可以理解为权利要求中所述的第二视频码流，其中包含的视频数据可以理解为权利要求中所述的第二视频数据。在一个或多个实施例中，FOV视频码流中的视频数据图像质量要高于全景视频码流中的视频数据图像质量。具体而言，相比于全景视频码流中的视频数据，服务器侧在生成FOV视频码流的视频数据时采用的压缩码率更高，或者FOV视频码流中视频数据的信噪比要高于全景视频码流中视频数据的信噪比。It should be noted that the number of the client-side client devices in the embodiment of the present application may be any number. The specific number of the client in the embodiment is only for convenience of description, and does not constitute a limitation on the application scenario of the present application. In one or more embodiments of this embodiment, the FOV video code stream can be understood as the first video code stream described in the claims, wherein the included video data can be understood as the first video described in the claims. Data; a panoramic video stream can be understood as a second video stream as described in the claims, and the video data contained therein can be understood as the second video data recited in the claims. In one or more embodiments, the video data image quality in the FOV video stream is higher than the video data image quality in the panoramic video stream. Specifically, the server side uses a higher compression code rate when generating video data of the FOV video stream than the video data in the panoramic video stream, or the signal-to-noise ratio of the video data in the FOV video stream is higher. The signal-to-noise ratio of the video data in the panoramic video stream.
现有技术中的VR全景视频播放***的组成如图1所示。VR全景视频播放***100包括服务器侧110、客户端侧120和网络设备侧130，其中服务器侧110可以是任何提供媒体服务的设备，比如服务器组、电脑甚至手机；客户端侧120可以为一个设备，也可以为多个设备，具体数量取决于同时观看同一全景视频内容的用户数量，本实施例中为客户端121和客户端122两台设备；网络设备侧130可以实现为路由器或交换机。当多个用户同时观看同一全景视频内容时，各客户端侧的设备单独通过网络侧设备与服务器侧进行通信，具体的，服务器侧110通过网络设备侧130向客户端侧120的各客户端发送全景视频分块信息，客户端121和客户端122分别向网络侧130发送请求数据，请求数据中包含本客户端的FOV对应子块信息，所述FOV对应子块信息由客户端根据全景视频分块信息和本机用户视角信息计算得出。网络侧130将所述请求数据发送给服务器侧110。服务器侧110在接收到请求数据后，向网络侧130返回应答数据，分别是两路FOV视频码流和两路全景视频码流。网络侧130将所述两路FOV视频码流分别发送给客户端121和客户端122，并将两路全景视频码流分别发送给客户端121和客户端122。两个客户端在接收到所述应答数据后，对其中包含的视频内容进行显示，呈现给用户。The composition of the VR panoramic video playback system in the prior art is as shown in FIG. 1. The VR panoramic video playback system 100 includes a server side 110, a client side 120, and a network device side 130, wherein the server side 110 can be any device that provides media services, such as a server group, a computer, or even a mobile phone; the client side 120 can be a device. The number of users that can view the same panoramic video content at the same time, in this embodiment, the two devices of the client 121 and the client 122; the network device side 130 can be implemented as a router or a switch. When a plurality of users simultaneously view the same panoramic video content, the devices on the client side communicate with the server side through the network side device. Specifically, the server side 110 sends the client side to the client side 120 through the network device side 130. For the panoramic video segmentation information, the client 121 and the client 122 respectively send request data to the network side 130, where the request data includes the FOV corresponding sub-block information of the client, and the FOV corresponding sub-block information is segmented by the client according to the panoramic video. Information and local user perspective information are calculated. The network side 130 transmits the request data to the server side 110. After receiving the request data, the server side 110 returns response data to the network side 130, which are two FOV video code streams and two panoramic video code streams. The network side 130 sends the two FOV video streams to the client 121 and the client 122, and sends the two panoramic video streams to the client 121 and the client 122, respectively. After receiving the response data, the two clients display the video content contained therein and present it to the user.
VR全景视频播放***100的具体运作方式请参见图2。所述***的通信交互在三个层次完成，分别是服务器侧110、客户端侧120和网络设备侧130。客户端侧120可以为一个设备，也可以为多个设备，具体数量取决于同时观看同一全景视频内容的用户数量，本实施例中为客户端121和客户端122。See Figure 2 for the specific operation of the VR panoramic video playback system 100. The communication interaction of the system is completed at three levels, namely, the server side 110, the client side 120, and the network device side 130. The client side 120 can be a device or a plurality of devices, and the number of users depends on the number of users who simultaneously view the same panoramic video content. In this embodiment, the client 121 and the client 122 are used.
首先，服务器侧110生成分块的全景视频内容111，然后将视频分块信息通过网络设备侧130发送给客户端侧120。本示例中全景视频内容111被分为了1～8八个相同大小的子视频块。在不同的实施方案中，全景视频内容111的分块方式可以为任意大小和数量的矩形。客户端侧120的客户端121向网络设备侧发送请求信息1，其中包含客户端121的FOV对应子块信息，所述FOV对应子块信息由客户端根据视频分块信息和当前用户视角信息确定，描述了客户端121当前的视角范围覆盖了具体哪些子块的视频，本实施例中是子块2、3、6和7。客户端122向网络设备侧发送请求信息2，其中包含客户端122的FOV对应子块信息，所述FOV对应子块信息描述了客户端122当前的视角范围覆盖了具体哪些子块的视频，本实施例中是子块1、2、5和6。所述请求信息1和请求信息2通过网络设备侧被传输给服务器侧110。然后服务器侧110根据所述请求信息1和请求信息2通过网络设备侧130同时向客户端侧120以单播形式发送四路视频码流，分别是针对客户端121发送的FOV视频码流1，其内容为FOV视频数据1，即客户端121所请求的视频子块内容；全景视频码流1，内容为全景视 频数据112；针对客户端122发送的FOV视频码流2，其内容为FOV视频数据2，即客户端122所请求的视频子块内容；全景视频码流2，内容为全景视频数据112。客户端121在接收所述FOV视频码流1和所述全景视频码流1后，将二者中的视频内容进行组合，把所述子块2、3、6、7中的子块视频内容按发送前在服务器110中的关系进行拼接，然后将拼接好的内容与所述全景视频数据112中的全景视频内容进行组合，覆盖全景视频内容中与其对应的部分，形成客户端视频123，并呈现给用户；客户端122在接收所述FOV视频码流2和所述全景视频码流2后，将二者中的视频内容进行组合，把所述子块1、2、5、6中的子块视频内容按发送前在服务器110中的关系进行拼接，然后将拼接好的内容与所述全景视频数据112中的全景视频内容进行组合，覆盖全景视频内容中与其对应的部分，形成客户端视频124，并呈现给用户。First, the server side 110 generates the chunked panoramic video content 111 and then transmits the video chunking information to the client side 120 through the network device side 130. The panoramic video content 111 in this example is divided into 1 to 8 sub-video blocks of the same size. In various embodiments, the tiled manner of the panoramic video content 111 can be any size and number of rectangles. The client 121 of the client side 120 sends the request information 1 to the network device side, where the FOV corresponding sub-block information of the client 121 is included, and the FOV corresponding sub-block information is determined by the client according to the video segmentation information and the current user view information. It is described that the current view range of the client 121 covers the videos of which sub-blocks, and in this embodiment, the sub-blocks 2, 3, 6, and 7. The client 122 sends the request information 2 to the network device side, where the FOV corresponding sub-block information of the client 122 is included, and the FOV corresponding sub-block information describes the video of the specific viewing angle range of the client 122 covering the specific sub-blocks. In the embodiment are sub-blocks 1, 2, 5 and 6. The request information 1 and the request information 2 are transmitted to the server side 110 through the network device side. The server side 110 then sends the four video streams in the unicast form to the client side 120 through the network device side 130 according to the request information 1 and the request information 2, which are respectively the FOV video code stream 1 sent by the client 121. The content is FOV video data 1, that is, the video sub-block content requested by the client 121; the panoramic video stream 1 is the panoramic video data 112; and the FOV video stream 2 sent to the client 122 is the FOV video. Data 2, that is, video sub-block content requested by the client 122; panoramic video stream 2, the content is panoramic video data 112. After receiving the FOV video stream 1 and the panoramic video stream 1, the client 121 combines the video content in the two, and the sub-block video content in the sub-blocks 2, 3, 6, and 7. Splicing according to the relationship in the server 110 before sending, and then combining the stitched content with the panoramic video content in the panoramic video data 112, covering the corresponding portion of the panoramic video content, forming a client video 123, and Presented to the user; after receiving the FOV video stream 2 and the panoramic video stream 2, the client 122 combines the video content in the two, and the sub-blocks 1, 2, 5, and 6 The sub-block video content is spliced according to the relationship in the server 110 before being sent, and then the spliced content is combined with the panoramic video content in the panoramic video data 112 to cover the corresponding portion of the panoramic video content to form a client. Video 124 is presented to the user.
在现有技术的传输方案下客户端视频容易出现卡顿等情况。客户端FOV对应子块信息是由客户端根据视频描述信息和当前用户用户视角信息来计算的，频繁计算FOV对应子块信息会占用客户端大量运算资源，尤其是当视频分块比较复杂时。另外，当多用户同时观看视频时，网络侧设备与服务器侧之间带宽紧张。针对上述问题，本申请创造性的提出了以下视频传输方案。In the prior art transmission scheme, the client video is prone to jams and the like. The client FOV corresponding sub-block information is calculated by the client according to the video description information and the current user user perspective information. Frequently calculating the FOV corresponding sub-block information will occupy a large amount of computing resources of the client, especially when the video partitioning is complicated. In addition, when multiple users watch video at the same time, the bandwidth between the network side device and the server side is tight. In response to the above problems, the present application creatively proposes the following video transmission scheme.
请参阅图3，是本发明的一种实施例，内容为一种VR全景视频播放***200。VR全景视频播放***200包括服务器侧210、客户端侧220和网络设备侧230，其中服务器侧210可以是任何提供媒体服务的设备，比如服务器组、电脑甚至手机。客户端侧220可以为一个设备，也可以为多个设备，具体数量取决于同时观看同一全景视频内容的用户数量，本实施例中为客户端221和客户端222两台设备。须知，本实施例中客户端数量仅为方便说明，并不对本申请构成限定，客户端的个数可以为任意数量。网络设备侧230可以实现为支持组播功能的路由器或交换机。当多个用户同时观看同一全景视频内容时，各客户端侧的设备单独通过网络侧设备与服务器侧进行通信。具体的，服务器侧210首先生成分块的全景视频内容，所述分块的全景视频具体细节参见图2中的相关描述，此处不再赘述。客户端221和客户端222分别通过网络侧230向服务器侧210发送请求数据，所述数据中包含本客户端当前用户的用户视角信息，所述用户视角信息描述了客户端当前视角覆盖范围，可以包含视觉中心点信息和视觉覆盖角度信息，以确定客户端用户视角的具体覆盖范围。服务器侧210接收所述请求数据，并通过全景视频分块信息和所述请求数据中客户端221的用户视角信息确定客户端221的FOV对应子块信息，通过全景视频分块信息和所述请求数据中客户端222的用户视角信息确定客户端222的FOV对应子块信息。所述全景视频分块信息和所述FOV对应子块信息的含义请参见图2中的相关描述。之后，服务器侧210分别根据客户端221和客户端222的FOV对应子块信息生从分块的全景视频内容中提取对应子块，生成FOV视频码流1和FOV视频码流2，所述FOV视频码流1中包含客户端221所需要的子块视频，所述FOV视频码流2中包含客户端222所需要的子块视频。之后，服务器侧210以单播和组播相结合的方式向网络侧230返回应答数据，分别是以单播形式向客户端221发送的FOV视频码流1，以单播形式向客户端222发送的FOV视频码流2，以及以组播形式向所有客户端发送的全景视频码流。所述全景视频码流中包含了整体的全景视频内容。网络侧230将所述应答数据中的FOV视频码流分别发送给对应的客户端，即将FOV视频码流1发送给客户端221，将FOV视频码流2发送给客户端222，此外还分别向两个客户端发送全景视频码流。客户端侧220接收所述应答数据，具体的，客户端221接收所述FOV视频码流1和所述组播的全景视频码流；客户端222接收所述FOV视频码流2和所述组播的全景视频码流。在接收到所述应答数据后，客户端221 和客户端222分别对其中包含的视频内容进行显示，具体呈现方式参见图2中的具体描述，此处不再赘述。Please refer to FIG. 3, which is an embodiment of the present invention, and is a VR panoramic video playback system 200. The VR panoramic video playback system 200 includes a server side 210, a client side 220, and a network device side 230, wherein the server side 210 can be any device that provides media services, such as a server group, a computer, or even a mobile phone. The client side 220 can be a device or a plurality of devices, and the number of users depends on the number of users who simultaneously view the same panoramic video content. In this embodiment, the client 221 and the client 222 are two devices. It should be noted that the number of clients in this embodiment is only for convenience of description, and is not limited to the present application, and the number of clients may be any number. Network device side 230 can be implemented as a router or switch that supports multicast functionality. When multiple users simultaneously view the same panoramic video content, the devices on each client side communicate with the server side through the network side device. Specifically, the server side 210 first generates the blocked panoramic video content, and the specific details of the divided panoramic video are referred to the related description in FIG. 2, and details are not described herein again. The client 221 and the client 222 respectively send request data to the server side 210 through the network side 230, where the data includes user view information of the current user of the client, and the user view information describes the current view coverage of the client. Contains visual center point information and visual coverage angle information to determine the specific coverage of the client's user perspective. The server side 210 receives the request data, and determines the FOV corresponding sub-block information of the client 221 through the panoramic video blocking information and the user perspective information of the client 221 in the request data, and the panoramic video blocking information and the request. The user view information of the client 222 in the data determines the FOV corresponding sub-block information of the client 222. For the meanings of the panoramic video segmentation information and the FOV corresponding sub-block information, refer to the related description in FIG. 2. After that, the server side 210 extracts corresponding sub-blocks from the blocked panoramic video content according to the FOV corresponding sub-block information of the client 221 and the client 222, respectively, to generate the FOV video stream 1 and the FOV video stream 2, the FOV. The video stream 1 contains the sub-block video required by the client 221, and the FOV video stream 2 contains the sub-block video required by the client 222. Then, the server side 210 returns the response data to the network side 230 in a combination of unicast and multicast, and the FOV video code stream 1 sent to the client 221 in unicast mode is sent to the client 222 in unicast form. The FOV video stream 2, and the panoramic video stream sent to all clients in multicast form. The panoramic video stream includes the entire panoramic video content. The network side 230 sends the FOV video code stream in the response data to the corresponding client, that is, the FOV video stream 1 is sent to the client 221, and the FOV video stream 2 is sent to the client 222, and Two clients send a panoramic video stream. The client side 220 receives the response data. Specifically, the client 221 receives the FOV video stream 1 and the multicast panoramic video stream; the client 222 receives the FOV video stream 2 and the group. Broadcast panoramic video stream. After receiving the response data, the client 221 and the client 222 respectively display the video content contained therein. For the specific presentation manner, refer to the detailed description in FIG. 2, and details are not described herein again.
请参阅图4，是本发明的另一实施例，内容为一种VR全景视频播放***中的信息传输方法S300，具体包括：Referring to FIG. 4, it is another embodiment of the present invention, which is an information transmission method S300 in a VR panoramic video playing system, and specifically includes:
S310，客户端侧向服务器侧发送请求数据，所述请求数据中包含当前用户视角信息。客户端侧的客户端获取当前用户视角信息，所述用户视角信息中可以包含视觉中心点信息和视觉覆盖角度信息。视觉中心点信息和视觉覆盖角度信息用来确定客户端视角的具体覆盖范围。所述客户端侧通过网络设备侧向服务器侧发送请求数据。S310. The client side sends request data to the server side, where the request data includes current user perspective information. The client side client obtains current user perspective information, and the user perspective information may include visual center point information and visual coverage angle information. Visual center point information and visual coverage angle information are used to determine the specific coverage of the client's perspective. The client side sends request data to the server side through the network device side.
S320，服务器侧接收所述请求数据，并根据所述请求数据中的所述当前用户视角信息确定FOV对应子块信息。所述服务器侧通过所述当前用户视角信息和全景视频分块方式确定用户视角覆盖范围内应显示的具体视频子块，即FOV对应子块信息。所述FOV对应子块信息请参见图1相关内容中的描述。S320. The server side receives the request data, and determines FOV corresponding sub-block information according to the current user view information in the request data. The server side determines, by using the current user view information and the panoramic video block mode, a specific video sub-block that should be displayed in the coverage of the user's perspective, that is, FOV corresponding sub-block information. For the FOV corresponding sub-block information, please refer to the description in the related content of FIG.
S330，服务器侧向客户端侧发送应答信息，所述应答信息包含单播的FOV视频码流和组播的全景视频码流。具体的，S331，服务器侧向客户端侧的各客户端以单播的形式发送FOV视频码流，所述各FOV视频码流中分别包含各客户端所需的对应FOV子块视频；S332，服务器侧向客户端侧的各客户端以组播的形式发送全景视频码流，所述全景视频码流中包含全景视频内容。S330. The server side sends response information to the client side, where the response information includes a unicast FOV video code stream and a multicast panoramic video code stream. Specifically, in S331, the server side sends the FOV video code stream to each client on the client side in a unicast manner, where each FOV video code stream includes a corresponding FOV sub-block video required by each client; S332, The server side transmits the panoramic video code stream to each client on the client side in a multicast manner, and the panoramic video code stream includes the panoramic video content.
S340，客户端侧接收应答数据，即各客户端分别接收发送给本机的FOV视频码流和全景视频码流。客户端接收发送给本机的FOV视频码流，解析并获取其中的子块视频，同时还接收发送给本机的全景视频码流，解析并获取其中的全景视频。S340. The client side receives the response data, that is, each client separately receives the FOV video code stream and the panoramic video code stream sent to the local machine. The client receives the FOV video stream sent to the local machine, parses and acquires the sub-block video therein, and also receives the panoramic video code stream sent to the local machine, parses and acquires the panoramic video therein.
在具体实施过程中，VR全景视频播放***中的信息传输方法300可以实施于RTSP协议，具体如下：In a specific implementation process, the information transmission method 300 in the VR panoramic video playing system can be implemented in the RTSP protocol, as follows:
1)客户端向服务器发送用户视角信息。具体的，可以是，客户端向服务器发送请求数据，所述请求数据中包括该用户视角信息。1) The client sends user perspective information to the server. Specifically, the client may send request data to the server, where the request data includes the user view information.
在具体实施方式中，该请求数据可以是RTCP源描述报告。该RTCP源描述报告可以是根据服务器发送的根据客户端发送的建立连接命令的应答进行发送。其中，该建立连接命令用于建立用于传输子区域的视频码流的会话连接，该会话连接用于单播子区域视频的码流，子区域可以是将全景视频划分后得到的tile。In a specific implementation, the request data may be an RTCP source description report. The RTCP source description report may be sent according to a response sent by the server according to a connection establishment command sent by the client. The establishing a connection command is used to establish a session connection for transmitting a video code stream of a sub-area, where the session is connected to a code stream for unicast sub-area video, and the sub-area may be a tile obtained by dividing the panoramic video.
在具体实施方式中，客户端还会可以发送用于建立传输全景视频的码流的会话连接，该会话连接用于组播的全景视频的码流。其中，全景视频的码流对应的图像质量可以低于子区域视频的码流对应的图像质量，其中图像质量可以用码率或者分辨率来指示。其中，子区域为全景视频区域中的子区域，子区域视频的内容为全景视频中的内容中的部分，通过对子区域视频的内容进行拼接可以得到全景视频的内容，只是子区域视频的码流对应的图像质量更高。In a specific implementation, the client may also send a session connection for establishing a code stream for transmitting the panoramic video, the session connecting the code stream for the multicast panoramic video. The image quality corresponding to the code stream of the panoramic video may be lower than the image quality corresponding to the code stream of the sub-region video, wherein the image quality may be indicated by a code rate or a resolution. The sub-area is a sub-area in the panoramic video area, and the content of the sub-area video is a part of the content in the panoramic video. The content of the panoramic video can be obtained by splicing the content of the sub-area video, but only the code of the sub-area video. The image corresponding to the stream is of higher quality.
在具体实施方式中，该用户视角信息可以包括用户视角的位置信息和用户视角的大小信息至少一项，用户视角的位置信息可以是用户视角的中心点的位置信息，或者用户视角的左上角点的位置信息，其中位置可以分别对应VR球面坐标的yaw，pitch，roll值，也可以只保 留两项(X和Y)分别对应视角区域的中心点二维坐标值或视角区域左上角坐标值。；用户视角的大小信息可以是用户视角的覆盖角度信息，具体来说，可以是用户视角的高度和宽度，其中，宽和高可以是VR球面坐标的水平方向覆盖角度和垂直方向覆盖角度，也可以是二维图像的宽度和高度像素个数值。In a specific implementation, the user view information may include at least one of location information of the user's perspective and size information of the user's perspective. The location information of the user's perspective may be location information of a center point of the user's perspective, or an upper left corner of the user's perspective. The position information, wherein the position can respectively correspond to the yaw, pitch, and roll values of the VR spherical coordinates, or only two (X and Y) corresponding to the central point two-dimensional coordinate value of the viewing angle region or the upper left corner coordinate value of the viewing angle region. The size information of the user's perspective may be the coverage angle information of the user's perspective, specifically, the height and width of the user's perspective, wherein the width and height may be the horizontal coverage angle and the vertical coverage angle of the VR spherical coordinates, It can be the width and height pixel values of a two-dimensional image.
下面介绍本发明实施例提供的一种会话连接的建立过程：The process of establishing a session connection provided by the embodiment of the present invention is as follows:
1.服务器按照图2中所示准备好视频内容，该内容包括低质量的全景视频内容和高质量的子区域视频内容，并发布该内容的地址。地址为RTSP协议格式，作为如下：rtsp://server.example.com/video1. The server prepares the video content as shown in Figure 2, which includes low quality panoramic video content and high quality sub-regional video content, and publishes the address of the content. The address is in the RTSP protocol format as follows: rtsp://server.example.com/video
2.客户端向上述地址发送描述命令，服务器对描述命令进行应答，该应答还可以描述该视频内容，其中需描述用于单播的高质量的子区域视频的码流的会话信息以及用于组播的低质量全景视频的码流的会话信息。2. The client sends a description command to the above address, and the server responds to the description command, and the response can also describe the video content, wherein the session information of the code stream for the high quality sub-area video for unicast needs to be described and used for Session information for the stream of multicast low quality panoramic video.
客户端发送的描述命令可以如下：The description command sent by the client can be as follows:
Figure PCTCN2018100670-appb-000001
Figure PCTCN2018100670-appb-000001
上述表格中的DESCRIBE字段表示描述命令。The DESCRIBE field in the above table represents the description command.
服务器对描述命令的应答可以如下：The server's response to the description command can be as follows:
Figure PCTCN2018100670-appb-000002
Figure PCTCN2018100670-appb-000002
在上述应答种的SDP描述文件中总共描述了两路视频会话。一路是子区域(tiles)的视频码流，码率为5000kbps。一路是全景(panoramic)视频码流，码率为1000kbps。 两路会话均采用RTP协议进行传输。A total of two video sessions are described in the SDP description file of the above response species. All the way is the video stream of the tiles, the code rate is 5000kbps. All the way is a panoramic video stream with a code rate of 1000 kbps. Both sessions are transmitted using the RTP protocol.
3.客户端根据上述应答，与服务器建立两路会话连接，其中一路会话连接用来单播子区域码流，一路会话连接用来组播全景码流。具体过程可以如下：3. The client establishes two-way session connection with the server according to the above response, wherein one session connection is used for unicast sub-area code stream, and one session connection is used for multicast panoramic code stream. The specific process can be as follows:
客户端可以发送如下两个建立连接命令：The client can send the following two connection establishment commands:
Figure PCTCN2018100670-appb-000003
Figure PCTCN2018100670-appb-000003
上述命令中的SETUP字段表示该命令为建立连接命令。第一个命令表明与应答中的SDP(会话描述协议，Session Description Protocol)描述中的track1建立连接，其中的Transport字段表明子区域视频码流通过RTP协议以单播方式传输，用于接收数据流的RTP端口号为20000，用于接收控制流的RTCP端口号为20001。第二个命令表明与应答中SDP描述中的track2建立连接，其中的Transport字段表明全景视频码流通过RTP协议以组播方式传输，用于接收数据流的RTP端口号为20010，用于接收控制流的RTCP端口号为20011。The SETUP field in the above command indicates that the command is a connection establishment command. The first command indicates that a connection is established with track1 in the description of the SDP (Session Description Protocol) in the response, wherein the Transport field indicates that the sub-area video code stream is transmitted in a unicast manner through the RTP protocol for receiving the data stream. The RTP port number is 20000, and the RTCP port number used to receive the control flow is 20001. The second command indicates that a connection is established with track2 in the SDP description in the response. The Transport field indicates that the panoramic video stream is transmitted in multicast mode through the RTP protocol, and the RTP port number used to receive the data stream is 20010, which is used for receiving control. The RTCP port number of the stream is 20011.
服务器针对客户端发送的建立连接命令可以做如下应答：The server can respond to the connection establishment command sent by the client as follows:
Figure PCTCN2018100670-appb-000004
Figure PCTCN2018100670-appb-000004
上述应答为对客户端第一个命令的应答，表明接受客户端的用于单播的建立连接命令，其中的Transport字段表明单播地址为10.70.144.123，客户端的RTP接收端口为20000，RTCP接收端口为20001，服务器端的RTP接收端口为50000，RTCP接收端口为50001.该路连接的会话号为12345678。The above response is a response to the first command of the client, indicating that the client is configured to establish a connection command for unicast, wherein the Transport field indicates that the unicast address is 10.70.144.123, and the RTP receiving port of the client is 20000, and the RTCP receiving port For 20001, the RTP receiving port on the server side is 50000, and the RTCP receiving port is 50001. The session number of the connection is 12345678.
Figure PCTCN2018100670-appb-000005
Figure PCTCN2018100670-appb-000005
上述应答为客户端第二个命令的应答，表明接受客户端的用于组播的建立连接命令，其中的Transport字段表明组播地址为238.225.225.225，客户端的RTP接收端口为20010，RTCP接收端口为20011，服务器端的RTP接收端口为50010，RTCP接收端口为50011.该路连接的会话号为22345678。The above response is a response from the client's second command, indicating that the client's connection establishment command for multicast is accepted. The Transport field indicates that the multicast address is 238.225.225.225, the RTP receiving port of the client is 20010, and the RTCP receiving port is In 20011, the RTP receiving port on the server side is 50010, and the RTCP receiving port is 50011. The session number of the connection is 22345678.
以上内容即为本发明实施里公开的一种会话连接的建立过程，需要说明的是，基于上述会话连接的建立方法，客户端也可以不像服务器发送用户视角信息，客户端可以自己根据用户视角信息，确定子区域，并向服务器申请对应的子区域码流。The above content is a process for establishing a session connection disclosed in the implementation of the present invention. It should be noted that, based on the establishment method of the session connection, the client may not send the user view information like the server, and the client may view the user according to the user. Information, determine the sub-area, and apply to the server for the corresponding sub-area code stream.
基于上述会话连接的建立过程，用于描述客户端的用户视角的覆盖角度信息的RTCP源描述报告格式可以如下：Based on the establishment process of the session connection described above, the RTCP source description report format for describing the coverage angle information of the user perspective of the client may be as follows:
Figure PCTCN2018100670-appb-000006
Figure PCTCN2018100670-appb-000006
在上述格式中COVERAGE值为9(本发明实施里不限于其他的定值)可以表示这个RTCP源描述报告描述的是客户端的用户视角的覆盖角度信息，H Angle表示水平方向覆盖角度(可以称为用户视角的宽度)，V Angle表示垂直方向覆盖角度(可以称为用户视角的高度)。一个用于描述客户端的用户视角的水平方向覆盖角度为110度，垂直方向覆盖角度为90度的RTCP源描述报告的示例可以如下：In the above format, the COVERAGE value is 9 (the implementation of the present invention is not limited to other fixed values), which may indicate that the RTCP source description report describes the coverage angle information of the user's user perspective, and H Angle represents the horizontal coverage angle (may be called The width of the user's perspective), V Angle represents the vertical coverage angle (which may be referred to as the height of the user's perspective). An example of an RTCP source description report for describing a user's perspective of the client with a horizontal coverage angle of 110 degrees and a vertical coverage angle of 90 degrees may be as follows:
Figure PCTCN2018100670-appb-000007
Figure PCTCN2018100670-appb-000007
用于描述用户视角的位置信息的RTCP源描述报告的格式可以如下：The format of the RTCP source description report used to describe the location information of the user's perspective can be as follows:
Figure PCTCN2018100670-appb-000008
Figure PCTCN2018100670-appb-000008
在上述格式中COVERAGE值为10(本发明实施里不限于其他的定值)可以表示这个RTCP源描述报告描述的是用户视角的中心点位置信息，X,Y,Z分别表示中心点的yaw，pitch，roll值。一个用于描述用户视角的中心点位于-45,-45,0的RTCP源描述报告示例可以如下：In the above format, the COVERAGE value is 10 (not limited to other fixed values in the implementation of the present invention), which may indicate that the RTCP source description report describes the central point position information of the user's perspective, and X, Y, and Z respectively represent the yaw of the center point. Pitch, roll value. An example of an RTCP source description report with a center point at -45, -45, 0 for describing the user's perspective can be as follows:
Figure PCTCN2018100670-appb-000009
Figure PCTCN2018100670-appb-000009
Figure PCTCN2018100670-appb-000010
Figure PCTCN2018100670-appb-000010
2)服务器接收用户视角信息。2) The server receives user view information.
3)服务器向客户端发送应答信息，该应答信息可以包括子区域的视频码流。具体的，该应答信息还可以包括全景视频的码流。3) The server sends a response message to the client, and the response message may include a video code stream of the sub-area. Specifically, the response information may further include a code stream of the panoramic video.
在具体实现过程中，子区域的视频码流可以是用于单播的子区域视频的码流，全景视频的码流可以是用于组播的全景视频的码流。具体的，可以通过上述两路会话连接分别向客户端发送高质量子区域的视频码流和低质量全景视频的码流。其中，码流可以通过RTP协议包携带，其中的子区域的视频码流中需携带该子区域(一个或多个)的中心点坐标以及水平和垂直方向覆盖角度。子区域的视频码流对应的子区域由服务器根据用户视角信息以及子区域划分方式计算得出。其中，用户视角信息用于确定用户视角对应的区域，结合用户视角对应的区域和子区域的划分方式(该划分方式指示全景视频的区域如何划分得到多个子区域)，可以确定用于发送给客户端的子区域的视频码流对应的子区域，用于发送给客户端的子区域的视频码流对应的子区域(如果子区域为至少两个的话，需要将子区域进行拼接得到新的子区域)可以覆盖用户视角对应的区域。例如当用户视角中心点位于(0,0,0)覆盖角度为水平90度垂直90时，若子区域划分方式如图2所示分为4x2总共8个子区域，则可计算得到需传输的子区域包括2,3,6,7总共4个子区域。In a specific implementation process, the video code stream of the sub-area may be a code stream of the sub-area video for unicast, and the code stream of the panoramic video may be a code stream of the panoramic video for multicast. Specifically, the video stream of the high quality sub-area and the code stream of the low-quality panoramic video may be separately sent to the client through the two-way session connection. The code stream may be carried by the RTP protocol packet, where the video code stream of the sub-area needs to carry the center point coordinates of the sub-area (one or more) and the horizontal and vertical coverage angles. The sub-region corresponding to the video code stream of the sub-area is calculated by the server according to the user view information and the sub-area division manner. The user view information is used to determine an area corresponding to the user's perspective, and the area and the sub-area corresponding to the user's perspective are combined (the division mode indicates how the area of the panoramic video is divided to obtain multiple sub-areas), and may be determined for sending to the client. The sub-area corresponding to the video code stream of the sub-area is used to send the sub-area corresponding to the video code stream of the sub-area of the client (if the sub-area is at least two, the sub-area needs to be spliced to obtain a new sub-area) Covers the area corresponding to the user's perspective. For example, when the user's perspective center point is located at (0, 0, 0) and the coverage angle is 90 degrees horizontally 90, if the sub-area division mode is divided into 4x2 and 8 sub-areas as shown in FIG. 2, the sub-area to be transmitted can be calculated. Includes a total of 4 sub-areas of 2, 3, 6, and 7.
用于携带码流的RTP包头格式可以如下：The RTP header format used to carry the code stream can be as follows:
Figure PCTCN2018100670-appb-000011
Figure PCTCN2018100670-appb-000011
其中当X为1时，RTP包头可以增加扩展格式如下：When X is 1, the RTP header can be added with the following extended format:
Figure PCTCN2018100670-appb-000012
Figure PCTCN2018100670-appb-000012
例如，一个携带中心点位于(-45,-45,0)，水平方向覆盖角度90度，垂直方向覆盖角度90度的子区域视频数据的RTP包头可以如下所示：For example, an RTP header with a sub-area video data with a center point at (-45, -45, 0), a horizontal coverage angle of 90 degrees, and a vertical coverage angle of 90 degrees can be as follows:
Figure PCTCN2018100670-appb-000013
Figure PCTCN2018100670-appb-000013
Figure PCTCN2018100670-appb-000014
Figure PCTCN2018100670-appb-000014
4)客户端接收应答信息。在具体实现方式中，该传输方法300还可以包括：客户端基于应答信息中的子区域视频码流，播放子区域视频码流对应的视频中用户视角对应的区域的视频。可选的，也可以客户端基于应答信息中的子区域视频码流，播放子区域视频码流对应的视频。可选的，也可以客户端基于应答信息中的子区域视频码流和全景视频码流，播放子区域视频码流对应的视频和全景视频码流对应的视频中子区域视频码流对应的区域之外的区域的视频。4) The client receives the response message. In a specific implementation manner, the transmission method 300 may further include: the client plays a video of an area corresponding to a user perspective in a video corresponding to the sub-area video code stream, based on the sub-area video code stream in the response information. Optionally, the client may also play the video corresponding to the sub-region video stream based on the sub-area video stream in the response message. Optionally, the client may also play the area corresponding to the sub-area video stream corresponding to the sub-area video stream corresponding to the sub-area video stream and the panoramic video stream based on the sub-area video stream and the panoramic video stream in the response information. Video outside the area.
在具体实现方式中，该传输方法300还可以包括：In a specific implementation, the transmission method 300 may further include:
5)当用户视角发生变化时，客户端向服务器发送新的RTCP源描述报告，更新用户视角信息(类似步骤1)。服务器则根据新的用户视角信息计算子区域并向客户端发送新的子区域视频码流。5) When the user's perspective changes, the client sends a new RTCP source description report to the server, updating the user perspective information (similar to step 1). The server calculates the sub-area based on the new user view information and sends a new sub-area video stream to the client.
请参阅图5，是本发明的另一实施例，内容为一种VR全景视频播放客户端400。用于实现上述方法中客户端所进行的方法，客户端400可以包括主功能模块410和显示模块420两个部分。其中，主功能模块410包括用户视角信息获取单元411，数据发送单元412，第一数据接收单元413，第一数据解码单元414，第二数据接收单元415，第二数据解码单元416和图像处理单元417。Please refer to FIG. 5, which is another embodiment of the present invention, and is a VR panoramic video playing client 400. For implementing the method performed by the client in the above method, the client 400 may include two parts: a main function module 410 and a display module 420. The main function module 410 includes a user view information acquiring unit 411, a data transmitting unit 412, a first data receiving unit 413, a first data decoding unit 414, a second data receiving unit 415, a second data decoding unit 416, and an image processing unit. 417.
用户视角信息获取单元411用于收集当前的用户视角信息。所述用户视角信息描述了客户端当前视角覆盖范围，可以包含视觉中心点信息和视觉覆盖角度信息。之后用户视角信息单元411将所述用户视角信息传输给数据发送单元412。数据发送单元412用于生成并发送请求信息，所述请求信息中包含所述用户视角信息。The user view information obtaining unit 411 is configured to collect current user view information. The user perspective information describes a current perspective coverage of the client, and may include visual center point information and visual coverage angle information. The user view information unit 411 then transmits the user view information to the data transmitting unit 412. The data sending unit 412 is configured to generate and send request information, where the request information includes the user perspective information.
第一数据接收单元413用于接收单播的FOV视频码流，并将其传输给第一数据解码单元414。所述FOV视频码流的含义请参见图2中的相关描述。第一数据解码单元414用于对接收到的FOV视频码流进行解码，并将解码得到的子块视频信息发送给图像处理单元417。The first data receiving unit 413 is configured to receive the unicast FOV video code stream and transmit it to the first data decoding unit 414. For the meaning of the FOV video code stream, please refer to the related description in FIG. 2. The first data decoding unit 414 is configured to decode the received FOV video code stream, and send the decoded sub-block video information to the image processing unit 417.
第二数据接收单元415用于接收组播的全景视频码流，并将其传输给第二数据解码单元416。所述全景视频码流中包含全景视频信息。数据解码单元416用于对接收到的全景视频码流进行解码，并将解码得到的全景视频信息发送给图像处理单元417。The second data receiving unit 415 is configured to receive the multicast panoramic video code stream and transmit it to the second data decoding unit 416. The panoramic video code stream includes panoramic video information. The data decoding unit 416 is configured to decode the received panoramic video code stream, and send the decoded panoramic video information to the image processing unit 417.
图像处理单元417用于将所接收到的子块视频信息中的内容与全景视频信息中的内容进行组合，得到图像信息，具体组合方式参见图1相关内容中的描述，此处不再赘述。之后，图像处理单元417将图像信息发送给显示模块420。显示模块420用于显示所接收到的图像信息，输出图像。The image processing unit 417 is configured to combine the content in the received sub-block video information with the content in the panoramic video information to obtain the image information. For the specific combination, refer to the description in the related content in FIG. 1 , and details are not described herein again. Thereafter, the image processing unit 417 transmits the image information to the display module 420. The display module 420 is configured to display the received image information and output an image.
请参阅图6，是本发明的另一实施例，内容为一种VR全景视频播放服务器500，用于实现上述方法中服务器所进行的方法。服务器500可以包括图像生成模块510，数据接收模块520，子块信息计算模块530，第一视频数据获取模块540，第一数据发送模块550，第二视频数据获取模块560和第二数据发送模块570。Referring to FIG. 6, another embodiment of the present invention is a VR panoramic video playing server 500, which is used to implement a method performed by a server in the foregoing method. The server 500 may include an image generation module 510, a data receiving module 520, a sub-block information calculation module 530, a first video data acquisition module 540, a first data transmission module 550, a second video data acquisition module 560, and a second data transmission module 570. .
图像生成模块510用于生成分块的全景视频内容。所述分块的全景视频可以被划分为任 意大小和数量的矩形子块。图像生成模块还向子块信息计算模块530发送全景视频分块信息，所述信息描述了全景视频的具体分块方式。The image generation module 510 is configured to generate chunked panoramic video content. The tiled panoramic video can be divided into rectangular sub-blocks of any size and number. The image generation module also transmits panoramic video segmentation information to the sub-block information calculation module 530, which describes a specific blocking mode of the panoramic video.
数据接收模块520用于接收请求信息，所述请求信息中包含用户视角信息。所述用户视角信息描述了客户端当前视角覆盖范围，可以包含视觉中心点信息和视觉覆盖角度信息。数据接收模块520将所述请求信息中的所述用户视角信息发送给子块信息计算模块530。The data receiving module 520 is configured to receive request information, where the request information includes user perspective information. The user perspective information describes a current perspective coverage of the client, and may include visual center point information and visual coverage angle information. The data receiving module 520 sends the user view information in the request information to the sub-block information calculation module 530.
子块信息计算模块530用于根据所接收到的用户视角信息和全景视频分块信息计算FOV对应子块信息。所述FOV对应子块信息描述了发送请求信息的客户端具体需要哪些子块视频。之后子块信息计算模块530将所述FOV对应子块信息发送给第一视频数据获取模块540。The sub-block information calculation module 530 is configured to calculate FOV corresponding sub-block information according to the received user view information and the panoramic video block information. The FOV corresponding sub-block information describes which sub-block videos are specifically needed by the client that sends the request information. The sub-block information calculation module 530 then sends the FOV corresponding sub-block information to the first video data acquisition module 540.
第一视频数据获取模块540用于根据所述FOV对应子块信息从分块的全景视频中提取对应子块的视频信息，编码为FOV视频码流并发送给第一数据发送模块550。第一数据发送模块550用于将所接收的FOV视频码流以单播的形式发送给客户端。The first video data obtaining module 540 is configured to extract, according to the FOV corresponding sub-block information, the video information of the corresponding sub-block from the block-divided panoramic video, and encode the video information into an FOV video code stream and send the data to the first data sending module 550. The first data sending module 550 is configured to send the received FOV video code stream to the client in a unicast manner.
第二视频数据获取模块560用于提取全景视频信息，编码为全景视频码流并发送给第二数据发送模块570。第二数据发送模块570用于将所接收的全景视频码流以组播的形式发送给客户端。The second video data acquiring module 560 is configured to extract the panoramic video information, encode the panoramic video code stream, and send the data to the second data sending module 570. The second data sending module 570 is configured to send the received panoramic video code stream to the client in a multicast manner.
请参阅图7，为本发明的另一实施例，内容为一种VR全景视频播放客户端600，用于实现上述方法中客户端所进行的方法。客户端600包括传感器610，通信端口620，处理器630和显示器640。Referring to FIG. 7, another embodiment of the present invention is a VR panoramic video playback client 600, which is used to implement a method performed by a client in the foregoing method. Client 600 includes sensor 610, communication port 620, processor 630, and display 640.
传感器610用于收集当前的用户视角信息。所述用户视角信息描述了客户端当前视角覆盖范围，可以包含视觉中心点信息和视觉覆盖角度信息。之后传感器610将所述用户视角信息传输给处理器630。The sensor 610 is used to collect current user perspective information. The user perspective information describes a current perspective coverage of the client, and may include visual center point information and visual coverage angle information. Sensor 610 then transmits the user view information to processor 630.
通信端口620用于与服务器之间收发信息，具体的，用于发送请求信息和接收视频码流。The communication port 620 is configured to send and receive information to and from the server, and specifically, to send request information and receive a video code stream.
处理器630用于生成并通过通信端口620发送请求信息，所述请求信息中包含所述的用户视角信息。The processor 630 is configured to generate and send request information through the communication port 620, where the request information includes the user view information.
处理器630还用于通过通信端口620接收组播的全景视频码流和FOV视频码流，所述的全景视频码流中包含全景视频信息，FOV视频码流中包含子块视频信息。处理器630对两个视频码流分别进行解码得到其中的全景视频数据和子块视频数据，并将子块视频数据与全景视频数据进行组合，得到图像信息，具体组合方式参见图1相关内容中的描述，此处不再赘述。之后，处理器630将图像信息发送给显示器640。显示器640用于显示所接收到的图像信息，输出图像。The processor 630 is further configured to receive the multicast panoramic video code stream and the FOV video code stream through the communication port 620, where the panoramic video code stream includes panoramic video information, and the FOV video code stream includes sub-block video information. The processor 630 separately decodes the two video code streams to obtain the panoramic video data and the sub-block video data, and combines the sub-block video data with the panoramic video data to obtain the image information. For the specific combination manner, refer to the related content in FIG. Description, no longer repeat here. Processor 630 then transmits the image information to display 640. The display 640 is for displaying the received image information and outputting an image.
请参阅图8，为本发明的又一实施例，内容为一种VR全景视频播放服务器700，用于实现上述方法中服务器所进行的方法。服务器700包括通信端口810和处理器820。Please refer to FIG. 8 , which is a VR panoramic video playing server 700 for implementing the method performed by the server in the foregoing method. Server 700 includes a communication port 810 and a processor 820.
通信端口810用于与客户端之间收发信息，具体的，用于接收请求信息和发送视频码流。The communication port 810 is configured to send and receive information to and from the client, and specifically, to receive the request information and send the video code stream.
处理器820用于生成分块的全景视频内容。所述分块的全景视频可以被划分为任意大小和数量的矩形子块。处理器820通过通信端口810接收客户端发送的请求信息，所述请求信息中包含用户视角信息。所述用户视角信息描述了客户端当前视角覆盖范围，可以包含视觉中心点信息和视觉覆盖角度信息。之后，处理器820根据所述用户视角信息和全景视频分块信息计算FOV对应子块信息，所述FOV对应子块信息描述了发送请求信息的客户端具体需要哪些子块视频。然后，根据所述FOV对应子块信息提取对应子块的视频信息并编码为FOV视频码流，提取所述全景视频编码为全景视频码流，并通过通信端口以单播的形式向客户端发送所述FOV视频码流，以组播的方式向客户端发送所述全景视频码流。The processor 820 is configured to generate chunked panoramic video content. The tiled panoramic video can be divided into rectangular sub-blocks of any size and number. The processor 820 receives the request information sent by the client through the communication port 810, where the request information includes user perspective information. The user perspective information describes a current perspective coverage of the client, and may include visual center point information and visual coverage angle information. Then, the processor 820 calculates FOV corresponding sub-block information according to the user view information and the panoramic video block information, and the FOV corresponding sub-block information describes which sub-block videos are specifically required by the client that sends the request information. Then, the video information of the corresponding sub-block is extracted according to the FOV corresponding sub-block information and encoded into a FOV video code stream, and the panoramic video code is extracted into a panoramic video code stream, and sent to the client in a unicast manner through the communication port. The FOV video code stream transmits the panoramic video code stream to a client in a multicast manner.
在一个或多个实施例中，所描述的高质量分块视频和低质量全景视频相比，当分辨率相 同时，高质量分块视频的码率要高于低质量全景视频。In one or more embodiments, the high quality tiled video has a higher code rate than the low quality panoramic video when the resolution is the same as the high quality tiled video and the low quality panoramic video.
在一个或多个实例中，所描述的功能可以硬件、软件、固件或其任何组合来实施。如果以软件实施，则功能可作为一个或多个指令或代码而存储于计算机可读媒体上或经由计算机可读媒体而发送，且通过基于硬件的处理单元执行。计算机可读媒体可包含计算机可读存储媒体(其对应于例如数据存储媒体等有形媒体)或通信媒体，通信媒体包含(例如)根据通信协议促进计算机程序从一处传送到另一处的任何媒体。以此方式，计算机可读媒体大体上可对应于(1)非瞬时的有形计算机可读存储媒体，或(2)例如信号或载波等通信媒体。数据存储媒体可为可由一个或多个计算机或一个或多个处理器存取以检索指令、代码及/或数据结构以用于实施本发明中所描述的技术的任何可用媒体。计算机程序产品可包含计算机可读媒体。In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted as one or more instructions or code via a computer-readable medium and executed by a hardware-based processing unit. The computer readable medium can comprise a computer readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium comprising, for example, any medium that facilitates transfer of the computer program from one place to another in accordance with a communication protocol. . In this manner, computer readable media generally may correspond to (1) a non-transitory tangible computer readable storage medium, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for use in carrying out the techniques described herein. The computer program product can comprise a computer readable medium.
可由例如一个或多个数字信号处理器(DSP)、通用微处理器、专用集成电路(ASIC)、现场可编程逻辑阵列(FPGA)或其它等效集成或离散逻辑电路等一个或多个处理器来执行指令。因此，如本文中所使用的术语“处理器”可指代前述结构或适于实施本文中所描述的技术的任何其它结构中的任一者。One or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits To execute the instructions. Accordingly, the term "processor," as used herein, may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein.
应理解，说明书通篇中提到的“一个实施例”或“一实施例”意味着与实施例有关的特定特征、结构或特性包括在本发明的至少一个实施例中。因此，在整个说明书各处出现的“在一个实施例中”或“在一实施例中”未必一定指相同的实施例。此外，这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施例中。It is to be understood that the phrase "one embodiment" or "an embodiment" or "an" Thus, "in one embodiment" or "in an embodiment" or "an" In addition, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
在本发明的各种实施例中，应理解，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本发明实施例的实施过程构成任何限定。In the various embodiments of the present invention, it should be understood that the size of the sequence numbers of the above processes does not mean the order of execution, and the order of execution of each process should be determined by its function and internal logic, and should not be taken to the embodiments of the present invention. The implementation process constitutes any limitation.
在本申请所提供的实施例中，应理解，“与A相应的B”表示B与A相关联，根据A可以确定B。但还应理解，根据A确定B并不意味着仅仅根据A确定B，还可以根据A和/或其它信息确定B。In the embodiments provided herein, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined from A. However, it should also be understood that determining B from A does not mean that B is only determined based on A, and that B can also be determined based on A and/or other information.
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明的范围。Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both, for clarity of hardware and software. Interchangeability, the composition and steps of the various examples have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods for implementing the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的***、装置和单元的具体工作过程，可以参考所述方法实施例中的对应过程，在此不再赘述。A person skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the system, the device and the unit described above can refer to the corresponding process in the method embodiment, and details are not described herein again.
在本申请所提供的几个实施例中，应该理解到，所揭露的***、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个***，或一些特征可以忽略，或不执行。In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Claims (25)

  1. 一种VR全景视频播放客户端，其特征在于，包括主功能模块和显示模块：A VR panoramic video playing client, comprising: a main function module and a display module:
    所述主功能模块用于收集并向服务器发送用户视角信息，所述用户视角信息用于确定第一视频数据,所述第一视频数据对应用户视角范围区域FOV内视频内容；接收服务器发送的单播的第一视频码流，解码所述第一视频码流得到第一视频数据；接收服务器发送的组播的第二视频码流，解码所述第二视频码流得到第二视频数据，所述第二视频数据包括全景视频内容；将所述第一视频数据替代所述第二视频数据中与FOV内视频内容对应的部分，得到图像信息，并发送给所述显示模块；The main function module is configured to collect and send user view information to the server, where the user view information is used to determine first video data, where the first video data corresponds to video content in a user view range area FOV; Transmitting the first video code stream, decoding the first video code stream to obtain first video data; receiving a multicast second video code stream sent by the server, and decoding the second video code stream to obtain second video data, The second video data includes the panoramic video content; the first video data is replaced with the portion of the second video data corresponding to the video content in the FOV, and the image information is obtained and sent to the display module;
    所述显示模块用于接收并显示所述图像信息。The display module is configured to receive and display the image information.
  2. 如权利要求1所述的VR全景视频播放客户端，其特征在于，所述主功能模块包括：The VR panoramic video playing client according to claim 1, wherein the main function module comprises:
    用户视角信息获取单元，用于收集用户视角信息；a user view information obtaining unit, configured to collect user view information;
    数据发送单元，用于将所述用户视角信息发送给服务器，所述用户视角信息用于确定所述第一视频数据；a data sending unit, configured to send the user view information to a server, where the user view information is used to determine the first video data;
    第一数据接收单元，用于接收服务器发送的单播的第一视频码流，并发送给第一数据解码单元；a first data receiving unit, configured to receive a unicast first video code stream sent by the server, and send the first video code stream to the first data decoding unit;
    第一数据解码单元，用于解码所述第一视频码流得到第一视频数据，并将所述第一视频数据发送给图像处理单元；a first data decoding unit, configured to decode the first video code stream to obtain first video data, and send the first video data to an image processing unit;
    第二数据接收单元，用于接收服务器发送的组播的第二视频码流，并发送给第二数据解码单元；a second data receiving unit, configured to receive a multicast second video code stream sent by the server, and send the second video code stream to the second data decoding unit;
    第二数据解码单元，用于解码所述第二视频码流得到所述第二视频数据，并将所述第二视频数据发送给图像处理单元；a second data decoding unit, configured to decode the second video code stream to obtain the second video data, and send the second video data to an image processing unit;
    图像处理单元，用于将所述第一视频数据替代所述第二视频数据中的与FOV内视频内容相应的视频信息，得到图像信息，并将所述图像信息发送给显示模块。And an image processing unit, configured to replace the video information corresponding to the video content in the FOV in the second video data to obtain image information, and send the image information to the display module.
  3. 如权利要求1或2所述的VR全景视频播放客户端，其特征在于，所述第一视频数据的图像质量高于所述第二视频数据的图像质量。The VR panoramic video playback client according to claim 1 or 2, wherein the image quality of the first video data is higher than the image quality of the second video data.
  4. 如权利要求1、2或3所述的VR全景视频播放客户端，其特征在于，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息；所述用户视角信息用于确定第一视频数据具体为：所述视觉中心点信息和所述视觉覆盖角度信息用于确定用户视角的具体覆盖范围，所述用户的视角具体覆盖范围用于确定第一视频数据。The VR panoramic video playing client according to claim 1, 2 or 3, wherein the user perspective information includes visual center point information and visual coverage angle information; and the user perspective information is used to determine the first video data. Specifically, the visual center point information and the visual coverage angle information are used to determine a specific coverage of the user's perspective, and the specific coverage of the user's perspective is used to determine the first video data.
  5. 如权利要求1-4中任意一项所述的VR全景视频播放客户端，其特征在于，所述全景视频内容分为多个子块，所述FOV内视频内容指FOV所覆盖到的子块的视频内容。The VR panoramic video playing client according to any one of claims 1 to 4, wherein the panoramic video content is divided into a plurality of sub-blocks, and the video content in the FOV refers to a sub-block covered by the FOV. Video content.
  6. 如权利要求1-5中任意一项所述的VR全景视频播放客户端，其特征在于，所述第一视频数据由所述服务器根据所述用户视角信息获得。The VR panoramic video playing client according to any one of claims 1 to 5, wherein the first video data is obtained by the server according to the user view information.
  7. 一种VR全景视频播放服务器，其特征在于，所述服务器包括：A VR panoramic video playing server, wherein the server comprises:
    图像生成模块，用于生成VR全景视频内容，且所述全景视频被分为若干个子块；An image generating module, configured to generate VR panoramic video content, and the panoramic video is divided into several sub-blocks;
    数据接收模块，用于接收用户视角信息；a data receiving module, configured to receive user view information;
    子块信息计算模块，用于根据全景视频分块信息和所述用户视角信息确定FOV对应子块信息，所述FOV对应子块信息描述了所述用户视角所覆盖的子块；a sub-block information calculation module, configured to determine FOV corresponding sub-block information according to the panoramic video block information and the user view information, where the FOV corresponding sub-block information describes a sub-block covered by the user view;
    第一视频数据获取模块，用于根据所述FOV对应子块信息从所述全景视频中提取与所述子块信息对应的子块，并编码为第一视频码流；a first video data acquiring module, configured to extract a sub-block corresponding to the sub-block information from the panoramic video according to the FOV corresponding sub-block information, and encode the first video code stream;
    第一数据发送模块，用于以单播形式向客户端发送所述第一视频码流；a first data sending module, configured to send the first video code stream to a client in a unicast manner;
    第二视频数据获取模块，用于提取全景视频并编码为第二视频码流；a second video data acquiring module, configured to extract a panoramic video and encode the second video code stream;
    第二数据发送模块，用于以组播形式向客户端发送所述第二视频码流。And a second data sending module, configured to send the second video code stream to the client in a multicast manner.
  8. 如权利要求7所述的VR全景视频播放服务器，其特征在于，所述第一视频数据的图像质量高于所述第二视频数据的图像质量。The VR panoramic video playing server according to claim 7, wherein an image quality of said first video data is higher than an image quality of said second video data.
  9. 如权利要求7或8所述的VR全景视频播放服务器，其特征在于，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息；子块信息计算模块用于根据所述视觉中心点信息和所述视觉覆盖角度信息确定用户视角的具体覆盖范围，并根据全景视频分块方式和所述用户视角的具体覆盖范围确定FOV对应子块信息。The VR panoramic video playing server according to claim 7 or 8, wherein the user view information includes visual center point information and visual coverage angle information; and the sub-block information calculating module is configured to use the visual center point information and The visual coverage angle information determines a specific coverage range of the user perspective, and determines FOV corresponding sub-block information according to the panoramic video blocking manner and the specific coverage of the user perspective.
  10. 一种VR全景视频播放客户端，其特征在于，包括传感器、处理器、通信端口和显示器，其中：A VR panoramic video playback client, comprising a sensor, a processor, a communication port and a display, wherein:
    所述传感器用于收集用户视角信息，并发送给所述处理器；The sensor is configured to collect user view information and send the information to the processor;
    所述处理器用于通过所述端口发送所述用户视角信息，所述用户视角信息用于确定第一视频数据，所述第一视频数据对应用户视角范围区域FOV内的视频内容；通过所述通信端口接收服务器发送的单播的第一视频码流，解码所述第一视频码流得到第一视频数据；通过所述通信端口接收服务器发送的组播的第二视频码流，解码所述第二视频码流得到第二视频数据，所述第二视频数据包括全景视频内容；将所述第一视频数据替代所述第二视频数据中与FOV内视频内容对应的部分，得到图像信息，并发送给所述显示器；The processor is configured to send the user view information through the port, where the user view information is used to determine first video data, where the first video data corresponds to video content in a user view range area FOV; The port receives the unicast first video code stream sent by the server, decodes the first video code stream to obtain the first video data, and receives the multicast second video code stream sent by the server through the communication port, and decodes the first The second video stream obtains the second video data, the second video data includes the panoramic video content; the first video data is replaced with the portion of the second video data corresponding to the video content in the FOV, to obtain image information, and Sended to the display;
    所述通信端口，用于与服务器收发信息；The communication port is configured to send and receive information with a server;
    所述显示器，用于显示所述图像信息。The display is configured to display the image information.
  11. 如权利要求10所述的VR全景视频播放客户端，其特征在于，所述第一视频数据的图像质量高于所述第二视频数据的图像质量。The VR panoramic video playing client according to claim 10, wherein the image quality of the first video data is higher than the image quality of the second video data.
  12. 如权利要求10或11所述的VR全景视频播放客户端，其特征在于，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息；所述所述用户视角信息用于确定第一视频数据具体为：所述视觉中心点信息和所述视觉覆盖角度信息用于确定用户视角的具体覆盖范围，所述用户的视角具体覆盖范围用于确定第一视频数据。The VR panoramic video playing client according to claim 10 or 11, wherein the user perspective information includes visual center point information and visual coverage angle information; and the user perspective information is used to determine the first video data. Specifically, the visual center point information and the visual coverage angle information are used to determine a specific coverage of the user's perspective, and the specific coverage of the user's perspective is used to determine the first video data.
  13. 如权利要求10-12中任意一项所述的VR全景视频播放客户端，其特征在于，所述全景视频内容分为多个子块，所述FOV内视频内容指FOV所覆盖到的子块的视频内容。The VR panoramic video playing client according to any one of claims 10 to 12, wherein the panoramic video content is divided into a plurality of sub-blocks, and the video content in the FOV refers to a sub-block covered by the FOV. Video content.
  14. 如权利要求10-13中任意一项所述的VR全景视频播放客户端，其特征在于，所述第一视频数据由所述服务器根据所述用户视角信息获得。The VR panoramic video playing client according to any one of claims 10-13, wherein the first video data is obtained by the server according to the user perspective information.
  15. 一种VR全景视频播放服务器，其特征在于，包括通信端口和处理器，其中：A VR panoramic video playing server, comprising: a communication port and a processor, wherein:
    所述通信端口用于与客户端收发信息；The communication port is configured to send and receive information with a client;
    所述处理器用于生成VR全景视频内容，且所述全景视频内容被分为若干个子块；通过 所述通信端口接收客户端发送的用户视角信息；根据全景视频的分块方式和所述用户视角信息确定FOV对应子块信息，所述FOV对应子块信息描述了用户视角所覆盖的子块；根据所述FOV对应子块信息从所述全景视频中提取与所述子块信息对应的子块，并编码为第一视频码流；提取全景视频信息并编码为第二视频码流；通过所述通信端口以单播形式向客户端发送所述第一视频码流，通过所述通信端口以组播形式向客户端发送所述第二视频码流。The processor is configured to generate VR panoramic video content, and the panoramic video content is divided into a plurality of sub-blocks; the user view information sent by the client is received through the communication port; and the block mode of the panoramic video and the user view are Determining FOV corresponding sub-block information, the FOV corresponding sub-block information describing a sub-block covered by a user view; extracting a sub-block corresponding to the sub-block information from the panoramic video according to the FOV corresponding sub-block information And encoding the first video code stream; extracting the panoramic video information and encoding the second video code stream; transmitting, by the communication port, the first video code stream to the client in a unicast manner, through the communication port The multicast form sends the second video code stream to the client.
  16. 如权利要求15所述的VR全景视频播放服务器，其特征在于，所述第一视频数据的图像质量高于所述第二视频数据的图像质量。The VR panoramic video playing server according to claim 15, wherein an image quality of said first video data is higher than an image quality of said second video data.
  17. 如权利要求15或16所述的VR全景视频播放服务器，其特征在于，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息；所述处理器还用于根据所述视觉中心点信息和所述视觉覆盖角度信息确定用户视角的具体覆盖范围，并根据全景视频分块方式和所述用户视角的具体覆盖范围确定FOV对应子块信息。The VR panoramic video playing server according to claim 15 or 16, wherein the user perspective information includes visual center point information and visual coverage angle information; the processor is further configured to: according to the visual center point information and The visual coverage angle information determines a specific coverage range of the user perspective, and determines FOV corresponding sub-block information according to the panoramic video blocking manner and the specific coverage of the user perspective.
  18. 一种VR全景视频播放方法，其特征在于：A VR panoramic video playing method, characterized in that:
    客户端收集并向服务器发送用户视角信息，所述用户视角信息用于确定第一视频数据,所述第一视频数据对应用户视角范围区域FOV内的视频内容；The client collects and sends user view information to the server, where the user view information is used to determine first video data, where the first video data corresponds to video content in a user view range area FOV;
    客户端接收单播的第一视频码流，解码所述第一视频码流得到第一视频数据；接收组播的第二视频码流，解码所述第二视频码流得到第二视频数据，所述第二视频数据包括全景视频内容；The client receives the first video stream of the unicast, decodes the first video stream to obtain the first video data, receives the second video stream of the multicast, and decodes the second video stream to obtain the second video data. The second video data includes panoramic video content;
    将所述第一视频数据替代所述第二视频数据中与FOV内视频内容对应的部分，得到图像信息，并显示所述图像信息中的图像内容。Substituting the first video data for a portion of the second video data corresponding to the video content in the FOV, obtaining image information, and displaying the image content in the image information.
  19. 如权利要求18所述的VR全景视频播放方法，其特征在于，所述第一视频数据的图像质量高于所述第二视频数据的图像质量。The VR panoramic video playing method according to claim 18, wherein the image quality of the first video data is higher than the image quality of the second video data.
  20. 如权利要求18或19所述的VR全景视频播放方法，其特征在于，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息；所述用户视角信息用于确定第一视频数据具体为：所述视觉中心点信息和所述视觉覆盖角度信息用于确定用户视角的具体覆盖范围，所述用户的视角具体覆盖范围用于确定第一视频数据。The VR panoramic video playing method according to claim 18 or 19, wherein the user perspective information includes visual center point information and visual coverage angle information; and the user perspective information is used to determine that the first video data is: The visual center point information and the visual coverage angle information are used to determine a specific coverage of a user's perspective, and the user's perspective specific coverage is used to determine the first video data.
  21. 如权利要求18-20中任意一项所述的VR全景视频播放方法，其特征在于，所述全景视频内容分为多个子块，所述FOV内视频内容指FOV所覆盖到的子块的视频内容。The VR panoramic video playing method according to any one of claims 18 to 20, wherein the panoramic video content is divided into a plurality of sub-blocks, and the video content in the FOV refers to a video of a sub-block covered by the FOV. content.
  22. 如权利要求18-21中任意一项所述的VR全景视频播放方法，其特征在于，所述第一视频数据由所述服务器根据所述用户视角信息获得。The VR panoramic video playing method according to any one of claims 18 to 21, wherein the first video data is obtained by the server according to the user perspective information.
  23. 一种VR全景视频播放方法，其特征在于：A VR panoramic video playing method, characterized in that:
    服务器生成VR全景视频内容，所述全景视频被分为若干个子块；The server generates VR panoramic video content, and the panoramic video is divided into several sub-blocks;
    接收客户端发送的用户视角信息；Receiving user perspective information sent by the client;
    根据全景视频分块信息和所述用户视角信息确定FOV对应子块信息，所述FOV对应子块信息描述了所述用户视角所覆盖的对应子块；Determining FOV corresponding sub-block information according to the panoramic video segmentation information and the user view information, where the FOV corresponding sub-block information describes a corresponding sub-block covered by the user perspective;
    根据所述FOV对应子块信息从所述全景视频中提取对应子块，编码为第一视频码流，并以单播形式发送所述第一视频码流；And extracting a corresponding sub-block from the panoramic video according to the FOV corresponding sub-block information, encoding the first video code stream, and transmitting the first video code stream in a unicast manner;
    提取所述全景视频，编码为第二视频码流，并以组播形式发送所述第二视频码流；Extracting the panoramic video, encoding the second video code stream, and transmitting the second video code stream in a multicast manner;
    所述客户端接收所述以单播形式发送的所述第一视频码流和所述以组播形式发送的所述第二视频码流，解码并播放所述第一视频码流和所述第二视频码流中的内容。Receiving, by the client, the first video code stream sent in a unicast manner and the second video code stream sent in a multicast form, decoding and playing the first video code stream and the The content in the second video stream.
  24. 如权利要求23所述的VR全景视频播放方法，其特征在于，所述第一视频数据的图像质量高于所述第二视频数据的图像质量。The VR panoramic video playing method according to claim 23, wherein the image quality of the first video data is higher than the image quality of the second video data.
  25. 如权利要求23或24所述的VR全景视频播放方法，其特征在于，所述用户视角信息包括视觉中心点信息和视觉覆盖角度信息；所述服务器根据所述视觉中心点信息和所述视觉覆盖角度信息确定用户视角的具体覆盖范围，并根据全景视频分块方式和所述用户视角的具体覆盖范围确定FOV对应子块信息。The VR panoramic video playing method according to claim 23 or 24, wherein the user perspective information includes visual center point information and visual coverage angle information; and the server according to the visual center point information and the visual coverage The angle information determines a specific coverage of the user's perspective, and determines the FOV corresponding sub-block information according to the panoramic video blocking mode and the specific coverage of the user perspective.
PCT/CN2018/100670 2018-02-13 2018-08-15 Transmission control method WO2019157803A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810148373.1A CN110149542B (en) 2018-02-13 2018-02-13 Transmission control method
CN201810148373.1 2018-02-13

Publications (1)

Publication Number Publication Date
WO2019157803A1 true WO2019157803A1 (en) 2019-08-22

Family

ID=67589077

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/100670 WO2019157803A1 (en) 2018-02-13 2018-08-15 Transmission control method

Country Status (2)

Country Link
CN (1) CN110149542B (en)
WO (1) WO2019157803A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112714315A (en) * 2019-10-24 2021-04-27 上海交通大学 Layered buffering method and system based on panoramic video
CN114900508A (en) * 2022-05-16 2022-08-12 深圳市瑞云科技有限公司 Method for transmitting VR application data based on webrtc
CN114979762A (en) * 2022-04-12 2022-08-30 北京字节跳动网络技术有限公司 Video downloading and transmission method, device, terminal equipment, server and medium

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021087843A1 (en) * 2019-11-07 2021-05-14 Intel Corporation Heterogeneous real-time streaming and decoding of ultra-high resolution video content
CN110944239A (en) * 2019-11-28 2020-03-31 重庆爱奇艺智能科技有限公司 Video playing method and device
CN111698513A (en) * 2020-05-22 2020-09-22 深圳威尔视觉传媒有限公司 Image acquisition method, display method, device, electronic equipment and storage medium
CN112153401B (en) * 2020-09-22 2022-09-06 咪咕视讯科技有限公司 Video processing method, communication device and readable storage medium
CN112130667A (en) * 2020-09-25 2020-12-25 深圳市佳创视讯技术股份有限公司 Interaction method and system for ultra-high definition VR (virtual reality) video
CN114697876B (en) * 2020-12-30 2023-08-22 华为技术有限公司 Local area network screen projection method and device and electronic equipment
CN115314730A (en) * 2022-08-10 2022-11-08 中国电信股份有限公司 Video streaming transmission method and device applied to virtual reality VR scene
CN116668779B (en) * 2023-08-01 2023-10-10 中国电信股份有限公司 Virtual reality view field distribution method, system, device, equipment and medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150181294A1 (en) * 2013-12-19 2015-06-25 Electronics And Telecommunications Research Institute Method and system for providing and receiving multi-screen based content
CN106385587A (en) * 2016-09-14 2017-02-08 三星电子（中国）研发中心 Method, device and system for sharing virtual reality view angle
CN106919248A (en) * 2015-12-26 2017-07-04 华为技术有限公司 It is applied to the content transmission method and equipment of virtual reality
CN107135237A (en) * 2017-07-07 2017-09-05 三星电子（中国）研发中心 A kind of implementation method and device that targets improvement information is presented
CN107529064A (en) * 2017-09-04 2017-12-29 北京理工大学 A kind of self-adaptive encoding method based on VR terminals feedback

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6331869B1 (en) * 1998-08-07 2001-12-18 Be Here Corporation Method and apparatus for electronically distributing motion panoramic images
US6466254B1 (en) * 1997-05-08 2002-10-15 Be Here Corporation Method and apparatus for electronically distributing motion panoramic images
CN101119166A (en) * 2006-07-31 2008-02-06 北京凯诚高清电子技术有限公司 Multiplex real-time network monitoring method and apparatus
CN105005964B (en) * 2015-06-30 2018-11-16 南京师范大学 Geographic scenes panorama sketch rapid generation based on video sequence image
CN106856484A (en) * 2015-12-08 2017-06-16 南京迈瑞生物医疗电子有限公司 Control information transmission method based on Digital Operating Room, apparatus and system
US10380800B2 (en) * 2016-04-18 2019-08-13 Disney Enterprises, Inc. System and method for linking and interacting between augmented reality and virtual reality environments

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150181294A1 (en) * 2013-12-19 2015-06-25 Electronics And Telecommunications Research Institute Method and system for providing and receiving multi-screen based content
CN106919248A (en) * 2015-12-26 2017-07-04 华为技术有限公司 It is applied to the content transmission method and equipment of virtual reality
CN106385587A (en) * 2016-09-14 2017-02-08 三星电子（中国）研发中心 Method, device and system for sharing virtual reality view angle
CN107135237A (en) * 2017-07-07 2017-09-05 三星电子（中国）研发中心 A kind of implementation method and device that targets improvement information is presented
CN107529064A (en) * 2017-09-04 2017-12-29 北京理工大学 A kind of self-adaptive encoding method based on VR terminals feedback

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112714315A (en) * 2019-10-24 2021-04-27 上海交通大学 Layered buffering method and system based on panoramic video
CN114979762A (en) * 2022-04-12 2022-08-30 北京字节跳动网络技术有限公司 Video downloading and transmission method, device, terminal equipment, server and medium
CN114979762B (en) * 2022-04-12 2024-06-07 北京字节跳动网络技术有限公司 Video downloading and transmitting method and device, terminal equipment, server and medium
CN114900508A (en) * 2022-05-16 2022-08-12 深圳市瑞云科技有限公司 Method for transmitting VR application data based on webrtc
CN114900508B (en) * 2022-05-16 2023-08-29 深圳市瑞云科技有限公司 Method for transmitting VR application data based on webrtc

Also Published As

Publication number Publication date
CN110149542A (en) 2019-08-20
CN110149542B (en) 2021-12-03

Similar Documents

Publication Publication Date Title
WO2019157803A1 (en) Transmission control method
US11303826B2 (en) Method and device for transmitting/receiving metadata of image in wireless communication system
US11044455B2 (en) Multiple-viewpoints related metadata transmission and reception method and apparatus
US10469820B2 (en) Streaming volumetric video for six degrees of freedom virtual reality
WO2018014495A1 (en) Real-time panoramic live broadcast network camera and system and method
CN110519652B (en) VR video playing method, terminal and server
US9635252B2 (en) Live panoramic image capture and distribution
CN104980697A (en) Video transmission method for web camera
WO2021074005A1 (en) Immersive viewport dependent multiparty video communication
KR20170008725A (en) Methods and apparatus for streaming content
WO2018126702A1 (en) Streaming media transmission method applied to virtual reality technology and client
US10958950B2 (en) Method, apparatus and stream of formatting an immersive video for legacy and immersive rendering devices
US20190320156A1 (en) Multiple-viewpoints related metadata transmission and reception method and apparatus
WO2014044059A1 (en) Method, device and system for video conference recording and playing
JP2002330440A (en) Image transmission method, program for the image transmission method, recording medium for recording the program for the image transmission method, and image transmitter
WO2017198143A1 (en) Video processing method, video playback method, set-top box, and vr apparatus
CN110798707B (en) Method, client and server for transmitting media data
WO2019048733A1 (en) Transmission of video content based on feedback
US20180194465A1 (en) System and method for video broadcasting
CN111385590A (en) Live broadcast data processing method and device and terminal
Zeng et al. A new architecture of 8k vr fov video end-to-end technology
JP2024519747A (en) Split rendering of extended reality data over 5G networks
EP3386203B1 (en) Signalling of auxiliary content for a broadcast signal
US20230146498A1 (en) A Method, An Apparatus and a Computer Program Product for Video Encoding and Video Decoding
US20240007603A1 (en) Method, an apparatus and a computer program product for streaming of immersive video

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18906358

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18906358

Country of ref document: EP

Kind code of ref document: A1