CN111064985A - System, method and device for realizing video streaming - Google Patents

System, method and device for realizing video streaming Download PDF

Info

Publication number
CN111064985A
CN111064985A CN201811203090.9A CN201811203090A CN111064985A CN 111064985 A CN111064985 A CN 111064985A CN 201811203090 A CN201811203090 A CN 201811203090A CN 111064985 A CN111064985 A CN 111064985A
Authority
CN
China
Prior art keywords
data
streaming
software
application
equipment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201811203090.9A
Other languages
Chinese (zh)
Inventor
冉瑞元
张佳宁
张道宁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nolo Co ltd
Original Assignee
Nolo Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nolo Co ltd filed Critical Nolo Co ltd
Priority to CN201811203090.9A priority Critical patent/CN111064985A/en
Priority to PCT/CN2019/111315 priority patent/WO2020078354A1/en
Priority to US17/286,387 priority patent/US11500455B2/en
Publication of CN111064985A publication Critical patent/CN111064985A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/4363—Adapting the video or multiplex stream to a specific local network, e.g. a IEEE 1394 or Bluetooth® network
    • H04N21/43637—Adapting the video or multiplex stream to a specific local network, e.g. a IEEE 1394 or Bluetooth® network involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/443—OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video

Abstract

The invention discloses a method and a system for realizing video streaming, wherein the method comprises the following steps: acquiring attitude data and positioning data; sending the acquired attitude data and the positioning data to a VR application for picture rendering; and acquiring the rendered picture, and sending the rendered picture to VR equipment for display. The method ensures that the PC is responsible for running the VR application, the VR equipment is responsible for only displaying the picture, and the picture processing capability of the PC can be utilized to ensure that the VR equipment obtains a high-quality picture.

Description

System, method and device for realizing video streaming
Technical Field
The invention relates to a system for realizing video streaming; meanwhile, the video streaming method realized by the system belongs to the technical field of virtual reality.
Background
Video streaming (video streaming) is a video playing technology that compresses a series of video data and then transmits the compressed video data in segments to transmit video and audio on a network in real time for viewing. The video streaming technology is used by the players, such as the QuickTime Player, Real Player, etc., which have been popular. At present, with the deep development of industries such as online games, live games and the like, video streaming is more and more widely applied.
Virtual Reality (VR) is a Virtual environment generated by a modern high-technology means with a computer technology as a core, and a user obtains the same feeling as the real world through vision, hearing, touch and the like by means of special input/output devices. The virtual reality technology is a high-level man-machine interaction technology which comprehensively applies computer graphics, man-machine interface technology, sensor technology, artificial intelligence and the like, makes a vivid artificial simulation environment and can effectively simulate various perceptions of a human in a natural environment.
Head-mounted displays (head displays for short) are one of the core devices for virtual reality display, and are mainly classified into three types: the first type is PC head display, which needs to connect the head display with the PC by a data line, and the head display can process data on the PC and display the picture on the head display; the second type is an integrated head display, which is essentially a VR device with operation storage capability, display capability and positioning capability, and does not need external equipment, so the integrated head display is also called as a VR all-in-one machine; the third type is a mobile head display, which is used by matching a VR box with a mobile terminal.
For the integrated head display and the mobile head display, usually, a user can only use the application matched with the head display, the applications are installed in the head display and the mobile phone in a pre-installed mode, but the applications which can only be used on the PC head display cannot be used, namely, the applications which can not be used for data processing at the PC end cannot be used, and the application range of the applications is limited.
Disclosure of Invention
In view of the deficiencies of the prior art, the primary technical problem to be solved by the present invention is to provide a system for implementing video streaming.
Another technical problem to be solved by the present invention is to provide a method for implementing video streaming.
In order to achieve the purpose, the invention adopts the following technical scheme:
according to a first aspect of the embodiments of the present invention, a system for implementing video streaming is provided, including a terminal and a VR device;
the terminal is provided with a server of application platform software and streaming software;
the VR equipment is provided with a client of streaming software, and the client of the streaming software sends attitude data to a server of the streaming software on the terminal; and the server side of the streaming software sends the gesture data to the application platform software, and the application platform software renders a picture.
According to a second aspect of the embodiments of the present invention, a system for implementing video streaming is provided, including a terminal, a VR device, and a position tracking apparatus;
the terminal is provided with a server of application platform software and streaming software;
the positioning tracking device is used for acquiring positioning data and sending the positioning data to VR equipment;
the VR equipment is provided with a client of streaming software, and the client of the streaming software sends attitude data and the positioning data to a server of the streaming software on the terminal; and the server side of the streaming software sends the posture data and the positioning data to the application platform software, and the application platform software renders a picture.
Preferably, the server of the streaming software comprises a control interface and a server driver, and the server driver is loaded when the application platform software is started on the terminal.
Preferably, the server side of the streaming software acquires the rendered pictures and sends the rendered pictures to the client side of the streaming software; and the client of the streaming software sends the pictures to the VR equipment for display.
Preferably, the client of the streaming software sends the attitude data and/or the positioning data to the server of the streaming software in a wireless manner, wherein the wireless manner is 5G.
According to a third aspect of the embodiments of the present invention, there is provided a method for implementing video streaming, including the following steps:
acquiring attitude data of VR equipment;
sending the acquired attitude data to a VR application for picture rendering;
and acquiring the rendered picture, and sending the rendered picture to VR equipment for display.
Preferably, the acquiring of the posture data of the VR device includes:
a client of streaming software on the VR equipment acquires attitude data of the VR equipment;
and the streaming software server installed on the terminal acquires the attitude data sent by the client of the streaming software in a 5G wireless transmission mode through UDP.
Preferably, the step of sending the acquired gesture data to the VR application for screen rendering includes:
sending the acquired attitude data to OpenVR, and transmitting the attitude data to VR application through OpenVR;
rendering a picture through an application engine according to the posture data and the application logic obtained by VR application;
and storing the data rendered by the application engine in a video memory of the video card.
According to a fourth aspect of the embodiments of the present invention, there is provided a method for implementing video streaming, including the following steps:
acquiring attitude data and positioning data;
sending the acquired attitude data and the positioning data to a VR application for picture rendering;
and acquiring the rendered picture, and sending the rendered picture to VR equipment for display.
Preferably, the acquiring the attitude data and the positioning data comprises the following steps:
acquiring positioning data and/or posture data of a user through a positioning and tracking device;
sending the positioning data and/or the posture data acquired by the positioning and tracking device to VR equipment;
and the streaming software server acquires positioning data and attitude data sent by the VR equipment through UDP.
Preferably, the step of sending the acquired pose data and the positioning data to the VR application for image rendering includes the steps of:
sending the attitude data and the positioning data acquired by the streaming software server to OpenVR, and transmitting the attitude data and the positioning data to VR application through OpenVR;
rendering a picture through an application engine according to the positioning data, the posture data and the application logic obtained by the VR application;
and storing the data rendered by the application engine in a video memory of the video card.
Preferably, the step of obtaining the rendered picture and sending the rendered picture to the VR device for display includes the following steps:
acquiring texture data corresponding to a rendered picture, and encoding a frame of picture into a plurality of data packets;
and sending the encoded multiple data packets to the VR device for decoding and displaying.
Preferably, the method further comprises the steps of obtaining control information, sending the obtained control information to the VR application, and performing picture rendering;
wherein the control information is from the VR device or from a position tracking device.
According to a fifth aspect of the embodiments of the present invention, there is provided an apparatus for implementing video streaming, including a processor and a memory, where the processor is configured to execute a program for implementing video streaming stored in the memory to implement the method for implementing video streaming as described above.
According to the method for realizing the video streaming, provided by the invention, the service of streaming software acquires positioning data and attitude data; sending the acquired attitude data and the positioning data to a VR application for picture rendering; and acquiring the rendered picture, and sending the rendered picture to VR equipment for display. The method is characterized in that a server of streaming software is installed on a PC terminal, and the method ensures that a PC is responsible for running VR application and only picture display is required for VR equipment; therefore, the picture processing can be carried out through the hardware of the PC, and satisfactory pictures can be obtained on the screen of the VR equipment.
Drawings
Fig. 1 is a schematic structural diagram of a system for implementing video streaming according to a first embodiment of the present invention;
FIG. 2 is a flowchart illustrating a method for video streaming according to a first embodiment of the present invention;
fig. 3 is a schematic structural diagram of a system for implementing video streaming according to a second embodiment of the present invention;
fig. 4 is a flowchart of a method for implementing video streaming according to a second embodiment of the present invention.
Detailed Description
The technical contents of the invention are described in detail below with reference to the accompanying drawings and specific embodiments.
Example one
As shown in fig. 1, the system for implementing video streaming provided by the present invention includes a terminal and a VR device.
The terminal is provided with a server of application platform software and streaming software; in the embodiment provided by the present invention, the terminal is exemplified by a pc (personal computer), and may also be a terminal with data processing capability, such as a tablet computer, a smart television, and a smart phone. The application platform software installed on the PC is, illustratively, the steadvr platform software (corresponding APP on the smartphone). Of course, other application platforms such as a VIVEPORT platform, a HYPEREAL platform, an Ant-Vision VR application platform, a grand assistant, Tencent WEGAME, an OGP application platform, etc. may be used. VR applications in application platform software use application engines (Universal Engine 4, Universal 3D, etc.), and have integrated SDKs provided by data interfaces, such as the SDK provided by OpenVR of the data interface of the Steam VR platform software, so that the screen of the application can be seen on the display of the PC. The server side of the streaming software may be set as the a side of the nomogram software, for example.
The service end of the streaming software comprises two parts, one part is a control interface, and the other part is a server driver. The server driver is preferably a dll file, but may be implemented in other forms, such as an SDK, API file, etc. When application platform software, such as SteamVR platform software, is started on a PC, the server driver is loaded accordingly.
The client installed with the streaming software on the VR device may be set as the B-side of the nomogram software, for example. The VR equipment is provided with various sensors, such as a nine-axis sensor, an inertial sensor and the like, and can sense attitude actions, namely pitching, rolling, yawing and the like. The VR equipment sends the attitude data to a streaming software server on the PC through a streaming software client; and sending the data to the application platform software through a server of streaming software on the PC, so that the application platform software renders a real-time picture. The VR equipment can be integrated VR equipment, then the client side of the series flow software is installed in the system of the integrated VR equipment, the picture is also displayed on the display screen of the integrated VR equipment, and the sensor is fixedly installed on the integrated VR equipment. The VR equipment can be mobile VR equipment, then the client of series flow software is installed in the smart mobile phone of the mobile VR equipment, the picture can be displayed on the smart mobile phone of the mobile VR equipment and also can be displayed on the display screen of the mobile VR equipment, and the sensor can be fixedly installed in the shell of the mobile VR equipment and also can be used as the sensor of the smart mobile phone installed in the mobile VR equipment.
The PC and the VR device are connected by a wired/wireless method, and when the PC and the VR device are connected by a wireless method, they preferably operate in a WLAN (wireless local area network) or 5G communication environment. Due to the characteristics of high speed, low delay and the like of 5G communication, the actual delay generated by the PC and the VR equipment under the 5G communication environment is basically negligible.
In order to stream the frames of the VR application to the VR device, according to the stream architecture shown in fig. 1, several core modules that need to be implemented are: a server driver of a streaming software server, VR equipment and a client of streaming software installed in the VR equipment. The VR equipment is used for acquiring posture data of the VR equipment; and the client and the server driver of the streaming software are used for data transmission and processing.
Fig. 2 is a flowchart illustrating a method for implementing video streaming according to the present invention, which includes the following steps: and starting a client of the streaming software on the VR device, such as the NOLOHOME software at the B end, and starting a server of the streaming software on the PC, such as the NOLOHOME software at the A end. The control interface UI of the streaming software server comprises various control keys, streaming software is started through the control keys, and the A end is connected with the B end. The VR equipment can send data such as the attitude, the control information and the like to a server driver of a streaming software server on the PC through a client of the streaming software, the server driver processes the received data and sends the processed data to the application platform software for picture rendering, and the server driver sends the rendered picture to the VR equipment for picture display. The method specifically comprises the following steps:
and S1, acquiring the posture data of the VR equipment.
Wherein, the gesture data of VR equipment is obtained through installing sensor on VR equipment, for example nine sensor, inertial sensor, six sensor, gyroscope, earth magnetometer etc..
And transmitting the attitude data of the VR equipment to a client of streaming software installed on the VR equipment, and transmitting the attitude data to a server driver of a streaming software server in a 5G wireless mode through the client of the streaming software and UDP. Udp (user data program) is a connectionless transport layer protocol in the open system interconnection reference model, and provides transaction-oriented simple unreliable information transfer service. Through the step, the server side of the streaming software acquires the attitude data of the VR equipment.
Preferably, the control information of the VR device may also be obtained, and the control information of the VR device may also be sent to a server driver of a streaming software server via UDP through a client of the streaming software.
And S2, sending the acquired attitude data to the VR application for picture rendering.
The method specifically comprises the following steps:
and S21, sending the acquired gesture data of the VR equipment to a data interface, and transmitting the gesture data to VR application through the data interface.
And transmitting the attitude data acquired by the server driver of the streaming software server to a data interface, using an application engine for VR application in SteamVR application platform software, integrating SDKs provided by OpenVR data interfaces, and enabling OpenVR to transmit the attitude data to VR application.
Preferably, the control information of the VR device obtained by the server driver of the streaming software server is also sent to the VR application for screen rendering. And sending the control information acquired by the server side of the streaming software to a data interface, and transmitting the control information to the VR application through the data interface.
And S22, rendering the picture through the application engine according to the posture data and the application logic obtained by the VR application.
And the VR application transmits the obtained attitude data and the application logic to an application engine to obtain the exact rendering picture content and perform picture rendering. The application Engine is Unreal Engine 4, Universal 3D and the like.
Preferably, the VR application further transmits the obtained control information to the application engine to obtain the exact rendered screen content, and performs the rendering of the screen.
And S23, storing the data rendered by the application engine in a video memory of the video card.
In the embodiment provided by the invention, the data rendered by the application engine is stored in a video memory of a video card, for example, a video memory of an Nvidia video card, and notifies the VR application that the picture is rendered, the VR application notifies the data interface OpenVR, and the data interface OpenVR notifies the server driver of the streaming software server of the event that the rendering is completed.
And S3, acquiring the rendered picture, and sending the picture to VR equipment for display.
The method specifically comprises the following steps:
and S31, acquiring texture data corresponding to the rendered picture, and encoding a frame of picture into a plurality of data packets.
When the server driver of the streaming software server side knows the events of the rendered pictures, the corresponding texture data, namely the data of one frame of picture, is found in the video memory through the texture address transmitted by the OpenVR, and the one frame of picture is coded into a plurality of data packets.
In the embodiment provided by the invention, the special library for video coding and decoding provided by great is adopted, namely, the NvCodec library.
When initialization is performed, the encoding format and the picture format of the ncodec library are notified in advance. In the embodiment provided by the present invention, the data is encoded using H264. Regarding the picture FORMAT, using the NV _ ENC _ BUFFER _ FORMAT _ ABGR formatted picture, the NvCodec library will encode a frame of a picture into a plurality of small packets as required in the current frame.
And S32, sending the encoded data packets to VR equipment for decoding and displaying.
After the encoding is finished, a server driver of the streaming software server side sends a plurality of encoded data packets to a streaming software client installed on the VR device, the streaming software client is transmitted to the VR device, and the VR device decodes the received data packets after receiving a complete frame picture data to form a complete image on the VR device and displays the complete image.
The method and related hardware for displaying the picture by the VR device may use any method and hardware that are available and are not specifically required here.
In summary, in the method for implementing video streaming provided by the present invention, a server of streaming software obtains attitude data and control information of VR devices; sending the acquired attitude data and the acquired control information to a VR application for picture rendering; and acquiring the rendered picture, and sending the rendered picture to VR equipment for display. The method is characterized in that a server side of streaming software is installed on a PC terminal, and the method ensures that the PC side is responsible for running VR application and only picture display is required for VR equipment; therefore, the picture processing can be carried out through the hardware of the PC, and satisfactory pictures can be obtained on the screen of the VR equipment; moreover, the PC and the VR equipment are in wireless connection through the 5G router, the technical problem of 'wireless VR' which troubles numerous manufacturers is solved, and the actual delay generated by the wireless connection of the PC and the VR equipment through the 5G router can be basically ignored.
Example two
As shown in fig. 3, the system for implementing video streaming provided by the present invention includes a terminal, a VR device, and a position tracking apparatus.
The terminal is provided with a server of application platform software and streaming software; in the embodiment provided by the present invention, the terminal is exemplified by a pc (personal computer), and may also be a terminal with data processing capability, such as a tablet computer, a smart television, and a smart phone. The application platform software installed on the PC is, illustratively, the steadvr platform software (corresponding APP on the smartphone). Of course, other application platforms such as a VIVEPORT platform, a HYPEREAL platform, an Ant-Vision VR application platform, a grand assistant, Tencent WEGAME, an OGP application platform, etc. may be used. The VR application in the application platform software uses an application engine (UE4, U3D, etc.) to integrate the SDK provided by the data interface, such as the SDK provided by the data interface OpenVR of the Steam VR platform software, so that the screen of the application can be seen on the display of the PC. The server side of the streaming software may be set as the a side of the nomogram software, for example.
The service end of the streaming software comprises two parts, one part is a control interface, and the other part is a server driver. The server driver is preferably a dll file, but may be implemented in other forms, such as an SDK, API file, etc. When application platform software, such as the Steam VR platform software, is started on the PC, the server driver is loaded accordingly.
The client installed with the streaming software on the VR device may be set as the B-side of the nomogram software, for example. The VR equipment can be integrated VR equipment, then the client side of the series flow software is installed in the system of the integrated VR equipment, the picture is also displayed on the display screen of the integrated VR equipment, and the sensor is fixedly installed on the integrated VR equipment. The VR equipment can be mobile VR equipment, then the client of series flow software is installed in the smart mobile phone of the mobile VR equipment, the picture can be displayed on the smart mobile phone of the mobile VR equipment and also can be displayed on the display screen of the mobile VR equipment, and the sensor can be fixedly installed in the shell of the mobile VR equipment and also can be used as the sensor of the smart mobile phone installed in the mobile VR equipment.
The PC and the VR device are connected by a wired/wireless method, and when the PC and the VR device are connected by a wireless method, they preferably operate in a WLAN (wireless local area network) or 5G communication environment. Due to the characteristics of high speed, low delay and the like of 5G communication, the actual delay generated by the PC and the VR equipment under the 5G communication environment is basically negligible.
Preferably, most existing VR devices can only view video, that is, only three degrees of freedom of attitude tracking (pitch, roll and yaw) are needed, and if six degrees of freedom of head and hand position positioning (including pitch, roll, yaw and spatial X, Y, Z coordinates) are needed, a positioning tracking device is needed. The positioning and tracking device is used for tracking the position of a user, and for example, may include a handle held on the hand of the user for tracking the position of the hand of the user; a locator, either internally or externally mounted on the VR device, to track the position of the user's head. The handle can pass to the locator with the location data, and the locator is again with the location data of the location of locator and the location data of handle give VR equipment, and perhaps, handle and locator all transmit the location data directly to VR equipment.
The VR equipment acquires the positioning data acquired by the positioning and tracking device, acquires the attitude data of the positioning and tracking device and the attitude data of the VR equipment, and then sends the positioning data and the attitude data to a server of streaming software on the terminal through a UDP (user Datagram protocol) by utilizing a client of the streaming software. And the server side of the streaming software on the terminal sends the positioning data and the posture data to the application platform software, so that the application platform software renders a real-time picture.
In order to stream the frames of the VR application to the VR device, the requirement is implemented according to the architecture shown in fig. 3, in the stream architecture shown in fig. 3, several core modules that need to be implemented are: the device comprises a server driver of a streaming software server, VR equipment, a client of streaming software installed on the VR equipment, and a positioning and tracking device. The positioning and tracking device is used for acquiring positioning data and posture data of a user body; the VR equipment is used for acquiring positioning data and attitude data and transmitting the data to the server driver; and the client and the server driver of the streaming software are used for data transmission and processing.
Fig. 4 is a flowchart illustrating a method for implementing video streaming according to the present invention, which includes the following steps: and starting a client of the streaming software on the VR device, such as the NOLOHOME software at the B end, and starting a server of the streaming software on the PC, such as the NOLOHOME software at the A end. The control interface UI of the streaming software server comprises various control keys, streaming software is started through the control keys, and the A end is connected with the B end. The client of the streaming software on the VR equipment can send attitude data, control information, positioning data and the like to a server driver of a streaming software server of the PC, the server driver processes the received data and sends the processed data to the application platform software for picture rendering, and the server driver sends the rendered picture to the VR equipment for picture display. The method specifically comprises the following steps:
and S1, acquiring the posture data and the positioning data.
Acquiring attitude data and positioning data, specifically comprising the following steps:
and S11, acquiring positioning data and/or posture data of the user through the positioning and tracking device.
The positioning and tracking device may include a positioner mounted on the VR device, a handle held on a hand of the user, and the like, and the positioning data of the head and/or the hand of the user may be obtained by obtaining the positioning data of the positioner and/or the handle. The positioning data of the user can be obtained by a three-dimensional positioning method and system disclosed in patent application No. 201610917518.0, or can be obtained by other existing three-dimensional positioning methods and systems.
The posture data of the head of the user may be obtained by a sensor attached to the VR device, or may be obtained by a sensor attached to a positioner on the VR device, such as a nine-axis sensor, an inertial sensor, a six-axis sensor, a gyroscope, a magnetometer, or the like. Pose data for other parts of the user, such as the hand, is obtained by sensors mounted on the handle of the localization tracking device.
And S12, sending the acquired attitude data and the positioning data to VR equipment.
The VR equipment can read attitude data and positioning data through wired modes such as an OTG data line, and also can read attitude data and positioning data through wireless modes such as Bluetooth and wifi. For the integrated VR device, the data is directly sent to a system of the integrated VR device; for a mobile VR device, the data can then be sent to a smartphone installed in the mobile VR device housing.
And S13, sending the acquired attitude data and positioning data to a streaming software server.
And the VR equipment transmits the acquired attitude data and the positioning data to a client of streaming software installed on the VR equipment, and then transmits the attitude data and the positioning data to a server driver of a streaming software server installed on the terminal in a 5G wireless mode through UDP. Through the steps, the server side of the streaming software acquires the positioning data and the attitude data.
Preferably, the server of the streaming software may further obtain control information, and the control information may also be sent to the server driver of the server of the streaming software through the client of the streaming software via UDP. The control information may be from the VR device or from the position tracking device.
S2, sending the acquired attitude data and positioning data to a VR application for picture rendering, specifically comprising the following steps:
and S21, sending the acquired attitude data and positioning data to a data interface, and transmitting the attitude data and the positioning data to VR application through the data interface.
And transmitting the attitude data and the positioning data acquired by a server driver of the streaming software server to a data interface, using an application engine for VR application in application platform software SteamVR, integrating an SDK provided by a data interface OpenVR, and transmitting the attitude data and the positioning data to VR application by the data interface OpenVR.
Preferably, the control information obtained by the server driver of the streaming software server is also sent to the VR application. And sending the control information acquired by the server driver of the streaming software server to a data interface OpenVR, and transmitting the control information to VR application through the data interface OpenVR.
And S22, rendering the picture through the application engine according to the positioning data, the posture data and the application logic obtained by the VR application.
At this time, the VR application transmits the exact rendering picture content to the application engine according to the obtained positioning data, the obtained posture data and the obtained application logic, and performs picture rendering. The application Engine is Unreal Engine 4, Universal 3D and the like.
Preferably, the VR application transmits the obtained control information to the application engine to exactly render the screen content, so as to render the screen.
And S23, storing the data rendered by the application engine in a video memory of the video card.
In the embodiment provided by the invention, the data rendered by the application engine is stored in the video memory of the video card and informs the VR application that the picture is rendered, the VR application informs the data interface, and the data interface informs the streaming software server of the event of finishing the driving rendering.
And S3, acquiring the rendered picture, and sending the picture to VR equipment for display.
The method specifically comprises the following steps:
and S31, acquiring texture data corresponding to the rendered picture, and encoding a frame of picture into a plurality of data packets.
When the streaming software server driver knows the event of the rendered picture, the corresponding texture data, namely the data of one frame of picture, is found in the video memory through the texture address transmitted from the data interface, and the one frame of picture is coded into a plurality of data packets.
In the embodiment provided by the invention, the special library for video coding and decoding provided by great is adopted, namely, the NvCodec library.
When initialization is performed, the encoding format and the picture format of the ncodec library are notified in advance. In the embodiment provided by the present invention, the data is encoded using H264. Regarding the picture FORMAT, using the NV _ ENC _ BUFFER _ FORMAT _ ABGR formatted picture, the NvCodec library will encode a frame of a picture into a plurality of small packets as required in the current frame.
And S32, sending the encoded data packets to VR equipment for decoding and displaying.
After the encoding is finished, a server driver of a streaming software server side sends a plurality of encoded data packets to a client side of streaming software installed on VR equipment, the client side of the streaming software is transmitted to the VR equipment, and after receiving a complete frame image data, the VR equipment decodes the received data packets to form a complete image on the VR equipment and displays the complete image. The method and related hardware for displaying the picture by the VR device may use any method and hardware that are available and are not specifically required here.
In summary, in the method for implementing video streaming provided by the present invention, the service of the streaming software obtains the positioning data and the posture data; sending the acquired attitude data and the positioning data to a VR application for picture rendering; and acquiring the rendered picture, and sending the rendered picture to VR equipment for display. The method is characterized in that a server of streaming software is installed on a PC terminal, and the method ensures that a PC is responsible for running VR application and only picture display is required for VR equipment; therefore, the picture processing can be carried out through the hardware of the PC, and satisfactory pictures can be obtained on the screen of the VR equipment; moreover, the PC and the VR equipment are in wireless connection through the 5G router, the technical problem of 'wireless VR' which troubles numerous manufacturers is solved, and the actual delay generated by the wireless connection of the PC and the VR equipment through the 5G router can be basically ignored.
The embodiment of the present invention further provides an apparatus for implementing video streaming, where the apparatus includes a processor and a memory, and the processor is configured to execute a program for implementing video streaming stored in the memory to implement the method for implementing video streaming as described above. The memory herein stores one or more programs. Wherein the memory may comprise volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, a hard disk, or a solid state disk; the memory may also comprise a combination of memories of the kind described above. When executed by one or more processors, the one or more programs in the memory may implement some or all of the steps of the above-described method for implementing video streaming in the above-described method embodiments.
The system and method for implementing video streaming provided by the present invention are described in detail above. Any obvious modifications to the invention, which would occur to those skilled in the art, without departing from the true spirit of the invention, would constitute a violation of the patent rights of the invention and would carry a corresponding legal responsibility.

Claims (14)

1. A system for realizing video streaming is characterized by comprising a terminal and a VR device;
the terminal is provided with a server of application platform software and streaming software;
the VR equipment is provided with a client of streaming software, and the client of the streaming software sends attitude data to a server of the streaming software on the terminal; and the server side of the streaming software sends the gesture data to the application platform software, and the application platform software renders a picture.
2. A system for realizing video streaming is characterized by comprising a terminal, VR equipment and a positioning and tracking device;
the terminal is provided with a server of application platform software and streaming software;
the positioning tracking device is used for acquiring positioning data and sending the positioning data to VR equipment;
the VR equipment is provided with a client of streaming software, and the client of the streaming software sends attitude data and the positioning data to a server of the streaming software on the terminal; and the server side of the streaming software sends the posture data and the positioning data to the application platform software, and the application platform software renders a picture.
3. The system for video streaming according to claim 1 or 2, wherein:
the server side of the streaming software comprises a control interface and a server driver, and the server driver is loaded when the application platform software is started on the terminal.
4. The system for video streaming according to claim 1 or 2, wherein: the server side of the streaming software acquires rendered pictures and sends the rendered pictures to the client side of the streaming software; and the client of the streaming software sends the pictures to the VR equipment for display.
5. The system for video streaming according to claim 1 or 2, wherein: and the client of the streaming software sends the attitude data and/or the positioning data to the server of the streaming software in a wireless mode, wherein the wireless mode is 5G.
6. A method for implementing video streaming, comprising the steps of:
acquiring attitude data of VR equipment;
sending the acquired attitude data to a VR application for picture rendering;
and acquiring the rendered picture, and sending the rendered picture to VR equipment for display.
7. The method of claim 6, wherein the obtaining pose data of the VR device includes:
a client of streaming software installed on the VR equipment acquires attitude data of the VR equipment;
and a streaming software server installed on the terminal acquires the attitude data of the VR equipment sent by the client of the streaming software.
8. The method of claim 6, wherein the step of sending the obtained pose data to the VR application for rendering comprises the steps of:
sending the acquired attitude data to a data interface, and transmitting the attitude data to VR application through the data interface;
rendering a picture through an application engine according to the posture data and the application logic obtained by VR application;
and storing the data rendered by the application engine in a video memory of the video card.
9. A method for implementing video streaming, comprising the steps of:
acquiring attitude data and positioning data;
sending the acquired attitude data and the positioning data to a VR application for picture rendering;
and acquiring the rendered picture, and sending the rendered picture to VR equipment for display.
10. The method for video streaming according to claim 9, wherein the obtaining of pose data and positioning data comprises the steps of:
acquiring positioning data and/or posture data of a user through a positioning and tracking device;
sending the positioning data and/or the posture data acquired by the positioning and tracking device to VR equipment;
and the streaming software server acquires the positioning data and the attitude data which are sent by the VR equipment through UDP.
11. The method for video streaming according to claim 9, wherein the step of sending the obtained pose data and positioning data to a VR application for screen rendering comprises the steps of:
sending the attitude data and the positioning data acquired by the server of the streaming software to a data interface, and transmitting the attitude data and the positioning data to VR application through the data interface;
rendering a picture through an application engine according to the positioning data, the posture data and the application logic obtained by the VR application;
and storing the data rendered by the application engine in a video memory of the video card.
12. The method of claim 6 or 9, wherein the step of obtaining the rendered frames for sending to a VR device for displaying comprises the steps of:
acquiring texture data corresponding to a rendered picture, and encoding a frame of picture into a plurality of data packets;
and sending the encoded multiple data packets to the VR device for decoding and displaying.
13. The method of claim 6 or 9, further comprising obtaining control information, sending the obtained control information to the VR application for rendering;
wherein the control information is from the VR device or from a position tracking device.
14. An apparatus for implementing video streaming, comprising a processor and a memory, wherein the processor is configured to execute a program for implementing video streaming stored in the memory to implement the method for implementing video streaming according to claim 6 or 8.
CN201811203090.9A 2018-10-16 2018-10-16 System, method and device for realizing video streaming Pending CN111064985A (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201811203090.9A CN111064985A (en) 2018-10-16 2018-10-16 System, method and device for realizing video streaming
PCT/CN2019/111315 WO2020078354A1 (en) 2018-10-16 2019-10-15 Video streaming system, video streaming method and apparatus
US17/286,387 US11500455B2 (en) 2018-10-16 2019-10-15 Video streaming system, video streaming method and apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811203090.9A CN111064985A (en) 2018-10-16 2018-10-16 System, method and device for realizing video streaming

Publications (1)

Publication Number Publication Date
CN111064985A true CN111064985A (en) 2020-04-24

Family

ID=70296540

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811203090.9A Pending CN111064985A (en) 2018-10-16 2018-10-16 System, method and device for realizing video streaming

Country Status (1)

Country Link
CN (1) CN111064985A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111831353A (en) * 2020-07-09 2020-10-27 平行云科技(北京)有限公司 OpenXR standard-based runtime library, data interaction method, device and medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104035760A (en) * 2014-03-04 2014-09-10 苏州天魂网络科技有限公司 System capable of realizing immersive virtual reality over mobile platforms
US20170186231A1 (en) * 2015-12-28 2017-06-29 Oculus Vr, Llc Increasing field of view of head-mounted display using a mirror
CN106998409A (en) * 2017-03-21 2017-08-01 华为技术有限公司 A kind of image processing method, head-mounted display and rendering apparatus
CN206541288U (en) * 2017-01-07 2017-10-03 北京国承万通信息科技有限公司 A kind of virtual reality system, main frame and head-mounted display apparatus
CN107315470A (en) * 2017-05-25 2017-11-03 腾讯科技（深圳）有限公司 Graphic processing method, processor and virtual reality system
CN107979763A (en) * 2016-10-21 2018-05-01 阿里巴巴集团控股有限公司 A kind of virtual reality device generation video, playback method, apparatus and system
CN108111839A (en) * 2017-12-22 2018-06-01 北京轻威科技有限责任公司 A kind of series flow wireless dummy reality helmet

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104035760A (en) * 2014-03-04 2014-09-10 苏州天魂网络科技有限公司 System capable of realizing immersive virtual reality over mobile platforms
US20170186231A1 (en) * 2015-12-28 2017-06-29 Oculus Vr, Llc Increasing field of view of head-mounted display using a mirror
CN107979763A (en) * 2016-10-21 2018-05-01 阿里巴巴集团控股有限公司 A kind of virtual reality device generation video, playback method, apparatus and system
CN206541288U (en) * 2017-01-07 2017-10-03 北京国承万通信息科技有限公司 A kind of virtual reality system, main frame and head-mounted display apparatus
CN106998409A (en) * 2017-03-21 2017-08-01 华为技术有限公司 A kind of image processing method, head-mounted display and rendering apparatus
CN107315470A (en) * 2017-05-25 2017-11-03 腾讯科技（深圳）有限公司 Graphic processing method, processor and virtual reality system
CN108111839A (en) * 2017-12-22 2018-06-01 北京轻威科技有限责任公司 A kind of series flow wireless dummy reality helmet

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111831353A (en) * 2020-07-09 2020-10-27 平行云科技(北京)有限公司 OpenXR standard-based runtime library, data interaction method, device and medium
CN111831353B (en) * 2020-07-09 2024-02-20 平行云科技(北京)有限公司 Operation library based on OpenXR standard, data interaction method, device and medium

Similar Documents

Publication Publication Date Title
US10229651B2 (en) Variable refresh rate video capture and playback
EP3180911B1 (en) Immersive video
EP4057109A1 (en) Data processing method and apparatus, electronic device and storage medium
KR20210151114A (en) Hybrid rendering
CN108668168B (en) Android VR video player based on Unity3D and design method thereof
US10437055B2 (en) Master device, slave device, and control method therefor
US20180336069A1 (en) Systems and methods for a hardware agnostic virtual experience
US20240098344A1 (en) Video modification and transmission using tokens
KR102503337B1 (en) Image display method, apparatus and system
US20190295324A1 (en) Optimized content sharing interaction using a mixed reality environment
CN111064981B (en) System and method for video streaming
US11756251B2 (en) Facial animation control by automatic generation of facial action units using text and speech
WO2021196973A1 (en) Virtual content display method and apparatus, and electronic device and storage medium
CN111064985A (en) System, method and device for realizing video streaming
CN109003336A (en) Processing method, system and the computer readable storage medium of virtual reality imagery
KR102261739B1 (en) System and method for adaptive streaming of augmented reality media content
US11500455B2 (en) Video streaming system, video streaming method and apparatus
Dermawan et al. Moving Mixed Reality Game to the Cloud: A Survey on Feasibility
CN111065053B (en) System and method for video streaming
CN113568744A (en) Resource processing method, device, server and storage medium
KR20200116380A (en) Mobile interlocking VR device compatible to VR sports device
KR102608117B1 (en) System and method for correcting image
US20240062467A1 (en) Distributed generation of virtual content
US20230122666A1 (en) Cloud xr-based program virtualizing method
JP2023092729A (en) Communication device, communication system, display method, and program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20200424

RJ01 Rejection of invention patent application after publication