CN115696439A

CN115696439A - Data transmission method, device, equipment and medium

Info

Publication number: CN115696439A
Application number: CN202211334547.6A
Authority: CN
Inventors: 李金霖
Original assignee: Goertek Techology Co Ltd
Current assignee: Goertek Techology Co Ltd
Priority date: 2022-10-28
Filing date: 2022-10-28
Publication date: 2023-02-03

Abstract

The application discloses a data transmission method, a device, equipment and a medium, comprising the following steps: acquiring user posture data sent by VR equipment; generating a frame of a picture to be displayed on the VR device based on the user gesture data; transmitting the picture frame to the VR device based on an SRT protocol. Therefore, the frame of the picture to be displayed on the VR equipment is generated based on the user posture data acquired from the VR equipment, and then the frame of the picture is transmitted to the VR equipment based on the SRT protocol, so that the data transmission delay between the computer equipment and the VR equipment can be reduced on the premise of guaranteeing reliable transmission.

Description

Data transmission method, device, equipment and medium

Technical Field

The present application relates to the field of data transmission technologies, and in particular, to a data transmission method, apparatus, device, and medium.

Background

In recent years, VR (Virtual Reality) devices are becoming popular, and in the VR device use experience, the picture transmission and timely motion feedback between the VR device and a PC (Personal Computer) device are important factors affecting the user experience.

At present, data streaming between a VR device and a PC device is usually direct transmission based on wifi hot spots, and although the transmission mode is very low in delay, there may be problems such as data loss in large-traffic transmission. For the transmission of streaming media, real-Time audio fluid is required, and most of the encoding compression performed by mobile-mpeg is HTTP (Hyper Text Transfer Protocol) or RTMP (Real Time Messaging Protocol) transmission Protocol. HTTP is not real-time high. The RTMP is a streaming media transmission protocol which is widely applied at present, is widely applied to the field of live broadcast, and is more suitable for large-range distribution and stream pushing of multiple users. However, when the RTMP protocol is used for end-to-end video transmission, the RTMP server needs to be transferred, and the transmission delay is large. If the user establishes the RTMP server by himself, the delay is about 1 s. The RTMP has an accumulated delay because it does not lose packets based on TCP (Transmission Control Protocol). The RTSP server is complex to realize and is not penetrated by a firewall of a router.

In summary, how to reduce the delay of data transmission between the computer device and the VR device on the premise of ensuring reliable transmission is an urgent problem to be solved at present.

Disclosure of Invention

In view of this, an object of the present application is to provide a data transmission method, apparatus, device and medium, which can reduce the delay of data transmission between a computer device and a VR device on the premise of guaranteeing reliable transmission. The specific scheme is as follows:

in a first aspect, the present application discloses a data transmission method applied to a computer device, including:

acquiring user posture data sent by VR equipment;

generating a frame of a picture to be displayed on the VR device based on the user gesture data;

transmitting the picture frame to the VR device based on an SRT protocol.

Optionally, the transmitting the picture frame to the VR device based on the SRT protocol includes:

compressing the picture frame by using a preset compression processing logic to obtain a compressed frame;

transmitting the compressed frame to the VR device based on an SRT protocol.

Optionally, the compressing the picture frame by using a preset compression processing logic to obtain a compressed frame includes:

grouping the picture frames;

determining a basic frame and a non-basic frame in any group, predicting difference information of the non-basic frame based on the basic frame, and compressing the basic frame and the difference information to obtain a compressed frame corresponding to the basic frame and a compressed frame corresponding to the non-basic frame.

Optionally, the method further includes: establishing a wireless hotspot connection with the VR device; accordingly, the method has the advantages that,

the acquiring user gesture data sent by the VR device includes: and acquiring user posture data sent by the VR equipment based on the wireless hotspot connection.

Optionally, the generating a frame to be displayed on the VR device based on the user gesture data includes:

rendering by using a game engine based on the user posture data to obtain a picture frame to be displayed on the VR equipment.

sending a handshake request to the VR equipment, and receiving a handshake response returned by the VR equipment;

sending a parameter notification to the VR equipment, and receiving a parameter response returned by the VR equipment;

transmitting the picture frame and a timestamp of the picture frame to the VR device.

In a second aspect, the present application discloses a data transmission method, which is applied to a VR device, and includes:

sending user gesture data to a computer device;

receiving a picture frame sent by computer equipment based on an SRT protocol, wherein the picture frame is a picture frame to be displayed on the VR equipment generated by the computer equipment based on the user posture data;

in a third aspect, the present application discloses a data transmission apparatus, which is applied to a computer device, and includes:

the user data acquisition module is used for acquiring user posture data sent by the VR equipment;

a frame generation module, configured to generate a frame to be displayed on the VR device based on the user gesture data;

and the picture frame transmission module is used for transmitting the picture frame to the VR equipment based on an SRT protocol.

In a fourth aspect, the present application discloses an electronic device comprising a memory and a processor, wherein:

the memory is used for storing a computer program;

the processor is configured to execute the computer program to implement the foregoing data transmission method.

In a fifth aspect, the present application discloses a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned data transmission method.

Therefore, the method and the device for displaying the picture frame on the VR equipment acquire user posture data sent by the VR equipment, generate the picture frame to be displayed on the VR equipment based on the user posture data, and transmit the picture frame to the VR equipment based on an SRT protocol. That is, in the present application, a frame to be displayed on the VR device is generated based on user gesture data acquired from the VR device, and then the frame is transmitted to the VR device based on the SRT protocol, so that, with the SRT protocol, since the SRT protocol is based on bidirectional UDP point-to-point connection and has a good packet loss retransmission mechanism and a good congestion control mechanism, delay of data transmission between the computer device and the VR device can be reduced on the premise of ensuring reliable transmission.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

FIG. 1 is a flow chart of a data transmission method disclosed herein;

fig. 2 is a transmission diagram of a specific SRT protocol disclosed in the present application;

fig. 3 is a schematic diagram of data transmission between a specific VR device and a PC according to the present disclosure;

FIG. 4 is a flow chart of another data transmission method disclosed herein;

FIG. 5 is a schematic diagram of a data transmission device disclosed in the present application;

fig. 6 is a block diagram of an electronic device disclosed in the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

At present, data streaming between a VR device and a PC device is usually direct transmission based on wifi hot spots, and although the transmission mode is very low in delay, there may be problems such as data loss in large-traffic transmission. For the transmission of streaming media requiring real-time audio streaming, the HTTP or RTMP transmission protocol is mostly adopted for encoding compression by the mobile-mpeg. HTTP is not real-time high. The RTMP is a streaming media transmission protocol which is widely applied at present, widely used in the field of live broadcast, and more suitable for wide-range distribution of push streams to multiple users. However, when the RTMP protocol is used for end-to-end video transmission, an RTMP server needs to be transferred, and the transmission delay is large. If the user establishes the RTMP server, the time delay is about 1 s. RTMP has accumulated delay because RTMP does not lose packets based on TCP. The RTSP server is complex to realize and is not penetrated by a firewall of a router. Therefore, the data transmission scheme is provided, and the data transmission delay between the computer equipment and the VR equipment can be reduced on the premise of guaranteeing reliable transmission.

Referring to fig. 1, an embodiment of the present application discloses a data transmission method applied to a computer device, including:

step S11: and acquiring user posture data sent by the VR equipment.

In one embodiment, the user gesture data may be head gesture data of the user, but may be gesture data of other parts of the user in other embodiments.

Moreover, the embodiment of the application can establish wireless hotspot connection with the VR equipment; correspondingly, user gesture data sent by the VR device is obtained based on the wireless hotspot connection. It is noted that because the gesture data is small, the VR device may directly transmit the user gesture data to the computer device based on the wireless hotspot connection. Wherein the computer device may be a personal computing device.

Step S12: generating a frame of a picture to be displayed on the VR device based on the user gesture data.

In a specific embodiment, a game engine may be used to render based on the user gesture data, so as to obtain a frame of a picture to be displayed on the VR device.

Step S13: transmitting the picture frame to the VR device based on an SRT protocol.

In a specific embodiment, a handshake request may be sent to the VR device, and a handshake response returned by the VR device may be received; sending a parameter notification to the VR equipment, and receiving a parameter response returned by the VR equipment; transmitting the picture frame and a timestamp of the picture frame to the VR device. It can be understood that, in the embodiments of the present application, the picture frame may be transmitted to the VR device based on an SRT protocol by using a wireless hotspot connection.

For example, referring to fig. 2, fig. 2 is a schematic diagram of a specific SRT protocol transmission disclosed in the embodiment of the present application. The PC terminal is used as a caller terminal, encodes pictures and sends the coded pictures to the VR terminal which is a listener. The streaming media in the transmission process is totally decomposed into picture frames and time stamps through the SRT protocol for transmission. Meanwhile, small sensing data such as user gesture data of the VR end can be transmitted back to the PC end from the VR end through the wireless hotspot to respond. The SRT protocol includes two types of data packets: the information Packet (Data Packet) and the Control Packet (Control Packet) are distinguished by the highest bit (flag bit) of the SRT header, 0 represents the information Packet, and 1 represents the Control Packet. The control data packet also comprises various types of handshaking (Handshake), acknowledgement (ACK), negative Acknowledgement (NAK), acknowledgement to acknowledgement (ACKACK), keep connection (keep), close connection (Shutdown), and the like.

Further, in a specific embodiment, the picture frame may be compressed by using a preset compression processing logic to obtain a compressed frame; transmitting the compressed frame to the VR device based on an SRT protocol.

In one embodiment, picture frames may be grouped; determining a basic frame and a non-basic frame in any group, predicting difference information of the non-basic frame based on the basic frame, and compressing the basic frame and the difference information to obtain a compressed frame corresponding to the basic frame and a compressed frame corresponding to the non-basic frame.

Through testing, the SRT protocol is about 2.5 times to 3.2 times faster than the RTMP protocol. It should be noted that the SRT protocol is improved from the UDT protocol, and the SRT protocol retains most of the core concepts and mechanisms of the UDT protocol, and introduces some improvements and enhanced functions, mainly including flow control, enhanced congestion control, modification of control data, and improvement of encryption mechanism for real-time audio and video. The UDP protocol is an unreliable transport protocol based on datagram, which can only provide unreliable data transmission service with the main design principle of increasing the transmission rate of data, and the best effort is delivered. Although the UDP protocol is an unreliable transmission protocol and is unstable in the network environment of jitter and packet loss of the internet, by means of the strong data recovery capability of the SRT protocol, a forward correction technique (FEC) is applied to minimize the possibility of network packet loss, thereby ensuring the transmission stability of the SRT protocol, and meanwhile, the SRT protocol can also perform AES (Advanced Encryption Standard) Encryption, thereby ensuring the security of data in the transmission process. That is, the SRT protocol is based on a bidirectional UDP point-to-point connection, has the characteristics of safety, reliability, and low delay, allows a connection to be directly established between a signal source and a target, and is suitable for point-to-point high-quality low-delay reliable transmission.

In the following, taking a 6DoF Head Mounted Display (HMD) as an example to elaborate the data transmission scheme provided in the present application, referring to fig. 3, fig. 3 is a schematic diagram of data transmission between a specific VR device and a PC disclosed in an embodiment of the present application, the 6DoF head mounted display tracks and captures head pose data, since the pose data is much smaller than video data, the data can be directly transmitted to the PC by using a protocol hotspot (hostop), and a game engine on the PC generates a rendering frame (i.e. a picture frame) based on the head pose data and sends it back to the VR device, and decodes, sends and displays it on the device. Wherein the 6DoF engine determines head pose data based on an eye buffer (D3D 11 texture), the PC-side game engine transmits the rendered frames and the head pose data to an encoding module, an encoder in the encoding module encodes the head pose data into h.264 format for use by other references, and encodes the rendered frames into h.264 format and sends back to the VR-side based on SRT protocol. The VR end receives the SRT encoded bitstream of compressed rendered frames, synchronizes and decompresses the compressed frames, and then performs color correction and re-projection operations on the rendered frames, sending them to the display of the HMD for projection. The wireless module is responsible for encoding and decoding 6DoF head posture data and compressing SRT data streams of rendering frames, mainly uses an SRT wireless transmission technology to carry out low-delay transmission, and can obtain better user experience. The VR player transmits the HMD pose position information to the VR renderer at a specified rate. The player receives the encoded bitstream of rendered frames via wireless transmission and decodes/synchronizes the frames before transferring them to the VR stack to perform the required lens distortion and color difference correction. The VR renderer receives HMD pose position information from the player and sets a direction and position of the VR camera using the HMD pose position information. The VR device passes the latest head pose data to the PC side. The embodiment of the application adopts H.264 compression, so that the compression is carried out through the H.264 before the PC end transmits the picture, and the data streaming between the VR equipment and the PC end is realized by using SRT transmission, so that the transmission speed is improved on the premise of keeping low delay when the picture transmission is synchronous. Of course, in other embodiments, H.265 compression may also be employed. Therefore, on the basis of end-to-end wifi connection, the SRT protocol is used for transmitting streaming media, the stream pushing of large-flow pictures (such as high-definition video resources) is realized on the premise of keeping low delay, a good packet loss retransmission mechanism and rich congestion control statistical information are possessed, and the picture quality of a user is improved. Based on the UDT protocol, the time delay is low, and the encryption security requirement of short-distance end-to-end transmission is low.

Wherein, H.264 compression: three frames are defined in h.264: i frame: the completely coded frame is called I frame; p frame: a frame generated by referring to the previous I frame and only containing difference part codes is called a P frame; b frame: b frame is called as frame code frame after reference; the core algorithms adopted by H264 are intraframe compression and interframe compression: intra-frame compression is an algorithm for generating I-frames, and inter-frame compression is an algorithm for generating B-frames and P-frames. The compression method comprises the following steps: grouping: several frames of images are divided into a group (GOP, i.e., a sequence), and the number of frames is not preferable to be large in order to prevent motion variation; defining a frame: defining each frame image in each group into three types, namely I frame, B frame and P frame; predicting a frame: taking the I frame as a basic frame, predicting a P frame by the I frame, and predicting a B frame by the I frame and the P frame; data transmission: and finally, storing and transmitting the I frame data and the predicted difference information.

Therefore, the user posture data sent by the VR device is obtained firstly, then the picture frame to be displayed on the VR device is generated based on the user posture data, and the picture frame is transmitted to the VR device based on the SRT protocol. That is, in the present application, a frame to be displayed on the VR device is generated based on user gesture data acquired from the VR device, and then the frame is transmitted to the VR device based on the SRT protocol, so that, with the SRT protocol, since the SRT protocol is based on bidirectional UDP point-to-point connection and has a good packet loss retransmission mechanism and a good congestion control mechanism, delay of data transmission between the computer device and the VR device can be reduced on the premise of ensuring reliable transmission.

Compressing the picture frame by using a preset compression processing logic to obtain a compressed frame; and transmitting the compressed frame to the VR equipment based on the SRT protocol, and improving the transmission speed on the premise of keeping low delay.

Referring to fig. 4, an embodiment of the present application discloses a data transmission method applied to a VR device, including:

step S21: user gesture data is sent to the computer device.

In particular embodiments, a wireless hotspot connection is established with a computer device; accordingly, user gesture data is transmitted based on the wireless hotspot connection.

Step S22: and receiving a picture frame sent by computer equipment based on an SRT protocol, wherein the picture frame is a picture frame to be displayed on the VR equipment generated by the computer equipment based on the user posture data.

In a specific embodiment, a handshake response may be returned when a handshake request sent by a computer device is received; after receiving a parameter notification sent by the computer equipment, returning a parameter response; and then receiving the picture frame sent by the computer equipment and the time stamp of the picture frame.

And the picture frame is obtained by rendering the computer equipment by using a game engine based on the user gesture data. And, in an embodiment, a compressed frame of the picture frame is received, and the compressed frame is obtained by compressing the picture frame by the computer device by using a preset compression processing logic.

Therefore, the method and the device for processing the user gesture data send the user gesture data to the computer equipment; and receiving a picture frame sent by computer equipment based on an SRT protocol, wherein the picture frame is a picture frame to be displayed on the VR equipment generated by the computer equipment based on the user posture data. Therefore, by adopting the SRT protocol, the SRT protocol is based on bidirectional UDP point-to-point connection and has a good packet loss retransmission mechanism and a congestion control mechanism, so that the delay of data transmission between the computer equipment and the VR equipment can be reduced on the premise of ensuring reliable transmission.

Referring to fig. 5, an embodiment of the present application discloses a data transmission apparatus, which is applied to a computer device, and includes:

the user data acquisition module 11 is configured to acquire user gesture data sent by the VR device;

a frame generating module 12, configured to generate a frame to be displayed on the VR device based on the user gesture data;

a picture frame transmission module 13, configured to transmit the picture frame to the VR device based on an SRT protocol.

Therefore, the user posture data sent by the VR device is obtained firstly, then the picture frame to be displayed on the VR device is generated based on the user posture data, and the picture frame is transmitted to the VR device based on the SRT protocol. That is, according to the method and the device, a picture frame to be displayed on the VR device is generated based on user posture data acquired from the VR device, and then the picture frame is transmitted to the VR device based on the SRT protocol.

The picture frame transmission module 13 includes:

the compression submodule is used for compressing the picture frame by utilizing a preset compression processing logic to obtain a compressed frame;

and the picture frame transmission sub-module is used for transmitting the compressed frame to the VR equipment based on an SRT protocol.

In one embodiment, the compression submodule is specifically configured to group picture frames; determining a basic frame and a non-basic frame in any group, predicting difference information of the non-basic frame based on the basic frame, and compressing the basic frame and the difference information to obtain a compressed frame corresponding to the basic frame and a compressed frame corresponding to the non-basic frame.

Further, the device further comprises a wireless connection establishing module, configured to establish a wireless hotspot connection with the VR equipment; correspondingly, the user data obtaining module 11 is specifically configured to obtain the user gesture data sent by the VR device based on the wireless hotspot connection.

In an embodiment, the frame generating module 12 is configured to render, by using a game engine and based on the user gesture data, a frame to be displayed on the VR device.

Further, the picture frame transmission module 13 is specifically configured to send a handshake request to the VR device, and receive a handshake response returned by the VR device; sending a parameter notification to the VR equipment, and receiving a parameter response returned by the VR equipment; transmitting the picture frame and a timestamp of the picture frame to the VR device.

Referring to fig. 6, an embodiment of the present application discloses an electronic device, which includes a processor 21 and a memory 22; wherein, the memory 22 is used for saving computer programs; the processor 21 is configured to execute the computer program and the data transmission method disclosed in the foregoing embodiments.

For the specific process of the data transmission method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.

Further, the present application also discloses a computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the data transmission method disclosed in the foregoing embodiments.

For the specific process of the data transmission method, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.

In the present specification, the embodiments are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same or similar parts between the embodiments are referred to each other. The device disclosed in the embodiment corresponds to the method disclosed in the embodiment, so that the description is simple, and the relevant points can be referred to the description of the method part.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The data transmission method, apparatus, device and medium provided by the present application are introduced in detail, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A data transmission method is applied to computer equipment and comprises the following steps:

acquiring user posture data sent by VR equipment;

transmitting the picture frame to the VR device based on an SRT protocol.

2. The data transmission method of claim 1, wherein transmitting the picture frame to the VR device based on the SRT protocol comprises:

transmitting the compressed frame to the VR device based on an SRT protocol.

3. The data transmission method according to claim 2, wherein said compressing the picture frame by using a preset compression processing logic to obtain a compressed frame comprises:

grouping the picture frames;

4. The data transmission method according to claim 2, further comprising: establishing a wireless hotspot connection with the VR device; accordingly, the method has the advantages that,

the obtaining of user gesture data sent by VR equipment includes: and acquiring user posture data sent by the VR equipment based on the wireless hotspot connection.

5. The data transmission method of claim 1, wherein the generating a frame of a picture to be displayed on the VR device based on the user gesture data comprises:

rendering by using a game engine based on the user gesture data to obtain a frame of the picture to be displayed on the VR equipment.

6. The data transmission method of any of claims 1 to 5, wherein transmitting the picture frame to the VR device based on the SRT protocol includes:

7. A data transmission method is applied to VR equipment and comprises the following steps:

sending user gesture data to a computer device;

and receiving a picture frame sent by computer equipment based on an SRT protocol, wherein the picture frame is a picture frame to be displayed on the VR equipment generated by the computer equipment based on the user posture data.

8. A data transmission device applied to computer equipment comprises:

a picture frame transmission module for transmitting the picture frame to the VR device based on an SRT protocol.

9. An electronic device comprising a memory and a processor, wherein:

the memory is used for storing a computer program;

the processor for executing the computer program to implement the data transmission method according to any one of claims 1 to 7.

10. A computer-readable storage medium for storing a computer program, wherein the computer program when executed by a processor implements the data transmission method according to any one of claims 1 to 7.