CN117768649A

CN117768649A - Data stream transmission method and device and electronic equipment

Info

Publication number: CN117768649A
Application number: CN202311811332.3A
Authority: CN
Inventors: 杨明明; 李昱锋; 邱绪东
Original assignee: Goertek Techology Co Ltd
Current assignee: Goertek Techology Co Ltd
Priority date: 2023-12-26
Filing date: 2023-12-26
Publication date: 2024-03-26

Abstract

The invention discloses a data stream transmission method, a device and electronic equipment, and relates to the technical field of head-mounted display equipment, wherein the method is applied to the head-mounted display equipment and comprises the following steps: buffering the fragmented data frames of the image frames to be displayed, which are sent by the computing equipment, by utilizing a data frame buffer; wherein the sliced data frames comprise image frame codes, sliced codes and image sliced data; framing the image slice data according to the image frame coding and the slice coding to obtain an image frame to be displayed; displaying the image frame to be displayed by using a display screen of the head-mounted display device; the invention can carry out framing on all image slice data of the same image frame to be displayed in the data frame buffer area through the image frame coding and the setting of the slice coding in the slice data frame to obtain the image frame to be displayed, thereby avoiding the problem of display screen display caused by the error of the restored image frame; and the problem of abnormal display caused by disordered displayed image frames can be avoided, and the user experience is improved.

Description

Data stream transmission method and device and electronic equipment

Technical Field

The present invention relates to the field of head-mounted display devices, and in particular, to a data stream transmission method and apparatus, and an electronic device.

Background

With the development of modern society technologies, applications of head-mounted display devices such as AR (Augmented Reality) devices and VR (Virtual Reality) devices are becoming wider and wider. Data streaming of a head-mounted display device refers to sending scene content and sound generated by a computing device (e.g., a host or a cell phone) to the head-mounted display device (e.g., AR glasses) for display and audio playback.

In the prior art, the scheme of data stream transmission generally adopts UDP (User Datagram Protocol ) protocol to transmit audio and video data, and because the image frames in the video data are relatively large and far beyond the MTU (Maximum Transmission Unit ) of the network interface, the image frames need to be fragmented before transmission, as shown in fig. 1, by row and by column. Because the UDP protocol is an unreliable transport layer protocol, and a network between the computing device and the head-mounted display device may pass through the multi-level router, the sequence of different image frames received by the head-mounted display device and the sequence of different slices of the same image frame may be different from the sequence of the slices of the image frames sent by the computing device, so that if the head-mounted display device directly resumes the image frame display according to the sequence of the received slices of the image frames, the problems of disordered image frames and screen display of the displayed image frames may occur, and abnormal display of the head-mounted display device may be caused.

Therefore, how to solve the problem of abnormal display of the head-mounted display device caused by disordered transmission data, and improve the user experience is a problem which needs to be solved in the present day.

Disclosure of Invention

The invention aims to provide a data stream transmission method, a data stream transmission device and electronic equipment, so as to solve the problem of abnormal display of head-mounted display equipment caused by disordered transmission data and improve user experience.

In order to solve the above technical problems, the present invention provides a data stream transmission method, which is applied to a head-mounted display device, and includes:

buffering the fragmented data frames of the image frames to be displayed, which are sent by the computing equipment, by utilizing a data frame buffer; wherein the head-mounted display device is wirelessly connected with the computing device, and the sliced data frames comprise image frame codes, sliced codes and image sliced data;

framing the image slice data according to the image frame code and the slice code to acquire the image frame to be displayed;

displaying the image frame to be displayed by using a display screen of the head-mounted display device; the display sequence number corresponding to the image frame code of the current image frame to be displayed on the display screen is larger than the display sequence number corresponding to the image frame code of the last image frame to be displayed on the display screen, and the current image frame to be displayed is any image frame to be displayed.

In some embodiments, the framing the image slice data according to the image frame encoding and the slice encoding to obtain the image frame to be displayed includes:

according to the cached historical image frames in the historical frame cache area and the cached slicing codes in the slicing data frames of the current image frames, framing the image slicing data of the current image frames, and obtaining combined image frames corresponding to the current image frames; the current image frame is any image frame to be displayed corresponding to the image frame code, and the historical image frame comprises a preset number of combined image frames corresponding to the image frames to be displayed before the current image frame;

correspondingly, the displaying the image frame to be displayed by using the display screen of the head-mounted display device includes:

and updating the historical image frames cached in the historical frame cache area according to the combined image frames corresponding to the current image frames, and displaying the combined image frames corresponding to the current image frames by utilizing the display screen of the head-mounted display device.

In some embodiments, the framing the image slice data of the current image frame according to the slice coding in the historical image frame buffered in the historical frame buffer area and the slice data frame of the current image frame buffered, to obtain a combined image frame corresponding to the current image frame, includes:

After the buffering of all the sliced data frames of the current image frame is completed, if the number of the buffered all the sliced data frames of the current image frame is equal to the number of the sliced data frames of the current image frame, framing all the image sliced data of the current image frame according to the sliced codes in all the sliced data frames of the current image frame to obtain a combined image frame corresponding to the current image frame;

if the number of all the cached sliced data frames of the current image frame is smaller than the sliced number, the missing sliced code of the current image frame is obtained according to the sliced number and the sliced codes in all the sliced data frames of the current image frame; wherein the number of missing slice encodings is the difference between the number of slices and the number of full slice data frames of the current image frame;

according to the missing slice coding and the slice coding in all slice data frames of the current image frame, framing all image slice data of the current image frame and target slice data in the historical image frame, and obtaining a combined image frame corresponding to the current image frame; the target slice data is image slice data corresponding to the missing slice code.

In some embodiments, the framing is performed on the image slice data of the current image frame according to the slice code in the historical image frame cached in the historical frame cache area and the slice data frame of the cached current image frame, so as to obtain a combined image frame corresponding to the current image frame, and the method further includes:

Determining whether all the sliced data frames of the current image frame are cached according to the received image frame codes in the sliced data frames of the image frame to be displayed;

if yes, executing the step of framing all image slice data of the current image frame according to slice coding in all image slice data frames of the current image frame if the number of all slice data frames of the cached current image frame is equal to the number of slices of the current image frame, or the step of missing slice coding of the current image frame according to the number of slices and slice coding in all slice data frames of the current image frame if the number of all slice data frames of the cached current image frame is smaller than the number of slices.

In some embodiments, the determining whether buffering of all the sliced data frames of the current image frame is completed according to the received image frame codes in the sliced data frames of the image frame to be displayed includes:

judging whether the image frame code in the current slicing data frame is larger than the image frame code corresponding to the current image frame or not; the current sliced data frame is any sliced data frame of the received image frame to be displayed, and the image frame is coded into a digital code;

If the frame code is larger than the frame code corresponding to the current image frame, determining that the buffering of all the slicing data frames of the current image frame is completed;

if the current slice data frame is equal to the image frame code corresponding to the current image frame, taking the current slice data frame as one slice data frame of the current image frame, and caching the current slice data frame in a position corresponding to the current image frame in a data frame buffer area according to the slice code in the current slice data frame;

and if the current slice data frame is smaller than the image frame code corresponding to the current image frame, discarding the current slice data frame.

In some embodiments, the method further comprises:

and determining the number of fragments and the capacities of the data frame buffer area and the historical frame buffer area according to the maximum transmission unit information sent by the computing equipment and the transmission resolution of the image frames to be displayed.

In some embodiments, the encoding of slices in the full slice data frame according to the number of slices and the current image frame, after the encoding of the missing slices in the current image frame, further includes:

judging whether the quotient of the number of the missing slice codes and the number of the slices is larger than a proportion threshold value or not;

and if the ratio threshold is larger than the ratio threshold, sending packet loss serious prompt information to the computing equipment so as to prompt the computing equipment to reduce the transmission resolution of the image frame to be displayed.

In some embodiments, the historical image frame is a combined image frame corresponding to a nearest image frame to be displayed before the current image frame.

The invention also provides a data stream transmission device, which is applied to the head-mounted display equipment and comprises:

the frame buffer module is used for buffering the sliced data frames of the image frames to be displayed, which are sent by the computing equipment, by utilizing the data frame buffer area; wherein the head-mounted display device is wirelessly connected with the computing device, and the sliced data frames comprise image frame codes, sliced codes and image sliced data;

the framing module is used for framing the image slice data according to the image frame codes and the slice codes to acquire the image frames to be displayed;

the display module is used for displaying the image frames to be displayed by utilizing a display screen of the head-mounted display device; the display sequence number corresponding to the image frame code of the current image frame to be displayed on the display screen is larger than the display sequence number corresponding to the image frame code of the last image frame to be displayed on the display screen, and the current image frame to be displayed is any image frame to be displayed.

The invention also provides a data stream transmission method, which is applied to the computing equipment and comprises the following steps:

Acquiring an image frame to be displayed;

fragmenting the image frames to be displayed to generate respective fragment data frames of the image frames to be displayed; wherein the sliced data frames comprise image frame codes, sliced codes and image sliced data;

transmitting the fragmented data frames to a head-mounted display device to display the image frames to be displayed by using a display screen of the head-mounted display device; wherein the head mounted display device is wirelessly connected with the computing device.

The invention also provides a data stream transmission device, which is applied to the computing equipment and comprises:

the image acquisition module is used for acquiring an image frame to be displayed;

the image segmentation module is used for segmenting the image frames to be displayed according to the display sequence of the image frames to be displayed, and generating respective segmentation data frames of the image frames to be displayed; the image frame to be displayed is any image frame to be displayed, wherein the image frame to be displayed comprises image frame codes, fragment codes and image fragment data, and the display sequence number corresponding to the image frame codes of the current image frame to be displayed is larger than the display sequence number corresponding to the image frame codes of the previous image frame to be displayed;

The image sending module is used for sending the fragmented data frames to a head-mounted display device so as to display the image frames to be displayed by utilizing a display screen of the head-mounted display device; wherein the head mounted display device is wirelessly connected with the computing device.

In addition, the invention also provides electronic equipment, which comprises:

a memory for storing a computer program;

a processor for implementing the steps of the data stream transmission method as described above for a head mounted display device and/or the data stream transmission method as described above for a computing device when executing the computer program.

The invention provides a data stream transmission method, which is applied to head-mounted display equipment and comprises the following steps: buffering the fragmented data frames of the image frames to be displayed, which are sent by the computing equipment, by utilizing a data frame buffer; the head-mounted display device is in wireless connection with the computing device, and the slicing data frames comprise image frame codes, slicing codes and image slicing data; framing the image slice data according to the image frame coding and the slice coding to obtain an image frame to be displayed; displaying the image frame to be displayed by using a display screen of the head-mounted display device; the display sequence number corresponding to the image frame code of the current image frame to be displayed on the display screen is larger than the display sequence number corresponding to the image frame code of the last image frame to be displayed on the display screen, and the current image frame to be displayed is any image frame to be displayed;

Therefore, the invention can group frames of all image slice data of the same image frame to be displayed in the data frame buffer zone through the image frame coding and the slice coding in the slice data frame, so as to obtain the image frame to be displayed, and avoid the problem of display screen display caused by the error of the restored image frame; and through the setting of the image frame codes corresponding to the display sequence, the problem of abnormal display caused by disordered display of the displayed image frames can be avoided, the problem of abnormal display of the head-mounted display equipment caused by disordered transmission data is solved, and the user experience is improved. In addition, the invention also provides a data stream transmission device applied to the head-mounted display equipment, a data stream transmission method and device applied to the computing equipment and the electronic equipment, and the data stream transmission device and the data stream transmission method and device have the same beneficial effects.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a prior art two-image frame slicing scheme;

fig. 2 is a flowchart of a data stream transmission method according to an embodiment of the present invention;

fig. 3 is a schematic diagram of a structure of a sliced data frame of another data stream transmission method according to an embodiment of the present invention;

fig. 4 is a schematic diagram of another data stream transmission method according to an embodiment of the present invention;

fig. 5 is a flowchart of another data stream transmission method according to an embodiment of the present invention;

fig. 6 is a block diagram of a data stream transmission apparatus according to an embodiment of the present invention;

fig. 7 is a flowchart of another data stream transmission method according to an embodiment of the present invention;

fig. 8 is a block diagram of another data stream transmission apparatus according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Referring to fig. 2, fig. 2 is a flowchart of a data stream transmission method according to an embodiment of the invention. The method is applied to the head-mounted display device and can comprise the following steps:

step 101: buffering the fragmented data frames of the image frames to be displayed, which are sent by the computing equipment, by utilizing a data frame buffer; the head-mounted display device is in wireless connection with the computing device, and the slicing data frames comprise image frame codes, slicing codes and image slicing data.

It is to be understood that the head-mounted display device in this embodiment may be a device body for displaying contents, such as AR glasses or VR glasses, which is worn on the head of a user. The head-mounted display device in this embodiment may display image frames (i.e., image frames to be displayed) transmitted by a computing device (e.g., a host or a mobile phone), such as image frames in video data of scene content generated by the computing device. The head-mounted display device in this embodiment may be connected wirelessly with the computing device, such as a WiFi connection, to receive the tile data frame of the image frame to be displayed sent by the computing device.

For example, the head-mounted display device may be a head-mounted display device main body in a split head-mounted display device system, and a non-intelligent operating system such as a Linux (one type of operating system) system and an RTOS (Real-time operating system, real-time system) system is installed; the computing devices in the split head-mounted display device system can be provided with an intelligent operating system, such as an Android (an operating system) system; the computing device may access the internet, and the scene content displayed by the head-mounted display device may be computationally generated by the computing device, e.g., the computing device may send image frames corresponding to the scene content (i.e., image frames to be displayed) to the head-mounted display device via a WiFi connection, so as to be displayed on a display screen of the head-mounted display device.

Correspondingly, the image frame to be displayed in this embodiment may be an image frame that the computing device needs to send to the head-mounted display device, that is, an initial image frame that the head-mounted display device needs to display. Because the data size of the image frame to be displayed in each frame is larger, the computing device in this embodiment may segment the image frame to be displayed and then sequentially send the segmented data frame to the head-mounted display device, that is, the segmented data frame of the image frame to be displayed in this embodiment may be a data frame corresponding to the segmented data of the image frame to be displayed, that is, a data frame actually sent by the computing device to the head-mounted display device, for example, a data frame sent by the computing device through wireless connection and adopting a UDP protocol.

Accordingly, for the specific content of the sliced data frame of the image frame to be displayed in this embodiment, the designer may set the sliced data frame according to the usage scenario and the user requirement, as shown in fig. 3, the sliced data frame may include image frame coding (frame id), sliced coding (slice id) and sliced image data (i.e. image sliced data); the image frame codes can be codes corresponding to each frame of image frames to be displayed, namely, the image frame codes of different image frames to be displayed can be different, so that the segmented data frames of different image frames to be displayed can be distinguished through the arrangement of the image frame codes; the image frame code of each frame of image frame to be displayed may correspond to the display sequence number of the frame of image frame to be displayed, that is, the image frame code of the image frame to be displayed may correspond to the display sequence of the image frame to be displayed, and when the display sequence number corresponding to the image frame code of one image frame to be displayed is greater than the display sequence number corresponding to the image frame code of another image frame to be displayed, the other image frame to be displayed may be displayed in the display screen of the head-mounted display device prior to the image frame to be displayed. The slicing coding can be coding corresponding to each image slicing data obtained after each frame of image frame to be displayed is sliced, namely the slicing coding of different image slicing data of the same image frame to be displayed can be different, so that the slicing data frames of different image slicing data of the same image frame to be displayed are distinguished through the arrangement of the slicing coding, and the framing restoration of the image frame to be displayed is realized. The image slicing data may be image data obtained after slicing an image frame to be displayed. Correspondingly, the sliced data frame may further include other information such as the number of slices, the slice mode identifier or the transmission resolution identifier, so long as the head-mounted display device can group frames according to the information in the received sliced data frame of the image frame to be displayed, and restore the image frame to be displayed before slicing, which is not limited in this embodiment.

It should be noted that, the data frame buffer in this embodiment may be a buffer for buffering the received sliced data frames of the image frames to be displayed, which is set in the head-mounted display device, so that the image sliced data of the sliced data frames buffered in the data frame buffer is utilized to restore the image frames to be displayed before slicing.

Correspondingly, for the specific mode that the head-mounted display device utilizes the data frame buffer area to buffer the sliced data frames of the image frames to be displayed, which are sent by the computing device, the method can be set by a designer according to practical scenes and user requirements, for example, after the head-mounted display device receives the sliced data frames of the image frames to be displayed, which are sent by the computing device, the head-mounted display device can buffer the received sliced data frames to the data frame buffer area, namely, the data frame buffer area can buffer the complete sliced data frames; for example, according to the image frame code and the slice code in a slice data frame (i.e., the current slice data frame) currently received, the slice data frame is buffered to a buffer position corresponding to the slice code in a buffer area of the image frame code in a data frame buffer. After the head-mounted display device receives the sliced data frames of the image frames to be displayed, which are sent by the computing device, the head-mounted display device can buffer the image sliced data in the received sliced data frames to a data frame buffer zone, namely the data frame buffer zone can only buffer the image sliced data in the sliced data frames; for example, according to the image frame code and the slice code in one slice data frame (i.e. the current slice data frame) which are currently received, the image slice data in the slice data frame is cached to a cache position corresponding to the slice code in a cache region of the image frame code in a data frame buffer.

Correspondingly, for the specific capacity of the data frame buffer area in the embodiment, the specific capacity can be set by a designer according to a practical scene and user requirements, for example, the data frame buffer area can be used for simultaneously storing the sliced data frames or the image sliced data with the preset buffer quantity at most; the preset buffer number may be greater than or equal to the number of slices of the image frame to be displayed.

Correspondingly, the method provided by the embodiment can further include a configuration process of a data frame buffer, for example, the head-mounted display device can determine the number of fragments of the image frame to be displayed according to the maximum transmission unit information sent by the computing device and the transmission resolution of the image frame to be displayed; determining the capacity of a data frame buffer area according to the number of fragments of an image frame to be displayed, and configuring the data frame buffer area with the capacity; as shown in fig. 4, when the sending end (i.e. the computing device) can configure and adjust the resolution of the image frame to be displayed that needs to be transmitted after being acquired, the receiving end (i.e. the head-mounted display device) can automatically calculate the number of fragments of the image frame to be displayed according to the maximum transmission unit information sent by the computing device and the transmission resolution of the image frame to be displayed, so as to configure the data frame buffer area correspondingly; for example, the resolution of the image frame to be displayed (i.e., the transmission resolution) may be configured by the user's own choice, such as supporting 720p, 1080p, and 2K alike, and after the user selects the configured resolution on the computing device, the resolution may be sent to the head mounted display device. The head-mounted display device may also determine a capacity of the data frame buffer according to the number of tiles in the received tile data frame, and configure the data frame buffer of the capacity.

Step 102: and framing the image slice data according to the image frame coding and the slice coding to obtain the image frame to be displayed.

It can be understood that in this step, the head-mounted display device may perform framing on the image slice data in the slice data frame of the same image to be displayed according to the image frame code and the slice code in the slice data frame, and resume obtaining the image frame to be displayed before the slice, so as to realize streaming transmission of the image frame to be displayed. The specific framing method of the head-mounted display device for framing the image slice data in the embodiment may correspond to the slicing method of slicing the image to be displayed in the computing device. For example, the head-mounted display device may frame the image tile data according to a framing manner (such as line-by-line framing or column-by-column framing) corresponding to the preset tile manner of the computing device. The head-mounted display device can also frame the image fragment data according to the fragment mode information sent by the computing device and the framing mode corresponding to the fragment mode information; as shown in fig. 4, when the sending end (i.e. the computing device) can configure and adjust the slicing mode of the image frame to be displayed, the receiving end (i.e. the head-mounted display device) can automatically determine the framing mode adopted for framing the image slicing data according to the slicing mode information sent by the computing device, i.e. the framing mode corresponding to the slicing mode information.

Correspondingly, after encoding (such as compression encoding) the image frame to be displayed, the computing device in this embodiment may segment the encoded image frame obtained by encoding, so as to obtain and send the segmented data frame of the image frame to be displayed to the head-mounted display device. Correspondingly, in the step, the head-mounted display device can group frames and decode the image slice data according to the image frame codes and the slice codes to acquire the image frames to be displayed.

Correspondingly, for the embodiment, the specific mode of framing the image sliced data according to the image frame coding and the slice coding to obtain the image frame to be displayed can be set by a designer according to a practical scene and user requirements, because the sliced data frame of the image frame to be displayed transmitted between the computing device and the head-mounted display device may have a packet loss condition, that is, the number of all sliced data frames of a certain image frame to be displayed actually received by the head-mounted display device may be smaller than the number of sliced data frames (that is, the number of slices) of the image frame to be displayed sent by the computing device, in order to avoid the problem that the head-mounted display device cannot display and update in time due to the fact that the number of the received sliced data frames of the certain image frame to be displayed is smaller than the number of slices, in this step, the head-mounted display device may frame the image sliced data of the current image frame according to the historical image frame buffered in the historical frame buffer area and the slice coding of the current image frame buffered, and obtain the combined image frame corresponding to the current image frame; the current image frame is any image frame to be displayed corresponding to encoding, and the historical image frame comprises a preset number of combined image frames corresponding to the image frames to be displayed before the current image frame. That is, the head-mounted display device may correct and display a to-be-displayed image frame by using a previously displayed image frame (i.e., a combined image frame) buffered in the history frame buffer when there is a packet loss in a slice data frame of the to-be-displayed image frame.

Correspondingly, the head-mounted display device can also store the image frames to be displayed obtained by framing into an asynchronous time warping queue (such as a ring queue); after the display synchronization signal is acquired, determining a target image frame according to the gesture data acquired by the gesture detection sensor of the head-mounted display device and the latest image frame to be displayed in the asynchronous time warping queue; the latest image frame to be displayed comprises one image frame to be displayed which is stored in the asynchronous time warping queue, and the target image frame is an image frame which is not displayed on a display screen of the head-mounted display device; the target image frame is displayed using a display screen of the head-mounted display device. That is, the head-mounted display device may utilize an ATW (Asynchronous Timewarp, asynchronous time warping) technology, after acquiring a display synchronization signal that needs to update display content, use a to-be-displayed image frame that is obtained by framing a non-packet-dropped slice data frame stored in an asynchronous time warping queue, and determine each image frame that needs to be displayed, for example, when there is a packet loss in a newly received slice data frame of the to-be-displayed image frame, directly discard the slice data frame of the to-be-displayed image frame; after the display synchronization signal is acquired, if the latest image frame to be displayed stored in the asynchronous time warp queue is not displayed, the image frame to be displayed stored in the asynchronous time warp queue is directly displayed; if the latest image frame to be displayed stored in the asynchronous time warp queue is displayed, generating and displaying an image frame corresponding to the gesture data acquired by a gesture detection sensor (such as a 6DOF (degree of freedom) sensor) of the head-mounted display device according to the image frame to be displayed stored in the asynchronous time warp queue, so that the frame inserting process of ATW is realized, the head-mounted display device is ensured to display and update in time, and the user experience is improved.

Correspondingly, the process of framing the image slice data of the current image frame according to the slice code in the historical image frame cached in the historical frame cache area and the slice data frame of the cached current image frame to obtain the combined image frame corresponding to the current image frame may include: after the buffering of all the sliced data frames of the current image frame is completed, if the number of the buffered all the sliced data frames of the current image frame is equal to the number of the sliced data frames of the current image frame, framing all the image sliced data of the current image frame according to the sliced codes in all the sliced data frames of the current image frame to obtain a combined image frame corresponding to the current image frame; if the number of all the cached sliced data frames of the current image frame is smaller than the sliced number, the missing sliced code of the current image frame is coded according to the sliced number and the sliced codes in all the sliced data frames of the current image frame; wherein the number of missing slice codes is the difference between the number of slices and the number of all slice data frames of the current image frame; according to the missing slice coding and the slice coding in all slice data frames of the current image frame, framing all image slice data of the current image frame and target slice data in the historical image frame, and obtaining a combined image frame corresponding to the current image frame; the target slice data is image slice data corresponding to the missing slice code.

Step 103: displaying the image frame to be displayed by using a display screen of the head-mounted display device; the display sequence number corresponding to the image frame code of the current image frame to be displayed on the display screen is larger than the display sequence number corresponding to the image frame code of the last image frame to be displayed on the display screen, and the current image frame to be displayed is any image frame to be displayed.

It can be understood that, in this embodiment, the head-mounted display device may display an image frame (i.e., an image frame to be displayed) obtained after framing the image slice data by using the display screen, so as to implement sequential display of the image frame to be displayed to be sent by the computing device, that is, since the image frame code in the slice data frame corresponds to the display sequence of the image frame to be displayed, the display sequence number corresponding to the image frame code of the current image frame to be displayed by the display screen may be greater than the display sequence number corresponding to the image frame code of the last image frame to be displayed by the display screen, that is, the current image frame to be displayed is later than the last image frame to be displayed in the display sequence configured by the image frame code by the computing device, so that it is ensured that no other image frames to be displayed before but later than the image frame to be displayed are received by the image frame to be displayed are displayed after displaying a certain image frame to be displayed, thereby avoiding occurrence of disorder of displayed image frames.

It should be noted that, in this embodiment, the split transmission of image data between the computing device and the head-mounted display device is taken as an example, and the transmission of other data (such as audio data) between the computing device and the head-mounted display device may be implemented in the same or similar manner as the method provided in this embodiment, which is not limited in any way.

In the embodiment of the invention, through the image frame coding and the setting of the fragment coding in the fragment data frame, all image fragment data of the same image frame to be displayed in the data frame buffer area can be framed to obtain the image frame to be displayed, and the problem of display screen display caused by the error of the restored image frame is avoided; and through the setting of the image frame codes corresponding to the display sequence, the problem of abnormal display caused by disordered display of the displayed image frames can be avoided, the problem of abnormal display of the head-mounted display equipment caused by disordered transmission data is solved, and the user experience is improved.

Based on the above embodiments, the embodiments of the present invention further provide a flowchart of another data stream transmission method, so as to improve user experience. Specifically, referring to fig. 5, fig. 5 is a flowchart of another data stream transmission method according to an embodiment of the present invention. The method is applied to the head-mounted display device and can comprise the following steps:

Step 201: the fragmented data frames of the current image frame sent by the computing device are buffered using a data frame buffer.

The slice data frame in this embodiment may include image frame encoding, slice encoding, and image slice data. The current image frame in this embodiment may encode a corresponding image frame to be displayed for an image frame currently received by the head-mounted display device and buffered in the data frame buffer.

Step 202: after the buffering of all the sliced data frames of the current image frame is completed, judging whether the number of the buffered all the sliced data frames of the current image frame is equal to the number of the sliced frames of the current image frame; if yes, go to step 203; if not, go to step 204.

It can be understood that, in this step, after determining that buffering of all the sliced data frames of the current image frame is completed, that is, after determining that all the receivable sliced data frames of the current image frame have been received and buffered, the head-mounted display device determines whether packet loss occurs in all the sliced data frames of the current image frame by determining whether the number of the buffered all the sliced data frames of the current image frame is equal to the number of the slices of the current image frame, so that when the number of the buffered all the sliced data frames of the current image frame is equal to the number of the slices of the current image frame, it is determined that packet loss does not occur; and when the number of all the cached sliced data frames of the current image frame is not equal to (i.e. smaller than) the number of the sliced data frames of the current image frame, determining that the packet loss condition occurs.

Correspondingly, the head-mounted display device in this embodiment may determine whether all the sliced data frames of the current image frame are buffered according to the image frame codes in the received sliced data frames. For example, after receiving a slice data frame of a preset number of frames (e.g., 1 or 2) of the next slice data frame, it is determined that buffering of all slice data frames of the current image frame is completed. For example, when the preset frame number is 1 and the image frame codes in the sliced data frames are digital codes, the image frame codes can be associated with the display sequence, that is, the larger the image frame codes, the larger the display sequence number, and the more the image frame codes need to be displayed later; the head-mounted display device can judge whether the image frame code in the current slicing data frame is larger than the image frame code corresponding to the current image frame or not; wherein the current sliced data frame is any sliced data frame currently received; if the current frame is greater than the corresponding image frame code of the current image frame, determining that all the slicing data frames of the current image frame are cached, and taking the current slicing data frame as one slicing data frame of the next image frame, and caching the current slicing data frame in a position corresponding to the next image frame in a data frame buffer area according to the slicing code in the current slicing data frame; if the current slice data frame is equal to the image frame code corresponding to the current image frame, taking the current slice data frame as one slice data frame of the current image frame, and caching the current slice data frame in a position corresponding to the current image frame in a data frame buffer area according to the slice code in the current slice data frame; if the frame code is smaller than the frame code corresponding to the current image frame, discarding the current sliced data frame so as to avoid caching the sliced data frames of the disordered image frames to be displayed in a receiving and caching stage of the sliced data frames.

That is, after the head-mounted display device receives the sliced data frame, if the frame id (i.e. the image frame code) in the sliced data frame is the same as the frame id in the previous sliced data frame, it indicates that the sliced data frame is the same image frame to be displayed, and the sliced data frame can be cached in a specified position according to the slice id (i.e. the sliced code) in the sliced data frame; if the frame id in the sliced data frame is different from the frame id in the previous sliced data frame, if the frame id is smaller than the previous frame id, the old data of a frame is received, and the old data can be discarded; if the frame id is greater than the previous frame id, it indicates that a new image frame to be displayed (i.e., the next image frame) is received, i.e., the buffering of all the sliced data frames of the current image frame is completed. For example, the slice data frame of the newly received image frame to be displayed is the 3 rd image frame to be displayed in the display sequence, and at this time, it may be determined that the slice id of the buffered current image frame (e.g., the 2 nd image frame to be displayed) has been received; if the 5 pieces of sliced data frames of the current image frame (the sliced number of the current image frame) are already cached, framing and screening the image sliced data in the 1 st to 5 th sliced data frames, and updating the cached content in the historical frame cache area; if the sliced data frames of less than 5 slices of the current image frame are cached, the image sliced data in the missing sliced data frames are complemented by the historical image frames cached in the historical frame cache region so as to carry out framing and screen-up, and the cached content in the historical frame cache region is updated; if the 1 st, 2 nd, 3 rd and 5 th image slicing data of the current image frame are cached, the missing 3 rd image slicing data can be obtained from the historical frame cache area, the 5 th image slicing data is complemented, and the combined image frames corresponding to the current image frame are generated for displaying in a framing mode.

Correspondingly, the method provided by the embodiment may further include a process of obtaining the number of slices of the current image frame, for example, the head-mounted display device may determine the number of slices and the capacities of the data frame buffer and the history frame buffer according to the Maximum Transmission Unit (MTU) information sent by the computing device and the transmission resolution of the image frame to be displayed, so as to configure the data frame buffer and the history frame buffer by using the determined capacities of the data frame buffer and the history frame buffer, and complete initialization or configuration adjustment of the data frame buffer and the history frame buffer. For example, if the transmission resolution sent by the computing device is 720p and the mtu information is 1500 bytes, the head-mounted display device may determine that the number of slices is 1844 slices, i.e., (720×1280×3/1500.0) is rounded up.

Step 203: and framing all image slice data of the current image frame according to slice coding in all slice data frames of the current image frame to obtain a combined image frame corresponding to the current image frame.

It can be understood that, in this embodiment, when the number of all the cached fragmented data frames of the current image frame is equal to the number of fragments of the current image frame, that is, when no packet loss occurs in the transmission of all the fragmented data frames of the current image frame, the head-mounted display device may directly use all the image fragmented data of the current image frame to perform framing, so as to obtain a combined image frame corresponding to the current image frame, that is, the initial current image frame.

Step 204: according to the number of slices and the slice codes in all slice data frames of the current image frame, the missing slice codes of the current image frame; wherein the number of missing slice encodings is the difference between the number of slices and the number of full slice data frames of the current image frame.

It can be understood that, in this embodiment, the head-mounted display device may determine the slice code (i.e. the missing slice code) of the slice data frame that is not received when the number of all the slice data frames of the cached current image frame is smaller than the slice number of the current image frame, that is, the transmission of all the slice data frames of the current image frame is lost, so as to obtain the corresponding image slice data (i.e. the image slice data) from the history frame buffer by using the determined missing slice code.

Furthermore, the head-mounted display device in this embodiment may further determine whether a quotient of the number of missing slice codes and the number of slices is greater than a proportional threshold; if the ratio is not greater than the ratio threshold, directly ending; if the transmission resolution of the image frames to be displayed is larger than the proportion threshold value, sending packet loss serious prompt information to the computing equipment so as to prompt the computing equipment to reduce the transmission resolution of the image frames to be displayed, wherein the computing equipment can automatically lower the transmission resolution by one file; or outputting transmission resolution reduction prompt information to prompt a user to configure to reduce the transmission resolution. Accordingly, for cases where the ratio threshold is greater than the ratio threshold, the head mounted display device may also output computing device resolution adjustment information using a display screen or speaker to prompt a user wearing the head mounted display device to operate the computing device to reduce the transmission resolution.

Step 205: according to the missing slice coding and the slice coding in all slice data frames of the current image frame, framing all image slice data of the current image frame and target slice data in the history image frame cached in the history frame cache area, and obtaining a combined image frame corresponding to the current image frame; the target slice data is image slice data corresponding to the missing slice code.

It can be understood that, in this step, the head-mounted display device may acquire the target tile data for replacing the missing image tile data of the current image frame by using the historical image frames buffered in the historical frame buffer, that is, the previously displayed combined image frames, so as to perform framing by using the target tile data and all the image tile data that are not missing in the current image frame, and acquire the image frames (that is, the combined image frames) displayed on the display screen corresponding to the current image frame.

Correspondingly, for the specific mode that the head-mounted display device in the step groups frames according to the missing slice codes and the slice codes in all slice data frames of the current image frame, all image slice data of the current image frame and target slice data in the historical image frame cached in the historical frame cache region to obtain the combined image frame corresponding to the current image frame, the specific mode can be set by a designer according to a practical scene and user requirements, for example, the head-mounted display device can obtain the image slice data (namely the target slice data) corresponding to the missing slice codes from the latest combined image frame cached in the historical frame cache region according to the missing slice codes; according to the missing slice coding and the slice coding in all slice data frames of the current image frame, framing all image slice data and target slice data of the current image frame, and obtaining a combined image frame corresponding to the current image frame; that is, the historical image frames buffered in the historical frame buffer area are the combined image frames corresponding to the nearest image frame to be displayed before the current image frame, that is, the historical frame buffer area can be only used for buffering the latest combined image frames, so that the occupied space of the historical frame buffer area is reduced. The head-mounted display device can acquire image fragment data corresponding to the missing fragment codes from a plurality of newer combined image frames cached in the historical frame cache region according to the missing fragment codes; generating one image slice data (namely target slice data) corresponding to the missing slice code according to the plurality of image slice data corresponding to the missing slice code; and framing all image slice data and target slice data of the current image frame according to the missing slice code and the slice code in all slice data frames of the current image frame, and obtaining a combined image frame corresponding to the current image frame. The present embodiment does not impose any limitation on this.

Step 206: and updating the historical image frames cached in the historical frame cache area according to the combined image frames corresponding to the current image frames, and displaying the combined image frames corresponding to the current image frames by utilizing a display screen of the head-mounted display device.

In this embodiment, when the display screen of the head-mounted display device is used to display the combined image frame corresponding to the current image frame, the displayed combined image frame may be cached as a historical image frame in the historical frame cache region, so as to facilitate subsequent use.

Corresponding to the above method embodiment, the embodiment of the present invention further provides a data stream transmission device applied to the head-mounted display device, and a data stream transmission device applied to the head-mounted display device and a data stream transmission method applied to the head-mounted display device described below can be referred to correspondingly.

Referring to fig. 6, fig. 6 is a block diagram illustrating a data stream transmission apparatus according to an embodiment of the invention. The apparatus is applied to a head-mounted display device, and may include:

a frame buffer module 10, configured to buffer, with a data frame buffer, a sliced data frame of an image frame to be displayed, where the sliced data frame is sent by the computing device; the head-mounted display device is in wireless connection with the computing device, and the slicing data frames comprise image frame codes, slicing codes and image slicing data;

The framing module 20 is configured to frame the image slice data according to the image frame coding and the slice coding, and obtain an image frame to be displayed;

a display module 30 for displaying an image frame to be displayed using a display screen of the head-mounted display device; the display sequence number corresponding to the image frame code of the current image frame to be displayed on the display screen is larger than the display sequence number corresponding to the image frame code of the last image frame to be displayed on the display screen, and the current image frame to be displayed is any image frame to be displayed.

In some embodiments, the framing module 20 may be specifically configured to perform framing on the image slice data of the current image frame according to the historical image frame cached in the historical frame cache area and the slice code in the slice data frame of the cached current image frame, so as to obtain a combined image frame corresponding to the current image frame; the current image frame is any image frame to be displayed corresponding to encoding, and the historical image frame comprises a preset number of combined image frames corresponding to the image frames to be displayed before the current image frame;

correspondingly, the display module 30 may be specifically configured to update the historical image frames cached in the historical frame buffer area according to the combined image frame corresponding to the current image frame, and display the combined image frame corresponding to the current image frame by using the display screen of the head-mounted display device.

In some embodiments, framing module 20 may include:

the first framing sub-module is used for framing all image sliced data of the current image frame according to the sliced codes in all the sliced data frames of the current image frame if the number of the all the sliced data frames of the cached current image frame is equal to the number of the sliced data frames of the current image frame after the caching of all the sliced data frames of the current image frame is completed, so as to obtain a combined image frame corresponding to the current image frame;

the deletion determination sub-module is used for determining the number of the cached all sliced data frames of the current image frame after the caching of the all sliced data frames of the current image frame is completed, and if the number of the cached all sliced data frames of the current image frame is smaller than the number of the slices, the deletion sliced coding of the current image frame is performed according to the number of the slices and the sliced coding in the all sliced data frames of the current image frame; wherein the number of missing slice codes is the difference between the number of slices and the number of all slice data frames of the current image frame;

the second framing sub-module is used for framing all image slice data of the current image frame and target slice data in the historical image frame according to the missing slice coding and the slice coding in all slice data frames of the current image frame, and obtaining a combined image frame corresponding to the current image frame; the target slice data is image slice data corresponding to the missing slice code.

In some embodiments, framing module 20 may further include:

the buffer memory determining submodule is used for determining whether all the fragment data frames of the current image frame are buffered according to the received image frame codes in the fragment data frames of the image frame to be displayed; if yes, sending a starting signal to the first group of frame sub-modules and the deletion determination sub-module.

In some embodiments, the cache determination submodule may include:

a frame code judging unit for judging whether the image frame code in the current sliced data frame is larger than the image frame code corresponding to the current image frame; the current sliced data frame is any sliced data frame of the received image frame to be displayed, and the image frame is coded into a digital code;

the determining unit is used for determining that the buffering of all the slicing data frames of the current image frame is completed if the image frame code corresponding to the current image frame is larger than the image frame code corresponding to the current image frame;

the buffer unit is used for taking the current sliced data frame as one sliced data frame of the current image frame if the current sliced data frame is equal to the image frame code corresponding to the current image frame, and buffering the current sliced data frame in a position corresponding to the current image frame in the data frame buffer area according to the sliced code in the current sliced data frame;

And the discarding unit is used for discarding the current slicing data frame if the image frame code corresponding to the current image frame code is smaller than the image frame code corresponding to the current image frame code.

In some embodiments, the apparatus may further comprise:

the fragmentation determining module is used for determining the number of fragments and the capacities of the data frame buffer area and the historical frame buffer area according to the maximum transmission unit information sent by the computing equipment and the transmission resolution of the image frames to be displayed.

In some embodiments, the apparatus may further comprise:

the missing judging module is used for judging whether the quotient of the number of missing slice codes and the number of slices is larger than a proportional threshold value or not;

and the prompting module is used for sending the packet loss serious prompting information to the computing equipment if the packet loss serious prompting information is larger than the proportion threshold value so as to prompt the computing equipment to reduce the transmission resolution of the image frame to be displayed.

In some embodiments, the historical image frame is a combined image frame corresponding to the nearest image frame to be displayed prior to the current image frame.

Corresponding to the above method embodiment, the present invention further provides a data stream transmission method applied to a computing device, and a data stream transmission method applied to a computing device and a data stream transmission method applied to a head-mounted display device described above can be referred to correspondingly.

Referring to fig. 7, fig. 7 is a flowchart of another data stream transmission method according to an embodiment of the invention. The method is applied to a computing device and can include:

step 301: and acquiring an image frame to be displayed.

The image frame to be displayed in this embodiment may be an image frame that the computing device needs to send to the head-mounted display device, so as to display the image frame to be displayed by using a display screen of the head-mounted display device.

Step 302: fragmenting the image frames to be displayed to generate respective fragment data frames of the image frames to be displayed; wherein the slice data frames include image frame encoding, slice encoding, and image slice data.

It can be understood that, in this embodiment, the computing device may segment the image frames to be displayed according to a preset segmentation mode, so as to generate respective segmentation data frames of each image frame to be displayed; the number of the slice data frames of each image frame to be displayed may be a preset number of slices. For example, the computing device may determine the number of tiles based on the transmission resolution of the image frame to be displayed and the MTU information. As shown in fig. 4, when the sending end (i.e., the computing device) configures and adjusts the resolution of the image frame to be displayed that needs to be transmitted after acquisition, the computing device may determine, according to the acquired resolution configuration instruction, the resolution corresponding to the resolution configuration instruction as the transmission resolution of the image frame to be displayed.

Correspondingly, when the computing device configures the preset slicing mode, the computing device can determine the slicing mode corresponding to the slicing mode configuration instruction as the preset slicing mode according to the acquired slicing mode configuration instruction.

Further, in this step, the computing device may encode (e.g., compression encode) the image frame to be displayed, to obtain an encoded image frame; and fragmenting the coded image frames to generate respective fragment data frames of each image frame to be displayed, so that the transmission data quantity is reduced through the coding of the image frames to be displayed, and the information safety is ensured.

Step 303: transmitting the fragmented data frames to a head-mounted display device to display image frames to be displayed by using a display screen of the head-mounted display device; wherein the head mounted display device is wirelessly connected with the computing device.

Further, the computing device in this embodiment may also receive the packet loss serious prompt information sent by the head-mounted display device; outputting transmission resolution reduction prompt information according to the packet loss severity prompt information to prompt a user to configure to reduce the transmission resolution, or adjusting the transmission resolution according to the packet loss severity prompt information, wherein the transmission resolution after adjustment is smaller than the transmission resolution before adjustment, for example, after the packet loss severity prompt information is acquired, automatically reducing the transmission resolution by one file.

In the embodiment of the invention, the head-mounted display device can frame all image fragment data of the same image frame to be displayed through the image frame coding and the fragment coding in the fragment data frame to obtain the image frame to be displayed, so that the problem of display screen display caused by the error of the recovered image frame is avoided; and through the setting of the image frame codes corresponding to the display sequence, the problem of abnormal display of the head-mounted display equipment caused by disordered display of the image frames can be avoided, the problem of abnormal display of the head-mounted display equipment caused by disordered transmission data is solved, and the user experience is improved.

Corresponding to the above method embodiment, the present invention further provides a data stream transmission device applied to a computing device, and a data stream transmission device applied to a computing device and a data stream transmission method applied to a computing device described below can be referred to correspondingly.

Referring to fig. 8, fig. 8 is a block diagram illustrating another data stream transmission apparatus according to an embodiment of the present invention. The apparatus is applied to a computing device and may include:

an image acquisition module 40 for acquiring an image frame to be displayed;

The image slicing module 50 is configured to slice the image frames to be displayed according to the display sequence of the image frames to be displayed, and generate respective slicing data frames of each image frame to be displayed; the method comprises the steps that a slice data frame comprises image frame codes, slice codes and image slice data, a display sequence number corresponding to the image frame codes of the current image frame to be displayed is larger than a display sequence number corresponding to the image frame codes of the previous image frame to be displayed, and the current image frame to be displayed is any image frame to be displayed;

an image transmission module 60 for transmitting the fragmented data frames to the head-mounted display device to display the image frames to be displayed with the display screen of the head-mounted display device; wherein the head mounted display device is wirelessly connected with the computing device.

Corresponding to the above method embodiment, the embodiment of the present invention further provides an electronic device, and an electronic device described below and a data stream transmission method applied to a head-mounted display device and a data stream transmission method applied to a computing device described above may be referred to correspondingly.

Referring to fig. 9, fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the invention. The electronic device may include:

A memory D1 for storing a computer program;

the processor D2 is configured to implement the data stream transmission method applied to the head-mounted display device and/or the data stream transmission method applied to the computing device provided in the above method embodiments when executing the computer program.

The embodiment is not limited to a specific device type of the electronic device, and the electronic device provided in the embodiment may be a head-mounted display device, such as AR glasses; the electronic device provided in this embodiment may also be a computing device, such as a host or a mobile phone.

Corresponding to the above method embodiments, the embodiments of the present invention further provide a computer readable storage medium, where a data stream transmission method applied to a head-mounted display device and a data stream transmission method applied to a computing device described in the following description are referred to correspondingly.

A computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the data stream transmission method applied to a head mounted display device and/or the data stream transmission method applied to a computing device provided in the above method embodiments.

The computer readable storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, which may store various program codes.

In the description, each embodiment is described in a progressive manner, and each embodiment is mainly described by the differences from other embodiments, so that the same similar parts among the embodiments are mutually referred. The apparatus, the electronic device and the computer readable storage medium disclosed in the embodiments have a relatively simple description, and the relevant points refer to the description of the method section since the apparatus, the electronic device and the computer readable storage medium correspond to the method disclosed in the embodiments.

The method, the device and the electronic equipment for transmitting the data stream provided by the invention are described in detail. The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to facilitate an understanding of the method of the present invention and its core ideas. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the invention can be made without departing from the principles of the invention and these modifications and adaptations are intended to be within the scope of the invention as defined in the following claims.

Claims

1. A data stream transmission method, applied to a head-mounted display device, comprising:

2. The method for data streaming according to claim 1, wherein said framing the image slice data according to the image frame encoding and the slice encoding to obtain the image frame to be displayed comprises:

3. The method for transmitting data streams according to claim 2, wherein the framing the image slice data of the current image frame according to the slice codes in the buffered historical image frames in the historical frame buffer and the buffered slice data frames of the current image frame to obtain the combined image frame corresponding to the current image frame includes:

4. The data streaming transmission method according to claim 3, wherein the framing the image slice data of the current image frame according to the slice code in the historical image frame buffered in the historical frame buffer and the slice data frame of the buffered current image frame, and obtaining the combined image frame corresponding to the current image frame, further comprises:

5. The method for data streaming according to claim 4, wherein said determining whether buffering of all the sliced data frames of the current image frame is completed according to the received image frame codes in the sliced data frames of the image frame to be displayed comprises:

judging whether the image frame code in the received current slicing data frame is larger than the image frame code corresponding to the current image frame or not; the current sliced data frame is any sliced data frame of the received image frame to be displayed, and the image frame is coded into a digital code;

6. The data stream transmission method of claim 3, further comprising:

7. The data stream transmission method according to claim 6, wherein the encoding of slices in all slices of the current image frame according to the number of slices and the slice encoding of the current image frame, further comprises, after the encoding of the missing slices of the current image frame:

8. The method of claim 3, wherein the historical image frame is a combined image frame corresponding to a nearest image frame to be displayed before the current image frame.

9. A data stream transmission apparatus, applied to a head-mounted display device, comprising:

10. A method for data streaming, applied to a computing device, comprising:

acquiring an image frame to be displayed;

11. A data stream transmission apparatus, for use in a computing device, comprising:

12. An electronic device, comprising:

a memory for storing a computer program;

processor for implementing the data stream transmission method according to any one of claims 1 to 8 and/or the data stream transmission method according to claim 10 when executing the computer program.