CN111298427A - Method for reducing picture jitter in virtual reality cloud game system - Google Patents
Method for reducing picture jitter in virtual reality cloud game system Download PDFInfo
- Publication number
- CN111298427A CN111298427A CN202010079711.8A CN202010079711A CN111298427A CN 111298427 A CN111298427 A CN 111298427A CN 202010079711 A CN202010079711 A CN 202010079711A CN 111298427 A CN111298427 A CN 111298427A
- Authority
- CN
- China
- Prior art keywords
- virtual reality
- client
- cloud game
- image
- video stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 18
- 238000005516 engineering process Methods 0.000 claims abstract description 13
- 238000012937 correction Methods 0.000 claims description 2
- 230000005540 biological transmission Effects 0.000 abstract description 10
- 238000012545 processing Methods 0.000 abstract description 5
- 210000003128 head Anatomy 0.000 description 4
- 238000009877 rendering Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000033001 locomotion Effects 0.000 description 2
- 230000004043 responsiveness Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 208000002173 dizziness Diseases 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000007654 immersion Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000035807 sensation Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/30—Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
- A63F13/31—Communication aspects specific to video games, e.g. between several handheld game devices at close range
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/40—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterised by details of platform network
- A63F2300/402—Communication between platforms, i.e. physical link to protocol
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Information Transfer Between Computers (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses a method for reducing picture jitter in a virtual reality cloud game system. The method comprises the following specific steps: s1, acquiring a virtual reality image to be transmitted at a server end; s2, coding the image into video stream by adopting a block parallel coding mode; s3 transmitting the video stream by using network transmission protocol; s4, receiving the video stream transmitted from the server side by the client side in a buffer area mode; s5, decoding the video frame to be displayed into a display image in a parallel decoding mode; s6, using asynchronous distortion technology to complete the on-screen display of the image at the display client; s7 obtains the current client pose and predicts the client pose after a period of time. By the method, the problem of picture jitter caused by hardware processing time fluctuation and network transmission time fluctuation in the virtual reality cloud game system can be effectively solved, and the subjective experience quality of a user in the virtual reality cloud game is improved.
Description
Technical Field
The invention relates to the field of cloud games and computing vision, in particular to a method for reducing picture jitter in a virtual reality cloud game system.
Background
Virtual Reality (VR) is a new technology in the field of computers that has been developed by integrating various scientific technologies such as computer graphics, human-computer interface technology, network technology, stereoscopic display technology, simulation technology, and the like. VR technology can create a highly realistic virtual environment that acts on the user's vision, hearing, and touch, immerses the user in it, and interacts with the virtual world to create an immersive experience. This immersive environment can resemble the real world creating an experience that is not possible in ordinary physical reality. Augmented reality systems can be considered a form of VR that feeds virtual information through a live camera to headphones or through a smartphone or tablet device, enabling a user to view three-dimensional images. VR technology is currently used in many areas such as medicine, military, machine manufacturing, aerospace, etc. Current VR technology most often uses virtual reality headphones or multi-projection environments, sometimes in combination with physical environments or props, to generate realistic images, sounds, and other sensations that simulate the physical presence of a user in a virtual or fictional environment. A person using a virtual reality device can "look around" the artificial world, move within it, and interact with virtual features or items.
By 2016, at least 230 companies developed VR related products. Facebook owns 400 employees dedicated to VR development; ***, apple, amazon, microsoft, sony, and samsung all have specialized AR and VR communities. On 5/4/2016, HTC distributed the first HTC view head, the first major version of sensor-based tracking, allowing users to move freely in a certain space.
With the development of VR devices, VR devices with more powerful functions and more reasonable prices are affecting the traditional gaming industry. Such head-up devices, represented by Oculus Rift, HTC Vive, and PlayStation VR, make it easier for game players to experience VR games. In the gaming industry, more and more companies are investing large resources to develop games for emerging VR devices. VR games provide game players with an unparalleled experience of immersion in the virtual world, richer control, and more direct interaction with game elements than traditional games.
However, to maintain a highly immersive VR experience, the picture quality and operational responsiveness of the head display is essential. For picture quality, a realistic and high definition game environment needs to be presented in the head-up display. With respect to responsiveness, user actions, particularly head actions, require that the backend system that processes VR content be reflected as quickly as possible in the virtual world. If this requirement is not met, it can cause the user to feel dizzy, which greatly affects the player's VR experience. Therefore, VR games typically require higher specification hardware devices with higher requirements and higher costs than traditional games. According to great research, it is shown that to obtain a smooth VR game experience, games need to be run at 2K resolution and 90fps frame rate, whereas traditional PC games only need 1080P resolution and 30fps frame rate. Thus, the demand for more demanding gaming environments still hinders the popularity of many potential users who cannot afford such gaming systems.
To maintain a smooth VR experience, it has been proposed in recent years to centralize computing devices, utilizing cloud gaming technology to improve accessibility of the VR experience. Cloud gaming is an emerging gaming service that provides a novel and highly accessible gaming experience for millions of users. By utilizing elastic and high-performance computing resources, cloud games shift the intensive workload of game processing from the client device to the cloud server. Cloud games relieve game players from purchasing expensive gaming machines and allow them to play high quality games using any device.
Modern gaming is a computationally intensive application. Computer games employ advanced graphics rendering to create realistic scenes and smooth movements. Thus, games typically require powerful hardware, which is beyond the capabilities of many mobile devices and even personal computers. Meanwhile, with the high-speed mobile and the increasing popularization of broadband internet and mobile devices such as smart phones and tablet computers, it is very desirable to play high-quality videos in mobile. Cloud games use cloud computing to play games, rather than equipping mobile devices with powerful but power-hungry computing engines, thus providing an emerging green solution that can bring a high-quality immersive gaming experience to customers or mobile clients. Cloud games utilize a communication infrastructure to transfer heavy computing to a cloud server.
Fast response and high definition graphics quality are two key factors in maintaining a highly immersive VR experience. However, jitter problems in VR cloud games greatly affect the experience of VR cloud games due to unavoidable network delays and limited transmission bandwidth. However, a scheme for reducing the jitter problem in the virtual reality cloud game system is not proposed yet, and how to effectively reduce the image jitter in the cloud game system and improve the experience quality of the virtual reality cloud game system becomes an important topic.
Disclosure of Invention
In view of the above circumstances, the present invention provides a method for reducing picture jitter in a virtual reality cloud game system, which solves the problem of picture jitter in a virtual reality cloud game by using multiple ways to ensure a client display frame rate and an end-to-end delay.
The technical scheme adopted by the invention is as follows:
a method for reducing picture jitter in a virtual reality cloud game system comprises the following steps:
s1, the server side firstly obtains the gesture data collected from the client side, and then renders the image in the virtual reality game according to the gesture data;
s2, the image of step S1 is divided into blocks, and the blocks are encoded into a video stream which is convenient to transmit in parallel and transmitted to the client;
s3, the client decodes the acquired video stream into display images in parallel;
s4, displaying the display image of step S3 in a virtual reality helmet, specifically: the client side firstly acquires the attitude information of a current user according to the gyroscope device embedded in the helmet device, then acquires the attitude prediction interval parameter sent from the server side, finally predicts to obtain a predicted attitude by utilizing the attitude information and the attitude prediction interval parameter, and renders a display picture according to the predicted attitude information by using an asynchronous distortion technology, so that the frame rate of the client side is ensured, and meanwhile, the jitter of the picture is reduced.
The invention reduces the coding and decoding processing time of the virtual reality picture by using a parallel coding and parallel decoding mode at the receiving end and the server end of the virtual reality equipment, ensures that the equipment can display at a high frame rate, and improves the user experience quality. Meanwhile, a buffer area and a head motion prediction mode are used at a receiving end of the virtual reality equipment to ensure that the equipment can display at a stable frame rate, and the user experience quality is improved. The method can effectively reduce the problem of picture jitter caused by the fluctuation of hardware processing time and the fluctuation of network transmission time in the virtual reality cloud game system, and improve the subjective experience quality of the user in the virtual reality cloud game.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
fig. 2 is a virtual reality cloud game architecture timeline diagram.
Detailed Description
The invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
In this embodiment, a virtual reality device VIVEFocus of the HTC is used as an experience client of the virtual reality cloud game.
Referring to fig. 1, a method for reducing frame jitter in a virtual reality cloud game system according to the present invention includes the following steps:
s1, before the user starts browsing, the server side firstly obtains the gesture data collected from the client side, and then renders the image in the virtual reality game according to the gesture data;
s2, in order to minimize the time spent on encoding the image, the image is directly obtained from the server graphics card.
And S3, coding the acquired image into a video stream convenient for transmission. In this step, since the virtual reality game generally has two pictures, for encoding, the total encoding time can be reduced by using one encoder for encoding simultaneously and parallelly for the left and right eyes;
s4, the video stream encoded in step S3 is transmitted to the client, in which step different transmission protocols can be adopted for different application scenarios. For the use scene of the home local area network, the transmission mode of the TCP can be adopted to improve the picture quality. For the use scene of the internet, a transmission mode of UDP + forward error correction code can be adopted to reduce the transmission delay;
s5, after step S4, the client receives the video data transmitted from the server. In order to reduce the influence of network fluctuation and the fluctuation of image processing (rendering and encoding) time of a server on the display of a client, and ensure that the client displays a virtual reality game picture at a relatively stable frame rate, three video frame buffer areas are used at the client to receive video data transmitted from the server.
S6, for the video stream obtained in step S5, in order to reduce the time taken for image decoding to the maximum extent and ensure that the client can operate at a stable frame rate, the client decodes the left and right images simultaneously by using parallel decoding.
S7, in this step, images of left and right eyes are required to be displayed on the virtual reality helmet for user experience, and since the pose information used at the rendering time point of the server end is different from the pose information currently required to be rendered and displayed at the client end, an asynchronous warping technique is used to predict a pose information, which mainly includes, first, obtaining the current angle θ (rad), angular velocity ω (rad/sec), and angular acceleration a (rad/sec) from the gyroscope embedded in the virtual reality device2) Then, receiving a predicted time length parameter t (sec) sent from the server, and finally calculating the helmet posture after t (sec) according to the following formula:
the picture is rendered and displayed according to the posture information, and the jitter of the picture is reduced while the frame rate of the client side is ensured. The user starts browsing the current picture.
And S8, acquiring the current stable posture information of the user at a high frame rate through a gyroscope device embedded in the virtual reality helmet while the user browses the current picture. And considering the time delay caused by transmission, the user posture can be predicted for a certain time while the current user posture information is acquired, and the predicted user posture is sent to the server side. And repeating the steps S1 to S8 to complete the experience of the virtual reality cloud game.
Claims (5)
1. A method for reducing picture jitter in a virtual reality cloud game system is characterized by comprising the following steps:
s1, the server side firstly obtains the gesture data collected from the client side, and then renders the image in the virtual reality game according to the gesture data;
s2, the image of step S1 is divided into blocks, and the blocks are encoded into a video stream which is convenient to transmit in parallel and transmitted to the client;
s3, the client decodes the acquired video stream into display images in parallel;
s4, displaying the display image of step S3 in a virtual reality helmet, specifically: the client side firstly acquires the attitude information of a current user according to the gyroscope device embedded in the helmet device, then acquires the attitude prediction interval parameter sent from the server side, finally predicts to obtain a predicted attitude by utilizing the attitude information and the attitude prediction interval parameter, and renders a display picture according to the predicted attitude information by using an asynchronous distortion technology, so that the frame rate of the client side is ensured, and meanwhile, the jitter of the picture is reduced.
2. The method of claim 1, wherein after step S4, when the user browses the current screen, the client obtains information about the current user posture and adds a prediction of the user posture for a certain duration, and sends the predicted user posture to the server.
3. The method of claim 1 or 2, wherein in the step S2, the total encoding duration is reduced by using a single encoder for encoding the left and right eyes simultaneously in parallel.
4. The method for reducing picture jitter in a virtual reality cloud game system according to claim 1 or 2, wherein in step S2, the video stream is transmitted to the client by using TCP or UDP plus forward error correction code.
5. The method for reducing frame jitter in a virtual reality cloud game system as claimed in claim 1 or 2, wherein in step S3, a buffer is used at the client to receive the video stream transmitted from the server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010079711.8A CN111298427A (en) | 2020-02-04 | 2020-02-04 | Method for reducing picture jitter in virtual reality cloud game system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010079711.8A CN111298427A (en) | 2020-02-04 | 2020-02-04 | Method for reducing picture jitter in virtual reality cloud game system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111298427A true CN111298427A (en) | 2020-06-19 |
Family
ID=71152895
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010079711.8A Pending CN111298427A (en) | 2020-02-04 | 2020-02-04 | Method for reducing picture jitter in virtual reality cloud game system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111298427A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112073363A (en) * | 2020-07-08 | 2020-12-11 | 北京兰亭数字科技有限公司 | Cloud rendering low-delay high-guarantee transmission protocol operation method |
| WO2022218209A1 (en) * | 2021-04-14 | 2022-10-20 | 华为技术有限公司 | Information processing method, apparatus and system |
| CN115278289A (en) * | 2022-09-27 | 2022-11-01 | 海马云（天津）信息技术有限公司 | Cloud application rendering video frame processing method and device |
| CN115643449A (en) * | 2022-09-16 | 2023-01-24 | 阿里巴巴（中国）有限公司 | Video display method, device, equipment, storage medium and system of cloud service |
| CN116886974A (en) * | 2023-09-07 | 2023-10-13 | 海马云（天津）信息技术有限公司 | Method and device for optimizing decoding rendering performance of terminal |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108702503A (en) * | 2016-02-17 | 2018-10-23 | 诺基亚技术有限公司 | For Video coding and decoded device, method and computer program |
| CN109126122A (en) * | 2017-06-16 | 2019-01-04 | 上海拆名晃信息科技有限公司 | A kind of cloud game network system realization for virtual reality |
| CN110505518A (en) * | 2019-08-05 | 2019-11-26 | 青岛小鸟看看科技有限公司 | One kind wearing display equipment, data transmission method and system |
| CN110519247A (en) * | 2019-08-16 | 2019-11-29 | 上海乐相科技有限公司 | A kind of one-to-many virtual reality display method and device |
-
2020
- 2020-02-04 CN CN202010079711.8A patent/CN111298427A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108702503A (en) * | 2016-02-17 | 2018-10-23 | 诺基亚技术有限公司 | For Video coding and decoded device, method and computer program |
| CN109126122A (en) * | 2017-06-16 | 2019-01-04 | 上海拆名晃信息科技有限公司 | A kind of cloud game network system realization for virtual reality |
| CN110505518A (en) * | 2019-08-05 | 2019-11-26 | 青岛小鸟看看科技有限公司 | One kind wearing display equipment, data transmission method and system |
| CN110519247A (en) * | 2019-08-16 | 2019-11-29 | 上海乐相科技有限公司 | A kind of one-to-many virtual reality display method and device |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112073363A (en) * | 2020-07-08 | 2020-12-11 | 北京兰亭数字科技有限公司 | Cloud rendering low-delay high-guarantee transmission protocol operation method |
| WO2022218209A1 (en) * | 2021-04-14 | 2022-10-20 | 华为技术有限公司 | Information processing method, apparatus and system |
| CN115643449A (en) * | 2022-09-16 | 2023-01-24 | 阿里巴巴（中国）有限公司 | Video display method, device, equipment, storage medium and system of cloud service |
| WO2024056095A1 (en) * | 2022-09-16 | 2024-03-21 | 杭州阿里云飞天信息技术有限公司 | Cloud service video display method and apparatus, device, storage medium, and system |
| CN115278289A (en) * | 2022-09-27 | 2022-11-01 | 海马云（天津）信息技术有限公司 | Cloud application rendering video frame processing method and device |
| CN116886974A (en) * | 2023-09-07 | 2023-10-13 | 海马云（天津）信息技术有限公司 | Method and device for optimizing decoding rendering performance of terminal |
| CN116886974B (en) * | 2023-09-07 | 2023-12-19 | 海马云（天津）信息技术有限公司 | Method and device for optimizing decoding rendering performance of terminal |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111298427A (en) | Method for reducing picture jitter in virtual reality cloud game system | |
| EP3760287B1 (en) | Method and device for generating video frames | |
| US10049495B2 (en) | System and method for providing virtual reality content | |
| EP3596931B1 (en) | Method and apparatus for packaging and streaming of virtual reality media content | |
| WO2019074313A1 (en) | Method and apparatus for rendering three-dimensional content | |
| EP3691280B1 (en) | Video transmission method, server, vr playback terminal and computer-readable storage medium | |
| US20190012826A1 (en) | Asynchronous time warp with depth data | |
| US20160225188A1 (en) | Virtual-reality presentation volume within which human participants freely move while experiencing a virtual environment | |
| JP2022525414A (en) | Systems and methods for predicting state using a distributed game engine | |
| CN112915537B (en) | Virtual scene picture display method and device, computer equipment and storage medium | |
| Müller et al. | PanoVC: Pervasive telepresence using mobile phones | |
| CN112291590A (en) | Video processing method and device | |
| KR102441514B1 (en) | Hybrid streaming | |
| US20180336069A1 (en) | Systems and methods for a hardware agnostic virtual experience | |
| US20220189105A1 (en) | 3d Conversations in An Artificial Reality Environment | |
| CN115639976B (en) | Multi-mode multi-angle synchronous display method and system for virtual reality content | |
| US11900530B1 (en) | Multi-user data presentation in AR/VR | |
| US20220172440A1 (en) | Extended field of view generation for split-rendering for virtual reality streaming | |
| Wanick et al. | Virtual transcendence experiences: Exploring technical and design challenges in multi-sensory environments | |
| CN112565883A (en) | Video rendering processing system and computer equipment for virtual reality scene | |
| GB2585479A (en) | Reduction of the effects of latency for extended reality experiences | |
| KR20200144702A (en) | System and method for adaptive streaming of augmented reality media content | |
| KR20160136160A (en) | Virtual Reality Performance System and Performance Method | |
| US11294615B2 (en) | Incorporating external guests into a virtual reality environment | |
| CN114053694A (en) | Application server, application service method thereof and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200619 |
|
| RJ01 | Rejection of invention patent application after publication |