WO2022218209A1 - Information processing method, apparatus and system - Google Patents

Information processing method, apparatus and system Download PDF

Info

Publication number
WO2022218209A1
WO2022218209A1 PCT/CN2022/085663 CN2022085663W WO2022218209A1 WO 2022218209 A1 WO2022218209 A1 WO 2022218209A1 CN 2022085663 W CN2022085663 W CN 2022085663W WO 2022218209 A1 WO2022218209 A1 WO 2022218209A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
terminal device
image information
server
attitude
Prior art date
Application number
PCT/CN2022/085663
Other languages
French (fr)
Chinese (zh)
Inventor
吴亮
吴昊
朱勇平
刘红波
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2022218209A1 publication Critical patent/WO2022218209A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
    • H04N21/25808—Management of client data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel

Definitions

  • Embodiments of the present invention relate to the technical field of virtual reality, and in particular, to an information processing method, device, and system.
  • VR technology is a computer simulation system that can create and experience virtual worlds. It uses computers to generate a simulated environment and immerse users in the environment. That is, VR technology is a multi-source information fusion, interactive three-dimensional dynamic scene and system simulation of entity behavior.
  • the cloud VR technology renders the image by the cloud device (server side), and displays the image by the client (VR terminal device, such as VR helmet, etc.).
  • the cloud device and the client need to interact, that is, the cloud device can receive the data collected from the client and render it, and the client can receive the data from the cloud device and display VR images.
  • the cloud device after the cloud device receives the gesture information from the client, it cannot render the image information according to the gesture information in time, so that the terminal device has a large delay in displaying the image information.
  • the embodiments of the present invention disclose an information processing method, device and system, which are used for reducing the time delay of image information display.
  • a first aspect discloses an information processing method.
  • the information processing method can be applied to a server or a module (eg, a chip) in the server.
  • the server is used as an example for description below.
  • the information processing method may include: when receiving attitude information from a terminal device, rendering first image information according to the attitude information to obtain second image information, where the attitude information is the user's attitude information, and the first image information is obtained.
  • the first image information is virtual reality VR image information; the second image information is sent to the terminal device.
  • the server since the server renders the image information once after waiting for the same time interval, when the server receives the gesture information, it usually needs to wait for a period of time before it can start to process the image information. Rendering will result in a certain delay in displaying image information.
  • the server when the server receives the gesture information, the image information can be rendered immediately, thereby reducing the waiting delay in the server processing process, and further reducing the delay in displaying the image information by the terminal device.
  • the terminal device can refresh the received image information in time, so that the images viewed by the user are smoother, thereby improving the user experience.
  • the rendering of the first image information according to the attitude information to obtain the second image information includes: determining a frame rate (FR) according to the attitude information; The first image information is rendered to obtain second image information.
  • FR frame rate
  • the server renders the image information according to a fixed frame rate.
  • the decoding capabilities and/or refresh capabilities of different terminal devices are often different. Some terminal devices have relatively high decoding capabilities and/or refresh capabilities, and can decode and display them in a timely manner when receiving image information from the server; while some terminal devices have relatively low decoding capabilities and/or refresh capabilities.
  • the server can determine the frame rate corresponding to the rendered image information according to the received attitude information, so that it can adapt to the decoding capability and/or refresh capability of different terminal devices, thereby improving the adaptability of the server to the terminal device. , it can also ensure the reliability of the picture information displayed by the terminal device. Further, when the reliability of displaying the image information by the terminal device is improved, the correctness of the image picture viewed by the user can be improved, thereby improving the user experience.
  • the determining the frame rate according to the gesture information includes: determining the frame rate according to the frequency of receiving the gesture information.
  • the server may determine the frame rate of the rendered gesture information according to the frequency of receiving the gesture information.
  • the terminal device adjusts the frequency of collecting attitude information according to its own decoding ability and/or refresh ability, and sends attitude information to the server according to the above frequency.
  • the server After the server receives the gesture information, it can determine the frequency of rendering according to the frequency of receiving the gesture information, so as to determine the frame rate of the rendered image information. Therefore, the server can adjust the frame rate of the rendered image information according to the frequency of receiving the gesture information, so that the image information rendered by the server can be adapted to the decoding capability and/or refresh capability of the terminal device, and can also be better adapted to the terminal device. current display needs.
  • the method may further include: encoding the second image information to obtain third image information; the sending the second image information to the terminal device includes: sending the second image information to the terminal device The device sends the third image information.
  • the server may encode the rendered image information, and then may send the rendered image information to the server.
  • the server encodes the rendered image information, which may be compression of the image information, thereby reducing the amount of transmitted image information, thereby saving transmission resources.
  • the encoded image information may also be encryption of the image information, so that the security and reliability of the image information can be improved.
  • a second aspect discloses an information processing method.
  • the information processing method can be applied to a terminal device, and can also be applied to a module (eg, a chip) in the terminal device.
  • the terminal device is used as an example for description below.
  • the information processing method may include: when it is detected that the terminal device is in an unused state, stopping collecting the first attitude information, and the unused state includes that the screen of the terminal device is in an off-screen state and/or the terminal device is not in use. It is worn by the user, and the first posture information is the posture information of the user.
  • the terminal device when the user does not use a terminal device such as a VR helmet, the terminal device still continuously displays images, which will cause waste of processing resources and energy. Therefore, when the display screen of the terminal device is in an off-screen state and/or when the user takes off the helmet, the user does not view the screen, and the terminal device can stop collecting the first posture information.
  • the terminal device When the terminal device does not collect the gesture information, the terminal device will not send the gesture information to the server, therefore, the server will not receive the gesture information, and at this time, the server will not render the image information. Therefore, when the user does not use the terminal device, the terminal device can stop collecting information, and the server can stop rendering image information, so that processing resources can be saved and energy waste can be avoided.
  • the method may further include: when the second gesture information is collected, sending the second gesture information to the server; receiving the second image information from the server; displaying the second gesture information the second image information.
  • the terminal device when the terminal device collects attitude information, it can immediately send the attitude information to the server, thereby ensuring that the frequency at which the terminal equipment collects attitude information is basically consistent with the frequency at which the server receives attitude information. Since the server immediately renders the first image information after receiving the attitude information, it can ensure that the frame rate of the second image information rendered by the server is consistent with the sending frequency of the terminal device, so that the terminal device can adjust the frequency of collecting attitude information. The frame rate of the image information rendered by the server can be adjusted accordingly, thereby ensuring that the terminal device can adapt to the frequency at which the server sends the second image information. In addition, when the terminal device receives the second image information, the second image information can be displayed, so that the integrity of the solution can be guaranteed.
  • the sending the second attitude information to the server includes: adjusting the second attitude information according to the decoding capability and/or refresh capability of the terminal device to obtain third attitude information, and the The decoding capability is the fastest decoding speed of the terminal device, and the refresh capability is the maximum frame rate displayed by the terminal device; the third gesture information is sent to the server.
  • the decoding capability and/or refresh capability of the terminal device cannot be synchronized with the server, when the server renders image information at a fixed frame rate and sends the above image information to the terminal device, the decoding capability and/or Terminal devices with weak refresh capabilities may not be able to decode and/or display image information in time, which may cause frame loss and frame skipping. Since the server immediately renders the image information when it receives the gesture information, the terminal device can adjust the gesture information according to its own decoding capability and/or refresh capability, so that the frame rate of the image information rendered by the server can match the decoding capability of the terminal device. and/or refresh capability. After the above-mentioned adjustment of the attitude information by the terminal device, it can be ensured that the terminal device can decode and/or refresh the image information in time, thereby improving user experience.
  • the receiving the second image information from the server includes: receiving third image information from the server; decoding the third image information to obtain the second image information image information.
  • the terminal device when the terminal device receives the third image information from the server, it needs to decode the third image information first.
  • the decoding process of the terminal device can correspond to the encoding process of the server, thereby ensuring the integrity of the information processing process.
  • the above-mentioned encoding and decoding process of image information can not only reduce the amount of transmitted information, save transmission resources, but also improve the reliability and stability of information.
  • the method may further include: when it is detected that the terminal device is in a use state, collecting the fourth posture information, and the use state includes that the screen is in a display state and/or the terminal device has been User wears.
  • the terminal device when the user resumes using the terminal device, the terminal device can continue to collect gesture information, so that the server can resume the rendering of the image information, and then the server can send the rendered image information to the terminal device. Further, when the terminal device receives the rendered image information, it can be displayed. When the user suspends using the terminal device to resume using the terminal device, the terminal device can be quickly resumed. In this way, not only can the user experience be guaranteed, but also processing resources can be saved, and energy waste can be avoided.
  • a third aspect discloses an information processing apparatus.
  • the information processing apparatus may be a server or a module (eg, a chip) in the server.
  • the information processing apparatus may include:
  • a rendering unit configured to render the first image information according to the attitude information when receiving the attitude information from the terminal device, and obtain the second image information, the attitude information is the attitude information of the user, the first image
  • the information is virtual reality VR image information
  • a sending unit configured to send the second image information to the terminal device.
  • the server since the server renders the image information once after waiting for the same time interval, when the server receives the gesture information, it usually needs to wait for a period of time before it can start to process the image information. Rendering will result in a certain delay in displaying image information.
  • the server when the server receives the gesture information, the image information can be rendered immediately, thereby reducing the waiting delay in the server processing process, and further reducing the delay in displaying the image information by the terminal device.
  • the terminal device can refresh the received image information in time, so that the images viewed by the user are smoother, thereby improving the user experience.
  • the rendering unit is specifically used for:
  • the first image information is rendered according to the frame rate to obtain second image information.
  • the server renders the image information according to a fixed frame rate.
  • the decoding capabilities and/or refresh capabilities of different terminal devices are often different. Some terminal devices have relatively high decoding capabilities and/or refresh capabilities, and can decode and display in time when receiving image information from the server; while some terminal devices have relatively low decoding capabilities and/or refresh capabilities.
  • the server can determine the frame rate corresponding to the rendered image information according to the received attitude information, so that it can adapt to the decoding capability and/or refresh capability of different terminal devices, thereby improving the adaptability of the server to the terminal device. , it can also ensure the reliability of the picture information displayed by the terminal device. Further, when the reliability of displaying the image information by the terminal device is improved, the correctness of the image picture viewed by the user can be improved, thereby improving the user experience.
  • the rendering unit determining the frame rate according to the gesture information includes:
  • the frame rate is determined according to the frequency at which the attitude information is received.
  • the server may determine the frame rate of the rendered gesture information according to the frequency of receiving the gesture information.
  • the terminal device adjusts the frequency of collecting attitude information according to its own decoding ability and/or refresh ability, and sends attitude information to the server according to the above frequency.
  • the server After the server receives the gesture information, it can determine the frequency of rendering according to the frequency of receiving the gesture information, so as to determine the frame rate of the rendered image information. Therefore, the server can adjust the frame rate of the rendered image information according to the frequency of receiving the gesture information, so that the image information rendered by the server can be adapted to the decoding capability and/or refresh capability of the terminal device, and can also be better adapted to the terminal device. current display needs.
  • the device may also include:
  • an encoding unit configured to encode the second image information to obtain third image information
  • the sending unit is specifically configured to send the third image information to the terminal device.
  • the server may encode the rendered image information, and then may send the rendered image information to the server.
  • the server encodes the rendered image information, which may be compression of the image information, thereby reducing the amount of transmitted image information, thereby saving transmission resources.
  • the encoded image information may also be encryption of the image information, so that the security and reliability of the image information can be improved.
  • a fourth aspect discloses an information processing apparatus.
  • the information processing apparatus may be a terminal device or a module (eg, a chip) in the terminal device, and the information processing apparatus may include:
  • a collection unit configured to stop collecting the first attitude information when it is detected that the terminal device is in an unused state, where the unused state includes that the screen of the terminal device is in an off-screen state and/or the terminal device is not worn by the user , the first gesture information is the user's gesture information.
  • the terminal device when the user does not use a terminal device such as a VR helmet, the terminal device still continuously displays images, which will cause waste of processing resources and energy. Therefore, when the display screen of the terminal device is in an off-screen state and/or when the user takes off the helmet, the user does not view the screen, and the terminal device can stop collecting the first posture information.
  • the terminal device When the terminal device does not collect the gesture information, the terminal device will not send the gesture information to the server, therefore, the server will not receive the gesture information, and at this time, the server will not render the image information. Therefore, when the user does not use the terminal device, the terminal device can stop collecting information, and the server can stop rendering image information, so that processing resources can be saved and energy waste can be avoided.
  • the device may also include:
  • a sending unit configured to send the second attitude information to the server when the second attitude information is collected
  • a receiving unit configured to receive the second image information from the server
  • the display unit is used for displaying the second image information.
  • the terminal device when the terminal device collects attitude information, it can immediately send the attitude information to the server, thereby ensuring that the frequency at which the terminal equipment collects attitude information is basically consistent with the frequency at which the server receives attitude information. Since the server immediately renders the first image information after receiving the attitude information, it can ensure that the frame rate of the second image information rendered by the server is consistent with the sending frequency of the terminal device, so that the terminal device can adjust the frequency of collecting attitude information. The frame rate of the image information rendered by the server can be adjusted accordingly, thereby ensuring that the terminal device can adapt to the frequency at which the server sends the second image information. In addition, when the terminal device receives the second image information, the second image information can be displayed, so that the integrity of the solution can be guaranteed.
  • the sending unit is specifically used for:
  • the decoding capability and/or refresh capability of the terminal device cannot be synchronized with the server, when the server renders image information at a fixed frame rate and sends the above image information to the terminal device, the decoding capability and/or Terminal devices with weak refresh capabilities may not be able to decode and/or display image information in time, which may cause frame loss and frame skipping. Since the server immediately renders the image information when it receives the gesture information, the terminal device can adjust the gesture information according to its own decoding capability and/or refresh capability, so that the frame rate of the image information rendered by the server can match the decoding capability of the terminal device. and/or refresh capability. After the above-mentioned adjustment of the attitude information by the terminal device, it can be ensured that the terminal device can decode and/or refresh the image information in time, thereby improving user experience.
  • the receiving unit is specifically used for:
  • the terminal device when the terminal device receives the third image information from the server, it needs to decode the third image information first.
  • the decoding process of the terminal device can correspond to the encoding process of the server, thereby ensuring the integrity of the information processing process.
  • the above-mentioned encoding and decoding process of image information can not only reduce the amount of transmitted information, save transmission resources, but also improve the reliability and stability of information.
  • the collection unit is further configured to collect the fourth posture information when it is detected that the terminal device is in a use state, and the use state includes that the screen is in a display state and/or the terminal device is in a state of being displayed. worn by the user.
  • the terminal device when the user resumes using the terminal device, the terminal device can continue to collect gesture information, so that the server can resume the rendering of the image information, and then the server can send the rendered image information to the terminal device. Further, when the terminal device receives the rendered image information, it can be displayed. When the user suspends using the terminal device to resume using the terminal device, the terminal device can be quickly resumed. In this way, not only can the user experience be guaranteed, but also processing resources can be saved, and energy waste can be avoided.
  • a fifth aspect discloses an information processing apparatus, which may be a server or a module (eg, a chip) within the server.
  • the information processing apparatus may include a processor, a memory, an input interface for receiving information from devices other than the information processing apparatus, and an output interface for sending information to the information processing apparatus The other device outputs information, and when the processor executes the computer program stored in the memory, the processor causes the processor to execute the information processing method disclosed in the first aspect or any implementation manner of the first aspect.
  • a sixth aspect discloses an information processing apparatus, where the information processing apparatus may be a terminal device or a module (eg, a chip) in the terminal device.
  • the information processing apparatus may include a processor, a memory, an input interface for receiving information from devices other than the information processing apparatus, and an output interface for sending information to the information processing apparatus The other device outputs information, and when the processor executes the computer program stored in the memory, the processor causes the processor to execute the information processing method disclosed in the second aspect or any embodiment of the second aspect.
  • a seventh aspect discloses an information processing system including the information processing apparatus of the fifth aspect and the information processing apparatus of the sixth aspect.
  • An eighth aspect discloses a computer-readable storage medium, where a computer program or computer instruction is stored thereon, and when the computer program or computer instruction is executed, the information processing method disclosed in the above aspects is implemented.
  • a ninth aspect discloses a chip including a processor for executing a program stored in a memory, and when the program is executed, causes the chip to execute the above method.
  • the memory is located off-chip.
  • FIG. 1 is a schematic diagram of a system architecture disclosed in an embodiment of the present invention.
  • FIG. 2 is a schematic diagram of a product form disclosed in an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of a gesture information interaction form disclosed in an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of the structure of a cloud VR solution disclosed in an embodiment of the present invention.
  • FIG. 5 is a schematic flowchart of an image information processing disclosed in an embodiment of the present invention.
  • FIG. 6 is a schematic flowchart of an information processing method disclosed in an embodiment of the present invention.
  • FIG. 7 is a schematic flowchart of another image information processing disclosed in an embodiment of the present invention.
  • FIG. 8 is a schematic flowchart of another information processing method disclosed in an embodiment of the present invention.
  • FIG. 9 is a schematic structural diagram of an information processing apparatus disclosed in an embodiment of the present invention.
  • FIG. 10 is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention.
  • FIG. 11 is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention.
  • the embodiments of the present application disclose an information processing method, device and system, which are used to reduce the time delay of displaying image information. Each of them will be described in detail below.
  • FIG. 1 is a schematic diagram of a system architecture disclosed in an embodiment of the present application.
  • the above system architecture may include a server and a terminal device, and the server may interact with the terminal device, that is, the server may send data to the terminal device, and may also receive data from the terminal device.
  • the embodiments of the present application are described by taking a server as an image and/or video processing device as an example, that is, a cloud virtual reality (cloud VR) device implementing the functions of an image and/or video processing device as an example for description.
  • cloud VR cloud virtual reality
  • a server is a remote server deployed in the cloud, with image processing capabilities, data computing capabilities, and coding capabilities, for example, it can perform rendering operations, logical operations, and the like.
  • the server can be an ultra-multi-core server, a computer deployed with a graphics processing unit (GPU) cluster, a large-scale distributed computer, a cluster computer with hardware resource pooling, and the like.
  • the server may also be a cloud device such as a deployment edge computing node, cloud computing node, etc., which is not limited here.
  • Terminal devices may include devices worn on the user's head, such as VR glasses, VR helmets, VR all-in-one machines, mobile phones + VR glasses (VR Glass), etc., and may also include devices worn on other parts of the user, such as devices at the hands, elbows, feet, knees, etc., such as gamepads, etc.
  • the terminal device can display the video image of the VR video to the user through the display. Wherein, the terminal device may not save the data of VR video and jump information locally, but save the relevant data in the video server device.
  • the video image of the VR video is displayed, and the user's data is sent to the video server device. posture information.
  • VR technology has gradually entered many fields such as entertainment, education, medical care, environmental protection, transportation, public health, etc., and has great application value and commercial potential.
  • a series of products such as VR helmets and VR eyes have been promoted in the market, and entertainment forms such as watching VR videos and playing VR games have been accepted by consumers.
  • FIG. 2 is a schematic diagram of a product form disclosed in an embodiment of the present application.
  • VR product forms are mainly divided into two types, one is an integrated helmet VR, and the other is a tethered VR.
  • the VR application of the integrated helmet VR is installed in the helmet, and the helmet VR can use the embedded processor for VR calculation and display.
  • the tethered VR is to connect the VR headset and a personal computer (PC) through a high definition multimedia interface (HDMI) cable.
  • the PC installs the VR application and runs it, and the headset displays the screen received through HDMI. .
  • HDMI high definition multimedia interface
  • FIG. 3 is a schematic diagram of a gesture information interaction form disclosed by an embodiment of the present application.
  • the interactive forms of attitude information can be divided into two types, namely three degrees of freedom (3 degrees of freedom, 3Dof) and six degrees of freedom (6 degrees of freedom, 6Dof).
  • 3Dof refers to the gesture interaction that takes the interaction center as the coordinate origin and rotates around the three coordinate axes of x, y, and z; 6Dof refers to the addition of positive and negative along the three coordinate axes of x, y, and z on the basis of 3Dof.
  • 3Dof only includes the interaction of "rotation” around the three axes of x, y, and z, and does not include the interaction of "translation” along the three axes of x, y, and z; while 6Dof includes the interaction of "rotation” and “translation” two interactions. It should be understood that in the evolution of future technologies, other gesture information may appear, which is not limited here.
  • the VR terminal device After the 3/6Dof information of the player is collected by the VR terminal device, it can drive the VR application to make each VR video frame and display it on the screen in the VR helmet.
  • ordinary VR headsets and home PCs cannot meet large-scale VR applications.
  • more realistic and smooth images in VR games or immersive and grand VR games require greater computing power for the VR system. Therefore, with the evolution of VR technology and the development of network technology, the VR form is developing from local VR forms such as tethered VR and all-in-one helmets to cloud VR.
  • cloud computing or edge computing can provide unlimited computing power, allowing VR applications to freely exert their immersive advantages.
  • the VR end-side device ie, the terminal device
  • the VR end-side device can collect the attitude information of 3Dof/6Dof, and send the attitude information of 3Dof/6Dof to the cloud device (server).
  • the server starts the VR application, it can receive the terminal-side information in real time, and can render the gesture information in real time.
  • the server can compress the rendered VR image information, and then can send the above compressed image information to the terminal device.
  • the terminal device After receiving the compressed information from the server, the terminal device can decode the compressed information to obtain image information, and then can display the image information. In this process, the cloud VR interaction is completed.
  • FIG. 4 is a schematic diagram of the structure of a cloud VR solution disclosed in an embodiment of the present application.
  • the server of cloud VR can provide the running, rendering and encoding push streaming of VR applications.
  • Servers and end devices eg, VR headsets
  • the terminal device can send gesture information (3/6Dof information) to the server through 5G
  • the server can send the rendered image information (ie VR video stream) to the terminal device through 5G.
  • the communication process between the server and the terminal device can be carried out through a cellular mobile communication network (for example, 5G), WiFi, or cloud VR technology.
  • the binocular resolution generally pushed is about 3K to 4K, and the frame rate is 60. ⁇ 90 frames per second (FPS) video stream, after the VR terminal device receives the video stream, it can be decoded and displayed.
  • the terminal device generally uploads the 3Dof/6Dof attitude information to the server at a fixed time interval (for example, 16ms or 11ms).
  • FIG. 5 is a schematic flowchart of an image information processing disclosed in an embodiment of the present application.
  • a cloud VR terminal device After a cloud VR terminal device establishes a VR service, it can periodically Send 3Dof/6Dof pose information to the server.
  • the server After receiving the gesture information from the terminal device, the server may periodically render the image information, and then may send the rendered image information to the terminal device.
  • the terminal device After receiving the image information from the server, the terminal device can display the image information.
  • the terminal device and the server do not cooperate, the terminal device periodically sends gesture information, and the server periodically renders image information.
  • the server triggers a vertical synchronization at the same time interval (for example, the time interval is 16ms or 11ms to trigger a vertical synchronization), and then the server can immediately render the pose information.
  • the time when the server receives the attitude information is often between two vertical synchronizations.
  • the vertical synchronization refers to a process in which horizontal scan lines are stacked in the vertical direction to form a complete picture, and the horizontal scan lines may be composed of a single pixel. After the terminal device scans a frame, a vertical synchronization signal can be generated, and a vertical synchronization signal can trigger an image rendering process.
  • the server After the server receives the attitude information from the terminal device, it still needs to wait for about 6-9 ms before rendering the attitude data, which will cause a large delay.
  • the lower the information display delay the better the user experience. Therefore, how to reduce the information display delay is an urgent problem to be solved.
  • FIG. 6 is a schematic flowchart of an information processing method disclosed in an embodiment of the present application.
  • the functions performed by the terminal device in the embodiments of the present application may also be performed by a module (eg, a chip) in the terminal device, and the function performed by the server may also be performed by a module (eg, a chip) in the server.
  • the information processing method may include the following steps.
  • the terminal device collects attitude information.
  • the terminal device can collect gesture information.
  • the gesture information may refer to the user's body gesture information.
  • the gesture information can represent the rotation and movement of the user's body, for example, the gesture information of 3Dof and 6Dof.
  • the terminal device can obtain 3Dof or 6Dof attitude information through local sensors.
  • the terminal device can perform motion capture (motion capture) through the sensors on the VR headset and the handle, so as to collect gesture information.
  • motion capture can be a technology of recording and processing human or animal movements. It should be understood that the manner of collecting attitude information is illustrated here as an example, and is not intended to be limiting.
  • the terminal device can trigger the collection of attitude information in different situations.
  • the terminal device when the terminal device acquires the start-up operation, it can start to collect attitude information. For example, the user can turn on the terminal device by touching the screen or pressing a button, and in response to the above start-up operation, the terminal device can start collecting gesture information.
  • the terminal device may start to collect gesture information. After the user determined by the terminal device starts the VR application of the terminal device, the terminal device may start to collect gesture information. After the user determined by the terminal device closes the VR application of the terminal device, the terminal device may stop collecting gesture information.
  • the terminal device detects that the user starts a VR application (eg, VR game, VR video, etc.)
  • the terminal device may establish a connection with the server in response to the start-up operation. After the terminal device establishes a connection with the server, the terminal device can start to collect attitude information.
  • the terminal device detects that the user terminates the VR application, the terminal device may disconnect from the server in response to the termination operation. After the terminal device is disconnected from the server, the terminal device may stop collecting attitude information.
  • attitude information when the terminal device is not in use, the collection of attitude information (first attitude information) may be stopped; when the terminal device is in use, attitude information (fourth attitude information) may be collected.
  • the use state may include that the screen is in a display state and/or the terminal device has been worn by the user, and the unused state may include that the screen of the terminal device is in a screen-off state and/or the terminal device is not worn by the user.
  • the terminal device can confirm whether the screen is in the off-screen state through the display screen (for example, the user presses the pause or screen-off control key) to determine whether the terminal device is in use, and can also detect whether the user is wearing the current terminal device through the sensor (for example, Wear a VR helmet) to determine whether the terminal device is in use.
  • the first attitude information may be attitude information when the terminal device suspends collecting attitude information
  • the fourth attitude information may be attitude information when the terminal device resumes collecting attitude information.
  • the terminal device can stop collecting attitude information. That is, when it is detected that the user changes from using the terminal device to not using the terminal device, the terminal device can stop collecting attitude information. When the collection of attitude information is stopped, the terminal device stops sending attitude information to the server. At this time, since the server will not receive the gesture information, and thus will not trigger the rendering of the first image information, when the terminal device stops collecting the gesture information, the server may stop rendering.
  • the terminal device can continue to collect attitude information. That is, when it is detected that the user never uses the terminal device to use the terminal device (when the screen is turned off to the display state, and/or the user continues to wear the terminal device), the terminal device can resume collecting attitude information. That is, the terminal device changes from a state of stopping collecting attitude information to a state of starting to collect. After the terminal device collects the attitude information, it can send the attitude information to the server. At this point, the server can receive the gesture information, and can start rendering the first image information.
  • the terminal device when the user does not use the terminal device, the terminal device continues to collect gesture information, and the server continues to perform image rendering according to the gesture information, which will cause waste of processing resources and energy. Therefore, when the terminal device detects that the user does not use the terminal device temporarily, it can stop collecting gesture information, so that the server can stop image rendering, thereby saving processing resources and energy.
  • the terminal device when the terminal device receives the second image information from the server, it can continue to collect gesture information.
  • the second image information is virtual reality VR image information.
  • the terminal device may be triggered to continue to perform motion capture and collect gesture information.
  • the terminal device Before collecting attitude information, the terminal device can start the cloud VR service. That is, users can start cloud VR applications through the display screen, such as cloud VR games, cloud VR live broadcasts, and cloud VR videos.
  • the terminal device may send the first request to the server in response to the user's start-up operation.
  • the server may receive the first request from the terminal device.
  • the first request may be request information for the terminal device to request the server to initiate a VR service, and the first request may include information about the VR service corresponding to the VR application started by the terminal device, and may also include connection mode information between the terminal device and the server.
  • the server may negotiate with the server to start the VR service according to the first request. That is, it can be understood that the server can determine the connection mode with the current terminal device and the VR service (eg, VR scene information) to be used through the first request. That is, the server may determine the VR service currently used by the terminal device and the corresponding image information according to the first request, and may also determine the connection mode between the terminal device and the server according to the first request.
  • the server can determine the connection mode with the current terminal device and the VR service (eg, VR scene information) to be used through the first request. That is, the server may determine the VR service currently used by the terminal device and the corresponding image information according to the first request, and may also determine the connection mode between the terminal device and the server according to the first request.
  • the server may negotiate with the server to start the VR service according to the first request. That is, it can be understood that the server can determine the connection mode with the current terminal device and the VR service (eg, VR scene information) to be used through the first request. That is, the server may
  • the terminal device can start various local functional components.
  • the functional components may include components for receiving image information (eg, initializing streaming), decoding components (eg, decoding function), display components (eg, VR image texture refresh function), and collecting attitude information (eg, motion capture function) Components and screen components (eg, controlling the screen switch function), etc.
  • the terminal device sends gesture information to the server.
  • the terminal device When the terminal device collects the gesture information (second gesture information), it can send the second gesture information to the server.
  • the server may receive gesture information from the terminal device.
  • the terminal device can adjust the second attitude information according to the decoding capability and/or refresh capability of the terminal device to obtain the third attitude information, and then can send the third attitude information to the server.
  • the server may receive the third gesture information from the terminal device.
  • the decoding capability may be the fastest decoding speed of the terminal device
  • the refresh capability may be the maximum frame rate displayed by the terminal device. That is, the terminal device can determine the maximum frequency of collecting attitude information according to its own decoding ability and/or refresh ability, so as to determine the third attitude information.
  • the decoding capability of the terminal device is 70 (frames per second, FPS), and the refresh capability is 60 FPS.
  • the terminal device can first determine its own decoding capability and/or refresh capability, and then determine a frequency within the range of the decoding capability and/or refresh capability (that is, the maximum frequency range for collecting the third attitude information), and then immediately send the third attitude. information. That is, it can be understood that the frequency at which the terminal device collects the third attitude information needs to be less than or equal to the range of the decoding capability and/or the refresh capability of the terminal device. For example, when the decoding capability of the terminal device is 70FPS, the terminal device can determine that the frequency of collecting the third attitude information is at most 70 sets of 3Dof or 6Dof/second; when the refresh capability of the terminal device is 60FPS, the terminal device can determine to collect the third attitude information.
  • the maximum frequency of information is 60 sets of 3Dof or 6Dof information/second; when the decoding capability of the terminal device is 70FPS and the refresh capability is 60FPS, the terminal device can determine that the frequency of collecting the third attitude information is at most 60 sets of 3Dof or 6Dof information/ second.
  • the terminal device may also adjust the collection frequency of the third attitude information according to other factors.
  • the terminal device determines the maximum frequency of sending attitude information by itself, it can determine the frequency of sending attitude information within the maximum frequency range according to requirements. That is, when the currently displayed image information is relatively stuck, the frequency of collecting attitude information can be increased; when the scene does not need to display too smooth image information, the frequency of collecting attitude information can be reduced.
  • the currently collected attitude information is 40 sets of 6Dof information per second and the maximum collection frequency of attitude information is 60 sets of 6Dof information per second.
  • the device can increase the frequency of collecting attitude information to 60 sets of 6Dof information per second. It should be understood that the above is only an example, and does not constitute a limitation.
  • the terminal device adjusts the collection frequency according to other factors within the maximum frequency range of collecting attitude information, it can better adapt to the display requirements of the current terminal device, thereby improving user experience.
  • the terminal device After the terminal device collects the attitude information, it should immediately send the collected attitude information, which can ensure the consistency of the frequency of collecting attitude information and the frequency of sending attitude information.
  • the "rhythm" of server rendering, terminal device acquisition and display, and interaction is determined by the terminal device's acquisition "rhythm", so that the terminal device can adjust the frequency of collecting attitude information to adapt to its own decoding capabilities and/or refresh ability.
  • the server When receiving the gesture information from the terminal device, the server renders the first image information according to the gesture information to obtain the second image information.
  • the server may render the first image information according to the gesture information to obtain the second image information.
  • the first image information may be virtual reality VR image information, that is, the first image information may be image information of a cloud VR application used by the current user.
  • the server may determine the image information to be rendered (ie, the first image information) according to the gesture information, and then render the first image information to obtain the second image information.
  • FIG. 7 is a schematic flowchart of another image information processing disclosed in an embodiment of the present application.
  • the server can be triggered to perform vertical synchronization immediately.
  • the server can immediately start rendering the first image information to obtain the second image information.
  • the server can render image information through its own central processing unit (CPU) and graphics processing unit (graphics processing unit, GPU).
  • the second image information may be VR video frame data. That is, it can be understood that the terminal device can passively trigger the rendering of the first image information by receiving the gesture information.
  • the server when the server triggers vertical synchronization by receiving the attitude information, the server can immediately perform image rendering on the first image information according to the received attitude information, thereby reducing the waiting time until the vertical synchronization is periodically triggered before image rendering. In turn, the processing delay can be reduced. Further, when the time delay in the image rendering process is reduced, the time delay for the user to receive the second image information is also reduced, so that the time delay for the terminal device to display the second image information can be reduced, so as to improve the smoothness of the image display, and further. It can improve the user experience.
  • the server passively triggers the rendering process by receiving attitude information, and the terminal device can reduce the image display delay of 6-9 ms.
  • the server may continue to wait for receiving the gesture information from the terminal device.
  • the server may determine the frame rate according to the attitude information, and then render the first image information according to the frame rate to obtain the second image information.
  • the server can determine the frame rate based on how often the gesture information is received. For example, when the server receives a set of 3Dof or 6Dof attitude information, it can immediately render a frame of first image data to obtain a frame of second image data. That is, it can be understood that when the frequency of the gesture information received by the server may be inconsistent, the frequency of the received gesture information can determine the frame rate of the second image information.
  • the frequency of receiving the attitude information is faster, the frequency of rendering the first image information is also faster, and thus the frame rate of obtaining the second image information is higher. Conversely, when the frequency of attitude information received by the server is slower, the frequency of rendering is slower, and the frame rate of obtaining the second image information is smaller.
  • the server when the server periodically performs image rendering, the server performs rendering at a fixed frame rate, but different terminal devices have different decoding capabilities and/or refresh capabilities. Therefore, the server performs fixed frame rate rendering on the first image information, It may cause that the terminal device with weak decoding capability cannot decode in time, and/or the terminal device with weak refresh capability cannot display in time. For example, when the display capability of the VR headset is only 60FPS, when the second image data of 90FPS specification is received from the server, it may cause frame loss and frame skipping during the display process of the terminal device, or some decoding capabilities and/or Terminal devices with low refresh capability cannot access the cloud VR server of fixed specifications.
  • the terminal device determines the frequency of sending the attitude information, and when the server receives the attitude information, it can immediately render the first image information to obtain the second image information, so that the server can change the rendered image information from a fixed period to that of the terminal equipment. It is driven by the frequency of collecting attitude information, so that the rendering speed of the server can be adapted to the decoding capability and/or refresh capability of the terminal device.
  • a power-saving state such as silent, screen-off, etc.
  • the server can reduce the trigger rendering, so that the server and the terminal device can enter the power-saving state at the same time. In order to avoid the energy consumption caused by the server rendering, encoding and sending image information all the time.
  • the server sends the second image information to the terminal device.
  • the server After the server obtains the second image information, it can send the second image information to the terminal device.
  • the terminal device may receive the second image information from the server.
  • the server After the server obtains the second image information, it can immediately send the second image information to the terminal device, which can ensure the consistency of the terminal device's collection of attitude information and display image information, and the "rhythm" of the server rendering the image information.
  • the second image information may be encoded by an encoder to obtain third image information (eg, VR video frame information).
  • the server may send the third image information to the terminal device.
  • the terminal device can receive the third image information from the server, and then can decode the third image information through the decoder to obtain the second image information.
  • the server After the server sends the second image information to the terminal device, it may continue to wait for the gesture information, and render the first image information.
  • the terminal device displays the second image information.
  • the second image information can be displayed.
  • the second image information may be a VR picture, and the terminal device may display the second image information through the display screen.
  • the terminal device may further process the pixels of the second image information.
  • pixel processing may include asynchronous warping and the like.
  • the terminal device may continue to wait for receiving the second image information from the server.
  • the terminal device may also stop displaying the second image information.
  • the unused state may include that the screen of the terminal device is in an off-screen state and/or the terminal device is not worn by the user.
  • the terminal device may continue to display the second image information. That is, it can be understood that the terminal device can detect whether it is in an unused state.
  • the usage state includes that the screen is in a display state and/or the terminal device has been re-worn by the user.
  • the terminal device when the user does not use the terminal device, the terminal device continues to display the second image information, which will cause waste of processing resources and energy. Therefore, when the terminal device detects that the user temporarily does not use the terminal device, it can stop displaying the second image information, thereby saving energy and processing resources.
  • the terminal device may send a second request to the server.
  • the user can end the current cloud VR application by touching the screen or pressing a button.
  • the server may receive the second request from the terminal device, and then may end the current connection according to the second request, and end the task of the current cloud VR service corresponding to each functional component.
  • the terminal device can end the tasks of each functional component of the VR application.
  • the terminal device After collecting the attitude information, the terminal device sends the attitude information to the server. After receiving the gesture information, the server renders the first image information to obtain the second image information, and sends the second image information to the terminal device. After receiving the image information from the server, the terminal device displays the second image information, and can continue to collect image information.
  • the above process forms a VR service cycle.
  • the current VR service cycle can be suspended; when the user resumes using the terminal device, the current VR service cycle can be resumed.
  • the VR service cycle can be started, and when the VR service is ended, the VR service cycle can be ended.
  • FIG. 8 is a schematic flowchart of another information processing method disclosed by an embodiment of the present application.
  • the functions performed by the terminal device in the embodiments of the present application may also be performed by a module (eg, a chip) in the terminal device, and the function performed by the server may also be performed by a module (eg, a chip) in the server.
  • the information processing method may include the following steps.
  • the terminal device starts the VR application.
  • the terminal device sends a first request to the server.
  • the server starts the VR service.
  • step 801-step 803 may refer to the relevant description of step 601 .
  • the terminal device collects attitude information.
  • step 804 reference may be made to the relevant description of step 601, and details are not repeated here.
  • the terminal device sends gesture information to the server.
  • step 805 reference may be made to the relevant description of step 602, and details are not repeated here.
  • the server renders the first image information according to the gesture information to obtain second image information.
  • step 806 reference may be made to the relevant description of step 603, and details are not repeated here.
  • the server encodes the second image information to obtain third image information.
  • the server sends the third image information to the terminal device.
  • the terminal device decodes the third image information to obtain the second image information.
  • steps 807-809 reference may be made to the relevant description of step 604, and details are not repeated here.
  • the terminal device displays the second image information.
  • step 810 reference may be made to the relevant description of step 605.
  • a service loop may be entered, that is, steps 804 to 810 may be repeatedly performed, and when the terminal device obtains and displays the second image information, it may continue to collect gesture information.
  • the server encodes the second image information and obtains the third image information, it can continue to wait for receiving the gesture information from the terminal device.
  • step 811 and step 812 may refer to the relevant description of step 601 .
  • step 811 is an action performed when a screen-on-screen event occurs on the terminal device or the user takes off the VR terminal device
  • step 812 is an action performed when the terminal device changes from a screen-on-screen to a screen display or when the user wears the VR terminal device.
  • Step 811 may suspend the above-mentioned process of the VR service loop; and step 812 may resume the above-mentioned process of the VR service loop.
  • the terminal device ends the VR application.
  • the terminal device sends a second request to the server.
  • the server terminates the VR service.
  • step 813-step 815 may refer to the relevant description of step 605 .
  • FIG. 6 and FIG. 8 is a general process in an interactive process, and is not limited to cloud VR, it can be applied to implement interactive scenarios, for example, cloud extended reality (XR), cloud VR games etc.
  • the cloud XR may include cloud VR, cloud augmented reality (AR), cloud mixed reality (MR), cloud holography reality (HR), and the like. This is not a limitation.
  • FIG. 9 is a schematic structural diagram of an information processing apparatus disclosed in an embodiment of the present invention.
  • the information processing apparatus may include:
  • the rendering unit 901 is configured to render the first image information according to the attitude information when receiving the attitude information from the terminal device, and obtain the second image information, the attitude information is the attitude information of the user, the first image information
  • the image information is virtual reality VR image information
  • the sending unit 902 is configured to send the second image information to the terminal device.
  • the rendering unit 901 is specifically used for:
  • the first image information is rendered according to the frame rate to obtain second image information.
  • the rendering unit 901 determining the frame rate according to the gesture information includes:
  • the frame rate is determined according to the frequency at which the attitude information is received.
  • the device may also include:
  • an encoding unit 903 configured to encode the second image information to obtain third image information
  • the sending unit 902 is specifically configured to send the third image information to the terminal device.
  • rendering unit 901 sending unit 902 and encoding unit 903 can be obtained directly by referring to the relevant descriptions of the servers in the method embodiments shown in FIG. 6 and FIG. 8 , and details are not repeated here.
  • FIG. 10 is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention.
  • the information processing apparatus may include:
  • a collection unit 1001 configured to stop collecting the first attitude information when it is detected that the terminal device is in an unused state, where the unused state includes that the screen of the terminal device is in an off-screen state and/or the terminal device is not used by a user When wearing, the first posture information is the posture information of the user.
  • the device may also include:
  • a receiving unit 1003, configured to receive the second image information from the server
  • the display unit 1004 is configured to display the second image information.
  • the sending unit 1002 is specifically configured to:
  • the receiving unit 1003 is specifically configured to:
  • the collection unit 1001 is further configured to collect the fourth posture information when it is detected that the terminal device is in a use state, and the use state includes that the screen is in a display state and/or the terminal device is in a display state.
  • the device has been worn by the user.
  • FIG. 11 is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention.
  • the information processing apparatus may include a processor 1101 , a memory 1102 , an input interface 1103 , an output interface 1104 and a bus 1105 .
  • the memory 1102 may exist independently, and may be connected to the processor 1101 through the bus 1105 .
  • the memory 1102 may also be integrated with the processor 1101 .
  • the bus 1105 is used to realize the connection between these components.
  • the information processing apparatus may be a server or a module (eg, a chip) in the server, and when the computer program instructions stored in the memory 1102 are executed, the processor 1101 is configured to control the sending unit 902 to execute the above-mentioned embodiments
  • the processor 1101 is further configured to perform the operations performed by the rendering unit 901 and the encoding unit 903 in the foregoing embodiments
  • the output interface 1104 is configured to perform the operations performed by the sending unit 902 in the foregoing embodiments.
  • the foregoing server or modules within the server may also be used to execute various methods executed by the server in the foregoing method embodiments in FIG. 6 and FIG. 8 , which will not be described again.
  • the information processing apparatus may be a terminal device or a module (eg, a chip) in the terminal device.
  • the processor 1101 is used to control the receiving unit 1003 and send the The unit 1002 performs the operations performed in the above-mentioned embodiments
  • the processor 901 is further configured to perform the operations performed by the acquisition unit 1001 and the display unit 1004 in the above-mentioned embodiments
  • the input interface 1103 is used to perform the operations performed by the receiving unit 1003 in the above-mentioned embodiments.
  • the output interface 1104 is used to perform the operations performed by the sending unit 1002 in the above embodiments.
  • the foregoing terminal device or modules in the terminal device may also be used to execute various methods performed by the terminal device in the foregoing method embodiments in FIG. 6 and FIG. 8 , which will not be described again.
  • FIG. 12 is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention.
  • the information processing apparatus may include an input interface 1201 , a logic circuit 1202 and an output interface 1203 .
  • the input interface 1201 and the output interface 1203 are connected through the logic circuit 1202 .
  • the input interface 1201 is used for receiving information from other devices, and the output interface 1203 is used for outputting, scheduling or sending information to other devices.
  • the logic circuit 1202 is configured to perform operations other than the operations of the input interface 1201 and the output interface 1203, for example, to implement the functions implemented by the processor 1101 in the above-mentioned embodiment.
  • the information processing apparatus may be a terminal device or a module in the terminal device, or may be a server or a module in the server.
  • the more detailed description about the input interface 1201 , the logic circuit 1202 and the output interface 1203 can be obtained directly by referring to the relevant description of the terminal device or the server in the above method embodiments, which will not be repeated here.
  • the embodiment of the present invention also discloses a computer-readable storage medium, on which an instruction is stored, and when the instruction is executed, the method in the foregoing method embodiment is performed.
  • the embodiment of the present invention also discloses a computer program product including an instruction, when the instruction is executed, the method in the foregoing method embodiment is performed.
  • An embodiment of the present invention also discloses an information processing system, the information processing system includes a terminal device and a server, and for a specific description, reference may be made to the information processing methods shown in FIG. 6 and FIG. 8 .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

Embodiments of the present invention disclose an information processing method, apparatus and system, the method comprising: upon receiving posture information from a terminal device, rendering first image information according to the posture information so as to obtain second image information, the posture information being posture information of a user, the first image information being virtual reality (VR) image information; and sending the second image information to the terminal device. The embodiments of the present application can reduce the delay of information display.

Description

一种信息处理方法、装置及***An information processing method, device and system
本申请要求于2021年04月14日提交中国专利局、申请号为202110402796.3、申请名称为“一种信息处理方法、装置及***”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims the priority of the Chinese patent application with the application number 202110402796.3 and the application title "An Information Processing Method, Device and System" filed with the China Patent Office on April 14, 2021, the entire contents of which are incorporated herein by reference Applying.
技术领域technical field
本发明实施例涉及虚拟现实技术领域，尤其涉及一种信息处理方法、装置及***。Embodiments of the present invention relate to the technical field of virtual reality, and in particular, to an information processing method, device, and system.
背景技术Background technique
虚拟现实(virtual reality，VR)技术是一种可以创建和体验虚拟世界的计算机仿真***，它利用计算机生成一种模拟环境，使用户沉浸到该环境中。即VR技术是一种多源信息融合的、交互式的三维动态景和实体行为的***仿真。Virtual reality (VR) technology is a computer simulation system that can create and experience virtual worlds. It uses computers to generate a simulated environment and immerse users in the environment. That is, VR technology is a multi-source information fusion, interactive three-dimensional dynamic scene and system simulation of entity behavior.
与本地VR技术相比，云(cloud)VR技术由云端设备(服务器侧)对图像进行渲染，由客户端(VR的终端设备，如VR头盔等)对图像进行显示。云端设备和客户端需要进行交互，即云端设备可以接收来自客户端采集的数据，并进行渲染，以及客户端可以接收来自云端设备的数据，并进行VR图像显示。然而，在目前的云VR交互技术中，云端设备接收到来自客户端的姿态信息之后，无法及时地根据姿态信息对图像信息进行渲染，以致终端设备显示图像信息的时延较大。Compared with the local VR technology, the cloud VR technology renders the image by the cloud device (server side), and displays the image by the client (VR terminal device, such as VR helmet, etc.). The cloud device and the client need to interact, that is, the cloud device can receive the data collected from the client and render it, and the client can receive the data from the cloud device and display VR images. However, in the current cloud VR interaction technology, after the cloud device receives the gesture information from the client, it cannot render the image information according to the gesture information in time, so that the terminal device has a large delay in displaying the image information.
发明内容SUMMARY OF THE INVENTION
本发明实施例公开了一种信息处理方法、装置及***，用于降低图像信息显示的时延。The embodiments of the present invention disclose an information processing method, device and system, which are used for reducing the time delay of image information display.
第一方面公开一种信息处理方法，该信息处理方法可以应用于服务器，也可以应用于服务器中的模块(例如，芯片)，以下以服务器为例进行说明。该信息处理方法可以包括：当接收到来自终端设备的姿态信息时，根据所述姿态信息对第一图像信息进行渲染，得到第二图像信息，所述姿态信息是用户的姿态信息，所述第一图像信息为虚拟现实VR图像信息；向所述终端设备发送所述第二图像信息。A first aspect discloses an information processing method. The information processing method can be applied to a server or a module (eg, a chip) in the server. The server is used as an example for description below. The information processing method may include: when receiving attitude information from a terminal device, rendering first image information according to the attitude information to obtain second image information, where the attitude information is the user's attitude information, and the first image information is obtained. The first image information is virtual reality VR image information; the second image information is sent to the terminal device.
本申请实施例中，在现有的技术中，由于服务器是在等待相同的时间间隔后渲染一次图像信息，因此，当服务器接收到姿态信息时，通常需要等待一段时间，才可以开始对图像信息进行渲染，导致显示图像信息会产生一定的时延。本申请实施例中，当服务器接收到姿态信息时，可以立即对图像信息进行渲染，从而可以减少服务器处理过程中的等待时延，进而可以降低终端设备显示图像信息的时延。此外，当降低了终端设备显示图像信息的时延时，可以使得终端设备及时地刷新接收到的图像信息，以使得用户观看的图像画面更加流畅，从而可以提高用户体验。In the embodiment of the present application, in the prior art, since the server renders the image information once after waiting for the same time interval, when the server receives the gesture information, it usually needs to wait for a period of time before it can start to process the image information. Rendering will result in a certain delay in displaying image information. In this embodiment of the present application, when the server receives the gesture information, the image information can be rendered immediately, thereby reducing the waiting delay in the server processing process, and further reducing the delay in displaying the image information by the terminal device. In addition, when the time delay for the terminal device to display the image information is reduced, the terminal device can refresh the received image information in time, so that the images viewed by the user are smoother, thereby improving the user experience.
作为一种可能的实施方式，所述根据所述姿态信息对第一图像信息进行渲染，得到第二图像信息包括：根据所述姿态信息确定帧率(frame rate，FR)；根据所述帧率对所述第一图像信息进行渲染，得到第二图像信息。As a possible implementation manner, the rendering of the first image information according to the attitude information to obtain the second image information includes: determining a frame rate (FR) according to the attitude information; The first image information is rendered to obtain second image information.
本申请实施例中，当前，服务器按照固定的帧率对图像信息进行渲染。然而，不同的终端设备的解码能力和/或刷新能力往往不同。有的终端设备的解码能力和/或刷新能力相对较高，当接收到服务器的图像信息时，能够及时地进行解码和显示；而有的终端设备的解码能力和/ 或刷新能力相对较低，当接收到服务器的图像信息时，来不及进行解码和刷新的时候，从而可能会出现丢帧和跳帧等问题。在本申请实施例中，服务器可以根据接收到的姿态信息确定渲染的图像信息对应的帧率，从而可以适应不同终端设备的解码能力和/或刷新能力，进而可以提高服务器对终端设备的适应能力，也可以保证终端设备显示图片信息的可靠性。进一步地，当终端设备显示图像信息的可靠性提高时，可以提高用户看到的图像画面的正确性，从而可以提高用户体验。In the embodiment of the present application, currently, the server renders the image information according to a fixed frame rate. However, the decoding capabilities and/or refresh capabilities of different terminal devices are often different. Some terminal devices have relatively high decoding capabilities and/or refresh capabilities, and can decode and display them in a timely manner when receiving image information from the server; while some terminal devices have relatively low decoding capabilities and/or refresh capabilities, When the image information from the server is received, it is too late to decode and refresh, which may cause problems such as frame loss and frame skipping. In this embodiment of the present application, the server can determine the frame rate corresponding to the rendered image information according to the received attitude information, so that it can adapt to the decoding capability and/or refresh capability of different terminal devices, thereby improving the adaptability of the server to the terminal device. , it can also ensure the reliability of the picture information displayed by the terminal device. Further, when the reliability of displaying the image information by the terminal device is improved, the correctness of the image picture viewed by the user can be improved, thereby improving the user experience.
作为一种可能的实施方式，所述根据所述姿态信息确定帧率包括：根据接收到所述姿态信息的频率确定所述帧率。As a possible implementation manner, the determining the frame rate according to the gesture information includes: determining the frame rate according to the frequency of receiving the gesture information.
本申请实施例中，服务器可以根据接收到姿态信息的频率确定渲染的姿态信息的帧率。终端设备根据自身的解码能力和/或刷新能力调整采集姿态信息的频率，并根据上述频率向服务器发送姿态信息。服务器接收到姿态信息之后，可以根据接收姿态信息的频率确定渲染的频率，从而可以确定渲染出的图像信息的帧率。因此，服务器可以根据接收到姿态信息的频率对应调整渲染图片信息的帧率，从而可以使得服务器渲染出的图像信息可以适应终端设备的解码能力和/或刷新能力，也可以更好地适应终端设备当前的显示需要。In this embodiment of the present application, the server may determine the frame rate of the rendered gesture information according to the frequency of receiving the gesture information. The terminal device adjusts the frequency of collecting attitude information according to its own decoding ability and/or refresh ability, and sends attitude information to the server according to the above frequency. After the server receives the gesture information, it can determine the frequency of rendering according to the frequency of receiving the gesture information, so as to determine the frame rate of the rendered image information. Therefore, the server can adjust the frame rate of the rendered image information according to the frequency of receiving the gesture information, so that the image information rendered by the server can be adapted to the decoding capability and/or refresh capability of the terminal device, and can also be better adapted to the terminal device. current display needs.
作为一种可能的实施方式，该方法还可以包括：对所述第二图像信息进行编码，得到第三图像信息；所述向所述终端设备发送所述第二图像信息包括：向所述终端设备发送所述第三图像信息。As a possible implementation manner, the method may further include: encoding the second image information to obtain third image information; the sending the second image information to the terminal device includes: sending the second image information to the terminal device The device sends the third image information.
本申请实施例中，服务器对图像信息进行渲染之后，可以对渲染之后的图像信息进行编码，之后可以向服务器发送渲染后的图像信息。服务器对渲染后的图像信息进行编码，可以是对图像信息的压缩，从而可以减小传输图像信息的量，进而可以节约传输资源。此外，编码图像信息也可以是对图像信息的加密，从而可以提高图像信息的安全性和可靠性。In this embodiment of the present application, after rendering the image information, the server may encode the rendered image information, and then may send the rendered image information to the server. The server encodes the rendered image information, which may be compression of the image information, thereby reducing the amount of transmitted image information, thereby saving transmission resources. In addition, the encoded image information may also be encryption of the image information, so that the security and reliability of the image information can be improved.
第二方面公开一种信息处理方法，该信息处理方法可以应用于终端设备，也可以应用于终端设备中的模块(例如，芯片)，以下以终端设备为例进行说明。该信息处理方法可以包括：当检测到终端设备处于未使用状态时，停止采集第一姿态信息，所述未使用状态包括所述终端设备的屏幕处于息屏状态和/或所述终端设备未被用户佩戴，所述第一姿态信息是用户的姿态信息。A second aspect discloses an information processing method. The information processing method can be applied to a terminal device, and can also be applied to a module (eg, a chip) in the terminal device. The terminal device is used as an example for description below. The information processing method may include: when it is detected that the terminal device is in an unused state, stopping collecting the first attitude information, and the unused state includes that the screen of the terminal device is in an off-screen state and/or the terminal device is not in use. It is worn by the user, and the first posture information is the posture information of the user.
本申请实施例中，当用户不使用VR头盔等终端设备时，终端设备依然不停地显示图像会造成处理资源和能源的浪费。因此，当终端设备的显示屏处于息屏状态时和/或用户摘下头盔时，用户不会查看屏幕，终端设备可以停止采集第一姿态信息。当终端设备不采集姿态信息时，终端设备不会向服务器发送姿态信息，因此，服务器不会接收到姿态信息，此时，服务器不会对图像信息进行渲染。因此，当用户不使用终端设备的时候，终端设备可以停止采集信息，以及服务器可以停止渲染图像信息，从而可以节约处理资源，以及可以避免能源的浪费。In the embodiment of the present application, when the user does not use a terminal device such as a VR helmet, the terminal device still continuously displays images, which will cause waste of processing resources and energy. Therefore, when the display screen of the terminal device is in an off-screen state and/or when the user takes off the helmet, the user does not view the screen, and the terminal device can stop collecting the first posture information. When the terminal device does not collect the gesture information, the terminal device will not send the gesture information to the server, therefore, the server will not receive the gesture information, and at this time, the server will not render the image information. Therefore, when the user does not use the terminal device, the terminal device can stop collecting information, and the server can stop rendering image information, so that processing resources can be saved and energy waste can be avoided.
作为一种可能的实施方式，该方法还可以包括：当采集到第二姿态信息时，向所述服务器发送所述第二姿态信息；接收来自所述服务器的所述第二图像信息；显示所述第二图像信息。As a possible implementation manner, the method may further include: when the second gesture information is collected, sending the second gesture information to the server; receiving the second image information from the server; displaying the second gesture information the second image information.
本申请实施例中，当终端设备采集姿态信息时候，可以立即向服务器发送姿态信息，从而可以保证终端设备的采集姿态信息的频率与服务器接收到姿态信息的频率基本一致。由于服务器接收到姿态信息便立即对第一图像信息进行渲染，从而可以保证服务器渲染得到的第二图像信息的帧率与终端设备的发送频率对应一致，从而可以使得终端设备调整采集姿态信 息的频率可以对应调整服务器渲染的图像信息的帧率，进而可以保证终端设备调能够适应服务器发送第二图像信息的频率。此外，当终端设备接收到第二图像信息时，可以显示第二图像信息，从而可以保证方案的完整性。In this embodiment of the present application, when the terminal device collects attitude information, it can immediately send the attitude information to the server, thereby ensuring that the frequency at which the terminal equipment collects attitude information is basically consistent with the frequency at which the server receives attitude information. Since the server immediately renders the first image information after receiving the attitude information, it can ensure that the frame rate of the second image information rendered by the server is consistent with the sending frequency of the terminal device, so that the terminal device can adjust the frequency of collecting attitude information. The frame rate of the image information rendered by the server can be adjusted accordingly, thereby ensuring that the terminal device can adapt to the frequency at which the server sends the second image information. In addition, when the terminal device receives the second image information, the second image information can be displayed, so that the integrity of the solution can be guaranteed.
作为一种可能的实施方式，所述向所述服务器发送所述第二姿态信息包括：根据终端设备的解码能力和/或刷新能力调整所述第二姿态信息，得到第三姿态信息，所述解码能力为所述终端设备解码的最快速度，所述刷新能力为所述终端设备显示的最大帧率；向所述服务器发送所述第三姿态信息。As a possible implementation manner, the sending the second attitude information to the server includes: adjusting the second attitude information according to the decoding capability and/or refresh capability of the terminal device to obtain third attitude information, and the The decoding capability is the fastest decoding speed of the terminal device, and the refresh capability is the maximum frame rate displayed by the terminal device; the third gesture information is sent to the server.
本申请实施例中，由于终端设备的解码能力和/或刷新能力不能够与服务器同步，因此，当服务器按照固定帧率渲染图像信息，并向终端设备发送上述图像信息时，解码能力和/或刷新能力较弱的终端设备可能来不及解码和/或显示图像信息，从而可能造成丢帧和跳帧等问题。由于服务器接收到姿态信息时，便立即渲染图像信息，因此，终端设备可以根据自身的解码能力和/或刷新能力调整姿态信息，使得服务器渲染后的图像信息的帧率能够与终端设备的解码能力和/或刷新能力相适应。经过上述终端设备对姿态信息的调整，可以保证终端设备能够及时地解码和/或刷新图像信息，从而可以提高用户体验。In this embodiment of the present application, since the decoding capability and/or refresh capability of the terminal device cannot be synchronized with the server, when the server renders image information at a fixed frame rate and sends the above image information to the terminal device, the decoding capability and/or Terminal devices with weak refresh capabilities may not be able to decode and/or display image information in time, which may cause frame loss and frame skipping. Since the server immediately renders the image information when it receives the gesture information, the terminal device can adjust the gesture information according to its own decoding capability and/or refresh capability, so that the frame rate of the image information rendered by the server can match the decoding capability of the terminal device. and/or refresh capability. After the above-mentioned adjustment of the attitude information by the terminal device, it can be ensured that the terminal device can decode and/or refresh the image information in time, thereby improving user experience.
作为一种可能的实施方式，所述接收来自所述服务器的所述第二图像信息包括：接收来自所述服务器的第三图像信息；对所述第三图像信息进行解码，得到所述第二图像信息。As a possible implementation manner, the receiving the second image information from the server includes: receiving third image information from the server; decoding the third image information to obtain the second image information image information.
本申请实施例中，当终端设备接收到来自服务器的第三图像信息时，需要先对第三图像信息进行解码。终端设备的解码过程可以对应服务器的编码过程，从而可以保证信息处理过程的完整性。此外，上述图像信息的编码和解码过程，不但可以减少传输信息的数量，节约传输资源，还可以提高信息的可靠性和稳定性。In this embodiment of the present application, when the terminal device receives the third image information from the server, it needs to decode the third image information first. The decoding process of the terminal device can correspond to the encoding process of the server, thereby ensuring the integrity of the information processing process. In addition, the above-mentioned encoding and decoding process of image information can not only reduce the amount of transmitted information, save transmission resources, but also improve the reliability and stability of information.
作为一种可能的实施方式，该方法还可以包括：当检测到终端设备处于使用状态时，采集所述第四姿态信息，所述使用状态包括屏幕处于显示状态和/或所述终端设备已被用户佩戴。As a possible implementation manner, the method may further include: when it is detected that the terminal device is in a use state, collecting the fourth posture information, and the use state includes that the screen is in a display state and/or the terminal device has been User wears.
本申请实施例中，当用户恢复使用终端设备时，终端设备可以继续采集姿态信息，从而可以使得服务器恢复对图像信息的渲染，之后服务器可以向终端设备发送渲染之后的图像信息。进一步地，当终端设备接收到渲染之后的图像信息时，可以进行显示。当用户从暂停使用终端设备到恢复使用终端设备时，终端设备可以快速恢复使用。由此，不但可以保证用户的使用体验，还可以节约处理资源，以及避免能源的浪费。In the embodiment of the present application, when the user resumes using the terminal device, the terminal device can continue to collect gesture information, so that the server can resume the rendering of the image information, and then the server can send the rendered image information to the terminal device. Further, when the terminal device receives the rendered image information, it can be displayed. When the user suspends using the terminal device to resume using the terminal device, the terminal device can be quickly resumed. In this way, not only can the user experience be guaranteed, but also processing resources can be saved, and energy waste can be avoided.
第三方面公开一种信息处理装置，该信息处理装置可以为服务器，也可以为服务器中的模块(例如，芯片)，该信息处理装置可以包括：A third aspect discloses an information processing apparatus. The information processing apparatus may be a server or a module (eg, a chip) in the server. The information processing apparatus may include:
渲染单元，用于当接收到来自终端设备的姿态信息时，根据所述姿态信息对第一图像信息进行渲染，得到第二图像信息，所述姿态信息是用户的姿态信息，所述第一图像信息为虚拟现实VR图像信息；a rendering unit, configured to render the first image information according to the attitude information when receiving the attitude information from the terminal device, and obtain the second image information, the attitude information is the attitude information of the user, the first image The information is virtual reality VR image information;
发送单元，用于向所述终端设备发送所述第二图像信息。A sending unit, configured to send the second image information to the terminal device.
本申请实施例中，在现有的技术中，由于服务器是在等待相同的时间间隔后渲染一次图像信息，因此，当服务器接收到姿态信息时，通常需要等待一段时间，才可以开始对图像信息进行渲染，导致显示图像信息会产生一定的时延。本申请实施例中，当服务器接收到姿态信息时，可以立即对图像信息进行渲染，从而可以减少服务器处理过程中的等待时延，进而可以降低终端设备显示图像信息的时延。此外，当降低了终端设备显示图像信息的时延时，可以使得终端设备及时地刷新接收到的图像信息，以使得用户观看的图像画面更加流畅，从而可以提高用户体验。In the embodiment of the present application, in the prior art, since the server renders the image information once after waiting for the same time interval, when the server receives the gesture information, it usually needs to wait for a period of time before it can start to process the image information. Rendering will result in a certain delay in displaying image information. In this embodiment of the present application, when the server receives the gesture information, the image information can be rendered immediately, thereby reducing the waiting delay in the server processing process, and further reducing the delay in displaying the image information by the terminal device. In addition, when the time delay for the terminal device to display the image information is reduced, the terminal device can refresh the received image information in time, so that the images viewed by the user are smoother, thereby improving the user experience.
作为一种可能的实施方式，所述渲染单元具体用于：As a possible implementation manner, the rendering unit is specifically used for:
根据所述姿态信息确定帧率；determine the frame rate according to the attitude information;
根据所述帧率对所述第一图像信息进行渲染，得到第二图像信息。The first image information is rendered according to the frame rate to obtain second image information.
本申请实施例中，当前，服务器按照固定的帧率对图像信息进行渲染。然而，不同的终端设备的解码能力和/或刷新能力往往不同。有的终端设备的解码能力和/或刷新能力相对较高，当接收到服务器的图像信息时，能够及时地进行解码和显示；而有的终端设备的解码能力和/或刷新能力相对较低，当接收到服务器的图像信息时，来不及进行解码和刷新的时候，从而可能会出现丢帧和跳帧等问题。在本申请实施例中，服务器可以根据接收到的姿态信息确定渲染的图像信息对应的帧率，从而可以适应不同终端设备的解码能力和/或刷新能力，进而可以提高服务器对终端设备的适应能力，也可以保证终端设备显示图片信息的可靠性。进一步地，当终端设备显示图像信息的可靠性提高时，可以提高用户看到的图像画面的正确性，从而可以提高用户体验。In the embodiment of the present application, currently, the server renders the image information according to a fixed frame rate. However, the decoding capabilities and/or refresh capabilities of different terminal devices are often different. Some terminal devices have relatively high decoding capabilities and/or refresh capabilities, and can decode and display in time when receiving image information from the server; while some terminal devices have relatively low decoding capabilities and/or refresh capabilities, When the image information from the server is received, it is too late to decode and refresh, which may cause problems such as frame loss and frame skipping. In this embodiment of the present application, the server can determine the frame rate corresponding to the rendered image information according to the received attitude information, so that it can adapt to the decoding capability and/or refresh capability of different terminal devices, thereby improving the adaptability of the server to the terminal device. , it can also ensure the reliability of the picture information displayed by the terminal device. Further, when the reliability of displaying the image information by the terminal device is improved, the correctness of the image picture viewed by the user can be improved, thereby improving the user experience.
作为一种可能的实施方式，所述渲染单元根据所述姿态信息确定帧率包括：As a possible implementation manner, the rendering unit determining the frame rate according to the gesture information includes:
根据接收到所述姿态信息的频率确定所述帧率。The frame rate is determined according to the frequency at which the attitude information is received.
本申请实施例中，服务器可以根据接收到姿态信息的频率确定渲染的姿态信息的帧率。终端设备根据自身的解码能力和/或刷新能力调整采集姿态信息的频率，并根据上述频率向服务器发送姿态信息。服务器接收到姿态信息之后，可以根据接收姿态信息的频率确定渲染的频率，从而可以确定渲染出的图像信息的帧率。因此，服务器可以根据接收到姿态信息的频率对应调整渲染图片信息的帧率，从而可以使得服务器渲染出的图像信息可以适应终端设备的解码能力和/或刷新能力，也可以更好地适应终端设备当前的显示需要。In this embodiment of the present application, the server may determine the frame rate of the rendered gesture information according to the frequency of receiving the gesture information. The terminal device adjusts the frequency of collecting attitude information according to its own decoding ability and/or refresh ability, and sends attitude information to the server according to the above frequency. After the server receives the gesture information, it can determine the frequency of rendering according to the frequency of receiving the gesture information, so as to determine the frame rate of the rendered image information. Therefore, the server can adjust the frame rate of the rendered image information according to the frequency of receiving the gesture information, so that the image information rendered by the server can be adapted to the decoding capability and/or refresh capability of the terminal device, and can also be better adapted to the terminal device. current display needs.
作为一种可能的实施方式，该装置还可以包括：As a possible implementation manner, the device may also include:
编码单元，用于对所述第二图像信息进行编码，得到第三图像信息；an encoding unit, configured to encode the second image information to obtain third image information;
所述发送单元，具体用于向所述终端设备发送所述第三图像信息。The sending unit is specifically configured to send the third image information to the terminal device.
本申请实施例中，服务器对图像信息进行渲染之后，可以对渲染之后的图像信息进行编码，之后可以向服务器发送渲染后的图像信息。服务器对渲染后的图像信息进行编码，可以是对图像信息的压缩，从而可以减小传输图像信息的量，进而可以节约传输资源。此外，编码图像信息也可以是对图像信息的加密，从而可以提高图像信息的安全性和可靠性。In this embodiment of the present application, after rendering the image information, the server may encode the rendered image information, and then may send the rendered image information to the server. The server encodes the rendered image information, which may be compression of the image information, thereby reducing the amount of transmitted image information, thereby saving transmission resources. In addition, the encoded image information may also be encryption of the image information, so that the security and reliability of the image information can be improved.
第四方面公开一种信息处理装置，该信息处理装置可以为终端设备，也可以为终端设备中的模块(例如，芯片)，该信息处理装置可以包括：A fourth aspect discloses an information processing apparatus. The information processing apparatus may be a terminal device or a module (eg, a chip) in the terminal device, and the information processing apparatus may include:
采集单元，用于当检测到终端设备处于未使用状态时，停止采集第一姿态信息，所述未使用状态包括所述终端设备的屏幕处于息屏状态和/或所述终端设备未被用户佩戴，所述第一姿态信息是用户的姿态信息。A collection unit, configured to stop collecting the first attitude information when it is detected that the terminal device is in an unused state, where the unused state includes that the screen of the terminal device is in an off-screen state and/or the terminal device is not worn by the user , the first gesture information is the user's gesture information.
本申请实施例中，当用户不使用VR头盔等终端设备时，终端设备依然不停地显示图像会造成处理资源和能源的浪费。因此，当终端设备的显示屏处于息屏状态时和/或用户摘下头盔时，用户不会查看屏幕，终端设备可以停止采集第一姿态信息。当终端设备不采集姿态信息时，终端设备不会向服务器发送姿态信息，因此，服务器不会接收到姿态信息，此时，服务器不会对图像信息进行渲染。因此，当用户不使用终端设备的时候，终端设备可以停止采集信息，以及服务器可以停止渲染图像信息，从而可以节约处理资源，以及可以避免能源的浪费。In the embodiment of the present application, when the user does not use a terminal device such as a VR helmet, the terminal device still continuously displays images, which will cause waste of processing resources and energy. Therefore, when the display screen of the terminal device is in an off-screen state and/or when the user takes off the helmet, the user does not view the screen, and the terminal device can stop collecting the first posture information. When the terminal device does not collect the gesture information, the terminal device will not send the gesture information to the server, therefore, the server will not receive the gesture information, and at this time, the server will not render the image information. Therefore, when the user does not use the terminal device, the terminal device can stop collecting information, and the server can stop rendering image information, so that processing resources can be saved and energy waste can be avoided.
作为一种可能的实施方式，该装置还可以包括：As a possible implementation manner, the device may also include:
发送单元，用于当采集到第二姿态信息时，向所述服务器发送所述第二姿态信息；a sending unit, configured to send the second attitude information to the server when the second attitude information is collected;
接收单元，用于接收来自所述服务器的所述第二图像信息；a receiving unit, configured to receive the second image information from the server;
显示单元，用于显示所述第二图像信息。The display unit is used for displaying the second image information.
本申请实施例中，当终端设备采集姿态信息时候，可以立即向服务器发送姿态信息，从而可以保证终端设备的采集姿态信息的频率与服务器接收到姿态信息的频率基本一致。由于服务器接收到姿态信息便立即对第一图像信息进行渲染，从而可以保证服务器渲染得到的第二图像信息的帧率与终端设备的发送频率对应一致，从而可以使得终端设备调整采集姿态信息的频率可以对应调整服务器渲染的图像信息的帧率，进而可以保证终端设备调能够适应服务器发送第二图像信息的频率。此外，当终端设备接收到第二图像信息时，可以显示第二图像信息，从而可以保证方案的完整性。In this embodiment of the present application, when the terminal device collects attitude information, it can immediately send the attitude information to the server, thereby ensuring that the frequency at which the terminal equipment collects attitude information is basically consistent with the frequency at which the server receives attitude information. Since the server immediately renders the first image information after receiving the attitude information, it can ensure that the frame rate of the second image information rendered by the server is consistent with the sending frequency of the terminal device, so that the terminal device can adjust the frequency of collecting attitude information. The frame rate of the image information rendered by the server can be adjusted accordingly, thereby ensuring that the terminal device can adapt to the frequency at which the server sends the second image information. In addition, when the terminal device receives the second image information, the second image information can be displayed, so that the integrity of the solution can be guaranteed.
作为一种可能的实施方式，所述发送单元具体用于：As a possible implementation manner, the sending unit is specifically used for:
根据终端设备的解码能力和/或刷新能力调整所述第二姿态信息，得到第三姿态信息，所述解码能力为所述终端设备解码的最快速度，所述刷新能力为所述终端设备显示的最大帧率；Adjust the second attitude information according to the decoding capability and/or refresh capability of the terminal device to obtain the third attitude information, where the decoding capability is the fastest decoding speed of the terminal device, and the refresh capability is the display capability of the terminal device the maximum frame rate;
向所述服务器发送所述第三姿态信息。Sending the third gesture information to the server.
本申请实施例中，由于终端设备的解码能力和/或刷新能力不能够与服务器同步，因此，当服务器按照固定帧率渲染图像信息，并向终端设备发送上述图像信息时，解码能力和/或刷新能力较弱的终端设备可能来不及解码和/或显示图像信息，从而可能造成丢帧和跳帧等问题。由于服务器接收到姿态信息时，便立即渲染图像信息，因此，终端设备可以根据自身的解码能力和/或刷新能力调整姿态信息，使得服务器渲染后的图像信息的帧率能够与终端设备的解码能力和/或刷新能力相适应。经过上述终端设备对姿态信息的调整，可以保证终端设备能够及时地解码和/或刷新图像信息，从而可以提高用户体验。In this embodiment of the present application, since the decoding capability and/or refresh capability of the terminal device cannot be synchronized with the server, when the server renders image information at a fixed frame rate and sends the above image information to the terminal device, the decoding capability and/or Terminal devices with weak refresh capabilities may not be able to decode and/or display image information in time, which may cause frame loss and frame skipping. Since the server immediately renders the image information when it receives the gesture information, the terminal device can adjust the gesture information according to its own decoding capability and/or refresh capability, so that the frame rate of the image information rendered by the server can match the decoding capability of the terminal device. and/or refresh capability. After the above-mentioned adjustment of the attitude information by the terminal device, it can be ensured that the terminal device can decode and/or refresh the image information in time, thereby improving user experience.
作为一种可能的实施方式，所述接收单元具体用于：As a possible implementation manner, the receiving unit is specifically used for:
接收来自所述服务器的第三图像信息；receiving third image information from the server;
对所述第三图像信息进行解码，得到所述第二图像信息。Decoding the third image information to obtain the second image information.
本申请实施例中，当终端设备接收到来自服务器的第三图像信息时，需要先对第三图像信息进行解码。终端设备的解码过程可以对应服务器的编码过程，从而可以保证信息处理过程的完整性。此外，上述图像信息的编码和解码过程，不但可以减少传输信息的数量，节约传输资源，还可以提高信息的可靠性和稳定性。In this embodiment of the present application, when the terminal device receives the third image information from the server, it needs to decode the third image information first. The decoding process of the terminal device can correspond to the encoding process of the server, thereby ensuring the integrity of the information processing process. In addition, the above-mentioned encoding and decoding process of image information can not only reduce the amount of transmitted information, save transmission resources, but also improve the reliability and stability of information.
作为一种可能的实施方式，所述采集单元，还用于当检测到终端设备处于使用状态时，采集所述第四姿态信息，所述使用状态包括屏幕处于显示状态和/或所述终端设备已被用户佩戴。As a possible implementation manner, the collection unit is further configured to collect the fourth posture information when it is detected that the terminal device is in a use state, and the use state includes that the screen is in a display state and/or the terminal device is in a state of being displayed. worn by the user.
本申请实施例中，当用户恢复使用终端设备时，终端设备可以继续采集姿态信息，从而可以使得服务器恢复对图像信息的渲染，之后服务器可以向终端设备发送渲染之后的图像信息。进一步地，当终端设备接收到渲染之后的图像信息时，可以进行显示。当用户从暂停使用终端设备到恢复使用终端设备时，终端设备可以快速恢复使用。由此，不但可以保证用户的使用体验，还可以节约处理资源，以及避免能源的浪费。In the embodiment of the present application, when the user resumes using the terminal device, the terminal device can continue to collect gesture information, so that the server can resume the rendering of the image information, and then the server can send the rendered image information to the terminal device. Further, when the terminal device receives the rendered image information, it can be displayed. When the user suspends using the terminal device to resume using the terminal device, the terminal device can be quickly resumed. In this way, not only can the user experience be guaranteed, but also processing resources can be saved, and energy waste can be avoided.
第五方面公开一种信息处理装置，该信息处理装置可以为服务器或者服务器内的模块(例如，芯片)。该信息处理装置可以包括处理器、存储器、输入接口和输出接口，所述输入接口用于接收来自所述信息处理装置之外的其它装置的信息，所述输出接口用于向所述信息处理装置之外的其它装置输出信息，当所述处理器执行所述存储器存储的计算机程序时，使得 所述处理器执行第一方面或第一方面的任一实施方式公开的信息处理方法。A fifth aspect discloses an information processing apparatus, which may be a server or a module (eg, a chip) within the server. The information processing apparatus may include a processor, a memory, an input interface for receiving information from devices other than the information processing apparatus, and an output interface for sending information to the information processing apparatus The other device outputs information, and when the processor executes the computer program stored in the memory, the processor causes the processor to execute the information processing method disclosed in the first aspect or any implementation manner of the first aspect.
第六方面公开一种信息处理装置，该信息处理装置可以为终端设备或者终端设备内的模块(例如，芯片)。该信息处理装置可以包括处理器、存储器、输入接口和输出接口，所述输入接口用于接收来自所述信息处理装置之外的其它装置的信息，所述输出接口用于向所述信息处理装置之外的其它装置输出信息，当所述处理器执行所述存储器存储的计算机程序时，使得所述处理器执行第二方面或第二方面的任一实施方式公开的信息处理方法。A sixth aspect discloses an information processing apparatus, where the information processing apparatus may be a terminal device or a module (eg, a chip) in the terminal device. The information processing apparatus may include a processor, a memory, an input interface for receiving information from devices other than the information processing apparatus, and an output interface for sending information to the information processing apparatus The other device outputs information, and when the processor executes the computer program stored in the memory, the processor causes the processor to execute the information processing method disclosed in the second aspect or any embodiment of the second aspect.
第七方面公开一种信息处理***，该信息处理***包括第五方面的信息处理装置和第六方面的信息处理装置。A seventh aspect discloses an information processing system including the information processing apparatus of the fifth aspect and the information processing apparatus of the sixth aspect.
第八方面公开一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序或计算机指令，当该计算机程序或计算机指令运行时，实现如上述各方面公开的信息处理方法。An eighth aspect discloses a computer-readable storage medium, where a computer program or computer instruction is stored thereon, and when the computer program or computer instruction is executed, the information processing method disclosed in the above aspects is implemented.
第九方面公开一种芯片，包括处理器，用于执行存储器中存储的程序，当程序被执行时，使得芯片执行上面的方法。A ninth aspect discloses a chip including a processor for executing a program stored in a memory, and when the program is executed, causes the chip to execute the above method.
作为一种可能的实施方式，存储器位于芯片之外。As a possible implementation, the memory is located off-chip.
附图说明Description of drawings
图1是本发明实施例公开的一种***架构示意图；1 is a schematic diagram of a system architecture disclosed in an embodiment of the present invention;
图2是本发明实施例公开的一种产品形态的示意图；2 is a schematic diagram of a product form disclosed in an embodiment of the present invention;
图3是本发明实施例公开的一种姿态信息交互形式的示意图；3 is a schematic diagram of a gesture information interaction form disclosed in an embodiment of the present invention;
图4是本发明实施例公开的一种云VR方案结构的示意图；4 is a schematic diagram of the structure of a cloud VR solution disclosed in an embodiment of the present invention;
图5是本发明实施例公开的一种图像信息处理的流程示意图；5 is a schematic flowchart of an image information processing disclosed in an embodiment of the present invention;
图6是本发明实施例公开的一种信息处理方法的流程示意图；6 is a schematic flowchart of an information processing method disclosed in an embodiment of the present invention;
图7是本发明实施例公开的另一种图像信息处理的流程示意图；7 is a schematic flowchart of another image information processing disclosed in an embodiment of the present invention;
图8是本发明实施例公开的另一种信息处理方法的流程示意图；8 is a schematic flowchart of another information processing method disclosed in an embodiment of the present invention;
图9是本发明实施例公开的一种信息处理装置的结构示意图；9 is a schematic structural diagram of an information processing apparatus disclosed in an embodiment of the present invention;
图10是本发明实施例公开的另一种信息处理装置的结构示意图；10 is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention;
图11是本发明实施例公开的又一种信息处理装置的结构示意图；11 is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention;
图12是本发明实施例公开的又一种信息处理装置的结构示意图。FIG. 12 is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention.
具体实施方式Detailed ways
本申请实施例公开了一种信息处理方法、装置及***，用于降低图像信息显示的时延。以下分别进行详细说明。The embodiments of the present application disclose an information processing method, device and system, which are used to reduce the time delay of displaying image information. Each of them will be described in detail below.
为了更好地理解本申请实施例公开的一种信息处理方法、装置及***，下面先对本申请实施例中使用的***架构进行描述。请参阅图1，图1是本申请实施例公开的一种***架构示意图。如图1所示，上述***构架可以包括服务器和终端设备，服务器可以与终端设备进行交互，即服务器可以向终端设备发送数据，也可以接收来自终端设备的数据。需要说明的是，本申请实施例以服务器作为图像和/或视频处理设备为例说明，也就是说云虚拟现实(cloud VR)设备来实现图像和/或视频处理设备的功能为示例进行说明。In order to better understand an information processing method, device, and system disclosed in the embodiments of the present application, the system architecture used in the embodiments of the present application is first described below. Please refer to FIG. 1. FIG. 1 is a schematic diagram of a system architecture disclosed in an embodiment of the present application. As shown in FIG. 1 , the above system architecture may include a server and a terminal device, and the server may interact with the terminal device, that is, the server may send data to the terminal device, and may also receive data from the terminal device. It should be noted that the embodiments of the present application are described by taking a server as an image and/or video processing device as an example, that is, a cloud virtual reality (cloud VR) device implementing the functions of an image and/or video processing device as an example for description.
服务器是部署在云端的远程服务器，具有图像处理能力、数据计算能力以及编码能力，例如，可以执行渲染操作、逻辑运算功能等。服务器可以是超多核服务器、部署有图形处理器(graphics processing unit，GPU)集群的计算机、大型的分布式计算机、硬件资源池化的集群 计算机等等。服务器还可以是部署边缘计算节点、云计算节点等云端设备，此处不构成限定。A server is a remote server deployed in the cloud, with image processing capabilities, data computing capabilities, and coding capabilities, for example, it can perform rendering operations, logical operations, and the like. The server can be an ultra-multi-core server, a computer deployed with a graphics processing unit (GPU) cluster, a large-scale distributed computer, a cluster computer with hardware resource pooling, and the like. The server may also be a cloud device such as a deployment edge computing node, cloud computing node, etc., which is not limited here.
终端设备可以包括佩戴在用户的头部的设备，如VR眼镜、VR头盔、VR一体机、手机+VR眼镜(VR Glass)等，还可以包括佩戴在用户的其它部位的设备，如佩戴在用户的手部、肘部、脚部、膝盖处的设备等，例如，游戏手柄等。终端设备可以通过显示器向用户显示VR视频的视频图像。其中，终端设备可以不在本地保存VR视频和跳转信息的数据，而将相关数据都保存在视频服务器设备中，在播放VR视频时，显示VR视频的视频图像，并向视频服务器设备发送用户的姿态信息。Terminal devices may include devices worn on the user's head, such as VR glasses, VR helmets, VR all-in-one machines, mobile phones + VR glasses (VR Glass), etc., and may also include devices worn on other parts of the user, such as devices at the hands, elbows, feet, knees, etc., such as gamepads, etc. The terminal device can display the video image of the VR video to the user through the display. Wherein, the terminal device may not save the data of VR video and jump information locally, but save the relevant data in the video server device. When playing the VR video, the video image of the VR video is displayed, and the user's data is sent to the video server device. posture information.
近年来，VR技术已经逐步进入娱乐、教育、医疗、环保、交通运输、公共卫生等众多领域，具有极大的应用价值和商业潜力。VR头盔和VR眼睛等一系列的产品已经在市场上进行推广，看VR视频，玩VR游戏等娱乐形式已经被消费者接受。In recent years, VR technology has gradually entered many fields such as entertainment, education, medical care, environmental protection, transportation, public health, etc., and has great application value and commercial potential. A series of products such as VR helmets and VR eyes have been promoted in the market, and entertainment forms such as watching VR videos and playing VR games have been accepted by consumers.
请参阅图2，图2是本申请实施例公开的一种产品形态的示意图。如图2所示，VR的产品形态主要分为两种，一种是一体式头盔VR，另一种是系留VR。一体式头盔VR的VR应用安装在头盔中，头盔VR可以利用嵌入式处理器做VR计算和显示。系留VR则是通过高清晰度多媒体接口(high definition multimedia interface，HDMI)线连接VR头盔和个人电脑(personal computer，PC)，PC安装VR应用，并进行运行，头盔显示通过HDMI接受到的画面。Please refer to FIG. 2 , which is a schematic diagram of a product form disclosed in an embodiment of the present application. As shown in Figure 2, VR product forms are mainly divided into two types, one is an integrated helmet VR, and the other is a tethered VR. The VR application of the integrated helmet VR is installed in the helmet, and the helmet VR can use the embedded processor for VR calculation and display. The tethered VR is to connect the VR headset and a personal computer (PC) through a high definition multimedia interface (HDMI) cable. The PC installs the VR application and runs it, and the headset displays the screen received through HDMI. .
上述两种形式均需要用户的头部和/或肘部与VR传感器产生姿态信息的交互，来达到视觉沉浸式体验。请参阅图3，图3是本申请实施例公开的一种姿态信息交互形式的示意图。如图3所示，姿态信息的交互形式可以分为两种，即三自由度(3degree of freedom，3Dof)和六自由度(6degree of freedom，6Dof)。其中，3Dof是指以交互中心作为坐标原点，围绕x，y，z三个坐标轴进行旋转的姿态交互；6Dof是指在3Dof的基础上，增加沿x，y，z三个坐标轴正负方向的平移的位置交互。即3Dof仅包括围绕x，y，z三个坐标轴“旋转”的交互，不包括沿着x，y，z三个坐标轴“平移”的交互；而6Dof包括“旋转”和“平移”的两种交互。应理解，在未来技术的演进过程中，可以出现其它的姿态信息，此处不加限定。The above two forms both require the interaction of the user's head and/or elbow with the VR sensor to generate gesture information to achieve a visually immersive experience. Please refer to FIG. 3 , which is a schematic diagram of a gesture information interaction form disclosed by an embodiment of the present application. As shown in Figure 3, the interactive forms of attitude information can be divided into two types, namely three degrees of freedom (3 degrees of freedom, 3Dof) and six degrees of freedom (6 degrees of freedom, 6Dof). Among them, 3Dof refers to the gesture interaction that takes the interaction center as the coordinate origin and rotates around the three coordinate axes of x, y, and z; 6Dof refers to the addition of positive and negative along the three coordinate axes of x, y, and z on the basis of 3Dof. The position of the translation of the direction interacts. That is, 3Dof only includes the interaction of "rotation" around the three axes of x, y, and z, and does not include the interaction of "translation" along the three axes of x, y, and z; while 6Dof includes the interaction of "rotation" and "translation" two interactions. It should be understood that in the evolution of future technologies, other gesture information may appear, which is not limited here.
在本地VR***中，VR终端设备采集到的玩家的3/6Dof信息之后，可以驱动VR应用制作每一个VR视频帧画面，并可以显示在VR头盔里的屏幕上。然而，随着VR应用复杂度的提升，普通的一体式头盔VR和家用PC无法满足大型的VR应用。例如，VR游戏中更加逼真和流畅的画面或者沉浸式宏大的VR大型游戏，对VR***有更大的算力要求。因此，随时VR技术的演进和网络技术的发展，VR形态正在从系留VR和一体式头盔等本地VR形态向云VR方向发展。在云VR技术中，云计算或者边缘计算可以提供无限制的算力让VR应用自由发挥其沉浸式优势。此时，VR端侧设备(即终端设备)可以采集3Dof/6Dof的姿态信息，并向云端设备(服务器)发送3Dof/6Dof的姿态信息。服务器开启VR应用之后，可以实时接收端侧的信息，并可以将姿态信息进行实时渲染。之后服务器可以把渲染后的VR图像信息进行压缩后，之后可以向终端设备发送上述压缩的图像信息。终端设备接收到来自服务器的压缩信息之后，可以解码压缩信息，得到图像信息，之后可以显示图像信息。在这个过程中，完成云VR交互。In the local VR system, after the 3/6Dof information of the player is collected by the VR terminal device, it can drive the VR application to make each VR video frame and display it on the screen in the VR helmet. However, with the increasing complexity of VR applications, ordinary VR headsets and home PCs cannot meet large-scale VR applications. For example, more realistic and smooth images in VR games or immersive and grand VR games require greater computing power for the VR system. Therefore, with the evolution of VR technology and the development of network technology, the VR form is developing from local VR forms such as tethered VR and all-in-one helmets to cloud VR. In cloud VR technology, cloud computing or edge computing can provide unlimited computing power, allowing VR applications to freely exert their immersive advantages. At this time, the VR end-side device (ie, the terminal device) can collect the attitude information of 3Dof/6Dof, and send the attitude information of 3Dof/6Dof to the cloud device (server). After the server starts the VR application, it can receive the terminal-side information in real time, and can render the gesture information in real time. After that, the server can compress the rendered VR image information, and then can send the above compressed image information to the terminal device. After receiving the compressed information from the server, the terminal device can decode the compressed information to obtain image information, and then can display the image information. In this process, the cloud VR interaction is completed.
5G的发展推动了云VR的普及，市场上很多厂商，如华为、视博云、英伟达、电信天翼云等都有云VR的解决方案和相关产品。请参阅图4，图4是本申请实施例公开的一种云VR方案结构的示意图。如图4所示，当前，云VR的服务器可以提供VR应用的运行、渲染和编码推流。服务器和终端设备(如，VR头盔)可以通过5G网络进行通信。即终端设备可以通过5G向服务器发送姿态信息(3/6Dof信息)，服务器可以通过5G可以向终端设备发送渲染之后的图像信息(即VR视频流)。应理解，服务器与终端设备的通信过程可以通过蜂窝移动通信网络(例如，5G)进行、也可以WiFi进行、还可以云VR技术一般推送的双目分辨率为3K～4K左右，帧 率为60～90每秒传输帧数(frames per second，FPS)的视频流，VR终端设备接收到视频流之后，可以进行解码和显示。终端设备一般以固定时间间隔(例如，16ms或11ms)向服务器上传一次3Dof/6Dof的姿态信息。The development of 5G has promoted the popularity of cloud VR. Many manufacturers in the market, such as Huawei, Shiboyun, NVIDIA, and Telecom Tianyi Cloud, have cloud VR solutions and related products. Please refer to FIG. 4. FIG. 4 is a schematic diagram of the structure of a cloud VR solution disclosed in an embodiment of the present application. As shown in Figure 4, currently, the server of cloud VR can provide the running, rendering and encoding push streaming of VR applications. Servers and end devices (eg, VR headsets) can communicate via 5G networks. That is, the terminal device can send gesture information (3/6Dof information) to the server through 5G, and the server can send the rendered image information (ie VR video stream) to the terminal device through 5G. It should be understood that the communication process between the server and the terminal device can be carried out through a cellular mobile communication network (for example, 5G), WiFi, or cloud VR technology. The binocular resolution generally pushed is about 3K to 4K, and the frame rate is 60. ~90 frames per second (FPS) video stream, after the VR terminal device receives the video stream, it can be decoded and displayed. The terminal device generally uploads the 3Dof/6Dof attitude information to the server at a fixed time interval (for example, 16ms or 11ms).
请参阅图5，图5是本申请实施例公开的一种图像信息处理的流程示意图，如图5所述，在目前的云VR技术方案中，云VR终端设备建立VR业务之后，可以周期性地向服务器发送3Dof/6Dof姿态信息。服务器接收到来自终端设备的姿态信息之后，可以周期性地渲染图像信息，之后可以向终端设备发送渲染后的图像信息。终端设备接收到来自服务器的图像信息之后，可以显示这一图像信息。在上述交互过程中，终端设备和服务器的未进行协同，终端设备周期性地发送姿态信息，服务器周期性地渲染图像信息。服务器间隔相同的时间间隔触发一次垂直同步(例如，时间间隔为16ms或11ms触发一次垂直同步)，之后服务器可以立即对姿态信息进行渲染。在交互过程中，服务器接收到姿态信息的时间点往往处于两次垂直同步之间。其中，垂直同步是指水平扫描线在垂直方向的堆积形成了完整的画面的过程，水平扫描线可以由单个像素组成。当终端设备扫描一帧之后，可以产生一个垂直同步信号，一个垂直同步信号可以触发一次图像渲染过程。当服务器接收到来自终端设备的姿态信息之后，还需要等待约6～9ms，之后才能够对姿态数据进行渲染，从而会产生较大的时延。在云VR技术中，信息显示的时延越低，用户体验越好，因此，如何降低信息显示的时延是一个亟待解决的问题。Please refer to FIG. 5. FIG. 5 is a schematic flowchart of an image information processing disclosed in an embodiment of the present application. As shown in FIG. 5, in the current cloud VR technical solution, after a cloud VR terminal device establishes a VR service, it can periodically Send 3Dof/6Dof pose information to the server. After receiving the gesture information from the terminal device, the server may periodically render the image information, and then may send the rendered image information to the terminal device. After receiving the image information from the server, the terminal device can display the image information. In the above interaction process, the terminal device and the server do not cooperate, the terminal device periodically sends gesture information, and the server periodically renders image information. The server triggers a vertical synchronization at the same time interval (for example, the time interval is 16ms or 11ms to trigger a vertical synchronization), and then the server can immediately render the pose information. In the interaction process, the time when the server receives the attitude information is often between two vertical synchronizations. The vertical synchronization refers to a process in which horizontal scan lines are stacked in the vertical direction to form a complete picture, and the horizontal scan lines may be composed of a single pixel. After the terminal device scans a frame, a vertical synchronization signal can be generated, and a vertical synchronization signal can trigger an image rendering process. After the server receives the attitude information from the terminal device, it still needs to wait for about 6-9 ms before rendering the attitude data, which will cause a large delay. In the cloud VR technology, the lower the information display delay, the better the user experience. Therefore, how to reduce the information display delay is an urgent problem to be solved.
基于上述网络架构，请参阅图6，图6是本申请实施例公开的一种信息处理方法的流程示意图。其中，本申请实施例中由终端设备执行的功能也可以由终端设备中的模块(例如，芯片)来执行，由服务器执行的功能也可以由服务器中的模块(例如，芯片)来执行。如图6所示，该信息处理方法可以包括以下步骤。Based on the above network architecture, please refer to FIG. 6 , which is a schematic flowchart of an information processing method disclosed in an embodiment of the present application. The functions performed by the terminal device in the embodiments of the present application may also be performed by a module (eg, a chip) in the terminal device, and the function performed by the server may also be performed by a module (eg, a chip) in the server. As shown in FIG. 6 , the information processing method may include the following steps.
601、终端设备采集姿态信息。601. The terminal device collects attitude information.
在用户使用VR终端设备的过程中，终端设备可以采集姿态信息。姿态信息可以是指用户的身体姿态信息。姿态信息可以表示用户身体转动以及移动的情况，例如，3Dof和6Dof的姿态信息。终端设备可以通过本地的传感器获取3Dof或6Dof的姿态信息。例如，终端设备可以通过VR头盔，以及手柄等设备上的传感器进行动作捕捉(motion capture)，从而可以采集姿态信息。其中，动作捕捉可以为记录和处理人体或动物动作的技术。应理解，此处为举例说明采集姿态信息的方式，不构成限定。When a user uses a VR terminal device, the terminal device can collect gesture information. The gesture information may refer to the user's body gesture information. The gesture information can represent the rotation and movement of the user's body, for example, the gesture information of 3Dof and 6Dof. The terminal device can obtain 3Dof or 6Dof attitude information through local sensors. For example, the terminal device can perform motion capture (motion capture) through the sensors on the VR headset and the handle, so as to collect gesture information. Among them, motion capture can be a technology of recording and processing human or animal movements. It should be understood that the manner of collecting attitude information is illustrated here as an example, and is not intended to be limiting.
终端设备可以在不同的情况下触发采集姿态信息。The terminal device can trigger the collection of attitude information in different situations.
一种情况下，当终端设备获取到启动操作时，可以开始采集姿态信息。例如，用户可以通过屏幕触控或者按下按钮等方式开启终端设备，响应于上述启动操作，终端设备可以开始采集姿态信息。In one case, when the terminal device acquires the start-up operation, it can start to collect attitude information. For example, the user can turn on the terminal device by touching the screen or pressing a button, and in response to the above start-up operation, the terminal device can start collecting gesture information.
当终端设备确定的用户启动终端设备的VR应用之后，终端设备可以开始采集姿态信息。当终端设备确定的用户关闭终端设备的VR应用之后，终端设备可以停止采集姿态信息。当终端设备检测存在用户启动VR应用(例如，VR游戏、VR视频等)时，终端设备可以响应于启动操作，与服务器建立连接。终端设备与服务器建立连接之后，终端设备可以开始采集姿态信息。当终端设备检测存在用户结束VR应用时，终端设备可以响应于结束操作，与服务器断开连接。终端设备与服务器断开连接之后，终端设备可以停止采集姿态信息。After the user determined by the terminal device starts the VR application of the terminal device, the terminal device may start to collect gesture information. After the user determined by the terminal device closes the VR application of the terminal device, the terminal device may stop collecting gesture information. When the terminal device detects that the user starts a VR application (eg, VR game, VR video, etc.), the terminal device may establish a connection with the server in response to the start-up operation. After the terminal device establishes a connection with the server, the terminal device can start to collect attitude information. When the terminal device detects that the user terminates the VR application, the terminal device may disconnect from the server in response to the termination operation. After the terminal device is disconnected from the server, the terminal device may stop collecting attitude information.
另一种情况下，当终端设备处于未使用状态时，可以停止采集姿态信息(第一姿态信息)；当终端设备处于使用状态时，采集姿态信息(第四姿态信息)。In another case, when the terminal device is not in use, the collection of attitude information (first attitude information) may be stopped; when the terminal device is in use, attitude information (fourth attitude information) may be collected.
在终端设备启动VR应用之后，可以检测终端设备是否处于使用状态。其中，使用状态可以包括屏幕处于显示状态和/或终端设备已被用户佩戴，未使用状态可以包括终端设备的屏幕处于息屏状态和/或终端设备未被用户佩戴。终端设备可以通过显示屏确认屏幕是否处于息屏状态(例如，用户按下暂停或息屏控键)确定终端设备是否处于使用状态，还可以通过传感器检测是否存在用户佩戴当前的终端设备(例如，佩戴VR头盔)确定终端设备是否使用状态。需要说明的是，第一姿态信息可以是终端设备暂停采集姿态信息时的姿态信息；第四姿态信息可以是终端设备恢复采集姿态信息时的姿态信息。After the terminal device starts the VR application, it can be detected whether the terminal device is in a use state. Wherein, the use state may include that the screen is in a display state and/or the terminal device has been worn by the user, and the unused state may include that the screen of the terminal device is in a screen-off state and/or the terminal device is not worn by the user. The terminal device can confirm whether the screen is in the off-screen state through the display screen (for example, the user presses the pause or screen-off control key) to determine whether the terminal device is in use, and can also detect whether the user is wearing the current terminal device through the sensor (for example, Wear a VR helmet) to determine whether the terminal device is in use. It should be noted that the first attitude information may be attitude information when the terminal device suspends collecting attitude information; the fourth attitude information may be attitude information when the terminal device resumes collecting attitude information.
当检测到终端设备处于未使用状态时，终端设备可以停止采集姿态信息。即当检测到用户从使用终端设备到不使用终端设备时，终端设备可以停止采集姿态信息。当停止采集姿态信息时，终端设备停止向服务器发送姿态信息。此时，由于服务器不会接收到姿态信息，进而不会触发对第一图像信息的渲染，因此，当终端设备停止采集姿态信息时，服务器便可以停止渲染。When it is detected that the terminal device is not in use, the terminal device can stop collecting attitude information. That is, when it is detected that the user changes from using the terminal device to not using the terminal device, the terminal device can stop collecting attitude information. When the collection of attitude information is stopped, the terminal device stops sending attitude information to the server. At this time, since the server will not receive the gesture information, and thus will not trigger the rendering of the first image information, when the terminal device stops collecting the gesture information, the server may stop rendering.
当检测到终端设备处于使用状态时，终端设备可以继续采集姿态信息。即当检测到用户从不使用终端设备到使用终端设备(当屏幕从熄灭到显示状态，和/或，用户继续佩戴好终端设备)时，终端设备可以恢复采集姿态信息。即终端设备从停止采集姿态信息的状态变为开始采集的状态。当终端设备采集到姿态信息之后，可以向服务器发送姿态信息。此时，服务器可以接收到姿态信息，并可以开始对第一图像信息进行渲染。When it is detected that the terminal device is in use, the terminal device can continue to collect attitude information. That is, when it is detected that the user never uses the terminal device to use the terminal device (when the screen is turned off to the display state, and/or the user continues to wear the terminal device), the terminal device can resume collecting attitude information. That is, the terminal device changes from a state of stopping collecting attitude information to a state of starting to collect. After the terminal device collects the attitude information, it can send the attitude information to the server. At this point, the server can receive the gesture information, and can start rendering the first image information.
应理解，当用户不使用终端设备时，终端设备继续采集姿态信息，以及服务器继续根据姿态信息进行图像渲染会造成处理资源和能源的浪费。因此，当终端设备检测到用户暂时不使用终端设备时，可以停止采集姿态信息，从而可以使得服务器停止进行图像渲染，从而可以节约处理资源和能源。It should be understood that when the user does not use the terminal device, the terminal device continues to collect gesture information, and the server continues to perform image rendering according to the gesture information, which will cause waste of processing resources and energy. Therefore, when the terminal device detects that the user does not use the terminal device temporarily, it can stop collecting gesture information, so that the server can stop image rendering, thereby saving processing resources and energy.
又一种情况下，当终端设备接收到来自服务器的第二图像信息时，可以继续采集姿态信息。其中，第二图像信息为虚拟现实VR图像信息。例如，当终端设备接收到第二图像信息之后，可以触发终端设备继续进行动作捕捉，采集姿态信息。In yet another case, when the terminal device receives the second image information from the server, it can continue to collect gesture information. The second image information is virtual reality VR image information. For example, after the terminal device receives the second image information, the terminal device may be triggered to continue to perform motion capture and collect gesture information.
在采集姿态信息之前，终端设备可以先启动云VR业务。即用户可以通过显示屏启动云VR的应用，例如，云VR游戏、云VR直播、云VR视频等。终端设备响应用户的启动操作，可以向服务器发送第一请求。对应地，服务器可以接收来自终端设备的第一请求。其中，第一请求可以为终端设备向服务器请求发起VR业务的请求信息，第一请求可以包括终端设备启动的VR应用的对应VR业务的信息，还可以包括终端设备与服务器的连接方式信息。Before collecting attitude information, the terminal device can start the cloud VR service. That is, users can start cloud VR applications through the display screen, such as cloud VR games, cloud VR live broadcasts, and cloud VR videos. The terminal device may send the first request to the server in response to the user's start-up operation. Correspondingly, the server may receive the first request from the terminal device. The first request may be request information for the terminal device to request the server to initiate a VR service, and the first request may include information about the VR service corresponding to the VR application started by the terminal device, and may also include connection mode information between the terminal device and the server.
服务器接收来自终端设备的第一请求之后，可以根据第一请求与服务器协商启动VR业务。即可以理解为，服务器可以通过第一请求确定与当前终端设备的连接方式以及需要使用的VR业务(如，VR场景信息)。即服务器可以根据第一请求确定当前终端设备所使用到的VR业务，以及对应的图像信息，还可以根据第一请求确定终端设备与服务器的连接方式。After receiving the first request from the terminal device, the server may negotiate with the server to start the VR service according to the first request. That is, it can be understood that the server can determine the connection mode with the current terminal device and the VR service (eg, VR scene information) to be used through the first request. That is, the server may determine the VR service currently used by the terminal device and the corresponding image information according to the first request, and may also determine the connection mode between the terminal device and the server according to the first request.
当终端设备与服务器建立起连接之后，终端设备可以启动本地的各个功能组件。其中功能组件可以包括接收图像信息的组件(如，初始化收流)、解码组件(如，解码功能)、显示组件(如，VR画面纹理刷新功能)、采集姿态信息(如，动作捕捉功能)的组件和屏幕组件(如，控制屏幕开关功能)等。After the terminal device establishes a connection with the server, the terminal device can start various local functional components. The functional components may include components for receiving image information (eg, initializing streaming), decoding components (eg, decoding function), display components (eg, VR image texture refresh function), and collecting attitude information (eg, motion capture function) Components and screen components (eg, controlling the screen switch function), etc.
602、终端设备向服务器发送姿态信息。602. The terminal device sends gesture information to the server.
当终端设备采集到姿态信息(第二姿态信息)时，可以向服务器发送第二姿态信息。对应地，服务器可以接收来自终端设备的姿态信息。When the terminal device collects the gesture information (second gesture information), it can send the second gesture information to the server. Correspondingly, the server may receive gesture information from the terminal device.
终端设备可以根据终端设备的解码能力和/或刷新能力调整所述第二姿态信息，得到第三 姿态信息，之后可以向服务器发送第三姿态信息。对应地，服务器可以接收来自终端设备的第三姿态信息。其中，解码能力可以为终端设备解码的最快速度，刷新能力可以为终端设备显示的最大帧率。即终端设备可以根据自身的解码能力和/或刷新能力确定采集姿态信息的最大频率，从而可以确定第三姿态信息。例如，终端设备解码能力是70(frames per second，FPS)，以及刷新能力为60FPS。终端设备可以先确定自身的解码能力和/或刷新能力，之后可以确定一个在解码能力和/或刷新能力范围内的频率(即采集第三姿态信息最大频率范围)，之后可以立即发送第三姿态信息。即可以理解为，终端设备采集第三姿态信息的频率需要小于或等于终端设备的解码能力和/或刷新能力的范围。例如，当终端设备的解码能力是70FPS时，终端设备可以确定采集第三姿态信息的频率最大为70组3Dof或6Dof/秒；当终端设备的刷新能力为60FPS，终端设备可以确定采集第三姿态信息的频率最大为60组3Dof或6Dof信息/秒；当终端设备的解码能力是70FPS，且刷新能力为60FPS时，终端设备可以确定采集第三姿态信息的频率最大为60组3Dof或6Dof信息/秒。The terminal device can adjust the second attitude information according to the decoding capability and/or refresh capability of the terminal device to obtain the third attitude information, and then can send the third attitude information to the server. Correspondingly, the server may receive the third gesture information from the terminal device. The decoding capability may be the fastest decoding speed of the terminal device, and the refresh capability may be the maximum frame rate displayed by the terminal device. That is, the terminal device can determine the maximum frequency of collecting attitude information according to its own decoding ability and/or refresh ability, so as to determine the third attitude information. For example, the decoding capability of the terminal device is 70 (frames per second, FPS), and the refresh capability is 60 FPS. The terminal device can first determine its own decoding capability and/or refresh capability, and then determine a frequency within the range of the decoding capability and/or refresh capability (that is, the maximum frequency range for collecting the third attitude information), and then immediately send the third attitude. information. That is, it can be understood that the frequency at which the terminal device collects the third attitude information needs to be less than or equal to the range of the decoding capability and/or the refresh capability of the terminal device. For example, when the decoding capability of the terminal device is 70FPS, the terminal device can determine that the frequency of collecting the third attitude information is at most 70 sets of 3Dof or 6Dof/second; when the refresh capability of the terminal device is 60FPS, the terminal device can determine to collect the third attitude information. The maximum frequency of information is 60 sets of 3Dof or 6Dof information/second; when the decoding capability of the terminal device is 70FPS and the refresh capability is 60FPS, the terminal device can determine that the frequency of collecting the third attitude information is at most 60 sets of 3Dof or 6Dof information/ second.
进一步地，终端设备也可以根据其它因素调整第三姿态信息的采集频率。终端设备在确定自身发送姿态信息的最大频率之后，可以在最大频率范围内根据需求确定发送姿态信息的频率。即当前显示的图像信息比较卡顿时，可以提高采集姿态信息的频率；当场景不需要显示过于流畅的图像信息时，可以降低采集姿态信息的频率。例如，已知当前采集的姿态信息为40组6Dof信息/秒且姿态信息的最大采集频率为60组6Dof信息/秒，当用户通过屏幕触控操作提高显示的画面流畅度和清晰度时，终端设备可以将采集姿态信息的频率提高至60组6Dof信息/秒。应理解，上述仅是举例说明，并不构成限定。当终端设备在采集姿态信息最大频率范围内根据其它因素调整采集频率时，可以更好地适应当前终端设备的显示需求，从而可以提高用户体验。Further, the terminal device may also adjust the collection frequency of the third attitude information according to other factors. After the terminal device determines the maximum frequency of sending attitude information by itself, it can determine the frequency of sending attitude information within the maximum frequency range according to requirements. That is, when the currently displayed image information is relatively stuck, the frequency of collecting attitude information can be increased; when the scene does not need to display too smooth image information, the frequency of collecting attitude information can be reduced. For example, it is known that the currently collected attitude information is 40 sets of 6Dof information per second and the maximum collection frequency of attitude information is 60 sets of 6Dof information per second. The device can increase the frequency of collecting attitude information to 60 sets of 6Dof information per second. It should be understood that the above is only an example, and does not constitute a limitation. When the terminal device adjusts the collection frequency according to other factors within the maximum frequency range of collecting attitude information, it can better adapt to the display requirements of the current terminal device, thereby improving user experience.
需要说明的是，当终端设备采集到姿态信息之后，应当立即发送采集的姿态信息，可以保证终端设备的采集姿态信息频率和发送姿态信息频率的一致性。此时，服务器渲染与终端设备采集与显示，以及交互的“节奏”均由终端设备的采集“节奏”来确定，从而终端设备可以调整采集姿态信息的频率来适应自身的解码能力和/或刷新能力。It should be noted that, after the terminal device collects the attitude information, it should immediately send the collected attitude information, which can ensure the consistency of the frequency of collecting attitude information and the frequency of sending attitude information. At this time, the "rhythm" of server rendering, terminal device acquisition and display, and interaction is determined by the terminal device's acquisition "rhythm", so that the terminal device can adjust the frequency of collecting attitude information to adapt to its own decoding capabilities and/or refresh ability.
603、当接收到来自终端设备的姿态信息时，服务器根据姿态信息对第一图像信息进行渲染，得到第二图像信息。603. When receiving the gesture information from the terminal device, the server renders the first image information according to the gesture information to obtain the second image information.
当接收到来自终端设备的姿态信息时，服务器可以根据姿态信息对第一图像信息进行渲染，得到第二图像信息。其中，第一图像信息可以为虚拟现实VR图像信息，即第一图像信息可以为当前用户使用的云VR应用的图像信息。服务器可以根据姿态信息确定需要渲染的图像信息(即第一图像信息)，之后可以对第一图像信息进行渲染，得到第二图像信息。When receiving the gesture information from the terminal device, the server may render the first image information according to the gesture information to obtain the second image information. The first image information may be virtual reality VR image information, that is, the first image information may be image information of a cloud VR application used by the current user. The server may determine the image information to be rendered (ie, the first image information) according to the gesture information, and then render the first image information to obtain the second image information.
请参阅图7，图7是本申请实施例公开的另一种图像信息处理的流程示意图。如图7所示，当服务器接收到来自终端设备的姿态信息之后，可以立即触发服务器执行垂直同步。服务器执行垂直同步之后，可以立即开始对第一图像信息进行渲染，得到第二图像信息。服务器可以通过自身的中央处理器(central processing unit，CPU)和图形处理器(graphics processing unit，GPU)对图像信息进行渲染。其中，第二图像信息可以为VR视频帧数据。即可以理解为，终端设备可以通过接收姿态信息被动地触发对第一图像信息的渲染。Please refer to FIG. 7 , which is a schematic flowchart of another image information processing disclosed in an embodiment of the present application. As shown in FIG. 7 , after the server receives the attitude information from the terminal device, the server can be triggered to perform vertical synchronization immediately. After the server performs vertical synchronization, it can immediately start rendering the first image information to obtain the second image information. The server can render image information through its own central processing unit (CPU) and graphics processing unit (graphics processing unit, GPU). The second image information may be VR video frame data. That is, it can be understood that the terminal device can passively trigger the rendering of the first image information by receiving the gesture information.
应理解，当服务器由接收到姿态信息触发垂直同步时，服务器可以根据接收到的姿态信息可以立即对第一图像信息进行图像渲染，从而可以减少图像渲染之前等到周期性触发垂直同步的等待时间，进而可以减少处理时延。进一步地，当图像渲染过程中的时延降低，用户接收到第二图像信息的时延也会降低，从而可以降低终端设备显示第二图像信息的时延，以 提高图像显示的流畅度，进而可以提高用户的使用体验。It should be understood that when the server triggers vertical synchronization by receiving the attitude information, the server can immediately perform image rendering on the first image information according to the received attitude information, thereby reducing the waiting time until the vertical synchronization is periodically triggered before image rendering. In turn, the processing delay can be reduced. Further, when the time delay in the image rendering process is reduced, the time delay for the user to receive the second image information is also reduced, so that the time delay for the terminal device to display the second image information can be reduced, so as to improve the smoothness of the image display, and further. It can improve the user experience.
需要说明的是，服务器被动地通过接收姿态信息触发渲染的过程，终端设备可以降低6～9ms的图像显示时延。It should be noted that the server passively triggers the rendering process by receiving attitude information, and the terminal device can reduce the image display delay of 6-9 ms.
此外，服务器接收到姿态信息之后，可以继续等待接收来自终端设备的姿态信息。In addition, after receiving the gesture information, the server may continue to wait for receiving the gesture information from the terminal device.
服务器可以根据姿态信息确定帧率，之后可以根据帧率对所述第一图像信息进行渲染，得到第二图像信息。服务器可以根据接收到姿态信息的频率确定帧率。例如，当服务器接收到一组3Dof或6Dof的姿态信息时，可以立即对一帧第一图像数据进行渲染，得到一帧第二图像数据。即可以理解为，当服务器接收到的姿态信息的频率可以能是不一致的，接收到的姿态信息的频率可以确定第二图像信息的帧率。当接收到姿态信息的频率越快，对第一图像信息进行渲染的频次也越快，进而得到第二图像信息的帧率越大。反之，当服务器接收到的姿态信息的频率越慢，渲染的频次越慢，则得到第二图像信息的帧率越小。The server may determine the frame rate according to the attitude information, and then render the first image information according to the frame rate to obtain the second image information. The server can determine the frame rate based on how often the gesture information is received. For example, when the server receives a set of 3Dof or 6Dof attitude information, it can immediately render a frame of first image data to obtain a frame of second image data. That is, it can be understood that when the frequency of the gesture information received by the server may be inconsistent, the frequency of the received gesture information can determine the frame rate of the second image information. When the frequency of receiving the attitude information is faster, the frequency of rendering the first image information is also faster, and thus the frame rate of obtaining the second image information is higher. Conversely, when the frequency of attitude information received by the server is slower, the frequency of rendering is slower, and the frame rate of obtaining the second image information is smaller.
应理解，当服务器周期性的进行图像渲染时，服务器固定帧率进行渲染，然而不同的终端设备的解码能力和/或刷新能力不相同，因此，服务器对第一图像信息进行固定帧率渲染，可能会导致解码能力较弱的终端设备来不及解码，和/或，刷新能力较弱的终端设备来不及显示。例如，VR头盔的显示能力只有的60FPS时，当从服务器接收到90FPS规格的第二图像数据时，可能会导致终端设备在显示过程中出现丢帧和跳帧的现象或者一些解码能力和/或刷新能力低的终端设备无法接入固定规格的云VR服务器。因此，终端设备确定发送姿态信息的频率，以及服务器接收到姿态信息时，便可以立即对第一图像信息进行渲染，得到第二图像信息，可以使得服务器渲染图像信息从固定周期变更到由终端设备采集姿态信息的频率来驱动，从而可以使服务器的渲染速度能够与终端设备的解码能力和/或刷新能力相适应。此外，当终端设备进入节电状态(如，静默、息屏等)时，不再继续向服务器发送姿态信息，此时，服务器可以减少触发渲染，从而服务器和终端设备可以同时进入节电状态，以避免服务器一直渲染、编码和发送图像信息等带来的能耗。It should be understood that when the server periodically performs image rendering, the server performs rendering at a fixed frame rate, but different terminal devices have different decoding capabilities and/or refresh capabilities. Therefore, the server performs fixed frame rate rendering on the first image information, It may cause that the terminal device with weak decoding capability cannot decode in time, and/or the terminal device with weak refresh capability cannot display in time. For example, when the display capability of the VR headset is only 60FPS, when the second image data of 90FPS specification is received from the server, it may cause frame loss and frame skipping during the display process of the terminal device, or some decoding capabilities and/or Terminal devices with low refresh capability cannot access the cloud VR server of fixed specifications. Therefore, when the terminal device determines the frequency of sending the attitude information, and when the server receives the attitude information, it can immediately render the first image information to obtain the second image information, so that the server can change the rendered image information from a fixed period to that of the terminal equipment. It is driven by the frequency of collecting attitude information, so that the rendering speed of the server can be adapted to the decoding capability and/or refresh capability of the terminal device. In addition, when the terminal device enters a power-saving state (such as silent, screen-off, etc.), it does not continue to send gesture information to the server. At this time, the server can reduce the trigger rendering, so that the server and the terminal device can enter the power-saving state at the same time. In order to avoid the energy consumption caused by the server rendering, encoding and sending image information all the time.
604、服务器向终端设备发送第二图像信息。604. The server sends the second image information to the terminal device.
当服务器得到第二图像信息之后，可以向终端设备发送第二图像信息。对应地，终端设备可以接收来自服务器的第二图像信息。After the server obtains the second image information, it can send the second image information to the terminal device. Correspondingly, the terminal device may receive the second image information from the server.
当服务器得到第二图像信息之后，可以立即向终端设备发送第二图像信息，可以保证终端设备采集姿态信息和显示图像信息，以及服务器渲染图像信息的“节奏”的一致性。After the server obtains the second image information, it can immediately send the second image information to the terminal device, which can ensure the consistency of the terminal device's collection of attitude information and display image information, and the "rhythm" of the server rendering the image information.
如图7所示，当服务器得到第二图像信息(例如，VR画面信息)之后，可以通过编码器对第二图像信息进行编码，得到第三图像信息(例如，VR视频帧信息)。之后服务器可以向终端设备发送第三图像信息。对应地，终端设备可以接收来自服务器的第三图像信息，之后可以通过解码器对第三图像信息进行解码，得到第二图像信息。As shown in FIG. 7 , after the server obtains the second image information (eg, VR picture information), the second image information may be encoded by an encoder to obtain third image information (eg, VR video frame information). Afterwards, the server may send the third image information to the terminal device. Correspondingly, the terminal device can receive the third image information from the server, and then can decode the third image information through the decoder to obtain the second image information.
当服务器向终端设备发送第二图像信息之后，可以继续等待姿态信息，并进行对第一图像信息渲染。After the server sends the second image information to the terminal device, it may continue to wait for the gesture information, and render the first image information.
605、终端设备显示第二图像信息。605. The terminal device displays the second image information.
当终端设备得到第二图像信息之后，可以显示第二图像信息。其中，第二图像信息可以是一幅VR画面，终端设备可以通过显示屏显示第二图像信息。After the terminal device obtains the second image information, the second image information can be displayed. The second image information may be a VR picture, and the terminal device may display the second image information through the display screen.
在显示第二图像信息之前，终端设备还可以对第二图像信息的像素进行处理。其中，像素处理可以包括异步扭曲等。Before displaying the second image information, the terminal device may further process the pixels of the second image information. Among them, pixel processing may include asynchronous warping and the like.
终端设备显示第二图像信息之后，可以继续等待接收来自服务器的第二图像信息。After displaying the second image information, the terminal device may continue to wait for receiving the second image information from the server.
当检测到终端设备处于未使用状态时，终端设备也可以停止显示第二图像信息。其中， 未使用状态可以包括终端设备的屏幕处于息屏状态和/或终端设备未被用户佩戴。当检测到终端设备处于使用状态时，终端设备可以继续显示第二图像信息。即可以理解为，终端设备可以检测自身是否处于未使用状态。使用状态包括屏幕处于显示状态和/或终端设备已被用户重新佩戴上。When it is detected that the terminal device is in an unused state, the terminal device may also stop displaying the second image information. The unused state may include that the screen of the terminal device is in an off-screen state and/or the terminal device is not worn by the user. When it is detected that the terminal device is in a use state, the terminal device may continue to display the second image information. That is, it can be understood that the terminal device can detect whether it is in an unused state. The usage state includes that the screen is in a display state and/or the terminal device has been re-worn by the user.
应理解，当用户不使用终端设备的时候，终端设备继续显示第二图像信息会造成处理资源和能量的浪费。因此，当终端设备检测到用户暂时不使用终端设备时，可以停止显示第二图像信息，从而可以节约能源和处理资源。It should be understood that, when the user does not use the terminal device, the terminal device continues to display the second image information, which will cause waste of processing resources and energy. Therefore, when the terminal device detects that the user temporarily does not use the terminal device, it can stop displaying the second image information, thereby saving energy and processing resources.
当检测到用户结束VR应用时，终端设备可以向服务器发送第二请求。其中，用户可以通过屏幕触控或按钮按键等方法结束当前云VR应用。对应地，服务器可以接收来自终端设备的第二请求，之后可以根据第二请求结束当前连接，以及结束当前各个功能组件的对应的云VR业务的任务。When detecting that the user ends the VR application, the terminal device may send a second request to the server. Among them, the user can end the current cloud VR application by touching the screen or pressing a button. Correspondingly, the server may receive the second request from the terminal device, and then may end the current connection according to the second request, and end the task of the current cloud VR service corresponding to each functional component.
当服务器与终端设备断开连接之后，终端设备可以结束VR应用各个功能组件的任务。After the server is disconnected from the terminal device, the terminal device can end the tasks of each functional component of the VR application.
应理解，终端设备采集姿态信息后，向服务器发送姿态信息。服务器接收到姿态信息之后，对第一图像信息进行渲染，得到第二图像信息，并向终端设备发送第二图像信息。终端设备接收到来自服务器的图像信息之后，显示第二图像信息，并且可以继续采集图像信息。由此，上述的过程形成了VR业务的循环，当用户暂停使用终端设备时，可以暂停当前VR业务循环；当用户恢复使用终端设备时，可以恢复当前VR业务循环。当开启VR业务时，可以开启VR业务循环，当结束VR业务时，可以结束VR业务循环。It should be understood that after collecting the attitude information, the terminal device sends the attitude information to the server. After receiving the gesture information, the server renders the first image information to obtain the second image information, and sends the second image information to the terminal device. After receiving the image information from the server, the terminal device displays the second image information, and can continue to collect image information. Thus, the above process forms a VR service cycle. When the user suspends using the terminal device, the current VR service cycle can be suspended; when the user resumes using the terminal device, the current VR service cycle can be resumed. When the VR service is started, the VR service cycle can be started, and when the VR service is ended, the VR service cycle can be ended.
基于上述网络架构，请参阅图8，图8是本申请实施例公开的另一种信息处理方法的流程示意图。其中，本申请实施例中由终端设备执行的功能也可以由终端设备中的模块(例如，芯片)来执行，由服务器执行的功能也可以由服务器中的模块(例如，芯片)来执行。如图8所示，该信息处理方法可以包括以下步骤。Based on the above network architecture, please refer to FIG. 8 , which is a schematic flowchart of another information processing method disclosed by an embodiment of the present application. The functions performed by the terminal device in the embodiments of the present application may also be performed by a module (eg, a chip) in the terminal device, and the function performed by the server may also be performed by a module (eg, a chip) in the server. As shown in FIG. 8 , the information processing method may include the following steps.
801、终端设备启动VR应用。801. The terminal device starts the VR application.
802、终端设备向服务器发送第一请求。802. The terminal device sends a first request to the server.
803、服务器启动VR业务。803. The server starts the VR service.
其中，步骤801-步骤803可以参考步骤601的相关描述。Wherein, step 801-step 803 may refer to the relevant description of step 601 .
应理解，当终端设备尚未开启VR应用时，需要先启动VR业务，才能够进入VR业务循环的过程。It should be understood that, when the VR application has not been started on the terminal device, the VR service needs to be started before entering the process of the VR service cycle.
804、终端设备采集姿态信息。804. The terminal device collects attitude information.
其中，步骤804可以参考步骤601的相关描述，此处不加赘述。Wherein, for step 804, reference may be made to the relevant description of step 601, and details are not repeated here.
805、终端设备向服务器发送姿态信息。805. The terminal device sends gesture information to the server.
其中，步骤805可以参考步骤602的相关描述，此处不加赘述。Wherein, for step 805, reference may be made to the relevant description of step 602, and details are not repeated here.
806、服务器根据姿态信息对第一图像信息进行渲染，得到第二图像信息。806. The server renders the first image information according to the gesture information to obtain second image information.
其中，步骤806可以参考步骤603的相关描述，此处不加赘述。Wherein, for step 806, reference may be made to the relevant description of step 603, and details are not repeated here.
807、服务器对第二图像信息进行编码，得到第三图像信息。807. The server encodes the second image information to obtain third image information.
808、服务器向终端设备发送第三图像信息。808. The server sends the third image information to the terminal device.
809、终端设备对第三图像信息进行解码，得到第二图像信息。809. The terminal device decodes the third image information to obtain the second image information.
其中，步骤807-步骤809可以参考步骤604的相关描述，此处不加赘述。Wherein, for steps 807-809, reference may be made to the relevant description of step 604, and details are not repeated here.
810、终端设备显示第二图像信息。810. The terminal device displays the second image information.
其中，步骤810可以参考步骤605的相关描述。For step 810, reference may be made to the relevant description of step 605.
应理解，当VR业务启动之后，可以进入业务循环，即步骤804-步骤810可以重复执行，当终端设备得到第二图像信息并进行显示时，可以继续采集姿态信息。服务器对第二图像信息进行编码，得到第三图像信息之后，可以继续等待接收来自终端设备的姿态信息。It should be understood that after the VR service is started, a service loop may be entered, that is, steps 804 to 810 may be repeatedly performed, and when the terminal device obtains and displays the second image information, it may continue to collect gesture information. After the server encodes the second image information and obtains the third image information, it can continue to wait for receiving the gesture information from the terminal device.
811、当终端设备处于未使用状态时，停止采集姿态信息。811. When the terminal device is in an unused state, stop collecting attitude information.
812、当终端设备处于使用状态时，采集姿态信息。812. Collect attitude information when the terminal device is in a use state.
其中，步骤811和步骤812可以参考步骤601的相关描述。Wherein, step 811 and step 812 may refer to the relevant description of step 601 .
应理解，步骤811为终端设备发生息屏事件或者用户摘下VR终端设备时执行的动作，步骤812为终端设备从息屏变为屏幕显示或者用户佩戴上VR终端设备时执行的动作。步骤811可以暂停上述VR业务循环的过程；步骤812可以恢复上述VR业务循环的过程。It should be understood that step 811 is an action performed when a screen-on-screen event occurs on the terminal device or the user takes off the VR terminal device, and step 812 is an action performed when the terminal device changes from a screen-on-screen to a screen display or when the user wears the VR terminal device. Step 811 may suspend the above-mentioned process of the VR service loop; and step 812 may resume the above-mentioned process of the VR service loop.
813、终端设备结束VR应用。813. The terminal device ends the VR application.
814、终端设备向服务器发送第二请求。814. The terminal device sends a second request to the server.
815、服务器结束VR业务。815. The server terminates the VR service.
其中，步骤813-步骤815可以参考步骤605的相关描述。Wherein, step 813-step 815 may refer to the relevant description of step 605 .
应理解，当终端设备和服务器结束VR业务时，会结束VR业务循环的过程。It should be understood that when the terminal device and the server terminate the VR service, the process of the VR service cycle will be terminated.
应理解，图6和图8的信息处理方法是一种交互过程中通用的过程，并不局限于云VR，它可以适用于实施互动场景，例如，云扩展现实(extended reality，XR),云游戏等。其中云XR可以包括云VR、云增强现实(augmented reality，AR)、云混合现实(mixed reality，MR)、云全息现实(holography reality，HR)等。此处不构成限定。It should be understood that the information processing method of FIG. 6 and FIG. 8 is a general process in an interactive process, and is not limited to cloud VR, it can be applied to implement interactive scenarios, for example, cloud extended reality (XR), cloud VR games etc. The cloud XR may include cloud VR, cloud augmented reality (AR), cloud mixed reality (MR), cloud holography reality (HR), and the like. This is not a limitation.
基于上述网络架构，请参阅图9，图9是本发明实施例公开的一种信息处理装置的结构示意图。如图9所示，该信息处理装置可以包括：Based on the above network architecture, please refer to FIG. 9 , which is a schematic structural diagram of an information processing apparatus disclosed in an embodiment of the present invention. As shown in FIG. 9, the information processing apparatus may include:
渲染单元901，用于当接收到来自终端设备的姿态信息时，根据所述姿态信息对第一图像信息进行渲染，得到第二图像信息，所述姿态信息是用户的姿态信息，所述第一图像信息为虚拟现实VR图像信息；The rendering unit 901 is configured to render the first image information according to the attitude information when receiving the attitude information from the terminal device, and obtain the second image information, the attitude information is the attitude information of the user, the first image information The image information is virtual reality VR image information;
发送单元902，用于向所述终端设备发送所述第二图像信息。The sending unit 902 is configured to send the second image information to the terminal device.
作为一种可能的实施方式，所述渲染单元901具体用于：As a possible implementation manner, the rendering unit 901 is specifically used for:
根据所述姿态信息确定帧率；determine the frame rate according to the attitude information;
根据所述帧率对所述第一图像信息进行渲染，得到第二图像信息。The first image information is rendered according to the frame rate to obtain second image information.
作为一种可能的实施方式，所述渲染单元901根据所述姿态信息确定帧率包括：As a possible implementation manner, the rendering unit 901 determining the frame rate according to the gesture information includes:
根据接收到所述姿态信息的频率确定所述帧率。The frame rate is determined according to the frequency at which the attitude information is received.
作为一种可能的实施方式，该装置还可以包括：As a possible implementation manner, the device may also include:
编码单元903，用于对所述第二图像信息进行编码，得到第三图像信息；an encoding unit 903, configured to encode the second image information to obtain third image information;
所述发送单元902，具体用于向所述终端设备发送所述第三图像信息。The sending unit 902 is specifically configured to send the third image information to the terminal device.
有关上述渲染单元901、发送单元902和编码单元903更详细的描述可以直接参考上述图6和图8所示的方法实施例中服务器的相关描述直接得到，此处不加赘述。More detailed descriptions of the foregoing rendering unit 901 , sending unit 902 and encoding unit 903 can be obtained directly by referring to the relevant descriptions of the servers in the method embodiments shown in FIG. 6 and FIG. 8 , and details are not repeated here.
基于上述网络架构，请参阅图10，图10是本发明实施例公开的另一种信息处理装置的结构示意图。如图10所示，该信息处理装置可以包括：Based on the above network architecture, please refer to FIG. 10 , which is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention. As shown in FIG. 10, the information processing apparatus may include:
采集单元1001，用于当检测到终端设备处于未使用状态时，停止采集第一姿态信息，所述未使用状态包括所述终端设备的屏幕处于息屏状态和/或所述终端设备未被用户佩戴，所述第一姿态信息是用户的姿态信息。A collection unit 1001, configured to stop collecting the first attitude information when it is detected that the terminal device is in an unused state, where the unused state includes that the screen of the terminal device is in an off-screen state and/or the terminal device is not used by a user When wearing, the first posture information is the posture information of the user.
作为一种可能的实施方式，该装置还可以包括：As a possible implementation manner, the device may also include:
发送单元1002，用于当采集到第二姿态信息时，向所述服务器发送所述第二姿态信息；A sending unit 1002, configured to send the second attitude information to the server when the second attitude information is collected;
接收单元1003，用于接收来自所述服务器的所述第二图像信息；a receiving unit 1003, configured to receive the second image information from the server;
显示单元1004，用于显示所述第二图像信息。The display unit 1004 is configured to display the second image information.
作为一种可能的实施方式，所述发送单元1002具体用于：As a possible implementation manner, the sending unit 1002 is specifically configured to:
根据终端设备的解码能力和/或刷新能力调整所述第二姿态信息，得到第三姿态信息，所述解码能力为所述终端设备解码的最快速度，所述刷新能力为所述终端设备显示的最大帧率；Adjust the second attitude information according to the decoding capability and/or refresh capability of the terminal device to obtain the third attitude information, where the decoding capability is the fastest decoding speed of the terminal device, and the refresh capability is the display capability of the terminal device the maximum frame rate;
向所述服务器发送所述第三姿态信息。Sending the third gesture information to the server.
作为一种可能的实施方式，所述接收单元1003具体用于：As a possible implementation manner, the receiving unit 1003 is specifically configured to:
接收来自所述服务器的第三图像信息；receiving third image information from the server;
对所述第三图像信息进行解码，得到所述第二图像信息。Decoding the third image information to obtain the second image information.
作为一种可能的实施方式，所述采集单元1001，还用于当检测到终端设备处于使用状态时，采集所述第四姿态信息，所述使用状态包括屏幕处于显示状态和/或所述终端设备已被用户佩戴。As a possible implementation manner, the collection unit 1001 is further configured to collect the fourth posture information when it is detected that the terminal device is in a use state, and the use state includes that the screen is in a display state and/or the terminal device is in a display state. The device has been worn by the user.
有关上述采集单元1001、发送单元1002、接收单元1003和显示单元1004更详细的描述可以直接参考上述图6和图8所示的方法实施例中终端设备的相关描述直接得到，此处不加赘述。More detailed descriptions of the above collection unit 1001, sending unit 1002, receiving unit 1003, and display unit 1004 can be obtained directly by referring to the relevant descriptions of the terminal devices in the method embodiments shown in FIG. 6 and FIG. 8, and will not be repeated here. .
基于上述网络架构，请参阅图11，图11是本发明实施例公开的又一种信息处理装置的结构示意图。如图11所示，该信息处理装置可以包括处理器1101、存储器1102、输入接口1103、输出接口1104和总线1105。存储器1102可以是独立存在的，可以通过总线1105与处理器1101相连接。存储器1102也可以和处理器1101集成在一起。其中，总线1105用于实现这些组件之间的连接。Based on the above network architecture, please refer to FIG. 11 . FIG. 11 is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention. As shown in FIG. 11 , the information processing apparatus may include a processor 1101 , a memory 1102 , an input interface 1103 , an output interface 1104 and a bus 1105 . The memory 1102 may exist independently, and may be connected to the processor 1101 through the bus 1105 . The memory 1102 may also be integrated with the processor 1101 . Among them, the bus 1105 is used to realize the connection between these components.
在一个实施例中，该信息处理装置可以为服务器或者服务器内的模块(例如，芯片)，存储器1102中存储的计算机程序指令被执行时，该处理器1101用于控制发送单元902执行上述实施例中执行的操作，该处理器1101还用于执行渲染单元901和编码单元903上述实施例中执行的操作，输出接口1104用于执行上述实施例中发送单元902执行的操作。上述服务器或者服务器内的模块还可以用于执行上述图6和图8方法实施例中服务器执行的各种方法，不再赘述。In one embodiment, the information processing apparatus may be a server or a module (eg, a chip) in the server, and when the computer program instructions stored in the memory 1102 are executed, the processor 1101 is configured to control the sending unit 902 to execute the above-mentioned embodiments The processor 1101 is further configured to perform the operations performed by the rendering unit 901 and the encoding unit 903 in the foregoing embodiments, and the output interface 1104 is configured to perform the operations performed by the sending unit 902 in the foregoing embodiments. The foregoing server or modules within the server may also be used to execute various methods executed by the server in the foregoing method embodiments in FIG. 6 and FIG. 8 , which will not be described again.
在一个实施例中，该信息处理装置可以为终端设备或者终端设备内的模块(例如，芯片)，存储器1102中存储的计算机程序指令被执行时，该处理器1101用于控制接收单元1003和发送单元1002执行上述实施例中执行的操作，该处理器901还用于执行采集单元1001和显示单元1004上述实施例中执行的操作，输入接口1103用于执行上述实施例中接收单元1003执行的操作，输出接口1104用于执行上述实施例中发送单元1002执行的操作。上述终端设备或者终端设备内的模块还可以用于执行上述图6和图8方法实施例中终端设备执行的各种方法，不再赘述。In one embodiment, the information processing apparatus may be a terminal device or a module (eg, a chip) in the terminal device. When the computer program instructions stored in the memory 1102 are executed, the processor 1101 is used to control the receiving unit 1003 and send the The unit 1002 performs the operations performed in the above-mentioned embodiments, the processor 901 is further configured to perform the operations performed by the acquisition unit 1001 and the display unit 1004 in the above-mentioned embodiments, and the input interface 1103 is used to perform the operations performed by the receiving unit 1003 in the above-mentioned embodiments. , and the output interface 1104 is used to perform the operations performed by the sending unit 1002 in the above embodiments. The foregoing terminal device or modules in the terminal device may also be used to execute various methods performed by the terminal device in the foregoing method embodiments in FIG. 6 and FIG. 8 , which will not be described again.
基于上述网络架构，请参阅图12，图12是本发明实施例公开的又一种信息处理装置的结构示意图。如图12所示，该信息处理装置可以包括输入接口1201、逻辑电路1202和输出接口1203。输入接口1201与输出接口1203通过逻辑电路1202相连接。其中，输入接口1201用于接收来自其它装置的信息，输出接口1203用于向其它装置输出、调度或者发送信息。逻辑电路1202用于执行除输入接口1201与输出接口1203的操作之外的操作，例如实现上述实施例中处理器1101实现的功能。其中，该信息处理装置可以为终端设备或者终端设备内的模块，也可以为服务器或者服务器内的模块。其中，有关输入接口1201、逻辑电路1202和输出接口1203 更详细的描述可以直接参考上述方法实施例中终端设备或服务器的相关描述直接得到，这里不加赘述。Based on the above network architecture, please refer to FIG. 12 , which is a schematic structural diagram of another information processing apparatus disclosed in an embodiment of the present invention. As shown in FIG. 12 , the information processing apparatus may include an input interface 1201 , a logic circuit 1202 and an output interface 1203 . The input interface 1201 and the output interface 1203 are connected through the logic circuit 1202 . The input interface 1201 is used for receiving information from other devices, and the output interface 1203 is used for outputting, scheduling or sending information to other devices. The logic circuit 1202 is configured to perform operations other than the operations of the input interface 1201 and the output interface 1203, for example, to implement the functions implemented by the processor 1101 in the above-mentioned embodiment. The information processing apparatus may be a terminal device or a module in the terminal device, or may be a server or a module in the server. The more detailed description about the input interface 1201 , the logic circuit 1202 and the output interface 1203 can be obtained directly by referring to the relevant description of the terminal device or the server in the above method embodiments, which will not be repeated here.
本发明实施例还公开一种计算机可读存储介质，其上存储有指令，该指令被执行时执行上述方法实施例中的方法。The embodiment of the present invention also discloses a computer-readable storage medium, on which an instruction is stored, and when the instruction is executed, the method in the foregoing method embodiment is performed.
本发明实施例还公开一种包括指令的计算机程序产品，该指令被执行时执行上述方法实施例中的方法。The embodiment of the present invention also discloses a computer program product including an instruction, when the instruction is executed, the method in the foregoing method embodiment is performed.
本发明实施例还公开一种信息处理***，该信息处理***包括终端设备和服务器，具体描述可以参考图6和图8所示的信息处理方法。An embodiment of the present invention also discloses an information processing system, the information processing system includes a terminal device and a server, and for a specific description, reference may be made to the information processing methods shown in FIG. 6 and FIG. 8 .
以上所述的具体实施方式，对本申请的目的、技术方案和有益效果进行了进一步详细说明，所应理解的是，以上所述仅为本申请的具体实施方式而已，并不用于限定本申请的保护范围，凡在本申请的技术方案的基础之上，所做的任何修改、等同替换、改进等，均应包括在本申请的保护范围之内。The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above descriptions are only specific embodiments of the present application, and are not intended to limit the The protection scope, any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present application shall be included within the protection scope of the present application.

Claims (22)

  1. 一种信息处理方法，应用于服务器，其特征在于，所述方法包括：An information processing method, applied to a server, characterized in that the method comprises:
    当接收到来自终端设备的姿态信息时，根据所述姿态信息对第一图像信息进行渲染，得到第二图像信息，所述姿态信息是用户的姿态信息，所述第一图像信息为虚拟现实VR图像信息；When receiving the gesture information from the terminal device, the first image information is rendered according to the gesture information to obtain the second image information, the gesture information is the user's gesture information, and the first image information is the virtual reality VR image information;
    向所述终端设备发送所述第二图像信息。The second image information is sent to the terminal device.
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述姿态信息对第一图像信息进行渲染，得到第二图像信息包括：The method according to claim 1, wherein the rendering of the first image information according to the posture information to obtain the second image information comprises:
    根据所述姿态信息确定帧率；determine the frame rate according to the attitude information;
    根据所述帧率对所述第一图像信息进行渲染，得到第二图像信息。The first image information is rendered according to the frame rate to obtain second image information.
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述姿态信息确定帧率包括：The method according to claim 2, wherein the determining the frame rate according to the gesture information comprises:
    根据接收到所述姿态信息的频率确定所述帧率。The frame rate is determined according to the frequency at which the attitude information is received.
  4. 根据权利要求1-3任一项所述的方法，其特征在于，所述方法还包括：The method according to any one of claims 1-3, wherein the method further comprises:
    对所述第二图像信息进行编码，得到第三图像信息；Encoding the second image information to obtain third image information;
    所述向所述终端设备发送所述第二图像信息包括：The sending the second image information to the terminal device includes:
    向所述终端设备发送所述第三图像信息。Sending the third image information to the terminal device.
  5. 一种信息处理方法，应用于终端设备，其特征在于，所述方法包括：An information processing method, applied to a terminal device, characterized in that the method comprises:
    当检测到终端设备处于未使用状态时，停止采集第一姿态信息，所述未使用状态包括所述终端设备的屏幕处于息屏状态和/或所述终端设备未被用户佩戴，所述第一姿态信息是用户的姿态信息。When it is detected that the terminal device is in an unused state, the collection of the first posture information is stopped. The unused state includes that the screen of the terminal device is in an off-screen state and/or the terminal device is not worn by the user. The gesture information is the gesture information of the user.
  6. 根据权利要求5所述的方法，其特征在于，所述方法还包括：The method according to claim 5, wherein the method further comprises:
    当采集到第二姿态信息时，向所述服务器发送所述第二姿态信息；When the second attitude information is collected, sending the second attitude information to the server;
    接收来自所述服务器的所述第二图像信息；receiving the second image information from the server;
    显示所述第二图像信息。The second image information is displayed.
  7. 根据权利要求6所述的方法，其特征在于，所述向所述服务器发送所述第二姿态信息包括：The method according to claim 6, wherein the sending the second gesture information to the server comprises:
    根据终端设备的解码能力和/或刷新能力调整所述第二姿态信息，得到第三姿态信息，所述解码能力为所述终端设备解码的最快速度，所述刷新能力为所述终端设备显示的最大帧率；Adjust the second attitude information according to the decoding capability and/or refresh capability of the terminal device to obtain the third attitude information, where the decoding capability is the fastest decoding speed of the terminal device, and the refresh capability is the display capability of the terminal device the maximum frame rate;
    向所述服务器发送所述第三姿态信息。Sending the third gesture information to the server.
  8. 根据权利要求6或7所述的方法，其特征在于，所述接收来自所述服务器的所述第二图像信息包括：The method according to claim 6 or 7, wherein the receiving the second image information from the server comprises:
    接收来自所述服务器的第三图像信息；receiving third image information from the server;
    对所述第三图像信息进行解码，得到所述第二图像信息。Decoding the third image information to obtain the second image information.
  9. 根据权利要求5-8任一项所述的方法，其特征在于，所述方法还包括：The method according to any one of claims 5-8, wherein the method further comprises:
    当检测到终端设备处于使用状态时，采集所述第四姿态信息，所述使用状态包括屏幕处于显示状态和/或所述终端设备已被用户佩戴。When it is detected that the terminal device is in a use state, the fourth gesture information is collected, and the use state includes that the screen is in a display state and/or the terminal device is worn by the user.
  10. 一种信息处理装置，所述装置为服务器，其特征在于，所述装置包括：An information processing device, wherein the device is a server, characterized in that the device comprises:
    渲染单元，用于当接收到来自终端设备的姿态信息时，根据所述姿态信息对第一图像信 息进行渲染，得到第二图像信息，所述姿态信息是用户的姿态信息，所述第一图像信息为虚拟现实VR图像信息；a rendering unit, configured to render the first image information according to the attitude information when receiving the attitude information from the terminal device, and obtain the second image information, the attitude information is the attitude information of the user, the first image The information is virtual reality VR image information;
    发送单元，用于向所述终端设备发送所述第二图像信息。A sending unit, configured to send the second image information to the terminal device.
  11. 根据权利要求10所述的装置，其特征在于，所述渲染单元具体用于：The apparatus according to claim 10, wherein the rendering unit is specifically configured to:
    根据所述姿态信息确定帧率；determine the frame rate according to the attitude information;
    根据所述帧率对所述第一图像信息进行渲染，得到第二图像信息。The first image information is rendered according to the frame rate to obtain second image information.
  12. 根据权利要求11所述的装置，其特征在于，所述渲染单元根据所述姿态信息确定帧率包括：The apparatus according to claim 11, wherein the rendering unit determining the frame rate according to the gesture information comprises:
    根据接收到所述姿态信息的频率确定所述帧率。The frame rate is determined according to the frequency at which the attitude information is received.
  13. 根据权利要求10-12任一项所述的装置，其特征在于，所述装置还包括：The device according to any one of claims 10-12, wherein the device further comprises:
    编码单元，用于对所述第二图像信息进行编码，得到第三图像信息；an encoding unit, configured to encode the second image information to obtain third image information;
    所述发送单元，具体用于向所述终端设备发送所述第三图像信息。The sending unit is specifically configured to send the third image information to the terminal device.
  14. 一种信息处理装置，所述装置为终端设备，其特征在于，所述装置包括：An information processing apparatus, the apparatus is a terminal device, characterized in that, the apparatus comprises:
    采集单元，用于当检测到终端设备处于未使用状态时，停止采集第一姿态信息，所述未使用状态包括所述终端设备的屏幕处于息屏状态和/或所述终端设备未被用户佩戴，所述第一姿态信息是用户的姿态信息。A collection unit, configured to stop collecting the first attitude information when it is detected that the terminal device is in an unused state, where the unused state includes that the screen of the terminal device is in an off-screen state and/or the terminal device is not worn by the user , the first gesture information is the user's gesture information.
  15. 根据权利要求14所述的装置，其特征在于，所述装置还包括：The apparatus of claim 14, wherein the apparatus further comprises:
    发送单元，用于当采集到第二姿态信息时，向所述服务器发送所述第二姿态信息；a sending unit, configured to send the second attitude information to the server when the second attitude information is collected;
    接收单元，用于接收来自所述服务器的所述第二图像信息；a receiving unit, configured to receive the second image information from the server;
    显示单元，用于显示所述第二图像信息。The display unit is used for displaying the second image information.
  16. 根据权利要求15所述的装置，其特征在于，所述发送单元具体用于：The apparatus according to claim 15, wherein the sending unit is specifically configured to:
    根据终端设备的解码能力和/或刷新能力调整所述第二姿态信息，得到第三姿态信息，所述解码能力为所述终端设备解码的最快速度，所述刷新能力为所述终端设备显示的最大帧率；Adjust the second attitude information according to the decoding capability and/or refresh capability of the terminal device to obtain the third attitude information, where the decoding capability is the fastest decoding speed of the terminal device, and the refresh capability is the display capability of the terminal device the maximum frame rate;
    向所述服务器发送所述第三姿态信息。Sending the third gesture information to the server.
  17. 根据权利要求15或16所述的装置，其特征在于，所述接收单元具体用于：The device according to claim 15 or 16, wherein the receiving unit is specifically configured to:
    接收来自所述服务器的第三图像信息；receiving third image information from the server;
    对所述第三图像信息进行解码，得到所述第二图像信息。Decoding the third image information to obtain the second image information.
  18. 根据权利要求14-17任一项所述的装置，其特征在于，所述采集单元，还用于当检测到终端设备处于使用状态时，采集所述第四姿态信息，所述使用状态包括屏幕处于显示状态和/或所述终端设备已被用户佩戴。The apparatus according to any one of claims 14-17, wherein the collection unit is further configured to collect the fourth posture information when it is detected that the terminal device is in a use state, and the use state includes a screen is in the displayed state and/or the terminal device has been worn by the user.
  19. 一种信息处理装置，所述装置为服务器，其特征在于，包括处理器、存储器、输入接口和输出接口，所述输入接口用于接收来自所述信息处理装置之外的其它装置的信息，所述输出接口用于向所述信息处理装置之外的其它装置输出信息，所述处理器调用所述存储器中存储的计算机程序实现如权利要求1-4任一项所述的方法。An information processing device, the device is a server, characterized in that it includes a processor, a memory, an input interface and an output interface, the input interface is used to receive information from other devices other than the information processing device, and the The output interface is used for outputting information to other devices other than the information processing device, and the processor invokes a computer program stored in the memory to implement the method according to any one of claims 1-4.
  20. 一种信息处理装置，所述装置为终端设备，其特征在于，包括处理器、存储器、输入接口和输出接口，所述输入接口用于接收来自所述信息处理装置之外的其它装置的信息，所述输出接口用于向所述信息处理装置之外的其它装置输出信息，所述处理器调用所述存储器中存储的计算机程序实现如权利要求5-9任一项所述的方法。An information processing device, the device is a terminal device, characterized in that it comprises a processor, a memory, an input interface and an output interface, the input interface is used to receive information from other devices other than the information processing device, The output interface is used for outputting information to other devices than the information processing device, and the processor invokes a computer program stored in the memory to implement the method according to any one of claims 5-9.
  21. 一种信息处理***，其特征在于，所述信息处理***包括如权利要求19所述的装置 以及如权利要求20所述的装置。An information processing system, characterized in that the information processing system includes the apparatus of claim 19 and the apparatus of claim 20.
  22. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有计算机程序或计算机指令，当所述计算机程序或计算机指令被运行时，实现如权利要求1-9任一项所述的方法。A computer-readable storage medium, characterized in that a computer program or computer instruction is stored in the computer-readable storage medium, and when the computer program or computer instruction is executed, any one of claims 1-9 is realized. the method described.
PCT/CN2022/085663 2021-04-14 2022-04-07 Information processing method, apparatus and system WO2022218209A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110402796.3A CN115209178A (en) 2021-04-14 2021-04-14 Information processing method, device and system
CN202110402796.3 2021-04-14

Publications (1)

Publication Number Publication Date
WO2022218209A1 true WO2022218209A1 (en) 2022-10-20

Family

ID=83573808

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/085663 WO2022218209A1 (en) 2021-04-14 2022-04-07 Information processing method, apparatus and system

Country Status (2)

Country Link
CN (1) CN115209178A (en)
WO (1) WO2022218209A1 (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180224930A1 (en) * 2015-08-04 2018-08-09 Board Of Regents Of The Nevada System Of Higher Education, On Behalf Of The University Of Nevada, Immersive virtual reality locomotion using head-mounted motion sensors
CN110070812A (en) * 2019-04-12 2019-07-30 华为技术有限公司 Wear electronic equipment and its control method
CN110868581A (en) * 2018-08-28 2020-03-06 华为技术有限公司 Image display method, device and system
CN111298427A (en) * 2020-02-04 2020-06-19 南京大学 Method for reducing picture jitter in virtual reality cloud game system
CN111563027A (en) * 2020-04-30 2020-08-21 北京视博云信息技术有限公司 Application operation monitoring method, device and system
CN111831353A (en) * 2020-07-09 2020-10-27 平行云科技(北京)有限公司 OpenXR standard-based runtime library, data interaction method, device and medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180224930A1 (en) * 2015-08-04 2018-08-09 Board Of Regents Of The Nevada System Of Higher Education, On Behalf Of The University Of Nevada, Immersive virtual reality locomotion using head-mounted motion sensors
CN110868581A (en) * 2018-08-28 2020-03-06 华为技术有限公司 Image display method, device and system
CN110070812A (en) * 2019-04-12 2019-07-30 华为技术有限公司 Wear electronic equipment and its control method
CN111298427A (en) * 2020-02-04 2020-06-19 南京大学 Method for reducing picture jitter in virtual reality cloud game system
CN111563027A (en) * 2020-04-30 2020-08-21 北京视博云信息技术有限公司 Application operation monitoring method, device and system
CN111831353A (en) * 2020-07-09 2020-10-27 平行云科技(北京)有限公司 OpenXR standard-based runtime library, data interaction method, device and medium

Also Published As

Publication number Publication date
CN115209178A (en) 2022-10-18

Similar Documents

Publication Publication Date Title
Liu et al. Cutting the cord: Designing a high-quality untethered vr system with low latency remote rendering
US11303881B2 (en) Method and client for playing back panoramic video
Cai et al. Toward gaming as a service
US10229651B2 (en) Variable refresh rate video capture and playback
CN105791977B (en) Virtual reality data processing method, equipment and system based on cloud service
WO2019074313A1 (en) Method and apparatus for rendering three-dimensional content
RU2627222C2 (en) Power efficient content transfer over wireless connection
US20110199304A1 (en) Systems and Methods for Providing Enhanced Motion Detection
WO2018076939A1 (en) Video file processing method and apparatus
CN205068298U (en) Interaction system is wandered to three -dimensional scene
JP2018521550A (en) Method, client and computer storage medium for playing video
CN111298427A (en) Method for reducing picture jitter in virtual reality cloud game system
CN112367542A (en) Terminal playing system and method for mirror image screen projection
WO2020093843A1 (en) Distributed multi-terminal and multi-network supporting system for android online game
CN113273219A (en) Distributed video and graphics rendering system
US20170221174A1 (en) Gpu data sniffing and 3d streaming system and method
CN114554277B (en) Multimedia processing method, device, server and computer readable storage medium
KR20200133127A (en) System for providing real-tim immersive content and method for transmitting haptic effect thereof
CN108271056B (en) Video interaction method, user client, server and storage medium
WO2022218209A1 (en) Information processing method, apparatus and system
CN112565883A (en) Video rendering processing system and computer equipment for virtual reality scene
CN111901628A (en) Cloud rendering method based on zSpace desktop VR all-in-one machine
JP5268991B2 (en) Information processing apparatus and video content reproduction method
WO2023093792A1 (en) Image frame rendering method and related apparatus
CN116347158A (en) Video playing method and device, electronic equipment and computer readable storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22787435

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22787435

Country of ref document: EP

Kind code of ref document: A1