WO2023066054A1 - 图像处理方法、云服务器、vr终端、存储介质 - Google Patents

图像处理方法、云服务器、vr终端、存储介质 Download PDF

Info

Publication number
WO2023066054A1
WO2023066054A1 PCT/CN2022/124175 CN2022124175W WO2023066054A1 WO 2023066054 A1 WO2023066054 A1 WO 2023066054A1 CN 2022124175 W CN2022124175 W CN 2022124175W WO 2023066054 A1 WO2023066054 A1 WO 2023066054A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame
rendered frame
rendering
information
difference data
Prior art date
Application number
PCT/CN2022/124175
Other languages
English (en)
French (fr)
Inventor
郑征世
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2023066054A1 publication Critical patent/WO2023066054A1/zh

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194Transmission of image signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/398Synchronisation thereof; Control thereof

Definitions

  • This application relates to but is not limited to the field of data processing, for example, an image processing method, a cloud server, a VR terminal, and a storage medium.
  • Cloud virtual reality (Virtual Reality, VR) technology renders VR applications in the cloud through cloud rendering technology, which can reduce the rendering and calculation of VR terminals, not only reducing delay and device power consumption, but also reducing the impact of local storage on content clarity. Reducing the performance requirements of VR terminals is an important development direction of VR technology. After the high-bit-rate audio and video streams rendered in the cloud, they need to be transmitted to the VR terminal through the network, which puts forward higher requirements on the transmission delay. The end-to-end delay in cloud VR is related to various factors such as network transmission efficiency, cloud rendering, cloud encoding method, and terminal decoding and rendering.
  • the full-view transmission solution is to transmit all 360-degree surround images to the VR terminal in the cloud, which requires encoding and decoding of ultra-high resolution, and the encoding delay, network transmission delay, and VR terminal decoding delay will increase several times. Higher requirements are placed on hardware resources.
  • the FOV transmission scheme can perform real-time picture rendering according to the user's perspective, and only render and transmit the picture of the user's current perspective. Compared with the panoramic transmission scheme, it has great advantages and has been widely used.
  • the embodiment of the present application provides an image processing method, a cloud server, a VR terminal, and a storage medium, which can reduce the amount of transmitted data and reduce the transmission delay.
  • the embodiment of the present application provides an image processing method, which is applied to a cloud server, and the cloud server is communicatively connected to a VR terminal, and the cloud server is configured with a first FOV rendering camera and a second FOV rendering camera, so
  • the methods described include:
  • the embodiment of the present application provides an image processing method, which is applied to a VR terminal, and the VR terminal is communicatively connected to a cloud server, and the cloud server is configured with a first FOV rendering camera and a second FOV rendering camera, so
  • the methods described include:
  • a display image is obtained according to the first rendered frame and the second rendered frame.
  • an embodiment of the present application provides a cloud server, including: a memory, a processor, and a computer program stored on the memory and operable on the processor, when the processor executes the computer program, the following In one aspect, the image processing method.
  • an embodiment of the present application provides a VR terminal, including: a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor executes the computer program to implement the following: The image processing method described in the second aspect.
  • the embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to execute the image processing method described in the first aspect, or to execute the image processing method described in the first aspect.
  • the image processing method described in the second aspect is a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to execute the image processing method described in the first aspect, or to execute the image processing method described in the first aspect.
  • Fig. 1 is a flowchart of an image processing method applied to a cloud server provided by an embodiment of the present application
  • Fig. 2 is a rendering schematic diagram provided by another embodiment of the present application.
  • FIG. 3 is a flow chart of configuring a rendering camera provided by another embodiment of the present application.
  • Fig. 4 is a flow chart of obtaining difference data provided by another embodiment of the present application.
  • Fig. 5 is a flow chart of restoring the second rendered frame provided by another embodiment of the present application.
  • Fig. 6 is a flow chart of updating the attitude matrix provided by another embodiment of the present application.
  • FIG. 7 is a flow chart of encoding and decoding provided by another embodiment of the present application.
  • FIG. 8 is a flowchart of an image processing method applied to a VR terminal provided by another embodiment of the present application.
  • Fig. 9 is a flow chart of sending device information and attitude matrix provided by another embodiment of the present application.
  • Fig. 10 is a flow chart of restoring the second rendered frame provided by another embodiment of the present application.
  • Fig. 11 is a flow chart of restoring the second rendered frame provided by another embodiment of the present application.
  • Fig. 12 is a flow chart of obtaining second screen coordinates provided by another embodiment of the present application.
  • Fig. 13 is a flowchart of obtaining the coordinates of the first device provided by another embodiment of the present application.
  • Fig. 14 is a flow chart of updating the attitude matrix provided by another embodiment of the present application.
  • Fig. 15 is a flow chart of encoding and decoding provided by another embodiment of the present application.
  • Fig. 16 is a flow chart of a specific example provided by another embodiment of the present application.
  • Fig. 17 is a device diagram of a cloud server provided by another embodiment of the present application.
  • Fig. 18 is a device diagram of a VR terminal provided by another embodiment of the present application.
  • the present application provides an image processing method, a cloud server, a VR terminal, and a storage medium.
  • the image processing method includes: acquiring a first rendered frame and a second rendered frame, wherein the first rendered frame comes from the first FOV Rendering the camera, the second rendering frame is from the second FOV rendering camera; obtaining difference data from the second rendering frame, wherein the difference data is from the second rendering frame that is not related to the An area where the first rendered frame repeats; sending the first rendered frame and the difference data to the VR terminal, so that the VR terminal restores the first rendered frame and the difference data according to the first rendered frame and the difference data two rendering frames, so as to obtain a display image according to the first rendering frame and the second rendering frame.
  • the data transmission in the overlapping area can be omitted, and the load pressure of image processing and the transmission code stream are reduced, thereby reducing the direct transmission delay between the cloud server and the VR terminal, and improving user experience.
  • Figure 1 is an image processing method provided by an embodiment of the present application, which is applied to a cloud server, and the cloud server is connected to a VR terminal in communication, and the cloud server is configured with a first FOV rendering camera and a second FOV rendering camera , including but not limited to step S110, step S120 and step S130.
  • Step S110 acquiring a first rendered frame and a second rendered frame, wherein the first rendered frame comes from the first FOV rendering camera, and the second rendered frame comes from the second FOV rendering camera.
  • the rendering camera is a virtual device in the VR system, and the FOV rendering camera can render the images captured in the virtual scene according to the FOV angle, so as to obtain the rendered picture.
  • the VR system of the rendering camera The definition of , will not be repeated here.
  • the first FOV rendering camera and the second FOV rendering camera can correspond to the user's eyes respectively, for example, the first FOV rendering camera corresponds to the left For the eye area, the second FOV rendering camera corresponds to the right eye area, or the first FOV rendering camera corresponds to the right eye area, and the second FOV rendering camera corresponds to the left eye area.
  • the first FOV rendering camera and the second FOV rendering camera The corresponding relationship with the eyes is not limited.
  • first rendering frame and the second rendering frame in this embodiment are rendering frames at the same frame time, and since the display image is composed of multiple frames, in the case of multiple consecutive frames, this implementation is executed multiple times The technical solution of the example is enough, so I won’t go into details in the follow-up.
  • the first rendering frame and the second rendering frame can be saved in the frame buffer of the Graphics Processing Unit (GPU) in real time, of course , the first rendered frame and the second rendered frame can be stored in different frame buffers, which is convenient for parallel reading and writing, and the specific saving method can be adjusted according to the timing requirements.
  • GPU Graphics Processing Unit
  • Step S120 acquiring difference data from the second rendered frame, wherein the difference data comes from an area in the second rendered frame that does not overlap with the first rendered frame.
  • the first FOV rendering camera and the second FOV rendering camera correspond to the left and right eyes respectively. Since the interpupillary distance of a person is limited, the first rendering frame and the second rendering frame will not completely overlap, but there will be overlapping areas. For example, as shown in FIG. 2 , taking the left eye corresponding to the first FOV rendering camera and the right eye corresponding to the second FOV rendering camera as an example, there is an overlapping area 24 between the first rendering frame 25 and the second rendering frame 23 in FIG. 2 , if Using the existing technology, it is necessary to encode, decode and transmit the first rendered frame and the second rendered frame separately, and the amount of data is large, which is not conducive to the optimization of the transmission delay. However, this embodiment extracts the difference data from the second rendered frame , which can effectively reduce the amount of codec and transmission data.
  • the difference data can be from the second rendering frame or from For the first rendered frame, when the difference data is obtained from the first rendered frame, the full amount of frame data of the second rendered frame is sent to the VR terminal for subsequent operations.
  • the difference data can be from the second rendering frame or from For the first rendered frame, when the difference data is obtained from the first rendered frame, the full amount of frame data of the second rendered frame is sent to the VR terminal for subsequent operations.
  • the difference data obtained from the second rendering frame can be used to distinguish the display-related parameters of each pixel in the region, such as the coordinates of each pixel, and the red, green and blue (Red Green Blue, RGB) value of each pixel , which can be used to display images of difference regions.
  • RGB Red Green Blue
  • Step S130 sending the first rendered frame and the difference data to the VR terminal, so that the VR terminal restores the second rendered frame according to the first rendered frame and the difference data, so as to obtain a display image according to the first rendered frame and the second rendered frame.
  • the display image of the VR terminal is obtained by synthesizing the first rendered frame and the second rendered frame, and the full frame data of the first rendered frame is sent to the VR terminal by the cloud server. Extract the frame data of the repeated area from the first rendered frame, and then combine the frame data and the difference data of the repeated area to restore the second rendered frame.
  • the method of combining the frame data and the difference data of the repeated area is not too much in this embodiment limited.
  • the cloud server After the cloud server completes the rendering, it needs to encode the rendering result, and then send the encoding result to the VR terminal through the transmission network, and decode the encoding result through the VR terminal to obtain the displayed image.
  • the technical solution can effectively reduce the amount of data in the encoding stage, thereby reducing the amount of data transmitted, effectively reducing the transmission delay, and improving the user experience. provides the basis.
  • step S110 of the embodiment shown in FIG. 1 it also includes but is not limited to the following steps:
  • Step S310 obtaining the device information and attitude matrix sent by the VR terminal, the device information includes interpupillary distance information and FOV angle information, and the attitude matrix includes a first pose matrix and a second pose matrix;
  • Step S320 determining the distance between the first FOV rendering camera and the second FOV rendering camera according to the interpupillary distance information
  • Step S330 configuring a first rendering camera according to the first pose matrix and FOV angle information, and configuring a second rendering camera according to the second pose matrix and FOV angle information.
  • the corresponding FOV angle information is also different, so the device information can be obtained through the detection of the VR terminal.
  • Those skilled in the art are familiar with how to obtain the user's FOV angle information through the VR terminal during the user's use.
  • Pupillary distance information and FOV angle information for example, as shown in FIG.
  • the distance between the cameras is D, and images are acquired at an FOV angle of 21 for rendering. This embodiment does not place too many limitations on the information acquisition process, as long as the corresponding information can be acquired.
  • the attitude matrix can include position information and rotation information of the VR terminal, and the input images of the first FOV rendering camera and the second FOV rendering camera can be determined through the attitude matrix. Therefore, the attitude matrix can include the first pose matrix and The second pose matrix corresponds to the left and right eye regions respectively. For example, the first pose matrix corresponds to the left eye region, and the second pose matrix corresponds to the right eye region. This embodiment does not limit this much. According to the first FOV rendering camera and The configuration of the second FOV rendering camera can be adjusted.
  • the cloud server can create and start a rendering thread, and adjust the distance between the first FOV rendering camera and the second FOV rendering camera through the interpupillary distance information, thereby simulating the temporal distance between the eyes, And configure the first FOV rendering camera through the first pose matrix and FOV angle information, and configure the second FOV rendering camera through the second pose matrix and FOV angle information, so that the first FOV rendering camera and the second FOV rendering camera can be configured according to Real-time rendering with good rendering parameters to ensure the real-time performance of displayed images.
  • step S120 of the embodiment shown in FIG. 1 also includes but is not limited to the following steps:
  • Step S410 according to the FOV angle information and the interpupillary distance information, determine the overlapping area of the first rendered frame and the second rendered frame;
  • Step S420 determining an area other than the repeated area in the second rendered frame as a difference area
  • Step S430 acquiring difference data from the second rendered frame according to the difference area.
  • the area other than the duplicated area in the second rendered frame and the first rendered frame is a difference area.
  • the difference from the second rendered frame can be The region obtains difference data and sends it to the VR terminal, thereby reducing the amount of coded, decoded and transmitted data, and optimizing the transmission delay.
  • step S130 of the embodiment shown in FIG. 1 also includes but is not limited to the following steps:
  • Step S510 adding the identification information of the repeated area in the first rendered frame
  • Step S520 sending the first rendered frame carrying the identification information and the difference data to the VR terminal, so that the VR terminal determines the repeated area according to the identified information, obtains the first repeated frame data from the first rendered frame according to the repeated area, and according to The second rendered frame is restored from the first repeated frame data and the difference data.
  • the VR terminal After the VR terminal obtains the data sent by the cloud server, it needs to restore the second rendered frame based on the first rendered frame and the difference data, so it is necessary to extract the first repeated frame of the repeated area from the first rendered frame Based on this, the identification information of the repeated area can be added to the first rendered frame.
  • the shapes of the rendered frames are usually the same and parallel, so the repeated area can be determined with the coordinates of the upper left corner vertex.
  • identification information can also be added to the coordinates in the entire repeated area.
  • This embodiment does not specify the specific identification method. As a limitation, it only needs to enable the VR terminal to determine the repeated area from the first rendered frame.
  • the imaging of the left and right eyes is not the same, but has a certain mapping relationship, so it is necessary to analyze the first repeated frame Transform the data.
  • the first rendered frame corresponds to the left eye.
  • the pose matrix perform matrix transformation on the first repeated frame data according to the first pose matrix and the second pose matrix, so as to obtain the right
  • the second repeated frame data corresponding to the eye is combined with the difference data to obtain the second rendered frame.
  • step S110 of the embodiment shown in FIG. 1 is executed, the following steps are also included but not limited to:
  • Step S610 acquiring posture update information sent by the VR terminal
  • Step S620 updating the attitude matrix according to the attitude update information.
  • the attitude matrix includes terminal position information and terminal rotation information, and the above information will change during the actual use of the user.
  • the real-time report of the VR terminal can be obtained
  • the attitude update information can be used to update the attitude matrix in real time.
  • the specific form of the attitude update information can be adjusted according to actual needs, or it can be directly reported to the real-time attitude matrix for replacement, which is not limited here.
  • the rendering parameters of the first FOV rendering camera and the second FOV rendering camera can be adjusted according to the latest attitude matrix, which will not be repeated here.
  • step S130 of the embodiment shown in FIG. 1 also includes but is not limited to the following steps:
  • Step S710 acquiring the full amount of frame data of the first rendered frame, and encoding the full amount of frame data to obtain a first encoding result
  • Step S720 encoding the difference data to obtain a second encoding result
  • Step S730 sending the first encoding result and the second encoding result to the VR terminal in parallel, so that the VR terminal can decode the first rendered frame according to the first encoding result, and obtain difference data according to the second encoding result.
  • the cloud server can create and start two coding threads, one of which obtains the full frame data of the first rendered frame from the GPU cache that saves the first rendered frame for coding, and the other One encoding thread acquires difference data from the GPU cache storing the second rendered frame for encoding, thereby realizing parallel encoding and improving encoding efficiency.
  • the data volume of the second encoding result is effectively reduced, which can reduce the transmission delay.
  • the two encoding results are sent to the VR terminal in parallel, making VR The terminal can perform decoding in a parallel manner, which effectively improves decoding efficiency and effectively improves user experience.
  • FIG. 8 another embodiment of the present application provides an image processing method, which is applied to a VR terminal, and the VR terminal is connected to a cloud server in communication, and the cloud server is configured with a first FOV rendering camera and a second FOV rendering camera , including but not limited to step S810, step S820 and step S830.
  • Step S810 obtaining the first rendered frame and difference data sent by the cloud server, wherein the first rendered frame comes from the first FOV rendering camera, and the difference data comes from an area in the second rendered frame that does not overlap with the first rendered frame.
  • the second rendering frame comes from the second FOV rendering camera;
  • Step S820 restore the second rendered frame according to the first rendered frame and the difference data
  • Step S830 obtaining a display image according to the first rendered frame and the second rendered frame.
  • step S810 of the embodiment shown in FIG. 8 is executed, the following steps are also included but not limited to:
  • Step S910 acquiring device information and an attitude matrix, where the equipment information includes interpupillary distance information and FOV angle information, and the attitude matrix includes a first pose matrix and a second pose matrix;
  • Step S920 sending the device information and attitude matrix to the cloud server, so that the cloud server determines the distance between the first FOV rendering camera and the second FOV rendering camera according to the interpupillary distance information, and configures according to the first pose matrix and FOV angle information
  • the first rendering camera configures the second rendering camera according to the second pose matrix and FOV angle information.
  • step S820 of the embodiment shown in FIG. 8 also includes but is not limited to the following steps:
  • Step S1010 determine the repeated area according to the identification information, and acquire the first repeated frame data from the first rendered frame according to the repeated area;
  • Step S1020 restore the second rendered frame according to the first repeated frame data and the difference data.
  • the first repeated frame data includes the first screen coordinates of each pixel in the screen space coordinate system, and also includes the RGB values of the pixels corresponding to the first screen coordinates.
  • FIG. 11 shown in FIG. 10
  • the step S1020 of the illustrated embodiment also includes but not limited to the following steps:
  • Step S1110 performing matrix transformation on the first screen coordinates to obtain the second screen coordinates
  • Step S1120 according to the mapping relationship between the first screen coordinates and the second screen coordinates, assign the pixel RGB value to the corresponding second screen coordinates to obtain the second repeated frame data;
  • step S1130 a second rendered frame is obtained by combining the second repeated frame data and the difference data.
  • the screen space coordinate system is the space coordinate system corresponding to the display screen in the VR terminal
  • the first rendering frame and the second rendering frame may include the coordinates corresponding to the screen space coordinate system and the RGB values corresponding to each coordinate. value, so as to realize the display on the VR terminal.
  • the overlapping area is the area where the image displayed by the left eye and the image displayed by the right eye overlap, the imaging of the left eye and the right eye are not the same, and because the VR terminal can rotate, the first rendered frame and the second rendered frame are not necessarily in the same position.
  • the first repeated frame data and the difference data are directly merged, this will cause an error in the displayed image.
  • the first repeated frame can be transformed by matrix transformation. The data is converted into second repeated frame data, so as to ensure that the restored image is the second rendered frame.
  • the first screen coordinates of each pixel of the first repeated frame data can be converted into the second screen coordinates of the second repeated frame data, and the RGB value of the pixel corresponding to the first screen coordinates is assigned Give the second screen coordinates so that each pixel of the second repeated frame data has an RGB value, thereby restoring the image of the repeated area, and after obtaining the second repeated frame data, restore the second repeated frame data by combining the second repeated frame data and the difference data
  • the second rendering frame therefore, after the cloud server finishes rendering, only the difference data of the second rendering frame can be delivered, and the second rendering frame can be completed on the VR terminal, which effectively reduces the amount of data encoded, decoded and transmitted.
  • step S1110 of the embodiment shown in FIG. 11 also includes but is not limited to the following steps:
  • Step S1210 converting the first screen coordinates of each pixel into first device coordinates in the device coordinate system through screen mapping
  • Step S1220 obtaining a preset projection matrix, and converting the first device coordinates of each pixel into world coordinates according to the first pose matrix and the projection matrix;
  • Step S1230 converting the world coordinates of each pixel into second device coordinates according to the projection matrix and the second pose matrix;
  • Step S1240 converting the second device coordinates of each pixel into second screen coordinates through screen mapping.
  • the first pose matrix, the projection matrix and the second pose matrix can all be obtained, and the specific obtaining methods will not be repeated here.
  • a VR terminal not only a screen space coordinate system, but also a device coordinate system and a world coordinate system are provided.
  • the above coordinate systems are technologies well known to those skilled in the art, and will not be repeated here.
  • the first screen coordinates can be converted to the first device coordinates through the coordinate system first, and according to the mapping relationship between the device coordinate system and the world coordinate system, each pixel world coordinates, and then use the second pose matrix and projection matrix to convert the world coordinates to obtain the second screen coordinates.
  • a specific example is proposed below:
  • the first rendering frame corresponds to the left eye
  • the second rendering frame corresponds to the right eye.
  • the left eye pose matrix M_Left and projection matrix P are obtained, and the The coordinates in the screen space coordinate system are converted into standardized device coordinates (x_left, y_left, z_left, w_left) T , and then the standardized device coordinates (x_left, y_left, z_left, w_left) T of each left-eye repeated frame are converted into the world coordinate system
  • the first repeated frame data also includes the depth information of each pixel.
  • step S1210 of the embodiment shown in FIG. 12 also includes but is not limited to the following steps:
  • Step S1310 obtaining pre-configured screen width information, screen height information and default weight
  • Step S1320 determining the abscissa of the first device coordinate according to the abscissa of the first screen coordinate and the screen width information
  • Step S1330 determining the ordinate of the first device coordinate according to the ordinate of the first screen coordinate and the screen height information
  • step S1340 the depth information is determined as the depth coordinate of the first device coordinate, and the default weight is determined as the weight of the first device coordinate.
  • the coordinates of the left-eye repeat frame in the screen space coordinate system are (X_left, Y_left) T , and the color buffer ColorBuffer and depth information DepthBuffer are obtained from the left-eye repeat frame data, and the RGB value of the pixel is obtained according to the ColorBuffer, and the coordinates of the screen space coordinate system
  • the depth corresponding to the point is DepthBuffer(X_Left, Y_Left), and the predefined screen width ScreenWidth and screen height ScreenHeight have 0 ⁇ X_Left ⁇ ScreenWidth, 0 ⁇ Y_Left ⁇ ScreenHeight; therefore, the screen space coordinate system coordinates are converted to standardized device coordinates (x_left, y_left, z_left, w_left)
  • step S810 of the embodiment shown in FIG. 8 after executing step S810 of the embodiment shown in FIG. 8 , it also includes but is not limited to the following steps:
  • Step S1410 when it is detected that the attitude matrix is changed, acquire the attitude update information of the attitude matrix
  • Step S1420 sending the attitude update information to the cloud server, so that the cloud server updates the attitude matrix according to the attitude update information.
  • step S810 of the embodiment shown in FIG. 8 also includes but is not limited to the following steps:
  • Step S1510 obtain the first encoding result and the second encoding result sent by the cloud server in parallel, wherein the first encoding result is obtained by the cloud server by encoding the full amount of frame data of the first rendered frame, and the second encoding result is obtained by the cloud server according to the difference
  • the data is encoded to obtain;
  • Step S1520 obtaining the first rendered frame by decoding the first encoding result
  • Step S1530 obtaining difference data by decoding the second encoding result.
  • Step S1610 obtain the device information and real-time attitude matrix of the VR terminal, and upload them to the cloud server, wherein the attitude matrix includes terminal position information and terminal rotation information, and the device information includes pupil distance information and FOV angle information of the VR terminal;
  • Step S1620 the cloud server rendering application side initializes the left-eye rendering camera and the right-eye rendering camera, wherein the rendering parameter is FOV angle information, the interpupillary distance information is the relative distance between the two rendering cameras, and the cloud server receives the uplink data of the VR terminal in real time Attitude matrix, camera position and camera rotation information are updated in real time by the upstream attitude matrix;
  • the rendering parameter is FOV angle information
  • the interpupillary distance information is the relative distance between the two rendering cameras
  • the cloud server receives the uplink data of the VR terminal in real time Attitude matrix
  • camera position and camera rotation information are updated in real time by the upstream attitude matrix
  • Step S1630 the cloud server creates and starts a rendering thread, the left-eye rendering camera and the right-eye rendering camera perform real-time rendering respectively under the control of the attitude matrix, and the rendering results are updated in real time and saved in the GPU frame buffer, wherein the left-eye rendering results are saved to the left-eye frame buffer, and the right-eye rendering result is saved to the right-eye frame buffer;
  • Step S1640 perform calculation according to the FOV angle information and the interpupillary distance information, and extract the difference area rendering frame data from the difference area of the right eye rendering result;
  • Step S1650 the cloud server creates and starts two parallel encoding threads, the left encoding thread encodes the full amount of frame data rendered by the left eye, and the right encoding thread only encodes the frame data rendered in the right eye difference area, and sends the encoding result For VR terminal;
  • Step S1660 the VR terminal starts two decoding threads, and the left decoding thread decodes to obtain the full amount of frame data for the left eye, and directly displays it on the screen;
  • Step S1670 extracting the left-eye repeated frame data from the left-eye full frame data, and converting the coordinates of the left-eye repeated frame data in the screen space coordinate system into the coordinates of the right-eye repeated frame in the screen space coordinate system by matrix transformation;
  • Step S1680 assign the pixel RGB value of the coordinates of the left-eye repeated frame in the screen space coordinate system to the coordinates of the corresponding right-eye repeated frame in the screen space coordinate system, so as to obtain the right-eye repeated frame image;
  • step S1690 the right-eye repeated frame picture and the right-eye difference area picture are synthesized, and displayed on the screen.
  • step S1670 is as follows:
  • the coordinates in the screen space coordinate system are (X_left, Y_left) T , including the color buffer ColorBuffer and the depth information DepthBuffer.
  • the coordinate point of the screen space coordinate system corresponds to the depth of DepthBuffer(X_Left, Y_Left), and the predefined screen width ScreenWidth and screen height ScreenHeight have 0 ⁇ X_Left ⁇ ScreenWidth, 0 ⁇ Y_Left ⁇ ScreenHeight; therefore, the coordinates of the screen space coordinate system are converted to Normalized device coordinates (x_left, y_left, z_left, w_left)
  • the normalized device coordinates of the frame for the right eye are converted into coordinates (X_right, Y_right) T in the screen space coordinate system of the right eye, and the specific formula is:
  • an embodiment of the present application also provides a cloud server.
  • the cloud server 1700 includes: a memory 1710 , a processor 1720 , and a computer program stored in the memory 1710 and operable on the processor 1720 .
  • the processor 1720 and the memory 1710 may be connected through a bus or in other ways.
  • the non-transitory software programs and instructions required to realize the image processing method of the above embodiment are stored in the memory 1710, and when executed by the processor 1720, the image processing method applied to the cloud server in the above embodiment is executed, for example, executing Method step S110 to step S130 in Fig. 1 described above, method step S310 to step S330 in Fig. 3, method step S410 to step S430 in Fig. 4, method step S510 to step S520 in Fig. 5, in Fig. 6
  • the method step S610 to step S620, and the method step S710 to step S730 in FIG. 7 The method step S610 to step S620, and the method step S710 to step S730 in FIG. 7 .
  • an embodiment of the present application also provides a VR terminal.
  • the VR terminal 1800 includes: a memory 1810 , a processor 1820 , and a computer program stored in the memory 1810 and operable on the processor 1820 .
  • the processor 1820 and the memory 1810 may be connected through a bus or in other ways.
  • the non-transitory software programs and instructions required to implement the image processing method of the above embodiment are stored in the memory 1810, and when executed by the processor 1820, the image processing method applied to the VR terminal in the above embodiment is executed, for example, executing Method steps S810 to step S830 in FIG. 8 described above, method steps S910 to step S920 in FIG. 9 , method steps S1010 to step S1020 in FIG. 10 , method steps S1110 to step S1130 in FIG. method step S1210 to step S1240, the method step S1310 to step S1340 in FIG. 13 , the method step S1410 to step S1420 in FIG. 14 , and the method step S1510 to step S1530 in FIG. 15 .
  • the device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • an embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by a processor or a controller, for example, by the above-mentioned Execution by a processor in the cloud server embodiment can make the above-mentioned processor execute the image processing method applied to the cloud server device in the above-mentioned embodiment, for example, execute the method step S110 to step S130 in FIG. 1 described above, FIG. Method step S310 to step S330 in 3, method step S410 to step S430 in Fig. 4, method step S510 to step S520 in Fig. 5, method step S610 to step S620 in Fig. 6, method step S710 in Fig.
  • step S730 Go to step S730; as another example, it is executed by a processor in the above-mentioned VR terminal embodiment, which can make the above-mentioned processor execute the image processing method applied to the cloud server device in the above-mentioned embodiment, for example, execute the above-described FIG. 8 method step S810 to step S830, method step S910 to step S920 in FIG. 9, method step S1010 to step S1020 in FIG. 10, method step S1110 to step S1130 in FIG. S1240, method step S1310 to step S1340 in FIG. 13 , method step S1410 to step S1420 in FIG. 14 , method step S1510 to step S1530 in FIG. 15 .
  • Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer.
  • communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请提供了一种图像处理方法、云服务器、VR终端、存储介质,图像处理方法包括:获取第一渲染帧和第二渲染帧,其中,第一渲染帧来自于第一FOV渲染相机,第二渲染帧来自于第二FOV渲染相机(S110);从第二渲染帧中获取差异数据,其中,差异数据来自于第二渲染帧中不与第一渲染帧重复的区域(S120);将第一渲染帧和差异数据发送至VR终端,以使VR终端根据第一渲染帧和差异数据还原出第二渲染帧,从而根据第一渲染帧和第二渲染帧得到显示图像(S130)。

Description

图像处理方法、云服务器、VR终端、存储介质
相关申请的交叉引用
本申请基于申请号为202111225795.2、申请日为2021年10月21日的中国专利申请提出,并要求该中国专利申请的优先权,该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请涉及但不限于数据处理领域,例如涉及一种图像处理方法、云服务器、VR终端、存储介质。
背景技术
云化虚拟现实(Virtual Reality,VR)技术通过云端渲染技术在云端对VR应用进行渲染,可以减少VR终端的渲染和计算,不仅减少了延迟和设备功耗,还可以降低本地存储对内容清晰度的限制,降低VR终端的性能要求,是VR技术重要的发展方向。云端渲染得到的高码率音视频流之后,需要通过网络传输到VR终端,这对传输时延提出了更高的要求。云化VR中端到端延时大小与网络传输效率、云端渲染、云端编码方式以及终端解码渲染等多种因素相关。
在传统的云化VR解决方案中,通常采用全视角传输和视觉场(Field Of View,FOV)两种传输方案。全视角传输方案是云端将360度环绕的画面全部传输至VR终端,这就需要对超高分辨率进行编解码,编码延时和网络传输延时以及VR终端解码延时将成数倍增加,同时对硬件资源提出较高的要求。FOV传输方案能够根据用户视角进行实时画面渲染,只渲染和传输用户当前视角的画面,相比于全景传输方案存在较大优势,得到了较为广泛的应用。
但是,为了应用FOV传输方案,需要模拟双眼进行渲染,然后将左右眼渲染结果分别进行编码和传输到VR终端,再由VR终端进行左右眼解码上屏。由于双眼瞳距不大,左右眼渲染帧结果存在很大比例的重复渲染区域,编解码时延较高,从而影响传输时延。
发明内容
以下是对本文详细描述的主题的概述。本概述并非是为了限制权利要求的保护范围。
本申请实施例提供了一种图像处理方法、云服务器、VR终端、存储介质,能够减少传输的数据量,减少传输时延。
第一方面,本申请实施例提供了一种图像处理方法,应用于云服务器,所述云服务器与VR终端通信连接,所述云服务器配置有第一FOV渲染相机和第二FOV渲染相机,所述方法包括:
获取第一渲染帧和第二渲染帧,其中,所述第一渲染帧来自于所述第一FOV渲染相机,所述第二渲染帧来自于所述第二FOV渲染相机;
从所述第二渲染帧中获取差异数据,其中,所述差异数据来自于所述第二渲染帧中不与所述第一渲染帧重复的区域;
将所述第一渲染帧和所述差异数据发送至所述VR终端,以使所述VR终端根据所述第一渲染帧和所述差异数据还原出所述第二渲染帧,从而根据所述第一渲染帧和所述第二渲染帧得到显示图像。
第二方面,本申请实施例提供了一种图像处理方法,应用于VR终端,所述VR终端与云服务器通信连接,所述云服务器配置有第一FOV渲染相机和第二FOV渲染相机,所述方法包括:
获取所述云服务器发送的第一渲染帧和差异数据,其中,所述第一渲染帧来自于所述第一FOV渲染相机,所述差异数据来自于所述第二渲染帧中不与所述第一渲染帧重复的区域,所述第二渲染帧来自于所述第二FOV渲染相机;
根据所述第一渲染帧和所述差异数据还原出所述第二渲染帧;
根据所述第一渲染帧和所述第二渲染帧得到显示图像。
第三方面,本申请实施例提供了一种云服务器,包括:存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序,所述处理器执行所述计算机程序时实现如第一方面所述的图像处理方法。
第四方面,本申请实施例提供了一种VR终端,包括:存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序,所述处理器执行所述计算机程序时实现如第二方面所述的图像处理方法。
第五方面,本申请实施例提供了一种计算机可读存储介质,存储有计算机可执行指令,所述计算机可执行指令用于执行如第一方面所述的图像处理方法,或者,执行如第二方面所述的图像处理方法。
本申请的其它特征和优点将在随后的说明书中阐述,并且,部分地从说明书中变得显而易见,或者通过实施本申请而了解。本申请的目的和其他优点可通过在说明书、权利要求书以及附图中所特别指出的结构来实现和获得。
附图说明
附图用来提供对本申请技术方案的进一步理解,并且构成说明书的一部分,与本申请的实施例一起用于解释本申请的技术方案,并不构成对本申请技术方案的限制。
图1是本申请一个实施例提供的应用于云服务器的图像处理方法的流程图;
图2是本申请另一个实施例提供的渲染示意图;
图3是本申请另一个实施例提供的配置渲染相机的流程图;
图4是本申请另一个实施例提供的获取差异数据的流程图;
图5是本申请另一个实施例提供的还原第二渲染帧的流程图;
图6是本申请另一个实施例提供的更新姿态矩阵的流程图;
图7是本申请另一个实施例提供的编解码的流程图;
图8是本申请另一个实施例提供的应用于VR终端的图像处理方法的流程图;
图9是本申请另一个实施例提供的发送设备信息和姿态矩阵的流程图;
图10是本申请另一个实施例提供的还原第二渲染帧的流程图;
图11是本申请另一个实施例提供的还原第二渲染帧的流程图;
图12是本申请另一个实施例提供的得到第二屏幕坐标的流程图;
图13是本申请另一个实施例提供的得到第一设备坐标的流程图;
图14是本申请另一个实施例提供的更新姿态矩阵的流程图;
图15是本申请另一个实施例提供的编解码的流程图;
图16是本申请另一个实施例提供的具体示例流程图;
图17是本申请另一个实施例提供的云服务器的装置图;
图18是本申请另一个实施例提供的VR终端的装置图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白,以下结合附图及实施例,对本申请进行进一步详细说明。应当理解,此处所描述的具体实施例仅用以解释本申请,并不用于限定本申请。
需要说明的是,虽然在装置示意图中进行了功能模块划分,在流程图中示出了逻辑顺序,但是在某些情况下,可以以不同于装置中的模块划分,或流程图中的顺序执行所示出或描述的步骤。说明书、权利要求书或上述附图中的术语“第一”、“第二”等是用于区别类似的对象,而不必用于描述特定的顺序或先后次序。
本申请提供了一种图像处理方法、云服务器、VR终端、存储介质,图像处理方法包括:获取第一渲染帧和第二渲染帧,其中,所述第一渲染帧来自于所述第一FOV渲染相机,所述第二渲染帧来自于所述第二FOV渲染相机;从所述第二渲染帧中获取差异数据,其中,所述差异数据来自于所述第二渲染帧中不与所述第一渲染帧重复的区域;将所述第一渲染帧和所述差异数据发送至所述VR终端,以使所述VR终端根据所述第一渲染帧和所述差异数据还原出所述第二渲染帧,从而根据所述第一渲染帧和所述第二渲染帧得到显示图像。根据本实施例的技术方案,能够省去重叠区域的数据传输,减少了图像处理的负载压力和传输码流,从而减少了云服务器与VR终端直接的传输时延,提高了用户体验。
下面结合附图,对本申请实施例作进一步阐述。
如图1所示,图1是本申请一个实施例提供的一种图像处理方法,应用于云服务器,云服务器与VR终端通信连接,云服务器配置有第一FOV渲染相机和第二FOV渲染相机,包括但不限于有步骤S110、步骤S120和步骤S130。
步骤S110,获取第一渲染帧和第二渲染帧,其中,第一渲染帧来自于第一FOV渲染相机,第二渲染帧来自于第二FOV渲染相机。
需要说明的是,渲染相机为VR***中的虚拟装置,FOV渲染相机能够对根据FOV角在虚拟场景中拍摄得到的图像进行渲染,从而得到渲染画面,本领域技术人员熟知渲染相机的VR***中的定义,在此不多作赘述。
值得注意的是,对于应用FOV技术的VR***,需要根据用户双眼的FOV进行图像显示,因此第一FOV渲染相机和第二FOV渲染相机可以分别对应用户的双眼,例如第一FOV渲染相机对应左眼区域,第二FOV渲染相机对应右眼区域,也可以是第一FOV渲染相机对应右眼区域,第二FOV渲染相机对应左眼区域,本实施例第一FOV渲染相机和第二FOV渲染相机与双眼的对应关系并不多做限定。
需要说明的是本实施例的第一渲染帧和第二渲染帧是同一帧时刻的渲染帧,由于显示图像是由多帧画面组成,在涉及多个连续帧的情况下,多次执行本实施例的技术方案即可,后续不多作赘述。
值得注意的是,在通过渲染得到第一渲染帧和第二渲染帧之后,可以将第一渲染帧和第二渲染帧实时保存到图形处理器(Graphics Processing Unit,GPU)的帧缓存中,当然,可以将第一渲染帧和第二渲染帧分别保存在不同的帧缓存中,便于实现并行读写,具体的保存方式根据时机需求调整即可。
步骤S120,从第二渲染帧中获取差异数据,其中,差异数据来自于第二渲染帧中不与第一渲染帧重复的区域。
需要说明的是,第一FOV渲染相机和第二FOV渲染相机分别与左右眼对应,由于人的瞳距有限,因此第一渲染帧和第二渲染帧不会完全重叠,而是存在重复区域,例如图2所示,以左眼对应第一FOV渲染相机,右眼对应第二FOV渲染相机为例,图2中的第一渲染帧25和第二渲染帧23之间有重复区域24,若采用现有技术,则需要分别对第一渲染帧和第二渲染帧进行编解码和传输,数据量较大,不利于传输时延的优化,而本实施例从第二渲染帧中提取差异数据,能够有效减少编解码和传输的数据量。
值得注意的是,由于差异数据是从不重复的区域进行获取,而重复区域对于第一渲染帧和第二渲染帧是相同的,因此差异数据既可以是来自于第二渲染帧,也可以来自于第一渲染帧,当从第一渲染帧获取差异数据,将第二渲染帧的全量帧数据发送至VR终端进行后续操作即可,在本实施例的技术方案的基础上,本领域技术人员有动机根据实际需求从任意一个渲染帧中获取差异数据,并将另一个渲染帧的全量帧数据发送至VR终端,在此不多作限定。
需要说明的是,从第二渲染帧获取的差异数据可以差异区域中每一个像素与显示相关的参数,例如每个像素的坐标,以及每个像素的红绿蓝(Red Green Blue,RGB)值,能够用于显示差异区域的图像即可。
步骤S130,将第一渲染帧和差异数据发送至VR终端,以使VR终端根据第一渲染帧和差异数据还原出第二渲染帧,从而根据第一渲染帧和第二渲染帧得到显示图像。
需要说明的是,VR终端的显示图像通过合成第一渲染帧和第二渲染帧得到,而第一渲染帧的全量帧数据由云服务器下发至VR终端,在具备差异数据的前提下,可以从第一渲染帧提取重复区域的帧数据,再组合重复区域的帧数据与差异数据,从而还原出第二渲染帧,组合重复区域的帧数据与差异数据的方法在本实施例中不作过多限定。
需要说明的是,云服务器完成渲染后,需要对渲染结果进行编码,再将编码结果通过传输网络下发至VR终端,通过VR终端对编码结果进行解码,从而得到显示图像,采用本实施例的技术方案,由于省去了第二渲染帧中与第一渲染帧重复的区域的数据,能够有效减少编码阶段的数据量,从而减少数据传输的数据量,有效减少传输时延,为提高用户体验提供了基础。
另外,在一实施例中,参照图3,在执行图1所示实施例的步骤S110之前,还包括但不限于有以下步骤:
步骤S310,获取VR终端发送的设备信息和姿态矩阵,设备信息包括瞳距信息和FOV角度信息,姿态矩阵包括第一位姿矩阵和第二位姿矩阵;
步骤S320,根据瞳距信息确定第一FOV渲染相机和第二FOV渲染相机之间的距离;
步骤S330,根据第一位姿矩阵和FOV角度信息配置第一渲染相机,根据第二位姿矩阵和FOV角度信息配置第二渲染相机。
需要说明的是,由于不同用户的瞳距不同,对应的FOV角度信息也不同,因此设备信息可以通过VR终端进行检测得到,本领域技术人员熟知如何在用户使用过程中,通过VR终端获取用户的瞳距信息和FOV角度信息,例如,参考图2所示,通过VR终端20检测到瞳距信息和FOV角21的大小,其中眼瞳距为D,则第一FOV渲染相机和第二FOV渲染相机之间的距离为D,并且以FOV角21获取图像进行渲染,本实施例并不对信息获取过程作过多限定,能 够获取到对应的信息即可。
需要说明的是,姿态矩阵可以包括VR终端的位置信息和旋转信息,通过姿态矩阵能够确定第一FOV渲染相机和第二FOV渲染相机的输入图像,因此,姿态矩阵可以包括第一位姿矩阵和第二位姿矩阵,分别对应左右眼区域,例如第一位姿矩阵对应左眼区域,第二位姿矩阵对应右眼区域,本实施例对此不多做限定,根据第一FOV渲染相机和第二FOV渲染相机的配置调整即可。
需要说明的是,在获取到姿态矩阵后,云服务器可以创建并启动渲染线程,可以通过瞳距信息调整第一FOV渲染相机和第二FOV渲染相机之间的距离,从而模拟双眼时间的距离,并且通过第一位姿矩阵和FOV角度信息配置第一FOV渲染相机,以及第二位姿矩阵和FOV角度信息配置第二FOV渲染相机,使得第一FOV渲染相机和第二FOV渲染相机能够根据配置好的渲染参数进行实时渲染,确保显示图像的实时性。
另外,在一实施例中,参照图4,图1所示实施例的步骤S120,还包括但不限于有以下步骤:
步骤S410,根据FOV角度信息和瞳距信息,确定第一渲染帧和第二渲染帧的重复区域;
步骤S420,将第二渲染帧中除重复区域以外的区域确定为差异区域;
步骤S430,根据差异区域从第二渲染帧中获取差异数据。
参考图2所示,在已知瞳距为D以及FOV角21的数值的情况下,本领域技术人员熟知利用三角函数计算出视线重复区域22的边长,由于视线重复区域22与重复区域24的两边是相对应的,因此能够确定重复区域24的左边界和右边界,再根据第一渲染帧和第二渲染帧之间的位置信息,可以确定重复区域24的上边界和下边界,从而确定出重复区域24。当然,也可以根据第一渲染帧和第二渲染帧每个像素的坐标,将重复的坐标所对应的区域作为重复区域,在此不作过多限定。
需要说明的是,在确定重复区域后,第二渲染帧和第一渲染帧中除了重复区域以外的区域为差异区域,在第一渲染帧全量传输的前提下,可以从第二渲染帧的差异区域获取差异数据,并下发至VR终端,从而减少编解码和传输的数据量,优化传输时延。
另外,在一实施例中,参照图5,图1所示实施例的步骤S130,还包括但不限于有以下步骤:
步骤S510,在第一渲染帧中添加重复区域的标识信息;
步骤S520,将携带有标识信息的第一渲染帧和差异数据发送至VR终端,以使VR终端根据标识信息确定重复区域,根据重复区域从第一渲染帧中获取第一重复帧数据,并根据第一重复帧数据和差异数据还原出第二渲染帧。
需要说明的是,VR终端在获取到云服务器发送的数据之后,需要根据第一渲染帧和差异数据还原出第二渲染帧,因此需要从第一渲染帧中提取出重复区域的第一重复帧数据,基于此,可以在第一渲染帧中添加重复区域的标识信息,例如,参考图2,在发送第一渲染帧时携带重复区域的左上角顶点的坐标,由于第一渲染帧和第二渲染帧的形状通常相同且平行,因此能够在具备左上角顶点的坐标的情况下确定出重复区域,当然,也可以对整个重复区域中的坐标添加标识信息,本实施例对具体的标识方式不做限定,能够使得VR终端从第一渲染帧中确定重复区域即可。
需要说明的是,在获取到第一重复帧数据之后,虽然该区域是左右眼重复的区域,但是 左右眼的成像并不相同,而是具有一定的映射关系,因此还需要对第一重复帧数据进行变换处理,例如,第一渲染帧与左眼对应,在具备姿态矩阵的情况下,根据第一位姿矩阵和第二位姿矩阵对第一重复帧数据进行矩阵变换,从而得到与右眼对应的第二重复帧数据,再与差异数据进行组合得到第二渲染帧,本领域技术人员可以根据实际需求选取由左眼视图转换到右眼视图的计算方式,在此不多作限定。
另外,在一实施例中,参照图6,在执行图1所示实施例的步骤S110之后,还包括但不限于有以下步骤:
步骤S610,获取VR终端发送的姿态更新信息;
步骤S620,根据姿态更新信息更新姿态矩阵。
需要说明的是,基于上述实施例的描述,姿态矩阵包括终端位置信息和终端旋转信息,而上述信息在用户实际使用过程中会发生变化,为了确保渲染图像的准确性,可以获取VR终端实时上报的姿态更新信息,从而实时更新姿态矩阵,姿态更新信息的具体形式可以根据实际需求调整,也可以直接上报实时的姿态矩阵进行替换,在此不多作限定。
值得注意的是,在更新姿态矩阵后,可以根据最新的姿态矩阵调整第一FOV渲染相机和第二FOV渲染相机的渲染参数,在此不多作赘述。
另外,在一实施例中,参照图7,图1所示实施例的步骤S130,还包括但不限于有以下步骤:
步骤S710,获取第一渲染帧的全量帧数据,针对全量帧数据进行编码,得到第一编码结果;
步骤S720,针对差异数据进行编码,得到第二编码结果;
步骤S730,将第一编码结果和第二编码结果并行发送至VR终端,以使VR终端根据第一编码结果解码得到第一渲染帧,根据第二编码结果解码得到差异数据。
需要说明的是,在获取到差异数据后,云服务器可以创建并启动两路编码线程,其中一路编码线程从保存第一渲染帧的GPU缓存中获取第一渲染帧的全量帧数据进行编码,另一路编码线程从保存第二渲染帧的GPU缓存中获取差异数据进行编码,从而实现并行编码,提高编码效率。
需要说明的是,由于在编码过程中只有差异数据,因此第二编码结果的数据量得到了有效的减少,能够减少传输时延,同时两路编码结果以并行的方式发送至VR终端,使得VR终端能够以并行的方式进行解码,有效提高了解码效率,有效提高用户体验。
另外,参照图8,在本申请的另一个实施例提供了一种图像处理方法,应用于VR终端,VR终端与云服务器通信连接,云服务器配置有第一FOV渲染相机和第二FOV渲染相机,包括但不限于有步骤S810、步骤S820和步骤S830。
步骤S810,获取云服务器发送的第一渲染帧和差异数据,其中,第一渲染帧来自于第一FOV渲染相机,差异数据来自于第二渲染帧中不与第一渲染帧重复的区域,第二渲染帧来自于第二FOV渲染相机;
步骤S820,根据第一渲染帧和差异数据还原出第二渲染帧;
步骤S830,根据第一渲染帧和第二渲染帧得到显示图像。
需要说明的是,本实施例的技术方案与图1所示实施例记载的技术方案相类似,区别在于本实施例的技术方案是以VR终端作为执行主体进行描述,为了叙述简便在此不重复赘述。
另外,在一实施例中,参照图9,在执行图8所示实施例的步骤S810之后,还包括但不限于有以下步骤:
步骤S910,获取设备信息和姿态矩阵,设备信息包括瞳距信息和FOV角度信息,姿态矩阵包括第一位姿矩阵和第二位姿矩阵;
步骤S920,向云服务器发送设备信息和姿态矩阵,以使云服务器根据瞳距信息确定第一FOV渲染相机和第二FOV渲染相机之间的距离,并根据第一位姿矩阵和FOV角度信息配置第一渲染相机,根据第二位姿矩阵和FOV角度信息配置第二渲染相机。
需要说明的是,本实施例的技术方案与图3所示实施例记载的技术方案相类似,区别在于本实施例的技术方案是以VR终端作为执行主体进行描述,为了叙述简便在此不重复赘述。
另外,在一实施例中,第一渲染帧携带有重复区域的标识信息,重复区域为第一渲染帧和第二渲染帧重复的区域,重复区域由云服务器根据FOV角度信息和瞳距信息确定,参照图10,图8所示实施例的步骤S820,还包括但不限于有以下步骤:
步骤S1010,根据标识信息确定重复区域,根据重复区域从第一渲染帧中获取第一重复帧数据;
步骤S1020,根据第一重复帧数据和差异数据还原出第二渲染帧。
需要说明的是,本实施例的技术方案与图5所示实施例记载的技术方案相类似,区别在于本实施例的技术方案是以VR终端作为执行主体进行描述,为了叙述简便在此不重复赘述。
另外,在一实施例中,第一重复帧数据包括每个像素在屏幕空间坐标系下的第一屏幕坐标,还包括与第一屏幕坐标相对应的像素RGB值,参照图11,图10所示实施例的步骤S1020,还包括但不限于有以下步骤:
步骤S1110,针对第一屏幕坐标进行矩阵变换,得到第二屏幕坐标;
步骤S1120,根据第一屏幕坐标与第二屏幕坐标的映射关系,将像素RGB值赋值于对应的第二屏幕坐标,得到第二重复帧数据;
步骤S1130,通过组合第二重复帧数据和差异数据得到第二渲染帧。
需要说明的是,屏幕空间坐标系为VR终端中显示屏幕所对应的空间坐标系,第一渲染帧和第二渲染帧可以包括屏幕空间坐标系所对应的坐标,以及每个坐标所对应的RGB值,从而实现在VR终端的显示。而重复区域虽然是左眼显示图像和右眼显示图像重叠的区域,但是左眼和右眼的成像并不相同,并且由于VR终端能够旋转,第一渲染帧和第二渲染帧并不一定在同一水平线,若直接将第一重复帧数据与差异数据合并,这会导致显示图像出错,在具备坐标系、姿态矩阵和第一屏幕坐标的情况下,可以通过矩阵变换的方式将第一重复帧数据转换成第二重复帧数据,从而确保还原得到的图像为第二渲染帧。
值得注意的是,通过矩阵变换,能够将第一重复帧数据的每个像素的第一屏幕坐标转换成第二重复帧数据的第二屏幕坐标,讲第一屏幕坐标所对应的像素RGB值赋值给第二屏幕坐标,使得第二重复帧数据的每个像素具备RGB值,从而还原得到重复区域的图像,在得到第二重复帧数据后,通过组合第二重复帧数据和差异数据还原出第二渲染帧,因此,可以在云服务器完成渲染后,只下发第二渲染帧的差异数据部分,即可在VR终端还完出第二渲染帧,有效减少了编解码和传输的数据量。
另外,在一实施例中,参照图12,图11所示实施例的步骤S1110,还包括但不限于有以下步骤:
步骤S1210,通过屏幕映射将每个像素的第一屏幕坐标转换为设备坐标系下的第一设备坐标;
步骤S1220,获取预先设定的投影矩阵,根据第一位姿矩阵和投影矩阵,将每个像素的第一设备坐标转换成世界坐标;
步骤S1230,根据投影矩阵和第二位姿矩阵,将每个像素的世界坐标转换成第二设备坐标;
步骤S1240,通过屏幕映射将每个像素的第二设备坐标转换为第二屏幕坐标。
需要说明的是,对于VR终端而言,第一位姿矩阵、投影矩阵和第二位姿矩阵都是可以获取的,具体的获取方式在此不多作赘述。
需要说明的是,对于VR终端而言,不仅仅具备屏幕空间坐标系,还具备设备坐标系和世界坐标系,上述坐标系都为本领域技术人员熟知的技术,在此不做多赘述。
值得注意的是,为了将第一屏幕坐标转换成第二屏幕坐标,可以先将第一屏幕坐标通过坐标系的转换得到第一设备坐标,根据设备坐标系与世界坐标系的映射关系,得到每个像素的世界坐标,再利用第二位姿矩阵和投影矩阵,将世界坐标转换得到第二屏幕坐标,为了更好地说明本实施例的技术方案,以下提出一个具体示例:
在本示例中,第一渲染帧对应左眼,第二渲染帧对应右眼,以左眼重复帧的坐标为(X_Left,Y_Left)为例,获取左眼位姿矩阵M_Left、投影矩阵P,将屏幕空间坐标系下的坐标转换为标准化设备坐标(x_left,y_left,z_left,w_left) T,再将每个左眼重复帧的标准化设备坐标(x_left,y_left,z_left,w_left) T转化为世界坐标系下的坐标(x_world,y_world,z_world,w_world) T,具体公式如下:(x_world,y_world,z_world,w_world) T=M_Left -1×P -1×(x_left,y_left,z_left,w_left) T
同时,根据右眼姿态矩阵M_right和投影矩阵P将世界坐标系下的坐标(x_world,y_world,z_world,w_world) T转换成右眼帧标准化设备坐标(x_right,y_right,z_right,w_right) T,具体公式如下:(x_right,y_right,z_right,w_right) T=M_right×P×(x_world,y_world,z_world,w_world) T
最后,将右眼帧标准化设备坐标转换为右眼的屏幕空间坐标系下坐标(X_right,Y_right) T,具体公式为:
Figure PCTCN2022124175-appb-000001
Figure PCTCN2022124175-appb-000002
另外,在一实施例中,第一重复帧数据还包括每个像素的深度信息,参照图13,图12所示实施例的步骤S1210,还包括但不限于有以下步骤:
步骤S1310,获取预先配置的屏幕宽度信息、屏幕高度信息和默认权重;
步骤S1320,根据第一屏幕坐标的横坐标和屏幕宽度信息确定第一设备坐标的横坐标;
步骤S1330,根据第一屏幕坐标的纵坐标和屏幕高度信息确定第一设备坐标的纵坐标;
步骤S1340,将深度信息确定为第一设备坐标的深度坐标,将默认权重确定为第一设备坐标的权重。
需要说明的是,在图12所示实施例的示例基础上,进一步进行举例说明:
左眼重复帧在屏幕空间坐标系下的坐标为(X_left,Y_left) T,从左眼重复帧数据中获取包 括颜色缓存ColorBuffer和深度信息DepthBuffer,根据ColorBuffer获取像素RGB值,屏幕空间坐标系的坐标点对应深度为DepthBuffer(X_Left,Y_Left),预先定义的屏幕宽度ScreenWidth和屏幕高度ScreenHeight,则有0≤X_Left≤ScreenWidth,0≤Y_Left≤ScreenHeight;因此屏幕空间坐标系坐标转为标准化设备坐标(x_left,y_left,z_left,w_left) T的数值可以通过以下公式得到:
Figure PCTCN2022124175-appb-000003
z=DepthBuffer(X_Left,Y_Left),w=1.0,其中w为权重。
另外,在一实施例中,参照图14,在执行图8所示实施例的步骤S810之后,还包括但不限于有以下步骤:
步骤S1410,当检测到姿态矩阵发生变更,获取姿态矩阵的姿态更新信息;
步骤S1420,将姿态更新信息发送至云服务器,以使云服务器根据姿态更新信息更新姿态矩阵。
需要说明的是,本实施例的技术方案与图6所示实施例记载的技术方案相类似,区别在于本实施例的技术方案是以VR终端作为执行主体进行描述,为了叙述简便在此不重复赘述。
另外,在一实施例中,参照图15,图8所示实施例的步骤S810,还包括但不限于有以下步骤:
步骤S1510,获取云服务器并行发送的第一编码结果和第二编码结果,其中,第一编码结果由云服务器根据第一渲染帧的全量帧数据进行编码得到,第二编码结果由云服务器根据差异数据进行编码得到;
步骤S1520,通过解码第一编码结果得到第一渲染帧;
步骤S1530,通过解码第二编码结果得到差异数据。
需要说明的是,本实施例的技术方案与图7所示实施例记载的技术方案相类似,区别在于本实施例的技术方案是以VR终端作为执行主体进行描述,为了叙述简便在此不重复赘述。
另外,为了更好地说明本实施例的技术方案,以下提出一个具体示例,参照图16,该具体示例包括但不限于有以下步骤:
步骤S1610,获取VR终端的设备信息和实时的姿态矩阵,并上传至云服务器,其中,姿态矩阵包括终端位置信息和终端旋转信息,设备信息包括VR终端的瞳距信息和FOV角信息;
步骤S1620,云服务器渲染应用侧初始化左眼渲染相机和右眼渲染相机,其中,渲染参数为FOV角信息,瞳距信息为两个渲染相机的相对距离,并且,云服务器实时接收VR终端上行的姿态矩阵,相机位置和相机旋转信息由上行的姿态矩阵实时控制更新;
步骤S1630,云服务器创建并启动渲染线程,左眼渲染相机和右眼渲染相机在姿态矩阵的控制下分别进行实时渲染,渲染结果实时更新并保存到GPU帧缓存中,其中,左眼渲染结果保存到左眼帧缓存中,右眼渲染结果保存到右眼帧缓存中;
步骤S1640,根据FOV角信息和瞳距信息进行计算,从右眼渲染结果的差异区域提取出差异区域渲染帧数据;
步骤S1650,云服务器创建并启动两路并行的编码线程,左路编码线程对左眼渲染全量帧数据进行编码,右路编码线程只对右眼差异区域渲染帧数据进行编码,将编码结果下发给VR终端;
步骤S1660,VR终端启动两路解码线程,左路解码线程解码得到左眼全量帧数据,并直 接进行上屏显示;
步骤S1670,从左眼全量帧数据中提取左眼重复帧数据,通过矩阵变换方式将左眼重复帧数据在屏幕空间坐标系的坐标转换成右眼重复帧在屏幕空间坐标系的坐标;
步骤S1680,将左眼重复帧帧时刻在屏幕空间坐标系的坐标的像素RGB值赋值给对应的右眼重复帧在屏幕空间坐标系的坐标,从而得到右眼重复帧画面;
步骤S1690,将右眼重复帧画面和右眼差异区域画面进行合成,并上屏显示。
在一实施例中,步骤S1670具体的实现方式为:
读取左眼重复帧数据,在屏幕空间坐标系下的坐标为(X_left,Y_left) T,包括颜色缓存ColorBuffer和深度信息DepthBuffer。屏幕空间坐标系的坐标点对应深度为DepthBuffer(X_Left,Y_Left),预先定义的屏幕宽度ScreenWidth和屏幕高度ScreenHeight,则有0≤X_Left≤ScreenWidth,0≤Y_Left≤ScreenHeight;因此屏幕空间坐标系坐标转为标准化设备坐标(x_left,y_left,z_left,w_left) T的数值可以通过以下公式得到:
Figure PCTCN2022124175-appb-000004
Figure PCTCN2022124175-appb-000005
z=DepthBuffer(X_Left,Y_Left),w=1.0。
在一实施例中,根据左眼位姿矩阵M_Left、投影矩阵P,将每个左眼重复帧的标准化设备坐标(x_left,y_left,z_left,w_left) T转化为世界坐标系下的坐标(x_world,y_world,z_world,w_world) T,具体公式如下:(x_world,y_world,z_world,w_world) T=M_Left -1×P -1×(x_left,y_left,z_left,w_left) T
在一实施例中,根据右眼姿态矩阵M_right和投影矩阵P将世界坐标系下的坐标(x_world,y_world,z_world,w_world) T转换成右眼帧标准化设备坐标(x_right,y_right,z_right,w_right) T,具体公式如下:(x_right,y_right,z_right,w_right) T=M_right×P×(x_world,y_world,z_world,w_world) T
在一实施例中,将右眼帧标准化设备坐标转换为右眼的屏幕空间坐标系下坐标(X_right,Y_right) T,具体公式为:
Figure PCTCN2022124175-appb-000006
Figure PCTCN2022124175-appb-000007
另外,参照图17,本申请的一个实施例还提供了一种云服务器,该云服务器1700包括:存储器1710、处理器1720及存储在存储器1710上并可在处理器1720上运行的计算机程序。
处理器1720和存储器1710可以通过总线或者其他方式连接。
实现上述实施例的图像处理方法所需的非暂态软件程序以及指令存储在存储器1710中,当被处理器1720执行时,执行上述实施例中的应用于云服务器的图像处理方法,例如,执行以上描述的图1中的方法步骤S110至步骤S130、图3中的方法步骤S310至步骤S330、图4中的方法步骤S410至步骤S430、图5中的方法步骤S510至步骤S520、图6中的方法步骤S610至步骤S620、图7中的方法步骤S710至步骤S730。
另外,参照图18,本申请的一个实施例还提供了一种VR终端,该VR终端1800包括:存储器1810、处理器1820及存储在存储器1810上并可在处理器1820上运行的计算机程序。
处理器1820和存储器1810可以通过总线或者其他方式连接。
实现上述实施例的图像处理方法所需的非暂态软件程序以及指令存储在存储器1810中, 当被处理器1820执行时,执行上述实施例中的应用于VR终端的图像处理方法,例如,执行以上描述的图8中的方法步骤S810至步骤S830、图9中的方法步骤S910至步骤S920、图10中的方法步骤S1010至步骤S1020、图11中的方法步骤S1110至步骤S1130、图12中的方法步骤S1210至步骤S1240、图13中的方法步骤S1310至步骤S1340、图14中的方法步骤S1410至步骤S1420、图15中的方法步骤S1510至步骤S1530。
以上所描述的装置实施例仅仅是示意性的,其中作为分离部件说明的单元可以是或者也可以不是物理上分开的,即可以位于一个地方,或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
此外,本申请的一个实施例还提供了一种计算机可读存储介质,该计算机可读存储介质存储有计算机可执行指令,该计算机可执行指令被一个处理器或控制器执行,例如,被上述云服务器实施例中的一个处理器执行,可使得上述处理器执行上述实施例中的应用于云服务器备的图像处理方法,例如,执行以上描述的图1中的方法步骤S110至步骤S130、图3中的方法步骤S310至步骤S330、图4中的方法步骤S410至步骤S430、图5中的方法步骤S510至步骤S520、图6中的方法步骤S610至步骤S620、图7中的方法步骤S710至步骤S730;又如,被上述VR终端实施例中的一个处理器执行,可使得上述处理器执行上述实施例中的应用于云服务器备的图像处理方法,例如,执行以上描述的图8中的方法步骤S810至步骤S830、图9中的方法步骤S910至步骤S920、图10中的方法步骤S1010至步骤S1020、图11中的方法步骤S1110至步骤S1130、图12中的方法步骤S1210至步骤S1240、图13中的方法步骤S1310至步骤S1340、图14中的方法步骤S1410至步骤S1420、图15中的方法步骤S1510至步骤S1530。本领域普通技术人员可以理解,上文中所公开方法中的全部或某些步骤、***可以被实施为软件、固件、硬件及其适当的组合。某些物理组件或所有物理组件可以被实施为由处理器,如中央处理器、数字信号处理器或微处理器执行的软件,或者被实施为硬件,或者被实施为集成电路,如专用集成电路。这样的软件可以分布在计算机可读介质上,计算机可读介质可以包括计算机存储介质(或非暂时性介质)和通信介质(或暂时性介质)。如本领域普通技术人员公知的,术语计算机存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质。计算机存储介质包括但不限于RAM、ROM、EEPROM、闪存或其他存储器技术、CD-ROM、数字多功能盘(DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。此外,本领域普通技术人员公知的是,通信介质通常包含计算机可读指令、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据,并且可包括任何信息递送介质。
以上是对本申请的若干实施进行了具体说明,但本申请并不局限于上述实施方式,熟悉本领域的技术人员在不违背本申请本质的前提下还可作出种种的等同变形或替换,这些等同的变形或替换均包含在本申请权利要求所限定的范围内。

Claims (17)

  1. 一种图像处理方法,应用于云服务器,所述云服务器与VR终端通信连接,所述云服务器配置有第一FOV渲染相机和第二FOV渲染相机,所述方法包括:
    获取第一渲染帧和第二渲染帧,其中,所述第一渲染帧来自于所述第一FOV渲染相机,所述第二渲染帧来自于所述第二FOV渲染相机;
    从所述第二渲染帧中获取差异数据,其中,所述差异数据来自于所述第二渲染帧中不与所述第一渲染帧重复的区域;
    将所述第一渲染帧和所述差异数据发送至所述VR终端,以使所述VR终端根据所述第一渲染帧和所述差异数据还原出所述第二渲染帧,从而根据所述第一渲染帧和所述第二渲染帧得到显示图像。
  2. 根据权利要求1所述的方法,其中,在所述获取第一渲染帧和第二渲染帧之前,所述方法还包括:
    获取所述VR终端发送的设备信息和姿态矩阵,所述设备信息包括瞳距信息和FOV角度信息,所述姿态矩阵包括第一位姿矩阵和第二位姿矩阵;
    根据所述瞳距信息确定所述第一FOV渲染相机和所述第二FOV渲染相机之间的距离;
    根据所述第一位姿矩阵和所述FOV角度信息配置所述第一渲染相机,根据所述第二位姿矩阵和所述FOV角度信息配置所述第二渲染相机。
  3. 根据权利要求2所述的方法,其中,所述从所述第二渲染帧中获取差异数据,包括:
    根据所述FOV角度信息和所述瞳距信息,确定所述第一渲染帧和所述第二渲染帧的重复区域;
    将所述第二渲染帧中除所述重复区域以外的区域确定为差异区域;
    根据所述差异区域从所述第二渲染帧中获取所述差异数据。
  4. 根据权利要求3所述的方法,其中,所述将所述第一渲染帧和所述差异数据发送至所述VR终端,以使所述VR终端根据所述第一渲染帧和所述差异数据还原出所述第二渲染帧,包括,
    在所述第一渲染帧中添加所述重复区域的标识信息;
    将携带有所述标识信息的所述第一渲染帧和所述差异数据发送至所述VR终端,以使所述VR终端根据所述标识信息确定所述重复区域,根据所述重复区域从所述第一渲染帧中获取第一重复帧数据,并根据所述第一重复帧数据和所述差异数据还原出所述第二渲染帧。
  5. 根据权利要求2所述的方法,其中,在所述获取第一渲染帧和第二渲染帧之后,所述方法还包括:
    获取所述VR终端发送的姿态更新信息;
    根据所述姿态更新信息更新所述姿态矩阵。
  6. 根据权利要求1所述的方法,其中,所述将所述第一渲染帧和所述差异数据发送至所述VR终端,包括:
    获取所述第一渲染帧的全量帧数据,针对所述全量帧数据进行编码,得到第一编码结果;
    针对所述差异数据进行编码,得到第二编码结果;
    将所述第一编码结果和所述第二编码结果并行发送至所述VR终端,以使所述VR终端根 据所述第一编码结果解码得到所述第一渲染帧,根据所述第二编码结果解码得到所述差异数据。
  7. 一种图像处理方法,应用于VR终端,所述VR终端与云服务器通信连接,所述云服务器配置有第一FOV渲染相机和第二FOV渲染相机,所述方法包括:
    获取所述云服务器发送的第一渲染帧和差异数据,其中,所述第一渲染帧来自于所述第一FOV渲染相机,所述差异数据来自于所述第二渲染帧中不与所述第一渲染帧重复的区域,所述第二渲染帧来自于所述第二FOV渲染相机;
    根据所述第一渲染帧和所述差异数据还原出所述第二渲染帧;
    根据所述第一渲染帧和所述第二渲染帧得到显示图像。
  8. 根据权利要求7所述的方法,其中,在所述获取所述云服务器发送的第一渲染帧和差异数据之前,所述方法还包括:
    获取设备信息和姿态矩阵,所述设备信息包括瞳距信息和FOV角度信息,所述姿态矩阵包括第一位姿矩阵和第二位姿矩阵;
    向所述云服务器发送所述设备信息和所述姿态矩阵,以使所述云服务器根据所述瞳距信息确定所述第一FOV渲染相机和所述第二FOV渲染相机之间的距离,并根据所述第一位姿矩阵和所述FOV角度信息配置所述第一渲染相机,根据所述第二位姿矩阵和所述FOV角度信息配置所述第二渲染相机。
  9. 根据权利要求8所述的方法,其中,所述第一渲染帧携带有重复区域的标识信息,所述重复区域为所述第一渲染帧和所述第二渲染帧重复的区域,所述重复区域由所述云服务器根据所述FOV角度信息和所述瞳距信息确定,所述根据所述第一渲染帧和所述差异数据还原出所述第二渲染帧,包括:
    根据所述标识信息确定所述重复区域,根据所述重复区域从所述第一渲染帧中获取第一重复帧数据;
    根据所述第一重复帧数据和所述差异数据还原出所述第二渲染帧。
  10. 根据权利要求9所述的方法,其中,所述第一重复帧数据包括每个像素在屏幕空间坐标系下的第一屏幕坐标,还包括与所述第一屏幕坐标相对应的像素RGB值,所述根据所述第一重复帧数据和所述差异数据还原出所述第二渲染帧,包括:
    针对所述第一屏幕坐标进行矩阵变换,得到第二屏幕坐标;
    根据所述第一屏幕坐标与所述第二屏幕坐标的映射关系,将所述像素RGB值赋值于对应的所述第二屏幕坐标,得到第二重复帧数据;
    通过组合所述第二重复帧数据和所述差异数据得到所述第二渲染帧。
  11. 根据权利要求10所述的方法,其中,所述针对所述第一屏幕坐标进行矩阵变换,得到第二屏幕坐标,包括:
    通过屏幕映射将每个像素的所述第一屏幕坐标转换为设备坐标系下的第一设备坐标;
    获取预先设定的投影矩阵,根据所述第一位姿矩阵和所述投影矩阵,将每个像素的所述第一设备坐标转换成世界坐标;
    根据所述投影矩阵和所述第二位姿矩阵,将每个像素的所述世界坐标转换成第二设备坐标;
    通过屏幕映射将每个像素的所述第二设备坐标转换为所述第二屏幕坐标。
  12. 根据权利要求11所述的方法,其中,所述第一重复帧数据还包括每个像素的深度信息,所述通过屏幕映射将每个像素的所述第一屏幕坐标转换为设备坐标系下的第一设备坐标,包括:
    获取预先配置的屏幕宽度信息、屏幕高度信息和默认权重;
    根据所述第一屏幕坐标的横坐标和所述屏幕宽度信息确定所述第一设备坐标的横坐标;
    根据所述第一屏幕坐标的纵坐标和所述屏幕高度信息确定所述第一设备坐标的纵坐标;
    将所述深度信息确定为所述第一设备坐标的深度坐标,将所述默认权重确定为所述第一设备坐标的权重。
  13. 根据权利要求8所述的方法,其中,在所述获取所述云服务器发送的第一渲染帧和差异数据之后,所述方法还包括:
    当检测到所述姿态矩阵发生变更,获取所述姿态矩阵的姿态更新信息;
    将所述姿态更新信息发送至所述云服务器,以使所述云服务器根据所述姿态更新信息更新所述姿态矩阵。
  14. 根据权利要求7所述的方法,其中,所述获取所述云服务器发送的第一渲染帧和差异数据,包括:
    获取所述云服务器并行发送的第一编码结果和第二编码结果,其中,所述第一编码结果由所述云服务器根据所述第一渲染帧的全量帧数据进行编码得到,所述第二编码结果由所述云服务器根据所述差异数据进行编码得到;
    通过解码所述第一编码结果得到所述第一渲染帧;
    通过解码所述第二编码结果得到所述差异数据。
  15. 一种云服务器,包括:存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序,其中,所述处理器执行所述计算机程序时实现如权利要求1至6中任意一项所述的图像处理方法。
  16. 一种VR终端,包括:存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序,其中,所述处理器执行所述计算机程序时实现如权利要求7至14中任意一项所述的图像处理方法。
  17. 一种计算机可读存储介质,存储有计算机可执行指令,其中,所述计算机可执行指令用于执行如权利要求1至14中任意一项所述的图像处理方法。
PCT/CN2022/124175 2021-10-21 2022-10-09 图像处理方法、云服务器、vr终端、存储介质 WO2023066054A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111225795.2A CN116016885A (zh) 2021-10-21 2021-10-21 图像处理方法、云服务器、vr终端、存储介质
CN202111225795.2 2021-10-21

Publications (1)

Publication Number Publication Date
WO2023066054A1 true WO2023066054A1 (zh) 2023-04-27

Family

ID=86032220

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/124175 WO2023066054A1 (zh) 2021-10-21 2022-10-09 图像处理方法、云服务器、vr终端、存储介质

Country Status (2)

Country Link
CN (1) CN116016885A (zh)
WO (1) WO2023066054A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117278780A (zh) * 2023-09-06 2023-12-22 上海久尺网络科技有限公司 视频编解码方法、装置、设备及存储介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117170953B (zh) * 2023-11-02 2024-01-12 北京麟卓信息科技有限公司 一种基于多路视频编码的渲染差异比对方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108989784A (zh) * 2018-07-09 2018-12-11 歌尔科技有限公司 虚拟现实设备的图像显示方法、装置、设备及存储介质
CN109714585A (zh) * 2019-01-24 2019-05-03 京东方科技集团股份有限公司 图像传输方法及装置、显示方法及装置、存储介质
WO2019209259A1 (en) * 2018-04-24 2019-10-31 Hewlett-Packard Development Company, L.P. Display devices including switches for transmitting identical pixel data to first and second display regions

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019209259A1 (en) * 2018-04-24 2019-10-31 Hewlett-Packard Development Company, L.P. Display devices including switches for transmitting identical pixel data to first and second display regions
CN108989784A (zh) * 2018-07-09 2018-12-11 歌尔科技有限公司 虚拟现实设备的图像显示方法、装置、设备及存储介质
CN109714585A (zh) * 2019-01-24 2019-05-03 京东方科技集团股份有限公司 图像传输方法及装置、显示方法及装置、存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117278780A (zh) * 2023-09-06 2023-12-22 上海久尺网络科技有限公司 视频编解码方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN116016885A (zh) 2023-04-25

Similar Documents

Publication Publication Date Title
WO2023066054A1 (zh) 图像处理方法、云服务器、vr终端、存储介质
CN107454468B (zh) 对沉浸式视频进行格式化的方法、装置和流
JP7504953B2 (ja) 画像を合成するための方法及び装置
US20140092439A1 (en) Encoding images using a 3d mesh of polygons and corresponding textures
JP2019534606A (ja) ライトフィールドデータを使用して場面を表す点群を再構築するための方法および装置
WO2017118078A1 (zh) 一种图像处理方法、播放方法及相关的装置和***
WO2018234258A1 (en) PROCESSING 3D IMAGE INFORMATION BASED ON TEXTURE AND MESH CARDS
CN106303289A (zh) 一种将真实对象与虚拟场景融合显示的方法、装置及***
WO2019095830A1 (zh) 基于增强现实的视频处理方法、装置及电子设备
US20130039597A1 (en) Comfort Noise and Film Grain Processing for 3 Dimensional Video
US10958950B2 (en) Method, apparatus and stream of formatting an immersive video for legacy and immersive rendering devices
US20130027389A1 (en) Making a two-dimensional image into three dimensions
CN112714357B (zh) 视频播放方法、视频播放装置、电子设备和存储介质
EP4120181A2 (en) Method and apparatus of fusing image, and method of training image fusion model
WO2023169283A1 (zh) 双目立体全景图像的生成方法、装置、设备、存储介质和产品
JP2012522285A (ja) データ及び三次元レンダリングを符号化するためのシステム及びフォーマット
TWI524730B (zh) 處理視頻之方法及其系統
WO2024022086A1 (zh) 基于三维显示的视频通信方法及***
US11240512B2 (en) Intra-prediction for video coding using perspective information
WO2023065961A1 (zh) 视频植入方法、装置、设备及计算机可读存储介质
CN116962743A (zh) 视频图像编码、抠图方法和装置及直播***
TWM630947U (zh) 立體影像播放裝置
TW202332263A (zh) 立體影像播放裝置及其立體影像產生方法
EP3821602A1 (en) A method, an apparatus and a computer program product for volumetric video coding
US20190052868A1 (en) Wide viewing angle video processing system, wide viewing angle video transmitting and reproducing method, and computer program therefor

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22882664

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2022882664

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022882664

Country of ref document: EP

Effective date: 20240507

NENP Non-entry into the national phase

Ref country code: DE