WO2021223577A1 - 一种视频处理的方法、相关装置、存储介质及程序产品 - Google Patents
一种视频处理的方法、相关装置、存储介质及程序产品 Download PDFInfo
- Publication number
- WO2021223577A1 WO2021223577A1 PCT/CN2021/086784 CN2021086784W WO2021223577A1 WO 2021223577 A1 WO2021223577 A1 WO 2021223577A1 CN 2021086784 W CN2021086784 W CN 2021086784W WO 2021223577 A1 WO2021223577 A1 WO 2021223577A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- video
- frame
- rendered
- target
- sequence
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 46
- 238000012545 processing Methods 0.000 claims abstract description 192
- 238000009877 rendering Methods 0.000 claims abstract description 42
- 238000004590 computer program Methods 0.000 claims abstract description 4
- 238000000034 method Methods 0.000 claims description 93
- 108091026890 Coding region Proteins 0.000 claims description 31
- 238000003780 insertion Methods 0.000 claims description 24
- 230000037431 insertion Effects 0.000 claims description 24
- 238000010586 diagram Methods 0.000 description 42
- 238000005516 engineering process Methods 0.000 description 21
- 238000012549 training Methods 0.000 description 20
- 238000013213 extrapolation Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 8
- 230000033001 locomotion Effects 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000002156 mixing Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 230000002457 bidirectional effect Effects 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/587—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/50—Controlling the output signals based on the game progress
- A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/103—Selection of coding mode or of prediction mode
- H04N19/114—Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234381—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440281—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the temporal resolution, e.g. by frame skipping
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
- H04N21/4781—Games
Definitions
- This application relates to an industry application field based on cloud technology, and specifically relates to a video processing technology.
- Cloud gaming is an online gaming technology based on cloud computing technology.
- all games are run on the server side, and the rendered game screen is compressed and transmitted to the user through the network.
- the terminal device used by the user does not require a high-end processor and graphics card, only basic video decompression capabilities.
- the business process based on cloud games is as follows: firstly, the terminal device is connected to the cloud game server, and then the interaction between the user and the game is realized through data flow and control flow.
- the data flow mainly includes game screen data, that is, the cloud game server first.
- the game screen is encoded, and then the encoded game screen data is transmitted to the terminal device, which is decoded by the terminal device and displayed on the interface.
- the cloud game server needs to render each frame of the game screen, and then encodes based on the rendered video frame, the whole process needs to consume more processing resources, resulting in excessive processor overhead , Reduce the business processing capacity of the cloud game server.
- One aspect of this application provides a video processing method, which is executed by a server, and includes:
- the original video sequence includes P video frames obtained after rendering, and P is an integer greater than or equal to 2;
- the target video sequence includes P video frames and Q unrendered video frames inserted into the P video frames, where Q is an integer greater than or equal to 1;
- Another aspect of the present application provides a video processing method, which is executed by a terminal device, and includes:
- the video sequence to be rendered includes X video frames that have not been rendered, and X is an integer greater than or equal to 2;
- the target video sequence includes X unrendered video frames and Y unrendered video frames inserted into the X video frames, where Y is an integer greater than or equal to 1;
- Another aspect of the present application provides the above-mentioned video processing method, the corresponding video processing device, computer equipment, computer-readable storage medium, and program product.
- the server first obtains the original video sequence, which includes P video frames obtained after rendering, and then obtains the target video sequence according to the original video sequence, and the target video sequence also includes inserting P Q unrendered video frames out of 4 video frames, then the server encodes the target video sequence to obtain the video encoding sequence, and finally the server sends the video encoding sequence to the terminal device so that the terminal device can decode the video encoding sequence to obtain The video sequence to be rendered.
- the server only needs to render part of the video frame, and then perform frame interpolation processing based on the rendered video frame, thereby obtaining the target video sequence, and the resources consumed by the frame interpolation processing are less than the resources required for rendering, so , Can save the processing resources on the server side, reduce the overhead of the processor, and help improve the service processing capacity of the server.
- FIG. 1 is a schematic diagram of an architecture of a video processing system in an embodiment of the application
- FIG. 2 is a schematic diagram of a cloud architecture of the video processing system in an embodiment of the application
- FIG. 3 is a schematic diagram of a cloud game architecture of the video processing system in an embodiment of the application
- FIG. 4 is a schematic diagram of an embodiment of a video processing method in an embodiment of the application.
- FIG. 5 is a schematic diagram of an embodiment of implementing interpolation frame processing based on a server in an embodiment of the application
- FIG. 6 is a schematic diagram of generating a target video frame in an interpolation frame mode in an embodiment of the application
- FIG. 7 is a schematic diagram of an embodiment of training an interpolation frame prediction model in an embodiment of the application.
- FIG. 8 is a schematic diagram of an embodiment of verifying an interpolation frame prediction model in an embodiment of the application.
- FIG. 9 is a schematic diagram of an embodiment of realizing extrapolation frame processing based on a server in an embodiment of the application.
- FIG. 10 is a schematic diagram of generating a target video frame in an extrapolated frame mode in an embodiment of the application.
- FIG. 11 is a schematic diagram of another embodiment of a video processing method in an embodiment of the application.
- FIG. 12 is a schematic diagram of an embodiment of implementing interpolation frame processing based on a terminal device in an embodiment of the application
- FIG. 13 is a schematic diagram of an embodiment of implementing extrapolation frame processing based on a terminal device in an embodiment of the application
- FIG. 14 is a schematic diagram of an interaction flow of the video processing method in an embodiment of the application.
- 15 is a schematic diagram of an embodiment of implementing interpolation frame processing based on a video processing system in an embodiment of the application;
- 16 is a schematic diagram of an embodiment of implementing extrapolated frame processing based on a video processing system in an embodiment of the application;
- FIG. 17 is a schematic diagram of an embodiment of implementing interpolation frame processing based on a video processing system in an embodiment of the application.
- FIG. 18 is a schematic diagram of another embodiment for implementing interpolation frame processing based on a video processing system in an embodiment of the application;
- FIG. 19 is a schematic diagram of an embodiment of a video processing device in an embodiment of the application.
- FIG. 20 is a schematic diagram of another embodiment of a video processing device in an embodiment of this application.
- FIG. 21 is a schematic diagram of a structure of a server in an embodiment of the application.
- FIG. 22 is a schematic diagram of a structure of a terminal device in an embodiment of the application.
- FIG. 23 is a schematic structural diagram of a video processing system in an embodiment of the application.
- the embodiments of the present application provide a video processing method, related device, and storage medium, which can save server-side processing resources, reduce processor overhead, and help improve the service processing capability of the server.
- the video processing method provided in this application is suitable for business scenarios under remote networks and business scenarios under cloud technology, where the business scenarios under cloud technology include but are not limited to cloud game services or cloud video services.
- this application uses dynamic frame insertion technology. Under the requirement of players to experience 60 frames per second (frame) of game screens, the cloud game server only needs to run 30 frames of game screens per second. Save processor overhead.
- this application uses the technology of dynamic frame insertion. Under the demand of the audience to experience 24 frames of animation per second, the cloud video server only needs to run 12 frames of animation per second, thereby saving processor overhead. .
- this application proposes a video processing method, which can be applied to the video processing system shown in FIG. 1.
- server 1 can establish a communication connection with M terminal devices
- server 2 can establish a communication connection with N terminals.
- M and N depend on the server.
- N and M can take the value 100.
- the server shown in FIG. 1 may be an independent physical server, or may be a server cluster or a distributed system composed of multiple physical servers, which is not limited here.
- FIG. 2 is a cloud architecture of the video processing system in an embodiment of the application.
- multiple cloud servers together form a cloud server cluster.
- One cloud server cluster can provide services for multiple terminal devices.
- a cloud server cluster includes 4 cloud servers, which can be combined with M
- the terminal device establishes a communication connection.
- the cloud server shown in Figure 2 can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (content delivery network, CDN) and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.
- the terminal devices shown in FIGS. 1 and 2 may be smart phones, tablet computers, TVs, and network set-top boxes (over the top, OTT), notebook computers, desktop computers, smart speakers, smart watches, etc., but are not limited to these.
- a client terminal is deployed on the terminal device, for example, a video client or a game client.
- the terminal device and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application.
- cloud technology is used in the architecture shown in Figure 2 to achieve business communication.
- cloud technology refers to the unification of hardware, software, and network resources in a wide area network or a local area network to achieve data integration.
- Cloud technology is based on the general term of network technology, information technology, integration technology, management platform technology, application technology, etc. applied in the cloud computing business model, which can form a resource pool, which can be used as needed, which is flexible and convenient. Cloud computing technology will become an important support.
- the background service of the technical network system requires a large amount of computing and storage resources, such as video websites, image websites and more portal websites.
- each item may have its own identification mark, which needs to be transmitted to the back-end system for logical processing.
- Data of different levels will be processed separately, and all types of industry data need to be powerful
- the backing of the system can only be achieved through cloud computing.
- Figure 3 is a schematic diagram of a cloud game architecture of the video processing system in an embodiment of the application.
- the control flow is mainly responsible for sending control signals.
- an input device such as a keyboard, mouse or joystick
- the signal will be encoded into the sending list and transmitted to Cloud gaming server.
- the cloud game server needs to transmit a data stream to the terminal device.
- the data stream includes an audio data stream and a video data stream.
- the terminal device de-decodes the data stream and then presents it on the screen.
- Cloud gaming also known as gaming on demand, is an online gaming technology based on cloud computing technology.
- Cloud gaming technology enables thin clients with relatively limited graphics processing and data computing capabilities to run high-quality games.
- the game does not run on the player's game terminal, but runs on the cloud server, and the cloud server renders the game scene into a video and audio stream, which is transmitted to the player's game terminal through the network.
- the player's game terminal does not need to have powerful graphics computing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to obtain player input commands and send them to the cloud server.
- cloud games can greatly reduce the cost of equipment for players to play games. For many high-quality games that require long-term updates, cloud games can also reduce the cost of publishing, updating and maintaining games for game companies.
- the quality of the multimedia stream rendered by the game scene depends on the network communication bandwidth.
- the multimedia stream of cloud games needs to consume more bandwidth, and the higher the quality of the multimedia stream, the bandwidth it consumes The resources will be higher.
- the video processing method provided by the present application can not only reduce the resource overhead on the cloud server side, but also reduce the transmission amount of video frames, thereby saving network bandwidth while ensuring the quality of the game picture.
- MMO massive multiplayer online
- the consumption of a graphics processing unit (GPU) is 36%.
- the GPU consumption is 20%.
- a maximum of 3 game processes can be run on a GPU.
- 5 game processes can be run on a GPU.
- An embodiment of the video processing method in the embodiment of this application includes:
- the server obtains an original video sequence, where the original video sequence includes P video frames obtained after rendering, and P is an integer greater than or equal to 2.
- the server acquires continuous P frame pictures, that is, P video frames are acquired, and these P video frames may constitute the original video sequence.
- the video frame may refer to a rendered video picture, may also refer to a rendered game picture, or may also refer to other types of pictures that have been rendered.
- the server involved in this application can be a local server (for example, a game server or a video server, etc.), or a cloud server (for example, a cloud game server or a cloud video server).
- a cloud game server applied to the cloud game business is introduced as an example, but this should not be understood as a limitation of this application.
- Picture rendering requires programs to perform operations on the geometry and vertices of the drawn graphics, and then get the image. In this process, the computer's processor needs to perform a lot of calculations. In practical applications, both the central processing unit (CPU) and GPU can perform rendering tasks. As the requirements for screen resolution and quality are getting higher and higher, the single-precision floating-point performance of the CPU is difficult to meet the rendering needs of complex screens. Therefore, the GPU can take over most of the graphics rendering work, and the CPU needs to schedule the GPU through some instructions Perform rendering.
- CPU central processing unit
- GPU can take over most of the graphics rendering work, and the CPU needs to schedule the GPU through some instructions Perform rendering.
- the server obtains a target video sequence according to the original video sequence, where the target video sequence includes P video frames and Q unrendered video frames inserted in the P video frames, and Q is an integer greater than or equal to 1.
- the server performs frame interpolation processing on P video frames in the original video sequence, and can perform frame interpolation processing on two adjacent video frames, for example, perform frame interpolation processing on video frame No. 1 and video frame No. 2 , And then insert the frame processing on the 2nd video frame and the 3rd video frame, and so on. It is also possible to perform frame interpolation processing on two related video frames, for example, perform frame interpolation processing on video frame No. 1 and video frame 2, and then perform frame interpolation processing on video frame No. 3 and video frame No. 4, and so on. Interval video frames can also be used for frame insertion processing, for example, frame insertion processing is performed on video frame No. 1 and video frame No. 2, and then frame insertion processing is performed on video frame No. 5 and video frame No. 6. Based on the original video sequence, generating Q video frames obtained after inserting frames, these Q video frames and P video frames together constitute a target video sequence.
- the method of inserting frames includes but is not limited to frame sampling, frame mixing, motion compensation and optical flow method.
- frame sampling refers to lengthening the display time of each key frame, which is equivalent to inserting the same several key frames.
- Frame blending means to increase the transparency of key frames before and after inserting a frame, and then synthesize a new frame.
- Motion compensation refers to recognizing the movement of an object, and then compensating and inserting frames.
- the optical flow method is to find the motion trajectory of the pixel in the previous and next frames when the gray value (or brightness value) of the same pixel is constant in the previous and next frames, and then perform prediction and interpolation processing based on the motion trajectory.
- the server performs encoding processing on the target video sequence to obtain a video encoding sequence.
- the server uses an encoding algorithm to encode the target video sequence, thereby generating a video encoding sequence.
- the encoding process can run on the CPU of the server, on the GPU, or on other encoding hardware, for example, the encoding chip in the GPU or a dedicated encoding chip independent of the GPU.
- the coding algorithm can adopt H264, H265, VP8 or VP9, etc., which is not limited here.
- the server s encoding mainly uses forward encoding reference frames (ie P frames) and key frames (ie I frames) instead of bidirectional encoding reference frames (ie B frame).
- the video coding sequence includes at least one group of pictures (GOP).
- the GOP is composed of one I frame and multiple B frames (or P frames), and is also the basic unit of encoder and decoder access.
- the I frame refers to an independent frame that carries all the information, and can be decoded independently without referring to other video frames.
- the P frame needs to refer to the previous I frame to be encoded.
- the B frame records the difference between this video frame and the previous two video frames.
- the server sends the video coding sequence to the terminal device, so that the terminal device decodes the video coding sequence to obtain the video sequence to be rendered.
- the server sends the video encoding sequence to the terminal device, so the terminal device can decode the video encoding sequence to obtain the video sequence to be rendered, and finally perform rendering processing on the video sequence to be rendered to generate the target video, and Display the target video on the screen of the terminal device.
- the decoding process can run on the CPU of the terminal device, or on the GPU, or on other decoding hardware, for example, a decoding chip in the GPU or a dedicated decoding chip independent of the GPU. After a video frame is decoded, the video frame can be read by the CPU or GPU on the terminal device side, rendered, and displayed on the interface.
- a method for video processing is provided.
- the server first obtains an original video sequence, which includes P video frames obtained after rendering, and then obtains a target video sequence according to the original video sequence.
- the target video sequence It also includes Q video frames obtained after frame insertion processing, and then the server encodes the target video sequence to obtain the video encoding sequence, and finally the server sends the video encoding sequence to the terminal device so that the terminal device can decode the video encoding sequence To get the video sequence to be rendered.
- the server only needs to render part of the video frame, and then perform frame interpolation processing based on the rendered video frame, thereby obtaining the target video sequence, and the resources consumed by the frame interpolation processing are less than the resources required for rendering, so , Can save the processing resources on the server side, reduce the overhead of the processor, and help improve the service processing capacity of the server.
- the server obtains the target video sequence according to the original video sequence, which may include:
- the server obtains the first rendered video frame and the second rendered video frame from the original video sequence, where the first rendered video frame is the previous image adjacent to the second rendered video frame;
- the server performs frame insertion processing on the first rendered video frame and the second rendered video frame to obtain the target video frame, where the target video frame is the next frame image adjacent to the first rendered video frame, and the target video frame is the first rendered video frame. 2.
- the image of the previous frame adjacent to the rendered video frame, and the target video frame is an unrendered video frame;
- the server generates a first video sub-sequence in the target video sequence according to the first rendered video frame, the target video frame, and the second rendered video frame, where the first video sub-sequence consists of the first rendered video frame and the target video in sequence. Frame and the second rendered video frame.
- a method of obtaining a target video sequence based on interpolated frames is introduced.
- the original video sequence may include a large number of rendered video frames, for ease of description, any two phases will be used below.
- the adjacent rendered video frame is introduced as an example. In practical applications, the processing methods for the other two adjacent frames are similar, so I will not repeat them here.
- the server obtains two adjacent video frames from the original video sequence, that is, the first rendered video frame and the second rendered video frame, and then generates a new video frame based on the two rendered video frames, That is, the target video frame is generated, where the target video frame is a frame between the first rendered video frame and the second rendered video frame, which is equivalent to inserting an additional frame of image.
- the first video subsequence in the target video sequence is generated.
- a series of video subsequences are generated in a similar manner, and finally Generate the target video sequence.
- Figure 5 is a schematic diagram of an embodiment of implementing interpolation frame processing based on a server in an embodiment of the application. As shown in the figure, taking a cloud game scene as an example, the cloud game server captures the rendered first The n frame game screen and the n+1 frame game screen are interpolated frame processing on the n frame game screen and the n+1 frame game screen to obtain an additional frame of the game screen, namely the n+0.5 frame game screen.
- the cloud game server continues to obtain the n+2th frame of the game screen, and performs interpolation frame processing on the n+1th frame of the game screen and the n+2th frame of the game screen to obtain an additional frame of the game screen, that is, the n+th frame. 1.5 frames of game screen.
- the target video sequence is composed of game pictures such as the nth frame, the n+0.5th frame, the n+1th frame, the n+1.5th frame, and the n+2th frame, and then the target video sequence is encoded to obtain Video coding sequence.
- the cloud game server transmits the video encoding sequence to the terminal device through the network, and the terminal device decodes the video encoding sequence to obtain the to-be-rendered video sequence, which is finally rendered and displayed on the interface of the terminal device.
- Figure 6 is a schematic diagram of the target video frame generated in the interpolation frame mode in the embodiment of the application.
- the first rendered video frame is the nth frame
- the second already rendered The rendered video frame is the n+1th frame
- the target video frame is generated after interpolation frame processing, where the target video frame is the n+0.5th frame.
- the server renders 30 frames of pictures
- the actual encoded pictures can be 60 frames
- the pictures output to the terminal device are also 60 frames. Therefore, for the terminal device, the number of decoded pictures is 60 frames.
- the image effect of the interpolated frame is better, and it is suitable for services that do not require high latency but require high image quality, such as non-real-time battle games.
- a method for obtaining the target video sequence based on the interpolated frame is provided.
- the target video sequence obtained by using the interpolated frame has better image effects, but due to the cloud service scenario One frame delay, therefore, is more suitable for services that do not require high delay, but require high picture quality.
- the picture quality can also be improved.
- the server performs processing on the first rendered video frame and the second rendered video frame.
- Interpolate frame processing to obtain the target video frame which can include:
- the server obtains the first frame number corresponding to the first rendered video frame
- the server obtains the second frame number corresponding to the second rendered video frame
- the server obtains the target frame number according to the first frame number and the second frame number
- the server can calculate the average value of the first frame number and the second frame number to obtain the target frame number.
- the server may also determine the target frame number in other ways.
- the server obtains K pixel information corresponding to the target video frame through the interpolated frame prediction model based on the target frame number, wherein the K pixel information is used to determine the target video frame, and K is an integer greater than 1.
- a method for interpolating frames based on the interpolated frame prediction model is introduced. It should be noted that since the original video sequence may include a large number of rendered video frames, for ease of description, any two The adjacent rendered video frames are introduced as an example. In practical applications, the processing methods for the other two adjacent frames are similar, so I won’t repeat them here.
- each rendered video frame corresponds to a frame number, where the frame number of the first rendered video frame is the first frame number, and the frame number of the second rendered video frame is the second frame number, assuming the first
- the frame number is n
- the second frame number is n+1
- the target frame number is calculated as follows:
- U represents the target frame number, which is n+0.5.
- K represents the total number of pixels included in a video frame.
- the pixel information may be expressed in the form of luminance-bandwidth-chrominance (YUV), or may be expressed in the form of red green blue (RGB).
- the following will introduce the way to train the interpolated frame prediction model. Take the cloud game scene as an example. Generally, you can train a corresponding interpolated frame prediction model for each game, or you can train the same interpolated frame prediction model for multiple games. There is no limit. First, it is necessary to obtain the video to be trained.
- the video to be trained includes multiple frames of training images. During the training process, the training image from the mth frame to the rth frame of training image can be extracted from the video to be trained, where 0 ⁇ m ⁇ r, In different interpolation frame prediction models, m can take different values.
- the interpolation frame prediction model can be trained.
- This application uses the following method to represent the frame interpolation prediction model:
- t represents the frame number, that is, the value range of t is a value greater than or equal to m and less than or equal to r.
- pos represents the coordinate point (x, y) corresponding to the video frame.
- frame (t, pos) represents the pixel information of the coordinate point corresponding to pos in the t video frames.
- the pixel information can be expressed in RGB format, or in YUV format, or in other formats. It is not exhaustive here.
- the interpolated frame prediction model f (t, pos) can be a one-time fitting function, or a multiple-time fitting function, and a functional relationship determined by a neural network or a deep learning method can also be used.
- FIG. 7 is a schematic diagram of an embodiment of training the interpolation frame prediction model in the embodiment of the application.
- the training image of the mth frame and the training image of the m+2th frame are input to the frame interpolation prediction model to be trained, and the target image is output by the frame interpolation prediction model, where the target image may be the predicted training image of the m+1th frame. And so on, until each training image in the known image frame sequence is similarly processed.
- the interpolated frame prediction model used when the extrapolated frame is trained, you can input the mth frame training image and the m+1th frame training image to the interpolated frame prediction model to be trained, and the interpolated frame prediction model outputs the target image, where ,
- the target image may be the predicted training image of the m+2th frame. And so on, until each training image in the known image frame sequence is similarly processed.
- a feasible evaluation method is to use the following loss function for calculation:
- L represents the loss value
- FIG. 8 is a schematic diagram of an embodiment for verifying the frame insertion prediction model in the embodiment of the application.
- the number of frames per second (FPS) is 60.
- the predicted images corresponding to all odd-numbered frames are used as the input of the interpolated frame prediction model, and the target image corresponding to the even-numbered frames is output through the interpolated frame prediction model.
- the target image corresponding to the even-numbered frame is compared with the predicted image in the sequence of known image frames.
- the first frame predicted image and the third frame predicted image are input to the interpolated frame prediction model, the interpolated frame predictive model outputs the target image, and the pixel information of each pixel on the target image is compared with the pixel information of each pixel of the second frame predicted image.
- the pixel information is compared, and if the difference between the two is less than the threshold, it means that the interpolation frame prediction model has been trained.
- a method for interpolating frames based on the interpolation frame prediction model is provided.
- the trained interpolation frame prediction model can be used to predict each pixel information in the target video frame, and then The target video frame is reconstructed from these pixel information, so as to realize the process of interpolating one frame to the video frame, thereby improving the feasibility and operability of the solution.
- the server obtains the target video sequence according to the original video sequence, which may include:
- the server obtains the first rendered video frame and the second rendered video frame from the original video sequence, where the first rendered video frame is the previous image adjacent to the second rendered video frame;
- the server performs frame insertion processing on the first rendered video frame and the second rendered video frame to obtain the target video frame, where the target video frame is the next image adjacent to the second rendered video frame, and the target video frame is Unrendered video frames;
- the server generates a second video sub-sequence in the target video sequence according to the first rendered video frame, the target video frame, and the second rendered video frame, where the second video sub-sequence consists of the first rendered video frame and the second rendered video frame.
- the rendered video frame and the target video frame are the first rendered video frame, the target video frame, and the second rendered video frame.
- a method for obtaining a target video sequence based on extrapolated frames is introduced.
- the original video sequence may include a large number of rendered video frames, for ease of description, any two phases will be used below.
- the adjacent rendered video frame is introduced as an example. In practical applications, the processing methods for the other two adjacent frames are similar, so I will not repeat them here.
- the server obtains two adjacent video frames from the original video sequence, that is, the first rendered video frame and the second rendered video frame, and then generates a new video frame based on the two rendered video frames, That is, the target video frame is generated, where the target video frame is a frame between the second rendered video frame and the next rendered video frame, which is equivalent to inserting an additional frame of image.
- the second video subsequence in the target video sequence is generated.
- a series of video subsequences are generated in a similar manner, and finally Generate the target video sequence.
- Figure 9 is a schematic diagram of an embodiment of implementing extrapolated frame processing based on a server in an embodiment of the application.
- the cloud game server captures the rendered first For the n frame game screen and the n+1 frame game screen, extrapolate frame processing is performed on the n frame game screen and the n+1 frame game screen to obtain an additional frame of game screen, namely the n+1.5th frame game screen.
- the cloud game server continues to obtain the n+2th frame of the game picture, and performs extrapolation processing on the n+1th frame of the game picture and the n+2th frame of the game picture to obtain an extra frame of the game picture, that is, the n+th frame. 2.5 frames of game screen.
- the target video sequence is composed of game pictures such as the nth frame, the n+1th frame, the n+1.5th frame, the n+2th frame, and the n+2.5th frame, and then the target video sequence is encoded to obtain Video coding sequence.
- the cloud game server transmits the video encoding sequence to the terminal device through the network, and the terminal device decodes the video encoding sequence to obtain the to-be-rendered video sequence, which is finally rendered and displayed on the interface of the terminal device.
- Figure 10 is a schematic diagram of the target video frame generated in the extrapolated frame mode in the embodiment of the application.
- the first rendered video frame is the nth frame
- the second rendered video frame is The video frame is the n+1th frame
- the target video frame is generated after the extrapolation frame processing, where the target video frame is the n+1.5th frame.
- the server renders 30 frames of pictures
- the actual encoded pictures can be 60 frames
- the pictures output to the terminal device are also 60 frames. Therefore, for the terminal device, the number of decoded pictures is 60 frames.
- Extrapolated frames usually do not generate additional delay, and are suitable for services that require high delay, such as real-time battle games.
- a method for obtaining the target video sequence based on the extrapolated frame is provided.
- the target video sequence obtained by using the extrapolated frame usually does not cause additional delay, so it is more suitable for delay requirements. Businesses that are relatively high, but do not require high picture quality. Therefore, while saving the server-side processor overhead, it can also avoid the phenomenon of picture delay caused by time delay.
- the server performs processing on the first rendered video frame and the second rendered video frame.
- Interpolate frame processing to obtain the target video frame which can include:
- the server obtains the second frame number corresponding to the second rendered video frame
- the server determines the next adjacent frame number of the second frame number as the third frame number
- the server obtains the target frame number according to the second frame number and the third frame number
- the server Based on the target frame number, the server obtains K pixel information corresponding to the target video frame through the interpolation frame prediction model, where the K pixel information is used to determine the target video frame, and K is an integer greater than 1.
- a method for extrapolating frames based on the interpolation frame prediction model is introduced. It should be noted that since the original video sequence may include a large number of rendered video frames, for ease of description, any two The adjacent rendered video frames are introduced as an example. In practical applications, the processing methods for the other two adjacent frames are similar, so I won’t repeat them here.
- each rendered video frame corresponds to a frame number, where the frame number of the first rendered video frame is the first frame number, and the frame number of the second rendered video frame is the second frame number.
- the frame number is n+1, and the third frame number is n+2.
- the calculation method of the target frame number is as follows:
- U represents the target frame number, which is n+1.5.
- K represents the total number of pixels included in a video frame.
- the pixel information may be expressed in YUV format, or may be expressed in RGB format.
- a method for extrapolating frames based on the interpolation frame prediction model is provided.
- the trained interpolation frame prediction model can be used to predict each pixel information in the target video frame, and then The target video frame is reconstructed from these pixel information, so as to realize the process of extrapolating a frame to the video frame, thereby improving the feasibility and operability of the solution.
- FIG. 11 another embodiment of the video processing method in the embodiment of this application includes:
- the terminal device receives the video encoding sequence sent by the server.
- the server obtains P consecutive video frames, and these P video frames may constitute the original video sequence.
- the video frame may refer to a rendered video picture, may also refer to a rendered game picture, or may also refer to other types of pictures that have been rendered.
- the server involved in this application can be a local server (for example, a game server or a video server, etc.), or a cloud server (for example, a cloud game server or a cloud video server).
- a cloud game server applied to the cloud game business is introduced as an example, but this should not be understood as a limitation of this application.
- the server uses an encoding algorithm to encode the original video sequence to generate a video encoding sequence.
- the encoding process can run on the CPU of the server, on the GPU, or on other encoding hardware, for example, the encoding chip in the GPU or a dedicated encoding chip independent of the GPU.
- the coding algorithm can adopt H264, H265, VP8 or VP9, etc., which is not limited here.
- the terminal device decodes the video encoding sequence to obtain a video sequence to be rendered, where the video sequence to be rendered includes X video frames that have not been rendered, and X is an integer greater than or equal to 2.
- the server sends a video coding sequence to the terminal device, and the video coding sequence includes at least two video frames. Therefore, the terminal device can decode the video encoding sequence to obtain the video sequence to be rendered.
- the video sequence to be rendered includes X video frames that have not been rendered, and the number of video frames included in the video sequence to be rendered is the same as the number of video frames in the video encoding sequence. The number of video frames included is the same.
- the terminal device obtains a target video sequence according to the video sequence to be rendered, where the target video sequence includes X unrendered video frames and Y unrendered video frames inserted into the X video frames, and Y is greater than or equal to 1. Integer.
- the terminal device performs frame interpolation processing on X video frames in the video sequence to be rendered, and can perform frame interpolation processing on two adjacent video frames, for example, perform interpolation on video frame 1 and video frame 2.
- Frame processing and then insert frame processing on the 2nd video frame and the 3rd video frame, and so on.
- frame interpolation processing on two related video frames, for example, perform frame interpolation processing on video frame No. 1 and video frame 2, and then perform frame interpolation processing on video frame No. 3 and video frame No. 4, and so on.
- Interval video frames can also be used for frame insertion processing, for example, frame insertion processing is performed on video frame No. 1 and video frame No. 2, and then frame insertion processing is performed on video frame No. 5 and video frame No. 6.
- Y video frames are generated after the frame is inserted, and the X video frames and the Y video frames together constitute the target video sequence.
- the resources consumed by inserting a frame of image are less than the resources consumed by rendering a frame of image.
- the method of inserting frames includes, but is not limited to, frame sampling, frame mixing, motion compensation, and optical flow method, which will not be repeated here.
- the terminal device performs rendering processing on the target video sequence to obtain the target video.
- the terminal device performs rendering processing on the target video sequence to generate the target video and display the target video on the screen of the terminal device.
- the decoding process can run on the CPU of the terminal device, on the GPU, or on other decoding hardware, for example, a decoding chip in the GPU or a dedicated decoding chip independent of the GPU. After a video frame is decoded, the video frame can be read by the CPU or GPU on the terminal device side, rendered, and displayed on the interface.
- the server generates 30 frames of images
- the corresponding bit rate is 10 megabits per second (Mbps)
- the terminal device can achieve the effect of 60 frames of images after frame insertion.
- the corresponding code rate is 20 Mbps, and the terminal device can achieve the effect of 60 frames of images without inserting frames. Therefore, the method of inserting frames in the terminal device can save transmission bandwidth.
- a video processing method is provided.
- the terminal device first receives the video coding sequence sent by the server, and then decodes the video coding sequence to obtain the video sequence to be rendered, and then obtains the video sequence according to the video sequence to be rendered.
- the target video sequence, and finally the target video sequence is rendered to obtain the target video.
- the server only needs to render fewer video frames, and then transmit them to the terminal device.
- the terminal device generates the target video sequence by inserting frames, and renders the target video. Therefore, the whole process corresponds to the server. It can save its processing resources and coding overhead, which is conducive to improving the service processing capabilities of the server. The whole process saves transmission bandwidth for the client.
- the terminal device obtains the target video sequence according to the video sequence to be rendered, which may include:
- the terminal device obtains the first unrendered video frame and the second unrendered video frame from the video sequence to be rendered, where the first unrendered video frame is the previous image adjacent to the second unrendered video frame;
- the terminal device performs frame insertion processing on the first unrendered video frame and the second unrendered video frame to obtain the target video frame, where the target video frame is the next frame image adjacent to the first unrendered video frame, and the target video frame Is the previous frame image adjacent to the second unrendered video frame;
- the terminal device generates a first video sub-sequence in the target video sequence according to the first unrendered video frame, the target video frame, and the second unrendered video frame, where the first video sub-sequence consists of the first unrendered video frame and the target video frame in turn. It is composed of a video frame and a second unrendered video frame.
- a method for obtaining a target video sequence based on interpolated frames is introduced.
- the video sequence to be rendered may include a large number of unrendered video frames, for ease of description, any two Two adjacent unrendered video frames are introduced as an example. In practical applications, the processing methods for the other two adjacent frames are similar, and will not be repeated here.
- the terminal device first obtains two adjacent video frames from the video sequence to be rendered, that is, the first unrendered video frame and the second unrendered video frame, and then generates a new one based on the two unrendered video frames
- the video frame is to generate a target video frame, where the target video frame is a frame located between the first unrendered video frame and the second unrendered video frame, which is equivalent to inserting an additional frame of image.
- the first video subsequence in the target video sequence is generated. In practical applications, a series of video subsequences are generated in a similar manner, and finally Generate the target video sequence.
- Figure 12 is a schematic diagram of an embodiment of implementing interpolation frame processing based on a terminal device in an embodiment of the application.
- the cloud game server captures the rendered One frame of game picture, when multiple frames of rendered game pictures are obtained, the original video sequence is obtained, and then the original video sequence is coded to obtain the video coding sequence.
- the cloud game server transmits the video encoding sequence to the terminal device through the network, and then the terminal device decodes the video encoding sequence to obtain the video sequence to be rendered.
- the terminal device can perform interpolation frame processing according to the nth frame of game screen and the n+1th frame of game screen to obtain an additional frame of game screen, that is, the n+0.5th frame of game screen.
- the terminal device continues to obtain the n+2th frame of the game picture, and performs interpolation frame processing on the n+1th frame of the game picture and the n+2th frame of the game picture to obtain an additional frame of the game picture, that is, the n+1.5th frame.
- Frame the game screen is composed of game images such as frame n, frame n+0.5, frame n+1, frame n+1.5, and frame n+2, which are finally displayed on the interface of the terminal device after rendering .
- the first unrendered video frame is the nth frame and the second unrendered video frame is the n+1th frame.
- the target video frame is generated after interpolated frame processing , Where the target video frame is the n+0.5th frame.
- the server renders 30 frames of pictures
- the actual encoded pictures can be 30 frames
- the pictures output to the terminal device are also 30 frames. Therefore, for the terminal device, the number of decoded pictures is 30 frames.
- 60 frames of images can be rendered.
- the image effect of the interpolated frame is better, and it is suitable for services that do not require high latency but require high image quality, such as non-real-time battle games.
- a method for obtaining the target video sequence based on the interpolated frame is provided.
- the target video sequence obtained by using the interpolated frame has better image effects, but due to the cloud service scenario One frame delay, therefore, is more suitable for services that do not require high delay, but require high picture quality.
- the picture quality can also be improved.
- the terminal device performs the processing of the first unrendered video frame and the second unrendered video frame Perform frame insertion processing to obtain the target video frame, which may include:
- the terminal device obtains the first frame number corresponding to the first unrendered video frame
- the terminal device obtains the second frame number corresponding to the second unrendered video frame
- the terminal device obtains the target frame number according to the first frame number and the second frame number
- the terminal device obtains K pixel information corresponding to the target video frame through the interpolation frame prediction model based on the target frame number, wherein the K pixel information is used to determine the target video frame, and K is an integer greater than 1.
- a method for interpolating frames based on an interpolated frame prediction model is introduced. It should be noted that since the video sequence to be rendered may include a large number of unrendered video frames, for ease of description, any Two adjacent unrendered video frames are introduced as an example. In practical applications, the processing methods for the other two adjacent frames are similar, and will not be repeated here.
- each unrendered video frame corresponds to a frame number, where the frame number of the first unrendered video frame is the first frame number, and the frame number of the second unrendered video frame is the second frame number.
- the frame number is n
- the second frame number is n+1
- the target frame number is calculated as follows:
- U represents the target frame number, which is n+0.5.
- K represents the total number of pixels included in a video frame.
- the pixel information may be expressed in YUV format, or may be expressed in RGB format.
- a method for interpolating frames based on the interpolation frame prediction model is provided.
- the trained interpolation frame prediction model can be used to predict each pixel information in the target video frame, and then The target video frame is reconstructed from these pixel information, so as to realize the process of interpolating one frame to the video frame, thereby improving the feasibility and operability of the solution.
- the terminal device obtains the target video sequence according to the video sequence to be rendered, which may include:
- the terminal device obtains the first unrendered video frame and the second unrendered video frame from the video sequence to be rendered, where the first unrendered video frame is the previous image adjacent to the second unrendered video frame;
- the terminal device performs frame insertion processing on the first unrendered video frame and the second unrendered video frame to obtain a target video frame, where the target video frame is the next frame image adjacent to the second unrendered video frame;
- the terminal device generates a second video subsequence in the target video sequence according to the first unrendered video frame, the target video frame, and the second unrendered video frame, where the second video subsequence consists of the first unrendered video frame and the second unrendered video frame. 2. It is composed of unrendered video frames and target video frames.
- a method of acquiring a target video sequence based on extrapolated frames is introduced.
- the video sequence to be rendered may include a large number of unrendered video frames, for ease of description, any two Two adjacent unrendered video frames are introduced as an example. In practical applications, the processing methods for the other two adjacent frames are similar, and will not be repeated here.
- the terminal device first obtains two adjacent video frames from the video sequence to be rendered, that is, the first unrendered video frame and the second unrendered video frame, and then generates a new one based on the two unrendered video frames
- a video frame is to generate a target video frame, where the target video frame is a frame between the second unrendered video frame and the next unrendered video frame, which is equivalent to inserting an additional frame of image.
- the second video subsequence in the target video sequence is generated. In practical applications, a series of video subsequences are generated in a similar manner, and finally Generate the target video sequence.
- FIG. 13 is a schematic diagram of an embodiment of implementing extrapolation frame processing based on a terminal device in an embodiment of the application.
- the cloud game server captures the rendered One frame of game picture, when multiple frames of rendered game pictures are obtained, the original video sequence is obtained, and then the original video sequence is coded to obtain the video coding sequence.
- the cloud game server transmits the video encoding sequence to the terminal device through the network, and then the terminal device decodes the video encoding sequence to obtain the video sequence to be rendered.
- the terminal device can perform extrapolation processing according to the nth frame of game screen and the n+1th frame of game screen to obtain an additional frame of game screen, that is, the n+1.5th frame of game screen. Similarly, the terminal device continues to obtain the n+2th frame of the game picture, and performs extrapolation processing on the n+1th frame of the game picture and the n+2th frame of the game picture to obtain an extra frame of the game picture, that is, the n+2.5th frame. Frame the game screen.
- the target video sequence is composed of game images such as the nth frame, the n+1th frame, the n+1.5th frame, and the n+2th frame, which are finally rendered and displayed on the interface of the terminal device.
- the first unrendered video frame is the nth frame and the second unrendered video frame is the n+1th frame.
- the target video frame is generated after the extrapolation frame processing , Where the target video frame is the n+1.5th frame.
- the server renders 30 frames of pictures
- the actual encoded picture can be 30 frames
- the picture output to the terminal device is also 30 frames. Therefore, for the terminal device, the number of decoded pictures is 30 frames.
- 60 frames of images can be rendered. Extrapolated frames usually do not generate additional delay, and are suitable for services that require high delay, such as real-time battle games.
- a method for obtaining the target video sequence based on the extrapolated frame is provided.
- the target video sequence obtained by using the extrapolated frame usually does not cause additional delay, so it is more suitable for delay requirements. Businesses that are relatively high, but do not require high picture quality. Therefore, while saving the server-side processor overhead, it can also avoid the phenomenon of picture delay caused by time delay.
- the terminal device performs the processing of the first unrendered video frame and the second unrendered video frame Perform frame insertion processing to obtain the target video frame, which may include:
- the terminal device obtains the second frame number corresponding to the second unrendered video frame
- the terminal device determines the next adjacent frame number of the second frame number as the third frame number
- the terminal device obtains the target frame number according to the average value of the second frame number and the third frame number;
- the terminal device obtains K pixel information corresponding to the target video frame through the interpolation frame prediction model based on the target frame number, wherein the K pixel information is used to determine the target video frame, and K is an integer greater than 1.
- a method for extrapolating frames based on the interpolation frame prediction model is introduced. It should be noted that since the video sequence to be rendered may include a large number of unrendered video frames, for ease of description, any Two adjacent unrendered video frames are introduced as an example. In practical applications, the processing methods for the other two adjacent frames are similar, and will not be repeated here.
- each unrendered video frame corresponds to a frame number, where the frame number of the first unrendered video frame is the first frame number, and the frame number of the second unrendered video frame is the second frame number.
- the frame number is n+1, and the third frame number is n+2.
- the calculation method of the target frame number is as follows:
- U represents the target frame number, which is n+1.5.
- K represents the total number of pixels included in a video frame.
- the pixel information may be expressed in YUV format, or may be expressed in RGB format.
- a method for extrapolating frames based on the interpolation frame prediction model is provided.
- the trained interpolation frame prediction model can be used to predict each pixel information in the target video frame, and then The target video frame is reconstructed from these pixel information, so as to realize the process of extrapolating a frame to the video frame, thereby improving the feasibility and operability of the solution.
- FIG. 14 Another embodiment of the video processing method in the embodiment of this application includes:
- the server obtains an original video sequence, where the original video sequence includes P video frames obtained after rendering, and P is an integer greater than or equal to 2.
- the server acquires continuous P frame pictures, that is, P video frames are acquired, and these P video frames may constitute the original video sequence.
- the video frame may refer to a rendered video picture, may also refer to a rendered game picture, or may also refer to other types of pictures that have been rendered.
- the server involved in this application can be a local server (for example, a game server or a video server, etc.), or a cloud server (for example, a cloud game server or a cloud video server).
- a cloud game server applied to the cloud game business is introduced as an example.
- the CPU and the graphics GPU can perform rendering tasks, which should not be construed as a limitation of this application.
- the server obtains a first target video sequence according to the original video sequence, where the first target video sequence includes P video frames obtained after rendering and Q unrendered video frames inserted into the P video frames, where Q is greater than or An integer equal to 1.
- the server performs frame insertion processing on P video frames in the original video sequence to obtain the first target video sequence.
- the resources consumed by inserting one frame of image are less than the resources consumed for rendering one frame of image. It is understandable that the frame insertion processing method is similar to that of step 102, so it will not be repeated here.
- the server performs encoding processing on the first target video sequence to obtain a video encoding sequence.
- the server uses an encoding algorithm to encode the first target video sequence, thereby generating a video encoding sequence.
- the encoding process can run on the CPU of the server, on the GPU, or on other encoding hardware, for example, the encoding chip in the GPU or a dedicated encoding chip independent of the GPU.
- the coding algorithm can adopt H264, H265, VP8 or VP9, etc., which is not limited here. It is understandable that the encoding process can refer to the content described in step 103, so it will not be repeated here.
- the server sends the video coding sequence to the terminal device.
- the server transmits the video coding sequence to the terminal device via the network, and the terminal device receives the video coding sequence, where the video coding sequence includes (P+Q) video frames.
- the terminal device decodes the video encoding sequence to obtain a video sequence to be rendered, where the video sequence to be rendered includes (P+Q) video frames that have not been rendered.
- the terminal device can decode the video coding sequence to obtain the video sequence to be rendered, and the video sequence to be rendered includes (P+Q) video frames that have not been rendered, that is, the video sequence to be rendered includes The number of video frames is the same as the number of video frames included in the video coding sequence.
- the terminal device obtains a second target video sequence according to the video sequence to be rendered, where the second target video sequence includes (P+Q) video frames that have not been rendered and Y inserted into (P+Q) video frames. Unrendered video frames, Y is an integer greater than or equal to 1.
- the terminal device performs frame interpolation processing on (P+Q) video frames in the video sequence to be rendered. It can perform frame interpolation processing on two adjacent video frames, or it can perform interpolation processing on two related video frames. For frame processing, interval video frames can also be used for frame interpolation processing, which is not limited here.
- Y video frames are generated after the frame is inserted, and these (P+Q) video frames and Y video frames together constitute a second target video sequence.
- the method of inserting frames includes but is not limited to frame sampling, frame mixing, motion compensation, and optical flow method, which will not be repeated here.
- the terminal device performs rendering processing on the second target video sequence to obtain the target video.
- the terminal device performs rendering processing on the second target video sequence to generate the target video and display the target video on the screen of the terminal device.
- the decoding process can run on the CPU of the terminal device, or on the GPU, or on other decoding hardware, for example, a decoding chip in the GPU or a dedicated decoding chip independent of the GPU. After a video frame is decoded, the video frame can be read by the CPU or GPU on the terminal device side, rendered, and displayed on the interface.
- a video processing method is provided. Both sides of the server and the terminal device generate video frames by inserting frames.
- the performance of the server and the terminal device can be integrated.
- the server only It is necessary to render some video frames, and then perform frame interpolation processing based on the rendered video frames to obtain the video sequence to be encoded.
- the target video sequence is generated in the manner of frame interpolation, and the target video is rendered.
- the frame insertion processing consumes less resources than the rendering resources. Therefore, it can not only save server-side processing resources, reduce processor overhead, and help improve the service processing capabilities of the server, but also imposes requirements on the performance of terminal equipment.
- the reduction effectively balances the performance of the server and terminal equipment. The whole process saves transmission bandwidth for the client.
- the first method is that both the server and the terminal device adopt the processing method of interpolating frames.
- FIG. 15 is a schematic diagram of an embodiment of implementing interpolation frame processing based on a video processing system in an embodiment of the application.
- the cloud game server captures the rendered nth frame For the game screen and the n+1th frame of the game screen, interpolating frame processing is performed on the nth frame of the game screen and the n+1th frame of the game screen to obtain an additional frame of the game screen, that is, the n+0.5th frame of the game screen.
- the cloud game server continues to obtain the n+2th frame of the game screen, and performs interpolation frame processing on the n+1th frame of the game screen and the n+2th frame of the game screen to obtain an additional frame of the game screen, that is, the n+th frame. 1.5 frames of game screen.
- the first target video sequence is composed of game pictures such as the nth frame, the n+0.5th frame, the n+1th frame, the n+1.5th frame, and the n+2th frame, and then the first target video sequence Perform encoding to obtain a video encoding sequence.
- the cloud game server transmits the video encoding sequence to the terminal device through the network, and the terminal device decodes the video encoding sequence to obtain the video sequence to be rendered. Then, the interpolation frame processing is performed according to the nth frame of game picture and the n+0.5th frame of game picture to obtain an additional frame of game picture, that is, the n+0.25th frame of game picture. Similarly, the terminal device continues to obtain the n+1th frame of the game picture, and performs interpolation frame processing on the n+0.5th frame of the game picture and the n+1th frame of the game picture to obtain an additional frame of the game picture, that is, the n+0.75th frame. Frame the game screen.
- the game images of the nth frame, n+0.25th frame, n+0.5th frame, n+0.75th frame, and n+1th frame together constitute the second target video sequence, which is displayed on the terminal device after rendering interface.
- the second method is that both the server and the terminal device adopt the processing method of extrapolating frames.
- Figure 16 is a schematic diagram of an embodiment of implementing extrapolated frame processing based on a video processing system in an embodiment of the application. As shown in the figure, taking a cloud game scene as an example, the cloud game server captures the rendered nth frame For the game screen and the n+1th frame of the game screen, extrapolate frame processing is performed on the nth frame of the game screen and the n+1th frame of the game screen to obtain an additional frame of the game screen, namely the n+1.5th frame of the game screen.
- the cloud game server continues to obtain the n+2th frame of the game picture, and performs extrapolation processing on the n+1th frame of the game picture and the n+2th frame of the game picture to obtain an extra frame of the game picture, that is, the n+th frame. 2.5 frames of game screen.
- the game images of the nth frame, n+1th frame, n+1.5th frame, n+2th frame and n+2.5th frame together constitute the first target video sequence, and then the first target video sequence Perform encoding to obtain a video encoding sequence.
- the cloud game server transmits the video encoding sequence to the terminal device through the network, and the terminal device decodes the video encoding sequence to obtain the video sequence to be rendered.
- extrapolate frame processing is performed according to the n+1th frame of the game screen and the n+1.5th frame of the game screen to obtain an additional frame of game screen, that is, the n+1.75th frame of game screen.
- the terminal device continues to obtain the n+2th frame of the game picture, and performs extrapolation processing on the n+1.5th frame of the game picture and the n+2th frame of the game picture to obtain an extra frame of the game picture, that is, the n+2.25th frame. Frame the game screen.
- the game screens such as the nth frame, the n+1th frame, the n+1.5th frame, the n+1.75th frame, the n+2th frame, the n+2.25th frame, and the n+2.5th frame together constitute the game screen.
- the target video sequence is rendered and displayed on the interface of the terminal device.
- the third method is that the server adopts the interpolation frame, and the terminal equipment adopts the processing method of the extrapolation frame.
- Figure 17 is a schematic diagram of an embodiment of implementing interpolation frame processing based on a video processing system in an embodiment of the application. As shown in the figure, taking a cloud game scene as an example, the cloud game server captures the rendered nth frame The game screen and the n+1th frame of the game screen are interpolated frame processing on the nth frame of the game screen and the n+1th frame of the game screen to obtain an additional frame of the game screen, that is, the n+0.5th frame of the game screen.
- the cloud game server continues to obtain the n+2th frame of the game screen, and performs interpolation frame processing on the n+1th frame of the game screen and the n+2th frame of the game screen to obtain an additional frame of the game screen, that is, the n+th frame. 1.5 frames of game screen.
- the first target video sequence is composed of game pictures such as the nth frame, the n+0.5th frame, the n+1th frame, the n+1.5th frame, and the n+2th frame, and then the first target video sequence Perform encoding to obtain a video encoding sequence.
- the cloud game server transmits the video encoding sequence to the terminal device through the network, and the terminal device decodes the video encoding sequence to obtain the video sequence to be rendered. Then, extrapolate frame processing is performed according to the nth frame of game screen and the n+0.5th frame of game screen to obtain an additional frame of game screen, that is, the n+0.75th frame of game screen. Similarly, the terminal device continues to obtain the n+1th frame of the game picture, and performs extrapolation processing on the n+0.5th frame of the game picture and the n+1th frame of the game picture to obtain an additional frame of the game picture, that is, the n+1.25th frame. Frame the game screen.
- the fourth method is that the server adopts the extrapolated frame, and the terminal device adopts the processing method of the interpolated frame.
- FIG. 18 is a schematic diagram of another embodiment of implementing interpolation frame processing based on a video processing system in an embodiment of the application. As shown in the figure, taking a cloud game scene as an example, the cloud game server captures the rendered nth Frame the game screen and the n+1th frame of the game screen, and perform extrapolation processing on the nth frame of the game screen and the n+1th frame of the game screen to obtain an additional frame of the game screen, that is, the n+1.5th frame of the game screen.
- the cloud game server continues to obtain the n+2th frame of the game picture, and performs extrapolation processing on the n+1th frame of the game picture and the n+2th frame of the game picture to obtain an extra frame of the game picture, that is, the n+th frame. 2.5 frames of game screen.
- the game images of the nth frame, n+1th frame, n+1.5th frame, n+2th frame and n+2.5th frame together constitute the first target video sequence, and then the first target video sequence Perform encoding to obtain a video encoding sequence.
- the cloud game server transmits the video encoding sequence to the terminal device through the network, and the terminal device decodes the video encoding sequence to obtain the video sequence to be rendered.
- the interpolation frame processing is performed according to the nth frame of the game screen and the n+1th frame of the game screen to obtain an additional frame of the game screen, that is, the n+0.5th frame of the game screen.
- the terminal device continues to obtain the n+1.5th frame of the game picture, and performs interpolation frame processing on the n+1th frame of the game picture and the n+1.5th frame of the game picture to obtain an additional frame of the game picture, that is, the n+1.25th frame. Frame the game screen.
- the n+2.5th frame and other game pictures together constitute the second target video sequence, which is displayed on the interface after rendering.
- FIG. 19 is a schematic diagram of an embodiment of the video processing device in an embodiment of the application.
- the video processing device 40 includes:
- the obtaining module 401 is configured to obtain an original video sequence, where the original video sequence includes P video frames obtained after rendering, and P is an integer greater than or equal to 2;
- the obtaining module 401 is further configured to obtain a target video sequence according to the original video sequence, where the target video sequence includes P video frames and Q unrendered video frames inserted into the P video frames, and Q is an integer greater than or equal to 1;
- the encoding module 402 is configured to perform encoding processing on the target video sequence to obtain a video encoding sequence
- the sending module 403 is used to send the video coding sequence to the terminal device, so that the terminal device can decode the video coding sequence to obtain the video sequence to be rendered.
- the acquiring module 401 is specifically configured to acquire the first rendered video frame and the second rendered video frame from the original video sequence, where the first rendered video frame is the previous image adjacent to the second rendered video frame;
- the image of the previous frame adjacent to the rendered video frame, and the target video frame is an unrendered video frame
- the first video subsequence in the target video sequence is generated, where the first video subsequence consists of the first rendered video frame and the target video frame in turn.
- the second rendered video frame composition is generated.
- the obtaining module 401 is specifically configured to obtain the first frame number corresponding to the first rendered video frame
- K pixel information corresponding to the target video frame is obtained through the interpolation frame prediction model, where the K pixel information is used to determine the target video frame, and K is an integer greater than 1.
- the acquiring module 401 is specifically configured to acquire the first rendered video frame and the second rendered video frame from the original video sequence, where the first rendered video frame is the previous image adjacent to the second rendered video frame;
- a second video subsequence in the target video sequence is generated. Rendered video frame and target video frame composition.
- the obtaining module 401 is specifically configured to obtain the second frame number corresponding to the second rendered video frame
- the target frame number is obtained
- K pixel information corresponding to the target video frame is obtained through the interpolation frame prediction model, where the K pixel information is used to determine the target video frame, and K is an integer greater than 1.
- FIG. 20 is a schematic diagram of an embodiment of a video processing device in an embodiment of this application.
- the video processing device 50 includes:
- the receiving module 501 is configured to receive the video coding sequence sent by the server;
- the decoding module 502 is configured to decode the video encoding sequence to obtain a video sequence to be rendered, where the video sequence to be rendered includes X video frames that have not been rendered, and X is an integer greater than or equal to 2;
- the obtaining module 503 is configured to obtain a target video sequence according to the video sequence to be rendered, where the target video sequence includes X unrendered video frames and Y unrendered video frames inserted into the X video frames, where Y is An integer greater than or equal to 1;
- the rendering module 504 is configured to perform rendering processing on the target video sequence to obtain the target video.
- the obtaining module 503 is specifically configured to obtain the first unrendered video frame and the second unrendered video frame from the video sequence to be rendered, where the first unrendered video frame is the previous frame adjacent to the second unrendered video frame image;
- the first video subsequence in the target video sequence is generated, wherein the first video subsequence is sequentially composed of the first unrendered video frame and the target video frame.
- the second unrendered video frame composition is generated.
- the obtaining module 503 is specifically configured to obtain the first frame number corresponding to the first unrendered video frame
- K pixel information corresponding to the target video frame is obtained through the interpolation frame prediction model, where the K pixel information is used to determine the target video frame, and K is an integer greater than 1.
- the obtaining module 503 is specifically configured to obtain the first unrendered video frame and the second unrendered video frame from the video sequence to be rendered, where the first unrendered video frame is the previous frame adjacent to the second unrendered video frame image;
- a second video subsequence in the target video sequence is generated, wherein the second video subsequence is sequentially composed of the first unrendered video frame and the second unrendered video frame Rendered video frame and target video frame composition.
- the obtaining module 503 is specifically configured to obtain the second frame number corresponding to the second unrendered video frame
- the target frame number is obtained
- K pixel information corresponding to the target video frame is obtained through the interpolation frame prediction model, where the K pixel information is used to determine the target video frame, and K is an integer greater than 1.
- FIG. 21 is a schematic diagram of a server structure provided by an embodiment of the present application.
- the server 600 may have relatively large differences due to different configurations or performances, and may include one or more CPUs 622 (for example, one or more processors) and The memory 632, one or more storage media 630 (for example, one or one storage device with a large amount of storage) storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 may be short-term storage or persistent storage.
- the program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of command operations on the server.
- the central processing unit 622 may be configured to communicate with the storage medium 630, and execute a series of instruction operations in the storage medium 630 on the server 600.
- the server 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and/or one or more operating systems 641, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM and so on.
- operating systems 641 such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM and so on.
- the steps performed by the server in the foregoing embodiment may be based on the server structure shown in FIG. 21.
- the embodiment of the present application also provides another video processing device, as shown in FIG. 22.
- the terminal device can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a car computer, a TV, an OTT box, etc.
- the terminal device is a mobile phone
- PDA personal digital assistant
- POS point of sales
- car computer a TV, an OTT box, etc.
- the terminal device is a mobile phone
- FIG. 22 shows a block diagram of a part of the structure of a mobile phone related to a terminal device provided in an embodiment of the present application.
- the mobile phone includes components such as a radio frequency (RF) circuit 710, a memory 720, and a processor 780.
- RF radio frequency
- FIG. 22 does not constitute a limitation on the mobile phone, and may include more or less components than those shown in the figure, or a combination of certain components, or different component arrangements.
- the RF circuit 710 can be used for receiving and sending signals during the process of sending and receiving information or talking. In particular, after receiving the downlink information of the base station, it is processed by the processor 780; in addition, the designed uplink data is sent to the base station.
- the RF circuit 710 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like.
- the RF circuit 710 can also communicate with the network and other devices through wireless communication.
- the above-mentioned wireless communication can use any communication standard or protocol, including but not limited to global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (code division multiple access, GSM) multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.
- GSM global system of mobile communication
- GPRS general packet radio service
- code division multiple access code division multiple access
- GSM code division multiple access
- CDMA code division multiple access
- WCDMA wideband code division multiple access
- LTE long term evolution
- email short messaging service
- SMS short messaging service
- the memory 720 may be used to store software programs and modules.
- the processor 780 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 720.
- the memory 720 may mainly include a program storage area and a data storage area.
- the program storage area may store an operating system, an application program required by at least one function (such as a sound playback function, an image playback function, etc.), etc.; Data created by the use of mobile phones (such as audio data, phone book, etc.), etc.
- the memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.
- the processor 780 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. Various functions and processing data of the mobile phone can be used to monitor the mobile phone as a whole.
- the processor 780 may include one or more processing units; optionally, the processor 780 may integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, and application programs. And so on, the modem processor mainly deals with wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 780.
- the mobile phone may also include a camera, a Bluetooth module, etc., which will not be repeated here.
- the steps performed by the terminal device in the foregoing embodiment may be based on the terminal device structure shown in FIG. 22.
- FIG. 23 is a schematic structural diagram of the video processing system in an embodiment of this application.
- the server 801 obtains the original video sequence, where the original video sequence includes For P video frames obtained after rendering, P is an integer greater than or equal to 2.
- the server 801 obtains the first target video sequence according to the original video sequence, where the first target video sequence includes P video frames obtained after rendering and Q unrendered video frames inserted into the P video frames, where Q is greater than or equal to An integer of 1.
- the server 801 performs encoding processing on the first target video sequence to obtain a video encoding sequence.
- the server 801 sends the video coding sequence to the terminal device 802.
- the terminal device 802 decodes the video encoding sequence to obtain a video sequence to be rendered, where the video sequence to be rendered includes (P+Q) video frames that have not been rendered.
- the terminal device 802 obtains a second target video sequence according to the video sequence to be rendered, where the second target video sequence includes (P+Q) video frames that have not been rendered and inserted into the (P+Q) video frames Y unrendered video frames, Y is an integer greater than or equal to 1.
- the terminal device 802 performs rendering processing on the second target video sequence to obtain the target video.
- the embodiments of the present application also provide a computer-readable storage medium.
- the computer-readable storage medium stores a computer program. When it runs on a computer, the computer executes the video processing device in the method described in the foregoing various embodiments. The steps performed.
- the embodiments of the present application also provide a computer program product including a program, which when running on a computer, causes the computer to execute the steps performed by the video processing apparatus in the methods described in the foregoing embodiments.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Television Systems (AREA)
Abstract
一种视频处理的方法,包括:获取原始视频序列,其中,原始视频序列包括经过渲染后得到的P个视频帧,P为大于或等于2的整数;根据原始视频序列获取目标视频序列,目标视频序列包括经过渲染后得到的P个视频帧以及***P个视频帧中的Q个未渲染视频帧,Q为大于或等于1的整数;对目标视频序列进行编码处理,得到视频编码序列;向终端设备发送视频编码序列,以使终端设备对视频编码序列进行解码处理,得到待渲染的视频序列。本申请还提供了装置、存储介质及计算机程序产品,本申请可以节省服务器侧的处理资源,减少处理器的开销,有利于提升服务器的业务处理能力。
Description
本申请要求于2020年5月8日提交中国专利局、申请号为202010381049.1、申请名称为“一种视频处理的方法、相关装置及存储介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及基于云技术的行业应用领域,具体涉及一种视频处理技术。
云游戏是一种以云计算技术为基础的在线游戏技术。在云游戏的运行模式下,所有游戏都在服务器端运行,并将渲染完毕后的游戏画面压缩后通过网络传送给用户。用户使用的终端设备不需要高端处理器和显卡,只需要基本的视频解压能力即可。
目前,基于云游戏的业务流程为,首先终端设备连接至云游戏服务器,然后通过数据流和控制流实现用户与游戏之间的交互,其中,数据流主要包括游戏画面数据,即云游戏服务器先对游戏画面进行编码,然后将编码后的游戏画面数据传输至终端设备,由终端设备进行解码后呈现在界面上。
然而,在现有的业务流程中,云游戏服务器对需要对每帧游戏画面进行渲染,再基于渲染后的视频帧进行编码,整个过程需要消耗较多的处理资源,造成处理器的开销过大,降低云游戏服务器的业务处理能力。
发明内容
本申请一方面提供一种视频处理的方法,由服务器执行,包括:
获取原始视频序列,原始视频序列包括经过渲染后得到的P个视频帧,P为大于或等于2的整数;
根据原始视频序列获取目标视频序列,目标视频序列包括P个视频帧以及***所述P个视频帧中的Q个未渲染视频帧,Q为大于或等于1的整数;
对目标视频序列进行编码处理,得到视频编码序列;
向终端设备发送视频编码序列,以使终端设备对视频编码序列进行解码处理,得到待渲染的视频序列。
本申请另一方面提供一种视频处理的方法,由终端设备执行,包括:
接收服务器发送的视频编码序列;
对视频编码序列进行解码处理,得到待渲染的视频序列,待渲染的视频序列包括未经过渲染的X个视频帧,X为大于或等于2的整数;
根据待渲染的视频序列获取目标视频序列,目标视频序列包括未经过渲染的X个视频帧以及***所述X个视频帧中的Y个未渲染视频帧,Y为大于或等于1的整数;
对目标视频序列进行渲染处理,得到目标视频。
本申请另一方面提供上述视频处理的方法,所对应的视频处理装置、计算机设备、计算机可读存储介质以及程序产品。
上述本申请实施例的视频处理的方法,服务器先获取原始视频序列,该原始视频序列包括经过渲染后得到的P个视频帧,然后根据原始视频序列获取目标视频序列,目标视频序列还包括***P个视频帧中的Q个未渲染视频帧,然后服务器对目标视频序列进行编码处理,得到视频编码序列,最后服务器向终端设备发送视频编码序列,以使终端设备对视频编码序列进行解码处理,得到待渲染的视频序列。通过上述方式,服务器只需要对部分视频帧进行渲染,再基于渲染后的视频帧进行插帧处理,由此得到目标视频序列,而插帧处理所消耗的资源少于渲染所需的资源,因此,可以节省服务器侧的处理资源,减少处理器的开销,有利于提升服务器的业务处理能力。
图1为本申请实施例中视频处理***的一个架构示意图;
图2为本申请实施例中视频处理***的一个云架构示意图;
图3为本申请实施例中视频处理***的一个云游戏架构示意图;
图4为本申请实施例中视频处理方法的一个实施例示意图;
图5为本申请实施例中基于服务器实现内插帧处理的一个实施例示意图;
图6为本申请实施例中内插帧模式下生成目标视频帧的一个示意图;
图7为本申请实施例中训练插帧预测模型的一个实施例示意图;
图8为本申请实施例中验证插帧预测模型的一个实施例示意图;
图9为本申请实施例中基于服务器实现外插帧处理的一个实施例示意图;
图10为本申请实施例中外插帧模式下生成目标视频帧的一个示意图;
图11为本申请实施例中视频处理方法的另一个实施例示意图;
图12为本申请实施例中基于终端设备实现内插帧处理的一个实施例示意图;
图13为本申请实施例中基于终端设备实现外插帧处理的一个实施例示意图;
图14为本申请实施例中视频处理方法的一个交互流程示意图;
图15为本申请实施例中基于视频处理***实现内插帧处理的一个实施例示意图;
图16为本申请实施例中基于视频处理***实现外插帧处理的一个实施例示意图;
图17为本申请实施例中基于视频处理***实现内外插帧处理的一个实施例示意图;
图18为本申请实施例中基于视频处理***实现内外插帧处理的另一个实施例示意图;
图19为本申请实施例中视频处理装置的一个实施例示意图;
图20为本申请实施例中视频处理装置的另一个实施例示意图;
图21为本申请实施例中服务器的一个结构示意图;
图22为本申请实施例中终端设备的一个结构示意图;
图23为本申请实施例中视频处理***的一个结构示意图。
本申请实施例提供了一种视频处理的方法、相关装置及存储介质,可以节省服务器侧的处理资源,减少处理器的开销,有利于提升服务器的业务处理能力。
应理解,本申请提供的视频处理方法适用于远程网络下的业务场景以及云技术下的业务场景,其中,云技术下的业务场景包含但不仅限于云游戏业务或者云视频业务等。以云游戏业务为例,本申请利用动态插帧的技术,在玩家体验每秒60帧(frame)游戏画面的需求下,云游戏服务器上只需要每秒运行30帧游戏画面即可,由此节省处理器的开销。以云视频业务为例,本申请利用动态插帧的技术,在观众体验每秒24帧动画的需求下,云视频服务器上只需要每秒运行12帧动画即可,由此节省处理器的开销。
为了便于理解,本申请提出了一种视频处理的方法,该方法可应用于图1所示的视频处理***,请参阅图1,图1为本申请实施例中视频处理***的一个架构示意图,如图所示,一台服务器可以为多台终端设备提供业务,例如,服务器1可以与M台终端设备建立通信连接,服务器2可以与N台终端建立通信连接,M和N的取值视服务器的处理能力而定,通常情况下,N和M可以取值100。
可以理解的是,图1所示的服务器可以是独立的物理服务器,也可以是多个物理服务器构成的服务器集群或者分布式***,此处不做限定。
为了便于理解,本申请提出了一种视频处理的方法,该方法还可应用于图2所示的视频处理***,请参阅图2,图2为本申请实施例中视频处理***的一个云架构示意图,如图所示,多台云服务器共同构成一个云服务器集群,一个云服务器集群可以为多台终端设备提供业务,例如,云服务器集群包括4台云服务器,该云服务器集群可以与M台终端设备建立通信连接。
可以理解的是,图2所示的云服务器可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、内容分发网络(content delivery network,CDN)以及大数据和人工智能平台等基础云计算服务的云服务器。图1和图2所示的终端设备可以是智能手机、平板电脑、电视以及网络机顶盒(over the top,OTT)、笔记本电脑、台式计算机、智能音箱、智能手表等,但并不局限于此。终端设备上部署有客户端,例如,视频客户端或者游戏客户端等。终端设备以及服务器可以通过有线或无线通信方式进行直接或间接地连接,本申请在此不做限制。
基于此,在图2所示的架构中采用了云技术(cloud technology)以实现 业务通信,其中,云技术是指在广域网或局域网内将硬件、软件、网络等系列资源统一起来,实现数据的计算、储存、处理和共享的一种托管技术。云技术基于云计算商业模式应用的网络技术、信息技术、整合技术、管理平台技术、应用技术等的总称,可以组成资源池,按需所用,灵活便利。云计算技术将变成重要支撑。技术网络***的后台服务需要大量的计算、存储资源,如视频网站、图片类网站和更多的门户网站。伴随着互联网行业的高度发展和应用,将来每个物品都有可能存在自己的识别标志,都需要传输到后台***进行逻辑处理,不同程度级别的数据将会分开处理,各类行业数据皆需要强大的***后盾支撑,只能通过云计算来实现。
下面将以应用于云游戏业务场景为例进行介绍,请参阅图3,图3为本申请实施例中视频处理***的一个云游戏架构示意图,如图所示,当终端设备和云游戏服务器交互时会存在控制流传输和数据流传输,控制流主要负责发送控制信号,当用户通过输入设备(例如键盘、鼠标或者摇杆)触发操作时,信号会被编码到发送列表中,通过网络传输至云游戏服务器。同时,云游戏服务器需要向终端设备传输数据流,数据流包括音频数据流和视频数据流,终端设备再将数据流进行反解码,然后呈现在屏幕上。
云游戏(cloud gaming)又可称为游戏点播(gaming on demand),是一种以云计算技术为基础的在线游戏技术。云游戏技术使图形处理与数据运算能力相对有限的轻端设备(thin client)能运行高品质游戏。在云游戏场景下,游戏并不在玩家游戏终端,而是在云端服务器中运行,并由云端服务器将游戏场景渲染为视频音频流,通过网络传输给玩家游戏终端。玩家游戏终端无需拥有强大的图形运算与数据处理能力,仅需拥有基本的流媒体播放能力与获取玩家输入指令并发送给云端服务器的能力即可。与传统游戏模式相比,云游戏能在很大程度上减小玩家游玩游戏的设备成本。对于许多需要长期更新的高品质游戏而言,云游戏也能减少游戏商发行与更新维护游戏的成本。
在保证玩家游戏体验上,游戏场景渲染的多媒体流质量取决于网络通信带宽,与传统网络游戏相比云游戏的多媒体流需要消耗更多带宽,并且画质越高的多媒体流,其消耗的带宽资源也会越高。本申请提供的视频处理方法,不但能够减少云服务器侧的资源开销,而且还可以减少视频帧的传输量,从而在保证游戏画面质量的情况下,节省网络带宽。以大型多人在线(massive multiplayer online,MMO)游戏为例,一个图形处理器(graphics processing unit,GPU)的消耗是36%,采用本申请提供的方法后,GPU的消耗为20%,在传统模式下,一个GPU上最多运行3个游戏进程,采用本申请提供的方法后,一个GPU上可运行5个游戏进程。
结合上述介绍,下面将从服务器的角度,对本申请中视频处理的方法进行介绍,请参阅图4,本申请实施例中视频处理方法的一个实施例包括:
101、服务器获取原始视频序列,其中,原始视频序列包括经过渲染后得 到的P个视频帧,P为大于或等于2的整数。
本实施例中,由服务器获取连续P帧画面,即获取到P个视频帧,这P个视频帧可构成原始视频序列。其中,视频帧可以指经过渲染后的视频画面,也可以指经过渲染后的游戏画面,还可以指经过渲染的其他类型画面。需要说明的是,本申请涉及的服务器可以是本地服务器(例如,游戏服务器或者视频服务器等),也可以是云服务器(例如云游戏服务器或者云视频服务器),为了便于介绍,本申请将以服务器为应用于云游戏业务中的云游戏服务器为例进行介绍,然而这不应理解为对本申请的限定。
画面渲染需要通过程序对绘制的图形的几何和顶点等信息进行运算,然后得到的图像。在这个过程中计算机的处理器需要进行大量运算。在实际应用中,中央处理器(central processing unit,CPU)和GPU都可以进行渲染任务。由于对画面分辨率和质量的要求越来越高,CPU的单精度浮点性能难以满足复杂画面的渲染需求,因此,可以由GPU接管大部分的图形渲染工作,而CPU需要通过一些指令调度GPU进行渲染。
102、服务器根据原始视频序列获取目标视频序列,其中,目标视频序列包括P个视频帧以及***所述P个视频帧中的Q个未渲染视频帧,Q为大于或等于1的整数。
本实施例中,服务器对原始视频序列中的P个视频帧进行插帧处理,可以对相邻两个视频帧进行插帧处理,例如,对1号视频帧和2号视频帧进行插帧处理,再对2号视频帧和3号视频帧进行插帧处理,以此类推。也可以对相关两个视频帧进行插帧处理,例如,对1号视频帧和2号视频帧进行插帧处理,再对3号视频帧和4号视频帧进行插帧处理,以此类推。还可以采用间隔视频帧进行插帧处理,例如,对1号视频帧和2号视频帧进行插帧处理,再对5号视频帧和6号视频帧进行插帧处理。基于原始视频序列生成插帧后得到的Q个视频帧,这Q个视频帧和P个视频帧共同构成目标视频序列。
可以理解的是,***一帧图像所消耗的资源少于渲染一帧图像所消耗的资源。插帧的方式包含但不仅限于帧采样,帧混合,运动补偿以及光流法。其中,帧采样是指使用拉长每个关键帧的显示时间,相当于***了相同的若干个相同的关键帧。帧混合是指***一帧后提升前后关键帧的透明度,在合成新的一帧。运动补偿是指对物体的运动进行识别,然后再进行补偿插帧。光流法是指同一个像素点的灰度值(或亮度值)在前后帧中恒定的情况下,找到像素点在前后帧的运动轨迹,再基于该运动轨迹进行预测插帧处理。
103、服务器对目标视频序列进行编码处理,得到视频编码序列。
本实施例中,服务器采用编码算法对目标视频序列进行编码处理,从而生成视频编码序列。编码过程可以运行于服务器的CPU上,也可以运行于GPU上,还可以运行于其他编码硬件上,例如,GPU内的编码芯片或者独立于GPU之外的专用编码芯片。编码算法可采用H264、H265、VP8或VP9等, 此处不做限定。
在云游戏的场景中,由于云游戏通常为低时延业务,因此,在编码过程中不会采用后向编码参考帧或者双向编码参考帧。这是因为,如果考虑后向编码参考帧或者双向编码参考帧(即B帧),则终端设备在收到当前视频帧时,还需要等待下一个视频帧的到来才能实现对当前视频帧的解码,从而造成一帧的延迟,基于此,在本申请中,服务器的编码主要采用前向编码参考帧(即P帧)以及关键帧(即I帧),而不使用双向编码参考帧(即B帧)。视频编码序列中包括至少一个图像组(group of pictures,GOP),GOP是由一个I帧和多个B帧(或P帧)组成,也是编码器和解码器存取的基本单位。其中,I帧是指携带全部信息的独立帧,无需参考其他视频帧便可进行独立解码。P帧需要参考前面的I帧才能进行编码。B帧记录本视频帧与前后两个视频帧之间的差别。
104、服务器向终端设备发送视频编码序列,以使终端设备对视频编码序列进行解码处理,得到待渲染的视频序列。
本实施例中,服务器向终端设备发送视频编码序列,于是终端设备可以对视频编码序列进行解码,从而得到待渲染的视频序列,最后对待渲染的视频序列进行渲染处理,即可生成目标视频,并在终端设备的屏幕上展示该目标视频。解码过程可以运行于终端设备的CPU上,也可以运行于GPU上,还可以运行于其他解码硬件上,例如,GPU内的解码芯片或者独立于GPU之外的专用解码芯片。解码得到一个视频帧之后,即可通过终端设备侧的CPU或者GPU读取该视频帧,并进行渲染后显示于界面。
本申请实施例中,提供了一种视频处理的方法,服务器先获取原始视频序列,该原始视频序列包括经过渲染后得到的P个视频帧,然后根据原始视频序列获取目标视频序列,目标视频序列还包括经过插帧处理后得到的Q个视频帧,然后服务器对目标视频序列进行编码处理,得到视频编码序列,最后服务器向终端设备发送视频编码序列,以使终端设备对视频编码序列进行解码处理,得到待渲染的视频序列。通过上述方式,服务器只需要对部分视频帧进行渲染,再基于渲染后的视频帧进行插帧处理,由此得到目标视频序列,而插帧处理所消耗的资源少于渲染所需的资源,因此,可以节省服务器侧的处理资源,减少处理器的开销,有利于提升服务器的业务处理能力。
可选地,在上述图4对应的各个实施例的基础上,本申请实施例提供的视频处理方法另一个可选实施例中,服务器根据原始视频序列获取目标视频序列,可以包括:
服务器从原始视频序列中获取第一已渲染视频帧以及第二已渲染视频帧,其中,第一已渲染视频帧为第二已渲染视频帧相邻的前一帧图像;
服务器对第一已渲染视频帧以及第二已渲染视频帧进行插帧处理,得到目标视频帧,其中,目标视频帧为第一已渲染视频帧相邻的后一帧图像,目 标视频帧为第二已渲染视频帧相邻的前一帧图像,且目标视频帧为未渲染视频帧;
服务器根据第一已渲染视频帧、目标视频帧以及第二已渲染视频帧,生成目标视频序列中的第一视频子序列,其中,第一视频子序列依次由第一已渲染视频帧、目标视频帧以及第二已渲染视频帧组成。
本实施例中,介绍了一种基于内插帧获取目标视频序列的方式,需要说明的是,由于原始视频序列中可能包括大量的已渲染视频帧,为了便于说明,下面将以任意两个相邻的已渲染视频帧为例介绍,在实际应用中,对其他相邻两帧的处理方式类似,此处不做赘述。
具体地,首先服务器从原始视频序列中获取相邻的两个视频帧,即第一已渲染视频帧以及第二已渲染视频帧,然后基于这两个已渲染视频帧生成新的一个视频帧,即生成目标视频帧,其中,目标视频帧是位于第一已渲染视频帧和第二已渲染视频帧之间的一帧,相当于额外***了一帧图像。按照第一已渲染视频帧、目标视频帧以及第二已渲染视频帧的顺序,生成目标视频序列中的第一视频子序列,在实际应用中,采用类似地方式生成一系列视频子序列,最终生成目标视频序列。
为了便于理解,请参阅图5,图5为本申请实施例中基于服务器实现内插帧处理的一个实施例示意图,如图所示,以云游戏场景为例,云游戏服务器捕获经过渲染的第n帧游戏画面以及第n+1帧游戏画面,对第n帧游戏画面以及第n+1帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+0.5帧游戏画面。类似地,云游戏服务器继续获取第n+2帧游戏画面,对第n+1帧游戏画面以及第n+2帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+1.5帧游戏画面。以此类推,由第n帧、第n+0.5帧、第n+1帧、第n+1.5帧以及第n+2帧等游戏画面共同构成目标视频序列,再对目标视频序列进行编码,得到视频编码序列。云游戏服务器通过网络将视频编码序列传输至终端设备,于是,终端设备对视频编码序列进行解码处理,得到待渲染视频序列,最后经过渲染显示于终端设备的界面。
基于上述介绍,请参阅图6,图6为本申请实施例中内插帧模式下生成目标视频帧的一个示意图,如图所示,假设第一已渲染视频帧为第n帧,第二已渲染视频帧为第n+1帧,经过内插帧处理后生成目标视频帧,其中,目标视频帧为第n+0.5帧。由此可见,假设服务器渲染30帧画面,则实际编码的画面可以为60帧,输出至终端设备的画面也为60帧,因此,对于终端设备而言,解码的画面数量为60帧。内插帧的图像效果较好,适用于对时延要求不高,而对画质要求较高的业务,例如,非实时对战类游戏等。
其次,本申请实施例中,提供了一种基于内插帧获取目标视频序列的方式,通过上述方式,利用内插帧得到的目标视频序列具有更好的图像效果,但由于云业务场景会产生一帧时延,因此,更适用于对时延要求不高,但是 对画面质量要求较高的业务。从而在节省服务器侧处理器开销的同时,还可以提升画面质量。
可选地,在上述图4对应的各个实施例的基础上,本申请实施例提供的视频处理方法另一个可选实施例中,服务器对第一已渲染视频帧以及第二已渲染视频帧进行插帧处理,得到目标视频帧,可以包括:
服务器获取第一已渲染视频帧对应的第一帧号;
服务器获取第二已渲染视频帧对应的第二帧号;
服务器根据第一帧号以及第二帧号,得到目标帧号;
其中,服务器可以计算第一帧号和第二帧号的平均值,得到目标帧号。在一些可能的实现方式中,服务器也可以通过其他方式确定目标帧号。进一步地,服务器基于目标帧号,通过插帧预测模型获取目标视频帧对应的K个像素信息,其中,K个像素信息用于确定目标视频帧,K为大于1的整数。
本实施例中,介绍了一种基于插帧预测模型进行内插帧的方式,需要说明的是,由于原始视频序列中可能包括大量的已渲染视频帧,为了便于说明,下面将以任意两个相邻的已渲染视频帧为例介绍,在实际应用中,对其他相邻两帧的处理方式类似,此处不做赘述。
具体地,每个已渲染视频帧对应于一个帧号,其中,第一已渲染视频帧的帧号为第一帧号,第二已渲染视频帧的帧号为第二帧号,假设第一帧号为n,第二帧号为n+1,则目标帧号的计算方式如下:
U=[n+(n+1)]/2;
其中,U表示目标帧号,即为n+0.5。
将目标帧号输入至已训练好的插帧预测模型中,通过插帧预测模型输出K个像素信息,K表示一个视频帧所包括的像素总个数,在得到K个像素对应的像素信息时,即获取到目标视频帧。像素信息可以表示为亮度带宽色度(luminance-bandwidth-chrominance,YUV)形式,或者可以表示为红绿蓝(red green blue,RGB)形式。
下面将介绍训练插帧预测模型的方式,以云游戏场景为例,通常情况下,可以为每个游戏训练一个对应插帧预测模型,也可以为多个游戏训练同一个插帧预测模型,此处不做限定。首先需要获取待训练视频,该待训练视频包括多帧训练图像,在训练的过程中,可以从待训练视频中抽取第m帧训练图像至第r帧训练图像,其中,0<m<r,在不同的插帧预测模型中,m可取不同的值。将第m帧训练图像至第r帧训练图像作为已知图像帧序列,再分别提取已知图像帧序列中每帧训练图像的像素信息。利用已知图像帧序列中每帧训练图像的像素信息,可以训练得到插帧预测模型。
本申请采用如下方式表示插帧预测模型:
f
(t,pos)≈frame
(t,pos);
其中,t表示帧号,即t的取值范围为大于或等于m,且小于或等于r的 数值。pos表示视频帧对应的坐标点(x,y)。frame
(t,pos)表示在t个视频帧中pos对应坐标点的像素信息,该像素信息可以表示为RGB形式,也可以表示为YUV形式,还可以表示为其他形式,此处不做穷举。插帧预测模型f
(t,pos)可以为一次拟合函数,或者多次拟合函数,还可以采用神经网络或深度学习方法确定的函数关系。
为了便于理解,请参阅图7,图7为本申请实施例中训练插帧预测模型的一个实施例示意图,如图所示,若需要训练内插帧时使用的插帧预测模型,则可以将第m帧训练图像和第m+2帧训练图像输入至待训练的插帧预测模型,由插帧预测模型输出目标图像,其中,该目标图像可以是预测出的第m+1帧训练图像。以此类推,直至对已知图像帧序列中各个训练图像均进行类似处理。
若需要训练外插帧时使用的插帧预测模型,则可以将第m帧训练图像和第m+1帧训练图像输入至待训练的插帧预测模型,由插帧预测模型输出目标图像,其中,该目标图像可以是预测出的第m+2帧训练图像。以此类推,直至对已知图像帧序列中各个训练图像均进行类似处理。
在预测得到多个目标图像后,还需评估插帧预测模型的质量,一种可行的评估方式为,采用如下损失函数进行计算:
L=∑(f
(t,pos)-frame
(t,pos))
2;
其中,L表示损失值。
为了便于说明,请参阅图8,图8为本申请实施例中验证插帧预测模型的一个实施例示意图,如图所示,以每秒传输帧数(frames per second,FPS)为60的已知图像帧序列为例,将所有奇数帧对应的预测图像作为插帧预测模型的输入,通过插帧预测模型后输出偶数帧对应的目标图像。将偶数帧对应的目标图像与已知图像帧序列中的预测图像进行比较。例如,将第1帧预测图像和第3帧预测图像输入至插帧预测模型,由插帧预测模型输出目标图像,将目标图像上各个像素点的像素信息与第2帧预测图像各个像素点的像素信息进行比对,如果两者差距小于阈值,则表示插帧预测模型已训练完成。
再次,本申请实施例中,提供了一种基于插帧预测模型进行内插帧的方式,通过上述方式,利用训练好的插帧预测模型能够预测出目标视频帧中的每个像素信息,再由这些像素信息重构出目标视频帧,从而实现对视频帧内插一帧的过程,由此提升方案的可行性和可操作性。
可选地,在上述图4对应的各个实施例的基础上,本申请实施例提供的视频处理方法另一个可选实施例中,服务器根据原始视频序列获取目标视频序列,可以包括:
服务器从原始视频序列中获取第一已渲染视频帧以及第二已渲染视频帧,其中,第一已渲染视频帧为第二已渲染视频帧相邻的前一帧图像;
服务器对第一已渲染视频帧以及第二已渲染视频帧进行插帧处理,得到目标视频帧,其中,目标视频帧为第二已渲染视频帧相邻的后一帧图像,且目标视频帧为未渲染视频帧;
服务器根据第一已渲染视频帧、目标视频帧以及第二已渲染视频帧,生成目标视频序列中的第二视频子序列,其中,第二视频子序列依次由第一已渲染视频帧、第二已渲染视频帧以及目标视频帧组成。
本实施例中,介绍了一种基于外插帧获取目标视频序列的方式,需要说明的是,由于原始视频序列中可能包括大量的已渲染视频帧,为了便于说明,下面将以任意两个相邻的已渲染视频帧为例介绍,在实际应用中,对其他相邻两帧的处理方式类似,此处不做赘述。
具体地,首先服务器从原始视频序列中获取相邻的两个视频帧,即第一已渲染视频帧以及第二已渲染视频帧,然后基于这两个已渲染视频帧生成新的一个视频帧,即生成目标视频帧,其中,目标视频帧是位于第二已渲染视频帧与下一帧已渲染视频帧之间的一帧,相当于额外***了一帧图像。按照第一已渲染视频帧、第二已渲染视频帧以及目标视频帧的顺序,生成目标视频序列中的第二视频子序列,在实际应用中,采用类似地方式生成一系列视频子序列,最终生成目标视频序列。
为了便于理解,请参阅图9,图9为本申请实施例中基于服务器实现外插帧处理的一个实施例示意图,如图所示,以云游戏场景为例,云游戏服务器捕获经过渲染的第n帧游戏画面以及第n+1帧游戏画面,对第n帧游戏画面以及第n+1帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+1.5帧游戏画面。类似地,云游戏服务器继续获取第n+2帧游戏画面,对第n+1帧游戏画面以及第n+2帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+2.5帧游戏画面。以此类推,由第n帧、第n+1帧、第n+1.5帧、第n+2帧以及第n+2.5帧等游戏画面共同构成目标视频序列,再对目标视频序列进行编码,得到视频编码序列。云游戏服务器通过网络将视频编码序列传输至终端设备,由终端设备对视频编码序列进行解码处理,得到待渲染视频序列,最后经过渲染显示于终端设备的界面。
基于上述介绍,请参阅图10,图10为本申请实施例中外插帧模式下生成目标视频帧的一个示意图,如图所示,假设第一已渲染视频帧为第n帧,第二已渲染视频帧为第n+1帧,经过外插帧处理后生成目标视频帧,其中,目标视频帧为第n+1.5帧。由此可见,假设服务器渲染30帧画面,则实际编码的画面可以为60帧,输出至终端设备的画面也为60帧,因此,对于终端设备而言,解码的画面数量为60帧。外插帧通常不会产生额外的时延,适用于对时延要求较高的业务,例如,实时对战类游戏等。
其次,本申请实施例中,提供了一种基于外插帧获取目标视频序列的方式,利用外插帧得到的目标视频序列通常不会产生额外的时延,因此,更适 用于对时延要求较高,但是对画面质量要求不高的业务。从而在节省服务器侧处理器开销的同时,还可以避免因时延造成画面延迟的现象。
可选地,在上述图4对应的各个实施例的基础上,本申请实施例提供的视频处理方法另一个可选实施例中,服务器对第一已渲染视频帧以及第二已渲染视频帧进行插帧处理,得到目标视频帧,可以包括:
服务器获取第二已渲染视频帧对应的第二帧号;
服务器将第二帧号的下一个相邻帧号确定为第三帧号;
服务器根据第二帧号以及第三帧号,得到目标帧号;
服务器基于目标帧号,通过插帧预测模型获取目标视频帧对应的K个像素信息,其中,K个像素信息用于确定目标视频帧,K为大于1的整数。
本实施例中,介绍了一种基于插帧预测模型进行外插帧的方式,需要说明的是,由于原始视频序列中可能包括大量的已渲染视频帧,为了便于说明,下面将以任意两个相邻的已渲染视频帧为例介绍,在实际应用中,对其他相邻两帧的处理方式类似,此处不做赘述。
具体地,每个已渲染视频帧对应于一个帧号,其中,第一已渲染视频帧的帧号为第一帧号,第二已渲染视频帧的帧号为第二帧号,假设第二帧号为n+1,第三帧号为n+2,则目标帧号的计算方式如下:
U=[n+1+(n+2)]/2;
其中,U表示目标帧号,即为n+1.5。
将目标帧号输入至已训练好的插帧预测模型中,通过插帧预测模型输出K个像素信息,K表示一个视频帧所包括的像素总个数,在得到K个像素对应的像素信息时,即获取到目标视频帧。像素信息可以表示为YUV形式,或者可以表示为RGB形式。
需要说明的是,插帧预测模型的训练方式已在上述实施例中介绍,故此处不做赘述。
再次,本申请实施例中,提供了一种基于插帧预测模型进行外插帧的方式,通过上述方式,利用训练好的插帧预测模型能够预测出目标视频帧中的每个像素信息,再由这些像素信息重构出目标视频帧,从而实现对视频帧外插一帧的过程,由此提升方案的可行性和可操作性。
结合上述介绍,下面将从终端设备的角度,对本申请中视频处理的方法进行介绍,请参阅图11,本申请实施例中视频处理方法的另一个实施例包括:
201、终端设备接收服务器发送的视频编码序列。
本实施例中,由服务器获取连续的P个视频帧,这P个视频帧可构成原始视频序列。其中,视频帧可以指经过渲染后的视频画面,也可以指经过渲染后的游戏画面,还可以指经过渲染的其他类型画面。需要说明的是,本申请涉及的服务器可以是本地服务器(例如,游戏服务器或者视频服务器等),也可以是云服务器(例如云游戏服务器或者云视频服务器),为了便于介绍, 本申请将以服务器为应用于云游戏业务中的云游戏服务器为例进行介绍,然而这不应理解为对本申请的限定。
服务器采用编码算法对原始视频序列进行编码处理,从而生成视频编码序列。编码过程可以运行于服务器的CPU上,也可以运行于GPU上,还可以运行于其他编码硬件上,例如,GPU内的编码芯片或者独立于GPU之外的专用编码芯片。编码算法可采用H264、H265、VP8或VP9等,此处不做限定。
202、终端设备对视频编码序列进行解码处理,得到待渲染的视频序列,其中,待渲染的视频序列包括未经过渲染的X个视频帧,X为大于或等于2的整数。
本实施例中,服务器向终端设备发送视频编码序列,视频编码序列包括至少两个视频帧。于是终端设备可以对视频编码序列进行解码,从而得到待渲染的视频序列,待渲染的视频序列包括未经过渲染的X个视频帧,且待渲染视频序列中包括的视频帧数量与视频编码序列中包括的视频帧数量相同。
203、终端设备根据待渲染的视频序列获取目标视频序列,其中,目标视频序列包括未经过渲染的X个视频帧以及***X个视频帧中的Y个未渲染视频帧,Y为大于或等于1的整数。
本实施例中,终端设备对待渲染的视频序列中的X个视频帧进行插帧处理,可以对相邻两个视频帧进行插帧处理,例如,对1号视频帧和2号视频帧进行插帧处理,再对2号视频帧和3号视频帧进行插帧处理,以此类推。也可以对相关两个视频帧进行插帧处理,例如,对1号视频帧和2号视频帧进行插帧处理,再对3号视频帧和4号视频帧进行插帧处理,以此类推。还可以采用间隔视频帧进行插帧处理,例如,对1号视频帧和2号视频帧进行插帧处理,再对5号视频帧和6号视频帧进行插帧处理。基于待渲染的视频序列生成插帧后得到的Y个视频帧,这X个视频帧和Y个视频帧共同构成目标视频序列。
可以理解的是,***一帧图像所消耗的资源少于渲染一帧图像所消耗的资源。插帧的方式包含但不仅限于帧采样,帧混合,运动补偿以及光流法,此处不做赘述。
204、终端设备对目标视频序列进行渲染处理,得到目标视频。
本实施例中,终端设备对目标视频序列进行渲染处理,即可生成目标视频,并在终端设备的屏幕上展示该目标视频。解码过程可以运行于终端设备的CPU上,也可以运行于GPU上,还可以运行于其他解码硬件上,例如,GPU内的解码芯片或者独立于GPU之外的专用解码芯片。解码得到一个视频帧之后,即可通过终端设备侧的CPU或者GPU读取该视频帧,并进行渲染后显示于界面。
下面将以一个场景为例,假设服务器生成30帧图像,对应的码率为10 兆比特每秒(million bits per second,Mbps),终端设备经过插帧后可实现60帧图像的效果。又假设服务器生成60帧图像,对应的码率为20Mbps,终端设备在不插帧的情况下,也能实现60帧图像的效果。因此,在终端设备插帧的方式能够节省传输带宽。
本申请实施例中,提供了一种视频处理的方法,终端设备先接收服务器发送的视频编码序列,然后对视频编码序列进行解码处理,得到待渲染的视频序列,再根据待渲染的视频序列获取目标视频序列,最后对目标视频序列进行渲染处理,得到目标视频。通过上述方式,服务器只需要对较少的视频帧进行渲染,再传输至终端设备,由终端设备以插帧的方式生成目标视频序列,并渲染出目标视频,因此,整个过程对应服务器而言,能够节省其处理资源以及编码开销,有利于提升服务器的业务处理能力。整个过程对于客户端而言,节省了传输带宽。
可选地,在上述图11对应的各个实施例的基础上,本申请实施例提供的视频处理方法另一个可选实施例中,终端设备根据待渲染的视频序列获取目标视频序列,可以包括:
终端设备从待渲染的视频序列中获取第一未渲染视频帧以及第二未渲染视频帧,其中,第一未渲染视频帧为第二未渲染视频帧相邻的前一帧图像,;
终端设备对第一未渲染视频帧以及第二未渲染视频帧进行插帧处理,得到目标视频帧,其中,目标视频帧为第一未渲染视频帧相邻的后一帧图像,且目标视频帧为第二未渲染视频帧相邻的前一帧图像;
终端设备根据第一未渲染视频帧、目标视频帧以及第二未渲染视频帧,生成目标视频序列中的第一视频子序列,其中,第一视频子序列依次由第一未渲染视频帧、目标视频帧以及第二未渲染视频帧组成。
本实施例中,介绍了一种基于内插帧获取目标视频序列的方式,需要说明的是,由于待渲染的视频序列中可能包括大量的未渲染视频帧,为了便于说明,下面将以任意两个相邻的未渲染视频帧为例介绍,在实际应用中,对其他相邻两帧的处理方式类似,此处不做赘述。
具体地,首先终端设备从待渲染的视频序列中获取相邻的两个视频帧,即第一未渲染视频帧以及第二未渲染视频帧,然后基于这两个未渲染视频帧生成新的一个视频帧,即生成目标视频帧,其中,目标视频帧是位于第一未渲染视频帧和第二未渲染视频帧之间的一帧,相当于额外***了一帧图像。按照第一未渲染视频帧、目标视频帧以及第二未渲染视频帧的顺序,生成目标视频序列中的第一视频子序列,在实际应用中,采用类似地方式生成一系列视频子序列,最终生成目标视频序列。
为了便于理解,请参阅图12,图12为本申请实施例中基于终端设备实现内插帧处理的一个实施例示意图,如图所示,以云游戏场景为例,云游戏服务器捕获经过渲染的一帧游戏画面,在获取到多帧渲染后的游戏画面时, 即得到原始视频序列,再对原始视频序列进行编码,得到视频编码序列。云游戏服务器通过网络将视频编码序列传输至终端设备,于是,终端设备对视频编码序列进行解码处理,得到待渲染的视频序列。于是,终端设备可以根据第n帧游戏画面以及第n+1帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+0.5帧游戏画面。类似地,终端设备继续获取第n+2帧游戏画面,对第n+1帧游戏画面以及第n+2帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+1.5帧游戏画面。以此类推,由第n帧、第n+0.5帧、第n+1帧、第n+1.5帧以及第n+2帧等游戏画面共同构成目标视频序列,最后经过渲染显示于终端设备的界面。
基于上述介绍,请再次参阅图6,如图所示,假设第一未渲染视频帧为第n帧,第二未渲染视频帧为第n+1帧,经过内插帧处理后生成目标视频帧,其中,目标视频帧为第n+0.5帧。由此可见,假设服务器渲染30帧画面,则实际编码的画面可以为30帧,输出至终端设备的画面也为30帧,因此,对于终端设备而言,解码的画面数量为30帧,经过内插帧处理后,可渲染出60帧图像。内插帧的图像效果较好,适用于对时延要求不高,而对画质要求较高的业务,例如,非实时对战类游戏等。
其次,本申请实施例中,提供了一种基于内插帧获取目标视频序列的方式,通过上述方式,利用内插帧得到的目标视频序列具有更好的图像效果,但由于云业务场景会产生一帧时延,因此,更适用于对时延要求不高,但是对画面质量要求较高的业务。从而在节省服务器侧处理器开销的同时,还可以提升画面质量。
可选地,在上述图11对应的各个实施例的基础上,本申请实施例提供的视频处理方法另一个可选实施例中,终端设备对第一未渲染视频帧以及第二未渲染视频帧进行插帧处理,得到目标视频帧,可以包括:
终端设备获取第一未渲染视频帧对应的第一帧号;
终端设备获取第二未渲染视频帧对应的第二帧号;
终端设备根据第一帧号以及第二帧号,得到目标帧号;
终端设备基于目标帧号,通过插帧预测模型获取目标视频帧对应的K个像素信息,其中,K个像素信息用于确定目标视频帧,K为大于1的整数。
本实施例中,介绍了一种基于插帧预测模型进行内插帧的方式,需要说明的是,由于待渲染的视频序列中可能包括大量的未渲染视频帧,为了便于说明,下面将以任意两个相邻的未渲染视频帧为例介绍,在实际应用中,对其他相邻两帧的处理方式类似,此处不做赘述。
具体地,每个未渲染视频帧对应于一个帧号,其中,第一未渲染视频帧的帧号为第一帧号,第二未渲染视频帧的帧号为第二帧号,假设第一帧号为n,第二帧号为n+1,则目标帧号的计算方式如下:
U=[n+(n+1)]/2;
其中,U表示目标帧号,即为n+0.5。
将目标帧号输入至已训练好的插帧预测模型中,通过插帧预测模型输出K个像素信息,K表示一个视频帧所包括的像素总个数,在得到K个像素对应的像素信息时,即获取到目标视频帧。像素信息可以表示为YUV形式,或者可以表示为RGB形式。
再次,本申请实施例中,提供了一种基于插帧预测模型进行内插帧的方式,通过上述方式,利用训练好的插帧预测模型能够预测出目标视频帧中的每个像素信息,再由这些像素信息重构出目标视频帧,从而实现对视频帧内插一帧的过程,由此提升方案的可行性和可操作性。
可选地,在上述图11对应的各个实施例的基础上,本申请实施例提供的视频处理方法另一个可选实施例中,终端设备根据待渲染的视频序列获取目标视频序列,可以包括:
终端设备从待渲染的视频序列中获取第一未渲染视频帧以及第二未渲染视频帧,其中,第一未渲染视频帧为第二未渲染视频帧相邻的前一帧图像;
终端设备对第一未渲染视频帧以及第二未渲染视频帧进行插帧处理,得到目标视频帧,其中,目标视频帧为第二未渲染视频帧相邻的后一帧图像;
终端设备根据第一未渲染视频帧、目标视频帧以及第二未渲染视频帧,生成目标视频序列中的第二视频子序列,其中,第二视频子序列依次由第一未渲染视频帧、第二未渲染视频帧以及目标视频帧组成。
本实施例中,介绍了一种基于外插帧获取目标视频序列的方式,需要说明的是,由于待渲染的视频序列中可能包括大量的未渲染视频帧,为了便于说明,下面将以任意两个相邻的未渲染视频帧为例介绍,在实际应用中,对其他相邻两帧的处理方式类似,此处不做赘述。
具体地,首先终端设备从待渲染的视频序列中获取相邻的两个视频帧,即第一未渲染视频帧以及第二未渲染视频帧,然后基于这两个未渲染视频帧生成新的一个视频帧,即生成目标视频帧,其中,目标视频帧是位于第二未渲染视频帧与下一帧未渲染视频帧之间的一帧,相当于额外***了一帧图像。按照第一未渲染视频帧、第二未渲染视频帧以及目标视频帧的顺序,生成目标视频序列中的第二视频子序列,在实际应用中,采用类似地方式生成一系列视频子序列,最终生成目标视频序列。
为了便于理解,请参阅图13,图13为本申请实施例中基于终端设备实现外插帧处理的一个实施例示意图,如图所示,以云游戏场景为例,云游戏服务器捕获经过渲染的一帧游戏画面,在获取到多帧渲染后的游戏画面时,即得到原始视频序列,再对原始视频序列进行编码,得到视频编码序列。云游戏服务器通过网络将视频编码序列传输至终端设备,于是,终端设备对视频编码序列进行解码处理,得到待渲染的视频序列。于是,终端设备可以根据第n帧游戏画面以及第n+1帧游戏画面进行外插帧处理,得到额外的一帧 游戏画面,即第n+1.5帧游戏画面。类似地,终端设备继续获取第n+2帧游戏画面,对第n+1帧游戏画面以及第n+2帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+2.5帧游戏画面。以此类推,由第n帧、第n+1帧、第n+1.5帧以及第n+2帧等游戏画面共同构成目标视频序列,最后经过渲染显示于终端设备的界面。
基于上述介绍,请再次参阅图10,如图所示,假设第一未渲染视频帧为第n帧,第二未渲染视频帧为第n+1帧,经过外插帧处理后生成目标视频帧,其中,目标视频帧为第n+1.5帧。由此可见,假设服务器渲染30帧画面,则实际编码的画面可以为30帧,输出至终端设备的画面也为30帧,因此,对于终端设备而言,解码的画面数量为30帧,经过外插帧处理后,可渲染出60帧图像。外插帧通常不会产生额外的时延,适用于对时延要求较高的业务,例如,实时对战类游戏等。
其次,本申请实施例中,提供了一种基于外插帧获取目标视频序列的方式,利用外插帧得到的目标视频序列通常不会产生额外的时延,因此,更适用于对时延要求较高,但是对画面质量要求不高的业务。从而在节省服务器侧处理器开销的同时,还可以避免因时延造成画面延迟的现象。
可选地,在上述图11对应的各个实施例的基础上,本申请实施例提供的视频处理方法另一个可选实施例中,终端设备对第一未渲染视频帧以及第二未渲染视频帧进行插帧处理,得到目标视频帧,可以包括:
终端设备获取第二未渲染视频帧对应的第二帧号;
终端设备将第二帧号的下一个相邻帧号确定为第三帧号;
终端设备根据第二帧号以及第三帧号的平均值,得到目标帧号;
终端设备基于目标帧号,通过插帧预测模型获取目标视频帧对应的K个像素信息,其中,K个像素信息用于确定目标视频帧,K为大于1的整数。
本实施例中,介绍了一种基于插帧预测模型进行外插帧的方式,需要说明的是,由于待渲染的视频序列中可能包括大量的未渲染视频帧,为了便于说明,下面将以任意两个相邻的未渲染视频帧为例介绍,在实际应用中,对其他相邻两帧的处理方式类似,此处不做赘述。
具体地,每个未渲染视频帧对应于一个帧号,其中,第一未渲染视频帧的帧号为第一帧号,第二未渲染视频帧的帧号为第二帧号,假设第二帧号为n+1,第三帧号为n+2,则目标帧号的计算方式如下:
U=[n+1+(n+2)]/2;
其中,U表示目标帧号,即为n+1.5。
将目标帧号输入至已训练好的插帧预测模型中,通过插帧预测模型输出K个像素信息,K表示一个视频帧所包括的像素总个数,在得到K个像素对应的像素信息时,即获取到目标视频帧。像素信息可以表示为YUV形式,或者可以表示为RGB形式。
需要说明的是,插帧预测模型的训练方式已在上述实施例中介绍,故此处不做赘述。
再次,本申请实施例中,提供了一种基于插帧预测模型进行外插帧的方式,通过上述方式,利用训练好的插帧预测模型能够预测出目标视频帧中的每个像素信息,再由这些像素信息重构出目标视频帧,从而实现对视频帧外插一帧的过程,由此提升方案的可行性和可操作性。
结合上述介绍,下面将从视频处理***的角度,对本申请中视频处理的方法进行介绍,请参阅图14,本申请实施例中视频处理方法的另一个实施例包括:
301、服务器获取原始视频序列,其中,原始视频序列包括经过渲染后得到的P个视频帧,P为大于或等于2的整数。
本实施例中,由服务器获取连续P帧画面,即获取到P个视频帧,这P个视频帧可构成原始视频序列。其中,视频帧可以指经过渲染后的视频画面,也可以指经过渲染后的游戏画面,还可以指经过渲染的其他类型画面。需要说明的是,本申请涉及的服务器可以是本地服务器(例如,游戏服务器或者视频服务器等),也可以是云服务器(例如云游戏服务器或者云视频服务器),为了便于介绍,本申请将以服务器为应用于云游戏业务中的云游戏服务器为例进行介绍,此外,在实际应用中CPU和图GPU)可以进行渲染任务,这均不应理解为对本申请的限定。
302、服务器根据原始视频序列获取第一目标视频序列,其中,第一目标视频序列包括经过渲染后得到的P个视频帧以及***P个视频帧中的Q个未渲染视频帧,Q为大于或等于1的整数。
本实施例中,服务器对原始视频序列中的P个视频帧进行插帧处理,从而得到第一目标视频序列,***一帧图像所消耗的资源少于渲染一帧图像所消耗的资源。可以理解的是,插帧处理的方式与步骤102类似,故此处不做赘述。
303、服务器对第一目标视频序列进行编码处理,得到视频编码序列。
本实施例中,服务器采用编码算法对第一目标视频序列进行编码处理,从而生成视频编码序列。编码过程可以运行于服务器的CPU上,也可以运行于GPU上,还可以运行于其他编码硬件上,例如,GPU内的编码芯片或者独立于GPU之外的专用编码芯片。编码算法可采用H264、H265、VP8或VP9等,此处不做限定。可以理解的是,编码过程可以参考步骤103所描述的内容,故此处不做赘述。
304、服务器向终端设备发送视频编码序列。
本实施例中,服务器通过网络向终端设备传输视频编码序列,于是终端设备接收该视频编码序列,其中,视频编码序列包括(P+Q)个视频帧。
305、终端设备对视频编码序列进行解码处理,得到待渲染的视频序列, 其中,待渲染的视频序列包括未经过渲染的(P+Q)个视频帧。
本实施例中,终端设备可以对视频编码序列进行解码,从而得到待渲染的视频序列,且待渲染的视频序列包括未经过渲染的(P+Q)个视频帧,即待渲染视频序列中包括的视频帧数量与视频编码序列中包括的视频帧数量相同。
306、终端设备根据待渲染的视频序列获取第二目标视频序列,其中,第二目标视频序列包括未经过渲染的(P+Q)个视频帧以及***(P+Q)个视频帧中的Y个未渲染视频帧,Y为大于或等于1的整数。
本实施例中,终端设备对待渲染的视频序列中的(P+Q)个视频帧进行插帧处理,可以对相邻两个视频帧进行插帧处理,也可以对相关两个视频帧进行插帧处理,还可以采用间隔视频帧进行插帧处理,此处不做限定。基于待渲染的视频序列生成插帧后得到的Y个视频帧,这(P+Q)个视频帧和Y个视频帧共同构成第二目标视频序列。
可以理解的是,插帧的方式包含但不仅限于帧采样,帧混合,运动补偿以及光流法,此处不做赘述。
307、终端设备对第二目标视频序列进行渲染处理,得到目标视频。
本实施例中,终端设备对第二目标视频序列进行渲染处理,即可生成目标视频,并在终端设备的屏幕上展示该目标视频。解码过程可以运行于终端设备的CPU上,也可以运行于GPU上,还可以运行于其他解码硬件上,例如,GPU内的解码芯片或者独立于GPU之外的专用解码芯片。解码得到一个视频帧之后,即可通过终端设备侧的CPU或者GPU读取该视频帧,并进行渲染后显示于界面。
本申请实施例中,提供了一种视频处理的方法,服务器与终端设备两侧均采用插帧的方式生成视频帧,通过上述方式,可以综合服务器与终端设备的性能,对于服务器而言,只需要对部分视频帧进行渲染,再基于渲染后的视频帧进行插帧处理,得到待编码的视频序列,对于终端设备而言,以插帧的方式生成目标视频序列,并渲染出目标视频。而插帧处理所消耗的资源少于渲染所需的资源,因此,不但可以节省服务器侧的处理资源,减少处理器的开销,有利于提升服务器的业务处理能力,而且对终端设备的性能要求有所降低,有效地平衡了服务器和终端设备的性能。整个过程对于客户端而言,节省了传输带宽。
基于图14所介绍视频处理方法,下面将分别介绍四种插帧处理方式。
第一种方式为服务器与终端设备均采用内插帧的处理方式。请参阅图15,图15为本申请实施例中基于视频处理***实现内插帧处理的一个实施例示意图,如图所示,以云游戏场景为例,云游戏服务器捕获经过渲染的第n帧游戏画面以及第n+1帧游戏画面,对第n帧游戏画面以及第n+1帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+0.5帧游戏画面。类似 地,云游戏服务器继续获取第n+2帧游戏画面,对第n+1帧游戏画面以及第n+2帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+1.5帧游戏画面。以此类推,由第n帧、第n+0.5帧、第n+1帧、第n+1.5帧以及第n+2帧等游戏画面共同构成第一目标视频序列,再对第一目标视频序列进行编码,得到视频编码序列。云游戏服务器通过网络将视频编码序列传输至终端设备,于是,终端设备对视频编码序列进行解码处理,得到待渲染视频序列。再根据第n帧游戏画面以及第n+0.5帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+0.25帧游戏画面。类似地,终端设备继续获取第n+1帧游戏画面,对第n+0.5帧游戏画面以及第n+1帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+0.75帧游戏画面。以此类推,由第n帧、第n+0.25帧、第n+0.5帧、第n+0.75帧以及第n+1帧等游戏画面共同构成第二目标视频序列,经过渲染显示于终端设备的界面。
第二种方式为服务器与终端设备均采用外插帧的处理方式。请参阅图16,图16为本申请实施例中基于视频处理***实现外插帧处理的一个实施例示意图,如图所示,以云游戏场景为例,云游戏服务器捕获经过渲染的第n帧游戏画面以及第n+1帧游戏画面,对第n帧游戏画面以及第n+1帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+1.5帧游戏画面。类似地,云游戏服务器继续获取第n+2帧游戏画面,对第n+1帧游戏画面以及第n+2帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+2.5帧游戏画面。以此类推,由第n帧、第n+1帧、第n+1.5帧、第n+2帧以及第n+2.5帧等游戏画面共同构成第一目标视频序列,再对第一目标视频序列进行编码,得到视频编码序列。云游戏服务器通过网络将视频编码序列传输至终端设备,于是,终端设备对视频编码序列进行解码处理,得到待渲染视频序列。再根据第n+1帧游戏画面以及第n+1.5帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+1.75帧游戏画面。类似地,终端设备继续获取第n+2帧游戏画面,对第n+1.5帧游戏画面以及第n+2帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+2.25帧游戏画面。以此类推,由第n帧、第n+1帧、第n+1.5帧、第n+1.75帧、第n+2帧、第n+2.25帧以及第n+2.5帧等游戏画面共同构成第二目标视频序列,经过渲染显示于终端设备的界面。
第三种方式为服务器采用内插帧,而终端设备采用外插帧的处理方式。请参阅图17,图17为本申请实施例中基于视频处理***实现内外插帧处理的一个实施例示意图,如图所示,以云游戏场景为例,云游戏服务器捕获经过渲染的第n帧游戏画面以及第n+1帧游戏画面,对第n帧游戏画面以及第n+1帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+0.5帧游戏画面。类似地,云游戏服务器继续获取第n+2帧游戏画面,对第n+1帧游戏画面以及第n+2帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+1.5帧游戏画面。以此类推,由第n帧、第n+0.5帧、第n+1帧、第 n+1.5帧以及第n+2帧等游戏画面共同构成第一目标视频序列,再对第一目标视频序列进行编码,得到视频编码序列。云游戏服务器通过网络将视频编码序列传输至终端设备,于是,终端设备对视频编码序列进行解码处理,得到待渲染视频序列。再根据第n帧游戏画面以及第n+0.5帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+0.75帧游戏画面。类似地,终端设备继续获取第n+1帧游戏画面,对第n+0.5帧游戏画面以及第n+1帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+1.25帧游戏画面。以此类推,由第n帧、第n+0.5帧、第n+0.75帧、第n+1帧、第1.25帧、第n+1.5帧、第n+1.75帧以及第n+2帧等游戏画面共同构成第二目标视频序列,经过渲染显示于终端设备的界面。
第四种方式为服务器采用外插帧,而终端设备采用内插帧的处理方式。请参阅图18,图18为本申请实施例中基于视频处理***实现内外插帧处理的另一个实施例示意图,如图所示,以云游戏场景为例,云游戏服务器捕获经过渲染的第n帧游戏画面以及第n+1帧游戏画面,对第n帧游戏画面以及第n+1帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+1.5帧游戏画面。类似地,云游戏服务器继续获取第n+2帧游戏画面,对第n+1帧游戏画面以及第n+2帧游戏画面进行外插帧处理,得到额外的一帧游戏画面,即第n+2.5帧游戏画面。以此类推,由第n帧、第n+1帧、第n+1.5帧、第n+2帧以及第n+2.5帧等游戏画面共同构成第一目标视频序列,再对第一目标视频序列进行编码,得到视频编码序列。云游戏服务器通过网络将视频编码序列传输至终端设备,于是,终端设备对视频编码序列进行解码处理,得到待渲染视频序列。再根据第n帧游戏画面以及第n+1帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+0.5帧游戏画面。类似地,终端设备继续获取第n+1.5帧游戏画面,对第n+1帧游戏画面以及第n+1.5帧游戏画面进行内插帧处理,得到额外的一帧游戏画面,即第n+1.25帧游戏画面。以此类推,由第n帧、第n+0.5帧、第n+1帧、第n+1.25帧、第n+1.5帧、第n+1.75帧、第n+2帧、第n+2.25帧以及第n+2.5帧等游戏画面共同构成第二目标视频序列,经过渲染显示于界面。
下面对本申请中的视频处理装置进行详细描述,请参阅图19,图19为本申请实施例中视频处理装置的一个实施例示意图,视频处理装置40包括:
获取模块401,用于获取原始视频序列,其中,原始视频序列包括经过渲染后得到的P个视频帧,P为大于或等于2的整数;
获取模块401,还用于根据原始视频序列获取目标视频序列,其中,目标视频序列包括P个视频帧以及***P个视频帧中的Q个未渲染视频帧,Q为大于或等于1的整数;
编码模块402,用于对目标视频序列进行编码处理,得到视频编码序列;
发送模块403,用于向终端设备发送视频编码序列,以使终端设备对视 频编码序列进行解码处理,得到待渲染的视频序列。
可选地,在上述图19所对应的实施例的基础上,本申请实施例提供的视频处理装置40的另一实施例中,
获取模块401,具体用于从原始视频序列中获取第一已渲染视频帧以及第二已渲染视频帧,其中,第一已渲染视频帧为第二已渲染视频帧相邻的前一帧图像;
对第一已渲染视频帧以及第二已渲染视频帧进行插帧处理,得到目标视频帧,其中,目标视频帧为第一已渲染视频帧相邻的后一帧图像,且目标视频帧为第二已渲染视频帧相邻的前一帧图像,且目标视频帧为未渲染视频帧;
根据第一已渲染视频帧、目标视频帧以及第二已渲染视频帧,生成目标视频序列中的第一视频子序列,其中,第一视频子序列依次由第一已渲染视频帧、目标视频帧以及第二已渲染视频帧组成。
可选地,在上述图19所对应的实施例的基础上,本申请实施例提供的视频处理装置40的另一实施例中,
获取模块401,具体用于获取第一已渲染视频帧对应的第一帧号;
获取第二已渲染视频帧对应的第二帧号;
根据第一帧号以及第二帧号,得到目标帧号;
基于目标帧号,通过插帧预测模型获取目标视频帧对应的K个像素信息,其中,K个像素信息用于确定目标视频帧,K为大于1的整数。
可选地,在上述图19所对应的实施例的基础上,本申请实施例提供的视频处理装置40的另一实施例中,
获取模块401,具体用于从原始视频序列中获取第一已渲染视频帧以及第二已渲染视频帧,其中,第一已渲染视频帧为第二已渲染视频帧相邻的前一帧图像;
对第一已渲染视频帧以及第二已渲染视频帧进行插帧处理,得到目标视频帧,其中,目标视频帧为第二已渲染视频帧相邻的后一帧图像,且目标视频帧为未渲染视频帧;
根据第一已渲染视频帧、目标视频帧以及第二已渲染视频帧,生成目标视频序列中的第二视频子序列,其中,第二视频子序列依次由第一已渲染视频帧、第二已渲染视频帧以及目标视频帧组成。
可选地,在上述图19所对应的实施例的基础上,本申请实施例提供的视频处理装置40的另一实施例中,
获取模块401,具体用于获取第二已渲染视频帧对应的第二帧号;
将第二帧号的下一个相邻帧号确定为第三帧号;
根据第二帧号以及第三帧号,得到目标帧号;
基于目标帧号,通过插帧预测模型获取目标视频帧对应的K个像素信息,其中,K个像素信息用于确定目标视频帧,K为大于1的整数。
下面对本申请中的视频处理装置进行详细描述,请参阅图20,图20为本申请实施例中视频处理装置的一个实施例示意图,视频处理装置50包括:
接收模块501,用于接收服务器发送的视频编码序列;
解码模块502,用于对视频编码序列进行解码处理,得到待渲染的视频序列,其中,待渲染的视频序列包括未经过渲染的X个视频帧,X为大于或等于2的整数;
获取模块503,用于根据待渲染的视频序列获取目标视频序列,其中,目标视频序列包括未经过渲染的X个视频帧以及***所述X个视频帧中的Y个未渲染视频帧,Y为大于或等于1的整数;
渲染模块504,用于对目标视频序列进行渲染处理,得到目标视频。
可选地,在上述图20所对应的实施例的基础上,本申请实施例提供的视频处理装置50的另一实施例中,
获取模块503,具体用于从待渲染的视频序列中获取第一未渲染视频帧以及第二未渲染视频帧,其中,第一未渲染视频帧为第二未渲染视频帧相邻的前一帧图像;
对第一未渲染视频帧以及第二未渲染视频帧进行插帧处理,得到目标视频帧,其中,目标视频帧为第一未渲染视频帧相邻的后一帧图像,且目标视频帧为第二未渲染视频帧相邻的前一帧图像;
根据第一未渲染视频帧、目标视频帧以及第二未渲染视频帧,生成目标视频序列中的第一视频子序列,其中,第一视频子序列依次由第一未渲染视频帧、目标视频帧以及第二未渲染视频帧组成。
可选地,在上述图20所对应的实施例的基础上,本申请实施例提供的视频处理装置50的另一实施例中,
获取模块503,具体用于获取第一未渲染视频帧对应的第一帧号;
获取第二未渲染视频帧对应的第二帧号;
根据第一帧号以及第二帧号,得到目标帧号;
基于目标帧号,通过插帧预测模型获取目标视频帧对应的K个像素信息,其中,K个像素信息用于确定目标视频帧,K为大于1的整数。
可选地,在上述图20所对应的实施例的基础上,本申请实施例提供的视频处理装置50的另一实施例中,
获取模块503,具体用于从待渲染的视频序列中获取第一未渲染视频帧以及第二未渲染视频帧,其中,第一未渲染视频帧为第二未渲染视频帧相邻的前一帧图像;
对第一未渲染视频帧以及第二未渲染视频帧进行插帧处理,得到目标视频帧,其中,目标视频帧为第二未渲染视频帧相邻的后一帧图像;
根据第一未渲染视频帧、目标视频帧以及第二未渲染视频帧,生成目标视频序列中的第二视频子序列,其中,第二视频子序列依次由第一未渲染视 频帧、第二未渲染视频帧以及目标视频帧组成。
可选地,在上述图20所对应的实施例的基础上,本申请实施例提供的视频处理装置50的另一实施例中,
获取模块503,具体用于获取第二未渲染视频帧对应的第二帧号;
将第二帧号的下一个相邻帧号确定为第三帧号;
根据第二帧号以及第三帧号,得到目标帧号;
基于目标帧号,通过插帧预测模型获取目标视频帧对应的K个像素信息,其中,K个像素信息用于确定目标视频帧,K为大于1的整数。
图21是本申请实施例提供的一种服务器结构示意图,该服务器600可因配置或性能不同而产生比较大的差异,可以包括一个或一个以上CPU 622(例如,一个或一个以上处理器)和存储器632,一个或一个以上存储应用程序642或数据644的存储介质630(例如一个或一个以上海量存储设备)。其中,存储器632和存储介质630可以是短暂存储或持久存储。存储在存储介质630的程序可以包括一个或一个以上模块(图示没标出),每个模块可以包括对服务器中的一系列指令操作。更进一步地,中央处理器622可以设置为与存储介质630通信,在服务器600上执行存储介质630中的一系列指令操作。
服务器600还可以包括一个或一个以上电源626,一个或一个以上有线或无线网络接口650,一个或一个以上输入输出接口658,和/或,一个或一个以上操作***641,例如Windows Server
TM,Mac OS X
TM,Unix
TM,Linux
TM,FreeBSD
TM等等。
上述实施例中由服务器所执行的步骤可以基于该图21所示的服务器结构。
本申请实施例还提供了另一种视频处理装置,如图22所示,为了便于说明,仅示出了与本申请实施例相关的部分,具体技术细节未揭示的,请参照本申请实施例方法部分。该终端设备可以为包括手机、平板电脑、个人数字助理(personal digital assistant,PDA)、销售终端设备(point of sales,POS)、车载电脑、电视以及OTT盒子等任意终端设备,以终端设备为手机为例:
图22示出的是与本申请实施例提供的终端设备相关的手机的部分结构的框图。参考图22,手机包括:射频(radio frequency,RF)电路710、存储器720、和处理器780等部件。本领域技术人员可以理解,图22中示出的手机结构并不构成对手机的限定,可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件布置。
下面结合图22对手机的各个构成部件进行具体的介绍:
RF电路710可用于收发信息或通话过程中,信号的接收和发送,特别地,将基站的下行信息接收后,给处理器780处理;另外,将设计上行的数据发送给基站。通常,RF电路710包括但不限于天线、至少一个放大器、收发信机、耦合器、低噪声放大器(low noise amplifier,LNA)、双工器等。此外, RF电路710还可以通过无线通信与网络和其他设备通信。上述无线通信可以使用任一通信标准或协议,包括但不限于全球移动通讯***(global system of mobile communication,GSM)、通用分组无线服务(general packet radio service,GPRS)、码分多址(code division multiple access,CDMA)、宽带码分多址(wideband code division multiple access,WCDMA)、长期演进(long term evolution,LTE)、电子邮件、短消息服务(short messaging service,SMS)等。
存储器720可用于存储软件程序以及模块,处理器780通过运行存储在存储器720的软件程序以及模块,从而执行手机的各种功能应用以及数据处理。存储器720可主要包括存储程序区和存储数据区,其中,存储程序区可存储操作***、至少一个功能所需的应用程序(比如声音播放功能、图像播放功能等)等;存储数据区可存储根据手机的使用所创建的数据(比如音频数据、电话本等)等。此外,存储器720可以包括高速随机存取存储器,还可以包括非易失性存储器,例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。
处理器780是手机的控制中心,利用各种接口和线路连接整个手机的各个部分,通过运行或执行存储在存储器720内的软件程序和/或模块,以及调用存储在存储器720内的数据,执行手机的各种功能和处理数据,从而对手机进行整体监控。可选的,处理器780可包括一个或多个处理单元;可选的,处理器780可集成应用处理器和调制解调处理器,其中,应用处理器主要处理操作***、用户界面和应用程序等,调制解调处理器主要处理无线通信。可以理解的是,上述调制解调处理器也可以不集成到处理器780中。
尽管未示出,手机还可以包括摄像头、蓝牙模块等,在此不再赘述。
上述实施例中由终端设备所执行的步骤可以基于该图22所示的终端设备结构。
本申请提供了一种视频处理***,请参阅图23,图23为本申请实施例中视频处理***的一个结构示意图,如图所示,服务器801获取原始视频序列,其中,原始视频序列包括经过渲染后得到的P个视频帧,P为大于或等于2的整数。服务器801根据原始视频序列获取第一目标视频序列,其中,第一目标视频序列包括经过渲染后得到的P个视频帧以及***P个视频帧中的Q个未渲染视频帧,Q为大于或等于1的整数。服务器801对第一目标视频序列进行编码处理,得到视频编码序列。服务器801向终端设备802发送视频编码序列。终端设备802对视频编码序列进行解码处理,得到待渲染的视频序列,其中,待渲染的视频序列包括未经过渲染的(P+Q)个视频帧。终端设备802根据待渲染的视频序列获取第二目标视频序列,其中,第二目标视频序列包括未经过渲染的(P+Q)个视频帧以及***所述(P+Q)个视频帧中的Y个未渲染视频帧,Y为大于或等于1的整数。终端设备802对第二 目标视频序列进行渲染处理,得到目标视频。
本申请实施例中还提供一种计算机可读存储介质,该计算机可读存储介质中存储有计算机程序,当其在计算机上运行时,使得计算机执行如前述各个实施例描述的方法中视频处理装置所执行的步骤。
本申请实施例中还提供一种包括程序的计算机程序产品,当其在计算机上运行时,使得计算机执行前述各个实施例描述的方法中视频处理装置所执行的步骤。
所属领域的技术人员可以清楚地了解到,为描述的方便和简洁,上述描述的***,装置和单元的具体工作过程,可以参考前述方法实施例中的对应过程,在此不再赘述。
Claims (16)
- 一种视频处理的方法,由服务器执行,包括:获取原始视频序列,所述原始视频序列包括经过渲染后得到的P个视频帧,所述P为大于或等于2的整数;根据所述原始视频序列获取目标视频序列,所述目标视频序列包括所述P个视频帧以及***所述P个视频帧中的Q个未渲染视频帧,所述Q为大于或等于1的整数;对所述目标视频序列进行编码处理,得到视频编码序列;向终端设备发送所述视频编码序列,以使所述终端设备对所述视频编码序列进行解码处理,得到待渲染的视频序列。
- 根据权利要求1所述的方法,所述根据所述原始视频序列获取目标视频序列,包括:从所述原始视频序列中获取第一已渲染视频帧以及第二已渲染视频帧,所述第一已渲染视频帧为所述第二已渲染视频帧相邻的前一帧图像;对所述第一已渲染视频帧以及所述第二已渲染视频帧进行插帧处理,得到目标视频帧,所述目标视频帧为所述第一已渲染视频帧相邻的后一帧图像,所述目标视频帧为所述第二已渲染视频帧相邻的前一帧图像,且所述目标视频帧为未渲染视频帧;根据所述第一已渲染视频帧、所述目标视频帧以及所述第二已渲染视频帧,生成所述目标视频序列中的第一视频子序列,其中,所述第一视频子序列依次由所述第一已渲染视频帧、所述目标视频帧以及所述第二已渲染视频帧组成。
- 根据权利要求2所述的方法,所述对所述第一已渲染视频帧以及所述第二已渲染视频帧进行插帧处理,得到目标视频帧,包括:获取所述第一已渲染视频帧对应的第一帧号;获取所述第二已渲染视频帧对应的第二帧号;根据所述第一帧号以及所述第二帧号,得到目标帧号;基于所述目标帧号,通过插帧预测模型获取所述目标视频帧对应的K个像素信息,其中,所述K个像素信息用于确定所述目标视频帧,所述K为大于1的整数。
- 根据权利要求1所述的方法,所述根据所述原始视频序列获取目标视频序列,包括:从所述原始视频序列中获取第一已渲染视频帧以及第二已渲染视频帧,所述第一已渲染视频帧为所述第二已渲染视频帧相邻的前一帧图像;对所述第一已渲染视频帧以及所述第二已渲染视频帧进行插帧处理,得到目标视频帧,所述目标视频帧为所述第二已渲染视频帧相邻的后一帧图像,且所述目标视频帧为未渲染视频帧;根据所述第一已渲染视频帧、所述目标视频帧以及所述第二已渲染视频帧,生成所述目标视频序列中的第二视频子序列,其中,所述第二视频子序列依次由所述第一已渲染视频帧、所述第二已渲染视频帧以及所述目标视频帧组成。
- 根据权利要求4所述的方法,所述对所述第一已渲染视频帧以及所述第二已渲染视频帧进行插帧处理,得到目标视频帧,包括:获取所述第二已渲染视频帧对应的第二帧号;将所述第二帧号的下一个相邻帧号确定为第三帧号;根据所述第二帧号以及所述第三帧号,得到目标帧号;基于所述目标帧号,通过插帧预测模型获取所述目标视频帧对应的K个像素信息,其中,所述K个像素信息用于确定所述目标视频帧,所述K为大于1的整数。
- 一种视频处理的方法,由终端设备执行,包括:接收服务器发送的视频编码序列;对所述视频编码序列进行解码处理,得到待渲染的视频序列,所述待渲染的视频序列包括未经过渲染的X个视频帧,所述X为大于或等于2的整数;根据所述待渲染的视频序列获取目标视频序列,所述目标视频序列包括所述未经过渲染的X个视频帧以及***所述X个视频帧中的Y个未渲染视频帧,所述Y为大于或等于1的整数;对所述目标视频序列进行渲染处理,得到目标视频。
- 根据权利要求6所述的方法,所述根据所述待渲染的视频序列获取目标视频序列,包括:从所述待渲染的视频序列中获取第一未渲染视频帧以及第二未渲染视频帧,所述第一未渲染视频帧为所述第二未渲染视频帧相邻的前一帧图像;对所述第一未渲染视频帧以及所述第二未渲染视频帧进行插帧处理,得到目标视频帧,所述目标视频帧为所述第一未渲染视频帧相邻的后一帧图像,且所述目标视频帧为所述第二未渲染视频帧相邻的前一帧图像,所述目标视频帧为未渲染视频帧;根据所述第一未渲染视频帧、所述目标视频帧以及所述第二未渲染视频帧,生成所述目标视频序列中的第一视频子序列,其中,所述第一视频子序列依次由所述第一未渲染视频帧、所述目标视频帧以及所述第二未渲染视频帧组成。
- 根据权利要求7所述的方法,所述对所述第一未渲染视频帧以及所述第二未渲染视频帧进行插帧处理,得到目标视频帧,包括:获取所述第一未渲染视频帧对应的第一帧号;获取所述第二未渲染视频帧对应的第二帧号;根据所述第一帧号以及所述第二帧号,得到目标帧号;基于所述目标帧号,通过插帧预测模型获取所述目标视频帧对应的K个像素信息,其中,所述K个像素信息用于确定所述目标视频帧,所述K为大于1的整数。
- 根据权利要求6所述的方法,所述根据所述待渲染的视频序列获取目标视频序列,包括:从所述待渲染的视频序列中获取第一未渲染视频帧以及第二未渲染视频帧,所述第一未渲染视频帧为所述第二未渲染视频帧相邻的前一帧图像;对所述第一未渲染视频帧以及所述第二未渲染视频帧进行插帧处理,得到目标视频帧,所述目标视频帧为所述第二未渲染视频帧相邻的后一帧图像;根据所述第一未渲染视频帧、所述目标视频帧以及所述第二未渲染视频帧,生成所述目标视频序列中的第二视频子序列,其中,所述第二视频子序列依次由所述第一未渲染视频帧、所述第二未渲染视频帧以及所述目标视频帧组成。
- 根据权利要求9所述的方法,所述对所述第一未渲染视频帧以及所述第二未渲染视频帧进行插帧处理,得到目标视频帧,包括:获取所述第二未渲染视频帧对应的第二帧号;将所述第二帧号的下一个相邻帧号确定为第三帧号;根据所述第二帧号以及所述第三帧号,得到目标帧号;基于所述目标帧号,通过插帧预测模型获取所述目标视频帧对应的K个像素信息,其中,所述K个像素信息用于确定所述目标视频帧,所述K为大于1的整数。
- 一种视频处理的方法,包括:服务器获取原始视频序列,所述原始视频序列包括经过渲染后得到的P个视频帧,所述P为大于或等于2的整数;所述服务器根据所述原始视频序列获取第一目标视频序列,所述第一目标视频序列包括所述P个视频帧以及***所述P个视频帧中的Q个未渲染视频帧,所述Q为大于或等于1的整数;所述服务器对所述第一目标视频序列进行编码处理,得到视频编码序列;所述服务器向终端设备发送所述视频编码序列;所述终端设备对所述视频编码序列进行解码处理,得到待渲染的视频序列,所述待渲染的视频序列包括未经过渲染的(P+Q)个视频帧;所述终端设备根据所述待渲染的视频序列获取第二目标视频序列,所述第二目标视频序列包括所述未经过渲染的(P+Q)个视频帧以及***所述(P+Q)个视频帧中的Y个未渲染视频帧,所述Y为大于或等于1的整数;所述终端设备对所述第二目标视频序列进行渲染处理,得到目标视频。
- 一种视频处理装置,包括:获取模块,用于获取原始视频序列,所述原始视频序列包括经过渲染后 得到的P个视频帧,所述P为大于或等于2的整数;所述获取模块,还用于根据所述原始视频序列获取目标视频序列,所述目标视频序列包括所述P个视频帧以及***所述P个视频帧中的Q个未渲染视频帧,所述Q为大于或等于1的整数;编码模块,用于对所述目标视频序列进行编码处理,得到视频编码序列;发送模块,用于向终端设备发送所述视频编码序列,以使所述终端设备对所述视频编码序列进行解码处理,得到待渲染的视频序列。
- 一种视频处理装置,包括:接收模块,用于接收服务器发送的视频编码序列;解码模块,用于对所述视频编码序列进行解码处理,得到待渲染的视频序列,所述待渲染的视频序列包括未经过渲染的X个视频帧,所述X为大于或等于2的整数;获取模块,用于根据所述待渲染的视频序列获取目标视频序列,所述目标视频序列包括所述未经过渲染的X个视频帧以及***所述X个视频帧中的Y个未渲染视频帧,所述Y为大于或等于1的整数;渲染模块,用于对所述目标视频序列进行渲染处理,得到目标视频。
- 一种计算机设备,包括:存储器、收发器、处理器以及总线***;其中,所述存储器用于存储程序;所述处理器用于执行所述存储器中的程序,所述处理器用于根据所述程序代码中的指令执行权利要求1至5中任一项所述的方法,或,执行权利要求6至10中任一项所述的方法;所述总线***用于连接所述存储器以及所述处理器,以使所述存储器以及所述处理器进行通信。
- 一种计算机可读存储介质,包括指令,当其在计算机上运行时,使得计算机执行如权利要求1至5中任一项所述的方法,或,执行如权利要求6至10中任一项所述的方法。
- 一种计算机程序产品,包括指令,当其在计算机上运行时,使得所述计算机执行如上述权利要求1至5或者6至10中任一项所述的方法。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/720,669 US20220239920A1 (en) | 2020-05-08 | 2022-04-14 | Video processing method, related apparatus, storage medium, and program product |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010381049.1 | 2020-05-08 | ||
CN202010381049.1A CN113630621B (zh) | 2020-05-08 | 2020-05-08 | 一种视频处理的方法、相关装置及存储介质 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/720,669 Continuation US20220239920A1 (en) | 2020-05-08 | 2022-04-14 | Video processing method, related apparatus, storage medium, and program product |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021223577A1 true WO2021223577A1 (zh) | 2021-11-11 |
Family
ID=78377142
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/086784 WO2021223577A1 (zh) | 2020-05-08 | 2021-04-13 | 一种视频处理的方法、相关装置、存储介质及程序产品 |
Country Status (3)
Country | Link |
---|---|
US (1) | US20220239920A1 (zh) |
CN (1) | CN113630621B (zh) |
WO (1) | WO2021223577A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116708931A (zh) * | 2022-11-14 | 2023-09-05 | 荣耀终端有限公司 | 图像处理方法及电子设备 |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111310744B (zh) * | 2020-05-11 | 2020-08-11 | 腾讯科技(深圳)有限公司 | 图像识别方法、视频播放方法、相关设备及介质 |
CN118283340A (zh) * | 2022-12-29 | 2024-07-02 | 华为技术有限公司 | 端云协同***、编解码方法及电子设备 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8147339B1 (en) * | 2007-12-15 | 2012-04-03 | Gaikai Inc. | Systems and methods of serving game video |
CN109743626A (zh) * | 2019-01-02 | 2019-05-10 | 京东方科技集团股份有限公司 | 一种图像显示方法、图像处理方法和相关设备 |
CN110138769A (zh) * | 2019-05-09 | 2019-08-16 | 深圳市腾讯网域计算机网络有限公司 | 一种图像传输的方法以及相关装置 |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106686472B (zh) * | 2016-12-29 | 2019-04-26 | 华中科技大学 | 一种基于深度学习的高帧率视频生成方法及*** |
US11284107B2 (en) * | 2017-08-22 | 2022-03-22 | Google Llc | Co-located reference frame interpolation using optical flow estimation |
US10659788B2 (en) * | 2017-11-20 | 2020-05-19 | Google Llc | Block-based optical flow estimation for motion compensated prediction in video coding |
CN107483960B (zh) * | 2017-09-15 | 2020-06-02 | 信阳师范学院 | 一种基于空间预测的运动补偿帧率上转换方法 |
CN108322685B (zh) * | 2018-01-12 | 2020-09-25 | 广州华多网络科技有限公司 | 视频插帧方法、存储介质以及终端 |
CN109803175B (zh) * | 2019-03-12 | 2021-03-26 | 京东方科技集团股份有限公司 | 视频处理方法及装置、设备、存储介质 |
CN110324664B (zh) * | 2019-07-11 | 2021-06-04 | 南开大学 | 一种基于神经网络的视频补帧方法及其模型的训练方法 |
CN110740380A (zh) * | 2019-10-16 | 2020-01-31 | 腾讯科技(深圳)有限公司 | 视频处理方法和装置、存储介质及电子装置 |
-
2020
- 2020-05-08 CN CN202010381049.1A patent/CN113630621B/zh active Active
-
2021
- 2021-04-13 WO PCT/CN2021/086784 patent/WO2021223577A1/zh active Application Filing
-
2022
- 2022-04-14 US US17/720,669 patent/US20220239920A1/en active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8147339B1 (en) * | 2007-12-15 | 2012-04-03 | Gaikai Inc. | Systems and methods of serving game video |
CN109743626A (zh) * | 2019-01-02 | 2019-05-10 | 京东方科技集团股份有限公司 | 一种图像显示方法、图像处理方法和相关设备 |
CN110138769A (zh) * | 2019-05-09 | 2019-08-16 | 深圳市腾讯网域计算机网络有限公司 | 一种图像传输的方法以及相关装置 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116708931A (zh) * | 2022-11-14 | 2023-09-05 | 荣耀终端有限公司 | 图像处理方法及电子设备 |
CN116708931B (zh) * | 2022-11-14 | 2024-03-15 | 荣耀终端有限公司 | 图像处理方法及电子设备 |
Also Published As
Publication number | Publication date |
---|---|
CN113630621A (zh) | 2021-11-09 |
US20220239920A1 (en) | 2022-07-28 |
CN113630621B (zh) | 2022-07-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112104879B (zh) | 一种视频编码方法、装置、电子设备及存储介质 | |
WO2021223577A1 (zh) | 一种视频处理的方法、相关装置、存储介质及程序产品 | |
US20180270496A1 (en) | Composite video streaming using stateless compression | |
JP5307958B1 (ja) | 動画配信サーバ、動画再生装置、制御方法、プログラム、及び記録媒体 | |
US11265599B2 (en) | Re-encoding predicted picture frames in live video stream applications | |
US8238420B1 (en) | Video content transcoding for mobile devices | |
CN112533059B (zh) | 图像渲染方法、装置、电子设备以及存储介质 | |
WO2023142716A1 (zh) | 编码方法、实时通信方法、装置、设备及存储介质 | |
CN113973224B (zh) | 媒体信息的传输方法、计算设备及存储介质 | |
US10841591B2 (en) | Systems and methods for deferred post-processes in video encoding | |
WO2021057686A1 (zh) | 视频解码方法和装置、视频编码方法和装置、存储介质及电子装置 | |
KR20160015134A (ko) | 클라우드 스트리밍 서비스 시스템, 이미지 타입에 따른 클라우드 스트리밍 서비스 방법 및 이를 위한 장치 | |
EP3791595A1 (en) | Client side data stream processing | |
WO2022100645A1 (zh) | 一种数据编码方法、装置以及计算机可读存储介质 | |
CN116980392A (zh) | 媒体流处理方法、装置、计算机设备和存储介质 | |
JP6483850B2 (ja) | データ処理方法および装置 | |
CN110798715A (zh) | 基于图像串的视频播放方法及*** | |
CN113038277B (zh) | 一种视频处理方法和装置 | |
CN111435995B (zh) | 用于生成动态图片的方法、装置和*** | |
KR20160044732A (ko) | 클라우드 스트리밍 서비스 시스템, 스틸 이미지 기반 클라우드 스트리밍 서비스 방법 및 이를 위한 장치 | |
CN117336531A (zh) | 视频画面的渲染方法、装置、终端、存储介质及程序产品 | |
WO2024010588A1 (en) | Cloud-based gaming system for supporting legacy gaming applications with high frame rate streams | |
KR102265419B1 (ko) | 클라우드 스트리밍 서비스 시스템, 선택적 인코딩 처리 유닛을 이용한 클라우드 스트리밍 서비스 방법 및 이를 위한 장치 | |
CN114501027A (zh) | 可软硬件混合视频编码的卡上视频编码加速*** | |
CN115529491A (zh) | 一种音视频解码的方法、音视频解码的装置以及终端设备 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21799944 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 13.04.2023) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21799944 Country of ref document: EP Kind code of ref document: A1 |