CN116777744A - Image superdivision method and system for cooperation of server and client - Google Patents
Image superdivision method and system for cooperation of server and client Download PDFInfo
- Publication number
- CN116777744A CN116777744A CN202310720931.8A CN202310720931A CN116777744A CN 116777744 A CN116777744 A CN 116777744A CN 202310720931 A CN202310720931 A CN 202310720931A CN 116777744 A CN116777744 A CN 116777744A
- Authority
- CN
- China
- Prior art keywords
- image
- coordinates
- pixel matrix
- interpolation
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 82
- 238000009877 rendering Methods 0.000 claims abstract description 38
- 239000011159 matrix material Substances 0.000 claims description 77
- 238000004364 calculation method Methods 0.000 claims description 25
- 238000013507 mapping Methods 0.000 claims description 18
- 230000000694 effects Effects 0.000 abstract description 12
- 238000012545 processing Methods 0.000 description 17
- 230000005540 biological transmission Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000001303 quality assessment method Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000008570 general process Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/50—Controlling the output signals based on the game progress
- A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23412—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
- H04N21/4781—Games
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The image superdivision method and system for the cooperation of the server and the client comprises the following steps: s101: the server side renders the first resolution image to obtain a rendered image; s102: the server zooms the rendered image to obtain a second resolution image; s103: the server encodes the second resolution image to obtain an encoded image, and sends the encoded image to the client; s104: the client receives the coded image and decodes the coded image to obtain a decoded image; s105: the client performs image super-division on the decoded image to obtain a super-division image; the first resolution is greater than the second resolution. An image superdivision system for cooperation of a server and a client, comprising: a server-side assembly comprising: a rendering module; a scaling module; a coding module; a client component, comprising: a decoding module; and an image superdivision module. Better rendering and superdivision effects can be obtained by using the method and the system of the invention.
Description
Technical Field
The invention relates to an image superdivision method for cooperation of a server and a client and an image superdivision system for cooperation of the server and the client.
Background
With the development of display resolution of a user side, a requirement of supporting display of a high-definition picture on the client side is required, and when resolution is poor due to image quality of an original picture subjected to photographing equipment, photographing conditions and network transmission conditions, super-resolution processing is required to be performed on the user side to match the requirement of display of the client side, which is called image super resolution (super resolution). In online games, generally, because the problem of insufficient computing power exists in the super-resolution of the terminal side, the computing power requirement of the super-resolution of the terminal side can be reduced by using smaller original resolution, so that the resolution of a picture is reduced by using a common server side, the picture is transmitted to a client side for rendering, super-resolution and other processing.
Patent document CN113842635a discloses a method and a system for improving the smoothness of a cloud game, specifically: the method comprises the steps of receiving a game picture video stream rendered by a server side by using low resolution; decoding the video stream to obtain a video picture; and upsampling the decoded video picture to generate a game picture for client display. In the related art, a server side adopts a low resolution to render a picture, and upsamples after decoding by a client side, so that there is a problem of poor picture quality.
Disclosure of Invention
The invention aims to solve the problem of poor image quality caused by the fact that a low-resolution picture is transmitted to a client side and then rendered or super processed.
Aiming at the limitations, the invention provides an image superdivision method for cooperation of a server and a client, which comprises the following steps:
s101: the server side renders the first resolution image to obtain a rendered image;
s102: the server zooms the rendered image to obtain a second resolution image;
s103: the server encodes the second resolution image to obtain an encoded image, and sends the encoded image to a client;
s104: the client receives the coded image and decodes the coded image to obtain a decoded image;
s105: the client performs image super-division on the decoded image to obtain a super-division image;
the first resolution is greater than the second resolution.
Further: and rendering by using a Unreal and Unity rendering engine.
Further: in step S103, the encoding is performed using the h.264 or HEVC standard;
in step S104, the decoding uses the h.264 or HEVC standard for decoding.
Further:
in step S102, the scaling method maps the coordinates i, j of the ith row and jth column in the target channel pixel matrix T into the original channel pixel matrix S to obtain mapped coordinates i ', j' in the original channel pixel matrix S, and obtains the element value T corresponding to the coordinates i, j in the target channel pixel matrix T by interpolation according to the mapped coordinates i ', j' and the element value in the original channel pixel matrix S i,j The method for determining the mapping coordinates i ', j' comprises the following steps:
w 1 、h 1 the columns and the rows of the original channel pixel matrix S; w (w) 2 、h 2 The number of columns and the number of rows of the target channel pixel matrix T;
the interpolation method is any one of nearest neighbor interpolation, bilinear interpolation, bicubic interpolation and antialiasing bicubic interpolation. Further: the interpolation method is nearest neighbor interpolation, and the calculation formula is as follows:
wherein the method comprises the steps ofRepresenting the largest integer not greater than x.
Further: the interpolation method is bilinear interpolation, and the calculation formula is as follows:
c w ＝[i 1 -i′,i′-i 0 ]
c h ＝[j 1 -j′,j′-j 0 ]
c w 、c h the superscript T represents the transpose for the vector;representing coordinate i 0 ,j 0 Values in the original channel pixel matrix S;representing coordinate i 1 ,j 0 Values in the original channel pixel matrix S; />Representing coordinate i 0 ,j 1 Values in the original channel pixel matrix S; />Representing coordinate i 1 ,j 1 The value in the original channel pixel matrix S, coordinate i 0 ,j 0 Coordinates i 1 ,j 0 Coordinates i 0 ,j 1 Coordinates i 1 ,j 1 The four coordinates closest to the coordinates i ', j' in the original channel pixel matrix S.
Further: the interpolation method is bicubic interpolation, and the calculation formula is as follows: according to the following:
g i (k)＝f(i′-i k )
g j (k)＝f(j′-j k )
the method comprises the following steps:
w 1 、h 1 the columns and the rows are the columns and the rows of the original channel pixel matrix; w (w) 2 、h 2 The number of columns and the number of rows of the pixel matrix of the target channel; coordinates i 0 ,j 0 Coordinates i 1 ,j 0 Coordinates i 0 ,j 1 Coordinates i 1 ,j 1 Is the four coordinates closest to the coordinates i ', j' in the original channel pixel matrix S, and meets the requirement of i 1 ＝i 0 +1,j 1 ＝j 0 +1；i -1 ＝i 0 -1,i 2 ＝i 1 +1，j -1 ＝j 0 -1，j 2 ＝j 1 +1。
An image superdivision system for cooperation of a server and a client, comprising:
a server-side assembly comprising:
the rendering module is used for rendering the first resolution image to obtain a rendered image;
the scaling module scales the rendered image to obtain a second resolution image;
the encoding module is used for encoding the second resolution image to obtain an encoded image, and transmitting the encoded image to a client;
a client component, comprising:
and a decoding module: receiving the coded image and decoding to obtain a decoded image;
and an image super-division module: and performing image super-division on the decoded image to obtain a super-division image.
Further: the scaling module uses a scaling method that is one or more of the following:
nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, antialiasing bicubic interpolation.
A computer readable storage medium storing computer executable program code which when run performs the method.
Compared with the related art, the invention has the following advantages:
the invention discloses an image superdivision method for the cooperation of a server and a client, which is characterized in that the server and the client perform cooperative division on rendering and superdivision, the server performs rendering, performs scaling and encoding processing, transmits the processing to the client, and performs decoding and superdivision processing after decoding by the client, so that a high-quality superdivision image can be obtained. Because the service end has stronger calculation power, the calculation power pressure of the client end for rendering can be saved by performing rendering processing on the service end, so that the hardware requirement on the client end can be reduced. Rendering and scaling are carried out on the high-resolution image and then super-division is carried out, so that the effect superior to that of directly rendering or super-division by using the low-resolution image through scaling is obtained.
The image superdivision system with the cooperation of the server and the client in the other invention can realize the method, so the invention also has the advantages.
The computer readable storage medium according to another aspect of the present invention can perform the above method, and has the above advantages.
Drawings
FIG. 1 is a flowchart of an image superdivision method for cooperation of a server and a client according to an embodiment of the present invention;
FIG. 2 is a block diagram illustrating a server-client collaborative image superdivision system according to an embodiment of the present invention;
FIG. 3 is a graph showing the comparison of the effects of the super-resolution picture (a) obtained by the method of the present invention and the super-resolution picture (b) obtained by the related art;
FIG. 4 is a graph showing the comparison of the effects of the super-resolution picture (a) obtained by the method of the present invention and the super-resolution picture (b) obtained by the related art;
FIG. 5 shows the bi-cubic interpolation calculated as i in one embodiment of the present invention 0 、j 0 A method of selecting 16 coordinates nearest to the center.
Detailed Description
The present invention will be described in further detail below in order to make the objects, technical solutions and advantages of the present invention more apparent. It is to be understood that the description is only intended to illustrate the invention and is not intended to limit the scope of the invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, and the terms used herein in this description of the invention are for the purpose of describing particular embodiments only and are not intended to be limiting of the invention. The characterization means referred to herein are all referred to in the related description of the prior art, and are not described herein in detail.
For a further understanding of the present invention, the present invention will be described in further detail with reference to the following preferred embodiments.
Example 1
A collaborative image superdivision method for a server and a client comprises the following steps:
s101: the server side renders the first resolution image to obtain a rendered image;
s102: the server zooms the rendered image to obtain a second resolution image;
s103: the server encodes the second resolution image to obtain an encoded image, and sends the encoded image to a client;
s104: the client receives the coded image and decodes the coded image to obtain a decoded image;
s105: the client performs image super-division on the decoded image to obtain a super-division image;
s106: the client displays the super-resolution image;
the first resolution is greater than the second resolution.
The invention discloses an image superdivision method for the cooperation of a server and a client, which is characterized in that the server and the client perform cooperative division on rendering and superdivision, the server performs rendering, performs scaling and encoding processing, transmits the processing to the client, and performs decoding and superdivision processing after decoding by the client, so that a high-quality superdivision image can be obtained. Because the service end has stronger calculation power, the calculation power pressure of the client end for rendering can be saved by performing rendering processing on the service end, so that the hardware requirement on the client end can be reduced. Rendering and scaling are carried out on the high-resolution image and then super-division is carried out, so that the effect superior to that of directly rendering or super-division by using the low-resolution image through scaling is obtained.
The server side is a server side.
The computer converts the content to be displayed into a corresponding process that is actually drawn on the screen is called rendering. The general process of image rendering is that a CPU operates and changes an image, generated primitive information is delivered to a GPU, and the GPU performs geometric processing, rasterization and pixelation processing.
Image scaling (Image scale): the method is a process of mapping pixel points in an original image to pixel points of a target image through an algorithm, and specifically comprises the following steps: and finding out the pixel points of the original image corresponding to the pixel points in the target image, filling the pixel points of the original image into the pixel points of the corresponding target image in a specific mode, and finally forming the target image. Image scaling generally uses an interpolation method, which is to assign values to pixels that cannot be directly obtained by mapping when filling pixels into a target image. For example, when the original image is enlarged, the pixel value of the redundant pixel points which cannot be directly mapped is determined by interpolation. When the original image is reduced, reverse mapping is needed, when the pixel point value in the target image is mapped into the original image, the mapping is a non-integer position, but the non-integer position does not exist in the original image actually, and at the moment, interpolation processing is also needed to be carried out on the pixel point so as to finish the mapping. The first resolution is greater than the second resolution, i.e., the image size is reduced. The mapping is typically performed in a pixel matrix of the image, and when the image has a plurality of channels, an image scaling operation with the same scaling rate is required for the plurality of channels, respectively.
The image coding, the image after the server side rendering is subjected to the image coding, the size of the image transmitted by the network can be compressed, the transmission speed can be seriously influenced when the uncoded image is directly transmitted, and the problems of serious blocking, display delay and the like occur. Standard and technology of h.264, h.265, VP8, VP9, HEVC coding, etc. can be used at present.
After receiving the image, the client needs to decode the image, and the standard used for decoding needs to be decoded according to the standard used for encoding.
The client performs the image superdivision model, and generally uses a model with smaller model structure and smaller calculation amount and simultaneously ensures the required superdivision quality, including but not limited to an ECBSR model. When training the model, training data is consistent with the reasoning input, so that the images processed by the steps S101-S104 are used for training to optimize the corresponding superscore result.
Example 2
Based on example 1, further: and rendering by using a Unreal and Unity rendering engine. The Unity, unreal engine is an Integrated Development Environment (IDE) for creating an interactable multimedia experience, in particular a video game.
Further: in step S103, the encoding is performed using the h.264 or HEVC standard;
in step S104, the decoding uses the h.264 or HEVC standard for decoding.
The h.264 standard is characterized by high compression and high quality and supports streaming media transmission of multiple networks, and three frames are defined in h.264: the fully encoded I frame, the I frame information before reference, only includes the P frame encoded by the difference part information, and the B frame encoded by the frames before and after reference.
The coding module of the HEVC coding core comprises intra/inter prediction, transformation and quantization, entropy coding, in-loop filtering and the like, and three modes of intra coding, low-delay coding and random access coding are respectively set according to the characteristics of the GOP group of the video frame.
Further:
in step S102, the scaling method maps the coordinates i, j of the ith row and jth column in the target channel pixel matrix T into the original channel pixel matrix S to obtain mapped coordinates i ', j' in the original channel pixel matrix S, and obtains the element value T corresponding to the coordinates i, j in the target channel pixel matrix T by interpolation according to the mapped coordinates i ', j' and the element value in the original channel pixel matrix S i,j The method for determining the mapping coordinates i ', j' comprises the following steps:
(equation I)
w 1 、h 1 The columns and the rows of the original channel pixel matrix S; w (w) 2 、h 2 The number of columns and the number of rows of the target channel pixel matrix T; i, j are non-negative integers.
In actual operation, i and j are also taken from the subscript of the pixel matrix, i and j are non-negative integers, and the pixel matrix of each channel can be regarded as a matrix of w columns and h rows, wherein the subscript of the j-th element of the i-th row is i and j; in the scaling operation, the pixel coordinates are represented by using the pixel matrix index i, j for performing correlation calculation, and the method of representing may include direct representation or representation by using values obtained through mapping, transformation, and the like.
According to the mapping coordinates i ', j' and the element values in the original channel pixel matrix S, obtaining the element values T corresponding to the coordinates i, j in the target channel pixel matrix T by using an interpolation method i,j When the root isAccording to different interpolation methods, different element values are used, the element value of which the mapping coordinate is closest to the nearest coordinate is used in the nearest neighbor algorithm, and the element value of which the mapping coordinate is closest to 4 coordinates is used in bilinear interpolation.
The interpolation method is any one of nearest neighbor interpolation, bilinear interpolation, bicubic interpolation and antialiasing bicubic interpolation. Further: the interpolation method is nearest neighbor interpolation, and the calculation formula is as follows:
wherein the method comprises the steps ofRepresenting the largest integer not greater than x.
Further: the interpolation method is bilinear interpolation, and the calculation formula is as follows:
c w ＝[i 1 -i′,i′-i 0 ]
c h ＝[j 1 -j′,j′-j 0 ]
T i,j the element value of the ith row and the jth column of the target channel pixel matrix T; i ', j' are the mapped coordinates of coordinates i, j in the target channel pixel matrix T in the original channel pixel matrix S. c w 、c h The superscript T represents the transpose for the vector;representing coordinate i 0 ,j 0 Values in the original channel pixel matrix S; />Representing coordinate i 1 ,j 0 Values in the original channel pixel matrix S; />Representing coordinate i 0 ,j 1 Values in the original channel pixel matrix S; />Representing coordinate i 1 ,j 1 Values in the original channel pixel matrix S. w (w) 1 、h 1 Is the width and the height of the original picture, w 2 、h 2 Is the width and height of the target picture. I.e. w 1 、h 1 The columns and the rows are the columns and the rows of the original channel pixel matrix; w (w) 2 、h 2 Is the number of columns and rows of the target channel pixel matrix. Coordinates i 0 ,j 0 Coordinates i 1 ,j 0 Coordinates i 0 ,j 1 Coordinates i 1 ,j 1 The four coordinates closest to the coordinates i ', j' in the original channel pixel matrix S.
Further: the interpolation method is bicubic interpolation, and the calculation formula is as follows:
and (3) making:
(equation II)
g i (k)＝f(i′-i k )
g j (k)＝f(j′-j k )
The method comprises the following steps:
w 1 、h 1 is the width and the height of the original picture, w 2 、h 2 Is the width and height of the target picture. I.e. w 1 、h 1 The columns and the rows are the columns and the rows of the original channel pixel matrix; w (w) 2 、h 2 Is the number of columns and rows of the target channel pixel matrix. Coordinates i 0 ,j 0 Coordinates i 1 ,j 0 Coordinates i 0 ,j 1 Coordinates i 1 ,j 1 Is the four coordinates closest to the coordinates i ', j' in the original channel pixel matrix S, and meets the requirement of i 1 ＝i 0 +1,j 1 ＝j 0 +1. In addition, i -1 ＝i 0 -1,i 2 ＝i 1 +1，j -1 ＝j 0 -1，j 2 ＝j 1 +1。
Above-mentionedIn g i (-1) in the case of an example of this,
and then carrying out calculation by using a formula II.Other calculation methods are similar and will not be described in detail. Wherein the coordinates i 0 ，j 0 In order to calculate the coordinate i, j according to the formula I, the coordinate value obtained by projecting the coordinate i, j into the matrix S does not exceed the point of projection and the Euclidean distance of the coordinate is nearest.
In the above calculation, the coordinate i is used 0 ，j 0 Calculation of the pixel value T from values of 16 coordinates in the vicinity i,j . The 16 coordinates are selected as shown in fig. 5.
Further: the interpolation method is anti-sawing bicubic interpolation, and the formula is as follows:
T i,j the element value of the ith row and the jth column of the target channel pixel matrix T; i ', j' are the mapped coordinates of coordinates i, j in the target channel pixel matrix T in the original channel pixel matrix S. w (w) 1 、h 1 Is the width and the height of the original picture, w 2 、h 2 Is the width and height of the target picture. I.e. w 1 、h 1 The columns and the rows are the columns and the rows of the original channel pixel matrix; w (w) 2 、h 2 Is the number of columns and rows of the target channel pixel matrix.
The principle of the image scaling and interpolation method will be described below.
Image scaling (resolution) is the calculation of the pixel value of the target image by a certain rule from the pixel value of the original image. Two key issues with image scaling are which pixel values in the original image should be selected and how to determine the weights of the selected pixel values.
If the picture is w wide and h high, the picture information may look at one or more w x h matrices, each matrix being a channel. The different channels may be color information, brightness information, transparency information, etc.
Will be a w 1 ×h 1 Scaling of a picture to w 2 ×h 2 The value of each element in the target image information matrix is required. For a multi-channel image, when the image is scaled, the same image scaling operation is performed for each channel.
For one channel of the original picture, scaling is carried out to obtain the pixel value of the image information matrix of the target channel, which is hereinafter referred to as the original channel pixel matrix and the target channel pixel matrix. When the element value of the ith row and the jth column of the target channel pixel matrix is obtained, the coordinates (i, j) in the target channel pixel matrix T are subjected to space transformation and mapped into the original channel pixel matrix S, so that the coordinates (i ', j') are obtained.
First, the coordinate locations are mapped into the original channel pixel matrix:
let the original channel pixel matrix be S and the target channel pixel matrix be T.
Wherein …, i -3 ,i -2 ,i -1 ,i 0 ,i 1 ,i 2 ,i 3 …, is a continuous sequence of integers and satisfies i 0 ≤i′<i 1 ；
…,j -3 ,j -2 ,j -1 ,j 0 ,j 1 ,j 2 ,j 3 …, is a continuous sequence of integers and satisfies j 0 ≤j′<j 1 . The negative subscript indicates that its position is i 0 ,j 0 Before. i.e 1 ＝i 0 +1,j 1 ＝j 0 +1. In addition, i -1 ＝i 0 -1,i 2 ＝i 1 +1，j -1 ＝j 0 -1，j 2 ＝j 1 +1。
In actual image rendering, since each pixel is not a dot but a rectangle (square), the length and width of the rectangle is generally 1, and the color value at the position of the exact center of the rectangle, that is, the actual coordinates (i+0.5, j+0.5), is the color value of the pixel dot. Therefore, the actual coordinates of the color point at the time of calculation are shifted by (0.5 ) from the position corresponding to the subscript thereof in the matrix. Therefore, the actual coordinates are as follows according to the coordinate mapping rule of the interpolation method:
and obtaining a formula of mapping the coordinate position into the original channel pixel matrix through the above formula and the term shifting combination.
In actual display, the pixel shape of the rectangle may not be square, i.e. its actual physical width and height may not be equal, and the calculation of the above formula is not affected, but 1 in width and 1 in height may correspond to different lengths on a physical device (such as a display).
The pixel value T of the target channel pixel matrix i,j The method can be obtained by the following steps:
the nearest neighbor algorithm is to find the nearest neighbor of the mapping coordinate (i ', j'), calculate the pixel value of the point (i, j) to be interpolated by using the pixel value, and the pixel value calculation formula of the point (i, j) to be interpolated is as follows:
wherein the method comprises the steps ofRepresents a maximum integer not greater than x;
the nearest neighbor algorithm has small calculated amount and high speed, but can cause a blocky effect and has the problems of image blurring, mosaic feeling and the like.
Bilinear interpolation:
bilinear interpolation is a linear interpolation performed in the horizontal and vertical directions, respectively, due to i 0 ≤i′<i 1 ，j 0 ≤j′<j 1 . Therefore, the pixel value T of the point coordinate (i, j) to be interpolated i,j Four adjacent pixel coordinates (i) of the mapping coordinates (i ', j') are taken 0 ，j 0 )、(i 0 ，j 1 )、(i 1 ，j 0 )、(i 1 ，j 1 ) Is calculated by the following method:
c w ＝[i 1 -i′,i′-i 0 ]
c h ＝[j 1 -j′,j′-j 0 ]
bilinear interpolation can damage high frequency components, resulting in fine structural changes in the edges of the picture.
Bicubic interpolation (Bicubic interpolation):
the bicubic interpolation is also called cubic convolution interpolation, and is performed by performing cubic linear interpolation according to the pixel values of 16 pixels around the target pixel, so that the influence of the values of adjacent 4 pixels is considered, the influence of the change rate of the values of each adjacent point is considered, the image interpolation precision is higher, the effect is good, and the calculated amount is larger.
According to the following:
g i (k)＝f(i′-i k )
g j (k)＝f(j′-j k )
the method comprises the following steps:
the method comprises the following steps:
bicubic interpolation with antialiasing (antialiasing Bicub ic interpolation):
and (3) making:
the image quality scaled by the interpolation method described above may be subjected to an image quality assessment (IQA, image Quality Assessment) to assess the quality loss due to scaling. Existing methods such as blind/no reference image space quality estimators, etc. may be used or the estimation method may be self-directed.
In the practical application of the client, the occupation and delay of the GPU are actually measured through simulating the online environment, and one or more combinations of the interpolation methods with optimal effects under the hardware condition are selected in an allowable range. And the comprehensive evaluation is generally carried out according to the effect promotion and GPU occupation promotion of the superdivision result. The practical test results show that the interpolation method is used for occupying no more than 1% of the GPU.
The method occupies resources, and the nearest neighbor interpolation method and the bilinear interpolation method are realized by GPU hardware, so that the resource consumption is low. Bicubic interpolation and antialiasing bicubic interpolation have no specific hardware support and are relatively computationally complex and therefore relatively high in resource consumption.
As shown in fig. 3 and fig. 4, the effect comparison graph of the superminute picture obtained by the method of the present invention and the superminute picture obtained by the related technology is shown; (a) In the related technology, a low-resolution picture is directly rendered and super-divided on the client side, so that the detail definition of the face of the person or the definition of the text can be seen, and the method and the device have higher promotion than the related technology.
Example 3
An image superdivision system for cooperation of a server and a client, comprising:
a server-side assembly comprising:
the rendering module is used for rendering the first resolution image to obtain a rendered image;
the scaling module scales the rendered image to obtain a second resolution image;
the encoding module is used for encoding the second resolution image to obtain an encoded image, and transmitting the encoded image to a client;
a client component, comprising:
and a decoding module: receiving the coded image and decoding to obtain a decoded image;
and an image super-division module: and performing image super-division on the decoded image to obtain a super-division image.
Further: further comprises: and the on-screen module is used for displaying the superdivision image on the display equipment of the client.
The system comprises the components and the modules, wherein the rendering and superdivision are performed by the cooperative division between the server and the client, the rendering is performed on the server, the scaling and encoding processing is performed on the rendering and the encoding processing and transmitted to the client, and the decoding and superdivision processing is performed after the decoding by the client, so that a high-quality superdivision image can be obtained. Because the service end has stronger calculation power, the calculation power pressure of the client end for rendering can be saved by performing rendering processing on the service end, so that the hardware requirement on the client end can be reduced. Rendering and scaling are carried out on the high-resolution image and then super-division is carried out, so that the effect superior to that of directly rendering or super-division by using the low-resolution image through scaling is obtained.
Example 4
Based on example 2 or 3, further: the scaling module uses a scaling method that is one or more of the following:
nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, antialiasing bicubic interpolation.
Example 5
A computer readable storage medium storing computer executable program code which when run performs the method.
In the embodiment of the invention, the superdivision method and system for the cooperation of the server and the client can be used for cloud games, and it can be understood that the method and the system are not limited to the application, and can be used in scenes such as online video, online live broadcast, meta-space activities and the like.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, or alternatives falling within the spirit and principles of the invention.
Claims (10)
1. A collaborative image superdivision method for a server and a client is characterized in that: comprising the following steps:
s101: the server side renders the first resolution image to obtain a rendered image;
s102: the server zooms the rendered image to obtain a second resolution image;
s103: the server encodes the second resolution image to obtain an encoded image, and sends the encoded image to a client;
s104: the client receives the coded image and decodes the coded image to obtain a decoded image;
s105: the client performs image super-division on the decoded image to obtain a super-division image;
the first resolution is greater than the second resolution.
2. The method of claim 1, wherein:
the rendering uses a Unreal, unity rendering engine to render.
3. The method of claim 1, wherein:
in step S103, the encoding is performed using the h.264 or HEVC standard;
in step S104, the decoding uses the h.264 or HEVC standard for decoding.
4. The method of claim 1, wherein:
in step S102, the scaling method maps the coordinates i, j of the ith row and jth column in the target channel pixel matrix T into the original channel pixel matrix S to obtain the mapped coordinates i in the original channel pixel matrix S ′ ，j ′ According to the mapping coordinate i ′ ，j ′ The element values in the original channel pixel matrix S are used for obtaining element values T corresponding to coordinates i and j in the target channel pixel matrix T by interpolation i,j The mapping coordinates i ′ ，j ′ The determining method of (1) comprises the following steps:
w 1 、h 1 the columns and the rows of the original channel pixel matrix S; w (w) 2 、h 2 The number of columns and the number of rows of the target channel pixel matrix T; the interpolation method is any one of nearest neighbor interpolation, bilinear interpolation, bicubic interpolation and antialiasing bicubic interpolation.
5. The method of claim 4, wherein:
the interpolation method is nearest neighbor interpolation, and the calculation formula is as follows:
wherein the method comprises the steps ofRepresenting the largest integer not greater than x.
6. The method of claim 4, wherein:
the interpolation method is bilinear interpolation, and the calculation formula is as follows:
c w ＝[i 1 -i′，i′-i 0 ]
c h ＝[j 1 -j′，j′-j 0 ]
c w 、c h the superscript T represents the transpose for the vector;representing coordinate i 0 ，j 0 Values in the original channel pixel matrix S; />Representing coordinate i 1 ，j 0 Values in the original channel pixel matrix S; />Representing coordinate i 0 ，j 1 Values in the original channel pixel matrix S; />Representing coordinate i 1 ，j 1 The value in the original channel pixel matrix S, coordinate i 0 ，j 0 Coordinates i 1 ，j 0 Coordinates i 0 ，j 1 Coordinates i 1 ，j 1 The four coordinates closest to the coordinates i ', j' in the original channel pixel matrix S.
7. The method of claim 4, wherein:
the interpolation method is bicubic interpolation, and the calculation formula is as follows: according to the following:
g i (k)＝f(i′-i k )
g j (k)＝f(j′-j k )
the method comprises the following steps:
w 1 、h 1 the columns and the rows are the columns and the rows of the original channel pixel matrix; w (w) 2 、h 2 The number of columns and the number of rows of the pixel matrix of the target channel; coordinates i 0 ，j 0 Coordinates i 1 ，j 0 Coordinates i 0 ，j 1 Coordinates i 1 ，j 1 Is the four coordinates closest to the coordinates i ', j' in the original channel pixel matrix S, and meets the requirement of i 1 ＝i 0 +1，j 1 ＝j 0 +1；i- 1 ＝i 0 -1，i 2 ＝i 1 +1，j- 1 ＝j 0 -1，j 2 ＝j 1 +1。
8. An image superdivision system for cooperation of a server and a client, wherein the method for implementing any one of claims 1-7 comprises:
a server-side assembly comprising:
the rendering module is used for rendering the first resolution image to obtain a rendered image;
the scaling module scales the rendered image to obtain a second resolution image;
the encoding module is used for encoding the second resolution image to obtain an encoded image, and transmitting the encoded image to a client;
a client component, comprising:
and a decoding module: receiving the coded image and decoding to obtain a decoded image;
and an image super-division module: and performing image super-division on the decoded image to obtain a super-division image.
9. The system as recited in claim 8, wherein:
the scaling module uses a scaling method that is one or more of the following:
nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, antialiasing bicubic interpolation.
10. A computer-readable storage medium, characterized by: computer executable program code is stored, which code when run performs the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310720931.8A CN116777744A (en) | 2023-06-16 | 2023-06-16 | Image superdivision method and system for cooperation of server and client |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310720931.8A CN116777744A (en) | 2023-06-16 | 2023-06-16 | Image superdivision method and system for cooperation of server and client |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116777744A true CN116777744A (en) | 2023-09-19 |
Family
ID=87987325
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310720931.8A Pending CN116777744A (en) | 2023-06-16 | 2023-06-16 | Image superdivision method and system for cooperation of server and client |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116777744A (en) |
-
2023
- 2023-06-16 CN CN202310720931.8A patent/CN116777744A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106960416B (en) | A kind of video satellite that content complexity is adaptive compression image super-resolution method | |
| Fenney | Texture compression using low-frequency signal modulation | |
| US11172220B2 (en) | Video encoding method, and storage medium thereof | |
| JP5722761B2 (en) | Video compression apparatus, image processing apparatus, video compression method, image processing method, and data structure of video compression file | |
| JP4455364B2 (en) | Resolution conversion method and apparatus | |
| CN108921910B (en) | JPEG coding compressed image restoration method based on scalable convolutional neural network | |
| US20020191699A1 (en) | Detection system and method for enhancing digital video | |
| US20050094899A1 (en) | Adaptive image upscaling method and apparatus | |
| US20120257679A1 (en) | System and method for encoding and decoding video data | |
| JP2019075783A (en) | Method and encoder for encoding video stream in video coding format supporting auxiliary frames | |
| CN111510739B (en) | Video transmission method and device | |
| US11330254B2 (en) | Video encoding method and apparatus, video decoding method and apparatus, computer device, and storage medium | |
| JP5158096B2 (en) | Encoding data generation apparatus, encoding data generation method, decoding apparatus, and decoding method | |
| US11494870B2 (en) | Method and apparatus for reducing artifacts in projection-based frame | |
| US20190272616A1 (en) | Method for processing projection-based frame that includes projection faces packed in cube-based projection layout with padding | |
| US8514932B2 (en) | Content adaptive and art directable scalable video coding | |
| JP2010098352A (en) | Image information encoder | |
| JP2001136527A (en) | Device and method for resolution conversion of orthogonal image | |
| CN115311145B (en) | Image processing method and device, electronic equipment and storage medium | |
| KR100754735B1 (en) | Method of an effective image expansion using edge signal component and the apparatus therefor | |
| CN116777744A (en) | Image superdivision method and system for cooperation of server and client | |
| CN109413445B (en) | Video transmission method and device | |
| CN113344786B (en) | Video transcoding method, device, medium and equipment based on geometric generation model | |
| TW440804B (en) | Method and apparatus for resizing image information | |
| CN106664387B9 (en) | Computer device and method for post-processing video image frame and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |