WO2019056898A1 - 一种编码、解码方法及装置 - Google Patents
一种编码、解码方法及装置 Download PDFInfo
- Publication number
- WO2019056898A1 WO2019056898A1 PCT/CN2018/101040 CN2018101040W WO2019056898A1 WO 2019056898 A1 WO2019056898 A1 WO 2019056898A1 CN 2018101040 W CN2018101040 W CN 2018101040W WO 2019056898 A1 WO2019056898 A1 WO 2019056898A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- resolution image
- image
- full
- encoder
- residual
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/117—Filters, e.g. for pre-processing or post-processing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/146—Data rate or code amount at the encoder output
- H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
- H04N19/86—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving reduction of coding artifacts, e.g. of blockiness
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
- H04N19/91—Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
Definitions
- the present application relates to image processing technologies, and in particular, to a coding and decoding method and apparatus.
- 4K TV has gradually moved toward thousands of households, providing users with ultra-high definition visual enjoyment.
- the UK market research organization's future consulting report pointed out that it is estimated that global 4K TV shipments will reach 100 million units in 2018, and the Chinese market accounts for 70% of the global 4K TV market demand.
- 4K video has a large demand for transmission bandwidth.
- the basic 4K60P video requires a code rate of 30 to 50 Mbps to ensure good performance.
- China's current access network bandwidth is only 20Mbps, the peak rate is 18.4Mbps, and the average speed is only 3.4Mbps.
- the traditional universal video codec technology cannot guarantee the transmission of high quality 4K video in the existing bandwidth environment.
- the present application provides an encoding method and apparatus, which can solve the transmission of high quality video in an existing bandwidth environment.
- an encoding apparatus including: a down sampler, an up converter, a calculator, a self encoder, a main encoder, and a secondary encoder, wherein
- the down sampler is configured to receive an original resolution image, perform down sampling processing on the original resolution image, acquire a low resolution image, and send the acquired low resolution image to the up converter and the Main encoder
- the primary encoder is connected to the downsampler for encoding the received low resolution image to obtain primary coding information
- the up converter is coupled to the downsampler, configured to convert the low resolution image sent by the downsampler into a first full resolution image, and send the first full resolution image to The calculator;
- the calculator is configured to receive the original resolution image and the first full resolution image, and calculate a residual image according to the original resolution image and the first full resolution image, and the residual image
- the difference image is sent to the self-encoder for processing
- the self-encoder is connected to the calculator, and is configured to encode the residual image according to a self-encoding algorithm to obtain residual image encoding information, and send the residual image encoding information to the secondary encoder. ;
- the secondary encoder is connected to the self-encoder for entropy encoding the residual image coding information from the self-encoder to obtain secondary coding information.
- the encoding apparatus further includes: a filter booster connected to the up converter, configured to acquire a first full resolution image from the up converter, to the first full The resolution image is filtered and enhanced to obtain a second full resolution image; the calculator is further configured to calculate a residual image according to the original resolution image and the second full resolution image.
- the filter enhancer is specifically configured to perform filtering enhancement on the first full-resolution image by using a bilateral filtering algorithm to obtain a second full-resolution image.
- the calculator calculates a residual image according to the original resolution image and the second full resolution image, including:
- the original resolution image is subtracted from the pixel value at the same position as the second full resolution image to obtain a residual image.
- the self-encoder encodes the residual image according to a self-encoding algorithm to obtain residual image coding information, including:
- Each image block is separately encoded according to a preset self-encoder network parameter to obtain residual image coding information.
- a decoding apparatus including: a primary decoder, an up converter, a secondary decoder, a self decoder, and a synthesizer, wherein
- the primary decoder is configured to obtain primary coding information, and decode the primary coding information to obtain a low resolution image
- the up converter is connected to the main decoder for receiving the low resolution image from the main decoder, and converting the low resolution image to obtain a first full resolution image;
- the secondary decoder is configured to obtain secondary coding information, and perform entropy decoding on the secondary coding information to obtain residual image coding information.
- the self-decoder is connected to the auxiliary decoder, and is configured to self-decode the residual image coding information from the secondary decoder to acquire a residual image;
- a synthesizer coupled to the upconverter and the self decoder for transmitting the first full resolution image from the upconverter and the residual image from the self decoder The synthesis is performed to obtain a first original resolution image.
- the decoding apparatus further includes a filter enhancer coupled to the up converter for performing filter enhancement processing on the first full resolution image to obtain a second full resolution image.
- the synthesizer is further configured to synthesize the residual image and the second full resolution image to obtain a second original resolution image.
- the synthesizer is specifically configured to:
- an encoding method including:
- the original resolution image is input to a downsampler for downsampling processing to obtain a low resolution image
- the residual image coding information is input to a secondary encoder for entropy coding to obtain secondary coding information.
- a decoding method including:
- the residual image is synthesized with the first full resolution image to obtain a first original resolution image.
- an encoding apparatus having the function of implementing the encoding apparatus in the method described in the first aspect or the method described in the second aspect.
- This function can be implemented in hardware or in hardware by executing the corresponding software.
- the hardware or software includes one or more modules (or units) corresponding to the functions described above.
- a decoding apparatus having the function of implementing the decoding apparatus in the method of the second aspect described above.
- This function can be implemented in hardware or in hardware by executing the corresponding software.
- the hardware or software includes one or more modules (or units) corresponding to the functions described above.
- a computer program product comprising executable program code, wherein the program code comprises instructions that, when executed by the processor, cause the encoding device to perform the method as described in the above aspect Coding method.
- a computer program product comprising executable program code, wherein the program code comprises instructions that, when executed by the processor, cause the decoding device to perform the method as described in the above aspect Decoding method.
- embodiments of the present application provide a computer storage medium for storing computer software instructions for use in an encoding apparatus as described above, including a program designed to perform the above aspects.
- embodiments of the present application provide a computer storage medium for storing computer software instructions for use in a decoding apparatus as described above, including a program designed to perform the above aspects.
- a chip system comprising a processor for supporting an encoding device or a decoding device as described above to implement an encoding method or a decoding method as referred to in the above aspect.
- the chip system further includes a memory for holding program instructions and data necessary for the communication device.
- the chip system can be composed of chips, and can also include chips and other discrete devices.
- FIG. 1 is a schematic structural diagram of an encoding apparatus according to an embodiment of the present application.
- FIG. 2 is a schematic structural diagram of a decoding apparatus according to an embodiment of the present application.
- FIG. 3 is a schematic flowchart of an encoding method according to an embodiment of the present application.
- FIG. 4 is a schematic flowchart of a decoding method according to an embodiment of the present application.
- FIG. 5 is a schematic structural diagram of a self-encoder according to an embodiment of the present application.
- FIG. 6 is a schematic flowchart of a self-encoding method according to an embodiment of the present application.
- the encoding apparatus includes: a down sampler 101, an up converter 102, a calculator 104, a self-encoder 105, a secondary encoder 106, and a main Encoder 107.
- the down sampler 101 is configured to receive an original resolution image, perform down sampling processing on the original resolution image, acquire a low resolution image, and send the acquired low resolution image to the up converter 102 and the main
- the encoder 107 performs processing; the main encoder 107 is connected to the downsampler 101 for encoding the received low resolution image to output main coding information; the up converter 102 and the down sampler 101 Connected to convert the low resolution image sent by the downsampler 101 into a first full resolution image, send the first full resolution image to the calculator 104; the calculator 104 is configured to receive And calculating the residual image according to the original resolution image and the first full resolution image, and transmitting the residual image to the encoder 105.
- the encoder 105 is connected to the calculator 104, and configured to encode the residual image according to a self-encoding algorithm to obtain residual image encoding information, and send the residual image encoding information.
- the secondary encoder 106; a secondary encoder 106, and from the encoder 105 is connected, for future entropy encoding the residual image encoding information to the encoder 105 is obtained from the secondary encoded information.
- the encoding device further includes a filter booster 103 connected to the up converter 102, configured to acquire a first full resolution image from the up converter 102, and perform filtering enhancement on the first full resolution image. , obtaining a second full resolution image.
- a filter booster 103 connected to the up converter 102, configured to acquire a first full resolution image from the up converter 102, and perform filtering enhancement on the first full resolution image. , obtaining a second full resolution image.
- FIG. 2 is a schematic structural diagram of a decoding apparatus according to an embodiment of the present application, as shown in FIG. 2.
- the decoding apparatus includes a main decoder 201, an up converter 202, a synthesizer 204, a self decoder 205, and a sub decoder 206.
- the main decoder 201 is configured to obtain main coding information, and decode the main coding information to obtain a low resolution image.
- the up converter 202 is connected to the main decoder 201 for receiving from the main decoder 201.
- the low-resolution image is subjected to conversion processing to obtain a first full-resolution image; the secondary decoder 206 is configured to acquire secondary encoding information, and perform entropy decoding on the secondary encoding information to obtain a residual image.
- the difference image coding information is output to the self-decoder 205 for processing; the self-decoder 205 is coupled to the secondary decoder 206 for the residual image from the secondary decoder 206.
- the encoded information is self-decoded to obtain a residual image; a synthesizer 204 is coupled to the upconverter 202 and the self decoder 205, respectively, for the first full resolution image and the source from the upconverter 202
- the residual image described from the decoder is synthesized to obtain a first original resolution image.
- the decoding device further includes a filter enhancer 203, connected to the up converter 202, configured to perform filter enhancement processing on the first full resolution image to obtain a second full resolution image.
- a filter enhancer 203 connected to the up converter 202, configured to perform filter enhancement processing on the first full resolution image to obtain a second full resolution image.
- the self-encoder and the self-decoder may be a module or device, such as self-encoding.
- Auto Encoder which itself contains both an encoding part and a decoding part, the self-encoder can correspond to the encoding part of the self-encoder, and the self-decoder corresponds to the decoding part of the self-encoder.
- the encoder encodes the primary encoding information and the secondary encoding information, or the decoder pairs the primary encoding information and the secondary encoding.
- the decoding process of information can be implemented by a software program executing on a programmable device and/or other hardware device, such as a graphics processing unit (GPU), a field programmable gate array (FPGA), a central processing unit (CPU), and The implementation of the software program is executed on the computing device.
- the encoding process of the primary encoding information and the secondary encoding information by the encoding device, or the decoding process of the primary encoding information and the secondary encoding information by the decoding device may be at least partially hardware and/or embedded in a specific application integrated circuit ( The code in ASIC) is implemented together.
- the encoding method of an embodiment of the present application is described below with reference to FIG. 3. As shown in FIG. 3, the encoding method can be implemented by the encoding apparatus shown in FIG. 1, and includes the following steps:
- the downsampler functions to reduce the image and reduce the resolution of the image.
- the original resolution image of the input size of M ⁇ N it is downsampled by s times to obtain an image of (M/s) ⁇ (N/s) size.
- s should be the common divisor of M and N. Row.
- An alternative method is to convert the image in the original resolution image s ⁇ s window into a pixel whose value is the mean of all pixels in the window.
- Another method is to take a pixel of the original resolution image every (s-1) line and every (s-1) column to form a new image.
- the image of 4K resolution (3840 ⁇ 2160) is reduced to 1080P resolution (1920 ⁇ 1080), which is actually 2 times downsampling of 4K objects, and the image in 2 ⁇ 2 window in 4K image can be turned into one.
- Pixels which represent the pixels by the mean of all the pixels in the window; or one pixel at every other row and every other column of the original 4K image to form a reduced 1080P image.
- the primary encoder can be encoded using any coding standard, such as the H.264, H.265, and VP9 standards.
- the low-resolution images are sequentially subjected to predictive coding, transform coding, quantization loop post-processing, and entropy coding to obtain a binary code stream file.
- the role of the upconverter is to convert a low resolution image into a high resolution image, also known as upsampling, such as upsampling a 1080P image into a 4K image.
- Interpolation methods are generally used to insert new elements between pixels using appropriate interpolation algorithms based on the original image pixels. For example, a classical bicubic interpolation algorithm can be employed.
- the full resolution image obtained at this time is of low quality, so it is called a low quality full resolution image.
- the higher quality full resolution image is improved in image quality after filtering enhancement processing with respect to the low quality full resolution image as described above.
- the filter booster uses a Bilateral filter algorithm to enhance the low-quality full-resolution image obtained in step 303, so as to remove the image particle noise and maintain the image edge and detail texture, and the human eye visual attention.
- the mechanism is sensitive to information such as edge texture, and the subjective visual quality of the image can be improved by using the bilateral filtering algorithm.
- the bilateral filtering algorithm is a classical filtering enhancement algorithm. It is a nonlinear filtering method. It is a kind of compromise processing that combines the spatial proximity of the image and the similarity of the pixel values. At the same time, the spatial domain information and the grayscale similarity are considered to achieve the edge preservation.
- the purpose of denoising is simple, non-iterative, and local.
- step 304 is an optional step.
- step 305 Perform difference processing on the image of the original resolution and the high-quality full-resolution image obtained in step 304 to obtain a residual image.
- the image of the original resolution is A
- the image of the higher quality full resolution is B
- the pixel values at all the same positions in A and B are subtracted, and a residual image is obtained.
- the network structure of the constructed self-encoder encodes the plurality of image blocks of the residual image separately. For the specific implementation process, reference may be made to the self-encoding method as shown in FIG. 6.
- the so-called entropy coding refers to a lossless coding method according to the principle of information entropy. Encode the input information into a binary stream file.
- the entropy coding method used here may be Context-based Adaptive Binary Arithmetic Coding (CABAC), which is also widely used by the H.265 coding standard.
- CABAC Context-based Adaptive Binary Arithmetic Coding
- the self-encoded code output corresponding to each image block in step 306 is input to an entropy encoder, and CABAC is used as a binary code stream file to obtain auxiliary coded information of each image block.
- a decoding method according to an embodiment of the present application is described below based on FIG. 4. As shown in FIG. 4, the decoding method includes the following steps:
- the main coding information is input to the main decoder for decoding, and the main decoder can adopt any decoding standard, for example, adopting standards such as H.264, H.265, and VP9.
- the main decoder can adopt any decoding standard, for example, adopting standards such as H.264, H.265, and VP9.
- H.265 adopting standards such as H.264, H.265, and VP9.
- the binary code stream file obtained after encoding is sequentially subjected to entropy decoding, inverse quantization, inverse transform and the like to decode a low resolution image.
- this step may adopt the same method as step 303 of FIG. 3, such as using a classical bicubic interpolation algorithm to convert a low resolution image into a low quality full resolution image.
- step 304 of FIG. 3 the same method as step 304 of FIG. 3 may be used, such as using a Bilateral filter algorithm to enhance the low-quality full-resolution image, so as to remove image particle noise and maintain image edge and detail texture. purpose.
- the secondary coding information is input to the secondary decoder, and the residual image coding information is obtained after entropy decoding.
- This step is opposite to step 307 of FIG. 3, and the binary stream file is decoded by the CABAC entropy decoding algorithm, and each image block is decoded in turn, and the self-encoded L 2 layer output corresponding to each image block is obtained.
- the resolution of the residual image obtained in step 405 is the same as the higher quality full resolution image obtained in step 403.
- the residual image recovered in step 405 is A
- the high-quality full-resolution image obtained in step 403 is B
- the pixel values at all the same positions in A and B are added to obtain reconstruction. After the original resolution image.
- the decoding method of this embodiment is the inverse of the encoding method of the embodiment of FIG. 3, and the related details may be referred to and applied to each other, and details are not described herein again.
- the so-called self-encoder is a fully connected neural network, setting the target output value of the network equal to the input value.
- a self-encoder with 6 nodes is shown in Figure 5:
- the self-encoder includes three layers.
- the input layer L1 has six common nodes and one offset node.
- the circle labeled "+1" is called a bias node;
- the L2 layer contains three common nodes and one offset.
- Node; output layer L3 contains 6 ordinary nodes.
- Each common node of L2 is connected to all nodes of the L1 layer by one edge, and each connected node has a weight parameter w.
- Each common node of L3 is connected with one edge of all nodes of the L2 layer, each There will also be a weight parameter w between the connected nodes, taking the output of the L2 layer as an example, and the output of the first node of the L2 layer. for:
- the weight parameter between the first node of the L1 layer and the first node of the L2 layer The weight parameter between the second node of the L1 layer and the first node of the L2 layer, The weight parameter between the third node of the L1 layer and the first node of the L2 layer, and so on.
- f(.) represents the activation function, here the sigmoid function is used.
- W l-1 represents the set of weight parameters between the l-1 layer and the l layer
- b l-1 represents the set of the bias term parameters of the l-1 layer and the first layer
- the key is the calculation of the weight w of the edge connected between each node, which is generally divided into the following steps:
- the training samples are generally selected to be more than 10,000.
- loss function J(w,b) that is, the degree of the original input loss compared to the output of the self-encoder, is generally expressed by the square of the difference between the two.
- the back propagation algorithm (BP) is used to calculate the weight w layer by layer.
- the output of the L3 layer is calculated according to the equation (3), that is, the recovery of the original actual input 6-dimensional vector is obtained, which is the decoding process of the self-encoder.
- the self-encoding method in the embodiment of the present application is further introduced in the following with reference to FIG. 6.
- the self-encoding method includes the following steps:
- each image block includes N pixels, the corresponding input layer L1 has N corresponding to the pixel points.
- the output layer L3 contains N ordinary nodes.
- Each common node of L2 is connected to all nodes of the L1 layer by one edge, and each connected node has a weight parameter w; similarly, each common node of L3 uses one edge with all nodes of the L2 layer. Connected, there will be a weight parameter w between each connected node.
- the output of the first node of the L2 layer for:
- a weight parameter is the weight between the second node N and L L. 1 layer 2 layer of the first node.
- Weight is the weight parameter bias node between L and L. 1 layer 2 layer of the first node.
- f(.) represents the activation function, here the sigmoid function is used.
- the L 3 layer has a total of N nodes. Similarly, the output of the first, second, and Nth nodes of the L 3 layer is:
- the output L N layer 3 comprises the original input image block of N pixels equal.
- Another possible loss function construction method is to add sparse constraints to the intermediate layer L 2 .
- the so-called sparse constraint means that when the output of a node of the L 2 layer is close to 1, we think that it is activated, and when the output of the node is close to 0, it is considered to be suppressed, and as many as possible, the number of suppression nodes of the L 2 layer is large. Then the sparse constraint is reached.
- the jth output of the L 2 layer The average activity relative to an image block containing N pixels (x 1 to x N ) is:
- join restrictions ⁇ is the sparsity coefficient, usually a small value close to zero, such as 0.05.
- an additional penalty factor needs to be added to the constructed loss function, which will punish those that are significantly different from ⁇ . Therefore, the average activity of each node in the L 2 layer is kept in a small range, and the penalty factor is:
- our goal is to determine the optimal parameters W and b, taking J(W,b) to a minimum, which needs to be achieved by the gradient descent method.
- the network parameters of the self-encoder are initialized with random numbers, and the training samples are sequentially input into the self-encoder; then the loss function J(W, b) is obtained according to the difference between the output of the self-encoder and the original input; (a total of three layers) parameters are expressed as W l , the first layer bias term is expressed as b l , and the partial derivative is calculated:
- ⁇ is called the learning speed, and the range of values is generally [0.01, 0.1], which is a smaller real number.
- the key to solving the gradient descent method is to calculate the partial derivative of the cost function J(W,b) for each layer of parameters, which needs to be implemented by the Back Propagation (BP) algorithm (this is a classic algorithm, no longer here. Brief description).
- BP Back Propagation
- the final optimal weight parameter W l and the offset term parameter b l of each layer of the self-encoder are obtained.
- each image block of the residual image is input to an encoding portion of the self-encoder to be encoded.
- the coding portion of the self-encoder refers to the input layer Layer L 1 and the intermediate layer Layer L 2 and the network structure therebetween.
- the decoding process of the self-encoder corresponds to the encoding process inversely. Specifically, as shown in step 405 of FIG. 4, the entropy decoded result is input to the decoding part of the self-encoder, and the residual image is restored.
- the specific process is as follows:
- the decoding portion of the self-encoder refers to the intermediate layer L 2 and the output layer L 3 and the network structure therebetween. After constructing the network structure of the self-encoder and training the network parameters, what you need to do is to decode the entropy and get the output of the L 2 layer of the self-encoder. to Connected to the L 3 layer and calculated according to the formula (6), the output of the L 3 layer is obtained, which is the decoding result of the self-encoder, that is, the approximate recovery of the original image block containing N pixel points.
- the entropy decoding results of all k image blocks m 1 , . . . m k in the original residual image are sequentially input to the decoding portion of the self-encoder, and the approximate restorations m ' 1 , . . . m ' k of the image blocks are obtained after decoding.
- These approximate restored image blocks are combined in accordance with the positions of m 1 , . . . m k in the original residual image to obtain a restored residual image.
- the codec method based on the self-encoder is nonlinear, and the distortion of the original residual image is less in the encoding and decoding process, so that the reconstructed video image has better subjective quality and objective quality.
- both the pre-processing module and the post-processing module need to store a reference image matrix with a large amount of data
- the self-described embodiment is based on The encoding and decoding method of the encoder only needs to store the parameters of the encoding part in the encoder (the parameters between the input layer L1 and the intermediate layer L2) and the decoding parameters (the parameters between the intermediate layer L2 and the output layer L3) stored from the encoder. ), the amount of data is small, saving storage space.
- the disclosed systems, devices, and methods may be implemented in other manners.
- the device embodiments described above are merely illustrative.
- the division of the unit is only a logical function division.
- there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
- the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, or an electrical, mechanical or other form of connection.
- the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the embodiments of the present application.
- each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
- the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
- the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
- the technical solution of the present application may be in essence or part of the contribution to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium.
- a number of instructions are included to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present application.
- the foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like. .
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
本发明实施例提供一种编解码方法和装置,其中,编码方法包括:先将原始分辨率图像进行下采样处理,获取低分辨率图像,然后将该低分辨率图像输入主编码器进行编码,获得主编码信息;另外还将该低分辨率图像输入上转换器进行上转换处理,获取全分辨率图像,然后根据所述原始分辨率图像与该全分辨率图像计算得到残差图像,随后将所述残差图像输入自编码器进行编码,获取残差图像编码信息,将所述残差图像编码信息输入辅编码器进行熵编码获得辅编码信息。这样可将4K或更高质量的图像拆分成主编码信息和辅编码信息分开传输,能够满足当前网络环境下,实现4K或更高质量的视频传输,并且能够保证重构后的视频图像质量。
Description
本申请要求于2017年9月21日提交中国专利局、申请号为201710861871.6、发明名称为“一种编码、解码方法及装置”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及图像处理技术,具体的,涉及一种编码、解码方法及装置。
随着用户群体对4K视频的需求日益增大,4K电视逐渐走向千家万户,为用户提供了超高清的视觉享受。英国市场研究机构未来咨询报告指出,预计2018年全球4K电视出货量将达到1亿台,中国市场占据全球4K电视市场需求总量的70%。但是,相比1080P视频仅需8~10Mbps的码率就能得到较好画质而言,4K视频对传输带宽的需求大增,基本的4K60P视频就需要30~50Mbps的码率才能保证良好的画面体验,根据统计报告,中国现在的接入网带宽仅为20Mbps,峰值速率为18.4Mbps,平均速度更是仅有可怜的3.4Mbps。如此情况,传统通用的视频编解码技术已无法保证在现有带宽环境下传输高质量的4K视频。
发明内容
有鉴于此,本申请提供一种编码方法和装置,可以解决在现有带宽环境下传输高质量的视频。
第一方面,提供一种编码装置,包括:下采样器,上转换器,计算器,自编码器、主编码器和辅编码器,其中,
所述下采样器,用于接收原始分辨率图像,对所述原始分辨率图像进行下采样处理,获取低分辨率图像,将所述获取的低分辨率图像发送给所述上转换器和所述主编码器;
所述主编码器,与所述下采样器相连,用于对接收的所述低分辨率图像进行编码,获得主编码信息;
所述上转换器,与所述下采样器相连,用于将所述下采样器发送的所述低分辨率图像转换为第一全分辨率图像,将所述第一全分辨率图像发送给所述计算器;
所述计算器,用于接收所述原始分辨率图像和所述第一全分辨率图像,根据所述原始分辨率图像与所述第一全分辨率图像计算得到残差图像,将所述残差图像发给所述自编码器进行处理;
所述自编码器,与所述计算器相连,用于根据自编码算法对所述残差图像进行编码,获得残差图像编码信息,将所述残差图像编码信息发送给所述辅编码器;
所述辅编码器,与所述自编码器相连,用于将来自所述自编码器的残差图像编码信息进行熵编码获得辅编码信息。
参考第一方面的可选方式中,该编码装置还包括:与所述上转换器相连的滤波增强器,用于从所述上转换器获取第一全分辨率图像,对所述第一全分辨率图像进行滤波增强,获得第二全分辨率图像;所述计算器还用于,根据所述原始分辨率图像和第二全分辨率图像计算得到残差图像。
参考第一方面的可选方式中,所述滤波增强器具体用于,采用双边滤波算法对所述第一全分辨率图像进行滤波增强,获得第二全分辨率图像。
参考第一方面的可选方式中,所述计算器根据所述原始分辨率图像与所述第二全分辨率图像计算得到残差图像,包括:
将所述原始分辨率图像与所述第二全分辨率图像相同位置处的像素值相减,获得残差图像。
参考第一方面的可选方式中,所述自编码器根据自编码算法对所述残差图像进行编码获得残差图像编码信息,包括:
将所述残差图像划分为多个图像块;
根据预置的自编码器网络参数,对每个图像块分别进行编码,获得残差图像编码信息。
第二方面,提供一种解码装置,包括:主解码器,上转换器,辅解码器,自解码器以及合成器,其中,
所述主解码器,用于获取主编码信息,将所述主编码信息进行解码,获得低分辨率图像;
所述上转换器,与所述主解码器相连,用于接收来自所述主解码器的所述低分辨率图像,将所述低分辨率图像进行转换处理,获得第一全分辨率图像;
所述辅解码器,用于获取辅编码信息,将所述辅编码信息进行熵解码,获得残差图像编码信息;
所述自解码器,与所述辅解码器相连,用于对来自所述辅解码器的所述残差图像编码信息进行自解码,获取残差图像;
所述合成器,与所述上转换器和所述自解码器相连,用于将来自所述上转换器的所述第一全分辨率图像和来自所述自解码器的所述残差图像进行合成,获得第一原始分辨率图像。
参考第二方面的可选方式中,该解码装置还包括滤波增强器,与所述上转换器相连,用于对所述第一全分辨率图像进行滤波增强处理,获得第二全分辨率图像;所述合成器还用于将所述残差图像与所述第二全分辨率图像合成,获得第二原始分辨率图像。
参考第二方面的可选方式中,所述合成器具体用于:
将所述第一或者第二全分辨率图像与所述残差图像的相同位置处的像素值相加,获得第一或第二原始分辨率图像。
第三方面,提供一种编码方法,包括:
将原始分辨率图像输入下采样器进行下采样处理,获取低分辨率图像;
将所述低分辨率图像输入主编码器进行编码,获得主编码信息;
将所述低分辨率图像输入上转换器进行转换处理,获取第一全分辨率图像;
根据所述原始分辨率图像与所述第一全分辨率图像计算得到残差图像;
将所述残差图像输入自编码器进行编码,获取残差图像编码信息;
将所述残差图像编码信息输入辅编码器进行熵编码获得辅编码信息。
第四方面,提供一种解码方法,包括:
获取主编码信息,将所述主编码信息进行解码,获得低分辨率图像;
将所述低分辨率图像输入上转换器进行上转换处理,获得第一全分辨率图像;
获取辅编码信息,将所述辅编码信息输入辅解码器进行熵解码,获得残差图像编码信息;
将所述残差图像编码信息输入自解码器进行自解码,获取残差图像;
将所述残差图像与所述第一全分辨率图像合成,获得第一原始分辨率图像。
第五方面,提供一种编码装置,该装置具有实现上述第一方面所述的方法或第二方面中所述的方法中编码装置的功能。该功能可以通过硬件实现,也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块(或单元)。
第六方面,提供一种解码装置,该解码装置具有实现上述第二方面所述的方法中解码装置的功能。该功能可以通过硬件实现,也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块(或单元)。
第七方面,提供一种计算机程序产品,包括可执行程序代码,其中所述程序代码包括指令,当所述处理器执行所述指令时,所述指令使编码装置可执行如上述方面所述的编码方法。
第八方面,提供一种计算机程序产品,包括可执行程序代码,其中所述程序代码包括指令,当所述处理器执行所述指令时,所述指令使解码装置可执行如上述方面所述的解码方法。
第九方面,本申请实施例提供了一种计算机存储介质,用于储存为如上所述的编码装置所用的计算机软件指令,其包含用于执行上述方面所设计的程序。
第十方面,本申请实施例提供了一种计算机存储介质,用于储存为如上所述的解码装置所用的计算机软件指令,其包含用于执行上述方面所设计的程序。
第十一方面,提供了一种芯片***,该芯片***包括处理器,用于支持如上所述的编码装置或解码装置实现如上述方面中所涉及的编码方法或解码方法。在一种可能的设计中,所述芯片***还包括存储器,所述存储器,用于保存通信设备必要的程序指令和数据。该芯片***,可以由芯片构成,也可以包含芯片和其他分立器件。
通过上述方面,可以使得满足当前网络环境下,实现4K或更高质量的视频传输,并且能够保证重构后的视频图像质量。
图1为本申请一实施例的编码装置结构示意图;
图2为本申请一实施例的解码装置结构示意图;
图3为本申请一实施例的编码方法流程示意图;
图4为本申请一实施例的解码方法流程示意图;
图5为本申请一实施例的自编码器结构示意图;
图6为本申请一实施例的自编码方法流程示意图。
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行描述。
图1为本申请一实施例的编码装置结构示意图,如图1所示,该编码装置包括:下采样器101、上转换器102、计算器104、自编码器105、辅编码器106和主编码器107。其中,下采样器101,用于接收原始分辨率图像,对所述原始分辨率图像进行下采样处理,获取低分辨率图像,将所述获取的低分辨率图像发送给上转换器102和主编码器107分别进行处理;主编码器107,与所述下采样器101相连,用于对接收的所述低分辨率图像进行编码,输出主编码信息;上转换器102,与下采样器101相连,用于将所述下采样器101发送的所述低分辨率图像转换为第一全分辨率图像,将所述第一全分辨率图像发送给计算器104;计算器104,用于接收所述原始分辨率图像和所述第一全分辨率图像,根据所述原始分辨率图像与所述第一全分辨率图像计算得到残差图像,将所述残差图像发给自编码器105进行处理;自编码器105,与所述计算器104相连,用于根据自编码算法对所述残差图像进行编码,获得残差图像编码信息,将所述残差图像编码信息发送给辅编码器106;辅编码器106,与自编码器105相连,用于将来于自编码器105的残差图像编码信息进行熵编码获得辅编码信息。
可选的,该编码装置还包括与上转换器102相连的滤波增强器103,用于从所述上转换器102获取第一全分辨率图像,对所述第一全分辨率图像进行滤波增强,获得第二全分辨率图像。
图2为本申请一实施例中解码装置结构示意图,如图2所示。该解码装置包括:主解码器201,上转换器202,合成器204、自解码器205和辅解码器206。其中,主解码器201,用于获取主编码信息,将所述主编码信息进行解码,获得低分辨率图像;上转换器202,与主解码器201相连,用于接收来自主解码器201的所述低分辨率图像,将所述低分辨率图像进行转换处理,获得第一全分辨率图像;辅解码器206,用于获取辅编码信息,将所述辅编码信息进行熵解码,获得残差图像编码信息,将所述残差图像编码信息输出给自解码器205进行处理;自解码器205,与辅解码器206相连,用于对来自所述辅解码器206的所述残差图像编码信息进行自解码,获取残差图像;合成器204,分别与上转换器202和自解码器205相连,用于将来自所述上转换器202的所述第一全分辨率图像和来自所述自解码器的所述残差图像进行合成,获得第一原始分辨率图像。
可选的,该解码装置还包括滤波增强器203,与所述上转换器202相连,用于对所述第一全分辨率图像进行滤波增强处理,获得第二全分辨率图像。
需要注意的是,为了便于说明和区分,本实施例中,使用了自编码器和自解码器的概念,在实际应用中,自编码器和自解码器可以是一个模块或者装置,比如自编码器(Auto Encoder,AE),其本身既包含编码部分也包含解码部分,自编码器可以对应自编码器的编码部分,自解码器对应自编码器的解码部分。
需要注意的是,上述编码装置和解码装置的各个模块可以通过硬件或者软件实现,在一个示例中,编码器对主编码信息和辅编码信息的编码过程,或者解码器对主编码信息和辅编码信息的解码过程可以由在可编程设备和/或其他硬件设备上执行的软件程序来实现,诸如在图形处理器单元(GPU),现场可编程门阵列(FPGA),中央处理单元(CPU)以及计算设备上执行软件程序实现。在另一个示例中,编码装置对主编码信息和辅编码信息的编码过程,或者解码装置对主编码信息和辅编码信息的解码过程可以通过至少部分地硬件和/或嵌入在特定应用集成电路(ASIC)中的代码共同实现。
下面结合图3对本申请一实施例的编码方法进行阐述,如图3所示,该编码方法可由如 图1所示的编码装置实现,包括如下步骤:
301、将原始分辨率图像输入下采样器模块,计算得到低分辨率的图像。
具体的,下采样器的作用是缩小图像,降低图像的分辨率。对于输入的尺寸为M×N的原始分辨率图像,对其进行s倍下采样,即得到(M/s)×(N/s)尺寸的图像,当然s应该是M和N的公约数才行。
一种可选的方法,就是把原始分辨率图像s×s窗口内的图像变成一个像素,这个像素点的值就是窗口内所有像素的均值。另一种办法就是将原始分辨率图像每隔(s-1)行、每隔(s-1)列取一个像素点,组成一个新的图像。比如将4K分辨率(3840×2160)的图像缩小到1080P分辨率(1920×1080),实际上是对4K对象进行2倍下采样,可以将4K图像中2×2窗口内的图像变成一个像素,用窗口内所有像素的均值代表该像素;或者对原4K图像每隔一行、每隔一列去一个像素点,组成一幅缩小的1080P图像。
302、将低分辨率图像输入主编码器进行编码,输出主编码信息。
可选的,主编码器可采用任意的编码标准进行编码,比如,H.264、H.265、VP9标准。以H.265编码标准为例,将低分辨率图像分别依次进行预测编码、变换编码、量化环路后处理、熵编码,得到二进制的码流文件。
303、将低分辨率图像送入上转换器,获得低质量的全分辨率图像。
上转换器的作用是将低分辨率图像转换为高分辨率图像,又称为上采样,如将1080P图像上采样为4K图像。一般都采用内插值方法,即在原有图像像素的基础上在像素点之间采用合适的插值算法***新的元素。例如可采用经典的双三次插值算法。此时得到的全分辨率图像质量较低,故称为低质量的全分辨率图像。
304、将低质量的全分辨率视频图像进行滤波增强处理,得到质量较高的全分辨率图像。
该质量较高的全分辨率图像是相对于如上所述的低质量的全分辨率图像而言,经过滤波增强处理后,图像质量有提升。
具体的,滤波增强器采用双边滤波(Bilateral filter)算法对步骤303得到的低质量全分辨率图像进行增强处理,以便达到去除图像颗粒噪声,保持图像边缘、细节纹理的目的,而人眼视觉注意机制对于这些边缘纹理等信息比较敏感,采用双边滤波算法增强后可提高图像的主观视觉质量。双边滤波算法是经典的滤波增强算法,是一种非线性的滤波方法,是结合图像的空间邻近度和像素值相似度的一种折衷处理,同时考虑空域信息和灰度相似性,达到保边去噪的目的,具有简单、非迭代、局部的特点。
需要注意的是,步骤304为可选步骤。
305、将原始分辨率的图像与步骤304中获得的质量较高的全分辨率图像进行差值处理,得到残差图像;
具体而言,假设原始分辨率的图像为A,质量较高的全分辨率图像为B,将A与B中所有相同位置处的像素值相减,就得到残差图像。
306、对残差图像采用自编码器进行编码,获得残差图像编码信息。
具体的,首先将残差图像划分为多个n×n的图像块,每个图像块共包含N=n×n个像素点;其次构造自编码器的网络结构,可参考图5,然后利用构造好的自编码器的网络结构对该残差图像的多个图像块分别进行编码。具体实施过程可参考如图6所示的自编码方法。
307、将自编码器编码后的残差图像编码信息送入辅编码器进行熵编码,生成最终的辅 编码信息。
具体的,所谓熵编码是指按照信息熵原理进行的无损编码方式。将输入信息编码为二进制码流文件。这里采用的熵编码编码方法可为上下文自适应算法二进制编码(Context-based Adaptive Binary Arithmetic Coding,CABAC),该方法也被H.265编码标准广泛使用。具体而言,将步骤306中每个图像块对应的自编码后的编码输出,输入到熵编码器中,采用CABAC编码为二进制码流文件,得到该各个图像块的辅助编码信息。
下面基于图4对本申请一实施例的解码方法进行说明,如图4所示,该解码方法包括如下步骤:
401、获取主编码信息,对所述主编码信息进行解码,获得低分辨率图像。
具体的,将主编码信息输入主解码器进行解码,主解码器可采用任意的解码标准,比如采用H.264、H.265、VP9等标准。以H.265标准为例,将编码后得到的二进制的码流文件依次进行熵解码、反量化、反变换等步骤,解码出低分辨率的图像。
402、将低分辨率图像输入上转换器进行转换处理,获得低质量的全分辨率图像。
可选的,本步骤可采用如图3步骤303相同的方法,如采用经典的双三次插值算法,将低分辨率图像转化为低质量的全分辨率图像。
403、将低质量的全分辨率视频图像进行滤波增强处理,得到质量较高的全分辨率图像。
可选的,本步骤可采用如图3步骤304相同的方法,如使用双边滤波(Bilateral filter)算法对低质量全分辨率图像进行增强,以便达到去除图像颗粒噪声,保持图像边缘、细节纹理的目的。
404、获取辅编码信息,对辅编码信息进行熵解码,获得残差图像编码信息。
具体的,将辅编码信息输入辅解码器,熵解码后得到残差图像编码信息。
405、将熵解码后的残差图像编码信息输入到自编码器的解码部分,恢复出残差图像。
具体实现方式可参考如下文所述的自编码器的解码过程。
需要注意的是,步骤405获得的残差图像的分辨率与步骤403得到的质量较高的全分辨率图像一样。
406、将步骤405恢复出的残差图像与步骤403得到的质量较高的全分辨率图像合成,得到重构后的原始分辨率图像。
具体而言,假设步骤405恢复出的残差图像为A,步骤403得到的质量较高的全分辨率图像为B,将A与B中所有相同位置处的像素值相加,就得到重构后的原始分辨率图像。
本实施例的解码方法是图3实施例的编码方法的逆过程,相关细节可以互相参考和应用,在此不再赘述。
下面参考图5对本申请实施例中的自编码器作进一步介绍。
所谓自编码器就是一个全连接的神经网络,设定网络的目标输出值等于输入值,一个包含6个节点的自编码器如图5所示:
该自编码器是对一个6维的向量进行x=[x1,x2,…x6]进行自编码,使得输出h
w,b(x)=x。该自编码器共包含三层,输入层L1有6个普通节点和一个偏置节点,标上“+1”的圆圈被称为偏置节点;L2层共含有3个普通节点和一个偏置节点;输出层L3包含6个普通节点。L2的每个 普通节点都与L1层的所有节点用一条边相连,每个相连节点之间都会有一个权重参数w,L3的每个普通节点都与L2层的所有节点用一条边相连,每个相连节点之间也都会有一个权重参数w,以L2层的输出为例,L2层第一个节点的输出
为:
其中,
为L1层的第一个节点与L2层的第一个节点之间的权重参数,
为L1层的第二个节点与L2层的第一个节点之间的权重参数,
为L1层的第三个节点与L2层的第一个节点之间的权重参数,以此类推。
为L1层的偏置节点与L2层的第一个节点之间的权重参数。f(.)表示激活函数,这里采用sigmoid函数。以此类推,可以得到L
2层第二个、第三个节点的输出
和
假设用W
l-1表示l-1层与l层之间的权重参数的集合,用b
l-1表示l-1层与第l层的偏置项参数的集合,则第l个全连接层的输出可简单表示为:
X
l=W
l-1X
l-1+b
l-1。
以此类推,与L
2层相似,输出层L
3的每个节点的输出值为:
并且输出值与输入值相等,如此便形成了一个自编码器。其关键是每个节点之间相连的边的权重w的计算,一般分为如下几个步骤:
首先,收集大量的6维训练样本,作为自编码器的输入,为保证参数训练的准确性,训练样本一般选取10000个以上。
其次,构造损失函数J(w,b),即自编码器的输出相比原始的输入损失的程度,一般采用两者差值的平方来表示。
最后,根据损失函数,采用反向传播算法(Back propagation Algorithm,BP),逐层计算权重w。
当训练好自编码器的参数后,可对实际输入的6维向量x=[x1,x2,…x6]进行自编码:即根据L1层和L2层之间的参数,按式(1)、(2)计算得到L2层的3维输出a=[a1,a2,a3],这便是自编码器的编码过程,此过程能够很好地实现原始数据的压缩。如果对L2层添加稀疏约束,使尽量多的L2层的节点输出值为0,便可进一步压缩数据量。
根据L2层和L3层之间的参数,按式(3)计算得到L3层的输出,即得到了对原始实际输入 的6维向量的恢复,这便是自编码器的解码过程。
下面结合图6对本申请实施例中自编码方法作进一步介绍,该自编码方法包括如下步骤:
601、将残差图像划分为多个图像块。
具体的,可将残差图像划分为n×n的图像块,每个图像块共包含N=n×n个像素点。
602、构造自编码器的网络结构。
一种具体实施方案中,可参照图5所示的三层网络,不过节点数目有所变动:由于每个图像块包含N个像素点,那么对应的输入层L1有N个与像素点对应的普通节点和一个偏置节点;L2层共含有M=N/2个普通节点(或取一个小于N的数)和一个偏置节点;输出层L3包含N个普通节点。L2的每个普通节点都与L1层的所有节点用一条边相连,每个相连节点之间都会有一个权重参数w;同样的,L3的每个普通节点都与L2层的所有节点用一条边相连,每个相连节点之间都会有一个权重参数w。以L2层的输出为例,L2层第一个节点的输出
为:
其中,
为L
1层的第一个节点与L
2层的第一个节点之间的权重参数,
为L
1层的第二个节点与L
2层的第一个节点之间的权重参数,以此类推,
为L
1层的第N个节点与L
2层的第一个节点之间的权重参数。
为L
1层的偏置节点与L
2层的第一个节点之间的权重参数。f(.)表示激活函数,这里采用sigmoid函数。以此类推,可以得到L
2层第二个直至第M个节点的输出:
L
3层共有N个节点,类似的,L
3层第一个、第二个直至第N个节点的输出为:
自编码器的目标是使输出等于输入,这里就是使L
3层的N个输出与原始的包含N个像素点的图像块输入相等。
603、训练自编码器的网络参数,即训练每个节点之间相连的边的权重w。
首先,选取多种类型的视频图像,包括动画片、室内场景、室外场景等,将图像划分为N=n×n的图像块,作为训练样本。为保证网络参数学习的充分性,选取的训练样本最好足够多,一般取10000个以上;
其次,构造损失函数J(W,b),即自编码器的输出相比原始的输入损失的程度:
另一种可行的损失函数构造方式是对中间层L
2增加稀疏约束。所谓稀疏约束是指:当L
2 层某节点输出接近于1的时候我们认为它被激活,而节点输出接近于0的时候认为它被抑制,尽可能的使L
2层的抑制节点数目多,则达到了稀疏约束。具体而言,L
2层的第j个输出
相对于包含N个像素点(x
1到x
N)的图像块的平均活跃度为:
加入限制
ρ是稀疏性系数,通常是一个接近于0的较小值,如0.05。为了实现这一限制,需要在构造的损失函数中增加一个额外的惩罚因子,这个惩罚因子会将惩罚那些与ρ有显著不同的
从而使L
2层各节点的平均活跃度保持在较小范围内,惩罚因子为:
其中M表示L
2层节点数目。该惩罚因子可表达为相对熵(KL divergence)的形式:
那么,增加了稀疏约束后的损失函数为:
其中β是控制稀疏性惩罚因子的权重。此时便构造好了损失函数。
我们的目标是确定最优的参数W和b,使J(W,b)取最小值,这需要通过梯度下降法实现。首先,用随机数初始化自编码器的网络参数,将训练样本依次输入到自编码器中;接着根据自编码器的输出和原始输入的差异得到损失函数J(W,b);假设第l层(一共三层)参数表示为W
l,第l层偏置项表示为b
l,计算偏导数:
然后,利用偏导数更新W
l和b
l:
其中α称为学习速度,取值范围一般为[0.01,0.1],为一较小的实数。梯度下降法求解的关键是计算代价函数J(W,b)对每层参数的偏导数,这需要通过反向传播(Back propagation Algorithm,BP)算法实现(这是一个经典的算法,这里不再赘述)。
当依次训练完所有的输入训练样本后,便得到自编码器每一层最终的最优权重参数W
l和偏置项参数b
l。
604、根据训练好的自编码器网络参数,将残差图像的每个图像块输入到自编码器的编码部分进行编码。
所谓自编码器的编码部分,是指输入层Layer L
1和中间层Layer L
2及其之间的网络结构。图像块包含N=n×n个像素点,那么自编码器L
1层就有N个普通节点和一个偏置节点;L
2层含有M=N/2个普通节点(或取一个小于N的数)和一个偏置节点;当按照步骤603训练好各个节点之间相连的边权重w之后,便可按照公式(4)~(5)计算得到L
2层的各个节点的输出
至
由于M<N,L
2层的输出
至
就实现了对原始数据的压缩,也即实现了编码过程。
至
能够更好反应原始数据之间的一种内在特征。更进一步,当增加了步骤603中描述的稀疏约束,即L
2层中尽量多的节点输出接近于0,则可更好地在后续熵编码步骤中进一步压缩数据量。
自编码器的解码过程与编码过程逆对应,具体,如图4所示的步骤405,将熵解码后的结果输入到自编码器的解码部分,恢复出残差图像,具体过程如下:
所谓自编码器的解码部分,是指中间层L
2和输出层L
3及其之间的网络结构。当按照构造好自编码器的网络结构并训练好网络参数后,所需做的就是将熵解码后得到的“自编码器的L
2层的输出
至
”与L
3层相连,按照公式(6)计算,得到L
3层的输出,就是自编码器的解码结果,即包含N个像素点的原图像块的近似恢复。
将原残差图像中所有k个图像块m
1,…m
k的熵解码结果都依次输入到自编码器的解码部分,解码后得到这些图像块的近似恢复m
’
1,…m
’
k,将这些近似恢复的图像块按照m
1,…m
k在原残差图像中的位置进行组合,便得到恢复出的残差图像。
如上所述的编解码方法,基于自编码器的编解码方法是非线性的,在编解码过程中相对原始残差图像的失真较少,从而重构后的视频图像主观质量、客观质量更优。另外,相比传统的重构视频编码(Reconstructive Video Coding,RVC)方法中需要在预处理模块和后处理模块都存储一份数据量较大的参考图像矩阵,而本实施例所述的基于自编码器的编解码方法仅需存储自编码器中编码部分的参数(输入层L1与中间层L2之间的参数)和存储自编码器的解码参数(中间层L2与输出层L3之间的参数),数据量较小,节省存储空间。
本领域普通技术人员可以意识到,结合本文中所公开的实施例描述的各示例的单元及算法步骤,能够以电子硬件、计算机软件或者二者的结合来实现,为了清楚地说明硬件和软件的可互换性,在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行,取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能,但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到,为了描述的方便和简洁,上述描述的***、装置和单元的具体工作过程,可以参考前述方法实施例中的对应过程,在此不再赘述。
在本申请所提供的几个实施例中,应该理解到,所揭露的***、装置和方法,可以通过其它的方式实现。例如,以上所描述的装置实施例仅仅是示意性的,例如,所述单元的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如多个单元或组件可以结合或者可以集成到另一个***,或一些特征可以忽略,或不执行。另外,所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口、装置或单元的间接耦合或通信连接,也可以是电的,机械的或其它的形式连接。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的,作为单元显示的部件可以是或者也可以不是物理单元,即可以位于一个地方,或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本申请实施例方案的目的。
另外,在本申请各个实施例中的各功能单元可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以是两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时,可以 存储在一个计算机可读取存储介质中。基于这样的理解,本申请的技术方案本质上或者说对现有技术做出贡献的部分,或者该技术方案的全部或部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得一台计算机设备(可以是个人计算机,服务器,或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括:U盘、移动硬盘、只读存储器(ROM,Read-Only Memory)、随机存取存储器(RAM,Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述,仅为本申请的具体实施方式,但本申请的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本申请揭露的技术范围内,可轻易想到各种等效的修改或替换,这些修改或替换都应涵盖在本申请的保护范围之内。因此,本申请的保护范围应以权利要求的保护范围为准。
Claims (17)
- 一种编码装置,其特征在于,包括:下采样器,上转换器,计算器,自编码器、主编码器和辅编码器,其中,所述下采样器,用于接收原始分辨率图像,对所述原始分辨率图像进行下采样处理,获取低分辨率图像,将所述获取的低分辨率图像发送给所述上转换器和所述主编码器;所述主编码器,与所述下采样器相连,用于对接收的所述低分辨率图像进行编码,获得主编码信息;所述上转换器,与所述下采样器相连,用于将所述下采样器发送的所述低分辨率图像转换为第一全分辨率图像,将所述第一全分辨率图像发送给所述计算器;所述计算器,用于接收所述原始分辨率图像和所述第一全分辨率图像,根据所述原始分辨率图像与所述第一全分辨率图像计算得到残差图像,将所述残差图像发给所述自编码器进行处理;所述自编码器,与所述计算器相连,用于根据自编码算法对所述残差图像进行编码,获得残差图像编码信息,将所述残差图像编码信息发送给所述辅编码器;所述辅编码器,与所述自编码器相连,用于将来自所述自编码器的残差图像编码信息进行熵编码获得辅编码信息。
- 如权利要求1所述的装置,其特征在于,还包括:与所述上转换器相连的滤波增强器,用于从所述上转换器获取第一全分辨率图像,对所述第一全分辨率图像进行滤波增强,获得第二全分辨率图像;所述计算器还用于,根据所述原始分辨率图像和所述第二全分辨率图像计算得到残差图像。
- 如权利要求2所述的装置,其特征在于,所述滤波增强器具体用于,采用双边滤波算法对所述第一全分辨率图像进行滤波增强,获得所述第二全分辨率图像。
- 如权利要求3所述的装置,其特征在于,所述计算器根据所述原始分辨率图像与所述第二全分辨率图像计算得到残差图像,包括:将所述原始分辨率图像与所述第二全分辨率图像相同位置处的像素值相减,获得残差图像。
- 如权利要求1-4任一项所述的装置,其特征在于,所述自编码器根据自编码算法对所述残差图像进行编码获得残差图像编码信息,包括:将所述残差图像划分为多个图像块;根据预置的自编码器网络参数,对每个图像块分别进行编码,获得残差图像编码信息。
- 一种解码装置,其特征在于,包括:主解码器,上转换器,辅解码器,自解码器以及合成器,其中,所述主解码器,用于获取主编码信息,将所述主编码信息进行解码,获得低分辨率图像;所述上转换器,与所述主解码器相连,用于接收来自所述主解码器的所述低分辨率图像,将所述低分辨率图像进行上转换处理,获得第一全分辨率图像;所述辅解码器,用于获取辅编码信息,将所述辅编码信息进行熵解码,获得残差图像编码信息;所述自解码器,与所述辅解码器相连,用于对来自所述辅解码器的所述残差图像编码信息进行解码,获取残差图像;所述合成器,与所述上转换器和所述自解码器相连,用于将来自所述上转换器的所述第一全分辨率图像和来自所述自解码器的所述残差图像进行合成,获得第一原始分辨率图像。
- 如权利要求6所述的装置,其特征在于,还包括:滤波增强器,与所述上转换器相连,用于对所述第一全分辨率图像进行滤波增强处理,获得第二全分辨率图像;所述合成器还用于将所述残差图像与所述第二全分辨率图像合成,获得第二原始分辨率图像。
- 如权利要求6或7所述的装置,其特征在于,所述合成器具体用于:将所述第一或者第二全分辨率图像与所述残差图像的相同位置处的像素值相加,获得第一或第二原始分辨率图像。
- 一种编码方法,其特征在于,包括:将原始分辨率图像输入下采样器进行下采样处理,获取低分辨率图像;将所述低分辨率图像输入主编码器进行编码,获得主编码信息;将所述低分辨率图像输入上转换器进行上转换处理,获取第一全分辨率图像;根据所述原始分辨率图像与所述第一全分辨率图像计算得到残差图像;将所述残差图像输入自编码器进行编码,获取残差图像编码信息;将所述残差图像编码信息输入辅编码器进行熵编码获得辅编码信息。
- 如权利要求9所述的方法,其特征在于,还包括:将所述第一全分辨率图像输入滤波增强器进行滤波增强处理,获得第二全分辨率图像;根据所述原始分辨率图像和所述第二全分辨率图像计算得到残差图像。
- 如权利要求9或10所述的方法,其特征在于,所述将所述第一全分辨率图像输入滤波增强器进行滤波增强处理,获得第二全分辨率图像,包括:所述滤波增强器采用双边滤波算法对所述第一全分辨率图像进行滤波增强,获得所述第二全分辨率图像。
- 如权利要求10所述的方法,其特征在于,所述根据所述原始分辨率图像和所述第二全分辨率图像计算得到残差图像,包括:将所述原始分辨率图像与所述第二全分辨率图像相同位置处的像素值相减,获得残差图像。
- 如权利要求9-12任一项所述的方法,其特征在于,所述将所述残差图像输入自编码器进行编码,获取残差图像编码信息,包括:将所述残差图像划分为多个图像块;根据预置的自编码器网络参数,对每个图像块分别进行编码,获取残差图像编码信息。
- 一种解码方法,其特征在于,包括:获取主编码信息,将所述主编码信息进行解码,获得低分辨率图像;将所述低分辨率图像输入上转换器进行上转换处理,获得第一全分辨率图像;获取辅编码信息,将所述辅编码信息输入辅解码器进行熵解码,获得残差图像编码信 息;将所述残差图像编码信息输入自解码器进行解码,获取残差图像;将所述残差图像与所述第一全分辨率图像合成,获得第一原始分辨率图像。
- 如权利要求14所述的方法,其特征在于,所述方法还包括:将所述第一全分辨率图像输入滤波增强器进行滤波增强处理,获得第二全分辨率图像;将所述第二全分辨率图像与所述残差图像合成,获得第二原始分辨率图像。
- 如权利要求14或15所述的方法,其特征在于,所述将第一或第二全分辨率图像与所述残差图像合成,获得第一或第二原始分辨率图像,包括:将所述第一或者第二全分辨率图像与所述残差图像的相同位置处的像素值相加,获得第一或第二原始分辨率图像。
- 一种编解码***,其特征在于,包括:如权利要求1-5任一项所述的编码装置,和如权利要求6-8任一项所述的解码装置。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710861871.6A CN109547784A (zh) | 2017-09-21 | 2017-09-21 | 一种编码、解码方法及装置 |
CN201710861871.6 | 2017-09-21 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019056898A1 true WO2019056898A1 (zh) | 2019-03-28 |
Family
ID=65811031
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2018/101040 WO2019056898A1 (zh) | 2017-09-21 | 2018-08-17 | 一种编码、解码方法及装置 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN109547784A (zh) |
WO (1) | WO2019056898A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111343458A (zh) * | 2020-01-15 | 2020-06-26 | 浙江工业大学 | 一种基于重建残差的稀疏灰度图像编解码方法及*** |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110971903A (zh) * | 2019-10-17 | 2020-04-07 | 西安万像电子科技有限公司 | 编码方法、设备及*** |
CN110677681A (zh) * | 2019-11-01 | 2020-01-10 | 合肥图鸭信息科技有限公司 | 一种视频编解码方法、装置及终端设备 |
KR20210067783A (ko) * | 2019-11-29 | 2021-06-08 | 삼성전자주식회사 | 전자 장치, 그 제어 방법 및 시스템 |
CN111246206B (zh) * | 2020-01-14 | 2021-09-21 | 山东浪潮科学研究院有限公司 | 一种基于自编码器的光流信息压缩方法及装置 |
CN113574886B (zh) * | 2020-02-28 | 2024-06-18 | 华为技术有限公司 | 图像传输方法及装置 |
CN112055223A (zh) * | 2020-08-21 | 2020-12-08 | 浙江大华技术股份有限公司 | 一种图像编解码方法及编解码器 |
CN112053285B (zh) * | 2020-09-01 | 2024-05-31 | 平安科技(深圳)有限公司 | 图像处理方法、装置、计算机设备及存储介质 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2003036980A1 (en) * | 2001-10-26 | 2003-05-01 | Koninklijke Philips Electronics N.V. | Spatial scalable compression scheme using spatial sharpness enhancement techniques |
CN1939067A (zh) * | 2004-04-02 | 2007-03-28 | 汤姆森许可贸易公司 | 用于复杂度可伸缩视频编码器的方法和设备 |
CN103607591A (zh) * | 2013-10-28 | 2014-02-26 | 四川大学 | 结合超分辨率重建的图像压缩方法 |
CN105959705A (zh) * | 2016-05-10 | 2016-09-21 | 武汉大学 | 一种面向可穿戴设备的视频直播方法 |
CN107018422A (zh) * | 2017-04-27 | 2017-08-04 | 四川大学 | 基于深度卷积神经网络的静止图像压缩方法 |
CN107181949A (zh) * | 2017-06-23 | 2017-09-19 | 四川大学 | 一种结合超分辨率和残差编码技术的图像压缩框架 |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR100703740B1 (ko) * | 2004-10-21 | 2007-04-05 | 삼성전자주식회사 | 다 계층 기반의 모션 벡터를 효율적으로 부호화하는 방법및 장치 |
CN105163121B (zh) * | 2015-08-24 | 2018-04-17 | 西安电子科技大学 | 基于深度自编码网络的大压缩比卫星遥感图像压缩方法 |
CN106254879B (zh) * | 2016-08-31 | 2019-11-08 | 广东精点数据科技股份有限公司 | 一种应用自编码神经网络的有损图像压缩方法 |
-
2017
- 2017-09-21 CN CN201710861871.6A patent/CN109547784A/zh active Pending
-
2018
- 2018-08-17 WO PCT/CN2018/101040 patent/WO2019056898A1/zh active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2003036980A1 (en) * | 2001-10-26 | 2003-05-01 | Koninklijke Philips Electronics N.V. | Spatial scalable compression scheme using spatial sharpness enhancement techniques |
CN1939067A (zh) * | 2004-04-02 | 2007-03-28 | 汤姆森许可贸易公司 | 用于复杂度可伸缩视频编码器的方法和设备 |
CN103607591A (zh) * | 2013-10-28 | 2014-02-26 | 四川大学 | 结合超分辨率重建的图像压缩方法 |
CN105959705A (zh) * | 2016-05-10 | 2016-09-21 | 武汉大学 | 一种面向可穿戴设备的视频直播方法 |
CN107018422A (zh) * | 2017-04-27 | 2017-08-04 | 四川大学 | 基于深度卷积神经网络的静止图像压缩方法 |
CN107181949A (zh) * | 2017-06-23 | 2017-09-19 | 四川大学 | 一种结合超分辨率和残差编码技术的图像压缩框架 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111343458A (zh) * | 2020-01-15 | 2020-06-26 | 浙江工业大学 | 一种基于重建残差的稀疏灰度图像编解码方法及*** |
CN111343458B (zh) * | 2020-01-15 | 2022-03-01 | 浙江工业大学 | 一种基于重建残差的稀疏灰度图像编解码方法及*** |
Also Published As
Publication number | Publication date |
---|---|
CN109547784A (zh) | 2019-03-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2019056898A1 (zh) | 一种编码、解码方法及装置 | |
US20200145692A1 (en) | Video processing method and apparatus | |
CN108737823B (zh) | 基于超分辨技术的图像编码方法和装置、解码方法和装置 | |
CN105474642A (zh) | 使用频域差对图像集合进行重新编码 | |
WO2023000179A1 (zh) | 视频超分辨网络及视频超分辨、编解码处理方法、装置 | |
CN111510739B (zh) | 一种视频传输方法及装置 | |
CN115606179A (zh) | 用于使用学习的下采样特征进行图像和视频编码的基于学习的下采样的cnn滤波器 | |
WO2018190199A1 (ja) | 画像処理装置、画像処理システム、情報処理システム及び画像処理方法 | |
CN111800629A (zh) | 视频解码方法、编码方法以及视频解码器和编码器 | |
WO2023279961A1 (zh) | 视频图像的编解码方法及装置 | |
Agustsson et al. | Extreme learned image compression with gans | |
KR20200050284A (ko) | 영상 적응적 양자화 테이블을 이용한 영상의 부호화 장치 및 방법 | |
CN115552905A (zh) | 用于图像和视频编码的基于全局跳过连接的cnn滤波器 | |
WO2022266955A1 (zh) | 图像解码及处理方法、装置及设备 | |
CN113747242A (zh) | 图像处理方法、装置、电子设备及存储介质 | |
WO2023193629A1 (zh) | 区域增强层的编解码方法和装置 | |
WO2021168827A1 (zh) | 图像传输方法及装置 | |
JP7125559B2 (ja) | ビットレート削減のためのビデオストリーム適応フィルタリング | |
WO2023010981A1 (zh) | 编解码方法及装置 | |
Zhang et al. | Dual-layer image compression via adaptive downsampling and spatially varying upconversion | |
JP2024511587A (ja) | ニューラルネットワークベースのピクチャ処理における補助情報の独立した配置 | |
Gupta et al. | Lossless compression based Kmp technique | |
WO2023279968A1 (zh) | 视频图像的编解码方法及装置 | |
CN111246205B (zh) | 基于方向双四元数滤波器组的图像压缩方法 | |
KR102604657B1 (ko) | 영상 압축 성능 개선 방법 및 장치 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18859839 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 18859839 Country of ref document: EP Kind code of ref document: A1 |