WO2020192034A1 - 滤波方法及装置、计算机存储介质 - Google Patents

滤波方法及装置、计算机存储介质 Download PDF

Info

Publication number
WO2020192034A1
WO2020192034A1 PCT/CN2019/105799 CN2019105799W WO2020192034A1 WO 2020192034 A1 WO2020192034 A1 WO 2020192034A1 CN 2019105799 W CN2019105799 W CN 2019105799W WO 2020192034 A1 WO2020192034 A1 WO 2020192034A1
Authority
WO
WIPO (PCT)
Prior art keywords
filter
filtering
neural network
parameters
parameter
Prior art date
Application number
PCT/CN2019/105799
Other languages
English (en)
French (fr)
Inventor
马彦卓
万帅
霍俊彦
张伟
王铭泽
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Priority to KR1020217032829A priority Critical patent/KR20210134397A/ko
Priority to EP19921828.0A priority patent/EP3941066A4/en
Priority to CN201980094265.3A priority patent/CN113574897A/zh
Priority to JP2021555871A priority patent/JP2022525235A/ja
Publication of WO2020192034A1 publication Critical patent/WO2020192034A1/zh
Priority to US17/475,237 priority patent/US11985313B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117—Filters, e.g. for pre-processing or post-processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/184—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
    • H04N19/82—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation involving filtering within a prediction loop
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20192—Edge enhancement; Edge preservation

Definitions

  • the embodiments of the present application relate to the technical field of video coding and decoding, in particular to a filtering method and device, and a computer storage medium.
  • image/video filtering is realized by using filters.
  • the pre-processing filter is used to pre-process the original image to reduce the video resolution, because the video resolution that needs to be encoded is higher than The resolution of the original video is low, which can use fewer bits to represent, thereby improving the overall coding efficiency;
  • the post-processing filter processes the filtered video in the loop to output the video to improve the video resolution;
  • loop The path filter is used to improve the subjective and objective quality of the reconstructed image.
  • pre-processing filters For pre-processing filters, loop filters and post-processing filters, they are all implemented using convolutional neural networks, and filters based on convolutional neural networks can be divided into two categories, one is offline training, the other The species is trained online.
  • all weight parameters of the neural network can be set at the encoding end and the decoding end at the same time after the training is completed, but because the weight coefficients are fixed, the filter performance may be in the case of some video content Down degradation.
  • the weight parameters of the network need to be retrained and updated frequently, so the weight coefficients need to be transmitted in the code stream, which is computationally expensive and complex, and is suitable for processing a narrow range of video content. Relatively limited.
  • the embodiments of the present application provide a filtering method and device, and a computer storage medium, which can improve the filtering performance of the filtering device and have a wide application range.
  • the embodiment of the present application provides a filtering method, including:
  • the pixel information to be filtered and the side information are input into a filter based on a neural network to output filtered pixels, where the filter is obtained by an online filtering part combined with an offline filtering part.
  • the embodiment of the present application also provides a filtering method for encoding video, including:
  • the first filter parameter is encoded and written into the video code stream.
  • the embodiment of the present application also provides a filtering method for decoding a video bitstream, including:
  • the adaptive filter is used to filter the input pixels to obtain filtered pixels.
  • An embodiment of the present application provides a filtering device, including:
  • the first acquiring part is configured to acquire pixel information to be filtered
  • the first determining part is configured to determine side information
  • the first filtering part is configured to input the pixel information to be filtered and the side information into a filter based on a neural network to output filtered pixels, wherein the filter is combined with an online filtering part and an offline filtering part get.
  • An embodiment of the present application also provides a filtering device for encoding video, including:
  • the second determining part is configured to determine filter parameters of the adaptive filter
  • the second filtering part is configured to use the adaptive filter to filter the input pixels according to the filtering parameters and side information to obtain filtered pixels;
  • the second determining part is further configured to determine a first filter parameter, where the first filter parameter is a part of the filter parameter in the filter parameter that needs to be encoded;
  • the second writing part is configured to encode the first filter parameter and write the video code stream.
  • An embodiment of the present application also provides a filtering device for decoding video, including:
  • the third determining part is configured to analyze the video code stream and determine the first filter parameter of the adaptive filter, wherein the first filter parameter is a part of the filter parameters of all the filter parameters of the adaptive filter; and Determine all filter parameters of the adaptive filter according to the first filter parameter;
  • the third filtering part is configured to use the adaptive filter to filter the input pixels according to all the filtering parameters and side information to obtain filtered pixels.
  • An embodiment of the present application also provides a filtering device, including:
  • Memory used to store executable instructions
  • the processor is configured to implement the filtering method provided in the embodiment of the present application when executing the executable instructions stored in the memory.
  • the embodiment of the present application provides a computer storage medium that stores executable instructions for causing a processor to execute, to implement the filtering method provided in the embodiment of the present application.
  • the filtering device aims at the pixel information to be filtered; determines the side information of the video frame to be filtered; inputs the side information and the pixels to be filtered into the filter for filtering, and outputs the filtered pixels. Since the filter is obtained by the online filtering part and the offline filtering part, When filtering, you can use the offline filtering part, which is suitable for filtering a wide range of videos. At the same time, it can also ensure the update of the parameters of the online filtering part to avoid the degradation of filtering performance, that is, to improve the filtering performance of the filtering device, and it is applicable wide range.
  • FIG. 1 is a schematic structural diagram of a coding block diagram provided by an embodiment of the application
  • FIG. 2 is a schematic structural diagram of a decoding block diagram provided by an embodiment of the application.
  • FIG. 3 is an optional flowchart of a filtering method provided by an embodiment of the application.
  • FIG. 4 is a schematic structural diagram of a block division matrix provided by an embodiment of the application.
  • FIG. 5 is a schematic diagram 1 of a connection mode of a filter provided by an embodiment of the application.
  • FIG. 6 is a second schematic diagram of a connection mode of a filter provided by an embodiment of the application.
  • FIG. 7 is a third schematic diagram of a connection mode of a filter provided by an embodiment of the application.
  • FIG. 8 is a fourth schematic diagram of a connection mode of a filter provided by an embodiment of the application.
  • FIG. 9 is a fifth schematic diagram of a connection mode of a filter provided by an embodiment of the application.
  • FIG. 10 is another optional flowchart of a filtering method provided by an embodiment of this application.
  • FIG. 11 is a schematic diagram of an optional structure of a filtering device provided by an embodiment of the application.
  • FIG. 12 is a schematic diagram of another optional structure of a filtering device provided by an embodiment of the application.
  • the video to be coded includes the original video frame, and the original video frame includes the original image.
  • a variety of processing is performed on the original image, such as prediction, transformation, quantization, reconstruction, and filtering. During these processes , The processed video image may have shifted in pixel value relative to the original image, causing visual impairment or artifacts.
  • CU block coding unit
  • QP quantization Parameter
  • prediction methods different reference image frames, etc.
  • the size of the error introduced by each coding block and its distribution characteristics are independent of each other, and the discontinuity of the boundaries of adjacent coding blocks produces block effects.
  • These distortions affect the subjective and objective quality of the reconstructed image block. If the reconstructed image block is used as a reference image for subsequent coded pixels, it will even affect the prediction accuracy of the subsequent codec, and further affect the size of the bits in the video bitstream. Therefore, in the video codec system, pre-processing filters, post-processing filters, and loop filters (In-Loop Filter) are often added to improve the subjective and objective quality of reconstructed images.
  • pre-processing filters, post-processing filters, and loop filters In-Loop Filter
  • FIG. 1 is a schematic structural diagram of a traditional coding block diagram.
  • the traditional coding block diagram 10 may include a transform and quantization unit 101, an inverse transform and inverse quantization unit 102, a prediction unit 103, an in-loop filtering unit 104, and an entropy coding unit 105.
  • Components such as a pre-processing filtering unit 106 and a post-processing filtering unit 107; wherein the prediction unit 103 further includes an intra prediction unit 1031 and an inter prediction unit 1032.
  • the coding tree unit (CTU, Coding Tree Unit) can be obtained through preliminary division, and the content adaptive division of a CTU can be continued to obtain the CU.
  • CTU Coding Tree Unit
  • a CU generally contains one or more coding blocks (CB, Coding Block).
  • CB Coding Block
  • the unit 104 removes the block artifacts, and then adds the reconstructed residual block to the decoded image buffer unit to generate a reconstructed reference image; the entropy encoding unit
  • the in-loop filtering unit 104 is a loop filter, also called an in-loop filter, which may include a de-blocking filter (DBF, De-Blocking Filter), sample point adaptive Compensation (SAO, Sample Adaptive Offset) filter and adaptive loop filter (ALF, Adaptive Loop Filter), etc.
  • DPF de-blocking filter
  • SAO Sample Adaptive Offset
  • ALF adaptive Loop Filter
  • the deblocking filter is used to implement deblocking filtering.
  • next-generation video coding standard H.266/Versatile Video Coding for all coding block boundaries in the original image, first The coding parameters determine the boundary strength, and determine whether to perform deblocking filtering decision based on the calculated block boundary texture value, and finally correct the pixel information on both sides of the coding block boundary according to the boundary strength and filtering decision.
  • SAO technology is also introduced, that is, the sample adaptive compensation filter; further, starting from the pixel domain, add pixels at the peak Add a positive value to the pixel at the negative value and valley for compensation.
  • VVC VVC
  • the deblocking filter and sample adaptive compensation filter are executed, it is necessary to further use the adaptive loop filter for filtering; for the adaptive loop filter, it is based on the pixel value of the original image and the distorted image Calculate the optimal filter in the sense of mean square.
  • the preprocessing filtering unit 106 is used to receive the input original video frame, perform preprocessing and filtering on the original video frame to reduce the resolution of the video
  • the post-processing filtering unit 107 is used to receive
  • the video frame filtered in the loop is post-processed and filtered to the video frame filtered in the loop to improve the resolution of the video. In this way, less bits can be used in the video encoding and decoding process to obtain the reconstructed video frame, which can Improve the overall coding and decoding efficiency.
  • the input of the neural network adopted by both the pre-processing filter and the post-processing filter is single input or multiple input, that is, a single image component or multiple image components are input to realize image reconstruction.
  • Figure 2 is a schematic structural diagram of a traditional decoding block diagram.
  • the traditional decoding block diagram 20 may include an entropy coding unit 201, an inverse quantization and inverse transformation unit 202, a prediction unit 203, and an in-loop
  • the filtering unit 204 and the post-processing filtering unit 205 are components; wherein, the prediction unit 203 further includes an intra prediction unit 2031 and an inter prediction unit 2032.
  • the video decoding process is the opposite or inverse process of the video encoding process, in which the post-processing filtered image obtained in the video decoding process is determined as the reconstructed video frame, as can be seen from Figure 2.
  • the decoding process does not involve the pre-processing filtering unit in the encoding process, only the post-processing filtering unit and the in-loop filtering unit.
  • the pre-processing filtering unit, the post-processing filtering unit, and the in-loop filtering unit can all be said to be one type of filter, and the filter in the embodiment of the present application may be a convolutional neural network (Convolutional Neural Networks). , CNN) filter, may also be other filters established by deep learning, and the embodiment of the present application does not specifically limit it.
  • CNN convolutional Neural Networks
  • the convolutional neural network filter can not only replace the preprocessing filter unit, postprocessing filter unit, and in-loop filter unit in Figure 1, but can also partially replace the preprocessing filter in Figure 1. Any one or two of the unit, post-processing filter unit and in-loop filter unit can even be used in combination with any one or more of the pre-processing filter unit, post-processing filter unit and in-loop filter unit in Figure 1 . It should also be noted that, for each of the components shown in FIG.
  • transform and quantization unit 101 such as transform and quantization unit 101, inverse transform and inverse quantization unit 102, prediction unit 103, in-loop filtering unit 104, entropy encoding unit 105, preprocessing filter
  • the unit 106 and the post-processing filtering unit 107, these components may be virtual modules or hardware modules.
  • these units do not constitute a limitation on the coding block diagram, and the coding block diagram may include more or less components than those shown in the figure, or a combination of certain components, or different component arrangements.
  • the convolutional neural network filter when used as the in-loop filtering unit, the convolutional neural network filter can be directly deployed on the encoding end and the decoding end after performing filter network training. Moreover, the convolutional neural network filter can also process side information and other auxiliary information with the input image to be filtered; in this way, it not only makes full use of the relationship between the image side information, and further improves the subjectivity and objectiveness of the video reconstruction image in the encoding and decoding process. quality.
  • the convolutional neural network filter is used as the post-processing filter unit, the convolutional neural network filter can be directly deployed on the decoding side after the filter network training.
  • the convolutional neural network filter is used as the preprocessing filter unit, the convolutional neural network The filter can be directly deployed on the encoding end after filtering network training.
  • the filtering method in the embodiments of the present application can be applied to an encoding system and/or a decoding system when the filter types are different.
  • the in-loop filter of the embodiment of the present application must be deployed in the coding system and the decoding system simultaneously.
  • filters based on a convolutional neural network can be divided into two categories, one is offline training, and the other is online training.
  • offline-trained filters all weight parameters of the neural network can be set at the encoding end and the decoding end at the same time after the training is completed, but because the weight coefficients are fixed, the filter performance may be in the case of some video content Down degradation.
  • online training filters the weight parameters of the network need to be retrained and updated frequently, so the weight coefficients need to be transmitted in the code stream, which is computationally expensive and complex, and is suitable for processing a narrow range of video content. Relatively limited.
  • an embodiment of the present application provides a filtering method, which is applied to a filtering device.
  • the filtering device can be set in the preprocessing filter and the in-loop filter in the encoder, or in the decoder.
  • the in-loop filter and the post-processing filter can also be used in other filters used in the prediction process, which is not specifically limited in the embodiment of the present application.
  • the neural network-based filter is suitable for post-processing filtering, in-loop filtering, pre-processing filtering and prediction processes.
  • the neural network-based filter when the neural network-based filter is suitable for post-processing filtering, it is set at the decoding end; when the neural network-based filter is suitable for in-loop processing filtering, it is set at the decoding end and the encoding end; based on neural network
  • the filter is suitable for preprocessing filtering and is set at the encoding end.
  • FIG. 3 is a schematic flowchart of an optional filtering method provided by an embodiment of this application.
  • the filtering method may include:
  • S101 Acquire pixel information to be filtered.
  • S103 Input the pixel information to be filtered and the side information into the filter based on the neural network to output the filtered pixels, where the filter is obtained by combining the online filtering part with the offline filtering part.
  • the video frame to be filtered is generated during the video encoding process of the original image in the video to be encoded.
  • the video to be encoded includes the original image frame, and the original image frame includes the original image.
  • the video frame to be filtered includes multiple frames of images, and the filtering device filters the pixel information of each frame of image to be filtered when filtering.
  • each frame of the video frame to be filtered has corresponding side information, that is, the side information corresponding to the pixel information of each frame of image to be filtered.
  • side information represents the boundary information of each frame of image.
  • the original image can be divided into CTUs or CTUs into CUs; that is, the side information in the embodiments of this application can refer to CTU division information or CUs. Divide information; in this way, the filtering method in the embodiment of the present application can be applied not only to CU-level filtering, but also to CTU-level filtering, which is not specifically limited in the embodiment of the present application.
  • the original image is divided into coding unit CU to obtain CU division information.
  • the CU partition information fill the first value at each pixel position corresponding to the CU boundary and fill the second value at other pixel positions to obtain the first matrix corresponding to the CU partition information; where the first value is different from the second value;
  • the first matrix here is the side information of each frame of image.
  • the first value can be a preset value, letter, etc.
  • the second value can also be a preset value, letter, etc., as long as the first value is different from the second value, for example ,
  • the first value can be set to 2, and the second value can be set to 1, which is not limited in the embodiment of the application.
  • the filtering device filters the to-be-filtered pixel information of the to-be-filtered video frame through the filter and the side information to obtain the filtered pixels, which can be understood as the final filtered image, where the filter is combined with the online filtering part and the offline filtering part get.
  • the filtering device can use CU information as auxiliary information to assist the filtering process of the video frame to be filtered, that is, in the process of video encoding the original image in the video to be encoded, the CU division can be fully utilized Information, fused with the video frame to be filtered and then guided filtering.
  • the CU division information is converted into a coding unit map (Coding Unit Map, CUmap), and represented by a two-dimensional matrix, that is, the CUmap matrix, that is, the first matrix in the embodiment of the present application; That is to say, taking the original image as an example, it can be divided into multiple CUs; each pixel position corresponding to the boundary of each CU is filled with the first value, and other pixel positions are filled with the second value.
  • a first matrix reflecting CU partition information can be constructed.
  • FIG. 4 shows a schematic structural diagram of a block division matrix provided by an embodiment of the present application.
  • the CTU can be divided into 9 CUs; suppose the first value is set to 2 and the second value is set to 1; in this way, each pixel corresponding to the boundary of each CU The dot position is filled with 2 and the other pixel positions are filled with 1. That is to say, the pixel position filled with 2 represents the boundary of the CU, so that the CU division information can be determined, that is, the video frame to be filtered Side information of an image.
  • the CU division information may also be corresponding based on the image component level, which is not limited in the embodiment of the present application.
  • the filter is formed by cascading the online filtering model and the offline filtering model; or, the filter is formed by cascading the online filtering model and the offline filtering model; wherein, the offline filtering model has some online training parameters; Alternatively, the filter is formed by an offline filtering model, where there are some online training parameters in the offline filtering model.
  • the filtering device filters the to-be-filtered pixel information of the video frame to be filtered through the filter and combined with the side information.
  • the realization process of obtaining the filtered pixels can be: the filtering device adopts the offline filtering model and combines the side information.
  • the information is filtered on the video frame to be filtered to obtain a filtered image, that is, the pixel information to be filtered and the side information are input to an offline filtering model based on a neural network to output the filtered pixels.
  • the filter is formed by cascading the online filtering model and the offline filtering model; or, when the filter is formed by cascading the online filtering model and the offline filtering model, the filtering device passes through the filter and combines the side information to filter the pixel information of the video frame to be filtered Perform filtering to obtain a filtered image, that is, the realization process of filtered pixels: the filtering device adopts an offline filtering model and combines the side information to filter the video frame to be filtered to obtain an intermediate filtered image; adopts an online filtering model and combines the side information to filter the intermediate image Perform filtering to obtain a filtered image.
  • the filtering device adopts an online filtering model and combines the side information to filter the to-be-filtered video frame to obtain an intermediate filtered image; adopts an offline filtering model and combines the side information to filter the intermediate filtered image to obtain a filtered image. That is, input the pixel information and side information to be filtered into the neural network-based offline filtering model to output the intermediate filtered pixels; input the intermediate filtered pixels and side information into the neural network-based online filtering model to output the filtered pixels; or , Input the pixel information and side information to be filtered into the neural network-based online filtering model to output the intermediate filtered pixels; input the intermediate filtered pixels and side information into the neural network-based offline filtering model to output the filtered pixels.
  • the filter may be formed by combining an online convolutional neural network and an offline convolutional neural network.
  • an offline model is cascaded with an online model, or an offline model, but part of it can be trained online, or an offline model (some of which can be trained online) is cascaded with an online model, this embodiment of the application No restrictions.
  • the offline filtering model is an offline training filter; the offline filtering model also includes offline training parameters.
  • the online filtering model is an online training filter; wherein, the online filtering model includes online training parameters.
  • the offline filtering model refers to a filter trained offline.
  • the convolutional neural network needs to be trained with a large number of pictures and tested on pictures divided from the training set. If the performance is very effective, it can be applied to filters such as in-loop/post-processing filters in video codec technology. All weight parameters (ie, parameters) in the convolutional neural network can be set on the encoding end and the decoding end at the same time after the training is completed.
  • Online filtering models refer to filters trained online.
  • Convolutional neural networks are often trained based on random access segments (some video frames that have just been coded in the video sequence) to obtain updated parameters and use updated online filtering with updated parameters.
  • the model processes subsequent frames of the same video sequence. This type of convolutional neural network is small in scale and can only be applied to the processing of a very narrow range of video sequences.
  • the filter is formed by cascading online filtering model 1 (On-line trained NN) and offline filtering model 2 (Off-line trained NN), where the cascading sequence shown in FIG. 5 or as shown in FIG. 6
  • the cascade sequence shown can be used.
  • the filtering device inputs the pixel information 4 to be filtered and its corresponding side information 3 (Side information) into the offline filtering model 2, and after offline filtering, the intermediate filtered pixel 5 is obtained; the intermediate filtered pixel 5
  • the sum-side information 3 is then input to the online filtering model 1, for online filtering, and filtered output (Filtered output) filtered pixel 6.
  • filtered output Filtered output
  • the filtering device inputs the pixel information 4 to be filtered and its corresponding side information 3 into the online filtering model 1, and after online filtering, the intermediate filtered pixel 5 is obtained; the intermediate filtered pixel 5 and side information 3 Then input to the offline filtering model 2, perform offline filtering, and output the filtered pixel 6 after filtering.
  • the filter is formed by cascading an online filtering model and an offline filtering model; wherein, there are some online training parameters in the offline filtering model.
  • the filtering device inputs the pixel information 4 to be filtered and its corresponding side information 3 into the offline filtering model 2, and after offline-online hybrid filtering, the intermediate filtered pixel 5 is obtained; the intermediate filtered pixel 5 and the side
  • the information 3 is then input to the online filtering model 1 for online filtering, and the filtered output pixel 6 is filtered.
  • the filtering device inputs the pixel information 4 to be filtered and its corresponding side information 3 into the online filtering model 1, and after online filtering, the intermediate filtered pixel 5 is obtained; the intermediate filtered pixel 5 and side information 3 Then input to the offline filtering model 2, perform offline-online hybrid filtering, and output filtered pixel 6.
  • the filter is formed by an offline filtering model, where there are some online training parameters in the offline filtering model.
  • the filtering device inputs the pixel information 1 to be filtered and its corresponding side information 2 into the offline filtering model 3, after offline-online hybrid filtering, the filtered pixel 4 is output by filtering.
  • the advantage of the offline filtering model is that it has outstanding performance and does not require additional transmission weight coefficients.
  • the disadvantage is that the adaptive ability of the sequence is lacking; the advantage of the online filtering model is that it has the adaptive ability to the sequence, but it requires transmission weights. coefficient.
  • the combination of the offline filtering part and the online filtering part can not only use the performance of the offline filtering model, but also use the online filtering model to improve the objective quality of the video again. That is to say, the filtering device of the embodiment of the present application can obtain a trade-off between generalization ability and sequence adaptability in filtering processing of different videos, and can bring better coding performance under the condition of low complexity.
  • the filtering device inputs the pixel information to be filtered and the side information into the filter based on the neural network to output the filtered pixels, and then performs the online filter based on the filtered pixels.
  • Training to obtain online filtering parameters after training the online part of the filter based on the online filtering parameters, input the subsequent pixel information and side information to be filtered into the updated filter for filtering to obtain the subsequent filtered pixels; write the online filtering parameters into the video Code stream.
  • the neural network is often trained based on random access segments (the pixel information of some video frames that have just been encoded in the video sequence), and it is immediately used for subsequent follow-ups of the same sequence after training.
  • Frame that is, the pixel information of subsequent frames.
  • Such neural networks are small in scale and can only be applied to a very narrow range of video content.
  • the weight parameters obtained by training (that is, online filtering parameters) need to be retrained and updated frequently. Therefore, the weight coefficients need to be transmitted in the code stream.
  • the following takes the pixel information to be filtered of the video frame to be filtered as an example to describe the online filtering parameters.
  • an optional implementation process of the implementation of S103 may include: S1031-S1036. as follows:
  • the filters both have an offline filtering part and an online filtering part
  • the difference is that some online filtering parts are implemented directly through the online filtering model, and some are implemented through the offline filtering model.
  • Part of the parameters of is obtained through online training, and the online filtering part is realized. The latter is realized by combining the above two online filtering.
  • the online filtering part is trained based on random access segments (some video frames that have just been filtered in the video sequence) to obtain updated parameters, and the updated online filtering part with updated parameters is used to process the same video sequence Of subsequent frames.
  • the video frame to be filtered may contain N frames of images, N is the total number of video frames to be filtered, and N is a positive integer greater than 1, and the value of N is determined by the number of frames to be filtered.
  • N is the total number of video frames to be filtered
  • N is a positive integer greater than 1
  • the value of N is determined by the number of frames to be filtered.
  • the filtering device when the filtering device performs filtering on the i-th frame image in the video frame to be filtered, the i-th frame image and the side information of the i-th frame image are input into the filter, and the filter is processed online and offline. After filtering, output the filtered image of the i-th frame, where i starts from 1, that is, the filtering device starts to filter the image of the first frame. After filtering the image of the i-th frame, the filtering device continues to filter the i+1 The frame image is filtered until the filtering process of the i+H frame image is completed, and the i-th frame filtered image to the i+H frame filtered image are obtained, where H is greater than 1 and less than Ni.
  • the filtering device adopts a filter including an online filtering part and an offline filtering part to filter the to-be-filtered video frame, after filtering a sequence of frame images, the filtered i+H frame image and i +H frame filtered image (i-th frame filtered image to i+H frame filtered image) is used as the training sample, and the online part of the filter is trained again, and the training result is the closest to the output result of the i+H frame filtered image
  • the filtering device can use the updated online filtering part to combine the existing offline filtering part with the i+H+th For the side information of 1 frame of image, filter the filtered image of the i+H+1 frame to obtain the filtered image of the i+H+1 frame, and continue to filter the image of the i+H+2 frame until the Nth frame is filtered.
  • the filtering process of the frame image is completed,
  • the filter device can start to update the online filtering part of the filter after filtering the Hi frame, and the specific value of H can be based on actual needs and specific design It is decided that the embodiments of this application are not limited.
  • the filtering process for the i+H+1 frame image to the Nth frame image can still be performed after the Hi frame image has been processed, but the Nth frame image has not been processed yet again.
  • the update of the online filtering part uses the updated online filtering part to continue filtering of subsequent frames until the filtering of the Nth frame of image is completed.
  • the filtering device may start to update the online filtering parameters of the online filtering part after filtering a fixed number of frames of the image, or the number of filtered frames may be different every time the online filtering part is updated.
  • the embodiments of this application are not limited.
  • the condition for stopping the update is that the filtering of the last frame of image is completed.
  • the online filtering part may be an online filtering model, or a part of the convolutional neural network level in the offline model can update parameters online to obtain parameter update information (that is, online filter parameters), and It may be a combination of the previous two online filtering parts, which is not limited in the embodiment of the present application.
  • the filter is formed by cascading online filtering model 1 (On-line trained NN) and offline filtering model 2 (Off-line trained NN), where the cascading sequence shown in FIG. 5 or as shown in FIG. 6
  • the cascade sequence shown can be used.
  • the filtering device inputs the pixel information 4 to be filtered and its corresponding side information 3 (Side information) into the offline filtering model 2 and the online filtering model 1.
  • Side information Side information
  • online filtering model 1 is trained to obtain parameter update information (that is, online filtering parameters); in order to train the online part of the filter based on the online filtering parameters, it will be later
  • parameter update information that is, online filtering parameters
  • the filtered pixel information and side information are input to the updated filter for filtering to obtain subsequent filtered pixels.
  • the filter is formed by cascading an online filtering model and an offline filtering model; wherein, there are some online training parameters in the offline filtering model.
  • an online filtering model and an offline filtering model
  • there are some online training parameters in the offline filtering model either the cascade sequence shown in FIG. 7 or the cascade sequence shown in FIG. 8 may be used.
  • the filtering device inputs pixel information 4 to be filtered and its corresponding side information 3 (Side information) into offline filtering model 2 (offline-online hybrid filtering) and online filtering model 1 during the filtering process .
  • Side information pixel information 4 to be filtered and its corresponding side information 3 (Side information) into offline filtering model 2 (offline-online hybrid filtering) and online filtering model 1 during the filtering process .
  • the subsequent pixel information and side information to be filtered are input into the updated filter for filtering to obtain the subsequent filtered pixels.
  • the filter is formed by an offline filtering model, where there are some online training parameters in the offline filtering model.
  • the filtering device inputs the pixel information 1 to be filtered and its corresponding side information 2 into the offline filtering model 3.
  • the filtered pixel 4 is filtered and output, and the filtering of the previous frame can also be used.
  • the online part of the offline filter model 3 is trained online, that is, the online part of the offline filter model 3 is trained to obtain parameter update information (that is, online filter parameters); in order to train the online part of the filter based on the online filter parameters, Subsequent to-be-filtered pixel information and side information are input to the updated filter for filtering to obtain subsequent filtered pixels.
  • the filtering device is aimed at the video frame to be filtered; determines the side information of the video frame to be filtered; inputs the side information and the video frame to be filtered into the filter for filtering, and outputs the filtered image, because the filter is combined offline by the online filtering part
  • the filtering part is obtained, so that the offline filtering part can be used in filtering, which is suitable for filtering a wide range of videos, and at the same time, it can also ensure the update of the online filtering parameters of the online filtering part, and use the updated online filtering part to be filtered
  • the filtering of subsequent frames of the video frame avoids the degradation of filtering performance, improves the filtering performance of the filtering device, and has a wide application range.
  • the embodiment of the application also provides a filtering method for encoding video and applying it to the encoding end, including: determining filter parameters of the adaptive filter; according to the filter parameters and side information, using the adaptive filter to input The pixels are filtered to obtain filtered pixels; the first filter parameter is determined, where the first filter parameter is a part of the filter parameter that needs to be encoded; the first filter parameter is encoded and written into the video code stream.
  • the first filter parameter is an online filter parameter, that is, parameter update information.
  • the adaptive filter is a neural network filter.
  • the adaptive filter is a cascade filter of the first neural network filter and the second neural network filter.
  • determining the filter parameters of the adaptive filter includes: using offline training to determine the second filter parameters of the neural network filter, where the second filter parameters are all parameters of the neural network filter; use Online training determines the third filter parameter of the neural network filter; among them, the second filter parameter is part of the neural network filter parameters; the third filter parameter is used to update the corresponding filter parameter in the second filter parameter, and the updated second filter parameter
  • the filter parameters are used as all the filter parameters of the neural network filter.
  • determining the first filter parameter includes: using the third filter parameter as the first filter parameter.
  • determining the filter parameters of the adaptive filter includes: using offline training to determine all filter parameters of the first neural network filter; using online training to determine the fourth filter parameter of the second neural network filter , Where the fourth filter parameter is all the parameters of the second neural network filter.
  • determining the first filter parameter includes: using the fourth filter parameter as the first filter parameter.
  • determining the filter parameters of the adaptive filter includes: using offline training to determine the fifth filter parameter of the first neural network filter, where the fifth filter parameter is that of the first neural network filter. All parameters; use online training to determine the sixth filter parameter of the neural network filter, where the sixth filter parameter is part of the parameter of the first neural network filter; use the sixth filter parameter to update the corresponding filter parameter in the fifth filter parameter, Use the updated fifth filter parameter as the filter parameter of the first neural network filter; in the process of encoding the video or image, use online training to determine the seventh filter parameter of the second neural network filter, where the seventh filter The parameters are all the parameters of the second neural network filter.
  • determining the first filter parameter includes: using the sixth filter parameter and the seventh filter parameter as the first filter parameter.
  • offline training is a process of training the neural network filter using one or more images before starting to encode the video or image
  • Online training is a process of training the neural network filter using one or more images in the video sequence to be encoded in the process of encoding a video or image.
  • the adaptive filter is a pre-processing filter used in encoding video, or an in-loop filter.
  • the method for updating the parameters of the wave device can adopt random access segment, or sequence adaptive, which is not limited in the embodiment of this application.
  • the filter parameters of the adaptive filter are determined; according to the filter parameters and side information, the adaptive filter is used to filter the input pixels to obtain filtered pixels; the first filter parameter is determined, Among them, the first filter parameter is a part of the filter parameter (online filter parameter) in the filter parameter that needs to be encoded; the first filter parameter is encoded and written into the video code stream. Since the filter is obtained by the online filtering part combined with the offline filtering part, the offline filtering part can be used during filtering, and it is suitable for filtering a wide range of videos. At the same time, it can also ensure that some models of the online filtering part are updated to avoid The degradation of the filtering performance means that the filtering performance of the filtering device is improved, and the application range is wide.
  • the embodiment of the present application also provides a filtering method, which is used to decode the video and applied to the decoding end, including: parsing the video code stream and determining the first filtering parameter of the adaptive filter, where the first filtering parameter is Part of the filter parameters in all the filter parameters of the adaptive filter;
  • the input pixels are filtered by an adaptive filter to obtain filtered pixels.
  • the first filter parameter is an online filter parameter, that is, parameter update information.
  • the adaptive filter is a neural network filter.
  • the adaptive filter is two or more cascaded neural network filters of different types.
  • the adaptive filter is a cascade filter of the first neural network filter and the second neural network filter.
  • determining all the filter parameters of the adaptive filter includes: determining the second filter parameter of the neural network filter, where the second filter parameter is all the parameters of the neural network filter; using the first The filter parameter updates the corresponding filter parameter in the second filter parameter, and uses the updated second filter parameter as all the filter parameters of the neural network filter.
  • determining the second filter parameter of the neural network filter includes: using offline training to determine the second filter parameter of the neural network filter; or, before decoding the video stream, obtaining the second filter parameter .
  • determining all filter parameters of the adaptive filter includes: determining all filter parameters of the first neural network filter; and using the first filter parameter as all filter parameters of the second neural network filter.
  • determining all filter parameters of the first neural network filter includes: using offline training to determine all filter parameters of the first neural network filter; or, before decoding the video code stream, obtaining the first All filter parameters of neural network filter.
  • determining all filter parameters of the adaptive filter includes: determining a fifth filter parameter of the first neural network filter; wherein the fifth filter parameter is all parameters of the first neural network filter ; Use part of the first filter parameter as the sixth filter parameter, where the sixth filter parameter is part of the first neural network filter. Use the sixth filter parameter to update the corresponding filter parameter in the fifth filter parameter, which will update The following fifth filter parameter is used as all filter parameters of the first neural network filter; another part of the first filter parameters is used as all parameters of the second neural network filter.
  • determining the fifth filter parameter of the first neural network filter includes: using offline training to determine the fifth filter parameter; or, before decoding the video bitstream, obtaining the fifth filter parameter.
  • offline training is a process of training the neural network filter using one or more images before parsing the video code stream.
  • the adaptive filter is an in-loop filter or a post-processing filter used in decoding a video bitstream.
  • the method (picture adaptive, or sequence adaptive) for updating the parameters of the wave device can be sequence adaptive, which is not limited in the embodiment of this application.
  • the first filter parameter of the adaptive filter is determined by analyzing the video code stream.
  • the first filter parameter is part of the filter parameters of all the filter parameters of the adaptive filter (online filter Parameters); and according to the first filter parameter, determine all filter parameters of the adaptive filter (online filter parameters and offline filter parameters); according to all filter parameters and side information, use the adaptive filter to filter the input pixels to obtain the filter
  • the offline filtering part can be used in filtering, and it is suitable for filtering a wide range of videos, and at the same time, it can also ensure the quality of some models in the online filtering part. It is updated to avoid the degradation of filtering performance, that is, to improve the filtering performance of the filtering device, and the application range is wide.
  • FIG. 11 is a schematic structural diagram of an optional filtering device provided by an embodiment of the application.
  • the filtering device 1 may include: an acquiring part 11, a determining part 12, and a filtering part 13. ,among them,
  • the first acquiring part 11 is configured to acquire pixel information to be filtered
  • the first determining part 12 is configured to determine side information
  • the first filtering part 13 is configured to input the pixel information to be filtered and the side information into a neural network-based filter to output filtered pixels, wherein the filter is combined with the offline filtering part by the online filtering part.
  • the filtering part is obtained.
  • the filter is formed by cascading an online filtering model and an offline filtering model; or,
  • the filter is formed by cascading an online filtering model and an offline filtering model; wherein, there are some online training parameters in the offline filtering model.
  • the filter is formed by an offline filtering model, where some online training parameters exist in the offline filtering model;
  • the first filtering part 13 is further configured to input the pixel information to be filtered and the side information into the offline filtering model based on a neural network to output the filtered pixels.
  • the first filtering part 13 is further configured to input the pixel information to be filtered and the side information into the offline filtering model based on the neural network to output the intermediate filtering Pixels; input the intermediate filtered pixels and the side information to the online filtering model based on a neural network to output the filtered pixels; or, input the pixel information to be filtered and the side information to
  • the online filtering model based on the neural network is used to output intermediate filtered pixels; the intermediate filtered pixels and the side information are input to the offline filtering model based on the neural network to output the filtered pixels.
  • the filter further includes a training part 14 and a writing part 15;
  • the first training part 14 is configured to input the pixel information to be filtered and the side information into a filter based on a neural network to output filtered pixels, based on the filtered pixels, Online training is performed on the filter to obtain online filtering parameters.
  • the first filtering part 13 is also configured to, after training the online part of the filter based on the online filtering parameters, input the subsequent pixel information to be filtered and the side information into the updated filter for filtering to obtain subsequent Filtered pixels;
  • the first writing part 15 is configured to write the online filtering parameters into a video code stream.
  • the offline filtering model is an offline trained filter; wherein, the offline filtering model further includes offline training parameters.
  • the online filtering model is an online training filter; wherein, the online filtering model includes online training parameters.
  • the neural network-based filter is suitable for post-processing filtering, in-loop filtering, pre-processing filtering and prediction processes.
  • the neural network-based filter when the neural network-based filter is suitable for post-processing filtering, it is set at the decoding end;
  • the neural network-based filter When the neural network-based filter is suitable for in-loop processing and filtering, it is set at the decoding end and the encoding end;
  • the filter based on neural network When the filter based on neural network is suitable for preprocessing filtering, it is set at the encoding end.
  • An embodiment of the present application also provides a filtering device for encoding video, including:
  • the second determining part 20 is configured to determine filter parameters of the adaptive filter
  • the second filtering part 21 is configured to use the adaptive filter to filter the input pixels according to the filtering parameters and side information to obtain filtered pixels;
  • the second determining part 20 is further configured to determine a first filter parameter, where the first filter parameter is a part of the filter parameter in the filter parameter that needs to be encoded;
  • the second writing part 22 is configured to encode the first filter parameter and write the video bitstream.
  • the adaptive filter is a neural network filter.
  • the adaptive filter is a cascade filter of the first neural network filter and the second neural network filter.
  • the second determining part 20 is further configured to use offline training to determine the second filter parameters of the neural network filter, where the second filter parameters are all parameters of the neural network filter; Use online training to determine the third filter parameter of the neural network filter; where the second filter parameter is part of the neural network filter parameter; use the third filter parameter to update the corresponding filter parameter in the second filter parameter, and the updated first filter parameter
  • the second filter parameter serves as all the filter parameters of the neural network filter.
  • the second determining part 20 is further configured to use the third filter parameter as the first filter parameter.
  • the second determining part 20 is further configured to: use offline training to determine all filter parameters of the first neural network filter; use online training to determine the fourth Filtering parameters, where the fourth filtering parameter is all the parameters of the second neural network filter.
  • the second determining part 20 is further configured to use the fourth filter parameter as the first filter parameter.
  • the second determining part 20 is further configured to use offline training to determine the fifth filter parameter of the first neural network filter, wherein the fifth filter parameter is the first neural network filter Use online training to determine the sixth filter parameter of the neural network filter, where the sixth filter parameter is part of the first neural network filter; use the sixth filter parameter to update the corresponding filter parameter in the fifth filter parameter , Use the updated fifth filter parameter as the filter parameter of the first neural network filter; in the process of encoding the video or image, use online training to determine the seventh filter parameter of the second neural network filter, where the seventh The filter parameters are all parameters of the second neural network filter.
  • the second determining part 20 is further configured to use the sixth filter parameter and the seventh filter parameter as the first filter parameter.
  • offline training is a process of training the neural network filter using one or more images before starting to encode the video or image
  • Online training is a process of training the neural network filter using one or more images in the video sequence to be encoded in the process of encoding a video or image.
  • the adaptive filter is a pre-processing filter used in encoding video, or an in-loop filter.
  • the filter parameters of the adaptive filter are determined; according to the filter parameters and side information, the adaptive filter is used to filter the input pixels to obtain filtered pixels; the first filter parameter is determined, Among them, the first filter parameter is a part of the filter parameter (online filter parameter) in the filter parameter that needs to be encoded; the first filter parameter is encoded and written into the video code stream. Since the filter is obtained by the online filtering part combined with the offline filtering part, the offline filtering part can be used during filtering, and it is suitable for filtering a wide range of videos. At the same time, it can also ensure that some models of the online filtering part are updated to avoid The degradation of the filtering performance means that the filtering performance of the filtering device is improved, and the application range is wide.
  • An embodiment of the application also provides a filtering device for decoding a video bitstream, including:
  • the third determining part 30 is configured to analyze the video code stream and determine the first filter parameter of the adaptive filter, where the first filter parameter is a part of the filter parameters of all the filter parameters of the adaptive filter; And determining all filter parameters of the adaptive filter according to the first filter parameter;
  • the third filtering part 31 is configured to use the adaptive filter to filter the input pixels according to all the filtering parameters and side information to obtain filtered pixels.
  • the adaptive filter is a neural network filter.
  • the adaptive filter is two or more cascaded neural network filters of different types.
  • the adaptive filter is a cascade filter of the first neural network filter and the second neural network filter.
  • the third determining part 30 is further configured to determine the second filter parameter of the neural network filter, where the second filter parameter is all the parameters of the neural network filter; using the first The filter parameter updates the corresponding filter parameter in the second filter parameter, and uses the updated second filter parameter as all the filter parameters of the neural network filter.
  • the third determining part 30 is further configured to use offline training to determine the second filter parameter of the neural network filter; or, before decoding the video bitstream, obtain the second filter parameter.
  • the third determining part 30 is further configured to determine all the filter parameters of the first neural network filter; use the first filter parameters as all the filter parameters of the second neural network filter.
  • the third determining part 30 is further configured to use offline training to determine all filtering parameters of the first neural network filter; or, before decoding the video stream, obtain the first neural network All filter parameters of the filter.
  • the third determining part 30 is further configured to determine a fifth filter parameter of the first neural network filter; wherein, the fifth filter parameter is all parameters of the first neural network filter ; Use part of the first filter parameter as the sixth filter parameter, where the sixth filter parameter is part of the first neural network filter. Use the sixth filter parameter to update the corresponding filter parameter in the fifth filter parameter, which will update The following fifth filter parameter is used as all filter parameters of the first neural network filter; another part of the first filter parameters is used as all parameters of the second neural network filter.
  • the third determining part 30 is further configured to use offline training to determine the fifth filter parameter; or, before decoding the video bitstream, obtain the fifth filter parameter.
  • offline training is a process of training the neural network filter using one or more images before parsing the video code stream.
  • the adaptive filter is an in-loop filter or a post-processing filter used in decoding a video bitstream.
  • the first filter parameter of the adaptive filter is determined by analyzing the video code stream.
  • the first filter parameter is part of the filter parameters of all the filter parameters of the adaptive filter (online filter Parameters); and according to the first filter parameter, determine all filter parameters of the adaptive filter (online filter parameters and offline filter parameters); according to all filter parameters and side information, use the adaptive filter to filter the input pixels to obtain the filter
  • the offline filtering part can be used in filtering, and it is suitable for filtering a wide range of videos, and at the same time, it can also ensure the quality of some models in the online filtering part. It is updated to avoid the degradation of filtering performance, that is, to improve the filtering performance of the filtering device, and the application range is wide.
  • a "unit" may be a part of a circuit, a part of a processor, a part of a program, or software, etc., of course, may also be a module, or may be non-modular.
  • the various components in this embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be realized in the form of hardware or software function module.
  • the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of this embodiment is essentially or It is said that the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product.
  • the computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can A personal computer, server, or network device, etc.) or a processor (processor) executes all or part of the steps of the method described in this embodiment.
  • the aforementioned storage media include: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.
  • FIG. 12 is a schematic structural diagram of an optional filtering device according to an embodiment of the application.
  • An embodiment of the application provides a filtering device, including:
  • the memory 17 is used to store executable instructions
  • the processor 16 is configured to implement the filtering method provided in the embodiment of the present application when executing the executable instructions stored in the memory 17.
  • the various components in the terminal are coupled together through the communication bus 18.
  • the communication bus 18 is used to implement connection and communication between these components.
  • the communication bus 18 also includes a power bus, a control bus, and a status signal bus.
  • various buses are marked as the communication bus 18 in FIG. 12.
  • the embodiment of the present application provides a computer storage medium that stores executable instructions for causing a processor to execute, to implement the filtering method provided in the embodiment of the present application.
  • the memory in the embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory.
  • the non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), and electrically available Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory.
  • the volatile memory may be a random access memory (Random Access Memory, RAM), which is used as an external cache.
  • RAM static random access memory
  • DRAM dynamic random access memory
  • DRAM synchronous dynamic random access memory
  • DDRSDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • Enhanced SDRAM, ESDRAM Synchronous Link Dynamic Random Access Memory
  • Synchlink DRAM Synchronous Link Dynamic Random Access Memory
  • DRRAM Direct Rambus RAM
  • the processor may be an integrated circuit chip with signal processing capabilities.
  • the steps of the above method can be completed by hardware integrated logic circuits in the processor or instructions in the form of software.
  • the aforementioned processor may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (ASIC), a ready-made programmable gate array (Field Programmable Gate Array, FPGA) or other Programming logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • DSP Digital Signal Processor
  • ASIC application specific integrated circuit
  • FPGA ready-made programmable gate array
  • the methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
  • the embodiments described herein can be implemented by hardware, software, firmware, middleware, microcode, or a combination thereof.
  • the processing unit can be implemented in one or more Application Specific Integrated Circuits (ASIC), Digital Signal Processing (DSP), Digital Signal Processing Equipment (DSP Device, DSPD), programmable Logic device (Programmable Logic Device, PLD), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, and others for performing the functions described in this application Electronic unit or its combination.
  • ASIC Application Specific Integrated Circuits
  • DSP Digital Signal Processing
  • DSP Device Digital Signal Processing Equipment
  • PLD programmable Logic Device
  • PLD Field-Programmable Gate Array
  • FPGA Field-Programmable Gate Array
  • the technology described in this article can be implemented through modules (such as procedures, functions, etc.) that perform the functions described in this article.
  • the software codes can be stored in the memory and executed by the processor.
  • the memory can be implemented in the processor or external to the processor.
  • the method of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. ⁇
  • the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes a number of instructions to enable a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the method described in each embodiment of the present application.
  • the filtering device determines the side information of the video frame to be filtered for the pixel information to be filtered of the video frame to be filtered; inputs the side information and the pixel information to be filtered into the filter for filtering, and outputs the filtered pixels.
  • the filter is obtained by combining the online filtering part and the offline filtering part, so that the offline filtering part can be used during filtering, which is suitable for filtering a wide range of videos, and at the same time, it can also ensure that some models of the online filtering part are updated to avoid filtering.
  • the performance degradation means that the filtering performance of the filtering device is improved, and the application range is wide.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本申请实施例提供了滤波方法及装置、计算机存储介质；方法包括：获取待滤波像素信息；确定边信息；将待滤波像素信息和边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素，其中，滤波器由在线滤波部分结合离线滤波部分得到，其中，所述滤波器由在线滤波部分结合离线滤波部分得到。

Description

滤波方法及装置、计算机存储介质 技术领域
本申请实施例涉及视频编解码的技术领域，尤其涉及一种滤波方法及装置、计算机存储介质。
背景技术
在视频编解码***中，图像/视频滤波是采用滤波器来实现的，其中，预处理滤波器用于对原始图像进行预处理，用来降低视频分辨率，因为需要编码表示的视频分辨率要比原始视频的分辨率低，这样可以使用更少的比特表示，从而能够提高总体的编码效率；后处理滤波器对环内滤波后的视频进行处理，以输出视频，用来提高视频分辨率；环路滤波器被使用来提升重建图像的主客观质量。
针对预处理滤波器、环路滤波器和后处理滤波器，都是采用卷积神经网络来实现的，而基于卷积神经网络的滤波器可以分成两类，一种是离线训练的，另一种是在线训练的。对于离线训练的滤波器，神经网络的所有权重参数在训练完成后就可以同时设置在编码端和解码端，但是由于权重系数是固定好的，因此滤波器性能可能会在某些视频内容的情况下退化。对于在线训练的滤波器，网络的权重参数需要被频繁地重新训练和更新，因此权重系数需要在码流中进行传输，计算量和复杂度高，且适用于很窄范围的视频内容的处理，比较局限。
发明内容
本申请实施例提供了一种滤波方法及装置、计算机存储介质，能够提高滤波装置的滤波性能，且适用范围广。
本申请实施例的技术方案可以如下实现：
本申请实施例提供了一种滤波方法，包括：
获取待滤波像素信息；
确定边信息；
将所述待滤波像素信息和所述边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素，其中，所述滤波器由在线滤波部分结合离线滤波部分得到。
本申请实施例还提供了一种滤波方法，用于对视频进行编码，包括：
确定自适应滤波器的滤波参数；
根据所述滤波参数和边信息，使用所述自适应滤波器对输入像素进行滤波，得到滤波后像素；
确定第一滤波参数，其中，所述第一滤波参数是需要进行编码的、所述滤波参数中的部分滤波参数；
对所述第一滤波参数进行编码，写入视频码流。
本申请实施例还提供了一种滤波方法，用于对视频码流进行解码，包括：
解析视频码流，确定自适应滤波器的第一滤波参数，其中，所述第一滤波参数是 所述自适应滤波器的全部滤波参数中的部分滤波参数；
根据所述第一滤波参数，确定所述自适应滤波器的全部滤波参数；
根据所述全部滤波参数和边信息，使用所述自适应滤波器对输入像素进行滤波，得到滤波后像素。
本申请实施例提供了一种滤波装置，包括：
第一获取部分，被配置为获取待滤波像素信息；
第一确定部分，被配置为确定边信息；
第一滤波部分，被配置将所述待滤波像素信息和所述边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素，其中，所述滤波器由在线滤波部分结合离线滤波部分得到。
本申请实施例还提供了一种滤波装置，用于对视频进行编码，包括：
第二确定部分，被配置为确定自适应滤波器的滤波参数；
第二滤波部分，被配置为根据所述滤波参数和边信息，使用所述自适应滤波器对输入像素进行滤波，得到滤波后像素；
所述第二确定部分，还被配置为确定第一滤波参数，其中，所述第一滤波参数是需要进行编码的、所述滤波参数中的部分滤波参数；
第二写入部分，被配置为对所述第一滤波参数进行编码，写入视频码流。
本申请实施例还提供了一种滤波装置，用于对视频进行解码，包括：
第三确定部分，被配置为解析视频码流，确定自适应滤波器的第一滤波参数，其中，所述第一滤波参数是所述自适应滤波器的全部滤波参数中的部分滤波参数；及根据所述第一滤波参数，确定所述自适应滤波器的全部滤波参数；
第三滤波部分，被配置为根据所述全部滤波参数和边信息，使用所述自适应滤波器对输入像素进行滤波，得到滤波后像素。
本申请实施例还提供了一种滤波装置，包括：
存储器，用于存储可执行指令；
处理器，用于执行所述存储器中存储的可执行指令时，实现本申请实施例提供的滤波方法。
本申请实施例提供了一种计算机存储介质，存储有可执行指令，用于引起处理器执行时，实现本申请实施例提供的滤波方法。
本申请实施例具有以下有益效果：
滤波装置针对待滤波像素信息；确定出待滤波视频帧的边信息；将边信息和待滤波像素输入滤波器进行滤波，输出滤波后像素，由于滤波器由在线滤波部分结合离线滤波部分得到，这样在滤波时即可以使用离线滤波部分，而适用于很广的范围的视频的过滤，同时还可以保证在线滤波部分的参数的更新，避免滤波性能的退化，即提高滤波装置的滤波性能，且适用范围广。
附图说明
图1为本申请实施例提供的一种编码框图的结构示意图；
图2为本申请实施例提供的一种解码框图的结构示意图；
图3为本申请实施例提供的一种滤波方法的一种可选的流程图；
图4为本申请实施例提供的一种块划分矩阵的结构示意图；
图5为本申请实施例提供的滤波器的连接方式示意图一；
图6为本申请实施例提供的滤波器的连接方式示意图二；
图7为本申请实施例提供的滤波器的连接方式示意图三；
图8为本申请实施例提供的滤波器的连接方式示意图四；
图9为本申请实施例提供的滤波器的连接方式示意图五；
图10为本申请实施例提供的一种滤波方法的另一种可选的流程图；
图11为本申请实施例提供的一种滤波装置的一种可选的结构示意图；
图12为本申请实施例提供的一种滤波装置的另一种可选的结构示意图。
具体实施方式
为了能够更加详尽地了解本申请实施例的特点与技术内容，下面结合附图对本申请实施例的实现进行详细阐述，所附附图仅供参考说明之用，并非用来限定本申请实施例。
在视频编解码***中，待编码视频包括原始视频帧，而原始视频帧中包括原始图像，对该原始图像进行多种处理，诸如预测、变换、量化、重建和滤波等，在这些处理过程中，已处理的视频图像相对原始图像可能已经发生像素值偏移，导致视觉障碍或假象。此外，在大多数视频编解码***采用的基于块形编码单元(CU，Coding Unit)的混合编码框架下，由于相邻的编码块采用不同的编码参数(比如不同的变换过程、不同的量化参数(QP，Quantization Parameter)、不同的预测方式、不同的参考图像帧等)，各个编码块引入的误差大小及其分布特性的相互独立，相邻编码块边界的不连续性，产生块效应。这些失真影响了重建图像块的主客观质量，若重建图像块作为后续编码像素的参考图像，甚至还会影响后续编解码的预测准确性，进而影响了视频码流中比特的大小。因此，在视频编解码***中，往往会加入预处理滤波器、后处理滤波器和环路滤波器(In-Loop Filter)等来提升重建图像的主客观质量。
图1为传统编码框图的结构示意图，如图1所示，该传统编码框图10可以包括变换及量化单元101、反变换及反量化单元102、预测单元103、环内滤波单元104、熵编码单元105、预处理滤波单元106和后处理滤波单元107等部件；其中，预测单元103还包括帧内预测单元1031和帧间预测单元1032。针对输入的原始图像，通过初步划分可以得到编码树单元(CTU，Coding Tree Unit)，而对一个CTU继续进行内容自适应划分，可以得到CU，CU一般包含一个或多个编码块(CB，Coding Block)。对编码块进行帧内预测单元1031的帧内预测或者帧间预测单元1032的帧间预测，可以得到残差信息；将该残差信息通过变换及量化单元101对该编码块进行变换，包括将残差信息从像素域变换到变换域，以及对所得到的变换系数进行量化，用以进一步减少比特率；在确定出预测模式之后，预测单元103还用于将所选择的帧内预测数据或者帧间预测数据提供给熵编码单元105；此外，反变换与反量化单元102是用于该编码块的重构建，在像素域中重构建残差块，该重构建残差块通过环内滤波单元104去除方块效应伪影，然后将该重构残差块添加到解码图像缓存单元中，用以产生经重构建的参考图像；熵编码单元105是用于编码各种编码参数及量化后的变换系数，比如熵编码单元105采用头信息编码及基于上下文的自适应二进制算术编码(CABAC，Context-based Adaptive Binary Arithmatic Coding)算法，可以用于编码指示所确定的预测模式的编码信息，输出对应的码流。
针对图1中的传统编码框图10，环内滤波单元104为环路滤波器，也称之为环内滤波器，它可以包括去方块滤波器(DBF，De-Blocking Filter)、样点自适应补偿(SAO，Sample Adaptive Offset)滤波器和自适应环路滤波器(ALF，Adaptive Loop Filter)等。其中，去方块滤波器用于实现去方块滤波，在下一代视频编码标准H.266/多功能视频编码(Versatile Video Coding，VVC)中，对于原始图像中所有的编码块边界，首先根据 边界两边的编码参数进行边界强度的判定，以及根据计算得到的块边界纹理度值来判断是否进行去方块的滤波决策，最后再根据边界强度、滤波决策对编码块边界两侧的像素信息进行修正处理。在VVC中，当去方块滤波执行之后，为了减轻高频交流系数的量化失真，还引入了SAO技术，即样点自适应补偿滤波器；进一步地，从像素域入手，对波峰处的像素添加负值、波谷处的像素添加正值进行补偿处理。在VVC中，当去方块滤波和样点自适应补偿滤波执行之后，还需要进一步使用自适应环路滤波器进行滤波处理；对于自适应环路滤波，它是根据原始图像的像素值和失真图像的像素值来计算得到均方意义下的最优滤波器。
针对图1中的传统编码框图10，预处理滤波单元106用于接收到输入的原始视频帧，对原始视频帧进行预处理滤波，以降低视频的分辨率，后处理滤波单元107用于接收到环内滤波后的视频帧，对环内滤波后的视频帧进行后处理滤波，以提高视频的分辨率，这样，在视频的编解码过程中可以使用较少的比特得到重建视频帧，从而可以提高总体编解码的效率。目前预处理滤波器和后处理滤波器均采用的神经网络的输入为单输入或者多输入，即输入时单种图像分量或者多种图像分量，实现重建图像。
与图1中的编码框图类似，图2为传统解码框图的结构示意图，如图2所示，该传统解码框图20可以包括熵编码单元201、反量化反变换单元202、预测单元203、环内滤波单元204和后处理滤波单元205等部件；其中，预测单元203还包括帧内预测单元2031和帧间预测单元2032。这里，需要说明的是，视频解码过程是与视频编码过程相反或相逆的过程，其中，视频解码过程中将得到的后处理滤波后的图像确定为重建视频帧，由图2可以看出，在解码过程中并不涉及编码过程中的预处理滤波单元，只有后处理滤波单元和环内滤波单元。
在本申请实施例中，预处理滤波单元、后处理滤波单元和环内滤波单元都可以说是滤波器的一种，并且本申请实施例中的滤波器可以是卷积神经网络(Convolutional Neural Networks，CNN)滤波器，也可以是其他深度学习所建立的滤波器，本申请实施例不作具体限定。
以卷积神经网络滤波器为例，卷积神经网络滤波器不仅可以代替图1中的预处理滤波单元、后处理滤波单元和环内滤波单元，而且也可以部分替代图1中的预处理滤波单元、后处理滤波单元和环内滤波单元中的任意一个或者两个，甚至也可以和图1中的预处理滤波单元、后处理滤波单元和环内滤波单元中的任意一个或者多个组合使用。还需要注意的是，针对图1中所示的每一个部件，比如变换及量化单元101、反变换及反量化单元102、预测单元103、环内滤波单元104、熵编码单元105、预处理滤波单元106和后处理滤波单元107，这些部件可以是虚拟模块，也可以是硬件模块。另外，本领域技术人员可以理解，这些单元并不构成对编码框图的限定，编码框图可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
本申请实施例中，当卷积神经网络滤波器作为环内滤波单元时，卷积神经网络滤波器进行滤波网络训练之后可以直接部署在编码端和解码端。而且该卷积神经网络滤波器还可以将边信息等辅助信息与输入的待滤波图像进行处理；这样，不仅充分利用了图像边信息的关系，进一步提升了编解码过程中视频重建图像的主客观质量。当卷积神经网络滤波器作为后处理滤波单元时，卷积神经网络滤波器进行滤波网络训练之后可以直接部署在解码端，当卷积神经网络滤波器作为预处理滤波单元时，卷积神经网络滤波器进行滤波网络训练之后可以直接部署在编码端。
需要说明的是，本申请实施例的滤波方法，在滤波器类型不同时，可以应用于编码***和/或解码***。例如，为了节省编码码率，同时保证解码***可以进行正确的解码处理，在编码***和解码***必须同步部署本申请实施例的环内滤波器。
在本申请实施例中，不论哪种滤波器采用卷积神经网络滤波器基于卷积神经网络的滤波器可以分成两类，一种是离线训练的，另一种是在线训练的。对于离线训练的滤波器，神经网络的所有权重参数在训练完成后就可以同时设置在编码端和解码端，但是由于权重系数是固定好的，因此滤波器性能可能会在某些视频内容的情况下退化。对于在线训练的滤波器，网络的权重参数需要被频繁地重新训练和更新，因此权重系数需要在码流中进行传输，计算量和复杂度高，且适用于很窄范围的视频内容的处理，比较局限。
针对上述情况，本申请实施例提供一种滤波方法，该方法应用于一滤波装置中，该滤波装置可以设置于编码器中的预处理滤波器和环内滤波器中，也可以设置于解码器的环内滤波器和后处理滤波器中，还可以作用于其他预测过程中使用的滤波器中，这里本申请实施例不作具体限定。
也就是说，基于神经网络的滤波器适用于后处理滤波、环内滤波、预处理滤波和预测过程。
在本申请实施例中，基于神经网络的滤波器适用于后处理滤波时，设置在解码端；基于神经网络的滤波器适用于环内处理滤波时，设置在解码端和编码端；基于神经网络的滤波器适用于预处理滤波时，设置在编码端。
图3为本申请实施例提供的一种可选的滤波方法的流程示意图，参考图3所示，该滤波方法可以包括：
S101、获取待滤波像素信息。
S102、确定边信息。
S103、将待滤波像素信息和边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素，其中，滤波器由在线滤波部分结合离线滤波部分得到。
在S101中，在本申请实施例中，待滤波视频帧是在对待编码视频中的原始图像进行视频编码过程中所产生的，待编码视频包括原始图像帧，原始图像帧包括原始图像。
需要说明的是，在本申请实施例中，待滤波视频帧中包含多帧图像，滤波装置滤波时是针对每帧图像的待滤波像素信息进行滤波的。
在S102中，针对待滤波视频帧中的每帧图像都有对应的边信息，即每帧图像的待滤波像素信息对应的边信息。其中，边信息表征每帧图像的边界信息的。
需要说明的是，在本申请实施例中，原始图像可以划分为CTU，或者由CTU划分为CU；也就是说，本申请实施例中的边信息可以是指CTU划分信息，也可以是指CU划分信息；这样，本申请实施例的滤波方法不仅可以应用于CU级别的滤波，也可以应用于CTU级别的滤波，本申请实施例不作具体限定。下面以CU划分信息作为边信息为例进行描述。
在本申请实施例中，基于待编码视频中的原始图像，分别对原始图像进行编码单元CU划分，得到CU划分信息。针对CU划分信息，在CU边界对应的各个像素点位置填充第一值，在其他像素点位置填充第二值，得到与CU划分信息对应的第一矩阵；其中第一值与第二值不同；这里的第一矩阵为每帧图像的边信息。
需要说明的是，第一值可以是预设设定的数值、字母等；第二值也可以是是预设设定的数值、字母等，只要第一值与第二值不同即可，例如，第一值可以设置为2，第二值可以设置为1，本申请实施例不作限制。
在S103中，滤波装置通过滤波器，结合边信息对待滤波视频帧的待滤波像素信息进行滤波，得到滤波后像素，可以理解为最后得到滤波图像，其中，滤波器由在线滤波部分结合离线滤波部分得到。
在本申请实施例中，滤波装置可以将CU信息作为辅助信息来辅助待滤波视频帧进行滤波处理，也就是说，在对待编码视频中的原始图像进行视频编码的过程中，可以充 分利用CU划分信息，将其与待滤波视频帧的进行融合后来指导滤波。
在本申请的一些实施例中，将CU划分信息转换成一个编码单元图(Coding Unit Map，CUmap)，并以二维矩阵表示，即CUmap矩阵，也即本申请实施例中的第一矩阵；也就是说，以原始图像为例，可以将其划分为多个CU；在每个CU边界对应的各个像素点位置用第一值进行填充，而在其他像素点位置用第二值进行填充，这样就可以构造出一张反映CU划分信息的第一矩阵。示例性地，参见图4，其示出了本申请实施例提供的一种块划分矩阵的结构示意图。如图4所示，如果该图表示一个CTU，那么可以将该CTU划分为9个CU；假定第一值设置为2，第二值设置为1；这样，在每个CU边界对应的各个像素点位置用2进行填充，而在其他像素点位置用1进行填充，也就是说，利用2所填充的像素点位置表示了CU的边界，从而可以确定出CU划分信息，即待滤波视频帧的一帧图像的边信息。
需要说明的是，在本申请实施例中，CU划分信息也可以是基于图像分量级别来对应的，本申请实施例不作限制。
在本申请实施例中，滤波器由在线滤波模型与离线滤波模型级联形成；或者，滤波器由在线滤波模型与离线滤波模型级联形成；其中，离线滤波模型中存在部分在线训练的参数；或者，滤波器由离线滤波模型形成，其中，离线滤波模型中存在部分在线训练的参数。
针对滤波器由离线滤波模型形成时，滤波装置通过滤波器，结合边信息对待滤波视频帧的待滤波像素信息进行滤波，得到滤波后像素的实现过程可以为：滤波装置采用离线滤波模型，结合边信息对待滤波视频帧进行滤波，得到滤波图像，即将待滤波像素信息和边信息输入至基于神经网络的离线滤波模型，以输出滤波后像素。
针对滤波器由在线滤波模型与离线滤波模型级联形成；或者，滤波器由在线滤波模型与离线滤波模型级联形成时，滤波装置通过滤波器，结合边信息对待滤波视频帧的待滤波像素信息进行滤波，得到滤波图像，即滤波后像素的实现过程为：滤波装置采用离线滤波模型，结合边信息对待滤波视频帧进行滤波，得到中间滤波图像；采用在线滤波模型，结合边信息对中间滤波图像进行滤波，得到滤波图像。或者，滤波装置采用在线滤波模型，结合边信息对待滤波视频帧进行滤波，得到中间滤波图像；采用离线滤波模型，结合边信息对中间滤波图像进行滤波，得到滤波图像。即将待滤波像素信息和边信息输入至基于神经网络的离线滤波模型，以输出中间滤波后像素；将中间滤波后像素和边信息输入至基于神经网络的在线滤波模型，以输出滤波后像素；或者，将待滤波像素信息和边信息输入至基于神经网络的在线滤波模型，以输出中间滤波后像素；将中间滤波后像素和边信息输入至基于神经网络的离线滤波模型，以输出滤波后像素。
也就是说，在本申请实施例中，滤波器可以是采用在线卷积神经网络结合离线卷积神经网络结合形成的。例如，一个离线模型与一个在线模型级联起来，或者，一个离线模型，但是其中一部分可以在线训练，或者，一个离线模型(其中一部分可以在线训练)与一个在线模型级联起来，本申请实施例不作限制。
其中，离线滤波模型为离线训练的滤波器；离线滤波模型还包括离线训练的参数。
在线滤波模型为在线训练的滤波器；其中，在线滤波模型包括在线训练的参数。
详细地，离线滤波模型指离线训练的滤波器，卷积神经网络需要用海量的图片进行训练，并在从训练集中划分出的图片上测试。如果性能很有效，就可以被应用为视频编解码技术中的环内/后处理滤波器等滤波器中。卷积神经网络中的所有权重参数(即参数)在训练完成后就可以同时设置在编码端和解码端。
在线滤波模型指在线训练的滤波器，卷积神经网络往往是基于随机接入片段(视 频序列中刚刚被编码的一些视频帧)来训练的，得到更新的参数，采用更新了参数的更新在线滤波模型处理同一视频序列的后续帧。此类卷积神经网络的规模较小，只能适用于很窄范围的视频序列的处理。
示例性的，滤波器由在线滤波模型1(On-line traind NN)与离线滤波模型2(Off-line traind NN)级联形成，其中，如图5所示的级联顺序或如图6所示的级联顺序均可。针对图5的滤波器，滤波装置将待滤波像素信息4和其对应的边信息3(Side information)输入离线滤波模型2，经过离线滤波，得到了中间滤波后像素5；将中间滤波后像素5和边信息3再输入至在线滤波模型1，进行在线滤波，滤波输出(Filtered output)滤波后像素6。针对图6的滤波器，滤波装置将待滤波像素信息4和其对应的边信息3输入在线滤波模型1，经过在线滤波，得到了中间滤波后像素5；将中间滤波后像素5和边信息3再输入至离线滤波模型2，进行离线滤波，滤波输出滤波后像素6。
示例性的，滤波器由在线滤波模型与离线滤波模型级联形成；其中，离线滤波模型中存在部分在线训练的参数。其中，如图7所示的级联顺序或如图8所示的级联顺序均可。如图7所示，滤波装置将待滤波像素信息4和其对应的边信息3输入离线滤波模型2，经过离线-在线混合滤波，得到了中间滤波后像素5；将中间滤波后像素5和边信息3再输入至在线滤波模型1，进行在线滤波，滤波输出滤波后像素6。针对图8的滤波器，滤波装置将待滤波像素信息4和其对应的边信息3输入在线滤波模型1，经过在线滤波，得到了中间滤波后像素5；将中间滤波后像素5和边信息3再输入至离线滤波模型2，进行离线-在线混合滤波，滤波输出滤波后像素6。
示例性的，滤波器由离线滤波模型形成，其中，离线滤波模型中存在部分在线训练的参数。如图9所示，滤波装置将待滤波像素信息1和其对应的边信息2输入离线滤波模型3，经过离线-在线混合滤波，滤波输出滤波后像素4。
可以理解的是，离线滤波模型的优点是性能突出、不需要额外传输权重系数，缺点是对序列的自适应能力有所欠缺；在线滤波模型的优点是对序列具有自适应能力，但需要传输权重系数。离线滤波部分结合在线滤波部分的方式既能够用到离线滤波模型的性能，又能用在线滤波模型再次提高视频的客观质量。也就是说，本申请实施例的滤波装置可以在不同的视频的滤波处理中，得到泛化能力和序列适应性的折衷，在低复杂度的条件下，能带来更优的编码性能。
需要说明的是，在本申请实施例中，滤波装置将待滤波像素信息和边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素之后，基于滤波后像素，对滤波器进行在线训练，得到在线滤波参数；基于在线滤波参数训练滤波器的在线部分后，将后续待滤波像素信息和边信息输入更新后滤波器中进行滤波，得到后续滤波后像素；将在线滤波参数写入视频码流。
也就是说，在本申请实施例中，神经网络往往是基于随机接入片段(视频序列中刚刚被编码的一些视频帧的像素信息)来训练的，训练后就立即被用于同一序列的后续帧，即后续帧的像素信息。此类神经网络的规模较小，只能适用于很窄范围的视频内容。训练得到的权重参数(即在线滤波参数)需要被频繁地重新训练和更新。因此权重系数需要在码流中进行传输。
下面以待滤波视频帧的待滤波像素信息为例进行在线滤波参数的说明。
在本申请的一些实施例中，基于图3，如图10所示，S103的实现的一种可选的实现过程可以包括：S1031-S1036。如下：
S1031、通过滤波器，结合边信息中第i帧图像的边信息，对待滤波视频帧的第i帧图像的待滤波像素信息进行滤波，得到第i帧滤波图像，其中，i大于等于1，且小于等于N，N为待滤波视频帧的总帧数。
S1032、继续对第i+1帧图像的待滤波像素信息进行滤波处理，直至对第i+H帧图像的滤波处理完成，得到第i帧滤波图像至第i+H帧滤波图像，其中，H大于1且小于N-i。
S1033、采用第i帧滤波图像至第i+H帧滤波图像，对滤波器进行在线滤波部分进行训练，得到在线滤波部分的参数更新信息。
S1034、利用参数更新信息，得到更新后在线滤波部分。
S1035、利用更新后在线滤波部分和离线滤波部分，结合边信息中第i+H+1帧图像的边信息，对第i+H+1帧滤波图像参数更新信息进行滤波处理，得到第i+H+1帧滤波图像。
S1036、继续对第i+H+2帧图像参数更新信息进行滤波处理，直至对第N帧图像的滤波处理完成，得到第i+H+1帧滤波图像至第N帧滤波图像，第1帧滤波图像至第N帧滤波图像为滤波图像。
在本申请实施例中，由于滤波器均是存在离线滤波部分和在线滤波部分的，不同的是，有的在线滤波部分是直接通过在线滤波模型实现的，而有的则是通过离线滤波模型中的部分参数是在线训练得到的，而实现在线滤波部分的，后者是上述两种在线滤波结合来实现的。
可以理解的是，在线滤波部分是基于随机接入片段(视频序列中刚刚被滤波过的一些视频帧)来训练的，得到更新的参数，采用更新了参数的更新在线滤波部分处理同一视频序列的后续帧的。
在本申请实施例中，待滤波视频帧可以为包含N帧图像，N为待滤波视频帧的总帧数，且N为大于1的正整数，N的数值由待滤波视频帧的帧数决定，本申请实施例不作限制。
于是，在本申请实施例中，滤波装置在对待滤波视频帧中的第i帧图像进行滤波时，将第i帧图像和该第i帧图像的边信息输入进滤波器，经过在线滤波和离线滤波后，输出第i帧滤波图像，其中，i是从1开始的，即滤波装置是从第1帧图像开始滤波的，完成第i帧图像的滤波后，该滤波装置继续对第i+1帧图像进行滤波处理，直至对第i+H帧图像的滤波处理完成，得到第i帧滤波图像至第i+H帧滤波图像，其中，H大于1且小于N-i。也就是说，滤波装置在采用包含在线滤波部分和离线滤波部分的滤波器对待滤波视频帧进行滤波时，在滤波了一段序列的帧图像后，可以采用已经滤波过的i+H帧图像和i+H帧过滤图像(第i帧滤波图像至第i+H帧滤波图像)作为训练样本，再次对滤波器中的在线部分进行训练，得到训练结果与i+H帧过滤图像最接近的输出结果时，获取此时在线滤波部分的参数更新信息，利用参数更新信息，得到更新后在线滤波部分，滤波装置就可以利用更新后在线滤波部分，在结合已有的离线滤波部分和第i+H+1帧图像的边信息，对第i+H+1帧滤波图像进行滤波处理，得到第i+H+1帧滤波图像，继续对第i+H+2帧图像进行滤波处理，直至对第N帧图像的滤波处理完成，得到第i+H+1帧滤波图像至第N帧滤波图像，第1帧滤波图像至第N帧滤波图像为滤波图像。
需要说明的是，在本申请实施例中，滤波装置在滤波了H-i帧后，就可以开始对滤波器的在线滤波部分进行更新了，而具体的H的取值则可以根据实际需求和具体设计决定，本申请实施例不作限制。
还需要说明的是，针对第i+H+1帧图像至第N帧图像的滤波处理中，还是可以进行在进行了H-i帧图像的处理后，还没有处理到第N帧图像时，再次进行在线滤波部分的更新，使用更新后的在线滤波部分继续进行后续帧的滤波，直至第N帧图像的滤波完成。也就是说，滤波装置可以针对一个固定帧数的图像滤波后，就开始进行在线滤波部分的在线滤波参数的更新，也可以每次进行在线滤波部分更新时的已滤波帧数每次都不 一样，本申请实施例不作限制。停止更新的条件为对最后一帧图像的滤波完成即可。
在本申请的一些实施例中，在线滤波部分可以为在线滤波模型，也可以为离线模型中的部分卷积神经网络的层级是可在线更新参数，得到参数更新信息(即在线滤波参数)，还可以是前面二种在线滤波部分的结合，本申请实施例不作限制。
示例性的，滤波器由在线滤波模型1(On-line traind NN)与离线滤波模型2(Off-line traind NN)级联形成，其中，如图5所示的级联顺序或如图6所示的级联顺序均可。针对图5的滤波器和图6的滤波器，滤波装置将待滤波像素信息4和其对应的边信息3(Side information)输入离线滤波模型2和在线滤波模型1滤波过程中，还可以采用上一帧的滤波结果对滤波器进行在线部分进行在线训练，即对在线滤波模型1训练，得到参数更新信息(即在线滤波参数)；以便基于在线滤波参数训练滤波器的在线部分后，将后续待滤波像素信息和边信息输入更新后滤波器中进行滤波，得到后续滤波后像素。
示例性的，滤波器由在线滤波模型与离线滤波模型级联形成；其中，离线滤波模型中存在部分在线训练的参数。其中，如图7所示的级联顺序或如图8所示的级联顺序均可。如图7所示和图8所示，滤波装置将待滤波像素信息4和其对应的边信息3(Side information)输入离线滤波模型2(离线-在线混合滤波)和在线滤波模型1滤波过程中，还可以采用上一帧的滤波结果对滤波器进行在线部分进行在线训练，即对离线滤波模型2中的在线部分以及在线滤波模型1训练，得到参数更新信息(即在线滤波参数)；以便基于在线滤波参数训练滤波器的在线部分后，将后续待滤波像素信息和边信息输入更新后滤波器中进行滤波，得到后续滤波后像素。
示例性的，滤波器由离线滤波模型形成，其中，离线滤波模型中存在部分在线训练的参数。如图9所示，滤波装置将待滤波像素信息1和其对应的边信息2输入离线滤波模型3，经过离线-在线混合滤波，滤波输出滤波后像素4，同时还可以采用上一帧的滤波结果对离线滤波模型3进行在线部分进行在线训练，即对离线滤波模型3的在线部分进行训练，得到参数更新信息(即在线滤波参数)；以便基于在线滤波参数训练滤波器的在线部分后，将后续待滤波像素信息和边信息输入更新后滤波器中进行滤波，得到后续滤波后像素。
可以理解的是，滤波装置针对待滤波视频帧；确定出待滤波视频帧的边信息；将边信息和待滤波视频帧输入滤波器进行滤波，输出滤波图像，由于滤波器由在线滤波部分结合离线滤波部分得到，这样在滤波时即可以使用离线滤波部分，而适用于很广的范围的视频的过滤，同时还可以保证在线滤波部分的在线滤波参数的更新，使用更新的在线滤波部分进行待滤波视频帧的后续帧的滤波，从而避免滤波性能的退化，既提高滤波装置的滤波性能，且适用范围广。
在本申请实施例还提供了一种滤波方法，用于对视频进行编码，应用于编码端，包括：确定自适应滤波器的滤波参数；根据滤波参数和边信息，使用自适应滤波器对输入像素进行滤波，得到滤波后像素；确定第一滤波参数，其中，第一滤波参数是需要进行编码的、滤波参数中的部分滤波参数；对第一滤波参数进行编码，写入视频码流。
其中，第一滤波参数为在线滤波参数，即参数更新信息。
在本申请的一些实施例中，自适应滤波器是一个神经网络滤波器。
在本申请的一些实施例中，自适应滤波器是第一神经网络滤波器和第二神经网络滤波器的级联滤波器。
在本申请的一些实施例中，确定自适应滤波器的滤波参数，包括：使用离线训练确定神经网络滤波器的第二滤波参数，其中，第二滤波参数是神经网络滤波器的全部参数；使用在线训练确定神经网络滤波器的第三滤波参数；其中，第二滤波参数是神 经网络滤波器的部分参数；使用第三滤波参数更新第二滤波参数中对应的滤波参数，将更新后的第二滤波参数作为神经网络滤波器的全部滤波参数。
在本申请的一些实施例中，确定第一滤波参数，包括：将第三滤波参数作为第一滤波参数。
在本申请的一些实施例中，确定自适应滤波器的滤波参数，包括：使用离线训练确定第一神经网络滤波器的全部滤波参数；使用在线训练确定第二神经网络滤波器的第四滤波参数，其中，第四滤波参数是第二神经网络滤波器的全部参数。
在本申请的一些实施例中，确定第一滤波参数，包括：将第四滤波参数作为第一滤波参数。
在本申请的一些实施例中，确定自适应滤波器的滤波参数，包括：使用离线训练确定第一神经网络滤波器的第五滤波参数，其中，第五滤波参数是第一神经网络滤波器的全部参数；使用在线训练确定神经网络滤波器的第六滤波参数，其中，第六滤波参数是第一神经网络滤波器的部分参数；使用第六滤波参数更新第五滤波参数中对应的滤波参数，将更新后的第五滤波参数作为第一神经网络滤波器的滤波参数；在对视频或图像编码的过程中，使用在线训练确定第二神经网络滤波器的第七滤波参数，其中，第七滤波参数是第二神经网络滤波器的全部参数。
在本申请的一些实施例中，确定第一滤波参数，包括：将第六滤波参数和第七滤波参数作为第一滤波参数。
在本申请的一些实施例中，其中，离线训练是在对视频或图像开始编码之前，使用一个或多个图像对神经网络滤波器进行训练的过程；
在线训练是在对视频或图像编码的过程中，使用待编码视频序列中的一个或多个图像对神经网络滤波器进行训练的过程。
在本申请的一些实施例中，自适应滤波器是对视频进行编码中使用的预处理滤波器，或者环内滤波器。
需要说明的是，波器参数更新的方式(picture adaptive，or sequence adaptive)可以采用random access segment，或sequence adaptive，本申请实施例不作限制。
可以理解的是，滤波装置在编码端时，确定自适应滤波器的滤波参数；根据滤波参数和边信息，使用自适应滤波器对输入像素进行滤波，得到滤波后像素；确定第一滤波参数，其中，第一滤波参数是需要进行编码的、滤波参数中的部分滤波参数(在线滤波参数)；对第一滤波参数进行编码，写入视频码流。由于滤波器由在线滤波部分结合离线滤波部分得到，这样在滤波时即可以使用离线滤波部分，而适用于很广的范围的视频的过滤，同时还可以保证在线滤波部分的一些模型的更新，避免滤波性能的退化，即提高滤波装置的滤波性能，且适用范围广。
在本申请实施例还提供了一种滤波方法，用于对视频进行解码，应用于解码端，包括：解析视频码流，确定自适应滤波器的第一滤波参数，其中，第一滤波参数是自适应滤波器的全部滤波参数中的部分滤波参数；
根据第一滤波参数，确定自适应滤波器的全部滤波参数；
根据全部滤波参数和边信息，使用自适应滤波器对输入像素进行滤波，得到滤波后像素。
需要说明的是，第一滤波参数为在线滤波参数，即参数更新信息。
在本申请的一些实施例中，自适应滤波器是一个神经网络滤波器。
在本申请的一些实施例中，自适应滤波器是两个或多个级联的、不同类型的神经网络滤波器。
在本申请的一些实施例中，自适应滤波器是第一神经网络滤波器和第二神经网络 滤波器的级联滤波器。
在本申请的一些实施例中，确定自适应滤波器的全部滤波参数，包括：确定神经网络滤波器的第二滤波参数，其中，第二滤波参数是神经网络滤波器的全部参数；使用第一滤波参数更新第二滤波参数中对应的滤波参数，将更新后的第二滤波参数作为神经网络滤波器的全部滤波参数。
在本申请的一些实施例中，确定神经网络滤波器的第二滤波参数，包括：使用离线训练确定神经网络滤波器的第二滤波参数；或者，在解码视频码流之前，获得第二滤波参数。
在本申请的一些实施例中，确定自适应滤波器的全部滤波参数，包括：确定第一神经网络滤波器的全部滤波参数；将第一滤波参数作为第二神经网络滤波器的全部滤波参数。
在本申请的一些实施例中，确定第一神经网络滤波器的全部滤波参数，包括：使用离线训练确定第一神经网络滤波器的全部滤波参数；或者，在解码视频码流之前，获得第一神经网络滤波器的全部滤波参数。
在本申请的一些实施例中，确定自适应滤波器的全部滤波参数，包括：确定第一神经网络滤波器的第五滤波参数；其中，第五滤波参数是第一神经网络滤波器的全部参数；将第一滤波参数中的部分参数作为第六滤波参数，其中，第六滤波参数是第一神经网络滤波器的部分参数使用第六滤波参数更新第五滤波参数中对应的滤波参数，将更新后的第五滤波参数作为第一神经网络滤波器的全部滤波参数；将第一滤波参数中的另一部分参数作为第二神经网络滤波器的全部参数。
在本申请的一些实施例中，确定第一神经网络滤波器的第五滤波参数，包括：使用离线训练确定第五滤波参数；或者，在解码视频码流之前，获得第五滤波参数。
在本申请的一些实施例中，离线训练是在解析视频码流之前，使用一个或多个图像对神经网络滤波器进行训练的过程。
在本申请的一些实施例中，自适应滤波器是对视频码流进行解码中使用的环内滤波器或者后处理滤波器。
需要说明的是，波器参数更新的方式(picture adaptive，or sequence adaptive)可以采用sequence adaptive，本申请实施例不作限制。
可以理解的是，滤波装置在解码端时，通过解析视频码流，确定自适应滤波器的第一滤波参数，第一滤波参数是自适应滤波器的全部滤波参数中的部分滤波参数(在线滤波参数)；及根据第一滤波参数，确定自适应滤波器的全部滤波参数(在线滤波参数和离线滤波参数)；根据全部滤波参数和边信息，使用自适应滤波器对输入像素进行滤波，得到滤波后像素，由于滤波器由在线滤波部分结合离线滤波部分得到，这样在滤波时即可以使用离线滤波部分，而适用于很广的范围的视频的过滤，同时还可以保证在线滤波部分的一些模型的更新，避免滤波性能的退化，即提高滤波装置的滤波性能，且适用范围广。
基于相同的发明构思，图11为本申请实施例提供的一种可选的滤波装置的结构示意图，如图11所示，该滤波装置1可以包括：获取部分11、确定部分12和滤波部分13，其中，
第一获取部分11，被配置为获取待滤波像素信息；
第一确定部分12，被配置为确定边信息；
第一滤波部分13，被配置为将所述待滤波像素信息和所述边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素，其中，所述滤波器由在线滤波部分结合离线滤波部分得到。
在本申请的一些实施例中，所述滤波器由在线滤波模型与离线滤波模型级联形成；或者，
所述滤波器由在线滤波模型与离线滤波模型级联形成；其中，所述离线滤波模型中存在部分在线训练的参数。
在本申请的一些实施例中，所述滤波器由离线滤波模型形成，其中，离线滤波模型中存在部分在线训练的参数；
所述第一滤波部分13，还被配置为将所述待滤波像素信息和所述边信息输入至基于神经网络的所述离线滤波模型，以输出所述滤波后像素。
在本申请的一些实施例中，所述第一滤波部分13，还被配置为将所述待滤波像素信息和所述边信息输入至基于神经网络的所述离线滤波模型，以输出中间滤波后像素；将所述中间滤波后像素和所述边信息输入至基于神经网络的所述在线滤波模型，以输出所述滤波后像素；或者，将所述待滤波像素信息和所述边信息输入至基于神经网络的所述在线滤波模型，以输出中间滤波后像素；将所述中间滤波后像素和所述边信息输入至基于神经网络的所述离线滤波模型，以输出所述滤波后像素。
在本申请的一些实施例中，所述滤波器还包括训练部分14和写入部分15；
所述第一训练部分14，被配置为所述将所述待滤波像素信息和所述边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素之后，基于所述滤波后像素，对所述滤波器进行在线训练，得到在线滤波参数。
所述第一滤波部分13，还被配置为基于所述在线滤波参数训练所述滤波器的在线部分后，将后续待滤波像素信息和所述边信息输入更新后滤波器中进行滤波，得到后续滤波后像素；
所述第一写入部分15，被配置为将所述在线滤波参数写入视频码流。
在本申请的一些实施例中，所述离线滤波模型为离线训练的滤波器；其中，所述离线滤波模型还包括离线训练的参数。
所述在线滤波模型为在线训练的滤波器；其中，所述在线滤波模型包括在线训练的参数。
在本申请的一些实施例中，基于神经网络的所述滤波器适用于后处理滤波、环内滤波、预处理滤波和预测过程。
在本申请的一些实施例中，基于神经网络的所述滤波器适用于后处理滤波时，设置在解码端；
基于神经网络的所述滤波器适用于环内处理滤波时，设置在解码端和编码端；
基于神经网络的所述滤波器适用于预处理滤波时，设置在编码端。
本申请实施例还提供了一种滤波装置，用于对视频进行编码，包括：
第二确定部分20，被配置为确定自适应滤波器的滤波参数；
第二滤波部分21，被配置为根据所述滤波参数和边信息，使用所述自适应滤波器对输入像素进行滤波，得到滤波后像素；
所述第二确定部分20，还被配置为确定第一滤波参数，其中，所述第一滤波参数是需要进行编码的、所述滤波参数中的部分滤波参数；
第二写入部分22，被配置为对所述第一滤波参数进行编码，写入视频码流。
在本申请的一些实施例中，自适应滤波器是一个神经网络滤波器。
在本申请的一些实施例中，自适应滤波器是第一神经网络滤波器和第二神经网络滤波器的级联滤波器。
在本申请的一些实施例中，所述第二确定部分20，还被配置为使用离线训练确定神经网络滤波器的第二滤波参数，其中，第二滤波参数是神经网络滤波器的全部参 数；使用在线训练确定神经网络滤波器的第三滤波参数；其中，第二滤波参数是神经网络滤波器的部分参数；使用第三滤波参数更新第二滤波参数中对应的滤波参数，将更新后的第二滤波参数作为神经网络滤波器的全部滤波参数。
在本申请的一些实施例中，所述第二确定部分20，还被配置为将第三滤波参数作为第一滤波参数。
在本申请的一些实施例中，所述第二确定部分20，还被配置为：使用离线训练确定第一神经网络滤波器的全部滤波参数；使用在线训练确定第二神经网络滤波器的第四滤波参数，其中，第四滤波参数是第二神经网络滤波器的全部参数。
在本申请的一些实施例中，所述第二确定部分20，还被配置为将第四滤波参数作为第一滤波参数。
在本申请的一些实施例中，所述第二确定部分20，还被配置为使用离线训练确定第一神经网络滤波器的第五滤波参数，其中，第五滤波参数是第一神经网络滤波器的全部参数；使用在线训练确定神经网络滤波器的第六滤波参数，其中，第六滤波参数是第一神经网络滤波器的部分参数；使用第六滤波参数更新第五滤波参数中对应的滤波参数，将更新后的第五滤波参数作为第一神经网络滤波器的滤波参数；在对视频或图像编码的过程中，使用在线训练确定第二神经网络滤波器的第七滤波参数，其中，第七滤波参数是第二神经网络滤波器的全部参数。
在本申请的一些实施例中，所述第二确定部分20，还被配置为将第六滤波参数和第七滤波参数作为第一滤波参数。
在本申请的一些实施例中，其中，离线训练是在对视频或图像开始编码之前，使用一个或多个图像对神经网络滤波器进行训练的过程；
在线训练是在对视频或图像编码的过程中，使用待编码视频序列中的一个或多个图像对神经网络滤波器进行训练的过程。
在本申请的一些实施例中，自适应滤波器是对视频进行编码中使用的预处理滤波器，或者环内滤波器。
可以理解的是，滤波装置在编码端时，确定自适应滤波器的滤波参数；根据滤波参数和边信息，使用自适应滤波器对输入像素进行滤波，得到滤波后像素；确定第一滤波参数，其中，第一滤波参数是需要进行编码的、滤波参数中的部分滤波参数(在线滤波参数)；对第一滤波参数进行编码，写入视频码流。由于滤波器由在线滤波部分结合离线滤波部分得到，这样在滤波时即可以使用离线滤波部分，而适用于很广的范围的视频的过滤，同时还可以保证在线滤波部分的一些模型的更新，避免滤波性能的退化，即提高滤波装置的滤波性能，且适用范围广。
本申请实施例还提供了一种滤波装置，用于对视频码流进行解码，包括：
第三确定部分30，被配置为解析视频码流，确定自适应滤波器的第一滤波参数，其中，所述第一滤波参数是所述自适应滤波器的全部滤波参数中的部分滤波参数；及根据所述第一滤波参数，确定所述自适应滤波器的全部滤波参数；
第三滤波部分31，被配置为根据所述全部滤波参数和边信息，使用所述自适应滤波器对输入像素进行滤波，得到滤波后像素。
在本申请的一些实施例中，自适应滤波器是一个神经网络滤波器。
在本申请的一些实施例中，自适应滤波器是两个或多个级联的、不同类型的神经网络滤波器。
在本申请的一些实施例中，自适应滤波器是第一神经网络滤波器和第二神经网络滤波器的级联滤波器。
在本申请的一些实施例中，所述第三确定部分30，还被配置为确定神经网络滤 波器的第二滤波参数，其中，第二滤波参数是神经网络滤波器的全部参数；使用第一滤波参数更新第二滤波参数中对应的滤波参数，将更新后的第二滤波参数作为神经网络滤波器的全部滤波参数。
在本申请的一些实施例中，所述第三确定部分30，还被配置为使用离线训练确定神经网络滤波器的第二滤波参数；或者，在解码视频码流之前，获得第二滤波参数。
在本申请的一些实施例中，所述第三确定部分30，还被配置为确定第一神经网络滤波器的全部滤波参数；将第一滤波参数作为第二神经网络滤波器的全部滤波参数。
在本申请的一些实施例中，所述第三确定部分30，还被配置为使用离线训练确定第一神经网络滤波器的全部滤波参数；或者，在解码视频码流之前，获得第一神经网络滤波器的全部滤波参数。
在本申请的一些实施例中，所述第三确定部分30，还被配置为确定第一神经网络滤波器的第五滤波参数；其中，第五滤波参数是第一神经网络滤波器的全部参数；将第一滤波参数中的部分参数作为第六滤波参数，其中，第六滤波参数是第一神经网络滤波器的部分参数使用第六滤波参数更新第五滤波参数中对应的滤波参数，将更新后的第五滤波参数作为第一神经网络滤波器的全部滤波参数；将第一滤波参数中的另一部分参数作为第二神经网络滤波器的全部参数。
在本申请的一些实施例中，所述第三确定部分30，还被配置为使用离线训练确定第五滤波参数；或者，在解码视频码流之前，获得第五滤波参数。
在本申请的一些实施例中，离线训练是在解析视频码流之前，使用一个或多个图像对神经网络滤波器进行训练的过程。
在本申请的一些实施例中，自适应滤波器是对视频码流进行解码中使用的环内滤波器或者后处理滤波器。
可以理解的是，滤波装置在解码端时，通过解析视频码流，确定自适应滤波器的第一滤波参数，第一滤波参数是自适应滤波器的全部滤波参数中的部分滤波参数(在线滤波参数)；及根据第一滤波参数，确定自适应滤波器的全部滤波参数(在线滤波参数和离线滤波参数)；根据全部滤波参数和边信息，使用自适应滤波器对输入像素进行滤波，得到滤波后像素，由于滤波器由在线滤波部分结合离线滤波部分得到，这样在滤波时即可以使用离线滤波部分，而适用于很广的范围的视频的过滤，同时还可以保证在线滤波部分的一些模型的更新，避免滤波性能的退化，即提高滤波装置的滤波性能，且适用范围广。
可以理解地，在本实施例中，“单元”可以是部分电路、部分处理器、部分程序或软件等等，当然也可以是模块，还可以是非模块化的。而且在本实施例中的各组成部分可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。
所述集成的单元如果以软件功能模块的形式实现并非作为独立的产品进行销售或使用时，可以存储在一个计算机可读取存储介质中，基于这样的理解，本实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或processor(处理器)执行本实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
图12为本申请实施例提供的一种可选的滤波装置的结构示意图，本申请实施例提供了一种滤波装置，包括：
存储器17，用于存储可执行指令；
处理器16，用于执行所述存储器17中存储的可执行指令时，实现本申请实施例提供的滤波方法。
需要说明的是，实际应用时，终端中的各个组件通过通信总线18耦合在一起。可理解，通信总线18用于实现这些组件之间的连接通信。通信总线18除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图12中将各种总线都标为通信总线18。
本申请实施例提供了一种计算机存储介质，存储有可执行指令，用于引起处理器执行时，实现本申请实施例提供的滤波方法。
可以理解，本申请实施例中的存储器可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable ROM，PROM)、可擦除可编程只读存储器(Erasable PROM，EPROM)、电可擦除可编程只读存储器(Electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(Static RAM，SRAM)、动态随机存取存储器(Dynamic RAM，DRAM)、同步动态随机存取存储器(Synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SDRAM，DDRSDRAM)、增强型同步动态随机存取存储器(Enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(Synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM，DRRAM)。本文描述的***和方法的存储器旨在包括但不限于这些和任意其它适合类型的存储器。
而处理器可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器可以是通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的信息，结合其硬件完成上述方法的步骤。
可以理解的是，本文描述的这些实施例可以用硬件、软件、固件、中间件、微码或其组合来实现。对于硬件实现，处理单元可以实现在一个或多个专用集成电路(Application Specific Integrated Circuits，ASIC)、数字信号处理器(Digital Signal Processing，DSP)、数字信号处理设备(DSP Device，DSPD)、可编程逻辑设备(Programmable Logic Device，PLD)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)、通用处理器、控制器、微控制器、微处理器、用于执行本申请所述功能的其它电子单元或其组合中。
对于软件实现，可通过执行本文所述功能的模块(例如过程、函数等)来实现本文所 述的技术。软件代码可存储在存储器中并通过处理器执行。存储器可以在处理器中或在处理器外部实现。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端(可以是手机、计算机、服务器、或者网络设备等)执行本申请各个实施例所述的方法。
上面结合附图对本申请的实施例进行了描述，但是本申请并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本申请的启示下，在不脱离本申请宗旨和权利要求所保护的范围情况下，还可做出很多形式，这些均属于本申请的保护之内。
工业实用性
在本申请实施例中，滤波装置针对待滤波视频帧的待滤波像素信息；确定出待滤波视频帧的边信息；将边信息和待滤波像素信息输入滤波器进行滤波，输出滤波后像素，由于滤波器由在线滤波部分结合离线滤波部分得到，这样在滤波时即可以使用离线滤波部分，而适用于很广的范围的视频的过滤，同时还可以保证在线滤波部分的一些模型的更新，避免滤波性能的退化，即提高滤波装置的滤波性能，且适用范围广。

Claims (43)

  1. 一种滤波方法，包括：
    获取待滤波像素信息；
    确定边信息；
    将所述待滤波像素信息和所述边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素，其中，所述滤波器由在线滤波部分结合离线滤波部分得到。
  2. 根据权利要求1所述的方法，其中，
    所述滤波器由在线滤波模型与离线滤波模型级联形成；或者，
    所述滤波器由在线滤波模型与离线滤波模型级联形成；其中，所述离线滤波模型中存在部分在线训练的参数。
  3. 根据权利要求1所述的方法，其中，
    所述滤波器由离线滤波模型形成，其中，离线滤波模型中存在部分在线训练的参数；
    其中，所述将所述待滤波像素信息和所述边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素，包括：
    将所述待滤波像素信息和所述边信息输入至基于神经网络的所述离线滤波模型，以输出所述滤波后像素。
  4. 根据权利要求2所述的方法，其中，所述将所述待滤波像素信息和所述边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素，包括：
    将所述待滤波像素信息和所述边信息输入至基于神经网络的所述离线滤波模型，以输出中间滤波后像素；将所述中间滤波后像素和所述边信息输入至基于神经网络的所述在线滤波模型，以输出所述滤波后像素；或者，
    将所述待滤波像素信息和所述边信息输入至基于神经网络的所述在线滤波模型，以输出中间滤波后像素；将所述中间滤波后像素和所述边信息输入至基于神经网络的所述离线滤波模型，以输出所述滤波后像素。
  5. 根据权利要求1至4任一项所述的方法，其中，所述将所述待滤波像素信息和所述边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素之后，所述方法还包括：
    基于所述滤波后像素，对所述滤波器进行在线训练，得到在线滤波参数；
    基于所述在线滤波参数训练所述滤波器的在线部分后，将后续待滤波像素信息和所述边信息输入更新后滤波器中进行滤波，得到后续滤波后像素；
    将所述在线滤波参数写入视频码流。
  6. 根据权利要求1至4任一项所述的方法，其中，
    所述离线滤波模型为离线训练的滤波器；其中，所述离线滤波模型还包括离线训练的参数。
    所述在线滤波模型为在线训练的滤波器；其中，所述在线滤波模型包括在线训练的参数。
  7. 根据权利要求6所述的方法，其中，
    基于神经网络的所述滤波器适用于后处理滤波、环内滤波、预处理滤波和预测过程。
  8. 根据权利要求7所述的方法，其中，
    基于神经网络的所述滤波器适用于后处理滤波时，设置在解码端；
    基于神经网络的所述滤波器适用于环内处理滤波时，设置在解码端和编码端；
    基于神经网络的所述滤波器适用于预处理滤波时，设置在编码端。
  9. 一种滤波方法，用于对视频进行编码，包括：
    确定自适应滤波器的滤波参数；
    根据所述滤波参数和边信息，使用所述自适应滤波器对输入像素进行滤波，得到滤波后像素；
    确定第一滤波参数，其中，所述第一滤波参数是需要进行编码的、所述滤波参数中的部分滤波参数；
    对所述第一滤波参数进行编码，写入视频码流。
  10. 根据权利要求9所述的方法，其中，所述自适应滤波器，包括：
    所述自适应滤波器是一个神经网络滤波器。
  11. 根据权利要求9所述的方法，其中，所述自适应滤波器，包括：
    所述自适应滤波器是第一神经网络滤波器和第二神经网络滤波器的级联滤波器。
  12. 根据权利要求9所述的方法，其中，所述确定自适应滤波器的滤波参数，包括：
    使用离线训练确定所述神经网络滤波器的第二滤波参数，其中，所述第二滤波参数是所述神经网络滤波器的全部参数；
    使用在线训练确定所述神经网络滤波器的第三滤波参数；其中，所述第二滤波参数是所述神经网络滤波器的部分参数；
    使用所述第三滤波参数更新所述第二滤波参数中对应的滤波参数，将更新后的所述第二滤波参数作为所述神经网络滤波器的全部滤波参数。
  13. 根据权利要求12所述的方法，其中，所述确定第一滤波参数，包括：
    将所述第三滤波参数作为所述第一滤波参数。
  14. 根据权利要求11所述的方法，其中，所述确定自适应滤波器的滤波参数，包括：
    使用离线训练确定所述第一神经网络滤波器的全部滤波参数；
    使用在线训练确定所述第二神经网络滤波器的第四滤波参数，其中，所述第四滤波参数是所述第二神经网络滤波器的全部参数。
  15. 根据权利要求14所述的方法，其中，所述确定第一滤波参数，包括：
    将所述第四滤波参数作为所述第一滤波参数。
  16. 根据权利要求11所述的方法，其中，所述确定自适应滤波器的滤波参数，包括：
    使用离线训练确定所述第一神经网络滤波器的第五滤波参数，其中，所述第五滤波参数是所述第一神经网络滤波器的全部参数；
    使用在线训练确定所述神经网络滤波器的第六滤波参数，其中，所述第六滤波参数是所述第一神经网络滤波器的部分参数；
    使用所述第六滤波参数更新所述第五滤波参数中对应的滤波参数，将更新后的所述第五滤波参数作为所述第一神经网络滤波器的滤波参数；
    在对所述视频或图像编码的过程中，使用在线训练确定所述第二神经网络滤波器的第七滤波参数，其中，所述第七滤波参数是所述第二神经网络滤波器的全部参数。
  17. 根据权利要求16所述的方法，其中，所述确定第一滤波参数，包括：
    将所述第六滤波参数和所述第七滤波参数作为所述第一滤波参数。
  18. 根据权利要求12至17中任一项所述的方法，其中，还包括：
    所述离线训练是在对所述视频或图像开始编码之前，使用一个或多个图像对神经 网络滤波器进行训练的过程；
    所述在线训练是在对所述视频或图像编码的过程中，使用待编码视频序列中的一个或多个图像对神经网络滤波器进行训练的过程。
  19. 根据权利要求9至17中任一项所述的方法，其特征在于，还包括：
    所述自适应滤波器是所述对视频进行编码中使用的预处理滤波器，或者环内滤波器。
  20. 一种滤波方法，用于对视频码流进行解码，包括：
    解析视频码流，确定自适应滤波器的第一滤波参数，其中，所述第一滤波参数是所述自适应滤波器的全部滤波参数中的部分滤波参数；
    根据所述第一滤波参数，确定所述自适应滤波器的全部滤波参数；
    根据所述全部滤波参数和边信息，使用所述自适应滤波器对输入像素进行滤波，得到滤波后像素。
  21. 根据权利要求20所述的方法，其中，所述自适应滤波器，包括：
    所述自适应滤波器是一个神经网络滤波器。
  22. 根据权利要求20所述的方法，其中，所述自适应滤波器，包括：
    所述自适应滤波器是两个或多个级联的、不同类型的神经网络滤波器。
  23. 根据权利要求22所述的方法，其中，所述自适应滤波器，包括：
    所述自适应滤波器是第一神经网络滤波器和第二神经网络滤波器的级联滤波器。
  24. 根据权利要求21所述的方法，其中，所述确定自适应滤波器的全部滤波参数，包括：
    确定所述神经网络滤波器的第二滤波参数，其中，所述第二滤波参数是所述神经网络滤波器的全部参数；
    使用所述第一滤波参数更新所述第二滤波参数中对应的滤波参数，将更新后的所述第二滤波参数作为所述神经网络滤波器的全部滤波参数。
  25. 根据权利要求24所述的方法，其中，确定所述神经网络滤波器的第二滤波参数，包括：
    使用离线训练确定所述神经网络滤波器的第二滤波参数；或者，
    在解码所述视频码流之前，获得所述第二滤波参数。
  26. 根据权利要求23所述的方法，其中，所述确定自适应滤波器的全部滤波参数，包括：
    确定所述第一神经网络滤波器的全部滤波参数；
    将所述第一滤波参数作为所述第二神经网络滤波器的全部滤波参数。
  27. 根据权利要求26所述的方法，其中，所述确定所述第一神经网络滤波器的全部滤波参数，包括：
    使用离线训练确定所述第一神经网络滤波器的全部滤波参数；或者，
    在解码所述视频码流之前，获得所述第一神经网络滤波器的全部滤波参数。
  28. 根据权利要求23所述的方法，其中，所述确定自适应滤波器的全部滤波参数，包括：
    确定所述第一神经网络滤波器的第五滤波参数；其中，所述第五滤波参数是所述第一神经网络滤波器的全部参数；
    将所述第一滤波参数中的部分参数作为第六滤波参数，其中，所述第六滤波参数是所述第一神经网络滤波器的部分参数；
    使用所述第六滤波参数更新所述第五滤波参数中对应的滤波参数，将更新后的所述第五滤波参数作为所述第一神经网络滤波器的全部滤波参数；
    将所述第一滤波参数中的另一部分参数作为所述第二神经网络滤波器的全部参数。
  29. 根据权利要求28所述的方法，其中，所述确定所述第一神经网络滤波器的第五滤波参数，包括：
    使用离线训练确定所述第五滤波参数；或者，
    在解码所述视频码流之前，获得所述第五滤波参数。
  30. 根据权利要求24至29中任一项所述的方法，其特征在于，还包括：
    所述离线训练是在解析所述视频码流之前，使用一个或多个图像对神经网络滤波器进行训练的过程。
  31. 根据权利要求24至29中任一项所述的方法，其特征在于，还包括：
    所述自适应滤波器是所述对视频码流进行解码中使用的环内滤波器或者后处理滤波器。
  32. 一种滤波装置，包括：
    第一获取部分，被配置为获取待滤波像素信息；
    第一确定部分，被配置为确定边信息；
    第一滤波部分，被配置将所述待滤波像素信息和所述边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素，其中，所述滤波器由在线滤波部分结合离线滤波部分得到。
  33. 根据权利要求32所述的滤波装置，其中，
    所述滤波器由在线滤波模型与离线滤波模型级联形成；或者，
    所述滤波器由在线滤波模型与离线滤波模型级联形成；其中，所述离线滤波模型中存在部分在线训练的参数。
  34. 根据权利要求32所述的滤波装置，其中，
    所述滤波器由离线滤波模型形成，其中，离线滤波模型中存在部分在线训练的参数；
    所述第一滤波部分，还被配置为将所述待滤波像素信息和所述边信息输入至基于神经网络的所述离线滤波模型，以输出所述滤波后像素。
  35. 根据权利要求33所述的滤波装置，其中，
    所述第一滤波部分，还被配置为将所述待滤波像素信息和所述边信息输入至基于神经网络的所述离线滤波模型，以输出中间滤波后像素；将所述中间滤波后像素和所述边信息输入至基于神经网络的所述在线滤波模型，以输出所述滤波后像素；或者，将所述待滤波像素信息和所述边信息输入至基于神经网络的所述在线滤波模型，以输出中间滤波后像素；将所述中间滤波后像素和所述边信息输入至基于神经网络的所述离线滤波模型，以输出所述滤波后像素。
  36. 根据权利要求32至35任一项所述的滤波装置，其中，所述滤波器还包括训练部分和写入部分；
    所述第一训练部分，被配置为所述将所述待滤波像素信息和所述边信息输入至基于神经网络的滤波器中，以输出得到滤波后像素之后，基于所述滤波后像素，对所述滤波器进行在线训练，得到在线滤波参数。
    所述第一滤波部分，还被配置为基于所述在线滤波参数训练所述滤波器的在线部分后，将后续待滤波像素信息和所述边信息输入更新后滤波器中进行滤波，得到后续滤波后像素；
    所述第一写入部分，被配置为将所述在线滤波参数写入视频码流。
  37. 根据权利要求32至35任一项所述的滤波装置，其中，
    所述离线滤波模型为离线训练的滤波器；其中，所述离线滤波模型还包括离线训练的参数。
    所述在线滤波模型为在线训练的滤波器；其中，所述在线滤波模型包括在线训练的参数。
  38. 根据权利要求37所述的滤波装置，其中，
    基于神经网络的所述滤波器适用于后处理滤波、环内滤波、预处理滤波和预测过程。
  39. 根据权利要求38所述的滤波装置，其中，
    基于神经网络的所述滤波器适用于后处理滤波时，设置在解码端；
    基于神经网络的所述滤波器适用于环内处理滤波时，设置在解码端和编码端；
    基于神经网络的所述滤波器适用于预处理滤波时，设置在编码端。
  40. 一种滤波装置，用于对视频进行编码，包括：
    第二确定部分，被配置为确定自适应滤波器的滤波参数；
    第二滤波部分，被配置为根据所述滤波参数和边信息，使用所述自适应滤波器对输入像素进行滤波，得到滤波后像素；
    所述第二确定部分，还被配置为确定第一滤波参数，其中，所述第一滤波参数是需要进行编码的、所述滤波参数中的部分滤波参数；
    第二写入部分，被配置为对所述第一滤波参数进行编码，写入视频码流。
  41. 一种滤波装置，用于对视频码流进行解码，包括：
    第三确定部分，被配置为解析视频码流，确定自适应滤波器的第一滤波参数，其中，所述第一滤波参数是所述自适应滤波器的全部滤波参数中的部分滤波参数；及根据所述第一滤波参数，确定所述自适应滤波器的全部滤波参数；
    第三滤波部分，被配置为根据所述全部滤波参数和边信息，使用所述自适应滤波器对输入像素进行滤波，得到滤波后像素。
  42. 一种滤波装置，包括：
    存储器，用于存储可执行指令；
    处理器，用于执行所述存储器中存储的可执行指令时，实现权利要求1至8任一项所述的方法，或者9至19任一项所述的方法，或者20至31任一项所述的方法。
  43. 一种计算机存储介质，存储有可执行指令，用于引起处理器执行时，实现权利要求1至8任一项所述的方法，或者9至19任一项所述的方法，或者20至31任一项所述的方法。
PCT/CN2019/105799 2019-03-24 2019-09-12 滤波方法及装置、计算机存储介质 WO2020192034A1 (zh)

Priority Applications (5)

Application Number Priority Date Filing Date Title
KR1020217032829A KR20210134397A (ko) 2019-03-24 2019-09-12 필터링 방법 및 장치, 컴퓨터 저장 매체
EP19921828.0A EP3941066A4 (en) 2019-03-24 2019-09-12 FILTER METHOD AND APPARATUS AND COMPUTER STORAGE MEDIUM
CN201980094265.3A CN113574897A (zh) 2019-03-24 2019-09-12 滤波方法及装置、计算机存储介质
JP2021555871A JP2022525235A (ja) 2019-03-24 2019-09-12 フィルタリング方法及び装置、コンピュータ記憶媒体
US17/475,237 US11985313B2 (en) 2019-03-24 2021-09-14 Filtering method and apparatus, and computer storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201962822949P 2019-03-24 2019-03-24
US62/822,949 2019-03-24

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/475,237 Continuation US11985313B2 (en) 2019-03-24 2021-09-14 Filtering method and apparatus, and computer storage medium

Publications (1)

Publication Number Publication Date
WO2020192034A1 true WO2020192034A1 (zh) 2020-10-01

Family

ID=72609590

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/105799 WO2020192034A1 (zh) 2019-03-24 2019-09-12 滤波方法及装置、计算机存储介质

Country Status (6)

Country Link
US (1) US11985313B2 (zh)
EP (1) EP3941066A4 (zh)
JP (1) JP2022525235A (zh)
KR (1) KR20210134397A (zh)
CN (1) CN113574897A (zh)
WO (1) WO2020192034A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115956363A (zh) * 2021-05-27 2023-04-11 腾讯美国有限责任公司 用于后滤波的内容自适应在线训练方法及装置
CN116320410A (zh) * 2021-12-21 2023-06-23 腾讯科技（深圳）有限公司 一种数据处理方法、装置、设备以及可读存储介质
JP7522860B2 (ja) 2021-04-12 2024-07-25 テンセント・アメリカ・エルエルシー ビデオストリームにおけるニューラル・ネットワーク・トポロジ、パラメータ、および処理情報をシグナリングするための技術

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021170901A1 (en) * 2020-02-24 2021-09-02 Nokia Technologies Oy A method, an apparatus and a computer program product for video encoding and video decoding
WO2024019343A1 (ko) * 2022-07-20 2024-01-25 현대자동차주식회사 다양한 잡음 및 특성에 적응적인 비디오 인루프 필터
WO2024073145A1 (en) * 2022-09-30 2024-04-04 Beijing Dajia Internet Information Technology Co., Ltd. Methods and devices for adaptive loop filtering and cross-component adaptive loop filter

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101841701A (zh) * 2009-03-20 2010-09-22 华为技术有限公司 基于宏块对的编解码方法及装置
WO2017036370A1 (en) * 2015-09-03 2017-03-09 Mediatek Inc. Method and apparatus of neural network based processing in video coding
CN108184129A (zh) * 2017-12-11 2018-06-19 北京大学 一种视频编解码方法、装置及用于图像滤波的神经网络
CN108932697A (zh) * 2017-05-26 2018-12-04 杭州海康威视数字技术股份有限公司 一种失真图像的去失真方法、装置及电子设备

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8572184B1 (en) * 2007-10-04 2013-10-29 Bitdefender IPR Management Ltd. Systems and methods for dynamically integrating heterogeneous anti-spam filters
US9762906B2 (en) * 2013-02-18 2017-09-12 Mediatek Inc. Method and apparatus for video decoding using multi-core processor
GB2539845B (en) 2015-02-19 2017-07-12 Magic Pony Tech Ltd Offline training of hierarchical algorithms
US10531111B2 (en) * 2015-11-06 2020-01-07 Microsoft Technology Licensing, Llc Flexible reference picture management for video encoding and decoding
US11132619B1 (en) * 2017-02-24 2021-09-28 Cadence Design Systems, Inc. Filtering in trainable networks
US20180293486A1 (en) * 2017-04-07 2018-10-11 Tenstorrent Inc. Conditional graph execution based on prior simplified graph execution
MX2019014443A (es) * 2017-05-31 2020-02-10 Interdigital Vc Holdings Inc Un metodo y un dispositivo para codificacion y decodificacion de imagenes.
CN108134932B (zh) 2018-01-11 2021-03-30 上海交通大学 基于卷积神经网络的视频编解码环路内滤波实现方法及***

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101841701A (zh) * 2009-03-20 2010-09-22 华为技术有限公司 基于宏块对的编解码方法及装置
WO2017036370A1 (en) * 2015-09-03 2017-03-09 Mediatek Inc. Method and apparatus of neural network based processing in video coding
CN108932697A (zh) * 2017-05-26 2018-12-04 杭州海康威视数字技术股份有限公司 一种失真图像的去失真方法、装置及电子设备
CN108184129A (zh) * 2017-12-11 2018-06-19 北京大学 一种视频编解码方法、装置及用于图像滤波的神经网络

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7522860B2 (ja) 2021-04-12 2024-07-25 テンセント・アメリカ・エルエルシー ビデオストリームにおけるニューラル・ネットワーク・トポロジ、パラメータ、および処理情報をシグナリングするための技術
CN115956363A (zh) * 2021-05-27 2023-04-11 腾讯美国有限责任公司 用于后滤波的内容自适应在线训练方法及装置
EP4128764A4 (en) * 2021-05-27 2023-09-06 Tencent America Llc CONTENT-ADAPTIVE ONLINE TRAINING METHOD AND DEVICE FOR POST-FILTERING
US11979565B2 (en) 2021-05-27 2024-05-07 Tencent America LLC Content-adaptive online training method and apparatus for post-filtering
CN116320410A (zh) * 2021-12-21 2023-06-23 腾讯科技（深圳）有限公司 一种数据处理方法、装置、设备以及可读存储介质

Also Published As

Publication number Publication date
EP3941066A1 (en) 2022-01-19
KR20210134397A (ko) 2021-11-09
JP2022525235A (ja) 2022-05-11
US20220007015A1 (en) 2022-01-06
EP3941066A4 (en) 2022-06-22
US11985313B2 (en) 2024-05-14
CN113574897A (zh) 2021-10-29

Similar Documents

Publication Publication Date Title
WO2020192034A1 (zh) 滤波方法及装置、计算机存储介质
US11589041B2 (en) Method and apparatus of neural network based processing in video coding
US20210409783A1 (en) Loop filter implementation method and apparatus, and computer storage medium
US11627342B2 (en) Loop filtering implementation method and apparatus, and computer storage medium
US20220021905A1 (en) Filtering method and device, encoder and computer storage medium
WO2021134706A1 (zh) 环路滤波的方法与装置
CN111699686B (zh) 用于视频编解码的分组神经网络的方法以及装置
JP2024095842A (ja) 画像予測方法、エンコーダー、デコーダー及び記憶媒体
CN110619607B (zh) 图像去噪和包含图像去噪的图像编解码方法及装置
WO2021056216A1 (zh) 预测值的确定方法、编码器、解码器以及计算机存储介质
WO2020181554A1 (zh) 预测值的确定方法、解码器以及计算机存储介质
WO2022227082A1 (zh) 块划分方法、编码器、解码器以及计算机存储介质
KR20210139327A (ko) 화상 예측 방법, 인코더, 디코더 및 저장 매체
US20220007042A1 (en) Colour component prediction method, encoder, decoder, and computer storage medium
WO2023050439A1 (zh) 编解码方法、码流、编码器、解码器、存储介质和***
EP4224852A1 (en) Video encoding and decoding methods, encoder, decoder, and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19921828

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021555871

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20217032829

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2019921828

Country of ref document: EP

Effective date: 20211015