CN110087087B

CN110087087B - VVC inter-frame coding unit prediction mode early decision and block division early termination method

Info

Publication number: CN110087087B
Application number: CN201910279316.1A
Authority: CN
Inventors: 张冬冬; 钱鹏飞
Original assignee: Tongji University
Current assignee: Tongji University
Priority date: 2019-04-09
Filing date: 2019-04-09
Publication date: 2023-05-12
Anticipated expiration: 2039-04-09
Also published as: CN110087087A

Abstract

The method for deciding the prediction mode of the VVC inter-frame coding unit in advance and stopping the block division in advance is characterized in that motion changes among video frames are analyzed, and the coding cost of the inter-frame prediction mode in the optimal mode and the coding cost of the inter-frame prediction mode in the subsequent coding depth are compared through comparing the rate distortion cost among all the prediction modes in the inter-frame coding unit with the coding depth of 1, so that the coding unit meeting the conditions directly skips the check of the coding cost of the inter-frame prediction mode, and the calculation complexity is reduced. In addition, the optimal dividing size of the current coding unit is predicted through the ratio relation between the current coding depth of the coding unit and the selected optimal prediction mode and the rate distortion cost among all the prediction modes, and the recursive dividing process with extremely high calculating complexity is avoided through the early termination of the dividing of the coding unit, so that the calculating complexity of an encoder is reduced, the time consumption of the encoder can be effectively reduced on the premise that the video coding quality is not influenced, and the coding efficiency is improved.

Description

VVC inter-frame coding unit prediction mode early decision and block division early termination method

Technical Field

The invention relates to the field of multifunctional video coding (Versatile Video Coding, VVC), in particular to a prediction mode early decision and block partition early termination technology of a VVC inter-frame coding unit.

Background

In recent years, with the development of VR technology and the increasing demand of consumers for viewing 360-degree panoramic video, there has been an explosive growth in the internet about the content of 360-degree panoramic video. The most common 360-degree panoramic video format is greatly different from the traditional two-dimensional plane video. Taking the most popular equidistant projection (EquiRectangular Projection, ERP) format as an example, due to the special projection mode, the video content is deformed to a certain extent, and the deformation is more obvious when the video content is closer to the upper part and the lower part of the image. In addition, in order to bring a more realistic and immersive viewing experience to the viewer, the resolution and frame rate of a general 360-degree panoramic video are high, which also presents a great challenge to the current video coding algorithms. So that the international telecommunication union's video coding expert group (Video Coding Experts Group, VCEG) and moving picture expert group (Moving Picture Experts Group, MPEG) jointly developed the latest video coding standard, the versatile video coding standard (VVC), in order to be able to better process such videos. It is an improvement over the previous generation video coding standard HEVC in many respects. In terms of coding structure: a quadtree-binary division model is introduced on the basis of the quadtree division model in HEVC, so that the method can adapt to more complex image textures and improve the coding efficiency. In addition, in inter prediction: many new techniques have also been developed and introduced, including local illumination compensation (Local Illumination Compensation, LIC), two new mere modes: affine Merge and FRUC Merge. Meanwhile, quick algorithms are added, for example, when the intra-frame prediction detection is carried out on the inter-frame video frames, the intra-frame prediction detection can be directly skipped when the skip condition is met, and the coding speed is increased. Due to the change of the coding structure and the introduction of new techniques in intra-and inter-frame coding, on the one hand the coding efficiency is improved, but the computational complexity of the encoder is also increased. So if it is desired to apply VVC to the actual traffic scenario, all that is needed is to optimize its encoder.

The inter-frame coding flow of the general test platform JEM7.0 of the VVC is as follows: firstly, starting from depth 0, checking prediction modes of the current maximum coding unit and coding units with different coding depths layer by layer, and selecting the optimal prediction mode of each coding unit under each coding depth. And then comparing all the coded depths based on the rate distortion cost to select a final coding unit division mode. It is therefore necessary to try various inter prediction modes and different coding unit partitions, which gives the encoder a very high computational complexity. At present, some researchers optimize the inter-frame coding complexity of VVC and HEVC, and achieve a certain effect. An algorithm for fast decision on coding unit partitioning is proposed as in document 1 (K.Choi, S.H.Park, and e.s. jang, coding tree pruning based CU early termination, JCTVC-F092,2011.) which, through statistical analysis, finds that if the current optimal prediction mode is SKIP (a special Merge mode) during inter prediction coding, the probability of partitioning the current coding unit into optimal is over 95%. Therefore, the document judges whether the current coding unit division can be terminated in advance on the condition that whether the optimal prediction mode is SKIP or not, so that the optimal division size of the coding unit is rapidly determined. Document 2 (X.L.Shen, Y.Lu, CU splitting early termination based on weighted SVM [ J ], eurasip Journal on Image & Video Processing,2013 (1): 1-11,2013.) then, based on a support vector machine (Support Vector Machine, SVM), a classifier is constructed from the coding unit pixel gradient information, rate distortion cost, block coding flags, etc., to decide whether the current coding unit continues to recursively divide down. Document 3 (M.Xu, T.Li, Z.Wang, X.Deng, R.Yang and z. Guard, reducing Complexity of HEVC: A Deep Learning Approach, in IEEE Transactions on Image Processing, vol.27, no.10, pp.5044-5059, oct.2018.) then uses convolutional neural networks (Convolutional Neural Networks, CNN) and Long Short Term Memory (LSTM) to divide depth correlations in the time domain by analyzing coding units based on HEVC. Because the closer in the time domain, the more likely its coding unit partition depth is. The document predicts the division of the current coding unit in advance by constructing an inter-frame training set, inputting the coding unit residual error into the CNN, and utilizing the correlation of the LSTM learning previous and subsequent frames on the coding unit division depth.

Disclosure of Invention

The invention aims to provide a VVC Inter-frame coding unit prediction mode early decision and block division early termination method, which is characterized in that motion changes among video frames are analyzed, and the coding cost of Inter-frame prediction modes (Inter2Nx2N) in an optimal mode and a subsequent coding depth selected by comparing the rate distortion cost among all prediction modes in an Inter-frame coding unit with a coding depth of 1 is compared, so that the check of the coding cost of the Inter-frame prediction modes is directly skipped for a coding unit meeting the conditions, thereby reducing the computational complexity. In addition, the optimal dividing size of the current coding unit is predicted through the ratio relation between the current coding depth of the coding unit and the selected optimal prediction mode and the rate distortion cost among all the prediction modes, and the recursive dividing process with extremely high calculating complexity is avoided through the early termination of the dividing of the coding unit, so that the calculating complexity of an encoder is reduced, the time consumption of the encoder can be effectively reduced on the premise that the video coding quality is hardly influenced, and the coding efficiency is improved.

The technical scheme of the invention is as follows:

(1) When the Inter coding unit prediction mode is selected, in the coding units with depth of 1, whether to skip the Inter prediction mode in the following coding units is determined according to the selected optimal prediction mode and the relation of the rate distortion cost between the current Merge prediction mode and the Inter prediction mode (Inter_2Nx2N).

(2) When the inter-frame coding unit block division decision is made, the relation between the final coding depth of the coding unit and the selected optimal prediction mode is firstly counted, and the counting result shows that the final coding depth and the selected optimal prediction mode have strong correlation. We therefore again base our ratio MI on the rate-distortion cost for each prediction mode _rate To assist in deciding whether or not the partitioning of the current coding unit needs to be terminated. MI (MI) _rate The calculation of (2) is shown in formula (1), wherein Merge _RDC Representing the rate distortion cost of the current coding unit using the Merge prediction mode, inter _RDC The representation is a rate distortion cost using Inter prediction mode (Inter 2N x 2N).

By adopting the scheme, the invention has the beneficial effects that:

1. the invention skips the checking of the inter-frame prediction mode in the coding unit meeting the condition by comparing the rate distortion cost of the inter-frame prediction mode under different coding depths, thereby removing redundant calculation in the mode search and reducing the calculation complexity.

2. The invention discovers that the ratio of the rate distortion cost between the coded depth and the optimal prediction mode of the coded depth in different types of video sequences has certain correlation by analyzing the correlation, thereby rapidly terminating the recursion division of the current coding unit.

3. The invention comprehensively considers the characteristics of 360-degree panoramic video, and can effectively reduce redundant calculation in the decision of the optimal division mode of the prediction mode searching and encoding unit, thereby improving the encoding efficiency of the VVC encoder on the premise of almost not losing the encoding quality.

Drawings

Fig. 1 is a case of inter-layer prediction mode rate distortion cost comparison at different coding depths.

Fig. 2 is a graph of the percentage of coding units selecting Merge as the optimal prediction mode at different coding depths.

Fig. 3 is the final experimental result and is compared with a standard algorithm.

Fig. 4 is an overall flowchart of the VVC inter-frame rapid algorithm of the present invention.

Detailed Description

The main idea of the invention is to determine whether to skip the checking of the inter prediction mode in the following coding depth by comparing the coding cost of the inter prediction mode in different coding depths by utilizing the motion change relation between video frames. In addition, the division of the current coding unit is terminated in advance by the ratio relation between the optimal prediction mode selected in the coding unit and the rate distortion cost of different prediction modes. The computational complexity of the VVC inter-frame coding is reduced through optimization of the two aspects. The test video sequences used in the embodiment of the invention are all 4K video (resolution: 3840×1920), the original video is in mp4 format, the test video sequences are used after being processed into YUV420 format, and the video is from (http:// zy.dmgeek.com/c/vrshipin).

The invention firstly counts the difference of rate distortion cost among coding layers when Merge is selected as an optimal prediction mode in a coding unit with coding depth of 1 and the inter prediction mode is used under each subsequent coding depth. Statistical results are shown in FIG. 1, where RDC _depth Depth=0,..5 represents the sum of rate distortion costs for all coding units of each layer using inter prediction mode at each coding depth in the current maximum coding unit (Largest Coding Unit, LCU). RDC (RDC) _skip Is a threshold for whether to skip a subsequent inter prediction mode. It can be seen that when the coding unit with coding depth 1 selects Merge as the optimal prediction mode, for four different test sequences, the ratio of the following inter prediction modes is only 0.72% on average, which is superior to RDC ₁ ×RDC _skip (0.923). That is, if the rate distortion cost of the current Merge prediction mode is less than 0.923 times that of the inter prediction mode in the coding unit having depth of 1, the finer motion search process in the subsequent coding unit does not hardly exceed the coding effect of the current Merge prediction mode.

Meanwhile, we also count the relation between the selected optimal prediction mode and the dividing depth of the coding unit under different coding depths, and the statistical result is shown in fig. 2. From the statistics it can be seen that: when the coding depth is 0, on average about 98.04% of the inter-coded units select Merge as the optimal prediction mode. And when the coded depth is 1, this ratio is 77.07%. This illustrates that the optimal prediction mode selected by the inter-coding unit has a greater relationship with whether it continues to recursively divide, and therefore the continued recursion division of the inter-coding unit can be terminated in advance accordingly.

Based on the technical scheme idea of the invention, the invention is further described with reference to a flow chart shown in fig. 4.

Step 1: general test platform JEM7.0 based on VVC. After starting to encode an LCU, the current encoded depth (initially 0, 1 plus each recursively divided encoded depth) is first determined. If the coding depth is 0, go to step 2, otherwise go to step 4.

Step 2: judging whether the optimal prediction mode of the current coding unit is Merge, if so, turning to step 3, otherwise turning to step 9.

Step 3: calculation of MI _rate If MI _rate ＜MI _Threshold (MI at coding depths of 0 and 1) _Threshold 0.63 and 0.95 respectively), the current coding unit is stopped and the partitioning down continues, otherwise step 9 is proceeded to.

Step 4: if the current coding depth is 1, go to step 5, otherwise go to step 7.

Step 5: rate distortion cost RDC for obtaining Merge prediction mode _merge Rate distortion cost RDC for inter prediction modes ₁ And set RDC _skip =0.923. Turning to step 6.

Step 6: if RDC _merge ＜RDC ₁ ×RDC _skip The variable skip inter is set to true. Otherwise, turning to the step 3.

Step 7: it is checked whether the variable skip inter is true. If true, the check for inter prediction mode in the current coded depth is skipped, otherwise step 8 is passed.

Step 8: proceed according to JEM7.0 standard procedure.

Step 9: continuing to carry out the coding according to JEM7.0 standard flow until the coding of the current LCU is finished.

Finally, the fast algorithm proposed in the present invention was validated based on JEM7.0 platform and compared to the standard algorithm. It can be seen that an average of 30.34% of the coding time can be saved without substantial loss of coding efficiency, with statistics see fig. 3.

The above description is only illustrative of the preferred embodiments of the invention and is not intended to limit the scope of the invention in any way. Any alterations or modifications of the invention, which are obvious to those skilled in the art based on the teachings disclosed above, are intended to be equally effective embodiments, and are intended to be within the scope of the appended claims.

Claims

1. A VVC inter-coding unit prediction mode early decision and block partition early termination method, characterized by comprising:

(1) When the inter-frame coding unit prediction mode is selected, in the coding unit with depth of 1, deciding whether to skip the inter-frame prediction mode in the subsequent coding unit according to the selected optimal prediction mode and the relation of the rate distortion cost between the current Merge prediction mode and the inter-frame prediction mode;

(2) When the inter-frame coding unit block division decision is made, firstly, the relation between the final coding depth of the coding unit and the selected optimal prediction mode is counted, and the counting result shows that the final coding depth and the selected optimal prediction mode have correlation; then based on the ratio MI of rate distortion cost for each prediction mode _rate To assist in deciding whether or not the partitioning of the current coding unit needs to be terminated;

MI _rate the calculation of (2) is as follows:

wherein, merge _RDC Representing the rate distortion cost of the current coding unit using the Merge prediction mode, inter _RDC Then the representation is a rate-distortion cost using inter prediction mode;

the method specifically comprises the following steps:

step 1: after starting to code one LCU, the universal test platform JEM7.0 based on the VVC firstly judges the current coding depth, and initially is 0, and the coding depth is divided once every recursion and added with 1, if the coding depth is 0, the step 2 is switched, otherwise, the step 4 is switched;

step 2: judging whether the optimal prediction mode of the current coding unit is a Merge prediction mode, if so, turning to step 3, otherwise turning to step 9;

step 3: calculation of MI _rate If MI _rate ＜MI _Threshold MI for coded depth 0 and 1 _Threshold 0.63 and 0.95 respectively, stopping the current coding unit to continue dividing downwards, otherwise turning to the step 9;

step 4: if the current coding depth is 1, turning to step 5, otherwise turning to step 7;

step 5: rate distortion cost RDC for obtaining Merge prediction mode _merge Rate distortion cost RDC for inter prediction modes ₁ And set RDC _skip =0.923, go to step 6;

step 6: if RDC _merge ＜RDC ₁ ×RDC _skip Setting a variable skip Inter to be true, otherwise turning to the step 3;

step 7: checking whether the variable skip inter is true, if true, skipping the checking of the inter prediction mode in the current coding depth, otherwise turning to step 8;

step 8: continuing to perform according to JEM7.0 standard flow;