AU2019232802A1 - Method, apparatus and system for encoding and decoding an Image frame from a bistream - Google Patents
Method, apparatus and system for encoding and decoding an Image frame from a bistream Download PDFInfo
- Publication number
- AU2019232802A1 AU2019232802A1 AU2019232802A AU2019232802A AU2019232802A1 AU 2019232802 A1 AU2019232802 A1 AU 2019232802A1 AU 2019232802 A AU2019232802 A AU 2019232802A AU 2019232802 A AU2019232802 A AU 2019232802A AU 2019232802 A1 AU2019232802 A1 AU 2019232802A1
- Authority
- AU
- Australia
- Prior art keywords
- transform coefficients
- kernel
- zero
- primary
- primary transform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims abstract description 101
- 230000015654 memory Effects 0.000 claims description 58
- 238000009826 distribution Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 7
- 230000000694 effects Effects 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 description 58
- 239000013598 vector Substances 0.000 description 37
- 241000023320 Luma <angiosperm> Species 0.000 description 36
- OSWPMRLSEDHDFF-UHFFFAOYSA-N methyl salicylate Chemical compound COC(=O)C1=CC=CC=C1O OSWPMRLSEDHDFF-UHFFFAOYSA-N 0.000 description 36
- 230000002829 reductive effect Effects 0.000 description 25
- 239000000523 sample Substances 0.000 description 24
- 238000003860 storage Methods 0.000 description 24
- 230000002776 aggregation Effects 0.000 description 19
- 238000004220 aggregation Methods 0.000 description 19
- 238000004891 communication Methods 0.000 description 18
- 230000008569 process Effects 0.000 description 18
- 238000012549 training Methods 0.000 description 16
- 238000001914 filtration Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 12
- 238000012545 processing Methods 0.000 description 11
- 238000007906 compression Methods 0.000 description 10
- 238000012360 testing method Methods 0.000 description 10
- 230000006835 compression Effects 0.000 description 9
- 230000008901 benefit Effects 0.000 description 8
- 230000001419 dependent effect Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000013507 mapping Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 230000009467 reduction Effects 0.000 description 6
- 230000003044 adaptive effect Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 5
- 238000013461 design Methods 0.000 description 5
- 239000013074 reference sample Substances 0.000 description 5
- 230000011664 signaling Effects 0.000 description 5
- 238000000354 decomposition reaction Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 239000004065 semiconductor Substances 0.000 description 4
- 238000000638 solvent extraction Methods 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000002250 progressing effect Effects 0.000 description 3
- 230000002123 temporal effect Effects 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 2
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 229940050561 matrix product Drugs 0.000 description 2
- 230000036961 partial effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 229910052710 silicon Inorganic materials 0.000 description 2
- 239000010703 silicon Substances 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 101100379080 Emericella variicolor andB gene Proteins 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000003466 anti-cipated effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 239000013078 crystal Substances 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000003319 supportive effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/129—Scanning of coding units, e.g. zig-zag scan of transform coefficients or flexible macroblock ordering [FMO]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/18—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a set of transform coefficients
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/117—Filters, e.g. for pre-processing or post-processing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/12—Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/146—Data rate or code amount at the encoder output
- H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/157—Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
- H04N19/159—Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/625—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using discrete cosine transform [DCT]
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
METHOD, APPARATUS AND SYSTEM FOR ENCODING AND DECODING AN
IMAGE FRAME FROM A BISTREAM
A system and method of decoding an image frame from a video bitstream using a non-separable
secondary transform kernel. The method comprises arranging a set of decoded intermediate
transform coefficients of a coding block of the bitstream based on a scan order (1610); applying
an inverse non-separable secondary transform (NSST) kernel to the decoded intermediate
transform coefficients to generate a set of primary transform coefficients (1630), the NSST
kernel being orthogonal and having at least one row column of the NSST kernel composed of
zero-only weights, wherein positions of the set of primary transform coefficients have zero
magnitude in positions corresponding to the rows columns of zero-only weights, and the
corresponding positions are discretely distributed in an area of a two-dimensional scan array;
and decoding the image frame based on the generated set of primary transform coefficients of
the coding block.
23470274_1
18/19
1600
Start
1610 Diagonal scan
1620 Fetch kernel
1630 Multiply kernel
1640 Raster scan
End
Fig. 16
Description
18/19
1600
Start
1610 Diagonal scan
1620 Fetch kernel
1630 Multiply kernel
1640 Raster scan
End
Fig. 16
[0001] The present invention relates generally to digital video signal processing and, in particular, to a method, apparatus and system for encoding and decoding a block of video samples. The present invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for encoding and decoding a block of video samples.
[0002] Many applications for video coding currently exist, including applications for transmission and storage of video data. Many video coding standards have also been developed and others are currently in development. Recent developments in video coding standardisation have led to the formation of a group called the "Joint Video Experts Team" (JVET). The Joint Video Experts Team (JVET) includes members of Study Group 16, Question 6 (SG16/Q6) of the Telecommunication Standardisation Sector (ITU-T) of the International Telecommunication Union (ITU), also known as the "Video Coding Experts Group" (VCEG), and members of the International Organisations for Standardisation / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO/IEC JTC1/SC29/WG11), also known as the "Moving Picture Experts Group" (MPEG).
[0003] The Joint Video Experts Team (JVET) issued a Call for Proposals (CfP), with responses analysed at its 10 th meeting in San Diego, USA. The submitted responses demonstrated video compression capability significantly outperforming that of the current state-of-the-art video compression standard, i.e.: "high efficiency video coding" (HEVC). On the basis of this outperformance it was decided to commence a project to develop a new video compression standard, to be named 'versatile video coding' (VVC). VVC is anticipated to address ongoing demand for ever-higher compression performance, especially as video formats increase in capability (e.g., with higher resolution and higher frame rate) and address increasing market demand for service delivery over WANs, where bandwidth costs are relatively high. At the same time, VVC must be implementable in contemporary silicon processes and offer an acceptable trade-off between the achieved performance versus the implementation cost (for example, in terms of silicon area, CPU processor load, memory utilisation and bandwidth).
23470274_1
[0004] Video data includes a sequence of frames of image data, each of which include one or more colour channels. Generally, one primary colour channel and two secondary colour channels are needed. The primary colour channel is generally referred to as the 'luma' channel and the secondary colour channel(s) are generally referred to as the 'chroma' channels. Although video data is typically displayed in an RGB (red-green-blue) colour space, this colour space has a high degree of correlation between the three respective components. The video data representation seen by an encoder or a decoder is often using a colour space such as YCbCr. YCbCr concentrates luminance, mapped to 'luma' according to a transfer function, in a Y (primary) channel and chroma in Cb and Cr (secondary) channels. Moreover, the Cb and Cr channels may be sampled spatially at a lower rate (subsampled) compared to the luma channel, for example half horizontally and half vertically - known as a '4:2:0 chroma format'. The 4:2:0 chroma format is commonly used in 'consumer' applications, such as internet video streaming, broadcast television, and storage on Blu-Ray T M disks. Subsampling the Cb and Cr channels at half-rate horizontally and not subsampling vertically is known as a '4:2:2 chroma format'. The 4:2:2 chroma format is typically used in professional applications, including capture of footage for cinematic production and the like. The higher sampling rate of the 4:2:2 chroma format makes the resulting video more resilient to editing operations such as colour grading. Prior to distribution to consumers, 4:2:2 chroma format material is often converted to the 4:2:0 chroma format and then encoded for distribution to consumers. In addition to chroma format, video is also characterised by resolution and frame rate. Example resolutions are ultra-high definition (UHD) with a resolution of 3840x2160 or '8K' with a resolution of 7680x4320 and example frame rates are 60 or 120Hz. Luma sample rates may range from approximately 500 mega samples per second to several giga samples per second. For the 4:2:0 chroma format, the sample rate of each chroma channel is one quarter the luma sample rate and for the 4:2:2 chroma format, the sample rate of each chroma channel is one half the luma sample rate.
[0005] The VVC standard is a 'block based' codec, in which frames are firstly divided into a square array of regions known as 'coding tree units' (CTUs). CTUs generally occupy a relatively large area, such as 128x128 luma samples. However, CTUs at the right and bottom edge of each frame may be smaller in area. Associated with each CTU is a 'coding tree' for the luma channel and an additional coding tree for the chroma channels. A coding tree defines a decomposition of the area of the CTU into a set of blocks, also referred to as 'coding blocks' (CBs). It is also possible for a single coding tree to specify blocks both for the luma channel and the chroma channels, in which case the collections of collocated coding blocks are referred to as 'coding units' (CUs), i.e., each CU having a coding block for each colour channel. The
23470274_1
CBs are processed for encoding or decoding in a particular order. As a consequence of the use of the 4:2:0 chroma format, a CTU with a luma coding tree for a 128x128 luma sample area has a corresponding chroma coding tree for a 64x64 chroma sample area, collocated with the 128x128 luma sample area. When a single coding tree is in use for the luma channel and the chroma channels, the collections of collocated blocks for a given area are generally referred to as 'units', for example the above-mentioned CUs, as well as 'prediction units' (PUs), and 'transform units' (TUs). When separate coding trees are used for a given area, the above mentioned CBs, as well as 'prediction blocks' (PBs), and 'transform blocks' (TBs) are used.
[0006] Notwithstanding the above distinction between 'units' and 'blocks', the term 'block' may be used as a general term for areas or regions of a frame for which operations are applied to all colour channels.
[0007] For each CU a prediction unit (PU) of the contents (sample values) of the corresponding area of frame data is generated (a 'prediction unit'). If the PU is generated from sample values in a previously signalled frame, the prediction is called inter prediction. If the PU is generated from previous samples in the same frame, the prediction is called intra prediction. Further, a representation of the difference (or 'residual' in the spatial domain) between the prediction and the contents of the area as seen at input to the encoder is formed. The difference in each colour channel may be transformed and coded as a block of residual coefficients, forming one or more TUs for a given CU. The residual coefficients may be transformed by a primary transform such as a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), or other transform. This primary transform is applied separably, that is the two-dimensional transform is performed in two passes. The block is firstly transformed by applying a one-dimensional transform to each row of samples in the block. Then, the partial result is transformed by applying a one dimensional transform to each column of the partial result to produce a final block of primary transform coefficients that substantially decorrelates the residual samples.
[0008] While the separable application of a primary transform is computationally efficient, it is unable to fully exploit angular features that may exist in the block. For example, when a form of intra prediction called intra angular prediction is used, the block of residual samples often contains angular features corresponding to the angle of intra angular prediction. A non separable secondary transform may be applied to achieve further compression efficiency. The non-separable secondary transform may be selected from a number of non-separable secondary transform candidates, where selection may be explicitly signalled or depend on other already
23470274_1 known decision, such as for example the intra prediction angular mode. Application of the secondary transform, while desirable from the perspective of further reducing the coding cost of the residual coefficients, introduces complexity into the design with respect to the size of the secondary transform including all candidates. Application of the secondary transform also increases complexity in the encoder to test and select from the application of possible combinations of primary transforms and non-separable secondary transforms.
[0009] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.
[00010] One aspect of the present invention provides a method of decoding an image frame from a video bitstream using a non-separable secondary transform kernel, the method comprising: arranging a set of decoded intermediate transform coefficients of a coding block of the bitstream based on a scan order; applying an inverse non-separable secondary transform (NSST) kernel to the decoded intermediate transform coefficients to generate a set of primary transform coefficients, the NSST kernel being orthogonal and having at least one column of the NSST kernel composed of zero-only weights, wherein positions of the set of primary transform coefficients have zero magnitude in positions corresponding to the columns of zero-only weights, and the corresponding positions are discretely distributed in an area of a two dimensional scan array; and decoding the image frame based on the generated set of primary transform coefficients of the coding block.
[00011] According to another aspect, the at least one column of zero-only weights is determined based upon preferential order of applying zero-value weighting to columns [63, 15, 47,62,7,31,55,23,22,14,54,30,38,39,6,21,45,57,61,13,46,29,5,59,53,60,52,58, 12,20,50,36,37,49,42,28,35,56,41,43,44,27,51,34,48,19,26,4,18,11,33,40,3,25, 17, 10, 9, 2, 16, 8, 32, 24, 0, 1] for a block of sixty-four primary transform coefficients.
[00012] According to another aspect, the discrete distribution of corresponding positions having zero magnitude primary transform coefficients are based on an order of [63, 15, 47, 62, 7,31,55,23,22,14,54,30,38,39,6,21,45,57,61,13,46,29,5,59,53,60,52,58,12,20, 50,36,37,49,42,28,35,56,41,43,44,27,51,34,48,19,26,4,18,11,33,40,3,25,17,10, 9, 2, 16, 8, 32, 24, 0, 1] for a block of sixty-four primary transform coefficients.
23470274_1
[00013] According to another aspect, the at least one column of zero-only weights is determined based upon preferential order of applying zero-value weighting to columns as the indices [15, 31, 23, 7, 22, 14, 30, 6, 29, 21, 13, 5, 45, 47, 46, 41, 20, 12, 44, 28, 42, 39, 27, 38, 37, 43, 35, 34, 26, 40, 19, 4, 33, 18, 11, 25, 36, 10, 3, 24, 9, 2, 17, 32, 16, 1, 8, 0] for a block of forty-eight primary transform coefficients.
[00014] According to another aspect, the discrete distribution of corresponding positions having zero magnitude primary transform coefficients are based on an order of indices [15, 31, 23,7,22,14,30,6,29,21,13,5,45,47,46,41,20,12,44,28,42,39,27,38,37,43,35,34, 26, 40, 19, 4, 33, 18, 11, 25, 36, 10, 3, 24, 9, 2, 17, 32, 16, 1, 8, 0] for a block of forty-eight transform coefficients.
[00015] According to another aspect, the primary transform coefficients relate to a block of forty-eight transform coefficients and the discrete distribution of transform coefficients relates to positions 15, 31, 23, 7, 22, 14, 30, 6, 29, 21, 13 and 5.
[00016] According to another aspect, the at least one column of zero-only weights is determined based upon effect of the corresponding primary transform positions to coding gain.
[00017] According to another aspect, the zero-only columns are determined based upon a selection of an intra prediction mode set and a candidate set.
[00018] According to another aspect, the zero-only columns are determined based upon a selection of an intra prediction mode set.
[00019] According to another aspect, the discrete distribution is asymmetrical asymmetry and relates to removing primary transform coefficients which correspond to horizontal high frequencies.
[00020] Another aspect of the present invention provides a non-transitory computer-readable medium having a computer program stored thereon to implement a method of decoding an image frame from a video bitstream using a non-separable secondary transform kernel, the method comprising: arranging a set of decoded intermediate transform coefficients of a coding block of the bitstream based on a scan order; applying an inverse non-separable secondary transform (NSST) kernel to the decoded intermediate transform coefficients to generate a set of primary transform coefficients, the NSST kernel being orthogonal and having at least one
23470274_1 column of the NSST kernel composed of zero-only weights, wherein positions of the set of primary transform coefficients have zero magnitude in positions corresponding to the columns of zero-only weights, and the corresponding positions are discretely distributed in an area of a two-dimensional scan array; and decoding the image frame based on the generated set of primary transform coefficients of the coding block.
[00021] Another aspect of the present invention provides a video decoder configured to implement a method of decoding an image frame from a video bitstream using a non-separable secondary transform kernel, the method comprising: arranging a set of decoded intermediate transform coefficients of a coding block of the bitstream based on a scan order; applying an inverse non-separable secondary transform (NSST) kernel to the decoded intermediate transform coefficients to generate a set of primary transform coefficients, the NSST kernel being orthogonal and having at least one column of the NSST kernel composed of zero-only weights, wherein positions of the set of primary transform coefficients have zero magnitude in positions corresponding to the columns of zero-only weights, and the corresponding positions are discretely distributed in an area of a two-dimensional scan array; and decoding the image frame based on the generated set of primary transform coefficients of the coding block.
[00022] Another aspect of the present invention provides a system, comprising: a memory; and a processor, wherein the processor is configured to execute code stored on the memory for implementing a method of decoding an image frame from a video bitstream using a non separable secondary transform kernel, the method comprising: arranging a set of decoded intermediate transform coefficients of a coding block of the bitstream based on a scan order; applying an inverse non-separable secondary transform (NSST) kernel to the decoded intermediate transform coefficients to generate a set of primary transform coefficients, the NSST kernel being orthogonal and having at least one column of the NSST kernel composed of zero-only weights, wherein positions of the set of primary transform coefficients have zero magnitude in positions corresponding to the columns of zero-only weights, and the corresponding positions are discretely distributed in an area of a two-dimensional scan array; and decoding the image frame based on the generated set of primary transform coefficients of the coding block.
[00023] Other aspects are also described.
23470274_1
[00024] At least one embodiment of the present invention will now be described with reference to the following drawings and and appendices, in which:
[00025] Fig. 1 is a schematic block diagram showing a video encoding and decoding system;
[00026] Figs. 2A and 2B form a schematic block diagram of a general purpose computer system upon which one or both of the video encoding and decoding system of Fig. 1 may be practiced;
[00027] Fig. 3 is a schematic block diagram showing functional modules of a video encoder;
[00028] Fig. 4 is a schematic block diagram showing functional modules of a video decoder;
[00029] Fig. 5 is a schematic block diagram showing the available divisions of a block into one or more blocks in the tree structure of versatile video coding;
[00030] Fig. 6 is a schematic illustration of a dataflow to achieve permitted divisions of a block into one or more blocks in a tree structure of versatile video coding;
[00031] Figs. 7A and 7B show an example division of a coding tree unit (CTU) into a number of coding units (CUs);
[00032] Fig. 8 shows a two-level backward diagonal scan;
[00033] Figs. 9A, 9B, 9C and 9D show forward and inverse non-separable secondary transforms performed according to different sizes of transform blocks;
[00034] Fig. 10 shows an example mapping of intra-angular angles to intra prediction modes;
[00035] Figs. 11A and 11B show modified patterns of primary transform coefficients for transform blocks of size 8x8 or greater;
[00036] Figs. 12A and 12B shows raster scan indexing for patterns of 8x8 primary transform coefficients and forty-eight primary transform coefficients;
[00037] Figs. 13 and 14 show example patterns of primary transform coefficients used by a non-separable secondary transform modified by setting weights to zero for coefficient indices selected according to a preferential order;
23470274_1
[00038] Fig. 15 shows a method for applying a forward non-separable secondary transform;
[00039] Fig. 16 shows a method for applying an inverse non-separable secondary transform; and
[00040] Fig. 17 shows examples patterns of primary transform coefficients used by a non separable secondary transform.
[00041] Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.
[00042] As described above, the use of a non-separable secondary transform can produce coding gains not achievable with a separable primary transform, because a non-separable secondary transform can exploit angular features. Angular features present in the video signal may be transformed by a non-separable secondary transform into a sparse number of secondary transform coefficients. However, use of a non-separable secondary transform introduces additional complexity into design of an encoder or decoder. The secondary transform "kernel" is the set of the weights representing the secondary transform basis vectors. The size of the secondary transform kernel, or equivalently the total number of secondary transform weights, is directly proportional to the additional memory that must be provisioned in the design to store the secondary transform kernel, and the number of additional multiplications that are required when the secondary transform is used. Therefore, minimising the size of the secondary transform kernel while maintaining the coding performance of the secondary transform, reduces the cost of implementation of the video coder design.
[00043] Fig. 1 is a schematic block diagram showing functional modules of a video encoding and decoding system 100. The system 100 may utilise constraints on the secondary transform kernel, such that the non-separable secondary transform may be performed with reduced complexity, while achieving similar coding performance to an unconstrained secondary transform kernel.
[00044] The system 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from the
23470274_1 source device 110 to the destination device 130. In some arrangements, the source device 110 and destination device 130 may either or both comprise respective mobile telephone handsets or "smartphones", in which case the communication channel 120 is a wireless channel. In other arrangements, the source device 110 and destination device 130 may comprise video conferencing equipment, in which case the communication channel 120 is typically a wired channel, such as an internet connection. Moreover, the source device 110 and the destination device 130 may comprise any of a wide range of devices, including devices supporting over the-air television broadcasts, cable television applications, internet video applications (including streaming) and applications where encoded video data is captured on some computer-readable storage medium, such as hard disk drives in a file server.
[00045] As shown in Fig. 1, the source device 110 includes a video source 112, a video encoder 114 and a transmitter 116. The video source 112 typically comprises a source of captured video frame data (shown as 113), such as an image capture sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote image capture sensor. The video source 112 may also be an output of a computer graphics card, for example displaying the video output of an operating system and various applications executing upon a computing device, for example a tablet computer. Examples of source devices 110 that may include an image capture sensor as the video source 112 include smart-phones, video camcorders, professional video cameras, and network video cameras.
[00046] The video encoder 114 converts (or 'encodes') the captured frame data (indicated by an arrow 113) from the video source 112 into a bitstream (indicated by an arrow 115) as described further with reference to Fig. 3. The bitstream 115 is transmitted by the transmitter 116 over the communication channel 120 as encoded video data (or "encoded video information"). It is also possible for the bitstream 115 to be stored in a non-transitory storage device 122, such as a "Flash" memory or a hard disk drive, until later being transmitted over the communication channel 120, or in-lieu of transmission over the communication channel 120.
[00047] The destination device 130 includes a receiver 132, a video decoder 134 and a display device 136. The receiver 132 receives encoded video data from the communication channel 120 and passes received video data to the video decoder 134 as a bitstream (indicated by an arrow 133). The video decoder 134 then outputs decoded frame data (indicated by an arrow 135) to the display device 136. The decoded frame data 135 has the same chroma format as the frame data 113. Examples of the display device 136 include a cathode ray tube, a liquid
23470274_1 crystal display, such as in smart-phones, tablet computers, computer monitors or in stand-alone television sets. It is also possible for the functionality of each of the source device 110 and the destination device 130 to be embodied in a single device, examples of which include mobile telephone handsets and tablet computers.
[00048] Notwithstanding the example devices mentioned above, each of the source device 110 and destination device 130 may be configured within a general purpose computing system, typically through a combination of hardware and software components. Fig. 2A illustrates such a computer system 200, which includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227, which may be configured as the video source 112, and a microphone 280; and output devices including a printer 215, a display device 214, which may be configured as the display device 136, and loudspeakers 217. An external Modulator-Demodulator (Modem) transceiver device 216 may be used by the computer module 201 for communicating to and from a communications network 220 via a connection 221. The communications network 220, which may represent the communication channel 120, may be a wide-area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. Where the connection 221 is a telephone line, the modem 216 may be a traditional "dial-up" modem. Alternatively, where the connection 221 is a high capacity (e.g., cable or optical) connection, the modem 216 may be a broadband modem. A wireless modem may also be used for wireless connection to the communications network 220. The transceiver device 216 may provide the functionality of the transmitter 116 and the receiver 132 and the communication channel 120 may be embodied in the connection 221.
[00049] The computer module 201 typically includes at least one processor unit 205, and a memory unit 206. For example, the memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read only memory (ROM). The computer module 201 also includes an number of input/output (1/0) interfaces including: an audio-video interface 207 that couples to the video display 214, loudspeakers 217 and microphone 280; an I/O interface 213 that couples to the keyboard 202, mouse 203, scanner 226, camera 227 and optionally a joystick or other human interface device (not illustrated); and an interface 208 for the external modem 216 and printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is generally the output of a computer graphics card. In some implementations, the modem 216 may be incorporated within the computer module 201, for example within the interface 208. The computer module 201 also has a local network interface 211, which permits
23470274_1 coupling of the computer system 200 via a connection 223 to a local-area communications network 222, known as a Local Area Network (LAN). As illustrated in Fig. 2A, the local communications network 222 may also couple to the wide network 220 via a connection 224, which would typically include a so-called "firewall" device or device of similar functionality. The local network interface 211 may comprise an EthernetTM circuit card, a BluetoothTM wireless arrangement or an IEEE 802.11 wireless arrangement; however, numerous other types of interfaces may be practiced for the interface 211. The local network interface 211 may also provide the functionality of the transmitter 116 and the receiver 132 and communication channel 120 may also be embodied in the local communications network 222.
[00050] The I/O interfaces 208 and 213 may afford either or both of serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated). Storage devices 209 are provided and typically include a hard disk drive (HDD) 210. Other storage devices such as a floppy disk drive and a magnetic tape drive (not illustrated) may also be used. An optical disk drive 212 is typically provided to act as a non-volatile source of data. Portable memory devices, such optical disks (e.g. CD-ROM, DVD, Blu ray DiscTM), USB-RAM, portable, external hard drives, and floppy disks, for example, may be used as appropriate sources of data to the computer system 200. Typically, any of the HDD 210, optical drive 212, networks 220 and 222 may also be configured to operate as the video source 112, or as a destination for decoded video data to be stored for reproduction via the display 214. The source device 110 and the destination device 130 of the system 100 may be embodied in the computer system 200.
[00051] The components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner that results in a conventional mode of operation of the computer system 200 known to those in the relevant art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Likewise, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by connections 219. Examples of computers on which the described arrangements can be practised include IBM-PC's and compatibles, Sun SPARCstations, Apple MacTM or alike computer systems.
[00052] Where appropriate or desired, the video encoder 114 and the video decoder 134, as well as methods described below, may be implemented using the computer system 200. In particular, the video encoder 114, the video decoder 134 and methods to be described, may be
23470274_1 implemented as one or more software application programs 233 executable within the computer system 200. In particular, the video encoder 114, the video decoder 134 and the steps of the described methods are effected by instructions 231 (see Fig. 2B) in the software 233 that are carried out within the computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the described methods and a second part and the corresponding code modules manage a user interface between the first part and the user.
[00053] The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer system 200 from the computer readable medium, and then executed by the computer system 200. A computer readable medium having such software or computer program recorded on the computer readable medium is a computer program product. The use of the computer program product in the computer system 200 preferably effects an advantageous apparatus for implementing the video encoder 114, the video decoder 134 and the described methods.
[00054] The software 233 is typically stored in the HDD 210 or the memory 206. The software is loaded into the computer system 200 from a computer readable medium, and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., CD-ROM) 225 that is read by the optical disk drive 212.
[00055] In some instances, the application programs 233 may be supplied to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212, or alternatively may be read by the user from the networks 220 or 222. Still further, the software can also be loaded into the computer system 200 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and/or data to the computer system 200 for execution and/or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-ray DiscTM, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 201. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of the software, application programs, instructions and/or video data or encoded video data to the computer module 401 include radio or infra-red transmission channels, as well as a network connection to
23470274_1 another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.
[00056] The second part of the application program 233 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 214. Through manipulation of typically the keyboard 202 and the mouse 203, a user of the computer system 200 and the application may manipulate the interface in a functionally adaptable manner to provide controlling commands and/or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via the loudspeakers 217 and user voice commands input via the microphone 280.
[00057] Fig. 2B is a detailed schematic block diagram of the processor 205 and a "memory" 234. The memory 234 represents a logical aggregation of all the memory modules (including the HDD 209 and semiconductor memory 206) that can be accessed by the computer module 201 in Fig. 2A.
[00058] When the computer module 201 is initially powered up, a power-on self-test (POST) program 250 executes. The POST program 250 is typically stored in a ROM 249 of the semiconductor memory 206 of Fig. 2A. A hardware device such as the ROM 249 storing software is sometimes referred to as firmware. The POST program 250 examines hardware within the computer module 201 to ensure proper functioning and typically checks the processor 205, the memory 234 (209, 206), and a basic input-output systems software (BIOS) module 251, also typically stored in the ROM 249, for correct operation. Once the POST program 250 has run successfully, the BIOS 251 activates the hard disk drive 210 of Fig. 2A. Activation of the hard disk drive 210 causes a bootstrap loader program 252 that is resident on the hard disk drive 210 to execute via the processor 205. This loads an operating system 253 into the RAM memory 206, upon which the operating system 253 commences operation. The operating system 253 is a system level application, executable by the processor 205, to fulfil various high level functions, including processor management, memory management, device management, storage management, software application interface, and generic user interface.
[00059] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has sufficient memory in which to execute without colliding with memory allocated to another process. Furthermore, the different
23470274_1 types of memory available in the computer system 200 of Fig. 2A must be used properly so that each process can run effectively. Accordingly, the aggregated memory 234 is not intended to illustrate how particular segments of memory are allocated (unless otherwise stated), but rather to provide a general view of the memory accessible by the computer system 200 and how such is used.
[00060] As shown in Fig. 2B, the processor 205 includes a number of functional modules including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes called a cache memory. The cache memory 248 typically includes a number of storage registers 244-246 in a register section. One or more internal busses 241 functionally interconnect these functional modules. The processor 205 typically also has one or more interfaces 242 for communicating with external devices via the system bus 204, using a connection 218. The memory 234 is coupled to the bus 204 using a connection 219.
[00061] The application program 233 includes a sequence of instructions 231 that may include conditional branch and loop instructions. The program 233 may also include data 232 which is used in execution of the program 233. The instructions 231 and the data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending upon the relative size of the instructions 231 and the memory locations 228-230, a particular instruction may be stored in a single memory location as depicted by the instruction shown in the memory location 230. Alternately, an instruction may be segmented into a number of parts each of which is stored in a separate memory location, as depicted by the instruction segments shown in the memory locations 228 and 229.
[00062] In general, the processor 205 is given a set of instructions which are executed therein. The processor 205 waits for a subsequent input, to which the processor 205 reacts to by executing another set of instructions. Each input may be provided from one or more of a number of sources, including data generated by one or more of the input devices 202, 203, data received from an external source across one of the networks 220, 202, data retrieved from one of the storage devices 206, 209 or data retrieved from a storage medium 225 inserted into the corresponding reader 212, all depicted in Fig. 2A. The execution of a set of the instructions may in some cases result in output of data. Execution may also involve storing data or variables to the memory 234.
[00063] The video encoder 114, the video decoder 134 and the described methods may use input variables 254, which are stored in the memory 234 in corresponding memory
23470274_1 locations 255, 256, 257. The video encoder 114, the video decoder 134 and the described methods produce output variables 261, which are stored in the memory 234 in corresponding memory locations 262, 263, 264. Intermediate variables 258 may be stored in memory locations 259, 260, 266 and 267.
[00064] Referring to the processor 205 of Fig. 2B, the registers 244, 245, 246, the arithmetic logic unit (ALU) 240, and the control unit 239 work together to perform sequences of micro operations needed to perform "fetch, decode, and execute" cycles for every instruction in the instruction set making up the program 233. Each fetch, decode, and execute cycle comprises:
a fetch operation, which fetches or reads an instruction 231 from a memory location 228, 229, 230;
a decode operation in which the control unit 239 determines which instruction has been fetched; and
an execute operation in which the control unit 239 and/or the ALU 240 execute the instruction.
[00065] Thereafter, a further fetch, decode, and execute cycle for the next instruction may be executed. Similarly, a store cycle may be performed by which the control unit 239 stores or writes a value to a memory location 232.
[00066] Each step or sub-process in the method of Figs. 15 and 16, to be described, is associated with one or more segments of the program 233 and is typically performed by the register section 244, 245, 247, the ALU 240, and the control unit 239 in the processor 205 working together to perform the fetch, decode, and execute cycles for every instruction in the instruction set for the noted segments of the program 233.
[00067] Fig. 3 is a schematic block diagram showing functional modules of the video encoder 114. Fig. 4 is a schematic block diagram showing functional modules of the video decoder 134. Generally, data passes between functional modules within the video encoder 114 and the video decoder 134 in groups of samples or coefficients, such as divisions of blocks into sub-blocks of a fixed size, or as arrays. The video encoder 114 and video decoder 134 may be implemented using a general-purpose computer system 200, as shown in Figs. 2A and 2B, where the various functional modules may be implemented by dedicated hardware within the
23470274_1 computer system 200, by software executable within the computer system 200 such as one or more software code modules of the software application program 233 resident on the hard disk drive 205 and being controlled in its execution by the processor 205. Alternatively the video encoder 114 and video decoder 134 may be implemented by a combination of dedicated hardware and software executable within the computer system 200. The video encoder 114, the video decoder 134 and the described methods may alternatively be implemented in dedicated hardware, such as one or more integrated circuits performing the functions or sub functions of the described methods. Such dedicated hardware may include graphic processing units (GPUs), digital signal processors (DSPs), application-specific standard products (ASSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or one or more microprocessors and associated memories. In particular, the video encoder 114 comprises modules 310-386 and the video decoder 134 comprises modules 420-496 which may each be implemented as one or more software code modules of the software application program 233.
[00068] Although the video encoder 114 of Fig. 3 is an example of a versatile video coding (VVC) video encoding pipeline, other video codecs may also be used to perform the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames, each frame including one or more colour channels. The frame data 113 may be in any chroma format, for example 4:0:0, 4:2:0, 4:2:2, or 4:4:4 chroma format. A block partitioner 310 firstly divides the frame data 113 into CTUs, generally square in shape and configured such that a particular size for the CTUs is used. The size of the CTUs may be 64x64, 128x128, or 256x256 luma samples for example. The block partitioner 310 further divides each CTU into one or more CBs according to a luma coding tree and a chroma coding tree. The CBs have a variety of sizes, and may include both square and non-square aspect ratios. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Thus, a current CB, represented as 312, is output from the block partitioner 310, progressing in accordance with an iteration over the one or more blocks of the CTU, in accordance with the luma coding tree and the chroma coding tree of the CTU. Options for partitioning CTUs into CBs are further described below with reference to Figs. 5 and 6.
[00069] The CTUs resulting from the first division of the frame data 113 may be scanned in raster scan order and may be grouped into one or more 'slices'. A slice may be an 'intra' (or 'I') slice. An intra slice (I slice) indicates that every CU in the slice is intra predicted. Alternatively, a slice may be uni- or bi-predicted ('P' or 'B'slice, respectively), indicating additional availability of uni- and bi-prediction in the slice, respectively.
23470274_1
[00070] For each CTU, the video encoder 114 operates in two stages. In the first stage (referred to as a 'search' stage), the block partitioner 310 tests various potential configurations of a coding tree. Each potential configuration of a coding tree has associated 'candidate' CBs. The first stage involves testing various candidate CBs to select CBs providing high compression efficiency with low distortion. The testing generally involves a Lagrangian optimisation whereby a candidate CB is evaluated based on a weighted combination of the rate (coding cost) and the distortion (error with respect to the input frame data 113). The 'best' candidate CBs (the CBs with the lowest evaluated rate/distortion) are selected for subsequent encoding into the bitstream 115. Included in evaluation of candidate CBs is an option to use a CB for a given area or to further split the area according to various splitting options and code each of the smaller resulting areas with further CBs, or split the areas even further. As a consequence, both the CBs and the coding tree themselves are selected in the search stage.
[00071] The video encoder 114 produces a prediction block (PB), indicated by an arrow 320, for each CB, for example the CB 312. The PB 320 is a prediction of the contents of the associated CB 312. A subtracter module 322 produces a difference, indicated as 324 (or 'residual', referring to the difference being in the spatial domain), between the PB 320 and the CB 312. The difference 324 is a block-size difference between corresponding samples in the PB 320 and the CB 312. The difference 324 is transformed, quantised and represented as a transform block (TB), indicated by an arrow 336. The PB 320 and associated TB 336 are typically chosen from one of many possible candidate CBs, for example based on evaluated cost or distortion.
[00072] A candidate coding block (CB) is a CB resulting from one of the prediction modes available to the video encoder 114 for the associated PB and the resulting residual. Each candidate CB results in one or more corresponding TBs, as described hereafter with reference to Fig. 8. The TB 336 is a quantised and transformed representation of the difference 324. When combined with the predicted PB in the video decoder 114, the TB 336 reduces the difference between decoded CBs and the original CB 312 at the expense of additional signalling in a bitstream.
[00073] Each candidate coding block (CB), that is prediction block (PB) in combination with a transform block (TB), thus has an associated coding cost (or 'rate') and an associated difference (or 'distortion'). The rate is typically measured in bits. The distortion of the CB is typically estimated as a difference in sample values, such as a sum of absolute differences (SAD) or a
23470274_1 sum of squared differences (SSD). The estimate resulting from each candidate PB may be determined by a mode selector 386 using the difference 324 to determine a prediction mode (represented by an arrow 388). Estimation of the coding costs associated with each candidate prediction mode and corresponding residual coding can be performed at significantly lower cost than entropy coding of the residual. Accordingly, a number of candidate modes can be evaluated to determine an optimum mode in a rate-distortion sense.
[00074] Determining an optimum mode in terms of rate-distortion is typically achieved using a variation of Lagrangian optimisation. Selection of the prediction mode 388 typically involves determining a coding cost for the residual data resulting from application of a particular prediction mode. The coding cost may be approximated by using a 'sum of absolute transformed differences' (SATD) whereby a relatively simple transform, such as a Hadamard transform, is used to obtain an estimated transformed residual cost. In some implementations using relatively simple transforms, the costs resulting from the simplified estimation method are monotonically related to the actual costs that would otherwise be determined from a full evaluation. In implementations with monotonically related estimated costs, the simplified estimation method may be used to make the same decision (i.e. prediction mode) with a reduction in complexity in the video encoder 114. To allow for possible non-monotonicity in the relationship between estimated and actual costs, the simplified estimation method may be used to generate a list of best candidates. The non-monotonicity may result from further mode decisions available for the coding of residual data, for example. The list of best candidates may be of an arbitrary number. A more complete search may be performed using the best candidates to establish optimal mode choices for coding the residual data for each of the candidates, allowing a final selection of the prediction mode 388 along with other mode decisions.
[00075] Prediction modes fall broadly into two categories. A first category is 'intra-frame prediction' (also referred to as 'intra prediction'). In intra-frame prediction, a prediction for a block is generated, and the generation method may use other samples obtained from the current frame. Types of intra prediction include intra planar, intra DC, intra angular, and matrix weighted intra prediction (MIP). For an intra-predicted PB, it is possible for different intra prediction modes to be used for luma and chroma, and thus intra prediction is described primarily in terms of operation upon PBs. Additionally, chroma CBs may be predicted from co-located luma samples by a cross-component linear model prediction.
23470274_1
[00076] The second category of prediction modes is 'inter-frame prediction' (also referred to as 'inter prediction'). In inter-frame prediction a prediction for a block is produced using samples from one or two frames preceding the current frame in an order of coding frames in the bitstream. Moreover, for inter-frame prediction, a single coding tree is typically used for both the luma channel and the chroma channels. The order of coding frames in the bitstream may differ from the order of the frames when captured or displayed. When one frame is used for prediction, the block is said to be 'uni-predicted' and has one associated motion vector. When two frames are used for prediction, the block is said to be 'bi-predicted' and has two associated motion vectors. For a P slice, each CU may be intra predicted or uni-predicted. For a B slice, each CU may be intra predicted, uni-predicted, or bi-predicted. Frames are typically coded using a 'group of pictures' structure, enabling a temporal hierarchy of frames. A temporal hierarchy of frames allows a frame to reference a preceding and a subsequent picture in the order of displaying the frames. The images are coded in the order necessary to ensure the dependencies for decoding each frame are met.
[00077] A subcategory of inter prediction is referred to as 'skip mode'. Inter prediction and skip modes are described as two distinct modes. However, both inter prediction mode and skip mode involve motion vectors referencing blocks of samples from preceding frames. Inter prediction involves a coded motion vector delta, specifying a motion vector relative to a motion vector predictor. The motion vector predictor is obtained from a list of one or more candidate motion vectors, selected with a 'merge index'. The coded motion vector delta provides a spatial offset to a selected motion vector prediction. Inter prediction also uses a coded residual in the bitstream 133. Skip mode uses only an index (also named a 'merge index') to select one out of several motion vector candidates. The selected candidate is used without any further signalling. Also, skip mode does not support coding of any residual coefficients. The absence of coded residual coefficients when the skip mode is used means that there is no need to perform transforms for the skip mode. Therefore, skip mode does not typically result in pipeline processing issues. Pipeline processing issues may be the case for intra predicted CUs and inter predicted CUs. Due to the limited signalling of the skip mode, skip mode is useful for achieving very high compression performance when relatively high quality reference frames are available. Bi-predicted CUs in higher temporal layers of a random-access group-of-picture structure typically have high quality reference pictures and motion vector candidates that accurately reflect underlying motion.
23470274_1
[00078] The samples are selected according to a motion vector and reference picture index. The motion vector and reference picture index applies to all colour channels and thus inter prediction is described primarily in terms of operation upon PUs rather than PBs. Within each category (that is, intra- and inter-frame prediction), different techniques may be applied to generate the PU. For example, intra prediction may use values from adjacent rows and columns of previously reconstructed samples, in combination with a direction to generate a PU according to a prescribed filtering and generation process. Alternatively, the PU may be described using a small number of parameters. Inter prediction methods may vary in the number of motion parameters and their precision. Motion parameters typically comprise a reference frame index, indicating which reference frame(s) from lists of reference frames are to be used plus a spatial translation for each of the reference frames, but may include more frames, special frames, or complex affine parameters such as scaling and rotation. In addition, a pre determined motion refinement process may be applied to generate dense motion estimates based on referenced sample blocks.
[00079] Lagrangian or similar optimisation processing can be employed to both select an optimal partitioning of a CTU into CBs (by the block partitioner 310) as well as the selection of a best prediction mode from a plurality of possibilities. Through application of a Lagrangian optimisation process of the candidate modes in the mode selector module 386, the prediction mode with the lowest cost measurement is selected as the 'best' mode. The lowest cost mode is the selected prediction mode 388 and is also encoded in the bitstream 115 by an entropy encoder 338. The selection of the prediction mode 388 by operation of the mode selector module 386 extends to operation of the block partitioner 310. For example, candidates for selection of the prediction mode 388 may include modes applicable to a given block and additionally modes applicable to multiple smaller blocks that collectively are collocated with the given block. In cases including modes applicable to a given block and smaller collocated blocks, the process of selection of candidates implicitly is also a process of determining the best hierarchical decomposition of the CTU into CBs.
[00080] In the second stage of operation of the video encoder 114 (referred to as a 'coding' stage), an iteration over the selected luma coding tree and the selected chroma coding tree, and hence each selected CB, is performed in the video encoder 114. In the iteration, the CBs are encoded into the bitstream 115, as described further herein.
23470274_1
[00081] The entropy encoder 338 supports both variable-length coding of syntax elements and arithmetic coding of syntax elements. Arithmetic coding is supported using a context-adaptive binary arithmetic coding process. Arithmetically coded syntax elements consist of sequences of one or more 'bins'. Bins, like bits, have a value of '0' or '1'.However bins are not encoded in the bitstream 115 as discrete bits. Bins have an associated predicted (or 'likely' or 'most probable') value and an associated probability, known as a 'context'. When the actual bin to be coded matches the predicted value, a 'most probable symbol' (MPS) is coded. Coding a most probable symbol is relatively inexpensive in terms of consumed bits. When the actual bin to be coded mismatches the likely value, a 'least probable symbol' (LPS) is coded. Coding a least probable symbol has a relatively high cost in terms of consumed bits. The bin coding techniques enable efficient coding of bins where the probability of a '0' versus a '1' is skewed. For a syntax element with two possible values (that is, a 'flag'), a single bin is adequate. For syntax elements with many possible values, a sequence of bins is needed.
[00082] The presence of later bins in the sequence may be determined based on the value of earlier bins in the sequence. Additionally, each bin may be associated with more than one context. The selection of a particular context can be dependent on earlier bins in the syntax element, the bin values of neighbouring syntax elements (i.e. those from neighbouring blocks) and the like. Each time a context-coded bin is encoded, the context that was selected for that bin (if any) is updated in a manner reflective of the new bin value. As such, the binary arithmetic coding scheme is said to be adaptive.
[00083] Also supported by the video encoder 114 are bins that lack a context ('bypass bins'). Bypass bins are coded assuming an equiprobable distribution between a '0' and a '1'. Thus, each bin occupies one bit in the bitstream 115. The absence of a context saves memory and reduces complexity, and thus bypass bins are used where the distribution of values for the particular bin is not skewed. One example of an entropy coder employing context and adaption is known in the art as CABAC (context adaptive binary arithmetic coder) and many variants of this coder have been employed in video coding.
[00084] The entropy encoder 338 encodes the prediction mode 388 using a combination of context-coded and bypass-coded bins. For example, when the prediction mode 388 is an intra prediction mode, a list of 'most probable modes' is generated in the video encoder 114. The list of most probable modes is typically of a fixed length, such as three or six modes, and may include modes encountered in earlier blocks. A context-coded bin encodes a flag indicating if
23470274_1 the prediction mode is one of the most probable modes. If the intra prediction mode 388 is one of the most probable modes, further signalling, using bypass-coded bins, is encoded. The encoded further signalling is indicative of which most probable mode corresponds with the intra prediction mode 388, for example using a truncated unary bin string. Otherwise, the intra prediction mode 388 is encoded as a 'remaining mode'. Encoding as a remaining mode uses an alternative syntax, such as a fixed-length code, also coded using bypass-coded bins, to express intra prediction modes other than those present in the most probable mode list.
[00085] A multiplexer module 384 outputs the PB 320 according to the determined best prediction mode 388, selecting from the tested prediction mode of each candidate CB. The candidate prediction modes need not include every conceivable prediction mode supported by the video encoder 114.
[00086] Having determined and selected the PB 320, and subtracted the PB 320 from the original sample block at the subtractor 322, a residual with lowest coding cost, represented as 324, is obtained and subjected to lossy compression. The lossy compression process comprises the steps of transformation, quantisation and entropy coding. A forward primary transform module 326 applies a forward transform to the residual 324, converting the residual 324 from the spatial domain to the frequency domain, and producing primary transform coefficients represented by an arrow 328. The primary transform coefficients 328 are passed to a forward secondary transform module 330 to produce transform coefficients represented by an arrow 332 by performing a non-separable secondary transform (NSST) operation. The forward primary transform is typically separable, transforming a set of rows and then a set of columns of each block, typically using a type-II discrete cosine transform (DCT-2), although a type-VII discrete sine transform (DST-7) and a type-VIII discrete cosine transform (DCT-8) may also be available, for example horizontally for block widths not exceeding 16 samples and vertically for block heights not exceeding 16 samples. The transformation of each set of rows and columns is performed by applying one-dimensional transforms firstly to each row of a block to produce an intermediate result and then to each column of the intermediate result to produce a final result. The forward secondary transform is generally a non-separable transform, which is only applied for the residual of intra-predicted CUs and may nonetheless also be bypassed. The forward secondary transform operates either on 16 samples (arranged as the upper-left 4x4 sub-block of the primary transform coefficients 328) or 64 samples (arranged as the upper-left 8 x8 coefficients, arranged as four 4x4 sub-blocks of the primary transform coefficients 328). Moreover, the matrix coefficients of the forward secondary transform are selected from
23470274_1 multiple sets according to the intra prediction mode of the CU such that two sets of coefficients are available for use. The use of one of the sets of matrix coefficients, or the bypassing of the forward secondary transform, is signalled with an "nsst-index" syntax element, coded using a truncated unary binarisation to express the values zero (secondary transform not applied), one (first set of matrix coefficients selected), or two (second set of matrix coefficients selected).
[00087] The video encoder 114 may also choose to skip both the primary and secondary transforms, known as 'transform skip' mode. Skipping the transforms is suited to residual data that lacks adequate correlation for reduced coding cost via expression as transform basis functions. Certain types of content, such as relatively simple computer generated graphics may exhibit similar behaviour. When transform skip mode is used, the transform coefficients 332 are the same as the residual coefficients 324.
[00088] The transform coefficients 332 are passed to a quantiser module 334. At the module 334, quantisation in accordance with a 'quantisation parameter' is performed to produce residual coefficients, represented by the arrow 336. The quantisation parameter is constant for a given TB and thus results in a uniform scaling for the production of residual coefficients for a TB. A non-uniform scaling is also possible by application of a 'quantisation matrix', whereby the scaling factor applied for each residual coefficient is derived from a combination of the quantisation parameter and the corresponding entry in a scaling matrix, typically having a size equal to that of the TB. The scaling matrix may have a size that is smaller than the size of the TB, and when applied to the TB a nearest neighbour approach is used to provide scaling values for each residual coefficient from a scaling matrix smaller in size than the TB size. The residual coefficients 336 are supplied to the entropy encoder 338 for encoding in the bitstream 115. Typically, the residual coefficients of each TB with at least one significant residual coefficient of the TU are scanned to produce an ordered list of values, according to a scan pattern. The scan pattern generally scans the TB as a sequence of 4x4 'sub-blocks', providing a regular scanning operation at the granularity of 4x4 sets of residual coefficients, with the arrangement of sub-blocks dependent on the size of the TB. Additionally, the prediction mode 388 and the corresponding block partitioning are also encoded in the bitstream 115.
[00089] As described above, the video encoder 114 needs access to a frame representation corresponding to the frame representation seen by the video decoder 134. Thus, the residual coefficients 336 are also inverse quantised by a dequantiser module 340 to produce inverse
23470274_1 transform coefficients, represented by an arrow 342. The inverse transform coefficients 342 are passed through an inverse secondary transform module 344 to produce intermediate inverse transform coefficients, represented by an arrow 346. The intermediate inverse transform coefficients 346 are passed to an inverse primary transform module 348 to produce residual samples, represented by an arrow 350, of the TU. The types of inverse transform performed by the inverse secondary transform module 344 correspond with the types of forward transform performed by the forward secondary transform module 330. The types of inverse transform performed by the inverse primary transform module 348 correspond with the types of primary transform performed by the primary transform module 326. A summation module 352 adds the residual samples 350 and the PU 320 to produce reconstructed samples (indicated by an arrow 354) of the CU.
[00090] The reconstructed samples 354 are passed to a reference sample cache 356 and an in loop filters module 368. The reference sample cache 356, typically implemented using static RAM on an ASIC (thus avoiding costly off-chip memory access) provides minimal sample storage needed to satisfy the dependencies for generating intra-frame PBs for subsequent CUs in the frame. The minimal dependencies typically include a 'line buffer' of samples along the bottom of a row of CTUs, for use by the next row of CTUs and column buffering the extent of which is set by the height of the CTU. The reference sample cache 356 supplies reference samples (represented by an arrow 358) to a reference sample filter 360. The sample filter 360 applies a smoothing operation to produce filtered reference samples (indicated by an arrow 362). The filtered reference samples 362 are used by an intra-frame prediction module 364 to produce an intra-predicted block of samples, represented by an arrow 366. For each candidate intra prediction mode the intra-frame prediction module 364 produces a block of samples, that is 366.
[00091] The in-loop filters module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a 'deblocking filter' (DBF) which applies smoothing aligned to the CU boundaries to reduce artefacts resulting from discontinuities. Another filtering stage present in the in-loop filters module 368 is an 'adaptive loop filter' (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further availablefiltering stage in the in-loop filters module 368 is a 'sample adaptive offset' (SAO) filter. The SAO filter operates by firstly classifying reconstructed samples into one or multiple categories and, according to the allocated category, applying an offset at the sample level.
23470274_1
[00092] Filtered samples, represented by an arrow 370, are output from the in-loop filters module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has the capacity to store several (for example up to 16) pictures and thus is stored in the memory 206. The frame buffer 372 is not typically stored using on-chip memory due to the large memory consumption required. As such, access to the frame buffer 372 is costly in terms of memory bandwidth. The frame buffer 372 provides reference frames (represented by an arrow 374) to a motion estimation module 376 and a motion compensation module 380.
[00093] The motion estimation module 376 estimates a number of 'motion vectors' (indicated as 378), each being a Cartesian spatial offset from the location of the present CB, referencing a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (represented as 382) is produced for each motion vector. The filtered reference samples 382 form further candidate modes available for potential selection by the mode selector 386. Moreover, for a given CU, the PU 320 may be formed using one reference block ('uni-predicted') or may be formed using two reference blocks ('bi-predicted'). For the selected motion vector, the motion compensation module 380 produces the PB 320 in accordance with a filtering process supportive of sub-pixel accuracy in the motion vectors. As such, the motion estimation module 376 (which operates on many candidate motion vectors) may perform a simplified filtering process compared to that of the motion compensation module 380 (which operates on the selected candidate only) to achieve reduced computational complexity. When the video encoder 114 selects inter prediction for a CU the motion vector 378 is encoded into the bitstream 115.
[00094] Although the video encoder 114 of Fig. 3 is described with reference to versatile video coding (VVC), other video coding standards or implementations may also employ the processing stages of modules 310-386. The frame data 113 (and bitstream 115) may also be read from (or written to) memory 206, the hard disk drive 210, a CD-ROM, a Blu-ray diskTM or other computer readable storage medium. Additionally, the frame data 113 (and bitstream 115) may be received from (or transmitted to) an external source, such as a server connected to the communications network 220 or a radio-frequency receiver.
[00095] The video decoder 134 is shown in Fig. 4. Although the video decoder 134 of Fig. 4 is an example of a versatile video coding (VVC) video decoding pipeline, other video codecs may also be used to perform the processing stages described herein. As shown in Fig. 4, the bitstream 133 is input to the video decoder 134. The bitstream 133 may be read from
23470274_1 memory 206, the hard disk drive 210, a CD-ROM, a Blu-ray diskTM or other non-transitory computer readable storage medium. Alternatively, the bitstream 133 may be received from an external source such as a server connected to the communications network 220 or a radio frequency receiver. The bitstream 133 contains encoded syntax elements representing the captured frame data to be decoded.
[00096] The bitstream 133 is input to an entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 by decoding sequences of 'bins' and passes the values of the syntax elements to other modules in the video decoder 134. One example of a syntax element extracted from the bitstream 133 are residual coefficients 424. The entropy decoder module 420 uses an arithmetic decoding engine to decode each syntax element as a sequence of one or more bins. Each bin may use one or more 'contexts', with a context describing probability levels to be used for coding a 'one' and a 'zero' value for the bin. Where multiple contexts are available for a given bin, a 'context modelling' or 'context selection' step is performed to choose one of the available contexts for decoding the bin. The process of decoding bins forms a sequential feedback loop. The number of operations in the feedback loop is preferably minimised to enable the entropy decoder 420 to achieve a high throughput in bins/second. Context modelling depends on other properties of the bitstream known to the video decoder 134 at the time of selecting the context, that is, properties preceding the current bin. For example, a context may be selected based on the quad-tree depth of the current CU in the coding tree. Dependencies are preferably based on properties that are known well in advance of decoding a bin, or are determined without requiring long sequential processes.
[00097] The residual coefficients 424 are input to a dequantiser module 428. The dequantiser module 428 performs inverse quantisation (or 'scaling') on the residual coefficients 424 to create reconstructed intermediate transform coefficients, represented by an arrow 432, according to a quantisation parameter. Should use of a non-uniform inverse quantisation matrix be indicated in the bitstream 133, the video decoder 134 reads a quantisation matrix from the bitstream 133 as a sequence of scaling factors and arranges the scaling factors into a matrix. The inverse scaling uses the quantisation matrix in combination with the quantisation parameter to create the reconstructed intermediate transform coefficients 432. The reconstructed intermediate transform coefficients 432 are passed to an inverse secondary transform module 436 where a secondary transform is applied or no operation (bypass), in accordance with a decoded "nsst-index" syntax element. The "nsst-index" is decoded from the
23470274_1 bitstream 133 by the entropy decoder 420, under execution of the processor 205. The inverse secondary transform module 436 produces reconstructed transform coefficients 440.
[00098] The reconstructed transform coefficients 440 are passed to an inverse primary transform module 444. The module 444 transforms the coefficients from the frequency domain back to the spatial domain. The result of operation of the module 444 is a block of residual samples, represented by an arrow 448. The block of residual samples 448 is equal in size to the corresponding CU. The residual samples 448 are supplied to a summation module 450. At the summation module 450 the residual samples 448 are added to a decoded PB (represented as 452) to produce a block of reconstructed samples, represented by an arrow 456. The reconstructed samples 456 are supplied to a reconstructed sample cache 460 and an in-loop filtering module 488. The in-loop filtering module 488 produces reconstructed blocks of frame samples, represented as 492. The frame samples 492 are written to a frame buffer 496.
[00099] The reconstructed sample cache 460 operates similarly to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for reconstructed sample needed to intra predict subsequent CBs without the memory 206 (for example by using the data 232 instead, which is typically on-chip memory). Reference samples, represented by an arrow 464, are obtained from the reconstructed sample cache 460 and supplied to a reference sample filter 468 to produce filtered reference samples indicated by arrow 472. The filtered reference samples 472 are supplied to an intra-frame prediction module 476. The module 476 produces a block of intra-predicted samples, represented by an arrow 480, in accordance with an intra prediction mode parameter 458 signalled in the bitstream 133 and decoded by the entropy decoder 420.
[000100] When the prediction mode of a CB is indicated to be intra prediction in the bitstream 133, the intra-predicted samples 480 form the decoded PB 452 via a multiplexor module 484. Intra prediction produces a prediction block (PB) of samples, that is, a block in one colour component, derived using 'neighbouring samples' in the same colour component. The neighbouring samples are samples adjacent to the current block and by virtue of being preceding in the block decoding order have already been reconstructed. Where luma and chroma blocks are collocated, the luma and chroma blocks may use different intra prediction modes. However, the two chroma channels each share the same intra prediction mode.
[000101] Intra prediction for luma blocks consist of four types. "DC intra prediction" involves populating a PB with a single value representing the average of the neighbouring samples.
23470274_1
"Planar intra prediction" involves populating a PB with samples according to a plane, with a DC offset and a vertical and horizontal gradient being derived from the neighbouring samples. "Angular intra prediction" involves populating a PB with neighbouring samples filtered and propagated across the PB in a particular direction (or 'angle'). In VVC a PB may select from up to 65 angles, with rectangular blocks able to utilise different angles not available to square blocks. "Matrix intra prediction" involves populating a PB by multiplying a reduced set of neighbouring samples by one of a number of available matrices available to the video decoder 134. The reduced set of neighbouring samples is produced by filtering and subsampling the neighbouring samples. Then, a reduced set of prediction samples is produced by multiplying the reduced set of samples by a matrix, and adding an offset vector. The matrix and associated offset vector are selected from a number of possible matrices depending on the size of the PB, with a particular selection of matrix and offset vector being indicated by a "MIP mode" syntax element. For example, for PBs with size greater than 8x8 there are 11 MIP modes, while for PBs of size 8x8 there are 19 MIP modes. Finally, the PB produced by matrix intra prediction is populated from the reduced set of prediction samples by interpolation.
[000102] A fifth type of intra prediction is available to chroma PBs, whereby the PB is generated from collocated luma reconstructed samples according to a 'cross-component linear model' (CCLM) mode. Three different CCLM modes are available, each of which uses a different model derived from the neighbouring luma and chroma samples. The derived model is used to generate a block of samples for the chroma PB from the collocated luma samples.
[000103] When the prediction mode of a CB is indicated to be inter prediction in the bitstream 133, a motion compensation module 434 produces a block of inter-predicted samples, represented as 438, using a motion vector and reference frame index to select and filter a block of samples 498 from a frame buffer 496. The block of samples 498 is obtained from a previously decoded frame stored in the frame buffer 496. For bi-prediction, two blocks of samples are produced and blended together to produce samples for the decoded PB 452. The frame buffer 496 is populated with filtered block data 492 from an in-loop filtering module 488. As with the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies any of the DBF, the ALF and SAO filtering operations. Generally, the motion vector is applied to both the luma and chroma channels, although the filtering processes for sub-sample interpolation luma and chroma channel are different.
23470274_1
[000104] Fig. 5 is a schematic block diagram showing a collection 500 of available divisions or splits of a region into one or more sub-regions in the tree structure of versatile video coding. The divisions shown in the collection 500 are available to the block partitioner 310 of the encoder 114 to divide each CTU into one or more CUs or CBs according to a coding tree, as determined by the Lagrangian optimisation, as described with reference to Fig. 3.
[000105] Although the collection 500 shows only square regions being divided into other, possibly non-square sub-regions, it should be understood that the diagram 500 is showing the potential divisions but not requiring the containing region to be square. If the containing region is non-square, the dimensions of the blocks resulting from the division are scaled according to the aspect ratio of the containing block. Once a region is not further split, that is, at a leaf node of the coding tree, a CU occupies that region. The particular subdivision of a CTU into one or more CUs by the block partitioner 310 is referred to as the 'coding tree' of the CTU.
[000106] The process of subdividing regions into sub-regions must terminate when the resulting sub-regions reach a minimum CU size. In addition to constraining CUs to prohibit block areas smaller than a predetermined minimum size, for example 16 samples, CUs are constrained to have a minimum width or height of four. Other minimums, both in terms of width and height or in terms of width or height are also possible. The process of subdivision may also terminate prior to the deepest level of decomposition, resulting in a CU larger than the minimum CU size. It is possible for no splitting to occur, resulting in a single CU occupying the entirety of the CTU. A single CU occupying the entirety of the CTU is the largest available coding unit size. Due to use of subsampled chroma formats, such as 4:2:0, arrangements of the video encoder 114 and the video decoder 134 may terminate splitting of regions in the chroma channels earlier than in the luma channels.
[000107] At the leaf nodes of the coding tree exist CUs, with no further subdivision. For example, a leaf node 510 contains one CU. At the non-leaf nodes of the coding tree exist a split into two or more further nodes, each of which could be a leaf node that forms one CU, or a non leaf node containing further splits into smaller regions. At each leaf node of the coding tree, one coding block exists for each colour channel. Splitting terminating at the same depth for both luma and chroma results in three collocated CBs. Splitting terminating at a deeper depth for luma than for chroma results in a plurality of luma CBs being collocated with the CBs of the chroma channels.
23470274_1
[000108] A quad-tree split 512 divides the containing region into four equal-size regions as shown in Fig. 5. Compared to HEVC, versatile video coding (VVC) achieves additional flexibility with the addition of a horizontal binary split 514 and a vertical binary split 516. Each of the splits 514 and 516 divides the containing region into two equal-size regions. The division is either along a horizontal boundary (514) or a vertical boundary (516) within the containing block.
[000109] Further flexibility is achieved in versatile video coding with addition of a ternary horizontal split 518 and a ternary vertical split 520. The ternary splits 518 and 520 divide the block into three regions, bounded either horizontally (518) or vertically (520) along /4and/4 of the containing region width or height. The combination of the quad tree, binary tree, and ternary tree is referred to as 'QTBTTT'. The root of the tree includes zero or more quadtree splits (the 'QT' section of the tree). Once the QT section terminates, zero or more binary or ternary splits may occur (the 'multi-tree' or 'MT' section of the tree), finally ending in CBs or CUs at leaf nodes of the tree. Where the tree describes all colour channels, the tree leaf nodes are CUs. Where the tree describes the luma channel or the chroma channels, the tree leaf nodes are CBs.
[000110] Compared to HEVC, which supports only the quad tree and thus only supports square blocks, the QTBTTT results in many more possible CU sizes, particularly considering possible recursive application of binary tree and/or ternary tree splits. The potential for unusual (non square) block sizes can be reduced by constraining split options to eliminate splits that would result in a block width or height either being less than four samples or in not being a multiple of four samples. Generally, the constraint would apply in considering luma samples. However, in the arrangements described, the constraint can be applied separately to the blocks for the chroma channels. Application of the constraint to split options to chroma channels can result in differing minimum block sizes for luma versus chroma, for example when the frame data is in the 4:2:0 chroma format or the 4:2:2 chroma format. Each split produces sub-regions with a side dimension either unchanged, halved or quartered, with respect to the containing region. Then, since the CTU size is a power of two, the side dimensions of all CUs are also powers of two.
[000111] Fig. 6 is a schematic flow diagram illustrating a data flow 600 of a QTBTTT (or 'coding tree') structure used in versatile video coding. The QTBTTT structure is used for each CTU to define a division of the CTU into one or more CUs. The QTBTTT structure of each CTU is determined by the block partitioner 310 in the video encoder 114 and encoded into the
23470274_1 bitstream 115 or decoded from the bitstream 133 by the entropy decoder 420 in the video decoder 134. The data flow 600 further characterises the permissible combinations available to the block partitioner 310 for dividing a CTU into one or more CUs, according to the divisions shown in Fig. 5.
[000112] Starting from the top level of the hierarchy, that is at the CTU, zero or more quad-tree divisions are first performed. Specifically, a Quad-tree (QT) split decision 610 is made by the block partitioner 310. The decision at 610 returning a '1' symbol indicates a decision to split the current node into four sub-nodes according to the quad-tree split 512. The result is the generation of four new nodes, such as at 620, and for each new node, recursing back to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further split is to be performed (returns a '0' symbol), quad-tree partitioning ceases and multi-tree (MT) splits are subsequently considered.
[000113] Firstly, an MT split decision 612 is made by the block partitioner 310. At 612, a decision to perform an MT split is indicated. Returning a '0' symbol at decision 612 indicates that no further splitting of the node into sub-nodes is to be performed. If no further splitting of a node is to be performed, then the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if the MT split 612 indicates a decision to perform an MT split (returns a '1' symbol), the block partitioner 310 proceeds to a direction decision 614.
[000114] The direction decision 614 indicates the direction of the MT split as either horizontal ('H' or '0') or vertical ('V' or '1').The block partitioner 310 proceeds to a decision 616 if the decision 614 returns a '0' indicating a horizontal direction. The block partitioner 310 proceeds to a decision 618 if the decision 614 returns a '1'indicating a vertical direction.
[000115] At each of the decisions 616 and 618, the number of partitions for the MT split is indicated as either two (binary split or 'BT' node) or three (ternary split or 'TT') at the BT/TT split. That is, a BT/TT split decision 616 is made by the block partitioner 310 when the indicated direction from 614 is horizontal and a BT/TT split decision 618 is made by the block partitioner 310 when the indicated direction from 614 is vertical.
[000116] The BT/TT split decision 616 indicates whether the horizontal split is the binary split 514, indicated by returning a '0', or the ternary split 518, indicated by returning a '1'. When the BT/TT split decision 616 indicates a binary split, at a generate HBT CTU nodes step 625 two nodes are generated by the block partitioner 310, according to the binary
23470274_1 horizontal split 514. When the BT/TT split 616 indicates a ternary split, at a generate HTT CTU nodes step 626 three nodes are generated by the block partitioner 310, according to the ternary horizontal split 518.
[000117] The BT/TT split decision 618 indicates whether the vertical split is the binary split 516, indicated by returning a '0', or the ternary split 520, indicated by returning a '1'. When the BT/TT split 618 indicates a binary split, at a generate VBT CTU nodes step 627 two nodes are generated by the block partitioner 310, according to the vertical binary split 516. When the BT/TT split 618 indicates a ternary split, at a generate VTT CTU nodes step 628 three nodes are generated by the block partitioner 310, according to the vertical ternary split 520. For each node resulting from steps 625-628 recursion of the data flow 600 back to the MT split decision 612 is applied, in a left-to-right or top-to-bottom order, depending on the direction 614. As a consequence, the binary tree and ternary tree splits may be applied to generate CUs having a variety of sizes.
[000118] Figs. 7A and 7B provide an example division 700 of a CTU 710 into a number of CUs or CBs. An example CU 712 is shown in Fig. 7A. Fig. 7A shows a spatial arrangement of CUs in the CTU 710. The example division 700 is also shown as a coding tree 720 in Fig. 7B.
[000119] At each non-leaf node in the CTU 710 of Fig. 7A, for example nodes 714, 716 and 718, the contained nodes (which may be further divided or may be CUs) are scanned or traversed in a 'Z-order' to create lists of nodes, represented as columns in the coding tree 720. For a quad-tree split, the Z-order scanning results in top left to right followed by bottom left to right order. For horizontal and vertical splits, the Z-order scanning (traversal) simplifies to a top-to-bottom scan and a left-to-right scan, respectively. The coding tree 720 of Fig. 7B lists all nodes and CUs according to the applied scan order. Each split generates a list of two, three or four new nodes at the next level of the tree until a leaf node (CU) is reached.
[000120] Having decomposed the image into CTUs and further into CUs by the block partitioner 310, and using the CUs to generate each residual block (324) as described with reference to Fig. 3, residual blocks are subject to forward transformation by the video encoder 114. An equivalent inverse transform process is performed in the video decoder 134 to obtain TBs from the bitstream 133.
[000121] In the video encoder 114, the transform coefficients 332 are rearranged to a one dimensional list by performing a two-level backward diagonal scan. Similarly, in the video decoder 134, the reconstructed intermediate transform coefficients 432 are rearranged from a
23470274_1 one-dimensional list to a two-dimensional collection of sub-blocks by the same two-level backward diagonal scan.
[000122] Fig. 8 shows a two-level backward diagonal scan 810 of an example 8x8 TB 800. The scan 810 is shown progressing from the bottom-right residual coefficient position of the TB 800 back to the top-left (DC) residual coefficient position of the TB 800. The path of the scan 910 progresses through 4x4 regions, known as sub-blocks, and from one sub-block to the next. For TBs of width or height of two, sub-block sizes of 2x2, 2x8, or 8x2 are available. Scanning within a particular sub-block is either performed or the sub-block skipped, according to a 'coded sub-block flag'. When scanning of a sub-block is skipped all residual coefficients within the sub-block are inferred to have a value of zero. Although the scan 810 is shown commencing from the bottom-right residual coefficient position of the TB 800, for a given set of residual coefficients scanning commences from the position of the 'last significant coefficient', the coefficient being 'last' when order of coefficients is considered as progressing from the DC coefficient instead of the scan order.
[000123] Figs. 9A, 9B, 9C and 9D show examples of forward and inverse non-separable secondary transforms that are performed according to different sizes of transform blocks (TBs). Fig. 9A shows a set of relationships 900 between primary transform coefficients 902 and secondary transform coefficients 904 for a 4x4 TB size. The primary transform coefficients 902 consist of 4x4 coefficients, while the secondary transform coefficients 904 consist of eight coefficients. The eight secondary transform coefficients are arranged in a pattern 906. The pattern 906 corresponds to the last eight positions in a backward diagonal scan of the TB. The first eight positions in the backward diagonal scan are zeroes. A forward non-separable secondary transform 910 for 4x4 TBs therefore receives sixteen primary transform coefficients, and produces as output eight secondary transform coefficients. The forward secondary transform 910 for 4x4 TBs can therefore be represented by an 8x16 matrix of weights. Similarly, an inverse secondary transform 912 can be represented by a 16x8 matrix of weights.
[000124] Fig. 9B shows a set of relationships 918 between primary transform coefficients and secondary transform coefficients for 4xN and Nx4 TB sizes, where N is greater than 4. In both cases, a top-left 4x4 sub-block of primary coefficients 920 is associated with a top-left 4x4 sub block of secondary transform coefficients 924. Then in the video encoder 114, the forward non-separable secondary transform 930 takes sixteen primary transform coefficients, and produces as output sixteen secondary transform coefficients. Remaining primary transform coefficients 922 are zeroed out. After the forward non-separable secondary transform 930 is
23470274_1 performed, coefficient positions 926 associated with the coefficients 922 are not populated and remain zeroes as well.
[000125] The forward secondary transform 930 for 4xN or Nx4 TBs can be represented by a 16x16 matrix of weights. The matrix representing the forward secondary transform 930 is defined as A. Similarly, a corresponding inverse secondary transform 932 can be represented by a 16x16 matrix of weights. The matrix representing the inverse secondary transform 932 is defined as B. For the inverse secondary transform 932 to be the inverse of the forward secondary transform 930, the property: B = A- should be present. Moreover, it is highly desirable if A has the property of orthogonality. A kernel satisfying the property of orthogonality consists of basis vectors which have unit length (in an L2 norm sense) and are orthogonal to each other. The basis vectors of a kernel expressed in the matrix form of A are the rows. If A is orthogonal, then A- = A T, andB =AT. Based on the orthogonal relationship only A needs be stored in the video encoder 114 and video decoder 134, as the weights of B can be looked up from A. In practice, it is sufficient for orthogonality to be approximately satisfied, as limitations due to integer precision may prevent exact equality.
[000126] Approximate orthogonality may be tested by calculating the matrix product AAT, where the weights of matrix A incorporate any bit shift operations that may be employed in an integerised implementation of the non-separable secondary transform. A is perfectly orthogonal if AAT is the identity matrix. Alternatively, A may be considered as approximately orthogonal if all weights on the diagonal of AA T deviate from the value one by less than a threshold a, and all weights not on the diagonal of AA T have an absolute value of less than the same threshold a. In practice 8-bit integerised implementations of the non-separable transform typically achieve approximate orthogonality with a = 0.02. In the context of the arrangements described, a kernel achieving a = 0.02 is sufficient to be considered a kernel with rows having orthogonality.
[000127] The storage requirement of the non-separable transform kernel is further reduced by reusing parts of A for the forward secondary transform 910 and the inverse secondary transform 912 for 4x4 TBs. The first eight rows of A are used for the forward secondary transform 910, and the transpose of the first eight rows of A are used for the inverse secondary transform 912.
[000128] Fig. 9C shows a relationship 955 between primary transform coefficients 940 and secondary transform coefficients 942 for 8x8 TB sizes. The primary transform coefficients 940 consist of 8x8 coefficients, while the secondary transform coefficients 942 only consist of eight
23470274_1 transform coefficients. The eight secondary transform coefficients 942 are arranged in a pattern corresponding to the last eight positions in a backward diagonal scan of the TB. The remaining secondary transform coefficients in the TB are all zeroes. Then, the forward non-separable secondary transform 950 for an 8x8 TB takes sixty-four primary transform coefficients, and produces as output eight secondary transform coefficients. The forward secondary transform 950 for 8x8 TBs can be represented by an 8x64 matrix of weights. Similarly, a corresponding inverse secondary transform 952 for 8x8 TBs can be represented by a 64x8 matrix of weights.
[000129] Fig. 9D shows a relationship 975 between primary transform coefficients 960 and secondary transform coefficients 962 for TBs of size greater than 8x8. A top-left 8x8 block of primary coefficients 960 (composed of four 4x4 sub-blocks) is associated with a top-left 4x4 sub-block of secondary transform coefficients 962. In the video encoder 114, a forward non separable secondary transform 970 takes sixty-four primary transform coefficients, and produces as output sixteen secondary transform coefficients. Remaining primary transform coefficients 964 are zeroed out. Secondary transform coefficient positions 966 outside of the top-left 4x4 sub-block of secondary transform coefficients 962 are not populated and remain as zeroes.
[000130] The forward secondary transform 970 for TBs of size greater than 8x8 can be represented by a 16x64 matrix of weights. A matrix representing the forward secondary transform 970 is defined as F. Similarly, a corresponding inverse secondary transform 932 can be represented by a 64x16 matrix of weights. A matrix representing the inverse secondary transform 972 is defined as G. As described above with reference to matrices A and B, it is desirable that F has the property of orthogonality. The property of orthogonality means G = FT, and only F needs be stored in the video encoder 114 and video decoder 134. An orthogonal matrix can be described as a matrix in which the rows have orthogonality.
[000131] The storage requirement of the non-separable transform kernel is further reduced by reusing parts of F for the forward secondary transform 950 and the inverse secondary transform 952 for 8x8 TBs. The first eight rows of F are used for the forward secondary transform 910, and the transpose of the first eight rows of F are used for the inverse secondary transform 912.
[000132] Non-separable secondary transforms may achieve coding improvement over the use of separable primary transforms alone, because non-separable secondary transforms are able to sparsify two-dimensional features in the residual signal, such as angular features. As angular features in the residual signal may be dependent on the type of intra prediction mode 388
23470274_1 selected, it is advantageous for the non-separable secondary transform matrix to be adaptively selected depending on the intra prediction mode. As described above, intra prediction modes consist of "intra-DC", "intra-planar", "intra-angular" modes, and "matrix intra prediction" modes. The intra prediction mode parameter 458 takes the value of 0 when intra-DC prediction is used. The intra prediction mode parameter 458 takes the value of1 when intra-planar prediction is used. The intra prediction mode parameter 458 takes a value between 2 and 66 inclusive, when intra-angular prediction on square TBs is used.
[000133] Fig. 10 shows an example mapping 1000 of intra-angular mode angles to the intra prediction mode parameter 458. Further angles relating to wide-angle prediction on rectangular TBs as well as other modes can be used for intra-angular mode but are not shown in the example of Fig. 10.
[000134] Table 1 below shows an example mapping from the intra prediction mode parameter 458 to a mode set index. The mode set index is used to adaptively select a non-separable secondary matrix according to the type of intra prediction mode. In the example of Table 1, four mode sets are stored in the video encoder 114 and video decoder 134. However, for intra angular prediction modes the non-separable secondary matrix is applied differently depending on whether the angle of prediction is dominantly horizontal (IntraPredMode 34), or the angle of prediction is dominantly vertical (IntraPredMode > 34). As described above with reference to Fig. 8, the transform coefficients 332 are rearranged as a one-dimensional list from a two-dimensional collection of sub-blocks by a two-level diagonal scan.
[000135] When the non-separable secondary transform is used, the transform coefficients 332 are secondary transform coefficients. The transform coefficients 332 are rearranged by a two level forward diagonal scan, which is the opposite order of the two-level backward diagonal scan 810. When the non-separable secondary transform is used and IntraPredMode < 34, the primary transform coefficients are rearranged by a raster scan in row-major order. When the non-separable secondary transform is used and IntraPredMode > 34, the primary transform coefficients are rearranged by a raster scan in column-major order. Therefore, the same non separable matrix may be utilised by two oppositely symmetric intra prediction angles.
23470274_1
IntraPredMode Mode set mndex IntraPredMode < 0 1 0 <= IntraPredMode <= 1 0 2 <= IntraPredMode <= 12 1 13 <= IntraPredMode <= 23 2 24 <= IntraPredMode <= 44 3 45 <= IntraPredMode <= 55 2 56 <= IntraPredMode<= 80 1 81 <= IntraPredMode<= 83 0 Table 1: Intra prediction mode to mode set index mapping
[000136] In addition to adaptively selecting a non-separable secondary transform matrix based on the size of the TB, or based on the intra prediction mode parameter 458, the non-separable secondary transform matrix may also be selected from a number of candidate sets. The "nsst-index" is used to indicate which candidate set is selected. In VVC, two candidate sets are supported. If the nsstindex takes the value of 0, then the non-separable secondary transform is disabled. If the nsstindex takes the value of 1, then candidate set 0 is selected. If the nsst-index takes the value of 2, then candidate set 1 is selected.
[000137] In summary, the non-separable secondary transform described above may be implemented by storing two kernels in the video encoder 114 and video decoder 134, a smaller kernel applying to TBs of size less than 8x8, and a larger kernel applying to TBs of size 8x8 or greater. The smaller kernel has size 4x2x16x16. The larger kernel has size 4x2x16x64.
[000138] The non-separable secondary transform kernel weights may be obtained by an offline training procedure. The training is referred to as offline as the training is not implemented when the encoder 114 or decoder 134 are encoding or decoding video data for user purposes. Rather, the training can be implemented at setup or update of 114 or 134 prior to encoding or decoding video data for user purposes. For example, the video encoder 114 may be applied to a training dataset with inter prediction methods disabled, resulting in a number of intra predicted TBs. Let Bi, with j = 0 be the set of TBs of size less than 8x8 predicted by one of the intra
prediction modes corresponding to mode set index i as indicated above in Table 1. Let Bi,
with j = 1 be the set of TBs of size 8x8 or greater predicted by one of the intra prediction modes corresponding to mode set index i as indicated above in Table 1. In a first initialisation step, each Bi, is divided randomly into three subsets Bijk corresponding to the number of
values of the nsstindex. For each subset Bi,j,k where k > 0, an optimal non-separable
secondary transform matrix may be solved by applying the method of singular value
23470274_1 decomposition (SVD) to the covariance matrix of Bilk. Only the primary transform coefficient positions taken as input to the non-separable secondary transform are used to construct the covariance matrix. For example, the zeroed-out coefficient positions 964 are not considered. The eigenvectors of the covariance matrix form the basis vectors of the non-separable transform matrix. For non-square transform matrices (ie, the larger kernel applying to TBs of size 8x8 or greater), the eigenvectors associated with the largest magnitude eigenvalues are selected. For each subset Bi,j,k where k = 0, the non-separable secondary transform matrix is forced to the identity matrix (ie non-separable secondary transform disabled).
[000139] In subsequent iterative steps of the offline training procedure, each TB in Bi, for all
i,j is transformed by each of the transform matrices determined in a prior step and associated with the subsets Bij,,. Then each TB in Bi, is assigned to the one of the three subsetsBi,j,k
with the best performance in a rate-distortion optimisation. After reassignment of the TBs to the subsets Bijk the optimal non-separable secondary transform matrices may be recalculated
with the SVD method described above. The iterative steps may be terminated after a fixed number of iterations, or when the total rate-distortion cost stops decreasing.
[000140] Minimising the size of the secondary transform kernel provides an advantage by reducing the implementation cost of the video coder design. In one arrangement, the larger kernel applying to TBs of size 8x8 or greater is reduced from 4x2x16x64 to 4x2x16x48, that is the input to the non-separable secondary transform is reduced from sixty-four primary transform coefficients to forty-eight primary transform coefficients.
[000141] The arrangements described provide a basis for determining how to reduce or minimise the size of a kernel of the non-separable secondary transform of the module 330 (and correspondingly 436). The arrangements described provide methods for reducing the size of the kernel based on minimisation of coding costs and maintaining orthogonality. The reduced size kernel has particular zero-valued weights in some columns, resulting in primary coefficients at decoding (output by the module 436) that have zero magnitude at corresponding positions to the zero-value weighted rows when arranged in two-dimensional form. The effect of operation of using the zero-weight values of the non-separable secondary transform when inversely applied at decoding results in a corresponding (modified) pattern of zeros in the decoded primary transform coefficients.
[000142] Fig. 11A shows a modified pattern 1110 of primary transform coefficients for TBs of size 8x8. The modified pattern 1110 corresponds to the first three 4x4 sub-blocks in a forward
23470274_1 diagonal scan of the TB. A remaining sub-block of primary coefficients 1120 is zeroed out. A set 1115 of secondary transform coefficient results. Fig. 1lB shows a modified pattern 1130 of primary transform coefficients for TBs of size greater than 8x8. The modified pattern 1130 also corresponds to the first three 4x4 sub-blocks in a forward diagonal scan of the TB. Remaining primary transform coefficients 1140 are zeroed out. A set 1150 of 4x4 secondary transform coefficients results. Transform kernel weights for the arrangements shown in Fig. 11A and 1lB may be obtained by offline training in the same manner as described above, with the covariance matrices constructed from the reduced number of primary transform coefficients.
[000143] In another arrangement, the modified (resulting) pattern of primary transform coefficients may be selected based on primary transform coefficients positions that are identified to be most influential to the output of the non-separable secondary transform. F is set to be a larger kernel applying to TBs of size8x8 or greater, with dimensionality 4x2x16xP, and weights obtained by an offline training procedure such as described above performed on an initial pattern of P primary coefficients. The notation F[m][c][s][p] indicates a weight in F corresponding to mode set m, candidate set c, secondary transform coefficient s and primary transform coefficient p. The notation F[ ][ ][ ][p] indicates a multi-dimensional tensor of weights in F corresponding to primary transform coefficient p.
[000144] For a given mode set m and a given candidate set c, the basis vectors of the non separable secondary transform are the rows F[m][c][s][ ] for each s. The weight F[m][c][s][p] indicates the amount by which primary transform coefficient p contributes to the result of an inner product between F[m][c][s][ ] and a TB of primary transform coefficients. For example, if the weight F[m][c][s] [p] = 0, then primary transform coefficient p has no contribution to the result of the inner product. Similarly, the tensor F[ ][ ][ ][p] indicates the amount by which primary transform coefficient p contributes to the result of applying the non separable secondary transform, for all secondary transform coefficients and across all mode sets and candidate sets. The degree to which each primary transform coefficient p contributes to the overall result of applying the non-separable secondary transform may be estimated by applying an aggregation measure to the F[ ][ ][ ][p] tensor. For example, the aggregation measure may be the LI norm, the L2 norm, or the L-infinity norm of F[ ][ ][ ][p], where the norm is applied treating F[ ][ ][ ][p] as a vector. The aggregation measure may also be calculated by applying a combination of vector norms across the dimensions of F[ ][ ][ ][p]. In one example, the aggregation measure is provided by V[p] = ||||F[][][][p]|II|mc. That is, the aggregation
measure V[p] may be calculated by taking the L2 norm of each column F[ m][c ][ ][p] for all
23470274_1 mode sets m and candidate sets c, and then taking the L-infinity norm of the L2 norms across all m and c.
[000145] If the aggregate measure V[p] for a primary transform coefficient p is relatively small, then p is identified as not being influential to the output of the non-separable secondary transform. Accordingly, the set of weights F[ ][ ][ ][p] may all be set to zero with little change to the performance of the non-separable secondary transform. Setting all the weights of F[ ][ ][ ][p] to zero is equivalent to removing p from the pattern of primary transform coefficients that are input to the non-separable secondary transform. An advantage of setting all the weights of F[ ][ ][ ][p] for some primary transform coefficients p is that the portion F[ ][ ][ ][p] of the non-separable secondary transform kernel does not need to be stored in the video encoder 114 or video decoder 134.
[000146] In one arrangement, the kernel F is obtained by offline training performed on an initial pattern of sixty-four primary coefficients. A vector of aggregate measures V[ ] is arranged in increasing order, which produces a preferential order in which primary transform coefficients are identified for removal from the pattern of primary transform coefficients. Fig. 12A shows a raster scan indexing of coefficients 1210 for a pattern of8x8 primary transform coefficients. The 8x8 primary transform coefficients may correspond to the 8x8 primary coefficients 940 of TBs of size 8x8, or the8x8 primary transform coefficients may correspond to the top-left 8x8 primary coefficients 960 of TBs of size greater than 8x8. The raster indexing 1210 is in row-major order and corresponds with the non-separable secondary transform used when IntraPredMode 34. If IntraPredMode > 34 the raster indexing is in column-major order, and the coefficient indices will be the transpose of indices shown in Fig. 12A. In the arrangement described in relation to Fig. 12A, the preferential order has been identified with reference to the raster indexing 1210 as the indices [63, 15, 47, 62, 7, 31, 55, 23, 22, 14, 54, 30, 38,39,6,21,45,57,61,13,46,29,5,59,53,60,52,58,12,20,50,36,37,49,42,28,35,56, 41,43,44,27,51,34,48, 19,26,4, 18, 11,33,40,3,25, 17, 10,9,2, 16,8,32,24,0, 1].
[000147] Fig. 13 shows a collection 1300 of example patterns of primary transform coefficients used by a non-separable secondary transform that has been modified by setting weights of F[ ][ ][ ][p] to zero for coefficient indices p selected according to the preferential order described above. In one arrangement, the first eight coefficients in the preferential order above (63, 15, 47, 62, 7, 31, 55, 23) are selected. The corresponding columns of the non-separable secondary transform kernel (implemented by the module 330) are set to zero value. Application of the non-separable secondary transform kernel results in a modified pattern 1310.
23470274_1
The modified pattern 1310 has fifty-six primary residual coefficients. As can be seen with reference to Fig. 12A the modified pattern of 1310 has zero value coefficients (shown as 1311) at positions corresponding to the zero-value weighted columns. In another arrangement, the first sixteen coefficients in the preferential order above are selected (63, 15, 47, 62, 7, 31, 55, 23, 22, 14, 54, 30, 38, 39, 6, 21), resulting in a modified pattern 1320. In yet another arrangement, the first twenty-four coefficients in the preferential order above are selected, resulting in a modified pattern 1330.
[000148] In another arrangement, the kernel F is obtained by offline training performed on an initial pattern of forty-eight primary coefficients. Fig. 12B shows a raster scan indexing of coefficients 1220 for a pattern of forty-eight primary transform coefficients. The forty-eight primary transform coefficients may correspond to the modified pattern 1110 for TBs of size 8x8, or the forty-eight primary transform coefficients may correspond to the modified pattern 1130 for TBs of size greater than 8x8. As described above, the raster indexing 1220 is in row major order and corresponds with the non-separable transform used when IntraPredMode 34. When IntraPredMode > 34 the raster indexing is in column-major order, and the coefficient indices are transposed. In the arrangement described in relation to Fig. 12B, the preferential order in which primary transform coefficients are removed from the pattern has been identified with reference to the raster indexing 1220 as the indices [15, 31, 23, 7, 22, 14, 30,6,29,21,13,5,45,47,46,41,20,12,44,28,42,39,27,38,37,43,35,34,26,40,19,4, 33,18,11,25,36,10,3,24,9,2,17,32,16,1,8,0].
[000149] Fig. 14 shows a collection 1400 of example patterns of primary transform coefficients used by a non-separable secondary transform that has been modified by setting weights of F[ ][ ][ ][p] to zero for coefficient indices p selected according to the preferential order described above. In one arrangement, the first four coefficients in the preferential order above (15, 31, 23, 7) are selected, resulting in a modified pattern 1410. In another arrangement, the first eight coefficients in the preferential order above are selected, resulting in a modified pattern 1420. In yet another arrangement, the first twelve coefficients in the preferential order above are selected, resulting in a modified pattern 1430.
[000150] The kernel F for forty-eight primary coefficients (described with reference to Fig. 12B) is not a subset of the kernel F for sixty-four primary coefficients (described with reference to Fig. 12A). Because the weights for both kernels are obtained by an offline training procedure, the preferential order for removing primary transform coefficients is data driven and can accordingly vary across both kernels. However, the example patterns of Figs. 13 and 14
23470274_1 show similar behaviour. Firstly, the example patterns are highly asymmetric. Secondly, the asymmetry is caused by preferentially removing primary transform coefficients which correspond to horizontal high frequencies, when IntraPredMode 34. That is, horizontal high frequencies are identified as having little influence on the output of the non-separable secondary transform when the intra prediction mode is predominantly horizontal.
[000151] As shown in Figs. 13 and 14, the modified pattern of the primary transform coefficients generated relates to primary transform coefficients with a zero magnitude in positions corresponding to columns of the secondary transform kernel having zero-valued weights. The resulting pattern of zero magnitude positions in the primary transform coefficients, as indicated by the listed order of preferences and by Figs. 13 and 14, is discretely distributed about an area of the 2-dimensional scan array. A discrete distribution occurs irrespective of whether the primary coefficients are arranged in row-major order or column major order. The discrete distribution corresponds to distribution of the zero-value weighted columns in the secondary transform kernel F.
[000152] The advantage of the arrangements described above resulting in the example primary residual coefficient transform patterns of Figs. 13 and 14 is a reduction in complexity. Complexity is reduced both in terms of memory required for storage of the secondary transform kernel, and in the number of multiplications that are required to perform the secondary transform. Rather than arbitrary selection of a primary transform pattern such as the modified pattern 1110, the secondary transform kernel is reduced in size by neglecting the primary transform coefficients that have been identified as having the least influence on the output of the non-separable secondary transform. There may be some trade-off with lower coding performance if the number of primary transform coefficients neglected is too large. One test to measure the potential trade-off is by performing an orthogonality test, such as described above with reference to calculation of the matrix production AA T. The secondary transform kernel is firstly modified by setting the weights F[ ][ ][ ][p] to zero for each of the coefficient indices p removed, resulting in a modified secondary transform kernel F. The modified secondary transform kernel may be tested for approximate orthogonality by calculating the matrix product
[m][c ][][ ][m][c][][]T for each mode set m and candidate set s. If the modified secondary transform kernel F is approximately orthogonal, then negligible coding loss can be expected. For example, a test of the modified pattern 1420 using the Common Test Conditions of the VVC Test Model 6.0 in All Intra configuration shows coding performance loss of 0.02%
23470274_1 in luma, -0.01% in Cb chroma, and 0.02% in Cr chroma where coding performance is measured with the Bjontegaard distortion-rate metric.
[000153] The preferential orders of removing coefficients described above and illustrated using Figs. 13 and 14 account for coding cost as well as preservation of orthogonality of the secondary transform kernel. Effectively, removing coefficients earlier in the preferential order has reduced coding cost compared to removing coefficients later in the order. The preferential orders identified above can be used to maintain orthogonality dependent upon the level at which the columns of the secondary transform F are pruned (assigned zero-value weighting). Further, if larger numbers of columns of the kernel are to be removed, orthogonality (or approximate orthogonality) of the secondary transform kernel can still be maintained as described below.
[000154] In another arrangement, the modified secondary transform kernel is not used for the non-separable secondary transform. Instead, the offline training procedure described above is performed again, with the modified pattern of primary transform coefficients associated withF used to construct the covariance matrix. One advantage of the arrangement performing the offline training procedure again is that even if P is not orthogonal, the offline training procedure will produce an orthogonal secondary transform kernel. Therefore, the arrangement performing the offline training procedure again is suitable if a significant reduction in the size of the secondary transform kernel is desired.
[000155] Fig. 15 shows a method 1500 for applying a forward non-separable secondary transform in the video encoder 114 to the primary transform coefficients 328 for a current transform block with size greater than or equal to8x8, producing secondary transform coefficients 332. The method 1500 is typically implemented as one or more modules of the application 233, stored in the memory 206 and controlled under execution of the processor 205. The method 1500 is implemented by the application 233 using the primary transform coefficients generated by the primary transform module 326. The method 1500 begins at a raster scan step 1510.
[000156] At the raster scan step 1510, the primary transform coefficients 328 are rearranged from a two-dimensional representation to a one-dimensional list x by a raster scan over a modified pattern of P primary transform coefficients. The modified pattern may be the modified pattern 1310, the modified pattern 1320, the modified pattern 1330, the modified pattern 1410, the modified pattern 1420, the modified pattern 1430, or may be any pattern of primary transform coefficients obtained by excluding coefficient indices in the preferential
23470274_1 order described above with reference to Figs. 12 , 13 and 14. The raster scan over the modified pattern is in row major order if IntraPredMode 34. Otherwise, if IntraPredMode > 34 the raster scan over the modified pattern is in column major order. Coefficient indices not in the modified pattern are excluded from the one-dimensional list, and the corresponding primary transform coefficients are zeroed out. The method 1500 proceeds from step 1510 to a fetch kernel step 1520.
[000157] At the fetch kernel step 1520, a mode set index m is determined from the intra prediction mode 388 according to Table 1, and a candidate set index c is determined from a nsst-index syntax element selected by the video encoder 114 according to a rate-distortion optimisation. The mode set index and candidate set index are used to index into a 4x2xl6xP secondary transform kernel P stored in the video encoder 114. If the current transform block has size greater than 8x8, then the method 1500 retrieves a 16xP secondary transform matrix F from P[m][c][][ ]. Otherwise, if the current transform block has size 8x8, then the method 1500 retrieves an 8xP secondary transform matrix F from the first eight rows of F[m][c][][]. The method 1500 proceeds from step 1520 to a multiply kernel step 1530.
[000158] At the multiply kernel step 1530, a one-dimensional list of secondary transform coefficients y is calculated from the list of primary transform coefficients x and the secondary transform matrix F retrieved in step 1520. For an 8-bit integerised implementation, the secondary transform coefficients y are calculated by Equation 1 below:
y[i] = Clip(-1 F[i][] * x[j] + 64)»7) (1)
The step 1530 is implemented by the secondary transform module 330. The method 1500 proceeds from step 1530 to a diagonal scan step 1540.
[000159] At the diagonal scan step 1540, the list of secondary transform coefficients y is rearranged to a two-dimensional representation of secondary transform coefficients 332 by a forward diagonal scan. If the current transform block has size greater than 8x8, the secondary transform coefficients 332 fully populate a top-left 4x4 sub-block 962. Otherwise, if the current transform block has size 8x8, the secondary transform coefficients 332 populate the first eight coefficients in a forward diagonal scan order of the top-left 4x4 sub-block. The method 1500 terminates upon completion of step 1540. The secondary transform coefficients generated by operation of step 1540 are encoded into the bitstream by the entropy encoder 338.
23470274_1
[000160] Fig. 16 shows a method 1600 for applying an inverse non-separable secondary transform in the video decoder 134 to the intermediate transform coefficients 432 decoded for a current transform block with size greater than or equal to8x8, producing reconstructed transform coefficients 440. The method 1600 is typically implemented as one or more modules of the application 233, stored in the memory 206 and controlled under execution of the processor 205. The method 1600 begins at a diagonal scan step 1610.
[000161] The intermediate transform coefficients decoded by the entropy decoder 420 are used by the application 233 at step 1610. At the diagonal scan step 1610, the intermediate transform coefficients 432 are rearranged to a one-dimensional list y of S secondary transform coefficients by a forward diagonal scan. If the current transform block has size greater than 8x8, S = 16 (y has sixteen secondary transform coefficients). If the current transform block has size 8x8, S = 8 (y has eight secondary transform coefficients). The method 1600 proceeds from step 1610 to a fetch kernel step 1620.
[000162] At the fetch kernel step 1620, a mode set index m is determined from the intra prediction mode 458 according to Table 1, and a candidate set index c is determined from a nsst-index syntax element decoded from the bitstream 133. The mode set index and candidate set index are used to index into a 4x2x16xP secondary transform kernel P stored in the video decoder 134. If the current transform block has size greater than 8x8, the method 1600 retrieves a 16xP secondary transform matrix F from F[m][c][][ ]. Otherwise, if the current transform block has size 8x8, then the method 1600 retrieves an 8xP secondary transform matrix F from the first eight rows of [m][c][ ][].The method 1600 proceeds from step 1620 to a multiply kernel step 1630.
[000163] At the multiply kernel step 1630, a one-dimensional list of primary transform coefficients x is calculated from the list of secondary transform coefficients y and the secondary transform matrix F retrieved in step 1620. For an 8-bit integerised implementation, the primary transform coefficients x are determined using Equation (2) below:
x[i] = Clip( -LFU][i]* y[j] + 64) » 7) (2)
The step 1530 is implemented by the inverse secondary transform module 436. The method 1600 proceeds from step 1630 to a raster scan step 1640.
23470274_1
[000164] At the raster scan step 1640, the list of primary transform coefficients x is rearranged to a two-dimensional representation of reconstructed transform coefficients 440 by a raster scan over a modified pattern of P primary transform coefficients. The modified pattern may be the modified pattern 1310, the modified pattern 1320, the modified pattern 1330, the modified pattern 1410, the modified pattern 1420, the modified pattern 1430, or may be any pattern of primary transform coefficients obtained by excluding coefficient indices in the preferential order described above with reference to Figs. 13 or 14. The raster scan over the modified pattern is in row major order if IntraPredMode 34. Otherwise, if IntraPredMode > 34 the raster scan over the modified pattern is in column major order. Coefficient indices not in the modified pattern are not populated (ie, remain zero). The method 1600 terminates upon completion of the step 1640. The modified pattern of primary transform coefficients is provided to the inverse primary transform module 444.
[000165] In the arrangements above, the aggregation measure is applied to F[ ][][][p] for each p, resulting in a single preferential order by which primary transform coefficients may be removed from the initial pattern. However, aggregation measures can also be applied to smaller subsets of F, resulting in more than one preferential order for removing primary transform coefficients.
[000166] In one arrangement relating to smaller subsets of F, an aggregation measure is applied to the tensor F[m][ ][ ][p] for each mode set m and primary transform coefficient p, resulting in aggregates V[m][p] =||F[m][ ][][p]|I|I|c°. Then, for each mode set m, the vector V[m][] may be arranged in increasing order, producing a preferential order in which primary transform coefficients are identified for removal from the initial pattern of primary transform coefficients. The preferential orders for each mode set m may be different, therefore modified patterns corresponding to each mode set m resulting from removal of primary transform coefficients may also be different. Typically, the modified pattern of primary transform coefficients is dependent on the mode set m. For example, for the initial pattern of forty-eight primary transform coefficients 1130, the arrangement applying an aggregational measure for each mode set m and primary transform coefficient p identifies preferential orders of coefficient indices with reference to the raster indexing 1220 as follows:
[45,31,46,47,15,23,43,22,42,30,41,14,44,7,39,29,37,40,38,6,21,13,5,36,34,27, 28, 35, 33, 20, 12, 26, 19, 32, 4, 25, 18, 11, 24, 10, 16, 17, 1, 3, 8, 9, 2, 0] when m = 0
23470274_1
[15,31,23,7,14,6,22,30,21,13,29,44,5,4,40,12,47,45,46,20,41,28,27,42,19,35, 39, 36, 38, 37, 43, 34, 26, 33, 11, 18, 32, 25, 10, 17, 3, 24, 9, 2, 16, 0, 1, 8] when m = 1
[31,15,23,30,14,22,7,29,13,47,21,46,6,39,43,45,42,28,20,12,5,35,27,4,41,19, 38, 37, 44, 34, 11, 26, 3, 18, 40, 10, 33, 25, 2, 36, 9, 24, 17, 32, 16, 1, 8, 0] when m = 2
[15,31,7,23,6,22,44,14,13,30,5,29,21,45,40,4,47,46,41,20,12,42,43,39,11,28, 36, 37, 27, 38, 35, 34, 19, 26, 33, 32, 18, 25, 3, 10, 17, 2, 9, 16, 24, 0, 1, 8] when m = 3
[000167] In another arrangement relating to subsets of F, an aggregation measure is applied to the tensor F[ ][c][ ][p] for each candidate set c and primary transform coefficient p, resulting in aggregates V[c] [p] = IIF[ ][c][][p]|Ill°°. Then, for each candidate set c, the vector V[c][] may be arranged in increasing order, producing a preferential order in which primary transform coefficients are identified for removal from the initial pattern of primary transform coefficients. Typically, the modified pattern of primary transform coefficients is dependent on the candidate set c. For example, for the initial pattern of forty-eight primary transform coefficients 1130, the arrangement applying an aggregation measure for each candidate set c and primary transform coefficient p identifies preferential orders of coefficient indices with reference to the raster indexing 1220 as follows:
[15,31,23,7,45,22,14,30,47,6,13,46,29,21,5,41,42,43,37,20,12,44,38,28,39,27, 35, 34, 26, 19, 40, 18, 4, 11, 33, 25, 3, 36, 10, 9, 24, 2, 17, 32, 16, 1, 8, 0] when c = 0
[15,23,31,7,14,22,30,6,29,21,5,13,45,47,28,46,41,20,12,44,27,42,35,39,38,37, 43, 34, 40, 33, 26, 4, 19, 25, 18, 11, 36, 17, 10, 16, 24, 3, 2, 9, 32, 1, 0, 8] when c = 1
[000168] In yet another arrangement relating to subsets of F, an aggregation measure is applied to the vector F[m][c][ ][p] for each combination of mode set m and candidate set c and primary transform coefficient p, resulting in aggregates V[m][c] [p] lIF ][c][ [p]|112. Then, for each combination of mode set m and candidate set c, the vector V[m][c][ ]may be arranged in increasing order, producing a preferential order in which primary transform coefficients are identified for removal from the initial pattern of primary transform coefficients. Typically, the modified pattern of primary transform coefficients is dependent on both the mode set m and the candidate set c. For example, for the initial pattern of fortyeight primary transform coefficients 1130, the arrangement applying an aggregation measure for each combination of mode set m and candidate set c and primary transform coefficient p identifies preferential orders of coefficient indices with reference to the raster indexing 1220 as follows:
23470274_1
[31,45,23,46,47,15,30,43,22,42,41,14,44,7,29,39,21,38,37,40,6,13,33,5,36,12, 28, 35, 27, 20, 34, 26, 19, 32, 18, 4, 11, 25, 24, 3, 9, 10, 16, 1, 17, 8, 2, 0] when m = 0, c = 0
[15,23,31,7,14,6,13,22,30,45,44,21,40,47,29,5,46,4,12,36,41,42,43,37,20,28, 27, 35, 38, 39, 19, 32, 34, 26, 33, 11, 18, 25, 10, 17, 3, 24, 9, 2, 16, 0, 1, 8] when m = 1, c = 0
[31,47,15,39,30,23,43,29,14,22,13,45,46,7,28,21,27,35,42,6,12,20,41,11,38,5, 34, 37, 4, 19, 44, 26, 3, 18, 40, 10, 33, 25, 2, 36, 9, 24, 17, 32, 16, 1, 8, 0] when m = 2, c = 0
[15,31,7,23,6,45,44,22,47,14,13,30,46,40,41,5,29,21,43,4,37,36,42,20,12,38, 32, 39, 11, 28, 27, 35, 34, 26, 19, 33, 18, 25, 3, 17, 10, 2, 24, 9, 16, 1, 0, 8] when m = 3, c = 0
[45,47,44,15,31,46,23,42,43,41,22,7,14,40,30,36,37,6,39,29,38,21,5,13,32,34, 27, 28, 35, 33, 20, 12, 26, 4, 19, 25, 18, 11, 24, 17, 10, 0, 16, 8, 1, 3, 2, 9] when m = 0, c = 1
[15,7,31,23,14,22,6,30,5,21,29,13,44,4,12,40,47,45,20,3,28,46,41,11,19,27,42, 35, 39, 36, 38, 37, 43, 10, 34, 26, 33, 18, 2, 32, 25, 17, 24, 9, 1, 16, 8, 0] when m = 1, c = 1
[31,15,23,30,14,7,22,29,6,13,47,21,46,39,43,20,45,5,42,28,12,35,27,4,41,19, 38, 37, 44, 34, 11, 26, 3, 18, 25, 33, 40, 10, 17, 2, 36, 9, 16, 24, 32, 1, 0, 8] when m = 2, c = 1
[7,23,14,15,31,22,6,30,5,44,21,13,29,12,4,20,45,40,47,28,3,46,41,11,27,19,39, 42, 43, 35, 36, 37, 38, 34, 26, 33, 25, 32, 18, 10, 17, 2, 9, 16, 24, 8, 0, 1] when m = 3, c = 1
[000169] The arrangements using subsets of F demonstrate the freedom of preferential orders that may be generated dependent on both mode set m and candidate set c. One advantage of the arrangements is that each of the modified patterns corresponds more closely to the primary transform coefficients that are important for a non-separable secondary transform with the associated mode set and candidate set. However, supporting eight different patterns may be costly to implement. Fewer patterns may be used while preserving the advantage of the previous arrangements, by grouping mode sets with similar properties. For example, the mode set m = 0 is selected when either the Intra DC or the Intra Planar prediction mode is used. Both Intra DC and Intra Planar are non-directional prediction modes. In contrast, mode sets m = 1,2,3 correspond to intra angular prediction modes which are directional.
[000170] In another arrangement an aggregation measure is applied to the tensor F[0][ ][][p] for each primary transform coefficient p. Another aggregation measure is applied to the tensor F[m][ ][ ][p] for each mode set m # 0 and primary transform coefficient p. Then for the initial
23470274_1 pattern of forty-eight primary transform coefficients 1130, the arrangement applying both aggregation measures identifies preferential orders of coefficient indices with reference to the raster indexing 1220 as follows:
[45,31,46,47,15,23,43,22,42,30,41,14,44,7,39,29,37,40,38,6,21,13,5,36,34,27, 28, 35, 33, 20, 12, 26, 19, 32, 4, 25, 18, 11, 24, 10, 16, 17, 1, 3, 8, 9, 2, 0] when m = 0
[15,31,23,7,22,14,30,6,13,29,21,5,45,47,4,46,20,41,12,44,28,42,39,27,38,37, 43, 35, 34, 19, 26, 11, 40, 18, 33, 3, 10, 25, 36, 2, 9, 24, 17, 32, 16, 1, 8, 0] when m # 0
[000171] Fig. 17 shows a set 1700 of example patterns of primary transform coefficients used by a non-separable secondary transform. The non-separable secondary transform has been modified by setting weights of F[m][ ][ ][p] to zero for coefficient indices p selected according to the preferential orders described above for m = 0 and m # 0. In one arrangement, when m = 0, the first twelve coefficients in the corresponding preferential order above (45, 31, 46, 47, 15, 23, 43, 22, 42, 30, 41, 14) are selected, resulting in a modified pattern 1710. When m , the first twelve coefficients in the corresponding preferential order above (15, 31, 23, 7, 22, 14, 30, 6, 13, 29, 21, 5) are selected, resulting in a modified pattern 1720.
[000172] In the arrangement applying both aggregation measures, the preferential order for mode set m = 0 results in modified patterns of primary transform coefficients which are highly symmetrical. The modified patterns correspond well with the Intra DC and Intra Planar prediction modes in terms of limiting coding cost. In contrast, the preferential order for mode sets m # 0 results in modified patterns of primary transform coefficients which are relatively highly asymmetrical. For IntraPredMode 34, primary transform coefficients corresponding to horizontal high frequencies are preferentially removed first. This corresponds well with intra angular modes with IntraPredMode < 34, as such intra angular modes are predominantly horizontal. Intra angular modes that are predominantly horizontal will tend to produce residuals with horizontal features, and thus have relatively little energy in the horizontal high frequencies. Similarly, for IntraPredMode > 34, the modified pattern for mode sets m # 0 is scanned in column-major order, and therefore the primary transform coefficients corresponding to vertical high frequencies are preferentially removed first.
[000173] In another arrangement, a reduced number of mode sets are stored in the video encoder 114 and the video decoder 134. Table 2 below shows an example mapping from the intra prediction mode parameter 458 to one of three mode set indices. One advantage of
23470274_1 reducing the number of mode sets from four to three is a 25% reduction in the storage requirements for the non-separable secondary transform kernel.
IntraPredMode Mode set mndex IntraPredMode < 0 1 0 <= IntraPredMode <= 1 0 2 <= IntraPredMode <= 18 1 19 <= IntraPredMode <= 49 2 50 <= IntraPredMode<= 80 1 81 <= IntraPredMode<= 83 0 Table 2: Intra prediction mode to mode set index mapping
[000174] In the arrangement with reduced number of mode sets as shown in Table 2, an aggregation measure is applied to the tensor F[O][][ ][p] for each primary transform coefficient p. Another aggregation measure is applied to the tensor F[m][ ][ ][p] for each mode set m # 0 and primary transform coefficient p. For the initial pattern of forty-eight primary transform coefficients 1130, the arrangement with reduced number of mode sets identifies preferential orders of coefficient indices with reference to the raster indexing 1220 as follows:
[45,31,46,47,15,23,43,22,42,30,41,14,44,7,39,29,37,40,38,6,21,13,5,36,34,27, 28, 35, 33, 20, 12, 26, 19, 32, 4, 25, 18, 11, 24, 10, 16, 17, 1, 3, 8, 9, 2, 0] when m = 0
[15,31,23,7,6,22,14,30,13,44,5,29,21,40,4,45,47,46,20,41,12,28,42,39,36,27, 38, 37, 43, 35, 34, 19, 26, 11, 33, 32, 18, 25, 3, 10, 17, 9, 2, 16, 24, 0, 1, 8] when m # 0
[000175] In the arrangements above, the non-separable secondary transform kernel is stored in both the video encoder 114 and the video decoder 134, and has predetermined, fixed weights. In another arrangement, the bitstream 133 may contain a flag allowing the stored non-separable secondary transform kernel to be overridden. If the flag is set, then the bitstream 133 also signals a user-defined secondary transform kernel. The user-defined secondary transform kernel is used instead of the stored secondary transform kernel. One advantage of the arrangements described is that the user-defined secondary transform kernel may be trained on a narrow class of video content that corresponds to the video source 112. A user-defined secondary transform kernel may produce superior coding performance to a predetermined, fixed secondary transform kernel.
[000176] The matrix intra prediction (MIP) mode also uses predetermined, fixed weights which are determined by a training procedure, and stored in the video encoder 114 and video decoder
23470274_1
134. In another arrangement, the bitstream 133 may contain a flag allowing the stored MIP kernel to be overridden. If the flag is set, then the bitstream 133 signals a user-defined MIP kernel. The user-defined MIP kernel is used instead of the stored MIP kernel.
[000177] The arrangements described are applicable to the computer and data processing industries and particularly for the digital signal processing for the encoding a decoding of signals such as video and image signals, achieving high compression efficiency.
[000178] The arrangements described herein allow reduced complexity in a video encoder and corresponding decoder by using the modified patterns described. In particular, required storage capacity required for a secondary transform kernel is to be reduced. Additionally, by reducing the number of residual coefficients used with the secondary transform, the complexity of computations, and thereby the computational congestion incurred in applying the secondary transform is reduced. The reduction in complexity is achieved without unacceptable coding cost as the reduction is based upon columns that are determined to have reduced effect thereon.
[000179] The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.
[000180] In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including", and not "consisting only of'. Variations of the word "comprising", such as "comprise" and "comprises" have correspondingly varied meanings.
23470274_1
Claims (13)
1. A method of decoding an image frame from a video bitstream using a non-separable secondary transform kernel, the method comprising:
arranging a set of decoded intermediate transform coefficients of a coding block of the bitstream based on a scan order;
applying an inverse non-separable secondary transform (NSST) kernel to the decoded intermediate transform coefficients to generate a set of primary transform coefficients, the NSST kernel being orthogonal and having at least one column of the NSST kernel composed of zero-only weights, wherein positions of the set of primary transform coefficients have zero magnitude in positions corresponding to the columns of zero-only weights, and the corresponding positions are discretely distributed in an area of a two-dimensional scan array; and
decoding the image frame based on the generated set of primary transform coefficients of the coding block.
2. The method according to claim 1, wherein the at least one column of zero-only weights is determined based upon preferential order of applying zero-value weighting to columns [63, 15,47,62,7,31,55,23,22,14,54,30,38,39,6,21,45,57,61,13,46,29,5,59,53,60,52, 58,12,20,50,36,37,49,42,28,35,56,41,43,44,27,51,34,48,19,26,4,18,11,33,40,3, 25, 17, 10, 9, 2, 16, 8, 32, 24, 0, 1] for a block of sixty-four primary transform coefficients.
3. The method according to claim 1, wherein the discrete distribution of corresponding positions having zero magnitude primary transform coefficients are based on an order of [63, 15,47,62,7,31,55,23,22,14,54,30,38,39,6,21,45,57,61,13,46,29,5,59,53,60,52, 58,12,20,50,36,37,49,42,28,35,56,41,43,44,27,51,34,48,19,26,4,18,11,33,40,3, 25, 17, 10, 9, 2, 16, 8, 32, 24, 0, 1] for a block of sixty-four primary transform coefficients.
23470274_1
4. The method according to claim 1, wherein the at least one column of zero-only weights is determined based upon preferential order of applying zero-value weighting to columns as the indices [15, 31, 23, 7, 22, 14, 30, 6, 29, 21, 13, 5, 45, 47, 46, 41, 20, 12, 44, 28, 42, 39, 27, 38, 37, 43, 35, 34, 26, 40, 19, 4, 33, 18, 11, 25, 36, 10, 3, 24, 9, 2, 17, 32, 16, 1, 8, 0] for a block of forty-eight primary transform coefficients.
5. The method according to claim 1, wherein the discrete distribution of corresponding positions having zero magnitude primary transform coefficients are based on an order of indices
[15,31,23,7,22,14,30,6,29,21,13,5,45,47,46,41,20,12,44,28,42,39,27,38,37,43, 35, 34, 26, 40, 19, 4, 33, 18, 11, 25, 36, 10, 3, 24, 9, 2, 17, 32, 16, 1, 8, 0] for a block of forty eight transform coefficients.
6. The method according to claim 1, wherein the primary transform coefficients relate to a block of forty-eight transform coefficients and the discrete distribution of transform coefficients relates to positions 15, 31, 23, 7, 22, 14, 30, 6, 29, 21, 13 and 5.
7. The method according to claim 1, wherein the at least one column of zero-only weights is determined based upon effect of the corresponding primary transform positions to coding gain.
8. The method according to claim 1, wherein the zero-only columns are determined based upon a selection of an intra prediction mode set and a candidate set.
9. The method according to claim 1, wherein the zero-only columns are determined based upon a selection of an intra prediction mode set.
10. The method according to claim 1, wherein the discrete distribution is asymmetrical asymmetry and relates to removing primary transform coefficients which correspond to horizontal high frequencies.
11. A non-transitory computer-readable medium having a computer program stored thereon to implement a method of decoding an image frame from a video bitstream using a non separable secondary transform kernel, the method comprising:
23470274_1 arranging a set of decoded intermediate transform coefficients of a coding block of the bitstream based on a scan order; applying an inverse non-separable secondary transform (NSST) kernel to the decoded intermediate transform coefficients to generate a set of primary transform coefficients, the NSST kernel being orthogonal and having at least one column of the NSST kernel composed of zero-only weights, wherein positions of the set of primary transform coefficients have zero magnitude in positions corresponding to the columns of zero-only weights, and the corresponding positions are discretely distributed in an area of a two-dimensional scan array; and decoding the image frame based on the generated set of primary transform coefficients of the coding block.
12. A video decoder configured to implement a method of decoding an image frame from a video bitstream using a non-separable secondary transform kernel, the method comprising:
arranging a set of decoded intermediate transform coefficients of a coding block of the bitstream based on a scan order;
applying an inverse non-separable secondary transform (NSST) kernel to the decoded intermediate transform coefficients to generate a set of primary transform coefficients, the NSST kernel being orthogonal and having at least one column of the NSST kernel composed of zero-only weights, wherein positions of the set of primary transform coefficients have zero magnitude in positions corresponding to the columns of zero-only weights, and the corresponding positions are discretely distributed in an area of a two-dimensional scan array; and
decoding the image frame based on the generated set of primary transform coefficients of the coding block.
23470274_1
13. A system, comprising:
a memory; and
a processor, wherein the processor is configured to execute code stored on the memory for implementing a method of decoding an image frame from a video bitstream using a non separable secondary transform kernel, the method comprising:
arranging a set of decoded intermediate transform coefficients of a coding block of the bitstream based on a scan order;
applying an inverse non-separable secondary transform (NSST) kernel to the decoded intermediate transform coefficients to generate a set of primary transform coefficients, the NSST kernel being orthogonal and having at least one column of the NSST kernel composed of zero-only weights, wherein positions of the set of primary transform coefficients have zero magnitude in positions corresponding to the columns of zero-only weights, and the corresponding positions are discretely distributed in an area of a two-dimensional scan array; and
decoding the image frame based on the generated set of primary transform coefficients of the coding block.
CANON KABUSHIKI KAISHA Patent Attorneys for the Applicant Spruson & Ferguson
23470274_1
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2019232802A AU2019232802A1 (en) | 2019-09-17 | 2019-09-17 | Method, apparatus and system for encoding and decoding an Image frame from a bistream |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2019232802A AU2019232802A1 (en) | 2019-09-17 | 2019-09-17 | Method, apparatus and system for encoding and decoding an Image frame from a bistream |
Publications (1)
Publication Number | Publication Date |
---|---|
AU2019232802A1 true AU2019232802A1 (en) | 2021-04-01 |
Family
ID=75267698
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
AU2019232802A Abandoned AU2019232802A1 (en) | 2019-09-17 | 2019-09-17 | Method, apparatus and system for encoding and decoding an Image frame from a bistream |
Country Status (1)
Country | Link |
---|---|
AU (1) | AU2019232802A1 (en) |
-
2019
- 2019-09-17 AU AU2019232802A patent/AU2019232802A1/en not_active Abandoned
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US12022133B2 (en) | Method, apparatus and system for encoding and decoding a tree of blocks of video samples | |
AU2020201753B2 (en) | Method, apparatus and system for encoding and decoding a block of video samples | |
US11949857B2 (en) | Method, apparatus and system for encoding and decoding a tree of blocks of video samples | |
US20220150509A1 (en) | Method, apparatus and system for encoding and decoding a tree of blocks of video samples | |
WO2021127723A1 (en) | Method, apparatus and system for encoding and decoding a block of video samples | |
WO2021051155A1 (en) | Method, apparatus and system for encoding and decoding a block of video samples | |
CN114365491A (en) | Method, apparatus and system for encoding and decoding a block of video samples | |
AU2021273633B2 (en) | Method, apparatus and system for encoding and decoding a block of video samples | |
AU2019232802A1 (en) | Method, apparatus and system for encoding and decoding an Image frame from a bistream | |
AU2020202057A1 (en) | Method, apparatus and system for encoding and decoding a block of video samples | |
AU2020202285A1 (en) | Method, apparatus and system for encoding and decoding a block of video samples | |
AU2019203981A1 (en) | Method, apparatus and system for encoding and decoding a block of video samples |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
MK4 | Application lapsed section 142(2)(d) - no continuation fee paid for the application |