TW201946468A

TW201946468A - Encoding device, decoding device, encoding method, and decoding method

Info

Publication number: TW201946468A
Application number: TW108112911A
Authority: TW
Inventors: 安倍清史; 加納龍一; 遠間正真; 西孝啓
Original assignee: 美商松下電器（美國）知識產權公司
Priority date: 2018-04-17
Filing date: 2019-04-12
Publication date: 2019-12-01
Also published as: WO2019203036A1

Abstract

This encoding device performs inter-prediction when encoding an encoding object block constituting a moving image, and is provided with a circuit and a memory, wherein in the case when the circuit performs, by using the memory, a first process for correcting a prediction image of the encoding object block by utilizing the luminance around the encoding object block and the luminance around a reference block, the circuit performs inter-prediction on the encoding object block without performing one or more processes including a second process for correcting the prediction image using the spatial gradient of the luminance obtained from the reference block.

Description

編碼裝置、解碼裝置、編碼方法及解碼方法Encoding device, decoding device, encoding method and decoding method

發明領域
本揭示是關於在構成動態圖像之編碼對象區塊之編碼中進行幀間預測之編碼裝置等。FIELD OF THE DISCLOSURE The present disclosure relates to an encoding device and the like for performing inter prediction in encoding of a coding target block constituting a moving image.

發明背景
習知，有被稱作HEVC(High Efficiency Video Coding，高效率視訊編碼)之H.265來作為用於將動態圖像編碼之規格(非專利文獻1)。
先行技術文獻BACKGROUND OF THE INVENTION It is conventionally known that there is H.265 called High Efficiency Video Coding (HEVC) as a specification for encoding a moving image (Non-Patent Document 1).
Advance technical literature

非專利文獻
非專利文獻1：H.265(ISO/IEC 23008-2 HEVC)/HEVC(High Efficiency Video Coding)Non-Patent Literature Non-Patent Literature 1: H.265 (ISO / IEC 23008-2 HEVC) / HEVC (High Efficiency Video Coding)

發明概要
發明欲解決之課題
然而，在動態圖像之編碼等，如果未有效率地進行處理，則可能發生處理之延遲。SUMMARY OF THE INVENTION Problems to be Solved by the Invention However, delays in processing may occur if coding of moving images is not performed efficiently.

於是，本揭示是提供可在動態圖像之編碼中將處理之延遲予以抑制之編碼裝置等。
用以解決課題之手段Therefore, the present disclosure is to provide an encoding device and the like which can suppress processing delay in encoding of a moving image.
Means to solve the problem

與本揭示之一態樣相關之編碼裝置是在構成動態圖像之編碼對象區塊之編碼中進行幀間預測之編碼裝置，前述編碼裝置具備電路與記憶體，前述電路是使用前述記憶體，當進行使用前述編碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述編碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述編碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。An encoding device related to one aspect of the present disclosure is an encoding device that performs inter prediction in encoding of an encoding target block constituting a moving image. The foregoing encoding device includes a circuit and a memory. The foregoing circuit uses the foregoing memory. When the first process of correcting the predicted image of the coding target block using the brightness of the coding target block's periphery and the reference block's surrounding luminance is performed, one or more processes including the second process are not performed. Processing to perform inter prediction of the encoding target block, and the second processing is to correct the predicted image using a spatial gradient of luminance obtained from the reference block.

另，該等總括或具體之態樣可以是藉由系統、裝置、方法、積體電路、電腦程式、或電腦可讀取之CD-ROM等之非暫時性記錄媒體而實現，亦可以是藉由系統、裝置、方法、積體電路、電腦程式、及記錄媒體之任意組合而實現。
發明效果In addition, these collective or specific aspects may be realized by a non-transitory recording medium such as a system, a device, a method, an integrated circuit, a computer program, or a computer-readable CD-ROM. It is realized by any combination of system, device, method, integrated circuit, computer program, and recording medium.
Invention effect

與本揭示之一態樣相關之編碼裝置等可在動態圖像之編碼將處理之延遲予以抑制。An encoding device or the like related to one aspect of the present disclosure can suppress processing delay in encoding of a moving image.

用以實施發明之形態
(作為本揭示之基礎之見解)
例如，編碼裝置是依照各區塊而將圖像編碼。編碼裝置在依照各區塊而將圖像編碼時，可以是使用幀間預測，亦可以是使用幀內預測。當在處理對象區塊之編碼使用幀間預測的情況下，編碼裝置是特定出參考區塊，並參考參考區塊而生成處理對象區塊之預測圖像。然後，編碼裝置是將處理對象區塊之預測圖像與處理對象區塊之原圖像的差分圖像編碼，藉此減少碼量。Forms used to implement the invention
(The insights that underlie this disclosure)
For example, the encoding device encodes an image in accordance with each block. When the encoding device encodes an image according to each block, it may use inter prediction or intra prediction. When inter-prediction is used for the encoding of the processing target block, the encoding device specifies a reference block and refers to the reference block to generate a prediction image of the processing target block. Then, the encoding device encodes a difference image between the predicted image of the processing target block and the original image of the processing target block, thereby reducing the code amount.

又，解碼裝置是在解碼圖像時將差分圖像解碼。然後，解碼裝置是參考參考區塊而生成處理對象區塊之預測圖像，藉由將預測圖像與差分圖像相加而重建原圖像。藉此，解碼裝置可解碼圖像。The decoding device decodes a differential image when decoding an image. Then, the decoding device refers to the reference block to generate a prediction image of the processing target block, and reconstructs the original image by adding the prediction image and the difference image. Thereby, the decoding device can decode an image.

又，例如，編碼裝置及解碼裝置亦可以為了令預測圖像接近原圖像，而對預測圖像進行修正處理。修正處理亦可以包含被稱作LIC(local illumination compensation，局部光照補償)處理之亮度修正處理。具體而言，編碼裝置及解碼裝置是使用處理對象區塊之周邊之亮度與參考區塊之周邊之亮度而導出亮度修正參數，並使用亮度修正參數而修正預測圖像。藉此，令碼量更加減少。In addition, for example, the encoding device and the decoding device may perform correction processing on the predicted image in order to bring the predicted image closer to the original image. The correction process may include a brightness correction process called a LIC (local illumination compensation, local illumination compensation) process. Specifically, the encoding device and the decoding device use the brightness of the periphery of the processing target block and the brightness of the periphery of the reference block to derive brightness correction parameters, and use the brightness correction parameters to correct the predicted image. With this, the code size is further reduced.

然而，在生成處理對象區塊之周邊之重建圖像前，編碼裝置及解碼裝置無法利用處理對象區塊之周邊之亮度。所以，有因為亮度修正處理等而發生延遲之可能性。However, before generating a reconstructed image around the processing target block, the encoding device and decoding device cannot use the brightness around the processing target block. Therefore, there is a possibility that a delay occurs due to brightness correction processing and the like.

於是，例如，本揭示之一態樣之編碼裝置是在構成動態圖像之編碼對象區塊之編碼中進行幀間預測之編碼裝置，前述編碼裝置具備電路與記憶體，前述電路是使用前述記憶體，當進行使用前述編碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述編碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述編碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。Therefore, for example, an encoding device of one aspect of the present disclosure is an encoding device that performs inter prediction in encoding of a coding target block constituting a moving image. The encoding device includes a circuit and a memory. The circuit uses the memory In the case of performing the first process of correcting the predicted image of the aforementioned coding target block using the brightness of the periphery of the aforementioned coding target block and the brightness of the periphery of the reference block, one including the second process is not performed. The above processing performs inter prediction of the encoding target block, and the second processing uses the spatial gradient of the luminance obtained from the reference block to correct the predicted image.

藉此，編碼裝置可在進行被設想成處理時間長之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行幀間預測，前述第2處理是與第1處理同樣地被設想成處理時間長。所以，編碼裝置可縮短包含第1處理之階段的處理時間，可在動態圖像之編碼將處理之延遲予以抑制。With this, the encoding device can perform inter prediction without performing one or more processes including the second process when the first process is supposed to take a long processing time. The second process is the same as the first process. The ground is envisaged as a long processing time. Therefore, the encoding device can shorten the processing time including the first processing stage, and can suppress the processing delay in the encoding of the moving image.

又，例如，前述1個以上之處理包含第3處理，前述第3處理是使用參考圖像來修正前述預測圖像，且前述參考圖像是使用前述編碼對象區塊之周邊之已編碼區塊之運動向量而從前述編碼對象區塊指出之參考圖像。In addition, for example, the one or more processes include a third process. The third process uses a reference image to modify the predicted image, and the reference image is a coded block using a periphery of the coding target block. The motion vector is a reference image pointed out from the aforementioned coding target block.

藉此，編碼裝置可在進行第1處理的情況下，不進行第2處理及第3處理而進行幀間預測。所以，編碼裝置可縮短包含第1處理之階段的處理時間，可在動態圖像之編碼中將處理之延遲予以抑制。This allows the encoding device to perform inter prediction without performing the second processing and the third processing when the first processing is performed. Therefore, the encoding device can shorten the processing time including the first processing stage, and can suppress the processing delay in the encoding of the moving image.

又，例如，前述1個以上之處理是修正前述預測圖像之複數個處理中之與前述第1處理不同之所有處理。In addition, for example, the one or more processes are all processes different from the first process among the plurality of processes for correcting the predicted image.

藉此，編碼裝置可在進行第1處理的情況下，不進行第1處理以外之修正處理而進行幀間預測。所以，編碼裝置可縮短包含第1處理之階段的處理時間，可在動態圖像之編碼中將處理之延遲予以抑制。This allows the encoding device to perform inter prediction without performing correction processing other than the first processing when the first processing is performed. Therefore, the encoding device can shorten the processing time including the first processing stage, and can suppress the processing delay in the encoding of the moving image.

又，例如，前述電路在進行前述第1處理後，不修正前述預測圖像而結束前述編碼對象區塊之幀間預測。In addition, for example, after performing the first process, the circuit ends the inter prediction of the encoding target block without modifying the prediction image.

藉此，編碼裝置可在進行第1處理的情況下，在第1處理之後不進行修正處理而進行幀間預測。所以，編碼裝置可縮短包含第1處理之階段的處理時間，可在動態圖像之編碼中將處理之延遲予以抑制。With this, when the encoding device performs the first processing, it is possible to perform inter prediction without performing the correction processing after the first processing. Therefore, the encoding device can shorten the processing time including the first processing stage, and can suppress the processing delay in the encoding of the moving image.

又，例如，前述電路判定是否以低延遲模式進行前述編碼對象區塊之編碼，當在判定成以前述低延遲模式進行前述編碼對象區塊之編碼的情況下進行前述第1處理的情況下，不進行前述1個以上之處理而進行前述編碼對象區塊之幀間預測。In addition, for example, the circuit determines whether to encode the encoding target block in a low-latency mode, and when the first processing is performed when it is determined to encode the encoding target block in the low-latency mode, Inter-prediction of the encoding target block is performed without performing the aforementioned one or more processes.

藉此，編碼裝置可在低延遲模式縮短包含第1處理之階段的處理時間，可在動態圖像之編碼中將處理之延遲予以抑制。Thereby, the encoding device can shorten the processing time of the stage including the first processing in the low-latency mode, and can suppress the processing delay in the encoding of the moving image.

又，例如，前述電路將表示是否以前述低延遲模式進行前述編碼對象區塊之編碼之資訊，編碼在前述動態圖像之編碼串流之序列標頭區域、圖片標頭區域、圖塊(tile)標頭區域、片段(slice)標頭區域及補助資訊區域之至少其中1者。Also, for example, the foregoing circuit will indicate whether to encode the encoding target block in the low-latency mode, and encode the sequence header area, picture header area, and tile (tile) of the encoding stream of the moving image. ) At least one of a header area, a slice header area, and a subsidy information area.

藉此，編碼裝置可將是否為低延遲模式之資訊給予解碼裝置。Thereby, the encoding device can give the decoding device information about whether it is a low-latency mode.

又，例如，前述電路是依循包含前述編碼對象區塊之編碼對象圖片的大小，而判定是否以前述低延遲模式進行前述編碼對象區塊之編碼。In addition, for example, the circuit determines whether to encode the encoding target block in the low delay mode according to the size of the encoding target picture including the encoding target block.

藉此，編碼裝置可依循圖片的大小，而適應性地控制是否以低延遲模式進行編碼。With this, the encoding device can adaptively control whether to encode in a low-latency mode according to the size of the picture.

又，例如，前述電路是依循解碼裝置之處理能力，而判定是否以前述低延遲模式進行前述編碼對象區塊之編碼。In addition, for example, the circuit determines whether to encode the encoding target block in the low-latency mode according to the processing capability of the decoding device.

藉此，編碼裝置可依循解碼裝置之處理能力，而適應性地控制是否以低延遲模式進行編碼。Thereby, the encoding device can adaptively control whether to encode in the low-latency mode according to the processing capability of the decoding device.

又，例如，前述電路是依循分配給前述動態圖像之編碼串流之設定檔(profile)或等級(level)資訊，而判定是否以前述低延遲模式進行前述編碼對象區塊之編碼。In addition, for example, the circuit determines whether to encode the encoding target block in the low-latency mode according to profile or level information of the encoding stream allocated to the moving image.

藉此，編碼裝置可依循設定檔或等級，而適應性地控制是否以低延遲模式進行編碼。Thereby, the encoding device can adaptively control whether to encode in a low-latency mode according to a profile or level.

又，例如，本揭示之一態樣之解碼裝置是在構成動態圖像之解碼對象區塊之解碼中進行幀間預測之解碼裝置，前述解碼裝置具備電路與記憶體，前述電路是使用前述記憶體，當進行使用前述解碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述解碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述解碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。In addition, for example, a decoding device according to one aspect of the present disclosure is a decoding device that performs inter prediction during decoding of a decoding target block constituting a moving image. The decoding device includes a circuit and a memory, and the circuit uses the memory. In the case of performing the first processing of correcting the predicted image of the decoding target block by using the brightness of the surroundings of the decoding target block and the brightness of the reference block, the first processing including the second processing is not performed. The above processing performs inter prediction of the decoding target block, and the second processing is to use the spatial gradient of the luminance obtained from the reference block to correct the predicted image.

藉此，解碼裝置可在進行被設想成處理時間長之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行幀間預測，前述第2處理是與第1處理同樣地被設想成處理時間長。所以，解碼裝置可縮短包含第1處理之階段的處理時間，可在動態圖像之解碼中將處理之延遲予以抑制。This allows the decoding device to perform inter prediction without performing one or more processes including the second process when the first process is supposed to take a long processing time. The second process is the same as the first process. The ground is envisaged as a long processing time. Therefore, the decoding device can shorten the processing time including the first processing stage, and can suppress the processing delay in decoding the moving image.

又，例如，前述1個以上之處理包含第3處理，前述第3處理是使用參考圖像來修正前述預測圖像，且前述參考圖像是使用前述解碼對象區塊之周邊之已解碼區塊之運動向量而從前述解碼對象區塊指出之參考圖像。In addition, for example, the one or more processes include a third process. The third process uses a reference image to modify the predicted image, and the reference image is a decoded block using a periphery of the decoding target block. The motion vector is a reference image pointed out from the aforementioned decoding target block.

藉此，解碼裝置可在進行第1處理的情況下，不進行第2處理及第3處理而進行幀間預測。所以，解碼裝置可縮短包含第1處理之階段的處理時間，可在動態圖像之解碼中將處理之延遲予以抑制。This allows the decoding device to perform inter prediction without performing the second processing and the third processing when the first processing is performed. Therefore, the decoding device can shorten the processing time including the first processing stage, and can suppress the processing delay in decoding the moving image.

藉此，解碼裝置可在進行第1處理的情況下，不進行第1處理以外之修正處理而進行幀間預測。所以，解碼裝置可縮短包含第1處理之階段的處理時間，可在動態圖像之解碼中將處理之延遲予以抑制。This allows the decoding device to perform inter prediction without performing correction processing other than the first processing when the first processing is performed. Therefore, the decoding device can shorten the processing time including the first processing stage, and can suppress the processing delay in decoding the moving image.

又，例如，前述電路在進行前述第1處理後，不修正前述預測圖像的方式而結束前述解碼對象區塊之幀間預測。In addition, for example, after performing the first process, the circuit ends the inter prediction of the decoding target block without correcting the manner of predicting the image.

藉此，解碼裝置可在進行第1處理的情況下，在第1處理之後不進行修正處理而進行幀間預測。所以，解碼裝置可縮短包含第1處理之階段的處理時間，可在動態圖像之解碼中將處理之延遲予以抑制。With this, when the decoding device performs the first processing, it is possible to perform inter prediction without performing the correction processing after the first processing. Therefore, the decoding device can shorten the processing time including the first processing stage, and can suppress the processing delay in decoding the moving image.

又，例如，前述電路判定是否以低延遲模式進行前述解碼對象區塊之解碼，當在判定成以前述低延遲模式進行前述解碼對象區塊之解碼的情況下進行前述第1處理的情況下，不進行前述1個以上之處理而進行前述解碼對象區塊之幀間預測。In addition, for example, the circuit determines whether the decoding target block is decoded in the low-latency mode, and when the first process is performed when it is determined that the decoding target block is decoded in the low-latency mode, Inter-prediction of the decoding target block is performed without performing the aforementioned one or more processes.

藉此，解碼裝置可在低延遲模式縮短包含第1處理之階段的處理時間，可在動態圖像之解碼中將處理之延遲予以抑制。Thereby, the decoding device can shorten the processing time of the stage including the first processing in the low-latency mode, and can suppress the processing delay in the decoding of the moving image.

又，例如，前述電路從前述動態圖像之編碼串流之序列標頭區域、圖片標頭區域、圖塊標頭區域、片段標頭區域及補助資訊區域之至少其中1者，解碼表示是否以前述低延遲模式進行前述解碼對象區塊之解碼之資訊。Also, for example, the foregoing circuit decodes whether at least one of a sequence header area, a picture header area, a tile header area, a fragment header area, and an auxiliary information area of the encoded stream of the moving image is decoded to indicate whether The aforementioned low-latency mode performs decoding information of the aforementioned decoding target block.

藉此，解碼裝置可從編碼裝置獲得是否為低延遲模式之資訊。Thereby, the decoding device can obtain information from the encoding device as to whether it is a low-latency mode.

又，例如，前述電路是依循分配給前述動態圖像之編碼串流之設定檔或等級資訊，而判定是否以前述低延遲模式進行前述解碼對象區塊之解碼。In addition, for example, the circuit determines whether to decode the decoding target block in the low-latency mode according to the profile or level information of the encoded stream allocated to the moving image.

藉此，解碼裝置可依循設定檔或等級，而適應性地控制是否以低延遲模式進行解碼。Thereby, the decoding device can adaptively control whether to perform decoding in a low-latency mode according to a set file or level.

又，例如，本揭示之一態樣之編碼方法是在構成動態圖像之編碼對象區塊之編碼中進行幀間預測之編碼方法，當進行使用前述編碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述編碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述編碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。In addition, for example, one aspect of the present disclosure is an encoding method for performing inter prediction in the encoding of an encoding target block constituting a moving image. When performing the use of the brightness and reference area around the encoding target block, In the case of the first processing of correcting the predicted image of the coding target block by the brightness around the block, the inter prediction of the coding target block is performed without performing one or more processes including the second processing. The 2 process is to use the spatial gradient of the brightness obtained from the aforementioned reference block to modify the aforementioned predicted image.

藉此，可在進行被設想成處理時間長之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行幀間預測，前述第2處理是與第1處理同樣地被設想成處理時間長。所以，可縮短包含第1處理之階段的處理時間，可在動態圖像之編碼中將處理之延遲予以抑制。Thereby, when the first process which is supposed to have a long processing time is performed, the inter prediction is performed without performing one or more processes including the second process. The second process is performed in the same manner as the first process. It is assumed that the processing time is long. Therefore, the processing time in the stage including the first processing can be shortened, and the processing delay can be suppressed in the coding of the moving image.

又，例如，本揭示之一態樣之解碼方法是在構成動態圖像之解碼對象區塊之解碼中進行幀間預測之解碼方法，當進行使用前述解碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述解碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述解碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。Also, for example, a decoding method of one aspect of the present disclosure is a decoding method for performing inter prediction in decoding of a decoding target block constituting a moving image. When performing the use of the brightness and reference area around the decoding target block, In the case of the first processing of correcting the predicted image of the decoding target block by the brightness around the block, the inter prediction of the decoding target block is performed without performing one or more processes including the second processing. The 2 process is to use the spatial gradient of the brightness obtained from the aforementioned reference block to modify the aforementioned predicted image.

藉此，可在進行被設想成處理時間長之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行幀間預測，前述第2處理是與第1處理同樣地被設想成處理時間長。所以，可縮短包含第1處理之階段的處理時間，可在動態圖像之解碼中將處理之延遲予以抑制。Thereby, when the first process which is supposed to have a long processing time is performed, the inter prediction is performed without performing one or more processes including the second process. The second process is performed in the same manner as the first process. It is assumed that the processing time is long. Therefore, the processing time including the first processing stage can be shortened, and the processing delay can be suppressed in the decoding of the moving image.

又，例如，本揭示之一態樣之編碼裝置具備分割部、幀內預測部、幀間預測部、轉換部、量化部、熵編碼部、迴路濾波部。前述分割部將動態圖像分割成複數個區塊。前述幀內預測部對包含於前述複數個區塊之區塊進行幀內預測。前述幀間預測部對前述區塊進行幀間預測。前述轉換部將藉由前述幀內預測或前述幀間預測所獲得之預測圖像與原圖像的預測誤差轉換，生成轉換係數。前述量化部將前述轉換係數量化而生成量化係數。前述熵編碼部將前述量化係數編碼而生成編碼位元串流。前述迴路濾波部對使用前述預測圖像而生成之重建圖像適用濾波。In addition, for example, the encoding device according to one aspect of the present disclosure includes a division unit, an intra prediction unit, an inter prediction unit, a conversion unit, a quantization unit, an entropy encoding unit, and a loop filtering unit. The division unit divides the moving image into a plurality of blocks. The intra prediction unit performs intra prediction on a block included in the plurality of blocks. The inter prediction unit performs inter prediction on the block. The conversion unit converts a prediction error between the predicted image obtained by the intra prediction or the inter prediction and the original image to generate a conversion coefficient. The quantization unit quantizes the conversion coefficient to generate a quantization coefficient. The entropy encoding unit encodes the quantization coefficient to generate a coded bit stream. The loop filtering unit applies filtering to a reconstructed image generated using the predicted image.

又，例如，前述編碼裝置是在構成動態圖像之編碼對象區塊之編碼中進行幀間預測。前述幀間預測部是當進行使用前述編碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述編碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述編碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。In addition, for example, the encoding device performs inter prediction during encoding of an encoding target block constituting a moving image. When the inter prediction unit performs the first process of correcting the predicted image of the coding target block using the brightness of the coding target block's periphery and the reference block's surrounding luminance, the second prediction process is not performed. One or more processes perform inter prediction of the encoding target block, and the second process corrects the predicted image using a spatial gradient of luminance obtained from the reference block.

又，例如，本揭示之一態樣之解碼裝置具備熵解碼部、反量化部、反轉換部、幀內預測部、幀間預測部、迴路濾波部。前述熵解碼部從編碼位元串流解碼構成動態圖像之區塊之量化係數。前述反量化部將前述量化係數反量化而取得轉換係數。前述反轉換部將前述轉換係數反轉換而取得預測誤差。前述幀內預測部對前述區塊進行幀內預測。前述幀間預測部對前述區塊進行幀間預測。前述迴路濾波部對使用藉由前述幀內預測或前述幀間預測所獲得之預測圖像與前述預測誤差而生成之重建圖像適用濾波。In addition, for example, a decoding device according to one aspect of the present disclosure includes an entropy decoding unit, an inverse quantization unit, an inverse conversion unit, an intra prediction unit, an inter prediction unit, and a loop filtering unit. The aforementioned entropy decoding unit decodes a quantization coefficient of a block constituting a moving image from a coded bit stream. The inverse quantization unit inversely quantizes the quantization coefficient to obtain a conversion coefficient. The inverse conversion unit inversely converts the conversion coefficient to obtain a prediction error. The intra prediction unit performs intra prediction on the block. The inter prediction unit performs inter prediction on the block. The in-loop filtering section applies filtering to a reconstructed image generated by using a prediction image obtained by the intra prediction or the inter prediction and the prediction error.

又，例如，前述解碼裝置是在構成動態圖像之解碼對象區塊之解碼中進行幀間預測。前述幀間預測部是當進行使用前述解碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述解碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述解碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。In addition, for example, the aforementioned decoding device performs inter prediction during decoding of a decoding target block constituting a moving image. When the inter prediction unit performs the first process of correcting the predicted image of the decoding target block by using the brightness of the periphery of the decoding target block and the brightness of the reference block, it does not include the second One or more processes perform inter prediction of the decoding target block, and the second process corrects the predicted image using a spatial gradient of luminance obtained from the reference block.

再者，該等總括或具體之態樣可以是藉由系統、裝置、方法、積體電路、電腦程式、或電腦可讀取之CD-ROM等之非暫時性記錄媒體而實現，亦可以是藉由系統、裝置、方法、積體電路、電腦程式、及記錄媒體之任意組合而實現。Furthermore, these collective or specific aspects may be realized by a non-transitory recording medium such as a system, device, method, integrated circuit, computer program, or computer-readable CD-ROM, or may be It is realized by any combination of a system, a device, a method, an integrated circuit, a computer program, and a recording medium.

以下，一面參考圖面一面具體地說明實施形態。Hereinafter, embodiments will be specifically described with reference to the drawings.

另，以下說明之實施形態皆是顯示總括或具體之例。在以下之實施形態顯示之數值、形狀、材料、構成要素、構成要素之配置位置及連接形態、步驟、步驟之順序等只是一例，其主旨並非限定申請專利範圍。又，以下之實施形態之構成要素之中，針對未記載於顯示最上位概念之獨立請求項之構成要素，是作為任意之構成要素而說明。
(實施形態1)It should be noted that the embodiments described below are all summary or specific examples. The numerical values, shapes, materials, constituent elements, arrangement positions and connection forms, steps, and order of the steps shown in the following embodiments are merely examples, and the gist is not intended to limit the scope of patent application. In addition, among the constituent elements of the following embodiments, constituent elements that are not described in the independent request item showing the highest-level concept are described as arbitrary constituent elements.
(Embodiment 1)

首先，說明實施形態1之概要，來當作可適用後述之本揭示之各態樣中說明之處理及/或構成之編碼裝置及解碼裝置的一例。不過，實施形態1只不過是可適用本揭示之各態樣中說明之處理及/或構成之編碼裝置及解碼裝置的一例，在本揭示之各態樣中說明之處理及/或構成亦可在與實施形態1不同之編碼裝置及解碼裝置中實施。First, the outline of the first embodiment will be described as an example of an encoding device and a decoding device to which the processing and / or configuration described in the aspects of the present disclosure described later can be applied. However, the first embodiment is merely an example of an encoding device and a decoding device to which the processing and / or configuration described in each aspect of the present disclosure can be applied, and the processing and / or configuration described in each aspect of the present disclosure may also be applied. It is implemented in an encoding device and a decoding device different from those in the first embodiment.

關於對實施形態1適用在本揭示之各態樣中說明之處理及/或構成，例如亦可以是進行以下之任一者：
(1)對於實施形態1之編碼裝置或解碼裝置，將構成該編碼裝置或解碼裝置之複數個構成要素中之與在本揭示之各態樣中說明之構成要素對應之構成要素，替換成在本揭示之各態樣中說明之構成要素；
(2)對於實施形態1之編碼裝置或解碼裝置，針對構成該編碼裝置或解碼裝置之複數個構成要素中之一部分之構成要素來進行功能或實施之處理之追加、替換、刪除等任意之變更，並且，將與在本揭示之各態樣中說明之構成要素對應之構成要素，替換成在本揭示之各態樣中說明之構成要素；
(3)對於實施形態1之編碼裝置或解碼裝置所實施之方法，進行處理之追加、及/或針對該方法所含有之複數個處理中之一部分之處理進行替換、刪除等任意之變更，並且，將與在本揭示之各態樣中說明之處理對應之處理，替換成在本揭示之各態樣中說明之處理；
(4)把構成實施形態1之編碼裝置或解碼裝置之複數個構成要素中之一部分之構成要素，與以下之構成要素組合而實施：在本揭示之各態樣中說明之構成要素、將在本揭示之各態樣中說明之構成要素所具備之功能之一部分含括在內之構成要素、或是將在本揭示之各態樣中說明之構成要素所實施之處理之一部分予以實施之構成要素；
(5)把將構成實施形態1之編碼裝置或解碼裝置之複數個構成要素中之一部分之構成要素所具備之功能之一部分含括在內之構成要素、或是將構成實施形態1之編碼裝置或解碼裝置之複數個構成要素中之一部分之構成要素所實施之處理之一部分予以實施之構成要素，與以下之構成要素組合而實施：在本揭示之各態樣中說明之構成要素、將在本揭示之各態樣中說明之構成要素所具備之功能之一部分含括在內之構成要素、或是將在本揭示之各態樣中說明之構成要素所實施之處理之一部分予以實施之構成要素；
(6)對於實施形態1之編碼裝置或解碼裝置所實施之方法，將該方法含有之複數個處理中之與在本揭示之各態樣中說明之處理對應之處理，替換成在本揭示之各態樣中說明之處理；
(7)把實施形態1之編碼裝置或解碼裝置所實施之方法含有之複數個處理中之一部分之處理，與在本揭示之各態樣中說明之處理組合而實施。
另，在本揭示之各態樣中說明之處理及/或構成之實施法並非限定於上述之例。例如，亦可以實施於以與在實施形態1揭示之動態圖像/圖像編碼裝置或動態圖像/圖像解碼裝置不同之目的而利用之裝置，亦可以將在各態樣中說明之處理及/或構成予以單獨實施。又，亦可以將在不同之態樣中說明之處理及/或構成予以組合而實施。
[編碼裝置之概要]Regarding application of the processing and / or configuration described in each aspect of the present disclosure to Embodiment 1, for example, any one of the following may be performed:
(1) For the encoding device or the decoding device according to the first embodiment, among the plurality of components constituting the encoding device or the decoding device, the components corresponding to the components described in the aspects of the present disclosure are replaced with The constituent elements described in the aspects of this disclosure;
(2) With regard to the encoding device or the decoding device according to the first embodiment, arbitrary changes such as additions, replacements, and deletions of functions or processes performed on the constituent elements constituting one of a plurality of constituent elements of the encoding device or the decoding device And replace the constituent elements corresponding to the constituent elements described in the aspects of the present disclosure with the constituent elements described in the aspects of the present disclosure;
(3) the method implemented by the encoding device or the decoding device according to the first embodiment, the processing is added, and / or any one of a plurality of processing included in the method is replaced or deleted, and , Replace the processing corresponding to the processing described in the aspects of this disclosure with the processing described in the aspects of this disclosure;
(4) The components constituting one of the plurality of components constituting the encoding device or the decoding device according to the first embodiment are combined with the following components and implemented. The components described in the aspects of this disclosure will be described in The constituent elements included in the aspects described in the aspects of the present disclosure include a constituent element that is included in the functional elements described in the aspects of the present disclosure, or a component that implements a part of the processing performed by the constituent elements described in the aspects of the present disclosure. Elements
(5) A constituent element including a part of the functions of the constituent elements constituting one of the plural constituent elements of the encoding apparatus or the decoding apparatus according to the first embodiment, or the encoding apparatus constituting the first embodiment. Or a component that is implemented by one of the components of a plurality of components of the decoding device and is implemented in combination with the following components: The components described in the aspects of this disclosure will be described in The constituent elements included in the aspects described in the aspects of the present disclosure include a constituent element that is included in the functional elements described in the aspects of the present disclosure, or a component that implements a part of the processing performed by the constituent elements described in the aspects of the present disclosure. Elements
(6) For the method implemented by the encoding device or the decoding device according to the first embodiment, the processing corresponding to the processing described in the aspects of the present disclosure among the plurality of processes included in the method is replaced with the one described in the present disclosure. The treatment described in each aspect;
(7) A combination of the processing included in the plurality of processing included in the method implemented by the encoding device or the decoding device of Embodiment 1 and the processing described in each aspect of the present disclosure is implemented.
In addition, the implementation method of the process and / or structure demonstrated in each aspect of this indication is not limited to the said example. For example, it may be implemented in a device that is used for a different purpose from the moving image / image encoding device or the moving image / image decoding device disclosed in the first embodiment, and the processing described in each aspect may be implemented And / or composition is implemented separately. Furthermore, the processes and / or configurations described in different aspects may be combined and implemented.
[Outline of encoding device]

首先，說明實施形態1之編碼裝置之概要。圖1是顯示與實施形態1之編碼裝置100之功能構成的方塊圖。編碼裝置100是將動態圖像/圖像以區塊單位來編碼之動態圖像/圖像編碼裝置。First, the outline of the encoding device according to the first embodiment will be described. FIG. 1 is a block diagram showing a functional configuration of the encoding device 100 according to the first embodiment. The encoding device 100 is a moving image / image encoding device that encodes moving images / images in units of blocks.

如圖1所示，編碼裝置100是將圖像以區塊單位來編碼之裝置，具有分割部102、相減部104、轉換部106、量化部108、熵編碼部110、反量化部112、反轉換部114、相加部116、區塊記憶體118、迴路濾波部120、幀記憶體122、幀內預測部124、幀間預測部126、預測控制部128。As shown in FIG. 1, the encoding device 100 is a device that encodes an image in units of blocks. The inverse conversion unit 114, the addition unit 116, the block memory 118, the loop filtering unit 120, the frame memory 122, the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128.

編碼裝置100例如是藉由通用處理器及記憶體而實現。此情況下，當藉由處理器來執行儲存在記憶體之軟體程式時，處理器是作為分割部102、相減部104、轉換部106、量化部108、熵編碼部110、反量化部112、反轉換部114、相加部116、迴路濾波部120、幀內預測部124、幀間預測部126及預測控制部128來發揮功能。又，編碼裝置100亦可以以對應於分割部102、相減部104、轉換部106、量化部108、熵編碼部110、反量化部112、反轉換部114、相加部116、迴路濾波部120、幀內預測部124、幀間預測部126及預測控制部128之專用之1個以上之電子電路而實現。The encoding device 100 is implemented by, for example, a general-purpose processor and a memory. In this case, when the software program stored in the memory is executed by the processor, the processor functions as the division unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy encoding unit 110, and the inverse quantization unit 112. The inverse conversion unit 114, the addition unit 116, the loop filtering unit 120, the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 function. The encoding device 100 may correspond to the division unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy encoding unit 110, the inverse quantization unit 112, the inverse conversion unit 114, the addition unit 116, and the loop filter unit. 120. The one or more electronic circuits dedicated to the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 are implemented.

以下，針對編碼裝置100含有之各構成要素進行說明。
[分割部]Hereinafter, each component included in the encoding device 100 will be described.
[Division]

分割部102是將輸入動態圖像所含有之各圖片分割成複數個區塊，將各區塊朝相減部104輸出。例如，分割部102首先是將圖片分割成固定尺寸(例如128x128)之區塊。前述固定尺寸之區塊有時被稱作編碼樹單元(CTU)。然後，分割部102是基於遞迴性之四元樹(quadtree)及/或二元樹(binary tree)區塊分割而將固定尺寸之各區塊分割成可變尺寸(例如64x64以下)之區塊。前述可變尺寸之區塊有時被稱作編碼單元(CU)、預測單元(PU)或轉換單元(TU)。另，在本實施形態並不需要區分CU、PU及TU，亦可以令圖片內之一部分或全部之區塊是CU、PU、TU之處理單位。The dividing unit 102 divides each picture included in the input moving image into a plurality of blocks, and outputs each block to the subtraction unit 104. For example, the segmentation unit 102 first divides a picture into blocks of a fixed size (for example, 128 × 128). The aforementioned fixed-size block is sometimes called a coding tree unit (CTU). Then, the dividing unit 102 divides blocks of a fixed size into regions of variable size (for example, 64x64 or less) based on recursive quadtree and / or binary tree block division. Piece. The aforementioned variable-sized blocks are sometimes referred to as coding units (CU), prediction units (PU), or conversion units (TU). In addition, in this embodiment, there is no need to distinguish between CU, PU, and TU, and a part or all of the blocks in the picture may be the processing units of CU, PU, and TU.

圖2是顯示實施形態1之區塊分割之一例的圖。在圖2，實線是表示由四元樹區塊分割形成之區塊邊界，虛線是表示由二元樹區塊分割形成之區塊邊界。FIG. 2 is a diagram showing an example of block division in the first embodiment. In FIG. 2, the solid line indicates a block boundary formed by partitioning a quaternary tree block, and the dotted line indicates a block boundary formed by partitioning a quaternary tree block.

在此，區塊10是128x128像素之正方形區塊(128x128區塊)。前述128x128區塊10首先是分割成4個正方形之64x64區塊(四元樹區塊分割)。Here, the block 10 is a square block of 128x128 pixels (128x128 blocks). The aforementioned 128x128 block 10 is first divided into 64 square 64x64 blocks (quaternary tree block partition).

左上之64x64區塊是進一步垂直地分割成2個矩形之32x64區塊，左邊之32x64區塊是進一步垂直地分割成2個矩形之16x64區塊(二元樹區塊分割)。結果，左上之64x64區塊是分割成：2個16x64區塊11、12；以及32x64區塊13。The upper left 64x64 block is further vertically divided into two rectangular 32x64 blocks, and the left 32x64 block is further vertically divided into two rectangular 16x64 blocks (binary tree block division). As a result, the upper left 64x64 block is divided into: two 16x64 blocks 11, 12; and a 32x64 block 13.

右上之64x64區塊是水平地分割成2個矩形之64x32區塊14、15(二元樹區塊分割)。The upper 64x64 block is horizontally divided into two rectangular 64x32 blocks 14, 15 (binary tree block partition).

左下之64x64區塊是分割成4個正方形之32x32區塊(四元樹區塊分割)。進一步分割4個32x32區塊中之左上之區塊及右下之區塊。左上之32x32區塊是垂直地分割成2個矩形之16x32區塊，右邊之16x32區塊是進一步水平地分割成2個16x16區塊(二元樹區塊分割)。右下之32x32區塊是水平地分割成2個32x16區塊(二元樹區塊分割)。結果，左下之64x64區塊是分割成：16x32區塊16；2個16x16區塊17、18；2個32x32區塊19、20；以及2個32x16區塊21、22。The 64x64 block at the bottom left is a 32x32 block (quaternary tree block partition) divided into 4 squares. The upper left block and the lower right block of the four 32x32 blocks are further divided. The upper left 32x32 block is vertically divided into two rectangular 16x32 blocks, and the right 16x32 block is further horizontally divided into two 16x16 blocks (binary tree block division). The bottom right 32x32 block is horizontally divided into two 32x16 blocks (binary tree block partition). As a result, the lower left 64x64 block is divided into: 16x32 block 16; two 16x16 blocks 17, 18; two 32x32 blocks 19 and 20; and two 32x16 blocks 21 and 22.

右下之64x64區塊23未分割。The lower right 64x64 block 23 is not divided.

如以上，在圖2，區塊10是基於遞迴性之四元樹及二元樹區塊分割而分割成13個可變尺寸之區塊11~23。如此之分割有時被稱作QTBT(quad-tree plus binary tree，四元樹加二元樹)分割。As described above, in FIG. 2, the block 10 is divided into 13 variable-size blocks 11 to 23 based on the recursive quaternary tree and binary tree block division. Such segmentation is sometimes called QTBT (quad-tree plus binary tree) segmentation.

另，雖然在圖2是將1個區塊分割成4個或2個區塊(四元樹或二元樹區塊分割)，但分割並非限定於此。例如，亦可以將1個區塊分割成3個區塊(三元樹區塊分割)。如此之包含三元樹區塊分割之分割有時被稱作MBT(multi type tree，多類型樹)分割。
[相減部]In FIG. 2, one block is divided into four or two blocks (quaternary tree or binary tree block division), but the division is not limited to this. For example, one block may be divided into three blocks (ternary tree block division). Such a partition including a ternary tree block partition is sometimes referred to as MBT (multi type tree) partition.
[Subtraction Division]

相減部104是以被分割部102分割之區塊單位來從原訊號(原樣本)減去預測訊號(預測樣本)。亦即，相減部104是算出編碼對象區塊(以下稱作當前區塊)之預測誤差(亦稱作殘差)。然後，相減部104是將算出之預測誤差朝轉換部106輸出。The subtraction unit 104 subtracts the prediction signal (prediction sample) from the original signal (original sample) in units of blocks divided by the division unit 102. That is, the subtraction unit 104 calculates a prediction error (also referred to as a residual) of a coding target block (hereinafter referred to as a current block). Then, the subtraction unit 104 outputs the calculated prediction error to the conversion unit 106.

原訊號是編碼裝置100之輸入訊號，且是表示構成動態圖像之各圖片之圖像的訊號(例如亮度(luma)訊號及2個色差(chroma)訊號)。在以下，有時將表示圖像的訊號稱作樣本。
[轉換部]The original signal is an input signal of the encoding device 100, and is a signal (for example, a luma signal and two chroma signals) indicating an image of each picture constituting a moving image. Hereinafter, a signal representing an image is sometimes referred to as a sample.
[Conversion Department]

轉換部106是將空間領域之預測誤差轉換成頻率領域之轉換係數，並將轉換係數朝量化部108輸出。具體而言，轉換部106例如是對空間領域之預測誤差進行事先決定之離散餘弦轉換(DCT)或離散正弦轉換(DST)。The conversion unit 106 converts the prediction error in the space domain into a conversion coefficient in the frequency domain, and outputs the conversion coefficient to the quantization unit 108. Specifically, the conversion unit 106 is, for example, a discrete cosine transform (DCT) or a discrete sine transform (DST) that determines a prediction error in a spatial domain in advance.

另，轉換部106亦可以從複數個轉換類型中適應性地選擇轉換類型，並使用與選擇之轉換類型對應之轉換基底函數(transform basis function)而將預測誤差轉換成轉換係數。如此之轉換有時被稱作EMT(explicit multiple core transform，顯式多重核心轉換)或AMT(adaptive multiple transform，適應性多重轉換)。In addition, the conversion unit 106 may adaptively select a conversion type from a plurality of conversion types, and use a transform basis function corresponding to the selected conversion type to convert the prediction error into a conversion coefficient. Such a conversion is sometimes called an EMT (explicit multiple core transform) or an AMT (adaptive multiple transform).

複數個轉換類型例如是包含DCT-II、DCT-V、DCT-VIII、DST-I及DST-VII。圖3是顯示與各轉換類型對應之轉換基底函數的表。在圖3，N是表示輸入像素之數。關於從該等複數個轉換類型中選擇轉換類型，例如，可以取決於預測之種類(幀內預測及幀間預測)，亦可以取決於幀內預測模式。The plurality of conversion types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. FIG. 3 is a table showing conversion basis functions corresponding to each conversion type. In FIG. 3, N is the number of input pixels. Regarding the selection of a conversion type from the plurality of conversion types, for example, it may depend on the type of prediction (intra prediction and inter prediction), or it may depend on an intra prediction mode.

表示是否適用如此之EMT或AMT的資訊(例如稱作AMT旗標)及表示被選擇之轉換類型的資訊是以CU層級來訊號化。另，該等資訊之訊號化並非一定要限定在CU層級，亦可以是其他層級(例如序列層級、圖片層級、片段層級、圖塊層級或CTU層級)。Information indicating whether such EMT or AMT is applicable (for example, called AMT flag) and information indicating the selected conversion type are signaled at the CU level. In addition, the signalization of such information is not necessarily limited to the CU level, but may be other levels (such as sequence level, picture level, fragment level, tile level, or CTU level).

又，轉換部106亦可以將轉換係數(轉換結果)再轉換。如此之再轉換有時被稱作AST(adaptive secondary transform，適應性二次轉換)或NSST(non-separable secondary transform，不可分離二次轉換)。例如，轉換部106是依照與幀內預測誤差對應之轉換係數之區塊所含有之各子區塊(例如4x4子區塊)而進行再轉換。表示是否適用NSST的資訊、以及與用於NSST之轉換矩陣相關的資訊是以CU層級來訊號化。另，該等資訊之訊號化並非一定要限定在CU層級，亦可以是其他層級(例如序列層級、圖片層級、片段層級、圖塊層級或CTU層級)。The conversion unit 106 may convert the conversion coefficient (conversion result) again. Such re-transformation is sometimes referred to as AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the conversion unit 106 performs re-conversion according to each sub-block (for example, a 4 × 4 sub-block) included in a block having a conversion coefficient corresponding to the intra prediction error. Information indicating whether or not NSST is applicable, and information related to the conversion matrix used for NSST are signaled at the CU level. In addition, the signalization of such information is not necessarily limited to the CU level, but may be other levels (such as sequence level, picture level, fragment level, tile level, or CTU level).

在此，Separable之轉換是如下之方式：隨著輸入之維之數，依各方向予以分離而進行複數次轉換；Non-Separable之轉換是如下之方式：當輸入為多維時，將2個以上之維一起視為1維而一起進行轉換。Here, the transformation of Separable is as follows: As the number of dimensions of the input is separated in each direction, multiple transformations are performed; the conversion of Non-Separable is as follows: When the input is multi-dimensional, two or more The dimensions are treated together as one dimension and converted together.

可舉出例如，Non-Separable之轉換之1例是當輸入為4×4之區塊的情況下，將其視為具有16個元素之一陣列，並以16×16之轉換矩陣對前述陣列進行轉換處理。For example, one example of Non-Separable conversion is when the input is a 4 × 4 block, it is regarded as an array with 16 elements, and the aforementioned array is converted by a 16 × 16 conversion matrix. Perform conversion processing.

又，Non-Separable之轉換之例亦有如下之例：在同樣地將4×4之輸入區塊視為具有16個元素之一陣列後，對前述陣列進行複數次Givens旋轉(Hypercube Givens Transform，超立方體吉文斯轉換)。
[量化部]In addition, the example of Non-Separable transformation is as follows: After similarly considering a 4 × 4 input block as an array of 16 elements, the aforementioned array is subjected to multiple Givens rotation (Hypercube Givens Transform, Hypercube Givens Transformation).
[Quantization Department]

量化部108是將從轉換部106輸出之轉換係數量化。具體而言，量化部108是以預定之掃描順序來掃描當前區塊之轉換係數，基於與被掃描之轉換係數對應之量化參數(QP)而將該轉換係數量化。然後，量化部108是將當前區塊之經過量化之轉換係數(以下稱作量化係數)朝熵編碼部110及反量化部112輸出。The quantization unit 108 quantizes the conversion coefficient output from the conversion unit 106. Specifically, the quantization unit 108 scans the conversion coefficient of the current block in a predetermined scanning order, and quantizes the conversion coefficient based on a quantization parameter (QP) corresponding to the scanned conversion coefficient. The quantization unit 108 outputs the quantized conversion coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy encoding unit 110 and the inverse quantization unit 112.

預定之順序是為了轉換係數之量化/反量化之順序。例如，預定之掃描順序是以頻率之升序(從低頻至高頻之順序)或降序(從高頻至低頻之順序)來定義。The predetermined order is the order of quantization / inverse quantization of the conversion coefficients. For example, the predetermined scanning order is defined in ascending order of frequency (order from low frequency to high frequency) or descending order (order from high frequency to low frequency).

量化參數是定義量化步距(量化寬度)之參數。例如，若量化參數之值增加則量化步距亦增加。亦即，若量化參數之值增加則量化誤差增大。
[熵編碼部]The quantization parameter is a parameter that defines a quantization step (quantization width). For example, if the value of the quantization parameter is increased, the quantization step is also increased. That is, as the value of the quantization parameter increases, the quantization error increases.
[Entropy coding section]

熵編碼部110是藉由對從量化部108輸入之量化係數進行可變長度編碼，而生成編碼訊號(編碼位元串流)。具體而言，熵編碼部110例如是將量化係數二值化，對二值訊號進行算術編碼。
[反量化部]The entropy coding unit 110 generates a coding signal (coded bit stream) by variable-length coding the quantization coefficient input from the quantization unit 108. Specifically, for example, the entropy coding unit 110 binarizes the quantization coefficient and performs arithmetic coding on the binary signal.
[Dequantization Department]

反量化部112是對從量化部108輸入之量化係數進行反量化。具體而言，反量化部112是以預定之掃描順序對當前區塊之量化係數進行反量化。然後，反量化部112是將當前區塊之經過反量化之轉換係數朝反轉換部114輸出。
[反轉換部]The inverse quantization unit 112 performs inverse quantization on the quantization coefficient input from the quantization unit 108. Specifically, the inverse quantization unit 112 performs inverse quantization on the quantization coefficients of the current block in a predetermined scanning order. Then, the inverse quantization section 112 outputs the inverse quantized conversion coefficient of the current block to the inverse conversion section 114.
[Anti conversion department]

反轉換部114是藉由對從反量化部112輸入之轉換係數進行反轉換，而令預測誤差復原。具體而言，反轉換部114是藉由對轉換係數進行與轉換部106之轉換對應之反轉換，而令當前區塊之預測誤差復原。然後，反轉換部114是將經過復原之預測誤差朝相加部116輸出。The inverse conversion unit 114 restores the prediction error by inversely converting the conversion coefficient input from the inverse quantization unit 112. Specifically, the inverse conversion unit 114 restores the prediction error of the current block by performing an inverse conversion corresponding to the conversion of the conversion unit 106 on the conversion coefficient. Then, the inverse conversion unit 114 outputs the restored prediction error to the adding unit 116.

另，經過復原之預測誤差因量化而失去資訊，故與相減部104所算出之預測誤差不一致。亦即，經過復原之預測誤差含有量化誤差。
[相加部]In addition, since the restored prediction error loses information due to quantization, it does not agree with the prediction error calculated by the subtraction unit 104. That is, the restored prediction error contains a quantization error.
[Addition department]

相加部116是藉由將從反轉換部114輸入之預測誤差與從預測控制部128輸入之預測樣本予以相加，而重建當前區塊。然後，相加部116是將重建之區塊朝區塊記憶體118及迴路濾波部120輸出。重建區塊有時被稱作局部解碼區塊。
[區塊記憶體]The adding section 116 reconstructs the current block by adding the prediction error input from the inverse conversion section 114 and the prediction sample input from the prediction control section 128. Then, the adding unit 116 outputs the reconstructed block to the block memory 118 and the loop filtering unit 120. The reconstructed block is sometimes called a locally decoded block.
[Block memory]

區塊記憶體118是用於儲存在幀內預測被參考之區塊且為編碼對象圖片(以下稱作當前圖片)內之區塊之記憶部。具體而言，區塊記憶體118是將從相加部116輸出之重建區塊儲存。
[迴路濾波部]The block memory 118 is a memory portion for storing a block to be referenced in the intra prediction and a block in a coding target picture (hereinafter referred to as a current picture). Specifically, the block memory 118 stores the reconstructed blocks output from the adding unit 116.
[Loop Filtering Division]

迴路濾波部120是對被相加部116重建之區塊施加迴路濾波，將經過濾波之重建區塊朝幀記憶體122輸出。迴路濾波是在編碼迴路內使用之濾波(內嵌式迴路濾波，In-loop filter)，例如，包含去區塊濾波(DF)、取樣適應性偏移(SAO)及適應性迴路濾波(ALF)等。The loop filtering unit 120 applies loop filtering to the blocks reconstructed by the adding unit 116, and outputs the filtered reconstructed blocks to the frame memory 122. Loop filtering is the filtering used in the coding loop (in-loop filter). For example, it includes deblocking filtering (DF), sampling adaptive offset (SAO), and adaptive loop filtering (ALF). Wait.

在ALF是適用用於將編碼失真去除之最小平方誤差濾波器，對例如當前區塊內之各個2x2子區塊，基於局部性的梯度(gradient)之方向及活性度(activity)而適用從複數個濾波器中選擇之1個濾波器。The ALF is a least square error filter suitable for removing coding distortion. For example, for each 2x2 sub-block in the current block, it is applicable from the complex number based on the direction of the local gradient and activity. One of the filters selected.

具體而言，首先，子區塊(例如2x2子區塊)是分類成複數個類別(例如15或25類別)。子區塊之分類是基於梯度之方向及活性度而進行。例如，使用梯度之方向值D(例如0~2或0~4)與梯度之活性值A(例如0~4)來算出分類值C(例如C=5D+A)。然後，基於分類值C而將子區塊分類成複數個類別(例如15或25類別)。Specifically, first, a sub-block (for example, a 2 × 2 sub-block) is classified into a plurality of categories (for example, 15 or 25 categories). The classification of sub-blocks is based on the direction and activity of the gradient. For example, the direction value D (for example, 0 ~ 2 or 0 ~ 4) of the gradient and the activity value A (for example, 0 ~ 4) of the gradient are used to calculate the classification value C (for example, C = 5D + A). Then, the sub-blocks are classified into a plurality of categories (for example, 15 or 25 categories) based on the classification value C.

梯度之方向值D例如是藉由將複數個方向(例如水平、垂直及2個對角方向)之梯度進行比較而導出。又，梯度之活性值A例如是將複數個方向之梯度相加，藉由將相加結果量化而導出。The direction value D of the gradient is derived by comparing the gradients in a plurality of directions (for example, horizontal, vertical, and two diagonal directions). The activity value A of the gradient is obtained by, for example, adding gradients in a plurality of directions and quantifying the addition result.

基於如此分類之結果，而從複數個濾波器中決定用在子區塊之濾波器。Based on the results of such classification, a filter to be used for a sub-block is determined from a plurality of filters.

關於在ALF使用之濾波器之形狀，例如利用圓對稱形狀。圖4A~圖4C是顯示在ALF使用之濾波器之形狀之複數例的圖。圖4A是顯示5x5菱形形狀濾波器，圖4B是顯示7x7菱形形狀濾波器，圖4C是顯示9x9菱形形狀濾波器。表示濾波器之形狀之資訊是以圖片層級來訊號化。另，表示濾波器之形狀之資訊之訊號化並非一定要限定在圖片層級，亦可以是其他層級(例如序列層級、片段層級、圖塊層級、CTU層級或CU層級)。The shape of the filter used in ALF is, for example, a circularly symmetric shape. 4A to 4C are diagrams showing plural examples of the shape of a filter used in ALF. FIG. 4A shows a 5x5 diamond shape filter, FIG. 4B shows a 7x7 diamond shape filter, and FIG. 4C shows a 9x9 diamond shape filter. The information representing the shape of the filter is signaled at the picture level. In addition, the signalization of the information representing the shape of the filter is not necessarily limited to the picture level, but may be other levels (such as sequence level, fragment level, tile level, CTU level, or CU level).

ALF之開啟/關閉例如是以圖片層級或CU層級來決定。例如，關於亮度是以CU層級來決定是否適用ALF，關於色差是以圖片層級來決定是否適用ALF。表示ALF之開啟/關閉之資訊是以圖片層級或CU層級來訊號化。另，表示ALF之開啟/關閉之資訊之訊號化並非一定要限定在圖片層級或CU層級，亦可以是其他層級(例如序列層級、片段層級、圖塊層級或CTU層級)。The on / off of ALF is determined at the picture level or the CU level, for example. For example, regarding brightness, it is determined at the CU level whether to apply ALF, and regarding color difference, it is determined at the picture level whether to apply ALF. Information indicating that ALF is turned on / off is signaled at the picture level or the CU level. In addition, the signalization of the information indicating the on / off of ALF is not necessarily limited to the picture level or the CU level, but may also be other levels (such as sequence level, fragment level, tile level, or CTU level).

可選擇之複數個濾波器(例如到15或25為止之濾波器)之係數組是以圖片層級來訊號化。另，係數組之訊號化並非一定要限定在圖片層級，亦可以是其他層級(例如序列層級、片段層級、圖塊層級、CTU層級、CU層級或子區塊層級)。
[幀記憶體]The coefficient sets of the plurality of filters that can be selected (for example, filters up to 15 or 25) are signaled at the picture level. In addition, the signalization of the coefficient group is not necessarily limited to the picture level, and may also be other levels (such as sequence level, fragment level, tile level, CTU level, CU level, or sub-block level).
[Frame memory]

幀記憶體122是用於儲存幀間預測用之參考圖片之記憶部，有時被稱作幀緩衝器。具體而言，幀記憶體122是儲存被迴路濾波部120濾波後之重建區塊。
[幀內預測部]The frame memory 122 is a memory unit for storing reference pictures for inter prediction, and is sometimes called a frame buffer. Specifically, the frame memory 122 stores the reconstructed blocks filtered by the loop filtering unit 120.
[Intra prediction section]

幀內預測部124是參考儲存在區塊記憶體118之當前圖片內之區塊來進行當前區塊之幀內預測(亦稱作畫面內預測)，藉此生成預測訊號(幀內預測訊號)。具體而言，幀內預測部124是參考與當前區塊鄰接之區塊之樣本(例如亮度值、色差值)來進行幀內預測，藉此生成幀內預測訊號，將幀內預測訊號朝預測控制部128輸出。The intra-prediction unit 124 refers to the blocks stored in the current picture of the block memory 118 to perform intra-prediction of the current block (also referred to as intra-frame prediction), thereby generating a prediction signal (intra-prediction signal). . Specifically, the intra prediction unit 124 performs intra prediction with reference to samples (e.g., luminance values and color difference values) of blocks adjacent to the current block, thereby generating an intra prediction signal, and directing the intra prediction signal toward The prediction control unit 128 outputs.

例如，幀內預測部124是使用事先規定之複數個幀內預測模式之其中1者來進行幀內預測。複數個幀內預測模式包含有1個以上之非方向性預測模式、以及複數個方向性預測模式。For example, the intra prediction unit 124 performs intra prediction using one of a plurality of predetermined intra prediction modes. The plurality of intra prediction modes include one or more non-directional prediction modes and a plurality of directional prediction modes.

1個以上之非方向性預測模式例如是包含H.265/HEVC(High-Efficiency Video Coding)規格(非專利文獻1)所規定之Planar預測模式及DC預測模式。The one or more non-directional prediction modes include, for example, the Planar prediction mode and the DC prediction mode specified in the H.265 / HEVC (High-Efficiency Video Coding) specification (Non-Patent Document 1).

複數個方向性預測模式例如是包含H.265/HEVC規格所規定之33方向之預測模式。另，複數個方向性預測模式亦可以是除了33方向之外還更包含32方向之預測模式(合計65個方向性預測模式)。圖5A是顯示幀內預測之67個幀內預測模式(2個非方向性預測模式及65個方向性預測模式)的圖。實線箭頭是表示H.265/HEVC規格所規定之33方向，虛線箭頭是表示追加之32方向。The plurality of directional prediction modes are, for example, prediction modes including 33 directions specified in the H.265 / HEVC standard. In addition, the plurality of directional prediction modes may be prediction modes including 32 directions in addition to 33 directions (a total of 65 directional prediction modes). FIG. 5A is a diagram showing 67 intra prediction modes (2 non-directional prediction modes and 65 directional prediction modes) of intra prediction. The solid arrows indicate the 33 directions prescribed by the H.265 / HEVC standard, and the dotted arrows indicate the additional 32 directions.

另，在色差區塊之幀內預測中，亦可以參考亮度區塊。亦即，亦可以基於當前區塊之亮度成分來預測當前區塊之色差成分。如此之幀內預測有時被稱作CCLM(cross-component linear model，跨成分線性模型)預測。亦可以將如此之參考亮度區塊來進行的色差區塊之幀內預測模式(例如稱作CCLM模式)當作色差區塊之幀內預測模式之其中1者而加入。In addition, in the intra prediction of the color difference block, the luminance block can also be referred to. That is, the color difference component of the current block can also be predicted based on the brightness component of the current block. Such intra prediction is sometimes called CCLM (cross-component linear model) prediction. The intra prediction mode of the color difference block (such as the CCLM mode) performed with reference to the luminance block may also be added as one of the intra prediction modes of the color difference block.

幀內預測部124亦可基於水平/垂直方向之參考像素之梯度來修正幀內預測後之像素值。如此之伴隨著修正的幀內預測有時被稱作PDPC(position dependent intra prediction combination，位置相關幀內預測組合)。表示有無適用PDPC之資訊(例如稱作PDPC旗標)例如是以CU層級來訊號化。另，前述資訊之訊號化並非一定要限定在CU層級，亦可以是其他層級(例如序列層級、圖片層級、片段層級、圖塊層級或CTU層級)。
[幀間預測部]The intra prediction unit 124 may also modify the pixel values after intra prediction based on the gradient of the reference pixels in the horizontal / vertical direction. As such, the intra prediction accompanied by the correction is sometimes referred to as a PDPC (position dependent intra prediction combination). Information indicating whether there is applicable PDPC (for example, called PDPC flag) is signaled at the CU level, for example. In addition, the signalization of the foregoing information is not necessarily limited to the CU level, and may be other levels (such as a sequence level, a picture level, a fragment level, a tile level, or a CTU level).
[Inter prediction section]

幀間預測部126是參考儲存在幀記憶體122之參考圖片且與當前圖片不同之參考圖片來進行當前區塊之幀間預測(亦稱作畫面間預測)，藉此生成預測訊號(幀間預測訊號)。幀間預測是以當前區塊或當前區塊內之子區塊(例如4x4區塊)之單位來進行。例如，幀間預測部126是針對當前區塊或子區塊而在參考圖片內進行運動估計(motion estimation)。然後，幀間預測部126是使用藉由運動估計而獲得之運動資訊(例如運動向量)進行運動補償，藉此生成當前區塊或子區塊之幀間預測訊號。然後，幀間預測部126是將生成之幀間預測訊號朝預測控制部128輸出。The inter prediction unit 126 refers to a reference picture stored in the frame memory 122 and is different from the current picture to perform inter prediction (also called inter-frame prediction) of the current block, thereby generating a prediction signal (inter-frame Forecast signal). Inter prediction is performed in units of a current block or a sub-block (for example, a 4x4 block) within the current block. For example, the inter prediction unit 126 performs motion estimation in the reference picture for the current block or subblock. Then, the inter prediction section 126 performs motion compensation using motion information (for example, a motion vector) obtained by motion estimation, thereby generating an inter prediction signal of a current block or a subblock. Then, the inter prediction unit 126 outputs the generated inter prediction signal to the prediction control unit 128.

用於運動補償之運動資訊是被訊號化。運動向量之訊號化亦可以使用運動向量預測子(motion vector predictor)。亦即，亦可以讓運動向量與運動向量預測子之間之差分被訊號化。The motion information used for motion compensation is signaled. Signaling of motion vectors can also use motion vector predictors. That is, the difference between the motion vector and the motion vector predictor can also be signaled.

另，亦可以除了藉由運動估計所獲得之當前區塊之運動資訊之外，還使用鄰接區塊之運動資訊，而生成幀間預測訊號。具體而言，亦可以將基於藉由運動估計所獲得之運動資訊而來之預測訊號、以及基於鄰接區塊之運動資訊而來之預測訊號進行加權相加，藉此，以當前區塊內之子區塊單位來生成幀間預測訊號。如此之幀間預測(運動補償)有時被稱作OBMC(overlapped block motion compensation，重疊區塊移動補償)。In addition, in addition to the motion information of the current block obtained by motion estimation, the motion information of the adjacent block can also be used to generate the inter prediction signal. Specifically, the prediction signal based on the motion information obtained by motion estimation and the prediction signal based on the motion information of adjacent blocks can also be weighted and added, so that the child in the current block can be added. Block unit to generate inter prediction signals. Such inter prediction (motion compensation) is sometimes called OBMC (overlapped block motion compensation).

在如此之OBMC模式，表示用於OBMC之子區塊的尺寸之資訊(例如稱作OBMC區塊尺寸)是以序列層級來訊號化。又，表示是否適用OBMC模式之資訊(例如稱作OBMC旗標)是以CU層級來訊號化。另，該等資訊之訊號化層級並非一定要限定在序列層級及CU層級，亦可以是其他層級(例如圖片層級、片段層級、圖塊層級、CTU層級或子區塊層級)。In such an OBMC mode, information (for example, called OBMC block size) indicating the size of a sub-block for OBMC is signaled at a sequence level. In addition, information (for example, called OBMC flag) indicating whether the OBMC mode is applicable is signaled at the CU level. In addition, the signalization level of such information is not necessarily limited to the sequence level and the CU level, but may also be other levels (such as the picture level, the fragment level, the tile level, the CTU level, or the sub-block level).

針對OBMC模式進行更具體之說明。圖5B及圖5C是用於說明利用OBMC處理之預測圖像修正處理之概要的流程圖及概念圖。The OBMC mode is described in more detail. 5B and 5C are a flowchart and a conceptual diagram for explaining an outline of a prediction image correction process using OBMC processing.

首先，使用分配給編碼對象區塊之運動向量(MV)來取得利用通常之運動補償之預測圖像(Pred)。First, a motion vector (MV) allocated to a coding target block is used to obtain a prediction image (Pred) using normal motion compensation.

接著，將已編碼之左鄰接區塊之運動向量(MV_L)適用至編碼對象區塊而取得預測圖像(Pred_L)，藉由將前述預測圖像與Pred_L加權重疊而進行預測圖像之第1次修正。Next, apply the motion vector (MV_L) of the coded left-neighboring block to the coding target block to obtain a predicted image (Pred_L), and perform weighted overlap of the aforementioned predicted image and Pred_L to perform the first prediction image. Corrections.

同樣地，將已編碼之上鄰接區塊之運動向量(MV_U)適用至編碼對象區塊而取得預測圖像(Pred_U)，藉由將進行了前述第1次修正之預測圖像與Pred_U加權重疊而進行預測圖像之第2次修正，將其當作最終之預測圖像。Similarly, the motion vector (MV_U) of the coded upper adjacent block is applied to the coding target block to obtain a prediction image (Pred_U), and the prediction image subjected to the first correction is weighted to overlap with Pred_U. The second correction of the predicted image is performed as the final predicted image.

另，雖然在此說明的是使用左鄰接區塊與上鄰接區塊之2階段之修正方法，但亦可以是使用右鄰接區塊、下鄰接區塊而進行比2階段還多次之修正的構成。In addition, although the two-stage correction method using the left adjacent block and the upper adjacent block is described here, the right adjacent block and the lower adjacent block may also be used to perform multiple corrections more than the two steps. Make up.

另，亦可僅以區塊邊界附近之一部分之區域來進行重疊，而不是以區塊整體之像素區域來進行重疊。In addition, the overlapping may be performed only by a part of the area near the block boundary instead of the entire pixel area of the block.

另，雖然在此說明的是由1張參考圖片之預測圖像修正處理，但由複數張參考圖片修正預測圖像的情況亦是同樣的，在取得由各參考圖片修正之預測圖像後，藉由將獲得之預測圖像進一步重疊而成為最終之預測圖像。In addition, although the prediction image correction processing by one reference picture is described here, the same applies to the case where the prediction image is corrected by a plurality of reference pictures. After obtaining the prediction image corrected by each reference picture, The obtained predicted images are further overlapped to become the final predicted image.

另，前述處理對象區塊可以是預測區塊單位，亦可以是將預測區塊進一步分割之子區塊單位。In addition, the foregoing processing target block may be a prediction block unit, or may be a sub-block unit that further divides the prediction block.

關於判定是否適用OBMC處理之方法，例如有使用obmc_flag之方法，前述obmc_flag為表示是否適用OBMC處理之訊號。作為具體之一例：在編碼裝置中，判定編碼對象區塊是否屬於運動複雜之區域，當屬於運動複雜之區域的情況下，將obmc_flag設定成值1而適用OBMC處理來進行編碼，當不屬於運動複雜之區域的情況下，將obmc_flag設定成值0，不適用OBMC處理而進行編碼。另一方面，解碼裝置是將描述在串流之obmc_flag解碼，藉此因應其值來切換是否適用OBMC處理而進行解碼。As a method for determining whether OBMC processing is applicable, for example, there is a method using obmc_flag. The aforementioned obmc_flag is a signal indicating whether OBMC processing is applicable. As a specific example: In the encoding device, it is determined whether the coding target block belongs to a region with complex motion. When it belongs to a region with complex motion, set obmc_flag to a value of 1 and apply OBMC processing to perform encoding. In the case of a complex area, the obmc_flag is set to a value of 0, and encoding is performed without applying OBMC processing. On the other hand, the decoding device will describe obmc_flag decoding in the stream, and thereby perform decoding by switching whether or not to apply OBMC processing according to its value.

另，運動資訊亦可以不被訊號化而在解碼裝置側被導出。例如，亦可以使用H.265/HEVC規格所規定之合併模式。又，例如，亦可以藉由在解碼裝置側進行運動估計而導出運動資訊。此時，不使用當前區塊之像素值而進行運動估計。In addition, the motion information may be derived on the decoding device side without being signaled. For example, the merge mode specified by the H.265 / HEVC specification can also be used. In addition, for example, motion information may be derived by performing motion estimation on the decoding device side. At this time, motion estimation is performed without using the pixel values of the current block.

在此，針對在解碼裝置側進行運動估計之模式進行說明。此在解碼裝置側進行運動估計之模式有時被稱作PMMVD(pattern matched motion vector derivation，型樣匹配移動向量導出)模式或FRUC(frame rate up-conversion，幀率提升轉換)模式。Here, a mode for performing motion estimation on the decoding device side will be described. This mode of performing motion estimation on the decoding device side is sometimes called a PMMVD (pattern matched motion vector derivation) mode or a FRUC (frame rate up-conversion) mode.

在圖5D顯示FRUC處理之一例。首先，參考在空間或時間上與當前區塊鄰接之已編碼區塊之運動向量，而生成分別具有運動向量預測子之複數個候選之清單(亦可以與合併清單共通)。接著，從登記在候選清單中之複數個候選MV中選擇最佳候選MV。例如，將候選清單所含有之各候選的評價值算出，基於評價值而選擇1個候選。An example of FRUC processing is shown in FIG. 5D. First, referring to the motion vectors of the coded blocks that are adjacent to the current block in space or time, a list of a plurality of candidates each having a motion vector predictor is generated (also common to the merged list). Next, the best candidate MV is selected from a plurality of candidate MVs registered in the candidate list. For example, the evaluation value of each candidate included in the candidate list is calculated, and one candidate is selected based on the evaluation value.

然後，基於被選擇之候選之運動向量而導出用於當前區塊之運動向量。具體而言，例如，直接將被選擇之候選之運動向量(最佳候選MV)當作用於當前區塊之運動向量而導出。又，例如，亦可以藉由在與被選擇之候選之運動向量對應之參考圖片內的位置之周邊區域進行型樣匹配(pattern matching)，而導出用於當前區塊之運動向量。亦即，亦可以對最佳候選MV之周邊之區域以同樣之方法進行估計，當有評價值為更佳值之MV的情況下，將最佳候選MV更新成前述MV，將其當作當前區塊之最終之MV。另，亦可以採用不實施該處理之構成。Then, a motion vector for the current block is derived based on the motion vector of the selected candidate. Specifically, for example, the motion vector (best candidate MV) of the selected candidate is directly derived as the motion vector for the current block. In addition, for example, a pattern matching may be performed on a peripheral area of a position in a reference picture corresponding to the selected candidate motion vector to derive a motion vector for the current block. That is, the area around the best candidate MV can also be estimated in the same way. When there is a MV with a better evaluation value, the best candidate MV is updated to the aforementioned MV, which is regarded as the current MV. The final MV of the block. It is also possible to adopt a configuration in which this processing is not performed.

亦可以在以子區塊單位進行處理的情況下亦採用完全同樣之處理。It is also possible to use the exact same processing in the case of processing in sub-block units.

另，評價值是藉由如下而算出：利用與運動向量對應之參考圖片內之區域、以及預定之區域之間之型樣匹配，求出重建圖像之差分值。另，亦可以是除了差分值之外還使用其他之資訊而算出評價值。In addition, the evaluation value is calculated by using a pattern matching between a region in the reference picture corresponding to the motion vector and a predetermined region to obtain a difference value of the reconstructed image. Alternatively, the evaluation value may be calculated using other information in addition to the difference value.

關於型樣匹配是使用第1型樣匹配或第2型樣匹配。第1型樣匹配及第2型樣匹配有時分別被稱作雙向匹配(bilateral matching)及模板匹配(template matching)。Regarding pattern matching, the first pattern matching or the second pattern matching is used. The first pattern matching and the second pattern matching are sometimes called bilateral matching and template matching, respectively.

第1型樣匹配是在既為相異之2個參考圖片內之2個區塊亦為沿著當前區塊之運動軌跡(motion trajectory)之2個區塊之間進行型樣匹配。所以，第1型樣匹配是使用沿著當前區塊之運動軌跡之其他參考圖片內的區域，來作為上述之用於算出候選之評價值的預定之區域。The first pattern matching is to perform pattern matching between two blocks in two different reference pictures and two blocks along the motion trajectory of the current block. Therefore, the first pattern matching is to use the regions in other reference pictures along the motion trajectory of the current block as the above-mentioned predetermined regions for calculating candidate evaluation values.

圖6是用於說明在沿著運動軌跡之2個區塊間之型樣匹配(雙向匹配)之一例的圖。如圖6所示，第1型樣匹配是藉由估計既為沿著當前區塊(Cur block)之運動軌跡之2個區塊亦為相異之2個參考圖片(Ref0、Ref1)內之2個區塊的配對中最匹配的配對，而導出2個運動向量(MV0、MV1)。具體而言，對於當前區塊，導出藉由候選MV而指定之第1已編碼參考圖片(Ref0)內的指定位置之重建圖像、以及藉由對稱MV而指定之第2已編碼參考圖片(Ref1)內的指定位置之重建圖像的差分，並使用獲得之差分值來算出評價值，前述對稱MV是將前述候選MV以顯示時間間隔來縮放(scaling)而獲得的MV。選擇在複數個候選MV中評價值為最佳值之候選MV來作為最終MV即可。FIG. 6 is a diagram for explaining an example of pattern matching (two-way matching) between two blocks along a motion trajectory. As shown in FIG. 6, the first type of matching is to estimate that two blocks along the trajectory of the current block (Cur block) are within two reference pictures (Ref0, Ref1) that are different. Among the pairing of the two blocks, the best matching pair is derived, and two motion vectors (MV0, MV1) are derived. Specifically, for the current block, a reconstructed image at a specified position in the first coded reference picture (Ref0) designated by the candidate MV and a second coded reference picture designated by the symmetric MV ( The difference between the reconstructed image at a specified position in Ref1) and the evaluation value is calculated using the obtained difference value. The symmetrical MV is an MV obtained by scaling the candidate MV at a display time interval. The candidate MV with the best evaluation value among the plurality of candidate MVs may be selected as the final MV.

在假設是連續之運動軌跡之下，則指出2個參考區塊的運動向量(MV0、MV1)是相對於當前圖片(Cur Pic)和2個參考圖片(Ref0、Ref1)之間的時間上的距離(TD0、TD1)成比例。例如，當前圖片在時間上是位於2個參考圖片之間、且從當前圖片往2個參考圖片之時間距離是相等時，第1型樣匹配會導出鏡射對稱之雙向之運動向量。Under the assumption of a continuous motion trajectory, it is pointed out that the motion vectors (MV0, MV1) of the two reference blocks are relative to the time between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). The distances (TD0, TD1) are proportional. For example, when the current picture is located between two reference pictures in time and the time distance from the current picture to the two reference pictures is equal, the first pattern matching will derive a mirror-symmetric bidirectional motion vector.

第2型樣匹配是在當前圖片內之模板(在當前圖片內與當前區塊鄰接之區塊(例如上及/或左鄰接區塊))與參考圖片內之區塊之間進行型樣匹配。所以，第2型樣匹配是使用當前圖片內之與當前區塊鄰接之區塊，來作為上述之用於算出候選之評價值的預定之區域。The second pattern matching is pattern matching between the template in the current picture (blocks adjacent to the current block in the current picture (such as upper and / or left adjacent blocks)) and blocks in the reference picture. . Therefore, the second type of matching is to use the block adjacent to the current block in the current picture as the above-mentioned predetermined area for calculating the candidate evaluation value.

圖7是用於說明在當前圖片內之模板與參考圖片內之區塊之間的型樣匹配(模板匹配)之一例的圖。如圖7所示，第2型樣匹配是藉由在參考圖片(Ref0)內估計與在當前圖片(Cur Pic)內鄰接於當前區塊(Cur block)之區塊最匹配的區塊，而導出當前區塊之運動向量。具體而言，對於當前區塊，導出左鄰接及上鄰接雙方或其中一方的已編碼區域之重建圖像、以及藉由候選MV而指定之已編碼參考圖片(Ref0)內的同等位置之重建圖像的差分，並使用獲得之差分值而算出評價值，選擇在複數個候選MV中評價值為最佳值之候選MV來作為最佳候選MV即可。FIG. 7 is a diagram for explaining an example of pattern matching (template matching) between a template in a current picture and a block in a reference picture. As shown in FIG. 7, the second type of matching is by estimating in the reference picture (Ref0) the block that most closely matches the block in the current picture (Cur Pic) that is adjacent to the current block (Cur block), and Derive the motion vector of the current block. Specifically, for the current block, a reconstructed image of the coded area of both the left and upper adjacent sides or one of them is derived, and the reconstructed map of the equivalent position in the encoded reference picture (Ref0) specified by the candidate MV The difference of the image is calculated using the obtained difference value, and the candidate MV with the best evaluation value among the plurality of candidate MVs may be selected as the best candidate MV.

表示是否適用如此之FRUC模式的資訊(例如稱作FRUC旗標)是以CU層級來訊號化。又，當適用FRUC模式的情況下(例如FRUC旗標為真的情況下)，表示型樣匹配之方法(第1型樣匹配或第2型樣匹配)的資訊(例如稱作FRUC模式旗標)是以CU層級來訊號化。另，該等資訊之訊號化並非一定要是CU層級，亦可以是其他層級(例如序列層級、圖片層級、片段層級、圖塊層級、CTU層級或子區塊層級)。Information indicating whether such a FRUC mode is applicable (for example, called a FRUC flag) is signaled at the CU level. In addition, when the FRUC mode is applied (for example, when the FRUC flag is true), information indicating a method of pattern matching (first pattern matching or second pattern matching) (for example, called a FRUC mode flag) ) Is signaled at the CU level. In addition, the signaling of such information does not necessarily have to be at the CU level, but also at other levels (such as sequence level, picture level, fragment level, tile level, CTU level, or sub-block level).

在此說明的是基於假設等速直線運動之模型而導出運動向量之模式。此模式有時被稱作BIO(bi-directional optical flow，雙向光流)模式。Described here is a mode for deriving a motion vector based on a model that assumes constant velocity linear motion. This mode is sometimes called the BIO (bi-directional optical flow, bidirectional optical flow) mode.

圖8是用於說明假設等速直線運動之模型的圖。在圖8，(v_x ,v_y )是表示速度向量，τ₀ 、τ₁ 分別表示當前圖片(Cur Pic)和2個參考圖片(Ref₀ ,Ref₁ )之間的時間距離。(MVx₀ ,MVy₀ )是表示與參考圖片Ref₀ 對應之運動向量，(MVx₁ 、MVy₁ )是表示與參考圖片Ref₁ 對應之運動向量。FIG. 8 is a diagram for explaining a model that assumes constant velocity linear motion. In FIG. 8, (v _x , v _y ) represents a velocity vector, and τ ₀ and τ ₁ represent a time distance between a current picture (Cur Pic) and two reference pictures (Ref ₀ , Ref ₁ ), respectively. (MVx ₀ , MVy ₀ ) are motion vectors corresponding to the reference picture Ref ₀ , and (MVx ₁ , MVy ₁ ) are motion vectors corresponding to the reference picture Ref ₁ .

此時在速度向量(v_x ,v_y )之等速直線運動之假設下，(MVx₀ ,MVy₀ )及(MVx₁ ,MVy₁ )分別表示成(v_x τ₀ ,v_y τ₀ )及(-v_x τ₁ ,-v_y τ₁ )，以下之光流等式(1)成立。
[式子1]
At this time, under the assumption of constant velocity linear motion of the velocity vector (v _x , v _y ), (MVx ₀ , MVy ₀ ) and (MVx ₁ , MVy ₁ ) are expressed as (v _x τ ₀ , v _y τ ₀ ), respectively. And (-v _x τ ₁ , -v _y τ ₁ ), the following optical flow equation (1) holds.
[Formula 1]

在此，I^(k) 是表示運動補償後之參考圖像k(k=0,1)之亮度值。此光流等式是表示(i)、(ii)及(iii)的和等於零，其中(i)為亮度值之時間微分，(ii)為水平方向之速度及參考圖像之空間梯度之水平成分之積，(iii)為垂直方向之速度及參考圖像之空間梯度之垂直成分之積。基於前述光流等式與赫米特內插(Hermite interpolation)之組合，而令從合併清單等獲得之區塊單位之運動向量以像素單位被修正。Here, I ^(k) is a luminance value representing a reference image k (k = 0, 1) after motion compensation. This optical flow equation means that the sum of (i), (ii), and (iii) is equal to zero, where (i) is the time differential of the luminance value, and (ii) is the horizontal velocity and the level of the spatial gradient of the reference image The product of components, (iii) is the product of the vertical component of the velocity in the vertical direction and the spatial gradient of the reference image. Based on the combination of the aforementioned optical flow equation and Hermite interpolation, the motion vector of the block unit obtained from the merged list or the like is corrected in pixel units.

另，亦可以是以與基於假設等速直線運動之模型而將運動向量導出不同之方法，在解碼裝置側將運動向量導出。例如，亦可以基於複數個鄰接區塊之運動向量而以子區塊單位將運動向量導出。In addition, the motion vector may be derived on the decoding device side in a different method from the method of deriving a motion vector based on a model that assumes a constant-speed linear motion. For example, the motion vector may be derived in units of sub-blocks based on the motion vectors of a plurality of adjacent blocks.

在此，針對基於複數個鄰接區塊之運動向量而以子區塊單位將運動向量導出之模式進行說明。此模式有時被稱作仿射運動補償預測(affine motion compensation prediction)模式。Here, a mode of deriving a motion vector in units of sub-blocks based on motion vectors of a plurality of adjacent blocks will be described. This mode is sometimes called the affine motion compensation prediction mode.

圖9A是用於說明基於複數個鄰接區塊之運動向量而導出子區塊單位之運動向量的圖。在圖9A，當前區塊含有16個4x4子區塊。在此是基於鄰接區塊之運動向量而導出當前區塊之左上角控制點之運動向量v₀ ，且基於鄰接子區塊之運動向量而導出當前區塊之右上角控制點之運動向量v₁ 。然後，使用2個運動向量v₀ 及v₁ ，藉由以下之式子(2)，導出當前區塊內之各子區塊之運動向量(v_x ,v_y )。
[式子2]
FIG. 9A is a diagram for explaining that a motion vector of a sub-block unit is derived based on motion vectors of a plurality of adjacent blocks. In FIG. 9A, the current block contains 16 4x4 sub-blocks. Here, the motion vector v ₀ of the upper left corner control point of the current block is derived based on the motion vector of the adjacent block, and the motion vector v ₁ of the upper right corner control point of the current block is derived based on the motion vector of the adjacent subblock. . Then, using two motion vectors v ₀ and v ₁ , the motion vector (v _x , v _y ) of each sub-block in the current block is derived by the following formula (2).
[Formula 2]

在此，x及y分別表示子區塊之水平位置及垂直位置，w表示事先決定之加權係數。Here, x and y represent the horizontal position and vertical position of the sub-block, respectively, and w represents a weighting coefficient determined in advance.

在如此之仿射運動補償預測模式，亦可以令左上及右上角控制點之運動向量之導出方法是含有不同之幾個模式。表示如此之仿射運動補償預測模式之資訊(例如稱作仿射旗標)是以CU層級被訊號化。另，此表示仿射運動補償預測模式之資訊之訊號化並非一定要限定在CU層級，亦可以是其他層級(例如序列層級、圖片層級、片段層級、圖塊層級、CTU層級或子區塊層級)。
[預測控制部]In such an affine motion-compensated prediction mode, the method of deriving the motion vectors of the upper left and upper right control points can also include several different modes. Information (such as an affine flag) indicating such an affine motion-compensated prediction mode is signaled at the CU level. In addition, this means that the signalization of the information of the affine motion-compensated prediction mode is not necessarily limited to the CU level, but may also be other levels (such as sequence level, picture level, fragment level, tile level, CTU level, or sub-block level). ).
[Predictive Control Department]

預測控制部128是選擇幀內預測訊號及幀間預測訊號之任一者，並將選擇之訊號當作預測訊號而朝相減部104及相加部116輸出。The prediction control unit 128 selects either an intra prediction signal or an inter prediction signal, and outputs the selected signal to the subtraction unit 104 and the addition unit 116 as a prediction signal.

在此，說明藉由合併模式而導出編碼對象圖片之運動向量之例。圖9B是用於說明利用合併模式之運動向量導出處理之概要的圖。Here, an example of deriving a motion vector of a picture to be encoded by a merge mode will be described. FIG. 9B is a diagram for explaining the outline of the motion vector derivation process using the merge mode.

首先，生成登記了預測MV之候選的預測MV清單。關於預測MV之候選，有空間鄰接預測MV、時間鄰接預測MV、結合預測MV、零預測MV等，空間鄰接預測MV是以空間而言位於編碼對象區塊之周邊之複數個已編碼區塊所具有之MV，時間鄰接預測MV是將已編碼參考圖片中之編碼對象區塊之位置投影後之近處之區塊所具有之MV，結合預測MV是將空間鄰接預測MV與時間鄰接預測MV之MV值組合而生成之MV，零預測MV是值為零之MV。First, a list of predicted MVs in which candidates for predicted MVs are registered is generated. As for candidates for prediction MV, there are spatial adjacency prediction MV, temporal adjacency prediction MV, combined prediction MV, zero prediction MV, etc. The spatial adjacency prediction MV is a plurality of coded blocks located spatially around the encoding target block With MV, temporal adjacency prediction MV is the MV possessed by the nearby block after projecting the position of the encoding target block in the encoded reference picture, and combined prediction MV is the spatial adjacency prediction MV and temporal adjacency prediction MV MVs are generated by combining MV values. Zero-prediction MVs are MVs with a value of zero.

接著，從登記在預測MV清單中之複數個預測MV中選擇1個預測MV，藉此決定編碼對象區塊之MV。Next, one predictive MV is selected from the plurality of predictive MVs registered in the predictive MV list, thereby determining the MV of the encoding target block.

再者，可變長度編碼部將merge_idx描述在串流來編碼，前述merge_idx是表示選擇了哪一個預測MV之訊號。Furthermore, the variable-length encoding unit encodes merge_idx by describing it in a stream, and the aforementioned merge_idx is a signal indicating which prediction MV is selected.

另，在圖9B說明之登記在預測MV清單中之預測MV是一例，亦可以令圖中之個數是不同之個數、或者構成為不包含圖中之預測MV之一部分種類、或者構成為追加圖中之預測MV的種類以外的預測MV。In addition, the predicted MV registered in the predicted MV list described in FIG. 9B is an example, and the number in the figure may be a different number, or may be configured not to include a part of the types of the predicted MV in the figure, or may be configured as A prediction MV other than the type of prediction MV in the figure is added.

另，亦可以使用藉由合併模式而導出之編碼對象區塊之MV來進行後述之DMVR處理，藉此決定最終之MV。In addition, the MV of the encoding target block derived by the merge mode may be used to perform DMVR processing described later, thereby determining the final MV.

在此，針對使用DMVR處理來決定MV之例進行說明。Here, an example of MV determination using DMVR processing will be described.

圖9C是用於說明DMVR處理之概要的概念圖。FIG. 9C is a conceptual diagram for explaining the outline of the DMVR process.

首先，將設定在處理對象區塊之最合適MVP當作候選MV，依循前述候選MV，從L0方向之已處理圖片即第1參考圖片、及L1方向之已處理圖片即第2參考圖片分別取得參考像素，取各參考像素之平均，藉此生成模板。First, the most suitable MVP set in the processing target block is regarded as a candidate MV. According to the foregoing candidate MVs, the processed reference picture in the L0 direction is the first reference picture, and the processed picture in the L1 direction is the second reference picture. The reference pixels are averaged to generate a template.

接著，使用前述模板，分別估計第1參考圖片及第2參考圖片之候選MV之周邊區域，將成本最小之MV決定為最終之MV。另，成本值是使用模板之各像素值與估計區域之各像素值的差分值及MV值等而算出。Then, using the aforementioned template, the surrounding areas of the candidate MVs of the first reference picture and the second reference picture are estimated respectively, and the MV with the lowest cost is determined as the final MV. The cost value is calculated using a difference value, an MV value, and the like of each pixel value of the template and each pixel value of the estimated area.

另，在此說明之處理之概要對編碼裝置及解碼裝置而言基本上是共通的。The outline of the processing described here is basically common to the encoding device and the decoding device.

另，即便與在此說明之處理不同，只要是可估計候選MV之周邊而將最終之MV導出之處理，則使用其他處理亦無妨。In addition, even if it is different from the processing described here, as long as the surroundings of the candidate MV can be estimated and the final MV is derived, other processing may be used.

在此，針對使用LIC處理來生成預測圖像之模式進行說明。Here, a mode for generating a predicted image using LIC processing will be described.

圖9D是用於說明用到利用LIC處理之亮度修正處理之預測圖像生成方法之概要的圖。FIG. 9D is a diagram for explaining an outline of a prediction image generation method using a brightness correction process using a LIC process.

首先，導出用於從已編碼圖片即參考圖片取得與編碼對象區塊對應之參考圖像的MV。First, an MV for obtaining a reference image corresponding to an encoding target block from an encoded picture, that is, a reference picture, is derived.

接著，對於編碼對象區塊，使用左鄰接及上鄰接之已編碼周邊參考區域之亮度像素值、以及藉由MV來指定之參考圖片內之同等位置之亮度像素值，擷取表示亮度值在參考圖片與編碼對象圖片是如何變化之資訊而算出亮度修正參數。Next, for the encoding target block, the luminance pixel values of the coded surrounding reference areas adjacent to the left and the top adjacent, and the luminance pixel values of the equivalent positions in the reference picture specified by the MV are used to extract the luminance values in the reference. Information on how the picture and the encoding target picture change to calculate brightness correction parameters.

對於藉由MV而指定之參考圖片內之參考圖像，使用前述亮度修正參數來進行亮度修正處理，藉此生成對應於編碼對象區塊之預測圖像。For the reference image in the reference picture specified by the MV, the aforementioned brightness correction parameter is used to perform brightness correction processing, thereby generating a predicted image corresponding to the coding target block.

另，圖9D之前述周邊參考區域之形狀是一例，亦可以使用其以外之形狀。The shape of the peripheral reference area in FIG. 9D is an example, and other shapes may be used.

又，雖然在此說明的是由1張參考圖片來將預測圖像生成之處理，但從複數張參考圖片生成預測圖像的情況亦同樣，在從各參考圖片取得之參考圖像以同樣之方法進行亮度修正處理後，生成預測圖像。In addition, although the process of generating a predicted image from one reference picture is described here, the same applies to the case of generating a predicted image from a plurality of reference pictures. The same applies to the reference image obtained from each reference picture. After performing brightness correction processing, a prediction image is generated.

關於判定是否適用LIC處理之方法，例如有使用lic_flag之方法，前述lic_flag為表示是否適用LIC處理之訊號。具體之一例是如下：在編碼裝置，判定編碼對象區塊是否屬於發生亮度變化之區域，當屬於發生亮度變化之區域的情況下，將lic_flag設定成值1而適用LIC處理來進行編碼，當不屬於發生亮度變化之區域的情況下，將lic_flag設定成值0，不適用LIC處理而進行編碼。另一方面，解碼裝置是將描述在串流之lic_flag解碼，藉此因應其值來切換是否適用LIC處理而進行解碼。As a method for determining whether LIC processing is applicable, for example, there is a method using lic_flag, and the aforementioned lic_flag is a signal indicating whether LIC processing is applicable. A specific example is as follows: In the encoding device, it is determined whether the encoding target block belongs to an area where brightness changes occur. When it belongs to an area where brightness changes occur, set lic_flag to a value of 1 and apply LIC processing to perform encoding. In the case where the luminance changes, the lic_flag is set to a value of 0, and encoding is performed without applying LIC processing. On the other hand, the decoding device decodes the lic_flag described in the stream, thereby switching whether or not to apply the LIC processing according to its value.

關於判定是否適用LIC處理之別的方法，例如，還有依循周邊區塊是否適用了LIC處理而判定之方法。具體之一例是如下：當編碼對象區塊是合併模式的情況下，判定將合併模式處理中之MV導出時所選擇的周邊之已編碼區塊是否適用了LIC處理而編碼，並因應其結果來切換是否適用LIC處理而進行編碼。另，此例的情況下，解碼之處理亦全部同樣。
[解碼裝置之概要]As another method for determining whether the LIC processing is applicable, for example, there is a method for determining whether the LIC processing is applicable to the surrounding block. A specific example is as follows: When the encoding target block is the merge mode, it is determined whether the surrounding encoded blocks selected when the MV in the merge mode process is exported are encoded using the LIC process, and according to the result, It is switched whether to apply LIC processing for encoding. In addition, in the case of this example, the decoding process is all the same.
[Outline of decoding device]

接著，說明可解碼從上述之編碼裝置100輸出之編碼訊號(編碼位元串流)之解碼裝置的概要。圖10是顯示實施形態1之解碼裝置200之功能構成的方塊圖。解碼裝置200是將動態圖像/圖像以區塊單位來解碼之動態圖像/圖像解碼裝置。Next, an outline of a decoding device capable of decoding the encoded signal (encoded bit stream) output from the encoding device 100 described above will be described. FIG. 10 is a block diagram showing a functional configuration of the decoding device 200 according to the first embodiment. The decoding device 200 is a moving image / image decoding device that decodes moving images / images in units of blocks.

如圖10所示，解碼裝置200具有熵解碼部202、反量化部204、反轉換部206、相加部208、區塊記憶體210、迴路濾波部212、幀記憶體214、幀內預測部216、幀間預測部218、預測控制部220。As shown in FIG. 10, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse conversion unit 206, an addition unit 208, a block memory 210, a loop filtering unit 212, a frame memory 214, and an intra prediction unit. 216. An inter prediction unit 218 and a prediction control unit 220.

解碼裝置200例如是藉由通用處理器及記憶體而實現。此情況下，當藉由處理器而執行儲存在記憶體之軟體程式時，處理器是作為熵解碼部202、反量化部204、反轉換部206、相加部208、迴路濾波部212、幀內預測部216、幀間預測部218及預測控制部220而發揮功能。又，解碼裝置200亦可以是以對應於熵解碼部202、反量化部204、反轉換部206、相加部208、迴路濾波部212、幀內預測部216、幀間預測部218及預測控制部220之專用之1個以上之電子電路而實現。The decoding device 200 is implemented by, for example, a general-purpose processor and a memory. In this case, when the software program stored in the memory is executed by the processor, the processor functions as the entropy decoding unit 202, the inverse quantization unit 204, the inverse conversion unit 206, the addition unit 208, the loop filter unit 212, and the frame. The intra prediction section 216, the inter prediction section 218, and the prediction control section 220 function. The decoding device 200 may correspond to the entropy decoding unit 202, the inverse quantization unit 204, the inverse conversion unit 206, the addition unit 208, the loop filtering unit 212, the intra prediction unit 216, the inter prediction unit 218, and prediction control. The part 220 is realized by one or more electronic circuits.

以下，針對解碼裝置200含有之各構成要素進行說明。
[熵解碼部]Hereinafter, each constituent element included in the decoding device 200 will be described.
[Entropy decoding section]

熵解碼部202是對編碼位元串流進行熵解碼。具體而言，熵解碼部202例如是從編碼位元串流算術解碼成二值訊號。然後，熵解碼部202是令二值訊號多值化(debinarize)。藉此，熵解碼部202以區塊單位將量化係數朝反量化部204輸出。
[反量化部]The entropy decoding unit 202 performs entropy decoding on a coded bit stream. Specifically, the entropy decoding unit 202 arithmetically decodes the encoded bit stream into a binary signal, for example. The entropy decoding unit 202 debinarizes the binary signal. Thereby, the entropy decoding unit 202 outputs the quantization coefficient to the inverse quantization unit 204 in units of blocks.
[Dequantization Department]

反量化部204是對從熵解碼部202輸入之解碼對象區塊(以下稱作當前區塊)之量化係數進行反量化。具體而言，反量化部204是對於每個當前區塊之量化係數，基於與該量化係數對應之量化參數而將該量化係數反量化。然後，反量化部204是將當前區塊之經過反量化之量化係數(亦即轉換係數)朝反轉換部206輸出。
[反轉換部]The inverse quantization unit 204 performs inverse quantization on the quantization coefficients of the decoding target block (hereinafter referred to as the current block) input from the entropy decoding unit 202. Specifically, the inverse quantization unit 204 inversely quantizes the quantization coefficient of each current block based on a quantization parameter corresponding to the quantization coefficient. Then, the inverse quantization unit 204 outputs the inverse quantized quantized coefficients (that is, conversion coefficients) of the current block to the inverse conversion unit 206.
[Anti conversion department]

反轉換部206是藉由對從反量化部204輸入之轉換係數進行反轉換，而復原預測誤差。The inverse conversion unit 206 restores the prediction error by inversely converting the conversion coefficient input from the inverse quantization unit 204.

例如，當從編碼位元串流解讀之資訊是表示適用EMT或AMT的情況下(例如AMT旗標為真)，反轉換部206是基於解讀之表示轉換類型之資訊而將當前區塊之轉換係數反轉換。For example, when the information decoded from the encoded bit stream indicates that EMT or AMT is applicable (for example, the AMT flag is true), the inverse conversion unit 206 converts the current block based on the decoded information indicating the conversion type. Coefficient inverse conversion.

又例如，當從編碼位元串流解讀之資訊是表示適用NSST的情況下，反轉換部206是對轉換係數適用反再轉換。
[相加部]As another example, when the information decoded from the coded bit stream indicates that the NSST is applicable, the inverse conversion unit 206 applies the inverse reconversion to the conversion coefficient.
[Addition department]

相加部208是藉由將從反轉換部206輸入之預測誤差與從預測控制部220輸入之預測樣本予以相加，而重建當前區塊。然後，相加部208是將重建之區塊朝區塊記憶體210及迴路濾波部212輸出。
[區塊記憶體]The adding section 208 reconstructs the current block by adding the prediction error input from the inverse conversion section 206 and the prediction sample input from the prediction control section 220. Then, the adding unit 208 outputs the reconstructed block to the block memory 210 and the loop filtering unit 212.
[Block memory]

區塊記憶體210是用於儲存在幀內預測被參考之區塊且為解碼對象圖片(以下稱作當前圖片)內之區塊的記憶部。具體而言，區塊記憶體210儲存從相加部208輸出之重建區塊。
[迴路濾波部]The block memory 210 is a memory unit for storing a block to be referenced in the intra prediction and a block in a picture to be decoded (hereinafter referred to as a current picture). Specifically, the block memory 210 stores the reconstructed blocks output from the adding section 208.
[Loop Filtering Division]

迴路濾波部212是對藉由相加部208而重建之區塊施加迴路濾波，並將經過濾波之重建區塊朝幀記憶體214及顯示裝置等輸出。The loop filtering section 212 applies loop filtering to the blocks reconstructed by the adding section 208, and outputs the filtered reconstructed blocks to the frame memory 214, the display device, and the like.

當從編碼位元串流解讀之表示ALF之開啟/關閉之資訊是表示開啟ALF的情況下，基於局部性梯度之方向及活性度而從複數個濾波器中選擇1個濾波器，對重建區塊適用選擇之濾波器。
[幀記憶體]When the information indicating the on / off of ALF interpreted from the coded bit stream indicates that ALF is on, a filter is selected from a plurality of filters based on the direction and activity of the local gradient, and the reconstruction area is The block applies the selected filter.
[Frame memory]

幀記憶體214是用於儲存幀間預測用之參考圖片的記憶部，有時被稱作幀緩衝器。具體而言，幀記憶體214儲存被迴路濾波部212濾波後之重建區塊。
[幀內預測部]The frame memory 214 is a memory unit for storing reference pictures for inter prediction, and is sometimes called a frame buffer. Specifically, the frame memory 214 stores the reconstructed blocks filtered by the loop filtering unit 212.
[Intra prediction section]

幀內預測部216是基於從編碼位元串流解讀之幀內預測模式，參考儲存在區塊記憶體210之當前圖片內之區塊而進行幀內預測，藉此生成預測訊號(幀內預測訊號)。具體而言，幀內預測部216是參考與當前區塊鄰接之區塊之樣本(例如亮度值、色差值)而進行幀內預測，藉此生成幀內預測訊號，並將幀內預測訊號朝預測控制部220輸出。The intra prediction section 216 is based on the intra prediction mode decoded from the encoded bit stream, and performs intra prediction with reference to the blocks stored in the current picture of the block memory 210, thereby generating a prediction signal (intra prediction) Signal). Specifically, the intra prediction unit 216 performs intra prediction with reference to samples (e.g., luminance values and color difference values) of blocks adjacent to the current block, thereby generating an intra prediction signal, and converting the intra prediction signal. Output to the prediction control unit 220.

另，當在色差區塊之幀內預測中選擇參考亮度區塊之幀內預測模式的情況下，幀內預測部216亦可以基於當前區塊之亮度成分而預測當前區塊之色差成分。In addition, when the intra prediction mode of the reference luminance block is selected in the intra prediction of the color difference block, the intra prediction section 216 may also predict the color difference component of the current block based on the luminance component of the current block.

又，當從編碼位元串流解讀之資訊是表示適用PDPC的情況下，幀內預測部216基於水平/垂直方向之參考像素之梯度來修正幀內預測後之像素值。
[幀間預測部]When the information decoded from the encoded bit stream indicates that PDPC is applicable, the intra prediction unit 216 corrects the pixel values after intra prediction based on the gradient of the reference pixels in the horizontal / vertical direction.
[Inter prediction section]

幀間預測部218是參考儲存在幀記憶體214之參考圖片來預測當前區塊。預測是以當前區塊或當前區塊內之子區塊(例如4x4區塊)之單位來進行。例如，幀間預測部218是使用從編碼位元串流解讀之運動資訊(例如運動向量)而進行運動補償，藉此生成當前區塊或子區塊之幀間預測訊號，並將幀間預測訊號朝預測控制部220輸出。The inter prediction unit 218 refers to a reference picture stored in the frame memory 214 to predict a current block. The prediction is performed in units of the current block or a sub-block (for example, a 4x4 block) within the current block. For example, the inter prediction unit 218 performs motion compensation using motion information (for example, a motion vector) decoded from the encoded bit stream, thereby generating an inter prediction signal of the current block or a subblock, and performing inter prediction. The signal is output to the prediction control unit 220.

另，當從編碼位元串流解讀之資訊是表示適用OBMC模式的情況下，幀間預測部218是除了藉由運動估計所獲得之當前區塊之運動資訊之外，還使用鄰接區塊之運動資訊，而生成幀間預測訊號。In addition, when the information decoded from the coded bit stream indicates that the OBMC mode is applicable, the inter prediction unit 218 uses the motion information of the current block obtained by motion estimation, and also uses the adjacent block information. Motion information while generating inter prediction signals.

又，當從編碼位元串流解讀之資訊是表示適用FRUC模式的情況下，幀間預測部218是依循從編碼串流解讀之型樣匹配之方法(雙向匹配或模板匹配)來進行運動估計，藉此導出運動資訊。然後，幀間預測部218是使用導出之運動資訊來進行運動補償。When the information decoded from the encoded bit stream indicates that the FRUC mode is applicable, the inter prediction unit 218 performs motion estimation according to the pattern matching method (two-way matching or template matching) interpreted from the encoded stream. To export exercise information. Then, the inter prediction unit 218 performs motion compensation using the derived motion information.

又，幀間預測部218是在適用BIO模式的情況下，基於假設等速直線運動之模型而導出運動向量。又，當從編碼位元串流解讀之資訊是表示適用仿射運動補償預測模式的情況下，幀間預測部218是基於複數個鄰接區塊之運動向量而以子區塊單位將運動向量導出。
[預測控制部]When the BIO mode is applied, the inter prediction unit 218 derives a motion vector based on a model that assumes constant-speed linear motion. When the information decoded from the encoded bit stream indicates that the affine motion compensation prediction mode is applied, the inter prediction unit 218 derives the motion vector in units of sub-blocks based on the motion vectors of a plurality of adjacent blocks. .
[Predictive Control Department]

預測控制部220是選擇幀內預測訊號及幀間預測訊號之任一者，將選擇之訊號當作預測訊號而朝相加部208輸出。
[幀間預測處理之第1具體例]The prediction control unit 220 selects either an intra prediction signal or an inter prediction signal, and outputs the selected signal to the addition unit 208 as a prediction signal.
[First specific example of inter prediction processing]

圖11是顯示本具體例之管線(pipeline)構造的概略圖。本具體例之管線構造例是用於解碼動態圖像之管線構造例，亦可用在解碼裝置200。FIG. 11 is a schematic diagram showing a pipeline structure of this specific example. The pipeline configuration example of this specific example is a pipeline configuration example for decoding a moving image, and can also be used in the decoding device 200.

又，本具體例之管線構造包含第1階段、第2階段及第3階段之3個階段。又，第1階段包含熵解碼(S101)。第2階段包含決定MV(S102)、LIC(S103)、MC/BIO(S104)、OBMC(S105)、幀內預測(S106)、切換(S107)、反量化反轉換(S108)、及、相加(S109)。第3階段包含迴路濾波(S110)。In addition, the pipeline structure of this specific example includes three stages of a first stage, a second stage, and a third stage. The first stage includes entropy decoding (S101). The second stage includes determining the MV (S102), LIC (S103), MC / BIO (S104), OBMC (S105), intra prediction (S106), handover (S107), inverse quantization and inverse conversion (S108), and phase. Add (S109). The third stage includes loop filtering (S110).

在熵解碼(S101)，例如，解碼裝置200之熵解碼部202對輸入串流進行可變長度解碼等之熵解碼，藉此，取得用於解碼動態圖像之資訊。具體而言，熵解碼部202藉由熵解碼而取得量化係數等。In entropy decoding (S101), for example, the entropy decoding unit 202 of the decoding device 200 performs entropy decoding such as variable-length decoding on an input stream, thereby obtaining information for decoding a moving image. Specifically, the entropy decoding unit 202 obtains a quantization coefficient and the like by entropy decoding.

決定MV(S102)、LIC(S103)、MC/BIO(S104)、及、OBMC(S105)是構成幀間預測。在幀間預測，解碼裝置200之幀間預測部218是參考與處理對象圖片不同之參考圖片內之重建圖像，而生成處理對象圖片內之處理對象編碼單元(處理對象CU)之預測圖像。It is determined that MV (S102), LIC (S103), MC / BIO (S104), and OBMC (S105) constitute inter prediction. In inter prediction, the inter prediction unit 218 of the decoding device 200 refers to a reconstructed image in a reference picture different from the processing target picture, and generates a predicted image of a processing target coding unit (processing target CU) in the processing target picture. .

在決定MV(S102)，幀間預測部218是將處理對象編碼單元之運動向量預測子(MVP)導出。例如，幀間預測部218是將處理對象編碼單元之複數個運動向量預測子分別當作處理對象編碼單元之運動向量之複數個候選而導出。然後，幀間預測部218是從複數個候選中決定處理對象編碼單元之運動向量。In determining the MV (S102), the inter prediction unit 218 derives a motion vector predictor (MVP) of the coding unit to be processed. For example, the inter prediction unit 218 derives a plurality of motion vector predictors of the processing target coding unit as a plurality of candidates of motion vectors of the processing target coding unit. Then, the inter prediction unit 218 determines a motion vector of a coding unit to be processed from a plurality of candidates.

在LIC(S103)，幀間預測部218是藉由LIC(local illumination compensation)處理來進行處理對象編碼單元之預測圖像之生成或修正。In LIC (S103), the inter prediction unit 218 generates or corrects a prediction image of a processing target coding unit by performing LIC (local illumination compensation) processing.

例如，幀間預測部218是使用與處理對象編碼單元不同之區域之像素值來導出亮度修正參數。具體而言，幀間預測部218是使用處理對象編碼單元之周邊之亮度、及、藉由處理對象編碼單元之運動向量來特定出之特定之參考區塊之周邊之亮度，而導出亮度修正參數。然後，幀間預測部218是使用亮度修正參數來進行處理對象編碼單元之預測圖像之生成或修正。For example, the inter prediction unit 218 derives a brightness correction parameter using the pixel values of a region different from the processing target coding unit. Specifically, the inter prediction unit 218 derives a brightness correction parameter by using the brightness of the periphery of the processing target coding unit and the brightness of the periphery of a specific reference block specified by the motion vector of the processing target coding unit. . Then, the inter prediction unit 218 generates or corrects a prediction image of a processing target coding unit by using a brightness correction parameter.

更具體而言，如圖9D，藉由處理對象編碼單元(編碼對象區塊或解碼對象區塊)之運動向量而特定出參考圖片內之參考區塊。然後，依循與處理對象編碼單元鄰接之圖像區域、以及與參考區塊鄰接之圖像區域之間之亮度的變化，來算出亮度修正參數。More specifically, as shown in FIG. 9D, the reference block in the reference picture is specified by processing the motion vector of the target coding unit (coding target block or decoding target block). Then, the brightness correction parameter is calculated in accordance with the change in brightness between the image region adjacent to the processing target coding unit and the image region adjacent to the reference block.

然後，使用亮度修正參數對參考區塊之參考圖像進行亮度修正處理，藉此生成預測圖像。或者，使用亮度修正參數對從參考區塊之參考圖像導出之預測圖像進行亮度修正處理，藉此修正預測圖像。Then, the brightness correction parameter is used to perform brightness correction processing on the reference image of the reference block, thereby generating a predicted image. Alternatively, the brightness correction parameter is used to perform brightness correction processing on the prediction image derived from the reference image of the reference block, thereby correcting the prediction image.

在MC/BIO(S104)，幀間預測部218是藉由BIO(bi-directional optical flow)處理來進行處理對象編碼單元之預測圖像之生成或修正。例如，幀間預測部218是藉由運動補償(MC)處理而生成處理對象編碼單元之預測圖像，藉由BIO處理而修正處理對象編碼單元之預測圖像。In MC / BIO (S104), the inter prediction unit 218 generates or corrects a prediction image of a processing target coding unit through BIO (bi-directional optical flow) processing. For example, the inter prediction unit 218 generates a prediction image of the processing target coding unit by motion compensation (MC) processing, and corrects the prediction image of the processing target coding unit by BIO processing.

具體而言，幀間預測部218是藉由處理對象編碼單元之運動向量而特定出參考區塊，使用從參考區塊獲得之亮度之空間上的梯度來修正預測圖像。從參考區塊獲得之亮度之空間上的梯度亦可以是從參考區塊獲得之圖像中之亮度之空間上的梯度。從參考區塊獲得之圖像亦可以是參考圖像或預測圖像。預測圖像可以是小數像素精度之運動補償圖像。Specifically, the inter prediction unit 218 specifies a reference block by using a motion vector of a target coding unit, and corrects a predicted image using a spatial gradient of luminance obtained from the reference block. The spatial gradient of the brightness obtained from the reference block may also be the spatial gradient of the brightness in the image obtained from the reference block. The image obtained from the reference block may also be a reference image or a predicted image. The predicted image may be a motion-compensated image with a fractional pixel accuracy.

例如，如圖8，幀間預測部218是藉由處理對象編碼單元之2個運動向量預測子而特定出2個參考區塊，並從2個參考區塊取得2個預測圖像。然後，幀間預測部218是對2個預測圖像分別取得亮度之空間上的梯度，藉此取得2個梯度圖像。For example, as shown in FIG. 8, the inter prediction unit 218 specifies two reference blocks by using two motion vector predictors of the processing target coding unit, and obtains two predicted images from the two reference blocks. Then, the inter prediction unit 218 obtains two gradient images by obtaining the spatial gradient of the brightness for each of the two predicted images.

然後，幀間預測部218是基於2個預測圖像之複數個像素位置之複數個像素值、2個梯度圖像之複數個像素位置之複數個梯度值、及、光流等式等，而導出複數個像素位置之新的複數個預測像素值。藉此，幀間預測部218生成適用了BIO處理之預測圖像。亦即，幀間預測部218是藉由BIO處理來修正預測圖像。Then, the inter prediction unit 218 is based on the plurality of pixel values of the plurality of pixel positions of the two predicted images, the plurality of gradient values of the plurality of pixel positions of the two gradient images, and the optical flow equation, etc., and Derive a new plurality of predicted pixel values for the plurality of pixel positions. Thereby, the inter prediction unit 218 generates a prediction image to which BIO processing is applied. That is, the inter prediction unit 218 corrects the predicted image by BIO processing.

在OBMC(S105)，幀間預測部218是藉由OBMC(overlapped block motion compensation)處理而進行處理對象編碼單元之預測圖像之生成或修正。例如，使用藉由處理對象編碼單元之運動向量來特定出之參考圖像、及、藉由與處理對象編碼單元鄰接之鄰接編碼單元之運動向量而特定出之參考圖像，進行處理對象編碼單元之預測圖像之生成或修正。In OBMC (S105), the inter prediction unit 218 generates or corrects a predicted image of a processing target coding unit by performing OBMC (overlapped block motion compensation) processing. For example, a processing target coding unit is performed using a reference image specified by a motion vector of a processing target coding unit and a reference image specified by a motion vector of an adjacent coding unit adjacent to the processing target coding unit. Generation or correction of predicted images.

具體而言，如圖5B及圖5C，幀間預測部218特定出藉由處理對象編碼單元之運動向量而指出之參考區塊。又，幀間預測部218特定出藉由與處理對象編碼單元鄰接之1個以上之編碼單元之1個以上之運動向量而指出之1個以上之參考區塊。Specifically, as shown in FIG. 5B and FIG. 5C, the inter prediction unit 218 specifies a reference block indicated by the motion vector of the processing target coding unit. In addition, the inter prediction unit 218 specifies one or more reference blocks specified by one or more motion vectors of one or more coding units adjacent to the processing target coding unit.

然後，幀間預測部218是將由該等複數個參考區塊之複數個參考圖像獲得之複數個預測圖像重疊，藉此生成新的預測圖像。換句話說，幀間預測部218是使用藉由已處理之周邊編碼單元之運動向量而獲得之參考圖像，來修正藉由處理對象編碼單元之運動向量而獲得之預測圖像。在此，已處理之周邊編碼單元是經過處理之編碼單元，且是處理對象編碼單元之周邊之編碼單元。Then, the inter prediction unit 218 overlaps the plurality of prediction images obtained from the plurality of reference images of the plurality of reference blocks, thereby generating a new prediction image. In other words, the inter prediction unit 218 uses the reference image obtained by processing the motion vectors of the surrounding coding units to correct the predicted image obtained by processing the motion vectors of the target coding units. Here, the processed surrounding coding unit is a processed coding unit, and is a coding unit around a processing target coding unit.

在幀內預測(S106)，解碼裝置200之幀內預測部216是參考處理對象圖片內之重建圖像而生成處理對象編碼單元之預測圖像。In the intra prediction (S106), the intra prediction unit 216 of the decoding device 200 refers to the reconstructed image in the processing target picture to generate a prediction image of the processing target coding unit.

在切換(S107)，將藉由決定MV(S102)、LIC(S103)、MC/BIO(S104)、OBMC(S105)等之幀間預測而獲得之預測圖像、以及、藉由幀內預測(S106)而獲得之預測圖像予以切換。In switching (S107), a prediction image obtained by determining inter prediction of MV (S102), LIC (S103), MC / BIO (S104), OBMC (S105), etc., and, by intra prediction (S106) The predicted image obtained is switched.

在反量化反轉換(S108)，解碼裝置200之反量化部204及反轉換部206是使用藉由熵解碼而獲得之資訊，進行反量化反轉換處理。具體而言，反量化部204及反轉換部206是對量化係數進行反量化反轉換處理。藉此，反量化部204及反轉換部206將身為原圖像與預測圖像之預測誤差之差分圖像復原。In the inverse quantization and inverse conversion (S108), the inverse quantization section 204 and the inverse conversion section 206 of the decoding device 200 perform inverse quantization and inverse conversion processing using information obtained by entropy decoding. Specifically, the inverse quantization unit 204 and the inverse conversion unit 206 perform inverse quantization and inverse conversion processing on the quantized coefficients. Thereby, the inverse quantization unit 204 and the inverse conversion unit 206 restore the difference image which is a prediction error between the original image and the predicted image.

在相加(S109)，解碼裝置200之相加部208是將差分圖像與預測圖像相加，藉此重建原圖像。亦即，相加部208生成重建圖像。In the addition (S109), the addition unit 208 of the decoding device 200 adds the difference image and the predicted image to reconstruct the original image. That is, the adding section 208 generates a reconstructed image.

在迴路濾波(S110)，解碼裝置200之迴路濾波部212是對重建圖像適用迴路濾波處理。藉此，例如，迴路濾波部212將編碼單元間之歪曲予以抑制。然後，迴路濾波部212是將適用了濾波之重建圖像當作解碼圖像而生成。In the loop filtering (S110), the loop filtering unit 212 of the decoding device 200 applies a loop filtering process to the reconstructed image. Thereby, for example, the loop filter unit 212 suppresses distortion between coding units. The loop filter unit 212 generates the reconstructed image to which the filtering is applied as a decoded image.

在第2階段生成之重建圖像有可能在後續之編碼單元之LIC(S103)及幀內預測(S106)，被當作進行LIC(S103)及幀內預測(S106)之編碼單元之周邊之重建圖像來參考。因此，在第2階段生成之重建圖像是作為LIC(S103)及幀內預測(S106)之輸入而反饋。The reconstructed image generated in the second stage may be used in the LIC (S103) and intra prediction (S106) of the subsequent coding unit as the surrounding of the coding unit that performs the LIC (S103) and intra prediction (S106). Reconstructed images for reference. Therefore, the reconstructed image generated in the second stage is fed back as input of LIC (S103) and intra prediction (S106).

在本具體例之管線構造中，反饋是包含於第2階段內而未橫跨複數個階段，故在LIC(S103)及幀內預測(S106)中，可將以處理順序而言位於前一個之編碼單元之重建圖像拿來參考。另一方面，在本具體例之管線構造中，第2階段含有多個處理。所以，第2階段之處理時間長。In the pipeline structure of this specific example, the feedback is included in the second stage without crossing multiple stages. Therefore, in the LIC (S103) and the intra prediction (S106), the processing order can be placed in the previous one. The reconstructed image of the coding unit is used for reference. On the other hand, in the pipeline structure of this specific example, the second stage includes a plurality of processes. Therefore, the processing time of the second stage is long.

另，處理時間是對應於處理量或處理週期數。又，本具體例之管線構造是管線構造之一例，可以對處理進行部分性之去除，亦可以對處理進行部分性之追加，亦可以改變階段之分割方法。例如，亦可以是在決定MV之後進行運動補償(MC)而生成預測圖像。之後，亦可以適應性地進行LIC處理、BIO處理及OBMC處理而修正預測圖像。The processing time corresponds to the processing amount or the number of processing cycles. In addition, the pipeline structure of this specific example is an example of the pipeline structure, and the processing may be partially removed, the processing may be partially added, and the division method of the phases may be changed. For example, it is also possible to generate a prediction image by performing motion compensation (MC) after determining the MV. After that, the LIC process, the BIO process, and the OBMC process can also be adaptively performed to correct the predicted image.

圖12是顯示使用在管線處理之說明之區塊分割例的示意圖。圖12所示之區塊分割例顯示有2個編碼樹單元。其中一方之編碼樹單元含有2個編碼單元CU0及CU1，另一方之編碼樹單元含有3個編碼單元CU2、CU3及CU4。FIG. 12 is a diagram showing an example of block division used in the description of pipeline processing. The block division example shown in FIG. 12 shows two coding tree units. One coding tree unit contains two coding units CU0 and CU1, and the other coding tree unit contains three coding units CU2, CU3 and CU4.

編碼單元CU0、CU1及CU4互為相同之尺寸。編碼單元CU2及CU3互為相同之尺寸。編碼單元CU0、CU1及CU4之各者之尺寸是編碼單元CU2及CU3之各者之尺寸的2倍。The coding units CU0, CU1 and CU4 are the same size as each other. The coding units CU2 and CU3 are the same size as each other. The size of each of the coding units CU0, CU1, and CU4 is twice the size of each of the coding units CU2 and CU3.

另，編碼樹單元是含有1個以上之編碼單元之區塊，且是具有事先決定之尺寸之正方形區塊。例如，構成動態圖像之複數個圖片所含有之複數個編碼樹單元具有事先決定之共通之尺寸。編碼單元是包含於編碼樹單元之矩形區塊，具有可變的尺寸。例如，亦可以是依各編碼樹單元而決定包含於編碼樹單元之編碼單元的尺寸。In addition, the coding tree unit is a block containing one or more coding units, and is a square block having a size determined in advance. For example, a plurality of coding tree units included in a plurality of pictures constituting a moving image have a common size determined in advance. A coding unit is a rectangular block contained in a coding tree unit and has a variable size. For example, the size of a coding unit included in a coding tree unit may be determined for each coding tree unit.

圖13是顯示圖11所說明之本具體例之管線構造的處理時機之一例的時序圖。圖13顯示有圖12所示之5個編碼單元CU0~CU4的處理時機。又，圖13之S1~S3是表示圖11之第1階段~第3階段之處理時間。FIG. 13 is a timing chart showing an example of processing timing of the pipeline structure of the specific example described in FIG. 11. FIG. 13 shows processing timings of the five coding units CU0 to CU4 shown in FIG. 12. Note that S1 to S3 in FIG. 13 show the processing times of the first stage to the third stage in FIG. 11.

由於編碼單元CU0、CU1及CU4之各者之尺寸是編碼單元CU2及CU3之各者之尺寸的2倍，故關於編碼單元CU0、CU1及CU4，各階段之處理時間亦為2倍。又，由於第2階段含有多個處理，故第2階段之處理時間是其他階段之處理時間的2倍。又，關於各階段，當對以處理順序而言位於前一個之編碼單元進行之同一階段之處理結束後，才讓對下一個編碼單元進行之同一階段之處理開始。Since the size of each of the coding units CU0, CU1, and CU4 is twice the size of each of the coding units CU2 and CU3, the processing time for each stage of the coding units CU0, CU1, and CU4 is also doubled. In addition, since the second stage includes a plurality of processes, the processing time in the second stage is twice that of the other stages. With regard to each stage, the processing of the same stage for the next coding unit is not started until the processing of the same stage for the coding unit located in the previous one in terms of processing order ends.

例如，對編碼單元CU1進行之第2階段之處理是從對編碼單元CU0進行之第2階段之處理結束之時間t6開始。由於對編碼單元CU0進行之第2階段之處理時間是第1階段的2倍，故就編碼單元CU1而言，從第1階段之處理結束之時間t4至第2階段之處理開始之時間t6為止，發生了等待時間。For example, the second-stage processing performed on the coding unit CU1 starts at time t6 when the second-stage processing performed on the coding unit CU0 ends. Since the processing time of the second stage of the coding unit CU0 is twice as long as that of the first stage, as far as the coding unit CU1 is concerned, from the time t4 when the processing of the first stage ends to the time t6 when the processing of the second stage starts , A wait time has occurred.

自編碼單元CU1以後，在第2階段之處理之開始，總是發生等待時間。然後，就編碼單元CU0、CU2、CU3及CU4之各者而言，等待時間被累積。然後，就編碼單元CU4而言，從第1階段之處理結束之時間t8至第2階段之處理開始之時間t14為止，發生了冗長的等待時間。Since the encoding unit CU1, the waiting time always occurs at the beginning of the processing in the second stage. Then, for each of the coding units CU0, CU2, CU3, and CU4, the waiting time is accumulated. Then, for the encoding unit CU4, a long waiting time occurs from the time t8 when the processing in the first stage ends to the time t14 when the processing in the second stage starts.

結果，在1圖片之處理中，將等待時間含括在內之處理時間是增加至不將等待時間含括在內之原來之處理時間的約2倍，有可能造成難以在分配給1圖片之時間內，完成1圖片內之全部之編碼單元之處理。As a result, in the processing of 1 picture, the processing time including the waiting time is increased to about 2 times the original processing time without including the waiting time, which may make it difficult to allocate to the 1 picture. In time, complete the processing of all coding units in one picture.

圖14是顯示圖11所說明之本具體例之管線構造的幀間預測之動作的流程圖。例如，解碼裝置200之幀間預測部218進行圖14所示之動作。又，編碼裝置100之幀間預測部126亦可以與解碼裝置200之幀間預測部218同樣地進行圖14所示之動作。FIG. 14 is a flowchart showing the operation of the inter prediction of the pipeline structure of the specific example described in FIG. 11. For example, the inter prediction unit 218 of the decoding device 200 performs the operation shown in FIG. 14. The inter prediction unit 126 of the encoding device 100 may perform the operation shown in FIG. 14 in the same manner as the inter prediction unit 218 of the decoding device 200.

幀間預測之模式是可從包含一般幀間模式及合併模式等之複數個模式中來選擇。幀間預測部218是從複數個模式中選擇幀間預測之模式，使用所選擇之模式來決定運動向量(MV)。The inter prediction mode can be selected from a plurality of modes including a general inter mode and a merge mode. The inter prediction unit 218 selects a mode of inter prediction from among a plurality of modes, and determines a motion vector (MV) using the selected mode.

一般幀間模式的情況下(在S201為一般幀間模式)，幀間預測部218是參考處理對象編碼單元之周邊之複數個已處理編碼單元之複數個運動向量，而取得複數個運動向量預測子(MVP)。藉此，幀間預測部218製作包含複數個運動向量預測子之一般用MVP清單。然後，幀間預測部218從製作之一般用MVP清單中選擇1個運動向量預測子。In the case of the general inter mode (the general inter mode in S201), the inter prediction unit 218 refers to a plurality of motion vectors of a plurality of processed coding units around a processing target coding unit, and obtains a plurality of motion vector predictions. Child (MVP). As a result, the inter prediction unit 218 creates a general MVP list including a plurality of motion vector predictors. Then, the inter prediction unit 218 selects one motion vector predictor from the list of general MVPs to be created.

然後，幀間預測部218是在選擇之運動向量預測子加上差分運動向量(MVD)，藉此決定處理對象單元之運動向量(S202)。Then, the inter prediction unit 218 adds a differential motion vector (MVD) to the selected motion vector predictor, thereby determining the motion vector of the processing target unit (S202).

另，在編碼裝置100，幀間預測部126是藉由運動檢測等來決定運動向量，將運動向量與運動向量預測子之差分運動向量導出。然後，熵編碼部110將差分運動向量編碼。然後，在解碼裝置200，熵解碼部202將差分運動向量解碼，藉此導出差分運動向量。In the encoding device 100, the inter prediction unit 126 determines a motion vector by motion detection or the like, and derives a difference motion vector between the motion vector and the motion vector predictor. Then, the entropy encoding unit 110 encodes a differential motion vector. Then, in the decoding device 200, the entropy decoding unit 202 decodes the differential motion vector, thereby deriving a differential motion vector.

又，合併模式的情況下(在S201為合併模式)，幀間預測部218是參考處理對象編碼單元之周邊之複數個已處理編碼單元之複數個運動向量，而取得複數個運動向量預測子(MVP)。藉此，幀間預測部218製作包含複數個運動向量預測子之合併用MVP清單。然後，幀間預測部218從製作之合併用MVP清單中選擇1個運動向量預測子。In the case of the merge mode (the merge mode in S201), the inter prediction unit 218 refers to the plurality of motion vectors of the plurality of processed coding units around the processing target coding unit, and obtains a plurality of motion vector predictors ( MVP). Thereby, the inter prediction unit 218 creates a merge MVP list including a plurality of motion vector predictors. Then, the inter prediction unit 218 selects one motion vector predictor from the created merge MVP list.

然後，幀間預測部218是將選擇之運動向量預測子直接決定為處理對象編碼單元之運動向量(S203)。Then, the inter prediction unit 218 directly determines the selected motion vector predictor as the motion vector of the processing target coding unit (S203).

接著，幀間預測部218判定當使用藉由一般幀間模式或合併模式決定之運動向量進行運動補償時，是否適用LIC處理(S204)。Next, the inter prediction unit 218 determines whether the LIC process is applied when motion compensation is performed using a motion vector determined by a general inter mode or a merge mode (S204).

在此，當判定成適用LIC處理的情況下(在S204為「是」)，幀間預測部218是依循處理對象編碼單元之運動向量，而進行伴隨著將處理對象編碼單元之周邊之重建圖像予以參考之LIC處理之運動補償(S206)。藉此，幀間預測部218生成預測圖像。Here, when it is determined that the LIC processing is applicable (YES in S204), the inter prediction unit 218 performs a reconstruction image along with the motion vector of the processing target coding unit, and the surroundings of the processing target coding unit. Motion compensation (S206) like LIC processing for reference. Thereby, the inter prediction unit 218 generates a predicted image.

另一方面，當判定成不適用LIC處理的情況下(在S204為「否」)，幀間預測部218是依循處理對象編碼單元之運動向量而進行不伴隨著LIC處理之通常之運動補償(S205)。藉此，幀間預測部218生成預測圖像。On the other hand, when it is determined that the LIC processing is not applicable (NO in S204), the inter prediction unit 218 performs normal motion compensation without accompanying LIC processing in accordance with the motion vector of the processing target coding unit ( S205). Thereby, the inter prediction unit 218 generates a predicted image.

例如，當適用LIC處理的情況下，幀間預測部218使用與處理對象編碼單元鄰接之編碼單元之重建圖像。因此，使用圖11所示之反饋處理。另，關於是否適用LIC處理之判定，可以是依循處理對象編碼單元之預測模式、預測方向或尺寸等之條件而進行。又，亦可以將表示是否適用LIC處理之訊號明示地描述在串流。而且，亦可以依循此訊號來進行判定。For example, when LIC processing is applied, the inter prediction unit 218 uses a reconstructed image of a coding unit adjacent to the processing target coding unit. Therefore, the feedback processing shown in FIG. 11 is used. The determination as to whether the LIC process is applicable may be performed in accordance with conditions such as a prediction mode, a prediction direction, and a size of a coding unit to be processed. In addition, a signal indicating whether LIC processing is applicable may be explicitly described in the stream. Moreover, you can follow this signal to make a decision.

接著，幀間預測部218針對處理對象編碼單元來判定是否適用BIO處理(S207)。在此，當判定成適用BIO處理的情況下(在S207為「是」)，幀間預測部218使用BIO處理來對生成之預測圖像進行修正(S208)。另一方面，當判定成不適用BIO處理的情況下(在S207為「否」)，幀間預測部218不對生成之預測圖像進行基於BIO處理之修正。Next, the inter prediction unit 218 determines whether or not the BIO processing is applied to the processing target coding unit (S207). If it is determined that the BIO process is applicable (YES in S207), the inter prediction unit 218 uses the BIO process to correct the generated predicted image (S208). On the other hand, when it is determined that the BIO processing is not applicable (NO in S207), the inter prediction unit 218 does not perform correction on the generated prediction image based on the BIO processing.

關於是否適用BIO處理之判定，亦可以是依循處理對象編碼單元之預測模式、預測方向或尺寸等之條件而進行。又，亦可以將表示是否適用BIO處理之訊號明示地描述在串流。而且，亦可以依循此訊號來進行判定。The determination as to whether or not the BIO processing is applicable may be performed in accordance with conditions such as a prediction mode, a prediction direction, or a size of a processing target coding unit. In addition, a signal indicating whether BIO processing is applicable may be explicitly described in the stream. Moreover, you can follow this signal to make a decision.

接著，幀間預測部218針對處理對象編碼單元來判定是否適用OBMC處理(S209)。在此，當判定成適用OBMC處理的情況下(在S209為「是」)，幀間預測部218使用OBMC處理來對生成之預測圖像進行修正處理，藉此生成最終之預測圖像(S210)。另一方面，當判定成不適用OBMC處理的情況下(在S209為「否」)，幀間預測部218不使用OBMC處理，而將生成之預測圖像直接當作最終之預測圖像。Next, the inter prediction unit 218 determines whether or not the OBMC process is applied to the processing target coding unit (S209). Here, when it is determined that the OBMC processing is applicable (YES in S209), the inter prediction unit 218 uses the OBMC processing to perform correction processing on the generated prediction image, thereby generating a final prediction image (S210 ). On the other hand, when it is determined that the OBMC process is not applicable (NO in S209), the inter prediction unit 218 does not use the OBMC process, and directly uses the generated prediction image as the final prediction image.

關於是否適用OBMC處理之判定，亦可以是依循處理對象編碼單元之預測模式、預測方向或尺寸等之條件而進行。又，亦可以將表示是否適用OBMC處理之訊號明示地描述在串流。而且，亦可以依循此訊號來進行判定。The determination as to whether the OBMC process is applicable may be performed in accordance with conditions such as a prediction mode, a prediction direction, and a size of a coding unit to be processed. In addition, a signal indicating whether the OBMC processing is applicable may be explicitly described in the stream. Moreover, you can follow this signal to make a decision.

上述之處理(S201~S210)是依各編碼單元而反覆進行。另，幀間預測並非限定於依各編碼單元，可以是依各預測單元而進行，亦可以是依各其他之區塊而進行。The above processing (S201 ~ S210) is performed repeatedly according to each coding unit. In addition, the inter prediction is not limited to each coding unit, and may be performed according to each prediction unit, or may be performed according to other blocks.

如上述，當在處理對象編碼單元適用LIC處理的情況下，要進行之處理比不適用LIC處理的情況還多。結果，處理時間變長，如圖13之說明，在管線控制中，有增加用於開始第2階段之處理之等待時間的可能性。As described above, when the LIC processing is applied to the processing target coding unit, more processing is performed than when the LIC processing is not applied. As a result, the processing time becomes longer. As illustrated in FIG. 13, in the pipeline control, there is a possibility that the waiting time for starting the processing of the second stage may be increased.

另，上述之處理流程是一例，亦可以將上述之處理之一部分予以去除，亦可以將未顯示在上述之處理予以追加。例如，幀間預測部218亦可以在藉由進行通常之運動補償而生成預測圖像後，藉由適應性地進行LIC處理、或、藉由適應性地進行BIO處理及OBMC處理，而修正預測圖像。In addition, the above-mentioned processing flow is an example, and a part of the above-mentioned processing may be removed, or the processing not shown above may be added. For example, the inter prediction unit 218 may modify the prediction by adaptively performing LIC processing or adaptively performing BIO processing and OBMC processing after generating a predicted image by performing normal motion compensation. image.

又，雖然編碼裝置100及解碼裝置200具有將用於處理之訊號編碼在串流、或從串流予以解碼之差異，但對編碼裝置100與解碼裝置200而言，在此說明之處理流程基本上是共通的。例如，亦可以在編碼裝置100，藉由幀間預測部126進行與在解碼裝置200藉由幀間預測部218進行之處理相同之處理。In addition, although the encoding device 100 and the decoding device 200 have a difference in encoding or decoding a signal for processing in a stream, the processing flow described here is basically the same for the encoding device 100 and the decoding device 200. The above is common. For example, the encoding device 100 may perform the same processing in the inter prediction unit 126 as the decoding device 200 in the inter prediction unit 218.

又，雖然在此是依各編碼單元而進行處理，但亦可以依各任意決定之區塊而進行處理。所以，亦可以將在此之編碼單元替換成區塊。
[幀間預測處理之第2具體例]In addition, although processing is performed for each coding unit here, processing may be performed for each arbitrarily determined block. Therefore, the coding unit here can also be replaced with a block.
[Second specific example of inter prediction processing]

本具體例可將進行LIC處理之情況下之幀間預測之處理量之增加予以抑制。In this specific example, it is possible to suppress an increase in the processing amount of inter prediction when the LIC processing is performed.

圖15是顯示本具體例之管線構造的概略圖。與第1具體例之管線構造例同樣，本具體例之管線構造例是用於解碼動態圖像之管線構造例，可以是用在解碼裝置200。FIG. 15 is a schematic diagram showing a pipeline structure of this specific example. Similar to the pipeline structure example of the first specific example, the pipeline structure example of this specific example is a pipeline structure example for decoding a moving image, and may be used in the decoding device 200.

又，與第1具體例之管線構造例同樣，本具體例之管線構造包含第1階段、第2階段及第3階段之3個階段。又，第1階段包含熵解碼(S101)。第2階段包含決定MV(S102)、LIC(S103)、MC(S111)、MC/BIO(S104)、OBMC(S105)、幀內預測(S106)、切換(S107)、反量化反轉換(S108)、及、相加(S109)。第3階段包含迴路濾波(S110)。Also, similar to the pipeline structure example of the first specific example, the pipeline structure of this specific example includes three stages of a first stage, a second stage, and a third stage. The first stage includes entropy decoding (S101). The second stage includes the decision MV (S102), LIC (S103), MC (S111), MC / BIO (S104), OBMC (S105), intra prediction (S106), handover (S107), inverse quantization and inverse conversion (S108 ), And, add (S109). The third stage includes loop filtering (S110).

在MC(S111)，解碼裝置200之幀間預測部218是藉由運動補償(MC)處理而生成處理對象編碼單元之預測圖像。熵解碼(S101)、決定MV(S102)、LIC(S103)、MC/BIO(S104)、OBMC(S105)、幀內預測(S106)、切換(S107)、反量化反轉換(S108)、相加(S109)、及、迴路濾波(S110)之各者之具體處理是與第1具體例相同。In MC (S111), the inter prediction unit 218 of the decoding device 200 generates a prediction image of a processing target coding unit through motion compensation (MC) processing. Entropy decoding (S101), decision MV (S102), LIC (S103), MC / BIO (S104), OBMC (S105), intra prediction (S106), switching (S107), inverse quantization and inverse conversion (S108), phase The specific processing of each of the addition (S109) and the loop filtering (S110) is the same as that of the first specific example.

不過，以下是與第1具體例相異：在幀間預測中，當進行LIC(S103)的情況下，不進行身為與生成預測圖像相關之處理之BIO/MC(S104)及OBMC(S105)，只進行MC(S111)，而生成預測圖像。在幀間預測中，當不進行LIC(S103)的情況下，與第1具體例同樣，適應性地進行MC/BIO(S104)及OBMC(S105)，而生成預測圖像。However, the following is different from the first specific example: In the case of LIC (S103) during inter prediction, BIO / MC (S104) and OBMC ( S105), perform only MC (S111), and generate a prediction image. In the inter prediction, when LIC is not performed (S103), similar to the first specific example, MC / BIO (S104) and OBMC (S105) are adaptively performed to generate a predicted image.

在第2階段生成之重建圖像有可能在後續之編碼單元之LIC(S103)及幀內預測(S106)，被當作進行幀內預測(S106)及LIC(S106)之編碼單元之周邊之重建圖像來參考。因此，在第2階段生成之重建圖像是作為LIC(S103)及幀內預測(S106)之輸入而反饋。The reconstructed image generated in the second stage may be used in the surrounding LIC (S103) and intra prediction (S106) of the coding unit as the surrounding of the coding unit for intra prediction (S106) and LIC (S106). Reconstructed images for reference. Therefore, the reconstructed image generated in the second stage is fed back as input of LIC (S103) and intra prediction (S106).

相較於第1具體例，本具體例是令第2階段之與幀間預測相關之連續之處理數減少。因此，即便與第1具體例同樣地進行反饋處理，第2階段整體之處理時間亦大幅地縮短。Compared with the first specific example, this specific example reduces the number of consecutive processes related to inter prediction in the second stage. Therefore, even if the feedback processing is performed in the same manner as the first specific example, the overall processing time in the second stage is greatly shortened.

另，本具體例之管線構造是管線構造之一例，可以對處理進行部分性之去除，亦可以對處理進行部分性之追加，亦可以改變階段之分割方法。例如，亦可以是在決定MV之後進行運動補償(MC)而生成預測圖像。之後，亦可以是適應性地進行LIC處理、BIO處理及OBMC處理而修正預測圖像。In addition, the pipeline structure of this specific example is an example of the pipeline structure. The processing can be partially removed, the processing can be partially added, and the division method of the phases can be changed. For example, it is also possible to generate a prediction image by performing motion compensation (MC) after determining the MV. After that, the LIC process, the BIO process, and the OBMC process may be adaptively performed to correct the predicted image.

圖16是顯示圖15所說明之本具體例之管線構造的處理時機之一例的時序圖。在圖16顯示著圖12所示之5個編碼單元CU0~CU4的處理時機。與第1具體例同樣，由於編碼單元CU0、CU1及CU4之各者之尺寸是編碼單元CU2及CU3之各者之尺寸的2倍，故關於編碼單元CU0、CU1及CU4，各階段之處理時間亦為2倍。FIG. 16 is a timing chart showing an example of processing timing of the pipeline structure of the specific example described in FIG. 15. FIG. 16 shows processing timings of the five coding units CU0 to CU4 shown in FIG. 12. As in the first specific example, since the size of each of the coding units CU0, CU1, and CU4 is twice the size of each of the coding units CU2 and CU3, the processing time for each stage of the coding units CU0, CU1, and CU4 It is also 2 times.

另一方面，相較於第1具體例，本具體例中之第2階段之處理時間短，本具體例中之第2階段之處理時間的長度是與其他階段之處理時間的長度相同。而且，在各階段，當對以處理順序而言位於前一個之編碼單元進行之同一階段之處理結束後，才讓對下一個編碼單元進行之同一階段之處理開始。On the other hand, compared with the first specific example, the processing time of the second stage in this specific example is shorter, and the length of the processing time of the second stage in this specific example is the same as that of the other stages. Moreover, in each stage, the processing of the same stage for the next coding unit is not started until the processing of the same stage for the coding unit located in the previous one in terms of processing order is completed.

例如，對編碼單元CU1進行之第2階段之處理是從對編碼單元CU0進行之第2階段之處理結束之時間t4開始。由於對編碼單元CU0進行之第2階段之處理時間是與其他階段相同，故在對編碼單元CU1進行之第1階段之處理結束後，可以無等待時間地開始第2階段之處理。For example, the second-stage processing performed on the coding unit CU1 starts from the time t4 when the second-stage processing performed on the coding unit CU0 ends. Since the processing time of the second stage of the coding unit CU0 is the same as that of the other stages, after the processing of the first stage of the coding unit CU1 is completed, the processing of the second stage can be started without waiting time.

另一方面，由於編碼單元CU2的尺寸比以處理順序而言位於前一個之編碼單元CU1的尺寸小，故就編碼單元CU2而言，在第1階段與第2階段之間發生等待時間。不過，此等待時間不會累積，在編碼單元CU4之處理就沒有等待時間。On the other hand, since the size of the coding unit CU2 is smaller than the size of the coding unit CU1 that is located immediately before in terms of processing order, as for the coding unit CU2, a waiting time occurs between the first stage and the second stage. However, this waiting time does not accumulate, and there is no waiting time in the processing of the coding unit CU4.

結果，在1圖片之處理中，將等待時間含括在內之處理時間是與不將等待時間含括在內之原來之處理時間幾乎相等。所以，在分配給1圖片之時間內令1圖片之處理完成之可能性變高。As a result, in the processing of one picture, the processing time including the waiting time is almost equal to the original processing time not including the waiting time. Therefore, there is a high possibility that the processing of one picture is completed within the time allocated to one picture.

圖17是顯示圖15所說明之本具體例之管線構造的幀間預測之動作的流程圖。例如，解碼裝置200之幀間預測部218進行圖17所示之動作。又，編碼裝置100之幀間預測部126亦可以與解碼裝置200之幀間預測部218同樣地進行圖17所示之動作。基本上，圖17所示之本具體例之複數個處理(S201~S210)是分別與圖14所示之第1具體例之複數個處理(S201~S210)相同。FIG. 17 is a flowchart showing the operation of inter prediction of the pipeline structure of the specific example described in FIG. 15. For example, the inter prediction unit 218 of the decoding device 200 performs the operation shown in FIG. 17. The inter prediction unit 126 of the encoding device 100 may perform the operation shown in FIG. 17 in the same manner as the inter prediction unit 218 of the decoding device 200. Basically, the plural processes (S201 to S210) of the specific example shown in FIG. 17 are the same as the plural processes (S201 to S210) of the first specific example shown in FIG. 14, respectively.

相較於第1具體例，在本具體例中，幀間預測部218是當判定成在運動補償適用LIC處理的情況下(在S204為「是」)，不進行身為與生成預測圖像相關之處理之BIO處理(S208)及OBMC處理(S210)。於是，幀間預測部218是只進行伴隨LIC處理之運動補償(S206)，而生成最終之預測圖像。Compared with the first specific example, in this specific example, when the inter prediction unit 218 determines that the LIC processing is applied to the motion compensation (YES in S204), it does not perform the appearance and generation of a prediction image. The related processing is BIO processing (S208) and OBMC processing (S210). Then, the inter prediction unit 218 performs motion compensation only with the LIC process (S206), and generates a final predicted image.

藉此，可將對處理對象編碼單元適用LIC處理之情況下之幀間預測之處理量之增加予以抑制。而且，可設想的是，適用LIC處理之情況下之幀間預測之處理時間會比適用BIO處理及OBMC處理之情況還短。結果，如圖16之說明，在管線控制中，將用於讓第2階段之處理開始之等待時間予以抑制之可能性變高。This makes it possible to suppress an increase in the processing amount of inter prediction when the LIC processing is applied to the processing target coding unit. Moreover, it is conceivable that the processing time of the inter prediction when the LIC processing is applied will be shorter than when the BIO processing and the OBMC processing are applied. As a result, as illustrated in FIG. 16, in pipeline control, there is a high possibility that the waiting time for starting the processing in the second stage is suppressed.

在圖17之例中，當對處理對象編碼單元適用LIC處理的情況下，不進行BIO處理與OBMC處理雙方而生成預測圖像。然而，此情況下，亦可以只不進行BIO處理而生成預測圖像，亦可以只不進行OBMC處理而生成預測圖像。藉此，雖然處理量及處理時間之削減效果降低，但可能生成精度較高之預測圖像。所以，編碼效率提升之可能性變高。In the example of FIG. 17, when the LIC processing is applied to the processing target coding unit, the prediction image is generated without performing both the BIO processing and the OBMC processing. However, in this case, the prediction image may be generated without performing BIO processing only, or the prediction image may be generated without performing OBMC processing only. Thereby, although the reduction effect of the processing amount and processing time is reduced, a highly accurate prediction image may be generated. Therefore, the possibility of improving coding efficiency becomes higher.

尤其，BIO處理是使用基於光流等式等之複雜之算術式。因此，可設想成BIO處理之處理時間長。所以，藉由控制成當適用LIC處理的情況下不進行BIO處理，可抑制第2階段之處理時間變得過長之情形。In particular, BIO processing uses complex arithmetic expressions based on optical flow equations and the like. Therefore, it can be considered that the processing time of BIO processing is long. Therefore, by controlling such that the BIO process is not performed when the LIC process is applied, it is possible to prevent the processing time in the second stage from becoming excessively long.

又，若在判定是否適用LIC處理後，後續有BIO處理及OBMC處理以外之與生成預測圖像相關之別的處理，則亦可依循是否適用LIC處理之判定結果而將該別的處理也跳過。例如，與生成預測圖像相關之處理是為了令預測圖像接近原圖像而修正預測圖像之處理。In addition, after determining whether the LIC process is applicable, if there are other processes related to the prediction image generation other than the BIO process and the OBMC process, the other processes may also be skipped according to the determination result of whether the LIC process is applicable. Too. For example, the process related to generating the predicted image is a process of correcting the predicted image so that the predicted image approaches the original image.

又，上述之處理流程是一例，亦可以將上述之處理之一部分予以去除，亦可以將未顯示在上述之處理予以追加。例如，幀間預測部218亦可以在藉由進行通常之運動補償而生成預測圖像後，藉由適應性地進行LIC處理、或、藉由適應性地進行BIO處理及OBMC處理，而修正預測圖像。In addition, the above-mentioned processing flow is an example, and a part of the above-mentioned processing may be removed, or the processing not shown above may be added. For example, the inter prediction unit 218 may modify the prediction by adaptively performing LIC processing or adaptively performing BIO processing and OBMC processing after generating a predicted image by performing normal motion compensation. image.

又，雖然編碼裝置100及解碼裝置200具有將處理用之訊號編碼在串流、或從串流予以解碼之差異，但對編碼裝置100與解碼裝置200而言，在此說明之處理流程基本上是共通的。例如，亦可以在編碼裝置100，藉由幀間預測部126進行與在解碼裝置200藉由幀間預測部218進行之處理相同之處理。In addition, although the encoding device 100 and the decoding device 200 have differences in encoding or decoding a signal for processing, the processing flow described here is basically the same for the encoding device 100 and the decoding device 200. Are common. For example, the encoding device 100 may perform the same processing in the inter prediction unit 126 as the decoding device 200 in the inter prediction unit 218.

藉由使用圖15、圖16及圖17所說明之本具體例，可將對處理對象編碼單元適用LIC處理之情況下之處理量之增加予以抑制、在管線控制中將包含LIC處理之階段的處理時間縮短。而且，可在管線控制中削減用於開始包含LIC處理之階段之處理的等待時間。因此，即便在解碼裝置200之處理性能低的情況下，在分配給1圖片之時間內令1圖片之處理完成之可能性亦變高。By using the specific examples described in FIG. 15, FIG. 16, and FIG. 17, it is possible to suppress an increase in the processing amount in the case where LIC processing is applied to a processing target coding unit, and pipeline control will include the stage of LIC processing. Reduced processing time. In addition, in the pipeline control, the waiting time for starting the process including the LIC process can be reduced. Therefore, even in a case where the processing performance of the decoding device 200 is low, the possibility that the processing of one picture is completed within the time allocated to the one picture becomes high.

另，雖然在此是依各編碼單元而進行處理，但亦可以依各任意決定之區塊而進行處理。所以，亦可以將在此之編碼單元替換成區塊。
[幀間預測處理之第3具體例]In addition, although processing is performed for each coding unit here, processing may be performed for each arbitrarily determined block. Therefore, the coding unit here can also be replaced with a block.
[Third specific example of inter prediction processing]

本具體例可將在低延遲模式中進行LIC處理之情況下之幀間預測之處理量之增加予以抑制。在本具體例，藉由是否為低延遲模式，而將第1具體例之管線構造與第2具體例之管線構造切換使用。In this specific example, it is possible to suppress an increase in the processing amount of inter prediction when the LIC processing is performed in the low-latency mode. In this specific example, the pipeline structure of the first specific example and the pipeline structure of the second specific example are switched and used depending on whether it is a low-latency mode.

圖18是顯示本具體例之幀間預測之動作的流程圖。例如，解碼裝置200之幀間預測部218進行圖18所示之動作。又，編碼裝置100之幀間預測部126亦可以與解碼裝置200之幀間預測部218同樣地進行圖18所示之動作。基本上，圖18所示之本具體例之複數個處理(S201~S210)分別與圖17所示之第2具體例之複數個處理(S201~S210)相同。FIG. 18 is a flowchart showing the operation of inter prediction in this specific example. For example, the inter prediction unit 218 of the decoding device 200 performs the operation shown in FIG. 18. The inter prediction unit 126 of the encoding device 100 may perform the operation shown in FIG. 18 in the same manner as the inter prediction unit 218 of the decoding device 200. Basically, the plural processes (S201 to S210) of the specific example shown in FIG. 18 are the same as the plural processes (S201 to S210) of the second specific example shown in FIG. 17, respectively.

相較於第2具體例，在本具體例中，幀間預測部218是當判定成對處理對象編碼單元適用LIC處理(S206)的情況下，判定是否以低延遲模式來處理處理對象編碼單元(S211)。Compared with the second specific example, in this specific example, when the inter prediction unit 218 determines that the LIC processing is applied to the processing target coding unit (S206), it determines whether to process the processing target coding unit in the low-latency mode. (S211).

而且，當判定成以低延遲模式來處理處理對象編碼單元的情況下(在S211為「是」)，幀間預測部218不進行身為與生成預測圖像相關之處理之BIO處理(S208)及OBMC處理(S210)。此情況下，幀間預測部218將只進行伴隨LIC處理之運動補償而生成之預測圖像當作最終之預測圖像。When it is determined that the processing target coding unit is processed in the low-latency mode (YES in S211), the inter prediction unit 218 does not perform BIO processing which is processing related to the generation of a predicted image (S208) And OBMC processing (S210). In this case, the inter prediction unit 218 regards the prediction image generated by performing only motion compensation with LIC processing as the final prediction image.

又，當判定成不以低延遲模式來處理處理對象編碼單元的情況下(在S211為「否」)，幀間預測部218判定是否進行BIO處理、及、是否進行OBMC處理(S207、S209)。而且，幀間預測部218是依循判定結果而適應性地進行BIO處理(S208)及OBMC處理(S210)，藉此生成最終之預測圖像。When it is determined that the processing target coding unit is not processed in the low-latency mode (NO in S211), the inter prediction unit 218 determines whether to perform BIO processing and whether to perform OBMC processing (S207, S209). . Then, the inter prediction unit 218 adaptively performs BIO processing (S208) and OBMC processing (S210) according to the determination result, thereby generating a final predicted image.

藉此，在低延遲模式，對處理對象編碼單元適用LIC處理之情況下之幀間預測之處理量之增加受到抑制。所以，如圖16之說明，在管線控制中，將用於開始第2階段之處理之等待時間予以抑制之可能性變高。Accordingly, in the low-latency mode, an increase in the amount of inter-frame prediction processing when the LIC processing is applied to the processing target coding unit is suppressed. Therefore, as illustrated in FIG. 16, in pipeline control, there is a high possibility that the waiting time for starting the processing of the second stage is suppressed.

另一方面，在低延遲模式以外，對處理對象編碼單元適用LIC處理之情況下之處理量增加。所以，如圖13之說明，在管線控制中，發生用於開始第2階段之處理之等待時間。然而，可能生成更高精度之預測圖像。所以，編碼效率提升之可能性變高。On the other hand, when the LIC processing is applied to a processing target coding unit other than the low-latency mode, the processing amount increases. Therefore, as illustrated in FIG. 13, in the pipeline control, a waiting time for starting the processing of the second stage occurs. However, it is possible to generate a more accurate prediction image. Therefore, the possibility of improving coding efficiency becomes higher.

編碼裝置100之熵編碼部110亦可以將表示是否以低延遲模式來處理編碼單元之資訊生成，並編碼在串流。而且，解碼裝置200之熵解碼部202亦可以從串流解碼而取得表示是否以低延遲模式來處理編碼單元之資訊。藉此，編碼裝置100及解碼裝置200能夠以獲得相同之判定結果的方式，判定是否以低延遲模式來處理編碼單元。The entropy encoding unit 110 of the encoding device 100 may also generate information indicating whether to process the encoding unit in a low-latency mode and encode the information in a stream. Further, the entropy decoding unit 202 of the decoding device 200 may also obtain information indicating whether the coding unit is processed in the low-delay mode from the stream decoding. Thereby, the encoding device 100 and the decoding device 200 can determine whether to process the encoding unit in the low-latency mode in a manner to obtain the same determination result.

例如，表示是否以低延遲模式來處理編碼單元之資訊是描述在編碼串流之序列標頭區域、圖片標頭區域、片段標頭區域及補助資訊區域之中至少其中1者。此情況下，編碼裝置100是將表示是否以低延遲模式來處理編碼單元之資訊編碼在該等之至少其中1者。又，解碼裝置200是從該等之至少其中1者解碼出表示是否以低延遲模式來處理編碼單元之資訊。For example, information indicating whether a coding unit is processed in a low-latency mode is described in at least one of a sequence header region, a picture header region, a fragment header region, and a supplementary information region of a coded stream. In this case, the encoding device 100 encodes information indicating whether to process the encoding unit in a low-latency mode into at least one of these. In addition, the decoding device 200 decodes information indicating whether to process the coding unit in a low-latency mode from at least one of these.

在此，序列標頭區域是儲存對動態圖像共通之參數的區域。序列標頭區域例如可以是SPS(序列參數組)。又，圖片標頭區域是儲存對構成動態圖像之圖片共通之參數的區域。圖片標頭區域例如可以是PPS(圖片參數組)。又，片段標頭區域是儲存對構成圖片之片段共通之參數的區域。Here, the sequence header area is an area storing parameters common to a moving image. The sequence header area may be, for example, an SPS (Sequence Parameter Set). The picture header area is an area storing parameters common to pictures constituting a moving picture. The picture header area may be, for example, a PPS (Picture Parameter Set). The slice header area is an area storing parameters common to the fragments constituting the picture.

又，補助資訊區域是將有別於儲存在序列標頭區域、圖片標頭區域及片段標頭區域之各參數的參數予以儲存的區域。補助資訊區域例如可以是SEI(Supplemental Enhancement Information，補充增強資訊)、圖塊標頭區域。圖塊標頭區域是儲存對構成圖片之圖塊共通之參數的區域。In addition, the assistance information area is an area in which parameters different from each parameter stored in the sequence header area, the picture header area, and the fragment header area are stored. The supplementary information area may be, for example, SEI (Supplemental Enhancement Information, supplemental enhancement information), or a tile header area. The tile header area is an area storing parameters common to the tiles constituting a picture.

又，例如，編碼裝置100亦可以是依循處理對象圖片之尺寸，切換是否以低延遲模式來處理處理對象圖片內之編碼單元。與編碼裝置100同樣，解碼裝置200也亦可以是依循處理對象圖片之尺寸，切換是否以低延遲模式來處理處理對象圖片內之編碼單元。例如，圖片之尺寸是對應於圖片之像素數。In addition, for example, the encoding device 100 may switch whether to process the coding unit in the processing target picture in a low-latency mode according to the size of the processing target picture. Similar to the encoding device 100, the decoding device 200 may also switch whether to process the coding unit in the processing target picture in a low-latency mode according to the size of the processing target picture. For example, the size of a picture is the number of pixels corresponding to the picture.

具體而言，例如，當處理對象圖片之尺寸小的情況下，處理對象圖片內之編碼單元之數量少，處理時間有餘裕。因此，此情況下，編碼裝置100不將低延遲模式適用。另一方面，當處理對象圖片之尺寸大的情況下，處理對象圖片內之編碼單元之數量多，處理時間沒有餘裕。因此，此情況下，編碼裝置100將低延遲模式適用。Specifically, for example, when the size of the picture to be processed is small, the number of coding units in the picture to be processed is small, and there is a margin for processing time. Therefore, in this case, the encoding device 100 does not apply the low-latency mode. On the other hand, when the size of the processing target picture is large, the number of coding units in the processing target picture is large, and there is no room for processing time. Therefore, in this case, the encoding device 100 applies the low-latency mode.

在此，亦可以藉由預定之基準而判定處理對象圖片之尺寸是否大。例如，亦可以藉由處理對象圖片之尺寸是否比閾值大，而判定處理對象圖片之尺寸是否大。Here, it is also possible to determine whether the size of the processing target picture is large based on a predetermined reference. For example, it is also possible to determine whether the size of the processing target picture is large based on whether the size of the processing target picture is larger than a threshold.

又，例如，編碼裝置100亦可以是依循將編碼串流接收而解碼動態圖像之解碼裝置200的處理能力，切換是否以低延遲模式來處理編碼單元。又，解碼裝置200也亦可以是依循解碼裝置200的處理能力，切換是否以低延遲模式來處理編碼單元。In addition, for example, the encoding device 100 may switch the processing of the encoding unit in a low-latency mode according to the processing capability of the decoding device 200 that receives the encoded stream and decodes the moving image. In addition, the decoding device 200 may switch whether to process the coding unit in the low-latency mode according to the processing capability of the decoding device 200.

具體而言，例如，當解碼裝置200之處理能力高的情況下，可在一定之處理時間內進行大量之處理。因此，此情況下，編碼裝置100不將低延遲模式適用。另一方面，當解碼裝置200之處理能力低的情況下，要在一定之處理時間內進行大量之處理會有困難。因此，此情況下，編碼裝置100將低延遲模式適用。Specifically, for example, when the processing capability of the decoding device 200 is high, a large amount of processing can be performed within a certain processing time. Therefore, in this case, the encoding device 100 does not apply the low-latency mode. On the other hand, when the processing capability of the decoding device 200 is low, it may be difficult to perform a large amount of processing within a certain processing time. Therefore, in this case, the encoding device 100 applies the low-latency mode.

在此，亦可以依循預定之基準來判定解碼裝置200的處理能力是否高。例如，亦可以藉由表示解碼裝置200之處理能力的數值是否比閾值大，而判定解碼裝置200的處理能力是否高。Here, it can also be determined whether the processing capability of the decoding device 200 is high according to a predetermined reference. For example, it is also possible to determine whether the processing capability of the decoding apparatus 200 is high by determining whether the value of the processing capability of the decoding apparatus 200 is greater than a threshold.

又，例如，解碼裝置200亦可以從解碼裝置200之各構成要素等取得解碼裝置200的處理能力。而且，編碼裝置100亦可以藉由事先進行之資訊交換，而從解碼裝置200取得解碼裝置200的處理能力。或者，當編碼裝置100及解碼裝置200一體化的情況下，編碼裝置100及解碼裝置200亦可以不進行資訊交換而判定各者的處理能力。In addition, for example, the decoding device 200 may obtain the processing capability of the decoding device 200 from each component of the decoding device 200 and the like. In addition, the encoding device 100 can also obtain the processing capability of the decoding device 200 from the decoding device 200 by exchanging information in advance. Alternatively, when the encoding device 100 and the decoding device 200 are integrated, the encoding device 100 and the decoding device 200 may determine the processing capabilities of each without exchanging information.

又，例如，編碼裝置100亦可以是依循分配給編碼串流之設定檔或等級資訊，切換是否以低延遲模式來處理編碼單元。又，解碼裝置200也亦可以是依循分配給編碼串流之設定檔或等級資訊，切換是否以低延遲模式來處理編碼單元。In addition, for example, the encoding device 100 may switch whether to process the encoding unit in a low-latency mode according to the profile or level information allocated to the encoding stream. In addition, the decoding device 200 may also switch whether to process the coding unit in a low-latency mode according to the profile or level information allocated to the coding stream.

上述之設定檔是表示對圖像之串流決定之技術要件。技術要件是對應於功能之集合。更具體而言，技術要件是對應於色彩空間之形式、色彩深度之尺寸、及、可否利用熵片段等。又，上述之等級是將對圖像之串流決定之參數要件以數值表示。參數要件是對應於處理之負荷、及、使用記憶體量等。更具體而言，參數要件是對應於最大位元率、最大取樣率、及、最大圖片尺寸等。The above-mentioned profile is a technical element indicating the streaming decision of the image. Technical requirements are a collection of functions. More specifically, the technical requirements are the form corresponding to the color space, the size of the color depth, and whether entropy fragments can be used. In addition, the above-mentioned levels are expressed by numerical values of parameter requirements determined on the stream of the image. The parameter requirements correspond to the processing load and the amount of memory used. More specifically, the parameter requirements correspond to a maximum bit rate, a maximum sampling rate, and a maximum picture size.

具體而言，例如，編碼裝置100對於被設想成解碼動態圖像之解碼裝置200具有充分之處理能力的設定檔及等級，不適用低延遲模式。另一方面，編碼裝置100對於被設想成解碼裝置200不具有充分之處理能力的設定檔及等級，適用低延遲模式。亦可以事先針對各設定檔或各等級而決定是否適用低延遲模式。Specifically, for example, the encoding device 100 does not apply the low-latency mode to a profile and a level having sufficient processing power for the decoding device 200 which is supposed to decode a moving image. On the other hand, the encoding device 100 applies a low-latency mode to a profile and a level that is assumed to be insufficient for the decoding device 200 to have a processing capability. You can also decide whether to apply the low-latency mode for each profile or level in advance.

例如，亦可以藉由將表示被分配給編碼串流之設定檔或等級之資訊予以編碼及解碼，而將表示是否適用低延遲模式之資訊予以編碼及解碼。又，亦可以是如下：關於預定之設定檔或預定之等級，依各預定之處理單位適應性地決定是否適用低延遲模式，且依各預定之處理單位將表示是否適用低延遲模式之資訊予以編碼及解碼。For example, it is also possible to encode and decode information indicating whether a low-latency mode is applicable by encoding and decoding information indicating a profile or level assigned to an encoding stream. In addition, it may be as follows: regarding the predetermined profile or the predetermined level, the low-latency mode is adaptively determined according to each predetermined processing unit, and the information indicating whether the low-latency mode is applicable is given according to each predetermined processing unit. Encoding and decoding.

又，在上述，基本上是依循解碼裝置200的處理能力來決定是否適用低延遲模式。亦可以不是依循解碼裝置200的處理能力而是依循編碼裝置100的處理能力、或者、除了解碼裝置200的處理能力之外還依循編碼裝置100的處理能力，來決定是否適用低延遲模式。與解碼裝置200的處理能力同樣，編碼裝置100的處理能力可在編碼裝置100及解碼裝置200取得。In addition, as described above, it is basically determined whether to apply the low-latency mode according to the processing capability of the decoding device 200. It is also possible to determine whether or not to apply the low-latency mode according to the processing capability of the encoding apparatus 100 instead of the processing capability of the decoding apparatus 200, or in addition to the processing capability of the decoding apparatus 200. Similar to the processing capability of the decoding device 200, the processing capability of the encoding device 100 can be obtained at the encoding device 100 and the decoding device 200.

例如，亦可以依循編碼裝置100的處理能力與解碼裝置200的處理能力中之較低的處理能力，來決定是否適用低延遲模式。For example, it is also possible to determine whether to apply the low-latency mode according to the lower processing capacity of the processing capacity of the encoding device 100 and the processing capacity of the decoding device 200.

又，編碼裝置100及解碼裝置200亦可以是依循圖片的尺寸、編碼裝置100的處理能力、解碼裝置200的處理能力、設定檔及等級中之1種以上之組合，來決定是否適用低延遲模式。例如，編碼裝置100及解碼裝置200亦可以是當該等組合滿足預定之基準的情況下決定為適用低延遲模式，當該等組合不滿足預定之基準的情況下決定為不適用低延遲模式。In addition, the encoding device 100 and the decoding device 200 may determine whether to apply the low-latency mode according to one or more combinations of the size of the picture, the processing capacity of the encoding device 100, the processing capacity of the decoding device 200, the profile, and the level. . For example, the encoding device 100 and the decoding device 200 may decide to apply the low-latency mode when the combinations satisfy a predetermined reference, and determine that the low-latency mode is not applicable when the combinations do not satisfy the predetermined reference.

在編碼裝置100，可以是由幀間預測部126決定是否適用低延遲模式，亦可以是由其他之構成要素決定是否適用低延遲模式。同樣地，在解碼裝置200，可以是由幀間預測部218決定是否適用低延遲模式，亦可以是由其他之構成要素決定是否適用低延遲模式。In the encoding device 100, the inter prediction unit 126 may decide whether or not to apply the low-latency mode, or other constituent elements may determine whether or not to apply the low-latency mode. Similarly, in the decoding device 200, the inter prediction unit 218 may determine whether to apply the low-latency mode, or other components may determine whether to apply the low-latency mode.

又，表示是否以低延遲模式來處理編碼單元之資訊亦可以非嚴謹地作為表示是否以低延遲模式來處理編碼單元之資訊的訊號而編碼在編碼串流。亦即，表示是否以低延遲模式來處理編碼單元之資訊亦可以作為帶有其他意義之訊號而編碼在編碼串流。In addition, the information indicating whether the coding unit is processed in the low-delay mode may be non-rigorously encoded in the coded stream as a signal indicating whether the information in the coding unit is processed in the low-delay mode. That is, the information indicating whether the coding unit is processed in the low-latency mode can also be encoded in the encoded stream as a signal with other meanings.

例如，亦可以是有編碼如下之2個訊號：表示對適用LIC處理之編碼單元強制禁止適用BIO處理的訊號、及表示對適用LIC處理之編碼單元強制禁止適用OBMC處理的訊號。又，作為別的例，表示是否以低延遲模式來處理編碼單元之資訊亦可以直接與設定檔或等級構成對應關係，藉此，只依循表示設定檔或等級的訊號來判定是否以低延遲模式來處理編碼單元。For example, there may be two signals encoded as follows: a signal indicating that the BIO process is forbidden to be applied to the coding unit to which LIC processing is applied, and a signal indicating that the OBMC process is to be forbidden to be coded unit that is applicable to LIC. Also, as another example, the information indicating whether to process the coding unit in a low-latency mode can also directly correspond to a profile or level, whereby only the signal indicating the profile or level is used to determine whether to use the low-latency mode. To process the coding unit.

又，在圖18中，當在低延遲模式對處理對象單元適用LIC處理的情況下，幀間預測部218不進行BIO處理與OBMC處理之雙方而生成預測圖像。然而，此情況下，幀間預測部218亦可以是只不進行BIO處理而生成預測圖像，亦可以是只不進行OBMC處理而生成預測圖像。藉此，雖然處理量及處理時間之削減效果降低，但可能生成精度較高之預測圖像。所以，編碼效率提升之可能性變高。In FIG. 18, when the LIC process is applied to the processing target unit in the low-latency mode, the inter prediction unit 218 does not perform both the BIO process and the OBMC process, and generates a predicted image. However, in this case, the inter prediction unit 218 may generate a prediction image without performing BIO processing only, or may generate a prediction image without performing OBMC processing only. Thereby, although the reduction effect of the processing amount and processing time is reduced, a highly accurate prediction image may be generated. Therefore, the possibility of improving coding efficiency becomes higher.

又，在判定是否適用LIC處理後，後續有BIO處理及OBMC處理以外之與生成預測圖像相關之別的處理的情況下，幀間預測部218亦可依循是否適用LIC處理之判定結果而將此別的處理也跳過。In addition, after determining whether the LIC process is applicable, if there are subsequent processes other than the BIO process and the OBMC process that are related to the generation of the predicted image, the inter prediction unit 218 may follow the determination result of whether the LIC process is applicable. This other processing is also skipped.

又，雖然編碼裝置100及解碼裝置200具有將處理用之訊號編碼在串流、或從串流予以解碼之差異，但對編碼裝置100與解碼裝置200而言，在此說明之處理流程基本上是共通的。例如，亦可以在編碼裝置100中，藉由幀間預測部126進行與在解碼裝置200藉由幀間預測部218進行之處理相同之處理。In addition, although the encoding device 100 and the decoding device 200 have differences in encoding or decoding a signal for processing, the processing flow described here is basically the same for the encoding device 100 and the decoding device 200. Are common. For example, the encoding device 100 may perform the same processing in the inter prediction unit 126 as the decoding device 200 in the inter prediction unit 218.

又，雖然在此是依各編碼單元而進行處理，但亦可以依各任意決定之區塊而進行處理。所以，亦可以將在此之編碼單元替換成區塊。In addition, although processing is performed for each coding unit here, processing may be performed for each arbitrarily determined block. Therefore, the coding unit here can also be replaced with a block.

藉由使用圖18所說明之本具體例，可將在低延遲模式對處理對象編碼單元適用LIC處理之情況下之處理量之增加予以抑制、在管線控制中將包含LIC處理之階段的處理時間縮短。另一方面，雖然在低延遲模式以外對處理對象編碼單元適用LIC處理之情況下之處理量未被削減，但可提高編碼效率。By using this specific example illustrated in FIG. 18, it is possible to suppress an increase in the processing amount when the LIC processing is applied to the processing target coding unit in the low-latency mode, and to include the processing time of the LIC processing stage in the pipeline control. shorten. On the other hand, although the processing amount is not reduced when the LIC processing is applied to the processing target coding unit other than the low-latency mode, coding efficiency can be improved.

藉此，可因應編碼裝置100及解碼裝置200的處理能力或編碼條件而選擇最合適之處理方法。所以，對廣泛用途之可利用性變高。
[幀間預測處理之變形例]Thereby, the most suitable processing method can be selected according to the processing capabilities or encoding conditions of the encoding device 100 and the decoding device 200. Therefore, the availability for a wide range of uses becomes high.
[Modification of inter prediction processing]

亦可以在上述各具體例之幀間預測中，進行下述之複數個處理之任一者。In the inter prediction of each of the above specific examples, any one of the plurality of processes described below may be performed.

(處理1)亦可以是當對尺寸比特定值大之處理對象編碼單元進行LIC處理的情況下，不進行伴隨預測圖像的生成而來之處理之一部份，而進行預測圖像的生成。藉此，可將處理時間變長之情形予以抑制。(Process 1) When LIC processing is performed on a processing target coding unit having a size larger than a specific value, a part of the processing accompanying the generation of the predicted image may not be performed, and the predicted image may be generated. . This can suppress a situation in which the processing time becomes longer.

(處理2)亦可以是當對尺寸比前一個進行編碼之編碼單元大之處理對象編碼單元進行LIC處理的情況下，不進行伴隨預測圖像的生成而來之處理之一部分，而進行預測圖像的生成。藉此，可將等待時間變長之情形予以抑制。(Process 2) When LIC processing is performed on a processing target coding unit whose size is larger than that of the previous coding unit, the prediction map may not be performed as part of the processing accompanying the generation of the predicted image. Image generation. This can suppress a situation in which the waiting time becomes longer.

(處理3)亦可以是當在一定期間內進行了LIC處理之編碼單元的個數比特定值大的情況下對處理對象編碼單元進行LIC處理時，不進行伴隨預測圖像的生成而來之處理之一部份，而進行預測圖像的生成。藉此，可將等待時間被累積之情形予以抑制。(Process 3) When the number of coding units that have been subjected to LIC processing within a certain period is greater than a specific value, the LIC processing may be performed on the processing target coding unit without generating a prediction image. Part of the processing is to generate a predicted image. Thereby, a situation where the waiting time is accumulated can be suppressed.

(處理4)亦可以是當在一定期間內進行了LIC處理之編碼單元的比例比特定值大的情況下對處理對象編碼單元進行LIC處理時，不進行伴隨預測圖像的生成而來之處理之一部份，而進行預測圖像的生成。藉此，可將等待時間被累積之情形予以抑制。(Process 4) When the proportion of coding units that have been subjected to LIC processing within a certain period is larger than a specific value, the processing of the coding unit to be processed may be performed without performing processing accompanying the generation of the predicted image. Part of the prediction image generation. Thereby, a situation where the waiting time is accumulated can be suppressed.

(處理5)上述之一定期間可以是處理對象編碼單元之前面一定數量之編碼樹單元(CTU)的處理期間。藉此，依循一定數量之編碼樹單元之處理的結果等而抑制處理延遲之增加。(Process 5) The above-mentioned certain period may be a process period in which a certain number of coding tree units (CTUs) precede the processing target coding unit. Thereby, an increase in processing delay is suppressed in accordance with the processing result of a certain number of coding tree units and the like.

(處理6)上述之一定期間可以是處理對象編碼單元所屬之片段整體或圖片整體的處理期間。藉此，依循片段或圖片之處理的結果等而抑制處理延遲之增加。(Process 6) The above-mentioned certain period may be a processing period of the entire fragment or the picture to which the processing target coding unit belongs. Thereby, an increase in processing delay is suppressed in accordance with a result of processing of a segment or a picture, and the like.

(處理7)亦可以是當對處理對象編碼單元進行LIC處理時，當在此之前為止連續進行了LIC處理之複數個編碼單元的個數比特定值大的情況下，不進行伴隨預測圖像的生成而來之處理之一部份，而進行預測圖像的生成。藉此，可將等待時間被累積之情形予以抑制。(Process 7) When LIC processing is performed on the coding unit to be processed, if the number of coding units that have been continuously subjected to LIC processing before that is larger than a specific value, the accompanying prediction image may not be performed. As part of the processing, the prediction image is generated. Thereby, a situation where the waiting time is accumulated can be suppressed.

(處理8)編碼裝置100可以進行將管線控制之等待時間予以推定之模擬處理。而且，當對處理對象編碼單元進行LIC處理時，當在其處理開始時間點推定之等待時間比特定值大的情況下，不進行伴隨預測圖像的生成而來之處理之一部份，而進行預測圖像的生成。藉此，可將等待時間變長之情形予以抑制。
[LIC處理](Process 8) The encoding device 100 may perform simulation processing for estimating the waiting time of the pipeline control. In addition, when performing LIC processing on the processing target coding unit, if the estimated waiting time at the processing start time point is greater than a specific value, a part of the processing accompanying the generation of the predicted image is not performed, and The prediction image is generated. This can suppress a situation in which the waiting time becomes longer.
[LIC processing]

以下，針對LIC處理更具體地進行說明。LIC處理是使用處理對象區塊之周邊參考區域之亮度像素值、及參考區塊之周邊參考區域之亮度像素值，來導出亮度修正參數，並依循亮度修正參數來生成或修正預測圖像(參考圖9D)。Hereinafter, the LIC processing will be described more specifically. LIC processing is to use the brightness pixel values of the surrounding reference area of the processing target block and the brightness pixel values of the surrounding reference area of the reference block to derive brightness correction parameters, and generate or modify the predicted image according to the brightness correction parameters (reference Figure 9D).

具體而言，在LIC處理之前，解碼裝置200之幀間預測部218是對於處理對象編碼單元，取得用於特定出參考圖片內之參考區塊的運動向量。在此，參考圖片是已處理之圖片。然後，幀間預測部218取得藉由此運動向量而指出之參考區塊之圖像即參考圖像。然後，幀間預測部218取得處理對象編碼單元之周邊參考區域之亮度像素值、及、參考區塊之周邊參考區域之亮度像素值。Specifically, before the LIC processing, the inter prediction unit 218 of the decoding device 200 obtains a motion vector for specifying a reference block in a reference picture for a processing target coding unit. Here, the reference picture is a processed picture. Then, the inter prediction unit 218 obtains a reference image, which is an image of a reference block pointed out by this motion vector. Then, the inter prediction unit 218 obtains the luminance pixel values of the surrounding reference area of the processing target coding unit and the luminance pixel values of the surrounding reference area of the reference block.

在此，處理對象編碼單元之周邊參考區域是被決定在判定成可參考之1個以上之已處理周邊編碼單元內。參考區塊之周邊參考區域是以如下方式而決定：參考區塊之周邊參考區域之與參考區塊的相對位置，與處理對象編碼單元之周邊參考區域之與處理對象編碼單元的相對位置相同的方式。Here, the surrounding reference area of the processing target coding unit is determined to be in one or more processed surrounding coding units determined to be referenceable. The surrounding reference area of the reference block is determined as follows: The relative position of the surrounding reference area of the reference block to the reference block is the same as the relative position of the surrounding reference area of the processing target coding unit and the relative position of the processing target coding unit. the way.

具體而言，例如，當處理對象編碼單元之周邊參考區域是決定在對處理對象編碼單元而言位於左側之位置的情況下，參考區塊之周邊參考區域是決定在對參考區塊而言位於左側之位置。又，例如，當處理對象編碼單元之周邊參考區域是決定在對處理對象編碼單元而言位於上側之位置的情況下，參考區塊之周邊參考區域是決定在對參考區塊而言位於上側之位置。Specifically, for example, when the surrounding reference area of the processing target coding unit is determined to be located on the left side for the processing target coding unit, the surrounding reference area of the reference block is determined to be located on the reference block. Position on the left. Also, for example, when the surrounding reference area of the processing target coding unit is determined to be located on the upper side for the processing target coding unit, the surrounding reference area of the reference block is determined to be on the upper side for the reference block. position.

處理對象編碼單元之周邊參考區域可以是被判定成可參考之1個以上之已處理周邊編碼單元內之一部分之區域。例如，處理對象編碼單元之周邊參考區域可以是被判定成可參考之1個以上之已處理周邊編碼單元所含有之區域中，與處理對象編碼單元鄰接之像素之區域。The surrounding reference area of the processing target coding unit may be an area of a part of the processed surrounding coding unit that is determined to be referenced. For example, the surrounding reference area of the processing target coding unit may be a region of pixels adjacent to the processing target coding unit among the areas included in the processed surrounding coding unit that are determined to be referenced.

接著，幀間預測部218是使用處理對象編碼單元之周邊參考區域之亮度像素值、及、參考區塊之周邊參考區域之亮度像素值，來導出亮度修正參數。例如，幀間預測部218是使用該等亮度像素值，將表示亮度值在包含處理對象編碼單元之處理對象圖片、及、包含參考區塊之參考圖片之間如何變化之資訊擷取，並依循此資訊而導出亮度修正參數。Next, the inter prediction unit 218 derives the brightness correction parameter by using the brightness pixel values of the surrounding reference area of the processing target coding unit and the brightness pixel values of the surrounding reference area of the reference block. For example, the inter prediction unit 218 uses the luminance pixel values to extract information indicating how the luminance value changes between a processing target picture including a processing target coding unit and a reference picture including a reference block, and follows This information is used to derive brightness correction parameters.

亮度修正參數例如是用於亮度像素值之線性轉換的參數。具體而言，亮度修正參數可以是用於以 α×亮度像素值+β 來進行亮度像素值之線性轉換的α及β。又，幀間預測部218是以當對參考區塊之周邊參考區域之亮度像素值進行線性轉換時可獲得與處理對象編碼單元之周邊參考區域之亮度像素值接近之結果的方式，將亮度修正參數導出。The luminance correction parameter is, for example, a parameter for linear conversion of luminance pixel values. Specifically, the brightness correction parameter may be α and β for linearly converting the brightness pixel value with α × luminance pixel value + β. In addition, the inter prediction unit 218 corrects the luminance in such a manner that when the luminance pixel values in the peripheral reference area of the reference block are linearly converted to obtain results close to the luminance pixel values in the peripheral reference area of the processing target coding unit Parameter export.

然後，幀間預測部218是使用亮度修正參數來生成預測圖像。例如，幀間預測部218是使用亮度修正參數，對藉由處理對象編碼單元之運動向量而特定出之參考圖像進行亮度修正處理，藉此由參考圖像生成處理對象編碼單元之預測圖像。換句話說，幀間預測部218是使用亮度修正參數來修正由參考圖像生成之預測圖像。Then, the inter prediction unit 218 generates a predicted image using a luminance correction parameter. For example, the inter prediction unit 218 performs luminance correction processing on a reference image specified by a motion vector of a processing target coding unit using a brightness correction parameter, thereby generating a predicted image of the processing target coding unit from the reference image. . In other words, the inter prediction unit 218 uses a brightness correction parameter to correct a predicted image generated from a reference image.

幀間預測部218是依各編碼單元而反覆進行上述之處理。另，幀間預測部218亦可以不是依各編碼單元、而是依各藉由將編碼單元進一步分割而獲得之子區塊，進行與上述同樣之處理。The inter prediction unit 218 performs the above-mentioned processing repeatedly for each coding unit. In addition, the inter prediction unit 218 may perform the same processing as described above, instead of depending on each coding unit, but on each subblock obtained by further dividing the coding unit.

又，雖然在上述之說明中是由解碼裝置200之幀間預測部218進行LIC處理，但編碼裝置100之幀間預測部126亦與解碼裝置200之幀間預測部218同樣地進行LIC處理。又，在解碼裝置200，亦可以由與幀間預測部218不同之構成要素進行LIC處理。同樣地，在編碼裝置100，亦可以由與幀間預測部126不同之構成要素進行LIC處理。In the above description, the inter prediction unit 218 of the decoding device 200 performs LIC processing, but the inter prediction unit 126 of the encoding device 100 also performs LIC processing in the same manner as the inter prediction unit 218 of the decoding device 200. In addition, in the decoding device 200, LIC processing may be performed using components different from those of the inter prediction unit 218. Similarly, in the encoding device 100, the LIC processing may be performed using components different from those of the inter prediction unit 126.

如上述，處理對象編碼單元之周邊參考區域例如是以與處理對象編碼單元之左側或上側鄰接之像素之區域來構成。依循如此之處理對象編碼單元之周邊參考區域之亮度像素值、及參考區塊之周邊參考區域之亮度像素值，而導出亮度修正參數。然後，使用亮度修正參數來生成或修正預測圖像。As described above, the surrounding reference area of the processing target coding unit is, for example, a region of pixels adjacent to the left or upper side of the processing target coding unit. The brightness correction parameters are derived by following the brightness pixel values of the surrounding reference area of the processing target coding unit and the brightness pixel values of the surrounding reference area of the reference block. Then, a brightness correction parameter is used to generate or modify the predicted image.

另，周邊參考區域之形狀並不限定於圖9D之例，亦可以是其他之形狀。例如，周邊參考區域亦可以含有不與處理對象編碼單元直接鄰接之像素之區域。The shape of the peripheral reference area is not limited to the example shown in FIG. 9D, and may be other shapes. For example, the peripheral reference region may include a region of pixels that are not directly adjacent to the processing target coding unit.

在LIC處理，導出適切之亮度修正參數被設想成會花費長時間。在上述之複數例，當進行LIC處理的情況下不進行其他一部分之處理，因此，可抑制包含LIC處理之階段的處理時間變得過長之情形。
[安裝例]In LIC processing, deriving appropriate brightness correction parameters is envisaged to take a long time. In the plural examples described above, when LIC processing is performed, other processing is not performed. Therefore, it is possible to suppress a situation in which the processing time at the stage including the LIC processing becomes excessively long.
[Example of installation]

圖19是顯示編碼裝置100之安裝例的方塊圖。編碼裝置100具備電路160及記憶體162。例如，在圖1顯示之編碼裝置100之複數個構成要素是藉由在圖19顯示之電路160及記憶體162而安裝。FIG. 19 is a block diagram showing an installation example of the encoding device 100. The encoding device 100 includes a circuit 160 and a memory 162. For example, a plurality of constituent elements of the encoding device 100 shown in FIG. 1 are installed by the circuit 160 and the memory 162 shown in FIG. 19.

電路160是可對記憶體162存取之電子電路，且進行資訊處理。例如，電路160是使用記憶體162來編碼動態圖像之專用或通用之電子電路。電路160亦可以是如CPU般之處理器。又，電路160亦可以是複數個電子電路之集合體。The circuit 160 is an electronic circuit that can access the memory 162 and performs information processing. For example, the circuit 160 is a dedicated or general-purpose electronic circuit that uses the memory 162 to encode a moving image. The circuit 160 may also be a processor such as a CPU. In addition, the circuit 160 may be an aggregate of a plurality of electronic circuits.

又，例如，電路160亦可以發揮在圖1顯示之編碼裝置100之複數個構成要素中，除了用於記憶資訊的構成要素以外之複數個構成要素的功能。亦即，電路160亦可以進行上述之動作來作為該等構成要素之動作。In addition, for example, the circuit 160 can also function as a plurality of constituent elements other than the constituent elements for storing information among the plurality of constituent elements of the encoding device 100 shown in FIG. 1. That is, the circuit 160 may perform the operations described above as the operations of the constituent elements.

記憶體162是將電路160用來編碼動態圖像之資訊予以記憶之專用或通用之記憶體。記憶體162亦可以是電子電路，亦可以是與電路160連接，亦可以是包含於電路160。The memory 162 is a dedicated or general-purpose memory that stores information used by the circuit 160 to encode moving images. The memory 162 may also be an electronic circuit, may be connected to the circuit 160, or may be included in the circuit 160.

又，記憶體162亦可以是複數個電子電路之集合體，亦可以是以複數個副記憶體來構成。又，記憶體162亦可以是磁碟片或光碟片等，亦可以表現為儲存器或記錄媒體等。又，記憶體162亦可以是非依電性記憶體，亦可以是依電性記憶體。In addition, the memory 162 may be an aggregate of a plurality of electronic circuits, or may be constituted by a plurality of auxiliary memories. In addition, the memory 162 may be a magnetic disk, an optical disk, or the like, and may also be expressed as a memory or a recording medium. In addition, the memory 162 may be a non-electrically-dependent memory or an electrically-dependent memory.

例如，記憶體162亦可以發揮在圖1顯示之編碼裝置100之複數個構成要素中，用於記憶資訊之構成要素的功能。具體而言，記憶體162亦可以發揮在圖1顯示之區塊記憶體118及幀記憶體122的功能。For example, the memory 162 may also play a function of storing the constituent elements of the information among the constituent elements of the encoding device 100 shown in FIG. 1. Specifically, the memory 162 can also function as the block memory 118 and the frame memory 122 shown in FIG. 1.

又，記憶體162亦可以記憶被編碼之動態圖像，亦可以記憶與已編碼之動態圖像對應之位元串。又，記憶體162亦可以記憶電路160用來編碼動態圖像之程式。In addition, the memory 162 can also store the encoded dynamic image, and can also store the bit string corresponding to the encoded dynamic image. In addition, the memory 162 may also store a program that the circuit 160 uses to encode a moving image.

另，編碼裝置100亦可以不安裝在圖1顯示之複數個構成要素之全部，亦可以不進行上述之複數個處理之全部。亦可以令在圖1顯示之複數個構成要素之一部分包含於其他裝置，亦可以令上述之複數個處理之一部分藉由其他裝置執行。而且，藉由在編碼裝置100安裝圖1顯示之複數個構成要素中之一部分、進行上述之複數個處理之一部分，可將處理之延遲予以抑制。In addition, the encoding device 100 may not be installed on all of the plurality of constituent elements shown in FIG. 1, or may not be performed on all of the plurality of processes described above. A part of the plurality of constituent elements shown in FIG. 1 may also be included in other devices, and a part of the plurality of processes described above may also be performed by other devices. Furthermore, by installing one of the plurality of constituent elements shown in FIG. 1 in the encoding device 100 and performing one of the plurality of processes described above, the delay in processing can be suppressed.

圖20是顯示圖19所示之編碼裝置100之動作例的流程圖。例如，編碼裝置100之電路160使用記憶體162來進行圖20所示之動作，藉此，編碼裝置100在構成動態圖像之編碼對象區塊之編碼中進行幀間預測。FIG. 20 is a flowchart showing an operation example of the encoding device 100 shown in FIG. 19. For example, the circuit 160 of the encoding device 100 uses the memory 162 to perform the operation shown in FIG. 20, whereby the encoding device 100 performs inter prediction during encoding of a coding target block constituting a moving image.

具體而言，電路160是當進行第1處理的情況下，不進行包含第2處理之1個以上之處理而進行編碼對象區塊之幀間預測(S301)。在此，第1處理是使用編碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正編碼對象區塊之預測圖像之處理。第2處理是使用由參考區塊獲得之亮度之空間上的梯度來修正預測圖像之處理。Specifically, in the case where the first process is performed, the circuit 160 performs inter prediction of the encoding target block without performing one or more processes including the second process (S301). Here, the first process is a process of correcting the predicted image of the coding target block by using the brightness of the periphery of the coding target block and the brightness of the periphery of the reference block. The second process is a process of correcting a predicted image using a spatial gradient of brightness obtained from a reference block.

藉此，編碼裝置100可在進行被設想成處理時間長之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行幀間預測，前述第2處理是與第1處理同樣地被設想成處理時間長。所以，編碼裝置100可縮短包含第1處理之階段的處理時間，可在動態圖像之編碼中將處理之延遲予以抑制。As a result, the encoding device 100 can perform inter prediction without performing one or more processes including the second process when the first process is supposed to take a long processing time. The second process is the same as the first process. It is also envisaged that the processing time is long. Therefore, the encoding device 100 can shorten the processing time including the first processing stage, and can suppress the processing delay in the encoding of the moving image.

例如，上述1個以上之處理亦可以包含第3處理，前述第3處理是使用藉由編碼對象區塊之周邊之已編碼區塊之運動向量而從編碼對象區塊指出之參考圖像，來修正預測圖像。藉此，編碼裝置100可在進行第1處理的情況下，不進行第2處理及第3處理而進行幀間預測。所以，編碼裝置100可縮短包含第1處理之階段的處理時間，可在動態圖像之編碼將處理之延遲予以抑制。For example, the above one or more processes may include a third process. The third process is to use a reference image pointed out from the coding target block by using the motion vectors of the coded blocks around the coding target block. Correct the predicted image. This allows the encoding device 100 to perform inter prediction without performing the second processing and the third processing when the first processing is performed. Therefore, the encoding device 100 can shorten the processing time including the first processing stage, and can suppress the processing delay in the encoding of the moving image.

又，例如，上述1個以上之處理亦可以是修正預測圖像之複數個處理中之與第1處理不同之所有處理。藉此，編碼裝置100可在進行第1處理的情況下，不進行第1處理以外之修正處理而進行幀間預測。所以，編碼裝置100可縮短包含第1處理之階段的處理時間，可在動態圖像之編碼將處理之延遲予以抑制。In addition, for example, the one or more processes described above may be all processes different from the first process among a plurality of processes for correcting a predicted image. This allows the encoding device 100 to perform inter prediction without performing correction processing other than the first processing when the first processing is performed. Therefore, the encoding device 100 can shorten the processing time including the first processing stage, and can suppress the processing delay in the encoding of the moving image.

又，例如，電路160亦可以在進行第1處理後，以不修正預測圖像的方式而令編碼對象區塊之幀間預測結束。藉此，編碼裝置100可在進行第1處理的情況下，在第1處理之後不進行修正處理而進行幀間預測。所以，編碼裝置100可縮短包含第1處理之階段的處理時間，可在動態圖像之編碼將處理之延遲予以抑制。In addition, for example, the circuit 160 may end the inter-prediction of the encoding target block without performing correction on the prediction image after performing the first processing. This allows the encoding device 100 to perform inter prediction without performing correction processing after the first processing when performing the first processing. Therefore, the encoding device 100 can shorten the processing time including the first processing stage, and can suppress the processing delay in the encoding of the moving image.

又，例如，電路160亦可以判定是否以低延遲模式進行編碼對象區塊之編碼。而且，電路160亦可以當在判定成以低延遲模式進行編碼對象區塊之編碼的情況下進行第1處理的情況下，不進行上述1個以上之處理而進行編碼對象區塊之幀間預測。藉此，編碼裝置100可在低延遲模式縮短包含第1處理之階段的處理時間，可在動態圖像之編碼將處理之延遲予以抑制。In addition, for example, the circuit 160 may determine whether to encode the encoding target block in the low-latency mode. In addition, when the circuit 160 performs the first process when it is determined that the encoding target block is encoded in the low-latency mode, the inter-prediction of the encoding target block is not performed without performing the above one or more processes. . Thereby, the encoding device 100 can shorten the processing time including the first processing stage in the low-latency mode, and can suppress the processing delay in the encoding of the moving image.

又，例如，電路160亦可以將表示是否以低延遲模式進行編碼對象區塊之編碼之資訊，編碼在動態圖像之編碼串流之序列標頭區域、圖片標頭區域、圖塊標頭區域、片段標頭區域及補助資訊區域之至少其中1者。藉此，編碼裝置100可將是否為低延遲模式之資訊給予解碼裝置200。In addition, for example, the circuit 160 may also encode information indicating whether to encode the encoding target block in the low-latency mode, in a sequence header area, a picture header area, and a tile header area of an encoding stream of a moving image. , At least one of the fragment header area and the subsidy information area. Thereby, the encoding device 100 can give the decoding device 200 information on whether it is in a low-latency mode.

又，例如，電路160亦可以是依循包含編碼對象區塊之編碼對象圖片的大小，而判定是否以低延遲模式進行編碼對象區塊之編碼。藉此，編碼裝置100可依循圖片的大小，而適應性地控制是否以低延遲模式進行編碼。In addition, for example, the circuit 160 may determine whether to encode the encoding target block in a low-latency mode according to the size of the encoding target picture including the encoding target block. Thereby, the encoding device 100 can adaptively control whether to perform encoding in a low-latency mode according to the size of the picture.

又，例如，電路160亦可以是依循解碼裝置200之處理能力，而判定是否以低延遲模式進行編碼對象區塊之編碼。藉此，編碼裝置100可依循解碼裝置200之處理能力，而適應性地控制是否以低延遲模式進行編碼。In addition, for example, the circuit 160 may determine whether to encode the encoding target block in a low-latency mode according to the processing capability of the decoding device 200. Thereby, the encoding device 100 can adaptively control whether to perform encoding in a low-latency mode according to the processing capability of the decoding device 200.

又，例如，電路160亦可以是依循分配給動態圖像之編碼串流之設定檔或等級資訊，而判定是否以低延遲模式進行編碼對象區塊之編碼。藉此，編碼裝置100可依循設定檔或等級，而適應性地控制是否以低延遲模式進行編碼。In addition, for example, the circuit 160 may determine whether to encode the encoding target block in the low-latency mode according to the profile or level information of the encoding stream allocated to the moving image. Thereby, the encoding device 100 can adaptively control whether to perform encoding in a low-latency mode according to a set file or level.

圖21是顯示解碼裝置200之安裝例的方塊圖。解碼裝置200具備電路260及記憶體262。例如，在圖10顯示之解碼裝置200之複數個構成要素是藉由在圖21顯示之電路260及記憶體262而安裝。FIG. 21 is a block diagram showing an installation example of the decoding device 200. The decoding device 200 includes a circuit 260 and a memory 262. For example, a plurality of constituent elements of the decoding device 200 shown in FIG. 10 are installed by the circuit 260 and the memory 262 shown in FIG. 21.

電路260是可對記憶體262存取之電子電路，且進行資訊處理。例如，電路260是使用記憶體262來解碼動態圖像之專用或通用之電子電路。電路260亦可以是如CPU般之處理器。又，電路260亦可以是複數個電子電路之集合體。The circuit 260 is an electronic circuit accessible to the memory 262 and performs information processing. For example, the circuit 260 is a dedicated or general-purpose electronic circuit that uses the memory 262 to decode moving images. The circuit 260 may also be a processor such as a CPU. In addition, the circuit 260 may be an aggregate of a plurality of electronic circuits.

又，例如，電路260亦可以發揮在圖10顯示之解碼裝置200之複數個構成要素中之除了用於記憶資訊的構成要素以外之複數個構成要素的功能。亦即，電路260亦可以進行上述之動作來作為該等構成要素之動作。In addition, for example, the circuit 260 may also function as a plurality of constituent elements other than the constituent elements for storing information among the plurality of constituent elements of the decoding device 200 shown in FIG. 10. That is, the circuit 260 may perform the operations described above as operations of these constituent elements.

記憶體262是將電路260用來解碼動態圖像之資訊予以記憶之專用或通用之記憶體。記憶體262亦可以是電子電路，亦可以是與電路260連接，亦可以是包含於電路260。The memory 262 is a dedicated or general-purpose memory that stores information used by the circuit 260 to decode moving images. The memory 262 may also be an electronic circuit, may be connected to the circuit 260, or may be included in the circuit 260.

又，記憶體262亦可以是複數個電子電路之集合體，亦可以是以複數個副記憶體來構成。又，記憶體262亦可以是磁碟片或光碟片等，亦可以表現為儲存器或記錄媒體等。又，記憶體262亦可以是非依電性記憶體，亦可以是依電性記憶體。In addition, the memory 262 may be an aggregate of a plurality of electronic circuits, or may be constituted by a plurality of auxiliary memories. In addition, the memory 262 may be a magnetic disk, an optical disk, or the like, and may also be expressed as a memory or a recording medium. In addition, the memory 262 may be a non-electrically-dependent memory or an electrically-dependent memory.

例如，記憶體262亦可以發揮在圖10顯示之解碼裝置200之複數個構成要素中，用於記憶資訊之構成要素的功能。具體而言，記憶體262亦可以發揮在圖10顯示之區塊記憶體210及幀記憶體214的功能。For example, the memory 262 may also play a function of storing the constituent elements of the information among the constituent elements of the decoding device 200 shown in FIG. 10. Specifically, the memory 262 can also function as the block memory 210 and the frame memory 214 shown in FIG. 10.

又，記憶體262亦可以記憶與已編碼之動態圖像對應之位元串，亦可以記憶已解碼之動態圖像。又，記憶體262亦可以記憶電路260用來解碼動態圖像之程式。In addition, the memory 262 can also store the bit string corresponding to the encoded moving picture, and can also store the decoded moving picture. In addition, the memory 262 may also store a program used by the circuit 260 to decode a moving image.

另，解碼裝置200亦可以不安裝在圖10顯示之複數個構成要素之全部，亦可以不進行上述之複數個處理之全部。亦可以令在圖10顯示之複數個構成要素之一部分包含於其他裝置，亦可以令上述之複數個處理之一部分藉由其他裝置執行。而且，藉由在解碼裝置200安裝圖10顯示之複數個構成要素中之一部分、進行上述之複數個處理之一部分，可將處理之延遲予以抑制。In addition, the decoding device 200 may not be installed on all of the plurality of constituent elements shown in FIG. 10, or may not be performed on all of the plurality of processes described above. A part of the plurality of constituent elements shown in FIG. 10 may be included in other devices, and a part of the plurality of processes described above may also be performed by other devices. Further, by installing one of the plurality of constituent elements shown in FIG. 10 in the decoding device 200 and performing one of the plurality of processes described above, it is possible to suppress processing delay.

圖22是顯示圖21所示之解碼裝置200之動作例的流程圖。例如，解碼裝置200之電路260使用記憶體262來進行圖22所示之動作，藉此，解碼裝置200在構成動態圖像之解碼對象區塊之解碼中進行幀間預測。FIG. 22 is a flowchart showing an operation example of the decoding device 200 shown in FIG. 21. For example, the circuit 260 of the decoding device 200 uses the memory 262 to perform the operation shown in FIG. 22, whereby the decoding device 200 performs inter prediction during decoding of a decoding target block constituting a moving image.

具體而言，電路260是當進行第1處理的情況下，不進行包含第2處理之1個以上之處理而進行解碼對象區塊之幀間預測(S401)。在此，第1處理是使用解碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正解碼對象區塊之預測圖像之處理。第2處理是使用由參考區塊獲得之亮度之空間上的梯度來修正預測圖像之處理。Specifically, when the circuit 260 performs the first process, it performs inter prediction of the decoding target block without performing one or more processes including the second process (S401). Here, the first process is a process of correcting the predicted image of the decoding target block by using the brightness of the periphery of the decoding target block and the brightness of the periphery of the reference block. The second process is a process of correcting a predicted image using a spatial gradient of brightness obtained from a reference block.

藉此，解碼裝置200可在進行被設想成處理時間長之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行幀間預測，前述第2處理是與第1處理同樣地被設想成處理時間長。所以，解碼裝置200可縮短包含第1處理之階段的處理時間，可在動態圖像之解碼將處理之延遲予以抑制。With this, the decoding device 200 can perform inter prediction without performing one or more processes including the second process when performing the first process that is assumed to have a long processing time. The second process is the same as the first process. It is also envisaged that the processing time is long. Therefore, the decoding device 200 can shorten the processing time including the first processing stage, and can suppress the processing delay in the decoding of the moving image.

例如，上述1個以上之處理亦可以包含第3處理，前述第3處理是使用藉由解碼對象區塊之周邊之已解碼區塊之運動向量而從解碼對象區塊指出之參考圖像，來修正預測圖像。藉此，解碼裝置200可在進行第1處理的情況下，不進行第2處理及第3處理而進行幀間預測。所以，解碼裝置200可縮短包含第1處理之階段的處理時間，可在動態圖像之解碼將處理之延遲予以抑制。For example, the above one or more processes may include a third process. The third process is to use a reference image pointed out from the decoding target block by using the motion vectors of the decoded blocks around the decoding target block. Correct the predicted image. This allows the decoding device 200 to perform inter prediction without performing the second processing and the third processing when the first processing is performed. Therefore, the decoding device 200 can shorten the processing time including the first processing stage, and can suppress the processing delay in the decoding of the moving image.

又，例如，上述1個以上之處理亦可以是修正預測圖像之複數個處理中之與第1處理不同之所有處理。藉此，解碼裝置200可在進行第1處理的情況下，不進行第1處理以外之修正處理而進行幀間預測。所以，解碼裝置200可縮短包含第1處理之階段的處理時間，可在動態圖像之解碼將處理之延遲予以抑制。In addition, for example, the one or more processes described above may be all processes different from the first process among a plurality of processes for correcting a predicted image. This allows the decoding device 200 to perform inter prediction without performing correction processing other than the first processing when performing the first processing. Therefore, the decoding device 200 can shorten the processing time including the first processing stage, and can suppress the processing delay in the decoding of the moving image.

又，例如，電路260亦可以在進行第1處理後，以不修正預測圖像的方式而結束解碼對象區塊之幀間預測。藉此，解碼裝置200可在進行第1處理的情況下，在第1處理之後不進行修正處理而進行幀間預測。所以，解碼裝置200可縮短包含第1處理之階段的處理時間，可在動態圖像之解碼中將處理之延遲予以抑制。In addition, for example, the circuit 260 may end the inter prediction of the decoding target block without modifying the prediction image after performing the first processing. This allows the decoding device 200 to perform inter-prediction without performing correction processing after the first processing when performing the first processing. Therefore, the decoding device 200 can shorten the processing time including the first processing stage, and can suppress the processing delay in decoding the moving image.

又，例如，電路260亦可以判定是否以低延遲模式進行解碼對象區塊之解碼。而且，電路260亦可以當在判定成以低延遲模式進行解碼對象區塊之解碼的情況下進行第1處理的情況下，不進行上述1個以上之處理而進行解碼對象區塊之幀間預測。藉此，解碼裝置200可在低延遲模式縮短包含第1處理之階段的處理時間，可在動態圖像之解碼將處理之延遲予以抑制。In addition, for example, the circuit 260 may determine whether to decode the decoding target block in the low-latency mode. In addition, when the circuit 260 performs the first processing when it is determined that the decoding target block is decoded in the low-latency mode, the inter-prediction of the decoding target block is not performed without performing the above one or more processes. . Thereby, the decoding device 200 can shorten the processing time including the first processing stage in the low-latency mode, and can suppress the processing delay in the decoding of the moving image.

又，例如，電路260亦可以從動態圖像之編碼串流之序列標頭區域、圖片標頭區域、圖塊標頭區域、片段標頭區域及補助資訊區域之至少其中1者，解碼出表示是否以低延遲模式進行解碼對象區塊之解碼之資訊。藉此，解碼裝置200可從編碼裝置100獲得是否為低延遲模式之資訊。In addition, for example, the circuit 260 may also decode at least one of a sequence header area, a picture header area, a tile header area, a fragment header area, and a supplementary information area of a coded stream of a moving image. Whether to decode the decoding target block in low-latency mode. Thereby, the decoding device 200 can obtain information from the encoding device 100 as to whether it is a low-latency mode.

又，例如，電路260亦可以是依循分配給動態圖像之編碼串流之設定檔或等級資訊，而判定是否以低延遲模式進行解碼對象區塊之解碼。藉此，解碼裝置200可依循設定檔或等級，而適應性地控制是否以低延遲模式進行解碼。
[補充]In addition, for example, the circuit 260 may determine whether to decode the decoding target block in the low-latency mode according to the profile or level information of the encoded stream allocated to the moving image. Thereby, the decoding device 200 can adaptively control whether to perform decoding in a low-latency mode according to a set file or level.
[supplement]

本實施形態之編碼裝置100及解碼裝置200可以分別作為圖像編碼裝置及圖像解碼裝置來利用。或者，編碼裝置100及解碼裝置200可以分別作為幀間預測裝置來利用。亦即，編碼裝置100及解碼裝置200亦可以分別只與幀間預測部126及幀間預測部218對應。The encoding device 100 and the decoding device 200 of this embodiment can be used as an image encoding device and an image decoding device, respectively. Alternatively, the encoding device 100 and the decoding device 200 may be used as inter prediction devices, respectively. That is, the encoding device 100 and the decoding device 200 may correspond to only the inter prediction unit 126 and the inter prediction unit 218, respectively.

又，在上述之說明中，與處理對象區塊、已處理區塊及已處理周邊區塊等對應之處理是例如LIC處理，具體而言，亦可以是使用亮度修正參數來修正預測圖像之處理。又，此處理亦可以是將使用亮度修正參數來修正預測圖像之處理含括在內之階段的處理。又，已處理區塊及已處理周邊區塊亦可以是因為將處理跳過而結束處理之區塊。In the above description, the processing corresponding to the processing target block, the processed block, and the processed surrounding block is, for example, LIC processing. Specifically, the brightness correction parameter may be used to correct the predicted image. deal with. This process may be a process including a process of correcting a predicted image using a brightness correction parameter. In addition, the processed block and the processed surrounding blocks may also be blocks that are terminated because processing is skipped.

又，與區塊之周邊對應之範圍可以是與區塊鄰接之範圍，亦可以是從區塊起算之預定像素數之範圍。The range corresponding to the periphery of the block may be a range adjacent to the block, or a range of a predetermined number of pixels from the block.

又，雖然在本實施形態是藉由專用之硬體來構成各構成要素，但亦可以藉由執行適合各構成要素之軟體程式來實現。各構成要素亦可以藉由如下方式而實現：CPU或處理器等之程式執行部將記錄在硬碟或半導體記憶體等之記錄媒體的軟體程式讀取而執行。In addition, although each constituent element is configured by dedicated hardware in this embodiment, it can also be implemented by executing a software program suitable for each constituent element. Each constituent element can also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

具體而言，編碼裝置100及解碼裝置200亦可以分別具備處理電路(Processing Circuitry)、以及與該處理電路電連接之可從該處理電路存取之記憶裝置(Storage)。例如，處理電路是與電路160或260對應，記憶裝置是與記憶體162或262對應。Specifically, the encoding device 100 and the decoding device 200 may each include a processing circuit and a storage device that is electrically connected to the processing circuit and accessible from the processing circuit. For example, the processing circuit corresponds to the circuit 160 or 260, and the memory device corresponds to the memory 162 or 262.

處理電路包含有專用之硬體及程式執行部之至少其中一者，並使用記憶裝置來執行處理。又，當處理電路包含有程式執行部的情況下，記憶裝置是記憶藉由該程式執行部來執行的軟體程式。The processing circuit includes at least one of dedicated hardware and a program execution unit, and performs processing using a memory device. When the processing circuit includes a program execution unit, the memory device stores a software program executed by the program execution unit.

在此，實現本實施形態之編碼裝置100或解碼裝置200等之軟體是如下之程式。Here, software for realizing the encoding device 100, the decoding device 200, and the like of the present embodiment is the following program.

例如，此程式令電腦執行在構成動態圖像之編碼對象區塊之編碼中進行幀間預測之編碼方法，前述編碼方法是當進行使用前述編碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述編碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述編碼對象區塊之幀間預測，前述第2處理是使用由前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。For example, this program causes a computer to execute an encoding method for performing inter prediction in the encoding of an encoding target block constituting a moving image. The foregoing encoding method is to use the brightness of the periphery of the encoding target block and the periphery of the reference block. When the brightness is used to correct the first process of the predicted image of the coding target block, the inter prediction of the coding target block is performed without performing one or more processes including the second process. The second process is A spatial gradient of the brightness obtained from the aforementioned reference block is used to correct the aforementioned predicted image.

又，例如，此程式令電腦執行在構成動態圖像之解碼對象區塊之解碼中進行幀間預測之解碼方法，前述解碼方法是當進行使用前述解碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述解碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述解碼對象區塊之幀間預測，前述第2處理是使用由前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。Also, for example, this program causes a computer to execute a decoding method for performing inter prediction in the decoding of a decoding target block constituting a moving image. The foregoing decoding method is to use the brightness and reference blocks around the decoding target block. In the case of the first processing of correcting the predicted image of the decoding target block by the peripheral brightness, the inter prediction of the decoding target block is performed without performing one or more processings including the second processing. The processing is to use the spatial gradient of the brightness obtained from the aforementioned reference block to modify the aforementioned predicted image.

又，如上述，各構成要素亦可以是電路。該等電路可以是整體構成1個電路，亦可以分別是個別之電路。又，各構成要素可以藉由通用之處理器而實現，亦可以藉由專用之處理器而實現。As described above, each constituent element may be a circuit. These circuits may constitute a single circuit as a whole, or they may be separate circuits. In addition, each constituent element may be realized by a general-purpose processor, or may be realized by a dedicated processor.

又，亦可以由別的構成要素來執行特定之構成要素所執行之處理。又，亦可以變更處理之執行順序，亦可以並行地執行複數個處理。又，亦可以是編碼解碼裝置具備編碼裝置100及解碼裝置200。In addition, the processing performed by a specific component may be performed by another component. In addition, the execution order of the processes may be changed, or a plurality of processes may be executed in parallel. The encoding / decoding device may include the encoding device 100 and the decoding device 200.

亦可以將在說明中使用之第1及第2等之序數適宜地更換。又，可以對構成要素等賦予新的序數，亦可以對構成要素等去除序數。The first and second ordinal numbers used in the description may be replaced as appropriate. In addition, a new ordinal number may be added to the constituent elements and the like, and the ordinal number may be removed from the constituent elements and the like.

雖然以上是基於實施形態來說明編碼裝置100及解碼裝置200之態樣，但編碼裝置100及解碼裝置200之態樣並非限定於此實施形態。只要未脫離本揭示之主旨，則所屬技術領域中具有通常知識者將所能想到之各種變形施加在本實施形態而成者、將不同實施形態之構成要素組合而建構之形態，亦可以包含於編碼裝置100及解碼裝置200之態樣之範圍內。Although the aspects of the encoding device 100 and the decoding device 200 have been described above based on the embodiment, the aspects of the encoding device 100 and the decoding device 200 are not limited to this embodiment. As long as it does not deviate from the gist of the present disclosure, a person with ordinary knowledge in the technical field who can conceive various deformations that can be conceived in this embodiment, and forms constructed by combining the constituent elements of different embodiments can also be included Within the scope of aspects of the encoding device 100 and the decoding device 200.

亦可以將本態樣與本揭示之其他態樣之至少一部分組合而實施。又，亦可以將在本態樣之流程圖記載之一部分處理、裝置之一部分構成、語法之一部分等與其他態樣組合而實施。
(實施形態2)This aspect may also be implemented in combination with at least a part of other aspects of the present disclosure. In addition, a part of the processing described in the flowchart of this aspect, a part of the device configuration, a part of the grammar, and the like may be implemented in combination with other aspects.
(Embodiment 2)

在以上之各實施形態，各功能區塊通常可由MPU及記憶體等而實現。又，各功能區塊之處理通常是藉由如下方式而實現：處理器等之程式執行部將記錄在ROM等之記錄媒體的軟體(程式)讀取而執行。該軟體可以藉由下載等而分發，亦可以記錄在半導體記憶體等之記錄媒體來分發。另，當然亦可藉由硬體(專用電路)來實現各功能區塊。In each of the above embodiments, each functional block can usually be implemented by an MPU, a memory, or the like. The processing of each functional block is usually realized by a program execution unit such as a processor reading and executing software (program) recorded on a recording medium such as a ROM. The software may be distributed by downloading or the like, or may be distributed on a recording medium such as a semiconductor memory. In addition, of course, each functional block can also be realized by hardware (dedicated circuit).

又，在各實施形態說明之處理可以是使用單一之裝置(系統)予以集中處理而實現，或者，亦可以是使用複數個裝置予以分散處理而實現。又，執行上述程式之處理器可以是單個，亦可以是複數個。亦即，可以進行集中處理，或者亦可以進行分散處理。In addition, the processing described in each embodiment may be realized by centralized processing using a single device (system), or may be realized by distributed processing using a plurality of devices. In addition, the processor executing the above program may be a single processor or a plurality of processors. That is, centralized processing may be performed, or distributed processing may be performed.

本揭示之態樣並非限定於以上之實施例，可進行各式各樣之變更，且這些亦包含於本揭示之態樣之範圍內。The aspects of the present disclosure are not limited to the above embodiments, and various changes can be made, and these are also included in the scope of the aspects of the present disclosure.

再者，在此說明在上述各實施形態顯示之動態圖像編碼方法(圖像編碼方法)或動態圖像解碼方法(圖像解碼方法)之應用例與使用其之系統。該系統之特徵是具有：用到圖像編碼方法之圖像編碼裝置、用到圖像解碼方法之圖像解碼裝置、及具備雙方之圖像編碼解碼裝置。系統中之其他構成可因應情況而適切地變更。
[使用例]In addition, application examples of the moving image encoding method (image encoding method) or the moving image decoding method (image decoding method) and the system using the moving image encoding method (image encoding method) displayed in the above embodiments will be described. This system is characterized by having an image encoding device using an image encoding method, an image decoding device using an image decoding method, and an image encoding and decoding device having both. Other components in the system can be appropriately changed according to the situation.
[Example of use]

圖23是顯示將內容發布服務實現之內容供給系統ex100之整體構成的圖。將通訊服務之提供區域分割成想要之大小，在各細胞(cell)內分別設置有固定無線台即基地台ex106、ex107、ex108、ex109、ex110。FIG. 23 is a diagram showing the overall configuration of a content supply system ex100 that implements a content distribution service. The communication service providing area is divided into desired sizes, and fixed wireless stations, namely base stations ex106, ex107, ex108, ex109, and ex110, are provided in each cell.

在該內容供給系統ex100中，電腦ex111、遊戲機ex112、相機ex113、家電ex114、及智慧型手機ex115等之各機器是透過網際網路服務供應商ex102或通訊網ex104、及基地台ex106~ex110而連接於網際網路ex101。該內容供給系統ex100亦可以將上述之任一要素組合而連接。各機器亦可以不透過固定無線台即基地台ex106~ex110，而是透過電話網或近距離無線等直接或間接地相互連接。又，串流伺服器ex103是透過網際網路ex101等而與電腦ex111、遊戲機ex112、相機ex113、家電ex114、及智慧型手機ex115等之各機器連接。又，串流伺服器ex103是透過衛星ex116而與飛機ex117內之熱點內之終端等連接。In this content supply system ex100, each device such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 is passed through an Internet service provider ex102 or a communication network ex104, and a base station ex106 to ex110. Connected to the internet ex101. The content supply system ex100 may be connected by combining any of the above-mentioned elements. Each device can also be directly or indirectly connected to each other through a fixed wireless station, ie, base stations ex106 to ex110, but through a telephone network or short-range wireless. The streaming server ex103 is connected to various devices such as a computer ex111, a game machine ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 via the Internet ex101 and the like. The streaming server ex103 is connected to a terminal or the like in a hot spot in the aircraft ex117 through the satellite ex116.

另，亦可以使用無線存取點或熱點等來取代基地台ex106~ex110。又，串流伺服器ex103亦可以不透過網際網路ex101或網際網路服務供應商ex102而直接與通訊網ex104連接，亦可以不透過衛星ex116而直接與飛機ex117連接。In addition, a wireless access point or a hotspot may be used instead of the base stations ex106 to ex110. In addition, the streaming server ex103 may be directly connected to the communication network ex104 without using the Internet ex101 or the Internet service provider ex102, or may be directly connected to the aircraft ex117 without using the satellite ex116.

相機ex113是數位相機等之可拍攝靜止圖及拍攝動態圖之機器。又，智慧型手機ex115一般而言是與2G、3G、3.9G、4G、以及今後被稱作5G之行動通訊系統之方式對應之智慧型手機機器、行動電話、或PHS(Personal Handyphone System，個人手持電話系統)等。The camera ex113 is a digital camera or the like capable of taking still pictures and moving pictures. In addition, the smartphone ex115 is generally a smartphone device, a mobile phone, or a PHS (Personal Handyphone System, personal) that corresponds to the method of 2G, 3G, 3.9G, 4G, and a mobile communication system to be referred to as 5G in the future. Handheld phone system) and so on.

家電ex118是冰箱、或包含於家庭用燃料電池熱電共生系統之機器等。The home appliance ex118 is a refrigerator or a device included in a domestic fuel cell cogeneration system.

在內容供給系統ex100，具有拍攝功能之終端是藉由通過基地台ex106等與串流伺服器ex103連接，而可進行實況發布等。在實況發布，終端(電腦ex111、遊戲機ex112、相機ex113、家電ex114、智慧型手機ex115、及飛機ex117內之終端等)是對使用者使用該終端而拍攝之靜止圖或動態圖內容，進行在上述各實施形態說明之編碼處理，將藉由編碼而獲得之影像資料、以及與影像對應的聲音經編碼後之聲音資料予以多工，將獲得之資料朝串流伺服器ex103發送。亦即，各終端是作為與本揭示之一態樣相關之圖像編碼裝置而發揮功能。In the content supply system ex100, a terminal having a shooting function is connected to the streaming server ex103 through a base station ex106 and the like, and can perform live distribution and the like. In live release, the terminals (computer ex111, game console ex112, camera ex113, home appliances ex114, smart phones ex115, and terminals in aircraft ex117, etc.) are content for still pictures or dynamic pictures taken by users using the terminal. In the encoding process described in each of the above embodiments, the image data obtained by encoding and the encoded audio data of the sound corresponding to the image are multiplexed, and the obtained data is sent to the streaming server ex103. That is, each terminal functions as an image coding device related to one aspect of the present disclosure.

另一方面，串流伺服器ex103是把對於有要求之客戶端發送的內容資料進行串流發布。客戶端是可將經過上述編碼處理之資料解碼之電腦ex111、遊戲機ex112、相機ex113、家電ex114、智慧型手機ex115、或飛機ex117內之終端等。接收到發布之資料的各機器是對接收到之資料進行解碼處理而予以播放。亦即，各機器是作為與本揭示之一態樣之圖像解碼裝置而發揮功能。
[分散處理]On the other hand, the streaming server ex103 performs streaming distribution on content data sent by a requesting client. The client is a computer ex111, a game machine ex112, a camera ex113, a home appliance ex114, a smart phone ex115, or a terminal in an aircraft ex117 that can decode the data processed by the above encoding process. Each machine that receives the published data decodes the received data and plays it. That is, each device functions as an image decoding device according to one aspect of the present disclosure.
[Decentralized processing]

又，串流伺服器ex103亦可以是複數個伺服器或複數個電腦，將資料分散地處理、記錄、或發布。例如，串流伺服器ex103亦可以是藉由CDN(Contents Delivery Network，內容傳遞網路)而實現，藉由將在世界中分散之多數個邊緣伺服器與邊緣伺服器間連繫之網路而實現內容發布。CDN是因應客戶端而動態地分配物理上較近之邊緣伺服器。而且，可藉由在該邊緣伺服器進行內容之快取及發布而令延遲減少。又，由於可在發生某些錯誤的情況下、或因為流量增加等而造成通訊狀態改變的情況下，藉由複數個邊緣伺服器而令處理分散、或將發布主體切換成其他邊緣伺服器、或繞過發生障礙之網路部分而繼續發布，故可實現高速且穩定之發布。In addition, the streaming server ex103 may be a plurality of servers or a plurality of computers, and the data may be processed, recorded, or distributed in a distributed manner. For example, the streaming server ex103 can also be implemented by a CDN (Contents Delivery Network, Content Delivery Network), and by a network that connects the majority of edge servers and edge servers scattered throughout the world. Achieve content publishing. The CDN dynamically allocates physically closer edge servers in response to clients. In addition, latency can be reduced by caching and publishing content on the edge server. In addition, because some errors may occur, or the communication status changes due to increased traffic, etc., the processing may be distributed by a plurality of edge servers, or the publishing subject may be switched to another edge server. Or bypass the part of the network where the obstacle occurs and continue to publish, so you can achieve high-speed and stable release.

又，不僅止於發布本身之分散處理，拍攝之資料之編碼處理亦可以在各終端進行，亦可以在伺服器側進行，亦可以互相分擔而進行。作為一例，一般而言，編碼處理是進行2次處理迴路。在第1次之迴路，檢測出幀或場景單位之圖像的複雜度或編碼量。又，在第2次之迴路，進行維持畫質且提升編碼效率之處理。例如，終端進行第1次之編碼處理，接收到內容之伺服器側進行第2次之編碼處理，藉此，可一面減少在各終端之處理負荷，一面提升內容之質與效率。此情況下，若有幾乎即時地接收且解碼之要求，則亦可令終端已進行第一次編碼之資料在其他終端接收而播放，故亦可更彈性地進行即時發布。In addition, not only the distributed processing of the release itself, but also the encoding processing of the photographed data can be performed at each terminal, it can also be performed on the server side, and it can also be performed by sharing each other. As an example, in general, the encoding process is performed in a secondary processing loop. In the first loop, the complexity or encoding amount of the image in frames or scenes is detected. In the second pass, processing is performed to maintain image quality and improve coding efficiency. For example, the terminal performs the first encoding process, and the server side that receives the content performs the second encoding process, thereby reducing the processing load on each terminal and improving the quality and efficiency of the content. In this case, if there is a requirement to receive and decode almost instantaneously, the data that the terminal has encoded for the first time can also be received and played on other terminals, so real-time publishing can also be more flexible.

作為其他之例，相機ex113等是從圖像進行特徵量擷取，將關於特徵量之資料作為元資料(metadata)而壓縮並朝伺服器發送。伺服器是例如從特徵量來判斷物件(object)之重要性而切換量化精度等，因應圖像之意義而進行壓縮。特徵量資料是對於在伺服器再次壓縮時之運動向量預測的精度及效率提升尤其有效。又，亦可以在終端進行VLC(可變長度編碼)等之簡易的編碼、在伺服器進行CABAC(內文適應性二值算術編碼方式)等處理負荷大之編碼。As another example, the camera ex113 or the like extracts feature quantities from an image, compresses data about the feature quantities as metadata, and sends the data to a server. For example, the server judges the importance of an object from a feature quantity, switches the quantization accuracy, etc., and compresses it according to the meaning of the image. Feature data is especially effective for improving the accuracy and efficiency of motion vector prediction when the server is compressed again. In addition, simple coding such as VLC (Variable Length Coding) can be performed on the terminal, and coding with a large processing load such as CABAC (text adaptive binary arithmetic coding method) can be performed on the server.

進而以其他例來說，在運動場、購物中心或工廠等有時會有藉由複數個終端拍攝幾乎同一場景而有複數個影像資料存在的情況。此情況下，使用進行拍攝之複數個終端、且因應需要而使用未拍攝之其他終端及伺服器，例如以GOP(Group of Picture，圖片組)單位、圖片單位、或將圖片分割而成之圖塊單位等將編碼處理分別分配而進行分散處理。藉此，可減少延遲、更加實現即時性。Furthermore, for another example, in a sports field, a shopping mall, a factory, or the like, there may be cases where a plurality of video materials exist when a plurality of terminals shoot almost the same scene. In this case, use a plurality of terminals for shooting, and use other terminals and servers that are not taken as needed, such as GOP (Group of Picture) units, picture units, or a picture divided into pictures The block unit or the like distributes encoding processing separately and performs distributed processing. As a result, delays can be reduced and immediateness can be achieved.

又，由於複數個影像資料幾乎是同一場景，故亦可以由伺服器管理及/或指示將各終端所拍攝之影像資料互相參考。或者，亦可以由伺服器接收來自各終端之已編碼資料，且在複數個資料間變更參考關係、或者修正或抽換圖片本身並重新編碼。藉此，可生成令一個個資料之質與效率提高之串流。In addition, since the plurality of image data are almost the same scene, the server can also manage and / or instruct the image data captured by each terminal to refer to each other. Alternatively, the server may also receive the encoded data from each terminal, and change the reference relationship between the plurality of data, or modify or replace the picture itself and re-encode. In this way, a stream that improves the quality and efficiency of individual data can be generated.

又，伺服器亦可以進行將影像資料之編碼方式變更之轉碼，再發布影像資料。例如，伺服器亦可以將MPEG系之編碼方式轉換成VP系，亦可以將H.264轉換成H.265。In addition, the server can also transcode the encoding method of the image data and publish the image data. For example, the server can also convert the encoding method of MPEG system to VP system, and it can also convert H.264 to H.265.

如此，編碼處理可藉由終端或1個以上之伺服器而進行。所以，雖然以下是使用「伺服器」或「終端」等之記載來作為進行處理之主體，但亦可以令以伺服器進行之處理之一部分或全部是在終端進行，亦可以令以終端進行之處理之一部分或全部是在伺服器進行。又，關於這些，解碼處理亦是同樣的。
[3D、多角度]In this way, the encoding process can be performed by a terminal or one or more servers. Therefore, although the following uses the description of "server" or "terminal" as the main body for processing, some or all of the processing performed by the server can be performed on the terminal, or it can be performed by the terminal. Part or all of the processing is performed on the server. The same applies to the decoding process.
[3D, multi-angle]

近年來，將藉由幾乎互相同步之複數個相機ex113及/或智慧型手機ex115等終端所拍攝之不同場景、或者從不同角度拍攝同一場景的圖像或影像統合而利用之情形亦逐漸增加。各終端所拍攝之影像是基於另外取得之終端間的相對位置關係、或者影像含有之特徵點一致的區域等而統合。In recent years, the use of different scenes taken by terminals such as multiple cameras ex113 and / or smart phones ex115, which are almost synchronized with each other, or integration of images or images of the same scene taken from different angles has also gradually increased. The images captured by each terminal are integrated based on the relative positional relationship between the terminals obtained separately or the regions with the same feature points included in the image.

伺服器亦可以不只將2維之動態圖像編碼，還基於動態圖像之場景解析等而自動地或在使用者指定之時刻將靜止圖編碼，朝接收終端發送。伺服器更可在可取得拍攝終端間之相對位置關係的情況下，不只2維之動態圖像，還基於從不同角度拍攝同一場景之影像，而生成該場景之3維形狀。另，伺服器可以將藉由點雲等而生成之3維之資料另外編碼，亦可以基於使用3維資料而辨識或追蹤人物或物件之結果，將要朝接收終端發送之影像從複數個終端所拍攝之影像中選擇、或者重建而生成。The server may not only encode the 2-dimensional moving image, but also automatically or at the time specified by the user based on the scene analysis of the moving image to encode the still image and send it to the receiving terminal. When the server can obtain the relative positional relationship between the shooting terminals, not only the two-dimensional dynamic image, but also the three-dimensional shape of the scene based on the images of the same scene taken from different angles. In addition, the server can additionally encode the 3D data generated by the point cloud, etc., and can also recognize or track people or objects based on the use of the 3D data. The images to be sent to the receiving terminal are obtained from multiple terminals. It is generated by selecting or reconstructing the captured image.

如此，使用者可任意地選擇與各拍攝終端對應之各影像而享受場景，亦可享受從使用複數圖像或影像而重建之3維資料剪出來的任意視點之影像的內容。再者，與影像同樣，聲音亦可以從複數個不同之角度來收音，伺服器亦可以配合影像而將來自特定之角度或空間的聲音與影像多工而發送。In this way, the user can arbitrarily select each image corresponding to each shooting terminal to enjoy the scene, and can also enjoy the content of the image at any viewpoint cut out from the three-dimensional data reconstructed using the plural images or images. Furthermore, like video, sound can also be received from a number of different angles, and the server can also multiplex and send sound and video from a specific angle or space in cooperation with the video.

又，近年來，Virtual Reality(VR，虛擬實境)及Augmented Reality(AR，擴增實境)等將現實世界與虛擬世界加上對應關係之內容亦逐漸普及。VR圖像的情況下，伺服器可以分別製作右眼用及左眼用之視點圖像，藉由Multi-View Coding(MVC，多視角編碼)等進行容許在各視點影像間參考之編碼，亦可以互不參考地當作相異的串流來編碼。在解碼不同的串流時，因應使用者之視點，以可將虛擬之3維空間再現的方式互相同步而播放即可。In addition, in recent years, contents such as Virtual Reality (VR, Augmented Reality) (AR, Augmented Reality) that integrate the real world with the virtual world have gradually become popular. In the case of VR images, the server can separately create right-view and left-view viewpoint images, and perform multi-view coding (MVC, multi-view coding), etc., to allow encoding between reference images at each viewpoint. Can be coded as distinct streams without reference to each other. When decoding different streams, it can be played in synchronization with each other in a manner that can reproduce the virtual three-dimensional space according to the user's point of view.

AR圖像的情況下，伺服器是基於3維之位置或使用者之視點之運動，而將虛擬空間上之虛擬物體資訊重疊在現實空間之相機資訊。解碼裝置亦可以取得或保持虛擬物體資訊及3維資料，因應使用者之視點之運動而生成2維圖像並流暢地連繫，藉此製作重疊資料。或者，亦可以是解碼裝置除了虛擬物體資訊之請求之外，還將使用者之視點之運動朝伺服器發送，伺服器配合接收到之視點之運動而從伺服器所保持之3維資料製作重疊資料，將重疊資料編碼而朝解碼裝置發布。另，亦可以是重疊資料除了RGB以外還具有表示透明度之α值，伺服器將從3維資料製作之物件以外的部分之α值設定成0等，以該部分為透明之狀態而編碼。或者，伺服器亦可以如色度鍵般地將預定值之RGB值設定成背景，生成令物件以外之部分是背景色之資料。In the case of AR images, the server overlays the virtual object information in the virtual space with the camera information in the real space based on the 3D position or the movement of the user's viewpoint. The decoding device can also obtain or maintain virtual object information and 3D data, and generate 2D images based on the movement of the user's viewpoint and connect them smoothly to create overlapping data. Alternatively, in addition to the request of the virtual object information, the decoding device may also send the user's viewpoint motion to the server, and the server will make an overlay from the 3D data held by the server in accordance with the received viewpoint motion. The data is encoded and distributed to the decoding device. In addition, the superimposed data may have an alpha value indicating transparency in addition to RGB, and the server may set the alpha value of a part other than the object created from the three-dimensional data to 0, etc., and encode the part as transparent. Alternatively, the server can set the RGB value of the predetermined value as the background like a chroma key, and generate data that makes the part other than the object a background color.

同樣地，發布之資料之解碼處理可以在身為客戶端之各終端進行，亦可以在伺服器側進行，亦可以是互相分擔而進行。作為一例，亦可以是某終端暫且朝伺服器傳送接收要求，由其他終端將與要求對應之內容接收而進行解碼處理，將已解碼之訊號發送至具有顯示器之裝置。可藉由將處理分散、選擇適切之內容，而無關可通訊之終端本身之性能地播放畫質佳之資料。又，作為其他例，亦可以一面藉由TV等接收大尺寸之圖像資料，一面在觀賞者之個人終端令分割圖片而成之圖塊等一部分的區域解碼而顯示。藉此，可一面令整體像共用化，一面在手邊確認自己所負責之領域或想更詳細確認之區域。Similarly, the decoding process of published data can be performed at each terminal that is a client, it can also be performed at the server side, or it can be performed by sharing each other. As an example, a terminal may temporarily transmit a receiving request to a server, and other terminals may receive content corresponding to the request, perform decoding processing, and send the decoded signal to a device having a display. By dispersing the processing and selecting the appropriate content, it is possible to play back high-quality data without regard to the performance of the communicable terminal itself. Also, as another example, a large-size image data may be received by a TV or the like, and a part of an area such as a tile obtained by dividing a picture may be decoded and displayed on a viewer's personal terminal. With this, you can confirm the area you are responsible for or the area you want to confirm in more detail while making the overall image common.

又，未來可預想的是不管室內外，當近距離、中距離、或長距離之無線通訊可使用複數種之狀況下，利用MPEG-DASH等之發布系統規格，一面對連接中之通訊切換適切之資料、一面無縫地接收內容。藉此，使用者不受限於自己之終端，可一面自由地選擇在室內外設置之顯示器等之解碼裝置或顯示裝置，一面即時地切換。又，可基於自己之位置資訊等而一面切換解碼之終端及顯示之終端，一面進行解碼。藉此，在往目的地之移動中，可一面令埋有可顯示之器件的旁邊建築物之壁面或地面之一部分顯示地圖資訊一面移動。又，亦可以基於編碼資料是被快取在接收終端能以短時間存取之伺服器、或者是被複製在內容傳遞服務之邊緣伺服器、等之在網路上存取編碼資料之容易程度，而切換接收資料之位元率。
[可適性編碼]In the future, it is expected that regardless of indoor and outdoor, when multiple types of wireless communication can be used at short, medium, or long distances, publish system specifications such as MPEG-DASH will be used to switch communication during connection. Receiving content with the right information. Thereby, the user is not limited to his own terminal, and can switch in real time while freely selecting a decoding device or a display device such as a display installed indoors and outdoors. In addition, it is possible to perform decoding while switching between the terminal to be decoded and the terminal to be displayed based on its own location information. Thereby, in the movement to the destination, it is possible to move a part of the wall or the ground of a building next to the displayable device to display map information while moving. In addition, the ease of accessing the encoded data on the network can also be based on whether the encoded data is cached on a server that the receiving terminal can access in a short time, or it is copied on the edge server of the content delivery service, etc. The bit rate of the received data is switched.
[Adaptability coding]

關於內容之切換，使用在圖24顯示之應用上述各實施形態所顯示之動態圖像編碼方法而壓縮編碼的可適性串流來說明。雖然伺服器能以當作個別之串流的方式而具有內容相同、品質不同之複數個串流，但亦可以是如下之構成：活用藉由如圖示般地分層進行編碼而實現之時間/空間可適性之串流的特徵來切換內容。亦即，解碼側是因應所謂性能之內在因素與通訊頻帶狀態等之外在因素而決定要解碼到哪一層為止，藉此，解碼側可將低解析度之內容與高解析度之內容自由地切換而解碼。例如，當想要在回家後以網際網路TV等機器來收看在移動中以智慧型手機ex115收看之影像之後續的情況下，該機器只要將相同之串流解碼至不同之層即可，故可減輕伺服器側之負擔。The content switching will be described using an adaptive stream that is compression-encoded by applying the moving picture encoding method shown in each of the above embodiments shown in FIG. 24. Although the server can have a plurality of streams with the same content and different quality as a separate stream, it can also have a structure that utilizes the time realized by layering and encoding as shown in the figure / Spatial adaptability of streaming features to switch content. That is, the decoding side decides which layer to decode based on the so-called internal factors of performance and external factors such as the state of the communication frequency band. Thus, the decoding side can freely define low-resolution content and high-resolution content. Switch while decoding. For example, when you want to use a device such as Internet TV to watch the follow-up video on a mobile phone ex115 when you return home, the device only needs to decode the same stream to different layers. , So it can reduce the burden on the server side.

再者，除了如上述般依各層而編碼有圖片、將在基礎層之上位存在有增強層的可適性予以實現之構成外，亦可以是：增強層含有基於圖像之統計資訊等的元資訊，解碼側基於元資訊而將基礎層之圖片超高解析度成像，藉此生成高畫質化之內容。超高解析度成像可以是同一解析度之SN比之提升、及解析度之擴大之任一者。元資訊包含有以下資訊：用於特定出在超高解析度成像處理中使用之線性或非線性之濾波係數的資訊、或者特定出在超高解析度成像處理中使用之濾波處理、機械學習或最小平方演算之參數值的資訊等。In addition, in addition to the structure in which pictures are encoded according to each layer as described above, and the adaptability of an enhancement layer existing above the base layer is realized, the enhancement layer may include meta-information such as statistical information based on images. On the decoding side, based on the meta-information, the picture of the base layer is super-high-resolution imaged, thereby generating high-quality content. The ultra-high-resolution imaging may be any one of an increase in the SN ratio and an increase in the resolution of the same resolution. The meta information contains the following information: information for specifying linear or non-linear filter coefficients used in ultra-high-resolution imaging processing, or filter processing, mechanical learning or Information on parameter values of least squares calculations, etc.

或者，亦可以是如下之構成：因應圖像內之物件等的內涵而將圖片分割成圖塊等，解碼側藉由選擇欲解碼之圖塊而只解碼一部分之區域。又，可將物件之屬性(人物、車、球等)與影像內之位置(同一圖像中之座標位置等)作為元資訊而儲存，藉此，解碼側可基於元資訊而找出想要之物件的位置，決定含有該物件之圖塊。例如，如圖25所示，元資訊是使用HEVC之SEI訊息等與像素資料不同之資料儲存構造而被儲存。此元資訊例如是表示主物件之位置、尺寸、或色彩等。Alternatively, it may have a structure in which a picture is divided into tiles or the like according to the connotation of an object or the like in the image, and the decoding side decodes only a part of the area by selecting a tile to be decoded. In addition, the attributes of objects (people, cars, balls, etc.) and positions in the image (positions of coordinates in the same image, etc.) can be stored as meta information, whereby the decoding side can find the The position of the object determines the block containing the object. For example, as shown in FIG. 25, the meta information is stored using a data storage structure different from the pixel data, such as the SEI message of HEVC. This meta information indicates, for example, the position, size, or color of the main object.

又，亦可以是以串流、序列或隨機存取單位等由複數個圖片構成之單位來儲存元資訊。藉此，解碼側可取得特定人物在影像內出現之時刻等，可藉由配合圖片單位之資訊而找出物件存在之圖片、及在圖片內之物件之位置。
[Web頁面之最佳化]In addition, the meta information may be stored in units of a plurality of pictures, such as a stream, a sequence, or a random access unit. With this, the decoding side can obtain the moment when a specific character appears in the image, etc., and can find the picture of the object and the location of the object in the picture by matching the information of the picture unit.
[Optimization of Web pages]

圖26是顯示電腦ex111等之web頁面之顯示畫面例的圖。圖27是顯示智慧型手機ex115等之web頁面之顯示畫面例的圖。如圖26及圖27所示，web頁面有時含有複數個往圖像內容的連結即連結圖像，其外貌會隨著瀏覽之器件而不同。當在畫面上看到複數個連結圖像的情況下，直到使用者明示地選擇連結圖像為止、或者直到連結圖像靠近畫面中央附近或連結圖像整體進入畫面內為止，顯示裝置(解碼裝置)是將各內容具有之靜止圖或I圖片當作連結圖像而顯示、或者藉由複數個靜止圖或I圖片等而顯示如gif動畫般之影像、或者僅接收基礎層而將影像解碼及顯示。FIG. 26 is a diagram showing an example of a display screen of a web page such as a computer ex111. 27 is a diagram showing an example of a display screen of a web page such as a smartphone ex115. As shown in FIG. 26 and FIG. 27, a web page sometimes contains a plurality of links to image content, that is, a linked image, and its appearance may vary depending on the device being viewed. When a plurality of connected images are seen on the screen, the display device (decoding device) is not displayed until the user explicitly selects the connected image, or until the connected image is near the center of the screen or the entire connected image enters the screen. ) Is to display the still image or I picture of each content as a linked image, or to display a gif animation like a plurality of still images or I pictures, etc., or to decode the image by receiving only the base layer and display.

當由使用者選擇了連結圖像的情況下，顯示裝置是將基礎層最優先解碼。另，若在構成web頁面之HTML中有表示其為可適性內容之資訊，則顯示裝置亦可以解碼至增強層。又，為了保障即時性，當選擇前或通訊頻帶非常嚴苛的情況下，顯示裝置僅將前方參考之圖片(I圖片、P圖片、僅前方參考之B圖片)解碼及顯示，藉此，可降低前頭圖片之解碼時刻與顯示時刻之間的延遲(從開始解碼內容至開始顯示內容為止之延遲)。又，顯示裝置亦可以刻意無視圖片之參考關係，令全部之B圖片及P圖片為前方參考而粗糙地解碼，隨著時間經過、接收之圖片增加，再進行正常之解碼。
[自動行駛]When the user selects the linked image, the display device decodes the base layer with the highest priority. In addition, if there is information in the HTML constituting the web page indicating that it is adaptable content, the display device can also decode to the enhancement layer. In addition, in order to ensure the immediacy, when the front selection or the communication band is very strict, the display device decodes and displays only the forward reference pictures (I picture, P picture, and only forward reference B picture). Reduce the delay between the decoding time and the display time of the previous picture (the delay from the start of decoding the content to the start of displaying the content). In addition, the display device can also deliberately ignore the reference relationship of pictures, so that all B pictures and P pictures are roughly decoded as forward reference, and as time passes, the received pictures increase, and then normal decoding is performed.
[Automatic driving]

又，當接收發送用於車之自動行駛或支援行駛的2維或3維之地圖資訊等之靜止圖或影像資料的情況下，接收終端亦可以是除了接收屬於1個以上的層之圖像資料之外，還接收天候或施工之資訊等作為元資訊，並將這些附上對應關係而解碼。另，元資訊亦可以是屬於層，亦可以是單純與圖像資料多工。In addition, when receiving and transmitting still pictures or image data such as 2D or 3D map information used for automatic driving of the car or supporting driving, the receiving terminal may also receive images belonging to more than one layer. In addition to the data, it also receives weather or construction information as meta-information, and decodes these with a corresponding relationship. In addition, the meta-information can also belong to layers, or it can simply be multiplexed with image data.

此情況下，由於含有接收終端之車、無人機或飛機等會移動，故接收終端是將該接收終端之位置資訊在接收要求時發送，藉此，可一面切換基地台ex106~ex110，一面實現無縫之接收及解碼。又，接收終端可以因應使用者之選擇、使用者之狀況或通訊頻帶之狀態，而動態地切換將元資訊接收到何種程度、或將地圖資訊逐漸更新到何種程度。In this case, since the car, drone, or airplane containing the receiving terminal moves, the receiving terminal sends the location information of the receiving terminal when receiving the request, thereby enabling the base station ex106 to ex110 to be switched while switching Seamless reception and decoding. In addition, the receiving terminal can dynamically switch to what degree the meta-information is received or to which degree the map information is gradually updated according to the user's selection, the user's condition, or the state of the communication band.

如以上，在內容供給系統ex100，可令使用者發送之經過編碼的資訊即時地被客戶端接收而解碼、播放。
[個人內容之發布]As described above, in the content supply system ex100, the encoded information sent by the user can be received, decoded, and played by the client in real time.
[Public Content Release]

又，內容供給系統ex100並不僅止於影像發布業者之高畫質、長時間之內容，亦可以進行個人之低畫質、短時間之內容之單播或多播發布。又，可想到的是，如此之個人之內容以後亦會逐漸增加。為了令個人內容成為更佳之內容，伺服器亦可以在進行編輯處理後才進行編碼處理。關於此，例如，可藉由如以下之構成而實現。In addition, the content supply system ex100 is not limited to the high-quality and long-term content of the video distribution industry, but also can be used for unicast or multicast distribution of personal low-quality and short-term content. Also, it is conceivable that such personal content will gradually increase in the future. In order to make personal content better, the server can also perform encoding processing after editing. This can be achieved by, for example, the following configuration.

在拍攝時即時或累積而於拍攝後，伺服器從原圖或已編碼資料進行拍攝錯誤、場景估計、意義解析、及物件檢測等之辨識處理。然後，伺服器基於辨識結果而手動或自動地進行以下編輯：修正失焦或手晃等，將明度比其他圖片低或焦點不合之場景等重要性低的場景刪除，強調物件之邊緣，令色調變化等。伺服器基於編輯結果而將編輯後之資料編碼。又，拍攝時刻過長則收視率會下降是已為人知，伺服器亦可以因應拍攝時間，以成為特定時間範圍內之內容的方式，除了如上述般之重要性低之場景之外，還將運動少之場景等基於圖像處理結果而自動地剪去。或者，伺服器亦可以基於場景之意義解析之結果而生成摘要並編碼。Immediately or cumulatively when shooting, and after shooting, the server performs recognition processing for shooting errors, scene estimation, meaning analysis, and object detection from the original image or encoded data. Then, the server manually or automatically performs the following editing based on the recognition results: correction of out-of-focus or hand shake, deletion of less important scenes such as scenes with lower brightness than other pictures, or out-of-focus scenes, emphasizing the edges of objects, and making color Change, etc. The server encodes the edited data based on the editing results. In addition, it is known that the shooting rate will decrease when the shooting time is too long. The server can also respond to the shooting time to become a content within a specific time range. In addition to the scenes of low importance as described above, Scenes with little movement are automatically clipped based on image processing results. Alternatively, the server may generate and encode the digest based on the result of the meaning analysis of the scene.

另，個人內容有時會拍到維持原樣的話會侵害著作權、著作人格權或肖像權等的東西，亦可能發生共用之範圍超過意圖之範圍等對個人而言不便的情況。所以，例如，伺服器亦可以刻意將畫面之周邊部之人臉或家中等變更成焦點不合之圖像而進行編碼。又，伺服器亦可以辨識在編碼對象圖像內是否有照到與事先登記之人物不同之人物的臉，當有照到的情況下，進行在臉之部分加上馬賽克等之處理。或者，亦可以作為編碼之前處理或後處理，由著作權等之觀點，讓使用者指定圖像中想加工之人物或背景區域，伺服器則進行將指定之區域換成別的影像、或令焦點模糊等之處理。若是人物，可在動態圖像中一面追蹤人物、一面將臉之部分之影像換掉。In addition, personal content may be captured in a way that would infringe copyright, personality, or portrait rights, and inconvenience may be caused to individuals, such as when the scope of sharing exceeds the scope of intention. Therefore, for example, the server can also intentionally change the face or home in the periphery of the screen into an out-of-focus image for encoding. In addition, the server can also recognize whether a face of a person different from a previously registered person is photographed in the encoding target image, and if it is photographed, perform processing such as adding a mosaic to the face portion. Alternatively, it can also be used as a pre- or post-coding process. From the viewpoint of copyright, the user can specify the person or background area in the image to be processed, and the server can replace the specified area with another image or focus. Blur processing. If it is a person, you can track the person in the moving image and change the image of the face part.

又，由於資料量小之個人內容之收看有較強之即時性之要求，故雖然還要考慮頻帶寬，但解碼裝置首先是最優先接收基礎層而進行解碼及播放。解碼裝置亦可以在此期間接收增強層，當循環播放的情況等播放2次以上的情況下，將增強層亦包含在內而播放高畫質之影像。若是進行如此之可適性編碼之串流，則可提供如下之體驗：雖然在未選擇時或剛開始看之階段是較粗糙之動態圖，但漸漸地串流變流暢、圖像變佳。除了可適性編碼之外，將第1次播放之較粗糙之串流、以及參考第1次之動態圖而編碼之第2次之串流當作1個串流而構成，亦可以提供同樣之體驗。
[其他之使用例]In addition, since the viewing of personal content with a small amount of data has a strong requirement for immediateness, although the frequency bandwidth must also be considered, the decoding device first receives the base layer for decoding and playback first. The decoding device can also receive the enhancement layer during this period. When the loop is played twice or more, the enhancement layer is included and the high-quality image is played. If such adaptively encoded streaming is provided, it can provide the following experience: Although it is a rough motion picture when it is not selected or at the beginning of viewing, the stream gradually becomes smooth and the image becomes better. In addition to the adaptive encoding, the rougher stream played at the first time and the second stream encoded with reference to the first motion picture are constructed as one stream, and the same can be provided. Experience.
[Other use cases]

又，該等編碼或解碼處理一般是在各終端所具有之LSIex500中處理。LSIex500可以是單晶片，亦可以是由複數晶片構成。另，亦可以是：將動態圖像編碼或解碼用之軟體裝進電腦ex111等可讀取之某些記錄媒體(CD-ROM、軟性磁碟、或硬碟等)中，使用該軟體進行編碼或解碼處理。再者，當智慧型手機ex115附有相機的情況下，亦可以將藉由該相機取得之動態圖資料發送。此時之動態圖資料是在智慧型手機ex115所具有之LSIex500中經過編碼處理之資料。In addition, these encoding or decoding processes are generally processed in the LSIex500 that each terminal has. The LSIex500 may be a single chip or a plurality of chips. In addition, it is also possible to install software for encoding or decoding moving images into some readable recording media (CD-ROM, flexible disk, or hard disk, etc.) such as computer ex111, and use the software to encode Or decode processing. In addition, when the smart phone ex115 is equipped with a camera, the moving picture data obtained by the camera can also be sent. The dynamic map data at this time is data processed by encoding in the LSIex500 of the smart phone ex115.

另，LSIex500亦可以是下載應用軟體而啟用之構成。此情況下，終端首先判定該終端是否對應於內容之編碼方式、或者是否有特定服務之執行能力。當終端未對應於內容之編碼方式的情況下、或者不具有特定服務之執行能力的情況下，終端下載編解碼器或應用軟體，之後進行內容取得及播放。In addition, the LSIex500 can be configured by downloading application software and enabling it. In this case, the terminal first determines whether the terminal corresponds to the encoding method of the content, or whether it has the execution capability of a specific service. When the terminal does not correspond to the encoding method of the content or does not have the ability to execute specific services, the terminal downloads a codec or application software and then obtains and plays the content.

又，並非限定於透過網際網路ex101之內容供給系統ex100，數位播送用系統中亦可組裝上述各實施形態之至少動態圖像編碼裝置(圖像編碼裝置)或動態圖像解碼裝置(圖像解碼裝置)之任一者。由於利用衛星等將影像與聲音經過多工之多工資料乘載在播送用之電波而發送接收，故雖然有相對於內容供給系統ex100是易於進行單播之構成，而其則是適合多播這樣的差異，但關於編碼處理及解碼處理是可同樣地應用。
[硬體構成]It is not limited to the content supply system ex100 via the Internet ex101, and at least a moving image encoding device (image encoding device) or a moving image decoding device (image Decoding device). Images and sounds are transmitted and received by multiplexing multiplexed data using satellites, etc., on radio waves for broadcasting. Therefore, although it is easier to unicast than the content supply system ex100, it is suitable for multicasting. Such a difference is similarly applicable to the encoding process and the decoding process.
[Hardware composition]

圖28是顯示智慧型手機ex115的圖。又，圖29是顯示智慧型手機ex115之構成例的圖。智慧型手機ex115具備：天線ex450，用於在與基地台ex110之間發送接收電波；相機部ex465，可拍攝影像及靜止圖；顯示部ex458，顯示相機部ex465所拍攝之影像、及藉由天線ex450接收之影像等經解碼之資料。智慧型手機ex115更具備：操作部ex466，即觸控面板等；聲音輸出部ex457，即用於輸出聲音或音響之揚聲器等；聲音輸入部ex456，即用於輸入聲音之麥克風等；記憶體部ex467，可保存拍攝之影像或靜止圖、錄音之聲音、接收之影像或靜止圖、郵件等經過編碼之資料或經過解碼之資料；插槽部ex464，即對於SIMex468的介面部，前述SIMex468是用來認出使用者、進行以網路為首之各種資料之存取的認證。另，亦可以使用外接之記憶體來取代記憶體部ex467。FIG. 28 is a diagram showing a smart phone ex115. FIG. 29 is a diagram showing a configuration example of a smartphone ex115. The smart phone ex115 includes: an antenna ex450 for transmitting and receiving radio waves to and from the base station ex110; a camera section ex465 for capturing images and still images; a display section ex458 for displaying images captured by the camera section ex465, and an antenna decoded data such as images received by ex450. The smart phone ex115 further includes: an operation unit ex466, which is a touch panel, etc .; a sound output unit ex457, which is a speaker for outputting sound or sound, etc .; a sound input unit ex456, which is a microphone for inputting sound, etc .; a memory unit. ex467, which can store captured or still images, recorded audio, received images or still images, mail, and other encoded or decoded data; slot part ex464, which is the interface of SIMex468, the aforementioned SIMex468 is used To identify users and authenticate access to various data including the Internet. Alternatively, an external memory can be used instead of the memory section ex467.

又，透過匯流排ex470而連接：主控制部ex460，將顯示部ex458及操作部ex466等總括地控制；電源電路部ex461；操作輸入控制部ex462；影像訊號處理部ex455；相機介面部ex463；顯示器控制部ex459；調變/解調部ex452；多工/分離部ex453；聲音訊號處理部ex454；插槽部ex464；及記憶體部ex467。In addition, it is connected via a bus ex470: the main control unit ex460 controls the display unit ex458 and the operation unit ex466; the power supply circuit unit ex461; the operation input control unit ex462; the image signal processing unit ex455; The control section ex459; the modulation / demodulation section ex452; the multiplexing / demultiplexing section ex453; the audio signal processing section ex454; the slot section ex464; and the memory section ex467.

電源電路部ex461是當因為使用者之操作而令電源鍵成為開啟狀態時，藉由從電池組對各部供給電力，而將智慧型手機ex115啟動成可動作之狀態。When the power supply circuit part ex461 is turned on due to a user's operation, the smart phone ex115 is activated to operate by supplying power from the battery pack to each part.

智慧型手機ex115是基於具有CPU、ROM及RAM等的主控制部ex460之控制而進行通話及資料通訊等之處理。通話時是將在聲音輸入部ex456收音之聲音訊號以聲音訊號處理部ex454轉換成數位聲音訊號，以調變/解調部ex452對其進行展頻處理，以發送/接收部ex451施加數位類比轉換處理及頻率轉換處理後透過天線ex450而發送。又，將接收資料放大而施加頻率轉換處理及類比數位轉換處理，以調變/解調部ex452進行解展頻處理，以聲音訊號處理部ex454轉換成類比聲音訊號後，將其從聲音輸出部ex457輸出。資料通訊模式時是藉由本體部之操作部ex466等之操作而令文字、靜止圖、或影像資料透過操作輸入控制部ex462朝主控制部ex460送出，同樣地進行發送接收處理。在資料通訊模式時發送影像、靜止圖、或影像與聲音的情況下，影像訊號處理部ex455是藉由在上述各實施形態顯示之動態圖像編碼方法而對保存在記憶體部ex467之影像訊號或從相機部ex465輸入之影像訊號進行壓縮編碼，將經過編碼之影像資料朝多工/分離部ex453送出。又，聲音訊號處理部ex454將藉由相機部ex465拍攝影像或靜止圖等時以聲音輸入部ex456收音之聲音訊號編碼，將經過編碼之聲音資料朝多工/分離部ex453送出。多工/分離部ex453將已編碼影像資料與已編碼聲音資料以預定之方式進行多工，以調變/解調部(調變/解調電路部)ex452、及發送/接收部ex451施加調變處理及轉換處理，透過天線ex450發送。The smart phone ex115 performs processing such as calling and data communication based on the control of the main control unit ex460 including the CPU, ROM, and RAM. During a call, the sound signal received in the sound input section ex456 is converted into a digital sound signal by the sound signal processing section ex454, and the modulation / demodulation section ex452 is used for spread spectrum processing, and the sending / receiving section ex451 applies digital analog conversion. After processing and frequency conversion processing, it is transmitted through the antenna ex450. In addition, the received data is amplified and subjected to frequency conversion processing and analog digital conversion processing. The modulation / demodulation unit ex452 performs despreading processing, and the audio signal processing unit ex454 converts the analog data to an analog audio signal, and then removes it from the audio output unit ex457 output. In the data communication mode, text, still images, or image data are sent to the main control unit ex460 through the operation input control unit ex462 through operations of the operation unit ex466 and the like of the main body unit, and the same process is performed for transmission and reception. In the case of sending an image, a still image, or an image and sound in the data communication mode, the image signal processing unit ex455 uses the moving image coding method displayed in each of the above embodiments to process the image signal stored in the memory portion ex467. Or the image signal input from the camera part ex465 is compression-encoded, and the encoded image data is sent to the multiplexing / separation part ex453. In addition, the sound signal processing unit ex454 encodes the sound signal received by the sound input unit ex456 when the camera unit ex465 captures an image or a still image, and sends the encoded sound data to the multiplexing / separation unit ex453. The multiplexing / separating section ex453 multiplexes the encoded video data and the encoded audio data in a predetermined manner, and applies modulation to the modulation / demodulation section (modulation / demodulation circuit section) ex452 and the transmission / reception section ex451. Variable processing and conversion processing are transmitted through the antenna ex450.

當接收到附加在電子郵件或聊天的影像、或連結於網頁等的影像的情況下，為了將透過天線ex450而接收到之多工資料解碼，多工/分離部ex453藉由分離多工資料而將多工資料分成影像資料之位元串流與聲音資料之位元串流，透過同步匯流排ex470而將經過編碼之影像資料朝影像訊號處理部ex455供給，並將經過編碼之聲音資料朝聲音訊號處理部ex454供給。影像訊號處理部ex455是藉由與在上述各實施形態顯示之動態圖像編碼方法對應之動態圖像解碼方法而將影像訊號解碼，透過顯示器控制部ex459而從顯示部ex458顯示連結之動態圖像檔案所含有之影像或靜止圖。又，聲音訊號處理部ex454將聲音訊號解碼，從聲音輸出部ex457輸出聲音。另，由於即時串流普及，故隨著使用者之狀況，亦可能發生就社會性而言播放聲音並不恰當的情況。因此，宜採用如下之構成：作為初期值，不播放聲音訊號，只播放影像資料。亦可以是只在使用者進行了點擊影像資料等操作的情況下，才將聲音同步播放。When receiving an image attached to an e-mail or chat, or an image linked to a web page, in order to decode the multiplexed data received through the antenna ex450, the multiplexing / separating unit ex453 separates the multiplexed data to The multiplexed data is divided into a bit stream of image data and a bit stream of sound data, and the encoded image data is supplied to the image signal processing unit ex455 through the synchronous bus ex470, and the encoded sound data is directed toward the sound. The signal processing unit ex454 is supplied. The video signal processing unit ex455 decodes the video signal by a video decoding method corresponding to the video coding method displayed in each of the above embodiments, and displays the linked video from the display unit ex458 through the display control unit ex459. The image or still image contained in the file. The audio signal processing unit ex454 decodes the audio signal and outputs audio from the audio output unit ex457. In addition, due to the popularity of real-time streaming, depending on the user's situation, it may happen that the sound is not appropriate for social reasons. Therefore, the following configuration should be adopted: As an initial value, no audio signal is played and only the video data is played. It is also possible to play the sound synchronously only when the user has performed operations such as clicking on the image data.

又，雖然在此是以智慧型手機ex115來舉例說明，但關於終端，可以想到如下之3種安裝形式：除了具有編碼器及解碼器雙方之發送接收型終端之外，還有只具有編碼器之發送終端、及、只具有解碼器之接收終端。再者，雖然在數位播送用系統中說明的是接收或發送已將聲音資料等多工在影像資料之多工資料，但亦可以令多工資料是除了聲音資料之外將與影像相關之文字資料等多工，亦可以令接收或發送的是影像資料本身而非多工資料。In addition, although the smart phone ex115 is used as an example here, as for the terminal, the following three types of installations are conceivable: In addition to the transmitting and receiving terminal having both an encoder and a decoder, there are only encoders. A transmitting terminal, and a receiving terminal having only a decoder. Furthermore, although the digital broadcasting system is described as receiving or transmitting multiplexed data that has multiplexed audio data, etc. into video data, it is also possible to make the multiplexed data be text that is related to the image in addition to audio data The multiplexing of data, etc., can also cause the image data itself to be received or sent instead of the multiplexed data.

另，雖然說明的是含有CPU之主控制部ex460控制編碼或解碼處理，但終端常常具備GPU。所以，亦可以是如下之構成：藉由在CPU與GPU共通化之記憶體、或者、以可共通使用的方式管理位址之記憶體，活用GPU之性能而一併處理寬廣的區域。藉此，可縮短編碼時間，確保即時性，可實現低延遲。尤其，若不是藉由CPU，而是藉由GPU將運動估計、解塊濾波、SAO(Sample Adaptive Offset，取樣自適應偏移)、及轉換/量化之處理以圖片等單位一併進行，則有效率。
產業上之可利用性In addition, although it is explained that the main control unit ex460 including the CPU controls the encoding or decoding process, the terminal often includes a GPU. Therefore, it can be configured as follows: by using a memory common to the CPU and the GPU, or a memory that manages addresses in a way that can be used in common, the performance of the GPU is used to process a wide area together. This can reduce encoding time, ensure immediateness, and achieve low latency. In particular, if the motion estimation, deblocking filtering, SAO (Sample Adaptive Offset), and conversion / quantization processing are performed in units of pictures, etc., not by the CPU, but by the GPU effectiveness.
Industrial availability

本揭示例如可用在電視機、數位視訊記錄器、汽車導航、行動電話、數位相機、數位視訊攝影機、電視會議系統、或電子鏡等。The present disclosure can be used in, for example, a television, a digital video recorder, a car navigation system, a mobile phone, a digital camera, a digital video camera, a video conference system, or an electronic mirror.

10~23‧‧‧區塊10 ~ 23‧‧‧block

100‧‧‧編碼裝置 100‧‧‧ encoding device

102‧‧‧分割部 102‧‧‧Division

104‧‧‧相減部 104‧‧‧Subtraction Division

106‧‧‧轉換部 106‧‧‧ Conversion Department

108‧‧‧量化部 108‧‧‧Quantitative Department

110‧‧‧熵編碼部 110‧‧‧Entropy coding department

112、204‧‧‧反量化部 112, 204‧‧‧Dequantization Department

114、206‧‧‧反轉換部 114, 206‧‧‧Anti-Conversion Department

116、208‧‧‧相加部 116, 208‧‧‧ Addition Department

118、210‧‧‧區塊記憶體 118, 210‧‧‧ block memory

120、212‧‧‧迴路濾波部 120, 212‧‧‧loop filtering department

122、214‧‧‧幀記憶體 122, 214‧‧‧frame memory

124、216‧‧‧幀內預測部 124, 216‧‧‧Intra prediction department

126、218‧‧‧幀間預測部 126, 218‧‧‧Inter prediction department

128、220‧‧‧預測控制部 128, 220‧‧‧ Predictive Control Department

160、260‧‧‧電路 160, 260‧‧‧circuit

162、262‧‧‧記憶體 162, 262‧‧‧Memory

200‧‧‧解碼裝置 200‧‧‧ decoding device

202‧‧‧熵解碼部 202‧‧‧Entropy Decoding Department

CU0~CU4‧‧‧編碼單元 CU0 ~ CU4‧‧‧‧coding unit

ex100‧‧‧內容供給系統 ex100‧‧‧Content Supply System

ex101‧‧‧網際網路 ex101‧‧‧Internet

ex102‧‧‧網際網路服務供應商 ex102‧‧‧Internet Service Provider

ex103‧‧‧串流伺服器 ex103‧‧‧streaming server

ex104‧‧‧通訊網 ex104‧‧‧Communication Network

ex106~ex110‧‧‧基地台 ex106 ~ ex110‧‧‧Base Station

ex111‧‧‧電腦 ex111‧‧‧Computer

ex112‧‧‧遊戲機 ex112‧‧‧Game console

ex113‧‧‧相機 ex113‧‧‧Camera

ex114‧‧‧家電 ex114‧‧‧Household appliances

ex115‧‧‧智慧型手機 ex115‧‧‧Smartphone

ex116‧‧‧衛星 ex116‧‧‧ satellite

ex117‧‧‧飛機 ex117‧‧‧plane

ex450‧‧‧天線 ex450‧‧‧antenna

ex451‧‧‧發送/接收部 ex451‧‧‧Sending / Receiving Department

ex452‧‧‧調變/解調部 ex452‧‧‧Modulation / Demodulation Department

ex453‧‧‧多工/分離部 ex453‧‧‧Multiplexing / Separation Department

ex454‧‧‧聲音訊號處理部 ex454‧‧‧Sound Signal Processing Department

ex455‧‧‧影像訊號處理部 ex455‧‧‧Image Signal Processing Department

ex456‧‧‧聲音輸入部 ex456‧‧‧Voice input section

ex457‧‧‧聲音輸出部 ex457‧‧‧Sound output

ex458‧‧‧顯示部 ex458‧‧‧Display

ex459‧‧‧LCD控制部 ex459‧‧‧LCD Control Department

ex460‧‧‧主控制部 ex460‧‧‧Main Control Department

ex461‧‧‧電源電路部 ex461‧‧‧Power circuit department

ex462‧‧‧操作輸入控制部 ex462‧‧‧Operation input control section

ex463‧‧‧相機I/F部 ex463‧‧‧Camera I / F

ex464‧‧‧插槽部 ex464‧‧‧Slot

ex465‧‧‧相機部 ex465‧‧‧Camera Department

ex466‧‧‧操作部 ex466‧‧‧Operation Department

ex467‧‧‧記憶體部 ex467‧‧‧Memory

ex468‧‧‧SIM ex468‧‧‧SIM

ex470‧‧‧匯流排 ex470‧‧‧Bus

ex500‧‧‧LSI ex500‧‧‧LSI

S101~S111、S201~S211、S301、S401‧‧‧處理 S101 ~ S111, S201 ~ S211, S301, S401‧‧‧Process

圖1是顯示實施形態1之編碼裝置之功能構成的方塊圖。FIG. 1 is a block diagram showing a functional configuration of an encoding device according to the first embodiment.

圖2是顯示實施形態1之區塊分割之一例的圖。FIG. 2 is a diagram showing an example of block division in the first embodiment.

圖3是顯示與各轉換類型對應之轉換基底函數的表。FIG. 3 is a table showing conversion basis functions corresponding to each conversion type.

圖4A是顯示在ALF使用之濾波器之形狀之一例的圖。FIG. 4A is a diagram showing an example of the shape of a filter used in ALF.

圖4B是顯示在ALF使用之濾波器之形狀之其他之一例的圖。FIG. 4B is a diagram showing another example of the shape of a filter used in ALF.

圖4C是顯示在ALF使用之濾波器之形狀之其他之一例的圖。FIG. 4C is a diagram showing another example of the shape of a filter used in ALF.

圖5A是顯示幀內預測之67個幀內預測模式的圖。FIG. 5A is a diagram showing 67 intra prediction modes of intra prediction.

圖5B是用於說明利用OBMC處理之預測圖像修正處理之概要的流程圖。FIG. 5B is a flowchart for explaining the outline of a prediction image correction process using OBMC processing.

圖5C是用於說明利用OBMC處理之預測圖像修正處理之概要的概念圖。FIG. 5C is a conceptual diagram for explaining an outline of a prediction image correction process using OBMC processing.

圖5D是顯示FRUC之一例的圖。FIG. 5D is a diagram showing an example of FRUC.

圖6是用於說明在延著運動軌跡之2個區塊間之型樣匹配(雙向匹配)的圖。FIG. 6 is a diagram for explaining pattern matching (two-way matching) between two blocks extending along a motion trajectory.

圖7是用於說明在當前圖片內之模板與參考圖片內之區塊之間之型樣匹配(模板匹配)的圖。FIG. 7 is a diagram for explaining pattern matching (template matching) between a template in a current picture and a block in a reference picture.

圖8是用於說明假設等速直線運動之模型的圖。FIG. 8 is a diagram for explaining a model that assumes constant velocity linear motion.

圖9A是用於說明基於複數個鄰接區塊之運動向量而導出子區塊單位之運動向量的圖。FIG. 9A is a diagram for explaining that a motion vector of a sub-block unit is derived based on motion vectors of a plurality of adjacent blocks.

圖9B是用於說明利用合併模式之運動向量導出處理之概要的圖。FIG. 9B is a diagram for explaining the outline of the motion vector derivation process using the merge mode.

圖10是顯示實施形態1之解碼裝置之功能構成的方塊圖。Fig. 10 is a block diagram showing a functional configuration of a decoding device according to the first embodiment.

圖11是顯示第1具體例之管線構造的概略圖。FIG. 11 is a schematic diagram showing a pipeline structure of a first specific example.

圖12是顯示使用在管線處理之說明之區塊分割例的示意圖。FIG. 12 is a diagram showing an example of block division used in the description of pipeline processing.

圖13是顯示第1具體例之處理時機之一例的時序圖。FIG. 13 is a timing chart showing an example of processing timing of the first specific example.

圖14是顯示第1具體例之幀間預測之動作的流程圖。FIG. 14 is a flowchart showing the operation of inter prediction in the first specific example.

圖15是顯示第2具體例之管線構造的概略圖。FIG. 15 is a schematic diagram showing a pipeline structure of a second specific example.

圖16是顯示第2具體例之處理時機之一例的時序圖。FIG. 16 is a timing chart showing an example of processing timing of the second specific example.

圖17是顯示第2具體例之幀間預測之動作的流程圖。FIG. 17 is a flowchart showing the operation of inter prediction in the second specific example.

圖18是顯示第3具體例之幀間預測之動作的流程圖。FIG. 18 is a flowchart showing the operation of inter prediction in the third specific example.

圖19是顯示實施形態1之編碼裝置之安裝例的方塊圖。Fig. 19 is a block diagram showing an installation example of the encoding device of the first embodiment.

圖20是顯示實施形態1之編碼裝置之動作例的流程圖。Fig. 20 is a flowchart showing an example of the operation of the encoding device of the first embodiment.

圖21是顯示實施形態1之解碼裝置之安裝例的方塊圖。Fig. 21 is a block diagram showing an installation example of the decoding device of the first embodiment.

圖22是顯示實施形態1之解碼裝置之動作例的流程圖。Fig. 22 is a flowchart showing an operation example of the decoding device according to the first embodiment.

圖23是將內容發布服務實現之內容供給系統的整體構成圖。FIG. 23 is an overall configuration diagram of a content supply system implementing a content distribution service.

圖24是顯示可適性編碼時之編碼構造之一例的圖。FIG. 24 is a diagram showing an example of a coding structure during adaptive coding.

圖25是顯示可適性編碼時之編碼構造之一例的圖。FIG. 25 is a diagram showing an example of a coding structure during adaptive coding.

圖26是顯示web頁面之顯示畫面例的圖。FIG. 26 is a diagram showing an example of a display screen displaying a web page.

圖27是顯示web頁面之顯示畫面例的圖。FIG. 27 is a diagram showing an example of a display screen displaying a web page.

圖28是顯示智慧型手機之一例的圖。FIG. 28 is a diagram showing an example of a smartphone.

圖29是顯示智慧型手機之構成例的方塊圖。FIG. 29 is a block diagram showing a configuration example of a smartphone.

Claims

一種編碼裝置，是在構成動態圖像之編碼對象區塊之編碼中進行幀間預測之編碼裝置，前述編碼裝置具備：電路；及記憶體，前述電路是使用前述記憶體，當進行使用前述編碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述編碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述編碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。An encoding device is an encoding device that performs inter prediction in encoding of a coding target block constituting a moving image. The foregoing encoding device includes: Electrical circuits; and Memory, The aforementioned circuit uses the aforementioned memory, When the first process of correcting the predicted image of the coding target block using the brightness of the coding target block's periphery and the reference block's surrounding luminance is performed, one or more processes including the second process are not performed. Processing to perform inter prediction of the encoding target block, and the second processing is to correct the predicted image using a spatial gradient of luminance obtained from the reference block.

如請求項1之編碼裝置，其中前述1個以上之處理包含第3處理，前述第3處理是使用參考圖像來修正前述預測圖像，且前述參考圖像是使用前述編碼對象區塊之周邊之已編碼區塊之運動向量而從前述編碼對象區塊指出之參考圖像。For example, the encoding device of claim 1, wherein the one or more processes include a third process, and the third process is to modify the predicted image using a reference image, and the reference image uses a periphery of the encoding target block. The motion vector of the encoded block is the reference image pointed out from the aforementioned encoding target block.

如請求項1或2之編碼裝置，其中前述1個以上之處理是修正前述預測圖像之複數個處理中之與前述第1處理不同之所有處理。For example, the encoding device of claim 1 or 2, wherein the one or more processes described above are all processes different from the first process among the plurality of processes for correcting the predicted image.

如請求項1或2之編碼裝置，其中前述電路在進行前述第1處理後，以不修正前述預測圖像的方式而結束前述編碼對象區塊之幀間預測。For example, the encoding device according to claim 1 or 2, wherein after performing the first processing, the circuit ends the inter prediction of the encoding target block without modifying the prediction image.

如請求項1或2之編碼裝置，其中前述電路判定是否以低延遲模式進行前述編碼對象區塊之編碼，當在判定成以前述低延遲模式進行前述編碼對象區塊之編碼的情況下進行前述第1處理的情況下，不進行前述1個以上之處理而進行前述編碼對象區塊之幀間預測。For example, the encoding device of claim 1 or 2, wherein the circuit determines whether to encode the encoding target block in a low-latency mode, and performs the foregoing when determining to encode the encoding target block in the low-latency mode. In the case of the first process, the inter prediction of the coding target block is performed without performing one or more of the aforementioned processes.

如請求項5之編碼裝置，其中前述電路將表示是否以前述低延遲模式進行前述編碼對象區塊之編碼之資訊，編碼在前述動態圖像之編碼串流之序列標頭區域、圖片標頭區域、圖塊標頭區域、片段標頭區域及補助資訊區域之至少其中1者。For example, the encoding device of claim 5, wherein the aforementioned circuit will indicate whether to encode the encoding target block in the aforementioned low-latency mode, and encode the information in the sequence header area and picture header area of the encoding stream of the moving image. , At least one of the tile header area, the fragment header area, and the subsidy information area.

如請求項5之編碼裝置，其中前述電路是依循包含前述編碼對象區塊之編碼對象圖片的大小，而判定是否以前述低延遲模式進行前述編碼對象區塊之編碼。For example, the encoding device of claim 5, wherein the aforementioned circuit determines whether to encode the encoding target block in the aforementioned low-latency mode according to the size of the encoding target picture including the encoding target block.

如請求項5之編碼裝置，其中前述電路是依循解碼裝置之處理能力，而判定是否以前述低延遲模式進行前述編碼對象區塊之編碼。For example, if the encoding device of item 5 is requested, the aforementioned circuit is based on the processing capability of the decoding device, and determines whether to encode the aforementioned encoding target block in the aforementioned low-latency mode.

如請求項5之編碼裝置，其中前述電路是依循分配給前述動態圖像之編碼串流之設定檔或等級資訊，而判定是否以前述低延遲模式進行前述編碼對象區塊之編碼。For example, the encoding device according to claim 5, wherein the circuit is configured to determine whether to encode the encoding target block in the low-latency mode according to the profile or level information of the encoding stream allocated to the moving image.

一種解碼裝置，是在構成動態圖像之解碼對象區塊之解碼中進行幀間預測之解碼裝置，前述解碼裝置具備：電路；及記憶體，前述電路是使用前述記憶體，當進行使用前述解碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述解碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述解碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。A decoding device is a decoding device that performs inter prediction in decoding of a decoding target block constituting a moving image. The foregoing decoding device includes: Electrical circuits; and Memory, The aforementioned circuit uses the aforementioned memory, When performing the first process of correcting the predicted image of the decoding target block using the brightness of the surroundings of the decoding target block and the brightness of the reference block, the first processing including the second processing is not performed Processing to perform inter prediction of the decoding target block, and the second processing is to correct the predicted image using a spatial gradient of luminance obtained from the reference block.

如請求項10之解碼裝置，其中前述1個以上之處理包含第3處理，前述第3處理是使用參考圖像來修正前述預測圖像，且前述參考圖像是使用前述解碼對象區塊之周邊之已解碼區塊之運動向量而從前述解碼對象區塊指出之參考圖像。For example, the decoding device of claim 10, wherein the one or more processes include a third process, and the third process uses a reference image to modify the predicted image, and the reference image uses a periphery of the decoding target block. The motion vector of the decoded block is the reference image pointed out from the aforementioned decoding target block.

如請求項10或11之解碼裝置，其中前述1個以上之處理是修正前述預測圖像之複數個處理中之與前述第1處理不同之所有處理。For example, the decoding device of claim 10 or 11, wherein the one or more processes described above are all processes different from the first process among the plurality of processes for correcting the predicted image.

如請求項10或11之解碼裝置，其中前述電路在進行前述第1處理後，以不修正前述預測圖像的方式而令前述解碼對象區塊之幀間預測結束。For example, the decoding device of claim 10 or 11, wherein after performing the first processing, the circuit ends the inter prediction of the decoding target block without modifying the predicted image.

如請求項10或11之解碼裝置，其中前述電路判定是否以低延遲模式進行前述解碼對象區塊之解碼，當在判定成以前述低延遲模式進行前述解碼對象區塊之解碼的情況下進行前述第1處理的情況下，不進行前述1個以上之處理而進行前述解碼對象區塊之幀間預測。For example, the decoding device of claim 10 or 11, wherein the circuit determines whether to decode the decoding target block in a low-latency mode, and performs the foregoing when determining to decode the decoding target block in the low-latency mode. In the case of the first process, the inter prediction of the decoding target block is performed without performing one or more of the aforementioned processes.

如請求項14之解碼裝置，其中前述電路從前述動態圖像之編碼串流之序列標頭區域、圖片標頭區域、圖塊標頭區域、片段標頭區域及補助資訊區域之至少其中1者，解碼出表示是否以前述低延遲模式進行前述解碼對象區塊之解碼之資訊。The decoding device according to claim 14, wherein the aforementioned circuit selects at least one of a sequence header area, a picture header area, a tile header area, a fragment header area, and a subsidy information area from the encoding stream of the moving image. , Decoding information indicating whether the decoding of the decoding target block is performed in the aforementioned low-latency mode.

如請求項14之解碼裝置，其中前述電路是依循分配給前述動態圖像之編碼串流之設定檔或等級資訊，而判定是否以前述低延遲模式進行前述解碼對象區塊之解碼。For example, the decoding device of claim 14, wherein the aforementioned circuit determines whether to decode the decoding target block in the aforementioned low-latency mode according to the profile or level information of the encoded stream allocated to the aforementioned moving image.

一種編碼方法，是在構成動態圖像之編碼對象區塊之編碼中進行幀間預測之編碼方法，當進行使用前述編碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述編碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述編碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。A coding method is a coding method for performing inter prediction in coding of a coding target block constituting a moving image. When the first process of correcting the predicted image of the coding target block using the brightness of the coding target block's periphery and the reference block's surrounding luminance is performed, one or more processes including the second process are not performed. Processing to perform inter prediction of the encoding target block, and the second processing is to correct the predicted image using a spatial gradient of luminance obtained from the reference block.

一種解碼方法，是在構成動態圖像之解碼對象區塊之解碼中進行幀間預測之解碼方法，當進行使用前述解碼對象區塊之周邊之亮度與參考區塊之周邊之亮度來修正前述解碼對象區塊之預測圖像之第1處理的情況下，不進行包含第2處理之1個以上之處理而進行前述解碼對象區塊之幀間預測，前述第2處理是使用從前述參考區塊獲得之亮度之空間上的梯度來修正前述預測圖像。A decoding method is a decoding method for performing inter prediction in decoding of a decoding target block constituting a moving image. In the case of performing the first process of correcting the predicted image of the decoding target block using the brightness of the surroundings of the decoding target block and the brightness of the reference block, the processing including one or more of the second processing is not performed. Processing to perform inter prediction of the decoding target block, and the second processing is to correct the predicted image using a spatial gradient of luminance obtained from the reference block.