WO2013076888A1 - 画像処理装置および画像処理方法 - Google Patents
画像処理装置および画像処理方法 Download PDFInfo
- Publication number
- WO2013076888A1 WO2013076888A1 PCT/JP2012/005256 JP2012005256W WO2013076888A1 WO 2013076888 A1 WO2013076888 A1 WO 2013076888A1 JP 2012005256 W JP2012005256 W JP 2012005256W WO 2013076888 A1 WO2013076888 A1 WO 2013076888A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- processing
- image
- unit
- prediction
- image processing
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/129—Scanning of coding units, e.g. zig-zag scan of transform coefficients or flexible macroblock ordering [FMO]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/157—Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/18—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a set of transform coefficients
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/12—Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
- H04N19/122—Selection of transform size, e.g. 8x8 or 2x4x8 DCT; Selection of sub-band transforms of varying structure or type
Definitions
- the present invention relates to an image processing apparatus that executes a process for decoding an image encoded by an encoding process including a prediction process.
- Patent Document 1 As a technique for decoding an image encoded by an encoding process including a prediction process, there are techniques described in Patent Document 1, Non-Patent Document 1, and Non-Patent Document 2.
- delays may occur in the process for decoding the encoded image.
- the present invention provides an image processing apparatus capable of reducing a delay generated in a process for decoding an encoded image.
- An image processing apparatus is a code by encoding processing including prediction processing for one or more prediction units and frequency conversion processing for a plurality of transform units included in a region including the one or more prediction units.
- An image processing apparatus that executes a process for decoding an encoded image, and relates to predicted images of the one or more prediction units, depending on a predetermined conversion order for the plurality of conversion units
- a processing unit that performs image processing is provided.
- the image processing apparatus of the present invention can reduce the delay generated in the process for decoding the encoded image.
- FIG. 1 is a diagram showing division of a macroblock.
- FIG. 2 is a diagram showing the configuration of the image processing apparatus according to the first embodiment.
- FIG. 3 is a diagram showing a configuration related to motion compensation according to the first embodiment.
- FIG. 4A is a diagram showing an example of a sequence according to Embodiment 1.
- FIG. 4B is a diagram illustrating an example of a picture according to Embodiment 1.
- FIG. 4C is a diagram illustrating an example of a coded stream according to Embodiment 1.
- FIG. 5A is a diagram showing a configuration example of a coding unit and coding unit layer data according to Embodiment 1.
- FIG. 5B is a diagram showing an example of configuration of coding unit data according to Embodiment 1.
- FIG. 5A is a diagram showing a configuration example of a coding unit and coding unit layer data according to Embodiment 1.
- FIG. 5B is a diagram showing an example of configuration of coding unit data according to
- FIG. 6A is a diagram illustrating an example of the size of a prediction unit according to Embodiment 1.
- FIG. 6B is a diagram showing an example of the size of the conversion unit according to Embodiment 1.
- FIG. 7 is a flowchart showing the operation of the image processing apparatus according to the first embodiment.
- FIG. 8 is a flowchart showing a process of decoding a coding unit according to the first embodiment.
- FIG. 9 is a diagram showing an operation of motion compensation according to the first embodiment.
- FIG. 10A is a diagram showing a prediction unit and motion vectors according to Embodiment 1.
- FIG. 10B is a diagram showing division of a prediction unit according to Embodiment 1.
- FIG. 11 is a flowchart showing an operation of motion compensation according to the first embodiment. 12A] FIG.
- FIG. 12A is a diagram showing an example of configuration of a coding unit according to Embodiment 1.
- FIG. 12B is a diagram illustrating a configuration example of a prediction unit according to Embodiment 1.
- 12C is a diagram showing an example of configuration of a conversion unit according to Embodiment 1.
- FIG. 12D is a diagram illustrating a configuration example of blocks after division of a prediction unit according to Embodiment 1.
- FIG. 12E is a diagram illustrating an example of a configuration of blocks after division of the coding unit according to Embodiment 1.
- FIG. 13A is a time chart showing a first example of pipeline processing according to the first embodiment.
- FIG. 13B is a time chart showing a second example of pipeline processing according to the first embodiment.
- FIG. 14A is a diagram showing a prediction unit and motion vectors according to Embodiment 2.
- FIG. 14B is a diagram showing division of a prediction unit according to Embodiment 2.
- FIG. 15 is a flowchart showing an operation of motion compensation according to the second embodiment.
- FIG. 16A is a diagram illustrating a configuration example of a coding unit according to a second embodiment.
- FIG. 16B is a diagram illustrating a configuration example of a prediction unit according to Embodiment 2.
- FIG. 16C is a diagram showing an example configuration of a conversion unit according to Embodiment 2.
- FIG. 16D is a diagram illustrating a configuration example of blocks after division of a prediction unit according to Embodiment 2.
- FIG. 16A is a diagram showing a prediction unit and motion vectors according to Embodiment 2.
- FIG. 14B is a diagram showing division of a prediction unit according to Embodiment 2.
- FIG. 15 is a flowchart showing an operation of motion compensation according
- FIG. 16E is a diagram illustrating a configuration example of blocks after division of the coding unit according to Embodiment 2.
- FIG. 17A is a time chart showing a first example of pipeline processing according to the second embodiment.
- FIG. 17B is a time chart showing a second example of pipeline processing according to the second embodiment.
- FIG. 18 is a diagram showing a prediction unit and motion vectors according to Embodiment 3.
- FIG. 19 is a flowchart showing an operation of motion compensation according to the third embodiment.
- FIG. 20A is a diagram illustrating a configuration example of a coding unit according to a third embodiment.
- FIG. 20B is a diagram showing an example of a configuration of a prediction unit according to Embodiment 3.
- FIG. 20C is a diagram showing an exemplary configuration of a conversion unit according to Embodiment 3.
- FIG. 20D is a diagram showing a prediction order according to Embodiment 3.
- FIG. 20E is a diagram illustrating a configuration example of blocks after division of the coding unit according to Embodiment 3.
- FIG. 21A is a time chart showing a first example of pipeline processing according to the third embodiment.
- FIG. 21B is a time chart showing a second example of pipeline processing according to the third embodiment.
- FIG. 22 is a diagram showing a configuration related to in-plane prediction according to the fourth embodiment.
- FIG. 23 is a flowchart showing an operation of in-plane prediction according to the fourth embodiment.
- FIG. 24 is a diagram illustrating a prediction unit of in-plane prediction according to a fourth embodiment.
- FIG. 25A is a diagram showing the configuration of an image processing apparatus according to the fifth embodiment.
- FIG. 25B is a flowchart showing an operation of the image processing apparatus according to the fifth embodiment.
- FIG. 26A is a diagram showing the configuration of an image processing apparatus according to the sixth embodiment.
- FIG. 26B is a flowchart showing an operation of the image processing apparatus according to the sixth embodiment.
- FIG. 27A is a diagram showing the configuration of an image processing apparatus according to a seventh embodiment.
- FIG. 27B is a flowchart showing an operation of the image processing apparatus according to the seventh embodiment.
- FIG. 28A is a diagram illustrating a first example of a prediction order according to a seventh embodiment.
- FIG. 28B is a diagram illustrating a second example of the prediction order according to the seventh embodiment.
- FIG. 28C is a diagram illustrating a third example of the prediction order according to Embodiment 7.
- FIG. 28D is a diagram illustrating a fourth example of the prediction order according to Embodiment 7.
- FIG. 29A is a diagram showing the configuration of an image processing apparatus according to an eighth embodiment.
- FIG. 29B is a flowchart showing an operation of the image processing apparatus according to the eighth embodiment.
- FIG. 30A is a diagram showing the configuration of an image processing apparatus according to a ninth embodiment.
- FIG. 30B is a flowchart showing the operation of the image processing apparatus according to the ninth embodiment.
- FIG. 31A is a diagram showing the configuration of an image processing apparatus according to the tenth embodiment.
- FIG. 31B is a flowchart showing the operation of the image processing apparatus according to the tenth embodiment.
- FIG. 32A is a diagram showing the configuration of an image processing apparatus according to an eleventh embodiment.
- FIG. 32B is a flowchart showing an operation of the image processing apparatus according to the eleventh embodiment.
- FIG. 33A is a diagram showing the configuration of an image processing apparatus according to a twelfth embodiment.
- FIG. 33B is a flowchart showing an operation of the image processing apparatus according to the twelfth embodiment.
- FIG. 34 is an overall configuration diagram of a content supply system for realizing content distribution service.
- FIG. 35 is an overall configuration diagram of a digital broadcasting system.
- FIG. 36 is a block diagram showing a configuration example of a television.
- FIG. 37 is a block diagram showing a configuration example of an information reproducing / recording unit for reading and writing information on a recording medium which is an optical disk.
- FIG. 38 is a view showing an example of the structure of a recording medium which is an optical disc.
- FIG. 39 is a block diagram showing an example of the configuration of an integrated circuit for realizing the image decoding process.
- FIG. 40 is a block diagram showing an example of the configuration of an integrated circuit for realizing the image decoding process and the image encoding process.
- An image coding apparatus for coding an image divides each picture constituting the image into a plurality of macroblocks (also called Macroblock, sometimes abbreviated as MB) each of which is composed of 16 ⁇ 16 pixels. Then, the image coding apparatus codes each macroblock in raster scan order. An image coding apparatus generates a coded stream by coding and compressing an image. An image processing apparatus that decodes an image decodes this encoded stream for each macroblock in raster scan order, and reproduces each picture of the original image.
- macroblocks also called Macroblock, sometimes abbreviated as MB
- ITU-T H.264 There are H.264 standards (see, for example, Non-Patent Document 1).
- the image processing apparatus is described in the following. In order to decode an image encoded according to the H.264 standard, first, the encoded stream is read. Then, the image processing apparatus performs variable-length decoding after decoding various header information. The image processing apparatus inversely quantizes the coefficient information obtained by the variable length decoding and performs inverse frequency transform. Thereby, a difference image is generated.
- the image processing apparatus performs in-plane prediction or motion compensation according to the macroblock type obtained by variable-length decoding.
- motion compensation is performed on up to 16 ⁇ 16 pixels.
- the image processing apparatus generates a predicted image.
- the image processing apparatus performs a reconstruction process by adding the difference image to the predicted image.
- the image processing apparatus decodes the decoding target image by performing in-loop filter processing on the reconstructed image.
- the in-loop filter is a filter that is applied before the reconstructed image is stored in the frame memory as a reference image.
- a deblocking filter As an in-loop filter, a deblocking filter, a sample adaptive offset filter, and an active loop filter may be used.
- the filter applied at the time of display is called an out loop filter.
- the image coding apparatus codes an image in units of macroblocks configured by 16 ⁇ 16 pixels.
- 16 ⁇ 16 pixels are not always optimal as a unit of encoding.
- the image encoding apparatus can further improve the compression efficiency by increasing the unit of encoding.
- Non-Patent Document 2 the conventional H.
- the size of the coding unit block corresponding to the H.264 standard becomes variable.
- the image coding apparatus according to this technique can also code an image in a block larger than the conventional 16 ⁇ 16 pixels, and can appropriately code an ultra high definition image.
- a coding unit (CU: Coding Unit) is defined as a data unit of coding.
- This encoding unit is a data unit capable of switching between intra prediction performing intra prediction and inter prediction performing motion compensation, as in the case of a macroblock in a conventional encoding standard, and is the most basic of encoding. Is defined as a basic block.
- the size of this encoding unit is either 8x8 pixels, 16x16 pixels, 32x32 pixels, or 64x64 pixels.
- the largest coding unit is called the largest coding unit (LCU).
- a transform unit (TU: also referred to as a transform unit) is defined.
- This conversion unit is set to various rectangular sizes of 4 ⁇ 4 pixels or more, such as 4 ⁇ 4 pixels, 8 ⁇ 8 pixels, 16 ⁇ 16 pixels, 16 ⁇ 12 pixels, 32 ⁇ 32 pixels, and so on.
- a prediction unit is defined as a data unit of intra prediction or inter prediction.
- the prediction unit is set to various rectangular sizes of 4 ⁇ 4 pixels or more, such as 64 ⁇ 64 pixels, 64 ⁇ 32 pixels, 32 ⁇ 64 pixels, 32 ⁇ 32 pixels, 32 ⁇ 16 pixels, 16 pixels ⁇ 32 pixels, 16 ⁇ 12 pixels, etc. inside the encoding unit.
- Patent Document 1 As an image decoding device according to the H.264 standard, there is an image decoding device described in Patent Document 1.
- the image decoding apparatus according to Patent Document 1 divides a macroblock into a plurality of subblocks each configured to have a predetermined size, and performs motion compensation on each of the plurality of subblocks.
- FIG. 1 is a diagram showing division of a macroblock.
- the image decoding apparatus according to Patent Document 1 divides a 16 ⁇ 16 pixel macro block into a plurality of sub blocks each configured of 4 ⁇ 4 pixels. Then, the image decoding apparatus performs motion compensation processing on each of the plurality of sub blocks. As a result, regardless of the size of the macroblock, the data size of motion compensation becomes a predetermined size. Therefore, the operation processing of motion compensation is simplified.
- Non-Patent Document 2 when the encoding unit, the prediction unit, and the conversion unit are respectively configured in various sizes, pipeline processing including inverse frequency conversion processing, motion compensation processing, reconstruction processing, etc. Smooth operation may be hindered.
- the required processing time varies in each of the inverse frequency conversion process, the motion compensation process, the reconstruction process, and the like. This may cause unnecessary waiting time.
- the size of the transform unit is configured in various sizes.
- frequency conversion is performed at various sizes, it is difficult to perform inverse frequency conversion at a predetermined size. Therefore, even with the technology of Patent Document 1, the size and the shape of the data unit are not unified.
- Non-Patent Document 2 the occurrence of useless waiting time according to Non-Patent Document 2 is not resolved even by the technology of Patent Document 1. This useless waiting time increases the delay generated in the entire process.
- an image processing apparatus includes encoding processing including prediction processing for one or more prediction units and frequency conversion processing for a plurality of conversion units included in a region including the one or more prediction units.
- An image processing apparatus that executes a process for decoding an image encoded by the method, wherein the prediction of the one or more prediction units is performed depending on a predetermined conversion order for the plurality of conversion units.
- a processing unit is provided to execute image processing on an image.
- the processing unit may acquire the conversion order and execute the image processing in accordance with the acquired conversion order.
- the processing unit may execute the image processing in the conversion order for each block that is equal to or smaller than the conversion unit.
- image processing on the predicted image is performed for each conversion unit in accordance with the conversion order.
- image processing on the predicted image is performed for each block smaller than the transform unit so as to conform to the transform order. Therefore, image processing on the predicted image is appropriately performed in the conversion order.
- the processing unit may execute the processing of outputting the predicted image as the image processing along the conversion order.
- the processing unit may execute the image processing in accordance with the conversion order when the prediction order determined in advance for the one or more prediction units does not conform to the conversion order.
- the conversion order is used for image processing on the prediction image, not the predetermined prediction order. Therefore, pipeline processing is performed smoothly, and the delay generated in the entire processing is reduced.
- the image processing apparatus further includes a division unit that divides the one or more prediction units into a plurality of blocks along edges of the plurality of conversion units, and the processing unit further includes the one or more predictions.
- the image processing may be performed in the conversion order for each block obtained by dividing a unit.
- the division unit divides one prediction unit corresponding to the one or more prediction units into the plurality of blocks along the edges of the plurality of conversion units, and the processing unit further includes the one unit.
- the image processing may be performed along the conversion order for each block obtained by dividing a prediction unit.
- prediction units configured of regions of a plurality of transform units are also divided along the edges of these transform units. Therefore, image processing on the prediction image of such prediction unit is also appropriately performed along the conversion order.
- the division unit may set a plurality of prediction units included in the coding unit including the plurality of conversion units as the one or more prediction units along the edges of the plurality of conversion units to the plurality of blocks.
- the division may be performed, and the processing unit may execute the image processing in the conversion order for each block obtained by dividing the plurality of prediction units.
- the plurality of prediction units in the coding unit are divided according to the plurality of transform units. Therefore, image processing on the prediction image corresponding to the encoding unit is appropriately performed.
- the processing unit may execute, as the image processing, the processing of generating the predicted image and the processing of outputting the generated predicted image in the conversion order.
- the processing unit may perform a process of acquiring information for generating the predicted image, a process of generating the predicted image using the acquired information, and output the generated predicted image.
- the processing may be performed as the image processing along the conversion order.
- the process of acquiring information for generating a predicted image, the process of generating a predicted image, and the process of outputting a predicted image are performed in the order of conversion. Therefore, pipeline processing is performed smoothly, and the delay generated in the overall processing is reduced.
- the processing unit may execute, as the image processing, a process of outputting the predicted image generated by the motion compensation process in accordance with the conversion order.
- the processing unit may execute, as the image processing, the processing of generating the predicted image by performing the motion compensation processing and the processing of outputting the predicted image in the conversion order.
- the process of generating a predicted image by motion compensation and the process of outputting a predicted image generated by motion compensation are performed in the order of conversion. Therefore, pipeline processing is performed smoothly, and the delay generated in the entire processing is reduced.
- the processing unit may perform processing of acquiring information for performing motion compensation processing, and processing of generating the predicted image by executing the motion compensation processing using the acquired information.
- the process of outputting the generated predicted image may be performed as the image processing along the conversion order.
- processing for acquiring information for motion compensation, processing for generating a prediction image by motion compensation, and processing for outputting a prediction image generated by motion compensation are executed in the order of conversion. Therefore, pipeline processing is performed smoothly, and the delay generated in the entire processing is reduced.
- the processing unit may execute, as the image processing, a process of outputting the predicted image generated by the in-plane prediction process according to the conversion order.
- the processing unit executes, as the image processing, the processing of generating the predicted image by executing the in-plane prediction processing and the processing of outputting the generated predicted image in the conversion order.
- the processing unit is a process of acquiring information for executing the in-plane prediction process, and a process of generating the predicted image by executing the in-plane prediction process using the acquired information. And processing of outputting the generated predicted image may be performed as the image processing along the conversion order.
- processing for acquiring information for in-plane prediction, processing for generating a predicted image by in-plane prediction, and processing for outputting a predicted image generated by in-plane prediction are executed according to the conversion order. Be done. Therefore, pipeline processing is performed smoothly, and the delay generated in the entire processing is reduced.
- the processing unit may execute a process of reconstructing the image using the predicted image as the image process along the conversion order.
- the processing unit may further execute in-loop filter processing, which is filter processing on the reconstructed image, along the conversion order.
- the processing unit may further execute processing of storing the image subjected to the in-loop filter processing in a frame memory in the conversion order.
- the image processing apparatus decodes the image encoded by the encoding process including the in-plane prediction process of generating the prediction image of the prediction unit according to the prediction mode of the prediction unit And performing in-plane prediction on each of the plurality of blocks and a division unit that divides the prediction unit into a plurality of blocks according to the prediction mode of the prediction unit.
- the image processing apparatus may be provided with a processing unit.
- the image processing apparatus can perform in-plane prediction in small data units. Therefore, the image processing apparatus can smoothly execute pipeline processing in smaller data units. Thus, the image processing apparatus can reduce the delay occurring in the entire processing.
- expressions such as 64 ⁇ 64 pixels and 32 ⁇ 32 pixels mean sizes such as 64 pixels ⁇ 64 pixels and 32 pixels ⁇ 32 pixels, respectively.
- expressions such as a block, a data unit, a coding unit (CU) and the like respectively mean an integrated area. Each of them may mean an image area. Alternatively, they may mean data areas in the coded stream, respectively.
- the image may be any of a plurality of pictures, one picture, part of a picture, etc. constituting still pictures or moving pictures.
- Embodiment 1 (1-1. Overview) First, an outline of the image processing apparatus according to the present embodiment will be described.
- the image processing apparatus according to the present embodiment decodes a coded stream generated by coding processing including motion compensation.
- the size of motion compensation i.e. the size of the prediction unit, is variable, up to 64x64 pixels.
- the image processing apparatus divides the prediction unit into a plurality of blocks (divided blocks) at the edge of the conversion unit, when the edge of the conversion unit is included in the rectangle of the prediction unit for which motion compensation is performed. Then, the image processing apparatus performs reference image transfer processing, motion compensation processing, and prediction image output processing on the plurality of blocks obtained by division in the processing order (conversion order) of the conversion unit. The image processing apparatus also performs processing in the processing order of the conversion unit with respect to the subsequent reconstruction processing and in-loop filter processing.
- the image processing device divides the prediction unit and performs motion compensation in the processing order of the conversion unit. This reduces the required memory capacity.
- the image processing apparatus performs transfer processing of a reference image and processing of motion compensation in a pipelined manner. This reduces processing delays. Also, the image processing apparatus can start reconstruction processing and in-loop filter processing early. Therefore, the image processing apparatus can accelerate the entire decoding process.
- FIG. 2 is a block diagram of an image processing apparatus according to the present embodiment.
- the image processing apparatus according to the present embodiment includes a control unit 501, a frame memory 502, a reconstructed image memory 509, a variable length decoding unit 503, an inverse quantization unit 504, an inverse frequency conversion unit 505, a motion compensation unit 506, and
- the prediction unit 507, the reconstruction unit 508, the in-loop filter unit 510, and the motion vector calculation unit 511 are provided.
- a control unit 501 controls the entire image processing apparatus.
- the frame memory 502 is a memory for storing decoded image data.
- the reconstructed image memory 509 is a memory for storing a part of the generated reconstructed image.
- the variable length decoding unit 503 reads the encoded stream and decodes a variable length code.
- the inverse quantization unit 504 performs inverse quantization.
- the inverse frequency converter 505 performs inverse frequency conversion.
- the motion vector calculation unit 511 calculates a motion vector based on the predicted motion vector, the differential motion vector, and the like, and outputs the motion vector to the motion compensation unit 506.
- the motion compensation unit 506 reads a reference image from the frame memory 502, performs motion compensation, and generates a predicted image.
- the in-plane prediction unit 507 reads out the reference image from the reconstructed image memory 509, performs in-plane prediction (also referred to as intra prediction), and generates a predicted image.
- the reconstruction unit 508 adds the difference image and the predicted image to generate a reconstructed image, and stores a part of the reconstructed image in the reconstructed image memory 509.
- the in-loop filter unit 510 removes noise of the reconstructed image to improve the quality of the reconstructed image.
- FIG. 3 is a block diagram of the periphery of the motion compensation unit 506 according to the present embodiment.
- symbol is allocated to the same component as FIG. 2, and description is abbreviate
- FIG. 3 shows a DMA (Direct Memory Access) control unit 512, a reference image storage unit 513, and a predicted image storage unit 514. These may be included in the motion compensation unit 506.
- DMA Direct Memory Access
- the DMA control unit 512 transfers the reference image from the frame memory 502 to the reference image storage unit 513 based on the motion vector calculated by the motion vector calculation unit 511.
- the reference image storage unit 513 stores the reference image transferred by the DMA control unit 512.
- the predicted image storage unit 514 stores the predicted image generated by the motion compensation unit 506.
- the motion compensation unit 506 performs motion compensation based on the motion vector. At this time, the motion compensation unit 506 obtains, from the variable length decoding unit 503, information on the processing order of the conversion unit. Then, the motion compensation unit 506 performs motion compensation according to the processing order of the conversion unit to generate a predicted image. Thereafter, the motion compensation unit 506 stores the prediction image in the prediction image storage unit 514. The reconstruction unit 508 executes reconstruction processing using the predicted image stored in the predicted image storage unit 514.
- the coded stream decoded by the image processing apparatus according to the present embodiment is composed of a coding unit (CU), a transform unit (TU), and a prediction unit (PU).
- CU coding unit
- TU transform unit
- PU prediction unit
- the encoding unit is set with a size of 64 ⁇ 64 pixels to 8 ⁇ 8 pixels, and is a switchable data unit of intra prediction and inter prediction.
- the transform unit is set to a size of 64 ⁇ 64 pixels to 4 ⁇ 4 pixels in an area inside the encoding unit.
- the prediction unit is configured with a size of 64 ⁇ 64 pixels to 4 ⁇ 4 pixels in the region inside the coding unit, and has a prediction mode for in-plane prediction or a motion vector for inter prediction. The configuration of the coded stream will be described below with reference to FIGS. 4A to 6B.
- FIGS. 4A and 4B show hierarchical configurations of images decoded by the image processing apparatus according to the present embodiment.
- a group of a plurality of pictures is called a sequence.
- each picture is divided into slices, and each slice is further divided into coding units. The picture may not be divided into slices.
- the size of the largest coding unit is 64 ⁇ 64 pixels.
- FIG. 4C is a diagram showing a coded stream according to the present embodiment.
- the data shown in FIGS. 4A and 4B are hierarchically coded to obtain the coded stream shown in FIG. 4C.
- the coded stream shown in FIG. 4C includes a sequence header that controls a sequence, a picture header that controls a picture, a slice header that controls a slice, and coding unit layer data (CU layer data).
- sequence header is also called SPS (Sequence Parameter Set)
- PPS Picture Parameter Set
- FIG. 5A is a diagram showing a configuration example of a coding unit and coding unit layer data according to the present embodiment.
- the coding unit layer data corresponding to the coding unit is configured of a CU division flag and CU data (coding unit data). In the case of “1”, this CU division flag indicates that the coding unit is divided into four, and in the case of “0”, this indicates that the coding unit is not divided into four.
- the 64x64 pixel encoding unit is not split. That is, the CU division flag is "0".
- FIG. 5B is a diagram showing a configuration example of CU data according to the present embodiment.
- the CU data includes CU type, motion vector or in-plane prediction mode, and coefficients.
- the CU type determines the size of the prediction unit.
- FIG. 6A is a diagram showing an example of selectable prediction unit sizes. Specifically, prediction units such as 64x64 pixels, 32x64 pixels, 64x32 pixels, 32x32 pixels, 16x32 pixels, 32x16 pixels, 16x16 pixels, 16x8 pixels, 8x16 pixels, 8x8 pixels, 8x4 pixels, 4x8 pixels, 4x4 pixels, etc. It is shown.
- the size of the prediction unit can be selected from the size of 4 ⁇ 4 pixels or more. Also, the shape of the prediction unit may be rectangular.
- a motion vector or an in-plane prediction mode is specified for each prediction unit.
- only motion vectors are used, so only motion vectors are shown in FIG. 5B.
- a 16 ⁇ 64 pixel prediction unit and a 48 ⁇ 64 pixel prediction unit obtained by dividing a square into 1: 3 may be selected.
- FIG. 6B is a diagram showing an example of selectable conversion unit sizes. Specifically, conversion units such as 32 ⁇ 32 pixels, 16 ⁇ 32 pixels, 32 ⁇ 16 pixels, 16 ⁇ 16 pixels, 16 ⁇ 8 pixels, 8 ⁇ 16 pixels, 8 ⁇ 8 pixels, 8 ⁇ 4 pixels, 4 ⁇ 8 pixels, and 4 ⁇ 4 pixels are shown. As shown in FIG. 6B, a conversion unit of 8 ⁇ 32 pixels and a conversion unit of 24 ⁇ 32 pixels obtained by dividing the square into 1: 3 may be selected.
- FIG. 7 is a flowchart showing the decoding operation of one sequence included in the coded stream.
- the operation of the image processing apparatus shown in FIG. 2 will be described using the flowchart shown in FIG.
- the image processing apparatus first decodes the sequence header (S901).
- the variable-length decoding unit 503 decodes the encoded stream based on the control of the control unit 501.
- the image processing apparatus similarly decodes the picture header (S902) and decodes the slice header (S903).
- the image processing apparatus decodes the coding unit (S904). Decoding of the coding unit will be described in detail later. After decoding the coding unit, the image processing apparatus determines whether the decoded coding unit is the last coding unit of the slice (S905). Then, if the decoded coding unit is not the end of the slice (No in S905), the image processing apparatus decodes the next coding unit again (S904).
- the image processing apparatus determines whether the slice including the decoded coding unit is the last slice of the picture (S906). Then, if the slice is not the end of the picture (No in S906), the image processing apparatus decodes the slice header again (S903).
- the image processing apparatus determines whether the picture including the decoded encoding unit is the last picture of the sequence (S907). Then, if the picture is not the end of the sequence (No in S907), the image processing apparatus decodes the picture header again (S902). The image processing apparatus ends a series of decoding operations after decoding of all the pictures of the sequence.
- FIG. 8 is a flowchart showing the decoding operation of one coding unit. The operation of decoding (S904) of the coding unit of FIG. 7 will be described using the flowchart shown in FIG.
- variable-length decoding unit 503 performs variable-length decoding on a coding unit to be processed included in the input coded stream (S1001).
- variable length decoding unit 503 outputs coding information such as a coding unit type, an intra prediction (intra prediction) mode, motion vector information, and a quantization parameter.
- the output coding information includes the size of the coding unit, the processing order of the coding unit, the size of the prediction unit, the size of the transform unit, and the processing order of the transform unit.
- the variable-length decoding unit 503 outputs coefficient information corresponding to each pixel data.
- the coding information is output to the control unit 501 and then input to each processing unit.
- the coefficient information is output to the next inverse quantization unit 504.
- the inverse quantization unit 504 performs inverse quantization processing (S1002).
- the inverse frequency transform unit 505 performs inverse frequency transform to generate a difference image (S1003).
- control unit 501 determines whether inter prediction is used for the coding unit to be processed or in-plane prediction is used (S1004).
- the control unit 501 activates the motion vector calculation unit 511.
- the motion vector calculation unit 511 calculates a motion vector (S1005).
- the motion vector computing unit 511 transfers, from the frame memory 502, a reference image indicated by the motion vector.
- the control unit 501 activates the motion compensation unit 506.
- the motion compensation unit 506 generates a predicted image with 1/2 pixel accuracy or 1/4 pixel accuracy (S1006).
- the control unit 501 activates the in-plane prediction unit 507.
- the in-plane prediction unit 507 performs in-plane prediction processing to generate a predicted image (S1007).
- the reconstruction unit 508 generates a reconstructed image by adding the predicted image output by the motion compensation unit 506 or the in-plane prediction unit 507 and the difference image output by the inverse frequency conversion unit 505 (S1008). ).
- the generated reconstructed image is input to the in-loop filter unit 510.
- the part used in the in-plane prediction is stored in the reconstructed image memory 509.
- the in-loop filter unit 510 performs in-loop filter processing to reduce noise on the obtained reconstructed image.
- the in-loop filter unit 510 stores the result in the frame memory 502 (S1009).
- FIG. 9 is an explanatory view showing an outline of the motion compensation process.
- the motion compensation process extracts a part of a picture decoded in the past pointed to by a motion vector v (vx, vy) decoded from a coded stream, and performs a filter operation, It is a process of generating a predicted image.
- the reference image extracted from the reference picture is 71 ⁇ 71 pixels. If the top left coordinates of the prediction unit are (x, y), the reference image is a 71 ⁇ 71 pixel rectangle whose top left coordinates are (x + vx ⁇ 3, y + vy ⁇ 3).
- FIG. 10A is a diagram showing a prediction unit and motion vectors according to the present embodiment.
- the prediction unit PU 0 of 64 ⁇ 64 pixels shown in FIG. 10A has one motion vector v.
- FIG. 10B is a diagram showing division of prediction unit PU0 shown in FIG. 10A.
- the prediction unit PU0 of 64 ⁇ 64 pixels is divided into 16 blocks PU0a to PU0p of 16 ⁇ 16 pixels.
- One motion vector v for the 64 ⁇ 64 pixel prediction unit PU0 shown in FIG. 10A is the same for any pixel of this prediction unit PU0. That is, as shown in FIG. 10B, even when the prediction unit PU0 is divided into 16 blocks, the motion vectors of the respective blocks are all the same motion vector v. Therefore, the 64x64 pixel prediction unit PU0 is processed as 16 blocks with the same motion vector v.
- the start of reconstruction processing is delayed according to the processing order of 16 blocks, that is, the order (prediction order) of the transfer processing of the reference image of each block, the motion compensation processing, and the output processing of prediction images May be
- the image processing apparatus divides prediction units and performs transfer processing of reference images of respective blocks, motion compensation processing, and output processing of prediction images in the processing order of the conversion units.
- the processing order of transform units may be, for example, the storage order of a plurality of transform units in a coded stream.
- FIG. 11 is a flowchart showing an operation of motion compensation according to the present embodiment. The operations of the motion vector calculator 511 and the motion compensator 506 shown in FIG. 3 will be described using FIG.
- the motion vector calculation unit 511 calculates the motion vector of the prediction unit (PU) according to a method defined by the standard (S1100).
- the motion vector computing unit 511 determines the edge (edge or side) of the transform unit within the rectangle of the prediction unit based on the in-screen coordinates and size for the prediction unit in the coding unit (CU) and the transform unit (TU). It is determined whether it includes the TU boundary (S1101).
- the motion vector calculation unit 511 determines the reference image position and the reference from the motion vector, the coordinates of the prediction unit, and the size of the prediction unit.
- the image size is calculated (S1102).
- the motion vector calculation unit 511 sets the obtained reference image position and reference image size in the DMA control unit 512.
- the DMA control unit 512 transfers the reference image from the frame memory 502 to the reference image storage unit 513 (S1103).
- the motion compensation unit 506 performs a motion compensation operation using the reference image transferred to the reference image storage unit 513 (S1104). Then, the motion compensation unit 506 writes the predicted image obtained as a result of the motion compensation in the predicted image storage unit 514 (S1110).
- the motion vector calculator 511 divides the prediction unit by the edge of the conversion unit (S1105).
- the motion vector calculation unit 511 calculates, for each of the plurality of blocks obtained by the division, a reference image position and a reference image size required in the motion compensation process (S1106).
- the motion vector calculation unit 511 sets the obtained reference image position and reference image size in the DMA control unit 512.
- the DMA control unit 512 transfers the reference image from the frame memory 502 to the reference image storage unit 513 (S1107).
- the motion compensation unit 506 performs a motion compensation operation using the reference image transferred to the reference image storage unit 513 (S1108). Then, the motion compensation unit 506 writes the predicted image obtained as a result of the motion compensation in the predicted image storage unit 514 (S1111).
- the motion vector calculation unit 511 determines whether or not there is an unprocessed block in the original prediction unit (S1109). If there is an unprocessed block (Yes in S1109), the motion vector calculator 511 calculates a reference image position and a reference image size for the block (S1106). If there is no unprocessed block (No in S1109), the motion compensation process ends.
- the image processing apparatus performs the above-described processes (S1106, S1107, S1108, S1111, and S1109) in the processing order of the conversion units.
- FIG. 12A is a diagram showing an example of configuration of a coding unit according to the present embodiment.
- the coding unit CU0 shown in FIG. 12A has a size of 64 ⁇ 64 pixels.
- FIG. 12B is a diagram showing a configuration example of a prediction unit in the coding unit CU0 shown in FIG. 12A.
- the prediction unit PU0 shown in FIG. 12B is composed of the entire region of the coding unit CU0.
- FIG. 12C is a diagram showing a configuration example of the transform unit in the coding unit CU0 shown in FIG. 12A.
- 16 conversion units TU0 to TU15 are shown.
- the 16 conversion units TU0 to TU15 are processed in the order of TU0, TU1, TU2,..., TU15.
- the processes for the 16 transform units TU0 to TU15 include frequency transform, inverse frequency transform, quantization and inverse quantization, and the like.
- FIG. 12D is a diagram showing a configuration example of blocks after division of prediction units in coding unit CU0 shown in FIG. 12A.
- sixteen blocks PU0a to PU0p are shown.
- the sixteen blocks PU0a to PU0p are processed in the order of PU0a, PU0b, PU0c,..., PU0p.
- processing for the 16 blocks PU0a to PU0p there are transfer processing of a reference image, motion compensation processing, output processing of a predicted image, and the like.
- the processing order of 16 blocks PU0a to PU0p shown in FIG. 12D follows the conversion order which is the processing order of 16 conversion units TU0 to TU15 shown in FIG. 12C. That is, prediction unit PU0 is processed according to the conversion order.
- FIG. 12E is a diagram showing a configuration example of blocks after division of the coding unit CU0 shown in FIG. 12A.
- FIG. 12E shows sixteen blocks BK0 to BK15.
- the sixteen blocks BK0 to BK15 are processed in the order of BK0, BK1, BK2,..., BK15.
- the processing for the 16 blocks BK 0 to BK 15 includes reconstruction processing, in-loop filter processing, storage processing in the frame memory 502, and the like.
- FIG. 13A is a time chart illustrating an example of pipeline processing when the procedure of FIG. 11 is not applied.
- variable length decoding processing is performed on the coding unit CU0.
- inverse quantization processing and inverse frequency conversion processing are performed on the conversion units TU0 to TU15 in the order of TU0, TU1, TU2,..., TU15.
- motion compensation processing is performed on the prediction unit PU0.
- a reconstruction process is performed on the coding unit CU0.
- in-loop filtering is performed on the coding unit CU0.
- reconstruction processing starts until motion compensation processing ends for prediction unit PU0, and inverse quantization processing and inverse frequency conversion processing end for conversion units TU0 to TU15. Can not.
- FIG. 13B is a time chart showing an example of pipeline processing when the procedure of FIG. 11 is applied.
- variable length decoding processing is performed on the coding unit CU0.
- inverse quantization processing and inverse frequency conversion processing are performed on the conversion units TU0 to TU15 in the order of TU0, TU1, TU2,..., TU15.
- motion compensation processing is performed on the blocks PU0a to PU0p in the order of PU0a, PU0b, PU0c,.
- the reconstruction processing is executed on the blocks BK0 to BK15 in the order of BK0, BK1, BK2, ..., BK15.
- the reconstruction processing of the block BK0 starts.
- the reconstruction processing of blocks BK1 to BK15 sequentially starts at the timing when the inverse quantization processing and inverse frequency conversion processing of conversion units TU1 to TU15 end sequentially and the motion compensation processing of blocks PU0 b to PU0 p end sequentially .
- in-loop filter processing is performed on the blocks BK0 to BK15 in the order of BK0, BK1, BK2, ..., BK15.
- the in-loop filter processing of the block BK0 starts.
- in-loop filter processing of the blocks BK1 to BK15 sequentially starts at timing when the reconstruction processing of the blocks BK1 to BK15 ends in order.
- the predicted images of the blocks after division are output in the same order as the conversion units. Therefore, compared with the case of FIG. 13A, both the difference image required for the reconstruction processing and the predicted image in the same area as the difference image are aligned earlier. Therefore, the delay in the start of reconstruction processing and in-loop filtering is reduced. Therefore, the decoding process is speeded up.
- the image processing apparatus can execute the transfer processing of the reference image, the motion compensation processing, and the output processing of the prediction image in parallel by pipeline processing by dividing the prediction unit. Therefore, the image processing apparatus can shorten the processing time required for motion compensation.
- the processing of the conversion units TU0 to TU15 and the processing of the blocks PU0a to PU0p start at the same timing.
- these processes do not have to start at the same timing, and can be started independently of each other when the necessary information is obtained.
- the image processing apparatus can reduce the memory capacity by dividing the prediction unit and performing processing in the processing order of the conversion unit. Furthermore, the image processing apparatus can reduce processing delay by performing transfer processing of a reference image and processing of motion compensation in a pipelined manner. Also, the image processing apparatus can accelerate the start of the reconstruction process and the in-loop filter process. Therefore, the entire decoding process is speeded up.
- the image processing apparatus calculates the reference image position, calculates the reference image size, transmits the reference image, one by one in the processing order of the conversion unit for the plurality of blocks after the division of the prediction unit. Motion compensation calculation and prediction image output are performed.
- the image processing apparatus may perform processing on a plurality of blocks two by two, three each, or may perform processing on blocks corresponding to 256 pixels.
- the image processing apparatus divides the prediction unit after motion vector calculation.
- the image processing apparatus may divide the reference image necessary for the motion compensation process into a plurality of blocks by the edge of the conversion unit. Then, the image processing apparatus may perform the motion compensation calculation and the prediction image output one by one on the plurality of blocks after the division of the reference image in the processing order of the conversion unit.
- the required capacity of the reference image storage unit 513 is the same as in the case where the division is not performed.
- the required capacity of the predicted image storage unit 514 is reduced.
- the image processing apparatus can start reconstruction processing and in-loop filter processing early. Therefore, the entire decoding process is speeded up. In addition, the frequency of access to the frame memory 502 is reduced.
- the image processing device divides the predicted image necessary for the reconstruction processing into a plurality of blocks by the edge of the conversion unit. It is also good. Then, the image processing apparatus may perform predicted image output, reconstruction processing, and in-loop filter processing one by one on the plurality of blocks after division of the predicted image in the processing order of the conversion unit.
- the required capacity of the reference image storage unit 513 does not decrease.
- the required capacity of the predicted image storage unit 514 is reduced.
- the image processing apparatus can start reconstruction processing and in-loop filter processing early. Therefore, the entire decoding process is speeded up. In addition, the frequency of access to the frame memory 502 is reduced.
- each processing unit may be realized by a circuit using dedicated hardware or may be realized by a program executed by a processor.
- the frame memory 502, the reference image storage unit 513, and the predicted image storage unit 514 are shown as a memory or a storage unit. However, as long as they are storage elements capable of storing data, they may have any configuration such as a flip flop or a register. Furthermore, a part of the memory area of the processor or a part of the cache memory may be used as the frame memory 502, the reference image storage unit 513, and the predicted image storage unit 514.
- an image processing apparatus which is an image decoding apparatus is shown.
- the image processing apparatus is not limited to the image decoding apparatus.
- an image coding apparatus that executes decoding processing in the reverse procedure can also divide a prediction unit and perform motion compensation processing. For example, even in an image coding apparatus, motion compensation may be performed to generate a reference image. In such a case, the image coding apparatus may divide the prediction unit and perform motion compensation processing.
- motion compensation is described as an example.
- the image processing apparatus can obtain the same effect by executing the same process.
- the sizes and shapes of the coding unit, prediction unit and transform unit shown in the present embodiment are examples, and these sizes and shapes may be any size and shape.
- the image processing apparatus decodes a coded stream generated by coding processing including motion compensation.
- the size of motion compensation i.e. the size of the prediction unit, is variable, up to 64x64 pixels.
- the image processing apparatus divides the prediction unit into a plurality of blocks at the edge of the transform unit when the edge of the transform unit is included in the rectangle of the prediction unit for each of all the prediction units included in the encoding unit. Do. Then, the image processing apparatus performs reference image transfer processing, motion compensation processing, and prediction image output processing on the plurality of blocks obtained by division in the processing order of the conversion unit.
- inter prediction is used among intra prediction and inter prediction.
- the image processing apparatus also performs processing in the processing order of the conversion unit with respect to the subsequent reconstruction processing and in-loop filter processing.
- the image processing apparatus can perform processing on all prediction units included in the encoding unit in the same processing order as the conversion unit processing order. Therefore, the image processing apparatus may be able to speed up the decoding process more than in the first embodiment.
- FIG. 2 is a block diagram of an image processing apparatus according to the present embodiment.
- FIG. 3 is a block diagram of the periphery of the motion compensation unit 506 according to the present embodiment.
- the configuration of the image processing apparatus according to the present embodiment is the same as that of the first embodiment, so the description will be omitted.
- FIG. 14A is a diagram showing a prediction unit and motion vectors according to the present embodiment.
- one coding unit includes two prediction units PU0 and PU1.
- the prediction unit PU0 has one motion vector v0 and the prediction unit PU1 has one motion vector v1.
- FIG. 14B is a diagram showing the division of the two prediction units PU0, PU1 shown in FIG. 14A.
- the prediction unit PU0 is divided into two blocks PU0a and PU0b
- the prediction unit PU1 is divided into two blocks PU1a and PU1b.
- One motion vector v0 for prediction unit PU0 shown in FIG. 14A is the same for any pixel of this prediction unit. That is, as shown in FIG. 14B, even when the prediction unit PU0 is divided into two blocks PU0a and PU0b, the two blocks PU0a and PU0b each have the same motion vector v0. Similarly, the two blocks PU1a, PU1b each have the same motion vector v1.
- the image processing apparatus divides a plurality of prediction units included in a coding unit into a plurality of blocks. Then, the image processing apparatus performs the transfer process of the reference image, the motion compensation process, and the output process of the predicted image for each block in the process order of the conversion unit.
- FIG. 15 is a flowchart showing the motion compensation operation according to the present embodiment. The operations of motion vector operation unit 511 and motion compensation unit 506 according to the present embodiment will be described using FIG. FIG. 15 shows the motion compensation operation for one coding unit.
- the motion vector calculation unit 511 calculates the motion vector of the prediction unit included in the coding unit according to a method defined by the standard (S1400). Next, the motion vector calculation unit 511 determines whether or not the edge of the transform unit is included in the rectangle of the prediction unit based on the in-screen coordinates and the size of the transform unit included in the same encoding unit (S1401) .
- the motion vector computing unit 511 determines the reference image position based on the motion vector, the coordinates of the prediction unit, and the size of the prediction unit. And the reference image size is calculated (S1402).
- the prediction unit is divided by the edge of the conversion unit (S1405).
- the motion vector calculation unit 511 calculates, for a plurality of blocks obtained by division, a reference image position and a reference image size necessary for motion compensation processing (S1406).
- the motion vector calculation unit 511 has performed a series of processing (S1400, S1401, S1402, S1405, S1406) of calculation of the reference image position and the reference image size for all prediction units in the encoding unit It is determined whether or not it is (S1408). If there is an unprocessed prediction unit (No in S1408), the motion vector calculator 511 performs motion vector calculation on the prediction unit (S1400).
- the motion vector calculation unit 511 sets the obtained reference image position and reference image size in the DMA control unit 512.
- the DMA control unit 512 transfers the reference image from the frame memory 502 to the reference image storage unit 513 (S1403).
- the motion compensation unit 506 performs a motion compensation operation using the reference image transferred to the reference image storage unit 513 (S1404), and writes the result in the predicted image storage unit 514 (S1407).
- the motion vector calculation unit 511 determines whether or not a series of processing (S1403, S1404, and S1407) of calculation of the reference image position and the reference image size has been performed on all prediction units in the encoding unit. (S1409). If there is an unprocessed block (No in S1409), the motion vector calculator 511 transfers a reference image for that block (S1403). If there is no unprocessed block (Yes in S1409), the process ends.
- the motion vector calculation unit 511, the DMA control unit 512, and the motion compensation unit 506 perform the above-described processing (S1403, S1404, and S1407) in the processing order of the conversion unit.
- FIG. 16A is a diagram showing an example of configuration of a coding unit according to the present embodiment.
- the coding unit CU0 shown in FIG. 16A has a size of 64 ⁇ 64 pixels.
- FIG. 16B is a diagram showing a configuration example of a prediction unit in coding unit CU0 shown in FIG. 16A.
- the coding unit CU0 comprises two prediction units PU0, PU1.
- FIG. 16C is a diagram showing a configuration example of a transform unit in coding unit CU0 shown in FIG. 16A.
- two conversion units TU0 and TU1 are shown.
- the two transform units TU0 and TU1 are processed in the order of TU0 and TU1.
- FIG. 16D is a diagram showing a configuration example of blocks after division of two prediction units PU0 and PU1 in the coding unit CU0 shown in FIG. 16A.
- Four blocks PU0a, PU0b, PU1a, PU1b are shown in FIG. 16D.
- the four blocks PU0a, PU0b, PU1a, and PU1b are processed in the order of PU0a, PU1a, PU0b, and PU1b.
- the processing order of the four blocks PU0a, PU0b, PU1a, PU1b shown in FIG. 16D follows the conversion order which is the processing order of the two conversion units TU0, TU1 shown in FIG. 16C. That is, two prediction units PU0 and PU1 are processed according to the conversion order.
- FIG. 16E is a diagram showing a configuration example of blocks after division of the coding unit CU0 shown in FIG. 16A. Two blocks BK0 and BK1 are shown in FIG. 16E. The two blocks BK0 and BK1 are processed in the order of BK0 and BK1.
- FIG. 17A is a time chart illustrating an example of pipeline processing when the procedure of FIG. 15 is not applied.
- variable length decoding processing is performed on the coding unit CU0.
- inverse quantization processing and inverse frequency conversion processing are performed on the conversion units TU0 and TU1 in the order of TU0 and TU1.
- motion compensation processing is performed on the prediction units PU0, PU1.
- a reconstruction process is performed on the coding unit CU0.
- in-loop filtering is performed on the coding unit CU0.
- reconstruction processing is performed until motion compensation processing ends for prediction units PU0 and PU1, and inverse quantization processing and inverse frequency conversion processing end for conversion units TU0 and TU1. Can not start.
- FIG. 17B is a time chart showing an example of pipeline processing when the procedure of FIG. 15 is applied.
- variable length decoding processing is performed on the coding unit CU0.
- inverse quantization processing and inverse frequency conversion processing are performed on the conversion units TU0 and TU1 in the order of TU0 and TU1.
- motion compensation processing is performed on the blocks PU0a, PU0b, PU1a, and PU1b in the order of PU0a, PU1a, PU0b, and PU1b.
- the reconstruction processing is executed on blocks BK0 and BK1 in the order of BK0 and BK1.
- the reconstruction processing of the block BK0 starts.
- the reconstruction processing of the block BK1 starts at the timing when the inverse quantization processing and inverse frequency conversion processing of the conversion unit TU1 end and the motion compensation processing of the blocks PU0 b and PU1 b ends.
- in-loop filter processing is performed on blocks BK0 and BK1 in the order of BK0 and BK1.
- the in-loop filter processing of the block BK0 starts.
- the in-loop filter processing of the block BK1 starts.
- the image processing apparatus can execute the transfer processing of the reference image, the motion compensation processing, and the output processing of the prediction image in parallel by pipeline processing by dividing the prediction unit. Therefore, the image processing apparatus can shorten the processing time required for motion compensation.
- the image processing apparatus can perform processing on all prediction units included in the encoding unit in the same processing order as the conversion unit. Therefore, the image processing apparatus according to the present embodiment may be able to speed up the decoding process more than in the case of the first embodiment.
- reference image transfer and motion compensation are performed on the plurality of blocks after division of all prediction units included in the encoding unit in the processing order of the conversion unit. Perform calculation and prediction image output.
- the image processing apparatus may perform processing on a plurality of blocks two by two, three each, or may perform processing on blocks corresponding to 256 pixels.
- the image processing apparatus divides the prediction unit after motion vector calculation. However, after the image processing apparatus performs reference image position calculation, reference image size calculation, and reference image transfer on all prediction units included in the encoding unit, the image processing apparatus performs a plurality of blocks of reference images necessary for motion compensation processing. It may be divided into Then, motion compensation calculation and predicted image output may be performed one by one on the plurality of blocks after division of the reference image in the processing order of the conversion unit.
- the required capacity of the reference image storage unit 513 is the same as in the case where the division is not performed.
- the required capacity of the predicted image storage unit 514 is reduced.
- the image processing apparatus can start reconstruction processing and in-loop filter processing early. Therefore, the entire decoding process is speeded up. In addition, the frequency of access to the frame memory 502 is reduced.
- the image processing apparatus performs reference image position calculation, reference image size calculation, reference image transfer, and motion compensation processing on all prediction units included in the coding unit, and then performs prediction images necessary for reconstruction processing. May be divided into a plurality of blocks. Then, the image processing apparatus may perform predicted image output, reconstruction processing, and in-loop filter processing one by one on the plurality of blocks after division of the predicted image in the processing order of the conversion unit.
- the required capacity of the reference image storage unit 513 does not decrease.
- the required capacity of the predicted image storage unit 514 is reduced.
- the image processing apparatus can start reconstruction processing and in-loop filter processing early. Therefore, the entire decoding process is speeded up. In addition, the frequency of access to the frame memory 502 is reduced.
- each processing unit may be realized by a circuit using dedicated hardware or may be realized by a program executed by a processor.
- the frame memory 502, the reference image storage unit 513, and the predicted image storage unit 514 are shown as a memory or a storage unit. However, as long as they are storage elements capable of storing data, they may have any configuration such as a flip flop or a register. Furthermore, a part of the memory area of the processor or a part of the cache memory may be used as the frame memory 502, the reference image storage unit 513, and the predicted image storage unit 514.
- an image processing apparatus which is an image decoding apparatus is shown.
- the image processing apparatus is not limited to the image decoding apparatus.
- an image coding apparatus that executes decoding processing in the reverse procedure can also divide a prediction unit and perform motion compensation processing.
- motion compensation is described as an example.
- the image processing apparatus can obtain the same effect by executing the same process.
- the sizes and shapes of the coding unit, prediction unit and transform unit shown in the present embodiment are examples, and these sizes and shapes may be any size and shape.
- the image processing apparatus decodes a coded stream generated by coding processing including motion compensation.
- the size of motion compensation i.e. the size of the prediction unit, is variable, up to 64x64 pixels.
- the image processing apparatus does not include the edge of the conversion unit in the rectangle of the prediction unit, and if the processing order of the conversion unit and the processing order of the prediction unit are different, the processing order of the reference image in the conversion unit Transfer processing, motion compensation processing, and output processing of a predicted image are performed.
- inter prediction is used among intra prediction and inter prediction.
- the image processing apparatus also performs processing in the processing order of the conversion unit with respect to the subsequent reconstruction processing and in-loop filter processing.
- the image processing apparatus can process the prediction unit in the same processing order as the conversion unit without dividing the prediction unit. Therefore, the decoding process is speeded up.
- FIG. 2 is a block diagram of an image processing apparatus according to the present embodiment.
- FIG. 3 is a block diagram of the periphery of the motion compensation unit 506 according to the present embodiment.
- the configuration of the image processing apparatus according to the present embodiment is the same as that of the first embodiment, so the description will be omitted.
- FIG. 18 is a diagram showing a prediction unit and motion vectors according to the present embodiment.
- one coding unit includes four prediction units PU0 to PU3.
- the prediction unit PU0 has one motion vector v0.
- prediction units PU1 to PU3 have motion vectors v1 to v3.
- the image processing apparatus transfers reference images to a plurality of prediction units in the processing order of conversion units without dividing the plurality of prediction units included in the encoding unit, and performs motion compensation processing. And output processing of a predicted image.
- FIG. 19 is a flowchart showing an operation of motion compensation according to the present embodiment. The operations of motion vector operation unit 511 and motion compensation unit 506 according to the present embodiment will be described using FIG. FIG. 19 shows the motion compensation operation for one coding unit.
- the motion vector calculator 511 calculates the motion vector of the prediction unit included in the coding unit according to the method defined by the standard (S1800).
- the motion vector computing unit 511 calculates a reference image position and a reference image size based on the motion vector, the coordinates of the prediction unit, and the size of the prediction unit (S1802).
- the motion vector computing unit 511 determines whether or not a series of processing (S1800, S1802) of calculation of the reference image position and the reference image size has been performed on all prediction units in the encoding unit (S1800 and S1802). S1808). If there is an unprocessed prediction unit (No in S1808), the motion vector calculator 511 performs motion vector calculation on the prediction unit (S1800).
- the motion vector calculation unit 511 sets the obtained reference image position and reference image size in the DMA control unit 512.
- the DMA control unit 512 transfers the reference image from the frame memory 502 to the reference image storage unit 513 (S1803).
- the motion compensation unit 506 performs calculation of motion compensation using the reference image transferred to the reference image storage unit 513 (S1804), and writes the result in the predicted image storage unit 514 (S1807).
- the motion vector calculation unit 511 determines whether or not a series of processing (S1803, S1804, S1807) of calculation of the reference image position and the reference image size has been performed on all prediction units in the encoding unit. (S1809). If there is an unprocessed block (No in S1809), the motion vector calculator 511 transfers a reference image for that block (S1803). If there is no unprocessed block (Yes in S1809), the process ends.
- the motion vector calculation unit 511, the DMA control unit 512, and the motion compensation unit 506 perform the above-described processing (S1803, S1804, and S1807) in the processing order of the conversion unit.
- FIG. 20A is a diagram showing a configuration example of a coding unit according to the present embodiment.
- the coding unit CU0 shown in FIG. 20A has a size of 64 ⁇ 64 pixels.
- FIG. 20B is a diagram showing a configuration example of a prediction unit in coding unit CU0 shown in FIG. 20A.
- the coding unit CU0 includes four prediction units PU0 to PU3.
- FIG. 20C is a diagram showing a configuration example of a transform unit in coding unit CU0 shown in FIG. 20A.
- two conversion units TU0 and TU1 are shown.
- the two transform units TU0 and TU1 are processed in the order of TU0 and TU1.
- FIG. 20D is a diagram showing the processing order of the four prediction units PU0 to PU3 in the coding unit CU0 shown in FIG. 16A.
- the four prediction units PU0 to PU3 are processed in the order of PU0, PU2, PU1, and PU3.
- the processing order of the four prediction units PU0 to PU3 shown in FIG. 20D follows the conversion order which is the processing order of the two conversion units TU0 and TU1 shown in FIG. 20C. That is, the four prediction units PU0 to PU3 are processed according to the conversion order.
- FIG. 20E is a diagram showing a configuration example of blocks after division of coding unit CU0 shown in FIG. 20A. Two blocks BK0 and BK1 are shown in FIG. 20E. The two blocks BK0 and BK1 are processed in the order of BK0 and BK1.
- FIG. 21A is a time chart illustrating an example of pipeline processing when the procedure of FIG. 19 is not applied.
- variable length decoding processing is performed on the coding unit CU0.
- inverse quantization processing and inverse frequency conversion processing are performed on the conversion units TU0 and TU1 in the order of TU0 and TU1.
- motion compensation processing is performed on the prediction units PU0 to PU3.
- a reconstruction process is performed on the coding unit CU0.
- in-loop filtering is performed on the coding unit CU0.
- reconstruction processing is performed until motion compensation processing ends for prediction units PU0 to PU3, and inverse quantization processing and inverse frequency conversion processing end for conversion units TU0 and TU1. Can not start.
- FIG. 21B is a time chart illustrating an example of pipeline processing when the procedure of FIG. 19 is applied.
- variable length decoding processing is performed on the coding unit CU0.
- inverse quantization processing and inverse frequency conversion processing are performed on the conversion units TU0 and TU1 in the order of TU0 and TU1.
- motion compensation processing is performed on blocks PU0 to PU3 in the order of PU0, PU2, PU1, and PU3.
- the reconstruction processing is executed on blocks BK0 and BK1 in the order of BK0 and BK1.
- the reconstruction processing of the block BK0 starts.
- the reconstruction processing of the block BK1 starts at the timing when the inverse quantization processing and inverse frequency transformation processing of the conversion unit TU1 end and the motion compensation processing of the blocks PU1 and PU3 ends.
- in-loop filter processing is performed on blocks BK0 and BK1 in the order of BK0 and BK1.
- the in-loop filter processing of the block BK0 starts.
- the in-loop filter processing of the block BK1 starts.
- the prediction images are output in the same order as the conversion units. Therefore, compared with the case of FIG. 21A, both the difference image necessary for the reconstruction processing and the predicted image in the same area as the difference image are aligned earlier. Therefore, the delay in the start of reconstruction processing and in-loop filtering is reduced. Therefore, the decoding process is speeded up.
- the image processing apparatus performs processing of the prediction unit in the same processing order as the conversion unit without dividing the prediction unit. Therefore, the decoding process is speeded up.
- the image processing apparatus performs reference image transfer, motion compensation calculation, and predicted image output one by one for all prediction units included in the encoding unit in the processing order of the conversion unit.
- the image processing apparatus may perform processing on a plurality of prediction units two by two, three each, or may perform processing on blocks corresponding to 256 pixels.
- each processing unit may be realized by a circuit using dedicated hardware or may be realized by a program executed by a processor.
- the frame memory 502, the reference image storage unit 513, and the predicted image storage unit 514 are shown as a memory or a storage unit. However, as long as they are storage elements capable of storing data, they may have any configuration such as a flip flop or a register. Furthermore, a part of the memory area of the processor or a part of the cache memory may be used as the frame memory 502, the reference image storage unit 513, and the predicted image storage unit 514.
- an image processing apparatus which is an image decoding apparatus is shown.
- the image processing apparatus is not limited to the image decoding apparatus.
- An image coding apparatus that performs decoding processing in the reverse procedure can also perform motion compensation processing in the same manner.
- motion compensation is described as an example.
- the image processing apparatus can obtain the same effect by executing the same process.
- the sizes and shapes of the coding unit, prediction unit and transform unit shown in the present embodiment are examples, and these sizes and shapes may be any size and shape.
- Embodiment 4 (4-1. Overview) First, an outline of the image processing apparatus according to the present embodiment will be described.
- the image processing apparatus according to the present embodiment decodes a coded stream generated by coding processing including motion compensation.
- the size of in-plane prediction i.e. the size of the prediction unit, is variable, up to 64x64 pixels.
- the image processing device divides the prediction unit at the edge of the transform unit, when the rectangle of the prediction unit includes the edge of the transform unit. Then, the image processing apparatus performs in-plane prediction processing and output of a predicted image on the plurality of blocks obtained by division in the processing order of the conversion unit. The image processing apparatus also performs processing in the processing order of the conversion unit with respect to the subsequent reconstruction processing and in-loop filter processing.
- the image processing apparatus divides the prediction unit and performs in-plane prediction in the processing order of the conversion unit. This reduces the memory capacity required to store the predicted image. Furthermore, the image processing device can start reconstruction processing and in-loop filtering processing early. Therefore, the entire decoding process is speeded up.
- FIG. 2 is a block diagram of an image processing apparatus according to the present embodiment.
- the overall configuration of the image processing apparatus according to the present embodiment is the same as that of the first embodiment, so the description will be omitted.
- FIG. 22 is a configuration diagram of the periphery of the in-plane prediction unit 507 according to the present embodiment.
- symbol is allocated to the same component as FIG. 2, and description is abbreviate
- FIG. 22 shows a variable-length decoding unit 503, a prediction mode calculation unit 515, an in-plane prediction unit 507, a reconstructed image memory 509, a reconstruction unit 508, and a predicted image storage unit 514.
- the prediction mode calculation unit 515 and the prediction image storage unit 514 may be included in the in-plane prediction unit 507.
- the prediction mode calculation unit 515 calculates the in-plane prediction mode of the in-plane prediction based on the decoding information decoded by the variable-length decoding unit 503.
- the in-plane prediction unit 507 performs in-plane prediction based on the in-plane prediction mode. At this time, the in-plane prediction unit 507 may obtain the conversion order for the conversion unit from the variable length decoding unit 503.
- FIG. 23 is a flowchart showing an operation of in-plane prediction according to the present embodiment. The operation of prediction mode calculation unit 515 and in-plane prediction unit 507 shown in FIG. 22 will be described using FIG.
- the prediction mode calculation unit 515 calculates the in-plane prediction mode of the prediction unit according to the method defined by the standard (S2200).
- the in-plane prediction unit 507 determines whether or not the edge of the conversion unit is included in the rectangle of the prediction unit based on the in-screen coordinates and the size of the conversion unit (S2201).
- the in-plane prediction unit 507 makes a reference based on the in-plane prediction mode, the coordinates of the prediction unit, and the size of the prediction unit.
- the peripheral pixel position is calculated (S2202).
- the in-plane prediction unit 507 performs in-plane prediction using the reference pixel of the obtained reference peripheral pixel position and the in-plane prediction mode (S2204). Then, the in-plane prediction unit 507 writes the result of the in-plane prediction in the predicted image storage unit 514 (S2210).
- the in-plane prediction unit 507 divides the prediction unit by the edge of the conversion unit (S2205). Then, the in-plane prediction unit 507 calculates the reference peripheral pixel position of the block based on the in-plane prediction mode, the coordinates of the block, and the size of the block for a plurality of blocks generated by division (S2206 ).
- the in-plane prediction unit 507 performs in-plane prediction using the reference pixel at the obtained reference peripheral pixel position and the in-plane prediction mode (S2208). Then, the in-plane prediction unit 507 writes the result of the in-plane prediction in the predicted image storage unit 514 (S2211).
- the in-plane prediction unit 507 calculates the reference peripheral pixel position of the block (S2206). If there is no unprocessed block (No in S2209), the process ends.
- the in-plane prediction unit 507 performs the above-described processes (S2206, S2208, S2211, and S2209) in the processing order of the conversion units.
- the image processing apparatus can obtain the same effect as in the case of the first embodiment with respect to the in-plane prediction by the above processing. That is, as shown in FIG. 13B, predicted images are output in the same order as the conversion units. Therefore, the difference image necessary for the reconstruction processing and the predicted image in the same area as the difference image are aligned quickly. Therefore, compared with the case of FIG. 13A, the delay of the start of the reconstruction process and the in-loop filter process becomes smaller. Therefore, the decoding process is speeded up.
- in-plane prediction processing, reconstruction processing and in-loop filter processing are performed in small data units. Therefore, the required capacity of the predicted image storage unit 514 for holding the predicted image is reduced, and the required capacity of the memory used for the reconstruction processing and the in-loop filter processing is reduced.
- FIG. 24 is a diagram illustrating an example of a prediction unit and reference peripheral pixels necessary for in-plane prediction processing of the prediction unit.
- reference peripheral pixels 2302 and 2303 located around the prediction unit are used.
- in-plane prediction is performed according to Equation 1 below.
- nS of Formula 1 represents the size of a prediction unit.
- nS is 64.
- p [x ', -1] indicates the reference peripheral pixel 2302
- x' indicates the value (x coordinate value) of the coordinate axis with the left end of the prediction unit as the origin and the right direction as the positive direction.
- p [ ⁇ 1, y ′] indicates a reference peripheral pixel 2303
- y ′ indicates a value (y coordinate value) of a coordinate axis with the upper end of the prediction unit as an origin and the lower direction as a positive direction.
- K is a variable that can be derived from the reference peripheral pixels. Further, predSamples [x, y] in Expression 1 indicates predicted pixel values as a result of performing in-plane prediction.
- Equation 1 the predicted pixel values are the same at any position in the prediction unit. Therefore, even if the prediction unit is divided into arbitrary rectangles, in-plane prediction processing for each block is possible. In addition, after the processing of in-plane prediction, it is possible to output a predicted image in any order.
- the image processing apparatus divides the prediction unit and executes the in-plane prediction processing in the processing order of the conversion unit. This reduces the required memory capacity. Also, faster start up of reconstruction and in-loop filtering is possible. Thus, the entire decoding process is speeded up.
- the image processing apparatus calculates the reference peripheral pixel position, the in-plane prediction calculation, and the prediction for the plurality of blocks after the division of the prediction unit, one by one in the processing order of the conversion unit. Perform image output.
- the image processing apparatus may perform processing on a plurality of blocks two by two, three each, or may perform processing on blocks corresponding to 256 pixels.
- the image processing apparatus divides the prediction unit after the in-plane prediction mode calculation.
- the image processing apparatus divides the predicted image by the edge of the conversion unit after reference peripheral image position calculation and in-plane prediction processing of the prediction unit, and outputs the predicted image according to the processing order of the conversion unit. Good.
- each processing unit may be realized by a circuit using dedicated hardware or may be realized by a program executed by a processor.
- the frame memory 502, the reference image storage unit 513, and the predicted image storage unit 514 are shown as a memory or a storage unit. However, as long as they are storage elements capable of storing data, they may have any configuration such as a flip flop or a register. Furthermore, a part of the memory area of the processor or a part of the cache memory may be used as the frame memory 502, the reference image storage unit 513, and the predicted image storage unit 514.
- an image processing apparatus which is an image decoding apparatus is shown.
- the image processing apparatus is not limited to the image decoding apparatus.
- the image coding apparatus that executes the decoding process in the reverse procedure can also divide the prediction unit to perform the in-plane prediction process.
- the sizes and shapes of the coding unit, prediction unit and transform unit shown in the present embodiment are examples, and these sizes and shapes may be any size and shape.
- the in-plane prediction mode called DC (Direct Current) prediction or average value prediction is used, but other in-plane prediction modes may be used.
- DC Direct Current
- an in-plane prediction mode called vertical prediction may be used
- an in-plane prediction mode called horizontal prediction may be used
- other in-plane prediction modes described in Non-Patent Document 2 are used. May be
- FIG. 25A is a diagram showing the configuration of an image processing apparatus according to the fifth embodiment.
- the image processing apparatus 2400 shown in FIG. 25A decodes the image encoded by the encoding process.
- the encoding process includes prediction processes for one or more prediction units and frequency conversion processes for a plurality of transform units included in a region including the one or more prediction units.
- the image processing apparatus 2400 includes a prediction unit 2401.
- the prediction unit 2401 corresponds to the motion compensation unit 506, the in-plane prediction unit 507, and the like described in the first embodiment.
- FIG. 25B is a flowchart showing an operation of the image processing apparatus 2400 shown in FIG. 25A.
- the prediction unit 2401 selects one or more prediction images corresponding to one or more prediction units when the prediction order predetermined for one or more prediction units does not follow the conversion order predetermined for a plurality of conversion units. It outputs according to the conversion order for every block (S2401). For example, the prediction unit 2401 generates a prediction image by prediction processing on one or more prediction units. Then, the prediction unit 2401 outputs the predicted image in the conversion order.
- the predetermined prediction order may be, for example, the order of prediction units in the stream or any other order.
- the predetermined conversion order may be, for example, the order of conversion units in the stream, or any other order. Then, the prediction unit 2401 controls the output of the prediction unit such that the area in which the processing result of the conversion unit is output and the area in which the processing result of the prediction unit is output are the same.
- the timing of the output of the predicted image is close to the timing of the output of the difference image obtained by the conversion. Therefore, it is possible to start the reconstruction processing at the later stage early. Therefore, the image processing apparatus 2400 can reduce the delay generated in the image processing.
- an inverse frequency transform unit 505 may be added which performs inverse frequency transform on the plurality of transform units in accordance with the transform order.
- the image processing apparatus 2400 shown in the fifth embodiment may be incorporated into the configuration shown in the other embodiments.
- the image processing apparatus according to the sixth embodiment is a specific configuration example of the image processing apparatus 2400 according to the fifth embodiment.
- FIG. 26A is a diagram showing the configuration of an image processing apparatus according to the sixth embodiment. Similar to the image processing device 2400 according to the fifth embodiment, the image processing device 2500 shown in FIG. 26A decodes the image encoded by the encoding process. Similar to Embodiment 5, this encoding process includes prediction processes for one or more prediction units, and frequency conversion processes for a plurality of transform units included in a region including the one or more prediction units.
- the image processing apparatus 2500 includes a prediction unit 2501.
- the prediction unit 2501 corresponds to the prediction unit 2401 shown in the fifth embodiment.
- the prediction unit 2501 further includes an information acquisition unit 2511, a prediction image generation unit 2512, and a prediction image output unit 2513.
- FIG. 26B is a flowchart showing an operation of the image processing apparatus 2500 shown in FIG. 26A.
- the information acquisition unit 2511 acquires information for generating a predicted image (S2501).
- information for generating a predicted image includes a motion vector, an in-plane prediction mode, a reference image, and reference peripheral pixels.
- the predicted image generation unit 2512 generates a predicted image by executing a prediction process using the acquired information (S2502).
- the predicted image output unit 2513 outputs a predicted image (S2503). At this time, when the prediction order does not follow the conversion order, the prediction image output unit 2513 outputs the prediction image block by block according to the conversion order.
- the timing of the output of the predicted image is close to the timing of the output of the difference image obtained by the conversion. Therefore, it is possible to start the reconstruction processing at the later stage early.
- the image processing apparatus 2500 can reduce the delay generated in image processing.
- the prediction image generation unit 2512 may generate a prediction image for each block according to the conversion order. Thereby, the generation process and the output process are smoothly performed.
- the information acquisition unit 2511 may acquire information for generating a prediction image according to the conversion order for each block. Thereby, the information acquisition process, the generation process, and the output process are smoothly performed. In addition, the memory capacity required for prediction decreases.
- the predicted image generation unit 2512 may generate a predicted image by performing in-plane prediction, or may generate a predicted image by executing motion compensation.
- FIG. 27A is a diagram showing the configuration of an image processing apparatus according to a seventh embodiment. Similar to the image processing apparatus 2400 according to the fifth embodiment, the image processing apparatus 2600 shown in FIG. 27A decodes the image encoded by the encoding process. Similar to Embodiment 5, this encoding process includes prediction processes for one or more prediction units, and frequency conversion processes for a plurality of transform units included in a region including the one or more prediction units.
- the image processing apparatus 2600 includes a division unit 2601 and a prediction unit 2602.
- the prediction unit 2602 corresponds to the prediction unit 2401 shown in the fifth embodiment.
- FIG. 27B is a flowchart showing the operation of the image processing apparatus 2600 shown in FIG. 27A.
- the dividing unit 2601 divides one or more prediction units into a plurality of blocks along the edges of a plurality of conversion units (S2601).
- the prediction unit 2602 outputs a predicted image according to the conversion order for each block obtained by dividing one or more prediction units (S2602).
- FIGS. 28A to 28D show whether or not the prediction order conforms to the transform order, and the prediction order applied to the prediction unit.
- the numerical values shown in FIGS. 28A to 28D indicate the order.
- the transform units and prediction units shown in FIGS. 28A-28D are hierarchically divided. Each block obtained by hierarchically dividing the conversion unit is also a conversion unit. Also, each block obtained by hierarchically dividing a prediction unit is also a prediction unit.
- FIG. 28A is a diagram showing a first example of the prediction order used by the image processing device 2600 of FIG. 27A.
- the prediction order of the fourth prediction unit does not follow the conversion order of the fourth to seventh conversion units.
- the fourth prediction unit is split and processed according to the transform order of transform units.
- FIG. 28B is a diagram showing a second example of the prediction order used by the image processing device 2600 of FIG. 27A.
- the second and third prediction units do not follow the transformation order of the first and second transformation units.
- the second and third prediction units are processed according to the transformation order of the first and second transformation units.
- FIG. 28C is a diagram showing a third example of the prediction order used by the image processing device 2600 of FIG. 27A.
- the conversion order of the first to fourth conversion units matches the prediction order of the first to fourth prediction units. That is, the prediction order follows the conversion order. Thus, the prediction order may not be changed.
- FIG. 28D is a diagram showing a fourth example of the prediction order used by the image processing device 2600 of FIG. 27A.
- the prediction order of the fourth to seventh prediction units does not match the conversion order of the fourth conversion unit.
- the prediction order of the fourth to seventh prediction units is equivalent to the order obtained by subdividing the conversion order of the fourth conversion unit. Therefore, the prediction order of the fourth to seventh prediction units follows the conversion order of the fourth conversion unit. Thus, the prediction order may not be changed.
- whether or not the prediction order conforms to the conversion order is equivalent to whether or not the prediction order conforms to the conversion order, or whether the prediction order conforms to the conversion order.
- FIG. 29A is a diagram showing the configuration of an image processing apparatus according to an eighth embodiment. Similar to the image processing apparatus 2400 according to the fifth embodiment, the image processing apparatus 2800 shown in FIG. 29A decodes the image encoded by the encoding process. Similar to Embodiment 5, this encoding process includes prediction processes for one or more prediction units, and frequency conversion processes for a plurality of transform units included in a region including the one or more prediction units.
- the image processing apparatus 2800 includes a prediction unit 2801 and a reconstruction processing unit 2802.
- the prediction unit 2801 corresponds to the prediction unit 2401 shown in the fifth embodiment.
- the reconfiguration processing unit 2802 corresponds to the reconfiguration unit 508 shown in the first embodiment.
- FIG. 29B is a flowchart showing the operation of the image processing apparatus 2800 shown in FIG. 29A.
- the prediction unit 2801 outputs the prediction image for each block according to the conversion order (S2801).
- the reconfiguration processing unit 2802 executes reconfiguration processing for each block in the conversion order (S2802).
- the image processing apparatus 2800 can start the reconstruction processing early. Therefore, the image processing apparatus 2800 can reduce the delay generated in the image processing.
- FIG. 30A is a diagram showing the configuration of an image processing apparatus according to a ninth embodiment. Similar to the image processing apparatus 2400 according to the fifth embodiment, the image processing apparatus 2900 shown in FIG. 30A decodes the image encoded by the encoding process. Similar to Embodiment 5, this encoding process includes prediction processes for one or more prediction units, and frequency conversion processes for a plurality of transform units included in a region including the one or more prediction units.
- the image processing apparatus 2900 includes a prediction unit 2901, a reconstruction processing unit 2902, and an in-loop filter processing unit 2903.
- the prediction unit 2901 corresponds to the prediction unit 2401 shown in the fifth embodiment.
- the reconfiguration processing unit 2902 corresponds to the reconfiguration unit 508 described in the first embodiment.
- in-loop filter processing unit 2903 corresponds to in-loop filter unit 510 shown in the first embodiment.
- FIG. 30B is a flowchart showing an operation of the image processing apparatus 2900 shown in FIG. 30A.
- the prediction unit 2901 outputs the prediction image for each block according to the conversion order (S2901).
- the reconfiguration processing unit 2902 executes reconfiguration processing for each block in the conversion order (S2902).
- the in-loop filter processing unit 2903 executes in-loop filter processing for each block in the conversion order (S2903).
- the image processing apparatus 2900 can start the in-loop filter process early. Therefore, the image processing apparatus 2900 can reduce the delay generated in the image processing.
- the in-loop filter processing unit 2903 may store the image data obtained by performing the in-loop filter processing in the frame memory in the conversion order for each block.
- the image processing apparatus 2900 can quickly complete the in-loop filter processing. Therefore, the image processing apparatus 2900 can speed up a series of pipeline processes.
- FIG. 31A is a diagram showing the configuration of an image processing apparatus according to the tenth embodiment.
- the image processing apparatus 3100 shown in FIG. 31A decodes the image encoded by the encoding process.
- the encoding process includes prediction processes for one or more prediction units and frequency conversion processes for a plurality of transform units included in a region including the one or more prediction units.
- the image processing apparatus 3100 includes a processing unit 3101.
- the processing unit 3101 corresponds to the motion compensation unit 506, the in-plane prediction unit 507, and the like described in the first embodiment.
- the processing unit 3101 may correspond to the reconfiguring unit 508 described in the first embodiment.
- FIG. 31B is a flowchart showing the operation of the image processing apparatus 3100 shown in FIG. 31A.
- the processing unit 3101 executes image processing on predicted images of one or more prediction units, depending on a predetermined conversion order for a plurality of conversion units (S3101).
- the image processing apparatus 3100 reduces the waiting time in pipeline processing and reduces the processing delay.
- the image processing regarding a prediction image is a processing which handles a prediction image, for example, the processing which acquires the information for generating a prediction image, the processing which generates a prediction image, the processing which outputs a prediction image, the prediction image is used. Processing for reconstructing an image, or a combination of these processing.
- the processing unit 3101 may generate a prediction image by executing motion compensation, or may generate a prediction image by executing in-plane prediction.
- the processing unit 3101 shown in the tenth embodiment may execute other processing.
- the processing unit 3101 may perform inverse frequency transform on the plurality of transform units in accordance with the transform order.
- the image processing apparatus 3100 or the processing unit 3101 may have part or all of the configuration shown in the other embodiments. Also, the image processing apparatus 3100 or the processing unit 3101 may be incorporated into the configuration shown in the other embodiments.
- the processing unit 3101 may acquire the conversion order, and may execute the image processing in accordance with the acquired conversion order.
- the processing unit 3101 acquires information indicating the conversion order, and executes image processing in accordance with the conversion order indicated by the acquired information.
- the processing unit 3101 may determine the order of image processing in accordance with the conversion order, and execute the image processing in accordance with the determined order.
- the processing unit 3101 may execute the image processing in the conversion order for each block that is equal to or smaller than the conversion unit.
- the processing unit 3101 may execute the image processing in accordance with the conversion order when the prediction order determined in advance for one or more prediction units does not conform to the conversion order. In addition, the processing unit 3101 may execute the image processing in the conversion order when the prediction order is not determined in advance for one or more prediction units. The processing unit 3101 may execute the image processing in the conversion order when the edge format of the plurality of transform units is different from the edge format of one or more prediction units.
- the processing unit 3101 may execute in-loop filter processing, which is filter processing on the reconstructed image, in accordance with the conversion order. Further, the processing unit 3101 may execute processing of storing the image on which the in-loop filter processing has been performed in the frame memory in the order of conversion.
- FIG. 32A is a diagram showing the configuration of an image processing apparatus according to an eleventh embodiment. Similar to the image processing apparatus 3100 according to the tenth embodiment, the image processing apparatus 3200 shown in FIG. 32A decodes the image encoded by the encoding process. Similar to Embodiment 10, this encoding process includes prediction processes for one or more prediction units and frequency conversion processes for a plurality of transform units included in a region including the one or more prediction units.
- the image processing apparatus 3200 includes a dividing unit 3201 and a processing unit 3202.
- the processing unit 3202 corresponds to the processing unit 3101 described in the tenth embodiment.
- FIG. 32B is a flowchart showing the operation of the image processing apparatus 3200 shown in FIG. 32A.
- the dividing unit 3201 divides one or more prediction units into a plurality of blocks along the edges of a plurality of conversion units (S3201).
- the processing unit 3202 executes image processing on a predicted image in the conversion order for each block obtained by dividing one or more prediction units (S3202).
- the dividing unit 3201 may divide one prediction unit into a plurality of blocks along the edges of a plurality of transform units.
- the division unit 3201 may divide a plurality of prediction units included in the coding unit into a plurality of blocks along the edges of the plurality of transform units.
- FIG. 33A is a diagram showing the configuration of an image processing apparatus according to a twelfth embodiment.
- the image processing apparatus 3000 shown in FIG. 33A decodes the image encoded by the encoding process.
- the encoding process includes an in-plane prediction process of generating a predicted image corresponding to the prediction unit according to the prediction mode of the prediction unit.
- the image processing apparatus 3000 includes a dividing unit 3001 and a processing unit 3002.
- the processing unit 3002 corresponds to the in-plane prediction unit 507 or the like shown in the first embodiment.
- FIG. 33B is a flowchart showing an operation of the image processing apparatus 3000 shown in FIG. 33A.
- the dividing unit 3001 divides the prediction unit into a plurality of blocks (S3001).
- the processing unit 3002 performs in-plane prediction on each of the plurality of blocks according to the prediction mode of the prediction unit (S3002).
- the image processing apparatus 3000 can perform in-plane prediction in small data units. Therefore, the image processing apparatus 3000 can smoothly execute pipeline processing in smaller data units. Therefore, the image processing apparatus 3000 can reduce the delay generated in the image processing.
- the components shown in the other embodiments may be added to the image processing apparatus 3000 shown in the twelfth embodiment. Also, the image processing apparatus 3000 shown in the twelfth embodiment may be incorporated into the configuration shown in the other embodiments.
- each component may be configured by dedicated hardware or implemented by executing a software program suitable for each component.
- Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.
- program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.
- software for realizing the image processing apparatus and the like according to each of the above-described embodiments is the following program.
- this program is encoded by encoding processing including, in a computer, prediction processing for one or more prediction units and frequency conversion processing for a plurality of conversion units included in a region including the one or more prediction units.
- An image processing method for executing a process for decoding an image wherein the image process for the predicted image of the one or more prediction units is performed depending on a predetermined conversion order for the plurality of conversion units. Execute the image processing method to be executed.
- another processing unit may execute a process performed by a specific processing unit. Further, the order of executing the processing may be changed, or a plurality of processing may be executed in parallel.
- the present invention can be realized not only as an image processing apparatus but also as a method having processing means constituting the image processing apparatus as steps. For example, those steps are performed by a computer. And this invention can be implement
- the image processing apparatus and the image processing method according to the present invention are also applicable to an image coding apparatus, an image decoding apparatus, an image coding method and an image decoding method.
- the plurality of components included in the image processing apparatus may be realized as an LSI (Large Scale Integration) which is an integrated circuit. These components may be individually made into one chip, or may be made into one chip so as to include part or all. For example, components other than the memory may be integrated into one chip.
- an LSI is used here, it may be called an IC (Integrated Circuit), a system LSI, a super LSI, or an ultra LSI depending on the degree of integration.
- the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. It is also possible to use a programmable field programmable gate array (FPGA) or a reconfigurable processor that can reconfigure connection and setting of circuit cells in the LSI.
- FPGA field programmable gate array
- reconfigurable processor that can reconfigure connection and setting of circuit cells in the LSI.
- the storage medium may be a magnetic disk, an optical disk, a magneto-optical disk, an IC card, a semiconductor memory, or the like as long as the program can be recorded.
- FIG. 34 is a diagram showing an overall configuration of a content supply system ex100 for realizing content distribution service.
- the area for providing communication service is divided into desired sizes, and base stations ex106 to ex110, which are fixed wireless stations, are installed in each cell.
- devices such as a computer ex111, a PDA (Personal Digital Assistant) ex112, a camera ex113, a mobile phone ex114, and a game machine ex115 mutually communicate with each other via a telephone network ex104 and base stations ex106 to ex110.
- each device is connected to the Internet ex101 via the Internet service provider ex102.
- each device may be directly connected to the telephone network ex104 without passing through the base stations ex106 to ex110, which are fixed wireless stations.
- the devices may be directly connected to each other via near field communication or the like.
- the camera ex113 is an apparatus capable of shooting moving pictures such as a digital video camera
- the camera ex116 is an apparatus capable of shooting still pictures and moving pictures such as a digital camera
- the mobile phone ex114 is a GSM (registered trademark) (Global System for Mobile Communications) system, a CDMA (Code Division Multiple Access) system, a W-CDMA (Wideband-Code Division Multiple Access) system, an LTE (Long Term Evolution) system
- GSM Global System for Mobile Communications
- CDMA Code Division Multiple Access
- W-CDMA Wideband-Code Division Multiple Access
- LTE Long Term Evolution
- HSPA high speed packet access
- PHS personal handy phone system
- live distribution and the like become possible by connecting the camera ex113 and the like to the streaming server ex103 through the base station ex109 and the telephone network ex104.
- live distribution encoding processing is performed on content (for example, a video of a music live, etc.) captured by the user using the camera ex113 as described in the above embodiments, and the encoded content is transmitted to the streaming server ex103.
- the streaming server ex 103 streams the transmitted content data to the requested client.
- the clients include the computer ex111, the PDA ex112, the camera ex113, the mobile phone ex114, the game machine ex115 and the like capable of decoding the above-mentioned encoded data.
- Each device that has received the distributed data decrypts and reproduces the received data.
- encoding processing of captured data may be performed by the camera ex113, may be performed by the streaming server ex103 that performs data transmission processing, or may be performed sharing each other.
- the decryption processing of similarly distributed data may be performed by the client, may be performed by the streaming server ex 103, or may be performed sharing each other.
- not only the camera ex113 but also still images and / or moving image data captured by the camera ex116 may be transmitted to the streaming server ex103 via the computer ex111.
- the encoding process in this case may be performed by any of the camera ex 116, the computer ex 111, and the streaming server ex 103, or may be performed sharing each other.
- the encoding process and the decoding process are generally executed by a computer ex 111 and an LSI (Large Scale Integration) ex 500 included in each device.
- the LSI ex 500 may be a single chip or a plurality of chips.
- Software for image coding or software for image decoding is incorporated in any recording medium (CD-ROM, flexible disk, hard disk, etc.) readable by computer ex111 etc., and the coding process or decoding process is performed using the software.
- the streaming server ex103 may be a plurality of servers or a plurality of computers, and may process, record, or distribute data in a distributed manner.
- the client can receive and reproduce the encoded data.
- the client can receive, decode, and reproduce the information transmitted by the user in real time, and even a user who does not have special rights and facilities can realize personal broadcasting.
- At least the image encoding device or the image processing device of each of the above embodiments can be incorporated into the digital broadcasting system ex200.
- a bit stream of video information is communicated or transmitted to the satellite ex202 via radio waves.
- This bit stream is a coded bit stream coded by the image coding method described in each of the above embodiments.
- the broadcast satellite ex202 receiving this transmits a radio wave for broadcasting, and this radio wave is received by a home antenna ex204 capable of receiving satellite broadcasting.
- a device such as a television (receiver) ex300 or a set top box (STB) ex217 decodes and reproduces the received bit stream.
- the image processing apparatus shown in the above-described embodiment can also be mounted on the reproducing apparatus ex 212 that reads and decodes the bit stream recorded on the recording medium ex 214 such as a CD and a DVD as a recording medium.
- the reproduced video signal is displayed on the monitor ex 213.
- the image processing described in each of the above embodiments is also applied to a reader / recorder ex 218 that reads and decodes an encoded bit stream recorded on a recording medium ex 215 such as DVD or BD, or encodes and writes a video signal on the recording medium ex 215
- a device or an image coding device In this case, the reproduced video signal is displayed on the monitor ex 219, and the video signal can be reproduced in another apparatus and system by the recording medium ex 215 on which the encoded bit stream is recorded.
- the image processing apparatus may be mounted in the set top box ex217 connected to the cable ex203 for cable television or the antenna ex204 for satellite / terrestrial broadcast, and this may be displayed on the monitor ex219 of the television. At this time, the image processing apparatus may be incorporated in the television instead of the set top box.
- FIG. 36 is a diagram showing a television (receiver) ex300 that uses the image decoding method described in each of the above embodiments.
- the television ex300 is a tuner ex301 that acquires or outputs a bit stream of video information via the antenna ex204 that receives the broadcast, the cable ex203, or the like, and the encoded data that is to be received to demodulate or transmit the encoded data to the outside.
- a modulation / demodulation unit ex302 for modulation, and a multiplexing / demultiplexing unit ex303 for separating demodulated video data and audio data, or multiplexing encoded video data and audio data are provided.
- the television ex300 decodes the audio data and the video data, or the audio signal processing unit ex304 that encodes each information, a signal processing unit ex306 having the video signal processing unit ex305, and outputs the decoded audio signal.
- the television ex300 includes an interface unit ex317 including an operation input unit ex312 and the like that receive an input of a user operation.
- the television ex300 includes a control unit ex310 that centrally controls each unit, and a power supply circuit unit ex311 that supplies power to each unit.
- the interface unit ex317 is, besides the operation input unit ex312, a bridge ex313 connected to an external device such as a reader / recorder ex218, a slot unit ex314 for enabling attachment of a recording medium ex216 such as an SD card, external recording such as a hard disk It may have a driver ex 315 for connecting to a medium, a modem ex 316 connected to a telephone network, and the like. Note that the recording medium ex216 can electrically record information by a nonvolatile / volatile semiconductor memory element to be stored.
- the components of the television ex300 are connected to one another via a synchronization bus.
- the television ex300 decodes data acquired from the outside with the antenna ex204 and the like and reproduces the data.
- the television ex300 receives the user operation from the remote controller ex220 and the like, and demultiplexes the video data and audio data demodulated by the modulation / demodulation unit ex302 by the multiplexing / demultiplexing unit ex303 based on the control of the control unit ex310 having a CPU etc. .
- the television ex300 decodes the separated audio data by the audio signal processing unit ex304, and decodes the separated video data by the video signal processing unit ex305 using the decoding method described in each of the above embodiments.
- the decoded audio signal and video signal are output from the output unit ex309 to the outside.
- these signals may be temporarily stored in the buffers ex318, ex319, etc. so that the audio signal and the video signal are reproduced synchronously.
- the television ex300 may read the encoded bit stream not from the broadcast or the like, but from the recording media ex215 and ex216 such as a magnetic / optical disc and an SD card.
- television ex300 encodes an audio signal and a video signal and externally transmits or writes the audio signal and the video signal to a recording medium or the like.
- the television ex300 receives the user operation from the remote controller ex220 and the like, and based on the control of the control unit ex310, encodes the audio signal by the audio signal processing unit ex304, and the video signal processing unit ex305 executes the video signal in each of the above embodiments. Coding is performed using the coding method described above.
- the encoded audio signal and video signal are multiplexed by multiplexer / demultiplexer ex303 and output to the outside. At the time of multiplexing, these signals may be temporarily stored in the buffers ex320, ex321, etc. so that the audio signal and the video signal are synchronized.
- a plurality of buffers ex318 to ex321 may be provided as illustrated, or one or more buffers may be shared. Furthermore, besides being illustrated, data may be stored in a buffer as a buffer material to avoid system overflow and underflow even between, for example, the modulation / demodulation unit ex302 and the multiplexing / demultiplexing unit ex303.
- television ex300 In addition to acquiring audio data and video data from broadcasts and recording media, etc., television ex300 is also configured to receive AV input from a microphone and a camera, and even if data acquired from them is encoded. Good. Although television ex300 is described here as a configuration capable of the above encoding processing, multiplexing, and external output, such processing can not be performed, and a configuration capable of only the above reception, decoding processing, and external output It may be
- the decoding process or the encoding process may be performed by either the television ex300 or the reader / recorder ex218, or the television
- the ex 300 and the reader / recorder ex 218 may share each other.
- FIG. 37 shows a configuration of an information reproducing / recording unit ex400 in the case of reading or writing data from an optical disc.
- the information reproducing / recording unit ex400 includes elements ex401 to ex407 described below.
- the optical head ex401 irradiates a laser spot on the recording surface of the recording medium ex215 which is an optical disk to write information, detects reflected light from the recording surface of the recording medium ex215, and reads the information.
- the modulation recording unit ex402 electrically drives the semiconductor laser incorporated in the optical head ex401 and modulates the laser light according to the recording data.
- the reproduction / demodulation unit ex403 amplifies the reproduction signal obtained by electrically detecting the reflected light from the recording surface by the photodetector incorporated in the optical head ex401, separates and demodulates the signal component recorded in the recording medium ex215, and Play back information.
- the buffer ex 404 temporarily holds information to be recorded on the recording medium ex 215 and information reproduced from the recording medium ex 215.
- the disk motor ex405 rotates the recording medium ex215.
- the servo control unit ex406 moves the optical head ex401 to a predetermined information track while controlling the rotational drive of the disk motor ex405, and performs the laser
- the system control unit ex407 controls the entire information reproducing / recording unit ex400.
- the system control unit ex407 uses the various information held in the buffer ex404, and generates and adds new information as necessary.
- the modulation recording unit ex402 and the reproduction / demodulation unit This is realized by performing recording and reproduction of information through the optical head ex401 while cooperatively operating the servo control unit ex406.
- the system control unit ex 407 is configured of, for example, a microprocessor, and executes the processing of reading and writing by executing the program.
- the optical head ex401 may be configured to perform higher-density recording using near-field light.
- FIG. 38 shows a schematic view of the recording medium ex 215 which is an optical disc.
- a guide groove (groove) is formed in a spiral shape on the recording surface of the recording medium ex215, and in the information track ex230, address information indicating the absolute position on the disc is recorded in advance by the change of the groove shape.
- This address information includes information for specifying the position of the recording block ex231, which is a unit for recording data, and the recording and reproducing apparatus identifies the recording block by reproducing the information track ex230 and reading the address information.
- the recording medium ex215 includes a data recording area ex233, an inner circumference area ex232, and an outer circumference area ex234.
- An area used to record user data is data recording area ex233, and inner circumference area ex232 and outer circumference area ex234 arranged on the inner circumference or the outer circumference of data recording area ex233 are used for specific applications other than user data recording. Used.
- the information reproducing / recording unit ex400 reads / writes encoded audio data, video data, or encoded data obtained by multiplexing those data from / to the data recording area ex233 of such a recording medium ex215.
- an optical disc such as a single layer DVD or BD has been described as an example, but the optical disc is not limited to these, and may be an optical disc having a multilayer structure and capable of recording other than the surface.
- multi-dimensional recording / reproduction such as recording information in the same place of the disc using light of colors of different wavelengths, recording layers of information different from different angles, etc. It may be an optical disc.
- the digital broadcasting system ex200 it is possible to receive data from the satellite ex202 and the like by the car ex210 having the antenna ex205 and reproduce a moving image on a display device such as a car navigation system ex211 which the car ex210 has.
- the configuration of the car navigation system ex211 may be, for example, a configuration in which a GPS reception unit is added, and the same may be considered for the computer ex111, the mobile phone ex114, and the like.
- the terminal such as the above-mentioned mobile phone ex114 is, like the television ex300, in addition to a transceiving type terminal having both an encoder and a decoder, a transmitter terminal of only an encoder and a receiver terminal of only a decoder.
- the implementation style of can be considered.
- the image processing apparatus shown in the first embodiment is realized as an LSI, which is typically a semiconductor integrated circuit.
- the realized form is shown in FIG.
- the frame memory 502 is implemented on a DRAM, and other circuits and memories are configured on an LSI.
- a bit stream buffer for storing the encoded stream may be implemented on the DRAM.
- an LSI may be called an IC, a system LSI, a super LSI, or an ultra LSI depending on the degree of integration.
- the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible.
- a field programmable gate array FPGA
- a reconfigurable processor that can reconfigure connection and setting of circuit cells in the LSI may be used.
- a semiconductor chip on which the image processing apparatus of the present embodiment is integrated and a display for drawing an image can be combined to construct a drawing device according to various applications.
- the present invention can be used as an information drawing means in a mobile phone, a television, a digital video recorder, a digital video camera, car navigation and the like.
- the display can be combined with a cathode ray tube (CRT), a liquid crystal, a flat display such as a PDP (plasma display panel) and an organic EL, a projection type display typified by a projector, and the like.
- the LSI in the present embodiment performs encoding processing or decoding processing in cooperation with a DRAM (Dynamic Random Access Memory) including a bit stream buffer for storing a coded stream, a frame memory for storing an image, and the like. You may Further, the LSI in the present embodiment may cooperate with another storage device such as an eDRAM (embeded DRAM), a static random access memory (SRAM), or a hard disk instead of a DRAM.
- eDRAM embeded DRAM
- SRAM static random access memory
- the image coding device, the image processing device, the image coding method, and the image decoding method described in the above embodiments are typically realized by an LSI which is an integrated circuit.
- FIG. 40 shows a configuration of an LSI ex500 that is made into one chip.
- the LSI ex 500 includes elements ex 502 to ex 509 described below, and the elements are connected via the bus ex 510.
- the power supply circuit unit ex505 starts up to an operable state by supplying power to each unit when the power is on.
- the LSI ex 500 receives an input of an AV signal from the microphone ex 117, the camera ex 113, and the like by the AV I / O ex 509.
- the input AV signal is temporarily stored in an external memory ex 511 such as an SDRAM.
- the accumulated data is appropriately divided into a plurality of times according to the processing amount and the processing speed, and sent to the signal processing unit ex507.
- the signal processing unit ex 507 performs coding of an audio signal and / or coding of a video signal.
- the coding process of the video signal is the coding process described in the above embodiment.
- the signal processing unit ex 507 further performs processing such as multiplexing of encoded audio data and encoded video data as needed, and outputs the multiplexed data from the stream I / O ex 504 to the outside.
- the output bit stream is transmitted to the base station ex 107 or written to the recording medium ex 215.
- the LSI ex 500 is, based on control of the microcomputer (microcomputer) ex 502, encoded data obtained from the base station ex 107 by the stream I / O ex 504 or from the recording medium ex 215 The encoded data obtained by reading is temporarily stored in the memory ex 511 or the like. Based on the control of the microcomputer ex502, the accumulated data is divided into a plurality of times as appropriate according to the processing amount and processing speed and sent to the signal processing unit ex507, and the signal processing unit ex507 decodes audio data and / or video data Decoding is performed.
- the video signal decoding process is the decoding process described in each of the above embodiments.
- the decoded output signal is output from the AV I / O ex 509 to the monitor ex 219 or the like through the memory ex 511 or the like as appropriate.
- the memory controller ex503 is used.
- the memory ex 511 has been described as an external configuration of the LSI ex 500, but may be included in the LSI ex 500.
- the LSI ex 500 may be integrated into one chip or a plurality of chips.
- LSI LSI
- IC system LSI
- super LSI ultra LSI
- the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible.
- a field programmable gate array FPGA
- a reconfigurable processor that can reconfigure connection and setting of circuit cells in the LSI may be used.
- the invention can be used in a variety of applications. For example, it can be used for high-resolution information display devices such as televisions, digital video recorders, car navigation systems, mobile phones, digital cameras, digital video cameras, or imaging devices, and the utility value is high.
- high-resolution information display devices such as televisions, digital video recorders, car navigation systems, mobile phones, digital cameras, digital video cameras, or imaging devices, and the utility value is high.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
Description
画像を符号化する画像符号化装置は、画像を構成する各ピクチャを16x16画素でそれぞれが構成される複数のマクロブロック(Macroblock、略してMBと呼ぶこともある)に分割する。そして、画像符号化装置は、ラスタースキャン順に各マクロブロックを符号化する。画像符号化装置は、画像を符号化し圧縮することにより、符号化ストリームを生成する。画像を復号する画像処理装置は、この符号化ストリームをラスタースキャン順でマクロブロック毎に復号し、元の画像の各ピクチャを再生する。
(1-1.概要)
まず、本実施の形態に係る画像処理装置の概要について説明する。本実施の形態に係る画像処理装置は、動き補償を含む符号化処理によって生成された符号化ストリームを復号する。符号化において、動き補償のサイズ、すなわち、予測ユニットのサイズは、可変であり、最大で64x64画素である。
次に、本実施の形態に係る画像処理装置の構成について説明する。
次に、本実施の形態に係る画像処理装置の動作を説明する。本実施の形態に係る画像処理装置が復号する符号化ストリームは、符号化ユニット(CU)と、変換ユニット(TU)と、予測ユニット(PU)とで構成される。
このように、画像処理装置は、予測ユニットのサイズが大きい場合でも、予測ユニットを分割して変換ユニットの処理順序で処理することにより、メモリ容量を小さくすることができる。さらに、画像処理装置は、参照画像の転送処理と、動き補償の処理をパイプライン状に行うことで、処理の遅延を小さくすることができる。また、画像処理装置は、再構成処理およびインループフィルタ処理の開始を早くすることができる。したがって、復号処理全体が高速化する。
なお、本実施の形態に係る画像処理装置は、予測ユニットの分割後の複数のブロックに対して、変換ユニットの処理順序で1つずつ、参照画像位置算出、参照画像サイズ算出、参照画像転送、動き補償演算、および、予測画像出力を行う。しかし、画像処理装置は、複数のブロックに対する処理を2つずつ行ってもよいし、3つずつ行ってもよいし、256画素相当分のブロック毎に行ってもよい。
(2-1.概要)
まず、本実施の形態に係る画像処理装置の概要について説明する。本実施の形態に係る画像処理装置は、動き補償を含む符号化処理によって生成された符号化ストリームを復号する。符号化において、動き補償のサイズ、すなわち、予測ユニットのサイズは、可変であり、最大で64x64画素である。
図2は、本実施の形態に係る画像処理装置の構成図である。図3は、本実施の形態に係る動き補償部506の周辺の構成図である。本実施の形態に係る画像処理装置の構成は、実施の形態1と同様であるので、説明を省略する。
本実施の形態では、実施の形態1と同様に、図4A~図6Bに示された符号化ストリームの構造が用いられる。本実施の形態に係る画像処理装置の動作フローは、図7および図8に示された実施の形態1の動作フローと同様であるので、説明を省略する。
これにより、本実施の形態に係る画像処理装置は、符号化ユニットに含まれるすべての予測ユニットに対して、変換ユニットと同じ処理順序で処理を行うことができる。そのため、本実施の形態に係る画像処理装置は、実施の形態1の場合よりも復号処理を高速化することができる可能性がある。
なお、本実施の形態に係る画像処理装置は、符号化ユニットに含まれるすべての予測ユニットの分割後の複数のブロックに対して、変換ユニットの処理順序で1つずつ、参照画像転送、動き補償演算、および、予測画像出力を行う。しかし、画像処理装置は、複数のブロックに対する処理を2つずつ行ってもよいし、3つずつ行ってもよいし、256画素相当分のブロック毎に行ってもよい。
(3-1.概要)
まず、本実施の形態に係る画像処理装置の概要について説明する。本実施の形態に係る画像処理装置は、動き補償を含む符号化処理によって生成された符号化ストリームを復号する。符号化において、動き補償のサイズ、すなわち、予測ユニットのサイズは、可変であり、最大で64x64画素である。
図2は、本実施の形態に係る画像処理装置の構成図である。図3は、本実施の形態に係る動き補償部506の周辺の構成図である。本実施の形態に係る画像処理装置の構成は、実施の形態1と同様であるので、説明を省略する。
本実施の形態では、実施の形態1と同様に、図4A~図6Bに示された符号化ストリームの構造が用いられる。本実施の形態に係る画像処理装置の動作フローは、図7および図8に示された実施の形態1の動作フローと同様であるので、説明を省略する。
このように、画像処理装置は、予測ユニットを分割せずに、変換ユニットと同じ処理順序で、予測ユニットの処理を行う。したがって、復号処理が高速化する。
なお、本実施の形態に係る画像処理装置は、符号化ユニットに含まれるすべての予測ユニットを変換ユニットの処理順序で1つずつ、参照画像転送、動き補償演算、および、予測画像出力を行う。しかし、画像処理装置は、複数の予測ユニットに対する処理を2つずつ行ってもよいし、3つずつ行ってもよいし、256画素相当分のブロック毎に行ってもよい。
(4-1.概要)
まず、本実施の形態に係る画像処理装置の概要について説明する。本実施の形態に係る画像処理装置は、動き補償を含む符号化処理によって生成された符号化ストリームを復号する。符号化において、面内予測のサイズ、すなわち、予測ユニットのサイズは、可変であり、最大で64x64画素である。
次に、本実施の形態に係る画像処理装置の構成について説明する。図2は本実施の形態に係る画像処理装置の構成図である。本実施の形態に係る画像処理装置の全体構成は、実施の形態1と同様であるので、説明を省略する。
図23は、本実施の形態に係る面内予測の動作を示すフローチャートである。図22に示された予測モード演算部515および面内予測部507の動作について図23を用いて説明する。
本実施の形態に係る画像処理装置は、予測ユニットのサイズが大きい場合でも、予測ユニットを分割して、変換ユニットの処理順序で、面内予測処理を実行する。これにより、必要なメモリ容量が小さくなる。また、再構成処理およびインループフィルタ処理のより早い開始が可能になる。よって、復号処理全体が高速化する。
なお、本実施の形態に係る画像処理装置は、予測ユニットの分割後の複数のブロックに対して、変換ユニットの処理順序で1つずつ、参照周辺画素位置算出、面内予測演算、および、予測画像出力を行う。しかし、画像処理装置は、複数のブロックに対する処理を2つずつ行ってもよいし、3つずつ行ってもよいし、256画素相当分のブロック毎に行ってもよい。
以下、実施の形態5~12において、上述の複数の実施の形態で示された特徴的な構成要素を備える画像処理装置を示す。
実施の形態6に係る画像処理装置は、実施の形態5に係る画像処理装置2400の具体的な構成例である。
図27Aは、実施の形態7に係る画像処理装置の構成を示す図である。図27Aに示された画像処理装置2600は、実施の形態5に係る画像処理装置2400と同様に、符号化処理によって符号化された画像を復号する。この符号化処理は、実施の形態5と同様に、1以上の予測ユニットに対する予測処理と、その1以上の予測ユニットを含む領域に含まれる複数の変換ユニットに対する周波数変換処理とを含む。
図29Aは、実施の形態8に係る画像処理装置の構成を示す図である。図29Aに示された画像処理装置2800は、実施の形態5に係る画像処理装置2400と同様に、符号化処理によって符号化された画像を復号する。この符号化処理は、実施の形態5と同様に、1以上の予測ユニットに対する予測処理と、その1以上の予測ユニットを含む領域に含まれる複数の変換ユニットに対する周波数変換処理とを含む。
図30Aは、実施の形態9に係る画像処理装置の構成を示す図である。図30Aに示された画像処理装置2900は、実施の形態5に係る画像処理装置2400と同様に、符号化処理によって符号化された画像を復号する。この符号化処理は、実施の形態5と同様に、1以上の予測ユニットに対する予測処理と、その1以上の予測ユニットを含む領域に含まれる複数の変換ユニットに対する周波数変換処理とを含む。
図31Aは、実施の形態10に係る画像処理装置の構成を示す図である。図31Aに示された画像処理装置3100は、符号化処理によって符号化された画像を復号する。この符号化処理は、1以上の予測ユニットに対する予測処理と、その1以上の予測ユニットを含む領域に含まれる複数の変換ユニットに対する周波数変換処理とを含む。
図32Aは、実施の形態11に係る画像処理装置の構成を示す図である。図32Aに示された画像処理装置3200は、実施の形態10に係る画像処理装置3100と同様に、符号化処理によって符号化された画像を復号する。この符号化処理は、実施の形態10と同様に、1以上の予測ユニットに対する予測処理と、その1以上の予測ユニットを含む領域に含まれる複数の変換ユニットに対する周波数変換処理とを含む。
図33Aは、実施の形態12に係る画像処理装置の構成を示す図である。図33Aに示された画像処理装置3000は、符号化処理によって符号化された画像を復号する。この符号化処理は、予測ユニットの予測モードに従って予測ユニットに対応する予測画像を生成する面内予測処理を含む。
上記各実施の形態で示した画像符号化方法および画像復号方法の構成を実現するためのプログラムを記憶メディアに記録することにより、上記各実施の形態で示した処理を独立したコンピュータシステムにおいて簡単に実施することが可能となる。記憶メディアは、磁気ディスク、光ディスク、光磁気ディスク、ICカード、半導体メモリ等、プログラムを記録できるものであればよい。
本実施の形態では、実施の形態1に示した画像処理装置を、典型的には半導体集積回路であるLSIとして実現する。実現した形態を図39に示す。フレームメモリ502をDRAM上に実現し、その他の回路およびメモリをLSI上に構成している。符号化ストリームを格納するビットストリームバッファをDRAM上に実現してもよい。
上記各実施の形態で示した画像符号化装置、画像処理装置、画像符号化方法および画像復号方法は、典型的には集積回路であるLSIで実現される。一例として、図40に1チップ化されたLSIex500の構成を示す。LSIex500は、以下に説明する要素ex502~ex509を備え、各要素はバスex510を介して接続している。電源回路部ex505は電源がオン状態の場合に各部に対して電力を供給することで動作可能な状態に起動する。
502 フレームメモリ
503 可変長復号部
504 逆量子化部
505 逆周波数変換部
506 動き補償部
507 面内予測部
508 再構成部
509 再構成画像メモリ
510 インループフィルタ部
511 動きベクトル演算部
512 DMA制御部
513 参照画像記憶部
514 予測画像記憶部
515 予測モード演算部
2302、2303 参照周辺画素
2400、2500、2600、2800、2900、3000、3100、3200 画像処理装置
2401、2501、2602、2801、2901 予測部
2511 情報取得部
2512 予測画像生成部
2513 予測画像出力部
2601、3001、3201 分割部
2802、2902 再構成処理部
2903 インループフィルタ処理部
3002、3101、3202 処理部
ex100 コンテンツ供給システム
ex101 インターネット
ex102 インターネットサービスプロバイダ
ex103 ストリーミングサーバ
ex104 電話網
ex106、ex107、ex108、ex109、ex110 基地局
ex111 コンピュータ
ex112 PDA(Personal Digital Assistant)
ex113、ex116 カメラ
ex114 携帯電話
ex115 ゲーム機
ex117 マイク
ex200 デジタル放送用システム
ex201 放送局
ex202 放送衛星(衛星)
ex203 ケーブル
ex204、ex205 アンテナ
ex210 車
ex211 カーナビゲーション(カーナビ)
ex212 再生装置
ex213、ex219 モニタ
ex214、ex215、ex216 記録メディア
ex217 セットトップボックス(STB)
ex218 リーダ/レコーダ
ex220 リモートコントローラ
ex230 情報トラック
ex231 記録ブロック
ex232 内周領域
ex233 データ記録領域
ex234 外周領域
ex300 テレビ(受信機)
ex301 チューナ
ex302 変調/復調部
ex303 多重/分離部
ex304 音声信号処理部
ex305 映像信号処理部
ex306、ex507 信号処理部
ex307 スピーカ
ex308 表示部
ex309 出力部
ex311、ex505 電源回路部
ex312 操作入力部
ex313 ブリッジ
ex314 スロット部
ex315 ドライバ
ex316 モデム
ex317 インタフェース部
ex318、ex319、ex320、ex321、ex404 バッファ
ex400 情報再生/記録部
ex401 光ヘッド
ex402 変調記録部
ex403 再生復調部
ex405 ディスクモータ
ex406 サーボ制御部
ex407 システム制御部
ex500 LSI
ex502 マイコン(マイクロコンピュータ)
ex503 メモリコントローラ
ex504 ストリームI/O
ex509 AV I/O
ex510 バス
ex511 メモリ
Claims (23)
- 1以上の予測ユニットに対する予測処理と、前記1以上の予測ユニットを含む領域に含まれる複数の変換ユニットに対する周波数変換処理とを含む符号化処理によって符号化された画像を復号するための処理を実行する画像処理装置であって、
前記複数の変換ユニットに対して予め定められている変換順序に依存させて、前記1以上の予測ユニットの予測画像に関する画像処理を実行する処理部を備える
画像処理装置。 - 前記処理部は、前記変換順序を取得し、取得された前記変換順序に沿って、前記画像処理を実行する
請求項1に記載の画像処理装置。 - 前記処理部は、変換ユニットに等しい、または、変換ユニットよりも小さいブロック毎に、前記画像処理を前記変換順序に沿って実行する
請求項1または2に記載の画像処理装置。 - 前記処理部は、前記予測画像を出力する処理を前記画像処理として前記変換順序に沿って実行する
請求項1~3のいずれか1項に記載の画像処理装置。 - 前記処理部は、前記1以上の予測ユニットに対して予め定められている予測順序が前記変換順序に沿っていない場合、前記画像処理を前記変換順序に沿って実行する
請求項1~4のいずれか1項に記載の画像処理装置。 - 前記画像処理装置は、さらに、前記1以上の予測ユニットを前記複数の変換ユニットのエッジに沿って複数のブロックに分割する分割部を備え、
前記処理部は、前記1以上の予測ユニットを分割することにより得られたブロック毎に、前記画像処理を前記変換順序に沿って実行する
請求項1~5のいずれか1項に記載の画像処理装置。 - 前記分割部は、前記1以上の予測ユニットに対応する1つの予測ユニットを前記複数の変換ユニットの前記エッジに沿って前記複数のブロックに分割し、
前記処理部は、前記1つの予測ユニットを分割することにより得られたブロック毎に、前記画像処理を前記変換順序に沿って実行する
請求項6に記載の画像処理装置。 - 前記分割部は、前記複数の変換ユニットを含む符号化ユニットに含まれる複数の予測ユニットを前記1以上の予測ユニットとして前記複数の変換ユニットの前記エッジに沿って前記複数のブロックに分割し、
前記処理部は、前記複数の予測ユニットを分割することにより得られたブロック毎に、前記画像処理を前記変換順序に沿って実行する
請求項6に記載の画像処理装置。 - 前記処理部は、前記予測画像を生成する処理と、生成された前記予測画像を出力する処理とを前記画像処理として前記変換順序に沿って実行する
請求項1~8のいずれか1項に記載の画像処理装置。 - 前記処理部は、前記予測画像を生成するための情報を取得する処理と、取得された前記情報を用いて前記予測画像を生成する処理と、生成された前記予測画像を出力する処理とを前記画像処理として前記変換順序に沿って実行する
請求項1~9のいずれか1項に記載の画像処理装置。 - 前記処理部は、動き補償処理によって生成された前記予測画像を出力する処理を前記画像処理として前記変換順序に沿って実行する
請求項1~10のいずれか1項に記載の画像処理装置。 - 前記処理部は、動き補償処理を実行することにより前記予測画像を生成する処理と、前記予測画像を出力する処理とを前記画像処理として前記変換順序に沿って実行する
請求項1~11のいずれか1項に記載の画像処理装置。 - 前記処理部は、動き補償処理を実行するための情報を取得する処理と、取得された前記情報を用いて前記動き補償処理を実行することにより前記予測画像を生成する処理と、生成された前記予測画像を出力する処理とを前記画像処理として前記変換順序に沿って実行する
請求項1~12のいずれか1項に記載の画像処理装置。 - 前記処理部は、面内予測処理によって生成された前記予測画像を出力する処理を前記画像処理として前記変換順序に沿って実行する
請求項1~10のいずれか1項に記載の画像処理装置。 - 前記処理部は、面内予測処理を実行することにより前記予測画像を生成する処理と、生成された前記予測画像を出力する処理とを前記画像処理として前記変換順序に沿って実行する
請求項1~10および14のいずれか1項に記載の画像処理装置。 - 前記処理部は、面内予測処理を実行するための情報を取得する処理と、取得された前記情報を用いて前記面内予測処理を実行することにより前記予測画像を生成する処理と、生成された前記予測画像を出力する処理とを前記画像処理として前記変換順序に沿って実行する
請求項1~10、14および15のいずれか1項に記載の画像処理装置。 - 前記処理部は、前記予測画像を用いて前記画像を再構成する処理を前記画像処理として前記変換順序に沿って実行する
請求項1~16のいずれか1項に記載の画像処理装置。 - 前記処理部は、さらに、再構成された前記画像に対するフィルタ処理であるインループフィルタ処理を前記変換順序に沿って実行する
請求項17に記載の画像処理装置。 - 前記処理部は、さらに、前記インループフィルタ処理が実行された前記画像をフレームメモリに格納する処理を前記変換順序に沿って実行する
請求項18に記載の画像処理装置。 - 予測ユニットの予測モードに従って前記予測ユニットの予測画像を生成する面内予測処理を含む符号化処理によって符号化された画像を復号するための処理を実行する画像処理装置であって、
前記予測ユニットを複数のブロックに分割する分割部と、
前記複数のブロックのそれぞれに対して、前記予測ユニットの前記予測モードに従って面内予測を実行する処理部とを備える
画像処理装置。 - 1以上の予測ユニットに対する予測処理と、前記1以上の予測ユニットを含む領域に含まれる複数の変換ユニットに対する周波数変換処理とを含む符号化処理によって符号化された画像を復号するための処理を実行する画像処理方法であって、
前記複数の変換ユニットに対して予め定められている変換順序に依存させて、前記1以上の予測ユニットの予測画像に関する画像処理を実行する
画像処理方法。 - 請求項18に記載の画像処理方法をコンピュータに実行させるための
プログラム。 - 1以上の予測ユニットに対する予測処理と、前記1以上の予測ユニットを含む領域に含まれる複数の変換ユニットに対する周波数変換処理とを含む符号化処理によって符号化された画像を復号するための処理を実行する集積回路であって、
前記複数の変換ユニットに対して予め定められている変換順序に依存させて、前記1以上の予測ユニットの予測画像に関する処理を実行する処理部を備える
集積回路。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201280003052.3A CN103222266B (zh) | 2011-11-21 | 2012-08-22 | 图像处理装置以及图像处理方法 |
JP2013509334A JP5899447B2 (ja) | 2011-11-21 | 2012-08-22 | 画像処理装置および画像処理方法 |
US13/823,129 US9674528B2 (en) | 2011-11-21 | 2012-08-22 | Image processing apparatus and image processing method |
US15/586,797 US20170238016A1 (en) | 2011-11-21 | 2017-05-04 | Image processing apparatus and image processing method |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011254421 | 2011-11-21 | ||
JP2011-254421 | 2011-11-21 |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/823,129 A-371-Of-International US9674528B2 (en) | 2011-11-21 | 2012-08-22 | Image processing apparatus and image processing method |
US15/586,797 Continuation US20170238016A1 (en) | 2011-11-21 | 2017-05-04 | Image processing apparatus and image processing method |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2013076888A1 true WO2013076888A1 (ja) | 2013-05-30 |
Family
ID=48469360
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2012/005256 WO2013076888A1 (ja) | 2011-11-21 | 2012-08-22 | 画像処理装置および画像処理方法 |
Country Status (4)
Country | Link |
---|---|
US (2) | US9674528B2 (ja) |
JP (5) | JP5899447B2 (ja) |
CN (2) | CN108040255A (ja) |
WO (1) | WO2013076888A1 (ja) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015145504A1 (ja) * | 2014-03-25 | 2015-10-01 | 株式会社ソシオネクスト | 画像復号装置、画像復号方法、及び集積回路 |
JPWO2015146646A1 (ja) * | 2014-03-28 | 2017-04-13 | ソニー株式会社 | 画像復号装置および方法 |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8913656B2 (en) * | 2011-10-14 | 2014-12-16 | Mediatek Inc. | Method and apparatus for in-loop filtering |
JP5899447B2 (ja) * | 2011-11-21 | 2016-04-06 | パナソニックIpマネジメント株式会社 | 画像処理装置および画像処理方法 |
CN111885380B (zh) * | 2015-03-23 | 2024-04-12 | Lg电子株式会社 | 在内预测模式的基础上处理图像的方法及其装置 |
KR20180107083A (ko) * | 2016-02-16 | 2018-10-01 | 삼성전자주식회사 | 적응적인 블록 파티셔닝 방법 및 장치 |
KR102584349B1 (ko) * | 2016-03-28 | 2023-10-04 | 로즈데일 다이나믹스 엘엘씨 | 인터 예측 모드 기반 영상 처리 방법 및 이를 위한 장치 |
EP3506632A4 (en) | 2016-10-10 | 2019-08-07 | Samsung Electronics Co., Ltd. | DEVICE AND METHOD FOR ENCODING OR DECODING IMAGE CONTOUR ENCODING UNIT |
WO2019059107A1 (ja) * | 2017-09-20 | 2019-03-28 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ | 符号化装置、復号装置、符号化方法及び復号方法 |
AU2018217333A1 (en) * | 2018-08-17 | 2020-03-05 | Canon Kabushiki Kaisha | Method, apparatus and system for encoding and decoding a transformed block of video samples |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006311526A (ja) * | 2005-03-31 | 2006-11-09 | Matsushita Electric Ind Co Ltd | 画像復号装置、画像復号方法、画像復号プログラム、画像復号集積回路 |
JP2008079170A (ja) * | 2006-09-25 | 2008-04-03 | Sony Corp | 画像データ処理装置、画像データ処理方法、画像データ処理方法のプログラム及び画像データ処理方法のプログラムを記録した記録媒体 |
WO2011004577A1 (ja) * | 2009-07-06 | 2011-01-13 | パナソニック株式会社 | 画像復号装置、画像符号化装置、画像復号方法、画像符号化方法、プログラムおよび集積回路 |
WO2011161949A1 (ja) * | 2010-06-23 | 2011-12-29 | パナソニック株式会社 | 画像復号装置、画像復号方法、集積回路およびプログラム |
WO2012046435A1 (ja) * | 2010-10-04 | 2012-04-12 | パナソニック株式会社 | 画像処理装置、画像符号化方法および画像処理方法 |
Family Cites Families (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6426975B1 (en) * | 1997-07-25 | 2002-07-30 | Matsushita Electric Industrial Co., Ltd. | Image processing method, image processing apparatus and data recording medium |
GB2348064A (en) * | 1999-03-16 | 2000-09-20 | Mitsubishi Electric Inf Tech | Motion vector field encoding |
US7728878B2 (en) * | 2004-12-17 | 2010-06-01 | Mitsubishi Electric Research Labortories, Inc. | Method and system for processing multiview videos for view synthesis using side information |
US8588304B2 (en) | 2005-03-31 | 2013-11-19 | Panasonic Corporation | Video decoding device, video decoding method, video decoding program, and video decoding integrated circuit |
JP5089878B2 (ja) * | 2005-10-28 | 2012-12-05 | パナソニック株式会社 | 画像符号化装置 |
KR101266168B1 (ko) * | 2006-08-16 | 2013-05-21 | 삼성전자주식회사 | 영상의 부호화, 복호화 방법 및 장치 |
WO2008047300A2 (en) * | 2006-10-16 | 2008-04-24 | Nokia Corporation | System and method for using parallelly decodable slices for multi-view video coding |
US8428133B2 (en) * | 2007-06-15 | 2013-04-23 | Qualcomm Incorporated | Adaptive coding of video block prediction mode |
CN101170688B (zh) * | 2007-11-26 | 2010-12-01 | 电子科技大学 | 一种宏块模式的快速选择方法 |
WO2010091504A1 (en) * | 2009-02-13 | 2010-08-19 | Research In Motion Limited | In-loop deblocking for intra-coded images or frames |
JP5326828B2 (ja) * | 2009-06-03 | 2013-10-30 | 富士通株式会社 | 動画像符号化装置及び動画像符号化方法 |
KR101474756B1 (ko) * | 2009-08-13 | 2014-12-19 | 삼성전자주식회사 | 큰 크기의 변환 단위를 이용한 영상 부호화, 복호화 방법 및 장치 |
KR102195687B1 (ko) * | 2010-01-12 | 2020-12-28 | 엘지전자 주식회사 | 비디오 신호의 처리 방법 및 장치 |
WO2011126277A2 (en) * | 2010-04-05 | 2011-10-13 | Samsung Electronics Co., Ltd. | Low complexity entropy-encoding/decoding method and apparatus |
US8804826B2 (en) * | 2010-04-13 | 2014-08-12 | Blackberry Limited | Methods and devices for incorporating deblocking into encoded video |
WO2011158867A1 (ja) | 2010-06-17 | 2011-12-22 | シャープ株式会社 | 画像復号装置、及び画像符号化装置 |
WO2012046436A1 (ja) * | 2010-10-05 | 2012-04-12 | 旭化成株式会社 | 文書分析装置、文書分析方法および文書分析プログラム |
JP5899447B2 (ja) * | 2011-11-21 | 2016-04-06 | パナソニックIpマネジメント株式会社 | 画像処理装置および画像処理方法 |
-
2012
- 2012-08-22 JP JP2013509334A patent/JP5899447B2/ja not_active Expired - Fee Related
- 2012-08-22 CN CN201810092859.8A patent/CN108040255A/zh active Pending
- 2012-08-22 US US13/823,129 patent/US9674528B2/en active Active
- 2012-08-22 CN CN201280003052.3A patent/CN103222266B/zh not_active Expired - Fee Related
- 2012-08-22 WO PCT/JP2012/005256 patent/WO2013076888A1/ja active Application Filing
-
2015
- 2015-12-08 JP JP2015239647A patent/JP6089311B2/ja not_active Expired - Fee Related
-
2017
- 2017-01-17 JP JP2017006022A patent/JP6238096B2/ja active Active
- 2017-05-04 US US15/586,797 patent/US20170238016A1/en not_active Abandoned
- 2017-10-18 JP JP2017202034A patent/JP6390883B2/ja not_active Expired - Fee Related
-
2018
- 2018-08-07 JP JP2018148786A patent/JP2018191338A/ja active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006311526A (ja) * | 2005-03-31 | 2006-11-09 | Matsushita Electric Ind Co Ltd | 画像復号装置、画像復号方法、画像復号プログラム、画像復号集積回路 |
JP2008079170A (ja) * | 2006-09-25 | 2008-04-03 | Sony Corp | 画像データ処理装置、画像データ処理方法、画像データ処理方法のプログラム及び画像データ処理方法のプログラムを記録した記録媒体 |
WO2011004577A1 (ja) * | 2009-07-06 | 2011-01-13 | パナソニック株式会社 | 画像復号装置、画像符号化装置、画像復号方法、画像符号化方法、プログラムおよび集積回路 |
WO2011161949A1 (ja) * | 2010-06-23 | 2011-12-29 | パナソニック株式会社 | 画像復号装置、画像復号方法、集積回路およびプログラム |
WO2012046435A1 (ja) * | 2010-10-04 | 2012-04-12 | パナソニック株式会社 | 画像処理装置、画像符号化方法および画像処理方法 |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015145504A1 (ja) * | 2014-03-25 | 2015-10-01 | 株式会社ソシオネクスト | 画像復号装置、画像復号方法、及び集積回路 |
JPWO2015145504A1 (ja) * | 2014-03-25 | 2017-04-13 | 株式会社ソシオネクスト | 画像復号装置、画像復号方法、及び集積回路 |
US10306255B2 (en) | 2014-03-25 | 2019-05-28 | Socionext Inc. | Image decoding device, image decoding method, and integrated circuit |
JPWO2015146646A1 (ja) * | 2014-03-28 | 2017-04-13 | ソニー株式会社 | 画像復号装置および方法 |
US10298927B2 (en) | 2014-03-28 | 2019-05-21 | Sony Corporation | Image decoding device and method |
Also Published As
Publication number | Publication date |
---|---|
US20130315312A1 (en) | 2013-11-28 |
CN103222266A (zh) | 2013-07-24 |
US9674528B2 (en) | 2017-06-06 |
JP2018191338A (ja) | 2018-11-29 |
JP6390883B2 (ja) | 2018-09-19 |
JPWO2013076888A1 (ja) | 2015-04-27 |
JP6238096B2 (ja) | 2017-11-29 |
JP2017092977A (ja) | 2017-05-25 |
US20170238016A1 (en) | 2017-08-17 |
JP6089311B2 (ja) | 2017-03-08 |
JP5899447B2 (ja) | 2016-04-06 |
CN108040255A (zh) | 2018-05-15 |
CN103222266B (zh) | 2018-02-23 |
JP2016034161A (ja) | 2016-03-10 |
JP2018029379A (ja) | 2018-02-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6390883B2 (ja) | 画像処理装置 | |
JP5518069B2 (ja) | 画像復号装置、画像符号化装置、画像復号方法、画像符号化方法、プログラムおよび集積回路 | |
WO2012046435A1 (ja) | 画像処理装置、画像符号化方法および画像処理方法 | |
EP2618579B1 (en) | Image decoding device, image encoding device, method for said devices, program, integrated circuit, and transcoder | |
WO2011161949A1 (ja) | 画像復号装置、画像復号方法、集積回路およびプログラム | |
CN102197652B (zh) | 解码装置、解码方法、程序以及集成电路 | |
WO2011111341A1 (ja) | 動画像復号装置、動画像符号化装置、動画像復号回路及び動画像復号方法 | |
JP5999515B2 (ja) | 画像処理装置、及び、画像処理方法 | |
WO2013108330A1 (ja) | 画像復号装置、画像符号化装置、画像復号方法および画像符号化方法 | |
JP6260921B2 (ja) | 画像処理装置および画像処理方法 | |
WO2013076897A1 (ja) | 画像処理装置および画像処理方法 | |
JPWO2010131422A1 (ja) | 画像復号装置、集積回路、画像復号方法及び画像復号システム | |
WO2011129052A1 (ja) | 画像復号装置、画像符号化装置、画像復号方法および画像符号化方法 | |
JP2011182132A (ja) | 画像符号化装置、画像復号装置、画像符号化方法、画像復号方法、集積回路およびプログラム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
ENP | Entry into the national phase |
Ref document number: 2013509334 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13823129 Country of ref document: US |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12852258 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 12852258 Country of ref document: EP Kind code of ref document: A1 |