WO2019225337A1 - Encoding device, decoding device, encoding method, decoding method, encoding program and decoding program - Google Patents
Encoding device, decoding device, encoding method, decoding method, encoding program and decoding program Download PDFInfo
- Publication number
- WO2019225337A1 WO2019225337A1 PCT/JP2019/018568 JP2019018568W WO2019225337A1 WO 2019225337 A1 WO2019225337 A1 WO 2019225337A1 JP 2019018568 W JP2019018568 W JP 2019018568W WO 2019225337 A1 WO2019225337 A1 WO 2019225337A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- encoding
- decoding
- auxiliary information
- unit
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/46—Embedding additional information in the video signal during the compression process
- H04N19/463—Embedding additional information in the video signal during the compression process by compressing encoding parameters before transmission
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
Definitions
- the present invention relates to an encoding device, a decoding device, an encoding method, a decoding method, an encoding program, and a decoding program.
- MPEG-4 and H.264 are standards for compressing and encoding video data.
- H.264 / AVC, H.H. H.265 / HEVC (hereinafter referred to as “HEVC”) is known.
- HEVC H.H. H.265 / HEVC
- standardization of a new standard following HEVC is being studied.
- processing is performed in units obtained by dividing an image into rectangular blocks, the prediction block adjacent to the prediction target block is referred to, the pixel value of the prediction target block is predicted, and the prediction residual signal only Is used.
- HEVC intra-prediction coding method for predicting a pixel signal by closing in a frame will be described.
- the entire screen is divided into blocks of 64 pixels ⁇ 64 pixels (hereinafter referred to as “64 ⁇ 64”), and each unit is defined as a CTU (Coding Tree Unit).
- the CTU can be divided into four squares called CUs (Coding Units).
- CUs Coding Units
- the CTU is divided into fine blocks.
- four types of 64 ⁇ 64, 32 ⁇ 32, 16 ⁇ 16, and 8 ⁇ 8 CU sizes can be used, and prediction processing is performed in units called PUs (Prediction Units) obtained by further dividing the CU. .
- PUs Prediction Units
- Each PU can selectively apply 35 types of prediction parameters. For example, a prediction parameter that minimizes a prediction residual signal with the original image is selected on the encoding side, and the prediction parameter and the prediction residual signal are decoded on the decoding side. Send to.
- the prediction method can be selected from three types of Planar prediction, DC (Direct Current) prediction, and directionality prediction, and 33 prediction parameters are assigned to the directionality prediction.
- the total number is 35.
- each prediction method performs prediction using pixel values of reference pixels located on the left and top of the prediction target block, and refers to one direction from the defined 33 directions in the directional prediction.
- a prediction pixel of the prediction target block is generated.
- Planar prediction the lower left and upper right of the prediction target block, the left and upper four pixels of the prediction target pixel are referred to, and each pixel in the prediction target block is predicted as a weighted average thereof.
- DC prediction a prediction value of a prediction target block is generated as a single average as the average of reference pixels located on the left and above the prediction target block.
- an input image is decomposed by Cartoon-Texture signal decomposition, and a non-synthetic component image represented by the sum of a Cartoon component and a non-synthetic Texture component, and a representative of the synthetic Texture
- the texture and the area information corresponding to the composite texture are transmitted.
- the area information is expressed by an image, and includes a synthesis area and a synthesis method corresponding to the area.
- the decoded image is obtained by adding the image reconstructed by the texture synthesis using the representative texture of the synthesized texture and the region information.
- an existing coding standard is used as a method of coding and decoding region information corresponding to a non-synthesized component image and a synthesized texture.
- the technique described in Patent Document 1 can encode an image with a large number of texture components with a smaller code amount.
- an object of the present invention is to provide a technique capable of transmitting and receiving a region to be reconfigured with a smaller code amount.
- One aspect of the present invention is an encoding device that encodes an image, a determination unit that determines whether or not an input image is a reconstruction target, and an image that is determined to be the reconstruction target
- An auxiliary information extraction unit that extracts auxiliary information that is information used for reconstruction, a conversion unit that converts the image determined to be the reconstruction target and obtains a converted image, and encodes and encodes the converted image
- An encoding unit that obtains data
- the conversion unit is an encoding device that converts the input image so as to have a smaller code amount than when the input image is encoded when the encoding unit encodes. is there.
- One embodiment of the present invention is the above encoding device, in which the determination unit acquires the estimated generated code amount and the estimated distortion amount and performs the rate distortion optimization to obtain the input image. It is determined whether or not to be reconfigured.
- one aspect of the present invention is the encoding device described above, in which the auxiliary information is the reconstruction target while maintaining the characteristics of the image that is determined to be the reconstruction image. This is information for inverse conversion into an image having a smaller code amount than the determined image.
- Another embodiment of the present invention is a decoding device that decodes encoded data obtained by encoding an image, a decoding unit that decodes input encoded data to obtain a decoded image, and the decoded image is reconstructed
- a determination unit that determines whether or not the image is a target image; and auxiliary information that is information used for reconstruction is acquired, and the decoded image that is determined to be the image to be reconstructed is converted to the auxiliary information.
- a reconfiguration unit that reconfigures the decoding device.
- One embodiment of the present invention is an encoding method by an encoding device that encodes an image, the determination step for determining whether or not the input image is a reconstruction target, and the reconstruction target Then, an auxiliary information extraction step for extracting auxiliary information, which is information for use in reconstruction, from the determined image, and an image determined to be the reconstruction target than when the input image is encoded
- the encoding method includes a conversion step of converting the code amount so as to obtain a converted image and an encoding step of encoding the converted image to obtain encoded data.
- Another embodiment of the present invention is a decoding method by a decoding device that decodes encoded data in which an image is encoded, the decoding step of decoding input encoded data to obtain a decoded image, and the decoded image Determining whether or not is an image to be reconstructed, and acquiring auxiliary information that is information for use in reconstruction, and decoding the image that has been determined to be the image to be reconstructed, And a reconstruction step of reconstructing using auxiliary information.
- one embodiment of the present invention is an encoding program for causing a computer to function as the above encoding device.
- one embodiment of the present invention is a decoding program for causing a computer to function as the above-described decoding device.
- a region to be reconfigured can be transmitted and received with a smaller code amount.
- the present invention is not limited to HEVC and intra prediction. That is, the present invention can be applied to image coding methods other than HEVC and inter prediction.
- reconstruction refers to a process of generating a pseudo image that matches a target region of an image by texture synthesis, image interpolation synthesis processing, or the like.
- the pseudo image referred to here is, for example, an image in which it is difficult to feel a difference from a subjective viewpoint as compared with an input image.
- the reconstruction target block is subjected to uniform image processing on the entire block so as to reduce the amount of information of the prediction residual in the HEVC intra prediction, and is input to the HEVC encoder.
- a block with low prediction accuracy in HEVC or a block related to a subject that does not need to accurately reproduce pixels of an image before encoding if a certain level of subjective image quality can be ensured is set as a reconstruction target block.
- the reconstruction target block is determined by determining whether or not uniform image processing is performed on the entire block.
- FIG. 1 shows a processing flow of the encoding apparatus according to the first embodiment of the present invention.
- the shape of the encoding processing block is determined from the input picture (step S101).
- the output block division shape follows CTU, CU, and PU as shown in FIG. 17, and this block is used as a unit of reconstruction processing on the decoding side and a unit of HEVC encoding processing.
- a method for determining the division shape in addition to a method for determining a uniform rectangle such as CTU, a CU division shape determined by rate distortion optimization as implemented in a HEVC test model (HM; HEVC Test Model)
- HM HEVC Test Model
- a determination method, a method of determining a result obtained by performing region division for each object used in image recognition as an approximation in block units, or the like can be used.
- the coding method determination process it is determined whether the block is divided into blocks by the block division process, whether to be a reconstruction target block or a non-reconstruction target block (step S102).
- the estimated generated code amount and the estimated distortion amount are respectively derived for the case of the reconfiguration target and the case of the non-reconfiguration target, and the determination is made by applying rate distortion optimization. The method can be used.
- auxiliary information to be transmitted to the decoding device to assist the reconstruction process is extracted from the reconstruction target block by the auxiliary information extraction process (step S104).
- the reconstruction process is a process of inversely transforming, on the decoding side, a block obtained by performing some kind of transformation described later on the block to be reconstructed.
- the auxiliary information extraction process for example, when reconstruction is performed by synthesizing a reconstruction target block by image synthesis, a representative texture used at the time of synthesis or a label identifying an object is extracted as auxiliary information.
- the extracted auxiliary information is entropy encoded by the auxiliary information entropy encoding process, and becomes encoded data of the auxiliary information.
- any encoding method such as Huffman encoding or run-length encoding can be used (step S105).
- the block to be reconstructed after the auxiliary information is extracted is converted into an image that can be transmitted with a smaller code amount by HEVC by image conversion processing (step S106).
- the reconstruction target block may be replaced with the average value of the block, and the prediction residual when predicting with an arbitrary or specific mode number in HEVC intra direction prediction is asymptotic to zero. Such conversion may be performed.
- the mode number of HEVC intra prediction used for conversion may be transmitted to the decoding side as a part of the auxiliary information, and the specific mode number of HEVC intra prediction corresponds to the reconstruction processing method on the decoding side.
- image conversion may be performed, and the correspondence relationship may be transmitted to the decoding side as part of the auxiliary information.
- an intra prediction mode number and a representative texture may be associated with each other and the corresponding relationship may be transmitted as auxiliary information to the decoding side.
- the image conversion method may be a method other than conversion based on HEVC intra prediction.
- An arbitrary conversion method capable of obtaining an output that does not exist in the input picture may be selected from those defined or previously defined in the course of the image conversion process, and the conversion method may be transmitted to the decoding side as auxiliary information.
- converted image (hereinafter referred to as “converted image”) is encoded by the converted image intra encoding process to obtain encoded data of the converted image (step S107).
- step S108 and step S109 The above processing is applied to all blocks in the order of processing (step S108 and step S109), and encoded data of auxiliary information and encoded data of a converted image are obtained as transmission information.
- FIG. 2 shows a configuration example of the encoding device 10 in the first embodiment.
- the encoding apparatus 10 includes a block division unit 101, an encoding scheme determination unit 102, an auxiliary information extraction unit 103, an auxiliary information entropy encoding unit 104, an image conversion unit 105, and an intra prediction unit. 107, a transform / quantization unit 108, an entropy coding unit 109, an inverse quantization / inverse transform unit 110, and a prediction memory 111.
- the block division unit 101 performs block division processing with the input picture as an input.
- the block division unit 101 outputs an input picture that has been divided into blocks.
- the encoding method determination unit 102 performs an encoding method determination process using an input picture divided into blocks as an input.
- the encoding method determination unit 102 outputs a determination result of the block encoding method.
- the auxiliary information extraction unit 103 performs auxiliary information extraction processing with the reconstruction target block and the reference block as inputs.
- the reference block is a block including a pixel to be referred to in the reconstruction process.
- the reference block is a block including a pixel referred to in the interpolation process.
- the auxiliary information extraction unit 103 outputs auxiliary information.
- the auxiliary information entropy encoding unit 104 performs entropy encoding on the input auxiliary information to obtain encoded data of the auxiliary information.
- the auxiliary information entropy encoding unit 104 outputs encoded data of auxiliary information.
- the image conversion unit 105 performs an image conversion process with the reconstruction target block as an input.
- the image conversion unit 105 outputs the converted block.
- the post-conversion block and the non-reconstruction target block are encoded by intra encoding.
- the prediction residual with the predicted image output from the intra prediction unit 107 is subjected to orthogonal transform and quantization by the transform / quantization unit 108 and encoded by the entropy coding unit 109.
- encoded data of the image is obtained.
- the entropy encoding unit 109 that encodes the prediction residual and the auxiliary information entropy encoding unit 104 that encodes auxiliary information are configured as separate functional blocks. You may be comprised by the same functional block. That is, the encoding residual encoding and the auxiliary information encoding may be performed by one encoding unit, for example, using a common entropy encoding scheme.
- the prediction residual quantized by the transform / quantization unit 108 is subjected to inverse quantization and inverse transform processing by the inverse quantization / inverse transform unit 110 and is stored in the prediction memory 111.
- the data stored in the prediction memory 111 is used for intra prediction processing by the intra prediction unit 107 and auxiliary information report extraction processing by the auxiliary information extraction unit 103.
- FIG. 3 shows a processing flow of the decoding device according to the first embodiment.
- the post-conversion image decoding process decodes the encoded data of the post-conversion image to obtain a block of the decoded image of the post-conversion image (step S201).
- the decoded image may be a unit image corresponding to the input image, or a unit image corresponding to a block obtained by blocking the input image. In the following processes, the description will be continued assuming that the decoded image is an image of a unit corresponding to a block.
- a block converted by the image conversion method used by the image conversion unit 105 of the encoding device 10 is determined as a reconstruction target block (step S202). For example, when the image conversion unit 105 of the encoding device 10 performs the process of uniformly replacing the reconstruction target block with the average value, the encoding method determination process is performed on the block obtained from the decoded image of the converted image. The processed block is determined as a reconstruction target block.
- the coding method determination process corresponds to the reconstruction target block based on the coding method applied by the auxiliary information entropy coding unit 104 of the coding device 10.
- the encoded data of the auxiliary information to be decoded is decoded (step S204).
- the auxiliary information and the reference block that can be referred to by the reconstruction target block are input, and the reconstruction process is performed (step S205).
- step S206 and step S207 The above processing is applied to all blocks in the order of processing (step S206 and step S207), and a final decoded image is obtained.
- FIG. 4 shows a configuration example of the decoding device 20 in the first embodiment.
- the decoding device 20 includes an entropy decoding unit 201, an inverse transform / inverse quantization unit 202, an intra prediction unit 203, a prediction memory 204, a reconstruction unit 205, and an encoding scheme determination unit 206. And an auxiliary information entropy decoding unit 207.
- the encoded data of the converted image is decoded by HEVC.
- decoding by HEVC first, encoded data of a converted image is entropy-decoded by an entropy decoding unit 201, and inverse transformation / inverse quantization processing is performed by an inverse transformation / inverse quantization unit 202. Accordingly, the prediction residual image is decoded, and the prediction result by the intra prediction unit 203 is added, so that a block of the decoded image of the converted image is obtained.
- the decoded converted image is accumulated in the prediction memory 204 and used as an input to the intra prediction unit 203 and the reconstruction unit 205.
- the encoding method determination unit 206 receives the decoded image block of the converted image, performs an encoding method determination process, and outputs a determination result.
- the auxiliary information entropy decoding unit 207 performs entropy decoding on the encoded data of the input auxiliary information to obtain auxiliary information.
- the auxiliary information entropy decoding unit 207 outputs auxiliary information to the reconstruction unit 205.
- the reconstruction unit 205 performs reconstruction processing with the auxiliary information, the reference pixel that can be referred to by the reconstruction target block, and the reconstruction target block as inputs, and outputs a final output picture.
- the encoding method and the decoding method according to the above-described embodiment unlike the related art, whether the input image is to be reconstructed in units of processing blocks or not to be reconstructed. Classify and apply reconstruction process.
- the encoding method and the decoding method according to the above embodiment can reduce the amount of code when transmitting boundary information by restricting the processing in units of blocks.
- the boundary information is transmitted by sharing a rule of replacing the inside of the reconstruction target block with an average value between the encoding device 10 and the decoding device 20. Therefore, it is possible to specify the position of the reconstruction target block.
- a reconstruction target block is specified for each block, and the specified reconstruction target block is encoded by HEVC with a smaller code amount on the encoding side. Processing that can be performed (for example, processing that replaces the entire block with an average value) is performed, and processing for determining the presence or absence of the processing is performed on the decoding side.
- the reconstruction method can be transmitted to the decoding side at the same time.
- FIG. 5 shows the configuration of the encoding device 30 according to the second embodiment.
- the encoding device 30 includes a preprocessing device 31 and a conventional encoding device 32.
- the preprocessing device 31 includes a block division unit 301, an encoding scheme determination unit 302, an auxiliary information extraction unit 303, an auxiliary information entropy encoding unit 304, an image conversion unit 305, and a post-conversion image memory 306. Consists of including.
- the conventional coding apparatus 32 includes an intra prediction unit 307, a transform / quantization unit 308, an entropy coding unit 309, an inverse quantization / inverse transform unit 310, and a prediction memory 311.
- the conventional coding apparatus 32 includes an intra prediction unit 307, a transform / quantization unit 308, an entropy coding unit 309, an inverse quantization / inverse transform unit 310, and a prediction memory 311.
- the difference between the encoding device 30 in the second embodiment and the encoding device 10 in the first embodiment is that a block division unit, an encoding method determination unit, an image conversion unit, and auxiliary information extraction
- the apparatus provided with a part and an entropy encoding part is a point provided as the pre-processing apparatus 31 independently from other structural parts (namely, the structural part with which a conventional encoding apparatus is provided).
- the converted image is stored in the converted image memory 306, and the auxiliary information extraction unit 303 refers to the converted image stored in the converted image memory 306. May be.
- Components other than the components included in the preprocessing device 31 are configured independently as the conventional encoding device 32.
- the conventional encoding device 32 for example, an HEVC intra encoding device, an encoding device conforming to an image encoding standard such as JPEG (JointoPhotographic Experts Group), or the like can be used.
- processing flow of the encoding device 30 is the same as the processing flow shown in FIG.
- the decoding device 40 includes a conventional decoding device 41 and a post-processing device 42.
- the conventional decoding device 41 includes an entropy decoding unit 401, an inverse transform / inverse quantization unit 402, an intra prediction unit 403, and a prediction memory 404.
- the post-processing device 42 includes a reconstruction unit 405, an encoding scheme determination unit 406, and an auxiliary information entropy decoding unit 407.
- the difference between the decoding apparatus 40 in the second embodiment and the decoding apparatus 20 in the first embodiment is that an apparatus including an encoding scheme determination unit, an auxiliary information entropy decoding unit, and a reconstruction unit.
- the post-processing device 42 is provided independently from other components (that is, components included in the conventional decoding device).
- the output picture memory 408 may store the output picture
- the reconstruction unit 405 may refer to the output picture stored in the output picture memory 408.
- Components other than the components included in the post-processing device 42 are configured independently as the conventional decoding device 41.
- processing flow of the decoding device 40 is the same as the processing flow shown in FIG.
- the pre-processing device 31 and the post-processing device 42 that can be used in combination with the conventional encoding device and decoding device can be realized.
- the improvement of the encoding efficiency is additive in the standard and the pre-processing device 31 and the post-processing device 42, according to the encoding method and the decoding method according to the second embodiment, the standard When the efficiency of the encoding device based on is improved, the encoding efficiency of the entire encoding device 30 can be improved.
- each prediction method (Planar prediction, DC prediction, and directionality prediction) that can be selected in HEVC refers to a referenceable pixel and performs prediction based on a simple prediction rule.
- prediction efficiency is lowered in an image in which components are randomly distributed. In such an image, since the amount of information of the prediction residual signal is large, when encoding is performed with the quantization width of the prediction residual signal being constant, the amount of code is excessively generated.
- an interpolation network constituted by a convolutional neural network and an interpolation image constituted by a convolutional neural network and interpolated by the interpolation network are interpolated.
- the interpolation network can reconstruct the missing region of the image in a pseudo manner.
- a region to be reconstructed is selected on the decoding side by image interpolation from the input image, a loss image is generated by loss, and output together with the loss region information indicating the loss region (step S301).
- the missing area information is a binary image or the like showing the missing area.
- the defect area information encoding process since the defect area information is transmitted to the decoding side, a process for encoding the defect area information is performed by using a conventional image encoding method such as JPEG (Joint Photographic Experts Group) or HEVC, or a run length. This is performed by an entropy encoding method such as encoding.
- the missing area information encoding process obtains encoded data of the missing area information (step S302).
- the missing image is encoded using a conventional image encoding method such as JPEG or HEVC. Thereby, the image encoding process obtains encoded data of the missing image (step S303).
- a decoded missing image is obtained from the encoded data of the missing image (step S304).
- the missing area information decoding process obtains missing area information from the encoded data of the missing area information (step S305).
- the decoded missing image and the missing area information are input to the interpolation network of the conventional technique 1 to obtain a final output image.
- the processing unit of the encoding process and the decoding process may be the entire screen, or may be a block unit obtained by dividing the screen using a structure such as HEVC CTU (step S306).
- FIG. 8 shows a configuration example of the encoding device 50 and the decoding device 60 that realize the above encoding processing and decoding processing.
- the encoding device 50 includes an image loss processing unit 501, an image encoding unit 502, and a missing region information encoding unit 503.
- the image loss processing unit 501 receives the input image and performs image loss processing. As a result, the image defect processing unit 501 outputs a defect image and defect area information.
- the image encoding unit 502 receives the missing image and performs image encoding processing. As a result, the image encoding unit 502 outputs encoded data of the missing image.
- the missing area information encoding unit 503 receives the missing area information as input and performs a missing area information encoding process. Thereby, the missing area information encoding unit 503 outputs encoded data of the missing area information.
- the encoded data of the missing image and the encoded data of the missing area information are transmitted to the decoding device 60.
- the decoding device 60 includes an image decoding unit 601, a missing area information decoding unit 602, and an image interpolation unit 603.
- the image decoding unit 601 receives the encoded data of the missing image and performs an image decoding process. Thereby, the image decoding unit 601 obtains a decoded missing image.
- the missing area information decoding unit 602 receives the encoded data of the missing area information as input and performs a missing area information decoding process. Thereby, defect area information is obtained.
- the image interpolation unit 603 includes an image interpolation network 604, and receives the decoded missing image and missing area information as input, and performs image interpolation processing. Thereby, the image interpolation unit 603 obtains a final output image.
- the subjective image quality of the output image greatly depends on the area of the missing area of the missing image in the image interpolation process. Specifically, the larger the area of the missing area to be interpolated, the smaller the amount of information input to the interpolation network, making it difficult to estimate the missing area in the image interpolation process and degrading the subjective image quality of the output image. Further, in the above configuration, if the missing region to be interpolated includes a complex element that cannot be inferred from the referenceable region, it is not reconstructed on the decoding side, or the subjective image quality of the output deteriorates.
- the third embodiment of the present invention will be described using learning by a hostile generation network using a convolutional neural network and an identification network as an example.
- the present invention describes image interpolation and hostile generation network by a convolutional neural network. It is not limited to learning by the framework of That is, any learning model in which the image interpolation method is acquired by learning can be applied to image interpolation. In addition, a learning method using an arbitrary error function can be applied to the learning method.
- the encoding device performs feature extraction with reference to the original image, and transmits image interpolation auxiliary information for assisting image interpolation to the decoding device.
- the decoding device performs image interpolation using the image interpolation auxiliary information.
- the networks used for extraction of image interpolation auxiliary information and image interpolation are individually optimized for each network, and then the networks are combined to be optimized as a whole.
- FIG. 9 shows the flow of encoding processing and decoding processing according to the third embodiment.
- an area to be reconstructed is selected on the decoding side by image interpolation from the input image.
- a defective image is generated by deleting the area by a process such as replacing the area with an average value.
- the generated defect image is output together with the defect area information indicating the position of the defect area, which is a set of pixel values of the defect area.
- the defect area information for example, a binary mask image (hereinafter, a defect area mask image) indicating a defect area can be used.
- a region selection method in image loss processing a method of selecting a region with a large amount of generated codes when using a fixed quantization width in HEVC intra coding, or region division for each object used in image recognition Can be used to select a region that can be interpolated (step S401).
- image interpolation auxiliary information is extracted from an area corresponding to a missing area derived from the missing area information in the input image or the input image itself using a network for extracting image interpolation auxiliary information. (Step S402). Details of the network for extracting image interpolation auxiliary information will be described later.
- the auxiliary information encoding process encodes the image interpolation auxiliary information extracted by the auxiliary information extraction process by a conventional entropy encoding method such as Huffman encoding.
- the auxiliary information encoding process obtains encoded data of the image interpolation auxiliary information (step S403).
- the process for encoding the reconstruction target area is performed using a conventional image encoding method such as JPEG or HEVC, or entropy such as run-length encoding. This is done according to the encoding method. Thereby, the missing area information encoding process obtains encoded data of the missing area information (step S404).
- a defective image is encoded using a conventional image encoding method such as JPEG or HEVC.
- the image encoding process obtains encoded data of the missing image (step S405).
- a decoded missing image is obtained from the encoded data of the missing image (step S406).
- the missing area information decoding process obtains missing area information from the encoded data of the missing area information (step S407).
- the auxiliary information decoding process obtains image interpolation auxiliary information from the encoded data of the image interpolation auxiliary information (step S407).
- the decoded missing image, the missing region information, and the image interpolation auxiliary information are input to a network for image interpolation, and a final output image is obtained. Details of the network for image interpolation will be described later (step S408).
- the processing unit of the encoding process and the decoding process may be the entire screen, or may be a block unit obtained by dividing the screen using a structure such as HEVC CTU.
- FIG. 10 shows a configuration example of an encoding device and a decoding device that realize the above encoding processing and decoding processing.
- the encoding device 70 includes an image loss processing unit 701, an image encoding unit 702, a missing region information encoding unit 703, an auxiliary information extracting unit 704, and an auxiliary information encoding unit 705. Composed.
- the image loss processing unit 701 receives an input image and performs image loss processing. Accordingly, the image defect processing unit 701 outputs a defect image and defect area information.
- the image encoding unit 702 receives the missing image and performs image encoding processing. As a result, the image encoding unit 702 outputs encoded data of the missing image.
- the missing area information encoding unit 703 receives the missing area information as input and performs a missing area information encoding process. Thereby, the missing area information encoding unit 703 outputs encoded data of the missing area information.
- the auxiliary information extraction unit 704 performs an auxiliary information extraction process by using, as input, an area corresponding to the missing area derived from the missing area information in the input image or an entire image including an area that is not a missing area. As a result, the auxiliary information extraction unit 704 extracts image interpolation auxiliary information.
- the auxiliary information encoding unit 705 receives the image interpolation auxiliary information and performs auxiliary information encoding processing. Thereby, the auxiliary information encoding unit 705 outputs encoded data of the image interpolation auxiliary information.
- the encoded data of the missing image, the encoded data of the missing area information, and the encoded data of the image interpolation auxiliary information are transmitted to the decoding device 80.
- the decoding device 80 includes an image decoding unit 801, a missing region information decoding unit 802, an image interpolation unit 803, and an auxiliary information decoding unit 805.
- the image decoding unit 801 receives the encoded data of the missing image and performs an image decoding process. Thereby, the image decoding unit 801 obtains a decoded missing image.
- the missing area information decoding unit 802 receives the encoded data of the missing area information as input and performs a missing area information decoding process. Thereby, the missing area information decoding unit 802 obtains missing area information.
- the auxiliary information decoding unit 805 receives the encoded data of the image interpolation auxiliary information and performs auxiliary information decoding processing. Thereby, the auxiliary information decoding unit 805 obtains image interpolation auxiliary information.
- the image interpolation unit 803 receives the decoded missing image, the missing region information, and the image interpolation auxiliary information, and performs an image interpolation process with reference to the image interpolation auxiliary information. Thereby, the image interpolation unit 803 obtains a final output image.
- FIG. 11 shows a network configuration of the auxiliary information extraction unit 704 and the image interpolation unit 803.
- the auxiliary information extraction unit 704 includes an auxiliary information extraction network 7041 for extracting image interpolation auxiliary information to be transmitted to the decoding side.
- the auxiliary information extraction network 7041 is a network that receives the input image and the missing area information and outputs image interpolation auxiliary information.
- the auxiliary information extraction network 7041 configures an intermediate layer by a convolutional layer, a fully connected layer, or the like, for example, with an input as an input image and a defective area mask image as two images and an output as an arbitrary number of units.
- the image interpolation unit 803 refers to the auxiliary information reference network 8031 for predicting the missing area with reference to the image interpolation auxiliary information, and the missing image reference for predicting the missing area with reference to the missing image.
- the auxiliary information reference network 8031 is a network that receives the image interpolation auxiliary information and outputs an intermediate image by referring to the auxiliary information.
- the auxiliary information reference network 8031 has, for example, the same number of units as the image interpolation auxiliary information and the output as an intermediate image by referring to one auxiliary information, and the intermediate layer is formed by a fully connected layer, a deconvolution layer, a convolution layer, and the like. Constitute.
- the missing image reference network 8032 is a network that outputs the intermediate image by referring to the missing image with the missing image and the missing area mask image of the input image as inputs.
- the missing image reference network 8032 has, for example, a convolutional layer, a fully connected layer, and a deconvolution using, as input, two images, a missing image of the input image and a missing region mask image, and an output as an intermediate image by referring to one missing image.
- An intermediate layer is constituted by layers and the like.
- the reconstruction network 8033 is a network that receives an intermediate image based on auxiliary information reference and an intermediate image based on missing image reference and outputs a final output image in which a missing area is interpolated.
- the reconstruction network 8033 includes, for example, two intermediate images as input and one output image as output, and forms an intermediate layer including a convolution layer, a fully connected layer, a deconvolution layer, and the like.
- the auxiliary information extraction unit 704 and the image interpolation unit 803 are learned.
- the framework of the hostile generation network can be used as in the prior art 1.
- the identification network 9000 for evaluating the naturalness of the interpolated region receives the output image of the image interpolating unit 803 as an input, and calculates the probability that the output image is a true image that has not been interpolated. Output.
- the mean square error of the pixels of the original image and the output image of the network (hereinafter referred to as mean square error) and the framework of the hostile generation network are applied, and the output image of the network is identified by the identification network.
- error hereinafter referred to as “identification network error” or an error due to a weighted sum of the mean square error and the identification network error (hereinafter referred to as weighted error) can be used.
- the missing image reference network 8032 and the identification network 9000 shown in FIG. 11 are cut out and combined as shown in FIG. 13, and the output of the missing image reference network 8032 is regarded as an input to the identification network 9000.
- the image reference network 8032 is learned (step S501).
- the missing image and missing area information of the original image are input to the missing image reference network 8032, and the output image approaches the original image by the error back propagation method. Update the parameters.
- learning is performed by first applying a mean square error as an error function, and then performing learning by applying a weighted error. In the subsequent learning processing of each network, learning is similarly performed using the mean square error, and then learning is performed using the weighted error.
- the auxiliary information extraction network 7041, the auxiliary information reference network 8031, and the identification network 9000 shown in FIG. 11 are cut out and combined as shown in FIG. 14 to identify the output of the auxiliary information reference network 8031. It is regarded as an input to the network 9000, and the auxiliary information extraction network 7041 and the auxiliary information reference network 8031 are learned (step S502).
- the original image and the missing area information are input to a network in which the auxiliary information extraction network 7041 and the auxiliary information reference network 8031 are combined.
- the mean square error and the weighted error are sequentially applied so that the output image approaches the original image, and the network parameters are updated by the error back propagation method.
- the reconstruction network learning process includes a missing image reference network 8032, an auxiliary information extraction network 7041, an auxiliary information reference network 8031, a reconstruction network 8033, and a defect image reference network learning process and an auxiliary information extraction / reference network learning process.
- the identification networks 9000 are combined as shown in FIG. 11, and only the reconfiguration network 8033 is learned (step S503).
- the reconstruction network learning process inputs the original image, the missing image of the original image, and the missing area information to the combined network, and the mean square error and the weight so that the output image approaches the original image.
- the attached error is applied in order, and only the parameters of the reconstruction network are updated by the error back propagation method.
- the whole learning process simultaneously learns the missing image reference network 8032, the auxiliary information extraction network 7041, the auxiliary information reference network 8031, and the reconstruction network 8033 that are combined as shown in FIG. 11 in the reconstruction network learning process (step S504). ).
- the whole learning process is performed by inputting the original image, the missing image of the original image, and the missing area information into the combined network, and the mean square error and the weighted error so that the output image approaches the original image.
- the parameters of all networks are updated by the error back propagation method.
- only the auxiliary information extraction network may be configured to learn with fixed network parameters.
- the order of application of the above error functions is an example, and learning may be performed without using the framework of the hostile generation network including the identification network 9000, and the identification network error, the mean square error, or the weighted error is learned. You may apply, changing at any time according to the number of repetitions.
- the identification network 9000 is learned according to the number of iterations and the accuracy rate of the identification network 9000 independently of the learning process of each network in FIG. May be.
- the network output image and the original image used in each learning process of FIG. 12 are alternately input to the identification network 9000, and the probability that the input is the original image is output.
- the error from the correct value of 1 may be evaluated by an error function such as a mutual information amount, and the parameters may be updated by the error back propagation method.
- each learning process may be determined by using a threshold process for the number of iterations or a reduction in error.
- the unit of processing may be the entire screen or may be a block unit obtained by dividing the screen using a structure such as HEVC CTU.
- the encoding method and the decoding method in the third embodiment are different from the method of obtaining the output image by generating the image by applying the interpolation network in the prior art to the decoding side, and using the image interpolation auxiliary information. Generate an image.
- the encoding method and the decoding method in the third embodiment can improve the prediction accuracy over the method using the conventional technique, and can realize the generation using the feature of the original picture.
- the encoding method and the decoding method in the third embodiment can determine the image interpolation auxiliary information to be transmitted by learning, the image interpolation auxiliary information determined by manual trial and error such as conventional HEVC. Compared to the extraction, it is possible to extract image interpolation auxiliary information that can obtain a more accurate reconstruction result. Furthermore, the encoding method and the decoding method according to the third embodiment acquire an intended operation for each network having a complicated configuration to be learned by controlling the network learning order and the error function to be applied. Can be made.
- the encoding method and decoding method in the third embodiment solve this problem by providing an auxiliary information extraction unit 704 on the encoding side and providing image interpolation auxiliary information to the interpolation network.
- the auxiliary information extraction network 7041 that defines the image interpolation auxiliary information is also acquired by learning, so that the encoding method and the decoding method in the third embodiment can be performed manually like image encoding such as HEVC.
- image interpolation auxiliary information with higher image generation accuracy can be extracted.
- the configuration of the encoding method and the decoding method in the third embodiment includes the auxiliary information extraction unit 704 that generates image interpolation auxiliary information and acquires network parameters by learning, the auxiliary information extraction unit 704 and the image
- the interpolating unit 803 learns simultaneously, it is difficult for each network to learn the intended operation. In particular, when using the framework of the hostile generation network, this tendency becomes remarkable because it is difficult to adjust learning.
- the auxiliary information extraction unit 704 and the image interpolation unit 803 are divided into networks for each role, and the network to be learned and the error to be applied according to the number of learning iterations. By controlling the function, each network can acquire an intended operation.
- the fourth embodiment differs from the third embodiment in the configuration of the network of the auxiliary information extraction unit and the image interpolation unit, and generates image interpolation auxiliary information from the output of the missing image reference network and the difference between the input images. .
- FIG. 15 shows a network configuration in the fourth embodiment.
- the auxiliary information extraction unit 704 includes an auxiliary information extraction network 7041 and a missing image reference network 8032 using network parameters common to the image interpolation unit 803.
- the auxiliary information extraction network 7041 is a network that outputs the image interpolation auxiliary information by using the difference between the input image and the intermediate image based on the missing image and the missing area information as inputs.
- the auxiliary information extraction network 7041 has, for example, a difference image between an input image and an intermediate image by referring to a missing image, and two images of a missing region mask image as an input, an output as an arbitrary number of units, a convolution layer, and a fully connected
- An intermediate layer is constituted by layers and the like.
- the image interpolation unit 803 includes an auxiliary information reference network 8031, a missing image reference network 8032, and a reconstruction network 8033.
- the input / output of each network is the same as that of the third embodiment except for the missing image reference network 8032.
- the auxiliary information reference network 8031 is a network that receives the image interpolation auxiliary information and outputs an intermediate image by referring to the auxiliary information.
- the missing image reference network 8032 is a network that outputs the intermediate image based on the missing image by using the missing image of the input image and the missing area mask image as inputs.
- the intermediate image based on the missing image reference is input to the reconstruction network 8033 as a component of the image interpolation unit 803. Further, the difference between the intermediate image and the input image based on the missing image reference is input to the auxiliary information extraction network 7041 as a component of the auxiliary information extraction unit 704.
- the reconstruction network 8033 is a network that receives an intermediate image based on auxiliary information reference and an intermediate image based on missing image reference, and outputs a final output image in which a missing area is interpolated.
- auxiliary information extraction unit 704 and the image interpolation unit 803 is performed.
- the learning process is the same as in the third embodiment, but the network configuration in the auxiliary information extraction / reference network learning process is as shown in FIG. In this process, only the auxiliary information extraction network 7041 and the auxiliary information reference network 8031 are learned in the configuration of FIG.
- the auxiliary information extraction unit 704 can directly input the original image as in the third embodiment, but as described above, the auxiliary information extraction unit 704 can perform the decoding on the decoding side and the encoding side.
- the prediction result from the peripheral block intermediate image by referring to the missing image
- a difference image between the original image and the predicted image from the peripheral block can be input.
- a part or all of the encoding device and the decoding device in the above-described embodiment may be realized by a computer.
- a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed.
- the “computer system” includes an OS and hardware such as peripheral devices.
- the “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, and a hard disk incorporated in a computer system.
- the “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line.
- a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time.
- the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. It may be realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).
- FPGA Field Programmable Gate Array
- DESCRIPTION OF SYMBOLS 10,30 ... Coding apparatus, 101, 301 ... Block division part, 102, 302 ... Coding system determination part, 103, 303 ... Auxiliary information extraction part, 104.304 ... Auxiliary information entropy coding part, 105, 305 ... Image conversion unit, 306 ... post-conversion image memory, 107, 307 ... intra prediction unit, 108, 308 ... transformation / quantization unit, 109, 309 ... entropy coding unit, 110, 310 ... inverse quantization / inverse transformation unit, 111, 311 ... Prediction memory, 20 ... Decoding device, 201, 401 ...
- missing region information decoding unit 603, 803 ... image interpolation unit, 8031 ... auxiliary information reference network , 8032 ... Missing image reference network, 8033 ... Reconstruction network, 604 ... Image interpolation network, 805 ... Auxiliary information decoding unit, 9000 ... Identification network
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
This encoding device for encoding images is provided with: a determination unit which determines whether an input image is to be reconfigured; an auxiliary information extraction unit which extracts, from the image determined to be reconfigured, auxiliary information to be used in reconfiguration; a conversion unit which converts the image determined to be reconfigured to obtain a converted image; and an encoding unit which encodes the converted image to obtain encoded data, wherein the conversion unit performs conversion so that, when the encoding unit performs the encoding, a smaller encoded amount is obtained than when the input image is encoded. According to the present invention, transmission and reception can be performed with a smaller encoded amount for a region to be reconfigured.
Description
本発明は、符号化装置、復号装置、符号化方法、復号方法、符号化プログラム及び復号プログラムに関する。
The present invention relates to an encoding device, a decoding device, an encoding method, a decoding method, an encoding program, and a decoding program.
映像データを圧縮符号化するための標準規格として、MPEG-4やH.264/AVC、H.265/HEVC(以下、「HEVC」という。)が知られている。また、HEVCに次ぐ新たな規格の標準化も検討が進められている。これらの映像圧縮符号化規格では、画像を矩形のブロックに分割した単位で処理を行い、予測対象ブロックに隣接する予測ブロックを参照して予測対象ブロックの画素値を予測し、予測残差信号のみを送信する予測符号化方式が採用されている。以下、HEVCを例に、フレーム内に閉じて画素信号の予測を行うイントラ予測符号化の方法を述べる。
MPEG-4 and H.264 are standards for compressing and encoding video data. H.264 / AVC, H.H. H.265 / HEVC (hereinafter referred to as “HEVC”) is known. In addition, standardization of a new standard following HEVC is being studied. In these video compression coding standards, processing is performed in units obtained by dividing an image into rectangular blocks, the prediction block adjacent to the prediction target block is referred to, the pixel value of the prediction target block is predicted, and the prediction residual signal only Is used. Hereinafter, taking HEVC as an example, an intra-prediction coding method for predicting a pixel signal by closing in a frame will be described.
HEVCでは、図17に示すように画面全体を64画素×64画素(以下、「64×64」という。)のブロックに区切り、各ユニットをCTU(Coding Tree Unit)として定義する。CTUは、CU(Coding Unit)と呼ばれる4つの正方形に分割することができ、これを再帰的に処理することで、細かなブロックに分割を行う。HEVCでは、CUのサイズは64×64、32×32、16×16及び8×8の4種類を用いることができ、このCUをさらに分割したPU(Prediction Unit)と呼ばれる単位で予測処理を行う。
イントラ予測の場合、CUを4つの正方形に分割するか否かの2通りのPUを用いることができる。各PUは35種類の予測パラメータを選択的に適用可能であり、例えば原画像との予測残差信号が最小となる予測パラメータを符号化側で選択し、予測パラメータ及び予測残差信号を復号側に送信する。 In HEVC, as shown in FIG. 17, the entire screen is divided into blocks of 64 pixels × 64 pixels (hereinafter referred to as “64 × 64”), and each unit is defined as a CTU (Coding Tree Unit). The CTU can be divided into four squares called CUs (Coding Units). By recursively processing the CTU, the CTU is divided into fine blocks. In HEVC, four types of 64 × 64, 32 × 32, 16 × 16, and 8 × 8 CU sizes can be used, and prediction processing is performed in units called PUs (Prediction Units) obtained by further dividing the CU. .
In the case of intra prediction, it is possible to use two types of PUs for determining whether to divide a CU into four squares. Each PU can selectively apply 35 types of prediction parameters. For example, a prediction parameter that minimizes a prediction residual signal with the original image is selected on the encoding side, and the prediction parameter and the prediction residual signal are decoded on the decoding side. Send to.
イントラ予測の場合、CUを4つの正方形に分割するか否かの2通りのPUを用いることができる。各PUは35種類の予測パラメータを選択的に適用可能であり、例えば原画像との予測残差信号が最小となる予測パラメータを符号化側で選択し、予測パラメータ及び予測残差信号を復号側に送信する。 In HEVC, as shown in FIG. 17, the entire screen is divided into blocks of 64 pixels × 64 pixels (hereinafter referred to as “64 × 64”), and each unit is defined as a CTU (Coding Tree Unit). The CTU can be divided into four squares called CUs (Coding Units). By recursively processing the CTU, the CTU is divided into fine blocks. In HEVC, four types of 64 × 64, 32 × 32, 16 × 16, and 8 × 8 CU sizes can be used, and prediction processing is performed in units called PUs (Prediction Units) obtained by further dividing the CU. .
In the case of intra prediction, it is possible to use two types of PUs for determining whether to divide a CU into four squares. Each PU can selectively apply 35 types of prediction parameters. For example, a prediction parameter that minimizes a prediction residual signal with the original image is selected on the encoding side, and the prediction parameter and the prediction residual signal are decoded on the decoding side. Send to.
HEVCでは、予測方式はPlanar予測、DC(Direct Current;直流成分)予測及び方向性予測の3種類から選択可能であり、方向性予測には33の予測パラメータが割り当てられていることから、予測パラメータの総数は35である。各予測方式は、図18に示すように、予測対象ブロックの左および上に位置する参照画素の画素値を用いて予測を行い、方向性予測では、定義された33方向から1つの方向を参照方向として選択し、参照方向の画素値を参照ブロックに割り当てることで、予測対象ブロックの予測画素を生成する。Planar予測では、予測対象ブロックの左下、右上、予測対象画素の左、上の4画素を参照し、それらの重み付き平均として予測対象ブロック内の各画素を予測する。DC予測では、予測対象ブロックの左および上に位置する参照画素の平均として、予測対象ブロックの予測値を単一に生成する。
In HEVC, the prediction method can be selected from three types of Planar prediction, DC (Direct Current) prediction, and directionality prediction, and 33 prediction parameters are assigned to the directionality prediction. The total number is 35. As shown in FIG. 18, each prediction method performs prediction using pixel values of reference pixels located on the left and top of the prediction target block, and refers to one direction from the defined 33 directions in the directional prediction. By selecting a direction and assigning a pixel value in the reference direction to the reference block, a prediction pixel of the prediction target block is generated. In Planar prediction, the lower left and upper right of the prediction target block, the left and upper four pixels of the prediction target pixel are referred to, and each pixel in the prediction target block is predicted as a weighted average thereof. In DC prediction, a prediction value of a prediction target block is generated as a single average as the average of reference pixels located on the left and above the prediction target block.
画質を保持したまま符号量を削減する方法として、画素の完全再現を目的として上記に基づく予測符号化の予測方式の高精度化により予測残差の情報量を削減する方法がある。
これ以外の方法として、擬似的な画像を復号側で再構成する処理方式を導入し、従来の符号化方式と併用することで、上記の予測方式やその高精度化で効率的に符号化できない画像に対しても、画素の完全再現を目的とせず復号画像の主観品質を保持しながら符号量を削減する方法が提案されている(特許文献1参照)。特許文献1に記載の技術によれば、符号化側では入力画像をCartoon-Texture信号分解により分解し、Cartoon成分ならびに非合成Texture成分の和で表現される非合成成分画像と、合成Textureの代表Textureと、合成Textureに対応する領域情報とを送信する。領域情報は画像で表現され、合成領域と当該領域に対応する合成方法が含まれる。復号側では、非合成成分画像を復号後、合成Textureの代表Textureと領域情報とを用いて、Texture合成により再構成された画像との加算により復号画像を得る。ここで、非合成成分画像、及び、合成Textureに対応する領域情報の符号化及び復号の方法には、既存の符号化標準が用いられる。特許文献1に記載の技術は、特にTexture成分の多い画像に対してより少ない符号量で符号化できる。 As a method of reducing the code amount while maintaining the image quality, there is a method of reducing the information amount of the prediction residual by improving the prediction method of the predictive coding based on the above for the purpose of complete pixel reproduction.
As another method, a processing method for reconstructing a pseudo image on the decoding side is introduced, and when used together with the conventional encoding method, the above prediction method and its high accuracy cannot be efficiently encoded. Also for an image, a method has been proposed in which the amount of code is reduced while maintaining the subjective quality of a decoded image without aiming at complete pixel reproduction (see Patent Document 1). According to the technique described in Patent Document 1, on the encoding side, an input image is decomposed by Cartoon-Texture signal decomposition, and a non-synthetic component image represented by the sum of a Cartoon component and a non-synthetic Texture component, and a representative of the synthetic Texture The texture and the area information corresponding to the composite texture are transmitted. The area information is expressed by an image, and includes a synthesis area and a synthesis method corresponding to the area. On the decoding side, after decoding the non-synthesized component image, the decoded image is obtained by adding the image reconstructed by the texture synthesis using the representative texture of the synthesized texture and the region information. Here, an existing coding standard is used as a method of coding and decoding region information corresponding to a non-synthesized component image and a synthesized texture. The technique described in Patent Document 1 can encode an image with a large number of texture components with a smaller code amount.
これ以外の方法として、擬似的な画像を復号側で再構成する処理方式を導入し、従来の符号化方式と併用することで、上記の予測方式やその高精度化で効率的に符号化できない画像に対しても、画素の完全再現を目的とせず復号画像の主観品質を保持しながら符号量を削減する方法が提案されている(特許文献1参照)。特許文献1に記載の技術によれば、符号化側では入力画像をCartoon-Texture信号分解により分解し、Cartoon成分ならびに非合成Texture成分の和で表現される非合成成分画像と、合成Textureの代表Textureと、合成Textureに対応する領域情報とを送信する。領域情報は画像で表現され、合成領域と当該領域に対応する合成方法が含まれる。復号側では、非合成成分画像を復号後、合成Textureの代表Textureと領域情報とを用いて、Texture合成により再構成された画像との加算により復号画像を得る。ここで、非合成成分画像、及び、合成Textureに対応する領域情報の符号化及び復号の方法には、既存の符号化標準が用いられる。特許文献1に記載の技術は、特にTexture成分の多い画像に対してより少ない符号量で符号化できる。 As a method of reducing the code amount while maintaining the image quality, there is a method of reducing the information amount of the prediction residual by improving the prediction method of the predictive coding based on the above for the purpose of complete pixel reproduction.
As another method, a processing method for reconstructing a pseudo image on the decoding side is introduced, and when used together with the conventional encoding method, the above prediction method and its high accuracy cannot be efficiently encoded. Also for an image, a method has been proposed in which the amount of code is reduced while maintaining the subjective quality of a decoded image without aiming at complete pixel reproduction (see Patent Document 1). According to the technique described in Patent Document 1, on the encoding side, an input image is decomposed by Cartoon-Texture signal decomposition, and a non-synthetic component image represented by the sum of a Cartoon component and a non-synthetic Texture component, and a representative of the synthetic Texture The texture and the area information corresponding to the composite texture are transmitted. The area information is expressed by an image, and includes a synthesis area and a synthesis method corresponding to the area. On the decoding side, after decoding the non-synthesized component image, the decoded image is obtained by adding the image reconstructed by the texture synthesis using the representative texture of the synthesized texture and the region information. Here, an existing coding standard is used as a method of coding and decoding region information corresponding to a non-synthesized component image and a synthesized texture. The technique described in Patent Document 1 can encode an image with a large number of texture components with a smaller code amount.
特許文献1に記載の技術では、合成Textureに対応する領域情報を送信する必要があり、この領域情報を画像として復号側に送信する。このため、1フレームの送信のために非合成成分画像と領域情報画像の2フレームを送信する必要がある、これにより、補助情報の符号量が多くなるという課題がある。このように、復号側でTexture合成等の再構成を伴う符号化方式では、再構成対象となる領域をより少ない符号量で送受信可能な再構成処理を実現することが望まれる。
In the technique described in Patent Document 1, it is necessary to transmit region information corresponding to a composite texture, and this region information is transmitted to the decoding side as an image. For this reason, it is necessary to transmit two frames of a non-synthetic component image and a region information image for transmission of one frame, which causes a problem that the amount of code of auxiliary information increases. As described above, in an encoding method involving reconstruction such as texture synthesis on the decoding side, it is desired to realize a reconstruction process that enables transmission and reception of a region to be reconstructed with a smaller code amount.
上記事情に鑑み、本発明は、再構成対象となる領域をより少ない符号量で送受信することができる技術の提供を目的としている。
In view of the above circumstances, an object of the present invention is to provide a technique capable of transmitting and receiving a region to be reconfigured with a smaller code amount.
本発明の一態様は、画像を符号化する符号化装置であって、入力された画像を再構成対象とするか否かを判定する判定部と、前記再構成対象とすると判定された画像から、再構成に使うための情報である補助情報を抽出する補助情報抽出部と、前記再構成対象とすると判定された画像を変換し変換画像を得る変換部と、前記変換画像を符号化し符号化データを得る符号化部と、を備え、前記変換部は、前記符号化部が符号化する際、前記入力された画像を符号化した場合よりも少ない符号量になるよう変換する符号化装置である。
One aspect of the present invention is an encoding device that encodes an image, a determination unit that determines whether or not an input image is a reconstruction target, and an image that is determined to be the reconstruction target An auxiliary information extraction unit that extracts auxiliary information that is information used for reconstruction, a conversion unit that converts the image determined to be the reconstruction target and obtains a converted image, and encodes and encodes the converted image An encoding unit that obtains data, and the conversion unit is an encoding device that converts the input image so as to have a smaller code amount than when the input image is encoded when the encoding unit encodes. is there.
また、本発明の一態様は、上記の符号化装置であって、前記判定部は、推定発生符号量及び推定歪量を取得してレート歪最適化を行うことによって前記入力された画像を前記再構成対象とするか否かを判定する。
One embodiment of the present invention is the above encoding device, in which the determination unit acquires the estimated generated code amount and the estimated distortion amount and performs the rate distortion optimization to obtain the input image. It is determined whether or not to be reconfigured.
また、本発明の一態様は、上記の符号化装置であって、前記補助情報は、前記変換画像を、前記再構成対象とすると判定された画像の特徴を保ちつつ、前記再構成対象とすると判定された画像よりも少ない符号量の画像に逆変換するための情報である。
Further, one aspect of the present invention is the encoding device described above, in which the auxiliary information is the reconstruction target while maintaining the characteristics of the image that is determined to be the reconstruction image. This is information for inverse conversion into an image having a smaller code amount than the determined image.
また、本発明の一態様は、画像が符号化された符号化データを復号する復号装置であって、入力された符号化データを復号し復号画像を得る復号部と、前記復号画像が再構成対象の画像であるか否かを判定する判定部と、再構成に使うための情報である補助情報を取得し、前記再構成対象の画像であると判定された復号画像を、前記補助情報を用いて再構成する再構成部と、を備える復号装置である。
Another embodiment of the present invention is a decoding device that decodes encoded data obtained by encoding an image, a decoding unit that decodes input encoded data to obtain a decoded image, and the decoded image is reconstructed A determination unit that determines whether or not the image is a target image; and auxiliary information that is information used for reconstruction is acquired, and the decoded image that is determined to be the image to be reconstructed is converted to the auxiliary information. And a reconfiguration unit that reconfigures the decoding device.
また、本発明の一態様は、画像を符号化する符号化装置による符号化方法であって、入力された画像を再構成対象とするか否かを判定する判定ステップと、前記再構成対象とすると判定された画像から、再構成に使うための情報である補助情報を抽出する補助情報抽出ステップと、前記再構成対象とすると判定された画像を、前記入力された画像を符号化した場合よりも少ない符号量になるよう変換し変換画像を得る変換ステップと、前記変換画像を符号化し符号化データを得る符号化ステップと、を有する符号化方法である。
One embodiment of the present invention is an encoding method by an encoding device that encodes an image, the determination step for determining whether or not the input image is a reconstruction target, and the reconstruction target Then, an auxiliary information extraction step for extracting auxiliary information, which is information for use in reconstruction, from the determined image, and an image determined to be the reconstruction target than when the input image is encoded The encoding method includes a conversion step of converting the code amount so as to obtain a converted image and an encoding step of encoding the converted image to obtain encoded data.
また、本発明の一態様は、画像が符号化された符号化データを復号する復号装置による復号方法であって、入力された符号化データを復号し復号画像を得る復号ステップと、前記復号画像が再構成対象の画像であるか否かを判定する判定ステップと、再構成に使うための情報である補助情報を取得し、前記再構成対象の画像であると判定された復号画像を、前記補助情報を用いて再構成する再構成ステップと、を有する復号方法である。
Another embodiment of the present invention is a decoding method by a decoding device that decodes encoded data in which an image is encoded, the decoding step of decoding input encoded data to obtain a decoded image, and the decoded image Determining whether or not is an image to be reconstructed, and acquiring auxiliary information that is information for use in reconstruction, and decoding the image that has been determined to be the image to be reconstructed, And a reconstruction step of reconstructing using auxiliary information.
また、本発明の一態様は、上記の符号化装置としてコンピュータを機能させるための符号化プログラムである。
Further, one embodiment of the present invention is an encoding program for causing a computer to function as the above encoding device.
また、本発明の一態様は、上記の復号装置としてコンピュータを機能させるための復号プログラムである。
Further, one embodiment of the present invention is a decoding program for causing a computer to function as the above-described decoding device.
本発明により、再構成対象となる領域をより少ない符号量で送受信することができる。
According to the present invention, a region to be reconfigured can be transmitted and received with a smaller code amount.
以下、HEVCのイントラ予測符号化との併用を例に、本発明の実施形態について説明するが、本発明はHEVCならびにイントラ予測に限定されるものではない。つまり、本発明は、HEVC以外の画像符号化方式ならびにインター予測に対しても適用できるものである。
Hereinafter, although an embodiment of the present invention will be described using a combination with HEVC intra prediction coding as an example, the present invention is not limited to HEVC and intra prediction. That is, the present invention can be applied to image coding methods other than HEVC and inter prediction.
本発明では、HEVCのCTUやCU等のブロック毎に符号化側で再構成対象とするか非再構成対象とするかを判定し、再構成対象と判定されたブロック(以下、再構成対象ブロック)から補助情報を抽出し送信する。ここで、再構成とはTexture合成や画像の補間合成処理等によって、画像の着目領域に適合する擬似的な画像を生成する処理を指す。なお、ここでいう擬似的な画像とは、例えば、入力画像と比較して、主観的な観点から差異を感じ辛い画像のことである。
また、再構成対象ブロックには、HEVCのイントラ予測で予測残差の情報量が小さくなるよう、ブロック全体に均一な画像処理を施して、HEVC符号化器に入力する。言い換えると、HEVCでは予測精度が低いブロックや、一定の精度の主観画質を確保できれば符号化前の画像の画素を正確に再現する必要が少ない被写体に係るブロックを、再構成対象ブロックとし、HEVCが予測しやすい画素で構成させるようにすることで所望の画質を保ったまま符号化に要する符号量の低減を図る。復号側では、ブロック全体に均一な画像処理が施されているか否かを判定することにより、再構成対象ブロックを判別する。 In the present invention, for each block such as CTU or CU of HEVC, it is determined whether to be a reconstruction target or a non-reconstruction target on the encoding side, and a block determined as a reconstruction target (hereinafter, a reconstruction target block) ) Extract auxiliary information from Here, reconstruction refers to a process of generating a pseudo image that matches a target region of an image by texture synthesis, image interpolation synthesis processing, or the like. Note that the pseudo image referred to here is, for example, an image in which it is difficult to feel a difference from a subjective viewpoint as compared with an input image.
In addition, the reconstruction target block is subjected to uniform image processing on the entire block so as to reduce the amount of information of the prediction residual in the HEVC intra prediction, and is input to the HEVC encoder. In other words, a block with low prediction accuracy in HEVC or a block related to a subject that does not need to accurately reproduce pixels of an image before encoding if a certain level of subjective image quality can be ensured is set as a reconstruction target block. By configuring the pixels with predictable pixels, the amount of code required for encoding is reduced while maintaining a desired image quality. On the decoding side, the reconstruction target block is determined by determining whether or not uniform image processing is performed on the entire block.
また、再構成対象ブロックには、HEVCのイントラ予測で予測残差の情報量が小さくなるよう、ブロック全体に均一な画像処理を施して、HEVC符号化器に入力する。言い換えると、HEVCでは予測精度が低いブロックや、一定の精度の主観画質を確保できれば符号化前の画像の画素を正確に再現する必要が少ない被写体に係るブロックを、再構成対象ブロックとし、HEVCが予測しやすい画素で構成させるようにすることで所望の画質を保ったまま符号化に要する符号量の低減を図る。復号側では、ブロック全体に均一な画像処理が施されているか否かを判定することにより、再構成対象ブロックを判別する。 In the present invention, for each block such as CTU or CU of HEVC, it is determined whether to be a reconstruction target or a non-reconstruction target on the encoding side, and a block determined as a reconstruction target (hereinafter, a reconstruction target block) ) Extract auxiliary information from Here, reconstruction refers to a process of generating a pseudo image that matches a target region of an image by texture synthesis, image interpolation synthesis processing, or the like. Note that the pseudo image referred to here is, for example, an image in which it is difficult to feel a difference from a subjective viewpoint as compared with an input image.
In addition, the reconstruction target block is subjected to uniform image processing on the entire block so as to reduce the amount of information of the prediction residual in the HEVC intra prediction, and is input to the HEVC encoder. In other words, a block with low prediction accuracy in HEVC or a block related to a subject that does not need to accurately reproduce pixels of an image before encoding if a certain level of subjective image quality can be ensured is set as a reconstruction target block. By configuring the pixels with predictable pixels, the amount of code required for encoding is reduced while maintaining a desired image quality. On the decoding side, the reconstruction target block is determined by determining whether or not uniform image processing is performed on the entire block.
<第1の実施形態>
以下、第1の実施形態について、図面を参照しながら説明する。 <First Embodiment>
The first embodiment will be described below with reference to the drawings.
以下、第1の実施形態について、図面を参照しながら説明する。 <First Embodiment>
The first embodiment will be described below with reference to the drawings.
[符号化装置の処理]
まず、本発明による符号化装置の処理について説明する。
図1に、本発明の第1の実施形態における符号化装置の処理フローを示す。 [Processing of encoding apparatus]
First, the processing of the encoding device according to the present invention will be described.
FIG. 1 shows a processing flow of the encoding apparatus according to the first embodiment of the present invention.
まず、本発明による符号化装置の処理について説明する。
図1に、本発明の第1の実施形態における符号化装置の処理フローを示す。 [Processing of encoding apparatus]
First, the processing of the encoding device according to the present invention will be described.
FIG. 1 shows a processing flow of the encoding apparatus according to the first embodiment of the present invention.
ブロック分割処理は、入力ピクチャから符号化処理ブロックの形状を決定する(ステップS101)。出力されるブロック分割の形状は、図17に示したようなCTU、CU及びPUに倣い、このブロックを復号側の再構成処理の単位かつHEVC符号化処理の単位とする。
In the block division processing, the shape of the encoding processing block is determined from the input picture (step S101). The output block division shape follows CTU, CU, and PU as shown in FIG. 17, and this block is used as a unit of reconstruction processing on the decoding side and a unit of HEVC encoding processing.
分割形状の決定方法としては、CTUのように均一の矩形として決定する方法のほか、HEVCテストモデル(HM;HEVC Test Model)に実装されるようなレート歪最適化により決定されるCU分割形状として決定する方法、又は、画像認識で用いられる物体毎に領域分割を実行した結果をブロック単位で近似したものとして決定する方法等を用いることができる。
As a method for determining the division shape, in addition to a method for determining a uniform rectangle such as CTU, a CU division shape determined by rate distortion optimization as implemented in a HEVC test model (HM; HEVC Test Model) A determination method, a method of determining a result obtained by performing region division for each object used in image recognition as an approximation in block units, or the like can be used.
符号化方式判定処理は、ブロック分割処理により分割されたブロック単位で、再構成対象ブロックとするか又は非再構成対象ブロックとするかを判定する(ステップS102)。なお、判定方法としては、例えば、再構成対象とする場合と非再構成対象とする場合とで、推定発生符号量ならびに推定歪量をそれぞれ導出し、レート歪最適化を適用することで判定する方法を用いることができる。
In the coding method determination process, it is determined whether the block is divided into blocks by the block division process, whether to be a reconstruction target block or a non-reconstruction target block (step S102). As a determination method, for example, the estimated generated code amount and the estimated distortion amount are respectively derived for the case of the reconfiguration target and the case of the non-reconfiguration target, and the determination is made by applying rate distortion optimization. The method can be used.
再構成対象ブロックとして判定された場合(ステップS103・Yes)、補助情報抽出処理により、再構成処理を補助するために復号装置へ送信する補助情報を、再構成対象ブロックから抽出する(ステップS104)。なお、再構成処理とは、再構成対象のブロックに対し、後述するような何らかの変換を施したブロックを、復号側で逆変換する処理のことである。なお、補助情報抽出処理は、例えば画像合成により再構成対象ブロックを合成することによって再構成する場合には、合成時に使用する代表Texture又は物体を識別したラベル等を補助情報として抽出する。
When it is determined as the reconstruction target block (step S103: Yes), auxiliary information to be transmitted to the decoding device to assist the reconstruction process is extracted from the reconstruction target block by the auxiliary information extraction process (step S104). . Note that the reconstruction process is a process of inversely transforming, on the decoding side, a block obtained by performing some kind of transformation described later on the block to be reconstructed. In the auxiliary information extraction process, for example, when reconstruction is performed by synthesizing a reconstruction target block by image synthesis, a representative texture used at the time of synthesis or a label identifying an object is extracted as auxiliary information.
抽出された補助情報は、補助情報エントロピー符号化処理によりエントロピー符号化され、補助情報の符号化データとなる。なお、補助情報エントロピー符号化処理には、例えばハフマン符号化又はランレングス符号化等の任意の符号化方法を用いることができる(ステップS105)。
The extracted auxiliary information is entropy encoded by the auxiliary information entropy encoding process, and becomes encoded data of the auxiliary information. For the auxiliary information entropy encoding process, any encoding method such as Huffman encoding or run-length encoding can be used (step S105).
補助情報が抽出された後の再構成対象ブロックは、画像変換処理により、HEVCでより少ない符号量で送信可能な画像に変換される(ステップS106)。なお、画像変換処理は、例えば再構成対象ブロックを当該ブロックの平均値に置き換えてもよいし、HEVCイントラ方向性予測において任意又は特定のモード番号で予測した場合の予測残差がゼロに漸近するような変換を施してもよい。
The block to be reconstructed after the auxiliary information is extracted is converted into an image that can be transmitted with a smaller code amount by HEVC by image conversion processing (step S106). In the image conversion process, for example, the reconstruction target block may be replaced with the average value of the block, and the prediction residual when predicting with an arbitrary or specific mode number in HEVC intra direction prediction is asymptotic to zero. Such conversion may be performed.
また、変換に用いたHEVCイントラ予測のモード番号を、補助情報の一部として復号側に送信してもよく、HEVCイントラ予測の特定のモード番号と復号側での再構成処理の方法とを対応付けて画像変換を行い、その対応関係を補助情報の一部として復号側へ送信してもよい。
Further, the mode number of HEVC intra prediction used for conversion may be transmitted to the decoding side as a part of the auxiliary information, and the specific mode number of HEVC intra prediction corresponds to the reconstruction processing method on the decoding side. In addition, image conversion may be performed, and the correspondence relationship may be transmitted to the decoding side as part of the auxiliary information.
例えば、Texture合成を再構成処理とする場合、イントラ予測モード番号と代表Textureとを対応付けて、その対応関係を補助情報として復号側へ送信してもよい。また、画像変換の方法は、HEVCイントラ予測に基づく変換以外の方法でもよい。入力ピクチャに存在しない出力を得ることができる任意の変換方法を、画像変換処理の過程で定義又は事前に定義したものから選択し、その変換方法を補助情報として復号側へ送信してもよい。
For example, when texture synthesis is used as a reconstruction process, an intra prediction mode number and a representative texture may be associated with each other and the corresponding relationship may be transmitted as auxiliary information to the decoding side. The image conversion method may be a method other than conversion based on HEVC intra prediction. An arbitrary conversion method capable of obtaining an output that does not exist in the input picture may be selected from those defined or previously defined in the course of the image conversion process, and the conversion method may be transmitted to the decoding side as auxiliary information.
変換後の画像(以下、「変換後画像」という。)は、変換後画像イントラ符号化処理で符号化し、変換後画像の符号化データを得る(ステップS107)。
The converted image (hereinafter referred to as “converted image”) is encoded by the converted image intra encoding process to obtain encoded data of the converted image (step S107).
上記の処理を、全てのブロックに対して処理順に適用し(ステップS108及びステップS109)、送信情報として補助情報の符号化データならびに変換後画像の符号化データを得る。
The above processing is applied to all blocks in the order of processing (step S108 and step S109), and encoded data of auxiliary information and encoded data of a converted image are obtained as transmission information.
[符号化装置の構成例]
次に、上記の処理を実現するための符号化装置の構成例について説明する。
図2に、第1の実施形態における符号化装置10の構成例を示す。図示するように、符号化装置10は、ブロック分割部101と、符号化方式判定部102と、補助情報抽出部103と、補助情報エントロピー符号化部104と、画像変換部105と、イントラ予測部107と、変換/量子化部108と、エントロピー符号化部109と、逆量子化/逆変換部110と、予測用メモリ111と、を含んで構成される。 [Configuration Example of Encoding Device]
Next, a configuration example of an encoding device for realizing the above processing will be described.
FIG. 2 shows a configuration example of theencoding device 10 in the first embodiment. As illustrated, the encoding apparatus 10 includes a block division unit 101, an encoding scheme determination unit 102, an auxiliary information extraction unit 103, an auxiliary information entropy encoding unit 104, an image conversion unit 105, and an intra prediction unit. 107, a transform / quantization unit 108, an entropy coding unit 109, an inverse quantization / inverse transform unit 110, and a prediction memory 111.
次に、上記の処理を実現するための符号化装置の構成例について説明する。
図2に、第1の実施形態における符号化装置10の構成例を示す。図示するように、符号化装置10は、ブロック分割部101と、符号化方式判定部102と、補助情報抽出部103と、補助情報エントロピー符号化部104と、画像変換部105と、イントラ予測部107と、変換/量子化部108と、エントロピー符号化部109と、逆量子化/逆変換部110と、予測用メモリ111と、を含んで構成される。 [Configuration Example of Encoding Device]
Next, a configuration example of an encoding device for realizing the above processing will be described.
FIG. 2 shows a configuration example of the
ブロック分割部101は、入力ピクチャを入力として、ブロック分割処理を行う。ブロック分割部101は、ブロック分割された入力ピクチャを出力する。
The block division unit 101 performs block division processing with the input picture as an input. The block division unit 101 outputs an input picture that has been divided into blocks.
符号化方式判定部102は、ブロック分割された入力ピクチャを入力として、符号化方式判定処理を行う。符号化方式判定部102は、ブロックの符号化方式の判定結果を出力する。
The encoding method determination unit 102 performs an encoding method determination process using an input picture divided into blocks as an input. The encoding method determination unit 102 outputs a determination result of the block encoding method.
補助情報抽出部103は、再構成対象ブロック及び参照ブロックを入力として、補助情報抽出処理を行う。参照ブロックは、再構成処理で参照すべき画素を含むブロックであり、例えば再構成処理として画像の補間合成を用いる場合、補間処理で参照する画素を含むブロックである。補助情報抽出部103は、補助情報を出力する。
The auxiliary information extraction unit 103 performs auxiliary information extraction processing with the reconstruction target block and the reference block as inputs. The reference block is a block including a pixel to be referred to in the reconstruction process. For example, in the case of using interpolation of an image as the reconstruction process, the reference block is a block including a pixel referred to in the interpolation process. The auxiliary information extraction unit 103 outputs auxiliary information.
補助情報エントロピー符号化部104は、入力された補助情報に対してエントロピー符号化を行い、補助情報の符号化データを得る。補助情報エントロピー符号化部104は、補助情報の符号化データを出力する。
The auxiliary information entropy encoding unit 104 performs entropy encoding on the input auxiliary information to obtain encoded data of the auxiliary information. The auxiliary information entropy encoding unit 104 outputs encoded data of auxiliary information.
画像変換部105は、再構成対象ブロックを入力として、画像変換処理を行う。画像変換部105は、変換後ブロックを出力する。
The image conversion unit 105 performs an image conversion process with the reconstruction target block as an input. The image conversion unit 105 outputs the converted block.
変換後ブロック及び非再構成対象ブロックは、イントラ符号化により符号化される。イントラ符号化では、イントラ予測部107から出力される予測画像との予測残差が、変換/量子化部108により直行変換及び量子化され、エントロピー符号化部109により符号化される。これにより、画像の符号化データが得られる。
なお、本実施形態においては、上記予測残差を符号化するエントロピー符号化部109と補助情報を符号化する補助情報エントロピー符号化部104とが別々の機能ブロックである構成としたが、これらが同一の機能ブロックで構成されてもよい。すなわち、1つの符号化部によって、例えば共通のエントロピー符号化方式で、上記予測残差の符号化及び補助情報の符号化が行われる構成であってもよい。 The post-conversion block and the non-reconstruction target block are encoded by intra encoding. In the intra coding, the prediction residual with the predicted image output from theintra prediction unit 107 is subjected to orthogonal transform and quantization by the transform / quantization unit 108 and encoded by the entropy coding unit 109. Thereby, encoded data of the image is obtained.
In the present embodiment, theentropy encoding unit 109 that encodes the prediction residual and the auxiliary information entropy encoding unit 104 that encodes auxiliary information are configured as separate functional blocks. You may be comprised by the same functional block. That is, the encoding residual encoding and the auxiliary information encoding may be performed by one encoding unit, for example, using a common entropy encoding scheme.
なお、本実施形態においては、上記予測残差を符号化するエントロピー符号化部109と補助情報を符号化する補助情報エントロピー符号化部104とが別々の機能ブロックである構成としたが、これらが同一の機能ブロックで構成されてもよい。すなわち、1つの符号化部によって、例えば共通のエントロピー符号化方式で、上記予測残差の符号化及び補助情報の符号化が行われる構成であってもよい。 The post-conversion block and the non-reconstruction target block are encoded by intra encoding. In the intra coding, the prediction residual with the predicted image output from the
In the present embodiment, the
変換/量子化部108により量子化された予測残差は、逆量子化/逆変換部110により逆量子化及び逆変換の処理がなされ、予測用メモリ111に蓄積される。予測用メモリ111に蓄積されたデータは、イントラ予測部107によるイントラ予測処理、及び、補助情報抽出部103による補助情報報抽出処理に用いられる。
The prediction residual quantized by the transform / quantization unit 108 is subjected to inverse quantization and inverse transform processing by the inverse quantization / inverse transform unit 110 and is stored in the prediction memory 111. The data stored in the prediction memory 111 is used for intra prediction processing by the intra prediction unit 107 and auxiliary information report extraction processing by the auxiliary information extraction unit 103.
[復号装置の処理]
次に、上記の処理方法及び機能構成によって生成された符号化データから画像を復号する復号装置の処理について説明する。
図3に、第1の実施形態における復号装置の処理フローを示す。 [Processing of decoding apparatus]
Next, the process of the decoding device that decodes an image from the encoded data generated by the above processing method and functional configuration will be described.
FIG. 3 shows a processing flow of the decoding device according to the first embodiment.
次に、上記の処理方法及び機能構成によって生成された符号化データから画像を復号する復号装置の処理について説明する。
図3に、第1の実施形態における復号装置の処理フローを示す。 [Processing of decoding apparatus]
Next, the process of the decoding device that decodes an image from the encoded data generated by the above processing method and functional configuration will be described.
FIG. 3 shows a processing flow of the decoding device according to the first embodiment.
変換後画像復号処理は、変換後画像の符号化データを復号し、変換後画像の復号画像のブロックを得る(ステップS201)。なお、復号画像は、入力画像に相当する単位の画像でもよいし、入力画像がブロック化されたブロックに相当する単位の画像でもよい。以下の各処理では、復号画像は、ブロックに相当する単位の画像であるものとして説明を続ける。
The post-conversion image decoding process decodes the encoded data of the post-conversion image to obtain a block of the decoded image of the post-conversion image (step S201). The decoded image may be a unit image corresponding to the input image, or a unit image corresponding to a block obtained by blocking the input image. In the following processes, the description will be continued assuming that the decoded image is an image of a unit corresponding to a block.
符号化方式判定処理は、符号化装置10の画像変換部105によって用いられた画像変換方法で変換されたブロックを、再構成対象ブロックとして判定する(ステップS202)。例えば、符号化装置10の画像変換部105が再構成対象ブロックを平均値で均一に置き換える処理を行う場合、符号化方式判定処理は、変換後画像の復号画像から得たブロックに対して、当該処理がなされたブロックを再構成対象ブロックとして判定する。
In the encoding method determination process, a block converted by the image conversion method used by the image conversion unit 105 of the encoding device 10 is determined as a reconstruction target block (step S202). For example, when the image conversion unit 105 of the encoding device 10 performs the process of uniformly replacing the reconstruction target block with the average value, the encoding method determination process is performed on the block obtained from the decoded image of the converted image. The processed block is determined as a reconstruction target block.
再構成対象ブロックに対しては(ステップS203・Yes)、符号化方式判定処理は、符号化装置10の補助情報エントロピー符号化部104で適用した符号化方式に基づき、当該再構成対象ブロックに対応する補助情報の符号化データを復号する(ステップS204)。
For the reconstruction target block (Yes in step S203), the coding method determination process corresponds to the reconstruction target block based on the coding method applied by the auxiliary information entropy coding unit 104 of the coding device 10. The encoded data of the auxiliary information to be decoded is decoded (step S204).
再構成処理は、補助情報と当該再構成対象ブロックが参照できる参照ブロックとを入力として、再構成処理を行う(ステップS205)。
In the reconstruction process, the auxiliary information and the reference block that can be referred to by the reconstruction target block are input, and the reconstruction process is performed (step S205).
上記の処理を、全てのブロックに対して処理順に適用し(ステップS206及びステップS207)、最終的な復号画像を得る。
The above processing is applied to all blocks in the order of processing (step S206 and step S207), and a final decoded image is obtained.
[復号装置の構成例]
次に、上記の処理を実現するための復号装置の構成例について説明する。
図4に、第1の実施形態における復号装置20の構成例を示す。図示するように、復号装置20は、エントロピー復号部201と、逆変換/逆量子化部202と、イントラ予測部203と、予測用メモリ204と、再構成部205と、符号化方式判定部206と、補助情報エントロピー復号部207と、を含んで構成される。 [Configuration Example of Decoding Device]
Next, a configuration example of a decoding device for realizing the above processing will be described.
FIG. 4 shows a configuration example of thedecoding device 20 in the first embodiment. As illustrated, the decoding device 20 includes an entropy decoding unit 201, an inverse transform / inverse quantization unit 202, an intra prediction unit 203, a prediction memory 204, a reconstruction unit 205, and an encoding scheme determination unit 206. And an auxiliary information entropy decoding unit 207.
次に、上記の処理を実現するための復号装置の構成例について説明する。
図4に、第1の実施形態における復号装置20の構成例を示す。図示するように、復号装置20は、エントロピー復号部201と、逆変換/逆量子化部202と、イントラ予測部203と、予測用メモリ204と、再構成部205と、符号化方式判定部206と、補助情報エントロピー復号部207と、を含んで構成される。 [Configuration Example of Decoding Device]
Next, a configuration example of a decoding device for realizing the above processing will be described.
FIG. 4 shows a configuration example of the
変換後画像の符号化データは、HEVCにより復号される。HEVCによる復号では、まず変換後画像の符号化データがエントロピー復号部201によりエントロピー復号され、逆変換/逆量子化部202により逆変換及び逆量子化の処理が施される。これにより、予測残差画像が復号され、イントラ予測部203による予測結果が加算されることで、変換後画像の復号画像のブロックが得られる。
The encoded data of the converted image is decoded by HEVC. In decoding by HEVC, first, encoded data of a converted image is entropy-decoded by an entropy decoding unit 201, and inverse transformation / inverse quantization processing is performed by an inverse transformation / inverse quantization unit 202. Accordingly, the prediction residual image is decoded, and the prediction result by the intra prediction unit 203 is added, so that a block of the decoded image of the converted image is obtained.
復号された変換後画像は、予測用メモリ204に蓄積されて、イントラ予測部203及び再構成部205への入力として用いられる。
The decoded converted image is accumulated in the prediction memory 204 and used as an input to the intra prediction unit 203 and the reconstruction unit 205.
符号化方式判定部206は、変換後画像の復号画像のブロックを入力とし、符号化方式判定処理を行い、判定結果を出力する。
The encoding method determination unit 206 receives the decoded image block of the converted image, performs an encoding method determination process, and outputs a determination result.
補助情報エントロピー復号部207は、入力された補助情報の符号化データに対してエントロピー復号を行い、補助情報を得る。補助情報エントロピー復号部207は、補助情報を再構成部205へ出力する。
The auxiliary information entropy decoding unit 207 performs entropy decoding on the encoded data of the input auxiliary information to obtain auxiliary information. The auxiliary information entropy decoding unit 207 outputs auxiliary information to the reconstruction unit 205.
再構成部205は、補助情報、再構成対象ブロックが参照できる参照画素及び再構成対象ブロックを入力として再構成処理を行い、最終的な出力ピクチャを出力する。
The reconstruction unit 205 performs reconstruction processing with the auxiliary information, the reference pixel that can be referred to by the reconstruction target block, and the reconstruction target block as inputs, and outputs a final output picture.
以上説明したように、上記実施形態に係る符号化方法及び復号方法では、従来技術とは異なり、入力画像に対し、処理ブロックの単位で再構成対象とするか又は非再構成対象とするかに分類して、再構成処理を適用する。上記実施形態に係る符号化方法及び復号方法は、ブロック単位で処理を行うことに制約することによって、境界情報を送信する際の符号量を少なくすることができる。上記実施形態に係る符号化方法及び復号方法は、例えば再構成対象ブロック内を平均値で置き換えるという規則を符号化装置10と復号装置20との間で共有させることにより、境界情報を送信することなく再構成対象ブロック位置の特定を実現することができる。
As described above, in the encoding method and the decoding method according to the above-described embodiment, unlike the related art, whether the input image is to be reconstructed in units of processing blocks or not to be reconstructed. Classify and apply reconstruction process. The encoding method and the decoding method according to the above embodiment can reduce the amount of code when transmitting boundary information by restricting the processing in units of blocks. In the encoding method and the decoding method according to the above embodiment, for example, the boundary information is transmitted by sharing a rule of replacing the inside of the reconstruction target block with an average value between the encoding device 10 and the decoding device 20. Therefore, it is possible to specify the position of the reconstruction target block.
従来、任意の形状で再構成対象を指定できる一方で、領域毎に、再構成対象であるか否かの情報及び再構成の方法を復号側に補助情報として送信する必要があった。これにより、従来、補助情報の符号量が多くなるという課題があった。一方、上記実施形態に係る符号化方法及び復号方法では、ブロック毎に再構成対象ブロックを指定し、指定された再構成対象ブロックに対して、符号化側でより少ない符号量でHEVCによって符号化できる処理(例えば、ブロック全体を平均値に置き換える処理)を施し、復号側で当該処理の有無を判定する処理を施す。これにより、上記実施形態に係る符号化方法及び復号方法では、補助情報として境界情報を復号側へ送信しなくても、復号側で再構成ブロックを判定することができる。また、HEVCのモード番号と再構成の方法とを関連付けることにより、再構成の方法も復号側へ同時に送信することができる。
Conventionally, while it is possible to designate a reconstruction target in an arbitrary shape, it has been necessary to transmit information about whether or not to be a reconstruction target and a reconstruction method as auxiliary information to the decoding side for each region. As a result, there has conventionally been a problem that the code amount of the auxiliary information increases. On the other hand, in the encoding method and decoding method according to the above embodiment, a reconstruction target block is specified for each block, and the specified reconstruction target block is encoded by HEVC with a smaller code amount on the encoding side. Processing that can be performed (for example, processing that replaces the entire block with an average value) is performed, and processing for determining the presence or absence of the processing is performed on the decoding side. Thereby, in the encoding method and decoding method according to the above-described embodiment, it is possible to determine a reconstructed block on the decoding side without transmitting boundary information to the decoding side as auxiliary information. Also, by associating the HEVC mode number with the reconstruction method, the reconstruction method can be transmitted to the decoding side at the same time.
<第2の実施形態>
以下、第2の実施形態について、図面を参照しながら説明する。以下に説明する第2の実施形態では、上述した第1の実施形態に対して、符号化装置及び復号装置の構成が異なる。 <Second Embodiment>
Hereinafter, the second embodiment will be described with reference to the drawings. In the second embodiment described below, the configurations of the encoding device and the decoding device are different from those of the first embodiment described above.
以下、第2の実施形態について、図面を参照しながら説明する。以下に説明する第2の実施形態では、上述した第1の実施形態に対して、符号化装置及び復号装置の構成が異なる。 <Second Embodiment>
Hereinafter, the second embodiment will be described with reference to the drawings. In the second embodiment described below, the configurations of the encoding device and the decoding device are different from those of the first embodiment described above.
[符号化装置の構成例]
第2の実施形態における符号化装置30の構成を図5に示す。図示するように、符号化装置30は、前処理装置31と、従来型符号化装置32と、から構成される。前処理装置31は、ブロック分割部301と、符号化方式判定部302と、補助情報抽出部303と、補助情報エントロピー符号化部304と、画像変換部305と、変換後画像メモリ306と、を含んで構成される。従来型符号化装置32は、イントラ予測部307と、変換/量子化部308と、エントロピー符号化部309と、逆量子化/逆変換部310と、予測用メモリ311と、を含んで構成される。 [Configuration Example of Encoding Device]
FIG. 5 shows the configuration of theencoding device 30 according to the second embodiment. As illustrated, the encoding device 30 includes a preprocessing device 31 and a conventional encoding device 32. The preprocessing device 31 includes a block division unit 301, an encoding scheme determination unit 302, an auxiliary information extraction unit 303, an auxiliary information entropy encoding unit 304, an image conversion unit 305, and a post-conversion image memory 306. Consists of including. The conventional coding apparatus 32 includes an intra prediction unit 307, a transform / quantization unit 308, an entropy coding unit 309, an inverse quantization / inverse transform unit 310, and a prediction memory 311. The
第2の実施形態における符号化装置30の構成を図5に示す。図示するように、符号化装置30は、前処理装置31と、従来型符号化装置32と、から構成される。前処理装置31は、ブロック分割部301と、符号化方式判定部302と、補助情報抽出部303と、補助情報エントロピー符号化部304と、画像変換部305と、変換後画像メモリ306と、を含んで構成される。従来型符号化装置32は、イントラ予測部307と、変換/量子化部308と、エントロピー符号化部309と、逆量子化/逆変換部310と、予測用メモリ311と、を含んで構成される。 [Configuration Example of Encoding Device]
FIG. 5 shows the configuration of the
図5に示すように、第2の実施形態における符号化装置30と第1の実施形態における符号化装置10との違いは、ブロック分割部、符号化方式判定部、画像変換部、補助情報抽出部及びエントロピー符号化部を備える装置が、前処理装置31として、その他の構成部(すなわち、従来型の符号化装置が備える構成部)とは独立して備えられる点である。
As shown in FIG. 5, the difference between the encoding device 30 in the second embodiment and the encoding device 10 in the first embodiment is that a block division unit, an encoding method determination unit, an image conversion unit, and auxiliary information extraction The apparatus provided with a part and an entropy encoding part is a point provided as the pre-processing apparatus 31 independently from other structural parts (namely, the structural part with which a conventional encoding apparatus is provided).
この場合、図5に例示した構成のように、変換後画像メモリ306に変換後画像が蓄積され、補助情報抽出部303が変換後画像メモリ306に蓄積された変換後画像を参照する構成であってもよい。前処理装置31に含まれる構成部以外の構成部は、従来型符号化装置32として独立して構成される。従来型符号化装置32として、例えば、HEVCのイントラ符号化装置のほか、JPEG(Joint Photographic Experts Group)等の画像符号化標準に則った符号化装置等を用いることができる。
In this case, as in the configuration illustrated in FIG. 5, the converted image is stored in the converted image memory 306, and the auxiliary information extraction unit 303 refers to the converted image stored in the converted image memory 306. May be. Components other than the components included in the preprocessing device 31 are configured independently as the conventional encoding device 32. As the conventional encoding device 32, for example, an HEVC intra encoding device, an encoding device conforming to an image encoding standard such as JPEG (JointoPhotographic Experts Group), or the like can be used.
なお、符号化装置30の処理の流れは図1に示した処理フローと共通であるため、説明を省略する。
Note that the processing flow of the encoding device 30 is the same as the processing flow shown in FIG.
[復号装置の構成例]
次に、第2の実施形態における復号装置40の構成を図6に示す。図示するように、復号装置40は、従来型復号装置41と、後処理装置42と、から構成される。従来型復号装置41は、エントロピー復号部401と、逆変換/逆量子化部402と、イントラ予測部403と、予測用メモリ404と、を含んで構成される。後処理装置42は、再構成部405と、符号化方式判定部406と、補助情報エントロピー復号部407と、を含んで構成される。 [Configuration Example of Decoding Device]
Next, the configuration of thedecoding device 40 in the second embodiment is shown in FIG. As shown in the figure, the decoding device 40 includes a conventional decoding device 41 and a post-processing device 42. The conventional decoding device 41 includes an entropy decoding unit 401, an inverse transform / inverse quantization unit 402, an intra prediction unit 403, and a prediction memory 404. The post-processing device 42 includes a reconstruction unit 405, an encoding scheme determination unit 406, and an auxiliary information entropy decoding unit 407.
次に、第2の実施形態における復号装置40の構成を図6に示す。図示するように、復号装置40は、従来型復号装置41と、後処理装置42と、から構成される。従来型復号装置41は、エントロピー復号部401と、逆変換/逆量子化部402と、イントラ予測部403と、予測用メモリ404と、を含んで構成される。後処理装置42は、再構成部405と、符号化方式判定部406と、補助情報エントロピー復号部407と、を含んで構成される。 [Configuration Example of Decoding Device]
Next, the configuration of the
図6に示すように、第2の実施形態における復号装置40と第1の実施形態における復号装置20との違いは、符号化方式判定部、補助情報エントロピー復号部、再構成部を備える装置が、後処理装置42として、その他の構成部(すなわち、従来型の復号装置が備える構成部)とは独立して備えられる点である。
As shown in FIG. 6, the difference between the decoding apparatus 40 in the second embodiment and the decoding apparatus 20 in the first embodiment is that an apparatus including an encoding scheme determination unit, an auxiliary information entropy decoding unit, and a reconstruction unit. The post-processing device 42 is provided independently from other components (that is, components included in the conventional decoding device).
この場合、図6に例示した構成のように、出力ピクチャメモリ408に出力ピクチャが蓄積され、再構成部405が出力ピクチャメモリ408に蓄積された出力ピクチャを参照する構成であってもよい。後処理装置42に含まれる構成部以外の構成部は、従来型復号装置41として独立して構成される。
In this case, as shown in the configuration illustrated in FIG. 6, the output picture memory 408 may store the output picture, and the reconstruction unit 405 may refer to the output picture stored in the output picture memory 408. Components other than the components included in the post-processing device 42 are configured independently as the conventional decoding device 41.
なお、復号装置40の処理の流れは図3に示した処理フローと共通であるため、説明を省略する。
Note that the processing flow of the decoding device 40 is the same as the processing flow shown in FIG.
以上説明した第2の実施形態に係る符号化方法及び復号方法によれば、従来の符号化装置及び復号装置と併用が可能な、前処理装置31及び後処理装置42を実現することができる。これにより、標準規格と、前処理装置31及び後処理装置42とにおいて、符号化効率の改善が加算的となるため、第2の実施形態に係る符号化方法及び復号方法によれば、標準規格に基づく符号化装置が高効率化した場合に、符号化装置30全体の符号化効率を改善できる。
According to the encoding method and the decoding method according to the second embodiment described above, the pre-processing device 31 and the post-processing device 42 that can be used in combination with the conventional encoding device and decoding device can be realized. Thereby, since the improvement of the encoding efficiency is additive in the standard and the pre-processing device 31 and the post-processing device 42, according to the encoding method and the decoding method according to the second embodiment, the standard When the efficiency of the encoding device based on is improved, the encoding efficiency of the entire encoding device 30 can be improved.
以下、再構成対象ブロックを、機械学習を用いた画像の補間合成処理により復号側で再構成する手段について説明する。当然、本手段を上記第1及び第2の実施形態において用いることは可能である。
<第3の実施形態>
以下、第3の実施形態について、図面を参照しながら説明する。 Hereinafter, means for reconstructing the reconstruction target block on the decoding side by image interpolation and synthesis processing using machine learning will be described. Of course, this means can be used in the first and second embodiments.
<Third Embodiment>
Hereinafter, a third embodiment will be described with reference to the drawings.
<第3の実施形態>
以下、第3の実施形態について、図面を参照しながら説明する。 Hereinafter, means for reconstructing the reconstruction target block on the decoding side by image interpolation and synthesis processing using machine learning will be described. Of course, this means can be used in the first and second embodiments.
<Third Embodiment>
Hereinafter, a third embodiment will be described with reference to the drawings.
上述したように、HEVCにおいて選択可能な各予測方式(Planar予測、DC予測及び方向性予測)は参照可能な画素を参照し、単純な予測ルールに基づいて予測を行うが、例えば画面内に高周波成分が無作為に分布する画像では予測効率が低下するという課題がある。このような画像では、予測残差信号の情報量が多くなるため、予測残差信号の量子化幅を一定として符号化した場合、符号量が過剰に発生する。
As described above, each prediction method (Planar prediction, DC prediction, and directionality prediction) that can be selected in HEVC refers to a referenceable pixel and performs prediction based on a simple prediction rule. There is a problem that prediction efficiency is lowered in an image in which components are randomly distributed. In such an image, since the amount of information of the prediction residual signal is large, when encoding is performed with the quantization width of the prediction residual signal being constant, the amount of code is excessively generated.
このような画像に対しても主観品質を保持したまま符号量を削減する圧縮符号化を実現する方法として、予測方式の高精度化以外に、上記の予測とは異なり、擬似的に画像を再構成する処理方式を導入する方法が考えられる。
Unlike the above prediction, as a method for realizing compression coding for reducing the code amount while maintaining the subjective quality for such an image, unlike the above prediction, the image is reproduced in a pseudo manner. A method of introducing a processing method to be configured is conceivable.
非特許文献1に記載の技術(以下、「従来技術1」という。)によれば、畳み込みニューラルネットワークにより構成される補間ネットワークと、畳み込みニューラルネットワークにより構成され補間ネットワークが補間した補間画像と補間されていない真の画像を識別する識別ネットワークの2つのネットワークを、敵対的生成ネットワークの枠組みに倣って交互に学習することで、補間ネットワークが画像の欠損領域を擬似的に再構成できるようになる。
According to the technique described in Non-Patent Document 1 (hereinafter referred to as “Prior Art 1”), an interpolation network constituted by a convolutional neural network and an interpolation image constituted by a convolutional neural network and interpolated by the interpolation network are interpolated. By alternately learning the two networks of the identification network that identify the true image that is not, following the framework of the hostile generation network, the interpolation network can reconstruct the missing region of the image in a pseudo manner.
従来技術1の補間ネットワークを復号側に適用することで、上述の予測効率が低下する画像の領域に対し復号側で画像を再構成でき、再構成領域の送信が不要になることから、符号量を削減することができる。
By applying the interpolation network of the prior art 1 to the decoding side, it is possible to reconstruct an image on the decoding side with respect to the above-described image region where the prediction efficiency is reduced, and transmission of the reconstructed region is not necessary. Can be reduced.
[補間ネットワークを用いた画像符号化、復号処理の例]
補間ネットワークを用いた画像符号化、復号処理の例を図7に示す。 [Example of image encoding / decoding process using interpolation network]
An example of image encoding / decoding processing using an interpolation network is shown in FIG.
補間ネットワークを用いた画像符号化、復号処理の例を図7に示す。 [Example of image encoding / decoding process using interpolation network]
An example of image encoding / decoding processing using an interpolation network is shown in FIG.
画像欠損処理は、入力画像から画像補間により復号側で再構成対象とする領域を選択し、欠損させて欠損画像を生成し、欠損領域を示す欠損領域情報とともに出力する(ステップS301)。ここで、欠損領域情報は欠損領域を示す2値画像等である。
In the image loss processing, a region to be reconstructed is selected on the decoding side by image interpolation from the input image, a loss image is generated by loss, and output together with the loss region information indicating the loss region (step S301). Here, the missing area information is a binary image or the like showing the missing area.
欠損領域情報符号化処理は、欠損領域情報を復号側に送信するため、欠損領域情報を符号化する処理を、JPEG(Joint Photographic Experts Group)やHEVC等の従来の画像符号化方式や、ランレングス符号化等のエントロピー符号化方式により行う。これにより、欠損領域情報符号化処理は、欠損領域情報の符号化データを得る(ステップS302)。
In the defect area information encoding process, since the defect area information is transmitted to the decoding side, a process for encoding the defect area information is performed by using a conventional image encoding method such as JPEG (Joint Photographic Experts Group) or HEVC, or a run length. This is performed by an entropy encoding method such as encoding. Thus, the missing area information encoding process obtains encoded data of the missing area information (step S302).
画像符号化処理は、欠損画像をJPEGやHEVC等の従来の画像符号化方式を用いて符号化処理を行う。これにより、画像符号化処理は、欠損画像の符号化データを得る(ステップS303)。
In the image encoding process, the missing image is encoded using a conventional image encoding method such as JPEG or HEVC. Thereby, the image encoding process obtains encoded data of the missing image (step S303).
画像復号処理は、欠損画像の符号化データから復号済み欠損画像を得る(ステップS304)。
In the image decoding process, a decoded missing image is obtained from the encoded data of the missing image (step S304).
欠損領域情報復号処理は、欠損領域情報の符号化データから、欠損領域情報を得る(ステップS305)。
The missing area information decoding process obtains missing area information from the encoded data of the missing area information (step S305).
画像補間処理は、従来技術1の補間ネットワークに対し、復号済み欠損画像と欠損領域情報とを入力し、最終的な出力画像を得る。なお、符号化処理及び復号処理の処理単位は、画面全体としてもよいし、HEVCのCTUのような構造を用いて画面を分割したブロック単位としてもよい(ステップS306)。
In the image interpolation process, the decoded missing image and the missing area information are input to the interpolation network of the conventional technique 1 to obtain a final output image. The processing unit of the encoding process and the decoding process may be the entire screen, or may be a block unit obtained by dividing the screen using a structure such as HEVC CTU (step S306).
[符号化装置及び復号装置の構成例]
上記の符号化処理及び復号処理を実現する符号化装置50及び復号装置60の構成例を、図8に示す。図示するように、符号化装置50は、画像欠損処理部501と、画像符号化部502と、欠損領域情報符号化部503と、から構成される。 [Configuration Examples of Encoding Device and Decoding Device]
FIG. 8 shows a configuration example of theencoding device 50 and the decoding device 60 that realize the above encoding processing and decoding processing. As illustrated, the encoding device 50 includes an image loss processing unit 501, an image encoding unit 502, and a missing region information encoding unit 503.
上記の符号化処理及び復号処理を実現する符号化装置50及び復号装置60の構成例を、図8に示す。図示するように、符号化装置50は、画像欠損処理部501と、画像符号化部502と、欠損領域情報符号化部503と、から構成される。 [Configuration Examples of Encoding Device and Decoding Device]
FIG. 8 shows a configuration example of the
画像欠損処理部501は、入力画像を入力とし、画像欠損処理を行う。これにより、画像欠損処理部501は、欠損画像と欠損領域情報を出力する。
The image loss processing unit 501 receives the input image and performs image loss processing. As a result, the image defect processing unit 501 outputs a defect image and defect area information.
画像符号化部502は、欠損画像を入力とし、画像符号化処理を行う。これにより、画像符号化部502は、欠損画像の符号化データを出力する。
The image encoding unit 502 receives the missing image and performs image encoding processing. As a result, the image encoding unit 502 outputs encoded data of the missing image.
欠損領域情報符号化部503は、欠損領域情報を入力とし、欠損領域情報符号化処理を行う。これにより、欠損領域情報符号化部503は、欠損領域情報の符号化データを出力する。
The missing area information encoding unit 503 receives the missing area information as input and performs a missing area information encoding process. Thereby, the missing area information encoding unit 503 outputs encoded data of the missing area information.
欠損画像の符号化データならびに欠損領域情報の符号化データは、復号装置60に送信される。
The encoded data of the missing image and the encoded data of the missing area information are transmitted to the decoding device 60.
図8に示すように、復号装置60は、画像復号部601と、欠損領域情報復号部602と、画像補間部603と、から構成される。
As shown in FIG. 8, the decoding device 60 includes an image decoding unit 601, a missing area information decoding unit 602, and an image interpolation unit 603.
画像復号部601は、欠損画像の符号化データを入力とし、画像復号処理を行う。これにより、画像復号部601は、復号済み欠損画像を得る。
The image decoding unit 601 receives the encoded data of the missing image and performs an image decoding process. Thereby, the image decoding unit 601 obtains a decoded missing image.
欠損領域情報復号部602は、欠損領域情報の符号化データを入力とし、欠損領域情報復号処理を行う。これにより、欠損領域情報を得る。
The missing area information decoding unit 602 receives the encoded data of the missing area information as input and performs a missing area information decoding process. Thereby, defect area information is obtained.
画像補間部603は、画像補間ネットワーク604を備え、復号済み欠損画像と欠損領域情報を入力とし、画像補間処理を行う。これにより、画像補間部603は、最終的な出力画像を得る。
The image interpolation unit 603 includes an image interpolation network 604, and receives the decoded missing image and missing area information as input, and performs image interpolation processing. Thereby, the image interpolation unit 603 obtains a final output image.
上記の構成では、画像補間処理において欠損画像の欠損領域の面積に出力画像の主観画質が大きく依存する。具体的には、補間すべき欠損領域の面積が大きくなるほど、補間ネットワークに入力される情報量が少なくなるため、画像補間処理における欠損領域の推定が困難となり、出力画像の主観画質が劣化する。また、上記の構成では、補間すべき欠損領域に、参照可能な領域から推論できない複雑な要素が含まれていた場合に、復号側で再構成されない、もしくは出力の主観画質が劣化する。
In the above configuration, the subjective image quality of the output image greatly depends on the area of the missing area of the missing image in the image interpolation process. Specifically, the larger the area of the missing area to be interpolated, the smaller the amount of information input to the interpolation network, making it difficult to estimate the missing area in the image interpolation process and degrading the subjective image quality of the output image. Further, in the above configuration, if the missing region to be interpolated includes a complex element that cannot be inferred from the referenceable region, it is not reconstructed on the decoding side, or the subjective image quality of the output deteriorates.
よって、欠損領域の面積が大きな場合や欠損領域が複雑な場合にも、主観画質の劣化を抑制しながら画像補間処理を実行できる画像補間処理を含む符号化方式及び復号方式、ならびに構成要素となるネットワークの効率的な学習方法が望まれる。
Therefore, even when the area of the missing area is large or when the missing area is complex, it becomes an encoding method and decoding method including image interpolation processing capable of executing image interpolation processing while suppressing deterioration of subjective image quality, and a constituent element An efficient network learning method is desired.
以下、畳み込みニューラルネットワークを用い、識別ネットワークを用いた敵対的生成ネットワークによる学習を例に、本発明の第3の実施形態について説明するが、本発明は畳み込みニューラルネットワークによる画像補間及び敵対的生成ネットワークの枠組みによる学習に限定されるものではない。つまり、画像補間に対しては、学習によりその画像補間方法が獲得される任意の学習モデルを適用できる。また、その学習方法に対しては、任意の誤差関数を用いた学習方法を適用できる。
Hereinafter, the third embodiment of the present invention will be described using learning by a hostile generation network using a convolutional neural network and an identification network as an example. The present invention describes image interpolation and hostile generation network by a convolutional neural network. It is not limited to learning by the framework of That is, any learning model in which the image interpolation method is acquired by learning can be applied to image interpolation. In addition, a learning method using an arbitrary error function can be applied to the learning method.
第3の実施形態では、符号化装置は、原画像を参照して特徴抽出を行い、画像補間を補助するための画像補間補助情報を、復号装置へ送信する。復号装置は、画像補間補助情報を用いて画像補間を行う。また、画像補間補助情報の抽出及び画像補間に用いられるネットワークは、ネットワーク毎に個別に最適化がなされた後、各ネットワークが結合されて全体最適化される。
In the third embodiment, the encoding device performs feature extraction with reference to the original image, and transmits image interpolation auxiliary information for assisting image interpolation to the decoding device. The decoding device performs image interpolation using the image interpolation auxiliary information. Further, the networks used for extraction of image interpolation auxiliary information and image interpolation are individually optimized for each network, and then the networks are combined to be optimized as a whole.
[符号化処理及び復号処理の流れ]
まず、本発明による補間ネットワーク及び補助情報抽出ネットワークを用いた符号化処理及び復号処理について概要を説明する。
図9に、第3の実施形態による符号化処理及び復号処理の流れを示す。 [Flow of encoding process and decoding process]
First, an outline of encoding processing and decoding processing using the interpolation network and auxiliary information extraction network according to the present invention will be described.
FIG. 9 shows the flow of encoding processing and decoding processing according to the third embodiment.
まず、本発明による補間ネットワーク及び補助情報抽出ネットワークを用いた符号化処理及び復号処理について概要を説明する。
図9に、第3の実施形態による符号化処理及び復号処理の流れを示す。 [Flow of encoding process and decoding process]
First, an outline of encoding processing and decoding processing using the interpolation network and auxiliary information extraction network according to the present invention will be described.
FIG. 9 shows the flow of encoding processing and decoding processing according to the third embodiment.
画像欠損処理は、入力画像から画像補間により復号側で再構成対象とする領域を選択する。画像欠損処理は、当該領域を、例えば平均値に置き換える等の処理により欠損させて欠損画像を生成する。画像欠損処理は、生成した欠損画像を、欠損させた領域の画素値の集合である欠損領域の位置を示す欠損領域情報とともに出力する。
In the image loss process, an area to be reconstructed is selected on the decoding side by image interpolation from the input image. In the image loss process, a defective image is generated by deleting the area by a process such as replacing the area with an average value. In the image defect process, the generated defect image is output together with the defect area information indicating the position of the defect area, which is a set of pixel values of the defect area.
ここで、欠損領域情報としては、例えば欠損領域を示す2値マスク画像(以下、欠損領域マスク画像)を用いることができる。また、画像欠損処理における領域選択方法としては、HEVCのイントラ符号化において固定量子化幅を用いた際の発生符号量が多い領域を選択する方法、又は、画像認識で用いられる物体毎に領域分割を実行し補間可能な領域として選択する方法等を用いることができる(ステップS401)。
Here, as the defect area information, for example, a binary mask image (hereinafter, a defect area mask image) indicating a defect area can be used. In addition, as a region selection method in image loss processing, a method of selecting a region with a large amount of generated codes when using a fixed quantization width in HEVC intra coding, or region division for each object used in image recognition Can be used to select a region that can be interpolated (step S401).
補助情報抽出処理は、入力画像のうち欠損領域情報から導出される欠損領域に対応する領域、又は、入力画像そのものから、画像補間補助情報抽出のためのネットワークを用いて画像補間補助情報を抽出する(ステップS402)。画像補間補助情報抽出のためのネットワークの詳細は後述する。
In the auxiliary information extraction process, image interpolation auxiliary information is extracted from an area corresponding to a missing area derived from the missing area information in the input image or the input image itself using a network for extracting image interpolation auxiliary information. (Step S402). Details of the network for extracting image interpolation auxiliary information will be described later.
補助情報符号化処理は、補助情報抽出処理によって抽出された画像補間補助情報を、ハフマン符号化等の従来のエントロピー符号化方式により符号化する。これにより、補助情報符号化処理は、画像補間補助情報の符号化データを得る(ステップS403)。
The auxiliary information encoding process encodes the image interpolation auxiliary information extracted by the auxiliary information extraction process by a conventional entropy encoding method such as Huffman encoding. Thus, the auxiliary information encoding process obtains encoded data of the image interpolation auxiliary information (step S403).
欠損領域情報符号化処理は、欠損領域情報を復号側に送信するため、再構成対象領域を符号化する処理を、JPEGやHEVC等の従来の画像符号化方式や、ランレングス符号化等のエントロピー符号化方式により行う。これにより、欠損領域情報符号化処理は、欠損領域情報の符号化データを得る(ステップS404)。
In the missing area information encoding process, since the missing area information is transmitted to the decoding side, the process for encoding the reconstruction target area is performed using a conventional image encoding method such as JPEG or HEVC, or entropy such as run-length encoding. This is done according to the encoding method. Thereby, the missing area information encoding process obtains encoded data of the missing area information (step S404).
画像符号化処理は、欠損画像に対して、JPEGやHEVC等の従来の画像符号化方式を用いて符号化処理を行う。これにより、画像符号化処理は、欠損画像の符号化データを得る(ステップS405)。
In the image encoding process, a defective image is encoded using a conventional image encoding method such as JPEG or HEVC. Thus, the image encoding process obtains encoded data of the missing image (step S405).
画像復号処理は、欠損画像の符号化データから、復号済み欠損画像を得る(ステップS406)。
In the image decoding process, a decoded missing image is obtained from the encoded data of the missing image (step S406).
欠損領域情報復号処理は、欠損領域情報の符号化データから、欠損領域情報を得る(ステップS407)。
The missing area information decoding process obtains missing area information from the encoded data of the missing area information (step S407).
補助情報復号処理は、画像補間補助情報の符号化データから、画像補間補助情報を得る(ステップS407)。
The auxiliary information decoding process obtains image interpolation auxiliary information from the encoded data of the image interpolation auxiliary information (step S407).
画像補間処理は、画像補間のためのネットワークに、復号済み欠損画像、欠損領域情報、及び画像補間補助情報を入力し、最終的な出力画像を得る。画像補間のためのネットワークの詳細については、後述する(ステップS408)。
In the image interpolation process, the decoded missing image, the missing region information, and the image interpolation auxiliary information are input to a network for image interpolation, and a final output image is obtained. Details of the network for image interpolation will be described later (step S408).
なお、符号化処理及び復号処理の処理単位は、画面全体としてもよいし、HEVCのCTUのような構造を用いて画面を分割したブロック単位としてもよい。
The processing unit of the encoding process and the decoding process may be the entire screen, or may be a block unit obtained by dividing the screen using a structure such as HEVC CTU.
[符号化装置及び復号装置の構成例]
次に、上記の符号化処理及び復号処理を実現する符号化装置及び復号装置の構成例を、図10に示す。図示するように、符号化装置70は、画像欠損処理部701と、画像符号化部702と、欠損領域情報符号化部703と、補助情報抽出部704と、補助情報符号化部705と、から構成される。 [Configuration Examples of Encoding Device and Decoding Device]
Next, FIG. 10 shows a configuration example of an encoding device and a decoding device that realize the above encoding processing and decoding processing. As illustrated, theencoding device 70 includes an image loss processing unit 701, an image encoding unit 702, a missing region information encoding unit 703, an auxiliary information extracting unit 704, and an auxiliary information encoding unit 705. Composed.
次に、上記の符号化処理及び復号処理を実現する符号化装置及び復号装置の構成例を、図10に示す。図示するように、符号化装置70は、画像欠損処理部701と、画像符号化部702と、欠損領域情報符号化部703と、補助情報抽出部704と、補助情報符号化部705と、から構成される。 [Configuration Examples of Encoding Device and Decoding Device]
Next, FIG. 10 shows a configuration example of an encoding device and a decoding device that realize the above encoding processing and decoding processing. As illustrated, the
画像欠損処理部701は、入力画像を入力とし、画像欠損処理を行う。これにより、画像欠損処理部701は、欠損画像と欠損領域情報とを出力する。
The image loss processing unit 701 receives an input image and performs image loss processing. Accordingly, the image defect processing unit 701 outputs a defect image and defect area information.
画像符号化部702は、欠損画像を入力とし、画像符号化処理を行う。これにより、画像符号化部702は、欠損画像の符号化データを出力する。
The image encoding unit 702 receives the missing image and performs image encoding processing. As a result, the image encoding unit 702 outputs encoded data of the missing image.
欠損領域情報符号化部703は、欠損領域情報を入力とし、欠損領域情報符号化処理を行う。これにより、欠損領域情報符号化部703は、欠損領域情報の符号化データを出力する。
The missing area information encoding unit 703 receives the missing area information as input and performs a missing area information encoding process. Thereby, the missing area information encoding unit 703 outputs encoded data of the missing area information.
補助情報抽出部704は、入力画像のうち欠損領域情報から導出される欠損領域に対応する領域、又は、欠損領域でない領域を含む画像全体を入力とし、補助情報抽出処理を行う。これにより、補助情報抽出部704は、画像補間補助情報を抽出する。
The auxiliary information extraction unit 704 performs an auxiliary information extraction process by using, as input, an area corresponding to the missing area derived from the missing area information in the input image or an entire image including an area that is not a missing area. As a result, the auxiliary information extraction unit 704 extracts image interpolation auxiliary information.
補助情報符号化部705は、画像補間補助情報を入力とし、補助情報符号化処理を行う。これにより、補助情報符号化部705は、画像補間補助情報の符号化データを出力する。
The auxiliary information encoding unit 705 receives the image interpolation auxiliary information and performs auxiliary information encoding processing. Thereby, the auxiliary information encoding unit 705 outputs encoded data of the image interpolation auxiliary information.
欠損画像の符号化データ、欠損領域情報の符号化データ及び画像補間補助情報の符号化データは、復号装置80へ送信される。
The encoded data of the missing image, the encoded data of the missing area information, and the encoded data of the image interpolation auxiliary information are transmitted to the decoding device 80.
図10に示すように、復号装置80は、画像復号部801と、欠損領域情報復号部802と、画像補間部803と、補助情報復号部805と、から構成される。
As shown in FIG. 10, the decoding device 80 includes an image decoding unit 801, a missing region information decoding unit 802, an image interpolation unit 803, and an auxiliary information decoding unit 805.
画像復号部801は、欠損画像の符号化データを入力とし、画像復号処理を行う。これにより、画像復号部801は、復号済み欠損画像を得る。
The image decoding unit 801 receives the encoded data of the missing image and performs an image decoding process. Thereby, the image decoding unit 801 obtains a decoded missing image.
欠損領域情報復号部802は、欠損領域情報の符号化データを入力とし、欠損領域情報復号処理を行う。これにより、欠損領域情報復号部802は、欠損領域情報を得る。
The missing area information decoding unit 802 receives the encoded data of the missing area information as input and performs a missing area information decoding process. Thereby, the missing area information decoding unit 802 obtains missing area information.
補助情報復号部805は、画像補間補助情報の符号化データを入力とし、補助情報復号処理を行う。これにより、補助情報復号部805は、画像補間補助情報を得る。
The auxiliary information decoding unit 805 receives the encoded data of the image interpolation auxiliary information and performs auxiliary information decoding processing. Thereby, the auxiliary information decoding unit 805 obtains image interpolation auxiliary information.
画像補間部803は、復号済み欠損画像、欠損領域情報及び画像補間補助情報を入力とし、画像補間補助情報を参照した画像補間処理を行う。これにより、画像補間部803は、最終的な出力画像を得る。
The image interpolation unit 803 receives the decoded missing image, the missing region information, and the image interpolation auxiliary information, and performs an image interpolation process with reference to the image interpolation auxiliary information. Thereby, the image interpolation unit 803 obtains a final output image.
[補助情報抽出部と画像補間部の構成ならびに学習方法]
次に、補助情報抽出部704及び画像補間部803の構成、ならびに学習方法について説明する。 [Configuration and learning method of auxiliary information extraction unit and image interpolation unit]
Next, the configuration of the auxiliaryinformation extraction unit 704 and the image interpolation unit 803 and the learning method will be described.
次に、補助情報抽出部704及び画像補間部803の構成、ならびに学習方法について説明する。 [Configuration and learning method of auxiliary information extraction unit and image interpolation unit]
Next, the configuration of the auxiliary
補助情報抽出部704及び画像補間部803のネットワークの構成を図11に示す。図示するように、補助情報抽出部704は、復号側に送信する画像補間補助情報を抽出するための補助情報抽出ネットワーク7041から構成される。
FIG. 11 shows a network configuration of the auxiliary information extraction unit 704 and the image interpolation unit 803. As shown in the figure, the auxiliary information extraction unit 704 includes an auxiliary information extraction network 7041 for extracting image interpolation auxiliary information to be transmitted to the decoding side.
補助情報抽出ネットワーク7041は、入力画像及び欠損領域情報を入力として、画像補間補助情報を出力するネットワークである。補助情報抽出ネットワーク7041は、例えば入力を入力画像及び欠損領域マスク画像の2枚の画像とし、出力を任意の数のユニットとして、畳み込み層や全結合層等により中間層を構成する。
The auxiliary information extraction network 7041 is a network that receives the input image and the missing area information and outputs image interpolation auxiliary information. The auxiliary information extraction network 7041 configures an intermediate layer by a convolutional layer, a fully connected layer, or the like, for example, with an input as an input image and a defective area mask image as two images and an output as an arbitrary number of units.
図11に示すように、画像補間部803は、画像補間補助情報を参照して欠損領域を予測するための補助情報参照ネットワーク8031、欠損画像を参照して欠損領域を予測するための欠損画像参照ネットワーク8032、及び、前記2つのネットワークの出力から最終的な補間画像を生成するための再構成ネットワーク8033から構成される。
As illustrated in FIG. 11, the image interpolation unit 803 refers to the auxiliary information reference network 8031 for predicting the missing area with reference to the image interpolation auxiliary information, and the missing image reference for predicting the missing area with reference to the missing image. A network 8032 and a reconstruction network 8033 for generating a final interpolated image from the outputs of the two networks.
補助情報参照ネットワーク8031は、画像補間補助情報を入力として、補助情報参照による中間画像を出力するネットワークである。補助情報参照ネットワーク8031は、例えば入力を画像補間補助情報と同数のユニットとし、出力を1枚の補助情報参照による中間画像として、全結合層、逆畳み込み層、及び、畳み込み層等により中間層を構成する。
The auxiliary information reference network 8031 is a network that receives the image interpolation auxiliary information and outputs an intermediate image by referring to the auxiliary information. The auxiliary information reference network 8031 has, for example, the same number of units as the image interpolation auxiliary information and the output as an intermediate image by referring to one auxiliary information, and the intermediate layer is formed by a fully connected layer, a deconvolution layer, a convolution layer, and the like. Constitute.
欠損画像参照ネットワーク8032は、入力画像の欠損画像及び欠損領域マスク画像を入力として、欠損画像参照による中間画像を出力するネットワークである。欠損画像参照ネットワーク8032は、例えば入力を入力画像の欠損画像及び欠損領域マスク画像の2枚の画像、出力を1枚の欠損画像参照による中間画像として、畳み込み層、全結合層、及び、逆畳み込み層等により中間層を構成する。
The missing image reference network 8032 is a network that outputs the intermediate image by referring to the missing image with the missing image and the missing area mask image of the input image as inputs. The missing image reference network 8032 has, for example, a convolutional layer, a fully connected layer, and a deconvolution using, as input, two images, a missing image of the input image and a missing region mask image, and an output as an intermediate image by referring to one missing image. An intermediate layer is constituted by layers and the like.
再構成ネットワーク8033は、補助情報参照による中間画像及び欠損画像参照による中間画像を入力として、欠損領域が補間された最終的な出力画像を出力するネットワークである。再構成ネットワーク8033は、例えば入力を2枚の中間画像とし、出力を1枚の出力画像として、畳み込み層、全結合層、及び、逆畳み込み層等により中間層を構成する。
The reconstruction network 8033 is a network that receives an intermediate image based on auxiliary information reference and an intermediate image based on missing image reference and outputs a final output image in which a missing area is interpolated. The reconstruction network 8033 includes, for example, two intermediate images as input and one output image as output, and forms an intermediate layer including a convolution layer, a fully connected layer, a deconvolution layer, and the like.
以上の構成により、補助情報抽出部704及び画像補間部803を学習する。学習時は、従来技術1と同様に、敵対的生成ネットワークの枠組みを用いることができる。このとき、従来技術1と同様、補間した領域の自然さを評価するための識別ネットワーク9000は、画像補間部803の出力画像を入力とし、出力画像が補間されていない真の画像である確率を出力する。
With the above configuration, the auxiliary information extraction unit 704 and the image interpolation unit 803 are learned. At the time of learning, the framework of the hostile generation network can be used as in the prior art 1. At this time, as in the prior art 1, the identification network 9000 for evaluating the naturalness of the interpolated region receives the output image of the image interpolating unit 803 as an input, and calculates the probability that the output image is a true image that has not been interpolated. Output.
次に、図11の構成を用いたネットワークの学習方法について説明する。学習処理は、教師データとして、原画像と、原画像に欠損領域をランダムに与えて生成した原画像の欠損画像と、欠損領域情報との組を多数用意する。学習で用いる誤差関数としては、例えば原画像とネットワークの出力画像の画素の平均二乗誤差(以下、平均二乗誤差)、及び敵対的生成ネットワークの枠組みを適用し、識別ネットワークによってネットワークの出力画像が識別された誤差(以下、「識別ネットワーク誤差」という。)、又は平均二乗誤差と識別ネットワーク誤差の重み付き和による誤差(以下、重み付き誤差)のいずれかを用いることができる。
Next, a network learning method using the configuration of FIG. 11 will be described. In the learning process, a large number of sets of original images, missing images of original images generated by randomly assigning missing regions to the original images, and missing region information are prepared as teacher data. As the error function used in learning, for example, the mean square error of the pixels of the original image and the output image of the network (hereinafter referred to as mean square error) and the framework of the hostile generation network are applied, and the output image of the network is identified by the identification network. Error (hereinafter referred to as “identification network error”) or an error due to a weighted sum of the mean square error and the identification network error (hereinafter referred to as weighted error) can be used.
[ネットワークの学習方法]
学習処理の流れを図12に示す。 [Network learning method]
The flow of the learning process is shown in FIG.
学習処理の流れを図12に示す。 [Network learning method]
The flow of the learning process is shown in FIG.
欠損画像参照ネットワーク学習処理は、図11の欠損画像参照ネットワーク8032及び識別ネットワーク9000を切り出し、図13のように結合して、欠損画像参照ネットワーク8032の出力を識別ネットワーク9000への入力とみなし、欠損画像参照ネットワーク8032を学習する(ステップS501)。
In the missing image reference network learning process, the missing image reference network 8032 and the identification network 9000 shown in FIG. 11 are cut out and combined as shown in FIG. 13, and the output of the missing image reference network 8032 is regarded as an input to the identification network 9000. The image reference network 8032 is learned (step S501).
具体的には、欠損画像参照ネットワーク学習処理は、原画像の欠損画像と欠損領域情報とを欠損画像参照ネットワーク8032に入力し、出力される画像が原画像に近付くよう、誤差逆伝播法によりネットワークのパラメータを更新する。ここで、欠損画像参照ネットワーク学習処理は、誤差関数として、まず平均二乗誤差を適用して学習を行った後、重み付き誤差を適用して学習を行う。以降の各ネットワークの学習処理でも、同様に平均二乗誤差を用いて学習が行われた後、重み付き誤差を用いて学習が行われる。
Specifically, in the missing image reference network learning process, the missing image and missing area information of the original image are input to the missing image reference network 8032, and the output image approaches the original image by the error back propagation method. Update the parameters. Here, in the missing image reference network learning process, learning is performed by first applying a mean square error as an error function, and then performing learning by applying a weighted error. In the subsequent learning processing of each network, learning is similarly performed using the mean square error, and then learning is performed using the weighted error.
補助情報抽出/参照ネットワーク学習処理は、図11の補助情報抽出ネットワーク7041、補助情報参照ネットワーク8031、及び識別ネットワーク9000を切り出し、図14のように結合して、補助情報参照ネットワーク8031の出力を識別ネットワーク9000への入力とみなし、補助情報抽出ネットワーク7041と補助情報参照ネットワーク8031とを学習する(ステップS502)。
In the auxiliary information extraction / reference network learning process, the auxiliary information extraction network 7041, the auxiliary information reference network 8031, and the identification network 9000 shown in FIG. 11 are cut out and combined as shown in FIG. 14 to identify the output of the auxiliary information reference network 8031. It is regarded as an input to the network 9000, and the auxiliary information extraction network 7041 and the auxiliary information reference network 8031 are learned (step S502).
具体的には、補助情報抽出/参照ネットワーク学習処理は、原画像と欠損領域情報とを、補助情報抽出ネットワーク7041及び補助情報参照ネットワーク8031が結合されたネットワークに入力する。補助情報抽出/参照ネットワーク学習処理は、出力される画像が原画像に近付くよう、平均二乗誤差と重み付き誤差を順に適用して、誤差逆伝播法によりネットワークのパラメータを更新する。
Specifically, in the auxiliary information extraction / reference network learning process, the original image and the missing area information are input to a network in which the auxiliary information extraction network 7041 and the auxiliary information reference network 8031 are combined. In the auxiliary information extraction / reference network learning process, the mean square error and the weighted error are sequentially applied so that the output image approaches the original image, and the network parameters are updated by the error back propagation method.
再構成ネットワーク学習処理は、欠損画像参照ネットワーク学習処理及び補助情報抽出/参照ネットワーク学習処理で構築された欠損画像参照ネットワーク8032、補助情報抽出ネットワーク7041、補助情報参照ネットワーク8031、再構成ネットワーク8033、及び識別ネットワーク9000を図11のように結合し、再構成ネットワーク8033のみ学習する(ステップS503)。
The reconstruction network learning process includes a missing image reference network 8032, an auxiliary information extraction network 7041, an auxiliary information reference network 8031, a reconstruction network 8033, and a defect image reference network learning process and an auxiliary information extraction / reference network learning process. The identification networks 9000 are combined as shown in FIG. 11, and only the reconfiguration network 8033 is learned (step S503).
具体的には、再構成ネットワーク学習処理は、原画像、原画像の欠損画像、及び欠損領域情報を結合されたネットワークに入力し、出力される画像が原画像に近付くよう、平均二乗誤差と重み付き誤差を順に適用して、再構成ネットワークのパラメータのみを誤差逆伝播法により更新する。
Specifically, the reconstruction network learning process inputs the original image, the missing image of the original image, and the missing area information to the combined network, and the mean square error and the weight so that the output image approaches the original image. The attached error is applied in order, and only the parameters of the reconstruction network are updated by the error back propagation method.
全体学習処理は、再構成ネットワーク学習処理において図11のように結合された、欠損画像参照ネットワーク8032、補助情報抽出ネットワーク7041、補助情報参照ネットワーク8031、及び再構成ネットワーク8033を同時に学習する(ステップS504)。
The whole learning process simultaneously learns the missing image reference network 8032, the auxiliary information extraction network 7041, the auxiliary information reference network 8031, and the reconstruction network 8033 that are combined as shown in FIG. 11 in the reconstruction network learning process (step S504). ).
具体的には、全体学習処理は、原画像、原画像の欠損画像、及び欠損領域情報を結合されたネットワークに入力し、出力される画像が原画像に近付くよう、平均二乗誤差と重み付き誤差を順に適用して、全ネットワークのパラメータを誤差逆伝播法により更新する。
なお、補助情報抽出ネットワークのみネットワークのパラメータを固定して学習する構成であってもよい。 Specifically, the whole learning process is performed by inputting the original image, the missing image of the original image, and the missing area information into the combined network, and the mean square error and the weighted error so that the output image approaches the original image. Are applied in order, and the parameters of all networks are updated by the error back propagation method.
Note that only the auxiliary information extraction network may be configured to learn with fixed network parameters.
なお、補助情報抽出ネットワークのみネットワークのパラメータを固定して学習する構成であってもよい。 Specifically, the whole learning process is performed by inputting the original image, the missing image of the original image, and the missing area information into the combined network, and the mean square error and the weighted error so that the output image approaches the original image. Are applied in order, and the parameters of all networks are updated by the error back propagation method.
Note that only the auxiliary information extraction network may be configured to learn with fixed network parameters.
なお、上記の誤差関数の適用順は一例であり、識別ネットワーク9000を含む敵対的生成ネットワークの枠組みを用いずに学習してもよく、識別ネットワーク誤差や平均二乗誤差、もしくは重み付き誤差を、学習の反復回数等に応じて随時変更しながら適用してもよい。
Note that the order of application of the above error functions is an example, and learning may be performed without using the framework of the hostile generation network including the identification network 9000, and the identification network error, the mean square error, or the weighted error is learned. You may apply, changing at any time according to the number of repetitions.
また、敵対的生成ネットワークの枠組みで学習する場合には、図12の各ネットワークの学習処理とは独立に、識別ネットワーク9000を、反復回数や識別ネットワーク9000の正解率に応じて学習する構成であってもよい。
Further, in the case of learning in the framework of the hostile generation network, the identification network 9000 is learned according to the number of iterations and the accuracy rate of the identification network 9000 independently of the learning process of each network in FIG. May be.
識別ネットワーク9000の学習では、例えば図12の各学習処理で用いられるネットワークの出力画像と原画像とを交互に識別ネットワーク9000に入力して、入力が原画像である確率を出力させ、出力と0又は1の正解値との誤差を相互情報量等の誤差関数によって評価して、誤差逆伝播法によりパラメータを更新すればよい。
In learning of the identification network 9000, for example, the network output image and the original image used in each learning process of FIG. 12 are alternately input to the identification network 9000, and the probability that the input is the original image is output. Alternatively, the error from the correct value of 1 may be evaluated by an error function such as a mutual information amount, and the parameters may be updated by the error back propagation method.
また、各学習処理の終了は、反復回数や誤差の減少に対する閾値処理を用いて判定してもよい。なお、処理の単位は、画面全体としてもよいし、HEVCのCTUのような構造を用いて画面を分割したブロック単位としてもよい。
Further, the end of each learning process may be determined by using a threshold process for the number of iterations or a reduction in error. The unit of processing may be the entire screen or may be a block unit obtained by dividing the screen using a structure such as HEVC CTU.
以上説明したように、第3の実施形態における符号化方法及び復号方法は、従来技術における補間ネットワークを復号側に適用し画像生成により出力画像を得る方法とは異なり、画像補間補助情報を用いて画像生成を行う。これにより、第3の実施形態における符号化方法及び復号方法は、従来技術を用いる方法に対して予測精度を向上させることができ、原画の特徴を用いた生成を実現することができる。
As described above, the encoding method and the decoding method in the third embodiment are different from the method of obtaining the output image by generating the image by applying the interpolation network in the prior art to the decoding side, and using the image interpolation auxiliary information. Generate an image. Thereby, the encoding method and the decoding method in the third embodiment can improve the prediction accuracy over the method using the conventional technique, and can realize the generation using the feature of the original picture.
また、第3の実施形態における符号化方法及び復号方法は、送信する画像補間補助情報を学習により決定可能なことから、従来のHEVCのような人手の試行錯誤により決定された画像補間補助情報の抽出に比べて、より高精度な再構成結果が得られる画像補間補助情報を抽出することができる。さらに、第3の実施形態における符号化方法及び復号方法は、ネットワークの学習順序や適用する誤差関数を制御することで、学習すべき複雑な構成のネットワークに対し、意図する動作を各ネットワークに獲得させることができる。
In addition, since the encoding method and the decoding method in the third embodiment can determine the image interpolation auxiliary information to be transmitted by learning, the image interpolation auxiliary information determined by manual trial and error such as conventional HEVC. Compared to the extraction, it is possible to extract image interpolation auxiliary information that can obtain a more accurate reconstruction result. Furthermore, the encoding method and the decoding method according to the third embodiment acquire an intended operation for each network having a complicated configuration to be learned by controlling the network learning order and the error function to be applied. Can be made.
上述した従来技術1では、画像の補間ネットワークを学習により獲得する方法が提案されているが、この補間ネットワークを画像符号化の枠組みにおける復号側に適用する場合、特に広い面積を補間する場合や、補間したい領域が周囲から推論できない程度に複雑な場合に生成精度が低下する。
In the prior art 1 described above, a method of acquiring an image interpolation network by learning has been proposed. However, when this interpolation network is applied to the decoding side in the framework of image encoding, particularly when a large area is interpolated, When the region to be interpolated is so complex that it cannot be inferred from the surroundings, the generation accuracy decreases.
一方、第3の実施形態における符号化方法及び復号方法は、符号化側に補助情報抽出部704を設け、補間ネットワークに画像補間補助情報を与えることでこれを解決する。また、このとき、画像補間補助情報を定義する補助情報抽出ネットワーク7041も学習により獲得することで、第3の実施形態における符号化方法及び復号方法は、HEVC等の画像符号化のように、人手で設計した画像補間補助情報に比べて、画像生成の精度がより高まる画像補間補助情報を抽出することができる。
On the other hand, the encoding method and decoding method in the third embodiment solve this problem by providing an auxiliary information extraction unit 704 on the encoding side and providing image interpolation auxiliary information to the interpolation network. At this time, the auxiliary information extraction network 7041 that defines the image interpolation auxiliary information is also acquired by learning, so that the encoding method and the decoding method in the third embodiment can be performed manually like image encoding such as HEVC. As compared with the image interpolation auxiliary information designed in the above, image interpolation auxiliary information with higher image generation accuracy can be extracted.
第3の実施形態における符号化方法及び復号方法の構成は、画像補間補助情報を生成する補助情報抽出部704も含めて、学習によりネットワークのパラメータを獲得させることから、補助情報抽出部704及び画像補間部803を同時に学習した場合に、各ネットワークに意図する動作を学習させることが難しい。特に、敵対的生成ネットワークの枠組みを用いた場合は、学習の調整が難しいためこの傾向は顕著となる。
Since the configuration of the encoding method and the decoding method in the third embodiment includes the auxiliary information extraction unit 704 that generates image interpolation auxiliary information and acquires network parameters by learning, the auxiliary information extraction unit 704 and the image When the interpolating unit 803 learns simultaneously, it is difficult for each network to learn the intended operation. In particular, when using the framework of the hostile generation network, this tendency becomes remarkable because it is difficult to adjust learning.
しかしながら、第3の実施形態における符号化方法及び復号方法では、補助情報抽出部704及び画像補間部803を、役割毎のネットワークに分割し、学習の反復回数によって学習対象とするネットワーク及び適用する誤差関数を制御することで、各ネットワークに意図する動作を獲得させることができる。
However, in the encoding method and decoding method according to the third embodiment, the auxiliary information extraction unit 704 and the image interpolation unit 803 are divided into networks for each role, and the network to be learned and the error to be applied according to the number of learning iterations. By controlling the function, each network can acquire an intended operation.
<第4の実施形態>
以下、第4の実施形態について、図面を参照しながら説明する。 <Fourth Embodiment>
Hereinafter, a fourth embodiment will be described with reference to the drawings.
以下、第4の実施形態について、図面を参照しながら説明する。 <Fourth Embodiment>
Hereinafter, a fourth embodiment will be described with reference to the drawings.
第4の実施形態は、第3の実施形態と,補助情報抽出部及び画像補間部のネットワークの構成が異なり、画像補間補助情報を、欠損画像参照ネットワークの出力と入力画像の差分とから生成する。
The fourth embodiment differs from the third embodiment in the configuration of the network of the auxiliary information extraction unit and the image interpolation unit, and generates image interpolation auxiliary information from the output of the missing image reference network and the difference between the input images. .
第4の実施形態におけるネットワークの構成を、図15に示す。図示するように、補助情報抽出部704は、補助情報抽出ネットワーク7041と、画像補間部803と共通のネットワークのパラメータを用いた欠損画像参照ネットワーク8032と、から構成される。
FIG. 15 shows a network configuration in the fourth embodiment. As illustrated, the auxiliary information extraction unit 704 includes an auxiliary information extraction network 7041 and a missing image reference network 8032 using network parameters common to the image interpolation unit 803.
補助情報抽出ネットワーク7041は、入力画像と欠損画像参照による中間画像の差分、及び欠損領域情報を入力として、画像補間補助情報を出力するネットワークである。補助情報抽出ネットワーク7041は、例えば入力を入力画像と欠損画像参照による中間画像との差分画像、及び欠損領域マスク画像の2枚の画像とし、出力を任意の数のユニットとして、畳み込み層及び全結合層等により中間層を構成する。
The auxiliary information extraction network 7041 is a network that outputs the image interpolation auxiliary information by using the difference between the input image and the intermediate image based on the missing image and the missing area information as inputs. The auxiliary information extraction network 7041 has, for example, a difference image between an input image and an intermediate image by referring to a missing image, and two images of a missing region mask image as an input, an output as an arbitrary number of units, a convolution layer, and a fully connected An intermediate layer is constituted by layers and the like.
図15に示すように、画像補間部803は、補助情報参照ネットワーク8031と、欠損画像参照ネットワーク8032と、再構成ネットワーク8033と、から構成される。
これら各ネットワークの入出力は、欠損画像参照ネットワーク8032を除き第3の実施形態と共通である。 As illustrated in FIG. 15, theimage interpolation unit 803 includes an auxiliary information reference network 8031, a missing image reference network 8032, and a reconstruction network 8033.
The input / output of each network is the same as that of the third embodiment except for the missingimage reference network 8032.
これら各ネットワークの入出力は、欠損画像参照ネットワーク8032を除き第3の実施形態と共通である。 As illustrated in FIG. 15, the
The input / output of each network is the same as that of the third embodiment except for the missing
補助情報参照ネットワーク8031は、画像補間補助情報を入力として、補助情報参照による中間画像を出力するネットワークである。
The auxiliary information reference network 8031 is a network that receives the image interpolation auxiliary information and outputs an intermediate image by referring to the auxiliary information.
欠損画像参照ネットワーク8032は、入力画像の欠損画像と欠損領域マスク画像とを入力として、欠損画像参照による中間画像を出力するネットワークである。
The missing image reference network 8032 is a network that outputs the intermediate image based on the missing image by using the missing image of the input image and the missing area mask image as inputs.
欠損画像参照による中間画像は、画像補間部803の構成要素として、再構成ネットワーク8033に入力される。また、欠損画像参照による中間画像と入力画像との差分が、補助情報抽出部704の構成要素として、補助情報抽出ネットワーク7041に入力される。
The intermediate image based on the missing image reference is input to the reconstruction network 8033 as a component of the image interpolation unit 803. Further, the difference between the intermediate image and the input image based on the missing image reference is input to the auxiliary information extraction network 7041 as a component of the auxiliary information extraction unit 704.
再構成ネットワーク8033は、補助情報参照による中間画像と欠損画像参照による中間画像とを入力として、欠損領域が補間された最終的な出力画像を出力するネットワークである。
The reconstruction network 8033 is a network that receives an intermediate image based on auxiliary information reference and an intermediate image based on missing image reference, and outputs a final output image in which a missing area is interpolated.
以上の構成によって、補助情報抽出部704及び画像補間部803の学習が行われる。
なお、学習の処理は第3の実施形態と共通であるが、補助情報抽出/参照ネットワーク学習処理におけるネットワークの構成が図16のようになる。当該処理では、図16の構成で、補助情報抽出ネットワーク7041及び補助情報参照ネットワーク8031のみの学習が行われる。 With the above configuration, learning of the auxiliaryinformation extraction unit 704 and the image interpolation unit 803 is performed.
The learning process is the same as in the third embodiment, but the network configuration in the auxiliary information extraction / reference network learning process is as shown in FIG. In this process, only the auxiliaryinformation extraction network 7041 and the auxiliary information reference network 8031 are learned in the configuration of FIG.
なお、学習の処理は第3の実施形態と共通であるが、補助情報抽出/参照ネットワーク学習処理におけるネットワークの構成が図16のようになる。当該処理では、図16の構成で、補助情報抽出ネットワーク7041及び補助情報参照ネットワーク8031のみの学習が行われる。 With the above configuration, learning of the auxiliary
The learning process is the same as in the third embodiment, but the network configuration in the auxiliary information extraction / reference network learning process is as shown in FIG. In this process, only the auxiliary
以上説明したように、第4の実施形態による補助情報抽出部704は、第3の実施形態のように原画像を直接入力とすることもできるが、上述したように復号側と符号化側で周辺ブロックからの予測結果(欠損画像参照による中間画像)を共通にするという前提を置くことによって、原画像と周辺ブロックからの予測画像との差分画像を入力とすることができる。これにより、画像補間部803の出力画像が原画像から離れすぎないようにする制約を明示的に導入することができ、補間結果の主観品質が向上する。
As described above, the auxiliary information extraction unit 704 according to the fourth embodiment can directly input the original image as in the third embodiment, but as described above, the auxiliary information extraction unit 704 can perform the decoding on the decoding side and the encoding side. By assuming that the prediction result from the peripheral block (intermediate image by referring to the missing image) is shared, a difference image between the original image and the predicted image from the peripheral block can be input. Thereby, it is possible to explicitly introduce a constraint that the output image of the image interpolation unit 803 is not too far from the original image, and the subjective quality of the interpolation result is improved.
上述した実施形態における符号化装置及び復号装置の一部又は全部をコンピュータで実現するようにしてもよい。その場合、この機能を実現するためのプログラムをコンピュータ読み取り可能な記録媒体に記録して、この記録媒体に記録されたプログラムをコンピュータシステムに読み込ませ、実行することによって実現してもよい。なお、ここでいう「コンピュータシステム」とは、OSや周辺機器等のハードウェアを含むものとする。また、「コンピュータ読み取り可能な記録媒体」とは、フレキシブルディスク、光磁気ディスク、ROM、CD-ROM等の可搬媒体、コンピュータシステムに内蔵されるハードディスク等の記憶装置のことをいう。さらに「コンピュータ読み取り可能な記録媒体」とは、インターネット等のネットワークや電話回線等の通信回線を介してプログラムを送信する場合の通信線のように、短時間の間、動的にプログラムを保持するもの、その場合のサーバやクライアントとなるコンピュータシステム内部の揮発性メモリのように、一定時間プログラムを保持しているものも含んでもよい。また上記プログラムは、前述した機能の一部を実現するためのものであってもよく、さらに前述した機能をコンピュータシステムにすでに記録されているプログラムとの組み合わせで実現できるものであってもよく、FPGA(Field Programmable Gate Array)等のプログラマブルロジックデバイスを用いて実現されるものであってもよい。
A part or all of the encoding device and the decoding device in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. It may be realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).
以上、この発明の実施形態について図面を参照して詳述してきたが、具体的な構成はこの実施形態に限られるものではなく、この発明の要旨を逸脱しない範囲の設計等も含まれる。
As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design and the like within the scope not departing from the gist of the present invention.
10,30…符号化装置、101,301…ブロック分割部、102,302…符号化方式判定部、103,303…補助情報抽出部、104.304…補助情報エントロピー符号化部、105,305…画像変換部、306…変換後画像メモリ、107,307…イントラ予測部、108,308…変換/量子化部、109,309…エントロピー符号化部、110,310…逆量子化/逆変換部、111,311…予測用メモリ、20…復号装置、201,401…エントロピー復号部、202,402…逆変換/逆量子化部、203,403…イントラ予測部、204,404…予測用メモリ、205,405…再構成部、206,406…符号化方式判定部、207,407…補助情報エントロピー復号部、408…出力ピクチャメモリ、50,70…符号化装置、501,701…画像欠損処理部、502,702…画像符号化部、503,703…欠損領域情報符号化部、704…補助情報抽出部、7041…補助情報抽出ネットワーク、705…補助情報符号化部、60,80…復号装置、601,801…画像復号部、602,802…欠損領域情報復号部、603,803…画像補間部、8031…補助情報参照ネットワーク、8032…欠損画像参照ネットワーク、8033…再構成ネットワーク、604…画像補間ネットワーク、805…補助情報復号部、9000…識別ネットワーク
DESCRIPTION OF SYMBOLS 10,30 ... Coding apparatus, 101, 301 ... Block division part, 102, 302 ... Coding system determination part, 103, 303 ... Auxiliary information extraction part, 104.304 ... Auxiliary information entropy coding part, 105, 305 ... Image conversion unit, 306 ... post-conversion image memory, 107, 307 ... intra prediction unit, 108, 308 ... transformation / quantization unit, 109, 309 ... entropy coding unit, 110, 310 ... inverse quantization / inverse transformation unit, 111, 311 ... Prediction memory, 20 ... Decoding device, 201, 401 ... Entropy decoding unit, 202, 402 ... Inverse transformation / inverse quantization unit, 203, 403 ... Intra prediction unit, 204, 404 ... Prediction memory, 205 405: Reconstruction unit 206, 406 Coding method determination unit 207 407 Auxiliary information entropy decoding unit 408 Output picture memory 50, 70 ... coding device, 501, 701 ... image loss processing unit, 502, 702 ... image coding unit, 503, 703 ... missing region information coding unit, 704 ... auxiliary information extraction unit, 7041 ... auxiliary information Extraction network, 705 ... auxiliary information encoding unit, 60, 80 ... decoding device, 601, 801 ... image decoding unit, 602, 802 ... missing region information decoding unit, 603, 803 ... image interpolation unit, 8031 ... auxiliary information reference network , 8032 ... Missing image reference network, 8033 ... Reconstruction network, 604 ... Image interpolation network, 805 ... Auxiliary information decoding unit, 9000 ... Identification network
Claims (8)
- 画像を符号化する符号化装置であって、
入力された画像を再構成対象とするか否かを判定する判定部と、
前記再構成対象とすると判定された画像から、再構成に使うための情報である補助情報を抽出する補助情報抽出部と、
前記再構成対象とすると判定された画像を変換し変換画像を得る変換部と、
前記変換画像を符号化し符号化データを得る符号化部と、
を備え、
前記変換部は、前記符号化部が符号化する際、前記入力された画像を符号化した場合よりも少ない符号量になるよう変換する
符号化装置。 An encoding device for encoding an image, comprising:
A determination unit that determines whether the input image is to be reconstructed;
An auxiliary information extraction unit that extracts auxiliary information, which is information for use in reconstruction, from the image determined to be the reconstruction target;
A conversion unit that converts the image determined to be the reconstruction target and obtains a converted image;
An encoding unit that encodes the converted image to obtain encoded data;
With
The conversion unit performs conversion such that when the encoding unit encodes, the code amount is smaller than when the input image is encoded. - 前記判定部は、推定発生符号量及び推定歪量を取得してレート歪最適化を行うことによって前記入力された画像を前記再構成対象とするか否かを判定する
請求項1に記載の符号化装置。 The code according to claim 1, wherein the determination unit determines whether or not the input image is to be reconstructed by acquiring an estimated generated code amount and an estimated distortion amount and performing rate distortion optimization. Device. - 前記補助情報は、前記変換画像を、前記再構成対象とすると判定された画像の特徴を保ちつつ、前記再構成対象とすると判定された画像よりも少ない符号量の画像に逆変換するための情報である
請求項1又は請求項2に記載の符号化装置。 The auxiliary information is information for inversely converting the converted image into an image having a smaller code amount than the image determined to be the reconstruction target while maintaining the characteristics of the image determined to be the reconstruction target. The encoding device according to claim 1 or 2. - 画像が符号化された符号化データを復号する復号装置であって、
入力された符号化データを復号し復号画像を得る復号部と、
前記復号画像が再構成対象の画像であるか否かを判定する判定部と、
再構成に使うための情報である補助情報を取得し、前記再構成対象の画像であると判定された復号画像を、前記補助情報を用いて再構成する再構成部と、
を備える復号装置。 A decoding device for decoding encoded data in which an image is encoded,
A decoding unit that decodes input encoded data and obtains a decoded image;
A determination unit that determines whether or not the decoded image is an image to be reconstructed;
A reconstruction unit that obtains auxiliary information that is information for use in reconstruction, and reconstructs a decoded image determined to be the image to be reconstructed using the auxiliary information;
A decoding device comprising: - 画像を符号化する符号化装置による符号化方法であって、
入力された画像を再構成対象とするか否かを判定する判定ステップと、
前記再構成対象とすると判定された画像から、再構成に使うための情報である補助情報を抽出する補助情報抽出ステップと、
前記再構成対象とすると判定された画像を、前記入力された画像を符号化した場合よりも少ない符号量になるよう変換し変換画像を得る変換ステップと、
前記変換画像を符号化し符号化データを得る符号化ステップと、
を有する符号化方法。 An encoding method by an encoding device for encoding an image,
A determination step of determining whether or not the input image is to be reconstructed;
Auxiliary information extraction step for extracting auxiliary information that is information for use in reconstruction from the image determined to be the reconstruction target;
A conversion step of converting the image determined to be the reconstruction target into a code amount smaller than that when the input image is encoded to obtain a converted image;
An encoding step of encoding the converted image to obtain encoded data;
An encoding method comprising: - 画像が符号化された符号化データを復号する復号装置による復号方法であって、
入力された符号化データを復号し復号画像を得る復号ステップと、
前記復号画像が再構成対象の画像であるか否かを判定する判定ステップと、
再構成に使うための情報である補助情報を取得し、前記再構成対象の画像であると判定された復号画像を、前記補助情報を用いて再構成する再構成ステップと、
を有する復号方法。 A decoding method by a decoding device for decoding encoded data in which an image is encoded,
A decoding step of decoding the input encoded data to obtain a decoded image;
A determination step of determining whether or not the decoded image is an image to be reconstructed;
A reconstruction step of acquiring auxiliary information that is information for use in reconstruction, and reconstructing a decoded image determined to be the image to be reconstructed using the auxiliary information;
A decryption method. - 請求項1から請求項3のうちいずれか一項に記載の符号化装置としてコンピュータを機能させるための符号化プログラム。 An encoding program for causing a computer to function as the encoding device according to any one of claims 1 to 3.
- 請求項4に記載の復号装置としてコンピュータを機能させるための復号プログラム。 A decoding program for causing a computer to function as the decoding device according to claim 4.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2018-097427 | 2018-05-21 | ||
JP2018097427A JP2019205011A (en) | 2018-05-21 | 2018-05-21 | Encoding device, decoding device, encoding method, decoding method, encoding program, and decoding program |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019225337A1 true WO2019225337A1 (en) | 2019-11-28 |
Family
ID=68615960
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2019/018568 WO2019225337A1 (en) | 2018-05-21 | 2019-05-09 | Encoding device, decoding device, encoding method, decoding method, encoding program and decoding program |
Country Status (2)
Country | Link |
---|---|
JP (1) | JP2019205011A (en) |
WO (1) | WO2019225337A1 (en) |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2009157904A1 (en) * | 2008-06-27 | 2009-12-30 | Thomson Licensing | Methods and apparatus for texture compression using patch-based sampling texture synthesis |
WO2017183479A1 (en) * | 2016-04-22 | 2017-10-26 | ソニー株式会社 | Encoding device and encoding method, and decoding device and decoding method |
-
2018
- 2018-05-21 JP JP2018097427A patent/JP2019205011A/en active Pending
-
2019
- 2019-05-09 WO PCT/JP2019/018568 patent/WO2019225337A1/en active Application Filing
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2009157904A1 (en) * | 2008-06-27 | 2009-12-30 | Thomson Licensing | Methods and apparatus for texture compression using patch-based sampling texture synthesis |
WO2017183479A1 (en) * | 2016-04-22 | 2017-10-26 | ソニー株式会社 | Encoding device and encoding method, and decoding device and decoding method |
Non-Patent Citations (3)
Title |
---|
ADRIANA DUMITRAS ET AL.: "An Encoder-Decoder Texture Replacement Method With Application to Content-Based Movie Coding", IEEE TRANSACTION ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, vol. 14, no. 6, June 2004 (2004-06-01), pages 825 - 840, XP001196870, DOI: 10.1109/TCSVT.2004.828336 * |
BASTIAN WANDT ET AL.: "Extending HEVC Using Texture Synthesis", 2017 IEEE VISUAL COMMUNICATIONS AND IMAGE PROCESSING (VCIP, 4 December 2017 (2017-12-04), XP033325714 * |
DIMITAR DOSHKOV ET AL.: "IDENTIFICATION AND DISCUSSION OF OPEN ISSUES IN PERCEPTUAL VIDEO CODING BASED ON IMAGE ANALYSIS AND COMPLETION", IVMSP 2011, pages 30 - 35, XP031912076 * |
Also Published As
Publication number | Publication date |
---|---|
JP2019205011A (en) | 2019-11-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
KR102287947B1 (en) | Apparatus and method for performing artificial intelligence encoding and artificial intelligence decoding of image | |
US11159789B2 (en) | Generative adversarial network based intra prediction for video coding | |
JP7047119B2 (en) | Methods and equipment for residual code prediction in the conversion region | |
RU2492586C2 (en) | Method for automatic formation of procedure of generating predicted pixel value, image encoding method, mage decoding method, corresponding device, corresponding programmes and data media storing programmes | |
CN110383695B (en) | Method and apparatus for encoding and decoding digital image or video stream | |
JP6356913B2 (en) | Method and apparatus for decoding / encoding a video signal using a transformation derived from a graph template | |
JP6042899B2 (en) | Video encoding method and device, video decoding method and device, program and recording medium thereof | |
JP2019062576A (en) | Method for encoding current block of first image component relative to reference block of at least one second image component, encoding device, and corresponding computer program | |
WO2017023152A1 (en) | Device and method for performing transform by using singleton coefficient update | |
JP2010534015A (en) | Image processing method and corresponding electronic device | |
KR102012906B1 (en) | Systems and methods for processing a block of a digital image | |
WO2020261314A1 (en) | Image encoding method and image decoding method | |
CN116438796A (en) | Image prediction method, encoder, decoder, and computer storage medium | |
KR102648464B1 (en) | Method and apparatus for image enhancement using supervised learning | |
CN112601095B (en) | Method and system for creating fractional interpolation model of video brightness and chrominance | |
WO2019225344A1 (en) | Encoding device, image interpolation system and encoding program | |
US10869030B2 (en) | Method of coding and decoding images, a coding and decoding device, and corresponding computer programs | |
US20240163485A1 (en) | Multi-distribution entropy modeling of latent features in image and video coding using neural networks | |
WO2019225337A1 (en) | Encoding device, decoding device, encoding method, decoding method, encoding program and decoding program | |
CN107409216B (en) | Image encoding and decoding method, encoding and decoding device and corresponding computer program | |
CN116438567A (en) | In-loop filtering based on neural network for video coding | |
JP7303456B2 (en) | Encoding device, encoding method and program | |
US20180070109A1 (en) | Encoding of images by vector quantization | |
JP2011109390A (en) | Device and method for encoding image, and device and method for decoding image | |
JP2010041161A (en) | Image decoder, image decoding method, image encoder, and image encoding method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19807312 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19807312 Country of ref document: EP Kind code of ref document: A1 |